View Issue Details

IDProjectCategoryView StatusLast Update
0005299unrealtestspublic2019-11-27 10:44
Reportersyzop Assigned Tosyzop  
PrioritynormalSeverityfeatureReproducibilityN/A
Status resolvedResolutionfixed 
Fixed in Version5.0.0-rc2 
Summary0005299: U5: different test framework
DescriptionI hate our current test framework, since the beginning already. The only good thing about it is that we have SOME test framework :D

So... switch to something different.
The things I dislike most:
1) The auto-synchronization idea is missing, even though I provided this before the 1st test framework was coded :(. This results in lots of sleep() at various places. Causes unnecessary slowdown and random test failures if you miss cases. Very annoying.
2) Ruby... argh !

And... when looking at something new.... have a look at msgid and such extraction / comparison tests.

I'm thinking we probably will use some low level approach (more low level sockets) rather than the high level approach of the current test frame network (which uses an IRC library).
TagsNo tags attached.
3rd party modules

Relationships

child of 0005279 closedsyzop UnrealIRCd 5 master tracking issue 

Activities

syzop

2019-05-30 16:54

administrator   ~0020713

Wishlist:
* Auto syncher
* Parallel tasks with random nicks and channels (and some flag if it cannot run in parallel, or if that is too difficult then just split everything in two: parallel and non-parallel)
* msgid/timestamp/account comparisson across servers (and always expect it in channel events.. etc...)

Python would be preferred, PHP would be the lazy option

syzop

2019-06-01 07:59

administrator   ~0020717

I worked on this yesterday and it's looking nice. Auto synchronization works, across multiple servers even. Parallelization is no problem, all random based nicks and chans etc.
Next few days I probably won't do much work due to the nice weather (:D) but basically:
1) Clean up a bit
2) The parallel main thing is currently a bash script, perhaps this should be python too so we can more easily run on Windows
3) msgid/timestamp/account comparisson accross servers -> this can be added later tho
Main thing :( :
4) Write all tests to use this new framework

syzop

2019-06-03 13:41

administrator   ~0020719

The old test framework (ruby, by Heero) has been removed in U5 and the new one has been added (python, by me).
The new test framework currently has about 25% of the tests that the old framework had. More will follow soon...
TODO: run on Windows, currently uses a bash script with xargs

syzop

2019-06-24 15:04

administrator   ~0020741

Last edited: 2019-11-18 08:57

A lot more tests have been added. Some that existed in the previous test framework and some new.
Test framework now run on Windows too.

Some tests from the previous test framework still need to be migrated. Off the top of my head these are: extended bans and oper commands

syzop

2019-08-10 11:30

administrator   ~0020799

Still some tests needs to be migrated, indeed.

syzop

2019-11-11 17:47

administrator   ~0021071

Last edited: 2019-11-11 17:47

I added another ~25 tests, from scratch actually, forgot that I could look at the old tests :D.
Nearly all user commands are now tested.

Oper commands still need to be done. Some other areas as well probably, but not much, mostly oper commands. Oh yeah and extbans.

syzop

2019-11-18 08:55

administrator   ~0021075

I will have a go at including services in the tests, anope and atheme. Will probably run the tests twice then but that's OK.
Then at least we can test some basic functionality with services, something that is currently not done at all... and I normally don't use 1 of the 2 services brands myself which doesn't help either..

syzop

2019-11-27 10:44

administrator   ~0021120

Lots of tests were added in rc2, including the remaining things mentioned in the last notes. This can be closed, even though it is - of course - an always ongoing effort :)

Issue History

Date Modified Username Field Change
2019-05-25 08:26 syzop New Issue
2019-05-25 08:27 syzop Relationship added child of 0005279
2019-05-25 08:27 syzop Assigned To => syzop
2019-05-25 08:27 syzop Status new => acknowledged
2019-05-30 16:54 syzop Note Added: 0020713
2019-06-01 07:59 syzop Note Added: 0020717
2019-06-03 13:41 syzop Note Added: 0020719
2019-06-24 15:04 syzop Note Added: 0020741
2019-08-10 11:30 syzop Note Added: 0020799
2019-08-18 15:17 syzop View Status private => public
2019-11-11 17:47 syzop Note Added: 0021071
2019-11-11 17:47 syzop Note Edited: 0021071
2019-11-18 08:55 syzop Note Added: 0021075
2019-11-18 08:56 syzop Note Edited: 0020741
2019-11-18 08:57 syzop Note Edited: 0020741
2019-11-27 10:44 syzop Status acknowledged => resolved
2019-11-27 10:44 syzop Resolution open => fixed
2019-11-27 10:44 syzop Fixed in Version => 5.0.0-rc2
2019-11-27 10:44 syzop Note Added: 0021120