0002943: ircd logging: use fflush instead of fopen+fclose all the time

(0011828)
syzop (administrator)
2006-06-03 07:29

Logging never showed any (significant) cpu impact in my speedresults. So if there was any cpu increase or spike or even 99% cpu usage that must be some kind of bug.

I too, have wondered why we do the close+open thing all the time instead of fflush(), maybe the author was lazy. Though the (only?) good thing about it is that you don't have to worry if someone rm's the file ;)

(0011840)
codemastr (reporter)
2006-06-04 13:15

If I recall, it was because of Windows. I seem to remember there having been an issue where if we didn't fclose(), and I tried to open the file in notepad, it would give one of those fun "file is in use" errors but I think that was only a Win 9x issue so it could easily be changed now.

(0011844)
stskeeps (reporter)
2006-06-04 16:07

New material which has gotten us to realize what the problem is

// block repeting letters
spamfilter {
        regex "^.*(\w)\1{15,}.*$";
        target channel;
        action block;
        reason "Chars flood";
};

This, with a normal user flooding and moderate server traffic, causes the ircd to go high up in CPU (15%, where normal traffic is 1-2%. A ltrace is at http://bsd.tspre.org/~stskeeps/ltrace.out.gz [^] ). With a good amount of users doing the same, it'd reach 99%

(0011845)
codemastr (reporter)
2006-06-04 16:41

Backreferences are an NP-complete problem. That kinda makes it inefficient by definition, doesn't it?

(0011846)
syzop (administrator)
2006-06-04 16:42
edited on: 2006-06-04 16:45

codemastr: ack :P

stskeeps: yeah regexes can be a pain, especially with backrefs.. the TRE author even said to me "if you care about speed, don't use backreferences" ;P

EDIT: *grin* timing ;p

(0011855)
stskeeps (reporter)
2006-06-05 04:32
edited on: 2006-06-05 04:33

Well, true, but it would be a hell lot more efficient if it didn't go out and do a strncmp to compare one character.. and just did a char comparision instead :>

(0011863)
syzop (administrator)
2006-06-05 16:36

So..... now this bugreport is screwed up.. what to do about it...

Since we cannot do anything about regex, and the TRE author probably can't either, I suppose I'll keep it as-is for the fflush-logging tactic, and rename it to that ;p

(0011869)
codemastr (reporter)
2006-06-05 21:54

[quote]Well, true, but it would be a hell lot more efficient if it didn't go out and do a strncmp to compare one character.. and just did a char comparision instead :>[/quote]
Well, you know C, stop being lazy and go make a patch :P

(0011879)
syzop (administrator)
2006-06-06 10:17

Let's move this regex talk to 0002887, even though that is about ++ :P

(0014398)
stskeeps (reporter)
2007-06-21 15:43

Fixed in .2446

Issue History
Date Modified	Username	Field	Change
2006-06-03 07:25	stskeeps	New Issue
2006-06-03 07:29	syzop	Note Added: 0011828
2006-06-03 07:41	syzop	Relationship added	child of 0002936
2006-06-04 13:15	codemastr	Note Added: 0011840
2006-06-04 13:49	stskeeps	Note Added: 0011843
2006-06-04 16:07	stskeeps	Note Added: 0011844
2006-06-04 16:41	codemastr	Note Added: 0011845
2006-06-04 16:42	syzop	Note Added: 0011846
2006-06-04 16:45	syzop	Note Edited: 0011846
2006-06-05 04:32	stskeeps	Note Added: 0011855
2006-06-05 04:33	stskeeps	Note Edited: 0011855
2006-06-05 16:36	syzop	Note Added: 0011863
2006-06-05 16:38	syzop	Summary	Problem with 99% cpu with connection logging enabled => ircd logging: use fflush instead of fopen+fclose all the time
2006-06-05 16:38	syzop	Description Updated
2006-06-05 16:38	syzop	Additional Information Updated
2006-06-05 21:54	codemastr	Note Added: 0011869
2006-06-06 10:17	syzop	Note Added: 0011879
2006-11-03 17:40	syzop	Relationship added	child of 0003049
2006-11-03 17:40	syzop	Relationship deleted	child of 0002936
2007-04-27 03:25	stskeeps	Note Deleted: 0011843
2007-04-27 03:25	stskeeps	Status	new => acknowledged
2007-05-19 14:38	Trocotronic	Status	acknowledged => assigned
2007-05-19 14:38	Trocotronic	Assigned To	=> Trocotronic
2007-06-11 12:50	stskeeps	Assigned To	Trocotronic =>
2007-06-11 12:50	stskeeps	Status	assigned => acknowledged
2007-06-21 15:43	stskeeps	Status	acknowledged => resolved
2007-06-21 15:43	stskeeps	Fixed in Version	=> 3.3-alpha0
2007-06-21 15:43	stskeeps	Resolution	open => fixed
2007-06-21 15:43	stskeeps	Assigned To	=> stskeeps
2007-06-21 15:43	stskeeps	Note Added: 0014398

Notes
(0011828) syzop (administrator) 2006-06-03 07:29	Logging never showed any (significant) cpu impact in my speedresults. So if there was any cpu increase or spike or even 99% cpu usage that must be some kind of bug. I too, have wondered why we do the close+open thing all the time instead of fflush(), maybe the author was lazy. Though the (only?) good thing about it is that you don't have to worry if someone rm's the file ;)

(0011840) codemastr (reporter) 2006-06-04 13:15	If I recall, it was because of Windows. I seem to remember there having been an issue where if we didn't fclose(), and I tried to open the file in notepad, it would give one of those fun "file is in use" errors but I think that was only a Win 9x issue so it could easily be changed now.

(0011844) stskeeps (reporter) 2006-06-04 16:07	New material which has gotten us to realize what the problem is // block repeting letters spamfilter { regex "^.(\w)\1{15,}.$"; target channel; action block; reason "Chars flood"; }; This, with a normal user flooding and moderate server traffic, causes the ircd to go high up in CPU (15%, where normal traffic is 1-2%. A ltrace is at http://bsd.tspre.org/~stskeeps/ltrace.out.gz [^] ). With a good amount of users doing the same, it'd reach 99%

(0011845) codemastr (reporter) 2006-06-04 16:41	Backreferences are an NP-complete problem. That kinda makes it inefficient by definition, doesn't it?

(0011846) syzop (administrator) 2006-06-04 16:42 edited on: 2006-06-04 16:45	codemastr: ack :P stskeeps: yeah regexes can be a pain, especially with backrefs.. the TRE author even said to me "if you care about speed, don't use backreferences" ;P EDIT: grin timing ;p

(0011855) stskeeps (reporter) 2006-06-05 04:32 edited on: 2006-06-05 04:33	Well, true, but it would be a hell lot more efficient if it didn't go out and do a strncmp to compare one character.. and just did a char comparision instead :>

(0011863) syzop (administrator) 2006-06-05 16:36	So..... now this bugreport is screwed up.. what to do about it... Since we cannot do anything about regex, and the TRE author probably can't either, I suppose I'll keep it as-is for the fflush-logging tactic, and rename it to that ;p

(0011869) codemastr (reporter) 2006-06-05 21:54	[quote]Well, true, but it would be a hell lot more efficient if it didn't go out and do a strncmp to compare one character.. and just did a char comparision instead :>[/quote] Well, you know C, stop being lazy and go make a patch :P

(0011879) syzop (administrator) 2006-06-06 10:17	Let's move this regex talk to 0002887, even though that is about ++ :P

(0014398) stskeeps (reporter) 2007-06-21 15:43	Fixed in .2446