View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0001978unrealircdpublic2004-07-18 22:312015-07-23 22:26
ReportercodemastrAssigned Tosyzop  
PrioritynormalSeverityfeatureReproducibilityalways
Status closedResolutionfixed 
Fixed in Version3.4-alpha3 
Summary0001978: Saving module data during a /rehash
DescriptionI'm just wondering if anyone can come up with any good way to store module data during a /rehash. The problem is, for example, with usermodes. When you /rehash, the module is unloaded, then loaded again. So, what do we do about usermodes? We have to store the fact that the usermode is in use. Currently, I do this by simply having a 1/0 to indicate used/unused. That's not really that great though because, say the mode is +Y. Well you unloaded MyModeY and instead loaded YourModeY which uses +Y for a different purpose. So you might not want each user to remain +Y. So we need a better way to determine which info is still "in use" after a /rehash.

Additionally, I'd like to allow modules to store info specific to a user/channel. So we'd need some way to maintain that between rehashes. Since we really know nothing about what the data is, I was kinda thinking something like a database file. We write it out to file at /rehash, then when the new module is loaded, it reads it back in. That could work, but requires a bunch of work on the module coders part because they have to implement the code to write the data to a file.

I'm just wondering if anyone else can come up with a better solution to this...
TagsNo tags attached.
3rd party modules

Relationships

related to 0002153 closed custom var for Client and Channel (maybe others) 

Activities

syzop

2004-07-18 22:41

administrator   ~0007151

Last edited: 2004-07-18 22:46

 @usermodes.. note that it currently doesn't work at all due to bug 0001855 :[

@chan/user data.. yeah it's fun. especially if some module would change structs or something.
In my Monitor module I use the 'temporary file' aproach you just described, which works fine, but is indeed a mess to code :P.

When we are at it.. I have been thinking of the following feature in the past: when a loadmodule line for a PERM module is removed from the configuration file make it unload that mod. This would make it possible to 'upgrade' PERM mods without rebooting.
Usually the reason one has to make modules permanent is because data needs to be kept saved during rehash (whatever data it is.. usually chan/user data indeed, sometimes other stuff).
Perhaps that is an easier (and more simple) aproach to all this? Although granted it would require some more steps when upgrading a module (comment out loadmodule line, rehash, uncomment loadmodule line, rehash).
*edit* or make some /rehash flag to reload a particular existing module.. a la '/rehash -reloadmod src/modules/bla.so' or whateverrrrrrrr ;p */edit*

edited on: 2004-07-18 22:46

codemastr

2004-07-18 22:49

reporter   ~0007152

Last edited: 2004-07-18 22:51

 Well that's always something we can fallback to, but if we can, I'd like to avoid permanent modules altogether :P But, it also doesn't really solve all the problems. Meaning, umodes would still need to be reset. Because say perm mod = MyModeY and I remove the loadmodule line and add one for YourModeY, same problem.

For usermodes I was thinking the most accurate way to do it would be to store the name of the module that used it previously. It'd use a bit more memory, but it's not a big deal because the memory would be freed as soon as the /rehash finished. I could probably pull off something similar for cmodes/extbans.

For the module data, I was thinking something like:
ChannelDataCreate(free_function, serialize_function);
Then, you'd just implement those two functions. Unreal would handle telling it which file to write, etc. Then we could have some function that the module would call after the /rehash to get the name of the file the data was saved to and read it back in. I can't imagine that being too difficult. The thing is, remote includes make it a bit difficult. /rehash is reentrant. Meaning, if UserA quits, and a new UserA joins, we have to make sure not to load the data into the new user. So we'd need to make sure the file is timestamped and we don't load data for users who connected after that timestamp.

It certainly would be fun to try and implement this :)

*edit:
Yeah the /rehash -module type thing does sound like a better idea than removing the loadmodule line. Though it still would have some issues, especially with module data. Say v1.0 stores struct { char *someinfo; }; v1.1 stores struct { char *someinfo; int someflags; }; would cause some nice errors if the data wasn't freed/reloaded ;)

edited on: 2004-07-18 22:51

syzop

2004-07-18 23:12

administrator   ~0007153

> ChannelDataCreate(free_function, serialize_function);
Then, you'd just implement those two functions. Unreal would handle telling it which file to write, etc. Then we could have some function that the module would call after the /rehash to get the name of the file the data was saved to and read it back in. I can't imagine that being too difficult.

I think it would be nice if as much as possible is done by unreal / "abstracted". Obviously we will have to keep the encoding/whatever of the data up to the module but.. it would be nice if we could help them as much as possible.
I was also more thinking about storing this stuff in memory.
I know you are a low-memory freak ;p.. But I look at it like this: an IRC server should be able to handle like 80-200 mb of memory peaks due to floods (for buffers/sendq and stuff). The only memory peak this would generate is for the structs that get saved during rehash which I doubt will ever reach more than 3-5mb. Furthermore, it goes like: temporary store, free old, retrieve temporary -> new store.. so it would even be just for 1 second or so. IMO this isn't a problem at all.
Obviously, it depends on how you want to deal with writing the data / abstraction / etc if this is a good idea.
Like an ideal implementation for for example the per-user data might call the module callback for every struct which returns a string or something (eg: "<int>,<long>,<base64 string>,<base64string>") which then gets stored somewhere in memory (along with some data remembering of which user it belongs to), and then on the succesfull rehash store it back in the appropiate structs... So basically the module would only need a callback/whatever which serializes the data and unreal does the rest.
Ok, perhaps I just made your nightmare come trough by tossing this idea ;p.

> The thing is, remote includes make it a bit difficult. /rehash is reentrant. Meaning, if UserA quits, and a new UserA joins, we have to make sure not to load the data into the new user. So we'd need to make sure the file is timestamped and we don't load data for users who connected after that timestamp.

Not sure I understand. When we do the actual rehash (so unload/test/load) nothing else is going on so no user can quit/connect during that period (if it would we would have had huge problems anyway ;p).

codemastr

2004-07-18 23:58

reporter   ~0007154

Hmm, yeah I guess we could store the serialized data in memory. I wasn't ignoring that for memory savings, I guess it's just because the term serialize usually implies writing to a file or socket or something ;P Saving in memory could be a good idea. It would make things easier since instead of storing the user's name in the serialized data, we can just store an aClient *. The other thing is though, wouldn't the module still need to unserialize it? I mean, how do we know how to decode it if the module is the one that encoded it? I totally agree though, Unreal should do as much of the work as possible.

Oh and about the reentrant stuff, I don't know what the hell I was thinking, just ignore all that :P

aquanight

2004-07-19 02:03

reporter   ~0007156

(It's like 11:45PM for me here, so if nothing makes sense in this post, it's probably because I'm really tired :P )

As to all the MyModeY/YourModeY stuff... in an ideal world, there wouldn't be hundreds of modules trying to use the same mode character. Obviously the real world is far from ideal though :P .

>I guess it's just because the term serialize usually implies writing to a file or >socket or something ;P

By the way .NET defines "Serialize", it's simply writing to a stream. Where the other end of that Stream goes is, of course, up to the class implementing System.IO.Stream :P . So while there is a FileStream (but no SocketStream that I've seen :O ), there is also MemoryStream for writing into a byte array... :) . Obviously, you probably learned your definition from something else though ;) .

VB6 serializes controls into .frm/.ctl/etc (text)files by using a PropertyBag which is basically nothing more than a dictionary: each item is identified by a unique string, and the value is anything the control wishes to store (though there is no direct equivalent of Variant in C :( ). Typically, this is used for preserving Property values between closing and opening the designer, restarting VB6, going from design to run mode, etc, though theoretically it could be used for anything the control wants :) . I was thinking why not create a similar system (the value being char[] most likely, and modules can use memcpy to (de)serialize data)? It won't directly deal with module upgrading/structure changes - unless the module serialized the smart way :P (version identifier?). It would be useful for modules that need to serialize several otherwise unrelated pieces of data, though I can also see it would be a pain in the butt to do in a language that doesn't have classes :) (you can get pretty near to member functions in structs by using function pointers though... :) ).

syzop

2004-07-19 12:30

administrator   ~0007160

Last edited: 2004-07-19 12:30

 codemastr: Good to hear we think the same about this. And yeah, module should of course do the unserialize process too :).
aquanight: I personally know serialize() from PHP ;p. Anyway, this is C, which is not exactly a great highlevel language. The binary incompatability stuff you just mentioned means it's pretty much just the same as (saving) a struct + version thingy... And we are trying to solve the whole binary incompatability thing by converting it to 'parsable text' which the module can then easily parse (in a backward compatible manner). But yeah, the module could (should?) use a version parameter in the returned serialized string, that usually makes life easier ;p..

***

Another thing, in my more complex modules I sometimes refer from a user struct to another struct (which might or might not be allocated), or even a linked list.. perhaps we could also deal with that? ;).

Since we are probably be storing in memory, instead of <field1>,<field2>,<field3> we could store stuff in a parv[] alike array. Or perhaps even something with a type and data thingy:
typedef struct SModData ModData;
struct SModData {
   struct SModData *next, *prev;
   long type;
   char *data;
};
[or instead of data.. an union which provides int/long/char*/etc? But in that case we we should also make the module free the stuff etc]

UserData *myfunc([blablabla], ModData *mdata)
{
ModData *r;
MyStruct *e = MyMallocEx(sizeof(MyStruct));
 for (r = mdata; r; r=r->next)
 {
   switch(r->type)
   {
    case MY_ID: e->id = r->data.int; break;
    case MY_USERNAME: e->username = strdup(r->data.cp); break;
    case MY_IGNORELIST:
    {
       MyLList *x = MyMallocEx(sizeof(MyLList));
       x->target = strdup(r->data.cp);
       /* mmm would still have a problem with having multiple fields here ;p */
       AddListItem(x, e->ignorelist);
    }
[..]
   }
   /* Backward compatability.. fill missing fields.. */
   if (!e->username)
    e->username = strdup("unknown");
   [blabla]
   return (UserData *)e;
}

Hm *grin* ;).
*edit*readability*/edit*

edited on: 2004-07-19 12:30

codemastr

2004-07-19 13:40

reporter   ~0007162

> But yeah, the module could (should?) use a version parameter in the returned > serialized string, that usually makes life easier ;p..

Well, we could always store modulename,moduleversion at the beginning of the data.

But otherwise, yeah what you have is something like what I was thinking. If only C had a variant type...

The linked lists (and other complex structures) are somewhat difficult because each entry of the linked list itself would be a ModData structure *sigh*.

aquanight

2004-07-19 18:06

reporter   ~0007166

>If only C had a variant type...

Well, void* is about as close as you get - the module can interpret the data it's own way...

Or you could "borrow" the VARIANT structure from the COM headers... :P (of course - this means defining said structure yourself because *nix doesn't have the COM headers IIRC).

codemastr

2004-07-20 00:25

reporter   ~0007171

Yeah well COM pretty much just declares it as all the internal types and a void *. That doesn't help for MyStruct, only MyStruct *.

aquanight

2004-07-20 21:27

reporter   ~0007179

>Yeah well COM pretty much just declares it as all the internal types and a void*.
>That doesn't help for MyStruct, only MyStruct *.

Yes... but unfortunately there's little one can do... you can't possibly account for all possible structs... so there's little choice but to use the "BSTR" method - void* with some form of length prefix.

(Oh and BTW, IIRC the COM VARIANT also defines pointers to the internal types as well :P .)

Zell

2004-07-21 16:13

reporter   ~0007195

i actually have a real example.... one of AngryWolf's modules that uses a SnoMask to report usage of certain commands (configurable in the conf file), called CommandSno, is in use on my server. Upon rehashing, the snomask is unset. I had already figured out from disk activity that the server was unloading/reloading the modules. I hadnt thought about making them set [PERM] though... but as you all have said that would be ugly :[

syzop: that link to bug 0001855 tells me access denied. [is it marked private?]

syzop

2004-07-21 16:23

administrator   ~0007197

Yup it's private, to stop idiots from responding [HEY I GOT THAT TOO... HEY AND ME TOO AND WAIT YOU SHOULD IMPLEMENT IT LIKE X, HEY I GOT A *GREAT* IDEA.. YOU MUST IMPLEMENT IT BY Y]... {basically just what is happening here right now regarding .NET, COM, VB6 forms Wtf ;pp.. ok *run away* ;p}

syzop

2015-07-23 22:26

administrator   ~0018552

Although it doesn't save generic module data, we have a moddata system in place now to save user, client, channel and membership stuff, including serialization and unserialization. This is often sufficient.

Issue History

Date Modified Username Field Change
2004-07-18 22:31 codemastr New Issue
2004-07-18 22:41 syzop Note Added: 0007151
2004-07-18 22:46 syzop Note Edited: 0007151
2004-07-18 22:49 codemastr Note Added: 0007152
2004-07-18 22:51 codemastr Note Edited: 0007152
2004-07-18 23:12 syzop Note Added: 0007153
2004-07-18 23:58 codemastr Note Added: 0007154
2004-07-19 02:03 aquanight Note Added: 0007156
2004-07-19 12:30 syzop Note Added: 0007160
2004-07-19 12:30 syzop Note Edited: 0007160
2004-07-19 13:40 codemastr Note Added: 0007162
2004-07-19 18:06 aquanight Note Added: 0007166
2004-07-20 00:25 codemastr Note Added: 0007171
2004-07-20 21:27 aquanight Note Added: 0007179
2004-07-21 16:13 Zell Note Added: 0007195
2004-07-21 16:23 syzop Note Added: 0007197
2004-11-02 08:32 syzop Relationship added related to 0002153
2007-04-27 06:00 stskeeps Status new => feedback
2015-07-23 22:26 syzop Note Added: 0018552
2015-07-23 22:26 syzop Status feedback => closed
2015-07-23 22:26 syzop Assigned To => syzop
2015-07-23 22:26 syzop Resolution open => fixed
2015-07-23 22:26 syzop Fixed in Version => 3.4-alpha3