View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0001976 | unreal | ircd | public | 2004-07-17 19:59 | 2004-07-24 14:43 |
| Reporter | sh0 | Assigned To | |||
| Priority | normal | Severity | crash | Reproducibility | always |
| Status | closed | Resolution | open | ||
| Product Version | 3.2.1 | ||||
| Summary | 0001976: Rehashes may cause crashes if there are outstanding dns lookups | ||||
| Description | Outstanding asynchronous DNS lookups when a rehash is executed are likely to lead to problems; this may ultimately lead to ircd crashes. The typical less extreme reactions by the ircd usually look somewhat like this: [22:31:33] *** Resolving hostname 'HOSTNAME1'... [22:31:34] *** Notice -- $OPER is rehashing server config file [22:31:34] *** Notice -- Loading IRCd configuration .. [22:31:34] *** Notice -- oldcloak: WARNING: The 'oldcloak' module should *ONLY* be used temporarily for upgrading purposes since it is considered INSECURE, upgrade to 'cloak' AS SOON AS YOU CAN. [22:31:34] *** Notice -- Configuration loaded without any problems .. [22:31:35] *** Notice -- Hostname $HOSTNAME2 is unknown for server $SERVER2 (!?). [22:31:35] *** Notice -- Connecting to $SERVER2[$HOSTNAME2]. [22:31:36] *** LocOps -- ERROR :from $SERVER2 -- Server $SERVER0 already exists from $SERVER0 [22:31:36] *** LocOps -- ERROR :from $SERVER2 -- Closing Link: [$IP0] (Server Exists) [22:31:36] *** LocOps -- Server $SERVER2 closed the connection $SERVER0 is the server this happens on, $HOSTNAME0 its host, $IP0 its ip. $*1 refer to the server we are connecting to (and which has a host that has to be looked up before this can happen!) $*2 refer to an unrelated server that also has a linkblock on the server we are on, and is already linked. I.e. we trigger a connection, causing a DNS lookup (which has been artifically delayed), and then rehash the server before the lookup has been completed. When it is completed, the server attempts to connect to a completely unrelated third server - or coredumps instantly. Following an earlier suggestion by syzop, I've compiled the ircd with mpatrol and reproduced the bug to give you some more infos. mpatrol.log follows: """ @(#) mpatrol 1.4.8 (02/01/08) Copyright (C) 1997-2002 Graeme S. Roy This is free software, and you are welcome to redistribute it under certain conditions; see the GNU Library General Public License for details. For the latest mpatrol release and documentation, visit http://www.cbmamiga.demon.co.uk/mpatrol. operating system: UNIX system variant: Linux processor architecture: Intel 80x86 processor word size: 32-bit object file format: ELF32 dynamic linker type: SVR4 Log file generated on Sun Jul 18 01:31:52 2004 read 321 symbols from /usr/lib/i686/cmov/libssl.so.0.9.7 read 2444 symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.7 read 7 symbols from /lib/libcrypt.so.1 read 121 symbols from /lib/libnsl.so.1 read 62 symbols from /usr/lib/libz.so.1 read 9 symbols from /lib/libdl.so.2 read 303 symbols from /usr/lib/libmpatrol.so read 113 symbols from /usr/lib/libelf.so.0 read 577 symbols from /usr/lib/libbfd-2.14.90.0.7.so read 2013 symbols from /lib/libc.so.6 read 33 symbols from /lib/ld-linux.so.2 read 1122 symbols from /usr/lib/libstdc++-libc6.2-2.so.3 read 321 symbols from /lib/libm.so.6 read 850 symbols from ./ircd system page size: 4096 bytes default alignment: 4 bytes overflow size: 0 bytes overflow byte: 0xAA allocation byte: 0xFF free byte: 0x55 allocation stop: 0 reallocation stop: 0 free stop: 0 unfreed abort: 0 small boundary: 32 bytes medium boundary: 256 bytes large boundary: 2048 bytes lower check range: 0 upper check range: 0 check frequency: 1 failure frequency: 0 failure seed: 1090107112 prologue function: <unset> epilogue function: <unset> handler function: <unset> log file: mpatrol.log profiling file: mpatrol.out tracing file: mpatrol.trace program filename: ./ircd symbols read: 8296 autosave count: 0 freed queue size: 0 allocation count: 9330 allocation peak: 6347 (183449 bytes) allocation limit: 0 bytes allocated blocks: 5353 (122824 bytes) marked blocks: 0 (0 bytes) freed blocks: 0 (0 bytes) free blocks: 1187 (69688 bytes) internal blocks: 136 (2228224 bytes) total heap usage: 2420736 bytes total compared: 0 bytes total located: 78 bytes total copied: 17409 bytes total set: 695571 bytes total warnings: 0 total errors: 0 system page size: 4096 bytes default alignment: 4 bytes overflow size: 0 bytes overflow byte: 0xAA allocation byte: 0xFF free byte: 0x55 allocation stop: 0 reallocation stop: 0 free stop: 0 unfreed abort: 0 small boundary: 32 bytes medium boundary: 256 bytes large boundary: 2048 bytes lower check range: 0 upper check range: 0 check frequency: 1 failure frequency: 0 failure seed: 1090107112 prologue function: <unset> epilogue function: <unset> handler function: <unset> log file: mpatrol.log profiling file: mpatrol.out tracing file: mpatrol.trace program filename: ./ircd symbols read: 8296 autosave count: 0 freed queue size: 0 allocation count: 24064 allocation peak: 9781 (299120 bytes) allocation limit: 0 bytes allocated blocks: 6476 (170025 bytes) marked blocks: 0 (0 bytes) freed blocks: 0 (0 bytes) free blocks: 1331 (137175 bytes) internal blocks: 201 (3293184 bytes) total heap usage: 3600384 bytes total compared: 16 bytes total located: 78 bytes total copied: 37020 bytes total set: 936020 bytes total warnings: 0 total errors: 0 ERROR: [ILLMEM]: illegal memory access at address 0x00000000 0x00000000 not in heap call stack 0x08072331 connect_server+129 0x08072B65 do_dns_async+357 0x080720FA read_message+2282 0x080652BE main+2894 0x40295DC6 __libc_start_main+198 0x08053D21 _start+33 """ And for good measure, here is the output of gdb: """ ircd@support-unit-001:~/Unreal3.2_dev$ gdb bin/ircd core GNU gdb 6.1-debian Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-linux"...Using host libthread_db library "/lib/libthread_db.so.1". Core was generated by `./ircd'. Program terminated with signal 6, Aborted. Reading symbols from /usr/lib/i686/cmov/libssl.so.0.9.7...done. Loaded symbols for /usr/lib/i686/cmov/libssl.so.0.9.7 Reading symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.7...done. Loaded symbols for /usr/lib/i686/cmov/libcrypto.so.0.9.7 Reading symbols from /lib/libcrypt.so.1...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /lib/libnsl.so.1...done. Loaded symbols for /lib/libnsl.so.1 Reading symbols from /usr/lib/libz.so.1...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /lib/libdl.so.2...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /usr/lib/libmpatrol.so...done. Loaded symbols for /usr/lib/libmpatrol.so Reading symbols from /usr/lib/libelf.so.0...done. Loaded symbols for /usr/lib/libelf.so.0 Reading symbols from /usr/lib/libbfd-2.14.90.0.7.so...done. Loaded symbols for /usr/lib/libbfd-2.14.90.0.7.so Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /usr/lib/libstdc++-libc6.2-2.so.3...done. Loaded symbols for /usr/lib/libstdc++-libc6.2-2.so.3 Reading symbols from /lib/libm.so.6...done. Loaded symbols for /lib/libm.so.6 Reading symbols from tmp/9D5CC727.commands.so...done. Loaded symbols for tmp/9D5CC727.commands.so Reading symbols from tmp/E7D9308D.cloak.so...done. Loaded symbols for tmp/E7D9308D.cloak.so #0 0x402a9721 in kill () from /lib/libc.so.6 (gdb) (gdb) bt #0 0x402a9721 in kill () from /lib/libc.so.6 #1 0x402a94c5 in raise () from /lib/libc.so.6 #2 0x402aa9e8 in abort () from /lib/libc.so.6 #3 0x401b0022 in __mp_abort () from /usr/lib/libmpatrol.so #4 0x401aff24 in __mp_findsource () from /usr/lib/libmpatrol.so #5 <signal handler called> #6 0x402f5e60 in strncpy () from /lib/libc.so.6 #7 0x08072331 in connect_server (aconf=0x819311c, by=0x0, hp=0x81a5940) at s_bsd.c:2400 0000008 0x08072b65 in do_dns_async () at s_bsd.c:2617 #9 0x080720fa in read_message (delay=2, listp=0x8161340) at s_bsd.c:1757 #10 0x080652be in main (argc=0, argv=0x0) at ircd.c:1548 """ | ||||
| Steps To Reproduce | Basically three steps: 1.Create a linkblock for an unlinked server, and specify the hostname as hostname, not as ip. 2.Make sure that the hostname resolving will be delayed sufficiently for you to get a rehash in before it is completely, but not more than about 2-3 seconds (otherwise, unrealircd will abort the resolve attempt). 3.Connect the server, and immediatley afterwards, before the hostname has been resolved, rehash it. If you are an oper, the following commandsequence will do it: " DNS c REHASH CONNECT $SERVER1 REHASH " This should trigger the bug. If there are several linkblocks, and the unrealircd binary is not linked to mpatrol, this might cause the ircd to attempt to connect to an already linked server; if you do it sufficiently often, it ultimately will crash. If the unrealircd binary is linked to mpatrol, it will (apparently) always crash instantly. As a matter of fact, for unrealircd binaries linked to mpatrol you may not have to artifically delay the replies to dns lookups if you send the whole command-sequence at once (e.g. by pasting it into a netcat instance); it was possible to reproduce it here without delaying the dns requests made by those binaries. | ||||
| Additional Information | The way to reproduce this bug was discovered after another netadmin looked over the unrealircd 3.2 code and wondered what would happen in this situation. His description of the problem and suggestions to fix it follow: If unreal needs to asynchronously resolve a servername when connecting to another server, it saves a pointer to the corresponding link block along with the request. On rehash, this link block is deleted. If and when a reply to the dns request arrives (before the timeout), the pointer to the deleted link block will then cause an access to an invalid memory address, leading to undefined behaviour (usually connection attempts to an arbitrary server, or crashes). Suggestions for approaches to fix it: 1.Delete all ASYNC_CONNECT requests on rehash. 2.Use the reference counter. | ||||
| 3rd party modules | |||||
|
|
Thank you. I'll certainly take a look at this when I got some time (about a few days) :). |
|
|
I'd have to go for #1 (cancel+restart all pending DNS requests on rehash) because there are other reasons a rehash might have occured: /etc/resolv.conf was changed (though IIRC this needs a restart) link::hostname was changed (but the linkblock wasn't deleted), thus Unreal will still try to connect to the old host, which may have gone down (permanently?) or be a whole different machine (and what if this machine ran a different IRCd? Especially one that has a wierd protocol - could cause Unreal to crash since server-server stuff doesn't have a lot of checks on it (IIRC)! Uh oh...)! Pending /connect or link::options::autoconnect attempts waiting for a DNS reply may need to be canceled and restarted as well... I would probably say that even pending DNS requests for incoming connections need to be cancelled and restarted as well, though I can't see any major reason this would be necessary. *edit* Syzop beat me :/ */edit edited on: 2004-07-18 19:05 |
|
|
Should be fixed in CVS [.112] (using tactic 1). Can you confirm that it no longer crashes? |
|
|
Ack, afaict it has been fixed. Good work on that. |
|
|
Good. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2004-07-17 19:59 | sh0 | New Issue | |
| 2004-07-18 18:58 | syzop | Note Added: 0007148 | |
| 2004-07-18 19:04 | aquanight | Note Added: 0007149 | |
| 2004-07-18 19:05 | aquanight | Note Edited: 0007149 | |
| 2004-07-22 18:19 | syzop | Note Added: 0007224 | |
| 2004-07-22 18:20 | syzop | Status | new => confirmed |
| 2004-07-24 12:55 | sh0 | Note Added: 0007244 | |
| 2004-07-24 14:43 | syzop | Status | confirmed => closed |
| 2004-07-24 14:43 | syzop | Note Added: 0007245 |
