View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0006470 | unreal | installing | public | 2024-09-11 22:03 | 2024-09-25 19:41 |
| Reporter | rafaelgrether | Assigned To | |||
| Priority | low | Severity | minor | Reproducibility | always |
| Status | new | Resolution | open | ||
| Product Version | 6.1.7.2 | ||||
| Summary | 0006470: Building UnrealIrcd with AddressSanitizer Enabled - Fatal Error | ||||
| Description | Hello, I'm building UnrealIRCd with Address Sanitizer Enabled. At the end of the build, I got fatal error (related to OpenSSL): Checking for updates for third party modules... Reading module repository list from '/home/devnull/unrealircd-6.1.7.2/doc/conf/modules.sources.list'... Checking module repository https://modules.unrealircd.org/modules.list... [error] https://modules.unrealircd.org/modules.list: Internal OpenSSL error or protocol error [: internal error] Fatal error encountered Manually running ./src/ircd -m upgrade --no-install I got the same fatal error. I traced the syscalls (attached here) trying to find something. As this problem doesn't occur without Address Sanitizer, could it be something related to non thread safe OpenSSL functions? Thanks! | ||||
| Steps To Reproduce | $ fetch https://www.unrealircd.org/downloads/unrealircd-6.1.7.2.tar.gz $ tar -zxf unrealircd-6.1.7.2.tar.gz $ cd unrealircd-6.1.7.2 $ ./Config (answer Yes on Address Sanitizer question) $ gmake | ||||
| Additional Information | OS: FreeBSD 14.1p4 | ||||
| Tags | No tags attached. | ||||
| Attached Files | |||||
| 3rd party modules | |||||
|
|
I forgot to mention my OpenSSL version: OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024) |
|
|
I'm not sure what that could be, or at least not an immediate suggestion. I run myself with ASan enabled all the time during development (and even one or two live servers). The only OpenSSL issues I know with ASan is that it finds lots of fake/invalid memory leaks when it exits at the end (eg upon kill -15 or /die)... But I have not experienced a change in behavior, so not what you showed with like failing to connect to TLS servers. This is my OpenSSL version, it seems to be the same: OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024) I'm on Linux, Ubuntu 24.04 LTS. Compiler I use is: Ubuntu clang version 18.1.3 (1) |
|
|
Oh yeah, and of course: ASan uses a looooooot more memory. So make sure there's enough on the machine and that the user account is not limited. When actually running unrealircd (so not modulelist what you mentioned, but actually running the irc server) it will show up in "ps" as using 20.0 terrabyte of virtual memory. Of course that is not the real memory usage, which starts at around 300MB. When stress testing it can go up to 2GB, though. Performance-wise is similar, everything becoming slower. Especially rehashing which can easily take 15 seconds. Just a general warning in addition to previous comment :D |
|
|
Yesterday I tested on FreeBSD 14.1 (with Clang 18.1.5) , and today with on old FreeBSD 14.0 (with Clang 16.0.6). On booth occurs the Fatal Error. Your addition was pertinent, but I think that's not the case. One of the machines I tested (FreeBSD 14.0) has 64 Gb RAM, but just in case I monitored the memory usage (top/htop), and it is minimal. As UnrealIRCd was built (the fatal error occurred in the modulelist fase), today I will do more tests on the execution of UnrealIRCd itself. I think that if the problem is related to OpenSSL stuff, UnrealIRCd will not run either. I'm not an experienced programmer, but if I get the fatal error when running UnrealIRCd with ASan enabled, I believe I can debug the execution. But maybe I need help with that =/ It's not something I really need (ASan), but as I'm starting to learn and develop modules, I think it would be interesting to have ASan to catch any of my development mistakes. |
|
|
Syzop, Running UnrealIRCd with ASan, I got coredump. Running backtrace, coredump occur in abort() inside misc.c, after method EVP_DigestInit_ex() , line 2281. EVP_MD_CTX *mdctx = EVP_MD_CTX_new(); if (EVP_DigestInit_ex(mdctx, sha256_function, NULL) != 1) ---> abort(); I attached backtrace.txt. I changed -O2 flag to -O0 to remove optimization, to inspect some values in debug. I attached inspect.txt too. I don't know if it is the cause, but inspecting sha256_function, before abort() and coredump, these are many NULL (0x00) on init, update and final functions from sha256_function. init = 0x0000000000000000 update = 0x0000000000000000 final = 0x0000000000000000 copy = 0x0000000000000000 cleanup = 0x0000000000000000 |
|
|
Syzop, I think the problem is deeper. I wrote a simple code to hash a message (attached code EVP_DigestInit_ex.c). I executed both on Debian Linux and FreeBSD with success. But with -fsanitize=address flag, the code compiled on FreeBSD, coredump. On Linux, still working. Debug OpenSSL is very complex to me. In Dez/2023 I got a coredump in new anope 2.1 that Linux(with gcc/clang) doesn't caught somehow, but FreeBSD+clang do. It was some problem of null pointer dereference, we take some days to identify and fix. Initially I thought it might have something with the use of Deprecated openssl functions like SSL_load_error_strings() and SSLeay_add_ssl_algorithms() in src/tls.c But I wrote a very simple code, and same coredump. So I believe the problem may be on the OpenSSL side. Sorry, regarding my knowledge, I can't go any further than that :( |
|
|
Yeah it seems OpenSSL with ASan is failing on your system. I can't really help with that, maybe the FreeBSD guys or otherwise OpenSSL can help? You shouldn't be the first person with that, one would imagine. Have you searched? I dunno. I am a Linux guy myself so I must confess I won't be digging deeper into that. It's a shame because you are right, ASan is very useful when you are developing. ASan will catch bugs straight away "when the fault happens" rather than bugs that silently corrupt structures/memory/etc and "everything looks fine" and then a day later the program crashes and you have no clue as to where the problem is. I very much recommend ASan to everyone who develops modules, and that's also why I use it myself. |
|
|
Yes, I did all the research I could. I tested several codes from other softwares that use EVP_DigestInit_ex(). But whenever I change the flag to -fsanitize=address, the same coredump occurs. The FreeBSD community certainly believes that it is an internal OpenSSL memory leak, which somehow is not caught in Linux. They said this is not the first time this has happened with OpenSSL, and in particular in the function EVP_DigestInit_ex(), which has a history of memory leaks. I also asked for support in the OpenSSL community about this, I'm waiting to see if I receive some response, but I confess that I'm not hopeful. But thanks Syzop, if you want to close the issue, that's fine, I see there's nothing we can do. Or if you want to wait and see if I can get OpenSSL response back, I can update here too with new notes about this. |
|
|
Got an outdated freebsd VM (12.2) but it simply never ends, 100% CPU loop. Turns out to be this (completely different) bug: https://stackoverflow.com/questions/70452645/openssl-stucks-on-ssl-library-init-when-compiled-on-freebsd-clang9-and-clang1 https://github.com/openssl/openssl/issues/17339 Quick google on freebsd openssl sanitizer also shows recent issues like: https://github.com/openssl/openssl/issues/24793 One of them suggested to compile with "gcc" instead of "clang". Maybe worth trying for you? I don't have "gcc" installed so couldn't immediately check and.. not THAT much interest to be honest as it is not an unrealircd bug. |
|
|
I do not tested on FreeBSD < 14.0. While searching these days, I found many issues involving openssl x freebsd x clang. I'm still going to do two more tests, one of which is using GCC. And another is to use LibreSSL instead of OpenSSL. As soon as I finish I will keep you updated on the result. Even if it has no relation to UnrealIRCd, at least we will have some alternative (or workaround) for those who need to use ASan on FreeBSD. |
|
|
Syzop, Theo Buehler, OpenBSD and LibreSSL developer, helped me to find the problem. He identified that the problem is due to how FreeBSD and NetBSD's libasan is intercepting SHA256_Init. FreeBSD/NetBSD set SANITIZER_INTERCEPT_SHA2, and libasan replaces it with one of incompatible signature because this interception is designed for the libc version, which return NULL. It seems this problems doesn't occur with EVP_sha3_256(), another algorithm based on Keccak. He opened a bug on FreeBSD: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=281685 to solve the issue. For now, I think I will install some Linux on a VM to test module with Asan :P I think we can close this bug. Thanks for help and patience, Syzop! |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2024-09-11 22:03 | rafaelgrether | New Issue | |
| 2024-09-11 22:03 | rafaelgrether | File Added: syscall_trace.txt | |
| 2024-09-11 22:10 | rafaelgrether | Note Added: 0023343 | |
| 2024-09-12 09:10 | syzop | Note Added: 0023344 | |
| 2024-09-12 09:26 | syzop | Note Added: 0023345 | |
| 2024-09-12 16:16 | rafaelgrether | Note Added: 0023348 | |
| 2024-09-12 22:30 | rafaelgrether | Note Added: 0023349 | |
| 2024-09-12 22:30 | rafaelgrether | File Added: btall.txt | |
| 2024-09-12 22:30 | rafaelgrether | File Added: inspect.txt | |
| 2024-09-13 18:55 | rafaelgrether | Note Added: 0023350 | |
| 2024-09-13 18:55 | rafaelgrether | File Added: EVP_DigestInit_ex.c | |
| 2024-09-13 19:33 | syzop | Note Added: 0023351 | |
| 2024-09-13 19:47 | rafaelgrether | Note Added: 0023352 | |
| 2024-09-16 17:55 | syzop | Note Added: 0023355 | |
| 2024-09-16 19:50 | rafaelgrether | Note Added: 0023356 | |
| 2024-09-25 19:41 | rafaelgrether | Note Added: 0023382 |
