View Issue Details

IDProjectCategoryView StatusLast Update
0003888unrealinstallingpublic2010-02-08 17:56
Reporterohnobinki Assigned Tosyzop  
PrioritynormalSeverityminorReproducibilityhave not tried
Status resolvedResolutionfixed 
Platformamd64OSGentooOS Version2.0
Product Version3.2.8 
Fixed in Version3.2.9-RC1 
Summary0003888: cURL not handled well by buildsystem
Descriptiontons of problems.

./configure doesn't fail early when it should

./configure nor ./Config have sane defaults concerning --enable-libcurl

using old system curl + bundled new cares causes strange problems:
``
mode/#unreal-support [+v slushey] by [email protected]
<slushey> * Loading IRCd configuration ..
<slushey> [error] unrealircd.conf:9: include: error downloading '(url taken out)': Could not resolve host: www.synirc.net (Successful completion)
<slushey> [error] IRCd configuration failed to load
<slushey> $ nslookup www.synirc.net
<slushey> Server: 64.85.160.4
<slushey> Address: 64.85.160.4#53
<slushey> Non-authoritative answer:
<slushey> www.synirc.net canonical name = synirc.net.
<slushey> Name: synirc.net
<slushey> Address: 66.179.9.246
mode/#unreal-support [+b *!*@9CE2AB91.8D9E3B4C.8A7D5246.IP] by [email protected]
<slushey> $
slushey has been kicked by [email protected] [Banned: You are sending text too fast. DO NOT PASTE! Use http://unrealircd.pastebin.com - Otherwise, please try to put more text on each line you send so you aren't filling the channel with your nonsense. - 1 minute ban]
<binki> lol
<Stealth> I need to add that to the bot too
''
...
``
18-01-2010 22:29:08 > binki: slushey: pastebin your server's /etc/resolv.conf
[email protected] has joined #unreal-support
mode/#unreal-support [+o Nazca] by [email protected]
mode/#unreal-support [+o Nazca] by [email protected]
<binki> oh, nvmd
<slushey> its 2 lines so
<slushey> domain slushey.net
<slushey> nameserver 64.85.160.4
<binki> slushey: try getting the same URL using curl's CLI
<slushey> it worked
<binki> try upgrading curl
<slushey> worked
<slushey> thanks
[email protected] has left #unreal-support
''
Additional InformationMy patch adds some sane defaults and brings errors to light more quickly. I.e., the user will hopefully trigger an error in ./configure when messing with --enable-libcurl than at buildtime.

I also added a warning about system curl + bundled c-ares that nobody will read. Maybe I'll write a patch for ./Config to make system-cares and system-tre more available to users.
TagsNo tags attached.
Attached Files
3rd party modules

Activities

syzop

2010-01-25 12:00

administrator   ~0016002

Did you see 0003857 and the related solution, and the new stuff in CVS? I know your patch is against CVS, but from your talk I can't make up you really looked at the new CVS stuff (quite the contrary).
Especially since you talk about not having sane defaults, well.. with CVS if you say 'yes' during ./Config to remote includes and use the defaults, it will download and build curl if necessary, and use that, so not the "system curl", which solved all the problems 0003857 and others are having with regards to ABI incompatibility and such. (Unless something goes wrong in that download & build process, if so, please report in a new issue).

syzop

2010-01-25 12:29

administrator   ~0016003

Objectives:
1) Should work, no crash (due to ABI incompatability issues)
2) Should not stall the ircd when doing /REHASH (due to curl not being compiled with c-ares)
3) Don't use an insecure old c-ares
 
This means:
* System curl without c-ares is no go (conflicts with 1)
* System curl with c-ares is only a go if:
  * The c-ares it uses is equal to or newer than unreal's (makes 1&2&3 ok. needs to use --system-c-ares then)
  However there are still some issues (see further down)
  Is not a go if:
  * Well any other case, but in particular.. version too old (3), and if not using system-c-ares (1)
 
The argument in favor of system curl & c-ares is:
* Managability: curl & c-ares get upgraded automatically by the package manager etc in case of issues & updates
The argument against it is:
* Curl without c-ares is a no go
* Due to the automatic upgrading, you'll need to recompile Unreal whenever system c-ares or curl updates (and is ABI incompatible.. which has been frequent with c-ares..).
 
In my view this basically means:
* If Unreal is installed via a package manager, system curl & system c-ares (assuming they are too) are a good choice. Unreal package will also be upgraded if needed due to the new curl/c-ares.
* If Unreal is self-compiled, then it is debatable... a tradeoff. As Unreal will crash or misbehave whenever something in the system changes. This is where our automatic download & install fix from 0003857 helps and takes care of all that.

The only downside with the auto-install aproach is that curl updates don't take place automatically, and that it takes some more compile time and space.

As you can see, it's not a simple issue.

To my knowledge package mantainers are already using system-c-ares and curl.
And as for self-compiled, the solution to that is in CVS.

I don't mind applying parts of your patch though, there are some good parts in it. However, it should not warn the same way for system curl as the auto-download-and-install curl.

ohnobinki

2010-01-25 17:34

reporter   ~0016004

I do not know how much I have paid attention to 0003857 , but I did notice the changes to CVS concerning assisting the user in invoking ./curlinstall. In fact, that is why I filed this bug ;-).

Sorry. I now realize that I went overboard in my patch's comments, etc. But I will never be able to understand why people will opt for manually installing cURL into their ${HOME}---unless if they're forced to because they're using a shell account. I have observed that people do install curl locally regardless of how silly it is (in my understanding) to do so. So I suppose it'd be OK to support them even if I can't understand their motivation.

But may the ./configure script at least issue a warning when system and local versions of curl & c-ares (respectively) are requested by the user? I am pretty sure that my patch will not warn the user if he chose ./curlinstall... I tested to make sure that the cURL prefix was not ${HOME}/curl (which is where ./curlinstall places its files afaik).

FYI, and I forgot about this until now, the ``proper'' solution for preventing conflict between a bundled c-ares and system curl is symbol visibility: http://blog.flameeyes.eu/2008/02/10/why-would-an-executable-export-symbols . There should be some way to hide the c-ares symbols in the final ircd executable, but I'm not sure this is worth the work since using a system-installed c-ares is the ideal and avoids this problem altogether :-/

Thanks for considering even pieces of my patch, I look forward to seeing ``cvs update'' output some `U' lines :-).

ohnobinki

2010-01-25 21:12

reporter   ~0016005

Concerning visibility, here is the problem:
Run this command in a compiled UnrealIRCd-3.2.8.1 directory:
$ nm -D src/ircd | grep ares_version

You should see something like ``21341234 T ares_version'' (iirc). See the following output from my machine.

--with-system-cares:
$ nm -D $(which unrealircd) | grep ares_version
                 U ares_version

libcares.so:
$ nm -D /usr/lib64/libcares.so | grep ares_version
000000000000ce00 T ares_version

with the patch I have attached and --without-system-cares (uses bundled):
$ nm -D src/ircd | grep ares_version
$
(no output).

The output from the last one should be possible with unrealircd-3.2.8.1CVS-hidden-cares.patch . This will mean that if unrealircd links against a system curl, that curl will look where it normally does for c-ares, even when c-ares is bundled into src/ircd. Also, unrealircd will use the bundled version of c-ares. Something similar could be done for TRE, but there have been no issues so far.

syzop

2010-01-25 21:26

administrator   ~0016006

Patch applied, mostly unharmed :). Sorry, I overlooked the $HOME check indeed!

Changed a warning in an error. Added a warning when using a curl without c-ares (which, as you too pointed out, probably nobody reads :P). Etc.

- Added patch from ohnobinki (0003888), only slightly edited, which improves
  curl detection, added checks to see if curl actually works (print out a
  clear curl error during configure, instead of getting an error during
  'make'), and we now error when using --enable-libcurl without
  --with-system-cares if the system curl depends on c-ares. This is because
  this can cause ABI incompatability between curl's c-ares and our c-ares,
  which leads to odd issues such as:
  Could not resolve host: www.example.net (Successful completion)
  And possibly other weird issues, perhaps even crashes.

Thanks for the patch, definitely appreciated.

Improving ./Config is probably not a bad idea either.

I'll check your other patch on Wednesday.

syzop

2010-01-25 21:37

administrator   ~0016007

Seems you posted while I was committing & writing my comment.

Ehm... yeah is interesting, but.. I think we already solved the problem this way? It's probably a good idea to use system c-ares if it's available, anyway. And when curl is compiled with c-ares (which it should) then c-ares should be available too. (and if curl isn't compiled with c-ares then there's no problem).

Only thing I can think of is if you have some old c-ares on your system, which is insecure or does not support the features we require. Then you would need to upgrade your system c-ares first, before you can compile Unreal (as we require version X.YY). But that shouldn't be too much of a problem.
If it is, like you don't have root and the admin refuses, then you can always build your own.
Like.. the example pointed out that unreal didn't work because the system c-ares was a version 1.4.0 and unreal used 1.6.0 is not much of an issue to me, as <1.6.0 had a security issue in it and shouldn't be used anyway (incorrect memory allocation of multiple PTR records, in case you wonder, and possibly other issues).
Of course, if we at some point depend on some really new feature in c-ares, then it might be an issue. Then again, if you choose system c-ares, then it's your job to keep it recent.

syzop

2010-01-25 21:38

administrator   ~0016008

I'll think again about it on Wednesday :p

syzop

2010-01-27 09:58

administrator   ~0016009

Last edited: 2010-01-27 10:01

This patch seems to break building Unreal *without* cURL.

./configure --with-showlistmodes --enable-hub --enable-prefixaq --with-listen=5 --with-dpath=/home/syzop/Unreal3.2 --with-spath=/home/syzop/Unreal3.2/src/ircd --with-nick-history=2000 --with-sendq=3000000 --with-bufferpool=18 --with-hostname=vulnscan --with-permissions=0600 --with-fd-setsize=1024 --enable-dynamic-linking

gcc -I../include -I/home/syzop/Unreal3.2/extras/regexp/include -I/home/syzop/Unreal3.2/extras/c-ares/include -pipe -g -O2 -funsigned-char -fno-strict-aliasing -Wno-pointer-sign -export-dynamic   -c timesynch.c
gcc -I../include -I/home/syzop/Unreal3.2/extras/regexp/include -I/home/syzop/Unreal3.2/extras/c-ares/include -pipe -g -O2 -funsigned-char -fno-strict-aliasing -Wno-pointer-sign -export-dynamic   -c res.c
--snip--
gcc -I../include -I/home/syzop/Unreal3.2/extras/regexp/include -I/home/syzop/Unreal3.2/extras/c-ares/include -pipe -g -O2 -funsigned-char -fno-strict-aliasing -Wno-pointer-sign -export-dynamic    -o ircd timesynch.o res.o s_bsd.o auth.o aln.o channel.o cloak.o crule.o dbuf.o events.o fdlist.o hash.o help.o ircd.o ircsprintf.o list.o lusers.o match.o modules.o packet.o parse.o s_auth.o s_conf.o s_debug.o s_err.o s_extra.o s_kline.o s_misc.o s_numeric.o s_serv.o s_svs.o  socket.o ssl.o s_user.o charsys.o scache.o send.o support.o umodes.o version.o whowas.o zip.o cidr.o random.o extcmodes.o extbans.o md5.o api-isupport.o api-command.o url.o  -lcrypt -lnsl  -ldl -L/home/syzop/Unreal3.2/extras/regexp/lib -ltre   -L../extras/c-ares/lib -L/home/syzop/Unreal3.2/extras/c-ares/lib -lcares -lrt
url.o: In function `url_do_transfers_async':
/home/syzop/Unreal3.2/src/url.c:302: undefined reference to `curl_multi_perform'
/home/syzop/Unreal3.2/src/url.c:317: undefined reference to `curl_multi_fdset'
/home/syzop/Unreal3.2/src/url.c:333: undefined reference to `curl_multi_info_read'
/home/syzop/Unreal3.2/src/url.c:341: undefined reference to `curl_easy_getinfo'
/home/syzop/Unreal3.2/src/url.c:342: undefined reference to `curl_easy_getinfo'
/home/syzop/Unreal3.2/src/url.c:343: undefined reference to `curl_easy_getinfo'
/home/syzop/Unreal3.2/src/url.c:344: undefined reference to `curl_easy_getinfo'
/home/syzop/Unreal3.2/src/url.c:372: undefined reference to `curl_multi_remove_handle'
/home/syzop/Unreal3.2/src/url.c:377: undefined reference to `curl_easy_cleanup'
/home/syzop/Unreal3.2/src/url.c:327: undefined reference to `curl_multi_perform'
url.o: In function `url_init':
/home/syzop/Unreal3.2/src/url.c:221: undefined reference to `curl_global_init'
/home/syzop/Unreal3.2/src/url.c:222: undefined reference to `curl_multi_init'
url.o: In function `download_file_async':
/home/syzop/Unreal3.2/src/url.c:242: undefined reference to `curl_easy_init'
/home/syzop/Unreal3.2/src/url.c:264: undefined reference to `curl_easy_setopt'
/home/syzop/Unreal3.2/src/url.c:265: undefined reference to `curl_easy_setopt'
/home/syzop/Unreal3.2/src/url.c:266: undefined reference to `curl_easy_setopt'
/home/syzop/Unreal3.2/src/url.c:267: undefined reference to `curl_easy_setopt'
/home/syzop/Unreal3.2/src/url.c:272: undefined reference to `curl_easy_setopt'
url.o:/home/syzop/Unreal3.2/src/url.c:273: more undefined references to `curl_easy_setopt' follow
url.o: In function `download_file':
/home/syzop/Unreal3.2/src/url.c:144: undefined reference to `curl_easy_init'
/home/syzop/Unreal3.2/src/url.c:170: undefined reference to `curl_easy_setopt'
/home/syzop/Unreal3.2/src/url.c:171: undefined reference to `curl_easy_setopt'
/home/syzop/Unreal3.2/src/url.c:172: undefined reference to `curl_easy_setopt'
/home/syzop/Unreal3.2/src/url.c:173: undefined reference to `curl_easy_setopt'
/home/syzop/Unreal3.2/src/url.c:174: undefined reference to `curl_easy_setopt'
url.o:/home/syzop/Unreal3.2/src/url.c:175: more undefined references to `curl_easy_setopt' follow
url.o: In function `download_file':
/home/syzop/Unreal3.2/src/url.c:188: undefined reference to `curl_easy_perform'
/home/syzop/Unreal3.2/src/url.c:200: undefined reference to `curl_easy_getinfo'
/home/syzop/Unreal3.2/src/url.c:201: undefined reference to `curl_easy_cleanup'
/home/syzop/Unreal3.2/src/url.c:209: undefined reference to `curl_easy_cleanup'
url.o: In function `download_file_async':
/home/syzop/Unreal3.2/src/url.c:289: undefined reference to `curl_multi_add_handle'
collect2: ld returned 1 exit status
make[1]: *** [ircd] Error 1
make[1]: Leaving directory `/home/syzop/Unreal3.2/src'
make: *** [build] Error 2


syzop

2010-01-27 10:14

administrator   ~0016010

I'm quite busy today, and, I can't quickly spot the error to be honest (which isn't unusual with autoconf) :P.

I've reverted the patch in .793 until we fix this:
- Patch from above is (temp.) reverted, Unreal wouldn't compile without curl.

ohnobinki

2010-01-27 23:57

reporter   ~0016011

Sorry, it looks like I ended the AS_IF([test "x$enable_curl" != "no"]) with a comment that looks like ``dnl for emac's sh-mode ])''. This is now removed. Its presence caused url.o to be added to the list of files that's compiled and linked into UnrealIRCd and we somehow didn't see a warning about too many closing ``])'' sequences... which confuses me still :-/

Oh, disregard version 2, the ([]) are stil unbalanced I think...

ohnobinki

2010-01-28 00:01

reporter   ~0016012

Oh, I think that version 2 is properly balanced, just emacs was confused and thus confusing me ;-).

Compiling with and without remote includes now works for me.

syzop

2010-02-08 17:56

administrator   ~0016014

ok, redid the fix.

Issue History

Date Modified Username Field Change
2010-01-25 05:48 ohnobinki New Issue
2010-01-25 05:48 ohnobinki File Added: unrealircd-3.2.9_rc-curl-sanity.patch
2010-01-25 12:00 syzop Note Added: 0016002
2010-01-25 12:29 syzop Note Added: 0016003
2010-01-25 17:34 ohnobinki Note Added: 0016004
2010-01-25 21:12 ohnobinki Note Added: 0016005
2010-01-25 21:13 ohnobinki File Added: unrealircd-3.2.8.1CVS-hidden-cares.patch
2010-01-25 21:26 syzop QA => Not touched yet by developer
2010-01-25 21:26 syzop U4: Need for upstream patch => No need for upstream InspIRCd patch
2010-01-25 21:26 syzop Note Added: 0016006
2010-01-25 21:26 syzop Status new => resolved
2010-01-25 21:26 syzop Fixed in Version => 3.2.9-RC1
2010-01-25 21:26 syzop Resolution open => fixed
2010-01-25 21:26 syzop Assigned To => syzop
2010-01-25 21:37 syzop Note Added: 0016007
2010-01-25 21:38 syzop Note Added: 0016008
2010-01-27 09:58 syzop Note Added: 0016009
2010-01-27 09:58 syzop Status resolved => feedback
2010-01-27 09:58 syzop Resolution fixed => reopened
2010-01-27 10:01 syzop Note Edited: 0016009
2010-01-27 10:14 syzop Note Added: 0016010
2010-01-27 23:50 ohnobinki File Added: unrealircd-3.2.9_pre-curl-sanity-2.patch
2010-01-27 23:57 ohnobinki Note Added: 0016011
2010-01-28 00:01 ohnobinki Note Added: 0016012
2010-02-08 17:56 syzop Note Added: 0016014
2010-02-08 17:56 syzop Status feedback => resolved
2010-02-08 17:56 syzop Resolution reopened => fixed