Issue614791
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2002-09-26 04:44 by rbgrn, last changed 2022-04-10 16:05 by admin. This issue is now closed.
Messages (16) | |||
---|---|---|---|
msg12483 - (view) | Author: Rob Green (rbgrn) | Date: 2002-09-26 04:44 | |
I know this is because it is simply a wrapper for the libc gethostbyname, but it really, really hurts my app to have it block like this. I am requesting that by any means necessary, gethostbyname() not block when multiple threads call it simultaneously. It seriously ruins the ability to write effective network daemons in python, as it can take a very, very long time to get through even a small list of hostnames if one or a few of them need to timeout to be resolved. |
|||
msg12484 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2002-09-26 16:06 | |
Logged In: YES user_id=21627 Can you please report what platform you are using? |
|||
msg12485 - (view) | Author: Rob Green (rbgrn) | Date: 2002-09-26 16:09 | |
Logged In: YES user_id=590105 FreeBSD 4.5, although I don't think it's platform specific, as I've seen some messages from guido on certain mailing lists talking about this problem and how it will require a large rewrite to fix correctly. In all honesty, I'd be happy with a simple hack to make this work and then a rewrite of whatever needs it in the future to have it written correctly. |
|||
msg12486 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2002-09-26 16:29 | |
Logged In: YES user_id=21627 Bad choice of system :-) In general, it would be the easiest modification of the interpreter to just release the GIL before getaddrinfo (which gethostbyname uses); this is a simple change and makes everything free running. Unfortunately, on BSD, getaddrinfo is not thread-safe. So for this system, a much more complicated change must be implemented. I have semi-approved any change that releases the GIL unconditionally on "good" systems (Solaris, Linux, Tru64). That won't help you, though. If you can contribute a patch that adds a getaddrinfo lock for "bad" systems, and avoids this lock on "good" systems, there is a small chance that this can get included in 2.2.2. Without anybody acting quickly, this may or may not get fixed in 2.3. |
|||
msg12487 - (view) | Author: Rob Green (rbgrn) | Date: 2002-09-26 17:28 | |
Logged In: YES user_id=590105 This is a little over my head, but I'll take a look to see what you mean. Could you give me any more info to point me in the right direction? Thanks |
|||
msg12488 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2002-09-26 18:14 | |
Logged In: YES user_id=21627 socketmodule.c contains a number of calls to getaddrinfo and getnameinfo. On systems where those calls are not thread-safe, you should add a lock around them, see Python 2.1, socketmodule.c, gethostbyname_lock. On all systems, you should use Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS; again, see Python 2.1. There probably should be a single lock around both getaddrinfo and getnameinfo. By default, those should be considered thread-safe, giving explicit exceptions: *BSD, and the fallback implementations (i.e. if either getaddrinfo.c or getnameinfo.c is included). |
|||
msg12489 - (view) | Author: Rob Green (rbgrn) | Date: 2002-09-29 19:05 | |
Logged In: YES user_id=590105 I've done some testing and I partially see what you are talking about. I wrote a test case in C using pthreads and in FreeBSD 4.5 I was able to resolve tens of hostnames simultaneously with no errors or problems. Python of course has the lock wrapped around the call, and Java appears to as well. Should I expect problems with this? Or might this be an indication that somewhere along the lines in freebsd development this issue was resolved but never tested/incorporated with Python/Java? If this is the case, then I suppose we would need to figure out at what version this changed and have the python lock act accordingly. |
|||
msg12490 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2002-09-29 21:31 | |
Logged In: YES user_id=21627 As for FreeBSD bugs: Please consult the BUGS section of getaddrinfo(3). It may be that this man page is incorrect, I don't know. |
|||
msg12491 - (view) | Author: Rob Green (rbgrn) | Date: 2002-09-29 23:19 | |
Logged In: YES user_id=590105 http://orange.kame.net/dev/query-pr.cgi?pr=277 There's the bug. It was submitted over 2 years ago, and I'm willing to bet it has been fixed but their bug database isn't up- to-date. I ran the test code and it passed on my machine, meaning it's fixed in my version, so I emailed the author to find out what version this was fixed in. |
|||
msg12492 - (view) | Author: Rob Green (rbgrn) | Date: 2002-09-29 23:45 | |
Logged In: YES user_id=590105 http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libc/net/name6.c It appears as though this was made thread-safe in revision 1.17 of the file which was Jan 24, 2001. Unfortunately there are two branches of code here, and I have no idea what's going on with them. I apparently have code from 2000 that doesn't appear to be thread-safe but the test worked so I'm gonna send around some emails and see if I can't clear anything up. |
|||
msg12493 - (view) | Author: Rob Green (rbgrn) | Date: 2002-09-30 02:12 | |
Logged In: YES user_id=590105 Here's what I found out. The thread-safe version is available in FreeBSD-Current, which will be Release and Stable in v5.0. So you can take out the lock for FreeBSD 5.0 and up, and that should be incorporated into the next Python release, as it has been available for FreeBSD-Current users for a year now. |
|||
msg12494 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2002-09-30 07:30 | |
Logged In: YES user_id=21627 Thanks for the research. We'll leave the bug open until this is actually fixed. It would be still possible to implement a patch that replaces the GIL with a getaddrinfo lock on systems where getaddrinfo is not thread-safe, but I feel no urge to implement such a fix now. The only other system that would suffer is Win32, on which getaddrinfo is thread-safe if you compile with VC.NET. |
|||
msg12495 - (view) | Author: Rob Green (rbgrn) | Date: 2002-09-30 08:40 | |
Logged In: YES user_id=590105 Ok, well since it is confirmed fixed in FreeBSD 5.0, maybe the next release of python should consider it thread-safe. |
|||
msg12496 - (view) | Author: Jeremy Hylton (jhylton) | Date: 2002-09-30 13:56 | |
Logged In: YES user_id=31392 I should also take a look at the thread-safe implementation for FreeBSD. Perhaps it can be mined to make the default implementation that ships with Python thread-safe. |
|||
msg12497 - (view) | Author: Rob Green (rbgrn) | Date: 2002-09-30 17:22 | |
Logged In: YES user_id=590105 Here is a response from one of the FreeBSD developers working on resolver functions: Of course, HEAD branch has some newer implementation. However, basic facility of getaddrinfo(3) is same between branches. getaddrinfo(3) is expected as thead-safe from begining. This requirement is described in RFC2553: > The official specification for this function will be the final POSIX > standard, with the following additional requirements: > > - getaddrinfo() (along with the getnameinfo() function described > in the next section) must be thread safe. And, I believe getaddrinfo(3) itself is written in thread-safe manner. But, res_* functions is still not thread-safe. Since no one tries to audit if getaddrinfo(3) in FreeBSD is thread-safe, yet, I dunno if it is actually thread-safe. In any way, if getaddrinfo(3) is not thread-safe, we should fix it. |
|||
msg12498 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2003-06-14 14:52 | |
Logged In: YES user_id=21627 This was fixed with patch #731644 for Python 2.3. Please have a look at the implementation; FreeBSD is considered as a system which does not have a thread-safe getaddrinfo. If some release are thread-safe and some are not, patches telling those apart would be appreciated. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-10 16:05:42 | admin | set | github: 37222 |
2002-09-26 04:44:20 | rbgrn | create |