This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: gethostbyname() blocks when threaded
Type: Stage:
Components: Library (Lib) Versions:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: jhylton, loewis, rbgrn
Priority: normal Keywords:

Created on 2002-09-26 04:44 by rbgrn, last changed 2022-04-10 16:05 by admin. This issue is now closed.

Messages (16)
msg12483 - (view) Author: Rob Green (rbgrn) Date: 2002-09-26 04:44
I know this is because it is simply a wrapper for the libc 
gethostbyname, but it really, really hurts my app to have 
it block like this. I am requesting that by any means 
necessary, gethostbyname() not block when multiple 
threads call it simultaneously. It seriously ruins the 
ability to write effective network daemons in python, as it 
can take a very, very long time to get through even a 
small list of hostnames if one or a few of them need to 
timeout to be resolved.
msg12484 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2002-09-26 16:06
Logged In: YES 
user_id=21627

Can you please report what platform you are using?
msg12485 - (view) Author: Rob Green (rbgrn) Date: 2002-09-26 16:09
Logged In: YES 
user_id=590105

FreeBSD 4.5, although I don't think it's platform specific, as 
I've seen some messages from guido on certain mailing lists 
talking about this problem and how it will require a large 
rewrite to fix correctly. In all honesty, I'd be happy with a 
simple hack to make this work and then a rewrite of whatever 
needs it in the future to have it written correctly.
msg12486 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2002-09-26 16:29
Logged In: YES 
user_id=21627

Bad choice of system :-) In general, it would be the easiest
modification of the interpreter to just release the GIL
before getaddrinfo (which gethostbyname uses); this is a
simple change and makes everything free running.

Unfortunately, on BSD, getaddrinfo is not thread-safe. So
for this system, a much more complicated change must be
implemented. I have semi-approved any change that releases
the GIL unconditionally on "good" systems (Solaris, Linux,
Tru64). That won't help you, though.

If you can contribute a patch that adds a getaddrinfo lock
for "bad" systems, and avoids this lock on "good" systems,
there is a small chance that this can get included in 2.2.2.
Without anybody acting quickly, this may or may not get
fixed in 2.3.
msg12487 - (view) Author: Rob Green (rbgrn) Date: 2002-09-26 17:28
Logged In: YES 
user_id=590105

This is a little over my head, but I'll take a look to see what 
you mean. Could you give me any more info to point me in 
the right direction? Thanks
msg12488 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2002-09-26 18:14
Logged In: YES 
user_id=21627

socketmodule.c contains a number of calls to getaddrinfo and
getnameinfo. On systems where those calls are not
thread-safe, you should add a lock around them, see Python
2.1, socketmodule.c, gethostbyname_lock. On all systems, you
should use Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS;
again, see Python 2.1.

There probably should be a single lock around both
getaddrinfo and getnameinfo. By default, those should be
considered thread-safe, giving explicit exceptions: *BSD,
and the fallback implementations (i.e. if either
getaddrinfo.c or getnameinfo.c is included).
msg12489 - (view) Author: Rob Green (rbgrn) Date: 2002-09-29 19:05
Logged In: YES 
user_id=590105

I've done some testing and I partially see what you are talking 
about. I wrote a test case in C using pthreads and in 
FreeBSD 4.5 I was able to resolve tens of hostnames 
simultaneously with no errors or problems. Python of course 
has the lock wrapped around the call, and Java appears to as 
well. Should I expect problems with this? Or might this be an 
indication that somewhere along the lines in freebsd 
development this issue was resolved but never 
tested/incorporated with Python/Java? If this is the case, then 
I suppose we would need to figure out at what version this 
changed and have the python lock act accordingly.
msg12490 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2002-09-29 21:31
Logged In: YES 
user_id=21627

As for FreeBSD bugs: Please consult the BUGS section of
getaddrinfo(3). It may be that this man page is incorrect, I
don't know.
msg12491 - (view) Author: Rob Green (rbgrn) Date: 2002-09-29 23:19
Logged In: YES 
user_id=590105

http://orange.kame.net/dev/query-pr.cgi?pr=277

There's the bug. It was submitted over 2 years ago, and I'm 
willing to bet it has been fixed but their bug database isn't up-
to-date. I ran the test code and it passed on my machine, 
meaning it's fixed in my version, so I emailed the author to 
find out what version this was fixed in. 
msg12492 - (view) Author: Rob Green (rbgrn) Date: 2002-09-29 23:45
Logged In: YES 
user_id=590105

http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libc/net/name6.c

It appears as though this was made thread-safe in revision 
1.17 of the file which was Jan 24, 2001. Unfortunately there 
are two branches of code here, and I have no idea what's 
going on with them. I apparently have code from 2000 that 
doesn't appear to be thread-safe but the test worked so I'm 
gonna send around some emails and see if I can't clear 
anything up.
msg12493 - (view) Author: Rob Green (rbgrn) Date: 2002-09-30 02:12
Logged In: YES 
user_id=590105

Here's what I found out. The thread-safe version is available in 
FreeBSD-Current, which will be Release and Stable in v5.0. 
So you can take out the lock for FreeBSD 5.0 and up, and 
that should be incorporated into the next Python release, as 
it has been available for FreeBSD-Current users for a year 
now.
msg12494 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2002-09-30 07:30
Logged In: YES 
user_id=21627

Thanks for the research. 

We'll leave the bug open until this is actually fixed. It
would be still possible to implement a patch that replaces
the GIL with a getaddrinfo lock on systems where getaddrinfo
is not thread-safe, but I feel no urge to implement such a
fix now. The only other system that would suffer is Win32,
on which getaddrinfo is thread-safe if you compile with VC.NET.
msg12495 - (view) Author: Rob Green (rbgrn) Date: 2002-09-30 08:40
Logged In: YES 
user_id=590105

Ok, well since it is confirmed fixed in FreeBSD 5.0, maybe 
the next release of python should consider it thread-safe.
msg12496 - (view) Author: Jeremy Hylton (jhylton) (Python triager) Date: 2002-09-30 13:56
Logged In: YES 
user_id=31392

I should also take a look at the thread-safe implementation for 
FreeBSD.  Perhaps it can be mined to make the default 
implementation that ships with Python thread-safe.
msg12497 - (view) Author: Rob Green (rbgrn) Date: 2002-09-30 17:22
Logged In: YES 
user_id=590105

Here is a response from one of the FreeBSD developers 
working on resolver functions:

Of course, HEAD branch has some newer implementation.  
However, basic facility of getaddrinfo(3) is same between 
branches.

getaddrinfo(3) is expected as thead-safe from begining. This 
requirement is described in RFC2553:

>   The official specification for this function will be the final 
POSIX
>   standard, with the following additional requirements:
>
>      -  getaddrinfo() (along with the getnameinfo() function 
described
>         in the next section) must be thread safe.

And, I believe getaddrinfo(3) itself is written in thread-safe 
manner. But, res_* functions is still not thread-safe.  Since no 
one tries to audit if getaddrinfo(3) in FreeBSD is thread-safe, 
yet, I dunno if it is actually thread-safe. In any way, if 
getaddrinfo(3) is not thread-safe, we should fix it.
msg12498 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2003-06-14 14:52
Logged In: YES 
user_id=21627

This was fixed with patch #731644 for Python 2.3. Please
have a look at the implementation; FreeBSD is considered as
a system which does not have a thread-safe getaddrinfo. If
some release are thread-safe and some are not, patches
telling those apart would be appreciated.
History
Date User Action Args
2022-04-10 16:05:42adminsetgithub: 37222
2002-09-26 04:44:20rbgrncreate