[Split off from closed 1453341, which desired a Python-
level API. This request is for a C API, to be used in 
C extensions that call back into Python.]

I think it would be very useful to have an API that 
would tell Python not to release the GIL during call 
backs to Python code, until it is told again (back in 
C code) that it is ok to do so.

I'd like to quote Tim Peters, who explains it better 
than me (private conversation, June 2005):
----------
It would be more feasible to patch Python, introducing 
a way to tell the interpreter not to allow other 
_Python_ threads to run. It has such a mechanism 
already (the GIL -- the global interpreter lock), but 
as soon as C code calls back into Python, the 
interpreter may release the GIL and give ownership of 
it to a different thread. I think it would be pretty 
easy to introduce a Python C API to say "don't allow
other Python threads to run until I tell you 
otherwise". 
----------

The Use Case arose with a multi-threading problem in 
ZODB's Connection, where the C code must ensure that 
no other thread runs Python code accessing an object 
BUT the own thread needs to call back into Python. 

Any other mechanism that came to my mind would either 
require to halt *all* others threads, or run into race 
conditions. 

Best regards, Martin Gfeller

Logged In: YES 
user_id=21627

Would you like to work on a patch for that?

Logged In: YES 
user_id=884167

This would unfortunately be way beyond my competence in C 
programming. I would only be able to change the existing 
code to use the API, if it were available. Sorry! Martin

I'm not sure if such a API is feasible. The very nature of Python makes
it impossible to tell in advance what the interpreter will do when
getting called. This is even true for simple functions - think of your
function getting decorated...

Let's consider the following scenario:

- Python-Thread 1 is already running and owns a lock on some object.
While it still owns the lock, it releases the GIL.
- We are executing in Python-Thread 2 and call from Python to C. The C
function has the GIL, "locks" it and calls back to Python.
- The Python function executes in the same thread, still having the GIL.
It tries to acquire the lock on the same object as Thread 1. Preventing
a deadlock between those two threads, it releases the GIL and waits for
the object-lock.
- The GIL is "locked" to the current thread and the current thread is
the only one that we can allow to unlock it again; this narrows our
options down to Py_BEGIN_ALLOW_THREADS becoming a No-Op in such situations.
- Py_BEGIN_ALLOW_THREADS executed in the second C-function is silently
ignored. Thread 2 waits for the object-lock with Thread 1 never having a
chance to release it.
- The interpreter just deadlocked.


AFAICS we can't guarantee not to deadlock if there are other threads
running before we lock the GIL to the current thread.

Lukas, I'm afraid to admit you're right :-;. 

Assuming that the Python code called under the "not release the GIL" 
regime would not do anything that could be potentially blocking is 
probably dangerous and could result in an unstable interpreter. 

So I withdraw my feature request and thank you for your efforts and 
interest. 

Best regards, Martin