In order to maintain threaded code uncomplicated (VHL)
and in order to avoid massive use of locks, when
introducing few critical non-atomic sections, it should
be possible to put out that practical hammer ..

    try:
        last_ci=sys.setcheckinterval(sys.maxint)
        critical_function()  # now runs atomically
    finally:
        sys.setcheckinterval(last_ci)
    #(sys.setcheckinterval assumed to return the last
value)


..by an official function/class (in sys or thread):


Maybe:
======

atomic = sys.setatomicexecution(mode)
try:
    print "Executing critical section"
finally:
    sys.setatomicexecution(atomic)

there should be different modes/levels for blocking :

* 0=None
* 1=block only python execution in other threads
* 2=block signals
* 4/5/7/15=block threads at OS level (if OS supports)
* 8=(reserved: block future sub-/stackless switching
inside current thread..)

see:
http://groups.google.de/group/comp.lang.python/browse_frm/thread/bf5461507803975e/3bd7dfa9422a1200

compare:
http://www.stackless.com/wiki/Tasklets#critical-sections

---

Also Python could define officially its time atoms
beyond CPU atoms in the docs (That also defines the VHL
usability of the language).

Thus thinks like single-element changes ...

obj.var=x  , d[key]=x , l.append(x) .pop()

should be guaranteed to work atomic/correct FOR EVER. 

l[slice],  d.keys() .items(), .. questionable? 
If not guaranteed for the future, offer duplicates for
speed critical key building blocks like: 
l.slice_atomic(slice), d.keys_atomic() ,...  in order
to make code compatible for the future. 

---

Extra fun for blowing up python libs for poeple who
don't want to learn that try..finally all the time:

copy/deepcopy/dump maybe could be enriched by
copy_atomic , deepcopy_atomic, dump_atomic - or just
RuntimeError tolerant versions,   deepcopy_save  (no
use of .iterxxx)

Logged In: YES 
user_id=972995

or:

>>> atomic = sys.acquireatomicexecution(mode)
>>> try: 
... 	print 'critical section'
... finally:
... 	atomic.release()

>>> with sys.acquireatomicexecution(mode):
...     print 'critical section'

Logged In: YES 
user_id=884167

- sys.setcheckinterval(sys.maxint) does not prevent 
  thread switching when doing IO, does it? There is
  no way that I know of to prevent thread switching
  in this situation.

- When calling back into Python from C Code, there is
  no way to tell Python not to relinquish the GIL; such
  an option could be useful for some C code, because
  the GIL provides a "global critical section". 

Martin

Logged In: YES 
user_id=972995

thus the GIL could simply have a harder state 2 : "locked
hard for PyEval_AcquireThread/PyEval_AcquireLock/.." ?

Only PyEval_RestoreThread gets the lock again after
PyEval_SaveThread.

Robert

Logged In: YES 
user_id=972995

... only PyEval_RestoreThread with the harder execution
level in its tstate

Logged In: YES 
user_id=1038590

Raymond brought this idea up on python-dev at the time of
the c.l.p discussion - it was rejected on the basis that
thread synchronisation tools (Queues and Locks) are provided
for a reason. Python level access to the Global Interpreter
Lock is neither necessary nor desirable.

Avoiding the tools provided to permit threading to work
correctly and then finding that threaded code doesn't work
as desired really shouldn't be surprising.

FWIW, Python 2.5 aims to make normal locking easier to use
by permitting:

from __future__ import with_statement
from threading import Lock

sync_lock = Lock()

def my_func(*args, **kwds):
    with sync_lock:
        # Only one thread at a time can enter this section
        # regardless of IO or anything else
    # This section, on the other hand, is a free-for-all

If you genuinely have to avoid normal thread synchronisation
primitives, you can abuse (and I really do mean abuse) the
interpreter's import lock for this purpose:

imp.acquire_lock()
try: 
    print 'critical section'
finally:
    imp.release_lock()

Or even:

@contextlib.contextmanager
def CriticalSection()
    imp.acquire_lock()
    try:
        yield
    finally:
        imp.release_lock()

with CriticalSection():
    print 'critical section'

Logged In: YES 
user_id=1038590

On the other changes you suggest (which Raymond didn't bring
up on python-dev):

Python can't formally define as atomic any operations that
may execute arbitrary Python code, as the interpreter cannot
control what that code may do. All of the examples you give
are in that category.

Slowing down the common cases (unthreaded code and
Queue-based threaded code) by adding internal locking to
every data structure is also considered highly undesirable.

Logged In: YES 
user_id=884167

Nick Coghlan wrote:
> Python level access to the Global Interpreter Lock is 
neither necessary nor desirable.

I agree. But what I think would be very useful (as I 
already mentioned below) is C level API access that tells 
Python not to relinquish the GIL during a callback. 

I don't know of any other way to achieve this, apart from 
introducing a separate C-level locking mechanism. 

I think Tim Peters thinks thought this is feasible when he 
wrote the following (private conversation, June 2005):

> It would be more feasible to patch Python, introducing 
> a way to tell the interpreter not to allow other 
> _Python_ threads to run. It has such a mechanism 
> already (the GIL -- the global interpreter lock), but 
> as soon as C code calls back into Python, the interpreter
> may release the GIL and give ownership of it to a 
> different thread. I think it would be pretty easy to 
> introduce a Python C API to say "don't allow other Python
> threads to run until I tell you otherwise". 

Best regards, Martin Gfeller

Logged In: YES 
user_id=884167

Should I enter a separate feature request for this, 
separating it from this closed item?

Martin

Logged In: YES 
user_id=884167

[I split off a new item 1497532, asking for a C level API, 
so this item can remain closed.]