This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: embedding in multi-threaded & multi sub-interpreter environ
Type: Stage:
Components: Extension Modules Versions: Python 2.3
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: atulkshirsagar, brett.cannon, loewis
Priority: normal Keywords:

Created on 2004-03-22 15:39 by atulkshirsagar, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
multiSub.zip atulkshirsagar, 2004-04-01 20:12 Sample project and source files
extModule.zip atulkshirsagar, 2004-04-01 20:14 Extension module project and source files
consoleop.txt atulkshirsagar, 2004-04-01 20:17 Console output demonstrating the error
Messages (6)
msg20287 - (view) Author: Atul (atulkshirsagar) Date: 2004-03-22 15:39
I am embedding python in my C++ application. I am 
using Python *2.3.2* with a C++ extention DLL in multi-
threaded environment. I am using SWIG-1.3.19 to 
generate C++ to Python interface.

Now to explain it in details,
1. Python initialization [Py_Initialize()] and finalization 
[Py_Finalize()] is done in the *main* thread.
2. For each new thread I create a separate sub-
interpreter [Py_NewInterpreter()]. 
3. Using PyRun_String("import myModule"...) before 
execution of python script, extention module is imported.
4. Each thread executes *multiple* python script using 
PyEval_EvalCode() using the class objects in my 
extention DLL.
5. Each sub-interpreter is destroyed [Py_EndInterpreter
()] at the end of that particular thread.

I am observing that;
As explained above when multiple threads are running. 
And as one of these threads finishes, in other running 
threads I start getting 
"TypeError: 'NoneType' object is not callable" error on 
the methods called on class objects in extention module. 

The same code *works fine* with Python 2.2.2.

I have found these links more or less talking about the 
same problem
migrating from 2.2 to 2.3.
http://mail.python.org/pipermail/python-dev/2003-
September/038237.html
http://mail.python.org/pipermail/python-list/2004-
February/206851.html
http://mail.python.org/pipermail/python-list/2004-
January/204040.html

I *guess* what is happening is global variables are 
zapped to "NoneType"
when one thread finishes and other thread trying to 
access them through the
Python script (step 4.) this error is generated. But it 
*works* sometimes
when(*guess*) the running thread is at step 3. and by 
importing the  module
the global variables are re-initialized for "Type" 
information.

I tried using reload(myModule) to solve the problem but 
that is generating
big memory leak every time it is called.

Is this a know issue with 2.3 (interpreter?) ? Or is there 
a change for 2.3 in the way embedding should be done 
in a multi-threaded and multi-sub-interpreter 
environment ?
msg20288 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2004-03-29 22:28
Logged In: YES 
user_id=21627

Can you provide an example that demonstrates the supposed bug?
msg20289 - (view) Author: Atul (atulkshirsagar) Date: 2004-04-01 20:12
Logged In: YES 
user_id=1003784

Hello Martin,

I am attaching the sample program which demonstrates the 
issue;
1. I compiled it on Windows 2000 with VC++ compiler.
2. There is a "Extension Module" [flpythonmodulesu(d).dll] 
which is used by "Main" program

=========================================
Extension Module (extModule.dsp project) 
generates : (*u.* - release, *ud.* - debug)
1.  dll _flpythonmodulesud.dll
2. python script flpythonmodulesud.py 
------------------------------------------------------
1. Module has class FlPythonString and class Record defined
2. Using interface file include\flpythonmodulesud.i, SWIG-
1.3.19 generates include\flpythonmodulesud_wrap.cxx 
===========================================


=========================================
Main sample program (multiSub.dsp project) 
generates : 
1.  executable multiSub.exe
------------------------------------------------------
1. Links in the extension DLL
2. Initializes and De-initializes python in main thread
3. Main thread creates 2 separate threads: 
   (a) Each thread creating a new sub-interpreter
   (b) importing extension module
   (c) Calling following EXPRESSION using PyEval_EvalCode(),
       Thread 1:
myVar = FlPythonStringPtr(STRObject)
myRecord = myVar.NewRecord()
Thread=str(myVar.Get())
record=str(myRecord.GetValue())
print 'Inside EXPRESSION ' + Thread + ':' + 
record;myVar.DeleteRecord(myRecord)
       
    Thread 2:
myVar = FlPythonStringPtr(STRObject)
for recordNum in range(1, 11):
 myRecord = myVar.NewRecord()
 Thread=str(myVar.Get())
 record=str(myRecord.GetValue())
 print 'Inside EXPRESSION ' + Thread + ':' + 
record;myVar.DeleteRecord(myRecord)
      (d) Destroying sub-interpreter.

Comments in the code explain details about each step.
===========================================

==========================================
Console Output messages explanations
(Attaching a sample console output for the issue
[consoleop.txt])
-----------------------------------------------------
-Thread *ID*:IMPORT Module Called:*ITERATION*
-Inside EXPRESSION *Thread*:*Record::GetValue()
[hardcoded to 1000]*
-Thread *ID*:EXPRESSION Done:*ITERATION*
-Thread *ID*:ENDINTERPRETER Called
-Thread *ID*:EXPRESSION Failed!:*ITERATION*

Check the error message demonstrating the issue:
-----------------------------------------------
"Thread 2:EXPRESSION Failed!:3
Traceback (most recent call last):
  File "EXPRESSION", line 4, in ?
  
File "E:\VssLocal\Isis\python23_issue\multiSub\Debug\flpython
modulesud.py", line 69, in Get
    def Get(*args): return apply
(_flpythonmodulesud.FlPythonString_Get,args)
  
File "E:\VssLocal\Isis\python23_issue\multiSub\Debug\flpython
modulesud.py", line 51, in __init__
    _swig_setattr(self, Record, 'this', this)
TypeError: 'NoneType' object is not callable"

==========================================



Martin, Thanks for taking a look at the issue which has proved 
to be a major hurdle for our release and made us switch back 
to Python 2.2.2. Python 2.2.2 in itself has some issues for us; 
so if this can get fixed we would desperately be seeking a 
patch on Python 2.3.

Let me know if you need any more inputs from my side.

Thanks again,
Atul
msg20290 - (view) Author: Atul (atulkshirsagar) Date: 2004-04-05 20:16
Logged In: YES 
user_id=1003784

Hello,
Did you get a chance to take a look at this yet ?
Thanks,
Atul
msg20291 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2004-08-07 19:46
Logged In: YES 
user_id=357491

Have you tried this with the fixes added in 2.3.4 or 2.4?  Should print out 
proper tracebacks now.
msg20292 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2004-09-24 23:26
Logged In: YES 
user_id=357491

OK, since the OP has not responded to this and I suspect this is the same 
bug that got squashed a while back with exceptions not getting reported 
at teardown time by threads, I am closing this as outdated.
History
Date User Action Args
2022-04-11 14:56:03adminsetgithub: 40062
2004-03-22 15:39:10atulkshirsagarcreate