This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: embedding Python causes memory leaks
Type: Stage:
Components: Interpreter Core Versions:
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: andykt, loewis, nnorwitz, suresh_sf
Priority: normal Keywords:

Created on 2006-03-07 23:20 by andykt, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (9)
msg27700 - (view) Author: Andrew Trevorrow (andykt) Date: 2006-03-07 23:20
[This bug has been submitted by others but for some reason it
has been marked Closed.  I consider it to be an extremely serious
bug -- if I can't solve it I'm going to have to abandon Python as
my app's scripting language, even though I've fallen in love!]

I've added Python script support to my cross-platfom wxWidgets app
so that users can run .py scripts from within the app to automate the
GUI and do other fancy things.  It all works very nicely, except for
one nasty problem: *every* time a script is run there is a memory leak,
usually small (about 10K) but sometimes massive (about 4MB in the 
case of one rather complicated script).

The problem occurs on both Mac OS 10.3.9 and Windows 2000.
I'm using Python 2.3 on the Mac and 2.4.2 on Windows.

Every time the user runs a script, my app makes these calls:
(I've removed a lot of irrelevant stuff.)

   Py_Initialize();
   PyRun_SimpleString("execfile('foo.py')");
   Py_Finalize();

It's definitely not a wxWidgets problem.  In fact it's quite easy to
see the memory leak using a simple command-line program:

#include <stdio.h>
#include <Python.h>
main(int argc, char *argv[])
{
   int i;
   for (i=0; i<1000; i++) {
      Py_Initialize();
      Py_Finalize();
      printf(".");
      if ((i+1) % 50 == 0) printf("\n");
   }
}

Note that it doesn't even execute a script.  If I run this program on
my Mac and watch its memory usage with Activity Monitor, I see a leak
of about 10K each time through the loop.  Similar result on Windows.

Curiously, on both machines, the Py_Finalize() call takes longer and
longer to complete whatever it's doing.  The above program takes a
few *minutes* to complete on my 400MHz Mac.

Andrew
msg27701 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2006-03-08 08:32
Logged In: YES 
user_id=33168

Why do you call Py_Initialize/Py_Finalize more than once? 
Why not do something like this (I'm kinda mixing C and
Python for convenience):

 /* startup */
 Py_Initialize();

 /* do whatever */
 while (moreFiles()) {
   PyRun_SimpleString("execfile('%s')" % nextFile());
   /* do whatever */
 }

 /* shutdown */
 Py_Finalize();
msg27702 - (view) Author: Andrew Trevorrow (andykt) Date: 2006-03-08 09:46
Logged In: YES 
user_id=1281947

> Why do you call Py_Initialize/Py_Finalize more than once?

How else do I tell Python to free up memory after each PyRun_SimpleString
call?

I want users to be able to run scripts many times from within my app.
If I just keep calling PyRun_SimpleString then my app will leak more and
more memory until it becomes unusable.

From http://docs.python.org/api/embedding.html:

  Sometimes, it is desirable to ``uninitialize'' Python. For instance, the
  application may want to start over (make another call to Py_Initialize())
  or the application is simply done with its use of Python and wants to free
  all memory allocated by Python. This can be accomplished by calling
  Py_Finalize().

That's exactly what I want to do.  I want the interpreter to run a script
and then release all the resources used by that script.  Unfortunately,
Py_Finalize does *not* restore memory usage to what it was before the
Py_Initialize call.  I wouldn't mind if there was a one-off allocation
cost (the 1st time Py_Initialize is called), but my app is leaking more
memory *every* time a script is run!
msg27703 - (view) Author: Andrew Trevorrow (andykt) Date: 2006-03-08 09:50
Logged In: YES 
user_id=1281947

Bloody hell -- sorry for the bad line breaks.
One day I'll figure out how to use sf properly!
msg27704 - (view) Author: Andrew Trevorrow (andykt) Date: 2006-03-10 05:43
Logged In: YES 
user_id=1281947

See http://evanjones.ca/python-memory.html for some
useful info.  Apparently the Python memory allocator never
releases memory back to the OS!  So if a complicated script
happens to consume 100MB of interpreter memory then
that amount is no longer available to the app in which
Python is embedded.  Even worse, if a script has a
(Python) memory leak then there's nothing the app can
do about it.  It would be great if wrapping each script
inside Py_Initialize/Py_Finalize could avoid all that.

There should be some way to tell Python "release all
the memory you've ever allocated and start again
with a clean slate".

msg27705 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2006-04-13 07:29
Logged In: YES 
user_id=21627

That the documentation claims Py_Finalize releases all
memory is a bug; I just fixed this in r45344. The original
problem cannot be fixed (atleast not until Python 3000);
closing it as "won't fix".
msg27706 - (view) Author: suresh (suresh_sf) Date: 2007-06-06 06:22
I too am having the exact similar problem with embedded python. Is this a expected behaviour? if so how can we use python for long running tasks without running out of memory eventually? We even tried doing Py_Initialize once during startup and executing scripts before calling Py_Finalize. Even this runs out of memory after some time. BTW, I am running Python 2.5.1.
msg27707 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2007-06-06 18:12
If you are calling Py_Finalize multiple times, (small) memory leaks are expected. So don't call Py_Finalize until your program terminates.
msg27708 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2007-06-07 02:19
Also, it is quite common for third party C extension modules (those not distributed with Python) to contain memory leaks.  If you are using any third party extension, you should investigate it very carefully to verify it is not leaking memory.  You can use various memory debuggers to help find this problem.  Valrgind and purify come to mind.
History
Date User Action Args
2022-04-11 14:56:15adminsetgithub: 42993
2006-03-07 23:20:09andyktcreate