i have a c++ windows application with embedded python
interpreter.

this code crashs:

Py_Initialize();
PyRun_SimpleString("import gc");
Py_Finalize();
Py_Initialize();
PyRun_SimpleString("import gc");	// <--- BOOM


the problem seems to be the global var "garbage" in
gcmodule.c
the var isn't cleared in the reinitialization and
becomes an invalid pointer.
setting it to NULL in Py_Finalize fixes the crash.
i use python version 2.3.4 but a short look at the
2.4.1 source indicates that the problem is still there.

Logged In: YES 
user_id=6656

Bizarrely, I think this got fixed in CVS HEAD just a few days ago.  Are you 
in a position to check that?

Logged In: YES 
user_id=693638

yes, Neil Schemenauer (nascheme) added a Py_INCREF(garbage)
to initgc.
that *MIGHT* fix the crash. but only by simply never
releasing the garbage list. the garbage variable is still
never reset to NULL and therefore no new garbage list is
created after reinitialization.
besides: the incref is only in gcinit, but the list can be
created in gcinit OR in handle_finalizers (and while i was
looking at it in the debugger it always was handle_finalizers).

i guess the proper way to fix it would be a cleanup function
for the gcmodule that is called by Py_Finalize. the cleanup
function should do a decref (if Neils incref stays in, i'm
not sure if it is needed) and then reset the global vars.

Logged In: YES 
user_id=6656

At this point you might be thinking that Py_NewInterpreter/
Py_EndInterpreter loops are not the best tested things in the world.  You'd 
be right.

Neil's fix *does* fix the crash, but also shares PyObject's between 
interpreter states, which is a bit of a no-no, but otoh, this problem is far 
from unique to gcmodule.c.

> besides: the incref is only in gcinit, but the list can be
> created in gcinit OR in handle_finalizers (and while i was
> looking at it in the debugger it always was handle_finalizers).

That slightly misses the point -- the reason you got a crash was because 
the PyModule_AddObject initgc stole a reference to garbage, so when the 
module got cleaned up, it got deallocated despite there still being a 
reference to it in the module level global.  If initgc is never run, the 
refcount of garbage stays at 1.

> i guess the proper way to fix it would be a cleanup function
> for the gcmodule that is called by Py_Finalize.

Yeah, but quite a lot of modules could do with that, I think.  Do you have 
the energy to work on a 'proper' solution?  I don't.

Logged In: YES 
user_id=35752

Well, the fact that some things are broken is not a good
excuse to introduce more brokeness.  I guess decrefing the
gc.garbage list and letting any reference cycles contained
in it leak is better than sharing objects between interpreters.

The following still crashes (python 2.5.1):

        for (int i=0; i<1000; ++i)
        {
                Py_Initialize();
                PyRun_SimpleString("import tarfile\n");
                Py_Finalize();
        }

Bindings such as Swig are adding weird hacks just for avoiding the
finalize call and resetting the variables. Not being able to start from
a clean interpreter is unacceptable.

mwh, my threading patch is extensive enough and has enough overlap that
I'm not intimidating by fixing this.  It's low on my list of priorities
though.

So far my tendency is to rip out multiple interpreters, as I haven't
seen what it wants to accomplish.  It's certainly *NOT* completely
isolated interpreters.

As for Py_Finalize, I see 4 purposes to calling it:
1) Finish any lingering bits of the program, such as __del__ methods or
weakref callbacks
2) Delete lingering references so refcount errors or memory leaks may be
more easily discovered
3) Prepare for Py_Initialize to be called again
4) Return the interpreter to a quiescent state such that python.so can
be safely unloaded, then reloaded and Py_Initialize called again

Of these, 3 and 4 are not done.  3 would be easy if Py_Finalize was a
no-op, but that'd undermine 1 and 2.  Given the optimistic
documentation, it's surprising how limited Py_Finalize's scope is.

I've seen some fleeting discussion of allowing extension modules to be
unloaded, which IMO requires returning them to a quiescent state.  Doing
that may lay the foundation for doing it to the entire interpreter.

Anyway, real use cases are critical here.  Although you may think you
want your plugins to unload python, the costs mean you should only do so
very rarely, enough that there's really no advantage to unloading it
(except at program shutdown, for purposes 1 and 2.)

I find it surprising that something flagged up as causing a crash hasn't had a little more TLC, but is this still a genuine problem?

The initial test case was apparently fixed. It is not clear to me that the second, msg57834, is a supported use case. Nick, I know you are busy, but there is no C api category in the experts list. Any comment?

Unless someone can identify a bug in current 2.7 or 3.4+ or propose an enhancement for 3.5, we should close this. gc has been enhanced since 2.4, 2.5, and I would not expect anything said about those versions to apply to 3.4+.

Nick, please read my previous msg if you can find a moment.

I don't actually know the internal details of the GC - adding Antoine based on the list of interested devs for the gc module.

Someone interested in this issue should first check whether it still applies to 3.4 or 3.5.

I am closing since the initial problem has been fixed and there is no known current problem to fix.  If someone does discover one, they can reopen or, probably better, open a new issue.