I have an embedded use of the Python interpreter. I
want to compile and link the encapsulating application with
and without DEBUG. I have disabled the use of the
pragma in config.h to always use the library specified
in the link statement, which is always the standard
python library. I have ensured that all invocations of
Python.h undefine the DEBUG symbol before including them.

Compiling the application without debug: works fine if
you link non-debug. However, linking with debug causes
some change in the way python modules are looked up and
I'm getting undefined local modules. I have checked the
PYTHONPATH environment variable and it is correct, and
the .py module exists in the correct directory. (This
is a common scenario where some libraries are compiled
without DEBUG, some with DEBUG, and then the whole
application linked DEBUG.)

Compiling the application with debug: Link errors
result. Undefined -- _Py_RefTotal, _Py_Dealloc,
_PyInitModule4TraceRefs.

Python is making too many hidden assumptions about the
build environment when it is being used embedded. This
is a critical problem as it makes it impossible to
build the encapsulating application for debug purposes.

Logged In: NO 

More info: I've now narrowed it down to the fact that when
the final link is done with debug, the PYTHONPATH
environment variable is ignored altogether. In my case, I
cannot rely on the registry being set up for Python, and the
system is resorting to the default relative path names
(which, in my case, are of no use).

How does the behavior of the (non-debug) python library know
to change based on the final link? And why is PYTHONPATH
ignored if it is linked in debug?

Logged In: YES 
user_id=21627

I cannot understand what bug you are reporting. Of the
things you report, which of them did you not expect?

For example, that Py_RefTotal is undefined when you compile
with DEBUG, but link with the no-debug library is not a bug,
but by design: those functions are used in every debug
module, but available only in the debug library.

Your complete compilation approach is flawed: If you compile
with pydebug, you absolutely *must* link with the debug
library: the object layout of every object will change, so
crashes are likely if you link with the wrong library.

Logged In: NO 

I'm not recompiling the Python libraries. I am only
recompiling my personal libraries which use the Python
C API. I want to be able to compile my libraries with
debug and use the standard Python library.

This is not a "flawed" approach. It is extremely common
when using 3rd-party libraries to do this. Python should
be using a separate #define for its ref-counting debug
stuff so that clients can use the standard library and
still debug their own stuff.

Refer to my second comment regarding the use of PYTHONPATH
for another example of non-transparent substitution of
the debug and non-debug libraries.

Logged In: YES 
user_id=31435

Changed Category to Windows, since I'm almost certain 
that's the missing info here.

Assigned to MarkH hoping he has a slick <wink> reply.  This 
kind of complaint comes up rarely but regularly, and I'm 
never sure what to say:  since I build Python from source, 
and always did, our Windows debug scheme is fine by me.  
But what are pure end-users expected to do?

Logged In: YES 
user_id=21627

AFAICT, even on Windows, defining DEBUG does not change
anything whatsoever. It is only when you define _DEBUG that
things start to change.

I'm not sure whether this is a bug or a feature: If you just
want to enable debugging in your application, just don't
define _DEBUG. Notice that MSVC does that for you unless you
tell it not to. AFAICT, _DEBUG is what Microsoft uses in the
libraries to indicate "enable all debugging code that you
have". So Python enables Py_DEBUG.

Logged In: NO 

The only debug-type symbol I am defining when compiling my
code is DEBUG. I'm not deliberately defining _DEBUG or
Py_Debug.

Logged In: YES 
user_id=21627

What Python version are you using? You mentioned that you've
changed the pragma in config.h; atleast since Python 1.5.2,
this macro depends on _DEBUG, not on DEBUG. So if you didn't
define _DEBUG, you wouldn't need to change the pragma.Python
never changes its behaviour based on whether DEBUG is
defined, AFAICT.

Please check the project settings for the debug target in
your application again.

Logged In: YES 
user_id=31435

Well, Martin is right, you know:  if you're not defining 
_DEBUG, and not letting MSVC define it either, then neither 
will pyconfig.h define Py_DEBUG, and it gets harder and 
harder to believe that what you're reporting is the full 
truth.

Perhaps you could attach a tiny example, along with the 
MSVC project file, or Makefile, you use.  It may well be 
academic, though, as it's *normal* for _DEBUG to get 
defined under MSVC in a debug build (and whether or not you 
add it explicitly -- as Martin implied, you would have had 
to explicitly remove _DEBUG from the MSVC auto-generated 
list of preprocesser debug-build #defines).

In answer to one of your earlier questions, PYTHONPATH is 
not ignored after a debug link.  I understand that you 
believe it is, but I expect that's another illusion due to 
something you haven't told us-- or not discovered --yet.  
Perhaps you're linking with the wrong system libraries?  
Python requires the multithreaded libraries.

Logged In: NO 

It's not a question of *believing* that PYTHONPATH is
ignored: that is the observed behavior! (I am using
2.1.1.)

Whether I compile my client code with debug or not should
have nothing to do with what Python library I include.
(Note that I'm talking about *my* client code, not
compiling Python itself.) The Python #includes should not
be selecting a library for me, but should let me specify it
in my build environment. This is the way virtually every
3rd-party library is set up for both Windows and *nix
systems and the way I expected Python to behave. Whether
or not Windows defines _DEBUG shouldn't matter either --
I should have independent control. (I even when to the
lengths of undeffing DEBUG, _DEBUG, and Py_Debug prior to
including any Python headers -- still didn't help.)

Keep in mind that Python is not the core of my application.
It doesn't control the main loop. In fact I entirely
encapsulate the C API to Python so that my clients can
define commands without ever having to include Python.h
themselves. Python is just one piece of a very large system
(and a replacable piece at that) and can't dictate build
or compile requirements.

I'll check the system libraries, but I'm pretty sure they
are the multithreaded versions. (I think I'd be seeing
different symptoms if I had the wrong system libraries.)
Unfortunately, I can't send you samples due to
confidentiality concerns. Rest assured, however, that
I am describing the problems accurately.

Try it yourself! Uninstall python, just retaining the
normal link libraries and dlls and the headers. Now build
an application using just the C API. Not as simple as it
sounds (or is with competing tools such as Tcl).

Logged In: NO 

It's not a question of *believing* that PYTHONPATH is
ignored: that is the observed behavior! (I am using
2.1.1.)

Whether I compile my client code with debug or not should
have nothing to do with what Python library I include.
(Note that I'm talking about *my* client code, not
compiling Python itself.) The Python #includes should not
be selecting a library for me, but should let me specify it
in my build environment. This is the way virtually every
3rd-party library is set up for both Windows and *nix
systems and the way I expected Python to behave. Whether
or not Windows defines _DEBUG shouldn't matter either --
I should have independent control. (I even when to the
lengths of undeffing DEBUG, _DEBUG, and Py_Debug prior to
including any Python headers -- still didn't help.)

Keep in mind that Python is not the core of my application.
It doesn't control the main loop. In fact I entirely
encapsulate the C API to Python so that my clients can
define commands without ever having to include Python.h
themselves. Python is just one piece of a very large system
(and a replacable piece at that) and can't dictate build
or compile requirements.

I'll check the system libraries, but I'm pretty sure they
are the multithreaded versions. (I think I'd be seeing
different symptoms if I had the wrong system libraries.)
Unfortunately, I can't send you samples due to
confidentiality concerns. Rest assured, however, that
I am describing the problems accurately.

Try it yourself! Uninstall python, just retaining the
normal link libraries and dlls and the headers. Now build
an application using just the C API. Not as simple as it
sounds (or is with competing tools such as Tcl).

Logged In: YES 
user_id=31435

Anonymous, editorializing doesn't really help.  I 
understand that you want Python to work in a way other than 
it does, and repeating that isn't necessary.  I'm hoping 
Mark Hammond will comment on that.

In the meantime, I'm trying to determine why you're seeing 
the symptoms you are seeing.  I believe you're seeing them, 
but your analysis doesn't make sense, therefore something 
is going on you haven't told us about.  What *should* 
happen in your ideal world isn't relevant to determining 
what's actually happening.  Take this one step at a time, 
please.

Logged In: NO 

You'll have to excuse the editorializing: I've found working
with the C API immensely frustrating in a variety of ways,
and then to have my empirical observations dismissed...

Let me step back and explain where I am right now.

I've built the 2.1 debug version of the python library using
the source distribution. When doing a final link without
debug (regardless of how all my client code was compiled)
I use the non-debug version of the Python library. When
doing a final link with debug (again, regardless of how all
of my client code is compiled) I use the debug version of
the python library. To get the latter scenario to work, I
edit the registry -- as well as the PYTHONPATH -- before
initializing python through the C API. For these two
cases, my unit tests run correctly to completion.

However, this means that our developers must compile the
library I wrote that wraps up the Python C API to match
how they are going to link it. I would like to find a way
around this constraint so that any combination of compiling
and linking my wrapper library works correctly.

(Let me also mention that I am aware that you are all
volunteers and I do appreciate your efforts. I am hopeful
that by resolving issues such as this Python can be made
more easy to use in a wider variety of circumstances.)

Thanks.

Logged In: YES 
user_id=21627

AFAICT, PYTHONPATH is *not* ignored when running a debug
build. If you think it is, please run the debug and the
nodebug interpreter (i.e. python.exe and python_d.exe), and
report the values of sys.path for either case.

More likely, the cause of your problems is that the import
code looks for different DLL name *in sys.path*. I.e. when
you do 
import foo
in a nodebug build, it will try to find foo.pyd or foo.dll.
In a debug build, it will try to find foo_d.pyd or
foo_d.dll. So if you only have foo.pyd, then importing foo
will fail if Python was compiled with _DEBUG. 

Again, this is by design: foo.pyd might crash the debug
interpreter, since the object layout is different.

If you merely want to single-step through the Python
runtime, without making use of the Py_DEBUG features, you
should build the release version of Python, but activate the
generation of debug symbols.

Logged In: YES 
user_id=14198

There is not much I can add here.  I understand and 
appreciate your concerns.  However, as Tim and Martin have 
stated, something is missing from the puzzle.  PYTHONPATH 
is *not* ignored in debug builds.

The reason for the _d is 2-fold:
* As Martin said, the object layout may change.
* The Python core and extensions sometimes share CRT 
objects - notably file handles, and in some cases memory 
blocks.  It is critical that these come from the same CRT.

I could possibly agree that the "_d" convention is dumb and 
should be dropped.  However, it is unclear to me that this 
bug can be simply summarised as this.  This is probably 
because it is unclear to me how this bug can be summarised 
at all :)

Logged In: NO 

I guess I'm not making myself very clear. Let me try again...

I have a library that completely encapsulates all use of
Python.h from my clients. I need to be able to debug this
library by compiling it with DEBUG. I do not want to use
the debug Python library. Unfortunately, just compiling
the Python headers with DEBUG seems to change the object
layout and require the use of the debug library.

BTW, the sys.path values include my PYTHONPATH values when
I link non-debug, and they are the default relative path
values when I link with debug. I know the code looks like
it doesn't ignore PYTHONPATH (I've looked through
getpathp.c too) but something is screwing up sys.path.

Logged In: YES 
user_id=21627

In your code that includes Python.h, can you please add the
lines

#ifdef _DEBUG
#error You must not define _DEBUG
#endif
#ifdef Py_DEBUG
#error For some reason Py_DEBUG is still defined
#endif

immediately after the
#include <Python.h>

line? Then please compile your library with DEBUG, and
report whether it compiles correctly; if not, please report
the errors it gives.

Furthermore, could you please evaluate the string

"import sys;print sys.path"

in your embedded code and report the exact output that this
gives when running with and without DEBUG?

Without the results of these specific experiments, I do not
think we can help much.

Logged In: NO 

Compiling without debug worked fine with the additional
#ifdefs. Compiling with DEBUG resulted in the _DEBUG
message and the Py_Debug message.


sys.path seems to be ok now in all configurations. One
of the (many) experimental changes to the build I did
must have bypassed the problem.

Logged In: YES 
user_id=21627

I'm glad the PYTHONPATH issue is resolved.

On the _DEBUG issue: Somebody *must* be defining _DEBUG
somewhere, or else the #ifdef would not trigger. Please open
the MSVC++ project file of your project (the .dsp file) in a
text editor, and search for _DEBUG. 

If you cannot find it there, please try to activate full
command line printing in your build process, so that you see
what options are passed to the compiler. 

If you are absolutely sure that _DEBUG is not passed in the
command line, it must be defined in one of the header files.
In this case, please put the #ifdef _DEBUG block of the
previous test after each #include, e.g. in a form

#include <stdio.h>
#ifdef _DEBUG
#error After stdio.h
#endif

#include <nextfile.h>
#ifdef _DEBUG
#error After nextfile.h
#endif

I'm sure there are better ways to find out where _DEBUG is
defined; don't hesitate to apply them if you are aware of any.

Once you found the location of _DEBUG, you may remove it to
be able to use the standard Python .DLL. 

Please make sure you understand Mark's comment on CRTs,
though: Many problems will occur if you mix the debug and
the nodebug version of the MSVC libraries (e.g. msvcrt.dll
and msvcrtd.dll). Please see the Microsoft documentation for
details: If you use pythonxy.dll, you absolutely *must* make
sure that you do not link the debug versions of the CRT,
even when performing a debug build. Unfortunately, you won't
get a compilation error if you do not follow this guideline;
instead, very hard-to-debug problems will occur. You can use
depends.exe on the final executable to analyse the set of
libraries used.

Logged In: NO 

Turns out the _DEBUG is defined when using the /MDd
compiler option.

We have painstakingly worked out our build system to
deal with the library conflicts you mention. If I
understand what you're saying, these are the rules:

Linking with debug CRT -> link with python_d
Linking with python_d -> compile client code with DEBUG

The first rule is thanks to MS. The second rule is to
get internal object layouts to match up.

Is this right?

Logged In: YES 
user_id=21627

The second rule should read

Linking with python_d -> compile client code with _DEBUG
since Python does not care whether DEBUG is defined or not;
it only cares about _DEBUG (i.e. /MDd, as you point out).

Furthermore, the second dependcy is also thanks to
Microsoft: If you compile your library with /MDd, you must
also compile all client code with /MDd, see MSKB Q140584).
Python currently supports two configurations on Windows:
1. Neither of _DEBUG, Py_DEBUG is defined; link with
pythonxy.dll and msvcrt.dll (i.e. /MD); extensions end with
.pyd or .dll
2. Both _DEBUG, Py_DEBUG are defined, link with
pythonxy_d.dll and msvcrtd.dll, extensions end with _d.pyd
or _d.dll.

It would be possible to create a third configuration:
_DEBUG is defined, but Py_DEBUG isn't. Link with a no-debug
pythonxy.dll, but with msvcrtd.dll. You couldn't use the
existing pythonxy.dll and existing extension modules, since
those already link with msvcrt.dll. Adding such a
configuration would increase the mess, so it won't happen.

I'd advise you to pursue a fourth configuration: Do not use
the debug CRT (i.e. compile with /MD instead of /MDd), even
if you compile your own library for debugging. With that
configuration, you can use the standard Python distribution
with no changes.

Logged In: YES 
user_id=31435

Martin and Mark, I want to play devil's advocate here just 
a bit:  How do we get away with, e.g., using expat.dll in 
Windows Python?  That is, we use the same expat.dll with 
the release and debug Python builds.  Nothing bad happens 
that I've seen, and we don't get warnings about mixing CRT 
types either.  IOW, we use expat.dll the way Anonymous 
would like to use the Python DLL.  How do we get away with 
that?

If that one is easy <wink>, ee don't use different versions 
of the Tcl/Tk libraries either.

Logged In: YES 
user_id=14198

The problem only occurs when the 2 DLLs must cooperate WRT 
CRT resources.  Eg, if a "FILE *" was ever passed and used 
across the expat->Python boundary, or a memory block 
allocated in one to be freed in the other, then we would be 
in the same situation.  This probably doesn't happen for 
expat and Tk.  For Python and its extension modules (and 
even embedders), it may.

IIRC, the problem with memory blocks was solved some time 
ago, leaving FILE* the only real problem.  File objects are 
used in only 1 or 2 cases, and these could be solved by 
moving to "PyFile_Write()" style APIs.

So, I guess I could agree a good solution is to ensure that 
Python C API does not require CRT resource sharing, and 
that Py_DEBUG is defined independently of _DEBUG or DEBUG.  
Individual extensions would be responsible for setting 
Py_DEBUG via their makefiles in debug builds.  This should 
be stable.

This would blur the _d convention.  I quite like it, but I 
agree it is a PITA for many people.  We could try giving 
pythonxx_d.dll the ability to load non _d versions, in the 
hope that the extensions have been "fixed" so they don't 
directly share the CRT.

Tim - never quite understood your feelings on _d - what are 
your thoughts?

Logged In: YES 
user_id=21627

I thought the problem with memory blocks still exists: If
you malloc with the debug CRT, and free with the nodebug
CRT, you get a leak. At least Q140584 claims the problem
still exists in 6.0sp5. In addition to memory and files,
they also notice that setting the locale in one CRT doesn't
affect the other.

I still cannot tell whether Anonymous builds a DLL wrapper
or a static library around pythonxy.dll.If it was a DLL, it
would be indeed equivalent to the pyexpat case. However, in
that case, linking Python wouldn't matter at all to the
clients of his or her DLL, since the reference to
pythonxy.dll/pythonxy_d.dll would be only in the wrapper DLL.

Re: decoupling _DEBUG and Py_DEBUG. How exactly would that
work? Would a _d extension imply _DEBUG, or Py_DEBUG, or
would we stop using _d? 

If it would imply _DEBUG, I cannot understand Mark's comment
"Individual extensions would be responsible for setting
Py_DEBUG via their makefiles in debug builds.  This should
be stable." You have to use Py_DEBUG consistently throughout
all modules because it changes the struct _object layout.
Mixing them certainly isn't stable.

Perhaps we could try to completely avoid changing
PyObject_HEAD depending on Py_TRACE_REFS, and make it a type
flag instead. In that case, you could, at run-time, mix
objects that do trace_refs, and objects that don't.

Logged In: YES 
user_id=14198

*sigh* - forgot that the object layout changed :(

Re malloc issues: they are indeed still a problem - but 
Python will not *force* you to do this.  It used to be the 
case that an extension module would malloc an object, and 
whatever module did the final Py_DECREF (often the core) 
would perform the free().  I believe this is no longer 
true.  It *is* still true with FILE objects and the 
tp_print slot IIRC.

My point with the "_d" was that iff we can get things 
working so that the standard Python extensions 
are "safe", "_d" need not imply anything other than "the 
preferred extension for debug builds of the core" - and 
even that is only for b/w compat and convenience reasons.

How has Linux survived so long?  I assume Py_DEBUG builds 
are simply never made in the "wild".

Logged In: YES 
user_id=21627

On Linux, it is common to include debug information into
every  library and executable; in particular, it is common
to install python with debugging information. This is
possible since debugging information can be combined with
optimization almost seamless in gcc (distributors often
strip the debugging information, though).

So you use Py_DEBUG *only* when you want to do the refcount
debugging. You have to configure --with-pydebug for that,
and you never do that in the wild. *If* you would install a
pydebug binary, you would use a different --prefix, so that
it would not interfere with the standard installation.
Depending on which pyconfig.h extensions pick up, they would
then be suitable for use with one or the other installation.

Logged In: YES 
user_id=31435

Martin, the MS docs also say the linker generates a warning 
if you mix CRT types -- but it doesn't for mixing debug 
Python with expat or Tcl/Tk.  I don't always believe 
everything in the MS docs <wink>.  But, Mark, I do believe 
them when they say you're on thin ice, and they really 
don't document the ways in which mixing CRTs can be 
disastrous (they give three specific examples, "just a few" 
of "many [unspecified] ways to get into trouble").

Mark, I like _d fine!  It's better than MS's plain "d" 
convention (which I've seen other projects mimic), and so 
long as there *is* a reason that mixing debug and release 
builds can be catastrophic, it's Good to make it painfully 
obvious that something potentially catastrophic is going on 
(note that Anonymous starting getting in trouble when 
he "disabled the use of the pragma in config.h" -- we spent 
most of this bug report trying to convince him the pragma 
wasn't there just to irritate him <wink>).

M and M, I don't *know* that expat.dll and Tcl/Tk work fine 
with a debug Python.  All I've ever done with expat.dll 
then is run our test suite, and that simply doesn't crash; 
and I probably haven't run any Tcl/Tk scripts under a debug 
Python in years.

Mark, I suspect debug Python builds are extremely rare on 
Unix, even among Python developers.  For example, for more 
than a year, I believe I'm the only one who has found test-
suite failures specific to the debug build.

But "compile for debugging" to Unixoids usually means some 
combination of not defining NDEBUG, retaining symbol info, 
and/or cranking down optimization.  The Unix libraries 
don't care about that stuff -- there's nothing there as 
Draconian as the MS-specific _DEBUG effect on libraries.

Martin, It Hurts to tell people not to use _DEBUG on 
Windows:  as a prime example, the MS _DEBUG malloc+free are 
indispensable debugging tools, and indeed have caught bugs 
in Python that escaped even efence.  Users want that for 
their code too.

I don't know where this leaves us.  Probably right where we 
started, except we know better that we're lost <wink>.

Logged In: YES 
user_id=21627

Why does the linker report no error if expat.dll is
integrated into a debug build? Because the conflict is
detected only for object files compiled with different
settings; the linker doesn't look into DLLs to see what CRT
they use.

I believe MS documented all known problems with combining
CRTs; this is an issue that occurs in many projects, and all
aspects are well-understood. Since pyexpat doesn't export
any CRT objects created by expat, there is no problem with
using expat.dll in a debug build.

Anonymous, given all this information, do you still think
there is a bug in Python? If so, can you please explain from
scratch what this problem is (stating only what you believe
to be facts)? If not, I propose to close this report as
"Won't fix".

Logged In: NO 

(Not anonymous: Jeff Kotula, Vital Images, Inc.)

"Bug" is probably not the right classification. I believe
it is a weakness in the configuration management though.

Here's the problem: As a client of the C API, I want to
be able to create a static library that is compiled
with DEBUG and/or _DEBUG and be able to select either
the debug or non-debug version of the python library
independently. This helps me stay as close as possible to
production-level configuration even when debugging some
parts of the system.

From your preceding discussion it sounds like the only
real barrier to this is the coupling of _DEBUG to Py_DEBUG.
If this is the case, it should be a small matter to
decouple the two. Changing the layout of objects could also
be avoided simply by always including the reference counting
field -- would it be that big of a percentage increase in
object size?

Logged In: YES 
user_id=31435

Martin, where does MS document "all known problems with 
combining CRTs"?  The KB article you referenced before 
(Q140584) says "There are many ways to get into trouble 
with two CRTs. Here are just a few:", and then lists the 
same three specific examples that have been brought up in 
this bug report.  There's not a clue there about what the 
rest of the many ways may be.

Logged In: YES 
user_id=21627

MS doesn't document it as a complete list; I still believe
it is a complete list. Of course, you have no reason to
trust me more than you trust Microsoft :-)

Logged In: YES 
user_id=14198

I believe there is no intention to "fix" this - it is by
design.  Even if we overcome the memory API issues, the fact
we pass FILE * pointers around is still going to be a
problem for some users.

Please reopen if you feel this is wrong.