Issue563303
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2002-06-01 14:10 by cliffo, last changed 2022-04-10 16:05 by admin. This issue is now closed.
Messages (5) | |||
---|---|---|---|
msg11005 - (view) | Author: Cliff Owen (cliffo) | Date: 2002-06-01 14:10 | |
PyObject_DebugMalloc() returns a pointer that is +8 bytes to detect for underwriting the pointer. Sometimes, PyObject_Free() is called instead of going through PyObject_DebugFree(). This means the pointer is off by 8. PyObject_Free() takes the pointer passed in as being valid, and will update the pool to indicate the passed pointer is a pointer to the new free block. Corruption of the heap occurs when PyObject_DebugMalloc() is called and there are multiple free blocks in a row that are properly aligned. Writing the FORBIDDENBYTE set to the end of the newly allocated block may stomp the next free pointer. 2 more allocations later on this arena, and you will AV. Why is PyObject_Free() being called instead of PyObject_DebugFree() (from ste_dealloc)? I'm assuming it was built incorrectly or there's an ordering issue in objimpl.h where PyObject_Free() is defined before the macro, and referenced by PyObject_Del. The fact remains that a simple alignment check in PyObject_Free() will handle any misaligned pointer. If I've got this completely screwed up and it is in fact building incorrectly, please let me know. I've traced through it about 100 times. I'm fairly certain I understand how/why it works the way it does, but I could be wrong. This problem would only show up in debug mode. |
|||
msg11006 - (view) | Author: Tim Peters (tim.peters) * | Date: 2002-06-01 16:35 | |
Logged In: YES user_id=31435 Do you have an actual program that shows corruption, or are you speculating? Note that in a debug build, it's not possible to call PyObject_Free: the name PyObject_Free is remapped to _PyObject_DebugFree by objimpl.h then. In a debug build when pymalloc is enabled, all calls to all PyMem_XYZ and PyObject_XYZ memory functions are redirected to _PyObject_DebugXYZ, except within obmalloc.c itself (which #undefs some name substitutions). ste_dealloc doesn't call PyObject_Free. It calls PyObject_Del. In a debug pymalloc build, that ends up calling _PyObject_DebugFree. Step through it in a debugger if you don't believe it <wink>. |
|||
msg11007 - (view) | Author: Cliff Owen (cliffo) | Date: 2002-06-01 22:22 | |
Logged In: YES user_id=556609 Yes, I do have a program which demonstrates the problem. However, I cannot send it in because it is commercial in nature. :( I understand this poses a bit of a problem for debugging, but I'm hoping you understand. I do have other things that require more of my time than browsing the Python sources and making speculations on what I see. This did AV, and I did spend a while debugging it, with a debugger. Being the nice developer who's using a free product that's not being maintained by me, I thought I'd report this. Yes, I also have seen what happens inside of objimpl.h, and do know that PyObject_Del is called in ste_dealloc. These are details I didn't think necessary to mention because you would obviously be familiar with the code and since I mentioned ste_dealloc, you might assume I would be familiar with these mappings as well since ste_dealloc() clearly calls PyObject_Del, which is nothing but a macro to PyObject_Free. Regardless and bickering aside, I have added the following line to PyObject_Free() in my own code and it resolves the problem by assuring that any pointer passed in is properly aligned on the correct block size. This one line covers a multitude of sins from the caller. p = (char *) p - ((char *)p - ((char *)pool + POOL_OVERHEAD)) % INDEX2SIZE(pool->szidx); Right after: if (ADDRESS_IN_RANGE(p, pool->arenaindex)) If you choose to use this line or not is up to you. If the user passes in a pointer that is aligned, this line does nothing. If the user passes in a pointer which resides anywhere in a valid block, that block is released. Is this the correct behavior? Probably not. It should probably raise an exception rather than allow the user to get away with it. This would at least force the user to figure out why they were passing in a bogus pointer in the first place. Some additional information: I am building with MSVC 6.x (SP3), using a snapshot of the python 2.3 build. I am compiling into a static library using these defines only. WIN32,_DEBUG,_MBCS,_LIB,USE_DL_EXPORT If you need additional information, let me know. I'm not sure what else I can tell you. |
|||
msg11008 - (view) | Author: Tim Peters (tim.peters) * | Date: 2002-06-01 23:56 | |
Logged In: YES user_id=31435 Could you try to describe exactly how the problem arises, holding off on speculation about causes and cures? What I still don't grasp at all is how you get into this situation to begin with: it should be impossible to enter PyObject_Free () in a pymalloc debug build *except* when _PyObject_DebugFree() calls it. Your report begins with """ Sometimes, PyObject_Free() is called instead of going through PyObject_DebugFree(). """ and I'm still lost on that sentence: it shouldn't be possible to do that in a debug build. If there is a way to do it, it's a serious problem, and your suggested code won't fix it (it may hide bad symptoms by accident for a while, but the problem still exists and will come back to bite you later). It's unclear to me why you mentioned ste_dealloc, but I've stepped thru that in the MSVC 6 debugger and it does call _PyObject_DebugFree in a debug build. It's doing what it's supposed to do. 1. Is it your claim that ste_dealloc actually calls PyObject_Free directly in *your* debug build? "Yes" or "no" would be more helpful than outrage <wink>. 2. What does "a snapshot of the python 2.3 build" mean, exactly? Did you compile it yourself? If not, from where did you get it? 3. Are you writing your own C extensions? If so, are you mixing _DEBUG with non-_DEBUG between Python and your own code? If you are so mixing, it's never going to work (Python and extension modules on Windows must both be built with _DEBUG, or neither). |
|||
msg11009 - (view) | Author: Tim Peters (tim.peters) * | Date: 2002-06-03 16:50 | |
Logged In: YES user_id=31435 Offline, it was determined that: + The OP was compiling Python as well as his app. + Some Python modules weren't getting compiled with _DEBUG, via an accident in his build setup. So closing as invalid -- the problem was due to a flawed custom build setup. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-10 16:05:22 | admin | set | github: 36683 |
2002-06-01 14:10:24 | cliffo | create |