Bug #1733973 is related.
What is happening in this bug, is that tokenize.tokenize is using generate_tokens to generate the tokens. Since tokeneater() fails while the iteration is occuring, the exception is raised and it unwinds the tokenize() call. This garbage collects the generate_tokens generator -- which throws a GeneratorExit into the generator.
This time, sysmodule.c's trace_trampoline is the one assuming it can overwrite PyErr_*.
Maybe the correct way of fixing the bug is saving+clearing/restoring PyErr_* in ceval.c:call_trace, rather than in each trace/profile handler.
The upside is that they all seem to assume that they can overwrite PyErr_* and this will fix them all.
The downside is that the PyErr_* that's been set in the frame will not be accessible to the trace function. This can be fixed by storing it in some global.
Basically, the fix is to call PyErr_Fetch before the trace/profile func, and PyErr_Restore after it. The problem is that the trace/profile func is a function ptr in the PyThreadState that's used by more than Python's ceval.c. It seems to be liberally used by pyexpat and others. This means that this function pointer should always point at the wrapper function rather than the given function, which means storing more information in the thread state, and then extracting it from the thread state again.
This got more complicated and intricate than I intended, and the whole thing has a wrong feel to it - as it seems to me that all calls to the trace/profile func should be in one centralized place, and that place should be wrapped.
I hope someone else will implement the fix or at least comment on this. If anyone is not clear on what the fix is, please post questions (and if this bug system has mailing notifications) and I'll further explain.
|