The following code generates an validation error.  It's an 
invalid ast tree where the global_stmt doesn't have the 
keyword 'global' as a child.

I tracked down the fix to validate_global_stmt() this time.

Python 2.3a1 (#38, Dec 31 2002, 17:53:59) [MSC 
v.1200 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more 
information.
>>> import symbol,token,parser
>>> ast = [symbol.file_input,
...  [symbol.stmt, [symbol.simple_stmt, 
[symbol.small_stmt, [symbol.global_stmt,
 [1, 'foo']]], [token.NEWLINE, '']]],
...  [token.NEWLINE, ''],
...  [token.ENDMARKER, '']]
>>> a = parser.sequence2ast(ast)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
parser.ParserError: VALIDATION FAILURE: report this 
to the maintainer!
>>>

Logged In: YES 
user_id=699438

And I found a few more by doing a roundtrip 
ast2Tuple/tuple2ast on all files in pythonpath.  These actually 
affect valid AST's, so they're probably a bigger deal.

Valid assert statements were failing.  The patch was trivial, 
so its attached.

Also, files that took advantage of encoding_decl (from PEP 
263) were failing.  I couldn't tell if PEP 263 was offically 
closed, and didn't know what the offical grammar was, so I 
didn't do anything there.

I'll also attach the test script that turned up these errors.

Logged In: YES 
user_id=6656

Thanks for doing this.  I'll probably check these in after
I've finished crunching my way through monday morning email...

Logged In: YES 
user_id=33168

Michael were these patches ok to check in?

Logged In: YES 
user_id=6656

Patches as such are probably fine.  The reason I haven't
checked them in is a wierdness that I may or may not be
related, but is bothering me.  Consider:

->> import parser, sys,glob,os, traceback
/>> for path in sys.path:
|..     for fil in glob.glob(os.path.join(path,"*.py")):
|..       try:
|..         try:
|..             a = parser.suite(file(fil).read())
|..             b = parser.ast2tuple(a)
|..             c = parser.sequence2ast(b)
|..         except parser.ParserError:
|..             print "Parse of %s" % fil, "FAILED"
|..             traceback.print_exc()
|..             print '----------------------------'
|..       except:
|..          print "waaa?"
|.. else:
|..     print "what?"
\__ 
Parse of /home/mwh/src/sf/python/dist/src/Lib/tarfile.py FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Parse of /home/mwh/src/sf/python/dist/src/Lib/heapq.py FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Parse of /home/mwh/src/sf/python/dist/src/Lib/getopt.py FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Parse of /home/mwh/src/sf/python/dist/src/Lib/inspect.py FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Parse of /home/mwh/src/sf/python/dist/src/Lib/pydoc.py FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Parse of /home/mwh/src/sf/python/dist/src/Lib/lib-tk/Tix.py
FAILED
Traceback (most recent call last):
  File "<input>", line 7, in ?
ParserError
----------------------------
Traceback (most recent call last):
  File "<input>", line 2, in ?
ParserError: Illegal terminal: expected "__assert__"
->> 

Notice that an exception seems to have made it through the
except clauses!  The scary bit is that a (seemingly) random
number of parser errors *will* caught, and then one will get
through.

The only explanation I have for this is some kind of
returning non-NULL with exception set kind of bug.

Off to play with a debug build, I guess.

Logged In: YES 
user_id=6656

Pff, I was right, but logistix's patch fix it.

logistix, did you forget to attach the encoding patch?  I
still get errors in module's with a cookie.

Logged In: YES 
user_id=699438

Michael,

At the time I didn't have an encoding patch and punted.  I did 
figure it out last night though.  I'll post it later today since I'm 
at the office now.

And yes, this is the error that'll print out "XXX unhandled 
exception" out of ceval.c when you run the test from debug 
mode.

It comes down to the bastardized way that encoding is 
implemented.  It doesn't fit the current definition of a token or 
a symbol.  There was some code added to add the encoding 
type in the ast2sequence functions, but not the 
sequence2ast functions, so when it hit scripts with 
encoding_decl's, it bombed big time.

Also, as a matter of style, should I add the stuff in my test 
script to the test suite or is that overkill? Should I just have it 
check4 or 5 random files?

Logged In: YES 
user_id=6656

> At the time I didn't have an encoding patch and punted.

Oh, I misread you then.

> I did figure it out last night though.  I'll post it later
today 
> since I'm at the office now.

Ta.

> And yes, this is the error that'll print out "XXX 
> unhandled exception" out of ceval.c when you run
> the test from debug mode.

Are you sure?  It seems that was the assert one.

> There was some code added to add the encoding 
> type in the ast2sequence functions,

By me :-)  I didn't really think about the sequence2ast 
functions as all I was doing was trying to get the compiler
package to work...

> Also, as a matter of style, should I add the stuff in 
> my test script to the test suite or is that overkill? Should
> I just have it check 4 or 5 random files?

It might be best to get it to check a few files we knew there
were problems with -- test_pep263, maybe.  Checking the
whole lot is probably overkill.

Logged In: YES 
user_id=699438

You're right, the assert statement was the unhandled 
exception.  It was raising an error but returning true on 
the "__assert__" check.

Just posted the whole patch

Logged In: YES 
user_id=6656

OK, I've checked in revision 2.76 of Modules/parsermodule.c.

Do you have a real name?  For Misc/ACKS.

Also, would you be interested in putting an example of each
failing syntax into test_parser?

Thanks!

Logged In: YES 
user_id=699438

I attached some changes for the testsuite.

My name is Grant Olson.

Looks like we can close this ticket out ;-)

Logged In: YES 
user_id=33168

I added Grant to ACKS 1.227 and removed a duplicate test I'm
guessing was introduced by patching parsermodule.c twice. 
Closing.

Logged In: YES 
user_id=33168

Added test_parser.py changes as 1.15
Also added NEWS entry 1.647 and backported the assert and
global changes as:
 test_parser.py 1.10.12.2
 parsermodule.c 2.68.6.2