The attached code snippet 'testcase.py' should produce an 
IndentationError, but does not.  The code in tokenize.py is too 
trusting, and needs to add a check against bad indentation as it 
yields DEDENT tokens.

I'm including a diff to tokenize.py that should at least raise an 
exception on bad indentation like this.

Just in case, I'm including testcase.py here too:
------
import tokenize
from StringIO import StringIO
sampleBadText = """
def foo():
    bar
  baz
"""
print list(tokenize.generate_tokens(
    StringIO(sampleBadText).readline))

Logged In: YES 
user_id=80475

Fixed.  
See Lib/tokenize.py 1.38 and 1.36.4.1

Logged In: YES 
user_id=4771

Reopening this bug report: this might fix the problem at
hand, but it breaks inspect.getsource() on cases where it
used to work.  See attached example.

Logged In: YES 
user_id=4771

Here is a proposed patch.  It relaxes the dedent policy a
bit.  It assumes that the first line may already have some
initial indentation, as is the case when tokenizing from the
middle of a file (as inspect.getsource() does).

It should also be back-ported to 2.4, given that the
previous patch was.  For 2.4, only the non-test part of the
patch applies cleanly; I suggest to ignore the test part and
just apply it, given that there are much more tests in 2.5
for inspect.getsource() anyway.

The whole issue of inspect.getsource() being muddy anyway, I
will go ahead and check this patch in unless someone spots a
problem.  For now the previously-applied patch makes parts
of PyPy break with an uncaught IndentationError.

Logged In: YES 
user_id=149084

Tokenize Rev 39046 21Jun05 breaks tabnanny.

tabnanny doesn't handle the IndentationError exception
when tokenize detects a dedent.

I patched up ScriptBinding.py in IDLE.  The 
IndentationError probably should pass the same parms as
TokenError and tabnanny should catch it.

Logged In: YES 
user_id=849994

tabnanny's been taken care of in r51284.