The product I'm working on uses a PEP 302 importer hook.  It's a home-grown ltihooks clone.  (We don't use ltihooks because it's GPLed, and we don't want to be.)

Our importer worked on 2.4.3, is broken on 2.5.0.

What's broken is that if the hook's find_module() method returns None for a given module name, say 'spam', then that is supposed to cause the import machinery to fall back on the regular unhooked import behavior -- that is, find 'spam.py' (or 'spam.pyc', 'spam.pyo') in the directory in question.  This used to be what happened, but no longer.

Tracing through the code, the problem seems to be occurring due to the 'continue' at line 1289 (in the 2.5 tarball) of Python/import.c.

Slogging through SVN (aside: this would have been easier if your ViewSVN supported annotate/blame -- any chance you could add that?), it appears that the offending continue statement was added in revision 46372, whose checkin comment claims that it was done for performance reasons.  I'm all for performance improvements, but not at the expense of breakage.

Attached is a tarball with some files that reproduce the problem.  (The LibtoolImporter.py file is a stripped-down toy version of what we are really using.)  Unwind the tarball, cd to the directory, and run script.py.  Here's what I get:



shell prompt> pwd
/home/phil/pep302_bug
shell prompt> ls -CF
eggs.la  LibtoolImporter.py  script.py*  spam.py
shell prompt> python2.4.3 script.py
.la file(s) found in /home/phil/pep302_bug, LibtoolImporter will be used.
LibtoolImporter.find_module() couldn't find spam.la or spammodule.la
in /home/phil/pep302_bug.  Returning None.  This is *supposed* to cause a
fallback to the default import code looking for spam.py
in /home/phil/pep302_bug
module spam loaded
shell prompt> python2.5 script.py
.la file(s) found in /home/phil/pep302_bug, LibtoolImporter will be used.
LibtoolImporter.find_module() couldn't find spam.la or spammodule.la
in /home/phil/pep302_bug.  Returning None.  This is *supposed* to cause a
fallback to the default import code looking for spam.py
in /home/phil/pep302_bug
Traceback (most recent call last):
  File "script.py", line 4, in <module>
    import spam
ImportError: No module named spam
shell prompt>

Georg, I believe this was your checkin.

I don't agree with this interpretation of PEP 302 for this instance.  If you read the PEP there is no explicit mention that if an importer for an entry on sys.path fails that it falls back on the default import behaviour.  The only mention of using the default behaviour is if a value of None is stored in sys.path_importer_cache (which also occurs when no entry on sys.path_hooks returns a usable importer).

In my interpretation of PEP 302 (and how I implemented it for my pure Python import implementation), if an importer exists for an entry on sys.path then it essentially "owns" that entry.  The default import semantics only kick in for unclaimed sys.path entries in my view.

Now I could be wrong and if I am I hope Phil can point out where in PEP 302 I am wrong.  Otherwise we have either undocumented behaviour that changed (and that is always messy) or implied semantics that got fixed.

In other words, this probably needs to go to python-dev if Phil can't point out exactly where PEP 302 says the semantics he is expecting is supposed to be that way.

You're correct, Brett.

I found the implicit fallback at the Iceland sprint and we decided to change
that in 2.5:

http://mail.python.org/pipermail/python-dev/2006-May/065174.html

PEP 302 has even been edited to say:

    [...] Note
    that if the callable returns an importer object for a specific
    sys.path entry, the builtin import machinery will not be invoked
    to handle that entry any longer, even if the importer object later
    fails to find a specific module.

Closing as "Won't fix", if more need for discussion arises, please take this to python-dev.

Ok.  I concede that I misread the PEP.  And I won't dispute the NO CHANGE resolution.  But before I drop it entirely, I wish to make one more comment.

From PEP 302: "The hooks proposed in this PEP...would allow the __import__-based tools to get rid of most of their import.c emulation code."

This implementation change causes hooks that handle directories containing both custom modules and "normal" modules to have to do some of that import.c emulation.  Which is what I guess I will do.

I am planning to work on the import machinery in general for this year so hopefully things will improve.