This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Prefix search is filesystem-centric
Type: enhancement Stage: test needed
Components: Interpreter Core Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: carljm, eric.araujo, eric.snow, gregory.p.smith, holdenweb, ncoghlan, patriki, steve.dower, superluser, terry.reedy
Priority: low Keywords:

Created on 2005-02-04 22:16 by holdenweb, last changed 2022-04-11 14:56 by admin.

Messages (12)
msg60644 - (view) Author: Steve Holden (holdenweb) * (Python committer) Date: 2005-02-04 22:16
With the introduction of zipimport I experimented to
determine how much of the standard library could be
provided in zip format.

I discovered that I could entirely remove the standard
library, replacing it with /lib/python24.zip, with the
following caveats (this is under Cygwin, where /usr/lib
appears to be a loopback mount of /lib: paths will
differ on other platforms):

1. The /lib/python2.4/lib-dynload directory had to be
copied to the /lib directory to make zlib available to
zipimport;

2. The interpreter now produced three error messages:
"Could not find platform independent libraries <prefix>"
"Could not find platform dependent libraries <exec_prefix>"
"Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]"

With the move towards esoteric import mechanisms it
seems that the searches for os.py in the filesystem
might no longer be an appropriate way to start
executing the interpreter.

Should some import hook API be available to determine
whether standard libraries are available without
actually importing anything?
msg113379 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-08-09 04:30
Is this request still relevant for 3.2?
msg113406 - (view) Author: Steve Holden (holdenweb) * (Python committer) Date: 2010-08-09 11:37
Personally I think it's just as relevant as it always was, particularly with the introduction of importlib, but Brett will have a more informed opinion. I won't be surprised if this issue is closed as wontfix.
msg154337 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-02-26 12:26
Yeah, I'm pretty sure the bootstrap mechanism needs to be able to get hold of os.py directly so it can be injected into the importlib._bootstrap namespace.

However, it may be worth figuring out and documenting the bare minimum that has to exist on the filesystem in order for importlib to get going.

It's even possible that Brett freezes enough into the interpreter binary that the required set has shrunk to zero.
msg154370 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-26 16:59
Importlib actually requires no files from disk; everything that is required for importlib are built-in modules or are constants in importlib itself (e.g. os.sep). So technically this should be doable since my bootstrap work freezes importlib itself.
msg222630 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-09 19:17
Is this something that we could get into 3.5?
msg283964 - (view) Author: Patrik Iselind (patriki) * Date: 2016-12-24 17:34
How much smaller would the stdlib for 3.5 become if you compress it with zip?
msg283965 - (view) Author: Patrik Iselind (patriki) * Date: 2016-12-24 17:50
What about the performance penalty for zipping stdlib? is it significant?

When would you like to zip stdlib? For embedded systems with limited disk
space?

Patrik Iselind

On Sat, Dec 24, 2016 at 6:34 PM, Patrik Iselind <report@bugs.python.org>
wrote:

>
> Patrik Iselind added the comment:
>
> How much smaller would the stdlib for 3.5 become if you compress it with
> zip?
>
> ----------
> nosy: +patriki
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue1116520>
> _______________________________________
>
msg284001 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-12-25 16:31
Originally zip file importing was faster than standard importing from disk because of the fewer stat calls, but importlib caches such things so I don't know if it's still beneficial. As for space savings, I have no idea; you can try zipping the files yourself to find out the space savings.
msg284005 - (view) Author: Patrik Iselind (patriki) * Date: 2016-12-25 17:48
Is it enough to include everything in the Lib folder, excluding 
__pycache__, site-packages and the test folder in Lib? Would that be 
representative enough?

Patrik Iselind

Den 2016-12-25 kl. 17:31, skrev Brett Cannon:
> Brett Cannon added the comment:
>
> Originally zip file importing was faster than standard importing from disk because of the fewer stat calls, but importlib caches such things so I don't know if it's still beneficial. As for space savings, I have no idea; you can try zipping the files yourself to find out the space savings.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue1116520>
> _______________________________________
msg284006 - (view) Author: Steve Holden (holdenweb) * (Python committer) Date: 2016-12-25 18:22
Don't forget that the built-in modules may need to be available before the
zipimporter is. A long time ago (when sys.metapath was introduced) I
experimented with imports from non-filesystem sources and that hit me until
I realised what was going on.

S

Steve Holden

On Sun, Dec 25, 2016 at 5:48 PM, Patrik Iselind <report@bugs.python.org>
wrote:

>
> Patrik Iselind added the comment:
>
> Is it enough to include everything in the Lib folder, excluding
> __pycache__, site-packages and the test folder in Lib? Would that be
> representative enough?
>
> Patrik Iselind
>
> Den 2016-12-25 kl. 17:31, skrev Brett Cannon:
> > Brett Cannon added the comment:
> >
> > Originally zip file importing was faster than standard importing from
> disk because of the fewer stat calls, but importlib caches such things so I
> don't know if it's still beneficial. As for space savings, I have no idea;
> you can try zipping the files yourself to find out the space savings.
> >
> > ----------
> >
> > _______________________________________
> > Python tracker <report@bugs.python.org>
> > <http://bugs.python.org/issue1116520>
> > _______________________________________
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue1116520>
> _______________________________________
>
msg284011 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-12-26 01:40
Note that Steve Dower made some significant changes to sys.path initialisation on Windows in Python 3.6 that could potentially be generalised to other platforms: https://docs.python.org/3/using/windows.html#finding-modules

(There's nothing inherently Windows specific about them, they're mainly useful for the case of embedding CPython inside a larger application)

One of those was to allow pythonXY.zip to be used as a sentinel indicating the location of PYTHONHOME.
History
Date User Action Args
2022-04-11 14:56:09adminsetgithub: 41525
2019-12-19 00:29:24brett.cannonsetnosy: - brett.cannon
2016-12-26 01:40:14ncoghlansetnosy: + steve.dower
messages: + msg284011
2016-12-25 18:22:00holdenwebsetmessages: + msg284006
2016-12-25 17:48:03patrikisetmessages: + msg284005
2016-12-25 16:31:59brett.cannonsetmessages: + msg284001
2016-12-25 07:07:23BreamoreBoysetnosy: - BreamoreBoy
2016-12-24 17:50:38patrikisetmessages: + msg283965
2016-12-24 17:34:07patrikisetnosy: + patriki
messages: + msg283964
2015-08-05 15:58:01eric.snowsetnosy: + gregory.p.smith, superluser

versions: + Python 3.6, - Python 3.5
2014-07-09 19:17:59BreamoreBoysetnosy: + BreamoreBoy

messages: + msg222630
versions: + Python 3.5, - Python 3.3
2012-02-27 18:55:06eric.snowsetnosy: + eric.snow
2012-02-26 16:59:17brett.cannonsetmessages: + msg154370
2012-02-26 12:26:35ncoghlansetnosy: + ncoghlan
messages: + msg154337
2012-02-25 07:18:42eric.araujosetnosy: + carljm, eric.araujo

versions: - Python 3.2
2012-02-24 11:05:52ezio.melottisetversions: + Python 3.3
2010-08-09 11:37:35holdenwebsetmessages: + msg113406
2010-08-09 04:30:55terry.reedysetnosy: + terry.reedy

messages: + msg113379
versions: + Python 3.2, - Python 3.1, Python 2.7
2009-02-15 22:33:44ajaksu2setnosy: + brett.cannon
stage: test needed
type: enhancement
versions: + Python 3.1, Python 2.7
2005-02-04 22:16:00holdenwebcreate