This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Hint for speeding up cPickle
Type: Stage:
Components: Library (Lib) Versions: Python 2.2
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: jhylton, jiba
Priority: normal Keywords:

Created on 2002-08-07 15:31 by jiba, last changed 2022-04-10 16:05 by admin. This issue is now closed.

Messages (3)
msg11854 - (view) Author: Jiba (jiba) Date: 2002-08-07 15:31
There is a strange behaviour in cPickle (in Python 2.2.1).
For big serialized file, the following:

        cPickle.loads(open("my_file").read())

can be much (e.g. 7 times) faster than:

        cPickle.load(open("my_file"))

It seems that cPickle use unbuffered reading or what ?
The pickle implementation doesn't have this behaviour,
and can even be
faster than the C one!
msg11855 - (view) Author: Jeremy Hylton (jhylton) (Python triager) Date: 2002-08-08 17:15
Logged In: YES 
user_id=31392

I expect this has to do with extra overhead in Python to
read from the file, independent of buffering.  The cPickle
code is trying to read small numbers of bytes at a time. 
Each read used fread().  This is slow, because fread is slow
for small numbers of bytes.  It's also slow because cPickle
releases and acquires the global interpeter lock around the
fread() calls.
msg11856 - (view) Author: Jeremy Hylton (jhylton) (Python triager) Date: 2003-06-19 17:24
Logged In: YES 
user_id=31392

Really no time to deal with this.
History
Date User Action Args
2022-04-10 16:05:34adminsetgithub: 36996
2002-08-07 15:31:47jibacreate