This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: pickle files should be opened in binary mode
Type: Stage:
Components: Documentation Versions: Python 2.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: georg.brandl Nosy List: fdrake, georg.brandl, irmen, loewis, sjmachin, tim.peters
Priority: normal Keywords:

Created on 2005-01-14 21:58 by sjmachin, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (8)
msg23965 - (view) Author: John Machin (sjmachin) Date: 2005-01-14 21:58
pickle (and cPickle):

At _each_ mention of the pickle file, the docs should say 
that it should be opened with 'wb' or 'rb' mode as 
appropriate, so that a pickle written on one OS can be 
read reliably on another.

The example code at the end of the section should be 
updated to use the 'b' flag.
msg23966 - (view) Author: Irmen de Jong (irmen) (Python triager) Date: 2005-01-16 15:07
Logged In: YES 
user_id=129426

Can't the pickle code just freopen() the file itself, using
binary mode?

Or is this against Python's rule "explicit is better than
implicit"
msg23967 - (view) Author: Fred Drake (fdrake) (Python committer) Date: 2005-01-19 05:06
Logged In: YES 
user_id=3066

Is this true in all cases?  Shouldn't files containing text
pickles (protocol 0) be opened in text mode?  (A problem,
given that all protocols should be readable without prior
knowledge of the protocol used to write the pickle.)
msg23968 - (view) Author: Fred Drake (fdrake) (Python committer) Date: 2005-01-19 05:09
Logged In: YES 
user_id=3066

In response to irmin's comment:

freopen() is only an option for real file objects; pickles
are often stored or read from other sources.  These other
sources are usually binary to begin with, fortunately,
though this issue probably deserves some real coverage in
the documentation either way.
msg23969 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2005-01-19 13:45
Logged In: YES 
user_id=31435

Yes, binary mode should always be used, regardless of 
protocol.  Else pickles aren't portable across boxes (in 
particular, Unix can't read a protocol 0 pickle produced on 
Windows if the latter was written to a text-mode file).  "text 
mode" was a horrible name for protocol 0.
msg23970 - (view) Author: John Machin (sjmachin) Date: 2005-01-19 13:51
Logged In: YES 
user_id=480138

Re Fred's question:
Refer to thread starting at 
http://mail.python.org/pipermail/python-dev/2003-
February/033362.html

Looks like the story is like this:

For pickle mode 1 or higher, always use binary mode for 
reading/writing.

For pickle mode 0, either (a) read/write in text mode and if 
moving to another OS, do so in text mode i.e. convert the line 
endings where necessary or (b) as for pickle mode 1+, stick 
with binary throughout.

Also should add a generalisation of Tim's comment re 
NotePad, e.g. something like """A file written with pickle mode 
0 and file mode 'wb' will contain lone linefeeds as line 
terminators. This will cause it to "look funny" when viewed on 
Windows or MacOS as a text file by editors like Notepad that 
do not understand this format."""
msg23971 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2005-02-24 20:08
Logged In: YES 
user_id=21627

sjmachin, it seems pretty clear what to do now. Would you
volunteer to come up with a patch yourself?
msg23972 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2006-02-20 13:12
Logged In: YES 
user_id=849994

Added a note to the docs in rev. 42520, 42521.
History
Date User Action Args
2022-04-11 14:56:09adminsetgithub: 41441
2005-01-14 21:58:27sjmachincreate