This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: pickle / cPickle can't load lambdas
Type: Stage:
Components: Library (Lib) Versions:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gvanrossum Nosy List: davecole, gmcm, gvanrossum
Priority: low Keywords:

Created on 2001-08-16 13:33 by gmcm, last changed 2022-04-10 16:04 by admin. This issue is now closed.

Messages (12)
msg5956 - (view) Author: Gordon B. McMillan (gmcm) Date: 2001-08-16 13:33
pickle and cPickle will happily dump a lambda, but on 
load, both report:
SystemError: Failed to import class <lambda> from 
module __main__

Seen on Py 2.1 & 1.5.2

>>> f = lambda x: x in (1,2,3)
>>> o = cPickle.dumps(f)
>>> f2 = cPickle.loads(o)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
SystemError: Failed to import class <lambda> from 
module __main__
>>> o = pickle.dumps(f)
>>> f2 = pickle.loads(o)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "c:\python21\lib\pickle.py", line 951, in loads
    return Unpickler(file).load()
  File "c:\python21\lib\pickle.py", line 567, in load
    dispatch[key](self)
  File "c:\python21\lib\pickle.py", line 780, in 
load_global
    klass = self.find_class(module, name)
  File "c:\python21\lib\pickle.py", line 790, in 
find_class
    raise SystemError, \
SystemError: Failed to import class <lambda> from 
module __main__

msg5957 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2001-08-16 15:11
Logged In: YES 
user_id=6380

I think I'll have to close this with a won't fix, or "then
don't do that" resolution.

The problem is that whenever a function or class is pickled,
pickle must accept on blind faith that it can also be
unpickled. How would you check that this is indeed the case?
msg5958 - (view) Author: Gordon B. McMillan (gmcm) Date: 2001-08-16 16:03
Logged In: YES 
user_id=4923

Hmm. In the (simplistic) case I was trying, I can't see any 
significant difference between the lambda and the 
equivalent function (names differ, and the func has 2 
appended & apparently unreachable bytecodes, but otherwise 
the func_* and func_code.co_* attributes match). So what am 
I missing?

Lowering priority - I can live without it easily enough.

But if lambda's won't load, they probably shouldn't dump.
msg5959 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2001-08-16 16:31
Logged In: YES 
user_id=6380

I guess what you're missing is that pickling a function
doesn't pickle the bytecode! It pickles the name instead. So
a reference to "foo.bar" is pickled as "foo.bar", and the
unpickler imports bar from foo.
msg5960 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2001-08-17 18:51
Logged In: YES 
user_id=6380

I figured it out.

In save_global(), I now try to import the module and then
extract the name from the module; this must give us the same
object. If it doesn't, raise PicklingError.

Checked in as pickle.py rev 1.50.  I can't close this yet
because cPickle needs
a similar patch.
msg5961 - (view) Author: Gordon B. McMillan (gmcm) Date: 2001-08-17 21:45
Logged In: YES 
user_id=4923

Patch # 452239 is my attempt to patch cPickle.c to match 
Guido's patch to pickle.py.
msg5962 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2001-08-18 21:23
Logged In: YES 
user_id=6380

I applied Gordon's patch, so this can be closed now.
msg5963 - (view) Author: Dave Cole (davecole) (Python triager) Date: 2002-10-25 13:44
Logged In: YES 
user_id=28658

I seem to be suffering from some unintentional consequences
of this fix.  I do not think that the pickle should fail in
the follwing case.  Wouldn't it be a better idea to just
check for a successful of the pickled class instead of
requiring the imported class to be stored at the same memory
location?

>>> import pickle, copy
>>> o = copy._EmptyClass()
>>> reload(copy)
<module 'copy' from '/usr/lib/python2.2/copy.pyc'>
>>> pickle.dumps(o, 1)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib/python2.2/pickle.py", line 978, in dumps
    Pickler(file, bin).dump(object)
  File "/usr/lib/python2.2/pickle.py", line 115, in dump
    self.save(object)
  File "/usr/lib/python2.2/pickle.py", line 225, in save
    f(self, object)
  File "/usr/lib/python2.2/pickle.py", line 477, in save_inst
    save(cls)
  File "/usr/lib/python2.2/pickle.py", line 225, in save
    f(self, object)
  File "/usr/lib/python2.2/pickle.py", line 524, in save_global
    raise PicklingError(
pickle.PicklingError: Can't pickle <class copy._EmptyClass
at 0x819493c>: it's not the same object as copy._EmptyClass
msg5964 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-10-28 13:26
Logged In: YES 
user_id=6380

Reopening so I won't forget about this.
msg5965 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-10-29 22:21
Logged In: YES 
user_id=6380

Well, the problem is that there are lots of other cases that
the fix also catches. For example:

>>> class C:
          class C:
              pass
    
>>> a = C.C()
>>> pickle.dumps(a,1)

I tend to think that catching this is more important than
handling reload() -- since reload() has lots of other
problems like this, it's better to blame reload() and not
try to fix it at the cost of other situations.
msg5966 - (view) Author: Dave Cole (davecole) (Python triager) Date: 2002-10-29 23:52
Logged In: YES 
user_id=28658

I realise that there are lots of things that you can do to
prevent or confuse introspection working in the pickle
module.  You can dynamically create classes on the fly too.
 It just seems a shame that it is possible to prevent the
most basic of class definitions from working in the pickler.

I can program around the problem but it took the better part
of a day to work out why I was experiencing the problem.

Maybe the correct bug fix is not to change code but to
improve the documentation of the pickle module.
msg5967 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-10-30 23:47
Logged In: YES 
user_id=6380

The problem really lies with reload(); pickle is just one of
the many victims. E.g. this fails too:

import foo
x = foo.C()
assert isinstance(x, foo.C) # OK
reload(foo)
assert isinstance(x, foo.C) # Fails

If you have a suggestion for how to warn about reload(),
please submit a *new* bug report with Category set to
Documentation.
History
Date User Action Args
2022-04-10 16:04:19adminsetgithub: 34974
2001-08-16 13:33:45gmcmcreate