This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Decorated functions are unpickleable
Type: Stage:
Components: Library (Lib) Versions: Python 2.4
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, sswamida
Priority: normal Keywords:

Created on 2005-02-12 19:48 by sswamida, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (4)
msg24286 - (view) Author: S Joshua Swamidass (sswamida) Date: 2005-02-12 19:48
The decorator feature renders functions impossible to 
pickle if you end up wrapping the function in either a 
trivial lambda or a callable class. This is a 
significant/artificial limitation of the decorator which 
should not exist and prevents decorators from being 
used for many of my purposes. 

==========================
Examples:
==========================

>>> def dec(f):
...     return lambda a: f(a)
...
>>> @dec
... def func(a):
...     return a
...
>>> import pickle
>>> pickle.dumps(func)
Traceback (most recent call last):
...
pickle.PicklingError: Can't pickle <function <lambda> at 
0x40160ae4>: it's not found as __main__.<lambda>



>>> class C:
...         def __init__(self, f):
...                 self.f=f
...         def __call__(self, a):
...                 return f(a)
...
>>> def dec(f):
...         return C(f)
>>> @dec
>>> def func(a):
...        return a
>>> import pickle
>>> pickle.dumps(func)
Traceback (most recent call last):
   .....
pickle.PicklingError: Can't pickle <function func at 
0x40160a74>: it's not the same object as __main__.func


==============================
I've found a syntacically ugly ways of working around 
the class wrapper bug that don't use decorators. 
Perhaps this could be used to create a decorator patch:
==================================

>>> class C:
...         def __init__(self, f):
...                 self.f=f
...         def __call__(self, a):
...                 return f(a)
...
>>> def _func(a):
...        return a
>>> func=C(_func)
>>>
>>> import pickle
>>> pickle.dumps(func)
(No error)
msg24287 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2005-02-16 01:21
Logged In: YES 
user_id=357491

If you read the pickle docs it says you can only pickle "functions defined 
at the top level of a module".  The lambdas are you having your 
decorator return are not defined at the module level and thus do not 
meet that requirement.

The same basic issue goes with your callable class.  The function you are 
storing in self.f is not the same as was is defined at the module level 
since self.f is 'func' prior to wrapping while what 'func' has at the global 
level is an instance of 'C'.

The reason your example works is because '_func` is defined at the 
module level and thus pickle can find it to pickle.  Decorators skip the 
intermediate step of storing the undecorated function at the module 
level.

It would be better to use the pickling protocol to define methods to allow 
your callable class to help with the pickling process.

And just so you know, your class examples should be calling self.f 
instead of f.

Closing as "won't fix".
msg24288 - (view) Author: S Joshua Swamidass (sswamida) Date: 2005-02-16 01:45
Logged In: YES 
user_id=786138

Yes, everything you said is correct, though i think this is not 
ideal behavior. If we decorate a function at the top level of a 
module, shouldn't it be pickleable? If not, then as my original 
point was, decorated functions are not compatible with pickle.

Is there any case you can give me where a decorated 
function is defined in the top level of a module and can be 
pickled? I can't think of one, but i could be wrong.

msg24289 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2005-02-17 05:10
Logged In: YES 
user_id=357491

It would be nice if you could pickle decorated functions, but the way 
pickle works won't allow it unless the decorator just passes the function 
right along:

>>> def dec(func):
...  func.meta = "foo"
...  return func
... 
>>> @dec
... def blah(): pass
... 
>>> pickle.dumps(blah)
'c__main__\nblah\np0\n.'

So it is not a hard rule that decorated functions are not pickleable.  It all 
depends on how the decorator messes with the function it is given.

If you think the documentation for pickle is misleading then please 
submit a documentation patch to clarify that decorators could cause 
issues if they return the function wrapped.
History
Date User Action Args
2022-04-11 14:56:09adminsetgithub: 41570
2005-02-12 19:48:31sswamidacreate