Python: pickle and copyreg
multiprocessing must pickle things to sling them among processes, and bound methods are not picklable. The workaround (whether you consider it "easy" or not;-) is to add the infrastructure to your program to allow such methods to be pickled, registering it with the copy_regstandard library method.
Following is from the python doc
Following is from the python doc
The
copyreg
module offers a way to define functions used while pickling specific objects. The pickle
and copy
modules use those functions when pickling/copying those objects. The module provides configuration information about object constructors which are not classes. Such constructors may be factory functions or class instances.copyreg.
constructor
(object)- Declares object to be a valid constructor. If object is not callable (and hence not valid as a constructor), raises
TypeError
.
copyreg.
pickle
(type, function, constructor=None)- Declares that function should be used as a βreductionβ function for objects of type type. function should return either a string or a tuple containing two or three elements.The optional constructor parameter, if provided, is a callable object which can be used to reconstruct the object when called with the tuple of arguments returned by function at pickling time.
TypeError
will be raised if object is a class or constructor is not callable.See thepickle
module for more details on the interface expected of function and constructor. Note that thedispatch_table
attribute of a pickler object or subclass ofpickle.Pickler
can also be used for declaring reduction functions.
12.2.1. Example
The example below would like to show how to register a pickle function and how it will be used:
import copy_reg | ||||
import types | ||||
__all__ = ["register_method_pickler"] | ||||
# Python can't pickle instancemethods (e.g. bound endpoints on the client) so we have to provide | ||||
# methods to pickle and unpickle methods and register them to be used for pickling methods. | ||||
# This is mostly lifted from an answer to: https://stackoverflow.com/a/1816969 | ||||
def register_method_pickler(): | ||||
copy_reg.pickle(types.MethodType, _method_pickler, _method_unpickler) | ||||
def _method_pickler(method): | ||||
func_name = method.im_func.__name__ | ||||
obj = method.im_self | ||||
cls = method.im_class | ||||
return _method_unpickler, (func_name, obj, cls) | ||||
def _method_unpickler(fn_name, obj, cls): | ||||
for cls in cls.mro(): | ||||
try: | ||||
fn = cls.__dict__[fn_name] | ||||
except KeyError: | ||||
pass | ||||
else: | ||||
break | ||||
return fn.__get__(obj, cls) |
Comments
Post a Comment