Python: pickle and copyreg
multiprocessing must pickle things to sling them among processes, and bound methods are not picklable. The workaround (whether you consider it "easy" or not;-) is to add the infrastructure to your program to allow such methods to be pickled, registering it with the copy_regstandard library method.
Following is from the python doc
Following is from the python doc
The
copyreg module offers a way to define functions used while pickling specific objects. The pickle and copy modules use those functions when pickling/copying those objects. The module provides configuration information about object constructors which are not classes. Such constructors may be factory functions or class instances.copyreg.constructor(object)- Declares object to be a valid constructor. If object is not callable (and hence not valid as a constructor), raises
TypeError.
copyreg.pickle(type, function, constructor=None)- Declares that function should be used as a “reduction” function for objects of type type. function should return either a string or a tuple containing two or three elements.The optional constructor parameter, if provided, is a callable object which can be used to reconstruct the object when called with the tuple of arguments returned by function at pickling time.
TypeErrorwill be raised if object is a class or constructor is not callable.See thepicklemodule for more details on the interface expected of function and constructor. Note that thedispatch_tableattribute of a pickler object or subclass ofpickle.Picklercan also be used for declaring reduction functions.
12.2.1. Example
The example below would like to show how to register a pickle function and how it will be used:
| import copy_reg | ||||
| import types | ||||
| __all__ = ["register_method_pickler"] | ||||
| # Python can't pickle instancemethods (e.g. bound endpoints on the client) so we have to provide | ||||
| # methods to pickle and unpickle methods and register them to be used for pickling methods. | ||||
| # This is mostly lifted from an answer to: https://stackoverflow.com/a/1816969 | ||||
| def register_method_pickler(): | ||||
| copy_reg.pickle(types.MethodType, _method_pickler, _method_unpickler) | ||||
| def _method_pickler(method): | ||||
| func_name = method.im_func.__name__ | ||||
| obj = method.im_self | ||||
| cls = method.im_class | ||||
| return _method_unpickler, (func_name, obj, cls) | ||||
| def _method_unpickler(fn_name, obj, cls): | ||||
| for cls in cls.mro(): | ||||
| try: | ||||
| fn = cls.__dict__[fn_name] | ||||
| except KeyError: | ||||
| pass | ||||
| else: | ||||
| break | ||||
| return fn.__get__(obj, cls) |
Comments
Post a Comment