Python: pickle and copyreg

multiprocessing must pickle things to sling them among processes, and bound methods are not picklable. The workaround (whether you consider it "easy" or not;-) is to add the infrastructure to your program to allow such methods to be pickled, registering it with the copy_regstandard library method.

Following is from the python doc

The copyreg module offers a way to define functions used while pickling specific objects. The pickle and copy modules use those functions when pickling/copying those objects. The module provides configuration information about object constructors which are not classes. Such constructors may be factory functions or class instances.
copyreg.constructor(object)
Declares object to be a valid constructor. If object is not callable (and hence not valid as a constructor), raises TypeError.
copyreg.pickle(typefunctionconstructor=None)
Declares that function should be used as a β€œreduction” function for objects of type typefunction should return either a string or a tuple containing two or three elements.
The optional constructor parameter, if provided, is a callable object which can be used to reconstruct the object when called with the tuple of arguments returned by function at pickling time. TypeError will be raised if object is a class or constructor is not callable.
See the pickle module for more details on the interface expected of function and constructor. Note that thedispatch_table attribute of a pickler object or subclass of pickle.Pickler can also be used for declaring reduction functions.

12.2.1. Example

The example below would like to show how to register a pickle function and how it will be used:
>>>
>>> import copyreg, copy, pickle
>>> class C(object):
...     def __init__(self, a):
...         self.a = a
...
>>> def pickle_c(c):
...     print("pickling a C instance...")
...     return C, (c.a,)
...
>>> copyreg.pickle(C, pickle_c)
>>> c = C(1)
>>> d = copy.copy(c)
pickling a C instance...
>>> p = pickle.dumps(c)
pickling a C instance...



import copy_reg
import types
__all__ = ["register_method_pickler"]
# Python can't pickle instancemethods (e.g. bound endpoints on the client) so we have to provide
# methods to pickle and unpickle methods and register them to be used for pickling methods.
# This is mostly lifted from an answer to: https://stackoverflow.com/a/1816969
def register_method_pickler():
copy_reg.pickle(types.MethodType, _method_pickler, _method_unpickler)
def _method_pickler(method):
func_name = method.im_func.__name__
obj = method.im_self
cls = method.im_class
return _method_unpickler, (func_name, obj, cls)
def _method_unpickler(fn_name, obj, cls):
for cls in cls.mro():
try:
fn = cls.__dict__[fn_name]
except KeyError:
pass
else:
break
return fn.__get__(obj, cls)

Comments

Popular posts from this blog

iOS Accessibility Tutorial: Getting Started

Swift: Codable