How to Solve Python Multiprocessing's 'PicklingError: Can't pickle '?

PicklingError in Python Multiprocessing

When using Python multiprocessing, you may encounter a "PicklingError: Can't pickle : attribute lookup __builtin__.function failed" error. This occurs when you attempt to send a function to a worker process, but the function is not defined at the top level of a module.

Nested Function Issue

In your specific case, the error may be caused by the function you are pickling calling a nested function. Even though the outer function f() is defined at the top level, if it calls a function g() that contains a nested function h(), multiprocessing will attempt to pickle the nested function and fail.

Solution

To resolve this issue, move the function you wish to pickle to the top level of the module. Alternatively, you can create a separate function at the top level that calls the nested function and pickle that function instead.

For example:

# Original code
def f():
    g()

def g():
    h()

# Corrected code
def f():
    wrapper_function()

def wrapper_function():
    g()
    h()

By defining wrapper_function() at the top level and pickling it, you can avoid the pickling error.

Additional Considerations

• Pickling lambda functions and function pointers is not supported.
• Functions that access non-picklable objects, such as file handles or database connections, cannot be pickled.
• It is generally a good practice to define functions at the top level of a module if you intend to use them in multiprocessing.

The above is the detailed content of How to Solve Python Multiprocessing's 'PicklingError: Can't pickle '?. For more information, please follow other related articles on the PHP Chinese website!