Should Readonly Shared Data be Copied in Multiprocessing Environments?

Multiprocessing Shared Readonly Data

Issue Summary

When using multiprocessing, the question arises if shared readonly data is copied to different processes. If so, how can this copying be prevented or encouraged?

Solution: Shared Memory

The provided code snippet, which uses a global array as a default parameter within a function executed by multiple processes, raises concerns about memory duplication. To prevent individual processes from obtaining a separate copy of the data, consider utilizing shared memory.

Shared Memory with Numpy

Multiprocessing and Numpy can be combined to share data efficiently:

<code class="python">import multiprocessing
import ctypes
import numpy as np

shared_array_base = multiprocessing.Array(ctypes.c_double, 10*10)
shared_array = np.ctypeslib.as_array(shared_array_base.get_obj())
shared_array = shared_array.reshape(10, 10)</code>

Parallel Processing Example

Within the function executed by each process, alterations made to the shared array are visible to all:

<code class="python">def my_func(i, def_param=shared_array):
    shared_array[i,:] = i</code>

Conclusion

The appropriate mechanism for this issue depends on the specific requirements and access patterns of the application. Shared memory offers an effective approach to prevent data duplication in this context, ensuring shared access among multiple processes.

The above is the detailed content of Should Readonly Shared Data be Copied in Multiprocessing Environments?. For more information, please follow other related articles on the PHP Chinese website!