How Can I Use NumPy Arrays in Shared Memory for Multiprocessing in Python?

Using NumPy Arrays in Shared Memory for Multiprocessing

When working with multiprocessing in Python, sharing arrays between processes can be challenging. Using a NumPy array in multiprocessing may present difficulties if you require it to behave like a NumPy array rather than a simple ctypes array.

Problem Statement

In the provided example, the NumPy array is wrapped in a ctypes Array() object. While this allows access to the array in a ctypes manner, it loses the functionality of a NumPy array. Operations like -1*arr or arr.sum() cannot be performed, and converting it back to a NumPy array would break the shared memory.

Solution using shared_arr.get_lock()

To retain NumPy array functionality while using a shared memory array, you can use the get_lock() method available from a shared array:

shared_arr = mp.Array(ctypes.c_double, N)

def f(i):
    with shared_arr.get_lock():
        arr = np.frombuffer(shared_arr.get_obj())
        arr[i] = -arr[i]

By using the get_lock() method, you can synchronize access to the shared array and ensure that it is accessed like a NumPy array within the process.

Example

The following code offers an improved example that retains NumPy array functionality while utilizing shared memory:

import multiprocessing as mp
import numpy as np

N = 100
shared_arr = mp.Array(ctypes.c_double, N)
arr = np.frombuffer(shared_arr.get_obj())

# Fill the shared array with random values
arr[:] = np.random.uniform(size=N)

# Create a pool of processes
with mp.Pool(initializer=init, initargs=(shared_arr,)) as pool:

    # Define functions that modify the shared array
    def f(i):
        with shared_arr.get_lock():
            arr -= 1 # Subtract 1 from each array element within the process

    pool.map(f, range(N))

# Check the modified shared array
assert np.allclose(arr, -1)

In this example, the init() function sets up the shared_arr for each process, and the f() function operates on the shared array within the lock. The modified shared array is then accessible by the main process after joining the pool. This method provides a synchronized and NumPy-compatible way to use a shared array in multiprocessing.

The above is the detailed content of How Can I Use NumPy Arrays in Shared Memory for Multiprocessing in Python?. For more information, please follow other related articles on the PHP Chinese website!