Why Does Multiprocessing Stall on a Single Core When NumPy Is Imported?

Why Multiprocessing Stalls on a Single Core When NumPy Is Imported

When attempting to parallelize a CPU-intensive loop using joblib, you may encounter an issue where all worker processes are assigned to the same core, leading to no performance gain. This occurs due to certain Python modules, including NumPy, manipulating core affinity on import.

Root Cause:

NumPy links against multithreaded OpenBLAS libraries, which interferes with core affinity settings.

Solution:

To resolve this issue, reset the task affinity using the following command:

<code class="python">os.system("taskset -p 0xff %d" % os.getpid())</code>

Place this line after importing the module where the issue arises. This forces the process to run on all available cores.

Alternative Solutions:

• Environment Variable: Set the OPENBLAS_MAIN_FREE (or GOTOBLAS_MAIN_FREE) environment variable to 1 at run-time to disable the CPU affinity-resetting behavior of OpenBLAS.
• Build-Time Modification: Disable CPU affinity resetting permanently by modifying the Makefile.rule to include the line NO_AFFINITY=1 when compiling OpenBLAS from source.

The above is the detailed content of Why Does Multiprocessing Stall on a Single Core When NumPy Is Imported?. For more information, please follow other related articles on the PHP Chinese website!