An in-depth explanation of the MultiProcessing library in python

MultiProcessing module is an excellent multi-threaded MultiThreading module that handles concurrency.
I have come into contact with this library a little bit before, but did not study it in depth. This time I studied it when I was idle and bored to solve my doubts.
Today we will study the apply_async and map methods. Rumor has it that these two methods allocate processes in the process pool to related functions. I want to verify it.
Look at the official website’s explanation of these two:
apply_async(func[, args[, kwds[, callback[, error_callback]]]])
A variant of the apply() method which returns a result object.

If callback is specified then it should be a callable which accepts a single argument. When the result becomes ready callback is applied to it, that is unless the call failed, in which case the error_callback is applied instead.

If error_callback is specified then it should be a callable which accepts a single argument. If the target function fails, then the error_callback is called with the exception instance.

Callbacks should complete immediately since otherwise the thread which handles the results will get blocked.

map(func, iterable[, chunksize])
A parallel equivalent of the map() built-in function (it supports only one iterable argument though). It blocks until the result is ready.

This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks. The (approximate) size of these chunks can be specified by setting chunksize to a positive integer.

Pool can provide a specified number of processes for users to call. When a new request is submitted to the pool, if the pool is not full, a new one will be created. The process is used to execute the request; but if the number of processes in the pool has reached the specified maximum, then the request will wait until a process in the pool ends, and then a new process will be created to run it

Let’s take a look at the program:

from multiprocessing import Poolimport timeimport osdef func(msg):print('msg: %s %s' % (msg, os.getpid()))
    time.sleep(3)print("end")if __name__ == '__main__':
    pool = Pool(4)for i in range(4):
        msg = 'hello %d' % (i)
        pool.apply_async(func, (msg, ))# pool.map(func, range(4))print("Mark~ Mark~ Mark~~~~~~~~~~~~~~~~~~~~~~")
    pool.close()
    pool.join()   # 调用join之前，先调用close函数，否则会出错。执行完close后不会有新的进程加入到pool,join函数等待所有子进程结束print("Sub-process(es) done.")

Run results:

##Remove the map comment and add a comment to the apply_async function

Look at the program and running results when there are not enough processes in the process pool:

from multiprocessing import Poolimport timeimport osdef func(msg):print('msg: %s %s' % (msg, os.getpid()))
    time.sleep(3)print("end")if __name__ == '__main__':
    pool = Pool(3)'''for i in range(4):
        msg = 'hello %d' % (i)
        pool.apply_async(func, (msg, ))'''pool.map(func, range(4))print("Mark~ Mark~ Mark~~~~~~~~~~~~~~~~~~~~~~")
    pool.close()
    pool.join()   # 调用join之前，先调用close函数，否则会出错。执行完close后不会有新的进程加入到pool,join函数等待所有子进程结束print("Sub-process(es) done.")

Program results:

You can see that if the number of processes in the process pool is greater than or equal to the number of times the function needs to be run, it will go smoothly, and the results will be natural; but if the number of processes in the process pool is less than the number of times the function is to be run, then one process will be blocked, that is, two or more functions share one process.

Moreover, the second parameter of the apply_async function is passed in a parameter value. Once it is run This function will allocate a process to the function. Note that it is asynchronous, so if you need to allocate multiple processes, you need a for loop or while loop; for the map function, its second parameter value receives an iteration device, so there is no need to use a for loop. Remember, what these two functions implement is to assign processes in the process pool to functions in turn.

By the way, the official website of MultiProcessing, which is all in English, is very confusing and painful to read, but also very interesting. I have to say that it is still very helpful for English...

The above is the detailed content of An in-depth explanation of the MultiProcessing library in python. For more information, please follow other related articles on the PHP Chinese website!