How to Share a Queue Between Multiple Asynchronous Worker Processes in Multiprocessing?

Sharing Results Across Multiple Processes

In multiprocessing, sharing resources, such as queues, between different processes can be a challenge. This article explores how to share a result queue among multiple asynchronous worker processes using the apply_async function.

The provided sample code, using multiprocessing.Process, fails due to the limitation of sharing queue objects across processes through inheritance. To overcome this restriction, the multiprocessing.Manager class offers a solution.

Using multiprocessing.Manager

multiprocessing.Manager provides a way to create and manage objects that can be shared across processes. The following code demonstrates how to use Manager to share a queue among worker processes:

<code class="python">import multiprocessing

def worker(name, que):
    que.put("%d is done" % name)

if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=3)
    m = multiprocessing.Manager()
    q = m.Queue()
    workers = pool.apply_async(worker, (33, q))</code>

By using m.Queue() instead of multiprocessing.Queue(), the queue is created and managed by the Manager object. This allows the worker processes to access and modify the queue without facing the "Queue objects should only be shared between processes through inheritance" error.

The above is the detailed content of How to Share a Queue Between Multiple Asynchronous Worker Processes in Multiprocessing?. For more information, please follow other related articles on the PHP Chinese website!