An article to help you understand Python's distributed process interface

1. Preface

Among Thread and Process, Process should be preferred because Process is more stable and Process can be distributed to multiple machines. Thread can only be distributed to multiple CPUs on the same machine at most.

Python's multiprocessing module not only supports multiple processes, but the managers submodule also supports distributing multiple processes to multiple machines. You can write a service process as a scheduler to distribute tasks to multiple other processes and rely on network communication for management.

2. Case Analysis

When doing a crawler program, crawl all the images of a website. If you use multiple processes, generally One process is responsible for grabbing the link address of the image and putting the link address into the queue. Another process is responsible for getting the link address from the queue to download and store it locally.

How to use distributed process?

The process on one machine is responsible for grabbing the link address, and the processes on other machines are responsible for storing it. The main problem encountered is to expose the queue to the network so that other machine processes can access it. The distributed process encapsulates this process. This process can be called the networking of local queues.

Example:

1.py

from multiprocessing.managers import BaseManager
from multiprocessing import freeze_support, Queue
# 任务个数
task_number = 10


# 收发队列
task_quue = Queue(task_number)
result_queue = Queue(task_number)


def get_task():
    return task_quue


def get_result():
    return result_queue
# 创建类似的queueManager
class QueueManager(BaseManager):
    pass


def win_run():
    # 注册在网络上，callable 关联了Queue 对象
    # 将Queue对象在网络中暴露
    # window下绑定调用接口不能直接使用lambda，所以只能先定义函数再绑定
    QueueManager.register('get_task_queue', callable=get_task)
    QueueManager.register('get_result_queue', callable=get_result)
    # 绑定端口和设置验证口令
    manager = QueueManager(address=('127.0.0.1', 8001), authkey='qiye'.encode())
    # 启动管理，监听信息通道
    manager.start()


    try:
        # 通过网络获取任务队列和结果队列
        task = manager.get_task_queue()
        result = manager.get_result_queue()


        # 添加任务
        for url in ["ImageUrl_" + str(i) for i in range(10)]:
            print('url is %s' % url)
            task.put(url)
            
        print('try get result')
        for i in range(10):
            print('result is %s' % result.get(timeout=10))


    except:
        print('Manager error')
    finally:
        manager.shutdown()


if __name__ == '__main__':
    freeze_support()
    win_run()

Connect to the server. Please keep the port and verification password exactly the same as those in the server process to obtain the Queue from the network. Perform localization, obtain tasks from the task queue, and write the results to the result queue

2.py

#coding:utf-8
import time
from multiprocessing.managers import BaseManager
# 创建类似的Manager:
class Manager(BaseManager):
    pass
#使用QueueManager注册获取Queue的方法名称
Manager.register('get_task_queue')
Manager.register('get_result_queue')
#连接到服务器:
server_addr = '127.0.0.1'
print('Connect to server %s...' % server_addr)
# 端口和验证口令注意保持与服务进程设置的完全一致:
m = Manager(address=(server_addr, 8001), authkey='qiye')
# 从网络连接:
m.connect()
#获取Queue的对象:
task = m.get_task_queue()
result = m.get_result_queue()
#从task队列取任务,并把结果写入result队列:
while(not task.empty()):
        image_url = task.get(True,timeout=5)
        print('run task download %s...' % image_url)
        time.sleep(1)
        result.put('%s--->success'%image_url)
#结束:
print('worker exit.')

The task process must be connected to the service process through the network, so To specify the IP of the service process.

The running results are as follows:

Get the image address and pass the address to 2.py.

Receive the address passed by 1.py, download the image, and the console displays the crawling results.

3. Summary

This article is based on the basics of Python. Python’s distributed process interface is simple and encapsulated Good, suitable for environments where heavy tasks need to be distributed to multiple machines. By explaining that the role of Queue is to deliver tasks and receive results.

The above is the detailed content of An article to help you understand Python's distributed process interface. For more information, please follow other related articles on the PHP Chinese website!