Install and use the asynchronous task queue package Celery in Python environment

1. Introduction

celery (celery) is an asynchronous task queue/job queue based on distributed messaging. It focuses on real-time operations, but also has good scheduling support.
celery is used in production systems to handle millions of tasks every day.
celery is written in Python, but the protocol can be implemented in any language. It can also be implemented with other languages ​​via webhooks.
The recommended message broker is RabbitMQ, but provides limited support for Redis, Beanstalk, MongoDB, CouchDB, and databases (using SQLAlchemy or Django ORM).
celery is easy to integrate Django, Pylons and Flask, just use the django-celery, celery-pylons and Flask-Celery add-on packages.

2. Installation
With the above concepts, you need to install these few things: RabbitMQ, SQLAlchemy, Celery
The installation method is also very simple: RabbitMQ:
Mac:

brew install rabbitmq

linux:

sudo apt-get install rabbitmq-server

The remaining two are all Python things. Just install them directly with pip. Students who have never installed the MySQL driver may need to install MySQL-python.
After the installation is completed, start the service:

$ rabbitmq-server[回车]

Do not close the window after starting, the following operation creates a new window (Tab)

3. Simple case
Make sure your previous RabbitMQ has been started.
Still the example on the official website, create a tasks.py file in any directory with the following content:

from celery import Celery

app = Celery('tasks', broker='amqp://guest@localhost//')

@app.task
def add(x, y):
  return x + y

Execute in the same directory :

$ celery -A tasks worker --loglevel=info

This command means to start a worker, add the tasks in tasks (add(x,y)) and put the tasks into the queue middle.
Keep the window open, open a new window to enter interactive mode, python or ipython:

>>> from tasks import add
>>> add.delay(4, 4)

So far, you can already use celery The task is executed. The add task is simply called in the python interactive mode above and the 4, 4 parameters are passed.
But there is a problem at this time. You suddenly want to know the execution results and status of this task, and whether it is finished or not. Therefore, you need to set up a backend.
Modify the code in the previous tasks.py as:

# coding:utf-8
import subprocess
from time import sleep

from celery import Celery

backend = 'db+mysql://root:@192.168.0.102/celery'
broker = 'amqp://guest@192.168.0.102:5672'

app = Celery('tasks', backend=backend, broker=broker)


@app.task
def add(x, y):
  sleep(10)
  return x + y


@app.task
def hostname():
  return subprocess.check_output(['hostname'])

In addition to adding backend, a who method is also added above Used to test multi-server operation. After the modification is completed, it will still be started as before.
Also enter the python interaction model:

>>> from tasks import add, hostname
>>> r = add.delay(4, 4)
>>> r.ready() # 10s内执行，会输出False，因为add中sleep了10s
>>>
>>> r = hostname.delay()
>>> r.result # 输出你的hostname

4. Test multiple servers
Complete the above After the test, a doubt arose. Celery is called distributed task management, so where is its distributed representation? How are its tasks performed? On which machine was it executed?
Without closing the celery service on the current server, install Celery on another server in the same way and start it:

$ celery -A tasks worker --loglevel=info

It is found that the Celery service of the previous server outputs the hostname of the server you just started, provided that the server is connected to your rabbitmq.
Then enter python interactive mode:

>>> from tasks import hostname
>>>
>>> for i in range(10):
...   r = hostname.delay()
...   print r.result # 输出你的hostname
>>>

Look at what you input and observe the output of when you start the celery service on both servers.

5. RabbitMQ remote connection problem
At the beginning of the test, the remote server could not connect to the local RabbitMQ service. Later, it was discovered that permissions need to be set in /usr/local/etc/rabbitmq In the file /rabbitmq-env.conf, modify the ip in NODE_IP_ADDRESS=127.0.0.1 to 0.0.0.0.

6. To summarize
This article briefly introduces the use of Celery, with the focus still on distributed use. The uncomfortable part is that when expanding, you need to redeploy the code (tasks.py) instead of directly sharing the tasks. Maybe Celery uses tasks to match different workers? I don’t know much about it yet, I’ll talk about it after using it in depth.

#For more articles related to installing and using the asynchronous task queue package Celery in the Python environment, please pay attention to the PHP Chinese website!