Learn Python Celery and easily complete asynchronous tasks

WBOY
Release: 2023-04-23 15:31:16
forward
1731 people have browsed it

Although modern web applications are faster and more convenient than ever, there are still many situations where heavy tasks need to be transferred to other parts of the system instead of working on the main thread.

Examples of these situations are as follows:

  • Periodic tasks – jobs that are scheduled to run at specific intervals. For example, daily, monthly report generation.
  • Third Party Tools - Applications should return responses to the user quickly, rather than waiting for other tasks to complete first. For example, send emails, notifications, communicate update progress to internal tools.
  • Long-running jobs – jobs that perform complex or resource-expensive work and for which the user needs to wait for the job to complete. For example. DAG workflows, Map-Reduce based tasks, long-running Spark jobs, etc.

So, how to deal with these situations? At this time, Celery comes in handy.

What is Celery?

Celery is an open source task queue implementation, often combined with Python-based web frameworks such as Flask and Django, to execute tasks asynchronously outside of the typical request-response cycle.

So, Celery is essentially a task queue based on distributed messaging. Execution units or tasks are executed simultaneously on one or more workers using multiprocessing, gevent or eventlets. These tasks can be executed synchronously (i.e., wait until ready) or asynchronously (i.e., in the background).

轻松完成异步任务,一文搞懂Python Celery

How does Celery work?

Celery is a distributed task queue based on the producer-consumer model.

Task Queue is a mechanism for distributing work across threads and machines. It is essentially a message intermediary between producers (web applications) and consumers (Celery workers).

Celery interacts through messages, and the broker acts as an intermediary between clients (producers) and workers (consumers). To start a task, the client pushes a message to the queue, and the broker delivers the message to the worker.

The Celery system can be composed of multiple workers and brokers, which provides the possibility of high availability and horizontal expansion.

In short, the Celery client is the producer, which adds new tasks to the queue through the message broker. Then, Celery workers also obtain new tasks from the queue through the message broker. Once processed, the results are stored in the results backend.

Working Example

The following example will use RedisMQ as the message broker.

Set up Redis

On the linux/macOS system, run the Redis server locally by executing the following command:

$ wget http://download.redis.io/redis-stable.tar.gz
$ tar xvzf redis-stable.tar.gz
$ rm redis-stable.tar.gz
$ cd redis-stable
$ make
Copy after login

After setting up Redis, run the Redis server by executing the following command:

$ redis-server
Copy after login

The server runs on the default port 6379.

Set up the application

First, set up the Python project locally.

Celery can be installed through standard tools such as pip or easy_install. Install Celery and Redis through the following command:

pip install celery redis==4.3.4
Copy after login

Now you need a Celery instance to run the application. Any task implemented by Celery starts with an instance, such as creating and managing tasks.

Create a file tasks.py in the project:

From celery import Celery

broker_url = 'redi://localhost:6379/0'

app = Celery('tasks',broker = broker_url)

@app.task
def add(x, y):
return x+y
Copy after login

A simple task add() is defined here to return the sum of two numbers.

Run Celery Worker

In the terminal, switch to the project location and run Celery worker with the following command:

$ celery -A tasks worker - loglevel=info
Copy after login

For more information about the Celery worker command line, you can use help:

$ celery worker - help
Copy after login

Calling tasks

In Celery, use the delay() method to call tasks.

Open another terminal window for the project and run the following command:

$ python
Copy after login

This will open the Python command line.

>> from tasks import add
>> add.delay(1,2)
Copy after login

This will return an AsyncResult instance, which can be used to check the status of the task, get its return value, wait for the task to complete, and also get exceptions and tracebacks on failure.

After running the add.delay() command, the task will be pushed to the queue and then acquired by the worker. This can be verified on the Celery worker terminal, where you can clearly see the task being received and then completed successfully.

The above is the detailed content of Learn Python Celery and easily complete asynchronous tasks. For more information, please follow other related articles on the PHP Chinese website!

Related labels:
source:51cto.com
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Popular Tutorials
More>
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template