How to apply concurrent programming in Python crawlers
What is concurrent programming
Concurrent programming refers to a program design that can perform multiple operations within a period of time. It is usually represented by multiple tasks in the program that are started at the same time and can run and interact with each other. There will be no impact. The benefit of concurrent programming is that it can improve the performance and responsiveness of the program.
Application of concurrent programming in crawlers
Crawler programs are typical I/O-intensive tasks. For I/O-intensive tasks, multi-threading and asynchronous I/O are A good choice, because when a certain part of the program is blocked due to I/O operations, other parts of the program can still run, so we don't have to waste a lot of time waiting and blocking.
Single-threaded version
Let’s first look at the single-threaded version of the crawler program. This crawler program uses the requests
library to obtain JSON data and saves the image locally through the open
function.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
|
On macOS or Linux systems, we can use the time
command to understand the execution time of the above code and the CPU utilization, as shown below.
time python3 example04.py
The following is the result of the single-threaded crawler code executed on my computer.
python3 example04.py 2.36s user 0.39s system 12% cpu 21.578 total
Here we only need to pay attention to the total time consumption of the code which is 21.578
seconds, CPU utilization is 12%
.
Multi-threaded version
We use the thread pool technology mentioned before to modify the above code into a multi-threaded version.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
|
Execute the command shown below.
time python3 example05.py
The execution result of the code is as follows:
python3 example05.py 2.65s user 0.40s system 95% cpu 3.193 total
Asynchronous I/O version
We use aiohttp
to modify the above code to the asynchronous I/O version. In order to achieve network resource acquisition and file writing operations in asynchronous I/O, we must first install the third-party libraries aiohttp
and aiofile
.
pip install aiohttp aiofile
The following is the asynchronous I/O version of the crawler code.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
|
Execute the command shown below.
time python3 example06.py
The execution result of the code is as follows:
python3 example06.py 0.92s user 0.27s system 290% cpu 0.420 total
Compared with the single-threaded version of the crawler program, the execution time of the multi-threaded version and the asynchronous I/O version of the crawler program has been significantly improved, and the asynchronous I/O version The /O version of the crawler performs best.
The above is the detailed content of How to apply concurrent programming in Python crawlers. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

Regarding the problem of removing the Python interpreter that comes with Linux systems, many Linux distributions will preinstall the Python interpreter when installed, and it does not use the package manager...

About Pythonasyncio...

Pylance type detection problem solution when using custom decorator In Python programming, decorator is a powerful tool that can be used to add rows...

Loading pickle file in Python 3.6 environment error: ModuleNotFoundError:Nomodulenamed...

Using python in Linux terminal...

Compatibility issues between Python asynchronous libraries In Python, asynchronous programming has become the process of high concurrency and I/O...

The problem and solution of the child process continuing to run when using signals to kill the parent process. In Python programming, after killing the parent process through signals, the child process still...

Error loading Pickle file in Python 3.6 environment: ModuleNotFoundError:Nomodulenamed...
