How to apply concurrent programming in Python crawlers
What is concurrent programming
Concurrent programming refers to a program design that can perform multiple operations within a period of time. It is usually represented by multiple tasks in the program that are started at the same time and can run and interact with each other. There will be no impact. The benefit of concurrent programming is that it can improve the performance and responsiveness of the program.
Application of concurrent programming in crawlers
Crawler programs are typical I/O-intensive tasks. For I/O-intensive tasks, multi-threading and asynchronous I/O are A good choice, because when a certain part of the program is blocked due to I/O operations, other parts of the program can still run, so we don't have to waste a lot of time waiting and blocking.
Single-threaded version
Let’s first look at the single-threaded version of the crawler program. This crawler program uses the requests
library to obtain JSON data and saves the image locally through the open
function.
""" example04.py - 单线程版本爬虫 """ import os import requests def download_picture(url): filename = url[url.rfind('/') + 1:] resp = requests.get(url) if resp.status_code == 200: with open(f'images/beauty/{filename}', 'wb') as file: file.write(resp.content) def main(): if not os.path.exists('images/beauty'): os.makedirs('images/beauty') for page in range(3): resp = requests.get(f'<https://image.so.com/zjl?ch=beauty&sn=>{page * 30}') if resp.status_code == 200: pic_dict_list = resp.json()['list'] for pic_dict in pic_dict_list: download_picture(pic_dict['qhimg_url']) if __name__ == '__main__': main()
On macOS or Linux systems, we can use the time
command to understand the execution time of the above code and the CPU utilization, as shown below.
time python3 example04.py
The following is the result of the single-threaded crawler code executed on my computer.
python3 example04.py 2.36s user 0.39s system 12% cpu 21.578 total
Here we only need to pay attention to the total time consumption of the code which is 21.578
seconds, CPU utilization is 12%
.
Multi-threaded version
We use the thread pool technology mentioned before to modify the above code into a multi-threaded version.
""" example05.py - 多线程版本爬虫 """ import os from concurrent.futures import ThreadPoolExecutor import requests def download_picture(url): filename = url[url.rfind('/') + 1:] resp = requests.get(url) if resp.status_code == 200: with open(f'images/beauty/{filename}', 'wb') as file: file.write(resp.content) def main(): if not os.path.exists('images/beauty'): os.makedirs('images/beauty') with ThreadPoolExecutor(max_workers=16) as pool: for page in range(3): resp = requests.get(f'<https://image.so.com/zjl?ch=beauty&sn=>{page * 30}') if resp.status_code == 200: pic_dict_list = resp.json()['list'] for pic_dict in pic_dict_list: pool.submit(download_picture, pic_dict['qhimg_url']) if __name__ == '__main__': main()
Execute the command shown below.
time python3 example05.py
The execution result of the code is as follows:
python3 example05.py 2.65s user 0.40s system 95% cpu 3.193 total
Asynchronous I/O version
We use aiohttp
to modify the above code to the asynchronous I/O version. In order to achieve network resource acquisition and file writing operations in asynchronous I/O, we must first install the third-party libraries aiohttp
and aiofile
.
pip install aiohttp aiofile
The following is the asynchronous I/O version of the crawler code.
""" example06.py - 异步I/O版本爬虫 """ import asyncio import json import os import aiofile import aiohttp async def download_picture(session, url): filename = url[url.rfind('/') + 1:] async with session.get(url, ssl=False) as resp: if resp.status == 200: data = await resp.read() async with aiofile.async_open(f'images/beauty/{filename}', 'wb') as file: await file.write(data) async def main(): if not os.path.exists('images/beauty'): os.makedirs('images/beauty') async with aiohttp.ClientSession() as session: tasks = [] for page in range(3): resp = await session.get(f'<https://image.so.com/zjl?ch=beauty&sn=>{page * 30}') if resp.status == 200: pic_dict_list = (await resp.json())['list'] for pic_dict in pic_dict_list: tasks.append(asyncio.ensure_future(download_picture(session, pic_dict['qhimg_url']))) await asyncio.gather(*tasks) if __name__ == '__main__': loop = asyncio.get_event_loop() loop.run_until_complete(main())
Execute the command shown below.
time python3 example06.py
The execution result of the code is as follows:
python3 example06.py 0.92s user 0.27s system 290% cpu 0.420 total
Compared with the single-threaded version of the crawler program, the execution time of the multi-threaded version and the asynchronous I/O version of the crawler program has been significantly improved, and the asynchronous I/O version The /O version of the crawler performs best.
The above is the detailed content of How to apply concurrent programming in Python crawlers. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics



In VS Code, you can run the program in the terminal through the following steps: Prepare the code and open the integrated terminal to ensure that the code directory is consistent with the terminal working directory. Select the run command according to the programming language (such as Python's python your_file_name.py) to check whether it runs successfully and resolve errors. Use the debugger to improve debugging efficiency.

VS Code can be used to write Python and provides many features that make it an ideal tool for developing Python applications. It allows users to: install Python extensions to get functions such as code completion, syntax highlighting, and debugging. Use the debugger to track code step by step, find and fix errors. Integrate Git for version control. Use code formatting tools to maintain code consistency. Use the Linting tool to spot potential problems ahead of time.

VS Code extensions pose malicious risks, such as hiding malicious code, exploiting vulnerabilities, and masturbating as legitimate extensions. Methods to identify malicious extensions include: checking publishers, reading comments, checking code, and installing with caution. Security measures also include: security awareness, good habits, regular updates and antivirus software.

VS Code can run on Windows 8, but the experience may not be great. First make sure the system has been updated to the latest patch, then download the VS Code installation package that matches the system architecture and install it as prompted. After installation, be aware that some extensions may be incompatible with Windows 8 and need to look for alternative extensions or use newer Windows systems in a virtual machine. Install the necessary extensions to check whether they work properly. Although VS Code is feasible on Windows 8, it is recommended to upgrade to a newer Windows system for a better development experience and security.

Python excels in automation, scripting, and task management. 1) Automation: File backup is realized through standard libraries such as os and shutil. 2) Script writing: Use the psutil library to monitor system resources. 3) Task management: Use the schedule library to schedule tasks. Python's ease of use and rich library support makes it the preferred tool in these areas.

VS Code is the full name Visual Studio Code, which is a free and open source cross-platform code editor and development environment developed by Microsoft. It supports a wide range of programming languages and provides syntax highlighting, code automatic completion, code snippets and smart prompts to improve development efficiency. Through a rich extension ecosystem, users can add extensions to specific needs and languages, such as debuggers, code formatting tools, and Git integrations. VS Code also includes an intuitive debugger that helps quickly find and resolve bugs in your code.

VS Code not only can run Python, but also provides powerful functions, including: automatically identifying Python files after installing Python extensions, providing functions such as code completion, syntax highlighting, and debugging. Relying on the installed Python environment, extensions act as bridge connection editing and Python environment. The debugging functions include setting breakpoints, step-by-step debugging, viewing variable values, and improving debugging efficiency. The integrated terminal supports running complex commands such as unit testing and package management. Supports extended configuration and enhances features such as code formatting, analysis and version control.

Yes, VS Code can run Python code. To run Python efficiently in VS Code, complete the following steps: Install the Python interpreter and configure environment variables. Install the Python extension in VS Code. Run Python code in VS Code's terminal via the command line. Use VS Code's debugging capabilities and code formatting to improve development efficiency. Adopt good programming habits and use performance analysis tools to optimize code performance.
