Distributed crawlers in Scrapy and methods to improve data crawling efficiency

Scrapy is an efficient Python web crawler framework that can write crawler programs quickly and flexibly. However, when processing large amounts of data or complex websites, stand-alone crawlers may encounter performance and scalability issues. At this time, distributed crawlers need to be used to improve data crawling efficiency. This article introduces distributed crawlers in Scrapy and methods to improve data crawling efficiency.

1. What is a distributed crawler?

In the traditional stand-alone crawler architecture, all crawlers run on the same machine. When faced with large amounts of data or high-pressure crawling tasks, machine performance is often tight. Distributed crawlers distribute crawler tasks to multiple machines for processing. Through distributed computing and storage, the burden on a single machine is reduced, thereby improving the efficiency and stability of the crawler.

Distributed crawlers in Scrapy are usually implemented using the open source distributed scheduling framework Distributed Scrapy (DSC for short). DSC distributes Scrapy crawler programs to multiple machines for parallel processing, and uniformly summarizes the results to the central scheduling node.

2. How to implement distributed crawler?

1. Install Distributed Scrapy

Run the following command to install DSC:

pip install scrapy_redis

pip install pymongo

2. Modify Scrapy configuration file

Add the following configuration in the settings.py file of the Scrapy project:

Use redis scheduler

SCHEDULER = "scrapy_redis.scheduler.Scheduler"

Use redis deduplication strategy

DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"

If you do not clear the redis record, you can pause/resume crawling

SCHEDULER_PERSIST=True

Set the connection parameters of redis

REDIS_HOST='localhost'
REDIS_PORT=6379

3. Write the crawler code

In the Scrapy crawler program , you need to modify the starting request method, use the starting method of scrapy-redis:

encoding:utf-8

import scrapy,re,json
from ..items import DouyuItem

from scrapy_redis.spiders import RedisSpider

class DouyuSpider(RedisSpider):

# 爬虫名字
name = 'douyu'
# redis-key，从redis中pop数据进行爬取
redis_key = 'douyu:start_urls'

def parse(self, response):
    # scrapy爬虫代码

4. Start the redis service

Execute the following command in the terminal to start the redis service ：

redis-server

5. Start Distributed Scrapy

Enter the following command in the terminal to start the DSC node:

scrapy crawl douyu -s JOBDIR= job1

Among them, job1 can be a custom name, which is used for DSC to record the crawler status.

3. Optimize Scrapy crawler

Scrapy provides many methods to optimize crawler efficiency. If used with distributed crawlers, data crawling efficiency can be further improved.

1. Using CrawlerRunner

CrawlerRunner requires a Twisted class to extend the application. Compared to simply running a Python file, it allows you to run multiple crawlers simultaneously in the same process without using multiple processes or multiple machines. This can make task management easier.

The way to use CrawlerRunner is as follows:

from twisted.internet import reactor,defer
from scrapy.crawler import CrawlerRunner
from scrapy.utils.project import get_project_settings
from my_spider.spiders.my_spider import MySpider

runner = CrawlerRunner(get_project_settings())

@defer.inlineCallbacks
def crawl():

yield runner.crawl(MySpider)
reactor.stop()

crawl()
reactor.run()

2. Reduce the priority of the download middleware

If you need to process a large amount or complex data in the download middleware, you can use CONCURRENT_REQUESTS_PER_DOMAIN to reduce the priority of the download middleware. Priority:

CONCURRENT_REQUESTS_PER_DOMAIN = 2
DOWNLOAD_DELAY = 0.5
DOWNLOADER_MIDDLEWARES = {
'myproject.middlewares.MyCustomDownloaderMiddleware': 543,
}

3. Adjustment The CONCURRENT_REQUESTS and DOWNLOAD_DELAY parameters

CONCURRENT_REQUESTS indicate the maximum number of requests that each domain name can handle simultaneously, and can be reasonably adjusted according to machine configuration and task requirements.

DOWNLOAD_DELAY represents the delay time between each request. The crawler efficiency can be improved by increasing the delay or asynchronous requests.

4. Summary

Scrapy’s distributed crawler can help us quickly process large amounts of data and improve crawler efficiency. At the same time, crawler efficiency can be further improved by lowering the priority of the download middleware, adjusting the number of coroutines, and increasing the request delay. Distributed crawler is one of the important functions of Scrapy. Learning it can allow us to easily handle various crawler tasks.

The above is the detailed content of Distributed crawlers in Scrapy and methods to improve data crawling efficiency. For more information, please follow other related articles on the PHP Chinese website!