Using proxy IP and anti-crawling strategies in Scrapy crawler

Using proxy IP and anti-crawler strategies in Scrapy crawlers

In recent years, with the development of the Internet, more and more data needs to be obtained through crawlers, and the anti-crawler strategies for crawlers have become more and more important. Becoming more and more strict. In many scenarios, using proxy IP and anti-crawler strategies have become essential skills for crawler developers. In this article, we will discuss how to use proxy IP and anti-crawling strategies in Scrapy crawlers to ensure the stability and success rate of crawling data.

1. Why you need to use a proxy IP

When a crawler accesses the same website, it will often be identified as the same IP address, which can easily be blocked or restricted. To prevent this from happening, a proxy IP needs to be used to hide the real IP address, thus better protecting the identity of the crawler.

2. How to use proxy IP

Using proxy IP in Scrapy can be achieved by setting the DOWNLOADER_MIDDLEWARES attribute in the settings.py file.

1. Add the following code in the settings.py file:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 1,
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'your_project.middlewares.RandomUserAgentMiddleware': 400,
    'your_project.middlewares.RandomProxyMiddleware': 410,
}

2. Define the RandomProxyMiddleware class in the middlewares.py file to implement the random proxy IP function:

import requests
import random


class RandomProxyMiddleware(object):
    def __init__(self, proxy_list_path):
        with open(proxy_list_path, 'r') as f:
            self.proxy_list = f.readlines()

    @classmethod
    def from_crawler(cls, crawler):
        settings = crawler.settings
        return cls(settings.get('PROXY_LIST_PATH'))

    def process_request(self, request, spider):
        proxy = random.choice(self.proxy_list).strip()
        request.meta['proxy'] = "http://" + proxy

Among them, the path to the proxy IP list needs to be set in the settings.py file:

PROXY_LIST_PATH = 'path/to/your/proxy/list'

When crawling, Scrapy will randomly select a proxy IP for access, thus This ensures the concealment of identity and the success rate of crawling.

3. About anti-crawler strategies

At present, anti-crawler strategies for websites are very common, ranging from simple User-Agent judgment to more complex verification codes and sliding bar verification. Below, we will discuss how to deal with several common anti-crawling strategies in Scrapy crawlers.

1. User-Agent anti-crawler

In order to prevent crawler access, websites often determine the User-Agent field. If the User-Agent is not the browser's method, it will Intercept it. Therefore, we need to set a random User-Agent in the Scrapy crawler to avoid the User-Agent being recognized as a crawler.

Under middlewares.py, we define the RandomUserAgentMiddleware class to implement the random User-Agent function:

import random
from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware


class RandomUserAgentMiddleware(UserAgentMiddleware):
    def __init__(self, user_agent):
        self.user_agent = user_agent

    @classmethod
    def from_crawler(cls, crawler):
        s = cls(crawler.settings.get('user_agent', 'Scrapy'))
        crawler.signals.connect(s.spider_closed, signal=signals.spider_closed)
        return s

    def process_request(self, request, spider):
        ua = random.choice(self.user_agent_list)
        if ua:
            request.headers.setdefault('User-Agent', ua)

At the same time, set the User-Agent list in the settings.py file:

USER_AGENT_LIST = ['Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36']

2. IP Anti-Crawler

In order to prevent a large number of requests from the same IP address, the website may restrict or prohibit access to requests from the same IP address. For this situation, we can use proxy IP to avoid IP anti-crawlers by randomly switching IP addresses.

3. Cookies and Session Anti-Crawler

Websites may identify the identity of the request by setting Cookies and Session, etc. These methods are often bound to accounts, and also The frequency of requests per account will be limited. Therefore, we need to simulate Cookies and Session in the Scrapy crawler to avoid being identified as illegal requests.

In Scrapy's settings.py file, we can configure the following:

COOKIES_ENABLED = True
COOKIES_DEBUG = True

At the same time, define the CookieMiddleware class in the middlewares.py file to simulate the Cookies function:

from scrapy.exceptions import IgnoreRequest


class CookieMiddleware(object):
    def __init__(self, cookies):
        self.cookies = cookies

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            cookies=crawler.settings.getdict('COOKIES')
        )

    def process_request(self, request, spider):
        request.cookies.update(self.cookies)

Among them, the COOKIES settings are as follows:

COOKIES = {
    'cookie1': 'value1',
    'cookie2': 'value2',
    ...
}

Cookies should be added to the cookies field of the request before the request is sent. If the request does not carry cookies, it is likely to be identified as an illegal request by the website.

4. Summary

The above is an introduction to the use of proxy IP and anti-crawler strategies in Scrapy crawlers. Using proxy IP and anti-crawler strategies is an important means to prevent crawlers from being restricted and banned. Of course, anti-crawler strategies emerge in endlessly, and we need to deal with different anti-crawler strategies accordingly.

The above is the detailed content of Using proxy IP and anti-crawling strategies in Scrapy crawler. For more information, please follow other related articles on the PHP Chinese website!