Example of forging random request headers for crawlers in Pyspider-Python Tutorial-php.cn

This article mainly introduces examples of forging random request headers for crawlers in Pyspider. It has a certain reference value. Now I share it with you. Friends in need can refer to it.

Pyspider uses tornado The library makes http requests. Various parameters can be added during the request process, such as request link timeout, request transmission data timeout, request headers, etc. However, according to the original framework of pyspider, adding parameters to the crawler can only be done through the crawl_config Python dictionary. To complete (as shown below), the framework code converts the parameters in this dictionary into task data and makes an http request. The disadvantage of this parameter is that it is inconvenient to make random request headers for each request.

crawl_config = {
"user_agent": "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36",
"timeout": 120,
"connect_timeout": 60,
"retries": 5,
"fetch_type": &#39;js&#39;,
"auto_recrawl": True,
}

Copy after login

Here is how to add random request headers to the crawler:

1 , write a script, place the script in the libs folder of pyspider, and name it header_switch.py

#!/usr/bin/env python
# -*- coding:utf-8 -*-
# Created on 2017-10-18 11:52:26
import random
import time
class HeadersSelector(object):
  """
  Header 中缺少几个字段 Host 和 Cookie
  """
  headers_1 = {
    "Proxy-Connection": "keep-alive",
    "Pragma": "no-cache",
    "Cache-Control": "no-cache",
    "User-Agent": "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "DNT": "1",
    "Accept-Encoding": "gzip, deflate, sdch",
    "Accept-Language": "zh-CN,zh;q=0.8,en-US;q=0.6,en;q=0.4",
    "Referer": "https://www.baidu.com/s?wd=%BC%96%E7%A0%81&rsv_spt=1&rsv_iqid=0x9fcbc99a0000b5d7&issp=1&f=8&rsv_bp=1&rsv_idx=2&ie=utf-8&rqlang=cn&tn=baiduhome_pg&rsv_enter=0&oq=If-None-Match&inputT=7282&rsv_t",
    "Accept-Charset": "gb2312,gbk;q=0.7,utf-8;q=0.7,*;q=0.7",
  } # 网上找的浏览器
  headers_2 = {
    "Proxy-Connection": "keep-alive",
    "Pragma": "no-cache",
    "Cache-Control": "no-cache",
    "User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0",
    "Accept": "image/gif,image/x-xbitmap,image/jpeg,application/x-shockwave-flash,application/vnd.ms-excel,application/vnd.ms-powerpoint,application/msword,*/*",
    "DNT": "1",
    "Referer": "https://www.baidu.com/link?url=c-FMHf06-ZPhoRM4tWduhraKXhnSm_RzjXZ-ZTFnPAvZN",
    "Accept-Encoding": "gzip, deflate, sdch",
    "Accept-Language": "zh-CN,zh;q=0.8,en-US;q=0.6,en;q=0.4",
  } # window 7 系统浏览器
  headers_3 = {
    "Proxy-Connection": "keep-alive",
    "Pragma": "no-cache",
    "Cache-Control": "no-cache",
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0",
    "Accept": "image/x-xbitmap,image/jpeg,application/x-shockwave-flash,application/vnd.ms-excel,application/vnd.ms-powerpoint,application/msword,*/*",
    "DNT": "1",
    "Referer": "https://www.baidu.com/s?wd=http%B4%20Pragma&rsf=1&rsp=4&f=1&oq=Pragma&tn=baiduhome_pg&ie=utf-8&usm=3&rsv_idx=2&rsv_pq=e9bd5e5000010",
    "Accept-Encoding": "gzip, deflate, sdch",
    "Accept-Language": "zh-CN,zh;q=0.8,en-US;q=0.7,en;q=0.6",
  } # Linux 系统 firefox 浏览器
  headers_4 = {
    "Proxy-Connection": "keep-alive",
    "Pragma": "no-cache",
    "Cache-Control": "no-cache",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:55.0) Gecko/20100101 Firefox/55.0",
    "Accept": "*/*",
    "DNT": "1",
    "Referer": "https://www.baidu.com/link?url=c-FMHf06-ZPhoRM4tWduhraKXhnSm_RzjXZ-ZTFnP",
    "Accept-Encoding": "gzip, deflate, sdch",
    "Accept-Language": "zh-CN,zh;q=0.9,en-US;q=0.7,en;q=0.6",
  } # Win10 系统 firefox 浏览器
  headers_5 = {
    "Connection": "keep-alive",
    "Pragma": "no-cache",
    "Cache-Control": "no-cache",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64;) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "Referer": "https://www.baidu.com/link?url=c-FMHf06-ZPhoRM4tWduhraKXhnSm_RzjXZ-",
    "Accept-Encoding": "gzip, deflate, sdch",
    "Accept-Language": "zh-CN,zh;q=0.9,en-US;q=0.7,en;q=0.6",
    "Accept-Charset": "gb2312,gbk;q=0.7,utf-8;q=0.7,*;q=0.7",
  } # Win10 系统 Chrome 浏览器
  headers_6 = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "Accept-Encoding": "gzip, deflate, sdch",
    "Accept-Language": "zh-CN,zh;q=0.8",
    "Pragma": "no-cache",
    "Cache-Control": "no-cache",
    "Connection": "keep-alive",
    "DNT": "1",
    "Referer": "https://www.baidu.com/s?wd=If-None-Match&rsv_spt=1&rsv_iqid=0x9fcbc99a0000b5d7&issp=1&f=8&rsv_bp=1&rsv_idx=2&ie=utf-8&rq",
    "Accept-Charset": "gb2312,gbk;q=0.7,utf-8;q=0.7,*;q=0.7",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0",
  } # win10 系统浏览器
  def __init__(self):
    pass
  def select_header(self):
    n = random.randint(1, 6)
    switch={
    1: self.headers_1
    2: self.headers_2
    3: self.headers_3
    4: self.headers_4
    5: self.headers_5
    6: self.headers_6
    }
    headers = switch[n]
    return headers

Copy after login

Among them, I only write There are 6 request headers. If the number of crawlers is very large, you can write more request headers, even hundreds, and then expand the random range of random for selection.

2. Write the following code in the pyspider script:

#!/usr/bin/env python
# -*- encoding: utf-8 -*-
# Created on 2017-08-18 11:52:26
from pyspider.libs.base_handler import *
from pyspider.addings.headers_switch import HeadersSelector
import sys
defaultencoding = &#39;utf-8&#39;
if sys.getdefaultencoding() != defaultencoding:
  reload(sys)
  sys.setdefaultencoding(defaultencoding)
class Handler(BaseHandler):
  crawl_config = {
    "user_agent": "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36",
    "timeout": 120,
    "connect_timeout": 60,
    "retries": 5,
    "fetch_type": &#39;js&#39;,
    "auto_recrawl": True,
  }
  @every(minutes=24 * 60)
  def on_start(self):
    header_slt = HeadersSelector()
    header = header_slt.select_header() # 获取一个新的 header
    # header["X-Requested-With"] = "XMLHttpRequest"
    orig_href = &#39;http://sww.bjxch.gov.cn/gggs.html&#39;
    self.crawl(orig_href,
          callback=self.index_page,
          headers=header) # 请求头必须写在 crawl 里，cookies 从 response.cookies 中找
  @config(age=24 * 60 * 60)
  def index_page(self, response):
    header_slt = HeadersSelector()
    header = header_slt.select_header() # 获取一个新的 header
    # header["X-Requested-With"] = "XMLHttpRequest"
    if response.cookies:
      header["Cookies"] = response.cookies

Copy after login

The most important one What's more, in each callback function on_start, index_page, etc., a header selector will be instantiated every time it is called, and a different header will be added to each request. Pay attention to the following code added:

    header_slt = HeadersSelector()
    header = header_slt.select_header() # 获取一个新的 header
    # header["X-Requested-With"] = "XMLHttpRequest"
    header["Host"] = "www.baidu.com"
    if response.cookies:
      header["Cookies"] = response.cookies

Copy after login

When using XHR to send an AJAX request, a Header will be included, which is often used to determine whether it is an Ajax request. The headers must be Add {'X-Requested-With': 'XMLHttpRequest'} to capture the content.

Once the url is determined, the Host in the request header is determined, which needs to be added as needed. The urlparse package provides a method function for parsing the host based on the url. Just call netloc directly.

If there is a cookie in the response, the cookie needs to be added to the request header.

If you have other disguise requirements, add them yourself.

In this way, random request headers can be realized, complete.

Related recommendations:

Using Python’s Pyspider as an example to analyze the search engine web crawler implementation method

The above is the detailed content of Example of forging random request headers for crawlers in Pyspider. For more information, please follow other related articles on the PHP Chinese website!