Python crawler programming framework Scrapy introductory learning tutorial

1. Introduction to Scrapy
Scrapy is an application framework written to crawl website data and extract structured data. It can be used in a series of programs including data mining, information processing or storing historical data.
It was originally designed for page scraping (more specifically, web scraping), but can also be used to obtain data returned by APIs (such as Amazon Associates Web Services) or general web crawlers. Scrapy is widely used and can be used for data mining, monitoring and automated testing
Scrapy uses the Twisted asynchronous network library to handle network communication. The overall structure is roughly as follows

Scrapy mainly includes the following components:

(1) Engine (Scrapy): used to handle data flow processing of the entire system and trigger transactions (core framework)

(2) Scheduler (Scheduler): Used to accept requests from the engine, push them into the queue, and return when the engine requests again. It can be imagined as a URL (the URL or link of the web page that is captured ), which determines the next URL to be crawled and removes duplicate URLs

(3) Downloader: used to download web content and return the web content to the spider (Scrapy downloader is built on twisted, an efficient asynchronous model)

(4) Spiders: Crawlers are mainly used to extract the information they need from specific web pages, which are so-called entities (Items). Users can also extract links from it and let Scrapy continue to crawl the next page

Project Pipeline: Responsible for processing entities extracted from web pages by crawlers. The main function is to persist entities, verify the validity of entities, and remove unnecessary information. When the page is parsed by the crawler, it will be sent to the project pipeline and the data will be processed through several specific sequences.

(5) Downloader Middlewares: A framework located between the Scrapy engine and the downloader. It mainly handles requests and responses between the Scrapy engine and the downloader.

(6) Spider Middlewares: A framework between the Scrapy engine and the crawler. Its main job is to process the spider's response input and request output.

(7) Scheduler Middewares: The middleware between the Scrapy engine and the scheduler, sending requests and responses from the Scrapy engine to the scheduler.

The Scrapy operation process is roughly as follows:

First, the engine takes out a link (URL) from the scheduler for subsequent crawling
The engine encapsulates the URL into a request (Request) and passes it to the downloader. The downloader downloads the resource and encapsulates it into a response package (Response)
Then, the crawler parses Response
If the entity (Item) is parsed, it is handed over to the entity pipeline for further processing.
If the parsed result is a link (URL), the URL will be handed over to the Scheduler to wait for crawling

2. Install Scrapy
Use the following command:

sudo pip install virtualenv #安装虚拟环境工具
virtualenv ENV #创建一个虚拟环境目录
source ./ENV/bin/active #激活虚拟环境
pip install Scrapy
#验证是否安装成功
pip list

#输出如下
cffi (0.8.6)
cryptography (0.6.1)
cssselect (0.9.1)
lxml (3.4.1)
pip (1.5.6)
pycparser (2.10)
pyOpenSSL (0.14)
queuelib (1.2.2)
Scrapy (0.24.4)
setuptools (3.6)
six (1.8.0)
Twisted (14.0.2)
w3lib (1.10.0)
wsgiref (0.1.2)
zope.interface (4.1.1)

For more virtual environment operations, please check my blog post

3. Scrapy Tutorial
Before scraping, you need to create a new Scrapy project. Enter a directory where you want to save the code, and then execute:

$ scrapy startproject tutorial

This command will create a new directory tutorial in the current directory, with the following structure:

.
├── scrapy.cfg
└── tutorial
 ├── __init__.py
 ├── items.py
 ├── pipelines.py
 ├── settings.py
 └── spiders
  └── __init__.py

These files are mainly:

(1) scrapy.cfg: project configuration file
(2) tutorial/: project python module, you will add code here later
(3) tutorial/items.py: project items file
(4) tutorial/pipelines.py: project pipeline file
(5) tutorial/settings.py: project configuration file
(6) tutorial/spiders: directory where spiders are placed

3.1. Define Item
Items is a container that will be loaded with scraped data. It works like a dictionary in python, but it provides more protection, such as padding undefined fields to prevent spelling errors

Declare an Item by creating a scrapy.Item class and defining a class attribute of type scrapy.Field.
We control the site data obtained from dmoz.org by modeling the required items. For example, if we want to obtain the site name, URL and site description, we define the fields of these three attributes. Edit the items.py file in the tutorial directory

from scrapy.item import Item, Field


class DmozItem(Item):
 # define the fields for your item here like:
 name = Field()
 description = Field()
 url = Field()

3.2. Writing Spider
Spider is a user-written class used to crawl information from a domain (or domain group), defining a preliminary list of URLs for downloading, how to follow the links, and how to parse the content of these web pages for Extract items.

To create a Spider, inherit the scrapy.Spider base class and identify three main, mandatory properties:

name：爬虫的识别名，它必须是唯一的，在不同的爬虫中你必须定义不同的名字.
start_urls：包含了Spider在启动时进行爬取的url列表。因此，第一个被获取到的页面将是其中之一。后续的URL则从初始的URL获取到的数据中提取。我们可以利用正则表达式定义和过滤需要进行跟进的链接。
parse()：是spider的一个方法。被调用时，每个初始URL完成下载后生成的 Response 对象将会作为唯一的参数传递给该函数。该方法负责解析返回的数据(response data)，提取数据(生成item)以及生成需要进一步处理的URL的 Request 对象。
这个方法负责解析返回的数据、匹配抓取的数据(解析为 item )并跟踪更多的 URL。
在 /tutorial/tutorial/spiders 目录下创建 dmoz_spider.py

import scrapy

class DmozSpider(scrapy.Spider):
 name = "dmoz"
 allowed_domains = ["dmoz.org"]
 start_urls = [
  "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
  "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
 ]

 def parse(self, response):
  filename = response.url.split("/")[-2]
  with open(filename, 'wb') as f:
   f.write(response.body)

3.3. 爬取
当前项目结构

├── scrapy.cfg
└── tutorial
 ├── __init__.py
 ├── items.py
 ├── pipelines.py
 ├── settings.py
 └── spiders
  ├── __init__.py
  └── dmoz_spider.py

到项目根目录, 然后运行命令:

$ scrapy crawl dmoz

运行结果:

2014-12-15 09:30:59+0800 [scrapy] INFO: Scrapy 0.24.4 started (bot: tutorial)
2014-12-15 09:30:59+0800 [scrapy] INFO: Optional features available: ssl, http11
2014-12-15 09:30:59+0800 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tutorial.spiders', 'SPIDER_MODULES': ['tutorial.spiders'], 'BOT_NAME': 'tutorial'}
2014-12-15 09:30:59+0800 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2014-12-15 09:30:59+0800 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2014-12-15 09:30:59+0800 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2014-12-15 09:30:59+0800 [scrapy] INFO: Enabled item pipelines:
2014-12-15 09:30:59+0800 [dmoz] INFO: Spider opened
2014-12-15 09:30:59+0800 [dmoz] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2014-12-15 09:30:59+0800 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2014-12-15 09:30:59+0800 [scrapy] DEBUG: Web service listening on 127.0.0.1:6080
2014-12-15 09:31:00+0800 [dmoz] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/> (referer: None)
2014-12-15 09:31:00+0800 [dmoz] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> (referer: None)
2014-12-15 09:31:00+0800 [dmoz] INFO: Closing spider (finished)
2014-12-15 09:31:00+0800 [dmoz] INFO: Dumping Scrapy stats:
 {'downloader/request_bytes': 516,
  'downloader/request_count': 2,
  'downloader/request_method_count/GET': 2,
  'downloader/response_bytes': 16338,
  'downloader/response_count': 2,
  'downloader/response_status_count/200': 2,
  'finish_reason': 'finished',
  'finish_time': datetime.datetime(2014, 12, 15, 1, 31, 0, 666214),
  'log_count/DEBUG': 4,
  'log_count/INFO': 7,
  'response_received_count': 2,
  'scheduler/dequeued': 2,
  'scheduler/dequeued/memory': 2,
  'scheduler/enqueued': 2,
  'scheduler/enqueued/memory': 2,
  'start_time': datetime.datetime(2014, 12, 15, 1, 30, 59, 533207)}
2014-12-15 09:31:00+0800 [dmoz] INFO: Spider closed (finished)

3.4. 提取Items
3.4.1. 介绍Selector
从网页中提取数据有很多方法。Scrapy使用了一种基于 XPath 或者 CSS 表达式机制： Scrapy Selectors

出XPath表达式的例子及对应的含义:

• /html/head/title: 选择HTML文档中 标签内的