Detailed tutorial on how to use scrapy shell to verify the results of xpath selection

1. scrapy shell

is a good interactive tool of scrapy package. Currently I use it mainly to verify the results of xpath selection. After installing scrapy, you can operate the scrapy shell directly on cmd.

Scrapy Shell

The Scrapy terminal is an interactive terminal. We can try and debug the code without starting the spider. It can also be used to test XPath or CSS expressions and see how they work. way to facilitate the extraction of data from the web pages we crawl.

If IPython is installed, the Scrapy terminal will use IPython (instead of the standard Python terminal). The IPython terminal is more powerful than others, providing intelligent auto-completion, highlighted output, and other features. (It is recommended to install IPython)

Start Scrapy Shell

Enter the root directory of the project and execute the following command to start the shell:

scrapy shell "http://www.itcast. cn/channel/teacher.shtml"

Scrapy Shell will automatically create some convenient objects based on the downloaded page, such as Response object and Selector object (for HTML and XML content).

When the shell is loaded, you will get a local response variable containing response data. Entering response.body will output the response body, and output response.headers to see the response header.

When you enter response.selector, you will get an object of class Selector initialized by response. At this time, you can query the response by using response.selector.xpath() or response.selector.css().

Scrapy also provides some shortcuts, such as response.xpath() or response.css() which can also take effect (as in the previous case).

Selectors selector

Scrapy Selectors built-in XPath and CSS Selector expression mechanism

Selector has four basic methods, the most commonly used is xpath:

xpath(): Pass in the xpath expression and return the selector list of all nodes corresponding to the expression

extract(): Serialize the node into a Unicode string and return the list

css(): Pass in a CSS expression and return the selector list of all nodes corresponding to the expression. The syntax is the same as BeautifulSoup4

#re(): Extract data based on the passed in regular expression. Return the Unicode string list list

2. ipython

on the official website It is recommended to use ipython to run scrapy shell, so I tried to install it. Because my python environment was configured through conda before (see the previous article), it is very convenient to install ipython through conda

conda install -c conda-forge ipython

Then the entire ipython package will be downloaded, because all It is compiled, and there is no annoying compilation failure process.

3. Run ipython and run scrapy shell on ipython

In the current cmd run box, because The system environment has been configured and you can run the python package directly, so directly typing ipython in the cmd run box will enter the ipython run box, which is similar to the system standard cmd, but has richer functions, richer colors, and layout. It can be good too.

But when I type the scrapy shell command directly on it, it keeps saying that there is no such command and fails. Stuck here.

Later by carefully reading the instructions of scrapy shell

If you have IPython installed, the Scrapy shell will use it (instead of the standard Python console).

It means that scrapy shell will find the ipython running box by itself.

So directly enter scrapy shell in the standard run box of cmd, and the returned result is directly called to the ipython run box.

The above is the detailed content of Detailed tutorial on how to use scrapy shell to verify the results of xpath selection. For more information, please follow other related articles on the PHP Chinese website!