How to install php crawler framework

When it comes to making a crawler, the first thing that everyone may think of is Python. In fact, PHP can also be used to write crawler programs. PHP has always been simple and easy to use. I personally tested that I can write a simple crawler program in 10 minutes using the PHPspider framework.

1. PHP environment installation

Like python, PHP also needs an environment. You can use PHP downloaded from the official website, or you can use XAMPP, PHPstudy and other integrated environments. PHP. An integrated environment is recommended, eliminating the need to install the Mysql database separately.

2. Composer installation

composer is a dependency package management tool under PHP, similar to PIP in Python.

The Chinese official website is https://www.phpcomposer.com/

. Just download and install it. Run cmd in win R and enter the composer command. If the following picture appears, the installation is successful.

3. PHPspider installation

Create a folder in any location. For example, if we want to capture the data of Jianshu, we You can create the jianshu folder on the D drive, then enter the folder with the cmd command, and run the command:

composer require owner888/phpspider

The following result is a successful installation.

Related recommendations: "php environment construction"

4. Start writing the first crawler

Now open the jianshu folder and you will find that there are some more things in it. Don't worry about it. Create a php file and start coding.

The development documentation is here: https://doc.phpspider.org/demo-start.html

I won’t talk about the basics here, just go to the code. , because we are doing a 10-minute quick tutorial.

The matching method uses XPach syntax.

<?php
require &#39;/vendor/autoload.php&#39;;
use phpspider\core\phpspider;
/* Do NOT delete this comment */
/* 不要删除这段注释 */
$configs = array(
&#39;name&#39; => &#39;简书&#39;,
&#39;log_show&#39; =>false,
&#39;tasknum&#39; => 1,
//数据库配置
&#39;db_config&#39; => array(
&#39;host&#39;  => &#39;127.0.0.1&#39;,
&#39;port&#39;  => 3306,
&#39;user&#39;  => &#39;root&#39;,
&#39;pass&#39;  => &#39;&#39;,
&#39;name&#39;  => &#39;demo&#39;,
),
&#39;export&#39; => array(
&#39;type&#39; => &#39;db&#39;,
&#39;table&#39; => &#39;jianshu&#39;,  // 如果数据表没有数据新增请检查表结构和字段名是否匹配
),
//爬取的域名列表  
&#39;domains&#39; => array(
    &#39;jianshu&#39;,
    &#39;www.jianshu.com&#39;
), 
//抓取的起点
&#39;scan_urls&#39; => array(
    &#39;https://www.jianshu.com/c/V2CqjW?utm_medium=index-collections&utm_source=desktop&#39;
),
//列表页实例
&#39;list_url_regexes&#39; => array(
    "https://www.jianshu.com/c/\d+"
),
//内容页实例
//  \d+  指的是变量
&#39;content_url_regexes&#39; => array(
    "https://www.jianshu.com/p/\d+",
),
&#39;max_try&#39; => 5,
&#39;fields&#39; => array(
    array(
        &#39;name&#39; => "title",
        &#39;selector&#39; => "//h1[@class=&#39;title&#39;]",
        &#39;required&#39; => true,
    ),
    array(
        &#39;name&#39; => "content",
        &#39;selector&#39; => "//div[@class=&#39;show-content-free&#39;]",
        &#39;required&#39; => true,
    ),
),
);
$spider = new phpspider($configs);
$spider->start();

Let’s explain the meaning of the syntax a little bit:

//h1[@class='title']

Get all h1 nodes with class value of title

//div[@class='show-content-free']

Get all divs with class value of show-content-free After finishing the code for node

, remember to create the corresponding database and data table according to the content to be captured, and the fields must be aligned.

Then cmd, enter:

php -f d:\jianshu\spider.php

Run as follows:

Open the data and take a look. Have you captured everything?

The above is the detailed content of How to install php crawler framework. For more information, please follow other related articles on the PHP Chinese website!