Concurrency and multi-threading techniques for PHP crawlers

Concurrency and multi-thread processing skills of PHP crawlers

Introduction:
With the rapid development of the Internet, a large amount of data information is stored on various websites , obtaining this data has become a requirement in many business scenarios. As a tool for automatically obtaining network information, crawlers are widely used in data collection, search engines, public opinion analysis and other fields. This article will introduce a concurrency and multi-threading processing technique for a PHP-based crawler class, and illustrate its implementation through code examples.

1. The basic structure of the reptile class
Before implementing concurrency and multi-thread processing of the reptile class, let's first take a look at the structure of a basic reptile class.

class Crawler {
    private $startUrl;

    public function __construct($startUrl) {
        $this->startUrl = $startUrl;
    }

    public function crawl() {
        // 获取初始页面的内容
        $content = $this->getContent($this->startUrl);

        // 解析页面内容，获取需要的信息
        $data = $this->parseContent($content);

        // 处理获取到的信息，进行业务逻辑处理或存储
        $this->processData($data);

        // 获取页面中的链接，并递归抓取
        $urls = $this->getUrls($content);
        foreach ($urls as $url) {
            $content = $this->getContent($url);
            $data = $this->parseContent($content);
            $this->processData($data);
        }
    }

    private function getContent($url) {
        // 发起HTTP请求，获取页面内容
        // ...
        return $content;
    }

    private function parseContent($content) {
        // 解析页面内容，提取需要的信息
        // ...
        return $data;
    }

    private function processData($data) {
        // 处理获取到的信息，进行逻辑处理或存储
        // ...
    }

    private function getUrls($content) {
        // 获取页面中的链接
        // ...
        return $urls;
    }
}

In the above code, we first define a Crawler class and pass in a starting URL through the constructor. In the crawl() method, we first obtain the content of the starting page, then parse the page content and extract the required information. Afterwards, we can process the obtained information, such as storing it in a database. Finally, we get the links in the page and recursively crawl other pages.

2. Concurrent processing
Normally, crawlers need to process a large number of URLs, and the IO operations of network requests are very time-consuming. If we use sequential execution, requesting the next one after one request is completed will greatly reduce our crawling efficiency. In order to improve concurrent processing capabilities, we can use PHP's multi-process extension to achieve this.

class ConcurrentCrawler {
    private $urls;

    public function __construct($urls) {
        $this->urls = $urls;
    }

    public function crawl() {
        $workers = [];
        $urlsNum = count($this->urls);
        $maxWorkersNum = 10; // 最大进程数

        for ($i = 0; $i < $maxWorkersNum; $i++) {
            $pid = pcntl_fork();
            if ($pid == -1) {
                die('fork failed');
            } else if ($pid == 0) {
                for ($j = $i; $j < $urlsNum; $j += $maxWorkersNum) {
                    $this->processUrl($this->urls[$j]);
                }
                exit();
            } else {
                $workers[$pid] = true;
            }
        }

        while (count($workers)) {
            $pid = pcntl_wait($status, WUNTRACED);
            if ($status == 0) {
                unset($workers[$pid]);
            } else {
                $workers[$pid] = false;
            } 
        }
    }

    private function processUrl($url) {
        // 发起HTTP请求，获取页面内容
        // ...
        // 解析页面内容，获取需要的信息
        // ...
        // 处理获取到的信息，进行逻辑处理或存储
        // ...
    }
}

In the above code, we first define a ConcurrentCrawler class and pass in a set of URLs that need to be crawled through the constructor. In the crawl() method, we use multi-process method for concurrent processing. By using the pcntl_fork() function, a portion of the URL is processed in each child process, while the parent process is responsible for managing the child process. Finally, wait for the end of all child processes through the pcntl_wait() function.

3. Multi-threaded processing
In addition to using multiple processes for concurrent processing, we can also use PHP's Thread extension to implement multi-threaded processing.

class MultithreadCrawler extends Thread {
    private $url;

    public function __construct($url) {
        $this->url = $url;
    }

    public function run() {
        // 发起HTTP请求，获取页面内容
        // ...
        // 解析页面内容，获取需要的信息
        // ...
        // 处理获取到的信息，进行逻辑处理或存储
        // ...
    }
}

class Executor {
    private $urls;

    public function __construct($urls) {
        $this->urls = $urls;
    }

    public function execute() {
        $threads = [];
        foreach ($this->urls as $url) {
            $thread = new MultithreadCrawler($url);
            $thread->start();
            $threads[] = $thread;
        }

        foreach ($threads as $thread) {
            $thread->join();
        }
    }
}

In the above code, we first define a MultithreadCrawler class, which inherits from the Thread class, and rewrites the run() method as the main logic of the thread. In the Executor class, we create multiple threads through a loop and start them for execution. Finally, wait for the end of all threads through the join() method.

Conclusion:
Through the introduction of concurrency and multi-thread processing techniques of PHP crawlers, we can find that both concurrency processing and multi-thread processing can greatly improve the crawler's crawling efficiency. However, in the actual development process, we need to choose the appropriate processing method according to the specific situation. At the same time, in order to ensure the safety of multi-threads or multi-processes, we also need to perform appropriate synchronization operations during processing.

The above is the detailed content of Concurrency and multi-threading techniques for PHP crawlers. For more information, please follow other related articles on the PHP Chinese website!