PHP Web Crawling Basics Tutorial: Using cURL Library to Access Websites

With the development of the Internet and the increasing growth of data, web crawlers have become one of the important ways to obtain Internet information. A web crawler is an automated program that accesses a website through network requests, crawls information on the website, processes and analyzes it. In this case, we will introduce how to write a basic web crawler in PHP, use the cURL library to access the website that needs to be crawled, and process the obtained information.

1. cURL library installation

cURL library is a very powerful tool for URL conversion tools that work under the command line. It also supports HTTP/HTTPS. /FTP/TELNET and other network protocols. Using the cURL library, you can easily crawl web data, upload files via FTP, HTTP POST and PUT data, and access remote site resources using basic, digest, or GSS-Negotiate authentication methods. Because the cURL library is very convenient and easy to use, it is widely used in web crawler writing.

In this tutorial, we will demonstrate how to use cURL by using the CURL extension, so first you need to install the cURL extension library in PHP. You can use the following command line to install the cURL extension:

sudo apt-get install php-curl

After installation, we need to restart the php-fpm service to ensure that the extension library can run normally.

2. Basic crawler script skeleton

We will next write a basic web crawler to access a specified URL to obtain some of the URL web pages Basic Information. The following is a basic crawler script skeleton:

<?php
$curl = curl_init();           
$url = "https://www.example.com/";
curl_setopt($curl, CURLOPT_URL, $url);     
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($curl);  
curl_close($curl);       
echo $result;
?>

The above code performs the following operations:

• Initializes a cURL session.
• Set the URL from which we want to extract information.
• Set options to make cURL return data instead of outputting it directly to the screen.
• Execute the request and obtain the data.
• Release cURL session.

You can also customize curl_setopt options as needed to fit your needs. For example, you can add an option to set a timeout using the following line of code:

curl_setopt($curl, CURLOPT_TIMEOUT, 5); // 5秒超时

Additionally, you can use the curl_setopt option to set an HTTP header to simulate a browser sending a request when a website is requested. If you need to set a cookie, you can use the curl_setopt option to set the cookie placeholder or use the related functions in cURL Cookie.

After obtaining the data, you may need to extract, parse and filter it. In this process, you may need to use PHP's string processing functions, regular expressions, or other parsing libraries.

3. Example: Extracting information from a target website

To better understand the process of writing a web crawler, here is an example that demonstrates how to extract information from a website. This website (www.example.com) is a test website from which we can obtain meaningful data.

First, we need to use the cURL library to obtain data from the specified website. The following is the code snippet used to obtain the data:

<?php
$curl = curl_init();
$url = "https://www.example.com/";
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($curl);
curl_close($curl);
echo $result;
?>

Running the above code will output the complete www.example.com website HTML content. Since we need to extract specific information from the obtained website, we need to parse the HTML. We will use the DOMDocument class to parse HTML, such as the following code:

<?php
$curl = curl_init();
$url = "https://www.example.com/";
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($curl);
curl_close($curl);
$dom = new DOMDocument;
$dom->loadHTML($result);
foreach ($dom->getElementsByTagName('a') as $link) {
    echo $link->getAttribute('href'), PHP_EOL;
}
?>

The above code uses the DOMDocument class to load HTML and use the getElementsByTagName() method to obtain all elements. After that, we can use the getAttribute() method to get the href attribute of the corresponding element. Running the code, we can see that the output parses and outputs the URL contained in the HTML tag.

4. Summary

In this article, we introduced how to use the cURL library to write a basic web crawler. We also covered how to extract data from websites and how to parse HTML documents. By understanding these basic concepts, you will be able to better understand how web crawlers work and start writing your own. Of course, there are many complex techniques and issues involved in writing web crawlers, but we hope this article helps you get off to a good start on your web crawler writing journey.

The above is the detailed content of PHP Web Crawling Basics Tutorial: Using cURL Library to Access Websites. For more information, please follow other related articles on the PHP Chinese website!