Use file_get_contents and curl to write collection

It is often found that the data to be used is all on the same website, and the data presentation format is the same. For example, there are thousands of products on Taobao or Amazon. If the information is entered manually, the workload will be too much. At this time, we can write a collection program to directly collect and display it. The server supports file_get_contents and curl

First add a text box and submit button to the page. The text box is used to enter the collection page address.

Collection needs to use the regular interception function

function preg_substr($start, $end, $str) // 正则截取函数      
{      
    $temp = preg_split($start, $str);      
    $content = preg_split($end, $temp[1]);      
    return $content[0];      
}

Collection needs to use the string interception function

function str_substr($start, $end, $str) // 字符串截取函数      
{      
    $temp = explode($start, $str, 2);      
    $content = explode($end, $temp[1], 2);      
    return $content[0];      
}

There is also a function to save the collected content:

function writelog($str)
{
@unlink("log.txt");
$open=fopen("log.txt","a" );
fwrite($open,$str);
fclose($open);
}

Sometimes the collected content is inconsistent with the content we view through the browser, causing us to not be able to find the correct regular expression, here you can Open the saved txt file and find the correct intercepted string in it.

If you need to collect even pictures, you need to use the picture function:

function getImage($url, $filename='', $dirName, $fileType, $type=0)
{
    if($url == ''){return false;}
    //获取文件原文件名
    $defaultFileName = basename($url);
    //获取文件类型
    $suffix = substr(strrchr($url,'.'), 1);
    if(!in_array($suffix, $fileType)){
        return false;
    }
    //设置保存后的文件名
    $filename = $filename == '' ? time().rand(0,9).'.'.$suffix : $defaultFileName;
          
    //获取远程文件资源
    if($type){
        $ch = curl_init();
        $timeout = 5;
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
        $file = curl_exec($ch);
        curl_close($ch);
    }else{
        ob_start();
        readfile($url);
        $file = ob_get_contents();
        ob_end_clean();
    }
    //设置文件保存路径
    $dirName = $dirName.'/'.date('Y', time()).'/'.date('m', time()).'/'.date('d',time()).'/';
    if(!file_exists($dirName)){
        mkdir($dirName, 0777, true);
    }
    //保存文件
    $res = fopen($dirName.$filename,'a');
    fwrite($res,$file);
    fclose($res);
    return $dirName.$filename;
}

Add the collection code. Since the collection code is added here to prevent submission, you can directly Above;

Let’s take a product page on Amazon as an example: enter a product link:

Look at the collection results as shown below. Only the content is displayed here. It is relatively simple to add it to the database. There will be time to introduce the collection of automatically entering lower-level links or automatically turning pages.

The above is the detailed content of Use file_get_contents and curl to write collection. For more information, please follow other related articles on the PHP Chinese website!