php - snoopy crawler reports error 405 Not Allowed

Question

Code

$httpClass = new Snoopy();
$httpClass->fetch('https://v.qq.com/');
$url = $httpClass->results;
print_r($url);die();

Crawling https://www.baidu.com/ keeps reporting 405 errors

Crawling https://v.qq.com/ is running normally

Looking for an answer, I’ve been working on it all afternoon and still can’t solve it

Answer 1

Baidu probably has a crawler-like setting. You need to disguise it and define a UA or something like that
Reference: http://www.4wei.cn/archives/396

Answer 2

This is not a Snoopy-type problem. It's because you don't know much about crawlers. Since there are crawlers, of course there will be anti-crawler technology. The simplest is based on the browser identifier or the referer in the request header, etc. Big websites like Baidu and Tencent will not want us to crawl data, so they must have taken many preventive measures. Therefore, it is recommended to understand the knowledge of crawlers before crawling data.