php curl抓取不到页面及来路问题?

WBOY
Release: 2016-06-17 08:31:58
Original
2469 people have browsed it

$url = "mp.weixinbridge.com/mp/";
$ch = curl_init();
$timeout = 1;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$contents = curl_exec($ch);
curl_close($ch);
echo $contents;
相对路径的图片不能显示,如何使相对路径的图片正常显示?

回复内容:

谢邀,哥们,抓不到数据是因为:
1.你没有写header
2.没有写cookie,
3.没有针对https的url特殊设置
所以没有抓到数据,好好研究我写的这个代码,这个是可以抓到数据的。
要是帮到了你,给哥点个赞,支持下。
<code class="language-text"><?php $url = "https://www.zhihu.com/";

	$ch = curl_init();
    // 设置浏览器的特定header
    curl_setopt($ch, CURLOPT_HTTPHEADER, array(
        "Host: www.zhihu.com",
        "Connection: keep-alive",
        "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Upgrade-Insecure-Requests: 1",
        "DNT:1",
        "Accept-Language: zh-CN,zh;q=0.8,en-GB;q=0.6,en;q=0.4,en-US;q=0.2",
        'Cookie:_za=4540d427-eee1-435a-a533-66ecd8676d7d; __utma=51854390.3169871.1440319332.1441339521.1442067491.5; __utmz=51854390.1442067491.5.5.utmcsr=baidu|utmccn=(organic)|utmcmd=organic; __utmv=51854390.100-1|2=registration_date=20140525=1^3=entry_date=20140525=1; q_c1=efa8c4ccdba04f63a0ba88845f485836|1451394239000|1440047640000; _xsrf=20c250b28098f92459cac05a3944d48d; cap_id="ZWQ5OGIzN2JiZWNmNGRlNGE3YTE1MTE0YTA5YjY1NjE=|1451394239|0efd13fc965c43c0fb6a7a2523b5dac4d1dac7e3"; z_c0="QUFCQXRLa3ZBQUFYQUFBQVlRSlZUY29ScWxZN0k3T1BHaFdqb1JNVlVZekNnZ0trU0xXdEdnPT0=|1451394250|02ed77acc81edbf2340fd0ce1b13618862b3674e"; unlock_ticket="QUFCQXRLa3ZBQUFYQUFBQVlRSlZUZEtMZ2xiM21FNDRmdzdsX1NnOVdieUp3M1VtY0RsaUVBPT0=|1451394250|8cf44cefb523b2973eca01f0918ef97fc03a4994"',
		
		));
    curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0');
    // 在HTTP请求头中"Referer: "的内容。
    curl_setopt($ch, CURLOPT_REFERER,"https://www.baidu.com/s?word=%E7%9F%A5%E4%B9%8E&tn=sitehao123&ie=utf-8&ssl_sample=normal&f=3&rsp=0");
    curl_setopt($ch, CURLOPT_ENCODING, "gzip, deflate, sdch");
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_TIMEOUT,120);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);//302redirect
    // 针对https的设置
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
    $html = curl_exec($ch);
    curl_close($ch);
    if($html === false) {
        echo 'Curl error: ' . curl_error($ch) . "<br>\n\r";
    } else {
		echo $html;
	}
</code>
Copy after login
我在暑假的时候爬过知乎,而且就是用的php+curl。知乎是有反爬虫机制的,你要尽量伪装成浏览器,包括header、useragent、cookie等等都设成浏览器上的一样,至于这些在哪可以看到请善用chrome的F12控制台。光伪装成浏览器是不够的,因为知乎有的页面是gzip加密的哦,所以你还要做好gzip解密的措施。如果知乎觉得你的行为可疑,知乎还会不定频率的给你返回空白页面,所以你还要做好数据验证的措施。总的来说爬知乎是不难的,但要稳定可靠的一口气爬完整个知乎还是很困难的。 curl配置增加cookie信息和header头部试试看,有些网站防采集需要尽可能的模拟。
Related labels:
source:php.cn
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Popular Tutorials
More>
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template