最快的速度获取网页全部图片的长和宽

WBOY
Release: 2016-06-13 12:08:55
Original
1074 people have browsed it

最快的速度获取网页所有图片的长和宽。
不知道大家有没有玩过 http://pinterest.com ?注册后,它有一个 add a pin, 当你提交一个网站的URL后,按Find Images时,它可以查找你提交网页上所有图片的(并进行长和宽条件的筛选),整个过程一般在10秒左右。

最近想模仿它,做一个小功能组件。已经摒弃掉万恶的 getimagesize() (需要48.64秒),换用 imagecreatefromstring()(还是需要26.13秒),和它10秒左右的成绩,简直是天壤之别。

要考虑 TCP 连接数,要做到服务器资源最省化,还要考虑执行时间最少化。求助万能的大虾们,如何继续优化代码?可以跑的更快些。

<br /><br />function ranger($url){<br />	$headers = array( "Range: bytes=0-32768" );<br />	$curl = curl_init($url);<br />	curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);<br />	curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);<br />	return curl_exec($curl);<br />	curl_close($curl);<br />}//curl设置<br /><br />require dirname(__FILE__) . '/simple_html_dom.php'; <br />//采用simple_html_dom.php分析HTML nod<br /><br />$url = 'http://www.huffingtonpost.com/';<br /><br />$html = file_get_html($url);<br />if($html->find('img')){<br />	foreach($html->find('img') as $element) {<br />		$raw = ranger($element->src);<br />		$im = @imagecreatefromstring($raw);<br />		$width = @imagesx($im);<br />		$height = @imagesy($im);<br />		if($width>=200||$height>=200){<br />			echo $element;//得出长大于大于200,宽大于等于200的图片<br />		}<br />	}<br />}<br /><br />
Copy after login


------解决思路----------------------
也许能走个弯路,减轻服务器网络压力。
服务器负责解析HTML数据,统计image标签信息,最后将收集的文本数据送回客户端。
加载图片由客户端来完成,只需读取width,height属性,就完全可以获取图片的原始大小。
好处多多,不过可能的麻烦是防盗链
------解决思路----------------------
顶楼上
PHP获取资源
javascript 取图片长和宽
------解决思路----------------------
读取并解析 2.8秒
读取图片(138个) 27秒
找到 7 个

仅从优化代码出发,应该油水不大
可考虑多路并发
------解决思路----------------------
读取并解析 3.6秒
启动读取图片进程(138个) 1.3秒
结果文件中记录数 7 个
http://s.huffpost.com/images/v/logos/v4/tagline.gif<br />http://s.huffpost.com/images/v/logos/v4/homepage.gif?v9<br />http://i.huffpost.com/gen/559399/thumbs/r-OLBERMANN-huge.jpg<br />http://s.huffpost.com/images/facebook_promo_connect.png?3<br />http://images.huffingtonpost.com/2012-04-04-michaeljfoxmarlo2SECOND.jpg<br />http://images.huffingtonpost.com/2012-04-05-Screenshot20120405at9.40.24AM.jpg<br />http://i.huffpost.com/gen/557914/thumbs/s-SCORSESE-large300.jpg<br />
Copy after login


原循环改为
    foreach($html->find('img') as $element) {<br />       tenor("tenorcall.php?v=$element->src");<br />    }<br />}<br />
Copy after login


tenorcall.php
function ranger($url){<br />    $headers = array( "Range: bytes=0-32768" );<br />    $curl = curl_init($url);<br />    curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);<br />    curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);<br />    return curl_exec($curl);<br />    curl_close($curl);<br />}//curl设置<br /><br />        $raw = ranger($_GET['v']);<br />        $im = @imagecreatefromstring($raw);<br />        $width = @imagesx($im);<br />        $height = @imagesy($im);<br />        if($width>=200<br><font color='#FF8000'>------解决思路----------------------</font><br>$height>=200){<br />            file_put_contents('tenorcall.txt', $_GET['v'].PHP_EOL, FILE_APPEND );//得出长大于大于200,宽大于等于200的图片<br />        }<br />
Copy after login


/**<br /> * 函数 tenor<br /> * 功能 启动一个url,但不等待返回<br /> * 参数 $page,待执行的页面程序<br /> * 返回 无<br /> **/<br />if(! function_exists('tenor')):<br />function tenor($page) {<br />        $host = $_SERVER["HTTP_HOST"];<br />        $fp = fsockopen($host, 80, $errno, $errmsg);<br />        if(!$fp) {<br />                echo "$errstr ($errno)<br>\n";<br />        } else {<br />                fputs($fp,"GET /$page HTTP/1.0\nHost: $host\n\n");<br />                fclose($fp);<br />        }<br />}<br />endif;<br />
Copy after login


代码还是原代码,非但没减少,反而增加了
但因为是并发,所以速度明显提高

值得注意的是:tenor 函数在某些web服务器中不能稳定的运行(比如iis6)原因不明
------解决思路----------------------
我觉得,让客户端加载的方案是可行的,

客户端再将符合要求的图片信息提交给服务器,服务器端再验证一次后保存。。。


另外32768是怎么得来的?1-200不够吗
------解决思路----------------------
学习! 是用PHP获取图片url后直接读取图片的头信息吗?
------解决思路----------------------
pinterest那个pin功能创意很好,而且技术很简单,就是书签一串js代码,然后你点这个书签就相当于往当前页面文档append入一个js文件,这个js文件怎么写,就很简单了,主要就是遍历document.getElementsByTagName('img')
------解决思路----------------------
本帖最后由 xuzuning 于 2012-04-06 15:25:06 编辑

138个照片并发,是不是就消耗了138个连接数


是否需要修改php.ini,增加连接数
否,连接是向外的,如果要改,也是对方改

CPU和内存开销如何
这个不太好测试

,关于使用 js 判断的问题,由于他们没有给出代码,无法测试
自己写了两个方案都不理想,也就作罢了

用JS并发和直接PHP并发,2者从资源消耗角度来比,哪个会更少
资源消耗角度来比 都一样,都要完整的加载图片
不过前者是消耗客户端资源,后者是消耗服务器端资源
另外浏览器的机制不很了解,是否真的是并发也未可知
------解决思路----------------------
这段代码在我这里大约 1.8秒,不计算 file_get_html ( $url ) 时间

$res [] = $url ;//$temp;
这样就是网络地址了

他是保存为本地文件后用 getimagesize 获取尺寸的

他应该是通过 curl 并发的,这个机制我不太了解
------解决思路----------------------
但是 if(in_array($absUrl, $visited))continue; 这行报错。 Warning: in_array() expects parameter 2 to be array, null。

他的代码中并没有你说的出错的代码
应该是 file_get_html 在报错吧
file_get_html 使用 file_get_contents 读取 url 成功率较低
经常要刷两三次才可独到数据
------解决思路----------------------
JS可以通过获取图片的头部信息,而直接获取到图片的高度,
这种方式比用图片加载完成以后在获取他的搞定效率至少快10倍以上,
之前记得有在一个播客里面看到过这么个帖子来着,
没收藏,这一时半会的找不到了,郁闷啊~

------解决思路----------------------
刚注册了http://pinterest.com。 它的做法就是用客户端来加载
点击Add 选择Pin ,贴上网址 http://www.huffingtonpost.com/
在chrome的Network中可以看到有一个请求
GET /pin/create/find_images/?url=http%253A%2F%2Fwww.huffingtonpost.com HTTP/1.1
返回的信息是一个json对象:
images: [http://s.huffpost.com/images/v/logos/v4/homepage.gif?v9,…]<br />0: "http://s.huffpost.com/images/v/logos/v4/homepage.gif?v9"<br />1: "http://s.huffpost.com/images/v/logos/v4/tagline.gif"<br />2: "http://s.huffpost.com/images/splash/t_mini-a.png"<br />3: "http://s.huffpost.com/images/splash/t_mini-a.png"<br />4: "http://s.huffpost.com/images/splash/t_mini-a.png"<br />5: "http://s.huffpost.com/images/splash/t_mini-a.png"<br />6: "http://s.huffpost.com/images/splash/t_mini-a.png"<br />7: "http://s.huffpost.com/images/splash/t_mini-a.png"<br />8: "http://s.huffpost.com/images/splash/t_mini-a.png"<br />9: "http://s.huffpost.com/images/splash/t_mini-a.png"<br />10: "http://s.huffpost.com/images/splash/t_mini-a.png"<br />11: "http://s.huffpost.com/images/splash/t_mini-a.png"<br />12: "http://s.huffpost.com/images/splash/t_mini-a.png"<br />13: "http://s.huffpost.com/images/splash/t_mini-a.png"<br />14: "http://s.huffpost.com/images/splash/t_mini-a.png"<br />15: "http://s.huffpost.com/images/splash/t_mini-a.png"<br />16: "http://s.huffpost.com/images/splash/t_mini-a.png"<br />17: "http://i.huffpost.com/gen/560770/thumbs/r-GSA-LAS-VEGAS-VIDEO-huge.jpg"<br />18: "http://s.huffpost.com/images/webslice12x12.png"<br />19: "http://s.huffpost.com/images/v/blog_column.png"<br />20: "http://s.huffpost.com/contributors/gary-hart/headshot.jpg"<br />21: "http://www.huffingtonpost.com/images/trans.gif"<br />22: "http://www.huffingtonpost.com/images/trans.gif"<br />23: "http://www.huffingtonpost.com/images/trans.gif"<br />24: "http://images.huffingtonpost.com/2012-04-06-campbellguitar.jpg"<br />25: "http://www.huffingtonpost.com/images/trans.gif"<br />26: "http://www.huffingtonpost.com/images/trans.gif"<br />27: "http://www.huffingtonpost.com/images/trans.gif"<br />28: "http://www.huffingtonpost.com/images/trans.gif"<br />29: "http://www.huffingtonpost.com/images/trans.gif"<br />30: "http://www.huffingtonpost.com/images/trans.gif"<br />31: "http://images.huffingtonpost.com/2012-04-06-Screenshot20120406at7.09.17PM.jpg"<br />32: "http://www.huffingtonpost.com/images/trans.gif"<br />33: "http://www.huffingtonpost.com/images/trans.gif"<br />34: "http://www.huffingtonpost.com/images/trans.gif"<br />35: "http://www.huffingtonpost.com/images/trans.gif"<br />36: "http://www.huffingtonpost.com/images/trans.gif"<br />37: "http://www.huffingtonpost.com/images/trans.gif"<br />38: "http://www.huffingtonpost.com/images/trans.gif"<br />39: "http://www.huffingtonpost.com/images/trans.gif"<br />40: "http://www.huffingtonpost.com/images/trans.gif"<br />41: "http://www.huffingtonpost.com/images/trans.gif"<br />42: "http://www.huffingtonpost.com/images/trans.gif"<br />43: "http://www.huffingtonpost.com/images/trans.gif"<br />44: "http://www.huffingtonpost.com/images/trans.gif"<br />45: "http://www.huffingtonpost.com/images/trans.gif"<br />46: "http://www.huffingtonpost.com/images/trans.gif"<br />47: "http://www.huffingtonpost.com/images/trans.gif"<br />48: "http://www.huffingtonpost.com/images/trans.gif"<br />49: "http://www.huffingtonpost.com/images/trans.gif"<br />50: "http://www.huffingtonpost.com/images/trans.gif"<br />51: "http://www.huffingtonpost.com/images/trans.gif"<br />52: "http://www.huffingtonpost.com/images/trans.gif"<br />53: "http://www.huffingtonpost.com/images/trans.gif"<br />54: "http://www.huffingtonpost.com/images/trans.gif"<br />55: "http://www.huffingtonpost.com/images/trans.gif"<br />56: "http://www.huffingtonpost.com/images/trans.gif"<br />57: "http://www.huffingtonpost.com/images/trans.gif"<br />58: "http://www.huffingtonpost.com/images/trans.gif"<br />59: "http://www.huffingtonpost.com/images/trans.gif"<br />60: "http://www.huffingtonpost.com/images/trans.gif"<br />61: "http://www.huffingtonpost.com/images/trans.gif"<br />62: "http://www.huffingtonpost.com/images/trans.gif"<br />63: "http://www.huffingtonpost.com/images/trans.gif"<br />64: "http://www.huffingtonpost.com/images/trans.gif"<br />65: "http://www.huffingtonpost.com/images/trans.gif"<br />66: "http://www.huffingtonpost.com/images/trans.gif"<br />67: "http://www.huffingtonpost.com/images/trans.gif"<br />68: "http://www.huffingtonpost.com/images/trans.gif"<br />69: "http://www.huffingtonpost.com/images/trans.gif"<br />70: "http://www.huffingtonpost.com/images/trans.gif"<br />71: "http://www.huffingtonpost.com/images/trans.gif"<br />72: "http://www.huffingtonpost.com/images/trans.gif"<br />73: "http://www.huffingtonpost.com/images/trans.gif"<br />74: "http://www.huffingtonpost.com/images/trans.gif"<br />75: "http://s.huffpost.com/images/blank.gif"<br />76: "http://s.huffpost.com/images/blank.gif"<br />77: "http://s.huffpost.com/images/blank.gif"<br />78: "http://s.huffpost.com/images/blank.gif"<br />79: "http://s.huffpost.com/images/blank.gif"<br />80: "http://s.huffpost.com/images/blank.gif"<br />81: "http://s.huffpost.com/images/blank.gif"<br />82: "http://s.huffpost.com/images/facebook_promo_connect.png?3"<br />83: "http://s.huffpost.com/images/loader.gif"<br />84: "http://www.huffingtonpost.com/images/trans.gif"<br />85: "http://www.huffingtonpost.com/images/trans.gif"<br />86: "http://www.huffingtonpost.com/images/trans.gif"<br />87: "http://www.huffingtonpost.com/images/trans.gif"<br />88: "http://www.huffingtonpost.com/images/trans.gif"<br />89: "http://www.huffingtonpost.com/images/trans.gif"<br />90: "http://s.huffpost.com/contributors/gary-hart/headshot.jpg"<br />91: "http://s.huffpost.com/contributors/mike-campbell/headshot.jpg"<br />92: "http://s.huffpost.com/contributors/roma-downey/headshot.jpg"<br />93: "http://s.huffpost.com/contributors/gavin-newsom/headshot.jpg"<br />94: "http://s.huffpost.com/contributors/sarah-shourd/headshot.jpg"<br />95: "http://s.huffpost.com/contributors/jacqueline-novogratz/headshot.jpg"<br />96: "http://s.huffpost.com/contributors/peggy-drexler/headshot.jpg"<br />97: "http://s.huffpost.com/contributors/mohamed-a-elerian/headshot.jpg"<br />98: "http://s.huffpost.com/contributors/bill-mckibben/headshot.jpg"<br />99: "http://s.huffpost.com/contributors/marlo-thomas/headshot.jpg"<br />100: "http://www.huffingtonpost.com/images/v/something_to_say_button.png"<br />101: "http://www.huffingtonpost.com/images/trans.gif"<br />102: "http://www.huffingtonpost.com/images/trans.gif"<br />103: "http://www.huffingtonpost.com/images/trans.gif"<br />104: "http://www.huffingtonpost.com/images/trans.gif"<br />105: "http://www.huffingtonpost.com/images/trans.gif"<br />106: "http://www.huffingtonpost.com/images/trans.gif"<br />107: "http://www.huffingtonpost.com/images/trans.gif"<br />108: "http://www.huffingtonpost.com/images/trans.gif"<br />109: "http://www.huffingtonpost.com/images/trans.gif"<br />110: "http://www.huffingtonpost.com/images/trans.gif"<br />111: "http://www.huffingtonpost.com/images/trans.gif"<br />112: "http://www.huffingtonpost.com/images/trans.gif"<br />113: "http://www.huffingtonpost.com/images/trans.gif"<br />114: "http://www.huffingtonpost.com/images/trans.gif"<br />115: "http://www.huffingtonpost.com/images/trans.gif"<br />116: "http://www.huffingtonpost.com/images/trans.gif"<br />117: "http://www.huffingtonpost.com/images/trans.gif"<br />118: "http://www.huffingtonpost.com/images/trans.gif"<br />119: "http://www.huffingtonpost.com/images/trans.gif"<br />120: "http://www.huffingtonpost.com/images/trans.gif"<br />121: "http://www.huffingtonpost.com/images/trans.gif"<br />122: "http://www.huffingtonpost.com/images/trans.gif"<br />123: "http://www.huffingtonpost.com/images/trans.gif"<br />124: "http://www.huffingtonpost.com/images/trans.gif"<br />125: "http://www.huffingtonpost.com/images/trans.gif"<br />126: "http://www.huffingtonpost.com/images/trans.gif"<br />127: "http://www.huffingtonpost.com/images/trans.gif"<br />128: "http://www.huffingtonpost.com/images/trans.gif"<br />129: "http://www.huffingtonpost.com/images/trans.gif"<br />130: "http://www.huffingtonpost.com/images/trans.gif"<br />131: "http://www.huffingtonpost.com/images/trans.gif"<br />132: "http://www.huffingtonpost.com/images/trans.gif"<br />133: "http://www.huffingtonpost.com/images/trans.gif"<br />134: "http://b.scorecardresearch.com/p?c1=2&c2=6723616&c3=&c4=&c5=front&c6=&c15=&cj=1"<br />135: "http://www.huffingtonpost.com//secure-us.imrworldwide.com/cgi-bin/m?ci=us-703240h&cg=0&cc=1&ts=noscript"<br />136: "http://vertical-stats.huffpost.com/?-1&&"<br />137: "http://www.huffingtonpost.com//pixel.quantserve.com/pixel/p-6fTutip1SMLM2.gif?labels=Home"<br />images_count: 138<br />redirected: false<br />status: "success"<br />title: "Breaking News and Opinion on The Huffington Post"<br />type: "text/html; charset=utf-8"
Copy after login


几乎是服务器返回的同时,浏览器开始加载图片。chrome监控如下。黄色的那个线表示提交url获取图片资源,后面的就都是加载图片了,加载的速度还是取决于我这儿的网络。

由于http://pinterest.com/的JS代码经过压缩,且使用了JQuery,所以找起来特别费劲。其实具体怎么干就很简单,谁都能想到。遍历json数据,创建img标签对象,设置src属性,保存对象。剩下的浏览器就会自己完成。
------解决思路----------------------

引用:
引用:

刚注册了http://pinterest.com。 它的做法就是用客户端来加载
点击Add 选择Pin ,贴上网址 http://www.huffingtonpost.com/
在chrome的Network中可以看到有一个请求
GET /pin/create/find_images/?url=http%253A%2F%2Fwww.huffingtonpo……

什么对象? 
你是说服务器返回的image链接的数据吗?不用保存呀。收到ajax请求后解析返回数据就完了
另外,浏览器加载外部资源都是异步。也就是说,不管是不是用的JQuery,都是异步加载的,相互不会影响。和老大写的php端的差不多。
Related labels:
source:php.cn
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Popular Tutorials
More>
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template
About us Disclaimer Sitemap
php.cn:Public welfare online PHP training,Help PHP learners grow quickly!