php file_get_contents抓取Gzip网页乱码的三种解决方法
Jun 20, 2016 pm 01:01 PM用 file_get_contents() 函数抓取网页会发生乱码现象。有两个原因会导致乱码,一个是编码问题,一个是目标页面开了Gzip,下面说的就是开了Gzip功能如何才能不乱码的方法
把抓取到的内容转下编码即可($content=iconv("GBK", "UTF-8//IGNORE", $content);),我们这里讨论的是如何抓取开了Gzip的页面。怎么判断呢?获取的头部当中有Content-Encoding: gzip说明内容是GZIP压缩的。用FireBug看一下就知道页面开了gzip没有。下面是用firebug查看我的博客的头信息,Gzip是开了的。
请求头信息原始头信息
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Encoding gzip, deflate Accept-Language zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3 Connection keep-alive Cookie __utma=225240837.787252530.1317310581.1335406161.1335411401.1537; __utmz=225240837.1326850415.887.3.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=%E4%BB%BB%E4%BD%95%E9%A1%B9%E7%9B%AE%E9%83%BD%E4%B8%8D%E4%BC%9A%E9%82%A3%E4%B9%88%E7%AE%80%E5%8D%95%20site%3Awww.nowamagic.net; PHPSESSID=888mj4425p8s0m7s0frre3ovc7; __utmc=225240837; __utmb=225240837.1.10.1335411401 Host www.nowamagic.net User-Agent Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20100101 Firefox/12.0
下面介绍一些解决方案:
1. 使用自带的zlib库
如果服务器已经装了zlib库,用下面的代码可以轻易解决乱码问题。
$data = file_get_contents("compress.zlib://".$url);
2. 使用CURL代替file_get_contents
function curl_get($url, $gzip=false){ $curl = curl_init($url); curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1); curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 10); if($gzip) curl_setopt($curl, CURLOPT_ENCODING, "gzip"); // 关键在这里 $content = curl_exec($curl); curl_close($curl); return $content; }
3. 使用gzip解压函数
function gzdecode($data) { $len = strlen($data); if ($len < 18 || strcmp(substr($data,0,2),"\x1f\x8b")) { return null; // Not GZIP format (See RFC 1952) } $method = ord(substr($data,2,1)); // Compression method $flags = ord(substr($data,3,1)); // Flags if ($flags & 31 != $flags) { // Reserved bits are set -- NOT ALLOWED by RFC 1952 return null; } // NOTE: $mtime may be negative (PHP integer limitations) $mtime = unpack("V", substr($data,4,4)); $mtime = $mtime[1]; $xfl = substr($data,8,1); $os = substr($data,8,1); $headerlen = 10; $extralen = 0; $extra = ""; if ($flags & 4) { // 2-byte length prefixed EXTRA data in header if ($len - $headerlen - 2 < 8) { return false; // Invalid format } $extralen = unpack("v",substr($data,8,2)); $extralen = $extralen[1]; if ($len - $headerlen - 2 - $extralen < 8) { return false; // Invalid format } $extra = substr($data,10,$extralen); $headerlen += 2 + $extralen; } $filenamelen = 0; $filename = ""; if ($flags & 8) { // C-style string file NAME data in header if ($len - $headerlen - 1 < 8) { return false; // Invalid format } $filenamelen = strpos(substr($data,8+$extralen),chr(0)); if ($filenamelen === false || $len - $headerlen - $filenamelen - 1 < 8) { return false; // Invalid format } $filename = substr($data,$headerlen,$filenamelen); $headerlen += $filenamelen + 1; } $commentlen = 0; $comment = ""; if ($flags & 16) { // C-style string COMMENT data in header if ($len - $headerlen - 1 < 8) { return false; // Invalid format } $commentlen = strpos(substr($data,8+$extralen+$filenamelen),chr(0)); if ($commentlen === false || $len - $headerlen - $commentlen - 1 < 8) { return false; // Invalid header format } $comment = substr($data,$headerlen,$commentlen); $headerlen += $commentlen + 1; } $headercrc = ""; if ($flags & 1) { // 2-bytes (lowest order) of CRC32 on header present if ($len - $headerlen - 2 < 8) { return false; // Invalid format } $calccrc = crc32(substr($data,0,$headerlen)) & 0xffff; $headercrc = unpack("v", substr($data,$headerlen,2)); $headercrc = $headercrc[1]; if ($headercrc != $calccrc) { return false; // Bad header CRC } $headerlen += 2; } // GZIP FOOTER - These be negative due to PHP's limitations $datacrc = unpack("V",substr($data,-8,4)); $datacrc = $datacrc[1]; $isize = unpack("V",substr($data,-4)); $isize = $isize[1]; // Perform the decompression: $bodylen = $len-$headerlen-8; if ($bodylen < 1) { // This should never happen - IMPLEMENTATION BUG! return null; } $body = substr($data,$headerlen,$bodylen); $data = ""; if ($bodylen > 0) { switch ($method) { case 8: // Currently the only supported compression method: $data = gzinflate($body); break; default: // Unknown compression method return false; } } else { // I'm not sure if zero-byte body content is allowed. // Allow it for now... Do nothing... } // Verifiy decompressed size and CRC32: // NOTE: This may fail with large data sizes depending on how // PHP's integer limitations affect strlen() since $isize // may be negative for large sizes. if ($isize != strlen($data) || crc32($data) != $datacrc) { // Bad format! Length or CRC doesn't match! return false; } return $data; }
使用:
$html=file_get_contents('http://www.jb51.net/'); $html=gzdecode($html);
就介绍这三个方法,应该能解决大部分gzip引起的抓取乱码问题了。

Artikel Panas

Alat panas Tag

Artikel Panas

Tag artikel panas

Notepad++7.3.1
Editor kod yang mudah digunakan dan percuma

SublimeText3 versi Cina
Versi Cina, sangat mudah digunakan

Hantar Studio 13.0.1
Persekitaran pembangunan bersepadu PHP yang berkuasa

Dreamweaver CS6
Alat pembangunan web visual

SublimeText3 versi Mac
Perisian penyuntingan kod peringkat Tuhan (SublimeText3)

Topik panas

Cara menggunakan PHP untuk membangunkan cache untuk meningkatkan pengalaman pengguna tapak web

Cara menggunakan Panel Pagoda untuk pengoptimuman laman web dan SEO

Kaedah pengoptimuman untuk meningkatkan prestasi dan kelajuan laman web

Konfigurasi algoritma pengimbangan beban Nginx untuk mengoptimumkan pengedaran perkhidmatan tapak web dengan cekap

Elemen utama untuk mengoptimumkan prestasi tapak web didedahkan: Bagaimana menggunakan analisis penunjuk untuk meningkatkan pengalaman pengguna?

Kajian mendalam tentang kepentingan kod status HTTP 301 dalam pengoptimuman tapak web

Optimumkan tapak web anda dan hapuskan ralat JavaScript

Kepentingan teknologi caching PHP dalam pengoptimuman laman web
