最近在做校园图书馆图书信息的采集程序，既然是图书馆图书的采集，肯定有提交搜索的页面，无非是post提交，让我想到了curl模拟提交，首先通过firebug进行抓包查询下post提交后的格式如下：
txtWxlx=CN&hidWxlx=spanCNLx&txtPY=HZ&txtTm=%D2%F4%C0%D6&txtLx=%25&txtSearchType=1&nMaxCount=100&nSetPageSize=10&cSortFld=%D5%FD%CC%E2%C3%FB&B1=%BC%EC+%CB%F7；搜索的关键字name=txtTm，随后代码如下：<br>
<?<span>php
</span><span>$keyword</span>="音乐"<span>;</span><span>$post</span>="txtWxlx=CN&hidWxlx=spanCNLx&txtPY=HZ&txtTm={<span>$keyword</span>}&txtLx=%25&txtSearchType=1&nMaxCount=100&nSetPageSize=20&cSortFld=%D5%FD%CC%E2%C3%FB&B1=%BC%EC+%CB%F7"<span>;
</span><span>$url</span> = "http://210.30.68.130/wxjs/tmjs.asp"<span>;
</span><span>//</span><span>查询地址</span>
<span>$ch</span> =<span> curl_init();
curl_setopt (</span><span>$ch</span>, CURLOPT_REFERER, "http://210.30.68.130/wxjs/tmjs_form.asp/ "); <span>//</span><span>模拟来源 </span>
curl_setopt(<span>$ch</span>, CURLOPT_URL, <span>$url</span>);<span>//</span><span>URL </span>
curl_setopt(<span>$ch</span>,CURLOPT_FOLLOWLOCATION,1<span>);
curl_setopt(</span><span>$ch</span>, CURLOPT_HEADER, <span>false</span><span>);
curl_setopt(</span><span>$ch</span>, CURLOPT_POST, 1); <span>//</span><span>模拟POST </span>
curl_setopt(<span>$ch</span>, CURLOPT_POSTFIELDS, <span>$post</span>);<span>//</span><span>POST内容 </span>
curl_exec(<span>$ch</span><span>);
</span><span>$output</span> = curl_close(<span>$ch</span><span>);
</span><span>echo</span> <span>$output</span><span>;
</span>?>
但返回的页面总是显示没有相关内容，如果把关键字改成英语或者数字就可以正常显示了，于是想到这应该是编码问题，然后在post上面可以看到txtTm=%D2%F4%C0%D6经过查询得知这是url编码，汉字会进行转化，如果是英语则不会发生任何改变，于是就把头信息也加了部分内容进去如下：
<span>$header</span> = <span>array</span><span>(); </span><span>$header</span>[] = 'User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:30.0) Gecko/20100101 Firefox/30.0'<span>; </span><span>$header</span>[] = 'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'<span>; </span><span>$header</span>[] = 'Connection: keep-alive'<span>; </span><span>$header</span>[] = 'Content-Type:application/x-www-form-urlencoded'<span>; curl_setopt ( </span><span>$ch</span>, CURLOPT_HTTPHEADER, <span>$header</span> );
其实主要是$header[] = 'Content-Type:application/x-www-form-urlencoded';再次加载页面后如果是汉字显示还是没有相关的内容，随后想到了一个很简单的问题所在，php程序是utf8的，然而图书馆的网站是gb2312，好吧，再加上一句，$keyword = iconv('UTF-8', 'GB2312', $keyword);再次加载成功，这个应该才是问题关键所在，然后我把头信息删除后添加一句$keyword = urlencode($keyword);再次加载也就是下面的代码：
<?<span>php
</span><span>$keyword</span>="世界"<span>;
</span><span>$keyword</span> = <span>iconv</span>('UTF-8', 'GB2312', <span>$keyword</span><span>);
</span><span>$keyword</span> = <span>urlencode</span>(<span>$keyword</span><span>);
</span><span>$post</span>="txtWxlx=CN&hidWxlx=spanCNLx&txtPY=HZ&txtTm={<span>$keyword</span>}&txtLx=%25&txtSearchType=1&nMaxCount=100&nSetPageSize=20&cSortFld=%D5%FD%CC%E2%C3%FB&B1=%BC%EC+%CB%F7"<span>;
</span><span>$url</span> = "http://210.30.68.130/wxjs/tmjs.asp"<span>;
</span><span>//</span><span>查询地址</span>
<span>$ch</span> =<span> curl_init();
curl_setopt (</span><span>$ch</span>, CURLOPT_REFERER, "http://210.30.68.130/wxjs/tmjs_form.asp/ "); <span>//</span><span>模拟来源 </span>
curl_setopt(<span>$ch</span>, CURLOPT_URL, <span>$url</span>);<span>//</span><span>URL </span>
curl_setopt(<span>$ch</span>,CURLOPT_FOLLOWLOCATION,1<span>);
curl_setopt(</span><span>$ch</span>, CURLOPT_HEADER, <span>false</span><span>);
curl_setopt(</span><span>$ch</span>, CURLOPT_POST, 1); <span>//</span><span>模拟POST </span>
curl_setopt(<span>$ch</span>, CURLOPT_POSTFIELDS, <span>$post</span>);<span>//</span><span>POST内容 </span>
curl_exec(<span>$ch</span><span>);
</span><span>$output</span> = curl_close(<span>$ch</span><span>);
</span><span>echo</span> <span>$output</span><span>;
</span>?>不出所料，正常显示了，随后就是页面内容的采集排版问题了。无非是正则了。（由于是校园网只能我们内网访问）
