snoopy (powerful PHP collection class) detailed introduction

Snoopy is a php class that is used to simulate the functions of a browser. It can obtain web content, send forms, and can be used to develop some collection programs and thief programs. This article introduces the usage tutorial of snoopy in detail. Some features of Snoopy:

Fetch the content of the web page fetch

Grab the text content of the web page (remove HTML tags) fetchtext

Fetch the links of the web page, form fetchlinks fetchform

Support proxy host

Support basic username/password authentication

Support setting user_agent, referer (source), cookies and header content (header file)

Support browser redirection and control Redirect depth

can expand links in web pages into high-quality URLs (default)

Submit data and get return values ​​

Support tracking HTML framework

Support delivery during redirection Cookies

only requires PHP 4 or above. Since it is a PHP class, there is no need to expand the support. The best choice when the server does not support curl,
Snoopy class methods and examples:
fetch( $URI)This is the method used to crawl the content of the web page. The $URI parameter is the URL address of the crawled web page. The crawled results are stored in $this->results. If you are scraping a frame, Snoopy will track each frame and store it in an array, and then store it in $this->results.
fetchtext($URI)This method is similar to fetch(). The only difference is that this method will remove HTML tags and other irrelevant data and only return the text content in the web page.
fetchform($URI)This method is similar to fetch(). The only difference is that this method will remove HTML tags and other irrelevant data, and only return the form content (form) in the web page.
fetchlinks($URI)This method is similar to fetch(). The only difference is that this method will remove HTML tags and other irrelevant data, and only return links in the web page. By default, relative links will be automatically completed and converted into complete URLs.
submit($URI,$formvars)This method sends a confirmation form to the link address specified by $URL. $formvars is an array that stores form parameters.
submittext($URI,$formvars)This method is similar to submit(). The only difference is that this method will remove HTML tags and other irrelevant data, and only return the text content in the web page after login.
submitlinks($URI)This method is similar to submit(). The only difference is that this method will remove HTML tags and other irrelevant data, and only return links in the web page. By default, relative links will be automatically completed and converted into complete URLs. Snoopy collection class attributes: (Default value is in brackets)
$host The connected host $port The connected port $proxy_host The proxy host used, if any $proxy_port The proxy host port used, if any $agent User agent camouflage (Snoopy v0.1) $referer Source information, if any $cookies cookies, if any $ rawheaders Other header information, if any $maxredirs Maximum number of redirects, 0=not allowed (5)$offsiteok whether or not to allow redirects off-site. (true)$expandlinks Whether to link All are completed to the complete address (true)$user authentication user name, if any $pass authentication user name, if any $accept http acceptance type (image/gif, image/x-xbitmap , image/jpeg, image/pjpeg, */*)$error Where the error is reported, if any $response_code The response code returned from the server $headers The header information returned from the server $maxlength The longest Return data length $read_timeout Read operation timeout (requires PHP 4 Beta 4+) Set to 0 for no timeout $timed_out If a read operation times out, this attribute returns true (requires PHP 4 Beta 4+ )$maxframes The maximum number of frames allowed to be tracked$status The status of the captured http$temp_dir The temporary file directory that the web server can write to (/tmp)$curl_path The directory of cURL binary, if there is no cURL Set binary to false
The following is an example:

include "Snoopy.class.php";
 $snoopy = new Snoopy;
  
 $snoopy->proxy_host = "http://www.9it.me";
 $snoopy->proxy_port = "80";
  
 $snoopy->agent = "(compatible; MSIE 4.01; MSN 2.5; AOL 4.0; Windows 98)";
 $snoopy->referer = "http://www.9it.me";
  
 $snoopy->cookies["SessionID"] = 238472834723489l;
 $snoopy->cookies["favoriteColor"] = "RED";
  
 $snoopy->rawheaders["Pragma"] = "no-cache";
  
 $snoopy->maxredirs = 2;
 $snoopy->offsiteok = false;
 $snoopy->expandlinks = false;
  
 $snoopy->user = "joe";
 $snoopy->pass = "bloe";
  
 if($snoopy->fetchtext("http://www.9it.me"))
 {
 echo "<PRE>".htmlspecialchars($snoopy->results)."

\n"; } else echo "error fetching document: ".$snoopy->error."\n";

Get the content of the specified url

<?php
 $url = "http://www.9it.me";
 include("snoopy.php");
 $snoopy = new Snoopy;
 $snoopy->fetch($url); //获取所有内容
 echo $snoopy->results; //显示结果
 //可选以下
 $snoopy->fetchtext //获取文本内容（去掉html代码）
 $snoopy->fetchlinks //获取链接
 $snoopy->fetchform  //获取表单
 ?>

表单提交

<?php
$formvars["username"] = "admin";
$formvars["pwd"] = "admin";
$action = "http://www.9it.me";//</a>表单提交地址
$snoopy->submit($action,$formvars);//$formvars为提交的数组
echo $snoopy->results; //获取表单提交后的 返回的结果
//可选以下
$snoopy->submittext; //提交后只返回 去除html的 文本
$snoopy->submitlinks;//提交后只返回 链接
?>

既然已经提交的表单 那就可以做很多事情 接下来我们来伪装ip,伪装浏览器

伪装浏览器

<?php
$formvars["username"] = "lanfengye";
$formvars["pwd"] = "lanfengye";
$action = "http://www.9it.me";
include "snoopy.php";
$snoopy = new Snoopy;
$snoopy->cookies["PHPSESSID"] = 'fc106b1918bd522cc863f36890e6fff7'; //伪装sessionid
$snoopy->agent = "(compatible; MSIE 4.01; MSN 2.5; AOL 4.0; Windows 98)"; //伪装浏览器
$snoopy->referer = "http://www.9it.me"; //伪装来源页地址 http_referer
$snoopy->rawheaders["Pragma"] = "no-cache"; //cache 的http头信息
$snoopy->rawheaders["X_FORWARDED_FOR"] = "127.0.0.101"; //伪装ip
$snoopy->submit($action,$formvars);
echo $snoopy->results;
?>

原来我们可以伪装session 伪装浏览器 ，伪装ip， haha 可以做很多事情了。例如 带验证码，验证ip 投票， 可以不停的投。ps:这里伪装ip ，其实是伪装http头, 所以一般的通过 REMOTE_ADDR 获取的ip是伪装不了，反而那些通过http头来获取ip的(可以防止代理的那种) 就可以自己来制造ip。关于如何验证码 ，简单说下：首先用普通的浏览器， 查看页面 ， 找到验证码所对应的sessionid，同时记下sessionid和验证码值，接下来就用snoopy去伪造 。原理:由于是同一个sessionid 所以取得的验证码和第一次输入的是一样的。
有时我们可能需要伪造更多的东西,snoopy完全为我们想到了

<?php
$snoopy->proxy_host = "http://www.9it.me";
$snoopy->proxy_port = "8080"; //使用代理
$snoopy->maxredirs = 2; //重定向次数
$snoopy->expandlinks = true; //是否补全链接 在采集的时候经常用到
// 例如链接为 /images/taoav.gif 可改为它的全链接 <a href="http://www.9it.me/images/taoav.gif">http://www.9it.me/images/taoav.gif</a>
$snoopy->maxframes = 5 //允许的最大框架数
//注意抓取框架的时候 $snoopy->results 返回的是一个数组
$snoopy->error //返回报错信息
?>

以上就介绍了snoopy（强大的PHP采集类） 详细介绍，包括了方面的内容，希望对PHP教程有兴趣的朋友有所帮助。