Suddenly I want to get some online data for fun, because there is SAE’s MySql database, and it is of no use leaving it there! So I started using PHP to write a small program that crawled the embarrassing things on the homepage of the Encyclopedia of Embarrassing Things. The data was all saved in MySql. Wouldn't it be fun!
Just do it! First determine the idea
Get HTML source code--->Parse HTML--->Save to database
Nothing difficult
1. Create the PHP file "getDataToDB.php",
2. Get the HTML source code of the specified URL
I am using the curl function here. For details, please refer to the PHP manual
The code is
<span new="" style="font-family:Times">// 获取对应链接的HTMLCODE function GetHtmlCode($url) { $ch = curl_init (); // 初始化一个cur对象 curl_setopt ( $ch, CURLOPT_URL, $url ); // 设置需要抓取的网页 curl_setopt ( $ch, CURLOPT_RETURNTRANSFER, 1 ); // 设置crul参数,要求结果保存到字符串中还是输出到屏幕上 curl_setopt ( $ch, CURLOPT_CONNECTTIMEOUT, 1000 ); // 设置链接延迟 $HtmlCode = curl_exec ( $ch ); // 运行curl,请求网页 return $HtmlCode; }</span>
I don’t have the ability to use regular expressions here, so I searched online and finally found this, just like using Jsoup in Java (using Jsoup to parse the official website of Chuzhou University to get the news list). For details, see BLOG
The code is as follows
<span new="" style="font-family:Times">function getFmlDataToDB() { $link = mysql_connect ( SAE_MYSQL_HOST_M . ':' . SAE_MYSQL_PORT, SAE_MYSQL_USER, SAE_MYSQL_PASS ); // 获取源码 $html = str_get_html ( GetHtmlCode ( http://www.qiushibaike.com/ ) ); if ($link) { mysql_select_db ( SAE_MYSQL_DB, $link ); mysql_query ( 'set names utf8' ); // class=article block untagged mb15 foreach ( $html->find ( 'div[class=article block untagged mb15]' ) as $per ) { $z = null; $t = null; $w = null; $d = null; $p = null; $ds = null; $ps = null; // //作者 $author = $per->find ( 'div[class=author]' ); if ($author != null) { $a = $author [0]->find ( 'a' ); $z = $a [1]->innertext; } else { $z = 'no author'; } // 头像链接 if ($author != null) { $icon = $author [0]->find ( 'a' ); $t = $icon [0]->src->innertext; } else { $t = '...............'; } // 文章内容 $content = $per->find ( 'div[class=content]' ); $w = $content [0]->innertext; // 点赞数 $vote1 = $per->find ( 'div[class=stats]' ); $vote2 = $vote1 [0]->find ( 'span[class=stats-vote]' ); $vote3 = $vote2 [0]->find ( 'i[class=number]' ); $d = $vote3 [0]->innertext; // 评论数 $comments1 = $vote1 [0]->find ( 'span[class=stats-comments]' ); $comments2 = $comments1 [0]->find ( 'a[class=qiushi_comments]' ); $comments3 = $comments2 [0]->find ( 'i[class=number]' ); $p = $comments3 [0]->innertext; // 顶 数 $up_down = $per->find ( 'div[class=stats-buttons bar clearfix]' ); $up_down1 = $up_down [0]->find ( 'ul' ); $li = $up_down1 [0]->find ( 'li' ); $up = $li [0]->find ( 'span[class=number hidden]' ); $ds = $up [0]->innertext; // 拍 数 $down = $li [1]->find ( 'span[class=number hidden]' ); $ps = $down [0]->innertext; } } else { echo '数据库链接KO'; } }</span>
4. Create a database and insert data into the database
Here I use MySQL in SAE. For specific connection methods, see Using PHP to connect to the MySql database in SAE
What you need to pay attention to is the encoding format. You should add this sentence before the execution statement
<span style="font-family:Microsoft">mysql_query ( 'set names utf8' );</span>
<span style="font-family:Microsoft"> $sql = INSERT INTO `app_bmhjqs`.`db_fml` (`id`, `author`, `icon_url`, `content`, `vote`, `comments`, `up`, `down`) VALUES (NULL, '$z', '$t', '$w', '$d', '$p', '$ds', '$ps');; // 解决乱码 mysql_query ( 'set names utf8' ); $result = mysql_query ( $sql );</span>
<span new="" style="font-family:Times">// 定时器 // ignore_user_abort (); // run script. in background // set_time_limit ( 0 ); // run script. forever // $interval = 30; // do every 15 minutes.. // do { // echo date ( 'Y-m-d H:i:s', time () ); // echo '写入数据库'; // //getFmlDataToDB (); // } while ( true );</span>
This morning, I couldn’t wait to turn on my computer and open the SAE database. The situation is as follows:
Oh my god! I couldn't stand it anymore, so I quickly turned off the timer and wrote a button to trigger the event! If this continues, the database will be crowded!