Python code example for grabbing NetEase news-Python Tutorial-php.cn

Python code example for grabbing NetEase news

Y2J

Release： 2017-05-03 16:03:42

Original

2108 people have browsed it

这篇文章主要介绍了Python正则抓取网易新闻的方法,结合实例形式较为详细的分析了Python使用正则进行网易新闻抓取操作的相关实现技巧与注意事项,需要的朋友可以参考下

本文实例讲述了Python正则抓取网易新闻的方法。分享给大家供大家参考，具体如下：

自己写了些关于抓取网易新闻的爬虫，发现其网页源代码与网页的评论根本就对不上，所以，采用了抓包工具得到了其评论的隐藏地址（每个浏览器都有自己的抓包工具，都可以用来分析网站）

如果仔细观察的话就会发现，有一个特殊的，那么这个就是自己想要的了

然后打开链接就可以找到相关的评论内容了。（下图为第一页内容）

接下来就是代码了(也照着大神的改改写写了)。

#coding=utf-8
import urllib2
import re
import json
import time
class WY():
  def __init__(self):
    self.headers = {&#39;User-Agent&#39;: &#39;Mozilla/5.0 (Windows NT 5.1) AppleWebKit/534.24 (KHTML, like &#39;}
    self.url=&#39;http://comment.news.163.com/data/news3_bbs/df/B9IBDHEH000146BE_1.html&#39;
  def getpage(self,page):
    full_url=&#39;http://comment.news.163.com/cache/newlist/news3_bbs/B9IBDHEH000146BE_&#39;+str(page)+&#39;.html&#39;
    return full_url
  def gethtml(self,page):
    try:
      req=urllib2.Request(page,None,self.headers)
      response = urllib2.urlopen(req)
      html = response.read()
      return html
    except urllib2.URLError,e:
      if hasattr(e,&#39;reason&#39;):
        print u"连接失败",e.reason
        return None
  #处理字符串
  def Process(self,data,page):
    if page == 1:
      data=data.replace(&#39;var replyData=&#39;,&#39;&#39;)
    else:
      data=data.replace(&#39;var newPostList=&#39;,&#39;&#39;)
    reg1=re.compile(" \[<a href=&#39;&#39;>")
    data=reg1.sub(&#39; &#39;,data)
    reg2=re.compile(&#39;<\\\/a>\]&#39;)
    data=reg2.sub(&#39;&#39;,data)
    reg3=re.compile(&#39;<br>&#39;)
    data=reg3.sub(&#39;&#39;,data)
    return data
  #解析json
  def dealJSON(self):
    with open("WY.txt","a") as file:
      file.write(&#39;ID&#39;+&#39;|&#39;+&#39;评论&#39;+&#39;|&#39;+&#39;踩&#39;+&#39;|&#39;+&#39;顶&#39;+&#39;\n&#39;)
    for i in range(1,12):
      if i == 1:
        data=self.gethtml(self.url)
        data=self.Process(data,i)[:-1]
        value=json.loads(data)
        file=open(&#39;WY.txt&#39;,&#39;a&#39;)
        for item in value[&#39;hotPosts&#39;]:
          try:
            file.write(item[&#39;1&#39;][&#39;f&#39;].encode(&#39;utf-8&#39;)+&#39;|&#39;)
            file.write(item[&#39;1&#39;][&#39;b&#39;].encode(&#39;utf-8&#39;)+&#39;|&#39;)
            file.write(item[&#39;1&#39;][&#39;a&#39;].encode(&#39;utf-8&#39;)+&#39;|&#39;)
            file.write(item[&#39;1&#39;][&#39;v&#39;].encode(&#39;utf-8&#39;)+&#39;\n&#39;)
          except:
            continue
        file.close()
        print &#39;--正在采集%d/12--&#39;%i
        time.sleep(5)
      else:
        page=self.getpage(i)
        data = self.gethtml(page)
        data = self.Process(data,i)[:-2]
        # print data
        value=json.loads(data)
        # print value
        file=open(&#39;WY.txt&#39;,&#39;a&#39;)
        for item in value[&#39;newPosts&#39;]:
          try:
            file.write(item[&#39;1&#39;][&#39;f&#39;].encode(&#39;utf-8&#39;)+&#39;|&#39;)
            file.write(item[&#39;1&#39;][&#39;b&#39;].encode(&#39;utf-8&#39;)+&#39;|&#39;)
            file.write(item[&#39;1&#39;][&#39;a&#39;].encode(&#39;utf-8&#39;)+&#39;|&#39;)
            file.write(item[&#39;1&#39;][&#39;v&#39;].encode(&#39;utf-8&#39;)+&#39;\n&#39;)
          except:
            continue
        file.close()
        print &#39;--正在采集%d/12--&#39;%i
        time.sleep(5)
if __name__ == &#39;__main__&#39;:
  WY().dealJSON()

Copy after login

以上就是我爬取的代码了。

The above is the detailed content of Python code example for grabbing NetEase news. For more information, please follow other related articles on the PHP Chinese website!