如何编写 Python 程序爬取新浪军事论坛？-Python Tutorial-php.cn

回复内容：

context_re = r'(.*?)
'

你准备的这个正则表达式啊，truncated！断在了
这里，所以只能爬第一段。

爬取新浪军事论坛需要做三件事：

一、

上CSDN汪海老师的专栏，http://blog.csdn.net/column/details/why-bug.html，学习一个。

二、

按F12看一下前端。

三、

<span class="kn">from</span> <span class="nn">bs4</span> <span class="kn">import</span> <span class="n">BeautifulSoup</span>
<span class="kn">import</span> <span class="nn">requests</span>

<span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">"http://club.mil.news.sina.com.cn/thread-666013-1-1.html?retcode=0"</span><span class="p">)</span> <span class="c">#硬点网址</span>
<span class="n">response</span><span class="o">.</span><span class="n">encoding</span> <span class="o">=</span> <span class="s">'gb18030'</span> <span class="c">#中文编码</span>
<span class="n">soup</span> <span class="o">=</span> <span class="n">BeautifulSoup</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">text</span><span class="p">,</span> <span class="s">'html.parser'</span><span class="p">)</span> <span class="c">#构建BeautifulSoup对象</span>


<span class="n">divs</span> <span class="o">=</span> <span class="n">soup</span><span class="p">(</span><span class="s">'div'</span><span class="p">,</span> <span class="s">'mainbox'</span><span class="p">)</span>
 <span class="c">#每个楼层</span>

<span class="k">for</span> <span class="n">div</span> <span class="ow">in</span> <span class="n">divs</span><span class="p">:</span>
    <span class="n">comments</span> <span class="o">=</span> <span class="n">div</span><span class="o">.</span><span class="n">find_all</span><span class="p">(</span><span class="s">'div'</span><span class="p">,</span><span class="s">'cont f14'</span><span class="p">)</span> <span class="c">#每个楼层的正文</span>
        
    <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'Sina_Military_Club.txt'</span><span class="p">,</span><span class="s">'a'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
        <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s">'</span><span class="se">\n</span><span class="s">'</span><span class="o">+</span><span class="nb">str</span><span class="p">(</span><span class="n">comments</span><span class="p">)</span><span class="o">+</span><span class="s">'</span><span class="se">\n</span><span class="s">'</span><span class="p">)</span>

Copy after login

刚好几个小时前就在写一个爬取网站会员（公司）资料的小程序
具体的编程问题就不回答了，跟用什么语言写代码无关，关键是你要分析好这个页面的html代码结构，写出合适的正则表达式来进行匹配，如果想简化的话，可以进行分次匹配（比如先得到

里面的第一个

里面的内容就是原帖的地址，然后再进一步处理）
大数据分析就不会了，还请赐教。

<span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">from</span> <span class="nn">bs4</span> <span class="kn">import</span> <span class="n">BeautifulSoup</span>

<span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">"http://club.mil.news.sina.com.cn/thread-666013-1-1.html"</span><span class="p">)</span>
<span class="n">r</span><span class="o">.</span><span class="n">encoding</span> <span class="o">=</span> <span class="n">r</span><span class="o">.</span><span class="n">apparent_encoding</span>
<span class="n">soup</span> <span class="o">=</span> <span class="n">BeautifulSoup</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">soup</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">attrs</span><span class="o">=</span><span class="p">{</span><span class="s">"class"</span><span class="p">:</span> <span class="s">"cont f14"</span><span class="p">})</span>
<span class="k">print</span> <span class="n">result</span><span class="o">.</span><span class="n">text</span>

Copy after login

用beautifulSoup吧,正则太多了看着都头疼. 先用了BeautifulSoup爬取数据

<span class="c"># -*- coding:utf-8 -*-</span>

<span class="kn">import</span> <span class="nn">re</span><span class="o">,</span> <span class="nn">requests</span>
<span class="kn">from</span> <span class="nn">bs4</span> <span class="kn">import</span> <span class="n">BeautifulSoup</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="nb">reload</span><span class="p">(</span><span class="n">sys</span><span class="p">)</span>
<span class="n">sys</span><span class="o">.</span><span class="n">setdefaultencoding</span><span class="p">(</span><span class="s">'utf-8'</span><span class="p">)</span>

<span class="n">url</span> <span class="o">=</span> <span class="s">"http://club.mil.news.sina.com.cn/viewthread.php?tid=666013&extra=page%3D1&page=1"</span>

<span class="n">req</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="n">req</span><span class="o">.</span><span class="n">encoding</span> <span class="o">=</span> <span class="n">req</span><span class="o">.</span><span class="n">apparent_encoding</span>
<span class="n">html</span> <span class="o">=</span> <span class="n">req</span><span class="o">.</span><span class="n">text</span>

<span class="n">soup</span> <span class="o">=</span> <span class="n">BeautifulSoup</span><span class="p">(</span><span class="n">html</span><span class="p">)</span>

<span class="nb">file</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s">'sina_club.txt'</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span>
<span class="n">x</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">for</span> <span class="n">tag</span> <span class="ow">in</span> <span class="n">soup</span><span class="o">.</span><span class="n">find_all</span><span class="p">(</span><span class="s">'div'</span><span class="p">,</span> <span class="n">attrs</span> <span class="o">=</span> <span class="p">{</span><span class="s">'class'</span><span class="p">:</span> <span class="s">"cont f14"</span><span class="p">}):</span>
    <span class="n">word</span> <span class="o">=</span> <span class="n">tag</span><span class="o">.</span><span class="n">get_text</span><span class="p">()</span>
    <span class="n">line1</span> <span class="o">=</span> <span class="s">"---------------评论"</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">+</span> <span class="s">"---------------------"</span> <span class="o">+</span> <span class="s">"</span><span class="se">\n</span><span class="s">"</span>
    <span class="n">line2</span> <span class="o">=</span> <span class="n">word</span> <span class="o">+</span> <span class="s">"</span><span class="se">\n</span><span class="s">"</span>
    <span class="n">line</span> <span class="o">=</span> <span class="n">line1</span> <span class="o">+</span> <span class="n">line2</span>
    <span class="n">x</span> <span class="o">+=</span> <span class="mi">1</span>
    <span class="nb">file</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="nb">file</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>

Copy after login

哎，扒就扒吧，发了paper能不能告诉我刊号页数让我看一下？我们自己都没做大数据分析…… 建议用一下正则测试工具你需要pyquery，可以使用jquery一样的语法。你值得拥有。
https://pythonhosted.org/pyquery/

Previous article：为什么 Python 不支持函数重载？ Next article：Python 程序如何高效地调试？

Statement of this Website

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn