Python crawling article example tutorial-Python Tutorial-php.cn

Home

Backend Development

Python Tutorial

Python crawling article example tutorial

巴扎黑

Aug 07, 2017 pm 05:37 PM

python Tutorial article

This article mainly introduces you to the relevant information of using python to crawl prose network articles. The introduction in the article is very detailed and has certain reference and learning value for everyone. Friends who need it can take a look below.

This article mainly introduces to you the relevant content about python crawling prose network articles. It is shared for your reference and study. Let’s take a look at the detailed introduction:

The rendering is as follows:

Configure python 2.7

 bs4

 requests

Copy after login

Installation using pipsudo pip install bs4

sudo pip install requests

Copy after login

Briefly explain the use of bs4 because it is crawling web pages, so I will introduce it. find and find_all

The difference between find and find_all is that they return different things. find returns the first matched tag and the content in the tag

find_all returns a list

For example, we write a test.html to test the difference between find and find_all.

The content is:

<html>
<head>
</head>
<body>
<p id="one"><a></a></p>
<p id="two"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >abc</a></p>
<p id="three"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >three a</a><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >three a</a><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >three a</a></p>
<p id="four"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >four<p>four p</p><p>four p</p><p>four p</p> a</a></p>
</body>
</html>

Copy after login

Then the code of test.py is:

from bs4 import BeautifulSoup
import lxml

if __name__==&#39;__main__&#39;:
 s = BeautifulSoup(open(&#39;test.html&#39;),&#39;lxml&#39;)
 print s.prettify()
 print "------------------------------"
 print s.find(&#39;p&#39;)
 print s.find_all(&#39;p&#39;)
 print "------------------------------"
 print s.find(&#39;p&#39;,id=&#39;one&#39;)
 print s.find_all(&#39;p&#39;,id=&#39;one&#39;)
 print "------------------------------"
 print s.find(&#39;p&#39;,id="two")
 print s.find_all(&#39;p&#39;,id="two")
 print "------------------------------"
 print s.find(&#39;p&#39;,id="three")
 print s.find_all(&#39;p&#39;,id="three")
 print "------------------------------"
 print s.find(&#39;p&#39;,id="four")
 print s.find_all(&#39;p&#39;,id="four")
 print "------------------------------"

Copy after login

After running, we can see the result. When getting the specified tag, there is not much difference between the two. When getting a group of tags, the difference between the two will be displayed.

So when we use it, we must pay attention to what we want, otherwise an error will occur

The next step is to obtain web page information through requests. I don’t quite understand why others write heard. With other things

I directly access the web page, obtain several classified secondary web pages of Prose Network through the get method, and then crawl all the web pages through a group test

def get_html():
 url = "https://www.sanwen.net/"
 two_html = [&#39;sanwen&#39;,&#39;shige&#39;,&#39;zawen&#39;,&#39;suibi&#39;,&#39;rizhi&#39;,&#39;novel&#39;]
 for doc in two_html:
 i=1
  if doc==&#39;sanwen&#39;:
  print "running sanwen -----------------------------"
  if doc==&#39;shige&#39;:
  print "running shige ------------------------------"
  if doc==&#39;zawen&#39;:
  print &#39;running zawen -------------------------------&#39;
  if doc==&#39;suibi&#39;:
  print &#39;running suibi -------------------------------&#39;
  if doc==&#39;rizhi&#39;:
  print &#39;running ruzhi -------------------------------&#39;
  if doc==&#39;nove&#39;:
  print &#39;running xiaoxiaoshuo -------------------------&#39;
 while(i<10):
 par = {&#39;p&#39;:i}
 res = requests.get(url+doc+&#39;/&#39;,params=par)
 if res.status_code==200:
  soup(res.text)
  i+=i

Copy after login

In this part of the code, I did not process the

res.status_code that is not 200. The resulting problem is that the error will not be displayed, and the crawled content will have lost. Then I analyzed the web page of prose website and found that it was www.sanwen.net/rizhi/&p=1

The maximum value of p is 10. I don’t understand this. The last time I crawled the disk, it was 100 pages. Forget it. Forget it and analyze it later. Then get the content of each page through the get method.

After getting the content of each page, we analyze the author and title. The code is like this

def soup(html_text):
 s = BeautifulSoup(html_text,&#39;lxml&#39;)
 link = s.find(&#39;p&#39;,class_=&#39;categorylist&#39;).find_all(&#39;li&#39;)
 for i in link:
 if i!=s.find(&#39;li&#39;,class_=&#39;page&#39;):
 title = i.find_all(&#39;a&#39;)[1]
 author = i.find_all(&#39;a&#39;)[2].text
 url = title.attrs[&#39;href&#39;]
 sign = re.compile(r&#39;(//)|/&#39;)
 match = sign.search(title.text)
 file_name = title.text
 if match:
 file_name = sign.sub(&#39;a&#39;,str(title.text))

Copy after login

There is something wrong when getting the title. Please tell me. Guys, why do you add slashes in the title when writing prose? Not only one but also two. This problem directly caused an error in the file name when I wrote the file later, so I wrote a regular expression and I changed it for you. Bar.

The last step is to get the prose content. Through the analysis of each page, we get the article address, and then get the content directly. I originally wanted to get it one by one by changing the web page address, which saves trouble.

def get_content(url):
 res = requests.get(&#39;https://www.sanwen.net&#39;+url)
 if res.status_code==200:
 soup = BeautifulSoup(res.text,&#39;lxml&#39;)
 contents = soup.find(&#39;p&#39;,class_=&#39;content&#39;).find_all(&#39;p&#39;)
 content = &#39;&#39;
 for i in contents:
 content+=i.text+&#39;\n&#39;
 return content

Copy after login

The last thing is to write the file and save it ok

 f = open(file_name+&#39;.txt&#39;,&#39;w&#39;)

 print &#39;running w txt&#39;+file_name+&#39;.txt&#39;
 f.write(title.text+&#39;\n&#39;)
 f.write(author+&#39;\n&#39;)
 content=get_content(url) 
 f.write(content)
 f.close()

Copy after login

Three functions get the prose from the prose network, but there are The problem, the problem is that I don’t know why some prose is lost. I can only get about 400 articles. This is much different from the articles on Prose.com, but it is indeed obtained page by page. I hope this problem will be solved. Please help me. Maybe we should make the web page inaccessible. Of course, I think it has something to do with the broken network in my dormitory

 f = open(file_name+&#39;.txt&#39;,&#39;w&#39;)
 print &#39;running w txt&#39;+file_name+&#39;.txt&#39;
 f.write(title.text+&#39;\n&#39;)
 f.write(author+&#39;\n&#39;)
 content=get_content(url) 
 f.write(content)
 f.close()

Copy after login

Almost forgot about the rendering

There may be a timeout phenomenon. I can only say that you must choose a good Internet connection when going to college!

The above is the detailed content of Python crawling article example tutorial. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress images for free

Clothoff.io

AI clothes remover

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

How to fix KB5055523 fails to install in Windows 11?

3 weeks ago By DDD

How to fix KB5055518 fails to install in Windows 10?

3 weeks ago By DDD

Roblox: Grow A Garden - Complete Mutation Guide

2 weeks ago By DDD

Roblox: Bubble Gum Simulator Infinity - How To Get And Use Royal Keys

3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

How to fix KB5055612 fails to install in Windows 10?

3 weeks ago By DDD

Hot Tools

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Java Tutorial

1662

CakePHP Tutorial

1419

Laravel Tutorial

1313

PHP Tutorial

1262

C# Tutorial

1235

Related knowledge

PHP and Python: Different Paradigms Explained Apr 18, 2025 am 12:26 AM

PHP is mainly procedural programming, but also supports object-oriented programming (OOP); Python supports a variety of paradigms, including OOP, functional and procedural programming. PHP is suitable for web development, and Python is suitable for a variety of applications such as data analysis and machine learning.

Choosing Between PHP and Python: A Guide Apr 18, 2025 am 12:24 AM

PHP is suitable for web development and rapid prototyping, and Python is suitable for data science and machine learning. 1.PHP is used for dynamic web development, with simple syntax and suitable for rapid development. 2. Python has concise syntax, is suitable for multiple fields, and has a strong library ecosystem.

PHP and Python: A Deep Dive into Their History Apr 18, 2025 am 12:25 AM

PHP originated in 1994 and was developed by RasmusLerdorf. It was originally used to track website visitors and gradually evolved into a server-side scripting language and was widely used in web development. Python was developed by Guidovan Rossum in the late 1980s and was first released in 1991. It emphasizes code readability and simplicity, and is suitable for scientific computing, data analysis and other fields.

Python vs. JavaScript: The Learning Curve and Ease of Use Apr 16, 2025 am 12:12 AM

Python is more suitable for beginners, with a smooth learning curve and concise syntax; JavaScript is suitable for front-end development, with a steep learning curve and flexible syntax. 1. Python syntax is intuitive and suitable for data science and back-end development. 2. JavaScript is flexible and widely used in front-end and server-side programming.

How to run sublime code python Apr 16, 2025 am 08:48 AM

To run Python code in Sublime Text, you need to install the Python plug-in first, then create a .py file and write the code, and finally press Ctrl B to run the code, and the output will be displayed in the console.

Where to write code in vscode Apr 15, 2025 pm 09:54 PM

Writing code in Visual Studio Code (VSCode) is simple and easy to use. Just install VSCode, create a project, select a language, create a file, write code, save and run it. The advantages of VSCode include cross-platform, free and open source, powerful features, rich extensions, and lightweight and fast.

Can visual studio code be used in python Apr 15, 2025 pm 08:18 PM

VS Code can be used to write Python and provides many features that make it an ideal tool for developing Python applications. It allows users to: install Python extensions to get functions such as code completion, syntax highlighting, and debugging. Use the debugger to track code step by step, find and fix errors. Integrate Git for version control. Use code formatting tools to maintain code consistency. Use the Linting tool to spot potential problems ahead of time.

How to run python with notepad Apr 16, 2025 pm 07:33 PM

Running Python code in Notepad requires the Python executable and NppExec plug-in to be installed. After installing Python and adding PATH to it, configure the command "python" and the parameter "{CURRENT_DIRECTORY}{FILE_NAME}" in the NppExec plug-in to run Python code in Notepad through the shortcut key "F6".

See all articles