It was my first time to learn crawler technology. I read a joke on Zhihu about how to crawl to the Encyclopedia of Embarrassing Things, so I decided to make one myself.
Achieve goals: 1. Crawling to the jokes in the Encyclopedia of Embarrassing Things
2. Crawling one paragraph every time and crawling to the next page every time you press Enter
Technical implementation: Based on the implementation of python, using the Requests library, re library, and the BeautifulSoup method of the bs4 library to implement
Main content: First, we need to clarify the ideas for crawling implementation , let’s build the main framework. In the first step, we first write a method to obtain web pages using the Requests library. In the second step, we use the BeautifulSoup method of the bs4 library to analyze the obtained web page information and use regular expressions to match relevant paragraph information. . The third step is to print out the obtained information. We all execute the above methods through a main function .
First, import the relevant libraries
import requests
from bs4 import BeautifulSoup
import bs4
import re
Copy after login
Second, first obtain the web page information
def getHTMLText(url):
try:
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
headers = {'User-Agent': user_agent}
r = requests.get(url,headers = headers)
r.raise_for_status()
r.encoding = r.apparent_encoding
return r.text
except:
return ""Copy after login
Third, put the information into r and then analyze it
soup = BeautifulSoup(html,"html.parser")
Copy after login
What we need is the content and publisher of the joke. By viewing the source code on the web page, we know that the publisher of the joke is:
'p', attrs={'class': 'content'}中Copy after login
The content of the joke is in
'p', attrs={'class': 'author clearfix'}中Copy after login
, so we pass bs4 Library method to extract the specific content of these two tags
def fillUnivlist(lis,li,html,count):
soup = BeautifulSoup(html,"html.parser")
try:
a = soup.find_all('p', attrs={'class': 'content'})
ll = soup.find_all('p', attrs={'class': 'author clearfix'})Copy after login
Then obtain the information through specific regular expressions
for sp in a:
patten = re.compile(r'(.*?)',re.S)
Info = re.findall(patten,str(sp))
lis.append(Info)
count = count + 1
for mc in ll:
namePatten = re.compile(r'(.*?)
', re.S)
d = re.findall(namePatten, str(mc))
li.append(d)
Copy after login
What we need to pay attention to is the return of find_all and re’s findall method They are all a list. When using regular expressions, we only roughly extract and do not remove the line breaks in the tags
Next, we only need to combine the contents of the two lists and output them
def printUnivlist(lis,li,count):
for i in range(count):
a = li[i][0]
b = lis[i][0]
print ("%s:"%a+"%s"%b)Copy after login
Then I make an input control function, enter Q to return an error, exit, enter Enter to return correct, and load the next page of paragraphs
def input_enter():
input1 = input()
if input1 == 'Q':
return False
else:
return True
Copy after login
We realize the input control through the main function. If If the control function returns an error, the output will not be performed. If the return value is correct, the output will continue. We load the next page through a for loop.
def main():
passage = 0
enable = True
for i in range(20):
mc = input_enter()
if mc==True:
lit = []
li = []
count = 0
passage = passage + 1
qbpassage = passage
print(qbpassage)
url = 'http://www.qiushibaike.com/8hr/page/' + str(qbpassage) + '/?s=4966318'
a = getHTMLText(url)
fillUnivlist(lit, li, a, count)
number = fillUnivlist(lit, li, a, count)
printUnivlist(lit, li, number)
else:
break
Copy after login
Here we need to note that every for loop will refresh lis[] and li[], so that the paragraph content of the webpage can be correctly output every time
Here is the source code :
import requests
from bs4 import BeautifulSoup
import bs4
import re
def getHTMLText(url):
try:
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
headers = {'User-Agent': user_agent}
r = requests.get(url,headers = headers)
r.raise_for_status()
r.encoding = r.apparent_encoding
return r.text
except:
return ""
def fillUnivlist(lis,li,html,count):
soup = BeautifulSoup(html,"html.parser")
try:
a = soup.find_all('p', attrs={'class': 'content'})
ll = soup.find_all('p', attrs={'class': 'author clearfix'})
for sp in a:
patten = re.compile(r'(.*?)',re.S)
Info = re.findall(patten,str(sp))
lis.append(Info)
count = count + 1
for mc in ll:
namePatten = re.compile(r'(.*?)
', re.S)
d = re.findall(namePatten, str(mc))
li.append(d)
except:
return ""
return count
def printUnivlist(lis,li,count):
for i in range(count):
a = li[i][0]
b = lis[i][0]
print ("%s:"%a+"%s"%b)
def input_enter():
input1 = input()
if input1 == 'Q':
return False
else:
return True
def main():
passage = 0
enable = True
for i in range(20):
mc = input_enter()
if mc==True:
lit = []
li = []
count = 0
passage = passage + 1
qbpassage = passage
print(qbpassage)
url = 'http://www.qiushibaike.com/8hr/page/' + str(qbpassage) + '/?s=4966318'
a = getHTMLText(url)
fillUnivlist(lit, li, a, count)
number = fillUnivlist(lit, li, a, count)
printUnivlist(lit, li, number)
else:
break
main()Copy after login
This is my first time doing it and there are still many areas that can be optimized. I hope everyone can point it out.
The above is the detailed content of Detailed explanation of the method of crawling to the Encyclopedia of Embarrassing Things using Python's crawler technology. For more information, please follow other related articles on the PHP Chinese website!
![[Web front-end] Node.js quick start](https://img.php.cn/upload/course/000/000/067/662b5d34ba7c0227.png)
