How to use Python crawler to get those valuable blog posts-Python Tutorial-php.cn

Home

Backend Development

Python Tutorial

How to use Python crawler to get those valuable blog posts

不言

Apr 04, 2018 pm 03:06 PM

python Obtain

The content of this article is how to use Python crawler to obtain those valuable blog posts. Now I share it with everyone. Friends in need can refer to the content of this article

How to use Python crawler to get those valuable blog posts

Author CDA Data Analyst

There are many wonderful technical blog articles on CSDN, We can crawl it and save it on the local disk, which is very convenient for reading and learning later. Now we will use python to write a crawler code to achieve this purpose.

What we want to do: automatically read blog articles, record titles, and save favorite articles to personal Save it on your computer hard drive for future study reference.

The process is roughly divided into the following steps:

1. Find the crawled target URL;
2. Analyze the web page, Find the information you want to save. Here we mainly save the content of blog articles;
3. Clean and organize the crawled information and save it on the local disk. .

Open the csdn web page. As an example, we randomly open a web page:

http://blog.csdn.net/u013088062/article/list/1.

It can be seen that the blogger is very interested in "C++ Convolutional Neural Network" and other articles about computer science. Well written.

How to use Python crawler to get those valuable blog posts

The crawler code is divided into three categories (classes) according to the idea. The following three with "#" give the beginning of each class respectively (the specific code is attached for everyone to actually run and implement):

How to use Python crawler to get those valuable blog posts

##The "class" method belongs to Python's object-oriented programming, which is sometimes better than the process-oriented programming we usually use. Convenient, object-oriented programming is often used in large projects. For beginners, object-oriented programming is not easy to master, but after learning and getting used to it, they will gradually transition from process-oriented to object-oriented programming.

Special note is that the RePage class mainly uses regular expressions to process information obtained from web pages. Regular expressions Set the string style as follows:

How to use Python crawler to get those valuable blog posts

## Use regular expressions to match the content to be crawled, which can be achieved using Python and other software tools. There are many rules for regular expressions, and each software uses them in similar ways. Making good use of regular expressions is an important part of crawlers and text mining.

The SaveText class saves the information locally. The effect is as follows:

How to use Python crawler to get those valuable blog posts

## Use python to write crawler code, which is concise and efficient. This article only explains the most basic usage of the crawler. Interested friends can download the code and take a look. I hope you will gain something from it.

Attached is the relevant Python code:

1#-*-coding:UTF-8-*- 2import re 3import urllib2 4import sys 5#Purpose : Read blog articles, record titles, and save article content in Htnl format 6# Version: python2.7.13 7#Function: Read web page content 8class GetHtmlPage(): 9 #Note the case 10 def __init__ (self,strPage): ##11 ##13 = URLLIB2.Request (Self.StRPAPGE) # Create page request 11 15 #Ate "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36 SE 2.X MetaSr 1.0" )16 to urllib2. ## # Web page encoding 20 #except urllib2.URLError, e: #Catch exception 23 code 24 26 print 'HTTP Error:' + e.reason 27 ##28 return rePage29#Regular expression, get the desired content30class RePage(): 31# Regular expression extracts content and returns linked list 32 def GetReText(self,page,recode): ##33 rePage = re.findall(recode,page,re.S) 34 return rePage 35#Save text 36class SaveText(): 37 def Save(self,text,tilte): 38 try: 39 t="blog\\"+tilte+".html" 40 because 42 45 if __name__ == "__main__":46 s = SaveText() 47 #File encoding 48 #Character decoding correctly 49 reload(sys) 50 sys.setdefaultencoding( "utf-8" ) #Get the default encoding of the system 51 #Get the web page 52 page = GetHtmlPage("http://blog.csdn.net/ u013088062/article/list/1") 53 htmlPage = page.GetPage() 54 #Extract content 55 reServer = RePage() 56 reBlog = reServer.GetReText(htmlPage,r'.*?(\s.+?)') #Get the URL link and title 57 #Go down to get the text 58 for ref in reBlog: 59 pageHeard = "http://blog.csdn.net/" #Add link header 60 strPage = pageHeard+ref[0] 61 tilte=ref[1 ].replace('[Top]', "") #Use the replacement function to remove complicated English 62 tilte=tilte.replace("\r\n","").lstrip().rstrip() 63 #Get the text 64 htmlPage = GetHtmlPage(strPage) 65 htmlPageData = htmlPage.GetPage() 66 reBlogText = reServer.GetReText(htmlPageData,'

##(.+?)

') 67 #Save file 68 for s1 in reBlogText : 69 s1='\n'+s1 70 s.Save(s1,tilte)

Related recommendations:

Development case of php implementing a simple crawler

Python crawler browser identification library

Record a simple Python crawler instance

The above is the detailed content of How to use Python crawler to get those valuable blog posts. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress images for free

Clothoff.io

AI clothes remover

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)

1 months ago By 尊渡假赌尊渡假赌尊渡假赌

R.E.P.O. Best Graphic Settings

1 months ago By 尊渡假赌尊渡假赌尊渡假赌

Assassin's Creed Shadows: Seashell Riddle Solution

2 weeks ago By DDD

R.E.P.O. How to Fix Audio if You Can't Hear Anyone

1 months ago By 尊渡假赌尊渡假赌尊渡假赌

R.E.P.O. Chat Commands and How to Use Them

1 months ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Where is the login entrance for gmail email?

7537

CakePHP Tutorial

1379

What is the format of the account name of steam

win11 activation key permanent

nyt connections hints and answers

Related knowledge

PHP and Python: Code Examples and Comparison Apr 15, 2025 am 12:07 AM

PHP and Python have their own advantages and disadvantages, and the choice depends on project needs and personal preferences. 1.PHP is suitable for rapid development and maintenance of large-scale web applications. 2. Python dominates the field of data science and machine learning.

How to train PyTorch model on CentOS Apr 14, 2025 pm 03:03 PM

Efficient training of PyTorch models on CentOS systems requires steps, and this article will provide detailed guides. 1. Environment preparation: Python and dependency installation: CentOS system usually preinstalls Python, but the version may be older. It is recommended to use yum or dnf to install Python 3 and upgrade pip: sudoyumupdatepython3 (or sudodnfupdatepython3), pip3install--upgradepip. CUDA and cuDNN (GPU acceleration): If you use NVIDIAGPU, you need to install CUDATool

How is the GPU support for PyTorch on CentOS Apr 14, 2025 pm 06:48 PM

Enable PyTorch GPU acceleration on CentOS system requires the installation of CUDA, cuDNN and GPU versions of PyTorch. The following steps will guide you through the process: CUDA and cuDNN installation determine CUDA version compatibility: Use the nvidia-smi command to view the CUDA version supported by your NVIDIA graphics card. For example, your MX450 graphics card may support CUDA11.1 or higher. Download and install CUDAToolkit: Visit the official website of NVIDIACUDAToolkit and download and install the corresponding version according to the highest CUDA version supported by your graphics card. Install cuDNN library:

Detailed explanation of docker principle Apr 14, 2025 pm 11:57 PM

Docker uses Linux kernel features to provide an efficient and isolated application running environment. Its working principle is as follows: 1. The mirror is used as a read-only template, which contains everything you need to run the application; 2. The Union File System (UnionFS) stacks multiple file systems, only storing the differences, saving space and speeding up; 3. The daemon manages the mirrors and containers, and the client uses them for interaction; 4. Namespaces and cgroups implement container isolation and resource limitations; 5. Multiple network modes support container interconnection. Only by understanding these core concepts can you better utilize Docker.

Python vs. JavaScript: Community, Libraries, and Resources Apr 15, 2025 am 12:16 AM

Python and JavaScript have their own advantages and disadvantages in terms of community, libraries and resources. 1) The Python community is friendly and suitable for beginners, but the front-end development resources are not as rich as JavaScript. 2) Python is powerful in data science and machine learning libraries, while JavaScript is better in front-end development libraries and frameworks. 3) Both have rich learning resources, but Python is suitable for starting with official documents, while JavaScript is better with MDNWebDocs. The choice should be based on project needs and personal interests.

How to choose the PyTorch version under CentOS Apr 14, 2025 pm 02:51 PM

When selecting a PyTorch version under CentOS, the following key factors need to be considered: 1. CUDA version compatibility GPU support: If you have NVIDIA GPU and want to utilize GPU acceleration, you need to choose PyTorch that supports the corresponding CUDA version. You can view the CUDA version supported by running the nvidia-smi command. CPU version: If you don't have a GPU or don't want to use a GPU, you can choose a CPU version of PyTorch. 2. Python version PyTorch

MiniOpen Centos compatibility Apr 14, 2025 pm 05:45 PM

MinIO Object Storage: High-performance deployment under CentOS system MinIO is a high-performance, distributed object storage system developed based on the Go language, compatible with AmazonS3. It supports a variety of client languages, including Java, Python, JavaScript, and Go. This article will briefly introduce the installation and compatibility of MinIO on CentOS systems. CentOS version compatibility MinIO has been verified on multiple CentOS versions, including but not limited to: CentOS7.9: Provides a complete installation guide covering cluster configuration, environment preparation, configuration file settings, disk partitioning, and MinI

How to install nginx in centos Apr 14, 2025 pm 08:06 PM

CentOS Installing Nginx requires following the following steps: Installing dependencies such as development tools, pcre-devel, and openssl-devel. Download the Nginx source code package, unzip it and compile and install it, and specify the installation path as /usr/local/nginx. Create Nginx users and user groups and set permissions. Modify the configuration file nginx.conf, and configure the listening port and domain name/IP address. Start the Nginx service. Common errors need to be paid attention to, such as dependency issues, port conflicts, and configuration file errors. Performance optimization needs to be adjusted according to the specific situation, such as turning on cache and adjusting the number of worker processes.

See all articles