Practical crawler combat in Python: 58 city crawler

With the rapid development of the Internet, people can obtain the information they need through various channels. In this information age, web crawlers have become an indispensable tool. In this article, we will introduce the actual crawler in Python-58 city crawler.

1. Introduction to crawlers

A web crawler is an automated program that accesses web pages through the HTTP protocol and extracts the required data. On the Internet, there is a lot of data, but not all of it is available through APIs. Therefore, crawlers have become an important means of obtaining data.

The workflow of a crawler is generally divided into three steps:

1. Download web pages: download web pages through HTTP protocol, usually implemented using the requests library;
2. Parse web pages: The downloaded web page parses and extracts the required data, which is generally implemented using the BeautifulSoup4 library;
3. Storage data: Save the required data locally or in a database.

2. Practical crawler combat: 58 city crawler

58 city is a national classified information website, where users can publish product information, rental information, recruitment information, etc. This article will introduce how to implement the 58 city crawler through Python to obtain rental information.

1. Analyze the website

Before crawling, you need to analyze the 58 city website. By entering the rental page and selecting the desired city, you can find that the URL contains city information. For example, the URL of the rental page is: "https://[city pinyin].58.com/zufang/". By modifying the city pinyin in the URL, you can crawl rental information in other cities.

After opening the rental page, you can find that the page structure is divided into two parts: the search bar and the rental information list. The rental information list includes the title, rent, area, geographical location, housing type and other information of each rental information.

2. Writing a crawler

After analyzing the 58.com website, just write a crawler. First, you need to import the requests and BeautifulSoup4 libraries. The code is as follows:

import requests
from bs4 import BeautifulSoup

Next, obtaining the rental information in each city requires constructing the correct URL. The code is as follows:

city_pinyin = "bj"
url = "https://{}.58.com/zufang/".format(city_pinyin)

After obtaining the correct URL, you can use the requests library to obtain the HTML source code of the page. The code is as follows:

response = requests.get(url)
html = response.text

Now that you have obtained the HTML source code of the rental page, you need to use the BeautifulSoup4 library to parse the HTML source code and extract the required data. According to the page structure, the rental information list is contained in a div tag with a class of "list-wrap". We can obtain all div tags with class "list-wrap" through the find_all() function in the BeautifulSoup4 library. The code is as follows:

soup = BeautifulSoup(html, "lxml")
div_list = soup.find_all("div", class_="list-wrap")

After obtaining the div tag, you can traverse the tag list and extract the data of each rental information. According to the page structure, each piece of rental information is contained in a div tag with class "des", including title, rent, area, geographical location, housing type and other information. The code is as follows:

for div in div_list:
    info_list = div.find_all("div", class_="des")
    for info in info_list:
        # 提取需要的租房数据

In the for loop, we used the find_all() function to obtain all div tags with class "des". Next, we need to traverse these div tags and extract the required rental data. For example, the code to extract the title and other information of the rental information is as follows:

title = info.find("a", class_="t").text
rent = info.find("b").text
size = info.find_all("p")[0].text.split("/")[1]
address = info.find_all("p")[0].text.split("/")[0]
house_type = info.find_all("p")[1].text

Through the above code, we have successfully obtained each piece of rental information on the 58 city rental page and encapsulated it into variables. Next, by printing the variables of each rental information, you can see the data output on the console. For example:

print("标题：{}".format(title))
print("租金：{}".format(rent))
print("面积：{}".format(size))
print("地理位置：{}".format(address))
print("房屋类型：{}".format(house_type))

3. Summary

This article introduces the actual crawler in Python-58 city crawler. Before the crawler was implemented, we first analyzed the 58 city rental page and determined the URL to obtain rental information and the data that needed to be extracted. Then, the crawler was implemented using requests and BeautifulSoup4 library. Through the crawler, we successfully obtained the rental information of the 58 city rental page and encapsulated it into variables to facilitate subsequent data processing.

The above is the detailed content of Practical crawler combat in Python: 58 city crawler. For more information, please follow other related articles on the PHP Chinese website!