A summary of the python standard library Beautiful Soup and MongoDb climbing Himalayan Radio

Beautiful Soup standard library is a Python library that can extract data from HTML/XML files. It can realize the usual way of document navigation, search and modification through your favorite converter. Beautiful Soup will save hours. working hours. The pymongo standard library is the bridge between the MongoDb NoSql database and the python language. Data is saved to MongoDb through pymongo. Use the two together to crawl the Himalayan radio data...

Beautiful Soup supports the HTML parser in the Python standard library, and also supports some third-party parsers, one of which is lxml. This article uses lxml. For the installation of this, please see the python 3.6 lxml standard library lxml installation and the use of etree. Note
At the same time, this article uses XPath to parse the part we want. For the introduction of XPath and Beautiful Soup For more information and usage, please see Beautiful Soup 4.4.0 Document XPath Introduction
The knowledge of Beautiful Soup and XPath involved in this article is not very deep. You can understand it by looking at the official documentation, and I also added comments...
Regarding the pymongo standard library, I will not talk too much. For details, please see the pymongo module experience of the python standard library

Sometimes, we need to determine the type of client currently making a request to the server, which is commonly known as User-Agent, referred to as UA. The browser we use when browsing the web is a type of UA. In other words, UA is the browser. In the HTTP protocol, the User-Agent request header indicates the type of the user's browser and the operation Identification of system, browser kernel and other information. Through this logo, different versions of the websites visited can be displayed to provide users with a better experience or to conduct information statistics. Some websites officially use UA to prevent hackers or boring people like us from crawling the website's data information.
Therefore, the code in this article first lists all UAs to facilitate subsequent crawling work.

Okay, let’s clarify what data we want to crawl and obtain:

What we need is the link to the image, alt, etc.

Then we click on the image link to get the details inside. If some radio stations have multiple pages, then we use xpath to access them one by one. At the same time, we obtain the sound_id of the sound module in the album on the page...

The procedure is as follows:

import random
import requests
from bs4 import BeautifulSoup
import json
from lxml import etree
import pymongo


clients = pymongo.MongoClient("localhost", 27017)
db = clients["XiMaLaYa"]
collection_1 = db["album"]
collection_2 = db["detail"]

UA_LIST = [
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
    "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
    "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
    "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SE 2.X MetaSr 1.0; SE 2.X MetaSr 1.0; .NET CLR 2.0.50727; SE 2.X MetaSr 1.0)",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
    "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
    "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
    "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
    "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
    "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
]
headers1 = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate, sdch',
    'Accept-Language': 'zh-CN,zh;q=0.8,en;q=0.6',
    'Cache-Control': 'max-age=0',
    'Proxy-Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': random.choice(UA_LIST)  # User_agence表示用户代理
}
headers2 = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate, sdch',
    'Accept-Language': 'zh-CN,zh;q=0.8,en;q=0.6',
    'Cache-Control': 'max-age=0',
    'Proxy-Connection': 'keep-alive',
    'Referer': 'http://www.ximalaya.com/dq/all/2',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': random.choice(UA_LIST)
}


# Beautiful库用来处理XML和HTML...
# 主要就是利用BeautifulSoup模块来处理requests模块获取的Html源码
# 利用lxml模块将html源码解析成树结构，xpath来处理树节点.
def get_url():
    start_urls = ["http://www.ximalaya.com/dq/all/{}".format(num) for num in range(1，85)]
    # start_urls = ["http://www.ximalaya.com/dq/all/1"]
    for start_url in start_urls:
        html = requests.get(start_url, headers=headers1).text
        soup = BeautifulSoup(html, "lxml")  # 使用lxml来处理
        for item in soup.find_all(class_="albumfaceOutter"):  # 解析并查找xml节点
            content = {
                'href': item.a["href"],
                'title': item.img['alt'],
                'img_url': item.img['src']
            }
            collection_1.insert(content)
            # another(item.a["href"])
    print('写入完成...')


# 进入电台具体页面 http://www.ximalaya.com/15836959/album/303085，并处理分页录音...
def another(url):
    html = requests.get(url, headers=headers1).text
    # / :表示从根节点选取....
    # // :表示匹配选择的当前节点选择文档中的节点，而不考虑他们的位置...
    ifanother = etree.HTML(html).xpath('//div[@class="pagingBar_wrapper"]/a[last()-1]/@data-page')  # 页面链接地址  ifanother是list类型...
    if len(ifanother):  # 判断一个video的录音是否分割成了多页....
        num = ifanother[0]  # 获取页面数...
        print('本频道保存在' + num + '个页面')
        for n in range(1, int(num)):
            url2 = url + '?page={}'.format(n)
            get_m4a(url2)
        get_m4a(url)


# 获取分页录音页面的详细数据...
def get_m4a(url):
    html = requests.get(url, headers=headers2).text
    numlist = etree.HTML(html).xpath('//div[@class="personal_body"]/@sound_ids')[0].split(',')
    for i in numlist:
        murl = 'http://www.ximalaya.com/tracks/{}.json'.format(i)
        html = requests.get(murl, headers=headers1).text
        dic = json.loads(html)
        collection_2.insert(dic)


if __name__ == "__main__":
    get_url()

The above is the detailed content of A summary of the python standard library Beautiful Soup and MongoDb climbing Himalayan Radio. For more information, please follow other related articles on the PHP Chinese website!