Crawler｜Python crawls pictures of girls from station B, motivation to learn!

In this issue, I will introduce to you how toUse Python to crawl the pictures of the ladies at Station B, hope it helps.

##1. Web page analysis

##Directly open station B (bilibili) and search for' Miss Sister':

There are a total of 5 pages of content, Take page 2 as an example, F12 to open the web page source code:

##Search for the first one title, we can find the corresponding XHR request. After careful analysis, we find that all the data exists in a json format data set, and our goal is resultList.

Check the Headers as follows:

##This is a get request. The number of entries in the request includes page and keywordTwo entries, corresponding to the requested page number and keyword respectively.

Check a few more pages to find the pattern:

# 第一页
'https://api.bilibili.com/x/web-interface/search/all/v2?context=&page=1&order=totalrank&keyword=%E5%B0%8F%E5%A7%90%E5%A7%90&duration=0&tids_2=&from_source=&from_spmid=333.337&platform=pc&__refresh__=true&_extra=&tids=0&highlight=1&single_column=0'
# 第二页
'https://api.bilibili.com/x/web-interface/search/type?context=&page=2&order=totalrank&keyword=%E5%B0%8F%E5%A7%90%E5%A7%90&duration=0&tids_2=&from_source=&from_spmid=333.337&platform=pc&__refresh__=true&_extra=&search_type=video&tids=0&highlight=1&single_column=0'
# 第三页
'https://api.bilibili.com/x/web-interface/search/type?context=&page=3&order=totalrank&keyword=%E5%B0%8F%E5%A7%90%E5%A7%90&duration=0&tids_2=&from_source=&from_spmid=333.337&platform=pc&__refresh__=true&_extra=&search_type=video&tids=0&highlight=1&single_column=0'

You can see that except for the first page, the other page URLsonly have different page parameters, then let’s try it Also use the URLs of other pages to request the first page, and you will find that the desired results can also be obtained (try it yourself).

in conclusion: All page URLs differ only in the page parameter, and the rest are the same.

2. 数据爬取

2.1 导入模块

# 导包
import re
import time
import json
import random
import requests
from fake_useragent import UserAgent

2.2 获取页面信息

根据分析的url请求数据：

# 获取页面信息
def get_datas(url,headers):
    r = requests.get(url, headers=headers)
    r.raise_for_status()
    r.encoding = chardet.detect(r.content)['encoding'] 
    datas = json.loads(r.text)
    return datas

2.3 获取具体图片信息

# 获取图片链接信息
def get_hrefs(datas):
    titles,hrefs = [],[]
    for data in datas['data']['result']:
        # 标题
        title = data['title']
        # 时长
        duration = data['duration']
        # 播放量
        video_review =data['video_review']
        # 发布时间
        date_rls = data['pubdate']
        pubdate = time.strftime('%Y-%m-%d %H:%M', time.localtime(date_rls))
        # 作者
        author = data['author']
        # 图片链接
        link_pic = data['pic']
        href_pic = 'https:' + link_pic
        
        titles.append(title)
        hrefs.append(href_pic)
        
        return titles, hrefs

代码解析了视频标题，时长，播放量，发布时间，作者，图片链接等参数，这里我们只取标题和图片链接，其他参数可根据需要自行增，删。

2.4 保存图片

# 保存图片
def download_jpg(titles, hrefs):
    path = "D:/B站小姐姐/"
    if not os.path.exists(path):
        os.mkdir(path)
    for i in range(len(hrefs)):
        title_t = titles[i].replace('/','').replace('，','').replace('?','')
     title_t = title_t.replace(' ','').replace('|','').replace('。','')
        filename = '{}{}.jpg'.format(path,title_t)
        with open(filename, 'wb') as f:
            req = requests.get(url=hrefs[i], headers=headers)
            f.write(req.content)
            time.sleep(random.uniform(1.5,3.4))

这里我们用标题作为图片名称进行存储，需要注意文件名称不能包含特殊符号，这里过滤了” / ，。｜“等4种（每天视频有增删，可能有出入，需要自己调整，也可以不使用标题做名称）。

3. Result

##Part of the picture:

##

The above is the detailed content of Crawler｜Python crawls pictures of girls from station B, motivation to learn!. For more information, please follow other related articles on the PHP Chinese website!