Introducing Python to crawl Bilibili videos
python video tutorial column introduces how to crawl videos
Related free learning recommendations: python video tutorial
This article is mainly for Please explain how to use python to crawl videos on Bilibili. First of all, I am a big data development engineer. Crawlers are just a hobby of mine. I like crawlers. Partners can communicate together. It’s much better, so I won’t say much more. If you like it, you can collect it, please reply to the original link for forwarding, thank you
1. Environment preparation
The environment I am using here is as follows for reference only:
Development tools: pycharm
python environment: python-3.8.0
Dependent packages : shutil, os, re, json, choice, requests, lxml
2. Page analysis
Here I will take the video of Teacher Ma that was very popular some time ago Let’s give an example.
Video link: https://www.bilibili.com/video/BV1Ef4y1i78b?from=search&seid=12072538764197074893
- Video link analysis here we only need BV1Ef4y1i78b That is
Behind the video? - The second part of the packet capture in front of the number, the video here on Bilibili is divided into multiple small segments After looking at the source code analysis, we can analyze it
The content in script><script></script>returns a json string and parses it to get the data we want. .
- Analysis returns the specific content in json
The information returned to us is as follows,The information that is really useful to us is in the data
Under data, we can clearly see the content we want, such as the video quality, video address, etc. Note: If you get the address and access it directly The cannot be accessed. Bilibili has added Referer. If you access it directly in the browser, there is no Referer and the page cannot be found.
The content we need to parse is as follows:
- Duration of video
- Quality of video
- URL of video
- URL of audio
- Audio and video merging
1 |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 |
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
def single_download(aid, acc_quality):
'''单个视频实现下载'''
# 请求视频链接，获取信息
origin_video_url = 'https://www.bilibili.com/video/' + aid
res = requests.get(origin_video_url, headers=headers)
html = etree.HTML(res.text)
title = html.xpath('//*[@id="viewbox_report"]/h1/span/text()')[0]
print('您当前正在下载：', title)
video_info_temp = re_video_info(res.text, '__playinfo__=(.*?)<script>')
video_info = {}
# 获取视频质量
quality = video_info_temp['data']['accept_description'][acc_quality]
# 获取视频时长
video_info['duration'] = video_info_temp['data']['dash']['duration']
# 获取视频链接
video_url = video_info_temp['data']['dash']['video'][acc_quality]['baseUrl']
# 获取音频链接
audio_url = video_info_temp['data']['dash']['audio'][acc_quality]['baseUrl']
# 计算视频时长
video_time = int(video_info.get('duration', 0))
video_minute = video_time // 60
video_second = video_time % 60
print('当前视频清晰度为{}，时长{}分{}秒'.format(quality, video_minute, video_second))
# 调用函数下载保存视频
download_video_single(origin_video_url, video_url, audio_url, title)</script>
Copy after login
3.3 Write download code1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
def download_video_single(referer_url, video_url, audio_url, video_name):
'''单个视频下载'''
# 更新请求头
headers.update({"Referer": referer_url})
print("视频下载开始：%s" % video_name)
# 下载并保存视频
video_content = requests.get(video_url, headers=headers)
print('%s\t视频大小：' % video_name, round(int(video_content.headers.get('content-length', 0)) / 1024 / 1024, 2), '\tMB')
received_video = 0
with open('%s_video.mp4' % video_name, 'ab') as output:
headers['Range'] = 'bytes=' + str(received_video) + '-'
response = requests.get(video_url, headers=headers)
output.write(response.content)
# 下载并保存音频
audio_content = requests.get(audio_url, headers=headers)
print('%s\t音频大小：' % video_name, round(int(audio_content.headers.get('content-length', 0)) / 1024 / 1024, 2), '\tMB')
received_audio = 0
with open('%s_audio.mp4' % video_name, 'ab') as output:
headers['Range'] = 'bytes=' + str(received_audio) + '-'
response = requests.get(audio_url, headers=headers)
output.write(response.content)
received_audio += len(response.content)
print("视频下载结束：%s" % video_name)
video_audio_merge_single(video_name)
Copy after login
3.4 Merge downloaded audio and video1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
|
1
2
3
4
5
6
7
8
def video_audio_merge_single(video_name):
'''使用ffmpeg单个视频音频合并'''
print("视频合成开始：%s" % video_name)
import subprocess
command = 'ffmpeg -i %s_video.mp4 -i %s_audio.mp4 -c copy %s.mp4 -y -loglevel quiet' % (
video_name, video_name, video_name)
subprocess.Popen(command, shell=True)
print("视频合成结束：%s" % video_name)
Copy after login
3.4 Running test test test
1 2 3 4 5 6 7 8 |
|
## 4. Summary
好## The video is
. If you feel that there is something you don’t understand or have any doubts about, you can leave a message in the background and I will answer it for you.php programming
(video)
The above is the detailed content of Introducing Python to crawl Bilibili videos. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1389
52
PHP and Python: Different Paradigms Explained
Apr 18, 2025 am 12:26 AM
PHP is mainly procedural programming, but also supports object-oriented programming (OOP); Python supports a variety of paradigms, including OOP, functional and procedural programming. PHP is suitable for web development, and Python is suitable for a variety of applications such as data analysis and machine learning.
Choosing Between PHP and Python: A Guide
Apr 18, 2025 am 12:24 AM
PHP is suitable for web development and rapid prototyping, and Python is suitable for data science and machine learning. 1.PHP is used for dynamic web development, with simple syntax and suitable for rapid development. 2. Python has concise syntax, is suitable for multiple fields, and has a strong library ecosystem.
Can vs code run in Windows 8
Apr 15, 2025 pm 07:24 PM
VS Code can run on Windows 8, but the experience may not be great. First make sure the system has been updated to the latest patch, then download the VS Code installation package that matches the system architecture and install it as prompted. After installation, be aware that some extensions may be incompatible with Windows 8 and need to look for alternative extensions or use newer Windows systems in a virtual machine. Install the necessary extensions to check whether they work properly. Although VS Code is feasible on Windows 8, it is recommended to upgrade to a newer Windows system for a better development experience and security.
Is the vscode extension malicious?
Apr 15, 2025 pm 07:57 PM
VS Code extensions pose malicious risks, such as hiding malicious code, exploiting vulnerabilities, and masturbating as legitimate extensions. Methods to identify malicious extensions include: checking publishers, reading comments, checking code, and installing with caution. Security measures also include: security awareness, good habits, regular updates and antivirus software.
How to run programs in terminal vscode
Apr 15, 2025 pm 06:42 PM
In VS Code, you can run the program in the terminal through the following steps: Prepare the code and open the integrated terminal to ensure that the code directory is consistent with the terminal working directory. Select the run command according to the programming language (such as Python's python your_file_name.py) to check whether it runs successfully and resolve errors. Use the debugger to improve debugging efficiency.
Can visual studio code be used in python
Apr 15, 2025 pm 08:18 PM
VS Code can be used to write Python and provides many features that make it an ideal tool for developing Python applications. It allows users to: install Python extensions to get functions such as code completion, syntax highlighting, and debugging. Use the debugger to track code step by step, find and fix errors. Integrate Git for version control. Use code formatting tools to maintain code consistency. Use the Linting tool to spot potential problems ahead of time.
Can vscode be used for mac
Apr 15, 2025 pm 07:36 PM
VS Code is available on Mac. It has powerful extensions, Git integration, terminal and debugger, and also offers a wealth of setup options. However, for particularly large projects or highly professional development, VS Code may have performance or functional limitations.
Can vscode run ipynb
Apr 15, 2025 pm 07:30 PM
The key to running Jupyter Notebook in VS Code is to ensure that the Python environment is properly configured, understand that the code execution order is consistent with the cell order, and be aware of large files or external libraries that may affect performance. The code completion and debugging functions provided by VS Code can greatly improve coding efficiency and reduce errors.
