H5 video live broadcast

Live video is so popular, if you don’t learn it, you will be out.

In order to keep up with the trend, this article will introduce to you the basic process and main technical points in live video broadcast, including but not limited to front-end technology.

1 Can H5 do live video streaming?

Of course, H5 has been popular for so long and covers all aspects of technology.

For video recording, you can use the powerful webRTC (Web Real-Time Communication), which is a technology that supports web browsers for real-time voice conversations or video conversations. The disadvantage is that it is only available on PCs. The support on chrome is good, but the mobile support is not ideal.

For video playback, you can use the HLS (HTTP Live Streaming) protocol to play the live stream. Both ios and android naturally support this protocol. The configuration is simple, just use the video tag directly.

webRTC compatibility:

video tag to play hls protocol video:

<video controls autoplay>  
    <source src="http://10.66.69.77:8080/hls/mystream.m3u8" type="application/vnd.apple.mpegurl" />  
    <p class="warning">Your browser does not support HTML5 video.</p>  
</video>

2 What exactly is the HLS protocol?

To put it simply, it is to divide the entire stream into small HTTP-based files to download, and only download a few at a time. The one mentioned earlier is used when playing live video in H5. .m3u8 file, this file is based on the HLS protocol and stores video stream metadata.

Each .m3u8 file corresponds to several ts files. These ts files are the actual data that stores the video. The m3u8 file only stores the configuration information and related paths of some ts files. When the video is played, .m3u8 is changed dynamically. The video tag will parse this file and find the corresponding ts file to play. Therefore, generally in order to speed up, .m3u8 is placed on the web server and the ts file is placed on the cdn.

.m3u8 file is actually an m3u file encoded in UTF-8. This file itself cannot be played. It is just a text file that stores playback information:

#EXTM3U                     m3u文件头
#EXT-X-MEDIA-SEQUENCE       第一个TS分片的序列号
#EXT-X-TARGETDURATION       每个分片TS的最大的时长
#EXT-X-ALLOW-CACHE          是否允许cache
#EXT-X-ENDLIST              m3u8文件结束符
#EXTINF                     指定每个媒体段(ts)的持续时间（秒），仅对其后面的URI有效
mystream-12.ts

ts file:

HLS request process is:
1 HTTP request m3u8 url
2 The server returns an m3u8 playlist, which is updated in real time , generally giving the URLs of 3 pieces of data at a time
3 The client parses the m3u8 playlist, and then requests the URLs of each piece in sequence to obtain the ts data stream

Simple process:

3 HLS live broadcast delay

When we know that the hls protocol divides the live stream into short videos for downloading and playing, so Assume that the list contains 5 TS files, and each TS file contains 5 seconds of video content, then the overall delay is 25 seconds. Because when you see these videos, the host has already recorded the video and uploaded it, so there is a delay caused by this. Of course, the length of the list and the size of a single TS file can be shortened to reduce the delay. In the extreme, the length of the list can be reduced to 1 and the duration of the TS to 1s. However, this will increase the number of requests and increase the pressure on the server. When the network speed is slow, Timing causes more buffering, so Apple’s officially recommended ts duration is 10s, so this will cause a 30s delay. Reference: https://developer.apple.com/library/ios/documentation/NetworkingInternet/Conceptual/StreamingMediaGuide/FrequentlyAskedQuestions/FrequentlyAskedQuestions.html

4 The entire video live broadcast What is the process?

Live video can be roughly divided into:

1 Video recording end: usually the audio and video input device on the computer or the camera or microphone on the mobile phone, which has been moved at present Mainly mobile phone videos.

2 Video player: It can be the player on the computer, the native player on the mobile phone, and the h5 video tag, etc. Currently, the native player on the mobile phone is the main one. .

3 Video server: usually an nginx server, used to accept the video source provided by the video recording end and provide streaming services to the video playback end.

Simple process:

5 How to collect audio and video?

Let’s first clarify a few concepts:

Video encoding: The so-called video encoding refers to converting a file in a certain video format into another through specific compression technology. A video format file method. The video we use to record on the iPhone must be encoded, uploaded, and decoded before it can actually be played in the user-side player.

编解码标准：视频流传输中最为重要的编解码标准有国际电联的H.261、H.263、H.264，其中HLS协议支持H.264格式的编码。
音频编码：同视频编码类似，将原始的音频流按照一定的标准进行编码，上传，解码，同时在播放器里播放，当然音频也有许多编码标准，例如PCM编码，WMA编码，AAC编码等等，这里我们HLS协议支持的音频编码方式是AAC编码。

下面将利用ios上的摄像头，进行音视频的数据采集，主要分为以下几个步骤：

1 音视频的采集，ios中，利用AVCaptureSession和AVCaptureDevice可以采集到原始的音视频数据流。
2 对视频进行H264编码，对音频进行AAC编码，在ios中分别有已经封装好的编码库来实现对音视频的编码。
3 对编码后的音、视频数据进行组装封包；
4 建立RTMP连接并上推到服务端。

ps：由于编码库大多使用c语言编写，需要自己使用时编译，对于ios，可以使用已经编译好的编码库。

x264编码：https://github.com/kewlbear/x264-ios

faac编码：https://github.com/fflydev/faac-ios-build

ffmpeg编码：https://github.com/kewlbear/FFmpeg-iOS-build-script

关于如果想给视频增加一些特殊效果，例如增加滤镜等，一般在编码前给使用滤镜库，但是这样也会造成一些耗时，导致上传视频数据有一定延时。

简单流程：

6 前面提到的ffmpeg是什么？

和之前的x264一样，ffmpeg其实也是一套编码库，类似的还有Xvid，Xvid是基于MPEG4协议的编解码器，x264是基于H.264协议的编码器，ffmpeg集合了各种音频，视频编解码协议，通过设置参数可以完成基于MPEG4,H.264等协议的编解码，demo这里使用的是x264编码库。

7 什么是RTMP？

Real Time Messaging Protocol（简称 RTMP）是 Macromedia 开发的一套视频直播协议，现在属于 Adobe。和HLS一样都可以应用于视频直播，区别是RTMP基于flash无法在ios的浏览器里播放，但是实时性比HLS要好。所以一般使用这种协议来上传视频流，也就是视频流推送到服务器。

这里列举一下hls和rtmp对比：

8 推流

简所谓推流，就是将我们已经编码好的音视频数据发往视频流服务器中，一般常用的是使用rtmp推流，可以使用第三方库librtmp-iOS进行推流，librtmp封装了一些核心的api供使用者调用，如果觉得麻烦，可以使用现成的ios视频推流sdk，也是基于rtmp的，https://github.com/runner365/LiveVideoCoreSDK

9 推流服务器搭建

简简单的推流服务器搭建，由于我们上传的视频流都是基于rtmp协议的，所以服务器也必须要支持rtmp才行，大概需要以下几个步骤：

1 安装一台nginx服务器。

2 安装nginx的rtmp扩展，目前使用比较多的是https://github.com/arut/nginx-rtmp-module

3 配置nginx的conf文件：

rtmp {  
  
    server {  
  
        listen 1935;  #监听的端口
  
        chunk_size 4000;  
        
         
        application hls {  #rtmp推流请求路径
            live on;  
            hls on;  
            hls_path /usr/local/var/www/hls;  
            hls_fragment 5s;  
        }  
    }  
}

4 重启nginx，将rtmp的推流地址写为rtmp://ip:1935/hls/mystream，其中hls_path表示生成的.m3u8和ts文件所存放的地址，hls_fragment表示切片时长，mysteam表示一个实例，即将来要生成的文件名可以先自己随便设置一个。更多配置可以参考：https://github.com/arut/nginx-rtmp-module/wiki/

根据以上步骤基本上已经实现了一个支持rtmp的视频服务器了。

10 在html5页面进行播放直播视频？

简单来说，直接使用video标签即可播放hls协议的直播视频：

<video autoplay webkit-playsinline>  
    <source src="http://10.66.69.77:8080/hls/mystream.m3u8" type="application/vnd.apple.mpegurl" />  
    <p class="warning">Your browser does not support HTML5 video.</p>  
</video>

需要注意的是，给video标签增加webkit-playsinline属性，这个属性是为了让video视频在ios的uiwebview里面可以不全屏播放，默认ios会全屏播放视频，需要给uiwebview设置allowsInlineMediaPlayback＝YES。业界比较成熟的videojs，可以根据不同平台选择不同的策略，例如ios使用video标签，pc使用flash等。

11 坑点总结

简根据以上步骤，笔者写了一个demo，从实现ios视频录制，采集，上传，nginx服务器下发直播流，h5页面播放直播视频者一整套流程，总结出以下几点比较坑的地方：

1 在使用AVCaptureSession进行采集视频时，需要实现AVCaptureVideoDataOutputSampleBufferDelegate协议，同时在- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection捕获到视频流，要注意的是didOutputSampleBuffer这个方法不是didDropSampleBuffer方法，后者只会触发一次，当时开始写的是didDropSampleBuffer方法，差了半天才发现方法调用错了。

2 在使用rtmp推流时，rmtp地址要以rtmp://开头，ip地址要写实际ip地址，不要写成localhost，同时要加上端口号，因为手机端上传时是无法识别localhost的。

这里后续会补充上一些坑点，有的需要贴代码，这里先列这么多。

12 业界支持

目前，腾讯云，百度云，阿里云都已经有了基于视频直播的解决方案，从视频录制到视频播放，推流，都有一系列的sdk可以使用，缺点就是需要收费，如果可以的话，自己实现一套也并不是难事哈。

demo地址：https://github.com/lvming6816077/LMVideoTest/