GB28181设备接入侧如何对接外部编码后音视频数据并实现预览播放

 技术背景

我们在对接GB28181设备接入模块的时候,遇到这样的技术诉求,好多开发者期望能提供编码后(H.264/H.265、AAC/PCMA)数据对接,确保外部采集设备,比如无人机类似回调过来的数据,直接通过模块,对接到GB28181平台侧,此外,还期望不支持或者内网没有外部网络权限的RTSP设备,也能间接接入到国标平台。

技术实现

编码后音视频数据

本文以Android平台为例,基于上述诉求,我们设计的接口如下,简单来说,GB28181交互流程不变,只要提供数据接入接口即可:

/**
     * 设置编码后视频数据(H.264)
     *
     * @param codec_id, H.264对应 1
     *
     * @param data 编码后的video数据
     *
     * @param size data length
     *
     * @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0.
     *
     * @param timestamp video timestamp
     *
     * @param pts Presentation Time Stamp, 显示时间戳
     *
     * @return {0} if successful
     */
    public native int SmartPublisherPostVideoEncodedData(long handle, int codec_id, ByteBuffer data, int size, int is_key_frame, long timestamp, long pts);

    /**
     * 设置编码后视频数据(H.264)
     *
     * @param codec_id, H.264对应 1
     *
     * @param data 编码后的video数据
     *
     *@param offset data的偏移
     *
     * @param size data length
     *
     * @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0.
     *
     * @param timestamp video timestamp
     *
     * @param pts Presentation Time Stamp, 显示时间戳
     *
     * @return {0} if successful
     */
    public native int SmartPublisherPostVideoEncodedDataV2(long handle, int codec_id,
                                                           ByteBuffer data, int offset, int size,
                                                           int is_key_frame, long timestamp, long pts,
                                                           byte[] sps, int sps_len,
                                                           byte[] pps, int pps_len);

    /**
     * 设置编码后视频数据(H.264),如需录制编码后的数据,用此接口,且设置实际宽高
     *
     * @param codec_id, H.264对应 1
     *
     * @param data 编码后的video数据
     *
     *@param offset data的偏移
     *
     * @param size data length
     *
     * @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0.
     *
     * @param timestamp video timestamp
     *
     * @param pts Presentation Time Stamp, 显示时间戳
     *
     * @param width, height: 编码后视频宽高
     *
     * @return {0} if successful
     */
    public native int SmartPublisherPostVideoEncodedDataV3(long handle, int codec_id,
                                                           ByteBuffer data, int offset, int size,
                                                           int is_key_frame, long timestamp, long pts,
                                                           byte[] sps, int sps_len,
                                                           byte[] pps, int pps_len,
                                                           int width, int height);

    /**
     * 设置音频数据(AAC/PCMA/PCMU/SPEEX)
     *
     * @param codec_id:
     *
     *  NT_MEDIA_CODEC_ID_AUDIO_BASE = 0x10000,
     *   NT_MEDIA_CODEC_ID_PCMA = NT_MEDIA_CODEC_ID_AUDIO_BASE,
     *   NT_MEDIA_CODEC_ID_PCMU,
     *   NT_MEDIA_CODEC_ID_AAC,
     *   NT_MEDIA_CODEC_ID_SPEEX,
     *   NT_MEDIA_CODEC_ID_SPEEX_NB,
     *   NT_MEDIA_CODEC_ID_SPEEX_WB,
     *   NT_MEDIA_CODEC_ID_SPEEX_UWB,
     *
     * @param data audio数据
     *
     * @param size data length
     *
     * @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0, audio忽略
     *
     * @param timestamp video timestamp
     *
     * @param parameter_info 用于AAC special config信息填充
     *
     * @param parameter_info_size parameter info size
     *
     * @return {0} if successful
     */
    public native int SmartPublisherPostAudioEncodedData(long handle, int codec_id, ByteBuffer data, int size, int is_key_frame, long timestamp,ByteBuffer parameter_info, int parameter_info_size);

    /**
     * 设置音频数据(AAC/PCMA/PCMU/SPEEX)
     *
     * @param codec_id:
     *
     *  NT_MEDIA_CODEC_ID_AUDIO_BASE = 0x10000,
     *   NT_MEDIA_CODEC_ID_PCMA = NT_MEDIA_CODEC_ID_AUDIO_BASE,
     *   NT_MEDIA_CODEC_ID_PCMU,
     *   NT_MEDIA_CODEC_ID_AAC,
     *   NT_MEDIA_CODEC_ID_SPEEX,
     *   NT_MEDIA_CODEC_ID_SPEEX_NB,
     *   NT_MEDIA_CODEC_ID_SPEEX_WB,
     *   NT_MEDIA_CODEC_ID_SPEEX_UWB,
     *
     * @param data audio数据
     *
     * @param offset data的偏移
     *
     * @param size data length
     *
     * @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0, audio忽略
     *
     * @param timestamp video timestamp
     *
     * @param parameter_info 用于AAC special config信息填充
     *
     * @param parameter_info_size parameter info size
     *
     * @return {0} if successful
     */
    public native int SmartPublisherPostAudioEncodedDataV2(long handle, int codec_id,
                                                           ByteBuffer data, int offset, int size,
                                                           int is_key_frame, long timestamp,
                                                           byte[] parameter_info, int parameter_info_size);


    /**
     * 设置音频数据(AAC/PCMA/PCMU/SPEEX)
     *
     * @param codec_id:
     *
     *  NT_MEDIA_CODEC_ID_AUDIO_BASE = 0x10000,
     *   NT_MEDIA_CODEC_ID_PCMA = NT_MEDIA_CODEC_ID_AUDIO_BASE,
     *   NT_MEDIA_CODEC_ID_PCMU,
     *   NT_MEDIA_CODEC_ID_AAC,
     *   NT_MEDIA_CODEC_ID_SPEEX,
     *   NT_MEDIA_CODEC_ID_SPEEX_NB,
     *   NT_MEDIA_CODEC_ID_SPEEX_WB,
     *   NT_MEDIA_CODEC_ID_SPEEX_UWB,
     *
     * @param data audio数据
     *
     * @param offset data的偏移
     *
     * @param size data length
     *
     * @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0, audio忽略
     *
     * @param timestamp video timestamp
     *
     * @param parameter_info 用于AAC special config信息填充
     *
     * @param parameter_info_size parameter info size
     *
     * @param sample_rate 采样率,如果需要录像的话必须传正确的值
     *
     *@param channels 通道数, 如果需要录像的话必须传正确的值, 一般是1或者2
     *
     * @return {0} if successful
     */
    public native int SmartPublisherPostAudioEncodedDataV3(long handle, int codec_id,
                                                           ByteBuffer data, int offset, int size,
                                                           int is_key_frame, long timestamp,
                                                           byte[] parameter_info, int parameter_info_size,
                                                           int sample_rate, int channels);

拉取RTSP流接入到GB28181平台

GB28181设备接入侧如何对接外部编码后音视频数据并实现预览播放_第1张图片

简单那来说,把摄像机的RTSP流数据拉下来,然后回调编码后的数据到上层,上层根据GB28181数据格式要求,实现PS打包,然后通过对接GB28181平台信令和数据交互,国标平台侧需要预览的时候,信令交互后,拉RTSP即可。

如何拉流:

private boolean StartPull()
  {
    if ( isPulling )
      return false;

    if (!OpenPullHandle())
      return false;

    libPlayer.SmartPlayerSetAudioDataCallback(playerHandle, new PlayerAudioDataCallback());
    libPlayer.SmartPlayerSetVideoDataCallback(playerHandle, new PlayerVideoDataCallback());

    int is_pull_trans_code  = 1;
    libPlayer.SmartPlayerSetPullStreamAudioTranscodeAAC(playerHandle, is_pull_trans_code);

    int startRet = libPlayer.SmartPlayerStartPullStream(playerHandle);

    if (startRet != 0) {
      Log.e(TAG, "Failed to start pull stream!");

      if(!isPlaying && !isRecording && isPushing && !isRTSPPublisherRunning)
      {
        libPlayer.SmartPlayerClose(playerHandle);
        playerHandle = 0;
      }

      return false;
    }

    isPulling = true;
    return true;
  }

  private void StopPull()
  {
    if ( !isPulling )
      return;

    libPlayer.SmartPlayerStopPullStream(playerHandle);

    if ( !isPlaying && !isRecording && !isPushing && !isRTSPPublisherRunning)
    {
      libPlayer.SmartPlayerClose(playerHandle);
      playerHandle = 0;
    }

    isPulling = false;
  }

拉到的音视频数据,投递到GB28181接入模块:

class PlayerAudioDataCallback implements NTAudioDataCallback
  {
    private int audio_buffer_size = 0;
    private int param_info_size = 0;

    private ByteBuffer audio_buffer_ = null;
    private ByteBuffer parameter_info_ = null;

    @Override
    public ByteBuffer getAudioByteBuffer(int size)
    {
      //Log.i("getAudioByteBuffer", "size: " + size);

      if( size < 1 )
      {
        return null;
      }

      if ( size <= audio_buffer_size && audio_buffer_ != null )
      {
        return audio_buffer_;
      }

      audio_buffer_size = size + 512;
      audio_buffer_size = (audio_buffer_size+0xf) & (~0xf);

      audio_buffer_ = ByteBuffer.allocateDirect(audio_buffer_size);

      // Log.i("getAudioByteBuffer", "size: " + size + " buffer_size:" + audio_buffer_size);

      return audio_buffer_;
    }

    @Override
    public ByteBuffer getAudioParameterInfo(int size)
    {
      //Log.i("getAudioParameterInfo", "size: " + size);

      if(size < 1)
      {
        return null;
      }

      if ( size <= param_info_size &&  parameter_info_ != null )
      {
        return  parameter_info_;
      }

      param_info_size = size + 32;
      param_info_size = (param_info_size+0xf) & (~0xf);

      parameter_info_ = ByteBuffer.allocateDirect(param_info_size);

      //Log.i("getAudioParameterInfo", "size: " + size + " buffer_size:" + param_info_size);

      return parameter_info_;
    }

    public void onAudioDataCallback(int ret, int audio_codec_id, int sample_size, int is_key_frame, long timestamp, int sample_rate, int channel, int parameter_info_size, long reserve)
    {

      if ( audio_buffer_ == null)
        return;

      audio_buffer_.rewind();

      if ( ret == 0 && (isPushing || isRTSPPublisherRunning || isGB28181StreamRunning)) {
        libPublisher.SmartPublisherPostAudioEncodedData(publisherHandle, audio_codec_id, audio_buffer_, sample_size, is_key_frame, timestamp, parameter_info_, parameter_info_size);
      }
    }
  }

  class PlayerVideoDataCallback implements NTVideoDataCallback
  {
    private int video_buffer_size = 0;

    private ByteBuffer video_buffer_ = null;

    @Override
    public ByteBuffer getVideoByteBuffer(int size)
    {

      if( size < 1 )
      {
        return null;
      }

      if ( size <= video_buffer_size &&  video_buffer_ != null )
      {
        return  video_buffer_;
      }

      video_buffer_size = size + 1024;
      video_buffer_size = (video_buffer_size+0xf) & (~0xf);

      video_buffer_ = ByteBuffer.allocateDirect(video_buffer_size);

      return video_buffer_;
    }

    public void onVideoDataCallback(int ret, int video_codec_id, int sample_size, int is_key_frame, long timestamp, int width, int height, long presentation_timestamp)
    {
      if ( video_buffer_ == null)
        return;

      video_buffer_.rewind();

      if ( ret == 0 &&  (isPushing || isRTSPPublisherRunning || isGB28181StreamRunning) ) {

        libPublisher.SmartPublisherPostVideoEncodedData(publisherHandle, video_codec_id, video_buffer_, sample_size, is_key_frame, timestamp, presentation_timestamp);
      }
    }
  }

如何预览播放外部音视频数据?

除了想把编码后的音视频数据转至GB28181外,有些场景下,还需要本地预览甚至对数据做二次处理(视频分析、实时水印字符叠加等,然后二次编码),基于这样的场景诉求,我们实现了Android平台外部编码数据实时预览播放模块。

GB28181设备接入侧如何对接外部编码后音视频数据并实现预览播放_第2张图片

外部(H.264/H.265)投递接口设计如下:

    // SmartPlayerJniV2.java
    // Author: daniusdk.com
    /**
	 * 投递视频包给外部Live Source
	 *
	 * @param codec_id: 编码id, 当前仅支持H264和H265, 1:H264, 2:H265
	 *
	 * @param packet: 视频数据, ByteBuffer必须是DirectBuffer, 包格式请参考H264/H265 Annex B Byte stream format, 例如:
	 *                0x00000001 nal_unit 0x00000001 ...
	 *                H264 IDR: 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....
	 *                H265 IDR: 0x00000001 vps 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....
	 *
	 * @param offset: 偏移量
	 * @param size: packet size
	 * @param timestamp_ms: 时间戳, 单位毫秒
	 * @param is_timestamp_discontinuity: 是否时间戳间断,0:未间断,1:间断
	 * @param is_key: 是否是关键帧, 0:非关键帧, 1:关键帧
	 * @param extra_data: 可选参数,可传null, 对于H264关键帧包, 如果packet不含sps和pps, 可传0x00000001 sps 0x00000001 pps
	 *                    ,对于H265关键帧包, 如果packet不含vps,sps和pps, 可传0x00000001 vps 0x00000001 sps 0x00000001 pps
	 * @param extra_data_size: extra_data size
	 * @param width: 图像宽, 可传0
	 * @param height: 图像高, 可传0
	 *
	 * @return {0} if successful
	 */
	public native int PostVideoPacketByteBuffer(long handle, int codec_id,
									  java.nio.ByteBuffer packet, int offset, int size, long timestamp_ms, int is_timestamp_discontinuity, int is_key,
									  byte[] extra_data, int extra_data_size, int width, int height);


	/*
	* 请参考 PostVideoPacketByteBuffer说明
	 */
	public native int PostVideoPacketByteArray(long handle, int codec_id,
												byte[] packet, int offset, int size, long timestamp_ms, int is_timestamp_discontinuity, int is_key,
												byte[] extra_data, int extra_data_size, int width, int height);

PostVideoPacketByteBuffer()和PostVideoPacketByteArray()接口设计基本类似,唯一的区别在于,一个数据类型是ByteBuffer,一个是byte数组。

其中codec_id,系编码id,目前仅支持H.264和H.265类型。

 packet视频数据,需要注意的是,ByteBuffer必须是DirectBuffer, 包格式请参考H264/H265 Annex B Byte stream format, 例如:

0x00000001 nal_unit 0x00000001 ...
H264 IDR: 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....
H265 IDR: 0x00000001 vps 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....

extra_data: 可选参数,可传null, 对于H264关键帧包,如果packet不含sps和pps,可传0x00000001 sps 0x00000001 pps,对于H265关键帧包,如果packet不含vps,sps和pps, 可传0x00000001 vps 0x00000001 sps 0x00000001 pps

音频(AAC/PCMA/PCMU)投递接口设计如下:

    /**
	 * 投递音频包给外部Live source, 注意ByteBuffer对象必须是DirectBuffer
	 *
	 * @param handle: return value from SmartPlayerOpen()
	 *
	 * @param codec_id: 编码id, 当前支持PCMA、PCMU和AAC, 65536:PCMA, 65537:PCMU, 65538:AAC
	 * @param packet: 音频数据
	 * @param offset:packet偏移量
	 * @param size: packet size
	 * @param pts_ms: 时间戳, 单位毫秒
	 * @param is_pts_discontinuity: 是否时间戳间断,false:未间断,true:间断
	 * @param extra_data: 如果是AAC的话,需要传 Audio Specific Configuration
	 * @param extra_data_offset: extra_data 偏移量
	 * @param extra_data_size: extra_data size
	 * @param sample_rate: 采样率
	 * @param channels: 通道数
	 *
	 * @return {0} if successful
	 */
	public native int PostAudioPacket(long handle, int codec_id,
									  java.nio.ByteBuffer packet, int offset, int size, long pts_ms, boolean is_pts_discontinuity,
									  java.nio.ByteBuffer extra_data, int extra_data_offset, int extra_data_size, int sample_rate, int channels);

	/*
	* 投递音频包给外部Live source, byte数组版本, 具体请参考PostAudioPacket
	*
	* @param is_pts_discontinuity: 是否时间戳间断,0:未间断,1:间断
	* @return {0} if successful
	*/
	public native int PostAudioPacketByteArray(long handle, int codec_id,
											   byte[] packet, int offset, int size, long pts_ms, int is_pts_discontinuity,
											   byte[] extra_data, int extra_data_size, int sample_rate, int channels);

总结

通过以上描述,大家可以看到,GB/T 28181音视频数据源接入,无论是编码前还是编码后数据,或外部RTSP流数据,包括数据预览,如果有技术积累的话,实现起来也没那么麻烦,感兴趣的开发者,可以尝试看。

你可能感兴趣的:(GB28181接入,Android,GB28181,GB28181,RTSP,GB28181,HEVC,GB28181安卓端,大牛直播SDK)