ffmpeg推流本地MP4 h264视频文件到rtsp服务器，rtsp客户端NVDECODE无法解码

问题描述：
最近在音视频遇到一个问题，因为需要，必须采用英伟达的NVDEC解码接受到的rtsp流，刚开始从摄像头拉流，解码非常正常，后来摄像头没了，就采用ffmpeg+ZLMediaKit进行rtsp推流，然后拉流解码，但是但是，结果突然就无法解码了。

解决：
首先想到的第一个是推的流有问题，结果用VLC又能打开。
第二个想到的是bsf的h264_mp4toannexb转换有误，
参考：https://github.com/gongluck/AnalysisAVP#%E6%B5%81%E5%AA%92%E4%BD%93%E5%8D%8F%E8%AE%AE
将bsf转换后的流打印出来，发现也没有问题，有startcodec ，但是惊奇的发现关键帧前面没有sps，pps。果然问题在这里，那么我们在将其送入到解码器之前，在关键帧之前加入sps和pps就可以了。

关于这个问题网上已经有人给了解决方案：
从mp4,flv文件中解析出h264和aac,送解码器解码失败
https://www.cnblogs.com/lihaiping/p/5285166.html

然后将摄像头找回来做同样的事情，用分析工具一览：
这个工具非常好，可以从网上找到如下：
可以发现摄像头的h264流是自带sps,pps,sei的。

特别注意67 sps 68 pps 106 sei 61

但是如何将这些信息补充上去呢，其实前面说了ffmpeg提供了bit stram filter函数：

av_bsf_get_by_name("h264_mp4toannexb")
av_bsf_alloc（）
av_bsf_init（）
av_bsf_send_packet（）
av_bsf_receive_packet（）

这组函数通过一个过滤器h264_mp4toannexb_filter，将AVCodecParameters中extradata中的数据转换为sps,pps,sei后补充上去。
补充上去之后，英伟达解码器就可以正常解码了。
参考：https://www.cnblogs.com/nsnow/p/3862709.html

那么我这里也是正常加载了过滤器h264_mp4toannexb，为什么还是没有sps，pps，sei呢？继续查找原因：
原因是推流过来的rtsp流已经进行过h264_mp4toannexb转换了，所以bsf过滤器失效了，但是流中又没有在IDR帧中插入sps pps 等信息，所以手动插入：

auto nbsfRet = av_bsf_receive_packet(m_bsfc, &m_pktFiltered);
//在其后边加入
appendPPSandSPS(m_pktFiltered.data, m_pktFiltered.size,
                    m_bsfc->par_in->extradata, m_bsfc->par_in->extradata_size);
//m_bsfc->par_in->extradata已经就是pps sps了
//0 00 00 01 67 64 00 32 AC 2C 6A 80 A0 02 D6 9B 80 80 80 A0 00 00 E1 00 00 2B F2 00 80 00 00 00 01 68 EE 3C B0
                    
bool appendPPSandSPS(uint8_t *src, unsigned int len1,
                                   uint8_t *inf, unsigned int len2) {
  bool ret = false;
  uint8_t *hdr = nullptr;
  if ((len1 > 5 && src[0] == 0 && src[1] == 0 && src[2] == 0 && src[3] == 1) &&
      (len2 > 5 && inf[0] == 0 && inf[1] == 0 && inf[2] == 0 && inf[3] == 67)) {
    src = (uint8_t *)realloc(src, len1 + len2);
    hdr = src;
    memmove(src + len2, src, len1);
    memcpy(hdr, inf, len2);
    ret = true;
  }
  return ret;
}

这里说一下，m_bsfc->par_in这个是AVCodecParameters，这个是输入流的解码参数，在创建avformat_open_input（）中获取到的codecpar,它里面的extradata存放的就是sps,pps，其实它还有一个参数siede_data可以参考：https://blog.csdn.net/bolitongyue/article/details/109053503

使用RTP传输H264的时候,需要用到sdp协议描述,其中有两项:Sequence Parameter Sets (SPS) 和Picture Parameter Set (PPS)需要用到,那么这两项从哪里获取呢?答案是从H264码流中获取.在H264码流中,都是以"0x00 0x00 0x01"或者"0x00 0x00 0x00 0x01"为开始码的,找到开始码之后,使用开始码之后的第一个字节的低5位判断是否为7(sps)或者8(pps), 及data[4] & 0x1f == 7 || data[4] & 0x1f == 8.然后对获取的nal去掉开始码之后进行base64编码,得到的信息就可以用于sdp.sps和pps需要用逗号分隔开来.

SDP中的H.264的SPS和PPS串，包含了初始化H.264解码器所需要的信息参数，包括编码所用的profile，level，图像的宽和高，deblock滤波器等。
sdp中正常是包含sps和pps内容的，但是有时候sdp中没有sps和pps这些内容，这时候就要从流中获取。
详细看这里：https://blog.csdn.net/Jody1989/article/details/46127561

H264码流中SPS PPS详解
https://zhuanlan.zhihu.com/p/27896239

在这里看一个sdp会话

v=0
o=- 0 0 IN IP4 192.168.1.112
s=Stream-0
i=N/A
c=IN IP4 192.168.1.104
t=0 0
a=recvonly
m=video 5006 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=42c016;sprop-parameter-sets=Z0LAFqtAUB7QgAAAAwCAAAAPR4sXUA==,aM48gA==;
a=control:trackID=1
————————————————
版权声明：本文为CSDN博主「Devil_Lee」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。
原文链接：https://blog.csdn.net/devil__lee/article/details/9717471

其中主要的3个参数即为 profile-level-id sprop-parameter-sets

其中sprop-parameter-sets 包含sps和pps的信息，以逗号隔开。

那么这3个数值从哪里获取？有什么含义？

可以参考：http://www.cnblogs.com/skyseraph/archive/2012/04/01/2429384.html

通过这段时间的研究发现，sprop-parameter-sets 后面的数值为码流中sps和pps获取的值经过base64编码以后的数值，

profile-level-id为sps数值67以后的3个字节对应的16进制字符串

那么获取了sps和pps的数值以后就可以获取sdp文件的关键信息了

如果采用ffmpeg拉流，ffmpeg会解析sps和pps,然后存放在AVFormatContext中streams[0]中的codecpar中的extradata中。

有兴趣继续看一下过滤函数吧。
在 av_bsf_init(bsfcCtx)的时候，有一个非常重要的参数：
AVBSFContext *bsfcCtx

typedef struct AVBSFContext {
    /**
     * A class for logging and AVOptions
     */
    const AVClass *av_class;

    /**
     * The bitstream filter this context is an instance of.
     */
    const struct AVBitStreamFilter *filter;

    /**
     * Opaque libavcodec internal data. Must not be touched by the caller in any
     * way.
     */
    AVBSFInternal *internal;

    /**
     * Opaque filter-specific private data. If filter->priv_class is non-NULL,
     * this is an AVOptions-enabled struct.
     */
    void *priv_data;//H264BSFContext这个结构体

    /**
     * Parameters of the input stream. This field is allocated in
     * av_bsf_alloc(), it needs to be filled by the caller before
     * av_bsf_init().
     */
    AVCodecParameters *par_in;//这个参数中存储的就是输入流的AVCodecParameters

    /**
     * Parameters of the output stream. This field is allocated in
     * av_bsf_alloc(), it is set by the filter in av_bsf_init().
     */
    AVCodecParameters *par_out;

    /**
     * The timebase used for the timestamps of the input packets. Set by the
     * caller before av_bsf_init().
     */
    AVRational time_base_in;

    /**
     * The timebase used for the timestamps of the output packets. Set by the
     * filter in av_bsf_init().
     */
    AVRational time_base_out;
} AVBSFContext;

我们看看它的创建：

const AVBitStreamFilter *bsf;
bsf = av_bsf_get_by_name("h264_mp4toannexb");    if (!bsf) {
        LOG_INFO("h264_mp4toannexb bitstream filter get failed");
        return false;
      }

const AVBitStreamFilter *av_bsf_get_by_name(const char *name)
{
    const AVBitStreamFilter *f = NULL;
    void *i = 0;

    if (!name)
        return NULL;
//遍历找到fileter,那么我们有多少过滤器呢
    while ((f = av_bsf_iterate(&i))) {//遍历返回一个bsf过滤器
        if (!strcmp(f->name, name))
            return f;
    }

    return NULL;
}
//
//ffmpeg有这么多过滤器
static const AVBitStreamFilter * const bitstream_filters[] = {
    &ff_aac_adtstoasc_bsf,
    &ff_av1_frame_split_bsf,
    &ff_av1_metadata_bsf,
    &ff_chomp_bsf,
    &ff_dump_extradata_bsf,
    &ff_dca_core_bsf,
    &ff_eac3_core_bsf,
    &ff_extract_extradata_bsf,
    &ff_filter_units_bsf,
    &ff_h264_metadata_bsf,
    &ff_h264_mp4toannexb_bsf,
    &ff_h264_redundant_pps_bsf,
    &ff_hapqa_extract_bsf,
    &ff_hevc_metadata_bsf,
    &ff_hevc_mp4toannexb_bsf,
    &ff_imx_dump_header_bsf,
    &ff_mjpeg2jpeg_bsf,
    &ff_mjpega_dump_header_bsf,
    &ff_mp3_header_decompress_bsf,
    &ff_mpeg2_metadata_bsf,
    &ff_mpeg4_unpack_bframes_bsf,
    &ff_mov2textsub_bsf,
    &ff_noise_bsf,
    &ff_null_bsf,
    &ff_prores_metadata_bsf,
    &ff_remove_extradata_bsf,
    &ff_text2movsub_bsf,
    &ff_trace_headers_bsf,
    &ff_truehd_core_bsf,
    &ff_vp9_metadata_bsf,
    &ff_vp9_raw_reorder_bsf,
    &ff_vp9_superframe_bsf,
    &ff_vp9_superframe_split_bsf,
    NULL };

/

auto ret=  av_bsf_alloc(bsf, &m_bsfc);
int av_bsf_alloc(const AVBitStreamFilter *filter, AVBSFContext **pctx)
{
    AVBSFContext *ctx;
    int ret;

    ctx = av_mallocz(sizeof(*ctx));
    if (!ctx)
        return AVERROR(ENOMEM);

    ctx->av_class = &bsf_class;
    //这里进行过滤器赋值
    ctx->filter   = filter;

    ctx->par_in  = avcodec_parameters_alloc();
    ctx->par_out = avcodec_parameters_alloc();
    if (!ctx->par_in || !ctx->par_out) {
        ret = AVERROR(ENOMEM);
        goto fail;
    }

    ctx->internal = av_mallocz(sizeof(*ctx->internal));
    if (!ctx->internal) {
        ret = AVERROR(ENOMEM);
        goto fail;
    }

    ctx->internal->buffer_pkt = av_packet_alloc();
    if (!ctx->internal->buffer_pkt) {
        ret = AVERROR(ENOMEM);
        goto fail;
    }

    av_opt_set_defaults(ctx);

    /* allocate priv data and init private options */
    if (filter->priv_data_size) {
    //这里存储的一般是option参数等
        ctx->priv_data = av_mallocz(filter->priv_data_size);
        if (!ctx->priv_data) {
            ret = AVERROR(ENOMEM);
            goto fail;
        }
        //将filter中的priv_class赋值为ctx中的priv_data
        if (filter->priv_class) {
            *(const AVClass **)ctx->priv_data = filter->priv_class;
            av_opt_set_defaults(ctx->priv_data);
        }
    }

    *pctx = ctx;
    return 0;
fail:
    av_bsf_free(&ctx);
    return ret;
}

接下来进行par_in赋值

//这里主要进行的就是extradata的拷贝
avcodec_parameters_copy(m_bsfc->par_in, m_param.codecParams);
int avcodec_parameters_copy(AVCodecParameters *dst, const AVCodecParameters *src)
{
    codec_parameters_reset(dst);
    memcpy(dst, src, sizeof(*dst));

    dst->extradata      = NULL;
    dst->extradata_size = 0;
    if (src->extradata) {
    //这里存储的就是sps以及pps
    //这些数据是ffmpeg从rtsp中的sdp中解析出来的，有时候sdp中没有sps和pps信息，那么说明流中包含了sps和pps
    //如果sdp中包含了sps和pps那么流中就不包含sps和pps
    //00 00 00 01 67 64 00 32 AC D9 40 28 00 B5 A6 C8 00 00 03 00 08 00 00 03 01 90 78 C1 8C B0 
    //00 00 00 01 68 EB E3 CB 22 C0 
        dst->extradata = av_mallocz(src->extradata_size + AV_INPUT_BUFFER_PADDING_SIZE);
        if (!dst->extradata)
            return AVERROR(ENOMEM);
        memcpy(dst->extradata, src->extradata, src->extradata_size);
        dst->extradata_size = src->extradata_size;
    }

    return 0;
}

其中codecParams参数如下：
接下来进行av_bsf_init，这个时候bsfc的值如下，其中initernal是存放输入packet的。

av_bsf_init(m_bsfc);
//检查了一下filter中编解码id（AV_CODEC_ID_H264）是否支持，然后对par_out赋值
int av_bsf_init(AVBSFContext *ctx)
{
    int ret, i;

    /* check that the codec is supported */
   /
   static void h264_mp4toannexb_flush(AVBSFContext *ctx)
{
    H264BSFContext *s = ctx->priv_data;

    s->idr_sps_seen = 0;
    s->idr_pps_seen = 0;
    s->new_idr      = s->extradata_parsed;
}

static const enum AVCodecID codec_ids[] = {
    AV_CODEC_ID_H264, AV_CODEC_ID_NONE,
};

const AVBitStreamFilter ff_h264_mp4toannexb_bsf = {
    .name           = "h264_mp4toannexb",
    .priv_data_size = sizeof(H264BSFContext),
    .init           = h264_mp4toannexb_init,
    .filter         = h264_mp4toannexb_filter,
    .flush          = h264_mp4toannexb_flush,
    .codec_ids      = codec_ids,
};
  
    if (ctx->filter->codec_ids) {
    //查看输入流的code_id是否被filter所支持
        for (i = 0; ctx->filter->codec_ids[i] != AV_CODEC_ID_NONE; i++)
            if (ctx->par_in->codec_id == ctx->filter->codec_ids[i])
                break;
         //如果遍历到了最后，那就说明不支持
        if (ctx->filter->codec_ids[i] == AV_CODEC_ID_NONE) {
            const AVCodecDescriptor *desc = avcodec_descriptor_get(ctx->par_in->codec_id);
            av_log(ctx, AV_LOG_ERROR, "Codec '%s' (%d) is not supported by the "
                   "bitstream filter '%s'. Supported codecs are: ",
                   desc ? desc->name : "unknown", ctx->par_in->codec_id, ctx->filter->name);
            for (i = 0; ctx->filter->codec_ids[i] != AV_CODEC_ID_NONE; i++) {
                desc = avcodec_descriptor_get(ctx->filter->codec_ids[i]);
                av_log(ctx, AV_LOG_ERROR, "%s (%d) ",
                       desc ? desc->name : "unknown", ctx->filter->codec_ids[i]);
            }
            av_log(ctx, AV_LOG_ERROR, "\n");
            return AVERROR(EINVAL);
        }
    }

    /* initialize output parameters to be the same as input
     * init below might overwrite that */
    ret = avcodec_parameters_copy(ctx->par_out, ctx->par_in);
    if (ret < 0)
        return ret;

    ctx->time_base_out = ctx->time_base_in;

    if (ctx->filter->init) {
    //h264_mp4toannexb_init
        ret = ctx->filter->init(ctx);
        if (ret < 0)
            return ret;
    }

    return 0;
}

static int h264_mp4toannexb_init(AVBSFContext *ctx)
{
    H264BSFContext *s = ctx->priv_data;
    //这里将sps pps等数据大小拷贝一下
    int extra_size = ctx->par_in->extradata_size;
    int ret;

    /* retrieve sps and pps NAL units from extradata */
    //在这里流程中The input looks like it is Annex B already，也就是
    //输入的extra_data已经是Annex B了
    /
#ifndef AV_RB24
#   define AV_RB24(x)                           \
    ((((const uint8_t*)(x))[0] << 16) |         \
     (((const uint8_t*)(x))[1] <<  8) |         \
      ((const uint8_t*)(x))[2])
#endif

#ifndef AV_RB32
#   define AV_RB32(x)                                \
    (((uint32_t)((const uint8_t*)(x))[0] << 24) |    \
               (((const uint8_t*)(x))[1] << 16) |    \
               (((const uint8_t*)(x))[2] <<  8) |    \
                ((const uint8_t*)(x))[3])
#endif
    
    //这里在做什么 AV_RB24
    //0x00 00 01
    //0x00 00 00 00 01
    //查看是否是上述包头
    if (!extra_size                                               ||
        (extra_size >= 3 && AV_RB24(ctx->par_in->extradata) == 1) ||
        (extra_size >= 4 && AV_RB32(ctx->par_in->extradata) == 1)) {
        av_log(ctx, AV_LOG_VERBOSE,
               "The input looks like it is Annex B already\n");
    } else if (extra_size >= 6) {
        ret = h264_extradata_to_annexb(ctx, AV_INPUT_BUFFER_PADDING_SIZE);
        if (ret < 0)
            return ret;

        s->length_size      = ret;
        s->new_idr          = 1;
        s->idr_sps_seen     = 0;
        s->idr_pps_seen     = 0;
        s->extradata_parsed = 1;
    } else {
        av_log(ctx, AV_LOG_ERROR, "Invalid extradata size: %d\n", extra_size);
        return AVERROR_INVALIDDATA;
    }

    return 0;
}

下面看一下h264_extradata_to_annexb
总的来说H264的码流的打包方式有两种,一种为annex-b byte stream format的格式，这个是绝大部分编码器的默认输出格式，就是每个帧的开头的3~4个字节是H264的start_code,0x00000001或者0x000001。
另一种是原始的NAL打包格式，就是开始的若干字节（1，2，4字节）是NAL的长度，而不是start_code,此时必须借助某个全局的数据来获得编码器的profile,level,PPS,SPS等信息才可以解码。

我一直疑问为什么有些视频解码时显示格式是:H264，大部分又是：AVC1
我在搜索编程资料时在微软的msdn上发现的：
原文:http://msdn.microsoft.com/en-us/library/dd757808(v=vs.85).aspx
FOURCC:AVC1 描述:H.264 bitstream without start codes.
FOURCC:H264 描述:H.264 bitstream with start codes.
mp4 文件中的h264 avc1格式介绍
https://blog.csdn.net/haima1998/article/details/50426944/

下面这篇文章很详细的介绍了h264_extradata_to_annexb的作用
ffmpeg 从mp4上提取H264的nalu
https://blog.csdn.net/gavinr/article/details/7183499

H264—MP4格式及在MP4文件中提取H264的SPS、PPS及码流
https://www.cnblogs.com/skyseraph/archive/2012/04/01/2429384.html

使用FFmpeg提取MP4中的H264和AAC
https://jiangdg.blog.csdn.net/article/details/102665541

static int h264_extradata_to_annexb(AVBSFContext *ctx, const int padding)
{
    H264BSFContext *s = ctx->priv_data;
    //
    typedef struct H264BSFContext {
    int32_t  sps_offset;
    int32_t  pps_offset;
    uint8_t  length_size;
    uint8_t  new_idr;
    uint8_t  idr_sps_seen;
    uint8_t  idr_pps_seen;
    int      extradata_parsed;
} H264BSFContext;
//
    uint16_t unit_size;
    uint64_t total_size                 = 0;
    uint8_t *out                        = NULL, unit_nb, sps_done = 0,
             sps_seen                   = 0, pps_seen = 0;
    const uint8_t *extradata            = ctx->par_in->extradata + 4;
    static const uint8_t nalu_header[4] = { 0, 0, 0, 1 };
    //0000 0011，因为nal包后5位是包的长度，这里计算出长度+1
    int length_size = (*extradata++ & 0x3) + 1; // retrieve length coded size

    s->sps_offset = s->pps_offset = -1;

    /* retrieve sps and pps unit(s) */
    //0x1f 0001 1111 下面这句就是提取 nal中的后5位
    unit_nb = *extradata++ & 0x1f; /* number of sps unit(s) */
    if (!unit_nb) {
        goto pps;
    } else {
        s->sps_offset = 0;
        sps_seen = 1;
    }

    while (unit_nb--) {
        int err;

        unit_size   = AV_RB16(extradata);
        total_size += unit_size + 4;
        if (total_size > INT_MAX - padding) {
            av_log(ctx, AV_LOG_ERROR,
                   "Too big extradata size, corrupted stream or invalid MP4/AVCC bitstream\n");
            av_free(out);
            return AVERROR(EINVAL);
        }
        if (extradata + 2 + unit_size > ctx->par_in->extradata + ctx->par_in->extradata_size) {
            av_log(ctx, AV_LOG_ERROR, "Packet header is not contained in global extradata, "
                   "corrupted stream or invalid MP4/AVCC bitstream\n");
            av_free(out);
            return AVERROR(EINVAL);
        }
        if ((err = av_reallocp(&out, total_size + padding)) < 0)
            return err;
        memcpy(out + total_size - unit_size - 4, nalu_header, 4);
        memcpy(out + total_size - unit_size, extradata + 2, unit_size);
        extradata += 2 + unit_size;
pps:
        if (!unit_nb && !sps_done++) {
            unit_nb = *extradata++; /* number of pps unit(s) */
            if (unit_nb) {
                s->pps_offset = total_size;
                pps_seen = 1;
            }
        }
    }

    if (out)
        memset(out + total_size, 0, padding);

    if (!sps_seen)
        av_log(ctx, AV_LOG_WARNING,
               "Warning: SPS NALU missing or invalid. "
               "The resulting stream may not play.\n");

    if (!pps_seen)
        av_log(ctx, AV_LOG_WARNING,
               "Warning: PPS NALU missing or invalid. "
               "The resulting stream may not play.\n");

    av_freep(&ctx->par_out->extradata);
    ctx->par_out->extradata      = out;
    ctx->par_out->extradata_size = total_size;

    return length_size;
}

接着就是循环处理了

auto ret = av_bsf_send_packet(m_bsfc, inPacket);
//这个函数的主要作用就是把pkt转到ctx->internal->buffer_pkt中
int av_bsf_send_packet(AVBSFContext *ctx, AVPacket *pkt)
{
    int ret;

    if (!pkt || (!pkt->data && !pkt->side_data_elems)) {
        ctx->internal->eof = 1;
        return 0;
    }

    if (ctx->internal->eof) {
        av_log(ctx, AV_LOG_ERROR, "A non-NULL packet sent after an EOF.\n");
        return AVERROR(EINVAL);
    }

    if (ctx->internal->buffer_pkt->data ||
        ctx->internal->buffer_pkt->side_data_elems)
        return AVERROR(EAGAIN);

    ret = av_packet_make_refcounted(pkt);
    if (ret < 0)
        return ret;
    av_packet_move_ref(ctx->internal->buffer_pkt, pkt);

    return 0;
}

auto nbsfRet = av_bsf_receive_packet(m_bsfc, &m_pktFiltered);
int av_bsf_receive_packet(AVBSFContext *ctx, AVPacket *pkt)
{
    return ctx->filter->filter(ctx, pkt);
}

static int h264_mp4toannexb_filter(AVBSFContext *ctx, AVPacket *out)
{
    H264BSFContext *s = ctx->priv_data;

    AVPacket *in;
    uint8_t unit_type;
    int32_t nal_size;
    uint32_t cumul_size    = 0;
    const uint8_t *buf;
    const uint8_t *buf_end;
    int            buf_size;
    int ret = 0, i;
//获取到pkt
    ret = ff_bsf_get_packet(ctx, &in);
    if (ret < 0)
        return ret;
/ff_bsf_get_packet
int ff_bsf_get_packet(AVBSFContext *ctx, AVPacket **pkt)
{
    AVBSFInternal *in = ctx->internal;
    AVPacket *tmp_pkt;

    if (in->eof)
        return AVERROR_EOF;

    if (!ctx->internal->buffer_pkt->data &&
        !ctx->internal->buffer_pkt->side_data_elems)
        return AVERROR(EAGAIN);

    tmp_pkt = av_packet_alloc();
    if (!tmp_pkt)
        return AVERROR(ENOMEM);

    *pkt = ctx->internal->buffer_pkt;
    ctx->internal->buffer_pkt = tmp_pkt;

    return 0;
}
/

    /* nothing to filter */
    //记得前面already annex B 这里就直接返回了
    if (!s->extradata_parsed) {
        av_packet_move_ref(out, in);
        av_packet_free(&in);
        return 0;
    }

    buf      = in->data;
    buf_size = in->size;
    buf_end  = in->data + in->size;

    do {
        ret= AVERROR(EINVAL);
        if (buf + s->length_size > buf_end)
            goto fail;

        for (nal_size = 0, i = 0; i<s->length_size; i++)
            nal_size = (nal_size << 8) | buf[i];

        buf += s->length_size;
        unit_type = *buf & 0x1f;

        if (nal_size > buf_end - buf || nal_size < 0)
            goto fail;

        if (unit_type == H264_NAL_SPS)
            s->idr_sps_seen = s->new_idr = 1;
        else if (unit_type == H264_NAL_PPS) {
            s->idr_pps_seen = s->new_idr = 1;
            /* if SPS has not been seen yet, prepend the AVCC one to PPS */
            if (!s->idr_sps_seen) {
                if (s->sps_offset == -1)
                    av_log(ctx, AV_LOG_WARNING, "SPS not present in the stream, nor in AVCC, stream may be unreadable\n");
                else {
                    if ((ret = alloc_and_copy(out,
                                         ctx->par_out->extradata + s->sps_offset,
                                         s->pps_offset != -1 ? s->pps_offset : ctx->par_out->extradata_size - s->sps_offset,
                                         buf, nal_size, 1)) < 0)
                        goto fail;
                    s->idr_sps_seen = 1;
                    goto next_nal;
                }
            }
        }

        /* if this is a new IDR picture following an IDR picture, reset the idr flag.
         * Just check first_mb_in_slice to be 0 as this is the simplest solution.
         * This could be checking idr_pic_id instead, but would complexify the parsing. */
        if (!s->new_idr && unit_type == H264_NAL_IDR_SLICE && (buf[1] & 0x80))
            s->new_idr = 1;

        /* prepend only to the first type 5 NAL unit of an IDR picture, if no sps/pps are already present */
        if (s->new_idr && unit_type == H264_NAL_IDR_SLICE && !s->idr_sps_seen && !s->idr_pps_seen) {
        //关键帧拷贝的时候同sps pps一起拷贝
            if ((ret=alloc_and_copy(out,
                               ctx->par_out->extradata, ctx->par_out->extradata_size,
                               buf, nal_size, 1)) < 0)
                goto fail;
            s->new_idr = 0;
        /* if only SPS has been seen, also insert PPS */
        } else if (s->new_idr && unit_type == H264_NAL_IDR_SLICE && s->idr_sps_seen && !s->idr_pps_seen) {
            if (s->pps_offset == -1) {
                av_log(ctx, AV_LOG_WARNING, "PPS not present in the stream, nor in AVCC, stream may be unreadable\n");
                if ((ret = alloc_and_copy(out, NULL, 0, buf, nal_size, 0)) < 0)
                    goto fail;
            } else if ((ret = alloc_and_copy(out,
                                        ctx->par_out->extradata + s->pps_offset, ctx->par_out->extradata_size - s->pps_offset,
                                        buf, nal_size, 1)) < 0)
                goto fail;
        } else {
            if ((ret=alloc_and_copy(out, NULL, 0, buf, nal_size, unit_type == H264_NAL_SPS || unit_type == H264_NAL_PPS)) < 0)
                goto fail;
            if (!s->new_idr && unit_type == H264_NAL_SLICE) {
                s->new_idr = 1;
                s->idr_sps_seen = 0;
                s->idr_pps_seen = 0;
            }
        }

next_nal:
        buf        += nal_size;
        cumul_size += nal_size + s->length_size;
    } while (cumul_size < buf_size);

    ret = av_packet_copy_props(out, in);
    if (ret < 0)
        goto fail;

fail:
    if (ret < 0)
        av_packet_unref(out);
    av_packet_free(&in);

    return ret;
}

typedef struct AVCodecParameters {
    /**
     * General type of the encoded data.
     */
    enum AVMediaType codec_type;
    /**
     * Specific type of the encoded data (the codec used).
     */
    enum AVCodecID   codec_id;
    /**
     * Additional information about the codec (corresponds to the AVI FOURCC).
     */
    uint32_t         codec_tag;

    /**
     * Extra binary data needed for initializing the decoder, codec-dependent.
     *
     * Must be allocated with av_malloc() and will be freed by
     * avcodec_parameters_free(). The allocated size of extradata must be at
     * least extradata_size + AV_INPUT_BUFFER_PADDING_SIZE, with the padding
     * bytes zeroed.
     * 0 00 00 01 67 64 00 32 AC D9 40 28 00 B5 A6 C8 00 00 03 00 08 00 00 03 01 90 78 C1 8C B0 00 00 00 01 68 EB E3 CB 22 C0 
     */
    uint8_t *extradata;//这个参数中存储就是pps sps
    /**
     * Size of the extradata content in bytes.
     */
    int      extradata_size;//

    /**
     * - video: the pixel format, the value corresponds to enum AVPixelFormat.
     * - audio: the sample format, the value corresponds to enum AVSampleFormat.
     */
    int format;

    /**
     * The average bitrate of the encoded data (in bits per second).
     */
    int64_t bit_rate;

    /**
     * The number of bits per sample in the codedwords.
     *
     * This is basically the bitrate per sample. It is mandatory for a bunch of
     * formats to actually decode them. It's the number of bits for one sample in
     * the actual coded bitstream.
     *
     * This could be for example 4 for ADPCM
     * For PCM formats this matches bits_per_raw_sample
     * Can be 0
     */
    int bits_per_coded_sample;

    /**
     * This is the number of valid bits in each output sample. If the
     * sample format has more bits, the least significant bits are additional
     * padding bits, which are always 0. Use right shifts to reduce the sample
     * to its actual size. For example, audio formats with 24 bit samples will
     * have bits_per_raw_sample set to 24, and format set to AV_SAMPLE_FMT_S32.
     * To get the original sample use "(int32_t)sample >> 8"."
     *
     * For ADPCM this might be 12 or 16 or similar
     * Can be 0
     */
    int bits_per_raw_sample;

    /**
     * Codec-specific bitstream restrictions that the stream conforms to.
     */
    int profile;
    int level;

    /**
     * Video only. The dimensions of the video frame in pixels.
     */
    int width;
    int height;

    /**
     * Video only. The aspect ratio (width / height) which a single pixel
     * should have when displayed.
     *
     * When the aspect ratio is unknown / undefined, the numerator should be
     * set to 0 (the denominator may have any value).
     */
    AVRational sample_aspect_ratio;

    /**
     * Video only. The order of the fields in interlaced video.
     */
    enum AVFieldOrder                  field_order;

    /**
     * Video only. Additional colorspace characteristics.
     */
    enum AVColorRange                  color_range;
    enum AVColorPrimaries              color_primaries;
    enum AVColorTransferCharacteristic color_trc;
    enum AVColorSpace                  color_space;
    enum AVChromaLocation              chroma_location;

    /**
     * Video only. Number of delayed frames.
     */
    int video_delay;

    /**
     * Audio only. The channel layout bitmask. May be 0 if the channel layout is
     * unknown or unspecified, otherwise the number of bits set must be equal to
     * the channels field.
     */
    uint64_t channel_layout;
    /**
     * Audio only. The number of audio channels.
     */
    int      channels;
    /**
     * Audio only. The number of audio samples per second.
     */
    int      sample_rate;
    /**
     * Audio only. The number of bytes per coded audio frame, required by some
     * formats.
     *
     * Corresponds to nBlockAlign in WAVEFORMATEX.
     */
    int      block_align;
    /**
     * Audio only. Audio frame size, if known. Required by some formats to be static.
     */
    int      frame_size;

    /**
     * Audio only. The amount of padding (in samples) inserted by the encoder at
     * the beginning of the audio. I.e. this number of leading decoded samples
     * must be discarded by the caller to get the original audio without leading
     * padding.
     */
    int initial_padding;
    /**
     * Audio only. The amount of padding (in samples) appended by the encoder to
     * the end of the audio. I.e. this number of decoded samples must be
     * discarded by the caller from the end of the stream to get the original
     * audio without any trailing padding.
     */
    int trailing_padding;
    /**
     * Audio only. Number of samples to skip after a discontinuity.
     */
    int seek_preroll;
} AVCodecParameters;

我们接着来分析：

static int h264_mp4toannexb_filter(AVBSFContext *ctx, AVPacket *out)
{
    H264BSFContext *s = ctx->priv_data;

    AVPacket *in;
    uint8_t unit_type;
    int32_t nal_size;
    uint32_t cumul_size    = 0;
    const uint8_t *buf;
    const uint8_t *buf_end;
    int            buf_size;
    int ret = 0, i;

    ret = ff_bsf_get_packet(ctx, &in);
    if (ret < 0)
        return ret;

    /* nothing to filter */
    if (!s->extradata_parsed) {
        av_packet_move_ref(out, in);
        av_packet_free(&in);
        return 0;
    }

    buf      = in->data;
    buf_size = in->size;
    buf_end  = in->data + in->size;

    do {
        ret= AVERROR(EINVAL);
        if (buf + s->length_size > buf_end)
            goto fail;

        for (nal_size = 0, i = 0; i<s->length_size; i++)
            nal_size = (nal_size << 8) | buf[i];

        buf += s->length_size;
        unit_type = *buf & 0x1f;

        if (nal_size > buf_end - buf || nal_size < 0)
            goto fail;

        if (unit_type == H264_NAL_SPS)
            s->idr_sps_seen = s->new_idr = 1;
        else if (unit_type == H264_NAL_PPS) {
            s->idr_pps_seen = s->new_idr = 1;
            /* if SPS has not been seen yet, prepend the AVCC one to PPS */
            if (!s->idr_sps_seen) {
                if (s->sps_offset == -1)
                    av_log(ctx, AV_LOG_WARNING, "SPS not present in the stream, nor in AVCC, stream may be unreadable\n");
                else {
                    if ((ret = alloc_and_copy(out,
                                         ctx->par_out->extradata + s->sps_offset,
                                         s->pps_offset != -1 ? s->pps_offset : ctx->par_out->extradata_size - s->sps_offset,
                                         buf, nal_size, 1)) < 0)
                        goto fail;
                    s->idr_sps_seen = 1;
                    goto next_nal;
                }
            }
        }

        /* if this is a new IDR picture following an IDR picture, reset the idr flag.
         * Just check first_mb_in_slice to be 0 as this is the simplest solution.
         * This could be checking idr_pic_id instead, but would complexify the parsing. */
        if (!s->new_idr && unit_type == H264_NAL_IDR_SLICE && (buf[1] & 0x80))
            s->new_idr = 1;

        /* prepend only to the first type 5 NAL unit of an IDR picture, if no sps/pps are already present */
        if (s->new_idr && unit_type == H264_NAL_IDR_SLICE && !s->idr_sps_seen && !s->idr_pps_seen) {
            if ((ret=alloc_and_copy(out,
                               ctx->par_out->extradata, ctx->par_out->extradata_size,
                               buf, nal_size, 1)) < 0)
                goto fail;
            s->new_idr = 0;
        /* if only SPS has been seen, also insert PPS */
        } else if (s->new_idr && unit_type == H264_NAL_IDR_SLICE && s->idr_sps_seen && !s->idr_pps_seen) {
            if (s->pps_offset == -1) {
                av_log(ctx, AV_LOG_WARNING, "PPS not present in the stream, nor in AVCC, stream may be unreadable\n");
                if ((ret = alloc_and_copy(out, NULL, 0, buf, nal_size, 0)) < 0)
                    goto fail;
            } else if ((ret = alloc_and_copy(out,
                                        ctx->par_out->extradata + s->pps_offset, ctx->par_out->extradata_size - s->pps_offset,
                                        buf, nal_size, 1)) < 0)
                goto fail;
        } else {
            if ((ret=alloc_and_copy(out, NULL, 0, buf, nal_size, unit_type == H264_NAL_SPS || unit_type == H264_NAL_PPS)) < 0)
                goto fail;
            if (!s->new_idr && unit_type == H264_NAL_SLICE) {
                s->new_idr = 1;
                s->idr_sps_seen = 0;
                s->idr_pps_seen = 0;
            }
        }

next_nal:
        buf        += nal_size;
        cumul_size += nal_size + s->length_size;
    } while (cumul_size < buf_size);

    ret = av_packet_copy_props(out, in);
    if (ret < 0)
        goto fail;

fail:
    if (ret < 0)
        av_packet_unref(out);
    av_packet_free(&in);

    return ret;
}