目前主要的编码方式为h264,h265虽然更好，但是ios11以上才支持，并且cpu负荷比较大
-
硬编码：基于GPU
- 视频：VideoToolBox
- 音频：AudioToolBox
-
软编码：基于CPU
- 视频压缩：视频编码MPEG，H264
X264把视频原数据YUV/RGB编码H264 - 音频：AudioToolBox
fdk_aac将音频数据PCM转AAC
- 视频压缩：视频编码MPEG，H264
H264基本概念.
I帧: 关键帧,采用帧内压缩技术.
- 举个例子,如果摄像头对着你拍摄,1秒之内,实际你发生的变化是非常少的.1秒钟之内实际少很少有大幅度的变化.摄像机一般一秒钟会抓取几十帧的数据.比如像动画,就是25帧/s,一般视频文件都是在30帧/s左右.对于一些要求比较高的,对动作的精细度有要求,想要捕捉到完整的动作的,高级的摄像机一般是60帧/s.那些对于一组帧的它的变化很小.为了便于压缩数据,那怎么办了?将第一帧完整的保存下来.如果没有这个关键帧后面解码数据,是完成不了的.所以I帧特别关键.
P帧: 向前参考帧.压缩时只参考前一个帧.属于帧间压缩技术.
- 视频的第一帧会被作为关键帧完整保存下来.而后面的帧会向前依赖.也就是第二帧依赖于第一个帧.后面所有的帧只存储于前一帧的差异.这样就能将数据大大的减少.从而达到一个高压缩率的效果.
B帧: 双向参考帧,压缩时即参考前一帧也参考后一帧.帧间压缩技术.
- B帧,即参考前一帧,也参考后一帧.这样就使得它的压缩率更高.存储的数据量更小.如果B帧的数量越多,你的压缩率就越高.这是B帧的优点,但是B帧最大的缺点是,如果是实时互动的直播,那时与B帧就要参考后面的帧才能解码,那在网络中就要等待后面的帧传输过来.这就与网络有关了.如果网络状态很好的话,解码会比较快,如果网络不好时解码会稍微慢一些.丢包时还需要重传.对实时互动的直播,一般不会使用B帧.
- 如果在泛娱乐的直播中,可以接受一定度的延时,需要比较高的压缩比就可以使用B帧.
- 如果我们在实时互动的直播,我们需要提高时效性,这时就不能使用B帧了.
二. GOF(Group of Frame)一组帧
两个I帧之间形成的一组图片,就是GOP(Group of Picture).
通常在编码器设置参数时，必须会设置gop_ size 的值其实就是代表2个|帧之间的帧数目.在一个GOP组中容量最大的就是I帧.所以相对而言, gop_ size 设置的越大，整个视频画面质量就会越好.但是解码端必须从接收的第一个|帧开始才可以正确解码出原始图像.否则无法正确解码.
image
SPS/PPS
SPS/PPS实际上就是存储GOP的参数.
SPS: (Sequence Parameter Set,序列参数集)存放帧数,参考帧数目,解码图像尺寸,帧场编码模式选择标识等.
- 一组帧的参数集.
PPS:(Picture Parameter Set,图像参数集).存放熵编码模式选择标识,片组数目,初始量化参数和去方块滤波系数调整标识等.(与图像相关的信息)
在一组帧之前我们首先收到的是SPS/PPS数据.如果没有这组参数的话,我们是无法解码.
如果我们在解码时发生错误,首先要检查是否有SPS/PPS.如果没有,是因为对端没有发送过来还是因为对端在发送过程中丢失了.
SPS/PPS数据,我们也把其归类到I帧.这2组数据是绝对不能丢的.
那么下面我们来看一下实际开发中遇到的问题.
视频花屏/卡顿原因
我们在观看视频时,会遇到花屏或者卡顿现象.那这个与我们刚刚所讲的GOF就息息相关了.
- 如果GOP分组中的P帧丢失就会造成解码端的图像发生错误.
- 为了避免花屏问题的发生,一般如果发现P帧或者I帧丢失.就不显示本GOP内的所有帧.只到下一个I帧来后重新刷新图像.
- 当这时因为没有刷新屏幕.丢包的这一组帧全部扔掉了.图像就会卡在哪里不动.这就是卡顿的原因.
所以总结起来,花屏是因为你丢了P帧或者I帧.导致解码错误. 而卡顿是因为为了怕花屏,将整组错误的GOP数据扔掉了.直达下一组正确的GOP再重新刷屏.而这中间的时间差,就是我们所感受的卡顿.
VideoToolBox
在iOS4.0，苹果就已经支持硬编解码但是硬编解码在当时属于私有API.不提供给开发者使用在2014年的WWDC大会上,iOS 8.0之后，苹果开放了硬编解码的APl。就是VideoToolbox. framework的API。VideoToolbox 是一套纯C语言API。其中包含了很多C语言函数. VideoToolbox . framework是基于Core Foundation库函数,基于C语言
VideoToolBox框架的流程
- 创建session
- 设置编码相关参数
- 开始编码
- 循环获取采集数据
- 获取编码后数据.
- 将数据写入H264文件
编码的输入与输出.png
h264编码采集
#import
#import
@interface ViewController ()
@property(nonatomic,strong)UILabel *cLabel;
@property(nonatomic,strong)AVCaptureSession *cCapturesession;//捕捉会话，用于输入输出设备之间的数据传递
@property(nonatomic,strong)AVCaptureDeviceInput *cCaptureDeviceInput;//捕捉输入
@property(nonatomic,strong)AVCaptureVideoDataOutput *cCaptureDataOutput;//捕捉输出
@property(nonatomic,strong)AVCaptureVideoPreviewLayer *cPreviewLayer;//预览图层
@end
@implementation ViewController
{
int frameID; //帧ID
dispatch_queue_t cCaptureQueue; //捕获队列
dispatch_queue_t cEncodeQueue; //编码队列
VTCompressionSessionRef cEncodeingSession;//编码session
CMFormatDescriptionRef format; //编码格式
NSFileHandle *fileHandele;
}
- (void)viewDidLoad {
[super viewDidLoad];
// Do any additional setup after loading the view, typically from a nib.
//基础UI实现
_cLabel = [[UILabel alloc]initWithFrame:CGRectMake(20, 20, 200, 100)];
_cLabel.text = @"cc课堂之H.264硬编码";
_cLabel.textColor = [UIColor redColor];
[self.view addSubview:_cLabel];
UIButton *cButton = [[UIButton alloc]initWithFrame:CGRectMake(200, 20, 100, 100)];
[cButton setTitle:@"play" forState:UIControlStateNormal];
[cButton setTitleColor:[UIColor whiteColor] forState:UIControlStateNormal];
[cButton setBackgroundColor:[UIColor orangeColor]];
[cButton addTarget:self action:@selector(buttonClick:) forControlEvents:UIControlEventTouchUpInside];
[self.view addSubview:cButton];
}
-(void)buttonClick:(UIButton *)button
{
//判断_cCapturesession 和 _cCapturesession是否正在捕捉
if (!_cCapturesession || !_cCapturesession.isRunning ) {
//修改按钮状态
[button setTitle:@"Stop" forState:UIControlStateNormal];
//开始捕捉
[self startCapture];
}else
{
[button setTitle:@"Play" forState:UIControlStateNormal];
//停止捕捉
[self stopCapture];
}
}
//开始捕捉
- (void)startCapture
{
self.cCapturesession = [[AVCaptureSession alloc]init];
//设置捕捉分辨率
self.cCapturesession.sessionPreset = AVCaptureSessionPreset640x480;
//使用函数dispath_get_global_queue去得到队列
cCaptureQueue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
cEncodeQueue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
AVCaptureDevice *inputCamera = nil;
//获取iPhone视频捕捉的设备，例如前置摄像头、后置摄像头......
NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo];
for (AVCaptureDevice *device in devices) {
//拿到后置摄像头
if ([device position] == AVCaptureDevicePositionBack) {
inputCamera = device;
}
}
//将捕捉设备 封装成 AVCaptureDeviceInput 对象
self.cCaptureDeviceInput = [[AVCaptureDeviceInput alloc]initWithDevice:inputCamera error:nil];
//判断是否能加入后置摄像头作为输入设备
if ([self.cCapturesession canAddInput:self.cCaptureDeviceInput]) {
//将设备添加到会话中
[self.cCapturesession addInput:self.cCaptureDeviceInput];
}
//配置输出
self.cCaptureDataOutput = [[AVCaptureVideoDataOutput alloc]init];
//设置丢弃最后的video frame 为NO
[self.cCaptureDataOutput setAlwaysDiscardsLateVideoFrames:NO];
//设置video的视频捕捉的像素点压缩方式为 420
[self.cCaptureDataOutput setVideoSettings:[NSDictionary dictionaryWithObject:[NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange] forKey:(id)kCVPixelBufferPixelFormatTypeKey]];
//设置捕捉代理 和 捕捉队列
[self.cCaptureDataOutput setSampleBufferDelegate:self queue:cCaptureQueue];
//判断是否能添加输出
if ([self.cCapturesession canAddOutput:self.cCaptureDataOutput]) {
//添加输出
[self.cCapturesession addOutput:self.cCaptureDataOutput];
}
//创建连接
AVCaptureConnection *connection = [self.cCaptureDataOutput connectionWithMediaType:AVMediaTypeVideo];
//设置连接的方向
[connection setVideoOrientation:AVCaptureVideoOrientationPortrait];
//初始化图层
self.cPreviewLayer = [[AVCaptureVideoPreviewLayer alloc]initWithSession:self.cCapturesession];
//设置视频重力
[self.cPreviewLayer setVideoGravity:AVLayerVideoGravityResizeAspect];
//设置图层的frame
[self.cPreviewLayer setFrame:self.view.bounds];
//添加图层
[self.view.layer addSublayer:self.cPreviewLayer];
//文件写入沙盒
NSString *filePath = [[NSSearchPathForDirectoriesInDomains(NSDocumentDirectory,NSUserDomainMask,YES)lastObject]stringByAppendingPathComponent:@"cc_video.h264"];
// NSString *filePath = [NSHomeDirectory()stringByAppendingPathComponent:@"/Documents/cc_video.h264"];
//先移除已存在的文件
[[NSFileManager defaultManager] removeItemAtPath:filePath error:nil];
//新建文件
BOOL createFile = [[NSFileManager defaultManager] createFileAtPath:filePath contents:nil attributes:nil];
if (!createFile) {
NSLog(@"create file failed");
}else
{
NSLog(@"create file success");
}
NSLog(@"filePaht = %@",filePath);
fileHandele = [NSFileHandle fileHandleForWritingAtPath:filePath];
//初始化videoToolbBox
[self initVideoToolBox];
//开始捕捉
[self.cCapturesession startRunning];
}
//停止捕捉
- (void)stopCapture
{
//停止捕捉
[self.cCapturesession stopRunning];
//移除预览图层
[self.cPreviewLayer removeFromSuperlayer];
//结束videoToolbBox
[self endVideoToolBox];
//关闭文件
[fileHandele closeFile];
fileHandele = NULL;
}
#pragma mark - AVCaptureVideoDataOutputSampleBufferDelegate
//AV Foundation 获取到视频流
-(void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
//开始视频录制，获取到摄像头的视频帧，传入encode 方法中
dispatch_sync(cEncodeQueue, ^{
[self encode:sampleBuffer];
});
}
//初始化videoToolBox
-(void)initVideoToolBox
{
dispatch_sync(cEncodeQueue, ^{
frameID = 0;
int width = 480,height = 640;
//1.调用VTCompressionSessionCreate创建编码session
//参数1：NULL 分配器,设置NULL为默认分配
//参数2：width，像素为单位，如果此数据非法，编码会改为合理的值
//参数3：height
//参数4：编码类型,如kCMVideoCodecType_H264
//参数5：NULL encoderSpecification: 编码规范。设置NULL由videoToolbox自己选择
//参数6：NULL sourceImageBufferAttributes: 源像素缓冲区属性.设置NULL不让videToolbox创建,而自己创建
//参数7：NULL compressedDataAllocator: 压缩数据分配器.设置NULL,默认的分配
//参数8：回调 当VTCompressionSessionEncodeFrame被调用压缩一次后会被异步调用.注:当你设置NULL的时候,你需要调用VTCompressionSessionEncodeFrameWithOutputHandler方法进行压缩帧处理,支持iOS9.0以上
//参数9：outputCallbackRefCon: 回调客户定义的参考值
//参数10：compressionSessionOut: 编码会话变量
OSStatus status = VTCompressionSessionCreate(NULL, width, height, kCMVideoCodecType_H264, NULL, NULL, NULL, didCompressH264, (__bridge void *)(self), &cEncodeingSession);
NSLog(@"H264:VTCompressionSessionCreate:%d",(int)status);
if (status != 0) {
NSLog(@"H264:Unable to create a H264 session");
return ;
}
/*
VTSessionSetProperty(VTSessionRef _Nonnull session, CFStringRef _Nonnull propertyKey, CFTypeRef _Nullable propertyValue)
* 参数设置对象 cEncodeingSession
*/
//设置实时编码输出（避免延迟）
VTSessionSetProperty(cEncodeingSession, kVTCompressionPropertyKey_RealTime, kCFBooleanTrue);
//舍弃B帧
VTSessionSetProperty(cEncodeingSession, kVTCompressionPropertyKey_ProfileLevel,kVTProfileLevel_H264_Baseline_AutoLevel);
//是否产生B帧(因为B帧在解码时并不是必要的,是可以抛弃B帧的)
VTSessionSetProperty(cEncodeingSession, kVTCompressionPropertyKey_AllowFrameReordering, kCFBooleanFalse);
//设置关键帧（GOPsize）间隔，GOP太小的话图像会模糊，太大视频体积增大
int frameInterval = 10;
//VTSessionSetProperty 不能直接设置int/float 作为属性值
/*
CFNumberCreate(CFAllocatorRef allocator, CFNumberType theType, const void *valuePtr)
* allocator : 分配器，一般默认kCFAllocatorDefault
* theType : 数据类型
* *valuePtr : 地址
*/
CFNumberRef frameIntervalRaf = CFNumberCreate(kCFAllocatorDefault, kCFNumberIntType, &frameInterval);
VTSessionSetProperty(cEncodeingSession, kVTCompressionPropertyKey_MaxKeyFrameInterval, frameIntervalRaf);
//设置期望帧率，不是实际帧率
int fps = 10;
CFNumberRef fpsRef = CFNumberCreate(kCFAllocatorDefault, kCFNumberIntType, &fps);
VTSessionSetProperty(cEncodeingSession, kVTCompressionPropertyKey_ExpectedFrameRate, fpsRef);
//码率的理解：码率大了话就会非常清晰，但同时文件也会比较大。码率小的话，图像有时会模糊，但也勉强能看
//码率计算公式，参考印象笔记
//设置码率、上限、单位是bps
int bitRate = width * height * 3 * 4 * 8;
CFNumberRef bitRateRef = CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &bitRate);
VTSessionSetProperty(cEncodeingSession, kVTCompressionPropertyKey_AverageBitRate, bitRateRef);
//设置码率，均值，单位是byte
int bigRateLimit = width * height * 3 * 4;
CFNumberRef bitRateLimitRef = CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &bigRateLimit);
VTSessionSetProperty(cEncodeingSession, kVTCompressionPropertyKey_DataRateLimits, bitRateLimitRef);
//开始编码
VTCompressionSessionPrepareToEncodeFrames(cEncodeingSession);
});
}
- (void) encode:(CMSampleBufferRef )sampleBuffer
{
//拿到每一帧未编码数据
CVImageBufferRef imageBuffer = (CVImageBufferRef)CMSampleBufferGetImageBuffer(sampleBuffer);
//设置帧时间，如果不设置会导致时间轴过长。
CMTime presentationTimeStamp = CMTimeMake(frameID++, 1000);
//同步，异步
VTEncodeInfoFlags flags;
//参数1：编码会话变量
//参数2：未编码数据
//参数3：获取到的这个sample buffer数据的展示时间戳。每一个传给这个session的时间戳都要大于前一个展示时间戳.
//参数4：对于获取到sample buffer数据,这个帧的展示时间.如果没有时间信息,可设置kCMTimeInvalid.
//参数5：frameProperties: 包含这个帧的属性.帧的改变会影响后边的编码帧.一般为null
//参数6：ourceFrameRefCon: 回调函数会引用你设置的这个帧的参考值. null
//参数7：infoFlagsOut: 指向一个VTEncodeInfoFlags来接受一个编码操作.如果使用异步运行,kVTEncodeInfo_Asynchronous被设置；同步运行,kVTEncodeInfo_FrameDropped被设置；设置NULL为不想接受这个信息.
OSStatus statusCode = VTCompressionSessionEncodeFrame(cEncodeingSession, imageBuffer, presentationTimeStamp, kCMTimeInvalid, NULL, NULL, &flags);
if (statusCode != noErr) {
NSLog(@"H.264:VTCompressionSessionEncodeFrame faild with %d",(int)statusCode);
VTCompressionSessionInvalidate(cEncodeingSession);
CFRelease(cEncodeingSession);
cEncodeingSession = NULL;
return;
}
NSLog(@"H264:VTCompressionSessionEncodeFrame Success");
}
//编码完成回调
/*
1.H264硬编码完成后，回调VTCompressionOutputCallback
2.将硬编码成功的CMSampleBuffer转换成H264码流，通过网络传播
3.解析出参数集SPS & PPS，加上开始码组装成 NALU。提现出视频数据，将长度码转换为开始码，组成NALU，将NALU发送出去。
*/
void didCompressH264(void *outputCallbackRefCon, void *sourceFrameRefCon, OSStatus status, VTEncodeInfoFlags infoFlags, CMSampleBufferRef sampleBuffer)
{
NSLog(@"didCompressH264 called with status %d infoFlags %d",(int)status,(int)infoFlags);
//状态错误
if (status != 0) {
return;
}
//没准备好
if (!CMSampleBufferDataIsReady(sampleBuffer)) {
NSLog(@"didCompressH264 data is not ready");
return;
}
//C转OC
ViewController *encoder = (__bridge ViewController *)outputCallbackRefCon;
//判断当前帧是否为关键帧
/* 分步骤判断
CFArrayRef array = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, true);
CFDictionaryRef dic = CFArrayGetValueAtIndex(array, 0);
bool isKeyFrame = !CFDictionaryContainsKey(dic, kCMSampleAttachmentKey_NotSync);
*/
bool keyFrame = !CFDictionaryContainsKey((CFArrayGetValueAtIndex(CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, true), 0)), kCMSampleAttachmentKey_NotSync);
//判断当前帧是否为关键帧
//获取sps & pps 数据 只获取1次，保存在h264文件开头的第一帧中
//sps(sample per second 采样次数/s),是衡量模数转换（ADC）时采样速率的单位，帧的参数信息
//pps()，单个图像的参数信息
if (keyFrame) {
//图像存储方式，编码器等格式描述
CMFormatDescriptionRef format = CMSampleBufferGetFormatDescription(sampleBuffer);
//sps
size_t sparameterSetSize,sparameterSetCount;
const uint8_t *sparameterSet;
OSStatus statusCode = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 0, &sparameterSet, &sparameterSetSize, &sparameterSetCount, 0);
if (statusCode == noErr) {
//获取pps
size_t pparameterSetSize,pparameterSetCount;
const uint8_t *pparameterSet;
//从第一个关键帧获取sps & pps
OSStatus statusCode = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 1, &pparameterSet, &pparameterSetSize, &pparameterSetCount, 0);
//获取H264参数集合中的SPS和PPS
if (statusCode == noErr)
{
//Found pps & sps
NSData *sps = [NSData dataWithBytes:sparameterSet length:sparameterSetSize];
NSData *pps = [NSData dataWithBytes:pparameterSet length:pparameterSetSize];
if(encoder)
{
[encoder gotSpsPps:sps pps:pps];
}
}
}
}
CMBlockBufferRef dataBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
size_t length,totalLength;
char *dataPointer;
/*
CMBlockBufferGetDataPointer(CMBlockBufferRef _Nonnull theBuffer, size_t offset, size_t * _Nullable lengthAtOffsetOut, size_t * _Nullable totalLengthOut, char * _Nullable * _Nullable dataPointerOut)
* theBuffer: 数据源
* offset : 偏移量
* lengthAtOffsetOut : 单个数据长度
* totalLengthOut : 总数据长度
* dataPointerOut : 数据块首地址
*/
OSStatus statusCodeRet = CMBlockBufferGetDataPointer(dataBuffer, 0, &length, &totalLength, &dataPointer);
if (statusCodeRet == noErr) {
/*
大端： 01 23 45 67
小端： 67 45 23 01
*/
size_t bufferOffset = 0;
static const int AVCCHeaderLength = 4;//返回的nalu数据前4个字节不是001的startcode,而是大端模式的帧长度length
//循环获取nalu数据
while (bufferOffset < totalLength - AVCCHeaderLength) {
uint32_t NALUnitLength = 0;
//读取 一单元长度的 nalu
memcpy(&NALUnitLength, dataPointer + bufferOffset, AVCCHeaderLength);
//从大端模式转换为系统端模式(小端)
NALUnitLength = CFSwapInt32BigToHost(NALUnitLength);
//获取nalu数据
NSData *data = [[NSData alloc]initWithBytes:(dataPointer + bufferOffset + AVCCHeaderLength) length:NALUnitLength];
//将nalu数据写入到文件
[encoder gotEncodedData:data isKeyFrame:keyFrame];
//move to the next NAL unit in the block buffer
//读取下一个nalu 一次回调可能包含多个nalu数据
bufferOffset += AVCCHeaderLength + NALUnitLength;
}
}
}
//第一帧写入 sps & pps
- (void)gotSpsPps:(NSData*)sps pps:(NSData*)pps
{
NSLog(@"gotSpsPp %d %d",(int)[sps length],(int)[pps length]);
//写入之前(起始位)
const char bytes[] = "\x00\x00\x00\x01";
//去除末尾的/0
size_t length = (sizeof bytes) - 1;
NSData *ByteHeader = [NSData dataWithBytes:bytes length:length];
[fileHandele writeData:ByteHeader];
[fileHandele writeData:sps];
[fileHandele writeData:ByteHeader];
[fileHandele writeData:pps];
}
- (void)gotEncodedData:(NSData*)data isKeyFrame:(BOOL)isKeyFrame
{
NSLog(@"gotEncodeData %d",(int)[data length]);
if (fileHandele != NULL) {
//添加4个字节的H264 协议 start code 分割符
//一般来说编码器编出的首帧数据为PPS & SPS
//H264编码时，在每个NAL前添加起始码 0x000001,解码器在码流中检测起始码，当前NAL结束。
/*
为了防止NAL内部出现0x000001的数据，h.264又提出'防止竞争 emulation prevention"机制，在编码完一个NAL时，如果检测出有连续两个0x00字节，就在后面插入一个0x03。当解码器在NAL内部检测到0x000003的数据，就把0x03抛弃，恢复原始数据。
总的来说H264的码流的打包方式有两种,一种为annex-b byte stream format 的格式，这个是绝大部分编码器的默认输出格式，就是每个帧的开头的3~4个字节是H264的start_code,0x00000001或者0x000001。
另一种是原始的NAL打包格式，就是开始的若干字节（1，2，4字节）是NAL的长度，而不是start_code,此时必须借助某个全局的数据来获得编 码器的profile,level,PPS,SPS等信息才可以解码。
*/
const char bytes[] ="\x00\x00\x00\x01";
//长度
size_t length = (sizeof bytes) - 1;
//头字节
NSData *ByteHeader = [NSData dataWithBytes:bytes length:length];
//写入头字节
[fileHandele writeData:ByteHeader];
//写入H264数据
[fileHandele writeData:data];
}
}
//结束VideoToolBox
-(void)endVideoToolBox
{
//完成
VTCompressionSessionCompleteFrames(cEncodeingSession, kCMTimeInvalid);
//释放
VTCompressionSessionInvalidate(cEncodeingSession);
CFRelease(cEncodeingSession);
cEncodeingSession = NULL;
}
结构.png
h264解码
一.解码的思路:
- 解析数据(NALU Unit) I/P/B...
- 初始化解码器
- 将解析后的H264 NALU Unit输入解码器
- 解码完成回调,输出解码数据
- 解码数据显示(OpenGL ES)
二.解码三个核心函数:
- 创建session, VTDecompressionSessionCreate
- 解码一个frame, VTDecompressionSessionDecodeFrame
- 销毁解码session, VTDecompressionSessionInvalidate
三.原理分析:
- H264原始码流-->NALU.
- I帧:保留了一张完整视频帧.解码关键!
- P帧:先前参考帧.差异数据.解码需要依赖于I帧
- B帧:双向参考帧,解码时既需要|帧，也需要P帧!
如果H264码流中I帧错误/丢失,就会导致错误传递,P/B帧单独是完成不了解码工作!花屏的现象产生. VideoToolBox硬编码编码H264帧.I帧!手动加入SPS/PPS.
解码时:需要使用SPS/PPS数据来对解码器进行初始化!
#import
#import
@interface CCVideoDecoder ()
@property (nonatomic, strong) dispatch_queue_t decodeQueue;
@property (nonatomic, strong) dispatch_queue_t callbackQueue;
/**解码会话*/
@property (nonatomic) VTDecompressionSessionRef decodeSesion;
@end
@implementation CCVideoDecoder{
uint8_t *_sps;
NSUInteger _spsSize;
uint8_t *_pps;
NSUInteger _ppsSize;
CMVideoFormatDescriptionRef _decodeDesc;
}
/**解码回调函数*/
/*
参数1: 回调引用
参数2: 帧引用
参数3: 状态标识
参数4: 同步/异步解码
参数5: 实际图像缓存
参数6: 出现时间戳
参数7: 出现持续时间
*/
void videoDecompressionOutputCallback(void * CM_NULLABLE decompressionOutputRefCon,
void * CM_NULLABLE sourceFrameRefCon,
OSStatus status,
VTDecodeInfoFlags infoFlags,
CM_NULLABLE CVImageBufferRef imageBuffer,
CMTime presentationTimeStamp,
CMTime presentationDuration ) {
if (status != noErr) {
NSLog(@"Video hard decode callback error status=%d", (int)status);
return;
}
//解码后的数据sourceFrameRefCon -> CVPixelBufferRef
CVPixelBufferRef *outputPixelBuffer = (CVPixelBufferRef *)sourceFrameRefCon;
*outputPixelBuffer = CVPixelBufferRetain(imageBuffer);
//获取self
CCVideoDecoder *decoder = (__bridge CCVideoDecoder *)(decompressionOutputRefCon);
//调用回调队列
dispatch_async(decoder.callbackQueue, ^{
//将解码后的数据给decoder代理.viewController
[decoder.delegate videoDecodeCallback:imageBuffer];
//释放数据
CVPixelBufferRelease(imageBuffer);
});
}
- (instancetype)initWithConfig:(CCVideoConfig *)config
{
self = [super init];
if (self) {
//初始化VideoConfig 信息
_config = config;
//创建解码队列与回调队列
_decodeQueue = dispatch_queue_create("h264 hard decode queue", DISPATCH_QUEUE_SERIAL);
_callbackQueue = dispatch_queue_create("h264 hard decode callback queue", DISPATCH_QUEUE_SERIAL);
}
return self;
}
/*初始化解码器**/
- (BOOL)initDecoder {
if (_decodeSesion) return true;
const uint8_t * const parameterSetPointers[2] = {_sps, _pps};
const size_t parameterSetSizes[2] = {_spsSize, _ppsSize};
int naluHeaderLen = 4;
/**
根据sps pps设置解码参数
param kCFAllocatorDefault 分配器
param 2 参数个数
param parameterSetPointers 参数集指针
param parameterSetSizes 参数集大小
param naluHeaderLen nalu nalu start code 的长度 4
param _decodeDesc 解码器描述
return 状态
*/
OSStatus status = CMVideoFormatDescriptionCreateFromH264ParameterSets(kCFAllocatorDefault, 2, parameterSetPointers, parameterSetSizes, naluHeaderLen, &_decodeDesc);
if (status != noErr) {
NSLog(@"Video hard DecodeSession create H264ParameterSets(sps, pps) failed status= %d", (int)status);
return false;
}
/*
解码参数:
* kCVPixelBufferPixelFormatTypeKey:摄像头的输出数据格式
kCVPixelBufferPixelFormatTypeKey，已测可用值为
kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange，即420v
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange，即420f
kCVPixelFormatType_32BGRA，iOS在内部进行YUV至BGRA格式转换
YUV420一般用于标清视频，YUV422用于高清视频，这里的限制让人感到意外。但是，在相同条件下，YUV420计算耗时和传输压力比YUV422都小。
* kCVPixelBufferWidthKey/kCVPixelBufferHeightKey: 视频源的分辨率 width*height
* kCVPixelBufferOpenGLCompatibilityKey : 它允许在 OpenGL 的上下文中直接绘制解码后的图像，而不是从总线和 CPU 之间复制数据。这有时候被称为零拷贝通道，因为在绘制过程中没有解码的图像被拷贝.
*/
NSDictionary *destinationPixBufferAttrs =
@{
(id)kCVPixelBufferPixelFormatTypeKey: [NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange], //iOS上 nv12(uvuv排布) 而不是nv21（vuvu排布）
(id)kCVPixelBufferWidthKey: [NSNumber numberWithInteger:_config.width],
(id)kCVPixelBufferHeightKey: [NSNumber numberWithInteger:_config.height],
(id)kCVPixelBufferOpenGLCompatibilityKey: [NSNumber numberWithBool:true]
};
//解码回调设置
/*
VTDecompressionOutputCallbackRecord 是一个简单的结构体，它带有一个指针 (decompressionOutputCallback)，指向帧解压完成后的回调方法。你需要提供可以找到这个回调方法的实例 (decompressionOutputRefCon)。VTDecompressionOutputCallback 回调方法包括七个参数：
参数1: 回调的引用
参数2: 帧的引用
参数3: 一个状态标识 (包含未定义的代码)
参数4: 指示同步/异步解码，或者解码器是否打算丢帧的标识
参数5: 实际图像的缓冲
参数6: 出现的时间戳
参数7: 出现的持续时间
*/
VTDecompressionOutputCallbackRecord callbackRecord;
callbackRecord.decompressionOutputCallback = videoDecompressionOutputCallback;
callbackRecord.decompressionOutputRefCon = (__bridge void * _Nullable)(self);
//创建session
/*!
@function VTDecompressionSessionCreate
@abstract 创建用于解压缩视频帧的会话。
@discussion 解压后的帧将通过调用OutputCallback发出
@param allocator 内存的会话。通过使用默认的kCFAllocatorDefault的分配器。
@param videoFormatDescription 描述源视频帧
@param videoDecoderSpecification 指定必须使用的特定视频解码器.NULL
@param destinationImageBufferAttributes 描述源像素缓冲区的要求 NULL
@param outputCallback 使用已解压缩的帧调用的回调
@param decompressionSessionOut 指向一个变量以接收新的解压会话
*/
status = VTDecompressionSessionCreate(kCFAllocatorDefault, _decodeDesc, NULL, (__bridge CFDictionaryRef _Nullable)(destinationPixBufferAttrs), &callbackRecord, &_decodeSesion);
//判断一下status
if (status != noErr) {
NSLog(@"Video hard DecodeSession create failed status= %d", (int)status);
return false;
}
//设置解码会话属性(实时编码)
status = VTSessionSetProperty(_decodeSesion, kVTDecompressionPropertyKey_RealTime,kCFBooleanTrue);
NSLog(@"Vidoe hard decodeSession set property RealTime status = %d", (int)status);
return true;
}
/**解码函数（private）*/
- (CVPixelBufferRef)decode:(uint8_t *)frame withSize:(uint32_t)frameSize {
// CVPixelBufferRef 解码后的数据，编码之前源视频数据
// CMBlockBufferRef 编码之后的数据
CVPixelBufferRef outputPixelBuffer = NULL;
CMBlockBufferRef blockBuffer = NULL;
CMBlockBufferFlags flag0 = 0;
//创建blockBuffer
/*!
参数1: structureAllocator kCFAllocatorDefault 默认内存分配
参数2: memoryBlock frame 内容
参数3: frame size
参数4: blockAllocator: Pass NULL
参数5: customBlockSource Pass NULL
参数6: offsetToData 数据偏移
参数7: dataLength 数据长度
参数8: flags 功能和控制标志
参数9: newBBufOut blockBuffer地址,不能为空
*/
OSStatus status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault, frame, frameSize, kCFAllocatorNull, NULL, 0, frameSize, flag0, &blockBuffer);
if (status != kCMBlockBufferNoErr) {
NSLog(@"Video hard decode create blockBuffer error code=%d", (int)status);
return outputPixelBuffer;
}
CMSampleBufferRef sampleBuffer = NULL;
const size_t sampleSizeArray[] = {frameSize};
//创建sampleBuffer
/*
参数1: allocator 分配器,使用默认内存分配, kCFAllocatorDefault
参数2: blockBuffer.需要编码的数据blockBuffer.不能为NULL
参数3: formatDescription,视频输出格式
参数4: numSamples.CMSampleBuffer 个数.
参数5: numSampleTimingEntries 必须为0,1,numSamples
参数6: sampleTimingArray. 数组.为空
参数7: numSampleSizeEntries 默认为1
参数8: sampleSizeArray
参数9: sampleBuffer对象
*/
status = CMSampleBufferCreateReady(kCFAllocatorDefault, blockBuffer, _decodeDesc, 1, 0, NULL, 1, sampleSizeArray, &sampleBuffer);
if (status != noErr || !sampleBuffer) {
NSLog(@"Video hard decode create sampleBuffer failed status=%d", (int)status);
CFRelease(blockBuffer);
return outputPixelBuffer;
}
//解码
//向视频解码器提示使用低功耗模式是可以的
VTDecodeFrameFlags flag1 = kVTDecodeFrame_1xRealTimePlayback;
//异步解码
VTDecodeInfoFlags infoFlag = kVTDecodeInfo_Asynchronous;
//解码数据
/*
参数1: 解码session
参数2: 源数据 包含一个或多个视频帧的CMsampleBuffer
参数3: 解码标志
参数4: 解码后数据outputPixelBuffer
参数5: 同步/异步解码标识
*/
status = VTDecompressionSessionDecodeFrame(_decodeSesion, sampleBuffer, flag1, &outputPixelBuffer, &infoFlag);
if (status == kVTInvalidSessionErr) {
NSLog(@"Video hard decode InvalidSessionErr status =%d", (int)status);
} else if (status == kVTVideoDecoderBadDataErr) {
NSLog(@"Video hard decode BadData status =%d", (int)status);
} else if (status != noErr) {
NSLog(@"Video hard decode failed status =%d", (int)status);
}
CFRelease(sampleBuffer);
CFRelease(blockBuffer);
return outputPixelBuffer;
}
// private
- (void)decodeNaluData:(uint8_t *)frame size:(uint32_t)size {
//数据类型:frame的前4个字节是NALU数据的开始码，也就是00 00 00 01，
// 第5个字节是表示数据类型，转为10进制后，7是sps, 8是pps, 5是IDR（I帧）信息
int type = (frame[4] & 0x1F);
// 将NALU的开始码转为4字节大端NALU的长度信息
uint32_t naluSize = size - 4;
uint8_t *pNaluSize = (uint8_t *)(&naluSize);
CVPixelBufferRef pixelBuffer = NULL;
frame[0] = *(pNaluSize + 3);
frame[1] = *(pNaluSize + 2);
frame[2] = *(pNaluSize + 1);
frame[3] = *(pNaluSize);
//第一次解析时: 初始化解码器initDecoder
/*
关键帧/其他帧数据: 调用[self decode:frame withSize:size] 方法
sps/pps数据:则将sps/pps数据赋值到_sps/_pps中.
*/
switch (type) {
case 0x05: //关键帧
if ([self initDecoder]) {
pixelBuffer= [self decode:frame withSize:size];
}
break;
case 0x06:
//NSLog(@"SEI");//增强信息
break;
case 0x07: //sps
_spsSize = naluSize;
_sps = malloc(_spsSize);
memcpy(_sps, &frame[4], _spsSize);
break;
case 0x08: //pps
_ppsSize = naluSize;
_pps = malloc(_ppsSize);
memcpy(_pps, &frame[4], _ppsSize);
break;
default: //其他帧（1-5）
if ([self initDecoder]) {
pixelBuffer = [self decode:frame withSize:size];
}
break;
}
}
// public
- (void)decodeNaluData:(NSData *)frame {
//将解码放在异步队列.
dispatch_async(_decodeQueue, ^{
//获取frame 二进制数据
uint8_t *nalu = (uint8_t *)frame.bytes;
//调用解码Nalu数据方法,参数1:数据 参数2:数据长度
[self decodeNaluData:nalu size:(uint32_t)frame.length];
});
}
//销毁
- (void)dealloc
{
if (_decodeSesion) {
VTDecompressionSessionInvalidate(_decodeSesion);
CFRelease(_decodeSesion);
_decodeSesion = NULL;
}
}
/**
nal_unit_type NAL类型 C
0 未使用
1 非IDR图像中不采用数据划分的片段 2,3,4
2 非IDR图像中A类数据划分片段 2
3 非IDR图像中B类数据划分片段 3
4 非IDR图像中C类数据划分片段 4
5 IDR图像的片 2,3
6 补充增强信息单元（SEI） 5
7 序列参数集 0
8 图像参数集 1
9 分界符 6
10 序列结束 7
11 码流结束 8
12 填充 9
13..23 保留
24..31 不保留（RTP打包时会用到）
NSString * const naluTypesStrings[] =
{
@"0: Unspecified (non-VCL)",
@"1: Coded slice of a non-IDR picture (VCL)", // P frame
@"2: Coded slice data partition A (VCL)",
@"3: Coded slice data partition B (VCL)",
@"4: Coded slice data partition C (VCL)",
@"5: Coded slice of an IDR picture (VCL)", // I frame
@"6: Supplemental enhancement information (SEI) (non-VCL)",
@"7: Sequence parameter set (non-VCL)", // SPS parameter
@"8: Picture parameter set (non-VCL)", // PPS parameter
@"9: Access unit delimiter (non-VCL)",
@"10: End of sequence (non-VCL)",
@"11: End of stream (non-VCL)",
@"12: Filler data (non-VCL)",
@"13: Sequence parameter set extension (non-VCL)",
@"14: Prefix NAL unit (non-VCL)",
@"15: Subset sequence parameter set (non-VCL)",
@"16: Reserved (non-VCL)",
@"17: Reserved (non-VCL)",
@"18: Reserved (non-VCL)",
@"19: Coded slice of an auxiliary coded picture without partitioning (non-VCL)",
@"20: Coded slice extension (non-VCL)",
@"21: Coded slice extension for depth view components (non-VCL)",
@"22: Reserved (non-VCL)",
@"23: Reserved (non-VCL)",
@"24: STAP-A Single-time aggregation packet (non-VCL)",
@"25: STAP-B Single-time aggregation packet (non-VCL)",
@"26: MTAP16 Multi-time aggregation packet (non-VCL)",
@"27: MTAP24 Multi-time aggregation packet (non-VCL)",
@"28: FU-A Fragmentation unit (non-VCL)",
@"29: FU-B Fragmentation unit (non-VCL)",
@"30: Unspecified (non-VCL)",
@"31: Unspecified (non-VCL)",
};
*/
@end
渲染
通过OpenGL渲染
#import
#import
#include
#import
#include
#include
#include
// Uniform index.
enum
{
UNIFORM_Y,
UNIFORM_UV,
UNIFORM_ROTATION_ANGLE,
UNIFORM_COLOR_CONVERSION_MATRIX,
NUM_UNIFORMS
};
GLint uniforms[NUM_UNIFORMS];
// Attribute index.
enum
{
ATTRIB_VERTEX,
ATTRIB_TEXCOORD,
NUM_ATTRIBUTES
};
// Color Conversion Constants (YUV to RGB) including adjustment from 16-235/16-240 (video range)
// BT.601, which is the standard for SDTV.
static const GLfloat kColorConversion601[] = {
1.164, 1.164, 1.164,
0.0, -0.392, 2.017,
1.596, -0.813, 0.0,
};
// BT.709, which is the standard for HDTV.
static const GLfloat kColorConversion709[] = {
1.164, 1.164, 1.164,
0.0, -0.213, 2.112,
1.793, -0.533, 0.0,
};
@interface AAPLEAGLLayer ()
{
// The pixel dimensions of the CAEAGLLayer.
GLint _backingWidth;
GLint _backingHeight;
EAGLContext *_context;
CVOpenGLESTextureRef _lumaTexture;
CVOpenGLESTextureRef _chromaTexture;
GLuint _frameBufferHandle;
GLuint _colorBufferHandle;
const GLfloat *_preferredConversion;
}
@property GLuint program;
@end
@implementation AAPLEAGLLayer
@synthesize pixelBuffer = _pixelBuffer;
-(CVPixelBufferRef) pixelBuffer
{
return _pixelBuffer;
}
- (void)setPixelBuffer:(CVPixelBufferRef)pb
{
if(_pixelBuffer) {
CVPixelBufferRelease(_pixelBuffer);
}
_pixelBuffer = CVPixelBufferRetain(pb);
int frameWidth = (int)CVPixelBufferGetWidth(_pixelBuffer);
int frameHeight = (int)CVPixelBufferGetHeight(_pixelBuffer);
[self displayPixelBuffer:_pixelBuffer width:frameWidth height:frameHeight];
}
- (instancetype)initWithFrame:(CGRect)frame
{
self = [super init];
if (self) {
CGFloat scale = [[UIScreen mainScreen] scale];
self.contentsScale = scale;
self.opaque = TRUE;
self.drawableProperties = @{ kEAGLDrawablePropertyRetainedBacking :[NSNumber numberWithBool:YES]};
[self setFrame:frame];
// Set the context into which the frames will be drawn.
_context = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2];
if (!_context) {
return nil;
}
// Set the default conversion to BT.709, which is the standard for HDTV.
_preferredConversion = kColorConversion709;
[self setupGL];
}
return self;
}
- (void)displayPixelBuffer:(CVPixelBufferRef)pixelBuffer width:(uint32_t)frameWidth height:(uint32_t)frameHeight
{
if (!_context || ![EAGLContext setCurrentContext:_context]) {
return;
}
if(pixelBuffer == NULL) {
NSLog(@"Pixel buffer is null");
return;
}
CVReturn err;
size_t planeCount = CVPixelBufferGetPlaneCount(pixelBuffer);
/*
Use the color attachment of the pixel buffer to determine the appropriate color conversion matrix.
*/
CFTypeRef colorAttachments = CVBufferGetAttachment(pixelBuffer, kCVImageBufferYCbCrMatrixKey, NULL);
if (CFStringCompare(colorAttachments, kCVImageBufferYCbCrMatrix_ITU_R_601_4, 0) == kCFCompareEqualTo) {
_preferredConversion = kColorConversion601;
}
else {
_preferredConversion = kColorConversion709;
}
/*
CVOpenGLESTextureCacheCreateTextureFromImage will create GLES texture optimally from CVPixelBufferRef.
*/
/*
Create Y and UV textures from the pixel buffer. These textures will be drawn on the frame buffer Y-plane.
*/
CVOpenGLESTextureCacheRef _videoTextureCache;
// Create CVOpenGLESTextureCacheRef for optimal CVPixelBufferRef to GLES texture conversion.
err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, _context, NULL, &_videoTextureCache);
if (err != noErr) {
NSLog(@"Error at CVOpenGLESTextureCacheCreate %d", err);
return;
}
glActiveTexture(GL_TEXTURE0);
err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
_videoTextureCache,
pixelBuffer,
NULL,
GL_TEXTURE_2D,
GL_RED_EXT,
frameWidth,
frameHeight,
GL_RED_EXT,
GL_UNSIGNED_BYTE,
0,
&_lumaTexture);
if (err) {
NSLog(@"Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", err);
}
glBindTexture(CVOpenGLESTextureGetTarget(_lumaTexture), CVOpenGLESTextureGetName(_lumaTexture));
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
if(planeCount == 2) {
// UV-plane.
glActiveTexture(GL_TEXTURE1);
err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
_videoTextureCache,
pixelBuffer,
NULL,
GL_TEXTURE_2D,
GL_RG_EXT,
frameWidth / 2,
frameHeight / 2,
GL_RG_EXT,
GL_UNSIGNED_BYTE,
1,
&_chromaTexture);
if (err) {
NSLog(@"Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", err);
}
glBindTexture(CVOpenGLESTextureGetTarget(_chromaTexture), CVOpenGLESTextureGetName(_chromaTexture));
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
}
glBindFramebuffer(GL_FRAMEBUFFER, _frameBufferHandle);
// Set the view port to the entire view.
glViewport(0, 0, _backingWidth, _backingHeight);
glClearColor(0.0f, 0.0f, 0.0f, 1.0f);
glClear(GL_COLOR_BUFFER_BIT);
// Use shader program.
glUseProgram(self.program);
// glUniform1f(uniforms[UNIFORM_LUMA_THRESHOLD], 1);
// glUniform1f(uniforms[UNIFORM_CHROMA_THRESHOLD], 1);
glUniform1f(uniforms[UNIFORM_ROTATION_ANGLE], 0);
glUniformMatrix3fv(uniforms[UNIFORM_COLOR_CONVERSION_MATRIX], 1, GL_FALSE, _preferredConversion);
// Set up the quad vertices with respect to the orientation and aspect ratio of the video.
CGRect viewBounds = self.bounds;
CGSize contentSize = CGSizeMake(frameWidth, frameHeight);
CGRect vertexSamplingRect = AVMakeRectWithAspectRatioInsideRect(contentSize, viewBounds);
// Compute normalized quad coordinates to draw the frame into.
CGSize normalizedSamplingSize = CGSizeMake(0.0, 0.0);
CGSize cropScaleAmount = CGSizeMake(vertexSamplingRect.size.width/viewBounds.size.width,
vertexSamplingRect.size.height/viewBounds.size.height);
// Normalize the quad vertices.
if (cropScaleAmount.width > cropScaleAmount.height) {
normalizedSamplingSize.width = 1.0;
normalizedSamplingSize.height = cropScaleAmount.height/cropScaleAmount.width;
}
else {
normalizedSamplingSize.width = cropScaleAmount.width/cropScaleAmount.height;
normalizedSamplingSize.height = 1.0;;
}
/*
The quad vertex data defines the region of 2D plane onto which we draw our pixel buffers.
Vertex data formed using (-1,-1) and (1,1) as the bottom left and top right coordinates respectively, covers the entire screen.
*/
GLfloat quadVertexData [] = {
-1 * normalizedSamplingSize.width, -1 * normalizedSamplingSize.height,
normalizedSamplingSize.width, -1 * normalizedSamplingSize.height,
-1 * normalizedSamplingSize.width, normalizedSamplingSize.height,
normalizedSamplingSize.width, normalizedSamplingSize.height,
};
// Update attribute values.
glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, quadVertexData);
glEnableVertexAttribArray(ATTRIB_VERTEX);
/*
The texture vertices are set up such that we flip the texture vertically. This is so that our top left origin buffers match OpenGL's bottom left texture coordinate system.
*/
CGRect textureSamplingRect = CGRectMake(0, 0, 1, 1);
GLfloat quadTextureData[] = {
CGRectGetMinX(textureSamplingRect), CGRectGetMaxY(textureSamplingRect),
CGRectGetMaxX(textureSamplingRect), CGRectGetMaxY(textureSamplingRect),
CGRectGetMinX(textureSamplingRect), CGRectGetMinY(textureSamplingRect),
CGRectGetMaxX(textureSamplingRect), CGRectGetMinY(textureSamplingRect)
};
glVertexAttribPointer(ATTRIB_TEXCOORD, 2, GL_FLOAT, 0, 0, quadTextureData);
glEnableVertexAttribArray(ATTRIB_TEXCOORD);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
glBindRenderbuffer(GL_RENDERBUFFER, _colorBufferHandle);
[_context presentRenderbuffer:GL_RENDERBUFFER];
[self cleanUpTextures];
// Periodic texture cache flush every frame
CVOpenGLESTextureCacheFlush(_videoTextureCache, 0);
if(_videoTextureCache) {
CFRelease(_videoTextureCache);
}
}
# pragma mark - OpenGL setup
- (void)setupGL
{
if (!_context || ![EAGLContext setCurrentContext:_context]) {
return;
}
[self setupBuffers];
[self loadShaders];
glUseProgram(self.program);
// 0 and 1 are the texture IDs of _lumaTexture and _chromaTexture respectively.
glUniform1i(uniforms[UNIFORM_Y], 0);
glUniform1i(uniforms[UNIFORM_UV], 1);
glUniform1f(uniforms[UNIFORM_ROTATION_ANGLE], 0);
glUniformMatrix3fv(uniforms[UNIFORM_COLOR_CONVERSION_MATRIX], 1, GL_FALSE, _preferredConversion);
}
#pragma mark - Utilities
- (void)setupBuffers
{
glDisable(GL_DEPTH_TEST);
glEnableVertexAttribArray(ATTRIB_VERTEX);
glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, GL_FALSE, 2 * sizeof(GLfloat), 0);
glEnableVertexAttribArray(ATTRIB_TEXCOORD);
glVertexAttribPointer(ATTRIB_TEXCOORD, 2, GL_FLOAT, GL_FALSE, 2 * sizeof(GLfloat), 0);
[self createBuffers];
}
- (void) createBuffers
{
glGenFramebuffers(1, &_frameBufferHandle);
glBindFramebuffer(GL_FRAMEBUFFER, _frameBufferHandle);
glGenRenderbuffers(1, &_colorBufferHandle);
glBindRenderbuffer(GL_RENDERBUFFER, _colorBufferHandle);
[_context renderbufferStorage:GL_RENDERBUFFER fromDrawable:self];
glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_WIDTH, &_backingWidth);
glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_HEIGHT, &_backingHeight);
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, _colorBufferHandle);
if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE) {
NSLog(@"Failed to make complete framebuffer object %x", glCheckFramebufferStatus(GL_FRAMEBUFFER));
}
}
- (void) releaseBuffers
{
if(_frameBufferHandle) {
glDeleteFramebuffers(1, &_frameBufferHandle);
_frameBufferHandle = 0;
}
if(_colorBufferHandle) {
glDeleteRenderbuffers(1, &_colorBufferHandle);
_colorBufferHandle = 0;
}
}
- (void) resetRenderBuffer
{
if (!_context || ![EAGLContext setCurrentContext:_context]) {
return;
}
[self releaseBuffers];
[self createBuffers];
}
- (void) cleanUpTextures
{
if (_lumaTexture) {
CFRelease(_lumaTexture);
_lumaTexture = NULL;
}
if (_chromaTexture) {
CFRelease(_chromaTexture);
_chromaTexture = NULL;
}
}
#pragma mark - OpenGL ES 2 shader compilation
const GLchar *shader_fsh = (const GLchar*)"varying highp vec2 texCoordVarying;"
"precision mediump float;"
"uniform sampler2D SamplerY;"
"uniform sampler2D SamplerUV;"
"uniform mat3 colorConversionMatrix;"
"void main()"
"{"
" mediump vec3 yuv;"
" lowp vec3 rgb;"
// Subtract constants to map the video range start at 0
" yuv.x = (texture2D(SamplerY, texCoordVarying).r - (16.0/255.0));"
" yuv.yz = (texture2D(SamplerUV, texCoordVarying).rg - vec2(0.5, 0.5));"
" rgb = colorConversionMatrix * yuv;"
" gl_FragColor = vec4(rgb, 1);"
"}";
const GLchar *shader_vsh = (const GLchar*)"attribute vec4 position;"
"attribute vec2 texCoord;"
"uniform float preferredRotation;"
"varying vec2 texCoordVarying;"
"void main()"
"{"
" mat4 rotationMatrix = mat4(cos(preferredRotation), -sin(preferredRotation), 0.0, 0.0,"
" sin(preferredRotation), cos(preferredRotation), 0.0, 0.0,"
" 0.0, 0.0, 1.0, 0.0,"
" 0.0, 0.0, 0.0, 1.0);"
" gl_Position = position * rotationMatrix;"
" texCoordVarying = texCoord;"
"}";
- (BOOL)loadShaders
{
GLuint vertShader = 0, fragShader = 0;
// Create the shader program.
self.program = glCreateProgram();
if(![self compileShaderString:&vertShader type:GL_VERTEX_SHADER shaderString:shader_vsh]) {
NSLog(@"Failed to compile vertex shader");
return NO;
}
if(![self compileShaderString:&fragShader type:GL_FRAGMENT_SHADER shaderString:shader_fsh]) {
NSLog(@"Failed to compile fragment shader");
return NO;
}
// Attach vertex shader to program.
glAttachShader(self.program, vertShader);
// Attach fragment shader to program.
glAttachShader(self.program, fragShader);
// Bind attribute locations. This needs to be done prior to linking.
glBindAttribLocation(self.program, ATTRIB_VERTEX, "position");
glBindAttribLocation(self.program, ATTRIB_TEXCOORD, "texCoord");
// Link the program.
if (![self linkProgram:self.program]) {
NSLog(@"Failed to link program: %d", self.program);
if (vertShader) {
glDeleteShader(vertShader);
vertShader = 0;
}
if (fragShader) {
glDeleteShader(fragShader);
fragShader = 0;
}
if (self.program) {
glDeleteProgram(self.program);
self.program = 0;
}
return NO;
}
// Get uniform locations.
uniforms[UNIFORM_Y] = glGetUniformLocation(self.program, "SamplerY");
uniforms[UNIFORM_UV] = glGetUniformLocation(self.program, "SamplerUV");
// uniforms[UNIFORM_LUMA_THRESHOLD] = glGetUniformLocation(self.program, "lumaThreshold");
// uniforms[UNIFORM_CHROMA_THRESHOLD] = glGetUniformLocation(self.program, "chromaThreshold");
uniforms[UNIFORM_ROTATION_ANGLE] = glGetUniformLocation(self.program, "preferredRotation");
uniforms[UNIFORM_COLOR_CONVERSION_MATRIX] = glGetUniformLocation(self.program, "colorConversionMatrix");
// Release vertex and fragment shaders.
if (vertShader) {
glDetachShader(self.program, vertShader);
glDeleteShader(vertShader);
}
if (fragShader) {
glDetachShader(self.program, fragShader);
glDeleteShader(fragShader);
}
return YES;
}
- (BOOL)compileShaderString:(GLuint *)shader type:(GLenum)type shaderString:(const GLchar*)shaderString
{
*shader = glCreateShader(type);
glShaderSource(*shader, 1, &shaderString, NULL);
glCompileShader(*shader);
#if defined(DEBUG)
GLint logLength;
glGetShaderiv(*shader, GL_INFO_LOG_LENGTH, &logLength);
if (logLength > 0) {
GLchar *log = (GLchar *)malloc(logLength);
glGetShaderInfoLog(*shader, logLength, &logLength, log);
NSLog(@"Shader compile log:\n%s", log);
free(log);
}
#endif
GLint status = 0;
glGetShaderiv(*shader, GL_COMPILE_STATUS, &status);
if (status == 0) {
glDeleteShader(*shader);
return NO;
}
return YES;
}
- (BOOL)compileShader:(GLuint *)shader type:(GLenum)type URL:(NSURL *)URL
{
NSError *error;
NSString *sourceString = [[NSString alloc] initWithContentsOfURL:URL encoding:NSUTF8StringEncoding error:&error];
if (sourceString == nil) {
NSLog(@"Failed to load vertex shader: %@", [error localizedDescription]);
return NO;
}
const GLchar *source = (GLchar *)[sourceString UTF8String];
return [self compileShaderString:shader type:type shaderString:source];
}
- (BOOL)linkProgram:(GLuint)prog
{
GLint status;
glLinkProgram(prog);
#if defined(DEBUG)
GLint logLength;
glGetProgramiv(prog, GL_INFO_LOG_LENGTH, &logLength);
if (logLength > 0) {
GLchar *log = (GLchar *)malloc(logLength);
glGetProgramInfoLog(prog, logLength, &logLength, log);
NSLog(@"Program link log:\n%s", log);
free(log);
}
#endif
glGetProgramiv(prog, GL_LINK_STATUS, &status);
if (status == 0) {
return NO;
}
return YES;
}
- (BOOL)validateProgram:(GLuint)prog
{
GLint logLength, status;
glValidateProgram(prog);
glGetProgramiv(prog, GL_INFO_LOG_LENGTH, &logLength);
if (logLength > 0) {
GLchar *log = (GLchar *)malloc(logLength);
glGetProgramInfoLog(prog, logLength, &logLength, log);
NSLog(@"Program validate log:\n%s", log);
free(log);
}
glGetProgramiv(prog, GL_VALIDATE_STATUS, &status);
if (status == 0) {
return NO;
}
return YES;
}
- (void)dealloc
{
if (!_context || ![EAGLContext setCurrentContext:_context]) {
return;
}
[self cleanUpTextures];
if(_pixelBuffer) {
CVPixelBufferRelease(_pixelBuffer);
}
if (self.program) {
glDeleteProgram(self.program);
self.program = 0;
}
if(_context) {
//[_context release];
_context = nil;
}
//[super dealloc];
}
@end