iOS渲染-将视频原始数据(RGB,YUV)渲染到屏幕上

栏目: IOS · 发布时间: 5年前

内容简介:在做如美颜,滤镜等功能时,我们不能使用相机原生的利用OpenGL完成高效的渲染功能.本例中仅提供简单流程讲解,具体每行代码含义可在开源库中查询.注意:

需求

在做如美颜,滤镜等功能时,我们不能使用相机原生的 AVCaptureVideoPreviewLayer ,而是需要通过其他方式将视频原始帧数据如RGB,NV12等等渲染到iOS界面上.

实现原理

利用OpenGL完成高效的渲染功能.本例中仅提供简单流程讲解,具体每行代码含义可在开源库中查询.

注意:

AVCaptureVideoPreviewLayer

阅读前提

  • 音视频基础
  • OpenGL基础

代码地址 : iOS视频渲染

掘金地址 : iOS视频渲染

简书地址 : iOS视频渲染

博客地址 :iOS视频渲染

具体步骤

1. 创建 EAGLContext 上下文对象

  • 创建OpenGL预览层
CAEAGLLayer *eaglLayer       = (CAEAGLLayer *)self.layer;
eaglLayer.opaque = YES;
eaglLayer.drawableProperties = @{kEAGLDrawablePropertyRetainedBacking   : [NSNumber numberWithBool:NO],
                                 kEAGLDrawablePropertyColorFormat       : kEAGLColorFormatRGBA8};
  • 创建OpenGL上下文对象
EAGLContext *context = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2];
[EAGLContext setCurrentContext:context];
  • 设置上下文渲染缓冲区

    - (void)setupBuffersWithContext:(EAGLContext *)context width:(int *)width height:(int *)height colorBufferHandle:(GLuint *)colorBufferHandle frameBufferHandle:(GLuint *)frameBufferHandle {
        glDisable(GL_DEPTH_TEST);
        
        glEnableVertexAttribArray(ATTRIB_VERTEX);
        glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, GL_FALSE, 2 * sizeof(GLfloat), 0);
        
        glEnableVertexAttribArray(ATTRIB_TEXCOORD);
        glVertexAttribPointer(ATTRIB_TEXCOORD, 2, GL_FLOAT, GL_FALSE, 2 * sizeof(GLfloat), 0);
        
        glGenFramebuffers(1, frameBufferHandle);
        glBindFramebuffer(GL_FRAMEBUFFER, *frameBufferHandle);
        
        glGenRenderbuffers(1, colorBufferHandle);
        glBindRenderbuffer(GL_RENDERBUFFER, *colorBufferHandle);
        
        [context renderbufferStorage:GL_RENDERBUFFER fromDrawable:(CAEAGLLayer *)self.layer];
        glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_WIDTH , width);
        glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_HEIGHT, height);
        
        glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, *colorBufferHandle);
    }
    
  • 加载着色器

着色器本Demo中只添加了NV12格式与RGB格式两种原始视频数据.

- (void)loadShaderWithBufferType:(XDXPixelBufferType)type {
    GLuint vertShader, fragShader;
    NSURL  *vertShaderURL, *fragShaderURL;
    
    NSString *shaderName;
    GLuint   program;
    program = glCreateProgram();
    
    if (type == XDXPixelBufferTypeNV12) {
        shaderName = @"XDXPreviewNV12Shader";
        _nv12Program = program;
    } else if (type == XDXPixelBufferTypeRGB) {
        shaderName = @"XDXPreviewRGBShader";
        _rgbProgram = program;
    }
    
    vertShaderURL = [[NSBundle mainBundle] URLForResource:shaderName withExtension:@"vsh"];
    if (![self compileShader:&vertShader type:GL_VERTEX_SHADER URL:vertShaderURL]) {
        log4cplus_error(kModuleName, "Failed to compile vertex shader");
        return;
    }
    
    fragShaderURL = [[NSBundle mainBundle] URLForResource:shaderName withExtension:@"fsh"];
    if (![self compileShader:&fragShader type:GL_FRAGMENT_SHADER URL:fragShaderURL]) {
        log4cplus_error(kModuleName, "Failed to compile fragment shader");
        return;
    }
    
    glAttachShader(program, vertShader);
    glAttachShader(program, fragShader);
    
    glBindAttribLocation(program, ATTRIB_VERTEX  , "position");
    glBindAttribLocation(program, ATTRIB_TEXCOORD, "inputTextureCoordinate");
    
    if (![self linkProgram:program]) {
        if (vertShader) {
            glDeleteShader(vertShader);
            vertShader = 0;
        }
        if (fragShader) {
            glDeleteShader(fragShader);
            fragShader = 0;
        }
        if (program) {
            glDeleteProgram(program);
            program = 0;
        }
        return;
    }
    
    if (type == XDXPixelBufferTypeNV12) {
        uniforms[UNIFORM_Y] = glGetUniformLocation(program , "luminanceTexture");
        uniforms[UNIFORM_UV] = glGetUniformLocation(program, "chrominanceTexture");
        uniforms[UNIFORM_COLOR_CONVERSION_MATRIX] = glGetUniformLocation(program, "colorConversionMatrix");
    } else if (type == XDXPixelBufferTypeRGB) {
        _displayInputTextureUniform = glGetUniformLocation(program, "inputImageTexture");
    }
    
    if (vertShader) {
        glDetachShader(program, vertShader);
        glDeleteShader(vertShader);
    }
    if (fragShader) {
        glDetachShader(program, fragShader);
        glDeleteShader(fragShader);
    }
}

- (BOOL)compileShader:(GLuint *)shader type:(GLenum)type URL:(NSURL *)URL {
    NSError *error;
    NSString *sourceString = [[NSString alloc] initWithContentsOfURL:URL
                                                            encoding:NSUTF8StringEncoding
                                                               error:&error];
    if (sourceString == nil) {
        log4cplus_error(kModuleName, "Failed to load vertex shader: %s", [error localizedDescription].UTF8String);
        return NO;
    }
    
    GLint status;
    const GLchar *source;
    source = (GLchar *)[sourceString UTF8String];
    
    *shader = glCreateShader(type);
    glShaderSource(*shader, 1, &source, NULL);
    glCompileShader(*shader);
    
    glGetShaderiv(*shader, GL_COMPILE_STATUS, &status);
    if (status == 0) {
        glDeleteShader(*shader);
        return NO;
    }
    return YES;
}

- (BOOL)linkProgram:(GLuint)prog {
    GLint status;
    glLinkProgram(prog);
    
    glGetProgramiv(prog, GL_LINK_STATUS, &status);
    if (status == 0) {
        return NO;
    }
    return YES;
}
  • 创建视频纹理缓存区
if (!*videoTextureCache) {
    CVReturn err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, context, NULL, videoTextureCache);
    if (err != noErr)
        log4cplus_error(kModuleName, "Error at CVOpenGLESTextureCacheCreate %d",err);
}

2. 将 pixelBuffer 渲染到屏幕

  • 开始渲染前先清空缓存数据

    - (void)cleanUpTextures {
        if (_lumaTexture) {
            CFRelease(_lumaTexture);
            _lumaTexture = NULL;
        }
        
        if (_chromaTexture) {
            CFRelease(_chromaTexture);
            _chromaTexture = NULL;
        }
        
        if (_renderTexture) {
            CFRelease(_renderTexture);
            _renderTexture = NULL;
        }
        
        CVOpenGLESTextureCacheFlush(_videoTextureCache, 0);
    }
    
  • 根据pixelBuffer格式确定视频数据类型

XDXPixelBufferType bufferType;
if (CVPixelBufferGetPixelFormatType(pixelBuffer) == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange || CVPixelBufferGetPixelFormatType(pixelBuffer) == kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange) {
    bufferType = XDXPixelBufferTypeNV12;
} else if (CVPixelBufferGetPixelFormatType(pixelBuffer) == kCVPixelFormatType_32BGRA) {
    bufferType = XDXPixelBufferTypeRGB;
}else {
    log4cplus_error(kModuleName, "Not support current format.");
    return;
}
  • 通过当前的pixelBuffer对象创建 CVOpenGLESTexture 对象

    CVOpenGLESTextureRef lumaTexture,chromaTexture,renderTexture;
        if (bufferType == XDXPixelBufferTypeNV12) {
            // Y
            glActiveTexture(GL_TEXTURE0);
            
            error = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                                 videoTextureCache,
                                                                 pixelBuffer,
                                                                 NULL,
                                                                 GL_TEXTURE_2D,
                                                                 GL_LUMINANCE,
                                                                 frameWidth,
                                                                 frameHeight,
                                                                 GL_LUMINANCE,
                                                                 GL_UNSIGNED_BYTE,
                                                                 0,
                                                                 &lumaTexture);
            if (error) {
                log4cplus_error(kModuleName, "Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", error);
            }else {
                _lumaTexture = lumaTexture;
            }
            
            glBindTexture(CVOpenGLESTextureGetTarget(lumaTexture), CVOpenGLESTextureGetName(lumaTexture));
            glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
            glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
            glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
            glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
            
            // UV
            glActiveTexture(GL_TEXTURE1);
            error = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                                 videoTextureCache,
                                                                 pixelBuffer,
                                                                 NULL,
                                                                 GL_TEXTURE_2D,
                                                                 GL_LUMINANCE_ALPHA,
                                                                 frameWidth / 2,
                                                                 frameHeight / 2,
                                                                 GL_LUMINANCE_ALPHA,
                                                                 GL_UNSIGNED_BYTE,
                                                                 1,
                                                                 &chromaTexture);
            if (error) {
                log4cplus_error(kModuleName, "Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", error);
            }else {
                _chromaTexture = chromaTexture;
            }
            
            glBindTexture(CVOpenGLESTextureGetTarget(chromaTexture), CVOpenGLESTextureGetName(chromaTexture));
            glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
            glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
            glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
            glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
            
        } else if (bufferType == XDXPixelBufferTypeRGB) {
            // RGB
            glActiveTexture(GL_TEXTURE0);
            error = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                                 videoTextureCache,
                                                                 pixelBuffer,
                                                                 NULL,
                                                                 GL_TEXTURE_2D,
                                                                 GL_RGBA,
                                                                 frameWidth,
                                                                 frameHeight,
                                                                 GL_BGRA,
                                                                 GL_UNSIGNED_BYTE,
                                                                 0,
                                                                 &renderTexture);
            if (error) {
                log4cplus_error(kModuleName, "Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", error);
            }else {
                _renderTexture = renderTexture;
            }
            
            glBindTexture(CVOpenGLESTextureGetTarget(renderTexture), CVOpenGLESTextureGetName(renderTexture));
            glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
            glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
            glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
            glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
        }
    
  • 选择OpenGL程序

    glBindFramebuffer(GL_FRAMEBUFFER, frameBufferHandle);
        
        glViewport(0, 0, backingWidth, backingHeight);
        
        glClearColor(0.1f, 0.0f, 0.0f, 1.0f);
        glClear(GL_COLOR_BUFFER_BIT);
        
        if (bufferType == XDXPixelBufferTypeNV12) {
            if (self.lastBufferType != bufferType) {
                glUseProgram(nv12Program);
                glUniform1i(uniforms[UNIFORM_Y], 0);
                glUniform1i(uniforms[UNIFORM_UV], 1);
                glUniformMatrix3fv(uniforms[UNIFORM_COLOR_CONVERSION_MATRIX], 1, GL_FALSE, preferredConversion);
            }
        } else if (bufferType == XDXPixelBufferTypeRGB) {
            if (self.lastBufferType != bufferType) {
                glUseProgram(rgbProgram);
                glUniform1i(displayInputTextureUniform, 0);
            }
        }
    
  • 计算非全屏尺寸

渲染的画面可能是全屏,可能是留黑边.

static CGSize normalizedSamplingSize;

if (self.lastFullScreen != self.isFullScreen || self.pixelbufferWidth != frameWidth || self.pixelbufferHeight != frameHeight
    || normalizedSamplingSize.width == 0 || normalizedSamplingSize.height == 0  || self.screenWidth != [UIScreen mainScreen].bounds.size.width) {
    
    normalizedSamplingSize = [self getNormalizedSamplingSize:CGSizeMake(frameWidth, frameHeight)];
    self.lastFullScreen = self.isFullScreen;
    self.pixelbufferWidth = frameWidth;
    self.pixelbufferHeight = frameHeight;
    self.screenWidth = [UIScreen mainScreen].bounds.size.width;
    
    quadVertexData[0] = -1 * normalizedSamplingSize.width;
    quadVertexData[1] = -1 * normalizedSamplingSize.height;
    quadVertexData[2] = normalizedSamplingSize.width;
    quadVertexData[3] = -1 * normalizedSamplingSize.height;
    quadVertexData[4] = -1 * normalizedSamplingSize.width;
    quadVertexData[5] = normalizedSamplingSize.height;
    quadVertexData[6] = normalizedSamplingSize.width;
    quadVertexData[7] = normalizedSamplingSize.height;
}
  • 渲染画面
glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, quadVertexData);
glEnableVertexAttribArray(ATTRIB_VERTEX);

glVertexAttribPointer(ATTRIB_TEXCOORD, 2, GL_FLOAT, 0, 0, quadTextureData);
glEnableVertexAttribArray(ATTRIB_TEXCOORD);

glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

glBindRenderbuffer(GL_RENDERBUFFER, colorBufferHandle);

if ([EAGLContext currentContext] == context) {
    [context presentRenderbuffer:GL_RENDERBUFFER];
}

以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持 码农网

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

跟我学Java Web

跟我学Java Web

2010-9 / 58.00元

Java Web开发是目前最流行、使用最广泛的网站开发技术。《跟我学Java Web》通过对Java Web开发中所运用到的各种技术循序渐进地进行讲解,使读者能尽快掌握开发Web应用程序的方法。《跟我学Java Web》内容包括搭建Web开发环境、HTML相关技术基础知识、JavaScript相关技术基础知识、JSP技术基础知识、Servlet技术基础知识、搭建MySQL数据库开发环境、JDBC技......一起来看看 《跟我学Java Web》 这本书的介绍吧!

JS 压缩/解压工具
JS 压缩/解压工具

在线压缩/解压 JS 代码

Markdown 在线编辑器
Markdown 在线编辑器

Markdown 在线编辑器