Clear a CVPixelBufferRef using the GPU

Question

I am writing some video processing code using AVComposition. Giving only the necessary background details, I receive a CVPixelBuffer from an apple API that I do not control. This CVPixel buffer, contains a previously rendered video frame, as they are apparently recycled by this Apple API I do not control. So my goal, is to set all the pixels in the CVPixelBufferRef to [0, 0, 0, 0] (in RGBA color space). I can do this on the CPU via this function:

- (void)processPixelBuffer: (CVImageBufferRef)pixelBuffer
{
    CVPixelBufferLockBaseAddress( pixelBuffer, 0 );

    int bufferWidth = CVPixelBufferGetWidth(pixelBuffer);
    int bufferHeight = CVPixelBufferGetHeight(pixelBuffer);
    unsigned char *pixel = (unsigned char *)CVPixelBufferGetBaseAddress(pixelBuffer);

    for( int row = 0; row < bufferHeight; row++ ) {
        for( int column = 0; column < bufferWidth; column++ ) {
            pixel[0] = 0;
            pixel[1] = 0;
            pixel[2] = 0;
            pixel[3] = 0;
            pixel += 4;
        }
    }
    CVPixelBufferUnlockBaseAddress( pixelBuffer, 0 );
}

Is there some way I can accomplish the same thing using the GPU? Additionally, is it possible to do this via CoreImage? as I don't know openGL and it appears quite complicated to set up.

Disannul

Answer 1

You can use Accelerate vImageBufferFill to do exactly that (not GPU though). This fills BGRA with black in Swift 4:

let width: Int = 3
let height: Int = 3

var pixelBuffer: CVPixelBuffer?
var imageBuffer: vImage_Buffer
var date: Date
let iterations: Int = 10000

let pixelBufferAttributes: [CFString: Any] = [kCVPixelBufferCGImageCompatibilityKey: true, kCVPixelBufferCGBitmapContextCompatibilityKey: true]
if CVPixelBufferCreate(kCFAllocatorDefault, width, height, kCVPixelFormatType_32BGRA, pixelBufferAttributes as CFDictionary, &pixelBuffer) != kCVReturnSuccess || pixelBuffer == nil { fatalError("Cannot create pixel buffer.") }
if CVPixelBufferLockBaseAddress(pixelBuffer!, []) != kCVReturnSuccess { fatalError("Cannot lock pixel buffer base address.") }
imageBuffer = vImage_Buffer(data: CVPixelBufferGetBaseAddress(pixelBuffer!), height: UInt(height), width: UInt(width), rowBytes: CVPixelBufferGetBytesPerRow(pixelBuffer!))
vImageBufferFill_ARGB8888(&imageBuffer, [0, 0, 0, 0xFF], UInt32(kvImageNoFlags))
if CVPixelBufferUnlockBaseAddress(pixelBuffer!, []) != kCVReturnSuccess { fatalError("Cannot unlock pixel buffer base address.") }

Testing plain memset to fill small buffer with zeroes in 10000 iterations is way faster in playground. In release build with real data results will probably not differ that much.

• accelerate: 1629.0
• memset: 314.0

Inwards

I have used memset_pattern16 based on your answer, it's great, thank you! One thing to add for those who want to render a CIImage into a CVPixelBuffer: do the zeroing out before the rendering, not after rendering and being done with the pixelBuffer. My AVAssetWriter with an AVAssetWrinterInputPixelBufferAdaptor was probably using my buffers in a delayed fashion and my images were added part black to the video. – Brahma

This approach of zeroing out the buffers worked for me to fix the issue of images "stacking on top of each other" when working with transparent HEVC video. The problem for me was when using a CVPixelBufferPool, the buffers still have old image data when you get them. I wasn't able to figure out how to use memset_pattern16 bc i'm not sure that PixelBuffers are always contiguous memory, which I understand memset requires. So I stuck with vImageBufferFill_ARGB8888() bc it should handle the non-contiguous case. Thanks! – Blent

Answer 2

Assuming your pixel buffer is attached to an image buffer backed by an IO surface, you can use CVOpenGLESTextureCacheCreateTextureFromImage to get a CVOpenGLESTextureCache to give you a CVOpenGLESTextureRef. That will be able to vend a texture target and name so that you can bind the thing in OpenGL.

In OpenGL you can use a framebuffer object to render to a texture. Having done that you could use glClear to clear the thing. Call glFinish to create a synchronisation point with the GL pipeline and when you next check it from the CPU, your memory should be cleared.

Benefice

Was able to get this going via some of apple's sample code. It was much more involved though, as I had to actually do a bunch of setup for openGL, that frankly I don't understand. – Disannul

Here is perhaps a better question, will this actually offer any performance improvement over using memset to zero out the Pixel Buffer? – Disannul

My instinct is that your best hope is you parallelise it — issue the glClear for buffer B, then do the CPU work on buffer A, then call glFinish only when the CPU wants to roll on to working on buffer B. But if you're not multithreaded already, doing the same thing by GCD would probably be equally efficient. A GPU isn't just innately going to be able to do a more efficient memset — you're going be bound by memory bandwidth. – Benefice

Answer 3

I have this same problem (forcing a recycled buffer supplied by -[AVVideoCompositionRenderContext newPixelBuffer] to be black).

What if you use Core Image to do the heavy lifting? Something like this:

EAGLContext *eaglContext = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2];
CIContext *ciContext = [CIContext contextWithEAGLContext:eaglContext];

CIImage *image = [CIImage imageWithColor:[CIColor colorWithRed:0 green:0 blue:0]];
CVPixelBufferRef destination = [request.renderContext newPixelBuffer];
[ciContext render:image toCVPixelBuffer:destination];
[request finishWithComposedVideoFrame:destination];
CVBufferRelease(destination);

You would have to test the performance to know if this is any better than memset, but this is a better approach for my money than trying to peer into OpenGL APIs just to set a few bits.

Firn

Answer 4

From this page it looks like you can directly access the CVImageBufferRef as an OpenGL Texture via:

glBindTexture( CVOpenGLTextureGetTarget( image ), CVOpenGLTextureGetName( image ) );

Once you have it as a texture you can use it as the draw buffer for an FBO and simply call glClear(GL_COLOR_BUFFER_BIT); on it.

Breakwater

Don't seem to be having any luck with: glBindTexture(CVOpenGLESTextureGetTarget(dstPixels),CVOpenGLESTextureGetName(dstPixels)); glClear(GL_COLOR_BUFFER_BIT); glFinish(); – Disannul

You'd still need to attach it to a framebuffer before clearing it would work. But if you're using iOS instead of MacOS, then you probably want Tommy's solution below. – Breakwater

The more I thought about it, the more I considered that simply using memset to zero out the pixel buffer might be equally as efficient as using glClear. Any thoughts on that? – Disannul