Is there any faster way to access the frame buffer than using glReadPixels? I would need read-only access to a small rectangular rendering area in the frame buffer to process the data further in CPU. Performance is important because I have to perform this operation repeatedly. I have searched the web and found some approach like using Pixel Buffer Object and glMapBuffer but it seems that OpenGL ES 2.0 does not support them.
Faster alternative to glReadPixels in iPhone OpenGL ES 2.0
Asked Answered
As of iOS 5.0, there is now a faster way to grab data from OpenGL ES. It isn't readily apparent, but it turns out that the texture cache support added in iOS 5.0 doesn't just work for fast upload of camera frames to OpenGL ES, but it can be used in reverse to get quick access to the raw pixels within an OpenGL ES texture.
You can take advantage of this to grab the pixels for an OpenGL ES rendering by using a framebuffer object (FBO) with an attached texture, with that texture having been supplied from the texture cache. Once you render your scene into that FBO, the BGRA pixels for that scene will be contained within your CVPixelBufferRef, so there will be no need to pull them down using glReadPixels().
This is much, much faster than using glReadPixels() in my benchmarks. I found that on my iPhone 4, glReadPixels() was the bottleneck in reading 720p video frames for encoding to disk. It limited the encoding from taking place at anything more than 8-9 FPS. Replacing this with the fast texture cache reads allows me to encode 720p video at 20 FPS now, and the bottleneck has moved from the pixel reading to the OpenGL ES processing and actual movie encoding parts of the pipeline. On an iPhone 4S, this allows you to write 1080p video at a full 30 FPS.
My implementation can be found within the GPUImageMovieWriter class within my open source GPUImage framework, but it was inspired by Dennis Muhlestein's article on the subject and Apple's ChromaKey sample application (which was only made available at WWDC 2011).
I start by configuring my AVAssetWriter, adding an input, and configuring a pixel buffer input. The following code is used to set up the pixel buffer input:
NSDictionary *sourcePixelBufferAttributesDictionary = [NSDictionary dictionaryWithObjectsAndKeys: [NSNumber numberWithInt:kCVPixelFormatType_32BGRA], kCVPixelBufferPixelFormatTypeKey,
[NSNumber numberWithInt:videoSize.width], kCVPixelBufferWidthKey,
[NSNumber numberWithInt:videoSize.height], kCVPixelBufferHeightKey,
nil];
assetWriterPixelBufferInput = [AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:assetWriterVideoInput sourcePixelBufferAttributes:sourcePixelBufferAttributesDictionary];
Once I have that, I configure the FBO that I'll be rendering my video frames to, using the following code:
if ([GPUImageOpenGLESContext supportsFastTextureUpload])
{
CVReturn err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, (__bridge void *)[[GPUImageOpenGLESContext sharedImageProcessingOpenGLESContext] context], NULL, &coreVideoTextureCache);
if (err)
{
NSAssert(NO, @"Error at CVOpenGLESTextureCacheCreate %d");
}
CVPixelBufferPoolCreatePixelBuffer (NULL, [assetWriterPixelBufferInput pixelBufferPool], &renderTarget);
CVOpenGLESTextureRef renderTexture;
CVOpenGLESTextureCacheCreateTextureFromImage (kCFAllocatorDefault, coreVideoTextureCache, renderTarget,
NULL, // texture attributes
GL_TEXTURE_2D,
GL_RGBA, // opengl format
(int)videoSize.width,
(int)videoSize.height,
GL_BGRA, // native iOS format
GL_UNSIGNED_BYTE,
0,
&renderTexture);
glBindTexture(CVOpenGLESTextureGetTarget(renderTexture), CVOpenGLESTextureGetName(renderTexture));
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, CVOpenGLESTextureGetName(renderTexture), 0);
}
This pulls a pixel buffer from the pool associated with my asset writer input, creates and associates a texture with it, and uses that texture as a target for my FBO.
Once I've rendered a frame, I lock the base address of the pixel buffer:
CVPixelBufferLockBaseAddress(pixel_buffer, 0);
and then simply feed it into my asset writer to be encoded:
CMTime currentTime = CMTimeMakeWithSeconds([[NSDate date] timeIntervalSinceDate:startTime],120);
if(![assetWriterPixelBufferInput appendPixelBuffer:pixel_buffer withPresentationTime:currentTime])
{
NSLog(@"Problem appending pixel buffer at time: %lld", currentTime.value);
}
else
{
// NSLog(@"Recorded pixel buffer at time: %lld", currentTime.value);
}
CVPixelBufferUnlockBaseAddress(pixel_buffer, 0);
if (![GPUImageOpenGLESContext supportsFastTextureUpload])
{
CVPixelBufferRelease(pixel_buffer);
}
Note that at no point here am I reading anything manually. Also, the textures are natively in BGRA format, which is what AVAssetWriters are optimized to use when encoding video, so there's no need to do any color swizzling here. The raw BGRA pixels are just fed into the encoder to make the movie.
Aside from the use of this in an AVAssetWriter, I have some code in this answer that I've used for raw pixel extraction. It also experiences a significant speedup in practice when compared to using glReadPixels(), although less than I see with the pixel buffer pool I use with AVAssetWriter.
It's a shame that none of this is documented anywhere, because it provides a huge boost to video capture performance.
@Steed - After a little experimentation, I've now tested this in several different cases and found it to have performance advantages in all of them. I show an example for grabbing raw pixels (that I use for on-CPU processing as well as image saving) here: https://mcmap.net/q/11141/-opengl-es-2d-rendering-into-image . You do have to make sure you use the
kCVPixelBufferIOSurfacePropertiesKey to get the direct data access to work well here. On every device I've tested, this is significantly faster than glReadPixels(). However, as you point out, the Simulator currently doesn't support these kinds of reads. –
Pylon Thanks for the update. I tried again but still cannot make it work without calling
CVOpenGLESTextureCacheCreateTextureFromImage every frame. According to your post, calling of the method is done in the set up but I can't make it work that way... –
Steed @Steed - Are you adjusting the size of your FBO regularly? That's the only reason I would see for needing to regenerate your texture on every frame. In my GPUImageRawDataOutput here: github.com/BradLarson/GPUImage I only create the texture once for a given FBO size, and I can just read from it on every incoming frame. You can check out the ColorObjectTracking sample (which has a slight flaw right now in its flipped coordinates), which uses this to read raw bytes back from the texture on every frame. It doesn't require this texture regeneration and it works just fine. –
Pylon
No I don't change the size of my FBO. Though I still cannot make it work with my app as expected yet. I made sure your method works faster than glReadPixels by writing a simple OpenGL app. Thanks for the help. I'm now marking your answer as accepted. –
Steed
@BradLarson Hey, I've been trying to implement realtime (or close) screenshotting on ipad/iphone using your code as an example, and i've run into the GL_FRAMEBUFFER_UNSUPPORTED error when creating my FBO. I use the same settings you do. What do you think might be causing this error? –
Musser
Note that if you are not creating a pixel buffer using an AVAssetWriterInputPixelBufferAdaptor pool, you will need to configure the CVPixelBufferRef to be an IOSurface in order for this technique to work. See Dennis Muhlestein's article for an example of how this is accomplished. –
Dalton
I posted my question here: #13431493 –
Wagram
@BradLarson Do you have to call
glFinish or glFlush before CVPixelBufferLockBaseAddress to ensure that the GPU isn't still processing commands in the framebuffer? –
Airbrush @MarkIngram - Yes,
glFinish() is necessary (glFlush() won't block in many cases on iOS) before -appendPixelBuffer:withPresentationTime: in the above code. Otherwise, you'll see screen tearing when recording video. I don't have it in the above code, but it is called as part of a rendering routine just before this in the framework where I use the above. –
Pylon @BradLarson What about reading from the texture at the same time as the texture is bound to the context, for drawing triangles? Can it be bound and read from at the same time? –
Airbrush
is this answer still valid considering IOSurface is not a public api. WIll apple not reject my app if I use this? –
Crashaw
@Crashaw - None of the above uses any private API. This is all fully supported by Apple, and they have not rejected any application that uses this, to my knowledge. Direct access to the screen via other IOSurface APIs might be what you're thinking of, since that does require private API to work. –
Pylon
@BradLarson: Are
CVPixelBuffers implemented above OpenGL ES? Is it possible to get the same effect using the (portable) OpenGL ES API? –
Awhirl @AdiShavit - CVPixelBuffers are Apple-specific, and are platform-dependent. The last mile of communicating between OpenGL ES and the OS will necessarily be vendor-specific. There's not going to be a portable version of this. –
Pylon
@BradLarson: Is it possible to create a 3-channel RGB/BGR texture (no alpha) with
CVOpenGLESTextureCacheCreateTextureFromImage? I cannot seem to find the right flags (#27130198). –
Awhirl Working with this method on iOS 8, and the resulting images are only a mirrored quarter of what is expected. AssetWriter, FBO and Texture are all created with the same dimensions. –
Monosyllable
Is there a reason you're using
glReadPixel in GPUImage github.com/BradLarson/GPUImage/blob/…? I could not get your example to work and I probably missed something: #43729960 thanks thought! –
Rammer @Rammer - That's on the Mac. As of the time I wrote that code, there was no direct equivalent on the Mac for the above (there were similar capabilities, but I didn't take the time to implement them). There are matching capabilities now, but I haven't updated that code. –
Pylon
Sorry for the confusion I meant to point to the iOS equivalent: github.com/BradLarson/GPUImage/blob/… –
Rammer
@Rammer - That's only used if the current context fails the
supportsFastTextureUpload check (it's running on an OS version that doesn't support the above, or in the Simulator, which doesn't). It's a fallback if the above can't be used. –
Pylon How about updating code example to include
kCVPixelBufferIOSurfacePropertiesKey? Because rendered pixels won't be available to CPU if the attribute is missing... I lost my two weeks for this... –
Departmentalism Can someone just post a youtube tutorial on this, elaborating on what Brad said? If so, please post the youtube link on here. –
Stultify
Regarding what atisman mentioned about the black screen, I had that issue as well. Do really make sure everything is fine with your texture and other settings. I was trying to capture AIR's OpenGL layer, which I did in the end, the problem was that when I didn't set "depthAndStencil" to true by accident in the apps manifest, my FBO texture was half in height(the screen was divided in half and mirrored, I guess because of the wrap texture param stuff). And my video was black.
That was pretty frustrating, as based on what Brad is posting it should have just worked once I had some data in texture. Unfortunately, that's not the case, everything has to be "right" for it to work - data in texture is not a guarantee for seeing equal data in the video. Once I added depthAndStencil my texture fixed itself to full height and I started to get video recording straight from AIR's OpenGL layer, no glReadPixels or anything :)
So yeah, what Brad describes really DOES work without the need to recreate the buffers on every frame, you just need to make sure your setup is right. If you're getting blackness, try playing with the video/texture sizes perhaps or some other settings (setup of your FBO?).
I also encountered a situation when no errors occur during rendering, however, only blackness appears in the recorded video file. I spent 3 days to make everything work. Here are two necessary points that were missed in my case:
- When creating a CVPixelBuffer (it doesn't matter from the pool or directly), you need to set kCVPixelBufferIOSurfacePropertiesKey to CFDictionary (can be empty). I mistakenly set this key to a Boolean value until it dawned on me.
...
let k = bounds.height / bounds.width
let vw = 720.0 // 1080.0
let vh = vw * k
let pixelBufferWidth = Int(vw) - Int(vw) % 16
let pixelBufferHeight = Int(vh) - Int(vh) % 16
...
var pixelBuffer:CVPixelBuffer? = nil
let ioSurfaceProperties = [:] as [String : Any] as CFDictionary
let pixelBufferAttributes = [
String(kCVPixelBufferIOSurfacePropertiesKey) : ioSurfaceProperties // <-- IMPORTANT
] as [String : Any] as CFDictionary
let cvReturn1 = CVPixelBufferCreate(kCFAllocatorDefault,
pixelBufferWidth,
pixelBufferHeight,
kCVPixelFormatType_32BGRA,
pixelBufferAttributes,
&pixelBuffer)
guard cvReturn1 == 0,
let sourceImage = pixelBuffer
else{
fatalError("createCVTextureCache fail CVPixelBufferCreate(nil, \(pixelBufferWidth), \(pixelBufferHeight), \(kCVPixelFormatType_32BGRA), \(pixelBufferAttributes)) return non zero (\(cvReturnToString(cvReturn1)))")
}
...
- The CVOpenGLESTexture value initialized by the CVOpenGLESTextureCacheCreateTextureFromImage function must be saved until the recording ends. If you do not do this, but use only the textureName and textureTarget values necessary for OpenGL, then after configuring FrameBufer you will get an INCOMPLETE_ATTACHMENT (when checking frame buffer status) and INVALID_FRAMEBUFFER_OPERATION (when checking glerrors), and the file will only be black.
....
var cvGlTexture:CVOpenGLESTexture? = nil
let cvReturn2 = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
textureCache,
sourceImage,
nil, //textureAttributes: CFDictionary
GLenum(GL_TEXTURE_2D),
GL_RGBA,
GLsizei(w), //width: GLsizei
GLsizei(h), //height: GLsizei
GLenum(GL_BGRA), //format: GLenum
GLenum(GL_UNSIGNED_BYTE), //type: GLenum
0, //planeIndex: Int
&cvGlTexture)//textureOut: unsafeMutablePointer<CVOpenGLESTexture?>
guard cvReturn2 == 0,
let cvTexture = cvGlTexture
else{
fatalError("createCVTextureCache fail CVOpenGLESTextureCacheCreateTextureFromImage(...) return non zero (\(cvReturnToString(cvReturn2)))")
}
GLView.cvTextureTarget = CVOpenGLESTextureGetTarget(cvTexture)
GLView.cvTextureName = CVOpenGLESTextureGetName(cvTexture)
GLView.cvTexture = cvTexture // <--- IMPORTANT (RETAIN)
....
© 2022 - 2024 — McMap. All rights reserved.
pixel_bufferis used because I branch based on whether you're running this on iOS 5.0 or 4.0. On 4.0, I pull a new pixel buffer from the pool, store it inpixel_buffer, and useglReadPixels()to pull data. On 5.0, it's just assigned from the existing cached pixel buffer. I'm not sure what's going wrong in your case, but getting the base address of the bytes in the pixel buffer should point you to the bytes for the texture after rendering to it in your FBO. – Pylon