CoreImage coordinate system

Question

I have CVPixelBufferRef from an AVAsset. I'm trying to apply a CIFilter to it. I use these lines:

CVPixelBufferRef pixelBuffer = ...
CVPixelBufferRef newPixelBuffer = // empty pixel buffer to fill
CIContex *context = // CIContext created from EAGLContext
CGAffineTransform preferredTransform = // AVAsset track preferred transform
CIImage *phase1 = [CIImage imageWithCVPixelBuffer:pixelBuffer];
CIImage *phase2 = [phase1 imageByApplyingTransform:preferredTransform];
CIImage *phase3 = [self applyFiltersToImage:phase2];

[context render:phase3 toCVPixelBuffer:newPixelBuffer bounds:phase3.extent colorSpace:CGColorSpaceCreateDeviceRGB()];

Unfortunately, the result I get have an incorrect orientation. For example, a video captured in the portrait mode is upside down. I guess the problem is in going from AVAsset to CoreImage coordinate system (showing a preview in XCode for phase2 also presents an incorrect result). How to fix it?

Cryptogam

Did you ever manage to figure out how to fix the orientation for portrait videos? I'm seeing the same thing and can't figure out if there's simply a setting that needs to be set – Devault

It was some time ago so I don't remember exactly, but I guess I hardcoded transformation like you did in your answer. – Lancey

Answer 1

I solved it by doing this, It should orientate everything correctly to the coordinate space

var preferredTransform = inst.preferredTransform
preferredTransform.b *= -1
preferredTransform.c *= -1

var outputImage = CIImage(cvPixelBuffer: videoFrameBuffer)
                    .applying(preferredTransform)
outputImage = outputImage.applying(CGAffineTransform(translationX: -outputImage.extent.origin.x, y: -outputImage.extent.origin.y))

Devault

Hey Jon, why is the translate needed? could it cause performance issues when doing it on many source frames? (filtering video for example) – Echinus

Hi there, I don't think it should have a huge performance hit since you'll always do the preferredTransform. I didn't notice anything in my App when I implemented it. – Devault

Hey Jon! Applying the correct transform and then scaling the CIImage causing the image to render in a stretched-kind-of-way (for a portrait video). Do you have any idea what can I do to solve this? – Echinus

Thanks for the working answer Jon, any chance you can elaborate what exactly this does and is there a more expanded version where we can read about this and why is this happening in first place? 👍 – Wilhoit