iOS Image Manipulation (Distortion)
Asked Answered
M

1

5

I initially approached this issue with CoreImage in mind (because I also need to do facial recognition), but realized that, unfortunately, the CI Distortion filters are not yet included on the iPhone.

I attempted to dive into GLImageProcessing, CImg, and ImageMagick, though I've had a lot of trouble finding a starting point for learning any of these.

Given the number of apps out there that do image distortion, I know this can't be incredibly difficult.

I don't know C or C++, and don't have the time to learn those languages unless absolutely necessary. It would become necessary if one of those libraries is the definitive library for handling this task.

Does anyone have experience with any of these libraries?

Any books out there that cover this for iOS5 specifically?

Resources I've found:

Muimuir answered 16/2, 2012 at 21:21 Comment(3)
I've used ImageMagick before and it was a bit of a hassle getting it built "just right" for our OS X product, but in the end it was worth it. Sadly I no longer have that makefile so I can't give it to you.Apiarian
If you want to use one of those libraries, you are going to have to take the time to learn the language it is written in. Maybe I don't understand your question.Krysta
Thanks for the answers guys, just not something I wanted to hear :( silly Apple for not having those filters available to the developers yet!Muimuir
D
15

As you say, the current capabilities of Core Image are a little limited on iOS. In particular, the lack of custom kernels like you find on the desktop is disappointing. The other alternatives you list (with the exception of GLImageProcessing, which wouldn't be able to do this kind of filtering) are all CPU-bound libraries and would be much too slow for doing live filtering on a mobile device.

However, I can point you to an open source framework called GPUImage that I just rolled out because I couldn't find something that let you pull off custom effects. As its name indicates, GPUImage does GPU-accelerated processing of still images and video using OpenGL ES 2.0 shaders. You can write your own custom effects using these, so you should be able to do just about anything you can think of. The framework itself is Objective-C, and has a fairly simple interface.

As an example of a distortion filter, the following shader (based on the code in Danny Pflughoeft's answer) does a sort of a fisheye effect:

varying highp vec2 textureCoordinate;
uniform sampler2D inputImageTexture;

const mediump float bulgeFactor = 0.5;

void main()
{
    mediump vec2 processedTextureCoordinate = textureCoordinate - vec2(0.5);
    mediump float radius = processedTextureCoordinate.x * processedTextureCoordinate.x + processedTextureCoordinate.y * processedTextureCoordinate.y;
    mediump vec2 distortedCoordinate = vec2(pow(radius, bulgeFactor)) * processedTextureCoordinate + vec2(0.5);

    gl_FragColor = texture2D(inputImageTexture, distortedCoordinate);
}

This produces this kind of effect on a video stream:

Fisheye effect filter

In my benchmarks, GPUImage processes images 4X faster than Core Image on an iPhone 4 (6X faster than CPU-bound processing) and video 25X faster than Core Image (70X faster than on the CPU). In even the worst case I could throw at it, it matches Core Image for processing speed.

The framework is still fairly new, so the number of stock filters I have in there right now is low, but I'll be adding a bunch more soon. In the meantime, you can write your own custom distortion shaders to process your images, and the source code for everything is available for you to tweak as needed. My introductory post about it has a little more detail on how to use this in your applications.

Descendible answered 17/2, 2012 at 20:27 Comment(6)
Very cool stuff Brad! Can we do image composting with your shaders and your framework?Nore
@JeshuaLacock - Not yet. I need to add a slight extension to the filters so that they can accept more than one image at a time. I was focusing on getting all the single-image filters working right first, but I'll see if I can quickly implement something for compositing.Descendible
Nice! Is there a place I can keep my eye on stuff or should I just check your GIT periodically? Also, I am really interested in a shader with the reverse of this effect (smaller in the middle and bigger on the edges). Should I post a new question?Nore
@JeshuaLacock - Watching the GitHub repository would probably be the best place to stay up to date on new additions. I occasionally mention stuff about this on Twitter, too: twitter.com/#!/bradlarson . As far as a pinch effect, changing the bulgeFactor to a -0.1 gives an interesting contraction, but doesn't look quite right overall. I'm sure there's a cleaner shader implementation of this you can find out there somewhere.Descendible
@JeshuaLacock - As an update, I've now added image blending capabilities, and have started adding blending mode filters.Descendible
This is a very complete looking library. You've got edge detection - canny and sobel, you've got blur, gaussian blur even, line detection, local adaptive filtering and even another one that OpenCV doesn't have. Plus erode and dilate.Norseman

© 2022 - 2024 — McMap. All rights reserved.