How to get pixel data from a UIImage (Cocoa Touch) or CGImage (Core Graphics)?

Question

I have a UIImage (Cocoa Touch). From that, I'm happy to get a CGImage or anything else you'd like that's available. I'd like to write this function:

- (int)getRGBAFromImage:(UIImage *)image atX:(int)xx andY:(int)yy {
  // [...]
  // What do I want to read about to help
  // me fill in this bit, here?
  // [...]

  int result = (red << 24) | (green << 16) | (blue << 8) | alpha;
  return result;
}

Kugler

Answer 1

FYI, I combined Keremk's answer with my original outline, cleaned-up the typos, generalized it to return an array of colors and got the whole thing to compile. Here is the result:

+ (NSArray*)getRGBAsFromImage:(UIImage*)image atX:(int)x andY:(int)y count:(int)count
{
    NSMutableArray *result = [NSMutableArray arrayWithCapacity:count];

    // First get the image into your data buffer
    CGImageRef imageRef = [image CGImage];
    NSUInteger width = CGImageGetWidth(imageRef);
    NSUInteger height = CGImageGetHeight(imageRef);
    CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
    unsigned char *rawData = (unsigned char*) calloc(height * width * 4, sizeof(unsigned char));
    NSUInteger bytesPerPixel = 4;
    NSUInteger bytesPerRow = bytesPerPixel * width;
    NSUInteger bitsPerComponent = 8;
    CGContextRef context = CGBitmapContextCreate(rawData, width, height,
                    bitsPerComponent, bytesPerRow, colorSpace,
                    kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);
    CGColorSpaceRelease(colorSpace);

    CGContextDrawImage(context, CGRectMake(0, 0, width, height), imageRef);
    CGContextRelease(context);

    // Now your rawData contains the image data in the RGBA8888 pixel format.
    NSUInteger byteIndex = (bytesPerRow * y) + x * bytesPerPixel;
    for (int i = 0 ; i < count ; ++i)
    {
        CGFloat alpha = ((CGFloat) rawData[byteIndex + 3] ) / 255.0f;
        CGFloat red   = ((CGFloat) rawData[byteIndex]     ) / alpha;
        CGFloat green = ((CGFloat) rawData[byteIndex + 1] ) / alpha;
        CGFloat blue  = ((CGFloat) rawData[byteIndex + 2] ) / alpha;
        byteIndex += bytesPerPixel;

        UIColor *acolor = [UIColor colorWithRed:red green:green blue:blue alpha:alpha];
        [result addObject:acolor];
    }

  free(rawData);

  return result;
}

Kugler

(Stripping out the array thing and getting it to return just 1 color should be easy enough. I wanted to grab a bunch of them, as long as I was creating the buffer.) – Kugler

Don't forget to unpremultiply the colors. Also, those multiplications by 1 don't do anything. – Whaley

@Peter: Correct, they're left-over from when I was doing color-fiddling. @Tom: to scan the entire image, it'd be count = width*height, right? For me, I added count because, in my special images, the first 6 pixels contained special color values, and that's what I wanted back in my array. – Kugler

What if I don't want the RGB values to be premultiplied? – Lawrence

Even if it's correct. When count is a large number (like the length of a row of the image or the entire image size!) I don't the performance of this method will be good since having too many UIColor objects is really heavy. In real life when I need a pixel then UIColor is ok, (An NSArray of UIColors is too much for me). And when you need bunch of colors I won't use UIColors since I will be probably using OpenGL,etc. In my opinion this method has no real life usage but is very good for educational purposes. – Mongolia

@Kugler you should use CGImageGetBytesPerRow(imageRef) i.s.o. bytesPerPixel*width, depending on the source of the image, bytesPerRow can be greater than bytesPerPixel*width (e.g. rows can be 128bit aligned). This also means the loop at the end could alter, if byteIndex gets in the last padding bytes area. – Masturbation

Way too cumbersome malloc a large space only to read one pixel. – Scramble

@Kugler thanks bunch for this major time saver. I've tweaked it a bit to make it an instance method on a UIImage category and to read into an NSData object. I've also added some methods for sampling a single pixel as well as a selection of colours along a gradient image. github.com/Club15CC/Marshmallows/tree/master/Lib/UIKit – Rowel

USE CALLOC INSTEAD OF MALLOC!!!! I was using this code to test if certain pixels were transparent and I was getting back bogus information where pixels were meant to be transparent because the memory wasn't cleared first. Using calloc(width*height, 4) instead of malloc did the trick. The rest of the code is great, THANKS! – Medlock

Also, cast that calloc to (unsigned char *) :) - xcode 4.3 will actually generate a compile-time error for this! – Commentative

Note that images taken vertically with the camera set the imageOrientation flag in the UIImage object instead of rotating the image. This can lead to some nasty bugs, since the conversion to CGImage ignores this. So if imageOrientation is UIImageOrientationUp (which it is for images taken with the phone sideways), then this will work fine, but if it is UIImageOrientationRight (which it is for images taken with the camera vertical) then the whole image will end up 90 degrees rotated in your array. Beware! – Trinatte

Maybe I am missing something but this code uses kCGImageAlphaPremultipliedLast for the CGContext yet when creating the UIColor, the RGB values are used as is. This will give wrong values when the pixel's alpha is less than 1.0 – Assisi

This is great. Thanks. Note on usage: x and y are the coordinates within the image to start getting RGBAs from and 'count' is the number of pixels from that point to get, going left to right, then row by row. Example Usage: To get RGBAs for an entire image: getRGBAsFromImage:image atX:0 andY:0 count:image.size.width*image.size.height To get RGBAs for the last row of an image: getRGBAsFromImage:image atX:0 andY:image.size.height-1 count:image.size.width – Winter

During assigning values to red, green, blue the numbers are not between 0 and 1. So UIColor that created with these values are almost always white. Each these code values must be divided to 255.0f like: UIColor *acolor = [UIColor colorWithRed:red/255.0f green:green/255.0f blue:blue/255.0f alpha:alpha]; – Impersonality

Thanks for sharing that. Apple really does a remarkable job when it comes to complicated, half finished, in other words crappy, APIs. – Windtight

Added comment: Make sure not to make the mistake of also dividing the alpha by 255.0f though. This messed me up for a few hours. – Bicameral

Answer 2

One way of doing it is to draw the image to a bitmap context that is backed by a given buffer for a given colorspace (in this case it is RGB): (note that this will copy the image data to that buffer, so you do want to cache it instead of doing this operation every time you need to get pixel values)

See below as a sample:

// First get the image into your data buffer
CGImageRef image = [myUIImage CGImage];
NSUInteger width = CGImageGetWidth(image);
NSUInteger height = CGImageGetHeight(image);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
unsigned char *rawData = malloc(height * width * 4);
NSUInteger bytesPerPixel = 4;
NSUInteger bytesPerRow = bytesPerPixel * width;
NSUInteger bitsPerComponent = 8;
CGContextRef context = CGBitmapContextCreate(rawData, width, height, bitsPerComponent, bytesPerRow, colorSpace, kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);
CGColorSpaceRelease(colorSpace);

CGContextDrawImage(context, CGRectMake(0, 0, width, height));
CGContextRelease(context);

// Now your rawData contains the image data in the RGBA8888 pixel format.
int byteIndex = (bytesPerRow * yy) + xx * bytesPerPixel;
red = rawData[byteIndex];
green = rawData[byteIndex + 1];
blue = rawData[byteIndex + 2];
alpha = rawData[byteIndex + 3];

Marlenmarlena

I did something similar in my app. To extract an arbitrary pixel, you just draw into a 1x1 bitmap with a known bitmap format, adjusting the origin of the CGBitmapContext appropriately. – Gautious

This leaks memory. Release rawData: free(rawData); – Mews

CGContextDrawImage requires three arguments and above only two are given...I think it should be CGContextDrawImage(context, CGRectMake(0, 0, width, height), image) – Partible

Answer 3

Apple's Technical Q&A QA1509 shows the following simple approach:

CFDataRef CopyImagePixels(CGImageRef inImage)
{
    return CGDataProviderCopyData(CGImageGetDataProvider(inImage));
}

Use CFDataGetBytePtr to get to the actual bytes (and various CGImageGet* methods to understand how to interpret them).

Meir

This is not a great approach because the pixel format tends to vary a lot per image. There are several things that can change, including 1. the orientation of the image 2. the format of the alpha component and 3. the byte order of RGB. I've personally spent some time trying to decode Apple's docs on how to do this, but I'm not sure it's worth it. If you want it done fast, just keremk's solution. – Emaciation

This method always gives me BGR format even if I save the image (from within my app in colorspace RGBA) – Kato

@Soumyaljit This is going to copy the data in the format which the CGImageRef data was originally stored. In most cases this is going to be BGRA, but could also be YUV. – Cleave

Answer 4

Couldn't believe that there is not one single correct answer here. No need to allocate pointers, and the unmultiplied values still need to be normalized. To cut to the chase, here is the correct version for Swift 4. For UIImage just use .cgImage.

extension CGImage {
    func colors(at: [CGPoint]) -> [UIColor]? {
        let colorSpace = CGColorSpaceCreateDeviceRGB()
        let bytesPerPixel = 4
        let bytesPerRow = bytesPerPixel * width
        let bitsPerComponent = 8
        let bitmapInfo: UInt32 = CGImageAlphaInfo.premultipliedLast.rawValue | CGBitmapInfo.byteOrder32Big.rawValue
    
        guard let context = CGContext(data: nil, width: width, height: height, bitsPerComponent: bitsPerComponent, bytesPerRow: bytesPerRow, space: colorSpace, bitmapInfo: bitmapInfo),
            let ptr = context.data?.assumingMemoryBound(to: UInt8.self) else {
            return nil
        }
    
        context.draw(self, in: CGRect(x: 0, y: 0, width: width, height: height))
    
        return at.map { p in
            let i = bytesPerRow * Int(p.y) + bytesPerPixel * Int(p.x)
            
            let a = CGFloat(ptr[i + 3]) / 255.0
            let r = (CGFloat(ptr[i]) / a) / 255.0
            let g = (CGFloat(ptr[i + 1]) / a) / 255.0
            let b = (CGFloat(ptr[i + 2]) / a) / 255.0
            
            return UIColor(red: r, green: g, blue: b, alpha: a)
        }
    }
}

The reason you have to draw/convert the image first into a buffer is because images can have several different formats. This step is required to convert it to a consistent format you can read.

Carew

How can I use this extension for my image? let imageObj = UIImage(named:"greencolorImg.png") – Hermann

let imageObj = UIImage(named:"greencolorImg.png"); imageObj.cgImage?.colors(at: [pt1, pt2]) – Carew

keep in mind that pt1 and pt2 are CGPoint variables. – Cupboard

saved my day :) – Elemi

@ErikAigner will this work if I wanted to get color of each pixel of an image? what's the performance cost? – Passageway

Drawing the whole image just to get the value of one pixel doesn't seem very performant. – Celtuce

Its not that bad, obviously its not as fast as direct access, but you'd need to know the internal format of the image for that, and a draw converts it to a known format/byte order we can query. That's why you can query for multiple colors in the image at once. – Carew

Answer 5

NSString * path = [[NSBundle mainBundle] pathForResource:@"filename" ofType:@"jpg"];
UIImage * img = [[UIImage alloc]initWithContentsOfFile:path];
CGImageRef image = [img CGImage];
CFDataRef data = CGDataProviderCopyData(CGImageGetDataProvider(image));
const unsigned char * buffer =  CFDataGetBytePtr(data);

Rebus

I posted this and it worked for me, but going from the buffer back to a UIImage I cant seem to figure out yet, any ideas? – Rebus

Answer 6

Here is a SO thread where @Matt renders only the desired pixel into a 1x1 context by displacing the image so that the desired pixel aligns with the one pixel in the context.

Goshawk

Answer 7

Swift 5 version

The answers given here are either outdated or incorrect because they don't take into account the following:

1. The pixel size of the image can differ from its point size that is returned by image.size.width/image.size.height.
2. There can be various layouts used by pixel components in the image, such as BGRA, ABGR, ARGB etc. or may not have an alpha component at all, such as BGR and RGB. For example, UIView.drawHierarchy(in:afterScreenUpdates:) method can produce BGRA images.
3. Color components can be premultiplied by the alpha for all pixels in the image and need to be divided by alpha in order to restore the original color.
4. For memory optimization used by CGImage, the size of a pixel row in bytes can be greater than the mere multiplication of the pixel width by 4.

The code below is to provide a universal Swift 5 solution to get the UIColor of a pixel for all such special cases. The code is optimized for usability and clarity, not for performance.

public extension UIImage {

    var pixelWidth: Int {
        return cgImage?.width ?? 0
    }

    var pixelHeight: Int {
        return cgImage?.height ?? 0
    }

    func pixelColor(x: Int, y: Int) -> UIColor {
        assert(
            0..<pixelWidth ~= x && 0..<pixelHeight ~= y,
            "Pixel coordinates are out of bounds")

        guard
            let cgImage = cgImage,
            let data = cgImage.dataProvider?.data,
            let dataPtr = CFDataGetBytePtr(data),
            let colorSpaceModel = cgImage.colorSpace?.model,
            let componentLayout = cgImage.bitmapInfo.componentLayout
        else {
            assertionFailure("Could not get a pixel of an image")
            return .clear
        }

        assert(
            colorSpaceModel == .rgb,
            "The only supported color space model is RGB")
        assert(
            cgImage.bitsPerPixel == 32 || cgImage.bitsPerPixel == 24,
            "A pixel is expected to be either 4 or 3 bytes in size")

        let bytesPerRow = cgImage.bytesPerRow
        let bytesPerPixel = cgImage.bitsPerPixel/8
        let pixelOffset = y*bytesPerRow + x*bytesPerPixel

        if componentLayout.count == 4 {
            let components = (
                dataPtr[pixelOffset + 0],
                dataPtr[pixelOffset + 1],
                dataPtr[pixelOffset + 2],
                dataPtr[pixelOffset + 3]
            )

            var alpha: UInt8 = 0
            var red: UInt8 = 0
            var green: UInt8 = 0
            var blue: UInt8 = 0

            switch componentLayout {
            case .bgra:
                alpha = components.3
                red = components.2
                green = components.1
                blue = components.0
            case .abgr:
                alpha = components.0
                red = components.3
                green = components.2
                blue = components.1
            case .argb:
                alpha = components.0
                red = components.1
                green = components.2
                blue = components.3
            case .rgba:
                alpha = components.3
                red = components.0
                green = components.1
                blue = components.2
            default:
                return .clear
            }

            // If chroma components are premultiplied by alpha and the alpha is `0`,
            // keep the chroma components to their current values.
            if cgImage.bitmapInfo.chromaIsPremultipliedByAlpha && alpha != 0 {
                let invUnitAlpha = 255/CGFloat(alpha)
                red = UInt8((CGFloat(red)*invUnitAlpha).rounded())
                green = UInt8((CGFloat(green)*invUnitAlpha).rounded())
                blue = UInt8((CGFloat(blue)*invUnitAlpha).rounded())
            }

            return .init(red: red, green: green, blue: blue, alpha: alpha)

        } else if componentLayout.count == 3 {
            let components = (
                dataPtr[pixelOffset + 0],
                dataPtr[pixelOffset + 1],
                dataPtr[pixelOffset + 2]
            )

            var red: UInt8 = 0
            var green: UInt8 = 0
            var blue: UInt8 = 0

            switch componentLayout {
            case .bgr:
                red = components.2
                green = components.1
                blue = components.0
            case .rgb:
                red = components.0
                green = components.1
                blue = components.2
            default:
                return .clear
            }

            return .init(red: red, green: green, blue: blue, alpha: UInt8(255))

        } else {
            assertionFailure("Unsupported number of pixel components")
            return .clear
        }
    }

}

public extension UIColor {

    convenience init(red: UInt8, green: UInt8, blue: UInt8, alpha: UInt8) {
        self.init(
            red: CGFloat(red)/255,
            green: CGFloat(green)/255,
            blue: CGFloat(blue)/255,
            alpha: CGFloat(alpha)/255)
    }

}

public extension CGBitmapInfo {

    enum ComponentLayout {

        case bgra
        case abgr
        case argb
        case rgba
        case bgr
        case rgb

        var count: Int {
            switch self {
            case .bgr, .rgb: return 3
            default: return 4
            }
        }

    }

    var componentLayout: ComponentLayout? {
        guard let alphaInfo = CGImageAlphaInfo(rawValue: rawValue & Self.alphaInfoMask.rawValue) else { return nil }
        let isLittleEndian = contains(.byteOrder32Little)

        if alphaInfo == .none {
            return isLittleEndian ? .bgr : .rgb
        }
        let alphaIsFirst = alphaInfo == .premultipliedFirst || alphaInfo == .first || alphaInfo == .noneSkipFirst

        if isLittleEndian {
            return alphaIsFirst ? .bgra : .abgr
        } else {
            return alphaIsFirst ? .argb : .rgba
        }
    }

    var chromaIsPremultipliedByAlpha: Bool {
        let alphaInfo = CGImageAlphaInfo(rawValue: rawValue & Self.alphaInfoMask.rawValue)
        return alphaInfo == .premultipliedFirst || alphaInfo == .premultipliedLast
    }

}

Coffelt

What about changes in orientation? Don't you need to check UIImage.Orientation as well? – Martin

Also, your componentLayout is missing the case when there's only alpha, .alphaOnly. – Martin

@Martin "Alpha only" is not supported (for being a very rare case). Image orientation is out of scope of this code, but there's plenty resource about how to normalize the orientation of a UIImage, which is what you would do before starting to read its pixel colors. – Coffelt

Answer 8

UIImage is a wrapper the bytes are CGImage or CIImage

According the the Apple Reference on UIImage the object is immutable and you have no access to the backing bytes. While it is true that you can access the CGImage data if you populated the UIImage with a CGImage (explicitly or implicitly), it will return NULL if the UIImage is backed by a CIImage and vice-versa.

Image objects not provide direct access to their underlying image data. However, you can retrieve the image data in other formats for use in your app. Specifically, you can use the cgImage and ciImage properties to retrieve versions of the image that are compatible with Core Graphics and Core Image, respectively. You can also use the UIImagePNGRepresentation(:) and UIImageJPEGRepresentation(:_:) functions to generate an NSData object containing the image data in either the PNG or JPEG format.

Common tricks to getting around this issue

As stated your options are

• UIImagePNGRepresentation or JPEG
• Determine if image has CGImage or CIImage backing data and get it there

Neither of these are particularly good tricks if you want output that isn't ARGB, PNG, or JPEG data and the data isn't already backed by CIImage.

My recommendation, try CIImage

While developing your project it might make more sense for you to avoid UIImage altogether and pick something else. UIImage, as a Obj-C image wrapper, is often backed by CGImage to the point where we take it for granted. CIImage tends to be a better wrapper format in that you can use a CIContext to get out the format you desire without needing to know how it was created. In your case, getting the bitmap would be a matter of calling

- render:toBitmap:rowBytes:bounds:format:colorSpace:

As an added bonus you can start doing nice manipulations to the image by chaining filters onto the image. This solves a lot of the issues where the image is upside down or needs to be rotated/scaled etc.

Cleave

Answer 9

Building on Olie and Algal's answer, here is an updated answer for Swift 3

public func getRGBAs(fromImage image: UIImage, x: Int, y: Int, count: Int) -> [UIColor] {

var result = [UIColor]()

// First get the image into your data buffer
guard let cgImage = image.cgImage else {
    print("CGContext creation failed")
    return []
}

let width = cgImage.width
let height = cgImage.height
let colorSpace = CGColorSpaceCreateDeviceRGB()
let rawdata = calloc(height*width*4, MemoryLayout<CUnsignedChar>.size)
let bytesPerPixel = 4
let bytesPerRow = bytesPerPixel * width
let bitsPerComponent = 8
let bitmapInfo: UInt32 = CGImageAlphaInfo.premultipliedLast.rawValue | CGBitmapInfo.byteOrder32Big.rawValue

guard let context = CGContext(data: rawdata, width: width, height: height, bitsPerComponent: bitsPerComponent, bytesPerRow: bytesPerRow, space: colorSpace, bitmapInfo: bitmapInfo) else {
    print("CGContext creation failed")
    return result
}

context.draw(cgImage, in: CGRect(x: 0, y: 0, width: width, height: height))

// Now your rawData contains the image data in the RGBA8888 pixel format.
var byteIndex = bytesPerRow * y + bytesPerPixel * x

for _ in 0..<count {
    let alpha = CGFloat(rawdata!.load(fromByteOffset: byteIndex + 3, as: UInt8.self)) / 255.0
    let red = CGFloat(rawdata!.load(fromByteOffset: byteIndex, as: UInt8.self)) / alpha
    let green = CGFloat(rawdata!.load(fromByteOffset: byteIndex + 1, as: UInt8.self)) / alpha
    let blue = CGFloat(rawdata!.load(fromByteOffset: byteIndex + 2, as: UInt8.self)) / alpha
    byteIndex += bytesPerPixel

    let aColor = UIColor(red: red, green: green, blue: blue, alpha: alpha)
    result.append(aColor)
}

free(rawdata)

return result
}

swift

Dix

Answer 10

To access the raw RGB values of an UIImage in Swift 5 use the underlying CGImage and its dataProvider:

import UIKit

let image = UIImage(named: "example.png")!

guard let cgImage = image.cgImage,
    let data = cgImage.dataProvider?.data,
    let bytes = CFDataGetBytePtr(data) else {
    fatalError("Couldn't access image data")
}
assert(cgImage.colorSpace?.model == .rgb)

let bytesPerPixel = cgImage.bitsPerPixel / cgImage.bitsPerComponent
for y in 0 ..< cgImage.height {
    for x in 0 ..< cgImage.width {
        let offset = (y * cgImage.bytesPerRow) + (x * bytesPerPixel)
        let components = (r: bytes[offset], g: bytes[offset + 1], b: bytes[offset + 2])
        print("[x:\(x), y:\(y)] \(components)")
    }
    print("---")
}

https://www.ralfebert.de/ios/examples/image-processing/uiimage-raw-pixels/

Naoma

Answer 11

Based on different answers but mainly on this, this works for what I need:

UIImage *image1 = ...; // The image from where you want a pixel data
int pixelX = ...; // The X coordinate of the pixel you want to retrieve
int pixelY = ...; // The Y coordinate of the pixel you want to retrieve

uint32_t pixel1; // Where the pixel data is to be stored
CGContextRef context1 = CGBitmapContextCreate(&pixel1, 1, 1, 8, 4, CGColorSpaceCreateDeviceRGB(), kCGImageAlphaNoneSkipFirst);
CGContextDrawImage(context1, CGRectMake(-pixelX, -pixelY, CGImageGetWidth(image1.CGImage), CGImageGetHeight(image1.CGImage)), image1.CGImage);
CGContextRelease(context1);

As a result of this lines, you will have a pixel in AARRGGBB format with alpha always set to FF in the 4 byte unsigned integer pixel1.

Biracial