Getting pixel format from CGImage
Asked Answered
P

3

15

I understand bitmap layout and pixel format subject pretty well, but getting an issue when working with png / jpeg images loaded through NSImage – I can't figure out if what I get is the intended behaviour or a bug.

let nsImage:NSImage = NSImage(byReferencingURL: …)
let cgImage:CGImage = nsImage.CGImageForProposedRect(nil, context: nil, hints: nil)!
let bitmapInfo:CGBitmapInfo = CGImageGetBitmapInfo(cgImage)
Swift.print(bitmapInfo.contains(CGBitmapInfo.ByteOrderDefault)) // True

My kCGBitmapByteOrder32Host is little endian, which implies that the pixel format is also little endian – BGRA in this case. But… png format is big endian by specification, and that's how the bytes are actually arranged in the data – opposite from what bitmap info tells me.

Does anybody knows what's going on? Surely the system somehow knows how do deal with this, since pngs are displayed correctly. Is there a bullet-proof way detecting pixel format of CGImage? Complete demo project is available at GitHub.


P. S. I'm copying raw pixel data via CFDataGetBytePtr buffer into another library buffer, which is then gets processed and saved. In order to do so, I need to explicitly specify pixel format. Actual images I'm dealing with (any png / jpeg files that I've checked) display correctly, for example:

But bitmap info of the same images gives me incorrect endianness information, resulting in bitmap being handled as BGRA pixel format instead of actual RGBA, when I process it the result looks like this:

The resulting image demonstrates the colour swapping between red and blue pixels, if RGBA pixel format is specified explicitly, everything works out perfectly, but I need this detection to be automated.


P. P. S. Documentation briefly mentions that CGColorSpace is another important variable that defines pixel format / byte order, but I found no mentions how to get it out of there.

Proconsul answered 4/9, 2016 at 19:1 Comment(6)
Does the image draw incorrectly? If it draws correctly, then it's safe to assume the image loader is swapping the bytes to host order.Sapajou
Rob, updated the question with an example. Image does display correctly, AppKit somehow knows that byte order is big endian (RGB), but bitmap info tells that its little endian (BGR). The loader doesn't change the byte order, but image information indicates that bytes are already swapped.Proconsul
"bitmap info of the same images gives me incorrect endianness information" are you saying that the apple CGImage class made from the original PNG or JPG file is not giving you the right big/little endian information? Or Is it incorrect AFTER you created the image after you made a copy of the buffer into the Other library object? If it's the second case, I would recommend to check out this library source code... might do something wrong with your data OR maybe you need to pass through the endianness to the library explicitly (after all it s not even sure you can detect it with Bytes only.)Ieper
The code above shows what's going on – CGBitmapInfo gives wrong endianness. The library is FFmpeg, I checked everywhere and narrowed down the issue to CoreGraphics. At the moment that's what I do – explicitly provide byte order, but this doesn't guarantee that it will be correct for all use cases, i.e., different image data sources use different byte order. Also added a link to a demo repository.Proconsul
Same problem for me. Trying to convert images to WebP, and some photos have swapped blue/red, CGBitmapInfo is always 5 =( Have you find a solution?Huddleston
See my answer below. 👍Proconsul
P
21

Some years later and after testing my findings in production I can share them with good confidence, but hoping someone with theory knowledge will explain things better here? Good places to refresh memory:

Based on that you can use following extensions:

public enum PixelFormat
{
    case abgr
    case argb
    case bgra
    case rgba
}

extension CGBitmapInfo
{
    public static var byteOrder16Host: CGBitmapInfo {
        return CFByteOrderGetCurrent() == Int(CFByteOrderLittleEndian.rawValue) ? .byteOrder16Little : .byteOrder16Big
    }

    public static var byteOrder32Host: CGBitmapInfo {
        return CFByteOrderGetCurrent() == Int(CFByteOrderLittleEndian.rawValue) ? .byteOrder32Little : .byteOrder32Big
    }
}

extension CGBitmapInfo
{
    public var pixelFormat: PixelFormat? {

        // AlphaFirst – the alpha channel is next to the red channel, argb and bgra are both alpha first formats.
        // AlphaLast – the alpha channel is next to the blue channel, rgba and abgr are both alpha last formats.
        // LittleEndian – blue comes before red, bgra and abgr are little endian formats.
        // Little endian ordered pixels are BGR (BGRX, XBGR, BGRA, ABGR, BGR).
        // BigEndian – red comes before blue, argb and rgba are big endian formats.
        // Big endian ordered pixels are RGB (XRGB, RGBX, ARGB, RGBA, RGB).

        let alphaInfo: CGImageAlphaInfo? = CGImageAlphaInfo(rawValue: self.rawValue & type(of: self).alphaInfoMask.rawValue)
        let alphaFirst: Bool = alphaInfo == .premultipliedFirst || alphaInfo == .first || alphaInfo == .noneSkipFirst
        let alphaLast: Bool = alphaInfo == .premultipliedLast || alphaInfo == .last || alphaInfo == .noneSkipLast
        let endianLittle: Bool = self.contains(.byteOrder32Little)

        // This is slippery… while byte order host returns little endian, default bytes are stored in big endian
        // format. Here we just assume if no byte order is given, then simple RGB is used, aka big endian, though…

        if alphaFirst && endianLittle {
            return .bgra
        } else if alphaFirst {
            return .argb
        } else if alphaLast && endianLittle {
            return .abgr
        } else if alphaLast {
            return .rgba
        } else {
            return nil
        }
    }
}

Note, that you should always pay attention to colour space – it directly affects how raw pixel data is stored. CGColorSpace(name: CGColorSpace.sRGB) is probably the safest one – it stores colours in plain format, for example, if you deal with red RGB it will be stored just like that (255, 0, 0) while device colour space will give you something like (235, 73, 53).

To see this in practice drop above and the following into a playground. You'll need two one-pixel red images with alpha and without, this and this should work.

import AppKit
import CoreGraphics

extension CFData
{
    public var pixelComponents: [UInt8] {
        let buffer: UnsafeMutablePointer<UInt8> = UnsafeMutablePointer.allocate(capacity: 4)
        defer { buffer.deallocate(capacity: 4) }
        CFDataGetBytes(self, CFRange(location: 0, length: CFDataGetLength(self)), buffer)
        return Array(UnsafeBufferPointer(start: buffer, count: 4))
    }
}

let color: NSColor = .red
Thread.sleep(forTimeInterval: 2)

// Must flip coordinates to capture what we want…
let screen: NSScreen = NSScreen.screens.first(where: { $0.frame.contains(NSEvent.mouseLocation) })!
let rect: CGRect = CGRect(origin: CGPoint(x: NSEvent.mouseLocation.x - 10, y: screen.frame.height - NSEvent.mouseLocation.y), size: CGSize(width: 1, height: 1))

Swift.print("Will capture image with \(rect) frame.")

let screenImage: CGImage = CGWindowListCreateImage(rect, [], kCGNullWindowID, [])!
let urlImageWithAlpha: CGImage = NSImage(byReferencing: URL(fileURLWithPath: "/Users/ianbytchek/Downloads/red-pixel-with-alpha.png")).cgImage(forProposedRect: nil, context: nil, hints: nil)!
let urlImageNoAlpha: CGImage = NSImage(byReferencing: URL(fileURLWithPath: "/Users/ianbytchek/Downloads/red-pixel-no-alpha.png")).cgImage(forProposedRect: nil, context: nil, hints: nil)!

Swift.print(screenImage.colorSpace!, screenImage.bitmapInfo, screenImage.bitmapInfo.pixelFormat!, screenImage.dataProvider!.data!.pixelComponents)
Swift.print(urlImageWithAlpha.colorSpace!, urlImageWithAlpha.bitmapInfo, urlImageWithAlpha.bitmapInfo.pixelFormat!, urlImageWithAlpha.dataProvider!.data!.pixelComponents)
Swift.print(urlImageNoAlpha.colorSpace!, urlImageNoAlpha.bitmapInfo, urlImageNoAlpha.bitmapInfo.pixelFormat!, urlImageNoAlpha.dataProvider!.data!.pixelComponents)

let formats: [CGBitmapInfo.RawValue] = [
    CGImageAlphaInfo.premultipliedFirst.rawValue,
    CGImageAlphaInfo.noneSkipFirst.rawValue,
    CGImageAlphaInfo.premultipliedLast.rawValue,
    CGImageAlphaInfo.noneSkipLast.rawValue,
]

for format in formats {

    // This "paints" and prints out components in the order they are stored in data.

    let context: CGContext = CGContext(data: nil, width: 1, height: 1, bitsPerComponent: 8, bytesPerRow: 32, space: CGColorSpace(name: CGColorSpace.sRGB)!, bitmapInfo: format)!
    let components: UnsafeBufferPointer<UInt8> = UnsafeBufferPointer(start: context.data!.assumingMemoryBound(to: UInt8.self), count: 4)

    context.setFillColor(red: 1 / 0xFF, green: 2 / 0xFF, blue: 3 / 0xFF, alpha: 1)
    context.fill(CGRect(x: 0, y: 0, width: 1, height: 1))
    Swift.print(context.colorSpace!, context.bitmapInfo, context.bitmapInfo.pixelFormat!, Array(components))
}

This will output the following. Pay attention how screen-captured image differs from ones loaded from disk.

Will capture image with (285.7734375, 294.5, 1.0, 1.0) frame.
<CGColorSpace 0x7fde4e9103e0> (kCGColorSpaceICCBased; kCGColorSpaceModelRGB; iMac) CGBitmapInfo(rawValue: 8194) bgra [27, 13, 252, 255]
<CGColorSpace 0x7fde4d703b20> (kCGColorSpaceICCBased; kCGColorSpaceModelRGB; Color LCD) CGBitmapInfo(rawValue: 3) rgba [235, 73, 53, 255]
<CGColorSpace 0x7fde4e915dc0> (kCGColorSpaceICCBased; kCGColorSpaceModelRGB; Color LCD) CGBitmapInfo(rawValue: 5) rgba [235, 73, 53, 255]
<CGColorSpace 0x7fde4d60d390> (kCGColorSpaceICCBased; kCGColorSpaceModelRGB; sRGB IEC61966-2.1) CGBitmapInfo(rawValue: 2) argb [255, 1, 2, 3]
<CGColorSpace 0x7fde4d60d390> (kCGColorSpaceICCBased; kCGColorSpaceModelRGB; sRGB IEC61966-2.1) CGBitmapInfo(rawValue: 6) argb [255, 1, 2, 3]
<CGColorSpace 0x7fde4d60d390> (kCGColorSpaceICCBased; kCGColorSpaceModelRGB; sRGB IEC61966-2.1) CGBitmapInfo(rawValue: 1) rgba [1, 2, 3, 255]
<CGColorSpace 0x7fde4d60d390> (kCGColorSpaceICCBased; kCGColorSpaceModelRGB; sRGB IEC61966-2.1) CGBitmapInfo(rawValue: 5) rgba [1, 2, 3, 255]

Playground

Proconsul answered 3/3, 2018 at 17:51 Comment(0)
I
1

Could you use NSBitmapFormat?

I wrote a class to source color schemes from images, and that's what I used to determine the bitmap format. Here's a snippet of how I used it:

var averageColorImage: CIImage?
var averageColorImageBitmap: NSBitmapImageRep

//... core image filter code

averageColorImage = filter?.outputImage

averageColorImageBitmap = NSBitmapImageRep(CIImage: averageColorImage!)

let red, green, blue: Int
switch averageColorImageBitmap.bitmapFormat {

    case NSBitmapFormat.NSAlphaFirstBitmapFormat:
        red = Int(averageColorImageBitmap.bitmapData.advancedBy(1).memory)
        green = Int(averageColorImageBitmap.bitmapData.advancedBy(2).memory)
        blue = Int(averageColorImageBitmap.bitmapData.advancedBy(3).memory)
    default:
        red = Int(averageColorImageBitmap.bitmapData.memory)
        green = Int(averageColorImageBitmap.bitmapData.advancedBy(1).memory)
        blue = Int(averageColorImageBitmap.bitmapData.advancedBy(2).memory)
}
Indigoid answered 8/9, 2016 at 20:27 Comment(7)
I'm interested in endianness, not alpha order. Haven't checked CIImage since dealing with CGImage, but might be an alternative. Will look into this and get back. Thanks!Proconsul
Good luck, no idea if you'll get different results from CIImage, but the NSBitmapFormat enum has some big/little endian cases.Indigoid
Nope, this didn't work at all – not creating NSBitmapImageRep, not getting the first one from NSImage, in both cases NSBitmapFormat is 0 and doesn't contain any endianness information whatsoever. I'm discussing this with Apple support, will post further info.Proconsul
Let's not put bounty to waste. I'd give another 100 just for being from Wyoming!Proconsul
Haha thanks @IanBytchek! I think that probably makes me Wyoming's #1 StackOverflow user :). I am going to try and confirm your findings about NSBitmatFormat. I have a project using it with drag/drop configured, so I should be able to print the value pretty easily. It needs updated to Swift 3 and has some dependencies, so it may be a few days though. I'll comment and tag you.Indigoid
@IanBytchek Did you ever find the solution? I'm too am trying to find the bitmap format from a CGImage. Please let me knowCarty
No, nothing solid unfortunately. Would appreciate any details if you able to find out more.Proconsul
C
0

Check out the answer to How to keep NSBitmapImageRep from creating lots of intermediate CGImages?.

The gist is that the NSImage/NSBitmapImageRepresentation implementation automatically handles the input format.

Apple's docs fail to note that the format parameter (for example in CIRenderDestination) specifies the desired output space.

If you want it in a particular format, the docs recommend drawing into that format (example in linked answer).

If you just need particular information, NSBitmapImageRepresentation provides easy access to individual parameters. I could not find a clear and direct route to a CIFormat without setting up cascading manual tests. I assume a way exists somewhere.

Congenial answered 18/4, 2022 at 21:23 Comment(1)
Thanks for the follow up on this. That's a great note on "drawing vs. copying" and it works great when working within Apple's ecosystem. But there are situations when you actually need the raw data and you need to know what format it is in, in this case when copying data to another non-Apple graphic processing framework. Here you would want to avoid fiddling with the data to get it in the "expected" format, because it would be a very expensive operation in real-time.Proconsul

© 2022 - 2024 — McMap. All rights reserved.