iOS Metal. Why does simply changing colorPixelFormat result in brighter imagery?

Question

In Metal on iOS the default colorPixelFormat is bgra8Unorm. When I change format to rgba16Float all imagery brightens. Why?

An example:

Artwork

MTKView with format bgra8Unorm. Texture-mapped quad. Texture created with SRGB=false.

MTKView with format rgba16Float. Texture-mapped quad. Texture created with SRGB=false.

Why is everything brighter with rgba16Float. My understanding is that SRGB=false implies that no gamma correction is done when importing artwork. The assumption is the artwork has no gamma applied.

What is going on here?

Hooked

You'll need to tell us what the data you're sending to the texture is. How do you convert it from 8bpp to 16bpp? Are you sending it as half float? – Beisel

Can you load the image data as sRGB and then capture it into a texture that is explicitly marked as sRGB? I would be willing to bet that the problem is not representing the value as linear float but actually the data is not being treated as sRGB all the way through your processing pipeline. Typically your input pixels would be sRGB so it is important to treat them as such in your pipeline. If your input is stored as sRGB pixel then your processing must remove the gamma correction when converting these values to linear, this is likely what is missing in your approach. – Agonized

MoDJ thanks for the insight. However, that does not address the fact that the background grey color is now brighter. In fact I did a test by applying a gamma function to the grey and it looks like the grey seen in the bgra8Unorm example. Why would that happen? – Hooked

When you take a gamma corrected image and incorrectly display it as linear, the result will be much "brighter" because the dark colors are not being adjusted down to their original "darker" values. This is why I am suggesting that you focus on your import path so that the original input sRGB values can be verified to be correct before you expand the values into 16 bit float in linear space. – Agonized

Ahh, I wasn't clear. The background grey is set as a color and is not part of the texture. That is a bit confusing. The texture is actually the 3x3 grid of figures with a transparent background. The color seen was set in my MTKView subclass as clearColor = MTLClearColor(red: .5, green: .5, blue: .5, alpha: 1.0). That produced the lighter then expected background. Using clearColor = MTLClearColor(red: value, green: value, blue: value, alpha: 1.0) where value is pow(0.5, 2.2) creates the darker - gamma'ed - grey. – Hooked

Well, sRGB does not use a linear gamma, so if you generate a color with pow(0.5, 2.2) then this may not exactly match your expectations. I don't see what this has to do with the problem that you are seeing which seems to be related to the original image pixels not being turned into linear data properly. Please try to fix your import so that the sRGB input pixels are treated as sRGB and are then are converted to linear 16 bit float data correctly. – Agonized

Answer 1

If your artwork has a gamma (it does per the first image you uploaded), you have to convert it to a linear gamma if you want to use it in a linear space.

What is happening here is you are displaying gamma encoded values of the image in a linear workspace, without using color management or transform to convert those values.

BUT: Reading some of your comments, is the texture not an image but an .svg?? Did you convert your color values to linear space?

Here's the thing: RGB values are meaningless numbers unless you define how those RGB values relate to a given space.

#00FF00 in sRGB is a different color than #00FF00 in Adobe98 for instance. In your case you are going linear, but what primaries? Still using sRGB primaries? P3 Primaries? I'm not seeing a real hue shift, so I assume you are using sRGB primaries and a linear transfer curve for the second example.

THAT SAID, an RGB value of the top middle kid's green shirt is #8DB54F, normalized to 0-1, that's 0.553 0.710 0.310 .These numbers by themselves don't know if they are gamma encoded or not.

THE RELATIONSHIP BETWEEN sRGB, Y, and Light:

For the purposes of this discussion, we will assume the SIMPLE sRGB gamma of 1/2.2 and not the piecewise version. Same for L*

In sRGB, #8DB54F when displayed on an sRGB monitor with a sRGB gamma curve, the luminance (Y) is 39

This can be found by

(0.553^2.2)*0.2126 + (0.710^2.2)*0.7152 + (0.310^2.2)*0.0722

or 0.057 + 0.33 + 0.0061 = 0.39 and 0.39 * 100 = 39 (Y)

But if color management is told the values are linear, then the gamma correction is discarded, and (more or less):

0.553*0.2126 + 0.710*0.7152 + 0.310*0.0722

or 0.1175 + 0.5078 + 0.0223 = 0.65 and 0.65 * 100 = 65 (Y)

(Assuming the same coefficients are used.)

Luminance (Y) is linear, like light. But human perception is not, and neither are sRGB values.

Y is the linear luminance from CIEXYZ, while it is spectrally weighted based on the eye's response to different wavelengths, it is NOT uniform in terms of lightness. On a scale of 0-100, 18.4 is perceived as the middle.

L* is a perceptual lightness from CIELAB (L* a* b*), it is (simplified curve of):

L* = Y^0.42 On a scale of 0-100, L* 50 is the "perceived middle" value. So that green shirt at Y 39 is L* 69 when interpreted and displayed as sRGB, and the Y 65 is about L* 84 (those numbers are based on the math, here are the values per the color picker on my MacBook):

sRGB is a gamma encoded signal, done to make the best use of the limited bit depth of 8bits per channel. The effective gamma curve is similar to human perception so that more bits are used to define darker areas as human perception is more sensitive to luminance changes in dark regions. As noted above it is a simplified curve of:

sRGB_Video = Linear_Video^0.455 (And to be noted, the MONITOR adds an exponent of about 1.1)

So if 0% is black and 100% is white, then middle gray, the point most humans will say is in between 0% and 100% is:

Y 18.4%. = L* 50% = sRGB 46.7%

That is, an sRGB hex value of #777777 will display a luminance of 18.4 Y, and is equivalent to a perceived lightness of 50 L*. Middle Grey.

BUT WAIT, THERE'S MORE

So what is happening, you are telling MTKView that you are sending it image data that references linear values. But you are actually sending it sRGB values which are lighter due to the applied gamma correction. And then color management is taking what it thinks are linear values, and transforming them to the needed values for the output display.

Color management needs to know what the values mean, what colorspace they relate to. When you set SRGB=false then you are telling it that you are sending it linear values, not gamma encoded values.

BUT you are clearly sending gamma encoded values into a linear space without transforming/decoding the values to linear. Linearization won't happen unless you implicitly do so.

SOLUTION

Linearize the image data OR set the flag SRGB=true

Please let me know if you have further questions. But also, you may wish to see the Poynton Gamma FAQ or also the Color FAQ for clarification.

Also, for your grey: A linear value of 0.216 is equivalent to an sRGB (0-1) value of 0.500

Loudermilk

Answer 2

I had to struggle with this issue recently and I finally figured out why changing the pixel format from bgra8Unorm to rgba16Float changes the visual appearance even though it should only be an implementation detail of how we are storing our pixel data.

The missing piece is how Core Animation composites the CAMetalLayer when no colorspace is specified. If the pixel format is bgra8Unorm, Core Animation assumes that your pixel data is in non-linear sRGB space. If the pixel format is rgba16Float, Core Animation assumes that your pixel data is in linear space. That's why all your colors appear brighter. Colors appearing too bright and washed out is a typical symptom of sRGB colors being interpreted as colors in the linear space.

The fix is actually quite straightforward. Simply set the colorspace of the CAMetalLayer to match the colorspace of your content (typically sRGB):

layer.colorspace = CGColorSpace(name: CGColorSpace.sRGB)!

and your content should look the same regardless of the pixel format.

Lytta