Color calibration with color checker using using Root-Polynomial Regression not giving correct results

Question

For a quantification project, I am in need of colour corrected images which produce the same result over and over again irrespective of lighting conditions.

Every image includes a X-Rite color-checker of which the colors are known in matrix format:

Reference=[[170, 189, 103],[46, 163, 224],[161, 133, 8],[52, 52, 52],[177, 128, 133],[64, 188, 157],[149, 86, 187],[85, 85, 85],[67, 108, 87],[108, 60, 94],[31, 199, 231],[121, 122, 122], [157, 122, 98],[99, 90, 193],[60, 54, 175],[160, 160, 160],[130, 150, 194],[166, 91, 80],[70, 148, 70],[200, 200, 200],[68, 82, 115],[44, 126, 214],[150, 61, 56],[242, 243, 243]]

For every image I calculate the same matrix for the color card present as an example:

Actual_colors=[[114, 184, 137], [2, 151, 237], [118, 131, 55], [12, 25, 41], [111, 113, 177], [33, 178, 188], [88, 78, 227], [36, 64, 85], [30, 99, 110], [45, 36, 116], [6, 169, 222], [53, 104, 138], [98, 114, 123], [48, 72, 229], [29, 39, 211], [85, 149, 184], [66, 136, 233], [110, 79, 90], [41, 142, 91], [110, 180, 214], [7, 55, 137], [0, 111, 238], [82, 44, 48], [139, 206, 242]]

Then I calibrate the entire image using a color correction matrix which was derived from the coefficient from the input and output matrices:

for im in calibrated_img:
    im[:]=colour.colour_correction(im[:], Actual_colors, Reference, "Finlayson 2015")

The results are as follows:

Where the top image represents the input and the down image the output. Lighting plays a key role in the final result for the color correction, but the first two images on the left should generate the same output. Once the images become too dark, white is somehow converted to red.. I am not able to understand why.

I have tried to apply a gamma correction before processing with no success. The other two models Cheung 2004 and Vandermonde gave worse results, as did partial least squares. The images are pretty well corrected from the yellow radiating lamps, but the final result is not clean white, instead they have a blueish haze over the image. White should be white.. What can I do to further improve these results?

---

Edit 23-08-2020: Based on @Kel Solaar his comments I have made changes to my script to include the steps mentioned by him as follows

#Convert image from int to float
Float_image=skimage.img_as_float(img)

#Normalise image to have pixel values from 0 to 1
Normalised_image = (Float_image - np.min(Float_image))/np.ptp(Float_image)

#Decoded the image with sRGB EOTF
Decoded_img=colour.models.eotf_sRGB(Normalised_image)  

#Performed Finlayson 2015 color correction to linear data:
for im in Decoded_img:
    im[:]=colour.colour_correction(im[:], Image_list, Reference, "Finlayson 2015")

#Encoded image back to sRGB
Encoded_img=colour.models.eotf_inverse_sRGB(Decoded_img)  

#Denormalized image to fit 255 pixel values
Denormalized_image=Encoded_img*255

#Converted floats back to integers
Integer_image=Denormalised_image.astype(int)

This greatly improved image quality as can be seen below:

However, lighting/color differences between corrected images are unfortunately still present.

Raw images can be found here but due note that they are upside down.

Measured values of color cards in images:

IMG_4244.JPG
[[180, 251, 208], [62, 235, 255], [204, 216, 126], [30, 62, 97], [189, 194, 255], [86, 250, 255], [168, 151, 255], [68, 127, 167], [52, 173, 193], [111, 87, 211], [70, 244, 255], [116, 185, 228], [182, 199, 212], [102, 145, 254], [70, 102, 255], [153, 225, 255], [134, 214, 255], [200, 156, 169], [87, 224, 170], [186, 245, 255], [44, 126, 235], [45, 197, 254], [166, 101, 110], [224, 255, 252]]

IMG_4243.JPG
[[140, 219, 168], [24, 187, 255], [148, 166, 73], [17, 31, 53], [141, 146, 215], [42, 211, 219], [115, 101, 255], [33, 78, 111], [24, 118, 137], [63, 46, 151], [31, 203, 255], [67, 131, 172], [128, 147, 155], [61, 98, 255], [42, 59, 252], [111, 181, 221], [88, 168, 255], [139, 101, 113], [47, 176, 117], [139, 211, 253], [19, 78, 178], [12, 146, 254], [110, 60, 64], [164, 232, 255]]

IMG_4241.JPG
[[66, 129, 87], [0, 90, 195], [65, 73, 26], [9, 13, 18], [60, 64, 117], [20, 127, 135], [51, 38, 176], [15, 27, 39], [14, 51, 55], [21, 15, 62], [1, 112, 180], [29, 63, 87], [54, 67, 69], [20, 33, 179], [10, 12, 154], [38, 92, 123], [26, 81, 178], [58, 44, 46], [23, 86, 54], [67, 127, 173], [5, 26, 77], [2, 64, 194], [43, 22, 25], [84, 161, 207]]

IMG_4246.JPG
[[43, 87, 56], [2, 56, 141], [38, 40, 20], [3, 5, 6], [31, 31, 71], [17, 85, 90], [19, 13, 108], [7, 13, 20], [4, 24, 29], [8, 7, 33], [1, 68, 123], [14, 28, 46], [28, 34, 41], [6, 11, 113], [0, 1, 91], [27, 53, 83], [11, 44, 123], [32, 21, 23], [11, 46, 26], [32, 77, 115], [2, 12, 42], [0, 29, 128], [20, 9, 11], [49, 111, 152]]

Actual colors of color card (or reference) are given in the top of this post and are in the same order as values given for images.

---

Edit 30-08-2020, I have applied @nicdall his comments:

#Remove color chips which are outside of RGB range
New_reference=[]
New_Actual_colors=[]
for L,K in zip(Actual_colors, range(len(Actual_colors))):
    if any(m in L for m in [0, 255]):
        print(L, "value outside of range")
    else:
        New_reference.append(Reference[K])
        New_Actual_colors.append(Actual_colors[K])

In addition to this, I realized I was using a single pixel from the color card, so I started to take 15 pixels per color chip and averaged them to make sure it is a good balance. The code is too long to post here completely but something in this direction (don't judge my bad coding here):

for i in Chip_list:
    R=round(sum([rotated_img[globals()[i][1],globals()[i][0],][0],
        rotated_img[globals()[i][1]+5,globals()[i][0],][0],
        rotated_img[globals()[i][1]+10,globals()[i][0],][0],
        rotated_img[globals()[i][1],(globals()[i][0]+5)][0],
        rotated_img[globals()[i][1],(globals()[i][0]+10)][0],
        rotated_img[globals()[i][1]+5,(globals()[i][0]+5)][0],
        rotated_img[globals()[i][1]+10,(globals()[i][0]+10)][0]])/(number of pixels which are summed up))

The result was dissapointing, as the correction seemed to have gotten worse but it is shown below:

New_reference = [[170, 189, 103], [161, 133, 8], [52, 52, 52], [177, 128, 133], [64, 188, 157], [85, 85, 85], [67, 108, 87], [108, 60, 94], [121, 122, 122], [157, 122, 98], [60, 54, 175], [160, 160, 160], [166, 91, 80], [70, 148, 70], [200, 200, 200], [68, 82, 115], [44, 126, 214], [150, 61, 56]]

#For Image: IMG_4243.JPG:
New_Actual_colors= [[139, 218, 168], [151, 166, 74], [16, 31, 52], [140, 146, 215], [44, 212, 220], [35, 78, 111], [25, 120, 137], [63, 47, 150], [68, 132, 173], [128, 147, 156], [40, 59, 250], [110, 182, 222], [141, 102, 115], [48, 176, 118], [140, 211, 253], [18, 77, 178], [12, 146, 254], [108, 59, 62]]

#The following values were omitted in IMG_4243:
[23, 187, 255] value outside of range
[115, 102, 255] value outside of range
[30, 203, 255] value outside of range
[61, 98, 255] value outside of range
[88, 168, 255] value outside of range
[163, 233, 255] value outside of range

I have started to approach the core of the problem but I am not a mathematician, however the correction itself seems to be the problem.. This is the color correction matrix for IMG4243.jpg generated and utilized by the colour package:

CCM=colour.characterisation.colour_correction_matrix_Finlayson2015(New_Actual_colors, New_reference, degree=1 ,root_polynomial_expansion=True)
print(CCM)
[[ 1.10079803 -0.03754644  0.18525637]
 [ 0.01519612  0.79700086  0.07502735]
 [-0.11301282 -0.05022718  0.78838144]]

Based on what I understand from the colour package code the New_Actual_colors is converted with the CCM as follows:

Converted_colors=np.reshape(np.transpose(np.dot(CCM, np.transpose(New_Actual_colors))), shape)

When we compare the Converted_colors with the New_reference, we can see that the correction is getting a long way, but differences are still present (so the endgoal is to convert New_Actual_colors with the color correction matrix (CCM) to Converted_colors which should exactly match the New_reference):

print("New_reference =",New_reference)
print("Converted_colors =",Converted_colors)
New_reference =    [[170, 189, 103],[161, 133, 8],[52, 52, 52],[177, 128, 133],[64, 188, 157],[85, 85, 85],[67, 108, 87],[108, 60, 94],[121, 122, 122],[157, 122, 98],[60, 54, 175],[160, 160, 160],[166, 91, 80],[70, 148, 70],[200, 200, 200],[68, 82, 115],[44, 126, 214],[150, 61, 56]]
Converted_colors = [[176, 188, 106],[174, 140, 33],[26, 29, 38],[188, 135, 146],[81, 186, 158],[56, 71, 80],[48, 106, 99],[95, 50, 109],[102, 119, 122],[164, 131, 101],[88, 66, 190],[155, 163, 153],[173, 92, 70],[68, 150, 79],[193, 189, 173],[50, 75, 134],[55, 136, 192],[128, 53, 34]]

When substracted the differences become clear, and the question is how to overcome these differences?:

list(np.array(New_reference) - np.array(Converted_colors))
[array([-6,  1, -3]),
 array([-13,  -7, -25]),
 array([26, 23, 14]),
 array([-11,  -7, -13]),
 array([-17,   2,  -1]),
 array([29, 14,  5]),
 array([ 19,   2, -12]),
 array([ 13,  10, -15]),
 array([19,  3,  0]),
 array([-7, -9, -3]),
 array([-28, -12, -15]),
 array([ 5, -3,  7]),
 array([-7, -1, 10]),
 array([ 2, -2, -9]),
 array([ 7, 11, 27]),
 array([ 18,   7, -19]),
 array([-11, -10,  22]),
 array([22,  8, 22])]

Embrocation

We had an implementation issue with the Root-Polynomial variant from Finlayson (2015). Which version of Colour are you using? Keep in mind that polynomial methods work really best with large amount of samples because anything not in the dataset is subject to extrapolation which in turns might result in rapid value explosions. – Washout

Thank you for your response. Currently I am running colour-science==0.3.15. The method I am currently using applies the Finlayson model pixel by pixel in the image. I would not know how to apply it to multiple samples for colour correction.. – Embrocation

Sorry but I am not able to understand your question properly. Correct me if I am wrong. You want to convert all of the images to similar to a reference image. For each image, you have the values of colours of the colour grid present in the image, and use you have the same values of the reference image. You wish to convert colours of the current image -> reference image using this colour grid values. – Wilburn

Also, please provide these images separately and the values of colours in it and the values of colours of the reference image. – Wilburn

I am sorry for the vague bounty explanation, after submission I was not able to edit it. Basically I would like to color correct all 4 input images (taken with different shutter speeds, top images in examples above this message) in such a way that (almost) the same output image is generated based on the true values of the color card. There is no reference image, the reference should be the color card of which the actual values are known. The raw images can be found here: imgur.com/a/p6AaGVe (do note they are upside down). – Embrocation

@Rahul Kedia I have added the color card values of the images into the original post! – Embrocation

I think your second approach is failing because the colors in the color chart are not covering all the color ranges. Due to which it is not able to do a proper transformation.I have an approach that finds the transformation matrix with the given color data but it is also failing in the same way because of the same reason. Try changing the color chart if possible and take a bigger one. My approach works perfectly fine for me but in that, I am currently using a color chart of 22*12 size. Its not necessary to take this big chart but at least take a chart which covers all the ranges of the colors. – Wilburn

Thank you for your effort. Could you advise me which 22*12 color chart you have? I seem to only find the conventional ones and the 14*10 (X-Rite Digital ColorChecker SG). I thought the 8*4 color card was designed for these kinds of approached but I might be wrong then.. – Embrocation

Answer 1

Here are a few recommendations:

• As stated in my comment above we had an implementation issue with the Root-Polynomial variant from Finlayson (2015) which should be fixed in the develop branch.
• You are passing integer and encoded values to the colour.colour_correction definition. I would strongly recommend that you:
  • Convert the datasets to floating-point representation.
  • Scale it from range [0, 255] to range [0, 1].
  • Decode it with the sRGB EOTF.
  • Perform the colour correction onto that linear data.
  • Encode back and scale back to integer representation.
• Your images seem to be an exposure wedge, ideally, you would compute a single matrix for the appropriate reference exposure, normalise the other images exposure to it and apply the matrix on it.

Washout

Thank you for your comments! I have implemented the suggestions from your second point. Regarding your third comment, the exposure wedge, I had noticed and lighting should be more uniform overall. I am not sure if I understand your concept here. With computing a matrix for the reference exposure, you mean a color card in the image? How would you then normalise the other images exposure? Anyhow I do still think color correction should be applicable as well to the given example in the question.. – Embrocation

> With computing a matrix for the reference exposure, you mean a color card in the image? Yes! I was talking about exposure normalisation because most of the polynomials methods are not exposure invariant, although the one your picked should be, i.e. Finlayson (2015) with Root-Polynomials. – Washout

Answer 2

An additional recommendation more on the physical side of the problem : I see some of the RGB values in the high and low exposure images are outside of the unsaturated range of the camera (0 and 255 values). This means that some information on the actual measured color is lost at the time of the image capture, because some of the calibration patches are either over- or under-exposed. This is a known problem in RGB colorimetry, and it is actually mentioned in (Finlayson, 2015) : "an additional assumption is that both v and kv are in the unsaturated range of the camera"

If possible, try to have a look at the histogram while you take the images so that all pixels have a value in the unsaturated range ([1, 254] at most).

Otherwise, if taking new images is out of the question, you can try ignoring the saturated patch (which have either 0 or 255 in any of R, G or B values) in the calibration process (make sure that you ignore the patches both in the image and in the reference). This could improve your calibration for the overall image as you do not make your model fit saturated values.

Menam