ImageMagick compare: Disregard white matches from the PSNR result

Question

I'm using compare to diff two similar color PNG files. They get a PSNR value of ~27.

The images contain a lot of white areas that will always match between the two images. Correct me if i'm wrong, but these white areas are increasing the PSNR values (making the images more similar as an average). Therefore I don't want to take white pixels into account, IF they match.

Is there a way to do this using compare or by factoring in some other tool?

I tried setting -transparent-color to white, but this had no effect.

Here is my command:

compare -compose src -metric PSNR img1.png img2.png diff.png

Edit:

compare --version
Version: ImageMagick 6.6.9-7 2014-03-06 Q16 http://www.imagemagick.org

convert --version
Version: ImageMagick 6.6.9-7 2014-03-06 Q16 http://www.imagemagick.org

My idea about ignoring white pixels may be rubbish, but I still want to achieve a better comparison.

To clarify:
Imagine this situation: you compare two images, they differ a bit. You then add a big white border to those images. You diff again and find that the resultant PSNR value has risen some amount. What I want is a way to negate this. Obviously the white borders are the same in both images...

Pip

im trying to compare the "important" parts of my image, and ignore other parts (which happen to be white) – Pip

Answer 1

Like Mark Setchell, I do not fully understand what your real intention is. You do not provide any (links to) sample images which may help us to grasp what you are up to.

Preparation

That's why I created 4 sample images myself. Here is the first pair:

• The left one is a JPEG.
• The right one is a PNG.

Differences are minimal -- you'll hardly be able to spot them with the naked eye when reproduced here. Both images are 482x642 pixels:

 

Here comes the second pair. Again,

• the left one is a JPEG;
• the right one is a PNG.

These are both enlarged to 1002x1002 pixels by adding a white frame around the initial pair. In the following reproduction they appear smaller because of the different scaling inside this web page:

 

So the "important" parts of the second pair are what is contained in the black frames. The white frames on the outer bound is identical.

Comparison

Now lets compare both pairs. But I do not want to limit the comparisons to just the PSNR metric. I want to see all available metrics. We can list available metrics with this command:

compare -list metric
 AE
 Fuzz
 MAE
 MEPP
 MSE
 NCC
 PAE
 PHASH
 PSNR
 RMSE

My command to return the metrics for the first pair is this:

for metric in $(compare -list metric) ; do  \
   echo -n "Metric ${metric} :  " ;         \
   compare                                  \
      -metric ${metric}                     \
       https://i.sstatic.net/TEjAd.jpg   \
       https://i.sstatic.net/p8JsE.png   \
       null: ;                              \
   echo ;                                   \
done

Here is the result (slightly re-formatted) for the first pair ("important" part only):

Metric AE    :  123789
Metric Fuzz  :     948.522      (0.0144735)
Metric MAE   :     381.318      (0.00581854)
Metric MEPP  :       3.5399e+08 (0.000209349, 0.32549)
Metric MSE   :      13.7285     (0.000209483)
Metric NCC   :       0.998307
Metric PAE   :   21331          (0.32549)
Metric PHASH :       5.43771
Metric PSNR  :      36.7885
Metric RMSE  :     948.522      (0.0144735)

My command to return the metrics for the second pair is this:

for metric in $(compare -list metric) ; do  \
   echo -n "Metric ${metric} :  " ;         \
   compare                                  \
      -metric ${metric}                     \
       https://i.sstatic.net/gBruS.jpg   \
       https://i.sstatic.net/8NJeB.png   \
       null: ;                              \
   echo ;                                   \
done

Here is the result (slightly re-formatted) for the second pair ("watered down" differences by added massive white frame):

Metric AE    :  133609
Metric Fuzz  :     611.952       (0.00933779)
Metric MAE   :     143.849       (0.00219499)
Metric MEPP  :       4.33273e+08 (8.71895e-05, 0.341176)
Metric MSE   :       5.71428     (8.71944e-05)
Metric NCC   :       0.998137
Metric PAE   :   22359           (0.341176)
Metric PHASH :       0.360076
Metric PSNR  :      40.5951
Metric RMSE  :     611.952       (0.00933779)

Here are both results in a common table:

+==============+=======================================+=========================================+
| Metric Type  | Results for "important" image parts   | Results including "unimportant" frames  |
+==============+=======================================+=========================================+
| Metric AE    | 123789                                | 133609                                  |
| Metric Fuzz  |    948.522      (0.0144735)           |    611.952       (0.00933779)           |
| Metric MAE   |    381.318      (0.00581854)          |    143.849       (0.00219499)           |
| Metric MEPP  |      3.5399e+08 (0.000209349, 0.32549)|      4.33273e+08 (8.71895e-05, 0.341176)|
| Metric MSE   |     13.7285     (0.000209483)         |      5.71428     (8.71944e-05)          |
| Metric NCC   |      0.998307                         |      0.998137                           |
| Metric PAE   |  21331          (0.32549)             |  22359           (0.341176)             |
| Metric PHASH |      5.43771                          |      0.360076                           |
| Metric PSNR  |     36.7885                           |     40.5951                             |
| Metric RMSE  |    948.522      (0.0144735)           |    611.952       (0.00933779)           |
+==============+========================================+=========================================+

^{Note: comparing two identical images with the PSNR metric would result in an inf (infinitiv) value.}

Now draw your own conclusions...

Discussion

Understanding comparison metrics is not a straight forward affair.

Your own understanding of the PSNR seems to be a bit off, from how I interpret your 'Edit:'

To clarify:
Imagine this situation: you compare two images, they differ a bit. You then add a big white border to those images. You diff again and find that the resultant PSNR value has risen some amount.

Because a rising value for PSNR means that the two compared images have become a bit more identical to each other! (Of course, your intention to remove white (or otherwise colored) frames/borders around images before comparing them is still a reasonable approach. To see how to do that, look at the end of my answer...)

In order to get a better feeling for image comparison metrics, you should create a few simple "images" first. Then start experimenting with these.

Here is a suggestion how to create a series of mono-colored "patches", sized 100x100 pixels each:

for col in black white blue green red; do               \
  convert -size 100x100 xc:${col} xc-100px-${col}.png ; \
done

    

Experimental Task: Compare each of the 100x100 pixels patches against each other.
Questions:

• What is notable if you juxtapose the "black-white" metrics against the "black-blue" or the "red-green" metrics?
• Are some results unexpected by you? Why so?

Now do the same for patches with 200x200 pixels:

for col in black white blue green red; do               \
  convert -size 200x200 xc:${col} xc-200px-${col}.png ; \
done

Experimental Task: Compare each of the 200x200 pixels patches against each other.
Questions:

• Do the respective "color1-color2" metrics for the 200x200 images deviate from the same "color1-color2" metrics of the 100x100 images?
• Are there some metrics which return identical results to their respective counter-parts? Why so?

Now add a 50 pixels wide red frame around each of the 100x100 pixel patches. The resulting images will also be of size 200x200 pixels:

for img in xc-100px-*.png ; do \
   convert                     \
     ${img}                    \
    -mattecolor red            \
    -frame 50x50               \
     redframed-${img} ;        \
done

Experimental Task: Make up your own comparison pairs. (You know can also compare 200x200 pixel 'mono-color' patches against 200x200 pixels 'redframed' patches...)
Questions:

• Which are reasonable comparisons?
• Did you know that the -metric phash is the only one which allows you to compare images using different dimensions (width x height) ?

---

How to 'trim' a mono-colored frame around images

You can remove any 'frame' around an image that is composed of identically colored pixels. The image operator -trim will achieve this automatically for you. (It works for colors different than white too.)

convert reframed-xc-100px-blue.png -trim +repage output.png

identify redframed-xc-100px-blue.png  output.png
 redframed-xc-100px-blue.png PNG 200x200 200x200+0+0 8-bit sRGB 3c 322B 0.000u 0:00.000
               output.png[1] PNG 100x100 100x100+0+0 8-bit sRGB 2c 285B 0.000u 0:00.000

 

Cabrales

the -trim option looks very promising, il run some tests – Pip

@ldgorman: Don't forget that you can add -fuzz X% to have also pixels trimmed off from the edges which deviate by X% from the exact color. And add +repage to reset the canvas size... – Cabrales

Answer 2

Updated

Ok, how about we make a mask of the areas you want to be compared first? So, if you want to ignore areas where both images are white, you can do this:

convert a.png b.png                   \
       -colorspace gray               \
       -compose multiply -composite   \
       -threshold 65534               \
       -negate PNG8:mask.png

Then, when you do your comparison, mask the images beforehand:

convert \( a.png mask.png -compose copy-opacity -composite \) \
        \( b.png mask.png -compose copy-opacity -composite \) \
        -metric PSNR -compare diff.png

or with compare like this in bash:

compare -metric PSNR \
    <(convert a.png mask.png -compose copy-opacity -composite PNG:-) \
    <(convert b.png mask.png -compose copy-opacity -composite PNG:-) \
    diff.png

I still don't see what you are trying to do, and I would still like to see your images and what result you expect... however, I have taken your advice and made two similar images from this page with the two s letters from the word missing blanked out.

a.png

b.png

So, if I now compare them, I get this:

convert a.png b.png -metric PSNR -compare -format "%[distortion]" info:
33.4539

and this image:

or if I do as I said, I get this:

convert -fill black \( a.png +opaque white \) \( b.png +opaque white \) -metric PSNR -compare -format "%[distortion]" info:
7.25418

and this image:

But I am still none the wiser about what you want because I have done all the work and you have not answered me.

Original Answer

It would be better if you could post your images so we can see what you mean, but try using convert like this to do the comparison:

convert a.png b.png -metric PSNR -compare diff.png

then you can add in pre-processing like this to make all white areas black before comparing. You can also add -fuzz 10% to catch near-white colours.

convert -fill black 
   \( a.png +opaque white \) \
   \( b.png +opaque white \) \
   -metric PSNR -compare diff.png

Elderberry

i'm guessing convert a.png b.png -metric PSNR -compare diff.png is absolutely equivalent to what im already doing – Pip

how will having black pixels instead of white, affect PSNR? – Pip

It's not very clear what you are actually trying to measure without seeing your images or understanding your application in its entirety - saying the PSNR between 2 images I cannot see is 27 does not help much! As you said white pixels are affecting your results, I assumed that changing them to black, or showing you how you can pre-process within a convert operation before comparing them would probably help - if not, please elaborate further what you are aiming for and I will try to assist further. – Elderberry

Il break it down: -ignore white pixels, don't compare white pixels -only ignore white pixels if they happen to match between the two images. i.e. if pixel bob is white in a.png and in b.png don't add to the mean squear error – Pip

you could use this page as a sample image, with some minor difference like a character missing – Pip

convert a.png b.png -metric PSNR -compare diff.png convert: unrecognized option -metric' @ error/convert.c/ConvertImageCommand/2030.` – Pip

convert -fill black \(a.png +opaque white\) \(b.png +opaque white\) -metric PSNR -compare diff.png convert: unable to open image (a.png': @ error/blob.c/OpenBlob/2587. convert: unable to open file (a.png' @ error/png.c/ReadPNGImage/3238. convert: unable to open image (b.png': @ error/blob.c/OpenBlob/2587. convert: unable to open file (b.png' @ error/png.c/ReadPNGImage/3238. convert: unrecognized option -metric' @ error/convert.c/ConvertImageCommand/2030.` – Pip

maybe we are using different versions? – Pip

yea so my version of convert is different to yours.. iv added my version info to the question – Pip

If you are only interested in the metric result, then you do not need to generate a diff.xxx image -- you can set it to null: for pure metric value results... – Cabrales