Choose luminosity (exposure) from HDR image

Question

I'm currently stuck on a video projet from pictures.

Problem :

I'm extracting pictures from UE4, due to a bug, not all lights are taken into account during the rendering of the screenshot. Output are HDR images. I want to get better brighteness because the exported picture are very dark, like the first exposure.

Using the "exposure bias" parameter in UE4 i'm able to real good luminosity of my scene, but cannot apply this parameter to the screenshot rendering :

Tries :

Using Tonemapper algorithm (speciafically cv::TonemapDrago) i'm able to get better image result :

The main problem, for my case, of the Tonemap Algorithm is because the global luminosity is changed depending of luminosity of areas : In the second image, the window add lots of light, so the algorithm low all the luminosity to adjust the mean. In the video rendered, the light change is very brutal.

I've tried to change brightness and saturation without success. I've modified the code of the TonemapDrago trying to use constants for some steps of the algorithm.

Question :

I would like to "choose the exposure time" from an HDR image. Tonemap is based on several exposure time of the same image, not interesting in my case.

Any other idea is welcome.

EDIT:

CV::Mat depth is 5, type is CV_32FC3

cout << mat.step give me 19200

Here are 2 samples i use to try solving my problem :

First Image

Image with light window

Edit 2 :

Cannot open .HDR picture with gimp, event with the "explosure blend" plugin. I'm able to get great enough result using Photoshop. Any idea of the algorithm behind that ? Any of the 6 Tonemap algos by OpenCV allow to choose an exposure correction.

EDIT 3:

I've followed the algorithm explain in this tuto for openGL, which is giving this C+ code to me :

cv::Mat exposureTonemap (cv::Mat m, float gamma = 2.2, float exposure = 1)
{
  // Exposure tone mapping
  cv::Mat exp;
  cv::exp( (-m) * exposure, exp );

  cv::Mat mapped = cv::Vec3f(1.0) - exp;
  // Gamma correction 
  cv::pow(exp, 1.0f / gamma, exp);

  cv::imshow("exposure tonemap", exp );
  cv::waitKey();

  return exp;
}

Applying this algo on my .HDR picture i got very bright result even with a correction of 1 and 1 for gamma and exposure :

Reading the code, there is something wrong because 1 and 1 as argument should not modify the picture. Fixed that, the answer is posted. thanks a lot to @user3896254 (Ge saw it too in comment)

Pyroelectricity

What format is the output image in? If it doesn't have enough bit depth you may not be able to recover the lost data. – Lithic

@MarkRansom The format of .hdr picture pixel open with opencv is CV_32FC3 – Pyroelectricity

If you upload an example of this image somewhere - it would be easier for us to help you. You can dump the raw image buffer to binary file and say image resolution to allow others use it. – Chadwell

@user3896254 : Done, i've added link, i guess easier than writing down binary stuff. I'm reading and trying ur answer – Pyroelectricity

regarding code for tonemapping that you have added - you should do v::pow(mapped, 1.0f / gamma, exp); instead of cv::pow(exp, 1.0f / gamma, exp); – Chadwell

@user3896254 Yes i've just was that :( I't seems to work i'm gonna post the fix as an answer i guess. Thanks a lot for all your help – Pyroelectricity

Since you can't place bounty on your own answer maybe I'll edit my answer with this fix so you can accept it? Since otherwise 100 reputation will be lost. – Chadwell

If you want to. Don't really care about the reputation, my goal was to find an solution anyway :D – Pyroelectricity

Answer 1

Consider using Retinex. It uses single image for input and is included in GIMP, so is easy to toy around, besides you can get its source code (or roll your own, which is pretty simple anyway). Since you have renders instead of photos - there's no noise, and you theoretically are able to adjust the colours to your needs.

But as @mark-ransom has already said, you may have trouble recovering information from your rendered output. you said you have HDR images as render output, but I am not sure what do you mean. Is it a single RGB image? What is the colour depth of each channel? I have tried to apply retinex to your sample, but obviously it doesn't look good, because of compression, and limited range that was applied before saving. If your output has high range and is not compressed - you'll get better results.

EDIT: I have tried retinex on your input and it turned out not very good - the bright parts of image (lamps/windows) introduced ugly dark halos around them.

In this case simple tonemapping&gamma correction looks a lot better. Your code was almost fine, you just had a little typo: instead of cv::pow(exp, 1.0f / gamma, exp); you should have had v::pow(mapped, 1.0f / gamma, exp);

I have messed around with your code, and noticed that this tonemapping seems to degrade color saturation. To overcome this I perform it only on V channel of HSV image. Compare results yourself (left - full space tonemapping, right - V only): Note floor color, sky in window and yellowish light color that got preserved with this approach.

Here is full code for the sake of completeness:

#include <opencv2/opencv.hpp>

using namespace cv;

Mat1f exposureTonemap (Mat1f m, float gamma = 2.2, float exposure = 1) {
  // Exposure tone mapping
  Mat1f exp;
  cv::exp( (-m) * exposure, exp );
  Mat1f mapped = 1.0f - exp;

  // Gamma correction
  cv::pow(mapped, 1.0f / gamma, mapped);

  return mapped;
}

Mat3f hsvExposureTonemap(Mat &a) {
  Mat3f hsvComb;
  cvtColor(a, hsvComb, COLOR_RGB2HSV);

  Mat1f hsv[3];
  split(hsvComb, hsv);

  hsv[2] = exposureTonemap(hsv[2], 2.2, 10);

  merge(hsv, 3, hsvComb);

  Mat rgb;
  cvtColor(hsvComb, rgb, COLOR_HSV2RGB);

  return rgb;
}

int main() {
  Mat a = imread("first.HDR", -1);
  Mat b = imread("withwindow.HDR", -1);

  imshow("a", hsvExposureTonemap(a));
  imshow("b", hsvExposureTonemap(b));
  waitKey();

  return 0;
}

Chadwell

Wooo nice, the tonemap on the 'value' from the HSV picture gives way better results. Thanks for this fix ! – Pyroelectricity

Answer 2

What kind of scene lighting are you currently using? It looks like you are using point lights where the lightbulbs would be, but they aren't bright enough. In your unrendered scene, the scene is going to be full brightness. In your rendered scene, you'll get darkness.

I would maybe recommend at least a minimal sky light so that you always have some light across your scene (unless you have areas of actual darkness)

Traps

The problem of this bug is that some of the ambiance lights and pointed lights arn't taken in the 360 rendering of a view point in UE. Adding a brighter skybox will increase the light from the window and the tonemapper algo will be even more brutal with the camera movement. Even with that, the extracted picture will be too dark to be used 'as is' – Pyroelectricity

Answer 3

cv::Mat exposureTonemap (cv::Mat m, float gamma = 2.2, float exposure = 1)
{
  // Exposure tone mapping
  cv::Mat exp;
  cv::exp( (-m) * exposure, exp );
  cv::Mat mapped = cv::Scalar(1.0f, 1.0f, 1.0f) - exp;

  // Gamma correction 
  cv::pow(mapped, 1.0f / gamma, mapped);

  /*
  cv::imshow("exposure tonemap", mapped );
  cv::waitKey();
  */

  return mapped;
}

This algorithm is a Tonemapper trying to simulate exposure bias in an HDR If you want to use it in openCv 3.0 don't forget to open with -1 as last argument of imread cv::Mat img = cv::imread("mypicture.HDR", -1);

Pyroelectricity