Does Kinect Infrared View Have an offset with the Kinect Depth View

Question

I am working on a Kinect project using the infrared view and depth view. In the infrared view, using CVBlob library, I am able to extract some 2D points of interest. I want to find the depth of these 2D points. So I thought that I can use the depth view directly, something like this:

coordinates3D[0] = coordinates2D[0];
coordinates3D[1] = coordinates2D[1];
coordinates3D[2] = (USHORT*)(LockedRect.pBits)
[(int)coordinates2D[1] * Width + (int)coordinates2D[0]] >> 3;

I don't think this is the right formula to get the depth. I am able to visualize the 2D points of interest in the depth view. If I get a point (x, y) in infrared view, then I draw it as a red point in the depth view at (x, y)
I noticed that the red points are not where I expect them to be (on an object). There is a systematic error in their locations.

I was of the opinion that the depth view and infrared views have one-to-one correspondence unlike the correspondence between the color view and depth view.
Is this indeed true or is there an offset between the IR and depth views? If there is an offset, can I somehow get the right depth value?

Pizzeria

Answer 1

Depth and Color streams are not taken from the same point so they do not correspond to each other perfectly. Also they FOV (field of view) is different.

1. cameras

   • IR/Depth FOV 58.5° x 45.6°
   • Color FOV 62.0° x 48.6°
   • distance between cameras 25mm

2. my corrections for 640x480 resolution for both streams

   if (valid depth)
    {
    ax=(((x+10-xs2)*241)>>8)+xs2;
    ay=(((y+30-ys2)*240)>>8)+ys2;
    }
   

   • x,y are in coordinates in depth image
   • ax,ay are out coordinates in color image
   • xs,ys = 640,480
   • xs2,ys2 = 320,240

   as you can see my kinect has also y-offset which is weird (even bigger then x-offset). My conversion works well on ranges up to 2 m I did not measure it further but it should work even then

3. do not forget to correct space coordinates from depth and depth image coordinates

   pz=0.8+(float(rawdepth-6576)*0.00012115165336374002280501710376283);
   px=-sin(58.5*deg*float(x-xs2)/float(xs))*pz;
   py=+sin(45.6*deg*float(y-ys2)/float(ys))*pz;
   pz=-pz;
   

   • where px,py,pz is point coordinate in [m] in space relative to kinect

   I use coordinate system for camera with opposite Z direction therefore the negation of sign

PS. I have old model 1414 so newer models have probably different calibration parameters

Excelsior

Answer 2

There is no offset between the "IR View" and "Depth View". Primarily because they are the same thing.

The Kinect has 2 cameras. A RGB color camera and a depth camera, which uses an IR blaster to generate a field light field that is used when processing the data. These give you a color video stream and a depth data stream; there is no "IR view" separate from the depth data.

UPDATE:

They are actually the same thing. What you are referring to as a "depth view" is simply a colorized version of of the "IR view"; the black-and-white image is the "raw" data, while the color image is a processed version of the same data.

In the Kinect for Windows Toolkit, have a look in the KinectWpfViewers project (if you installed the KinectExplorer-WPF example, it should be there). In there is the KinectDepthViewer and the DepthColorizer classes. They will demonstrate how the colorized "depth view" is created.

UPDATE 2:

Per comments below what I've said above is almost entirely junk. I'll likely go edit it out or just delete my answer in full in the near future, until then it shall stand as a testament to my once invalid beliefs on what was coming from where.

Anyways... Have a look at the CoordinateMapper class as another possible solution. The link will take you to the managed code docs (which is what I'm familiar with), I'm looking around the C++ docs to see if I can find the equivalent.

I've used this to map the standard color and depth views. It may also map the IR view just as well (I wouldn't see why not), but I'm not 100% sure of that.

Reliquiae

Thanks for reply. Actually there is a distinct IR view separate from Depth view. I will post a screenshot of the standard Kinect Explorer application showing both views. By looking at the images side-by-side, they appear to have one-to-one correspondence with no offsets; I want to confirm this to be a fact and not just an observation. – Pizzeria

The depth view can be used to infer the depth at a location (x, y) whereas the IR view cannot be used for doing the same directly. The IR image is processed by comparing it with a standard speckle pattern to infer the depth and then it is displayed such that the pixel intensities in the depth view depict the physical depth. So calling depth view as a colorized version of IR view is not accurate. Also, the IR view is the real image captured by Kinect's IR receiver and can be used for computer vision algorithms, whereas depth view is primarily meant for inferring depth only. – Pizzeria

Interestingly, I found an offset of 23 makes the two views aligned. This is purely through experimentation, but it is working for different distances from the Kinect along X, Y and Z. See below in comments for my new code: – Pizzeria

Indeed, after digging more and reviewing the ColorImageFormat class the IR data is read by the color camera. I was under the impression the depth camera read and processed the stream. The image I've included is just plain misleading... and the information I spewed forth if pure junk! Check answer updates for a few tips that might help... – Reliquiae

The depth camera (can also call IR camera) does indeed read the IR data. When I covered the rightmost camera, which is the IR receiver, with a finger, it closed both the depth and IR views. Covering the color camera had no effect on Depth and IR views. This clearly shows that the IR camera does all the depth processing. But I understand what you are saying. There are distinct Color and Depth streams, whereas the Infrared stream is not given a separate identity and can be obtained by tweaking the Color stream to output the IR view, which is a counter-intuitive API design decision, in my opinion – Pizzeria

Answer 3

I created a blog showing the IR and Depth views:

http://aparajithsairamkinect.blogspot.com/2013/06/kinect-infrared-and-depth-views_6.html

Pizzeria

Inserted the image into the post for you. See my other comment in the answer below for more... – Reliquiae

Answer 4

This code works for many positions of the trackers from the Kinect:

coordinates3D[0] = coordinates2D[0];
coordinates3D[1] = coordinates2D[1];
coordinates3D[2] = (USHORT*)(LockedRect.pBits)
[(int)(coordinates2D[1] + 23) * Width + (int)coordinates2D[0]] >> 3;

Pizzeria