Kinect: Converting from RGB Coordinates to Depth Coordinates

Question

I am using the Windows Kinect SDK to obtain depth and RGB images from the sensor.

Since the depth image and the RGB images do not align, I would like to find a way of converting the coordinates of the RGB image to that of the depth image, since I want to use an image mask on the depth image I have obtained from some processing on the RGB image.

There is already a method for converting depth coordinates to the color space coordinates:

NuiImageGetColorPixelCoordinatesFromDepthPixel

unfortunately, the reverse does not exist. There is only an arcane call in INUICoordinateMapper:

HRESULT MapColorFrameToDepthFrame(
     NUI_IMAGE_RESOLUTION eColorResolution,
     NUI_IMAGE_RESOLUTION eDepthResolution,
     DWORD cDepthPixels,
     NUI_DEPTH_IMAGE_PIXEL *pDepthPixels,
     DWORD cDepthPoints,
     NUI_DEPTH_IMAGE_POINT *pDepthPoints
)

How this method works is not very well documented. Has anyone used it before?

I'm on the verge of performing a manual calibration myself to calculate a transformation matrix, so I would be very happy for a solution.

Cline

I can't help you, but could you elaborate on how the images don't align? I thought they were the same. Is there a fixed offset or are they different field of view? – Groomsman

I am in the almost same situation now. And Found this page. social.msdn.microsoft.com/Forums/en-US/kinectsdknuiapi/thread/… I gonna try this out. but if you find the way, please let me know. – Wes

@Marian there's a certain amount of parallax error... if you look at the device, you may note that the two receivers are slightly separated. The conversion between the two coordinate systems isn't totally straightfoward. – Shawnna

Answer 1

Thanks to commenter horristic, I got a link to msdn with some useful information (thanks also to T. Chen over at msdn for helping out with the API). Extracted from T. Chen's post, here's the code that will perform the mapping from RGB to depth coordinate space:

INuiCoordinateMapper* pMapper;

mNuiSensor->NuiGetCoordinateMapper(&pMapper);

pMapper->MapColorFrameToDepthFrame(
        NUI_IMAGE_TYPE_COLOR,
        NUI_IMAGE_RESOLUTION_640x480,
        NUI_IMAGE_RESOLUTION_640x480,
        640 * 480, 
        (NUI_DEPTH_IMAGE_PIXEL*)LockedRect.pBits,
        640 * 480, 
        depthPoints);

Note: the sensor needs to be initialized and a depth frame locked for this to work.

The transformed coordinates can, e.g., be queried as follows:

/// transform RGB coordinate point to a depth coordinate point 
cv::Point TransformRGBtoDepthCoords(cv::Point rgb_coords, NUI_DEPTH_IMAGE_POINT *    depthPoints)
{
    long index = rgb_coords.y * 640 + rgb_coords.x;
    NUI_DEPTH_IMAGE_POINT depthPointAtIndex = depthPoints[index];
    return cv::Point(depthPointAtIndex.x, depthPointAtIndex.y); 
}

Cline

Answer 2

As far as I can tell, MapColorFrameToDepthFrame effectively runs the co-ordinate system conversion on every pixel of your RGB image, storing the depth image coordinates resulting from the conversion and the resultant depth value in the output NUI_DEPTH_IMAGE_POINT array. The definition of that structure is here: http://msdn.microsoft.com/en-us/library/nuiimagecamera.nui_depth_image_point.aspx

Possibly this is overkill for your needs however, and I've no idea how fast that method is. XBOX Kinect developers have a very fast implementation of that function that runs on the GPU at frame rate, Windows developers might not be quite so lucky!

Shawnna

The performance doesn't seem to suffer at all on Windows 7. However, I don't know if the GPU is being used or not. – Cline

@Cline that's good to know. I'm pretty certain the early SDK versions had no GPU accelleration at all, but I guess it is quite possible the more recent ones do. – Shawnna