Exact depth distance from Realsense D435 with X,Y coordinates

Question

I would like to calculate the depth distance with the x,y coordinates of a detect object.

In the image below, I used background subtraction in opencv to detect and track a new object that has entered in the cameras view. I am able to get the x,y coordinate with relative ease but have trouble getting the z depth with the realsense sdk. Is it possible in any way for me to input the x,y coordinates into the realsense sdk and get the z depth from there?

I'm using opencv python and realsense sdk 2 for reference.

Getting the z depth in the midpoint of the bounding box

import numpy as np
import cv2 as cv
import pyrealsense2 as rs

# Create a pipeline
pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 1280, 720, rs.format.bgr8, 30)

#Start pipeline
profile = pipeline.start(config)

erodeKernel = cv.getStructuringElement(cv.MORPH_RECT, (5,5))

fgbg = cv.createBackgroundSubtractorMOG2()
while True:
    frames = pipeline.wait_for_frames()
    depth_frame = frames.get_depth_frame()
    colour_frame = frames.get_color_frame()

    color_image = np.asanyarray(colour_frame.get_data())
    depth_image = np.asanyarray(depth_frame.get_data())

    # Apply colormap on depth image (image must be converted to 8-bit per pixel first)
    depth_colormap = cv.applyColorMap(cv.convertScaleAbs(depth_image, alpha=0.03), cv.COLORMAP_JET)

    blur = cv.GaussianBlur(color_image,(5,5),0)
    fgmask = fgbg.apply(blur)

    im2, contours, hierarchy = cv.findContours(fgmask.copy(), cv.RETR_EXTERNAL, cv.CHAIN_APPROX_SIMPLE)
    for c in contours:
        if cv.contourArea(c) < 200:
            continue

        (x,y,w,h) = cv.boundingRect(c)
        cv.rectangle(color_image, (x,y), (x + w, y + h), (0,255,0), 2)

    cv.imshow('RealSense', color_image)
    cv.imshow("Depth", depth_colormap)
    cv.imshow('Mask', fgmask)
    if cv.waitKey(25) == ord('q'):
        break
cv.destroyAllWindows()
pipeline.stop()

Esmaria

have you tried this answer? there is also a good link to github – Narrate

@Narrate thanks for the reply. I've found a solution but I will check your suggestion out as well. Thank you – Esmaria

Answer 1

It appears this is a rather simple solution. After poking around the c++ examples, the realsense sdk provides a function known as get_distance(x,y) where it will return the depth distance according to the x,y coords.

Note that this function in python is exactly the same but must be called from the depth frame and the x and y must be cast to integer

pipeline = rs.pipeline()
config = rs.config()
profile = pipeline.start(config)
frames = pipeline.wait_for_frames()
    while True:
        depth_frame = frames.get_depth_frame()
        zDepth = depth_frame.get_distance(int(x),int(y))

Esmaria

Can we use this code to get the Z of the color frame pixel? – Amendatory

@Schütze, Its mainly from the depth frame. You would need to align the colour and depth frame if you are using the x,y coordinates from the colour frame in order to get accurate depth reading. – Esmaria

Is the distance in Meters? – Bybidder

@RyanTin is there a faster way to get a distance map (Z values) for the whole image rather than looping each pixel? – Knapsack