How to draw 3D Coordinate Axes with OpenCV for face pose estimation?

Question

Suppose I have got the face euler angles (pitch, yaw, roll). How can draw the 3D Coordinate Axes which show the face pose?

Here is an example from here:

Haplosis

why not just take the code from your link ? – Lipophilic

I do not want to use OpenGL to draw the result. – Haplosis

Create the 3D axes as vertices/vectors. apply the rotation to them. project them to your camera/image plane – Southey

Thanks, @Micka. Would you please give me more details about your answer? Any code example or opensource reference for that? – Haplosis

no. read about camera model, computer graphics, rendering pipeline, camera projection. If you want the easy way, use OpenGL. If not, learn and understand how it is done (won't be too difficult for projecting just simple lines). - opencv has some functions to help you building up the camera model and transforming your vertices, but you will have to understand what you want to do and how to achieve it. - en.wikipedia.org/wiki/3D_projection – Southey

Do what @Southey suggested. If you do not want to use OpenGL, then what do you want? Do you want to use OpenCV? – Atharvaveda

You are being picky about the solution, so you should explain what your parameters are. What tools are you open to using @HoneyTidy – Sen

Yes, I want to use OpenCV to draw the result. – Haplosis

It is a standard technique to draw projected lines in OpenCV for visualizations. No opengl is necessary. – Jidda

Answer 1

This can be done in pure OpenCV as long as you have your camera parameters. You should be able to create three vectors corresponding to axis x, y, z (basically points [0,0,0] [1, 0, 0], [0, 1, 0], [0, 0, 1] which you will later project into the image plane. You should first rotate those points according to your yaw/pitch/roll (e.g. by multiplying them by a rotation matrix).

In order to project 3D points to the image plane, use the projectPoints function. It takes 3D points, camera parameters and generates 2D image points. Once you have the image points, you can simply use the line function to draw lines between the projected central point ([0,0,0] in 3D) and each of the resulting projections of the axis points.

Jidda

Answer 2

A simple example:

def draw_axis(img, R, t, K):
    # unit is mm
    rotV, _ = cv2.Rodrigues(R)
    points = np.float32([[100, 0, 0], [0, 100, 0], [0, 0, 100], [0, 0, 0]]).reshape(-1, 3)
    axisPoints, _ = cv2.projectPoints(points, rotV, t, K, (0, 0, 0, 0))
    img = cv2.line(img, tuple(axisPoints[3].ravel()), tuple(axisPoints[0].ravel()), (255,0,0), 3)
    img = cv2.line(img, tuple(axisPoints[3].ravel()), tuple(axisPoints[1].ravel()), (0,255,0), 3)
    img = cv2.line(img, tuple(axisPoints[3].ravel()), tuple(axisPoints[2].ravel()), (0,0,255), 3)
    return img

Prefecture

May I ask what R, t and K are in your example? – Visitant

@Visitant cs.cmu.edu/~16385/s17/Slides/11.1_Camera_matrix.pdf on page 30. They are the decomposition of the camera matrix – Diann

R can be any 3d rotation matrix. – Diann

Answer 3

I used thie code. It's from Basel Face model(BFM), you can find matlab code from their web sites

def draw_axis(img, yaw, pitch, roll, tdx=None, tdy=None, size = 100):

    pitch = pitch * np.pi / 180
    yaw = -(yaw * np.pi / 180)
    roll = roll * np.pi / 180

    if tdx != None and tdy != None:
        tdx = tdx
        tdy = tdy
    else:
        height, width = img.shape[:2]
        tdx = width / 2
        tdy = height / 2

    # X-Axis pointing to right. drawn in red
    x1 = size * (math.cos(yaw) * math.cos(roll)) + tdx
    y1 = size * (math.cos(pitch) * math.sin(roll) + math.cos(roll) * math.sin(pitch) * math.sin(yaw)) + tdy

    # Y-Axis | drawn in green
    #        v
    x2 = size * (-math.cos(yaw) * math.sin(roll)) + tdx
    y2 = size * (math.cos(pitch) * math.cos(roll) - math.sin(pitch) * math.sin(yaw) * math.sin(roll)) + tdy

    # Z-Axis (out of the screen) drawn in blue
    x3 = size * (math.sin(yaw)) + tdx
    y3 = size * (-math.cos(yaw) * math.sin(pitch)) + tdy

    cv2.line(img, (int(tdx), int(tdy)), (int(x1),int(y1)),(0,0,255),3)
    cv2.line(img, (int(tdx), int(tdy)), (int(x2),int(y2)),(0,255,0),3)
    cv2.line(img, (int(tdx), int(tdy)), (int(x3),int(y3)),(255,0,0),3)

    return img

Klutz

Answer 4

Some clarification of the code given above

def draw_axis(img, rotation_vec, t, K, scale=0.1, dist=None):
    """
    Draw a 6dof axis (XYZ -> RGB) in the given rotation and translation
    :param img - rgb numpy array
    :rotation_vec - euler rotations, numpy array of length 3,
                    use cv2.Rodrigues(R)[0] to convert from rotation matrix
    :t - 3d translation vector, in meters (dtype must be float)
    :K - intrinsic calibration matrix , 3x3
    :scale - factor to control the axis lengths
    :dist - optional distortion coefficients, numpy array of length 4. If None distortion is ignored.
    """
    img = img.astype(np.float32)
    dist = np.zeros(4, dtype=float) if dist is None else dist
    points = scale * np.float32([[1, 0, 0], [0, 1, 0], [0, 0, 1], [0, 0, 0]]).reshape(-1, 3)
    axis_points, _ = cv2.projectPoints(points, rotation_vec, t, K, dist)
    img = cv2.line(img, tuple(axis_points[3].ravel()), tuple(axis_points[0].ravel()), (255, 0, 0), 3)
    img = cv2.line(img, tuple(axis_points[3].ravel()), tuple(axis_points[1].ravel()), (0, 255, 0), 3)
    img = cv2.line(img, tuple(axis_points[3].ravel()), tuple(axis_points[2].ravel()), (0, 0, 255), 3)
    return img

Wellspoken

Answer 5

It is also possible to use the opencv function drawFrameAxes for this.

scale = 0.1 
img = cv2.drawFrameAxes(img, K, distortion, rotation_vec, translation_vec, scale)

Thoroughfare