How to implement camera pan like in 3dsMax?

Question

What are the necessary maths to achieve the camera panning effect that's used in 3ds max?

In 3ds max the distance between the cursor and the mesh will always remain the same throughout the entire movement (mouse_down+mouse_motion+mouse_up).

My naive and failed attempt has been trying to move the camera on the plane XY by using dt (frame time) multiplied by some hardcoded constant and the result is really ugly and uintuitive.

The code I've got so far is:

def glut_mouse(self, button, state, x, y):
    self.last_mouse_pos = vec2(x, y)
    self.mouse_down_pos = vec2(x, y)

def glut_motion(self, x, y):
    pos = vec2(x, y)
    move = self.last_mouse_pos - pos
    self.last_mouse_pos = pos
    self.pan(move)

def pan(self, delta):
    forward = vec3.normalize(self.target - self.eye)
    right = vec3.normalize(vec3.cross(forward, self.up))
    up = vec3.normalize(vec3.cross(forward, right))

    if delta.x:
        right = right*delta.x
    if delta.y:
        up = up*delta.y

    self.eye+=(right+up)
    self.target+=(right+up)

Could you explain how the maths of camera panning in 3dsmax work?

EDIT:

My question has already been answered initially by @Rabbid76 but there's still one case where his algorithm won't work properly. It doesn't handle properly the case where you panning is started from empty space (said otherwise, when depth buffer value takes the far value=1.0). In 3dsmax camera panning is handled correctly in all situations, no matter which value of the depth buffer.

Poverty

Provide the full code, including the class of self and its fields + behaviours e.g. how the eye and target fields are used to calculate the camera matrix. – Pennoncel

Answer 1

[...] but there's still one case where his algorithm won't work properly. It doesn't handle properly the case where you panning is started from empty space [...]

In the solution the depth of the object is taken from the depth buffer, at that position, where the mouse click occurs. If this is the "empty space", a position where no object was drawn, the depth is the maximum of the depth range (in common 1). This leads to a rapid paining.

A solution or workaround would be use the depth of an representative position of the scene. e.g. the origin of the world:

pt_drag = glm.vec3(0, 0, 0)

Of course this may not lead to a proper result in each case. If the objects of the scene are not around the origin of the world, this approach will fail. I recommend to calculate the center of the axis aligned bounding box of the scene. Use this point for the representative "depth":

box_min = ... # glm.vec3
box_max = ... # glm.vec3

pt_drag = (box_min + box_max) / 2

The depth of a point can computed by the transformation with the view and projection matrix and a final perspective divide:

o_clip = self.proj * self.view * glm.vec4(pt_drag, 1)
o_ndc  = glm.vec3(o_clip) / o_clip.w

This can be applied to the function glut_mouse:

def glut_mouse(self, button, state, x, y):
    self.drag = state == GLUT_DOWN
    self.last_mouse_pos = glm.vec2(x, self.height-y)
    self.mouse_down_pos = glm.vec2(x, self.height-y)

    if self.drag:
        depth_buffer = glReadPixels(x, self.height-y, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT)
        self.last_depth = depth_buffer[0][0]
        if self.last_depth == 1:
            pt_drag = glm.vec3(0, 0, 0)
            o_clip  = self.proj * self.view * glm.vec4(pt_drag, 1)
            o_ndc   = glm.vec3(o_clip) / o_clip.w
            if o_ndc.z > -1 and o_ndc.z < 1:
                self.last_depth = o_ndc.z * 0.5 + 0.5

Preview:

The key to a well feeling solution is to find the "correct" depth. At perspective projection the dragging, where the mouse movement effects the object in a 1:1 motion, projected on the viewport, only works correctly for a well defined depth. Objects with different depths are displaced by a different scale when they projected on the viewport, that's the "nature" of perspective.

To find the "correct" depth, there are different possibilities, which depend on your needs:

• Reading the depth from the depth buffer at the current mouse position:

depth_buffer = glReadPixels(x, self.height-y, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT)    
self.last_depth = depth_buffer[0][0]

• Get the minimum and maximum depth of the depth buffer (except the value for the far plane, 1.0) and calculate the mean depth. Of course the entire depth buffer has to be investigated in this case:

d_buf = glReadPixels(0, 0, self.width, self.height, GL_DEPTH_COMPONENT, GL_FLOAT)
d_vals = [float(d_buf[i][j]) for i in range(self.width) for j in range(self.height) if d_buf[i][j] != 1]
if len(d_vals) > 0:
    self.last_depth = (min(d_vals) + max(d_vals)) / 2 

• Use the origin of the world:

pt_drag = glm.vec3(0, 0, 0)
o_clip  = self.proj * self.view * glm.vec4(pt_drag, 1)
o_ndc   = glm.vec3(o_clip) / o_clip.w
if o_ndc.z > -1 and o_ndc.z < 1:
    self.last_depth = o_ndc.z * 0.5 + 0.5 

• Calculating the center of the bounding box of the scene.

• Implement a raycasting, which identifies an object by a ray, which starts at the point of view a runs trough the cursor (mouse) position. This algorithm can be advanced by identifying the object which is "closest" to the ray, when no object is hit.

See also Python OpenGL 4.6, GLM navigation

Selfimmolation

Answer 2

Your solution would work at orthographic projection, but it fails at perspective projection. Note, at Perspective Projection the projection matrix describes the mapping from 3D points in the world as they are seen from of a pinhole camera, to 2D points of the viewport.

The amount of displacement for the eye and target position depends on the depth of the object which is dragged on the viewport.

If the object is close to the eye position, then a translation on the viewport leads to a small displacement of the eye and target positions:

If the distance from the object to the eye is far, then a translation on the viewport leads to a large displacement of the eye and target positions:

To do what you want you have to know the size of the viewport, the view matrix and the projection matrix:

self.width   # width of the viewport
self.height  # height of the viewport
self.view    # view matrix
self.proj    # prjection matrix

Change the pane method, so that it receives the new and old mouse position. Note y axis has to be flipped (self.height-y). Get the depth of the hit point (object) by glReadPixels using the format type GL_DEPTH_COMPONENT:

def glut_mouse(self, button, state, x, y):
    self.drag = state == GLUT_DOWN
    self.last_mouse_pos = glm.vec2(x, self.height-y)
    self.mouse_down_pos = glm.vec2(x, self.height-y)
    if self.drag:
        depth_buffer = glReadPixels(x, self.height-y, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT)
        self.last_depth = depth_buffer[0][0]
        print(self.last_depth)

def glut_motion(self, x, y):
    if not self.drag:
       return
    old_pos = self.last_mouse_pos
    new_pos = glm.vec2(x, self.__vp_size[1]-y)
    self.last_mouse_pos = new_pos 
    self.pan(self.last_depth, old_pos, new_pos)

def pan(self, depth, old_pos, new_pos):
    # .....

The mouse position gives a position in window space, where the z coordinate is the depth of the hit point respectively object:

wnd_from    = glm.vec3(old_pos[0], old_pos[1], float(depth))
wnd_to      = glm.vec3(new_pos[0], new_pos[1], float(depth))

This positions can be transformed to world space by glm.unProject:

vp_rect     = glm.vec4(0, 0, self.width, self.height)
world_from  = glm.unProject(wnd_from, self.view, self.proj, vp_rect)
world_to    = glm.unProject(wnd_to, self.view, self.proj, vp_rect)

The world space displacement of the eye and target position is the distance from the old to the new world position:

world_vec   = world_to - world_from

Finally calculate the new eye and target position and update the view matrix:

self.eye    = self.eye - world_vec
self.target = self.target - world_vec
self.view   = glm.lookAt(self.eye, self.target, self.up)

See also Python OpenGL 4.6, GLM navigation

I tested the code with the following example:

Preview:

Full python code:

import os
import math
import numpy as np
import glm
from OpenGL.GLUT import *
from OpenGL.GL import *
from OpenGL.GL.shaders import *
from OpenGL.arrays import *
from ctypes import c_void_p

class MyWindow:

    __caption = 'OpenGL Window'
    __vp_size = [800, 600]
    __vp_valid = False
    __glut_wnd = None

    __glsl_vert = """
        #version 450 core

        layout (location = 0) in vec3 a_pos;
        layout (location = 1) in vec3 a_nv;
        layout (location = 2) in vec4 a_col;

        out vec3 v_pos;
        out vec3 v_nv;
        out vec4 v_color;

        uniform mat4 u_proj;
        uniform mat4 u_view;
        uniform mat4 u_model;

        void main()
        {
            mat4 model_view = u_view * u_model;
            mat3 normal     = transpose(inverse(mat3(model_view)));

            vec4 view_pos   = model_view * vec4(a_pos.xyz, 1.0);

            v_pos       = view_pos.xyz;
            v_nv        = normal * a_nv;  
            v_color     = a_col;
            gl_Position = u_proj * view_pos;
        }
    """

    __glsl_frag = """
        #version 450 core

        out vec4 frag_color;
        in  vec3 v_pos;
        in  vec3 v_nv;
        in  vec4 v_color;

        void main()
        {
            vec3  N    = normalize(v_nv);
            vec3  V    = -normalize(v_pos);
            float ka   = 0.1;
            float kd   = max(0.0, dot(N, V)) * 0.9;
            frag_color = vec4(v_color.rgb * (ka + kd), v_color.a);
        }
    """

    __program = None
    __vao = None
    __vbo = None
    __no_vert = 0

    def __init__(self, w, h):

        self.__vp_size = [w, h]

        glutInit()
        glutInitDisplayMode(GLUT_DOUBLE | GLUT_RGB | GLUT_DEPTH)
        glutInitWindowSize(self.__vp_size[0], self.__vp_size[1])
        __glut_wnd = glutCreateWindow(self.__caption)

        self.__program = compileProgram( 
            compileShader( self.__glsl_vert, GL_VERTEX_SHADER ),
            compileShader( self.__glsl_frag, GL_FRAGMENT_SHADER ),
        )

        self.___attrib = { a : glGetAttribLocation (self.__program, a) for a in ['a_pos', 'a_nv', 'a_col'] }
        print(self.___attrib)

        self.___uniform = { u : glGetUniformLocation (self.__program, u) for u in ['u_model', 'u_view', 'u_proj'] }
        print(self.___uniform)

        v = [ -1,-1,1,  1,-1,1,  1,1,1, -1,1,1, -1,-1,-1,  1,-1,-1,  1,1,-1, -1,1,-1 ]
        c = [ 1.0, 0.0, 0.0,   1.0, 0.5, 0.0,    1.0, 0.0, 1.0,   1.0, 1.0, 0.0,   0.0, 1.0, 0.0,   0.0, 0.0, 1.0 ]
        n = [ 0,0,1, 1,0,0, 0,0,-1, -1,0,0, 0,1,0, 0,-1,0 ]
        e = [ 0,1,2,3, 1,5,6,2, 5,4,7,6, 4,0,3,7, 3,2,6,7, 1,0,4,5 ]
        attr_array = []
        for si in range(6):
            for vi in range(6):
                ci = [0, 1, 2, 0, 2, 3][vi]
                i = si*4+ci
                attr_array.extend( [ v[e[i]*3], v[e[i]*3+1], v[e[i]*3+2] ] )
                attr_array.extend( [ n[si*3], n[si*3+1], n[si*3+2] ] )
                attr_array.extend( [ c[si*3], c[si*3+1], c[si*3+2], 1 ] ); 
        self.__no_vert = len(attr_array) // 10

        vertex_attributes = np.array(attr_array, dtype=np.float32)

        self.__vbo = glGenBuffers(1)
        glBindBuffer(GL_ARRAY_BUFFER, self.__vbo)
        glBufferData(GL_ARRAY_BUFFER, vertex_attributes, GL_STATIC_DRAW)

        self.__vao = glGenVertexArrays(1)
        glBindVertexArray(self.__vao)
        glVertexAttribPointer(0, 3, GL_FLOAT, False, 10*vertex_attributes.itemsize, None)
        glEnableVertexAttribArray(0)
        glVertexAttribPointer(1, 3, GL_FLOAT, False, 10*vertex_attributes.itemsize, c_void_p(3*vertex_attributes.itemsize))
        glEnableVertexAttribArray(1)
        glVertexAttribPointer(2, 4, GL_FLOAT, False, 10*vertex_attributes.itemsize, c_void_p(6*vertex_attributes.itemsize))
        glEnableVertexAttribArray(2)

        glEnable(GL_DEPTH_TEST)
        glUseProgram(self.__program)

        glutReshapeFunc(self.__reshape)
        glutDisplayFunc(self.__mainloop)
        glutMouseFunc(self.glut_mouse)
        glutMotionFunc(self.glut_motion)

        self.drag = False

        self.eye    = glm.vec3(-3, -7, 6)
        self.target = glm.vec3(0, 0, 0)
        self.up     = glm.vec3(0, 0, 1)

        self.near  = 0.1
        self.far   = 100.0
        aspect     = self.__vp_size[0]/self.__vp_size[1]
        self.proj  = glm.perspective(glm.radians(90.0), aspect, self.near, self.far)
        self.view  = glm.lookAt(self.eye, self.target, self.up)
        self.model = glm.mat4(1)  

    def run(self):
        self.__starttime = 0
        self.__starttime = self.elapsed_ms()
        glutMainLoop()

    def elapsed_ms(self):
      return glutGet(GLUT_ELAPSED_TIME) - self.__starttime

    def __reshape(self, w, h):
        self.__vp_valid = False

    def __mainloop(self):

        if not self.__vp_valid:
            self.width      = glutGet(GLUT_WINDOW_WIDTH)
            self.height     = glutGet(GLUT_WINDOW_HEIGHT)
            self.__vp_size  = [self.width, self.height]
            self.__vp_valid = True
            aspect          = self.width / self.height
            self.proj       = glm.perspective(glm.radians(90.0), aspect, self.near, self.far)

        glUniformMatrix4fv(self.___uniform['u_proj'], 1, GL_FALSE, glm.value_ptr(self.proj) )
        glUniformMatrix4fv(self.___uniform['u_view'], 1, GL_FALSE, glm.value_ptr(self.view) )
        glUniformMatrix4fv(self.___uniform['u_model'], 1, GL_FALSE, glm.value_ptr(self.model) )

        glClearColor(0.2, 0.3, 0.3, 1.0)
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)

        glDrawArrays(GL_TRIANGLES, 0, self.__no_vert)

        glutSwapBuffers()
        glutPostRedisplay()

    def glut_mouse(self, button, state, x, y):
        self.drag = state == GLUT_DOWN
        self.last_mouse_pos = glm.vec2(x, self.height-y)
        self.mouse_down_pos = glm.vec2(x, self.height-y)
        if self.drag:
            depth_buffer = glReadPixels(x, self.height-y, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT)
            self.last_depth = depth_buffer[0][0]
            print(self.last_depth)

    def glut_motion(self, x, y):
        if not self.drag:
          return
        old_pos = self.last_mouse_pos
        new_pos = glm.vec2(x, self.__vp_size[1]-y)
        self.last_mouse_pos = new_pos 
        self.pan(self.last_depth, old_pos, new_pos)

    def pan(self, depth, old_pos, new_pos):

        wnd_from    = glm.vec3(old_pos[0], old_pos[1], float(depth))
        wnd_to      = glm.vec3(new_pos[0], new_pos[1], float(depth))

        vp_rect     = glm.vec4(0, 0, self.width, self.height)
        world_from  = glm.unProject(wnd_from, self.view, self.proj, vp_rect)
        world_to    = glm.unProject(wnd_to, self.view, self.proj, vp_rect)

        world_vec   = world_to - world_from

        self.eye    = self.eye - world_vec
        self.target = self.target - world_vec
        self.view   = glm.lookAt(self.eye, self.target, self.up)

window = MyWindow(800, 600)
window.run()

Selfimmolation

Thank you very much! High quality answers like this one are inspiring... this type of content is one of the main reasons why I still have faith about asking in SO... ;) . And just for the record, your answer would deserve a lot of upvotes. Again... thank very much, the maths have become cristal clear now. – Poverty

Today finally I've had the time to implement this method on my engine and it works really well... i've catched a little issue though :) . Try panning when the depth buffer is exactly far (ie: 1.0 by default), you'll see the movements become really abrupt. Now, check it out in max and no matter where you start dragging the movements will be smooth... Still thinking about a possible workaround as we speaking :) – Poverty

@Poverty Possibly wnd_z = 0 if depth == 1 else float(depth), wnd_from = glm.vec3(old_pos[0], old_pos[1], wnd_z), wnd_to = glm.vec3(new_pos[0], new_pos[1], wnd_z). Note 1 is the far plane and 0 is the near plane. – Selfimmolation

no luck :) , with that tweak you won't get panning at all (or very very little).One little remark, near=0 and far=1 are the default values, you can change them by using glDepthRangeF(near,far) and query the values with glGetFloatV(GL_DEPTH_RANGE), they will be clamped to [0,1]. In fact, i've seen a lot of unproject routines that assume near=0 & far=1 and they will break otherwise ie: ndc.z = (2*wz-(f+n))/(f-n) in the general case... ndc.z = (2*wz-1) would be for the particular case – Poverty