Translate mouse coordinates to a 3D plan

Question

I'm building some kind of drag and drop app in javascript/jquery (DOM based, not canvas).

The idea is to be able to drag divs on a 3D scene (a div rotated in 3D).

It works on a 2D plan, the problem is when I rotate the scene in 3D, the objects position doesn't reflect the actual mouse position, but the coordinates translated in 3D

Illustrated exemple :

EXEMPLE ON JSFIDDLE

I want the objects moving relative to the absolute position of the mouse.

I calculate the mouse position like this :

document.addEventListener(gestureMove, function (event) {
  if (mouseDown == true) {
  event.preventDefault();
  moveX = (event.pageX - $('#scene').offset().left);
  moveY = (event.pageY - $('#scene').offset().top);
}

#scene { 
  width: 1000px;
  height: 1000px;
  -webkit-transform-style: preserve-3d;
  -webkit-transform: rotateX( 35deg ); 
}

An early solution was to calculate the difference between the mouse position and the object based on the initial position, and adding it to the object position during the drag. It was working, but the animation was really choppy and not smooth at all.

I'm sure there is a more simple way to get the mouse coordinates relative to the 3D plan, but wasn't able to find a real solution at this point.

Most of the search results on this subject point to gaming oriented languages, or canvas/webgl questions.

Any ideas ?

Thanks

Mclin

I think Google Sketch-up did a good job of this. – Inexplicable

You'll probably end up doing the reverse of perspective projection: en.wikipedia.org/wiki/3D_projection#Perspective_projection – Callis

Can you distill your code down enough to fit in a jsfiddle? – Benitobenjamen

Don't you have to account for the Z axis? as well as the X and Y? – Ssm

@Benitobenjamen I updated the question (jsfiddle.net/JulianG/Aubcr) – Mclin

#55655150 – Buffybuford

Answer 1

Assuming that your mouse position is an absolute screen position, and you want to grab and slide an object around on a 3D plane based directly on the mouse position:

You can represent your 3D target plane as:

• a 3D origin point O
• two 3D vectors U and V, representing the direction of the U- and V-axes

Then, a given 3D point corresponding to plane coordinates [u,v] is:

point3d P = O + u*U + v*V

Then, you can combine the operations that map this particular 3D point to the screen; this is usually described with 3D transformation matrices ModelMatrix, ViewMatrix, and ProjectionMatrix, and a viewport transform determined by a 2D screen origin point origin_2d and a 2D scaling vector scale_2d. To solve the problem in a readily invertible way, promote everything to homogeneous coordinates, by adding a .w-coordinate to each of them. This extra coordinate acts as a scaling factor -- to get cartesian coordinates back, you need to divide the homogeneous .x and .y values by the .w value:

P_hom = [u, v, 1] * [U.x, U.y, U.z, 0] = [u, v, 1] * TexMatrix
                    [V.x, V.y, V.z, 0]
                    [O.x, O.y, O.z, 1]

P_clip_hom = P_hom * ModelMatrix * ViewMatrix * ProjectionMatrix
           = P_hom * ModelViewProjectionMatrix

screenpos_hom = P_clip_hom * [scale_2d.x     0        0] = P_clip_hom * PortMatrix
                             [   0        scale_2d.y  0]
                             [   0           0        0]
                             [origin_2d.x origin_2d.y 1]

So, screenpos_hom = [u, v, 1] * TexMatrix * ModelViewProjectionMatrix * PortMatrix
                  = [u, v, 1] * TexToScreenMatrix

-> [screenpos.x, screenpos.y] = [screenpos_hom.x, screenpos_hom.y] / screenpos_hom.w

Note that TexToScreenMatrix is a 3x3 matrix; you should be able to invert it:

UV_2d_hom = [screenpos.x, screenpos.y, 1] * (TexToScreenMatrix)^-1

-> [u, v] = [UV_2d_hom.x, UV_2d_hom.y] / UV_2d_hom.w

Finally, you can either use the [u,v] coordinates directly, or you can use them to recreate the 3D point P as described above.

Agulhas

thanks! but I'm not sure to see how to use 3D transformation matrices in javascript. Also, what is "w" in my exemple ? – Mclin

I'm afraid I don't know how these are bound in on the javascript/webkit platform. If you can't get at the webkit transformation matrices directly, you may have to match the 3D math in javascript -- e.g., if you use rotateX(35deg), you would unfortunately need to set up your own 3D X-rotation matrix for 35 degrees and multiply it into your own ViewMatrix... – Agulhas

As for the "w", I tried to explain it briefly in my answer; division by "w" is what lets you do 3D perspective in the first place. If you want to understand it better than that, you need a longer and better presentation on basic 3D math; try wikipedia on 3D projection. – Agulhas

If the wikipedia article and other links you can find by googling on keywords like "3D", "perspective", "projection", and "homogenous" are no good, your best bet might be to use an actual book. I can recommend 3D Computer Graphics by Alan Watt, but there are a variety of good texts available -- see what you can find at your local university library. – Agulhas

It seems possible to get the webkit transformation matrices... I'll dig into that. Thanks – Mclin

distY = (moveY - startY); var rads = 60/360 * Math.PI * 2; distY = distY / Math.cos(rads); This is pretty close for the vertical translation: jsfiddle.net/Aubcr/3 – Benitobenjamen

@Benitobenjamen thanks, the result is a bit better, but even the vertical drag fails when you are close to the edges.. I'm trying to take the problem in the other way, I found a way to get the exact mouse coordinate relative to the 3D scene, not the screen, with this answer : https://mcmap.net/q/1581209/-receiving-transformed-mouse-event-data-from-dom-objects-with-a-css-3d-transform. But it's not perfect, now I have to manage objects and 3D scene mouse events in the same time – Mclin

@Mclin I'd love to see the final product. – Benitobenjamen