Mapbox/Maplibre raycast with only a projection matrix and georeferenced objects

Question

I guess this could be considered a continuation of this question so please check it out for context about his one, especially the JSFiddle reported in said question.

I'm basically trying to achieve the same, however I'm trying to plot georeferenced lines (builing them from points), but I still need to be able to make the raycaster functional.

My main issue was that if I plotted them without "moving" them closer to the center of the THREE.js world (by using what provided in the snippet below), otherwise the rendering was utterly imprecise (those lines need a precision of more or less 1/10 of meter to look decent and due to gpu tansforms their coordinates were being mangled).

...
this.center = MercatorCoordinate.fromLngLat(map.getCenter(), 0);
...
// lines is an array of lines, and each line is an array of georeferenced 3d points
lines.forEach((points) => { 
  const geometry = new THREE.BufferGeometry().setFromPoints(points.map(({lng, lat, elev}) => {
    const { x, y, z } = MercatorCoordinate.fromLngLat([lng, lat], elev);
    return new THREE.Vector3(x1 - center.x, y - center.y, z - center.z);
  }));
  const line = new THREE.Line(geometry, material);
  this.scene.add(line);
});

So, their visualization works correctly this way, but only if I don't consider scaling and rotating in the camera transforms (so by setting this.cameraTransform = new THREE.Matrix4().setPosition(x, y, z); in the fiddle mentioned above).

Clearly, by using this approach, the transformation about scale and rotation, that I supposed are indeed needed to make the raycaster work, aren't working anymore.

Whatever solution found online about positioning 3d objects with their lng/lat coordinates is quite superficial and lacks proper documentation, so I couldn't really figure out how to make this...

Any idea?

    class GraphsLayer {
      type = 'custom';
      renderingMode = '3d';
      constructor(id) {
        this.id = id;
      }
      async onAdd (map, gl) {
        this.map = map;
        const { width, height } = map.transform;
        this.camera = new THREE.PerspectiveCamera(28, width / height, 0.1, 1e6);
        const centerLngLat = map.getCenter();
        this.center = MercatorCoordinate.fromLngLat(centerLngLat, 0);
        const { x, y, z } = this.center;
        const s = this.center.meterInMercatorCoordinateUnits();
        const scale = new THREE.Matrix4().makeScale(s, s, -s);
        const rotation = new THREE.Matrix4().multiplyMatrices(
          new THREE.Matrix4().makeRotationX(-0.5 * Math.PI),
          new THREE.Matrix4().makeRotationY(Math.PI),
        );
        this.cameraTransform = new THREE.Matrix4().multiplyMatrices(scale, rotation).setPosition(x, y, z); // displaying of segments don't work with this
        this.cameraTransform = new THREE.Matrix4().setPosition(x, y, z); // this instead works for displaying but not for the raycaster
        this.scene = new THREE.Scene();
        const material = new THREE.LineBasicMaterial({ color: 0xff0000, linewidth: 3 });

        const segments = await (await fetch('lines.json')).json();

        segments.forEach((s) => {
          const geometry = new THREE.BufferGeometry().setFromPoints(s.coos.map(p => {
            const { x: x1, y: y1, z: z1 } = MercatorCoordinate.fromLngLat([p[0], p[1]], p[2]);
            return new THREE.Vector3(x1 - x, y1 - y, z1 - z);
          }));
          const line = new THREE.Line(geometry, material);
          this.scene.add(line);
        });

        this.renderer = new THREE.WebGLRenderer({
          canvas: map.getCanvas(),
          context: gl,
          antialias: true,
        });
        this.renderer.autoClear = false;
        this.raycaster = new THREE.Raycaster();
      }

      render (gl, matrix) {
        this.camera.projectionMatrix = new THREE.Matrix4()
          .fromArray(matrix)
          .multiply(this.cameraTransform);
        this.renderer.resetState();
        this.renderer.render(this.scene, this.camera);
      }

      raycast ({ x, y }) {
        const { width, height } = this.map.transform;
        const camInverseProjection = this.camera.projectionMatrix.clone().invert();
        const cameraPosition = new THREE.Vector3().applyMatrix4(camInverseProjection);
        const mousePosition = new THREE.Vector3(
          (x / width) * 2 - 1, 1 - (y / height) * 2, 1,
        ).applyMatrix4(camInverseProjection);
        const viewDirection = mousePosition.sub(cameraPosition).normalize();

        this.raycaster.set(cameraPosition, viewDirection);

        // calculate objects intersecting the picking ray
        var intersects = this.raycaster.intersectObjects(this.scene.children, true);
        console.log(intersects)
      }
    }

This works for displaying, but the raycaster isn't working.

UPDATE:

I put everything I got on here.

Shaker

Hi there! Could you please create a fiddle with what you have? Just take my original and adapt it with your code. Most important of all, please add some data, e.g. at least one line from lines.json as a constant array declared as part of the fiddle :) – Hark

Hi @Isolin, thanks for your reply! Appended a jsfiddle on the question, with some data as requested. – Shaker

Thank you, I hope I will find time to look on it in the following days. – Hark

Answer 1

You are using a Maplibre-gl, a different library than the referenced question. Their APIs are not compatible, that explains the issues you have experienced. Both of then are well visible using some simple debugging. If you console.log some variables in the raycast method, you will see that they are wrong: undefined and NaN.

Mouse move event

Deconstruction of the mousemove event fails, because you need to pass the point property. At least this is how it works on FF, I did not check any other browsers.

map.on('mousemove', (e) => graphsLayer.raycast(e.point)) // < add .point here

Canvas size

The API you are using is different to Mapbox. There are no width and height properties in the Map object. You need to get the canvas and extract them out from there.

const {
    clientWidth,
    clientHeight
} = this.map.getCanvas();

Of course you later use the new variable names.

const mousePosition = new THREE.Vector3(
    (x / clientWidth) * 2 - 1, 1 - (y / clientHeight) * 2, 1,
).applyMatrix4(camInverseProjection);

Scaling and rotation

In my answer to the original question I stressed out that MapBox was (perhaps still is) using a different coordinate system than OpenGL/WebGL/Three.js. So the scaling and rotation were needed to compensate for that. Maplibre-gl, as its name suggests, is oriented on OpenGL, thus there is no more conversion necessary. That is also why you correctly discovered, that omitting the rotation and scaling yields correct display.

// No need for this
// const s = this.center.meterInMercatorCoordinateUnits();
// const scale = new THREE.Matrix4().makeScale(s, s, -s);
// const rotation = new THREE.Matrix4().multiplyMatrices(
//     new THREE.Matrix4().makeRotationX(-0.5 * Math.PI),
//     new THREE.Matrix4().makeRotationY(Math.PI),
// );
// this.cameraTransform = new THREE.Matrix4().multiplyMatrices(scale, rotation).setPosition(x, y, z);

// Just keep this
this.cameraTransform = new THREE.Matrix4().setPosition(x, y, z); 

Hark