How to get the e.screenX/Y relative to the host screen on window.onmousemove when using a secondary monitor?

Question

I thought this one would be easy, but it turns out it's not.

Take the following situation: a primary desktop on the bottom, and a secondary desktop above that primary desktop.

The objective is to get the mouse X and Y position relative to the screen hosting the window on the secondary desktop. The naive approach would be:

const onMouseMove = e => {
  const hostScreenX = e.screenX;
  const hostScreenY = e.screenY;

  document.body.innerHTML = `${hostScreenX}x${hostScreenY}`;
};

window.addEventListener("mousemove", onMouseMove, false);

Turns out this returns a negative number for screenY when running on the secondary screen (the top one). The primary screen is seen as the "zero boundary" and everything before that is in a negative space.

We can account for this using screen.availTop and screen.availLeft:

const onMouseMove = e => {
  const hostScreenX = e.screenX - window.screen.availLeft;
  const hostScreenY = e.screenY - window.screen.availTop;

  document.body.innerHTML = `${hostScreenX}x${hostScreenY}`;
};

window.addEventListener("mousemove", onMouseMove, false);

Much better, except it's actually incorrect: because the availTop space includes the space of the permanent operating system UI (on MacOS the toolbar, and on Windows the toolbar if you moved the toolbar from the default bottom to the top), we are incorrectly subtracting the toolbar space.

E.g. on MacOS in chrome when moving the mouse to the far top this would give us a hostScreenY of 80, whereas it should be 103 (the macos toolbar is 23 pixels).

You might then think that we can solve for this by adding back the difference in height - availHeight:

const onMouseMove = e => {
  const hostScreenX = e.screenX - window.screen.availLeft + (window.screen.width - window.screen.availWidth);
  const hostScreenY = e.screenY - window.screen.availTop + (window.screen.height - window.screen.availHeight);

  document.body.innerHTML = `${hostScreenX}x${hostScreenY}`;
};

window.addEventListener("mousemove", onMouseMove, false);

And indeed this seems to be perfect on MacOS.

The problem is that this assumes that the toolbar of the operating system is on the top of the screen, which is usually not true on Windows (though sometimes it is if you move your toolbar to the top). As a result, the values are offset by the toolbar space.

How can we get the screenX and screenY relative to the screen hosting the current window when using secondary (or even tertiary) monitors on the mousemove event in Javascript?

Skin

This concept is useful in video conferencing apps with white board sharing. – Soelch

Answer 1

Is this solution suitable for your requirements?

It also works with 2nd monitor on the left of the 1st one, with the toolbar of the operating system on the right side of 1st monitor.

const onMouseMove = e => {
  const { screen } = window;
  const { availLeft, availTop } = screen;
  const hostScreenX=e.screenX-availLeft+ (availLeft ? (screen.width - screen.availWidth) : 0);
  const hostScreenY=e.screenY-availTop+ (availTop ? (screen.height - screen.availHeight) : 0);

  document.body.innerHTML = `${devicePixelRatio} ${hostScreenX}x${hostScreenY} ` +
    `${hostScreenX * devicePixelRatio}x${hostScreenY * devicePixelRatio}`;
};

window.addEventListener("mousemove", onMouseMove, false);

P.S.

I have only a Windows 10 PC, I could test this only on my platform

Edit: If we need the coordinates in actual screen resolution we can use the devicePixelRatio multiplayer. The problem is that devicePixelRatio could change between screens. If we need to know the coordinates only till the mouse pointer is in the screen hosting our window, no problem; if we need to continue getting the coordinates on the other screens, many other problems came in the scene. I suggest to discuss them in future edits if really required.

Moises

This doesn't work on windows when the secondary screen is to the left of the primary screen and the window is on the secondary screen. See my screen recording at app.usebubbles.com/68yqKUMBdzZguNnKGBtiic – Skin

Probably I didn't understood what you are looking for. Tested it right now and bringing the mouse pointer on top of the screen I get Y=0: what I was expecting. What do you expect when mouse is on the top of the screen? Why you are stating that at that offset Y should be 72 rather than 110? Is it a problem of scale or a problem of offset? – Moises

You wouldn't get Y=0 because we're looking for the screen position (not the client position) and the browser has its own address bar and UI elements, so the Y would always be > 0 even at the smallest value. I am expecting 72 rather than 110 simply because my mouse at that moment is at 72 pixels in the Y coordinate relative to the screen the window is on. – Skin

@Skin to test the script out of the client is enough to press primary mouse button in the test div then move the mouse where you need, that's how I can have Y=0. Once said that, 72 rather than 110 is due to an offset problem or due to a scale problem? Thank you – Moises

Tested this again and I suspect the reported problem is a scale problem, @Skin . I edited my answer to try to find a solution. – Moises

To be more specific, to test the script outside the test div, pres the mouse button and move the mouse while keeping pressed the mouse button, tested with chrome. – Moises

As previously mentioned, your approach is bugged on Windows when the window is on a secondary screen and the secondary screen is to the left of the primary screen. See further evidence in this new screen recording: app.usebubbles.com/c5vBuG249EoRSRoJ9Qyyqm/… – Skin

Answer 2

According to my research, it is currently not possible with JavaScript to get differentiated information about screens and screen sizes, not to mention location and/or size of OS-specific stuff like the macOS menu bar.

See for example Testing for multiple screens with javascript, where the best idea is to make educated guesses.

I tried the MediaDevices API (mediaDevices.enumerateDevices), but that doesn't reveal information about screen devices:

And I fear that we don't get this info anytime soon, at least without asking for users consent, because the browser vendors go to great lengths to prevent browser fingerprinting (see amiunique.org for an explanation).

---

If you can package your code as an Electron application you can access all info needed with the screen API:

Electron REPL, for example:

$ npx electron --interactive
> const { screen } = require('electron')
undefined
> screen.getAllDisplays()
[
  {
    id: 69733632,
    bounds: { x: 0, y: 0, width: 1440, height: 900 },
    workArea: { x: 0, y: 23, width: 1440, height: 877 },
    accelerometerSupport: 'unknown',
    monochrome: false,
    colorDepth: 30,
    colorSpace: '{primaries:SMPTEST432_1, transfer:IEC61966_2_1, matrix:RGB, range:FULL}',
    depthPerComponent: 10,
    size: { width: 1440, height: 900 },
    workAreaSize: { width: 1440, height: 877 },
    scaleFactor: 2,
    rotation: 0,
    internal: false,
    touchSupport: 'unknown'
  }
]

(I currently have no second display attached)

With bounds: { x: 0, y: 0, width: 1440, height: 900 } (and workArea: ...) you can calculate the size and position of the elements occupying screen space, like menu bars, toolbars or docks.

---

If you could package your code in a Chrome extension, you can use the system.display API, getInfo() ⇾ workArea

Adessive

I'm not trying to get information on the different screens than the current window is on. I already have the correct dimensions of the host screen, as it is available and correct from window.screens.width/height. I'm also not trying to get information about the OS toolbar. The only reason this is mentioned was because the e.screenX/Y did not actually return the host screen X or Y, but rather the virtual screen X or Y. When trying to resolve for this using screen.availTop and availLeft, the result was that there was a margin of error equal to the size of the OS toolbar. – Skin

The chrome extension API or the electron API do not help here because we already know the correct size of the window screen from window.screen.width/height. The problem is that the e.screenX/Y is incorrect, and we need to correct for it to get to the host screen location. Chrome extensions do not have an API to get this mouse location as far as I am aware. – Skin

Ok, I now think I initially misunderstood your question. But after re-reading (in fact multiple times) I think that you still need to get the screen information as I wrote: to solve your problem, you need to know the location of the OS toolbar on the primary screen, while the current window being on an secondary display. This can only be calculated using regular web API (availTop, availHeight) for the screen our window is on (= for the secondary display), right? – Adessive

availTop / availHeight won't tell you where a OS toolbar is located, neither will the APIs you referenced, right? The information those APIs expose is already available from web APIs – Skin

Those APIs expose this information for all screens, not just the screen the current window is on (web API), that's the difference AFAIU. And why can't you derive from available top, available height and full screen height the location of OS toolbars? – Adessive