Show current cursor position in Selenium

Question

Is there a way to show current cursor position or something like this? I have action chain that should click on the exact point on some object but I guess I've picked up wrong coordinates. I use Firefox webdriver. Here's how the script looks like:

from selenium import webdriver
import time
from selenium.webdriver.common.action_chains import ActionChains

driver = webdriver.Firefox()
driver.get("http://www.mysite/")
elem = driver.find_element_by_xpath('//div[@id="player"][1]/object[@type="application/x-shockwave-flash"]')
action_chains = ActionChains(driver)
action_chains.move_to_element_with_offset(elem, 660, 420).perform()
action_chains.click().perform()
time.sleep(10)
driver.close()

Inflationary

It's a very bad way of doing it. You will most likely come across problems as soon as you port this to another browser / computer / etc. There is almost always a way to make a client action without using the mouse, and even if it's the only way, there should still be a way to do it without using the cursor position. If that was the only way, then you would be able to click the element only when the browser window was set to a specific size. If you provide the URL of the website that you you are trying to access, then I might be able to find a better way for you to achieve what you're trying... – Stanislaw

@barakmanos Thanks for your comment. Unfortunately, there's no other way for me to work with that object on page. For example, lets say that I would like to start video and change its resolution on jwplayer.com but without using its API. – Inflationary

I agree with barak manos that in the general case you don't want to do this but I see that you're trying to click in to a flash player. I don't know of any solution that operates by controlling only the browser (like Selenium does) that will not depend on using coordinates to perform this test. However, maybe there are testing solutions designed for flash that would be able to find flash elements like Selenium is able to find DOM elements? – Annabelle

consider this javascript option: https://mcmap.net/q/1009649/-how-to-find-quot-cursor-quot-position-in-webpage-with-selenium-webdriver – Coccidioidomycosis

Answer 1

I had an opaque HTML5 canvas where I had to check values of tooltips at various offsets, and what trouble! After lots of banging my head, I learned several important things:

1. move_to_element_by_offset has issues prior to 2.42. See also https://code.google.com/p/selenium/issues/detail?id=4215

2. Even after an update, move_to_element_by_offset still seems persnickety. I switched to a combination of the ActionChains move_to_element hover (defaults to the center of the element) and a second ActionChains with move_by_offset

3. Now that you're in position, re-request the position-sensitive element. In my case, that's a tooltip object. This selenium.webdriver.remote.element will have the .location['x'] and .location['y'] members you need to sanity check your position.

Things that don't work for grabbing cursor position after a perform'd ActionChain to hover, to save everyone some time:

• .location['x'] and .location['y'] on the element you pass into move_to_element. These values are not updated by ActionChains.
• selenium.webdriver.remote.webdriver switch_to_active_element; I'm not clear what elements qualify as "active", but it doesn't acknowledge our tooltips as active elements and gave me back [0, 0] coordinates.
• selenium.webdriver.remote.webdriver get_window_position; the documentation was a bit vague, but this does just what the function suggests: a window rather than a cursor position.

Transalpine

Answer 2

If all you want to confirm is whether Selenium is clicking on the right element, instead of click()-ing on the element:

actions.move_to_element(my_element).click().perform()

context_click() on it:

actions.move_to_element(my_element).context_click().perform()

Selenium will right click on your element, which will result in the context menu being displayed at the bottom right corner of your cursor. Once you see the position of the context menu, you will know whether Selenium is clicking on the right element.

Syrup

Answer 3

Yeah, this problem is frustrating.

This method will allow you to see where the mouse is by painting a red dot on it: Start by adding the following CSS to your webpage, temporarily:

// TODO: Remove me after you've figured out where the mouse is moving to.
.dot {
  background: red;
  position: absolute;
  width: 2px;
  height: 2px;
  z-index: 10000;
}

Then, in your code, just before you're calling the *.moveTo() method, add a pause to allow you to inject code into the browser's console. This code is in JavaScript (JS / webdriver.io) but it's easy to add in any language:

// Other test stuff...

console.log("This gives you 10 seconds to inject the code..."); // <-- Add this and the next line
browser.pause(10000);

browser.moveTo(...gawdKnowsWhere);

Now, when you see the test command line console is waiting 10 seconds, open up Chrome/Firefox's developer tools (F12) and paste this snippet into the browser's "Console" tab:

  (function() {
    "use strict";

    document.onmousemove = handleMouseMove;
    function handleMouseMove(event) {
      var dot, eventDoc, doc, body, pageX, pageY;
      
      event = event || window.event; // IE-ism
      
      // If pageX/Y aren't available and clientX/Y
      // are, calculate pageX/Y - logic taken from jQuery
            // Calculate pageX/Y if missing and clientX/Y available
      if (event.pageX == null && event.clientX != null) {
        eventDoc = (event.target && event.target.ownerDocument) || document;
        doc = eventDoc.documentElement;
        body = eventDoc.body;

        event.pageX = event.clientX +
          (doc && doc.scrollLeft || body && body.scrollLeft || 0) -
          (doc && doc.clientLeft || body && body.clientLeft || 0);
        event.pageY = event.clientY +
          (doc && doc.scrollTop  || body && body.scrollTop  || 0) -
          (doc && doc.clientTop  || body && body.clientTop  || 0 );
      }

      // Add a dot to follow the cursor
      dot = document.createElement('div');
      dot.className = "dot";
      dot.style.left = event.pageX + "px";
      dot.style.top = event.pageY + "px";
      document.body.appendChild(dot);
    }
  })();

Now, go hunting for a red dot. Wherever the mouse is moved to, it will not be able to click there because there's a red square in the way now. But at least you can see where it's trying to go.

Can't see the dot?

Some tips:

• Make the dots bigger in the CSS by modifying the width & height.
• Move your mouse across the screen and watch all the red dots. Are they showing up when you move your mouse?
• Make sure you can paint a red dot where you expect it to be. The z-order might not be high enough, etc.

Bathsheba

Answer 4

I don't see any possible methods for changing the resolution directly as we can not use any locator for this.

For your example of JWPlayer, I am sure that there might be some javascript snippet which will allow changing the resolution. Using the Selenium JavascriptExecutor you can run that javacript code to simulate the resolution change.. although not by clicking the cursor.

WebDriver driver = new ChromeDriver();

if (driver instanceof JavascriptExecutor) {
    ((JavascriptExecutor) driver).executeScript(YOUR_JAVA_SCRIPT_STRING);
}

I noticed you mentioned you don't want to use any native JWPlayer API. I see this as the only possible solution.

Chore

Answer 5

It was a little hard for me as a selenium's newbie to understand that it just can't generate clicks on flash objects. The only possible solution I see here is to use sikuli-script.

I've found the project on GitHub: https://github.com/kevlened/sikuli_cpython. It allows to use sikuli scripts (written in jython) in pure python so I guess I will be able to write some sikuli scripts and integrate them with selenium tests.

Inflationary

you are not answering the question. The question ask how to check the cursor position – Metts

Answer 6

This answer gives the xy coordinates of the cursor:

def main():
    try:
        driver.get(LOGIN_URL)

        # Inject the JavaScript code
        js_code = """
        document.addEventListener('mousemove', updateCursorPosition);
        function updateCursorPosition(event) {
            window.cursorX = event.clientX;
            window.cursorY = event.clientY;
        }
        """
        driver.execute_script(js_code)
        time.sleep(2)  # Give the JavaScript code time to set up

        while True:
            # Retrieve the cursor position values
            cursor_x = driver.execute_script("return window.cursorX || 0;")
            cursor_y = driver.execute_script("return window.cursorY || 0;")
            print(f"Cursor Position: X = {cursor_x}, Y = {cursor_y}")

            time.sleep(0.5)  # Adjust the interval as needed

    except Exception as e:
        logging.error(f"An error occurred: {e}")
        # Optionally log more details only in development/debugging:
        # if __debug__:
        #     logging.error(traceback.format_exc())

    finally:
        driver.quit()```

this below js code can be used to mark the spot on the ui in browser (just replace it with the above js code )  


 document.addEventListener('mousemove', createCircle);

        function createCircle(event) {
            const circle = document.createElement('div');
            circle.style.position = 'absolute';
            circle.style.width = '20px';
            circle.style.height = '20px';
            circle.style.borderRadius = '50%';
            circle.style.backgroundColor = 'red';
            circle.style.left = (event.clientX - 10) + 'px'; // Center the circle
            circle.style.top = (event.clientY - 10) + 'px';  // Center the circle
            circle.style.zIndex = 9999; // Ensure it's on top
            document.body.appendChild(circle);
            setTimeout(() => {
                circle.remove();
            }, 1000); // Remove after 1 second
        }

Glanti