Human-like mouse movements via Selenium

Question

The Story:

One of the approaches to solve captchas, like Google ReCaptcha, is to try to imitate the human mouse actions: movements, hovering and clicks.

Some users reported that making mouse moves as B-spline curves worked for them.

The Question:

How to move the mouse to a particular element following the B-spline trajectory via Selenium?

---

Note that the regular browser.actions().mouseMove(elm).perform(); would "jump" to the element straight and far too quickly. My understanding is that it is a matter of slowing down the movement speed, "jumping" from point to point smoothly following the mathematical model for the B-spline trajectory.

^{We are using Protractor/JavaScript, but the question is really language-agnostic. Note that I'm not trying to solve the captcha, or contribute to the "captcha-solving making new evil bots violating terms of use here and there" space. I'm just curious and eager to obtain more skills in the test automation space.}

Nolitta

Answer 1

You can use scipy.interpolate to interpolate B-spline curves like you can see in this question.

Here I'll use one of the B-spline examples to get values to x and y:

import numpy as np
import scipy.interpolate as si

# Curve base:
points = [[0, 0], [0, 2], [2, 3], [4, 0], [6, 3], [8, 2], [8, 0]];
points = np.array(points)

x = points[:,0]
y = points[:,1]


t = range(len(points))
ipl_t = np.linspace(0.0, len(points) - 1, 100)

x_tup = si.splrep(t, x, k=3)
y_tup = si.splrep(t, y, k=3)

x_list = list(x_tup)
xl = x.tolist()
x_list[1] = xl + [0.0, 0.0, 0.0, 0.0]

y_list = list(y_tup)
yl = y.tolist()
y_list[1] = yl + [0.0, 0.0, 0.0, 0.0]

x_i = si.splev(ipl_t, x_list) # x interpolate values
y_i = si.splev(ipl_t, y_list) # y interpolate values

With values of x and y, you can move the mouse cursor with ActionChains:

from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains

url = "https://codepen.io/falldowngoboone/pen/PwzPYv"
driver = webdriver.Chrome(executable_path="/home/selenium/chromedriver2.25")
driver.get(url)

action =  ActionChains(driver);

startElement = driver.find_element_by_id('drawer')

# First, go to your start point or Element:
action.move_to_element(startElement);
action.perform();

for mouse_x, mouse_y in zip(x_i, y_i):
    action.move_by_offset(mouse_x,mouse_y);
    action.perform();
    print(mouse_x, mouse_y)

Kindly

full code github.com/guilhermebferreira/selenium-notebooks/blob/master/… – Kindly

the curve I used is just an example, it would be necessary to make one according to the intended course. Or to go through specific points of the site – Kindly

@NinoŠkopac with a Selenium PHP WebDriver (or similar) sure – Kindly

I don't get it. This makes the mouse just jump from the original position to the 1st position of the spline then back again, followed by a jump to the 2nd point and back again, etc. Doesn't look human at all, as long as you can't lock the cursor and travel along the spline without jumping back to the original point after every "sleep". It kind of works if you put the "action.perform()" after the loop and remove the sleep. However, this way you have no way to control the speed of the movement directly. You can just add more points to slow it down a little – Mistrial

@ODIUM You need to adjust this to your need. The start and end position should be the path you need the mouse to do on the screen (button positions). Then a curve is generated between these two points, which will be the way traveled by the mouse. – Kindly

I did not see Human-like mouse movements via Selenium in this answer. – Jeffery

@F.Vosnim the path taken by the mouse pointer, between the start point and destination, is made through a curve. Instead of a straight line / or magically appear at the destination point. This is a way to emulate a non-robotic movement. – Kindly

@Kindly Your answer is 5% of the question. Human-like movements can be divided into 2 parts: spatial and temporal. You do not show which b-spline curve can imitate human. Simple curves fails almost everywhere. And there is no temporal component at all. – Jeffery

@F.Vosnim My intention was only to answer the asked question ("How to move the mouse to a particular element following the B-spline trajectory via Selenium?"), It's not a bulletproof solution – Kindly

Why is there a delay between each action.perform? Does selenium implements this by default? – Stephaniastephanie

@JinhuaWang I know it has a option to set delay, but I'm not sure if it has a default value, or it's just the time spend to perform – Kindly

Answer 2

@ODIUM @Guilherme , or anyone still looking for a fix. What ODIUM described in Guilherme's answer as a jump to the first curve position then back to the beginning, then second curve position and back to the beginning again is caused by a small bug in the provided code. It will be fixed by "resetting" the action chain after each perform, like this:

action =  ActionChains(driver);
startElement = driver.find_element_by_id('drawer')

# First, go to your start point or Element:
action.move_to_element(startElement);
action.perform();

for mouse_x, mouse_y in zip(x_i, y_i):
    # Here you should reset the ActionChain and the 'jump' wont happen:
    action =  ActionChains(driver)
    action.move_by_offset(mouse_x,mouse_y);
    action.perform();
    print(mouse_x, mouse_y)

Measured

Answer 3

If you were running this from desktop wand wanted to use an actual mouse movement, with AutoIt you can make mouse movements delayed.

Fonda

Welcome to SO! A link-based answer is very risky as if the links disappears, there is no answer. Please, edit it and explain what can be found in the link. – Mestizo