Is there a way to send a click event to a window in the background in python?

Question

So I'm trying to build a bot to automate some actions in a mobile game that I'm running on my pc through Bluestacks.

My program takes a screenshot of the window, looks for certain button templates in the image and returns their coordinates.

I would now like to be able to send a click event to the window at those coordinates, but since I would also like to do other things while the bot runs in the background I'm looking for a way to send the mouse event directly to the window (even if it's minimized/in the background), without influencing the movement of the mouse while I'm doing other stuff or bringing the window to the foreground/unminimizing it. Is this possible?

Paraphrastic

Answer 1

Based on Dmitry's answer. The x and y must be the coordinates relative to the Bluestacks window not the screen.

def click(x, y):
    hWnd = win32gui.FindWindow(None, "BlueStacks")
    lParam = win32api.MAKELONG(x, y)

    hWnd1= win32gui.FindWindowEx(hWnd, None, None, None)
    win32gui.SendMessage(hWnd1, win32con.WM_LBUTTONDOWN, win32con.MK_LBUTTON, lParam)
    win32gui.SendMessage(hWnd1, win32con.WM_LBUTTONUP, None, lParam)

click(100,100)

Neuro

Answer 2

I solved this problem on windws 10 using win32api.

In Spy++ I looked at the mouse messages that occur when I click in Bluestacks. I found that I should find the hwnd of bluestacks child window with the title "BlueStacks Android PluginAndroid". And send them mouse click events:

lParam = win32api.MAKELONG(x, y)
win32api.PostMessage(hWnd, win32con.WM_LBUTTONDOWN, win32con.MK_LBUTTON, lParam)
win32api.PostMessage(hWnd, win32con.WM_LBUTTONUP, None, lParam)

This works for me even if the window is minimized to the tray.

Thankless

Hi, thanks for your answer. Could you post a bit more of your code? I tried this but it doesn't work for me, it doesn't give any errors but no mouse events are sent either. – Paraphrastic

Answer 3

I learned that you'll need to use the correct libraries for this to work properly. I discovered that pynput works very well. Apart from what you're trying to do, it can fetch user input on the mouse and keyboard before your OS processes it.

Let me help you save a lot of time of going through all the different libraries: I suggest pynput. You can read about it here. Below you can find a piece of example code that I copied from this website (should it ever go offline):

from pynput.mouse import Button, Controller

mouse = Controller()

# Read pointer position
print('The current pointer position is {0}'.format(
    mouse.position))

# Set pointer position
mouse.position = (10, 20)
print('Now we have moved it to {0}'.format(
    mouse.position))

# Move pointer relative to current position
mouse.move(5, -5)

# Press and release
mouse.press(Button.left)
mouse.release(Button.left)

# Double click; this is different from pressing and releasing
# twice on Mac OSX
mouse.click(Button.left, 2)

# Scroll two steps down
mouse.scroll(0, 2)

Hydromel

From what I understand from its documentation however pynput doesn't allow you to send events directly to a specific window, it simply clicks on the specified location. – Paraphrastic

Oh I misread your question. You can still use pynput to temporarily disable user input, open up the window, click, minimize an enable user input again. – Hydromel

That would not be ideal though as the bot needs to take action for several seconds every minute or so – Paraphrastic

Answer 4

If you really want to automate bluestacks, you could use adb (which could do the assigned work like clicking a button, even when the window is minimized) you could use Ppadb (pure Python adb module) for automating it. And ppadb could be used in mac and Linux also, but win32 is restricted to Windows.

Louden

Answer 5

Use Bluestacks inbuilt macros if you want it to run in background. I tried both pyautogui and win32gui, they need the window to be in focus (at least for my game).

Alvord