Screen scraping an application window and interacting with the mouse and keyboard

Question

The other day I found myself addicted to a Flash game and frustrated by the thing at the same time. In a moment of frustration with the game I thought I would make a 'bot' to beat it for me. Well, I really wouldn't, but it made me realize: I don't know how to interact with another application in a way to do this. Which brings me to the question, how would one take screenshots of another running application and interact with it with the keyboard and mouse. Ideally the solution would be in a managed language like c#.

When doing the background reading the net was drowning with articles on scraping HTML. There were not many articles on actually screen scraping an application.

I'm looking for a way to interface with another application rather than script/macro another application.

Could something like Xming be used to redirect the interface? http://www.straightrunning.com/XmingNotes/

Perhaps a Terminal Services client? http://www.codeproject.com/KB/cs/RemoteDesktop_CSharpNET.aspx

Forwarder

What exactly are you trying to accomplish? If you are looking for a "visual programming" approach, go for Sikuli. If you are looking for a programmable/selective screen recorder ("screen scraper"), maybe VNC is easiest to use. The server is already there, and the client is pretty simple to write (i wrote a simple Java client in ~300 lines of code, GUI and all). – Socman

I was looking for a way to capture screenshots of an application and manipulate the keyboard/mouse from something like c# (as opposed to a static script). The thought was a program would do some image work from the screenshot, and then respond with keyboard/mouse input. Sikuli looks great, however, it's a little different from what I am looking for. – Forwarder

Answer 1

Check out Sikuli, it is basically what you are looking for. It is written in Java however.

http://groups.csail.mit.edu/uid/sikuli/

Socman

Another +1 for Sikuli only wished they had a .NET api to programatically access that powerful engine. Maybe someday. – Peradventure

Answer 2

I ended up making the bot which did all this and documented it in a post

http://www.charlesrcook.com/archive/2010/09/05/creating-a-bejeweled-blitz-bot-in-c.aspx

Forwarder

Answer 3

I have used AutoHotKey for application automation.

Shannanshannen

Thanks for the suggestion, but i'm looking to interact with the application more than automate – Forwarder