Controlling iOS device via mouse/keyboard [closed]

Question

Some companies are offering manual testing of real iPhone/iPad devices. With your mouse and keyboard, you can control the device straight from your browser.

They probably use something like AirPlay to stream the device graphics to the browser. But how do they convert the mouse-clicks to touch events on iPhone/iPad? Since it's not possible to run a VNC server on the device, I'm wondering if there's another way to do this.

Penury

Are they using real devices, or could it be a simulator running remotely? Have an example? – Lewert

#6742727 , apple.stackexchange.com/questions/203178/… same as this – Henigman

@BeauYoung they're using real devices: perfectomobile.com - keynotedeviceanywhere.com – Penury

Based on this comment (#33185443), it seems like you're looking for a tool recommendation. Those kinds of questions are off-topic for StackOverflow. – Bellwort

@Bellwort I'm looking for a solution, which is on-topic for StackOverflow. – Penury

@Penury By that reasoning, "why isn't my laptop recognizing my USB drive" would be on-topic. I'm looking for a solution, after all. – Ferreira

By that reasoning, asking what you get when you dissolve salt in water would also be on topic. – Shane

@Shane thanks for your on-topic input, much appreciated – Penury

Answer 1

I do not know exactly how this services work, but i guess you have to include a framework from them in your app. This framework may contain a little server component which receives the mouse events and applies them to the app.

func applicationDidBecomeActive(application: UIApplication) {
    MyRemoteControlFramework.startServer()
}

If one mouse-clicks on a location on the screen, the framework may first perform a hittest to find the view which should receive the click (see Hittest on UIView). When the target view is found, the framework can call touchesBegan, touchesMoved, touchesEnded, ... on this view. It is a little work but i think it is not to complex. Also the screen can be captured easily and sent through the server component (see How to take a screenshot).

Cracy

A caveat of this solution is that you have to add extra code to support the incoming data from the framework, and have to handle the parsing. It seems that OP wants VNC-like solution, which is independent of any specific app and interacts with the phone itself. (But is that even possible? Hmm) – Dooley

It is for sure not possible to control the whole phone (without jailbreak it or having a robot arm touching the screen). But if you just want to remote control your app, then i think this is the way to go. – Cracy

You can attach hardware to the phone (lightning connector) which emulates touches like a stylus or keyboard etc., that would also work. It depends on what the use case is. – Cracy

I'm aware of this solution. What I'm looking for is an overall solution, which works on a vanilla iPhone without modifying apps. Something like github.com/iolate/SimulateTouch but without having to jailbreak the device – Penury