Android 10 injecting inputs to External Display / Virtual Display

Question

Concept:

I have an Android application written in Kotlin that simply launches an Android application on an external display. The display could be a AirServer, Miracast, Microsoft Connect, USB-OTG HDMI, USB-C HDMI, or a Simulated Secondary Display (see screenshot).

Here's the except of the code that does the "heavy lifting":

        val intent = packageManager.getLaunchIntentForPackage(info.id)
        val dm = recyclerView.context.getSystemService(Service.DISPLAY_SERVICE) as DisplayManager
        val displays = dm.getDisplays(DisplayManager.DISPLAY_CATEGORY_PRESENTATION)

        for (display in displays) {
            val options = ActivityOptions.makeBasic()
            options.launchDisplayId = display.displayId
            recyclerView.context.startActivity(intent, options.toBundle())
            break
        }

The applications start fine. See the screenshot, where I used it to launch Minecraft Pocket Edition to a Simulated Secondary Display (as if an external HDMI monitor was connected). The issue is that I cannot send touch or cursor inputs to the virtual displays.

What Works:

• adb shell input -d <display-id> tap <x> <y>
• Wireless Controllers
• Wireless Keyboards

What Doesn't Work:

• Wireless / Wired Mice (cursor hits the edges of the phone screen, never enters the Virtual Display / External Display screen)
• Touch (see screenshot)

What I've Tried:

The output of dumpsys display reveals that the Virtual Displays all lack the touch VIRTUAL parameter that the internal display has. I'm not sure if this means that the display itself doesn't support touchscreen input, or if it's simply not enabled.

I tried forcing the application into the foreground, which made the apps detect keyboard and controller inputs, but the cursor was still locked to the internal display window.

It's worth noting that the Android 10 Desktop Mode Developer Setting DOES put the cursor into the external display instead of the internal display. This is what I'm trying to accomplish.

There appears to be a hidden Java API within Androids SDK:

https://github.com/aosp-mirror/platform_frameworks_base/blob/a4ddee215e41ea232340c14ef92d6e9f290e5174/services/core/jni/com_android_server_input_InputManagerService.cpp#L825

I attempted to access this class via reflection and call setFocusedDisplay, but I kept getting ClassNotFound exceptions, even with the private API blacklist secure settings changed.

Any help would be appreciated here. The only other related threat to this one is here, and it was never solved:

Android Q VirtualDisplay touch input events

Loyce

I accidentally linked the JNI class. Here's the Java class: android.googlesource.com/platform/frameworks/base/+/refs/heads/… – Loyce

I'm trying to achieve something very similar, but I get stuck at launching external apps on virtual monitors. I get a permission error, and it seems that my app has to have a system-level permission to get by it. Can you kindly explain how did you achieve this? – Phagocyte

I never got an error like that. Check out ExtLaunch on my GitHub, it shows how I did it (username tytydraco). I just pass the display ID with the intent iirc. – Loyce

I checked it out. It seems the permission problem appears only if you try to launch foreign applications inside your "OWN" virtual displays, ones that are created through DisplayManager.createVirtualDisplay(). In Launcher.kt, you don't create a virtual display, you use existing ones. The virtual display you were referring to is probably the one created from the developer setting "Simulate secondary displays". So, that sorts it out! By the way, I've checked your Google Play account. Cool stuff! Thanks for supporting open-source, and for answering my question :) – Phagocyte

Answer 1

I've solve my problem. The input directed to the external display is controlled by the Android internal framework. By default, external mice are directed to the internal display. To have them go to the external display, Force Desktop Mode must be enabled in Developer Settings. There is currently no other way to forward the pointer.

Loyce