Background
When using a Steam Controller you can configure the touchpads to be complex menus that show up on the screen when you start touching them. These menus are very handy for games with many controls. They look like this:
However when playing a game in SteamVR none of these overlays appear in VR. They only appear on the desktop window that is rendered alongside with it.
This is a bug or a missing feature that is known. My goal was to write a fix for it. But that turned out to be quite tricky.
My Problem
I know how to render an overlay in OpenVR/SteamVR. Valve's Documentation is enough for me. But I didn't come up with a good way of capturing that overlay that is drawn on top of the game's window.
There are two ideas in my mind. But I don't know how to implement them properly.
Capture the overlay
I know nothing about the technical details of the Steam Controller Overlay. However I think it's not a bad assumption that it has the same or very similar implementation as the Steam Overlay.
According to the Steamworks API Documentation Steam listens for Present/SwapBuffers calls that the application fires and refreshes the overlay accordingly. (See https://partner.steamgames.com/doc/api/ISteamUtils#BOverlayNeedsPresent ) Also my guess is that Steam gets the hWnd of the window, creates an OpenGL context with its DC and renders to it every time it receives a SwapBuffers call that it is listening to.
But that's the end of the story. I don't know how to actually read what Steam is rendering on it. I have thought of hooking myself into the drawing message queues Steam is using and capturing what is rendered on the window before and after the overlay is drawn. I also thought of tricking Steam into thinking a different window with a green background is the one it should render to and then just capturing its contents. But these are just guesses into the wild. I have no clue how I would implement that.
Using Chroma Keying
The Selected Button has a very specific flat and opaque blue tone. I could make screenshots of the window and then only take the pixels that have this specific color. The disadvantage is quite obvious. All the other areas are semitransparent and thus unsuitable for chroma keying. Same goes for the antialiased text. Fortunately you can change the icon and their color like in the picture below. So that would kind of work, but it would be extremely ugly.
Does anyone have a better idea or a solution to my problem?
