Overlaying on a 3D fullscreen application

Question

I want to display some custom graphics on top of a 3rd party fullscreen Windows application.

Have you played any Steam games? It has an executable, GameOverlayUI.exe that lets you access Steam windows from within a game. (The GUI elements look custom-drawn, I don't know if that is a necessity; but they look exactly the same inside a game as they do on the desktop.) It even goes as far to 'dim' the game graphics in the background.

I'm wondering how one could go about writing such an application.

I'm also wondering how broad the scope of solutions would be. Could you use one method for all OpenGL applications? For all Direct3D minus DirectDraw applications? For all fullscreen Windows applications?

I'm looking for more 'modern' and generic solutions - overlaying a command prompt or a 320x240 indexed color application isn't a concern.

Edit: Some clarification: The fullscreen application isn't mine. It can be anything. Most probably, it will be a 3D game. I want to display, say a digital clock in the bottom right corner of the screen, on top of the game graphics. I would like to avoid tricks such as injecting code or debugging the process, if possible.

Toastmaster

Answer 1

Well, here's another example. Xfire is a chat program that overlays its interface into games so you can receive messages while playing. The way they do this is by editing the d3d/opengl DLL at the process memory level, like injecting assembly jumps. I know this because their method was causing my mod to crash, and I actually saw the strange assembly code they were injecting into the d3d9.dll.

So here's how Xfire does this:

1. target the process of the game
2. search its process memory for d3d.dll/opengl.dll
3. inject into the process a DLL containing the custom interface logic
4. connect interface's functions to the program flow by manually writing assembly jumps in the d3d/opengl functions found in the process memory

There's no other conceivable way since you need to hijack the graphics device that belongs to that game. I'm willing to bet that Steam does it the same way as Xfire. So yeah, no easy way to do this unless someone created a helpful library to do everything you need to do at the low level.

You also asked about dimming the game graphics under your overlay. This is just a simple DrawPrimitive call to draw a filled transparent rectangle over the screen.

Idona

Be careful when doing this with multiplayer games requiring anti cheat software like Punkbuster! Injecting custom libraries into running games is considered as intrusive game manipulation and cheating. Programs (like Xfire, Teamspeak Overlay, ...) are white-listed and won't cause cheating violations anymore but it's a learning process done by the Anti-Cheat-Vendor which can be time-consuming where your program can't be effectively used. – Interactive

Answer 2

Created a Gist with full code here

This is a pretty esoteric art, and there are several ways to do this. I prefer what's called a Direct3D Wrapper. It's been years since I've done this, but I did it with a game once. I may have left out some details, but I just hope to illustrate the idea.

All Direct3D9 games need to call IDirect3DDevice9::EndScene after the screen is rendered and ready to be drawn. So in theory, we can put custom code inside EndScene that draws what we want over the game. We do this by creating a class that looks like IDirect3DDevice9 to the game, but it's really just a wrapper that forwards all function calls to the real class. So now, in our custom class, our wrapper function for EndScene looks like this:

HRESULT IDirect3DDevice9::EndScene()
{
   // draw overlays here

   realDevice.EndScene();
}

Then we compile it into a DLL called d3d9.dll, and drop it in the game executable's directory. The real d3d9.dll should be in the /system32 folder, but the game first looks in the current directory, and instead loads ours. Once the game loads, it uses our DLL to create an instance of our wrapper class. And now each time EndScene is called, our code is executed to draw what we want to the screen.

I think you can try the same principle with OpenGL. But to prevent tampering, I think some modern games try to prevent this.

Idona

is there a wrapper that i can use the code in a c# application? thank you – Typewritten

Answer 3

In Hearthstone-Deck-Tracker

It create a tool over Hearthstone.

The base method is SetTopmost

It just like this:

private async void SetTopmost()
{
    if(User32.GetHearthstoneWindow() == IntPtr.Zero)
    {
        Log.Info("Hearthstone window not found");
        return;
    }

    for(var i = 0; i < 20; i++)
    {
        var isTopmost = User32.IsTopmost(new WindowInteropHelper(this).Handle);
        if(isTopmost)
        {
            Log.Info($"Overlay is topmost after {i + 1} tries.");
            return;
        }

        Topmost = false;
        Topmost = true;
        await Task.Delay(250);
    }

    Log.Info("Could not set overlay as topmost");
}

Diffidence

Answer 4

I've been thinking about this. For OpenGL, I'd hook WGL's SwapBuffers with Detours, drawing my stuff after the game and then swapping buffers myself.

Astronautics

Answer 5

I've done overlays with both DirectX & WPF viewports (same thing, really), using the overlay to paint some nice 2D GDI+ stuff on top of 3D visualisations.

I managed by using a panel on top of the "actual" visualisation, setting the colour to transparent except for the bits I wanted overlaid. Worked pretty well, but it was a bit of a pain passing click, drag, and other events through the "drawing" layer to the 3D control.

Outlying

I'm sorry if my question wasn't clear: The "actual" visualisation is drawn by a 3rd party app, in my case. I've done what you said, once, in the context of a single application. I don't know if your answer is relevant in the context of two different applications; if it is, how would I go about creating a child panel in the context of the fullscreen parent? – Toastmaster

@Outlying - can you elaborate how this technique could be used between a first part and 3rd party app? – Vellum