What happens when I double click an executable, technically

Question

tl;dr

I'm trying to understand the difference of running a program directly via double clicking the executable vs running it through a terminal or programatically via CreateProcess in windows 10.

---

long version

Because this is my problem, executing an old game (circa 2003 using D3D8) through double clicking in windows 10 works okay. Executing the game through an seconday executable (also circa 2003) using CreateProcess seems to sometimes work okay.

But executing it through my new golang executable never works. I get a very tiny screen instead. So I want to understand what's the technical difference.

For reference, my golang code goes like: (tiny screen)

cmd := exec.Command(filepath.Join(".", "Game.exe"))
err := cmd.Start()

Disassembling the secondary executable gives me this: (normal screen)

CPU Disasm
Address   Hex dump          Command                                  Comments
004043AF  |.  51            PUSH ECX                                 ; /pProcessInformation = 59C7F521 -> {hProcess=???,hThread=???,ProcessID=???,ThreadID=???}
004043B0  |.  52            PUSH EDX                                 ; |pStartupInfo => OFFSET LOCAL.16
004043B1  |.  68 28E54000   PUSH OFFSET 0040E528                     ; |CurrentDirectory = "."
004043B6  |.  50            PUSH EAX                                 ; |pEnvironment => NULL
004043B7  |.  50            PUSH EAX                                 ; |CreationFlags => 0
004043B8  |.  6A 01         PUSH 1                                   ; |InheritHandles = TRUE
004043BA  |.  50            PUSH EAX                                 ; |pThreadSecurity => NULL
004043BB  |.  50            PUSH EAX                                 ; |pProcessSecurity => NULL
004043BC  |.  68 10E54000   PUSH OFFSET 0040E510                     ; |CommandLine = "Game.exe"
004043C1  |.  50            PUSH EAX                                 ; |ApplicationName => NULL
004043C2  |.  FF15 8CB04000 CALL DWORD PTR DS:[<&KERNEL32.CreateProc ; \KERNEL32.CreateProcessA

When I say the game is a tiny screen it shows up like this:

versus this when it is executed directly with double click: (don't mind it being black its just the normal startup)

---

Additional info: Actually the problem only exists in windows 10. Windows 7 is completely fine.

For windows 10, the only way to make it normal screen is to use this setting:

When that is used, I get the normal screen when using double click or the secondary executable. But its still a tiny screen on my Golang app.

Gleanings

Beyond the programs environment and current working directory potentially being different, there should be no difference between running a program normally by typing its name in a shell or double-clicking it in some GUI environment. – Rawlins

Maybe you need to set the dwXSize and dwYSize members of a STARTUPINFO structure? – Bioclimatology

I wonder whether it could be possible to use ProcMon to peer into how CreateProcess[Ex] gets called in both the cases. – Membranous

Not sure I've read the problem description correctly. In Windows 10, before applying the compatibility settings, the program behaved the same (small window) for all three methods of launching (the old 2003 launcher, the new golang launcher, and Explorer)? Is that right? Then setting compatibility changed Explorer and the old (32-bit?) launcher, but not golang? – Epistrophe

@Epistrophe exactly. But I've figured it out. Adding the compatibility setting to the golang launcher itself instead of the game fixed the issue for the launcher but its again small window in double click. Having the compatibility setting on both the launcher and the exe makes it work ok. – Gleanings

@Gleanings Ah, so the issue seems to be that golang disregards its target's compatibility settings. It seems like you're in a position to answer your own question. – Epistrophe

When the shim engine is initialized at process startup, it sets the current compatibility layers in the environment variable __COMPAT_LAYER (e.g. to the value "16BitColor"). This allows propagating the compatibility layer to child processes that inherit the parent's environment variables. You can of course manually add the __COMPAT_LAYER variable. – Brummell

I don't know why CreateProcessW isn't writing this shim data in your case when it's set in the registry key "[HKLM|HKCU]\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers". In my test with a debugger and procmon, I can see it query the registry and write the data to the PEB pShimData in the child process. (It's kind of hacky that they implement this by having the parent process write directly to the PEB of the child. By now they should have migrated this to use a creation attribute that's passed to NtCreateUserProcess.) – Brummell

i just opened an app in 3 diff methods and ran procdump, they all checked AppCompatFlags correctly. Double clicking the exe, calling CreateProcess() on it in a C++ app, and using "start app.exe" on the command line. All 3 queried AppCompatFlags. I did not test with GoLang, you should try ProcMon on your golang app and see what happens – Karelian

Answer 1

The Windows-API function ShellExecute with "open" as second argument opens a file as if double-clicked. Example:

#include <windows.h>

int main(int, char**)
{
        ShellExecute(NULL, "open", "C:/Windows/System32/notepad.exe", NULL, NULL, SW_SHOWNORMAL);
        return 0;
}

See https://learn.microsoft.com/en-us/windows/win32/api/shellapi/nf-shellapi-shellexecutew

Reiterate

Answer 2

What actually happens when you double-click a file is a red herring but since you specifically asked about that I will answer (TLDR CreateProcess).

When you double-click a file in Explorer, it will

1. Use its IShellFolder instance and PIDL representing the file to call IShellFolder::GetUIObjectOf to get an IContextMenu instance.
2. Call IContextMenu::InvokeCommand on the default item ("Open" in your case).
3. IContextMenu::InvokeCommand will effectively call ShellExecuteEx (without SEE_MASK_INVOKEIDLIST set).
4. ShellExecuteEx will end up calling CreateProcess for files ending in .exe.

Any application can replicate this by calling ShellExecuteEx with the SEE_MASK_INVOKEIDLIST flag. It will then do the IContextMenu dance for you. For .exe files, this is unnecessary and the simpler ShellExecute can be used:

ShellExecuteW(NULL, L"open", L"c:\folder\file.exe", L"/switches /if /you /want", L"c:\folder", SW_SHOW);

The one exception is if the exe file needs UAC elevation (requested in its manifest or you set it in the compatibility tab). Then ShellExecute[Ex] will ask a Windows service to elevate on its behalf instead of calling CreateProcess. The service will call CreateProcessAsUser which is effectively CreateProcess with a different user token.

• When using cmd.exe, executing foo.exe will call CreateProcess and start foo.exe will call ShellExecuteEx.

• exec.go calls os.StartProcess which in turn calls CreateProcess.

---

Without knowing more about the game, it is hard to say if the problem is a bug in the game or a change in Direct 3D.

If you want to force compatibility on for the child process launched by Go, you can append the compatibility environment variable:

cmd := exec.Command(filepath.Join(".", "Game.exe"))
cmd.Env = append(os.Environ(), "__COMPAT_LAYER=16BitColor")
err := cmd.Start()

Also, using "." (the current directory) here is less than ideal. If your Go application is in the same directory as "Game.exe" then you should use the absolute path of your Go application (minus the file name) to join with "Game.exe". You should also set cmd.Dir to this directory.

You could also debug the game process and see which size it passes to USER32::CreateWindowExW.

Dinodinoflagellate