win32 ShellExecute drops messages from the queue

Question

Calling ShellExecute seems to remove messages from the Message Queue. The source for a C program to test for this behavior is available below. Note that I'm using ShellExecute to open URLs in the user's default browser (appears to be the recommended API for doing this).

Questions I have:

• Is this a bug?
• If it is a bug, how many versions of Windows does it affect?
• If it is a bug, how do I report it to Microsoft?

On one hand, this seems unlikely to be a bug since it would probably affect enough programs to have been fixed (if it's happening on enough versions of windows). On the other hand, dropping messages would only cause problems some of the time (i.e. no issues if the queue is empty or only has unimportant messages at the time). I also can't find any documentation that mentions this behavior or that there is any issues with calling ShellExecute on a thread that has a message queue. Also, ShellExecute likely isn't called often making this manifest even more intermittently.

If you're able to compile/run the program yourself on another version of Windows (mine is 10.0.19045) and report the results that would be helpful. You'll see and "Error: " message at the end if it reproduced the issue.

I've also tried submitting a bug to Microsoft via their "Feedback Hub" in the "Developer Platform" > "API Feedback" category but if anyone knows of a more appropriate place to submit this potential bug let me know.

// Compile from Visual Studio Command Prompt with:
//
//     cl main.c
//
#pragma comment(lib, "user32.lib")
#pragma comment(lib, "shell32.lib")

#define UNICODE
#include <windows.h>
#include <stdio.h>

static unsigned DropMessages()
{
    unsigned count = 0;
    MSG msg;
    while (PeekMessage(&msg, NULL, 0, 0, PM_REMOVE)) {
        printf("    droping message %d\n", msg.message);
        count++;
    }
    if (count == 0) {
        printf("    empty (no messages were dropped)\n");
    }
    return count;
}

// NOTE: using wWinMain doesn't help
//int WINAPI wWinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, WCHAR *pCmdLine, int nCmdShow)
int main()
{
    //
    // Show that the message queue starts out empty
    //
    printf("dropping messages, it should start out empty:\n");
    {
        unsigned dropped = DropMessages();
        if (dropped != 0) abort();
    }

    //
    // Post a message to the queue and show that it's in there
    //
    printf("posting message\n");
    if (!PostThreadMessage(GetCurrentThreadId(), WM_USER, 0, 0)) {
        printf("PostThreadMessage failed, error={}\n", GetLastError());
        return -1;
    }
    printf("dropping messages, it should drop 1:\n");
    {
        unsigned dropped = DropMessages();
        if (dropped != 1) abort();
    }
    
    //
    // Post a message to the queue, call ShellExecute which clears the queue!?!
    //
    printf("posting message\n");
    if (!PostThreadMessage(GetCurrentThreadId(), WM_USER, 0, 0)) {
        printf("PostThreadMessage failed, error={}\n", GetLastError());
        return -1;
    }

    static const int enable_bug = 1;
    if (enable_bug) {
        printf("calling ShellExecute, this clears the message queue!?!\n");
        INT_PTR result = (INT_PTR)ShellExecute(NULL, NULL, NULL, NULL, NULL, 0);
        if (result <= 32) {
            printf("ShellExecute failed, result={}, error={}\n", result, GetLastError());
            return -1;
        }
    }
    
    printf("dropping messages, you'd think it should have 1 but:\n");
    {
        unsigned dropped = DropMessages();
        if (dropped == 1) {
            printf("Success!\n");
        } else {
            printf("Error: expected to drop exactly 1 message but dropped %d\n", dropped);
        }

    }
    return 0;
}

Francklin

ShellExecute will invariably instantiate COM objects on the calling thread. Since this is assumed to be a STA thread it necessarily will have to dispatch messages. Since you are no longer in control of the thread, you can miss thread messages. This isn't specific to ShellExecute. Launching a modal dialog will exhibit identical behavior. This is by design and not a bug. If your requirement is to never miss thread messages, you cannot give up control over your thread's message loop. – Muriate

Wow that article was the ticket, thanks so much for the answer! – Francklin

Indeed. Because ShellExecute can delegate execution to Shell extensions (data sources, context menu handlers, verb implementations) that are activated using Component Object Model (COM), COM should be initialized before ShellExecute is called. Some Shell extensions require the COM single-threaded apartment (STA) type. When accessing objects that use the apartment threading model, COM will synchronize access to the object. In order for this synchronization to occur, COM must marshal calls to the object. – Madian

Thanks again for the help. My takeaways here are basically to stay away from any dependence on thread-only messages. I've fixed my issue by creating a "message only" window to handle the messages I was previously handling on my message loop. Another takeaway, if your message loop doesn't look like the default Get/Translate/Dispatch, then you're probably gonna have a bad time because any modal API won't know about any customization in your outer message loop. – Francklin

Answer 1

As IInspectable said, ShellExecute will invariably instantiate COM objects on the calling thread. Losing messages becasue you can not control thread any more.

The documnetation explains it: Because ShellExecute can delegate execution to Shell extensions (data sources, context menu handlers, verb implementations) that are activated using Component Object Model (COM), COM should be initialized before ShellExecute is called. Some Shell extensions require the COM single-threaded apartment (STA) type.

And it isn't specific to ShellExecute. Launching a modal dialog will exhibit identical behavior.

So losing meesages is by design.

Madian