What actions do I need to take to get a crash dump in ALL error scenarios?

Question

We're on Windows and we want to get a crash dump (possibly using MiniDumpWriteDump) for all scenarios where our application exit's unexpectedly.

So far we have identified, and set up, the following:

• SetUnhandledExceptionFilter for unhandled exception (Win32 as well as "normal" C++ ones.)
• _set_invalid_parameter_handler for the CRT invalid argument handling
• _set_abort_behaviorplus a SIGABRT handler to account for calls to abort()

Is there anything we missed? (Modulo some code non-legitimately calling ExitProcess, TerminateProcess or one of the exit variants.)

---

I'll note that this question here is orthogonal to how a crash dump is then obtained. E.g., if you want a crash dump in case of abort, you always must use _set_abort_behaviour because otherwise abort just exits.

I'll also note that on Windows7+, not setting SetUHEF and just setting up the "correct" WER dump settings in the registry is often a viable way.

Lesko

Have you thought about using something like crashrpt: It's oriented around generating a stack traceback (text) rather than a minidump, though. code.google.com/p/crashrpt – Antitoxin

Call stack file (text) can also be generated using StackWalk64 from within the global exception-handler. – Greenes

I recall a Microsoft claim that trying to do it in-process is dangerous. If the app hits an exception, you cannot trust the state of the app. While in many, many cases you'll get away with it (as lots of popular commercial software does), the only way to catch every case is to have a watchdog in a separate process. – Meaghan

@AdrianMcCarthy - Yes, in-process is suboptimal but works most of the time. I'll add a note though. – Lesko

Answer 1

I use exactly the ones you've listed, plus _set_purecall_handler, plus this handy snippet of code:

void EnableCrashingOnCrashes()
{
    typedef BOOL (WINAPI *tGetPolicy)(LPDWORD lpFlags);
    typedef BOOL (WINAPI *tSetPolicy)(DWORD dwFlags);
    static const DWORD EXCEPTION_SWALLOWING = 0x1;

    const HMODULE kernel32 = LoadLibraryA("kernel32.dll");
    const tGetPolicy pGetPolicy = (tGetPolicy)GetProcAddress(kernel32, "GetProcessUserModeExceptionPolicy");
    const tSetPolicy pSetPolicy = (tSetPolicy)GetProcAddress(kernel32, "SetProcessUserModeExceptionPolicy");
    if(pGetPolicy && pSetPolicy)
    {
        DWORD dwFlags;
        if(pGetPolicy(&dwFlags))
        {
            // Turn off the filter
            pSetPolicy(dwFlags & ~EXCEPTION_SWALLOWING);
        }
    }
}

Source: http://randomascii.wordpress.com/2012/07/05/when-even-crashing-doesnt-work/

These other articles on his site also helped me understand this: http://randomascii.wordpress.com/2011/12/07/increased-reliability-through-more-crashes/ http://randomascii.wordpress.com/2012/07/22/more-adventures-in-failing-to-crash-properly/

Watering

No problem :) Shame there isn't a way to handle exit and friends; the best I can come up with is to find instances of them in source and remove them! – Watering

Well, if one can API-hook SetUnhandledExceptionFilter as suggested elsewhere, I suspect that would be possible for ExitProcess and/or TerminateProcess as well. Whether it's a good idea is beyond me, though. – Lesko

Oh, and I guess hooking ExitProcess won't help completely with exit as I think that one does some more before calling ExitProcess. – Lesko

You won't no longer need EnableCrashingOnCrashes after 2014 – Weidman

Answer 2

SetUnhandledExceptionFilter is emphatically not enough to capture all unexpected exits. If an application accidentally calls a pure virtual function then a dialog will pop up. The application will hang, but not crash. Since there is no exception neither SetUnhandledExceptionFilter nor WER can help. There are a few variations on this theme.

Worse yet is the weirdness if you crash in a kernel callback such as a WindowProc. If this happens in a 32-bit application on 64-bit Windows then the exception is caught by the OS and execution continues. Yes, the crash is silently handed. I find that terrifying.

http://randomascii.wordpress.com/2012/07/05/when-even-crashing-doesnt-work/ should detail all of the tricks needed to handle these unusual cases.

Herrah

Answer 3

To expand on all the answers here's what I found to work best for 100M+ installs:

• SetErrorMode to prevent any WER dialogs showing up.
• EnableCrashingOnCrashes
• PreventSetUnhandledExceptionFilter
• SetUnhandledExceptionFilter for system exceptions.
• _set_invalid_parameter_handler for the CRT invalid argument handling
• _set_abort_behavior plus a SIGABRT handler to account for calls to abort()

std::set_terminate and std::set_unexpected perhaps should also be mentioned.

And the most important part to get it all right:

• all these handlers should call a function that executes under a mutex/critical section to ensure that if there are any other crashes happening in other threads at the same time they would all stop and wait instead of causing havoc.
• signal handler for SIGABRT must set itself back as a SIGABRT handler! Without this if you get crashes happening at the same time from other threads you process will exit immediately without giving you any time to handle the crash.
• actual handling of the error should ideally happen in another process, or at least in another thread that was started at the beginning of the process, otherwise you won't be able to handle low memory conditions or stackoverflow errors.

---

See setExceptionHandlers below for reference. Also, most likely you don't want to hook up all the handlers in debug builds or when IsDebuggerPresent.

#include <signal.h>
#include <windows.h>
#include <boost/thread/mutex.hpp>

void EnableCrashingOnCrashes();
void PreventSetUnhandledExceptionFilter();

static void exceptionHandler(EXCEPTION_POINTERS* excpInfo)
{
    // your code to handle the exception. Ideally it should
    // marshal the exception for processing to some other
    // thread and waif for the thread to complete the job
}

static boost::mutex unhandledExceptionMx;
static LONG WINAPI unhandledException(EXCEPTION_POINTERS* excpInfo = NULL)
{
    boost::mutex::scoped_lock lock(unhandledExceptionMx);
    if (!excpInfo == NULL)
    {
        __try // Generate exception to get proper context in dump
        {
            RaiseException(EXCEPTION_BREAKPOINT, 0, 0, NULL);
        }
        __except (exceptionHandler(GetExceptionInformation()), EXCEPTION_EXECUTE_HANDLER)
        {
        }
    }
    else
    {
        exceptionHandler(excpInfo);
    }

    return 0;
}

static void invalidParameter(const wchar_t* expr, const wchar_t* func,
    const wchar_t* file, unsigned int line, uintptr_t reserved)
{
    unhandledException();
}

static void pureVirtualCall()
{
    unhandledException();
}

static void sigAbortHandler(int sig)
{
    // this is required, otherwise if there is another thread
    // simultaneously tries to abort process will be terminated
    signal(SIGABRT, sigAbortHandler);
    unhandledException();
}

static void setExceptionHandlers()
{
    SetErrorMode(SEM_FAILCRITICALERRORS | SEM_NOGPFAULTERRORBOX);
    SetUnhandledExceptionFilter(unhandledException);
    _set_invalid_parameter_handler(invalidParameter);
    _set_purecall_handler(pureVirtualCall);
    signal(SIGABRT, sigAbortHandler);
    _set_abort_behavior(0, 0);
    EnableCrashingOnCrashes();
    PreventSetUnhandledExceptionFilter();
}

Latticework

Shouldn't if (!excpInfo == NULL) be if (excpInfo == NULL)? – Beslobber

Answer 4

Tall order, just in brief:

• You need not to use any other _set* functions, SetUnhandledExceptionFilter is enough for all.
• C runtime functions like abort will disable the global exception handler, which you setup using SetUnhandledExceptionFilter. CRT will simply call this same function will NULL parameter, and your exception-handler is disabled (not called), if CRT is causing crash! What you can do? [X]
• Disable all other running threads when the excption-handler gets called. Just lookup all threads using CreateToolhelp32Snapshot, and other functions. Look for this process, and suspend all other running threads (except current, ofcourse).
• Use SEH or no-SEH, global-exception handler gets called unless CRT interfers. Not to worry (in MOST cases).
• Do not any CLR in-between, it will not allow the exception handler to call, if any CLR/managed call (yes from C/C++) comes in between.
• You cannot handle one exception - Stack Overflow! Think! Running under a debugger is only solution, see below.

There is more to it, which I haven't tried (not found usefulness) - Vectored Exception Handling.

One other approach is to run the application into a debugger, which you can craft yourself! In the debugger, you can catch ALL exceptions, just like VS debugger catches. See my article. But, you know, this is not proper approach.

EDIT: Just read the last content about process-termination. You shouldn't control that. In any case, you can just hook the required APIs, which would act as what you code for (like displaying message box).

[X] You need to use API hooking. I dont have link and details handy. You'd hook other related APIs, but primarily the SetUnhandledExceptionFilter (after you'd called it for you). You dummy (hooked) function will look like:

xxx SetUnhandledExceptionFilter_DUMMY(xxx)
{
  // Dont do any thing
  return NULL;
}

I dont have link and details of API hooking handy.

---

And why not attempt to make your application more safe?

• Correct all warnings (yes, even level 4).
• Use static analysis. VS itself has (in higher versions, though. Except 2012 - all variants have). Other SA tools are available.
• Do careful code-reviewing. It pays!
• Run and Debug your RELEASE build from the debugger. Use all features.
• Look and correct all possible memory leaks.
• Use defensive approach to programming. Rather than checking if null, defend it using ASSERT, or your own assert. Pack it with assertion, log, return from function.

Greenes

"You need not to use any other _set* functions, SetUnhandledExceptionFilter is enough for all." -- I am pretty sure this is wrong. I have checked in the debugger in VS2005: SetUnhandledExceptionFilter(NULL) will only be called by the _invoke_watson function that is called by the default CRT _invalid_parameter function, and this default version will only call _invoke_watson iff no user defined handler is present. So, iff I use _set_invalid_parameter_handler then the CRT will not call SetUExF(NULL). – Lesko

And again for abort() - it will only call SetUnhExcF(NULL) if there is no user defined SIGABRT handler that does something else. – Lesko

Why are you running and testing the same within the debugger? I have tested the code with VS2008 Release build. abort and other CRT functions do call SetUExF to disable any use-defined handler. – Greenes

in the debugger I can look at the code in invarg.c and exactly see, in the code, that SetUExF is only called if no user defined handler is present. (As I said, I test with VS2005, but I really doubt this changed with VS2008.) – Lesko

That's what I am saying. I would not call SetUnhExcF in debug build - it of no use. – Greenes

Who's talking of the debug build? I was just talking of the debugger, and I was actually debugging the release configuration. – Lesko

Yeah! But the process is still running under a debugger, and it matters! Better put a DebugBreak, or use attach-to-process as soon as (or exactly "when") the process would crash. I am not saying you are wrong, or I am absolutely right. But crash must be tested in absolutely no-debugger environment! Your crash handler may get called, and may not get called in debugging/non-debugging environment. – Greenes

Heh. I actually did attach the debugger afterwards. But it doesn't matter. To repeat: Take a look at the source code of invarg.c in your VC folder. You don't even need to run anything to verify that SetUExF(NULL) won't be called when you have registered a handler. – Lesko

Answer 5

I will add a workaround that one can use in certain scenarios when running on Windows 7:

Windows Error Reporting (WER) offers the option to write full memory dumps on app crash.

So, if you are fine with that, you "just" have to make sure that the crash scenarios you're actually interested in will trigger WER. ... Which, really, leads us back to this very question, but still ...

Lesko

Answer 6

You can catch any exception with WER. As you have seen the CRT sometimes forces to call WER.

If you want to always catch excpetion in-process, you need to prevent the calling of SetUnhandledExceptionFilter(NULL) from the CRT. For more info see my blog entry: Improved “PreventSetUnhandledExceptionFilter”

Prohibitionist

That link is certainly appreciated. As I noted above, that WER stuff only works on Win7. And I still have to support WinXP :-/ – Lesko