Are there any compiler options to make x64 release crash dumps more usable?

Question

Whenever I get a crash dump for an x64 release build of my application I find its rare I can view the locals, this makes it hard or impossible to fix some issues. In x86 I can usually view all locals without any issues.

Are there any compiler options in the release build that will allow me to view the locals in release build crash dumps? Obviously I don't want to turn off optimizations but perhaps there is some way to force it to save the locals with minor performance impact?

Hoar

Did you consider switching to some other compiler like GCC or Clang/LLVM? You may compile with both debug info and optimizations (e.g. g++ or clang with -O2 -g) – Terza

Locals tend to get optimised away, so no, you can't "have your cake and eat it", in general. – Gomel

Don't expect the release/optimized build to be debug friendly. x86_64 has much more registers, so the compiler won't have to spill much variables to memory, that may be one of the reason why you didn't see the values. – Joscelin

Switching compiler is not an option, also I fully expect that things will be changed/moved around, but x64 vs x86 release crash dumps are pretty unusable by default.. – Hoar

always generate a debug build and share this debug version to the customer who has the issue and hope the customer gives you now better dumps. – Oira

Thats not really an option, plus most issues go away in debug builds too as it changes the timings of everything – Hoar

If you are using VC++ then use the /Zo option. It will track locals much better. It's not perfect (optimizing necessarily loses some information) but it is better than some comments claim is possible. Details here: randomascii.wordpress.com/2013/09/11/… – Aviation

Answer 1

You've said a couple things that hint at why you can't see locals...

#1 - It's a release build.

With certain optimizations turned on, the compiler is free to do a few things that make looking at locals more difficult.

1. It can inline a function. When this happens, the locals of the function that aren't optimized away are mixed with the calling stack frame.
2. It can free up a register and save a couple clock cycles on the function call using a trick called Frame-Pointer Omission.
3. In order to save stack space, the compiler can reuse the location that held a variable earlier in the function body to hold a different variable later in the function. This means that where you are in the function determines which locals you are actually able to see.

#2 - It's an x64 build.

MSVC uses a new calling convention for 64-bit code aptly called the x64 Calling Convention. The first 4 function arguments are stored in registers instead of on the stack. This means that even though you are looking at a stack frame, you will not see some of the arguments and you may not even be able to recover them if the registers have been reused for something else by the time you look at them.

---

So, now what?

Now that we know why you are going to have such a difficult time, let's see what you can do to get around the issues above. None of those issues are really show stoppers, but they all work together to make things just that much more difficult for you.

1. Turn off some optimizations. You can try making with a release build with optimizations at a level that doesn't impede debugging quite so much. You would probably want to start with the optimizations mentioned above that play with stack frames (/Oy and /Ob). Then you would need to hope that you can still reproduce the issue with those optimizations turned off. Also, depending on your internal policy and the contract that you have with your customer, you may need to involve a lawyer before sending an unofficial build to the customer -- probably not the most fun thing in the world.

2. Build a better symbol file. VS2012 and higher has a new compiler switch, /d2Zi+ in VS2012 and /Zo in VS2013, that generates better debug info when optimizations are turned on. This puts debugging optimized code on par with GCC/Clang. Even though it is undocumented in VS2012, I'd still consider it pretty safe since I saw no difference in the generated code -- only in the symbol file. You may even be able to rebuild locally with this flag and force windbg to use your new symbol file via .symopt+ 0x40. This gives you a chance to get more out of the dumps you already have.

3. Use windbg extensions to do the heavy lifting. In other StackOverflow answers, I've mentioned a tool called CMKD that has saved my bacon a couple times. It, among other things, attempts to reconstruct the arguments in the x64 calling convention that were passed in registers. It's not a sure thing, but it is probably the best hope of getting them back.

Anyway, I hope my ramblings will prove helpful in your debugging.

Overskirt

CMKD seems to be doing the job :) I'll try this /Zo option too – Hoar

You should definitely add /Zo to get better tracking of locals. It works especially well when debugging with windbg (it shows inlining), but it also works with VC++. There are a couple of provisos -- my blog post covers them: randomascii.wordpress.com/2013/09/11/… – Aviation