Stepping into a P/Invoke call in disassemby view

Question

My C# code is calling an unmanaged third-party library function via P/Invoke, and the unmanaged function is having some strange side effects. I want to debug into it and see what it's doing.

If I debug my C# code, and try to "Step Into" the P/Invoke call, it steps over instead. No surprise there -- I expected that; it doesn't have the source for this DLL, and I didn't tell it I was okay with seeing the disassembly view.

So I switch the debugger to disassembly view (Debug > Windows > Disassembly). Now I see the individual x86 instructions in my JITted code. Again I try to step into the P/Invoke call. Again, it steps over instead -- even though I clearly told it to Step Into an x86 CALL instruction. How hard is it to step into an x86 CALL?

My Googling thus far has shown me a couple of options that can affect this, and I've already set them:

• In Tools > Options > Debugging > General, "Enable Just My Code" is unchecked.
• In Project > Properties > Debug tab, "Enable unmanaged code debugging" is checked.

No good. Visual Studio still refuses to step in.

I don't have a PDB for the third-party DLL, but that shouldn't matter. I don't care about source code or symbol information. (Well, actually they'd be really nice, but I already know I'm not going to get them.) Visual Studio can do x86 debugging (that's what the Disassembly view is for), and all I want to do is step into the x86 code.

What else do I need to do to get VS to allow me to step into the x86 instructions inside a P/Invoke call?

Neogene

It's been a few years since I tried to do this, but as I recall, no, I never did get it to work. – Neogene

Answer 1

This may help you solve the problem: (by Graviton)

CallingConvention = CallingConvention.Cdecl

Also this mentions that you need to detach managed debugger and re-attach unmanaged when crossing the boundaries. You might need to check the capabilities of mixed debugger and it's preferences from MSDN.

And finally, using Ed Dore' answer:

Under Tools.Options dialog, select the Debugging category, and make sure the "Enable Just My Code" setting is unchecked. From the Project properties, select the Debug tab, and then ensure that "Enable unmanaged code debugging" is checked.

Once you've got these squared away, you should get the mixed mode debugging support working.

Also, if you use "Debug.Attach To Process" , be sure to hit the "Select..." button in the "Attach To Process" dialog, and select both Managed and Native debugging support.

Pablo

None of these help step into calls to framework methods in VS2017/Win10 :( – Dutton

Answer 2

One thing I would try is going from C# to C++/CLI code, and then from C++ to the third-party code. Once you're in C++ (and free of the P/Invoke infrastructure), you might have better luck with the disassembly view.

Discontinuous

I know pretty much nothing about writing C++/CLI code... got any links that could help get me started with this strategy? – Neogene

Answer 3

In your C# project properties, in the Debug tab, check Enable native code debugging. Worked for me in VS 2012.

Credit goes to billb.

Also, since it's a third party library, ensure Enable Just My Code is unchecked in Options > Debugging.

Heliotherapy

Answer 4

I had a similar issue where I was debugging a C# exe that called my own C++ dll via PInvoke, all part of the same solution. Enabling native code debugging in my c# project allowed me to debug my C++ code.

Flutterboard

Answer 5

Had same issue today. What helped in my case was to run Visual Studio as administrator.

My setup:

• Visual Studio 2019
• Windows 10

Oster