How do I debug a process that starts at boot time?

Question

I am trying to set a breakpoint into a Windows service that starts at boot time. Because of an unfortunate mistake on my end, the service forces the machine into a reboot loop: this means that I can't get to a stable state from which I could deploy a fix, and obviously I can't try to debug the service at a more convenient time.

I can use windbg in kernel mode. I'd very much like to break when the service hits the wmain function, but I'm having issues with that.

Up to now, I found that I can stop when the image is loaded by using the following commands:

!gflag +ksl
sxe ld MyServiceExecutable.exe

The problem is that once it breaks, I find myself in an empty process, in which I am apparently unable to set breakpoints. bm MyServiceExecutable!wmain says that it can't find the symbol and that the breakpoint will be "deferred", but it is effectively never set or reached. Setting a breakpoint on KERNEL32!BaseThreadInitThunk seems to work more or less at random across all the processes running and I didn't have a lot of luck with it to stop in my service so far.

Orthman

Cant you simply start the process from a command line and debug there (if you design it to work that way). A blocked boot normally means that ServiceInit never returns which should be easy to debug. – Spindle

@AloisKraus, in this specific instance I knew exactly what the bug was and how to fix it, the problem was that the machine just couldn't get into a state where I could deploy a fix. Basically, my options were "windbg your way to a functional state" or "wipe the machine". – Orthman

If the boot hangs cannot you force a blue screen and debug the created memory dump? msdn.microsoft.com/en-us/library/windows/hardware/… – Spindle

@AloisKraus, pausing the system with WinDbg in kernel mode is the easiest thing, you just hit Ctrl+Break and you're done. I wasn't trying to diagnose the issue, I was trying to get past it without wiping the machine. – Orthman

The other solution to this specific problem is to boot the machine to external media (e.g., WinPE or the Windows Setup recovery options) and disable the driver by modifying the registry. (Note that regedit has an option to load an arbitrary registry hive; make sure you unload the hive after making the changes, or they might not stick.) – Northcliffe

!gflag not !gflags apparently. – Cannabis

Answer 1

Alright, this might not the best way to do it, but it worked. MSFTs, please correct me if I'm doing something dumb!

The first part was good:

kd> !gflag +ksl
    New NtGlobalFlag contents: 0x00440000
        ksl - Enable loading of kernel debugger symbols
        ece - Enable close exception
kd> sxe ld MyServiceExecutable.exe
kd> g

In kernel mode, sxe ld will stop the first time the executable is loaded only.

When the debugger stops again, we're inside the freshly created process. We don't need the gflag anymore:

kd> !gflag -ksl
    New NtGlobalFlag contents: 0x00400000
        ece - Enable close exception

Though we're going to need the EPROCESS pointer. We can get it with .process or !process -1 0, but it is already in the $proc pseudo-register:

kd> r $proc
    $proc=0011223344556677
kd> .process
    Implicit process is now 00112233`44556677

From this point it's possible to set breakpoints on nt symbols, so let's use NtMapViewOfSection as it's called for each dll loaded.

kd> bp /p @$proc nt!NtMapViewOfSection
kd> g

On the next stop ntdll should be loaded (check with kn if it's on the stack, .reload /user if necessary), so you can set a breakpoint on RtlUserThreadStart. Also, we are overwriting breakpoint 0, because since we don't need to break on NtMapViewOfSection anymore(it would just be a nuisance).

kd> bp0 /p @$proc ntdll!RtlUserThreadStart
kd> g

All symbols should have been loaded by the time the first user thread starts, so you're free to set your breakpoint wherever you want.

kd> .reload /user
kd> bp /p @$proc MyServiceExecutable!wmain
kd> g

Orthman

You can set unresolved breakpoints using bu this will allow you to set breakpoints on modules yet to be loaded, you can put all of the above in a script which would streamline things a bit – Iniquitous

@EdChum, from what I read, bm breakpoints behave the same as bu, and I didn't have much luck with those--could it be because I was missing the /p in my first attempts? That said, now that I fixed my problem, you'll understand that I'm in no hurry to try again :) – Orthman

That is only true if the symbol can be matched, if the module has not been loaded then no symbols are matched, bm allows you to set multiple breakpoints on multiple matches, I can understand not wanting to go through this again! – Iniquitous

This worked for me! The executable shows up by running !process -1 0, but it is not shown in !process 0 0. Is -1 referring to some sort of most recent loaded executable? – Kelter

Answer 2

Use the technique that MS describes for debugging winlogon which involves using the kernel mode and user mode debuggers in tandem. See "Debugging WinLogon" in the debugger.chm file that comes with the "Debugging Tools for Windows" download.

Andros

"To attach a debugger to WinLogon, you must go through the registry so the process is debugged from the time it starts up," which I couldn't do. – Orthman

@zneak, you can set the reg key before rebooting machine, and the process will break into debugger when it starts up after reboot. It should work for you and it is much more powerful way to debug user mode processes this way, since you will be essentially in WinDbg, with all normal user mode extensions working as expected. In your approach with plain KD, it is more difficult to work, especially if some of the process memory is paged out. – Osteomyelitis

@SevaTitov, here's the deal: the service prevented the machine from booting. It was stuck in a reboot loop. This is what I meant by "the service blocks the boot process, so I can't debug it at a more convenient time, and I can't modify it" in my question. – Orthman

@zneak, can you boot into safe mode (F8 before windows logo appears), modify the registry, then boot normally? – Osteomyelitis

No need to even do that, you could also just boot into any Windows install/recovery media (I think there are some Linux utilities too) and offline edit the registry. – Chunky