Debug python dump file in windbg

Question

one of my python.exe (v3.5.2) process hang in remote Windows Server. I cannot attach it from remote site so I create a full memory dump and download it back to analyze.

however, windbg doesn't have the tool like py-bt in gdb. what I can do is downloading cpython symbol files and loading into windbg.

the question is: how can I print the python script call stack just like "py-bt" do?

the call stack is as follow, looks like hang on HTTPS connection

0:000> k
ChildEBP RetAddr  
0050dd24 74592242 ntdll!ZwWaitForSingleObject+0xc
0050dd6c 7459bdee mswsock!SockWaitForSingleObject+0x143
0050ddf8 777561e0 mswsock!WSPRecv+0x391
0050de48 614d8e76 ws2_32!recv+0x103
0050de68 614138bb _ssl!sock_read+0x26 [c:\build\cpython\externals\openssl-1.0.2h\crypto\bio\bss_sock.c @ 141]
0050de90 614bc229 _ssl!BIO_read+0x5b [c:\build\cpython\externals\openssl-1.0.2h\crypto\bio\bio_lib.c @ 210]
0050debc 614bc36e _ssl!ssl3_read_n+0x1c9 [c:\build\cpython\externals\openssl-1.0.2h\ssl\s3_pkt.c @ 255]
0050df78 614bd136 _ssl!ssl3_get_record+0x9e [c:\build\cpython\externals\openssl-1.0.2h\ssl\s3_pkt.c @ 339]
0050dfb8 614c0bb5 _ssl!ssl3_read_bytes+0x1a6 [c:\build\cpython\externals\openssl-1.0.2h\ssl\s3_pkt.c @ 1228]
0050dfdc 614c0c13 _ssl!ssl3_read_internal+0x45 [c:\build\cpython\externals\openssl-1.0.2h\ssl\s3_lib.c @ 4459]
0050dfec 61402880 _ssl!ssl3_read+0x13 [c:\build\cpython\externals\openssl-1.0.2h\ssl\s3_lib.c @ 4483]
0050e034 613ff8f7 _ssl!_ssl__SSLSocket_read_impl+0x1f0 [c:\build\cpython\modules\_ssl.c @ 1944]
0050e080 61b12ae3 _ssl!_ssl__SSLSocket_read+0x67 [c:\build\cpython\modules\clinic\_ssl.c.h @ 267]
0050e09c 61b7409f python35!PyCFunction_Call+0x113 [c:\build\cpython\objects\methodobject.c @ 109]
0050e0d0 61b712a5 python35!call_function+0x2ff [c:\build\cpython\python\ceval.c @ 4705]
0050e148 61b7301f python35!PyEval_EvalFrameEx+0x20a5 [c:\build\cpython\python\ceval.c @ 3239]
0050e194 61b74259 python35!_PyEval_EvalCodeWithName+0x75f [c:\build\cpython\python\ceval.c @ 4018]
0050e1dc 61b740f3 python35!fast_function+0x109 [c:\build\cpython\python\ceval.c @ 4813]
0050e210 61b712a5 python35!call_function+0x353 [c:\build\cpython\python\ceval.c @ 4730]
0050e288 61b7301f python35!PyEval_EvalFrameEx+0x20a5 [c:\build\cpython\python\ceval.c @ 3239]

Lowrance

Your best bet is going be using gdbinit as an example for how to get frame data. Start by looking at the stack frame python35!PyEval_EvalFrameEx+0x20a5. There should be a variable named co on the stack. From there, you can see co_name, co_filename which are compact PyASCIIObjects representing the module and filename. If you can figure out how v3.5.2 represents it, co_lnotab should let you figure out the line number. – Milli

For me, ?? (char*)((PyASCIIObject*)(co->co_name)+1) and ?? (char*)((PyASCIIObject*)(co->co_filename)+1) print the module and file, since compact PyASCIIObjects store their payload in the same allocation, but after the struct itself. Hopefully that's enough to at least give you an idea where in the python code things are hung. – Milli

Answer 1

I was surprised that nothing like CPython's gdbinit existed for WinDbg so I decided to write a debugger extension. I've uploaded the code to the PyExt repository on GitHub.

---

To install it:

• Grab the latest zip from the releases page
• Unzip pyext.dll into WinDbg's winext directory. (Make sure to use the x86 or x64 version that corresponds to the build of WinDbg you're using.)
• Open a dump file in WinDbg and load the extension with: .load pyext
• Add the Python symbol server to the symbol path with: .sympath+ srv*c:\symbols*http://pythonsymbols.sdcline.com/symbols/
• Use !pystack to see the Python backtrace at the time the dump was captured.

---

Example:

0:000> .load pyext
0:000> .sympath+ srv*c:\symbols*http://pythonsymbols.sdcline.com/symbols/
Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://pythonsymbols.sdcline.com/symbols/

************* Symbol Path validation summary **************
Response  Time (ms)     Location
Deferred                srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Deferred                srv*c:\symbols*http://pythonsymbols.sdcline.com/symbols/

0:000> ~*e!pystack
Thread 0:
    File "C:\python33\lib\threading.py", line 284, in wait
        [Globals] 
    File "C:\python33\lib\threading.py", line 1028, in join
        [Globals] 
    File "scripts\win32debug.py", line 153, in _launch_and_wait
        [Globals] 
    File "scripts\win32debug.py", line 180, in dump_process
        [Globals] 
    File "object_types.py", line 33, in <module>
        [Locals] [Globals] 

---

If you run into any problems or would find a new feature to be helpful, feel free to open an GitHub issue and let me know.

Milli

Thanks :-] it saves lots of my time to find out python script stack – Lowrance

That is great! Thanks a lot. – Faultfinding

Note: if .load pyext fails with The call to LoadLibrary(pyext) failed, Win32 error 0n126 "The specified module could not be found." despite pyext.dll being in the correct place, then you probably need to install Microsoft Visual C++ Redistributable for Visual Studio 2017 (or possibly 2015?) from support.microsoft.com/en-gb/help/2977003 because pyext.dll depends on MSVCP140.dll and VCRUNTIME140.dll – Blackmail

It works with IDAPython too (attach to IDA Pro with WinDbg). Thanks! – Partitive