Yes, you can definitely symbolicate kernel panic traces, but to do it retroactively, you need more information from the panic log than just the raw stack trace. As you say, the addresses only make sense relative to the load addresses.
Apple's official documentation on the topic, TN2063 is a little outdated. The example it gives is from Darwin 9, i.e. OS X 10.5, and things have changed a bit since then, with the introduction of Kernel ASLR and kext UUIDs. I'll try to give you a very quick up to date guide.
1. The Easy Way
If your panic is reproducible, the easiest thing to do is to get the kernel to symbolicate it for you. Using the kernel boot argument keepsyms=1
means the kernel won't discard whatever symbols are stored in the kernel and kext images, and will look up the return pointers in the stack trace in case of a panic.
Just add keepsyms=1
to either the Kernel Flags setting in /Library/Preferences/SystemConfiguration/com.apple.Boot.plist
, or to the boot-args
NVRAM variable. Reboot, and any subsequent panics will be automatically symbolicated. You can run mangled C++ symbols through the c++filt
command line utility to get the proper C++ function signatures. For example,
$ echo __ZN32IOPCIMessagedInterruptController17registerInterruptEP9IOServiceiPvPFvS2_S2_S2_iES2_ | c++filt
IOPCIMessagedInterruptController::registerInterrupt(IOService*, int, void*, void (*)(void*, void*, void*, int), void*)
2. The Manual Way
If you've got an unsymbolicated, mysterious panic you can't seem to reproduce, the easy way isn't much help.
Immediately following the stack trace, look for a section starting with "Kernel Extensions in backtrace:" in the panic log. This will list any kexts involved in the panic, including their load addresses, versions and UUIDs. The addresses are given as a range; the start address is to the left of the ->
and following the @
. The last address is to the right of the arrow. With this information you should be able to identify the kext in which each code address (the hex number on the right) listed in the stack trace is located.
Except some of them won't match any kext. Except in some odd circumstances, these will be from the kernel itself. The kernel image (kernel or mach_kernel) load address is further down, where it says "Kernel text base:"
Once you know which executable image to look in, the atos
command will let you symbolicate each address.
For example, let's say we have this line in a panic:
…
0xffffff8098c1bba0 : 0xffffff7f80c343f2
…
And we also find that:
Kernel Extensions in backtrace:
com.apple.iokit.IOPCIFamily(2.9)[BDA92C3B-AD86-33E5-A7F1-1603465350A7]@0xffffff7f80c1a000->0xffffff7f80c4dfff
Notice that 0xffffff7f80c343f2 is greater than (or equal to) 0xffffff7f80c1a000 and less than (or equal to) 0xffffff7f80c4dfff, so the code in question is in IOPCIFamily.
This leads me to the following command (and its output):
$ atos -o /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Extensions/IOPCIFamily.kext/IOPCIFamily -l 0xffffff7f80c1a000 0xffffff7f80c343f2
IOPCIMessagedInterruptController::registerInterrupt(IOService*, int, void*, void (*)(void*, void*, void*, int), void*) (in IOPCIFamily) (IOPCIMessagedInterruptController.cpp:85)
-o
specifies the executable file. This is generally in the Contents/MacOS/ subdirectory of the .kext package, but some of Apple's kexts have it directly inside the .kext directory. For functions in the kernel itself, provide the kernel image, e.g. /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Kernels/kernel
.
The -l
argument specifies the load address. I.e. start/text base.
Finally, just list all the addresses you want to symbolicate in that file. In this case, just one, but you can list multiple. You can also read them from stdin (if none are listed on the command line).
With this, you should be able to decode your whole trace.
A note on UUIDs
You'll notice that each kext in the trace, and the kernel itself, have a UUID listed. This is handy for making sure you're using the right version for symbolicating. This is the UUID from the LC_UUID loader command in the Mach-O binary file. You can check the UUID for a kext using:
$ otool -l /Library/Developer/KDKs/KDK_10.10.5_14F27.kdk/System/Library/Extensions/IOPCIFamily.kext/IOPCIFamily | grep uuid
uuid BDA92C3B-AD86-33E5-A7F1-1603465350A7
To confirm that the kext being used for symbolication does indeed match the one in the panic. This is great when you're ending up with weird versioning issues, or if you're having trouble with the kext cache.