Login/Register
How is Linux kernel live debugging done and what tools are used?
Asked Answered
Solved debugginglinux-kernelkernel
I

11

61

What are the most common and why are uncommon methods and tools used not to do live debugging on the Linux kernel?

I know that Linus, for example, is against this kind of debugging for the Linux Kernel or it least was and thus nothing much has been done in that sense in those years, but honestly a lot of time has passed since 2000 and I am interested if that mentality has changed regarding the Linux project and what current methods are used to do live debugging on the Linux kernel at the moment (either local or remote)?

References to walkthroughs and tutorials on mentioned techniques and tools are welcome.

Incudes answered 9/2, 2011 at 10:53 Comment(2)
Also see lwn.net/Articles/280912 (search for kgdb) – Vahe
What do you mean by live? The one you are currently running or? Or also one in a VM? – Windsor
L
30

Another option is to use an ICE or JTAG controller, and GDB. This 'hardware' solution is especially used with embedded systems.

But for instance QEMU offers similar features:

  • start QEMU with a GDB 'remote' stub which listens on 'localhost:1234' : qemu -s ...,

  • then with GDB, you open the kernel file vmlinux compiled with debug information (you can take a look a this mailing list thread where they discuss the unoptimization of the kernel).

  • connect GDB and QEMU: target remote localhost:1234

  • see your live kernel:

      (gdb) where
  #0  cpu_v7_do_idle () at arch/arm/mm/proc-v7.S:77
  #1  0xc0029728 in arch_idle () atarm/mach-realview/include/mach/system.h:36
  #2  default_idle () at arm/kernel/process.c:166
  #3  0xc00298a8 in cpu_idle () at arch/arm/kernel/process.c:199
  #4  0xc00089c0 in start_kernel () at init/main.c:713

Unfortunately, user-space debugging is not possible so far with GDB (no task list information, no memory management unit reprogramming to see different process contexts, ...), but if you stay in kernel-space, that's quite convenient.

  • info threads will give you the list and states of the different CPUs

You can get more details about the procedure in this PDF:

Debugging Linux systems using GDB and QEMU.

Labors answered 11/2, 2011 at 8:31 Comment(1)
I tried something similar to this Qemu technique some time ago, it's pretty cool. – Incudes
D
29

While debugging the Linux kernel we can use several tools, for example, debuggers (KDB and KGDB), dumping while crashed (LKCD), tracing toolkit (LTT, LTTV, and LTTng), and custom kernel instruments (dprobes and kprobes). In the following section I tried to summarized most of them.

The LKCD (Linux Kernel Crash Dump) tool allows the Linux system to write the contents of its memory when a crash occurs. These logs can be further analyzed for the root cause of the crash. Resources regarding LKCD:

Oops: when the kernel detects a problem, it prints an Oops message. Such a message is generated by printk statements in the fault handler (arch/*/kernel/traps.c). A dedicated ring buffer in the kernel is being used by the printk statements. Oops contains information like the CPU where the Oops occurred on, contents of CPU registers, number of Oops, description, stack back trace and others. Resources regarding kernel Oops:

Dynamic Probes is one of the popular debugging tool for Linux which developed by IBM. This tool allows the placement of a “probe” at almost any place in the system, in both user and kernel space. The probe consists of some code (written in a specialized, stack-oriented language) that is executed when control hits the given point. Resources regarding Dynamic Probe are listed below:

Linux Trace Toolkit is a kernel patch and a set of related utilities that allow the tracing of events in the kernel. The trace includes timing information and can create a reasonably complete picture of what happened over a given period of time. Resources of LTT, LTT Viewer and LTT Next Generation

MEMWATCH is an open source memory error detection tool. It works by defining MEMWATCH in gcc statement and by adding a header file to our code. Through this we can track memory leaks and memory corruptions. Resources regarding MEMWATCH

ftrace is a good tracing framework for Linux kernel. ftrace traces internal operations of the kernel. This tool included in the Linux kernel in 2.6.27. With its various tracer plugins, ftrace can be targeted at different static tracepoints, such as scheduling events, interrupts, memory-mapped I/O, CPU power state transitions, and operations related to file systems and virtualization. Also, dynamic tracking of kernel function calls is available, optionally restrictable to a subset of functions by using globs, and with the possibility to generate call graphs and provide stack usage. You can find a good tutorial of ftrace at https://events.linuxfoundation.org/slides/2010/linuxcon_japan/linuxcon_jp2010_rostedt.pdf

ltrace is a debugging utility in Linux, used to display the calls a user space application makes to shared libraries. This tool can be used to trace any dynamic library function call. It intercepts and records the dynamic library calls which are called by the executed process and the signals which are received by that process. It can also intercept and print the system calls executed by the program.

KDB is the in-kernel debugger of the Linux kernel. KDB follows simplistic shell-style interface. We can use it to inspect memory, registers, process lists, dmesg, and even set breakpoints to stop in a certain location. Through KDB we can set breakpoints and execute some basic kernel run control (Although KDB is not source level debugger). Several handy resources regarding KDB

KGDB is intended to be used as a source level debugger for the Linux kernel. It is used along with gdb to debug a Linux kernel. Two machines are required for using kgdb. One of these machines is a development machine and the other is the target machine. The kernel to be debugged runs on the target machine. The expectation is that gdb can be used to "break in" to the kernel to inspect memory, variables and look through call stack information similar to the way an application developer would use gdb to debug an application. It is possible to place breakpoints in kernel code and perform some limited execution stepping. Several handy resources regarding KGDB:

Dilantin answered 2/3, 2015 at 12:17 Comment(9)
In HP-UX there was a cool feature named "live dump" that created a kind of coredump of the running kernel without actuall terminating it. So if there was some kind of lock situation support could examine it. AFAIK that does not exist for Linux (yet). – Steffie
The first IBM link is broken: "An unexpected error occured" – Sall
The first Novell link is broken (but not the second): "HTTP 404 Page Not Found" – Sall
The first kernel.org link is broken (but not the second): "404 Not Found" – Sall
The madwifi-project.org link is broken (times out). – Sall
The events.linuxfoundation.org link (tutorial) is broken: "404. Sorry that page wasn't found" – Sall
The www.ellexus.com link is (effectively) broken. It redirects to a generic page. – Sall
The developerblog.redhat.com link is (effectively) broken. It redirects to a generic page. – Sall
The page.dev.man-online.org link is (effectively) broken. It redirects to a generic page. – Sall
M
21

According to the wiki, kgdb was merged into the kernel in 2.6.26 which is within the last few years. kgdb is a remote debugger, so you activate it in your kernel then you attach GDB to it somehow. I say somehow as there seems to be lots of options - see connecting GDB. Given that kgdb is now in the source tree, I'd say going forward this is what you want to be using.

So it looks like Linus gave in. However, I would emphasize his argument - you should know what you're doing and know the system well. This is kernel land. If anything goes wrong, you don't get a segmentation fault; you get anything from some obscure problem later on to the whole system coming down. Here be dragons. Proceed with care; you have been warned.

Makhachkala answered 9/2, 2011 at 11:15 Comment(6)
I stole that one from Mark Shuttleworth (the founder of Canonical). markshuttleworth.com. – Makhachkala
Linus is pretty on the mark in my opinion. The other thing to consider is that some bugs will break kgdb, possibly in subtle ways - can you trust it :) – Hyperaesthesia
@Hyperaesthesia I agree. Tools have bugs too. I think what he's saying is if you can only develop by relying on the tool to tell you when you're wrong, that's a problem - you need to also comprehend the tool and learn to interpret its output against your own knowledge. Blind trust in said tools leads to subtle bugs. – Makhachkala
If you're writing kernel code you really need to be able to understand the code, without a debugger. If you can, then maybe a debugger is a good tool, but it can't replace that fundamental code comprehension. – Hyperaesthesia
No tool can replace fundamental code comprehension, ever. That's my point. If all you do is rely on the tool to tell you when you're wrong, that's a problem because tools don't always tell you the right thing, sometimes because they can't (I'm thinking of valgrind/subtle memory bugs - they tell you there's an error, sure, but where is difficult for them to pinpoint) and sometimes because they have bugs themselves - errors in the kernel could affect kgdb output, as you say. It's structures and memory are no less immune than what you're trying to debug. Hence, here be dragons. – Makhachkala
I don't understand the hostility in the Linux community to kernel debugging. I want to use a kernel debugger to learn about the system as well as to debug problems. If it's OK to use printk() to debug (or to gain code comprehension), then by induction it's OK to use a real debugger. printk()'s are just really inefficient ways to implement breakpoints and variables watches (it's just a debugging technique that requires a full compile/reboot cycle to set a new breakpoint or watch). – Braud
I
16

Another good tool for "live" debugging is kprobes and Dynamic Probes.

This lets you dynamically build little tiny modules which run when certain addresses are executed - sort of like a breakpoint.

The big advantage of them are:

  1. They do not impact the system - i.e., when a location is hit—it just executes the code—it doesn't halt the whole kernel.

  2. You don't need two different systems interconnected (target and debug) like with KGDB

It is best for doing things like hitting a breakpoint, and seeing what data values are, or checking if things have been changed/overwritten, etc. If you want to "step through code" - it doesn't do that.

Another very powerful method is a program simply called "perf" which kind of rolls-up many tools (like dynamic probes) and kind of replaces/deprecates others (like oprofile).

In particular, the perf probe command can be used to easily create/add dynamic probes to the system, after which perf record can sample the system and report information (and backtraces) when the probe is hit for reporting via perf report (or perf script). If you have good debug symbols in the kernel you can get great intelligence out of the system without even taking the kernel down. Do a man perf (in Google or on your system) for more information on this tool or see this great page on it:

perf Examples

Intellectualize answered 9/2, 2011 at 20:58 Comment(1)
Another answer has more details on Dynamic Probes, KGDB, and other tools. – Sall
W
8

KGDB + QEMU step-by-step

KGDB is a kernel subsystem that allows you to step debug the kernel itself from a host GDB.

My QEMU + Buildroot example is a good way to get a taste of it without real hardware: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/1969cd6f8d30dace81d9848c6bacbb8bad9dacd8#kgdb

Pros and cons vs other methods:

  • advantage vs QEMU:
    • you often don't have software emulation for your device as hardware vendors don't like to release accurate software models for their devices
    • real hardware way faster than QEMU
  • advantage vs JTAG: no need for extra JTAG hardware, easier to setup
  • disadvantages vs QEMU and JTAG: less visibility and more intrusive. KGDB relies on the certain parts of the kernel working to be able to communicate with the host. So e.g. it breaks down in panic, you can't view the boot sequence.

The main steps are:

  1. Compile the kernel with:

    CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_INFO=y

CONFIG_CONSOLE_POLL=y
CONFIG_KDB_CONTINUE_CATASTROPHIC=0
CONFIG_KDB_DEFAULT_ENABLE=0x1
CONFIG_KDB_KEYBOARD=y
CONFIG_KGDB=y
CONFIG_KGDB_KDB=y
CONFIG_KGDB_LOW_LEVEL_TRAP=y
CONFIG_KGDB_SERIAL_CONSOLE=y
CONFIG_KGDB_TESTS=y
CONFIG_KGDB_TESTS_ON_BOOT=n
CONFIG_MAGIC_SYSRQ=y
CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
CONFIG_SERIAL_KGDB_NMI=n

    Most of those are not mandatory, but this is what I've tested.

  2. Add to your QEMU command:

    -append 'kgdbwait kgdboc=ttyS0,115200' \
-serial tcp::1234,server,nowait

  3. Run GDB with from the root of the Linux kernel source tree with:

    gdb -ex 'file vmlinux' -ex 'target remote localhost:1234'

  4. In GDB:

    (gdb) c

    and the boot should finish.

  5. In QEMU:

    echo g > /proc/sysrq-trigger

    And GDB should break.

  6. Now we are done, you can use GDB as usual:

    b sys_write
c

Tested in Ubuntu 14.04.

KGDB + Raspberry Pi

The exact same setup as above almost worked on a Raspberry Pi 2, Raspbian Jessie 2016-05-27.

You just have to learn to do the QEMU steps on the Pi, which are easily Googlable:

  • add the configuration options and recompile the kernel as explained at https://www.raspberrypi.org/documentation/linux/kernel/building.md There were unfortunately missing options on the default kernel build, notably no debug symbols, so the recompile is needed.

  • edit cmdline.txt of the boot partition and add:

    kgdbwait kgdboc=ttyAMA0,115200

  • connect gdb to the serial with:

    arm-linux-gnueabihf-gdb -ex 'file vmlinux' -ex 'target remote /dev/ttyUSB0'

    If you are not familiar with the serial, check out this: https://www.youtube.com/watch?v=da5Q7xL_OTo All you need is a cheap adapter like this one. Make sure you can get a shell through the serial to ensure that it is working before trying out KGDB.

  • do:

    echo g | sudo tee /proc/sysrq-trigger

    from inside an SSH session, since the serial is already taken by GDB.

With this setup, I was able to put a breakpoint in sys_write, pause program execution, list source and continue.

However, sometimes when I did next in sys_write GDB just hung and printed this error message several times:

Ignoring packet error, continuing...

so I'm not sure if something is wrong with my setup, or if this is expected because of what some background process is doing in the more complex Raspbian image.

I've also been told to try and disable multiprocessing with the Linux boot options, but I haven't tried it yet.

Windsor answered 28/5, 2017 at 10:41 Comment(4)
can u please pin point in your answer how can I compile linux 5.12.6 kernel with debug flags so I can debug specific kernel module and rest of the kernel with both printk and live debugging with gdb and kgdb – Quash
@Quash I think CONFIG_DEBUG_KERNEL=y enables debug flags. Why not just add all the options I mentioned? Or is the question more generally how to enable certain flags? For that consider: unix.stackexchange.com/a/450407/32558 – Windsor
Should I write all these options in makefile at root of downloaded and extracted linux-5-12-6 directory – Quash
@Quash the commands given at unix.stackexchange.com/questions/224887/… should just work (from inside the linux source tree as you say). – Windsor
W
6

QEMU + GDB step-by-step procedure tested on a Ubuntu 16.10 host

To get started from scratch quickly, I've made a minimal fully automated QEMU + Buildroot example at: https://github.com/cirosantilli/linux-kernel-module-cheat. The major steps are covered below.

First get a root filesystem rootfs.cpio.gz. If you need one, consider:

Then on the Linux kernel:

git checkout v4.9
make mrproper
make x86_64_defconfig
cat <<EOF >.config-fragment
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_KERNEL=y
CONFIG_GDB_SCRIPTS=y
EOF
./scripts/kconfig/merge_config.sh .config .config-fragment
make -j"$(nproc)"
qemu-system-x86_64 -kernel arch/x86/boot/bzImage \
                   -initrd rootfs.cpio.gz -S -s

On another terminal, supposing you want to start debugging from start_kernel:

gdb \
    -ex "add-auto-load-safe-path $(pwd)" \
    -ex "file vmlinux" \
    -ex 'set arch i386:x86-64:intel' \
    -ex 'target remote localhost:1234' \
    -ex 'break start_kernel' \
    -ex 'continue' \
    -ex 'disconnect' \
    -ex 'set arch i386:x86-64' \
    -ex 'target remote localhost:1234'

And we are done!!

For kernel modules, see How to debug Linux kernel modules with QEMU?.

For Ubuntu 14.04 (Trusty Tahr), GDB 7.7.1, hbreak was needed, and break software breakpoints were ignored. It is not the case anymore in Ubuntu 16.10 (Yakkety Yak). See also: https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/901944

The messy disconnect and what come after it are to work around the error:

Remote 'g' packet reply is too long: 000000000000000017d11000008ef4810120008000000000fdfb8b07000000000d352828000000004040010000000000903fe081ffffffff883fe081ffffffff00000000000e0000ffffffffffe0ffffffffffff07ffffffffffffffff9fffff17d11000008ef4810000000000800000fffffffff8ffffffffff0000ffffffff2ddbf481ffffffff4600000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007f0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000801f0000

Related threads:

See also:

Known limitations:

Windsor answered 18/2, 2017 at 14:43 Comment(0)
H
4

Actually, the joke is that Linux has had an in-kernel debugger since 2.2.12, xmon, but only for the PowerPC architecture (actually it was ppc back then).

It's not a source level debugger, and it's almost entirely undocumented, but still.

Hyperaesthesia answered 10/2, 2011 at 12:44 Comment(1)
"kdb" is the x86 equivalent of "xmon". – Intellectualize
H
4

As someone who writes kernel code a lot I have to say I have never used KGDB, and only rarely use kprobes, etc.

It is still often the best approach to throw in some strategic printks. In more recent kernels, trace_printk is a good way to do that without spamming dmesg.

Hyperaesthesia answered 10/2, 2011 at 12:52 Comment(0)
W
3

User mode Linux (UML)

https://en.wikipedia.org/wiki/User-mode_Linux

Another virtualization another method that allows step debugging kernel code.

UML is very ingenious: it is implemented as an ARCH, just like x86, but instead of using low level instructions, it implements the ARCH functions with userland system calls.

The result is that you are able to run Linux kernel code as a userland process on a Linux host!

First make a rootfs and run it as shown at: https://unix.stackexchange.com/questions/73203/how-to-create-rootfs-for-user-mode-linux-on-fedora-18/372207#372207

The um defconfig sets CONFIG_DEBUG_INFO=y by default (yup, it is a development thing), so we are fine.

On guest:

i=0
while true; do echo $i; i=$(($i+1)); done

On host in another shell:

ps aux | grep ./linux
gdb -pid "$pid"

In GDB:

break sys_write
continue
continue

And now you are controlling the count from GDB, and can see source as expected.

Pros:

  • fully contained in the Linux kernel mainline tree
  • more lightweight than QEMU's full system emulation

Cons:

  • very invasive, as it changes how the kernel itself is compiled.

    But the higher level APIs outside of ARCH specifics should remain unchanged.

  • arguably not very active: Is user mode linux (UML) project stopped?

See also: https://unix.stackexchange.com/questions/127829/why-would-someone-want-to-run-usermode-linux-uml

Windsor answered 21/6, 2017 at 7:25 Comment(0)
K
1

You guys are wrong. KGDB still works well for latest kernel. You need to take care of kernel configuration of split image, randomization optimization.

KGDB over serial port is useless, because no computer today supports DB9 on a motherboard serial port, and a USB serial port doesn't support the polling mode.

The new game is kgdboe, and following is the log trace:

The following is the host machine, and vmlinux is from the target machine.

gdb vmlinux

Reading symbols from vmlinux...done.
(gdb) target remote udp:192.168.1.22:31337
1077    kernel/debug/debug_core.c: No such file or directory.
(gdb) l oom_kill_process 
828 mm/oom_kill.c: No such file or directory.
(gdb) l oom_kill_process 
828 in mm/oom_kill.c
(gdb) break oom_kill_process
Breakpoint 1 at 0xffffffff8119e0c0: file mm/oom_kill.c, line 833.
(gdb) c
Continuing.
[New Thread 1779]
[New Thread 1782]
[New Thread 1777]
[New Thread 1778]
[New Thread 1780]
[New Thread 1781]
[Switching to Thread 1779]

Thread 388 hit Breakpoint 1, oom_kill_process (oc=0xffffc90000d93ce8, message=0xffffffff82098fbc "Out of memory")
at mm/oom_kill.c:833
833 in mm/oom_kill.c
(gdb) s
834 in mm/oom_kill.c
(gdb)

On the peer target machine, the following is how to get it to crash and to be captured by the host machine:

#swapoff -a
#stress -m 4 --vm-bytes=500m
Konstanze answered 22/6, 2019 at 5:1 Comment(0)
R
-4

KGDB and GDB are almost useless for debugging the kernel because the code is so optimized it bears no relation to the original source and many variables are optimized out. This makes stepping, hence stepping through the source is impossible, examining variables is impossible and is therefore almost pointless.

Actually it is worse than useless. It actually gives you false information and so detached is the code you are looking at to the actual running code.

And no, you can’t turn off optimizations in the kernel. It doesn't compile.

I have to say, coming from a Windows kernel environment, the lack of a decent debugger is annoying, given that there is junk code out there to maintain.

Recuperator answered 8/7, 2013 at 14:25 Comment(4)
make config ---> "Kernel hacking" --> "Compile-time checks and compiler options" -> "Compile the kernel with debug info" – Glyceryl
That isn't a problem with the kernel, but any program produced by a sufficiently optimizing compiler. True, the Linux kernel cannot be easily built without optimization, but you can enable CONFIG_READABLE_ASM. – Laith
CONFIG_READABLE_ASM gives you false hopes. Instruction reordering is there because -O2 and you still see gibberish while tracing. The only solution I've found is to set -O0 everywhere the build system doesn't complain, plus some more hacking. And then debug using qemu. I can see where the tone comes from :) – Vuillard
No -O0 really does suck stackoverflow.com/questions/29151235/… but at least it gives you an idea of what function was called. – Windsor

© 2022 - 2024 — McMap. All rights reserved.