How to debug the Linux kernel with QEMU and KGDB?

Question

I have been able to boot a powerpc based system (MPC8544DS to be specific) using the following way to invoke qemu (v1.7.0)

qemu-system-ppc -M mpc8544ds -m 512 -kernel zImage -s -nographic -initrd busyboxfs.img -append "root=/dev/ram rdinit=/bin/sh kgdboc=ttyS0,115200 kgdbwait"

where zImage is a custom cross compiled Linux Kernel (v2.6.32) which has KGDB enabled and compiled in (for startupcode debugging) and busyboxfs.img is the busybox based rootfs.

Since I'm using the -s flag to Qemu, I can break-in to the kernel using cross gdb like so:

(gdb) target remote localhost:1234
Remote debugging using localhost:1234
mem_serial_in (p=<value optimized out>, offset=5) at drivers/serial/8250.c:405
405  }

However if I remove the -s flag and try to break in to the kernel over /dev/ttyS0 it gives me a permission denied error:

(gdb) set remotebaud 115200
(gdb) target remote /dev/ttyS0
permission denied 

Is it because it has been held over by Qemu? Additionally in example across the internet, kgdboc has been set to ttyAMA0 which I've come to understand stands for the AMBAbus which is specific to ARM based systems. Do we have something similar for PowerPC? Am I doing something wrong here?

Crespi

It seems quite unlikely that your Qemu is pretending to be /dev/ttyS0 - shouldn't you be able to find out from the Qemu documentation what emulated serial device it might provide? – Clemenciaclemency

Thanks for the comment and the answer Chris. It's been a while I posted this question. I'm in the thick of some user-space programming assignment at work and will need some time to re-orient myself to this. Only then will I be able to appreciate your suggestions. I hope you'll bear with me – Crespi

I think this link would solve your query <br/> #13748885 – Rowdyism

Answer 1

KGDB + QEMU step-by-step

First, QEMU's -gdb option is strictly more powerful than KGDB, so you might want to use that instead: How to debug the Linux kernel with GDB and QEMU? QEMU is however an easy way to play around with KGDB in preparation for real hardware. I have posted some Raspberry Pi KGDB pointers at: Linux kernel live debugging, how it's done and what tools are used?

If you want to get started quickly from scratch, I've made a minimal fully automated Buildroot example at: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/d424380fe62351358d21406280bc7588d795209c#kgdb

The main steps are:

1. Compile the kernel with:

   CONFIG_DEBUG_KERNEL=y
   CONFIG_DEBUG_INFO=y
   
   CONFIG_CONSOLE_POLL=y
   CONFIG_KDB_CONTINUE_CATASTROPHIC=0
   CONFIG_KDB_DEFAULT_ENABLE=0x1
   CONFIG_KDB_KEYBOARD=y
   CONFIG_KGDB=y
   CONFIG_KGDB_KDB=y
   CONFIG_KGDB_LOW_LEVEL_TRAP=y
   CONFIG_KGDB_SERIAL_CONSOLE=y
   CONFIG_KGDB_TESTS=y
   CONFIG_KGDB_TESTS_ON_BOOT=n
   CONFIG_MAGIC_SYSRQ=y
   CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
   CONFIG_SERIAL_KGDB_NMI=n
   

   Most of those are not mandatory, but this is what I've tested.

2. Add to your QEMU command:

   -append 'kgdbwait kgdboc=ttyS0,115200' \
   -serial tcp::1234,server,nowait
   

3. Run GDB with from the root of the Linux kernel source tree with:

   gdb -ex 'file vmlinux' -ex 'target remote localhost:1234'
   

4. In GDB:

   (gdb) c
   

   and the boot should finish.

5. In QEMU:

   echo g > /proc/sysrq-trigger
   

   And GDB should break.

6. Now we are done, you can use GDB as usual:

   b sys_write
   c
   

Tested in Ubuntu 14.04.

ARM

Can't get it work. Possibly related to: How to use kgdb on ARM??

Detach

Answer 2

You appear to be confusing the host serial device /dev/ttyS0 for the guest one, and QEMU's own gdbserver for KGDB in the guest kernel.

There is normally no reason for QEMU to touch the host's serial port. Really the only reason for doing that would be if you wanted to have one physical machine host QEMU, and effectively give its physical serial port to the guest, so that you could then use a different physical machine connected by an actual serial cable to debug the guest.

When you use the -s flag, you tell QEMU to run its own GDB server (by default listening on host loopback TCP port 1234) allowing you to break into whatever program is running on the guest, be that a kernel or bootloader or something else. This is not the same as having the guest kernel itself cooperate with debugging via KGDB.

If you want to use KGDB, you are going to need to so something like configure KGDB in the kernel build to use the guest side of an emulated serial port, and then tell GDB on the host to use the host end of that emulated port. The QEMU command line documenation covers this in detail:

Debug/Expert options:

‘-serial dev’ Redirect the virtual serial port to host character device dev. The default device is vc in graphical mode and stdio in non graphical mode.

This option can be used several times to simulate up to 4 serial ports.

An abbreviated list of some of your more interesting options:

‘pty’ [Linux only] Pseudo TTY (a new PTY is automatically allocated)

‘/dev/XXX’ [Linux only] Use host tty, e.g. ‘/dev/ttyS0’. The host serial port parameters are set according to the emulated ones.

This is what you don't want - unless you want to use a serial cable to a different physical machine that will run GDB.

‘tcp:[host]:port[,server][,nowait][,nodelay]’ The TCP Net Console has two modes of operation. It can send the serial I/O to a location or wait for a connection from a location. By default the TCP Net Console is sent to host at the port. If you use the server option QEMU will wait for a client socket application to connect to the port before continuing, unless the nowait option was specified. The nodelay option disables the Nagle buffering algorithm. If host is omitted, 0.0.0.0 is assumed. Only one TCP connection at a time is accepted. You can use telnet to connect to the corresponding character device.

Example to send tcp console to 192.168.0.2 port 4444 -serial tcp:192.168.0.2:4444

Example to listen and wait on port 4444 for connection -serial tcp::4444,server

Example to not wait and listen on ip 192.168.0.100 port 4444 -serial tcp:192.168.0.100:4444,server,nowait

This is a good and common choice. You can use basically the same GDB syntax, for example if you specify the loopback interface address 127.0.0.1 and the port 1234 you can just use exactly the same GDB command as before.

‘unix:path[,server][,nowait]’ A unix domain socket is used instead of a tcp socket. The option works the same as if you >had specified -serial tcp except the unix domain socket path is used for connections.

This is a good choice too, assuming your GDB supports it.

You may need to configure one of these options first, run without KGDB and get a shell up and figure out what the guest end of the emulated device is called, then reboot with KGDB configured to use that.

Clemenciaclemency

I've restarted this project. Now I cannot even get qemu to boot the kernel using the very same arguments. In fact theres nothing but a blank screen. I've upgraded qemu to 2.0.2. But the rest is the same. How do I know what Qemu is doing in the background? Is there a way to check for it. I've added an explicit "console=tty1" argument to the kernel. – Crespi

@HighOnMeat if you've started qemu with both -s and -S then it will wait for something to attach to it's gdb server and issue a single-step or continue (free run) command. The screen is blank until QEMU runs some code that writes to the screen... – Suiter

Answer 3

If you have an ARM kernel, you might need to add more serial ports to your qemu VM and tell your kernel which port to use for KGDB and which for the terminal.

qemu-system-aarch64 \
    -machine virt  \
    -cpu max  \
    -m 1024  \
    -drive file=arch_aarch64.qcow2,format=qcow2 \
-   -serial stdio \
+   -chardev stdio,mux=on,id=char0 \
+   -chardev socket,path=/tmp/qemu_socket.sock,server=on,wait=off,id=gnc0 \
+   -mon chardev=char0,mode=readline \
+   -serial chardev:char0 \
+   -device pci-serial,id=serial0,chardev=gnc0 \
    -kernel "<path to linux directory>/arch/arm64/boot/Image.gz" \
-   -append "root=/dev/vda2"
+   -append "root=/dev/vda2 console=ttyAMA0 kgdboc=ttyS0 kgdbwait"

In this diff we add the pci-serial device, configure which serial port is available where (stdio and socket) and tell the kernel which serial ports to use (console=ttyAMA0 kgdboc=ttyS0).

Then to use KGDB you just have to connect to the socket:

gdb ./vmlinux

(gdb) target remote /tmp/qemu_socket.sock

I've also written a step-by-step guide: Debugging ARM kernel with kgdb in QEMU.

Jhansi