use trace-cmd/ftrace to get function_graph just before panic() happened

Question

I am trying to use trace-cmd to gather more information about a kernel crash that I am seeing. Unfortunately, kernel crashes with "kernel panic - not syncing" message (i.e. socket and file buffers are not flushed so whatever was in the buffers at the time of crash is inevitably lost).

Is there a way to:

1. force trace-cmd process to flush its buffers to filesystem or sockets? OR
2. make trace-cmd/ftrace to avoid buffering (ie call printk() behind scenes and use netconsole)?

Since I am running that trace-cmd command in virtualized environment then I don't care that hard drive may occasionally get corrupted, because I can simply revert to last good snapshot or redeploy the VM.

Geminate

Answer 1

I can't guarantee my recipe helps you, but who knows :)

At first try to unbuffer all output of trace-cmd. You could find some recipes to do this there https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe

Second, kernel treats writing FS on panic is very dangerous and that's ok, however you could try to store last bites of info before death outside of crashed machine. Try to redirect your critical info to serial-port or network connection and catch it on the other side. Like

unbuffer trace-cmd >/dev/ttyS0

Another way is to try avoid panic at all and let kernel works longer however with unpredictable result. You can disable panic for several cases. Look at panic_on_* settings in /proc/sys/kernel/ directory. More details can be found in the "Documentation for /proc/sys/kernel/"

Guadiana

This did not solve the problem, because I believe there is another buffer between ftrace and trace-cmd. The unbuffer only unbuffers the STDOUT, right? – Geminate

Yes, of course. This approach just a try to reduce buffering. Another way is to try avoid panic at all and let kernel works longer however with unpredictable result. You can disable panic for several cases. Look at "panic_on_" settings in the "Documentation for /proc/sys/kernel/" (kernel.org/doc/Documentation/sysctl/kernel.txt). – Guadiana