How to debug FUSE filesystem crash in Linux

Question

Currently I am developing an application using FUSE filesystem module in Linux (2.6 Kernel) in C language. Due to some programming error, the application crashes after mounting the filesystem. Since I am a novice developer in Linux/C environment. Could you please let me tell me possible options to debug such programs?

Soft

What do you mean "using"? Are you try implementing a use space file system based on fuse mechanism or something else? – Defrost

+1 - FUSE can be a bit of a pain to debug. – Merits

@arsane, yes I am implementing a user space file system based on FUSE. – Soft

Answer 1

There are a couple features of FUSE that can make it difficult to debug: it normally runs in the background (which means it detaches from stdin/out) and is multithreaded (which can introduce race conditions and is more complicated to debug with gdb). Luckily, both features can be disabled:

1. Use the -f switch to keep your application in the foreground. This will make your printf lines work.
2. Use the -s switch to disable multithreading. Disabling multithreading will limit performance, but will also hide certain bugs (race conditions), simplify the use of gdb, and ensure printf output is readable (when multiple threads call printf at about the same time their output can get mixed up).

I'd also recommend reading Geoff Kuenning's FUSE documentation.

Cocky

Thank you for the pointer. As far as I can see, this is the most helpful answer. – Egret

Answer 2

Run your fuse client with the -d option.

Sitarski

Answer 3

First, make sure you're compiling with debugging symbols enabled (-g option to gcc). Before you run your program, enable core dumps with the shell command:

ulimit -c unlimited

Then when the application crashes, it'll leave a core file in the current working directory (as long as it can write to it).

You can then load the core file in the gdb debugger:

gdb <executable file> <core file>

...and it'll show you where it crashed, and let you examine variables and so forth.

Mariel

Answer 4

You can use Valgrind with FUSE, however read this first to learn about a setuid work-around. I actually do the following as a convenience for others who might need to debug my file system:

#include <valgrind/valgrind.h>

if (RUNNING_ON_VALGRIND) {
    fprintf(stderr,
        "******** Valgrind has been detected by %s\n"
        "******** If you have difficulties getting %s to work under"
        " Valgrind,\n"
        "******** see the following thread:\n"
        "******** http://www.nabble.com/valgrind-and-fuse-file-systems"
        "-td13112112.html\n"
        "******** Sleeping for 5 seconds so this doesn't fly by ....",
            progname, progname);
    sleep(5);
    fprintf(stderr, "\n");
}

I work on FUSE a lot .. and 90% of the time my crashes are due to a leak which causes the OOM killer to take action, dereferencing a bad pointer, double free(), etc. Valgrind is a great tool to catch that. GDB is helpful, but I've found Valgrind to be indispensable.

Merits

Answer 5

UML is very good for debugging generic parts of linux kernel like filesystem, scheduling but not the hardware platform or drivers specific part of kernel.

http://www.csee.wvu.edu/~katta/uml/x475.html

http://valerieaurora.org/uml_tips.html

And looking at the diagram carefully:

You will see the application "hello" which is implementing all the FUSE callback handlers. So majority of debugging is in userspace program, as the FUSE kernel module (and libfuse) is generically meant to be used by ALL FUSE filesystem.

Nuclease