Can one use libSegFault.so to get backtraces for SIGABRT?

Question

The magic incantation

LD_PRELOAD=/lib/libSegFault.so someapp

runs someapp with libSegFault.so providing backtrace information on a SIGSEGV as described in many different places.

Other than using signal(7)-like approaches to cause SIGABRT to invoke the SIGSEGV handler, is there some way to get libSegFault to provide backtrace information for assert(3) failures?

Chumash

Answer 1

env SEGFAULT_SIGNALS="abrt segv" LD_PRELOAD=/lib/libSegFault.so someapp

Note that the actual path to the preload library may differ. On my machine, I'd use

env SEGFAULT_SIGNALS="abrt segv" LD_PRELOAD=/lib/x86_64-linux-gnu/libSegFault.so some-64bit-app

or

env SEGFAULT_SIGNALS="abrt segv" LD_PRELOAD=/lib/i386-linux-gnu/libSegFault.so some-32bit-app

depending whether the application I was running was compiled 64-bit or 32-bit. (You can use file to check.)

---

The source tells us there are three environment variables that define how libSegFault.so behaves:

• SEGFAULT_SIGNALS: The list of signals that cause a stack trace. The default is SIGSEGV. A defined but empty SEGFAULT_SIGNALS means no signals cause a stack trace. The supported values are segv, ill, abrt, fpe, bus on systems that have the SIGBUS signal, stkflt on systems that have the SIGSTKFLT signal, and all for all of these.

• SEGFAULT_USE_ALTSTACK: If defined in the environment, libSegFault.so uses an altenate stack for the stack trace signals. This may come in handy if you are debugging stack corruption.

• SEGFAULT_OUTPUT_NAME: If defined in the environment, the stack trace is written to this file instead of standard error.

To be honest, I found these initially by examining the library with strings /lib/libSegFault.so | sed -e '/[^0-9A-Z_]/ d'. All standard libraries (libSegFault.so having become a part of GNU C library) are tunable via environment variables, so using something like that command to dump any strings that look like environment variable names is a quick way to find stuff to search for. Doing a web search on "SEGFAULT_SIGNALS" "SEGFAULT_OUTPUT_NAME" produces a number of useful links; seeing that it was part of the GNU C library nowadays, I went to the source git archives, found the actual source file for the library, and posted my answer.

Illuminative

Thank you. Where is that behavior of SEGFAULT_SIGNALS documented? I spent some time digging around for details on libSegFault.so and kept coming up blank. – Chumash

@RhysUlerich: At the source, sourceware.org/git/?p=glibc.git;a=blob;f=debug/… -- I'll add the descriptions gleaned from the sources to my answer, in case it helps somebody later on. – Illuminative

Also, sorry for not awarding the bounty earlier. Dunno why I thought it would happen once I accepted the answer. Again, thanks for the time you spent on this. – Chumash

You don't need the fully qualified path name, LD_PRELOAD=libSegFault.so is fine if the library is in the dl path. – Kellby

@FernandoSilveira: I recommend not mentioning that to beginners, because they often end up with more than one version of their library installed in different library directions, causing grief and wails for help when nothing they do seems to have an effect. The fix, of course, is to verify that there is only one copy of the library in any of the standard library directories, but the locations vary from distribution to distribution (and even depending on whether you have a multilib installation or not). No, sir; it is definitely better to guide them to use full path names. – Illuminative

@NominalAnimal I still disagree (maybe even more): I don't see why someone would install two versions of libSegFault.so (as they come pre installed in 99% of the Linux distros, why and how would a beginner install two glibcs in a machine or even touch the libSegFault.so library?) and it is much more difficult for someone to find the right version of libSegFault.so than for LD to find the wrong version in the path. I'd never ask someone (specially a beginner) to call /bin/ls instead of ls. My 2 cents. – Kellby

@FernandoSilveira: Well, libSegFault.so is special, being a system library, and you do have a good argument there. However, I've answered here and elsewhere quite a few questions regarding how to interpose (standard) library functions, and the most common problem I've seen is mis-installing the (user's own) interposing library, and using only the library file name in LD_PRELOAD. It just seems to me that consistently using the full path is the easiest way to avoid the most common pitfall I've seen. I could be misguided or just plain wrong, though; so do feel free to disagree. – Illuminative

The source code moved here github.com/zatrazz/glibc-tools/blob/main/libSegFault/segfault.c – Bier

Answer 2

In a similar vein, the glibc exception handler writes a stack dump to /dev/console on heap corruption errors.

If you are running your executable in a non tty (i.e. a systemd process or other detached process), the crash output goes to /dev/null, which is not so useful.

There is an undocumented feature to redirect the output to /dev/stderr. Set the following environment variable:

export LIBC_FATAL_STDERR_=1

This can be used in conjunction with libSegFault.so for maximal forensics.

It is also worth mentioning that this might give you two stack traces if you also enable backtraces for SIGABRT, as the glibc first does a stack trace, then signals SIGABRT ... and then libSegFault gives a second stack trace.

Alis