Backtracing on Linux 64 bit from Signal Handler with malloc/free on callstack

Question

Below is an example of source I want to use on a machine running "Red Hat Enterprise Linux 5.5 (Tikanga) Kernel 2.6.18-194.el5xen x86_64" OS.

The general idea is that I want to have backtrace of some thread, so I am raising a SIGUSR1 signal for that thread and a handler does a backtrace() call.

In my scenario as below, FrameTwo function calls malloc and free in a loop. Whenever the signal is raised for this particular thread and free or malloc is on the callstack, the progream crashes when the signal handler calls backtrace().

(gdb) where (stack from gdb)
0  0x0000003e67207638 in ?? () 
1  0x0000003e672088bb in _Unwind_Backtrace
2  0x00000037ba0e5fa8 in backtrace () 
3  0x000000000040071a in handler ()
4  <signal handler called>
5  0x00000037ba071fac in _int_free () 
6  0x0000000a33605000 in ?? ()
7  0x000000004123b130 in ?? ()
8  0x00000000004007d4 in ThreadFunction ()
9  0x000000001f039020 in ?? ()
10 0x000000004123b940 in ?? ()
11 0x0000000000000001 in ?? ()
12 0x0000000000000000 in ?? ()

I learned from other sources that backtrace shouldn't be called from a signal handler, so I have written my own function grok_and_print_thread_stack() for this case.

It uses the RBP register to navigate the stack (RBP contains the base pointer of the current frame points to the previous frame's base pointer), but this algorithm does not work in this case either: when _int_free () is on the callstack, the RBP register navigation algorithm breaks, because the RBP of _int_free is some value like 0x20 which is not a valid frame's base pointer.

Does anyone know how a callstack can be navigated from the registers? Or how can I use backtrace for my purpose?

#include "stdio.h"
#include "stdlib.h"
#include "pthread.h"
#include "signal.h"
#include "syscall.h"
#include "string.h"
#include "inttypes.h"

//####################################################################

//gcc BacktraceTestProgram.c -o backtracetest -lpthread
//./backtracetest
//gdb -c core backtracetest

//####################################################################
volatile sig_atomic_t flag = 1;
int thlist[6] = {0};
int cnt = 0;
int *memory = NULL;

//####################################################################

void raiseUserSignal(int tid)
{
    union sigval value;
    value.sival_int = 1;
    sigqueue(tid,SIGUSR1, value);
}

//####################################################################

int grok_and_print_thread_stack()
{
    int ret = 0;
    register uint64_t* rbp asm("rbp");
    /*if buffer was built before, add separator */
    uint64_t *previous_bp;

    /*save pointers*/
    previous_bp = rbp;

    /* stack Traversal */
    while(previous_bp)
    {
        uint64_t *next_bp;

        next_bp = (uint64_t*)*previous_bp;
        printf("Read BP: %lx \n", next_bp);

        if ( NULL == (void*)next_bp )
        {
            printf("Reached the top of the stack\n");
            fflush(stdout);
            break;
        }

        previous_bp = next_bp;
    }
    return ret;
}

//####################################################################

void handler(int signum, siginfo_t *info, void *context)
{

    int nptrs = 0 ;
    void *buffer[100] = {NULL};
    char **strings = NULL;

    nptrs = backtrace(buffer, 100);

    flag = 1;
}

//####################################################################

void FrameTwo(const char A)
{
    do{
        if( memory == NULL)
            memory = (int *)malloc(sizeof(int) *5);

        if(memory != NULL) {
            free(memory);
            memory = NULL;
        }
    }while(1);
}

//####################################################################

void FrameOne(int no)
{
    FrameTwo('A');
}

//####################################################################

void *ThreadFunction( void *ptr )
{
    int tid = syscall(SYS_gettid);
    thlist[cnt++] = tid;

    FrameOne(10);
}

//####################################################################

void RegisterSignalHandler()
{
    /* Register a Signal Handler */
    struct sigaction usrsig_action;
    usrsig_action.sa_flags = SA_SIGINFO;
    usrsig_action.sa_sigaction = &handler;
    sigaction (SIGUSR1, &usrsig_action, NULL);
}

//####################################################################

int main(int no , char *argc[] )
{
    int iret1;
    pthread_t thread1;
    RegisterSignalHandler();

    /* Create independent threads each of which will execute function */
    iret1 = pthread_create( &thread1, NULL, ThreadFunction, NULL);

    while(cnt == 0);

    while(1) {
        if(flag == 1){
            flag = 0;
            raiseUserSignal(thlist[0]);
        }
    }

    pthread_join( thread1, NULL);
    return 0;
}

Dit

IIRC the signal handler has its own stack, so you wouldn't see the main stack of your program but that of your signal handler. – Stelliform

@Stelliform i am able to see main stack as well in most of the scenarios.. in fact in general while crash analysis people Suggest to implement the Signal handler for crash and call backtrace() from that handler to see main stack.. – Dit

Answer 1

In general x86_64 programs are likely to have been built with -fomit-frame-pointer as it is the default when optimisation is on.

What that means is that RBP is not usable for unwinding the stack and you will either need to use the DWARF unwind information (if you have debugging information available) or the exception unwind table.

Stuckey

in most of the scenarios this algorithm with RBP(Frame Pointer register in callstack) works and fails in very few scenarios like free and malloc... if i remove free and malloc from my stack.. this implementation works.. is "exception unwind table" available during execution or at the time of crash only? How can i use DWARF unwind information ? any reference? – Dit

Your code may have been built with frame pointers, but the system library routines (like malloc and free) won't have been. So backtracing with RBP will work in your code but not when you are tracing out of the C libary. – Stuckey

Answer 2

You may want to look at the libunwind project.

The primary goal of [libunwind] is to define a portable and efficient C programming interface (API) to determine the call-chain of a program. [...] As such, the API is useful in a number of applications. Some examples include:

• debuggers
  The libunwind API makes it trivial for debuggers to generate the call-chain (backtrace) of the threads in a running program/

In particular, have a look at the local unwinding section of their documentation, it contains explanations and the following code example (with you need to link with -lunwind) that prints the backtrace of the current function:

#define UNW_LOCAL_ONLY
#include <libunwind.h>

void show_backtrace (void) {
  unw_cursor_t cursor; unw_context_t uc;
  unw_word_t ip, sp;

  unw_getcontext(&uc);
  unw_init_local(&cursor, &uc);
  while (unw_step(&cursor) > 0) {
    unw_get_reg(&cursor, UNW_REG_IP, &ip);
    unw_get_reg(&cursor, UNW_REG_SP, &sp);
    printf ("ip = %lx, sp = %lx\n", (long) ip, (long) sp);
  }
}

Bet