How to trace dynamically loaded library calls with ltrace

Question

I have a C program using dynamically loaded library to load plugins. I would like to trace the library calls in order to debug the plugin's loading.

I looked at ltrace, but I can't seem to make it work:

Here is an example program:

#include <stdio.h>
#include <stdlib.h>

#include <dlfcn.h>

int main() {
    int *a = malloc(sizeof(int));
    void *handle;
    double (*cosine)(double);
    char *error;

    handle = dlopen ("/usr/lib/x86_64-linux-gnu/libm.so.6", RTLD_LAZY);
    if (!handle) {
        fputs (dlerror(), stderr);
        exit(1);
    }

    cosine = dlsym(handle, "cos");
    if ((error = dlerror()) != NULL)  {
        fputs(error, stderr);
        exit(1);
    }

    printf ("%f\n", (*cosine)(2.0));
    dlclose(handle);
    return 0;
}

Compiling (and removing PIE, otherwise ltrace won't see anything): gcc main.c -pg -ldl -no-pie

Running: ltrace ./a.out

Output

__monstartup(0x401170, 0x401431, 0x7fffe3875838, 0x7fffe3875838)                                                               = 0
__cxa_atexit(0x7f712aa98ba0, 0, 0, 0)                                                                                          = 0
malloc(4)                                                                                                                      = 0x76ea30
dlopen("/usr/lib/x86_64-linux-gnu/libm.s"..., 1)                                                                               = 0x76ea80
dlsym(0x76ea80, "cos")                                                                                                         = 0x7f712a8abd00
dlerror()                                                                                                                      = nil
printf("%f\n", -0.416147-0.416147
)                                                                                                      = 10
dlclose(0x76ea80)                                                                                                              = 0
+++ exited (status 0) +++

As you can see, the call to cos was skipped. How can I trace that with that arguments ?

I gave a try to uftrace

But again, It doesn't trace the cos call:

uftrace -a --libname --nest-libcall ./a.out
-0.416147
# DURATION     TID     FUNCTION
            [  8300] | main() {
   1.754 us [  8300] |   [email protected](4) = 0x15c6120;
 509.774 us [  8300] |   [email protected]("/usr/lib/x86_64-linux-gnu/libm.so.6", RTLD_LAZY) = 0x7ff70ae4d090;
   2.140 us [  8300] |   [email protected](0x7ff70ae4d090, "cos") = 0x7ff70aa61d00;
   0.463 us [  8300] |   [email protected]() = "NULL";
 332.451 us [  8300] |   [email protected]("%f\n") = 10;
   2.134 us [  8300] |   [email protected](0x7ff70ae4d090) = 0;
 958.926 us [  8300] | } /*

Which is surprising because on this comment it looks like it works.

Running on Ubuntu 20.04

• ltrace: 0.7.3
• uftrace: 0.9.3

Thank you for your help !

Haggar

Answer 1

You need to add a special flag -x pattern to force tracing of symbols in dlopen-ed libraries (the easiest would be -x '*', see @nayana's answer for more details.)

Older versions of ltrace do not include the relevant patch from https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=537781 (follow link to timetobleed.com) so they can't trace dlopen-ed libs.

Slovenly

AFAIK ltrace does have the -x option in version 0.7.3, that reference on timetobleed.com uses version 0.5 – Roundshouldered

Answer 2

The yugr answer is most relevant in this case. However even when using ltrace with the patch the usage is not straight-forward.

The important part is to use -x option otherwise the symbols from library load via dlopen will not show. This fact is stated in NEWS:

*** Support tracing of symbols from libraries opened with dlopen

These symbols are selected by -x.

Luckily we can set it to show all symbols matching our library address which is achieved by using special glob format described in ltrace man page.

My use case was to debug custom alsa-lib plugin which is loaded via dlopen:

# ltrace -x "@libdl.so.2" -x "*@libcustom.so" -f aplay -D custom test.wav
[pid 4183] snd_pcm_hw_params(0x5598dc8ce0, 0x7fe4d76fe0, 0xa3377eac3f8c3d00, 0 <unfinished ...>
[pid 4183] [email protected](0x5598dc5db0, 1, 0x5598dc8a4c, 0 <unfinished ...>
[pid 4183] _init@libdirac_dldsp.so(4, 0x7fe4d77858, 0x7fe4d77880, 0x7fa61ce0d8 <unfinished ...>
....
[pid 4183] [email protected](0x5598dc8ef0, 0x7fa65b1e48, 1, 0)                                                                                      = 0x7fa61cf890
[pid 4183] [email protected](0x7fa61cf890, 0, 0x7fa6627988, 1)                                                                       = 0x7fa61fe9e8
.....
[pid 4183] [email protected](0x7fe4d76770, 0x7fa61ffaf0, 0, 0 <unfinished ...>
[pid 4183] [email protected](0x7fe4d76770, 1, 3, 0)                                                                                      = 0
....

Hemichordate

Nice. Note that one can also do simply -x '*'. – Slovenly