Profile shared object without recompiling main program

Question

I currently am developing a shared lilbrary for loading into PostgreSQL (as C-Language functions, see here). Now I would like to profile the function in this library without recompiling PostgreSQL itself.

I tried callgrind using

valgrind --tool=callgrind path/to/postgres arguments-to-postgres

This gives me profiling information for PostgreSQL itself, but fails to log the shared library I am interested in.

I also tried sprof, but I have no idea how to get that one working.

Any ideas would be highly appriciated.

P.S.: Please do not suggest just pausing the application in a debugger. With function runtimes way below 0.01 seconds I need more detailled results.

Se

Just a short update: Upgrading to PostgreSQL 9.0 and Ubuntu 11.4 did the trick. Now everything works as expected. – Se

Answer 1

Using callgrind should work as expected. To test this I setup a simple project using a simple library and main functionMakefile:

CFLAGS=-fpic
exe:exe.o lib.so
        cc -o exe exe.o lib.so
lib.so:lib.o
        cc -shared lib.o -o lib.so
clean:
        rm -f exe lib.so *.o

lib.c is a simple library containing 2 functions:

#include <stdio.h>
void someOtherFunction() { printf("someOtherFunction\n"); }
void someFunction() { printf("someFunction\n"); someOtherFunction(); }

exe.c is a very simple executable:

int someFunction();
void main() { someFunction(); }

Use the Makefile to build the executable and run it using valgrind like so:

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD valgrind --tool=callgrind ./exe

If you examine the callgrind output, you will find the profiling data for both functions in the shared library. If you cannot see these functions you may be using a non-standard environment that does not support this functionality. I'm using Linux Mint 11 x64, with the latest patches.

Frond

Thanks for your input. This minimal example works. However, in my case above there are two major differences: The library is not linked at compile time, but at runtime using dlopen. Furthermore, my Postgres build has no debugging information enabled. Thus, callgrind seems to get kinda lost within the called functions. – Se

PostgreSQL is open source - recompile it with symbols. – Adjacent

Answer 2

Point 1: Valgrind seems to have issues with privilege escalation, which is VERY likely to be performed by Postgres, see http://www.mail-archive.com/[email protected]/msg02355.html

Point 2: Have you tried to prove (e.g. with strace) that your SL is actually loaded in the same process? Have you tried to --trace-children=yes?

Point 3: I've tried to modify the test by compiling exe.o and exe with -g0 and using dlopen to load the file, i.e.:

all: exe lib.so
exe : exe.c 
        cc -g0 -o exe exe.c -ldl

lib.so:lib.c
        cc -shared lib.c -o lib.so
clean:
        rm -f exe lib.so *.o

and

#include 
#include 

void main() { 
    void *handle;
    void (*p)();
    int i;

    handle = dlopen("./lib.so", RTLD_LAZY);
    if ( ! handle ) { printf("Object not found\n"); return; }
    p = dlsym(handle, "someFunction");
    if ( ! p ) { printf("Function not found\n"); return; }
    for (i = 0; i < 100; ++i) (*p)();
    dlclose(handle);
}

Works with callgrind. Maybe Postgres is not using -ldl to open the object files?

Sarchet

Thanks for your ideas. Regarding Point 2: The strace output does contain only one call to execve, which as far as I understand suggests that everything is running in the same process. However, adding --trace-children=yes has no effect. I will have a look at your point 1 later, now my time is running short. – Se

Got time now to look into your first point. This also seems very likely for me to be the cause of trouble. Which leaves one question: Any other suggestions? – Se

Running postgres as root to prevent privilege escalation? – Sarchet

Postgres prohibits running as root. However, I can start it in single user mode with it's own user. This does not solve the problem, however. – Se

Does it still call setuid()? If yes, you might be able to solve the issue by LD_PRELOAD -ing away the setuid call. – Sarchet

Good question. Just had a short look: It does not call setuid(). – Se

Well, fair enough. I am out of ideas, sorry :-(. – Sarchet

Still, thank you for your ideas. There were quite a few nice ones! – Se