Line number in Google Perftools CPU profiler on MacOSX

Question

I am trying to profile some C++ programs on MacOSX. So I built google-perftools, wrote a program, compiled using MacPorts g++ 4.7, with -g compiler flag, and linked to libprofiler. Then I ran:

CPUPROFILE=cpu.profile ./a.out

Then I ran pprof to generate the output:

[hidden ~]$ pprof --text ./a.out cpu.profile 
Using local file ./a.out.
Using local file cpu.profile.
Removing __sigtramp from all stack traces.
Total: 282 samples
     107  37.9%  37.9%      107  37.9% 0x000000010d72229e
      16   5.7%  43.6%       16   5.7% 0x000000010d721a5f
      12   4.3%  47.9%       12   4.3% 0x000000010d721de8
      11   3.9%  51.8%       11   3.9% 0x000000010d721a4e
       9   3.2%  55.0%        9   3.2% 0x000000010d721e13
       8   2.8%  57.8%        8   2.8% 0x000000010d721a64
       7   2.5%  60.3%        7   2.5% 0x000000010d7222f0
       6   2.1%  62.4%        6   2.1% 0x000000010d721a4c
       6   2.1%  64.5%        6   2.1% 0x000000010d721b1f
       6   2.1%  66.7%        6   2.1% 0x000000010d721e0c
       5   1.8%  68.4%        5   1.8% 0x000000010d721fba
    ......

It looks like the perftools don't convert the addresses to function names.

Does anyone know what I am missing here? What should I do to let the profiler generate the correct result.

EDIT: More information: it is not a problem of pprof or google-perftools, but more something like gcc or macosx, because Instrument.app also shows addresses instead of line numbers. I am not familiar with how debug symbols work under Mac OS X, so I would rather think it as my missing something here, instead of being bugs in gcc or Mac OS X. I wonder whether anyone can provide some hints on how debug info works for Mac OS X.

Gipps

"I am trying the profile some C++ programs" Then why do you tag this question as C? – Hebephrenia

This problem is not specific to C++ but also to C. – Gipps

I'm not familiar with OS X dev nor googleperf, but in I've used profiling libs before, and as well as the -g flag for debugging, we had to add the -p flag to the compiler and linker to get profiling details.. – Pentyl

@fduff, tried -p flag, as well as some combinations with -g -ggdb -gstas -pg, but still no use. What combination of flags did you use then? Thanks all the same. – Gipps

I'm running the test suite for the tools, and the 'out of memory' test has driven my VM size to over 5 TB (goodness only knows where it thinks it is storing the extra; I've only got 750 GB disk). It does make browsers rather slow. I think I'll report the problem to the maintainer. – Rampant

Answer 1

This seems to be related to the address space layout randomization (ASLR) introduced in OS X 10.5

I've filed issue #562 on the gperftools issue tracker. You can disable ASLR by passing -Wl,-no_pie.

Also, if you're not bound to use gperftools, Instruments (comes with Xcode) is worth a try.

Wanda

Thanks, this was exactly it. -Wl,-no_pie makes pprof work "out of the box" on Mac OS. – Laundrywoman

Answer 2

I believe in this platform the debug symbols stay in the .o file(s), they are not moved to the executable. To get symbols in gdb or profilers you need to save the .o files. This may mean that you need to compile your application in two steps (compile, then link) to preserve the .o files.

See this question for additional information.

In looking at the pprof Perl source code, symbol names are obtained via the use of nm and c++filt, so you could try to run those standalone and figure out why they don't work. From the pprof source it looks like it tries a bunch of different command line arguments to cover several versions of nm. This is a summary of the ways

nm [-D] -n [-f] [--demangle] object-file 2>/dev/nul [| cpp+filt]

The parts that I put in brackets are those that the script determines at runtime if are required for your platform and versions of nm and c++filt. Try all the combinations of the above and see what works. Then look at what the pprof script does, maybe by adding some printfs to it.

Good luck.

Couchant

It doesn't help. I tried saving .o files but pprof still prints address instead of line numbers. – Gipps

The problem is not pprof, I would work within gdb to get these symbols loaded, that may give you an idea of what is going on. Maybe the .o files are in a different directory than expected. – Couchant

I had some progress on this: if I start the program using CPUPROFILE=cpu.profile gdb ./a.out, then start and cont inside the gdb, I can get the symbols correctly. So I guess it might be the problem of google-perftools that doesn't load the symbols if I start the program normally, so that the address recorded inside the cpu.profile is incorrect. Since gdb is pretty slow and not practical to run every time, I really need to start the program normally without gdb. So do you have any idea how I can make it work? Thanks. – Gipps

I'm pretty sure that's not correct. If you compile with debug symbols correctly, you can delete the .o files and still have all the debug info you need. – Eisele

@MahmoudAl-Qudsi, I think you are right. I deleted the .o file and ran the program with gdb, then it everything works as expected. So the question is, why does google-perftools requires gdb to do the job correctly? I mean, basically, without gdb, the address recorded are something like 0x000000010550929e, whereas with gdb, the address reported are 0x000000010000129e. I think the ones with gdb are correct because nm ./a.out also gives out lower addresses like 0x0000000100000d46. Do you have any idea what I am missing here? Thanks a lot. – Gipps