What profiler should I use to measure _real_ time (including waiting for syscalls) spend in this function, not _CPU_ one

Question

The application does not calculate things, but does i/o, read files, uses network. I want profiler to show it.

I expect something like something like in callgrind that calls clock_gettime each proble.

Or like oprofile that interrupts my application (while it is sleeping or waiting for socket/file/whatever) to see what is it doing.

I want things like "read", "connect", "nanosleep", "send" and especially "fsync" (And all their callers) to be bold (not things like string or number functions that perform calculations).

Platform: GNU/Linux @ i386

Meilhac

At least related: #2449659. But answers are not clear. – Meilhac

Answer 1

Quickly hacked up trivial sampling profiler for linux: http://vi-server.org/vi/simple_sampling_profiler.html

It appends backtrace(3) to a file on SIGUSR1, and then converts it to annotated source.

As it probes the program periodically, we'll see functions that waits for something.

And as it walks the stack, we'll see callers too.

Also people from answers to similar questions recommends Zoom.

Meilhac

++ Now you're on the right track! Once you have stack samples, it's only a matter of what you present. The basic thing is percent of samples going through each line. (What Zoom does.) You don't need a large number of samples to get reasonable percents. Then, I prefer to look at individual samples, because they tell a lot more than just numbers do. – Pronounce

@Mike Dunlavey, Should I try to make it reusable? (github, *.deb packages, ...) – Meilhac

@Vi: I don't know. That's up to you. It is nice to see that you could make a simple sampling profiler so easily. Good work. – Pronounce

@Mike Dunlavey, 1. Thinking that it would be more nice to just convert it to callgrind format (to feed into kcachegrind). Already looked at that format. 2. Tried obtaining samples using lsstack (works, but not as I want) or gdb (slow and hacky). Can you propose other ways of obtaining stack traces fast enough with more-or-less reliable line numbers information? – Meilhac

@Vi: Well, I'm sure I'm slow and hacky, but I think that method gets the best results, when I've got an actual problem to solve. Just 2 days ago I sped up an ODE solver. Four samples showed it was being reinitialized more often than necessary. For something better-looking, I assume you have seen the Zoom demo. – Pronounce

Answer 2

There's no real way to answer that question without knowing your platform and a little more about what you are trying to do.

When I'm running on an Intel platform, I like to use the Time Stamp Counter (RDTSC). It is tough to beat resoultion in the sub-microsecond range. I just put a call to it before and after a chunk of code, and compare the difference.

Sartorius

Updated question to include platform – Meilhac

Yes, but I want callgrind to call it. – Meilhac