Confused about profiling result

Question

I have built my program with "-g -O2" and ran valgrind+cachegrind. I am unsure how to interpret the output. Here is the output:

http://daviddoria.com/Uploads/callgrind.CacheMisses

My "whole program" is the InpaintingAlgorithm function that is 98.4% of "main". So far so good. Now looking at the callees of InpaintingAlgorithm, 92.9% of InpaintingAlgorithm is LinearSearchKNNProperty::operator(). This is my "inner loop", and again I expect a huge amount of the time to be spent here.

Now here is where I get confused. Looking at the callees of LinearSearchKNNProperty::operator(), there is really nothing there?? The largest function is only 7.64%, and the rest are < 0.25%. I don't understand how the sum of all of the callees only adds to about 8%. Where is the other 92%?? (Presumably the stuff I would be looking for to make it go faster!)

If anyone could point me to my error in reading these results, I would appreciate it!

Calla

It's hard to make some conclusions without complete data. Perhaps someone who already has encountered such a situation can help. – Isentropic

What about the code inside the function + whatever function calls are inlined into that function body? Why would you presume this cannot consume 92% of that execution time? Posting the code for that function might be helpful. – Absorbefacient

Ah, I just had a thought. Could it be that "low level"/"built in" things (like calls to a normal * (multiplication for floats, for example) in the function itself (or in functions that have been inlined) are what are taking the rest of the time? – Calla

@Mikael Persson - yea, that has to be it then. I've never really used a profiler with optimized code, so I forgot about inlining. With "-g" only, usually the callees add up to about 100% of the caller. So how does this help me at all then? If I want to know where that 92% of the time is being spent, I have to just look manually at all of the functions that are called and assumedly inlined and guess? – Calla

@DavidDoria: Assumedly inlined? No. Inspect the assembly listings for that function. Guess? Hopefully, educated guesses. You can also try gprof instead, see if it is more helpful. I think some profilers allow you to add sequence-points in the code manually too. And for simple purposes like that, hand-coding a small in-code profiler to gather time-statistics about some manually placed points in the code is a pretty trivial task. – Absorbefacient

@DavidDoria: Don't mess with any of that. Stackoverflow is riddled with questions like yours. Do this instead. – Collaboration