Performance Overhead of Perf Event Subsystem in Linux Kernel

Question

Performance counters for Linux are a new kernel-based subsystem that provide a framework for all things performance analysis. It covers hardware level (CPU/PMU, Performance Monitoring Unit) features and software features (software counters, tracepoints) as well. Since 2.6.33, the kernel provide 'perf_event_create_kernel_counter' kernel api for developers to create kernel counter to collect system runtime information. What I concern most is the performance impact on overall system when tracepoint/ftrace is enabled. There are no docs I can find about them. I was once told that ftrace was implemented by dynamically patching code, will it slow the system dramatically?

Acclivity

perf is meant to have as little impact as possible, how much depends on what you do with it – Drachm

Answer 1

Dynamic patching is used as a way to minimize overhead when tracing isn't activated. The overhead of the PMU-based hardware counters should depend on how many different events you're sampling, and how often. It's probably low enough for you to find an acceptable configuration, depending on what you want. I would try enabling whatever information collection you're interested in, carefully set the frequency, and then check empirically whether the overhead is acceptable.

Chimaera

Answer 2

Those things are put in because people think they might be useful, and no doubt they can collect interesting information. If that is the goal, fine.

However, if the goal is to tune software for higher performance, they are of little diagnostic value.

They can provide interesting measurements, but they don't tell you what to fix.

This method does tell you what to fix.

Marela

Well, my goal is running long-term profiling on production machines, so the performance overhead concerns me most. – Acclivity

@Bo: I would think, for long-term monitoring of health, you might not need information collected at high frequency - that it would be more a matter of overall timing. If you have something that is collecting information at such high frequency that it could impact performance, I would think it would be way more information than you could possibly make sense of. Usually people do "instrumentation" (and are worried about overhead) in order to try to find "bottlenecks" (for which it is not a good method). – Marela