How to include google benchmark in continuous integration

Question

I like to include google benchmark tests within out continuous integration environment.

How can I check if the code under test has not become slower than it was before? Do I have to manually process the json output of the benchmark?

Is there an elegant way of accounting for different machines?

Is there an elegant way to check for every build on the same machine the performance? Is there a better way than writing a bash script and compare the output of the actual performance with a reference performance?

There has been a similar question, but no good answer to it.

Is there a way to integrate it into Jenkins.

Priceless

In case you are still looking for a way: Have you tried compare_bench.py tool – Eatton

Answer 1

To integrate Google Benchmark in Jenkins, I used this neat plugin:

https://plugins.jenkins.io/benchmark

I had the tests output .json files

benchmarks.exe --benchmark_out=benchmarktest_output.json

and wrote a custom JSON schema for the plugin to understand the output:

{
  "description": "Google Benchmark JSON schema",
  "failure": { "value": true },
  "type": "object",
  "properties": {
    "benchmarks": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": { "type": "name" },
          "real_time": { "type": "result" },
          "cpu_time": { "type": "result" },
          "iterations": { "type": "parameter" }
        }
      }
    }
  }
}

I'm not able to display all the output in the plugin though. E.g., I haven't figured out how to display the time unit together with both real and cpu time, only with one of them at the time.

Luciolucita

Answer 2

With this python tool one can automate the evaluation of google benchmark tests and thus integrate them into a continuous integration tool

Thanks you @MikeVanDyke for the hint.

Priceless

Could you elaborate on that ? Especially When a new git commit leads to performance degration, I want the CI to trigger a warning, even stop the new commit from being merged. Is that possible ? – Distefano

I wonder how to use compare.py to compare the performance between the base commit version and your commit version ? – Distefano