How to pass arguments to a Google Benchmark Program

Question

I have a C++ Google Benchmark Program. It uses Google's BENCHMARK_MAIN() method. Now I call and execute the compiled program with a Go script. Is there a way to pass arguments into my benchmark program? (I know the common way over a main method, but I'm not sure how to do it in Googletest, because it's implemented in benchmark_api.h and I can't just change that.)

Update:

So far I copied out the macro body into my benchmark.cpp and added a line. It is not a nice solution because possible changes from Google on this Macro (like a name changing or an added code line) wouldn't affect my copy. It is working at last.

int main (int argc, char** argv)
{
    MyNamespace::conf = {argv[1]};
    ::benchmark::Initialize (&argc, argv);
    ::benchmark::RunSpecifiedBenchmarks ();
}

Stray

What arguments do you want to pass to the program? It the arguments are for google benchmark then just call the program and append the arguments for example ./prog_name --benchmark_format=json. – Pretoria

Did you read the documentation? What did that tell you? – Andante

@MikevanDyke I want to pass two integer values and a string, because it is meant to be a generic benchmark. I wanna add a chunk size and an iteration count for a loop inside the used function to benchmark and the name of the used function in the benchmark test itself. This way I wanna run single, are modifiable Benchmarks manually trough a go script. – Stray

So far I just copied out the body of the 'BENCHMARK_MAIN()' makro (the main method). So now I don't use the makro anymore but the the body of that, which I modified now to use the first passed Argument and put it into a variable. – Stray

This approach is now mentioned in the official docs, so it ought to be stable: github.com/google/benchmark#using-registerbenchmarkname-fn-args You can also pass your args in to benchmark::RegisterBenchmark via ->Arg(...), rather than using a global variable. – Zannini

Answer 1

Hacking the whole BENCHMARK_MAIN function is of course one way to do it, but IMO it's really cumbersome and really ugly. So I'm just gonna propose a different approach:

// define your chunksize and iteration count combinations here (for i and j) 
static void CustomArguments(benchmark::internal::Benchmark* b) {
  for (int i = 0; i <= 10; ++i)
    for (int j = 0; j <= 50; ++j)
      b->Args({i, j});
}

// the string (name of the used function is passed later)
static void TestBenchmark(benchmark::State& state, std::string func_name) {
  // cout for testing purposes
  std::cout << state.range(0) /* = i */ << " " << state.range(1) /* = j */ 
  << " " << func_name << std::endl;
  for (auto _ : state) {
     // do whatever with i and j and func_name
  }
}

// This macro is used to pass the string "function_name1/2/3" 
// as a parameter to TestBenchmark
BENCHMARK_CAPTURE(TestBenchmark, benchmark_name1, "function_name1")
    ->Apply(CustomArguments);
BENCHMARK_CAPTURE(TestBenchmark, benchmark_name2, "function_name2")
    ->Apply(CustomArguments);
BENCHMARK_CAPTURE(TestBenchmark, benchmark_name3, "function_name3")
    ->Apply(CustomArguments);

BENCHMARK_MAIN()

And then in your go script, you call the benchmark with a regex filter:
./prog_name --benchmark_filter=InsertRegexFilterHere

So for example:
./prog_name --benchmark_filter=TestBenchmark/benchmark_name2/5/35

The above example will call the benchmark and will pass "function_name2", 5 and 35 (these are the values for your chunksize and iteration count) and so the output will be something like:

------------------------------------------------------------------------------
Benchmark                                       Time           CPU Iterations
------------------------------------------------------------------------------
TestBenchmark/benchmark_name2/5/35               2 ns          2 ns  308047644

Pretoria

Answer 2

I would like to extend Mike van Dyke's answer by adding that the library also supports the following:

static void BM_SetInsert(benchmark::State& state) {
  std::set<int> data;
  for (auto _ : state) {
    state.PauseTiming();
    data = ConstructRandomSet(state.range(0));
    state.ResumeTiming();
    for (int j = 0; j < state.range(1); ++j)
      data.insert(RandomNumber());
  }
}
BENCHMARK(BM_SetInsert)
    ->Args({1<<10, 128})
    ->Args({2<<10, 128})
    ->Args({4<<10, 128})
    ->Args({8<<10, 128})
    ->Args({1<<10, 512})
    ->Args({2<<10, 512})
    ->Args({4<<10, 512})
    ->Args({8<<10, 512});

In your benchmark, state.range(0) and state.range(1) now refers to the first and second argument respectively. See more @ https://github.com/google/benchmark#passing-arguments

Bendicta

Answer 3

As described in https://github.com/google/benchmark/blob/main/docs/user_guide.md#using-registerbenchmarkname-fn-args, benchmarks can be registered interactively like so:

auto BM_test = [](benchmark::State& st, auto Inputs) { /* ... */ };

int main(int argc, char** argv) {
  for (auto& test_input : { /* ... */ })
      benchmark::RegisterBenchmark(test_input.name(), BM_test, test_input);
  benchmark::Initialize(&argc, argv);
  benchmark::RunSpecifiedBenchmarks();
  benchmark::Shutdown();
}

It should be noted that normally benchmark aborts if it sees unknown arguments. This is not the case for benchmark::Initialize. This function removes all benchmark specific options like --benchmark_filter, --benchmark_out... from argv and it leaves unknown options and positional arguments intact (argc is modified too).

This means that one can call

benchmark::Initialize(&argc, argv);

and then get program specific arguments from the cleaned argc and argv.

Fluff