Google Benchmark Custom Setup And Teardown Method

Question

I am using benchmark library to benchmark some codes. I want to call a setup method before calling the actual benchmark code one time and not to be repeated everytime, for multiple benchmark method calls.. For e.g:

static void BM_SomeFunction(benchmark::State& state) {
  // Perform setup here
  for (auto _ : state) {
    // This code gets timed
  }
}

As we can see the setup code will be called multiple times here, for the range I specify. I did have a look at the fixture tests. But my question is can it be done without using fixture tests. And if yes then how can we do it?

Paratyphoid

Answer 1

As far as I can remember, the function is called multiple times, since benchmark dynamically decides how many times your benchmark needs to be run in order to get reliable results. If you don't want to use fixtures, there are multiple workarounds. You can use a global or static class member bool to check if the setup function was already called (don't forget to set it after the setup routine has run). Another possibility is to use a Singleton that calls the setup method in its ctor:

class Setup
{
    Setup()
    {
        // call your setup function
        std::cout << "singleton ctor called only once in the whole program" << std::endl;
    }

public:
    static void PerformSetup()
    {
        static Setup setup;
    }
};

static void BM_SomeFunction(benchmark::State& state) {
  Setup::PerformSetup()
  for (auto _ : state) {
    // ...
  }
}

However, fixtures are quite simple to use and are made for such use-cases.

Define a fixture class which inherits from benchmark::Fixture:

class MyFixture : public benchmark::Fixture
{
public:
    // add members as needed

    MyFixture()
    {
        std::cout << "Ctor only called once per fixture testcase hat uses it" << std::endl;
        // call whatever setup functions you need in the fixtures ctor 
    }
};

Then use the BENCHMARK_F macro to use your fixture in the test.

BENCHMARK_F(MyFixture, TestcaseName)(benchmark::State& state)
{
    std::cout << "Benchmark function called more than once" << std::endl;
    for (auto _ : state)
    {
        //run your benchmark
    }
}

However, if you use the fixture in multiple benchmarks, the ctor will be called multiple times. If you really need a certain setup function to be called only once during the whole benchmark, you can use a Singleton or a static bool to work around this as described earlier. Maybe benchmark also has a built-in solution for that, but I don't know it.

---

Alternative to Singleton

If you don't like the singleton class, you can also use a global function like this:

void Setup()
{
    static bool callSetup = true;
    if (callSetup)
    {
        // Call your setup function
    }
    callSetup = false;
}

Greetings

Peevish

Yeah, I was hoping for a built in solution. I am looking at its documentation. (if you use the fixture in multiple benchmarks, the ctor will be called multiple times.). this is my issue. I only want it to run one time. Since my setup step tends to be very time consuming. – Paratyphoid

@KishanKumar Have a look at the beginning of my answer. I edited in a possible solution using a singleton. – Peevish

sorry to say. It feels like a hack. I will test it. And it looks like something that will work. If no other answer comes, I will accept this as answer. But still if you come across built in method that will be great – Paratyphoid

@KishanKumar I didn't find a setup functionality the documentation. However, here is a link to a quite similar topic: #50426396 . The guy who answered sounds like being a benchmark developer. It seems like he also suggests global variables or static fixture members. Greetings – Peevish

@KishanKumar Maybe the Alternative function I just added at the end of my answer is more to your liking and does not feel so "hacky" ;) – Peevish