Should I create separate Benchmark project?

Question

I want to measure performance of some methods in my console application using BenchmarkDotNet library.

The question is: should I create a separate project in my solution where I will copy the methods I am interested in measuring and do the measuring there or should I add all the attributes necessary for measuring into the existing project?

What is the convention here?

Upheave

Answer 1

You can think about it as of adding unit tests for your console app. You don't add the tests to the app itself, but typically create a new project that references (not copies) the logic that you want to test.

In my opinion the best approach would be to:

1. Add a new console app for benchmarks to your solution.
2. In the benchmarks app, add a project reference to the existing console app.
3. Add new benchmarks that have all the BDN annotations to the benchmark project, but implement the benchmarks by referencing public types and methods exposed by your console app. Don't copy the code (over time you might introduce changes to one copy and end up testing outdated version).

Ingravescent

Thanks for the tips! Especially the last one since I was in the process of copying everything to new project :D Makes more sense to reference it. – Upheave

Any suggestions on naming conventions? For tests, folder tests in the root of the repo is quite common and the .Tests suffix in the project name is a standard, too. I have seen many variations when the unit, integration, and E2E tests are differentiated, but separate folders under the tests folder make sense to me and keeping the same suffix for all seems OK as only the unit tests will have 1:1 correspondence with the implementation projects. But where to put the benchmarks? For now, I opted for benches in the repo root and .Benchmarks suffix. – Respiration

@Respiration .Benchmarks suffix sounds very good to me. When it comes to the subfolder name I am not 100% sure myself. My current approach would be to keep it under tests if that is the only benchmark project for given repo solution. If you have more benchmark projects I would move them to dedicated, benchmarks subfolder (on the same level as src and tests) – Ingravescent

I went for benches to keep the folder name short. Though even about bench. The reason for a separate folder from tests is separate treatment by the pipelines -- benchmarks may need to run separately from tests, possibly not during all builds, and they have different outputs. – Respiration

@AdamSitnik sounds absolutely reasonable, I like the separation to avoid cluttering the console application with [Benchmark] attributes. But assuming that the class that we want to benchmark has some further dependencies, e.g. some interfaces to some services, repositories or whatever: how would you be able to inject these dependencies when you create an instance of the class to be benchmarked? I tried to use .NET Core's dependency injection but it looks as if it doesn't work when using BenchmarkDotNet. – Zoilazoilla

@Zoilazoilla Does https://mcmap.net/q/1304521/-how-to-use-dependencyinjection-in-benchmarkdotnet answer your question? – Ingravescent

@AdamSitnik I had seen that answer as well. I was just wondering if there was a more "automatic" way instead of building an own dependency container in addition to the one that's already provider by .NET. – Zoilazoilla

@Zoilazoilla We don't provide that and I doubt we ever will. – Ingravescent