Run single test against multiple configurations in Visual Studio

Question

We have our integration tests set up using xUnit and Microsoft.AspNetCore.TestHost.TestServer to run tests against Web API running on ASP.NET Core 2.2.

Our Web API is a single code base that would be deployed separately multiple times based on some configuration or application setting differences like country, currency, etc.

Below diagram tries to explain our deployment set up:

We want to ensure that our integration tests run against all the deployments.

For both deployments, X and X` the API endpoint, request, and response are absolutely same. Hence, We would like to avoid repeating ourselves when it comes to integration tests for each deployment.

Here is the sample code explaining our current test set up:

TestStartup.cs

public class TestStartup : IStartup
{
    public IServiceProvider ConfigureServices(IServiceCollection services)
    {
       var configuration = new ConfigurationBuilder()
           .SetBasePath(Directory.GetCurrentDirectory())
           .AddJsonFile("appsettings.json", false)
           .AddEnvironmentVariables()
           .Build();

        services.AddMvc()
            .SetCompatibilityVersion(version: CompatibilityVersion.Version_2_2);

        // Code to add required services based on configuration


        return services.BuildServiceProvider();
    }

    public void Configure(IApplicationBuilder app)
    {
        app.UseMvc();

        // Code to configure test Startup
    }
}

TestServerFixture.cs

public class TestServerFixture
{

    public TestServerFixture()
    {
        var builder = new WebHostBuilder().ConfigureServices(services =>
        {
            services.AddSingleton<IStartup>(new TestStartup());
        });

        var server = new TestServer(builder);
        Client = server.CreateClient();
    }

    public HttpClient Client { get; private set; }
}

MyTest.cs

public class MyTest : IClassFixture<TestServerFixture>
{
    private readonly TestServerFixture _fixture;

    public MyTest(TestServerFixture fixture)
    {
        _fixture = fixture;
    }

    [Fact]
    public void ItShouldExecuteTwice_AgainstTwoSeparateConfigurations()
    {
        //...
    }
}

Now, I'm looking to run ItShouldExecuteTwice_AgainstTwoSeparateConfigurations test in class MyTest more than once against two different configurations/ app settings or in other words against two different test deployments within Visual Studio.

• I know, I should be able to achieve this using a combination of build configurations (like DEBUG_SETTING1, DEBUG_SETTING2) and preprocessor directive (#if DEBUG_SETTING1).

• The other option could be to have a base test helper project with common methods and a separate integration project for each deployment.

Is there a better and more elegant way to achieve this?

Alpine

By configuration, do you mean 1) build configurations as in Release vs Debug? 2) or configuration files at runtime 3) or preprocessor macros? #1 and #2 are compile time only and you'll need different assemblies. Additionally, #3 is frowned upon and you should consider either writing multiple tests or refactoring the code to avoid the need in the first place. – Dishevel

@TanveerBadar I have updated the question with more info. Hope it makes it more clear. I'm also a surprised why I got a downvote, is the question not well-formed? Could downvoter explain how the question can be improved? – Alpine

I didn't downvote your question, but you never know people. It is dependent on the mood of whoever came across it. Also spotted a typo in my comment above, it should read "#1 and #3 are compile time". – Dishevel

I understand that. thanks for the clarification. I just don't know what I could have done differently to explain my problem. – Alpine

How is your deployment pipeline setup? Why not run the tests with different settings by triggering this as part of the deployment scripts? I belive at the CI/CD level you decide which deployments need to be made. It also then makes sense to run integration tests for each deployment. So simply in your deployment setup run a new task that will set the environment variables for a given deployment and run the tests. Repeat this for each deployment. This way you do not need to change the test code when there is a new deployment with different settings. – Ink

I see you mentioned running in "Visual Studio". I guess you mean you also want to be able to run the tests locally against different endpoints. In that case you can create appsettings.{environment}.json for each environment. Then use different build configurations to set which environment you are running. Your code in TestStartup will load the correct appsettings file. On the deployment pipeline you can do as I explained in my previous comment. Please let me know if this helps. – Ink

Answer 1

Refactor the test startup to allow for it to be modified as needed for its test

For example

public class TestStartup : IStartup {
    private readonly string settings;

    public TestStartup(string settings) {
        this.settings = settings;
    }

    public void ConfigureServices(IServiceCollection services) {
       var configuration = new ConfigurationBuilder()
           .SetBasePath(Directory.GetCurrentDirectory())
           .AddJsonFile(settings, false) //<--just an example
           .AddEnvironmentVariables()
           .Build();

        services.AddMvc()
            .SetCompatibilityVersion(version: CompatibilityVersion.Version_2_2);

        //...Code to add required services based on configuration

    }

    public void Configure(IApplicationBuilder app) {
        app.UseMvc();

        //...Code to configure test Startup
    }
}

And have that pattern filter up through the fixture

public class TestServerFixture {
    static readonly Dictionary<string, TestServer> cache = 
        new Dictionary<string, TestServer>();

    public TestServerFixture() {
        //...
    }

    public HttpClient GetClient(string settings) {
        TestServer server = null;
        if(!cache.TryGetValue(settings, out server)) {
            var startup = new TestStartup(settings); //<---
            var builder = new WebHostBuilder()
                .ConfigureServices(services => {
                    services.AddSingleton<IStartup>(startup);
                });
            server = new TestServer(builder);
            cache.Add(settings, server);
        }
        return server.CreateClient();
    }
}

And eventually the test itself

public class MyTest : IClassFixture<TestServerFixture> {
    private readonly TestServerFixture fixture;

    public MyTest(TestServerFixture fixture) {
        this.fixture = fixture;
    }

    [Theory]
    [InlineData("settings1.json")]
    [InlineData("settings2.json")]
    public async Task Should_Execute_Using_Configurations(string settings) {
        var client = fixture.CreateClient(settings);

        //...use client

    }
}

Bonspiel

Hi @Nkosi, this way I would need have to Arrange, Act and Assert twice for each client/server which is exactly what I was trying to avoid. – Alpine

Hi @Nkosi, your updated answer won't work as the settings are different at server level. The test server is already set up in Fixture.. In Fact or Theory would be very late to provide this setting.. – Alpine

@AnkitVijay No it wont. The test server is setup by the fixture when the client is requested. Note it is not being done in the constructor. If it was then I would have agreed with your statement. – Bonspiel

I think one side effect I see of this approach is that for the TestServer would have to be created for each test I have in a single class. This may have some performance implication.. But I should be able to mitigate it by creating static instance of test server for each setting/deployment – Alpine

@AnkitVijay That can be easily fixed with a local cache – Bonspiel

Yes, I realized that and updated my answer.. Thanks a lot for your help. 🙂 – Alpine

Hi @Bonspiel your answer is absolutely fine and works. We however took little bit different approach and I will post my answer and reasoning behind the approach. That said, I have marked your response as answer. – Alpine

Answer 2

@Nkosi's post fits very well with our scenario and my asked question. It's a simple, clean and easy to understand approach with maximum reusability. Full marks to the answer.

However, there were a few reasons why I could not go forward with the approach:

• In the suggested approach we couldn't run tests for only one particular setting. The reason it was important for us as in the future, there could two different teams maintaining their specific implementation and deployment. With Theory, it becomes slightly difficult to run only one setting for all the tests.

• There is a high probability that we may need two separate build and deployment pipelines for each setting/ deployment.

• While the API endpoints, Request, and Response are absolutely the same today, we do not know if it will continue to be the case as our development proceed.

Due to the above reasons we also considered the following two approaches:

Approach 1

Have a common class library which has common Fixture and Tests as abstract class

• Project Common.IntegrationTests

TestStartup.cs

public abstract class TestStartup : IStartup
{
    public abstract IServiceProvider ConfigureServices(IServiceCollection services);

    public void Configure(IApplicationBuilder app)
    {
        app.UseMvc();

        // Code to configure test Startup
    }
}

TestServerFixture.cs

public abstract class TestServerFixture
{

    protected TestServerFixture(IStartup startup)
    {
        var builder = new WebHostBuilder().ConfigureServices(services =>
        {
            services.AddSingleton<IStartup>(startup);
        });

        var server = new TestServer(builder);
        Client = server.CreateClient();
    }

    public HttpClient Client { get; private set; }
}

MyTest.cs

public abstract class MyTest
{
    private readonly TestServerFixture _fixture;

    protected MyTest(TestServerFixture fixture)
    {
        _fixture = fixture;
    }

    [Fact]
    public void ItShouldExecuteTwice_AgainstTwoSeparateConfigurations()
    {
        //...
    }
}

• Project Setting1.IntegrationTests (References Common.IntegrationTests)

TestStartup.cs

public class TestStartup : Common.IntegrationTests.TestStartup
{
    public override IServiceProvider ConfigureServices(IServiceCollection services)
    {
       var configuration = new ConfigurationBuilder()
           .SetBasePath(Directory.GetCurrentDirectory())
           .AddJsonFile("appsettings.json", false) // appsettings for Setting1
           .AddEnvironmentVariables()
           .Build();

        services.AddMvc()
            .SetCompatibilityVersion(version: CompatibilityVersion.Version_2_2);

        // Code to add required services based on configuration


        return services.BuildServiceProvider();
    }
}

TestServerFixture.cs

public class TestServerFixture : Fixtures.TestServerFixture
{
    public TestServerFixture() : base(new TestStartup())
    {
    }
}

MyTests.cs

public class MyTests : Common.IntegrationTests.MyTests, IClassFixture<TestServerFixture>
{
    public MyTests(TestServerFixture fixture) : base(fixture)
    {
    }
}

• Project Setting2.IntegrationTests (References Common.IntegrationTests)

A similar structure as Setting1.IntegrationTests

This approach provided a good balance of reusability and flexibility to run/ modify the tests independently. However, I was still not 100% convinced with this approach as it meant for each common Test class we would need to have an implementation where we are not doing anything other than calling the base constructor.

Approach 2

In the second approach, we took the Approach 1 further and try to fix the issue we had with Approach 1 with Shared Project. From the documentation:

Shared Projects let you write common code that is referenced by a number of different application projects. The code is compiled as part of each referencing project and can include compiler directives to help incorporate platform-specific functionality into the shared code base.

Shared Project gave us the best of both worlds without the ugliness of link files and unnecessary class inheritance or abstraction. Our new set up is as follows:

Edit: I wrote a blog post on this where I have talked about our use-case and the solution in detail. Here is the link:

https://ankitvijay.net/2020/01/04/running-an-asp-net-core-application-against-multiple-db-providers-part-2/

Alpine