Running Integration Testing with WebApplicationFactory throws CancellationTokenSource has been disposed on Test Class Cleanup

Question

I have a few Web Api Controller tests. These are built with the WebApplicationFactory provided by .NET Core Tests.

• When I run these Controller tests one-by-one, they all pass.
• When I run these Controller test all at once (in Visual Studio, right click the Test-project and choose "Run Tests") all the tests fail, except randomly one.

Print-screen 1: all success

Print-screen 2: all failed, except one:

All the tests consists of a [Theory] passing in MemberData to test various security combinations.

7)  Project.UserRolesControllerTests.GetUserDetailsById_ReturnsConfiguredResult
  Duration: 1 ms

  Message: 
    [Test Class Cleanup Failure (Project.Controllers.UserRolesControllerTests)]: System.ObjectDisposedException : The CancellationTokenSource has been disposed.
  Stack Trace: 
    CancellationTokenSource.ThrowObjectDisposedException()
    RedisStorage.Dispose()
    Disposer.Dispose(Boolean disposing) line 38
    Disposable.Dispose() line 32
    LifetimeScope.Dispose(Boolean disposing) line 414
    Disposable.Dispose() line 32
    Container.Dispose(Boolean disposing) line 142
    Disposable.Dispose() line 32
    AutofacServiceProvider.Dispose() line 121
    WebHost.Dispose()
    WebApplicationFactory`1.Dispose(Boolean disposing)
    WebApplicationFactory`1.Dispose()

I have found a Github issue on the Dotnet Runtime project here. However, I'm not sure it's the same issue. Also, I can't seem to get the workaround fixed.

Any one has experience with this? I can run these tests manually for now, but it will also be part of the CI pipeline on DevOps. I persume it will break there as well.

Many thanks in advance.

EDIT: add used code

public class MyCustomWebApplicationFactory<TStartup>
    : WebApplicationFactory<TStartup> where TStartup : class
{
    private UserConfiguration _userConfiguration;

    public void SetUser(UserConfiguration userConfiguration)
    {
        _userConfiguration = userConfiguration;
    }

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        base.ConfigureWebHost(builder);

        Environment.SetEnvironmentVariable("Statistics:RedisDb", "<conn. string>");

        builder.ConfigureTestServices(services =>
        {
            services.AddMvc(options =>
            {
                options.Filters.Add(new AllowAnonymousFilter()); // Remove entire JWT-requirement for all controllers during test
            });
            services.AddScoped<IJwtService>(instance =>
            {
                var customJwtService = new CustomJwtService(() => _userConfiguration);

                return customJwtService;
            });
            services.AddScoped<IHttpRequestHeaderService>(instance =>
            new CustomHttpRequestHeaderService(() => _userConfiguration));
        });

        builder.UseEnvironment("Test");
    }
}

public class CommunityPostsControllerTests : IClassFixture<MyCustomWebApplicationFactory<CommunityPostsController>>
{
    private MyCustomWebApplicationFactory<CommunityPostsController> _factory;

    public CommunityPostsControllerTests(MyCustomWebApplicationFactory<CommunityPostsController> factory)
    {
        _factory = factory;
    }

    public static IEnumerable<object[]> RoleAccess => new List<object[]>
    {
        new object[] { UserConfigurations.AdminCi1, Ci1EmployeePostInGroup, HttpStatusCode.OK},
        new object[] { UserConfigurations.EmployeeCi1, Ci1ParentPostInGroup, HttpStatusCode.Forbidden},
        // Omitted for readability
    };

    [Theory]
    [MemberData(nameof(RoleAccess))]
    public async Task GetPostById_ReturnsConfiguredResult(UserConfiguration userConfiguration, int communityPostId, HttpStatusCode expectedCode)
    {
        // Arrange
        _factory.SetUser(userConfiguration);
        var client = _factory.CreateClient();

        // Act
        var result = await client.GetAsync($"api/v1/communityposts/{communityPostId}", cancellationToken: CancellationToken.None);

        // Assert
        Assert.Equal(expectedCode, result.StatusCode);
    }
}

Edit: CI result

As expected, the Azure DevOps CI pipeline also encounters this issue. Below is the log/result:

Microsoft(R) Test Execution Command Line Tool Version 16.7.0
Copyright (c) Microsoft Corporation.All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
      No XML encryptor configured.Key { 8ad52f98 - 5736 - 4d40 - 85b9 - e758dd2823e4}
may be persisted to storage in unencrypted form.
warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
      No XML encryptor configured. Key {fd541f11-452b-4089-afd4-5f8cf81c43af} may be persisted to storage in unencrypted form.
  [xUnit.net 00:00:46.30][Test Class Cleanup Failure(Project.WebApi.IntegrationTests.Controllers.CommunityPostsControllerTests)] System.ObjectDisposedException
X Project.WebApi.IntegrationTests.Controllers.CommunityPostsControllerTests.GetPostById_ReturnsConfiguredResult [1ms]
Error Message:
   [Test Class Cleanup Failure(Project.WebApi.IntegrationTests.Controllers.CommunityPostsControllerTests)]: System.ObjectDisposedException : The CancellationTokenSource has been disposed.
  Stack Trace:
     at System.Threading.CancellationTokenSource.ThrowObjectDisposedException()
   at Hangfire.Pro.Redis.RedisStorage.Dispose()
   at Autofac.Core.Disposer.Dispose(Boolean disposing) in / home / appveyor / projects / autofac / src / Autofac / Core / Disposer.cs:line 38
   at Autofac.Util.Disposable.Dispose() in / home / appveyor / projects / autofac / src / Autofac / Util / Disposable.cs:line 32
   at Autofac.Core.Lifetime.LifetimeScope.Dispose(Boolean disposing) in / home / appveyor / projects / autofac / src / Autofac / Core / Lifetime / LifetimeScope.cs:line 414
   at Autofac.Util.Disposable.Dispose() in / home / appveyor / projects / autofac / src / Autofac / Util / Disposable.cs:line 32
   at Autofac.Core.Container.Dispose(Boolean disposing) in / home / appveyor / projects / autofac / src / Autofac / Core / Container.cs:line 142
   at Autofac.Util.Disposable.Dispose() in / home / appveyor / projects / autofac / src / Autofac / Util / Disposable.cs:line 32
   at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.Dispose() in / home / appveyor / projects / autofac - extensions - dependencyinjection / src / Autofac.Extensions.DependencyInjection / AutofacServiceProvider.cs:line 121
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.Dispose()
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.Dispose(Boolean disposing)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.Dispose()
Results File: /home/vsts/work/_temp/_fv-az6-181_2020-11-02_12_27_22.trx

Test Run Failed.
Total tests: 49
     Passed: 48
     Failed: 1

EDIT: Package configs

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.App" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="2.1.3" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.7.1" />
    <PackageReference Include="xunit" Version="2.4.1" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.3">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="coverlet.collector" Version="1.3.0">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>
</Project>

Weingarten

How does your tests look like? – Fourier

I've updated the post with usage. – Weingarten

Can you update to a newer version of xUnit? Maybe it is a known issue and already fixed... – Rascon

@Rascon Sadly not... I added packages & versions to the post. – Weingarten

Why are you passing CommunityPostsController into the generic of MyCustomWebApplicationFactory? This should be your Startup class and not your controller. I'm not sure if that's the problem, but you should probably update that. – Cypriot

Answer 1

IAsyncDisposable support was added to WebApplicationFactory on Jun 9, 2021. I think this means you can call DisposeAsync() instead of Dispose(), which, at least for me, seems to solve the problem.

Before:

[ClassCleanup]
public static void ClassCleanup()
{
    _webApplicationFactory.Dispose();
}

After:

[ClassCleanup]
public static async Task ClassCleanup()
{
    await _webApplicationFactory.DisposeAsync();
}

Misfeasor

[ClassCleanup] is not available anymore – Grape

Answer 2

In this case I added the IAsyncLifetime interface to the test class and implemented:

public async Task DisposeAsync()
{
   await _webApplicationFactory.DisposeAsync();   
}

prevents the error to happen. A nice explanation can be found async lifetime with xunit

Grape

Answer 3

One reason this can happen when running XUnit is that XUnit runs all tests in parallel (except the one in the same class or collection). WebApplicationFactory can mess this up a bit.

A solution is to mark all your test classes with a collection with the same name. This means that all test classes that exists in the same collection with that name not will run in parallel.

[Collection("DisableParallelRun")]
public class YourTestClass

Source: https://xunit.net/docs/running-tests-in-parallel

Note! This will of course make your tests run slower since they now doesn't run in parallel...

Dessertspoon