Asp.Net Core HealthChecks Concurrent Entity Framework DbContext Errors

Question

I read this: https://github.com/dotnet/aspnetcore/issues/14453 and it seems i have the same problem.

I have around 50 projects in my solution and each has their own RegisterServices.cs which defines the db context

 services.AddDbContextPool<Db.ACME.Context>(
          options => options.UseSqlServer(configuration.GetConnectionString("ACME"))); 

In there I also add the healthchecks but as soon as I add more than 1 e.g. in the project "PROJECTS" one for checking duplicate projectnames and one for checking for checking duplicate task names.

services.AddHealthChecks()
           .AddCheck<DuplicateTaskNamesHealthCheck>("Duplicate Task Names", HealthStatus.Unhealthy,
               tags: new[] { "org_project" });

        services.AddHealthChecks()
           .AddCheck<DuplicateProjectNamesHealthCheck>("Duplicate Project Names", HealthStatus.Unhealthy,
               tags: new[] { "org_project" });

Where the Healthcheck simply calls the service "projectservice" and the other "taskservice" (both in the same vs solution project)

It will fail with the known

A second operation was started on this context before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913.

The Healthchecks themselves do not contain any logic but simply call the specific service where the logic (EF linq queries) resides. When I comment either one out, they will work.

It seems the only solution is to copy functionality from the services inside the healthchecks but the disturbing thing is that this is not really handy. The more healthchecks, the more methods will be copied over and DRY is out of the window.

Is there any other known workaround?

---

example of a method called in the project service that fails

public async Task<List<string>> GetDuplicateProjectNamesAsync()
    {
        IQueryable<IGrouping<string, PROJECT>> DuplicateProjectNames = _context
            .PROJECT
            .AsNoTracking()
            .GroupBy(x => x.PROJECTNAME)
            .Where(x => x.Count() > 1);
        
        /* next line fails */
        var hasItems = await DuplicateProjectNames.AnyAsync();

        if (hasItems)
        {
            return await DuplicateProjectNames.Select(x=>x.Key).ToListAsync();
        }
        else 
        { 
            return new List<string>();
        }
    }

---

example of the healthcheck

 namespace ACME.Org.Project.Healthchecks
 { 
public class DuplicateProjectNamesHealthCheck : IHealthCheck
{
    public IACME6_PROJECT _ACME6_PROJECT;
    public Settings _settings;

    public DuplicateProjectNamesHealthCheck(IACME6_PROJECT PROJECT, IOptions<Settings> settings)
    {
        _ACME6_PROJECT = PROJECT;
        _settings = settings.Value;
    }

    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        List<string> duplicateProjectNames = await _ACME6_PROJECT.GetDuplicateProjectNamesAsync();

        if (duplicateProjectNames.Count > 0)
        {
            if (_settings.AutoFixDuplicateProjectNames)
            {                    
                await _ACME6_PROJECT.FixDuplicateProjectNamesAsync();                    
            }
            else
            {
                string errorMessage = "Duplicate Project Names found: ";
                foreach (string projectName in duplicateProjectNames)
                {
                    errorMessage += projectName + ") |";
                }
                errorMessage += "To autofix this set AutoFixDuplicateProjectNames to true in settings.";
                return new HealthCheckResult(status: context.Registration.FailureStatus, errorMessage);
            }
        }
        return HealthCheckResult.Healthy("OK");
    }
}

}

---

and the annotated line (see above) 236 complete error message as requested:

System.InvalidOperationException: A second operation was started on this context before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913.
   at Microsoft.EntityFrameworkCore.Internal.ConcurrencyDetector.EnterCriticalSection()
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
   at ACME.Org.Project.Services.ACME6.ACME6_PROJECT.GetDuplicateProjectNamesAsync() in E:\ACME\repo\acme-7-api\ACME.Org.Project\Services\ACME6\ACME6_PROJECT\ACME6_PROJECT.cs:line 236
   at ACME.Org.Project.Healthchecks.DuplicateProjectNamesHealthCheck.CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken) in E:\ACME\repo\acme-7-api\ACME.Org.Project\Healthchecks\DuplicateProjectNamesHealthCheck.cs:line 23
   at Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService.RunCheckAsync(IServiceScope scope, HealthCheckRegistration registration, CancellationToken cancellationToken)
Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService: Error: Health check "Duplicate Project Names" threw an unhandled exception after 562.888ms

Cumming

What are those custom healtchecks doing? There's no DuplicateTaskNamesHealthCheck or DuplicateProjectNamesHealthCheck in ASP.NET Core. What is the actual full exception text? Not just the message. Post the full text returned by Exception.ToString(). This will show where the error actually occurred, in what method and what calls led to it. – Commutation

simply call the specific service where the logic (EF linq queries) resides. post that code as well. Obviously it's not so simple, and two threads end up trying to use the same DbContext. Is that service a Singleton perhaps, using the same DbContext instance for all calls? That's a bug – Commutation

the project service that fails is that service a Singleton? – Commutation

i have added the complete healthcheck and the method it calls AND the place it errors out with the "a second operation has started". And as far as I can see this is what is described here: github.com/dotnet/aspnetcore/issues/14453 :: which probably means giving each healthcheck a factory dbcontext and thus its own methods instead of referencing existing services. – Cumming

You didn't add the actual exception. Again is the service a singleton? This is a bug in the code, not a bug in EF Core or Health Check – Commutation

No I have no singletons in the asp.net project ALL are defined as services.AddTransient(typeof(IACME_PROJECT), typeof(ACME_PROJECT)); – Cumming

Where is the full error? The error message is clear - you tried to use the same DbContext instance from different threads. Yoou need to find where that happens and fix it – Commutation

If you look in the text above you see the error it gives. "this fails with System.InvalidOperationException: A second operation was started on this context before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see go.microsoft.com/fwlink/?linkid=2097913." – Cumming

@PanagiotisKanavos I understand this, I understand the .core ef requires to await every operation since it is not thread safe. however, this is the case. until you add a second healthservice on the same context. – Cumming

Post the actual full error text. Not just the message. Post the actual full result of Exception.ToString(). That contains the source file location and the stack trace. This is SOP for any debug question. That will tell you where you need to create a new DbContext. – Commutation

As for why this error happens, one of the services is a singleton. Perhaps it's the health check itself – Commutation

from the healthservice i await the call in the specific service. in the service i await the call using await ... anyasync and even add asnotracking. – Cumming

Which actually makes it easier to call the same DbContext from multiple threads concurrently. If the health check is a singleton, the PROJECT service and the DbContext it contains will remain in scope forever and cause exactly this problem – Commutation

Did you register DuplicateTaskNamesHealthCheck as a singleton? That's the recommendation – Commutation

Can well be but i posted the exact lines in which i add the healthchecks above and would not know how to do this else than "services.AddHealthChecks() .AddCheck" etc as in docs – Cumming

@PanagiotisKanavos I posted above the exact lines – Cumming

Answer 1

The reason for this is that AddCheck will create and use a single instance of the healthcheck type

return builder.Add(new HealthCheckRegistration(name,
     s => ActivatorUtilities.GetServiceOrCreateInstance<T>(s), 
     failureStatus, tags, timeout));

ActivatorUtilities.GetServiceOrCreateInstance<T>(s) will try to retrieve an instance of the type from DI or creates a new one. In either case, this means that DuplicateTaskNamesHealthCheck behaves as a singleton and any DbContext instance that gets injected into it will remain active until shutdown.

To solve this, the health check's constructor should accept either IServiceProvider or IDbContextFactory and create context instances whenever needed.

For example:

public class ExampleHealthCheck : IHealthCheck
{
   private readonly IDbContextFactory<ApplicationDbContext> _contextFactory;

   public ExampleHealthCheck(IDbContextFactory<ApplicationDbContext> contextFactory)
   {
       _contextFactory = contextFactory;
   }

   public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context,
        CancellationToken cancellationToken = default(CancellationToken))
   {
       using (var context = _contextFactory.CreateDbContext())
       {
           // ...
       }
   }
}

IServiceProvider should be used to use a scoped or transient service like IACME6_PROJECT. IServiceProvider can be used to either retrieve transient services directly or create a scope to work with scoped services :

public class DuplicateProjectNamesHealthCheck : IHealthCheck
{
    public IServiceProvider _serviceProvider;
    public Settings _settings;

    public DuplicateProjectNamesHealthCheck(IServiceProvider serviceProvider, IOptions<Settings> settings)
    {
        _serviceProvider=serviceProvider;
        _settings = settings.Value;
    }

    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context,
        CancellationToken cancellationToken = default(CancellationToken))
    {
        using (var scope = _serviceProvider.CreateScope())
        {
            var project = scope.ServiceProvider.GetRequiredService< IACME6_PROJECT>();

            var duplicateProjectNames = await _ACME6_PROJECT.GetDuplicateProjectNamesAsync();
            ...
        }
    }

Commutation

so #50200700 "You shouldn't ever have to inject neither scope factory nor service provider into your classes, except for a few rare infrastructure cases " means "this is a rare infrastructure case" except that healthchecks are not rare but rather regular used things. – Cumming