EF Core internal caching and many DbContext types during testing

Question

I have many test classes, and each has dozens of tests. I want to isolate tests, so instead of a mega context MyDbContext, I use MyDbContextToTestFoo, MyDbContextToTestBar, MyDbContextToTestBaz, etc. So I have MANY DbContext subclasses.

In my unit tests with EF Core 5 I'm running into the ManyServiceProvidersCreatedWarning. They work individually, but many fail when run as a group:

System.InvalidOperationException : An error was generated for warning 'Microsoft.EntityFrameworkCore.Infrastructure.ManyServiceProvidersCreatedWarning': More than twenty 'IServiceProvider' instances have been created for internal use by Entity Framework. This is commonly caused by injection of a new singleton service instance into every DbContext instance. For example, calling 'UseLoggerFactory' passing in a new instance each time--see https://go.microsoft.com/fwlink/?linkid=869049 for more details. This may lead to performance issues, consider reviewing calls on 'DbContextOptionsBuilder' that may require new service providers to be built. This exception can be suppressed or logged by passing event ID 'CoreEventId.ManyServiceProvidersCreatedWarning' to the 'ConfigureWarnings' method in 'DbContext.OnConfiguring' or 'AddDbContext'.

I don't do anything weird with DbContextOptionsBuilder as that error suggests. I don't know how to diagnose "...that may require new service providers to be built". In most tests I create a context normally: new DbContextOptionsBuilder<TContext>().UseSqlite("DataSource=:memory:") where TContext is one of the context types I mentioned above.

I've read many issues on the repo, and discovered that EF does heavy caching of all sorts of things, but docs on that topic don't exist. The recommendation is to "find what causes so many service providers to be cached", but I don't know what to look for.

There are two workarounds:

• builder.EnableServiceProviderCaching(false) which is apparently very bad for perf
• builder.ConfigureWarnings(x => x.Ignore(CoreEventId.ManyServiceProvidersCreatedWarning)) which ignores the problem

I assume that "service provider" means EF's internal IoC container.

What I want to know is: does the fact that I have many DbContext types (and thus IModel types), affect service provider caching? Are the two related? (I know EF caches an IModel for every DbContext, does it also cache a service provider for each one?)

Yazbak

These aren't unit tests if you're interacting with a DbContext class - that doesn't mean they're not valid, just that they're not uint tests – Lebanon

@PaulKeister while I agree with you in principle, EF core provides and in memory DB context abstraction for testing and I think that confuses a lot of people. My way is still to use mocks. – Buckels

It seems like disabling caching is the way to go, performance be damned. Because although tests should be very fast, test isolation is even more important. – Buckels

I wonder what green tests mean for the context you're using in the application code. It sounds like the test environment and the application are two parallel universes. – Hypogastrium

@AluanHaddad Indeed my thoughts too. In that repo issue I linked others have the same opinion. There are too many static/cached things in EF, and so to get peace of mind I prefer isolation. – Yazbak

Answer 1

Service provider caching is purely based on the context options configuration - the context type, model etc. doesn't matter.

In EF Core 5.0, the key according to the source code is

static long GetCacheKey(IDbContextOptions options) => options.Extensions
    .OrderBy(e => e.GetType().Name)
    .Aggregate(0L, (t, e) => (t * 397) ^ ((long)e.GetType().GetHashCode() * 397) ^ e.Info.GetServiceProviderHashCode());

while in EF Core 6.0 the key is the options instance with overridden Equals method having similar semantics.

So something is different in the options you are using - either initially or after OnConfiguring call if you are overriding it and modifying the options inside. This is what you need to figure out (in 5.0 you could use the above method to check the key, in 6.0 you could use some static field for storing the first options instance and use it to check with Equals the next ones).

Note that EF Core caches both original and after OnConfiguring call options, so all they count. The code generating the warning btw is in the same place (class) - source.

Zeculon

Thank you for providing more info about this topic than even the official docs. And also, for advising a way to diagnose the problem! – Yazbak

You are welcome. I guess this is related to your previous questions, for instance retrieving the context options from the context (these are already configured and cached, hence should not be altered), or replacing EFC internal services with your own with additional dependencies etc. If you wish, please post a question with the real problem domain you are trying to solve, in order to be able to see how it could be solved w/o interfering with EFC internals. – Zeculon

We have some tools for the backend that I'm trying to simplify, and I always get lost in the "advanced topics" for EF. See here for all the things they did not yet document. They are doing a great job, but the non-standard stuff is complicated and undocumented. They should hire you! Thanks again and Happy New Year. – Yazbak

@IvanStoev I just wanted to say thank you. Your answer helped me resolve the same issue. In my case, I was providing a certificate callback (postgres) and even though it was the same cert, using a lambda for the callback was causing the cache key to change. Moving to a singleton delegate containing the singleton cert fixed it. Thanks again. – Fathometer

Answer 2

@IvanStoev : thank you! Your answer helped my to find the cause of my problem: I am using an interceptor for initializing some properties of the entities created by EF. I was using a new instance of my interceptor for each dbContext. Transforming it into a singleton fixed it!

optionsBuilder.AddInterceptors(new ServiceProviderInterceptor());
=>
optionsBuilder.AddInterceptors(ServiceProviderInterceptor.Instance);

Terricolous

Answer 3

Expanding on the answer from @ivan-stoev - this can happen with Npgsql if you always use a new NpgsqlDataSourceBuilder.

Before:

services.AddDbContext<DevaQueueDbContext>((options) => { 
    var dbDataSourceBuilder = new NpgsqlDataSourceBuilder(configuration.GetConnectionString("Database"));
    options.UseNpgsql(dbDataSourceBuilder.Build());
});

After:

var dbDataSource = new NpgsqlDataSourceBuilder(configuration.GetConnectionString("Database")).Build();

services.AddDbContext<DevaQueueDbContext>((options) => { 
    options.UseNpgsql(dbDataSource);
});

Procarp

Answer 4

Please upvote abou's answer. It put me onto the "look at your interceptors".

My contribution:

So instead of rolling my own singleton, I used the DotNet IoC.

using System;
using System.Collections.Generic;
using System.Data;
using System.Diagnostics.CodeAnalysis;
using System.Linq;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Diagnostics;


    public class MyCoolDbContext : DbContext
    {

        /* SOF answer primary item */
        public const string ErrorMessageIEnumerableIInterceptorIsNull = "IEnumerable<IInterceptor> is null";

        private readonly IEnumerable<IInterceptor> interceptors;



        public MyCoolDbContext(
            ILoggerFactory loggerFactory,

            /* SOF answer primary item */
            IEnumerable<IInterceptor> interceptors,


            DbContextOptions<MyCoolDbContext> options)
            : base(options)
        {

            
            /* SOF answer primary item */
            this.interceptors = interceptors ??
                                throw new ArgumentNullException(
                                    ErrorMessageIEnumerableIInterceptorIsNull,
                                    (Exception)null);

        }


        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            base.OnConfiguring(optionsBuilder);

            /* SOF answer primary item */
            if (this.interceptors.Any())
            {
                optionsBuilder.AddInterceptors(this.interceptors);
            }


        }
    }

Then your IoC/DI registrations:

//using Microsoft.EntityFrameworkCore.Diagnostics;
// using Microsoft.Extensions.DependencyInjection;

//      services is an     IServiceCollection

        services.AddSingleton<IInterceptor>(_ => new MyCustomInterceptorOne());

Even if you register just one (as above), the IoC will pick it up as part of the IEnumerable

But.. you have also followed the "open/closed" SOLID principle and you can do this .. without changing your MyCoolDbContext

        services.AddSingleton<IInterceptor>(_ => new MyCustomInterceptorOne());

        services.AddSingleton<IInterceptor>(_ => new MyCustomInterceptorTwo());

        services.AddSingleton<IInterceptor>(_ => new MyCustomInterceptorThree());

This is working for me with:

<PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
</PropertyGroup>

<ItemGroup>
    <PackageReference Include="Microsoft.EntityFrameworkCore" Version="6.0.5" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Relational" Version="6.0.5" />
</ItemGroup>

Honolulu