Replacing a entity collection in Entity Framework Core causes DbContext to fetch the new values when not saved to db. How to reload the collection?

Question

What I can't understand fully is exemplified in a unit test below. Basically I want to know why DbContext overrides collection values currently stored in the database even if Reload() and a new load for the collection has been called.

If I try to call dbContext.Entry(test.TestChildren).Reload(); I get the exception:

System.InvalidOperationException: 'The entity type 'List' was not found. Ensure that the entity type has been added to the model.'

I have also loaded TestChildren explicitly via dbContext.Entry(test).Collection(x => x.TestChildren).Load(); before calling Assert.

https://learn.microsoft.com/en-us/ef/core/querying/related-data/explicit#explicit-loading

I can get it working by creating a new context using (var context = new ApplicationDbContext(dbContextOptions)) and then execute a load on related entities.

However according to Microsoft Docs:

Queries are always executed against the database even if the entities returned in the result already exist in the context.

https://learn.microsoft.com/en-us/ef/core/querying/

The actual values are saved in LocalDb and SQLite file, SQLite in-memory and EF in-memory database looks correct.

https://learn.microsoft.com/en-us/ef/core/testing/sqlite

LocalDb after assert has failed:

EFC 3.1.9 with .NET Core 3.1:

EFC 5.0.2 with .NET 5:

As shown I can get simple properties to load correctly with Reload(). I have tried the solution from @IvanStoev to reload collections but it does not work for me.

https://mcmap.net/q/412022/-how-to-reload-collection-in-ef-core-2-x

I know that most examples have a new using statement per database call but it is normally not a problem to do several calls with one context and Microsoft has several examples with that as well.

https://learn.microsoft.com/en-us/ef/core/get-started/overview/first-app?tabs=netcore-cli#create-read-update--delete

My theory is that even though the values are in the database and then fetched DbContext still overrides with the tracked entities because they are probably marked as Modified. This would also explain why EntityEntry.Reload method sets EntityState to Unchanged.

https://learn.microsoft.com/en-us/dotnet/api/microsoft.entityframeworkcore.changetracking.entityentry.reload?view=efcore-3.1

https://learn.microsoft.com/en-us/dotnet/api/microsoft.entityframeworkcore.entitystate?view=efcore-3.1#Microsoft_EntityFrameworkCore_EntityState_Unchanged

https://learn.microsoft.com/en-us/dotnet/api/microsoft.entityframeworkcore.changetracking.entityentry.collection?view=efcore-3.1

My three questions:

1. How can I reload the collection for the entity? The answers I have already tried does not work.

https://mcmap.net/q/402498/-reload-an-entity-and-all-navigation-property-association-dbset-entity-framework https://mcmap.net/q/412022/-how-to-reload-collection-in-ef-core-2-x

2. Is my theory correct or why are values not updated when a new database call is made with Collection(x => x.TestChildren).Load()?

3. Is there anyway to disable the need to use Reload() behavior or can that have dire consequences? I have seen examples with dbContext.ChangeTracker.QueryTrackingBehavior = Microsoft.EntityFrameworkCore.QueryTrackingBehavior.NoTracking; but I have not dared to try it in production systems. I don't want to use Lazy loading.

https://learn.microsoft.com/en-us/ef/core/querying/related-data/lazy

https://mcmap.net/q/321899/-how-to-refresh-an-entity-framework-core-dbcontext

I have read a different question about a proper way to replace collection in one to many relationship in Entity Framework but I have not found an answer to my question there.

https://mcmap.net/q/412023/-entity-framework-proper-way-to-replace-collection-in-one-to-many

It is a bit similar to the question on how to refresh an Entity Framework Core DBContext but I would like to know why it happens and there is also no example with a collection.

How to refresh an Entity Framework Core DBContext?

To explain and keep it as simple as possible I have a model very similar to the Getting Started with EF Core model.

https://learn.microsoft.com/en-us/ef/core/get-started/overview/first-app?tabs=netcore-cli#create-read-update--delete

ApplicationDbContext:

public class ApplicationDbContext : DbContext
{
    public ApplicationDbContext(
        DbContextOptions options) : base(options)
    {
    }

    public DbSet<Test> Tests { get; set; }

    public DbSet<TestChild> TestChildren { get; set; }

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
    {
        base.OnConfiguring(optionsBuilder);
    }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        base.OnModelCreating(modelBuilder);
    }
}

Test model:

public class Test
{
    [DatabaseGenerated(DatabaseGeneratedOption.Identity)]
    public int Id { get; set; }

    public string Name { get; set; }

    public List<TestChild> TestChildren { get; set; } = new List<TestChild>();
}

public class TestChild
{
    [DatabaseGenerated(DatabaseGeneratedOption.Identity)]
    public int Id { get; set; }

    public string Name { get; set; }

    public int TestId { get; set; }

    public Test Test { get; set; }
}

UnitTest1:

using System;
using Xunit;
using EFCoreTest;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Infrastructure;
using System.Linq;
using System.Data.Common;
using Microsoft.Data.Sqlite;
using Microsoft.EntityFrameworkCore.ChangeTracking;

namespace XUnitTestEFCore
{
    public static class Extensions
    {
        public static void Reload(this CollectionEntry source)
        {
            if (source.CurrentValue != null)
            {
                foreach (var item in source.CurrentValue)
                    source.EntityEntry.Context.Entry(item).State = EntityState.Detached;
                source.CurrentValue = null;
            }
            source.IsLoaded = false;
            source.Load();
        }
    }

    public class UnitTest1
    {
        [Fact]
        public void TestSqliteFile()
        {
            var dbContextOptions = new DbContextOptionsBuilder<ApplicationDbContext>()
                .UseSqlite("Data Source=test.db")
                .Options;

            var dbContext = new ApplicationDbContext(dbContextOptions);

            TestSingleProprtyMethod(dbContext, dbContextOptions);
            TestCollectionMethod(dbContext, dbContextOptions);
        }

        [Fact]
        public void TestSqliteInMemory()
        {
            var dbContextOptions = new DbContextOptionsBuilder<ApplicationDbContext>()
                .UseSqlite(CreateInMemoryDatabase())
                .Options;

            var connection = RelationalOptionsExtension.Extract(dbContextOptions).Connection;

            var dbContext = new ApplicationDbContext(dbContextOptions);

            TestCollectionMethod(dbContext, dbContextOptions);
        }

        [Fact]
        public void TestEFInMemoryDatabase()
        {
            var dbContextOptions = new DbContextOptionsBuilder<ApplicationDbContext>()
                             .UseInMemoryDatabase(Guid.NewGuid().ToString())
                             .Options;

            var dbContext = new ApplicationDbContext(dbContextOptions);

            TestCollectionMethod(dbContext, dbContextOptions);
        }

        [Fact]
        public void TestLocalDb()
        {
            var dbContextOptions = new DbContextOptionsBuilder<ApplicationDbContext>()
                             .UseSqlServer("Server=(localdb)\\mssqllocaldb;Database=XUnitTestEFCore;Trusted_Connection=True;MultipleActiveResultSets=true")
                             .Options;

            var dbContext = new ApplicationDbContext(dbContextOptions);

            TestCollectionMethod(dbContext, dbContextOptions);
        }

        private static DbConnection CreateInMemoryDatabase()
        {
            var connection = new SqliteConnection("Data Source=:memory:");

            connection.Open();

            return connection;
        }

        private void TestCollectionMethod(ApplicationDbContext dbContext, DbContextOptions<ApplicationDbContext> dbContextOptions)
        {
            dbContext = new ApplicationDbContext(dbContextOptions);

            Seed(dbContext);
            //Works - Nothing in Db
            Assert.Null(dbContext.Tests.FirstOrDefault());

            dbContext.SaveChanges();

            //Works - First item in Db
            Assert.NotNull(dbContext.Tests.FirstOrDefault());

            //Works - TestChildren are correctly added
            Assert.Equal(2, dbContext.Tests.FirstOrDefault().TestChildren.Count);

            using (var context = new ApplicationDbContext(dbContextOptions))
            {
                var usingTest = context.Tests.FirstOrDefault();

                context.Entry(usingTest)
                    .Collection(x => x.TestChildren)
                    .Load();

                //Works
                Assert.Equal(2, usingTest.TestChildren.Count);
            }

            var test = dbContext.Tests.FirstOrDefault();
            test.TestChildren = test.TestChildren.Where(x => x.Id == 1).ToList();

            var count = dbContext.Entry(test)
                .Collection(x => x.TestChildren)
                .Query()
                .Count();
            //Works - Two items
            Assert.Equal(2, count);

            //Works - Two items
            Assert.Equal(2, dbContext.TestChildren.Where(x => x.TestId == 1).Count());

            using (var context = new ApplicationDbContext(dbContextOptions))
            {
                //Works - Two items
                Assert.Equal(2, context.TestChildren.Where(x => x.TestId == 1).Count());
            }

            using (var context = new ApplicationDbContext(dbContextOptions))
            {
                var usingTest = context.Tests.FirstOrDefault();

                context.Entry(usingTest)
                    .Collection(x => x.TestChildren)
                    .Load();

                //Works - Two items
                Assert.Equal(2, usingTest.TestChildren.Count);
            }

            dbContext.Entry(test).Reload();
            dbContext.Entry(test)
                .Collection(x => x.TestChildren)
                .Load();

            //Source - https://mcmap.net/q/412022/-how-to-reload-collection-in-ef-core-2-x
            dbContext.Entry(test).Collection(x => x.TestChildren).Reload();
            //dbContext.Entry(test.TestChildren).Reload(); Causes exception System.InvalidOperationException: 'The entity type 'List<TestChild>' was not found. Ensure that the entity type has been added to the model.'
            //Only one TestChild left even though no save has been performed, test is Reloaded and TestChildren has been explicitly loaded
            Assert.Equal(2, test.TestChildren.Count);
        }

        private void TestSingleProprtyMethod(ApplicationDbContext dbContext, DbContextOptions<ApplicationDbContext> dbContextOptions)
        {
            Seed(dbContext);

            Assert.Null(dbContext.Tests.FirstOrDefault());

            dbContext.SaveChanges();

            Assert.NotNull(dbContext.Tests.FirstOrDefault());

            var test2 = dbContext.Tests.FirstOrDefault();
            test2.Name = "test2";

            //Will be test2
            Assert.NotEqual("test1", dbContext.Tests.FirstOrDefault().Name);
            dbContext.Entry(test2).Reload();
            Assert.Equal("test1", dbContext.Tests.FirstOrDefault().Name);

            using (var context = new ApplicationDbContext(dbContextOptions))
            {
                Assert.Equal("test1", context.Tests.FirstOrDefault().Name);
            }
        }

        private void Seed(ApplicationDbContext dbContext)
        {
            dbContext.Database.EnsureDeleted();
            dbContext.Database.EnsureCreated();

            var test1 = new Test()
            {
                Name = "test1"
            };

            var testChild1 = new TestChild()
            {
                Name = "testChild1"
            };

            var testChild2 = new TestChild()
            {
                Name = "testChild2"
            };

            test1.TestChildren.Add(testChild1);
            test1.TestChildren.Add(testChild2);

            dbContext.Tests.Add(test1);
        }
    }
}

Giarla

Have you tried to debug that? :) After few hours of fail I'm starting debugging this monster https://mcmap.net/q/412024/-debug-net-core-source – Thermoelectricity

@SvyatoslavDanyliv Nope I'm not that far down the rabbit hole yet. :) – Giarla

The problem is that after this code test.TestChildren = test.TestChildren.Where(x => x.Id == 1).ToList(); the change tracker marks other items from the original collection as Deleted. My code for detaching cannot find them (they are not in the collection), and EFC removes them from any tracking query (client wins). Currently I'm not sure how to search the change tracker for entries which belong (currently or originally) to that collection. – Glanders

@IvanStoev Yes I suspected it had something to do with that. Would be good to be able to clear the change tracker of those entries but if that is not possible I will have to create a new Context. – Giarla

@Giarla Are you sure this is a valid use case? The aforementioned code is equivalent of dbContext.RemoveRange(test.TestChildren.Where(x => x.Id != 1), i.e. you are telling EFC that these items have to be deleted on SaveChanges, and "deleted" items are removed/not included in collections. So you are basically trying to "undo" that operation? EFC does not provide such service. In theory, setting CollectionEntry.IsModified = false; should do that, but in practice it's not implemented that way. Byw, what EFC version are you targeting (because change tracking behavior changes over versions)? – Glanders

@IvanStoev Correct this is an "undo" for those changes. I can absolutely work around it but If possible I would like to know how it can be done. This was using EFC 3.1.9 with .NET Core 3.1. The results were exactly the same using EFC 5.0.2 with .NET 5.0 though. – Giarla

Answer 1

The recommended way for dealing with changing data state is to keep the DbContext lifespan short, so dispose and use a new DbContext.

As for reloading a collection a DbContext is pretty "clingy" when it comes to navigation properties and collections. This occurs when making tracked changes that are not committed and trying to revert to data state, or trying to update DbContext state to reflect changes that may have been done in the database. Load and Reload work for refreshing fields, but when it comes to collections/references this isn't reliable.

To refresh a collection on an entity, about the only reliable option I have seen work outside of simply keeping DbContext lifespans short is something like this:

foreach(var child in test.TestChildren)
    context.Entry(child).State = EntityState.Detached;       
context.Entry(test).State = EntityState.Detached;
test.TestChildren.Clear();
context.Tests.Attach(test);

context.Entry(test).Collection(x => x.TestChildren).Query().Load();

We detach all of the children to stop the DbContext from tracking them, then also temporarily detach the parent from the DbContext as well. This is so that we can clear any object references from the parent's children collection. We then re-attach and tell the DbContext to reload the children. From there, test.TestChildren will reflect the persisted data state, so any changes made prior to reloading it are gone, and any database side changes since it was last loaded are brought up to date.

As a general rule with EF collections you should always avoid replacing the collection reference, instead adding, removing, and clearing via the EF proxies when tracking is happening.

I.e.

//(not good)
test.TestChildren = test.TestChildren.Where(x => x.Id == 1).ToList();

//(good)
var childrenToRemove = test.TestChildren.Where(x => x.Id != 1).ToList();
foreach(var child in childrenToRemove)
    test.TestChildren.Remove(child);

Replacing collection references can lead to all kinds of unexpected behavior and will vary depending on whether any/all have been eager loaded / present in the DbContext cache. In the case of unit/integration test code we can break rules for a fixed, known state. Detaching the parent first helps avoid tracked state changes against the set.

Charlena