When called from 'VisitLambda', rewriting a node of type 'System.Linq.Expressions.ParameterExpression' must return a non-null value of the same type

Question

I upgraded my data layer classlibrary project from .NETCore 2.2 to .NETCore3.1 and also upgraded the Nuget packages like EntityFrameWork Core from 2.2.0 to EFCore 3.1.0.

I then on validating the following code got an error as mentioned below:

Code:

private async Task<List<CountryDTO>> GetCountriesDataAsync(int languageId)
{
    int pageNo = 1, pageSize = 100;
    var images = await _cacheService.GetAllAsync("imagesCacheKey");
    return await _dbContext.Countries
                    .Where(cc => cc.IsPublished.Equals(true) 
                                     && cc.LanguageId.Equals(languageId))
                    .Select(co => new CountryDTO
                            {  Uuid = co.CountryId, 
                               PNGImagePath = images.FirstOrDefault(im => im.ImageId.Equals(co.PNGImageId))
                                                    .FilePath, 
                                SVGImagePath = images.FirstOrDefault(im => im.ImageId.Equals(co.SVGImageId))
                                                    .FilePath, 
                          DisplayName = co.DisplayName, 
                          DisplayNameShort = co.DisplayName, 
                          Name = Helper.ReplaceChars(co.DisplayName), 
                          Path = Helper.ReplaceChars(co.DisplayName), 
                          CompleteResponse = true})
                   .Skip((pageNo - 1) * 100)
                   .Take(pageSize)
                   .ToListAsync();
}

Error:

When called from 'VisitLambda', rewriting a node of type 'System.Linq.Expressions.ParameterExpression' must return a non-null value of the same type. Alternatively, override 'VisitLambda' and change it to not visit children of this type.
 ---> System.InvalidOperationException: When called from 'VisitLambda', rewriting a node of type 'System.Linq.Expressions.ParameterExpression' must return a non-null value of the same type. Alternatively, override 'VisitLambda' and change it to not visit children of this type.
   at System.Linq.Expressions.ExpressionVisitor.VisitAndConvert[T](T node, String callerName)
   at System.Dynamic.Utils.ExpressionVisitorUtils.VisitParameters(ExpressionVisitor visitor, IParameterProvider nodes, String callerName)
   at System.Linq.Expressions.ExpressionVisitor.VisitLambda[T](Expression`1 node)
   at Microsoft.EntityFrameworkCore.Cosmos.Query.Internal.CosmosProjectionBindingExpressionVisitor.Visit(Expression expression)
   at System.Dynamic.Utils.ExpressionVisitorUtils.VisitArguments(ExpressionVisitor visitor, IArgumentProvider nodes)
   at System.Linq.Expressions.ExpressionVisitor.VisitMethodCall(MethodCallExpression node)
   at Microsoft.EntityFrameworkCore.Cosmos.Query.Internal.CosmosProjectionBindingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Cosmos.Query.Internal.CosmosProjectionBindingExpressionVisitor.Visit(Expression expression)
   at Microsoft.EntityFrameworkCore.Cosmos.Query.Internal.CosmosProjectionBindingExpressionVisitor.VisitMember(MemberExpression memberExpression)
   at Microsoft.EntityFrameworkCore.Cosmos.Query.Internal.CosmosProjectionBindingExpressionVisitor.Visit(Expression expression)
   at Microsoft.EntityFrameworkCore.Cosmos.Query.Internal.CosmosProjectionBindingExpressionVisitor.VisitMemberAssignment(MemberAssignment memberAssignment)
   at Microsoft.EntityFrameworkCore.Cosmos.Query.Internal.CosmosProjectionBindingExpressionVisitor.VisitMemberInit(MemberInitExpression memberInitExpression)
   at System.Linq.Expressions.MemberInitExpression.Accept(ExpressionVisitor visitor)
   at Microsoft.EntityFrameworkCore.Cosmos.Query.Internal.CosmosProjectionBindingExpressionVisitor.Visit(Expression expression)
   at Microsoft.EntityFrameworkCore.Cosmos.Query.Internal.CosmosProjectionBindingExpressionVisitor.Translate(SelectExpression selectExpression, Expression expression)
   at Microsoft.EntityFrameworkCore.Cosmos.Query.Internal.CosmosQueryableMethodTranslatingExpressionVisitor.TranslateSelect(ShapedQueryExpression source, LambdaExpression selector)
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Storage.Database.CompileQuery[TResult](Expression query, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass12_0`1.<ExecuteAsync>b__0()
 at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQueryCore[TFunc](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync[TResult](Expression query, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.ExecuteAsync[TResult](Expression expression, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.GetAsyncEnumerator(CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.ConfiguredCancelableAsyncEnumerable`1.GetAsyncEnumerator()
 at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
   at Author.Query.Persistence.CountryService.GetCountriesDataAsync(Int32 languageId) in /src/QueryStack/Author.Query.Persistence/CountryService.cs:line 243
  at Author.Query.Persistence.CountryService.GetCountriesAsync(Int32 dftLanguageId, Int32 localeLangId) in /src/QueryStack/Author.Query.Persistence/CountryService.cs:line 228
  at Author.Query.Persistence.CountryService.GetAllCountriesAsync(LanguageDTO language) in /src/QueryStack/Author.Query.Persistence/CountryService.cs:line 164
  at GraphQL.DataLoader.DataLoaderBase`1.DispatchAsync(CancellationToken cancellationToken)
   at Author.Query.New.API.GraphQL.Resolvers.CountriesResolver.<>c__DisplayClass5_1.<<Resolve>b__3>d.MoveNext() in /src/QueryStack/Author.Query.New.API/GraphQL/Resolvers/CountriesResolver.cs:line 40
--- End of stack trace from previous location where exception was thrown ---
   at GraphQL.Types.ResolveFieldContext`1.TryAsyncResolve[TResult](Func`2 resolve, Func`2 error)
   --- End of inner exception stack trace ---

Can anyone help me here by providing their guidance to fix this issue?

Radbun

You already had a bug. This query simply can't be converted to SQL because it calls local functions. EF Core 2.2's client-side execution covered up the bug by pulling almost everything to the client and evaluating the rest of the query there. This seriously harms performance by loading more records than necessary, and/or loading all columns when only a few are needed. Fix the (very hard to read!) query so it doesn't use any local functions. – Verdieverdigris

BTW if you enabled EF Core 2.x logging you'd see that EF was already emitting warnings at runtime – Verdieverdigris

It's not just Helper.ReplaceChar that breaks this query. images.FirstOrDefault(...) searches into a local collection. There's no way this can be translated to SQL. The query is so complex though the LINQ provider failed with that error before trying to convert it into SQL – Verdieverdigris

images.FirstOrDefault shouldn't be used either. You're just looking up a FilePath by ImageId. The current code scans all images to find a matching one. Use a Dictionary instead to hold only the ImageID and FilePath values, and use Dictionary.TryGetValue to look up the path, eg images.TryGetValue(co.PNGImageID,out var img)? img.FileName:null – Verdieverdigris

Thanks @PanagiotisKanavos for your response. As per your inputs I have updated the code and it worked for me :) – Radbun

Answer 1

You have a couple of issues here that are quite easy to fix, break the query int two halves the first to query the database and return list of partially populated CountryDTOs the second iterate through the list populating the missing items:

private async Task<List<CountryDTO>> GetCountriesDataAsync(int languageId)
{
            int pageNo = 1, pageSize = 100;

            var results = _dbContext.Countries // <- do not use await on db context
                    .Where(cc => cc.IsPublished.Equals(true)
                      && cc.LanguageId.Equals(languageId))
                    .Select(co => new {
                        co.CountryId,
                        co.DisplayName,
                        co.PNGImageId,
                        co.SVGImageId
                    })
                     .Skip((pageNo - 1) * pageSize) // <- use page size no 100
                     .Take(pageSize)
                     .ToListAsync();

            var images = await _cacheService.GetAllAsync("imagesCacheKey");

            return Task.FromResult( results.Select(co => new CountryDTO
            {
                Uuid = co.CountryId,
                PNGImagePath = images.FirstOrDefault(im => im.ImageId.Equals(co.PNGImageId))?.FilePath, // <- could be null so use ?.FilePath
                SVGImagePath = images.FirstOrDefault(im => im.ImageId.Equals(co.SVGImageId))?.FilePath,
                DisplayName = co.DisplayName,
                DisplayNameShort = co.DisplayName,
                Name = Helper.ReplaceChars(co.DisplayName),
                Path = Helper.ReplaceChars(co.DisplayName),
                CompleteResponse = true
            }).ToList());

}

I have created an anonymous type to hold the return from the database as opposed to adding additional field to the DTO for image ids and the likes. if you need more from the database just add the fields to the anonymous type in the context select statement.

Lemal

Thanks @David Hinchliffe for your detailed responses :) – Radbun

Is there actually an issue with the OP's code or is this just still a bug in EF Core? – Orbicular

The issue with the code was that the link statement was trying to do something the underlying database didn't know how to, specifically the database doesn't how to do the Helper.ReplaceChars, so linq couldn't create a SQL statement to run on the database. Await can be used with databases but with care, the execution of the task must complete before the db context is disposed, and this can happen if the wrong provider pattern is used. – Lemal

Answer 2

This error just happened to me and for future visitors this what I was doing wrong:

I had two models, for exemple:

public User
{
    public int Id {get; set;}
    public string Name {get; set;}
    //just a silly method that does something
    public string GetName()
    {
      return this.Name + "something"
    }
}

public UserInfo
{
    public int Id {get; set;}
    // a bunch of other properties
}

And somewhere in code I do a projection with automapper like

Mapper.ProjectTo<UserInfo>(Repository.Get().ByUserName(userId)).FirstOrDefault();

This error When called from 'VisitLambda',... begin to show when I inserted the GetName methods that somehow broke the projectTo mapping.

Materials

Answer 3

My case is a bit different, but exact same error: "When called from 'VisitLambda', rewriting a node of type 'System.Linq.Expressions.ParameterExpression' must return a non-null value of the same type"

However, my collections were separate and I was not running the projection on the same database call. I retrieved an IQueryable<MyType> collection, and after that a Select projection to my view model. I was also not using any local functions in my projection.

SO, what happened was I was using two different collections in the same projection. I had an IQueryable<DbSet<T>> collection, and a different list of a certain DTO that I'd stored from the database. Thus it makes sense that it threw an exception for "must be a value of the same type."

The problem was using this other DTO list:

collection.Select(s => new MyDataClass(s.Id, s.Name, otherDTOList.Find(...) ...

My solution was to instead use a foreign key reference in my projection instead of trying to use the additional DTO list, since the FK table data was retrieved in the join:

collection.Select(s => new MyDataClass(s.Id, s.Name, s.MyFKClass.Find(...

Odd case, and if there a way to do this with two collections, please let me know!

Cherenkov