Add two expressions to create a predicate in Entity Framework Core 3 does not work

Question

I am trying to build an "And" predicate method using C# with Entity Framework Core 3 in a .NET Core application.

The function adds two expressions to each other and passes it to an IQueryable code:

public Expression<Func<T, bool>> AndExpression<T>
                    (Expression<Func<T, bool>> left, Expression<Func<T, bool>> right)
{
      var andExpression = Expression.AndAlso(
           left.Body, Expression.Invoke(right,
           left.Parameters.Single()));

      return Expression.Lambda<Func<T, bool>>(andExpression, left.Parameters);
}

The call for of the function

Expression<Func<Entity, bool>> left = t => t.Id == "id1";
Expression<Func<Entity, bool>> right = t => t.Id == "id2";
var exp = AndExpression(left, right);
this.dbContext.Set<Entity>().source.Where(exp).ToList();

My code works fine in version EF Core 2, but after I updated the version to version 3 it throws the following exception

The LINQ expression 'Where( source: DbSet, predicate: (s) => (t => t.Id == "id1") && Invoke(t => t.Id == "id2") )' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync().

I cannot translate the query to Enumerable due to memory issues. I understand the problem but I don't know if there is a way to avoid it.

If anyone has a tip for me, I would appreciate that. Thanks a lot!

Halliehallman

Answer 1

You need to unwrap the bodies of your lambdas, rather than using Expression.Invoke. You'll also have to rewrite at least one of the lambdas, so they both use the same parameters.

We'll use an ExpressionVisitor to replace right's parameter with the corresponding parameter from left. Then we'll construct the AndExpression using left's body and the rewritten body from right, and finally create a new lambda.

public class ParameterReplaceVisitor : ExpressionVisitor
{
    public ParameterExpression Target { get; set; }
    public ParameterExpression Replacement { get; set; }

    protected override Expression VisitParameter(ParameterExpression node)
    {
        return node == Target ? Replacement : base.VisitParameter(node);
    }
}

public static Expression<Func<T, bool>> AndExpression<T>(
    Expression<Func<T, bool>> left, Expression<Func<T, bool>> right)
{
    var visitor = new ParameterReplaceVisitor()
    {
        Target = right.Parameters[0],
        Replacement = left.Parameters[0],
    };

    var rewrittenRight = visitor.Visit(right.Body);
    var andExpression = Expression.AndAlso(left.Body, rewrittenRight);
    return Expression.Lambda<Func<T, bool>>(andExpression, left.Parameters);
}

This results in a lambda with the following DebugView:

.Lambda #Lambda1<System.Func`2[System.String,System.Boolean]>(Entity $t) {
    $t.Id == "id1" && $t.Id == "id2"
}

Whereas your code results in a lambda with the following DebugView:

.Lambda #Lambda1<System.Func`2[System.String,System.Boolean]>(System.String $t) {
    $t == "id1" && .Invoke (.Lambda #Lambda2<System.Func`2[System.String,System.Boolean]>)($t)
}

.Lambda #Lambda2<System.Func`2[System.String,System.Boolean]>(System.String $t) {
    $t == "id2"
}

See how yours is calling a lambda from within a lambda (something that EF can't handle), whereas mine just has a single lambda.

Ameliaamelie

Nice! So, the problem was with the invoke method as it is not being correctly translated. It worked! Thanks a lot – Halliehallman

The expression debug view can be misleading. I'm using a programmatically-built expression using parameter name 'x' and then I'm passing in a hard-coded expression, which also uses 'x'. I 'AndAlso' them. The debug view then shows "someList.Contains($x.PublicPackageID) && !$x.Expired". Same parameter names. Seemingly correct. However, the EF "can't translate" exception then reads "d => List<Nullable<int>> { 1, 2, 3 }.Contains(d.PublicPackageID) && !(x.Expired))". Different parameter names. Obviously incorrect. I'll see if your proposed solution can't help me solve my problem. – Myers

To make matters worse, if I replace my hardcoded lambda expression's 'x' param with a 'd' param, then the query shows seemingly correct in both debug view as well as the generated EF exception, in the sense that they both use a single param name consistently. Had EF coincidentally chosen the same param name as I had, or vice versa, then I might've been looking for days to even find the cause of this problem. – Myers

Canton7, your solution has solved my problem. It would've been a challenge to solve this myself. Many hours saved. Tnx! – Myers

ReplacingExpressionVisitor class from Microsoft.EntityFrameworkCore.Query can be useful here. learn.microsoft.com/en-us/dotnet/api/… – Flannery

Answer 2

The issue here is that your query didn't work, or at least not how you intended in EF Core 2 either - In EF3 it will throw an exception for things like this due to a behavioural change, within EF Core 2 this would have silently pulled into memory and done the operation there, so in essence your query would have always been pulling the data into memory and performing the operation there. However now its telling you that it would do that and wanting you to explicitly set it to, or modify your expression to something that it can translate into a sql statement correctly.

This is actually a pretty good example case of why the team did this, seemingly you were unaware that this was previously pulling all of the data into memory and performing the operation there - now that you are aware you can work on a fix to get it performing the operation on the sql server

For more info feel free to read up Here

A modification to your code to be somewhere along the lines of

public static Expression<Func<T, bool>> AndExpression
                    (this Expression<Func<T, bool>> left, Expression<Func<T, bool>> right)
{
      var invoked = var invokedExpr = Expression.Invoke (right, left.Parameters.Cast<Expression> ());
      return Expression.Lambda<Func<T, bool>>
      (Expression.AndAlso (left.Body, invoked), left.Parameters);
}

working from memory so code above may not be perfect - at work so I cant fully test

Manymanya

Answer 3

I took canton7's solution and made it into an extension that checks for null values:

    private static Expression<Func<T, bool>> _AndExpression<T>(Expression<Func<T, bool>> left, Expression<Func<T, bool>> right)
    {
        ParameterReplaceVisitor visitor = new ParameterReplaceVisitor()
        {
            Target      = right.Parameters[0],
            Replacement = left.Parameters[0],
        };

        Expression rewrittenRight = visitor.Visit(right.Body);
        Expression andExpression  = Expression.AndAlso(left.Body, rewrittenRight);
        return Expression.Lambda<Func<T, bool>>(andExpression, left.Parameters);
    }

    public static Expression<Func<T, bool>> AndExpression<T>(this Expression<Func<T, bool>> left, Expression<Func<T, bool>> right)
    {
        bool leftNotNull  = (left != null);
        bool rightNotNull = (right != null);

        return (leftNotNull && rightNotNull)
            ? _AndExpression(left, right)
            : left;
    }

Usage:

expression1 = expression1.AndExpression<T>(expression2);

Myers