LINQ where clause with lambda expression having OR clauses and null values returning incomplete results

Question

the problem in short

we have a lambda expression used in the Where clause, which is not returning the "expected" result.

quick summary

in the analysisObjectRepository object, there are certain objects which also contain the parent relationship in a property named Parent. we are querying this analysisObjectRepository to return some objects.

detail

what the code below supposed to do is, returning the root, the first children (immediate children) and grandchildren of a specific object containing the ID value.

in the code below, common sense says that all the results which makes any of the 3 seperate OR conditions true should be returned as in the results.

List<AnalysisObject> analysisObjects = 
    analysisObjectRepository
        .FindAll()
        .Where(x => x.ID               == packageId ||
                    x.Parent.ID        == packageId || 
                    x.Parent.Parent.ID == packageId)
        .ToList();

but the above code only returns the children and grandchildren, while not returning the root objects (with a null Parent value) which make the

x.ID == packageId

condition true.

only objects which make the second

x.Parent.ID == packageId

and third

x.Parent.Parent.ID == packageId

clauses are returned.

If we only write the code to return the root object with the below code, it is returned, so we are totally sure that analysisObjectRepository contains all the objects

List<AnalysisObject> analysisObjects = 
    analysisObjectRepository
        .FindAll()
        .Where(x => x.ID == packageId )
        .ToList();

However, when we rewrite it as a delegate, we get the expected result, returning all the expected objects.

List<AnalysisObject> analysisObjects = 
    analysisObjectRepository
        .FindAll()
        .Where(delegate(AnalysisObject x) 
        { 
            return 
              (x.ID == packageId) || 
              (x.Parent != null && x.Parent.ID == packageId) || 
                  (x.Parent != null && 
                   x.Parent.Parent != null && 
                   x.Parent.Parent.ID == packageId); })
        .ToList();

question

Are we missing something in the lambda expression? it is a really simple 3 part OR condition and we think that any object that makes any of the three conditions true should be returned. we suspected that the root object having a null Parent value might cause a problem but couldn't figure it out exactly.

any help would be great.

Cumulostratus

Is analysisObjectRepository a List<AnalysisObject>? If so, the call to FindAll is unnecessary. – Widen

What is it? Whether or not this is using LINQ-to-Objects or a custom implementation of IQueryable can make all the difference here. – Widen

Can you pare it down to one single object that displays the errant behavior? What is the value of x.ID and the value of packageId when this occurs? – Ical

Answer 1

Your second delegate is not a rewrite of the first in anonymous delegate (rather than lambda) format. Look at your conditions.

First:

x.ID == packageId || x.Parent.ID == packageId || x.Parent.Parent.ID == packageId

Second:

(x.ID == packageId) || (x.Parent != null && x.Parent.ID == packageId) || 
(x.Parent != null && x.Parent.Parent != null && x.Parent.Parent.ID == packageId)

The call to the lambda would throw an exception for any x where the ID doesn't match and either the parent is null or doesn't match and the grandparent is null. Copy the null checks into the lambda and it should work correctly.

Edit after Comment to Question

If your original object is not a List<T>, then we have no way of knowing what the return type of FindAll() is, and whether or not this implements the IQueryable interface. If it does, then that likely explains the discrepancy. Because lambdas can be converted at compile time into an Expression<Func<T>> but anonymous delegates cannot, then you may be using the implementation of IQueryable when using the lambda version but LINQ-to-Objects when using the anonymous delegate version.

This would also explain why your lambda is not causing a NullReferenceException. If you were to pass that lambda expression to something that implements IEnumerable<T> but not IQueryable<T>, runtime evaluation of the lambda (which is no different from other methods, anonymous or not) would throw a NullReferenceException the first time it encountered an object where ID was not equal to the target and the parent or grandparent was null.

Added 3/16/2011 8:29AM EDT

Consider the following simple example:

IQueryable<MyObject> source = ...; // some object that implements IQueryable<MyObject>

var anonymousMethod =  source.Where(delegate(MyObject o) { return o.Name == "Adam"; });    
var expressionLambda = source.Where(o => o.Name == "Adam");

These two methods produce entirely different results.

The first query is the simple version. The anonymous method results in a delegate that's then passed to the IEnumerable<MyObject>.Where extension method, where the entire contents of source will be checked (manually in memory using ordinary compiled code) against your delegate. In other words, if you're familiar with iterator blocks in C#, it's something like doing this:

public IEnumerable<MyObject> MyWhere(IEnumerable<MyObject> dataSource, Func<MyObject, bool> predicate)
{
    foreach(MyObject item in dataSource)
    {
        if(predicate(item)) yield return item;
    }
}

The salient point here is that you're actually performing your filtering in memory on the client side. For example, if your source were some SQL ORM, there would be no WHERE clause in the query; the entire result set would be brought back to the client and filtered there.

The second query, which uses a lambda expression, is converted to an Expression<Func<MyObject, bool>> and uses the IQueryable<MyObject>.Where() extension method. This results in an object that is also typed as IQueryable<MyObject>. All of this works by then passing the expression to the underlying provider. This is why you aren't getting a NullReferenceException. It's entirely up to the query provider how to translate the expression (which, rather than being an actual compiled function that it can just call, is a representation of the logic of the expression using objects) into something it can use.

An easy way to see the distinction (or, at least, that there is) a distinction, would be to put a call to AsEnumerable() before your call to Where in the lambda version. This will force your code to use LINQ-to-Objects (meaning it operates on IEnumerable<T> like the anonymous delegate version, not IQueryable<T> like the lambda version currently does), and you'll get the exceptions as expected.

TL;DR Version

The long and the short of it is that your lambda expression is being translated into some kind of query against your data source, whereas the anonymous method version is evaluating the entire data source in memory. Whatever is doing the translating of your lambda into a query is not representing the logic that you're expecting, which is why it isn't producing the results you're expecting.

Widen

the lambda expression is executing perfectly while only returning incomplete results. there are objects with non-matching ID values and null parent values. – Cumulostratus

Adam your comment shed some light on the issue, as you have guessed it implements IQueryable interface. We have tried the exact match of the delegate that works in the lambda expression and it is still not working in the "expected" way and it is NOT throwing a NullReferenceException. We do not want to lose time on this issue and we are going by using delegates on that part but it would remain as a mistery why it is not working in the "expected" way. – Cumulostratus

@Cihan: It isn't throwing an exception because the lambda (in this circumstance) and the anonymous method are working in fundamentally different ways. See my edit. – Widen

it is clear that they work in totally different ways. we have to dig deep into the provider class structure to find the implementation nuance that results in this seemingly "strange" behaviour. thanks for the support. – Cumulostratus

Answer 2

Try writting the lambda with the same conditions as the delegate. like this:

  List<AnalysisObject> analysisObjects = 
    analysisObjectRepository.FindAll().Where(
    (x => 
       (x.ID == packageId)
    || (x.Parent != null && x.Parent.ID == packageId)
    || (x.Parent != null && x.Parent.Parent != null && x.Parent.Parent.ID == packageId)
    ).ToList();

Australorp

we have tried that and it is not working in the "expected" way. – Cumulostratus

Answer 3

You are checking Parent properties for null in your delegate. The same should work with lambda expressions too.

List<AnalysisObject> analysisObjects = analysisObjectRepository
        .FindAll()
        .Where(x => 
            (x.ID == packageId) || 
            (x.Parent != null &&
                (x.Parent.ID == packageId || 
                (x.Parent.Parent != null && x.Parent.Parent.ID == packageId)))
        .ToList();

Clinkstone

we already tried that and it didn't work. and don't lambda expressions should take care of this? – Cumulostratus

@yaqari: Why would lambda expressions take care of null checks? – Widen

i was trying to say that lambda expression is not throwing an exception and the first OR condition is true so lambda expressions should be "taking care" of that and return the correct result. – Cumulostratus

Without null checks it will fail when x.ID != packegeId && x.Parent == null. – Clinkstone