Applying Distinct to OData query
Asked Answered
T

2

7

I want to get a list of distinct values from my OData endpoint. But distinct or group by isn't supported yet.

My URI query looks something like this

GET /odata/Products?$select=foo & $top=10 & $count=true & distinct=true

My Controller

[EnableQuery]
public IQueryable<FooBarBaz> Get(ODataQueryOptions<FooBarBaz> queryOptions, bool distinct)
{
        //I've tried the following
        return Repository.AsQueryable().Distinct();

        // and
        return Repository.AsQueryable().GroupBy(x => x.Foo);

        // and
        IQueryable query = queryOptions.ApplyTo(Repository.AsQueryable());
        return query.Distinct(); // Can't call .Distinct() here
}

None work :(

Toxinantitoxin answered 8/5, 2015 at 10:30 Comment(2)
The component AdaptiveLINQ can help you. Disclaimer: I'm the AdaptiveLINQ developer.Stuppy
The major reason this isn't supported is that the Entity records served by an OData controller by definition MUST be unique. You declare in the configuration the column that provides the unique key and if there is a unique column, then the records will always be distinct.Otology
D
8

The best solution to solve the problem by defining an collection Action on the resource.

First Step : configure the 'Distinct' action in WebApiConfig.cs

ODataConventionModelBuilder builder = new ODataConventionModelBuilder();
builder.EntitySet<FooBarBaz>("FooBarBazs");//Resource Name

ActionConfiguration Distinct = builder.Entity<FooBarBaz>().Collection.Action("Distinct");//Name of the action method
Distinct.ReturnsCollectionFromEntitySet<FooBarBaz>("FooBarBazs");//Return type of action
Distinct.Parameter<string>("On");//Property on which collection is filtered as Distinct

config.Routes.MapODataRoute("odata", "odata", builder.GetEdmModel());

Second Step : Add the Action in FooBarBazsController.cs which returns the collection of distinct entities

[EnableQuery]//enable the $select,$expend Queries
[HttpPost]//All the action methods are of post type in Web api
public IQueryable<FooBarBaz> Distinct(ODataActionParameters parameters)
{
        string on = "";
        if (!ModelState.IsValid)
        {
            throw new HttpResponseException(HttpStatusCode.BadRequest);
        }

        try
        {
             on = parameters["On"] as string;
        }
        catch (NullReferenceException ex)
        {
            HttpResponseMessage message = new HttpResponseMessage(HttpStatusCode.BadRequest);
            message.Content = new StringContent("{\"Error\":\"Invalid Query -> On property is not defined\"}");
            throw new HttpResponseException(message);
        }
        catch (Exception ex)
        {
            throw new HttpResponseException(HttpStatusCode.BadRequest);
        }


        PropertyInfo[] props = new FooBarBaz().GetType().GetProperties(BindingFlags.Public | BindingFlags.Instance);
        var isPropertyExist = false;
        for (int i = 0; i < props.Length; i++)
        {
            if (props[i].Name.Equals(on))
            {
                isPropertyExist = true;
                break;
            }
        }


        if (isPropertyExist)
        {
            var fooBarBazCollection = db.fooBarBazs.GroupBy(GetGroupKey(on)).Select(g => g.FirstOrDefault());//Select the Distinct Entity on the basis of a property
            return fooBarBazCollection ;
        }
        else
        {
            HttpResponseMessage message = new HttpResponseMessage(HttpStatusCode.BadRequest);
            message.Content = new StringContent("{\"Error\":\"Property '"+on+"' Not Exist}");
            throw new HttpResponseException(message);
        }
}

Third Step : Add a static method which returns an Expression for groupby on the basis of Property Name.

private static Expression<Func<fooBarBaz, string>> GetGroupKey(string property)
    {
        var parameter = Expression.Parameter(typeof(fooBarBaz));
        var body = Expression.Property(parameter, property);
        return Expression.Lambda<Func<fooBarBaz, string>>(body, parameter);
    } 

Now Build the project and You can query the Resource like this

POST /odata/FooBarBazs/Distinct HTTP/1.1
Host: localhost:9360
Content-Type: application/json
Cache-Control: no-cache
Postman-Token: 6d174086-7b97-76a2-679c-4dab3dfb5938

{"On":"PropertyName"} 

And can also use the $select and $expend like this

POST /odata/FooBarBazs/Distinct?$select=PropertyName1,PropertyName2 HTTP/1.1
Host: localhost:9360
Content-Type: application/json
Cache-Control: no-cache
Postman-Token: 6d174086-7b97-76a2-679c-4dab3dfb5938

{"On":"PropertyName"} 

I hope this solve the problem. +1 if it do.

Desist answered 14/6, 2015 at 11:33 Comment(2)
This does seem like the best approach until there's support for it :) [Queryable] should be [EnableQuery] though.Dauphine
@Toxinantitoxin I have updated the code block as you suggested.Desist
O
12

Because you have specified the EnableQuery attribute, you can use $apply to groupby your distinct fields, without having to add any custom functions or parameters, you get this for free out of the box:

GET /odata/Products?$apply=groupby((foo))&top=10&$count=true

This is simple OData v4 standard syntax that doesn't require any code modification to implement. Don't go changing each controller that you want to support a distinct query on, you can't know 100% in advance which controllers your client apps might want this functionality on, so use the functionality that is provided before you start customisations.

Of course there is a caveat to this approach that do not make it viable 100% of the time:

  • $filter and $orderby can only operate on the fields specified in your group by clause

This may require you to include additional fields into your grouping statement and for some complex filtering the resultant dataset may not be satisfactory, in cases such as these we found it easier to support passing in an additional pre-filter parameter through HTTP header that can be applied to the query before the passed in query options are applied, note that this was only necessary because our filter conditions were tenancy and security related and so the resultant data set had many more duplicate entries if you ignored the security descriptors.

Just for fun, here is our custom GET function that applies the pre-filter if it is passed in:

[EnableQuery]
public IQueryable<FooBarBaz> Get(ODataQueryOptions<FooBarBaz> queryOptions, bool distinct)
{
    DbQuery<FooBarBaz> query = Repository;
    query = this.ApplyUserPolicy(query);
    return Ok(query);
}

The following is implemented in a base class so that we don't have it in each controller:

/// <summary>
/// Apply default user policy to the DBQuery that will be used by actions on this controller.
/// The big one we support here is X-Filter HTTP headers, so now you can provide top level filtering in the header of the request 
/// before the normal OData filter and query parameters are applied.
/// This is useful when you want to use $apply and $filter together but on separate sets of conditions.
/// </summary>
/// <param name="dataTable">DBQuery to apply the policy to</param>
/// <returns>Returns IQueryable entity query ready for processing with the headers applied (if any)</returns>
private IQueryable<TEntity> ApplyUserPolicy(DbQuery<TEntity> dataTable)
{
    // Proprietary Implementation of Security Tokens
    //var tokenData = SystemController.CurrentToken(Request);
    //IQueryable<TEntity> query = ApplyUserPolicy(dataTable, tokenData);
    IQueryable<TEntity> query = dataTable.AsQueryable();

    // Now try and apply an OData filter passed in as a header.
    // This means we are applying a global filter BEFORE the normal OData query params
    // ... we can filter before $apply and group by

    System.Collections.Generic.IEnumerable<string> filters = null;
    if (Request.Headers.TryGetValues("X-Filter", out filters))
    {
        foreach (var filter in filters)
        {
            //var expressions = filter.Split(',');
            //foreach (var expression in expressions)
            {
                var expression = filter;
                Dictionary<string, string> options = new Dictionary<string, string>()
                {
                    { "$filter"  , expression },
                };

                var model = this.Request.ODataProperties().Model;
                IEdmNavigationSource source = model.FindDeclaredEntitySet(this.GetEntitySetName());
                var type = source.EntityType();
                Microsoft.OData.Core.UriParser.ODataQueryOptionParser parser
                    = new Microsoft.OData.Core.UriParser.ODataQueryOptionParser(model, type, source, options);
                var filterClause = parser.ParseFilter();     // parse $filter 

                FilterQueryOption option = new FilterQueryOption(expression, new ODataQueryContext(model, typeof(TEntity), this.Request.ODataProperties().Path), parser);
                query = (IQueryable<TEntity>)option.ApplyTo(query, new ODataQuerySettings());
            }
        }
    }


    return query;
}

If nothing else, it's cheaper that trying to sell AdaptiveLINQ to your manager :)

Otology answered 1/8, 2016 at 23:13 Comment(5)
I was able to get a Distinct response just by the $apply and groupby in my filter. Its likely you aren't getting more thanks because of the other verbose code in your response. I wanted to highlight to others that the $apply and groupby returns a distinct list.Cathar
true but $apply only works if you do not also need to $filter, unless of course your $filter is operating on the results columns in your $apply, and even then there are restrictions. So while I had a similar problem to OP, when I tried to implement my simple $apply in my own code first I quickly decided that the answer needed to be more robust. It was fun, hopefully someone from the ODataLib team takes notice and adds some similar support for us laterOtology
@ChrisSchaller - I have been fooling around with odata, as I want to perform a distinct query with it, but I also want pagination. Meaning, let's say that I have 330 distinct values, and with a page size of 100, I want 4 pages. And to do this, I'm executing my queryable (that's getting the odata applied to) using .Skip.Take.ToList, but here is what I'm finding: Executing the first page of 100 it's only getting the distinct values for that page and not the superset before paging. Is there a way to groupby on the superset and then page?Klondike
I would create a custom IQueryable Function endpoint to serve the distinct recordset in that case, or not page the data. 330 values is not a lot, you could bring that back in a single hit and manage the paging on the client if you really wanted a paged view. The need for Distinct usually indicates you do not have a sufficiently normalized data structure or you are executing the query from the wrong controller, serve the query from a controller that already provides the distinct records.Otology
@ChrisSchaller - Actually Distinct has a purpose. We have a lot of screens with grids. And each grid will have columns tied to the database. One of the features we have is "filtering". So, you can select any column and then "browse" the data associated with that column for data values to be part of your filter. Hence, the need for distinct. As for the 330, that was an example, all I know is that this has been very difficult! Fighting through some things now.Klondike
D
8

The best solution to solve the problem by defining an collection Action on the resource.

First Step : configure the 'Distinct' action in WebApiConfig.cs

ODataConventionModelBuilder builder = new ODataConventionModelBuilder();
builder.EntitySet<FooBarBaz>("FooBarBazs");//Resource Name

ActionConfiguration Distinct = builder.Entity<FooBarBaz>().Collection.Action("Distinct");//Name of the action method
Distinct.ReturnsCollectionFromEntitySet<FooBarBaz>("FooBarBazs");//Return type of action
Distinct.Parameter<string>("On");//Property on which collection is filtered as Distinct

config.Routes.MapODataRoute("odata", "odata", builder.GetEdmModel());

Second Step : Add the Action in FooBarBazsController.cs which returns the collection of distinct entities

[EnableQuery]//enable the $select,$expend Queries
[HttpPost]//All the action methods are of post type in Web api
public IQueryable<FooBarBaz> Distinct(ODataActionParameters parameters)
{
        string on = "";
        if (!ModelState.IsValid)
        {
            throw new HttpResponseException(HttpStatusCode.BadRequest);
        }

        try
        {
             on = parameters["On"] as string;
        }
        catch (NullReferenceException ex)
        {
            HttpResponseMessage message = new HttpResponseMessage(HttpStatusCode.BadRequest);
            message.Content = new StringContent("{\"Error\":\"Invalid Query -> On property is not defined\"}");
            throw new HttpResponseException(message);
        }
        catch (Exception ex)
        {
            throw new HttpResponseException(HttpStatusCode.BadRequest);
        }


        PropertyInfo[] props = new FooBarBaz().GetType().GetProperties(BindingFlags.Public | BindingFlags.Instance);
        var isPropertyExist = false;
        for (int i = 0; i < props.Length; i++)
        {
            if (props[i].Name.Equals(on))
            {
                isPropertyExist = true;
                break;
            }
        }


        if (isPropertyExist)
        {
            var fooBarBazCollection = db.fooBarBazs.GroupBy(GetGroupKey(on)).Select(g => g.FirstOrDefault());//Select the Distinct Entity on the basis of a property
            return fooBarBazCollection ;
        }
        else
        {
            HttpResponseMessage message = new HttpResponseMessage(HttpStatusCode.BadRequest);
            message.Content = new StringContent("{\"Error\":\"Property '"+on+"' Not Exist}");
            throw new HttpResponseException(message);
        }
}

Third Step : Add a static method which returns an Expression for groupby on the basis of Property Name.

private static Expression<Func<fooBarBaz, string>> GetGroupKey(string property)
    {
        var parameter = Expression.Parameter(typeof(fooBarBaz));
        var body = Expression.Property(parameter, property);
        return Expression.Lambda<Func<fooBarBaz, string>>(body, parameter);
    } 

Now Build the project and You can query the Resource like this

POST /odata/FooBarBazs/Distinct HTTP/1.1
Host: localhost:9360
Content-Type: application/json
Cache-Control: no-cache
Postman-Token: 6d174086-7b97-76a2-679c-4dab3dfb5938

{"On":"PropertyName"} 

And can also use the $select and $expend like this

POST /odata/FooBarBazs/Distinct?$select=PropertyName1,PropertyName2 HTTP/1.1
Host: localhost:9360
Content-Type: application/json
Cache-Control: no-cache
Postman-Token: 6d174086-7b97-76a2-679c-4dab3dfb5938

{"On":"PropertyName"} 

I hope this solve the problem. +1 if it do.

Desist answered 14/6, 2015 at 11:33 Comment(2)
This does seem like the best approach until there's support for it :) [Queryable] should be [EnableQuery] though.Dauphine
@Toxinantitoxin I have updated the code block as you suggested.Desist

© 2022 - 2024 — McMap. All rights reserved.