c# Nest and Elasticsearch Aggregations

Question

Does anyone know how to do multiple aggregations with nest? I have found quite a few examples unfortunately none of them work.

Here's what I have:

Vehicles fields = new Vehicles();

//create a terms query
var query = new TermsQuery
{
    IsVerbatim = true,
    Field = "VehicleOwnerId",
    Terms = new string[] { 25 },
};

var aggregations = new Dictionary<string, IAggregationContainer>
{
    { "years", new AggregationContainer
        {
            Terms = new TermsAggregation(nameof(fields.Year))
            {
                Field = new Field(nameof(fields.Year))
            }
        }
    }
    //,
    //{ "makes", new AggregationContainer
    //    {
    //        Terms = new TermsAggregation("Make")
    //        {
    //            Field = new Field(nameof(fields.Make))
    //        }
    //    }
    //}
};

//create the search request
var searchRequest = new SearchRequest
{
    Query = query,
    From = 0,
    Size = 100,
    Aggregations = aggregations
};

var result = client.SearchAsync<InventoryLiveView>(searchRequest).Result;

var years = result.Aggregations.Terms("years");
Dictionary<string, long> yearCounts = new Dictionary<string, long>();
foreach (var item in years.Buckets)
{
    yearCounts.Add(item.Key, item.DocCount ?? 0);
}

If I just execute the code like this it works. Years returns the aggregates as expected. If I try to add another field (like the one commented out above) it fails and I get zero records. How can I get multiple aggregates in one query? I see examples of it all over, but none of the examples I've tried seem to work and most seem to be outdated (including some in the Nest documentation). I have also tried this approach which is pretty close to the documentation.

//create the search request
var searchRequest = new SearchRequest
{
    Query = query,
    From = 0,
    Size = 100,
    //Aggregations = aggregations
    Aggregations = new AggregationDictionary
    {
        { 
            "childAgg", new ChildrenAggregation("childAgg", typeof(Vehicles ))
            {
                Aggregations = new AggregationDictionary
                {
                    {"years", new TermsAggregation(nameof(fields.VehicleYear))},
                    {"makes", new TermsAggregation(nameof(fields.VehicleMakeName))},
                    {"models", new TermsAggregation(nameof(fields.VehicleModelName))},
                }
            }
        }
    }
};

var result = client.SearchAsync<Vehicles>(searchRequest).Result;

This just produces a null reference exception.

Shiekh

Which NEST documentation is outdated? Have you seen elastic.co/guide/en/elasticsearch/client/net-api/current/… – Saleratus

I think that is the page that is out of date. It shows this example: new MaxAggregation("max_per_child", Field<CommitActivity>(p => p.ConfidenceFactor)) which should be MaxAggregation("max_per_child", new Field("ConfidenceFactor")) or at lease as far as I can tell. – Shiekh

Well now I see an error when I try one of the examples from the page. The error is "Type: null_pointer_exception Reason: """ not sure why it happens or what that means exactly. – Shiekh

The documentation on the page is generated from test source code: github.com/elastic/elasticsearch-net/blob/… These tests run on every check-in, to notify if there are compilation errors or changes in the client code so that ultimately, the documentation aligns with the latest client code in that branch. I can't see any issues with the docs in this case, but if you do come across some, please feel free to open an issue on github.com/elastic/elasticsearch-net/issues so we can address – Saleratus

The code that didn't seem to work for me was an example on the terms aggregation. It shows ` Field = Field<Project>(p => p.State)` but I couldn't get Field<T> to work. Field(string) worked fine however. I don't remember the exact error but It just didn't recognize it. Thank you for the information--I was probably doing something wrong. – Shiekh

That's a fair point - the documentation uses Nest.Infer static import: elastic.co/guide/en/elasticsearch/client/net-api/current/…. – Saleratus

Answer 1

I guess I'll never have too worry about getting to proud as a programmer :) It's too often that the solution to the problem makes me feel stupid when it reveals itself.

So my issue was that the field I was trying to use in the aggregation was text and couldn't be used. I switched everything to the ID fields and multiple aggregations work as expected.

So this version of the code works like a champ:

Vehicle fields = new Vehicle ();

//create a terms query
var query = new TermsQuery
{
    IsVerbatim = true,
    Field = "VehicleOwnerId",
    Terms = new string[] { "30" },
};

string[] Fields = new[]
{
    nameof(fields.Year),
    nameof(fields.MakeId),
    nameof(fields.ModelId)
};

var aggregations = new Dictionary<string, IAggregationContainer>();
foreach (string sField in Fields)
{
    var termsAggregation = new TermsAggregation(sField)
    {
        Field = sField
    };

    aggregations.Add(sField, new AggregationContainer { Terms = termsAggregation });
}

//create the search request
var searchRequest = new SearchRequest
{
    Query = query,
    From = 0,
    Size = 10,
    Aggregations = aggregations
};

var result = client.SearchAsync<InventoryLiveView>(searchRequest).Result;

var years = result.Aggregations.Terms(nameof(fields.Year));
Dictionary<string, long> yearCounts = new Dictionary<string, long>();
foreach (var item in years.Buckets)
{
    yearCounts.Add(item.Key, item.DocCount ?? 0);
}

The exact error from elasticsearch, which I saw using postman was:

Fielddata is disabled on text fields by default. Set fielddata=true on [MakeName] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

Shiekh

Answer 2

Here is my example using SearchDescriptors. My only problem is how to serialize returned results into a proper Key Value list. Is Looping through a fields list the best way to return results.

SearchDescriptor<Advert> agghDescriptor = new SearchDescriptor<Advert>();

agghDescriptor.Aggregations(ag => ag.Terms("make", a => a.Field(f => f.Make)) &&
                                  ag.Terms("region", a => a.Field(f => f.Region)) &&
                                  ag.Terms("city", a => a.Field(f => f.City)) &&
                                  ag.Terms("category", a => a.Field(f => f.Category)) &&
                                  ag.Terms("application", a => a.Field(f => f.Application)) &&
                                  ag.Terms("portalId", a => a.Field(f => f.PortalId)) &&
                                  ag.Terms("isActiveAuctionAdvert", a => a.Field(f => f.IsActiveAuctionAdvert)) &&
                                  ag.Terms("isBargainAccount", a => a.Field(f => f.IsBargainAccount)) &&
                                  ag.Terms("condition", a => a.Field(f => f.Condition))
);
agghDescriptor.Size(0);

var json2 = _client.RequestResponseSerializer.SerializeToString(agghDescriptor);
var aggregationResult = _client.Search<Advert>(agghDescriptor);

List<string> fields = new List<string>();
fields.Add("make");
fields.Add("category");
fields.Add("region");

List<Aggregation> aggregations = new List<Aggregation>();

foreach (var field in fields)
{
    var aggrs = aggregationResult.Aggregations.Terms(field);
    List<AggregateItem> aggregateItems = new List<AggregateItem>();
    
    foreach (var item in aggrs.Buckets)
    {
        aggregateItems.Add(new AggregateItem()
        {
            Count = item.DocCount ?? 0,
            Key = item.Key
        });
    }

    aggregations.Add(new Aggregation()
    {
        Name = field,
        Aggregates = aggregateItems
    });
}

Berard