How do you debug your Nest queries?

Question

I'm new to Nest, and I very likely am not creating my query like I think I am. My question is more along the lines of teach a man to fish rather than give me a fish. However, I'll use my current problem as an example.

I have several documents in ElasticSearch of the type Series. I'll stub it out below without the attributes and public modifiers with just the information pertinent to the query:

class Series
{
    string Id {get; set;}
    DateTime StartDate {get; set;}
    DateTime EndDate {get; set;}
    HashSet<Role> ReleasableTo {get; set;}
}

These are all fine and dandy. I can Get() a Series object no problem. The problem I'm running into is trying to figure out how Nest is formatting my query. My immediate goal is to find the most recent Series that is releasable to Role.Visitor. I set up the Nest query like this:

ISearchResponse<Series> response = client
    .Search<Series>(r => 
         r.Filter(f => 
             f.Term<Role>(t=>t.ReleasableTo.First(), Role.Visitor))
    .SortDescending(ser => ser.EndDate).Size(1));

In my mind, this should be producing a query that filters the Series so it only considers the ones that are ReleasableTo my Role.Visitor, reverse sorts by end date, and limits the results to one returned. That would be exactly what I want. In the multiple thousands of records I have for Series, about 90% fit this profile. Unfortunately the query returns 0 results. There is no error, just no results. What I don't know is if I'm using the API incorrectly, if Nest is producing a query structure that doesn't make sense, or I simply don't know ElasticSearch well enough. When I remove the Filter clause I get a result, but I'm not guaranteed everyone is allowed to see it.

How do I view the JSON that Nest produces and sends to ElasticSearch?

Ceremony

For future readers, take a look at the documentation on logging and debugging: elastic.co/guide/en/elasticsearch/client/net-api/current/… – Hypertonic

if you have typeORM integration you can try to set logging: true in your orm configuration (docs here) – Agnes

Answer 1

You can get the values of search request URL and JSON request body as under:

var requestURL = response.RequestInformation.RequestUrl;
var jsonBody = Encoding.UTF8.GetString(response.RequestInformation.Request);

You can find other useful properties in RequestInformation for debugging.

Weakness

That's very helpful, for what it's worth, if I do a "query": {"match":{"releasableTo": "Visitor")) I get what I want. I think my index needs some fine tuning. – Ceremony

Answer 2

NEST is Baroque of .NET APIs. For 2.1+ on call level:

IElasticClient client = new ElasticClient();
var searchDescriptor = new SearchDescriptor<Series>();
var query = Query<Series>.Term(...);
var pretty = query.ToPrettyString(query);
var json = client.ToRawRequest(searchDescriptor.Query(descriptor => query));

On configuration level:

    var settings = new ConnectionSettings()
                     .PrettyJson().DisableDirectStreaming()
                     .OnRequestCompleted(details=> Debug.WriteLine(Encoding.UTF8.GetString(details.RequestBodyInBytes)));

On response level look into CallDetails.RequestBodyInBytes.

Used extensions:

    /// <summary>
    /// Converts search to raw JSON request for debugging.
    /// </summary>
    /// <typeparam name="T">The type.</typeparam>
    /// <param name="self">The self.</param>
    /// <param name="searchDescriptor">The search descriptor.</param>
    /// <returns>The string.</returns>
    public static string ToRawRequest<T>(this IElasticClient self, SearchDescriptor<T> searchDescriptor) where T : class
    {
        using (var output = new MemoryStream())
        {
            self.Serializer.Serialize(searchDescriptor, output);
            output.Position = 0;
            var rawQuery = new StreamReader(output).ReadToEnd();
            return rawQuery;
        }
    }

    /// <summary>
    /// Prints query into string.
    /// </summary>
    /// <param name="self">The self.</param>
    /// <returns>The value.</returns>
    public static string ToPrettyString(this QueryContainer self)
    {
        using (var settings = new ConnectionSettings())
        {
            var visitor = new DslPrettyPrintVisitor(settings);
            self.Accept(visitor);
            return visitor.PrettyPrint.Replace(Environment.NewLine, string.Empty);
        }                                                                         
    }

Stricklin

PrettyJson, DisableDirectStreaming, and OnRequestCompleted can all be done in one call with EnableDebugMode in newer versions of nest. See my answer below for example. – Dada

Answer 3

I like to take it a step further than bsarkar suggests and eliminate the need for a roundtrip altogether:

var client = new ElasticClient();

var seriesSearch = new SearchDescriptor<Series>();
seriesSearch.Filter(f => f
    .Term<Role>(t => t.ReleasableTo.First(), Role.Visitor))
    .SortDescending(ser => ser.EndDate)
    .Size(1));

string searchJson = Encoding.UTF8.GetString(client.Serializer.Serialize(seriesSearch));

Note that your ElasticClient doesn't need any connection properties, so you have no dependency on an ES node.

Mesolithic

Note, in NEST 2.4.5 the Serialize function no longer returns a string and must take a target Stream as a parameter. e.g. using (var searchRequestStream = new MemoryStream()) { client.Serializer.Serialize(seriesSearch, searchRequestStream); var searchRequestString = Encoding.UTF8.GetString(searchRequestStream.GetBuffer()); } – Tuning

Answer 4

Really easily. If this is my code that searches:

var results = client.Search<SearchItem>(s => s.AllIndices()
    .Query(q =>
            q.Term(p => p.LastName, searchItem.LastName)
            && q.Term(p => p.FirstName, searchItem.FirstName)
            && q.Term(p => p.ApplicationCode, searchItem.ApplicationCode)
            )
    .Size(1000)
    );
var list = results.Documents.ToList();

I then set a breakpoint on the line above. Then, in Visual Studio Immediate Window, I enter this:

?results.ConnectionStatus

and it gives me this:

{StatusCode: 200, 
    Method: POST, 
    Url: http://localhost:9200/_all/searchitem/_search, 
    Request: {
  "size": 1000,
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "lastName": {
              "value": "carr"
            }
          }
        },
        {
          "term": {
            "firstName": {
              "value": "adrian"
            }
          }
        }
      ]
    }
  }
}

Hope this helps.

Christianachristiane

This is a really useful technique. – Rubricate

Not sure what I am missing, but I see no ConnectionStatus property on my results object – Aliber

Answer 5

Using the latest elastic search 5+, I was able to get mine (thanks to Adrian Carr's method) with the following:

var jsonOutput = System.Text.Encoding.UTF8.GetString(
    response.ApiCall.RequestBodyInBytes
)

Which gave me the following output:

{
   "from":0,
   "size":0,
   "query":{
      "bool":{
      ...
      }
   }
}

Gwendolyn

Answer 6

Tested in Nest 6.2.0

The call EnableDebugMode will automatically call DisableDirectStreaming, PrettyJson, and then OnRequestCompleted, passing in the lambda you give it.

So, now you can just do something like:

connectionSettings.EnableDebugMode(details =>
{
  Logger.Debug($"ES Request: {Encoding.UTF8.GetString(details.RequestBodyInBytes ?? new byte[0])}");
  Logger.Verbose($"ES Response: {Encoding.UTF8.GetString(details.ResponseBodyInBytes ?? new byte[0])}");
});

NOTE: The responses seemed to be lacking closing brackets for me. It would seem there may be a bug somewhere in the nest response stuff. Doesn't seem to affect anything but the logging.

Dada

Answer 7

Elasticsearch.Net and NEST: the .NET clients [7.x]

ref: https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/debug-information.html`

1. Enable DisableDirectStreaming in ES Global setting like following.

ref code

Var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));

var settings = new ConnectionSettings(connectionPool)
    .DisableDirectStreaming(true); 

var client = new ElasticClient(settings);

2.) var response = client.Search<Project>(s =>....)

3.)

var jsonOutput = System.Text.Encoding.UTF8.GetString(
            result.ApiCall.RequestBodyInBytes);

Above can be achieved on a per request basis with following.

var response = client.Search<Project>(s => s
    .RequestConfiguration(r => r
        .DisableDirectStreaming() 
    )
    .Query(q => q
        .MatchAll()
    )
);

Neuter

Answer 8

You can use EnableTrace or ConnectionStatusHandler. More details here.

Hemato

Answer 9

In my current version of Elasticsearch.NET/NEST (7.13.2), I can catch the ElasticsearchClientException and take advantage of the DebugInformation. If DisableDirectStreaming is set like in the other answers then the full request body is included.

try {
    // do some elastic stuff
} catch (ElasticsearchClientException e) {
    Logger.LogInformation(e.DebugInformation);
    // handle error
}

Shirty