Login/Register
ElasticSearch: inner_hits and hightlight_query
Asked Answered
Solved elasticsearchelasticsearch-2.0
L

1

8

Short version: Is it somehow possible to obtain the functionality delivered by highlight_query for the highlighting of inner_hits results?

Long version: Please consider the following mapping:

{
  "settings": {
    "number_of_replicas": 0,
    "number_of_shards": 1
  },
  "mappings": {
    "docs": {
      "properties": {
        "doctext": {
          "type": "string",
          "store": "yes"
        },
        "sentences": {
          "type": "nested",
          "properties": {
            "text": {
              "type": "string",
              "store": "yes"
            }
          }
        }
      }
    }
  }
}

As you can see, there is the doctext and the sentences field. The idea is that the document text is split into sentences to allow for a sentence-based search.

Let this be an example document:

{
  "doctext": "I will do a presentation. I talk about lions and show images of zebras. I hope it will be fun.",
  "sentences": [
    {
      "text": "I will do a presentation."
    },
    {
      "text": "I talk about lions and show images of zebras."
    },
    {
      "text": "I hope it will be fun."
    }
  ]
}

Now I can search the whole text as well as the single sentences and I can even highlight both:

{
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "doctext": "zebras"
          }
        },
        {
          "nested": {
            "path": "sentences",
            "query": {
              "match": {
                "sentences.text": "zebras"
              }
            },
            "inner_hits": {
              "highlight": {
                "fields": {
                  "sentences.text": {}
                }
              }
            }
          }
        }
      ]
    }
  },
  "_source": false,
  "highlight": {
    "fields": {
      "doctext": {
        "highlight_query": {
          "match": {
            "doctext": "lions"
          }
        }
      }
    }
  }
}

Please not the following:

  1. The nested query on sentences
  2. The inner_hits part of that query
  3. The highlight_query part of second highlight, NOT the one in the inner_hits

Issuing this query will result in this response:

"hits": [
      {
        "_index": "documents",
        "_type": "docs",
        "_id": "123456",
        "_score": 0.6360315,
        "highlight": {
          "doctext": [
            "I will do a presentation. I talk about <em>lions</em> and show images of zebras. I hope it will be fun."
          ]
        },
        "inner_hits": {
          "sentences": {
            "hits": {
              "total": 1,
              "max_score": 0.5291085,
              "hits": [
                {
                  "_index": "documents",
                  "_type": "docs",
                  "_id": "123456",
                  "_nested": {
                    "field": "sentences",
                    "offset": 1
                  },
                  "_score": 0.5291085,
                  "_source": {
                    "text": "I talk about lions and show images of zebras."
                  },
                  "highlight": {
                    "sentences.text": [
                      "I talk about lions and show images of <em>zebras</em>."
                    ]
                  }
                }
              ]
            }
          }
        }
      }
    ]

Please note how for the doctext field, lions is highlighted although we searched for zebras. The inner_hits do highlight those since we did not specify something else to do. But I WANT the inner hits to highlight lions, just as the doctext highlighting does.

I tried to change the inner_hits part of the query to 

 "inner_hits": {
          "highlight": {
            "fields": {
              "text": {
                "highlight_query": {
                  "match": {
                    "sentences.text": "lions"
                  }
                }
              }
            }
          }
        }

But this leads to the following exception:

Failed to execute phase [query_fetch], all shards failed; shardFailures {[9-pMHRPsRiyITgsRNFnkEA][documents][0]: RemoteTransportException[[Fafnir][127.0.0.1:9300][indices:data/read/search[phase/query+fetch]]]; nested:         SearchParseException[failed to parse search source [{
      "query": {
        "bool": {
          "should": [
            {
              "match": {
                "doctext": "fun"
              }
            },
            {
              "nested": {
                "path": "sentences",
                "query": {
                  "match": {
                    "sentences.text": "zebras"
                  }
                },
                "inner_hits": {
                  "highlight": {
                    "fields": {
                      "sentences.text": {
                        "highlight_query": {
                          "match": {
                            "doctext": "lions"
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          ]
        }
      },
      "_source": false,
      "highlight": {
        "fields": {
          "doctext": {
            "highlight_query": {
              "match": {
                "doctext": "lions"
              }
            }
          }
        }
      }
    }]]; nested: NullPointerException; }
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(        TransportSearchTypeAction.java:228)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(    TransportSearchTypeAction.    java:174)
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:46)
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:821)
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:799)
        at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:361)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
    Caused by: ; nested: NullPointerException;
        at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:382)
    at org.elasticsearch.action.search.SearchPhaseExecutionException.guessRootCauses(SearchPhaseExecutionException.    java:152)
        at org.elasticsearch.action.search.SearchPhaseExecutionException.getCause(SearchPhaseExecutionException.java:99)
        at java.lang.Throwable.printStackTrace(Throwable.java:665)
        at java.lang.Throwable.printStackTrace(Throwable.java:721)
        at org.apache.log4j.DefaultThrowableRenderer.render(DefaultThrowableRenderer.java:60)
        at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:87)
        at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:413)
        at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:313)
        at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
        at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
        at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
        at org.apache.log4j.Category.callAppenders(Category.java:206)
        at org.apache.log4j.Category.forcedLog(Category.java:391)
        at org.apache.log4j.Category.log(Category.java:856)
        at org.elasticsearch.common.logging.log4j.Log4jESLogger.internalInfo(Log4jESLogger.java:125)
        at org.elasticsearch.common.logging.support.AbstractESLogger.info(AbstractESLogger.java:90)
        at org.elasticsearch.rest.BytesRestResponse.convert(BytesRestResponse.java:131)
        at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)
        at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:87)
        at org.elasticsearch.rest.action.support.RestActionListener.onFailure(RestActionListener.java:60)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.raiseEarlyFailure(        TransportSearchTypeAction.java:316)
        ... 10 more
    Caused by: java.lang.NullPointerException
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:258)
        at org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:116)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:257)
        at org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:303)
        at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:206)
        at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:201)
        at org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:831)
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:651)
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:617)
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:460)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(        SearchServiceTransportAction.java:392)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(        SearchServiceTransportAction.java:389)
        at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        ... 3 more

Is there any way to make this work? Did I get the DSL wrong on this one? The documentation on inner_hits only states highlighting would work (https://www.elastic.co/guide/en/elasticsearch/reference/2.1/search-request-inner-hits.html) but does not go into any specifics.

Thanks very much for reading and any hints!

Liberate answered 11/12, 2015 at 10:50 Comment(0)
J
11

A good question looks like highlight-query is not supported withing the inner-hits if it in a query context.Not sure why this is so most probably is a bug.

A workaround is to wrap the highlighting nested query in the final highlight query. However it would require the original nested query to be the filter: Example workaround for the highlighting in the OP is shown below :

put test/docs/1
{
  "doctext": "I will do a presentation. I talk about lions and show images of zebras. I hope it will be fun.",
  "sentences": [
    {
      "text": "I will do a presentation."
    },
    {
      "text": "I talk about lions and show images of zebras."
    },
    {
      "text": "I hope it will be fun for lions"
    }
  ]
}


post test/_search
{
   "query": {
      "bool": {
         "should": [
            {
               "match": {
                  "doctext": "zebras"
               }
            },
            {
               "nested": {
                  "path": "sentences",
                  "query": {
                     "match": {
                        "sentences.text": "zebras"
                     }
                  },
                  "inner_hits": {
                     "highlight": {
                        "fields": {
                           "sentences.text": {}
                        }
                     }
                  }
               }
            }
         ]
      }
   },
   "_source": false,
   "highlight": {
      "fields": {
         "*": {
            "highlight_query": {
               "bool": {
                  "should": [
                     {
                        "match": {
                           "doctext": "lions"
                        }
                     },
                     {
                        "nested": {
                           "path": "sentences",
                           "query": {
                              "filtered": {
                                 "query": {
                                    "match": {
                                       "sentences.text": "lions"
                                    }
                                 },
                                 "filter": {
                                    "query": {
                                       "match": {
                                          "sentences.text": "zebras"
                                       }
                                    }
                                 }
                              }
                           },
                           "inner_hits": {
                              "highlight": {
                                 "fields": {
                                    "sentences.text": {}
                                 }
                              }
                           }
                        }
                     }
                  ]
               }
            }
         }
      }
   }
}

Result:

{
   "took": 1,
   "timed_out": false,
   "_shards": {
      "total": 1,
      "successful": 1,
      "failed": 0
   },
   "hits": {
      "total": 1,
      "max_score": 0.6360315,
      "hits": [
         {
            "_index": "test",
            "_type": "docs",
            "_id": "1",
            "_score": 0.6360315,
            "highlight": {
               "doctext": [
                  "I will do a presentation. I talk about <em>lions</em> and show images of zebras. I hope it will be fun."
               ]
            },
            "inner_hits": {
               "sentences": {
                  "hits": {
                     "total": 1,
                     "max_score": 0.40240064,
                     "hits": [
                        {
                           "_index": "test",
                           "_type": "docs",
                           "_id": "1",
                           "_nested": {
                              "field": "sentences",
                              "offset": 1
                           },
                           "_score": 0.40240064,
                           "_source": {
                              "text": "I talk about lions and show images of zebras."
                           },
                           "highlight": {
                              "sentences.text": [
                                 "I talk about <em>lions</em> and show images of zebras."
                              ]
                           }
                        }
                     ]
                  }
               }
            }
         }
      ]
   }
}
Jeffryjeffy answered 11/12, 2015 at 18:14 Comment(1)
That works perfectly. A bit more complicated, of course, but does exactly what I want. Thank you very much, I didn't realize I could put it all into the final highlight query. Nice! – Liberate

© 2022 - 2024 — McMap. All rights reserved.