How to provide highlighting with Spring data elasticsearch

Question

it seems that SpringData ES don't provide classes to fetch highlights returned by ES. Spring Data can return Lists of Objects but the highlights sections in the Json returned by ES is in a separated part that is not handled by the "ElasticSearchTemplate" class.

Code example :-

QueryBuilder query = QueryBuilders.matchQuery("name","tom"); 
SearchQuery searchQuery =new NativeSearchQueryBuilder().withQuery(query).
                               with HighlightFields(new Field("name")).build();
List<ESDocument> publications = elasticsearchTemplate.queryForList
                                                (searchQuery, ESDocument.class);

I might be wrong, but I can't figure out to do only with SpringDataES. Someone can post an example of how we can get highlights with Spring Data ES ?

Thanks in advance !

Vellicate

Answer 1

From the test cases in spring data elasticsearch I've found solution to this :

This can be helpful.

@Test
public void shouldReturnHighlightedFieldsForGivenQueryAndFields() {

    //given
    String documentId = randomNumeric(5);
    String actualMessage = "some test message";
    String highlightedMessage = "some <em>test</em> message";

    SampleEntity sampleEntity = SampleEntity.builder().id(documentId)
            .message(actualMessage)
            .version(System.currentTimeMillis()).build();

    IndexQuery indexQuery = getIndexQuery(sampleEntity);

    elasticsearchTemplate.index(indexQuery);
    elasticsearchTemplate.refresh(SampleEntity.class);

    SearchQuery searchQuery = new NativeSearchQueryBuilder()
            .withQuery(termQuery("message", "test"))
            .withHighlightFields(new HighlightBuilder.Field("message"))
            .build();

    Page<SampleEntity> sampleEntities = elasticsearchTemplate.queryForPage(searchQuery, SampleEntity.class, new SearchResultMapper() {
        @Override
        public <T> Page<T> mapResults(SearchResponse response, Class<T> clazz, Pageable pageable) {
            List<SampleEntity> chunk = new ArrayList<SampleEntity>();
            for (SearchHit searchHit : response.getHits()) {
                if (response.getHits().getHits().length <= 0) {
                    return null;
                }
                SampleEntity user = new SampleEntity();
                user.setId(searchHit.getId());
                user.setMessage((String) searchHit.getSource().get("message"));
                user.setHighlightedMessage(searchHit.getHighlightFields().get("message").fragments()[0].toString());
                chunk.add(user);
            }
            if (chunk.size() > 0) {
                return new PageImpl<T>((List<T>) chunk);
            }
            return null;
        }
    });

    assertThat(sampleEntities.getContent().get(0).getHighlightedMessage(), is(highlightedMessage));
}

Vellicate

Answer 2

Spring Data Elasticsearch 4.0 now has the SearchPage result type, which makes things a little easier if we need to return highlighted results:

This is a working sample:

    String query = "(id:123 OR id:456) AND (database:UCLF) AND (services:(sealer?), services:electronic*)"
    
    NativeSearchQuery searchQuery = new NativeSearchQueryBuilder()
            .withPageable(pageable)
            .withQuery(queryStringQuery(query))
            .withSourceFilter(sourceFilter)
            .withHighlightFields(new HighlightBuilder.Field("goodsAndServices"))
            .build();
    
    
    SearchHits<Trademark> searchHits = template.search(searchQuery, Trademark.class, IndexCoordinates.of("trademark"));
    SearchPage<Trademark> page = SearchHitSupport.searchPageFor(searchHits, searchQuery.getPageable());
    return (Page<Trademark>) SearchHitSupport.unwrapSearchHits(page);

And this would be the response from Page object in json:

{
    "content": [
        {
            "id": "123",
            "score": 12.10748,
            "sortValues": [],
            "content": {
                "_id": "1P0XzXIBdRyrchmFplEA",
                "trademarkIdentifier": "abc234",
                "goodsAndServices": null,
                "language": "EN",
                "niceClass": "2",
                "sequence": null,
                "database": "UCLF",
                "taggedResult": null
            },
            "highlightFields": {
                "goodsAndServices": [
                    "VARNISHES, <em>SEALERS</em>, AND NATURAL WOOD FINISHES"
                ]
            }
        }
    ],
    "pageable": {
        "sort": {
            "unsorted": true,
            "sorted": false,
            "empty": true
        },
        "offset": 0,
        "pageNumber": 0,
        "pageSize": 20,
        "unpaged": false,
        "paged": true
    },
    "searchHits": {
        "totalHits": 1,
        "totalHitsRelation": "EQUAL_TO",
        "maxScore": 12.10748,
        "scrollId": null,
        "searchHits": [
            {
                "id": "123",
                "score": 12.10748,
                "sortValues": [],
                "content": {
                    "_id": "1P0XzXIBdRyrchmFplEA",
                    "trademarkIdentifier": "abc234",
                    "goodsAndServices": null,
                    "language": "EN",
                    "niceClass": "2",
                    "sequence": null,
                    "database": "UCLF",
                    "taggedResult": null
                },
                "highlightFields": {
                    "goodsAndServices": [
                        "VARNISHES, <em>SEALERS</em>, AND NATURAL WOOD FINISHES"
                    ]
                }
            }
        ],
        "aggregations": null,
        "empty": false
    },
    "totalPages": 1,
    "totalElements": 1,
    "size": 20,
    "number": 0,
    "numberOfElements": 1,
    "last": true,
    "first": true,
    "sort": {
        "unsorted": true,
        "sorted": false,
        "empty": true
    },
    "empty": false
}

Invective

why unwrapping anything? The SearchHit class has methods to directly access the highlicht information – Jokester

The api is ugly ... and you can see the retured json has 2 copy of the page data, duplicated ... – Monck

Answer 3

Actually, you could do the following, with a custom ResultExtractor:

QueryBuilder query = QueryBuilders.matchQuery("name", "tom"); 
SearchQuery searchQuery = new NativeSearchQueryBuilder()
                           .withQuery(query)
                           .withHighlightFields(new Field("name")).build();
return elasticsearchTemplate.query(searchQuery.build(), new CustomResultExtractor());

And then

public class CustomResultExtractor implements ResultsExtractor<List<MyClass>> {

private final DefaultEntityMapper defaultEntityMapper;

public CustomResultExtractor() {
    defaultEntityMapper = new DefaultEntityMapper();
}


@Override
public List<MyClass> extract(SearchResponse response) {
    return StreamSupport.stream(response.getHits().spliterator(), false) 
        .map(this::searchHitToMyClass) 
        .collect(Collectors.toList());
}

private MyClass searchHitToMyClass(SearchHit searchHit) {
    MyElasticSearchObject myObject;
    try {
        myObject = defaultEntityMapper.mapToObject(searchHit.getSourceAsString(), MyElasticSearchObject.class);
    } catch (IOException e) {
        throw new ElasticsearchException("failed to map source [ " + searchHit.getSourceAsString() + "] to class " + MyElasticSearchObject.class.getSimpleName(), e);
    }
    List<String> highlights = searchHit.getHighlightFields().values()
        .stream() 
        .flatMap(highlightField -> Arrays.stream(highlightField.fragments())) 
        .map(Text::string) 
        .collect(Collectors.toList());
    // Or whatever you want to do with the highlights
    return new MyClass(myObject, highlights);
}}

Note that I used a list but you could use any other iterable data structure. Also, you could do something else with the highlights. Here I'm simply listing them.

Outstare

Answer 4

https://mcmap.net/q/1891517/-how-to-provide-highlighting-with-spring-data-elasticsearch The first answer does works,but I found some pageable problems with its returned result,which display with the wrong total elements and toalpages.Arter I checkout the DefaultResultMapper implementation, the returned statement shoud be return new AggregatedPageImpl((List<T>) chunk, pageable, totalHits, response.getAggregations(), response.getScrollId(), maxScore);,and then it works with paging.wish i could help you guys~

Pauwles

Answer 5

Please look this answer before read my answer. I run my query like this:

var indexCoordinates = IndexCoordinates.of(ESIndexNames.GLOBAL_SEARCH_INDICIES);
    var searchHits = operations.search(query, Object.class, indexCoordinates);
    return ESSearchHitsUtil.globalSearchPageFrom(searchHits, pageable);

I have parser like this:

public static Page<ESSearchHitResult<Object>> globalSearchPageFrom(SearchHits<Object> hits, Pageable pageable) {

    var content = hits.getSearchHits()
            .stream()
            .map(hit -> new ESSearchHitResult<>(
                    ObjectType.ofIndexName(hit.getIndex()),
                    map2DTO(hit),
                    getHighlightText(hit)
            ))
            .toList();

    return new PageImpl<>(content, pageable, hits.getTotalHits());
}

In my case I need to search from several indexes. You can make your own map2DTO. Here is my getHighlightText

private static Map<String, String> getHighlightText(SearchHit<Object> hit) {
    Map<String, String> result = new HashMap<>();
    try {
        var highlights = hit.getHighlightFields();

        var fieldNameOpt = highlights.keySet().stream()
                .filter(field -> !field.contains(KEYWORD))
                .findFirst();

        if (fieldNameOpt.isPresent()) {
            var fieldName = fieldNameOpt.get();
            var highlightList = highlights.get(fieldName);

            if (!highlightList.isEmpty()) {
                result.put(fieldName, highlightList.get(0));
            }
        }
    } catch (Exception e) {
        log.error("Global search getHighlightText error: ", e);
        return Collections.emptyMap();
    }

    return result.isEmpty() ? Collections.emptyMap() : result;
}

It was my solution, I am ready to hear if there is better solution. I would be glad if the solution safe your time.

Anyone