Elastic search query on indices with some date range

Question

I have below scenario:

I have following indices in elastic search.

• index-2016.04.10
• index-2016.04.11
• index-2016.04.12
• index-2016.04.15
• index-2016.04.16
• index-2016.04.18

Now suppose , I want to search some data in elastic search between dates - 2016.04.11 to 2016.04.16. My questions are:

1. Do we have any way to run a single query and define some filter parameters so that the search will happen only in the indices between these two dates ?

2. If not, then how can we optimize the search query If we need to search the data in some range of indices ?

3. Java implementation.

Please help..

Hydrochloride

You mean to do a date range search between those indices, or to query for data within a certain date range of an indice? – Incomputable

I mean, date range search between those indices. – Hydrochloride

I am facing similar problem in my code. I can get multiple indices using curl as follows ----> -XGET "https: ..........-2019-01-[14-15]/_search/ But not able to select multiple indices in this manner in kibana . – Gwendolin

Answer 1

I see two options.

You specify indexes when you search such as

GET /index-2016.04.10,index-2016.04.11,index-2016.04.12/_search?ignore_unavailable=true
{
    "query": {
        yourquery
    }
}

Or Filter in query (But this approach could be slow, and depends on amount of indexes could throw shard exception, since you will query all indexes which match pattern)

GET /index-*/_search
{
    "query": {
        "terms" : {
            "_index" : ["index1", "index2"]
        }
    }
}

I assume you will be indexing data in the past like last 7, 14, 30 days: so in this case i would definitely go with first approach, do index name calculation in your app

UPDATE 1: To prevent error for non existing indicies you can set flag ignore_unavailable

UPDATE 2: Well i you need to do search in the past one of the solution could be haveing an agregation job.

In ES there is reindex api

POST _reindex
{
  "source": {
    "index": ["twitter", "blog"]
  },
  "dest": {
    "index": "all_together"
  }
}

You will have daily index up to 7 days. Then on Monday 0:0 you aggregate data to weekly index.

You will have weekly up to 5 indexes. Again last day of the month you reindex to monthly index.

In query you combine multiple approach by providing which indexes you want to search and query filter.

Elevator

@Bilyachat, thanks for quick reply, this approach I know, to use this approach we must know all the existing indices, but suppose we want to search all the indices between two dates and It might be that for some dates in between there are no logs, so no index for those dates, in that case, I will get the index not found exception. – Hydrochloride

then in that case you can simply query the logs from all indices based on some timestamp or something for log creation. – Bonar

@Bilyachat, This first approach is good, but I got one problem, If we do the request for last 50 days, in that case we have to add 50 indices, and in this case , I got "Client request error: socket hang up" exception. – Hydrochloride

@pbajpai21 yes this is what i told you there is issue with too many shards. Can you give me examples of how you will choose dates? its random value or you have predefined? – Elevator

Answer 2

You can use aliasses and then query to alias like an index.

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-add-alias.html#indices-add-alias

Affected