How to log all executed elasticsearch queries

Asked 13/2, 2014 at 9:27 Answered 10/1 at 10:30

I want to see all queries executed against an elasticsearch instance. Is it possible to run elasticsearch in a debug mode, or to tell it to store all queries executed against it?

The purpose is to see which queries are launched from a software using elasticsearch for analysis.

Waterage answered 13/2, 2014 at 9:27 Comment(0)

In versions of ElasticSearch prior to 5, you can accomplish this by changing the ElasticSearch.yml configuration file. At the very bottom of this file, you can adjust the logging time to record all:

index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms

index.search.slowlog.threshold.fetch.warn: 1s  
index.search.slowlog.threshold.fetch.info: 800ms
index.search.slowlog.threshold.fetch.debug: 500ms
index.search.slowlog.threshold.fetch.trace: 200ms

index.indexing.slowlog.threshold.index.warn: 10s
index.indexing.slowlog.threshold.index.info: 5s
index.indexing.slowlog.threshold.index.debug: 2s
index.indexing.slowlog.threshold.index.trace: 500ms

Adjust the settings and restart your node, then consulting the logs to view the queries executed against your node. Note if in production log files will rapidly increase in size.

Floweret answered 13/2, 2014 at 10:17 Comment(8)

Does it log only the slow queries? I want to see all queries, they are still not in the log. – Waterage 13/2, 2014 at 12:57

@Waterage Adjust the time period so it will capture all queries rather than just those slow queries. – Floweret 13/2, 2014 at 14:3

To state it more precisely, you should be able to set the time threshold to zero and thus get every query logged. – Coleoptile 21/11, 2014 at 10:44

I am also google for it. Slow log will log the query in segment level not in cluster level... actually if any option to log the queries in cluster level that will be fine. – Shelah 19/12, 2014 at 12:49

This doesn't seem to work in current elastic. I set every one of these to zero and still get nothing in the slowlogs.... – Adhesive 22/4, 2016 at 13:4

do you know if you have to use logging.yml along with elasticsearch.yml for version 5.0.x? – Delectable 8/12, 2016 at 0:13

@Delectable I havent played with the latest elastic search as of, i'm afraid. Edit answer or add to, if you discover :) – Floweret 8/12, 2016 at 10:34

Where exactly is this logged? I mean where are the queries actually output to? – Harim 21/6, 2020 at 11:29

In version 5.x, you have to set slow log logging per index.

Command line:

curl -XPUT 'http://localhost:9200/myindexname/_settings' -d '{
"index.indexing.slowlog.threshold.index.debug" : "0s",
"index.search.slowlog.threshold.fetch.debug" : "0s",
"index.search.slowlog.threshold.query.debug" : "0s"
}'

Or, if you are using Kibana, go to the Dev Tools bar and enter:

PUT /myindexname/_settings 
{"index.indexing.slowlog.threshold.index.debug": "0s", 
"index.search.slowlog.threshold.fetch.debug" : "0s", 
"index.search.slowlog.threshold.query.debug": "0s"}

#1: Apply to ALL indices

You can apply the setting to ALL indices with the following command:

PUT /_all/_settings 
{"index.indexing.slowlog.threshold.index.debug": "0s", 
"index.search.slowlog.threshold.fetch.debug" : "0s", 
"index.search.slowlog.threshold.query.debug": "0s"}

#2: Preserve existing settings

If you don't want to overwrite existing settings, but just add new ones, add '''preserve_existing=true''' after _settings, like this:

PUT /_all/_settings?preserve_existing=true 
{"index.indexing.slowlog.threshold.index.debug": "0s", 
"index.search.slowlog.threshold.fetch.debug" : "0s", 
"index.search.slowlog.threshold.query.debug": "0s"}

The above request will ONLY add the settings if they don't exist. It will not change them if they are already there.

#3: All available log settings

All available slow log settings are here and below for your reference:

PUT /test_index/_settings
{
"index.search.slowlog.threshold.query.warn": "60s",
"index.search.slowlog.threshold.query.info": "5s",
"index.search.slowlog.threshold.query.debug": "1s",
"index.search.slowlog.threshold.query.trace": "0.1s",
"index.search.slowlog.threshold.fetch.warn": "30s",
"index.search.slowlog.threshold.fetch.info": "5s",
"index.search.slowlog.threshold.fetch.debug": "1s",
"index.search.slowlog.threshold.fetch.trace": "0.1s",
"index.indexing.slowlog.threshold.index.warn": "6s",
"index.indexing.slowlog.threshold.index.info": "5s",
"index.indexing.slowlog.threshold.index.debug": "1s",
"index.indexing.slowlog.threshold.index.trace": "0.1s",
"index.indexing.slowlog.level": "info",
"index.indexing.slowlog.source": "1000"
}

Saying answered 17/8, 2017 at 4:59 Comment(2)

In a single line so its convenient next time I'm back (in a month): curl -XPUT 'localhost:9200/_all/_settings' -d '{"index.indexing.slowlog.threshold.index.debug": "0s", "index.search.slowlog.threshold.fetch.debug" : "0s", "index.search.slowlog.threshold.query.debug": "0s"}' – Filiano 28/6, 2019 at 20:46

Note: the output now appears in a logfile including the name "slowlog". For me it was elastic_5_5_index_search_slowlog.log – Vivyanne 18/10, 2019 at 9:2

Starting with Version 5 ElasticSearch charges money for this functionality. It's called "Audit log" and is now part of X-Pack. There is a basic license available that is free, but this license only gives you a simplistic monitoring functionality. Authentication, query logging and all these rather basic things cost money now.

Fining answered 16/1, 2017 at 9:40 Comment(0)

Yes, it's possible to tell Elasticsearch to log all queries executed against it and you can configure logging levels, such as DEBUG. You can change it in ES 7.13.x using curl:

curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
  "transient": {
    "logger.org.elasticsearch.discovery": "DEBUG"
  }
}
'

On macOS log files are stored on $ES_HOME by default. Please check the docs about Elasticsearch Logging

Figwort answered 12/7, 2021 at 21:8 Comment(3)

I feel like this is the correct answer, but I am not so sure logger.org.elasticsearch.discovery is the correct Logging Hierarchy to catch queries. Do you know what are the available Logging Hierarchies? – Thom 8/3, 2022 at 15:52

You're right, the give example gives logs related to the discovery process. I don't know on top of my head the package for the search queries. – Figwort 8/3, 2022 at 19:3

@ElouanKeryell-Even, can you try something like logger.org.elasticsearch.search? Based on the docs, the log hierarchy seems to mimic the package name. See the search package. Please let me know if it works so I can update the answer! – Figwort 8/3, 2022 at 19:10

To update all indexes to log every query, you can set the slow query to 0s:

PUT /_all/_settings
{
  "index.search.slowlog.threshold.query.warn": "0ms",
  "index.search.slowlog.threshold.query.info": "0ms",
  "index.search.slowlog.threshold.query.debug": "0ms",
  "index.search.slowlog.threshold.query.trace": "0ms"
}

To reset this to the default:

PUT /_all/_settings
{
  "index.search.slowlog.threshold.query.warn": "-1",
  "index.search.slowlog.threshold.query.info": "-1",
  "index.search.slowlog.threshold.query.debug": "-1",
  "index.search.slowlog.threshold.query.trace": "-1"
}

Jujube answered 10/1 at 10:30 Comment(0)

Recommended topics

Hot tags