Removing Data From ElasticSearch

Question

I want to remove data from ElasticSearch. I have deleted my indexes. However, that doesn't seem to actually remove the data itself. The other stuff I've seen points to the Delete by Query feature. However, I'm not even sure what to query on. I know my indexes. Essentially, I'd like to figure out how to do a

DELETE FROM [Index]

From PostMan in Chrome. However, I'm not having any luck. It seems like no matter what I do, the data hangs around. Thus far, I've successfully deleted the indexes by using the DELETE HTTP Verb in PostMan and using a url like:

   http://localhost:9200/[indexName]

However, that doesn't seem to actually remove the data (aka docs) themselves.

Opium

I check this with postman and got reposne as "{ "acknowledged": true }" If you see this acknowledged response don't worry. The index is removed from elastic. – Unaesthetic

Answer 1

You can delete using cURL or visually using one of the many tools that open source enthusiasts have created for Elasticsearch.

Using cURL

curl -XDELETE localhost:9200/index/type/documentID

e.g.

curl -XDELETE localhost:9200/shop/product/1

You will then receive a reply as to whether this was successful or not. You can delete an entire index or types with an index also, you can delete a type by leaving out the document ID like so -

curl -XDELETE localhost:9200/shop/product

If you wish to delete an index -

curl -XDELETE localhost:9200/shop

If you wish to delete more than one index that follows a certain naming convention (note the *, a wildcard), -

curl -XDELETE localhost:9200/.mar* 

Visually

There are various tools as mentioned above, I wont list them here but I will link you to one which enables you to get started straight away, located here. This tool is called Cerebro (former KOPF), to connect to your host please click on the logo on top left hand corner and enter the URL of your cluster.

Once connected you will be able to administer your entire cluster, delete, optimise and tune your cluster.

Construct

is there any way I can delete 3 doc's of which id I know. – Bering

@JayeshJain to my current knowledge, no. You could put 3 modified curl -XDELETE commands into a bash script and execute or run 3 one after the other. – Construct

@JayeshJain so curl -XDELETE localhost:9200/index/type/docid1 // curl -XDELETE localhost:9200/index/type/docid2 // curl -XDELETE localhost:9200/index/type/docid3 – Construct

i did it the same way.but I was just thinking if there is a smarter way of deleting multiple docs. I could use term if I knew the field. But In this scenario,i just need to delete docs by their id. Thx anyways – Bering

How can I delete an index with an invalid character, e.g., logstash-eu-%{customer}-2016.11.22. I want to delete ALL indices logstash-eu-%{customer}-* or logstash-eu-%* – Overlay

curl: (7) Failed to connect to localhost port 9200: Connection refused – Salve

@Salve your elasticsearch instance may not be listening on localhost? – Construct

Shouldn’t there be a space between the -X and DELETE ? – Dormitory

Answer 2

If you ever need to delete all the indexes, this may come in handy:

curl -X DELETE 'http://localhost:9200/_all'

Powershell:

Invoke-WebRequest -method DELETE http://localhost:9200/_all

Note: This will delete all data, including your x-pack access credentials and Kibana dashboard and visualizations

Proportionate

this is very useful for development and needing to reset back to scratch (empty) database. Thanks!! – Perkins

in your bash_profile create an alias for this command and it will come in handy for development. – Deformity

'Wildcard expressions or all indices are not allowed' – Dally

Note that this will delete all data, including your x-pack access credentials. – Oleaceous

This is also deleting Kibana dashboards and visualizations – Burkett

I'm using AWS ES and when I run this command, I seem unable to recreate index patterns in Kibana UI. – Farceur

After doing this on a local Docker instance I'm getting an internal server error when going to localhost:5601 ("Error: Internal Server Error\n at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:61:19)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:177:34)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n at exports.Manager.execute [...cut...]") – Fret

Thanks, I could regex and delete the specific group using [curl -X DELETE 'localhost:9200/csvtest*'] – Motorboating

come in handy ... to remove everything: dashboards, x-pack access credentials – Rollway

Does deleting all indices equate to deleting all documents? Will there be any data left after running this command? – Sleepy

This answer still doesn't work in 8.9.0, same error as @Dally – Discretion

Answer 3

The documentation (or The Definitive Guide) says, that you can also use the next query to delete all indices:

curl -XDELETE 'http://localhost:9200/*'

And there's an important note:

For some, the ability to delete all your data with a single command is a very scary prospect. If you want to eliminate the possibility of an accidental mass-deletion, you can set the following to true in your elasticsearch.yml:

action.destructive_requires_name: true

Doublebreasted

Answer 4

You have to send a DELETE request to

http://[your_host]:9200/[your_index_name_here]

You can also delete a single document:

http://[your_host]:9200/[your_index_name_here]/[your_type_here]/[your_doc_id]

I suggest you to use elastichammer.

After deleting you can look up if the index still exists with the following URL: http://[your_host]:9200/_stats/

Good luck!

Jocasta

what is the way to delete indices older than 10 days ? I can not use curator because my server is not support. – Pastose

After deleting the index, you can also check its presence at http://localhost:9200/_cat/indices – Implication

Answer 5

#list all index:       curl -XGET http://localhost:9200/_cat/indices?v 

#delete index:         curl -XDELETE 'localhost:9200/index_name'
#delete all indices:   curl -XDELETE 'localhost:9200/_all'
#delete document   :   curl -XDELETE 'localhost:9200/index_name/type_name/document_id'

Install kibana. Kibana has a smarter dev tool which helps to build query easily.

Agility

what is the way to delete indices older than 10 days ? I can not use curator because my server is not support. – Pastose

Answer 6

Deleting the index will delete the mapping and type along. you can delete all rows by the following query

curl -XDELETE 'localhost:9200/twitter/tweet/_query?pretty' -d'
{
   "query": { 
      "match_all": 
   }
}'

However for above query you need to install delete-by-query plugin as of Elasticsearch's 2.0.0-beta1 delete-by-query was removed from main api

Install delete-by-query plugin

sudo bin/plugin install delete-by-query

For more

http://blog.appliedinformaticsinc.com/how-to-delete-elasticsearch-data-records-by-dsl-query/

Laissezfaire

Both before and after installing the plugin and restarting ES, I get "No handler found for uri and method". – Chiles

This does not work in Elasticsearch 6+. Use _delete_by_query instead. – Haulm

Answer 7

curl -X DELETE 'https://localhost:9200/_all'

Change http to https if you are using SSL certificate in you application

Gillum

Answer 8

You can delete the index by Kibana Console:

To get all index:

GET /_cat/indices?v

To delete a specific index:

DELETE /INDEX_NAME_TO_DELETE

Separation

Answer 9

To list down the indices curl -L localhost:9200/_cat/indices

9200 default port[change the port if using some other port]

You will likely find all indices starting with logstash-yyyy-mm-dd format(logstash-*)

You can see all the indices and use

To delete the indices and data trigger following command.

curl -XDELETE localhost:9200/index_name (Which will remove the data and indices both).

Kloof

Answer 10

You can delete an index in python as follows

from elasticsearch import Elasticsearch

es = Elasticsearch([{'host':'localhost', 'port':'9200'}])

es.index(index='grades',doc_type='ist_samester',id=1,body={
    "Name":"Programming Fundamentals",
    "Grade":"A"
})

es.indices.delete(index='grades')

Oomph

Answer 11

simplest way !

Endpoint :
http://localhost:9201/twitter/_delete_by_query

Payload :
{
  "query": { 
    "match": {
      "message": "some message"
    }
  }
}

where twitter is the index in elastic search

ref ; https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html

Binnie

Answer 12

1. Delete API

Removes a document from the specified index.

DELETE /<index>/_doc/<_id>

Example:

DELETE http://localhost:9200/my-index-000001/_doc/1

Reference : ES Guide >> Delete API

2. Delete by query API

Deletes documents that match the specified query.

Example:

POST http://localhost:9200/my-index-000001/_delete_by_query
{
  "query": {
      "match": {
           "user.id": "elkbee"
      }
   }
}

Reference : ES Guide >> Delete by query API

Garter

Answer 13

For mass-delete by query you may use special delete by query API:

$ curl -XDELETE 'http://localhost:9200/twitter/tweet/_query' -d '{
    "query" : {
        "term" : { "user" : "kimchy" }
    }
}

In history that API was deleted and then reintroduced again

Who interesting it has long history.

1. In first version of that answer I refer to documentation of elasticsearch version 1.6. In it that functionality was marked as deprecated but works good.
2. In elasticsearch version 2.0 it was moved to separate plugin. And even reasons why it became plugin explained.
3. And it again appeared in core API in version 5.0!

Gev

Be careful using delete by query. Its deprecated for a major reason. OutOfMemoryError! – Bagworm

Sure. But you may spy if it happened for you or you have anough memory. – Gev

This does not deprecated anymore: elastic.co/guide/en/elasticsearch/reference/6.4/… – Gev

Answer 14

You can delete either whole index,doc-type or a perticular id data. these are the three ways:

1. curl -XDELETE localhost:9200/index_name

2. curl -XDELETE localhost:9200/index_name/doc-type

3. curl -XDELETE localhost:9200/index_name/doc-type/documentId

and if you wish to delete all the index then go for wildcard.

Corriecorriedale

Hi, i hope before executing queries you started your elastic search and make sure by default it binds to all local addresses. and instead of localhost you can also use your IP address. like 10.80.15.45:9200 And once check your ES setting I can think to check in network.bind_host and make sure it is either not set or is set to 0.0.0.0 or ::0 or to the correct IP address for your network. – Corriecorriedale

Answer 15

I wanted to delete logstash index and searched a lot regarding different tools like curl. But found the solution at the end. Login into Kibana. Go to Dev Tools tab and type DELETE /logstash-* in query field and hit green arrow button. if you get "acknowledged": true in response that means the data has been cleared.

Rhea

THANK YOU!!! i tried many other options- this is the only one that worked for me. – Raynaraynah

Answer 16

You can delete one or more indices, which really deletes their files from disk. For example:

curl -XDELETE localhost:9200/$INDEXNAME

Where $INDEXNAME can be an index name (e.g. users_v2), N indices separated by comma (e.g. users_v2,users_v3). An index pattern (e.g. users_*) or _all, also works, unless it's blocked in the config via action.destructive_requires_name: true.

Deleting individual documents is possible, but this won't immediately purge them. A delete is only a soft delete, and documents are really removed during segment merges. You'll find lots of details about segments and merges in this presentation. It's about Solr, but merges are from Lucene, so you have the same options in Elasticsearch.

Back to the API, you can either delete individual documents by ID (provide a routing value if you index with routing):

curl -XDELETE localhost:9200/users_v2/_doc/user1

Or by query:

curl -XPOST -H 'Content-Type: application/json' localhost:9200/users_v2/_delete_by_query -d '{
  "query": {
    "match": {
      "description_field": "bad user"
    }
  }
}'

Hydrogenous

Answer 17

There are lots of good answers here, but there is also something i'd like to add:

• If you are running on AWS ElasticSearch service, you can´t drop/delete indexes. Instead of delete indexes, you must reindex them.

Chine

Just deleted an index on AWS ElasticSearch, my domain is running ES 5.1. – Highbrow

On AWS ES you can't open/close indexes - that requires reindexing. You can, however, delete indexes. I've only done it through the Kibana console, but it definitely works. – Dwight

Answer 18

I used Dev Tools to delete data

POST <index_name>/_delete_by_query
  {
   "query": {
       "match_all": {}
    }
 }

Example

POST vehicle-data/_delete_by_query
      {
       "query": {
           "match_all": {}
        }
     }

Bimetallic

Answer 19

You can also delete the index using DELETE action in 'elasticsearch head' (Chrome plugin ). Add it to your chrome and connect it to your host. There you will find all your indices and if you click on actions button below the index you want to delete, you will find a DELETE option in the drop down. click on it and enter DELETE in the pop-up. Your index will be deleted. 'Elasticsearch head' extension is an easy way to view and manage your indices and data.

Comprador

Answer 20

You can also use chrome extension elasticsearch-head to delete index

Gib

Answer 21

Say I need to delete an index filebeat-7.6.2-2020.04.30-000001 and I performed it using a curl DELETE option (curl -X DELETE "localhost:9200/filebeat-7.6.2-2020.04.30-000001?pretty") and results in an authentication problem as below;

{
  "error" : {
    "type" : "security_exception",
    "reason" : "missing authentication credentials for REST request [/filebeat-7.6.2-2020.04.30-000001?pretty]"
  },
  "status" : 401
}

Here you should authenticate the curl request using the username and password you have provided for Elasticsearch. Try then

curl -X DELETE -u myelasticuser:myelasticpassword "localhost:9200/filebeat-7.6.2-2020.04.30-000001?pretty"

will results in { "acknowledged" : true }.

Incapacity

Answer 22

Adding to the delete_by_query suggestion, if you want to quickly delete all records in a given index it may be also important to set the scroll_size and conflicts parameters.

• Scroll size
  • It is useful to use a small scroll size to avoid request timeouts while trying to delete many documents at the same time.
• Conflicts
  • It is useful to ignore conflicts during batch deletions, because version conflicts are common if you are deleting too many records at once.

---

 POST http://localhost:9200/my-index-000001/_delete_by_query?scroll_size=100&conflicts=proceed
 {
   "query": {
       "match_all": {}
    }
 }

Michal

Answer 23

You can try this curl:

curl --location --request DELETE 'http://<username>:<password>@<url>:9200/<index name>/<doc type>/<document id>

Or if you do not want to set the username and password at URL, then can also try this curl:

curl --location --request DELETE 'http://<url>:9200/<index name>/<doc type>/<document id>' --header 'Authorization: Basic <Base64 encoded username:password>'

After executing the response body will contains a result field. If this field's value is deleted, then it means that the document deleted successfully.

In this curl I assumed that you have configured your elastic to use http. If you are using https, simply change the protocol to https.

Angeles

Answer 24

A python script in order to delete all indexes:

import requests
import json

ES_HOST = "http://localhost:9200"
ES_URL = f"{ES_HOST}/_cat/indices?format=json"

indexes = requests.get(ES_URL).content
indexes = json.loads(index.decode())

for i in index:
    index_name = i['index']
    content = requests.delete(f"{ES_HOST}/{index_name}").content
    print(content)

Hydrogen

Answer 25

I am using Kibana as a tool for viewing and searching elastic search data and I recommend it based on good user experience.

Following queries would help remove data from elastic search -

Case 1: If you know the id of that data row:

DELETE /index_name/_doc/{id}

Case 2: Delete based on a column value: Suppose I have a column named abc.

Now I have to get _id field based in column_name to finally delete for that row.

GET /uts_checkout_configurations/_search { "query" : { "constant_score" : { "filter" : { "bool": { "must": {"exists": {"field": "ABC"}} } } } } }

Finally you would get your data with _id as one the fields in it.

Delete based on that -

DELETE /index_name/_doc/{_id}

Sycophancy

Answer 26

To delete all red indices you can execute:

es_host=127.0.0.1:9210
curl -s "$es_host/_cat/indices?v" |  grep red | tr -s " " | cut -d" " -f3 | xargs -I '{}' curl -XDELETE $es_host/{}

Festal

Answer 27

The year is 2023, I came across this question as I had the same problem, however, I needed to remove an item only from an index.

Scenario data:

Index name: requisitions-index
Item Id I needed to remove: 3d70b8eb-0889-445e-9e17-d5d96a97febc
Item type: _doc

CURL command used:

curl --request DELETE \
  --url http://localhost:9201/requisicoes-index/_doc/3d70b8eb-0889-445e-9e17-d5d96a97febc \
  --header 'Accept: application/json' \
  --header 'Authorization: Basic AWthc6GpYzpjvGFuZ2RhXW==' \
  --header 'Content-Type: application/json'

Remembering that the Elasticsearch instance required authentication, so I sent the authentication data using BASIC in the Insomnia tool

Details in the images below:

Insomnia client - for CURL DELETE command:

Indevout

Answer 28

Case 1: If you want to delete the index and its document too then,

command : DELETE /INDEX_NAME_TO_DELETE

example : DELETE student here student is index name.

Case 2: if you want delete only documents not index then,

command:

POST INDEX_NAME/_delete_by_query
{
  "query": {
    "match_all": {}
  }
}

Example :

POST students/_delete_by_query
{
  "query": {
    "match_all": {}
  }
}

Note : Use the POST method in case 2.

Bookrack