How to manually purge data from Graylog 2.1

Question

I have a Graylog 2.1 server that has been running for some time. I hadn't paid attention to my retention rate recently and came in this morning to find Graylog partially crashed because the disk was out of space. Nearly 100% of the disk space is currently being taken up by Elasticsearch Shards. The web interface for Graylog is not currently usable in the state it's in. I tried some of the standard Ubuntu tricks for freeing up disk space like apt-get autoremove and clean, but wasn't able to get enough to get the web interface functional.

The problem is all of the documentation I can currently find for changing the retention rate and cycling the shards, is via the web interface. The only config options no longer appear present in the Graylog config file.

Does anyone know of a manual, CLI, way of purging data from the Elasticsearch Shards in Graylog 2.1?

Naivete

Answer 1

First aid: check which indices are present:

curl http://localhost:9200/_cat/indices

Then delete the oldest indices (you should not delete all)

curl -XDELETE http://localhost:9200/graylog_1
curl -XDELETE http://localhost:9200/graylog_2
curl -XDELETE http://localhost:9200/graylog_3

Fix: You can then reduce the parameter elasticsearch_max_number_of_indices in /etc/graylog/server/server.conf to a value that fits your disk.

Chalcocite

Answer 2

If Elasticsearch is still starting, you can simply delete indices with the Delete Index API, which is, after using Graylog directly (System / Indices page in the web interface), the preferred way of getting rid of Elasticsearch indices.

If you're totally screwed (i. e. neither Graylog, nor Elasticsearch are starting), you can still delete the complete data from Elasticsearch's data path (see Directory Layout).

Backspace

I went to the physical shards which were under /var/lib/elasticsearch/data and located the oldest graylog shards and ended up deleting two of them to free up several GB of space. This freed enough space to get everything working and allowed me to modify my retention policy through the web interface. Thanks. – Naivete

Answer 3

There is list of indexes under graylog admin panel,

"/system/indices"

There is delete button for each index. You can check old indexes and delete them if not required.

You can also delete log files older that 7 days from elastic search,

sudo find /var/log/elasticsearch/ -type f -mtime +7 -delete

Legendary

Answer 4

You should set up a retention strategy from within graylog. If you manage the indices yourself and you delete the wrong index, you might break your graylog.

Go to system/indeces. Select default index set. Select edit index set and there you'll find index rotation and retention.

Avis