Prometheus pod crashlooping on error "opening storage failed" [closed]

Question

I restarted the Prometheus pod and now the Prometheus pod is crashlooping. Found this error in the logs:

level=error ts=2021-06-09T09:27:29.066Z caller=main.go:758 err="opening storage failed: block dir: \"/prometheus/01F6J0P4KBBWVJD2M8B1PE7C5E\": open /prometheus/01F6J0P4KBBWVJD2M8B1PE7C5E/meta.json: no such file or directory"

Noticed that the 01F6J0P4KBBWVJD2M8B1PE7C5E folder only has the chunks folder in it.

Any idea why this occurs and is there a way to fix this?

Plucky

Answer 1

So you have already found correct github related issues and asked there also.

• Still seeing "opening storage failed: block dir" on more recent version of prometheus #7090

• does not start up after corrupted meta.json file

And seems currently the only way to fix the problem is to delete 01F6J0P4KBBWVJD2M8B1PE7C5E folder that contains empty meta.json either nothing (link)

Scrubby

Yes, that was the only way to get the pod running. – Plucky