Prometheus metrics getting reset after each server restart or application redeploy : Java Spring Boot Application

Question

I am new to prometheus metrics. I am exposing my application's all endpoints (application Rest endpoint as well as the 3rd party APIs endpoints which my application is using).

As expected, I am getting metrics under "http_server_requests_seconds*"* for my APIs and "http_client_requests_seconds*"* for 3rd party APIs which I am using in /actuator/prometheus

My Questions are:

1. When I restart my server or redeploy my application, all metrics will reset to 0 and then fresh start will happen?
2. If above statement is true, how we get metrics information from past. Ex : I restarted my application today, but in my grafana dashboard , I want to see application metrics from last month?
3. If this metrics reset happens after each server restart or application redeploy, is there a way to avoid this? means via some configuration etc.

Sweeten

Answer 1

When I restart my server or redeploy my application, all metrics will reset to 0 and then fresh start will happen?

Metrics occupy RAM and unless initialized explicitly, they all become NULL upon restart. This means that not only they become zero, they also may not exist for some time after restart. This is because metrics are usually event-driven (e.g. it only appears if a certain event has happened).

If above statement is true, how we get metrics information from past. Ex : I restarted my application today, but in my grafana dashboard , I want to see application metrics from last month?

That is what Prometheus is for. It gathers metrics and stores them in its TimeSeries DataBase (TSDB). There are Grafana Mimir, Thanos, and several other solutions for (long-term) metric storage.

If this metrics reset happens after each server restart or application redeploy, is there a way to avoid this? means via some configuration etc.

Loosing metrics on restart is fine, since Prometheus keeps the history for you. However, there are use cases when persistent metrics is a necessity. For example, if you have a script that starts once a day and does not stay up for long (e.g. backup for example), then Prometheus may be not fast enough to scrape data before the app is down. In this case you might want to use either Pushgateway or NodeExporter's Textfile Collector to make metrics persistent.

Pani