Is there a way to increment counter in prometheus for similar metrics scraped from pushgateway [closed]

Question

I have a running instance of pushgateway and prometheus. And prometheus is configured to scrape from pushgateway. I am able to push metrics from my .NET applications to pushgateway which are successfully scraped by my prometheus. But when a two similar metrics are pushed to pushgateway, they are considered as one by the prometheus and the count is always displayed as one. Is there a way to increment the prometheus count value when similar metrics are present in the pushgateway.

Daugava

Answer 1

Pushgateway is not designed to handle this situation. As the official doc said, Usually, the only valid use case for the Pushgateway is for capturing the outcome of a service-level batch job.

https://prometheus.io/docs/practices/pushing/#should-i-be-using-the-pushgateway

Or you can use this instead, which is a aggregating push gateway for Prometheus: https://github.com/weaveworks/prom-aggregation-gateway

Megilp

Answer 2

The metrics from the two apps should be separated by adding a label that will distinguish between them. This should solve your issue. Note that the pushgateway docs explicitly state that there should be no duplicate metrics.

Pituri

I don't want them to be considered different. My actual use case is to increment the count when similar metrics are being scraped from prometheus – Daugava

In that case you should push 2 different metrics to pushgateway, and sum them together when you query them, across the different labels. I'll add a reference to my original answer. – Pituri

Answer 3

As @ImPeratOR_ mentioned, prom aggregation gateway is a solid alternative to Pushgateway but I ran into a couple of issues with it such as they additively combines gauges(idk why) and there's no TTL on metrics.

I created and use Metrics Accumulator instead.

Metrics Accumulator

1. handles counters and gauges. Counters are additively combined and the gauges are last one in are saved unless you use a special label.
2. Has partitions for your metrics known as metric groups
3. TTL on metric groups.
4. HTTP Service discovery mechanism

Both can be deployed as a standalone service.

For full transparency I'm the creator of Metrics Accumulator.

Solingen