Time based alerts in prometheus alertmanager

Question

I want to set an alert in Prometheus alertmanager that has to be triggered between a given time Eg: Condition to be checked between 09:15 to 15:30

Jacquez

Answer 1

Alerts in Prometheus are evaluated periodically, and you can't really set a schedule for them.

I think it can be acheived with some PromQL Kong Fu:

scalar(clamp(hour() > 9 and hour() < 15, 1, 1)) * <alert_promql>

hour() > 9 and hour() < 15 Define a range of time based on hour of the day (you can add minutes too)

clamp(..., 1, 1) ensure that the value will be 1 and nothing else

* - This is where the magic happens. If we get any value from the previous function it will be 1 so multiplying by 1 has no effect on the second expression. Otherwise, there is no series on the first expression, so the multiplication will return no results anyway.

Austine

Important to note that clamp() was not introduced until 2.26 promlabs.com/blog/2021/04/01/… – Dustin

There's a highly relevant question in #56428308 which argues in favour of using ON() (1 < hour() < 3). – Linda

Answer 2

Another option you can you is active_time_intervals in your alertmanager configuration

# Times when the route should be active. These must match the name of a
# time interval defined in the time_intervals section. An empty value
# means that the route is always active.
# Additionally, the root node cannot have any active times.
# The route will send notifications only when active, but otherwise
# acts normally (including ending the route-matching process
# if the `continue` option is not set).
active_time_intervals:
  [ - <string> ...]

so an example of your configuration would be

global:
  ...
route:
  group_by: ['...']
  receiver: 'black-hole-receiver'
  routes:
    - matchers:
        - env=dev
      continue: true
      receiver: 'black-hole-receiver'
      active_time_intervals:
        - workHours

time_intervals:
  - name: workHours
    time_intervals:
      - times:
          - start_time: 09:00
            end_time: 17:00
        weekdays: ['monday:friday']

receivers:
  - name: 'black-hole-receiver'

Jacquijacquie