How to aggregate health indicators in Spring boot

Question

I've added Actuator support for my Spring 2.0.4 application using this Baeldung article. In section 4.4 it talks about

A handy feature of health indicators is that we can aggregate them as part of a hierarchy

but it doesn't go into any discussion of how to do this aggregation. Nor have I been able to find any documentation on how to do this.

Question Do any of you know of a tutorial, example or other documentation on creating such an aggregation?

More Info I have a service in my application which relies on several sub-components. The service itself is only considered down if all these sub-components are down. So long as one is up then the service is up. Currently using the normal HealthIndicator mechanism if one of the sub-components is down it marks the server as down.

It seems I would want to use the CompositeHealthIndicator but it's not clear how I create the child HealthIndicators without the system picking them up. The caveat is that each of these sub-components uses the @Scheduled annotation, my understanding is that in order for that to work the class must use the @Component annotation(or some such) which will cause it to be created and sucked up into the application health.

Clarification I have added actuators and the health URL comes up as this:

{"status":"UP","details":{"MyServ1":{"status":"UP","details":{"Latency":...}},"MyServ2":{"status":"UP","details":{"Latency":...}},"diskSpace":{"status":"UP","details":{"total":...,"free":...,"threshold":...}}}}

but if 'MyServ1' or 'MySrv2' are down the overall status is down, but I only want that to happen if 'diskSpace' is down OR 'MyServ1' and 'MyServ2' is down.

It would appear that CompositeHealthIndicator would be the appropriate class for this, it is just unclear how I create the children health indicators for it (just use new)?

Thanks in advance

Peter

you just add that dependency and try http://localhost:port/actuator/health this should work – Lashley

I've done that and the URL works, but as I indicated in the more info section this causes the server to come back as down if any of my sub-components are down, I only want it to show the server as down if all of the sub-components are down (hence the need to aggregate these components) – Peter

CompositeHealthIndicator is deprecated now. This link: docs.spring.io/spring-boot/docs/2.2.0.RELEASE/reference/html//… Says you can create custom health groups in the properties file, but gives no information about how to actually add custom healthchecks to that group... Search the page for: "2.8.5. Health Groups" Also here... spring.io/blog/2019/10/16/… – Virelay

Answer 1

Aggregating the statuses from each health indicator into a single overall status is done by an implementation of org.springframework.boot.actuate.health.HealthAggregator. Spring Boot auto-configures an instance of OrderedHealthAggregator. If you provide your own bean that implements HealthAggregator the auto-configured aggregator will back off in favour of your custom implementation.

The aggregator's aggregate method is called with a Map<String, Status> where the keys are the names of the health indicators and the values are their statuses. Knowing the names of your sub-components' health indicators should allow you to perform custom aggregation for them.

Cod

Thank Andy, doing this makes my HealthAggregator used for the entire system. I want it to be used for my service. As an example, using default health actuator I get the following output from /actuator/health {"status":"UP","details":{"MyServ1":{"status":"UP","details":{...}},"MyServ2":{"status":"UP","details":{"...}},"diskSpace":{"status":"UP","details":{"total":...,"free":...,"threshold":...}}}} Right now, if any 1 of these is down the status is down. But I want to change it so that it is only down if 'diskSpace' is down or both 'myServ1' & 'MyServ2" are down – Peter

My suggestion is that you could implement that logic in a custom HealthAggregator implementation. – Cod

@AndyWilkinson the HealthAggregator class is an internal deprecated class used to determine a single status from a set of statuses. For example (UP, UP, DOWN) equals DOWN. Its not applicable to applying logical groupings of healthchecks in the json before they are aggregated. – Virelay

Answer 2

I came up with a solution, I posted a simple demonstration on GitHub. I have no idea if this is the correct way to do this or not but it seems to be working...

Example Spring Application

Peter

I could not figure out how to get a Health to convert to a Status naturally when upgrading from 2.1.5 up to 2.2.1. SimpleStatusAggregator would not return the needed health item. I ended up using this answer and breaking out my aggregated health checks into individual HealthIndicators. Unfortunate but this helped me past my issue. Thanks :) – Mirna

You can improve your answer by including the core of your approach (writing your own HealthAggregator). Links tend to go away (e.g. if you or GitHub terminate your GitHub account for whatever reason), and finding the core of the idea in a web page takes time, that makes the answer less valuable. – Saez

Answer 3

Well, Spring just rolled out some strange functionality. Here is how you use it!

Construct a couple healthchecks using a new "ReactiveHealthIndicator" class extension.

@Component("myindicator1")
public class CriticalAppHealthIndicator implements ReactiveHealthIndicator {

    @Override
    public Mono<Health> health() {

        return Mono.just(Health.up().build()); // obviously yours will do more..
    }
}

Second one:

@Component("myindicator2")
public class OptionalAppHealthIndicator implements ReactiveHealthIndicator {

    @Override
    public Mono<Health> health() {

        return Mono.just(Health.outOfService().build()); // obviously yours will do more..
    }
}

Now you can add a grouping. Modify your application.properties, or in this example your application.yml to add the "management.endpoint.health.group.{yourgroup}.include" key with a value of comma-delimited bean names.

management:
  endpoint:
    health:
      show-details: always
      # Include custom actuator grouping for any custom health checks with the "my-indicators" qualifier
      group:
        my-indicators:
          include: myindicator1,myindicator2

Now.. this is where it gets a bit strange. Now if you hit your health endpoint:

GET: {base}/health?detail=true

Your response will look like this:

{
  "status" : "DOWN",
  "components" : {
    "indicator2" : {
      "status" : "UP",
    },
    "indicator2" : {
      "status" : "DOWN",
    }
  },
  "groups" : [ "my-indicators" ]
}

Which doesn't actually group the two together like other dependencies are (ex: "db" for any spring-data hibernate groupings of data connections.)

However you can now navigate to:

GET: {base}/health/my-indicators

In order to see just the grouping you've setup.

PLEASE post if you figure out how to get a grouping in the parent endpoint - simply /health.

From what I can gather its setup this way so that groupings could be made dynamically based off of where you're deployed. E.G: Kubernetes..

It is beyond my understanding why they wouldn't apply a grouping at the top-level or give a setting to use a grouping as a default at the base /health level - when no other groupings exist. Making multiple requests to see a group isn't really convenient at-all if you're using postman or CURL. A front end dev could spend a bunch of time with this to make collapsible categories custom front-end, but this is also fraught with problems as the actuator endpoints don't strongly-type their response objects enough to make them work with any OpenAPI 3.0 code generator so today you'll have to disable actuator from being included in the swagger if you want to run any code generation..

References:

• https://spring.io/blog/2019/10/16/spring-boot-2-2-0#health-indicator-groups
• https://docs.spring.io/spring-boot/docs/2.2.0.RELEASE/reference/html//production-ready-features.html#health-groups

Seems to me the Spring Team was a bit lazy with documentation for all of this grouping nonsense.

Virelay

Output in regular health endpoint or even to Prometheus endpoint is a bit tedious: Provide a new SimpleStatusAggregator/ HealthAggregator and wrap its getAggregateStatus-method. Bind a MeterRegistry/ Gauge with appropriate name and publish the aggregated value through the gauge. – Fondea

Thanks for responding, so perhaps there is a solution to this conundrum! How would one find some example code or a link to a good and thorough tutorial or documentation? – Virelay

The aggregator class is already there. Simply extend it, inject a MeterRegistry and override getAggregateStatus. In the override use the MeterRegistry and get a gauge for the status, Save the result of super.getAggregateStatus in a variable. Put the variable into the gauge and return it. (Pattern is called delegate) – Fondea

Answer 4

I was struggling with this issue, and here is my solution. https://github.com/mohamed-taman/Spring-Actuator-health-aggregator

This demo based on the old spring version way of doing Health aggregation and latest spring boot versions 2.3.0.M4. The new idea I have developed based on new APIs in favor of old deprecated ones.

It will give you something like this:

{
   "status":"DOWN",
   "components":{
      "Core System Microservices":{
         "status":"DOWN",
         "components":{
            "Product Service":{
               "status":"UP"
            },
            "Recommendation Service":{
               "status":"DOWN",
               "details":{
                  "error":"java.lang.IllegalStateException: Not working"
               }
            },
            "Review Service":{
               "status":"UP"
            }
         }
      },
      "diskSpace":{
         "status":"UP",
         "details":{
            "total":255382777856,
            "free":86618931200,
            "threshold":10485760,
            "exists":true
         }
      },
      "ping":{
         "status":"UP"
      }
   }
}

I hope this helps you.

Vitale