ASP.NET Core Health Checks: Returning pre-evaluated results

Question

I'm evaluating the use of Microsoft Health Checks to improve routing of our internal load balancer. So far I'm very happy with the functionality provided by this feature and the community around it. However there's one thing I did not find yet and wanted to ask if it is possible out of the box:

The Health Checks seem to retrieve their own status as soon as they are requested. But because our service might have a hard time processing a lot of request in that given moment, the query to a thrid-party component like the SQL Server might take it's time to respond. Therefore, we would like to pre-evaluate that health check periodically (like every few seconds) and return that state when the health check api gets called.

The reason is, that we want our load balancer to get the health state as quickly as possible. Using pre-evaluated results seems to be good enough for our use case.

Now the question is: Is it possible to add a kind of "poll" or "auto-update" mechanism to the ASP.NET Core health checks? Or does this mean I have to implement my own health check returning values from a background service which pre-evaluates the results periodically?

Please note, I want to use pre-evaluated results on each request which is NOT HTTP Caching where the live result is cached for the next requests.

Carbuncle

You can turn that around and push metrics to a server periodically. HealthCheck supports pushing to eg Prometheus. This package contains a ton of checks and publishers for Prometheus (actually Prometheus Gateway), App Insights, Seq and Datadog – Canal

There's also a direct endpoint for Prometheus to poll. Prometheus does what you described, it polls its sources for events by calling an endpoint. That doesn't work for eg CLI applications, so a Prometheus Gateway is used as a "cache" to hold data published by applications until Prometheus asks for it. – Canal

Answer 1

Short Version

This is already available and can already integrate with common monitoring systems. You may be able to tie Health Check directly into your monitoring infrastructure.

The details

The Health Check middleware covers this by periodically publishing metrics to a target, through any registered classes that implement the IHealthCheckPublisher.PublishAsync interface method.

services.AddSingleton<IHealthCheckPublisher, ReadinessPublisher>();

Publishing can be configured through HealthCheckPublisherOptions. The default period is 30 seconds. The options can be used to add delays, filter the checks to run etc:

services.Configure<HealthCheckPublisherOptions>(options =>
{
    options.Delay = TimeSpan.FromSeconds(2);
    options.Predicate = (check) => check.Tags.Contains("ready");
});

One option would be to cache the results (the HealthReport instance) with a publisher and serve them from another HealthCheck endpoint.

Perhaps a better option would be to push them to a monitoring system like Application Insights or a time-series database like Prometheus. The AspNetCore.Diagnostics.HealthCheck package provides a ton of ready-made checks and publishers for App Insights, Seq, Datadog and Prometheus.

Prometheus uses polling itself. It calls all its registered sources periodically to retrieve metrics. While that works for services, it won't work for eg CLI applications. For that reason, applications can push results to a Prometheus Gateway that caches the metrics until Prometheus itself requests them.

services.AddHealthChecks()
        .AddSqlServer(connectionString: Configuration["Data:ConnectionStrings:Sample"])
        .AddCheck<RandomHealthCheck>("random")
        .AddPrometheusGatewayPublisher();

Apart from pushing to Prometheus Gateway, the Prometheus publisher also offers an endpoint to retrieve live metrics directly, through the AspNetcore.HealthChecks.Publisher.Prometheus package. The same endpoint could be used by other applications to retrieve those metrics :

// default endpoint: /healthmetrics
app.UseHealthChecksPrometheusExporter();

Canal

Thank you so much, Panagiotis. The tip with the publisher was pure gold and led me to a simple solution I have posted as second answer, however I want you to get the solution mark. Prometheus looks great to me btw, but is no option for our environment. – Carbuncle

I didn't know about this until I read the docs. I knew there was a way to publish health data to Prometheus but didn't know publishing was already built-in until I started searching for this question – Canal

I actually stopped at the mapping step because I didn't know Predicate would prevent metrics collection – Canal

Answer 2

Panagiotis answer is brilliant and brought me to an elegant solution I'd love to leave for the next developers stumbling over this ...

To achieve periodical updates without implementing a background service or any timers, I registered an IHealthCheckPublisher. With this, ASP.NET Core will automatically run the registered health checks periodically and publish their results to the corresponding implementation.

In my tests, the health report was published every 30 seconds by default.

// add a publisher to cache the latest health report
services.AddSingleton<IHealthCheckPublisher, HealthReportCachePublisher>();

I registered my implementation HealthReportCachePublisher which does nothing more than taking a published health report and keeping it in a static property.

I don't really like static properties but to me it seems adequate for this use case.

/// <summary>
/// This publisher takes a health report and keeps it as "Latest".
/// Other health checks or endpoints can reuse the latest health report to provide
/// health check APIs without having the checks executed on each request.
/// </summary>
public class HealthReportCachePublisher : IHealthCheckPublisher
{
    /// <summary>
    /// The latest health report which got published
    /// </summary>
    public static HealthReport Latest { get; set; }

    /// <summary>
    /// Publishes a provided report
    /// </summary>
    /// <param name="report">The result of executing a set of health checks</param>
    /// <param name="cancellationToken">A task which will complete when publishing is complete</param>
    /// <returns></returns>
    public Task PublishAsync(HealthReport report, CancellationToken cancellationToken)
    {
        Latest = report;
        return Task.CompletedTask;
    }
}

Now the real magic happens here

As seen in every Health Checks sample, I mapped the health checks to the route /health and use the UIResponseWriter.WriteHealthCheckUIResponse to return a beautiful json response.

But I mapped another route /health/latest. There, a predicate _ => false prevents any health checks to be executed at all. But instead of returning the empty results of zero health checks, I return the previously published health report by accessing the static HealthReportCachePublisher.Latest.

app.UseEndpoints(endpoints =>
{
    // live health data: executes health checks for each request
    endpoints.MapHealthChecks("/health", new Microsoft.AspNetCore.Diagnostics.HealthChecks.HealthCheckOptions()
    {
        ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
    });

    // latest health report: won't execute health checks but return the cached data from the HealthReportCachePublisher
    endpoints.MapHealthChecks("/health/latest", new Microsoft.AspNetCore.Diagnostics.HealthChecks.HealthCheckOptions()
    {
        Predicate = _ => false, // do not execute any health checks, we just want to return the latest health report
        ResponseWriter = (context, _) => UIResponseWriter.WriteHealthCheckUIResponse(context, HealthReportCachePublisher.Latest)
    });
});

This way, calling /health is returning the live health reports, by executing all the health checks on each request. This might take a while if there are many things to check or network requests to make.

Calling /health/latest will always return the latest pre-evaluated health report. This is extremely fast and may help a lot if you have a load balancer waiting for the health report to route incoming requests accordingly.

---

A little addition: The solution above uses the route mapping to cancel the execution of health checks and returning the latest health report. As suggested, I tried to build an further health check first which should return the latest, cached health report but this has two downsides:

• The new health check to return the cached report itself appears in the results as well (or has to be fitered by name or tags).
• There's no easy way to map the cached health report to a HealthCheckResult. If you copy over the properties and status codes this might work. But the resulting json is basically a health report containing an inner health report. That's not what you want to have.

Carbuncle

After all this searching, the complete answer is so simple, I wonder why it wasn't included as an example in the docs! Most examples are overcomplicated but this seemingly complex scenario is a lot easier than it seems – Canal

this is beautiful. nice to see a push-based way of doing this vs pull-based. – Auriga

Answer 3

Another alternative is using Scrutor, and decorating HealthCheckService. If you want to be paranoid about having multiple threads re-publishing, you'd have to add a locking mechanism while fetching the HealthCheckReport from the inner HealthCheckService. A decent example is here.

using System.Reflection;
using HealthCheckCache;
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Diagnostics.HealthChecks;

var builder = WebApplication.CreateBuilder(args);

// used by the Decorator CachingHealthCheckService
builder.Services.AddMemoryCache();
builder.Services.AddHttpContextAccessor();

// register all IHealthCheck types - basically builder.Services.AddTransient<AlwaysHealthy>(), but across all types in this assembly.
var healthServices = builder.Services.Scan(scan =>
    scan.FromCallingAssembly()
        .AddClasses(filter => filter.AssignableTo<IHealthCheck>())
        .AsSelf()
        .WithTransientLifetime()
);

// Register HealthCheckService, so it can be decorated.
var healthCheckBuilder = builder.Services.AddHealthChecks();
// Decorate the implementation with a cache
builder.Services.Decorate<HealthCheckService>((inner, provider) =>
    new CachingHealthCheckService(inner,
        provider.GetRequiredService<IHttpContextAccessor>(),
        provider.GetRequiredService<IMemoryCache>()
    )
);

// Register all the IHealthCheck instances in the container
// this has to be a for loop, b/c healthCheckBuilder.Add will modify the builder.Services - ServiceCollection
for (int i = 0; i < healthServices.Count; i++)
{
    ServiceDescriptor serviceDescriptor = healthServices[i];
    var isHealthCheck = serviceDescriptor.ServiceType.IsAssignableTo(typeof(IHealthCheck)) && serviceDescriptor.ServiceType == serviceDescriptor.ImplementationType;
    if (isHealthCheck)
    {
        healthCheckBuilder.Add(new HealthCheckRegistration(
            serviceDescriptor.ImplementationType.Name,
            s => (IHealthCheck)ActivatorUtilities.GetServiceOrCreateInstance(s, serviceDescriptor.ImplementationType),
            failureStatus: null,
            tags: null)
        );
    }

}

var app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.MapHealthChecks("/health", new HealthCheckOptions()
{
    AllowCachingResponses = true, // allow caching at Http level
});

app.Run();

public class CachingHealthCheckService : HealthCheckService
{
    private readonly HealthCheckService _innerHealthCheckService;
    private readonly IHttpContextAccessor _contextAccessor;
    private readonly IMemoryCache _cache;
    private const string CacheKey = "CachingHealthCheckService:HealthCheckReport";

    public CachingHealthCheckService(HealthCheckService innerHealthCheckService, IHttpContextAccessor contextAccessor, IMemoryCache cache)
    {
        _innerHealthCheckService = innerHealthCheckService;
        _contextAccessor = contextAccessor;
        _cache = cache;
    }

    public override async Task<HealthReport> CheckHealthAsync(Func<HealthCheckRegistration, bool>? predicate, CancellationToken cancellationToken = new CancellationToken())
    {
        HttpContext context = _contextAccessor.HttpContext;


        var forced = !string.IsNullOrEmpty(context.Request.Query["force"]);
        context.Response.Headers.Add("X-Health-Forced", forced.ToString());
        var cached = _cache.Get<HealthReport>(CacheKey);
        if (!forced && cached != null)
        {
            context.Response.Headers.Add("X-Health-Cached", "True");
            return cached;
        }
        var healthReport = await _innerHealthCheckService.CheckHealthAsync(predicate, cancellationToken);
        if (!forced)
        {
            _cache.Set(CacheKey, healthReport, TimeSpan.FromSeconds(30));
        }
        context.Response.Headers.Add("X-Health-Cached", "False");
        return healthReport;
    }
}

Auriga