Tracking hangfire background jobs with app insights

Question

I have set up app insights in Asp.net core application. All my web api requests are tracked on app insights and if I have any failures I can simply find them in Failures section.

However, I have also Hangfire background jobs running and if they are failing I can't find them on app insights. Also I have alert rule Whenever the total http server errors is greater than or equal to 1 count and I am not sure if hangfire 5xx errors will go under this condition.

So is there any way to track Hangfire jobs failures and get notified about them?

Aggress

Will need a bit more information. Have you configured logging with Hangfire using this documentation? docs.hangfire.io/en/latest/configuration/… – Pilarpilaster

@Clamsmash yes, but I don't see any data in app insights. – Aggress

Answer 1

There was a nice nuget package created Hangfire.Extensions.ApplicationInsights.

So, install the package:

Install-Package Hangfire.Extensions.ApplicationInsights

and add the line to ConfigureService method:

services.AddHangfireApplicationInsights();

If your solution requires some custom details you can adjust the code from github repository.

Aggress

This is actually working...easy but you must already have configured appInsight – Featherveined

Answer 2

Hangfire handles most exceptions under the hood, so App Insights is not going to pick them up by default. There is also a bunch of configuration you have to do with App Insights as well.

I wrote a JobFilter for Hangfire which allows you to connect with App Insights, this should be enough to get you going: https://github.com/maitlandmarshall/MIFCore/blob/master/MIFCore.Hangfire/Analytics/AppInsightsEventsFilter.cs

And for the App Insights configuration: https://github.com/maitlandmarshall/MIFCore/blob/master/MIFCore.Hangfire/Analytics/TelemetryConfigurationFactory.cs

To put everything together from the above links:

var appInsights = this.rootScope.ResolveOptional<AppInsightsConfig>();
var childScope = ServiceScope = this.rootScope.BeginLifetimeScope("HangfireServiceScope");
var activator = new AutofacLifecycleJobActivator(childScope);
var options = new BackgroundJobServerOptions()
{
    Activator = activator,
    Queues = new[] { JobQueue.Alpha, JobQueue.Beta, JobQueue.Default, JobQueue.Low }
};

this.globalConfig
    .UseFilter(new BackgroundJobContext());

if (!string.IsNullOrEmpty(appInsights?.InstrumentationKey))
{
    var telemetryClient = new TelemetryClient(TelemetryConfigurationFactory.Create(appInsights));
    this.globalConfig.UseFilter(new AppInsightsEventsFilter(telemetryClient));
}

using (var server = new BackgroundJobServer(options))
{
    await server.WaitForShutdownAsync(stoppingToken);
}

Pilarpilaster

What a great code example! I took your github code and modified it slightly so I could track the duration of a job. Was able to slim down your OnPerformed method a bunch moving the Job Succeeded assignment to the OnStateElection and implementing IElectStateFilter. – Dielectric