Hangfire - Configure AutomaticRetry for specific RecurringJob at runtime
Asked Answered
S

3

7

I'm using Hangfire v1.7.9 and I'm trying to configure a series of recurring background jobs within my MVC 5 application to automate the retrieval of external reference data into the application. I've tested this with one task and this works great, but I'd like administrators within the system to be able to configure the Attempts and DelayInSeconds attribute parameters associated with the method that is called in these background jobs.

The AutomaticRetryAttribute states that you have to use...

...a constant expression, typeof expression or an array creation expression of an attribute parameter type

... which from what I've read is typical of all Attributes. However, this means that I can't achieve my goal by setting a property value elsewhere and then referencing that in the class that contains the method I want to run.

Additionally, it doesn't look like there is any way to configure the automatic retry properties in the BackgroundJob.Enqueue or RecurringJob.AddOrUpdate methods. Lastly, I looked at whether you could utilise a specific retry count for each named Queue but alas the only properties about Hangfire queues you can set is their names in the BackgroundJobServerOptions class when the Hangfire server is initialised.

Have I exhausted every avenue here? The only other thing I can think of is to create my own implementation of the AutomaticRetryAttribute and set the values at compile time by using an int enum, though that in itself would create an issue in the sense that I would need to provide a defined list of each of the values that a user would need to select. Since I wanted the number of retries to be configurable from 5 minutes all the way up to 1440 minutes (24 hours), I really don't want a huge, lumbering enum : int with every available value. Has anyone ever encountered this issue or is this something I should submit as a request on the Hangfire GitHub?

Shakitashako answered 13/3, 2020 at 11:2 Comment(0)
V
6

I would take the approach of making a custom attribute that decorates AutomaticRetryAttribute:

public class MyCustomRetryAttribute : JobFilterAttribute, IElectStateFilter, IApplyStateFilter
{
    public void OnStateElection(ElectStateContext context)
    {
        GetAutomaticRetryAttribute().OnStateElection(context);
    }

    public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
    {
        GetAutomaticRetryAttribute().OnStateApplied(context, transaction);
    }

    public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
    {
        GetAutomaticRetryAttribute().OnStateUnapplied(context, transaction);
    }

    private AutomaticRetryAttribute GetAutomaticRetryAttribute()
    {
        // Somehow instantiate AutomaticRetryAttribute with dynamically fetched/set `Attempts` value
        return new AutomaticRetryAttribute { Attempts = /**/ };
    }
}

Edit: To clarify, this method allows you to reuse AutomaticRetryAttribute's logic, without duplicating it. However, if you need to change more aspects on per-job basis, you may need to duplicate the logic inside your own attribute.

Also, you can use context.GetJobParameter<T> to store arbitrary data on per-job basis

Verduzco answered 13/3, 2020 at 11:22 Comment(3)
Hi @Xymanek, this is useful... I think by using the context.GetJobParameter<T> method I can retrieve the configuration model object from the Job and then implement the number of retries from there :)Shakitashako
AutomaticRetryAttribute actually works using context.GetJobParameter<T>, so you can take inspiration from thereVerduzco
I tried the custom attribute approach but once the number of retries were exhausted it seems to fallback to the default settings i.e. it retried 10 times instead of my custom 3 times. At first it would report 'trying 2 of 3` and then changed to 'trying 4 of 10'!Byte
D
1

I ran into the same problem that jmatthias explains in a comment above. I think I may have a work around for it figured out, but this is only lightly tested so far:

The key change is:

int retryCount = context.GetJobParameter<int>("RetryCount");
if (retryCount >= attempts)
{
  context.CandidateState = new Hangfire.States.DeletedState();
}
    public class HgCustomRetryAttribute : JobFilterAttribute, IElectStateFilter, IApplyStateFilter
      {
        private AutomaticRetryAttribute _base;
        public HgCustomRetryAttribute()
        {
        }
        public void OnStateElection(ElectStateContext context)
        {
          var retryAttempts = (int)context.Job.Args[1];
          int retryCount = context.GetJobParameter<int>("RetryCount");
          if (retryCount >= retryAttempts )
          {
            context.CandidateState = new Hangfire.States.DeletedState();
          }
          GetAutomaticRetryAttribute(attempts).OnStateElection(context);
        }
        public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
        {
          var retryAttempts = (int)context.Job.Args[1];
          GetAutomaticRetryAttribute(retryAttempts ).OnStateApplied(context, transaction);
        }
        public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
        {
          var retryAttempts = (int)context.Job.Args[1];
          GetAutomaticRetryAttribute(retryAttempts ).OnStateUnapplied(context, transaction);
        }
        private AutomaticRetryAttribute GetAutomaticRetryAttribute(int retryAttempts )
        {
          _base = new AutomaticRetryAttribute { Attempts = retryAttempts , DelaysInSeconds = new int[] { 20 } }; 
          return _base;
        }
      }

Here is an example of how I use the attribute. Note the retryAttempts parameter that gets passed in to the method and then gets picked up by the HgCustomRetryAttribute.

[HgCustomRetryAttribute()]
public void ExecuteJob(string pathToDll, int retryAttempts)
{
...
...
}
Druce answered 20/4, 2023 at 9:21 Comment(0)
D
0

Building upon the answer provided by DotNetAndAngular Developer, I created a custom RetryAttempts attribute that allows you to specify the ordinal position of a retryAttempts method parameter, which matches the behavior of the existing Queue attribute and DisplayName attributes.

This custom RetryAttempts attribute allows you to dynamically specify the retry attempts for a given job at runtime.

Here is an example of using the custom RetryAttempts attribute along with using the aforementioned Queue attribute and DisplayName attribute:

var jobId = BackgroundJob.Enqueue(
    () => ProcessRequest("Default", "Process Pending Orders", 3, request));

// ...

/// <summary>
/// Processes the request.
/// </summary>
/// <param name="queueName">Queue to assign the job to. The Queue name should only include lowercase letters, digits, underscore, and hyphen characters.</param>
/// <param name="jobName">Job name.</param>
/// <param name="retryAttempts">Maximum retry attempts.</param>
/// <param name="request">Request to execute.</param>
[Queue("{0}")] // The first method parameter is the queue name
[DisplayName("{1}")] // The second method parameter is the job name
[RetryAttempts("{2}")] // The third method parameter is the max retry attempts
public async Task ProcessRequest(
#pragma warning disable IDE0060 // Remove unused parameter
    string queueName,
    string jobName,
    int retryAttempts,
#pragma warning restore IDE0060 // Remove unused parameter
    object request)
{
    // Process the thing
}

Here is the code for the RetryAttempts attribute:

using Hangfire;
using Hangfire.Common;
using Hangfire.States;
using Hangfire.Storage;
using System.Globalization;

namespace HangfireHelpers;

/// <summary>
/// Custom Hangfire attribute that allows dynamically setting the retry attempts for a job
/// by specifying the positional method parameter that defines the retry attempts.
/// </summary>
/// <remarks>
/// For example, if the retry attempts is defined as the 2nd positional method parameter (zero-indexing),
/// then you would decorate the method with the following attribute: [RetryAttempts("{2}")]
/// </remarks>
public class RetryAttemptsAttribute : JobFilterAttribute, IElectStateFilter, IApplyStateFilter
{
    private AutomaticRetryAttribute? _baseAttribute;
    private readonly string _retryAttempsParameterPosition;
    const string ValidationErrorMessage = "Invalid RetryAttemptsAttribute parameter. Example of valid value: {3}";

    public RetryAttemptsAttribute(string retryAttempsParameterPosition)
    {
        _retryAttempsParameterPosition = retryAttempsParameterPosition;
    }

    public void OnStateElection(ElectStateContext context)
    {
        var retryAttemptsValue = string.Format(CultureInfo.InvariantCulture, _retryAttempsParameterPosition, context.BackgroundJob.Job.Args.ToArray());
        if (!int.TryParse(retryAttemptsValue, out var retryAttempts))
        {
            throw new Exception(ValidationErrorMessage);
        }
        int retryCount = context.GetJobParameter<int>("RetryCount");
        if (retryCount >= retryAttempts)
        {
            context.CandidateState = new global::Hangfire.States.DeletedState();
        }
        GetAutomaticRetryAttribute(retryAttempts).OnStateElection(context);
    }

    public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
    {
        var retryAttemptsValue = string.Format(CultureInfo.InvariantCulture, _retryAttempsParameterPosition, context.BackgroundJob.Job.Args.ToArray());
        if (!int.TryParse(retryAttemptsValue, out var retryAttempts))
        {
            throw new Exception(ValidationErrorMessage);
        }
        GetAutomaticRetryAttribute(retryAttempts).OnStateApplied(context, transaction);
    }

    public void OnStateUnapplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
    {
        var retryAttemptsValue = string.Format(CultureInfo.InvariantCulture, _retryAttempsParameterPosition, context.BackgroundJob.Job.Args.ToArray());
        if (!int.TryParse(retryAttemptsValue, out var retryAttempts))
        {
            throw new Exception(ValidationErrorMessage);
        }
        GetAutomaticRetryAttribute(retryAttempts).OnStateUnapplied(context, transaction);
    }

    private AutomaticRetryAttribute GetAutomaticRetryAttribute(int retryAttempts)
    {
        _baseAttribute = new AutomaticRetryAttribute { Attempts = retryAttempts };
        return _baseAttribute;
    }
}
Disclaim answered 8/2 at 3:44 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.