Hangfire DistributedLockTimeoutException when calling the same static method concurrently

Question

I have a web service that, when posted to, queues up downloads of images in Hangfire, so that if the image download fails, Hangfire will automatically retry:

[AutomaticRetry(Attempts = 5, OnAttemptsExceeded = AttemptsExceededAction.Delete)]
public static void DownloadImage(string url)
{
    ...
}

Each time the web service is posted to, it will add a number of background jobs that call this method as follows:

Hangfire.BackgroundJob.Enqueue(() => Downloader.DownloadImage(o.SourceURL));

What I am seeing though, is the first of the background jobs succeeds, and the rest fail in RAPID succession i.e. in under a second, say 100 queued jobs will fail their allocated 5 times (as per AutomaticRetry attribute)

The error in the Hangfire tasks is:

Hangfire.Storage.DistributedLockTimeoutException

Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'HangFire:Downloader.DownloadImage' resource.

Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'HangFire:Downloader.DownloadImage' resource.
   at Hangfire.SqlServer.SqlServerDistributedLock.Acquire(IDbConnection connection, String resource, TimeSpan timeout)
   at Hangfire.SqlServer.SqlServerDistributedLock..ctor(SqlServerStorage storage, String resource, TimeSpan timeout)
   at Hangfire.SqlServer.SqlServerConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
   at Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContext filterContext)
   at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation)

Bernhard

Could this be related to this currently open pull request ? github.com/HangfireIO/Hangfire/pull/789 – Cecilycecity

I am not sure - that pull request refers to RecurringJobScheduler, DelayedJobScheduler, ExpirationManager background processes whereas this is calling BackgroundJob.Enqueue - possibly related? – Bernhard

@Jimbo, did you find a solution ? – Indigence

@Yanal-YvesFargialla I know I fixed the problem but dont recall how, sorry. I dont have access to that code any longer either. What I do recall is that it was more of a workaround than a fix (if that makes sense?) Perhaps check that you dont have DisableConcurrentExecution set by default on your method. #45164869 – Bernhard