Tips to avoid hangfire jobs to duplicate?

Question

I am having a problem that some jobs on hangfire are been queued more than one time with the same parameters, the jobs are almost queued at the same time.

I've tried to limit the numbers of workers for just one and then to decorate my method with DisableConcurrentExecution.

I'm using sqlserver as storage. Anyone had faced this problem, there are some tips to avoid it?

PS: I used DisableConcurrentExecution because on hangfire documentation says that mutex and semaphores could not guarantee the job being called just one time.

PS2: Check my hangfire servers I note that i had two instances, each one with 1 worker, so I think its a parallelism problem not a concurrent one.

Cogitate

Answer 1

This work arround works for me. I used MaximumConcurrentExecutions(1) decorating the interface method that hangfire runs.

This didn't solve my problem alone, because of having multiple servers (auto scale up).

So in the startup class I create another backgroundjobserveroptions to create an exclusive server for this job, to guarantee that when the application scale up this did not create another server with another queue, I had to query in the sqlserver hangfire.server and check if my queue already exists.

if(!HangfireServerInfo.QueueExists("queuename", AppSettings.GetConnectionString("Hangfire"))){
        var hangfireServerOptions = new BackgroundJobServerOptions { WorkerCount = 1, Queues = new[] {"queuename"} };
        app.UseHangfireServer(hangfireServerOptions);
    }


public static class HangfireServerInfo
    {
        public static bool QueueExists(string queueName, string connectionString)
        {
            using (var connection = new SqlConnection(connectionString))
            {
                var cmd = connection.CreateCommand();
                cmd.CommandText = 
                    $@"SELECT COUNT(*) 
                    FROM {YOURHANGFIRESERVER.server}
                    WHERE JSON_QUERY(data, '$.queues') = '[""{queueName}""]'";

                connection.Open();
                var result = (int)cmd.ExecuteScalar();
                connection.Close();

                return result > 0;
            }
        }
    }

Probably that are better ways to solve it, but this works.

Cogitate

could you post whole code. where you get checkIfQueueExists – Variolous

@Ganesh_Devlekar edit the answer, sorry for not put all the code. As I said there are probably better ways to solve this problem, but this work for me. – Cogitate

Answer 2

According to the Hangfire documentation the duplicate job problem occurs when the recurring jobs/tasks don't have identifiers: "Use unique identifiers for each recurring job, otherwise you’ll end with a single job.".

RecurringJob.AddOrUpdate("some-id", () => Console.WriteLine(), Cron.Hourly);

Sources:

1. https://gist.github.com/odinserj/a8332a3f486773baa009;
2. https://discuss.hangfire.io/t/how-do-i-prevent-creation-of-duplicate-jobs/1222/4;

Rissa

Thank for you help @iliass ! This is not a RecurringJob, the job started based on a webhook event that I received from a third part application. – Cogitate

@aucls. The point I was trying to make is that you assign unique id's to your jobs. Are you doing that? – Rissa