Use multiple instance of hangfire with single database

Question

Has anyone used multiple instances of Hangfire (in different applications) with same SQL DB for configuration. So instead of creating new SQL DB for each hangfire instance i would like to share same DB with multiple instances.

As per the hangfire documentation here it is supported since v1.5 However forum discussion here and here shows we still have issues running multiple instances with same db

Update 1
So based on suggestions and documentation i configired hangfire to use queue

   public void Configure(IApplicationBuilder app, IHostingEnvironment env, 
   ILoggerFactory loggerFactory)
   {
        app.UseHangfireServer(new BackgroundJobServerOptions()
        {
            Queues = new string[] { "instance1" }
        });
   }

Method to invoke

[Queue("instance1")]
public async Task Start(int requestID)
{

}

This is how i Enqueue job

 _backGroundJobClient.Enqueue<IPrepareService>(x => x.Start(request.ID));

however when i check [JobQueue] table the new job has queue name default and because of that hangfire will never pickup that job because it picks up jobs for queues.

I think is a bug

Update 2

Found one more thing. I am using instance of IBackgroundJobClient. The instance is automatically get injected by .Net Core's inbuilt container.

So if i use instance to enqueue the job then hangfire creates new job with default queue name

 _backGroundJobClient.Enqueue<IPrepareService>(x => x.Start(request.ID));

However if i use static method, then hangfire creates new job with configured queue name instance1

  BackgroundJob.Enqueue<IPrepareService>(x => x.Start(prepareRequest.ID));

How do i configure hangfire in .Net Core so the instance of IBackgroundJobClient will use configure queue name ?

Archil

Have you tried using different queue names for different instances? If yes, still has the same issue? I have not tried it, but think that using different queue names for different instances should not mix up the things. Can you also post your code, which you have tried? – Recollect

Doing it without problem. What issues do you have? – Holster

No i haven't used it yet. But that is the confusion. You are saying use different queue names but hangfire documentation says You aren’t required to have additional configuration to support multiple background processing servers in the same process since Hangfire 1.5, just skip the article. Server identifiers are now generated using GUIDs, so all the instance names are unique docs.hangfire.io/en/latest/background-processing/… – Archil

..and also the in generated SQL tables i don't see [Server].[Id] has foreign key relationship with any other table. So how does hangfire identify a job belongs to particular instance? – Archil

I run multiple instances on a single database. No queues necessary. – Cubical

@Cubical are all the instances of same application or different applications.? In my case i have two different applications trying to share same database. So they have different methods – Archil

We have one application that enqueues the job and another that processes the job, and we can have multiple instances of either application. In both cases, they're working off an interface that describes the job. – Cubical

it does make sense when you have microservice architecture. You dont want to end up with multiple small databases. – Archil

Most microservice architectures I've seen absolutely preach having small databases that are consumed by a single service rather than monolithic databases. Especially if there's no need for them to share the database, such as processing different sets of jobs in Hangfire. – Cubical

Answer 1

This is possible by simply setting the SQL server options with a different schema name for each instance.

Instance 1:

configuration.UseSqlServerStorage(
    configuration.GetConnectionString("Hangfire"),
    new SqlServerStorageOptions { SchemaName = "PrefixOne" }
);

Instance 2:

configuration.UseSqlServerStorage(
    configuration.GetConnectionString("Hangfire"),
    new SqlServerStorageOptions { SchemaName = "PrefixTwo" }
);

Both instances use same connection string and will create two instances of all the required tables with the prefix specified in the settings.

Queues are used for having separate queues in the same Hangfire instance. If you want to use different queues you'll need to specify the queues you want the IBackgroundJobClient to listen to and then specify that queue when creating jobs. This doesn't sound like what you're trying to accomplish.

Eastlake

when using instance of IBackgroundJobClient i dont see any option where i can specify queue name. The Enqueue method does not take queue name as parameter – Archil

RecurringJob manager has built in queue support. If you want to queue immediately in a specific queue you'll need to use IBackgroundJobClient.Create() with an EnqueuedState setting the queue for the IState parameter. Alternatively you can add the Queue attribute on the method you are calling. However none of this is required if you don't need different queues and just want two hangfire instances in same database. – Eastlake

Very useful. I thought Queue can be used to isolate different applications, but later I found in HF Queues are useful only in single application. Schemas are the best way to address the issue. Thanks – Successor

Hangfire.MAMQSqlExtension allows jobs to be isolated by queues. No code sharing is needed between multiple servers – Hornmad

@XiaoguoGe make your library available to .Net Core – Bernoulli

@JohnNyingi It is already. The usage examples contain both .NetFramework and .NetCore – Hornmad

I just saw the implementation its really good – Bernoulli