Hangfire server is trying to process job from other queue

Question

I would like to ask if there is a way how to define that HangFire server shouldn't process recurring jobs?

There are two application (windows service and web application) which are using HangFire for running background jobs, so there are two hangfire servers (queues are defined correctly).

Service creates only recurring jobs and it is using "default" queue. Web application creates fire-and-forgot jobs and it is using other queue (e.g. web_app_queue).

It seems that HangFire server, created by web application, is trying to schedule/process recurring jobs although there are defined for other queue, but web app HangFire server doesn't have access to required assembly which is available for windows service.

Web App HangFire server:

var backgroundJobServer = new BackgroundJobServer(new BackgroundJobServerOptions { Queues = new[] { "web_app_queue" } });

Web App HangFire job:

var client = new BackgroundJobClient();
var state = new EnqueuedState("web_app_queue");
var jobId = client.Create(methodCall, state);

Windows service HangFire server:

var server = new BackgroundJobServer();

Windows service HangFire job:

private static void AddRecurringJob<T>(string cronExpression) where T : IJob
{
    RecurringJob.AddOrUpdate<T>(job => job.Execute(), cronExpression, TimeZoneInfo.Local);
}

Name of queue is not defined for windows service HangFire server and jobs, so default queue ("default") is applied.

Exception message and detail is here:

message="Recurring job ... can not be scheduled due to job load exception."

detail="Hangfire.Common.JobLoadException: Could not load the job. See inner exception for the details. 
---> System.IO.FileNotFoundException: Could not load file or assembly 'XXX.YYY, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.;   
    at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMarkHandle stackMark, IntPtr pPrivHostBinder, Boolean loadTypeFromPartialName, ObjectHandleOnStack type);   
    at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean loadTypeFromPartialName);   
    at System.Type.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase);   
    at Hangfire.Storage.InvocationData.Deserialize();   
    --- End of inner exception stack trace 
    at Hangfire.Storage.InvocationData.Deserialize();  
    at Hangfire.Server.RecurringJobScheduler.TryScheduleJob(JobStorage storage, IStorageConnection connection, String recurringJobId, IReadOnlyDictionary`2 recurringJob);   
    at Hangfire.Server.RecurringJobScheduler.Execute(BackgroundProcessContext context)"

Regimentals

You should probably post the code that you're using to do the setup of the queues. – Interleaf

thank you for your suggestion, updated – Regimentals

Answer 1

I found solution for my issue here: https://github.com/HangfireIO/Hangfire/issues/775
So it is possible to create HangFire server (using class BackgroundProcessingServer) and define which processes can be handeled by this server.

This way I was able to define that my web app HangFire server should process only fire-and-forgot processes

Example bellow:

    int workerCount = Environment.ProcessorCount * 5;
    string queueName = "web_app_queue";

    List<IBackgroundProcess> processes = GetProcessesForBackgroundServer(queueName, workerCount);
    var backgroundJobServer = new BackgroundProcessingServer(JobStorage.Current, processes, new Dictionary<string, object> { { "Queues", new string[] { queueName } }, { "WorkerCount", workerCount } });


    private List<IBackgroundProcess> GetProcessesForBackgroundServer(string queueName, int workerCount)
    {
        var processes = new List<IBackgroundProcess>();

        for (var i = 0; i < workerCount; i++)
        {
            processes.Add(new Worker(queueName)); //only fire-and-forgot jobs will be processed by this server (important processes ServerHeartbeat, ServerWatchdog are included automatically by BackgroundProcessingServer)
        };

        return processes;
    }

Regimentals

Beware when using queues to partition jobs to different servers: failing jobs aren't automatically retried on the same queue. If a job fails then the queue it was supposed to be run on is 'lost' and any server can look at the job and determine if it should be executed on that server. You can tag jobs in some way to include the queue but (I'm pretty sure) they still require the job dlls/assemblies being available on all the Hangfire servers. e.g. github.com/HangfireIO/Hangfire/pull/502 – Tibetoburman