Quartz.NET scheduler doesn't fire jobs/triggers once deployed

Question

INTRODUCTION

I'm using Quartz.Net on an ASP.Net framework 4, webforms web site. Basically, the user should have the hability to fire manually a batch script that asynchronously processes thousands of records stored on a database. The user can stop or pause at any time, adjust some variables, and continue if needed with the process (remaining records).
The code is done and working locally (developer machine, win7, vs2010, sql server express 2008 R2).
It was also tested on a local server (win server 2008 R2, sql server express 2008 R2). It works fine on both enviroments, tested with all the code precompiled. The problem is that, once deployed on a remote server (win server 2008 R2), where it actually should be running on (hosting enviroment, not shared, not clustered), it doesnt completely works (se details below). Scheduler gets created, but the trigger, hence the job, doesn't fire.

(Note: I know some of you would suggest to use Quartz as windows service, but despite the benefits of doing so, I really would like to find out why it doesn't work as an embedded solution, since it should be working just fine like does locally)

DETAILS

Quartz 2.1.2  
Common.Logging 2.1.2  
Common.Logging.NLog 2.0.0  
NLog 2.0.1.2

global.asax

public static ISchedulerFactory SchedulerFactory;
public static IScheduler Scheduler;

void Application_Start(object sender, EventArgs e)
{
    SchedulerFactory = new StdSchedulerFactory();
    Scheduler = SchedulerFactory.GetScheduler();

    // Define a durable job instance (durable jobs can exist without triggers)
    IJobDetail job = JobBuilder.Create<MyJobClass>()
                                .WithIdentity("MyJob", "MyGroup")
                                .StoreDurably()
                                .Build();

    Scheduler.AddJob(job, false);
    Scheduler.Start();
}
void Application_End(object sender, EventArgs e)
{
    Scheduler.Shutdown(true);
}

process.aspx.cs (start button click)

// get records from DB, iterate, process, etc
...

IJobDetail job = ASP.global_asax.Scheduler.GetJobDetail(new JobKey("MyJob", "MyGroup"));
job.JobDataMap.Put("something1", 1);
job.JobDataMap.Put("something2", somevar);

ITrigger trigger = TriggerBuilder.Create()
                    .WithIdentity("MyTrigger", "MyGroup")
                    .StartNow()
                    .WithSimpleSchedule(x => x.WithIntervalInSeconds(5).RepeatForever())
                    .Build();

var triggersSet = new Quartz.Collection.HashSet<ITrigger> { trigger };

ASP.global_asax.Scheduler.ScheduleJob(job, triggersSet, true);

LOG OUTPUT

local log

Default Quartz.NET properties loaded from embedded resource file  
Using default implementation for object serializer  
Using default implementation for ThreadExecutor  
Initialized Scheduler Signaller of type: Quartz.Core.SchedulerSignalerImpl  
Quartz Scheduler v.2.1.2.400 created.  
RAMJobStore initialized.  
Scheduler meta-data: Quartz Scheduler (v2.1.2.400) 'DefaultQuartzScheduler' with instanceId 'NON_CLUSTERED'   Scheduler class: 'Quartz.Core.QuartzScheduler' - running locally.   NOT STARTED.   Currently in standby mode.   Number of jobs executed: 0   Using thread pool 'Quartz.Simpl.SimpleThreadPool' - with 10 threads.   Using job-store 'Quartz.Simpl.RAMJobStore' - which does not support persistence. and is not clustered.  
Quartz scheduler 'DefaultQuartzScheduler' initialized  
Quartz scheduler version: 2.1.2.400  
Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.  
Batch acquisition of 0 triggers  
Batch acquisition of 0 triggers

It continues logging Batch acquisition of 0 triggers until button click occurs:

Default Quartz.NET properties loaded from embedded resource file  
Batch acquisition of 1 triggers  
Producing instance of Job 'MyGroup.MyJob', class=MyJobClass  
Batch acquisition of 0 triggers  
Calling Execute on job MyGroup.MyJob  
Trigger instruction : NoInstruction  
Batch acquisition of 1 triggers  
Producing instance of Job 'MyGroup.MyJob', class=MyJobClass  
Batch acquisition of 0 triggers  
Calling Execute on job MyGroup.MyJob  
Trigger instruction : NoInstruction  
Batch acquisition of 1 triggers

deployed log

Default Quartz.NET properties loaded from embedded resource file  
Using default implementation for object serializer  
Using default implementation for ThreadExecutor  
Initialized Scheduler Signaller of type: Quartz.Core.SchedulerSignalerImpl  
Quartz Scheduler v.2.1.2.400 created.  
RAMJobStore initialized.  
Scheduler meta-data: Quartz Scheduler (v2.1.2.400) 'DefaultQuartzScheduler' with instanceId 'NON_CLUSTERED' Scheduler class: 'Quartz.Core.QuartzScheduler' - running locally. NOT STARTED. Currently in standby mode. Number of jobs executed: 0 Using thread pool 'Quartz.Simpl.SimpleThreadPool' - with 10 threads. Using job-store 'Quartz.Simpl.RAMJobStore' - which does not support persistence. and is not clustered.   
Quartz scheduler 'DefaultQuartzScheduler' initialized  
Quartz scheduler version: 2.1.2.400  
Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.

Here stays like this. As you see, compared to the other log, it's not trying to acquire triggers (line Batch acquisition of 0 triggers does not appear at all). If you click the process button anyway, the log adds one line:

Default Quartz.NET properties loaded from embedded resource file

But nothing else happens. The records are not processed (I know since every time a record is proccessed, is marked in the database). No errors occur, but the trigger is not fired, and the job is not executed. Also, the CPU usage run up to 50% or more on button click, and doesnt gets down unless you go to IIS, stop and restart the application pool. This cpu consumption doesn't happen locally.

---

update 1

Changed use of scheduler for a singleton, as suggested by LeftyX, but still get same behavior on remote server.

update 2

I also tried to use ADOJobStore (instead of RAMJobStore which I was using). Now it still works perfectly locally; but still doesn't execute the trigger (hence job) online. The only difference is that online the CPU usage doesn't run up to 50%. And now I can see that the job and trigger are created (I query the tables and see that those records exists), but never gets executed.

Lazo

If you think my answer was good enough for you, don't forget to accept it. Thanks. – Ringside

LeftyX, you have been very kind with your help, I'll see if I dont get any other suggestions and if it doesn't occur, I'll accept your answer. Also, I tried to upvote you but I don't have enough reputation. Please, also check my question below with updates. – Lazo

Zed, why don't you try and publish my sample code (you can download it from my skydrive repository) and see how it works? It might be a hosting provider problem ... – Ringside

What value do you have for quartz.threadPool.threadCount? (Don't know if it is even possible to set this to zero, but if so, perhaps the system thinks it doesn't have a free thread to execute the job). – Offer

@sgmoore, quartz.threadPool.threadCount is 10 by default and I also tested changing it to 5. I can see that it's successfuly changed in the log, where it says Using thread pool 'Quartz.Simpl.SimpleThreadPool' - with 5 threads. – Lazo

@LeftyX, I published and tested your sample code on same server, same app pool (tried both: different website, and also a new IIS app inside same website). Both cases worked! So, could it be a problem on web.config or compatibility issue? – Lazo

@zed: Guess so. Show us your web.config or zip your code somewhere so we can have a look. – Ringside

@LeftyX: You can check it here. I removed irrelevant parts (role, membership, profile providers, smtp and connection strings) – Lazo

@zed: I meant the whole application. your config does not say much. it seem and old config converted for framework 4.0. – Ringside

I know its bit unrelated but, can I use it in a commercial project for my client who is building a product and will distribute it to other clients of his, any hidden clauses or terms I need to be aware of? – Diabetes

Answer 1

One thing that I have noticed is the use of the Scheduler in your asp.net application.
You should use singleton objects.

in your process.aspx.cs this line

IScheduler scheduler = new StdSchedulerFactory().GetScheduler();

creates a new scheduler but you should use the one you've created as static in Application_Start.

If you want to get access to the singleton instance use a public memeber in your Global.asax.cs:

 public static ISchedulerFactory SchedulerFactory;
 public static IScheduler Scheduler;

and you can reference it in your process.aspx.cs:

MvcApplication.Scheduler.ScheduleJob(job, triggersSet, true);

Another solution is to use dependency injection. You can find some info here using StructureMap and here for Unity.

UPDATE:

You can download a sample application (asp.net 4.0) called AspNet_Quartz here and see how it works here.

Ringside

thanks for your help. From reading about the quartz scheduler, I thought that it manage the singleton on its own, and retrieved it every time that you invoke StdSchedulerFactory().GetScheduler(); Could you please point me in the right direction of use the scheduler as a singleton? Thanks! – Lazo

Thanks. Your update refers to MVC, and this is webforms. Anyway, I found that, like you said, adding the 'public' to the variable declared in the global.asax, then I could access it (the singleton) like this from aspx.cs: ASP.global_asax.Scheduler Nevertheless, the global behavior is the same: locally works like a charm, once uploaded to remote server, the scheduler seems not to be listening to triggers as described, and CPU start to work as crazy, upon 50%. Is it possible that I need to set special permissions or similar on the hosting enviroment? – Lazo

@zed: Quartz.net must work in a medium trusted environment. Are you using one of the latest versions? – Ringside

If I go to IIS (hosting enviroment) and check on website .Net Trust Levels it says "Full (internal)". Same as local server. Quartz version is 2.1.2, which I believe is the latest for .net – Lazo

@zed: I cannot figure out what your problem can be. I've created a test project. Check my updated answer. – Ringside

I mark this as answer, because of all good comments and accurate help provided. Although I couldn't solve the problem itself, maybe it is because this is a website and not a web application like LeftyX sample code (which works). Anyway, we are moving to quartz net as windows service now. – Lazo

I am having the same issue as @zed. I'm using an MVC application. Locally it works, but as soon as I deploy it, the trigger never fires. Has anything here changed? I may have to create a windows service at this point. But really wanted to figure this out! – Canaan

Answer 2

There are nothing wrong with Quartz, all because of IIS app pool recycling. I fixed the bug by stopping the pool that is used for Quartz from recycling:

1. Go to IIS manager -> Application Pools -> Create a new pool, I named it Scheduler (any name is ok)
   
2. Select Scheduler pool -> advanced Settings
   

• In General section, at Start Mode, Select AlwaysRunning (IIS 8.5) or true for (IIS 7.5, 8)
  
• In Process Model Section-> Idle Timeout(minutes) set to 0 (meaning: No Idel timeout)
  
• In Recycling section -> Regular time Interval set to 0 (meaning: no recycling)
  3. Deploy your Quartz site into that application pool. And send one request to the pool to "wake your app up" and it will run until u stop it.
  That's it.
  Update: Another solution to keep your app pool always alive is using Auto-Start ASP.NET Applications

Another option: Using some third party pinging tool (like uptimerobot or diy one ) to keep refreshing your site each and every x seconds (or minute)

Storekeeper

That configuration is the only thing that works :) – Ginkgo

Answer 3

The problem is related to IIS rather than the schedulers Quartz.NET, Hangfire, etc. On the other hand, there are lots of solution methods posted on the web, but only some of them is working. In my opinion, there is no need to apply lots of configuration settings. Just install Keep Alive Service For IIS 6.0/7.5 on the server to which you publish your application and enjoy. After that, your published application will be alive after application pool recycling, IIS/Application restarting, etc.

Smashing

Great answer, this is it. IIS recycles the app pool, and shuts down with inactivity. Which is why quartz repeat schedules don't fire. Check your IIS app pool settings!!! – Sight

Answer 4

I just ran into a similar issue that might bite someone else - it brought me to this SO question after turning on debug and getting that 'batch acquisition of 0 triggers' message.
I had a job that was every 2 hours like this:

"0 0 0/2 * * ?"

And I wanted to make it more often, so every 2 minutes like this:

"0 0/2 0 * ?"

I even tried https://cronexpressiondescriptor.azurewebsites.net/ which gave me a big clue I should have read more carefully: Every 2 hours, on day 0 of the month which eventually made me realize what I really meant was:

"0 0/2 * * ?"

So the lesson was, when 'shifting left' your cron, back-fill with *, not 0.

Hope that helps someone else.

Mainsail