Quartz.net Scheduler.Shutdown(true) not killing jobs
Asked Answered
S

1

7

I have a running quartz job and terminating my BackgroundService, for some reason despite calling scheduler.Shutdown(true) the job remains running.

Even when looping through and interrupting jobs, the program closes if before the threads exit.

Aside from my code below, would i be looking at writing a custom IScheduler to ensure runnings jobs are halted on shutdown?

This is my IJob Execute method:

public async Task Execute(IJobExecutionContext context)
{
    var cancellationToken = context.CancellationToken;

    while (cancellationToken.IsCancellationRequested == false)
    {
        // Extension method so we catch TaskCancelled exceptions.
        await TaskDelay.Wait(1000, cancellationToken);
        Console.WriteLine("keep rollin, rollin, rollin...");
    }
    Console.WriteLine("Cleaning up.");
    await Task.Delay(1000);
    Console.WriteLine("Really going now.");
}

This is my shutdown loop (calling shutdown directly doesn't interrupt any running jobs):

internal class QuartzHostedService : IHostedService
{
    // These are set by snipped constructor.
    private readonly IJobSettings jobSettings;
    private readonly ILogger logger;
    private readonly IScheduler scheduler;

    private async Task AddJobsToScheduler(CancellationToken cancellationToken = default)
    {
        var schedule = SchedulerBuilder.Create();

        var downloadJob = JobBuilder.Create<StreamTickersJob>().Build();

        var downloadJobTrigger = TriggerBuilder
            .Create()
            .ForJob(downloadJob)
            .WithDailyTimeIntervalSchedule(
                x => x.InTimeZone(serviceTimeZone)
                    .OnEveryDay()
                    .StartingDailyAt(new TimeOfDay(8,0))
                    .EndingDailyAt(new TimeOfDay(9,0)))
            .Build();

        await this.scheduler.ScheduleJob(downloadJob, downloadJobTrigger, cancellationToken);
    }

    public QuartzHostedService(IJobSettings jobSettings, IScheduler scheduler, ILogger<QuartzHostedService> logger)
    {
        this.jobSettings = jobSettings;
        this.scheduler = scheduler;
        this.logger = logger;
    }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        this.logger.LogInformation("Quartz started...");
        await AddJobsToScheduler(cancellationToken);
        await this.scheduler.Start(cancellationToken);
    }

    public async Task StopAsync(CancellationToken cancellationToken)
    {
        await this.scheduler.PauseAll(cancellationToken);

        foreach (var job in await this.scheduler.GetCurrentlyExecutingJobs(cancellationToken))
        {
            this.logger.LogInformation($"Interrupting job {job.JobDetail}");
            await this.scheduler.Interrupt(job.JobDetail.Key, cancellationToken);

        }
        await this.scheduler.Shutdown(cancellationToken);
    }
}

I can confirm IHost is not killing my app abruptly (at least not for a couple of seconds test pause) as I set a breakpoint at the end of the main program as below:

public static void Main(string[] args)
{
    // Wrap IHost in using statement to ensure disposal within scope.
    using (var host = CreateHostBuilder(args)
                                .UseSerilog<Settings>(Settings.Name)
                                .UseConsoleLifetime()
                                .Build()
                                .UseSimpleInjector(container))
    {
        // Makes no difference if I shutdown jobs here.
        // var lifetime = container.GetInstance<IHostApplicationLifetime>();            
        // lifetime.ApplicationStarted.Register(async () => { });
        // lifetime.ApplicationStopping.Register(async () => { });

        var logger = container.GetInstance<ILogger<Program>>();

        try
        {
            host.Run();
        }
        catch (Exception ex)
        {
            logger.LogCritical(ex, ex.Message);
        }

        // We reach here, whilst Jobs are still running :(
        logger.LogDebug($"Finish {nameof(Main)}().");
    }
}

I have also added from what I have found online the below, but still it doenst wait on shutdown:

var props = new NameValueCollection
{
    {"quartz.scheduler.interruptJobsOnShutdownWithWait", "true"},
};

var scheduler = AsyncContext.Run(async () => await new StdSchedulerFactory(props).GetScheduler());

My workaround with a delay to allow jobs to terminate below works, but is so dodgy - kindly advise how I can get this working properly without an brittle arbitrary delay:

public async Task StopAsync(CancellationToken cancellationToken)
{
    await this.scheduler.PauseAll(cancellationToken);
    foreach (var job in await this.scheduler.GetCurrentlyExecutingJobs(cancellationToken))
    {
        this.logger.LogInformation($"Interrupting job {job.JobDetail}");
        await this.scheduler.Interrupt(job.JobDetail.Key, cancellationToken);

    }
    await Task.Delay(3000);
    await this.scheduler.Shutdown(cancellationToken);
}
Struck answered 27/6, 2021 at 17:12 Comment(5)
Hi, I suggest to add graceful IHostedService shutdown to your BackgroundService. You can find great article. Or try with more simple way, add graceful shutdown in Startup.cs. Add dependecy in your method Configure - IHostApplicationLifetime lifetime. And register ApplicationStopping event - lifetime.ApplicationStopping.Register(() => { Task.Delay(TimeSpan.FromSeconds(10)).GetAwaiter().GetResult(); }); I think 10 seconds will be enough for Quartz to stop all jobs.Epicedium
Thanks @DarkSideMoon, I have posted up an answer inspired by your comments thank you. I still had to create StopAllJobs function but this seems to work at least per my use case and allow jobs to shutdown.Struck
Use await Task.Delay(2000) instead of Thread.Sleep(2000) in async methods.Likeminded
@Epicedium why not Thread.Sleep if you call Task.Delay in sync-over-async manner? .GetAwaiter().GetResult() for not completed yet Task kills its asynchronous nature.Likeminded
Thanks have updated that to use Task.Delay(). Same issue the worker doesn't shutdown and isn't waited to terminate.Struck
C
5

If you check the source code of generic host, you'll find that on host shutdown it waits for a default shutdown timeout, which is 5 seconds. It means that if your jobs take more time to complete, the host will exit by timeout and so will the application.

In addition, based on your comment, the scheduler has to be configured to interrupt the running jobs on shutdown:

var props = new NameValueCollection
{
    {"quartz.scheduler.interruptJobsOnShutdownWithWait", "true"},
};

var scheduler = AsyncContext.Run(async () => await new StdSchedulerFactory(props).GetScheduler());

and invoked with the waitForJobsToComplete parameter set to true for shutdown:

await this.scheduler.Shutdown(waitForJobsToComplete: true, cancellationToken);

to ensure that scheduler only exits when all jobs are completed.

To guarantee that the application exits only after all jobs are interrupted and completed, you could initiate shutdown after the host has exited:

public static Task Main(string[] args)
{
    using (var host = CreateHostBuilder(args)
        .UseSerilog<Settings>(Settings.Name)
        .UseConsoleLifetime()
        .Build()
        .UseSimpleInjector(container))
    {
        var logger = container.GetInstance<ILogger<Program>>();

        try
        {
            await host.RunAsync();

            var scheduller = container.GetInstance<IScheduler<Program>>();
            scheduller.Shutdown(true);
        }
        catch (Exception ex)
        {
            logger.LogCritical(ex, ex.Message);
        }

        logger.LogDebug($"Finish {nameof(Main)}().");
    }
}
Clam answered 15/7, 2021 at 22:5 Comment(5)
Thanks Andrii, I have ruled the shutdown time out, I have added a delay of a couple of seconds (within the 5 seconds) and can confirm the Quartz is immediately completing the shutdown function without waiting. It is not .net core IHost being terminated, this is definitely a quartz.net issue (or my misunderstanding of it).Struck
@Struck in that case you would want to call await this.scheduler.Shutdown(true, cancellationToken); to ensure that shutdown method doesn't exit before all jobs are stopped.Clam
thank you - that combined with props and {"quartz.scheduler.interruptJobsOnShutdownWithWait", "true"} setting I mentioned in the 2nd to last code block of my question, then works thank you. No need to to any looping just call await this.scheduler.Shutdown(true, cancellationToken); If you can please update your answer to include both of these aspects, I will then award :)Struck
I have update the answer, thanks for the hint on interruptJobsOnShutdownWithWait. I wasn't aware of it.Clam
Perfect thank you - will award bounty tomorrow its not allowing me to award so soon after issuing. Thanks for taking the time to respond and resolve the issue.Struck

© 2022 - 2024 — McMap. All rights reserved.