Why is IsCancellationRequested not set to true on stopping a BackgroundService in .NET Core 3.1?

Question

I've read most articles I can find about IHostApplicationLifetime and CancellationToken's in .NET Core 3.1, but I cannot find a reason why this is not working.

I have a simple BackgroundService which look like the following:

    public class AnotherWorker : BackgroundService
    {
        private readonly IHostApplicationLifetime _hostApplicationLifetime;

        public AnotherWorker(IHostApplicationLifetime hostApplicationLifetime)
        {
            _hostApplicationLifetime = hostApplicationLifetime;
        }

        public override Task StartAsync(CancellationToken cancellationToken)
        {
            Console.WriteLine($"Process id: {Process.GetCurrentProcess().Id}");
            _hostApplicationLifetime.ApplicationStarted.Register(() => Console.WriteLine("Started"));
            _hostApplicationLifetime.ApplicationStopping.Register(() => Console.WriteLine("Stopping"));
            _hostApplicationLifetime.ApplicationStopped.Register(() => Console.WriteLine("Stopped"));

            return Task.CompletedTask;
        }

        protected override Task ExecuteAsync(CancellationToken stoppingToken)
        {
            Console.WriteLine("Executing");
            return Task.CompletedTask;
        }

        public override async Task StopAsync(CancellationToken cancellationToken)
        {
        // This actually prints "Stop. IsCancellationRequested: False". Why?
            Console.WriteLine($"Stop. IsCancellationRequested: {cancellationToken.IsCancellationRequested}");
            await base.StopAsync(cancellationToken);
        }
    }

The ConsoleLifetime is added by default, which listens to Ctrl+C and SIGTERM and informs the IHostApplicationLifetime. I guess IHostApplicationLifetime in turn should then cancel all CancellationTokens? Here's a good article on the subject. So why is the output from the above code snippet the following?

Hosting starting
Started
Hosting started
(sends SIGTERM with `kill -s TERM <process_id>`)
Applicationis shuting down...
Stop. IsCancellationRequested: False
Stopped
Hosting stopped

I would expect it to log Stop. IsCancellationRequested: True

I want to be able to pass this token around, to other service calls, for them to have the capability to shutdown gracefully.

Hittel

learn.microsoft.com/en-us/dotnet/api/… – Padilla

I don't think it is the same the IsCancellationRequested from the same token. For shutting down gracefully you need to do that on the ExecuteAsync. When you've a long runnning task there and checking the IsCancellationRequested there it will be true. CloseAsync is just for cleaning up some data there not for gracefully shutdown. The token passed into ExecuteAsyncis not the same as CloseAsync. – Fogel

Answer 1

There are a lot of different cancellation tokens here, and several different abstractions (IHostApplicationLifetime, IHostedService, BackgroundService). It takes a while to untangle everything. The blog post you linked to is fantastic, but doesn't go into detail on the CancellationTokens.

First, if you're going to use BackgroundService, I recommend reading the code. Also, I strongly recommend not overriding StartAsync and StopAsync; BackgroundService uses these in a very particular way.

IHostedService has two methods. StartAsync starts the service running (possibly asynchronously); it takes a CancellationToken that indicates the "start" operation should be cancelled (I haven't checked, but I assume this token is only triggered if the app is shutdown almost immediately). Note that StartAsync needs to complete before the hosted service is considered in the "started" or "running" state. Similarly, StopAsync stops the service (possibly asynchronously). StopAsync is invoked when the application begins its graceful shutdown. There's a timeout for the graceful shutdown period, after which the application begins its "I'm serious now" shutdown. The CancellationToken for StopAsync represents the transition from "graceful" to "I'm serious now". So it's not set during that graceful shutdown timeout window.

If you use BackgroundService instead of IHostedService directly (like most people do), you get a different CancellationToken in ExecuteAsync. This one is set when BackgroundService.StopAsync is invoked - i.e., when the application has started its graceful shutdown. So it's roughly equivalent to IHostApplicationLifetime.ApplicationStopping, but scoped to a single hosted service. You can expect the BackgroundWorker.ExecuteAsync CancellationToken to be set shortly after IHostApplicationLifetime.ApplicationStopping is set.

Note that all of these CancellationTokens represent something different:

• IHostedService.StartAsync's CancellationToken means "abort the starting of this service".
• IHostedService.StopAsync's CancellationToken means "stop this service right now; you're out of the grace period".
• IHostApplicationLifetime.ApplicationStopping means "the graceful shutdown sequence for this entire application has started; everyone please stop what you are doing".
  • As part of the graceful shutdown sequence, all IHostedService.StopAsync methods are invoked.
• BackgroundService.ExecuteAsync's CancellationToken means "stop this service".

An interesting note is that BackgroundService types don't normally see the "I'm serious now" signal; they only see the "stop this service" signal. This is likely because the "I'm serious now" signal represented by a CancellationToken is somewhat confusing.

If you look into the code for Host, the shutdown sequence has even more cancellation tokens used in its shutdown sequence:

1. IHost.StopAsync takes a CancellationToken meaning "the stop should no longer be graceful".
2. It then starts a CancellationToken-based timeout for the graceful timeout period.
3. ... and another linked CancellationToken that is fired if either the IHost.StopAsync token is fired or if the timer elapsed. So this one also means "the stop should no longer be graceful".
4. Next it calls IHostApplicationLifetime.StopApplication, which cancels the IHostApplicationLifetime.ApplicationStopping CancellationToken.
5. It then invokes StopAsync for each IHostedService, passing the "stop should no longer be graceful" token.
   • All BackgroundService types have their own CancellationToken (which was passed to ExecuteAsync during startup), and those cancellation tokens are cancelled by StopAsync.
6. Finally, it invokes IHostApplicationLifetime.NotifyStopped, which cancels the IHostApplicationLifetime.ApplicationStopped CancellationToken.

I count 3 for the "no longer graceful" signal (one passed in, one timer, and one linking those two), plus 2 on IHostApplicationLifetime, plus 1 for each BackgroundService, for a total of 5 + n cancellation tokens used during shutdown. :)

Zobe

Now when you say it, it all feels more reasonable. Indeed there's a lot going on here. It's a bit unfortunate that the use of the same CancellationToken type everywhere makes the code a bit non self-explanatory. You'll have to treat them as very context-specific tokens, then it becomes a bit easier to reason about. Thanks Stephen! – Hittel

So if I had a client of some sort that I would like to close the connection to, let's say a service bus subscription client. This API is asynchronous and i'd like to await it, however i'm not sure exactly where to call await _subscriptionClient.CloseAsync (without overriding the StopAsync method that is). IHostApplicationLifetime doesn't support asynchronous operations as far as I can tell, and I can't use the CancellationToken passed to ExecuteAsync either (because token.Register is neither asynchroous). Do you have any ideas here? Thanks. – Hittel

If you're using BackgroundService, I'd just call CloseAsync in a finally branch just before the code exits ExecuteAsync. Bonus points: use IAsyncDisposable. :) – Zobe

The problem is that the code I have in ExecuteAsync is not itself asynchronous, but rather a synchronous call that just instantiates a subscription client and wires up a message handler. Also, I don't think I can use IAsyncDisposable here since I want to "gracefully" call CloseAsync to stop the subscription client from receiving further messages long before DisposeAsync would even be called. Update: I guess my real problem here is that the subscription client doesn't seem to support cancellation token, so there is no way for me to pass the token from ExecuteAsync further down. – Hittel

Right; it sounds like a client API issue. If it would be proper to call CloseAsync from a different thread than the one doing the synchronous Receive, then you could Register that up (with an async void lambda or sync-over-async hack) to the CancellationToken passed into ExecuteAsync. It wouldn't be pretty, but since the client doesn't support cancellation and requires an asynchronous teardown (an odd combo), you won't end up with a pretty solution anyway. – Zobe

Answer 2

The CancellationToken passed to the StopAsync indicate if the BackgroundService must execute a gracefull shutdown or a hard one.

You stop the process using the kill -s TERM so it send the SIGTERM signal asking the application to shutdown gracefully. Therefore the IsCancellationRequested property is still at false.

To pass a token to other services calls, you have to provide your own CancellationToken. You can use a CancellationTokenSource to manage token creation and cancellation.

Millard