.Net core Hosted Services guaranteed to complete

Question

I'm looking at .Net-Core 2.1 new feature Hosted Services, I see that they are very similarly modelled to QueueBackgroundWorkItem

The Queue Background work item seems to have a limitation that the task must execute within 90 seconds

The AppDomain shutdown can only be delayed 90 seconds (It’s actually the minimum of HttpRuntimeSection.ShutdownTimeout and processModel shutdownTimeLimit). If you have so many items queued that they can’t be completed in 90 seconds, the ASP.NET runtime will unload the AppDomain without waiting for the work items to finish.

Do hosted services have different behaviors or does this limitation still apply?

I'm worried that if I queue something on my hosted service, if it's a really long running task, is it still guaranteed to complete?

Abortion

If you take a look at the web host source github.com/aspnet/Hosting/blob/… specifically the stop propcess you will see that the cancellation token is based pm the Options.ShutdownTimeout What the default of that value is I am not certain, but it is somewhere to start looking. – Ossa

@Nkosi, thanks! even if you don't find anything that should suffice as an answer, my main concern is that it's still possible for tasks to not complete. I'll make a few edits to make that more clear – Abortion

This may be of interest learn.microsoft.com/en-us/dotnet/standard/… – Ossa

Interesting, it seems like they have the same behaviors of background services, I’m writing a rest framework, and I offload my notifications from my websockets. I was doing this by default so if a user makes an update request they don’t have to wait for the socket to push. I guess now I’ll just make a configuration for it. – Abortion

If you want to post an excerpt from the github code that shows the shutdown timer and a quote from the docs which mention application pool resets could cause issues. I’d mark this resolved :) – Abortion

How does one set the ShutdownTimeout value for HostBuilder ? I see it currently has no extension method UseShutdownTimeout as provided with WebHostbuilder ? Thanks. – Lebbie

@TimDude I'm not sure if you can set it. Nkosi, posted the source so you can track it back, I think the default is 90 seconds, which matches IIS default I believe but this is a while back so I don't fully remember – Abortion

Thanks @johnny5 I'm experiencing a 5 second timeout from StopAsync in the stack trace which seems to align to my GitHub search results. – Lebbie

Hi @johnny5 I have raised the ShutdownTimeout question - hopefully it will get some traction. #52911216 – Lebbie

@johnny5 I've found how to set ShutdownTimeout, answered on the above link. Note I have not experimented beyond the 90 second limit described in your original question. – Lebbie

Answer 1

As part of trying to gracefully shut down the web host builds a cancellation token in combination with the configured ShutdownTimeout

var timeoutToken = new CancellationTokenSource(Options.ShutdownTimeout).Token;
if (!cancellationToken.CanBeCanceled)
{
    cancellationToken = timeoutToken;
}
else
{
    cancellationToken = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, timeoutToken).Token;
}

Source

This becomes the shut down token when stopping hosted services

// Fire the IHostedService.Stop
if (_hostedServiceExecutor != null)
{
    await _hostedServiceExecutor.StopAsync(cancellationToken).ConfigureAwait(false);
}

In researching the potential issues with hosted services I came across the following from official documentation.

Deployment considerations and takeaways

It is important to note that the way you deploy your ASP.NET Core WebHost or .NET Core Host might impact the final solution. For instance, if you deploy your WebHost on IIS or a regular Azure App Service, your host can be shut down because of app pool recycles. But if you are deploying your host as a container into an orchestrator like Kubernetes or Service Fabric, you can control the assured number of live instances of your host. In addition, you could consider other approaches in the cloud especially made for these scenarios, like Azure Functions.

But even for a WebHost deployed into an app pool, there are scenarios like repopulating or flushing application’s in-memory cache, that would be still applicable.

The IHostedService interface provides a convenient way to start background tasks in an ASP.NET Core web application (in .NET Core 2.0) or in any process/host (starting in .NET Core 2.1 with IHost). Its main benefit is the opportunity you get with the graceful cancellation to clean-up code of your background tasks when the host itself is shutting down.

Now from this based on your concerns I would gather that there is no guarantee that your long running tasks will complete, but they may be given the opportunity for a graceful cancellation based on the hosting environment as explained in the quoted statement above.

Ossa

does this mean that hosted services are only run when requests come in? e.g if no request comes in no background task will run – Abortion

@johnny5 It depends how you build them. If they listen to queue then they do nothing until message appears on this queue. – Overboard

@PiotrPerak how can a background service run if iis gives no resources to the application – Abortion

In Program.Main you have host.Run() which stops application from stopping. – Overboard