How do Completion Port Threads of the Thread Pool behave during async I/O in .NET / .NET Core?

Question

The .NET / .NET Core Thread Pool uses two different categories of threads internally: worker threads and I/O Completion Port (IOCP) threads. Both are just usual managed threads, but used for different purposes. Via different APIs (e.g. Task.Start or ThreadPool.QueueUserWorkItem) I can start CPU-bound async operations on the worker threads (which shouldn't block, otherwise the Thread Pool would probably create additional worker threads).

But what about performing I/O-bound asynchronous operations? How do the IOCP threads behave exactly in these situations? Specifically, I have the following questions:

• If I start an async I/O operation (e.g. for file, pipe, or network), I suspect that the current thread dispatches the async request. I also know (via the book "CLR via C#") that the CLR registers to an I/O completion port that is used to perform overlapped async I/O. I suspect that this IOCP is bound to the async operation so that it can queue the async operation result to the Thread Pool later. Thus, is my assumption correct that no IOCP thread is touched when an async request is started?
• I suspect that when the result of the async I/O operation is reported via the I/O completion port of the CLR, this is the place where IOCP threads come into place. The result is queued to the Thread Pool and an IOCP thread is used to handle it. However, when reading through some forum threads like this one on MSDN, I get the feeling that IOCP threads are actually used to dispatch the request and then block until the result is back. Is this the case? Are IOCP threads blocking while the I/O operation is handled by the opposing system?
• What about async await and SynchronizationContext? Does an IOCP thread handle the async I/O response and then e.g. queue the continuation on the UI thread (assuming that ConfigureAwait(false) is not called)?
• What about .NET Core on Linux / MacOS X? There are no I/O completion ports - are they emulated in any kind of way?

Zora

There is no thread is always recommended reading here. – Hedveh

This question is too broad. In general, no, it doesn't take any async code to get an I/O request started. The I/O manager must already deal with multiple processes asking for I/O so it takes care of queuing the driver requests by itself. SynchronizationContext in general plays no role, except for XxxxAsync() methods in early .NET Framework versions that made an effort to get an event to raise on the expected thread. The unixes have their own flavor of it, macOS uses kqueue and Linux uses epoll. – Floeter

@Hedveh Thanks for the link, Stephen Toub's post answered almost all my questions. – Zora

@HansPassant you're right, the question is too broad (or not well-formulated). Thanks for the hints to kqueue and epoll. – Zora

Just for the record, Stephen Toub is the genius who designed a lot of the .NET async systems. Stephen Cleary (the "other Stephen") is the guy who just likes to write about it. – Sthenic

Answer 1

Damien and Hans pointed me into the right direction in the comments, which I want to sum up in this answer.

Damien pointed to Stephen Cleary's awesome blog post which answers the first three points:

• The async I/O operation is dispatched on the calling thread. No IOCP thread is involved.
• Consequently, no IOCP threads block during async I/O.
• When the result is returned to the .NET application, an IOCP thread is borrowed to mark the task complete. The continuation is queued to the target SynchronizationContext or the thread pool.

Hans pointed out that there are similar mechanisms to IOCP in Linux (epoll) and MacOS (kqueue).

UPDATE 2023-04-07: my original explanation of IOCP threads not blocking is wrong. When an IOCP thread is created, it is bound to an I/O Completion Port (on Windows) and then calls GetQueuedCompletionStatusEx in a loop. This call blocks until it is woken up because at least one new event is available on the IOCP. The IOCP thread makes its loop run to process all dequeued events (if you use the TPL, the corresponding task will be updated and the continuation will be queued, either on the caller's SynchronizationContext or on a Thread Pool worker thread). After that, the IOCP will call GetQueuedCompletionStatusEx and either block because no events are available, or the loop body can run again. This can be seen here: ThreadPoolPortable.IO.Windows.cs

While IOCP threads do block, the following statements are also true:

• No thread is created because an async method was called. The Thread Pool created and bound the IOCP threads beforehand, they probably have already been blocked before the async operation even started.
• The .NET Thread Pool will increase or decrease the number of IOCP threads depending on the async I/O workload that your app triggers.

UPDATE 2022-06-23: some people asked why IOCP threads do not block during I/O operations. It's important to understand how the Thread Pool manages its threads internally. The Thread Pool keeps a number of threads available, i.e. they reside in memory but are actually in a sleep state. This way when work comes in, you do not pay the cost of creating a new thread (my measurements on the topic show that creating a new thread instead of using an existing one is about 80 times slower). When work is available, it is queued to one of the sleeping threads, their state is changed from sleeping to ready to execute, and thus the operating system can pick this thread up in the next context switch (which usually occurs every 15ms) and assign it to one of your CPU cores. After your work is done, the thread is either put to sleep again or the next task can be executed on it. This is true for both worker threads and IOCP threads.

To conclude, IOCP threads do not block during the I/O operation, because only once the I/O completion port signals that the operation is complete, work is queued on an IOCP thread to mark the corresponding Task or Task<T> as completed and enqueue a possible continuation either on a worker thread or on the original caller thread if it has a synchronization context assigned to it and ConfigureAwait(false) was not called. During the I/O operation, the IOCP thread that will later execute the aforementioned work is either sleeping or handling completions from other I/O completion ports.

Zora

But in this url, https://mcmap.net/q/692023/-iocp-threads-clarification, the second answer have showed Worker threads: 0, Completion port threads: 30, Total threads: 34 when call I/O operation. ######## There are 30 I/O thread for I/O operation. But this post said, the I/O thread does not waiting for I/O. If it is true, it should show less 30 I/O thread for I/O operation. I can not understand that. – Illiterate

A completion port thread will only be used briefly to mark the corresponding task in .NET as completed. If you start several FileStream instances in a loop, then the thread pool will create more IOCP threads to handle all the completing I/O request packets. However, the IOCP threads will not block during the actual I/O operation. The operation is started on the calling thread, the IOCP thread marks the task as completed once the I/O operation is done, and a continuation will either be performed on a regular background thread or the calling thread if there was a SynchronizationContext. – Zora

Thank you and I have a more question. ## you mention that the async I/O operation is dispatched on the calling thread. No IOCP thread is involved. And you said that the thread pool will create more IOCP threads to handle all the completing I/O request packets. --> I got that IOCP threads will created when initial I/O operation, and wait for I/O complete. When IOCP threads waiting for I/O complete, they(IOCP threads) are blocking. If I have misunderstand, please tell me. thanks a lot. – Illiterate

No, you're wrong: 1) on your current thread, an I/O Request Packet is created and handed to the OS which starts the actual I/O operation. 2) The device driver is notified by the e.g. network or disk controller once the I/O operation is done. 3) Your .NET process is now notified by the OS that the I/O operation is done. The .NET runtime will only now use an IOCP thread to quickly mark the task as completed and enqueue the continuation (no blocking). 4) The continuation will be executed on the initial thread (if it has a SynchronizationContext) or on another background thread of the Thread Pool. – Zora

thanks, I understand it. And I also reference this repo github.com/dschenkelman/async-io-talk. – Illiterate

Can you explain how the I/O threads are not blocked? My understanding was that the I/O threads are essentially running in a while(true) loop calling the blocking GetQueuedCompletionStatus function. Is that not the case? – Tergum

@RobL Sorry for taking so long. Just noticed your question. Thread Pool threads are created and put in a sleep state when they are not used. This means they reside in memory but not code is executed on them. Only once you run tasks against the Thread Pool, one thread is picked and changes it state from sleeping to waiting for execution. The OS will can choose this thread during context switches and assign it to execute on a CPU core. Only then the code of the task will be executed. Once the work is done, the thread will not die but put to sleep or assigned the next task that is queued. – Zora

@RobL This way, the thread is not blocked when there is nothing to be done. It can be used for other concurrent work. It does not matter if tasks are queued to worker threads, or if I/O completion ports queue work to IOCP threads. The principle for both is the same. – Zora

@Zora When and how does the blocking call to GetQueuedCompletionStatus happen? I thought that when an IO operation was submitted to the OS, an IO thread was spun up and basically called GetQueuedCompletionStatus and once the operation completed, this method returned a result and later on the async callback runs. Is my understanding wrong? Thanks for your insights. – Tergum

@RobL I have never observed this behavior. The IOCP threads were never blocked during I/O operations. Can you point to some docs where you read about this? – Zora

@RobL I've updated the answer to indicate that you are right: IOCP threads do block until they are notified because an event is available on the bound IOCP. – Zora

Answer 2

The IOCP thread can be easily blocked if you process the I/O result directly in the thread (after GetQueuedCompletionStatus returns successfully). But usually the processing is transferred to another thread, for example by calling PostQueuedCompletionStatus,posting task or other method. In addition, you can set a timeout in GetQueuedCompletionStatus, and if nothing happens, do other work, although this is probably not the best option.

Darlenadarlene

While generally correct, my specific question was about IOCP threads in the .NET thread pool, and after some further research, I found out that they indeed do block until their state is set from blocked to ready-to-run when an IOCP event is available. They than update the corresponding .NET Task and queue the continuation, either on the SynchronizationContext of the initial caller, or a Thread Pool worker thread. – Zora