I’m trying to understand I/O Completion Ports and specifically how they relate to using
await for I/O.
The infamous article There is No Thread talks about IOCPs being borrowed briefly after the I/O is complete. Because the whole point of the article is to show that when the fancy hardware-level I/O stuff is in-flight, there is no thread that is consumed by a loop like
Is the I/O done yet? No. Is the I/O done yet? No. Is the I/O done yet? No. …
But then I’m looking at this article which says that a
“component is in charge of checking the completion port for queued
and gives an example like
public class IOCompletionWorker
public unsafe void Start(IntPtr completionPort)
var result = Interop.GetQueuedCompletionStatus(
var overlapped = Overlapped.Unpack(nativeOverlapped);
var asyncResult = ((FileReadAsyncResult)overlapped.AsyncResult);
var completionPortThread = new Thread(() => new IOCompletionWorker().Start(completionPortHandle))
IsBackground = true
which to me looks like there is some polling going on.
I guess my questions boil down to
- Is it true to say that a .NET application has 2 types of thread pools — (1) “worker threads” and (2) “I/O threads”?
- If it’s true, is there a fixed number, specified in a configuration, like M worker threads and N I/O threads? And what is usually the ratio of M to N?
- When exactly are I/O threads used?
Both articles are correct in their own way.
IOCPs are not threads. They can be seen as some kind of queue in which the kernel (or also regular user-mode code, through PostQueuedCompletionStatus) can post completion items. There is no inherent threading model or threads associated with IOCPs themselves, they are simply multiple producer-consummer queues.
Let’s take network Sockets as an example, but this would be true for any kind of asynchronous work:
- You call your WSARecv on your overlapped-mode socket bound to an IOCP, it’s up to the network driver to do whatever is necessary to setup the actual request for reception of data. There is no thread actively waiting for your data to arrive.
- Data arrives. The operating system is woken up by the hardware. The operating system will give the network driver some CPU time in the kernel to process the incoming event. The network driver processes the interrupt, and then because your socket was bound to an IOCP, posts a completion item to your IOCP queue. The request is complete.
There is no actual user mode thread from your process involved in any of this operation (beyond just the initial asynchronous call). If you want to act on the fact that your data has arrived (which I assume you do when you’re reading from a socket!), then you have to dequeue the completed items from your IOCP.
The point of IOCPs is that you can bind thousands of IO handles (sockets, files, …) to a single IOCP. You can then use a single thread to drive those thousands of asynchronous processes in parallel.
Yes, that one thread doing the GetQueuedCompletionStatus is blocked while there is no completion pending on the IOCP, so that’s probably where your confusion came from. But the point of IOCPs is that you block that one thread while you can have hundreds of thousands of network operations pending at any given time, all serviced by your one thread. You would never do a 1-to-1-to-1 mapping between IO handle/IOCP/Servicing Thread, because then you would lose any benefit from being asynchronous, and you might as well just use synchronous IO.
The main point of IOCPs is to achieve impressive parallelism of asynchronous operations under Windows.
I hope this clarifies the confusion.
As for the specific questions
- Yes, the .Net framework has two pools. One is purely for user-mode general purpose work, the “Worker” threadpool. The other is the “IO” threadpool. This second one is so that all the IOCP management can be hidden from you when writing high-level C# code and so that your asynchronous socket just works like magic.
- This is all implementation detail that can change at any time, but the answer is that both pools are independent. If you have massive work bandwidth happening on the worker threadpool and the framework decides that your overall throughput would increase by adding new threads, it will add threads to the worker pool alone, and not touch the IO pool. The same goes for the IO pool, if you have misbehaving code that blocks IO threads in their callbacks, it will spawn new IO pools and not touch the worker pool. You can customize the numbers using ThreadPool.SetMinThreads/SetMaxThreads, but this is usually a sign that your process misuses the threadpool.
- IO threads are used when there are items dequeued from the threadpool’s internal IOCP. In typical code, this will be when an asynchronous operation has completed on some IO handle. You can also queue items yourself through UnsafeQueueNativeOverlapped, but that’s a lot less common.
Pure managed asynchronous operations (like doing async-await with Task.Delay, for example) do not involve any IO handle, and so they don’t end up being posted to an IOCP by some driver, and so those would fall under the “Worker” category.
As a side note, you can tell Worker threads from IO threads by their callstack.
Worker threads will start their managed callstack with “ThreadPoolWorkQueue.Dispatch”, whereas IO threads will start their managed callstack with “_IOCompletionCallback.PerformIOCompletionCallback”. This is all implementation detail that can change at any time, but it can be helpful to know what you are dealing with when debugging your managed code.
IOCP is basically a queue, to which OS can add messages, each message representing completion of some IO operation. But for OS to know where to post completion of some IO operation – each IOCP can have multiple “file” handles associated with it. Despite its name – file handle can represent any IO device (file, socket, pipe, and so on). When IO operation on an object represented by file handle completes – completion message is posted to IOCP(s) associated with that handle.
Thread pool has just one IOCP. When asynchonous IO is needed – related handle is bound to that IOCP. For example, when you create
FileStream in asynchronous mode – related file handle will be bound to thread pool IOCP via
ThreadPool.BindHandle method. When IO operation (like
BeginRead) completes for that file – completion will be posted to thread pool IOCP.
Multiple threads can listen to the same IO completion port. They are sleeping (not consuming CPU) and when IO completion message arrives – only one of them is woken up to react on it (for example – execute callback if any, or pass that callback for “worker” thread to execute). Threads which are doing this (waiting on single IOCP of thread pool) are IO completion threads.
As you see, when IO operation is in progress – no thread is blocked by waiting for this specific operation to complete. IO completion threads are waiting for all IO completion operations. No thread is created just for this one. You can have 1 IO completion thread and then issue 100 asynchronous IO operations. All completions will be queued in IOCP and will be handled by one IO completion thread one at a time (that doesn’t mean all callbacks also will be executed sequentially – callback can be executed on “worker” thread pool thread, while IO completion thread can just process completion, queue callback to “worker” thread and go on to the next completion message).
There is just one thread pool, but threads in it might have different roles: “worker” threads (general purpose), and io completion threads (waiting on single IOCP of thread pool). They are created and destroyed as necessary (thread pool uses different heuristics to decide how many it needs), though you can affect this process with
ThreadPool.SetMinThreads(workerThreadsNumber, ioCompletionThreadsNumber) and