Why cpu bound is better with blocking I/O and I/O bound is better with non blocking I/O

Question

I have been told that for I/O bound applications, non blocking I/O would be better. For CPU bound applications, blocking I/O is much better. I could not find the reason for such a statement. Tried google, but few articles just touches the topic with not much details. Can someone provide the deep depth reason for it?

With this, I want to clear myself with what are the short coming of non blocking I/O as well.

After going through another thread here,a reason I could relate was out was if the I/O process is heavy enough then only we can see significant performance improvements using non blocking I/O. It also states that if the number of I/O operations is large(a typical web application scenario) where there are many requests looking out for I/O requests, then also we see significant improvements using non blocking I/O.

Thus my questions boil down to the following list:

1. In case of a CPU intensive applications, is it better to start a threadpool(or executionContext of scala) and divide the work between the threads of the threadpool.(I guess it has definitely an advantage over spawning your own threads and dividing the work manually. Also using asyn concepts of future, even CPU intensive work can be returned using callbacks hence avoiding the issues related to blocking of multi threading?). Also if there is a I/O which is fast enough, then do the I/O using blocking principles on the threads of thread pool itself. Am I right?

2. What are actually short comings or overheads of using a non blocking I/O technically? Why we don't see much performance gains of using non blocking I/O if the I/O is fast enough or if there are very less I/O operations required? Eventually it is the OS which is handling I/O's. Irrespective of whether the number of I/O's are large or small, let OS handle that pain. What makes the difference here.

Giles

Non-blocking IO would be "better" only if you had something else to do while waiting on IO (for example, process other connections). – Spokeshave

Which means using either multi threading or asynchronous programming? so that the blocking thread is not the one tying up. – Giles

Hmm. I can see why blocking I/O may be sufficient (because you cannot do anything else useful with your CPU anyway), but I don't see why it would be better (at least from a performance standpoint, it may be easier for the programmer). – Adlib

what would be your reason for blocking I/O being "sufficient"? – Giles

non-blocking I/O frees up your CPU thread for something else. If there is nothing else to do (or so little that a couple of threads are not too much of an overhead), then there is not too much to be gained by being non-blocking. – Adlib

Totally make sense. Now the next question is: what are the overheads related to a non blocking I/O? – Giles

Good question. I always thought I/O is inherently non-blocking (being based on hardware interrupts), and blocking I/O with threads is a convenient abstraction layered on top of that (which brings along some overhead). – Adlib

Yep. I agree with you. Everything looks good. But I can't figure out a reason where non blocking I/O would actually be harmful or of no use. In case of cpu bound applications also, if an I/O operation is done using non blocking I/O, what could possibly be the reason for less efficient as compared to do it using blocking I/O.. sigh!!! – Giles

There's plenty to be gained nowadays from freeing up a CPU thread even if the CPU has nothing else to do; the overwhelming majority of computers are laptops, tablets or phones, which all run on batteries and tend to be in contact with the user. Less power expended = less heat and longer use time between charges. It's also less likely that your OS' scheduler will schedule you out if you've not already been occupying time (as a rule of thumb, anyway). – Harwin

Can you provide a reference for the Statements "I have been told that for I/O bound applications, non blocking I/O would be better. For CPU bound applications, blocking I/O is much better."? Any web links? – Parmenides

blog.kgriffs.com/2012/09/18/demystifying-async-io.html .. There were two more links which mentioned about the statement. Can't locate them now. Also heard that in one of the technical conferences.. – Giles

Answer 1

From a programmer's perspective blocking I/O is easier to use than nonblocking I/O. You just call the read/write function and when it returns you are done. With nonblocking I/O you need to check if you can read/write, then read/write and then check the return values. If not everything was read or written you need mechanisms to read again or to write again now or later when write can be done.

Regarding performance: nonblocking I/O in one thread is not faster than blocking I/O in one thread. The speed of the I/O operation is determined by the device (for example the hard disc) that is read from or written to. The speed is not determined by someone waiting for (blocking on) or not waiting for (nonblocking on) it. Also if you call a blocking I/O function then the OS can do the blocking quite effectively. If you need to do the blocking/waiting in the application you might do that nearly as good as the OS, but you might also do it worse.

So why do programmers make their life harder and implement nonblocking I/O? Because, and that is the key point, their program has more to do than only that single I/O operation. When using blocking I/O you need to wait until the blocking I/O is done. When using nonblocking I/O you can do some calculations until the blocking I/O is done. Of course during nonblocking I/O you can also trigger other I/O (blocking or nonblocking).

Another approach to nonblocking I/O is to throw in more threads with blocking I/O, but as said in the SO post that you linked threads come with a cost. That cost is higher is than the cost for (OS supported) nonblocking I/O.

If you have an application with massive I/O but only low CPU usage like a web server with lots of clients in parallel, then use a few threads with nonblocking I/O. With blocking I/O you'll end up with a lot of threads -> high costs, so use only a few threads -> requires nonblocking I/O.

If you have an application that is CPU intensive like a program that reads a file, does intensive calculations on the complete data and writes the result to file, then 99% of the time will be spent in the CPU intensive part. So create a few threads (for example one per processor) and do as much calculation in parallel. Regarding the I/O you'll probably stick to one main thread with blocking I/O because it is easier to implement and because the main thread itself has nothing to do in parallel (given that the calculations are done in the other threads).

If you have an application that is CPU intensive and I/O intensive then you'ld also use a few threads and nonblocking I/O. You could think of a web server with lots of clients and web page requests where you are doing intensive calculations in a cgi script. While waiting for I/O on on connection the program could calculate the result for another connection. Or think of a program that reads a large file and could do intensive calculations on chunks of the file (like calculating an average value or adding 1 to all values). In that case you could use nonblocking reads and while waiting for the next read to finish you could already calculate on the data that is available. If the result file is only a small condensed value (like an average) you might use blocking write for the result. If the result file is as large as the input file and is like "all values +1", then you could write back the results nonblocking and while the write is being done you are free to do calculations on the next block.

Parmenides

Thanks Werner Henze for such a detailed explanation. :-) one quick question: "nonblocking I/O in one thread is not faster than blocking I/O in one thread." why is this so? Can you guide me to some web URL or something. I overall understands your concept and the nice explanation. My next obvious question to myself is the same thing I asked you. Cheers – Giles

@Giles I updated my answer and added: "The speed of the I/O operation is determined by the device (for example the hard disc) that is read from or written to. The speed is not determined by someone waiting for (blocking on) or not waiting for (nonblocking on) it." – Parmenides

@WernerHenze - So create a few threads (for example one per processor) and do as much calculation in parallel. Regarding the I/O you'll probably stick to blocking I/O because it is easier to implement and because the program has nothing to do in parallel. I didn't understand this. First part says execute them in parallel (using multithreaded/coroutine and/or multiprocessing). Last part says that the program has nothing to do in parallel. Thanks for the great explanation. Really helps grasping. – Icao

@technaz Thanks for the feedback. I rewrote the paragraph. I hope it is clearer now. – Parmenides