Why is MS-DOS “not reentrant” and a DOS interrupt can't call another one?

Question

This question is purely academic because no-one uses MS-DOS now, but I still want to know why.

In some books and articles they said if you call a DOS interrupt during another one, it may cause a deadlock. That is why MS-DOS is not reentrant. For example, RESIDENT PROGRAMS, and another book, as described below:

A  interrupt occurs
B  interrupt handling
C DOS command starts
D new interrupt occurs
E interrupt handling
F DOS COMMAND starts
G DOS command finished
H interrupt finished
I return to the original interrupt handling
J return to original DOS command

It says, when I is finished, going to J, which is trying to return to the point the first DOS command was interrupted, but as all DOS variables and stack are changed by F and G, when you try to go back to the original interrupt (B), you actually go back to the second interrupt (E), and that causes the deadlock.

But as far as I'm concerned, an interrupt is just like a call. Save the current CS:IP, check the vector, find the interrupt handler, execute, return to where the interrupt occurred. Exactly like call. It doesn't make sense how the deadlock is even possible.

So my question is what exactly may cause a deadlock? A specific example would be very appreciated.

Standstill

"because no one use MS-DOS now" I think you'll be surprised. My guess is there are still machines in manufacturing systems alive and well running on some version or variant of DOS. – Epperson

@MarkRotteveel DOS is very predictable in its operation, will probably never be updated again and has hard real time properties. This is very desirable for many embedded applications. – Nevins

Answer 1

There are two main reasons:

The DOS kernel is not designed for reentrancy

When code is supposed to be reentrant, it has to be designed with this in mind and you cannot use certain design patterns such as static buffers for temporary data. Avoiding these design patterns was not a priority for the authors of DOS, so the code is not generally reentrant.

With other functions, it's really hard to impossible to implement them in a reentrant way. For example, take a function that outputs a character to the screen. This is done by first advancing the cursor and then drawing the character into the frame buffer. Suppose an interrupt occurs after the cursor has been advanced but before the character is drawn. Then, the following happens:

1. outer call: cursor is advanced from position 1 to position 2
2. interrupt!
3. inner call: cursor is advanced from position 2 to position 3
4. inner call: character is drawn at position 3
5. interrupt ends
6. outer call: character is drawn at position 3

So instead of two characters, only one is drawn and there's a blank in between.

While this sort of problem can in some cases be avoided by turning off interrupts, it's actually not good design to do this all the time as that raises interrupt latency. Also, for more complicated subsystems like the file system, it may require a significantly different design to do so.

Another issue with reentrancy (as opposed to multi-threading safety) is that you cannot really use critical sections. When you are an interrupt handler and try to enter a critical section but cannot, you will not be able to wait for the critical section to become free as the code holding it will not continue execution until your interrupt handler finishes. So it's kind of a conundrum and it's really difficult to correctly deal with this sort of situation.

The DOS kernel has its own stack

DOS applications tend to have really small stacks. Meanwhile, DOS has grown over the years and may require significant amounts of stack space to execute its functions. To solve this problem, the DOS designers have added DOS-internal stacks starting with DOS 2. Whenever a DOS interrupt is called, the interrupt handler first switches to the DOS stack and then executes the functionality the user called for. If this is tried while already inside DOS, the stack switch would corrupt the call stack of the outer DOS call.

Luckily DOS prevents you from doing this: there's an “in DOS” flag that keeps track of whether a DOS call is running right now. If a call is running, the stack switch is aborted and your DOS call fails.

Incidentally this is a huge problem when writing pop-up TSRs and books on this topic devote long chapters on what DOS calls you can and cannot do when your TSR is called and how to work around these problems.

Nevins

You're supposed to check the InDOS flag yourself if you want to call DOS from a TSR's interrupt handler. – Oshea

@Oshea This is the TLDR answer, but there's a lot more to this. People did all sorts of crazy things including making copies of the internal DOS stacks and later restoring them. – Nevins