What is an uninterruptible process?

Question

Sometimes whenever I write a program in Linux and it crashes due to a bug of some sort, it will become an uninterruptible process and continue running forever until I restart my computer (even if I log out). My questions are:

• What causes a process to become uninterruptible?
• How do I stop that from happening?
• This is probably a dumb question, but is there any way to interrupt it without restarting my computer?

Elect

Is it possible that a program can be written to initiate a process that goes into a TASK_UNINTERUPTIBLE state whenever the the system isn't in an idle state, thereby forcibly collecting data, waiting to transmit once the super user exits? This would be a goldmine for hackers to retrieve information, return to zombie state, and transmit information through the network on idle. Some can argue that this is one way to create a Blackdoor for the powers that be, to enter and exit any system as desired. I strongly believe this loophole can be sealed for good, by eliminating the `TASK_UNINTERUPTIB – Anikaanil

would be please share the code? – Hale

Answer 1

An uninterruptible process is a process which happens to be in a system call (kernel function) that cannot be interrupted by a signal.

To understand what that means, you need to understand the concept of an interruptible system call. The classic example is read(). This is a system call that can take a long time (seconds) since it can potentially involve spinning up a hard drive, or moving heads. During most of this time, the process will be sleeping, blocking on the hardware.

While the process is sleeping in the system call, it can receive a Unix asynchronous signal (say, SIGTERM), then the following happens:

• The system call exits prematurely, and is set up to return -EINTR to user space.
• The signal handler is executed.
• If the process is still running, it gets the return value from the system call, and it can make the same call again.

Returning early from the system call enables the user space code to immediately alter its behavior in response to the signal. For example, terminating cleanly in reaction to SIGINT or SIGTERM.

On the other hand, some system calls are not allowed to be interrupted in this way. If the system calls stalls for some reason, the process can indefinitely remains in this unkillable state.

LWN ran a nice article that touched this topic in July.

To answer the original question:

• How to prevent this from happening: figure out which driver is causing you trouble, and either stop using, or become a kernel hacker and fix it.

• How to kill an uninterruptible process without rebooting: somehow make the system call terminate. Frequently the most effective manner to do this without hitting the power switch is to pull the power cord. You can also become a kernel hacker and make the driver use TASK_KILLABLE, as explained in the LWN article.

Anabasis

I pulled the power cord on my laptop and it is not working, sadly. ;-) – Glia

Isn't it EINTR instead of EAGAIN? Also read() returns -1 and errno is set to the error. – Limekiln

It seems EINTR is indeed the correct error code. I had memories of i/o calls returning EAGAIN without correct handling in Python library code a few years back, but I was apparently wrong. Thanks for that, I fixed the answer. – Anabasis

@Anabasis You can also become a kernel hacker and make the driver use TASK_KILLABLE This needs to be build in right into a kernel code. Perhaps change the kernel code in memory that is executing the system call to just return? Kill both the user-mode and kernel-mode part of the process? I bet there has to be a way to safely kill even those "uninterruptible" processes. Just make whatever it's waiting on IGNORE the result and bam, it's no longer uninterruptible! I really thought those kind of processes were Windows-only problem. – Ceremonious

@Anabasis somehow make the system call terminate ...or the OS kernel itself can, after finishing the system call, check if its calling process is still alive, and do nothing if it isn't? That way uninterruptible processes could be safely killed and their memory freed, and whatever it's been waiting on afterwards would ignore it. Am I missing something or is it actually that easy? – Ceremonious

@Dexter: You are indeed missing the point. Read the LWN article: lwn.net/Articles/288056. Those problems are caused by lazy device driver programmers, and they need to be fixed in the device driver code. – Anabasis

@Anabasis I've just read that LWN article and nothing in the article or comments made me think the uninterruptible process problem cannot really be solved just in the OS itself. An OS should be made in a way that makes ALL drivers always SAFELY interruptible. Any OS kernel not made that way is bad, period. – Ceremonious

@Anabasis "Unix tradition (and thus almost all applications) believe file store writes to be non signal interruptible. It would not be safe or practical to change that guarantee." -> This is exactly the most wrong part of all this IMO. Just interrupt the read/write request of the driver, and when the actual device (hard disk / network card / etc) delivers the data, ignore it. An OS kernel should be made in a way that NO developer can screw it up. – Ceremonious

@Ceremonious you sound like a microkernel fanboy. That is not what Linux is. Your troll is irrelevant. – Anabasis

@Anabasis I do know Linux is not a microkernel, though I am not sure what part of my comment relates to it... And then, does your comment mean that a microkernel OS does not have a problem with those "uninterruptible" processes? Because if it does not, maybe it's time for me to become a microkernel fan... :D – Ceremonious

Alternatively you can use a bazooka to get rid of the straggling process and free the poor computer out of existence. – Siberia

I see the LWN article, and it said "Unix tradition (and thus almost all applications) believe file store writes to be non signal interruptible. It would not be safe or practical to change that guarantee." , why it is not be safe or practical for interrupting file store writing? – Lydon

Answer 2

When a process is on user mode, it can be interrupted at any time (switching to kernel mode). When the kernel returns to user mode, it checks if there are any signals pending (including the ones which are used to kill the process, such as SIGTERM and SIGKILL). This means a process can be killed only on return to user mode.

The reason a process cannot be killed in kernel mode is that it could potentially corrupt the kernel structures used by all the other processes in the same machine (the same way killing a thread can potentially corrupt data structures used by other threads in the same process).

When the kernel needs to do something which could take a long time (waiting on a pipe written by another process or waiting for the hardware to do something, for instance), it sleeps by marking itself as sleeping and calling the scheduler to switch to another process (if there is no non-sleeping process, it switches to a "dummy" process which tells the cpu to slow down a bit and sits in a loop — the idle loop).

If a signal is sent to a sleeping process, it has to be woken up before it will return to user space and thus process the pending signal. Here we have the difference between the two main types of sleep:

• TASK_INTERRUPTIBLE, the interruptible sleep. If a task is marked with this flag, it is sleeping, but can be woken by signals. This means the code which marked the task as sleeping is expecting a possible signal, and after it wakes up will check for it and return from the system call. After the signal is handled, the system call can potentially be automatically restarted (and I won't go into details on how that works).
• TASK_UNINTERRUPTIBLE, the uninterruptible sleep. If a task is marked with this flag, it is not expecting to be woken up by anything other than whatever it is waiting for, either because it cannot easily be restarted, or because programs are expecting the system call to be atomic. This can also be used for sleeps known to be very short.

TASK_KILLABLE (mentioned in the LWN article linked to by ddaa's answer) is a new variant.

This answers your first question. As to your second question: you can't avoid uninterruptible sleeps, they are a normal thing (it happens, for instance, every time a process reads/writes from/to the disk); however, they should last only a fraction of a second. If they last much longer, it usually means a hardware problem (or a device driver problem, which looks the same to the kernel), where the device driver is waiting for the hardware to do something which will never happen. It can also mean you are using NFS and the NFS server is down (it is waiting for the server to recover; you can also use the "intr" option to avoid the problem).

Finally, the reason you cannot recover is the same reason the kernel waits until return to user mode to deliver a signal or kill the process: it would potentially corrupt the kernel's data structures (code waiting on an interruptible sleep can receive an error which tells it to return to user space, where the process can be killed; code waiting on an uninterruptible sleep is not expecting any error).

Edouard

Filesystem locking bug is also a likely cause, IME. – Bubo

I don't understand all of this. "you can't avoid uninterruptible sleeps" - can't the OS be made in such a way that uninterruptible sleep simply does not EXIST as a state? Then the part about corruption - can't the kernel-mode part of the process itself (or whatever COULD cause the corruption) be terminated or just its code modified right in memory to just return? Please explain why is this so hard / impossible to do that even Linux has not done it. (I thought this problem exists only on Windows) – Ceremonious

The only case I can think of that would make (safely) killing those processes really impossible (and not just, let's say, exceptionally hard) is if the hardware itself could cause the corruption. Hardware can't be controlled; kernel can. But it's the kernel that gets data from hardware and modifies the memory (that's why it must not be freed before the process returns to user mode and why could the corruption occur)... change the kernel code in memory and no more problems. – Ceremonious

@Ceremonious think of the kernel as if it were a single multi-threaded process, where the kernel-mode part of each process is a thread within the kernel. Your suggestion would be as bad as killing a single thread in a multi-threaded program: it could leave dangling locks, data structures temporarily modified or in the middle of being modified, and so on. – Edouard

@Edouard well you are right about killing a thread... But can't the "main" thread (that would be the OS kernel and other threads would be drivers for example) somehow handle it? Though those structures "in the middle of being modified" seem to be one really tough issue... maybe we will really never see an OS where uninterruptible processes would be impossible :( – Ceremonious

Answer 3

Uninterruptable processes are USUALLY waiting for I/O following a page fault.

Consider this:

• The thread tries to access a page which is not in core (either an executable which is demand-loaded, a page of anonymous memory which has been swapped out, or a mmap()'d file which is demand loaded, which are much the same thing)
• The kernel is now (trying to) load it in
• The process can't continue until the page is available.

The process/task cannot be interrupted in this state, because it can't handle any signals; if it did, another page fault would happen and it would be back where it was.

When I say "process", I really mean "task", which under Linux (2.6) roughly translates to "thread" which may or may not have an individual "thread group" entry in /proc

In some cases, it may be waiting for a long time. A typical example of this would be where the executable or mmap'd file is on a network filesystem where the server has failed. If the I/O eventually succeeds, the task will continue. If it eventually fails, the task will generally get a SIGBUS or something.

Godroon

If it eventually fails, the task will generally get a SIGBUS or something. Wait, can't the kernel be made so that, when killing those "uninterruptible" processes, it simply TELLS them the I/O operation failed? Then the process would go back to user mode and be gone? There HAS to be a way to safely kill those 'D' state processes. I guess it's just not easy and that's why neither Windows or Linux have that possibility yet. On the other side, I would want to be able to kill those processes at least unsafely. I don't care for possible system crash or whatever... – Ceremonious

@Ceremonious hmm, I've never experienced this problem with Windows. What is a way to reproduce it there? At least according to this post, all I/O requests can be interrupted in Windows. – Barnabe

see the "Notmyfault" example made by Sysinternals. Its driver named "myfault" has routines that cause a crash (blue screen) or hang the process in uninterruptible state. – Ceremonious

Answer 4

To your 3rd question: I think you can kill the uninterruptable processes by running sudo kill -HUP 1. It will restart init without ending the running processes and after running it, my uninterruptable processes were gone.

Blackness

Answer 5

If you are talking about a "zombie" process (which is designated as "zombie" in ps output), then this is a harmless record in the process list waiting for someone to collect its return code and it could be safely ignored.

Could you please describe what and "uninterruptable process" is for you? Does it survives the "kill -9 " and happily chugs along? If that is the case, then it's stuck on some syscall, which is stuck in some driver, and you are stuck with this process till reboot (and sometimes it's better to reboot soon) or unloading of relevant driver (which is unlikely to happen). You could try to use "strace" to find out where your process is stuck and avoid it in the future.

Wyman

Can't drivers be forcefully unloaded the same way a process could be killed? I know kernel mode has more privileged access than user mode, but it can never be more privileged that the operating system itself. Anything executing in kernel mode can always tamper with anything else executing in kernel mode - there is simply no control. – Ceremonious