What's the process of disabling interrupt in multi-processor system?

Question

I have a textbook statement says disabling interrupt is not recommended in multi-processor system, and it will take too much time. But I don't understand this, can anyone show me the process of multi-processor system disabling the interrupts? Thanks

Ul

on x86, enabling/disabling interrupts is on a per-core basis. You can't globally disable interrupts on all cores (unless you send messages to the kernel running on other cores, and causing a handler to run cli on those other cores). Is this talking about disabling interrupts as a way of making something atomic for the purposes of multithreading? What architecture are you asking about? – Irregularity

@PeterCordes: It's about making something atomic. I don't know which architecture the book is talking about, it's just a general statement put in the section of deadlock. But I think my confusion is that for me the difference between 1 core and 8 core is not a big number for me (maybe this is not true, so the reason my asking), why disabling all of them from interrupt is time consuming. – Ul

Disabling interrupts on one core doesn't make something atomic if other threads are running on other cores. Disabling interrupts works on uniprocessor machines because it makes a context-switch impossible. (Or it makes it impossible for the same interrupt handler to interrupt itself.) – Irregularity

@PeterCordes What does "kernel running on other cores" mean? Could you elaborate a little bit? Does that mean each core run a copy of kernel code? Isn't operating system living in memory, and cores/CPUs simply run instructions ? – Homerus

@Rick: Memory is shared, it's not a copy of the kernel. When you send an IPI (InterProc Interrupt), the interrupt handler on the other core is running kernel code. My point with that is that an SMP kernel is effectively a distributed algorithm, not a "master control program". The kernel, running on each core, decides what that core should be doing, e.g. disabling interrupts, or context-switching to a different user-space task after running a function like schedule(). Other cores can trigger that if it has interrupts enabled, but "the kernel" running on that core has to run instructions. – Irregularity

Answer 1

on x86 (and other architectures, AFAIK), enabling/disabling interrupts is on a per-core basis. You can't globally disable interrupts on all cores.

Software can communicate between cores with inter-processor interrupts (IPIs) or atomic shared variables, but even so it would be massively expensive to arrange for all cores to sit in a spin-loop waiting for a notification from this core that they can re-enable interrupts. (Interrupts are disabled on other cores, so you can't send them an IPI to let them know when you're done your block of atomic operations.) You have to interrupt whatever all 7 other cores (e.g. on an 8-way SMP system) are doing, with many cycles of round-trip communication overhead.

It's basically ridiculous. It would be clearer to just say you can't globally disable interrupts across all cores, and that it wouldn't help anyway for anything other than interrupt handlers. It's theoretically possible, but it's not just "slow", it's impractical.

---

Disabling interrupts on one core doesn't make something atomic if other threads are running on other cores. Disabling interrupts works on uniprocessor machines because it makes a context-switch impossible. (Or it makes it impossible for the same interrupt handler to interrupt itself.)

But I think my confusion is that for me the difference between 1 core and 8 core is not a big number for me; why disabling all of them from interrupt is time consuming.

Anything other than uniprocessor is a fundamental qualitative difference, not quantitative. Even a dual-core system, like early multi-socket x86 and the first dual-core-in-one-socket x86 systems, completely changes your approach to atomicity. You need to actually take a lock or something instead of just disabling interrupts. (Early Linux, for example, had a "big kernel lock" that a lot of things depended on, before it had fine-grained locking for separate things that didn't conflict with each other.)

The fundamental difference is that on a UP system, only interrupts on the current CPU can cause things to happen asynchronously to what the current code is doing. (Or DMA from devices...)

On an SMP system, other cores can be doing their own thing simultaneously.

---

For multithreading, getting atomicity for a block of instructions by disabling interrupts on the current CPU is completely ineffective; threads could be running on other CPUs.

For atomicity of something in an interrupt handler, if this IRQ is set up to only ever interrupt this core, disabling interrupts on this core will work. Because there's no threat of interference from other cores.

Irregularity

"it would be massively expensive to ... that they can re-enable interrupts. " So the most expensive part is since interrupts is disabled, the re-enable process will be complicated? Sounds reasonable for me! – Ul

Thanks a lot btw, great answer, learn a lot from it. And it's very interesting! – Ul

@ptr_user7813604: sending IPIs to every other core is probably just as expensive, and waiting for them to acknowledge that they've disabled interrupts and won't be running your critical section (by loading a shared variable) is probably even more expensive. The fact that part of this communication has to happen by stores / loads to shared variables basically proves that this process is at least as much work as just taking a lock, but it always blocks all cores. If you'd just used a lock, usually most other cores wouldn't ever be disturbed. (Massive contention is very rare in a good design). – Irregularity

Thanks for the very good answer! Is there a way to use a single core (reserve it) and disable intterrupts on that core and use it as an "old 1980s CPU"? – Pursuant

@Zibri: I think maybe, under Linux. lwn.net/Articles/816298 describes "A full task-isolation mode for the kernel". There's also the isolcpus=3,4 or whatever: elinux.org/CPU_Shielding_capability. This might not be the same as not even booting Linux on that core at all; there might be a separate kernel option for that. Or of course you could boot the whole system as uniprocessor, never bringing up other cores, and disable interrupts in kernel mode for some time. (Make sure your code isn't buggy because you won't be able to interrupt it via keyboard.) – Irregularity

@PeterCordes yes.. that I know, I was wondering if there are aother ways (which could be used also in windows AND linux) like KVM or some other virtual extension to have a synchronous and reserved cpu. hmmm – Pursuant

@Zibri: No idea; that sounds like a different question than this one. If you get a hypervisor to give you a whole CPU, you can boot your own freestanding code on that bare machine and it can do whatever it wants. (Although that would be limited to disabling guest-virtual interrupts, otherwise you could maybe stop the hypervisor from regaining control.) If you're doing that, neither Windows nor Linux are involved, except as other VMs under the hypervisor. (Or as the host for KVM?) You might formulate it in a way that's on-topic for SuperUser.com if you're clear what you want. – Irregularity

Very helpful answer and thanks. I am reading a book and it says disabling interrupts to implement a spin lock doesn't work for multiprocessors but doesn't say how would that be for mult cores, single processor. I found some useful info here. – Homerus

@Rick: Multiple cores in a single package/socket is exactly equivalent to a system with multiple cores across multiple sockets, as far as this goes. "multi-core" and "multi-processor" are synonyms. Related: What is socket, core, threads, CPU? / Is having multiple cores in a CPU for running multiple threads/processes at once, or for instruction-level parallelism? – Irregularity