Multi-core CPU interrupts

Question

How does multi-core processors handle interrupts?

I know of how single core processors handle interrupts. I also know of the different types of interrupts. I want to know how multi core processors handle hardware, program, CPU time sequence and input/output interrupt

Weywadt

How do interrupts work in multi-core system? – Muscadine

Answer 1

This should be considered as a continuation for or an expansion of the other answer.

Most multiprocessors support programmable interrupt controllers such as Intel's APIC. These are complicated chips that consist of a number components, some of which could be part of the chipset. At boot-time, all I/O interrupts are delivered to core 0 (the bootstrap processor). Then, in an APIC system, the OS can specify for each interrupt which core(s) should handle that interrupt. If more than one core is specified, it means that it's up to the APIC system to decide which of the cores should handle an incoming interrupt request. This is called interrupt affinity. Many scheduling algorithms have been proposed for both the OS and the hardware. One obvious technique is to load-balance the system by scheduling the interrupts in a round-robin fashion. Another is this technique from Intel that attempts to balance performance and power.

On a Linux system, you can open /proc/interrupts to see how many interrupts of each type were handled by each core. The contents of that file may look something like this on a system with 8 logical cores:

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
  0:         19          0          0          0          0          0          0          0  IR-IO-APIC   2-edge      timer
  1:          1          1          0          0          0          0          0          0  IR-IO-APIC   1-edge      i8042
  8:          0          0          1          0          0          0          0          0  IR-IO-APIC   8-edge      rtc0
  9:          0          0          0          0          1          0          0          2  IR-IO-APIC   9-fasteoi   acpi
 12:          3          0          0          0          0          0          1          0  IR-IO-APIC  12-edge      i8042
 16:         84    4187879          7          3          3   14044994          6          5  IR-IO-APIC  16-fasteoi   ehci_hcd:usb1
 19:          1          0          0          0          6          8          7          0  IR-IO-APIC  19-fasteoi 
 23:         50          2          0          3     273272          8          1          4  IR-IO-APIC  23-fasteoi   ehci_hcd:usb2
 24:          0          0          0          0          0          0          0          0  DMAR-MSI   0-edge      dmar0
 25:          0          0          0          0          0          0          0          0  DMAR-MSI   1-edge      dmar1
 26:          0          0          0          0          0          0          0          0  IR-PCI-MSI 327680-edge      xhci_hcd
 27:      11656        381        178   47851679       1170        481        593        104  IR-PCI-MSI 512000-edge      0000:00:1f.2
 28:          5   59208205          0          1          3          3          0          1  IR-PCI-MSI 409600-edge      eth0
 29:        274          8         29          4         15         18         40   64478962  IR-PCI-MSI 32768-edge      i915
 30:         19          0          0          0          2          2          0          0  IR-PCI-MSI 360448-edge      mei_me
 31:         96         18         23         11        386         18         40         27  IR-PCI-MSI 442368-edge      snd_hda_intel
 32:          8         88         17        275        208        301         43         76  IR-PCI-MSI 49152-edge      snd_hda_intel
NMI:          4         17         30         17          4          5         17         24   Non-maskable interrupts
LOC:  357688026  372212163  431750501  360923729  188688672  203021824  257050174  203510941   Local timer interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
PMI:          4         17         30         17          4          5         17         24   Performance monitoring interrupts
IWI:          2          0          0          0          0          0          0        140   IRQ work interrupts
RTR:          0          0          0          0          0          0          0          0   APIC ICR read retries
RES:   15122413   11566598   15149982   12360156    8538232   12428238    9265882    8192655   Rescheduling interrupts
CAL: 4086842476 4028729722 3961591824 3996615267 4065446828 4033019445 3994553904 4040202886   Function call interrupts
TLB: 2649827127 3201645276 3725606250 3581094963 3028395194 2952606298 3092015503 3024230859   TLB shootdowns
TRM:     169827     169827     169827     169827     169827     169827     169827     169827   Thermal event interrupts
THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
DFR:          0          0          0          0          0          0          0          0   Deferred Error APIC interrupts
MCE:          0          0          0          0          0          0          0          0   Machine check exceptions
MCP:       7194       7194       7194       7194       7194       7194       7194       7194   Machine check polls
ERR:          0
MIS:          0
PIN:          0          0          0          0          0          0          0          0   Posted-interrupt notification event
PIW:          0          0          0          0          0          0          0          0   Posted-interrupt wakeup event

The first column specifies the interrupt request (IRQ) number. All the IRQ numbers that are in use can be found in the list. The file /proc/irq/N/smp_affinity contains a single value that specifies the affinity of IRQ N. This value should be interpreted depending on the current mode of operation of the APIC.

A logical core can receive multiple I/O and IPI interrupts. At that point, local interrupt scheduling takes place, which is also configurable by assigning priorities to interrupts.

Other programmable interrupt controllers are similar.

Kolosick

Answer 2

In general, this depends on the particular system you have under test.
The broader approach is to have a specific chip in each processor¹ that is assigned, either statically or dinamically², a unique ID and that can send and receive interrupts over a shared or dedicated bus.
The IDs allows specific processors to be targets of interrupts.

Code running on the processor A can ask its interrupt chip to raise an interrupt on processor B, when this happens a message is sent along the above-mentioned bus, routed to processor B where the relative interrupt chip picks it up, decode it and raise the corresponding interrupt.

At the system level, one or more, general interrupt controllers are present to route interrupt requests from the IO devices (in any bus) to the processors.
These controllers are programmable, the OS can balance the interrupt load across all the processors (or implement any other convenient policy).

This is the most flexible approach, a wired approach is also possible.
In this case, processor A signals are wired directly to processor B inputs and vice versa; asserting these signals give rise to an interrupt on the target processor.

The general concept is called Inter-processor Interrupt (IPI).
The x86 architecture follows the first approach closely³ (beware of the nomenclature though, processor has a different meaning).
Other architectures may not, like the IBM OS/360 M65MP that uses a wired approach⁴.

Software generated interrupts are just instructions in a program, each processor executes their own instruction stream and thus if program X generate an exception when running on processor A, it is processor A that handles it.

Task scheduling is distributed across all the processors usually (that's what Linux does.

Time-keeping is usually done by a designated processor that serves a hardware timer interrupt.
This is not always the case, I haven't looked at the precise details of the implementations of the modern OSes.

---

¹ Usually an integrated chip, so we can think of it as a functional unit of the processor.
² By a power-on protocol.
³ Actually, this is reverse causality.
⁴ I'm following the Wikipedia examples.

Chicanery