Is Multi Message MSI implemented on Linux / x86?

Question

I am working on a network driver for an FPGA endpoint that supports multi-message MSI interrupts (not msix) on a PCIe bus. The host processor is an x86 Intel i7 620LM running on CentOS with a 4.2 kernel.

The FPGA endpoint correctly advertises multiple msi vectors in it's MSI capabilities register (0x101 = 32 total possible vectors).

From what I can tell, multi-message functionality was added in kernel 4.2. Unfortunately when I call pci_enable_msi_range(pdev, 1, 32); it only returns 1. When I call pci_msi_vec_count(pdev); it returns 32. I'm able to request an irq handler on the one vector and it works as expected.

Does anyone know if multi-message MSI vectors are actually supported in Linux on x86 architectures?

UPDATE: I was able to get all 32 MSI vectors enabled using a different SBC with an i7-4700EQ processor. This with a 4.4-rc1 kernel.

UPDATE: Works with 4.2 as well.

UPDATE: In this case, the problem was in coreboot. I was able to get multiple vectors working once the board vender provided an update.

Telescopy

Of course it is. I'm not sure why exactly you get the error. but I've written few drivers that used more than 1 vector. – Beaudoin

What platform(s) was this on. Have you found any processors, APIC, etc combos that are unable to remap multiple vectors? – Telescopy

I am very late to this .. but when I have a multi MSI endpoint detected in virtual box environment with ubuntu guest OS(x86 64 bit) running on it, I still get only 1 MSI even though I request for multiple MSIs !! I could see the PCI-MSI domain wasnt created properly. Any suggestions on how to enable this feature/how to use other controller(IR-PCI-MSI) to make use of MULTI MSI support ? @Telescopy any ideas on it ? – Tanka

Answer 1

Of course it is. I'm not sure why exactly you get the error. but I've written few drivers that used more than 1 vector.

I just checked my PC at home, I have various drivers which use more the 1 MSI IRQ:

$ cat /proc/interrupts | grep -i msi
 33:    5683962          0          0          0          0          0          0          0   PCI-MSI 524288-edge      radeon
 34:          0          0          0          0          0          0          0          0   PCI-MSI 1572864-edge      xhci_hcd
 35:          0          0          0          0          0          0          0          0   PCI-MSI 1572865-edge      xhci_hcd
 36:          0          0          0          0          0          0          0          0   PCI-MSI 1572866-edge      xhci_hcd
 37:          0          0          0          0          0          0          0          0   PCI-MSI 1572867-edge      xhci_hcd
 38:          0          0          0          0          0          0          0          0   PCI-MSI 1572868-edge      xhci_hcd
 39:          0          0          0          0          0          0          0          0   PCI-MSI 1572869-edge      xhci_hcd
 40:          0          0          0          0          0          0          0          0   PCI-MSI 1572870-edge      xhci_hcd
 41:          0          0          0          0          0          0          0          0   PCI-MSI 1572871-edge      xhci_hcd
 42:    3807594          0          0          0          0          0          0          0   PCI-MSI 512000-edge      0000:00:1f.2
 43:          0          0          0          0          0          0          0          0   PCI-MSI 2097152-edge      xhci_hcd
 44:          0          0          0          0          0          0          0          0   PCI-MSI 2097153-edge      xhci_hcd
 45:          0          0          0          0          0          0          0          0   PCI-MSI 2097154-edge      xhci_hcd
 46:          0          0          0          0          0          0          0          0   PCI-MSI 2097155-edge      xhci_hcd
 47:          0          0          0          0          0          0          0          0   PCI-MSI 2097156-edge      xhci_hcd
 48:          0          0          0          0          0          0          0          0   PCI-MSI 2097157-edge      xhci_hcd
 49:          0          0          0          0          0          0          0          0   PCI-MSI 2097158-edge      xhci_hcd
 50:          0          0          0          0          0          0          0          0   PCI-MSI 2097159-edge      xhci_hcd
 51:     310762          0          0          0          0          0          0          0   PCI-MSI 5242880-edge      0000:0a:00.0
 52:         11          0          0          0          0          0          0          0   PCI-MSI 360448-edge      mei_me
 54:   38991293          0          0          0          0          0          0          0   PCI-MSI 7340032-edge      enp14s0
 55:          1          0          0          0          0          0          0          0   PCI-MSI 32768-edge      i915
 56:       1169          0          0          0          0          0          0          0   PCI-MSI 442368-edge      snd_hda_intel
 57:        152          0          0          0          0          0          0          0   PCI-MSI 526336-edge      snd_hda_intel

$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 42
Model name:            Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Stepping:              7
CPU MHz:               1634.257
CPU max MHz:           3800.0000
CPU min MHz:           1600.0000
BogoMIPS:              6825.67
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-7
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid xsaveopt

Beaudoin

Are you sure they aren't using msi-x? I thought that is what xhci-hcd uses. can you do a sudo lspci -v | grep MSI:? – Telescopy

there's no difference between MSI and MSIX from APIC prospective. – Beaudoin

At some point, supposedly, Muilti-Message MSI support was added to the kernel for x86 architectures. It looks like it was added in 4.2. Before that, only multiple MSI-X vectors were available. Clearly there is some difference be it software or hardware that limited its implementation. I'm unable to get multiple MSI vectors on 4.2 and 4.4 kernels, I have run into others having the same problem. I haven't found a driver in the 4.4 kernel tree that implements more than one MSI vector. Did the drivers you wrote use MSI or MSI-X vectors? – Telescopy

See my update. I'll revert back when and if I figure out what the difference was in host hardware. – Telescopy

@Beaudoin .. could you post /proc/iomem output over here !! Apologies for the most delayed response.. ! – Tanka

Answer 2

I had problems using more than one MSI on an Intel machine independently of the Linux kernel version as well. After a lot of fiddling I realized that enabling VT-d in the bios fixed the issue (tested on Linux 4.6 kernel). I think this is because in order to support multiple MSIs on x86, the Linux kernel needs support for interrupt remapping which is one of the features provided by VT-d.

Leighannleighland

As it turned out the board vender had to provide a new version of coreboot. This solved the problem. – Telescopy

Answer 3

To add to what Adrien mentioned, turning on CONFIG_IRQ_REMAP solved my issue of being able to request multiple MSI's for the same device. pci_enable_msi_range is successful with my requested msi interrupts ( 8 in this case ).

Canzone