How can the linux kernel be forced to enumerate the PCI-e bus?

Question

Linux kernel 2.6

I've got an fpga that is loaded over GPIO connected to a development board running linux. The fpga will transmit and receive data over the pci-express bus. However, this is enumerated at boot and as such, no link is discovered (because the fpga is not loaded at boot).

How can I force re-enumeration of the pci-e bus in linux? Is there a simple command or will I have to make kernel changes? I need the capability to hotplug pcie devices.

Rousseau

Shouldn't the device driver take care of that? – Norvol

No, it seems the pci-e driver only enumerates automatically at boot, and if after boot a device is plugged in, nothing happens. lspci shows no devices and no device file exists – Rousseau

I think the answer is "no", although I am quite willing to be wrong. Altera seems to have a workaround method, google for "Altera FPGA Configuration via Protocol". If I understand this paper correctly, Altera has a flash available in (some?) Stratix V devices that is always available for PCIe bus enumeration while the bulk of the FPGA may be (re)programmed at leisure. – Aylesbury

Minimal QEMU example to play around with this: https://mcmap.net/q/282217/-why-is-the-probe-method-needed-in-linux-device-drivers-in-addition-to-init/… – Colchicine

Answer 1

I wonder what platform you are on: A work around (aka hack) for this that works on x86 systems is to have the BIOS basically statically configure a PCI device at whatever bus, device, function the FPGA normally lands on, then the OS will enumerate the device and reserve the PCI space for it (even though the device isn't really there). Then in your device driver you will have to do some extra things like setup the BARs and int lines manually after the fpga has been programmed. Of course this requires modifying the BIOS, which if you are working with a BIOS vendor you can contract them to make this change for you, if you are not working with a BIOS vendor then it will be much harder... Also keep in mind that I was working on VxWorks on x86, and we had a AMI make a custom BIOS for our boards...

If you don't have a BIOS, then consider programming it in the bootloader, there you already have the ability to read from disk, and adding GPIO capabilities probably isn't too difficult (assuming you are using jtag and GPIOs?), in fact depending on what bootloader you use it might already be able to do GPIO?

The issues with modifying the kernel to do this is that you have to find the sweet spot where you can read the bitfile, before the PCI enumeration... If for example the disk device drivers are initialized after PCI, then obviously you must do some radical changes to the kernel just to read the bitfile prior to PCI enumeration, which might cause other annoying problems...

One other option which you may have already discovered, and which is really only ok for development time: Power up the system, program the fpga board, then do a reset (without power cycle, for example: sudo reboot now), the FPGA should keep its configuration, and linux should enumerate it...

Rothrock

Reboot cycling is what we currently are doing. It's an Marvell ARM platform acting as a controller board to an Altera Arria II fpga. – Rousseau

I was also thinking it could be possible to modify the kernel such that it loads the fpga before it enumerates. I am not sure if this is practical/possible. – Rousseau

One more thing regarding the bootloader mods: if your bootloader is not field upgradable, then make sure you consider the consequences of adding complex functionality to it before you release it... Or consider methods to make it field up-gradable. – Rothrock

Answer 2

As root, try the following command:

echo "1" > /sys/bus/pci/rescan

See this link for more information: http://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci

Adductor

Yes I saw that in my google adventures trying to solve this problem. It does not work, even though I have CONFIG_HOTPLUG enabled in the linux build. – Rousseau

This does seem like an easy solution, but I can imagine that it might cause your PCI devices/drivers to freak out if this causes the PCI bus to go offline for a second or so... – Rothrock

If you need sudo, you will get "permission denied". In that case, use sudo sh -c "echo 1 > /sys/bus/pci/rescan". – Limbourg

eaiser: echo 1 | sudo tee /sys/bus/pci/rescan – Clouet

This doesn't work for me. It causes a kernel oops and attempting to use the pci bus utilities (such as lspci) hang the system. – Fortunna

This seems to be vary somewhat between kernels. It fails on a custom embedded system running 3.12.8, but works okay on an ubuntu desktop running 3.11.0. – Fortunna

Before rescanning, remove the linux-knowledge of card by writing 1 to /sys/bus/pci/devices/.../remove. – Imperceptive

We use this method with Xilinx and Altera FPGAs and kernel version 3.8 or newer, including step 2 above: echo 1 > /sys/bus/pci/devices/$device/remove – Comply

Answer 3

After turning on your computer, the BIOS enumerates the PCI bus and attempts to fulfill all IO space and memory mapped IO (MMIO) requests. It sets up these BAR's initially, and when the operating system loads these BAR's can be changed by the OS as it sees fit while the PCI bus driver enumerates the bus yet again. It is even possible for the superuser of the system to run the command setpci to change these BAR's after the BIOS has already attempted to configure them and the OS has loaded (may cause drivers to fail and several other bad things if done improperly).

I have had to do this in cases where the card in question was not assigned any resources by the BIOS since the region requested required a 64-bit address and the BIOS only operated with 32-bit address assignments. I was able to go in after-the-fact and change these addresses (originally assigned by the BIOS) to whatever addresses I saw fit, insert the kernel module, and my driver would map and use these newly-assigned addresses for the card without knowing the difference.

The problem that exists with hotplugging PCI-Express cards is that the power to the slot, itself, cannot be turned on/off without specific hotplug controllers that need to exist on the motherboard/backplane. Not having these hotplug controllers to turn the slot's power off may lead to shorts between the tiny pins when the card is physically inserted and/or removed if power is still present. Hotplug events, however, can be initiated by either end (the host or the endpoint device). This does not seem to be the case, however if your FPGA already has a link established with the root complex, a possible solution to your problem would be to generate hotplug interrupts to cause a bus rescan in the OS.

There is a major problem, though -- if your card does not actually obtain a link to the root complex, it won't be able to generate any hotplug events; which seems to be the case. After booting, the FPGA should toggle the PRESENT line on the PCIe bus to tell the OS there is a card ready to be enumerated. Once detected, the OS should attempt to establish a link to the card and assign memory regions to the device. After the OS enumerates the card you'll be able to load drivers against it and see it in lspci. You stated you're using kernel 2.6, which does have support for hotplugging and dynamic resource allocation so this method should work as long as your FPGA supports the ability to toggle the PRESENT PCIe line, too.

Theophylline