After going through some basics documents what I understood is, Base Address Register is Address space which can be accessed by PCIe IP. PCIe IP can either transmit data in Base Address Register or it can write received data on to it.
Am I right? Or missing anything?
Summary
Each BAR is a small 32-bit memory location that points to another (usually much larger) memory region which I'll call the corresponding "BAR region". Each BAR tells the CPU the base address + width + other properties of its BAR region. The CPU can then read and write to that BAR region to talk to the PCIe device.
When you read or write to offsets within the BAR region, TLP packets are sent back and forth between the CPU/memory and the PCIe device, which tells the PCIe device to do something or send something back.
Such reads and writes are the main way in which drivers interact with PCIe devices.
What reads and writes to specific addresses of a BAR region mean is defined by each specific PCIe device and completely device dependent, but typically:
configure how the device should operate
tell the device to start doing some work, e.g. write to disk, render a frame on the GPU, or send a packet over the network.
A very common pattern in which such operations happen is:
Here's how the configuration spaces and BAR regions might look like on physical memory of a hypothetical device, with some of the BARs pointing to corresponding memory regions:
+--------------+
| Func 64:00.0 |
| Conf. Space |
+--------------+
| BAR 0 |>--------------+
+--------------+ |
| BAR 1 |>----------+ |
+--------------+ | |
| BAR 2 | | |
+--------------+ | |
| BAR 3 | | |
+--------------+ | |
| BAR 4 | | |
+--------------+ | |
| BAR 5 | | |
+--------------+ | |
| |
+--------------+ | |
| Func 64:00.1 | | |
| Conf. Space | | |
+--------------+ | |
| BAR 0 |>------+ | |
+--------------+ | | |
| BAR 1 | | | |
+--------------+ | | |
| BAR 2 |>--+ | | |
+--------------+ | | | |
| BAR 3 | | | | |
+--------------+ | | | |
| BAR 4 | | | | |
+--------------+ | | | |
| BAR 5 | | | | |
+--------------+ | | | |
| | | |
| | | |
| | | |
+--------------+<--|---|---+ |
| Region of | | | |
| 64:00.0 BAR0 | | | |
| (1 MiB) | | | |
+--------------+ | | |
| | |
+--------------+<--|---+ |
| Region of | | |
| 64:00.1 BAR2 | | |
| (2 MiB) | | |
+--------------+ | |
| |
+--------------+<--|-----------+
| Region of | |
| 64:00.0 BAR1 | |
| (512 KiB) | |
+--------------+ |
|
+--------------+<--+
| Region of |
| 64:00.1 BAR0 |
| (512 KiB) |
+--------------+
Where BARs are located
Let's locate ourselves globally within the PCIe environment.
For each PCIe device, this is how things are logically organized hierarchically:
+--------+ +------------+ +------+
| device |>---------------->| function 0 |>----->| BAR0 |
| | | | +------+
| |>------------+ | |
| | | | | +------+
... ... | | |>----->| BAR1 |
| | | | | +------+
| |>--------+ | | |
+--------+ | | ... ... ...
| | | |
| | | | +------+
| | | |>----->| BAR5 |
| | +------------+ +------+
| |
| |
| | +------------+ +------+
| +--->| function 1 |>----->| BAR0 |
| | | +------+
| | |
| | | +------+
| | |>----->| BAR1 |
| | | +------+
| | |
| ... ... ...
| | |
| | | +------+
| | |>----->| BAR5 |
| +------------+ +------+
|
|
| ...
|
|
| +------------+ +------+
+------->| function 7 |>----->| BAR0 |
| | +------+
| |
| | +------+
| |>----->| BAR1 |
| | +------+
| |
... ... ...
| |
| | +------+
| |>----->| BAR5 |
+------------+ +------+
Each PCIe device can define from 1 to 8 "functions". Each function can be identified by the 2-byte "bus-device-function" (BDF) triplet:
These three values are often represented in the format
<bus>:<device>.<function>
e.g. doing lspci on my Lenovo ThinkPad P14s laptop gives among other lines:
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e8
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Device 14e9
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14ea
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14ea
00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14ee
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14ee
00:02.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14ee
64:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 (rev dd)
64:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller
64:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 19h (Model 74h) CCP/PSP 3.0 Device
64:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b9
64:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15ba
64:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor (rev 63)
65:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 14ec
65:00.1 Signal processing controller: Advanced Micro Devices, Inc. [AMD] Device 1502
so e.g. here we see that
00 contains several devices such as:
00:00, which contains two functions:
00:00.000:00.200:01 which has just one mandatory function 00:01.000:02 which has four functions:
00:02.000:02.100:02.200:02.4and so on for bus 64.
Each function has one "PCIe configuration space", which a standardized small chunk of memory used to get information or do certain operations common to all devices.
For PCI Type 0 (Non-Bridge) devices, the configuration space looks like this:
0 15 16 31
+------------------+-----------------+------------------+------------------+
| Vendor ID | Device ID | 00
+------------------+-----------------+------------------+------------------+
| Command | Status | 04
+------------------+-----------------+------------------+------------------+
| Revision ID | Class Code | 08
+------------------+-----------------+------------------+------------------+
| Cache Line Size | Latency Timer | Header Type | BIST | 0C
+------------------+-----------------+------------------+------------------+
| BAR0 | 10
+------------------+-----------------+------------------+------------------+
| BAR1 | 14
+------------------+-----------------+------------------+------------------+
| BAR2 | 18
+------------------+-----------------+------------------+------------------+
| BAR3 | 1C
+------------------+-----------------+------------------+------------------+
| BAR4 | 20
+------------------+-----------------+------------------+------------------+
| BAR5 | 24
+------------------+-----------------+------------------+------------------+
| Cardbus CIS Pointer | 28
+------------------+-----------------+------------------+------------------+
| Subsystem Vendor ID | Subsystem ID | 2C
+------------------+-----------------+------------------+------------------+
| Expansion ROM Base Address | 30
+------------------+-----------------+------------------+------------------+
| Cap. Pointer | | 34
+------------------+-----------------+------------------+------------------+
| | 38
+------------------+-----------------+------------------+------------------+
| Interrupt Line | Interrupt Pin | Min Gnt. | Max Lat. | 3C
+------------------+-----------------+------------------+------------------+
Adapted from: https://en.wikipedia.org/wiki/File:Pci-config-space.svg
Therefore we see that each such "PCIe function" has 6 32-bit BAR registers, BAR0 to BAR5, allowing it to register up to 6 BAR regions for itself. Each BAR can also be marked as inactive TODO how, all zeroes? Or width 0?
Therefore, each device can have up to 48 BAR registers: 8 functions times 6 BARs per function.
To give some context, other notable things present in the configuration space besides the BARs include:
Note that like the BAR registers, these operations are common to all PCIe devices. This is in opposition to writing to the BAR regions, which is highly device specific.
Layout of each BAR register
The wiki page gives the layout of each bar.
For "memory"-type BARs:
| Bits | Description | Values |
|---|---|---|
| 0 | Region Type | 0 = memory 1 = I/O (deprecated) |
| 2-1 | Locatable | 0 = any 32-bit 1 = < 1 MiB 2 = any 64-bit |
| 3 | Prefetchable | 0 = no 1 = yes |
| 31-4 | Base Address | 16-byte aligned |
and for "I/O" type BARs, there's just one big 31-bit Base Address field after the "Region Type" bit.
The most important field in the "memory"-type bar is the 27-bit "Base Address" field, which determines the initial address in memory of the corresponding BAR region, in multiples of 16-bytes.
From this we see that the maximum address of the start of a range is when all bits are '1' so:
(2^28 - 1) * 2^4 = 4 GiB - 16
Wikipedia however mentions that there is an option to have base addresses above 4 GiB:
If a platform supports the "Above 4G" option in system firmware, 64 bit BARs can be used.
Finding the region width
On the layout table, there is no mention of the region width, only the base address.
This is because determining the region width requires you to first write 1's to the BAR.
This modifies its value, so that when you next read it the result gives you the width: How is a PCI / PCIe BAR size determined?
This gives you a certain number or leading 1's, which maps to a power of two size between 16 bytes and 2 GiB.
Determine BAR regions on Linux
Running:
lspci -vv
clearly gives it to us on the "Region" entries of each PCI efunction. E.g. on my Lenovo ThinkPad P14s and slightly manually edited for brevity I see:
64:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 (rev dd) (prog-if 00 [VGA controller])
Region 0: Memory at 2400000000 (64-bit, prefetchable) [size=256M]
Region 2: Memory at 78000000 (64-bit, prefetchable) [size=2M]
Region 4: I/O ports at 1000 [size=256]
Region 5: Memory at 78500000 (32-bit, non-prefetchable) [size=512K]
64:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller
Region 0: Memory at 785c8000 (32-bit, non-prefetchable) [size=16K]
64:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 19h (Model 74h) CCP/PSP 3.0 Device
Region 2: Memory at 78400000 (32-bit, non-prefetchable) [size=1M]
Region 5: Memory at 785cc000 (32-bit, non-prefetchable) [size=8K]
so we understand that:
Play with it with QEMU and the Linux kernel
A good way to learn something is to interact with it, and the perfect setup for that is with the Linux kernel on QEMU with the QEMU "edu device"!
QEMU does not model PCIe at the TLP level; QEMU PCIe TLP Emulation data just magically flies from memory to devices and back. But all the software-level details are clearly visible so that's OK.
The QEMU "edu" device is an educational PCIe device, and therefore perfect for our purposes. It is clearly documented at: https://www.qemu.org/docs/master/specs/edu.html
And if anything is unclear, the source is at: https://github.com/qemu/qemu/blob/760b4dcdddba4a40b9fa0eb78fdfc7eda7cb83d0/hw/misc/edu.c
That device registers a single 1 MiB BAR region
memory_region_init_io(&edu->mmio, OBJECT(edu), &edu_mmio_ops, edu,
"edu-mmio", 1 * MiB);
pci_register_bar(pdev, 0, PCI_BASE_ADDRESS_SPACE_MEMORY, &edu->mmio);
and then the docs https://www.qemu.org/docs/master/specs/edu.html explain what reads and writes to the BAR region do https://www.qemu.org/docs/master/specs/edu.html#mmio-area-spec e.g.:
To enable the edu device it you have to launch QEMU with:
-device edu
Linux kernel PCIe interaction
The Linux kernel has extensive PCIe interaction facilities of course, given that it is used to interface with so many super important different PCIe hardware.
Here is a minimal kernel drive example I've created for edu: https://github.com/cirosantilli/linux-kernel-module-cheat/blob/366b1c1af269f56d6a7e6464f2862ba2bc368062/kernel_module/pci.c
Also that hole repo of mine automates building the root image for you with Buildroot, and cross compiling the kernel modules esaily: https://github.com/cirosantilli/linux-kernel-module-cheat
Several Linux kernel PCI functions take the BAR as a parameter to identify which communication channel is to be used, e.g.:
mmio = pci_iomap(pdev, BAR, pci_resource_len(pdev, BAR));
pci_resource_flags(dev, BAR);
pci_resource_start(pdev, BAR);
pci_resource_end(pdev, BAR);
Depending on the BAR "Region Type" (0/1), we should use different functions to read and write to the BAR region:
IORESOURCE_IO: must be accessed with inX and outXIORESOURCE_MEM: must be accessed with ioreadX and iowriteXBibliography:
I think this is a very basic question and I would suggest to read:
A Base Address Register (BAR) is used to:
- specify how much memory a device wants to be mapped into main memory, and
- after device enumeration, it holds the (base) address, where the mapped memory block begins.
A device can have up to six 32-bit BARs or combine two BARs to a 64-bit BAR.
BAR is record of the device address starting at memory.
root@Ubuntu:~$ lspci -s 00:04.0 -x
00:04.0 USB controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M) USB2 EHCI Controller (rev 10)
00: 86 80 cd 24 06 00 00 00 10 20 03 0c 10 00 00 00
10: 00 10 02 f3 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 f4 1a 00 11
30: 00 00 00 00 00 00 00 00 00 00 00 00 05 04 00 00
root@Ubuntu:~$ lspci -s 00:04.0 -v
00:04.0 USB controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M) USB2 EHCI Controller (rev 10) (prog-if 20 [EHCI])
Subsystem: Red Hat, Inc QEMU Virtual Machine
Physical Slot: 4
Flags: bus master, fast devsel, latency 0, IRQ 35
Memory at f3021000 (32-bit, non-prefetchable) [size=4K]
Kernel driver in use: ehci-pci
root@Ubuntu:~$ grep 00:04.0 /proc/iomem
f3021000-f3021fff : 0000:00:04.0
0xfff equals to 4095, which is 4K. Memory starts at 0xf3021000 is this USB device seen by CPU. This address is init during BIOS and in this example it is on BAR0. Why is BAR0?
Before that, one need to understand PCI spec, especially the below, type 0 and type 1:
Notice the header type is both defined at 0x0c third field, that is how BARs differ. In this example, it is 00, which means it is type 0. Thus BAR0 stores the address, which is 00 10 02 f3.
One may wonder why this is not exactly f3021000, this is because lspci goes with Little Endian. What is Endian? One may need to read "Gulliver's Travels".
BAR0 generally has three states, uninitialized, all 1s, and written address. And we now in the third since the device already init. Bit 11 ~ 4 is set to 0 at uninitialized state; Bit 3 means NP when set to 0, P when set to 1; Bit 2 ~ 1 means 32 bit when set to 00, 64 bit when set to 10; Bit 0 means memory request when set to 0, IO request when set to 1.
0xf3021000
====>>>>
11110011000000100001000000000000
From this, we can know this device is 32-bit, non-prefetchable, memory request. The uninitialized address is 32 ~ 12, since 2 ^ 12 = 4K.
For more device and vendor, one can find via https://pcilookup.com/
Roughly speaking, the root-complex (aka the host computer) acts as the "dealer" and talks to each end-point device in a process called enumeration, where each device has its own set of configuration registers. It does this access using configuration space, rather than normal memory space. memory space for the pci device doesn't exist until the bar registers are setup and mapped by the root complex. Using configuration space, the root-complex sequentially writes all 1's the bar register, in each PCI device, and read them back to determine the size of the bar address space assigned to each device. If the root complex sees zeros in the lower order bits above bit 4, this means that these are addressable space, then it picks a physical memory address and assigned it to the non-zero bits in the bar register...
For PCIe device with 32-bit bars the configuration space has the following 32-bit DWORDS:
UInt32 PCIEBAR32_0, PCIEBAR32_1, PCIEBAR32_2,
PCIEBAR32_3, PCIEBAR32_4, PCIEBAR32_5;
bool cond32_0 = (PCIeBAR32_0 & 0x7) == 0x00);
bool cond32_1 = (PCIeBAR32_1 & 0x7) == 0x00);
bool cond32_2 = (PCIeBAR32_2 & 0x7) == 0x00);
bool cond32_3 = (PCIeBAR32_3 & 0x7) == 0x00);
bool cond32_4 = (PCIeBAR32_4 & 0x7) == 0x00);
bool cond32_5 = (PCIeBAR32_5 & 0x7) == 0x00);
For PCIe device with 64-bit bars, the two adjacent 32-bit DWORDS are concatenated to form a 64-bit bar:
UInt64 PCIEBAR64_0, PCIEBAR64_1, PCIEBAR64_2;
bool cond64_0 = (PCIEBAR32_0 & 0x7) == 0x4);
bool cond64_1 = (PCIEBAR32_2 & 0x7) == 0x4);
bool cond64_2 = (PCIEBAR32_4 & 0x7) == 0x4);
if (!(cond64_0 && cond64_1 && cond64_2)) {
Console.Writeline("Whoops, we don't have 3 adjacent 64-bit bars");
return -1;
}
PCIEBAR64_0 = (UInt64)PCIEBAR32_1<<32 | (UInt64)PCIEBAR32_0;
PCIEBAR64_1 = (UInt64)PCIEBAR32_3<<32 | (UInt64)PCIEBAR32_2;
PCIEBAR64_2 = (UInt64)PCIEBAR32_5<<32 | (UInt64)PCIEBAR32_4;
//note that since lower 4-bits of Least significant
//bar indicate its a 64-bit bar, this means the
//next adjacent 32-bit bar doesn't knockout
//the bottom 4-bits of the bar. so that it can be concatenated.
Not really sure what happens for a system with a mix of 32-bit, and 64-bit bars... maybe you need to check the bars in order from 0 to 5 to find non-aligned cases...
© 2022 - 2024 — McMap. All rights reserved.