PCI Express BAR memory mapping basic understanding

Question

I am trying to understand how PCI Express works so i can write a windows driver that can read and write to a custom PCI Express device with no on-board memory.

I understand that the Base Address Registers (BAR) in the PCIE configuration space hold the memory address that the PCI Express should respond to / is allowed to write to. (Is that correct understood?)

My questions are the following:

• What is a "bus-specific address" compared to physical address when talking about PCIE?
• When and how is the BAR populated with addresses? Is the driver responsible for allocating memory and writing the address to the peripheral BAR?
• Is DMA used when transferring data from peripheral to host memory?

I appreciate your time.

Best regards,

Borstal

related: #30190550 – Brendanbrenden

Answer 1

From your query its clear that you want to write a driver for a PCIe slave device. To understand the scheme of things happening behind PCIe transfer, a lot of stuff is available on internet (Like PCIe Bus Enumeration, Peripheral Address Mapping to the memory etc).

Yes your understanding is correct regarding the mapping of PCIe registers to the memory and you can read/write them.( For e.g in case of linux PCIe device driver you can do this using "ioremap" ).

An address bus is used to specify a physical address. When a processor or DMA-enabled device needs to read or write to a memory location, it specifies that memory location on the address bus. Nothing more to add to that. "PCIe bus enumeration" topic will answer your 2nd question.

Your third question is vague. You mean slave PCIe device. Assuming it is, yes you can transfer data between a slave PCIe device and host using a DMA controller. I am working on a project which involves "PCIe-DMA" connected with the host over the PCIe bus. Really depends on your design and implementation. So in my case PCIe-DMA is itself a slave PCIe device on the target board connected to the host over PCIe.

Verdaverdant

Hello, thank you very much for your answer. So a "bus specific address" is the same as a physical address. E.g. they are the same seen from the host and the slave PCIe device. – Borstal

Most welcome. Yes. You just have to worry about the addresses of the registers specified in the PCIe slave device datasheet. So once you map those register addresses to your systems memory, you can access the PCIe device registers by accessing(read/write) the mapped system memory. – Verdaverdant

Hi @sumeet, Is this BAR window changes between one pci card to another ? I would assume it won't change becuase this is the window that the cpu use in order to get to the physical address of the pci card, right ? – Inaugurate

Answer 2

i'm also working on device driver (albeit on linux) with a custom board. Here is my attempt on answering your questions:

The BARs represent memory windows as seen by the host system (CPUs) to talk to the device. The device doesn't write into that window but merely answers TLPs (Transaction Layer Packets) requests (MRd*, MWr*).

I would say "bus-specific" = "physical" addresses if your architecture doesn't have a bus layer traslation mechanism. Check this thread for more info.

In all the x86 consumer PCs i've used so far, the BAR address seemed to be allocated either by the BIOS or at OS boot. The driver has to work with whatever address has been allocated.

The term DMA seems to abused instead of bus mastering which I believe is the correct term in PCIe. In PCIe every device may be a bus master (if allowed in its command register bit 2). It does so by sending MRd, MWr TLPs to other devices in the bus (but generally to the system memory) and signalling interrupts to the CPU.

Vice

Hi Claudio, Is this BAR window changes between one pci card to another ? I would assume it won't change becuase this is the window that the cpu use in order to get to the physical address of the pci card, right ? – Inaugurate

Hi @ransh, the BAR window size is defined by the PCI card. The location of this BAR is up to the software (BIOS or OS) to set-up. For e.g a PCI card could have BAR0 of size 1MB, another PCI card could have BAR0 of size 16kB. – Vice

Hi Cladio, Thank you. If I understood correctly, than "Base Address Registers" in PCI configuration space, ( the register which specify the physical address to device memory from cpu perspective) , would probably be the same when changing one PCI card with another, only the contents+size of this memory is different according to the pci device specific registers & memory map. Please correct me if wrong. Many thanks! – Inaugurate

Hi ransh, maybe there's still some confusion. The PCI configuration space (where the BAR registers are) is generally accessed through a special addressing which come in the form of bus/device/function or in linux (lspci) bus:slot.func (00:01.0). The PCIe protocol uses special packets for this kind addressing (Config Type 0/1 Read/Write Requests). These are not memory or IO space accesses (althought the mechanism to access them could be...thats another story). So if you replace a PCI card with another (in the same slot) they would probably use the same bus:slot address. Hope this helps. Claudio – Vice

Answer 3

clarification for your doubts/questions is here.

1> There are many devices thats sits on BUS like PCI which sees Memeory in terms that are different from a Physical address, those are called bus addresses. For example if you are initaiating DMA from a device sitting on bus to Main memory of system then destination address should be corresponding bus address of same physical address in Memmory

2> BARS gets populated at the time of enumeration, in a typical PC it is at boot time when your PCI aware frimware enumerate PCI devices presents on slot and allocate addresses and size to BARS.

3> yes you can use both DMA initiated or CPU initiated operations on these BARS.

-- flyinghigh

Calvo

who is reposnbile for a given BAR address ? Does it depends on the pci card ? or is it the cpu that choose address ( so that if I replace a card with another I will probably get the same address) ? – Inaugurate