PCIe 64-bit Non-Prefetchable Spaces

Question

I've been reading through the horror that is the PCIe spec, and still can't get any kind of resolution to the following question pair.

1. Does PCIe allow for mapping huge (say 16GB) 64-bit non-prefetchable memory spaces up above the 4GB boundary? Or are they still bound to the same 1GB that they were in the 32-bit days, and there's just no way to call for giant swaths of non-prefetchable space?

2. Assuming that the spec allows for it (and to my reading it does), do widely available BIOSes support it? Or is it allowed in theory but not done in practice?

Aroma

unless I'm missing something, I cannot see why not, at least spec-wise - in 64bit mode the device is able to address a 64bit space (well above 4GB). – Lore

Answer 1

TL;DR / Short Answer

No. BAR requests for non-prefetchable memory are limited to using the low 32-bit address space.

http://www.pcisig.com/reflector/msg03550.html

Long Answer

The reason why the answer is no has to do with PCI internals. The data structure which describes the memory ranges that a PCI bus encompasses only reserves enough space to store 32-bit base and limit addresses for non-prefetchable memory and for I/O memory ranges. However, it does reserve enough space to store a 64-bit base and limit for prefetchable memory.

Even Longer Answer

Specifically, look at http://wiki.osdev.org/PCI#PCI_Device_Structure, Figure 3 (PCI-to-PCI bridge). This shows a PCI Configuration Space Header Type 0x01 (the header format for a PCI-to-PCI bridge). Notice that starting at register 1C in that table, there are:

• 1C: 8 (middle) bits for I/O base address. Only top 4 bits are usable.
• 1D: 8 (middle) bits for I/O limit address. Only top 4 bits are usable.
• Ignore 1E-1F.
• 20: 16 bits for non-prefetchable memory base address. Only top 12 bits are usable.
• 22: 16 bits for non-prefetchable memory limit address. Only top 12 bits are usable.
• 24: 16 (middle) bits for prefetchable memory base address
• 26: 16 (middle) bits for prefetchable memory limit address
• 28: 32 upper bits for extended prefetchable memory base address
• 2C: 32 upper bits for extended prefetchable memory limit address
• 30: 16 upper bits for extended I/O base address
• 32: 16 upper bits for extended I/O limit address

The actual addresses are created by concatenating (parts of) these registers together with either 0s (for base addresses) or 1's (for limit addresses). The I/O and non-prefetchable base and limit addresses are 32-bits and formed thus:

Bit#                      31            20 19   16 15                       0

I/O Base:               [      16 upper bits      : 4 middle bits : 12 zeros ]
I/O Limit:              [      16 upper bits      : 4 middle bits : 12 ones  ]
Non-prefetchable Base:  [      12 bits     :              20 zeros           ]
Non-prefetchable Limit: [      12 bits     :              20 ones            ]

The prefetchable base and limit addresses are 64-bit and formed thus:

Prefetchable Base:
Bit#                      63                                                32

                        [                   32 upper bits                     ]
                        [  12 middle bits  :              20 zeros            ]

Bit#                      31                    16 15                        0


Prefetchable Limit:     
Bit#                      63                                               32

                        [                   32 upper bits                     ]
                        [  12 middle bits  :              20 ones             ]

Bit#                      31                    16 15                       0

As you can see, only the prefetchable memory base and limit registers are given enough bits to express a 64-bit address. All the other ones are limited to only 32.

Rouvin

could you then please explain this lspci output System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 (rev 02) Memory at fbf1c000 (64-bit, non-prefetchable) [size=16K] – Endblown

@Endblown That's the CPU's own root complex exposing a Header Type 0x00 with a 64-bit non-prefetchable BAR in the 32-bit address space (16K @ 0xfbf1c000, <0x100000000). Header Type 0x00 allows 64-bit non-prefetchable BARs at sub-20-bit alignment. However, you will never see non-prefetchable BARs behind bridges and above the 32-bit address space, and indeed this isn't an example of that. – Rouvin

ahha I see. Mm anyway I was digging around for a bit more, PCIe spec says PCIe Switches never buffer reads nor merge writes, so prefetchable doesn't really mean much anymore (legacy). But I guess it's still a nice hint to be able to say this is memory / these are registers. – Endblown

So - devices with large (>4G non-prefetchable) BARs should be okay if they are connected directly to root-ports?? – Unkennel

@IwillnotexistIdonotexist A lot of newer and newer chips (especially server SKUs) have more and more root ports - many of which are routed directly to physical slots without a bridge. (PCIe, NVMe, M.2, etc). – Unkennel

(....not root COMPLEX - just root PORT). – Unkennel

Answer 2

PCIe can define 64b memory addressing. The BARs (Base Address Registers) definition and usage is defined in the PCI 3.0 spec (chapter 6.2.5.1 "Address Maps") not in the PCIe spec.

Barren