PCI specification dictate root PCI bus address range?

[Hallam]

Hello All,

I sent this as a PM to a few users who suggested it would be a good question to post here.

I wondered if I could just get your views on something that's niggling me.

It's this ancient HP article: http://tinyurl.com/ocp6awt about why 32 bit systems can't use 4GB of memory and towards the end it says - " The PCI 2.2 specification (pages 202-204) dictates that root PCI bus must be allocated one block of MMIO addresses. This block of addresses is subdivided into the regions needed for each device on that PCI bus. And each of those device MMIO regions must be aligned on addresses that are multiples of the size of the region."

I have looked in the PCI 2.2 specs on pages 202 - 204 and can't find anything that says the root PCI bus must be allocated one block of addresses.

I know the PCI MMIO region is from TOLUD to 4GB but I didn't think the PCI specs enforced it to be in one block, is this just an error?

Pages 202 - 204 in the specs are just about memory and IO BARs? The only thing that I can relate on 202 - 204 is that regions are aligned on a natural boundary.

Any help would be appreciated as it's got me a bit confused.

Kind Regards,
Hallam

[Combuster]

While I know of no particular reason to have the root PCI bus contiguous, it does hold for all child PCI buses since the PCI-to-PCI bridge interface only allows you to specify just one range.

In reality, the PCI root bus is a fake. If you interpret it logically on many machines, you see a host-to-pci bridge, and then later on a pci-to-pci express device. Of course there's no way your 16X graphics card goes over a 33MHz bus link somewhere in the middle. The whole construction merely exists to make sure hardware detection sees all devices.

Beyond that, when the chipset finally routes addresses to what actually is the PCI bus - it might often just impose that address range restriction like any other pci-to-pci bridge. Similarly, any logic that configures the series of bridges would be trivial if it only has one range to assign from because the PCI bus address assignment algorithm would simply become an euler tour.


That said, I don't have the necessary quotes from the PCI standard at hand to look for any particular readings.

[Hallam]

Thanks for the reply Combuster, much appreciated.

While I know of no particular reason to have the root PCI bus contiguous, it does hold for all child PCI buses since the PCI-to-PCI bridge interface only allows you to specify just one range.

As you say one block of addresses will hold true for PCI - PCI bridges but as the host bridge is Chipset specific I wouldn't have that the PCI Specs dictated that the root PCI bus would have to be in one contiguous block, and it's not mentioned on Pages 202 - 204 of the PCI 2.2 spec so that's what was throwing me off.

In reality, the PCI root bus is a fake. If you interpret it logically on many machines, you see a host-to-pci bridge, and then later on a pci-to-pci express device. Of course there's no way your 16X graphics card goes over a 33MHz bus link somewhere in the middle. The whole construction merely exists to make sure hardware detection sees all devices.

Thanks, yes Brendan has explained this to me (thanks again Brendan!) that the root PCI but is just logical bus in the config apace, and not representative of a physical 'PCI Bus 0' in the system.

Beyond that, when the chipset finally routes addresses to what actually is the PCI bus - it might often just impose that address range restriction like any other pci-to-pci bridge. Similarly, any logic that configures the series of bridges would be trivial if it only has one range to assign from because the PCI bus address assignment algorithm would simply become an euler tour.

Not quite sure what you mean by this? Any physical PCI bus will be behind a PCI - PCI bridge won't it so the one block of addresses will have to be applied for that bus, is that correct?

----------------

So overall, I assume that the article is wrong.

Would in theory, a PCI host controller , be able to forward ranges 0xC0000000 - 0xCFFFFFFF and then 0xE0000000 - 0xFFFFFFFF onto the root PCI bus and then PCI devices gets addresses within those ranges? (Not that this would ever be done, just in theory)

Thanks again,
Hallam

[Hallam]

Hello, sorry to bump the thread. I was just wondering if you could clarify something?

Beyond that, when the chipset finally routes addresses to what actually is the PCI bus - it might often just impose that address range restriction like any other pci-to-pci bridge. Similarly, any logic that configures the series of bridges would be trivial if it only has one range to assign from because the PCI bus address assignment algorithm would simply become an euler tour.

Not quite sure what you mean by this? Any physical PCI bus will be behind a PCI - PCI bridge won't it so the one block of addresses will have to be applied for that bus, is that correct?

So overall, I assume that the article in my original post is wrong.

Would in theory, a PCI host controller , be able to forward ranges 0xC0000000 - 0xCFFFFFFF and then 0xE0000000 - 0xFFFFFFFF onto the root PCI bus and then PCI devices gets addresses within those ranges? (Not that this would ever be done, just in theory)

Any help would be appreciated,
Thanks.

[Combuster]

Any physical PCI bus will be behind a PCI - PCI bridge

I have yet to see bus 0 behind a pci-to-pci bridge. Rather you'll typically find a host-to-pci bridge at device 0, or whatever else identifies itself as the chipset's northbridge. The rest of the device numbers on bus 0 can be divided in a few chunks: PCI express bridges, the southbridge and its constituent logical components, and the device numbers that actually correspond to the physical PCI bus. Hence, there's already a lot of routing taking place to get chunks of the address space away from the physical PCI bus.

Actually, you won't find a true PCI-to-PCI bridge in the entire space on most systems, dropping the requirement for having it consecutive in that range.

The thing is, x86 hardware has traditionally had a memory hole between 3 and 4GB for 32-bit devices and operating systems. I haven't amassed enough video hardware in one machine to blow up that space and actually see a 64-bit address in use. At any rate, I wouldn't be surprised if a number of existing chipsets functionally use two separate ranges for the PCI device space already. There are chipsets already that abuse the PCI bus routing to post write requests in the 768K-1M on the PCI bus rather than memory as the implementation of making the BIOS "ROM" read-only.

[Hallam]

Thanks again Combuster

The thing is, x86 hardware has traditionally had a memory hole between 3 and 4GB for 32-bit devices and operating systems. I haven't amassed enough video hardware in one machine to blow up that space and actually see a 64-bit address in use. At any rate, I wouldn't be surprised if a number of existing chipsets functionally use two separate ranges for the PCI device space already. There are chipsets already that abuse the PCI bus routing to post write requests in the 768K-1M on the PCI bus rather than memory as the implementation of making the BIOS "ROM" read-only.

Thanks, yes I remember reading something about adding external GPU's and then having to edit the ACPI DSDT table so that it mapped it above 4GB as there wasn't enough space between TOLUD and 4GB, thereby creating 2 PCI windows on the root bus, like the image I found below shows -



I've done some further reading, mainly the EFI PCI Host Bridge Resource Allocation Protocol Spec v0.9, and it repeatedly says "The registers inside a PCI host bridge that control configuration of PCI root buses are not governed by the PCI specification and vary from chipset to chipset." So the HP article has to be wrong, not only does the PCI spec not dictate the root PCI bus range, but also having a memory windows above and below 4GB means it doesn't have to be in a contiguous block anyway.

Regarding the range from 768K-1M, I assume you mean the PAM registers. I'm not too sure how these exactly work, when one is set to forward reads/writes to the DMI, do they go straight to DMI or do they first have to go through a PCI root bridge and then onto the DMI. I've read a few Intel datasheets and can't seem to find a clear answer.

Thanks as always,
Hallam

[Candy]

The thing is, x86 hardware has traditionally had a memory hole between 3 and 4GB for 32-bit devices and operating systems. I haven't amassed enough video hardware in one machine to blow up that space and actually see a 64-bit address in use. At any rate, I wouldn't be surprised if a number of existing chipsets functionally use two separate ranges for the PCI device space already. There are chipsets already that abuse the PCI bus routing to post write requests in the 768K-1M on the PCI bus rather than memory as the implementation of making the BIOS "ROM" read-only.

I should check on that for you - I have a 2GB video card in my main computer, who only has a memory hole between 3G and 4G (and then memory up to 17G). Not sure it's mapped though, but the BARs may be telling me more.

[Hallam]

The thing is, x86 hardware has traditionally had a memory hole between 3 and 4GB for 32-bit devices and operating systems. I haven't amassed enough video hardware in one machine to blow up that space and actually see a 64-bit address in use. At any rate, I wouldn't be surprised if a number of existing chipsets functionally use two separate ranges for the PCI device space already. There are chipsets already that abuse the PCI bus routing to post write requests in the 768K-1M on the PCI bus rather than memory as the implementation of making the BIOS "ROM" read-only.

I should check on that for you - I have a 2GB video card in my main computer, who only has a memory hole between 3G and 4G (and then memory up to 17G). Not sure it's mapped though, but the BARs may be telling me more.

Hi Candy,

I don't think the full 2GB will be mapped into the address space, it usually only maps 256MB or 512MB to avoid this problem of exhausting the PCI hole.

Kind Regards,
Hallam