All about PCI

[sunnysideup]

You can consider me a computer hardware newbie here: Compared to most members in this forum, I wasn't even born when PCI was (and still is) ruling the lands. I want to know, at an abstract level, what PCI/PCIe does.

I have read https://www.tldp.org/LDP/tlk/dd/pci.html, as well as the osdev wiki pages to try understand PCI/PCIe. However, often I find that most resources focus on the CPU view of PCI.

I want to 'see' the world through the eyes of the PCI bus, i.e. an introduction to the interfaces provided by PCI bus, so that I'd be well equipped to read the specification if I want extremely specific knowledge.

Here's my understanding so far:

1. A PCI/PCIe bus can be thought of as an entity to which devices can be connected. (I don't think it is necessary that a CPU must be connected, and can just be thought of as just another device?)
2. Each PCI bus has a special device that controls the PCI bus?
3. Every device connected to the PCI bus has private memory. If we think of the CPU also as a device, this would be RAM?
4. There is something called configuration space too, but I'm not sure if this is per-device (that is connected to the PCI bus) or per-bus.

I'm sure I have a lot of flaws in my understanding. What I'd like to know from this discussion is a mental picture of PCI, from which I can build the specifics by reading the spec/any other wiki.

[bzt]

You can consider me a computer hardware newbie here: Compared to most members in this forum, I wasn't even born when PCI was (and still is) ruling the lands. I want to know, at an abstract level, what PCI/PCIe does.

No worries, everybody had to start somewhere

I have read https://www.tldp.org/LDP/tlk/dd/pci.html, as well as the osdev wiki pages to try understand PCI/PCIe. However, often I find that most resources focus on the CPU view of PCI.

I want to 'see' the world through the eyes of the PCI bus, i.e. an introduction to the interfaces provided by PCI bus, so that I'd be well equipped to read the specification if I want extremely specific knowledge.

Here's my understanding so far:

Not entirely sure what you are asking.

1. A PCI/PCIe bus can be thought of as an entity to which devices can be connected. (I don't think it is necessary that a CPU must be connected, and can just be thought of as just another device?)

No, not really. CPU is the CPU, and peripherals are peripherals. PCI is a standardized way to query peripheral configuration and to route IRQs. That's it (from your perspective, of course it is also a hardware spec that specifies voltage levels, pin layouts etc. for expansion cards).

2. Each PCI bus has a special device that controls the PCI bus?

Yep.

3. Every device connected to the PCI bus has private memory. If we think of the CPU also as a device, this would be RAM?

Most definitely not. RAM is free to use, while PCI memory is linked to the device and each address in it has a special meaning.

4. There is something called configuration space too, but I'm not sure if this is per-device (that is connected to the PCI bus) or per-bus.

Well, both? Each device has a configuration space, that is, a simple structure with data like numeric identifiers, IO port addresses, MMIO ranges etc. They describe the interface to be used to access that device. Since each PCI bus is a device itself, so yes, they have their own structures too. There are three kinds of structures: 0x00 PCI device, 0x01 PCI-to-PCI bridge (those are PCI bus devices) and 0x02 which are not really commonly use, except some hardware PCI-to-Cardbus devices.

I'm sure I have a lot of flaws in my understanding. What I'd like to know from this discussion is a mental picture of PCI, from which I can build the specifics by reading the spec/any other wiki.

I suggest to write a simple iteration, that loops through the PCI address space (bus, device, function), and print out what you can find. Then try it out, and see what you got. It will help understanding greatly.

In a nutshell, you decide how you can access the PCI address space. For old PCI, this means using IN/OUT instructions. For PCIe, you'll have a memory mapped structure (so just use a special address, exactly like with the VGA buffer). Then query the list (see the wiki page's section "Enumerating PCI Buses" has some code examples too.). You can do that by having three loops, one in another; or you could do a reentrant walk where you recursively call the enumeration function each time you find a 0x01 PCI bus device, etc.

Cheers,
bzt

[sunnysideup]

Not entirely sure what you are asking.

Well, I thought that PCI was something CPU-agnostic, and a "PCI bus with a CPU" was a special case (Although the most common). This is the reason that I wanted to 'explore' what PCI is, on an abstract level. (I'm not sure I'm making sense here either). I wanted to know what the mental picture that one gets when dealing with PCI. You can ignore it because I clearly had the wrong mental picture.

PCI memory is linked to the device and each address in it has a special meaning

What does this mean exactly? What does PCI memory mean? I'm guessing that it is a range of physical memory addresses which is technically MMIO. Is this right? And which is the device that is referred to here?

In a nutshell, you decide how you can access the PCI address space. For old PCI, this means using IN/OUT instructions. For PCIe, you'll have a memory-mapped structure (so just use a special address, exactly like with the VGA buffer).

So, I can choose how I access this configuration space? Meaning I can flexibly use any physical address that I want to access this space (That makes it MMIO??)? That sounds a bit weird, like I'm telling the hardware, "From now on, reads and writes to physical address 0x73420000 will access PCI configuration space." Is this how it works? I think I'm definitely wrong here.

Also something that comes up often is the concept of an address space. What does this mean? What does it mean when two devices have different address spaces? And I really hope that all the addresses we speak about here are physical addresses because the story with virtual memory ends with the processor I guess.

I apologize for sounding too dumb , just curious XD

[Octocontrabass]

1. A PCI/PCIe bus can be thought of as an entity to which devices can be connected. (I don't think it is necessary that a CPU must be connected, and can just be thought of as just another device?)

A CPU must be connected. PCI is a bus for connecting peripherals to a CPU.

2. Each PCI bus has a special device that controls the PCI bus?

Yes. It translates the CPU's requests into PCI bus cycles, routes IRQs, and arbitrates DMA so that PCI devices can access main memory without the CPU's intervention.

3. Every device connected to the PCI bus has private memory. If we think of the CPU also as a device, this would be RAM?

Most PCI devices do have memory to use as some sort of buffer, but it's not a requirement. The CPU is not a PCI device.

4. There is something called configuration space too, but I'm not sure if this is per-device (that is connected to the PCI bus) or per-bus.

It's both.

Well, I thought that PCI was something CPU-agnostic, and a "PCI bus with a CPU" was a special case (Although the most common).

It's architecture-agnostic, but you still need a CPU.

So, I can choose how I access this configuration space? Meaning I can flexibly use any physical address that I want to access this space (That makes it MMIO??)? That sounds a bit weird, like I'm telling the hardware, "From now on, reads and writes to physical address 0x73420000 will access PCI configuration space." Is this how it works? I think I'm definitely wrong here.

No, you can't do that with the configuration space. On modern PCs, you have the choice between the extended configuration space (which is MMIO at a fixed address reported in an ACPI table) or the legacy configuration space (which uses I/O ports at a standard address).

Also something that comes up often is the concept of an address space. What does this mean? What does it mean when two devices have different address spaces? And I really hope that all the addresses we speak about here are physical addresses because the story with virtual memory ends with the processor I guess.

An address space is just a range of addresses differentiated from other addresses in some way.

To use the configuration space example above, each PCI device has a set of configuration registers mapped to particular addresses. This range of addresses (or "space") is then mapped somewhere into a bigger range of addresses ("space") used only for that particular bus, which is then mapped into an even bigger space used only for PCI configuration registers. Finally, that big PCI configuration space is mapped somewhere in the CPU's memory or I/O space so that the CPU can configure PCI devices.

[bzt]

You can ignore it because I clearly had the wrong mental picture.

I'm not sure the phrase "mental picture" is what you were looking for either I guess you meant overall picture maybe?

What does this mean exactly? What does PCI memory mean? I'm guessing that it is a range of physical memory addresses which is technically MMIO. Is this right? And which is the device that is referred to here?

It is explained on the wiki in great detail. For legacy PCI, that's 256 bytes, for PCIe 4096 bytes. This could be mapped in the memory address space, but not necessarily. For example, you use IN/OUT for PCI. You write the required offset (from 0 to 255, otherwise called address) to the port 0xCF8 with an OUT instruction, then you can read in the data from port 0xCFC with an IN instruction. When it is MMIO, then you simply use a MOV instruction with a certain address to read the PCI memory, which is the base address of the configuration space plus the configuration space's offset.

So, I can choose how I access this configuration space? Meaning I can flexibly use any physical address

Definitely not. For PCI, you don't use physical memory address at all (you use IN/OUT), and for PCIe you must use the address you find in the MCFG ACPI table as base address. As I said, just like the VGA buffer. There it is fixed at 0xB8000, you can't move that address. When you write something to that address, you actually don't write to RAM, instead you communicate with the VGA card (that's how MMIO works). When you add offsets to that, for example 0xB8004, then you access different values in the VGA address space.

That sounds a bit weird, like I'm telling the hardware, "From now on, reads and writes to physical address 0x73420000 will access PCI configuration space." Is this how it works? I think I'm definitely wrong here.

Yep, that's wrong. You don't tell the hardware where to access PCI MMIO, you ask the hardware where to find the PCI MMIO (by parsing ACPI tables). And as I said before, if you can't find any MMIO in the ACPI tables, you have to use IN/OUT ports. In theory you might run on a computer that does not support IO ports either, but that's very unlikely, means pre-Pentium II. It is a safe assumption that without MMIO, you can use IO ports to access the PCI data.

Also something that comes up often is the concept of an address space. What does this mean?

Very simply put, it is an array of "slots" where each slot has a unique address to get/set the value stored there. I wrote "slot", because that could be really anything. For example, address space could be CMOS address space (or NVRAM), where you can read the current date/time from. It could mean physical memory, in which case addresses are pointing to bytes in RAM. It could also be an IO address (like 0xCF8 is an offset to the IO port address space) etc.

What does it mean when two devices have different address spaces?

Means that you can access different values at PCI configuration space offset 2C for example. You set it up for a specific device, then you can access that device's configuration space (256 bytes) at the 0xCF8/0xCF0 ports (or MMIO). Then you configure it for another device, and after that you can access a different configuration space, but the same address encodes the same kind of value. For example, on the wiki page you can see that reading PCI config space 2C will return the Subsystem Vendor ID. You change the bus and device, and then you can read another device's Subsystem Vendor ID at offset 2C.

And I really hope that all the addresses we speak about here are physical addresses because the story with virtual memory ends with the processor I guess.

Only if we're talking about MMIO. You can't access IO ports as memory (virtual or not). As far as virtual memory concerned, you can map MMIO just as RAM. For example, let's say that you parsed ACPI tables and you have found there's an MMIO at 0x1000 (just an example). Then that's in physical address space. You can map it at 0xFFFF000 for example, and then you can access that MMIO at 0xFFFF000 just as you would access it at 0x1000 without virtual memory. And yes, the chip responsible for the map translation, called MMU is part of the processor.

A "processor", or more accurately CPU (Central Processing Unit) consist of many parts: ALU (bit operations and logic), ARU (calculating with integer numbers), FPU (compute with floating point numbers), and MMU, Memory Management Unit which is responsible for memory addressing, etc.

I apologize for sounding too dumb , just curious XD

You should read a bit more on the topic, that's for sure. Understanding address spaces and how they differ (IO, NVRAM, physical RAM, virtual linear etc.) is a must in OS development.

Cheers,
bzt

[sunnysideup]

Thanks for the reply. It was very helpful!