PCI enumeration - do you cache the device list?

[gusc]

I've implemented some sort of PCI enumerator - I see what devices the system has and I see all the data that's coming from their configuration space, but I've yet to put it to some use. Now I'm having a design decision to make and I'd love to get some helpful nudge.

1. I couldn't find whether PCI specs say something about Hot-plugging, so I'm wondering can the PCI topography change at run-time?
2. What's the best practice to look up a specific device based on it's class and subclass? Do you cache device addresses and class/subclass information or do you re-enumeration at each request?
3. Is the approach mentioned in wiki (Recursive enumeration) 100% bulletproof? Is it safe to assume that the device 0:0 (bus:device) will always be PCI-to-PCI bus if there's any at all, not, for example, 0:1?

Thanks in advance.

[Jezze]

Yes I would say the recursive detection would be close to 100%.

I dont know what happens if you hotswap a PCI device but in general I don't think you need to cache the devices you found because if you use the recursive method it is usually pretty quick to do a lookup and you save alot of complexity. I dont think you can assume anything about what device goes where even if it is true in 99.9% of the cases. And why would you assume that to begin with?

EDIT: After looking at that example of recursive detection from the wiki I'm not sure it is 100%. Brendan wrote a good detection algorithm here on the forum a while ago and Brendan being himself I am pretty sure it is close to 100% reliable.

[Combuster]

The "simple" recursive search expects that the BIOS got everything right during set-up. The "advanced" recursive search with manually assigning bus numbers and IO/address ranges works even in the case of changed, misconfigured or hotplugged devices, but you'll need to know what share of the address space is used for PCI in the first place to be able to do that correctly.

And yes, I tried a specific hotplug PCI case: you can go access your chipset and re-enable the integrated video card when the firmware turned it off because it detected a superior replacement. (confirmed using an i945G)

[Brendan]

Hi,

1. I couldn't find whether PCI specs say something about Hot-plugging, so I'm wondering can the PCI topography change at run-time?

There's a separate specification called "PCI Hot-Plug Specification" that contains part of the information for hot-plug PCI. It doesn't give any details for the "hot-plug controller" (I assume controller itself isn't standardised and different systems can do different things) or the "hot-plug system driver" (for 80x86 I assume this is mostly done via. ACPI).

2. What's the best practice to look up a specific device based on it's class and subclass? Do you cache device addresses and class/subclass information or do you re-enumeration at each request?

I normally build a full device tree of hardware, starting with "computer" at the top; then NUMA nodes as children to "computer"; then things like CPUs, memory and PCI host controllers as children to the NUMA node/s. This tree is meant to include everything, including devices that aren't PCI devices (monitors, USB hubs and USB devices, hard disks, ISA devices, etc).

For example; for each piece of hardware in the device tree; I'd keep track of if things like:

• device identification information (e.g. vendorID and deviceID for PCI; EISA ID for ISA, etc), including a human readable text string (e.g. "generic PS/2 mouse" or "LG Flatron L246WHX monitor" or whatever) and (where possible) some sort of UUID or serial number
• if there's a device driver and what state it's in (not started, starting, running, crashed)
• some sort of reference to the device driver (process ID)
• what "power management state" the device is meant to be in (running, reduced power, sleeping, off). Note: if a device doesn't support any power management at all, then the device driver just pretends that does
• which resources (IRQ, IO ports, memory mapped areas) the device uses
• if the device has been assigned to a "user IO group", which group. Note: this is multi-user support. E.g. if there's 3 monitors and 2 keyboards then 2 monitors and one keyboard might be assigned to "user IO group #1", and the other monitor and keyboard might be assigned to "user IO group #2".
• error logging; where an error report might be "device driver crashed", or "controller timeout" or "CRC error" or whatever
• statistics for hardware maintenance and failure prediction. For example, for a mouse you might track how many meters it's been moved and how much time it's been in operation; and then you might setup rules like "needs to be cleaned every 500 meters" and "mean time between failures is 4 years", so that the system can automatically notify the maintenance department when the mouse is due for cleaning or replacing. Note: This is mostly intended for larger companies/businesses/LANs
• physical location. E.g. a printer might be "in corridor outside room #12". This is mostly just notes so maintenance people can find it easily.

There would also be a piece of software called the "device manager" that manages this information and manages device drivers. For example; if a USB controller driver crashes then the device manager would tell any drivers for USB devices that they won't be able to talk to the USB controller; or if there's been no activity for a "user IO group" for a while the device manager might tell device drivers for all devices assigned to that "user IO group" (but not others) to go into a power saving state.

Some of the information would be loaded from disk at boot (e.g. resources for ISA devices, error log, statistics, physical location) and saved to disk if it changes; and some of the information would be entered by humans (e.g. resources for ISA devices, physical location). The device manager would be responsible for that too.

Mostly what I'm trying to say is that PCI devices are a small part of a larger problem; and once you've got software in place to handle the larger problem (e.g. including ISA devices, USB devices, etc) it might not make sense to scan the PCI buses instead of caching the information for PCI devices.

3. Is the approach mentioned in wiki (Recursive enumeration) 100% bulletproof? Is it safe to assume that the device 0:0 (bus:device) will always be PCI-to-PCI bus if there's any at all, not, for example, 0:1?

As far as I know; the general idea behind the recursive enumeration is bulletproof; but if bus 0 device 0 is multi-function, then I'd check all functions of the multifunction device for host controllers (just in case).

However; it won't work for hot-plug PCI when devices are inserted after boot (as the hardware wouldn't have existed during boot when the firmware assigned bus numbers and/or resources to PCI devices). This means that if you intend to support hot-plug PCI then you'll need to be able to do the "Recursive Scan With Bus Configuration" thing. Note that you may also need to be able to reconfigure existing device's resources (possibly while the devices are being used). For example, if a bridge is configured to forward physical memory accesses in the range 0xD0000000 to 0xD8000000 to its secondary bus, and then someone (using hot-plug PCI) inserts 3 video cards (with 512 MiB of memory mapped IO each) onto that secondary bus; then you may have to reconfigure other bridges (and all devices connected to the other bridges) to create enough space; then reconfigure the bridge that the video cards got plugged into and any existing devices on that bridge; just so that you've got enough space for the inserted video cards and can start configuring them.

Note: This "reconfigure resources while a device is running" should sound scary. With careful design, it should be possible to be able to change a device's resources without the device driver knowing or caring - e.g. "freeze" the driver, then change the physical address of the device's memory mapped IO areas, then change the device driver's page tables to suit, then "unfreeze" the driver. You may need to rearrange IO ports too, and it's possible to come up with a scheme to allow the IO ports a device uses to be changed without the driver knowing/caring (e.g. make the driver use a kernel function to access IO ports; or emulate instructions like "IN" and "OUT" in the general protection fault handler; such that IO ports a driver uses are virtualised).


Cheers,

Brendan

[gusc]

Thank you for painting a broader picture here. Although I'm far away from user space, usb and other bells and whistles, I, for now, am trying to get some data from HDD through AHCI, thus I had to find the AHCI controller in the PCI space first.

So if I understand you correctly - it is OK to cache PCI devices. I'm not going to use PCI Hot-plug, as I think it's not widespread in the desktop PC world, is it? And it seems sane to gather this information at boot-time and store it in memory for run-time, as these IO operations seemed quite slow (the enumeration itself, I mean).

[Brendan]

Hi,

So if I understand you correctly - it is OK to cache PCI devices.

Yes; partly because you need extra information associated with a device anyway (e.g. if the PCI device has a driver and which driver it is, etc).

I'm not going to use PCI Hot-plug, as I think it's not widespread in the desktop PC world, is it? And it seems sane to gather this information at boot-time and store it in memory for run-time, as these IO operations seemed quite slow (the enumeration itself, I mean).

You can probalby split "hot plug PCI" into a few different categories:

• large fault tolerant servers (rare); involving special hardware to electrically isolate "PCI slots", little indicators to show which slot/s are ready for insertion/removal (to make it less likely that someone will remove the wrong PCI card), etc.
• Some laptop docking stations (probably not too common); where it's likely that the firmware/OS can know about the resources needed by the docking station when it's not plugged in and can make allowances for it (e.g. reserving areas of the physical address space for the docking station and configuring bridges to suit).
• Some types of devices designed for laptop/mobile use (e.g. ExpressCard).

However; I'm "strongly in favour" of separating the design of the OS (e.g. APIs, etc) from any specific implementation of that design; and making sure that the design is able to handle everything that might possibly be needed one day (even if a specific implementation of the design doesn't support various features).


Cheers,

Brendan

[Antti]

Warning: Explicit Content. For Entertainment Purposes Only.

Brendan's April Fool's Day OS:

- Available on Github.
- Written in C++ with assembly parts in gas syntax.
- GNU Autoconf configure+make etc. build scripts.
- GRUB boot loader.
- GPL3 license.

- Monolithic kernel.
- BIOS interrupts for I/O (real mode).
- Segmented memory model with paging (double security).
- POSIX system call interface + Linux compatibility layer (partial).
- Ported newlib, bash, binutils, gcc, and other common GNU tools.
- VGA text mode console (experimental support for 800x600 VBE graphics console).
- ELF executable format.
- Shared libraries.
- ACPICA ported (power off feature).
*Note! Hot plug power management also supported: OS stops if power cord is unplugged.

- PS/2 keyboard support (US layout supported).
- Highly dynamic device driver interface (no formal specification restrictions).
- PCI enumeration
- Enhanced round-robin scheduler (skips blocking tasks).
- Support for 16 concurrent processes.
*Note! Can be defined in compile time. Maximum is something around 64 because static task structures are accessed in real mode (below 0x100000).

[Kazinsal]

- PS/2 keyboard support (US layout supported).

Meanwhile, his real OS will support a completely unique keyboard layout (freely available keycap stickers are available, but in a format usable only by his OS), but if you sacrifice a sealed boxed copy of Windows ME as a burning effigy, you get ten minutes of Dvorak.