PCI initialization

[rdos]

I have a problem with two different UEFI BIOSes that apparently have not initialized PCI according to my expectations.

In one of them, the network chip doesn't generate interrupts, even if it has a BAR0 that appears to work (it's IO mapped). The strange thing is that if I change BIOS to boot from network (but don't have the network cable connected), then the network chip does generate interrupts like it should. So, apparently, the UEFI BIOS does some initialization on the network chip only if it is required to do so because it attempts network boot. The network chip is on it's own PCI bus.

What could cause this?

[rdos]

Apparently, a few bits differ in configuration byte 0x79. When BIOS has the network active it is 0x50 while when it is is inactive the value is 0x20. The problem is that 0x79 is not part of the standard configuration registers.

Edit: Seems to be capability 10h, and the difference is in the max read request size. This doesn't seem to relate to the issue with MSI interrupt messages not being delivered.

[nullplan]

Well, obviously, the problem can only possibly be something with the device itself, or something on the way. If it isn't the former, it must be the latter. Have you tried looking at the bridges between the host and the network card?

[rdos]

Well, obviously, the problem can only possibly be something with the device itself, or something on the way. If it isn't the former, it must be the latter. Have you tried looking at the bridges between the host and the network card?

I suppose I finally need to identify PCI devices based on bridges instead of the "brute force" method. That would give the bridge hierarchy so I can follow the chain up to root, something I currently cannot do.

I suspect it is either the power management setting that is wrong or the flags for memory & IO accesses.

[rdos]

According to my PCI-PCI bridge document, it is between the secondary bus number and the subordinate bus number that a bridge is supposed to handle traffic, but the wiki seems to suggest it is only the secondary bus number that is deciding this.

Which one is right? Or perhaps nobody is using the subordinate bus number?

Also, the wiki uses the class type rather than the header type to check for bridges. My PCI document suggests it is the header type that determines if a PCI function is a bridge or not.

[kzinti]

I would (mostly) ignore anything the wiki says if you have a more authoritative source. There are some bits of useful info in the wiki sometimes, they can sometime point you in a direction. But mostly I find that I am losing time when trying to understand something using the wiki.

[rdos]

Only scanning bus 0 for valid PCI bridges does not work. My threadripper system has bus 64-68 and there is no bridge link to it from bus 0. There are ACPI entries that match these, and there are also PCI devices that are useful (for example, a 4:rd XHCI controller that has ports on the rear panel).

That means that "brute force" is required anyway. It's necessary to scan all buses that are not scanned as part of finding bridges, which ends up more or less the same as "brute force", except that the topology is also found. Perhaps a possibility would be to only scan the buses when ACPI entries that point to these buses are found.

Another problem with the same system is that several ACPI entries match the same PCI function, but I've discovered that keeping the first and ignoring duplicates gives a sane name-space where bridges have the base level ACPI name of the devices behind the brigde.

There is also a global interrupt problem in that the system has some IRQ entries from 0x80 to 0x90, but those could also be allocated dynamically which can cause conflicts. Maybe any device that is not MSI or MSI-X capable needs to pre-register it's interrupts to avoid conflicts. My drivers always use MSI or MSI-X before PCI IRQs.

[nullplan]

Only scanning bus 0 for valid PCI bridges does not work. My threadripper system has bus 64-68 and there is no bridge link to it from bus 0. There are ACPI entries that match these, and there are also PCI devices that are useful (for example, a 4:rd XHCI controller that has ports on the rear panel).

That means that "brute force" is required anyway.

Or you could give up your stance of "ACPI is useless" and notice that in this case, it appears to be giving you the PCI roots. Interesting that they are not contiguous. It does make more sense with the rest of PCI that they aren't, but it also means that I've been modelling multiple PCI roots wrong. I believe I got the idea from the wiki, so I may need to change some articles.

There is also a global interrupt problem in that the system has some IRQ entries from 0x80 to 0x90, but those could also be allocated dynamically which can cause conflicts. Maybe any device that is not MSI or MSI-X capable needs to pre-register it's interrupts to avoid conflicts. My drivers always use MSI or MSI-X before PCI IRQs.

I think you are supposed to read the ACPI information and reserve those GSI numbers first, before initializing devices and allocating numbers for MSI and MSI-X. You don't actually have to use those numbers, but need to keep them clear in any case.

[rdos]

Only scanning bus 0 for valid PCI bridges does not work. My threadripper system has bus 64-68 and there is no bridge link to it from bus 0. There are ACPI entries that match these, and there are also PCI devices that are useful (for example, a 4:rd XHCI controller that has ports on the rear panel).

That means that "brute force" is required anyway.

Or you could give up your stance of "ACPI is useless" and notice that in this case, it appears to be giving you the PCI roots. Interesting that they are not contiguous. It does make more sense with the rest of PCI that they aren't, but it also means that I've been modelling multiple PCI roots wrong. I believe I got the idea from the wiki, so I may need to change some articles.

I think my conclusion is that ACPI alone will not handle this correctly. There are a number of issues:
1. There are PCI devices that have no ACPI entries
2. There are ACPI entries that are not backed with functional PCI devices
3. Sometimes the processor entries are missing all-together in ACPI

Right now I'm relying on ACPI having at least one entry for every root PCI bus device, but I'm not convinced this is always the case (see 1 above). In my setup, there are PCI buses that have no ACPI entries, but those are linked to bridges, so by following bridges this can be handled correctly. The safe method would be to scan all buses regardless if there are ACPI entries or not.

Some PCI devices are linked to multiple ACPI objects, which really sucks if you want to use the ACPI path to figure out the position of devices. The topology clearly must be discovered by following PCI bridges, and ACPI is not reliable.

There is also a global interrupt problem in that the system has some IRQ entries from 0x80 to 0x90, but those could also be allocated dynamically which can cause conflicts. Maybe any device that is not MSI or MSI-X capable needs to pre-register it's interrupts to avoid conflicts. My drivers always use MSI or MSI-X before PCI IRQs.

I think you are supposed to read the ACPI information and reserve those GSI numbers first, before initializing devices and allocating numbers for MSI and MSI-X. You don't actually have to use those numbers, but need to keep them clear in any case.

On my most advanced threadripper system, I don't use any PCI IRQ. All of the devices have MSI and/or MSI-X. That means I don't need ACPI for anything on that particular system.

I only think you need to reserve GSIs if the devices lack MSI or MSI-X. At least with my design since I never use PCI IRQs if there MSI or MSI-X exist.

I also note in my new PCI device dump that some devices don't appear to have PCI IRQ information from ACPI, rather report legacy IRQs.