PCI IRQs with I/O APIC

[BrightLight]

My OS uses the I/O APIC by default but I added PIC support for backwards-compatibility. But I have a question: every PCI device has the Interrupt Line field in its configuration space. So let's say for example, I have a PCI device with interrupt line 11, and the ACPI MADT table tells me that ISA IRQ 11 is I/O APIC IRQ 16. Can I use IRQ 16 for this device, then?
And if the I/O APIC IRQ is not gotten from the interrupt line but the interrupt pin, how do I use the interrupt pin? How do I know which IRQ, INTA# for example is using?

[Brendan]

Hi,

Let's start from the start...

For ISA; there was jumpers or switches on ISA cards used to select which ISA IRQ/s the card uses, and ISA IRQs were connected to PIC chip inputs. This was painful for everyone (adding a new ISA card meant investigating the existing cards and trying to find a free interrupt line manually).

Then they added PCI. PCI has 4 "PCI IRQs". They needed a way to connect them to whatever PIC IRQs happened to be unused by ISA cards (which could be using "random who knows what"). For this reason they added a "PCI IRQ router" so that PCI's IRQs could be connected to most of the PIC inputs and so it could be configured in BIOS settings. Each PCI device uses PCI IRQs in order, and (to reduce PCI IRQ sharing) the IRQ lines were connected to different PCI slots differently. That way with 4 PCI cards all using "pin #1 at the slot" they end up using different PCI IRQs at the PCI host controller. The "interrupt pin number" in PCI configuration space tells you which "pin # at the slot", which is completely useless. So that software (an OS) could figure out how PCI devices are connected to the PIC chips they used a "interrupt number" in PCI configuration space, where the BIOS (using chipset specific knowledge of how the motherboard is wired up and how it configured the "PCI IRQ router") sets this "interrupt number" in PCI configuration space for the OS.

Then they added IO APICs. These typically have 20 or more inputs and don't need any funky router thing (ISA can use 15 IO APIC inputs and PCI can use other/separate inputs); so they just connected PCI IRQs directly to IO APIC inputs. The "interrupt number" in PCI configuration space was already being used for PICs so to work around that (and because IO APICs were only really used in multi-CPU systems originally) they added the relevant information to the Multi-Processor Specification's tables.

Then ACPI came along. The problem with the older "Multi-Processor Specification's tables" is that it's a static table, and can't work for hot-plug PCI (where devices can be added/removed after the firmware creates the table). For this reason (at least that's what I assume) ACPI shifted the "which IO APIC input is used by which PCI device" into its AML. This means you need an AML interpreter just to figure out which IO APIC input a PCI device uses. The older "Multi-Processor Specification's tables" got deprecated (you can't assume they exist or are reliable on modern computers).

Then they added MSI (Message Signalled Interrupts) to PCI, first as an optional feature, then (for PCI-E) as a mandatory thing. This bypasses the IO APIC's inputs and sends a message directly. The nice thing about this is you can configure the IRQ vector (to send as a message to the CPU) in the device's PCI configuration space and don't need an AML interpreter.

So let's say for example, I have a PCI device with interrupt line 11, and the ACPI MADT table tells me that ISA IRQ 11 is I/O APIC IRQ 16. Can I use IRQ 16 for this device, then?

No, PCI devices are connected directly to IO APIC inputs and have nothing to do with ISA IRQs in this case.

And if the I/O APIC IRQ is not gotten from the interrupt line but the interrupt pin, how do I use the interrupt pin? How do I know which IRQ, INTA# for example is using?

You don't use the "interrupt pin" - it's useless without motherboard/chipset specific information. However ACPI's AML may use the "interrupt pin" in conjunction with chipset specific voodoo to figure out the IO APIC input for the device.


Cheers,

Brendan

[max]

Hey Brendan,

I'll dig out that old thread to ask a little question (your explanation is very nice, you should add this to the Wiki!)

As I had first understood, when using the IO APIC you have to do the following to find out which interrupt to listen to for a PCI device:

• read the interrupt number from the PCI configuration space, lets say this says 14
• then check the ACPI tables for an "interrupt source override" in the MADT that says "irq source" = 14
• if there is an entry, use the "global system interrupt" field
• this is the IRQ number that the CPU will actually receive and that can be used in the kernel

Is that understanding basically correct? But, you said since ACPI, you should get the information from AML - does this mean you can't reliably use the information from the MADT table but have to have an AML interpreter?

Thanks a lot in advance!
Greets

[Brendan]

Hi,

As I had first understood, when using the IO APIC you have to do the following to find out which interrupt to listen to for a PCI device:

• read the interrupt number from the PCI configuration space, lets say this says 14
• then check the ACPI tables for an "interrupt source override" in the MADT that says "irq source" = 14
• if there is an entry, use the "global system interrupt" field
• this is the IRQ number that the CPU will actually receive and that can be used in the kernel

Is that understanding basically correct?

Mostly, no.

Think of it a bit like this:

Code: Select all

    ______________
   |              |
   | PCI IRQ line |
   | (at the      |
   | device/slot) |
   |______________|
          :
    ______________
   |              |
   | Motherboard  |
   | or chipset   |
   | wiring       |
   |______________|
          :               _______________        ____________
    ______________       |               |      |            |       _____________
   |              | ---> | Legacy router | ---> | PIC inputs | <--- |             |
   | PCI IRQS     |      |_______________|      |____________|      |             |
   | (at the PCI  |                          ________________       | Legacy IRQs |
   | host bridge) |                         |                |      |             |
   |______________| ----------------------> | IO APIC inputs | <--- |_____________|
                                            |________________|

If the "interrupt number" field in PCI configuration space says 14; then it says "PIC chip input 14 (after the legacy router thing)" and says nothing about which IO APIC input it is or which PCI IRQ it is at the PCI host bridge. If the "interrupt line" field in PCI configuration space says "line #A"; then it says "line #A at the device/slot before it gets mangled by motherboard/chipset wiring" and says nothing about what the interrupt is at the PCI host bridge or which IO APIC input it is.

The "interrupt source overrides" in the MADT only tell you how legacy IRQs are connected to IO APIC inputs and don't tell you anything at all about PCI IRQs.

For IO APICs, the interrupt number that the CPU receives is whatever you feel like putting in the IO APIC's entry for the corresponding IO APIC input. You only need to know which IO APIC input is used.

ACPI's "global interrupt numbers" are mostly the IO APIC input numbers, but rather than saying which IO APIC input on which IO APIC (2 numbers) it only says (e.g.) "global interrupt number 19" (one number); and you have to figure out that (assuming the first IO APIC has 16 inputs) global interrupt numbers 0 to 15 are IO APIC inputs 0 to 15 on the first IO APIC, and global interrupt numbers 16 to 31 are IO APIC inputs 0 to 15 on the second IO APIC, and "global interrupt number 19" is the fourth IO APIC input on the second IO APIC.

But, you said since ACPI, you should get the information from AML - does this mean you can't reliably use the information from the MADT table but have to have an AML interpreter?

It's impossible to determine how PCI devices are connected to IO APIC inputs without using one of the following methods:

• Use an AML interpreter to get the information out of ACPI's AML.
• Use the older "MultiProcessor Specification" tables. Note: These tables are deprecated and may not exist (and will never exist for UEFI); and if they do exist they can't work for "hot plug PCI" (e.g. when a new PCI device is plugged in after the tables are generated by firmware).
• Have a different "motherboard driver" for every single motherboard, which knows how the motherboard's wiring was done (and can convert "PCI IRQ line at the slot" into "PCI IRQ line at the PCI host bridge") and also knows how "PCI IRQ lines at the PCI host bridge" are connected to IO APIC inputs.
• Use a clever/complex auto-detection scheme (starting with "this device could be using any interrupt" and monitoring which interrupts occur and when the driver says its device caused an interrupt to eliminate possibilities until you reach "the only interrupt that wasn't eliminated must be the interrupt the device uses"). Note that this is hard to design due to race conditions; and means that you have to assume all "unknown after parsing MADT" IO APIC inputs are safe to configure as PCI IRQs even if they aren't PCI IRQs at all.
• Don't do it at all. For example, if all PCI devices support MSI then you could use MSI for everything (without caring about IO APIC inputs), and if there's one or more PCI devices that don't support MSI then you could use PIC chips (if PIC chips exist).

Also don't forget that an AML interpreter and AML can be used for a lot of things (power management, etc); and there's no point avoiding AML for figuring out PCI IRQs and then relying on AML for other things.


Cheers,

Brendan

[max]

Mostly, no.

Haha I mostly expected this.

Thanks for the explanation. The entire interrupt wiring stuff is quite hard to understand, but I think I'm getting closer. I'll go with the AML interpreter, ACPICA seems a good implementation and easy to port. As you said it also provides some other nice stuff, also it validates the ACPI tables for me.

About the auto-detection, have you actually attempted something like this? Is it viable?

[Brendan]

Hi,

About the auto-detection, have you actually attempted something like this? Is it viable?

I haven't implemented it yet. As far as I know there's no reason it can't work, as long as it's designed to handle the race conditions properly, and as long as it's designed to handle potential problems caused by configuring an IO APIC input as "level triggered" when its not (IRQ flood detection and prevention).

Note that the first thing you'd need is for device drivers to return a "my device did/didn't cause an IRQ" status; but this can be beneficial for shared IRQs anyway (e.g. only inform less likely drivers that the IRQ occurred if/when more likely devices have said they didn't cause the IRQ, to avoid overhead/task switches in micro-kernels). The other thing you might want is a special "training mode" for drivers that are able to force their device to generate an IRQ; where the OS only allows one driver to use "training mode" at a time and keeps asking for IRQs until there can be no doubt which IO APIC input the device is using; which would help it determine which IO APIC inputs are used by which PCI devices faster.

Note that I'm planning to not use ACPI's AML - as far as I'm concerned it's a security problem at worst (e.g. "malicious AML" injected before OS boots), a stability problem at best (e.g. buggy AML) and an overcomplicated mess at all times. Instead, I'm planning to use a motherboard driver for each motherboard, and fall back to other approaches (e.g. IO APIC input auto-detection for any PCI devices that can't use MSI) when there is no motherboard driver. For this reason I've been thinking about (and trying to find problems with) the "IO APIC input auto-detection for PCI devices" scheme for a long time.


Cheers,

Brendan