I/O APIC and PCI interrupts

[madeeha]

I am curious to know how the 4 PCI interrupt lines are shared but this discussion did not ended conclusively.

Suppose there is device A with bdf 1:2:1 that and a device B with bdf 1:3:1 and that both share PCI INT A.

Device A's signal is connected to IOAPIC input pin# 17
Device B's signal is connected to IOAPIC input pin# 18

Since both devices share PCI INT A, which IOAPIC line will be asserted when both devices interrupt simultaneously?
Is this something thats handled in the handware?

Thanks!

[Brendan]

Hi,

Suppose there is device A with bdf 1:2:1 that and a device B with bdf 1:3:1 and that both share PCI INT A.

Device A's signal is connected to IOAPIC input pin# 17
Device B's signal is connected to IOAPIC input pin# 18

That doesn't make sense.

If both devices share PCI INT A; which IO APIC input pin is PCI INT A connected to?

If PCI INT A is connected to IOAPIC input pin# 17 then both devices end up using IOAPIC input pin# 17 (and neither device uses pin# 18). If PCI INT A is connected to IOAPIC input pin# 18 then both devices end up using IOAPIC input pin# 18 (and neither device uses pin# 17).


Cheers,

Brendan

[SpyderTL]

That's not what the example table (from 6 years ago) says.

Code: Select all

----------------------------------------
Bus   Device   IRQ Pin   I/O APIC Pin
----------------------------------------
0     2        A         16
0     26       A         16
0     26       B         21
0     26       D         19
0     26       C         18
0     27       A         22
0     28       A         16

Device 27 and 28 are both using what they think is pin A, but they are wired to two different I/O APIC pins, if this table is/was correct.

[Brendan]

Hi,

That's not what the example table (from 6 years ago) says.

Code: Select all

----------------------------------------
Bus   Device   IRQ Pin   I/O APIC Pin
----------------------------------------
0     2        A         16
0     26       A         16
0     26       B         21
0     26       D         19
0     26       C         18
0     27       A         22
0     28       A         16

Device 27 and 28 are both using what they think is pin A, but they are wired to two different I/O APIC pins, if this table is/was correct.

That table is talking about "interrupt pin A at the PCI slot" and not "PCI INT A at the host bridge".

Think of it like this:

Code: Select all

                                                    |                           |
                                                    |                           |
IO APIC pin #W-------PCI INT A at host-----------+  +--A at slot 1-----------+  +--A at slot 2
                                                 |                           |
IO APIC pin #X-------PCI INT B at host--------+  +-----B at slot 1--------+  +-----B at slot 2
                                              |                           |
IO APIC pin #Y-------PCI INT C at host-----+  +--------C at slot 1-----+  +--------C at slot 2
                                           |                           |
IO APIC pin #Z-------PCI INT D at host--+  +-----------D at slot 1--+  +-----------D at slot 2
                                        |                           |
                                        +-----------+               +-----------+
                                                    |                           |

..where "IO APIC pin #W is PCI INT A, but B at slot 1 and C at slot 2.

Note: In general every PCI card uses "A at slot" for its first function (and "B at slot" for its second function, etc); and every motherboard does this "barber pole" thing (where "PCI INT A" is connected to a different place at each slot) to try to reduce PCI IRQ sharing.


Cheers,

Brendan

[SpyderTL]

That was my understanding. But notice that the example table above has more than 4 distinct I/O APIC pins listed (16, 18, 19, 21, 22). How is this possible if there are only 4 interrupt pins connecting each bus to the I/O APIC? Am I missing something, or is the table just wrong?

[Brendan]

Hi,

That was my understanding. But notice that the example table above has more than 4 distinct I/O APIC pins listed (16, 18, 19, 21, 22). How is this possible if there are only 4 interrupt pins connecting each bus to the I/O APIC? Am I missing something, or is the table just wrong?

The number of IO APICs inputs per IO APIC, and the number of IO APICs, is not fixed. For a simple example, I have a computer here that has two IO APICs with 16 inputs per IO APIC (a total of 32 IO APIC inputs), and 2 PCI host controllers (a total of 8 PCI IRQs).

Each PCI host controller has 4 interrupt lines (but some systems have more than one PCI host controller); and in some cases devices built into the chipset may be connected to the I/O APIC using "extra" interrupt line/s (which I think is a violation of the PCI spec, but it does happen).

There would be nothing wrong with having (e.g.) four IO APICs with 32 inputs each and 32 separate PCI host controllers (with 4 PCI IRQs each); so that you end up with a total of 128 IO APIC inputs and a total of 128 PCI IRQs.


Cheers,

Brendan

[SpyderTL]

Each PCI host controller has 4 interrupt lines (but some systems have more than one PCI host controller); and in some cases devices built into the chipset may be connected to the I/O APIC using "extra" interrupt line/s (which I think is a violation of the PCI spec, but it does happen.

In the table above, all devices are listed as connected to bus zero, which means they should all be using the same host controller, I believe.

So how can they be on 5 different I/O APIC pins if there is only one host controller, and it only has 4 pins? The only possibility that I can think of is that one or more of these devices aren't "real" PCI devices, but are just emulated that way for enumeration purposes. Or this PCI host controller has more than 4 pins.

If the system has multiple I/O APICs, is each PCI controller interrupt pin physically connected to more than one I/O APIC?

[Brendan]

Hi,

Each PCI host controller has 4 interrupt lines (but some systems have more than one PCI host controller); and in some cases devices built into the chipset may be connected to the I/O APIC using "extra" interrupt line/s (which I think is a violation of the PCI spec, but it does happen.

In the table above, all devices are listed as connected to bus zero, which means they should all be using the same host controller, I believe.

So how can they be on 5 different I/O APIC pins if there is only one host controller, and it only has 4 pins? The only possibility that I can think of is that one or more of these devices aren't "real" PCI devices, but are just emulated that way for enumeration purposes. Or this PCI host controller has more than 4 pins.

Each PCI host controller has 4 interrupt lines (but some systems have more than one PCI host controller); and in some cases devices built into the chipset may be connected to the I/O APIC using "extra" interrupt line/s (which I think is a violation of the PCI spec, but it does happen.

If the system has multiple I/O APICs, is each PCI controller interrupt pin physically connected to more than one I/O APIC?

There's no guarantees for this - the motherboard manufacturer can happily connect IRQs from the same PCI host controller to multiple different IO APICs.


Cheers,

Brendan

[SpyderTL]

If the system has multiple I/O APICs, is each PCI controller interrupt pin physically connected to more than one I/O APIC?

There's no guarantees for this - the motherboard manufacturer can happily connect IRQs from the same PCI host controller to multiple different IO APICs.

Any idea how common this is? Almost never? Almost always?

What are the implications of this configuration at the OS level? It seems like at the very least, this would cause an interrupt on two different I/O APIC lines at the same time, and possibly two different IRQs on two different processors at the same time. Does the OS have to handle this situation?

[Brendan]

Hi,

If the system has multiple I/O APICs, is each PCI controller interrupt pin physically connected to more than one I/O APIC?

There's no guarantees for this - the motherboard manufacturer can happily connect IRQs from the same PCI host controller to multiple different IO APICs.

Any idea how common this is? Almost never? Almost always?

Most systems only have one IO APIC. For larger systems (with 2 or more IO APICs) I'm not sure but I'd suspect it's relatively rare, either because you've got an "IO hub" (IO APIC and PCI host bridge) for different NUMA domains, or simply because the motherboard designer had no reason to be "less logical".

What are the implications of this configuration at the OS level? It seems like at the very least, this would cause an interrupt on two different I/O APIC lines at the same time, and possibly two different IRQs on two different processors at the same time. Does the OS have to handle this situation?

It makes almost no difference to the OS at all. The local APIC handles ordering (not the IO APIC) and each CPU only has one local APIC; and if 2 IO APICs send IRQs at the same time then the local APIC tells CPU about the highest priority IRQ (until it gets EOI and sends the next).

ACPI tells you the "global IO APIC input number" for a PCI device, and it's relatively trivial to figure out that (e.g.) if the first IO APIC has 16 inputs then "global IO APIC input number is 22" must be "IO APIC input #6 on APIC #2".

The other thing that can complicate things slightly is the "directed EOI" feature introduced with x2APIC; where (instead of sending EOI to the local APIC where local APIC has to broadcast to all IO APICs) you send EOI to the IO APIC itself (to avoid the performance and correctness implications of "broadcast to all"). In this case the OS would have to keep track of which IO APIC is responsible for each interrupt to make sure it sends EOI to the correct IO APIC, but that can be a simple "which IO APIC" field in whatever structure the OS uses to manage IRQs.

Also note that for modern systems you'd use MSI for almost everything, and for MSI the IO APIC is effectively bypassed (device sends interrupt as a message directly to the local APIC/s without using an IO APIC input), so in that case it makes no difference how many IO APICs there are because you're not using them for much more than "legacy ISA" IRQs.


Cheers,

Brendan