Strange IOAPIC vector number and /proc/interrupts in Linux.

[wangt13]

I am writing code in Linux (ubuntu 18.04 and ubuntu 16.10) to understand Interrupt handling in X86 Linux.

With a tool, I dumped the registers of IOAPIC in my thinkpad laptop as followings.

Code: Select all

[14508.865716] RT Entry 1 : 0001000000000001
[14508.865721]   Dest           : 00
[14508.865725]   EDID           : 01
[14508.865729]   Mask           : not masked
[14508.865733]   Trigger mode   : edge
[14508.865736]   Remote IRR mode: 0
[14508.865740]   Polarity       : active high
[14508.865744]   Delivery status: idle
[14508.865747]   Dest mode      : physical
[14508.865751]   Delivery mode  : 000
[14508.865754]   vector         : 01
[14508.865778] RT Entry 2 : 0003000000000002
[14508.865792]   Dest           : 00
[14508.865806]   EDID           : 03
[14508.865821]   Mask           : not masked
[14508.865833]   Trigger mode   : edge
[14508.865847]   Remote IRR mode: 0
[14508.865859]   Polarity       : active high
[14508.865871]   Delivery status: idle
[14508.865881]   Dest mode      : physical
[14508.865884]   Delivery mode  : 000
[14508.865888]   vector         : 02
[14508.865979] RT Entry 8 : 000f000000000008
[14508.865983]   Dest           : 00
[14508.865987]   EDID           : 0F
[14508.865990]   Mask           : not masked
[14508.865994]   Trigger mode   : edge
[14508.865997]   Remote IRR mode: 0
[14508.866001]   Polarity       : active high
[14508.866005]   Delivery status: idle
[14508.866008]   Dest mode      : physical
[14508.866011]   Delivery mode  : 000
[14508.866015]   vector         : 08
[14508.866035] RT Entry 9 : 0011000000008009
[14508.866039]   Dest           : 00
[14508.866043]   EDID           : 11
[14508.866047]   Mask           : not masked
[14508.866050]   Trigger mode   : level
[14508.866054]   Remote IRR mode: 0
[14508.866057]   Polarity       : active high
[14508.866072]   Delivery status: idle
[14508.866086]   Dest mode      : physical
[14508.866101]   Delivery mode  : 000
[14508.866115]   vector         : 09
[14508.866171] RT Entry12 : 001700000000000c
[14508.866175]   Dest           : 00
[14508.866179]   EDID           : 17
[14508.866183]   Mask           : not masked
[14508.866187]   Trigger mode   : edge
[14508.866190]   Remote IRR mode: 0
[14508.866194]   Polarity       : active high
[14508.866197]   Delivery status: idle
[14508.866201]   Dest mode      : physical
[14508.866205]   Delivery mode  : 000
[14508.866208]   vector         : 0c
[14508.866269] RT Entry16 : 001f00000000a010
[14508.866282]   Dest           : 00
[14508.866296]   EDID           : 1F
[14508.866308]   Mask           : not masked
[14508.866312]   Trigger mode   : level
[14508.866315]   Remote IRR mode: 0
[14508.866319]   Polarity       : active low
[14508.866323]   Delivery status: idle
[14508.866327]   Dest mode      : physical
[14508.866330]   Delivery mode  : 000
[14508.866333]   vector         : 10
[14508.866433] RT Entry23 : 002300000000a017
[14508.866438]   Dest           : 00
[14508.866441]   EDID           : 23
[14508.866445]   Mask           : not masked
[14508.866449]   Trigger mode   : level
[14508.866453]   Remote IRR mode: 0
[14508.866465]   Polarity       : active low
[14508.866475]   Delivery status: idle
[14508.866479]   Dest mode      : physical
[14508.866482]   Delivery mode  : 000
[14508.866486]   vector         : 17

To my surprise, the vector number in IOAPIC does not between 0x10 and 0xFE.
At the same time, the /proc/interrupts in Linux shows something like below,

Code: Select all

            CPU0       CPU1       CPU2       CPU3       
   0:         23          0          0          0   IO-APIC    2-edge      timer
   1:         13          0        471          0   IO-APIC    1-edge      i8042
   8:          1          0          0          0   IO-APIC    8-edge      rtc0
   9:          0          0          0          0   IO-APIC    9-fasteoi   acpi
  12:         20          0       2144          0   IO-APIC   12-edge      i8042
  14:          0          0          0          0   IO-APIC   14-edge      ata_piix
  15:          0          0          0          0   IO-APIC   15-edge      ata_piix

I am trying to understand how the IOAPIC vector is mapped to the CPU core, does it need to go through local APIC to do re-routing?
Or is the vector number in IOAPIC directly delivered to the CPU core?

What are 24 redirection table entries in IOAPIC are used for?
The 1st entry is IRQ0, 2nd entry is IRQ1, 24th entry is IRQ25, is this correct?

Thanks,
-Tao

[Brendan]

Hi,

I am trying to understand how the IOAPIC vector is mapped to the CPU core, does it need to go through local APIC to do re-routing?
Or is the vector number in IOAPIC directly delivered to the CPU core?

What are 24 redirection table entries in IOAPIC are used for?
The 1st entry is IRQ0, 2nd entry is IRQ1, 24th entry is IRQ25, is this correct?

For IO APIC there's 2 cases. The first case is that the interrupt is sent by the IO APIC directly to a local APIC "as is" (and the local APIC forwards it to the CPU). For this case, vectors 0x00 to 0x1F are illegal, so it's safe to assume that is not what is happening on your computer.

The second case is that there's an IOMMU (including IRQ remapping), where interrupts from IO APIC (and from PCI devices using MSI) go to the IOMMU and then (potentially) get remapped to something else before being sent to a local APIC/CPU. This is necessary for systems with more than 255 CPUs (because the IO APIC only has an 8-bit "destination ID"), and it's "potentially convenient" for things like KVM (so that KVM can just mess with the IOMMU when a device is assigned to a guest and doesn't need to change both the IOMMU and the IO APIC).

What are 24 redirection table entries in IOAPIC are used for?

If the IO APIC has 24 inputs, then you'll have 24 redirection table entries in the IO APIC to determine what happens when the corresponding input is triggered. How devices are connected to IO APIC inputs is motherboard specific (you have to use ACPI to determine what they are, and if they should be edge triggered or level triggered, or active high or active low). What the IO APIC sends (the destination stuff and vector) is configurable; and where it's sent (to the local APIC/s in the destination, or to an IOMMU that uses the fields as indexes into a remapping table) depends on whether an IOMMU is in use.

The 1st entry is IRQ0, 2nd entry is IRQ1, 24th entry is IRQ25, is this correct?

IRQ numbering depends on context because each bus has its own separate IRQ numbering scheme.

Looking at your IO APIC register dump; (based on random educated guesswork and no facts) I'd assume that it's like:

• ISA IRQ 0 = IO APIC input #2 = global interrupt #2 = IOMMU remapping table entry 0
• ISA IRQ 1 = IO APIC input #1 = global interrupt #1 = IOMMU remapping table entry 1
• ISA IRQ 8 = IO APIC input #8 = global interrupt #8 = IOMMU remapping table entry 7
• ACPI SMI = IO APIC input #9 = global interrupt #9 = IOMMU remapping table entry 5
• ISA IRQ 12 = IO APIC input #12 = global interrupt #12 = IOMMU remapping table entry 11
• PCI IRQ A = IO APIC input #16 = global interrupt #16 = IOMMU remapping table entry 15
• PCI IRQ ? = IO APIC input #23 = global interrupt #23 = IOMMU remapping table entry 17

I have no idea what the numbers in "/proc/interrupts" are supposed to represent - the numbers at the start of each line don't correspond to any numbering scheme, and neither do the numbers in the second to last column. However, the last lines of "/proc/interrupts" (referring to a PATA disk controller from an ancient PIIX chipset from 20+ years ago, using edge triggered/ISA IRQs and not level triggered PCI IRQs) makes me wonder if you've posted "/proc/interrupts" information from inside a virtual machine that has nothing to do with IRQs on the real/host machine (and nothing to do with the information in the IO APIC dump from a real machine) .


Cheers,

Brendan

[wangt13]

Thank Brendan for the detailed answer.

Based on your answer, I think I missed 2 things.
The first one is the IOMMU, you are right, there is IOMMU in my system. I will read more about it.
The 2nd one is the ACPI, where the interrupt info is configured by BIOS, then read by OS. Could you tell me which ACPI table I should check for the IOAPIC configuration?

You are right, I am using VM.

Thanks,
-Tao

[Brendan]

Hi,

Based on your answer, I think I missed 2 things.
The first one is the IOMMU, you are right, there is IOMMU in my system. I will read more about it.

For that; the document you'll probably want is called "Intel® Virtualization Technology for Directed I/O" - I should've mentioned that last time.

The 2nd one is the ACPI, where the interrupt info is configured by BIOS, then read by OS. Could you tell me which ACPI table I should check for the IOAPIC configuration?

There's 2 parts. The first part is the MADT/APIC table which describes the IO APICs (how many, how many inputs each one has) and which IO APIC input each of the legacy IRQs uses (mostly only old hardware that's built into the chipset now, that used to use ISA bus and ISA IRQs - e.g. PIT, RTC/CMOS, PS/2, ...).

The second part is buried in AML and is used to figure out which IO APIC input each PCI device uses. This is the part that makes you wish ACPI never existed.


Cheers,

Brendan