OSDev.org

Hello everyone,

I just wanted to transition my hobby OS from using the PIC to using the IO APIC for interrupt delivery but I've stumbled on some problems.
What I've done so far:

• I programmed the PIC in cascade mode and I've masked all interrupts excepted the cascade one (bit 2 in master).
• I've parsed the ACPI MADT table to determine the IO APICs present in the system and the interrupt overrides. In my case there is 1 IO APIC and no interrupt overrides.
• I've enabled the LAPIC on the BSP and written the LDR register to 1.
• I've programmed the IO Apic redirection entries for the IRQs in which I'm interested leaving the interrupts masked. These are programmed with destination mode logical, destination 0xFF and delivery mode lowest priority. The interrupts in my case are IRQ 1 and 8 (keyboard and RTC) - the IO Apic GlobalIrqBase in the MADT was set to 0 => I programmed entries 1 and 8 in the IO Apic.
• I've enabled the LAPIC on the APs and written the LDR register to 1 << ApicId().
• I've unmasked the IO Apic entries in which I'm interested.
• I've enabled interrupts on all the processors.
• At the end of each interrupt (may be generated by the IO Apic or by the LAPIC - I also have LAPIC timer enabled on all the processors) I write to the EOI register.

My idea is to use lowest priority delivery mode to balance the workload among processors, but unfortunately the only processor to receive the interrupts is the BSP even if the the processor's priority is greater than that of the APs (I wrote a 0x2 in the BSP cr8 to make sure).
If I write in the destination the value 0xFE (all processors except BSP) then another CPU will be the one which always receives the interrupt - so far I've seen either CPU1 or CPU7.

I've read both the IO Apic and the local APIC documentation (Intel manual). The local APIC specification specifies the ability for a processor to send lowest priority interrupts is processor specific, but I did not see any limitations on receiving these kind of interrupts.

Is there something I'm missing?
Thanks in advance!

alexg wrote:In my case there is 1 IO APIC and no interrupt overrides.

How odd. How can there be no overrides? In the PIC, IRQ 0 is the PIT. In the I/O APIC, IRQ 0 is the local APIC timer and there should always be at least one interrupt override: ISA IRQ 0 (PIT) to I/O APIC IRQ 2 (master PIC cascade, which is unused on the I/O APIC).
Even in QEMU, there seem to be many interrupt overrides:

My kernel wrote:

Code: Select all

[ioapic] remapping PIC IRQ 0 to I/O APIC IRQ 2
[ioapic] remapping PIC IRQ 5 to I/O APIC IRQ 5
[ioapic] remapping PIC IRQ 9 to I/O APIC IRQ 9
[ioapic] remapping PIC IRQ 10 to I/O APIC IRQ 10
[ioapic] remapping PIC IRQ 11 to I/O APIC IRQ 11

Wow, thanks for that!

You seem to be right, I've looked over the code which parses the MADT and I've seem to have skipped over interrupt overrides . Unfortunately, there is only 1 interrupt override (the one you mentioned).

Where does it say that that the LAPIC timer is connected to the first IO Apic redirection entry?
I didn't configure the IO Apic in any way when I programmed the LAPIC timer.

Hi,

alexg wrote:Is there something I'm missing?

Sadly, no. As far as I can tell "send to lowest priority" should be working.

Note that while you didn't mention the local APIC's Logical Destination Format Register, it must be at least "correct enough" for CPU1 to have received the IRQ when destination=0xFE, so I know the Logical Destination Format Register isn't the problem.

My advice at this point would be to read from the Arbitrary Priority Register (at "local APIC base + 0x0090") on each CPU and see what they contain (and double check that AP CPU processor priority isn't higher than TPR due to a "stuck" interrupt or something); and to test on more/other computers to see if the problem effects all computers or only some.


Cheers,

Brendan

alexg wrote:Where does it say that that the LAPIC timer is connected to the first IO Apic redirection entry?
I didn't configure the IO Apic in any way when I programmed the LAPIC timer.

The OSDev wiki says that. To make space for the Local APIC timer, they put the PIT at I/O APIC IRQ 2. BTW, you should have another override for ACPI SCI IRQ.

@omarrx024: that is the only override I have on VMWare, on the physical machine I also have an override from IRQ 9 to 9 specifying the interrupt is level triggered.

The Logical Destination Format Register on each CPU has the value 1 << CpuApicId(). So CPU0 has 1, CPU1 has 2, etc...

Well I've read the APR and they return 0 on all processors, and the Intel manual states that this register is not supported on Pentium 4 and Xeon processors.

I've placed some asserts in my interrupt handler before the IRET (but after the EOI) to verify that both CR8 and PPR get to 0 and the assert seems to hold.

I think I've found something interesting: lowest priority interrupts seem to work fine amongst APs. I continued testing using the 0xFE destination and even though the same CPU is always targeted if they all have the same priority the mechanism works correctly if I increase CPUs TPR because the next CPU becomes the target.

Hi,

alexg wrote:The Logical Destination Format Register on each CPU has the value 1 << CpuApicId(). So CPU0 has 1, CPU1 has 2, etc...

I think we're both getting confused. There's the Logical Destination Register (at "local APIC base + 0x00D0") which can be set to "1 << CpuApicId()". Then there's the Destination Format Register (at "local APIC base + 0x00E0") which you haven't mentioned. Both of these registers are needed for logical destination to work properly.

alexg wrote:Well I've read the APR and they return 0 on all processors, and the Intel manual states that this register is not supported on Pentium 4 and Xeon processors.

"Xeon" is mostly a meaningless marketing term. When Intel's manual says something like "Pentium 4 and Xeon processors" typically they mean "on netburst CPUs" (and not Xeons based on P6). In any case, the Processor Priority Register (at "local APIC base + 0x00A0") is mostly the same.

alexg wrote:I've placed some asserts in my interrupt handler before the IRET (but after the EOI) to verify that both CR8 and PPR get to 0 and the assert seems to hold.

I think I've found something interesting: lowest priority interrupts seem to work fine amongst APs. I continued testing using the 0xFE destination and even though the same CPU is always targeted if they all have the same priority the mechanism works correctly if I increase CPUs TPR because the next CPU becomes the target.

Do you explicitly set the TPR on AP CPUs to zero (even though it should already be zero)? It could be that the chipset assumes that AP CPU's arbitration priorities are "max" to avoid sending IRQs to AP CPUs before they're started; and that you need to set TPR to zero (even though it should already be zero) to make the CPU inform the chipset of the AP CPU's true priority.


Cheers,

Brendan

The DFR is set to 0xFFFF'FFFF (flat model).

I tried setting the APs TPR explicitly to 0 but it still doesn't work, thanks for the suggestion anyway!