Page 1 of 1
IO APIC configuring
Posted: Mon Dec 23, 2024 10:02 pm
by mrjbom
I'm trying to configure the IO APIC, but I'm not quite sure how I can achieve the expected behavior.
I want my IDT to look like this:
0-31 CPU exceptions
32-47 PIC hardware interrupts (used before APIC initialization and PIC masking)
48 Local APIC Timer
49 Local APIC LINT0
50 Local APIC LINT1
41 Local APIC Error
42-57 PIC hardware interrupts after APIC initialization
255 Local APIC Spurious-Interrupt
But I don't understand if it is possible to make the interrupts (42-57) correspond to ISA IRQ, i.e. that the IO APIC sends interrupts as the PIC would do.
As far as I understand, the Interrupt Source Override table in the APIC contains the correspondence between the ISA IRQ and the IO APIC pin, and I have to configure the Redirection Table so that these pins refer to the IDT vector I want, right?
The problem is that my Interrupt Source Override has only 5 entries and I don't understand how the other interrupts will be handled ang how i should set redirection table entries for missing ISA IRQs.
For example, in the Interrupt Source Override, IRQ 0 is set to Global System Interrupt 2, does that mean I should set vector 42 in Redirection Table[2]?
But the strange thing is that the rest of the entries in Interrupt Source Override contain matches between IRQ and GSI, does that mean I should just set a non-standard Level and Trigger Mode(specified in Interrupt Source Override table)?
What about IRQs that are not in Interrupt Source Override?
Is it possible that IRQ 1 will be assigned to GSI 6 and there will be no entry for IRQ 6?
Is this how the IDT vector to be called is determined: IDT vector = Redirection Table[Global System Interrupt Base + Global System Interrupt].vector?
Re: IO APIC configuring
Posted: Mon Dec 23, 2024 11:34 pm
by Octocontrabass
mrjbom wrote: ↑Mon Dec 23, 2024 10:02 pmBut I don't understand if it is possible to make the interrupts (42-57) correspond to ISA IRQ, i.e. that the IO APIC sends interrupts as the PIC would do.
That's two different questions. If you want the IOAPIC to map ISA IRQs the same way the PICs would, that's possible and easy to do using interrupt source overrides in the MADT. If you want the IOAPIC to map
all IRQs the same way the PICs would, that's probably impossible and a terrible idea anyway. The IOAPIC supports more than 16 interrupt vectors, why wouldn't you want to use all of them?
mrjbom wrote: ↑Mon Dec 23, 2024 10:02 pmAs far as I understand, the Interrupt Source Override table in the APIC contains the correspondence between the ISA IRQ and the IO APIC pin, and I have to configure the Redirection Table so that these pins refer to the IDT vector I want, right?
Right.
mrjbom wrote: ↑Mon Dec 23, 2024 10:02 pmFor example, in the Interrupt Source Override, IRQ 0 is set to Global System Interrupt 2, does that mean I should set vector 42 in Redirection Table[2]?
Yes.
mrjbom wrote: ↑Mon Dec 23, 2024 10:02 pmBut the strange thing is that the rest of the entries in Interrupt Source Override contain matches between IRQ and GSI, does that mean I should just set a non-standard Level and Trigger Mode(specified in Interrupt Source Override table)?
Yes.
mrjbom wrote: ↑Mon Dec 23, 2024 10:02 pmWhat about IRQs that are not in Interrupt Source Override?
For ISA IRQs, the IRQ and GSI match, and the level and trigger mode are active-high edge-trigger. For other IRQs, you use ACPI (or legacy MP tables) to find how they're connected to the IOAPIC.
mrjbom wrote: ↑Mon Dec 23, 2024 10:02 pmIs it possible that IRQ 1 will be assigned to GSI 6 and there will be no entry for IRQ 6?
Yes. As far as I know, that situation could only happen when there are no ISA devices using IRQ 6.
mrjbom wrote: ↑Mon Dec 23, 2024 10:02 pmIs this how the IDT vector to be called is determined: IDT vector = Redirection Table[Global System Interrupt Base + Global System Interrupt].vector?
IDT vector = Redirection Table[Global System Interrupt - Global System Interrupt Base].vector
Re: IO APIC configuring
Posted: Tue Dec 24, 2024 12:22 am
by mrjbom
That's a great answer! Thank you!
Octocontrabass wrote: ↑Mon Dec 23, 2024 11:34 pm
That's two different questions. If you want the IOAPIC to map ISA IRQs the same way the PICs would, that's possible and easy to do using interrupt source overrides in the MADT. If you want the IOAPIC to map all IRQs the same way the PICs would, that's probably impossible and a terrible idea anyway. The IOAPIC supports more than 16 interrupt vectors, why wouldn't you want to use all of them?
I meant that there could be a possible scheme where:
Code: Select all
vector 42 - ISA IRQ 0
...
vector 57 - ISA IRQ 15
Re: IO APIC configuring
Posted: Tue Dec 31, 2024 3:10 pm
by rdos
You should focus on MSI and MSI-X whenever possible. These are programmed in the PCI device and doesn't need any input from MADT tables or ACPI. There is no reason to reserve IDT entries, rather these should be allocated on demand.
Re: IO APIC configuring
Posted: Tue Dec 31, 2024 5:36 pm
by mrjbom
rdos wrote: ↑Tue Dec 31, 2024 3:10 pm
You should focus on MSI and MSI-X whenever possible. These are programmed in the PCI device and doesn't need any input from MADT tables or ACPI.
Why should I use MSI/X? Is it easier to use them?
Re: IO APIC configuring
Posted: Wed Jan 01, 2025 2:45 pm
by nullplan
mrjbom wrote: ↑Tue Dec 31, 2024 5:36 pm
Why should I use MSI/X? Is it easier to use them?
Basically, yes. In order to use legacy interrupts, you have to parse and understand the wonky data structure that the AML in the ACPI tables returns and apply it to the situation you find in the system. In the system, you may find more devices than the AML mentions, or you may not find devices that are supposed to be there. You may indeed find bridges that aren't mentioned. And the whole logic with how many permutations are applied to the interrupt lines at each bridge are beyond me.
Or else, you use MSI, where you simply allocate a free interrupt for the device and then assign it dynamically, and you never have to look at the routing table ever, or run any AML code. And in MSI-X, you do the same thing, only it might be multiple interrupts. Try not to be overly greedy with them, you only have 224 to go around, and ca. 50 of those are typically in use by the interrupt controllers.
Re: IO APIC configuring
Posted: Wed Jan 01, 2025 4:02 pm
by rdos
nullplan wrote: ↑Wed Jan 01, 2025 2:45 pm
And in MSI-X, you do the same thing, only it might be multiple interrupts.
Another definite advantage of MSI-X is that each interrupt can be routed to different cores. It even should be possible to reuse the same IRQ but dispatch them to different cores. With MSI, you need to allocate a bunch of IRQs (with alignment and after each other), and they all will go to the same core.
I think it is possible to route IO-APIC IRQs to other cores than the BSP, but it's messy and I leave them assigned to BSP. The IO-APIC is connected to slow legacy devices anyway so no use in moving them.
My scheduler will also try to match IRQs with where the "server" thread is running, and then I move both MSI and MSI-X interrupts automatically to optimize load. IRQs connected to the IO-APIC are never moved, and the server threads are moved to BSP.
Re: IO APIC configuring
Posted: Wed Jan 01, 2025 11:59 pm
by mrjbom
@rdos and @nullplan, Thanks for the great answers!
Which should I implement, MSI-X or MSI?
Or both?(if so, which one should I start with).
I assume MSI-X support among devices is wide, right?
Are there many devices that only support MSI but not MSI-X?
Re: IO APIC configuring
Posted: Thu Jan 02, 2025 12:47 am
by nullplan
mrjbom wrote: ↑Wed Jan 01, 2025 11:59 pm
Which should I implement, MSI-X or MSI?
Or both?(if so, which one should I start with).
Both. Which one you use depends on which one the device supports. In the end, it is as simple as scanning the capability list for the right tag either way.
Re: IO APIC configuring
Posted: Thu Jan 02, 2025 1:59 am
by rdos
I think both, but MSI seems to be more widely supported than MSI-X, and devices that support MSI-X typically also support MSI. Therefore, it's probably best to start with MSI.
To support MSI and MSI-X you need to have a function to allocate IRQs. Also note that the higher bits in the IRQ # are related to priority, and you want high priority devices (or more sensitive devices) to allocate higher IRQ numbers than lower priority.