LAPIC IRQ limit per CPU core

[liviozanol]

Hi everyone.

I'm not sure if this is the right place to post this and not sure if anyone would be able to help me, but I'm trying anyway.

I'm running some computes on Ubuntu 20 (kernel 5.14) with isolated CPUs and holding 8 CPU cores (HT include) for house keeping.

I'm trying to isolated all IRQs to those 8 CPU cores.

The problem is that, considering the hardware and it's MSI/MSI-X configuration, I have more than ~ 200 MSI/MSI-X IRQs per CPU core.

When trying to pin some IRQs when I'm above 200 IRQs on those CPU cores I'm getting:

Code: Select all

echo 24 > /proc/irq/1234/smp_affinity_list
shell: echo: write error: No space left on device

Presumably, I'm hitting the LAPIC limit mentioned on section 10.5.2: https://www.intel.com/content/dam/www/ ... manual.pdf. "Local and I/O APICs support 240 of these vectors (in the range of 16 to 255) as valid interrupts."

Or maybe this(?) https://en.wikipedia.org/wiki/Advanced_ ... Controller " A single LAPIC may support up to 224 usable interrupt vectors from an I/O APIC. Vector numbers 0 to 31, out of 0 to 255, are reserved for exception handling by x86 processors."

I was trying to find some information on the Internet (this forum and the OSdev wiki included) but couldn't find anything reliable. In fact found some contradicting information...


I have the following questions:

• Should X2APIC expand the supported IRQs from 8 bits (256) to 32 bits?

• How can I see all the interrupts that are mapped on the LAPICs?

I really appreciate if anyone could help on these. I'm lost... Enabled X2APIC but I still can't pin more than 200 IRQs

[Octocontrabass]

Should X2APIC expand the supported IRQs from 8 bits (256) to 32 bits?

No. X2APIC expands the supported LAPIC identifiers from 8 bits to 32 bits, which means you can have more than 256 LAPICs and therefore more than 256 CPUs (since each CPU must have its own LAPIC). The interrupt vector is still only 8 bits, so IRQs can only use the 224 vectors that aren't reserved for CPU exceptions. Linux also reserves some of those 224 vectors for software interrupts.

How can I see all the interrupts that are mapped on the LAPICs?

I think you can find this information in /sys/kernel/debug/irq/ if your kernel was compiled with GENERIC_IRQ_DEBUGFS enabled.

[rdos]

You could potentially map the same IRQ to many different CPUs (using MSI or MSI-X), and let them handle different cases. It's a bit messy, but would increase the limit to 224 x number of CPU cores interrupts. Not sure if Linux supports this, but I decided against implementing it in my OS. It would preclude moving IRQs between cores, which is more useful than having more than 224 interrupts, or at least make it much more complicated.

[Octocontrabass]

Not sure if Linux supports this,

It's the only way Linux handles IRQs on x86.

[rdos]

Not sure if Linux supports this,

It's the only way Linux handles IRQs on x86.

So Linux doesn't support load balancing by moving IRQs and their server threads to less loaded cores?

[Octocontrabass]

So Linux doesn't support load balancing by moving IRQs and their server threads to less loaded cores?

Isn't that what you were describing?

Although Linux relies on userspace to manage the load balancing.