How to handle interrupt routing for hot-(un)plugged CPUs?

[ShukantPal]

Developing my IOAPIC driver, I have a table of IOAPIC input-signals that hold the ISRs for each device IRQ. I could circularly assign each IRQ to all CPUs, and when all CPUs have been assigned, go to the next local IRQ in each CPU.

Code: Select all

CPU      LocalVector     IOAPICInput

0          32                   0
1          32                   1
0          33                   2
1          33                   3
0          34                   4
1          34                   5

and so on

Now what is hitting hard my brain is that what would software do when one CPU is removed while the system is running? Should all IOAPICInputs be reassigned - that sounds simple but inefficient. Or is that the best solution?

[Solar]

Uh... what kind of hardware are you looking at that supports CPU hot unplugging in the first place?

Neither Windows nor Linux do support hot unplugging, so my guess would be no-one is selling x86-based hardware prepared for it either.

So you're looking at...?

[Octocontrabass]

Should all IOAPICInputs be reassigned - that sounds simple but inefficient. Or is that the best solution?

Hot unplugging a CPU sounds like a pretty rare event. As long as you don't lose IRQs or delay them too much, you should be fine.

A better optimization might be changing your IRQ assignment strategy, since your circular method can end up steering a device's IRQ away from the nearest CPU.

Neither Windows nor Linux do support hot unplugging

Windows might not, but Linux does.

[Brendan]

Hi,

Developing my IOAPIC driver, I have a table of IOAPIC input-signals that hold the ISRs for each device IRQ. I could circularly assign each IRQ to all CPUs, and when all CPUs have been assigned, go to the next local IRQ in each CPU.

Code: Select all

CPU      LocalVector     IOAPICInput

0          32                   0
1          32                   1
0          33                   2
1          33                   3
0          34                   4
1          34                   5

and so on

That looks "less than ideal" for multiple reasons. The first reason is that (for APICs) the "LocalVector" part determines the IRQs priority, so you risk having (e.g.) a high speed network card having to wait for a "higher priority" ancient/slow floppy disk's IRQ handler to finish. The next reason is MSI, where you may need to assign a range of interrupt vectors to a single device. What this adds up to is some kind of dynamic "interrupt vector allocator".

The third problem is IRQ balancing - e.g. if IO APIC inputs #1 and #3 are both generating lots of interrupts then you'd want to consider sending them to different CPU cores. This should include power management (e.g. if a core is in a sleep state then send IRQs to a different CPU to avoid waking it up, and avoid higher IRQ latency and avoid increased power consumption).

Of course if you support rebalancing IRQs for (relatively common) power management reasons, then it shouldn't be much extra work to use the same code to rebalance IRQs for (relatively rare) hot-(un)plugged CPUs.

Uh... what kind of hardware are you looking at that supports CPU hot unplugging in the first place?

In general, for real hardware I think "soft offline" is more useful, especially for hyper-threading (e.g. taking one logical CPU offline so that the other logical CPU gets full use of a core). Beyond that it's supported in a few virtual machines (VVMware, VirtualBox), possibly for the same reason "memory ballooning" exists.


Cheers,

Brendan