Multicore and IDT

[johnsa]

Hi,

Whats the best approach to managing IDTs/Handlers across multiple cores from a design/feasibility point of view?
I've been thinking of the below 3 options:

Option a)
Allocate an IDT per processor, some duplication of setup/assigning handlers to vectors, degree of flexibility, vector handlers can be by shared by assigning the same handler to each respective IDT entry,
but we'd need some way to pass the processor number/apicid to the handler.

Option b)
Use one IDT for all processors, no need to setup the handlers again, assigning a new handler from any core would automatically apply to all cores (Assuming the IDT resides in normal memory the cores should all see the up-to-date version, however locking would need to be used around adding/removing handlers), once again we'd have to have a way to let the handler know which processor/apicid.

Option c)
Every processor has it's own IDT, handlers are assigned per core and every core has a unique handler. No need to let it know which processor/apicid it is as it's implicit, however this is a lot of work creating duplicate handlers, for example
the LVT_Thermal, or LVT_Error handler is going to be identical per core.

[Brendan]

Hi,

Option b)
Use one IDT for all processors, no need to setup the handlers again, assigning a new handler from any core would automatically apply to all cores (Assuming the IDT resides in normal memory the cores should all see the up-to-date version, however locking would need to be used around adding/removing handlers), once again we'd have to have a way to let the handler know which processor/apicid.

This is the simplest and most common method. In this case you can send IRQs to a specific CPU (with the IO APIC or MSI) and you can also send an IRQ to a group of CPUs (using the IO APIC). Sending the IRQ to a CPU within a group of CPUs includes "send to lowest priority CPU"; which can be used to automatically send the IRQ to the CPU that isn't doing anything important (e.g. interrupt a CPU that's running a low priority task and avoid interrupting a CPU that's running a high priority/real time task).

The only real problem is what happens when you've got too many IRQs. Typically (out of 256 IDT entries) 32 are used for exceptions, a few are used by the OS for IPIs and/or software interrupts, a few are used for spurious IRQs (from IO APIC and PIC chips), and some are used by the local APIC. That leaves you with a maximum of maybe 215 IRQs that devices can use. Without MSI, this isn't likely to be a problem at all (you won't have that many IO APIC inputs anyway). With MSI, a single device might want a group of 32 IRQs and for a reasonably large server you can run out.

Also; due to the way the IRQ priority scheme works, even with 50 IRQs they might not be nicely distributed across priority levels (e.g. you might have 40 of them trying to use the 16 vectors for a specific priority) and you have to use "less than ideal" IRQ priorities.

Option c)
Every processor has it's own IDT, handlers are assigned per core and every core has a unique handler. No need to let it know which processor/apicid it is as it's implicit, however this is a lot of work creating duplicate handlers, for example
the LVT_Thermal, or LVT_Error handler is going to be identical per core.

This doesn't make too much sense to me; especially for things like exception handlers, IPIs, the local APIC's IRQs, etc. It also means you can't send an IRQ to a CPU within a group of CPUs (or do the "send to lowest priority" thing).

Option a)
Allocate an IDT per processor, some duplication of setup/assigning handlers to vectors, degree of flexibility, vector handlers can be by shared by assigning the same handler to each respective IDT entry,
but we'd need some way to pass the processor number/apicid to the handler.

That might make more sense if you're worried about running out of interrupt vectors; but it is more complex. I'm not too sure why you'd need a processor number or APIC ID - the handler can just read it from the local APIC if it cares (but it probably doesn't care).


There is a fourth option: support both. For example, you could have a single IDT used by all CPUs as default, but switch to one IDT per CPU if you determine (e.g. during PCI bus enumeration) that it's going to be beneficial. This is even more complicated (but probably not by very much). Of course if you chose this option you could implement the "one IDT shared by all CPUs" now, and then worry about adding support for "one IDT per CPU" later on.


Cheers,

Brendan

[rdos]

I don't think it makes sense to try to use the "lowest priority" thing, as it does't work on some (maybe even many) systems. It's a nice idea, but I doubt that anybody uses it, and that probably explains why it doesn't always work.

I think it is a better idea to migrate IRQs between cores based on load considerations.

[Brendan]

Hi,

I don't think it makes sense to try to use the "lowest priority" thing, as it does't work on some (maybe even many) systems. It's a nice idea, but I doubt that anybody uses it, and that probably explains why it doesn't always work.

It should work on all systems. Some CPUs don't support "lowest priority" for IPIs (but that has nothing to do with devices/IRQs and only effects IPIs).

I think it is a better idea to migrate IRQs between cores based on load considerations.

That'd likely have many race conditions, take far too long to adjust, and be less likely to work properly.


Cheers,

Brendan