Best way of mapping MSI-X interrupts to IDT interrupts

[Ethin]

So I'm curious what the best way of mapping MSI-X interrupts is. Right now I converge all the MSI-X vectors on a single IDT vector, but this has the major downside of making it impossible to tell what vector actually got triggered. I know that I can use the range 0x10-0xFE for the vector, but I'm hesitant to use anything below 0x20 because I don't want my MSI-X interrupts being considered CPU exceptions. I could map MSI-X vectors to any interrupt beyond interrupt 47, and though that works for devices with a low vector count, it becomes a problem for a device that has more than 207 vectors. So what's the best way of handling this? Is there any way to convey the "real" vector that was triggered if I mask the vector behind other IDT entries?

[Octocontrabass]

From what I've seen, hardware usually gives you so many MSI-X vectors so that you can direct each vector to a specific CPU. This way, each CPU can access the same piece of hardware (mostly) independently from other CPUs.

To use a NVMe controller as an example, you would use all of those vectors to assign each CPU its own submission and completion queues.

[Korona]

In general you do not want to use all possible vectors on all devices. Having more than one vector per device and CPU (which is what Octocontrabass suggests) does not really make sense.

[rdos]

I have an allocation mechanism for IRQs, which includes allocating a number of consequitive entries for MSI and MSI-X. Although, for MSI-X you don't need consequitive entries, but I use the same method there anyway. Having different IDTs per CPU or core might be possible, but I don't use that rather have global IRQs. The old-style IRQs have fixed mappings and so those must be shared between CPUs anyway.

The kernel automatically calculates links between IRQs and server threads, and when a link is detected the scheduler will make sure the IRQ is delivered to the same core as the server thread runs on. This allows for migrating IRQs and servers between cores.

I wouldn't say the maximum possible used IRQs should be used for each device since IRQs are a scarse resource. I usually put some limit like no more than four per device.