MSI - I think I need a tip..

[JulienDarc]

Hello,

I am at the stage where I will implement Message Signaled Interrupts. So I have the pcie 2.x/3.x specs spec under the elbow (and yes, i read it ), the intel dev guides and the linux/netbsd source code.

But there are two things I don't get :

Bits 31-20 — These bits contain a fixed value for interrupt messages (0xFEE). This value locates interrupts at the 1-MByte area with a base address of 4G – 18M. All accesses to this region are directed as interrupt messages. Care must to be taken to ensure that no other device claims the region as I/O space

In intel dev guide 3, sections 10.11.x

I couldn't find this address in the linux source (I may have missed the spot, but..).

1) Should I understand that , in the end, the os has to write to that address 0xFEE (logical, physical ?) to acknowledge the device ?
I am sure that the answer is no (even the spec doesn't mention it) but I am clueless

2) The vectors MSI uses : they are the one located in the IDT, right ? So we cannot have more than [256 -32(intel reserved) - 1 (spurious)] interrupts, right ? Even MSI-X could not allocate its 2048 interrupts (or is it not related with the 256 idt interrupts ?)

MSI(x) is a godsend to manage interrupts on modern hw ( I studied the pic, io-apic codes and they are a SLOW and INEFFICIENT mess).

I think I read so much that I cannot afford to understand anything more.

Your help will be greatly appreciated

[Brendan]

Hi,

1) Should I understand that , in the end, the os has to write to that address 0xFEE (logical, physical ?) to acknowledge the device ?
I am sure that the answer is no (even the spec doesn't mention it) but I am clueless

The interrupt would be received by the CPU's local APIC, which would forward it to the CPU (when IRQs are enabled, when no higher priority interrupt is in service). The CPU would start your interrupt handler, which would handle the interrupt and then send EOI to the local APIC.

Note that the address that MSI uses is a physical address and the highest 12 bits must be 0xFEE, which means the full address would be from 0xFEE00000 to 0xFEEFFFFF. This area is typically reserved for the local APICs. When Intel says "care must be taken to ensure no other device claims the area as IO space" they mean devices outside the CPU that might respond to message before it gets to the local APIC (and you do not need to tell the local APIC to use a different physical address).

2) The vectors MSI uses : they are the one located in the IDT, right ? So we cannot have more than [256 -32(intel reserved) - 1 (spurious)] interrupts, right ? Even MSI-X could not allocate its 2048 interrupts (or is it not related with the 256 idt interrupts ?)

PCI and MSI are intended to be "platform neutral" (and used by 80x86 and Itanium and PowerPC and ....). On 80x86 there's less than 223 vectors you can use, but on other platforms there's different limits.

Note that MSI-X allows you to give the PCI device less vectors than it requested (e.g. if a device actually does request 2048 interrupt vectors, you can give it 4 instead).

Also, on 80x86 the absolute limit is actually 223 vectors per CPU (because each CPU may have its own independent IDT). However, in practice you probably won't need so many interrupt vectors, and supporting "223 * number of CPUs" vectors adds complexity to the OS - you'd have to use "fixed delivery" for all IRQs and when you want an interrupt to be sent to a different CPU (e.g. because you're putting a CPU into a power saving sleep state or something) you'd have to reconfigure everything. More practical would be supporting ~223 vectors per NUMA domain (especially if/when multiple NUMA domains have IO hubs, and especially when you're using X2APIC and can send an interrupt to the lowest priority CPU within a specific NUMA domain).


Cheers,

Brendan

[JulienDarc]

Thanks a lot Brendan,

as usual, you put me back on track