message signaled interrupts (MSI)

[snooky]

Hey folks,

I am investigating on some information about message signaled interrupts (MSI) on the PCI* bus. So far I understand what MSI is, how it works in general and what is good for.

But: Where can I find information which addresses I have to use to program into the message address register and which values do I set in the data register? This is architecture specific, I know. I am on a x86 architecture - so which addresses / data I have to set? Some links or explanations would be nice!

Thanks for your time in advance!

--snooky

[bluecode]

But: Where can I find information which addresses I have to use to program into the message address register and which values do I set in the data register? This is architecture specific, I know. I am on a x86 architecture - so which addresses / data I have to set? Some links or explanations would be nice!

It is system-specific at least that's what is written in the PCI Local Bus Specification 3.0. So basically you decide where the PCI device send the MSI data to. At least that is how I understood the Specs.

[snooky]

So basically you decide where the PCI device send the MSI data to. At least that is how I understood the Specs.

Right, I fully agree. So the question is: To which address should the PCI device send the data?

Say, we let it write to 0x12345678, but then I get no interrupt. It has to be a specific address which is translated by the chipset (north bridge) to an interrupt cycle on the APIC bus or the FSB or so. I think the address has some base (I guess it is the Local APIC mmio base address or something similar) and there is something more encoded like to which CPU (in a smp scenario) it goes and so on.

Do you, bluecode, or someone know more about this?

--snooky

[bluecode]

Ah, I think now I understand it, too

It quite late in the evening so I will just post what I found: from the linux kernel and you might want to search throught the kernel yourself, I think it might help.

edit: Just a thought, but it can't be the LocalAPIC since it is local to the processor and they are by default on the same physical address in every processor. So I don't think it is there.

[Brendan]

Hi,

It is system-specific at least that's what is written in the PCI Local Bus Specification 3.0. So basically you decide where the PCI device send the MSI data to. At least that is how I understood the Specs.

AFAIK that is correct - the PCI device writes a pre-specified 32-bit value to a pre-specified address, where the data and the address is architecture specific.

For 80x86 the format of the 32-bit data value is detailed in the Intel manual - see section "8.11.1 Message Address Register Format" and section "8.11.2 Message Data Register Format" in my copy of Intel's the System Programming Guide (note: it may be different section numbers in newer or older copies of the manual).

The address register goes like this:

• Bits 31 to 20 = 0xFEE = address is in the local APIC area (see note)
  Bits 19 to 12 = Destination APIC ID for the IRQ
  Bit 3 = RH = Redirection Hint (similar to the "Destination Shorthand" for IPIs)
  Bit 2 = DM = Destination Mode (same meaning as for IPIs)
  Bits 0 to 1 = Don't care (leave zeroed)

Note: This makes an address within the local APIC area, but that doesn't prevent the chipset from intercepting it *before* it reaches any local APIC in any CPU.

The data register goes like this:

• Bit 15 = Trigger Mode (edge or level triggered)
  Bit 14 = polarity (only if level triggered)
  Bits 8 to 10 = Delivery mode (same meaning as for IPIs)
  Bits 0 to 7 = Interrupt vector (must be above or equal to 32 I guess)

If you're familiar with using I/O APICs and/or IPIs this should all seem fairly simple to you...

I should mention that while I've read about Message Signalled Interrupts and think I understand them, I've never tried to use them and have a few questions of my own (like how you're meant to tell if the device needs "edge triggered" or "level triggered", although I could probably guess)...


Cheers,

Brendan

[snooky]

AFAIK that is correct - the PCI device writes a pre-specified 32-bit value to a pre-specified address, where the data and the address is architecture specific.

For 80x86 the format of the 32-bit data value is detailed in the Intel manual - see section "8.11.1 Message Address Register Format" and section "8.11.2 Message Data Register Format"

Uhm, a little bit embarrassing. I searched for exactly this, but sometimes I don't see the wood for the trees. Thank you!

I should mention that while I've read about Message Signalled Interrupts and think I understand them, I've never tried to use them and have a few questions of my own (like how you're meant to tell if the device needs "edge triggered" or "level triggered", although I could probably guess)... ;)

From http://msdn2.microsoft.com/en-us/library/bb961238.aspx :

An interrupt message is a particular value that a device writes to a particular address to trigger an interrupt. Unlike line-based interrupts, message-based interrupts have edge semantics. The device sends a message but does not receive any hardware acknowledgement that the interrupt was received.

That makes perfect sense to me.

I also looked in the Linux kernel and if I didn't miss anything they are only using edge triggering.

So long
snooky

[snooky]

The address register goes like this:[...]
Note: This makes an address within the local APIC area, but that doesn't prevent the chipset from intercepting it *before* it reaches any local APIC in any CPU.

If I understand hardware correct, it cannot be that such a message reaches a local APIC at all (at least not as a write to some register or something like this). It wouldn't make any sense at all, because the address is made up artificially and doesn't mean any register or something.

The chipset has always to translate the "posted memory write" from PCI to either a memory access cycle on the memory bus or if it decides that it is a "special address" to do something "magical". In the "MSI on x86 architecture" case it translates a memory write to an address in the 0xFEExxxxx range to an interrupt cycle on the FSB or APIC bus.

If the chipset doesn't know anything about MSI (but I assume it knows about the presence of local APICs) and you configure a PCI device to do interrupts via a message the data you pre-specified will go to the pre-specified in the real physical RAM. You won't get any interrupts from the device. And the data written to memory will be "lost", because you can't access it: The local APIC is memory mapped in this region and memory access coming from the CPU will not go to the physical RAM but to it's APIC circuitry (maybe there will be a FSB transaction, but the chipset won't do a memory acces, because it is aware of the presence of local APICs).

I think they choose to assign this special meaning to the addresses 0xFEExxxxx because the physical memory there isn't used as a consequence of memory mapping the local APICs to this range and that this address "feels" like having something to do with "interrupt stuff" in the broadest sense of it. They could have choosen any other address (range) but, well, this is the way it is and IMHO that isn't the worst choice.

Conclusion: You have to know if the chipset supports MSI or not - and if it does not, don't use MSI. There may be (and there really are) some chipsets that are aware of MSI but don't handle it properly, so for them disable MSI, too.

And now the big question: Am I totally wrong, have I missed something or does it really make sense (I hope so)?

So long
snooky

[Brendan]

Hi,

Conclusion: You have to know if the chipset supports MSI or not - and if it does not, don't use MSI. There may be (and there really are) some chipsets that are aware of MSI but don't handle it properly, so for them disable MSI, too.

And now the big question: Am I totally wrong, have I missed something or does it really make sense (I hope so)?

That makes sense to me.

According to the Linux MSI HOWTO posted by Bluecode (and written by Intel AFAIK), it seems that the PCI device's inbound write is converted to an interrupt message before being sent to CPU/s by the PCI host bridge. Some PCI host bridges don't support MSI (host bridges need to support PCI 2.3 or later), and some PCI to PCI bridges don't support MSI either (which means that even if the device and the PCI host controller do support MSI, MSI still might not work).

This sounds fairly painful to me. The approach used by Linux is "try it and see if it works" - they expect end-users to figure it out, which IMHO is a poor approach (it fails the "I'm a normal user who doesn't know anything about the computer's internals" user-friendliness test).

A better idea may be to write drivers so that they can detect if MSI works or not (e.g. a device driver would enable MSI, force the device to generate an MSI/IRQ, then detect if the MSI/IRQ was received correctly). Once you know one device can send MSIs you know the PCI host controller supports it and any bridges between that device and the host controller (if any) support MSI too; and you could use this information to skip "MSI detection" for subsequent devices on the same PCI bus segment.


Cheers,

Brendan