How many ISRs installed if multiple MSI enabled ?

[liaoo]

According to PCI 2.2 spec, if multiple MSI is enabled, then a function modifies the lower message data bits to generate the allocated number of messages
For example, a Multiple Message Enable(MME) encoding of “010” indicates the function has been allocated 4 messages and is permitted to modify message data bits 1 and 0.

My question is: how many interrupt service routines are required(installed) in this case ?

If only one ISR(Interrupt Service Routine) is required, then within ISR is it possible to identify which message cause this interrupt

[Brendan]

Hi,

According to PCI 2.2 spec, if multiple MSI is enabled, then a function modifies the lower message data bits to generate the allocated number of messages
For example, a Multiple Message Enable(MME) encoding of “010” indicates the function has been allocated 4 messages and is permitted to modify message data bits 1 and 0.

My question is: how many interrupt service routines are required(installed) in this case ?

You'd need 4 consecutive IDT entries for that case.

Note that this mostly means the kernel needs some sort of "IDT manager", where you can ask it to allocate 'n' consecutive IDT entries at a certain interrupt priority; and if it can't find 'n' IDT entries it may find less instead. For example, if a function supports up to 16 messages but you can only allocate 4 consecutive IDT entries, then you can tell the function it has been allocated 4 messages.

If only one ISR(Interrupt Service Routine) is required, then within ISR is it possible to identify which message cause this interrupt

You can split a function's registers into 2 categories: "generic configuration registers" (in PCI configuration space) and "function specific operational registers" (mapped into the physical address space or assigned IO ports via. the function's BARs). The "generic configuration registers" are designed so that an OS/kernel can configure them without any device specific code (e.g. earlier during boot before any device drivers are started, even if no device driver exists for that device).

A device driver should only ever need to touch the "function specific operational registers". These would include something to indicate why the function has requested service. For example, a function might have a status register with 16 different flags that correspond to 16 different "causes of IRQs".

The device driver may know how "causes of IRQs" are mapped to MSI messages; and (if less than the maximum MSI messages were allocated, including if only 1 was allocated, and including if MSI isn't being used at all) may be able to determine which message would've been sent by the device if the maximum number of MSI message were allocated. Of course typically the device driver has no reason to care (and only uses the "causes of IRQs").


Cheers,

Brendan

[liaoo]

Hi Brendan,
Much thanks for your reply.

Can I say that either allocating 4 consecutive IDT entries or just installing one if hardware implements "function specific operational registers" which helps the ISR with the discrimination between multple MSI messages can solve this ?

Besides, if we use MSI-X then the above is still true or not ? ( 4 IDT entries required or just 1 )

[Brendan]

Hi,

Can I say that either allocating 4 consecutive IDT entries or just installing one if hardware implements "function specific operational registers" which helps the ISR with the discrimination between multple MSI messages can solve this ?

I'm not sure what you're trying to solve - neither the OS nor the driver needs to discriminate between multiple MSI messages.

Besides, if we use MSI-X then the above is still true or not ? ( 4 IDT entries required or just 1 )

For MSI-X, you still don't need to allocate all IDT entries that the function requested. It is more flexible though, as the allocated interrupts don't need to be consecutive for MSI-X and you can allocate a "non-power of 2" number of interrupts. Also note that (at least in theory) a function can request up to 2048 interrupts using MSI-X, and for 80x86 (where there's only 256 IDT entries) it'd be impossible to allocate the requested number (at least not without having different IDTs for different CPUs).


Cheers,

Brendan

[liaoo]

Let me describe the question in more detail first...

I got a trace file(captured via one protocol analyzer, OS unknown) and we can see how host initializes and access device.
The device supports multiple MSI and the MSI settings are: MMC=3, MME=3, MA = FEExxxxx, MD = xxA8, MSIE=1. Besides, we can see total 8 IVs(Interrupt vector) are assigned to device(0,1,2,3,..,7)

If we just check the interrupt messages device issued in the trace file then we can see below:

• - memory write to FEExxxxx with Data xxA8
  - memory write to FEExxxxx with Data xxA8
  - ...
  - memory write to FEExxxxx with Data xxA9
  - memory write to FEExxxxx with Data xxAC
  - ...
  - memory write to FEExxxxx with Data xxAD

Thus it is true that a function modifies the lower message data bits to generate the allocated number of messages. Because 8 IVs are assigned to the device I "guess" there should be 8 ISRs are allocated/installed for this device, am I right ?

Now my question is: Is it possible that we just install only 1 ISR for this device ?
Ex. within ISR, sw first identify the cause of MSI message and then execute accordingly.

The problem is: How to identify which cause ? (What we can access within ISR is the device PCI config registers, including MSI capability registers)

I was even wondering if sw can access local apic registers to identify the interrupt "cause" (because MSI messages are received by local apic...)

* Finally I have to say this task(servicing multiple MSIs) shall be executed in DOS, not Windows or Linux. Thus no APIs or system call can be invoked...

[rdos]

There are not a lot of devices where it is useful to use more than one MSI interrupt. The typical example is AHCI disc drivers where you can get one MSI per disc, which simplifies the detection algorithm of which disc caused the interrupt.

I don't think there is a way to detect which MSI caused the interrupt. You will need to either poll hardware for the cause (which is a bad idea with AHCI since then you need to check all discs, not just the one causing the interrupt), or have separate entry-points for the handlers.

Another advantage of using multiple MSIs is that you can spread interrupts over several cores.

[rdos]

Note that this mostly means the kernel needs some sort of "IDT manager", where you can ask it to allocate 'n' consecutive IDT entries at a certain interrupt priority; and if it can't find 'n' IDT entries it may find less instead. For example, if a function supports up to 16 messages but you can only allocate 4 consecutive IDT entries, then you can tell the function it has been allocated 4 messages.

I wouldn't do that. Even if the specification allows you to allocate any number of messages between 1 and what the function supports, you only want to allocate 1 or the number the function supports. This is because in the cases where multiple messages are supported, it also has a special function so you know which part of the function requested service simply by knowing which message it sent. When this advantage is lacking, there is no reason to allocate more than 1 message. It's usually undefined what function is requesting service if you allocated less messages than cases in the driver. For instance, if you have 5 SATA-drives connected to an AHCI controller that supports multiple messages, and you allocate only 4 messages, you don't know which device requests service, and instead need to check all 5 drives. That means you gain no benefit by using 4 messages over using a single message.

[Brendan]

Hi,

* Finally I have to say this task(servicing multiple MSIs) shall be executed in DOS, not Windows or Linux. Thus no APIs or system call can be invoked...

For DOS, you mostly have to use the PIC chips to avoid breaking everything (using both PIC and IO APIC at the same time is dodgy); which means that you shouldn't be using MSI at all. If the function doesn't support the older PCI IRQs (where each function is "hard-wired" to one PCI IRQ that is likely shared by many PCI devices) then you should poll without using any IRQs.


Cheers,

Brendan

[Brendan]

Hi,

Even if the specification allows you to allocate any number of messages between 1 and what the function supports, you only want to allocate 1 or the number the function supports. This is because in the cases where multiple messages are supported, it also has a special function so you know which part of the function requested service simply by knowing which message it sent. When this advantage is lacking, there is no reason to allocate more than 1 message. It's usually undefined what function is requesting service if you allocated less messages than cases in the driver. For instance, if you have 5 SATA-drives connected to an AHCI controller that supports multiple messages, and you allocate only 4 messages, you don't know which device requests service, and instead need to check all 5 drives. That means you gain no benefit by using 4 messages over using a single message.

There's 3 choices:

• Use one IRQ, have to poll the hardware for the cause in the ISR, and be unable handle different interrupts in parallel on different CPUs
• Use more than one and less than the maximum number of IRQs, be able handle different interrupts in parallel on different CPUs, and have to poll the hardware for the cause in the ISR
• Use the maximum number of IRQs, be able to handle different interrupts in parallel on different CPUs, and avoid the need to poll for the cause

If it is possible to allocate the maximum number of IRQs, then allocating the maximum number of IRQs is always the best possible choice.

If it's impossible to allocate the maximum number of IRQs, then allocating less than the maximum is better than only allocating one if there's more than one CPU (very likely for modern hardware).

Only allocating one only makes sense as a last resort (when neither of the other options are better).


Cheers,

Brendan

[liaoo]

Hi,

For DOS, you mostly have to use the PIC chips to avoid breaking everything (using both PIC and IO APIC at the same time is dodgy); which means that you shouldn't be using MSI at all. If the function doesn't support the older PCI IRQs (where each function is "hard-wired" to one PCI IRQ that is likely shared by many PCI devices) then you should poll without using any IRQs.

I did not know exactly why MSI can't be used in DOS mode. Could you explain in more detail ?

* My last project mainly focus on accessing sata drive via AHCI in DOS and 3 modes are supported(polling, pin-based, and single MSI)

* For multiple MSIs, the main question shall be how many "consecutive" interrupt vectors can be allocated in DOS. But for MSI-X s/w can config "independent" message/data thus it will be easier than in MSI case(I guess...)

[rdos]

Hi,

For DOS, you mostly have to use the PIC chips to avoid breaking everything (using both PIC and IO APIC at the same time is dodgy); which means that you shouldn't be using MSI at all. If the function doesn't support the older PCI IRQs (where each function is "hard-wired" to one PCI IRQ that is likely shared by many PCI devices) then you should poll without using any IRQs.

I did not know exactly why MSI can't be used in DOS mode. Could you explain in more detail ?

* My last project mainly focus on accessing sata drive via AHCI in DOS and 3 modes are supported(polling, pin-based, and single MSI)

* For multiple MSIs, the main question shall be how many "consecutive" interrupt vectors can be allocated in DOS. But for MSI-X s/w can config "independent" message/data thus it will be easier than in MSI case(I guess...)

I think MSI and multiple MSIs should work in DOS even when using the PIC. MSI sends messages directly to the processor cores, without going through the interrupt controller, and these interrupts should be reflected to the corresponding real-mode interrupt vectors.

[Brendan]

Hi,

For DOS, you mostly have to use the PIC chips to avoid breaking everything (using both PIC and IO APIC at the same time is dodgy); which means that you shouldn't be using MSI at all. If the function doesn't support the older PCI IRQs (where each function is "hard-wired" to one PCI IRQ that is likely shared by many PCI devices) then you should poll without using any IRQs.

I did not know exactly why MSI can't be used in DOS mode. Could you explain in more detail ?

For DOS, nothing protects anything from anything and the only thing that keeps it running properly is that everything expects (and maintains the illusion of) "legacy behaviour". As soon as you do something that isn't "legacy behaviour" you risk breaking applications and the OS itself.

For a simple example, an application (or a "DOS extender" being used by an application) may switch between real mode and protected mode; and setup its own protected mode IDT with ISRs for the 16 PIC IRQs that switch back to real mode and pass control to the normal real mode interrupt vectors or run the IRQ handlers in virtual8086 mode; where your driver will cause the application to crash (because the application expects that IRQs can only come from the PIC, but your MSI IRQ breaks that expectation).

For another example; an application and/or DOS itself ("himem.sys") can setup "unreal mode" and expect this to remain setup. To be able to send EOI your ISR will have to access the local APIC by switching to/from protected mode and will unexpectedly destroy "unreal mode" while other software is using it.

* For multiple MSIs, the main question shall be how many "consecutive" interrupt vectors can be allocated in DOS. But for MSI-X s/w can config "independent" message/data thus it will be easier than in MSI case(I guess...)

All IVT entries are either:

• used by BIOS or DOS interrupt handlers (including PIC IRQ handlers and APIs), or
• used by BIOS or DOS to store data that has nothing to do with interrupt handling at all (e.g. the IVT entry for "int 0x1D" contains the address of a video parameter table), or
• are free for TSRs to use for software APIs

Note: you can "steal" any IVT entries that aren't being used to store data; as long as your ISR detects if the interrupt was caused by your MSI (e.g. check the local APIC's "in service register") and passes control to the previous interrupt handler if the interrupt was not caused by your MSI. However, applications and TSRs that are started after your driver can also "steal" IVT entries and if it's not a standard/known interrupt the application may not bother passing control to your interrupt handler.

Also note that to use MSI you must enable the local APIC, and if you enable the local APIC you have to be prepared to handle spurious IRQs from the local APIC. This means that if you setup MSI for one interrupt vector, then you need 2 interrupt vectors (one for MSI and one for local APIC spurious IRQ).


Now; mostly what I'm trying to say with all the above is that it's dodgy/risky and virtually impossible to get everything 100% reliable in all cases (and the harder you try the more complex things get). On the other side, DOS/BIOS is a single-tasking system where only one thing happens at a time. You won't be able to support multiple reads/writes "in flight" at the same time because the APIs don't allow that to happen (they wait for one transfer to complete before any other transfer can begin); and without multiple reads/writes "in flight" there's no point caring about handling multiple IRQs from the function in parallel. Basically, you won't gain anything by using MSI (except more complexity and worse reliability). To make MSI worth bothering with, you need an OS that doesn't let applications do anything they like and doesn't kill any performance gains.


Cheers,

Brendan