IPI Hardly works

[MollenOS]

So, I'm doing a lot of SMP tests in my kernel, and it seems my IPI's are barely working.

Cases where my IPI works:
- When a core sends an IPI to itself, but only if not used while in an interrupt handler.

Cases where my IPI does not work:
- When a core sends an IPI to another core (it may actually even lock up afterwards).

Anyone know any possible causes for such troublesome interrupts?
The vector i try to send is my software-yield interrupt, which sends an EOI in it's interrupt handler.
I would like to add, the core it sends to is just idling (in a halt-loop with interrupts enabled). The IPI is trying to wake it up by inducing a Yield-interrupt.

My IPI code:

Code: Select all

               /* Setup flags
		* Logical Destination
		* Fixed Delivery
                * Assert
		* Edge Sensitive */
		IpiLow |= IrqVector;
		IpiLow |= (1 << 11);
		IpiLow |= (1 << 14);

		/* Setup Destination */
		IpiHigh |= ((ApicGetCpuMask(CpuTarget) << 24));

		/* Wait */
		ApicWaitForIcr();

		/* Disable interrupts */
		IntStatus_t IntStatus = InterruptDisable();

		/* Write upper 32 bits to ICR1 */
		ApicWriteLocal(APIC_ICR_HIGH, IpiHigh);

		/* Write lower 32 bits to ICR0 */
		ApicWriteLocal(APIC_ICR_LOW, IpiLow);

		/* Enable */
		InterruptRestoreState(IntStatus);

[Brendan]

Hi,

I have no idea what "ApicGetCpuMask(CpuTarget)" does, and no idea how you've configured each local APIC's "Logical Destination Register". This makes it impossible to determine if any CPU will claim the IPI (or if multiple CPUs will try to claim the IPI).

The "ApicWaitForIcr()" needs a timeout, so that you don't wait forever if something goes wrong.

Immediately after "ApicWaitForIcr()" and before "InterruptDisable();" an interrupt can occur and send an IPI, which means that there's no guarantee that the Delivery Status flag is clear when you're sending a new IPI. I'd have a "ApicWaitForIcrAndDisableIRQs()" that's sort of like this:

Code: Select all

    while(true) {
        if( Delivery Status flag is clear ) {
            IntStatus = InterruptDisable();
            if( Delivery Status flag is still clear ) {
                 return IntStatus;
            } else {
                InterruptRestoreState(IntStatus);
            }
        } else {
            if(time passed > too_long) {
                return NULL;   // timeout (maybe kernel panic?)
            }
        }
    }

Note that I'm not convinced Intel's advice is sane. Checking Delivery Status before sending an IPI (to make sure the previously sent IPI actually has been sent) is better for performance in theory maybe; but in most cases you need some sort of feedback (where the receiving CPU/s let you know they've invalidated a TLB, or accessed some data in memory that goes with the IPI, or whatever; so that the sending CPU knows its safe to continue), and therefore in most cases you have to wait until after the IPI is received anyway so it'd make more sense to check Delivery Status after sending an IPI (and not before sending the next IPI). Also note that if there's a problem (e.g. you get a time out because Delivery Status never becomes clear), displaying something like "Kernel Panic: Failed to send TLB shootdown IPI" is more fun for the person trying to debug the problem than "Kernel Panic: Failed to send something (I have no idea what)".


Cheers,

Brendan

[MollenOS]

I have no idea what "ApicGetCpuMask(CpuTarget)" does, and no idea how you've configured each local APIC's "Logical Destination Register". This makes it impossible to determine if any CPU will claim the IPI (or if multiple CPUs will try to claim the IPI).

Sorry Brendan, my fault for lack of information. In this particular case I have 4 cores running, each with following destination settings:

Destination Format: 0xFFFFFFFF
Logical Destination: 0x0100000 (cpu 0) or 0x0200000 (cpu 1) or 0x0400000 (cpu 2) or 0x0800000 (cpu 3)

This is what ApicGetCpuMask does, it simply just get's the bit for the cpu it wants to target, in this particular instance, it gets 0x0100000 since my Core 3 is trying to send IPI to Core 0.
I have tried to validate that the correct values get written, and it seems to check-out, but Core 0 never recieves the IPI.

The "ApicWaitForIcr()" needs a timeout, so that you don't wait forever if something goes wrong.

Yes indeed, it should have a timeout, but in this case it never gets stuck in that loop, since I have printf's both before and after trying to track down the issue

Generally I'm having issues trying to get the local apic to deliver my "apic" type interrupts (Other interrupts works just fine, both PCI and my hpet interrupts works). My Apic timer work's pretty well, as long as i reset the timer in my apic-timer interrupt handler, but if i try to reset it outside of the handler (i use it in one-shot mode), I have to do a (while (ApicReadLocal(TIMER_CURRENT) > 0);) before the interrupt actually fires, if I don't wait for it, it doesn't fire the interrupt.

And i'm testing on real hardware with an AMD phenon cpu if this helps any.

[Brendan]

Hi,

Sorry Brendan, my fault for lack of information. In this particular case I have 4 cores running, each with following destination settings:

Destination Format: 0xFFFFFFFF
Logical Destination: 0x0100000 (cpu 0) or 0x0200000 (cpu 1) or 0x0400000 (cpu 2) or 0x0800000 (cpu 3)

That's fairly standard I guess (at least I think that's what Intel describes).

Note that this can't work when there's more than 8 CPUs, and there's more flexible ways. For some random examples:

• You could have a "CPUs that aren't asleep" bit so you can broadcast an IPI to all CPUs that aren't asleep, and when a CPU goes to sleep (to save power) or wakes up again you change its logical destination.
• You could have a "first CPU in this NUMA domain" bit; so you can broadcast an IPI to one CPU in each NUMA domain (which can be nice if you're using a lot of "per NUMA domain" data structures for things like memory management and scheduling).
• You could have a "last CPU in this core" bit; so if you're updating anything that's shared by all logical CPUs in a core (e.g. power management, MTRRs, etc) you only interrupt CPUs that need to be interrupted
• You could use 4 bits for "current process ID & 0x0F" (and change a CPU's logical destination on task switches if necessary); so that when you do TLB shootdown (after a process' virtual address space was changed) you don't need to interrupt CPUs that can't be running the same process anyway. This could reduce the TLB shootdown overhead by a significant amount.

This is what ApicGetCpuMask does, it simply just get's the bit for the cpu it wants to target, in this particular instance, it gets 0x0100000 since my Core 3 is trying to send IPI to Core 0.

If you're only sending to a single specific CPU; it'd make more sense to use physical delivery instead of logical delivery. Logical delivery should still work though.

I have tried to validate that the correct values get written, and it seems to check-out, but Core 0 never recieves the IPI.

Have you checked Core 0's Task Priority Register? If TPR is higher than the vector then the CPU won't accept the interrupt (it'll be "pending" until TPR is low enough).

My Apic timer work's pretty well, as long as i reset the timer in my apic-timer interrupt handler, but if i try to reset it outside of the handler (i use it in one-shot mode), I have to do a (while (ApicReadLocal(TIMER_CURRENT) > 0);) before the interrupt actually fires, if I don't wait for it, it doesn't fire the interrupt.

That makes me think you're leaving interrupts disabled (e.g. "CLI") somewhere, or something else is going wrong. If you don't wait for local APIC timer count to reach zero; what is the OS doing when the local APIC timer is supposed to happen?


Cheers,

Brendan

[MollenOS]

That's fairly standard I guess (at least I think that's what Intel describes).

Note that this can't work when there's more than 8 CPUs, and there's more flexible ways. For some random examples:
You could have a "CPUs that aren't asleep" bit so you can broadcast an IPI to all CPUs that aren't asleep, and when a CPU goes to sleep (to save power) or wakes up again you change its logical destination.
You could have a "first CPU in this NUMA domain" bit; so you can broadcast an IPI to one CPU in each NUMA domain (which can be nice if you're using a lot of "per NUMA domain" data structures for things like memory management and scheduling).
You could have a "last CPU in this core" bit; so if you're updating anything that's shared by all logical CPUs in a core (e.g. power management, MTRRs, etc) you only interrupt CPUs that need to be interrupted
You could use 4 bits for "current process ID & 0x0F" (and change a CPU's logical destination on task switches if necessary); so that when you do TLB shootdown (after a process' virtual address space was changed) you don't need to interrupt CPUs that can't be running the same process anyway. This could reduce the TLB shootdown overhead by a significant amount.

Yea i know it's going to be a problem the moment I have more than 8 CPU's. I use "lowest priority" interrupts in the io-apic so I'm not sure how I should program the logical destination register if I want to maintain this mode with more than 8 CPU's. I like the idea about changing the logical id on task-switches.

f you're only sending to a single specific CPU; it'd make more sense to use physical delivery instead of logical delivery. Logical delivery should still work though.

The weird thing though, is the moment I use physical delivery instead, I can't even seem to interrupt the same core any more. I just tried and Core 0 won't even send the IPI to itself when using physical mode with the Apic Id of 0.

Have you checked Core 0's Task Priority Register? If TPR is higher than the vector then the CPU won't accept the interrupt (it'll be "pending" until TPR is low enough).

Im going to recheck later if it truly is set correct, but my core's set their TPR to 0 when they enter the "idle" task and write 0 to their Apic Timer Initial Count register.

That makes me think you're leaving interrupts disabled (e.g. "CLI") somewhere, or something else is going wrong. If you don't wait for local APIC timer count to reach zero; what is the OS doing when the local APIC timer is supposed to happen?

It simply enters a HLT loop after enabling the timer again (with interrupts enabled), since it's the end of the "init" task that get's converted to an idle task at the end.

And thanks for all the feedback Brendan, I feel like im running my head against a wall with this issue.

[MollenOS]

Ok, so now everything works perfectly, and I'm not even sure what changed, but thanks again Brendan! I played around with when I sent the IPI and now it works.