SMP query

[worldsapart]

Hi,

First of all thank you all for helping me out so far. I have managed to finally boot all the APs and jump them into protected mode [had to drop the idea of doing it with NASM.. GAS now! ]. The CPUs now print out a string to show they're up and then they just spin in an infinite loop for now.I want to work on it further. Here are a few doubts. Hope they dont sound too silly..

1. The APs currently, due to the trampoline code, have a common stack. I should allot different stacks to the APs, rite? So can I do that in the C code before they start to spin, using inline assembly? If so, then I can use the local apic ID register to identify which cpu it is and use that ID to map a different area in memory for stack for each CPU... Please do correct me, if I seem to be going wrong in the thought process. Or is there some other way to do it?

2. As of now the APs jus enter an infinite loop --> while (1) .. I know this wrong.. And it has to be corrected.. What is the usual way of doing this? And is there a way that the processor can be put to sleep and later be able to wake it up using an IPI, instead of just putting it in a loop polling a memory location?

3. My next step would be to setup interrupts. And use IPIs to communicate among processors.. something like a ping-pong for now -> BSP sends IPI to specific APs and it replies with.. say a "Hello World" or something like that... What all should I be careful about and what all needs to be setup before I can do that.

Hope you guys can help.

[Firestryke31]

I can answer question 2:

The assembly instruction "hlt". It waits for an interrupt, then (after the CPU deals with the interrupt) continues code execution. If you want the CPU to not do anything after the interrupt, then put it in a loop that simply executes "hlt" again. Also, the "cli" instruction turns off all maskable interrupts. This means that if you do "cli" then "hlt" you practically lock up the CPU in a low-power state until it gets a non-maskable interrupt. Finally, IIRC an IPI is a non-maskable interrupt.

Basically this does something like what you want:

Code: Select all

while(1)
{
  asm("cli");
  asm("hlt");
  // so domething interesting
}

If you ever want a CPU to be able to handle interrupts other than an IPI (& possibly family, IDK as I've never done MP programming) just leave out the "cli". The "cli" would also be the first thing I would try removing if that doesn't work.

[Brendan]

Hi,

1. The APs currently, due to the trampoline code, have a common stack. I should allot different stacks to the APs, rite? So can I do that in the C code before they start to spin, using inline assembly? If so, then I can use the local apic ID register to identify which cpu it is and use that ID to map a different area in memory for stack for each CPU... Please do correct me, if I seem to be going wrong in the thought process. Or is there some other way to do it?

When you're starting AP CPUs (one at a time), you can allocate some RAM, store the address of this RAM somewhere, then start the AP CPU. The AP CPU can read the address of the allocate area from wherever you put it and use it for it's stack space, then set some sort of flag to let the BSP know that the AP CPU started properly. The BSP would monitor this flag and wouldn't start another AP CPU until the flag is set by the AP CPU (and should probably have a time-out, in case the AP CPU fails to start).

2. As of now the APs jus enter an infinite loop --> while (1) .. I know this wrong.. And it has to be corrected.. What is the usual way of doing this? And is there a way that the processor can be put to sleep and later be able to wake it up using an IPI, instead of just putting it in a loop polling a memory location?

Eventually, what each CPU does will be determined by your scheduler. During boot the scheduler won't be ready yet, so the AP CPUs need to wait until the scheduler is ready. In this case I normally get the AP CPUs to constantly check a flag in RAM. For example:

Code: Select all

.wait:
    pause
    cmp [APflag],0      ;Is there anything to do?
    je .APwait          ; no, keep waiting

    mov eax,[value1]
    mov ebx,[value2]
    call [AProutine]
    lock inc [doneCPUs]
    jmp .APwait

This lets the BSP assign little jobs to AP CPUs during boot. For example:

Code: Select all

    mov [AProutine],APtestRoutine
    mov [value1],foo
    mov [value1],bar
    mov [doneCPUs],1
    mov [APflag],1           ;Make all AP CPUs call "APtestRoutine"

    mov eax,[totalCPUs]
.BSPwait:
    pause
    cmp [doneCPUs], eax
    jne .BSPwait

Of course this could be done with IPIs instead, so that AP CPUs can just do HLT in a loop (and the IPI handler could be used to allow the BSP to ask AP CPUs to execute abitrary code).

3. My next step would be to setup interrupts. And use IPIs to communicate among processors.. something like a ping-pong for now -> BSP sends IPI to specific APs and it replies with.. say a "Hello World" or something like that... What all should I be careful about and what all needs to be setup before I can do that.

For each CPU, make sure the local APIC is enabled and set the interrupt vector for the spurious interrupt handler, setup the logical destination register, set the destination format register and set the task priority register (note: some of this isn't actually necessary, but it's better to make sure - the logical APIC ID and logical destination format register are needed for broadcast IPIs only). You'll also need an IDT with at least one interrupt handler. After that you're ready to send IPIs.

The IPI interrupt handler needs to send an EOI (write a zero to the local APIC's EOI register) at the end of the IPI. Otherwise it's not very different to a software interrupt.


Cheers,

Brendan

[worldsapart]

Thank you very much for the answers. I really appreciate it. I have successfully been able to signal an AP and make it send an IPI to acknowledge receipt of the original IPI. Haven't done exhaustive testing on the code yet. But so far so good. Thanks again. Yet again.. I am still running on Qemu. Can't wait to see what happens on a real system!