Dynamically starting cores

[rdos]

It seems like it is not reliable to let AP cores stay "stopped" in BIOS. I have one machine that just refuses to boot when only BSP is booted. The reason for this seems to be missing interrupts. I use "lowest priority" delivery for interrupts. If I boot the AP core, but let it "hibernate" by doing this, it works:

Code: Select all

    cli

wait_for_start:
    test fs:ps_flags,PS_FLAG_ACTIVE      ; active flag is 0
    jz wait_for_start

Seems like BIOS in this system doesn't put the core to sleep in the proper manner. Perhaps they forget to disable interrupts?

IOW, it seems like it is necessary to boot all AP cores, and then put them to sleep in the proper manner until they are needed.

[Brendan]

Hi,

Seems like BIOS in this system doesn't put the core to sleep in the proper manner. Perhaps they forget to disable interrupts?

IOW, it seems like it is necessary to boot all AP cores, and then put them to sleep in the proper manner until they are needed.

There's isn't enough information here to form a conclusive conclusion. It's like saying "I'm wet, therefore it must be raining" - the assumption might be correct, but then again you could be wet for a completely different reason.

Normally (as I understand it), CLI has no effect on whether or not the local APIC accepts an IRQ. CLI only effects if/when the local APIC delivers a received interrupt to the CPU. For example, with interrupts disabled the local APIC can receive 20 different IRQs (and set the corresponding flags in its "Interrupt Received Register") and then wait until the CPU enables interrupts again. When the CPU does enable interrupts again the local APIC delivers the previously received IRQs to the CPU in order (highest priority IRQ first).

From this I'd assume that the BIOS probably does disable IRQs; and the problem is that the AP CPU's local APIC receives "lowest priority delivery" IRQs anyway. This is entirely possible if the AP CPU's "Logical Destination Register" is non-zero. I also have a suspicion that if the interrupt was sent with "destination = 0xFF" then it's treated as "all CPUs" regardless of whether it's "fixed mode" or "logical mode", and regardless of the local APIC's "Logical Destination Register". Note: This definitely is the case for x2APIC, but I'm not sure about xAPIC (including CPUs that support x2APIC but are running in "xAPIC mode").

I don't know if you program the IO APIC with "destination = 0xFF", but if you do, don't.

You also don't say if this behaviour occurs on a cold boot, or if it only occurs after a soft reset. For cold boot, the default contents of the local APIC's logical destination register is zero, and in that case it shouldn't accept "lowest priority delivery" IRQs at all (regardless of whether the BIOS does "CLI" or not). Perhaps your OS boots and sets the logical destination register, then you reset the computer and the BIOS doesn't clear the logical destination register, which leaves it non-zero and able to accept "lowest priority delivery" interrupts (regardless of whether the BIOS does "CLI" or not).

At a minimum, I'd be tempted to start the AP CPU and display the contents of all of its local APIC registers to see what the BIOS left in each of them. I guess you should also display the contents of the AP CPU's EFLAGS at startup too (but that shouldn't matter much).


Cheers,

Brendan

[rdos]

Normally (as I understand it), CLI has no effect on whether or not the local APIC accepts an IRQ. CLI only effects if/when the local APIC delivers a received interrupt to the CPU. For example, with interrupts disabled the local APIC can receive 20 different IRQs (and set the corresponding flags in its "Interrupt Received Register") and then wait until the CPU enables interrupts again. When the CPU does enable interrupts again the local APIC delivers the previously received IRQs to the CPU in order (highest priority IRQ first).

There is just one problem with this. When I send an INI, and let it boot up into my kernel, and then freeze it with cli/hlt, it doesn't affect the system boot in any way. That is, this could not be the reason why the "freeze" in BIOS seems to "steal" interrupts from the timer (PIT).

From this I'd assume that the BIOS probably does disable IRQs; and the problem is that the AP CPU's local APIC receives "lowest priority delivery" IRQs anyway. This is entirely possible if the AP CPU's "Logical Destination Register" is non-zero. I also have a suspicion that if the interrupt was sent with "destination = 0xFF" then it's treated as "all CPUs" regardless of whether it's "fixed mode" or "logical mode", and regardless of the local APIC's "Logical Destination Register". Note: This definitely is the case for x2APIC, but I'm not sure about xAPIC (including CPUs that support x2APIC but are running in "xAPIC mode").

I didn't reprogram the APIC before doing cli/hlt, so it should have the same state as in BIOS.

I don't know if you program the IO APIC with "destination = 0xFF", but if you do, don't.

I do. What would I program it with otherwise?

Besides, I think a reasonable action is to load TPR with the maximum value before putting the core into C3 state, and doing cli hlt. Then the APIC should no longer accept any interrupts, "lowest delivery" or other.

[Brendan]

Hi,

Normally (as I understand it), CLI has no effect on whether or not the local APIC accepts an IRQ. CLI only effects if/when the local APIC delivers a received interrupt to the CPU. For example, with interrupts disabled the local APIC can receive 20 different IRQs (and set the corresponding flags in its "Interrupt Received Register") and then wait until the CPU enables interrupts again. When the CPU does enable interrupts again the local APIC delivers the previously received IRQs to the CPU in order (highest priority IRQ first).

There is just one problem with this. When I send an INI, and let it boot up into my kernel, and then freeze it with cli/hlt, it doesn't affect the system boot in any way. That is, this could not be the reason why the "freeze" in BIOS seems to "steal" interrupts from the timer (PIT).

The INIT IPI causes the CPU to reset to default values.

I overlooked this in my last post - it means you won't be able to start the AP and see what was in it's local APIC registers before you sent the INIT IPI.

I don't know if you program the IO APIC with "destination = 0xFF", but if you do, don't.

I do. What would I program it with otherwise?

If there's less than 8 CPUs, then maybe 0x01, 0x03, 0x07, 0x0F, 0x1F, 0x3F or 0x7F. If there's more than 8 CPUs then the "one bit for each CPU" idea doesn't work anyway; so if there's 8 or more CPUs you could switch to something that isn't "one bit for each CPU" (e.g. "one bit for each group of IRQs" or something). Of course if there's lots of CPUs then it's fairly likely it's a modern system that supports x2APIC, where switching from "xAPIC mode" to "x2APIC mode" is probably a good idea anyway.

Besides, I think a reasonable action is to load TPR with the maximum value before putting the core into C3 state, and doing cli hlt. Then the APIC should no longer accept any interrupts, "lowest delivery" or other.

I wouldn't be too surprised if some BIOSs don't put the AP CPUs in C3 before stopping them, where doing it yourself saves a little power. Setting the TPR to 0xFF will only stop "lowest priority delivery" IRQs (unless all other CPUs have TPR = 0xFF too, making all CPUs the same priority).

You still didn't say if the problem occurs during cold boot, or if it only occurs when the computer does a warm boot/reset. I know that reset doesn't do a full reset in some cases. I've even seen "messed up local APIC" cause the BIOS to assume a CPU was faulty during POST, where the reset switch didn't help at all and I had to remove power completely to bring the CPU back.

If the problem only happens when the computer does a warm boot/reset; then it's very likely that the BIOS doesn't reset everything possible and you end up with left over values from before the reboot still left in the local APIC causing problems.


Cheers,

Brendan

[rdos]

If there's less than 8 CPUs, then maybe 0x01, 0x03, 0x07, 0x0F, 0x1F, 0x3F or 0x7F. If there's more than 8 CPUs then the "one bit for each CPU" idea doesn't work anyway; so if there's 8 or more CPUs you could switch to something that isn't "one bit for each CPU" (e.g. "one bit for each group of IRQs" or something). Of course if there's lots of CPUs then it's fairly likely it's a modern system that supports x2APIC, where switching from "xAPIC mode" to "x2APIC mode" is probably a good idea anyway.

I would probably leave it "as is" with xAPICs, and change it for newer CPUs that support x2APIC mode. Initially, I thought many processors actually supported x2APIC mode, but then I found out this was not so, and thus I removed this code to simplify the APIC device-driver. I'll probably build a new x2APIC device-driver instead for systems that support that instead of introducing variants in the present APIC module. As long as the processor doesn't have more than 8 cores, I really don't need x2APIC anyway.

I wouldn't be too surprised if some BIOSs don't put the AP CPUs in C3 before stopping them, where doing it yourself saves a little power. Setting the TPR to 0xFF will only stop "lowest priority delivery" IRQs (unless all other CPUs have TPR = 0xFF too, making all CPUs the same priority).

Yes, but I deliver all IO APIC interrupts and MSI interrupts in "lowest priority mode", so that should include everything except IPIs.

You still didn't say if the problem occurs during cold boot, or if it only occurs when the computer does a warm boot/reset. I know that reset doesn't do a full reset in some cases. I've even seen "messed up local APIC" cause the BIOS to assume a CPU was faulty during POST, where the reset switch didn't help at all and I had to remove power completely to bring the CPU back.

I think both. Besides, the soft RESET usually is a tripple-fault which should trigger a RESET signal. Don't know if that signal does a real cold reboot or not though.

[rdos]

The status of dynamically starting cores and adjusting P-states is that it works on 4 machines (6-core AMD, 4-core AMD, the 2-core AMD Athlon I, and my Compaq CQ57 portable with a 2-core AMD). It doesn't work anymore one the Intel Core-Duo based Vaio portable, but it probably was broken earlier than this. The AMD Geode processor doesn't seem to support P-states, and also only has a single core, so nothing happens there.

Yesterday I managed to load all 6-cores on my new AMD Phenom II computer to 100% by running 6 different processes that outputted random graphics. Everything seems to work perfectly well there now. It started with a single core at lowest P-state (800MHz), and ended up with all 6 cores running at 2800MHz with 100% load.

[rdos]

Now I can also shutdown cores as load decreases. It's actually a lot easier to start a new core than to shut it down. The shutdown must occur when there is no scheduling activities, and it must make sure no interrupts are accepted after shutdown is entered. I detect it in null-thread (which garantees there are no pending threads), and then save context of null-thread and prepare for a normal schedule. However, instead of loading a new thread, I put the core in low-power state. This currently only works if timers are global (HPET or PIT). It doesn't work with APIC timers, as those would also become shutdown.

I changed the logic so now a NMI IPI is used to wakeup / start a new core. It is a lot simpler, faster (and safer) than doing a full INI sequence. Instead, the INI sequence is only done at boot, and then the AP cores are immediately shutdown until needed.