Problems activating SMP on newer (core i5/i7) machines

[IanSeyler]

Does anyone else have issues with activating the AP's in newer computers? There are issues with Pure64 when run on core i5/i7 machines.

I think the issue is the delays that are required between the IPI's but I can't pinpoint it. The delay_EAX_microseconds function used using works fine on older machines but on a Core i5 or i7 it seems to lock the computer.

Any ideas?

wakeup code:
http://returninfinitylabs.svn.beanstalk ... it_smp.asm

Thanks,
-Ian

[xenos]

I had the experience that some newer machines seem to require only one Startup IPI instead of two as written in the multiprocessor specification. In the case of my kernel, they were started up by the first Startup IPI, walked into protected mode / long mode, received a second Startup IPI, jumped right back into the real mode startup code and crashed because they were still operating in protected / long mode. I fixed this by checking whether the AP has already booted up before sending the second Startup IPI.

[Brendan]

Hi,

There are issues with Pure64 when run on core i5/i7 machines.

I never had any problem with Nehalem CPUs (Xeon specifically, but they're all basically the same) - the AP startup code I've been using for ages worked fine.

If you search through the forums for "AP startup" and "Brendan" you'll probably find half a dozen of my posts that all repeat a few basic recommendations. You've managed to do the opposite of all my recommendations, so here they are:

1) Get the address of the local APIC from the MP specification tables or ACPI tables (and not from an MSR and definitely don't assume it's at 0xFEE00000). Just because Intel felt like calling the "APIC base MSR" an architectural MSR, it doesn't mean that older CPUs support it (Pentium or older CPUs don't), or that other CPU manufacturers also consider it architectural, or that no CPU manufacturer will break it future. Ironically, Intel was the first CPU manufacturer to break this when they introduced x2APIC.

2) Don't use "broadcast to all but self". It has several problems, including:

• For Netburst CPUs with hyper-threading, if hyper-threading is disabled in the BIOS you start the disable CPUs anyway. Note: Hyper-threading in Nehalem CPUs works a bit differently, because it can be totally disabled (rather than "pretend disabled" like in Netburst CPUs). I don't know if it's "safe" to broadcast the INIT/SIPI sequence in all cases on Nehalem (for e.g. it's possible for BIOS/chipset bugs to cause strange behaviour that nobody noticed because no sane OS broadcasts the INIT/SIPI sequence).
• If the firmware detected that some CPUs were faulty (and the firmware makes sure these faulty CPUs aren't listed in the MP specification tables and ACPI tables to make sure the OS doesn't attempt to use them), then using "broadcast to all but self" will attempt to start/use faulty CPUs.
• There's no way to tell if CPUs that should've started actually did start (for e.g. if there's 7 AP CPUs and only 5 of them start correctly, then your code doesn't notice that there's been any problems)
• I wouldn't trust the "delivery status" bit in the ICR for broadcast IPIs of any kind - it may only say if the IPI was sent correctly and if at least one CPU received it, and might reliably determine if all CPUs (rather than just one) received the IPI/s.
• It breaks backward compatibility for X2APIC (you can accidentally start CPUs that are in "x2APIC mode" that your OS can't handle)

3) The INIT/SIPI/SIPI startup sequence described by Intel (which is what you're using) is a pain in the neck - it requires precise timing (which typically isn't easily available in a standard way during early boot), it's possible for the AP CPU to start executing code after the receiving the first SIPI but before the receiving second SIPI, and it's slower (often the full 200 us delay isn't necessary). Don't use it. It's better to send the INIT IPI, then do the first delay (at least 10 ms, but longer seems fine), then send the first SIPI, then wait to see if the AP CPU starts or not (with a time-out that can be a lot longer than the 200 us that Intel say). If the first SIPI was enough to start the AP CPU then you're done (don't send the second SIPI at all). Otherwise, send the second SIPI then wait to see if the AP CPU starts or not (with a time-out that can be much longer than the 200 us that Intel say) and fail the CPU if it doesn't start after the second SIPI. In this case you will need some synchronisation between the AP CPU and the BSP to avoid race conditions (e.g. AP CPU sets a flag to tell the BSP it started correctly, and when the BSP notices this flag is set then the BSP set a flag to tell the AP CPU it can continue).

Also note that the way I do the AP CPU startup doesn't require precise timing - setting the PIT to 1000 Hz and using time-outs of "between 1 ms and 2 ms" for the SIPI delays is fine.

About x2APIC....

Nehalem CPUs support x2APIC; where the local APIC has a 2 modes - the "xAPIC" mode (for backward compatibility) that you're using, and the new "x2APIC" mode. When a CPU is using the x2APIC mode the local APIC's registers are accessed via. MSRs instead of being mapped into the physical address space, and there is no "local APIC base address". When a computer has lots of CPUs, some CPUs have APIC IDs that are larger than 256 and can't be represented by the 8-bit APIC IDs that used in "xAPIC" mode. In this case the firmware is required to put those CPUs into "X2APIC mode", and create different entries in the ACPI tables for these CPUs. This means that a normal OS that doesn't support x2APIC and only understands the normal ACPI entries with 8-bit APIC IDs won't start CPUs that need 32-bit APIC IDs and will work correctly (just those extra CPUs won't be used). By broadcasting the INIT/SIPI/SIPI sequence you bypass this and you'd start any CPUs that are in x2APIC mode, even if your OS can't handle CPUs that are in x2APIC mode.


Cheers,

Brendan

[IanSeyler]

Hello Brendan,

Thanks for taking the time to respond to this.

1) Pure64 targets 64-bit PC's only so the MSR will be there but I see your point. I'll stick to getting the APIC Base Address from the MP or ACPI tables. From reading one of your other posts you mentioned that all cores might not be included in the MP tables. Is this the case? Also it seems that newer PC's don't even have a MP table so it seems that ACPI is the way to go.

2) I'll update Pure64 to enable the AP's one-by-one. After parsing the MP tables you get all of the APIC ID's anyways. Are the APIC ID's also in the ACPI tables?

3) I'll dump the second SIPI. Like I mentioned earlier I am only targeting 64-bit computers so I don't have to deal with legacy hardware.

Thanks again,
-Ian

[Brendan]

Hi,

1) Pure64 targets 64-bit PC's only so the MSR will be there but I see your point. I'll stick to getting the APIC Base Address from the MP or ACPI tables. From reading one of your other posts you mentioned that all cores might not be included in the MP tables. Is this the case? Also it seems that newer PC's don't even have a MP table so it seems that ACPI is the way to go.

All cores that have an 8-bit APIC ID and aren't faulty should be listed in MP specification tables. For SMT (where there's multiple logical CPUs per core - e.g. hyper-threading), only one logical CPU per core should be listed in the MP specification tables. This is mostly for backward compatibility, to prevent old OSs that aren't designed to use hyper-threading efficiently (e.g. PAUSE in spinloops, etc) from using those extra logical CPUs.

In all cases, if ACPI tables exist (and pass checksums, etc) you should use the ACPI tables; and you should only ever use MP specification tables as a fall-back (when the ACPI tables can't be used).

2) I'll update Pure64 to enable the AP's one-by-one. After parsing the MP tables you get all of the APIC ID's anyways. Are the APIC ID's also in the ACPI tables?

The main thing is to avoid "broadcast to all but self". For large computers (with lots of CPUs) I'd be tempted to consider some sort of parallel startup. For an example, if there's 128 CPUs and it takes 11 ms to start each CPU, then it'll add up to 127*11 = 1397 ms to start all of them one at a time. However, the BSP could start the first CPU, then both these CPUs could start 2 more CPUs, then the 4 CPUs could start 4 more CPUs, etc. In this case, with 128 CPUs and 11 ms per CPU it only takes 7*11 = 77 ms.

For ACPI there's a table called "MADT" or Multiple APIC Description Table (which actually uses "APIC" as its identifier and not "MADT" just to confuse people). This table contains entries for local APICs (CPUs), I/O APICs, IRQs, etc, just like the MP Specification's tables (although the way IRQs are described is a bit different). On top of that there's a few other tables that allow the OS to optimise for NUMA (the "SLIT" or System Locality Information Table and the "SRAT" or System Resource Allocation Table) which don't have an equivalent in the MP Specification tables.

3) I'll dump the second SIPI. Like I mentioned earlier I am only targeting 64-bit computers so I don't have to deal with legacy hardware.

I wouldn't remove the second SIPI completely - it may or may not be required.


Cheers,

Brendan

[M2004]

Does anyone else have issues with activating the AP's in newer computers? There are issues with Pure64 when run on core i5/i7 machines.

I have done some testing with the pure64 code on Intel Core2 Quad Q9550 processor (real hardware) and on Virtualbox 4.0 (emulator).
Sometimes both fail to setup one of the four cores. It's propably the same bug you encounterd on core i5/i7.

Regards
Mac2004

[IanSeyler]

Thanks for the good advice Brendan,

I have reworked the SMP code in Pure64. It now parses the ACPI tables to get the APIC Base, IO-APIC Base, and the individual CPU APIC numbers.

Code is here:
http://returninfinitylabs.svn.beanstalk ... sm?rev=108
http://returninfinitylabs.svn.beanstalk ... sm?rev=108

It correctly detects 4 cores in the ACPI tables on the core i5 (ID's 0, 2, 4, and 6) but reboots sometime during the IPI's. Works fine in QEMU emulating 64 cores.

I could see it rebooting if I accidentally tried to init/startup the BSP but I check for that.

Thanks,
-Ian