NUMA assumptions

[devsau]

My OS (if you want to even call it that ) is not NUMA aware, and I had a question about the multi-processor bootstrap protocol.

Question 1. Will SIPI wake AP's on another NUMA node? (probably connected via quickpath interconnect or hypertransport).

Question 2. Will IPI with "all except self" be broadcast to the other AP's on other nodes?

Question 3. I would be correct to assume that I would need a CPUID from each CPU in a NUMA node, considering they are seperate physical packages(multiple cores per package or hyper-threading out of scope here), and I should not assume that CPUID from the BSP would report the same as packages from other nodes?

Thanks

[Brendan]

Hi,

My OS (if you want to even call it that ) is not NUMA aware, and I had a question about the multi-processor bootstrap protocol.

Question 1. Will SIPI wake AP's on another NUMA node? (probably connected via quickpath interconnect or hypertransport).

Yes.

Question 2. Will IPI with "all except self" be broadcast to the other AP's on other nodes?

Yes; potentially including faulty CPUs and disabled logical CPUs (e.g. if hyper-threading was disabled in firmware), and excluding any way to determine if all CPU/s actually started. Note: What I mean here is that "broadcast to all" for AP startup is broken and shouldn't be used, and is the same (broken) on NUMA and plain SMP (without NUMA).

Question 3. I would be correct to assume that I would need a CPUID from each CPU in a NUMA node, considering they are seperate physical packages(multiple cores per package or hyper-threading out of scope here), and I should not assume that CPUID from the BSP would report the same as packages from other nodes?

This depends on the OS. For some OSs (e.g. Windows, as far as I know) the OS assumes all CPUs are "similar enough that software needn't care" despite Intel's advice. A better OS would follow Intel's advice (e.g. different CPUs in the same system may support different features and/or run at different speeds and/or have different cache sizes); but this is more complicated. Of course this applies to both NUMA and plain SMP (without NUMA).


Cheers,

Brendan

[devsau]

Thanks, that was pretty much my guess. Question though about IPI (all except self). You mentioned it could potentially wake factory disabled cores and I presume these aren't reported in ACPI/MP tables. So that makes me curious, how does the firmware determine if it's supposed to be "disabled."?


Thanks!

[Brendan]

Hi,

Thanks, that was pretty much my guess. Question though about IPI (all except self). You mentioned it could potentially wake factory disabled cores and I presume these aren't reported in ACPI/MP tables. So that makes me curious, how does the firmware determine if it's supposed to be "disabled."?

It won't wake factory disabled cores.

When you turn a computer on the CPUs do a self test; then the firmware starts all of them (to setup things like the MTRRs, SMM area, etc) and the CPU tells the firmware if it failed its self test. If the CPU failed its self test then the firmware doesn't include it in the ACPI table (and/or MP specification table) so that the OS won't use it. If the OS broadcasts the startup sequence, then faulty CPUs get started/used.

Also, for some systems there's a BIOS option to enable/disable hyper-threading (where the user may have disabled it because it doesn't help performance, especially on older Pentium4/Netburst systems). For some of these systems (also Pentium4/Netburst systems) hyper-threading isn't "fully disabled" and the firmware just doesn't include them in the ACPI table (and/or MP specification table) so that the OS won't use it. If the OS broadcasts the startup sequence, then "deliberately disabled for performance reasons" CPUs get started/used.

In addition to both these problems; a CPU that does exist and should start may not start correctly. I good OS would detect this (e.g. using a time-out) and might or might not retry, and would definitely report the problem to the user. If the OS broadcasts the startup sequence, then it becomes very difficult to detect this.

Finally, either of the first 2 problems can occur at the same time as the third problem. For example, if the BIOS sees 3 working CPUs and one faulty CPU (and creates the ACPI/MP spec table with entries for the 3 working CPUs); and the OS broadcasts the startup sequence; then the OS might start 2 working CPUs and one faulty CPU, and one of the working CPUs might fail to start. In that cause the OS might be expecting 3 CPUs to start and 3 CPUs were started.

Mostly, in terms of software quality, there's a massive difference between "seems to work" and "actually works"; and broadcasting the startup sequence only "seems to work" (e.g. when there's no problems to notice) but does not work (when there are problems).


Cheers,

Brendan

[devsau]

Late reply but thanks again!