CPUID - how to determine number of physical cores?

[limp]

Hi all,

I would like to do some comments regrading some things previously said by Brendan.

If the HTT flag is clear then it's an old single-core CPU without hyper-threading. In that case "logical_CPU_bits = 0" and "core_bits = 0" and you're done.

I am afraid; it’s not that simple There are some multi-core processors with no HT support (Intel Pentium D, Intel Pentium Dual-core, Intel Core 2 Duo, Intel Core 2 Quad).

decided that Logical APIC ID (which shouldn't be confused with the "APIC ID") and the "Destination Format Register" would be hardwired; and that "Logical Destination

I guess you meant that LAPIC ID shouldn’t be confused with “ACPI ID” and not APIC ID (as LAPIC ID and APIC ID is the same thing).

For complete CPU topology, there's 4 levels:
Which logical CPU within a core
Which core within a chip
Which chip within a NUMA domain
Which NUMA domain

On the 2 last levels, you’re mentioning NUMA domains. On an SMP system, there is not possibility of having NUMA domains, so I guess what have you described applies only on NUMA system and not on SMP ones.

Please correct me if I interpreted something wrong.

Kind regards.

[Brendan]

Hi,

If the HTT flag is clear then it's an old single-core CPU without hyper-threading. In that case "logical_CPU_bits = 0" and "core_bits = 0" and you're done.

I am afraid; it’s not that simple There are some multi-core processors with no HT support (Intel Pentium D, Intel Pentium Dual-core, Intel Core 2 Duo, Intel Core 2 Quad).

It is that simple. All Intel multi-core CPUs set the "HTT" flag, regardless of whether they support hyper-threading or not. Basically that "HTT" flag should be called a "more than one logical CPU in chip" flag.

decided that Logical APIC ID (which shouldn't be confused with the "APIC ID") and the "Destination Format Register" would be hardwired; and that "Logical Destination

I guess you meant that LAPIC ID shouldn’t be confused with “ACPI ID” and not APIC ID (as LAPIC ID and APIC ID is the same thing).

I meant, "logical APIC ID" is extremely different to "local APIC ID". For example, you can set the logical APIC ID to 0x00000001 on all CPUs, and broadcast an IPI to all CPUs that have bit 0 set (using logical destination mode).

For complete CPU topology, there's 4 levels:
Which logical CPU within a core
Which core within a chip
Which chip within a NUMA domain
Which NUMA domain

On the 2 last levels, you’re mentioning NUMA domains. On an SMP system, there is not possibility of having NUMA domains, so I guess what have you described applies only on NUMA system and not on SMP ones.

For an OS designed to support NUMA, it's useful to have a different perspective: all computers are NUMA (just some only have one NUMA domain containing everything).


Cheers,

Brendan

[Combuster]

On an SMP system, there is not possibility of having NUMA domains.

Symmetric multiprocessing essentially comes down to each processor seeing the same memory at the same place. NUMA means that access to memory differs in speed and latency between processing units. Those are not mutually exclusive: if CPU1 sees memory unit 1 at 0G and memory unit 2 at 1G, and CPU 2 sees them at the exact same addresses, SMP is satisfied. If CPU1 controls memory 1 and asks CPU2 for anything in memory 2, whereas CPU2 is connected to memory 2 and asks CPU1 for anything in memory 1, NUMA is satisfied.

In other words: get a multisocket Opteron system

[limp]

Hi there,

If the HTT flag is clear then it's an old single-core CPU without hyper-threading. In that case "logical_CPU_bits = 0" and "core_bits = 0" and you're done.

I am afraid; it’s not that simple There are some multi-core processors with no HT support (Intel Pentium D, Intel Pentium Dual-core, Intel Core 2 Duo, Intel Core 2 Quad).

It is that simple. All Intel multi-core CPUs set the "HTT" flag, regardless of whether they support hyper-threading or not. Basically that "HTT" flag should be called a "more than one logical CPU in chip" flag.

Hmm...I haven’t checked it but I assumed that the HTT flag would be not-set on no HT-capable processors. They should definitely consider renaming the flag’s name.

decided that Logical APIC ID (which shouldn't be confused with the "APIC ID") and the "Destination Format Register" would be hardwired; and that "Logical Destination

I guess you meant that LAPIC ID shouldn’t be confused with “ACPI ID” and not APIC ID (as LAPIC ID and APIC ID is the same thing).

I meant, "logical APIC ID" is extremely different to "local APIC ID". For example, you can set the logical APIC ID to 0x00000001 on all CPUs, and broadcast an IPI to all CPUs that have bit 0 set (using logical destination mode).

I see…btw is there any apparent reason that I don't see in Intel's decision to use a different identifier (i.e. "Logical APIC IDs") for implementing logical destination mode (i.e. for sending IPIs) rather than using "Local APIC IDs"?

For complete CPU topology, there's 4 levels:
Which logical CPU within a core
Which core within a chip
Which chip within a NUMA domain
Which NUMA domain

On the 2 last levels, you’re mentioning NUMA domains. On an SMP system, there is not possibility of having NUMA domains, so I guess what have you described applies only on NUMA system and not on SMP ones.

For an OS designed to support NUMA, it's useful to have a different perspective: all computers are NUMA (just some only have one NUMA domain containing everything).

I agree with you however, as my OS is strictly for Intel SMPs, I considered the extra ID bits as a bit of an extra hassle without any real benefits in my case!

On an SMP system, there is not possibility of having NUMA domains.

Symmetric multiprocessing essentially comes down to each processor seeing the same memory at the same place. NUMA means that access to memory differs in speed and latency between processing units. Those are not mutually exclusive: if CPU1 sees memory unit 1 at 0G and memory unit 2 at 1G, and CPU 2 sees them at the exact same addresses, SMP is satisfied. If CPU1 controls memory 1 and asks CPU2 for anything in memory 2, whereas CPU2 is connected to memory 2 and asks CPU1 for anything in memory 1, NUMA is satisfied.

In other words: get a multisocket Opteron system

I guess what you're saying is that an NUMA system (like Opteron) satisfies SMP in certain cases however, an SMP system (like Intel's multicores) is not possible to satisfy NUMA in any case.

Regards.

[Brendan]

Hi,

Symmetric multiprocessing essentially comes down to each processor seeing the same memory at the same place. NUMA means that access to memory differs in speed and latency between processing units. Those are not mutually exclusive: if CPU1 sees memory unit 1 at 0G and memory unit 2 at 1G, and CPU 2 sees them at the exact same addresses, SMP is satisfied. If CPU1 controls memory 1 and asks CPU2 for anything in memory 2, whereas CPU2 is connected to memory 2 and asks CPU1 for anything in memory 1, NUMA is satisfied.

In other words: get a multisocket Opteron system

I guess what you're saying is that an NUMA system (like Opteron) satisfies SMP is certain case however, an SMP system (like Intel's multicores) is not possible to satisfy NUMA in any case.

All "multi-socket" 80x86 machines are NUMA now. AMD's have been NUMA for ages (ever since they switched to hyper-transport links, about a decade ago), and Intel's have been since about 2009 (when they switched to QuickPath links).

More specifically, they're cache coherent NUMA, where the NUMA ratio is quite low. This means you can probably pretend they're SMP without too much performance loss (especially if the number of CPUs is relatively low - e.g. 8 or less CPUs).

However, "number of CPUs" is increasing at a rate of roughly twice as many CPUs every 2 years. This causes a massive scalability problem for both hardware and software; and is the reason why both AMD and Intel switched to "point to point links" instead of continuing to use the old "front side bus" stuff. It's also why modern OSs need to be optimised for NUMA - so that the system as a whole scales well to "lots of CPUs" and doesn't suck for current "higher-end" workstation/server systems (and the "average" desktop/workstation/server systems we'll see in future).


Cheers,

Brendan

[limp]

Hi,

If the HTT flag is clear then it's an old single-core CPU without hyper-threading. In that case "logical_CPU_bits = 0" and "core_bits = 0" and you're done.

I am afraid; it’s not that simple There are some multi-core processors with no HT support (Intel Pentium D, Intel Pentium Dual-core, Intel Core 2 Duo, Intel Core 2 Quad).

It is that simple. All Intel multi-core CPUs set the "HTT" flag, regardless of whether they support hyper-threading or not. Basically that "HTT" flag should be called a "more than one logical CPU in chip" flag.

I did a quick check on my Intel Celeron M 440 and ACPI reports 2 LAPICs although HTT flag is not set (the processor should not support HT according to the specs). Specifically, ACPI founds a LAPIC entry with APIC ID 0, ACPI ID 1 and one with APIC ID 0x81 and ACPI ID 2. Does this makes any sense to you?

Regards.

[Brendan]

Hi,

I am afraid; it’s not that simple There are some multi-core processors with no HT support (Intel Pentium D, Intel Pentium Dual-core, Intel Core 2 Duo, Intel Core 2 Quad).

It is that simple. All Intel multi-core CPUs set the "HTT" flag, regardless of whether they support hyper-threading or not. Basically that "HTT" flag should be called a "more than one logical CPU in chip" flag.

I did a quick check on my Intel Celeron M 440 and ACPI reports 2 LAPICs although HTT flag is not set (the processor should not support HT according to the specs). Specifically, ACPI founds a LAPIC entry with APIC ID 0, ACPI ID 1 and one with APIC ID 0x81 and ACPI ID 2. Does this makes any sense to you?

The first part (HTT flag clear) makes perfect sense - according to Intel's site, Celeron M 440 is a single-core chip without hyper-threading.

The second part (ACPI results) look wrong. Even if there's 4 separate physical packages/chips (which doesn't make much sense for mobile parts), it'd be very rare for the local APIC IDs to be in such a strange order.


Cheers,

Brendan

[limp]

Hi Brendan,

The first part (HTT flag clear) makes perfect sense - according to Intel's site, Celeron M 440 is a single-core chip without hyper-threading.

The second part (ACPI results) look wrong. Even if there's 4 separate physical packages/chips (which doesn't make much sense for mobile parts), it'd be very rare for the local APIC IDs to be in such a strange order.

Thanks for the response. Apparently I was parsing the MADT correctly (I have verified that there are indeed 2 APIC entries) but I forgot the importance of the "Enable" field (I know, quite a big mistake). Specifically, the "APIC ID 0x81, ACPI ID 2" entry is marked as “disabled” (you were right that it looked wrong).

I remember you mentioning in other posts that Intel tends to disable/enable certain features in their processors, either because they have some defects or just because they seek of selling them under different series eliminating the need/cost of making new chips. I think that applies in my case. Either my processor is a defected HT processor (with 1 of its APIC not working) or a processor from a more expensive series (so they just disabled 1 APIC).

Now, a question I have regarding the CPU topology detection. When I run my topology detection code on my Celeron 440 M (which is a single-core processor without HT), I get (using CPUID leaf 4), max PACKAGE_ID = 1, max CORE_ID = 1 and max SMT_ID = 1.

I get max CORE_ID = 1 and max SMT_ID = 1 because CPUID says that max CORE_ID and max SMT_ID should be equal to the nearest power-of-2 integer that is not smaller than (1 + EAX[25:14]) and (1 + EAX[31:26]) respectively). So, although I get “0” from EAX[25:14] and EAX[31:26], I add “1” because that’s what it says to do. Also, I get PACKAGE_ID = 1 because although I get count=0 from EBX bits 16 to 23, I add 1 to the result as the manual says.

So, I end up with max PACKAGE_ID = 1, max CORE_ID = 1 and max SMT_ID = 1 which is certainly wrong. Do I have to assign all max PACKAGE_ID, max CORE_ID and max SMT_ID to 0 without reading the CPUID flags at all in this case (single-core processor without HT) ?

Something else that confuses me is that Intel’s manual 2A and “Intel’s Detecting Multi-Core Processor Topology in an IA-32 Platform” document seem to have a different opinion upon certain things.

While the former says as mentioned that:

that max CORE_ID and max SMT_ID should be equal to the nearest power-of-2 integer that is not smaller than (1 + EAX[25:14]) and (1 + EAX[31:26]) respectively)

the latter says that:

“One should not assume that the number of available threads or cores will be a power of two.”

Kind regards,

limp

[Brendan]

Hi,

The first part (HTT flag clear) makes perfect sense - according to Intel's site, Celeron M 440 is a single-core chip without hyper-threading.

The second part (ACPI results) look wrong. Even if there's 4 separate physical packages/chips (which doesn't make much sense for mobile parts), it'd be very rare for the local APIC IDs to be in such a strange order.

Thanks for the response. Apparently I was parsing the MADT correctly (I have verified that there are indeed 2 APIC entries) but I forgot the importance of the "Enable" field (I know, quite a big mistake). Specifically, the "APIC ID 0x81, ACPI ID 2" entry is marked as “disabled” (you were right that it looked wrong).

I remember you mentioning in other posts that Intel tends to disable/enable certain features in their processors, either because they have some defects or just because they seek of selling them under different series eliminating the need/cost of making new chips. I think that applies in my case. Either my processor is a defected HT processor (with 1 of its APIC not working) or a processor from a more expensive series (so they just disabled 1 APIC).

The other possibility is that the BIOS/motherboard is designed to support quad core CPUs and creates a MADT suitable for 4 CPUs; but then marks any unused entries as "disabled" (to simplify the code that generates the ACPI tables by using fixed length tables).

Now, a question I have regarding the CPU topology detection. When I run my topology detection code on my Celeron 440 M (which is a single-core processor without HT), I get (using CPUID leaf 4), max PACKAGE_ID = 1, max CORE_ID = 1 and max SMT_ID = 1.

I get max CORE_ID = 1 and max SMT_ID = 1 because CPUID says that max CORE_ID and max SMT_ID should be equal to the nearest power-of-2 integer that is not smaller than (1 + EAX[25:14]) and (1 + EAX[31:26]) respectively). So, although I get “0” from EAX[25:14] and EAX[31:26], I add “1” because that’s what it says to do. Also, I get PACKAGE_ID = 1 because although I get count=0 from EBX bits 16 to 23, I add 1 to the result as the manual says.

So, I end up with max PACKAGE_ID = 1, max CORE_ID = 1 and max SMT_ID = 1 which is certainly wrong. Do I have to assign all max PACKAGE_ID, max CORE_ID and max SMT_ID to 0 without reading the CPUID flags at all in this case (single-core processor without HT) ?

If the HTT flag is clear then it's a single-core CPU without hyper-threading, and looking further is a waste of time.

Something else that confuses me is that Intel’s manual 2A and “Intel’s Detecting Multi-Core Processor Topology in an IA-32 Platform” document seem to have a different opinion upon certain things.

While the former says as mentioned that:

that max CORE_ID and max SMT_ID should be equal to the nearest power-of-2 integer that is not smaller than (1 + EAX[25:14]) and (1 + EAX[31:26]) respectively)

the latter says that:

“One should not assume that the number of available threads or cores will be a power of two.”

That doesn't look contradictory at all.

If the max CORE ID is 4, then it doesn't imply that there's actually 4 cores and you could have 1 core with 3 unused IDs.


Cheers,

Brendan

[limp]

Hi,

Thanks for your reply Brendan, it has made things clearer to me.

If the HTT flag is clear then it's a single-core CPU without hyper-threading, and looking further is a waste of time.

I imagined that it would be waste of time too but I guess I thought that it would be possible to derive this conclusion using a different way as well.

That doesn't look contradictory at all.

If the max CORE ID is 4, then it doesn't imply that there's actually 4 cores and you could have 1 core with 3 unused IDs.

Now I got it, thanks.

One last question: In my Intel Atom 330 target, when I disable HT from the BIOS, I get the following:

Code: Select all

max PACKAGE_ID = 65, max CORE_ID = 2, max SMT_ID = 1

LAPIC (acpi id[0x01]  lapic id [0x00] enabled)
LAPIC (acpi id[0x02]  lapic id [0x01] enabled)
LAPIC (acpi id[0x03]  lapic id [0x82] disabled)
LAPIC (acpi id[0x04]  lapic id [0x83] disabled)

So, for HT-disabled, we have:

Code: Select all

max PACKAGE_ID = 65, max CORE_ID = 2, max SMT_ID = 1

LAPIC (lapic id [0x00]  => Package 0, Core 0, Log. CPU 0 enabled)
LAPIC (lapic id [0x01]  => Package 0, Core 0, Log. CPU 1 enabled)
LAPIC (lapic id [0x82]  => Package 0x20, Core 1, Log. CPU 0 disabled)
LAPIC (lapic id [0x83]  => Package 0x20, Core 1, Log. CPU 1 disabled)

This indicates (according to the topology masking bits/width) that when HT is disabled, we don’t have a single APIC per CPU core as I would expect but two APICs on a single CPU core and none on the other (which suggests that our CPU acts like a single-core HT CPU).

Could you (or anyone else) please help me understand why is that?

In the above case, I determined the CPU topological relationships using the LAPIC ID that APCI tables return and not the "Initial APIC ID" of each processor. Is it wrong to so something like that?

Kind regards,

limp.

[Brendan]

Hi,

One last question: In my Intel Atom 330 target, when I disable HT from the BIOS, I get the following:

Code: Select all

max PACKAGE_ID = 65, max CORE_ID = 2, max SMT_ID = 1

LAPIC (acpi id[0x01]  lapic id [0x00] enabled)
LAPIC (acpi id[0x02]  lapic id [0x01] enabled)
LAPIC (acpi id[0x03]  lapic id [0x82] disabled)
LAPIC (acpi id[0x04]  lapic id [0x83] disabled)

So, for HT-disabled, we have:

Code: Select all

max PACKAGE_ID = 65, max CORE_ID = 2, max SMT_ID = 1

LAPIC (lapic id [0x00]  => Package 0, Core 0, Log. CPU 0 enabled)
LAPIC (lapic id [0x01]  => Package 0, Core 0, Log. CPU 1 enabled)
LAPIC (lapic id [0x82]  => Package 0x20, Core 1, Log. CPU 0 disabled)
LAPIC (lapic id [0x83]  => Package 0x20, Core 1, Log. CPU 1 disabled)

This indicates (according to the topology masking bits/width) that when HT is disabled, we don’t have a single APIC per CPU core as I would expect but two APICs on a single CPU core and none on the other (which suggests that our CPU acts like a single-core HT CPU).

Could you (or anyone else) please help me understand why is that?

I probably should've mentioned this somewhere..

For local APIC ID's, there's the ID that happens to be in the local APIC's "Local APIC ID register". This can be changed by software at any time and therefore can't be relied on for topology detection. This also means that the firmware could've changed the "Local APIC ID register" to whatever it likes before creating the ACPI tables; and the local APIC IDs listed in the ACPI tables (and/or MP specification tables) can't be relied on for topology detection either.

"CPUID = 0x00000001" returns the "The 8-bit value assigned to the local APIC physical ID register at power-up" in EBX bits 24 to 31. Because this can't be changed by any software, it can be relied on for topology detection (and is the only "ID" that can be used reliably for topology detection).


Cheers,

Brendan

[limp]

I probably should've mentioned this somewhere..

For local APIC ID's, there's the ID that happens to be in the local APIC's "Local APIC ID register". This can be changed by software at any time and therefore can't be relied on for topology detection. This also means that the firmware could've changed the "Local APIC ID register" to whatever it likes before creating the ACPI tables; and the local APIC IDs listed in the ACPI tables (and/or MP specification tables) can't be relied on for topology detection either.

"CPUID = 0x00000001" returns the "The 8-bit value assigned to the local APIC physical ID register at power-up" in EBX bits 24 to 31. Because this can't be changed by any software, it can be relied on for topology detection (and is the only "ID" that can be used reliably for topology detection).

I see...thanks for clarifying this. So, for determining which LAPIC is in which Core/Package, we need to get the “Initial LAPIC” of each enabled LAPIC unit. But how can I do that without booting all the CPUs first (for instance, in the case in which HT is enabled, I want to be able to boot only one LAPIC / core)?

Thanks.

[Brendan]

Hi,

I see...thanks for clarifying this. So, for determining which LAPIC is in which Core/Package, we need to get the “Initial LAPIC” of each enabled LAPIC unit. But how can I do that without booting all the CPUs first (for instance, in the case in which HT is enabled, I want to be able to boot only one LAPIC / core)?

I don't think there's any way to do it without starting the other (enabled/present) CPUs.

If you want some (enabled/present) CPUs to remain "unstarted", then you could either skip them entirely (and maybe do the topology detection on them later on if/when you do want to start them), or start them and shut them down again afterwards.


Cheers,

Brendan

[limp]

Hi,

Hi,

I see...thanks for clarifying this. So, for determining which LAPIC is in which Core/Package, we need to get the “Initial LAPIC” of each enabled LAPIC unit. But how can I do that without booting all the CPUs first (for instance, in the case in which HT is enabled, I want to be able to boot only one LAPIC / core)?

I don't think there's any way to do it without starting the other (enabled/present) CPUs.

If you want some (enabled/present) CPUs to remain "unstarted", then you could either skip them entirely (and maybe do the topology detection on them later on if/when you do want to start them), or start them and shut them down again afterwards.

Well, I don't want to randomly start some CPUs and leave some others "unstarted" but to start only one CPU (LAPIC) per core. I guess from what you're saying that I can't really do that as I can't do a full topology detection unless I start all CPUs (as it depends on the "Initial APIC IDs").

So, I guess I need to start them all first, get the full topological relationship of the CPUs and then shutdown the extra CPUs so that I have one LAPIC / CPU core.

Thanks,

limp.