Hi,
Updating mtrr registers on an Intel processor requires a list of steps that includes disabling interrupts, cache flush and updating the MTRR (Complete list of steps to be done is listed in this linux source http://lxr.free-electrons.com/source/ar ... ain.c#L188). If the mtrr registers are edited (that includes editing current entry, adding new entry or removing an entry), the above activities need to be carried out on all cpus in the system.
My question is regarding MTRR initialization for a cpu coming online - When a cpu is moved to offline state and brought back to online state, do we need to stop the entire system to initialize the MTRR for the new online core. If my understanding is right, we are not editing MTRR but just populating the existing entries into MTRR for the new core. So, can we just execute the above steps on the new online cpu alone rather than all cpus?
- Sankar
MTRR Initialization
Re: MTRR Initialization
Hi,
For the second case (caches not disabled properly when CPU is offline), if you change MTTRs for the other CPUs then you also need to bring the offline CPU back online and program its MTRRs.
For the third case (hyper-threading); the MTTRs are shared by all logical CPUs in the core. In this case, if some logical CPUs in the core are offline and others aren't, then you can do the MTTRs changes (and cache disabling, etc) on the CPUs that are still online. However, you can't control the offline CPU's TLBs in this way. If all logical CPUs in the core are offline, then either all of them have all caches disabled or none of them do - effectively the core becomes one of the previous 2 cases.
For the last case (actual hot-plug); things are a little more complex. Often there's an L3 cache shared by all cores in the physical chip. This needs to be disabled and flushed before the CPU can be physically removed. For some CPUs (newer stuff) you probably don't have to worry. For older CPUs (Pentium 4/Netburst) you do.
My recommendations are:
a) When taking the a logical CPU offline, always disable paging and flush all TLBs. If other logical CPUs in the core are still online, then that's all. Otherwise (if the last logical CPU in the core is being taken offline) see "b)".
b) When taking a core offline; always disable paging and flush the TLBs; and always disable caches, flush caches and disable all MTRRs. If other cores in the physical chip are still online, then that's all. Otherwise (if the last core the physical chip is being taken offline) see "c)".
c) When taking a physical chip offline; check for and disable the chip's L3 cache when necessary (after disabling all caches, etc in all the cores).
d) When bringing a logical CPU back online; if other logical CPUs in the core are already online then you don't need to do anything special. Otherwise, see "e)"
e) When bringing a core back online; always reprogram the MTRRs to bring them up to date and enable the caches after that. If other cores within the chip are already online, then that's it. Otherwise see "f)".
f) When bringing a physical chip back online, always enable the L3 cache (if necessary, e.g. for Pentium 4/Netburst) before you enable anything else.
g) On power-up, do not assume the firmware disabled CPU's caches properly. This implies that you will need to deal with 2 different types of "offline" - "offline with all caches disabled properly" (done by you), and "offline with caches unknown" (done by firmware at power-on).
Note 1: For actual hot-plug CPUs (where the physical chip is actually removed from the motherboard); you will also need special code specifically for that specific motherboard (if you can find a motherboard that actually supports it). The same would apply to virtual machines.
Note 2: ACPI does not support hot-plug CPUs (either hot-add or hot-remove). Specifically, "Processor Local APIC" entries in ACPI's MADT that have the "enabled" flag clear do NOT represent "reserved for hot-add CPUs". It's far more likely for this to mean "CPU is present, but faulty and unusable". Be warned that Linux is a dodgy hack and does make stupid/non-standard assumptions about the meaning of the "enabled" flag in "Processor Local APIC" entries.
Cheers,
Brendan
There are 4 cases for "offline":sankarp wrote:Updating mtrr registers on an Intel processor requires a list of steps that includes disabling interrupts, cache flush and updating the MTRR (Complete list of steps to be done is listed in this linux source http://lxr.free-electrons.com/source/ar ... ain.c#L188). If the mtrr registers are edited (that includes editing current entry, adding new entry or removing an entry), the above activities need to be carried out on all cpus in the system.
My question is regarding MTRR initialization for a cpu coming online - When a cpu is moved to offline state and brought back to online state, do we need to stop the entire system to initialize the MTRR for the new online core. If my understanding is right, we are not editing MTRR but just populating the existing entries into MTRR for the new core. So, can we just execute the above steps on the new online cpu alone rather than all cpus?
- All CPU's caches were disabled and emptied before the CPU was put offline
- All CPU's caches where not disabled and emptied before the CPU was put offline
- Hyper-threading
- Hot-plug CPUs
For the second case (caches not disabled properly when CPU is offline), if you change MTTRs for the other CPUs then you also need to bring the offline CPU back online and program its MTRRs.
For the third case (hyper-threading); the MTTRs are shared by all logical CPUs in the core. In this case, if some logical CPUs in the core are offline and others aren't, then you can do the MTTRs changes (and cache disabling, etc) on the CPUs that are still online. However, you can't control the offline CPU's TLBs in this way. If all logical CPUs in the core are offline, then either all of them have all caches disabled or none of them do - effectively the core becomes one of the previous 2 cases.
For the last case (actual hot-plug); things are a little more complex. Often there's an L3 cache shared by all cores in the physical chip. This needs to be disabled and flushed before the CPU can be physically removed. For some CPUs (newer stuff) you probably don't have to worry. For older CPUs (Pentium 4/Netburst) you do.
My recommendations are:
a) When taking the a logical CPU offline, always disable paging and flush all TLBs. If other logical CPUs in the core are still online, then that's all. Otherwise (if the last logical CPU in the core is being taken offline) see "b)".
b) When taking a core offline; always disable paging and flush the TLBs; and always disable caches, flush caches and disable all MTRRs. If other cores in the physical chip are still online, then that's all. Otherwise (if the last core the physical chip is being taken offline) see "c)".
c) When taking a physical chip offline; check for and disable the chip's L3 cache when necessary (after disabling all caches, etc in all the cores).
d) When bringing a logical CPU back online; if other logical CPUs in the core are already online then you don't need to do anything special. Otherwise, see "e)"
e) When bringing a core back online; always reprogram the MTRRs to bring them up to date and enable the caches after that. If other cores within the chip are already online, then that's it. Otherwise see "f)".
f) When bringing a physical chip back online, always enable the L3 cache (if necessary, e.g. for Pentium 4/Netburst) before you enable anything else.
g) On power-up, do not assume the firmware disabled CPU's caches properly. This implies that you will need to deal with 2 different types of "offline" - "offline with all caches disabled properly" (done by you), and "offline with caches unknown" (done by firmware at power-on).
Note 1: For actual hot-plug CPUs (where the physical chip is actually removed from the motherboard); you will also need special code specifically for that specific motherboard (if you can find a motherboard that actually supports it). The same would apply to virtual machines.
Note 2: ACPI does not support hot-plug CPUs (either hot-add or hot-remove). Specifically, "Processor Local APIC" entries in ACPI's MADT that have the "enabled" flag clear do NOT represent "reserved for hot-add CPUs". It's far more likely for this to mean "CPU is present, but faulty and unusable". Be warned that Linux is a dodgy hack and does make stupid/non-standard assumptions about the meaning of the "enabled" flag in "Processor Local APIC" entries.
Cheers,
Brendan
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.
Re: MTRR Initialization
Thanks again for the response Brendan.
I should clarify that by hot-plug I refer to turning off core through software (and not physically removing or adding new processors). The software hot-plug mechanism supported in Linux is used by many mobile vendors for reasons of energy proportionality with respect to CPUs (turning off cores during period of low utilization).
I guess your first case of offline - "All CPU's caches were disabled and emptied before the CPU was put offline" - and recommendation at point (e) - "When bringing a core back online; always reprogram the MTRRs to bring them up to date and enable the caches after that. If other cores within the chip are already online, then that's it" - explains matches my case.
I should clarify that by hot-plug I refer to turning off core through software (and not physically removing or adding new processors). The software hot-plug mechanism supported in Linux is used by many mobile vendors for reasons of energy proportionality with respect to CPUs (turning off cores during period of low utilization).
I guess your first case of offline - "All CPU's caches were disabled and emptied before the CPU was put offline" - and recommendation at point (e) - "When bringing a core back online; always reprogram the MTRRs to bring them up to date and enable the caches after that. If other cores within the chip are already online, then that's it" - explains matches my case.