Page 1 of 1
Dell 1950 Halting when CR0 Bit 30 Clear Cache Disable Reset
Posted: Sat Oct 04, 2014 6:02 pm
by tsdnz
Hi all.
A couple of servers have arrived and one of them is not starting my OS.
If someone could find my fault it would be appreciated.
Here is my Init code for the CPU.
Code: Select all
FIL void tCPUs::InitThisCPU()
{
// Disable Cache
asm volatile ("mov %cr0, %rax");
asm volatile ("btr $29, %rax"); // Clear No Write Thru (Bit 29)
asm volatile ("bts $30, %rax"); // Set Cache Disable (Bit 30)
asm volatile ("mov %rax, %cr0");
// Flush Cache
asm volatile ("wbinvd");
// Disable Paging Global Extensions
asm volatile ("mov %cr4, %rax");
asm volatile ("btr $7, %rax"); // Clear Paging Global Extensions (Bit 7)
asm volatile ("mov %rax, %cr4");
// Disable MTRRs and Configure default memory type to UC
asm volatile ("mov $0x000002FF, %ecx");
asm volatile ("rdmsr");
asm volatile ("and $0xFFFFF300, %eax"); // Clear MTRR Enable (Bit 11), Fixed Range MTRR Enable (Bit 10), and Default Memory Type (Bits 7:0) to UC (0x00)
asm volatile ("wrmsr");
// Enable MTRRs
asm volatile ("mov $0x000002FF, %ecx");
asm volatile ("rdmsr");
asm volatile ("bts $11, %eax"); // Set MTRR Enable (Bit 11), Only enables Variable Range MTRR's
asm volatile ("wrmsr");
// Flush Cache
asm volatile ("wbinvd");
// Enable Cache
asm volatile ("mov %cr0, %rax");
asm volatile ("btr $29, %rax"); // Clear No Write Thru (Bit 29)
asm volatile ("btr $30, %rax"); // Clear Cache Disable (Bit 30)
asm volatile ("mov %rax, %cr0");
Progress("InitaliseCPU - Enable Cache Done");
// Enable Floating Point
asm volatile ("mov %cr0, %rax");
asm volatile ("bts $1, %rax"); // Set Monitor co-processor (Bit 1)
asm volatile ("btr $2, %rax"); // Clear Emulation (Bit 2)
asm volatile ("mov %rax, %cr0");
Progress("InitaliseCPU - Floating Point Done");
// Enable SSE
asm volatile ("mov %cr4, %rax");
asm volatile ("bts $9, %rax"); // Set Operating System Support for FXSAVE and FXSTOR instructions (Bit 9)
asm volatile ("bts $10, %rax"); // Set Operating System Support for Unmasked SIMD Floating-Point Exceptions (Bit 10)
asm volatile ("mov %rax, %cr4");
Progress("InitaliseCPU - FSSE Done");
// Enable Math Co-processor
asm volatile ("finit");
Progress("InitaliseCPU - finit Done");
asm volatile ("": : :"%rcx","%rax","cc","memory");
}
It locks when this line is included:
Code: Select all
asm volatile ("btr $30, %rax"); // Clear Cache Disable (Bit 30)
Any ideas ??
Alistair
Re: Dell 1950 Halting when CR0 Bit 30 Clear Cache Disable Re
Posted: Sat Oct 04, 2014 6:12 pm
by alexfru
No ideas. But your inline assembly code is almost certainly broken because you never tell the compiler what registers, variables, etc you use. The line "asm volatile ("": : :"%rcx","%rax","cc","memory");" does not apply to any previous asm statements, just to itself. Oops. You need to learn to write proper inline assembly code or just make a separate assembly source with that asm code.
Re: Dell 1950 Halting when CR0 Bit 30 Clear Cache Disable Re
Posted: Sat Oct 04, 2014 6:19 pm
by tsdnz
No ideas. But your inline assembly code is almost certainly broken because you never tell the compiler what registers, variables, etc you use. The line "asm volatile ("": : :"%rcx","%rax","cc","memory");" does not apply to any previous asm statements, just to itself. Oops. You need to learn to write proper inline assembly code or just make a separate assembly source with that asm code.
Good points thanks.
Re: Dell 1950 Halting when CR0 Bit 30 Clear Cache Disable Re
Posted: Sat Oct 04, 2014 9:50 pm
by Brendan
Hi,
I'm also not sure why anyone would want to disable the MTRRs so that everything is "uncached". This will completely destroy performance.
Cheers,
Brendan
Re: Dell 1950 Halting when CR0 Bit 30 Clear Cache Disable Re
Posted: Sat Oct 04, 2014 9:52 pm
by tsdnz
I'm also not sure why anyone would want to disable the MTRRs so that everything is "uncached". This will completely destroy performance.
Cheers
Re: Dell 1950 Halting when CR0 Bit 30 Clear Cache Disable Re
Posted: Sat Oct 04, 2014 11:29 pm
by tsdnz
Hi all, how does this look for inline syntax and setting up caching?
The code is working now on the dell 1950 server, it looks like [MY] inline code was the issue!!
Code: Select all
asm volatile (
// Enable SSE
// Set Operating System Support for FXSAVE and FXSTOR instructions (Bit 9)
// Set Operating System Support for Unmasked SIMD Floating-Point Exceptions (Bit 10)
"mov %%cr4, %%rax; " \
"bts $9, %%rax; " \
"bts $10, %%rax; " \
"mov %%rax, %%cr4; " \
// Enables MTRR's & Enables Fixed Range MTRR's, Bit 10, Bit 11, and WB—Writeback (0x6) Bits 0-7
"mov $0x000002FF, %%ecx; " \
"rdmsr; " \
"mov $0x00000C06, %%eax; " \
"wrmsr; " \
// Flush Cache
"wbinvd; " \
// Enable Cache
// Clear No Write Thru (Bit 29), Not used legacy olny.
// Clear Cache Disable (Bit 30)
"mov %%cr0, %%rax; " \
"btr $29, %%rax; " \
"btr $30, %%rax; " \
"mov %%rax, %%cr0; " \
// Enable Floating Point
// Set Monitor co-processor (Bit 1)
// Clear Emulation (Bit 2)
"mov %%cr0, %%rax; " \
"bts $1, %%rax; " \
"btr $2, %%rax; " \
"mov %%rax, %%cr0; " \
// Init Math Co-processor
"finit; " \
// Clobber used registers
"": : : "%rcx","%rax","cc","memory");
return;
Re: Dell 1950 Halting when CR0 Bit 30 Clear Cache Disable Re
Posted: Sun Oct 05, 2014 12:27 am
by Brendan
Hi,
tsdnz wrote:Hi all, how does this look for inline syntax and setting up caching?
Code: Select all
asm volatile (
// Enable SSE
// Set Operating System Support for FXSAVE and FXSTOR instructions (Bit 9)
// Set Operating System Support for Unmasked SIMD Floating-Point Exceptions (Bit 10)
"mov %%cr4, %%rax; " \
"bts $9, %%rax; " \
"bts $10, %%rax; " \
"mov %%rax, %%cr4; " \
OK, so far; but I wouldn't preserve the previous values in CR4 at all, and would reduce it down to the 2 instructions "mov $0x00000600,%%rax" and "mov %%rax, %%cr4".
Code: Select all
// Enables MTRR's & Enables Fixed Range MTRR's, Bit 10, Bit 11, and WB—Writeback (0x6) Bits 0-7
"mov $0x000002FF, %%ecx; " \
"rdmsr; " \
"mov $0x00000C06, %%eax; " \
"wrmsr; " \
// Flush Cache
"wbinvd; " \
// Enable Cache
// Clear No Write Thru (Bit 29), Not used legacy olny.
// Clear Cache Disable (Bit 30)
"mov %%cr0, %%rax; " \
"btr $29, %%rax; " \
"btr $30, %%rax; " \
"mov %%rax, %%cr0; " \
This is mostly wrong. To change the MTTRs (which should be 100% unnecessary to begin with); you must:
- disable caches in CR0 and then flush them (wbinvd); on all CPUs (not just one)
- wait until all CPUs have done the previous step before continuing
- update the MTTRs on all CPUs, so that the MTRRs are identical on all CPUs
- wait until all CPUs have done the previous step before continuing
- enable caches in CR0
In addition; you can not assume that "default memory type = write-back" is correct. For most systems the firmware sets the default memory type to "uncached" and then uses the variable range MTTRs to change anything that shouldn't be uncached to "write-back" (or whatever else); and therefore for most systems "default memory type = write-back" is wrong (e.g. you will end up with things like memory mapped PCI devices using "write-back" and breaking because of it).
However; for some systems the firmware sets the default memory type to "write-back" and then uses the variable range MTTRs to change anything that shouldn't be write-back to "uncached" (or whatever else); and therefore for some systems "default memory type = write-back" is correct.
Essentially there are only 2 sane choices. Either you don't touch the MTTRs at all and assume the firmware did it's job properly; or you reprogram all variable range MTTRs and the default memory type at the same time. Reprogramming all variable range MTTRs correctly and efficiently (e.g. using the least number of variable range MTTRs so you've got the most free for things like video) is complicated.
Code: Select all
// Enable Floating Point
// Set Monitor co-processor (Bit 1)
// Clear Emulation (Bit 2)
"mov %%cr0, %%rax; " \
"bts $1, %%rax; " \
"btr $2, %%rax; " \
"mov %%rax, %%cr0; " \
// Init Math Co-processor
"finit; " \
// Clobber used registers
"": : : "%rcx","%rax","cc","memory");
To be perfectly correct; you should tell the compiler that all floating point registers (and all MMX and 3DNow registers) are clobbered. This brings up another problem - what prevents the compiler from using FPU, MMX or 3DNow before this code is executed? There's only 2 ways to guarantee that doesn't happen. The first is to write the CPU initialisation code as pure assembly (not inline assembly) and use it as the kernel's entry point (where the pure assembly initialises the CPU before any C code is executed, and then passes control to the C code). The other way is to tell the compiler not to use FPU, MMX or 3DNow (but I'm not sure if that's possible to do for individual functions so you may end up never using FPU, MMX or 3DNow for anything anywhere in your entire kernel).
Note: I'd be willing to bet that you've already got some "pure assembly startup code" that (e.g.) sets up the stack.
Cheers,
Brendan
Re: Dell 1950 Halting when CR0 Bit 30 Clear Cache Disable Re
Posted: Sun Oct 05, 2014 12:49 am
by tsdnz
This is mostly wrong. To change the MTTRs (which should be 100% unnecessary to begin with); you must:
disable caches in CR0 and then flush them (wbinvd); on all CPUs (not just one)
wait until all CPUs have done the previous step before continuing
update the MTTRs on all CPUs, so that the MTRRs are identical on all CPUs
wait until all CPUs have done the previous step before continuing
enable caches in CR0
Essentially there are only 2 sane choices. Either you don't touch the MTTRs at all and assume the firmware did it's job properly; or you reprogram all variable range MTTRs and the default memory type at the same time. Reprogramming all variable range MTTRs correctly and efficiently (e.g. using the least number of variable range MTTRs so you've got the most free for things like video) is complicated.
Thanks, they are gone
In addition; you can not assume that "default memory type = write-back" is correct. For most systems the firmware sets the default memory type to "uncached" and then uses the variable range MTTRs to change anything that shouldn't be uncached to "write-back" (or whatever else); and therefore for most systems "default memory type = write-back" is wrong (e.g. you will end up with things like memory mapped PCI devices using "write-back" and breaking because of it).
However; for some systems the firmware sets the default memory type to "write-back" and then uses the variable range MTTRs to change anything that shouldn't be write-back to "uncached" (or whatever else); and therefore for some systems "default memory type = write-back" is correct.
I see, again nice, thanks.
To be perfectly correct; you should tell the compiler that all floating point registers (and all MMX and 3DNow registers) are clobbered
I do like to be tidy so I will add the registers you specified.
Yes, this code is run in the first few lines of code, there is no chance of MMX or 3DNow.
Code: Select all
void APStart()
{
tCPUs::DisableInterrupts();
tCPUs::InitThisCPU();
ABIT.CPUs->ConfigureLocalAPIC();
Code: Select all
void BSPStart()
{
tCPUs::DisableInterrupts();
InitaliseCPU();
Many thanks.
Re: Dell 1950 Halting when CR0 Bit 30 Clear Cache Disable Re
Posted: Sun Oct 05, 2014 1:06 am
by tsdnz
The servers crash if I just put 0x600 into rax
Code: Select all
FIL void tCPUs::InitThisCPU()
{
asm volatile (
// Enable SSE
// Set Operating System Support for FXSAVE and FXSTOR instructions (Bit 9)
// Set Operating System Support for Unmasked SIMD Floating-Point Exceptions (Bit 10)
"mov %%cr4, %%rax; " \
"bts $9, %%rax; " \
"bts $10, %%rax; " \
"mov %%rax, %%cr4; " \
// Flush Cache
"wbinvd; " \
// Enable Floating Point
// Set Monitor co-processor (Bit 1)
// Clear Emulation (Bit 2)
"mov %%cr0, %%rax; " \
"bts $1, %%rax; " \
"btr $2, %%rax; " \
"mov %%rax, %%cr0; " \
// Init Math Co-processor
"finit; " \
// Clobber used registers
"": : : "%rcx","%rax","cc","memory");
return;
Re: Dell 1950 Halting when CR0 Bit 30 Clear Cache Disable Re
Posted: Sun Oct 05, 2014 1:23 am
by tsdnz
Code: Select all
The servers crash if I just put 0x600 into rax
Silly me, lol!!
I am setting up paging etc before this point in assembly.
Code: Select all
;------------------------------------------------------------------------------
align 16
EnableExtendedProperties:
mov eax, cr4
or eax, 0x0000000B0 ; PGE (Bit 7), PAE (Bit 5), and PSE (Bit 4)
mov cr4, eax
ret
Re: Dell 1950 Halting when CR0 Bit 30 Clear Cache Disable Re
Posted: Sun Oct 05, 2014 1:32 am
by tsdnz
Here is the final Init:
Code: Select all
FIL void tCPUs::InitThisCPU()
{
asm volatile (
// Enable SSE
// Set Operating System Support for FXSAVE and FXSTOR instructions (Bit 9)
// Set Operating System Support for Unmasked SIMD Floating-Point Exceptions (Bit 10)
"mov %%cr4, %%rax; " \
"bts $9, %%rax; " \
"bts $10, %%rax; " \
"mov %%rax, %%cr4; " \
// Flush Cache
"wbinvd; " \
// Enable Floating Point
// Set Monitor co-processor (Bit 1)
// Clear Emulation (Bit 2)
"mov %%cr0, %%rax; " \
"bts $1, %%rax; " \
"btr $2, %%rax; " \
"mov %%rax, %%cr0; " \
// Init Math Co-processor
"finit; " \
// Clobber Registers
"": : : "%rax","%rbx","%rcx","%rdx","%rsp","%rsi","%rdi","%r8","%r9","%r10","%r11","%r12","%r13","%r14","%r15","cc","memory","%st","%st(1)","%st(2)","%st(3)","%st(4)","%st(5)","%st(6)","%st(7)","%xmm0","%xmm1","%xmm2","%xmm3","%xmm4","%xmm5","%xmm6","%xmm7");
}
Re: Dell 1950 Halting when CR0 Bit 30 Clear Cache Disable Re
Posted: Sun Oct 05, 2014 3:04 am
by Brendan
tsdnz wrote:Here is the final Init:
Almost
The "wbinvd" is now unnecessary (and is expensive); and you aren't clobbering rbx, rcx, rdx, rsp (which I'm surprised GCC accepts), rsi, rdi, r8, r9, r10, r11, r12, r13, r14 or r15.
Cheers,
Brendan
Re: Dell 1950 Halting when CR0 Bit 30 Clear Cache Disable Re
Posted: Sun Oct 05, 2014 1:21 pm
by tsdnz
Almost
The "wbinvd" is now unnecessary (and is expensive); and you aren't clobbering rbx, rcx, rdx, rsp (which I'm surprised GCC accepts), rsi, rdi, r8, r9, r10, r11, r12, r13, r14 or r15.
Awake to a new day, looking forward to my cold leaving me!
Yes, left that in.
Thought I would add all the other registers in for fun, a clean entry point, and tell the compiler to clobber everything.
And thanks heaps.