Dell 1950 Halting when CR0 Bit 30 Clear Cache Disable Reset

[tsdnz]

Hi all.
A couple of servers have arrived and one of them is not starting my OS.
If someone could find my fault it would be appreciated.
Here is my Init code for the CPU.

Code: Select all

FIL void tCPUs::InitThisCPU()
	{
		// Disable Cache
		asm volatile ("mov %cr0, %rax");
		asm volatile ("btr $29, %rax");			// Clear No Write Thru (Bit 29)
		asm volatile ("bts $30, %rax");			// Set Cache Disable (Bit 30)
		asm volatile ("mov %rax, %cr0");

		// Flush Cache
		asm volatile ("wbinvd");

		// Disable Paging Global Extensions
		asm volatile ("mov %cr4, %rax");
		asm volatile ("btr $7, %rax");				// Clear Paging Global Extensions (Bit 7)
		asm volatile ("mov %rax, %cr4");

		// Disable MTRRs and Configure default memory type to UC
		asm volatile ("mov $0x000002FF, %ecx");
		asm volatile ("rdmsr");
		asm volatile ("and $0xFFFFF300, %eax");	// Clear MTRR Enable (Bit 11), Fixed Range MTRR Enable (Bit 10), and Default Memory Type (Bits 7:0) to UC (0x00)
		asm volatile ("wrmsr");

		// Enable MTRRs
		asm volatile ("mov $0x000002FF, %ecx");
		asm volatile ("rdmsr");
		asm volatile ("bts $11, %eax");				// Set MTRR Enable (Bit 11), Only enables Variable Range MTRR's
		asm volatile ("wrmsr");

		// Flush Cache
		asm volatile ("wbinvd");

		// Enable Cache
		asm volatile ("mov %cr0, %rax");
		asm volatile ("btr $29, %rax");			// Clear No Write Thru (Bit 29)
		asm volatile ("btr $30, %rax");			// Clear Cache Disable (Bit 30)
		asm volatile ("mov %rax, %cr0");
		Progress("InitaliseCPU - Enable Cache Done");

		// Enable Floating Point
		asm volatile ("mov %cr0, %rax");
		asm volatile ("bts $1, %rax");				// Set Monitor co-processor (Bit 1)
		asm volatile ("btr $2, %rax");				// Clear Emulation (Bit 2)
		asm volatile ("mov %rax, %cr0");
		Progress("InitaliseCPU - Floating Point Done");

		// Enable SSE
		asm volatile ("mov %cr4, %rax");
		asm volatile ("bts $9, %rax");				// Set Operating System Support for FXSAVE and FXSTOR instructions (Bit 9)
		asm volatile ("bts $10, %rax");				// Set Operating System Support for Unmasked SIMD Floating-Point Exceptions (Bit 10)
		asm volatile ("mov %rax, %cr4");
		Progress("InitaliseCPU - FSSE Done");

		// Enable Math Co-processor
		asm volatile ("finit");
		Progress("InitaliseCPU - finit Done");

		asm volatile ("": : :"%rcx","%rax","cc","memory");
	}

It locks when this line is included:

Code: Select all

asm volatile ("btr $30, %rax");			// Clear Cache Disable (Bit 30)

Any ideas ??

Alistair

[alexfru]

No ideas. But your inline assembly code is almost certainly broken because you never tell the compiler what registers, variables, etc you use. The line "asm volatile ("": : :"%rcx","%rax","cc","memory");" does not apply to any previous asm statements, just to itself. Oops. You need to learn to write proper inline assembly code or just make a separate assembly source with that asm code.

[tsdnz]

No ideas. But your inline assembly code is almost certainly broken because you never tell the compiler what registers, variables, etc you use. The line "asm volatile ("": : :"%rcx","%rax","cc","memory");" does not apply to any previous asm statements, just to itself. Oops. You need to learn to write proper inline assembly code or just make a separate assembly source with that asm code.

Good points thanks.

[Brendan]

Hi,

I'm also not sure why anyone would want to disable the MTRRs so that everything is "uncached". This will completely destroy performance.


Cheers,

Brendan

[tsdnz]

I'm also not sure why anyone would want to disable the MTRRs so that everything is "uncached". This will completely destroy performance.

Cheers

[tsdnz]

Hi all, how does this look for inline syntax and setting up caching?

The code is working now on the dell 1950 server, it looks like [MY] inline code was the issue!!

Code: Select all

		asm volatile (
			// Enable SSE
			// Set Operating System Support for FXSAVE and FXSTOR instructions (Bit 9)
			// Set Operating System Support for Unmasked SIMD Floating-Point Exceptions (Bit 10)
			"mov %%cr4, %%rax;			" \
			"bts $9, %%rax;				" \
			"bts $10, %%rax;			" \
			"mov %%rax, %%cr4;			" \

			// Enables MTRR's & Enables Fixed Range MTRR's, Bit 10, Bit 11, and WB—Writeback (0x6) Bits 0-7
			"mov $0x000002FF, %%ecx;	" \
			"rdmsr;						" \
			"mov $0x00000C06, %%eax;	" \
			"wrmsr;						" \

			// Flush Cache
			"wbinvd;					" \

			// Enable Cache
			// Clear No Write Thru (Bit 29), Not used legacy olny.
			// Clear Cache Disable (Bit 30)
			"mov %%cr0, %%rax;			" \
			"btr $29, %%rax;			" \
			"btr $30, %%rax;			" \
			"mov %%rax, %%cr0;			" \

			// Enable Floating Point
			// Set Monitor co-processor (Bit 1)
			// Clear Emulation (Bit 2)
			"mov %%cr0, %%rax;			" \
			"bts $1, %%rax;				" \
			"btr $2, %%rax;				" \
			"mov %%rax, %%cr0;			" \

			// Init Math Co-processor
			"finit;						" \

			// Clobber used registers
			"": : : "%rcx","%rax","cc","memory");

		return;

[Brendan]

Hi,

Hi all, how does this look for inline syntax and setting up caching?

Code: Select all

		asm volatile (
			// Enable SSE
			// Set Operating System Support for FXSAVE and FXSTOR instructions (Bit 9)
			// Set Operating System Support for Unmasked SIMD Floating-Point Exceptions (Bit 10)
			"mov %%cr4, %%rax;			" \
			"bts $9, %%rax;				" \
			"bts $10, %%rax;			" \
			"mov %%rax, %%cr4;			" \

OK, so far; but I wouldn't preserve the previous values in CR4 at all, and would reduce it down to the 2 instructions "mov $0x00000600,%%rax" and "mov %%rax, %%cr4".

Code: Select all

			// Enables MTRR's & Enables Fixed Range MTRR's, Bit 10, Bit 11, and WB—Writeback (0x6) Bits 0-7
			"mov $0x000002FF, %%ecx;	" \
			"rdmsr;						" \
			"mov $0x00000C06, %%eax;	" \
			"wrmsr;						" \

			// Flush Cache
			"wbinvd;					" \

			// Enable Cache
			// Clear No Write Thru (Bit 29), Not used legacy olny.
			// Clear Cache Disable (Bit 30)
			"mov %%cr0, %%rax;			" \
			"btr $29, %%rax;			" \
			"btr $30, %%rax;			" \
			"mov %%rax, %%cr0;			" \

This is mostly wrong. To change the MTTRs (which should be 100% unnecessary to begin with); you must:

• disable caches in CR0 and then flush them (wbinvd); on all CPUs (not just one)
• wait until all CPUs have done the previous step before continuing
• update the MTTRs on all CPUs, so that the MTRRs are identical on all CPUs
• wait until all CPUs have done the previous step before continuing
• enable caches in CR0

In addition; you can not assume that "default memory type = write-back" is correct. For most systems the firmware sets the default memory type to "uncached" and then uses the variable range MTTRs to change anything that shouldn't be uncached to "write-back" (or whatever else); and therefore for most systems "default memory type = write-back" is wrong (e.g. you will end up with things like memory mapped PCI devices using "write-back" and breaking because of it).

However; for some systems the firmware sets the default memory type to "write-back" and then uses the variable range MTTRs to change anything that shouldn't be write-back to "uncached" (or whatever else); and therefore for some systems "default memory type = write-back" is correct.

Essentially there are only 2 sane choices. Either you don't touch the MTTRs at all and assume the firmware did it's job properly; or you reprogram all variable range MTTRs and the default memory type at the same time. Reprogramming all variable range MTTRs correctly and efficiently (e.g. using the least number of variable range MTTRs so you've got the most free for things like video) is complicated.

Code: Select all

			// Enable Floating Point
			// Set Monitor co-processor (Bit 1)
			// Clear Emulation (Bit 2)
			"mov %%cr0, %%rax;			" \
			"bts $1, %%rax;				" \
			"btr $2, %%rax;				" \
			"mov %%rax, %%cr0;			" \

			// Init Math Co-processor
			"finit;						" \

			// Clobber used registers
			"": : : "%rcx","%rax","cc","memory");

To be perfectly correct; you should tell the compiler that all floating point registers (and all MMX and 3DNow registers) are clobbered. This brings up another problem - what prevents the compiler from using FPU, MMX or 3DNow before this code is executed? There's only 2 ways to guarantee that doesn't happen. The first is to write the CPU initialisation code as pure assembly (not inline assembly) and use it as the kernel's entry point (where the pure assembly initialises the CPU before any C code is executed, and then passes control to the C code). The other way is to tell the compiler not to use FPU, MMX or 3DNow (but I'm not sure if that's possible to do for individual functions so you may end up never using FPU, MMX or 3DNow for anything anywhere in your entire kernel).

Note: I'd be willing to bet that you've already got some "pure assembly startup code" that (e.g.) sets up the stack.


Cheers,

Brendan

[tsdnz]

This is mostly wrong. To change the MTTRs (which should be 100% unnecessary to begin with); you must:
disable caches in CR0 and then flush them (wbinvd); on all CPUs (not just one)
wait until all CPUs have done the previous step before continuing
update the MTTRs on all CPUs, so that the MTRRs are identical on all CPUs
wait until all CPUs have done the previous step before continuing
enable caches in CR0

Essentially there are only 2 sane choices. Either you don't touch the MTTRs at all and assume the firmware did it's job properly; or you reprogram all variable range MTTRs and the default memory type at the same time. Reprogramming all variable range MTTRs correctly and efficiently (e.g. using the least number of variable range MTTRs so you've got the most free for things like video) is complicated.

Thanks, they are gone

In addition; you can not assume that "default memory type = write-back" is correct. For most systems the firmware sets the default memory type to "uncached" and then uses the variable range MTTRs to change anything that shouldn't be uncached to "write-back" (or whatever else); and therefore for most systems "default memory type = write-back" is wrong (e.g. you will end up with things like memory mapped PCI devices using "write-back" and breaking because of it).

However; for some systems the firmware sets the default memory type to "write-back" and then uses the variable range MTTRs to change anything that shouldn't be write-back to "uncached" (or whatever else); and therefore for some systems "default memory type = write-back" is correct.

I see, again nice, thanks.

To be perfectly correct; you should tell the compiler that all floating point registers (and all MMX and 3DNow registers) are clobbered

I do like to be tidy so I will add the registers you specified.


Yes, this code is run in the first few lines of code, there is no chance of MMX or 3DNow.

Code: Select all

	void APStart()
	{
		tCPUs::DisableInterrupts();
		tCPUs::InitThisCPU();
		ABIT.CPUs->ConfigureLocalAPIC();

Code: Select all

	void BSPStart()
	{
		tCPUs::DisableInterrupts();
		InitaliseCPU();

Many thanks.

[tsdnz]

The servers crash if I just put 0x600 into rax

Code: Select all

FIL void tCPUs::InitThisCPU()
	{
		asm volatile (
			// Enable SSE
			// Set Operating System Support for FXSAVE and FXSTOR instructions (Bit 9)
			// Set Operating System Support for Unmasked SIMD Floating-Point Exceptions (Bit 10)
			"mov %%cr4, %%rax;			" \
			"bts $9, %%rax;				" \
			"bts $10, %%rax;			" \
			"mov %%rax, %%cr4;			" \

			// Flush Cache
			"wbinvd;					" \

			// Enable Floating Point
			// Set Monitor co-processor (Bit 1)
			// Clear Emulation (Bit 2)
			"mov %%cr0, %%rax;			" \
			"bts $1, %%rax;				" \
			"btr $2, %%rax;				" \
			"mov %%rax, %%cr0;			" \

			// Init Math Co-processor
			"finit;						" \

			// Clobber used registers
			"": : : "%rcx","%rax","cc","memory");

		return;

[tsdnz]

Code: Select all

The servers crash if I just put 0x600 into rax

Silly me, lol!!

I am setting up paging etc before this point in assembly.

Code: Select all

;------------------------------------------------------------------------------

align 16
EnableExtendedProperties:

	mov		eax, cr4
	or		eax, 0x0000000B0	; PGE (Bit 7), PAE (Bit 5), and PSE (Bit 4)
	mov		cr4, eax

	ret

[tsdnz]

Here is the final Init:

Code: Select all

	FIL void tCPUs::InitThisCPU()
	{
		asm volatile (
			// Enable SSE
			// Set Operating System Support for FXSAVE and FXSTOR instructions (Bit 9)
			// Set Operating System Support for Unmasked SIMD Floating-Point Exceptions (Bit 10)
			"mov %%cr4, %%rax;			" \
			"bts $9, %%rax;				" \
			"bts $10, %%rax;			" \
			"mov %%rax, %%cr4;			" \

			// Flush Cache
			"wbinvd;					" \

			// Enable Floating Point
			// Set Monitor co-processor (Bit 1)
			// Clear Emulation (Bit 2)
			"mov %%cr0, %%rax;			" \
			"bts $1, %%rax;				" \
			"btr $2, %%rax;				" \
			"mov %%rax, %%cr0;			" \

			// Init Math Co-processor
			"finit;						" \

			// Clobber Registers
			"": : : "%rax","%rbx","%rcx","%rdx","%rsp","%rsi","%rdi","%r8","%r9","%r10","%r11","%r12","%r13","%r14","%r15","cc","memory","%st","%st(1)","%st(2)","%st(3)","%st(4)","%st(5)","%st(6)","%st(7)","%xmm0","%xmm1","%xmm2","%xmm3","%xmm4","%xmm5","%xmm6","%xmm7");
	}

[Brendan]

Here is the final Init:

Almost

The "wbinvd" is now unnecessary (and is expensive); and you aren't clobbering rbx, rcx, rdx, rsp (which I'm surprised GCC accepts), rsi, rdi, r8, r9, r10, r11, r12, r13, r14 or r15.


Cheers,

Brendan

[tsdnz]

Almost

The "wbinvd" is now unnecessary (and is expensive); and you aren't clobbering rbx, rcx, rdx, rsp (which I'm surprised GCC accepts), rsi, rdi, r8, r9, r10, r11, r12, r13, r14 or r15.

Awake to a new day, looking forward to my cold leaving me!

Yes, left that in.
Thought I would add all the other registers in for fun, a clean entry point, and tell the compiler to clobber everything.

And thanks heaps.