long mode activation and VMware Fusion vs VirtualBox

[mateuszb]

I am in the 32 bit protected mode with paging enabled and have the following problem.

I set up long mode as follows:

1. disable paging in CR0
2. enable PAE in CR4
3. load CR3 with PML4 root
4. enable long mode in Extended Feature MSR
5. enable paging

This should leave me in compatibility mode. It works in VMware Fusion but it does not in Virtual Box.

I identity map only the first 1GB. The PD with 2MB pages is located at 0x100000.
The PDPT is located at 0x101000, and the PML4 is located at 0x102000.

I set CR3 to 0x102000, here's memory dump for PML4

VBoxDbg> dq 0x102000
%0000000000102000: 0000000000101001 0000000000000000

VBoxDbg> dq 0x101000
%0000000000101000: 0000000000100001 0000000000000000

VBoxDbg> dq 0x100000
%0000000000100000: 0000000000000081 0000000000200081
%0000000000100010: 0000000000400081 0000000000600081
...

After enabling a paging bit in CR0 in Virtualbox the code stops to run.
I can only access memory up to 0x1FFFFF but it appears as everything is zeroed out.

EIP has value of 0x82cc which in my case is the address where a dummy interrupt handler is located.
In addition, ESP is set to 0 after the problem occurs.

I've done this in the past but I'm doing something wrong now and I can't figure out what.
Suggestions are welcome. I'm really stuck on this one now.

Here is a register dump right before enabling paging:

eax=00000000 ebx=00000048 ecx=c0000080 edx=00000000 esi=00000a48 edi=9e1c0000
eip=0000837c esp=00000900 ebp=00000900 iopl=0 nv up di pl zr na po nc
cs={0008 base=00000000 limit=ffffffff flags=c09a} dr0=00000000 dr1=00000000
ds={0010 base=00000000 limit=ffffffff flags=c093} dr2=00000000 dr3=00000000
es={0010 base=00000000 limit=ffffffff flags=c093} dr6=ffff0ff0 dr7=00000400
fs={0010 base=00000000 limit=ffffffff flags=c093} cr0=00000011 cr2=00000000
gs={0010 base=00000000 limit=ffffffff flags=c093} cr3=00102000 cr4=00000020
ss={0018 base=00000000 limit=ffffffff flags=c093} cr8=00000000
gdtr=00000a90:001f idtr=00000ab0:00ff eflags=00000046
ldtr={0000 base=00000000 limit=0000ffff flags=0082}
tr ={0000 base=00000000 limit=0000ffff flags=008b}
sysenter={cs=0000 eip=00000000 esp=00000000}
fcw=037f fsw=0000 ftw=ffff mxcsr=1f80 mxcsr_mask=0000
0008:0000837c 0f 20 c0 mov eax, cr0


Here is register dump after the problem occurs:

eax=80000011 ebx=00000048 ecx=c0000080 edx=00000000 esi=00000a48 edi=9e1c0000
eip=000082cc esp=00000000 ebp=00000900 iopl=0 nv up di ng nz na po nc
cs={0008 base=00000000 limit=ffffffff flags=c09a} dr0=00000000 dr1=00000000
ds={0010 base=00000000 limit=ffffffff flags=c093} dr2=00000000 dr3=00000000
es={0010 base=00000000 limit=ffffffff flags=c093} dr6=ffff0ff0 dr7=00000400
fs={0010 base=00000000 limit=ffffffff flags=c093} cr0=80000011 cr2=80000011
gs={0010 base=00000000 limit=ffffffff flags=c093} cr3=00102000 cr4=00000020
ss={0018 base=00000000 limit=ffffffff flags=c093} cr8=00000000
gdtr=00000a90:001f idtr=00000ab0:00ff eflags=00000086
ldtr={0000 base=00000000 limit=0000ffff flags=0082}
tr ={0000 base=00000000 limit=0000ffff flags=008b}
sysenter={cs=0000 eip=00000000 esp=00000000}
fcw=037f fsw=0000 ftw=ffff mxcsr=1f80 mxcsr_mask=0000

Here is GDT dump

VBoxDbg> Guest GDT (GCAddr=0000000000000a90 limit=1f):
0008 - 0000ffff 00cf9a00 - base=00000000 limit=ffffffff dpl=0 CodeER Present Page 32-bit
0010 - 0000ffff 00cf9300 - base=00000000 limit=ffffffff dpl=0 DataRW Accessed Present Page 32-bit
0018 - 0000ffff 00cf9300 - base=00000000 limit=ffffffff dpl=0 DataRW Accessed Present Page 32-bit

[xenos]

Could you provide some source code? It would probably be helpful to see the code in which the switch to long mode is done.

The fact that you end up in an interrupt handler suggest that something is going wrong, i.e., an exception occurs. Could you figure out which exception it is, and where in your code it happens?

I also use VBox as one of my testing platforms for long mode, but I haven't used its debug console yet. (Besides that, I don't stay in compatibility mode - the next to instructions after enabling long mode are loading a new GDT and jumping to a long mode code segment.) If possible, I would try to single-step through the long mode switching code to figure out what happens, and why.

[mateuszb]

I plan on switching to native mode but first I have to get this bug out of the way.

Here's the code

Code: Select all

	mov eax, cr4
	or eax, 1 << 5
	mov cr4, eax

	;; Load CR3
	mov eax, dword [PML4Root]
	mov cr3, eax
	
	mov ecx, 0xC0000080
	rdmsr
	or eax, 1 << 8
	wrmsr

	xor eax, eax
	mov cr2, eax

	nop
	;; set Paging bit
	mov eax, cr0
	or eax, 1 << 31
	mov cr0, eax

Here's the single stepping output:

VBoxDbg> rg
eax=80000011 ebx=00000048 ecx=c0000080 edx=00000000 esi=00000a48 edi=9e1c0000
eip=00008388 esp=00000900 ebp=00000900 iopl=0 nv up di ng nz na po nc
cs={0008 base=00000000 limit=ffffffff flags=c09a} dr0=00000000 dr1=00000000
ds={0010 base=00000000 limit=ffffffff flags=c093} dr2=00000000 dr3=00000000
es={0010 base=00000000 limit=ffffffff flags=c093} dr6=ffff0ff0 dr7=00000400
fs={0010 base=00000000 limit=ffffffff flags=c093} cr0=00000011 cr2=00000000
gs={0010 base=00000000 limit=ffffffff flags=c093} cr3=00102000 cr4=00000020
ss={0018 base=00000000 limit=ffffffff flags=c093} cr8=00000000
gdtr=00000a90:001f idtr=00000ab0:00ff eflags=00000086
ldtr={0000 base=00000000 limit=0000ffff flags=0082}
tr ={0000 base=00000000 limit=0000ffff flags=008b}
sysenter={cs=0000 eip=00000000 esp=00000000}
fcw=037f fsw=0000 ftw=ffff mxcsr=1f80 mxcsr_mask=0000

0008:00008388 0f 22 c0 mov cr0, eax

right after that instruction is single stepped I get

VBoxDbg> u32
Failed to disassemble instruction, skipping one byte.
u32: error: Too many disassembly failures. Giving up: VINF_SUCCESS
VBoxDbg> dq 0x8388
error: VERR_PAGE_TABLE_NOT_PRESENT: Reading memory at %0000000000008388.
VBoxDbg>

[xenos]

This is strange... According to the error message, there is no page table for 0x8388. But the page table looks fine to me. CR3 points to the PML4T at 0x102000, the first quadword points to the PDPT at 0x101000, the first quadword points to the PDT at 0x100000, and that one seems to be correctly filled with 2MB pages. What is the output if you enter "info mode" in the debugger, before and after enabling paging?

Have you checked your code with other simulators besides VBox and VMWare? I use VBox, Bochs, QEMU and SimNow! for testing long mode code. Sometimes one of them shows a problem which does not occur on the others, and it is often helpful to compare the different debugger outputs.

EDIT: I just tested my code in VBox debugger and I also get some strange error messages. First, it shows me only 32 bit registers, although I'm in long mode. Then it tells me that it can't disassemble the next instruction - of course, because there is not even a page table for 0x105132. But I know that there is a page table for 0x0xffffc00000105132 because I just stopped the VM while my kernel was running at that address, without any problems.

Code: Select all

VBoxDbg> stop
dbgf event: VM 00007f48f122c000 is halted! (other)
eax=00000001 ebx=00105130 ecx=00000000 edx=00000005 esi=000000a0 edi=001109e8
eip=00105132 esp=00112fd8 ebp=00109870 iopl=0 nv up ei ng nz na pe nc
cs=0020 ds=0000 es=0000 fs=0000 gs=0000 ss=0000               eflags=00000282
Failed to disassemble instruction, skipping one byte.

VBoxDbg> u64
Failed to disassemble instruction, skipping one byte.

VBoxDbg> dq 105132
error: VERR_PAGE_TABLE_NOT_PRESENT:  Reading memory at %0000000000105132.

VBoxDbg> dq ffffc00000105132
%ffffc00000105132: 00000000000000c3 b848000000000000
%ffffc00000105142: ffffc00000113800 c1489066c3078948
%ffffc00000105152: 46c74837034804e6 06c7480000000008
%ffffc00000105162: 1f0f90c300000000 8b48000000000084
%ffffc00000105172: 3044c604e6c14807 306480078b488905
%ffffc00000105182: 0c8966078b480f06 078b4c10e9c14830

VBoxDbg> info mode
Guest paging mode:  AMD64, changed 48035 times, A20 enabled
Shadow paging mode: EPT
Host paging mode:   AMD64+G+NX

[mateuszb]

I have had problems with VirtualBox on a new 2011 macbook pro so I had to disable virtualization feature in VBox which apparently cripples long mode (in fact simulates a CPU which does not support it), and I have not been checking EFER as I was sure the machine supports long mode. I had to re-enable the feature and everything works fine.

[quok]

I have not been checking EFER as I was sure the machine supports long mode.

Ah, assumptions. They'll get you every time.