[Solved] VirtualBox "mov es, ax" or "mov ss, ax" faults
Posted: Sat Dec 24, 2022 12:50 am
Solved. I was using VirtualBox 6.1.38 which apparently had broken VT-D virtualization. Switching to VirtualBox 6.1.6 solved it.
This code has worked fine in VirtualBox for a long time, but recently started causing a GPF. It's aborting the VM in the very first function after the jump to long mode. I'm testing on Virtualbox 6.1.38_Ubuntu r153438. I believe this problem started with VirtualBox 6.0.
It runs fine in bochs and qemu and on real hardware. VirtualBox doesn't log any error, it just shuts down the VM with "aborted."This is (most likely) a VirtualBox bug, but I'm wondering if there's something I could try to just get the code to run - I'm looking for a workaround.
Switching the order gives a clue - it seems to be triggered when SS and RSP are set:To be clear, since VirtualBox doesn't log any information and the 'p' single step command in the VirtualBox debugger causes VirtualBox to freeze, I'm using "1: jmp 1b" as suggested on the wiki to bisect what instruction is causing the abort.
While I'm here, I'm looking for ways to convince gcc 9.4.0-1ubuntu1 to omit the prolog on this function. The push and sub rsp instructions are relatively harmless but pointless.
This code has worked fine in VirtualBox for a long time, but recently started causing a GPF. It's aborting the VM in the very first function after the jump to long mode. I'm testing on Virtualbox 6.1.38_Ubuntu r153438. I believe this problem started with VirtualBox 6.0.
It runs fine in bochs and qemu and on real hardware. VirtualBox doesn't log any error, it just shuts down the VM with "aborted."
Code: Select all
push r15 ; 4157 (gcc-generated prolog)
push r14 ; 4156 (gcc-generated prolog)
push r13 ; 4155 (gcc-generated prolog)
push r12 ; 4154 (gcc-generated prolog)
push rbp ; 55 (gcc-generated prolog)
push rbx ; 53 (gcc-generated prolog)
sub rsp, 0x000000d8 ; 4881ecd8000000 (gcc-generated prolog)
xor rax, rax ; 4831c0 (handwritten assembly begins)
push rax ; 50
popf ; 9d
mov ax, 0x0010 ; 66b81000
mov ss, ax ; 8ed0
mov rsp, 0x00007dea ; 48c7c4ea7d0000
mov ds, ax ; 8ed8
mov es, ax ; 8ec0 (VirtualBox aborts)
mov fs, ax ; 8ee0
mov gs, ax ; 8ee8
xor eax, eax ; 31c0
Switching the order gives a clue - it seems to be triggered when SS and RSP are set:
Code: Select all
xor rax, rax ; 4831c0 (handwritten assembly begins)
push rax ; 50
popf ; 9d
mov ax, 0x0010 ; 66b81000
mov ds, ax ; 8ed8 (Rearranged this, now DS and ES come before SS)
mov es, ax ; 8ec0
mov ss, ax ; 8ed0 (VirtualBox aborts)
mov rsp, 0x00007dea ; 48c7c4ea7d0000
mov fs, ax ; 8ee0
mov gs, ax ; 8ee8
xor eax, eax ; 31c0
While I'm here, I'm looking for ways to convince gcc 9.4.0-1ubuntu1 to omit the prolog on this function. The push and sub rsp instructions are relatively harmless but pointless.