VirtualBox Triple Fault on far jump to 64bit code on AP core

[Oxmose]

Hi everyone,

I managed to switch to long mode and execute my kernel on the main core (works on Qemu x86-64 with 4 cores).
However, when I am enabling cores 1, 2 and 3 on VirtualBox, I get a Guru meditation. CPU 1 is triple faulting on the far jump to the 64bits part of the core boot code.
Everything works fine on Qemu though.

Here is the dump from virtualBox:

Code: Select all

dbgf event/1: DBGFSTOP (hyper)
File:     VINF_EM_TRIPLE_FAULT
Line:     0
Function: <NULL>
eax=00000000 ebx=0000808d ecx=0000808d edx=00000000 esi=00000000 edi=00000000
eip=00008086 esp=00000000 ebp=00000000 iopl=0 rf nv up di ng nz na po nc
cs=0008 ds=0010 es=0010 fs=0010 gs=0010 ss=0010               eflags=00010086
0008:00008086 ea 8d 80 00 00 08 00    jmp far 00008h:00000808dh

My GDT for 64 bits looks like this and is correctly loaded:

Code: Select all

VBoxDbg> dg
0008 CodeEO Bas=00000000 Lim=ffffffff DPL=0 P  NA G     AVL=0 L=1
0010 DataRW Bas=00000000 Lim=ffffffff DPL=0 P  NA G BIG AVL=0 L=0
0018 VERR_INVALID_SELECTOR

And my paging looks like this (and is correctly loaded as I am dumping this from the VBox debugger after loading it):

Code: Select all

VBoxDbg> dpt 0x8000
%%000000000034e040 (base %0000000000008000 / index 0x8):
008 %0000000000008000: 0000000000008021 4kb phys=0000000000008000 p  r s a  nd avl=00                 
009 %0000000000009000: 0000000000000000 4kb phys=0000000000000000 np r s na nd avl=00                 
00a %000000000000a000: 0000000000000000 4kb phys=0000000000000000 np r s na nd avl=00                 
00b %000000000000b000: 0000000000000000 4kb phys=0000000000000000 np r s na nd avl=00                 
00c %000000000000c000: 0000000000000000 4kb phys=0000000000000000 np r s na nd avl=00                 
00d %000000000000d000: 0000000000000000 4kb phys=0000000000000000 np r s na nd avl=00                 
00e %000000000000e000: 0000000000000000 4kb phys=0000000000000000 np r s na nd avl=00                 
00f %000000000000f000: 0000000000000000 4kb phys=0000000000000000 np r s na nd avl=00                 
010 %0000000000010000: 0000000000000000 4kb phys=0000000000000000 np r s na nd avl=00                 
011 %0000000000011000: 0000000000000000 4kb phys=0000000000000000 np r s na nd avl=00                 
012 %0000000000012000: 0000000000000000 4kb phys=0000000000000000 np r s na nd avl=00

And here is the full dump at the moment of the triple fault:

Code: Select all

00:00:11.943166 Guest CPUM (VCPU 1) state: 
00:00:11.943168 eax=00000000 ebx=0000808d ecx=0000808d edx=00000000 esi=00000000 edi=00000000
00:00:11.943169 eip=00008086 esp=00000000 ebp=00000000 iopl=0      rf nv up di nt zr na po nc
00:00:11.943169 cs={0008 base=0000000000000000 limit=ffffffff flags=0000c09b} dr0=00000000 dr1=00000000
00:00:11.943171 ds={0010 base=0000000000000000 limit=ffffffff flags=0000c093} dr2=00000000 dr3=00000000
00:00:11.943172 es={0010 base=0000000000000000 limit=ffffffff flags=0000c093} dr4=00000000 dr5=00000000
00:00:11.943172 fs={0010 base=0000000000000000 limit=ffffffff flags=0000c093} dr6=ffff0ff0 dr7=00000400
00:00:11.943173 gs={0010 base=0000000000000000 limit=ffffffff flags=0000c093} cr0=80010033 cr2=00000080
00:00:11.943174 ss={0010 base=0000000000000000 limit=ffffffff flags=0000c093} cr3=00133000 cr4=00000620
00:00:11.943175 gdtr=0000000000008120:0017  idtr=0000000000000000:ffff  eflags=00010086
00:00:11.943176 ldtr={0000 base=00000000 limit=0000ffff flags=00000082}
00:00:11.943176 tr  ={0000 base=00000000 limit=0000ffff flags=0000008b}
00:00:11.943177 SysEnter={cs=0000 eip=00000000 esp=00000000}
00:00:11.943199 xcr=0000000000000001 xcr1=0000000000000000 xss=0000000000000000 (fXStateMask=0000000000000000)
00:00:11.943201 FCW=037f FSW=0000 FTW=0000 FOP=0000 MXCSR=00001f80 MXCSR_MASK=0000ffff
00:00:11.943201 FPUIP=00000000 CS=0000 Rsrvd1=0000  FPUDP=00000000 DS=0000 Rsvrd2=0000
00:00:11.943203 ST(0)=FPR0={0000'00000000'00000000} t0 +0.0000000000000000000000 * 2 ^ -16383 (*)
00:00:11.943206 ST(1)=FPR1={0000'00000000'00000000} t0 +0.0000000000000000000000 * 2 ^ -16383 (*)
00:00:11.943207 ST(2)=FPR2={0000'00000000'00000000} t0 +0.0000000000000000000000 * 2 ^ -16383 (*)
00:00:11.943209 ST(3)=FPR3={0000'00000000'00000000} t0 +0.0000000000000000000000 * 2 ^ -16383 (*)
00:00:11.943210 ST(4)=FPR4={0000'00000000'00000000} t0 +0.0000000000000000000000 * 2 ^ -16383 (*)
00:00:11.943211 ST(5)=FPR5={0000'00000000'00000000} t0 +0.0000000000000000000000 * 2 ^ -16383 (*)
00:00:11.943212 ST(6)=FPR6={0000'00000000'00000000} t0 +0.0000000000000000000000 * 2 ^ -16383 (*)
00:00:11.943214 ST(7)=FPR7={0000'00000000'00000000} t0 +0.0000000000000000000000 * 2 ^ -16383 (*)
00:00:11.943215 XMM0 =00000000'00000000'00000000'00000000  XMM1 =00000000'00000000'00000000'00000000
00:00:11.943217 XMM2 =00000000'00000000'00000000'00000000  XMM3 =00000000'00000000'00000000'00000000
00:00:11.943218 XMM4 =00000000'00000000'00000000'00000000  XMM5 =00000000'00000000'00000000'00000000
00:00:11.943220 XMM6 =00000000'00000000'00000000'00000000  XMM7 =00000000'00000000'00000000'00000000
00:00:11.943221 XMM8 =00000000'00000000'00000000'00000000  XMM9 =00000000'00000000'00000000'00000000
00:00:11.943223 XMM10=00000000'00000000'00000000'00000000  XMM11=00000000'00000000'00000000'00000000
00:00:11.943225 XMM12=00000000'00000000'00000000'00000000  XMM13=00000000'00000000'00000000'00000000
00:00:11.943226 XMM14=00000000'00000000'00000000'00000000  XMM15=00000000'00000000'00000000'00000000
00:00:11.943228 EFER         =0000000000000d00
00:00:11.943228 PAT          =0007040600070406
00:00:11.943229 STAR         =0000000000000000
00:00:11.943229 CSTAR        =0000000000000000
00:00:11.943229 LSTAR        =0000000000000000
00:00:11.943229 SFMASK       =0000000000000000
00:00:11.943230 KERNELGSBASE =0000000000000000

The AP core kickstart is loaded at address 0x8000 and is the following:

Code: Select all

;-------------------------------------------------------------------------------
; INCLUDES
;-------------------------------------------------------------------------------
%include "config.inc"

;-------------------------------------------------------------------------------
; ARCH
;-------------------------------------------------------------------------------
[bits 16]

;-------------------------------------------------------------------------------
; DEFINES
;-------------------------------------------------------------------------------

%define CODE_LOCATION 0x8000
%define OFFSET_ADDR(ADDR)  (((ADDR) - __kinitApCores) + CODE_LOCATION)

%define CODE32 0x08
%define DATA32 0x10
%define CODE64 0x08
%define DATA64 0x10

;-------------------------------------------------------------------------------
; MACRO DEFINE
;-------------------------------------------------------------------------------

; None

;-------------------------------------------------------------------------------
; EXTERN DATA
;-------------------------------------------------------------------------------

extern _KERNEL_STACKS_BASE
extern _kernelPGDir
extern _bootedCPUCount
extern _fcw

;-------------------------------------------------------------------------------
; EXTERN FUNCTIONS
;-------------------------------------------------------------------------------

extern cpuApInit

;-------------------------------------------------------------------------------
; EXPORTED FUNCTIONS
;-------------------------------------------------------------------------------

; None

;-------------------------------------------------------------------------------
; EXPORTED DATA
;-------------------------------------------------------------------------------
; None

;-------------------------------------------------------------------------------
; CODE
;-------------------------------------------------------------------------------

section .low_ap_startup_code
;-------------------------------------------------------------------------------
; Kernel entry AP point
__kinitApCores:
    cli

    ; Set GDT
    lgdt [OFFSET_ADDR(_gdt16Ptr)]

    ; Set PE bit in cr0
    mov eax, cr0
    or  eax, 0x1
    mov cr0, eax

    ; Jump to protected mode
    jmp dword CODE32:OFFSET_ADDR(__kinitApPM)

[bits 32]
__kinitApPM:
    cli

    ; Set Segments
    mov ax, DATA32
    mov ds, ax
    mov es, ax
    mov fs, ax
    mov gs, ax
    mov ss, ax

    ; Init FPU
    fninit
    fldcw [_fcw]

    ; Enable SSE
    mov eax, cr0
    and al, ~0x04
    or  al, 0x22
    mov cr0, eax
    mov eax, cr4
    or  ax, 0x600
    mov cr4, eax

    ; Enable PAE
    mov eax, cr4
    or  eax, 0x20
    mov cr4, eax

    ; Switch to compatibility mode and NXE
    mov ecx, 0xC0000080
    rdmsr
    or  eax, 0x00000900
    wrmsr

    ; Set CR3
    mov eax, (_kernelPGDir - KERNEL_MEM_OFFSET)
    mov cr3, eax

    ; Enable paging and caches
    mov eax, cr0
    and eax, 0x0FFFFFFF
    or  eax, 0x80010000
    mov cr0, eax

    ; Set GDT
    lgdt [OFFSET_ADDR(_gdt64Ptr)]

    ; Far jump to 64 bit mode
    jmp CODE64:OFFSET_ADDR(__kinitApPM64bEntry) <------------------- Triple Fault here on VirtualBox

[bits 64]
__kinitApPM64bEntry:
    mov rax, __kinitApPM64bEntry    <--------------------- Not in the base code, but here just to ease debugging on Qemu at the moment (when removed, CPU boots correctly on Qemu)
    jmp rax

    ; Get our CPU id based on the booted CPU count and update booted CPU count
    mov ecx, [_bootedCPUCount]
    mov gs, ecx ; GS stores teh CPU ID
    mov eax, ecx
    add eax, 1
    mov [_bootedCPUCount], eax

    ; Set the stack base on the CPU id
    mov rbx, KERNEL_STACK_SIZE
    mul rbx
    mov rbx, _KERNEL_STACKS_BASE - 16
    add rax, rbx
    mov rsp, rax
    mov rbp, rsp

    ; RCX contains the CPU ID, set as first parameter
    mov rdi, rcx
    mov rax, cpuApInit
    call rax

; We should never return
__kinitAp64PMEnd:
    cli
    hlt
    jmp __kinitAp64PMEnd

align 32
; Temporary GDT for AP
_gdt16:
    .null:
        dd 0x00000000
        dd 0x00000000
    .code_32:
        dw 0xFFFF
        dw 0x0000
        db 0x00
        db 0x9A
        db 0xCF
        db 0x00
    .data_32:
        dw 0xFFFF
        dw 0x0000
        db 0x00
        db 0x92
        db 0xCF
        db 0x00
    .code_16:
        dw 0xFFFF
        dw 0x0000
        db 0x00
        db 0x9A
        db 0x0F
        db 0x00
    .data_16:
        dw 0xFFFF
        dw 0x0000
        db 0x00
        db 0x92
        db 0x0F
        db 0x00

_gdt16Ptr:                                 ; GDT pointer for 16bit access
    dw _gdt16Ptr - _gdt16 - 1              ; GDT limit
    dd _gdt16                              ; GDT base address

align 32
; Temporary GDT for AP
_gdt64:
    .null:
        dd 0x00000000
        dd 0x00000000

    .code_64:
        dq 0x00AF98000000FFFF
    .data_64
        dq 0x00CF92000000FFFF
_gdt64Ptr:                                 ; GDT pointer for 16bit access
    dw _gdt64Ptr - _gdt64 - 1              ; GDT limit
    dd _gdt64                              ; GDT base address

Again, everything is fine when using Qemu and Core0 also starts correctly on VirtualBox. So I suspect something with the virtualization is going on. Also the AP startup code really loks like the same as the main core startup (but for AP is added the PM switch but that is pretty much it).

Do you have any ideas on what could happen?

[Octocontrabass]

You need to set the "accessed" bit in your segment descriptors. (For example, use "dq 0x00AF99000000FFFF" for your 64-bit code segment.)

When the CPU loads a segment descriptor from a descriptor table, it checks the "accessed" bit and updates the descriptor to set that bit if it's not already set. Your GDT is in a read-only page, so the write to update the "accessed" bit causes a page fault.

[Oxmose]

Damn, how could I miss that! Thanks a lots, it works now

Another bug of Qemu (or missing check I'd say) to report.

Thanks!