SMP startup on x86_64 fails on AP with triple-fault

[max]

After debugging this for a very long time I'm kind of going crazy so I thought I'd post here maybe someone else has an idea

So I'm working on a port from x86 to x86_64 (https://github.com/maxdev1/ghost/pull/48) and it all looks fine on single-core so far, but when starting up the AP I'm triple-faulting when trying to access some memory in kernel space.

I'm booting up with Limine and just passing on the initial page directory to the AP. My AP startup code is this:

Code: Select all

; Loaded by the kernel to this location
org 0x1000

; Real mode
BITS 16
startup:
    ; No interrupts & clear segments
    cli
    xor ax, ax
    mov ds, ax
    mov es, ax
    mov ss, ax

    ; Load the GDT
    lgdt [gdtPointer]

    ; Enable protected mode
    mov eax, cr0
    or eax, 1
    mov cr0, eax

    ; Far-JMP into protected mode code
    jmp 0x8:protectedStart


; Protected mode
BITS 32
protectedStart:
    ; Code segments were set by far JMP, set data segments
    mov ax, 0x10
    mov ds, ax
    mov es, ax
    mov fs, ax
    mov gs, ax
    mov ss, ax

    ; Lock all cores
    acquireLock:
    lock bts dword [interlock], 0
    jc acquireLock

    ; Enable PAE and PGE
    mov eax, cr4
    or eax, (1 << 5) | (1 << 7)
    mov cr4, eax

    ; Set page directory
    mov eax, [0x500]
    mov cr3, eax

    ; Set up EFER MSR to enable long mode (EFER.LME)
    mov ecx, 0xC0000080
    rdmsr
    or eax, 1 << 8
    wrmsr

    ; Enable paging
    mov eax, cr0
    or eax, 1 << 31
    mov cr0, eax

    ; Get index of this AP and increment counter
    mov eax, [0x510]
    inc dword [0x510]

    ; Release lock
    lock btr dword [interlock], 0

    ; Far-jump to 64 bits
    jmp 0x18:longStart

; Long mode
[BITS 64]
longStart:
    ; 32-bit code provides AP index in EAX
    ; From the stack array, take the entry at offset multiplied by 8
    shl rax, 3
    mov rsp,  [0x518 + rax]
    mov rbp, rsp

    ; Set data segments
    mov bx, 0x20
    mov ds, bx
    mov es, bx
    mov fs, bx
    mov gs, bx
    mov ss, bx

    ; Jump into kernel code
    mov rax, [0x508]
    jmp rax

; Inter-core synchronization
interlock:
    dd 0

; Basic setup GDT
gdt:
    ; null descriptor
    dw 0x0000
    dw 0x0000
    dw 0x0000
    dw 0x0000

    ; code descriptor
    dw 0xFFFF
    dw 0x0000
    dw 0x9A00
    dw 0x00CF

    ; data descriptor
    dw 0xFFFF
    dw 0x0000
    dw 0x9200
    dw 0x00CF

    ; 64-bit code descriptor
    dw 0x0000
    dw 0x0000
    dw 0x9A00
    dw 0x0020

    ; 64-bit data descriptor
    dw 0x0000
    dw 0x0000
    dw 0x9200
    dw 0x0000

gdtEnd:

align 4
gdtPointer:
    dw gdtEnd - gdt - 1
    dd gdt
    

Basically the BSP provides the PML4 physical address in 0x500; a counter in 0x510 for each AP to know its index; as well as a stack for each AP in [0x518 + index]. Kernel entry address is stored in 0x508.

I would say the values are read correctly on the AP - but it's faulting like this:

Code: Select all

rax=ffffffff80600004 rbx=0000000000000580 rcx=00000000c0000080 rdx=0000000000000000
rsi=0000000000000000 rdi=0000000000000000 r8 =0000000000000000 r9 =0000000000000000
r10=0000000000000000 r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 iopl=0 rf nv up di ng nz na pe nc
rip=ffffffff8020c77e rsp=ffffff809000ffe8 rbp=ffffff809000fff8
cs=0018 ds=0020 es=0020 fs=0020 gs=0020 ss=0020                     rflags=4f6a368a00010082
%ffffffff8020c77e 8b 00                   mov eax, dword [rax]
// ^ this goes wrong, an attempt to read from the faulty address below

cpu1.cr0=0xe0000011 // PE, ET, NW, CD, PG are set
cpu1.cr2=0xffffffff80600004 // faulty address
cpu1.cr3=0x00000000dff86000 // same phys address of PML4 that BSP uses
cpu1.cr4=0x000000a0 // PGE & PAE set
cpu1.efer=0x00000500 // LME & LMA set

I dumped the paging entries for the faulty address 0xffffffff80600004 on BSP and it looks like this:

Code: Select all

PML4[511] = 0x00000000dff85027 flags: PWU--A---
  PDPT[510] = 0x00000000dff84027 flags: PWU--A---
    PD[3] = 0x00000000dff80027 flags: PWU--A---
      PT[0] = 0x80000000df735063 flags: PW---AD--

I'm lost, I think I've successfully switched to long mode, the page looks mapped and should be accessible, BSP can access the exact same address - did I mess something else up in my startup sequence? Pretty much the same worked fine on x86, but maybe there is a pitfall for x86_64 that I'm missing? I'm losing my mind lol thank you for any help on this

[sebihepp]

How do you start the APs? If I understood limines documentation correctly, you only need to write to

Code: Select all

limine_mp_response->limine_mp_info[i]->goto_address

and the AP jumps there already in long mode.

[max]

How do you start the APs? If I understood limines documentation correctly, you only need to write to

Code: Select all

limine_mp_response->limine_mp_info[i]->goto_address

and the AP jumps there already in long mode.

Basically with the classic INIT/SIPI startup sequence after reading the MADT: https://github.com/maxdev1/ghost/blob/x ... em/smp.cpp
I was expecting that limine doesn‘t tamper with the APs if I don‘t explicitly enable anything there but I might want to doublecheck. I kind of feel like my error is not related though since vbox debugger tells me cpu1 is failing here

[Octocontrabass]

Code: Select all

cpu1.cr0=0xe0000011 // PE, ET, NW, CD, PG are set

You probably should clear NW and CD. On some CPUs, setting NW disables cache coherency.

Code: Select all

      PT[0] = 0x80000000df735063 flags: PW---AD--

If you haven't set EFER.NXE, bit 63 of the page table entry is reserved and causes a page fault.

Pretty much the same worked fine on x86, but maybe there is a pitfall for x86_64 that I'm missing?

When I was browsing through your code I saw one or two things that weren't correctly updated for 64-bit mode.

[max]

Code: Select all

cpu1.cr0=0xe0000011 // PE, ET, NW, CD, PG are set

You probably should clear NW and CD. On some CPUs, setting NW disables cache coherency.

Code: Select all

      PT[0] = 0x80000000df735063 flags: PW---AD--

If you haven't set EFER.NXE, bit 63 of the page table entry is reserved and causes a page fault.

Pretty much the same worked fine on x86, but maybe there is a pitfall for x86_64 that I'm missing?

When I was browsing through your code I saw one or two things that weren't correctly updated for 64-bit mode.

Unbelievable. I set EFER.NXE and its working. How? How could you tell so quickly? You're amazing! Thank you so much for pointing this out.

Also thanks for the other issues. There is probably more I didn't consider yet, still some work to do. I will check them!

[Octocontrabass]

How could you tell so quickly?

You're not the first person to run into this. That leading "8" instead of "0" on the page table entry is hard to see in some fonts, though.

Also thanks for the other issues. There is probably more I didn't consider yet, still some work to do. I will check them!

They're nothing major, just an incorrect comment (SS/RSP are always pushed/popped) and a function you can delete (CPUID is always available in 64-bit mode).