[SOLVED] General Protection fault in higher half setup code.

[closos]

After writing a bunch of memory allocation code and whatnot, I realized that if I wanted to continue I should make my kernel higher-half mapped. So, I went back to the bootloader and the linker and modified them to allow for this. However, I'm getting a really strange #GP with error code 008. I manually inspected the page tables and I swear I didn't see any reserved bit which was set.

possibly important to note is that I'm loading this code with GRUB, on QEMU running a BIOS machine. I debug with lldb. The asm file with my multiboot2 header is not attached, but it also sets up a temporary stack. before calling enterLongMode.

I checked ancient posts here and couldn't find anything helpful. So, any help would be greatly appreciated.

GP fault:

Code: Select all

check_exception old: 0xffffffff new 0xd
     0: v=0d e=0008 i=0 cpl=0 IP=0010:0000000000007d52 pc=0000000000007d52 SP=0018:00000000001269b0 env->regs[R_EAX]=0000000080000011
EAX=80000011 EBX=0027fec0 ECX=c0000080 EDX=00000000
ESI=00000000 EDI=00127000 EBP=00000000 ESP=001269b0
EIP=00007d52 EFL=00000086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0018 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
CS =0010 00000000 ffffffff 00cf9a00 DPL=0 CS32 [-R-]
SS =0018 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
DS =0018 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
FS =0018 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
GS =0018 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT=     000000004e05e800 00000000
IDT=     0000000000000000 00000000
CR0=80000011 CR2=0000000000000000 CR3=0000000000127000 CR4=000000b0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=00200000 CCD=80000011 CCO=LOGICL
EFER=0000000000000500

long_mode.asm:

Code: Select all

section .lowtext
global enterLongmode

; This file serves to transition from 32 bit protected mode to 64 bit long mode
BITS 32

PAGE_TABLE_SIZE equ 4096 ; bytes

; Page table addresses are 4KiB aligned (1024 address bytes, or 0x1000)
; Because of this, the lower 12 bits are set as flags. They're zeroed for the purpose of physical address resolution.
PT_ADDR_MASK equ 0xfffff000
PAGE_PRESENT equ 1 << 0  ; Marks the entry as pointing to a real table
PAGE_RW equ 1 << 1       ; Marks the page as writeable
PAGE_PS equ 1 << 7       ; Marks large pages (in the PDT rather than the PT)

; In protected, each table stores 1024 entries at 32 bits each.
; There are 3 tables. However, we are setting up long.
; In long, there are four tables, 4KiB each, with 512 entires each.
; Each entry in the PT references a 4KiB page.
; Each higher page references an entire lower page table

; PML4T (256 TiB total)
; PDPT (512 GiB total)
; PDT (1 GiB total)
; PT (2 MiB total)

; Load (from the linker) the earliest safe location for the initial pages
extern __pml4t_addr
extern __kernel_start
extern __kernel_data_end
createPageTables:
  ; We are creating a basic page map.
  ; First, the PML4T will have a single entry, mapped to the immediately following PDPT, etc.
  ; Finally, we will fill the PDT with large page entries, identity mapping 1 GiB of lower half memory
  ; After, we add a second PML4T entry, to higher half map the kernel.
  ; So 1 PMlT at 0x0000, one PDPT at 0x1000, one PDT at 0x2000, one PDPT at 0x3000, and a last PDT at 0x4000

  ; =====================
  ; Blank the page tables
  ; =====================
  mov EDI, __pml4t_addr
  xor EAX, EAX
  mov ECX, (PAGE_TABLE_SIZE * 5) / 4
  ; rep stosd writes ECX number of double words, or 4 bytes.
  ; So we divide ECX by 4
  mov DWORD [EDI], 0
  rep stosd 

  ; This takes the PDPT address, masks it, marks the proper flags,
  ; And inserts it into the first entry in the PML4T (base address no offset)
  ; Then, we repeat it for the first two tables
  mov EDI, __pml4t_addr
  mov EAX, __pml4t_addr
  add EAX, 0x1000 ; Add 4K to get the PDPT address
  and EAX, PT_ADDR_MASK ; Mask it out and set the proper bits
  or EAX, PAGE_PRESENT | PAGE_RW 
  mov DWORD [edi], EAX

  ; Add first entry to the PDPT
  mov EDI, __pml4t_addr
  add EDI, 0x1000 ; The PDPT address
  mov EAX, __pml4t_addr
  add EAX, 0x2000 ; The PDT address
  and EAX, PT_ADDR_MASK ; Mask it out and set the proper bits
  or EAX, PAGE_PRESENT | PAGE_RW 
  mov DWORD [edi], EAX

  ; Now, we could do the same for the PDT.
  ; But, maybe it'd be nicer to instead use 2MiB large pages for now.
  ; After all, this is temporary. We could change it later.

  ; ============================
  ; Identity map the first 1 GiB
  ; ============================

  ; Prepare the first entry
  lea EAX, 0x0
  and EAX, PT_ADDR_MASK ; The mask is 64 bit, but EAX is 32 bit, so we mask the mask
  or EAX, PAGE_PRESENT | PAGE_RW | PAGE_PS
  
  mov EDI, __pml4t_addr
  add EDI, 0x2000 ; The PDT address

  mov ECX, 512 ; Number of entries in the PDT
  .loopPDT:
    mov DWORD [edi], EAX

    ; Increment the entry by 2 MiB, then increment the string pointer by 8 (size of entry)
    ; We only write the lower 4 of each entry, but that's fine, 
    ; because we only map 1 Gib and not more than 4 GiB
    add EAX, 0x00200000
    lea EDI, [EDI + 8]

    loop .loopPDT

  ; ====================================
  ; Apply the recursive page table trick
  ; ====================================
  lea EDI, [__pml4t_addr + (511*8)]
  mov EAX, __pml4t_addr
  and EAX, PT_ADDR_MASK ; Mask it out and set the proper bits
  or EAX, PAGE_PRESENT | PAGE_RW
  mov [EDI], EAX
  
  ret

mapHigherHalf:
  ; =======================
  ; Prepare the higher half
  ; =======================

  ; PML4 index for 0xFFFF800000000000 is 256
  ; So: PML4[256] will point to the PDPT at 0x3000
  mov EDI, __pml4t_addr
  add EDI, 256 * 8
  mov EAX, __pml4t_addr
  add EAX, 0x3000            ; PDPT for higher-half
  and EAX, PT_ADDR_MASK
  or EAX, PAGE_PRESENT | PAGE_RW
  mov [EDI], EAX             ; PML4[256] = PDPT

  ; PDPT[0] to PDT at 0x4000
  mov EDI, __pml4t_addr
  add EDI, 0x3000            ; PDPT for higher half
  mov EAX, __pml4t_addr
  add EAX, 0x4000            ; PDT for higher-half
  and EAX, PT_ADDR_MASK
  or EAX, PAGE_PRESENT | PAGE_RW
  mov [EDI], EAX

  ; ===================
  ; Map the kernel code
  ; ===================

  ; We need to calculate how many entries to add.
  lea eax, [__kernel_data_end]   
  lea ebx, [__kernel_start]      
  sub eax, ebx ; eax = size = end - start
  mov ecx, eax 
  shr ECX, 21 ; Divide by 2 MiB

  ; Prepare the first entry
  lea EAX, [__kernel_start]
  and EAX, PT_ADDR_MASK ; The mask is 64 bit, but EAX is 32 bit, so we mask the mask
  or EAX, PAGE_PRESENT | PAGE_RW | PAGE_PS
  
  mov EDI, __pml4t_addr
  add EDI, 0x4000 ; The PDT address, for higher half

  .loopHigherHalf:
    mov DWORD [edi], EAX

    ; Increment the entry by 2 MiB, then increment the string pointer by 8 (size of entry)
    ; We only write the lower 4 of each entry, but that's fine, 
    ; because we only map 1 Gib and not more than 4 GiB
    add EAX, 0x00200000
    lea EDI, [EDI + 8]

    loop .loopHigherHalf

  ret

ENTRIES_PT equ 512
PAGE_SIZE equ 0x1000

; CR4 flags for when we jump into long mode
FLAG_PSE equ 1 << 4 ; Page size extension
FLAG_PAE equ 1 << 5 ; Physical address extension
FLAG_PGE equ 1 << 7 ; Page global enable

; Data used to make the switch into 32 bit compatible long mode thing
EFER_MSR equ 0xC0000080
EFER_LONG_MODE_ENABLE equ 1 << 8
; And for long
CR0_PAGING_ENABLE equ 1 << 31
CR0_PROTECTED_ENABLE equ 1 << 0

; GRUB by default does NOT activate 32 bit paging
; And DOES activate the A20 line
; But keep that in mind if you decide to EFI stub this
enterLongmode:
  ; Disable 32 bit paging,
  ; Just in case
  mov EAX, CR0
  and EAX, ~CR0_PAGING_ENABLE
  mov CR0, EAX
  
  ; We need the actual page tables set up too
  call createPageTables
  call mapHigherHalf

  ; Load the GDT for some reason
  lgdt [GDT64.Pointer]

  ; Set the CR4 flags to configure paging (before we enable it)
  mov EAX, CR4
  or EAX, FLAG_PGE | FLAG_PAE | FLAG_PSE
  mov CR4, EAX

  ; First 12 bits of CR3 are assumed to be 0/ignored, because of page alignment
  ; Set CR3 to the start of the PML4T page table
  mov EDI, __pml4t_addr
  mov CR3, EDI

  ; MSRs are "machine specific registers"
  ; This one is no longer machine specific.
  ; We simply set bit 8 to enable long mode... ish.
  mov ECX, EFER_MSR
  rdmsr
  or EAX, EFER_LONG_MODE_ENABLE
  wrmsr

  ; Not quite. We need to enable paging in CR0 to enter long mode proper, then do a jump
  ; Right now, we are in compatibility mode (IA-32e)
  mov EAX, CR0
  or EAX, CR0_PAGING_ENABLE | CR0_PROTECTED_ENABLE
  mov CR0, EAX

  ; The last segment register, cs, can only be set with a long jump
  jmp GDT64.Code:trampoline64

; Weird holdover from memory segmentation days
; YES I copied it leave me alone
; Access bits
PRESENT        equ 1 << 7
NOT_SYS        equ 1 << 4
EXEC           equ 1 << 3
DC             equ 1 << 2
RW             equ 1 << 1
ACCESSED       equ 1 << 0

; Flags bits
GRAN_4K       equ 1 << 7
SZ_32         equ 1 << 6
LONG_MODE     equ 1 << 5

GDT64:
  dq 0 
  ; $ is current position. This is an offset
  .Code: equ $ - GDT64 
    .Code.limit_lo: dw 0xffff
    .Code.base_lo: dw 0
    .Code.base_mid: db 0
    .Code.access: db PRESENT | NOT_SYS | EXEC | RW
    .Code.flags: db GRAN_4K | LONG_MODE | 0xF   ; Flags & Limit (high, bits 16-19)
    .Code.base_hi: db 0
  .Data: equ $ - GDT64 
    .Data.limit_lo: dw 0xffff
    .Data.base_lo: dw 0
    .Data.base_mid: db 0
    .Data.access: db PRESENT | NOT_SYS | RW
    .Data.flags: db GRAN_4K | SZ_32 | 0xF       ; Flags & Limit (high, bits 16-19)
    .Data.base_hi: db 0
  .Pointer:
    dw $ - GDT64 - 1
    dq GDT64

section .lowtext
BITS 64
trampoline64:
  ; One last thing we need to do, is set the segment registers
  ; Despite the fact that nobody has used segmentation since the 80286
  ; I guess this tells what segment table (GDT) we are using to various things?
  mov AX, GDT64.Data
  mov SS, AX ; Stack segment
  mov DS, AX ; Data segment
  mov ES, AX ; Extra segment

  lea rax, [rel start64]
  jmp rax

BITS 64
section .text
extern _rust_start
start64:
  xor rbp, rbp
  jmp _rust_start

linker script:

Code: Select all

ENTRY(_start)

KERNEL_VMA = 0xFFFFFFFF80000000;
SECTIONS {
  . = 1M;
  __kernel_start = .;

  .boot ALIGN(4K) : AT(0x7C00) SUBALIGN(4) {
    KEEP(*(.multiboot_header*))
  }
  
  .lowtext : {
    KEEP(*(.lowtext*))
  }

  . += KERNEL_VMA;
  .text : AT (ADDR (.text) - KERNEL_VMA) {
    KEEP(*(.text*))
    KEEP(*(.ltext*))
  }

  __kernel_code_end = . - KERNEL_VMA;

  .rodata : AT (ADDR (.rodata) - KERNEL_VMA) {
    KEEP(*(.rodata*))
    KEEP(*(.lrodata*))
  }

  .bss ALIGN(16) : AT (ADDR (.bss) - KERNEL_VMA) {
    __temp_stack_bottom = LOADADDR(.bss) + (. - ADDR(.bss));
    . += 16K;
    __temp_stack_top = LOADADDR(.bss) + (. - ADDR(.bss));
    . = ALIGN(4K);
    __pml4t_addr = LOADADDR(.bss) + (. - ADDR(.bss));
    . += 20K;
  }

  . = ALIGN(0x200000) + __kernel_start;
  __kernel_data_end = . - KERNEL_VMA;
}

[MichaelPetch]

I did see this with the GPF:

Code: Select all

GDT=     000000004e05e800 00000000

That look like a corrupt GDTR was passed to LGDT. The address looks wrong as does the size (0). I'd recommends trying to use the BOCHS debugger to step through your code and see why this occurs.

[closos]

Yeah, that does look pretty weird:
Ghidra gives this

Code: Select all

                             GDT64.Pointer
        001001ef 17              ??         17h
        001001f0 00              ??         00h
        001001f1 d7              ??         D7h                                              ?  ->  001001d7
        001001f2 01              ??         01h
        001001f3 10              ??         10h
        001001f4 00              ??         00h
        001001f5 00              ??         00h
        001001f6 00              ??         00h
        001001f7 00              ??         00h
        001001f8 00              ??         00h

0x001001d7 looks normal to me as an address. Not sure what's going on

[closos]

Code: Select all

* thread #1, stop reason = instruction step into
    frame #0: 0x0000000000007dc6
->  0x7dc6: movq   %rax, %cr0
    0x7dc9: lgdtq  0x1001ef(%rip)
(lldb) memory read 0x1001ef
0x001001ef: 00 83 c4 10 eb 1f 01 f6 eb e2 89 f2 8b 03 e8 b2  ................
0x001001ff: 50 00 00 85 c0 75 07 a1 e0 7d 01 00 eb 07 89 03  P....u...}......
(lldb) memory read 0x1001ef -f x -c 4
0x001001ef: 0x10c48300 0xf6011feb 0xf289e2eb 0xb2e8038b
(lldb) 

Perhaps something is garbling the GDT before I can load it.

[MichaelPetch]

I don't think it is helping that you seem to be requesting code be loaded with a LMA of 0x7C00 and referencing things with a VMA of 0x001xxxxx. I'd be curious about the design decision to use 0x7C00 this way in the linker script:

Code: Select all

  . = 1M;
  __kernel_start = .;

  .boot ALIGN(4K) : AT(0x7C00) SUBALIGN(4) {
    KEEP(*(.multiboot_header*))
  }
  
  .lowtext : {
    KEEP(*(.lowtext*))
  } 

I'm assuming your _start entry point (which you don't show) is also in .lowtext as well.

[closos]

Oh, that was left in accidentally. I was playing around with forcing the linker to put the multiboot header at the start, because that was an issue I had been having a bit ago. Removing it, it boots properly still, and the error changes to something which seems to just be a bog-standard page fault. Thank you!

[MichaelPetch]

I also just noticed that your KERNEL VMA in the boot code (paging layout) is using 0xFFFF800000000000 but the linker script is using 0xFFFFFFFF80000000. You will have to modify things so you are using the same KERNEL VMA. I'd recommend going with 0xFFFFFFFF80000000 as you can then use the `-mcmodel=kernel` GCC option.

[MichaelPetch]

As well when you do the `mapHigherHalf` you will need to round `__kernel_start` down to the nearest 2MiB boundary because 2MiB pages will have to be aligned on 2MiB boundaries. You can do this by ANDing the address of`__kernel_start` with `-0x200000` (note the negative). When you compute the total size of the kernel to map you will want to round the result up to the nearest 2MiB to ensure all the code and data is captured. You can do this by adding (0x200000-1=0x1FFFFF) to the value and then shift that right 21 bits (2MiB).