Help fixing my userspace entry and syscall entry, please

[bunny]

Hihii all :)

( Reading/mood song: https://open.spotify.com/track/7vhtylsJ ... 3625774738 )

Intro:



I am making a Rust OS for fun and learning, so I guess I would fall into the hobby category, and I have made pretty good progress so far but I am blocked on something I would love some help with, please :)

I read the wiki article on asking questions so I want to make sure I get a few things out of the way first:

- I do have the necessary programming background
- I did research my problem as thoroughly as possible on the wiki, online, in Understanding the Linux Kernel book, and trying to find linux or other os source code (Im not a von duct tape and honestly I find it kinda cute you have all those archetypes lol)
- I will do my best to share the right amount of code: not to much, not too little. However you may wish to see more so just ask and I shall provide <3

I have been blocked on this for a bit now and Im really losing it :'(

Goal:

To get my basic/initial pass at entering user space and entering a syscall to work correctly and fully i.e. after loading a program (ELF) and setting up necessary resources (stack etc) I should be able to have it begin executing and trigger a syscall that I am able to then handle.

Background:

My OS is written in Rust and right now is architecture dependent (x86_64) as I have not gotten far enough in development to refactor out the architecturally dependent components. I am using the rust boot loader 0,11,13 crate (I saw Limine is big on r/osdev but again I want something simple & working for now and later I can refactor booting into something better longterm). I am relatively early on in my OS development and so far have the following components implemented and, seemingly, working:


- Paging
- Interrupts
- An allocator
- GDT
- ELF loading (basic - no PIE/ALSR/Dynamic Linking)
- Process abstraction & basic resources (VM Areas, PID, kernel stack, user stack)
- Some other misc utilities

Problem:



I am able to load a user program (ELF) into memory and setup its initial resources but when I go to call the non-returnable enter_userspace sequence nothing happens. By nothing I mean either triple fault or just getting hung, I can’t tell — I am using QEMU and when I pass the no-reboot flag it hangs, without it it, sometimes, reboots but gets to the same state.

I have verified, to the best of my ability/knowledge that the process’s resources are indeed properly initialized: for example the entry point address is correct, all relevant memory regions are mapped in both the kernels page table and the processes (I have even gone overkill a bit here in a temporary attempt to fix the issue), the stack is setup with the correct stack address getting set in RSP, etc. I have also made sure the raw bytes at various addresses match the disassembled user space test binary. (I’m happy to elaborate on this to whoever helps I am just trying to precent this post from becoming an essay/too long and then no one has the patience to help D: )



Now, onto the syscall side of things. The only way I can really tell if things are working (viz.. to see a write syscall from user space telling me its executing correctly) is if I can get to my syscall entry point where, for debugging purposes I have put a `ud2` instruction as the first instruction called. I have also verified as best I can that my sys call init is correct — the MSR seem to have the right values and my syscall entry point’s address is mapped in both page tables and also set in STAR. My segments/GDT seem to be right as well.

My initial assumption having done all this verifying is that my issue is somewhere in my userspace entry assembly and/or my syscall entry assembly. This is in part because assembly is my weakest thing right now so I’m less equipped to confidently check it. For what it’s worth I have put time into actually learning NASM and such but it’s obviously unforgiving so I wouldn’t be surprised if I have some small ‘gotcha’ in my code.

To start I have put below my process entry point which is a rust method attached to my scheduler that does a little process resource setup and then calls the user space entry assembly. I have also attached the entirety of my syscall management code (which is very little right now). Lastly I have my disassembled user space test program and some debug output from my terminal when running my OS. Again I am trying to keep this relatively minimal in the hopes my post isn’t too terribly long, but you can ask to see any other part of my code and I will send it. I have my full code in the repo linked if you wish to peruse :)



Code:

NOTE: while I am trying to learn and so am open to all sorts of advice and critique please do not judge me too much on the code shown as it is NOT at a clean checkpoint and, intentionally, contains lots of attempts and temporary fixes etc. I left it in so you can get an idea of what Ive tried just please know I would never leave my code in this state in general.




As reference here is the dissassembled userspace test program (I included the command for full transparency with how I am generating this):

Code: Select all

objdump -d init/target/x86_64-unknown-none/release/init.stripped

init/target/x86_64-unknown-none/release/init.stripped:	file format elf64-x86-64

Disassembly of section .text:

0000000000400000 <.text>:
  400000: b8 01 00 00 00               	movl	$0x1, %eax
  400005: bf 01 00 00 00               	movl	$0x1, %edi
  40000a: ba 15 00 00 00               	movl	$0x15, %edx
  40000f: 48 c7 c6 00 10 60 00         	movq	$0x601000, %rsi         # imm = 0x601000
  400016: 0f 05                        	syscall
  400018: b8 01 00 00 00               	movl	$0x1, %eax
  40001d: ba 01 00 00 00               	movl	$0x1, %edx
  400022: 48 c7 c6 15 10 60 00         	movq	$0x601015, %rsi         # imm = 0x601015
  400029: 0f 05                        	syscall
  40002b: b8 01 00 00 00               	movl	$0x1, %eax
  400030: ba 16 00 00 00               	movl	$0x16, %edx
  400035: 48 c7 c6 16 10 60 00         	movq	$0x601016, %rsi         # imm = 0x601016
  40003c: 0f 05                        	syscall
  40003e: b8 01 00 00 00               	movl	$0x1, %eax
  400043: ba 01 00 00 00               	movl	$0x1, %edx
  400048: 48 c7 c6 15 10 60 00         	movq	$0x601015, %rsi         # imm = 0x601015
  40004f: 0f 05                        	syscall
  400051: b8 01 00 00 00               	movl	$0x1, %eax
  400056: bf 02 00 00 00               	movl	$0x2, %edi
  40005b: ba 18 00 00 00               	movl	$0x18, %edx
  400060: 48 c7 c6 2c 10 60 00         	movq	$0x60102c, %rsi         # imm = 0x60102C
  400067: 0f 05                        	syscall
  400069: b8 3c 00 00 00               	movl	$0x3c, %eax
  40006e: 31 ff                        	xorl	%edi, %edi
  400070: 0f 05                        	syscall
  400072: 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00    	nopw	%cs:(%rax,%rax)
  400080: eb fe                        	jmp	0x400080 <.text+0x80>

The pertinent process/userspace_entry code and also here & here, again I left in multiple versions/attempts I took at this I know its not clean, also I had tried various debugging paths within the entry code but abandoned them as all I ended up doing was messing around with the state and clobbering registers on accident.

The pertinent syscall init and management code , the same disclaimer above applies, the goal as I mentioned before is to just hi the `ud2` so I know I got there.

Pretty much anything else immediately relevant will be in /proc

Most recent debug output:

Code: Select all

INFO : Framebuffer info: FrameBufferInfo { byte_len: 2764800, width: 1280, height: 720, pixel_format: Bgr, bytes_per_pixel: 3, stride: 1280 }
INFO : 4th Stage
INFO : BiosInfo { stage_4: Region { start: 130000, len: 25300 }, kernel: Region { start: 1000000, len: 4be170 }, ramdisk: Region { start: 14bf000, len: 0 }, config_file: Region { start: 14bf000, len: 0 }, last_used_addr: 14befff, framebuffer: BiosFramebufferInfo { region: Region { start: fd000000, len: 2a3000 }, width: 500, height: 2d0, bytes_per_pixel: 3, stride: 500, pixel_format: Bgr }, memory_map_addr: e244, memory_map_len: 7 }
INFO : BIOS boot
INFO : Elf file loaded at Pointer {
    addr: 0x0000000001000000,
    metadata: 4972912,
}
INFO : virtual_address_offset: 0x0
INFO : Handling Segment: Ph64(ProgramHeader64 { type_: Ok(Load), flags: Flags(4), offset: 0, virtual_addr: 200000, physical_addr: 200000, file_size: 11ab4, mem_size: 11ab4, align: 1000 })
INFO : Handling Segment: Ph64(ProgramHeader64 { type_: Ok(Load), flags: Flags(5), offset: 11ac0, virtual_addr: 212ac0, physical_addr: 212ac0, file_size: 43266, mem_size: 43266, align: 1000 })
INFO : Handling Segment: Ph64(ProgramHeader64 { type_: Ok(Load), flags: Flags(6), offset: 54d28, virtual_addr: 256d28, physical_addr: 256d28, file_size: 20, mem_size: 2d8, align: 1000 })
INFO : Mapping bss section
INFO : Handling Segment: Ph64(ProgramHeader64 { type_: Ok(Load), flags: Flags(6), offset: 54d48, virtual_addr: 257d48, physical_addr: 257d48, file_size: 8, mem_size: 24460, align: 1000 })
INFO : Mapping bss section
INFO : Entry point at: 0x2220b0
INFO : Creating GDT at PhysAddr(0x15ee000)
INFO : Map framebuffer
INFO : Map physical memory
INFO : Allocate bootinfo
INFO : Create Memory Map
INFO : Create bootinfo
INFO : Jumping to kernel entry point at VirtAddr(0x2220b0)


  INDYOS INITIALIZING...
  Frame allocator tracking 128448 frames (513792 KB of memory)
  PAGING INITIALIZED
Loading IDT...
IDT loaded successfully!
  IDT INITIALIZED
Loading GDT...
GDT loaded successfully!
  GDT INITIALIZED
Manual STAR value: 0x10000800000000
  Will set CS to: 0x23
  Will set SS to: 0x1b
STAR readback: SegmentSelector { index: 4, rpl: Ring0 }
LSTAR written: 0x21369c
LSTAR readback: 0x21369c
Match: true
============================

  SYSCALL INITIALIZED
LSTAR: 0x21369c
STAR: (SegmentSelector { index: 4, rpl: Ring0 }, SegmentSelector { index: 3, rpl: Ring0 }, SegmentSelector { index: 1, rpl: Ring0 }, SegmentSelector { index: 2, rpl: Ring0 })
SFMask: RFlags(DIRECTION_FLAG | INTERRUPT_FLAG | TRAP_FLAG)
[Kernel] Demand-paged at VirtAddr(0x444444440000) [PF: 1 total (1 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444441000) [PF: 2 total (2 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444442000) [PF: 3 total (3 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444443000) [PF: 4 total (4 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444444000) [PF: 5 total (5 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444445000) [PF: 6 total (6 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444446000) [PF: 7 total (7 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444447000) [PF: 8 total (8 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444448000) [PF: 9 total (9 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444449000) [PF: 10 total (10 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444444a000) [PF: 11 total (11 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444444b000) [PF: 12 total (12 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444444c000) [PF: 13 total (13 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444444d000) [PF: 14 total (14 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444444e000) [PF: 15 total (15 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444444f000) [PF: 16 total (16 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444450000) [PF: 17 total (17 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444451000) [PF: 18 total (18 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444452000) [PF: 19 total (19 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444453000) [PF: 20 total (20 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444454000) [PF: 21 total (21 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444455000) [PF: 22 total (22 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444456000) [PF: 23 total (23 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444457000) [PF: 24 total (24 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444458000) [PF: 25 total (25 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444459000) [PF: 26 total (26 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444445a000) [PF: 27 total (27 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444445b000) [PF: 28 total (28 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444445c000) [PF: 29 total (29 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444445d000) [PF: 30 total (30 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444445e000) [PF: 31 total (31 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444445f000) [PF: 32 total (32 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444460000) [PF: 33 total (33 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444461000) [PF: 34 total (34 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444462000) [PF: 35 total (35 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444463000) [PF: 36 total (36 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444464000) [PF: 37 total (37 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444465000) [PF: 38 total (38 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444466000) [PF: 39 total (39 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444467000) [PF: 40 total (40 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444468000) [PF: 41 total (41 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444469000) [PF: 42 total (42 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444446a000) [PF: 43 total (43 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444446b000) [PF: 44 total (44 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444446c000) [PF: 45 total (45 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444446d000) [PF: 46 total (46 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444446e000) [PF: 47 total (47 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444446f000) [PF: 48 total (48 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444470000) [PF: 49 total (49 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444471000) [PF: 50 total (50 paged, 0 failed)]
Zeroed out heap memory region init at VirtAddr(0x444444440000) -- amount 204800
Begin process initialization...
allocated pid: 1 -- about to setup virtual memory

=== PAGE TABLE COPY VERIFICATION ===
Syscall handler address: 0x21369c
Syscall handler PML4 index: 0
Kernel PML4[0]: mapped
Process PML4[0]: mapped
===================================

Created process 1...
[src/proc/elf.rs:218] header = ELFHeader {
    ident: ELFIdent {
        magic: [
            127,
            69,
            76,
            70,
        ],
        class: 2,
        data: 1,
        version: 1,
        osabi: 0,
        abiversion: 0,
        pad: [
            0,
            0,
            0,
            0,
            0,
            0,
            0,
        ],
    },
    elf_type: 2,
    machine: 62,
    version: 1,
    entry: 4194304,
    phoff: 64,
    shoff: 12320,
    flags: 0,
    ehsize: 64,
    phentsize: 56,
    phnum: 3,
    shentsize: 64,
    shnum: 5,
    shstrndx: 4,
}
about to validate page alignment....
about to validate segments...
about to load into memory...
  Page Page[4KiB](0x400000): copying 130 bytes at offset 0
Copied 130 bytes total
Successfully loaded program into memory!
about to validate page alignment....
about to validate segments...
about to load into memory...
  Page Page[4KiB](0x601000): copying 68 bytes at offset 0
Copied 68 bytes total
Successfully loaded program into memory!
Process created!
Done initializing process
Add process to scheduler -- about to run

Scheduled a process to run
argv_ptrs[0] = 0x7fffffffeffb

envp_ptrs: []

setup elf aux vector at: 0x7fffffffefd0

Stack has been setup with stack pointer: 0x7fffffffefb0
TSS updated: kernel stack = VirtAddr(0xffffa00000003ff0)
About to enter userspace via sysret path
  Entry: 0x400000
  Stack: 0x7fffffffefb0

=== MEMORY LAYOUT DEBUG ===
Entry point code disassembly shows:
  400000: b8 01 00 00 00  mov eax, 1
  400005: bf 01 00 00 00  mov edi, 1
  40000a: ba 15 00 00 00  mov edx, 0x15
  40000f: 48 c7 c6...     mov rsi, 0x601000
  400016: 0f 05           syscall  <-- FIRST SYSCALL

Checking user code at 0x400000:
  Bytes: b8 01 00 00 00 bf 01 00 00 00 ba 15 00 00 00 48 
         c7 c6 00 10 60 00 0f 05 b8 01 00 00 00 ba 01 00 
  Matches expected: mov eax,1; ...; syscall

Checking user data at 0x601000:
  Bytes: 48 65 6c 6c 6f 20 66 72 6f 6d 20 75 73 65 72 73 70 61 63 65 21 0a 54 68 69 73 20 69 73 20 61 20 
  ASCII: Hello from userspace!.This is a 

Checking syscall handler at 0x21369c:
  Mapped to phys 0x101269c
  First bytes: 0f 0b 50 48 
  Looks good: starts with UD2 (0f 0b)!
===========================



Final Words:



This is my first time posting here and honestly reaching out to anyone about my little OS. If I messed up anything in my post I apologize. I will be eternally grateful to anyone who can spare some time to help a girl out on this <3 Hopefully I can pay it forward and help others in the os dev journey!

Thanks <3

[Octocontrabass]

My initial assumption having done all this verifying is that my issue is somewhere in my userspace entry assembly and/or my syscall entry assembly.

You may be right about that. I found at least three problems in this block of inline assembly.

• Use input operands with single-register constraints to set RCX and R11. This both ensures that the compiler doesn't use those registers for any other inputs and allows you to remove two MOV instructions.
• Why are you setting RSP to the kernel stack? If you're following a typical one-kernel-stack-per-thread design, RSP already contains the correct kernel stack pointer here.
• Don't clobber EAX when you set CR3. You can MOV any general-purpose register to CR3, so just use whichever register the compiler chose for you.

...Aside from that, have you tried debugging with an actual debugger, like GDB or LLDB? You can attach the debugger to QEMU's GDB stub. Single-stepping through the code around the first SYSRETQ may be enlightening.

QEMU itself has some helpful debugging functions, too. For example, I find the "-d int" log is often helpful for figuring out what's causing a triple fault.

[bunny]

I didn't realize gdb worked with rust/qemu but thats a good point, will gdb allow me to step through the assembly on a single instruction level? From my basic understanding of how debuggers work they step through lines of source code which can correspond to multiple lines of assembly. I guess this is just another breakdown in my, lack of, assembly knowledge -- i.e. how to debug it.

I'll see what I can find through the debugging tips you mentioned though, thank you again! I'll try my best to solve this myself but if I hit a wall post debugging I might seek your help again if you dont mind <3


Happy Holidays!!!

[iansjack]

“stepi” and “nexti” work at the instruction level. gdb can also disassemble code. It’s worth reading the documentation: https://www.sourceware.org/gdb/documentation/

[bunny]

Yeah my bad, I'm working through this now, thank you both! If you have any tips for setting this up, on mac, with rust, and QEMU, I'll happily take them but no worries

And yes I did check out https://www.qemu.org/docs/master/system/gdb.html (sorry again for asking above haha I have a, bad, tendency to think 'out-loud' ) I'll figure it out either way!


<3

[bunny]

Ok so I have taken the advice, made some changes and got progress but hit another wall. Now instead of a triple fault/hang I am hitting a general protection fault.

My first iteration of fixes here simplified my assembly down to:

Code: Select all

unsafe {
    asm!(
        "mov cr3, {page_table}",
        "push {user_ss}",      // Pushed first (bottom) - popped last (5th)
        "push {sp}",           // Popped 4th
        "push {rflg}",         // Popped 3rd
        "push {user_cs}",      // Popped 2nd
        "push {ip}",           // Pushed last (top) - popped first (1st)
        "iretq",

        page_table = in(reg) page_table_addr.as_u64(),
        user_ss = in(reg) stack_frame.ss as u64,
        sp = in(reg) stack_frame.rsp,
        rflg = in(reg) stack_frame.rflags,
        user_cs = in(reg) stack_frame.cs as u64,
        ip = in(reg) stack_frame.rip,
        options(noreturn)
    );
}

Taking the advice to let the compiler decide registers and to just focus on setting up the interrupt frame for iretq correctly.

That was giving me a Double fault:

Code: Select all

KERNEL PANIC!
  Location: src/interrupts.rs:78
  Message: DOUBLE FAULT
InterruptStackFrame {
    instruction_pointer: VirtAddr(
        0x23,
    ),
    code_segment: SegmentSelector {
        index: 64,
        rpl: Ring2,
    },
    cpu_flags: RFlags(
        ID | VIRTUAL_INTERRUPT_PENDING | VIRTUAL_INTERRUPT | ALIGNMENT_CHECK | VIRTUAL_8086_MODE | RESUME_FLAG | NESTED_TASK | IOPL_HIGH | OVERFLOW_FLAG | DIRECTION_FLAG | INTERRUPT_FLAG | TRAP_FLAG | SIGN_FLAG | AUXILIARY_CARRY_FLAG | 0x7fffffc08020,
    ),
    stack_pointer: VirtAddr(
        0x1b,
    ),
    stack_segment: SegmentSelector {
        index: 0,
        rpl: Ring0,
    },
}

In the end: I noticed the values were correct but off by 8 bytes so I changed it up and went with a struct that holds the values and I just set rsp to it, to fix the 8 byte issue I just added a qword of padding before hand and I made progress to the now GPF.

Here is my new code I am using for the iretq:



Code: Select all

// struct

#[repr(C)]
#[derive(Debug, Clone, Copy)]
struct InterruptStackFrame {
    rip: u64,
    cs: u16,
    _pad1: [u8; 6],
    rflags: u64,
    rsp: u64,
    ss: u16,
    _pad2: [u8; 6],
}

impl InterruptStackFrame {
    fn new(rip: u64, cs: SegmentSelector, rsp: u64, ss: SegmentSelector) -> Self {
        Self {
            rip,
            cs: cs.0,
            _pad1: [0; 6],
            rflags: 0x202u64,
            rsp,
            ss: ss.0,
            _pad2: [0; 6],
        }
    }
}


user space entry:




Code: Select all


#[unsafe(naked)]
pub unsafe extern "C" fn jump_to_userspace(
    page_table: u64,
    frame_ptr: u64,
) -> ! {
    naked_asm!(
        "mov cr3, rdi",      // first arg (page_table) is in rdi
        "mov rsp, rsi",      // second arg (frame_ptr) is in rsi
        "iretq",
        );
}
pub fn run_current_process(&mut self) -> ! {
    let process = self.schedule().expect("No process to run!");

    let stack_pointer = process.setup_stack(
        &["init"],
        &[],
        vec![(6, 4096), (0, 0)],
    ).expect("Failed to setup stack");

    process.cpu_state_mut().rsp = stack_pointer;

    let kernel_stack_top = process.kernel_stack_top();
    let kernel_stack_ptr = kernel_stack_top.as_u64() - 16;
    let page_table_addr = process.page_table_addr();

    unsafe {
        crate::syscalls::management::CURRENT_KERNEL_RSP = kernel_stack_ptr;
        crate::memory::segmentation::set_kernel_stack(VirtAddr::new(kernel_stack_ptr));
    }

    let entry_point = process.cpu_state().rip;

    let stack_frame = InterruptStackFrame::new(
        entry_point,
        GDT.1.user_code,
        stack_pointer,
        GDT.1.user_data
    );

    let frame_size = size_of::<InterruptStackFrame>() as u64;
    let aligned_top = kernel_stack_top.as_u64() & !0xF;

    let frame_ptr = aligned_top - frame_size - 8;  // extra 8 bytes before frame
    unsafe {
        // write dummy qword
        *((frame_ptr) as *mut u64) = 0;

        // write frame 8 bytes higher
        let dst = (frame_ptr + 8) as *mut InterruptStackFrame;
        core::ptr::write(dst, stack_frame);

        }
    }

    dbg!(stack_frame);

    unsafe {
        Self::jump_to_userspace(page_table_addr.as_u64(), frame_ptr);
    }

    // old inline attempt swapped for naked function as a sanity check
    // unsafe {
    //     asm!(
    //         "mov cr3, {page_table}",
    //         "mov rsp, {frame_ptr}",
    //         "iretq",
    //
    //         page_table = in(reg) page_table_addr.as_u64(),
    //         frame_ptr = in(reg) frame_ptr,
    //         options(noreturn)
    //     );
    // }
}


I did some, basic, research on the GPF and given the error_code is 0 its NOT segment related and given the CPL in the fault is 0 (all the output is below) it looks like the issue is happening during the iretq itself.

On the output: I took the gdb advice and while it helped me a bit the issue is now seemingly during the iretq which, by its very nature, is hardware atomic so its not exactly something I can step through (unless I am missing something). One check in the output is to ensure Im in the right mode i.e. long and not compatibility (just something I saw come up when trying to search on this).

Debug Output:

Code: Select all

// checks to make sure struct is represented right in mem/to the iretq
sizeof InterruptStackFrame: 40
align of InterruptStackFrame: 8
kernel_stack_top: 0xffffa00000004000
frame_size: 40
frame_ptr: 0xffffa00000003fd0
[src/proc/process.rs:492] stack_frame = InterruptStackFrame {
    rip: 4194304,
    cs: 35,
    _pad1: [
        0,
        0,
        0,
        0,
        0,
        0,
    ],
    rflags: 514,
    rsp: 140737488351152,
    ss: 27,
    _pad2: [
        0,
        0,
        0,
        0,
        0,
        0,
    ],
}

=== WHAT IRETQ WILL READ ===
iretq will pop from: 0xffffa00000003fd8
  RIP:    0x400000
  CS:     0x23
  RFLAGS: 0x202
  RSP:    0x7fffffffefb0
  SS:     0x1b

=== CPU MODE CHECK ===
CR0: 0x80010011
CR4: 0x20
EFER: 0xd01
  Long Mode Active (LMA): 1
  Long Mode Enable (LME): 1

=== GDT ENTRY DECODE ===
User Code GDT[4]:
  Present: 1
  DPL: 3
  Executable: 1
  Long mode (L): 1
User Data GDT[3]:
  Present: 1
  DPL: 3
  Writable: 1
General Protection Fault
InterruptStackFrame {
    instruction_pointer: VirtAddr(
        0x2318e6,
    ),
    code_segment: SegmentSelector {
        index: 1,
        rpl: Ring0,
    },
    cpu_flags: RFlags(
        RESUME_FLAG | 0x2,
    ),
    stack_pointer: VirtAddr(
        0xffffa00000003fd0,
    ),
    stack_segment: SegmentSelector {
        index: 2,
        rpl: Ring0,
    },
}
Error: 0
KERNEL PANIC!
  Location: src/interrupts.rs:350
  Message: General Protection Fault

The debug output is a lot of sanity checks I wanted to do and also so anyone willing to help me has, hopefully, enough information to give me meaningful advice.



Thanks again <3



~ Bunny

[Octocontrabass]

That was giving me a Double fault:

[...]

In the end: I noticed the values were correct but off by 8 bytes

A double fault pushes an error code on the stack.

If the CPU pushes an error code when your exception handler isn't expecting one, everything will appear to be off by eight bytes. If this is this case, you should update your double fault handler to expect an error code, then check QEMU's interrupt log ("-d int") to see which exception happened before the double fault because a double fault means there's a problem with one of your other exception handlers.

If the CPU doesn't push an error code when your exception handler is expecting one, it's not a double fault, it's IRQ0 from the PIT and your code was working correctly.

[bunny]

Thanks for the reply, I worked on this a bit and really tried to dig into the QEMU log output to try to figure out what specific issue was truly causing the double fault (i.e. the 8 byte error code pushed during the iretq that then mis-aligns the interrupt frame thus causing the double fault) but I am having issues. If I look at the QEMU log I see a lot of hardware int=0x08 which based on what I have been able to find out is a timer interrupt. Currently interrupts (i.e. idt[32]+) are disabled (just what seemed to be enabled by default) so they get silently handled and so far have not interfered with my OS's normal operations so I doubt that somehow that is the culprit. There seems to be some state setup issue, or at least thats what seems reasonable to me at this point, thats outside of the interrupt stack frame since I have verified it pretty thoroughly.

Getting to my follow up question -- would you have any suggestions for either steps I can take to better investigate this further or suggestions as to what might be wrong? If it helps at all I will paste two QEMU logs (I tried -s & -S with gdb as well as -d int,cpu_reset) below -- one corresponds to the double fault scenario (no 8byte compensation on my part) and the GPF scenario (where I perform the 8 byte offset to just see what happens when the frame isn't malformed during exception handling). I have tried what I can to figure out what the first error code is and I just can't seem to get to whats really happening here, hence asking for some debugging advice because I dont really know what to investigate, or rather how to.

Note: For the logs I am omitting some of the hardware int lines as theres a decent amount of them, and I am omitting the system setup lines. If those would be relevant lmk okie?

Double Fault QEMU log:

Code: Select all

Servicing hardware INT=0x08
Servicing hardware INT=0x08
Servicing hardware INT=0x08
Servicing hardware INT=0x08
Servicing hardware INT=0x08
0: v=08 e=0000 i=0 cpl=3 IP=0023:0000000000400000 pc=0000000000400000 SP=001b:00007fffffffefb0 env->regs[R_EAX]=000000000001c000
RAX=000000000001c000 RBX=0000000000004fb8 RCX=ffffa00000003fd8 RDX=000000000000000a
RSI=00000100000ffa48 RDI=000000000001c000 RBP=0000000000000000 RSP=00007fffffffefb0
R8 =0000000000210340 R9 =0000000000000002 R10=000000000020c2b0 R11=00000100001000b0
R12=00000000000062a8 R13=00000000000075d8 R14=0000030000000000 R15=00000000000077c8
RIP=0000000000400000 RFL=00000202 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0
ES =0000 0000000000000000 00000000 00000000
CS =0023 0000000000000000 ffffffff 00affb00 DPL=3 CS64 [-RA]
SS =001b 0000000000000000 ffffffff 00cff300 DPL=3 DS [-WA]
DS =0000 0000000000000000 00000000 00000000
FS =0000 0000000000000000 00000000 00000000
GS =0000 0000000000000000 00000000 00000000
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0028 0000000000268750 00000067 00008900 DPL=0 TSS64-avl
GDT= 00000000002686f0 00000037
IDT= 0000000000268870 00000fff
CR0=80010011 CR2=0000000000000000 CR3=000000000001c000 CR4=00000020
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=0000000000000000 CCD=00000100000ffdb8 CCO=EFLAGS
EFER=0000000000000d01

General Protection Fault QEMU log:

(Yes I know this is most likely a red herring and my focus is the double fault I just wanted to include this scenario in case it helped illuminate something)

Code: Select all

Servicing hardware INT=0x08
Servicing hardware INT=0x08
Servicing hardware INT=0x08
Servicing hardware INT=0x08
Servicing hardware INT=0x08
check_exception old: 0xffffffff new 0xd
0: v=0d e=0000 i=0 cpl=0 IP=0008:0000000000241225 pc=0000000000241225 SP=0010:ffffa00000003fd0 env->regs[R_EAX]=000000000001c000
RAX=000000000001c000 RBX=0000000000004fb8 RCX=ffffa00000003fd0 RDX=000000000000000a
RSI=00000100000ffa28 RDI=000000000001c000 RBP=0000000000000000 RSP=ffffa00000003fd0
R8 =0000000000210370 R9 =000000000000000c R10=000000000020c2e0 R11=00000100001000b0
R12=00000000000062a8 R13=00000000000075d8 R14=0000030000000000 R15=00000000000077c8
RIP=0000000000241225 RFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 0000000000000000 00000000 00000000
CS =0008 0000000000000000 ffffffff 00af9b00 DPL=0 CS64 [-RA]
SS =0010 0000000000000000 ffffffff 00cf9300 DPL=0 DS [-WA]
DS =0000 0000000000000000 00000000 00000000
FS =0000 0000000000000000 00000000 00000000
GS =0000 0000000000000000 00000000 00000000
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0028 0000000000268840 00000067 00008900 DPL=0 TSS64-avl
GDT= 00000000002687e0 00000037
IDT= 0000000000268960 00000fff
CR0=80010011 CR2=0000000000000000 CR3=000000000001c000 CR4=00000020
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=0000000000000038 CCD=00000100000ffd98 CCO=ADDQ
EFER=0000000000000d01

As always I really appreciate your willingness to help me out, I was nervous initially posting here as I wasn't sure what to expect from the community (seeing as you are all way smarter and more capable than me ha). So thank you <3

[Octocontrabass]

Currently interrupts (i.e. idt[32]+) are disabled (just what seemed to be enabled by default) so they get silently handled and so far have not interfered with my OS's normal operations so I doubt that somehow that is the culprit.

You initialize the user-mode RFLAGS to 0x202, which sets the interrupt flag, which enables maskable interrupts. Did you disable interrupts some other way? If not, interrupts are enabled by the IRETQ into user mode. For annoying legacy reasons, the interrupt controllers are configured by default to use interrupt vector 8 for IRQ0, even though interrupt vector 8 is supposed to only be for double faults. Since the timer is also configured to send IRQs by default, as soon as you enable interrupts, you receive an IRQ0 that looks like a double fault. You're not the first person to run into this.

QEMU's interrupt log agrees that it's a hardware IRQ, since there's no "check_exception" line before the supposed double fault.

[bunny]

omg!! omg!!! thank youu :) that insight was all I needed to get the userspace transition working! I remapped the PIC so the hardware interrupt vectors are relocated to 0x20 (32) + and I was able to follow standard practice w/r/t the timer interrupt handling for now (as I dont have any scheduler and I think the timer interrupt would be for that purpose if Im not mistaken though I often am D: ). Now I am able to reach my debug ud2 at the top of my syscall entry so now begins my journey getting that to work. Seeing as thats also a critical bit of assembly I might end up reaching out again on this post if I really can't figure it out, but honestly the little tips and insights you all have given so far should be enough to get me by! So thank you again!! This problem was seriously getting me down/making me doubt myself so it feels like a weight of me now :D


Oh and my excitement almost completely distracted me from the interrupts thing you mentioned: you're right about rflags and I didn't catch that, I was stuck on the double fault and kept on thinking that I wasn't getting past the iretq successfully and since interrupts were disabled elsewhere up to that point there shouldn't be an issue but alas there was....don't judge me too hard okie? lol