Help fixing my userspace entry and syscall entry, please
Posted: Wed Dec 24, 2025 10:34 am
Hihii all :)
( Reading/mood song: https://open.spotify.com/track/7vhtylsJ ... 3625774738 )
Intro:
I am making a Rust OS for fun and learning, so I guess I would fall into the hobby category, and I have made pretty good progress so far but I am blocked on something I would love some help with, please :)
I read the wiki article on asking questions so I want to make sure I get a few things out of the way first:
- I do have the necessary programming background
- I did research my problem as thoroughly as possible on the wiki, online, in Understanding the Linux Kernel book, and trying to find linux or other os source code (Im not a von duct tape and honestly I find it kinda cute you have all those archetypes lol)
- I will do my best to share the right amount of code: not to much, not too little. However you may wish to see more so just ask and I shall provide <3
I have been blocked on this for a bit now and Im really losing it :'(
Goal:
To get my basic/initial pass at entering user space and entering a syscall to work correctly and fully i.e. after loading a program (ELF) and setting up necessary resources (stack etc) I should be able to have it begin executing and trigger a syscall that I am able to then handle.
Background:
My OS is written in Rust and right now is architecture dependent (x86_64) as I have not gotten far enough in development to refactor out the architecturally dependent components. I am using the rust boot loader 0,11,13 crate (I saw Limine is big on r/osdev but again I want something simple & working for now and later I can refactor booting into something better longterm). I am relatively early on in my OS development and so far have the following components implemented and, seemingly, working:
- Paging
- Interrupts
- An allocator
- GDT
- ELF loading (basic - no PIE/ALSR/Dynamic Linking)
- Process abstraction & basic resources (VM Areas, PID, kernel stack, user stack)
- Some other misc utilities
Problem:
I am able to load a user program (ELF) into memory and setup its initial resources but when I go to call the non-returnable enter_userspace sequence nothing happens. By nothing I mean either triple fault or just getting hung, I can’t tell — I am using QEMU and when I pass the no-reboot flag it hangs, without it it, sometimes, reboots but gets to the same state.
I have verified, to the best of my ability/knowledge that the process’s resources are indeed properly initialized: for example the entry point address is correct, all relevant memory regions are mapped in both the kernels page table and the processes (I have even gone overkill a bit here in a temporary attempt to fix the issue), the stack is setup with the correct stack address getting set in RSP, etc. I have also made sure the raw bytes at various addresses match the disassembled user space test binary. (I’m happy to elaborate on this to whoever helps I am just trying to precent this post from becoming an essay/too long and then no one has the patience to help D: )
Now, onto the syscall side of things. The only way I can really tell if things are working (viz.. to see a write syscall from user space telling me its executing correctly) is if I can get to my syscall entry point where, for debugging purposes I have put a `ud2` instruction as the first instruction called. I have also verified as best I can that my sys call init is correct — the MSR seem to have the right values and my syscall entry point’s address is mapped in both page tables and also set in STAR. My segments/GDT seem to be right as well.
My initial assumption having done all this verifying is that my issue is somewhere in my userspace entry assembly and/or my syscall entry assembly. This is in part because assembly is my weakest thing right now so I’m less equipped to confidently check it. For what it’s worth I have put time into actually learning NASM and such but it’s obviously unforgiving so I wouldn’t be surprised if I have some small ‘gotcha’ in my code.
To start I have put below my process entry point which is a rust method attached to my scheduler that does a little process resource setup and then calls the user space entry assembly. I have also attached the entirety of my syscall management code (which is very little right now). Lastly I have my disassembled user space test program and some debug output from my terminal when running my OS. Again I am trying to keep this relatively minimal in the hopes my post isn’t too terribly long, but you can ask to see any other part of my code and I will send it. I have my full code in the repo linked if you wish to peruse :)
Code:
NOTE: while I am trying to learn and so am open to all sorts of advice and critique please do not judge me too much on the code shown as it is NOT at a clean checkpoint and, intentionally, contains lots of attempts and temporary fixes etc. I left it in so you can get an idea of what Ive tried just please know I would never leave my code in this state in general.
As reference here is the dissassembled userspace test program (I included the command for full transparency with how I am generating this):
The pertinent process/userspace_entry code and also here & here, again I left in multiple versions/attempts I took at this I know its not clean, also I had tried various debugging paths within the entry code but abandoned them as all I ended up doing was messing around with the state and clobbering registers on accident.
The pertinent syscall init and management code , the same disclaimer above applies, the goal as I mentioned before is to just hi the `ud2` so I know I got there.
Pretty much anything else immediately relevant will be in /proc
Most recent debug output:
Final Words:
This is my first time posting here and honestly reaching out to anyone about my little OS. If I messed up anything in my post I apologize. I will be eternally grateful to anyone who can spare some time to help a girl out on this <3 Hopefully I can pay it forward and help others in the os dev journey!
Thanks <3
( Reading/mood song: https://open.spotify.com/track/7vhtylsJ ... 3625774738 )
Intro:
I am making a Rust OS for fun and learning, so I guess I would fall into the hobby category, and I have made pretty good progress so far but I am blocked on something I would love some help with, please :)
I read the wiki article on asking questions so I want to make sure I get a few things out of the way first:
- I do have the necessary programming background
- I did research my problem as thoroughly as possible on the wiki, online, in Understanding the Linux Kernel book, and trying to find linux or other os source code (Im not a von duct tape and honestly I find it kinda cute you have all those archetypes lol)
- I will do my best to share the right amount of code: not to much, not too little. However you may wish to see more so just ask and I shall provide <3
I have been blocked on this for a bit now and Im really losing it :'(
Goal:
To get my basic/initial pass at entering user space and entering a syscall to work correctly and fully i.e. after loading a program (ELF) and setting up necessary resources (stack etc) I should be able to have it begin executing and trigger a syscall that I am able to then handle.
Background:
My OS is written in Rust and right now is architecture dependent (x86_64) as I have not gotten far enough in development to refactor out the architecturally dependent components. I am using the rust boot loader 0,11,13 crate (I saw Limine is big on r/osdev but again I want something simple & working for now and later I can refactor booting into something better longterm). I am relatively early on in my OS development and so far have the following components implemented and, seemingly, working:
- Paging
- Interrupts
- An allocator
- GDT
- ELF loading (basic - no PIE/ALSR/Dynamic Linking)
- Process abstraction & basic resources (VM Areas, PID, kernel stack, user stack)
- Some other misc utilities
Problem:
I am able to load a user program (ELF) into memory and setup its initial resources but when I go to call the non-returnable enter_userspace sequence nothing happens. By nothing I mean either triple fault or just getting hung, I can’t tell — I am using QEMU and when I pass the no-reboot flag it hangs, without it it, sometimes, reboots but gets to the same state.
I have verified, to the best of my ability/knowledge that the process’s resources are indeed properly initialized: for example the entry point address is correct, all relevant memory regions are mapped in both the kernels page table and the processes (I have even gone overkill a bit here in a temporary attempt to fix the issue), the stack is setup with the correct stack address getting set in RSP, etc. I have also made sure the raw bytes at various addresses match the disassembled user space test binary. (I’m happy to elaborate on this to whoever helps I am just trying to precent this post from becoming an essay/too long and then no one has the patience to help D: )
Now, onto the syscall side of things. The only way I can really tell if things are working (viz.. to see a write syscall from user space telling me its executing correctly) is if I can get to my syscall entry point where, for debugging purposes I have put a `ud2` instruction as the first instruction called. I have also verified as best I can that my sys call init is correct — the MSR seem to have the right values and my syscall entry point’s address is mapped in both page tables and also set in STAR. My segments/GDT seem to be right as well.
My initial assumption having done all this verifying is that my issue is somewhere in my userspace entry assembly and/or my syscall entry assembly. This is in part because assembly is my weakest thing right now so I’m less equipped to confidently check it. For what it’s worth I have put time into actually learning NASM and such but it’s obviously unforgiving so I wouldn’t be surprised if I have some small ‘gotcha’ in my code.
To start I have put below my process entry point which is a rust method attached to my scheduler that does a little process resource setup and then calls the user space entry assembly. I have also attached the entirety of my syscall management code (which is very little right now). Lastly I have my disassembled user space test program and some debug output from my terminal when running my OS. Again I am trying to keep this relatively minimal in the hopes my post isn’t too terribly long, but you can ask to see any other part of my code and I will send it. I have my full code in the repo linked if you wish to peruse :)
Code:
NOTE: while I am trying to learn and so am open to all sorts of advice and critique please do not judge me too much on the code shown as it is NOT at a clean checkpoint and, intentionally, contains lots of attempts and temporary fixes etc. I left it in so you can get an idea of what Ive tried just please know I would never leave my code in this state in general.
As reference here is the dissassembled userspace test program (I included the command for full transparency with how I am generating this):
Code: Select all
objdump -d init/target/x86_64-unknown-none/release/init.stripped
init/target/x86_64-unknown-none/release/init.stripped: file format elf64-x86-64
Disassembly of section .text:
0000000000400000 <.text>:
400000: b8 01 00 00 00 movl $0x1, %eax
400005: bf 01 00 00 00 movl $0x1, %edi
40000a: ba 15 00 00 00 movl $0x15, %edx
40000f: 48 c7 c6 00 10 60 00 movq $0x601000, %rsi # imm = 0x601000
400016: 0f 05 syscall
400018: b8 01 00 00 00 movl $0x1, %eax
40001d: ba 01 00 00 00 movl $0x1, %edx
400022: 48 c7 c6 15 10 60 00 movq $0x601015, %rsi # imm = 0x601015
400029: 0f 05 syscall
40002b: b8 01 00 00 00 movl $0x1, %eax
400030: ba 16 00 00 00 movl $0x16, %edx
400035: 48 c7 c6 16 10 60 00 movq $0x601016, %rsi # imm = 0x601016
40003c: 0f 05 syscall
40003e: b8 01 00 00 00 movl $0x1, %eax
400043: ba 01 00 00 00 movl $0x1, %edx
400048: 48 c7 c6 15 10 60 00 movq $0x601015, %rsi # imm = 0x601015
40004f: 0f 05 syscall
400051: b8 01 00 00 00 movl $0x1, %eax
400056: bf 02 00 00 00 movl $0x2, %edi
40005b: ba 18 00 00 00 movl $0x18, %edx
400060: 48 c7 c6 2c 10 60 00 movq $0x60102c, %rsi # imm = 0x60102C
400067: 0f 05 syscall
400069: b8 3c 00 00 00 movl $0x3c, %eax
40006e: 31 ff xorl %edi, %edi
400070: 0f 05 syscall
400072: 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 nopw %cs:(%rax,%rax)
400080: eb fe jmp 0x400080 <.text+0x80>The pertinent syscall init and management code , the same disclaimer above applies, the goal as I mentioned before is to just hi the `ud2` so I know I got there.
Pretty much anything else immediately relevant will be in /proc
Most recent debug output:
Code: Select all
INFO : Framebuffer info: FrameBufferInfo { byte_len: 2764800, width: 1280, height: 720, pixel_format: Bgr, bytes_per_pixel: 3, stride: 1280 }
INFO : 4th Stage
INFO : BiosInfo { stage_4: Region { start: 130000, len: 25300 }, kernel: Region { start: 1000000, len: 4be170 }, ramdisk: Region { start: 14bf000, len: 0 }, config_file: Region { start: 14bf000, len: 0 }, last_used_addr: 14befff, framebuffer: BiosFramebufferInfo { region: Region { start: fd000000, len: 2a3000 }, width: 500, height: 2d0, bytes_per_pixel: 3, stride: 500, pixel_format: Bgr }, memory_map_addr: e244, memory_map_len: 7 }
INFO : BIOS boot
INFO : Elf file loaded at Pointer {
addr: 0x0000000001000000,
metadata: 4972912,
}
INFO : virtual_address_offset: 0x0
INFO : Handling Segment: Ph64(ProgramHeader64 { type_: Ok(Load), flags: Flags(4), offset: 0, virtual_addr: 200000, physical_addr: 200000, file_size: 11ab4, mem_size: 11ab4, align: 1000 })
INFO : Handling Segment: Ph64(ProgramHeader64 { type_: Ok(Load), flags: Flags(5), offset: 11ac0, virtual_addr: 212ac0, physical_addr: 212ac0, file_size: 43266, mem_size: 43266, align: 1000 })
INFO : Handling Segment: Ph64(ProgramHeader64 { type_: Ok(Load), flags: Flags(6), offset: 54d28, virtual_addr: 256d28, physical_addr: 256d28, file_size: 20, mem_size: 2d8, align: 1000 })
INFO : Mapping bss section
INFO : Handling Segment: Ph64(ProgramHeader64 { type_: Ok(Load), flags: Flags(6), offset: 54d48, virtual_addr: 257d48, physical_addr: 257d48, file_size: 8, mem_size: 24460, align: 1000 })
INFO : Mapping bss section
INFO : Entry point at: 0x2220b0
INFO : Creating GDT at PhysAddr(0x15ee000)
INFO : Map framebuffer
INFO : Map physical memory
INFO : Allocate bootinfo
INFO : Create Memory Map
INFO : Create bootinfo
INFO : Jumping to kernel entry point at VirtAddr(0x2220b0)
INDYOS INITIALIZING...
Frame allocator tracking 128448 frames (513792 KB of memory)
PAGING INITIALIZED
Loading IDT...
IDT loaded successfully!
IDT INITIALIZED
Loading GDT...
GDT loaded successfully!
GDT INITIALIZED
Manual STAR value: 0x10000800000000
Will set CS to: 0x23
Will set SS to: 0x1b
STAR readback: SegmentSelector { index: 4, rpl: Ring0 }
LSTAR written: 0x21369c
LSTAR readback: 0x21369c
Match: true
============================
SYSCALL INITIALIZED
LSTAR: 0x21369c
STAR: (SegmentSelector { index: 4, rpl: Ring0 }, SegmentSelector { index: 3, rpl: Ring0 }, SegmentSelector { index: 1, rpl: Ring0 }, SegmentSelector { index: 2, rpl: Ring0 })
SFMask: RFlags(DIRECTION_FLAG | INTERRUPT_FLAG | TRAP_FLAG)
[Kernel] Demand-paged at VirtAddr(0x444444440000) [PF: 1 total (1 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444441000) [PF: 2 total (2 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444442000) [PF: 3 total (3 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444443000) [PF: 4 total (4 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444444000) [PF: 5 total (5 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444445000) [PF: 6 total (6 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444446000) [PF: 7 total (7 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444447000) [PF: 8 total (8 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444448000) [PF: 9 total (9 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444449000) [PF: 10 total (10 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444444a000) [PF: 11 total (11 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444444b000) [PF: 12 total (12 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444444c000) [PF: 13 total (13 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444444d000) [PF: 14 total (14 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444444e000) [PF: 15 total (15 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444444f000) [PF: 16 total (16 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444450000) [PF: 17 total (17 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444451000) [PF: 18 total (18 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444452000) [PF: 19 total (19 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444453000) [PF: 20 total (20 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444454000) [PF: 21 total (21 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444455000) [PF: 22 total (22 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444456000) [PF: 23 total (23 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444457000) [PF: 24 total (24 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444458000) [PF: 25 total (25 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444459000) [PF: 26 total (26 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444445a000) [PF: 27 total (27 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444445b000) [PF: 28 total (28 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444445c000) [PF: 29 total (29 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444445d000) [PF: 30 total (30 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444445e000) [PF: 31 total (31 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444445f000) [PF: 32 total (32 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444460000) [PF: 33 total (33 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444461000) [PF: 34 total (34 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444462000) [PF: 35 total (35 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444463000) [PF: 36 total (36 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444464000) [PF: 37 total (37 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444465000) [PF: 38 total (38 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444466000) [PF: 39 total (39 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444467000) [PF: 40 total (40 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444468000) [PF: 41 total (41 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444469000) [PF: 42 total (42 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444446a000) [PF: 43 total (43 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444446b000) [PF: 44 total (44 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444446c000) [PF: 45 total (45 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444446d000) [PF: 46 total (46 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444446e000) [PF: 47 total (47 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x44444446f000) [PF: 48 total (48 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444470000) [PF: 49 total (49 paged, 0 failed)]
[Kernel] Demand-paged at VirtAddr(0x444444471000) [PF: 50 total (50 paged, 0 failed)]
Zeroed out heap memory region init at VirtAddr(0x444444440000) -- amount 204800
Begin process initialization...
allocated pid: 1 -- about to setup virtual memory
=== PAGE TABLE COPY VERIFICATION ===
Syscall handler address: 0x21369c
Syscall handler PML4 index: 0
Kernel PML4[0]: mapped
Process PML4[0]: mapped
===================================
Created process 1...
[src/proc/elf.rs:218] header = ELFHeader {
ident: ELFIdent {
magic: [
127,
69,
76,
70,
],
class: 2,
data: 1,
version: 1,
osabi: 0,
abiversion: 0,
pad: [
0,
0,
0,
0,
0,
0,
0,
],
},
elf_type: 2,
machine: 62,
version: 1,
entry: 4194304,
phoff: 64,
shoff: 12320,
flags: 0,
ehsize: 64,
phentsize: 56,
phnum: 3,
shentsize: 64,
shnum: 5,
shstrndx: 4,
}
about to validate page alignment....
about to validate segments...
about to load into memory...
Page Page[4KiB](0x400000): copying 130 bytes at offset 0
Copied 130 bytes total
Successfully loaded program into memory!
about to validate page alignment....
about to validate segments...
about to load into memory...
Page Page[4KiB](0x601000): copying 68 bytes at offset 0
Copied 68 bytes total
Successfully loaded program into memory!
Process created!
Done initializing process
Add process to scheduler -- about to run
Scheduled a process to run
argv_ptrs[0] = 0x7fffffffeffb
envp_ptrs: []
setup elf aux vector at: 0x7fffffffefd0
Stack has been setup with stack pointer: 0x7fffffffefb0
TSS updated: kernel stack = VirtAddr(0xffffa00000003ff0)
About to enter userspace via sysret path
Entry: 0x400000
Stack: 0x7fffffffefb0
=== MEMORY LAYOUT DEBUG ===
Entry point code disassembly shows:
400000: b8 01 00 00 00 mov eax, 1
400005: bf 01 00 00 00 mov edi, 1
40000a: ba 15 00 00 00 mov edx, 0x15
40000f: 48 c7 c6... mov rsi, 0x601000
400016: 0f 05 syscall <-- FIRST SYSCALL
Checking user code at 0x400000:
Bytes: b8 01 00 00 00 bf 01 00 00 00 ba 15 00 00 00 48
c7 c6 00 10 60 00 0f 05 b8 01 00 00 00 ba 01 00
Matches expected: mov eax,1; ...; syscall
Checking user data at 0x601000:
Bytes: 48 65 6c 6c 6f 20 66 72 6f 6d 20 75 73 65 72 73 70 61 63 65 21 0a 54 68 69 73 20 69 73 20 61 20
ASCII: Hello from userspace!.This is a
Checking syscall handler at 0x21369c:
Mapped to phys 0x101269c
First bytes: 0f 0b 50 48
Looks good: starts with UD2 (0f 0b)!
===========================
This is my first time posting here and honestly reaching out to anyone about my little OS. If I messed up anything in my post I apologize. I will be eternally grateful to anyone who can spare some time to help a girl out on this <3 Hopefully I can pay it forward and help others in the os dev journey!
Thanks <3
