Fun little retro os project
Posted: Tue Apr 28, 2026 10:47 am
Like many of you I always had an itch to write a hobby os. My dream was one where I could play my childhood dos games like PoP, keen, doom etc. natively, along with running normal linux programs. With the advent of AI coding, one can now focus on core design and have AI implement dos/dpmi/linux api's which makes getting a project to a fun state where you can actually play old games in your OS doable within a reasonable time investment.
https://github.com/gerben-stavenga/RetroOS
For a video showing multitasking dos + busybox linux.
https://youtu.be/Clh45xUpZNY
Before you are bemoaning AI slop, I put some real thought into the architecture which became actually quite nice. It's a 32-bit kernel supporting 16, 32 and 64 bit user programs. The "shell" is DN (a open source 16-bit Norton commander clone) from which you can start your 16 bit dos games (or 32 bit dpmi games) and also 32/64 bit linux binaries (of course not close to supporting full linux syscall API surface,, but some minimal set of file / process syscalls, with some rust compiled test programs using musl).
It's a high mem kernel 0xC000_0000 with recursive paging. The paging code is written so that the same code works for legacy (32 bit paging), PAE and x64 (PML4). The page tables are setup so that the kernel can seamlessly toggle between PAE and PML4 (compat mode). This way it supports 16-bit 8086 (using vm86 in old-style 32 bit protected mode), and 64 bit programs using 32-bit compat mode. Just before returning to userspace it checks if it's vm86 or 64bit and toggles if necessary.
The 32, 64 entry points all quickly merge into a single entry point with a canonical Regs frame. I tried to deal with hardware warts as much as possible in one place. The regs struct has the 64bit registers rax-r15, segments and the 64 bit entry stack frame. 32 bit entry follows the same layout by pushing 0 for the high dwords. The 32 bit entry frame is converted into 64 bit frame at entry and vice versa at exit. The extra segments that vm86 pushes are copied into their regular es,ds fields in Regs if vm86 mode entry is detected and vice versa on exit.
So ring-0 has a single entry point (call from entry.asm) and single exit point (return to entry.asm). The bulk of the kernel runs in ring-1 so that it can run in privileged pages but can make normal syscalls to ring-0. This makes ring-1 very easy to understand and makes ring-0 basically trivial.
Ring-0 flow
If userspace is interrupted
1) if pagefaults service directly/return if possible (COW/demand paging) else goto 3
2) if irq handle it and push interrupt-event in queue, then goto 3
3) swap frame with kernel frame, set (intno, extra) as return values (only execute syscall goes to userspace) and return (to ring-1 kernel)
if kernel space is interrupted
1) if pagefault handle (often kernel panic as ring-1 mostly shouldnt pagefault)
2) irq => handle and push in queue return to ring-1
3) syscall execute => swap frame with frame as provided return (to userspace)
4) few other syscalls like map mem that just return to ring-1
If ring-0 is interrupted service irq and return
Ring-1 flow
Just a simple event loop consisting of normal rust code. Because this design is free of hidden interrups/control flow, rust drops(RAII) just works. Hence from here on, AI is capable of producing ton of code implementing dos 21 functions, xms/ems/dpmi functions and some basic linux syscalls.
It's playing keen (it flickers with qemu but i suspect its qemu ega issue where wrapping 16bit segments is not implemented), original doom, original quake, prince of persia. You can multitask between them with F11 which background a DOS task (command.com does linux fork+exec).
I tried hard to minimize asm code. Only entry.asm contains actual logic, but it's only setting up a unified frame and upon exit a test to use 32 or 64 bit exit based on cpu-mode. The only other asm function is toggle between PAE and compat that needs to be identity mapped on a page (0xF000). One more trick is using segments on entry to make the kernel symbols lineup at the right high address at which they are linked. Often initial page tables are constructed in asm to map kernel high before calling into high level code. This allows building page tables using normal code (otherwise you have to be very careful that you don't use addresses that are at physical address because paging isnt active or use asm).
https://github.com/gerben-stavenga/RetroOS
For a video showing multitasking dos + busybox linux.
https://youtu.be/Clh45xUpZNY
Before you are bemoaning AI slop, I put some real thought into the architecture which became actually quite nice. It's a 32-bit kernel supporting 16, 32 and 64 bit user programs. The "shell" is DN (a open source 16-bit Norton commander clone) from which you can start your 16 bit dos games (or 32 bit dpmi games) and also 32/64 bit linux binaries (of course not close to supporting full linux syscall API surface,, but some minimal set of file / process syscalls, with some rust compiled test programs using musl).
It's a high mem kernel 0xC000_0000 with recursive paging. The paging code is written so that the same code works for legacy (32 bit paging), PAE and x64 (PML4). The page tables are setup so that the kernel can seamlessly toggle between PAE and PML4 (compat mode). This way it supports 16-bit 8086 (using vm86 in old-style 32 bit protected mode), and 64 bit programs using 32-bit compat mode. Just before returning to userspace it checks if it's vm86 or 64bit and toggles if necessary.
The 32, 64 entry points all quickly merge into a single entry point with a canonical Regs frame. I tried to deal with hardware warts as much as possible in one place. The regs struct has the 64bit registers rax-r15, segments and the 64 bit entry stack frame. 32 bit entry follows the same layout by pushing 0 for the high dwords. The 32 bit entry frame is converted into 64 bit frame at entry and vice versa at exit. The extra segments that vm86 pushes are copied into their regular es,ds fields in Regs if vm86 mode entry is detected and vice versa on exit.
So ring-0 has a single entry point (call from entry.asm) and single exit point (return to entry.asm). The bulk of the kernel runs in ring-1 so that it can run in privileged pages but can make normal syscalls to ring-0. This makes ring-1 very easy to understand and makes ring-0 basically trivial.
Ring-0 flow
If userspace is interrupted
1) if pagefaults service directly/return if possible (COW/demand paging) else goto 3
2) if irq handle it and push interrupt-event in queue, then goto 3
3) swap frame with kernel frame, set (intno, extra) as return values (only execute syscall goes to userspace) and return (to ring-1 kernel)
if kernel space is interrupted
1) if pagefault handle (often kernel panic as ring-1 mostly shouldnt pagefault)
2) irq => handle and push in queue return to ring-1
3) syscall execute => swap frame with frame as provided return (to userspace)
4) few other syscalls like map mem that just return to ring-1
If ring-0 is interrupted service irq and return
Ring-1 flow
Code: Select all
loop {
let event = ring_0_syscall_execute(&mut regs); // run process returns with some event (syscall, interrupt)
let action = match current_thread_os_personality {
Dos(dos_thread) => HandleDos(event, dos_thread, &mut regs),
Linux(linux_thread) => HandleLinux(event, linux_thread, &mut regs),
...
}
match action {
Switch(tid) => ring_0_syscall_switch_address_space(..)
...
}
}
It's playing keen (it flickers with qemu but i suspect its qemu ega issue where wrapping 16bit segments is not implemented), original doom, original quake, prince of persia. You can multitask between them with F11 which background a DOS task (command.com does linux fork+exec).
I tried hard to minimize asm code. Only entry.asm contains actual logic, but it's only setting up a unified frame and upon exit a test to use 32 or 64 bit exit based on cpu-mode. The only other asm function is toggle between PAE and compat that needs to be identity mapped on a page (0xF000). One more trick is using segments on entry to make the kernel symbols lineup at the right high address at which they are linked. Often initial page tables are constructed in asm to map kernel high before calling into high level code. This allows building page tables using normal code (otherwise you have to be very careful that you don't use addresses that are at physical address because paging isnt active or use asm).