TanOS 0.1 - from-scratch Rust microkernel: ring-3 isolation, synchronous IPC, fault isolation (QEMU/KVM)

[TanOS]

TanOS is a from-scratch x86-64 microkernel in Rust. It boots under QEMU/KVM and demonstrates:
4-level paging - each process gets its own address space
Ring 3 userspace - loads ELF binaries from an initrd via iretq
Synchronous IPC - cross-address-space call/receive/reply rendezvous
Fault isolation + restart-on-crash - a ring-3 crash kills only that process; the kernel can restart it from its image while others keep running. (Restart-from-image, not full reincarnation - clients aren't transparently re-bound to the restarted process.)
IPC latency - ~3,800 cycles/round-trip under KVM (unoptimized; two CR3 reloads, no PCID, spinlocks per trip)
Design decisions worth knowing:
Single kernel stack - every int 0x80 syscall saves the full register frame onto one reused kernel stack. Context switching copies that frame into the current PCB, restores another process's frame + CR3, and iretqs. This works safely because the syscall gate clears IF, preventing nested entries, so one stack always suffices.
Memory layout puts the kernel identity-mapped in the low 1 GB (2 MB huge pages) reachable from every address space, userspace starting at 0x40000000, and the user stack at 0xC0000000 - chosen specifically to avoid non-canonical address faults (we hit that bug; it's documented in the README).
What to look for when testing: build and boot per the README, then check /tmp/s.txt - you should see the driver crash, reincarnate, and the survivor confirm it stayed alive throughout.
No disk image yet - builds from source via cargo +nightly-2024-01-15. Full instructions in the README.
Disclosure: built with heavy AI assistance (Claude). Judge the output, not the prose.
GitHub: https://github.com/hawkaiglai/tanos

-----
EDITED

[Octocontrabass]

It boots under QEMU/KVM

Does it work on bare metal or any other VMs?

Fault isolation + reincarnation — ring-3 crash kills only that process; kernel restarts it from its image while other processes survive

How does that impact IPC for the surviving processes? Since your kernel is a microkernel, presumably other processes will still try to communicate with the process that crashed.

int 0x80 syscall

Why do your system calls use INT 0x80 instead of SYSCALL?

kernel identity-mapped

This will prevent your OS from booting on many systems, possibly including QEMU depending on how it's configured. Normally you want your kernel to use the higher half of the address space, to keep it cleanly separated from userspace in the lower half. On x86-64 specifically, you'd also want to link the kernel to run in the highest 2GiB of the address space (at or above 0xFFFFFFFF80000000) since it allows the compiler to produce smaller code.

Judge the output

Oh, I'm definitely judging it.

[TanOS]

Thanks - this is exactly the kind of scrutiny I was hoping for, and a couple of your points sent me straight back to the code. Going in order:

▎ Does it work on bare metal or any other VMs?

No - only QEMU (KVM and TCG). I haven't tested bare metal, VirtualBox, VMware, or Bochs, and I'd expect it to need real work first: it assumes a legacy PIC + PIT (no APIC/ACPI), relies entirely on a serial port for output (no framebuffer - so on a
machine with no COM port it'd boot to a blank screen), and makes the low-memory layout assumption you flag below. I've scoped the claim to "QEMU/KVM" deliberately rather than overstate the reach, and I've now written that out explicitly as a limitation
in the README.

▎ How does that impact IPC for the surviving processes? [...] other processes will still try to communicate with the process that crashed.

You were right that this was broken, and it was the most useful question of the lot - so I fixed it. Previously, if a process was blocked in a call awaiting a reply and its server crashed, it hung forever: the rendezvous still referenced the dead PID
and nothing woke the waiter.

Now the kernel tracks which process owes the reply (in_service), and on process death it wakes any caller blocked on the dead one with an error instead of leaving it stuck. Commit: https://github.com/hawkaiglai/tanos/commit/cfcd5be - there's an
ipc_crash_demo build (--features ipc_crash_demo) that shows a client recovering from a server that crashes mid-request.

To be precise about what that does and doesn't do yet, though: the caller gets an error and must retry - the kernel does not transparently re-bind it to the reincarnated server, which comes back with a new PID and no inherited service identity.
Proper client re-binding (stable service identity, dependency tracking, re-establishing endpoints) is the MINIX-3 reincarnation-server problem, and that part isn't done. So "reincarnation" here demonstrates the kernel-side mechanism plus not-hanging on
peer death - not the full client-recovery story.

▎ Why do your system calls use INT 0x80 instead of SYSCALL?

No principled reason - int 0x80 was the simpler bring-up (a DPL-3 IDT gate, no LSTAR/STAR/SFMASK setup, no swapgs). I actually had SYSCALL scaffolding and deleted it as dead code because it was never wired up. SYSCALL is faster and the right long-term choice; I haven't switched because it isn't the bottleneck - the ~3,800 cycles is dominated by the two CR3 reloads per round-trip, not the trap entry.

▎ [identity-mapped kernel] will prevent your OS from booting on many systems [...] use the higher half [...] 0xFFFFFFFF80000000

Agreed, and this is the most valuable pointer. The kernel is linked at 0x100000 and identity-maps the low 1 GB, so it runs in the low half - a deliberate bring-up shortcut (it mirrors the early boot tables and keeps all RAM trivially reachable while I
build page tables), but I know it's not the right design. Moving to a higher-half kernel (link at 0xFFFFFFFF80000000, -mcmodel=kernel, keep a low physmap for physical access) is on the list as a real fix, not a someday - I've scoped it and it's a
contained change in the linker script, the boot trampoline, and per-address-space mapping; mostly I'm wary of the silent-reset-during-the-transition debugging. It's also a prerequisite for the bare-metal answer above. I've noted the current low-half
layout as a known limitation in the README too.

▎ Judge the output - Oh, I'm definitely judging

Please do - it's already produced one real fix and a clear roadmap. Thank you for the time.

[dseller]

It boots under QEMU/KVM

Does it work on bare metal or any other VMs?

Fault isolation + reincarnation — ring-3 crash kills only that process; kernel restarts it from its image while other processes survive

How does that impact IPC for the surviving processes? Since your kernel is a microkernel, presumably other processes will still try to communicate with the process that crashed.

int 0x80 syscall

Why do your system calls use INT 0x80 instead of SYSCALL?

kernel identity-mapped

This will prevent your OS from booting on many systems, possibly including QEMU depending on how it's configured. Normally you want your kernel to use the higher half of the address space, to keep it cleanly separated from userspace in the lower half. On x86-64 specifically, you'd also want to link the kernel to run in the highest 2GiB of the address space (at or above 0xFFFFFFFF80000000) since it allows the compiler to produce smaller code.

Judge the output

Oh, I'm definitely judging it.

It looks like you're talking to an LLM here.

[Octocontrabass]

To be precise about what that does and doesn't do yet, though: the caller gets an error and must retry - the kernel does not transparently re-bind it to the reincarnated server, which comes back with a new PID and no inherited service identity.

So your "process reincarnation" is just restarting the process if it dies? That doesn't sound new or unique to me.

int 0x80 was the simpler bring-up

I suppose that's fitting.

mostly I'm wary of the silent-reset-during-the-transition debugging.

Perhaps it would be easier to debug if you wrote the code yourself.

[TanOS]

I corrected the conflation in the title, the post, and the repo (link https://github.com/hawkaiglai/tanos/commit/68fb7ad).
- Thanks for pointing this Octocontrabass.