implementing a linux-based os from scratch, how to start?

[mmdmine]

hello everyone.
i like writing an os, but i found writing a kernel that works well needs a lot of experience and information that i don't have. so i decided to start with an easier project. i want to write user space of my os on top of linux, by abstracting linux system calls and then porting it to my own kernel when it got ready. i dont want to use gnu libraries and common unix executables exists in every linux system (like bash, mount, kill, ...), i want to write my own user space from scratch without rewriting a c compiler, but i dont know how to start. is there any guide or reference on writing a linux user space?
i will need an initializer that scans for hardware and loads needed kernel modules and mounts disks, and a display server (i will call it visual host) that uses graphic driver through an abstraction to draw gui, i won't use xorg and framebuffer.

[bzt]

Ehem, Linux from scratch? Or try TinyCore Linux, start with core.gz only, that's quite a minimal user-space to start from.

Otherwise boot a linux kernel with /sbin/init being your hello world application, and build up from there. Learn how to create an initrd and how to add libraries to it and basic file system required by your init application. The theory is:
1. firmware loads the boot loader (isolinux, grub, whatever)
2. boot loader loads the kernel
3. the kernel starts /sbin/init which supposed to be the first user-space application with pid of 1 and be the root of all the other processes.
However with modern Linux kernels this is not the case any more, as Linux starts kernel processes as well (like kthreadd, ksoftirqd etc.) and requires minimal user-space support during boot too (often workarounded by an initrd or a read-only mounted root). Kernel processes are not real user-space processes, but they are not strictly part of the kernel either, as they have their own process id.

An alternative would be the Minix3 (microkernel) or Xv6 (monolithic), both being small, simple and POSIX compliant (of course Open Source and well documented too). Unlike Linux, their /sbin/init is truely the first user-space process.

Good luck!
bzt

[mmdmine]

Ehem, Linux from scratch? Or try TinyCore Linux, start with core.gz only, that's quite a minimal user-space to start from.

Otherwise boot a linux kernel with /sbin/init being your hello world application, and build up from there. Learn how to create an initrd and how to add libraries to it and basic file system required by your init application. The theory is:
1. firmware loads the boot loader (isolinux, grub, whatever)
2. boot loader loads the kernel
3. the kernel starts /sbin/init which supposed to be the first user-space application with pid of 1 and be the root of all the other processes.
However with modern Linux kernels this is not the case any more, as Linux starts kernel processes as well (like kthreadd, ksoftirqd etc.) and requires minimal user-space support during boot too (often workarounded by an initrd or a read-only mounted root). Kernel processes are not real user-space processes, but they are not strictly part of the kernel either, as they have their own process id.

An alternative would be the Minix3 (microkernel) or Xv6 (monolithic), both being small, simple and POSIX compliant (of course Open Source and well documented too). Unlike Linux, their /sbin/init is truely the first user-space process.

Good luck!
bzt

thank you.
already i know how to build an initrd and setup gnulinux by hand. i want to write my own user space, mean i won't use gnu coreutils and their c library, instead i want to write it myself, it's hard but easier than writing kernel from scratch. i don't know how can i call kernel to do something for me (like allocating memory or mounting a partition). should i use freestanding environment for writing c library?

[nullplan]

You may have bitten off more than you can chew here. A C library is a really complicated undertaking (there is a lengthy list of requirements in POSIX, and how to connect those requirements to the system call layer is also rather complicated). musl is a C library for Linux which should provide a base for what you want to do. It does not implement many of the GNU extensions to POSIX, and is way leaner, and also way saner, than glibc. Downside is, some packages fail to build with it. Adding C++ support is harder.

If you are looking for ready-made packages, I suggest you look into musl, clang, llvm, libc++, busybox/toybox/sbase and a decent shell, and you should be alright. You might still need the GNU binutils, though.

How to call the kernel is written into the ABI documents, and before you try to write your own library, you should really absorb those first. And finally, yes, you must always compile a C library in freestanding mode. For the simple reason that the functions you are defining are reserved in hosted implementations.

[mmdmine]

You may have bitten off more than you can chew here. A C library is a really complicated undertaking (there is a lengthy list of requirements in POSIX, and how to connect those requirements to the system call layer is also rather complicated). musl is a C library for Linux which should provide a base for what you want to do. It does not implement many of the GNU extensions to POSIX, and is way leaner, and also way saner, than glibc. Downside is, some packages fail to build with it. Adding C++ support is harder.

If you are looking for ready-made packages, I suggest you look into musl, clang, llvm, libc++, busybox/toybox/sbase and a decent shell, and you should be alright. You might still need the GNU binutils, though.

How to call the kernel is written into the ABI documents, and before you try to write your own library, you should really absorb those first. And finally, yes, you must always compile a C library in freestanding mode. For the simple reason that the functions you are defining are reserved in hosted implementations.

thank you. i'll search for abi or use musl.

[bzt]

Yeah, musl is a good advice, indeed. I don't think you can boot a Linux kernel without a minimal user space any more (not without a manually compiled kernel and dozens of dirty hacks that is). Check out TinyCore, it's core.gz is really minimal to start with. You can even get it smaller by throwing out things that you don't need for a Hello World /sbin/init.

As a bare minimum, you'll need /sbin/init (your first app), linux-vdso.so (or linux-gate.so), ld-linux.so (the mandatory ELF "interpreter" under Linux), probably libdl.so (for dlopen) too, and of course libc.so in the initrd. That's 5 user space files at least in addition to the kernel. I think it's pretty problematic to go below this with Linux, that's why I suggested Xv6 and Minix.

For the first user space program, I'd also suggest to take a look at busybox shell. It's a single binary which you can start with the Linux command line "init=/bin/sh", it has very minimal library dependency, far less than any other shell. Use "ldd" to figure out which libraries you'll need in the initrd (keep in mind that libraries might depend on other libraries).

Cheers,
bzt

[nullplan]

As a bare minimum, you'll need /sbin/init (your first app), linux-vdso.so (or linux-gate.so), ld-linux.so (the mandatory ELF "interpreter" under Linux), probably libdl.so (for dlopen) too, and of course libc.so in the initrd. That's 5 user space files at least in addition to the kernel. I think it's pretty problematic to go below this with Linux, that's why I suggested Xv6 and Minix.

No, this is false. You don't need an initramfs if the kernel has all drivers needed to find the root disk builtin, and you only need /sbin/init, which you can compile and link statically, and then there are no dependencies to any other files anymore. Of course, a system containing a single program is likely to be a supremely academic matter.

The VDSO (linux-vdso.so or linux-gate.so) is never a physical file, except during the kernel build process. It is linked into the kernel and injected into every process, with the base address given by the aux vector AT_SYSINFO_EHDR. Also, musl links all code into a single library file, so should you decide to do dynamic linking, you still only need libc.so (and none of the other names for it, but you still need /lib/ld-musl-$(ARCH)$(SUBARCH).so.1 as a symlink to the libc file). libdl, libm, librt, and libpthread (and I probably forgot a name) are all empty files present only so the linker does not error when presented with those names. But in musl, it is all linked into a single file. Because there is basically no benefit to splitting it up.

Regarding initramfs: The kernel has a builtin initramfs, which consists of /dev, /dev/console, and /root. Other initramfs-images can be appended to this. And if no /init is present in any initramfs given during boot time, then the kernel will interpret the root= command line argument to create /dev/root as the device file pointing to the correct root device, mount that to /root, and perform the whole switch_root() thing before it will execve() /sbin/init. This is why you see /dev/root in /proc/mounts, even though that file likely doesn't exist in your real rootfs. The code to do this is now in the Linux source code in init/do_mounts.c (search for prepare_namespace()).

The ELF interpreter is not mandatory, nor does it have a fixed name. An ELF file can request an interpreter by means of the PT_INTERP program header, and then that interpreter has to be present and not name an interpreter itself. /lib/ld-linux.so.2 is the name used by the glibc interpreter, so if you are linking against musl, that is about the last thing you need. However, the interpreter is mandatory for dynamically linked programs. If simplicity is your goal, you might want to circumvent this complication at first.

One more thing: /sbin/init exiting (due to crash, program end, or fatal signal) is a reason for a kernel panic. So, don't. Most simple /sbin/inits will just put themselves to sleep in an infinite system call, rather than end.

[bzt]

Ok, I see a little bit misunderstanding here.

No, this is false

In theory you're right it's false, but in practice it is true. That's what I meant by "not without a manually compiled kernel and dozens of dirty hacks that is". One of those dozen hacks is not using the default ld-linux.so interpreter or linking init statically for example as you mentioned. As you put it,

is likely to be a supremely academic matter

which is a perfect description of the situation imho.

However, the interpreter is mandatory for dynamically linked programs.

You mean under Linux, right? Because in my OS I use dynamically linked ELFs without interpreter so it is doable; and it is also true that the Linux kernel does not have a run-time linker, instead it delegates that task to the ELF interpreter.

But in musl, it is all linked into a single file.

I know, that's why I said it was a good advice. And I really think it is! The OP is better off with musl than with glibc. I forgot that libdl is also included in musl, sorry about that.

The kernel has a builtin initramfs, which consists of /dev, /dev/console, and /root

After boot, your environment is not yet suitable for user space even with that builtin ramfs. The kernel does not provide libc, and there are lots of things to be done by init, like remounting root rw, creating nodes in /dev (or mounting udev on /dev), mounting /proc (musl depends on it for some functions), starting getty etc., all described nicely in Linux from Scratch. That's all I meant by you'll need a user-space too to boot Linux properly, nothing more. For a viable and practically useful Linux you should create your own initrd (or have a root partition mounted read-only during boot) with all the dependencies. Not a small task, but definitely smaller than writing an entire OS.

I think we are saying the same thing, just with different words

Cheers,
bzt

[mmdmine]

in my archlinux setup I have configured kernel to boot without initramfs and it works perfect.
...
still you dont know what I mean. I dont want to create "yet another linux distrobute", and even I dont want to use a Unix shell and a Unix-like enviroment. my design has 3 layers, first a library that calls kernel, second my OS API, third my shell that runs on top of my API. if I try to write these in C, I will need a C Library then I could use musl or glibc or write my own library.

[nullplan]

In theory you're right it's false, but in practice it is true. That's what I meant by "not without a manually compiled kernel and dozens of dirty hacks that is". One of those dozen hacks is not using the default ld-linux.so interpreter or linking init statically for example as you mentioned.

Static linking is not a dirty hack. You do not actually want a real init daemon in your initramfs (initrd is the older mechanism), you only want a program there that can find the root file system, mount it, and change to it. That is all. The initramfs is only for finding the root FS. And due to all the mechanisms built into the kernel right now, it is even sufficient to build an initramfs consisting only of the modules needed for that. The kernel will try to find the root from the root= command line option, and will mount it, usually read-only (there are the ro and rw command line options to make sure).

Of course, you can take this game arbitrarily far: What if no root is found? Do you drop into a shell? Then that means we need a shell and all the stuff it needs. Do you want to check the root FS before continuing? What if that check fails? Etc. pp. Of course, "root-fs unavailable" is a fatal error, so just stopping the boot process with an explanatory message might also just work better.

You mean under Linux, right? Because in my OS I use dynamically linked ELFs without interpreter so it is doable; and it is also true that the Linux kernel does not have a run-time linker, instead it delegates that task to the ELF interpreter.

Yes, I mean under Linux, the OP is asking for an OS based on Linux. You are of course free to do as you like in your own OS, but I will note that I do know how much code goes into a dynamic interpreter, and just how data-dependent it is, and I do not want this stuff to run in ring 0. I mean, you have to interpret the PT_DYNAMIC section, which contains pointers, pointers everywhere. And you have to parse strings, and you have to go into recursion, and there are just so many ways bad or evil data can corrupt state that running that stuff in ring 0 is flat out irresponsible. With the dynamic interpreter, all of the dynamic linking stuff is controlled from user-space, and can only corrupt user space. And for setuid executables, it should only ever touch admin-controlled data, so there is no way, evil data can be implanted to subvert the control flow of anything with a higher privilege level. And anything at your own level, well, you are welcome to hack yourself.

As for you, mmdmine:

in my archlinux setup I have configured kernel to boot without initramfs and it works perfect.

My point, exactly.

still you dont know what I mean. I dont want to create "yet another linux distrobute", and even I dont want to use a Unix shell and a Unix-like enviroment. my design has 3 layers, first a library that calls kernel, second my OS API, third my shell that runs on top of my API. if I try to write these in C, I will need a C Library then I could use musl or glibc or write my own library.

I think I'm getting you. Something completely different on top of Linux. Something like Android, perhaps? Anyway, musl still does try to present a POSIX compatible interface, and requires your Linux to be configured in some ways, to be compatible with POSIX. For example, if you want pseudo-terminals, you really need a devpts file system, mounted on /dev/pts. And a surprising amount of interfaces requires proc to be mounted on /proc. But those are small things.

Bear in mind that musl sets up a POSIX compatible programming environment. What you do for user interface really is up to you. Also, musl and glibc contain not only the C library (stuff like strchr(), memcmp(), and stuff), but also wrappers for all the system calls, and therefore are needed pretty much by every language. There is this weird thing going on with Go at the moment, where they basically reinvent the wheel and do all the system calls from the Go runtime directly, but for threading they fall back onto libpthread. That doesn't work, and as a result, the Go runtime ends up stepping on musl's libpthread, and I have no idea if that conflict is resolved yet. Another example of needing to go whole hog or not at all: It is possible to have userspace in Linux without libc, but it is hard, and you end up doing a lot of what libc did again. If you want to do syscalls yourself, great, but then do it for everything, don't switch back to libc wrappers in the middle.

[bzt]

in my archlinux setup I have configured kernel to boot without initramfs and it works perfect.

Please read my whole post. Yes it works if you have a read-only mounted root fs. I've never said ramfs is the only way, I said it is the most reasonable and recommended way.

still you dont know what I mean. I dont want to create "yet another linux distrobute"

I understand that perfectly, my point is, the Linux kernel has user-space parts (independent to the distro files, common to all distros).

All I'm saying you'll need more than the kernel, no matter if you provide those from an initrd or a read-only root fs, but you have to provide those files somehow. Initial ramdisk is just the best way to do this, but not the only way.

I dont want to use a Unix shell and a Unix-like enviroment. my design has 3 layers, first a library that calls kernel, second my OS API, third my shell that runs on top of my API. if I try to write these in C, I will need a C Library then I could use musl or glibc or write my own library.

Which looks fine, just keep in mind that you'll have to mimic Unix to some extent if you build on the Linux kernel. I cannot tell you how much, it highly depends on which libraries are you going to use and how many libraries are you willing to rewrite from scratch. For example when you provide all the Linux-related libs (you don't want to rewrite those) and write your own libc (calling the kernel with int 0x80 or syscall instruction) then you can diverge from Unix more, but with a vanilla POSIX-compliant libc it will be more difficult.

In a nutshell: for Linux kernel, you'll need the kernel itself, some related libraries (like ld-linux), and a library that translates user-space function calls into kernel calls (easiest way to use a libc for that, but you can write your own), and one executable in user-space that does the OS initialization and with that finishes the booting procedure. This is all distro-independent. Now how that executable works (is it systemd, SysV init, or just a pure shell) and what other services it starts is distro-specific.

What you wrote so far, I'd suggest to fork musl and rewrite it's functions to your OS API. That way you'd have a single library that would provide your OS API to the user-space applications, but under the hood it would call the Linux kernel to implement those. Later, when you want to switch to your final kernel, then you'd have to rewrite this library, but only this library and your user-space could run just as-is.

Static linking is not a dirty hack.

Yes it is if we are talking about libc and not using ld-linux.

you only want a program there that can find the root file system, mount it, and change to it. That is all.

That's typical Linux user-space. You can diverge from that easily. TinyCore for example can provide a full environment including X in the initrd (or you may call it ramfs if you prefer that word). I use "initrd" as "initial ram disk" sense, not to refer to any specific implementation, sorry if that was misleading in any way.

I still think we're saying exactly the same thing.

For the OP: I think all has been said that can be said, there's nothing left for you than to start your experimenting. Create a Hello World init, include all the Linux kernel-related libraries, provide the rest to your liking. Then start to add functionality to the init and build up your own user-space from there.

Cheers,
bzt

[mmdmine]

Thank you for explanations.
Actually I mount root RW because it's btrfs and doesn't need a fsck on bootup.
I will start my OS as soon as possible.

[iocoder]

if I try to write these in C, I will need a C Library then I could use musl or glibc or write my own library.

If the intermediate API layer satisfies all what you need, then the C library is useless. It is not a rule.

[mmdmine]

if I try to write these in C, I will need a C Library then I could use musl or glibc or write my own library.

If the intermediate API layer satisfies all what you need, then the C library is useless. It is not a rule.

This was what I want to hear. I dont need to have a library same as glibc and others e.g. I can have memory_allocate() instead of malloc() or have a different string library. only problem is that it's not compatible with existing softwares that's not my goal, instead I can allow user to have a 3rd party C Library to install these softwarrs.

[iocoder]

if I try to write these in C, I will need a C Library then I could use musl or glibc or write my own library.

If the intermediate API layer satisfies all what you need, then the C library is useless. It is not a rule.

This was what I want to hear. I dont need to have a library same as glibc and others e.g. I can have memory_allocate() instead of malloc() or have a different string library. only problem is that it's not compatible with existing softwares that's not my goal, instead I can allow user to have a 3rd party C Library to install these softwarrs.

Exactly. There is still an option to provide a POSIX API (including C library) on top of your OS API if you want. This design pattern is quite common.

Having your own libraries (like your own memory_allocate() as you mentioned) is a great idea: it liberates your creative instinct from the UNIX prison. I am taking a similar approach for my UNIX-Unlike microkernel (separate library for each server: sched, hal, fs, strings, utils, mem, etc.).