What is the secret of creating a kernel from scratch?

[osdev199]

I am struggling with creating my own 64-bit Unix-like kernel from scratch for the past 1 year and 2 months. I have only succeeded with creating device drivers (including NVMe), interrupt handling, UEFI bootloader, and recently the physical memory manager.

I think (and I'm unsure if it is the "exact" issue) that I don't know about the Unix kernel design and architecture. I think reading books on OS concepts and on the design of Unix OS first is just too much theoretical. Every time, I give up. I prefer learning by doing and learn as you go. I believe in hacking. And at the same time I don't want to compromise on knowing the "needed" technical knowledge.

I am not being able to crack this problem - How to create a kernel from scratch? Let's say if I am done with physical memory manager, then what should to do next? I don't know if I miss the high level understanding or? I emailed a lot of people who have created their own kernels and also who are working in Linux and freebsd but no one replied. Also, there is no any latest and simple 64-bit Unix-like kernel for x86-64 PCs from which I can learn. Back then, Linus had Minix.

Lastly, I just don't know what am I struggling with? If osdev is hard, then why is it hard? How did people in the past and in the present made it simpler and easier? The end goal is obviously to run bash (or a shell) and to get the command prompt printed. Then the next goal is obviously to run the userspace programs from shell - I don't know - by porting them to my command-line OS. Like ls, grep, vim, gcc. Then I will have a "command-line OS". And it all begins from creating the kernel first. From scratch. And I always get stuck here as I have mentioned above.

Sorry for the long post. It is my burning desire and passion that made me to ask this question. I also could not found resources on how to create a "64-bit" Unix-like kernel for x86-64 PCs ... and "how to eventually run bash"! A rough roadmap would have been nice!

[nullplan]

How have you managed to write any drivers without a memory manager? That is like the first thing you need, because you need to allocate memory.

To write a UNIX-like kernel is actually pretty simple (that's why they are everywhere). You have to manage your virtual memory so the kernel is in the upper half (but that is just paging). Then you also need a VFS and file system drivers, of course. The UNIX VFS puts everything into a unified file system tree, but every directory might be the gateway to another file system, called a "mount point". It seems most UNIX kernels don't manage their mount points very well. On Linux, it is just a linked list. I don't think it was ever intended to be very big.

Part of that VFS is also device nodes. The VFS can contain special files, called "block special files" and "character special files" respectively, and they allow access to various devices, in particular the terminal. Applications only write stuff to the terminal file, and your OS has to make it appear on screen.

Anyway, once you have that, the only thing left is a syscall layer that contains the standard UNIX system calls like open(), read(), write(), and close(). Then you can port something like newlib to your system and then you can run bash.

[iansjack]

I think (and I'm unsure if it is the "exact" issue) that I don't know about the Unix kernel design and architecture. I think reading books on OS concepts and on the design of Unix OS first is just too much theoretical. Every time, I give up. I prefer learning by doing and learn as you go. I believe in hacking.

So how does your experience so far back up that approach?

There’s no point in trying to invent it all by yourself. Read some of the books on OS design to see how others have approached the subject. Have you read Tannenbaum’s Minix book? That will show you the basic components you need to implement in a relatively simple manner.

[rdos]

You need to implement things in the right order. There is no sense in writing an NVMe driver if you don't have at least a model of your VFS design. Otherwise, you will likely need to rewrite it later because it doesn't fit with your VFS.

Order to implement things:
1. Exception handling
2. Physical memory handling
3. Virtual memory handling
4. Design a device driver model
5. Threads and task switching
6. Write some basic device drivers (not disc related)
7. Write a kernel debugger
8. Design a VFS
9. Implement disc drivers
10. Implement file systems
11. Design your user space environment, including how syscals are implemented
12. Implement user space, including porting libc
13. Implement or port a user space debugger
14. Port bash or similar
15. Write or port a graphics engine

[eekee]

This sounds a lot like what I struggle with. There's a lot to learn, especially technical terms. There are some shortcuts, especially if you're prepared to limit your OS or make it a 'retro' design, but even then, there's quite a lot to learn if you don't want it to be awful. You just have to give yourself time to learn and practice. Some OS devs make an awful OS, then rewrite it better with what they've learned, then rewrite it better again. Some around here are on their 3rd rewrite but I've seen 5 or more. As you learn, you'll make some parts better than others. That's all part of the process. (Edit: Just seen 8 rewrites in 6 years for one OS. I knew I was lowballing it with 5.)

How have you managed to write any drivers without a memory manager? That is like the first thing you need, because you need to allocate memory.

Many OSs got by with fixed allocations until relatively recently. Plan 9 still has some, though it has kmalloc too.

But that's the only thing I disagree with in @nullplan's post. Unix is a remarkably simple design for its power with a small number of parts, though it could stand to lose the whole TTY lark and some other parts if you don't need compatibility. Its special files (dev nodes) map human-readable names to internal numbers, and they do so in the filesystem rather than in the kernel. But Thompson and Ritchie didn't see the need to implement that in their second OS, Plan 9.

You might want to study the source code of Plan 9, it's even smaller and simpler than Unix systems. They say 90% of Plan 9's code is for compatibility with external standards (such as TCP/IP) It has some disadvantages, no mmap() and an overuse of private namespaces, but it's got a lot going for it too.

You can build a really simple OS if you accept some pretty tight limitations. The RTOS field has become a place for people to make very simple OSs indeed. OSs supporting a single language, where the kernel is the interpreter, can be pretty simple too, especially Forth OSs.

If you can get your head around the Forth language, you can write an incredibly simple kernel which boots to an interpreter from which you can do anything! I say "interpreter", but really it's a sort of compiler, generating some fairly fast code. "Words" (functions) can read the input line after them so it's very easy to make command-line utilities. Never mind a VFS, you don't even need a filesystem; Forth traditionally divided the disk into numbered 1KB blocks with a text editor in a few KB of code. It's normal to have a built-in assembler in a Forth for writing those functions which need to be fast, and you can write an assembly-language definition as easily as a Forth one. It's also normal to have a hexdump and a decompiler, even on small systems such as 8-bit computers. Memory access is traditionally linear, on amd64 you could set up paging as one linear space. This isn't so good once you start dealing with a lot of temporary data, but there's at least one malloc-style allocator in the public domain. There's a whole lot of Forth code in the public domain.

On the other hand, Forth is... Forth. I have to use my fingers to keep track of stack elements. I recently tried using local variables extensively to make better use of all the registers in modern CPUs, but found it obscured the difference between code and data. It's not the best-performing language on modern CPUs as they take a performance hit if the stack size changes during a subroutine (function) call. And you may want a direct-threaded Forth for performance on modern CPUs (Pygmy Forth, eForth, hForth). The "double-indirect threaded" type of Forth (such as the popular tutorial Jonesforth) takes another performance hit by calling all subroutines via pointers.

Almost forgot, @rdos's list covers everything, but you're not likely to design parts well on your first attempt, unless perhaps you've studied other OSs.

Edit: Forgot to mention xv6 as a simple example of a Unix, even having support from former Unix @ Bell Labs coders. It's for RISC-V now, but early development was for x86. Though they might have moved because modern x86 is crazy-complex. Also, being a traditional early-type Unix, it probably doesn't and won't have mmap().

[sandras]

It's not the best-performing language on modern CPUs as they take a performance hit if the stack size changes during a subroutine (function) call.

Can you elaborate on that?

[eekee]

I'm afraid I can't add much, it was told to me years ago by someone much smarter than myself. If I remember right, it's something to do with the optimizations modern CPUs apply, but I might have imagined that. Guessing a bit, I think it would apply to any stack you could make with pre/post-in/de-crement register addressing. If it only applies to 1 stack, it wouldn't be bad as that stack could be the return stack which is almost always balanced in Forth, but I think it would be any stack on architectures like ARM where using a specific register is only a convention.

[sandras]

I can't say I understand. I'll let it go.

I do tend to worry about my Forth not being a good match for the hardware it runs on, though. What I think I do understand is that my indirect-threaded model is not good for branch prediction. On the other hand, at least I worked to separate code and data, so the caches stay clean. Speaking of caches, the entire progam should fit in a modern processor's L1 cache.

I tell myself the performance isn't really a problem. It isn't. And if it becomes a problem, then I can address it. It's just that the program doesn't map well to some of the hardware features, which kinda bugs me. : )

Anyway, I hope to not hijack the topic. To reply to the original post, I will second eekee, in that you might be well off looking at xv6. Also, the What Order Should I Make Things In? wiki page might come in handy.