OSDev.org

I've finally decided to start writing my kernel. I'm naming it Khaos, at least for the time being. It's not in all caps, the O and S do not stand for Operating System, and it's not about disorder.

As I only started it on Sunday, I don't have much to show so far: a simple (but stable) paging system and some library functions for screen output. Still, over the next few weeks I expect to have much of the kernel finished.

The design is unusual but realistic. In general, the philosophy is to be as lazy as possible at any point, and therefore end up with as much as possible left to the higher levels of the design (and out of the kernel), as long as it is actually useful to do so. That way, I can: a. test early and often b. change or combine things on the fly, and c. let multiple people work on different sections easily. The specific architecture is a bit of a mixed bag, not adhering specifically to any type of design. Although if anything, it's kind of microkernel-y.

The kernel is very simple, holding only things that are generally consistent. This means paging (and swapping, eventually), process management, scheduling, and a variant of message passing. It also has very simple execution code, only enough to load a flat binary. As it doesn't do much, it only has dynamic allocation on a page level (using an optimized bitmap scheme), which greatly simplifies it and makes the speed more consistent.

All of the I/O drivers, as well as the virtual filesystem itself, reside in a set of usermode processes. These processes have direct access to I/O ports, as well as special control over normal processes. The focus here is not reliability, like in a normal microkernel, but simply flexibility. Almost all security is implemented within these processes - there is nearly no security overhead in the kernel itself.

Message passing itself is very simple, and can only transfer a few words of data at a time. Large amounts of data are transferred asynchronously using a shared memory system, which is managed by the driver processes using special kernel calls. Using the combination of messages and shared memory, a stream-like data primitive is formed. I'm hoping this generally unbounded but still secure message passing system will be very fast.

One of the more interesting features is the C library design. Because I/O and other things are so low level, the C library is very important. Most interestingly, it has the main execution code. For an argument about why I'm doing this and how it could be done, see my posting "Exec call in userspace library" elsewhere in the forums. In a few words, it allows for completely different execution environments to be created on a single system. The library also (optionally) handles many events, such as page faults and other exceptions. Using the page fault handling and low level memory management, for example, the C library could implement dynamic loading. This makes the C library sort of like an exokernel's "LibOS", except without the need for a secure language or kernel protection.

If anyone would like to help, they are welcome to after I've gotten to a reasonable point in the kernel implementation. I'm not hosting anything yet though. The code will be under a GPL license when I officially release it. For now, a simple tarball of the current source. I can't attach a binary, sorry for whoever is using Windows.

One of the more interesting features is the C library design. Because I/O and other things are so low level, the C library is very important. Most interestingly, it has the main execution code. For an argument about why I'm doing this and how it could be done, see my posting "Exec call in userspace library" elsewhere in the forums. In a few words, it allows for completely different execution environments to be created on a single system. The library also (optionally) handles many events, such as page faults and other exceptions. Using the page fault handling and low level memory management, for example, the C library could implement dynamic loading. This makes the C library sort of like an exokernel's "LibOS", except without the need for a secure language or kernel protection.

You do realize anyone can link with a different C library don't you? will applications even be running in userland? It is insecure, if not impossible to run libc in a different ring than the application it is linked to

You should really read the thread I referred to. The library does not reside in a different ring, or have any special privileges. It has a relatively low level interface (a.k.a. mmap and some special signal handling), but can *not* affect other processes. It serves pretty much the same purpose as the C library in a normal OS, but will be a bit more complex. It's completely possible to run a program without a library, but there must be one there at first to load it.

The only issue (which is addressed in that thread) is a secure exec call for things like suid executables. In that case, a driver process is allowed to reload a new and trusted library to execute the executable.

Well, I just finished the first alpha release of my kernel. Note that this does not mean the entire system - just the kernel - so it still has no drivers, no filesystem, no libsys (the "libOS" equivalent I mentioned before), and no userspace utilities. However, it can do full multitasking and memory management with a (mostly complete) permission system, as well as very fast message passing and userspace handling of both faults and IRQs.

The scope and design of the Khaos project have changed a bit since my original post. It now will include not only the kernel, libsys, and drivers, but also a C library, C-interoperable bytecode VM, C compiler, and UNIX-esque shell utilities - essentially a full base system. None of these new things are written yet, of course, but the goal is to have a self hosting system written completely from scratch by the end of 2009. I think this is a reasonable goal considering that I got a kernel up and running in three months while doing schoolwork, plus I have some friends who will probably be helping me.

Here is my design so far:

The kernel handles memory management, tasking, and IPC. It is at addresses above 0xF8000000. It has a one shot ELF loader, which loads a copy of the libsys into memory at 0xF6000000 and above. There is only one true IPC primitive - the signal - which is a synchronous message with up to 3 pointers worth of data (12 bytes on the x86). Driver processes are allowed to map pages directly from physical memory or from other processes - this is used for DMA, mmapped I/O, and shared memory IPC.

The libsys is a set of wrapper functions for signals that emulates normal system calls like read() and write(). It also has default signal handlers, as said before. In essence, it turns the kernel's exclusively event driven interface into one more appropriate for the libc. For example, read() actually requires sending a signal, receiving another signal, and then sending a third one. Don't even ask about how it works on the driver's end; at least signals are very cheap in my design. The libsys is mapped read-only, and is mapped in every address space. When the process forks, it's memory is left alone, just like the kernel's. A table of functions at a specified address allows use without actual linking. Another function of the libsys is to execute programs - it is the only thing in the system with a true ELF loader. Once again, read that thread before saying that's impossible or insecure.

Signals are much like interrupts: when one is caught by a process, it calls a signal handler stored in a signal handler table (analogous to the IDT). Signals can be sent by other processes, or by the kernel when a fault happens in userspace. The libsys has default handlers for all faults, so it can fix them, but these can be overridden by the running program in most cases. Signals 0-16 can only be called by the kernel or a process that is more privileged than the receiving one. Signals are also preemptible - up to 60 signals can be sent at once to one process before it gets a fault.

The permission system is sort of two dimensional - each process has both a user ID and ring level. The general rule is that processes with the same user ID or a strictly lower ring level than a process have permissions over that process. Ring 0 also has direct I/O privileges, and rings 2 or less have privileges to all files. Ring 2 is usually administrator, and ring 3 is user, but there are 65536 possible rings for more complex configurations (it's 65536 because I had exactly 16 bits free in my process structure, and nothing else that would fit in 8 bits).

Drivers for I/O are run in userspace, and are responsible for all block caching, because the kernel's virtual memory space is very small (120 MB for the heap). Filesystem drivers (like ext2, fat32, etc.) run separately, and make requests to the I/O drivers. The virtual filesystem server keeps track of the filesystems, and helps to impose general structure. It's entirely possible to run multiple sets of filesystems and drivers completely independent of each other with relatively little effort; I'm not going to utilize this much in the standard base system though.

The general goals of this project are twofold. First, to make a fully functional self hosting operating system that is also very small and independent from the GNU project. I want to try and have an OS that gets rid of the bloat associated with modern *nix systems. I'm already on a good track - the kernel is less than 1000 sorted lines of C and fully functional. Second, to give the programmer and user as much control as possible, while still offering a reasonable but override-able interface. I.e., to separate mechanism from policy, but also interface from mechanism, so that you can have your worse-is-better designed system, and stay sane too.

That's what I've got as a design at this point.

Anyway, now that I've got the kernel at least appearing to work fully, I'm starting to look for some developers to help me fix bugs and write drivers and such. If anyone is interested, just email me at [url][email protected][/url]. You can look through the actual repository as well, it's just in my signature.