OSDev.org

Hello there,

I am an intermediate programmer, and I wish to learn about the more complex parts of an OS. Initially, I thought about trying to make a small OS, but the wiki doesn't recommend doing so before having obtained a lot of experience in programming. I'm not a beginner, but I'm far from writting drivers and kernel modules etc on the fly.

So my question is, what projects could I pick on to get good enough, if not in the best shape possible, to reach the stage where I can make my own OS.

I thought about it, and things like making drivers came to mind. Making system utilities, reimplementing existing tools... any suggestion are welcome.

I feel that I can do these kind of things in C and eventually get better at programming. But in assembly, what could I make that would have some kind of purpose ?

Nowadays, there's programs for everything and I feel like I cannot really make something useful that would not already be available.

What kind of projects do you recommend ? Ideally, things with a real purpose (no game dev), be it functional or educational.

Thanks you.

You may write a simple compiler. It will teach you assembly and how the target CPU works at the architectural level, object/executable files and relocation. This will prepare you for OS dev.

If you write a C compiler in C, it will teach you C (if you think you already know C, the exercise will likely prove you wrong), which is a good thing too. I find writing correct C code (in terms of the language, not business logic) from the get-go quite valuable. I don't need to spend time debugging stupid C-specific bugs, which are especially tough in OS dev, where debugging is severely limited (because of lack of tools and unstable kernel).

Note that this may distract you from OS dev for a couple of years easily.

I'm doing it in the reverse order, though. I got into OS dev (and such) long before compiler dev. Knowing assembly and the CPU helps with compilers too. IOW, it works both ways.

Thanks for you answer, it's a very good idea, hadn't thought of it. However, do you know some up to date resources I could use ? It seems most books on the subjects are more than a decade old.

While I'm at it, what kind of resources would be essential ? Language theory of course, probably Intel software books, probably a C book since I probably don't even know about half its syntax...

Are you making a linker as well, or are you outsourcing to ld ? I guess I should focus on just a simple compiler ?

If you want to write a compiler I suggest using a very simple grammar and writing a basic PEG (recursive descent) parser so that you don't have to bother with the more theoretical aspects of compiler dev.

You could even opt against linking and outputting binaries and just execute the code JIT by mmap()ing it with executable permissions and jumping to it. Or you could just output assembly file that you feed into GAS or some other assembler.

This will certainly teach you assembly language. If you know how assembly works jumping to OS dev shouldn't be too hard if you're able to understand the resources on the wiki and the basic parts of the Intel SDM. Other than knowing assembly, the most important part of kernel/driver dev is getting used to reading technical specifications.

orion40 wrote:However, do you know some up to date resources I could use ? It seems most books on the subjects are more than a decade old.

You don't need much if anything modern for a simple compiler. The 80386 was released in 1986. The first two versions of the C standard were released in 1989 and 1999 respectively. The ELF format (on UNIX/Linux) supports x86 since 1999. The PE format (on Windows) exists since 1993. The latest CPU documentation and compiler books and papers are needed for complex optimizing compilers.

About the only up to date documentation you need is for the OS system calls. And even those probably haven't changed much if we consider the most basic ones for file/console I/O, memory management, starting/terminating a process. For example, my Smaller C compiler supports Windows from XP (released in 2001) and up using the same subset of about 30 functions exported by kernel32.dll.

orion40 wrote:While I'm at it, what kind of resources would be essential ? Language theory of course, probably Intel software books, probably a C book since I probably don't even know about half its syntax...

Right. The Smaller C project on github lists a few useful ones in the readme. The OS dev wiki should give more links.

orion40 wrote:Are you making a linker as well, or are you outsourcing to ld ? I guess I should focus on just a simple compiler ?

I wrote everything except two things: the assembler (NASM/YASM or FASM is needed) and the preprocessor (I found ucpp and minimally adapted it).

Hi,

orion40 wrote:I am an intermediate programmer, and I wish to learn about the more complex parts of an OS. Initially, I thought about trying to make a small OS, but the wiki doesn't recommend doing so before having obtained a lot of experience in programming. I'm not a beginner, but I'm far from writting drivers and kernel modules etc on the fly.

The best way to learn about (e.g.) boot loaders is to read some tutorials and look at a few very different boot loaders and then write a boot loader, the best way to learn about (e.g.) the CPU's MMU is to read the CPU's manuals and then write code that uses the CPU's MMU, the best way to learn about (e.g.) scheduling is to read about scheduling and look at a few very different schedulers and then write a scheduler, ....

The wiki recommend having "experience in programming"; but what it really means is being familiar with whatever tools you're planning to use and having an understanding of basic concepts like pointers, memory management, boolean logic, algorithms, data structures, etc. Everything else is probably just a waste of time; and everything I mentioned you can learn while writing an OS anyway.

orion40 wrote:So my question is, what projects could I pick on to get good enough, if not in the best shape possible, to reach the stage where I can make my own OS.

To get good enough to write an OS; I'd write an OS.


Cheers,

Brendan

Thanks you all for your answers so far.

I agree that the best way to learn would be to actually make an OS, but I like to idea of making a small compiler first. I looked at some documentations online and it's clear to me that I don't know C enough, and I'm not even talking about assembly.

So I believe for now sticking on that might be a better idea.

I have another question then, where to start with a compiler ? I might be aiming for too big, as I want to try to make the whole thing from scratch, assembler, linker, compiler, preprocessor.

Of course I don't aim to rival nasm or gcc, they do a perfectly good job. My goal is to learn. How far should I go ? Should I just stick with "just" a compiler ? Is trying to make most of the toolchain myself realistic ?

Thanks again.

The easiest part of a compiler is probably the parser. That gives you plenty of room to learn C and play with a bunch of data structures.

Another good project for learning the language might be a game. It's a good contrast with a compiler- interactive instead of batch style, and thus uses a different set of OS-provided APIs and a different set of tradeoffs.

A once wrote a self-hosting JIT compiler (including a linker and an assembler) plus a kernel in that language that was capable to run a shell and execute programs from disk. Its definitely doable.

I did have some experience in OS and compiler dev so in your position I'd probably aim for a smaller goal (e.g. a working compiler that uses an existing linker/assembler) first.

Okay, so I think I'm going to stick with "just" a compiler for now. I've read some documentation on compilers in general, and it seems like it will be challenging enough for now. Any idea how long this will take ? There's the whole analysis process, then converting to pseudo machine code, and then actual machine code. Having an idea of how long this will take might motivate me further.

Depending on how ambitious the language is you want to translate, it doesnt have to take that much time. I build a compiler for a semester course, and even with a number of more complex topics (register allocation, intermediate language with optimizations, and polymorphic functions) that only took about 4 months. Have to mention that I already had some experience before starting, but even so, with less ambitious goals, a compiler for a simple language is perfectly doable in 3 months of dedicated evenings of work.

One bit of helpful advise, split the task into more manageable pieces. Making interfaces such that you can run just the lexer/parser, or just everything up to type checking can give you early feedback on how well those parts are functioning, and also give a feeling of accomplishment before the entire thing is done.

One option - which I realize might rub many the wrong way due to the language involved - is to follow the alternate compiler development process outlined in "An Incremental Approach to Compiler Construction" and covered in detail by Michaux in the "Scheme from Scratch" blog posts. In these, the (read) procedure, which comes most Scheme implementations and automatically tokenizes the text into Lisp atoms and lists, is leveraged to bypass the need for extensive lexical analysis, while the simple structure of Lisp s-expressions eliminates most of the need for significant amounts of parsing. This allows the developer to focus on code generation, which for an aspiring OS developer is the more pertinent aspect of compiler dev anyway.

This isn't to say that you won't need or want to learn the details of lexical analysis and parsing, but it does let the subject be put off in a way that would really be feasible with a language featuring a more conventional syntax. It's a good way to jump into the topic quickly, if you don't mind using Lisp for a while.

BTW, we really need help rebuilding the Compilerdev site after it crashed last month. The failure of both the server and the backups means that the almost all of the work done towards it up to now was lost. We aren't even sure if the wiki will be set up again, as right now the admin is hesitant to do that without any existing material. If anyone has anything to add to that forum, please do.

I don't really mind Lisp, but the problem would be that I'd need to learn it, and I don't think you can make an OS in Lisp. The main goal here is to be better in general with C/ASM so that I can work on OS dev without too much troubles that would be due to a lack of experience in programming. Maybe one day, but not today.

I think I'll aim for C, as it seem much easier to make a compiler for it than most other languages. I've not really started yet, beside some reading, and already I learn about parts of it I either never saw anywhere (volatile), or never really understood (extern). What is you guys opinion on that ?

A simple lisp-like interpreter or compiler can be far simpler than a C one. Learning enough lisp to do that is also probably simple enough to do in an hour or so- just expressions, `let`, functions, and maybe `lambda`.

The point of this is to get more familiar with C, though, so reading about `volatile` and `extern` is going to be valuable regardless.

I would suggest against trying to compile C, or even any reasonable subset of C. Because of it's design heritage, C has some major gotchas in the lexing/parsing department (especially around user defined types) that you probably dont want to try your hand on for a first compiler. Besides that, the kind of familiarity one gets with a language when building a compiler for it is quite different than one gets (and needs) for actually building projects in it. You get a really good idea of all the edgecases that a language supports, but won't learn as much on proper design of a larger software project, and good habits around building maintainable code in it, both of which are key for getting into something as complex osdev in the long run.

If you want to become more familiar with C through this project, the best option probably would be to program the compiler itself in C. Do note that this makes your job harder, as you will be missing any object oriented tools that can be very useful, but is perfectly doable. For what language to compile, my suggestion would be to go for something simple. There are various languages designed specifically for teaching compiler construction (for the project I did for university, the goal was to compile SPL, a language defined somewhat during the coarse, see the code.pdf target produced by this repo: https://github.com/davidv1992/SPLCompiler), such languages can also be found in instructional books on the topic of compiler construction, such as A.W. Appel's Modern Compiler Implementation in C or it's variants. It is also an option to construct your own simple language by choosing a number of language features you find interesting.

As for learning assembly during a project like this, a good option is to try to get your compiler to the point where it outputs assembly for an architecture you are interested in. This will teach you a lot about the assembly language for a platform, as well as the broader ecosystem surrounding it.