Top -> Down OS Development

[SoulofDeity]

So, I've been reading up on history a bit and recently came to the conclusion that writing an operating system from the bottom up is actually a bad idea. One of the first things each successful operating system had was an editor suitable for development, an assembler and/or compiler and debugger, and basic utilities for things like file management. eg. The Hurd still isn't in a useable state and Windows was riding on top of DOS until the NT kernel came out. DOS was also no exception to this, having been developed in assembly using a proprietary toolchain with a debugger running on CP/M; which in turn was written in custom language called PL/M running on TOPS-10 written in a custom language called MACRO-10 and BLISS which evolved from PDP-6 monitor software; where PDP-1 which among other things also included a text editor and debugger. The PDP-1 was based on TX-0 which (wouldn't you know it) also had text editors and debuggers. The TX-0 was a transister-ized version of the Whirlwind hardware which finally breaks this pattern, only because at that point the hardware was using vacuum tubes and programming involved flipping switches.

So, that mouthful out of the way, I think it's plain to see that the ladder of success has always been built on top of something else; and it's a regular pattern the development tools and utilities almost always exist before the kernel. The benefit I can see here is by the time you get around to actually writing the kernel, you already have a clear grasp of what functionality needs to be implemented to bootstrap your system.

As much as I prefer to hug the hardware, I've decided to work my way from the top down as well, writing all the basic utilities for my operating system before spending any time on the kernel itself.


So, what do you think, insightful or ignorant? Is there anything in particular that you think you could've done better in your own projects by following this idea?

[Muazzam]

It is not a good idea to start with writing programs and tools before kernel unless you're writing a clone or extending an existing operating system. For example hurd was clone and windows was to extend functionality of DOS. I can not consider my operating system completely from ground up. I had written Kernel (not (still) a real kernel, mono tasking), text editor and instead of writing an assembler I had ported. That is why I can not consider completely from ground up.

[alexfru]

I don't think that iterative development and reuse of hardware and software is necessarily a bad idea when there aren't that many alternatives available, especially on and for new platforms. You have to begin somewhere. Sometimes while others are still waiting for someone else to make the first step and create what they need for their ideal project (and they may need faster CPUs, larger and cheaper storage, better compilers, whatever), you become the pioneer, you create that something or you find a way to do the same kind of project (perhaps, less ideal) with the existing resources. This can get you an advantage, despite your project being imperfect. The right combination of features, price and timing can do wonders. You leverage what you have and what you know.

And what you know doesn't always include exact predictions of when and how technology will improve. IOW, you may design something nice for today, but tomorrow you'll look back and see how that something that once was a thing of art and beauty is unfit for the modern-day world.

That is not to say that all top to bottom designs are doomed to fail or necessarily suffer from enormous bloat resulting in from the mismatch between what has been anticipated and what has actually happened.

But there are some natural reasons why a lot of stuff happens to be done bottom-up.

[willedwards]

Different people make hobby OS for different motivations and want to get different things out of it.

You may have some strong technical vision about how an OS and/or apps should work, and set out to prove it to yourself.

For example, AtheOS was an excellent system with strong GUI and "massive multi-threading". He ported some stuff like bash and even webkit browser, but it would be disingenuous to say Kurt wasn't building the whole system from scratch in a true sense. Kurt got to the point where enthusiastic onlookers like myself actually tried to use it as a main OS

You may want to learn, and you may want to stick closely to a guide like Unix or Minix.

You may actually want to dabble in assembler. You can build a whole OS in assembly! MenuetOS.

Plan9, Obereon, Mosix, I can chuck in lots of names you can google to get technical inspiration about new ways of doing things. For example, I work on the Mill CPU which enables cheap peer protection transitions without thread switching... what does this mean for conventional OS? What would `ps` and `kill` show? Etc.

What other motivations are there?

[SoulofDeity]

True, there are many things better designed bottom up, but I don't think an operating system is one of them. With bottom up design, you lock yourself to a platform and it takes far more time to get any significant results. Just poke around the forums a bit and you'll see most os screenshots are very simplistic cli's or text/graphics glitches. People who've gotten to the loading stages have very primitive and tacky software (no offense). If you develop your utilities before hand, you can test them thoroughly and know they're tried and true. Then making a kernel bootstraps your entire system at once. You are finally independent and have all the time in the world to improve things or write new software.

[Combuster]

With top-down, where you build everything for the precursor OS first, you are equally liable for locking yourself in to the architecture of that platform and not making any real difference. There's no one true way in going about this.

[alexfru]

True, there are many things better designed bottom up, but I don't think an operating system is one of them. With bottom up design, you lock yourself to a platform and it takes far more time to get any significant results.

Is Linux locked to x86? Or Windows? There have existed and there exist ports to other platforms.

Just poke around the forums a bit and you'll see most os screenshots are very simplistic cli's or text/graphics glitches. People who've gotten to the loading stages have very primitive and tacky software (no offense). If you develop your utilities before hand, you can test them thoroughly and know they're tried and true. Then making a kernel bootstraps your entire system at once. You are finally independent and have all the time in the world to improve things or write new software.

This is exactly what I have advocated a while back (10 years ago, in fact) in a small article on hobby OS dev: OS Development Skeptically, specifically section Where to Start and Where not to.

But I wouldn't say that this is a top to bottom approach or a big part of it. I see it as a practical way of doing things. You create and polish building blocks, low-level and high-level (e.g. drivers or simple applications, doesn't matter), then you put them together and work on and around the seams. It may also be a good setting for experimentation. But it doesn't prevent doing things bottom-up (I'd say it naturally suggests it; remember, those are building blocks?) nor favors top-bottom (your building blocks can still be crappy and poorly thought out on their own or in the context of there being other blocks or both and therefore in need of improvement).

Perhaps, we should make a more clear distinction between design and implementation/development/coding? I'm afraid we might be confusing things a little.

[iansjack]

I'm not quite sure what you are saying in your post but it seems to be "you need good tools before writing an OS (or any other complicated software system)". It's difficult to argue with that.

You then appear to suggest that this means that you should write all those tools yourself before proceeding with the kernel. I can't see the logic of that. It's unlikely that I can produce a better editor, assembler, compiler, and (for those who like them) IDE than those that already exist. So why should I write a load of crappy tools rather than using those that have been well-honed and tested?

I'm not saying that people shouldn't wrie new editors, etc., if that's what interests them but as a prerequisite to writing a kernel - that's just crazy.

[Kevin]

One of the first things each successful operating system had was an editor suitable for development, an assembler and/or compiler and debugger, and basic utilities for things like file management. eg. The Hurd still isn't in a useable state

Not sure what point you want to make with this, because the Hurd completely contradicts what you're saying. GNU has an editor, assembler, compiler, debugger and more than just basic utilities. That's the successful part, almost everyone uses them with Linux. But the kernel, which is developed as the last part of the OS, never seems to get to a point where it's usable.

The benefit I can see here is by the time you get around to actually writing the kernel, you already have a clear grasp of what functionality needs to be implemented to bootstrap your system.

You probably want to test the software you're writing, so you'll probably not be writing it for your unique API. You might just end up copying the API of the host OS you use for development. And no doubt, using an existing, well documented API makes things a lot easier. On the other hand, it doesn't leave you much freedom in making your OS different and therefore interesting.

[SoulofDeity]

With top-down, where you build everything for the precursor OS first, you are equally liable for locking yourself in to the architecture of that platform and not making any real difference. There's no one true way in going about this.

This is definately true. Especially if you are relying on standard api's for everything. One repeating pattern you'll notice throughout history is that the operating systems almost always are written in a different language than the platform they are developed on.

Consider C. It was written specifically for Unix systems, and many of the system calls reflect that. There are many problems that can been seen in hindsight, but because the tool allows systems programmers to do the things they need effectively, we're naturally inclined to port libc rather than roll our own. If your kernel is tightly hugging the C language, then you're knocking off Unix.

Now, I'm not saying you have to go out and create a new language if you want to be innovative. Just put time into creating an API that suits your needs. For now, it can simply be a wrapper around libc, but when you write your kernel you can make it native.

@alexfru, when I used the word "platform", I'm talking about the environment, not the architecture.

[alexfru]

@alexfru, when I used the word "platform", I'm talking about the environment, not the architecture.

Can you elaborate on what constitutes said environment?

[SoulofDeity]

@alexfru, when I used the word "platform", I'm talking about the environment, not the architecture.

Can you elaborate on what constitutes said environment?

Generally, all of the resources you have at your disposal. A classic example of the horrors of being locked to an environment is Cygwin. It's just a giant clusterf#*@ of hacks to get Linux-specific code to work on Windows.

EDIT:

You then appear to suggest that this means that you should write all those tools yourself before proceeding with the kernel.

I'm not saying that at all. I'm saying that you should have the tools ready to work on your kernel before you actually make your kernel; regardless of whether you roll your own or are going to port open source software.

[embryo]

writing an operating system from the bottom up is actually a bad idea.

Yes. But sometime there is no choice but to use such approach. The best result can be achieved in a combination of both worlds. But combination should be very good, else the result will be far from the best.

The art of compromise and some combinatorics make the trick.

However, going bottom up you are learning the art. So, may be there is no way up without touching bottom.

More generally - it's about your existing knowledge and experience and about knowledge and experience elaboration.

[alexfru]

@alexfru, when I used the word "platform", I'm talking about the environment, not the architecture.

Can you elaborate on what constitutes said environment?

Generally, all of the resources you have at your disposal. A classic example of the horrors of being locked to an environment is Cygwin. It's just a giant clusterf#*@ of hacks to get Linux-specific code to work on Windows.

Right. It's not just the APIs, it's also the semantics that's different. Some things are pretty tough to emulate transparently and efficiently at the same time. That's why I prefer to stick to standard functionality when writing my C(++) code. Obviously, this doesn't cover everything because the language standard defines very little (no threads (until C++11 or 14), no network, no multimedia, nada). At work I rarely need to write multi-platform code, so it's less of a problem there, I can just use whatever the platform's got. Fortunately, with so many virtualization solutions and better computers we can do more and more of this stuff in a VM.

Back to the topic... If you think that the problem in "locking to a platform" is that it's unintentional, perhaps in order to avoid that one needs to intentionally "play" with a bunch of different systems and try to figure out their best and worst parts and consider that when designing another one. However, I wonder how reasonable it is to expect from hobbyists (especially noobs) to be able to do this kind of "research" work. But if you meant seasoned programmers, then there may be other issues getting in the way of better design (they may not have enough time (if it's a hobby or there are deadlines to meet), they may not have the final say (if it's a joint project, especially at work), the project may grow beyond what's easy to reason about and work with, etc). And it's never going to be ideal. For one thing, different people have different understanding of what ideal is. Perhaps it's therefore best to design differently for specific problems instead of trying to please everyone and alienating many.