On the practicality of OSdev

[Schol-R-LEA]

In a recent language thread, Solar brought up the matter of practicality:

That last point is what would keep me away from it. I see my hobby projects as much as a chance to improve my business skill portfolio as it is about fun.

I think that this is frankly a foolish position, and I suspect a disingenuous one. Independent OS development for anything other than Linux or *BSD won't help your portfolio, and in fact will probably hurt it; hobbyist OS-devers tend to be seen as cranks. At the very least, it would be unwise to mention it until you had a working system to show someone.

As for doing this for anything other than a hobby or academic study, well, let's be honest with ourselves for a moment, shall we? Operating systems design has been stuck in the VMS/Unix co-dominion for nearly 30 years now, and even the major commercial OSes - Windows and MacOS - are now split between those two camps, internally. Even the vast majority of OS dev projects that get beyond the DOS stage fall into one of those designs - usually the Unixoid one - often without the dev'er even realizing it. The field has frozen solid, and nothing short of a radical shift in design philosophy will change that.

This is because, in the end, the OS kernel and internal structure aren't what matter to the users, or even for the most part to the application developers; it's the UI and applications that matter. This has been amply demonstrated by MacOS X: if they weren't told about it, most users wouldn't have any idea that the underlying system is radically different from the previous versions. The Windows kernel has more quietly undergoes an equally drastic shift, dropping the older DOS/UI design that was underlying Windows 9x in favor of the VMS-style NT kernel. Linspire has shown that a Linux system can be made to seem virtually identical to Windows. Even the APIs have converged: ll the major systems are now POSIX-compliant (not that that means much), and there are portability libraries for windowing and othe rpreviously system-dependent things. Many - perhaps the majority - of applications have versions or equivalents on all of them. The only place where the underlying OS matters is in the system stability, and even there, the average users is willing to put up with an astonishing amount of crud and poor performance and inconvenience out of sheer inertia.

In fact, the only thing I see on the horizon that is likely to change things is a rather astonishing one, a resurgence of the IBM VM/370 system of late 1970s. The 'exokernel' designs just a reframing of this idea, as are 'virtualizers' such as Xen and VMware. These are becoming mainstream again as both Microsoft and Apple plan to release virtualized systems, intended to allow users to run competing systems 'inside' their own system. Linux distro makers are similiarly eyeing bundling a Xen based setup as a way of easing the the shift away from Windows. The effect is to reduce them to just UIs and applications. If Windows, MacOS and Linux become nothing more than the logical equivalent of a CMS user session, why have them be full operating systems at all? Eventually, if things go this way, all the differences between OSes will reduce down to a choice of window manager.

Mind you, I do think that there is room for improvement that goes beyond the superficial. The current designs are pretty lousy, to be honest, and even in the past better designs existed. However, I also recognize that the best designs rarely win, partly because there's no one yardstick for 'best', and partly because the economics of the field actually favor a certain amount of inefficiency.

So then why am I still involved in OS development, if I feel it is all (literally) academic? Well, to start with, I haven't been doing any for about a year, as I'm sure many of you have noticed; occupied with other things and fed up with my own lack of progress, I decided to set OS work aside for a while, figuring that I might come back to it eventually when I felt inspired again. Partly, I work in it because I do have some hope that things will eventually change; such miracles can happen in a mature field, if only rarely. But mostly, I do it because the field fascinates me, hold my interest in a way that has nothing to do with anything practical.

I do it because I love it. And that, in the end, is the definition of 'amateur'.

So don't be too afraid of impracticality, people. Oh, pay attention to efficiency, sure, and be aware of what tools are available; but still feel free to dream. After all, if we were practical people, we wouldn't be here.

[Schol-R-LEA]

That having been said (in a posting now too long to amend), I would say that most OS-devers would be better off writing a simple C/assembly language kernel first before contemplating a design involving a radically new language and design approach. If you can't master the existing systems, you won't have the skills to go beyond them.

I have heard it claimed that in traditional Chinese art, a student is expected to master fifteen different classical styles before undertaking any new work of their own. Similarly, architecture students today are expected to produce designs in several existing styles, and understand the strengths and weaknesses of each, before even going out into the field; Ayn Rand's egregious notions aside, such practice is necessary. Without knowing what has gone before, and why it was done that way, you cannot expect to do something new. Even the most radical of visionaries, the Bucky Fullers and the Nicolai Teslas and the Ted Nelsons, cannot escape this; when they try, they tend to get flattened down, beaten into the dust, and only later when someone like Brand or Steinmetz or Berners-Lee comes along and finds a way to sneak theier ideas into the mainstream (often losing important aspects along the way, admittedly) do their visions triumph - if at all. Even those rare visionaries who do know the past - da Vinci or Newton or Darwin or Einstein - often wait years to be vindicated, because it is very, very hard to incorporate something new into a field which is already stable, especially since there are always those who resent change.

[Crazed123]

I agree with you that current systems we can amply improve beyond the Unix and VMS designs which dominate today. However, I feel that if we all take the time to code a Unix-workalike kernel before setting out on our own projects, the field will have advanced of its own by the time we struggle out of the mire of the past ourselves.

We should learn the designs of past works, but we should not actually waste our time *recoding* them. After all, each of us is only one man.

[Combuster]

Ah philosophy. hehe.

Considering the chinese statement, i'd leave the following to think about. If we are to try to define laws behind things, we take into account our past knowledge and use it to understand what we see. The most radical ideas are generally come from those with a creative mind and little influence. Kids are generally more creative than adults, because adults are spoilt with the ideas fed upon them. If you ever watched the matrix you probably know the scene where morpheus excuses himself for freeing neo due to the age constraint, because of minds having trouble letting go. Basically, people have difficulty getting over what they were taught and take on something radical. Copernican revolution all over. In the end, i think that if you have people study all aspects of the field, their mind will end up troubled and all innovations are done from the point of view they have been put on. Back to China: Communism, Censor, Unidirectional thinking, nobody wanting to have their system thrown over by modern art.

In osdev context, just read the design thread and see how many people choose the microkernel road. Having people do unix, then someone else's kernel, then designing from scratch they probably figure that monolithics are easier, microkernels are more tastesensitive and all others are kindof taboo. If we force people to go through the standard route, they wont know better than to stick to it. The better inventions are still those "why didnt I think of that?", and the most innovative kernels will probably be by those who are the most open minded.

C and Assembler. Common stuff, same story. Still nobody's going to keep me from using freebasic as my main driver language. If i'd hang around a bit longer i might have been kicked off that idea as well. People just have to know what can and can not be done by their favorite language. even those miles away from C can write a decent os if they wanted to. (same for asm but as there arent any compilers that can do lgdt natively you'll be stuck at the bootstrap problem. Revolution anyone?)

Oh and, unlike that sig VB does allow for pretty decent and proper coding. I could for sarcasm say the same about c for not being readable and all that, but its a matter of taste. fact is, both are decent and its bad to kick off one idea for the other.

That said, there are people who lack the ingeniuity and will have to live by other people's design or get lost.

Now, go ahead hammering this

[Schol-R-LEA]

sigh I actually do agree, since that's my own view; revolutionary designs do tend to come from outside of the original context. If I weren't a dreamer myself, would I have spent so many years supporting Project Xanadu? But I've seen a lot of unfinished revolutions come and go, or worse, turn sour and become worse than what they were overthrowing, and sometimes my cynicism and weariness get the better of me.

The problem is, the original context is still there, and revolutionary ideas alone rarely are enough to change things. To change things, you need to know what it is you are trying to change. A lot of times, what sounds like a brilliant new idea is actually a much older one that the current ones actually supplanted.

Perhaps I should have just said that one should understand how the older systems work, first. I never meant that one should commit to a full OS in the monolithic Unix style, just a simple toy kernel that can switch tasks and manage memory. I also didn't necessary mean C - if you have another suitbale systems language, go ahead with that - though as pointed out elsewhere, most of the OS related texts are oriented towards C, so at least being able to read C code is pretty much necessary even if you despise the language.

[distantvoices]

It is not as if you're regarded as some kinda crackpot smoking all sorta weed if you're admitting to be osdever - especially with a working system: you're regarded as overqualified (that lad knows secrets we don't and we can't give him appropriate work so he won't grow bored nor do we want to pay him an appropriate wage)

That put aside, I enjoy all the experience gathered whilst working on my 08/15 microkernel os project. I've learned so much I could never have learned in any os course at university. Simply put: I enjoy doing osdev. That analysis & design have become like second nature to me is a nice effect of such a hobby I reckon.

It's like with all other things one is doing as a grown-up: you alone have the choice. you alone are to give sense to what you are doing.

[Candy]

Mind you, I do think that there is room for improvement that goes beyond the superficial. The current designs are pretty lousy, to be honest, and even in the past better designs existed. However, I also recognize that the best designs rarely win, partly because there's no one yardstick for 'best', and partly because the economics of the field actually favor a certain amount of inefficiency.

I'm working on it. Got a field test of my idea in a very very limited form to what I'm working on in my OS and it's surprised me in its versatility and ease of use. Considering that it's limited in about 4-5 ways in what it can do, I'm stunned. Been dreaming about what it could do for two weeks now, got the idea condensed enough to implement it in C++ now, except for one design problem related to use of templatized interfaces being instantiated with an inheritance tree. I think that'll take about 2 days and then it'll be done (the design). Implementation can take up to 2010 or 2015, if it's actually what I expect it to do it's good.

[Solar]

In a recent language thread, Solar brought up the matter of practicality:

That last point is what would keep me away from it. I see my hobby projects as much as a chance to improve my business skill portfolio as it is about fun.

I think that this is frankly a foolish position, and I suspect a disingenuous one. Independent OS development for anything other than Linux or *BSD won't help your portfolio, and in fact will probably hurt it; hobbyist OS-devers tend to be seen as cranks. At the very least, it would be unwise to mention it until you had a working system to show someone.

I won't go into the philosophical rest, but I have to set aright a misperception of what I tried to say.

When I speak about "extending my skill portfolio", I don't mean mentioning my work on Pro-POS prominently anywhere in my resume (if at all). It might be different with PDCLib once that's released, but that is not the point.

Since I started working with OS and C lib level code, I learned stuff about IPC, shared memory, locale and wide-character support, floating-point representations, details of the printf() formatting and loads of other stuff that's usually considered too "obscure" by app developers to really learn. Having tinkered with this stuff made me a much better C/C++ developer.

That's what I meant with "extending my skill portfolio". Today, I can look a prospective employer in the eye and tell him that I know the nitty-gritty of it.

[Schol-R-LEA]

Fair enough, Solar, fair enough. Certainly, having the experience in the subject is a worthy thing in and of itself.

[Brendan]

Hi,

The largest hurdle for OS development is that being practical itself is not practical.

The statement above doesn't seem to make much sense when you first see it, so let me share my experiences in this field...

I began OS developement party out of disgust with the operating systems available to me at the time (DOS/Win 3.11), and party out of natural curiousity. It didn't take long until I was fascinated.

I started without any of the knowledge necessary to actually acheive anything. After several rather dumb attempts at creating something (and after the internet became common, and the availability of resources that came with it), my initial stumbling seemed to settle into a "grand plan".

The basis of this plan revolves around a fully asynchronous (peer-to-peer) distributed OS based on message passing, to be implemented on top of a (slightly largish) micro-kernel. It's not really a new or original idea, but an idea I still consider "good".

Around 8 years ago I had a working prototype. It used 32 bit paging, supported single CPU only, wasn't portable and didn't have any of the bells & whistles, but it worked and did what it could efficiently. The next step was to take this "toy" and turn it into a usable, practical OS. This is where the problems begin.

The prototype lacked support for PAE (and 64 bit/long mode, which came much later), didn't support SMP (or NUMA, which also came later) and didn't support swap space or memory mapped files. There was no device detection, only a few simple/generic device drivers, no support for internationalization (Unicode, time zones, etc). No real support for video, sound, printers, scanners, etc. Nothing to handle OS installation or configuration, no useful applications, etc.

It took me around 18 months to write the prototype. So far I've spent most of the last 8 years working on a "practical" version of it, with little to show for it. Once you move out of the "toy" stage and start caring about decent hardware support, stability, features, etc it's an entirely different thing - it's like the difference between building a toy car and building a real, commercially viable car. Trying to pay your bills while also trying to make progress on the OS and do everything correctly to avoid the inevitable ("Doh, forgot to allow for that") rewrite is extremely hard.

For the last 6 months or so I've been playing computer games and experimenting with a C/SDL/OpenGL 3D MMORPG. I still haven't given up though - the "grand plan" lives on, even though it's hibernating for the time being.


Cheers,

Brendan

[paulbarker]

I've been thinking of a couple of things over the past few days which are relevant to this. It comes down to the question of what OS-dev really is, and I think its often seen as being "kernel dev" if you understand me.

The point is - theres more to the OS than the kernel. I realised that in terms of kernel I have very few (if any) innovative ideas (which is a message passing microkernel). The only real non-standard idea I have there is to implement the filesystem in kernel so that efficint caching can be done, and treating all disk files as block devices (the standard library can put the nice char-oriented face on them, the kernel just sees a file as a number of blocks).

I am more interested in the user-mode side of things, and how to adapt the unix philosophy to the 21st century. Argument passing can be improved greatly, and autoconf can hopefully be obliterated. Also, working on things like the standard libraries is good practice at designing and implementing algorithms. So my plan is to virtualize my system over the top of something like linux, until I get a working microkernel (either by writing my own or finding one I like). All I need to virtualize is send, recv and sendrecv functions, which can be implemented using unix sockets.

Getting back to my original point, I agree that there is very little innovation in kernel designs and that the easiest method of osdev is to follow the well travelled path. But there are a lot of good ideas out there (clicker32 has a fair few, although I don't really understand most of the implementation). Look beyond the unix/vms design patterns to ideas like treating everything as an object (or message target, or device or whatever other than a file or stream) and distributed systems and you will see the ideas of the future.

My personal "Grand Plan" is to create a system I will enjoy using and programming. Thats hard to do when you look at the pile of (expletive deleted) that we call the x86 architecture. Thats a subject for my electronics degree though (off to uni in less than 2 months now!).