Leveling the Playing Field

[NeoHacker]

As of November 2009:
91.26% internet clients use Windows.
5.66% internet clients use Macintosh.
1% use Linux.
And only 1% use something else. This may be BSD, and other *nix OSes.

Thus pretty much all computer users either use Windows, Macintosh, or Linux.

But the question is - why? With such a large community here, what is wrong? Why isn't Windows getting much competition? It's more expensive and slower than alternatives, right?

The problem is that Windows, Macintosh, AND Linux are all using anti-competitive strategy.

All modern books have copyrights on them. Just because you have one book doesn't mean you can make copies and distribute it to your friends.
Operating Systems also have copyrights on them. Just because you have a CD of Windows doesn't mean you can copy it and give it to your friends.
However, in the literature world, one book does not rule them all. New books are published all the time by different authors and may sometimes sell well for some period of time.
In the computer world, extremely few OSs dominate over everyone. New usable OSs are not published and sold.

So, what happened? Why are books easily written and OSs aren't?
The reason is lack of information. OS writers don't know how to write some parts of their code!
In particular, hardware data sheets are not free to come by. Either you have to agree to some non-disclosure statement, or you have to pay a huge sum of money, or the hardware company bluntly says "No"...
These hardware companies signed a contract with Microsoft stating that they will not give out the specifications to anyone but Microsoft.
The fundamental parts of an operating system are denied to independent operating system developers. Hence most of us are usually near the "Stage 1" of the OS development, even after years of learning and practicing the art.

This scheme that Microsoft and Apple have up is called "Exclusive dealing" ..a type of "Barrier to entry". Monopolies strive on this kind of stuff because it stops people like you from reaching up to their level.

Believe it or not, Microsoft and Apple are not the only ones being anticompetitive. Although Linux is free and open source, the details inner workings of Linux are rather obscure, even for people working on it.
Ever try to read through the source code of Linux? It reminds me of trying to fix a bug in Windows by calling up and chatting for hours on the phone with Microsoft's offshored customer service, only to end up having to reinstall the whole operating system.
Linux is a real beast - without good directions and useful information, one can easily be lost in its code.
When trying to read the assembly code of a C++ compiled "Hello World" problem, most people get lost. The program is not encrypted or hidden.. it just becomes very complicated to understand.
The same applies with Linux ASM/C/C++ code. Even with all of the code visible in full sight, having no encryption or hiding schemes, Linux is just way too complicated to understand.
Because of its code complexity, understanding how Linux handles hardware is very similar to understanding how Windows or Macintosh handles hardware - It's easier to reverse engineer the hardware than to read the code.

Because of this lack of useful information for operating system development, OS programmers simply get lost or stay near the beginner level of their development.

Hence Windows, Macintosh, and Linux dominate.

In order to get around this anticompetitive system, OS developers have to be granted easy access to the fundamentals. OS developers must have a predefined structure where all hardware is accessible as an API from stage 1 of development.
This includes the simple - switching to 32 and 64 bits, communication with mouse and keyboard, printing pictures in color, reading from and writing to a DVD.
But it also includes the hard - Interfacing a broadcom BCM4312 wireless card, using any 3d acceleration parallel processing GPU tricks available on a modern video card, and interfacing with a wireless USB webcam.
Basically, all chip specific hardware control must be abstracted from the OS programmer.
Allowing for a plug and play usb joystick should be extremely trivial for the OS programmer.
These are fundamentals that should be present in every Operating System without question.
There should also be libraries that handle the basic protocols and file formats - Interfacing TCP/UDP, the file format of a JPEG, the file format of a TTF.
The OS programmer should have easy access to file formats and protocols - no thinking should be involved here.

By having such a template system from stage 1 of operating system development, the OS programmer has a level playing field. No longer does one have to spend months trying reading through old partial manuals about interfacing the USB. No longer will one have to spend days attempting to experiment with some piece of hardware to figure out how it works.
Instead the programmer will focus on that which makes operating systems stand out:
Will it be graphics or console based?
What level of internet/user security do you want to place?
What will your API be?
How user friendly will your OS be?
Should some programs be hard coded into the kernel?
How fancy should the graphics be?
What programs should come with the OS?
etc.

By not having to think about the hardware, protocols, or file formats OS programmers will easily be able to compete with Windows, Macintosh, and Linux.
Ordinary users will have a much better selection of high speed, low cost OSs. High performance independent OSs will be ranging from completely home based graphical eye candy user friendly, to high security industry standard database servers.
Probably most of these diverse operating systems will be written by people like you.

So, how to implement such an idea? It's actually not as hard as it seems.
An organization of well minded individuals should form together and work as a team to achieve the following steps:

1. Copy the Minix3 code from here.
Minix3 in particular is nice because of how it handles hardware. The amount of hardware it has is rather limited (it has drivers for all companies that show their data sheets!), however Minix3 has a whole powerful system for handling the drivers. If something goes wrong with the driver, the OS doesn't freeze.. the driver just gets reset, and you're back in business. This is really good for us because we don't care about the drivers - as long as they work.
Minix has been one of the main pioneers of Operating System development. It has pretty much everything you need in the most basic operating system - Multitasking, transferring to 32-bits. It works on a PC. It is not portable because its lack of hardware implementation of other systems. This can be added later.
Minix3 is written with a BSD license, meaning anybody can copy the code without being forced into making their code open source.
Most important: Because of how Minix is built (small kernel) it is much more easy to understand and thus modify it than any other advanced Operating System.
The goal is to make a template system for any operating system. The smaller the original kernel, the easier it will be to make it into a template.

2. Figure out how to compile it from the code and run it in QEmu or Bochs.
Make sure you can compile it from your system. You will probably have to rename some of the commands in the Makefiles to get it to work.

3. Get rid of everything *nix!
To transform this code into a usable template structure, one must first remove everything having to do with *nix from it.
A good way to start this is by removing any code that has to do with touching the directories. There should be no such thing as "/root/" or "/tmp/" or even "/"
Using the new template one should be able to write up an advanced DOS with the usual "C:\" in it.
Remove any command line things that Minix has ("ls", "mkdir", "cd", etc) ..completely get rid of the shell - remember, an OS developer can choose not to have any console interaction at all.
Note that for all this time, you must keep Minix's ability to run applications, and use any drivers. Make sure to keep any protocols or file formats as libraries. If possible, the multitasking ability should be shoved into a library that can be turned on and off. (One may want to implement a limited form of non-preemptive multitasking).

4. Reflection, documentation.
Now if everything has been done properly, there should be about three things - the kernel, the hardware drivers, and extra useful protocol libraries.
In the kernel, there should be some driver initialization present... and afterwards, nothing. (Just freeze). It is fuzzy there how much control the programmer should have over the driver initialization. It seems that all of it should be controlled.
All controls that the user has should be extremely well documented in HTML or PDF form. (Regular text is boring to read.)
In some cases, extra functions should be added as a form of an API, so that there was an abstraction of the hardware.
All of these functions should be well explained in the HTML documentation. It is assumed that the APIs can be called from the "main area" within the kernel, however this idea can be modified so that the APIs can be called elsewhere.

5. Build a proof-of-concept example operating system
Using as much of the API as possible, build a small project OS. Preferably something opposite of Linux, but not something that takes too much time - Make an extremely simple Windows. Have it be able to run simple programs that use an extremely simple system API set. (createwindow, printmessage, waitkeypress, closewindow for example) Here one would realize what parts of the API are missing in order to accommodate any type of operating system.
Please note that here "API" was used in two different forms. One is where the OS programmer uses an API, the other is when the OS programmer supplies an API for applications. Do not confuse the two.
To continue on with the project, you are free to continue developing this sister project and thus enhancing the OS API even further.

6. Acquire new hardware drivers
Minix3 starts out with a nice supply of "open" drivers. It basically has all the capabilities of the bochs and qemu emulators, including monitor and networking. However after a while, that is not going to be enough. Luckily, Minix3's driver structure is relatively similar to Linux. Linux has most of the hardware needed. If you can figure out how to convert Linux drivers into Minix3 drivers, you will solve the dreams of many programmers. So, how to do it? Look through manuals about Linux and Minix3 drivers. Bug the developers of Minix3 and Linux a LOT, so that they can explain to you exactly what is the difference between the two drivers. You may get some help from FreeBSD, as they have some experience "emulating" Linux drivers.

7. 32 and 64-bit compatibility
Recompile the code and try and get it to work in both 32 bits and 64 bits. This shouldn't be too hard since mostly everything here is written in C.

8. 16-bit compatibility, and other system portability.
Using other compiler settings, or other compilers in general, compile the C code for other systems. 16-bit would be nice because some programmers consider the 32-bits to be extra useless bloat of memory. Some hardware however would not work. It would be best not to go crazy here by switching between real and protected mode. Portability to other systems will probably require hardware support on the other systems. Again, this is where the new hardware drivers (step 6) comes into play.
It would be a very good idea to separate this OS template into separate platform codes instead of having one big code that can handle everything.


If you can accomplish these eight goals, you will have completed the task of creating an equal playing field for OS programmers. Additional APIs may be added, but must always be described in detail in HTML, accessible to everyone. You may possibly develop this future into a monolithic kernel instead of a micro kernel, however it seems considerable amount of effort would be needed for that, with little benefit - The OS template should be relatively fast on its own.

Now any OS programmer would be able to use this OS template to create the OS of their dreams without thinking to hard about the hardware. Most important is to make the API very user-friendly and well documented in HTML form. Preferably in the format of functions in php.net.


I hope this post will influence enough people to finally bring OS developers out of Stage 1 of operating system development.

Veniamin Ilmer

[stephenj]

I was going to respond with a rebuttal, then I read the "beat the unix out of minix" section and couldn't stop laughing. I could stand the other non-sense, it wasn't funny after all, but the minix part was gold.

I'll tell you a story instead...

Over the past few weeks, I've had very little time on my hands. I've been working constant overtime at work, and have spent the remainder either sleeping or working on my language. I've been waiting for about a week to finish the final level of Halo 3: ODST, but I just can't find the time to do it.

Anyway, what has been on my thoughts, in what spare moments I have, is what features in mmap would be optimal for implementing my memory allocator. How would they impact the design and porting of my language... If I ever find the time to write an OS, how would I design it?

The point is, if I didn't have that kind of control, then the task at hand would be much less interesting. If I wanted to just do cool stuff to an OS, I'd be hacking Linux (truthfully Gentoo provides more kernel configurations then I'd ever need). But I want to see what can be done.

Sometimes the journey is more important than the destination.

Besides, I don't ever want to support users unless I'm getting paid for it. All they do is whine and ask for batshit insane things... Like beating the unix out of minix.

[MasterLee]

Its writing books really that easy? Writing an new Indiana Jones book may be easy. Writing an accurate book about history not.

[Solar]

No, writing an Indiana Jones book that people want to read isn't easy.

Sometimes the journey is more important than the destination.

Full ACK here. My project is my anchor in sanity. The place where I don't have to go through damn stupid bureaucracy, make estimates that get either ignored or turned into hard timetables, work towards my own standards of software quality and generally don't have to bother with the Dark Side of software engineering.

I'd probably have burned out, gone crazy or jumped off some building long ago without that outlet.

[Combuster]

May I add that it's widely considered a beginner mistake (even two) to get group involvement and plan on having everything and the kitchen sink. MS, Apple, and the linux community are all several thousands of people in size. Even with all decent programmers here, we simply don't have the manpower to achieve such goals.

Not to mention that our interests are too varied to ever be truly united.

[Creature]

As of November 2009:
91.26% internet clients use Windows.
5.66% internet clients use Macintosh.
1% use Linux.
And only 1% use something else. This may be BSD, and other *nix OSes.

As usual, statistics can be used to prove anything. A 1% in statistics can consist of 5 million people as far as I'm concerned. Percentages mean very little. Besides, how do you know for sure that the selections and tests are valid? How do you know if someone uses an OS? You ask them? You're always leaving a whole lot of people out if you do so. Where is the test performed? The USA? What about Europe?

And finally, if there is a part of your query that doesn't want to answer, your query will or might be faulty.

I'm not concluding that statistics are crap but they certainly aren't saints either.

[Love4Boobies]

Besides what everyone already said I'd like to add that hardware datasheets are not as hard to find as you make them sound - sure there's a lot of stuff that's not widely available (e.g., information on nVidia chipsets) but you'll usually find what you like. Also, PnP is not a feature that needs to be present in every OS. Not all OSes try to achieve the same goals so don't limit your thinking to general purpose desktops that you use to play your video games on.

1%? Seriously? I was surprised to see that here. It's probably not true but not far from it either. I did expect the Linux community to be larger, though. Oh wait, web client OSes.

[earlz]

this is a very interesting thing. i dont agree with everything you've said like the removing unix stuff. Have you tried implementing a good VFS? it is a lot harder to do than you would think.

i will edit this reply later when im not on my phone posting this

edit:
Ok, I'm back. I say that an OS Kit would be next to impossible to do right because it must be extremely open ended. For instance, some people like synchronous IO and others like async. This in itself means that you would need a significant par of code in the Kit duplicated to support both. And this of course will lead to major code bloat.

I think what we need is a layer of abstraction on top of the entire machine in order to level the playing field. This would not be easy however. I believe the solution there then is something like Xen. Where all of the hardware interfacing is one and consistent and Open(because that's how it's presented in the virtual machine). I believe this to be the only way of being capable of competing with the other big OSs.

What I will be waiting for is the day when we have a special chip that would help make the virtualization to be nearly as fast as native code.(as in, 95% or higher)

[Love4Boobies]

Someone had been trying to create a set o libraries for operating system development, you can find more about it here.

[earlz]

Someone had been trying to create a set o libraries for operating system development, you can find more about it here.

Yes, sadly it's very out of date now and a dead project..

[MXIIA]

I'm not entirely sure you understand what you are saying.

[kabure]

Because of this lack of useful information for operating system development, OS programmers simply get lost or stay near the beginner level of their development.

I agree with you--to a point. Actually, as people pointed out, there already is a lot of information out there about how to write an operating system, and most devices are well-documented. Take microkernels for example: there is a ton of information on how to write them, best practices, etc. and many modern microkernels publish full specifications for them, considering that most of them are research projects anyway. NICTA's seL4 and the EROS group's various microkernels are all very well-documented (except sometimes I want to wring Shapiro's neck for how terse and complicated he can be...)

In any case, while there is a lot of information on how to do something a certain way, unfortunately it can be difficult to find information on why to do something a certain way, which is just as important to an OS designer as the how. And that's a problem that all hobbyists will face. Here's a bunch of page replacement algorithms. Ok, great, but which methods are the best? Which methods are scalable? Which ones can be employed in many different circumstances effectively? Knowing this stuff takes a lot of practical experience, but practical experience tends not to be conveyed and passed down from generation to generation, unless you have a good teacher. Which you won't.

[gravaera]

Hi:

1. Copy the Minix3 code from here.

No.

Because of how Minix is built (small kernel) it is much more easy to understand and thus modify it than any other advanced Operating System.

That is incorrect: Linux is very easy to understand. There is a very simple explanation to give for the apparent complexity in Linux that beginners grapple with: they do not understand the basic abstractions that must be created for a kernel. After a while of reading and designing, you suddenly find that reading Linux source is easy: and you may even begin to see one or two areas where your own design trumps theirs.

There are areas in kernel design and development where all projects must converge, even if they have a radical end-goal. Therefore, the basic structure is there for any project, and reading the source is a matter of interpretation. So while you may think that Linux source code is in itself a barrier to new developers, it is actually a tribute to the Linux source that is designed to be able to neatly enough handle the complications of multiple chipsets and architectures all from one tree. When you begin to understand OS development enough, as much as you may find little things to jab at Linux about (Haw haw, I've got that done better in my design), there's a sort of awe at them all the same, since when you read their source, you see real depth of thought: therefore of course, if you're new on the block, you aren't going to be able to walk around and decipher all of that complex design.

So your argument is flawed, in that the only thing Linux should need to provide for new devs is its source. If you have an idea of how kernels work, looking through to find a particular subsystems, and a particular subfunction within that subsystem is going to be very easy. And in fact, at that stage, you won't be thinking at all about "Why is this so complicated?": you'll see it as gold, because the best, and pretty much the only thing you should be looking for from Linux source are comment lines like these:

"'What to do when a chipset repeatedly raises an IRQ that we have no driver for?' Each arch will have to define this for itself.\n Ingo Molnar says that ... not ACKing an unknown IRQ on the APIC will cause the queue to be filled, and the APIC will lock up."

To the extent that they have a treasure trove of comments of gold just like that, riddled all over the code: years and years of bug reports on hardware quirks, is the the extent that Linux source is phenomenally useful. They will not ever begin putting in useless comments all about to increase the size of the source tree. There are two reasons for this:

1. Nobody there wants an inexperienced coder submitting buggy code: Linux is run on all kinds of computers with all kinds of real-times requirements, etc.
2. Those who need to understand it can easily do so: all you need to do is read more. Read the Intel manuals. Go crazy on google. Read every design research paper you can get your hands on. Read about all those papers you see on obscure, seemingly useless things, like: "Lockless algorithms on the M68K" and all of those.

Read, read, read. And stop complaining.

3. Get rid of everything *nix!

I agree with you here, but I'm very sure that we don't think this way for the same reasons.

There should be no such thing as "/root/" or "/tmp/" or even "/"
Using the new template one should be able to write up an advanced DOS with the usual "C:\" in it.
Remove any command line things that Minix has ("ls", "mkdir", "cd", etc) ..completely get rid of the shell -

And here is the proof. Getting rid of the abstract filesystem layer in an OS which uniformly bridges together all of the mounted filesystems is not going to offer any tangible benefits. There is also the fact that any reasonably powerful, and respectable OS (which is taken seriously by anyone other than a Windows desktop luser) has a powerful command line environment: a command line shell is an inescapable requirement for fallback when the graphical environment is broken or is simply too much overhead for practical use.

Also, removing simple command line tools for creating and manipulating files isn't going to make an OS more usable at all. You can also completely hide a UNIX style VFS from the user by aliasing connected disks to "Windows styled" drive letters. So the user never accessed /media/foo/bar, but does "C:/bar". Your kernel does a lookup, and internally routes that to /media/foo/bar.

Now any OS programmer would be able to use this OS template to create the OS of their dreams

Yes, so we can now all just take a single, restrictive kernel design and write a fancy new GUI on top of it, then call it another OS altogether, and we can all be bashing heads in the market about whose window border style looks better.

I hope this post will influence enough people to finally bring OS developers out of Stage 1 of operating system development.

No. Reading, and taking time to read, and research, and read some more, and not whining and being lazy will get you out of "stage 1".

[Solar]

There should be no such thing as "/root/" or "/tmp/" or even "/"
Using the new template one should be able to write up an advanced DOS with the usual "C:\" in it.

Being opposed to the "one root" file system of Unix is one thing. Seriously suggesting drive letters as an alternative is ridiculous, though.

[TylerH]

As of November 2009:
91.26% internet clients use Windows.
5.66% internet clients use Macintosh.
1% use Linux.
And only 1% use something else. This may be BSD, and other *nix OSes.

There are three kinds of lies: lies, damned lies and statistics. -- Mark Twain, a man who wrote many good books

Sure, that's a great idea. Copy Minix, the inspiration of Linux. It'd only put you back in time ~10 years.

I'm not even going to take the time to respond to the rest. My time is almost worthless, but yet worth more than that...