Restructuring the Wiki to Reflect More Sensible Priorities

[Love4Boobies]

I feel that our wiki is structured in such a way that it misleads newcomers to focus on the wrong aspects of OS development. They all seem to spend a lot of time on something that has essentially very little to do with operating systems: how to set up a development environment and make a very basic piece of software boot on the bare metal. Operating systems are really about two things: managing resources and providing a clean programming interface. We go into details about conventions that anyone could look up, such as how to interact with some arcane piece of hardware or what the on-disk structure of a particular file system is but we fail to teach the core of the subject. And to make things worse, their code invariably ends up with procedural abstractions but no interface abstractions (in just the same manner the Bare_Bones terminal knows about VGA, which is something no sane programmer would ever design).

I propose we actually start recommending that people write their operating systems in user mode and move what are currently the introductory articles into some sort of Interesting Side-Projects category. Before you label my proposal as controversial, think of some of the advantages:

• People can start working on their operating systems right away and learn what they are all about.
• They get to use all the regular debugging tools they would when writing application programs.
• They are more or less forced to use nice abstractions, a practice that should hopefully rub off.

(PS: I also have a feeling it might keep some of the adventurous-yet-unwilling youngsters out of our hair because pretending a file is a disk, moving blocks of memory around in a process' address space, or getting signals and changing contexts is less flashy than the delusional "I'm working on the next Windows".)

[Antti]

I understand your point and pretty much agree on it. Unfortunately it does not work very well in practice. Phrases like managing resources and providing a clean programming interface are actually things an advanced developer appreciates and finds fascinating. However, I claim that an advanced developer very often makes a typical mistake when saying and thinking that beginners should immediately start "the right thing" and skip the "nonsense", e.g. tinkering with "hello, world" boot loaders. At the same time advanced developers have forgotten the path they followed, and if not, they think it was the wrong path. "Had I known all the things at the start, I'd have done the right thing." Just start with the advanced knowledge, skip the beginner experiments and everything is easier?

[Love4Boobies]

I do think there's real world value in tinkering with the ugly interface provided by hardware even if I disagree it was necessarily a good route to getting me where I am today. I don't propose deleting the existing content, only making it a specialized topic of use to people who want to progress in that direction and emphasizing other things to newcomers.

Someone on IRC pointed out that it's potentially dangerous to go this route because it might prematurely lock in the designs of aspiring OS developers. For instance, the solution to the CPU utilization problem during I/O used by most operating systems is to combine blocking I/O with multithreading. So if they use, say, the C standard library, they're likely to make a design decision without even realizing it. There are alternatives, such as event-driven systems with asynchronous I/O.

If enough people are interested, I could put together a little cross-platform OS development framework for user mode that provides a nice abstraction of hardware for anyone who doesn't want to invest in the boring and time-consuming activities of writing drivers or doing architecture-specific things just yet. I have in mind things like signals for interrupts, files for disks, a video framebuffer, etc. So, for instance, if the user wanted to write to a disk, they'd use the framework's API which would write to the appropriate file and send a signal back (setting up an interrupt table would be as simple registering some signals)---this would allow them to decide whether they wanted to block or not.

[Antti]

Sounds a very good plan to try out and I still agree on most of the points. However, you mentioned things like boring architecture-specific things but those could be very essential part of the fascination and an impulse to start doing operating system development in the first place. The clinical user space environment may sound like an excellent idea (and it will be) but it could be much less fascinating if you have not done your share of experiments on "bare metal" (whether it actually is very bare metal when you go into details).

Of course, for many people your user space enviroment approach works very well but please do not underestimate the opposite. These so called boring things are often the ones that differentiate this hobby from other programming fields (which may be, arguably, more productive). Are we trying to get rid of the very core of the difference?

[Love4Boobies]

It's not our job to hook people into OS development using whatever they might find fascinating as bait. Our responsibility is to provide them with advice on how to progress. So I'm not trying to get rid of any difference, whether it exists or not. There are plenty of communities dedicated to low-level programming and there is a fair bit of information here as well. Personally, I don't see any real distinction between OS development and any other kind of software (and, at least to some extent, hardware development), as long as we are comparing projects of similar scales, but that's another story altogether.

And what is so fascinating about interfacing hardware, anyway? If it's the ugly programming interface is working with an emulator just as fascinating? How about working on top of a software layer that is ugly in the same way? Would it be less fascinating if hardware designers put more effort into creating beautiful interfaces? Or is it just, perhaps, the idea that you are somewhat closer to some physical device (not very, but still)? The latter seems like a glitch in human thinking. I strongly believe the healthy attitude to have is to work on solving interesting problems, whatever those may be in your case, rather than aimless tinkering with arbitrary things. Incidentally, that is the only way to make something of any value. The tinkering should be seen as a means to an end.

[dozniak]

If you move any pages just don't forget to plug proper redirects, there's a bunch of sites pointing to osdev wiki, including BeagleBone and elinux.org

[Antti]

Or is it just, perhaps, the idea that you are somewhat closer to some physical device (not very, but still)?

I think that is not the issue. If we are working on top of hardware/firmware layer, the are no extra "hobby layers" but the baseline is exactly the same as it is for everyone else (including the big players in the field). It has not that much to do with the physical hardware. However, I do know that some details make this viewpoint a little bit naive but one of the coolest thing (in my opinion) in "80x86 BIOS/UEFI PC" hobby OS development is that we are able to make an OS that could be on a par with the "real stuff" and the platform is not some kind of hobby toolkit.

For the third time, your idea is good what I said is just an opinion.

[dchapiesky]

Perhaps recommending to beginners that researching and learning how existing projects are organized would be an additional wiki...

Noting the notable projects of http://wiki.osdev.org/Notable_Projects as a starting point, I have at least 30 different github hosted OS projects in a directory which I reference for examples on everything from gdb stubs to usb drivers....

Beginners probably should be schooled in the art of research.

[Antti]

@L4B: It is hard to win an argument because I mostly agree with you. But...

As a general note, have you thought that your little cross-platform OS development framework may itself (design and implementation) be exactly a thing people are interested in? In this regard, maybe people interested in things OSDev represents want to be exactly like the ones who design and implement things you are suggesting? You would always be a bit "above" when you are the one who creates these kind of frameworks.

[Love4Boobies]

@dchapiesky: Definitely. But even when they do research, a lot of people focus only on the coding without having thought about their upstream prerequisites (problem definition, architecture, design) or other aspects of development much thought. It's not that they are unwilling, just that they don't know that these things are very important as well. I suspect it has a lot to do with the fact that universities are oriented mostly towards practical theory but not practical... practice. (Don't misunderstand me, though, I am not one of those people who blames them for teaching unapplicable things, theory is just as important.)

@Antti: You don't have to keep repeating that you agree with most of what I've said. I acknowledged it, we're just having a conversation, not an argument. However, I didn't really understand your last message. Could you rephrase it?

[xenos]

Of course, if the wiki is mainly intended to be used by people, whose goal is to develop an operating system, and to provide them with the necessary information, then I mostly agree with you. To develop an OS, you need to know the tasks of an OS (process management etc.) and how to do them well and efficiently, and that requires some theoretical background as well (such as knowing things like deadlocks and priority inversion). So we should definitely have such topics covered.

The question is: Is that really the goal of the readers?

I'm pretty sure there are also readers who might have different goals, and part of these goals is certainly also hardware related stuff. They might not be interested in writing an actual kernel or OS, but simply enjoy tinkering with the bare metal, boot up a Raspberry Pi with their own code and let it do some simple tasks or whatever. Granted, we are talking about the OSDev Wiki, so already the name suggests it should be about OS development. But many people get attracted towards this topic from tinkering with bare metal, and also real developers might want to implement drivers at some point. So I wouldn't shift these topics into the darkest of all corners.

I think we should have sections on "OS Theory / Concepts", "OS Design / Implementation", "Hardware Interfaces", "OS Development Strategies" (like user space testing, unit testing), "OSDev Tools" etc. to offer something for every taste. The start page of the wiki indeed already has some separation between such topics, but I agree that hardware topics are very dominant there, and I would prefer a much shorter start page, just a short and concise overview of the covered topics. Stuff like "Hello World Bare Bones" would probably fall into pages concerning the boot process on a particular hardware, because you learn where the boot loader drops you off.

[Antti]

However, I didn't really understand your last message. Could you rephrase it?

Harsh and simple: you cannot unlearn what you have learnt already. Why not teaching people things you know and even more? It is very likely that you will know more than your framework users because you implemented it. Why make such a scenario because the users would most likely want to be just like you?

But many people get attracted towards this topic from tinkering with bare metal, and also real developers might want to implement drivers at some point.

People really have to know exactly what they will be doing if starting off with the concepts like "managing resources" and" providing a clean programming interface." I claim that usually people simply do not know what they do when starting a OSDev-like hobby. This is not an exception but a very common pattern and it is not wrong! They just know enough to be interested in "tinkering-with-bare-metal-like" things. This is a good starting position because in the end you will end up tinkering with bare metal.

[Antti]

@L4B: One other thing I would like to add to my previous unstructured reply. If we think about spending time on how to set up a development environment and make a very basic piece of software boot on the bare metal, like you mentioned in your original post, there could be some interesting statistics. I do not have any real ones, though. There seems to be users that very typically have gone through BIOS-phase, DOS-phase, GNU-phase and all kind of typical things in less than a year. Nowadays they are very stand-alone when it comes to OSDev and their "wasted years on wrong things" were actually important when you look at the big picture. They will start thinking about things you recommend they should start off with?

[onlyonemac]

The wiki is a collection of information that's been gathered about different hardware, different algorithms, and so on. It doesn't reflect how to approach OS development. There is however some useful beginner information here: http://wiki.osdev.org/Main_Page#Introduction.

[dchapiesky]

When you start writing an OS you do the minimum possible to get the x86 processor in a usable state, then you try to get as far away from it as possible.

Words to live by.