OSDev.org

I spent some time thinking about this topic and went through the wiki Expanded Main Page as it is right now, and I also thought what might be helpful for people who would like to get a deeper understanding of OS development and its advanced topics, while also keeping the interest of newcomers. I came up with some structure (note that this is not a complete list, but just a collection of ideas / example, what could be put in which category), so please feel free to comment, criticize, improve or write your own

• Preliminaries:
  • Introduction
  • Required knowledge
  • Beginner mistakes
  • Where to get help, how to ask smart questions...
  • FAQ
  • ...
• OSDev Theory / Concepts:
  • Goals of an OS: resource management, API, access rights, protection...
  • API concepts: user / kernel mode, system calls, IPC, shared memory, virtual file system...
  • Computer architectures: Turing, von Neumann...
  • Scheduling primitives: processes, threads, fibers...
  • Thread states & transitions: running, ready to run, blocked...
  • Memory management concepts: virtual memory, address spaces, paging, swap...
  • Synchronization primitives: semaphore, mutex, spinlock, lock-free algorithm, CAS...
  • Synchronization problems: deadlock, priority inversion...
  • Security problems: attack types, prevention strategies, stack protection, address space randomization...
  • ...
• OSDev Design / Implementation:
  • Kernel types: monolithic, micro, nano, exo...
  • Task models: monotasking, multitasking, real time...
  • Scheduling algorithms: round robin, priority based...
  • Physical memory allocators: bitmap, stack, buddy...
  • Virtual memory & heap management: slab allocator, next fit, best fit...
  • Paging / swap strategies: least recently used...
  • How other OSs handle(d) things: Linux, Windows, Mac OS, Minix, L4, DOS...
  • ...
• OSDev tools:
  • Tool chains: cross compiler, linker, assembler...
  • Image creation: ISO images, hard disk / USB drive / SD card images, how to copy...
  • Build / project management: makefiles, CMake, autotools...
  • Simulators / emulators / debuggers: bochs, QEMU, VMWare, VirtualBox, Simics, SimNow!...
  • Testing: test suites & tools, memory usage monitors, static code analysis, code coverage...
  • Automated builds & continuous integration: service providers, setups...
  • ...
• Standards:
  • Binary / executable / object file formats: ELF, PE, COFF, A.Out...
  • API / ABI: POSIX, System V, calling conventions...
  • File systems: Ext2/3/4, FAT, NTFS...
  • Hardware abstractions: UEFI, ACPI...
  • Network layers and protocols: Ethernet, IP, TCP, UDP...
  • ...
• Hardware:
  • CPU architectures and ISAs: x86, x86_64, ARM, MIPS, PowerPC, M68K...
  • Computer types: IBM compatible PC, Raspberry PI, Beagleboard, gaming consoles, mobile devices...
  • Boot process: boot sector, boot loader, multiboot...
  • Buses and interfaces: ISA, PCI, USB, SCSI, ATA, serial, parallel, I2C...
  • Interrupts: PIC, I/O APIC, local APIC, MSI, NMI...
  • Clocks and timers: RTC, PIT, APIC timer, HPET...
  • Human interface devices: keyboard, mouse, joystick...
  • Video: graphics cards, monitors, VGA, VESA...
  • Peripherals: printers, scanners, network cards, sound devices...
  • ...
• Practical help & troubleshooting:
  • Bare bones tutorials & "Hello world!"
  • OSDev using C / C++ / Pascal / Ada / D / C# / Rust / Haskell...
  • Troubleshooting
  • "Help! XYZ doesn't work!"
  • ...

Also, I wouldn't list all the subtopics on the main page, but just the first and second level categories ("OSDev Theory / Concepts" -> "Scheduling Primitives", but not "Threads"), at least not fully expanded, to get a better overview of which topics are covered. Otherwise people might just read the top of the page or miss important information. Also it would prevent "bottom topics" to fall completely out of sight. These categories could also nicely make use of wiki categories whenever appropriate.

Looks reasonable XenOS, I like it.

What would be best for wiki is to delete all of its content and employ an writer to make you articles. On scale of 10 quality of current wiki articles is somewhere around 2 and 3.

Through all "tutorial" pages, long code snippets, recipe-like articles (e.g. there is the first one recipe-like article I found), this like pages should be paramentally deleted.

And especially the ones with headings ending with question mark (?) should be deleted as soon possible.

Lukand wrote:What would be best for wiki is to delete all of its content and employ an writer to make you articles. On scale of 10 quality of current wiki articles is somewhere around 2 and 3.

Deleting all of the content on the wiki would be a terrible idea as it's a good source of information. Although I do agree some articles need to be altered slightly to give it more of an encyclopedic tone.

Lukand wrote:stupidity snipped

Do you know a definition of a wiki? Look it up once in a while.

Wiki gardening is a tedious process to keep wikis well maintained, but it does not include deleting all of the contents as a means of improving it.

Lukand wrote:What would be best for wiki is to delete all of its content and employ an writer to make you articles.

That is literally the worst suggestion I've seen in a long time. Yes, let's just take years of accumulated knowledge and delete it all because "the quality is too low".

Lukand wrote:Through all "tutorial" pages, long code snippets, recipe-like articles (e.g. there is the first one recipe-like article I found), this like pages should be paramentally deleted.

While long code snippets isn't the best way to teach someone about a topic, it's still better than nothing and demonstrates how to implement something. Often I've found that the whole code snippet isn't particularly useful but it features a few lines that demonstrate how to operate a piece of hardware that would have taken a few complicated paragraphs to explain in words. Articles like this could be improved by extracting the important code (i.e. the parts that are actually relevant to what the article's about, rather than the bits that just show how not to write assembly code) and working it into an article with an explanation of how it works/what it does (rather than just saying "here, use this code, it will do what you're trying to do" with no explanation). This would provide useful demonstrations of how to do specific tasks without encouraging blatant copying-and-pasting of whole functions.

By the way, I just discovered that the wiki already has a main page which essentially resembles my suggestion - and which would IMHO make a much better start page than the expanded one:

Categorized Main Page

Maybe one could work on this more, improve it, and eventually use this one as the start page?

There are a few things I didn't like about your proposal. First, there seem to be some inaccuracies (e.g., processes and threads are not scheduling primitives, deadlocks are not synchronization issues). Secondly, the categorization is a bit off (e.g., threads are presented where processes are but thread states are discussed elsewhere). Thirdly, and this is my main complaint, I don't really like all this inconsistent spreading around of articles: We keep theory separate from design but mush design and implementation together; also, in some cases there seems to be confusion as to what is architecture and what is design. I'm also not sure why there should be a separate page for the introduction and for OS goals. All of these are also the types of problems that currently exists on the wiki as well. Even though this is a little off-topic, my suggestion is to have something like this:

• ...
• Multiprogramming
  • Processes
    • ...
  • Threads
    • API's: C threads, POSIX threads
    • Designs: kernel threads, user threads, hybrid threads (with a section on scheduler activations), protothreads
    • Optimizations: thread pools, pop-up threads
  • IPC
    • ...
  • ...
• ...

Ideally, articles that are higher on this hierarchy would present some context for their subarticles. For instance, Multiprogramming might discuss the motivation for running processes concurrently and provide links to alternative solutions to the problem it is trying to solve, mention the idea of time-sharing, talk about how to use queuing theory and the degree of multiprogramming to model and improve CPU utilization, etc.

I haven't given the organization too much thought so don't consider this a blueprint. For instance, although books on OS theory don't really treat them this way, to my mind, it makes a lot more sense to put Threads under Processes, since to former are a type of resource and the latter are ultimately resource containers with privileges, relationships, etc. Textbooks typically discuss process states (by this I mean Ready, Running, Blocked, and Terminated) and then later say "threads are pretty much like that, too". How about making these a characteristic of threads only since processes can be single-threaded? I think there are a lot of pedagogical opportunities like that.

PS: I was also amused to see Turing machines mentioned under architectures. Surely we're not going to write a crash course into computability theory so such theoretical models are of little value to this wiki.

Love4Boobies wrote:There are a few things I didn't like about your proposal. First, there seem to be some inaccuracies (e.g., processes and threads are not scheduling primitives, deadlocks are not synchronization issues). Secondly, the categorization is a bit off (e.g., threads are presented where processes are but thread states are discussed elsewhere).

Of course, despite the formatting, this list was not meant to be a final tree structure, but just a broad list of keywords (and so the categories were not meant to be accurate). This definitely still needs some sorting and proper categorization, it's just a first draft.

Thirdly, and this is my main complaint, I don't really like all this inconsistent spreading around of articles: We keep theory separate from design but mush design and implementation together.

In this first draft, I used "design" and "implementation" synonymously in the sense that "it is a design decision, how I implement my memory manager, i.e., using a flat heap or slab allocator". But of course, one can have a more fine-grained separation, if that makes sense (different designs of memory allocators, vs. how they are implemented in some given programming paradigm). These terms are relative, after all - there is a whole hierarchy of "implementations", from the algorithm down to the bare machine instructions.

I haven't given the organization too much thought so don't consider this a blueprint. For instance, although books on OS theory don't really treat them this way, to my mind, it makes a lot more sense to put Threads under Processes, since to former are a type of resource and the latter are ultimately resource containers with privileges, relationships, etc. Textbooks typically discuss process states (by this I mean Ready, Running, Blocked, and Terminated) and then later say "threads are pretty much like that, too". How about making these a characteristic of threads only since processes can be single-threaded? I think there are a lot of pedagogical opportunities like that.

I think it should ideally be somewhere between a good pedagogical work and a practitioners guide. It might be neat to have a wiki which could also be used as an actual course material, but I don't think that this is the intention or use case of most of its users. So one goal of a good wiki organization should also be how to easily and quickly find the article you need when you are facing a particular task or problem in the course of writing your OS.

PS: I was also amused to see Turing machines mentioned under architectures. Surely we're not going to write a crash course into computability theory so such theoretical models are of little value to this wiki.

Again, this was just an example for some topic, to give another possible viewpoint on a computer from the theoretical side, besides von Neumann architectures and finite state machines, to name just two other examples. It might help at some point to view a computer as one or the other.

XenOS wrote:By the way, I just discovered that the wiki already has a main page which essentially resembles my suggestion - and which would IMHO make a much better start page than the expanded one:

Categorized Main Page

Maybe one could work on this more, improve it, and eventually use this one as the start page?

Categorized Main Page and Expanded Main Page are the two entry pages you can toggle with the "Short View / Expanded View" at the top.

The Categorized / Short view was the main entry page at one point, but it was considered to not give as well an overview of what is actually available from the Wiki, so the Expanded view (which was the "original" presentation all the way back to the static OSDev FAQ) was made the default again.

You can edit either view.

Personally I think that fussing over what's a thread and what's a process and what scheduling algorithms apply where is pretty stupid for a wiki like this. A lot of that stuff is fairly specific to the design of each OS, and it's up to the designer/developer to decide how they want to apply that information. What's important is that the information is easy to find, which currently it is.

If there's any issue with the wiki at the moment, it's that noobs just copy-and-paste code from the wiki and expect it to work without understanding how to use or adapt the code appropriately. Offering more in-depth explanations rather than code samples that look like an invitation to copy-and-paste, as I've already suggested, should deal with this well.

Solar wrote:The Categorized / Short view was the main entry page at one point, but it was considered to not give as well an overview of what is actually available from the Wiki, so the Expanded view (which was the "original" presentation all the way back to the static OSDev FAQ) was made the default again.

Personally, yes, if I had seen that page first when I had found this site I probably wouldn't have stayed around nearly as long. It makes the wiki look empty and like there isn't much on it at all.

Hmm, that seems self-contradictory to me. On the one hand, you want to avoid discussing theory because different goals imply different trade-offs. On the other hand, you want more in-depth explanations. What do you want more in-depth explanations of? How arbitrary pieces of hardware that people might want to support are supposed to be interfaced? No sane employer will be impressed if you know how to change the video mode, set up paging, probe the PCI bus, or do an USB request because any dummy could look that stuff up; it's just a bunch of conventions that require pretty much no knowledge or skill. We might as well only provide links to specifications, datasheets, and manuals for these.

As you suggest, how they apply the theoretical information depends on their goals. But I don't understand how that is an argument against discussing it. Having the information in the first place is a prerequisite to applying it. If you don't have it, you might get stuck, pick an ineffective solution, or fall victim to avoidable pitfalls. People have solved most problems encountered in operating systems and it's worth knowing how they did it. You've mentioned threads so let's take them as our example. If you don't understand what problems they are supposed to solve, you might end up thinking they are a silver bullet when in fact they are sometimes wrong, even in a highly parallel system. Or perhaps you've optimized your code to use thread pools without knowing pop-up threads are possible, or vice versa. Or perhaps you're using spin locks everywhere because you don't know that other synchronization primitives exist, each with its strengths and weaknesses. Or perhaps you haven't considered that you could be using message passing instead of shared memory. Or perhaps you're not aware that you could be writing a functional system, where synchronization isn't necessary because there are no side effects. Or perhaps you have but you don't understand what trade-offs you're making. What about other problems like priority inversion, starvation, etc?

Which do you think is more valuable, having a reference of what each bit in a hardware register does or understanding how to solve problems?

Love4Boobies wrote:Which do you think is more valuable, having a reference of what each bit in a hardware register does or understanding how to solve problems?

Problems...

being exposed to as many alternatives as possible and the reference materials to understand why those alternatives are available is extremely important.

A person learning OS Development may not have any idea what a lockless ring is but encountering it in a vacuum does nothing... however encountering lockless rings in the context of inter-process communication is invaluable.

Love4Boobies wrote:Hmm, that seems self-contradictory to me. On the one hand, you want to avoid discussing theory because different goals imply different trade-offs. On the other hand, you want more in-depth explanations.

No I was meaning that arguing about whether to classify "scheduling algorithms" under "processes" or "threads" (and similar such discussions) is a waste of time because it's up to the OS designer how they're going to apply scheduling algorithms. It's better to just explain scheduling algorithms in general and let the designer decide where and how to apply them. (Same goes for e.g. memory allocators, priority management, etc. - just explain the general idea and whether the priority management algorithm's applied to scheduling priorities, disk I/O priorities, or whatever is irrelevant.)

In terms of working with different kinds of hardware or protocols, that's where I think some of the copy-and-paste code samples need to be developed into explanations of how to interact with the hardware or how the protocol actually works, instead of (e.g.) showing a code sample called "read_byte" which reads a byte from a device and returns it as an unsigned char without explaining *how* to get the byte from the hardware. (Note: there's absolutely nothing wrong with showing the actual code to get the byte from the device, which often clarifies exact details like what memory addresses to read from or write to, what order to do things in, etc. that can sometimes be missed or difficult to explain in explanations in words - it's just the "complete functions" which encourage people to copy and paste without them understanding how to adapt the code for their situation.)

Love4Boobies wrote:No sane employer will be impressed if you know how to change the video mode, set up paging, probe the PCI bus, or do an USB request because any dummy could look that stuff up; it's just a bunch of conventions that require pretty much no knowledge or skill. We might as well only provide links to specifications, datasheets, and manuals for these.

Let's remember that OSdev isn't about impressing employers. But resources such as those that you list here are valuable to OSdevers because they explain in Plain English (tm) how to do stuff that would otherwise requiring finding, reading, and interpreting complex or even confusing specifications and datasheets. Sadly a lot of people here like to say "just read the specification", but the truth is if someone already knows how to do something (either because they read the specification, read someone else's code, or figured it out themselves) there's no reason for them not to share that information. Plus often specifications aren't followed correctly by hardware developers (usually because they're too vague or complicated - the same reason why OSdevers struggle to understand them), and the wiki is a useful place to document how stuff *actually* works rather than how it's *supposed* to work.

Love4Boobies wrote:Which do you think is more valuable, having a reference of what each bit in a hardware register does or understanding how to solve problems?

The former. That's the stuff that's difficult to find elsewhere. The latter is something you should already know if you're trying OSdev, and if you don't it's not the place of the OSdev wiki to explain it (note that with this statement I am *not* referring to solving problems in the sense of using debugging tools - the resources on debugging over serial links, connecting gdb to your kernel, etc. are very helpful and again stuff that's difficult to find out elsewhere).

onlyonemac wrote:

Love4Boobies wrote:Hmm, that seems self-contradictory to me. On the one hand, you want to avoid discussing theory because different goals imply different trade-offs. On the other hand, you want more in-depth explanations.

No I was meaning that arguing about whether to classify "scheduling algorithms" under "processes" or "threads" (and similar such discussions) is a waste of time because it's up to the OS designer how they're going to apply scheduling algorithms. It's better to just explain scheduling algorithms in general and let the designer decide where and how to apply them. (Same goes for e.g. memory allocators, priority management, etc. - just explain the general idea and whether the priority management algorithm's applied to scheduling priorities, disk I/O priorities, or whatever is irrelevant.)

Oh. Well, I was trying to argue that the distinction between processes and threads is actually independent of design: entities that have a control flow should be called threads and they belong to processes. It's a matter of vocabulary. For instance, I would say that DOS processes are single-threaded even if, in some sense, it has no notion of threads. Even in the versions of DOS that support multitasking, it is these threads (one per processes) that multiplex the CPU.

I agree that scheduling is a broader problem that can be discussed separately. And since some algorithms are more suitable for, say, thread scheduling than they are for disk scheduling, they can be linked to from the article describing threads. I imagine an article on threads with a section on scheduling that links to the main article on scheduling algorithms and then talks a little about what requirements and constraints threads typically have and why.

onlyonemac wrote:Sadly a lot of people here like to say "just read the specification", but the truth is if someone already knows how to do something (either because they read the specification, read someone else's code, or figured it out themselves) there's no reason for them not to share that information.

I sometimes do that and I will give you my reason for it: it scales way better. I believe that I am of more help to someone if I help them to become self-reliant than I am if I spoon-feed them. You know what they say: give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime.

onlyonemac wrote:

Love4Boobies wrote:Which do you think is more valuable, having a reference of what each bit in a hardware register does or understanding how to solve problems?

The former. That's the stuff that's difficult to find elsewhere. The latter is something you should already know if you're trying OSdev, and if you don't it's not the place of the OSdev wiki to explain it (note that with this statement I am *not* referring to solving problems in the sense of using debugging tools - the resources on debugging over serial links, connecting gdb to your kernel, etc. are very helpful and again stuff that's difficult to find out elsewhere).

Well, on this point we disagree greatly. Conventions are trivial and context-specific (e.g., language tips only apply if you use the right language, etc.). On the other hand, the problems encountered in a domain are typically more general. Not to mention that people who are just starting out in a particular area aren't necessarily equipped to deal with it yet (it's clearly true for our community as well judging from the forums). That is precisely why textbooks and courses on operating systems cover OS theory, not hardware. A wiki like the one you envision is esentially about hardware programming, not about operating systems, and there are places like that for anyone interested. Imagine a wiki on calculus. What does it cover?