OS theory questions

[smmalmansoori]

I am new to OS design and development and have a couple of questions that I could learn a great deal from your feedback on them.

1. I get the impression that most mainstream OSes today are bloated due to the years they've evolved through, the concentration on providing new features rather than refining their original code, and sometimes coding ugly workarounds for bugs rather than rewriting that code portion (since their are always short deadlines). If a company made an OS providing the already existing features with a much cleaner code would it actually be a better OS?

2. Is it worth it to implement a database file system in an OS where tags are used instead of a file hierarchy?

3. I often hear/read that programming an OS in C rather than C++ is better since C compilers are less error prone. If I follow that reasoning, would it be worthwhile to program an OS in Objective-C which could be a mid-point considering it is smaller than C++ but still get the advantage of OO programming?

[roboman]

1. I get the impression that most mainstream OSes today are bloated due to the years they've evolved through, the concentration on providing new features rather than refining their original code, and sometimes coding ugly workarounds for bugs rather than rewriting that code portion (since their are always short deadlines). If a company made an OS providing the already existing features with a much cleaner code would it actually be a better OS?

Is the XBox OS better then Windows? My personal view is there is some of what you said, mixed with business deciding direction... Back in the days of middle earth (after the mainframes, minis and s-100s, but before the 386) the OS was tiny, but so were the computers. Each piece of hardware was standard, in most cases you had no options or if you did it was only of one or two things or as in the case of hdd's, they all looked about the same to the software. Then came the sound cards and each program had to support several sound cards and the user had to select the one they had when the program started. Then companies made the jump past vga and no two companies did it the same way, so programs had to support a long list of video card options, if they expected to sell. In stepped Microsoft with the word virtual, all the hardware company had to do was write a windows driver and programmers could use their hardware with out having to write a custom section for each video card or sound card. Microsoft really hammered the nail into the coffin of the other systems and OS's by adding more and more chunks of pre written software to the OS, so programs could be written much faster since you had to write less. Then things kept moving on and things in both software and hardware had to be kept some what backward compatible or no one would buy the new system. Now that things are a near monopoly each new ver is made to break the old one so you have to upgrade, because there are no other options. But the hardware is still a pain, even little 'standard' things aren't, IE on the computer side of the usb there are at least 4 different controllers that work different. Even the hdd now has several 'standards' at the low level.

2. Is it worth it to implement a database file system in an OS where tags are used instead of a file hierarchy?

A case could likely be made for times when that would work better, ntfs takes a step in that direction.

3. I often hear/read that programming an OS in C rather than C++ is better since C compilers are less error prone. If I follow that reasoning, would it be worthwhile to program an OS in Objective-C which could be a mid-point considering it is smaller than C++ but still get the advantage of OO programming?

The language wars C and Unix have a very long history of being used to develop each other. Then there are the asm supporters or even the ones who wrote an os in basic. Then there is java script, but can you call it an OS with that much other software supporting your script? It's a personal choice and if you get comfortable in several languages, you will likely write different pieces of things in different languages. A general rule is that the closer you get to the hardware the more direct control you need over exactly what the software is doing to the hardware, the further you get from directly telling hardware what to do in detail the more the details don't matter and you can afford more abstraction (assuming the abstractions and tools that let you write things with out looking at every detail make it faster and easier, so you can finish a project in reasonable time)

[Colonel Kernel]

I am new to OS design and development and have a couple of questions that I could learn a great deal from your feedback on them.

1. I get the impression that most mainstream OSes today are bloated due to the years they've evolved through, the concentration on providing new features rather than refining their original code, and sometimes coding ugly workarounds for bugs rather than rewriting that code portion (since their are always short deadlines). If a company made an OS providing the already existing features with a much cleaner code would it actually be a better OS?

That is an excellent question, and it's actually applicable to all software, not just OSes.

I've worked on a few fairly big legacy products over the years, and while they may continue to work well for the end users, they suffer from a few problems:

• They are difficult to extend with new features. Usually version 1 was "designed" in a hurry, or not really designed at all, meaning that every new feature you add involves some kind of hacks that make the system less and less maintainable over the years. Eventually it gets so expensive to modify the code that you just have to throw out a lot of it and start over.
• It can be difficult to optimize performance. This is for much the same reason: the initial design made some assumptions that turned out to be bogus, making it hard to change.
• They can be full of security holes, which are hard to fix. Big, complex systems often have a lot of surface area to attack and aren't modular enough, making it hard to apply principles like least privilege.

Bringing this back to OSes specifically, yes, they are big and complicated. IMO the reason we still have them is because the companies maintaining them have a vast amount of resources to do so (for Linux, replaces "companies" with "communities", but it's the same idea). When it becomes more expensive to maintain the old system than to create a new one (expensive in terms of direct R&D costs, but also opportunity cost of lost share due to poor performance, lack of new features, and too many security holes), we'll see new OSes and OS architectures emerge. Until then, the status quo is good enough.

2. Is it worth it to implement a database file system in an OS where tags are used instead of a file hierarchy?

We've had this debate a few times on these forums, and the only conclusion I have drawn from them is that people don't agree on the definition of "database file system". It's better to define what kind of end-user scenarios you want to enable, then work backwards to what kind of file system features you will need. Literally replacing your garden-variety block-based hierarchical FS with a relational DBMS is probably not a good idea.

3. I often hear/read that programming an OS in C rather than C++ is better since C compilers are less error prone. If I follow that reasoning, would it be worthwhile to program an OS in Objective-C which could be a mid-point considering it is smaller than C++ but still get the advantage of OO programming?

There are tradeoffs involved in all of these, but here are a few important points:

• For sheer portability, you can't beat C.
• You will have to self-host your language of choice for OS dev. If you want to spend as little time as possible working on a run-time library to support your language, use C, and do not use Objective-C. Objective-C is a neat language for application development, but its runtime library is pretty big and its method dispatching mechanism is too slow for really latency-sensitive code like in the kernel.
• C++ can be a good choice if you're comfortable with the language, but it is pretty complicated. I think there has been a lot of FUD about the quality of C++ compilers, but I can tell you from experience that it is hard to write C++ that is truly portable between compilers unless you stick to a very limited subset of the language (and defining that subset will require experimenting with all the compilers). It may not be an issue if you're just going to pick one compiler for OS dev and stick with it, but if you ever want to switch compilers this may be a concern.

[Candy]

Let's keep this thread for the OP, shall we?

[Brendan]

Hi,

1. I get the impression that most mainstream OSes today are bloated due to the years they've evolved through, the concentration on providing new features rather than refining their original code, and sometimes coding ugly workarounds for bugs rather than rewriting that code portion (since their are always short deadlines). If a company made an OS providing the already existing features with a much cleaner code would it actually be a better OS?

I'd split "bloat" into several main categories:

• Hardware Complexity: This is simple enough. If you only need to care about one video card/monitor that's guaranteed to be using one "standard" video mode then you can avoid lots of work. If you only need to worry about one type of CPU (e.g. "core 2" and not Opteron, Athlon, Pentium 4, etc) then you can avoid lots of work. If you only need to worry about "single-CPU" and don't need to care about scalability, SMP, NUMA, etc then you can avoid a lot of work there too. If you need to support 30 different sound cards (rather than using a single hardware interface for all of them) then it's going to be a lot more work.
  Software Complexity: People just aren't happy with text mode anymore - they've come to expect full GUIs and widgets and many programs running at once and... It all adds complexity, which adds more code, etc.
  Managing Complexity: As hardware and software complexity increase, it becomes harder to manage a project. You start looking for tools that remove some of the work involved - higher level languages, run-time checks (both software checks like "assert()" and hardware checks like paging), splitting things into separate modules, etc. All of this takes more code, on top of the extra code you needed to add the extra complexity in the first place.
  Backward Compatibility: This effects everything sooner or later. You can write a nice "clean" OS today without any backward compatibility, and in 10 years time you'll be propping up old APIs hoping people won't kill you for breaking something...
  Media: People just aren't happy with ASCII art anymore - they've come to expect high resolution porn, and singing/dancing/animated web pages with flash and Java, and scalable font renders, and alpha blending and compositing effects and animated mouse cursors and thumbnail previews and... All the eye-candy (and ear-candy) isn't free.
  Accessibility: People still aren't happy with ASCII. Unicode and internationalisation and support for different time-zones and several different types of calendars. Interfaces for people with disadvantages (magnify/zoom, sticky-keys, text-to-speech, speech recognition, braille output, etc). None of this is free either.
  Performance: Better performance means more complex code (which is almost always larger).
  Wide Scope/generalisation: You invent a better mouse trap. Someone complains that it doesn't wash dishes, so you add that to the design. Next thing you know you've got a machine that does everything you can imagine that is as large as the average house. Then someone points out that it's too large to be a good mouse trap.
  Lazy Programmers: I'd like to think that this is actually the least common cause of bloat...

2. Is it worth it to implement a database file system in an OS where tags are used instead of a file hierarchy?

People have talked of similar ideas before (e.g. the "How about design the VFS like DB interface?" thread).

3. I often hear/read that programming an OS in C rather than C++ is better since C compilers are less error prone. If I follow that reasoning, would it be worthwhile to program an OS in Objective-C which could be a mid-point considering it is smaller than C++ but still get the advantage of OO programming?

Um. If you follow that reasoning, then you should be using a hex editor (because assemblers are more prone to errors)...

The best advice anyone can give is to use the tools that you are most familiar with. If you already know C++ but have never programmed in Objective-C, then...

However, there are advantages for using C (most example/tutorial code is in C, including full OSs like FreeBSD and Linux); and if you use a "less common" language you'll have less chance of getting help if you get stuck (and maybe less chance of getting volunteers to write drivers, etc when all the hard work is done). Of course I don't use C for my OS, so...


Cheers,

Brendan

[smmalmansoori]

Regarding point 1: and trying to solve it using a microkernel approach

Hardware Complexity: drivers could be separated from the kernel, but could the same be done for support of different CPUs where the a basic CPU class is developed and the other more specific ones are subclassed?

Software Complexity, Managing Complexity, Media, Accessibility: I could imagine how complex it might get but isn't that because those days they had to write it in low level languages since the hardware wasn't very capable back then (so basically historical reasons) ? would it help in lowering the complexity if for example the desktop environment get written in a high level language (i.e. imagine gnome or kde written in python)

Backward Compatibility: Why not break compatibility when needed and run the different versions simultaneously? I know that it is redundant, but it is practical and disk space is not usually an issue anymore.

Wide Scope/generalisation: how about plugins?

[Thomas]

Hi smmalmansoori,

I am new to OS design and development and have a couple of questions that I could learn a great deal from your feedback on them.

1. I get the impression that most mainstream OSes today are bloated due to the years they've evolved through, the concentration on providing new features rather than refining their original code, and sometimes coding ugly workarounds for bugs rather than rewriting that code portion (since their are always short deadlines). If a company made an OS providing the already existing features with a much cleaner code would it actually be a better OS?

For most of the mainstream/commercial operating systems it's actually $$$ at end of the day that matters. If anyone says something different, they are lying .If a feature sells the product, it will definitely be included in the release.The main reason becomes bloated / ugly is

• Bugs are real and sometimes they need to fixed fast to satisfy the customer
• No architect has enough foresight to imagine every scenario in real world ( corner case
  design is not really practical in real word -- having worked with a v1 product and a very
  stable product ( ~ 30 years ) , that's what my experience has taught me.)

You should think twice, or thrice before re-writing anything. Please read joel's blog : http://www.joelonsoftware.com/articles/ ... 00069.html.

--Thomas

[Brendan]

Hi,

Regarding point 1: and trying to solve it using a microkernel approach

Hardware Complexity: drivers could be separated from the kernel, but could the same be done for support of different CPUs where the a basic CPU class is developed and the other more specific ones are subclassed?

Drivers could be separated from the kernel, but that only increases the total size (which is why I mentioned "splitting things into separate modules" into the "managing complexity" part).

Having some sort of "CPU module" might work if all the rest of the code (e.g. the rest of the kernel, the drivers, the applications, etc) are written in some sort of byte-code, where the "CPU module" is responsible for executing (translating/interpreting) the byte-code. That's going to add a lot more complexity (bloat!) though.

Software Complexity, Managing Complexity, Media, Accessibility: I could imagine how complex it might get but isn't that because those days they had to write it in low level languages since the hardware wasn't very capable back then (so basically historical reasons) ? would it help in lowering the complexity if for example the desktop environment get written in a high level language (i.e. imagine gnome or kde written in python)

People actually were happy with simple stuff back then. For example, people used word processors that weren't WYSIWYG that only handled ASCII and didn't even support a mouse (or different fonts); and people thought it was wonderful because it was much much better than the type-writers they were used to...

Using an even higher level language might help to reduce the complexity even more; but I was only talking about bloat (and when it comes to bloat I really don't want to see Gnome or KDE implemented in python, thanks..).

Backward Compatibility: Why not break compatibility when needed and run the different versions simultaneously? I know that it is redundant, but it is practical and disk space is not usually an issue anymore.

Wouldn't running 2 different versions of the same thing add up to about 100% more bloat than running just one of them?

Wide Scope/generalisation: how about plugins?

Extra code for the actual work, plus extra code to implement the interface/s for the plugins, plus more code to manage plugins. Not sure how that is less bloated than just having the extra code for the actual work alone.


Cheers,

Brendan

[Colonel Kernel]

Lazy Programmers: I'd like to think that this is actually the least common cause of bloat...

In my experience, it isn't "lazy" programmers who cause trouble. It's ignorant ones, enabled by managers in a big rush with no respect for proper design.

[smmalmansoori]

Regarding point 2:

I know this might be way far fetched, but I would appreciate your analysis on this.

If files (all types text, images, audio, video, etc.) are implemented as tables in a database wouldn't they be read at a much faster rate than normal reads?

For example, Imagine a text file's implementation would be that the table's name is the actual file's name and every line in the file is actually a row in that table. That way in programming I can read the 3rd line of the file without calling the first 2 before it in order to reach it, I can even read lines simultaneously. Imagine how that would impact speed if all types of files are implemented that way (or at least the ones possible to implement in such a manner).

Your thoughts?

[Combuster]

If files (all types text, images, audio, video, etc.) are implemented as tables in a database wouldn't they be read at a much faster rate than normal reads

No. In the end you still need to load the same amount of data from the same places on the disk.

For example, Imagine a text file's implementation would be that the table's name is the actual file's name and every line in the file is actually a row in that table. That way in programming I can read the 3rd line of the file without calling the first 2 before it in order to reach it, I can even read lines simultaneously. Imagine how that would impact speed if all types of files are implemented that way (or at least the ones possible to implement in such a manner).

Your thoughts

Several problems:
- Parallel parsing of files leads to concurrency problems (and often records are dependent on previous ones)
- Who's going to make sure that indices are generated on a file's natural boundaries?
- How much time do you lose on writing that you can save on reading?

In reality, for the files that do matter, the index is maintained by the app in question, if not just stored as separate files.