Not-just-for-fun OS development

[vvaltchev]

Hello everyone,
I realized that most of people here develop their own OS for fun and to learn OS development itself, I guess. I started the development of Tilck exactly for the same reasons: at the beginning, I didn't plan to make it Linux-compatible and it was called "ExperimentOS", a place where I could have fun making experiments in the OS field. But, after some time, my practical & goal-oriented mindset kicked in and I decided that I cannot simply spend so much time working on something that big just for fun. Development could be funny, but it must have at least some tiny chance of becoming useful in the "real-world" one day in the future. That's when I decided to make it compatible with Linux and set in stone the rule that it must run many useful 3rd-party applications written for Linux/POSIX. Writing non-test userspace applications for Tilck was practically banned as idea. If I had to write something, if should work on Linux the same way as on Tilck.

As you can guess, at this point a fair amount of fun just went away. I had to spent quite some effort debugging stuff and trying to understand why it didn't work as expected. Also, had to support any fancy Linux syscall that libmusl wanted to use, instead of just sticking with the simplest ones (e.g. readv() instead read() etc.) because I didn't want to have "my own libc". So, it was a lot less fun, but it felt like more useful at the end. Indeed, a lot of people in the industry (Linux kernel devs) were impressed by that.

The side effect of my very conservative and practical choices (C language, monolithic architecture, combability with Linux) was that not only the whole project less fun and more complicated to develop, but it didn't trigger much interest in OSDEV community. Over time, I realized that almost the entire OSDEV community (here and outside) is either interested in:

- just-for-fun projects where people decide to re-write the whole software stack, from the kernel to the UI, including a browser, text editors etc.
- working on big mainstream projects like Linux and FreeBSD.

As you can see, my project doesn't fit in neither of those categories: it's not production-ready so that Linux hackers / embedded developers would be interested in patching it somehow, but it's neither fun like the other projects: there is a ton of focus on automated testing, coverage etc. and zero interest in making colorful windows to appear on the screen.

Therefore, my question is: how many of you are investing effort on developing something not-just-for-fun? I mean, do you have a long-term plan to make your project usable in the real world by the industry or the home user, even if it will have only a tiny chance of getting it there? What would be the advantages of your OS compared to what currently exists? I mean, if you're writing a desktop OS, you'll be inevitably competing at least with the mainstream Linux distributions: how do you plan to steal some of their market? And.. are you sure that your should be writing a kernel for achieving that, instead of just consuming Linux as kernel and making its GUI more powerful and user-friendly?

To me, writing the next "desktop OS" is so hard to achieve as goal that not even Linux succeeded in that in 30+ years, despite running on billions of machines. A more approachable (even if still insanely hard) goal would be write a special-purpose operating system, like a real time OS for embedded devices. Do anybody has goals like that? And again, what's your plan to make it, even if it's a 1 in 100 shot ?

Vlad

[devc1]

An OS that is not for fun must answer this question :
Do you or do you plan to have the billions necessary for that ?

I also plan to do this with my OS, but I am still trying to make other projects and sources of income from where I make (more accurate : take) money. You need to spend billions to make your OS compete with other OSes.

[thewrongchristian]

Therefore, my question is: how many of you are investing effort on developing something not-just-for-fun? I mean, do you have a long-term plan to make your project usable in the real world by the industry or the home user, even if it will have only a tiny chance of getting it there?

I develop my code mainly for fun. I've always wanted to do an OS kernel, with a particular interest in memory management. I'd long studied how OSes worked, and wanted to put what I'd learned into practice.

I'm aiming to get sufficient functionality to port busybox.

What would be the advantages of your OS compared to what currently exists?

While developing my memory allocator, I noticed how easy it was to go from a pointer to a memory block, and thus looking up allocated memory information from a raw pointer, even an internal pointer to a structure. From that, it seemed easy to implement a mark and sweep garbage collector. And thus, my OS gained conservative garbage collection.

The other thing I use in my OS is exception based error handling, using setjmp/longjmp. I much prefer exception based error handling, as exceptions can't be ignored or forgotten, and if code is written to be exception safe, then there is very little else that has to be done to properly handle errors. Return code error handling bites me daily in my day job, with a legacy C++ code base based on C++-03 and extensive use of COM objects.

The result being that my code is quite uncluttered and minimal looking, as the much of the memory management and error handling is concentrated at a smaller number of points.

I would hope these two points would make my kernel more maintainable and easier to learn, and so when ready for wider consumption, I can unleash the code on the world and I'd be flooded with patches for devices I don't have.

A more approachable (even if still insanely hard) goal would be write a special-purpose operating system, like a real time OS for embedded devices. Do anybody has goals like that? And again, what's your plan to make it, even if it's a 1 in 100 shot ?

In reality, I know full well that few on here would bother to look beyond a cursory glance (which I've no problem with, we're all here to work on our own pet projects.) And I'd expect little chance of it making it in the real world, especially as a desktop OS. You just have to look at the massive amounts of effort that have gone into the likes of ReactOS and Haiku to see that alternative OSes have a mountain to climb.

I think the best bet for my OS commercially would be as an embedded OS. I can see things like the garbage collection being a real boon for embedded systems, which often has a low duty cycle and can do garbage collection in idle time. Something like an IoT platform, that provides a vaguely POSIX like interface but with a lower overhead than something like Linux, and provides a smaller, more comprehensible code base.

With that in mind, I'm keeping my code base small, with a target to be usable in as little as 4MB of RAM (I'm currently above that, but probably because I use a ramdisk root.)

But other than that, I have no plan. As it is, I'll be happy to make it to a busybox based user space.

[nexos]

Therefore, my question is: how many of you are investing effort on developing something not-just-for-fun?

Yes. My plan is to research designs of OS that would otherwise be overlooked (like microkernels, various ideas for distributed systems, security ideas, FS ideas, and so on). What impact my research will have on the real OS world is probably minimal, but you never know.

In reality, developing my OS has been grueling at times due to my attempts to do things the right way.

I mean, do you have a long-term plan to make your project usable in the real world by the industry or the home user, even if it will have only a tiny chance of getting it there?

Not by the home user, but in high uptime systems that need maximum uptime. I plan on making my OS very stable, and for crashes to be greatly reduced.

I'm not planning to make it faster, but I do want close enough parity. Yes, I believe this is possible with microkernels.

An OS that is not for fun must answer this question :
Do you or do you plan to have the billions necessary for that ?

Not true. Linus Torvalds didn't pour billions into it, people got interested in his work and then he started receiving donations after that.

[klange]

Consider that the active participants in the community are, by and large, hobbyists. The people working on "serious" OS projects are grad students at universities working on research teams, or engineers at companies like Google, and neither group spends much time, if any, interacting with us. On top of that, I would guess that neither group is inclined to brag or share updates on a web forum - that's what papers are for.

As you can see, my project doesn't fit in neither of those categories: it's not production-ready so that Linux hackers / embedded developers would be interested in patching it somehow, but it's neither fun like the other projects: there is a ton of focus on automated testing, coverage etc. and zero interest in making colorful windows to appear on the screen.

I take this, and your earlier comment about text editors, as a slight against me specifically, and I want you to know that you have set up a strawman by declaring that "re-writing the whole software stack" is the antithesis to being useful. I also think you greatly exaggerate the binary partitioning of OSdevers in the community at large.

And.. are you sure that your should be writing a kernel for achieving that, instead of just consuming Linux as kernel and making its GUI more powerful and user-friendly?

My OS has been noted for its graphical environment, and this is no accident - I did contribute to making Linux GUIs better: I worked on one of the foremost X11 compositors before starting my own OS. I also spent some time at Apple working on macOS's window server. So, my desire for my OS to have colorful windows with fancy effects is supported by a professional background in that area. I did not set out to write a GUI from scratch, but it was a natural step for someone with that background.

Therefore, my question is: how many of you are investing effort on developing something not-just-for-fun? I mean, do you have a long-term plan to make your project usable in the real world by the industry or the home user, even if it will have only a tiny chance of getting it there? What would be the advantages of your OS compared to what currently exists?

There are more ways for an OS project to be useful than for the OS itself to actually be usable by end-users.

I started my OS project while I was a university student with the aim to learn myself. I wanted to understand everything from the ground up by building everything from the ground up. There are many components in an OS, many opportunities to learn something new not just in kernel space but beyond - be it how to design a windowing system, how TTY APIs work, how the terminal emulators on top of them work, TrueType rasterization, compilers... My OS covers a lot of ground because there is a lot of ground to cover.

It wasn't always that way. I did use an off-the-shelf libc (newlib - I predate the era when Musl was viable even for Linux) for a long time - and all sorts of third-party libraries. My compositor was originally built on Cairo, most of my GUI apps were written in Python, Freetype was rendering my text... but as you say, this got to a point where it wasn't fun, and each third-party component was another opportunity to learn. I was particularly inspired by an infamous software engineering interview question: "What happens when you type google.com into your web browser's address bar and hit enter?". I wanted to be able to answer that question line by line, and I wanted every one of those lines to be one I wrote. So, yes, I wrote my own libc (or, rather, am still in the process of writing - it remains hopelessly incomplete, and it is a miracle of coincidence it manages to support things like gcc), I wrote my own graphics libraries, I wrote my own TrueType engine, hell, I even wrote my own Python (one of the more fruitful and fulfilling of my "NIH" projects).

A few years ago, though, I became distraught at the state of my OS. The project I had spent so long building to learn myself had attracted learners of its own, hoping to use it as a study resource. ToaruOS pivoted - the student had become the teacher. And I felt quite strongly that I was doing a disservice to the community with all of the horrible designs and hacks in my kernel. I didn't want to be a bad teacher - I didn't want to be the butt of jokes like Mr. Molloy has unfortunately become with his old tutorial.

So, my goal has never been to build an OS people sit down and use to browse Facebook, or run a webserver, or control the dashboard on their car, but I do still have a goal: to build something that others can learn from.

If I had to write something, if should work on Linux the same way as on Tilck.

I share some of your idealism in this regard. Like you, I also put a focus on compatibility - I have tried quite hard to build things in ways that mimic "real" OSes, both so that I could better understand how "they" did things and so that, now, a decade later, I have a complete ecosystem where every interaction, every codepath, is simple to understand while still being a viable simulacrum of those "real" OSes. I do not like to build things just for my OS, which is why... I don't. My text editor isn't just part of my OS, it runs on Linux and macOS and even on Windows[* unmerged branch]. It's also not just a toy to check a box, I use my text editor as my daily driver - I write production code for my employer with it. That Python implementation I mentioned earlier also serves a useful function as it powers the syntax highlighting engine and configuration for the editor.

[Demindiro]

Therefore, my question is: how many of you are investing effort on developing something not-just-for-fun?

My motivation for making my own OS is that I believe that:

• The security model on modern OSes is fundamentally broken. Filesystems, network access, process listings etc are accessible by default. Restricting access is not trivial (with current tools, IMO) and/or inconvenient. Messing it up is usually an immediate security issue.
  I believe a more sane model restricts everything by default and only allows access to things by explicitly opting in. If you mess up it is much more likely that an application simply won't work properly instead of being a hole.
• It's too hard to make use of multiple, separate computers at the same time. There is a lot of compute power but a very large chunk of it is sitting idle. While there are tools such as distcc they are usually restricted to specific applications + I can't be arsed to set up a tool for every single application where I want/need it.
• Simulating devices is also too hard. I still haven't figured out how to fake a webcam that I can use with Discord to Rickroll people.

I do hope to eventually use my OS for serious things and perhaps my main driver, though it will take a long time.

I mean, do you have a long-term plan to make your project usable in the real world by the industry or the home user, even if it will have only a tiny chance of getting it there?

I don't have a strict roadmap because I still don't have a full picture of everything that a complete OS requires, but the long term plan is roughly:

• Make an OS that can be used for something. I achieved this when I was able to host a website with it, even if the network driver crashes from time to time.
• Make it useful for note-taking. This sounds simple but still involves a lot of work, from creating a GUI to supporting USB so I can actually transfer the notes to somewhere.
• Port third-party apps. Especially, port the Rust compiler and maybe a web browser too.
• Make it a distributed OS. Specifically, make distributing processes transparent.
• Clean up all APIs, fix glaring bugs and make an announcement on forums other than this one.

The development I split up in versions, where for each version I make a roadmap of planned features. This helps me with not wandering too far off track.

how do you plan to steal some of their market

Supporting their apps is going to be key for this. Porting all those apps is going to be far too much effort though, so instead I'm going to focus on emulating Linux's syscalls. It's probably more effort than porting a libc but I will then be able to support Linux programs without requiring them to be recompiled, which I think will be better for UX.

[AndrewAPrice]

My motivation is for fun. I want to challenge myself and have a sense of accomplishment.

I have a desire to make my build system/APIs cross platform so most end user programs could run on Windows/Mac/IOS/Android and my system. That way, if I build a cool game that runs on my OS, I can also share it on other platforms.

[Schol-R-LEA]

The security model on modern OSes is fundamentally broken. Filesystems, network access, process listings etc are accessible by default. Restricting access is not trivial (with current tools, IMO) and/or inconvenient. Messing it up is usually an immediate security issue.
I believe a more sane model restricts everything by default and only allows access to things by explicitly opting in. If you mess up it is much more likely that an application simply won't work properly instead of being a hole.

The most recent XKCD strip seems relevant here:


The weaknesses of modern security are more or less by design, because users - especially business users - value convenience over security. No matter how severe the problem may become, security simply doesn't sell.

[rdos]

My OS project started out for fun & a learning project (although it was motivated by the horrible design of MS-DOS). Over time, and because of luck, it's now running at thousands of installations, and used in a commercial environment. At this point most of the time I put into it is payed, and so I do very little as a hobby. About the only things I still do for fun is a few home automation projects and my spectrum analysator. So, I think the fun is still there, and as a professional programmer, I don't have to code for Windows or Linux, which is a big plus.

[vvaltchev]

Sorry for the late answer, but I was very sick. I'm not fully recovered yet.

Consider that the active participants in the community are, by and large, hobbyists. The people working on "serious" OS projects are grad students at universities working on research teams, or engineers at companies like Google, and neither group spends much time, if any, interacting with us. On top of that, I would guess that neither group is inclined to brag or share updates on a web forum - that's what papers are for.

Fair point.

I take this, and your earlier comment about text editors, as a slight against me specifically,

Not at all, man. I'm sorry that you felt somehow "attacked". I have nothing against your projects: I actually believe they're super-cool. It was a child-dream of mine to "re-write everything from scratch", including the kernel, a compiler, a desktop environment, a text editor etc. Unfortunately, I'll never have time for that, because that will not bring me an income to sustain my family. I've actually often wondered how could you (and some other people) afford to spend so much time working on your projects. IMHO, with a full-time job, that's hard or impossible to do.

Btw, if you really wanna know why I mentioned text editors and "colorful windows", I was thinking at all the projects with particular focus on having a GUI as early as possible, even when the "under-the-hood" software stack was not ready for that yet. I hope you won't ask me to point towards particular examples. Also, I was thinking about all the projects (including yours), that aim at re-writing the whole software stack. While that's cool and funny to do, it's not particularly useful, IMHO (see below)

and I want you to know that you have set up a strawman by declaring that "re-writing the whole software stack" is the antithesis to being useful.

Correct. I believe that re-writing the whole software stack is the antithesis to being useful. I could better explain that, but I'm not sure I have the physical strength to do it exhaustively today, in this post. But briefly, the open source community (and a few corporations) spent hundreds of thousands of man-years to develop the current software stack, after an insane amount of try-and-error. In other words, in particular open source projects, evolved naturally in decades to get to the current point. There is always room for improvement, of course, but re-writing everything from scratch is like re-inventing the wheel. Unless a group of people with deep experience in each component/layer has concrete ideas of how to improve the current state of the art, re-writing something would be just a fun exercise. And yes, I recognize improving the state of the art is insanely complex. Therefore, I aim myself at creating something that is not better (than the state of the art OSs) in the absolute scale, but has some advantages because it's simpler, smaller and better tested.

I also think you greatly exaggerate the binary partitioning of OSdevers in the community at large.

That's possible. Please, help me understand better the community.

And.. are you sure that your should be writing a kernel for achieving that, instead of just consuming Linux as kernel and making its GUI more powerful and user-friendly?

My OS has been noted for its graphical environment, and this is no accident - I did contribute to making Linux GUIs better: I worked on one of the foremost X11 compositors before starting my own OS. I also spent some time at Apple working on macOS's window server. So, my desire for my OS to have colorful windows with fancy effects is supported by a professional background in that area. I did not set out to write a GUI from scratch, but it was a natural step for someone with that background.

Sure, that makes perfect sense. As I hope you believe now, I didn't have specifically your OS in mind.

So, my goal has never been to build an OS people sit down and use to browse Facebook, or run a webserver, or control the dashboard on their car, but I do still have a goal: to build something that others can learn from.

You don't really need to defend yourself. There is absolutely nothing wrong in any of that. Writing something for fun or for educational purposes is great. I never wanted to attack people doing that. I've been doing the same thing for most of my life. I just wanted to see how many people have goals that go beyond fun/educational, because I noticed that at least in this community, the "just-for-fun projects" seemed to be the most successful ones while in other open source communities, I believe that the projects that solve the most painful practical problem are typically more successful.


P.S. I know that I didn't answer point by point on everything, but my head hurts like hell, when I'm in front of a monitor.

[thewrongchristian]

My OS project started out for fun & a learning project (although it was motivated by the horrible design of MS-DOS). Over time, and because of luck, it's now running at thousands of installations, and used in a commercial environment.

Just curious, what was the catalyst for that? Is it a story you can share?

[klange]

and I want you to know that you have set up a strawman by declaring that "re-writing the whole software stack" is the antithesis to being useful.

Correct. I believe that re-writing the whole software stack is the antithesis to being useful.

When I say that you have set up a strawman, I mean to say that you have taken it as a concluded fact that a project that aims to rewrite the software stack must not be useful, which you then use to support your claim that you are somehow alone in the OSdev community in trying to build something useful because you have automatically excluded any project with that aim. Not only do I think this view is wrong, I would point out that most commercial, proprietary systems - all of which seem to be quite serious and aimed at usefulness and not at all fun - are entirely in-house affairs, and have not only "rewritten" the entire software stack, they tend to invent entirely new interfaces (often without any regard to whether those interfaces are any better than existing ones).

Therefore, I aim myself at creating something that is not better (than the state of the art OSs) in the absolute scale, but has some advantages because it's simpler, smaller and better tested.

I will reiterate what I said in my previous post: You and I share a very similar goal. Simpler and smaller is my modus operandi as well. Rewriting the entire stack is not in opposition to the goal of being useful - for me, it is the only way to achieve that goal. I want the entire ecosystem to be present under the same license, written in the same style, following the same architectural design. How useful would an encyclopedia be if you turned the page to an article on a common subject and it merely said "this topic has already been written about quite thoroughly, go read another book"?

I also think you greatly exaggerate the binary partitioning of OSdevers in the community at large.

That's possible. Please, help me understand better the community.
[...]I just wanted to see how many people have goals that go beyond fun/educational[...]

Fun and education very rarely find common ground. I play video games for fun. I ride my motorcycle for fun. I do not write my operating system for fun.

[vvaltchev]

When I say that you have set up a strawman, I mean to say that you have taken it as a concluded fact that a project that aims to rewrite the software stack must not be useful, which you then use to support your claim that you are somehow alone in the OSdev community in trying to build something useful because you have automatically excluded any project with that aim. Not only do I think this view is wrong, I would point out that most commercial, proprietary systems - all of which seem to be quite serious and aimed at usefulness and not at all fun - are entirely in-house affairs, and have not only "rewritten" the entire software stack, they tend to invent entirely new interfaces (often without any regard to whether those interfaces are any better than existing ones).

I will reiterate what I said in my previous post: You and I share a very similar goal. Simpler and smaller is my modus operandi as well. Rewriting the entire stack is not in opposition to the goal of being useful - for me, it is the only way to achieve that goal. I want the entire ecosystem to be present under the same license, written in the same style, following the same architectural design. How useful would an encyclopedia be if you turned the page to an article on a common subject and it merely said "this topic has already been written about quite thoroughly, go read another book"?

OK, first of all, fortunately, I'm not alone in this. I was happy to see here other people with concrete goals. I just have yet to answer to each one of them. But I saw people focusing on built-in security, stability, small-size etc.

After that, about the "re-write the whole software stack" topic, I believe there has been a misunderstanding: I'm not opposing doing that on principle, I'm in reality arguing that it's unfeasible and unrealistic in 99.9% of the cases. Despite that, I like a lot the whole idea of the "integrated ecosystem" and probably will spend some time playing with SerenityOS, as soon as I have some extra time and feel better.

I'm simply arguing that, realistically, small groups of people (like OS devs here) have realistic chances to improve (e.g. re-write) just a small piece of the whole software stack, if the quality level has to be on-par with the state of the art. Sure, it will be possible to re-write more than a small piece of the stack, but the quality will suffer from that. Let me make a concrete example: just writing an SMP fair scheduler with support for NUMA nodes etc. will require years and years of work for a single person. For the memory management, it's the same. For drivers, same story (imagine supporting USB 3.2 etc.). Writing a web-browser from scratch, it's even worse: getting on par with Google Chrome will require thousands of man-years to do. Indeed, not even Apple itself is able to keep up with Chrome. Safari is constantly behind Chrome in terms of features. Microsoft gave up at the end on their own engine, and decided to reuse the same engine used by Chromium.

If today a group of 10 talented guys decide to re-write everything, how long it would take? Even if they are 10x more talented than the average Google developer, they won't get too far, simply because the scope of the task is too broad. If instead, they decided that's OK to skip most of the features of the big projects just to have something working then, sure, they'll be able to have a kernel, a GUI, a browser etc. but the quality and the features won't be on par with the state of the art. It's as simple as that.

Note: I'm making the crazy assumption that those super high-end developers will work for free 40-50 hours/week on their projects. Typically, that's not the case. Very few people can afford the luxury of not having a full-time job, in particular when you calculate-in the opportunity cost.

So, in conclusion, realistically, it makes sense for small communities to invest in re-writing a small piece of the stack (no matter which), in order to make it competitive with what already exists, given a reasonable amount of effort.

[vvaltchev]

My OS project started out for fun & a learning project (although it was motivated by the horrible design of MS-DOS). Over time, and because of luck, it's now running at thousands of installations, and used in a commercial environment. At this point most of the time I put into it is payed, and so I do very little as a hobby. About the only things I still do for fun is a few home automation projects and my spectrum analysator. So, I think the fun is still there, and as a professional programmer, I don't have to code for Windows or Linux, which is a big plus.

That's a fantastic success story, thanks for sharing! Writing an OS from scratch and getting to the point where it's used for commercial purposes must be amazing.

[klange]

After that, about the "re-write the whole software stack" topic, I believe there has been a misunderstanding: I'm not opposing doing that on principle, I'm in reality arguing that it's unfeasible and unrealistic in 99.9% of the cases. Despite that, I like a lot the whole idea of the "integrated ecosystem" and probably will spend some time playing with SerenityOS, as soon as I have some extra time and feel better.

Wow, that's insulting. You tell me I'm unfeasible and unrealistic, and then you say you're interested in another project with my goals instead of mine?

It was a child-dream of mine to "re-write everything from scratch", including the kernel, a compiler, a desktop environment, a text editor etc. Unfortunately, I'll never have time for that, [...]
[...]
So, in conclusion, realistically, it makes sense for small communities to invest in re-writing a small piece of the stack (no matter which), in order to make it competitive with what already exists, given a reasonable amount of effort.

I'm sorry you gave up on your dreams, but that's not realism, that's depression.