Rewrite from Scratch

[Combuster]

I will set up a website for this project. I want to own all the rights to the system but I prefer to show the code. Are there any legal problems if I just put "Copyright [...] All Rights Reserved." and just show the code? Is it still proprietary so that I can do whatever I want with it? It goes without saying that I do not use any GPL (or similar) code in it.

"All rights reserved" is legally sufficient - you used your own right to show the code on your website, which means that copying or distribution by others is still prohibited. People are however not lawyers and you might want to use a redirect based on a cookie to make it blatantly obvious that the availability of the code is only for such educational purposes.

[Jezze]

I'm questioning the reason to keeping the code proprietary especially when it comes to an operating system. I think the chance of someone paying for it, if that is the reason, is one in a million. That ship sailed a long time ago.

[Combuster]

That particular market saturation problem holds if you're primarily targeting a desktop, tablet or a cell phone and the like. There are a fair share of niches and examples of commercial OS successes even among the people of this forum.

[AndrewAPrice]

This problem tends to arise when you follow the Code and Fix methodology without having a good design. Your entire code was designed to have very limited functionality at the beginning, and all other functionality was implemented as quick fixes on top of it and now you have a huge unmaintainable spaghetti-code monstrosity.

Rewrites can waste a lot of time. It's not bad if it's 5-10 years from when you started your project and you need to seriously change the architecture because your requirements have changed over time. Rewriting your code willy-nilly just because you think you could use objects, clean up some variable names, etc is just silly and you'll waste a lot of time (and you'll have to go back and redo all those months/years of testing your code).

I recommend studying software architecture. There's plenty of resources online to teach you how design software architecture for large projects. For anything above a simple UNIX utility (ld, grep, more, etc), I recommend starting off with a good architecture and good design.

But good architecture all comes down to having a design document. Writing one can be tedious, but it's quite easy and could be fun once you get going. Lay out the requirements (what you want it to do), then lay out some general ideas of how the architect/code will be laid out. Try to design as much as possible, and go crazy with as many features as you can. This does not mean you have to implement all of it at once (I recommend against that) - split your implementation plan into several 'phases'. e.g.

- Phase 1 - Get it booting, memory initialization.
- Phase 2 - Loadable drivers, programs, IPC
- Phase 3 - Network stack
And so forth.

The biggest secret to being productive and writing good quality code is to code the interfaces first! In an object oriented language, this means declaring your classes and public members. In C or assembly, it means writing your exported function prototypes. This is the stage when you can find the most flaws in your design, and gets you thinking at a higher level, than individual logic. Do not write any code inside of these functions! If your language makes you (e.g. it won't compile unless you define the body), do something simple like:

Code: Select all

void *init_network_stack() {
   NOT_IMPLEMENTED();
   return 0;
}

Then implementation becomes simple. You know exactly what each function does, so when the time comes you have a pretty good idea of what code to put in there. Just implement the functions required for phase 1 (either comment out any references to phase 2 functions or handle them returning null/0). Once you test these functions you implemented you can release phase 1. Repeat these steps for any additional phases.

There is also no problem with refactoring. It's true that when you're playing with something new (your first kernel, a new framework, a new language) you don't really know enough about what you're dealing with to start off with a great architecture. In real life, I've seen design documents change during implementation, and this usually means refactoring your code.

If I were you, I'd write myself a design document with what I have learnt, take the code I already, and refactor it in stages towards your design. I'd probably get myself a good IDE/editor that allows you to refactor variables/functions and update all references to it.

[Brendan]

Hi,

That particular market saturation problem holds if you're primarily targeting a desktop, tablet or a cell phone and the like. There are a fair share of niches and examples of commercial OS successes even among the people of this forum.

In general (for all significant pieces of software, including OSs), users won't change from whatever they currently use to something else unless the benefits of changing outweigh the hassle.

For a normal desktop user who's always used Windows and has a collection of applications/games for Windows, the hassle of changing (getting used to a different OS and not being able to run the applications/games they've paid for) is large, so the benefits of changing need to also be large. Sadly, (from the user's perspective) Windows is relatively good, and it's extremely difficult to make an OS that is so much better than Windows that people will switch.

If you can find a niche where the hassle of changing is typically much smaller (e.g. embedded systems, special purpose servers) and the software people currently use can be improved on a lot easier, then it becomes a lot easier to convince people that the benefits of changing outweigh the hassle. For an example, a company that makes systems for Network Attached Storage might be using a cut-down version of Linux, and the hassle of switching to something else might be very small, and it might not be that hard for a new/different OS to provide sufficient benefits (e.g. easier to setup/install, leaner, tighter security, etc).

This problem tends to arise when you follow the Code and Fix methodology without having a good design.

While I agree in principle; there's a "catch 22" you're missing.

To create a good design you need experience in that area. For example, if I had to design and implement an accountancy package for Windows in C++, my first attempt would probably be very bad because I've never written software for accountancy before (I lack domain knowledge), I've never written an application for Windows before, and I have very little experience with C++. Of course the most obvious way to gain the experience I lack in all of these areas would be to attempt to design and implement an accountancy package for Windows in C++.

Basically, you end up with a cycle: design and implement (and gain experience); then retry (with the benefit of that experience).

If you don't gain enough experience during an iteration of this loop (and if the scope didn't change), then the design and implementation of the next iteration will be very similar and trying again won't be justified. Basically, the loop ends when you've failed to learn anything new.

Now; in general (for all significant pieces of software, including OSs), users won't change from whatever they currently use to something else unless the benefits of changing outweigh the hassle.

If your "design, implement, then retry" loop ends before you've gained all the experience needed to create something that is better than whatever people currently use; then you're screwed. What this mostly means is that the loop becomes something like this:

Code: Select all

    experience = ?;
    do {
        experience += design();
        if(design_changes > some_threshold) {
            experience += implement_from_scratch();
        } else {
            experience += modify_existing_code();
        }
    } while(experience < experience_required_to_surpass_competition);

Of course most people here are only gaining experience for fun, and aren't attempting to write an OS that people will want to use for something one day. In this case you can stop whenever you like (and most people seem stop after only a few iterations of the "design, implement, then retry" loop). If someone was trying to develop an OS that is significantly better than Windows for normal desktop users, then they'd probably need at least 10 iterations of the "design, implement, then retry" loop just to have a fraction of a hope.

Also note that I am simplifying this a lot - it's more like many loops (e.g. one for boot code, one for kernel, one for GUI, etc) each with it's own "experience" variable; where some fool has gone and inserted "if(rand() < foo) goto" statements that can cause you to (for example) jump from the middle of the "designing a networking stack" loop all the way back to the "designing a kernel" loop, and the "experience_required" for each of these loops keeps increasing (because the competition is improving too).


Cheers,

Brendan

[Antti]

This licensing issue gained too much attention. The license may change but I start with "All rights reserved."

I have realistic goals. I do know how to do basic steps for the first milestone because I have a reference implementation (my spaghetti-coded* OS). Now I just do it much better. Of course there are a lot of new features also. You have not seen new code yet so it is hard to say whether "Antti is capable of doing any OSs" or not.

Currently, I am quite helpless because I cannot show anything. I do not have anything to show. By showing my spaghetti-coded* OS (I started writing it without any OS development experience) I would crash my plausibility. The first impression is so important that I do not want to fail that. I do not expect the new OS to be more interesting than other OSs seen here. However, there probably are few people who will take a look at it. Just to see what it is all about. After this rewriting from scratch nonsense and all the other nonsenses I have posted.


*(It is not so bad but the rewritten one will be so much better)

[Kevin]

Currently, I am quite helpless because I cannot show anything. I do not have anything to show. By showing my spaghetti-coded* OS (I started writing it without any OS development experience) I would crash my plausibility. The first impression is so important that I do not want to fail that.

I wouldn't worry about that: Those people about whose opinion you should actually care have seen (and written) enough bad code, in varying degrees, that they should know which conclusions from the code of a first attempt at an OS are valid and which aren't. Maybe the only difficulty for you is to decide who is part of this group and who's not.

But then, why does it even make you helpless when you don't have anything to show? I think I've never shown the code of my private OS to anyone, and I don't feel as if I should. (And people already know how good or bad my code is because I've messed up often enough with tyndur.)

[Griwes]

@ "All rights reserved" notice: nowadays, it's redundant and implicit. From Wikipedia:

The requirement to add this notice became obsolete and essentially deprecated on August 23, 2000 when Nicaragua became the final member of the Buenos Aires Convention to also become a signatory to the Berne Convention. As of that date, every country that was a member of the Buenos Aires Convention (which is the only copyright treaty requiring this notice to be used) was also a member of Berne, which requires protection be granted without any formality of notice of copyright.

[bluemoon]

As Combuster said, not everyone a Lawyer, so you may still want to explicity state so to avoid misunderstanding.

And by the way, it's still arguable that if the source is opened, and it may be reasonable expectation in some country that some usage is granted (ie. including the right to read the material, thus not "all right reserved").

[Griwes]

Well, that's why we have ways to enforce copyrights, right? Ignorantia legis neminem excusat

[~]

You are basically talking about finding out that every rewrite is much more tedious than the previous one.

It took me a lot of time (my first 8 OS development years, from 2005 to this now ending year 2012) to find out how to solve that problem:

• It is usually an error trying to write any program (including a kernel) with features I currently don't need. It will lead to a very poor implementation. It is better to leave pride aside a little bit and start on things I truly need to learn and implement. For instance, right now I still have to learn a lot from the programs at Programmer's Heaven (things like graphics, math, and other miscellaneous things), and I also just need to load small programs from floppy and at most, from hard disks with FAT32. I also need PS/2 support for keyboard and mouse, and basic VGA modes, and VESA modes as well. I need mainly 32-bit code support.
  
  If I currently don't REALLY need multi-core, 64-bit support, network, paging/virtual memory or GPUs because I just haven't spontaneously experienced those problems and because I haven't implemented an easy-to-understand and easy-to-scalate code along with its very well defined and documented logistics, I will not only lose my time, but I won't understand what I am doing because they aren't real problems for me currently. And then an unnecessary rewrite will take place, when I could have just made a simple applications as a test for a future subsystem, instead of rewriting everything and using more effort than was really required.
  
  It is better to first come up with an excellent implemented solution for the problems I really know, understand or can relate to very well and are part of my life (if I'm an individual developer instead of a developer group), and find a way to get to understand and get to know the problems I want also being able to handle.
  
  At least as for me, the fact that practicing all of the above allowed me to create a minimal, yet extensible kernel, and more importantly, reusable explanations for the fundamental algorithms and the methodologies and tricks used for future code implementations (no matter how different) shows that it is more important to concentrate exclusively in solving my real problems.
  
  It is either that, or trying to leave those things abandoned while trying to achieve things I'm just not experiencing.
  
• To break the pressure and the tedious situation of rewriting everything year after year, it is necessary to read and do things I haven't done or did poorly in the past.
  
  It is important to give very small steps, and these steps must be reusable.
  
  One must be careful to not allow that the ratio of productivity/implemented stuff drops too much after studying; nor allowing the ratio of newly-learned things drop too much after implementing things to the point where they are starting to become sketchy and a bare attempt that barely works, if it even does well. And so on.
  
  It's also important to do things that are currently interesting, have numerous applications and allow me to process other things in the background. When returning to those, the mind will usually be refreshed an much more lucid than not allowing it to take a rest from such things. I have left many or all aspects of OS development for several weeks or months many times, and when I return, I find to my surprise that they have suddenly become much easier to understand before taking such a break. It might be counter-intuitive but it is true (allowing the mind to physically recover after a heavy cycle makes sense).
  
• I find that it is always good to talk about these things with programmers who aren't specifically in it. It's good from the point of view to psychologically regard past topics as "mastered" and "done", as if now everyone understand them; although it very probably will upset some of them, in which case it is better not to pay them too much attention, to avoid losing time.
  
• Things get much more interesting when combining OS-development-grade techniques in other types of programming. For instance, I managed to write a kernel that only has a console, FAT12 reading and simple program loading. Not very interesting. But to make it more educative, I am also building a PC emulator (or at least trying). It is allowing me to know much better the x86 architecture, and implement it myself, so I can know if what I am supposing is right or not (I study the manuals, then implement and then I see if things are coherent).

______________________________________
______________________________________
______________________________________
______________________________________

There surely is a huge gap between your knowledge level and mine. However, I am climbing up everyday. I started programming (in general) in early 2010 and I started my OS project ten months ago. I already have a simple basic system that is not anyhow based on any "tutorial operating systems." Here is a brief list of "keywords" (just to give some idea):

My interest in OS development started in November 2004, seemingly the right time, when general interest in it was at its peak point. And I've always had computers around and certain "ease" to program. Although I might not have the best of all educations in Computer Science unlike many people here. I learned assembly in 6 months, and I learned a lot of things that I couldn't connect to create a kernel, but I had a lot of fun (keyboard, handling CD-ROMs, hard disks, timers, the screen...), but now, after 8 years of being mainly focused in OS development, I'm in a much better position to start building a minimal kernel (which I have done: a basic text console with basic built-in commands, floppy support for standard FAT12, timers, and ability to run programs with minimal structure).

My current programs are using system calls directly and there is no any concept of shared libraries or dynamic library loading. I have not yet reached "the level" of porting the C library and start building a "POSIX-system-like user space". I mean the system that resembles Linux-Unix-bash-shell-something system (although this is not an accurate description nor technical, I hope you understand what I mean). Actually I am wondering whether or not I want to do this at all. Nevertheless, then I would notice how inefficient my system is when compared to the real ones.

To give an idea, I can currently run programs in physical space (no paging, no virtual addresses). You could easily have some sort of table of system calls with many empty slots, and you could, instead of loading a program, reserve a physical block of memory and the system call slot, to have very basic "dynamic loading". It might not be useable in a "final" system, but it will allow and will actually make it possible to have a starting point to create a true dynamic loading, specially if you have absolutely no previous experience on that.

Three years for the kernel/s and basics does not sound very bad at all. It is quite plausible. Is the kernel meant to be highly portable? I am just questioning the long gap (5 + 7 years) before the first public release. Of course there must be non-public releases for testing purposes? If it took so long to release the system publicly, I think the kernel should be considered being "just an extremely good design and stable API." I am not sure whether the kernel implementation details are very important when looking at the whole big picture. Of course it is quite difficult to forecast the future. It may be that the current hardware is still current in the future.

The language/toolchain/IDE would probably be the most interesting part of the project. It may be the way to do the next big thing. It could "live forever" if it were successful. The hardware evolution does not affect to it.

Three years for an useable kernel would mean that it was a team work or a full time activity for a very experienced individual. In either case, if it has all the basics (booting, memory, processes, GUI, mass storage, network), it almost always means that libraries were used; or if not, it means that the team or individual is truly top-notch and has exceptional abilities and/or complete knowledge.

Last few days have been terrible. I started rewriting the system from scratch. There is not much code yet. Mainly just stubs that are used for testing the build system. At first there are two targets: x86-32 and x86-64. I try to do things so that it would be easy to add more targets in the future. Portability and simplicity are design goals. Efficiency is not the primary concern.

I will set up a website for this project. I want to own all the rights to the system but I prefer to show the code. Are there any legal problems if I just put "Copyright [...] All Rights Reserved." and just show the code? Is it still proprietary so that I can do whatever I want with it? It goes without saying that I do not use any GPL (or similar) code in it.

In my first 8 years of attempts at OS development, I have tried almost everything you have said that you are doing: Trying to modularize the code and make it automatically portable, many full rewrites (10 major but that actually represent very little new knowledge that I could have acquired in less than a year if somebody had given me a structured and formal approach of learning and implementing).

For me, it's really about "intent" - e.g. if you intend for the OS to become a usable product that millions of people will use daily (regardless of whether anyone ever does use it).

That is the right attitude. These kind of projects are interesting to watch. In contrast to "just a hobby project that is discouraged (by the author) not to (even accidentally) gain attention."

In my case, my attitude is that of trying to make possible that understanding currently difficult things, becomes considerably easier to understand. Many things are difficult because they aren't publicly explained in a complete way. I don't know how far I can get, but I'm not worried.

Among other things, not even the fact of being essentially currently alone in my projects, having a deficient education on Programming/Computer Science (which I'm working with the best of my abilities to overturn), or seeing how small the general interest in such advanced topics is even among other "good" programmers, are even near enough to discourage me.

These 8 years of trying without stopping, and rewriting things, or for instance, just not returning ever again to this forum (among other things), demonstrate that when I say that I won't give up, I'm serious. I'm here after all, and I'm happy to see that I've been able to help people around here once or twice, at least in very small things.

I've had many chances to get discouraged and stop trying to learn more about OS development, but I haven't and don't even imagine suspend my attempts.

Of course I want to have my OS done in as short time as possible, but I also want to leave something that goes beyond it, so it can be used globally in anything other than OS development, and make it beneficial for others (that's why just like the OSDev Wiki, everything I program and document, that in in the Internet, is in the Public Domain / Creative Commons 0 1.0 Universal, so it has no limitations and can survive as long as it is something really valuable for even a small group of productive people).

I want that what I do can be free from all copyrights (but that can be reused in an individual basis even in copyrighted works). I want being able to help people who understand less than I.

+++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++

Now for me to also be productive and trying to demonstrate what I'm talking about, please allow me to gradually open new threads over time to post what I have managed to do and answer whatever I can (correctly). In this way I will do what I said, of not allowing my theoretical work level make drop too much my ratio of implemented achievements.

I hope it helps. If you have been iterating since 2010, then what I have done should be useful since I have been in that situation since 2004/2005 and still going, somehow as "bored" as you for rewriting things, but being much less bored because of remembering and now knowing that it's not a waste of time; just a demonstration of how many steps (iterations) are required to achieve the desired goals given their complexity, and learning and producing reusable documentation, code snippets, and algorithm descriptions constantly. In this way, everything I achieve will be the new starting point; it will no longer be from scratch but every time the starting point will be higher, and will have more quality.

It should probably serve as some sort of example as to what to expect to get done in the near future.

Keep in mind that you probably are twice or three times above my ability and knowledge if you have mastered things like the GNU toolchain, C and C++, memory management, USB, multicore support, parsers, text consoles, GUIs even if basic, data compression, sorting, image coding/decoding, Base64 et al, audio formats, and miscellaneous yet applied computer math, and all of the things that are in the Wiki (as well as indirectly related subjects such as full hardware, software and protocol specifications), to the point that you have implemented in applications or in your kernel.

[Antti]

I can give you my first source code file. This is all where it begun. I had no previous OS developing experience. It was still in use before I started this "rewriting from scratch". This file is public domain now. It contains all beginners mistatakes, hard-coded values, bad modularity, etc. This can be used as an example of bad design.

[Brendan]

Hi,

Just a "quick" note. I was planning to start the next version of my OS after my last exam, which was today. I actually started it a little early. So far I've rewritten my "build utility", imported/examined/updated about 5 different specifications, written a bunch of "include" files for generating various files (to comply with my file format specifications), imported/updated a "make floppy image" utility, and started a floppy boot loader.

What motivates me is trying to improve on the last version. For example, one of those specifications is my "Native File Format" specification; which defines the generic structure of all native file formats for my OS. When this got imported I tried to find ways to improve it and (based on suggestions from other people) ended up adding support for metadata and subfiles. What this means is that for the next version of my OS, it will be possible for a simple utility to examine any (native) file and obtain (and/or add) a file's metadata without knowing or caring what type of file it is; and (for an example) a GUI would be able to attach a small icon/thumbnail to any file without knowing or caring what type of file it is (and things like word-processor documents can have pictures, movies, sounds, spreadsheets, etc "embedded" in them in a standardised way, where a generic tool can find/extract these embedded files). No other OS (that I know of) is capable of doing these things.

For another example; my old "get physical memory map" code for PC BIOS was already better than anything I've seen anywhere else; but I found a way of generating much better errors/warnings. If the BIOS reports anything slightly dodgy or inconsistent the user gets a nice/clean report of exactly what the problem is and what caused it. No other OS (that I know of) is capable of doing this either (and most wouldn't even notice that the information the rest of the OS will rely on was messed up(!) ).

Of course I currently don't have any kernel or anything else, and only have a small part of the code I need to boot from floppy. Does this worry me? No - I can imagine what the newest version will become, and if I continue improving on the previous version/s and adding more features I think I'll end up with something truly amazing.

Currently, I am quite helpless because I cannot show anything. I do not have anything to show.

It's normal to see other people's projects and be a little disappointed when you compare them to your own (especially when you see awesome screenshots like this). However, it is important to realise that these people will all start again themselves sooner or later (and when they do you'll be ahead of them!).


Cheers,

Brendan

[rdos]

It's normal to see other people's projects and be a little disappointed when you compare them to your own (especially when you see awesome screenshots like this). However, it is important to realise that these people will all start again themselves sooner or later (and when they do you'll be ahead of them!).

No way. You will never get ahead of me because I start from scratch.

[Kevin]

Three years for an useable kernel would mean that it was a team work or a full time activity for a very experienced individual. In either case, if it has all the basics (booting, memory, processes, GUI, mass storage, network), it almost always means that libraries were used; or if not, it means that the team or individual is truly top-notch and has exceptional abilities and/or complete knowledge.

They are talking about a microkernel and userland basics. Basics most definitely do not include network or a GUI. Mass storage may or may not be considered part of them. In other words, what they are talking about is a kernel which, if you didn't want anything fancy but just get it to work with a hacked up design, could be written in like two or three weekends if you have some experience. If you want to do it right, and maybe add some interesting non-standard features (though such opportunities are limited with a microkernel), it could take quite a while longer. But three years is plenty of time for it.

It's normal to see other people's projects and be a little disappointed when you compare them to your own (especially when you see awesome screenshots like this). However, it is important to realise that these people will all start again themselves sooner or later (and when they do you'll be ahead of them!).

I won't fall into the rewrite from scratch trap any more. I've learnt that lesson. But then, my screenshots aren't quite as awesome.