Rewrite from Scratch

[Antti]

I have now started to realize that my code is not perfect (what a surprise). I was very inexperienced when I started my OS project and now I know much better how things should be done. Even now I am not an expert but I probably understand much better some important principles that I totally ignored in the past.

Now it might be the time for a rewrite from scratch. This rewritten OS would work exactly like the old one. I do not mean to change the actual design (I am happy with it). Why to do that rewrite at all?

The main advantage is that I know what to code. I could handle race conditions (the biggest problem) everywhere from the very beginning. It is quite tedious to do afterwards (I am doing it currently). I could also fully concentrate on having strict naming conventions, comments, and uniform coding style in general. There are a lot of little things that would be done better.

The main disadvantage is that I am not very excited to start this. It does not sound so interesting to do the same thing again.

However, it might be that I have to do this sooner or later anyway. On the other hand, it would be better if I waited more because then I would be even more experienced. This also means that I should wait forever. When is the right time? Do you have had the same kind of thoughts?

[bluemoon]

If you decide to rewrite your kernel, I suggest to do it early, and only at early phases.

As you realized, a rewrite from scratch clean things up; it benefit much at early stage of development, where designs are unclear and filled with hacks.
However, after 2~3 rewrite you more of less get a stable design and much cleaner code, and the need for a rewrite is reduced.

If you do a rewrite on mid-term scale, you will suffer from http://en.wikipedia.org/wiki/Second-system_effect.

The extreme case of a rewrite is major version change, at which like version 10.0 after a decade might be a complete rewrite...

[Antti]

It might also be reasonable to continue refactoring the current codebase.

Bad Reasons to Re-write Code from Scratch

- It's really ugly.
- We don't like the coding style.
- It's not done the way we would have done it.

It seems that I am doing the classical mistake. Of course this is not so simple...

[Antti]

One thing came to my mind: doing the rewrite by using some different programming language. That would probably solve the "doing the same thing again" psychological problem. However, there are not many choices* (currently Assembly + C).


*(Flame Warning, try to avoid the trap)

[cxzuk]

However, it might be that I have to do this sooner or later anyway.

I think some good analysis of your code to find out if you will have to "sooner or later" would help you greatly.

I could handle race conditions (the biggest problem) everywhere from the very beginning. It is quite tedious to do afterwards

As tedious as a rewrite?

Mike

[AJ]

Hi,

One thing came to my mind: doing the rewrite by using some different programming language. That would probably solve the "doing the same thing again" psychological problem.

The potential problem with this is that you end up in more of a mess than before. Why did you choose the current languages you are using: presumably C because you are familiar with it and asm because you need it for some system stuff? Assuming C is your "comfortable" language, then by switching to an alternative, you end up trying to learn the language and program and OS at the same time.

Let's assume you are currently using C and plan on switching to, for the sake of argument, C++. You do one of two things - either face the problem above of trying to learn a new way of programming, or you end up writing, effectively, C code but compiling it with a C++ compiler

Cheers,
Adam

[Antti]

As tedious as a rewrite?

Not so tedious.

C code but compiling it with a C++ compiler

That is probably a very common problem when moving from C to C++.

I do not really intend to change the programming language. If I were to do that, the language should be totally different. However, learning it while writing an OS is not a good idea.

One of the biggest problem is the testing. I am not sure whether or not I would test the rewritten code so carefully than I did with the current code. I had time to test the code because the development proceeded quite slowly (I had to study things first). The rewriting process would be much faster. I have not used Test Driven Development or similar approach. That would be worth considering to.

[Kevin]

I do not really intend to change the programming language. If I were to do that, the language should be totally different. However, learning it while writing an OS is not a good idea.

Depends. Learning a new programming language paradigma while writing an OS might not be the best idea indeed, but switching the language within a familiar paradigma shouldn't be that much of a problem. (In fact, I only really learnt C while writing an OS)

However, I would be surprised if it made rewriting the whole thing any less frustrating. Once your project has reached a certain size, incremental changes are the way to go.

I have not used Test Driven Development or similar approach. That would be worth considering to.

I think it's hard to do TDD at least for some parts of a kernel. You can automate some testing with emulators, but you can't really for the problems that you only get with real hardware. You also need some working infrastructure before you can actually run tests and communicate results back.

That said, TDD has worked great for us for things like file system drivers.

[Jezze]

A rewrite sounds like a big undertaking. I would not recommend it. Try to compartmentalize* your code instead more rigorously* and replace the parts you dislike one at a time. I think even though you will do a rewrite you will get to this same exact point once again.

*= Thank god for the english spellchecked** =)

**= HAHA SPELLCHECKER

[rdos]

In considering a complete rewrite, I would mostly consider issues of stability.

If I had a stable OS, or almost stable OS, I wouldn't consider a complete rewrite as this would most likely produce a less stable OS. That's because you would be less likely to test the new code as you already know how to write it, and less testing usually means more bugs.

It would also depend on your toolset and way of testing. For instance, when I write new OS-code, I usually single-step it in a debugger to verify that it does what I expect it to do, and also testing various error conditions. If I did a rewrite, I'm sure this testing would be less rigorous as I would test less error conditions, and possibly single-step less code. OTOH, if the original OS was written without error-checking and debugging, it would make more sense to rewrite it and analyse it with debuggers as you rewrite it to make it more solid.

Another issue is with SMP, multitasking and synchronization. If you want to write a new OS with SMP and multitasking, and you haven't created some sensible synchronization primitives that could be used for SMP, and used those throughout your current OS, it is a clear case of a complete rewrite, as adding SMP or multitasking to a single-tasking design is a hopeless task that will add race-conditions that would be very hard to discover and fix.

[Antti]

Thank you for the replies. I just looked my codebase and I realized that there surely is a lot to do if doing the rewrite from scratch. It is best that I will focus on the current code and try to improve and refactor it. It actually does not look that bad. There is hope.

That's because you would be less likely to test the new code as you already know how to write it, and less testing usually means more bugs.

Yes. That is exactly the thing I am afraid of. If doing this as a professional project, this would not be an issue. Because I am doing this as a hobby, it is.

Another issue is with SMP, multitasking and synchronization.

I am now working on this. The global lock is used and it is very inefficient because it means that the kernel code cannot be interrupted. As an offtopic note: it was the best moment in my OS developing history so far when I got all the other CPUs (cores) jump up on a trampoline from XXXXXh to FFFFFFFFCXXXXXXXh. It just felt awesome to "see" the real parallel code execution.

[Brendan]

Hi,

I have now started to realize that my code is not perfect (what a surprise). I was very inexperienced when I started my OS project and now I know much better how things should be done. Even now I am not an expert but I probably understand much better some important principles that I totally ignored in the past.

That's normal. While writing "version n" you gain experience, knowledge and confidence. The extra confidence leads to scope creep which stretches the original design; and sooner or later you realise that you could use the extra experience and knowledge to create a better design for "version n+1".

If I had a stable OS, or almost stable OS, I wouldn't consider a complete rewrite as this would most likely produce a less stable OS. That's because you would be less likely to test the new code as you already know how to write it, and less testing usually means more bugs.

New code that you know how to write is less likely to have bugs than old code that you didn't know how to write; and if you don't rewrite then you'd still need to make changes to existing code (and still risk introducing new bugs with every change). In the end; a rewrite can easily produce a more stable OS (and a more maintainable OS too).

However, it might be that I have to do this sooner or later anyway. On the other hand, it would be better if I waited more because then I would be even more experienced. This also means that I should wait forever. When is the right time? Do you have had the same kind of thoughts?

In my opinion what matters is the design, not the code itself. If it's well designed (e.g. it's easy to change/improve one piece without messing up other pieces) then you won't gain much from starting from scratch. On the other hand, if the design is becoming a limitation (e.g. you can't add a feature without effecting many different pieces) then it'd be wrong not to change the design (and therefore change a lot of code), and starting from scratch is easier.

Of course starting from scratch isn't necessarily as hard as it sounds either - you can use existing code as a reference and in a lot of cases (where a piece of code wasn't effected by design changes) you just cut & paste old code while making minor changes. The main benefit of starting again is that it's easy to keep track of what has been updated and what hasn't; without having "can't update <foo> because of <bar>" problems that you'd have with an "in place, mark and sweep" approach.

I'd recommend writing down a good description of how you want the OS to be designed (including details like how various data structures work and how they're protected by locks, etc). Then compare the design (while ignoring the implementation) of the existing code to your description. If you find that the design is adequate then don't rewrite, even if the existing code isn't very good (just change/improve the implementation). If the design is not adequate then rewrite (even if the existing code is very good).


Cheers,

Brendan

[piranha]

My current OS is actually the third iteration of an OS that I wrote. For the second one I had to start over completely because in writing the first one I was still learning about everything and so I really had no idea what I was doing. For the third one I had to rewrite it because a large part of the system was fundamentally broken.

Each time it was worth it. But its only worth it if theres a problem that cannot be fixed by rewriting small sections of the code. The biggest indicator of if you should do it or not (for a hobby OS) is: Do you want to? Forget about advantages and disadvantages. If you're unhappy with the current code base, start over if you think it'll help. You'll learn a lot in doing so, and even if it fails you'll be able to apply the new knowledge to your old code.

That doesn't mean that it's a good solution - running rm -rf ./ isn't the way to go for fairly minor things like going through the code and fixing race conditions and adding synchronization. While they're tedious, it's probably easier to add them in than to rewrite everything. It's generally better to rewrite certain sections of code that you know works but could work better than it is to start over. If your current problem is fixing race conditions, then do that. If you finish that, and you still hate the code, rewrite it. It'll be better in the long run if you try to fix the problems than to start over.

-JL

[eryjus]

Now it might be the time for a rewrite from scratch.

I recommend you take care with your statement. It can lead to a lot of trouble. I could go through the thread and reiterate all the separate quotes cautioning the same, but there is no need

So, the code isn't perfect. So, the implementation details are not what you would do now that you have a little more experience. So, your coding style has evolved over time. So, imagine you take on this effort.... Would all these "deficiencies" magically go away? Or, would they get worse?

The design sounds solid (or, at least not the driving need for a rewrite). It's like saying that your don't like the color of your walls, so it's time to move.

Consider the following:
* You can clean up the coding style function by function
* You can clean up the naming conventions module by module
* You can add comments source file by source file
* You can take care of these items easily while you continue to develop, alternating tasks to keep from getting bored

If the need is really for a complete rewrite, then you need to make that decision objectively; if the need is for code clean-up, that is different than a rewrite. I recommend you maintain objectivity.

[Antti]

The main benefit of starting again is that it's easy to keep track of what has been updated and what hasn't

It looks like that I will not do this rewriting from scratch. However, I am probably going to do a very simple thing that has almost the same benefit you mentioned:

Code: Select all

1: Create an empty folder
2: Take a source code file from the current code base
3: jnotfound 7f
4: Check the source file very carefully
5: Copy it to the new folder
6: jmp 2b
7: ret

Of course it is not needed to have a new folder at all. I could have some tag for files that are checked. The details are not very important here. The main point is to systematically check all the source code files.

Consider the following:
* You can clean up the coding style function by function
* You can clean up the naming conventions module by module
* You can add comments source file by source file

Most of the things that bother me with the current code are quite insignificant. It really is better to do just the clean up. Of course the real challenge is made the kernel being "fully preemptible." However, that is a little bit technical detail here in this topic.