OSDev.org

I am a new student interesteng on designing an O.S, please give me a clue where to start either using C compiler or anything else.

Where to start? Difficult call. The first move, which you may or may not have done already, is to read up on the subject as much as possible. This means not only getting an thorough background in each of the languages you mean to use - assembly language at minimum, along with the HLL of your choice - but also the Intel architecture, certain details of the assembler and compiler(s) you're going to be using (e.g., how gcc passes function arguments), the details of the hardware interfaces, and the memory management mechanisms on the x86.

And OS theory. As much as you can get your hands on. In fact, let me reverse that: read the theoretical stuff first. It will help put the rest in context, and let you get some idea of a direction to go in before getting into the hard details of the Intel platform. After all, while it is ubiquitous now, it is impossible to say that some other design won't supplant it five years from now - Intel could decide to push Itanium as a general platform, for example. Improbable, true, but stranger things have happened.

But I digress. The good news is, that you don't have to spend ten years studying before getting started; in fact, the best way to learn is to jump into the programming with the idea of learning how to do things first. I strongly suggest you start with the idea of 'build one to throw away'; that is, do a number of small trial programs practicing how to write, say, a mouse driver, and then use what you learned later to write the real thing.

This still leave the question fo where to start. Well, there are two main schools of thought, the low level and the high level. The low level school starts at the machine and builds up, usually starting with a boot loader, a bit of file system support, code to move into protected mode, low-level device drivers, etc. While these designs often get some impressive performance out of the machine, they often stall early (rarely geting past the boot loader, which is an excruciatingly difficult thing to get right), and the resulting monolithic system is often extremely difficult if not impossible to add on to without major hacking. This was probably the was DOS was originally written, as well as how many classic (read 'ancient') operating systems like ITS and TENEX were done.

The high-level approach is to start with the higher-level operations of the system, such as the task scheduler, filesystem, and memory manager, and then fill in the details. This may sound impossible, but the usual approach here is to launch the OS from a simple, easy to override system like DOS, and slowly move away from the system dependencies as you write the low level code later. IIRC, this was the approach taken by Linus Torvalds in the very earliest stages of what became Linux.

Generally speaking, while the high-level approach tends to result in less eficient systems, the development goes much quicker and the result is more flexible and more generally useful.

My own advice is to spend some time writing low-level material, then after a thorough review of the theory, start over again with a high-level approach. This is a very time-consuming process, but in the long run it will probably save as much time as it costs.

I do not recommend writing your own boot loader, except as an exercise; it is simply a waste of time, and it is the part which has driven off more would-be OS developers than anything else. You may want to try it, just to familiarize yourself with it, but don't commit yourself to it as a part of the overall project. I would suggest using GRUB intead. Though it isn't the easiest boot loader to use, it is probably the most powerful and flexible. Be certain to read both the GRUB manual and the Multiboot Specification very carefuly at least twice before trying to use it, paying particular attention to the examples given. Mind you, I'm a bit of a hypocrite here, as I did indeed start by writing a boot loader; but that was because I wanted the knowledge of how it worked in detail. While it is true I am writing an OS from it, it is still meant as a practice run; the whole LoIS project is meant as a sort of tutorial, for myself as much as anyone else (and it seems I've stalled on it, anyway). When I really get serious about things, I'll use GRUB, instead (in fact, I intend to write another tutoriasl based on that, as well).