OS Development: Starting / Kernel

[shikhin]

Hi All,

I have decided to make up a OS. And I know it would take loads of time, and wouldnt be the next WINDOWs Or Linux. So I started with first reading of on Assembly language and the x86 architecture. And now I am here, with only one thing in the whole OS Development process decided, i.e. Currently I would be using GRUB as my boot strapper / boot loader.

So I have Modern Operating Systems 3e here on my bedside, and am currently thinking what shall I start up with? Shall I read this book? What after that?

To be more specific I need some links to tutorials on Kernel Development. Some beginner links that is, which would start of from the beginning and would explain almost everything. Secondly I need links to books on OS Development, not design, as I have 1 book here with me, and have already ordered one more, which I would completely read and understand.

After that I also need some basic tutorials on how to implement a kernel, and other such things. Also I need to ask, what else shall I do. I am pretty focused and want to follow up with this project, how long it may take, and how much energy it would take up from me.

I hope you guys could understand what I want. I am pretty thick in explaining my mind, so if by chance you dont understand anything, feel free to say so.

Regards,
Shikhin

[AmA]

Hi!

I have decided to make up a OS. And I know it would take loads of time, and wouldnt be the next WINDOWs Or Linux. So I started with first reading of on Assembly language and the x86 architecture.

Congratulations!

Ok. At this point I cant talk to you in technical terms. At this point I can give you some councils:
1º: Don't be reading only. Yo need to code and read 50/50.
2º: It will be more easy if you start just coding something(print/draw OS-logo on screen). And with the time there will appear more and more questions in you mind and then will be the moment to read. This way you learn process will be guided. Yo have to read on demand, not just read to be reading. There are muuuch people that just read and don't code nothing, they have much posts in forums on internet and just bla bla bla about terminologies, and they can't code nothing. Don't end up like them!

I wish you good luck!

[shikhin]

Thanks AmA,

By the way, I am not reading just to be reading. At the present I know, I am a little weak on the subject of OS Design so I am reading about that. Before that, I had started coding a bootloader, and after that I had read on how shall I continue, shall I make my own bootloader, or probably use GRUB. I ended up with the desicion of using grub. So now can you please tell me where shall I continue. I mean, I cant just continue with coding, with what I know. I need something to build up my future base on kernels on. I wouldnt just started scrathing thourgh it now, as currently I am into OS Design, but for my future planning, I need all this information. Probably just a fact I shall know, probably a page I should read, probably some tutorials, probably a book, probably exercises. Anything that can help me build my future base. I am pretty confident on this project so thats why I need help.

Also which executable format does GRUB support? I tried with .SYS file but it displayed: "Error 13: Not Supportable Executable Format"/ Someone on another thread told me it is ELF format. If this is true, how shall I make a ELF supportable file format? Any hints on what to do?

[NickJohnson]

I think there's a way to get GRUB to boot PE files (the ones Windows makes/uses), but you'll probably have more long-term luck with ELF (just my opinion). Are you only using assembly, or are you also using C for your OS? And are you using Windows as I guessed?

Also, here are some suggestions for online reading:
- the wiki (that's what it's for)
- http://www.jamesmolloy.co.uk/tutorial_html/index.html (good tutorial under Linux)
- http://www.osdever.net/bkerndev/Docs/title.htm (good tutorial under Windows)

[shikhin]

I am using Linux SuSe Enterprise Edition 11.0. Further more, I am currently just using Assembly for my small OS. So how shall I convert it into ELF format? And by the way thank you for the links.

[AmA]

Hi!
I allways used "BIN" format, and I can't help you with ELF, SYS etc.
But you can be sure that the El Torito specification is more shorter than GRUB one, and that is easier to employ. Once you prepared the El Torito header, the only thing you have to do with a common Assembler is to copy-paste the header in the beginning of the code and it works. You can build the El Torito header with "DB" pre-processor instructions and then produce one binary archive with .ISO extension. All virtual machines boot fine from this and Win Vista(I guess Suse too) is writing the ISO on the CD directly without special third party programs.

In the computer world often who arrives first, owns the PC. If you execute third party code before you OS and this code have bugs(for sure) or malware, you OS will be the victim.

The finally decision is yours.

[Gaidheal]

Paranoia? ;¬)

Using a third-party (especially a fully open) bootloader is a good way to start and an OS that will 'play nice' with a chainloader is pretty much a must anyway but despite my teasing above, I'd want to be doing my own scratch-loading, too.

With regards to the reading, the comments about not reading exclusively are fair enough but I'd worry more about not reading enough - a lot of people read just enough to get going and then read bits and pieces as they get into trouble (or else ask in a forum like this), get past that obstacle and move on, never understanding that a lot of their issues are coming from not having a full grasp of the task to start with and not spending sufficient time on design rather than implementation.

Fun as it is to dive in and code and acknowledging that a 'proof of concept' type prototype (even if it has bugs, lacks functions and the end product will probably be nothing like it) is still useful, I'd advise that you carefully design your OS before getting too deep into the code. The pitfalls for not are winding up with a lot of code that you are reluctant to abandon, since a lot of time went into it, even though it doesn't quite work and needs a lot of tweaking to reflect later decisions or, writing quite a lot of things twice because you made decisions that invalidated earlier code and 'bit the bullet' (rewrote them rather than fudge it).

Just my thoughts, anyway.

[shikhin]

Thanks for all the help which has been given till now. Still any thing more?

[~]

I am currently trying to understand hardware and OS-related topics in more detail and have been trying to understand how others do things. Maybe this style of explaining seems good to you:

http://126.sytes.net/3party/tutorials/afpm17/01/docs/

I have also a list of mirrors of "downloadable" websites that have disappeared or changed/been reduced:

http://126.sytes.net/sitemirrors/

If you find a use for this, I can be directed to what to do next, and you could also help me by providing some material you have understood and then I could try to expand it. It would help me concentrate document and study topics that are more used by others and that stuff gets documented as this knowledge becomes better understood, from basic to advanced. Or you could wait for more explanations of things as I need to document them and actually understand them, and put them in my website.

Having said that, you can find plenty of tutorials, but you should at least read in detail the first parts so you understand the style of the author and better build gradually and actually learn something from them, but as I said it requires that you learn the tools and concepts the author is using to build the tutorial.


I have the habit of clearly and verbosely document things as far as I understand them in every detail, mostly because it later helps me manage my own source code and make it much more manageable, and is one of the things I do to understand stuff more easily when failure of getting something working gets extremely boring and more knowledge is an obvious need. I can tell you that two of the main reasons of not understanding something are that you lack some prerequisite knowledge and that you don't have an actual use of something to be immediately applied.

I can also tell you that almost all tutorials lack prerequisites and leave you asking yourself those things, and you won't get anywhere by just trying and wondering more and more. To solve this, you can try reading whole books on topics you are interested in. When you get bored you can read other book or actually try to apply what you have learned to see how much you have advanced. The key here is that when you program and document, you can manage to make reusable material that can be later improved easily so you don't have to start from scratch empty-handed.

If you want to learn about boot programs, maybe it will be better to try programming some simple boot code and then study the well-established boot specifications and boot programs, learn how to use and configure them, study their source code and then advance on what you have learned. It will take a good deal of time but it's necessary if you want to actually get to understand the boot managers/loaders not by chance, but because you have tampered with them and actually know things like how to boot an OS like Linux with, say, LILO or GRUB.

I personally studied and modified one of the simple FAT12-compatible floppy boot sectors at http://osdever.net and really understanding what I was doing took me several weeks, and I kept adding and improving several bits in the span of several months, in sporadic moments of inspiration. So you can get an idea of how much more would take to understand complete boot managers like GRUB and the specifications in which they are based.

You can see also at http://wiki.osdev.org/. Keep in mind that it is still a work in progress and may contain several misleading information. If you find it unclear or not working, it is a good measure that you will need additional material to be able to advance.

[AmA]

Gaidheal:

This is not paranoya!
Imagine a software company that connect to the net with the development computers(imperdonable). So imagine the tool-chain the company uses to produce software gets infected. A virus no necessary have malicious payload, but always lead to malfunction, this time of the tool-chain and every software produced with it. This is not paranoya, this is what profesionals do. The only tool I currently use is NASM. And soon I will try to code my own assembler integrated in my OS. This way I will not depend on other dom0 OS, but only on my own software.

Yes, you have all the right about the code overwriting. But imagine you are reading a book(the worst case, payed for it, not downloaded it) and in the middle in the book, autor says: "But all this of course applies only for **** hardware". Then you learn that **** hardware is no more produced nor sold. Lol 300 pages lose of time... Some autors are doing this, believe me...Both things have to be avoided.

And finally, when one is writing software to windows, he often use third party tools for optimizing, for debuging etc. But I think that if one has decided to do an OS, it will be normally to try to do it all himself, if not, I recommend the LFS project.

[~]

The only tool I currently use is NASM. And soon I will try to code my own assembler integrated in my OS. This way I will not depend on other dom0 OS, but only on my own software.

I have been working in my own toolchain which by now is a simple compiler. You have to expect having to work in it under a host OS like Linux or Windows, at least until it gives you total stability to not depend on other OS. However, think about that it will require you to handle filesystems, an editor, an actual assembly parser, a "screen driver", a keyboard driver, interrupt routines, among many other things. You can't make it all in one only pass if you are starting from scratch. Also, there is a moment in which purely programming in assembly language will not be an advantage but a time consuming activity that you could and should automate in more or less grade if you want to be more and more productive and keep getting smarter. It will take you several rewrites before you can migrate it to your own OS, at least if you want to make sure that your code is optimum and well tested and that your codebase will not have to be rewritten several times in a span of several inefficiently used years. It took me a year of "lazy" analysis and sporadic reading of a compiler book and actually trying to code several approaches before I could be able to come up with a truly working yet basic compiler interpreting code. I made it in Javascript because it is a very straightforward language and it has an excellent programming level for this task and a clean environment, and is very easy and fast to modify because it is interpreted (you don't need compilers, linkers or makefiles just for a simple conceptual test) and can be debugged in any system with at least Firefox. That's an advantage of a programming environment that is common and 100% standard across platforms. You will want to have it in one form or another for trying out concepts reliably. I won't be implementing it formally in a lower level language until my algorithms are optimum; doing otherwise would be duplicating programming effort and wasting too much time that could be used more naturally seeing the actual correctness of the raw algorithms and getting cleaner code for much less effort. If that is not enough, I already use my compiler for translating sources into assembly which I can readily do by hand but now I have better things to do to advance my projects. If I find errors (which by now is extremely frequent) I work to fix them. Once the code starts to look too bad, I will rewrite it and will implement it in better and more effective ways, with all of the concepts and code sections that were proven to be the best ones.

You can see most of what efforts it took me here and also the actual code and the latest version:

http://126.sytes.net/projects/realc/documentation/

You can see examples of source files in .CSM files that my compiler understands, here:

http://126.sytes.net/projects/x86/OS/LowEST/

Yes, you have all the right about the code overwriting. But imagine you are reading a book(the worst case, payed for it, not downloaded it) and in the middle in the book, autor says: "But all this of course applies only for **** hardware". Then you learn that **** hardware is no more produced nor sold. Lol 300 pages lose of time... Some autors are doing this, believe me...Both things have to be avoided.

Like what books?

Also, you will usually want to buy books once you have reached a minimum level of expertise, and once you have consumed all of the basic things you can get for free and also worked trying to do a kernel. By then you will probably understand and be familiar with up to half of what common books have to tell you and they will provide you with a new half of things you needed to know. And with a basic background and guidance that you can find here and in any other developer site, what you are fearing cannot happen. In the x86 PC environment specially, anything you learn will at least give you a historic background, which is a good thing and is fully functional and applicable to this day.

I haven't seen so far anything x86-related that is of practical nature (EGA, VGA, ModeX, timers, speakers, floppy, games, graphics, making compact assembly code and basic "optimizations") that even coming from the 8088 era had been completely useless. Remember that it seems that the more you go back in the timeline of PC history, the books and tutorials seem to get richer in the basics, which you probably won't find in "modern" books in a reasonable amount of time and simplicity. Even PC repairing books of that time will give you a few code snippets that still work. All of those things are fully integrated in even the latest x86 PC so you don't lose anything.

You can also stop reading any book always if you find that you don't require it in a given moment, but there is always something to learn from a well-written book.

And finally, when one is writing software to windows, he often use third party tools for optimizing, for debuging etc. But I think that if one has decided to do an OS, it will be normally to try to do it all himself, if not, I recommend the LFS project.

But think about how what you created is, for practical purposes, always below and "simpler" than the environment in which you created it and your own capabilities. Working in a sterile kernel environment will present the same barrier and will be an unnecessary delay and level of effort. Rushing would be like trying to solve a very complex math formula skipping key steps, or like trying to complete the programs you want for your OS with an undefined API or unimplemented design for the intended tasks. Why would you need to rush if you develop your own way to achieve things in a reasonable time? When you rush you demonstrate desperation because of not advancing as much as you wanted, and you must realize that studying and experimenting theory and practice in better detail is the way to go, and you will naturally become faster as you learn more things with time and a constant stream of clear information and action. You will certainly get very poor results.

Almost anything can be tested in user space and then integrated in a kernel. The most natural thing would be to program kernel-level things in a kernel environment as much as possible, and user space things in a user-space environment, and when you are just starting, the only user-space environment you have is a mature host OS like the ones everyone use daily. Since pure DOS-like operating systems without any drivers loaded have no protection, you could ease your work by using something like FreeDOS or MS-DOS as a path to kernel-level programming and tests, and DOS would be no more than a temporal shell you would use as long as there are tests that need it and as long as you don't have such a complete basic shell and environment.


It is a very common thing to try to use the goal as the path to the intended goal itself, and too often it is a too slow path of action that will yield too few results considering all of the work you still have ahead (see how much work it requires you to just boot the machine and load a "kernel", or to enter Protected Mode or Long Mode, and then you displayed a colorful animated GIF; in the end all you have done in practical terms is switching the way in which the CPU works and give it your own program to start executing and made a big effort to decode a graphic file for just showing a short animation, a very difficult thing that could be easier to do in other means, and even if you are very happy and proud you just did something very basic and there are still many other things waiting to be done, that are very difficult to create, program and make work but that have an amazingly simple actual effect in the real world and as helpful for the users (and for yourself), so one has to be very effective at reducing the number of purely failed tries as well as make sure that what you just achieved can be integrated with what you need to add next). It is better not to be too tied to existing technology as to not being able to innovate, and not too far as to not being compatible with the current world. It would be better to understand several concepts and ways to implement them (e.g, multitasking and the different ways in which it can be implemented), and then as you understand it program little related pieces that you can really test whether are working or not, until you have all of them that will make up the whole of your intended algorithm.


Another thing is that you shouldn't depend too much in optimizing tools but rather in improving your skills and algorithms. A bad algorithm will always malfunction, won't integrate well or will be too slow no matter how much optimization a tool applies to it. In this way you will truly be in charge of your programs and the optimization will just be an automation of a process you understand and could do yourself but that would be a waste of your time in a repetitive task. Using optimizing tools without knowing what they are doing is of few significance, and that is something you won't be able to do until you have tried to do it manually and have worked extensively at that for a very good period of time. Actually implementing one is just automating a task you have mastered inside out and that you are freeing yourself from repeating because a tool can do it as well as you, with the plus that you will have made it.

Read the Michael Abrash's Graphics Programming Black Book. Take your time and if possible read it from start to end. The writing style of the author and the structure of the document are more beneficial in that order. You will find many things that will improve the skills you might have at a basic level. It won't only talk you about graphics but about a very broad set of topics needed to be a better programmer. You most certainly won't be implementing all of what is discussed there but grasping things from it always helps.

http://www.gamedev.net/reference/articl ... le1698.asp

Don't forget to have more than one machine handy, preferably several machines from different generations, as much as it is possible for you. Even two identical machines are enough for a start. Otherwise you will have a very hard time working solely with emulators or having to restart your machine, if you want or need to test real hardware. And don't forget to get the best of your current OS and program your algorithms there before thinking about restarting your work machine, since it is a more natural, logical and rewarding path.

[gerryg400]

This is not paranoya!
Imagine a software company that connect to the net with the development computers(imperdonable). So imagine the tool-chain the company uses to produce software gets infected. A virus no necessary have malicious payload, but always lead to malfunction, this time of the tool-chain and every software produced with it. This is not paranoya, this is what profesionals do. The only tool I currently use is NASM. And soon I will try to code my own assembler integrated in my OS. This way I will not depend on other dom0 OS, but only on my own software.

You are correct. Where ever I have worked, build machines are not connected to the internet. They are specially built and maintained machines (not maintained by the IT department). However, it is not necessary to write all tools/libraries/software yourself. Being a professional requires tradeoffs and getting the job done in a timely and profitable manner requires re-using and building on other peoples work.

[AmA]

~:

I realize the problems with moving the production environment to my own OS. I understand that this is pretty hard and confusing. But I still have better things to do. Now I am trying to make my kernel to work(minimally, before loading drivers) in all past 2008 PCs. I still did not entered in the multiprocessor world so I will probably will depend of my host OS for looong time. And about using another languages to produce.... I am skilled with ASM, for me is more simple to jmp somewhere than using JavaScript stile coding. Often what is slowing me is the specification or the algorithm itself, once understood both, I need only half hour to code it in ASM(without the optimization).

About reading, in my case, I have to make an OS, without the help of nobody, I can't spend time reading literature. The only thing I read from months is specifications(and browsing on the net). So just as example, I don't know how to enable the A20, becouse when I see some explanation about it I dont read it. Never had the need to deal with A20.

[~]

~:

I realize the problems with moving the production environment to my own OS. I understand that this is pretty hard and confusing. But I still have better things to do. Now I am trying to make my kernel to work(minimally, before loading drivers) in all past 2008 PCs. I still did not entered in the multiprocessor world so I will probably will depend of my host OS for looong time. And about using another languages to produce.... I am skilled with ASM, for me is more simple to jmp somewhere than using JavaScript stile coding. Often what is slowing me is the specification or the algorithm itself, once understood both, I need only half hour to code it in ASM(without the optimization).

You should realize with time that you need low level languages like assembly for low level tasks, and other higher level languages for higher level tasks. Doing otherwise will greatly decrease your actual productivity. Programs are just that and there is all the time in the world to improve them, so trying to use exclusively assembly can be a disadvantage. It is better to start simple but actually functional and start improving from there. Think about maintainability. I am also skilled with assembly language and can perfectly code programs manually in it, to the point that I understand how to make a simple compiler that does the boring "temporary" bits for me with a style almost identical to my assembly style. But you will get very tired or bored after many failed attempts from scratch and having to repeat all "trivial" things again. If my compiler can save me from typing almost exactly the same assembly code I would write down, why not save that energy designing and debugging the implementation and leave the brute force, low-paying job to a tool?

And you can't understand how to manage multiprocessor systems if you don't understand single-CPU multitasking before.

About reading, in my case, I have to make an OS, without the help of nobody, I can't spend time reading literature. The only thing I read from months is specifications(and browsing on the net). So just as example, I don't know how to enable the A20, becouse when I see some explanation about it I dont read it. Never had the need to deal with A20.

But A20 is one of several vital things you need to use. You must enable it as soon as you start using 32-bit or 64-bit programs that access the whole memory address space. Otherwise, you will find mysterious errors and you won't know what is going on. In this case, you just will be unable to use odd Megabytes (the one after the first one for Real Mode, and every other: 1048576*1, 1048576*3rd, 1048576*5th, etc.).

Ignoring those seemingly useless bits will have the great effect of locking your development with bugs you don't know where they come from. This is almost starting to rush. You won't be able to figure everything for yourself, and specifications always lack the explanation of how to actually use it. You need to find proven knowledge as long as it exists, and investigate and develop what doesn't exist.

Otherwise you can expect to spend several years only in the basics without achieving much, unless you have already exceptional practical problem-solving capabilities and background of experience in several computing fields.

[AmA]

~:

Thanks for the councils! I am hearing all this for first time, you know.

But for me the funny part is to discover the things by myself. And this way for sure I will develop something different, maybe will take me more time than takes to you, but for sure will be different enough.