Bootloader development clarifications

[Gioele]

Hi at all,

I decided to develop a small OS in protected mode just as hobby and learn more about how an OS is working, I also decided to learn how bootloader works and how it enables the protected mode. So far I have some kind of working code where the first stage bootloader loads the second stage through a FAT file system on a floppy and the second stage loads the kernel, prepare everything to enter the protected mode and next enable it, then finally it jumps (32 bit long jump) to the kernel. I've done that by following a "tutorial" and consequently using various manuals/references to go deep into arguments separately (like A20 line, FAT bootsector, GDT, IDT, etc...) and learn how the single thing works (I know it's a weird way to learn but if it works I'm happy with that). Then I will focus more on the kernel which I will develop by myself only through docs because it's my goal. Also I will edit in the future the bootloader with more checks and features like fat32 support, etc...
Now I'm stuck because of a triple fault, which I've read it's caused by a misconfiguration of IDT or GDT and I just want to clarify some points:

1. The kernel is loaded to segment 0x2000 (linear address 0x20000) but AFAIK that's real mode RAM and it shouldn't work when protected mode is active right? If I'm right then I have to copy the kernel to protected mode RAM by using the real mode RAM as buffer or load directly with a FAT driver, even this is right?

2. I initialize the IDT without any interrupt because kernel will fill it as first thing when bootloader give the control to it and before jumping to kernel interrupts are disabled, is it ok (please don't blame me so far after days of research I didn't find anything about that)?
Also note the kernel is a sample shipped with the tutorial so if triple fault occours there, and I think it is so, because bochs before letting me know that triple fault occourred it's say: " 0x00000000000201c3>> jmp .-2 (0x000201c3) : EBFE" and the address match with the kernel linear address + an offset. So if that's confirmed I literally don't worry because as said my kernel is still in dev (0 to 0.0005% done ).

3. The GDT has only 1 entry (2 if we count the first empty descriptor) and it's the one of the kernel (code + data). It's set to fit all 4GB of RAM with base address 0x0000 and granularity actived, that's for both code and data. I'm worried of that because if kernel takes all RAM how can it be able to loads another process. Otherwise I think that if kernel hasn't access to all RAM how it can load things. The last thing is that Code descriptor, when thigs are more developed (my kernel) should be resized to fit in a small area of memory because data is where kernel should put things such as processes and not to mention the fact that now code section is overlapped to data one and that's really bad, I think.

Sorry if these questions seems noobish but really, I couldn't find any answer in a cuple of days of research (this forum included) (may be it's me that I didn't noticed things or researched worngly and if this is the case I want to apologize). Also sorry if my english is a bit clunky, I'm not english mother language.

If needed I will provide code as soon as I get access to it (now I'm at work, don't worry launch time)

Thanks in advance,
Gioele

[MDenham]

1) You'll want to look up identity mapping.

2/3) You're getting a triple fault because of multiple GPFs here: the first one is because you don't have distinct code and data descriptors (and therefore you're getting a GPF for trying to execute code when the access byte in the GDT doesn't allow it; if you "fix" this by just flipping that bit, you're going to run into issues as soon as you try to write to "data" space), the second one (which becomes a double fault) is because there's no interrupt handlers in the IDT, and the third one (which is your triple fault) is because there's no interrupt handlers in the IDT to handle the double fault, either.

The first thing your kernel needs to do is put together a sane IDT. (You'll need to make sure the GDT is at least marginally sane with separate code and data entries first, otherwise you'll just triple fault again.) Even if it just sends everything to a "show the interrupt number on the screen and halt" handler (the simplest method that comes to mind for doing this involves dirty tricks with mapping one page into multiple places in memory; your IDT just tells each interrupt to go to an appropriate copy of the page where a bit-shift can then recover the interrupt number), it's at least going to give you the opportunity to see what else is wrong with your code.

3) This is where memory management comes in - the kernel does "take up" all the memory initially, and hands it out as needed to other processes. During startup of another program, that initial handoff of memory will be enough to hold that program's code; the program is responsible for telling the kernel "hey, I need this much memory" and the kernel does what it can to fulfill that request (first by finding free blocks of memory, then by swapping out things in memory that haven't been used recently to disk, and finally by telling the program "sorry, I can't give you that much; I gave you this much instead (which might be none)").

[Octocontrabass]

The last thing is that Code descriptor, when thigs are more developed (my kernel) should be resized to fit in a small area of memory because data is where kernel should put things such as processes and not to mention the fact that now code section is overlapped to data one and that's really bad, I think.

Before you go any further, do some research on paging. Paging is very powerful, and much better than segmentation.

Most x86 operating systems have a bunch of overlapping segments with base 0 and limit 4GB, because this is equivalent to no segmentation at all.

[Gioele]

Thank you very much MDenham you cleared me some doubt I had and remotely I thought that the error involved GDT/IDT but I wasn't completly sure about that. Also thank you for letting me know some new things of which I will treasure, like watching the identity mapping, confirming to me that the first thing that kernel will do is to setup, of course, the IDT (as you said a sane one) and that’s right the kernel will take all the space on start (code + data filling all the remaining free space that code didn’t took right? Just to be more clear on that point).

One last thing, you said the program that is being loaded is responsible of letting the kernel know how much space does it need, this is done manually when creating the program or it can be done like what I believe an hosted implementation do, practically when I compile the source code, the compiler will append some info about that (so it comes to need a cross compiler or a modified one). That's for now it's just a curiosity of mine I want to satisfy in few words, because of course I will see that later, much later in depth.

Before you go any further, do some research on paging. Paging is very powerful, and much better than segmentation.

Most x86 operating systems have a bunch of overlapping segments with base 0 and limit 4GB, because this is equivalent to no segmentation at all.

Octocontrabass thanks for the advice I will treasure that too and I will do research about that for sure because I will do anything to improve myself and learn new powerfull stuff.

Again thanks at all, very appreciate your help.

[Octocontrabass]

One last thing, you said the program that is being loaded is responsible of letting the kernel know how much space does it need, this is done manually when creating the program or it can be done like what I believe an hosted implementation do, practically when I compile the source code, the compiler will append some info about that (so it comes to need a cross compiler or a modified one). That's for now it's just a curiosity of mine I want to satisfy in few words, because of course I will see that later, much later in depth.

Executable file formats (ELF, PE, COFF) include information about how much space the program needs in order to start. Most operating systems also provide a way for the program to request more memory while it's running, and return memory to the OS when it doesn't need it anymore.

[Gioele]

One last thing, you said the program that is being loaded is responsible of letting the kernel know how much space does it need, this is done manually when creating the program or it can be done like what I believe an hosted implementation do, practically when I compile the source code, the compiler will append some info about that (so it comes to need a cross compiler or a modified one). That's for now it's just a curiosity of mine I want to satisfy in few words, because of course I will see that later, much later in depth.

Executable file formats (ELF, PE, COFF) include information about how much space the program needs in order to start. Most operating systems also provide a way for the program to request more memory while it's running, and return memory to the OS when it doesn't need it anymore.

Many thanks for that. Now I can start to smash my head into the keyboard on my own and see what happens, but with logic of course.

[FallenAvatar]

Also, just wanted to correct a misunderstanding I saw in your OP. Namely, you said Real-Mode RAM is different than Protected-Mode RAM. This is incorrect. They are the same physical RAM, they are just addressed different. Please take a look here

- Monk

[jojo]

To be even more verbose, OP:

Let's say you want to load your kernel into memory in real mode, and you want to be able to access it in protected mode at 0x20000

You can achieve this by writing your kernel data to 0x2000:0x0000 in your real mode code. You can then enter protected mode and jump to 0x20000

0x2000:0x0000 real mode = 0x20000 physical = 0x20000 flat protected mode

This is all assuming the simple case in which you have set up your GDT so that your segments start at 0 and end at 4GB, which is what everyone does anyhow.

[Gioele]

Also, just wanted to correct a misunderstanding I saw in your OP. Namely, you said Real-Mode RAM is different than Protected-Mode RAM. This is incorrect. They are the same physical RAM, they are just addressed different. Please take a look here

- Monk

Sorry about that, I was intending what you said with wrong words (of course physical RAM is the same for real and protected mode, at least in this context) I need to dig more on correct terminology and thanks for correcting me, any constructive correction/criticism is always accepted to improve myself and I hope other people.

To be even more verbose, OP:

Let's say you want to load your kernel into memory in real mode, and you want to be able to access it in real mode at 0x20000

You can achieve this by writing your kernel data to 0x2000:0x0000 in your real mode code. You can then enter protected mode and jump to 0x20000

0x2000:0x0000 real mode = 0x20000 physical = 0x20000 flat protected mode

This is all assuming the simple case in which you have set up your GDT so that your segments start at 0 and end at 4GB, which is what everyone does anyhow.

Thanks for this explanation actually I knew that 0x2000:0x0000 (segment:offset) was the 0x20000 physical address (16 * segment + offset) but I didn't know that 0x20000 was also the flat protected mode address (assuming that GDT is setup, as you said, with the start of the segment to 0 and the end to 4GB).

Thanks at all again for all your input, I learned a lot today, usually I hate asking for things and help because I feel like I haven't put much effort in research, but when I start turning around on the same point I need a boost to unlock myself

[azblue]

I'm worried of that because if kernel takes all RAM how can it be able to loads another process. Otherwise I think that if kernel hasn't access to all RAM how it can load things.

This right here kept me confused for freaking ever!

Normally what you do is load your kernel at the upper 1GB, but have all your segments start at 0 and be 4 GB in length. Since your segments cover anything, your kernel can access everything -- as it needs to to load programs. However, for "kernel stuff" it won't touch that lower 3GB -- after all, that belongs to the user mode program.

When you switch to user mode, once again you have all your segments starting at 0 and covering the full 4 GB. Just as the kernel could access the user-mode's space, the user-mode program can "access" the kernel's space, in the sense that you're not going to get a segmentation fault when you read/write the upper 1GB. But you don't actually WANT that. So what you normally do is protect your kernel via paging, not segmentation, by setting the supervisor bit in virtual pages mapped from the upper 1 GB.

When you load a second program, you'll load a new value for CR3 pointing to new page tables. Once again, you'll have the kernel mapped to the upper 1GB (simply copy the same page table data for your upper 1GB in the first address space). Now your kernel is the same in 2 different address spaces, your kernel can access the entire 4GB in either address space, each program is completely seperated from the other, and the kernel is protected from each program.

When you load a 3rd, 4th, 5th program, etc, you just keep repeating that process -- load a new CR3, copy your kernel pages.

[Gioele]

I'm worried of that because if kernel takes all RAM how can it be able to loads another process. Otherwise I think that if kernel hasn't access to all RAM how it can load things.

This right here kept me confused for freaking ever!

Normally what you do is load your kernel at the upper 1GB, but have all your segments start at 0 and be 4 GB in length. Since your segments cover anything, your kernel can access everything -- as it needs to to load programs. However, for "kernel stuff" it won't touch that lower 3GB -- after all, that belongs to the user mode program.

When you switch to user mode, once again you have all your segments starting at 0 and covering the full 4 GB. Just as the kernel could access the user-mode's space, the user-mode program can "access" the kernel's space, in the sense that you're not going to get a segmentation fault when you read/write the upper 1GB. But you don't actually WANT that. So what you normally do is protect your kernel via paging, not segmentation, by setting the supervisor bit in virtual pages mapped from the upper 1 GB.

When you load a second program, you'll load a new value for CR3 pointing to new page tables. Once again, you'll have the kernel mapped to the upper 1GB (simply copy the same page table data for your upper 1GB in the first address space). Now your kernel is the same in 2 different address spaces, your kernel can access the entire 4GB in either address space, each program is completely seperated from the other, and the kernel is protected from each program.

When you load a 3rd, 4th, 5th program, etc, you just keep repeating that process -- load a new CR3, copy your kernel pages.

Thanks for that azblue, actually as already said by Octocontrabass I need to check paging, but this clarification will help me a lot understanding it as it introduces that in few words.