Bootloader design

[8infy]

I made a basic 2 stage bootloader (excluding MBR), all it does is switch to protected mode and read kernel from disk.
The bootloader uses the LBA read BIOS function to read the kernel before switching to protected mode.
This design seems OK for now, however I have some questions about improving it in the future.

1. Am I correct in thinking that there's no way for me to load the kernel anywhere above the 1MB range (including HMA) from real mode?
2. Does that mean that my kernel can only be like 500KB max (to fit in the guaranteed free range)?
3. What are BDA/EBDA and are they needed after switching to protected mode? Can I overwrite those areas?
4. What are some common memory locations to store GDT/IDT? Is 0x00000000 a conventional place for them?
5. If I want to load my kernel above the 1MB mark does that mean I have to implement actual separate disk drivers for the bootloader (since I can't use BIOS anymore)?
6. Would that mean that I have to make another stage for the bootloader that operates in protected mode and implements those drivers etc.
7. Is there any way to link 16 bit object file with a 32 bit object file (so that I can avoid creating another bootloader stage and write the "advanced" bootloader in a language like C at the same time)?
8. What are some of the common approaches for this? How does GRUB handle this? I've seen some wikipedia description of it but it was kinda vague, does it have like 3 stages as well?

Thanks!

[bzt]

Actually, you could do what I do: my bootloader runs in protected mode, and I have a load sector function which:
1. temporarily switches back to real mode
2. calls BIOS to load a sector into a buffer somewhere in conventional memory (below 640k)
3. then it switches back to protected mode
4. and copies the data from the temp buffer into its final location (probably above the 1M mark)

Here's my code. (Some notes, I also check for CDROM and update the number of sectors to read the same amount with 2048 bytes sectors, and it implements RAID-1 mirroring too.)

I can call this prot_readsecctorfunc from a protected mode code, and it doesn't have to know that the function switches back to real mode. From it's perspective the function reads data from disk to anywhere in the 4G address space.

Cheers,
bzt

[8infy]

Actually, you could do what I do: my bootloader runs in protected mode, and I have a load sector function which:
1. temporarily switches back to real mode
2. calls BIOS to load a sector into a buffer somewhere in conventional memory (below 640k)
3. then it switches back to protected mode
4. and copies the data from the temp buffer into its final location (probably above the 1M mark)

Here's my code. (Some notes, I also check for CDROM and update the number of sectors to read the same amount with 2048 bytes sectors, and it implements RAID-1 mirroring too.)

I can call this prot_readsecctorfunc from a protected mode code, and it doesn't have to know that the function switches back to real mode. From it's perspective the function reads data from disk to anywhere in the 4G address space.

Cheers,
bzt

Ohhh thats super smart, is that a decent way to do it tho? Feels hackish

[PeterX]

1. Am I correct in thinking that there's no way for me to load the kernel anywhere above the 1MB range (including HMA) from real mode?

You can load a sector (or several) and then switch to pmode to copy it high. There is also a BIOS call that does that for you, but it doesn't work properly on my computer, so I guess it will fail on some other PCs, too.

2. Does that mean that my kernel can only be like 500KB max (to fit in the guaranteed free range)?

All operating systems have this problem and they all solved it. In other words: Your kernel can be gigger than 500KB.

3. What are BDA/EBDA and are they needed after switching to protected mode? Can I overwrite those areas?

Maybe this explains it:
https://wiki.osdev.org/Memory_Map_(x86) ... _.28BDA.29
I don't know all ins and outs of memory usage. But I had some nasty kernel bugs because I used reserved memory.

4. What are some common memory locations to store GDT/IDT? Is 0x00000000 a conventional place for them?

0 isn't recommended. It's a reserved area. But the Wiki says it contains only the real mode interrupt table (IVT).

5. If I want to load my kernel above the 1MB mark does that mean I have to implement actual separate disk drivers for the bootloader (since I can't use BIOS anymore)?
6. Would that mean that I have to make another stage for the bootloader that operates in protected mode and implements those drivers etc.

As I said above, you can switch forth and back between the modes. Writing a disk driver for the boolotader seems a very bad idea to me; although you theoretically could do that. But I guess you like to avoid that.

7. Is there any way to link 16 bit object file with a 32 bit object file (so that I can avoid creating another bootloader stage and write the "advanced" bootloader in a language like C at the same time)?

I have tried that. But it didn't work. Both DOS linkers and Linux LD weren'T capable of mixing 16 and 32 bit. If anybody knows how to do that, tell me, please.

8. What are some of the common approaches for this? How does GRUB handle this? I've seen some wikipedia description of it but it was kinda vague, does it have like 3 stages as well?

AFAIK grub and the usual way to dis go like this:
Stage 1 (in MBR) loads partiton boot sector
Stage 1.5 (in partition bootsector) loads a file from a filesystem
Stage 2 (in the file) loads the kernel and/or presents an user interface to select what to boot.

Note that there are many possible variants to boot:
- If I remember correctly several years ago here was a bootloader (called "nuni") which switched to pmode and loaded a file, all in one bootsector
- You could place the stage 2 between MBR and start of the first partition
- You could (as Lilo did) write a kernel file, then use a tool to detect the sectors (or clusters). Then let stage 1 load the sectors from the list.
- DOS and AFAIK Windows do it this way: Create an empty filesystem (format it) and then place the kernel in the first file and the shell in the second file in the empty rootdir. So the loader simply loads the first entry in the rootdir and then the second.
- Old Linux was booting from floppy disk. The first sector ("boot") loaded the second stage in "raw" mode = without filesystem (The scond stage was"setup", in the sectors directly behind "boot") The second stage did setup the system (video mode, memory map, etc.) and then loaded the real kernel image (packed in tgz/ bz).
- You could take an unused parttion and load the stage 2 "raw"

[8infy]

1. Am I correct in thinking that there's no way for me to load the kernel anywhere above the 1MB range (including HMA) from real mode?

You can load a sector (or several) and then switch to pmode to copy it high. There is also a BIOS call that does that for you, but it doesn't work properly on my computer, so I guess it will fail on some other PCs, too.

2. Does that mean that my kernel can only be like 500KB max (to fit in the guaranteed free range)?

All operating systems have this problem and they all solved it. In other words: Your kernel can be gigger than 500KB.

3. What are BDA/EBDA and are they needed after switching to protected mode? Can I overwrite those areas?

Maybe this explains it:
https://wiki.osdev.org/Memory_Map_(x86) ... _.28BDA.29
I don't know all ins and outs of memory usage. But I had some nasty kernel bugs because I used reserved memory.

4. What are some common memory locations to store GDT/IDT? Is 0x00000000 a conventional place for them?

0 isn't recommended. It's a reserved area. But the Wiki says it contains only the real mode interrupt table (IVT).

5. If I want to load my kernel above the 1MB mark does that mean I have to implement actual separate disk drivers for the bootloader (since I can't use BIOS anymore)?
6. Would that mean that I have to make another stage for the bootloader that operates in protected mode and implements those drivers etc.

As I said above, you can switch forth and back between the modes. Writing a disk driver for the boolotader seems a very bad idea to me; although you theoretically could do that. But I guess you like to avoid that.

7. Is there any way to link 16 bit object file with a 32 bit object file (so that I can avoid creating another bootloader stage and write the "advanced" bootloader in a language like C at the same time)?

I have tried that. But it didn't work. Both DOS linkers and Linux LD weren'T capable of mixing 16 and 32 bit. If anybody knows how to do that, tell me, please.

8. What are some of the common approaches for this? How does GRUB handle this? I've seen some wikipedia description of it but it was kinda vague, does it have like 3 stages as well?

AFAIK grub and the usual way to dis go like this:
Stage 1 (in MBR) loads partiton boot sector
Stage 1.5 (in partition bootsector) loads a file from a filesystem
Stage 2 (in the file) loads the kernel and/or presents an user interface to select what to boot.

Note that there are many possible variants to boot:
- If I remember correctly several years ago here was a bootloader (called "nuni") which switched to pmode and loaded a file, all in one bootsector
- You could place the stage 2 between MBR and start of the first partition
- You could (as Lilo did) write a kernel file, then use a tool to detect the sectors (or clusters). Then let stage 1 load the sectors from the list.
- DOS and AFAIK Windows do it this way: Create an empty filesystem (format it) and then place the kernel in the first file and the shell in the second file in the empty rootdir. So the loader simply loads the first entry in the rootdir and then the second.
- Old Linux was booting from floppy disk. The first sector ("boot") loaded the second stage in "raw" mode = without filesystem (The scond stage was"setup", in the sectors directly behind "boot") The second stage did setup the system (video mode, memory map, etc.) and then loaded the real kernel image (packed in tgz/ bz).
- You could take an unused parttion and load the stage 2 "raw"

Thanks for the information!

[alexfru]

1. Am I correct in thinking that there's no way for me to load the kernel anywhere above the 1MB range (including HMA) from real mode?

Conceptually correct. However, take a closer look at int 0x15, function 0x87.

2. Does that mean that my kernel can only be like 500KB max (to fit in the guaranteed free range)?

No. Just load it in smaller chunks.

3. What are BDA/EBDA and are they needed after switching to protected mode? Can I overwrite those areas?

I wouldn't mess with them. They may be used by hardware.

4. What are some common memory locations to store GDT/IDT? Is 0x00000000 a conventional place for them?

Not very conventional and inconvenient if you still need to run realmode code, but otherwise would be OK.

5. If I want to load my kernel above the 1MB mark does that mean I have to implement actual separate disk drivers for the bootloader (since I can't use BIOS anymore)?

No. Just use BIOS.

6. Would that mean that I have to make another stage for the bootloader that operates in protected mode and implements those drivers etc.

Unnecessary.

7. Is there any way to link 16 bit object file with a 32 bit object file (so that I can avoid creating another bootloader stage and write the "advanced" bootloader in a language like C at the same time)?

Most tools (predominantly linkers) are inconvenient or unusable for linking 16-bit and 32-bit code together. The easiest is to compile either into a binary and include it in the other one (simple appending may work).

[8infy]

1. Am I correct in thinking that there's no way for me to load the kernel anywhere above the 1MB range (including HMA) from real mode?

Conceptually correct. However, take a closer look at int 0x15, function 0x87.

2. Does that mean that my kernel can only be like 500KB max (to fit in the guaranteed free range)?

No. Just load it in smaller chunks.

3. What are BDA/EBDA and are they needed after switching to protected mode? Can I overwrite those areas?

I wouldn't mess with them. They may be used by hardware.

4. What are some common memory locations to store GDT/IDT? Is 0x00000000 a conventional place for them?

Not very conventional and inconvenient if you still need to run realmode code, but otherwise would be OK.

5. If I want to load my kernel above the 1MB mark does that mean I have to implement actual separate disk drivers for the bootloader (since I can't use BIOS anymore)?

No. Just use BIOS.

6. Would that mean that I have to make another stage for the bootloader that operates in protected mode and implements those drivers etc.

Unnecessary.

7. Is there any way to link 16 bit object file with a 32 bit object file (so that I can avoid creating another bootloader stage and write the "advanced" bootloader in a language like C at the same time)?

Most tools (predominantly linkers) are inconvenient or unusable for linking 16-bit and 32-bit code together. The easiest is to compile either into a binary and include it in the other one (simple appending may work).

Awesome! Thanks.

[PeterX]

take a closer look at int 0x15, function 0x87.

Thanks. That is exactly the interrupt I meant. You have to setup a GDT to use this. And this interrupt didn't work on my machine. So I guess it has issues on other computers, too.

Happy hacking
Peter

[8infy]

take a closer look at int 0x15, function 0x87.

Thanks. That is exactly the interrupt I meant. You have to setup a GDT to use this. And this interrupt didn't work on my machine. So I guess it has issues on other computers, too.

Happy hacking
Peter

Did it not copy correctly or returned an error or what specifically?

[BenLunt]

I can't believe someone hasn't suggested unreal mode.

Using unreal mode, you can stay in unreal mode and use the BIOS as well as access all of the 4gig address space without having to switch modes back and forth.

Switch to unreal mode, then using the BIOS read your kernel (or parts of) to a memory location the BIOS can handle (below the 1 meg mark), then move that data to above the 1 meg mark using a simple rep movsb instruction (making sure to use the correct prefix).

Ben

[alexfru]

I can't believe someone hasn't suggested unreal mode.

Strictly speaking, it involves protected mode.

Using unreal mode, you can stay in unreal mode and use the BIOS...

I don't think the mode is guaranteed to persist when making BIOS calls for I/O. I wouldn't count on it.

[Octocontrabass]

This design seems OK for now, however I have some questions about improving it in the future.

1. The HMA is accessible from real mode (depending on A20), but any higher than that will require switching modes. There are some BIOS functions that may switch modes on your behalf, but I don't know of any that are reliable.
2. After you copy part of the kernel above 1MB, you can switch back to real mode to call the BIOS. If you use unreal mode, you can hide all of the mode switching in the GPF handler.
3. The EBDA is reserved for use by firmware. Do not overwrite it. Officially, the BDA is free for you to overwrite once you stop using real mode, but some buggy firmware is known to modify the first 64kB of memory, so it's best to avoid it.
4. OSes typically use paging to place the GDT and IDT at a virtual address somewhere within the highest 1 or 2 GB of address space. Since paging is in use, the physical address is irrelevant.
5 and 6. No, just switch back to real mode to call the BIOS.

I don't think the mode is guaranteed to persist when making BIOS calls for I/O. I wouldn't count on it.

Put the switch to unreal mode in your GPF handler. If the BIOS preserves unreal mode, nothing changes. If the BIOS does not, your attempt to access beyond the segment limit will cause a GPF and switch to unreal mode again.

[alexfru]

I don't think the mode is guaranteed to persist when making BIOS calls for I/O. I wouldn't count on it.

Put the switch to unreal mode in your GPF handler. If the BIOS preserves unreal mode, nothing changes. If the BIOS does not, your attempt to access beyond the segment limit will cause a GPF and switch to unreal mode again.

That's how my Smaller C supports unreal mode. But this wasn't mentioned specifically. And when you mention it, there isn't much difference between setting up the GDT for protected mode vs for unreal mode. Either way you have a dedicate subroutine for copying data from below 1MB to above, with some GDT management and mode switching in it.

[azblue]

5. If I want to load my kernel above the 1MB mark does that mean I have to implement actual separate disk drivers for the bootloader (since I can't use BIOS anymore)?
6. Would that mean that I have to make another stage for the bootloader that operates in protected mode and implements those drivers etc.
7. Is there any way to link 16 bit object file with a 32 bit object file (so that I can avoid creating another bootloader stage and write the "advanced" bootloader in a language like C at the same time)?
Thanks!

#5 depends on what you mean. You could put your kernel into the first 500kb of physical memory and map it higher in virtual memory before switching to protected mode.
#6 (and 5 continued) If you want it higher in physical memory you can switch to PM, then transfer control to a 32 bit loader (that no longer needs to access the disk) and then move it to higher memory.
#7 if you want to use C in your bootloader, you can use an old 16 bit compiler and link that to your assembly language bootloader.

[nexos]

How I plan on doing it is kind of complex, but here we go. The MBR code loads up stage 2. Stage 2 is two files concatenated with the cat command. The first one enters Protected Mode, and enables A20. It also relocates the second one, which is directly after. It then jumps to it. It is a C executable. When calling the BIOS, it will drop to real mode and call it, and then bounce back to PMode. Disk buffers will be put in a low memory temporary buffer, and then copied to the permanent buffer, which can be anywhere. Hopefully that makes sense and is of help.