Qestions about VESA/VBE

[Brendan]

Hi,

So the best idea is to write my own bootloaders (Both for BIOS and UEFI).

For the best OS, you should write your own boot loaders. However, this is possibly not the best idea for you.

Some people only want to learn about OS development and don't care about how good their OS is; and for these people writing their own boot loaders is probably a bad idea (it takes more time and you don't learn that much).

Some people do want to write the best OS they can. For these people the first step is to gain knowledge/experience; which mostly means writing an initial (bad) OS to learn about OS development, and then using what they've learnt to write a second (good) OS. Basically, for these people it's probably not worth the extra effort to write their own boot loaders for the initial (bad) OS; and by the time these people are actually ready to start writing a good OS they're able to see the advantages/disadvantages and make up their own mind.

But how can BIOS find my bootloader image in an ISO file? I mean that 512 bytes of code inside a FAT image?

The "El Torito" specification describes how this all works. Note: If you want a good OS, then you should be using "no emulation El Torito" (where the CD looks like a CD and isn't emulating a floppy disk or hard disk, and you don't have 512-byte sectors); and you should not be using FAT for anything (even on hard disks) because FAT sucks badly.


Cheers,

Brendan

[MadZarx]

Hi,

For the best OS, you should write your own boot loaders. However, this is possibly not the best idea for you.

Some people only want to learn about OS development and don't care about how good their OS is; and for these people writing their own boot loaders is probably a bad idea (it takes more time and you don't learn that much).

Some people do want to write the best OS they can. For these people the first step is to gain knowledge/experience; which mostly means writing an initial (bad) OS to learn about OS development, and then using what they've learnt to write a second (good) OS. Basically, for these people it's probably not worth the extra effort to write their own boot loaders for the initial (bad) OS; and by the time these people are actually ready to start writing a good OS they're able to see the advantages/disadvantages and make up their own mind.

Yeah I'm thinking about it. I'm working on a bad OS But I will write my own bootloader to get some more experience. Hopefully the BrokenThorn OS Development series has described everything about bootloaders and it has a tutorial for it. I will follow that but if you have any other good tutorials about bootloaders, I appreciate you if you tell me

The "El Torito" specification describes how this all works. Note: If you want a good OS, then you should be using "no emulation El Torito" (where the CD looks like a CD and isn't emulating a floppy disk or hard disk, and you don't have 512-byte sectors); and you should not be using FAT for anything (even on hard disks) because FAT sucks badly.


Cheers,

Brendan

The tutorial about the no emulation El Torito with grub legacy has this command for making iso with grub:

Code: Select all

genisoimage -R -b boot/grub/stage2_eltorito -no-emul-boot -boot-load-size 4
            -boot-info-table -o bootable.iso isofiles

Can I replace the stage2_eltorito with my bootloader? Should the bootloader be in 512 bytes or theres no limit for it? Then if I don't use a FAT image file, should I write a CDROM driver in assembly to find and load the kernel I have never done this before so I need a small help and information
Sorry if I ask too many questions

[Brendan]

Hi,

Yeah I'm thinking about it. I'm working on a bad OS But I will write my own bootloader to get some more experience. Hopefully the BrokenThorn OS Development series has described everything about bootloaders and it has a tutorial for it. I will follow that but if you have any other good tutorials about bootloaders, I appreciate you if you tell me

I don't know of any tutorials I'd call "good"...

The "El Torito" specification describes how this all works. Note: If you want a good OS, then you should be using "no emulation El Torito" (where the CD looks like a CD and isn't emulating a floppy disk or hard disk, and you don't have 512-byte sectors); and you should not be using FAT for anything (even on hard disks) because FAT sucks badly.

The tutorial about the no emulation El Torito with grub legacy has this command for making iso with grub:

Code: Select all

genisoimage -R -b boot/grub/stage2_eltorito -no-emul-boot -boot-load-size 4
            -boot-info-table -o bootable.iso isofiles

Can I replace the stage2_eltorito with my bootloader? Should the bootloader be in 512 bytes or theres no limit for it? Then if I don't use a FAT image file, should I write a CDROM driver in assembly to find and load the kernel I have never done this before so I need a small help and information

You might be able to replace stage2_eltorito with your own, as long as it's compatible with whatever GRUB's stage1 happens to feel like doing. A better idea would be to start with your own code (e.g. your own stage1) so that you only have to be compatible with official standards.

For "no emulation El Torito" your boot loader can be any size you like (as long as it's less than about 500 KiB) and the BIOS will load the entire thing (not just one sector). This means that (unlike floppy/hard disk) there's no real point bothering with a "stage2" and easier to put the entire boot loader in a single "stage1".

Also, for "no emulation El Torito" you'd be using the BIOS to load 2048-byte sectors from CD (not 512-byte sectors), and you'd want code to handle the ISO9660 file system. Fortunately, ISO9660 is relatively simple (no fragmented files and no "cluster allocation table" to worry about), and is a lot easier than FAT.


Cheers,

Brendan

[Antti]

For "no emulation El Torito" your boot loader can be any size you like (as long as it's less than about 500 KiB) and the BIOS will load the entire thing (not just one sector). This means that (unlike floppy/hard disk) there's no real point bothering with a "stage2" and easier to put the entire boot loader in a single "stage1".

Correct but it is still worth further consideration. We have to take bugs into account. Is the BIOS always well-tested to handle this correctly? I do not have any mainstream OS images at hand so that I could check how many sectors are usually read. It would be reasonable to assume that BIOSes are well-tested to handle booting from a Windows CD. If we had a 250 KiB "boot sector", for example, there would be many 16-bit segment boundary crossings. This could be prone to bugs and I would be careful.

If there were a relatively small first stage loader and additional sectors are read manually, there is more a programmer can do to solve problems. For example, return values from a read function can be analysed, a read can be retried etc. If we had this 250 KiB loader and the loading fails (whatever the reason is), a computer might not be able to run any of your code.

If mainstream operating systems use big first stage loaders and everything works fine, this post can be discarded and I am officially wrong about this.

[MadZarx]

Hmmm ...
Antti is right. I can use 2 stage bootloader to check for errors and do more stuff. But as Brendan said, I can load the kernel in a single stage bootloader. I don't think the whole bootloader size exceeds more than 10 kilobytes.
One question remains. How does BIOS or UEFI knows which file is the bootloader? For example I've written the stage 1 and made it the boot file. Then BIOS loads that booloader. But what about UEFI? Does it search for a special file inside the ISO? Or like BIOS, needs a boot file in a special position?

[Antti]

How does BIOS or UEFI knows which file is the bootloader?

I am not very good at explaining this. For BIOS, it is not necessary to have an actual file. El-Torito Specification defines a structure where you can set the "Load RBA" and "Sector Count". It is possible to make this pointing to file data (it makes sense so that you do not use any "hidden" loader). My explanation will make this more confusing so perhaps you should check the actual documents.

Standard ECMA-199 (ISO 9660)

"El-Torito Bootable CD Specification"

For UEFI, it has to be an actual file. To boot an x86-64 boot loader, the file is "/EFI/BOOT/BOOTX64.EFI" in the ISO 9660 file system. Although it is not hard to use UEFI, I recommend that you put it aside for a while and get back to it later. Your questions are good but it is hard to give step-by-step instructions of how to do all this. Of course, we are happy to answer your questions about details.

[Owen]

For UEFI, it has to be an actual file. To boot an x86-64 boot loader, the file is "/EFI/BOOT/BOOTX64.EFI" in the ISO 9660 file system. Although it is not hard to use UEFI, I recommend that you put it aside for a while and get back to it later. Your questions are good but it is hard to give step-by-step instructions of how to do all this. Of course, we are happy to answer your questions about details.

No shipping UEFI understands ISO 9660. Instead, we have to say hello again to our good old friend El-Torito, and to our less good old friend... FAT

You see, the people at Intel had interesting ideas about what functionality a "modern" firmware interface should include. Therefore, ISO 9660 was determined to be one file system too many, and instead you have to stuff a FAT file system image inside an El-Torito entry marked for UEFI.

I think the appropriate response to this is "You can't make this sh*t up..."

[Antti]

I am shocked! Thank you for clarifying this and forgive me for spreading misinformation. I guess I could throw my "UEFI boot CD" into a trash can. This information ruined my day...

[MadZarx]

Antti thank you for those great links. I needed them a lot

Qwen you mean I should put the bootloader inside a FAT image and put that inside the ISO for the UEFI?
Would you give me more information about this please?

[Brendan]

Hi,

Qwen you mean I should put the bootloader inside a FAT image and put that inside the ISO for the UEFI?
Would you give me more information about this please?

For "BIOS, no emulation El Torito", the firmware finds a suitable boot catalogue entry, which says the starting sector and number of sectors for the boot loader. Then the BIOS loads those sectors into memory and jumps to it. Once started, the boot loader uses BIOS functions to read raw sectors, and typically contains ISO9660 support (to load more files from the CD).

For "UEFI El Torito", the firmware finds a suitable boot catalogue entry, which says the starting sector and number of sectors for a "FAT system partition image". Then UEFI mounts this "FAT system partition image" and finds a usable boot loader inside it. The boot loaders may load more files from the "FAT UEFI system partition image"; but (instead) they may provide their own ISO9660 support and use UEFI functions to read raw sectors to load more files from the CD.

Now; for El Torito you can have several different boot catalogue entries (e.g. one for "BIOS, no emulation El Torito", one for "UEFI El Torito", one for "PowerPC/OpenFirmware El Torito", etc) with different boot loaders for different architectures; where the firmware will find the correct boot catalogue entry/boot loader to use. Also; for UEFI (in general) you can have several different bootloaders in the "FAT UEFI system partition image" and UEFI will find the right one for the architecture. For example, you could have boot loaders for "32-bit 80x86 UEFI", "64-bit 80x86 UEFI", "itanium UEFI" and "ARM UEFI" all in the same "FAT system partition image".

By combining these, you can have a single CD that supports "80x86 BIOS" and "32-bit 80x86 UEFI" and "64-bit 80x86 UEFI" (and any other different systems, with or without UEFI, that you feel like).

Of course for a "multi-architecture CD"; you're probably going to have some files that are used by all the boot loaders - things like a splash image containing the OS's logo or startup screen, files containing font data, files containing internationalisation information (a set of strings for each different language), etc. The question is, how do all the different boot loaders (with or without UEFI) access the same common files?

There's at least 4 solutions to this:

• the FAT way, where the common files are in the "FAT UEFI system image". In this case boot loaders that don't use UEFI (e.g. for 80x86 BIOS) have to be able to find the "FAT UEFI system image" (e.g. by including that image as a file in the ISO9660 file system) and also have to understand FAT (which is more bloated, more complex and slower to read from). This is easier for boot loaders designed for UEFI (and worse in every way for boot loaders that aren't designed for UEFI).
• the ISO9660 way, where the files are just normal files in the ISO9660 file system. In this case all boot loaders (those that use UEFI and those that don't) have to understand ISO9660 (which is easier for those that don't use UEFI, and faster for all of them).
• the other FAT way, where nothing uses ISO9660 at all and all files (for all architectures) are included in the "FAT UEFI system image". This is a bad idea because normally you want to have things like documentation and installation instructions that other OSs can read on the same CD (e.g. text or HTML files for Windows, Linux, etc).
• the other other way, where nothing uses ISO9660 at all, only UEFI boot loaders are in the "FAT UEFI system image", and you use any file system you like (e.g. design your own) instead of ISO9660 for almost all of the files. This is also a bad idea (same problem as the "other FAT way") but may have potential advantages maybe (e.g. your own file system might be better than ISO9660 and FAT for unknown reason/s - more efficient, support some feature you want, etc).

Additional Notes:

1) When booting from BIOS; your boot loader could auto-detect whether the CPU supports 64-bit or not, and could automatically load a 64-bit kernel (if the CPUs support it) or load a 32-bit kernel.
2) When booting from 32-bit UEFI; your boot loader could auto-detect whether the CPU supports 64-bit or not and decide which kernel to use (just like the BIOS boot loader)
3) If the OS is designed well there should be no reason why BIOS boot loaders and UEFI boot loaders can't start the exact same (32-bit and 64-bit) kernels

This means that you can have a "multi-architecture boot CD" that contains 3 boot loaders (BIOS, 32-bit UEFI and 64-bit UEFI) and 2 kernels (32-bit and 64-bit); and that CD can automatically work (and automatically start the best possible kernel) without any end user hassle at all, on every 80x86 that is capable of booting from CD. It also means that the kernels themselves are part of the "files that are shared by multiple different boot loaders".


Cheers,

Brendan

[Antti]

This is becoming clear. I have been reading the UEFI specification and everything Owen and Brendan said is true. However, I have a plain "/EFI/BOOT/BOOTX64.EFI" file in the ISO 9660 file system and VirtualBox boots it just fine so I thought it is officially supported. I wish it did not work, so I would not have had this misconception.

[Owen]

VirtualBox implements an ISO9660 file system driver for UEFI.

Being as this is presumably open source, one option for accessing files direct from the ISO 9660 FS might be to install and enable this driver in the boot program you place inside the FAT image; though I admit I haven't investigated the difficulty of doing so.

[MadZarx]

WoW thank you Brendan. Now everything is clear to me. I didn't know any of the things you said but now I know how to have a multi architecture CD image (Currently I work on 32bit kernel )

But as you said, both BIOS and UEFI use a boot catalog entry to find the boot (file/image). When making the ISO, I tell the mkisofs where the bootloader is with -b option and it will set it as the BIOS bootloader. But what about UEFI? Does this need another mkisofs parameter to tell it where is the UEFI bootloader image?

I think I should change the title of this topic to Questions about bootloaders