Making a bootloader for UEFI and BIOS at the same time.

[liwinux]

Hi ,

So I already have written my BIOS boot-loader that basically read a FAT16 filesystem and loads a file called "KERNEL.IMG" into memory and jumps to it. Now, I'm also interested into making my boot-loader compatible for UEFI. So, I grabbed a copy of debian.iso and looked at how partition were made as when as how the MBR and EFI partition where laid-out. From my understanding, they are using a hybrid-mbr kind of thing. I tried to look that up, and surprisingly, few results appeared during my research. I also saw that they were using "Xorriso" to make it boot with a CD.

I'm trying to wrap my head around that and assembling the parts, but I don't really understand how they managed to get it working.
How does the concept of hybrid-mbr works. I know that when a EFI BIOS boots, it reads the EFI tab partition and loads the bootloader, whereas a BIOS computer loads the mbr, executes the code of the boot-loader and that's it. But I'm having a hard time to image of both of them would work with a hybrid-mbr. Obviously two boot loader should be written (one for the BIOS and one for the UEFI) The only way I see them being "sort of connected", would be that they both read a FS, load the file and VOILA ! I'm not sure about that either..
If someone could possibly explain the process under the hood for example for GRUB (or any other UEFI-BIOS bootloader ) works and for example how debian used it to boot in BIOS and UEFI that would help trying to understand what's going on.

PS: I'm writing a boot-loader just to gain experience and better understand the lowest-level of a computer and nothing more, so reinventing the wheel is actually beneficial for me.
Thank you !

[nullplan]

Basic idea is that your kernel has two entry points. You make an entry point for your 16-bit loader, and the kernel is kinda left to its own devices there, and you make an entry point for the 64-bit UEFI loader. That one has to deal with the limitations of UEFI, to wit, you will not know what address the kernel ends up at ahead of time. UEFI loader is a UEFI application that interprets the file system to load the kernel file into memory, then call ExitBootServices() and transition to the kernel.

[rdos]

BIOS and UEFI rely on different partition schemes, so I don't think you can create a "hybrid mbr". UEFI doesn't use the mbr, and instead use the GPT partitioning scheme, which allows for supporting discs that has more than 4G sectors. In essence, a MBR partitioned disc must be booted with the legacy boot method, while the UEFI bootloader will only boot from a GPT partitioned disc containing an EFI system partition with the required files.

As for how to handle both long mode booting (64-bit UEFI) and legacy boot, you will need some common interface. You could decide to make UEFI native, and let the legacy boot setup your environment the same way as UEFI does. Since my OS was initially written to support MBR boot, and not UEFI, I have a stub that switches from long more to protected mode and then creates a similar protected mode environment as the MBR loader.

[Octocontrabass]

If someone could possibly explain the process under the hood for example for GRUB (or any other UEFI-BIOS bootloader ) works and for example how debian used it to boot in BIOS and UEFI that would help trying to understand what's going on.

It boots four ways: BIOS USB drive, UEFI USB drive, BIOS optical drive, and UEFI optical drive.

For the BIOS USB drive boot, it has a MBR. Things work mostly the same as any BIOS MBR, except some BIOSes are picky about the MBR contents and will misbehave if the partition tables aren't satisfactory. GRUB uses the MBR to load its next stage, and that stage includes enough code to load the rest of GRUB from a filesystem.

For the UEFI USB drive boot, it has a FAT32 filesystem in a MBR or GPT partition. The firmware locates the appropriate default EFI boot executable according to the CPU architecture and loads it, and that boot executable is GRUB.

For the optical drive boot, it has an El Torito boot record with two entries for BIOS and UEFI. The BIOS binary includes enough code to load the rest of GRUB, like the MBR. The UEFI "binary" is actually a FAT filesystem that contains the same default EFI executable.

GRUB smooths out the differences and provides a consistent environment for the kernel.

I don't think you can create a "hybrid mbr".

Linux distros have been doing it for years and it works fine.

UEFI doesn't use the mbr, [...] the UEFI bootloader will only boot from a GPT partitioned disc containing an EFI system partition with the required files.

Microsoft requires UEFI to support booting from ordinary FAT MBR partitions on USB drives. It's internal drives where UEFI needs GPT with an EFI system partition. (And legacy BIOS doesn't care how internal drives are formatted, so you can boot from GPT partitions there.)

[zaval]

Microsoft requires UEFI to support booting from ordinary FAT MBR partitions on USB drives. It's internal drives where UEFI needs GPT with an EFI system partition. (And legacy BIOS doesn't care how internal drives are formatted, so you can boot from GPT partitions there.)

It's not Microsoft, that "requires", it's simply the UEFI spec, that says (and it's logical imo), that every discovered FAT formatted volume should be mounted and exposed, regardless of the partitioning scheme and partition type ("normal" FAT vs. ESP FAT, except different IDs, they don't differ). UEFI uses MBR just as fine as it does GPT. What it doesn't is the MBR code.

You can install your loader into such an ordinary FAT volume and use it as "ESP" without marking it ESP. I do all my testing with MBR partitioned USB sticks, having ordinary, not ESP marked FAT volumes on them. It's much easier, because diskpart doesn't like more, than one ESP found. Because of this whim, I don't mark FAT volumes on my .vhds as "ESP" and still OVMF sees and uses them as supposed. Even as installation targets. For the loader, I mean: FilePathList[0] of the Load Option, which contains a Device Path instance, pointing the UEFI Boot Manager to the loader file. I've been making them to point to ordinary FAT volumes on vhds all the time and these Load Options work.

Allowance of booting from any FAT volume is actually a really powerful and aiming feature of UEFi, because it allows one to have an installation source to be merely a folder in the root directory of the USB stick FAT volume. No need to force the user to format that stick into something "speshul", overwriting stick contents, just unpack the installation archive into the directory with a predefined name and direct the UEFI Boot Manager (or "shell") to start your loader from there.


For the OP, share the code where possible, but create 2 loaders: a normal UEFI loader executable for UEFI, without uglifying the concept (as linux does) and a raw binary for the legacy BIOS.

[Octocontrabass]

It's not Microsoft, that "requires", it's simply the UEFI spec, that says (and it's logical imo), that every discovered FAT formatted volume should be mounted and exposed, regardless of the partitioning scheme and partition type ("normal" FAT vs. ESP FAT, except different IDs, they don't differ).

But it is Microsoft that requires it, since that's how the Windows installer boots. Firmware doesn't always follow the spec very well, but it does boot Windows.

[scdbackup]

Hi,

Debian ISOs are known to boot on the largest number of computers but their
partition layout is a bit weird in order to achieve that bootability.
(Originally invented by Matthew J. Garrett for Fedora.)
See for a short explanation:
https://wiki.debian.org/RepackBootableI ... 64_or_i386

The theory is described in
https://dev.lovelyhq.com/libburnia/libi ... ectors.txt
See there
"EL Torito CD booting, for PC-BIOS x86, PowerPC, (old) Mac, EFI."
"Boot Info Table and GRUB2 Boot Info"
(Debian uses ISOLINUX for BIOS. So only "Boot Info Table" applies.)
"Master Boot Record (MBR), for PC-BIOS x86 from (pseudo-) hard disk"
"- SYSLINUX isohybrid for MBR, UEFI and x86-Mac"
(Debian does not provide a HFS+ filesystem image. So the "x86-Mac"
does not really apply.)

Other distros have meanwhile switched to more regular partition layouts
by appending the EFI partition after the end of the ISO 9660 filesystem
and by deciding for either MBR partition table or GPT.
These layouts are more acceptable to partition editors but also leave
behind some oddly programmed firmwares which insist in MBR partition
boot flag or the existence of a GPT. Only one of both can be fulfilled
without bending the various involved specs.

See for example the output of

Code: Select all

xorriso -indev archlinux-2023.12.01-x86_64.iso -report_system_area plain

which marks an appended partition in MBR (but also has an invalid GPT for
luring the most dumbest EFI implementations).
Another example is

Code: Select all

xorriso -indev Fedora-Everything-netinst-x86_64-37-1.7.iso -report_system_area plain

which marks an appended partition by a valid GPT. Further it uses GRUB
as bootloader for BIOS. (See in boot_sectors.txt "GRUB2 Boot Info" and
"- GRUB2 grub-mkrescue MBR".)

If you replace in above commands "plain" by "as_mkisofs" then you will
be shown a best guess about which mkisofs emulation options might be used
for producing an ISO with similar boot lures.


> Obviously two boot loader should be written (one for the BIOS and one
> for the UEFI) The only way I see them being "sort of connected", would
> be that they both read a FS

Yes.

ISOLINUX and GRUB let their boot program for BIOS (i.e. the x86 machine
code in the MBR) jump onto their El Torito boot images for BIOS. That's
why xorriso has to patch the block address of the El Torito image into the
code in the MBR.
This way optical media and USB sticks get booted by nearly the same
bootloader code.

EFI needs a FAT filesystem with /EFI/BOOT/BOOT*.EFI programs. For 64-bit
x86 it is BOOTX64.EFI, for 32-bit x86 it is BOOTIA32.EFI.
The FAT filesystem is then advertised in both boot lures, El Torito
Catalog for optical media and partition table for USB sticks.
This way both media families get booted by exactly the same bootloader
code.

About all distros use in their ISOs GRUB for EFI, because SYSLINUX EFI
has a bug about block size which prevents it from working with optical
media. Knoppix ISOs are an example of SYSLINUX EFI.
(I dimly remember to have seen a patch for fixing that bug.)

Have a nice day

Thomas

[rdos]

Microsoft requires UEFI to support booting from ordinary FAT MBR partitions on USB drives. It's internal drives where UEFI needs GPT with an EFI system partition. (And legacy BIOS doesn't care how internal drives are formatted, so you can boot from GPT partitions there.)

At least one of my laptops will not boot an USB MBR partitioned USB drive if it has an internal drive that is GPT formatted. UEFI boot has priority over MBR boot. I don't think this is only an issue on my particular UEFI BIOS, rather common. What this means is that it's better to format USB drives with UEFI if the BIOS support UEFI, otherwise it will likely not want to boot from it.

As for dual partitioning, I don't see the point with it. My MBR boot sector requires an additional number of sectors after it (stage 2), and so this will waste sectors at the start of the drive for no valid reason.

[scdbackup]

Hi,

rdos wrote:
> At least one of my laptops will not boot an USB MBR partitioned USB drive
> if it has an internal drive that is GPT formatted. UEFI boot has priority
> over MBR boot. I don't think this is only an issue on my particular
> UEFI BIOS, rather common.

That's the first time i learn about this behavior. I know of EFIs which
demand the shadow of a GPT (namely older OVMF for qemu). I know of Legacy
BIOS which does not boot from MBR code if no partition with boot flag is
present (namely old HP laptops).
But unsurpassable influence by a neighboring hard disk is new to me.

Whatever:
It is important not to confuse the difference between MBR partition table
and GPT with the difference between Legacy BIOS and (U)EFI.

UEFI specs explicitely mention that a MBR partition of type 0xEF is to be
considered an EFI System Partition, as is prescribed for a GPT partition
with Type GUID C12A7328-F81F-11D2-BA4B-00A0C93EC93B.
(Actually i do not find in UEFI 2.6 the Microsoftish rule that any
partition with a FAT filesystem shall be considered for /EFI/BOOT/*.EFI.
But the author of program Rufus repeatedly stated that this is indeed a
reliable behavior with existing EFI implementations.)

As for booting Legacy BIOS from a GPT partitioned drive:
UEFI specs define that a GPT begins by an MBR with a single partition in
its table which must be of type 0xEE and begin at block 1. The code area
of the MBR is to be ignored by UEFI and thus can be used for the first
stage boot code which a Legacy BIOS will execute.

Have a nice day

Thomas

[rdos]

As for booting Legacy BIOS from a GPT partitioned drive:
UEFI specs define that a GPT begins by an MBR with a single partition in
its table which must be of type 0xEE and begin at block 1. The code area
of the MBR is to be ignored by UEFI and thus can be used for the first
stage boot code which a Legacy BIOS will execute.

Yes, and so you cannot put stage 2 of the bootloader directly after the MBR, which I do when I partition a drive for MBR boot. I know Linux doesn't do it this way, but you still need a redundant copy of stage 2 for dual boot. To avoid these redundant copies of stage 2 (or the EFI boot file), it's better to partition a USB drive either as MBR or UEFI.

Also, a second stage MBR loader that is able to boot both from MBR and GPT will need code for both schemes, which makes it larger and more complex.

[zaval]

But it is Microsoft that requires it, since that's how the Windows installer boots. Firmware doesn't always follow the spec very well, but it does boot Windows.

Whoever required that is a good person. Because it would be plain stupid to reject allowing to load files from a volume with the same FS type, just marked differently in the partition table.

Actually i do not find in UEFI 2.6 the Microsoftish rule that any
partition with a FAT filesystem shall be considered for /EFI/BOOT/*.EFI.

this "/EFI/BOOT/*.EFI" is not only not the main path for loading, but not even guaranteed. it is only for removable media. for non removable, it's optional at all. it's just a "default" or resort path when there is no other Load Options present. For the topic it is important to remember, that the Boot Manager's "Load from File" or the shell's start mechanism are also "Load Options" and they in my opinion are a much better way for starting the loader (preinstallation phase) than relying on this abused "\efi\boot\.efi" way. an OS developer has 0 reasons to stick with it. ah, except m0r0nic firmwares, that think, it's the only way.

In fact, in the chapter 3.3 2.4 Errata B, it says what you say it doesn't. It just doesn't call it "Microsoftish".

The default state of globally-defined variables is firmware vendor specific. However the boot
options require a standard default behavior in the exceptional case that valid boot options are not
present on a platform. The default behavior must be invoked any time the BootOrder variable
does not exist or only points to nonexistent boot options.

If no valid boot options exist, the boot manager will enumerate all removable media devices
followed by all fixed media devices. The order within each group is undefined. These new default
boot options are not saved to non volatile storage. The boot manger will then attempt to boot from
each boot option. If the device supports the EFI_SIMPLE_FILE_SYSTEM_PROTOCOL then the
removable media boot behavior (see Section 3.4.1.1) is executed. Otherwise, the firmware will
attempt to boot the device via the EFI_LOAD_FILE_PROTOCOL .

combine this with the requirement to instal SFSP instance one every FAT volume discovered
and you will get that "Microsoftish" rule.

btw. note, that "exceptional case". UEFI spec clearly tells, what it thinks "efi\boot" is. It's a resort point, that can
be avoided even for installation cases: one needs to rely on the "break into the firmware and start the loader via 'Load from file' or the shell" way. why it's much better? because it's more convenient and controlled, less confusing and way more powerful and because it is how it was supposed to be. before it got spoiled by the collective misunderstanding with this damn bootia32.efi/bootx64.efi nonsense.

UEFI boot has priority over MBR boot. I don't think this is only an issue on my particular UEFI BIOS, rather common. What this means is that it's better to format USB drives with UEFI if the BIOS support UEFI, otherwise it will likely not want to boot from it.

It's not true. UEFI Boot Manager (BDS phase, BDS is Boot Device Selection, just in case) processes Load Options created on the platform (BootXXXX vars) and if there are such, it picks the one at index 0 of BootOrder variable (or shows the whole array in form of Load Option table if the timeout is not 0 and Boot Manager is kind enough to show the the table) and starts the loader of the user picked Load Option. The loader is pointed to by FilePathList[0] DevicePath instance of the picked Load Option. It can point to a FAT volume, NOT marked as ESP and be sitting on a disk, partitoned with MBR. there is no any "priorities" other, than BootOrder array. Which is made by OS installers, blatantly modifying it for their discretion, good when in cooperation with the user.
If your laptop rejects to boot from a MBR partitioned USB with non ESP FAT, whereas it has GPT disks inside, it's that laptop FW bug, 100%. Your generalizing here looks a bit misleading for those, who would read this.

Forcing users to format USB stick with some special tools, yet with the requirement of making the stick ESP, will only decrease chances for your OS to be tried by those few enthusiasts, that were unlucky to think about that.

[scdbackup]

Hi,

rdos wrote:
> a second stage MBR loader that is able to boot both from MBR and GPT
> will need code for both schemes, which makes it larger and more complex.

GRUB has this code and is indeed fairly complex.

When creating a bootable hard disk one will normally decide for one
boot path depending on the mainboard firmware and the needs of the
operating system(s).
The need for hybrid jackalopes arises when a drive or a drive image
is supposed to encounter various machine firmwares. That's the case with
many Linux distro installation ISO 9660 images.

I hope to have given initial answers to the original question how the
Debian projects debian-cd and debian-live get this done by help of
xorriso. Another example of a multi-firmware xorriso customer would be
the GRUB program grub-mkrescue.

zaval wrote:
> In fact, in the chapter 3.3 2.4 Errata B, it says what you say it doesn't.

(nitpicking mode on)
But this text talks of "devices", not of "filesystems". The way how a
device is mapped to a filesystem and ultimately to a .EFI program is
not specified.
(nitpicking mode off)

Whatever, i do not doubt that acceptiance of partitions of arbitrary type
is as much a reliable firmware feature as anything that's easier to read
from the specs.

Have a nice day

Thomas

[rdos]

Forcing users to format USB stick with some special tools, yet with the requirement of making the stick ESP, will only decrease chances for your OS to be tried by those few enthusiasts, that were unlucky to think about that.

On the contrary. If you rely on bootloaders and special MBRs, your testers will have a lot of trouble testing your OS. The nice thing about the UEFI solution is that you can format the USB drive for GPT using Windows or Linux, create the EFI system partition, and then put the OS specific files in the filesystem. That's much easier than all the strange stuff needed for MBR boot or dual boot.

So, I recommend users to format USB drive for EFI boot on first installation, and then to use the RDOS command shell for creating the MBR stuff (or formating for GPT & creating EFI system partition).

this "/EFI/BOOT/*.EFI" is not only not the main path for loading, but not even guaranteed. it is only for removable media. for non removable, it's optional at all. it's just a "default" or resort path when there is no other Load Options present. For the topic it is important to remember, that the Boot Manager's "Load from File" or the shell's start mechanism are also "Load Options" and they in my opinion are a much better way for starting the loader (preinstallation phase) than relying on this abused "\efi\boot\.efi" way. an OS developer has 0 reasons to stick with it. ah, except m0r0nic firmwares, that think, it's the only way.

Yes, because some operating systems "mess up" stuff (Windows) so nothing else can be installed. In my experience, if you remove the system partition completely on a non-removable drive, the BIOS will typically revert back to "default". That's the only way this can be done on some BIOSes that don't have a "reset" option.

It's not true. UEFI Boot Manager (BDS phase, BDS is Boot Device Selection, just in case) processes Load Options created on the platform (BootXXXX vars) and if there are such, it picks the one at index 0 of BootOrder variable (or shows the whole array in form of Load Option table if the timeout is not 0 and Boot Manager is kind enough to show the the table) and starts the loader of the user picked Load Option. The loader is pointed to by FilePathList[0] DevicePath instance of the picked Load Option. It can point to a FAT volume, NOT marked as ESP and be sitting on a disk, partitoned with MBR. there is no any "priorities" other, than BootOrder array. Which is made by OS installers, blatantly modifying it for their discretion, good when in cooperation with the user.

Yes, this is the path Windows use to abuse things. They change the load options, and then the BIOS manufacturer helps them by not having an option to reset load options. As I noted above, this can sometimes be fixed by erasing all partitions on the non-removable media, let the BIOS try to boot (and fail), and then you have the default options instead that are not BIOS dependent.

[zaval]

(nitpicking mode on)
But this text talks of "devices", not of "filesystems". The way how a
device is mapped to a filesystem and ultimately to a .EFI program is
not specified.
(nitpicking mode off)

that's why I wrote this as a followup:

combine this with the requirement to instal SFSP instance one every FAT volume discovered
and you will get that "Microsoftish" rule.

"devices" are storage (block) devices (Block IO Protocol installed), "file systems" are those AND SFSP. And... this is any (as per the spec) FAT volume:

The firmware automatically creates handles for any block device that supports the following file
system formats:
•
FAT12
•
FAT16
•
FAT32

Whatever, i do not doubt that acceptiance of partitions of arbitrary type
is as much a reliable firmware feature as anything that's easier to read
from the specs.

sigh. okay, NOT ARBITRARY partition type, but of any FAT formatted
volume, not just ESP. is it clear now? and again, this is what one would read
from the spec if he wanted to.

On the contrary. If you rely on bootloaders and special MBRs, your testers will have a lot of trouble testing your OS. The nice thing about the UEFI solution is that you can format the USB drive for GPT using Windows or Linux, create the EFI system partition, and then put the OS specific files in the filesystem. That's much easier than all the strange stuff needed for MBR boot or dual boot.

and where did I say about "special MBRs"? I was talking about this: with UEFI, you create your installation package as an archive, your users download it and put into a directory, say named "rdos" in the root directory of a USB stick. then, the users insert it into their machines, stop in the FW, either in the Boot Manager or enter the UEFI shell and start your OS loader, via eg hitting the "load from file" option (did you see that? good, that means that FW isn't hopeless POS) or type in say fs0:\rdos\boot\loader.efi in the shell.
then the loader loads the OS in the preinstallation session and then installation (or live session) happens. No need to require user to do anything other than that. No USB stick format, backing up its data, no any other crazy trickery (hybrid/mutant MBR, hexediting manually MBR code/partition table). unfortunatelly, some lost FW implementations screw up even such a brilliant solution, but, what can be said here? as we can see, you already know, who is to blame and I don't want to add to this discussion, it already, without this "valuable" Microsoft bashing has deviated from what the OP asked about a lot.

So, I recommend users to format USB drive for EFI boot on first installation, and then to use the RDOS command shell for creating the MBR stuff (or formating for GPT & creating EFI system partition).

there is no such notion as "EFI boot". Honestly, no offense, judging by your usage of the UEFI related terminology, you need to strengthen a bit your understanding the standard. For example, the OS installer must create a Load Option and it should modify BootOrder. Whereas you think, it's "evil" Microsoft, who does this. How would you act, if your users want your OS to be the 1st in the OS table list?

[rdos]

and where did I say about "special MBRs"? I was talking about this: with UEFI, you create your installation package as an archive, your users download it and put into a directory, say named "rdos" in the root directory of a USB stick. then, the users insert it into their machines, stop in the FW, either in the Boot Manager or enter the UEFI shell and start your OS loader, via eg hitting the "load from file" option (did you see that? good, that means that FW isn't hopeless POS) or type in say fs0:\rdos\boot\loader.efi in the shell.

Well, that works if the BIOS has a EFI command line (shell), but if it doesn't you are better off with a USB stick that has /efi/boot/bootx64.efi (and /efi/boot/bootia32.efi if the machine is 32-bit). I'm aware that these locations and pathnames might not be in the UEFI specification, but they appear to work on all machines where I have tried them, so they are at least inofficial standards for x86.

I can also load my OS with multiboot, but it requires loading a "fake" loader and to load the OS image as a module. Still, this works too, but is not so useful as most users will not be able to do this.

For dual boot with Linux, I would hook onto Linux boot configuration. Still, I usually install RDOS as the only OS, and then I typically either create a new MBR + add stage 2 after it or create the EFI system partition and add the "standard" files to it.

there is no such notion as "EFI boot". Honestly, no offense, judging by your usage of the UEFI related terminology, you need to strengthen a bit your understanding the standard. For example, the OS installer must create a Load Option and it should modify BootOrder. Whereas you think, it's "evil" Microsoft, who does this. How would you act, if your users want your OS to be the 1st in the OS table list?

Actually, I have no idea how to modify Load Options or BootOrder outside of the EFI environment where the OS loader would run. So I depend on the "standard" on how the file system is organized in the absence of OSes fiddling with those.