MBR, VBR, Floppy, USB, CD, UEFI and hard drive intrinsics

[AJ]

That reminds me, I have a drawer full of floppies with stuff like MS-DOS, Star Trek 25th Anniversary, Hitchhikers Guide etc... I must get around to backing those up while my ancient external USB FDD still works!

Cheers,
Adam

[Antti]

Few weeks ago I started writing a boot loader again and now I re-read this old thread. I would like to bring up some new thoughts. First of all, I have a question for you all. How many of you have created an OS that can install itself on HDD? For example, booting from CD and running an installer?

I have planned to experiment with a toy-OS installer. In this case, the OS is not the main target of interest. It does not have to be a general-purpose OS at all. The main thing is just the ability of installing software on HDD that is capable of booting and starting something up. The main procedure from a user's perspective:

1. Download installation media, e.g. ISO 9660 image file (".ISO").
2. Burn the image on CD.
3. Boot your computer from the CD.
4. Run the installer and notify big warning messages (erasing all current data on disk).
5. Remove media.
6. Restart your computer.
7. Use the <whatever software I am shipping>.

Other media types for booting can be supported also. Like you can see, I will greatly simplify the installer because I do not care of keeping current partitions, data, or anything. Installer will always take the whole drive (or colloquially: "the computer"). No dual-booting. In practise, a user can take some abandoned computer and make it "useful" again. I find this very neat. To make things more fun, I planned to go on with the GUID Partition Table (GPT) partition system. Also, UEFI booting will be supported so the computer can be "legacy-free". Maybe there are some problems that I am not yet aware of but hopefully I can solve them as I proceed.

Any similar plans going on right now?

[Brendan]

Hi,

I have planned to experiment with a toy-OS installer. In this case, the OS is not the main target of interest. It does not have to be a general-purpose OS at all. The main thing is just the ability of installing software on HDD that is capable of booting and starting something up.

For El Torito, the firmware is meant to find a suitable "boot record". This means that the same CD can have boot loaders for 80x86 BIOS and 80x86 UEFI (and others for Itanium, PowerPC, etc) and the right boot loader will automatically be used. Of course the same applies for booting from network, or USB, or anything else (e.g. you can have BIOS and UEFI boot loaders on the same USB flash stick).

For my OS, there's several kernels and the boot code auto-detects some things (e.g. if the CPUs all support long mode or not) and determines which kernel is the most suitable kernel to use.

Most OSs have an "auto-update" feature where they'll ask a repository for newer versions. Because my OS is a micro-kernel, it should be possible to update most of the OS without rebooting.

Most OS's now have some sort of "live CD"; with an OS installer implemented as a normal utility that runs under the OS itself.

Combining these 4 things means that I can have a single CD (or USB flash stick, or "network boot server"), that supports BIOS and UEFI, and 32-bit and 64-bit 80x86; where the OS can update itself after boot, and can be used "live" or as an OS installer. It also means that you could boot, run the OS installer, remove the CD, then use the OS without rebooting at all.

Also note that during a typical OS installation a lot of time is spent copying files; which means reading data from one device (and optionally decompressing it) and writing data to a different device. Ideally you want to be reading, decompressing and writing all at the same time; and this requires native drivers (using the firmware means that you can only do one thing at a time, and it will take at least twice as long to install the OS). It'd be nice if the OS installer used "pretty graphics" too (e.g. a user-friendly GUI thing) - drop down "default language" selection, a graphical partition management, progress bar/s, etc. Of course the OS installer would also need networking support (for auto-updates, for using NTP to avoid asking the user if CMOS/RTC is set to local time or UTC, possible remote desktop, etc), user input devices (keyboard, mouse, touchpad, etc), file system support, partition management tools, etc.

Mostly what I'm suggesting here is that if you make a separate "OS installer" (that isn't built on top of your OS and does everything itself); then you'll end up writing most of your OS twice (once for the OS, and then again for the separate OS installer). Basically; it's probably a lot smarter to have an OS installer that relies on your OS's functionality to avoid duplicating everything.

First of all, I have a question for you all. How many of you have created an OS that can install itself on HDD? For example, booting from CD and running an installer?

I never seem to get that far - I concentrate on getting the base (kernel, drivers, etc) "perfect" before worrying about building things on top of that base..


Cheers,

Brendan

[Owen]

First things first: EFI is a "modern" firmware standard for the "modern" world. Being sufficiently modern, it does not support ISO9660, UDF, or any other such file system. Why would you possibly want to use those on a CD?

Instead, EFI requires you to use El-Torito. El-Torito encodes an EFI platform flag. Therefore, you need to embed multiple bootloaders inside one (multi-arch) image.

Now its' time to break the whiskey out, becasue the way you do this... is by embedding a FAT image as the El-Torito image.

You might want to also support what lots of Linux distributions do these days, namely allowing your ISO image to be written to a USB stick. This is where things get tricky:

• You need to use the 16 ISO9660 reserved sectors to embed an MBR and GPT. (to appease some over-sensitive BIOSes and EFI implementations respectively). This MBR contains your BIOS-from-USB bootloader
• One of the partitions in this GPT needs to be a "protective" partition which covers your ISO image
• The other partition needs to be a FAT partition, exactly the same one as you use in El-Torito boot... so, you have a partition table entry for an EFI System Partition which points into the middle of your ISO image to where your El-Torito image is

But then it gets better!

• OIder Macs don't support booting from El-Torito, so you need to embed an Apple Partition Map on your CD
• Apple Partition Maps start at byte 0 of sector 0
• This is where your MBR is...

Fortunately, the required bytes of the APM do things which are pretty harmless when interpreted as x86 machine code, so the fact that BIOS-based machines are going to have to execute a bit of APM is merely disgusting, not harmful.

Oh. By the way, a subset of those Macs which don't do El-Torito also don't boot from FAT. So you need a HFS+ image too.

So, now, the necessary contents of your partition tables:

• MBR: Whole disc with partition type 0 (Because you'll want to mount the ISO); needs to be type 0 because EFI sensitive implementations ignore that type when validating the disc; EFI system partition
• GPT: Whole disc (for same reason as MBR), EFI system partition,
• APM: Whole disc, HFS+ partition

Necessary El-Torito images:

• BIOS boot sector for CD-ROM
• EFI FAT partition
• EFI HFS partition (not necessary, but this way genisoimage will use the correct alignment)

Necessary boot loaders:

• BIOS boot sector for CD-ROM
• BIOS boot sector for HDD-with-ISO9660-image
• EFI boot loader for each architecture (duplicated twice)

All we have lost is sanity.

PS: Some EFI/BIOS hybrids get sufficiently impressed by this that they'll offer your users the following useful menu:

If you're lucky, they'll have remembered to enable the keyboard driver, and your user will manage to select the EFI option. I would suggest making the EFI image first in the list, to ensure this, but I suspect this would mean some whole other set of BIOSes would just crash, or try to load your UEFI image as a BIOS image.

PPS: Every Mac which understands EFI booting properly is now going to give your users the following menu:

1. CD
2. CD
3. Windows

Because they've spotted the APM, UEFI-El-Torito, and BIOS boot images, and to a Mac's firmware all BIOS boot loaders boot Windows

PPPS: Windows installs BOOTX86/BOOTX64.EFI images on the system partition. Some firmware manufacturers therefore deduced that their "EFI implementation" consequently didn't need to support the standard boot-variable based boot system. This is a novel interpretation of the UEFI specification

[Antti]

Basically; it's probably a lot smarter to have an OS installer that relies on your OS's functionality to avoid duplicating everything.

This makes sense.

You might want to also support what lots of Linux distributions do these days, namely allowing your ISO image to be written to a USB stick.

I am not going to do that. It feels like a hack. I ship multiple formats suitable for different media. When it comes to other points you made, I did not understand your stream of thoughts. That does mean your points are not valid but the image I made will not be "MBR+GPT+ElTorito+ISO9660+FAT" combination (what a mess).

[Owen]

Your CD will still be an El-Torito+FAT+HFS++ApplePartitionMap mess if you want to support UEFI and older Macs.

Your flash drive will still need an odd MBR+GPT hybrid in order to support the range of BIOSes and EFIs.

Personally, I lean towards the "all in one image" because the Linux distros have done all the work for you and named it isohybrid.

[Antti]

That does mean your points are not valid but...

I meant to say "That does not mean...".

Personally, I lean towards the "all in one image" because the Linux distros have done all the work for you and named it isohybrid.

This is a little bit problematic because now users know they can usually use ISOs as disk images. I have to think about this...

[Brendan]

Hi,

Your CD will still be an El-Torito+FAT+HFS++ApplePartitionMap mess if you want to support UEFI and older Macs.

Estimate a date when the OS will be "finished" (e.g. complete enough for people to use as their main OS). Then write a list of everything the OS needs and give each of those things a rating based on something like "rating = (how important it will be for end users when the OS is finished + how important it is for the development of the OS) * how hard it is to implement"; and sort that list in order of rating. Where would older Macs and 32-bit 80x86 UEFI systems be on that list?

For example: At the moment only about 4% of people use Apple, and it's probably reasonable to assume only a quarter of them are still using older 80x86 Macs, so that's 1% of people. If I estimate that my OS will be "finished" in 2020 and assume the number of older Mac users is going to halve every 2 years, then you could say that support for HFS+ and ApplePartitionMap is something that roughly 0.1% of potential end users will care about when the OS is "finished". Unlike FAT and ISO9660 (which are things your OS is likely to want for other reasons) HFS+ isn't a simple file system, so that pushes support for older Macs even further towards the bottom of the list. If you support BIOS then you don't need to care much about "generic white box" 32-bit 80x86 UEFI systems (as they're extremely rare and probably support BIOS anyway), so you'd mostly only really need to care about 32-bit 80x86 UEFI systems for older Macs; so that's another small increase in "how hard it is to implement" that pushes support for older Macs a little further towards the bottom of the list.

Basically; it's not worth bothering with support for older Macs or 32-bit 80x86 UEFI systems.

For a CD; for "BIOS, no emulation El Torito" the boot loader would use firmware to read sectors and understand enough ISO9660 to read files that the OS needs during boot (e.g. before it has native drivers, etc working); and for "UEFI, FAT image El Torito" the boot loader would use firmware to read sectors and understand enough ISO9660 to read files that the OS needs during boot.

Your flash drive will still need an odd MBR+GPT hybrid in order to support the range of BIOSes and EFIs.

For booting (where you're using firmware to read sectors); USB flash isn't very different to USB external hard drives, which aren't very different to SATA hard drives. For all these cases I'd use GPT; with a UEFI system partition containing a UEFI boot loader and "slightly hybrid" MBR that chain-loads a BIOS boot loader in the OS's partition. Both boot loaders (for UEFI and BIOS) would understand enough of GPT and the OS's native file system to read files that the OS needs during boot.

There's also network boot. For PXE, newer (UEFI) clients are supposed to send a "system architecture tag" and the DHCP server is meant to use this tag to determine the file name of the boot loader it tells the client to use. This means you need 2 boot loaders for network boot (one for BIOS and another for UEFI).

That adds up to a total of 6 boot loaders to cover every way of booting every 80x86 machine that's worth bothering with. Once these are done, the next thing on the "sorted list of things the OS needs" (that has anything to do with booting) is probably going to be ARM systems.


Cheers,

Brendan

[egos]

First of all, I have a question for you all. How many of you have created an OS that can install itself on HDD? For example, booting from CD and running an installer?

I have such installer. It is a usual program for my OS. It can be run from LiveCD or other media. GPT partitioning scheme not yet supported (I plan to implement BIOS booting from GPT disks first, then to implement UEFI booting). Installation to NTFS volumes will not be supported natively at first time (I plan to implement only "read-only" support for NTFS first).

[h0bby1]

for the sector size it can be detected for lba device using int 13h ah=48h, it can be a bit bothering, but not that much, at the moment my bootloader of 512 bytes can handle both chs and lba mode with any sector size, it is a bit bothering if you have to deal with an iso into which all entry address are in sector size of 2048 and that you have to load it from a floppy or hard drives whose sectors are 512 bytes long, sector address from the filesystem need to be changed to fit the sector size of the device, because one iso sector will cover 4 physical sector, and some translation need to be made, but it's not that hard to handle all together, my bootloader can load indifferently file of the iso from a floppy, cd rom, or any other device the bios can support, either chs or lba with any sector size