MBR, VBR, Floppy, USB, CD, UEFI and hard drive intrinsics

[Antti]

Release formats:

- Floppy "1440 KB format" image. FAT12.
- Disk image (basically for USB stick). FAT12 or FAT16. UEFI boot.
- El-Torito CD. ISO 9660. UEFI boot.
- Multiboot specification compatible "files".

The only headache-causing format for me is the disk image. Previously I had an "MBR boot sector" that had a hard-coded partition table (LBA only) with one partition entry of type 0xEF (EFI). That "EFI partition" had FAT12 file system and a single file (UEFI boot loader) in it. The image was raw-written to a USB stick (e.g. dd utility). It made an assumption of the sector size being 512 bytes. However, the old-school BIOS booting and the modern UEFI booting were roughly supported.

Now things will be made robust. Is this even possible for disk images? I cannot know the details of the USB stick / hard drive (e.g. capacity, sector size). This makes an elegant "old" MBR partition table almost impossible. CHS addressing is another pain but but even LBA is quite ambiguous because of sector sizes. The simplest solution would be just discard the "MBR" and use "VBR" directly. It means I would put the floppy image to the USB stick / hard drive. Would the UEFI boot without partitions (just FAT12 or FAT16)? I also looked at GUID Partition Table but it seems overkill for this purpose.

I also read something about "BIOS-USB-floppy-emulation." Without partions, the USB stick might be interpered as a floppy? Basically I do not want to rely on emulations. It would best to have "a native hard disk drive" and read sectors with BIOS "int 0x13, ah=0x42" Extended Read Sectors From Drive interface. As far as I know, it does not support floppies (but how about the emulated ones?).

[bluemoon]

For BIOS's USB HDD emulation, all you need is dd the sectors to /dev/usb (or /dev/disk* in mac) as if it is HDD, and use sector size of 512B. This is at least bootable on my mac book with rEFI and into my VBR on USB andf then my OS.

Basically I do not want to rely on emulations. It would best to have "a native hard disk drive" and read sectors with BIOS "int 0x13, ah=0x42" Extended Read Sectors From Drive interface.

This sentense is self contradicts. BIOS emulate USB as HDD on 0x42 service, and sure you know USB (stick) is not an HDD, (USB HDD is another story)

[Antti]

For BIOS's USB HDD emulation, all you need is dd the sectors to /dev/usb (or /dev/disk* in mac) as if it is HDD, and use sector size of 512B. This is at least bootable on my mac book with rEFI and into my VBR on USB andf then my OS.

The problem is not that I could not do this and I am not questioning the fact that things probably work just fine. If I have a hard-coded partition table (because it is a disk image), I would assume the sector size is always the same. Extended Read Drive Parameters interface could report the bytes per sector to be "anything". Right? The Extended Read Sectors From Drive interface uses sectors to determine from where to read and how many sectors. Things are easy because sector size happen to be 512 byte almost always when there is an MBR partition table. CD is a good example of different size sectors (usually 2048).

Should I just decide that "partition table uses sector size 512, convert that information before using""

This sentense is self contradicts.

That is true. There is some contradiction because emulation cannot actually be avoided.

[bluemoon]

I don't have a dirve with 4096 byte sector, but according to wiki http://en.wikipedia.org/wiki/Master_Boot_Record#PTE:

The sector size used to be considered fixed at 512 (29) bytes, and a broad range of important components including chipsets, operating systems, database engines, partitioning tools, backup and file system utilities and other software had this value hard-coded. Since the end of 2009, disk drives using a new technology known as Advanced Format and employing 4,096-byte sectors (4Kn) have been available, although the size of the sector for some of these drives was still reported as 512 bytes to the host system through conversion in the hard drive firmware and referred to as 512 emulation drives

I would also expect a lot of things break if the BIOS AH=42 service take sector size other than 512B (mainly caused by buffer overrun).
It looks like to utilize 4096B sector you may talk to the devices directly, but with BIOS service's it's always 512B.
And I'm not sure about EFI.

[Antti]

I would also expect a lot of things break if the BIOS AH=42 service take sector size other than 512B (mainly caused by buffer overrun).

It already does it. I had El-Torito CD (no-emulation) booting and I used that "BIOS INT 0x13, AH=0x42" service to load sectors. The reported sector size was 2048 bytes and it worked using that size.

I also had this snippet in old discarded code:

Code: Select all

	; dl is set by BIOS

	mov ax, 0x4100
	mov bx, 0x55AA
	int 0x13
	jc short .error
	cmp bx, 0xAA55
	jne short .error

	mov ax, 0x4800
	mov dx, [.drive_index]
	mov si, .drive_parameters
	int 0x13
	jc short .error

%ifdef BOOT_FROM_CD
	cmp word [.bytes_per_sector], 2048
%else
	cmp word [.bytes_per_sector], 512
%endif
	jne short .error

	mov ax, 0x4200
	mov dx, [.drive_index]
	mov si, .disk_address_packet
	int 0x13
	jc short .error

[Combuster]

I had El-Torito CD (no-emulation) booting (...) The reported sector size was 2048 bytes and it worked using that size.

Yet if you deliberately make an no-emulation El-Torito image, you already know in advance you won't be using the legacy PC-style boot sequence, but something much more modern in design.

[Antti]

Yet if you deliberately make an no-emulation El-Torito image, you already know in advance you won't be using the legacy PC-style boot sequence, but something much more modern in design.

That El-Torito CD was an example of "BIOS INT 0x13, AH=0x42" not being hard-coded to 512-byte sector size. Of course my modern design uses that service if it is available (only for booting, however). It works really good for loading sectors from USB-stick.

It seems that I do not use any MBR with the disk image release format. I cannot confirm it now but I think that UEFI can boot from "plain FAT12/FAT16" also. It would be ridiculous if it could not do that.

[Brendan]

Hi,

I cannot confirm it now but I think that UEFI can boot from "plain FAT12/FAT16" also. It would be ridiculous if it could not do that.

The UEFI specification (in 12.3.1.1 File System Format) says "The EFI firmware must support the FAT32, FAT16, and FAT12 variants of the EFI file system".

For sector sizes; the original PC was designed to support different sector sizes, and floppy disk drive hardware is still capable of using different sector sizes. For example, in theory, you can format a floppy with anything from 128-byte sectors to 16-KiB sectors and the BIOS functions will work. The only restriction is that your sectors need to be at least 512 bytes to boot from floppy, because the BIOS expects a "0xAA55 magic number" at offset 0x1FE in the first sector (e.g. at offset 0x1FE, near the start of an 8 KiB sector). Also in theory; a hard drive could use 4-KiB sectors (without the slow "512-byte emulation") and it should be fine.

In practice, after about 30 years of only using 512-byte sectors for floppies and hard disks, I'd assume that most software that uses BIOS and most BIOSs themselves are broken and lots of things will fail in different ways if anyone attempts to use different sector sizes for floppy disks or hard disks (or devices that appear to be floppy or hard disks to software, like USB flash). Of course this means no hardware or firmware manufacturer would want the risk involved in using different sector sizes, which also means there's no reason for software developers to make sure their code actually works for different sector sizes, which... creates a feedback loop.

For UEFI it's the same - the UEFI spec defines block size as "Not less than 512 bytes", and (just like BIOS) in theory other sector sizes are possible. Hopefully other sector sizes will become common quickly enough to prevent the old "512-byte sectors assumed" feedback loop from establishing again.


Cheers,

Brendan

[Antti]

I have also read from many sources that FAT12/FAT16/FAT32 file systems are supported by (U)EFI. However, it confuses me whether that file system must be in "EFI partition" to be bootable. Here is the simple illustration:

Code: Select all

- Hard disk (No GUID partitions)
  - MBR
    - Partition table with "EFI partition"
  - "EFI partition"
    - FAT12/FAT16/FAT32 file system
    - e.g. \EFI\BOOT\BOOTX64.EFI

- "USB Stick"
  - FAT12/FAT16/FAT32 file system
  - e.g. \EFI\BOOT\BOOTX64.EFI

Will the USB Stick boot (guaranteed by standards)? How about if I forcibly write the hard disk to have "USB Stick" layout? It is not sensible but would it boot?

The UEFI specification explicitly requires support for FAT32 for system partitions, and FAT12/FAT16 for removable media; specific implementations may support other file systems.

Does this mean that above "Hard disk setup" can only have the FAT32 file system (support guaranteed). Does it automatically include FAT12/FAT16 if FAT32 is supported? Does this also mean that "USB Stick setup" cannot use FAT32?

I am really sorry that I am making problems bigger than they are. You all may see this being perfectly unambiguous. However, I think that there are several small details that are not very clear.

[Antti]

I am not a floppy specialist (a highly considered title) but what actually means formatting? I can see that there is "INT 0x13, AH=0x05" service that can format a floppy disk. There surely is an option for the sector size.

Let say that the floppy contains just zeros. Where that sector size information is? Is it a "hardware thing of the floppy?" If I have a floppy disk image, I do an assumption that the sector size of the floppy is "formatted" to that particular size (universally 512 bytes).

[bluemoon]

If a disk is not "low level formatted", any check bits may be random garbage and the read may return things like "CRC error".

[Brendan]

Hi,

Will the USB Stick boot (guaranteed by standards)? How about if I forcibly write the hard disk to have "USB Stick" layout? It is not sensible but would it boot?

As far as I can tell; UEFI's boot order is:

• Check if one or more "boot variable" are set, and try to boot from whatever the boot variable/s say in order. I don't think there's any restrictions here - e.g. if the boot variable points to a file called "foo.bar" on a partition that the firmware can understand then it doesn't matter if that partition is the EFI system partition or not (or even if the partition is not FAT if the firmware can understand it).
• If that didn't work; then for each removable device:
  • If the device supports the EFI_SIMPLE_FILE_SYSTEM_PROTOCOL, check for a file/s with the name "\EFI\BOOT\BOOT{machine type short-name}.EFI" (for each supported machine type) and boot it if it exists. Here I'd also assume that it doesn't matter if the partition is an EFI system partition or not, and also that the partition may be anything (e.g. if the firmware has optional/extra support for ext4, NTFS or HPFS, then it would attempt to boot from that even though it's not FAT). I also wouldn't make assumptions about case sensitivity (e.g. if the firmware has optional/extra support for a file system that allows both uppercase and lowercase, then "efi\boot\..." may or may not work when "EFI\BOOT\..." does).
  • Otherwise, if the device supports the EFI_BLOCK_IO_PROTOCOL then it tries to connect drivers to it (e.g. if the block device has partitions, then it might connect an "MBR partitions" driver to it to create 4 partitions; and then (recursively) treat each of those partitions as a new removable device.
• If that didn't work; then the firmware may or may not do the same "removable device search" for non-removable devices, and if it does do this search at all then it may have additional (implementation defined) restrictions.

What this mostly means is that (to boot reliably) your OS would need to be on a removable device that uses (EFI's) El Torito, MBR partitions or GPT partitions, and uses FAT32, FAT16 or FAT12, and has a suitable file name; or it would need to be on a non-removable device that uses (EFI's) El Torito, MBR partitions or GPT partitions, and uses FAT32, FAT16 or FAT12, and has a "boot variable" set.

It would also mean that if a (removable) USB flash device does boot; then simply copying it to a (non-removable) hard disk (without setting a "boot variable") may not work.


Cheers,

Brendan

[Brendan]

Hi,

I am not a floppy specialist (a highly considered title) but what actually means formatting? I can see that there is "INT 0x13, AH=0x05" service that can format a floppy disk. There surely is an option for the sector size.

There is. That BIOS function has a table with one entry per sector (e.g. for "18 sectors per track" you'd have a table of 18 entries). Each of those entries has a sector size; which means that you can format a disk with different sector sizes on the same track (and the floppy hardware itself supports this).

I've wondered if this capability could be used to squeeze even more data onto a floppy - larger sectors mean less wasted space between sectors, but if the sectors are too large you waste space where only part of a whole large sector will fit. For example, maybe two 8192-byte sectors won't fit on one track, but maybe an 8192-byte sector, a 4096-byte sector and a 512-byte sector on each track would fit (and would let you put 2000 KiB of data onto a "1440 Kib" disk).

Let say that the floppy contains just zeros. Where that sector size information is? Is it a "hardware thing of the floppy?" If I have a floppy disk image, I do an assumption that the sector size of the floppy is "formatted" to that particular size (universally 512 bytes).

A floppy disk has an "index" hole in it. This hole is used by the drive to determine where the start of the track is (e.g. when the hole is aligned with a light the light passes through the disk onto a light sensor causing a "start of track" pulse to be sent to the disk drive). From the start of the track; there is a gap, then some metadata (cylinder number, sector number, sector size, a few magic numbers for sanity checks, etc), then another gap, then the sector's data; then this pattern repeats for subsequent sectors.

Formatting a track causes the gaps and metadata to be written. Without the gaps and metadata (e.g. if the disk is literally full of zeros), the drive can't find any sectors.


Cheers,

Brendan

[Antti]

Floppies are actually quite interesting (I need to buy a floppy drive). I have to say that an old PC with floppies was actually quite neat. There were more possibilities than actually were deployed. However, all the extensions to it have made it quite messy. Of course there are exceptions, e.g. VESA BIOS Extensions (VBE) was very good.

Another thread called "Usb floppy cannot read second cylinder" is active also. This floppy theme seems to be topical. It seems that El-Torito CD image is the most elegant format. I do not find any problems from it. After all, most of the Linux distributions are "officially" released as optical media images. However, optical media will be obsolete.

[Antti]

It is quite ironic that I said "optical media will be obsolete." At the same time, I am playing with floppies like they were modern.