What exactly are ISO files?

[bzt]

bzt wrote:
> And here's my hybrid PMBR GPT/ISO9660 image creator: mkimg.c

Wow. That's much smaller than any other ISO 9660 producer i know of.

Well thanks, that was the plan K.I.S.S.

Although the opportunities to add payload seem extremely sparse. )

It just reads the file bootpart.bin and creates a partition of it. You can put anything you want into it. (But creating a FAT image out of a directory wouldn't increase its complexity much. I was thinking about adding that too, but then I just used mkfs.vfat from the Makefile because I'm lazy ). Adding more partition image files to the generation is trivial too. I guess it should read all the files from the command line maybe?

You take the effort to produce a GPT. Do you know of EFI implementations
which would not boot from an MBR partition of type 0xef ?

Yes, all of them (at least all that I can test with). MBR type 0xef is there only for legacy systems and legacy tools to avoid accidental overwrite of the GPT. I can't boot the image under UEFI without an ESP in a GPT. (When I boot in legacy CSM, the /EFI/BOOT/BOOTX64.EFI file is not used at all, then it takes the x86_64-bios route.)

(I ask because the GPT in ISOLINUX isohybrid for EFI is invalid and EFI
is supposed to boot via MBR partition table.

You can only boot from the MBR with CSM, and does not work on modern machines which has no legacy CSM boot option. It is said that UEFI manufacturers will remove CSM this year, but I can imagine they'll still support it a bit longer, few more years perphaps, no more.

So i preach for abandoning the GPT entirely.

Not a good idea. There are lots of advantages in GPT over MBR. First, you can have more than 4 partition records. Second, you have larger sector number fields which can describe the capacity of disks these days.

> Volume Boot Record (VBR): [...] the first sector of a file system volume
> [...] There's no partitioning table in it for sure.

Well, someone should have told this program "mformat", which places a
partition table inside its FAT image

Yes, I was talking about standard VBRs, not hybrid ones. Hybrid boot sectors tend to have everything, because they are hybrid

> the first 64k is not used at all

It's the first 32k of an ISO 9660 filesystem, to be exacting.
At offset 32k comes the PVD, which is kindof the superblock of ISO 9660.

Oh, right! Should had more sleep I guess. 32k it is! Anyway the point is you have enough space to put a GPT there.

Cheers,
bzt

[scdbackup]

Hi,

bzt wrote:
> I can't boot the image under UEFI without an ESP in a GPT.

So this Debian installation ISO image will not boot for you if you put
it onto an USB stick and try with real EFI iron ?
https://cdimage.debian.org/debian-cd/cu ... etinst.iso

This classic isohybrid ISO contains an MBR partition table which is not
"protective" as specified for GPT. So the GPT header block and backup block
in the ISO are not valid. One can zeroize them by dd and the ISO is still
supposed to boot from USB stick.
Further the FAT partitions in the GPT tables are not marked by the proper
type GUID.

MBR partition type 0xef is mentioned in UEFI 2.4:

"12.3.1 System Partition
A System Partition is a partition in the conventional sense of a partition
on a legacy system. For a hard disk, a partition is a contiguous grouping
of sectors on the disk where the starting sector and size are defined by
the Master Boot Record (MBR), which resides on LBA 0 (i.e., the first sector
of the hard disk) (see Section 5.2), or the GUID Partition Table (GPT),
which resides on logical block 1 (the second sector of the hard disk)
(see Section 5.3.1)."

"5.2.2 OS Types
Unique types defined by this specification (other values are not defined
by this specification):
* 0xEF (i.e., UEFI System Partition) defines a UEFI system partition.
* 0xEE (i.e., GPT Protective) is used by a protective MBR (see 5.2.2)
to define a fake partition covering the entire disk."

So you can either have an MBR partition table with a partition of type
0xef plus maybe others, or you can have a "protective" MBR table with only
a single partition of type 0xee.

If you look at ISOs from CentOS, Fedora, Knoppix, Arch, Debian, Gentoo,
Ubuntu, SuSE, you will find MBR/"dos" partitions of type 0xef.

Installation ISOs with GPT are rare. Mostly they are made by grub-mkrescue:
Guix, Proxmox, Siduction, Super Grub Disk, ...
Also there are some GPT ISOs from the BSDs.

I wrote:
> > So i preach for abandoning the GPT entirely.

bzt wrote:
> Not a good idea.

In ISOs, not on hard disk, to be clear.

> There are lots of advantages in GPT over MBR.

The backup GPT at the end of the device is a problem, when an image file
is created for a device of unknown size.
32 bit LBA of MBR partition tables is still good for USB sticks up to 2 TB.

In the end it is up to the users to choose between neat MBR, dirty MBR
with invalid GPT (as nearly all Linux distros do), and neat GPT.


> Hybrid boot sectors tend to have everything, because they are hybrid

But mformat does not know about hybrid here. It rather believes in making
a virtual floppy and equips it with a partition table entry that starts
at LBA 0. But the FAT image of GRUB ends up as EFI partition. That drove
an old machine's EFI mad when it tried to boot a Guix ISO from USB stick.


Have a nice day

Thomas

[bzt]

Hi,

I'm not entirely sure what your question is here, but I'll try to answer.

So this Debian installation ISO image will not boot for you if you put
it onto an USB stick and try with real EFI iron ?

I can only boot it with legacy CSM and not in UEFI. BTW it has a GPT, but it does not have an ESP though, and fdisk can't recognize the partitions either (but you can see the magic "EFI PART" at 0x200 in a hex dump, it is there).

This classic isohybrid ISO contains an MBR partition table which is not
"protective" as specified for GPT. So the GPT header block and backup block
in the ISO are not valid.

The classic isohybrid ISO boots with El-Torito, and not with UEFI CDROM protocol. One should support that too (see my Makefile for the "eficdrom" rule).

Further the FAT partitions in the GPT tables are not marked by the proper
type GUID.

There are no valid GPT partitions in that image. Fdisk even reports invalid GPT and creates a new one.

I can imagine that some firmware implement a failsafe mechanism to locate the ESP in the MBR, but my firmware is not like that, and you shouldn't rely on this. If you have a GPT anyway, then just put a valid ESP record in it and go on. Simpler, and that's what the spec expects.

A System Partition is a partition in the conventional sense of a partition
on a legacy system.

See? It's for backward compatibility. It does not say that a non-legacy UEFI should use the MBR at all. Quite the contrary, the spec says non-legacy systems must use the GPT and only the GPT.

Installation ISOs with GPT are rare.

No, they are not. The debian iso you liked also has one (although invalid, but it is there). Second, we're not booting only from installation ISOs. My image creator creates disk images, which just happen to be CDROM-compatible in case you want to use it as a CDROM ISO. But you could just write it to a disk right away, it must boot.

In ISOs, not on hard disk, to be clear.

Ok, I see your confusion. An ISO (a file with the extension .iso) is nothing more than a dump of a storage media, one sector after another. In that sense an ISO can be a hard disk image, not just ISO9660 dumps. In my image creator examples I've used the extension ".img" to avoid this confusion.

The backup GPT at the end of the device is a problem, when an image file
is created for a device of unknown size.

Agreed, but I can't change the spec. One disadvantage over many.

In the end it is up to the users to choose between neat MBR, dirty MBR
with invalid GPT (as nearly all Linux distros do), and neat GPT.

Wrong. The user is tied to the solution their firmware supports. Sooner or later there'll be only GPT. Like it or not, you should get used to it.

Cheers,
bzt

[scdbackup]

Hi,

> > debian-10.4.0-amd64-netinst.iso

> I can only boot it with legacy CSM and not in UEFI.

What machine ? Does its firmware show any individual id text ?


> BTW it has a GPT,

For which nobody knows the motivation.
It stems from Matthew J. Garrett's layout for Fedora of 2012.
https://mjg59.dreamwidth.org/11285.html
But as you noticed, the Type GUID of EFI System Partition is not to see there.
It surely boots only via the MBR partition ... if it boots.


> The classic isohybrid ISO boots with El-Torito, and not with UEFI CDROM
> protocol.

Matthew J. Garrett added the EFI functionality to ISOLINUX isohybrid.c
back in 2012. Fedora and others make a FAT filesystem image with the
necessary files and submit it to the ISO producer (mkisofs, genisoimage,
xorriso) as El Torito boot image for platform EF. The isohybrid program
then creates the strange partition salad.
https://wiki.syslinux.org/wiki/index.ph ... ybrid#UEFI

libisofs and xorriso offer to do this already during ISO production,
so that no post-processing by the isohybrid program is necessary.


> [UEFI specs] "on a legacy system."

That's to distinguish "partition" from modern stuff like Logical Volume
Manager. The text clearly says that MBR and GPT are specified as table
types for marking the ESP.


> The user is tied to the solution their firmware supports.

My users are mostly the ISO preparers of Linux distros. Very conservative
and result oriented. Hard to convince of any change, but always interested
in bug reports.

I will forward your failure report to Debian if you tell me what machine
and firmware does not boot their netinst ISO via EFI.

I learned today that Guix went from GPT to MBR recently.
Nearly the whole spectrum of Linux distros seems not to get problem reports
about a need for GPT.


Have a nice day

Thomas

(editedt to correct publication year of https://mjg59.dreamwidth.org/11285.html)

[scdbackup]

Hi,

can it be that your machine has a 32-bit EFI which wants to see
file \EFI\BOOT\BOOTIA32.EFI in the ESP ?

In that case please test with 32 bit Debian "i386"
https://cdimage.debian.org/debian-cd/cu ... etinst.iso
because the "amd64" variant only offers BOOTX64.EFI.


Have a nice day

Thomas

[bzt]

Hi,

For which nobody knows the motivation.

Maybe because MBR is obsolete and soon not to be supported? According to this presentation page 15:

Intel is removing legacy BIOS support from client & data center platforms by 2020

Matthew J. Garrett added the EFI functionality to ISOLINUX isohybrid.c
back in 2012.

Which is no good if the firmware does not recognize the medium as UEFI-bootable because it can't locate a valid ESP in the GPT. But as I said, with a CDROM image partitioning table is not used at all, there a special Boot Catalog record shows the first sector of the ESP. So this only matters if you write out the ISO to an USB stick.

The text clearly says that MBR and GPT are specified as table
types for marking the ESP.

For legacy and backward compatible firmware. UEFI class 3 will only support GPT, it will not parse the MBR for legacy System Partitions for sure.

I will forward your failure report to Debian if you tell me what machine
and firmware does not boot their netinst ISO via EFI.

Thanks, but you misunderstood. I can boot Debian netinst ISO, just not the UEFI way.

Here there are two options, depending whether the end-users PC is supporting that MBR hack or not:
1. If it does, then UEFI is used
2. if it doesn't, then legacy CSM is used
Because the user can boot the ISO either way, there will be no bug reports.

It happens that I have an old and buggy firmware which does not support the MBR hack, but I still can boot Debian (with CSM). Then in the newly installed system there will be a valid GPT and ESP on the boot disk, so the installed kernel boots via UEFI.

Regardless you should file a bug report on the invalid GPT. According to the UEFI spec section 12.3.2 Partition Discovery, this shouldn't work:

if no GPT is present, each partition found in the legacy MBR partition tables.

It is pretty clear that since there's a GPT in the Debian netinst ISO, the firmware SHOULD NOT look for 0xef in the MBR, yet, you claim it does. According to this, it is not my firmware, but yours that does not follow the spec.

Nearly the whole spectrum of Linux distros seems not to get problem reports
about a need for GPT.

As I wrote above, as of today UEFI class 1-2 is widespread so if the MBR ESP hack is not supported, there's still CSM to boot those distro's installer. However this is a ticking bomb, as soon as the end users start to buy UEFI class 3 machines (probably during this year), they will start complaining because the installers won't boot. Just wait and see. That's why my image creator creates a GPT, to be future-proof. With a valid ESP in a GPT works for UEFI class 1-2-3 (and I have implemented a GPT parser in my boot loader for BIOS machines), so Intel's decision on removing legacy CSM don't bother me at all

Cheers,
bzt

[scdbackup]

Hi,

i see no connection between legacy CSM which goes via the 16 bit x86
machine code in the MBR and UEFI specs which allow MBR partition table
and GPT as alternatives.

Whatever, please tell me what machine and firmware version shows this
behavior, so that i can tell the maintainers of Debian ISOs to look out
for such a machine.


Have a nice day

Thomas

[bzt]

Hi,

i see no connection between legacy CSM which goes via the 16 bit x86
machine code in the MBR and UEFI specs which allow MBR partition table
and GPT as alternatives.

Other than the UEFI spec consistently calls that "legacy" too? Read section 12.3.2 Partition Discovery, it is pretty clear firmware should not parse the legacy MBR partitioning table for ESP if/when there's a GPT available.

Whatever, please tell me what machine and firmware version shows this
behavior, so that i can tell the maintainers of Debian ISOs to look out
for such a machine.

Doesn't matter. Enough to say that one is more than 10 years old, and the other has only EFI 1... There's a good reason why there are so many checks in my boot loader for the GOP structures and sanity checks for the memory map too

Cheers,
bzt

[scdbackup]

Hi,

i wrote:
> > machine and firmware version

bzt wrote:
> Doesn't matter.

You give away the opportunity to show the Debian developers where their ISO
boot layout fails. (They would also ask for a test with an ISO for 32 bit x86.)


> > UEFI specs which allow MBR partition table and GPT as alternatives.

> the UEFI spec consistently calls that "legacy"

UEFI uses this term for about everything that was invented before EFI.
I see "legacy" in the following contexts.

Related to CSM:

- Legacy Operating System Support (use of MBR x86 machine code by CSM)
- Legacy Platform (CSM itself)
- legacy interfaces (other name for CSM ?)
- legacy media (only equipped with BIOS/CSM boot stuff ?)
- legacy system (firmware doing what BIOS/CSM does)

Independent of CSM:

- Legacy Option ROM (extra BIOS features now supposed to be implemented
in EFI programs)
- Legacy Master Boot Record (MBR that is not "protective" for GPT.
It may mark ESP by partition type 0xEF.
x86 machine code is prescribed to be ignored
by EFI.)
- Legacy Floppy (hardware class)
- Legacy MBR Partition (to be offered if no GPT is detected)
- legacy ISA devices (hardware class)
- Legacy BIOS bootable (a flag in GPT)
- legacy hardware devices (possibly present time services)
- partition on a legacy system (as example of a contiguous block interval)
- legacy reduced system (not having a UART serial port)
- legacy infrastructure (everything that EFI shall finally make obsolete)
- "legacy" EFI application (they reside in the EFI Toolkit)

Not really obvious are the terms:

- legacy-API
- legacy code
- legacy devices and firmware code


The legacy MBR partition table is explicitely mentioned as one of the
three pointers to EFI visible partitions in general:

"12.3.2 Partition Discovery
This specification requires the firmware to be able to parse the legacy
master boot record(MBR) (see Section 5.2.1), GUID Partition Table (GPT)
(see Section 5.3.2), and El Torito (see Section 12.3.2.1) logical device
volumes.
The EFI firmware produces a logical EFI_BLOCK_IO_PROTOCOL device for:
* each GUID Partition Entry (see table 16 in 5.3.3) with bit 1 set to zero;
* each El Torito logical device volume; and
* if no GPT is present, each partition found in the legacy MBR partition
tables.
"

Note that this is not about legacy platform / CSM, but about which storage
devices get offered to programs which use the UEFI API.


Have a nice day

Thomas

[Ethin]

I can confirm this in the latest version of the UEFI spec (version 2. in section 13.3.2. Keep in mind that Intel is not the only company that decides what happens with UEFI. It is the UEFI forum, consisting of AMD, HP, AMI, Apple, Intel, ARM, and a large number of others. (Side note: I do agree with the complete removal of the MBR partition scheme. It makes perfect sense.)

[scdbackup]

Hi,

so for the records, the texts which i quoted from UEFI 2.4 are still in
https://uefi.org/sites/default/files/re ... _Feb14.pdf
with the same mentioning of legacy MBR partitions as valid ESP candidates.
Just the chapter numbers have changed from 12 to 13.

Ethin wrote:
> I do agree with the complete removal of the MBR partition scheme.

But it is in no way announced in UEFI 2.8.

> It makes perfect sense.

The prescription of a backup GPT is not suitable for images which get
on devices of which the size is not known at image production time.

I agree that it makes few sense to partition a hard disk by MBR.

I also agree (in advance) that the isohybrid --uefi layout of the main
Linux distro ISOs is an abomination with its invalid GPT debris.
But this does not mean that a neat MBR table would not be the most
convenient way to make a bootable ISO 9660 filesystem ready for USB stick.
I made my case towards Debian in
https://lists.debian.org/debian-cd/2019 ... 00007.html

Widespread abandoning of MBR partition table interpretation in EFI would
of course change the situation. Then the ISOs would need GPT and best two
ISO 9660 superblocks and directory trees. So there would be a mountable
ISO partition for USB stick and a mountable base device for DVD.

But for now i am sure that hundreds of thousands ISOs of Debian, Ubuntu,
Fedora, SuSE, etc. successfully booted on EFI from USB stick in the last
years.

Have a nice day

Thomas

[bzt]

with the same mentioning of legacy MBR partitions as valid ESP candidates.

But it is in no way announced in UEFI 2.8.

Wrong. Read your own post with the quote again:

* each GUID Partition Entry (see table 16 in 5.3.3) with bit 1 set to zero;
* each El Torito logical device volume; and
* if no GPT is present, each partition found in the legacy MBR partition tables.

So in other words, if there's a GPT, then the legacy MBR tables should not be supported with current class 1-2 firmware either (it is a bug if they do), and since class 3 will mandate GPT, this also means no legacy MBR table parsing for sure in the foreseeable future. (Just for the records, if the Debian installer boots in UEFI mode that is a bug, since it clearly has GPT. An invalid one, granted, but it is slippery-sloop at best!)

I honestly don't understand why are you against GPT that much. It is not that complicated to create. All Linux partitioning tools use that by default (fdisk, cfdisk, gdisk), plus all MacOSX tools (diskutil, Disk Utility), and Windows supports GPT too and it is the default for Win10. So if that's the de facto standard for all mainstream OSes, plus all the firmare (class 1-2-3) supports it, then just create a GPT and get on with it. There's no point in using an obsolete format and hoping that it will be still supported the next day...

But for now i am sure that hundreds of thousands ISOs of Debian, Ubuntu,
Fedora, SuSE, etc. successfully booted on EFI from USB stick in the last
years.

No way of knowing. As I explained before, if the installer itself boots using CSM, it still can install the system and the resulting system can boot using UEFI. The users wouldn't notice the difference, so there'll be no bug reports about this (well, not until class 3 and GPT gets widespread).

Cheers,
bzt

[scdbackup]

Hi,

bzt wrote:
> if there's a GPT, then the legacy MBR tables should not be supported

A GPT is valid only if the MBR partition table is "protective":
- The table contains exactly one partition entry that is not 0,
- type is 0xEE,
- start LBA is 1,
- size is number of blocks of device - 1.

So valid GPT and non-"protective" MBR partition table are mutally exclusive
and thus the prescription of UEFI does indeed invite for an ESP marked
by a partition of type 0xEF in a non-protective MBR.


> I honestly don't understand why are you against GPT that much

Two main reasons:

Because it makes cumbersome the addition of a partition to a hybrid ISO
on an USB stick. Being the upstream developer of xorriso, which is used
by most Linux distros for ISO production meanwhile, i have to repeatedly
explain reasonable people why their USB sticks appear unreasonable to
their more or less reasonable partition editors and auto-mounters.

Because a GPT partition cannot start at LBA 0 where the ISO 9660
filesystem for the first session of a DVD must start. So the developer of
the ISO has to refrain from offering a mountable ISO 9660 partition, or
has to invest storage space into a second superblock and directory tree.
The cost of the latter can be high if the ISO contains many files.


Whatever, i leave it to the developers who compose the content of the ISOs
whether they choose MBR or GPT as partition table. Most stay with the most
ugly variation, simply because it works since 2012.


> class 3 will mandate GPT

I read at various places that class 3 is about abandoning CSM (i.e. no more
booting via MBR x86 machine code) and 3+ is about unconditional Secure Boot.
GPT is not mentioned in conjunction with classes 1 to 3+.

Interestingly, UEFI specs do not mention this kind of classes at all.
There are many occurences of "class". But none which mentions CSM.
On uefi.org i found only
https://uefi.org/sites/default/files/re ... tforms.pdf

Have a nice day

Thomas

[bzt]

Hi,

bzt wrote:
> if there's a GPT, then the legacy MBR tables should not be supported

A GPT is valid only if the MBR partition table is "protective":
- The table contains exactly one partition entry that is not 0,
- type is 0xEE,
- start LBA is 1,
- size is number of blocks of device - 1.

So valid GPT and non-"protective" MBR partition table are mutally exclusive
and thus the prescription of UEFI does indeed invite for an ESP marked
by a partition of type 0xEF in a non-protective MBR.

First of all, only 0xEE record is mandatory. Second, if that exists, then the firmware SHOULD NOT look for any other legacy partitions, including the 0xEF record, the spec is pretty clear about that: "If no GPT present". But here's a newsflash for you: most software and firmware looks for the magic "EFI PART" and it doesn't care about the MBR any more. If you don't believe me, try this out: put some wrong LBA secs in the 0xEE record, nobody will care, firmware will read GPT regardless, because PMBR is only there to keep compatibility with legacy software that don't know GPT.

Because it makes cumbersome the addition of a partition to a hybrid ISO
on an USB stick.

No it does not. I solved GPT creation with a few SLoC, biggest part of which being the CRC calculation and lookup table. Yesterday I've even added FAT16/FAT32 image creation for the ESP. My image creator is ca. 500 SLoC, which includes everything without any dependency: FAT generation, GPT creation, ISO 9660 creation with Boot Catalog creation. I wouldn't call that "cumbersome". Feel free to study my code, it is MIT licensed. I'm also offer my help to you if you want it.

Being the upstream developer of xorriso, which is used
by most Linux distros for ISO production meanwhile, i have to repeatedly
explain reasonable people why their USB sticks appear unreasonable to
their more or less reasonable partition editors and auto-mounters.

Please don't get this wrong, but have you ever wondered maybe the problem is with xorriso? I mean all partition editors find my images valid, no errors reported ever. Even fdisk just prints a warning that it won't save the ISO9660 part on write, but that's all. No errors of any kind.

Because a GPT partition cannot start at LBA 0

MBR does not allow that either... As far as I remember, only Solaris slices allowed zero LBA for slice 2, and for all the other partitioning tables at least a warning is generated by all partitioning tool.

where the ISO 9660
filesystem for the first session of a DVD must start. So the developer of
the ISO has to refrain from offering a mountable ISO 9660 partition, or
has to invest storage space into a second superblock and directory tree.
The cost of the latter can be high if the ISO contains many files.

You are literally trying to solve a non-existent problem. Why on earth do you want to mount it as an ISO when there's a better alternative? It is enough to mount using the GPT. Like it or not, ISO 9660 is only used for booting from an obsolete media (I personally haven't used CDs nor DVDs for more than a decade, and I'm most definitely not using cutting-edge technologies.) So let ancient hardware use ISO9660 to locate the boot code, and there's the GPT for everyone else and for all the other use cases.

GPT is not mentioned in conjunction with classes 1 to 3+.

Indirectly yes. Class 3 means no legacy code at all, and the MBR partitioning scheme is consistently called "legacy" throughout the specification.

Cheers,
bzt

[scdbackup]

Hi,

the "hybrid" installation ISOs of GNU/Linux distros for x86 machines
are supposed to be put on USB stick by a simple run of dd or equivalent.
After that they boot from USB stick to their installer systems at least
on the vast majority of x86 machines which run in modern EFI mode.
The larger distros are meanwhile capable of Secure Boot from ISO (at the
price that the word "Microsoft" appears about 100 times in the BOOTX64.EFI
and BOOTIA32.EFI programs).

This fact cannot be washed away by holding up magic words like "legacy"
without regarding their context, or by predicting the future in newsflash
style.
Show released or officially announced specs which support your claim.
Show technical data about the firmware that does not boot an ISO with MBR
partition of type 0xEF and suitable software in it.

Have a nice day

Thomas