What exactly are ISO files?

[obiwac]

Hi everyone! (It's been like 2 years since my last post here oof, kinda miss this place)

I probably ought to know this already, but I think it's worth asking anyway so that I can clarify things, even if it's a bit stupid.

If you search "how do iso files work" on Google, you'll get results explaining at a very high level what ISO's are, how they're just images of a disk that (sometimes) have an OS installer on them, ... but that doesn't really help me. I wanna know how exactly data is stored in an ISO file. In what way is it different from a .img file? They're both just a mirror of some media that contains a bootable system either to use as a live system, either as an installer, right? So what's the difference, and what exactly do tools like Xorriso do that's different from just using dd to copy an OS installation to another media?

All this isn't very clear in my mind, so I'd greatly appriciate if someone could bring an element of explanation

Thanks for your time and have a nice day!

[klange]

https://wiki.osdev.org/ISO_9660

[obiwac]

Oh so OS ISO files are just an installer's files stored in a special file system? And if so, what's the advantage over simply having an image of a bootable partition on your media like what you'd have on your harddrive?

Thank you!

[klange]

ISO9660 is a special filesystem that was specifically designed for CDs, independent of the several competing hard disk file systems in use when the CD “rainbow books” were being written. It is a filesystem, though, and near universally supported in operating systems.

[Octocontrabass]

Oh so OS ISO files are just an installer's files stored in a special file system?

It doesn't have to be an installer. Any data can be put in the filesystem. The key feature is that ISO files follow optical disc standards, so you can burn it to a CD/DVD/whatever and other computers will be able to read it.

In the reverse direction, you can make an ISO file by using dd to copy any optical disc with 2048-byte sectors. (CDs may have different sector sizes. The ".iso" file extension typically indicates 2048-byte sectors, in order to differentiate it from the other formats.)

And if so, what's the advantage over simply having an image of a bootable partition on your media like what you'd have on your harddrive?

Hard drive partitions usually don't follow optical disc standards, so they wouldn't be bootable once you burn them to a CD.

Some clever folks figured out a way to squeeze hard disk boot code into ISO files while still following the optical disc standards, creating the "hybrid" ISO. You may be familiar with these hybrid ISOs, since many Linux distributions use them.

[obiwac]

Right, I understand better now, thanks alot. One last thing: why do people still mainly distribute ISO files if most people anyway use USB drives nowadays? I suspect it would be because you can boot ISO files just as well on USB drives so there'd be no point in changing and losing compatibility, but then why does FreeBSD provide downloads for both an ISO and normal image file?

[Octocontrabass]

One last thing: why do people still mainly distribute ISO files if most people anyway use USB drives nowadays?

It's usually more convenient to boot a virtual machine from a virtual CD/DVD (using the ISO file) than from a virtual USB drive. Virtual machines are very popular in the enterprise world.

I suspect it would be because you can boot ISO files just as well on USB drives so there'd be no point in changing and losing compatibility, but then why does FreeBSD provide downloads for both an ISO and normal image file?

Good question. They appear to be hybrid ISO files, so it's not clear why a separate USB boot image is necessary.

Note that an ordinary (non-hybrid) ISO file will not boot if you write it to a USB drive.

[obiwac]

Alright, I get it now. Thanks alot for your help, ISO files really seemed like black boxes to me!

[eekee]

Oh I know that "black box" feeling! I hate it.

Hard drive partitions usually don't follow optical disc standards, so they wouldn't be bootable once you burn them to a CD.

I'm almost sure I once did the reverse of this: booted an iso image written to a partition. Later I found another wouldn't boot. Perhaps it depends on which boot method the ISO uses; there are at least 3. Also, an OS distro configured to boot from ISO may not look at the partitions.

It's usually more convenient to boot a virtual machine from a virtual CD/DVD (using the ISO file) than from a virtual USB drive. Virtual machines are very popular in the enterprise world.

Yup, plus some people like to explore operating systems, and bringing up qemu with an iso file is still a little bit more convenient than messing with USB media.

[Octocontrabass]

I'm almost sure I once did the reverse of this: booted an iso image written to a partition. Later I found another wouldn't boot.

Sounds like the first one was a hybrid ISO and the second one was not. (Are you sure you wrote it to a partition? The ones I've seen only work when written to the entire disk.)

Perhaps it depends on which boot method the ISO uses; there are at least 3.

You might be thinking of the three boot modes available through El Torito. Those are only used when booting a CD (either real or virtual). For a flash drive, you need a MBR. Hybrid ISOs have a MBR in addition to the El Torito boot code.

[eekee]

I'm almost sure I once did the reverse of this: booted an iso image written to a partition. Later I found another wouldn't boot.

Sounds like the first one was a hybrid ISO and the second one was not. (Are you sure you wrote it to a partition? The ones I've seen only work when written to the entire disk.)

I'm not sure about the first ISO. It's possible I extracted the (Linux) kernel & wrote it to a boot partition, then booted it with instructions to use the iso partition as root. This would render my argument invalid, I guess, and also mean I was engaging in that very odd form of laziness which is characteristic of engineers. The second iso was definitely a hybrid. I remember blaming its complex boot scripts, (it was Knoppix,) because in my experience, distros with complex boot scripts are much less flexible than when the Linux kernel is left to its defaults.

I wonder if it's possible to write a bootsector which can function as MBR or PBR and load from an iso-9660 filesystem. That would allow a hybrid iso which could be written to a partition.

Perhaps it depends on which boot method the ISO uses; there are at least 3.

You might be thinking of the three boot modes available through El Torito. Those are only used when booting a CD (either real or virtual). For a flash drive, you need a MBR. Hybrid ISOs have a MBR in addition to the El Torito boot code.

Yes, I am. That makes sense.

[bzt]

I wonder if it's possible to write a bootsector which can function as MBR or PBR and load from an iso-9660 filesystem. That would allow a hybrid iso which could be written to a partition.

Requires a touch of a genius, but possible. I have a working PoC for this, see boot.asm. And here's my hybrid PMBR GPT/ISO9660 image creator: mkimg.c that uses this boot sector. Tested in BIOS disk, BIOS cdrom, UEFI disk, UEFI cdrom environments. Btw for a partition you'll need a VBR (not MBR nor PBR), and for CDROM you'll need a Boot Sector in Boot Catalog, but my code can work as both. The only thing that it can't support is a floppy, as it relies on LBA.

Of course I use this boot sector to load my BOOTBOOT loader, but you could load your own 2nd stage with it if you want. The only restriction on the second stage is that it has to start with these magic 6 bytes:

Code: Select all

0x55, 0xAA, 2nd stage size/512, 0xE9, el, eh

where "el" and "eh" are the low and high bytes of the entry point. The rest is up to you

Cheers,
bzt

[eekee]

Awesome! But what on Earth is a VBR, and how does it differ from a Partition Boot Record? (I'm teasing, but also trying to say, "it's not a good idea to be too strict about terminology.")

[bzt]

Awesome! But what on Earth is a VBR, and how does it differ from a Partition Boot Record? (I'm teasing, but also trying to say, "it's not a good idea to be too strict about terminology.")

Oh, you meant that! I though you were referring to the Protective Boot Record

Ok, let's clear things up, this is the terminology I was using:
- Master Boot Record (MBR): 512 bytes, is the first sector of a media, may contain BPB (for FAT floppies or USB sticks) and partitioning table as well (for larger media).
- Protective Master Boot Record (PMBR): 512 bytes, the first sector of a media, contains an old-style partitioning table with an entry to the new-style GPT partitioning table. Might contain a legacy boot code, but not necessarily.
- Partition Boot Record (PBR as you wrote it): the first sector of a partition (which may or may not contain a file system, therefore a volume).
- Volume Boot Record (VBR): 512 bytes, the first sector of a file system volume (regardless if it's on the entire disk or just in one partition). There's no partitioning table in it for sure. For FAT volumes, it contains BPB.
- CDROM Boor Record: 2048 bytes, any sector that's recorded in the Boot Catalog. Does not contain BPB nor partitioning table (unless it's a hybrid boot sector).

The problem here is, these terms are not well defined: you can define them by the sector position: MBR as the first sector of the disk, and VBR as the first sector of a partition. But you can also define them by their contents: first sector is an MBR if there's a partitioning table in it, and VBR if it's the first sector of a volume (that is, contains BPB for a FAT volume). As you can see, these are ambiguous definitions.

However we can conclude that MBR is always the first sector of a disk, PBR is never the first, and VBR could be the first of a disk and first of a partition as well.

I have already said this many times, but floppies can contain partitioning tables, and disks might contain only one file system without partitioning table. There's nothing in BIOS Boot Specification nor in the original MS-DOS code to prevent this, it is merely a choice of the user if they want to slice large media into smaller volumes or not. Windows XP could not handle partitioning tables on USB sticks until SP2 (or 3, my memory is fuzzy about which one, but I'm sure it required an SP). With CDROMs it is a bit clearer, as there can be only one file system on them, and the first 64k is not used at all, which opens up the possibility to create hybrid images.

Cheers,
bzt

[scdbackup]

Hi,

for ISOLINUX isohybrid from a partition instead of a base device see
https://wiki.syslinux.org/wiki/index.ph ... _selection

I am not aware whether GRUB would offer MBRs for that purpose. Its
official control program for making bootable ISOs offers no option for
preparing an ISO image for putting into a partition.


obivac wrote:
> what exactly do tools like Xorriso do that's different from just using dd

On the one hand xorriso packs up input files as payload of an ISO 9660
filesystem. On the other hand it advertises the block addresses and sizes
of boot image files in the ISO, which the computer firmware can use to
start an operating system. This advertisement is in form of El Torito
boot information
https://wiki.osdev.org/El-Torito
or of a partition table
https://wiki.osdev.org/EFI_System_Partition
or simply x86 machine code at the start of the ISO 9660 image.


bzt wrote:
> And here's my hybrid PMBR GPT/ISO9660 image creator: mkimg.c

Wow. That's much smaller than any other ISO 9660 producer i know of.
Although the opportunities to add payload seem extremely sparse. )

You take the effort to produce a GPT. Do you know of EFI implementations
which would not boot from an MBR partition of type 0xef ?

(I ask because the GPT in ISOLINUX isohybrid for EFI is invalid and EFI
is supposed to boot via MBR partition table. For most Linux distro ISOs it
seems to work. So i preach for abandoning the GPT entirely.)

------------------------------------------------------------------------
Anectotes and nitpicking:

> Volume Boot Record (VBR): [...] the first sector of a file system volume
> [...] There's no partitioning table in it for sure.

Well, someone should have told this program "mformat", which places a
partition table inside its FAT image if called like in
http://git.savannah.gnu.org/cgit/grub.g ... cue.c#n812
Some antique EFI really took offense and cycled endlessly.


> the first 64k is not used at all

It's the first 32k of an ISO 9660 filesystem, to be exacting.
At offset 32k comes the PVD, which is kindof the superblock of ISO 9660.


Have a nice day

Thomas