Making a bootable image

Question about which tools to use, bugs, the best way to implement a function, etc should go here. Don't forget to see if your question is answered in the wiki first! When in doubt post here.
User avatar
bzt
Member
Member
Posts: 1584
Joined: Thu Oct 13, 2016 4:55 pm
Contact:

Re: Making a bootable image

Post by bzt »

Octocontrabass wrote:
bzt wrote:There's nothing in the on-disk format that would prevent you from creating a 64Mb FAT32 image with 64 sectors per clusters.
Sure, but it wouldn't have enough clusters to be valid.
That's my point. Even with fewer clusters, many tools (including the Linux kernel) will identify it correctly as FAT32 and will use that file system without any problems.

If you simply remove "if (Volume->MaxCluster < FAT_MAX_FAT16_CLUSTER) {" check from TianoCore, it would work perfectly too, there's no drawback of any kind. That check does nothing but refuses some FAT32 file systems for no good reason (there's nothing in the TianoCore code that depends on that check).

Cheers,
bzt
Octocontrabass
Member
Member
Posts: 5568
Joined: Mon Mar 25, 2013 7:01 pm

Re: Making a bootable image

Post by Octocontrabass »

bzt wrote:That's my point. Even with fewer clusters, many tools (including the Linux kernel) will identify it correctly as FAT32 and will use that file system without any problems.
The Linux kernel is irrelevant. The only authoritative source on FAT filesystems is Microsoft, and Microsoft says it's invalid. I'm sure there are some tools that will correctly identify it as a FAT12 filesystem and delete all of your data trying to repair it.
nexos
Member
Member
Posts: 1081
Joined: Tue Feb 18, 2020 3:29 pm
Libera.chat IRC: nexos

Re: Making a bootable image

Post by nexos »

Bottom line: Microsoft says to go by clusters. I'll just obey them. Bill Gates was barely in college when he wrote FAT. I doubt he was looking ahead enough to see what was down the road for FAT. Saying, "It doesn't make sense" doesn't do anything :D .
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg
User avatar
bzt
Member
Member
Posts: 1584
Joined: Thu Oct 13, 2016 4:55 pm
Contact:

Re: Making a bootable image

Post by bzt »

Octocontrabass wrote:I'm sure there are some tools that will correctly identify it as a FAT12
No way. BPB_FATSz16 being zero plus the size being larger than 4M means it can't be a valid FAT12 file system. And that's my point, using the spec page 14-15 only would identify it incorrectly, for several reasons.
nexos wrote:I'll just obey them.
Well, your choice. It's sure as hell I won't, I rather write a driver that can handle real world devices (like mp3 players, cameras, mobile phone formatted SD cards etc.) too. Specially because that can work with Win formatted partitions as well no probs.
nexos wrote:I doubt he was looking ahead enough to see what was down the road for FAT.
Oh, but he did. FAT8 was so bad, that they had to create FAT12 ASAP. For that reason, there's a FilSysType field in the BPB, which should indicate the file system type (but they could have also used the MediaType field as Atari did, or they could have utilized the upper tetrad which is always 0xF0). What makes no sense is that the spec defines the magic bytes informational only. I haven't seen any other spec that would define magic bytes as optional, well, never ever. Just for the records, the magic bytes in exFAT are mandatory (as they should be).

Cheers,
bzt
Octocontrabass
Member
Member
Posts: 5568
Joined: Mon Mar 25, 2013 7:01 pm

Re: Making a bootable image

Post by Octocontrabass »

bzt wrote:No way. BPB_FATSz16 being zero plus the size being larger than 4M means it can't be a valid FAT12 file system.
I think your math might be a bit off there; FAT12 volumes can be up to around 128MB before you'll start having issues with overly large clusters.
bzt wrote:And that's my point, using the spec page 14-15 only would identify it incorrectly, for several reasons.
There is no "correct" way to identify an invalid filesystem. Either follow the spec or accept that some tools do not support your invalid filesystem.
bzt wrote:Oh, but he did. FAT8 was so bad, that they had to create FAT12 ASAP. For that reason, there's a FilSysType field in the BPB, which should indicate the file system type (but they could have also used the MediaType field as Atari did, or they could have utilized the upper tetrad which is always 0xF0). What makes no sense is that the spec defines the magic bytes informational only. I haven't seen any other spec that would define magic bytes as optional, well, never ever.
The FilSysType field didn't exist until OS/2 1.0, which means there are plenty of real-world examples of FAT12 and FAT16 filesystems without it. (I have no explanation for why Microsoft decided to make it optional for FAT32 though!)

The BPB itself was first introduced in DOS 2.0, in order to support hard disks. A floppy disk formatted with a prior version of DOS wouldn't have a BPB at all.
User avatar
bzt
Member
Member
Posts: 1584
Joined: Thu Oct 13, 2016 4:55 pm
Contact:

Re: Making a bootable image

Post by bzt »

Okay, this goes nowhere. Doesn't matter how much you want to protect a bad specification, that specification remains bad.
Octocontrabass wrote:I think your math might be a bit off there; FAT12 volumes can be up to around 128MB
Not my math. From the FAT specification, page 13:
Note that media size for FAT12 volumes must be <= 4MB.
Octocontrabass wrote:
Octocontrabass wrote:I'm sure there are some tools that will correctly identify it as a FAT12
There is no "correct" way to identify an invalid filesystem.
You said it, not me.

What I've said was that using the FAT specification page 14-15, section "3.5 Determination of FAT type when mounting the volume" is inadequate and wrong, because
The FAT type is determined solely by the count of clusters on the volume (CountOfClusters).
is obviously incorrect, and contradicts many other statements in the same specification. For example from page 8
BPB_RootEntCnt For FAT32 volumes, this field must be set to 0.
And from page 9
BPB_FATSz16 On FAT32 volumes this field must be 0
So the fields in the BPB also matters, so does the overall volume size (see on page 13), therefore the statement on page 14, "determined solely by the count of clusters" is unquestionably and undeniably wrong.
Octocontrabass wrote:The FilSysType field didn't exist until OS/2 1.0
But MediaType did existed. Long story short, they could have included a FAT type field in the BPB, but they didn't. This was a big mistake, anyone can see that.

Octocontrabass wrote:The BPB itself was first introduced in DOS 2.0, in order to support hard disks. A floppy disk formatted with a prior version of DOS wouldn't have a BPB at all.
I don't have any floppy drives any more (not to mention the really ancient 160Kb floppies), so I don't care how they were formatted by ancient and obsolete MS-DOS version which couldn't even handle directories. Furthermore, according to the FAT specification page 7:
All FAT volumes must have a BPB in the boot sector.
Conclusion: FAT file system is poorly designed, and the FAT specification is badly written with full of contradictions. Period. That's the truth. Resistance is futile ;-)

Cheers,
bzt
nullplan
Member
Member
Posts: 1790
Joined: Wed Aug 30, 2017 8:24 am

Re: Making a bootable image

Post by nullplan »

bzt wrote:Conclusion: FAT file system is poorly designed, and the FAT specification is badly written with full of contradictions. Period. That's the truth. Resistance is futile ;-)
Well, at least we agree on that. What's worse is that there's no excuse. That spec was published a long time after FAT had been rolled out, so some practical knowledge could have flown back into it. But no. Bare minimum of effort, I presume. Which describes the whole file system, now that I think about it.
Carpe diem!
Octocontrabass
Member
Member
Posts: 5568
Joined: Mon Mar 25, 2013 7:01 pm

Re: Making a bootable image

Post by Octocontrabass »

bzt wrote:Not my math. From the FAT specification, page 13:
Note that media size for FAT12 volumes must be <= 4MB.
That line appears only in the SD card version of the FAT specification, and applies only to the creation of new volumes. Either it's a limitation specific to SD cards for some reason, or it's an odd misinterpretation of a similar line in another version of the spec (where formatting volumes larger than 4MB as FAT12 is strongly discouraged, but not explicitly forbidden).

In any case, the number of clusters is still the only deciding factor when differentiating between FAT12, FAT16, and FAT32 when mounting an existing volume.
bzt wrote:So the fields in the BPB also matters, so does the overall volume size (see on page 13), therefore the statement on page 14, "determined solely by the count of clusters" is unquestionably and undeniably wrong.
The fields in the BPB matter only for determining whether the volume is valid, not which filesystem it contains. If a volume contains enough clusters to be FAT32 but has nonzero values in the must-be-zero BPB fields, the filesystem is corrupt and needs to be repaired. This is very important if you're writing a disk repair utility.

Tianocore is not a disk repair utility, so it doesn't matter if Tianocore detects the wrong filesystem before refusing to mount it.
bzt wrote:Conclusion: FAT file system is poorly designed, and the FAT specification is badly written with full of contradictions. Period. That's the truth. Resistance is futile ;-)
I don't think anyone disagrees there!
Post Reply