SFS change request

[Candy]

Hi all,

I was wondering. In the SFS spec, there's a line that requires a device to have a block size equal to the "sector" size, if there's such a concept on the device.

In the case of my OS, I'm trying to get all devices to share block buffers & to make them directly mappable if possible. That means that non-4k block sizes (lower, in particular) are a pain to support & will incur lots of copying. Is it possible to request the devices to be formatted with 4k blocks instead of their native sector size?

The change is strictly speaking changing the writing from "must be equal to the sector size" to "must be a whole multiple of the sector size"; that'll at least allow me to format them with 4k blocks and call it SFS.

[Brendan]

Hi,

I was wondering. In the SFS spec, there's a line that requires a device to have a block size equal to the "sector" size, if there's such a concept on the device.

In the case of my OS, I'm trying to get all devices to share block buffers & to make them directly mappable if possible. That means that non-4k block sizes (lower, in particular) are a pain to support & will incur lots of copying. Is it possible to request the devices to be formatted with 4k blocks instead of their native sector size?

The change is strictly speaking changing the writing from "must be equal to the sector size" to "must be a whole multiple of the sector size"; that'll at least allow me to format them with 4k blocks and call it SFS.

I think the wording should be changed, but for different reasons. Imagine copying an SFS floppy image to a CD-ROM (e.g. for "floppy emulation El Torito", or any other reason). You'd end up with 512 byte blocks even though the disk has 2 KiB sectors, and if you take the specification literally you'd need to convert the entire file system from 512-byte blocks to 2048-byte blocks in this case (therefore, preventing it from working for "floppy emulation El Torito").

IMHO the specification should be changed: "For media that is organized into "sectors" (floppy disks, hard disks, etc) the block size must match the size of a sector." should be changed to "When creating a new SFS file system, for media that is organized into "sectors" (floppy disks, hard disks, etc) it is recommended that the block size match the size of a sector.".

I'd also be tempted to add something like: "When using an existing SFS file system, for media that is organized into "sectors" the block size (at offset 0x01BC) shall be used regardless of the size of the device's sectors.".

Of course I'd assume that some implementations won't support anything except 512-byte sectors (despite the original wording of the specification, or any future changes), because this is exactly what happened with FAT. I'd also say that, given that the entire purpose of SFS is to allow files to be transferred between different computers (with different OSs), I'm not too sure it makes any sense for an implementation to support 4 KiB blocks only (especially when that OS would have to support 512-byte blocks for FAT, and 2048-byte blocks for ISO9660).


Cheers,

Brendan

[Candy]

I think the wording should be changed, but for different reasons. Imagine copying an SFS floppy image to a CD-ROM (e.g. for "floppy emulation El Torito", or any other reason). You'd end up with 512 byte blocks even though the disk has 2 KiB sectors, and if you take the specification literally you'd need to convert the entire file system from 512-byte blocks to 2048-byte blocks in this case (therefore, preventing it from working for "floppy emulation El Torito").

IMHO the specification should be changed: "For media that is organized into "sectors" (floppy disks, hard disks, etc) the block size must match the size of a sector." should be changed to "When creating a new SFS file system, for media that is organized into "sectors" (floppy disks, hard disks, etc) it is recommended that the block size match the size of a sector.".

I'd also be tempted to add something like: "When using an existing SFS file system, for media that is organized into "sectors" the block size (at offset 0x01BC) shall be used regardless of the size of the device's sectors.".

Sounds like a good case.

Of course I'd assume that some implementations won't support anything except 512-byte sectors (despite the original wording of the specification, or any future changes), because this is exactly what happened with FAT.

We'll call that noncompliance. Maybe it's an idea to create a testset of disk images that you have to be able to mount that vary in each of the features at least once.

I'd also say that, given that the entire purpose of SFS is to allow files to be transferred between different computers (with different OSs), I'm not too sure it makes any sense for an implementation to support 4 KiB blocks only (especially when that OS would have to support 512-byte blocks for FAT, and 2048-byte blocks for ISO9660).

The implementation would support 4k blocks only, but not be the only implementation of that filesystem. There'd be two implementations, one supporting 4k and multiple-of-4k blocks, and one supporting <4k blocks. The first would allow no-copy use, the latter would always copy (as it has no real choice). The FAT implementation would check that (reserved sectors + fat_count * fat_size) is a multiple of 4k and that the block size is 4k or more, and then it'd say "yeah, I'll load this filesystem". If it isn't, it'll get to the second FAT implementation that handles the non-4k case. For SFS, there's no option to make a fast implementation right now. I'd like SFS to be the exchange medium of choice in the future so I'd like it to use 4k sectors - at least, when formatting disks it'd be nice to create them for quick usage. For FAT and Ext2 I can enforce 4k+ blocks and proper alignment (for Ext2 there's no choice, for FAT you can force it through reserved_sectors) so I can create quick-read/write disks.

I'm trying to find out if there's some way to figure out if FAT file systems are always clustersize-aligned for their clusters; so far I can't find any definitive spec that says it's a good idea, but I also can't find any example filesystem (although I only looked at one so far) that didn't agree with that. Iso9660 is a special case that is probably going to be slow, but doesn't really matter that much because it's cdrom only.

[Owen]

Why do block buffers matter?

As far as I'm concerned, buffering should occur at the file system level; it is the FS' responsibility to buffer its structures, but the OS responsibility to cache file data. File data would always be manipulated in 4kb chunks; for unmatched tail ends of files, I'd say that any content beyond the end is undefined (and will be discarded if modified).

[Candy]

Why do block buffers matter?

As far as I'm concerned, buffering should occur at the file system level; it is the FS' responsibility to buffer its structures, but the OS responsibility to cache file data. File data would always be manipulated in 4kb chunks; for unmatched tail ends of files, I'd say that any content beyond the end is undefined (and will be discarded if modified).

I decided to make the block buffering at a partition level, which means that the stuff above it needs to be aligned on 4k boundaries for that filesystem level. That allows me to use the same caching mechanism for everything without extra code & special cases. If the rest of the filesystem uses 4k blocks and considers the start aligned on a block boundary, that makes it trivial to cache parts of files. The downside is that the FS needs to be 4k aligned for it to work. Mostly relevant for my native filesystem, but removable drives can have the optimization as well if they're 4k compatible.

Also, this makes it implicitly compatible with the WD 4k drives (and the others that will follow in the same vein).

[Combuster]

What stops you from having the FS driver aligning all files to 4k boundaries, i.e. starting all files on a multiple of 8 hardware sectors? You can still use the caching for all files you wrote, and you aren't violating the SFS principle. The only problem is that caching doesn't work with foreign SFS devices.

Also, fixing something like that to a 4k sounds rather hardware specific - in this case, the granularity of pages. How will you deal with hardware with different (larger) granularities, which are bound to appear on actual storage devices somewhere in the future.

[Candy]

What stops you from having the FS driver aligning all files to 4k boundaries, i.e. starting all files on a multiple of 8 hardware sectors? You can still use the caching for all files you wrote, and you aren't violating the SFS principle. The only problem is that caching doesn't work with foreign SFS devices.

The ability to determine that at mount time without searching through the entire file system to see if it conforms.

Also, fixing something like that to a 4k sounds rather hardware specific - in this case, the granularity of pages. How will you deal with hardware with different (larger) granularities, which are bound to appear on actual storage devices somewhere in the future.

It is hardware specific. The only requirement is 4k or a multiple of that - and future development will most likely be in powers of 2, so that's ok.

Remember, the only thing I'm trying to get changed in the SFS spec is to allow me to generate 4k blocksize filesystems on 512b sectorsize devices, so that filesystem I generate are both SFS conforming and quick.