Why not FAT file systems?

[PeterX]

I thought this might lead the original thread too off-topic. So I open this new thread.

Personally, I'd never suggest that anyone use any form of FAT. But I respect the right of others to suggest it.

1.) Why not FAT? It's not worse than the way ext2 handles the things.

Greetings
Peter

[bzt]

Well, I personally wouldn't use ext2 either

The main reasons against FAT:

• It is highly ineffective. The meta data overhead in a FAT is huge.
• Not well specified: to avoid overhead, you might want to increase the cluster size. But if the number of total clusters drops too low, some tools and firmware incorrectly assumes a different type (thinks it is a FAT12 when actually it is FAT16)
• You can't resize the partition easily. To increase the partition size, you have to increase the FAT table size, which means COPYING ALL the data on the partition.
• By default it uses 8+3 naming scheme, which is inadequate these days, and the LFN extension is patented, and must be workarounded to avoid lawsuits. Ehhh.
• Finally, it stores only very very limited file meta data, there's no place for access rights for example (which is good for an interchange format, but not so much for internal use)

With ext2, the last two points solved, and using node groups you can mitigate the effects for the third. However it has a total number of files allowed on a partition, which I really don't like to limit.

Cheers,
bzt

[Ethin]

Hate to drag this OT (sorta), but since we're talking about FSes, what would be a good FS to start with? I've looked at the wiki and the various FSes it provides. Was thinking of going with FAT32 just because EFI firmware has to implement that, or maybe even LEAN (that one looks pretty nice as a FAT32 alternative).

[nullplan]

By default it uses 8+3 naming scheme, which is inadequate these days, and the LFN extension is patented, and must be workarounded to avoid lawsuits. Ehhh.

To my knowledge, the last remaining patent expires this year, and only concerns representing long and short file names in a single namespace.

Hate to drag this OT (sorta), but since we're talking about FSes, what would be a good FS to start with?

Uh oh, let's hope this doesn't devolve into a flame war.

I'm personally going with ext2. It is extremely simple, yet feature-complete for a UNIX system, and ext4 is simply a souped-up version of it. So it can't be all wrong if it is shipped as the default FS. FAT12/16/32 of course must be implemented as well, though I would only use it as compatibility file system for information exchange with other OSes. That is basically how everyone else is using it right now. Other file systems seem to be playing a lot of the same notes, so there is not really a strong reason to prefer one over the other.

[Ethin]

Yeah, I'm hoping this doesn't devolve into a flame war either (I'd hope that we're all mature enough to avoid that, no offense intended...). I'm honestly divided. Ext2 is a good choice, and I've already got a disk formatted with that, but I've yet to add a VFS layer. I'm also not really sure how to go about implementing most of the functionality that EXT2 requires. I'd look at how Linux does it, but that seems like an over-complicated solution.

[iansjack]

I'm not sure that it matters whether the filesystem you use is supported by other OSs. To my mind, the easiest way to develop a FS is as a user program, which you can then transfer into a monolithic kernel or make into a service. It's hard enough testing and debugging a FS without debugging a kernel at the same time.

So you implement your own support. For a beginner, I'd choose ext2 or perhaps the Minix File System. Both are well documented and readily easy to implement (even if your implementation is initially sub-optimal).

[Octocontrabass]

Not well specified: to avoid overhead, you might want to increase the cluster size. But if the number of total clusters drops too low, some tools and firmware incorrectly assumes a different type (thinks it is a FAT12 when actually it is FAT16)

I'd say it's pretty well specified. The specification notes that you should avoid generating a filesystem with anywhere from 4068 to 4101 clusters, since that's the range where tools and firmware often get the type wrong.

You can have a FAT16 filesystem with fewer than 4085 clusters, but only on an Atari ST hard disk, so you might as well pretend it's not possible.

You can't resize the partition easily. To increase the partition size, you have to increase the FAT table size, which means COPYING ALL the data on the partition.

That's not true. If you need to increase the FAT size beyond the available reserved sectors, you only need to move files located within the area that will be occupied by the new larger FAT. For the other files, their cluster number changes so it's only necessary to generate new directory entries (and of course the new FAT).

The same process can also be used when moving the partition without resizing.

The same process can also be used when moving or resizing other filesystems: generate the appropriate metadata around existing file data, and only move the file data when necessary.

[Korona]

Neither FAT nor ext2 are good file systems. ext2 might be slightly more flexible and perform better compared to FAT but it's also slightly more complex. However, both are outclassed by any modern FS. Hence, I would argue that the choice doesn't really matter. Pick any file system you like when you start and put support for a good file system (like ext4, Btrfs, ZFS, XFS, ...) onto your "things to do at some point" list.

[nexos]

I will be initially using FAT, but switch to something else later. I think it is OK, besides the 8.3 naming convention. Personally, I think the only good filesystem in NTFS, but it is to undocumented

[iansjack]

Personally, I think the only good filesystem in NTFS, but it is to undocumented

Personally, I'd go for ZFS, but I've no idea how easy it is to implement. Apple's APFS is pretty good too. Both are miles ahead of NTFS IMO.

[PeterX]

Neither FAT nor ext2 are good file systems. ext2 might be slightly more flexible and perform better compared to FAT but it's also slightly more complex. However, both are outclassed by any modern FS. Hence, I would argue that the choice doesn't really matter. Pick any file system you like when you start and put support for a good file system (like ext4, Btrfs, ZFS, XFS, ...) onto your "things to do at some point" list.

But a beginner might start with tar (it's really easy) or FAT or ext2, because they are simple.

As you mention Btrfs: It's not in the list yet. Anyone want to write a text about Btrfs? I don't know enough about it to write it myself.

And, yes, I, too, have heard that journaling filesystems are much more robust, for example if the system crashes or the power goes off, then less data is lost than with an old FS.

Greetings
Peter

[bzt]

Huh, so many posts

@Ethin: true that EFI uses that, so you have to implement it anyway; however I don't think that's make a good root fs for your OS. But definitely a good and easy start. I was also messing around with LEAN, if there's a need I can add support for generating LEAN partitions on-the-fly from directories.

@nullplan: if the patent expires, that's cool Yes it only involves MICROSOF~1 names. Unfortunately the other problems still exists. Some of those were sorted out in exFAT, but not all.

@Octocontrabass: by "not well specified" I mean there's no "FATEntrySize" field in the BPB. Different tools implement FAT size differently, not all checks the total number of clusters, rather look for the magic bytes "FAT12", "FAT16" and at a different location, "FAT32". MSDN recommends the former, but I'd argue that the latter is a much much better solution, since it is perfectly viable to create a FAT16 with 2048 clusters for example or FAT32 with 4096 clusters. I can't recall it right now which one is which, but mtools and mkfs.vfat uses different approach to identify the size of clusters, that's for sure, I've spent a day with debugging because of it...

@Korona: you're perfectly correct. But for a beginner, specially for the first fs, I'd rather choose something simple. I'd go for FAT (because of UEFI and file interchange) and ext2 for my root (because of access rights). Both are easy to implement and having at least two file systems builds the bed for a VFS. One could run into trouble later if they tie their kernel to one fs at the beginning.

@nexos: ntfs is somewhat documented, but it is an extremely complicated file system. You'll spend months to get it right. The ntfs3g fuse driver is your best chance to figure out how things should be done the ntfs way. It is stable, and compatible with the undocumented parts too.

@iansjack: what other OS supports matter only when you want to exchange files. UEFI is an exception, because you should be able to interface with it, but you can still boot without implementing FAT (using EFISimpleFileSystem protocol). Otherwise doesn't really matter, I agree that you should focus on the features your kernel needs.

I haven't looked into APFS yet, but I have more than a decade of maintainer experience with ZFS under Solaris, and I've read its spec from top to bottom several times. Extremely complicated, and let me tell you, it sucks big time to use. Moreover, I seriously doubt you can implement that in a hobby OS. Just think about that for Sun it took 4 years to develop (and Solaris wasn't able to boot from it at that point) with a large dedicated team. And look at those poor FreeBSD guys how much they suck to port OpenZFS (they are not rewriting it, just porting it) for more than 7 years now and still not complete and has serious flaws.

FYI: unlike the name suggest, ZFS is actually not a file system. It is a complete volume and storage management solution totally incompatible with any other classic Windows/UNIX paradigms, which Sun developed to force Veritas out of its markets because that two company had some differences. It's like implementing RaiserFS+LVM+software RAID0,1,5 all at once as your first fs.

Cheers,
bzt

[bzt]

And, yes, I, too, have heard that journaling filesystems are much more robust, for example if the system crashes or the power goes off, then less data is lost than with an old FS.

Don't fall for the hype You can guarantee robustness and consistent file system in case of crash or power offs with soft update as well. Journaling is not the only way, and it is not the best solution either. Soft updates were used by FILES-11 for example. Designed by the same team as ntfs, but for an architecture which had way more resources than PCs. FILES-11 had all the features that modern fses are so proud of: ACLs, snapshots, quotas, extents, data checksums etc. etc. etc. Plus it also supported file versioning (per file snapshots if you like). I f*cking loved when I messed up my code I could copy SOURCE.C;-1 to get the version before the last save That was years before CVS, SVN or git was even invented (BTW, have you ever wondered where that ";1" file name part on ISO9660 come from?) And FILES-11 predates every other, btrfs, hammerfs, ntfs, ZFS etc. by a decade at least.

Cheers,
bzt

[eekee]

Bear in mind it took Linux and FreeBSD developers years to get BtrFS and ZFS working well enough for professionals. Plan 9's Fossil-Venti stack, which also offers snapshots, deduplication, & I think primitive forms of some other features, had a major data corruption bug unfixed for around 10 years. It's fair to say there were fewer people to maintain Plan 9 in that era. It was after "Ken has left the building." 9front, which forked before it was fixed, removed Fossil. Even now it's fixed, they refuse to include it. If I remember right, they don't want to deal with its complexity.

Description of Fossil unreliability & the problems it caused; begin at (search for) "Fossil-venti unreliability"; stop at "The Tribes" unless you like social problems:
http://doc.9gridchan.org/blog/200110.fossil.9front
There's also a comment, "Other users with large data sets have expressed some disappointment with performance."

Speaking of performance, I had such a horrible experience with ext3 that I twitch whenever anyone mentions journaling. Once, a simple :w command in gvim took 17 minutes to complete! i was running a complex search in the background with unix tools, and ext3's paranoid journaling defaults somehow prevented it from fitting gvim's write request into the flood of reads and stream of writes from the search. (There are articles about the problem.) I laugh about it now, but at the time, one of the things I most valued about Linux was its responsivity, so this was disturbing! I switched (back) to reiserfs, and was happy despite a little data loss. Years later I used ext4, in which I experienced approximately as much data loss as the very worst released version of reiserfs.

tl;dr:
* zfs & btrfs may very well be too much for a lone developer; they were almost too much for linux & freebsd teams
* i'm seriously unimpressed with linux's implementations of ext3 & ext4, and by extension, journalling

I remember a top-flight sysadmin of my acquaintance trying to discourage a friend from developing a filesystem. He said something like, "It's the worst thing for subtle bugs which aren't caught for years." So plan very carefully, I guess?

[OSwhatever]

From a hobbyist perspective, there is nothing wrong with FAT filesystems. They are simple, easily understood, tons of tools out there. If you want a file system for your hobby OS fast, that's one way to go.

Keep in mind that the FAT filesystem is starting to showing its age in terms of limitations. 128GB official maximum volume size (can be larger if using a non MS format tool), 4GB file size limit, not crash safe (FAT32 offers "double FATs" but still), proper long file support is complicated (USC2 and code pages like it was the 90s). If you want to learn about modern file system design, start with ext2 and go from there, read about Btrfs and XFS and so on. These more feature rich modern filesystems are a handful to implement and will take very long time for a single person to implement.