Using FUSE as an VFS

[thewrongchristian]

Still, is there any reason to support exFat at all?

It's mandatory for SDXC, and cards may misbehave if you don't format them according to the SD standard.

WARNING: fact check: FAKE
Oh, not this bullshit again, please. SDXC cards are perfectly capable of storing other filesystems, nobody ever reported a problem, never. Plus even the SD standard defines a register to tell that the card isn't using exFAT, which register must be supported by all compliant drivers. So no, not mandatory for several reasons, period.

Cheers,
bzt

Not sure it's fake. Misbehave doesn't mean fail. It might be that the SDXC card has optimisations that work with exFAT, and not using exFAT bypasses those optimisations, which might result in poor performance or excessive wear. For example, some cards have been shown to have FTL optimisations so the FAT area of the disk is optimised for small writes, with more fine grained FTL mapping in those areas, whereas general data areas tend to have coarser grained FTL.

So while they're perfectly capable of storing other filesystems, it is absolutely possible that doing so is not optimal.

[Octocontrabass]

No, their reasoning is invalid, because when you're checking file names in a case-insensitive way, you must convert all i variants (İ i I ı) to the same uppercase dotless I.

Do you have a source for this claim? Everything I can find says İ/i and I/ı are considered two separate and distinct letters in Turkish. In addition, the up-case table recommended in the exFAT specification doesn't provide a conversion to uppercase for ı.

Actually the current solution introduces a pretty nasty bug. Let's assume you have an application that wants to open a filename with an İ in it from an exFAT formatted image. If you compile that and create the image on a machine with user's language set to Turkish, it will work, but in every other case the app will return "File not found".

Why would it do that? Maybe I'm missing something here, but I don't see any reason why changing the user's language would suddenly make it impossible to open files with İ in the name.

Plus even the SD standard defines a register to tell that the card isn't using exFAT, which register must be supported by all compliant drivers.

I seem to have misplaced my copy of the SDXC specification. Can you quote the relevant paragraphs?

[nexos]

Plus even the SD standard defines a register to tell that the card isn't using exFAT, which register must be supported by all compliant drivers.

If only everyone followed the standard... I'm sure many drivers assume exFAT. With that being said, you are right for the most part, as Windows allows you to format SDXC cards as FAT32,

[rdos]

I have most of the disc buffering interface done now, and it occurred to me that instead of letting the filesystem block on IO when it would read data, it can instead create a request for x sectors, link these sectors to a request structure, and as soon as those are done the request is marked as done and the owner thread is signalled. This way, the filesystem driver would mostly handle meta data and create queued requests for file data. Maybe this is good enough so there is no need to create parallell threads in "fuse".

I have 16 bits available per sector buffer, and I initially thought that I'd create a linked list of selectors, but it might be better to use it as a bitmap instead. One bit can be for general IO and the 15 other bits for partitions. Since partitions have defined areas, which paritition a sector belongs to is rather easy to calculate, and then the 15 bits would point to one of 15 possible active requests in the partition.

So, when a partition wants to request 100 sectors of file data, it picks one of the 15 request structures, fills in the termination info, creates requests for the sectors and sets the bitmap bit based on the request # in each of them. The request structure will have a counter of currently pending sectors, and every time a sector is finished, the counter is decreased in active requests and when the counter goes to zero, the owner thread is signalled.

Bit 0 will be special. It would be used to read/write raw sector data (either from initialization code or applications). When a thread requests a sector, a request structure is dynamically allocated, the sector # is filled in, as well as the thread to notify on completion. This structure is then placed in a link list, and bit 0 in the bitmap is set in the sector buffer.

[thewrongchristian]

I have most of the disc buffering interface done now, and it occurred to me that instead of letting the filesystem block on IO when it would read data, it can instead create a request for x sectors, link these sectors to a request structure, and as soon as those are done the request is marked as done and the owner thread is signalled. This way, the filesystem driver would mostly handle meta data and create queued requests for file data. Maybe this is good enough so there is no need to create parallell threads in "fuse".

I was literally thinking the same thing today, in which case the filesystem just becomes a complex device mapper.

I was thinking how to handle data transfer from device to user space with minimal copying, and basically the solution you describe above seemed to be the answer.

But that would only work for block based file systems, and so would be no use to the likes of sshfs or similar filesystems, which aren't backed by a block device, so you'd still need a buffer mechanism to pass data to/from the FUSE module, in which case a shared mapping might be a better strategy for copy avoidance.

[bzt]

Okay, SD card one last time, here we go.

Not sure it's fake. Misbehave doesn't mean fail. It might be that the SDXC card has optimisations that work with exFAT, and not using exFAT bypasses those optimisations

First, "misbehave" suggests malfunctioning or corrupt filesystem, while "bypassing optimization" suggests it's slow, but correct. (Neither happens BTW)

which might result in poor performance or excessive wear. For example, some cards have been shown to have FTL optimisations so the FAT area of the disk is optimised for small writes, with more fine grained FTL mapping in those areas, whereas general data areas tend to have coarser grained FTL.

Second, no there's absolutely no performance drawback, nor any other drawback, because it is not the card that classifies the sectors, but the host (see simplified SD card specification section 4.13.2.6 Distinction of Data Type and section 4.13.2.8 Speed Class Control). The host is supposed to tell the controller if it's writing directory data, allocation data or file data. A high speed write consist of: "The Speed Class specification for SDXC defines the CI Update sequence and operation time. The sequence is similar to the directory entry update. CI is always written by a 512-byte single block write (either CMD24 or CMD25) and preceded by CMD20 Update CI command. "

An ext4 (or any other) driver for example can easily comply with that, use these commands in this order, while providing its own (non-fat) data structures in the sectors.

I seem to have misplaced my copy of the SDXC specification. Can you quote the relevant paragraphs?

Nope, you just don't want to read. Even spoken of in the "simplified" spec. I've already did quote it, several times actually... In the simplified SD specification

Code: Select all

4.13.2.7.3 Requirements of SD File System
This specification can be applied only to the SD file system formatted card defined by the File System Specification Version 3.00.

Pay good attention on the wording: it does not say other file system's can't be used, it says "the following is true if exFAT is being used".

The simplified SD specification does not go into details, but it names that register specifically: see section 5.3.2 CSD Register, bits FILE_FORMAT_GRP and FILE_FORMAT bits.
And on page 12, in Table 3-2 : SD Memory Card Registers, it says:

CSD 128 Card Specific Data; information about the card operation conditions(See 5.3). Mandatory

Note that last bold word. Supporting CSD bits (including FILE_FORMAT bits) is mandatory. According to this, SDXC cards are allowed to use just as any file systems they want, exFAT is just optional (but pushed by M$ marketing department aggressively because they have active patents on it and they are making lots of money from the licensing fees.)

Cheers,
bzt

[bzt]

Do you have a source for this claim? Everything I can find says İ/i and I/ı are considered two separate and distinct letters in Turkish.

You've answered your own question: "considered separate in Turkish". And only in Turkish may I add, so if my machine uses any other localization, those aren't separate.

Actually the current solution introduces a pretty nasty bug. Let's assume you have an application that wants to open a filename with an İ in it from an exFAT formatted image. If you compile that and create the image on a machine with user's language set to Turkish, it will work, but in every other case the app will return "File not found".

Why would it do that? Maybe I'm missing something here, but I don't see any reason why changing the user's language would suddenly make it impossible to open files

Isn't that obvious? There can be only two possibilities:
a) If the UNICODE upper-case table is the same for all localizations, then there's absolutely no point in saving it on each and every disk.
b) And if it's not the same, differs per localizations, then exFAT file systems aren't portable, because a file saved on a Turkish machine might have a different upper-case name than a file saved on an English machine, hence causing file name collisions or file name not found errors depending on the user's language.

Cheers,
bzt

[Octocontrabass]

An ext4 (or any other) driver for example can easily comply with that, use these commands in this order, while providing its own (non-fat) data structures in the sectors.

Does that work even with USB SD card adapters? Last I checked, those don't give you any control over which commands are sent to the SD card.

You've answered your own question: "considered separate in Turkish". And only in Turkish may I add, so if my machine uses any other localization, those aren't separate.

My machine is not localized for Turkish, yet I can save a file that's named "i" and it's considered separate from a file named "ı". Or, I can save a file named "I" and it's considered separate from a file named "İ". (Also, Turkish is not the only language where İ/i and I/ı are separate letters.)

b) And if it's not the same, differs per localizations, then exFAT file systems aren't portable, because a file saved on a Turkish machine might have a different upper-case name than a file saved on an English machine, hence causing file name collisions or file name not found errors depending on the user's language.

But in your example, the file name contains "İ" and the program trying to open the file is searching for a file named "İ". Even if converting the name to uppercase changes "İ" into something else, the names will still match because both will be converted using the same up-case table.

Name collisions don't happen until you start trying to move files between the exFAT disk and something else.

[bzt]

An ext4 (or any other) driver for example can easily comply with that, use these commands in this order, while providing its own (non-fat) data structures in the sectors.

Does that work even with USB SD card adapters?

Absolutely. I do that on a daily basis. Moreover, I don't use a single file system, I use multiple with a partitioning table. I transfer files to my Raspi and my PC using an USB adapter, and all the cards are formatted with MBR+FAT+ext4, as RaspiOS likes its boot partition as FAT32 and root partition as ext4 (neither being exFAT). Nobody (hundred thousands RaspiOS users all around the world) have reported issues with that.

Last I checked, those don't give you any control over which commands are sent to the SD card.

It doesn't matter if USB SD adapter doesn't give you control over commands, because it's still the host that classifies the sectors and not the card. Worst case scenario, your USB adapter driver is using simple sector read / writes instead of the high speed CMD20 command, so it's going to be slower, but perfectly working. I did not noticed lower performance with my Kingston USB adapter BTW.

My machine is not localized for Turkish, yet I can save a file that's named "i" and it's considered separate from a file named "ı".

Sure, there's no issue with one computer. The problem appears when you move that file system between two computers with different locales, which need different upper-case for the same UNICODE code-point.

the names will still match because both will be converted using the same up-case table.

That's the point, not the same up-case table. What if one machine creates an upper-case table with a filename, and another machine wants to create another file with a different upper-case, but it can't because according to its upper-case that would map to the same name? Should it overwrite the table and create a new file? Or will it silently overwrite the existing file? What if it overwrites the table, creates a new file, and the file system is moved back to the original machine?

To very simply put, machine A wants an upper-case table where names X and Y would be the same. Machine B wants an upper-case table where names X and Y are different. Obviously machine A and machine B can't save both their upper-case tables to the same file system, because there's only one table. Such a file system will fail to work either on machine A or on machine B.

Name collisions don't happen until you start trying to move files between the exFAT disk and something else.

Yes, it could happen. That's because upper-case translation is not bijective as it differs per localization. That's why you shouldn't store localized names, a file system must store names in a localization independent format, and only apply the localization in the driver according to the current locale so that the file system can be used on different machines.

Cheers,
bzt

[Octocontrabass]

Nobody (hundred thousands RaspiOS users all around the world) have reported issues with that.

Didn't zaval report having exactly this kind of issue? I'm sure I've seen at least one other report of an SD card corrupting files only when formatted with ext4, but I can't remember where I found it or what I was searching for.

That's the point, not the same up-case table.

Why not? The VFS should use the up-case table on the volume when accessing that volume.

What if one machine creates an upper-case table with a filename, and another machine wants to create another file with a different upper-case, but it can't because according to its upper-case that would map to the same name? Should it overwrite the table and create a new file? Or will it silently overwrite the existing file? What if it overwrites the table, creates a new file, and the file system is moved back to the original machine?

There is only one correct behavior: tell the user a file already exists with that name.

To very simply put, machine A wants an upper-case table where names X and Y would be the same. Machine B wants an upper-case table where names X and Y are different. Obviously machine A and machine B can't save both their upper-case tables to the same file system, because there's only one table. Such a file system will fail to work either on machine A or on machine B.

Or both machines will use the up-case table on the volume and behave the same way, even when this behavior might be inconsistent with the user's locale. User A might be confused if names X and Y are allowed to coexist in the same directory on an exFAT volume when they're not allowed to coexist elsewhere. User B might be confused if names X and Y are not allowed to coexist in the same directory on an exFAT volume when they're allowed to coexist elsewhere.

Consider the alternative: user B creates files X and Y, then user A tries to open one of them. On machine A, both files have the same name, so which file gets opened? I suspect the up-case table exists specifically to prevent this scenario.

That's why you shouldn't store localized names, a file system must store names in a localization independent format, and only apply the localization in the driver according to the current locale so that the file system can be used on different machines.

What does this even mean? Names are always localized!

[bzt]

Nobody (hundred thousands RaspiOS users all around the world) have reported issues with that.

Didn't zaval report having exactly this kind of issue? I'm sure I've seen at least one other report of an SD card corrupting files only when formatted with ext4, but I can't remember where I found it or what I was searching for.

Nope. Plus if only exFAT were supported, you couldn't use partitioning on the SD cards, but everybody can.

That's the point, not the same up-case table.

Why not? The VFS should use the up-case table on the volume when accessing that volume.

You still don't get it do you? Let me put it as simple as possible: Turkish machine would put Turkish table on the volume, English machine would put English table. Turkish table != English table.

There is only one correct behavior: tell the user a file already exists with that name.

So you admit, there can be a name collision.

Or both machines will use the up-case table on the volume and behave the same way

Nope. For the last time, I'll try to explain.

Let's say given application A, which operates on a removable device. A is installed on two (otherwise completely identical) computers, a Turkish and on an English one. Depending if the removable media used by the application was formatted on the Turkish machine or on the English machine, exactly the same A application on the Turkish machine would behave differently than exactly the same A application on the English machine. All it takes to trigger the bug is to use a removable media formatted on the other machine (hence using different tables).

What does this even mean? Names are always localized!

You are confusing name localization with name encoding, just like those M$ guys who created this terrible exFAT specification. You're wrong. UTF-8 encoding for example isn't localization dependent. It works exactly the same way on a Turkish machine and on an English machine. Filenames are encoded and stored to the file system exactly the same way, bit-by-bit and decoded on the other machine correctly no matter the user's locale setting. Upper-case matching should be implemented in the driver, independently to the file system, so both the Turkish machine and the English machine would display correctly the name in uppercase. Then there would be no issues when the media is moved from one machine to another, because the file system would only contain localization-independent encoding of the file names, no matter if it was formatted on a Turkish machine or on an English machine.

Cheers,
bzt

[Octocontrabass]

You still don't get it do you? Let me put it as simple as possible: Turkish machine would put Turkish table on the volume, English machine would put English table. Turkish table != English table.

Correct.

So you admit, there can be a name collision.

Yes, of course. You must be prepared for this anyway, since users may try to give the same name to two different files.

Let's say given application A, which operates on a removable device. A is installed on two (otherwise completely identical) computers, a Turkish and on an English one. Depending if the removable media used by the application was formatted on the Turkish machine or on the English machine, exactly the same A application on the Turkish machine would behave differently than exactly the same A application on the English machine. All it takes to trigger the bug is to use a removable media formatted on the other machine (hence using different tables).

Again, correct.

You are confusing name localization with name encoding, just like those M$ guys who created this terrible exFAT specification. You're wrong. UTF-8 encoding for example isn't localization dependent. It works exactly the same way on a Turkish machine and on an English machine. Filenames are encoded and stored to the file system exactly the same way, bit-by-bit and decoded on the other machine correctly no matter the user's locale setting.

I think you are confused here. exFAT stores file names in UTF-16, and they are encoded and decoded exactly the same on all machines regardless of the user's locale.

Upper-case matching should be implemented in the driver, independently to the file system, so both the Turkish machine and the English machine would display correctly the name in uppercase.

The uppercase file name is used only for case-insensitive file name comparison. It is never stored on the disk or displayed to the user.

Then there would be no issues when the media is moved from one machine to another, because the file system would only contain localization-independent encoding of the file names, no matter if it was formatted on a Turkish machine or on an English machine.

How does this resolve the problem when two files on the disk have different names in one locale and the same name in another locale? When two files have the same name, how does the user choose which one to access?

[bzt]

I think you are confused here. exFAT stores file names in UTF-16, and they are encoded and decoded exactly the same on all machines regardless of the user's locale.

That's where you're mistaken. The conversion table is stored in the file system, and it does depend on the user's locale. Meaning you'll get a different exFAT image if you format it on a Turkish machine and if you format it on an English machine.

The uppercase file name is used only for case-insensitive file name comparison. It is never stored on the disk or displayed to the user.

That's the problem, it is stored on disk, but not shown to the user. You can't see if a medium was formatted on a Turkish machine or on an English machine, yet it influences file names and therefore application behaviour greatly.

How does this resolve the problem when two files on the disk have different names in one locale and the same name in another locale?

It does not solve that, it makes the behaviour deterministic and predictable, consistent without hidden traps. For machines with locale A would always be different, for machines with locale B would always be the same, regardless in which machine the medium was formatted.

Cheers,
bzt

[Octocontrabass]

That's where you're mistaken. The conversion table is stored in the file system, and it does depend on the user's locale. Meaning you'll get a different exFAT image if you format it on a Turkish machine and if you format it on an English machine.

The conversion table is not part of the file name. The file names are stored exactly the same.

That's the problem, it is stored on disk, but not shown to the user. You can't see if a medium was formatted on a Turkish machine or on an English machine, yet it influences file names and therefore application behaviour greatly.

You're right, it's not a great solution, but it's still better than allowing two files with the same name.

It does not solve that, it makes the behaviour deterministic and predictable, consistent without hidden traps. For machines with locale A would always be different, for machines with locale B would always be the same, regardless in which machine the medium was formatted.

Two files are created in locale A. In locale B, the file names are equal. What happens when the user of machine B tries to open one of the files? Do they get the file they chose? Remember, both files have the same name! Machine B can't tell them apart!

[bzt]

It does not solve that, it makes the behaviour deterministic and predictable, consistent without hidden traps. For machines with locale A would always be different, for machines with locale B would always be the same, regardless in which machine the medium was formatted.

Two files are created in locale A. In locale B, the file names are equal. What happens when the user of machine B tries to open one of the files? Do they get the file they chose? Remember, both files have the same name! Machine B can't tell them apart!

Exactly the same thing like now. The difference is, it would only depend on the application's user locale (which the user can easy change if they experience problems), and not on some hidden property of the file system they can't see nor change.

(And let's not get into what happens if a medium is formatted on locale A, and then moved to machine with locale B which supposed to save a different conversion table. Will locale B's conversion be ignored? Or will it be merged with the table on the file system? Can you create a uppercase table with both Turkish and Devanagari uppercase conversions for example? Or is it either this or that? In theory it could be stored, but do drivers support that? Should they? So many usability questions with this custom table on disk solution...)

Cheers,
bzt