Using FUSE as an VFS

[zaval]

Didn't zaval report having exactly this kind of issue? I'm sure I've seen at least one other report of an SD card corrupting files only when formatted with ext4, but I can't remember where I found it or what I was searching for.

I did. It's a 16GB Kingston card, behaving severely erratic, being ext4 formatted and miraculously getting healed by the SD card formatter, getting back to the specified format. But it's bzt and the topic of SD card file system is one of his obsessions, he won't listen to or see any arguments. I posted here, several times, the quotes from the specification, where it is very clearly stated - FAT is mandatory but he keeps his <???>. Ignoring him would be the best approach.

Document, named "Part 2, File System Specification", it's not available as part of of the "simplified" version, but it is a part of the specification what is clearly seen in the Chapter 1. General Description, Figure 1-1: SD Specifications Documentation Structure of the part 1 of the specification. This specifation (p. 2) has Table 1-1: File System Type, under which, there is a text:

That is, all Standard Capacity SD Memory Cards whose capacity is 2048MB or less shall use FAT12 /
FAT16 file system, and never use the other file system. Similarly, all High Capacity SD Memory Cards
shall use FAT32 file system, and never use the other file system. And all Extended Capacity SD Memory
Cards shall use exFAT file system, and never use the other file system. This includes the prohibition of
partial format of SD Memory Card. For example, 8GB High Capacity SD Memory Card should not be
formatted as 2GB card with FAT12 / FAT16 file system. In this case, whole area of 8GB should be
formatted with FAT32 file system.

and the key words explanation added for avoiding uncertainty about if it is mandatory.

Key Words
• May: Indicates flexibility of choice with no implied recommendation or requirement.
• Shall: Indicates a mandatory requirement. Designers shall implement such mandatory
requirements to ensure interchangeability and to claim conformance with the specification.
• Should: Indicates a strong recommendation but not a mandatory requirement. Designers should
give strong consideration to such recommendations, but there is still a choice in
implementation.

What else could be said? bzt, in your universe, is there Part 2, File System Specification part? if so, read it, stop making yourself an arrogant laughingstock.

Nope, you just don't want to read. Even spoken of in the "simplified" spec. I've already did quote it, several times actually... In the simplified SD specification

Code: Select all

4.13.2.7.3 Requirements of SD File System
This specification can be applied only to the SD file system formatted card defined by the File System Specification Version 3.00.

Pay good attention on the wording: it does not say other file system's can't be used, it says "the following is true if exFAT is being used".

You are quoting the speed class specification and it really says, that only the standardly formatted cards will expose these behaviors, because it's "4.13.2.7 Measurement Conditions and Requirements of the Speed Class for SDXC". That is: without complying, you won't get these numbers. Jeez, it's not "yes, bzt, you can use whatever you want". But of course, screw such technicalties as context if you are obsessed to prove your delusion, right?

Of course, you can put incompatibe FS and partitioning scheme on the SD card and it might work. The mentioned card has been working for some time before going nuts. Of course, you can overclock the SD card and it might work. Despite the spec says "only 50Hz in legacy modes (pre UHS-I)". Rpi users do overclock. So what? It doesn't mean, that it doesn't cause card damage, or speeding up card wear out. Internal controller definitely expects standard specified format and it's not just FATs, there are more further specifications about what exactly parameters should be used. The fact, that many cards keep working being f&cked up with incompatible formats, just means, the internal FW resorts to some "worst" scenario and tries to handle it anyway, skipping loads of wear leveling technics or optimizations. Just as that "Others/Unknown" for FILE_FORMAT_GRP and FILE_FORMAT in CSD. It's there for the sake of completeness. Would be better if the spec told the controller just to return some "bad format" response to the screwed up card, forcing this way everybody to comply with standard and not invent square blinking wheels. bzt was pointing so hard to the CSD fields, but omitted the line below the Table 15-5, describing them. maybe because that line says:

A more detailed description is given in the File System Specification

And because the File System Specification clearly states the mandatory status of FATs, as shown in the quote above? There is no a single line in the part 2, something like "you can use other formats if CSD's FILE_FORMAT_GRP.FILE_FORMAT bits set to 0.3 and it will work or work the optimal way". It's apparently an "undefined" case and card manufacturer is free to make their own decision as of how the card should react to that. Who would want to rely on that? Moreover, p. 2 specifies exactly MBR and only 1 partition, so of all CSD file format bit field values, only 0.0 (Hard disk-like file system with partition table) are valid ones. And those bits are "writable once" btw.

All commercial OSs do use exactly these specified FSs. Only linux deviates, but it's just a clear incompliance. Did bzt measure, how many Rpi users have been experiencing card malfunction and sped up wear out and how their numbers correlate with such for normal users, using standard FSs? So who is here "fake", bzt? you see only what you want to see, your labeling people with that "fake" line in this light shows you yet once an arrogant person, whom it is better off to just not pay attention to.

[Octocontrabass]

Exactly the same thing like now. The difference is, it would only depend on the application's user locale (which the user can easy change if they experience problems), and not on some hidden property of the file system they can't see nor change.

My point is, the up-case table in exFAT prevents this situation from ever happening in the first place. If two files have the same name in the user's locale but different names according to the up-case table, the user will be able to access both files because the VFS uses the up-case table and not the user's locale to look up file names on an exFAT volume.

And let's not get into what happens if a medium is formatted on locale A, and then moved to machine with locale B which supposed to save a different conversion table. Will locale B's conversion be ignored? Or will it be merged with the table on the file system?

Locale B's conversion will be ignored. The volume must be formatted in order to modify the up-case table.

Can you create a uppercase table with both Turkish and Devanagari uppercase conversions for example? Or is it either this or that?

The default up-case table recommended in the exFAT specification is already suitable for multiple languages. If the default up-case table is not suitable for your chosen languages, you can provide your own, as long as you follow the limitations outlined in the exFAT specification. (Devanagari has no concept of "case" anyway, so it doesn't make sense to provide a table specifically for Devanagari.)

In theory it could be stored, but do drivers support that? Should they?

Yes and yes.

So many usability questions with this custom table on disk solution...

Wouldn't it be nice if Microsoft would give up on case-insensitive file names?

[rdos]

Wouldn't it be nice if Microsoft would give up on case-insensitive file names?

Definitely. I plan to handle FAT in a way that only 8.3 names are case-insensitive while LFN would be case-sensitive and upper-case conversions of Unicode would be unnecessary. It might create some inter-operational problems with other OSes, but I don't want this mess. I'm even amazed how Unicode could be localized. I always thought it was invented to get rid of that s*t, but apparently, it doesn't work fully like that.

[bzt]

Exactly the same thing like now. The difference is, it would only depend on the application's user locale (which the user can easy change if they experience problems), and not on some hidden property of the file system they can't see nor change.

My point is, the up-case table in exFAT prevents this situation from ever happening in the first place.

And my point is, it does not. It only prevents this if the exFAT was formatted on a particular machine with a particular user's locale, and never being used in another machine with another user's locale. Considering SD cards are primarily removable media, this is a failure by design.

Locale B's conversion will be ignored. The volume must be formatted in order to modify the up-case table.

The default up-case table recommended in the exFAT specification is already suitable for multiple languages. If the default up-case table is not suitable for your chosen languages, you can provide your own

Now you just got in a contradiction with yourself, on two accounts actually.

Plus you said it yourself, if the up-case table turns out to be unsuitable, you can't just simply provide your own table, there's no other option but to reformat (which will loose ALL THE DATA). You see, there's no such thing as "chosen languages". There's the user's locale of the machine on which the media was formatted, and there are other machine's user's locales where the SD card will be plugged in in the future, that you simply can't know in advance.

In theory it could be stored, but do drivers support that? Should they?

Yes and yes.

How so? A few sentences ago you said it yourself locale's B conversion will be ignored. And changing the up-case table on-the-fly isn't possible either, you've also said it yourself that the volume must be reformatted to change it. So even in the best case this can only be "No and no".

Wouldn't it be nice if Microsoft would give up on case-insensitive file names?

Or if they wouldn't try to push this poorly designed file system so badly and spread fake information that nothing else can be used just because they have a few patents on it. This is actually considered to be a crime according to EU law, "misleading customers". I hope M$ will get a huge fine because of this soon. They surely deserve it, they've scammed @zaval.

Cheers
bzt

[rdos]

The new low-level interface required for buffering disc sectors is now more or less done. The main difference between the typical fuse API (which use the file-IO API with a buffer) is that the function that requests disc data will return a mapped pointer to disc data (and thus will not take a buffer). The other difference is that the buffer pointer must be unmapped (released) when it is no longer needed, otherwise the sectors will continue to be locked and thus cannot be discarded in low-memory situations.

I will build my native API with C++ classes, which will make the map/unmap and lock/unlock process more automatic (the unmap & unlock with be part of the destructor).

There could be up to 15 requests per partition, and each request can have up to 255 sector ranges. For FAT, I think I will use one request for the first FAT table and one for the second FAT table, keeping the last x (maybe 8 or 16) sector ranges mapped & locked. The other requests I will use to read cluster chains for files & directories. This scheme will allow for creating much more efficient read operations that often can read complete file & directory data in just one operation, allowing the disc-driver to create large read operations to the device.

[rdos]

The issue of how to reliably share data between the FS code and applications is a pretty important one. Both sides should be able to terminate any link at any time. I don't want a full-blown Unix-style reference counting system per page as this is too complex and memory-hungry. Rather, I want a specific mechanism optimized for sharing meta data and file data.

I've started to look at meta-data sharing. I want to cache complete directories in FS memory space, and then I want the ability to take snap-shots of those for applications so I can implement opendir and readdir. It should be possible to map these either in kernel-space or user-space.

I need a new function to achieve this. I defined a header with reference count and page count. To create the object (in FS memory space), I allocate a single page frame, set reference count and page count 1. If I need more than 4k memory, I can extend it either by grabbing the next page in linear address space if it is free or by allocating a new continuous area and then move the existing physical pages to the new area. I then increment the page counter and add another physical page to the object. I can share this object to an application by incrementing reference count and then map the physical pages either in kernel space or user space. When the application no longer needs the object, it will decrement reference count (using lock prefix) and if it reaches zero, it must free the physical pages. The object will be read-only to the application. On the FS server side, if the directory content changes, the reference count is checked and if it is not 1 (it's been shared), then a copy is created with reference count = 1 and the reference count for the original object is decremented (and freed if it becomes 0). Now the FS server can forget about the old object and link the new object to the directory structure. This feature can also be used if the FS drive becomes unavailable (a removable drive). All the FS server needs to do is to decrement reference count and free those that reach zero and leave the rest allocated, and then detach and terminate itself.