Directory free FS and binary configs.

[Pype.Clicker]

i'd say it's bad because it's redundant, not because of the waste of space.

Imagine you have ID3 on a mpeg file that happens to mismatch the metadata in the database... which one is good ? you search for "author=Madonna" and the resulting file displays on your system as "Bon Jovi -- It's my life" ...

I fear that proper metadata support is not only the problem of the FS, but quickly becomes a concern for every applications.

[Solar]

The "waste" of space isn't the issue. Redundancy in data storage is bad, because (like Pype said) you will end up with contradictionary data sooner or later.

It's different in data safety, where redundancy is the solution of choice (checksums, RAID arrays, multiple backups of the root block etc.), but that is a type of redundancy that the user cannot see.

[JoeKayzA]

Also, why on earth would you need an exporter for every format? I mean sure, an intelligent exporter would spit out My\ Masterpiece.doc and Mesmerize - 5 - Radio/Video.ogg, but the file itself would just be like any other file and be accessed just like any other file.

If you design it that way, you wouldn't need an additional exporter per file type, right, as you wouldn't need self-designed file formats either. I think you'll come to something like BeFS (the BeOS' native FS) , which is also famous for it's custom metadata support and its fast and rich query functions - the only difference here is that BeFS decided to keep the hierarchical directory tree, in addition to metadata But it was possible to categorize and look up files without browsing the directory tree.

An ogg-jukebox app could create appropriate attributes for the ID3 tags at time of importing (so the app is the importer then), and use them for organization. But here lies exactly the problem that Pype and Solar mentioned above - when the user then uses the file browser to change an attribute (say, the song title), the information from the stream (ID3 tags) and the metadata attribute are no more synchronous, because the browser doesn't know about the ogg-specific tags in the data stream.

Aside from the redundancy issue: To keep them sync, you could provide a plugin for the file browser (or for the FS layer in general), which maps the attributes to pieces of data inside the data stream (again, the ID3 tags, in this example). But then again, you'd need a plugin for every specific file type!

cheers Joe

[stonedzealot]

Ahhh, now I see the problem here. So what's to keep the OS from reindexing a file when it's written to to avoid such contradictory information? Wouldn't that solve the problem?

[Solar]

Erm... I would consider that mandatory if you are keeping redundant information. But it doesn't solve the problem of keeping information that should be the same in two different locations.

[stonedzealot]

Well, why not? I mean... when the file is reindexed, shouldn't it be smart enough to make the information in the DB the same as the information in the flat file?

[Solar]

You have much to learn yet, young padavan.



Of course, if everything goes as planned, you will never have inconsistent data.

But what if your data gets borked anyway, and someone calls for a "remove all from Britney Spears", and wipes out all his source code?

[JoeKayzA]

Well, why not? I mean... when the file is reindexed, shouldn't it be smart enough to make the information in the DB the same as the information in the flat file?

Well, that's quite exactly what I mentioned above - you could provide a plugin for the FS layer, which extracts information out of the data stream and presents them as attributes in DB-style (and the other way round - changing these values). But again, this plugins need to be filetype specific.

cheers Joe

[Solar]

And we've not even started into the subject of compatibility, if you really want to make it "directory free" - many "portable" languages have a concept of directories. Perl, for example, or Java. If you do not provide directory primitives, how do you enable programs written in these languages?

You will end up "emulating" directories on top of your DBFS, and at that point, you're back at the question whether it's really worth the hassle, or if a "traditional" FS with a search index cache isn't easier to handle, both for the app programmer and the user.

[Pype.Clicker]

The concept i came with last time i was fighting against directories could be called to be "directory-free" rather than "directory-less".

I mean, keep directories for what they are: directories (exhaustive lists of data items). Usually, you build directories based on some criterion and you ensure everything finally ends up in one directory. I don't think there's a real need to have per-author and per-album sub-directories, for instance: that is nowadays more a problem for the tools that access the directory than a problem of the FS itself.

It may, however (or it may not, that's upon you, user) make sense to have a directory of "multimedia" stuff and a directory of "accounting documents".

Programmers need sub-directories for their own purpose ? fine. Each "program" (or each "project") is a suitable item for the "programs" directory. Sub-directories within one project are something internal to that project and there's no real need for files to be kept apart in those stuff. I mean, are you more frequently accessing the whole bunch of your files for a grep or a compile or accessing them just those files in include/net ?

I think directories is a legacy borrowed from the ancient time where there was so little data on computers that you could find a couple of criterion that allows them to be kept apart. beholding to a directory is a criterion, such as "extension" is a criterion (what do you perform most often: delete all .o files from a project or delete all files from a directory ?) "calling function foobar()" could be another criterion. "anything created today" is yet another criterion and "anything that #include this file" is another one.

shouldn't "view *.c" work same as "view file.h:includers" ?

[JoeKayzA]

And we've not even started into the subject of compatibility, if you really want to make it "directory free" - many "portable" languages have a concept of directories. Perl, for example, or Java. If you do not provide directory primitives, how do you enable programs written in these languages?

Isn't that more about API compliance rather than language support? When you want to provide a fully functional Perl or Java runtime, you are expected to provide the standardized APIs too, but IMO this goes into the same category as implementing, say, POSIX APIs for your OS. It depends on whether compatibility is an issue to you or not.

You will end up "emulating" directories on top of your DBFS, and at that point, you're back at the question whether it's really worth the hassle, or if a "traditional" FS with a search index cache isn't easier to handle, both for the app programmer and the user.

For the app programmer, when porting an existing app, yes, probably. Not sure about the user though...

But it's a good question: is it really neccessary to remove the directory hierarchy, as long as all the features we wanted (rich metadata, fast searches, easy locating of files without browsing the tree) are implemented, and the system takes full advantage of them?

What about providing a thing like the 'VFolders' in windows vista? Basically, a DB query (in vista without WinFS it's rather a traditional filesearch), stored under a custom name, which can then be accessed just like a real directory. But I'm afraid if you take that concept further, you'll end up with the above anyway....

cheers Joe

[DennisCGc]

Just think of high-end servers who need their files quick (with a 'simple' command, like: SELECT content FROM `files` WHERE `filename` LIKE 'foo', you get my point.. ?)

Honestly? No. How is that SELECT "simpler" than [tt]fopen( "foo", "r" );[/tt]??

I single-quoted simple. Anyways, I can imagine sometimes a SQL query can be easier than a simple fopen. Okay, let's have this scenario. User 'A' searches a file with the keywords 'foo'.

What would be simpler? Do a search and then a fopen? (the searching without at least an indexed FS requires a lot of fopen's) or entering a SQL like command? (either done by software or the user itself) Only for that sort of operations a DBFS can perform better.

But I still prefer a traditional filesystem with an XML like document, containing the keywords of a file, etc.

DennisCGc.

[Solar]

Okay, let's have this scenario. User 'A' searches a file with the keywords 'foo'.

What would be simpler? Do a search and then a fopen? (the searching without at least an indexed FS requires a lot of fopen's) or entering a SQL like command? (either done by software or the user itself)

It's not done with "an SQL like command". What does the command return? File handles? The file contents concatenated together? You still have to process whatever the SQL command returned, including error handling for empty returns. And you have to pass that SQL-like command through some custom API. When I suggest, like, [tt]FILE ** vfopen( void * predicate, char * mode )[/tt] where [tt]predicate[/tt] is a pointer to a function that decides whether a file fits the criteria or not (the "where" part of the SQL query)...

[Pype.Clicker]

About searching for keyword "foo", etc. You will map _that_ more like a "directory" open than a file open, imho, in the sense that it returns you a collection of files.

(that's rather funny the concept of "folder containing the result of a search" now appears in Windows VISTa as i'm trying to design something alike for roughly 2 or 3 years now with my documents managing system ;D )