FILE SYSTEM vs DataBase System

frank · Post by **frank** » Tue Jul 03, 2007 11:16 am

Alboin wrote:
frank wrote:It wouldn't be that hard to make the database engine understand the contents of most files and for the other files they could be stored with meta data just like on a real filesystem.
You could even have a module-type system so each application could have a module that would allow the database system to understand its own file type. (Blender, for instance, would load a module that would allow the system to preview, read, etc. 3D type files.) Then other applications could use each other's module's to read and write those types of files.

I think this is what WinFS was going for. Reading about it just makes me want to use it...

Yeah I was really hoping for WinFS. When they decided not to put it in Vista I lost just a little bit more of my respect for Microsoft.

Colonel Kernel · Post by **Colonel Kernel** » Tue Jul 03, 2007 4:03 pm

WinFS is not the only approach. "Database" does not necessarily imply "relational database".

<anti-MS-rant>

IMO WinFS is overkill. You can get the same benefits without the relational DBMS back-end. In fact, it's already been done. IMO it makes more sense to study a working implementation that's already been providing tangible benefits to users for 3 years than to study a half-baked piece of mostly-vaporware.

</anti-MS-rant>

As I said before, you don't need the relational model in order to have good search capabilities and extensible metadata. I have a hard time imagining what value a relational DBMS adds to such a scheme in the first place. Did WinFS have any support for transactions? Triggers? Referential integrity? I'd be surprised if these features make much sense on unstructured file data... For example, why bother with RDBMS-style transactions when journalling filesystems already exist?

df · Post by df » Fri Jul 06, 2007 1:15 pm

a DB as an FS is a great idea (id love to do an open source version of something like OS400)...

a normal FS is a DB..

when most people go 'oh a DB would make a great FS' they commonly instantly forget you need on-disk-structures and such to manage the DB.

some DB systems used raw disk (enterprisy stuff), bypassing the filesystem, but still you need the whole free space/bitmaps/inodes type deal to manage all the data.

the idea is great, but it comes down to the implementation which suddenly becomes hairy.

axilmar · Post by **axilmar** » Fri Jul 13, 2007 8:19 am

Users need DBMSs, not file systems. Consequently, O/Ses need to manage information, not bytes.

Once the jump to managing information takes places, a lot of 'protocols' will go away, and computing will be a much better experience.

JamesM · Post by **JamesM** » Mon Jul 23, 2007 6:55 am

@axilmar:

You are forgetting that bare hardware will never be able to support markup structures/complex data. They will ALWAYS only support numbers, and as such all OS's will ALWAYS need to transmit information in words (byte length standard could change). How much marshalling/unmarshalling actually takes place in kernel-space can be variable, but it's just another layer of abstraction.

(by always I mean in the forseeable future, not a millennium from now).

AndrewAPrice · Post by **AndrewAPrice** » Mon Jul 23, 2007 7:58 am

SQLFS? How more abstract do you want?

I've had an idea of standardising common file types (raw files, images, sounds, contacts, e-mails). My idea is about representing files as having properties based on their type rather than just as raw binary. For example, imaging the following is how a file-system stores data: (I'm only using XML to show my idea - in a real-world scenario you wouldn't store your FS in an ASCII-readable format)

A conventional file system has, around 3 properties for a file: name, permissions, stream of data.

What if, instead of just having a stream of data, the entry can have more properties depending on it's type. If it were a contact's details stored on the FS, it could be represented as:
<file type="contact" name="Joe Bloggs" e-mail="jblogs" picture="PNG)(*#@)$*#@)$*"></file>

An offline copy of a HTML file could be represented as:
<file type="offline-html" name="mysite.html" origional-url="http://www.mysite.com/mysite.html" last-updated=200700724>raw-html-data-here</file>

There could be a limitless number of fields (for associating attachments with a page). A binary could be represented as:
<file type="binary" developer="Joe Bloggs" homepage="http://www.mysite.com" version=1.4.2 icon="#@$*@)($*@" minimum-memory=4096>raw-binary-data-here</file>

A file could have an owner, and the database could look for the contact entry, and then list all other files also associated with that owner.

The operating system should have a 'plug-in' interface so it can understand different types of files, what to do with them, and how to associate them with other types.

Colonel Kernel · Post by **Colonel Kernel** » Mon Jul 23, 2007 12:59 pm

MessiahAndrw wrote:The operating system should have a 'plug-in' interface so it can understand different types of files, what to do with them, and how to associate them with other types.

As I said before, it's already been done.

AndrewAPrice · Post by **AndrewAPrice** » Tue Jul 24, 2007 4:01 am

Colonel Kernel wrote:
MessiahAndrw wrote:The operating system should have a 'plug-in' interface so it can understand different types of files, what to do with them, and how to associate them with other types.
As I said before, it's already been done.

Spotlight is getting close to what I mean, but not exactly. Spotlight stores the file in a conventional way but with extra metadata.

I'm thinking of a completely new way to store files. A file will not have 'data' in a traditional way, with meta-data wrapping around it. Instead, a file will just be an 'entry' in the FS, with various fields each with their own attributes (e.g. each file has a Name attribute an a type attribute as a bare minimum, an image will also have a width attribute, a height attribute, a bpp attribute, what format it is attribute (raw bitmap, compressed raster, compressed vector) and an image-data attribute). There will be no data this entry wraps around (i.e. it's not a file 'header'), rather all data is stored in the entry as an attribute.

Colonel Kernel · Post by **Colonel Kernel** » Tue Jul 24, 2007 6:51 am

MessiahAndrw wrote:Spotlight is getting close to what I mean, but not exactly. Spotlight stores the file in a conventional way but with extra metadata.

I'm thinking of a completely new way to store files.

What advantage is there to such a scheme? Wouldn't it make it harder to trade files with other file systems without stripping off the extra metadata (like Mac resource forks in the days of yore)? A lot of important file formats already include their own metadata (mp3, jpeg, most video formats I can think of, etc.)

AndrewAPrice · Post by **AndrewAPrice** » Tue Jul 24, 2007 7:31 am

Colonel Kernel wrote:
MessiahAndrw wrote:Spotlight is getting close to what I mean, but not exactly. Spotlight stores the file in a conventional way but with extra metadata.

I'm thinking of a completely new way to store files.
What advantage is there to such a scheme? Wouldn't it make it harder to trade files with other file systems without stripping off the extra metadata (like Mac resource forks in the days of yore)? A lot of important file formats already include their own metadata (mp3, jpeg, most video formats I can think of, etc.)

I'll have conversion tools.. Anyway, in another post I said I wanted to try out new ways of doing things in my OS, and not write yet another DOS or POSIX clone just for the sake of compatibility.

Colonel Kernel · Post by **Colonel Kernel** » Tue Jul 24, 2007 12:15 pm

MessiahAndrw wrote:I'll have conversion tools.. Anyway, in another post I said I wanted to try out new ways of doing things in my OS, and not write yet another DOS or POSIX clone just for the sake of compatibility.

IMO, it's better to be better than to just be different.

AndrewAPrice · Post by **AndrewAPrice** » Tue Jul 24, 2007 8:59 pm

Colonel Kernel wrote:
MessiahAndrw wrote:I'll have conversion tools.. Anyway, in another post I said I wanted to try out new ways of doing things in my OS, and not write yet another DOS or POSIX clone just for the sake of compatibility.
IMO, it's better to be better than to just be different.

I've already planned how it would work. My file open function would be:

FILE File::Open(const char *path, const char *field, const char *mode);

Where the field would be the type, size, the data, etc. Depending on what you want to access within the file.

And I'll have a legacy fopen which defaults the field to "data". When a conventional file is copied the to file system, it just has all it's data placed in the "data" field until a program converts it to the new file-system's format. Live conversion tools will convert the format back when it is being copied to another file system.

AJ · Post by AJ » Wed Jul 25, 2007 1:19 am

Sounds interesting. I'll be interested to see how it works.

Cheers,
Adam

pabloreda · Post by **pabloreda** » Wed Aug 01, 2007 6:08 pm

Hi all,

There are a old OS called Pick, the filesystem are a DB, very practical and functional, I work with this in a 128kb computer with 2 terminals in 1990 in the army of my country (la colimba).

I think this is the beter aproach posible, the infinite levels of folders not help for speed

The manuals for this OS is in the web..