How about design the VFS like DB interface?

Discussions on more advanced topics such as monolithic vs micro-kernels, transactional memory models, and paging vs segmentation should go here. Use this forum to expand and improve the wiki!
Post Reply
User avatar
nicesj
Member
Member
Posts: 27
Joined: Fri May 06, 2005 11:00 pm
Location: South Korea
Contact:

How about design the VFS like DB interface?

Post by nicesj »

Hello, developers.

I'm working on VFS design.
So I want to share the idea of it...

How about adpat the database interfaces instead of orignal VFS design?
I found some papers about DB oriented Filesystem.,.

of course, I will exposure the system related data(or functions) through the DB interfaces like "registry"(win) or "proc"(linux) filesystem and the name of syscall will be change as DB style interface...

Referenced DB is the SQLite,. it has a virtual machine to process the queries...
uhm, manipulating a query string is bad for the kernel, so the kernel will only supports VM APIs like the SQLite VM.

How about this idea?
or do you have any idea?
Could you share ur idea /w me?

Thanks..

ps. I'm not good at english, so if you feel bad by my post, tell me, and teach me.... :)
http://nicesj.com
With the software, What You Think Is What You Get.(WYTIWYG)
User avatar
prajwal
Member
Member
Posts: 154
Joined: Sat Oct 23, 2004 11:00 pm
Contact:

Re: How about design the VFS like DB interface?

Post by prajwal »

Is only the interface to fs will be changed to SQL like OR will the fs itself will be maintained as a database (relational ?) ?

If only interface, then I do not see the big advantage here. what would a query look like
select * from "/home/mydir"
select size from /home/mydir where filename in ('todo.txt', 'os.bin') ??

If the fs itself is maintained as a relational / object database then I see some value there
but what is actually a relational filesystem ? not able to comprehend this idea
object (relational) filesystem... hmmm.. would be good to implement and maintain

So, value add that I can think of from SQL like interface would be a generic langauge which everyone can understand...
complexity is the core of simplicity
exel0n
Posts: 1
Joined: Thu Dec 24, 2009 9:27 am

Re: How about design the VFS like DB interface?

Post by exel0n »

I've been playing around with a similiar idea for quite some time now, recently I've began writing a 'simulation' in Java as I haven't really bothered to look at C or ASM in depth.

Anyhow, my concept builds on completely removing the FS and replacing it with a hierarchical object DB in short. This means storing every piece of code (variables) as DB entries and then utilizing an execution model (not SQL) creating a 'kind of' powerful low level OO language. I'm trying to create something that doesn't require any kind of compiling by merging FS and code into one.

Anyway, still working on the specs and learning as I go, that's my idea, any thoughts?
cxzuk
Member
Member
Posts: 164
Joined: Mon Dec 21, 2009 6:03 pm

Re: How about design the VFS like DB interface?

Post by cxzuk »

a filesystem and database are actually very similar, they both share technology. I personally see it tho that a database is a filesystem, with added bits. Rather than the other way round.

a database is very specific in its role. I would not recommend replacing the filesystem with a database. Just extend the filesystem (or applications) to provide the features the database offers.

The main problem with replacing the filesystem with a database is this:
The things extra a database does is at an application level ("application" as in an understanding of the data, not just seeing it as a block of bits). Indexing, additional expression and functions (e.g. add(x,y)) and a language to query these indexes (SQL).

Beagle is an application that does it. It reads your data at an application level (it understands what a .doc file and a .jpg file is) and creates an index you can query. Just have something like beagle as a core service to your operating system. It would be much more flexable and easier to manage (coding and user wise).

MikeyB
User avatar
NickJohnson
Member
Member
Posts: 1249
Joined: Tue Mar 24, 2009 8:11 pm
Location: Sunnyvale, California

Re: How about design the VFS like DB interface?

Post by NickJohnson »

I've been planning to use a sort of database for a VFS for a while. The idea is that the VFS is its own process that only stores the information about which files belong to which drivers, but nothing about the permissions. User processes query it for file descriptors, and the rest of the metadata is handled by the drivers when read or write requests are made. At that level, the VFS really is just a database with an odd structure.
earlz
Member
Member
Posts: 1546
Joined: Thu Jul 07, 2005 11:00 pm
Contact:

Re: How about design the VFS like DB interface?

Post by earlz »

I have a few things I'd like to see in a filesystem(or VFS with emulation at the filesystem level). Constraints. A way to express that files depend on each other. Also, versioning.... And a way to express a file as parts. Basically, where (in a slightly unconventional DMBS) could be expressed as something like `select @file=foo from /usr; select versioninfo from @file;...` and such but also being able to express things like how an ELF file has section and headers.. how cool would it be if the OS knew how every file was built and how it all relates... course, this would add a pretty big layer of complexity as well..
User avatar
Jezze
Member
Member
Posts: 395
Joined: Thu Jul 26, 2007 1:53 am
Libera.chat IRC: jfu
Contact:

Re: How about design the VFS like DB interface?

Post by Jezze »

I just want to add that if you are gonna do this you might consider using an object-oriented database model instead of a typical table-row-type. It might also be easier to implement.
Fudge - Simplicity, clarity and speed.
http://github.com/Jezze/fudge/
robheus
Posts: 6
Joined: Sun Mar 14, 2010 1:14 pm

Re: How about design the VFS like DB interface?

Post by robheus »

robheus
Posts: 6
Joined: Sun Mar 14, 2010 1:14 pm

Re: How about design the VFS like DB interface?

Post by robheus »

To have your files stored in a database has several advantages. You can create multiple tags for your files, and store them with the file, for easy lookup. You don't have to know where your file is physically stored, just remember one or more tags for retrieving the file.

Also (althoug this is a seperate issue) for application development the utilization of a database instead of a flat file system would be great, since you can store the sources in a way that makes sense, For example every program variable you use, has an unique ID, and you store it only once. When chaning the name of that variable, there is only one location to change it, and all sources that use the variable are changed instantly.

This approach is used by Intentional Programming.
eddyb
Member
Member
Posts: 248
Joined: Fri Aug 01, 2008 7:52 am

Re: How about design the VFS like DB interface?

Post by eddyb »

robheus wrote:To have your files stored in a database has several advantages. You can create multiple tags for your files, and store them with the file, for easy lookup. You don't have to know where your file is physically stored, just remember one or more tags for retrieving the file.

Also (althoug this is a seperate issue) for application development the utilization of a database instead of a flat file system would be great, since you can store the sources in a way that makes sense, For example every program variable you use, has an unique ID, and you store it only once. When chaning the name of that variable, there is only one location to change it, and all sources that use the variable are changed instantly.

This approach is used by Intentional Programming.
While reading this, something came into my mind(I have no idea if there's such a thing already there): making a file system which, instead of plain structures, would use a database for storing *everything* about files. It'd provide great compatibility and it would allow different "flavours"/various features without any conflicts.
robheus
Posts: 6
Joined: Sun Mar 14, 2010 1:14 pm

Re: How about design the VFS like DB interface?

Post by robheus »

First, I think this idea is not entirely new. Second, there are rumours that Microsoft is planning to release this sort of thing in Windows 8, called WinFS.

In short, the whole idea here is to change the underlying paradigm of the operating system of being file based (most operating systems work on the idea of seeing everything as a file, this is the central idea of the UNIX operating system), to a relational (object/relation) database system.

It has the advantage that everything you store on disk is in relational format, and using smart indexes, this could speed up searching. Second, you don't need to bother where you store data, only how you will retrieve it. For example, instead of a normal file dialog in Save as, where you have to enter the exact location of your data, instead you will use tags or meta-tags for your data, and you don't bother where on disks it will be stored, that is up to the operating system.

The hierarchical directory structure and file system, still would be implemented in some form (mostly for compatibility issues), but then you could optionally have alternative parallel hierarchical directory tree/file structure.

In essence though this is a very far reaching shift in operating system design, affecting all applications that can run on this operating system (but, for compatibility, one might think of a special layer that represents the data as a tradiational hierarchical file system, so that existing application might be able to run on this OS).

This new design has various implications, as not only user data but also data used by the kernel/OS will be internally represented in the forms of relations (in fact: tables).

You could for instance implement object dependencie in this relational schema.
For example, suppose you made a document and included a picture. Now when you update the picture you included in your document, the document is then instantly updated too.

Another applicability is that you can store versioned information in the database. Every object in the database has a version number, and you can revert the system to a previous point in time. Of course you need to be able to limit the number of versions, or include features like automatic backup of old revisions (backup the data, but retain the meta data, so that searches will still be able to find older revisions and fetch it from backup).

For application development, this could open new frontiers. No longer source modules need to be stored as flat files, but can be stored in their gramatic/syntactic form. Of course you need specialised (grammar sensitive) editors to be able to do that.

In fact this approach is used in the Intentional Programming project. (see for example: Wikipedia: Intentional Programming and Intentional Software and Jetbrains/Metaprogramming: Intentional Programming).

Manipulating software would become a lot easier then. You can not introduce syntactic bugs, as the only valid code you could enter in the editor is syntactic correct, so only semantic checks need to be taken care of (and partly checked for automatically). In fact, we might in many cases not even to do coding ourselves, but use specialised code generators and/or using meta-programming features and/or a domain work bench to generate code.
Changing a function or variable name in one module will automaticall change the function or variable name in all the modules that reference the same function or variable. etc.

Build tools like the compiler, linker, make and other build tools would become (almost) obsolete, as the dependency of objects is included as meta data when creating those objects, and also what processing objects uses this object as a source for creating a target, etc.

The simple model is: everything is an object. A target object is created by a single object (process) with optional process-parameters, from one or more source objects (both processing and source objects are themselves target objects, and so on recursively), and the final source objects consists of user-sessions entering that data.

Implementing this OS on different platforms comes automatically with the feature that you have source compatibility between this OS on different platforms, you simply need to transfer the sources (you just point to the final target objects you want to transfer, and the system will figure out recursively from which sources it was built) to the other platform and there recompile and built it, and you have your target system.

In fact, such an Operating system has many advantages above traditional file system architectures, but why has nobody yet implemented this already?

Some form of implementation of this idea (at least partly) is already existing, and is called tagging filesystem. But this only deals with user data (adding meta data to files, but not based on a real relational file- or database system; see for example: oyepa and TagFS), I guess not the internal data of the underlying operating system.

I don't know if this concept would make the concept of a filesystem completely obsolete (I guess not, if only for the sake of interoperability). As to the question, how to implement a database without an underlying filesystem. This can be done, you just design the database to take a partition or extended partition as its data location. Perhaps only for the boot up procedure you need a very simple filesystem to be able to boot up and start the rdbms, and from then on the OS can take advantage of the rdbms.
Last edited by robheus on Tue Apr 20, 2010 6:55 pm, edited 4 times in total.
robheus
Posts: 6
Joined: Sun Mar 14, 2010 1:14 pm

Re: How about design the VFS like DB interface?

Post by robheus »

eddyb wrote: While reading this, something came into my mind(I have no idea if there's such a thing already there): making a file system which, instead of plain structures, would use a database for storing *everything* about files. It'd provide great compatibility and it would allow different "flavours"/various features without any conflicts.
Not just everything *about* files, but also the file itself! For instance an internal representation of a file could be a complex query from many different tables.
Adding time-dependency (historic or versioned info) the file could contain many other data (for instance what the file looked like 3 weeks ago) and object-dependency (what are the objects depending on this file, or on which this file depends) and tags (what tags or meta-tags were added when creating the file), etc.
User avatar
Neolander
Member
Member
Posts: 228
Joined: Tue Mar 23, 2010 3:01 pm
Location: Uppsala, Sweden
Contact:

Re: How about design the VFS like DB interface?

Post by Neolander »

Just an opinion about hierarchy : keep some kind of user-accessible hierarchy, even if it doesn't exist at the core of the system. All current search-based ways to locate some data are only working properly when the user exactly knows what file he is looking for. To the contrary, a good hierarchical file system can help someone finding a file that he doesn't know about yet.
Post Reply