How about design the VFS like DB interface?
How about design the VFS like DB interface?
Hello, developers.
I'm working on VFS design.
So I want to share the idea of it...
How about adpat the database interfaces instead of orignal VFS design?
I found some papers about DB oriented Filesystem.,.
of course, I will exposure the system related data(or functions) through the DB interfaces like "registry"(win) or "proc"(linux) filesystem and the name of syscall will be change as DB style interface...
Referenced DB is the SQLite,. it has a virtual machine to process the queries...
uhm, manipulating a query string is bad for the kernel, so the kernel will only supports VM APIs like the SQLite VM.
How about this idea?
or do you have any idea?
Could you share ur idea /w me?
Thanks..
ps. I'm not good at english, so if you feel bad by my post, tell me, and teach me....
I'm working on VFS design.
So I want to share the idea of it...
How about adpat the database interfaces instead of orignal VFS design?
I found some papers about DB oriented Filesystem.,.
of course, I will exposure the system related data(or functions) through the DB interfaces like "registry"(win) or "proc"(linux) filesystem and the name of syscall will be change as DB style interface...
Referenced DB is the SQLite,. it has a virtual machine to process the queries...
uhm, manipulating a query string is bad for the kernel, so the kernel will only supports VM APIs like the SQLite VM.
How about this idea?
or do you have any idea?
Could you share ur idea /w me?
Thanks..
ps. I'm not good at english, so if you feel bad by my post, tell me, and teach me....
http://nicesj.com
With the software, What You Think Is What You Get.(WYTIWYG)
With the software, What You Think Is What You Get.(WYTIWYG)
Re: How about design the VFS like DB interface?
Is only the interface to fs will be changed to SQL like OR will the fs itself will be maintained as a database (relational ?) ?
If only interface, then I do not see the big advantage here. what would a query look like
select * from "/home/mydir"
select size from /home/mydir where filename in ('todo.txt', 'os.bin') ??
If the fs itself is maintained as a relational / object database then I see some value there
but what is actually a relational filesystem ? not able to comprehend this idea
object (relational) filesystem... hmmm.. would be good to implement and maintain
So, value add that I can think of from SQL like interface would be a generic langauge which everyone can understand...
If only interface, then I do not see the big advantage here. what would a query look like
select * from "/home/mydir"
select size from /home/mydir where filename in ('todo.txt', 'os.bin') ??
If the fs itself is maintained as a relational / object database then I see some value there
but what is actually a relational filesystem ? not able to comprehend this idea
object (relational) filesystem... hmmm.. would be good to implement and maintain
So, value add that I can think of from SQL like interface would be a generic langauge which everyone can understand...
complexity is the core of simplicity
Re: How about design the VFS like DB interface?
I've been playing around with a similiar idea for quite some time now, recently I've began writing a 'simulation' in Java as I haven't really bothered to look at C or ASM in depth.
Anyhow, my concept builds on completely removing the FS and replacing it with a hierarchical object DB in short. This means storing every piece of code (variables) as DB entries and then utilizing an execution model (not SQL) creating a 'kind of' powerful low level OO language. I'm trying to create something that doesn't require any kind of compiling by merging FS and code into one.
Anyway, still working on the specs and learning as I go, that's my idea, any thoughts?
Anyhow, my concept builds on completely removing the FS and replacing it with a hierarchical object DB in short. This means storing every piece of code (variables) as DB entries and then utilizing an execution model (not SQL) creating a 'kind of' powerful low level OO language. I'm trying to create something that doesn't require any kind of compiling by merging FS and code into one.
Anyway, still working on the specs and learning as I go, that's my idea, any thoughts?
Re: How about design the VFS like DB interface?
a filesystem and database are actually very similar, they both share technology. I personally see it tho that a database is a filesystem, with added bits. Rather than the other way round.
a database is very specific in its role. I would not recommend replacing the filesystem with a database. Just extend the filesystem (or applications) to provide the features the database offers.
The main problem with replacing the filesystem with a database is this:
The things extra a database does is at an application level ("application" as in an understanding of the data, not just seeing it as a block of bits). Indexing, additional expression and functions (e.g. add(x,y)) and a language to query these indexes (SQL).
Beagle is an application that does it. It reads your data at an application level (it understands what a .doc file and a .jpg file is) and creates an index you can query. Just have something like beagle as a core service to your operating system. It would be much more flexable and easier to manage (coding and user wise).
MikeyB
a database is very specific in its role. I would not recommend replacing the filesystem with a database. Just extend the filesystem (or applications) to provide the features the database offers.
The main problem with replacing the filesystem with a database is this:
The things extra a database does is at an application level ("application" as in an understanding of the data, not just seeing it as a block of bits). Indexing, additional expression and functions (e.g. add(x,y)) and a language to query these indexes (SQL).
Beagle is an application that does it. It reads your data at an application level (it understands what a .doc file and a .jpg file is) and creates an index you can query. Just have something like beagle as a core service to your operating system. It would be much more flexable and easier to manage (coding and user wise).
MikeyB
- NickJohnson
- Member
- Posts: 1249
- Joined: Tue Mar 24, 2009 8:11 pm
- Location: Sunnyvale, California
Re: How about design the VFS like DB interface?
I've been planning to use a sort of database for a VFS for a while. The idea is that the VFS is its own process that only stores the information about which files belong to which drivers, but nothing about the permissions. User processes query it for file descriptors, and the rest of the metadata is handled by the drivers when read or write requests are made. At that level, the VFS really is just a database with an odd structure.
Re: How about design the VFS like DB interface?
I have a few things I'd like to see in a filesystem(or VFS with emulation at the filesystem level). Constraints. A way to express that files depend on each other. Also, versioning.... And a way to express a file as parts. Basically, where (in a slightly unconventional DMBS) could be expressed as something like `select @file=foo from /usr; select versioninfo from @file;...` and such but also being able to express things like how an ELF file has section and headers.. how cool would it be if the OS knew how every file was built and how it all relates... course, this would add a pretty big layer of complexity as well..
Re: How about design the VFS like DB interface?
I just want to add that if you are gonna do this you might consider using an object-oriented database model instead of a typical table-row-type. It might also be easier to implement.
Fudge - Simplicity, clarity and speed.
http://github.com/Jezze/fudge/
http://github.com/Jezze/fudge/
Re: How about design the VFS like DB interface?
I think this is already under development:
There is the project Btrfs:
http://btrfs.wiki.kernel.org/index.php/Main_Page
http://en.wikipedia.org/wiki/Btrfs
http://oss.oracle.com/projects/btrfs/
And there are:
http://download-uk.oracle.com/docs/cd/E ... lob_fs.htm
http://tech.inhelsinki.nl/dbfs/dbfs-screen.pdf
http://kevinclosson.wordpress.com/2010/ ... s-part-ii/
I heard there are plans for incorporating this idea into a new version of Windows.
There is the project Btrfs:
http://btrfs.wiki.kernel.org/index.php/Main_Page
http://en.wikipedia.org/wiki/Btrfs
http://oss.oracle.com/projects/btrfs/
And there are:
http://download-uk.oracle.com/docs/cd/E ... lob_fs.htm
http://tech.inhelsinki.nl/dbfs/dbfs-screen.pdf
http://kevinclosson.wordpress.com/2010/ ... s-part-ii/
I heard there are plans for incorporating this idea into a new version of Windows.
Re: How about design the VFS like DB interface?
To have your files stored in a database has several advantages. You can create multiple tags for your files, and store them with the file, for easy lookup. You don't have to know where your file is physically stored, just remember one or more tags for retrieving the file.
Also (althoug this is a seperate issue) for application development the utilization of a database instead of a flat file system would be great, since you can store the sources in a way that makes sense, For example every program variable you use, has an unique ID, and you store it only once. When chaning the name of that variable, there is only one location to change it, and all sources that use the variable are changed instantly.
This approach is used by Intentional Programming.
Also (althoug this is a seperate issue) for application development the utilization of a database instead of a flat file system would be great, since you can store the sources in a way that makes sense, For example every program variable you use, has an unique ID, and you store it only once. When chaning the name of that variable, there is only one location to change it, and all sources that use the variable are changed instantly.
This approach is used by Intentional Programming.
Re: How about design the VFS like DB interface?
While reading this, something came into my mind(I have no idea if there's such a thing already there): making a file system which, instead of plain structures, would use a database for storing *everything* about files. It'd provide great compatibility and it would allow different "flavours"/various features without any conflicts.robheus wrote:To have your files stored in a database has several advantages. You can create multiple tags for your files, and store them with the file, for easy lookup. You don't have to know where your file is physically stored, just remember one or more tags for retrieving the file.
Also (althoug this is a seperate issue) for application development the utilization of a database instead of a flat file system would be great, since you can store the sources in a way that makes sense, For example every program variable you use, has an unique ID, and you store it only once. When chaning the name of that variable, there is only one location to change it, and all sources that use the variable are changed instantly.
This approach is used by Intentional Programming.
Re: How about design the VFS like DB interface?
First, I think this idea is not entirely new. Second, there are rumours that Microsoft is planning to release this sort of thing in Windows 8, called WinFS.
In short, the whole idea here is to change the underlying paradigm of the operating system of being file based (most operating systems work on the idea of seeing everything as a file, this is the central idea of the UNIX operating system), to a relational (object/relation) database system.
It has the advantage that everything you store on disk is in relational format, and using smart indexes, this could speed up searching. Second, you don't need to bother where you store data, only how you will retrieve it. For example, instead of a normal file dialog in Save as, where you have to enter the exact location of your data, instead you will use tags or meta-tags for your data, and you don't bother where on disks it will be stored, that is up to the operating system.
The hierarchical directory structure and file system, still would be implemented in some form (mostly for compatibility issues), but then you could optionally have alternative parallel hierarchical directory tree/file structure.
In essence though this is a very far reaching shift in operating system design, affecting all applications that can run on this operating system (but, for compatibility, one might think of a special layer that represents the data as a tradiational hierarchical file system, so that existing application might be able to run on this OS).
This new design has various implications, as not only user data but also data used by the kernel/OS will be internally represented in the forms of relations (in fact: tables).
You could for instance implement object dependencie in this relational schema.
For example, suppose you made a document and included a picture. Now when you update the picture you included in your document, the document is then instantly updated too.
Another applicability is that you can store versioned information in the database. Every object in the database has a version number, and you can revert the system to a previous point in time. Of course you need to be able to limit the number of versions, or include features like automatic backup of old revisions (backup the data, but retain the meta data, so that searches will still be able to find older revisions and fetch it from backup).
For application development, this could open new frontiers. No longer source modules need to be stored as flat files, but can be stored in their gramatic/syntactic form. Of course you need specialised (grammar sensitive) editors to be able to do that.
In fact this approach is used in the Intentional Programming project. (see for example: Wikipedia: Intentional Programming and Intentional Software and Jetbrains/Metaprogramming: Intentional Programming).
Manipulating software would become a lot easier then. You can not introduce syntactic bugs, as the only valid code you could enter in the editor is syntactic correct, so only semantic checks need to be taken care of (and partly checked for automatically). In fact, we might in many cases not even to do coding ourselves, but use specialised code generators and/or using meta-programming features and/or a domain work bench to generate code.
Changing a function or variable name in one module will automaticall change the function or variable name in all the modules that reference the same function or variable. etc.
Build tools like the compiler, linker, make and other build tools would become (almost) obsolete, as the dependency of objects is included as meta data when creating those objects, and also what processing objects uses this object as a source for creating a target, etc.
The simple model is: everything is an object. A target object is created by a single object (process) with optional process-parameters, from one or more source objects (both processing and source objects are themselves target objects, and so on recursively), and the final source objects consists of user-sessions entering that data.
Implementing this OS on different platforms comes automatically with the feature that you have source compatibility between this OS on different platforms, you simply need to transfer the sources (you just point to the final target objects you want to transfer, and the system will figure out recursively from which sources it was built) to the other platform and there recompile and built it, and you have your target system.
In fact, such an Operating system has many advantages above traditional file system architectures, but why has nobody yet implemented this already?
Some form of implementation of this idea (at least partly) is already existing, and is called tagging filesystem. But this only deals with user data (adding meta data to files, but not based on a real relational file- or database system; see for example: oyepa and TagFS), I guess not the internal data of the underlying operating system.
I don't know if this concept would make the concept of a filesystem completely obsolete (I guess not, if only for the sake of interoperability). As to the question, how to implement a database without an underlying filesystem. This can be done, you just design the database to take a partition or extended partition as its data location. Perhaps only for the boot up procedure you need a very simple filesystem to be able to boot up and start the rdbms, and from then on the OS can take advantage of the rdbms.
In short, the whole idea here is to change the underlying paradigm of the operating system of being file based (most operating systems work on the idea of seeing everything as a file, this is the central idea of the UNIX operating system), to a relational (object/relation) database system.
It has the advantage that everything you store on disk is in relational format, and using smart indexes, this could speed up searching. Second, you don't need to bother where you store data, only how you will retrieve it. For example, instead of a normal file dialog in Save as, where you have to enter the exact location of your data, instead you will use tags or meta-tags for your data, and you don't bother where on disks it will be stored, that is up to the operating system.
The hierarchical directory structure and file system, still would be implemented in some form (mostly for compatibility issues), but then you could optionally have alternative parallel hierarchical directory tree/file structure.
In essence though this is a very far reaching shift in operating system design, affecting all applications that can run on this operating system (but, for compatibility, one might think of a special layer that represents the data as a tradiational hierarchical file system, so that existing application might be able to run on this OS).
This new design has various implications, as not only user data but also data used by the kernel/OS will be internally represented in the forms of relations (in fact: tables).
You could for instance implement object dependencie in this relational schema.
For example, suppose you made a document and included a picture. Now when you update the picture you included in your document, the document is then instantly updated too.
Another applicability is that you can store versioned information in the database. Every object in the database has a version number, and you can revert the system to a previous point in time. Of course you need to be able to limit the number of versions, or include features like automatic backup of old revisions (backup the data, but retain the meta data, so that searches will still be able to find older revisions and fetch it from backup).
For application development, this could open new frontiers. No longer source modules need to be stored as flat files, but can be stored in their gramatic/syntactic form. Of course you need specialised (grammar sensitive) editors to be able to do that.
In fact this approach is used in the Intentional Programming project. (see for example: Wikipedia: Intentional Programming and Intentional Software and Jetbrains/Metaprogramming: Intentional Programming).
Manipulating software would become a lot easier then. You can not introduce syntactic bugs, as the only valid code you could enter in the editor is syntactic correct, so only semantic checks need to be taken care of (and partly checked for automatically). In fact, we might in many cases not even to do coding ourselves, but use specialised code generators and/or using meta-programming features and/or a domain work bench to generate code.
Changing a function or variable name in one module will automaticall change the function or variable name in all the modules that reference the same function or variable. etc.
Build tools like the compiler, linker, make and other build tools would become (almost) obsolete, as the dependency of objects is included as meta data when creating those objects, and also what processing objects uses this object as a source for creating a target, etc.
The simple model is: everything is an object. A target object is created by a single object (process) with optional process-parameters, from one or more source objects (both processing and source objects are themselves target objects, and so on recursively), and the final source objects consists of user-sessions entering that data.
Implementing this OS on different platforms comes automatically with the feature that you have source compatibility between this OS on different platforms, you simply need to transfer the sources (you just point to the final target objects you want to transfer, and the system will figure out recursively from which sources it was built) to the other platform and there recompile and built it, and you have your target system.
In fact, such an Operating system has many advantages above traditional file system architectures, but why has nobody yet implemented this already?
Some form of implementation of this idea (at least partly) is already existing, and is called tagging filesystem. But this only deals with user data (adding meta data to files, but not based on a real relational file- or database system; see for example: oyepa and TagFS), I guess not the internal data of the underlying operating system.
I don't know if this concept would make the concept of a filesystem completely obsolete (I guess not, if only for the sake of interoperability). As to the question, how to implement a database without an underlying filesystem. This can be done, you just design the database to take a partition or extended partition as its data location. Perhaps only for the boot up procedure you need a very simple filesystem to be able to boot up and start the rdbms, and from then on the OS can take advantage of the rdbms.
Last edited by robheus on Tue Apr 20, 2010 6:55 pm, edited 4 times in total.
Re: How about design the VFS like DB interface?
Not just everything *about* files, but also the file itself! For instance an internal representation of a file could be a complex query from many different tables.eddyb wrote: While reading this, something came into my mind(I have no idea if there's such a thing already there): making a file system which, instead of plain structures, would use a database for storing *everything* about files. It'd provide great compatibility and it would allow different "flavours"/various features without any conflicts.
Adding time-dependency (historic or versioned info) the file could contain many other data (for instance what the file looked like 3 weeks ago) and object-dependency (what are the objects depending on this file, or on which this file depends) and tags (what tags or meta-tags were added when creating the file), etc.
Re: How about design the VFS like DB interface?
Just an opinion about hierarchy : keep some kind of user-accessible hierarchy, even if it doesn't exist at the core of the system. All current search-based ways to locate some data are only working properly when the user exactly knows what file he is looking for. To the contrary, a good hierarchical file system can help someone finding a file that he doesn't know about yet.