SQLite as storage

[OSwhatever]

The traditional way here I guess would be some VFAT implementation as the first file system support for your OS. However an interesting different approach would be to included some data base storage instead. My question here is if anyone here has successfully managed to implement SQLite on a bare bone with block device only rather than keeping the data base on a file.

If SQLite could efficiently work with block access that would be an interesting replacement for embedded operating systems.

[brain]

I would be interested to see if it could work. I assume you would build a non-hierarchical filesystem which locates files based on a set of search terms or tags in indexed tables?

I guess it depends on how much sqlite relies on a traditional filesystem rather than just a raw block device....

[OSwhatever]

I looked into the source code and documentation and it looks quite promising and you can adjust how large the blocks are supposed to be.

[brain]

please do document this filesystem if you create it, i can imagine it would be interesting to implement it...

[OSwhatever]

please do document this filesystem if you create it, i can imagine it would be interesting to implement it...

It's already created, not by me, and it's already well documented.

http://www.sqlite.org/about.html

It's an open source database written in C. However, if I port this to my OS, maybe wiki should be created to explain what steps are necessary.

[brain]

Sure sqlite is documented and already developed, I am aware of what it is however you will need to document two things:

1) changes needed to sqlite to make it work
2) required tables and database schema plus documentation of how errors are managed at the os level

also I wonder if it would be possible to implement the same fs over a more heavyweight sql server such as mysql, as a network filesystem... something interesting to consider...

[Solar]

@ OSwhatever:

A database is a database. A filesystem is something different.

There are several issues that must be solved when using SQLite as basis for a filesystem. Quoting from their FAQ:

When any process wants to write, it must lock the entire database file for the duration of its update.

There is a size limit for BLOBs; 1 billion by default, 2^31-1 bytes maximum, with the FAQ discouraging increasing the BLOB size limit beyond its default.

Then there is the question of how you want to map the file system structure - which would that be? - to database objects.

All that cannot be answered by a link to the SQLite homepage; that's why brain asked, I assume.

[OSwhatever]

@ OSwhatever:

A database is a database. A filesystem is something different.

There are several issues that must be solved when using SQLite as basis for a filesystem. Quoting from their FAQ:

When any process wants to write, it must lock the entire database file for the duration of its update.

There is a size limit for BLOBs; 1 billion by default, 2^31-1 bytes maximum, with the FAQ discouraging increasing the BLOB size limit beyond its default.

Then there is the question of how you want to map the file system structure - which would that be? - to database objects.

All that cannot be answered by a link to the SQLite homepage; that's why brain asked, I assume.

Agreed, a file system is a bit different but I think in case it would suit my needs. You could store data as it would be a file in a non-hierarchical way.

For small embedded purposes this could be a good way where you can store keys/values (think settings) together with a few files.

There are other questions also, how are "files" aligned in a database. In filesystem they are cluster aligned so that you swap in the data quickly using demand paging. Sqlite was never intended as a filesystem replacement but more a way to quickly get storage access for me.

BTW. Commercial OS developers like Microsoft seems to want a "database like filesystem" even if this was postponed for the future. I've not really understood what this filesystem/database merge really mean for them. BTRFS for example is more like database like when it comes how it arranges its internal structures, however still acts as an FS.

[CrypticalCode0]

SQLite does not have this as a intended use, However i do not see the harm in trying to figure out a way how to do this.

Personally i would use BFS (Be File System), and use a SQL library to extend what already is there.

[turdus]

You could store data as it would be a file in a non-hierarchical way.

Beg to differ. The problem is, many of you thinks in squares only. Try to think in abstractions, differentiate logic to representation. Filesystems can be interpreted as key/values easily, and hierarchy isn't a problem either.

Code: Select all

create table hierarchy {
  inode int,
  parentid int,
  filename varchar,
  unique key(inode),
  primary key (parentid,filename)
}
create table files {
  inode int,
  size int,
  created int,
  modified int,
  devmajor int,
  devminor int,
  content blob,
  primary key(inode)
}

Will do fine.

For small embedded purposes this could be a good way where you can store keys/values (think settings) together with a few files.

There are other questions also, how are "files" aligned in a database.

http://www.sqlite.org/fileformat2.html
"The main database file consists of one or more pages. The size of a page is a power of two between 512 and 65536 inclusive."

In filesystem they are cluster aligned so that you swap in the data quickly using demand paging. Sqlite was never intended as a filesystem replacement but more a way to quickly get storage access for me.

You are speaking as if it would applied to any filesystem, but as a matter of fact you're only speaking for fat. ISO9660 does not know any cluster for example.

BTW. Commercial OS developers like Microsoft seems to want a "database like filesystem" even if this was postponed for the future. I've not really understood what this filesystem/database merge really mean for them. BTRFS for example is more like database like when it comes how it arranges its internal structures, however still acts as an FS.

Let make this clear. When IBM invented the filesystem, there was only root directory. Files where referenced by their index in the catalog. Each catalog entry pointed to a portion of the storage (content). It looked like an MBR partition table. It was the ancestor of DB2, as well as CP/M's file "system". So it's not the future, it's the past.