OSDev.org

Hi All,

I have a question about designing the VFS layer.

What advantages are there for having a file descriptor represent both files AND directories?

Would having the VFS distinguish between directories and files (or file streams, to be more extensible/unified) as separate types/interfaces have any benefits? And what disadvantages would this entail?

I'm just thinking about operations that you perform on directories versus file streams.

For example: seeking on a directory seems pointless, but then other kinds of file streams are non-seekable too.

On the other hand: you can't open a file, and try and read it as a directory. However: if you were tricky, you could treat a file as a directory when that file is say a file archive, or a disk image.

So, I'm quite undecided. It seems everybody treats files & directories as a single base type (the file descriptor). Therefore, all your guys thoughts are very welcome =)

Jonathan

Hi,

purevoid wrote:What advantages are there for having a file descriptor represent both files AND directories?

The term "file descriptor" isn't entirely appropriate - a better name for it could be "item descriptor".

The VFS maintains a database of items and keeps track of the relationships between them.

An "item descriptor" consists of the item name, its type, some attributes and the item data.

If "type = file" then the name is the file name and the data is the file data.

If "type = directory" then the name is the directory name and the data is a list of "item descriptors".

If "type = device" then the name is the device name and the data is a reference to the device driver (for e.g. a message port number).

For IPC, most OS's have named pipes (or named sockets, or named message ports or whatever). Here "type = IPC" and the data is a reference to the receiver.

For all of these things, the attributes usually contain permission data which is used to determine who can read or modify which item.

All of the VFS code to maintain the database of items is the same regardless of the type of item/s. If you had seperate databases, then half the time you'd need to perform the same operation on all databases at the same time. For example, when a new item is created you'd need to check all databases to see if the name is already in use, or if you're trying to find out what type the item "/foo/bar" is.

Then there's "extensions".

Modern VFS systems support "notifications", where the VFS code will let your code know if a requested "item descriptor" or its data has changed. An example would be a GUI window that shows the contents of a directory - if some other software adds a new file to the directory then the VFS code tells the GUI that the directory was changed, and the GUI code updates the displayed window. Another example would be a text editor that tells the user "the file was changed on disk - do you want to reload?". It could also be used for devices - if the user unplugs some USB flash memory then the VFS code might tell the file system driver that the device is gone. As you can see the notification code would work the same regardless of the item type.

Then there's "converters". If software asks to open a text file called "/foo/bar" and it happens to be a directory, wouldn't it be nice if the VFS code started an "item type converter" that converted the directory contents into a text file? How about trying to open "/foor/bar.html" as a PDF file - a converter could convert the html data into PDF format (and back again if the file is saved). Then there's things like compressed archives, where software could ask for the directory contents of "foo/bar.zip" and the converter would extract directory information.

Allowing the VFS code to support "converter plugins" is a very powerful idea. For example, you can write one application that only ever handles one file format, and other people can write a pile of converters to allow that application to work with many different file formats. If none of your software can handle GIF files, just write some converters and all of your software will handle them.

Once you've got this, you could even have a set of "standard file formats", where all software only ever needs to support data that is in a standard format. For all other formats you'd use file converters. This can make writing software much easier than you'd think. Consider "cut & paste" and the clipboard - if a user selects an area of a JPG file and wants to paste it into a BMP editor, what format should the data be in when it's transfered?

Unfortunately, most software uses something like "fopen(fileName, mode)", where to take advantage of the file converters you'd need something more like "fopen(fileName, mode, expectedFileType)".


Cheers,

Brendan

Okay, so to be really powerful in terms of usefulness, we'd want to turn the VFS into almost a database in it's own right, that's extensible.

Then, all we need is an abstract set of operations that can be performed over any type. And this is how a file descriptor works on unix such that it can be a socket or a file, or whatever, right?

My ideal system would work on mime-types, and yes, I've had the idea for converters (much like BeOS for image types) already planned.

So, I'd just need the basic set of operators (open, close, read, write, seek), and whatever else to make my VFS tick over the way I want.

Just one last question then: how does a VFS deal with mount points, and know whether a directory (be it at the root of the FS, or in a random dir, such as a mounted image) is actually a directory, or a new file system? Does it need to know?

I really don't understand the workings of mount points .. that's probably the biggest obstacle I have in figuring out VFS.

Hi,

purevoid wrote: Okay, so to be really powerful in terms of usefulness, we'd want to turn the VFS into almost a database in it's own right, that's extensible.

Then, all we need is an abstract set of operations that can be performed over any type. And this is how a file descriptor works on unix such that it can be a socket or a file, or whatever, right?

For Linux, the VFS works on a database of "inodes", similar to what I described. See:
http://www.cse.unsw.edu.au/~neilb/oss/l ... y/vfs.html

purevoid wrote:So, I'd just need the basic set of operators (open, close, read, write, seek), and whatever else to make my VFS tick over the way I want.

I'd consider seperating the VFS into 2 seperate parts. The lower level part would be pure database management code, with functions to find an item, create an item, move an item, etc. The higher level part would contain the file I/O functions (open, close, read, write, mount, unmount, etc).

The "seek" function can be done in 2 different ways - I prefer to have functions like "read(fileDescriptor, fileOffset, size)" and let the caller (or the library the caller is using) keep track of it's own file position (with no need for a "seek()" function in the VFS itself). The alternative would be "read(fileDescriptor, size)" where the VFS code keeps track of the file position and a "seek()" function is necessary.

IMHO this makes it easier to support asynchronious file I/O. For example, an application could do something like:
For RAID arrays this can mean that different parts of the file are loaded from disk in parallel. For example, for 2 disk RAID 1 (mirroring) one hard drive might do 2 of the reads while the other hard drive does the other 2, and all reads could complete in about half the time.

purevoid wrote:Just one last question then: how does a VFS deal with mount points, and know whether a directory (be it at the root of the FS, or in a random dir, such as a mounted image) is actually a directory, or a new file system? Does it need to know?

I really don't understand the workings of mount points .. that's probably the biggest obstacle I have in figuring out VFS.

As part of the "item attributes" I'd have a "mount point" flag and a reference to the file system driver. If "item type = directory" and if the "mount point" flag is set, then the file system reference is valid, but the item data still contains references to other items, just like a normal directory.

When a file system is mounted, the VFS code finds the item corresponding to the directory, removes any previous item data, sets the "mount point" flag and sets the file system driver reference.

To complicate things, the "item attributes" should also have a flag for "up-to-date". If this flag is set, then the "item data" is present and correct (in the VFS's database). If the flag is clear then the VFS code would need to ask the file system driver for the "item data". This allows the VFS to contain part of the directory structure rather than all of it, and also allows the VFS to cache file data (so file system drivers don't need to).

As a (very rough) illustration, it'd be something like:
[EDIT: I got RAID 0 (striping) and RAID 1 (mirroring) mixed up the first time!]


Cheers,

Brendan

Let's concentrate on the VFS itself.

Does anyone have a practical guide to writing a VFS layer?

And one interesting question: how does the VFS instruct an FS module to create a directory?

Lastly, (I think I asked, but didn't get a reply...) is there any value in making directory & file handling completely separate in the VFS (own types, etc.)?

I've been looking at some sources, and Practical File System Design PDF, but I'm as confused as ever.

I see the base of the VFS is built upon i-nodes; so functionality of the system would be dependent on the VFS structure of this.

What I don't understand is that given an i-node, you can perform all needed operations on any given underlying file system.

Can someone explain the magic behind i-nodes?

Maybe I can get away with a high-level interface to build on top of for the moment to get things I need to compile; just completely stuck on how it all works.

Well, there's not that much 'magic' behind them. Simply that if you declare they have sufficient fields, they can encompass all the operations a (set of) traditionnal file systems support.

The inode is just a special block on the disk that contains all the information you might wish to have about one file and from which you can retrieve all the data blocks of the file you might need. Plus, the inode number immediately translates into a block to be read. With a custom function (that is, depending on the FS) that 'opens' the actual structure on disk, you can perform any translation required and fill the 'virtual-node' structure with what was present on disk...

Well, I don't think that a lot more then files can be represented well with files ...

Legend wrote: Well, I don't think that a lot more then files can be represented well with files ...

well, i don't think either. That's one of the reasons why i'm thinking about FLUIDs instead of i-nodes and file classes' with customizable and combinables interfaces rather than usual files/directories