Need advices for designing a simple file system

[pauldinhqd]

hi guys,

i'm writing the kernel, and a part of it is the
services for accessing data stored on disk (so called system data service).
however i don't have much idea of how to design a simple file system
which is easy to implement but comes with standard (or at least basic) features

let me know your suggestions if possible, tks in advance

[piranha]

http://wiki.osdev.org/File_Systems would be a good place to start. As a recommendation, you're going to want to have a good memory manager with a malloc like ability. It'll make things much easier to implement.

-JL

[Combuster]

Need advices for designing a simple file system

Do not design your own - it's a typical beginner mistake. There are enough simple filesystems out there and making your own at that stage only reinvents the wheel or more often than not yields something more broken. Basically you'll end up throwing it away anyway.

Only when you know why the existing filesystems don't work for you, will you have a good starting point for making something new.

[turdus]

part of it is the
services for accessing data stored on disk

I suppose you are at early stage, and your primary goal is to read files from disk. The good news is, you don't need an FS driver for that, many kernels (most notably Linux) use an archive file for that. Most widely used is cpio with ascii header format. It's just a mound of header+data blocks, with header aligned, what makes it extremly easy to get a file from. http://www.mkssoftware.com/docs/man4/cpio.4.asp
You can load that by grub as a module, or write your own loader. If the latter, you have to
- locate the boot fs on disk
- load the fs into ram (can be done by BIOS calls, at boot you don't have any disk driver yet)
- implement a readfile function, that parses the ram image for a file and returns a pointer to it's data.

For compatibility, you may consider to load the fs image from a file ona FAT disk. It's not so hard, and there're plenty of examples on the net. Or use the unix way, one small boot partition (be careful when assigning a partition type code for it).

Desinging a good FS is a really hard task, I suggest to study different FS's on disk format and write drivers for them before you start to get a clue.

[iansjack]

If you really want to design your own file system (which, along with others, I would say is a very advanced topic) you might want to look at this link: http://www.readwriteweb.com/hack/2011/0 ... file-s.php . In particular, follow the link to Practical File System Design With the Be File System.

[bluemoon]

I agree not to design another file system.
But use that effort to design a good file system driver.

For example, my boot loader originally do not cache FAT and take a minute to start the boot process on real machine with USB stick, but introducing cache there I reduced about half disk access(and thus half time) when loading initrd.

This is to say, even using existing file system design, there is still many architectural design can be done.

[OSwhatever]

Need advices for designing a simple file system

Do not design your own - it's a typical beginner mistake. There are enough simple filesystems out there and making your own at that stage only reinvents the wheel or more often than not yields something more broken. Basically you'll end up throwing it away anyway.

Only when you know why the existing filesystems don't work for you, will you have a good starting point for making something new.

We all reinvent the wheel here. Designing an own FS is a lot of work but if a person is truly interested in file systems and wants to design an FS then the person should of course do it. On the other hand if you quickly want file system support and get on with the rest of the OS then taking an existing FS would probably be a better choice.

[Solar]

We all reinvent the wheel here. Designing an own FS is a lot of work but if a person is truly interested in file systems and wants to design an FS then the person should of course do it.

Same as with OS projects overall: Only if you are interested in the experience itself, or know enough about the subject to actually surpass what is already available. If you are merely interested in having a file system, use / implement one of the readily available ones (SFS for simple, FAT/ext2 for data interchange, ext4 for powerful would be my choices).

[pauldinhqd]

We all reinvent the wheel here. Designing an own FS is a lot of work but if a person is truly interested in file systems and wants to design an FS then the person should of course do it.

Same as with OS projects overall: Only if you are interested in the experience itself, or know enough about the subject to actually surpass what is already available. If you are merely interested in having a file system, use / implement one of the readily available ones (SFS for simple, FAT/ext2 for data interchange, ext4 for powerful would be my choices).

Combuster, bluemoon, OSwhatever, Solar:
yeah, i don't have any plan to make a commercial product, just a real hobby that i do when i'm off work;
so i possibly should design something special coz the timing doesn't matter

iansjack, turdus, piranha, your links bring great resources

[AndrewAPrice]

If it's a read only file system, then you can have an extremely fast and efficient file system by simply having a table at the start of the disk containing the file name, where it starts, next file, where it starts, etc, then all of the files are stored sequentially on disk.

[gerryg400]

If it's a read only file system, then you can have an extremely fast and efficient file system by simply having a table at the start of the disk containing the file name, where it starts, next file, where it starts, etc, then all of the files are stored sequentially on disk.

tarfs ? That's the first fs I implemented.

[bluemoon]

my opinion is, if you goes for initrd (which contain basic set of drivers), there are 2 choices:

1. implement it as ramdisk and a file system. The kernel can mount initrd and access files there over VFS layer.
If you want to use tar or simplefs I recommend to pad the files to sector-aligned, which is more convenient to work with storage devices.

2. handle initrd with a data reader. you "extract" files from initrd archive.

[Brendan]

Hi,

let me know your suggestions if possible, tks in advance

If you want to develop a new high performance file system; I'd start by splitting it in half - a lower level part and a higher level part.

The lower level part would mainly manage "linked lists of extents" - allocating new extents, freeing them, resizing them, reading them and writing them. Naturally it'd also manage free space, handle things like bad sectors, etc; and it'd probably be the best place to do things like "sparse lists of extents" (used for sparse files support), encryption, compression, etc.

The higher level part relies on the lower level part. A file's data is just a linked lists of extents. Directories are linked lists of extents that contain directory entries instead of file data. More advanced things like snapshots and forks are just more linked lists of extents.

If you split things like this you should be able to test the lower level part on its own; and could possibly have several "lower level parts" and several "higher level parts" and combine them in various ways. For example, perhaps one "lower level part" that tries harder to avoid moving extents for SSD and USB flash, and another "lower level part" that tries harder to avoid fragmented extents for hard drives; and perhaps one "higher level part" designed to be efficient for lots of small files and one designed to be efficient for lots of large files (plus maybe a stable version of each and a beta/development version of each).


Cheers,

Brendan

[rdos]

The best idea is to start with the VFS, and implement FAT or something similar that is in extended use. Then you can access files on your ordinary harddrive, which is a definite bonus. You also need an IDE, SATA/AHCI or USB driver to read something from real hardware.

After that is functional, I'd start thinking about a possible custom FS. I've made several attempts at creating custom FSes, but none of them were finished for various reasons. Mostly I wish I had a power-failure safe FS, which has wear-leveling, but that design never made it into a functional FS. The reasons for this failure was several, one of the most important being that you need some type of extensive FS analysis program that can correct errors and display raw contents. If you go by FAT or some Linux variant, you can just start the native applications to correct the FS. If you do something home-brew, you are on your own. IOW, it is not just the FS driver that needs to be done, but also a fairly complex error-detection/correction program.

[pauldinhqd]

Hi,

let me know your suggestions if possible, tks in advance

If you want to develop a new high performance file system; I'd start by splitting it in half - a lower level part and a higher level part.

The lower level part would mainly manage "linked lists of extents" - allocating new extents, freeing them, resizing them, reading them and writing them. Naturally it'd also manage free space, handle things like bad sectors, etc; and it'd probably be the best place to do things like "sparse lists of extents" (used for sparse files support), encryption, compression, etc.

The higher level part relies on the lower level part. A file's data is just a linked lists of extents. Directories are linked lists of extents that contain directory entries instead of file data. More advanced things like snapshots and forks are just more linked lists of extents.

If you split things like this you should be able to test the lower level part on its own; and could possibly have several "lower level parts" and several "higher level parts" and combine them in various ways. For example, perhaps one "lower level part" that tries harder to avoid moving extents for SSD and USB flash, and another "lower level part" that tries harder to avoid fragmented extents for hard drives; and perhaps one "higher level part" designed to be efficient for lots of small files and one designed to be efficient for lots of large files (plus maybe a stable version of each and a beta/development version of each).


Cheers,

Brendan

These are great points however I'm still a bit confused whether linked-list
may give good performance, coz, let say when we need to find some file with its
name and its path, it seems that we to have to scan 'linearly' thru' certain linked-lists,
and what if these linked-lists contain lots of items

Possible to allow O(logN) file search on these linked-lists
What may happen when we seek to a byte in a very large file

rdos,
I'm trying not to do double work :p