Jupiter OS filesystem design problems

[amd64pager]

I am currently developing the filesystem for my OS.
This is the design:
Indexes are per directory and contain the sector numbers for the file records.
A directory can have many indexes if the large directory flag is set in the directory header.
And it supports physical journaling.

But where should the info(indexes and records) be,and the data be?
I tried using the info is at the start and data grows downward from the end of the disk,but that will get cumbersome.
Could anyone give any ideas since my filesystem should be fast and support huge amounts of data?

[gerryg400]

Keeping the indexes in one place and the data in another is not great for performance. You'll tend to do a lot of seeking between the two. Ext and XFS filesystems have the concept of block groups. Each block group will contain its own superblock, bitmaps, tables and data. Typical ext2 block group size in Linux is 128MB. Files and directories may span a block group though the FS attempts to keep related files in the same block group.

[amd64pager]

Thanks for that idea.

So now I'll implement block groups.

[tom9876543]

With all due respect:

Why are you designing yet another filesystem????

I see you have "indexes" for each directory. I would suggest copying the MacOS idea, simply have indexes created by a background daemon process. As far as I know MacOS indexes aren't built into the low level filesystem meta data.

[gerryg400]

I see no problem with amd64pager designing a new file system. Aside from the fact that the new FS won't be supported in other OSes, is it really that different from designing a new scheduler, new memory manager or new network stack ? I thought that was why we were here.

[tom9876543]

You are correct, people can choose their own priorities for what they want to build.
IMHO I would prefer to get a GUI up and running before implementing a new file system, but that is me. Maybe Jupiter OS does already have a Compiz gui working, if so that is very impressive.

[amd64pager]

Jupiter has no GUI.And no shell too,because it is still in the planning stage.

[turdus]

Keeping the indexes in one place and the data in another is not great for performance. You'll tend to do a lot of seeking between the two

Wrong about that, almost every modern filesystems do this. For example when creating a btrfs volume it will pre-allocate the 1/3 at the beginning for indexes (well, for b-trees) expanding up, and keep the rest 2/3 for data, which expands down. It's faster because the indexes are cached in memory whenever possible, so no extra seeks necessary.

Ext and XFS filesystems have the concept of block groups. Each block group will contain its own superblock, bitmaps, tables and data. Typical ext2 block group size in Linux is 128MB. Files and directories may span a block group though the FS attempts to keep related files in the same block group.

Block groups and inode tables are the old and stupid way of organizing data on disks (limited number of files, too many sector reads on random access, does not scale well etc.). I suggest to read more on the topic, before you design your own fs. Study hammerfs, btrfs and zfs at least. Filesystem theory evolved much lately (extents, dynamic inode allocation, protection against silent data corruption, index tree approach, live snapshots etc.)

[amd64pager]

Filesystem theory evolved much lately (extents, dynamic inode allocation, protection against silent data corruption, index tree approach, live snapshots etc.)

Do you have a general explanation of the terminology you used?(I don't understand what you put in the brackets)
Is there a wiki article(s) for those stuff?

[turdus]

Filesystem theory evolved much lately (extents, dynamic inode allocation, protection against silent data corruption, index tree approach, live snapshots etc.)

Do you have a general explanation of the terminology you used?(I don't understand what you put in the brackets)
Is there a wiki article(s) for those stuff?

I don't think there's a wiki here for that. I'm afraid you have to do a little research of your own.
Some terms:
http://en.wikipedia.org/wiki/Extent_%28file_systems%29
http://en.wikipedia.org/wiki/Data_corruption
http://en.wikipedia.org/wiki/B-tree
etc.
many of these are explained here: http://en.wikipedia.org/wiki/Btrfs and here: http://en.wikipedia.org/wiki/ZFS

Again, I think designing your own fs is good thing, but you have to learn a lot before you start.

[amd64pager]

Thanks for the helpful replies.
So my main design is now based on b-trees.
But actually the data and btree-keeping is based on both sides.
trees have the LBA of the next element of the tree,so there can be data in between two tree elements.
And there is a physical journal called the Log Journal,and a logical one called the Update Journal.

[turdus]

You don't have to use any of the existing methods, just because they seem good for the first glimpse, like b-trees here

Instead I suggest to get a paper and pencil, and think of the following:
- what are the features that I lack from existing implementations? What is my goal?
- is it really necessary? If so, why others fail on implementing it if they tried at all?
- how could I implement it? Is it a good way (by speed or by size or whatever important to my goal)?
- what other features exists there that's worth adopting? Do I need to start over to merge them properly?
- comparing to existing ones, can I improve it in aspects it's beaten?

It took me years to figure out how to implement all of my fs' goals. I've studied many fs' on disk format, implemented and throw out many ideas. Eg. I found journaling and extents are dead ends for my purposes, I found better alternatives. But it does not mean they do not fit your needs. Always check ideas from several different perspectives, and if you cannot find any drawbacks, well, that's the time to put the paper away and write the first byte of your code (most likely a readfile() in your bootloader).

And for an example of interesting goal, I once saw a hobby OS that stored extra info along with fstat like device parameters. The file /dev/eth0 had ip address and gateway as meta data, so no configuration file needed. Unfortunately it was cancelled years ago, and I forgot the OS' name, so I cannot check whether it was finally continued or not...