File System theory: Z-OS FS

[gravaera]

Before I finally post up my draft overall system, I'd like to post up the Filesystem I've drafted up for comments.

Here's some BG-info before I get into it: My OS will have a kernel loader that simply initializes interrupts, and several other basics, before memcpy()ing several modules from the HDD into memory, and then running the kernel "Tie-in" module, which will interface those, and load the GUI, and link it to the loaded modules. Then, that's that for bootup. (The Driver frame is one of the modules)

Now here's the FS:

Code: Select all

+------------------------------------------------------------------------------------+
| 1|    2|                                          File Data                        |
|  |     |                                                                           |
+------------------------------------------------------------------------------------+
Note that in a Z-OS drive (the drive where the OS is installed), there will be extra information packed into the are between (1) and (2), as well as certain pointers to physical locations on disk for OS purposes. The core system files will be packed into a read-only area between (1) and (2), and will have no means of being referenced within the FS. The Z-OS FS skips over this point using a special pointer system, and goes straight on to the Root Directory (2). So I believe this will at least aid in the prevention of virus corruption of system files.

Region 1: Volume Boot Sector. (The area with the boot magic: 0x55 and 0xAA, etc.)
Region 2: Root Directory.
File Data: Self explanatory...

From here: The FS has no FAT (like FAT32, etc.), or Master File Table (like NTFS), but the FS begins right after the Boot Sector. Hold on: before you fire off those questions, look at the format for a directory listing structure: (A directory)

Code: Select all

-> 32b |               16b|    32b|      256B|      32b|             16b|
+----------------------------------------------------------------------------------------+
| inode | HDD segment | offset | filename | filesize | permissions |
+----------------------------------------------------------------------------------------+

With an accessing structure like so:

Code: Select all

struct zos_fs_dir_format
{
   void *inode;
   uword16 segment;
   udword32 offset;
   char name[256];
   udword32 size;
   uword16 perm
} zos_fs_directory;

The last listing in any directory (the end marker) is a row with the zos_fs_directory->inode set to 0.
In every directory file there is a row (first row) with inode set to 1. This contains a pointer (segment, offset) which will indicate the previous directory file's physical address.

In the root directory listing, the row with inode set to 1 will, in the name field, hold the drive name, and a list of 'aliases' for the drive. Upon mounting the FS, these aliases will be read, and placed into a configuration buffer in memory. The aliases will be used for portability between *nix, W32, and my OS. A partition can have the name "John's drive", with aliases "/" and "C:".

Simple. The kernel's HDD driver handles the alias translation.

I'e already pseudocoded it, and wanted to see if it really seems wholesome to anyone other than myself. There's more to it, but I've explained the basics for now.

[Combuster]

Designing an FS for the sake of it is generally considered a Bad Idea. Not to mention that ext2 is thought out better and supports everything and more than the things you put into this system. Even FAT32 has features that are gone amiss in your system.

You should seriously consider whether Ext2/FAT32/SFS can do what you want to do, what improvements you get by using your own system, and how easy it is to get your system operational compared to existing ones, so that you do not end up making a beginner mistake.

[NickJohnson]

Designing an FS for the sake of it is generally considered a Bad Idea.

But arguably, designing an OS for the sake of it is a bad idea too. Designing and implementing something to learn how/why it works can be pretty effective. You just need to make sure you support some sort of common format for interoperability. But having your own FS could be useful too, because you could make sure it's extensible enough to make ad-hoc modifications for testing new features.

[Combuster]

And arguably, many people don't "design" a new OS at all

Seriously, if you are writing an OS its not a good idea to complicate things by writing an FS from scratch in the process for the sake of it. If someone wants to design an FS, they should better grab FUSE or the like.

[gravaera]

Honestly, I understood from the start that designing an FS may not be the best way to go, but..I'm in OSDev for the learning experience. I feel like I'd be missing out on a huge part of what it should be if I don't make a proprietary FS specially catering for my OS.

I dunno. Do you guys feel good implementing other peoples' systems in your code?

Except for the ELF spec, everytime I realize I may have to subsidize with an established system, I feel like I'm defeating the purpose of dev'ing my OS. Look at Windows.

The MS peeps implement their OWN systems, as they see FIT, and conform whenever it suits them. I understand the need for portability, etc, etc, but apart from Application portability, and generally obvious area where I should conform (It would be nothing but plain dumb to create a CDFS of my own, for example), I don't see any need to do much more than create a workable interface to cater for interoperability.

I understand the whole...POSIX, and whatnot, but I strongly believe that sticking to All these POSIX ideas, and standards will make me yet another "*nix" clone. It will limit me considerably.

I'm not even going to bother doing full ELF compatibility on a major scale. I've decided that ELF is perfect as is, and that it would do great for my OS. I'll implement it as the pure standard, and also add extra compatibility with, say, Ubuntu, and that's that.

I mean: do you guys not get a sickly feeling everytime you have to recycle code or systems? Like and "I wasn't good enough" feeling? Why the hell else am I in OSDev, if not to reach the pinnacle of programming? If nt to perfect my hobby? To see MY code come to life, and know it's not some kind of glued together mass of code from other experts? I WANT to re-invent the wheel.

[b.zaar]

Honestly, I understood from the start that designing an FS may not be the best way to go, but..I'm in OSDev for the learning experience. I feel like I'd be missing out on a huge part of what it should be if I don't make a proprietary FS specially catering for my OS.

...

I understand the whole...POSIX, and whatnot, but I strongly believe that sticking to All these POSIX ideas, and standards will make me yet another "*nix" clone. It will limit me considerably.

...

I mean: do you guys not get a sickly feeling everytime you have to recycle code or systems? Like and "I wasn't good enough" feeling? Why the hell else am I in OSDev, if not to reach the pinnacle of programming? If nt to perfect my hobby? To see MY code come to life, and know it's not some kind of glued together mass of code from other experts? I WANT to re-invent the wheel.

I agree with these points and think its good to invent your own filesystem for your OS if it will suit your system better than an existing filesystem. Using defined specs such as ELF and multiboot can be helpful but some things need to be personalised for your OS.

I've been working on a filesystem for venom which uses parts from BeFS and XFS and would be interested in comparing features and functions with you.

[Combuster]

I WANT to re-invent the wheel.

More power to you.

I mean: do you guys not get a sickly feeling everytime you have to recycle code or systems? Like and "I wasn't good enough" feeling? Why the hell else am I in OSDev, if not to reach the pinnacle of programming? If nt to perfect my hobby? To see MY code come to life, and know it's not some kind of glued together mass of code from other experts?

I grabbed a C library from somewhere, I grabbed a freebasic runtime from somewhere else, and I honestly think that I was good enough to not start wasting my time on totally uninteresting things.

Since you seem to have made up your mind already, let me post this rhetoric question:
Is it the mere fact of writing an filesystem that makes you feel good, or is it the fact of improving on existing filesystems that makes you feel good? Do you really think that developing a (less scrutinized) filesystem from scratch teaches you those things that you can not get from implementing a existing FS from scratch and some other OS development task?

And the bottom line:
If we know what you want to achieve with your FS, we can give a more directed comment on where things go. To be really honest, the original reason for suggesting not to design a FS is that this design appears as a mush-up of random features taken from Ext2/SFS, with little thought of the background processes and theory involved.

[gravaera]

Well, honestly, I never even read up up on Ext2, or the other FS you mentioned.

Is it the mere fact of writing an filesystem that makes you feel good, or is it the fact of improving on existing filesystems that makes you feel good?

No. The experience of having written out the code by myself, and seeing it come to life gives me pride in knowing that I was able to do it by myself with just the understanding of the concept.

I understand the idea of improving on other peoples' work, but I'd ffeel a hell of a lot better to know that I was able to do something of my own entirely instead.

[earlz]

Just remember, no one will want to use your FS if you must go through your whole disk to list a directory, and if adding files to that same directory involves defragmenting(with a noticeable delay)

[Combuster]

The reason I mentioned SFS is that it has the same search-defragment problems. It is however built for when the common case is many large files added, and deletion occurs in large batches.

For regular OS usage, that is not a valid assumption. Hence, practically all FSes support fragmentation and have some sort of structure indexing free areas. (FAT tables, the bitmap in ext2)

[VolTeK]

didnt ntfs fix the fragmentation problem? or did they just say that?

[frank]

didnt ntfs fix the fragmentation problem? or did they just say that?

While they significantly reduced the impact of fragmentation and the amount of fragmentation it is still present. I usually end up with a bunch of fragmented files after installing a new program. But it doesn't seem to hurt performance.

Here is something that talks about fragmentation in FAT vs NTFS
http://www.pcguide.com/ref/hdd/file/ntfs/relFrag-c.html

[VolTeK]

Hey thanks

[frank]

@op
Before you get to involved with your own filesystem maybe you should make some tools to read from common filesystems such as FAT and EXT2. That way you can see some of the benefits and drawbacks of both and figure out what you want in your own filesystem. Or you could try writting a ramdisk filesystem, they're much simpler.

[salil_bhagurkar]

@op
Before you get to involved with your own filesystem maybe you should make some tools to read from common filesystems such as FAT and EXT2. That way you can see some of the benefits and drawbacks of both and figure out what you want in your own filesystem. Or you could try writting a ramdisk filesystem, they're much simpler.

QFT
I think a simple hard-coded implementation of FAT12 would be good. Before you get to your own fs after that, try a simple write support and see the problems you face.