FAT (was "Making a File System")

[skandalOS]

Hello, I've also questions in this area.

First questions: When I write a second-stage bootloader and want to copy this second stage 12345678.SYS on the HDD not as
a raw binary, which should be registered in the Root Directory and FAT(12) or whatever, which should be able then to find it.

How is this information registered in FAT anyway?
I dont think that giving BPB parameters is enough for this purpose. On what depends this BPB parameters, on the HDD? or does
the HDD depend on the BPB?
How do I get control to the FAT, I mean have I go through the clusters to read for example the root directory, should I calculate it?

Second:
Every harddisk has got a filesystem, are there any without filesystem?
Ok, when I have a blank harddisk and I want to put a filesystem on it I will do it of course with a software and so on, but
as a OS programmer, when I work with QEMU, should I innately assume that there is a Filesystem already been integrated
on QEMU?

please excuse me for my stupid Questions, if there is anyone but could anybody give me enlightments?

Thanks

[Brendan]

Hi,

Hello, I've also questions in this area.

First questions: When I write a second-stage bootloader and want to copy this second stage 12345678.SYS on the HDD not as
a raw binary, which should be registered in the Root Directory and FAT(12) or whatever, which should be able then to find it.

How is this information registered in FAT anyway?

Anything "registered in the Root Directory" is just a normal file in the file system (possibly with the "system" flag set). To find any normal file in the file system, you start by scanning root directory entries.

Normally at the start of a FAT partition there's "hidden sectors". Your first stage boot loader would be in these hidden sectors, and would be large enough to contain code that understands the FAT file system (and can find files within the FAT file system). There's no reason for this to be limited to one 512-byte sector - it could be 123 sectors (where the BIOS loads the first sector and the first sector loads the remaining sectors).

I dont think that giving BPB parameters is enough for this purpose. On what depends this BPB parameters, on the HDD? or does
the HDD depend on the BPB?

The BPB is just a little structure that Microsoft invented. It's not used by the BIOS and not used by the hard drive itself. Whatever formats the FAT partition is responsible for creating a "BPB" for that FAT partition.

Every harddisk has got a filesystem, are there any without filesystem?

There's no strict requirement for hard disks to have file systems. For example, you could use an entire hard disk for swap space, split the disk space into 4 KiB pieces, and use RAM to track what each 4 KiB piece is used for. You could also have (at least in theory) some sort of advanced pieces of software (e.g. a database management system) that uses raw sectors to store data instead of files. There's also things like RAID, where each hard drive contains a fraction of one file system and not a whole file system.

However, because hard disks are relatively huge, using them to store files makes a lot of sense.

Also note that the reverse is also true - you can have a file system on something that isn't a hard drive. This includes file systems inside files, file systems in RAM, etc.

Ok, when I have a blank harddisk and I want to put a filesystem on it I will do it of course with a software and so on, but
as a OS programmer, when I work with QEMU, should I innately assume that there is a Filesystem already been integrated
on QEMU?

Qemu doesn't automatically create file systems; and it's not really possible for Qemu to guess what file system/s you might want or what they would contain. For example, there are no modern OSs that still use the FAT (because FAT is crap) - they all use better (more advanced) file systems, like NTFS, HFS, ext2/3/4, etc. You can use Qemu like a normal computer, and use tools (from a bootable CD or something) to create partitions and file systems (potentially as part of installing the OS inside Qemu). However, it's also possible to use tools on the host OS to create "hard disk images" that contain partitions and file systems, and then tell Qemu to use the pre-made hard disk image; so that you don't need to do it inside Qemu.


Cheers,

Brendan

[gerryg400]

For example, there are no modern OSs that still use the FAT (because FAT is crap)

Don't forget smaller USB drives. Many are formatted as FAT.

[Brendan]

Hi,

For example, there are no modern OSs that still use the FAT (because FAT is crap)

Don't forget smaller USB drives. Many are formatted as FAT.

You're right - FAT is still used for transferring data between different systems, and I should've said "There are no modern OSs that still use FAT as the file system the OS is installed on or boots from".


Cheers,

Brendan

[Combuster]

I should've said "There are no modern OSs that still use FAT as the file system the OS is installed on or boots from".

and therefore all our OSes are outdated?

[Brendan]

Hi,

I should've said "There are no modern OSs that still use FAT as the file system the OS is installed on or boots from".

and therefore all our OSes are outdated?

As an exercise, write down the 5 most important characteristics for a file system that you want your OS to be installed on (e.g. security, fault resilience, performance, etc). Using the characteristics you've chosen, compare FAT to NTFS, HFS+, ReiserFS and ext3/4. You can almost guarantee that FAT will be beaten by everything else for at least 4 of the 5 characteristics you chose.


Cheers,

Brendan

[Solar]

You could also have (at least in theory) some sort of advanced pieces of software (e.g. a database management system) that uses raw sectors to store data instead of files.

AFAIK this is standard practice on Oracle DBMS, and quite possibly others. They get assigned raw partitions, so they don't have to tunnel their persistent memory management through a file system handler (for a 5% to 10% performance advantage).

[Combuster]

You can almost guarantee that FAT will be beaten by everything else for at least 4 of the 5 characteristics you chose.

Since you asked: Simple, Portable, Well documented, Wide range of existing tools, Operates on a floppy

But that was not the point I tried to make. You can make a lot of progress in more important domains than dealing with filesystem stuff. Face it, every "modern OS" of yours will still have a FAT driver

[turdus]

Combuster: there's a huge difference between providing an fat driver for an OS, and building a whole OS on top of it. You should know that. Choosing a main fs is very important, and I suggest to implement your own, that fits your kernel's need best. If you use for example fat32, you'll suck with large disks, symlinks and inodes and mounted isos later, you can bet.

[gerryg400]

I don't think there is much to argue about here. There are lots of filesystems and they each have pros and cons. Because of that all operating systems need to support more than one filesystem to cover all the common use cases.

[Brendan]

Hi,

Combuster: there's a huge difference between providing an fat driver for an OS, and building a whole OS on top of it. You should know that. Choosing a main fs is very important, and I suggest to implement your own, that fits your kernel's need best. If you use for example fat32, you'll suck with large disks, symlinks and inodes and mounted isos later, you can bet.

Agreed. All OS's probably should provide FAT support (and support for other file systems that are primarily used for information exchange, like ISO9660). Installing the OS on one of these filesystems (and therefore needing to boot the OS from one of these file systems) is a serious design flaw (unless you're one of those people who are writing an OS as a hobby in an attempt to learn how to do things badly).

For a related example, consider UEFI. It uses FAT for an "EFI system partition", which means that any idiot (using almost any OS) can screw up anything in the "EFI system partition" (either accidentally or maliciously). There's no sane/easy way to guard against that, and you'd probably have to resort to something extreme (like requiring digitally signed executables) just to solve some of the security problems (where "some" doesn't include DoS).


Cheers,

Brendan

[Brynet-Inc]

I honestly believe foreign filesystems should be treated differently.

Include mtools-like utilities/libraries to access these filesystems, no native kernel driver, no FUSE, treat them as archives of files and handle it entirely in userland.

Having the kernel deal with a plethora of untrustworthy filesystems seems like a bad idea, it should only have to deal with native filesystems.

[Owen]

The kernel should have no concept of file systems

(i.e. that is monolithic thinking, and may be a reasonable design decision in a monolithic kernel based system, but a micro- or hybrid-kernel system need not follow such an exclusive approach)

Hi,

For a related example, consider UEFI. It uses FAT for an "EFI system partition", which means that any idiot (using almost any OS) can screw up anything in the "EFI system partition" (either accidentally or maliciously). There's no sane/easy way to guard against that, and you'd probably have to resort to something extreme (like requiring digitally signed executables) just to solve some of the security problems (where "some" doesn't include DoS).

Of course, that is no fault of the file system (and if the FS were something non-FAT, it would probably have hindered those implementing support)

[piranha]

Reading this thread reminded me that I wanted to write a FAT driver for my OS. So, I started in. Couldn't be too hard, right?

...good god. This is terrible. It isn't difficult, its just....bad...i don't know. I found ext2 to make much more sense. Whatever

-JL

[Kevin]

For a related example, consider UEFI. It uses FAT for an "EFI system partition", which means that any idiot (using almost any OS) can screw up anything in the "EFI system partition" (either accidentally or maliciously). There's no sane/easy way to guard against that, and you'd probably have to resort to something extreme (like requiring digitally signed executables) just to solve some of the security problems (where "some" doesn't include DoS).

So you're arguing for security by obscurity? Just use a file system that no OS implements (yet) and everything magically becomes secure?