OSDev.org

amn wrote: There are smart developers everywhere, but if your say, technical manager told you to implement raw disk access for the sake of maximum speed in Linux how will you circumvent the file system layer?

I would tell him to go back to excel and stop bothering me.


Anyway, from my point of view virtual memory and file systems provide the same type of abstraction to an application. Both provide logical address spaces from physical resources. The main difference here is that file systems provides the possibility to use multiple named logical address spaces.

As soon as your system becomes the least bit complex there will be a need for the kind of abstraction a file system (or something file system like) provides. The very notion that there can be multiple applications implies some kind of organization regarding how we store the executable entities and associated data.

It only really makes sense to discard the storage abstraction if you have a very well defined target system that is unlikely to change (i.e. tiny embedded system). If this is not the case the application is forced to deal with the existence of other applications (created by other developers!) in the same storage medium as well as changing characteristics of different storage mediums.

Discarding the use of a file system and you will suddenly be forced to take care of things such as device size, block size and where other applications store their data (even other running instances of your application). In addition if the underlying storage type is switched all applications need to be rewritten.

amn, there are 2 points that I don't think have been covered yet.

1) The standardized filesystem abstraction is the only thing that has kept the API for disk access stable for the last 40 years. Keeping APIs stable is an enormous benefit that almost everyone overlooks. It may not be perfectly efficient, but it's still very good for being 40 years obsolete. I'm not about to chuck it in for any wild experiment.

2) You said yourself that the way the thin memory abstraction works is that "you get a segfault if you violate the allocation space." You neglected to mention that the segfault is generated by the hardware, because the memory mapping hardware is specifically built to understand memory protection schemes. And initting and maintaining that memory protection in the hardware is mighty complicated (extending all the way to PML4 tables and assorted whatnot). If you want to make a disk work like RAM, then you need to come up with a hardware memory protection scheme for all disks for decades into the future. Look at the miserable job that intel has done with keeping their memory protection scheme standardized. And then the disks need to be able to generate a hardware fault if you violate your allocation. So you have one horrifying job of hardware redesign and standardization in front of you to complete your dream. The disk manufacturers already do a very very bad job of following their own standards.

My argument is merely for filesystem not to monopolize a single disk, or at least for the system to let users transparently re-partition their storage space. In other words, the file system should not own all the storage space, but merely like a file may shrink and grow, the filesystem should shrink and grow, letting other sort of customers utilize the storage it does not itself occupy. Indeed, as some suggested here, one can always setgid and setuid here and there and what not, but additionally, with direct disk access I didn't mean unprotected raw sector writing. Instead I was advocating a thin layer of abstraction doing the job in software what the MMU does for memory - protection. A filesystem even though doing exactly that, is much more than just that - and the abstraction layer needs to be broken and its independent parts encapsulated into their own subsystems. I am talking about protection vs. management.

Which ultimately means you're advocating an explicit Exokernel-style storage approach.

Which in turn means that there's going to be a default library that implements a filesystem together with the stdio.h that all but a few apps will end up using anyway.

If RAM was non volatile and we had terabytes of it, so that hard disks were redundant - except possibly for backup, we would still want to have a way of putting labels on those parts of memory where our documents were located - a file system in other words.

Well, if you are using a memory management system such as a bitmap to shows which memory blocks are allocated, this is technically a file system (or a block system).

BMW wrote:Well, if you are using a memory management system such as a bitmap to shows which memory blocks are allocated, this is technically a file system (or a block system).

That's just used for physical memory management where the paging architecture mandates a certain page size (4k on x86). Other than the paging software, the OS typically uses linear address which are not uniform memory blocks.

amn wrote:My argument is merely for filesystem not to monopolize a single disk, or at least for the system to let users transparently re-partition their storage space. In other words, the file system should not own all the storage space, but merely like a file may shrink and grow, the filesystem should shrink and grow, letting other sort of customers utilize the storage it does not itself occupy.

That can already be done with stuff like logical volume management.

And my filesystem (Bang!FS) already does dynamic allocation, so it grows to fill the partition as you need it. The rest of the partition is unallocated and can be used for swapspace or anything else.

The fact that a disk is a long-term storage device, may affect the benefits of using an abstraction layer (like a file system).

When the system is using memory, it (usually) sets up some sort of database, that does not fit well into files (GDT, IDT, TSS etc.). It would make no sense to store it underneath such abstraction, since only the system needs to keep track of this data.

With disk drives this is different. This is a place for permanent storage of data, available to the user. The user therefore needs some way of organizing this data, using names, extensions, dates of last edit etc.

My point is: I couldn't care less what's in RAM when using an OS. I only care about what is on my disk, and how accessable it is.




Using filesystems in memory is a possibility though, but I do not believe it is very effecient...?