OSDev.org

The OS I am designing right now will not use a traditional file system. I was wondering what you guys thought I should do about C standards in this case since basically all of the io functions are dependent on the existence of files.

They aren't. They are dependent on streams that are:

• either binary or text, where text may be identical to binary but is allowed to do strange things to newlines and unprintable characters,
• either byte- or widechar-oriented,
• can be read from, written from, appended to, and may support positioning,
• can be identified in some way by an implementation-defined string.

That gives quite some leeway for "strange" persistence implementations. Even MVS can support most of stdio.

So I should make a fopen, fread, and company even though it's not actually doing what it says it does?

If you wish to support the C Library, you need to do whatever it takes to make those calls work on your system. If you need some more processing for a more abstract/less traditional file system, then do so.

The sole purpose of the C Library in this respect is to create a common interface that can be used the same on any system. Of course there are always slight differences (i.e. fopen('/home/me/unix.txt') vs. fopen('C:\mystuff\windows.txt'). You can either conform your system/library to a current "standard", or force the programmer to conform to your standard.

You can add a compatibility subsystem to your OS, and set aside a chunk of storage with directories and files, where traditional programs can feel at $HOME

(Wondering just how non-traditional it is... )

geezusfreeek wrote: So I should make a fopen, fread, and company even though it's not actually doing what it says it does?

If they are identifying a data stream through a name, and make it available for reading / writing / appending, they're doing exactly what the standard says it does. It is called "FILE *", but that's for historic reason - the standard calls it "streams".

Well, the biggest problem isn't the fact that it's a stream, but the fact that it is referenced by a path. This is a bit incompatible with my goals as there will be no such thing for users or developers.

@zioba: It's barely resembles a traditional OS at all. The design is coming together, but it's a very far-off goal (maybe a decade or so?) since I have not gained enough experience yet in OS development, nor do I have much time to devote to it yet. I am intentionally being vague about the internals, but I did list a few of the features that can emerge from my design in this blog entry: http://geezusfreeek.wordpress.com/2006/03/02/my-os/

Really guys, I sound just like an ambitious newbie, don't I?

geezusfreeek wrote: Well, the biggest problem isn't the fact that it's a stream, but the fact that it is referenced by a path. This is a bit incompatible with my goals as there will be no such thing for users or developers.

Not a path. A string. That could be a filename, an URL, a select statement, a database primary key. Nothing forces you to consider the [tt]char *[/tt] passed to fopen() as a path.

I think what's being got at here is the File system is just an abstraction anyway. Remeber the HDD doesn't really contain files and folders, its just a long series of 0's and 1's (and even thats a massive abstraction).

Hell your file system could be set to work with co-ordinates based on hard drive geometry if you really wanted then what you'd pass to fopen wouldn't be "c:\folder\test.txt" but "platter 4 1inch to the left 0.2 inches forward" it won't matter to the C programs which you use provided they can read out the data in a flow of bits.

Actually, the standard explicitly allows the "filename" to denote an interactive device...

Okay I see what you guys are saying. I'm sure I could come up with something, though the use of such an abstraction would be next to useless in my OS. Also, there is still the complication of fstat, which reports permissions, inodes, hard links, and other information that will be difficult if not impossible to return in the form specified by the returned structure.

well first of all, just cause you stat has certain fields, doesn't mean they have to have meaningful values..

if it's a number just put a 0, if it's a string just put "", simple enough right?

secondly, what do you have planned for storage if not something that is even remotely close to files? do you plan to be able to remember data between reboots? if so, how will you reference it?

I mean, even if you want you "FS" to be a database like SQL instead of a file system and have data only be accessible by selects, it is very easy to make it _look_ like a normal FS to a given API like libc.

maybe we can give you better suggestions if you can be a little more specific of your intentions.

proxy

geezusfreeek wrote: Also, there is still the complication of fstat...

fstat() isn't part of the C standard.

@proxy: Functions that return meaningless values all the time? Yuck. As for explaining how data storage works... here we go.

Firstly, the hard drive is used as nothing more than an extension of RAM, just swap space, and that is all. Secondly, the OS is persistent; it periodically swaps everything in RAM out to the hard drive (or does so upon a soft power down, reboot, or maybe a manual "flush") so that it can all be retrieved the next time the computer is booted. Obviously, there must be a little structure to the swapped data, but nothing too fancy as long as the kernel knows how to load it in at boot. Thirdly, processes on the system generally last forever. They are not "stopped," just "suspended." Obviously this is not always going to be possible, particularly with buggy programs, but mechanisms can be put in place to be able to restore to previous (stable) points of execution when needed.

At this point, I have basically described a system where each process takes care of its own data and never loses it. Data "storage" and organization outside of the process that created the data can be achieved through the use of one or many database processes. A database process would simply store the data inside its own address space and communicate with the other processes in the system for the data they want. A database that is designed to be able to handle certain kinds of media can even facilitate concurrent access to the same data by multiple processes by implementing a protocol designed specifically for synchronization on that kind of data structure. There are probably many possibilities here that I have yet to realize. I believe that this is far more flexible than simple file systems that provide nearly raw access to chunks of data on a disk.

So far, I am not sure if I want to include such a database process as a true "part" of the OS or if it should be considered an extra. Either way, since such things could be added or taken away at any time by the user, there is not necessarily going to be any true standard for accessing generic data storage on the system. There could be a "standard" such that any database must conform to a certain protocol, but this is limiting in flexibility. Obviously, if I was to actually include a database like this as a part of the OS, I would probably be able to implement the complete C standard such that programs written with it must invariably use the included database, but this cuts them out from the flexibility of being able to add custom databases/storage.

@solar: Ah, good to know.

Orthogonal persistence is fine, but what if a user wants to send a file as an email attachment, or save it to a floppy (usb keychain, CD-RW, 5" floppy) for viewing on a different system (non-orthogonal), how are you going to do that? Or conversely, how about receiving and loading one?

Say, the user makes a snapshot of data called "resume sent to XYZ inc. on March 10, 2006".. (not to mention the fact that it has to be viewable on whatever computer they're using in XYZ inc.)

You can't seriously expect that everyone will be running your system, and using its magical facilities to communicate that stuff transparently..

Interoperability requires that there be a tangible piece of data in a well-defined format. Humans require it to have some human-friendly identification like, say, a name.

And if your database does all of that, it *is* a filesystem, might as well call it that. Database, filesystem, "My Documents", whatever. Pretending that it is not will just make it more awkward.

The abstraction of having to store data on disk may be fine, until you start to care where it is stored and how. Where is my %%%ing file?!?!

Obviously, there must be a little structure to the swapped data, but nothing too fancy as long as the kernel knows how to load it in at boot.

in other words, the data will be restricted to your computer, assuming it never crashes, the hard drive never dies, and you never change the "Nothing Fancy v.1.0" data format.

Incidentally, there's the law of leaky abstractions