Filesystem design

[Candy]

Back to topic however, it might also be a nice idea if the noncommercial programs are installed in this way. Commercial programs however, might just as well be distributed through the internet (personal opinion) since just about the entire world is going to be swamped with internet access within a few years (at most 15). If most (everybody) is going to be connected and have a reasonable bandwidth, why not get the programs directly from the authors?

They'd distribute a package of X MB to be dumped on disk directly, then to be executed directly. Combining this with online home space management (on company or ISP servers, if we're going to take over the world...) where you store your settings etc. on a fast server, that pretty much allows you to log into any computer and use it as your own. Makes it a lot better to work around the globe at random computers, not being forced to figure out what is installed at this computer and just using your own preference.

then, it may need storage for user's settings and/or data, obviously within the user's quota. no apps refusing to work under a nonpriviledged user.

Would be quite a good point, should fit under the user space somewhere. Possibly a second section, although I favor merging those two (for ease of management). The user should be able to view which application takes up which amount of space, I personally like an area-equivalent-with-space-taken approach, such as a pie chart or a square overview (with a square per file). This gives a lot better view what is taking up this load of space as opposed to a bunch of anonymous directories.

if the user wants to open a file with the app, they can "feed" the file to the app via a system-provided "open file" dialog. (this also adds the possibility of accessing more than just normal files - such as HTTP documents - with the OS or another app encapsulating the retrieval of the document, and the app not giving a d*mn about it, much less including a full HTTP implementation)

Exactly. Half of this is what Pype already suggested (including the "open file" dialog and the other half is something I had also thought about (the open protocol thing), but Solar can give quite a good word at that too.

[zloba]

Back to topic however, it might also be a nice idea if the noncommercial programs are installed in this way. Commercial programs however, might just as well be distributed through the internet

also a good idea. (saw this on Pype's site too)
this also may require the possibility of distributed filesystems (remote slices/sections over the network), how about that?

Not entirely agreed upon. I consider the programs locally installed to be more an operating system thing, similar to what current systems do. I strongly prefer it to be taken from the global "programs" slice into its own slice (special case ) such that the amount for programs in general can be limited, each program can be checked upon, but nobody is penalized for having a bunch of programs installed.

i'm not sure what you mean and what you disagree with. can you explain or rephrase?

[Candy]

Back to topic however, it might also be a nice idea if the noncommercial programs are installed in this way. Commercial programs however, might just as well be distributed through the internet

also a good idea. (saw this on Pype's site too)
this also may require the possibility of distributed filesystems (remote slices/sections over the network), how about that?

Is an idea of course... However, I don't think this falls under the on-disk filesystem anymore, and more under the implementation of the layer above this one. Which is also nice and important to discuss about, but doesn't help with this layer.

Of course, it's a good idea to not copy the entire slice right away, but to just copy whatever's needed. With a fast internet and somebody decent at programming (IE, not copying the entire program in a single run action would be a nice start) this could well be the faster way. We might need to store the remote slice location along with the local ones, that's a good point :S...

Not entirely agreed upon. I consider the programs locally installed to be more an operating system thing, similar to what current systems do. I strongly prefer it to be taken from the global "programs" slice into its own slice (special case ) such that the amount for programs in general can be limited, each program can be checked upon, but nobody is penalized for having a bunch of programs installed.

i'm not sure what you mean and what you disagree with. can you explain or rephrase?

I propose a single section for "programs" which is entirely empty, with just quota attached to it. Then, when you install a program, it's taken from this slice and not the user slice. This way, if somebody sets up a computer, he's not void of space for himself.

You should of course be able to see how much space a program takes in order to keep your harddisk with available space.


Something else I'd like to advise to setup programmers, make the setup dynamic. Dynamic as in, the size of the install shouldn't be a fixed thing set up at install time but more something you set at install time and can later change. It's nice to install every game with "full install" when your harddisks is empty. However, when it's full, those installs are hogging the harddisk for themselves, and you'd probably like to shrink them, but not delete them. Seeing as it's very possible, why didn't anybody do that?

Note for self, have to include a bit with file indicating importance, as thought up months ago. Cached files (internet, cdroms etc.) can be deleted pretty much without much thought.

Note to Pype, the tables I'm defining right now are things that have more than a superficial consequence for the filesystem. Including artist and song information is plain information with which the FS doesn't have to do anything. Including a backup copy is something that's at the FS level of responsibilities. The second should be defined here (imho), the first shouldn't.

[zloba]

I propose a single section for "programs" which is entirely empty, with just quota attached to it. Then, when you install a program, it's taken from this slice and not the user slice. This way, if somebody sets up a computer, he's not void of space for himself.

i'd think that when somebody sets up a computer, they are the administrator/root and they can't run out of quota because they own it all..

so the administrator would install programs into the programs section. quotas would be needed not to restrict them, but to keep track of how space is used.

but allowing users to install whatever programs into the programs section, as much as they want? that's fine when it's your own computer and you are also the administrator, but not in a multi-user environment - one user may want to install MS Office, the other - a bunch of games, yet another - a huge CAD package..

[bubach]

ohh ****, you guys really have some cool ideas.
i just can?t figure out why you aren?t working together with _one_ OS.
if it wasent for my much more simpler os design, one user/task only (mostly for personal use), i would have joined with pype..

/ Christoffer

[distantvoices]

Gonna take some time to read throu' this.

@bubach: How's the going wih yor kernel?

I've been stuck with debugging ext2-fs AND my floppy driver for days until suddenly a small knot sprang up in my mind - I reverted to an older state - copied over changes in other parts after some diff's, changed a small loop condition and suddenly it works like a charm - in virtual space and on a real computer.

Well - I think this Idea about slices/sections needs to be fleshed out. Currently I can't imagine it working - not understanding, that's pretty ok, but seeing a picture of it doing work in my head/with my inner eye.

[bubach]

@BI: good i suppose.. except the booting part.. other then my wothless attempt with my bootsector the kernel is fine and i have a simple commandline with: cls, reboot, regdump (for fun), about, help etc..
i becomes a bit lazy when i can?t see any real progress, right now i don?t even have any power at home due to a big storm in denmark,sweden,UK etc (sitting at school now).. + i have lots of school work to do.

[EDIT] it sounds like this FS isn?t going to be compatible with any other os. eg. a hardrive is for that FS alone.

/ Christoffer

[Candy]

[EDIT] it sounds like this FS isn?t going to be compatible with any other os. eg. a hardrive is for that FS alone.

It does support other OSes beside it, even with already-existing bootloaders. If you partition the disk however you like it and leave a partition for this filesystem, it can treat that as the entire "disk". In other words, it also works on a partition instead of the entire disk, so you can use other partitions for other operating systems. It even works with a mix between the two or two partitions on one disk (allowing you to use both as one! Not really a design goal but still nice if you have a linux/windows install in the middle).

The design precautions still to be taken for this (in my head they're there) are leaving bytes 448 to 510 free in the first sector (for a stub partition table so old OSes don't wreck the disk by thinking it's empty) and including a 64-bit sector pointer in the first sector to allow the bootloader to easily find the rest of the boot loader without searching through the disk finding the partition itself (after the other boot sectors already did that).

[zloba]

these things don't belong at the filesystem level - caching, encryption, compression, UID/GID, versioning, "the usual access control" ..
putting all these into one design complicates it too much. these may also conflict with how other OSdevers want to solve these problems.

these can be put in a separate layer, on top of a filesystem - call it "file management" or whatever..
the filesystem should provide generic attribute storage instead, as Pype said.

it should be structured into layers, so that each layer has specific responsibilities. and design should start with sketching those layers, defining their interfaces.

[Candy]

these things don't belong at the filesystem level - caching, encryption, compression, UID/GID, versioning, "the usual access control" ..
putting all these into one design complicates it too much. these may also conflict with how other OSdevers want to solve these problems.

these can be put in a separate layer, on top of a filesystem - call it "file management" or whatever..
the filesystem should provide generic attribute storage instead, as Pype said.

it should be structured into layers, so that each layer has specific responsibilities. and design should start with sketching those layers, defining their interfaces.

I strongly disagree.

Making all these attributes separate from the rigidly defined structure that the specification should become makes it less clear how and where to find them, and this is exactly the distinction I'd like to make from NTFS. Encryption doesn't work if it's an afterthought (or put nicely, a separate layer at a too high level). Security at the very least doesn't work when you leave it to a top layer (think people cutting below the security line, which is very well possible if it isn't properly defined). I consider the idea of making a sound and closed security concept not something OS devers should be worried with. At the very least not at times of implementation. Also, for sharing passwords etc. between OSes (which is also a design goal of this project, at least of my OS somewhat) it's required to make them a shared object and to thus use the same methodics for using and determining them.

My consideration is that everything with a direct influence on disk structures of the filesystem (which includes file backups, logging, applying the security policy (which must be done at a level where you can't bypass it by say taking a number of bytes on disk as a key), making it still accessible to all and securely tight and compression, should all be defined at the filesystem layer. These all affect how things are stored in a nonpredictable way, and as such should strongly be standardized before any wildgrowth is created. Nobody cared a while ago about the default FS in windows since you only had one choice and /EVERY/body used it. The point with a true standard is that if you follow it, you can communicate with all others. If the standard doesn't define a lot of essential parts you can't communicate. What good is a language if you can only say hello?

[zloba]

Encryption doesn't work if it's an afterthought (or put nicely, a separate layer at a too high level)

did i say anything about "TOO high level"? how about just high enough? And why does it have to be afterthought?
everything works nicely when it's in the right place.

[Candy]

Encryption doesn't work if it's an afterthought (or put nicely, a separate layer at a too high level)

did i say anything about "TOO high level"? how about just high enough? And why does it have to be afterthought?
everything works nicely when it's in the right place.

Ok, sorry bout that last post, was kind of not happy...

I personally think that it's not right to include things that should modify the file storage or usability for other operating systems at a level not inside the filesystem defined layers. Doing so will make the actual file unreadable to the other operating system even though it conforms to the file system. For the user, this will mean that both operating systems (and the file system) are less "reliable" because he can't share files securely between the two. On another hand, it might work but there would be a new "standard" on top of this one that people would have to conform with to work with the features one could expect this sole specification to supply.

In short, if you don't specify things changing files beforehand, others will experiment and develop incompatible standards on top of this one. One of the intents of this standard is to end incompatibilities between operating systems and leaving a lot of open ends concerning wanted features is exactly how not to do it.

If you think leaving them open or accessible to others also leaves the ability for interoperability, could you explain to me how?

[zloba]

suppose (for example) you define 2 levels:
-basic filesystem: provides directory structure, allows access to files and directores. these have data and metadata (attributes), attributes are kept in a black-box fashion.
the API (or object model) for this layer is one standard.
-file management, where such attributes are handled, perhaps with certain plugins which may be OS-specific, such as password-based encryption.
this forms another standard. or maybe a whole set of small standards for: date/time, access control, encryption..

Doing so will make the actual file unreadable to the other operating system even though it conforms to the file system.

quite the opposite.

you could pass the file around undamaged, including its attributes.
or boot into a different OS and still see the filesystem, minus the differences in attributes, if those use different standards (which may not even be the case if you define a generally acceptable standard).
which shouldn't matter much for the contents of ordinary files.

also, you could possibly load a plugin for the standard actually used in the file or filesystem, and thus use it the way it was intended.

On another hand, it might work but there would be a new "standard" on top of this one that people would have to conform with to work with the features one could expect this sole specification to supply.

which is the idea.
by making standards separate, you make it possible to accept some but not others.
for example, i may accept your versioning standard, but not the access control, while ignoring caching.

also, this allows a modular implementation, where modules for different standards and different layers can be mixed and matched, with each module implementing only those nontrivial features that are needed to make it work with the rest.

if you don't specify things changing files beforehand, others will experiment and develop incompatible standards on top of this one

if others don't like how you define these things, they will develop incompatible standards anyway, and not even on top but completely unrelated. that way, eveyone loses.
but if you provide the generic, extensible, modular filesystem abstraction, they may accept the basic layers, that way you would still be able to browse the filesystem and read the data, possibly even with full features.

so interoperability would be achieved at the filesystem level, despite different on-disk structures, storage/access methods and attribute interpretations.

One of the intents of this standard is to end incompatibilities between operating systems and leaving a lot of open ends concerning wanted features is exactly how not to do it.

do you think you will convince everyone to accept yet another swiss-army-knife standard and abandon their open ends?

(i don't think it will work either way, considering the wide range of OSdevers' objectives, motivations, fanatical preferencies for certain methods and overemphasis of certain problems)

everyone thinks they have a pretty reasonable solution, while in fact it's a solid block, tied to particular OS family, disk structures, design decisions, implementation..

[mystran]

I can see how you don't want too much stuff in your filesystem desing. Most of zloba's "these don't belong in" actually need SOME support from the filesystem to work in a sane way, but you can just have an extensible set of attributes for each file, and use those for plugging other features in. Actually, that's pretty standard way to handle it.

As for encryption specifically, I don't see how it is "too high level" or whatever. Where you want your encryption depends on what you want to protect with it. Want to protect local data from people who steal your hardware? Then easiest thing is to put the encryption layer between the filesystem and the disk-driver. That way the whole filesystem will be encrypted.

Or do you want to protect certain files? Forget about encryption for that. It makes no sense, unless you are going to play distributed filesystem, in which case you might want to encrypt the network communications. Or can't trust the nodes connecting to your distributed filesystem? Ok, now we might want to encrypt files, but now we have a huge revokation problem.

As a general usability design ideal: try to reduce the number of passwords user has to remember. Password encrypted files are stupid: either people just use their normal login password, or they use so easy passwords that you'll brute force them in minutes. Either way it's bad idea.

Personally I think that it is enough to be able to read the filesystem from one OS, as long as (1) you can mount local fs from a booting cdrom, and (2) it supports network filetransfers with other OS, or can mount FAT.

[Solar]

As a general usability design ideal: try to reduce the number of passwords user has to remember.

There is a deep truth in that sentence. Ideally, once a user has identified himself (password, smart card, biometrics), he shouldn't have to remember any passwords. Not for websites, not for PGP keys, not for SSH access, nothing. A system-wide KDE kwallet, so to speak.

But that's beyond the scope of the filesystem, really. Just a point that's been in the back of my mind for ages.