Ideas for changing how Programs interact with Files

[willedwards]

Back when I was due to start as an architect at Symbian, I published some articles to put out some ideas I had in the public domain.

I think I published them on osnews and a few other places. At the time, they were unflatteringly received. Anyway, for the curious, I think they are as good as ever ideas even today:

Network Transparency and Multi-root Filesystems

Files and Programmatic Streams

Openning Streams by Mime-Type

Feel free to use and abuse these ideas. Hope they give birth to creative useful new ideas too.

[embryo]

I published some articles to put out some ideas I had in the public domain

In my view all file system related ideas lack some generalization. Actually developer need not anything like this:

Code: Select all

File f=new File("http://www.microsoft.com");

Because the developer actually need just some part of the content at "http://www.microsoft.com". And bothering with some "library way" of accessing such content is not very pleasant. Developer needs tooling support for accessing some data and then for extraction of something useful. And this again leads us to the idea of metadata. If we have a world with pervasive metadata then we can ask our tools like this:

Code: Select all

String data[][]=Tooling.find("microsoft.statistics.sales.byRegions");
if (data!=null && data.length>0)
  .....

[willedwards]

Embryo, you describe the Wolfram Language. If you make any exciting Wolfram programs, please share.

Of course, you have to use lower level abstractions to implement the Wolfram runtime. Someone somewhere has to actually know how computers work.

[embryo]

Embryo, you describe the Wolfram Language. If you make any exciting Wolfram programs, please share.

Actually the language is Java. And the string with dotted notation is just a widely used method of declaring a path in a tree-like structure. An example of such notaion is - wiki.osdev.org.

Of course, you have to use lower level abstractions to implement the Wolfram runtime. Someone somewhere has to actually know how computers work.

The lower part of the job is done by a disk driver (in case of a disk access, for simplicity) and a piece of software like database server. The server parses the string and looks for mapping from string keys to disk sectors, then the server asks the disk driver to read the sectors. And there is no such entity as file, because it is useless in this case.

[willedwards]

Code: Select all

String data[][]=Tooling.find("microsoft.statistics.sales.byRegions");
if (data!=null && data.length>0)
  .....

And how would you know how to parse this array of strings? The data has structure more than simply being an array of strings.

If you read my three articles linked in the original, you'll see some thought went into them.

[embryo]

And how would you know how to parse this array of strings? The data has structure more than simply being an array of strings.

How a database knows what to show when parses SQL query?

If you read my three articles linked in the original, you'll see some thought went into them.

If you read some other articles then you'll also see some thought went into them. But if you wand to describe your opinion then the keyboard is near you.

[b.zaar]

FILE *f = fopen("ftp://www.microsoft.com/mission/statement.html","r");

I've wanted to integrate this into an OS for a while. By default all files would be opened with the 'file://' protocol and only extended protcols would need to be specified. This would be supported by drivers and could even be used with hardware like 'audio://0/' for recording from a mic or 'usb://3/' for input from a usb joystick. This would happen at the OS level so if you wanted to use a joystick library the library would detect and open the usb device automatically for you and then you just use the library functions.

[bluemoon]

FILE *f = fopen("ftp://www.microsoft.com/mission/statement.html","r");

I've wanted to integrate this into an OS for a while. By default all files would be opened with the 'file://' protocol and only extended protcols would need to be specified. This would be supported by drivers and could even be used with hardware like 'audio://0/' for recording from a mic or 'usb://3/' for input from a usb joystick. This would happen at the OS level so if you wanted to use a joystick library the library would detect and open the usb device automatically for you and then you just use the library functions.

This cause the same problem with unix as "everything is a file".
How do you handle seeking a joystick, "403 Moved" for a display, "Auth required" on the audio, or changing frequency/presentation format for a normal file?

Do they only exists on the interface and the programmer is told "certain cases is only bound to certain category of devices"?

[b.zaar]

I see devices being handled like this more as objects with a simple file name for opening the device initially. Changing any settings on a device like frequency/presentation format would be through I/O control system calls. I'm not sure what problems it would create that's different to other solutions. If a usb joystick is not attached when a program is looking for it on any OS it will appear as 'device not found' in some way. The naming of the devices could be anything such as usb://joystick/0/, usb://joystick/1/ or usb://joystick/logictechAwestick, usb://joystick/MSstuck etc and then every joystick found is listed in the game for a user to select. I haven't looked at other solutions to presenting network files or devices but these problems will still exist just in a different way.

[willedwards]

This cause the same problem with unix as "everything is a file".
How do you handle seeking a joystick, "403 Moved" for a display, "Auth required" on the audio, or changing frequency/presentation format for a normal file?

Do they only exists on the interface and the programmer is told "certain cases is only bound to certain category of devices"?

The mantra "everything is a file" doesn't actually mean everything is seekable. You can't seek in pipes nor sockets, nor even all files. In the old days, languages used to make you specify if you were opening a file for "random access". Unix made it so you just had to check seek() for error.

This is described fleetingly in my three articles, but thanks for bringing it up as it needs more awareness and attention.

[Combuster]

I'd personally start with making all storage accesses asynchronous by default (and ditch fopen). It pushes the developer towards making programs more responsive, easily allows multiple accesses to be performed concurrently without multithreading the application, and prevents architecturally-conceived shitstorms like linux kernel samba making it impossible to kill processes in case of an error.

[willedwards]

I'd personally start with making all storage accesses asynchronous by default (and ditch fopen).

Its an interesting orthogonal problem. I used to do a lot of async io, and have come to prefer lightweight threads.

Blocking io for IPC is the basic building block of L4, and it would be hard to imagine those kind of performance advantages being transferable to async io (for what I called "programmatic streams" in my articles).

On the new Mill CPU, you can actually move between peer protection domains (think process to process) without changing thread. This will lead hopefully to some interesting innovation at the OS level.

prevents architecturally-conceived shitstorms like linux kernel samba making it impossible to kill processes in case of an error.

I'm curious about the details of why this is? I recall hearing it before, but googling makes me no wiser.

[Combuster]

Try having a stable application when the kernel runs through several minutes of network recovery processes

[willedwards]

Well I've worked on kernels where disk-wait threads can be aborted, and this is orthogonal to blocking/non-blockingness of the IO calls.

At first glance I can't see why Linux doesn't support it. There may sply be the rationalization that it would affect too many old drivers to change it now, or there may be depths I am unfamiliar with.

If you make a kernel, please make IO tasks orphanable in their own right.