The OS-periment's IPC design is looking for "testers"

[NickJohnson]

How does the client learn about the server's PID ?

It looks it up using the VFS. Also, I didn't see anything about finding a particular driver in your list, unless you only have one driver implementing each function call so that looking for an adequate function equates to finding the right driver. If that is true, I would definitely place it under the "over-binding" category.

How do different versions of the client and the server communicate with each other ?

There's no explicit versioning, but you could easily name the commands based on version, or use a command to query a version. However, I've found that in general, interfaces are extended by adding more commands, not changing existing ones, assuming the interfaces are decently well-constructed in the first place. It's not like a library API: the VFS only uses about ten RPC commands, including locking and security, and few drivers implement more than a couple of their own. Also, you don't always use the RPC directly--VFS stuff at least is done via C library stubs--so as long as the interface can do everything you need it to, the exact form (name, argument order/type) of a command is unimportant.

Also, how would you manage shared memory for references and large parameter transfers ? And asynchronous operation ?

It's not a purely RPC system: the real primitive is asynchronous message passing, which uses shared memory. The RPC only uses strings, so there are no pointers: you have to serialize objects somehow if you want to send them. However, RPC is not used for I/O, so this almost never happens, especially not on a large scale. RPC, especially synchronous RPC, is great for a lot of things, but is pretty bad for certain ones like I/O and events.

[Neolander]

It looks it up using the VFS. Also, I didn't see anything about finding a particular driver in your list, unless you only have one driver implementing each function call so that looking for an adequate function equates to finding the right driver. If that is true, I would definitely place it under the "over-binding" category.

Not necessarily so, I think. If process don't communicate directly with the drivers, they should communicate with a higher-level entity. For disk drivers, as an example, that entity would be the VFS. And that higher-level entity can do the driver accounting job.

Imagine the following scenario :

• VFS process is started first
• Hardware parsing occurs to check the number and kind of disks, and driver for each is started accordingly
• Disk drivers register to the VFS process during their startup, using a standardized interface, so that the VFS knows about them and their PID
• There's only one VFS process, so drivers know how to find and contact it
• Drivers have registered to the VFS, so the VFS knows how to contact them

Sole issue which I can see with this is that if the VFS dies and is resurrected, it doesn't know about the disk drivers anymore. But that could be addressed if each time a new driver registers to the VFS, the VFS keeps track of it in some form of crash-proof storage that can be reset on each boot.

It's not a purely RPC system: the real primitive is asynchronous message passing, which uses shared memory. The RPC only uses strings, so there are no pointers: you have to serialize objects somehow if you want to send them. However, RPC is not used for I/O, so this almost never happens, especially not on a large scale. RPC, especially synchronous RPC, is great for a lot of things, but is pretty bad for certain ones like I/O and events.

Well, I'd like an IPC primitive which is good for that too, and I think that my implementation of async RPC, combined with shared memory, could do the trick. But it might be just me... Thing is, to test I have to implement, and implementing is sufficiently hard that I don't want theoretical problems lying around.

[Neolander]

I'm in particular interested in the dialog emitted by .pkg installers. To me, they sound like a step backwards from the cleanness of .app bundles, but also like a security risk since you give full root access to a script which can do whatever it wants. Like on Windows.

[NickJohnson]

Imagine the following scenario :

• VFS process is started first
• Hardware parsing occurs to check the number and kind of disks, and driver for each is started accordingly
• Disk drivers register to the VFS process during their startup, using a standardized interface, so that the VFS knows about them and their PID
• There's only one VFS process, so drivers know how to find and contact it
• Drivers have registered to the VFS, so the VFS knows how to contact them

That makes sense. I forgot to mention that my VFS is not contained within a server, but is instead distributed across all drivers, which sort of makes a big difference.

But what if you want to control a specific driver? Let's say I wanted to tell my graphics driver to change it's resolution, which is clearly an interface that not all drivers implement, but one that more than one driver could at a time. How would I do that? It seems like making that type of request go through the VFS would be sort of ugly.

[Neolander]

Asynchronous calls are inherently more complex, also they require some preliminary setup.

I agree that the setup is an annoying part of the equation, although I think there's some setup overhead with all IPC methods, either in kernel or user space. But is async always conceptually more complex ? For everything which resolves around sending a task to a process without any reply being required (events/notifications, interrupt handling), async is a more natural way to do things than synchronous. For things which do require a reply, that reply might not be needed right away, in which case it is stupid to wait for it (imagine that as part of its initialization, a process has to setup a display and a sound interface. It would be better for the process to send the setup requests to both hardware right away before starting to wait for the replies). What would be typical examples of tasks sent to distant system services that would be better handled by synchronous calls ?

How are you going to initialise a callback receiving thread pool before you have threads package up?

I won't. I'll setup the thread package before setting up the RPC one.

It also puts a limit on communication inside a kernel - because kernel will have to support a well-balanced pool of threads to process its own messages, asynchronously (oops).

Actually, considering how simplistic the kernel functionality will be (it is essentially about managing the thread and process abstractions and having processes communicate with each other), I think about doing user process->kernel calls using the traditional system call methodology, though RPC would be used for kernel->process calls and process->process calls. After all, RPC calls themselves have to be implemented in some way.

I speak in terms of setup overhead when used at the very low level.

Sure, but this happens only once isn't it ?

This is certainly a signature changing modification, from "malloc" to "malloc_kb" for example. Otherwise you risk semantic inconsistency. To your attempts to counter-example I'll just say that I put api version into the signature and that kinda solves it (e.g. malloc_v1 can be different from malloc_v2 both in arguments and in semantics, they could also peacefully co-exist in single system).

Fair point. Keeping the same signature without keeping the same behaviour is not a behaviour that should be encouraged anyway, so I schedule this extra versioning system for later removal unless someone complains. Will make things a bit lighter.

[Neolander]

That makes sense. I forgot to mention that my VFS is not contained within a server, but is instead distributed across all drivers, which sort of makes a big difference.

Indeed, that's quite a different structure. The idea of distributing VFS management across drivers is interesting, though I wonder how it works in practice when an applications wants to access the VFS : who does the application contact ?

But what if you want to control a specific driver? Let's say I wanted to tell my graphics driver to change it's resolution, which is clearly an interface that not all drivers implement, but one that more than one driver could at a time. How would I do that? It seems like making that type of request go through the VFS would be sort of ugly.

In the same way as previously, we could imagine a unifying "display manager" process that centralizes all graphics drivers behind big system abstractions, just like the VFS does for disk drivers.

Let's assume that it is safe to say that only one graphics driver is behind each screen for the sake of explanation clarity. When an application tells the display manager that she wants to change the resolution, the display manager checks on which screen(s) the application is, which graphics driver(s) are associated to each screen, and checks if all drivers have resolution changing capability. If not, it aborts the process, if so, it broadcasts the resolution changing order to all drivers.

Here I assumed that the display manager had support for resolution changing, because it's a very common feature of graphics adapters, but I suppose you might think about other situations where the application truly wants to communicate with the driver directly, either because it's faster ("direct rendering" comes to mind) or because it wants to check for the existence of some driver-specific extensions that are not supported by the system abstractions. While definitely not a good practice, and some pretty hell to secure, I guess it should definitely be supported.

In that case, we can imagine having a remote call by which the application may ask the display manager to give it the name of the driver(s) associated to the display(s) which it's on. After that, it can contact the drivers directly and do whatever it wants with them -- within the limits of its security permissions, that is.

[Neolander]

I don't like the .(m)pkg installers, they usually mess around with the system files and in a very uncontrollable manner, here I totally agree.

The thing is, as soon as your application needs to do anything that is beyond the basic user privileges, you'll need them, if I'm not misunderstood...

[NickJohnson]

That makes sense. I forgot to mention that my VFS is not contained within a server, but is instead distributed across all drivers, which sort of makes a big difference.

Indeed, that's quite a different structure. The idea of distributing VFS management across drivers is interesting, though I wonder how it works in practice when an applications wants to access the VFS : who does the application contact ?

Each driver contains a VFS tree that contains its own files, as well as mountpoints that "point" to other drivers. If you make a request to a particular driver to find a file and it reaches a mountpoint, that driver then requests that the mounted driver find the file instead. (In reality, the mechanism is more complex to prevent the obvious confused deputy problem here, but you get the idea.) Normal processes simply keep a pointer to their root directory, which is what they always request to find files from. By "pointer" here I mean a special 64-bit integer type that uniquely identifies a file or directory on my system. This whole thing has been implemented and actually works quite well.

Here I assumed that the display manager had support for resolution changing, because it's a very common feature of graphics adapters, but I suppose you might think about other situations where the application truly wants to communicate with the driver directly, either because it's faster ("direct rendering" comes to mind) or because it wants to check for the existence of some driver-specific extensions that are not supported by the system abstractions. While definitely not a good practice, and some pretty hell to secure, I guess it should definitely be supported.

In that case, we can imagine having a remote call by which the application may ask the display manager to give it the name of the driver(s) associated to the display(s) which it's on. After that, it can contact the drivers directly and do whatever it wants with them -- within the limits of its security permissions, that is.

I basically just do that for all requests: once a process gets a hold of a file pointer (and has proper permissions) it just talks directly to the driver for everything.

[Neolander]

Each driver contains a VFS tree that contains its own files, as well as mountpoints that "point" to other drivers. If you make a request to a particular driver to find a file and it reaches a mountpoint, that driver then requests that the mounted driver find the file instead. (In reality, the mechanism is more complex to prevent the obvious confused deputy problem here, but you get the idea.) Normal processes simply keep a pointer to their root directory, which is what they always request to find files from. By "pointer" here I mean a special 64-bit integer type that uniquely identifies a file or directory on my system. This whole thing has been implemented and actually works quite well.

Indeed, provided that the normal process keeps working on its root directory, things will work out quite well, but I wonder if such transparent switching from one driver to another couldn't harm performance a bit if process start to massively work with files on another disk.

I mean, imagine that a compiler stored on disk A compiles a program on disk B. Shouldn't there be a way for the disk A driver to tell the compiler that it should contact the disk B driver, instead of constantly redirecting calls ?

I basically just do that for all requests: once a process gets a hold of a file pointer (and has proper permissions) it just talks directly to the driver for everything.

That works as long as the driver implements the toplevel abstractions that processes directly use, which seems indeed to be your design choice considering the way your VFS is managed. Might make drivers more complicated to code, though, as they have to care about other things than abstracting their own hardware. Or not. Have to think about it some more, but I'm getting tired right now as it's 1h16 here...

[Neolander]

I happens once per IPC setup, which can be multiple times depending on how initialisation sequences are grouped (e.g. there can be a chunk of setup at start and then mostly closure-maintained IPC, or setup can be on demand, and then it's not very deterministic, may cause servers to start up and so on and so on).

Indeed, this may slow down the system's and programs' initialization if lots of IPC initializations are involved, especially since I tend to lean towards the "on demand" option. Have to think about it. The problem here is my lack of real-world data : if setting up a remote call is a matter of 100 µs and there are only, like, 100 RPC initializations during system startup, then that's not a big issue (we only add a few ms to system boot). On the other hand, if setting up a remote call takes 10 ms and there are 1000 RPC initializations on system startup, this may become an issue, because now RPC initializations take several seconds.

Maybe what I should do would be to (over)estimate (by a large margin) how much RPC initializations would take place during a typical system startup, then write some dummy code that basically does work that's similar to those RPC initializations, and then check how long that code takes to complete in a slow VM.

[NickJohnson]

Each driver contains a VFS tree that contains its own files, as well as mountpoints that "point" to other drivers. If you make a request to a particular driver to find a file and it reaches a mountpoint, that driver then requests that the mounted driver find the file instead. (In reality, the mechanism is more complex to prevent the obvious confused deputy problem here, but you get the idea.) Normal processes simply keep a pointer to their root directory, which is what they always request to find files from. By "pointer" here I mean a special 64-bit integer type that uniquely identifies a file or directory on my system. This whole thing has been implemented and actually works quite well.

Indeed, provided that the normal process keeps working on its root directory, things will work out quite well, but I wonder if such transparent switching from one driver to another couldn't harm performance a bit if process start to massively work with files on another disk.

I mean, imagine that a compiler stored on disk A compiles a program on disk B. Shouldn't there be a way for the disk A driver to tell the compiler that it should contact the disk B driver, instead of constantly redirecting calls ?

Compared to the time it takes to read the file itself plus the processing time of the file, the time taken finding the file pointer in the VFS will probably be negligible. Remember that the VFS has nothing to do with actual file I/O: file I/O is done directly using file pointers, avoiding the VFS entirely. It could probably be optimized by some sort of complex client-side caching, but for now it's not a problem, and I'm currently more occupied with getting my OS to beta.

I basically just do that for all requests: once a process gets a hold of a file pointer (and has proper permissions) it just talks directly to the driver for everything.

That works as long as the driver implements the toplevel abstractions that processes directly use, which seems indeed to be your design choice considering the way your VFS is managed. Might make drivers more complicated to code, though, as they have to care about other things than abstracting their own hardware. Or not. Have to think about it some more, but I'm getting tired right now as it's 1h16 here...

You're assuming that the drivers implement these interfaces completely from scratch. There is a driver library that implements the whole VFS mechanism and core I/O (including security,) and in theory you could have additional libraries for different types of drivers. I've spent quite a while getting the interface very nice, actually: for example, my ramdisk driver is only 88 SLOC and contains only 5 functions and my read-only initrd driver is only 100 SLOC with 3 functions. Of course, these libraries can be easily bypassed by the drivers (all the way down to the message passing layer, if needed) if they want to implement their own versions.

[Neolander]

Indeed, with shared libraries it works. Shared libraries and remote call interfaces may more or less be used interchangeably, though they have different characteristics (a privilege escalation may occur at a process boundary whereas with libraries code has to be privileged enough from the start, remote calls are more costly both in code complexity and performance, it is easier to append new parameters to an existing remote call without recompiling, etc...).

Thanks for the clarification !

[Neolander]

whereas with libraries code has to be privileged enough from the start

Not at all, shared libraries may include do_in_supervisor_mode() code which checks current PL and makes a syscall if in user mode, for example.

I was talking about fully library-based implementations, because if syscalls come into play you get something that's similar to IPC in the end.