I have thought about two modes of implementing IPC for message passing in a microkernel: client-server and point-to-point.
Client-Server would be like: The processes would have ports (also called server descriptors), and client descriptors (intergers actually in userspace). Each client descriptor points to a port (or server descriptor) on some process. This way a server can have several ports, and each of them would be reachable from a number of clients maybe in different processes. It is like a multipoint-to-point structure. This way fork() would need, in the client side, to simply copy or duplicate each client descriptor so that it points to the same server as the parent process, and the server would not need to know it or be notified.
In Point-to-Point there would be only one type of descriptor (also integers in userspace): each process would have a number of descriptors and each one would point exactly to other descriptor of potentially other process. And that other descriptor would point back only to the one descriptor that points to it. So they are symmetrically connected. Each descriptor reaches 1 other descriptor and is reachable by 1 other descriptor. This way any end point could act as a client or as a server indifferently. It is like a point-to-point structure. fork() would need to duplicate the parent's descriptor to the child, and the other end would have to be notified and allocate a new descriptor that would be connected with the child's duplicated descriptor.
In my design, I started with a point-to-point approach because it seemed to me more flexible and general, but after hitting some problems, I considered the client-server approach. That was because I thought that as the Operating System is structured like a stack of protocols and services, and has different levels or types of entities (kernel - drivers - servers - normal processes), so the main direct communications would be between processes of different level or type, and they would not be peers, instead ones would provide services (servers) for others (clients). Also, as the 'normal' processes would generally only request services through libc, they would not need any server functionality, so it would be a waste or resources to assign them potentially unused server infastructures. Also, I think that in point-to-point the system would need a greater total number of descriptors, so client-server might be more efficient.
So, any thoughts about this matter? Do you think that the client-server model is the most adequate for an Operating System's IPC mechanism? Any other model?
Microkernels and IPC: client-server or point-to-point?
-
OSwhatever
- Member
- Posts: 595
- Joined: Mon Jul 05, 2010 4:15 pm
Re: Microkernels and IPC: client-server or point-to-point?
I think you should do whatever fits your beliefs. There is no right or wrong here and you already said that you discovered some problems with the point-to-point design, which is good because that's how you learn.
What I found interesting here is that you mentioned fork(). My OS has an even more agnostic design, the kernel basically doesn't care about who talks to who and no kernel primitives are involved for the IPC, similar to L4. My idea was to keep the kernel simple and then let user space take of "who is allowed to talk to who". Basically the user processes themselves are responsible for rejecting or accepting an IPC messages. However, this approach doesn't fit well with fork() as the resources should be inherited and a service doesn't magically know that a child has been created. Of course this can be solved but it could be slow and complicated. QNX solved this by having having kernel controlled ports which are used for the IPC and therefore fork() works quite fast.
For me though fork() is not that important. Sure it can do some nice tricks but I don't find fork() that important anyway and have accepted it is not high on my priority list and neither is POSIX. It's a matter what your goals are. For commercial operating systems this might be more important as it could be a selling point but for hobbyists it might not.
Another thing is that I don't find client-server and point-to-point to be a contradiction. It's a matter to find a service and then you can assign a point-to-point communication for the service in question, like with a TCP connection. You just mentioned two ways to implement it or am I misunderstanding this?
Not sure if this helps but I found fork() to be the kind the hurdle here. In short for fast fork() I think the IPC "port" must be kernel controlled. Anyone who has more information regarding this if it is done entirely in user space? Anyway, interesting question.
What I found interesting here is that you mentioned fork(). My OS has an even more agnostic design, the kernel basically doesn't care about who talks to who and no kernel primitives are involved for the IPC, similar to L4. My idea was to keep the kernel simple and then let user space take of "who is allowed to talk to who". Basically the user processes themselves are responsible for rejecting or accepting an IPC messages. However, this approach doesn't fit well with fork() as the resources should be inherited and a service doesn't magically know that a child has been created. Of course this can be solved but it could be slow and complicated. QNX solved this by having having kernel controlled ports which are used for the IPC and therefore fork() works quite fast.
For me though fork() is not that important. Sure it can do some nice tricks but I don't find fork() that important anyway and have accepted it is not high on my priority list and neither is POSIX. It's a matter what your goals are. For commercial operating systems this might be more important as it could be a selling point but for hobbyists it might not.
Another thing is that I don't find client-server and point-to-point to be a contradiction. It's a matter to find a service and then you can assign a point-to-point communication for the service in question, like with a TCP connection. You just mentioned two ways to implement it or am I misunderstanding this?
Not sure if this helps but I found fork() to be the kind the hurdle here. In short for fast fork() I think the IPC "port" must be kernel controlled. Anyone who has more information regarding this if it is done entirely in user space? Anyway, interesting question.
Re: Microkernels and IPC: client-server or point-to-point?
There's also the mailbox model. It is very similar to the client-server model, but there the ports are known to all processes and communication is stateless, so that no local descriptors needed. This means you can fork() without caring about your IPC implementation.
Unline L4, in my OS the kernel does provide IPC primitives (actually my syscall interface is nothing more), and that's the only way to communicate between processes. It also forces a security policy on the system by checking the messages before they are delivered. Some of these checks are hardwired (for example only device drivers can send an IRQ acknowledged message), others are checked against an ACL.
Cheers,
bzt
Unline L4, in my OS the kernel does provide IPC primitives (actually my syscall interface is nothing more), and that's the only way to communicate between processes. It also forces a security policy on the system by checking the messages before they are delivered. Some of these checks are hardwired (for example only device drivers can send an IRQ acknowledged message), others are checked against an ACL.
Cheers,
bzt
Re: Microkernels and IPC: client-server or point-to-point?
Thanks for your replies!
I have read a bit about L4. It seems that the endpoints it uses are thread identifiers, so I think it is "multipoint-to-point", as several threads can send messages to a given thread.
In my design, the processes have client endpoint descriptors and server endpoint descriptors, which are numbers ("ints" or "longs") in userspace, but in kernelspace they have some corresponding data structure for storing the state (flags, queues, etc.) of each one. Each thread can listen to a server endpoint, or to a group of them.
About fork(): at the moment I think I can implement something similar to UNIX/POSIX, as after a process does fork(), the child process has the same client descriptors as the parent, so if the process had some open file (represented in its side by a client descriptor), then parent and child would end up sharing the same server endpoint, so they would share the open file and its offset, etc., as UNIX does.
The problem I am seeing now is that as each process' fd (file descriptor) is a client descriptor, and each client descriptor points to a server structure, I need a server structure for each open file. So if each process has 1024 open files, and there are 1024 processes, then the file server or servers would total 1024*1024 server structures (assuming there are no shared ones like the ones inherited by fork()). And the server structures are about 96 bytes each, so I am thinking about reducing memory consumption.
I agree. As I see now, the client-server model could also be implemented with a point-to-point structure. Maybe I should have called the two alternatives as point-to-point and multipoint-to-point.OSwhatever wrote: Another thing is that I don't find client-server and point-to-point to be a contradiction. It's a matter to find a service and then you can assign a point-to-point communication for the service in question, like with a TCP connection. You just mentioned two ways to implement it or am I misunderstanding this?
I have read a bit about L4. It seems that the endpoints it uses are thread identifiers, so I think it is "multipoint-to-point", as several threads can send messages to a given thread.
In my design, the processes have client endpoint descriptors and server endpoint descriptors, which are numbers ("ints" or "longs") in userspace, but in kernelspace they have some corresponding data structure for storing the state (flags, queues, etc.) of each one. Each thread can listen to a server endpoint, or to a group of them.
About fork(): at the moment I think I can implement something similar to UNIX/POSIX, as after a process does fork(), the child process has the same client descriptors as the parent, so if the process had some open file (represented in its side by a client descriptor), then parent and child would end up sharing the same server endpoint, so they would share the open file and its offset, etc., as UNIX does.
The problem I am seeing now is that as each process' fd (file descriptor) is a client descriptor, and each client descriptor points to a server structure, I need a server structure for each open file. So if each process has 1024 open files, and there are 1024 processes, then the file server or servers would total 1024*1024 server structures (assuming there are no shared ones like the ones inherited by fork()). And the server structures are about 96 bytes each, so I am thinking about reducing memory consumption.
Re: Microkernels and IPC: client-server or point-to-point?
Happy New Year to you!rod wrote:The problem I am seeing now is that as each process' fd (file descriptor) is a client descriptor, and each client descriptor points to a server structure, I need a server structure for each open file. So if each process has 1024 open files, and there are 1024 processes, then the file server or servers would total 1024*1024 server structures (assuming there are no shared ones like the ones inherited by fork()). And the server structures are about 96 bytes each, so I am thinking about reducing memory consumption.
It seems you have reached the level which we started to discuss in another topic. viewtopic.php?f=1&t=33381
I don't want to tell you how to do this, because there are several good solutions. Instead I've suggested there to draw boxes, and write the labels into them (process' fd, client descriptor, server structure etc.). Move those labels around until you are satisfied with your design. That means you can connect everything and you can imagine how would your design work. An advice, don't start coding until you get a clear picture of that.
Cheers,
bzt