OSDev.org

Message IPC can be used for piping data to different destinations and I'm currently wondering if there is a just as good way for doing this without mailboxes (or ports). Mailboxes are very well suited for this purpose as you can give these mailboxes around without the application knowing or care about it. If Message IPC is designed so that only threads can be addressed, changing where data should be received or sent for me seems to be more difficult.

Problem is that mailboxes doesn't fit very well with my programming model and I wonder if there is another way that gives you almost the same versatility as mailboxes. Mailboxes will be an extra primitive in the OS to care about and it will do the programming model more complex. Proxy threads is what first comes in my mind however, should the message be buffered or not? If buffered they become a mailbox. I'm undecided.

In my system, messages are only really addressed to processes. However, each message also contains an "index" field, which along with the PID forms a 64 bit unsigned integer "resource pointer" that uniquely refers to a resource (like a file, pipe, device, etc.) Resource pointers are then used like file descriptors, but in a global fashion. I suppose this is sort of like having a mailbox for each resource, although all queueing/event handling is done by the receiving process instead of the kernel or a central server, and almost all processes (other than drivers) only have a single resource.

NickJohnson wrote:In my system, messages are only really addressed to processes. However, each message also contains an "index" field, which along with the PID forms a 64 bit unsigned integer "resource pointer" that uniquely refers to a resource (like a file, pipe, device, etc.) Resource pointers are then used like file descriptors, but in a global fashion. I suppose this is sort of like having a mailbox for each resource, although all queueing/event handling is done by the receiving process instead of the kernel or a central server, and almost all processes (other than drivers) only have a single resource.

Does this mean that you have one thread that gets all messages and then will spread those around in the process to the threads where they belong?

NickJohnson wrote:In my system, messages are only really addressed to processes. However, each message also contains an "index" field, which along with the PID forms a 64 bit unsigned integer "resource pointer" that uniquely refers to a resource (like a file, pipe, device, etc.) Resource pointers are then used like file descriptors, but in a global fashion. I suppose this is sort of like having a mailbox for each resource, although all queueing/event handling is done by the receiving process instead of the kernel or a central server, and almost all processes (other than drivers) only have a single resource.

My IPC is thread-oriented. A thread defines the "mailbox" by a resource name, and then other threads can access it by the use of this name. This mechanism also works transparently over IP-networks. Buffering occurs both at the receiver-side, and at the sender side (reply). On the local machine, messages are always page-aligned, and are copied from sender to receiver (and receiver to sender) by copying page-tables. This created quite some problems with TLB-flushing on SMP systems, but it works ok now.

The IPC model I'm currently implementing is channel-based: Processes can register somthing like TCP/IP servers other processes can connect to, creating a channel. Each of the two endpoints of the channel can define the address for a message handler callback. When a message is received by a process, the kernel creates a new thread that starts at the address of the callback handler and copies/moves the message's contents to the thread's stack. Message handlers can decide to explicitly respond to the message using a system call or to send an implicit null-response, when they return or are stopped.

When sending a message, onecan decide on how responses should be delived: Discard disposes response messages, RPC lets the send system call block and the response message is copied/moved to the invoker's thread and normal mode delivers the response message using the channel, with the call type set to discard to present responses to the response to be sent.

When channels are closed, a termination message is sent to the other endpoint to allow for auto-cleanup. Messages are routed in an L4 manner, with the difference that sibling processes must communicate through their chief.

As this approach is channel-based it's not exactly without-'mailboxes', but it might give you another idea on how IPC can also be implemented.

OSwhatever wrote:

NickJohnson wrote:...

Does this mean that you have one thread that gets all messages and then will spread those around in the process to the threads where they belong?

Actually, each message received spawns a new thread, which then either handles it or puts it in a queue and exits. The former is more common, because it is faster and lower-latency. You can't address a particular thread with a message explicitly, only a resource, but it would be simple to make a relation between thread IDs and resource indices if you wanted to address threads. In this way I suppose it is a lot like mailboxes.

What I understand is the both Farok and NickJohnson are using a model similar to mine basically new threads runs in response to messages. I implement a thread pool model in kernel space. Currently there are threads, jobs and processes. Threads is a context where code is executed. Jobs are basically a start address which can start when the job receives a message or explicitly started by a start job system call. You can send a message to either a thread or a job. If it is a thread the message is first received when the thread does a receive system call.

In most cases you send messages to jobs which schedules a new thread to be executed. The thread then responds to the message that started the job. Simple and works well most of the time but it requires a client server model where the client must register to the server in order to subscribe get asynchronous messages. Not very convenient if you want relay data around.

I'm thinking about creating another primitive which is basically a single ended mailbox which buffers messages. Threads and jobs get automatically a mailbox when created but for jobs an external mailbox can be attached to it. This leads to a weird case where a job suddenly have two mailboxes. You can always let the other have higher priority than the other but still I'm not really convinced. Or you explicitly choose a mailbox when the job is created, otherwise it will get a default mailbox. Send a message to the job or the mailbox will in this case perform the same task.

It's quite difficult to create intuitive programming models that at the same time have maximum flexibility. Mach have good flexibility and abstraction as messages can only be sent to ports, however this limitation does not fit very well with the thread pool model otherwise I would have chosen that right away.