An IPC design for an microkernel

[Zoxc]

Hi,

So I'm trying to design a microkernel and I'm plotting the IPC and how to design a POSIX API on top of it.

My basic IPC would be port which you can asynchronously send and recive fixed size messages from, it would be controlled by the kernel using system calls and it could expand in size as needed. It would also allocate some shared memory which would be used to implement a userspace heap which larger data and messages can be stored. This reduces the amount of system calls and typically the larger data would be needed by the reciver for a while anyway. An unique id can optionally be allocated per message so that the kernel can block the sender and catch the returning message (which would share the same id).

Now I not sure at what level I should allocate ports, there is a few (many) possibilities:

• They could be allocated per process. This is simple, but would increase the port lock contention.
• They could be allocated per thread. This increases memory usage, but reduces the port lock contention.
• They could be allocated per object. This has the highest amount of memory usage and would require the servers to check a high amount of ports. It would make a POSIX implementation simpler however.

The first two options would allow the ports to be abstracted away and the per thread option could be viewed as just an optimization. The third option would be pretty expensive in memory usage (especially virtual memory) and it would be slow for the servers since they would have to check alot of ports for messages (which I guess could be optimized with message queue queues :X) so I'll just rule that option out.

Now for POSIX emulation I see two options, one is to implement it in a wrapper library or a wrapper service, which could help with descriptor inheritance with fork and execve. As this could depend upon the result of my process management I think I'll wait a bit with the design.

I also had an idea to have a pointer to a page of shared memory passed with a message instead of using the heap as storage. The typical message payload would be sized from 512 bytes to the size of a page which makes a page a nice medium. It would also have the benefit that you could just map the pages you'd want to swap out to the port. If this is suitable would depend on if the speed of a system call is faster than just copying the data (which could be accelerated with SSE).

I would like some feedback on the design and suggestions for improvement and I can start planning the process creation/management, memory management and page swapping...

[Hangin10]

What kind of things are you leaving in the kernel vs. userspace?

My basic IPC would be port which you can asynchronously send and receive fixed size messages from, it would be controlled by the kernel using system calls and it could expand in size as needed. It would also allocate some shared memory which would be used to implement a userspace heap which larger data and messages can be stored. This reduces the amount of system calls and typically the larger data would be needed by the receiver for a while anyway. An unique id can optionally be allocated per message so that the kernel can block the sender and catch the returning message (which would share the same id).

I think that sounds good. It's fairly similar to how I plan to do things, except my messages are variable sized (just bounded). How much does your kernel know about the messages? Does it check for correctness, or do you leave that to the apps? How do you determine who can read/write to which port?

Now I not sure at what level I should allocate ports, there is a few (many) possibilities:

• They could be allocated per process. This is simple, but would increase the port lock contention.
• They could be allocated per thread. This increases memory usage, but reduces the port lock contention.
• They could be allocated per object. This has the highest amount of memory usage and would require the servers to check a high amount of ports. It would make a POSIX implementation simpler however.

I'm not sure what you mean "per object", could you elaborate?
My IPC system allocated ports per thread. Port IDs are system global, but need to be opened by senders before they can be used (access checking happens at that point). Ports can only be read by owner thread. Driver/servers can write to any port.

The first two options would allow the ports to be abstracted away and the per thread option could be viewed as just an optimization. The third option would be pretty expensive in memory usage (especially virtual memory) and it would be slow for the servers since they would have to check alot of ports for messages (which I guess could be optimized with message queue queues :X) so I'll just rule that option out.

How can they be abstracted away?

I also had an idea to have a pointer to a page of shared memory passed with a message instead of using the heap as storage. The typical message payload would be sized from 512 bytes to the size of a page which makes a page a nice medium. It would also have the benefit that you could just map the pages you'd want to swap out to the port. If this is suitable would depend on if the speed of a system call is faster than just copying the data (which could be accelerated with SSE).

I think at the moment, my message limit is 1024 bytes, after that the sender needs to create kernel request obj which states read/write, start address, and length of data to be used by whatever the message is. The reader can then use syscalls to get at that data (my kernel reserves a whole 128MB address space for kernel request data).

I would like some feedback on the design and suggestions for improvement and I can start planning the process creation/management, memory management and page swapping...

Sounds good to me. I'd like to know more about the design of your kernel in general, what things are in the kernel vs. userspace, what your plans on driver interface are, etc.

[Zoxc]

What kind of things are you leaving in the kernel vs. userspace?

The idea was to have the message queues in the kernel and the heap in the userspace.

I think that sounds good. It's fairly similar to how I plan to do things, except my messages are variable sized (just bounded). How much does your kernel know about the messages? Does it check for correctness, or do you leave that to the apps? How do you determine who can read/write to which port?

The kernel manages the messages and delivers the data in them. It also has a mechanism to send a reply to messages and handles blocking/awakening of the sender. The two processes associated with each port is the only ones which can send/receive data, each at their respective side.

I'm not sure what you mean "per object", could you elaborate?

This would mean that every object, like a file, pipe or socket would have it's own port.

How can they be abstracted away?

They could be simply allocated by the underlying library since there is no state associated with then (all roads lead to Rome).

Sounds good to me. I'd like to know more about the design of your kernel in general, what things are in the kernel vs. userspace, what your plans on driver interface are, etc.

I will have to get back to you on that

Also I had an idea to queue asynchronous system calls in a kernel aware per thread structure. This queue would be flushed in the next kernel transition which could be caused by a blocking (synchronous) action, a task switch or that queue filling up. This could improve the performance alot on systems with expensive system calls and possibly allow me to move the heap into kernel space.