I've been gradually refining the design I want to use for my OS project, and I thought I'd post some details here and hopefully get some constructive feedback.
My current plan is to use a microkernel design, but it won't be rabidly "micro"; for example, I'm not going to split out memory management or scheduling. IPC will be with fixed-length messages, and longer messages will be sent using shared memory.
I'm planning on implementing both synchronous and asynchronous message passing. I was originally going to build the synchronous layer on top of the asynchronous one, but that leads to issues with out-of-order responses that could be solved in application code but would cause extra message copying and memory allocation in userspace programs. I haven't completely figured out the rules I'll use for the task switching that message passing requires, but I'm kind of hoping that threads won't have to completely give up their timeslices every time they pass a message.
There will be a relatively heavy emphasis on multithreading; since threads must make a system call to block until a message is ready, each process needs a thread that is devoted to message handling so it can process critical messages in a timely manner. The C build environment will automagically set the first thread to spawn off main() in a new thread and start listening for messages, so nothing fancy should be needed for basic C programs.
There will also be strong support for TLS; I'm planning on having a special section/segment in the ELF files rather than implementing it with syscalls; that way, TLS will feel like less of a second-class citizen, and it will fit very well with the threading model of D, which I'm hoping to use for a good portion of the userspace.
I'm thinking about a method to reduce the number of context switches needed for driver calls; what I might do is keep driver processes constantly mapped somewhere in memory, which means that they would need to be position-independent. Unfortunately, I'm currently working with a 32-bit machine (an ARM), so I'm not sure if I'll have enough virtual address space to keep all the drivers mapped. Of course, on an ARM machine, I'll probably run out of RAM long before I run out of addresses...
Another issue with keeping the drivers mapped is that the individual threads would probably be statically mapped as well, which means that there could be addressing/pointer issues when passing data between threads that reside at different addresses, but I suppose that some PIC techniques could sort that out.
I'm currently working on the kernel memory management, and I still can't decide whether I want to support multiple page sizes; the ARM MMU offers a tree of nested pages, but managing that will require more than a simple linked-list allocator, which will in turn slow the machine down, and support for the different page sizes will be useless if I ever port the kernel to a system with only one page size.
Anyway, that's my brain dump for the day. If you have any suggestions, I'd love to hear them.