Difficult things to do in a microkernel?

[rod]

Which things can be difficult to do in a microkernel with respect to a monolithic kernel?
I am not asking about performance or efficiency considerations, but about operations that the microkernel design makes difficult to do them correctly.
I can think of devices which are sensitive to timing, and some operations should be done in a sequence without interruption. In a monolithic kernel this could be solved by temporarily disabling interruptions. In a microkernel, when a driver thread wants to read/write from/to some device's memory or ports, I think that this could be solved by temporarily disabling preemption for that thread (if timer interruptions are still acceptable).
Any more issues you can think of? And... how can they be solved?

[nullplan]

Memory transfers between servers and normal programs come to mind. Say, you wish to open a file. In a monolithic kernel, you tell the kernel which file to open, and the kernel can just copy the path into kernel space, because the userspace is still mapped. In a microkernel, the open() system call is going to be handled by the VFS server (or else, to paraphrase Andy Tanenbaum, it isn't a microkernel), which runs in userspace, but in a different address space from the calling program. So now the VFS server has to ask the kernel to transfer the memory. So how are you going to do that, then? I see some options, most of them bad. At least in 64-bit mode you can cheat by mapping all physical memory into kernel space, so for such a copy you only have to look up the physical address of the source buffer, and copy from the kernel-space version of the physical memory.

Another problem that was already discussed before is the issue of process priorities. In a µK, you often have one server that handles PCI (enumerates devices and provides access to config space), and then servers to handle the devices on the bus. And how are you going to prioritize these processes relative to each other? Many choices, easy to get wrong.

[rod]

Memory transfers between servers and normal programs come to mind. Say, you wish to open a file. In a monolithic kernel, you tell the kernel which file to open, and the kernel can just copy the path into kernel space, because the userspace is still mapped. In a microkernel, the open() system call is going to be handled by the VFS server (or else, to paraphrase Andy Tanenbaum, it isn't a microkernel), which runs in userspace, but in a different address space from the calling program. So now the VFS server has to ask the kernel to transfer the memory. So how are you going to do that, then? I see some options, most of them bad. At least in 64-bit mode you can cheat by mapping all physical memory into kernel space, so for such a copy you only have to look up the physical address of the source buffer, and copy from the kernel-space version of the physical memory.

Another option that I see is (specially if there is much data to copy) to temporarily map, in kernel space, some level of the page tables of the the process that is not mapped, and copy between that place and the process that is already mapped.

[Korona]

I have not encountered devices that are so sensitive to timing that my microkernel (GitHub: managarm) cannot handle them. Most modern devices require you to set up structures describing operations in system memory before submitting those operation - the submission itself is only a single port or MMIO write and thus not sensitive to timing.

Another problem that was already discussed before is the issue of process priorities. In a µK, you often have one server that handles PCI (enumerates devices and provides access to config space), and then servers to handle the devices on the bus. And how are you going to prioritize these processes relative to each other? Many choices, easy to get wrong.

My personal experience here is that priorities without priority inheritance just do not work in situations like that and decrease performance and latency.

---

My two cents on the OP's question:

In general, microkernel designs tend to be more difficult because they have to do strictly more than monolithic kernel: In addition to everything that a monolithic kernel has to do, they need to take communication into account. So, while it is not necessarily the case that a single aspect will be much more complex than in a monolithic kernel, everything will be a bit more complex.

Furthermore, a main difficulty that I often encounter is how to make things fast. Thinking about how the VFS/FS interaction can work is a matter of minutes. Designing it in a way that is fast takes a lot of time.

---

As the OP asked for one particular difficulty: I found race-free handling of IRQs in userspace drivers to be somewhat complicated. I have scrapped at least 3 or so designs before I came up with the current one in managarm. It is difficult because the way that IRQ handling usually works (call a handler, do some work, perform EOI) does not work in an asynchronous design. Of course, monolithic kernels do not encounter this problem: they can just run handlers synchronously (and maybe delegate additional work to a thread after EOI). managarm however, cannot easily do that: it asynchronously delegates the handling to a userspace thread. If you're curious how it works, I'll scratch the design below but it is somewhat technical.

There are many complications: handling differs based on the trigger mode (level- vs. edge-triggered), IRQs can be shared, in some situations drivers might be unable to determine if an IRQ originated from their device (!) (mostly after a device reset), mask and unmask operations can race against IRQ delivery, devices need to be able to asynchronously ACK and NAK (!) IRQs etc. The way this is now solved in managarm involves managing multiple sequence numbers that are matched against each other to determine if some actions (like mask IRQ, unmask IRQ, notify userspace) need to be taken: the sequence number of the hardware IRQ (incremented in each ISR) and, derived from that, the sequence number of the IRQ that was seen last by the driver as well as the sequence number of the IRQ that the driver tried to ACK/NAK last. The latter might differ from the sequence number that was actually ACKed/NAKed last because another IRQ might have happened in the case of shared IRQs (in which case a NAK must be ignored but a ACK must still unmask the IRQ). While this interface sounds complicated, it's quite nice to userspace - most of the complexity is handled in the kernel and drivers only have to do

Code: Select all

uint64_t seq = 0;
// [...]
seq = wait_for_irq(some_irq_capability, seq);
if(from_my_device()) {
    ack_irq(seq);
    // Handle the IRQ.
}else{
    nak_irq(seq);
}

Using this system, managarm can reliably (i) handle IRQs asynchronously in all situations, (2) detect IRQs that are NAKed by all drivers and provide debugging information if that happens and (3) detect if drivers fail to either ACK or NAK IRQs in time which would either decrease I/O performance or lead to system lockup (if IRQs are not ACKed at all).

[OSwhatever]

Which things can be difficult to do in a microkernel with respect to a monolithic kernel?
I am not asking about performance or efficiency considerations, but about operations that the microkernel design makes difficult to do them correctly.
I can think of devices which are sensitive to timing, and some operations should be done in a sequence without interruption. In a monolithic kernel this could be solved by temporarily disabling interruptions. In a microkernel, when a driver thread wants to read/write from/to some device's memory or ports, I think that this could be solved by temporarily disabling preemption for that thread (if timer interruptions are still acceptable).
Any more issues you can think of? And... how can they be solved?

In general, microkernels are more challenging. Not having the ability to temporarily disable interrupts in user space is a serious limitation in my opinion. Not much because of HW access but more because of concurrent access to data structures, like lists or trees that aren't lock less. This often leads to more use of lock less algorithms but that isn't always possible. I previously suggested that CPU designers should allow disabling interrupts in user space for X cycles ahead, just like you do in kernel mode. if X cycles are exceeded an exception will occur. Instead we have to create incredibly complicated things like futexes when just that simple CPU addition could help a lot. Keep in mind you usually need both exclusive access and disabling preemption in a critical section which means you need some barrier to prevent other CPU executing the critical section when on CPU is in it. This usually involves kernel calls which just slows things down.

Not really much of a microkernel vs monolithic kernel problem but more a generic problem which is not that easy to address.

[pharos]

Hi,

Just to give my 2 cents, Pharos RTOS is more or less developed in the microkernel philosophy. However, it does allow for a partition to disable interrupts in very specific contexts. In short, the application is statically configured which means that it defines a function which will be executed in user mode and with interrupts disabled. As soon as the function ends the system enables the interrupts.

The advantage of this is that since the application is statically configured, the verifier of the system only needs to check specific portions of code that might damage the system. These functions that execute with interrupts disabled are one of these pieces of code. Typically these are rather small and simple pieces of code so it shouldn't be too hard to make sure they execute fast enough not to damage the rest of the application.

Of course, this is a method that only works with statically configured applications. It does not make sense for an OS like windows or Linux.

Hope it helps,
Pedro

[Korona]

Did you implement such a verifier? If yes, what techniques does it use? SMT solving?

[pharos]

Hi,

When I was referring to a verifier I meant a person, not a tool. So no, we have not created such a tool.

[fano1]

A probably way to make a microkernel easily is to "merge" it with the concept of "language based OS".

Cosmos in its current incarnation is probably the reverse of a microkernel that is a unikernel that permit to write "Application OS" in C# / VB:Net or F#.

Nevertheless some concepts are interesting for a microkernel too:

1. In Cosmos does not exists the concept of user space or kernel space code: all code runs in a FAT memory space, the memory isolation is guaranteed by the language itself (CLI - Common Language Infrastructure also called .NET)
2. There will be really no processes but only threads
3. The point 2 imply that the most important aspect of a microkernel that is message passing could be implemented in a special way that does effectively permit to "share" the Message without need to copy it. I say more as in a "language based OS" you control the compiler and could invent all sort of crazy optimization to do message passing efficiently
4. Verifiers in software as the ones @pharos was talking controlling the compiler can be done. For example we have the concepts of rings for example there is a "user ring" (but this does not imply that the code runs in user space!) and code originating from user ring cannot ever be unsafe code. This prohibition happens at compiler level cannot be more safe than this!