Microkernel and driver resource access.

[sandras]

So I was thinking about how would one make drivers in a microkernel system get access to the hardware resources. And I can't seem to find a nice solution for that. The only thing I can think of is that the kernel has a list of what kind of driver needs what resources, and when a privileged process registers itself as a certain kind of driver, the kernel gives it corresponding resources. If another process tries to register itself as a driver that already runs, it simply gets an error. What I don't like about this, is that the kernel has to have some sort of list of drivers, and resources, that should be allocated to those drivers - it would not be possible to load a driver which kernel has no data about. What other ways are there?

[Combuster]

The only thing I can think of is that the kernel has a list of what kind of driver needs what resources

There are too many devices existing, and the actual resources required depend on all the other hardware in the system. So any design based on a predefined list is a gross design failure at any rate.

So instead you'll have to get a list of devices and their resources dynamically. You load a few bus drivers that can tell you what other devices exist and what resources they need. When you have that list, you can go pair driver instances with those resources. Any driver will not be useful without a list of devices it can operate on so it might need to provide just that. That means the moment you add a driver you also get a list of supported devices to come with it.

[OSwhatever]

Actually a microkernel doesn't need to know about device driver classes or anything like that. The microkernel only provides resources like interrupts and IO memory or ports and that's it. Usually it is the first driver who asks for a resource that gets it, if some other driver tries to obtain it later then the kernel will not allow it.

When the driver in user space has obtained the required IO memory and interrupts it can really do what it wants to with those resources. The driver now owns the IO memory and interrupts in question and can also delegate that resource further to other drivers in the chain.

If you want to create a driver system with classes and hierarchy, that should be done in as a server(s) in user space. This alone is actually quite a complex task as there is myriad of different driver classes and associated services.

[gerryg400]

This is all IMHO, not everyone agrees.

A microkernel doesn't have to have any idea about drivers, driver classes or I/O space. It's only really interested in IPC and scheduling. BTW, the only reason that the kernel needs to know about interrupts is not so much because interrupts are a system resource but more because they are closely related to scheduling. Microkernels don't manage hardware or hardware resources.

You do need to manage hardware resources somehow however. This can easily be done in a trusted process, perhaps one for PCI, one for USB etc. If you have a microkernel OS then your OS will comprise a trusted microkernel, a set of trusted system processes and a set of possibly trusted device drivers. The level of trust you need to grant to the components depends on the hardware. On some hardware you can use IO maps to limit the hardware access of driver processes but this is not essential to building a reliable microkernel system.

On a PC it makes sense to stop applications from accessing IO space. This is easily accomplished by keeping processes in ring 3 and not giving them any IO privilege. If a process wants to become a device driver it could ask the kernel to raise its privilege level or to lower the IO privilege for that process to if you really want to be fancy to grant access to a portion of the IO map.

In my OS, the question of whether the kernel should grant IO access to a particular process is answered by the system administrator not by the kernel. Only processes started by the system administrator (root) can ask to have there privilege altered.

[Brendan]

Hi,

For me; there's a special process called "device manager" which is responsible for scanning PCI buses, etc, that is also responsible for starting device drivers. When device manager starts a driver, it tells the kernel which IO ports, IRQs and memory mapped IO areas the driver should be allowed to access. The kernel keeps track of which processes/drivers have been given access to which IO ports, IRQs and memory mapped IO areas and the kernel prevents processes from accessing anything they don't have permission to touch. Drivers ask the kernel "what am I allowed to access" to determine where the device's resources are.

The kernel may lie to drivers. For example, the kernel could tell several drivers that their devices use "IO port 0x0000", where the kernel translates IO port accesses into the device's actual IO ports without the driver knowing what they really are. This allows the device manager to change the IO ports a device uses without the drivers knowing or caring about the change. In a similar way the kernel may lie about IRQs so that the driver doesn't need to know/care about actual IRQ numbers.

Devices may not actually exist. For example, you might have a floppy controller driver that thinks it's using real IO ports and real IRQs, where the kernel is actually forwarding everything to/from a process that emulates the floppy controller hardware.

Also, virtual machines can be given access to raw devices. For example, you could tell device manager that a PCI device should be reserved for virtual machines, and then allow something like Bochs to have direct access to the real device. Of course the virtual machine wouldn't know if the device it's been given raw access to is real or if the kernel is using some other process to emulate that device. If you think about this; you'll see that you could implement a virtual machine/s that don't need to emulate most hardware, but (without knowing it) the virtual machines can use generic/external processes that emulate various devices.

Basically, this creates a "mix & match" environment, where the host OS can use real and emulated devices, and guest OSs inside virtual machines can use real or emulated devices, and anyone could write "device emulation processes" that could be used anywhere.

Unfortunately, there is one problem. A normal process can emulate a device that uses memory mapped IO, but it's very hard for the kernel to intercept a device driver's memory mapped IO accesses and forward them to a process that is emulating the memory mapped IO area. I'm still trying to find a good solution to this (so that it isn't limited to devices that don't use memory mapped IO).


Cheers,

Brendan

[turdus]

So I was thinking about how would one make drivers in a microkernel system get access to the hardware resources.

Not sure what you mean by that, but here's how I do it at low level:
1. I can differentiate between normal executables and drivers
2. If a driver type executable is loaded, the microkernel modifies IOPL to 3 (so it will be able to use io address space)
3. Any other resource is protected by ACLs. First thing every syscall do is, check the caller thread's ACLs if it has right for the required call with the given parameters. For example, you can mmap firmware memory (reported as type 2 by E820) if you have read or write access in "mmiomap" ACL. Another example: you can only mmap the LFB if you have read/write access to "vbelfb" ACL. You can create files under /dev if you have write access in "driver" ACL. And so on.

If you mean handling devices at high level,
1. I have a dummy ACPI parser in the microkernel that's been called on boot. It looks for buses in ACPI tables, and starts driver threads for them (like PCI)
2. the bus driver iterates on it's bus to discover devices
3. if a device is found, it's registered in filesystem server. There's a simple reason for this, every device appears in /dev.
Later, if an application want's to talk to a device, it opens it's /dev file. The server dispatches subsequent read/write operations to the associated driver thread(s) (also taking care of mutual exclusion, so device drivers don't have to worry at all). This is the same for IRQs. If you want to create an "ISR" in a device driver, you'll have to:
1. open device file (/dev/irq/1 for example)
2. assign a method for parsing incoming messages on returned file descriptor
When the appropriate IRQ fires, the low level microkernel code sends a message to the filesystem server. The fs server then looks up what threads keep the according /dev/irq/x file open, and sends IRQ event message to all of them (and awake them). Finally threads with pending messages are scheduled.
Yes, it introduce an overhead to inline kernel-mode ISRs, but keeps the design consistent and easy understable.

[OSwhatever]

BTW one problems that is arising when it comes to driver in user space is handling of critical regions. How do you handle this. Do you allow your drivers in user space to disable the interrupts for the CPU or have you added some kind of API?

[turdus]

BTW one problems that is arising when it comes to driver in user space is handling of critical regions. How do you handle this. Do you allow your drivers in user space to disable the interrupts for the CPU or have you added some kind of API?

In kernel:
Neither. I do not disable interrupts (except in ISRs for a short time) and I do not provide API either. What I do is a bitlock to prevent scheduler from preemption if something important is being done (for example when creating mapping structures for a thread, if scheduler switches away, it won't be able to switch back because address space is not ready). I do not stop ISRs from executing, only the preemption is locked. Also note that my ISRs are very small and fast, they just send a message, nothing device specific.

In userspace:
I have exactly 1 thread for 1 device (allowed to access the resource directly). If more threads wants to access this driver thread, no problem, because IPC messages are serialized and queued, they will be served in order.

As long as only process variables (common to all thread of the same process) or shared memory (global to all) concerned, I have a critical section construct in my language that's translated to a simple mutex test-and-set (and yield if locked). Similar to C#:

Code: Select all

critical {                 //hidden lock
  // do something with protected memory
  a+=1;
}                            //hidden unlock

The variables used for locking are shared and allocated by the system. If you specify a bitindex in "critical(x){", that particular lock will be used, otherwise one will be calculated based on source and line. (This may change in the future as implicit index (named after the resource) would make the source more readable, like "critical(rtclock)")
Currently I only have a very dummy malloc/free implementation, but I'm going to rewrite it in a lock-free way.

[sandras]

What about the kernel API? How do you speak with the kernel? How do you tell him to give certain resources to a certain process? Of course, it can be done, but, again, I can't seem to think of a nice solution. You need to allocate memory mapped IO, IO ports, interrupts. Most architectures don't have IO ports. On x86, DMA can be anywhere between 0 and 16mb (except where something get's in the way), aligned at 64kb boundary. I think that's different on other architectures. Can there be an architecture independent way of asking for resources? Or do you just make it different for every architecture?

[sandras]

What I did not include in the last post:

In case of DMA, you could provide either specific get_dma() call, or get_memory(uint lower_limit, uint upper_limit, uint size, uint alignment_size). Both are too nasty.

[gerryg400]

If you have mmap you can add some extra flags. Linux has lots of extra flags (MAP_32BIT, MAP_STACK) but actually a microkernel needs more options because much of the system is in user space. I have MAP_PHYS, MAP_DMA and MAP_COMMIT available for device drivers.

So for example my console driver does this.

Code: Select all

    vidmem = mmap(0, 8192, PROT_READ | PROT_WRITE, MAP_PHYS | MAP_PRIVATE, -1, 0xb8000);

My preference is to use a standard interface and modify it in a (hopefully) standard-complient way to support the extras that I need.

[sandras]

Hmm, it seems much nicer/cleaner to have some flags for an existing system call than having a complete new system call for these kind of special cases. After all if you already have flags for, say, read/write permissions, you might as well add architecture specific ones. I wouldn't like it only if every architecture would have to have flags argument for a certain system call just because another architecture has them.

Thanks for the replies everyone. : )

[turdus]

What about the kernel API? How do you speak with the kernel? How do you tell him to give certain resources to a certain process?

Again, microkernel like. So the answer: messages, messages, messages.

Of course, it can be done, but, again, I can't seem to think of a nice solution. You need to allocate memory mapped IO, IO ports, interrupts. Most architectures don't have IO ports.

And? What's the matter? It's just another namespace. Some computer has ISA namespace, others have SCSI. Some happened to have IO namespace too.

On x86, DMA can be anywhere between 0 and 16mb (except where something get's in the way), aligned at 64kb boundary. I think that's different on other architectures. Can there be an architecture independent way of asking for resources? Or do you just make it different for every architecture?

No. What kernels are for, to hide these details and provide a consistent environment. A device driver does not need to know where the DMA area is. Enough to know where it's mapped in its virtual address space.

[OSwhatever]

Again, microkernel like. So the answer: messages, messages, messages.

Why messages to kernel? I understand that we use messages to services in user space but to the kernel normal call gates can be used. I know some kernels use messages as kernel interface but not really understood the benefit.

[gerryg400]

Again, microkernel like. So the answer: messages, messages, messages.

Why messages to kernel? I understand that we use messages to services in user space but to the kernel normal call gates can be used. I know some kernels use messages as kernel interface but not really understood the benefit.

Maybe not the kernel. Lot's of microkernels have mm in user space, My mmap sends a message to the memory manager process and blocks waiting for a reply.

[Edit] OSwhatever, one benefit of using a messaging interface to your kernel is that it can be asynchronous. Your app can call mmap (or equivalent) and continue on doing stuff until the reply comes back.