Microkernel Driver Question

[frank]

I was just thinking about restarting my kernel and this time making it a microkernel. I rely like the added security that the separate address space adds for drivers. Well anyways I have a question about the drivers specifically. How do they do device IO? I have a few ideas. I was wondering if anyone wanted to comment on any of them.

1. Set the IOPL to 3 in the EFLAGS for the device drivers

2. Emulate the in/out instructions using GPF. This way you could only allow the driver to use certain ports

3. Make a system call for in/out instructions.

I am looking at #1 right now. It seems to be the best to me. Any other methods are also welcome.

Thanks

[speal]

There is an excellent chapter on operating system design in Andrew Tanenbaum's Modern Operating Systems (2nd edition) that covers some of the basics of microkernel design. The author's MINIX operating system serves as a basis for a lot of the "facts" about modern microkernel architectures, and would certainly be worth a look.

The general premise seems to be a simplified set of system calls: send, receive, or in some cases, send-then-block-until-reply. Drivers simply call send() and indicate that they want to send data to a port, what port number, and the data to be sent.

You could certainly design your own abstraction for the process of moving data between user and kernel privilege mode. You may want to take a look at the Object Pipelines implemented in Windows PowerShell, and CMS Pipelines in IBM mainframes. These don't directly affect user-mode drivers, but they may help in coming up with a system for message passing between the kernel and user-mode processes.

[Brendan]

Hi,

I was just thinking about restarting my kernel and this time making it a microkernel. I rely like the added security that the separate address space adds for drivers. Well anyways I have a question about the drivers specifically. How do they do device IO? I have a few ideas. I was wondering if anyone wanted to comment on any of them.

1. Set the IOPL to 3 in the EFLAGS for the device drivers[/quote]

This method doesn't make much sense to me - you choose a micro-kernel (or choose to sacrifice some performance to improve protection), but then ruin it by allowing any device driver to mess with any I/O ports it likes (including resetting the computer, trashing timers, messing up the PICs/IRQs, changing memory controller configuration, killing A20, etc).

2. Emulate the in/out instructions using GPF. This way you could only allow the driver to use certain ports

3. Make a system call for in/out instructions.

Both of these options work. Emulating the I/O port instructions isn't too hard and allows device drivers to use the I/O port instructions like normal, which is good for code size. Using system calls is faster though. It is possible to support both these methods at the same time (for e.g. so that device drivers can use I/O port instructions when code size is important, and system calls when performance is important).

Both of these methods also allow you to have "virtual I/O ports", which may be useful - a normal device driver might be talking to emulated hardware instead of real hardware, or the device driver might think it's using some I/O ports when it's actually using other I/O ports.

For example, all device drivers might be hard-coded to use I/O ports 0x0000 to N. When the sound card driver tries to access I/O port 0x0000 the kernel might actually access I/O port 0x0340 on it's behalf, and when the network card driver tries to access I/O port 0x0000 the kernel might actually access I/O port 0x02C0 on it's behalf. Also, if PCI configuration space is changed for the network card (so that the network card is reconfigured to use I/O port 0x1100 instead), then the kernel could change it's translation to suit without the network card driver caring or knowing about the change (basically, you could dynamically reconfigure resources at any time without the device driver's knowledge).

The other method is to use the I/O permission bitmap in the TSS. This isn't as flexible (you can't do I/O port virtualisation). It also increases task switch times as you need to change the I/O permission bitmap during task switches; however there is no "per access" overhead (when a device driver accesses an I/O port there's no CPL=3 -> CPL=0 -> CPL=3 context switching involved).


Cheers,

Brendan

[frank]

@Brendan

Yeah I was only thinking of using #1 for a temporary basis. Now of course I see the problems with it. For number 3 are you talking about an individual syscall for in and out or an IPC message to an IO manager? I'm only asking because it seems that one of the other responses talks about using IPC to communicate with ports.

Thanks everyone for your responses.

[elderK]

Hey Brendan!
Long time listener, short time caller

I would really appreciate it if you could go into some more detail on the topic of Port virtualization.

I tend to think of IO Protection as such:
- You can have a privileged interface. ie: System calls to gain access to
IO ranges. System calls to release.

And yahey, thats all well and spiffy and all, but its still not virtualized, if you are say... writing to port 0x3249, you write to that port, via whatever IO Sys call.

What would be *AWESOMELY* interesting, would be to emulate Memory mapped IO.

Like ...

"I want to claim this port range, 0x3f0 - 3f8' (or whatever).
KErnel says, Okay... They havent been claimed... (or even if they have, maybe the kernel could buffer the writes? Who knows)

Part of the system call could identify a variable in userspace, an array or something, that would act as the logical ports.

so, like, you could select the width.

Hhmmm (sorry, im pondering as I type man, you know how passion is! )

It we want 4 32bit IO links, started from physical IO 0 (hypothetical).

PHYS LOGICAL BUFFER IN USERSPACE
00 <------ port_dword[0]
04 <------ port_dword[1]
08 <------ ...
0C <------

So, the userspace app could just write data into the array it created for this use. The kernel would know where to find it, so... at specific times the kernel would perform the IO stuff for real.

Only problem I have is for IO Reads.

ie :
unsigned int fromPort = port_dword[0];
^--- youd want this to like, read the data from the port. Which, it wouldnt. UNLESS the kernel updated the array. Of course, the kernel would need to know if it should do ins from them all ... or not. Yknow?

Im rambling, but, Im shakey on Port virtualization, so, any info would be MASSSSIVEEELLLLLY appreciatified.

~Z

[elderK]

Actually, this reminds me of my time on an Atmel AVR Microcontroller.

You had three registers for a port.

say, PORTA, PINA, DDRA.

Port A out, port A in, port A direction.

In this case, we can do with just a Port and its direction.

If the direction is set OUT, Then the kernel will write the data in that entry (in the logical array or whatever), out to the proper address.

If the direction is set IN, then the Kernel will update.

And... like, you could have different modes for these kinds of things.
Like, you could have it completely asynchronous.

Ie : I am waiting for this .... Wake me up when this port says this *sleeps*

OR... You could do poll. Both systems would work withthe kernel automatically updating the IO stuff on preemption, or idle times... or whatever.

ALTERNATIVELY, you could do like...a Flush thing, like, ioSync.
And the kernel would then do it all post-haste for you via syscalls.

Hhmmm...ponderponderponderponder.
Feedback?

I LOVE COFFEEEE!:D
*wanders*
~Z

[Brendan]

Hi,

Yeah I was only thinking of using #1 for a temporary basis. Now of course I see the problems with it. For number 3 are you talking about an individual syscall for in and out or an IPC message to an IO manager?

I'd be tempted to use individual syscalls for IN and OUT (with an I/O manager that tells the kernel which I/O ports a process is allowed to access), however this isn't the only way things could be done...

I'm only asking because it seems that one of the other responses talks about using IPC to communicate with ports.

Your original question "How do microkernels do device IO" could be read in 2 different ways - "How does other software talk to device drivers" and "How do device drivers talk to hardware".

To make things more confusing, the word "ports" has many different meanings - it could mean I/O ports in hardware, or message ports used by IPC, or networking ports (e.g. "the HTTP protocol uses port #80").

I think Sean's post is talking about the interface/s between other software and device drivers (and not the interface/s used by device drivers to talk to hardware)...


Cheers,

Brendan

[frank]

Okay, thank you very much.

[Brendan]

Hi,

What would be *AWESOMELY* interesting, would be to emulate Memory mapped IO.

To do this you'd need (for example) code in the page fault handler to decode and emulate every instruction that reads or writes to memory, so that something like "mov eax,[foo]" generates a page fault and the page fault handler determines what you were trying to do and does "in eax,foo" instead.

It'd be a huge amount of code that doesn't really do anything except add overhead and bugs - much easier to just use the I/O port instructions to begin with.

I would really appreciate it if you could go into some more detail on the topic of Port virtualization.

Imagine you've got a system where all I/O port instructions cause a general protection fault, and the general protection fault handler decides if you have permission to access the I/O port and either terminates your code or emulates the I/O port access on your behalf.

Now, what sorts of tricks could you add to this?

First, it'd be simple for the general protection fault handler to check a flag, and if that flag is set it could display some sort of message in the system log (e.g. "process %s read the dword %x from I/O port %x\n"). This could be used as a debugging option, so that (for e.g.) you might set the flag for a specific driver and end up with a full log of all I/O port accesses done by that device driver.

Next, you could remap I/O ports. For most OSs a device driver needs to find out the base I/O port for it's device, and then does things like this:

Code: Select all

    mov dx, [ baseIOport ]
    add dx, offsetForRegister
    in al,dx

The general protection fault handler could keep track of the base I/O port for each process and automatically work out which I/O port to access. In this case the device driver doesn't need to find out what the base I/O port is for it's device, and virtual I/O port numbers" can be hard-coded into the device driver. The above code becomes:

Code: Select all

    in al, offsetForRegister

This is much shorter and cleaner.

Now most of us are familiar with the "device manager" in Windows, which lets you change resources (e.g. I/O ports, IRQs, etc) used by devices to fix any conflicts. The problem here is that you have to reboot the OS before the changes take effect. If our general protection fault handler is remapping I/O ports and allowing device drivers to use "pretend" I/O ports, then you could change a device's I/O ports and tell the general protection fault handler about the change, and the device driver wouldn't need to know. Basically, you could change device's resources without rebooting (and without adding a pile of mess to the device drivers).

Lastly, who says the devices actually need to be real devices? You could have (for e.g.) a real floppy disk driver which access I/O ports like any other driver, but the general protection fault handler might forward these accesses to some floppy emulation code instead. It'd be an interesting way of developing an emulator...


Cheers,

Brendan

[speal]

What is the reason for not using the same IPC mechanisms to give device drivers access to I/O ports? The overhead of the context switch would be present in all implementations, so I'm just curious what the benefits of port virtualization are over a unified system call.

The only real reason I see to use the standard IPC send/receive calls for port I/O would be for emulated devices. If standard IPC is used, a user-mode server could run as a device emulator, and if a separate device driver tries to interact with it there is no overhead of translating virtual ports into IPC calls. Of course there may be an overhead when converting the send/receive calls into actual port I/O implemented by the kernel, but this would certainly depend on how your IPC has been implemented.

I'm no veteran microkernel designer, so I may be missing something big here. Thoughts?

- Charlie

[elderK]

Well, from an architectural perspective, it all depends on what kind of environment you are planning to create.

Do you want something that is completely asynchronous? Do you want to support multiple processes using the EXACT same hardware at once? Are you super concerned about interrupt latency? Is your Kernel preemptive? Can your drivers be preempted? How do you schedule driver actions?

All of these are considerations.

Thank you for the explanation, Brendan.
From all I have heard lately about Port virtualization (from professors and suchlike), I had the odd feeling that maybe I had missed some cool hardware feature that made such protection fantastically easy.

~Z

[JamesM]

Brendan: Why don't you use the I/O permissions bitmap in the TSS? I currently hack around and set IOPL to 3, but I'm intending to switch to a TSS-based method. I believe this is also one of the few TSS features that linux makes use of.

It would be much quicker than implementing syscalls for In and OUT and would offer much the same capabilities.

[JAAman]

he already explained that:

The other method is to use the I/O permission bitmap in the TSS. This isn't as flexible (you can't do I/O port virtualisation). It also increases task switch times as you need to change the I/O permission bitmap during task switches; however there is no "per access" overhead (when a device driver accesses an I/O port there's no CPL=3 -> CPL=0 -> CPL=3 context switching involved).

basically, its a valid option but...
1) you increase task-switch overhead since you must modify the TSS on each task switch (although im thinking... what if device drivers each had their own TSS, then a generic one for all others... this would allow you to just change it instead of modifying it on each switch)

2) you cannot virtualize the ports -- ie you cannot redirect the driver to a virtual device, and the driver must know the exact address range used by the hardware


as an additional benefit:

For example, all device drivers might be hard-coded to use I/O ports 0x0000 to N. When the sound card driver tries to access I/O port 0x0000 the kernel might actually access I/O port 0x0340 on it's behalf, and when the network card driver tries to access I/O port 0x0000 the kernel might actually access I/O port 0x02C0 on it's behalf. Also, if PCI configuration space is changed for the network card (so that the network card is reconfigured to use I/O port 0x1100 instead), then the kernel could change it's translation to suit without the network card driver caring or knowing about the change (basically, you could dynamically reconfigure resources at any time without the device driver's knowledge).

though if your going to do this, you will also have to virtualize memory-mapped-I/O

[elderK]

don't get me wrong, but isn't that the point of a device driver?
I mean, A driver needs to know the port addresses of the hardware it is trying to access. It doesn't necessarily have to use the ABSOLUTE addresses in action.

I mean... lets look at this.
Driver:"Hi, I'm a driver for the VIA 8237 Audio chipset!"
Kernel: "Hey dude, no problem. What IRQs, Ports do you require to control the device?"
Driver:"Er... Im not so sure, man. last night is kind of blurry..."
Kernel: "I see..." <kernel kills driver>



I mean, yes, port virtualization is *important*, in terms of protecting the hardware and providing dynamic routing. But, a destination is still important.

for example:
http://www.osdev.org/

The destination is usually the same (who knows, maybe the DNS changes secretly ...) but how I get there, isn't.

Virtualized I/O, to me, is more about protection then holding the developer's hand.

It's like socket communications, we open a connection on whatever socket we create. That socket has an obvious destination, it has a obscured route. We are still communicating via te socket, which is really.. just a logical representation of the link.

Is Port virtualization not the same? Would such a mechanism NOT be useful? Flexible? Especially if the overhead was small?

Imagine the kinds of scheduling we could perform!

Program A creates an IO Socket, claims 0x0 - 0x61.
Program B creates an IO Sockets, claims 0x0 - 0x61.
>>> Kernel knows internally, that the range 0x60-0x61 is already ACQUIRED, so, it could either block Program B (to allow the program to 'continue' at a later time) [[lame]], OR... it could buffer the output to thsoe ports.

It all depends on implementation, it all depends on the logical binding.

Regardless, my god... Port virtualization makes my mind GLEEFUL!


(Oh, and to anyone who might have taken this post wrong, this isn't meant to be even SLIGHTLY flamey).

~Z

[Brendan]

Hi,

1) you increase task-switch overhead since you must modify the TSS on each task switch (although im thinking... what if device drivers each had their own TSS, then a generic one for all others... this would allow you to just change it instead of modifying it on each switch)

A bitmap with one bit per I/O port costs 8 KB. If you support a maximum of 8192 tasks, then it'll cost a maximum of 64 MB just for I/O permission bitmaps. That is far too much.

Instead, each process could have a "I don't use any I/O ports" flag, and you could have a shared "no I/O port access allowed" TSS. That would reduce the memory consumption a lot (e.g. for 50 device drivers it'd only cost 400 KB), but then your task switch code becomes something like:

Code: Select all

  if(new_task_uses_IO_ports) {
        copy_IO_bitmap();
    } else {
        if(old_task_used_IO_ports) {
            wipe_IO_bitmap();
        }
    }

That's 2 branches (and up to 2 branch mispredictions per task switch) plus the cost of "rep movsd" or "rep stosd" added to the worst case task switch times.

Instead, I can have an array that says which process owns each I/O port. It costs 256 KB (4 bytes per I/O port) and nothing needs to be done during task switches. Given that my OS/kernel is modular, this helps reduce the dependancies between modules (e.g. I can have a "scheduler module" and a "hardware support module", where both modules don't need to care what each other do). Also, because my OS uses a lot of event-driven programming there's lots more task switches than there would be in a normal monolithic system, so reducing the overhead of task switches makes more sense.

don't get me wrong, but isn't that the point of a device driver?
I mean, A driver needs to know the port addresses of the hardware it is trying to access. It doesn't necessarily have to use the ABSOLUTE addresses in action.

I mean... lets look at this.
Driver:"Hi, I'm a driver for the VIA 8237 Audio chipset!"
Kernel: "Hey dude, no problem. What IRQs, Ports do you require to control the device?"
Driver:"Er... Im not so sure, man. last night is kind of blurry..."
Kernel: "I see..." <kernel kills driver>

In a monolithic kernel, the kernel assumes that device drivers can be trusted because they're part of the kernel. For a micro-kernel, device drivers are just seperate pieces of code (like applications), and shouldn't be trusted. In this case the kernel should give them just enough access to do their job, but no more.

Consider this:

Driver: Hi, I'm a driver for the VIA 8237 Audio chipset!
Kernel: Hey dude, no problem. What IRQs, Ports do you require to control the device?
Driver: I want access to all these I/O ports [Driver sends list of I/O ports to Kernel]
Kernel: Hmm, OK I guess....
Driver: Sucker! [Computer crashes as malicious driver turns off A20]

For my OS it works more like this:

Device Manager: I just found a VIA 8237 Audio chipset during the PCI bus scan!
Device Manager: Hmm, according to the device's PCI configuration space, this device needs some I/O ports. I guess I can assign I/O ports 0x1234 to 0x1345 to this device.
Device Manager: Now I'll see if there's a device driver for it....
Device Manager: Woot! Found a device driver [various grunting noises as the device driver is started]
Device Manager: Hey kernel - I'm starting this device driver, and it needs to be able to access I/O ports 0x1234 to 0x1345. Can you make it happen?
Kernel: You are the official Device Manager, and there's no other problems. Approved.
Device Manager: Thanks Kernel!
Driver: Hello?
Device Manager: Hi audio driver. Today you'll be using IRQ #33, and I/O ports 0x1234 to 0x1345.
Driver: Cool - initializing
Driver: Hey, everything is working!
Device Manager: Good. I'll tell the user interface to talk to you
UI: Hello sound driver
Driver: Hello UI. I can do MIDI and have 5 channel surround sound!
UI: That's nice. For now just play this aweful startup jingle.
Driver: Ok

Of course security should go further than this. Should a keyboard driver be able to use networking and send keypresses to some dodgy server on the internet? No. Should a network card driver be able to open the file "/etc/passwd"? No. Should a sound card driver be able to tell the kernel it's a disk driver and get access to data that applications send to swap space? No.

For systems like Windows and Linux there really isn't enough device driver security built into the system, and to get around that there's digitally signed drivers and Windows Hardware Quality Lab (WHQL) tests, or you need to spend months arguing with Kernel maintainers to get your driver into the kernel source code tree. I simply don't have enough cash or time to do things that way.


Cheers,

Brendan