OSDev.org

With all the trouble about device drivers you get when developing an OS, I've recently found myself thinking about it and wondering if there wasn't a better solution. And indeed, I found something, a mere idea only, that could make the os developer's life a lot easier. Anyway, here it goes:

When the kernel detects a new device on the PCI, USB, etc. bus or when it does the startup check and gets a list of devices, it sends the driver a short message, in order to start the communication. This message could be anything (HELLO, HI, e.g.) and then the device driver sends an answer, and this answer is an URL (typically on the device manufacturer host, or some mirror). Then, the kernel loads the content of this URL. The resulting document must be an XML document, well-formed and valid upon a given DTD scheme which must be followed by the hardware manufacturers.

What follows here is somewhat implementation specific, and it assumes a microkernel/hybrid design. The kernel has a device driver program that is then loaden into memory and executed (as you would do normally with other device drivers). But this device driver is different, as you always load the same one, ideally the only driver you shall ever write. As a parameter, the program receives the XML document. After that, when a system call occurs and the driver is required to do something, it looks the action up in the XML document and does what it is told to do there. The same happens when the device sends an event, the driver has like a database containing everything it needs in order to operate.

So, the basic idea is that the developer must now only provide a single driver which is kind-of self-adapting to the circumstances. The system is also platform-independant, as the variable data is only XML, and only the driver must be ported.

This is like the basic architecture of the system. I hope one could understand it. In order to improve the performance of the system, one could store the XML document on the local harddisk and check it on the internet every month or so (or when the ethernet card is idle).

The keyword 'ethernet card' brings us to the bootstrap process of the system. The ethernet card or modem cannot be loaden in the same way, as it must run first to get information (a typical example of a bootstrap). But even then, it can follow the same procedure. One could have the most basic ethernet driver loaden as a module by GRUB and then as first thing, the ethernet driver updates itself et voil?.

This is where my ideas end. I hope you don't think something like 'calm down, B. Gates' or anything similar of me , but I must nevertheless admit that I've been thinking about this since a month and I haven't found really big drawbacks apart from the bootstrap problem. I also hope you could understand everything (I sometimes use a weird english ). But anyway, if only one or two device manufacturers would follow this 'standard', others would soon follow...

Hope you're interested,

Robin Hollenstein (call me Robin or Candamir, whatever you like )

PS: If this is crap, tell me

Candamir wrote: PS: If this is crap, tell me

It isn't crap. It's not going to be supported either. It'd be dog slow (I'm annoyed by my laptop that swaps out the caps-lock logic only to cause a 3-second delay later on, no talk about inet latency), unsafe (let's execute some code we found online with full permissions), uncertain (unplug the cord to your internet and you've lost all usability) and probably going to take forever to make as well. Also, you still have bootstrap which would require you to always have those at the ready - and if you're going to have them, why not the rest?

Just a small note, read up on the in-depth working of ACPI. It works exactly like you want it to but for power-management only. Then count the number of proper ACPI implementations and the number of times a curseword appears in its code. Now, use your reasoning capabilities.

I have a basic security test I always apply when looking at things like this.

If by inserting a usb key, but not touching the keyboard / mouse or other input devices, I can get a piece of malware onto the system it should be rejected.

Ideas based on automatically downloading a driver for a new piece of hardware, or embedding the driver on the hardware scare me.

I like your idea. It's just one problem. How are you going to get all hardware manufactures to agree to this idea. Moreover the problem is that there are no standards between manufactures. If the manufactures were to have a standard interface between the devices. Developers would only have to implement the extensions that the device provides to the standard, not write a completely new driver every time a new piece of hardware came out. And because of code reuse, the drivers would be less error prone.

Candy wrote:
It isn't crap. It's not going to be supported either. It'd be dog slow (I'm annoyed by my laptop that swaps out the caps-lock logic only to cause a 3-second delay later on, no talk about inet latency), unsafe (let's execute some code we found online with full permissions), uncertain (unplug the cord to your internet and you've lost all usability) and probably going to take forever to make as well. Also, you still have bootstrap which would require you to always have those at the ready - and if you're going to have them, why not the rest?

Well, I think Candy has hit the nail, 100%, more or less... :-\ But I think there are /some/ workaraounds.

The performance: I agree that when the OS starts up first time on a new machine, it would take some time to download all drivers, but afterwards, there would be no overhead as you could tell the controller to search updates /after/ the system is completely started up, but even then only when noone else is using the bandwidth (edit: like some Update Download Managers for programs (Adobe, e.g.)). And then, when it has the driver downloaded, it loads a new instance of the driver that isn't allowed to answer request at that point, then you wait for the first driver to be idle, shut it down, register the new one with the kernel and pass over control to the new one. You wouldn't even have to restart the system, as in Windows. This would only happen when CPU usage is less than 50%. So, once you have your controller XML files initially downloaded, it wouldn't generate (much) overhead anymore.

Then, next topic, security. Lets say I'm a cracker and want to distribute a virus. Trying to modify the internal firmware programming of a device would certainly not be the easiest way to do it. Why would device manufacturers want to distribute unsafe code? And it isn't even code, it could be pseudo-code and that could be checked by the system with a heuristic method if it wouldn't install any backdoors or such things.

But I agree with you in one term: You could have the bootstrap system working with all devices, and then, you wouldn't have to update /all/ drivers first thing an OS is installed, but following the approach described in the performance argument.

But it is a sad truth that some manufacturers will always write propietary drivers... :-[

But drop the idea of the internet for a moment (that *is* probably the weak point in the whole thing). One could use this system and write the driver framework (and of course it would be less error prone, because there is only *one* driver to code, *one* driver to test and *one* driver to debug). So, you could adapt your kernel to use this approach even if it's you writing the XML files... It would take you quite a time before you could actually /use/ the system, but afterwards, adding support for a new device is a piece of cake.

Candamir wrote: But drop the idea of the internet for a moment (that *is* probably the weak point in the whole thing). One could use this system and write the driver framework (and of course it would be less error prone, because there is only *one* driver to code, *one* driver to test and *one* driver to debug). So, you could adapt your kernel to use this approach even if it's you writing the XML files... It would take you quite a time before you could actually /use/ the system, but afterwards, adding support for a new device is a piece of cake.

Then it degrades into a high-level UDI, which incidentally also failed because nobody could be arsed to write drivers or to support it. Would also mean that you'd just be recreating UDI but in a higher level language.

Hi,

Candamir wrote:So, the basic idea is that the developer must now only provide a single driver which is kind-of self-adapting to the circumstances. The system is also platform-independant, as the variable data is only XML, and only the driver must be ported.

Have you tried to define the content of these XML files?

It'll need to support direct access to I/O ports, linear memory addresses, physical memory addresses, PCI configuration space locations. Then you'll generic variables with support for different variable sizes and types (signed/unsigned, byte/word/dword/qword, and floating point), some conditional branching (if/then/else), some form of looping (for/next, while, do/until) and switch/case would be nice. Of course you'd also need arithmetic and boolean operators (multiply/divide/add/subtract/negate, and/or/xor/not/shifts) and then something to control operator precedence (brackets). You'd also want support for single-dimension arrays (but multi-dimension array would be nice), and structures, and maybe arrays of structures and structures containing arrays, and perhaps allow types to be defined.

Then you'd need some things for interfacing with the OS - a way of sending error messages, a way to allocate resources (allocate I/O ports, IRQs, etc), some memory management for temporary data buffers (and maybe variables) like "malloc/free" (or maybe "objstacks" and/or garbage collection) but capable of allocating buffers that are below 16 MB (for ISA) or below 4 GB (for 32 bit PCI) with alignment restrictions. You'd also need a way for the driver to communicate with others (messaging, sockets or something), and some way for the OS to make requests (standardized OOP methods like init(), read(), write(), shutdown(), etc?).

To make it easier for developers you'd also want the XML language to look similar to an existing language, so that they don't have to learn something completely different.

To speed it all up, I'd suggest some form of compiling - either compiling the XML when it's downloaded, or doing JIT when it's run. Then use caches to minimize the time spent downloading and compiling, so that the OS can check the URL for a timestamp or version number and if nothing has changed it could use the cached XML (or the cached compiled code).

After all of this, you might as well allow the OS to do the same thing for applications - auto-download the XML for them too, with the same benefits.

Of course this leads to some obvious questions - would long is this going to take to design and implement, and why not just use an existing language to begin with?


Cheers,

Brendan

If you're interested, search through the history of the alt.os.development newgroup for Eli Gottlieb's "EDI" - Extensible Device (or driver I can't remember) Interface. It's a different approach to a similar thing, and has a draft of the interface. Most of the discussion is relevent to what Candamir has described.

Am I detecting the hint of someone who has jumped on the "XML is the answer to every problem ever" bandwagon? As far as I'm concerned, any kernel that is sitting there parsing XML documents is wasting its time, there are much better and faster ways of storing data (and yes, even data that might need to be extended in some way ) ::)

Gotta love this quote though, pretty much embodies the corporate attitude to XML right now:
"XML is like violence, if it's not working you're not using enough of it."
(swap violence for paperwork/meetings/anything else you fancy where appropriate)

Well, XML would make development of a new driver easier than other things (to me). But actualy, it would be possible to support various dialects, etc. and then compile the driver for every new device you want to monitor.

Here comes a small example for the PIT:
And so on, you could also write a beep method, as this is also handled by the PIT. It took me like 10 mins to figure out this XML code with my actual sources for the timer, but surely it would be faster normally (as I had to invent everything on the fly ;D)

The only disadvantage of this system is the slow interpreting of the XML code, but once you have your OS run a compiler and a linker, the system is able to bootstrap itself and compile the XML files into binaries...

And BTW, XML isn't the solution for all things (at least, to me ), but I found it quite readable and that's what makes it valuable, because this system is intended to generate very stable drivers in no time, so it must be easy to read and write. Of course the kernel will be slow parsing the files, but what about a low-level thread that does this when CPU-usage is low? In the meanwhile, the system could work with the bootstrap drivers (these ones could be typical drivers as we're used to, in C and ASM). So, when the system hasn't updated all of it's drivers yet, it could work only when the CPU is lower than 80% (normally when it's lower than 30%).

I'm also concient that this interface *requires* a modular kernel and that it is certainly not fit for low-level kernels that aren't capable of some basic tasks (such as mine ;D). Also, the point I mentioned: Would you expect some hardware manufacturers (not to be named here) to release the specifications and APIs for their devices? Sad, sad, sad...

But nevertheless, it's fun to make theories and outrageous designs from time to time, even if they're never implemented

BTW, if anyone has implemented a similar kind of solution, could you please PM me?

i think that is a good idea. Also I could try that in my OS after spring break. this looks good (but only hybrid with reguler drivers) so you could just open a text editor and write a driver for a something, like a text printer
hows that, hows the os know what func to call, i'm not sure about <init> stuff for text printer?

okay, the "PIT" example clarifies things ... but it is essentially nothing more than a "hello world". I suggest before you start implementing anything, you take the time to read documentation of a somewhat more complex device (e.g. NE2000 card or ATAPI drives) and figure what you could do with that.

iirc, there's also something like "device interface language" (devil or something alike) that has addressed the issue, not mentionning the multiple research projects on network processors from ETH zurich and friends

I think for any device more complex than something requiring simple OUT commands, the XML would get so complicated that you might as well just be distributing source and get it over with.

This reminds me of the SPIN kernel, which downloads driver code written in a "safe" language into the kernel and compiles it. The main difference being that you're using XML instead of a proper programming language, which IMO is like trying to use a hammer to chop down a tree.

I'm quite liking the angle with the psuedo-code btw. Automatically download the code, run a few safety checks, compile it to work on the system it's actually on, and use it. There would still be the problem that any drivers written for the system would have source viewable to anyone and the manufacturers might not like that, but for drivers written by the community it could work.