Unified Driver Interface

[Kevin]

I have also recently found that there have been a lot of UDI drivers out there, unfortunately closed-source - either written by SCO, individual hardware vendors or various other parties. Very few of them can be found on the Internet and the only open source implementations are those in udiref.

Then all your discussion with Brendan is moot. Not having drivers makes UDI pretty useless to support.

[Love4Boobies]

And that attitude won't help in bringing any drivers. However if more people wrote drivers, we'd all benefit from them. That's why we started the repository in the first place. I'll be commiting an AMD64 processor binding to the framework soon. I came up with it today, I just need to test it.

[Love4Boobies]

I went looking for UDI drivers. In the STG "UDI for Linux Reference Environment" I found:

• one driver for DEC 21140/21143 based ethernet cards
• one device driver for CMOS
• one device driver for some Adaptec SCSI controllers (6 versions of DPT SmartCACHE/SmartRAID controllers)
• some useless pseudo/test/example code
• one device driver for a "Guinea Pig" - I don't know if this is a real device or more useless pseudo/test/example code

For all of these the copyright is the same, and it looks like the code came from Project UDI back in the year 2000/2001 and hasn't been touched since.

I couldn't find any other UDI drivers of any kind. And if this list is a complete list of all UDI drivers then (if I ignore the other problems) it's going to cost me more time to support UDI than it'd save me.

Although this isn't a community way of thinking, you do have a point. Even if you didn't, I was trying to suggest, "Hey, let's all adopt this interface since it's pretty well-thought and we can all benefit from each other's work - since that's one of the points of open source in the first place!". I did some research of myself and found *a lot* of UDI drivers on SCO's FTP. Unfortunately they're spread all over the place (different OSes, different OS versions, unrelated "drivers" directories and so on). I estimate there are about 70 drivers there - I'll make a list; I'll look for others in the days to come.

Ah, one thing that I forgot to mention about udiref (the framework) - it features a complete USB stack so that is another thing to think about

[Firestryke31]

Also, don't forget that driver interface != operating system. Just because you support UDI doesn't mean your operating system is a Windows/Unix/whatever clone, it just means that your drivers might possibly be able to run on a different operating system with little to no modification, and another UDI-compliant operating system's drivers might work on yours as well.

[Brendan]

Hi,

But UDI itself doesn't make it impossible for you to add extra features, that's one of my points.

I might be able to implement my features in UDI drivers that I write, but how do I add those feature to UDI drivers that everyone else has already written (and alternatively, remove features in drivers from other OSs that my OS doesn't use)?

For example, if a user downloads a binary driver for a new SCSI controller from Adaptec's website, how does my OS automagically add support for virtualization and my I/O priorities scheme to that binary driver?

Also, how can I tell if the driver is a fully tested driver, or a beta release that my OS shouldn't allow on a mission critical server?

I have also recently found that there have been a lot of UDI drivers out there, unfortunately closed-source - either written by SCO, individual hardware vendors or various other parties. Very few of them can be found on the Internet and the only open source implementations are those in udiref.

Then all your discussion with Brendan is moot. Not having drivers makes UDI pretty useless to support.

I don't have any objection to my OS using proprietory/closed source drivers; as long as I can legally distribute them (in boot images for my OS's installer, and on a central device driver repository) and users can use them, without anyone paying $$ for it. For me, the worst part is "Very few of them can be found on the Internet".

And that attitude won't help in bringing any drivers. However if more people wrote drivers, we'd all benefit from them. That's why we started the repository in the first place. I'll be commiting an AMD64 processor binding to the framework soon. I came up with it today, I just need to test it.

Huh???

I thought UDI was a standard where (in theory) everyone can use the same drivers? Why are you inventing your own AMD64 processor binding, and what happens when 20 different people all invent their own different/incompatible AMD64 processor bindings?


Cheers,

Brendan

[Solar]

I doubt that there is much "personality", or performance, to be won by custom driver architecture. I think that the "personality" of an OS is defined by how good it is to administrate and use.

That's my personal opinion, of course, and Brendan is free to think different. Where I have a problem is that Brendan discourages the concept of a generic driver system, in general terms.

When there was first talk about Linux and its "open source" approach in the early 90'ies, I thought it would finally level the playing field so that OS's could compete in features, in usability, instead of leveraging their driver base (as Microsoft Windows so beautifully did, and still does). When it turned out that the Linux people had no intention of making their "open source" driver architecture usable by other operating systems, I feel somewhat cheated.

When I started my OS project (gosh, has it really been 9 years already?), Project UDI seems like the solution. Who cares if those who invented it didn't pack a big collection of drivers to go with it? If every hobby OS out there would contribute one UDI driver that is not already been done by someone else, we would have what, like, 1000% of the drivers we have now, for every hobby OS out there?

If your OS "lives" off its specific, custom, driver architecture, OK, UDI is not for you. But one nagging question remains: Is your custom driver architecture better, or is it merely different for difference's sake? ( <-- NOT trying to offend Brendan here, but it's a question you have to answer for yourself.)

[Brendan]

Hi,

I doubt that there is much "personality", or performance, to be won by custom driver architecture. I think that the "personality" of an OS is defined by how good it is to administrate and use.

That's my personal opinion, of course, and Brendan is free to think different. Where I have a problem is that Brendan discourages the concept of a generic driver system, in general terms.

Well, why stop at device drivers?

How would you feel about a generic scheduler architecture that all OSs can use, and a generic memory management architecture that all OSs can use, a generic IPC framework, and generic VFS and networking layers? That way the average script kiddie can grab a copy of GRUB, download the developer kits for everything (including UDI), cut & paste about 50 lines of code from the nearest tutorial, and have a complete OS that they've "written" themselves after about 2 days work (and then begin porting GCC, X and a bunch of GNU tools)? Would you assume that all these "script kiddie" OSs will be unique designs that help push "state of the art" past the current status quo?

You see "generic device driver framework" and think "Woot - lots of work I can avoid!". I see "generic device driver framework" and think "reduced to the lowest common denominator" - the set of features that are common to all OSs that rely on that generic driver framework.

But one nagging question remains: Is your custom driver architecture better, or is it merely different for difference's sake? ( <-- NOT trying to offend Brendan here, but it's a question you have to answer for yourself.)

A car isn't be better than a truck (especially if you want to shift large/heavy things from place to place), and a truck isn't be better than a car (especially if you just want to get to work on time), they're just different. The important thing is that these differences allow people to choose whatever product is best for their purpose.


Cheers,

Brendan

[pcmattman]

@Brendan: If your OS doesn't facilitate using this framework, then don't port the framework. Don't do it! We don't care! But please stop bashing it because it doesn't work for you.

I'd also ask that we wait for an actual implementation in a hobby OS to be written before any conclusions are jumped to as to complexity, difficulty, or how well it integrates.

[Solar]

Well, why stop at device drivers?

How would you feel about a generic scheduler architecture that all OSs can use, and a generic memory management architecture that all OSs can use, a generic IPC framework, and generic VFS and networking layers?

All of which you can handle yourself with a copy of the Intel manuals and a good book on OS architecture.

Drivers are different. For one, you need the hardware in question to test it - that alone is a kill shot for most hobbyists.

[Brendan]

Hi,

Well, why stop at device drivers?

How would you feel about a generic scheduler architecture that all OSs can use, and a generic memory management architecture that all OSs can use, a generic IPC framework, and generic VFS and networking layers?

All of which you can handle yourself with a copy of the Intel manuals and a good book on OS architecture.

Do you actually believe that?

Over the years I've collected a total of 26 computers (so far), specifically selected to ensure that I've got a wide variety of *different* CPUs to test on. There's a reason for this...

I guarantee that (even with as many Intel manuals and good books on OS architecture that you could possibly want), it's very likely that your OS (assuming it's not just a "Hello world') will not work on anything *except* the computers that you use regularly for testing. Nowhere will any Intel manual mention that Cyrix CPUs don't have a CR4 and crash (general protection fault) when you try to access it, nowhere do Intel manuals mention bugs in Transmeta's implementation of RDTSC. They won't mention how to use the encryption stuff built into VIA's CPUs, or AMD's power saving technologies, or anything else about any other 80x86 CPU manufacturer; and that doesn't even begin to cover all the bugs in your software that the Intel manuals *do* contain enough information to avoid, that you don't even know are there.

@Solar: These aren't just idle words - I've already backed this up with my wallet (I've spent close to $10000(Aust.) in the last 12 months alone). However; if your OS is capable of booting from network (as there's no other way to boot most of these test computers), I'm prepared to test your OS (assuming it's not just a "Hello world') on all of these computers, just so you know how badly your "I only need Intel manuals OS" fails.

Drivers are different. For one, you need the hardware in question to test it - that alone is a kill shot for most hobbyists.

Drivers are different - if you don't have suitable hardware you can skip it, and let someone else write the driver for your OS at a later time. You can't really do that with kernel code.


Cheers,

Brendan

[Solar]

Drivers are different - if you don't have suitable hardware you can skip it, and let someone else write the driver for your OS at a later time.

Or see your OS collect dust in some remote corner of the web because no-one will bother to get an antique ATA drive to bootit and get excited enough to do any driver development work for it...

Brendan, I see we are unlikely to agree on this. Let's agree not to agree.

[Love4Boobies]

Sorry, I'm on vacation so I'm rarely online to post anything.

First of all, standards sometimes do need processor bindings (e.g., the ELF file format). It's only natural they do. UDI defined processor bindings for a few platforms (alpha, arm32le, ia32, ia64, ppc, ppc-mach-o, sparc32, sparc64). Some of these were never formally described through documents but implemented in the framework nonetheless. It's pretty obvious how they should look - you can't really make incompatible interfaces. For example, writing AMD64 mode bindings is almost a copy & paste exercise between the IA-32 and IA-64 specs; it needs to have AMD64 calling convetion, 64-bit word sizes, ELF64, etc. If there would be a new wave of interest in UDI people would write bindings for future CPUs for sure.

[KurtGollhardt]

Hi, I heard about this discussion, and since I was part of the core team that developed UDI, I thought I would try to shed light on a few points. First, a brief bit of history and context:

I am no longer working for an O/S company, but at the time I worked on UDI, I had worked on many different parts of the UNIX System V kernel (VM, FS, boot, I/O, and others), both as developer and architect. I followed the "official" UNIX source base (from which all others were licensed) as its ownership (or "stewardship" as it turns out in SCO's case) moved through a series of companies including Bell Labs, USL, Novell, SCO, and then Caldera (which belatedly renamed itself SCO). I left soon after the Caldera acquisition and before the policy of sue-everything-that-moves, which I do not condone.

The UDI work started before SCO's involvement; I brought it over with the UNIX acquisition, but in the end SCO did become one of the most proactive in supporting it and actually using it in product and even producing driver-writer training materials (still available on the projectudi.org site--http://udi.certek.com/Presentations/SCO_Forum_1999).

From the start, this was a multi-company effort, with support from over a dozen companies, both O/S and driver providers, and always at least 5 or 6 of us actively involved in the spec design. And even though many of us were representing UNIX companies, we set a strong goal early on that UDI be completely O/S-neutral and NOT be tied to traditional UNIX APIs or models. I firmly believe that we succeeded.

Unfortunately, though we did complete a great and innovative piece of technology--which is still usable and appropriate today--we couldn't get it to go any further, because the effort was primarily driven by a group of engineers and architects, and we couldn't get enough momentum on the business side. If there had been one shrewd business person truly committed to it, it probably would be commonplace today. (No, I'm not bitter. )

So as of 2001 or so, activity on the specs, the website, the mailing lists, and general efforts to further the goals of Project UDI, petered off as we each lost momentum within our respective organizations and as each of those organizations continued to dwindle as part of the overall death of the UNIX O/S industry. But there are a number of us who, as individuals, would support any renewed efforts if they appear to be committed and productive. If new materials became available I could put them on the website.

But let's get to some specific points:

Does UDI stifle innovation or limit the ability to add features to the O/S or driver? No. In fact, it's the reverse. The UDI APIs were very carefully designed to work in a wide variety of O/S architectures: real-time, time-sharing, or virtual; server, desktop, or embedded; traditional kernels, micro-kernels, or distributed systems; single processors or highly parallel systems with or without shared memory; drivers running in kernel space, user space, on dedicated I/O processors, or over a network. All of these possibilities and more without ANY change to the driver code.

Many factors lead to this flexibility, including an asynchronous message-based API (even for resource allocation); a precisely-defined model for memory usage; O/S-driven driver instantiation, discovery, and I/O resource assignment; and, in general, hiding as much detail as possible from the driver about the O/S implementation and the underlying hardware platform. Have we covered every possible architecture? Probably not, but we've come a whole lot closer to it than any other driver API I've seen.

By narrowing the responsibility of the driver to dealing with the functions of its device and the bare minimum necessary to communicate with the surrounding environment (O/S and/or other drivers), UDI allows O/Ses to add features and make architectural changes at any time without having to worry about changing existing drivers or driver APIs. At the same time, it allows drivers to implement and export new functional features of their devices, and to experiment with different architectures for their own devices without affecting upper software layers.

There are plenty of opportunities for uniqueness and "reasons for people to be interested in" O/Ses that happen to use UDI, even within the I/O subsystem, let alone the rest of the O/S.

I commend Brendan for what sounds like--from his comments on this thread--a pretty sophisticated O/S that uses a number of the advanced architectural techniques we envisioned when we designed UDI to be able to go well beyond the traditional UNIX kernel. Without examining it more closely, I can't say whether the UDI API and architecture 100% covers all of his driver requirements, but I suspect that it does.

Could Brendan--or any O/S provider--develop a "proprietary" driver API that's more finely tuned to their specific O/S architecture (in its current version) than a generic portable interface like UDI? Sure, and Brendan quite likely already has. BUT, in general this a challenging piece of work that is at best likely to yield only a small increment in functionality or performance over what can be achieved with UDI. (In practice, all UDI implementations so far have shown increased performance and scalability over native drivers, even when implemented as a "wrapper" underneath the native APIs.) It also means you have to write (or port) all the drivers yourself, or convince someone else to port the drivers to your proprietary API.

You also potentially have to change your API and possibly revise all existing drivers when you make certain changes to you internal O/S architecture (so as to maintain the "fine-tuning" to your architecture), unless you built sufficient flexibility and abstraction into the API in the first place (but then it would be a generic portable interface just like UDI) or keep the API for compatibility and then sacrifice performance or features.

We all have limited resources, whether it's a single person working on a hobby O/S, a big open-source project, or a commercial O/S. Why not spend the effort on the core features and/or architectural design elements of the O/S, rather than designing yet another driver API and porting umpteen drivers (and/or convincing h/w vendors that choose to keep their device designs proprietary to port their drivers to your O/S)? Let h/w providers work on h/w drivers, and O/S providers work on O/S internals.

Which came first, the chicken or the egg? or Why should I implement UDI in my OS/driver when there aren't any UDI drivers/OSes out there? This was always one of our biggest challenges with the business folks, but the answer should be simple: Someone has to go first. If more OSes support UDI, more drivers will get written (even if some of those are ports done by the early OS providers themselves), and as more drivers are available, more OSes will support it, and then you reach critical mass.

Here's my favorite:

We already have x many h/w vendors and drivers supporting our OS. If we implement UDI, and get our vendors to rewrite their drivers, then our competitors will get all those drivers for free (by also implementing UDI) and we'll lose our competitive edge. Over the years, I heard this same argument from folks at every single UNIX vendor, as well as Novell's NetWare. Of course, every one of them claimed that THEY were the leader in IHV (Independent Hardware Vendor) support, so why should they give up that lead.

Come on folks! The ONLY one who could truly make that claim in the last 15 years, and for the foreseeable future, is Microsoft. They are the only ones for whom h/w vendors automatically write drivers. They are the only ones who really do have a marketing incentive to discourage portable drivers. For every other O/S, driver support (or lack thereof) is a negative differentiator. (In marketing terms this means that the lack of support for a device will hurt you more, when being compared to a competitor, than support for one will help.)

Where are all those UNIX vendors now? And NetWare; what's NetWare?

Of course, SCO didn't make it either, but then--as much as I appreciate the effort they did put into UDI--it was probably too little, too late. Of course, they also did things that pissed off a lot of people, and were also victims of the general UNIX market downturn.

Does UDI support ALL types of drivers and hardware? Well... Yes and No. This is one area where you definitely don't get something for nothing. The answer is that the core specifications cover 95% of what you need to write a driver. They cover all the things that are common across all types of drivers: driver instantiation, resource allocation, communication between drivers or between drivers and the O/S, interrupts, priorities, and so on.

However, each type (or "class") of device (such as mass storage, serial port, Human Interface Device, etc) needs a set of specific requests and responses that reflect the particular semantics of that type of device. These are defined in what UDI calls a "metalanguage", one metalanguage for each device class.

Project UDI has so far published a small number of "add-on" specs that define metalanguages for specific device classes (Network, and SCSI), plus a few specialized metalanguages in the core specs. The USB standards group has published a USB UDI metalanguage (http://www.usb.org/developers/devclass_docs/usbdi10.pdf)--though it's in need of some revision (for which I, personally, probably have the only complete set of notes--on paper, no less). I also have a mostly-complete HID metalanguage which has not yet been published. (Both victims of the timing of my leaving SCO.)

For other device classes, new metalanguages would have to be written. The core UDI spec provides procedures, policies, and tools for developing new metalanguages. These could be done ad hoc for specific OSes, but to get the most out of them, you would want them to be standardized and published by some related standards body or industry group, or at least a group of multiple implementors. Project UDI, having no active official resources at this time, could probably still scrounge up enough volunteers to review, "approve", and publish new driver-class specs, if the bulk of the design and writing were done by someone else and validated by at least one implementation.

Fortunately, with modern devices this is a relatively easy process, since the detailed semantics for device classes are often defined in hardware specs, such as USB. So we can take things like HID, Mass Storage, Power, and so forth, generalize them a little bit to not be USB-specific, and then simply cast them into the structure of a UDI metalanguage (for which the general rules are specified on the core spec).

Similarly, additional processor bindings (which specify binary formats and calling conventions, usually by reference to existing standards) can be published. I'll be happy to put the proposed AMD64 binding on the Project UDI site. Any given O/S, though, only needs to worry about the binary formats they choose to support. UDI allows for drivers to be distributed as source or binary, and any combination of source and/or multiple binaries can be included in a single driver package. (This is the point on which the FSF--i.e. Stallman--had the philosophical objection; UDI tried to be pragmatic, especially with respect to commercial OSes, allowing drivers to be source OR binary; the FSF wanted to require source; we said any particular OS can choose to support only source drivers, but that wasn't purist enough for them.)

The UDI Core specs are so big, it must be too complex and costly to implement, right? Wrong. The specs are big because they go into great detail about the APIs and required behaviors, in an attempt to eliminate all ambiguities that might interfere with the complete portability of the drivers. Also, the detailed APIs are a bit repetitive (with request and response parts of most operations, with caller and callee forms of each). Once you get the feel for it, you can skim through most of it pretty quickly.

P.S. If you haven't seen Deven Corzine's 2005 editorial on this topic, I recommend it: http://www.ties.org/deven/udi.html

More materials are available at http://projectudi.org

[Solar]

I'm stunned speechless.

...

Thank you for that very insightfull post. Would you mind me copying it over to udi.rootdirectory.de?

Also helpful - if I make such a bold request - would be a quick overview on what's dead, what's dormant, and what's still capable of responding if poked? I think we'd be better off not forking, or duplicating efforts, when there's still some active stewardship going on somewhere.

Edit: He responded via PM in the affirmative. Yay!

As I said to Love4Boobies here, all I can do at the moment is providing the website infrastructure at rootdirectory.de, and perhaps some communication "glue"; so I can't speak for the others. But for me, UDI sounds like one of the big great ideas of the IT industry that just hasn't made it to the spotlights yet. (Like multitasking desktops, which were available for many years already when Microsoft released Win95.)

[Deven]

It's nice to see that my 2005 editorial promoting UDI didn't go completely unnoticed. I haven't had time and energy to do anything like sitting down and actually writing an OS with UDI support or writing UDI drivers, but I've certainly tried to at least do some advocacy over the years.

I can say with certainty that if I ever do write my own OS -- and that's something I've always wanted to do -- it would definitely use UDI as the native device driver API, whether UDI appears to be dead or not. It's good technology, it's just suffering from a chicken-and-egg adoption problem.

Deven

P.S. I fixed the expansion of "UDI" in the subject line.