Extensible Driver Interface

[thepowersgang]

Ok, after my last post in this thread I managed to implement EDI in my OS (not quite a complete implementation, but it's able to load the sample driver) and I'm going to try to write a floppy driver for it.
See attached for the implementation from Acess (see the [link=http://code.google.com/p/acessos/]acess website[/link] for a look at the Acess driver interface and the EDI helper files)

[Kevin]

Okay: this is it: Why don't we all create a Portable Driver Interface? I'm willing to work on it and produce a specification, seeing as I'm currently in a design phase for my kernel, and a method has pretty much presented itself to me. I haven't really read through the specification for the EDI, seeing as it doesn't really seem to have a large base of implementation.

Because it doesn't help if everyone is doing his own portable interface. In this thread at least three different, more or less existing, interfaces have been mentioned until now. I'm not sure what a "large base" does mean for you, but I know that your new interface currently doesn't have a single user.

1. Drivers should be as absolutely bare-minimum as is possible.

I don't completely agree here. It should be possible to use drivers with a very small implementation, right. For a network card something like send packet, receive packet, that's it. But then, at some point you probably want to use more features of the hardware. So maybe consequently using HAVE_XYZ macros defined by the OS side implementation would make sense - then the OS can decide which part of the driver it wants to use.

Ok, after my last post in this thread I managed to implement EDI in my OS (not quite a complete implementation, but it's able to load the sample driver) and I'm going to try to write a floppy driver for it.
See attached for the implementation from Acess (see the [link=http://code.google.com/p/acessos/]acess website[/link] for a look at the Acess driver interface and the EDI helper files)

Amazing how much code EDI needs to achieve simple things like accessing IO ports.

You don't seem to have any EDI drivers yet in your repository? Would be curious how the driver side looks like.

[gravaera]

Okay: this is it: Why don't we all create a Portable Driver Interface? I'm willing to work on it and produce a specification, seeing as I'm currently in a design phase for my kernel, and a method has pretty much presented itself to me. I haven't really read through the specification for the EDI, seeing as it doesn't really seem to have a large base of implementation.

Because it doesn't help if everyone is doing his own portable interface. In this thread at least three different, more or less existing, interfaces have been mentioned until now. I'm not sure what a "large base" does mean for you, but I know that your new interface currently doesn't have a single user.

I'm creating it for easy, flexible use. I'm not saying anyone has to use it, or that anyone will. I'm more or less hoping that it will catch on when I DO start to promote it, though. I DO have a plan to make it seem more attractive later, after I've developed it fully. Don't worry: I think things through

1. Drivers should be as absolutely bare-minimum as is possible.

I don't completely agree here. It should be possible to use drivers with a very small implementation, right. For a network card something like send packet, receive packet, that's it. But then, at some point you probably want to use more features of the hardware. So maybe consequently using HAVE_XYZ macros defined by the OS side implementation would make sense - then the OS can decide which part of the driver it wants to use.

To the first boldened snippet: I didn't quite understand this part of what you said. It seems you're trying to say that it would be possible for people to use drivers with a bare-minimum interface. If that is what you were trying to say, then yes, I know that. That's why I believe in a bare-minimum driver interface being best for OS Portability.

To the second: My post does cover hardware with added features. And with the sketches I made out two days ago, there is a presentation layer for extra features. The PDI I'm going to release sometime from now will cover that.

HAVE_XYZ macros would be inefficient for automated driver compatibility checking. That would make it static. The PDI will define an interface that will consist of function pointers and relevant data all encapsulated within a struct. There are several compulsory PDI functions, such as autoDetect(), and identify(), which the PDI compliant struct must implement. The autoDetect() function, for example, will enable a kernel developer to write code that runs through a list of PDI compatible drivers, and just runs autoDetect() to see whether or one or more of them can be used on the hardware that needs to be configured. The PDI driver will return avalue of (1)True, or (0) False, depending on whether or not it detects a device in the system that it can work with.

It won't be ready anytime soon, but when it is, it will be simple, yet powerful. And will have full support and documentation. Whether or not anyone uses it is entirely up to them.

...But the lure of having a large range of divers already implemented in PDI form posted up on the Wiki will be most likely too strong. When I DO come out with it, I'll be posting up the source for all the drivers I'd have implemented with it so far. That would most likely include an HDD, console, mem-Mgr-interface, and IRQ-Handler interface. Because by the time I finish those, I would have already finished drafting out the PDI, since my OS would need to be using the PDI.

[Kevin]

Because it doesn't help if everyone is doing his own portable interface. In this thread at least three different, more or less existing, interfaces have been mentioned until now. I'm not sure what a "large base" does mean for you, but I know that your new interface currently doesn't have a single user.

I'm creating it for easy, flexible use. I'm not saying anyone has to use it, or that anyone will. I'm more or less hoping that it will catch on when I DO start to promote it, though. I DO have a plan to make it seem more attractive later, after I've developed it fully. Don't worry: I think things through

Well, I'm just saying that the approach of not even looking at existing models because the user base is too small and considering a new interface isn't consequent. A new interface, by definition, doesn't have any users and consequently wouldn't be worth to look at either. Looking at existing things and then doing something new because there wasn't the right one among them is a different thing.

To the first boldened snippet: I didn't quite understand this part of what you said. It seems you're trying to say that it would be possible for people to use drivers with a bare-minimum interface. If that is what you were trying to say, then yes, I know that. That's why I believe in a bare-minimum driver interface being best for OS Portability.

Right, this is what I was saying.

To the second: My post does cover hardware with added features. And with the sketches I made out two days ago, there is a presentation layer for extra features. The PDI I'm going to release sometime from now will cover that.

I had the impression that everything that goes beyond very basic operations would be driver specific. Not sure if I understood that correctly, though.

HAVE_XYZ macros would be inefficient for automated driver compatibility checking. That would make it static. The PDI will define an interface that will consist of function pointers and relevant data all encapsulated within a struct. There are several compulsory PDI functions, such as autoDetect(), and identify(), which the PDI compliant struct must implement

Oh, so you're actually working towards a different goal from what I thought. "Portable" sounds like "you can take the driver source and use it in your own OS easily". What you really mean is a binary driver interface, right? This makes the world look different, obviously, even though I don't think this approach helps you a lot more compared to the source base variant. It just makes the design a bit more complicated.

...But the lure of having a large range of divers already implemented in PDI form posted up on the Wiki will be most likely too strong. When I DO come out with it, I'll be posting up the source for all the drivers I'd have implemented with it so far. That would most likely include an HDD, console, mem-Mgr-interface, and IRQ-Handler interface. Because by the time I finish those, I would have already finished drafting out the PDI, since my OS would need to be using the PDI.

Again, what does a "large range of drivers" mean? I certainly wouldn't consider a HDD and a console driver a large range. In this case, the range of available CDI drivers (which I still consider small) is larger.

[thepowersgang]

[quote=Kevin]
You don't seem to have any EDI drivers yet in your repository? Would be curious how the driver side looks like.
[/quote]

I've uploaded the drivers directory, including the sample driver converted to C and the start of a FDD driver.
Also, the EDI headers there are different to the official versions to allow them to compile in GCC (it doesn't allow external variables to be initialised)

Does anyone know how I can get the FDD driver to allow the user to select the disk they want at construct time?

[gravaera]

As a slight update, I went and read the EDI specification, and it actually is very solid. I suppose that takes a load off my back where driver design is concerned. I'll be using it as soon as I start hardcore driver stuff.

Anyway: I hope to be able to lend a good bit of support to spreading EDI. It's a great interface, and just requires a bit of reading. Versatile enough to implemented on multiple, unrelated OSs without a problem.

I'm in the bandwagon.

[xvedejas]

I think someone ought to make a central repository for EDI conforming drivers. It'd be interesting at least.

[thepowersgang]

I've started a page on EDI at Extensible_Driver_Interface that has the start of a list of OSs and drivers that use EDI, if you've implemented EDI in your OS or have written a driver please add it to the list.

[Solar]

Well... together with Love4Boobies I've set up a central location for UDI drivers and references, because I still think it's the strongest concept of them all...

That's http://udi.rootdirectory.de (not much there yet but we only started yesterday).

I could do the same for EDI, but... well... I still think it's kind of stupid to spawn new standards when there's a strong one already existing, and I haven't been convinced yet on the advantages of EDI over UDI. Convince me, and I'll set up edi.rootdirectory.de as well.

[gravaera]

@Solar: I like the idea that UDI allows local inclusion of Win32 drivers. That's a big plus.

My OS is going to follow after Win32 more than Linux, where drivers and API are concerned. One of my main goals is to ensure that most important Win32 programs will run on my OS, naturally, without any problems.

I have read up on the legal implications of Compatibility Layers and it seems that locally implementing a Win32 API is not purely illegal. I need to do much more reading before I can confidently say that I'd be safe doing it, but if it can be safely done, then Win32 driver compatibility is definitely a plus for me.

At the end of all this, though, I may still, it seems, have to implement my own driver structure. I'm not sure how well this may go. It's pretty frustrating.

[Solar]

I like the idea that UDI allows local inclusion of Win32 drivers.

I am not sure what you mean by that, and I am afraid that you might misunderstand what UDI is about.

[Brendan]

Hi,

Let's try something, just for fun. Let's try to invent a common/generic device driver interface for video cards and nothing else (and forget about all the other types of devices for now).

Here's how it will be done. Everyone describes how the video device driver that they've already planned for their OS is meant to work, and the winning design is the design that's described by the most people. Also, when (if?) no video driver design is described more than once we all finally realize how futile any attempt at creating common/generic device driver interface actually is.

Are you ready? GO!


Cheers,

Brendan

[Solar]

I know what you are aiming at, Brendan, and indeed I don't think it is possible to create a generic driver interface e.g. for 3D-accelerated gfx cards.

But UDI is a functional specification (I just don't know enough about EDI to comment), and if "all" those generic approaches give to the OS dev community is a common set of drivers for hard drive controllers, serial and USB ports, and network cards, that's nothing to be sneered at IMHO.

[Brendan]

Hi,

Ok, I thought it's only fair that I start things off by describing my video device drivers...

----8<----

First, everything in my OS uses asynchronous messaging, where different pieces of software need to use the same protocol (e.g. same message identifiers, etc) to be able to reliably talk to each other. One of these messaging protocols is called the "user interface messaging protocol", and it covers everything that is used to get input from users and output data for the user (video, sound, keyboard, mouse, joystick, touchpad, microphones, etc). There may be a hierarchical tree of processes. For example, messages from applications (using this "user interface messaging protocol") get sent and received to/from the application's parent (e.g. a GUI), and this parent (e.g. the GUI) may send/receive messages (in the exact same "user interface messaging protocol") to/from the parent's parent (e.g. a virtual terminal layer). Eventually, near the top of this hierarchical tree of processes there's a layer that talks directly to device drivers. This is important because this particular layer is responsible for splitting things up - the sound related stuff gets forwarded to the sound card driver, the keyboard related stuff gets forwarded to the keyboard driver, and the video related stuff gets forwarded to the video driver.

Basically, the video driver must support the subset of the "user interface messaging protocol" that relates to video.

This subset of the "user interface messaging protocol" mainly involves a "video script" that describes a complete frame of video using graphics primitives - what should be drawn by the video driver, and where (on the screen) the video driver should drawn them. For example, the script might say "draw a solid box from (x1, y1, z1) to (x2, y2, z2) using a specific colour". In all cases, all colours are specified in CIE XYZ; and all co-ordinates are 3 dimensional virtual coordinates that range from 0x00000000 to 0xFFFFFFFF (for e.g. the middle of the screen would be at the co-ordinate (0x80000000,0x80000000,0x80000000). This means that the values used to represent colours and the values used to represent co-ordinates never have anything to do with the colour depth or resolution of the video mode currently being used by the video driver; and that the video driver is free to automatically change video modes for any reason. The video driver is responsible for choosing a video mode automatically (which must include negotiation with the monitor - DDC/EDID), and changing this video mode (and the quality of the primitives drawn) to ensure that it takes no more than 30 ms to drawn a complete video script.

In addition, the video driver may/should cache the results of drawing parts of a video script, so that these parts can be redrawn quickly. For example, a video script may describe something that is reused later, and the video driver can draw that thing once and cache the results, and then draw the cached results every time the any video script describes the same object. Video scripts will be subdivided to make this easier (for example, the video script from an application might be split into 4 separate zones, where the first zone describes the menus at the top, the second zone describes a toolbox on the left, the third zone describes the main working area, and the fourth zone describes a status bar; where the beginning of the script contains instructions on where these zones should be placed relative to each other). The video script may ask for these zones to be translated, rotated and scaled (in 3D space) by the video driver, so the video driver needs to support that to.

In addition the video script may include references to files that contain other video scripts, or bitmap data, etc. The video driver is responsible for loading and caching these files, and drawing their contents.

Video scripts may also include (Unicode, UTF-32) character data. It is the video driver's responsibility to communicate with the font engine to get any character data converted into suitable data of the appropriate size/shape and then draw the resulting data from the font engine.

It is recommended (but not necessarily required) that the video driver uses page flipping (or double buffering), and switches between frames using vertical synchronization (e.g. during a vertical retrace IRQ). It is also recommended (but not necessarily required) that the video driver is able to track performance metrics and sends warnings when certain conditions occur, including when its client is trying to update the video too frequently (e.g. if the frame rate from an application exceeds to vertical refresh rate of the current video mode), and when a new frame of video is exactly the same as the current/previous frame of video. When enabled, these warnings are sent by the video driver to a performance monitoring tool.

When the video driver is first started, the operating system will tell it which resources (I/O ports, IRQs and areas of the physical address space) correspond to the video card. The video driver must not attempt to use any other resources, except for RAM. Typically the operating system knows which resources correspond to the video card by examining the device's PCI configuration space (note: the video driver does not have access to the video card's PIC configuration space). In some cases (e.g. an ancient ISA video card) a separate piece of code is responsible for detecting the device and finding out which resources (I/O ports, IRQs and areas of the physical address space) the video card is using; but this has nothing to do with the video driver itself (it's beyond the scope of the video driver). These resource may be virtual resources, and may not be the actual resources used by the video card. For example, the video card might generate IRQ 0x09 when a vertical retrace occurs, and the operating system might tell the video device driver that it's using IRQ 0x00, and when IRQ 0x09 occurs the operating system tells the video driver that IRQ 0x00 occured.

Video drivers must also support virtualization. This is a separate interface to the video driver that uses a different messaging protocol (the video driver has 2 modes: normal and virtualized; and either the "user interface messaging protocol" or the "virtualized device messaging protocol" are used, never both at the same time). In the "virtualized" mode, something (e.g. an emulator) can connect to the video driver, and tell the video driver to read or write to/from certain I/O ports and memory mapped areas; and when an IRQ occurs the video driver forwards the IRQ to the client. In this case the video card must do what it's told by the emulator, except that anything that involves the device transferring data to/from RAM (e.g. DMA or bus mastering) must be virtualized by the device driver. For the purpose of testing, the operating system will allow one instance of the video driver (operating in "virtualized" mode, using the real hardware) to be used by another instance of the same video driver (operating in "normal" mode, using the first instance of the video driver instead of the real hardware *without* knowing it). All video drivers must operate correctly in this situation, and the operating system reserves the right to use this test as a compliance test (and refuse to start the device driver if it fails).

Also, all device drivers run as normal processes (e.g. at CPL=3). This means that a video driver can use multiple threads (and it's recommended that for any heavy processing the video driver does use multiple threads - e.g. one per CPU).

It's recommended (but not strictly required) that unused video card RAM be made available to the operating system in the form of swap space. The messaging protocol used for swap space is fairly simple (mostly it consists of "store page" and "fetch page"), but won't be covered here. For supporting "VRAM as swap space", the video driver should use a separate thread with a higher priority than all other threads used by the video driver.

----8<----

It's missing a lot of details, but it should provide a good enough idea of how video drivers for my OS should work.

I'm hoping that someone else's video drivers are designed in exactly the same way, so that this becomes the common/generic device driver interface for video cards...


Cheers,

Brendan

[Combuster]

Are you ready? GO!

Considering that I am developing from an exokernel perspective, that behavior is reflected in the main layout of the "graphics driver". In practice, the graphics interface is a layer between the application and the low-level driver, so that it can take any number of outputs, and multiplex the graphics between several (desktop) programs and other viewports, and it is this layer that provides X / OpenGL / xxx functionality to the actual programs. This functionality may be redirected or emulated, but it may also be a 1:1 mapping of driver calls.

The actual low-level drivers do not have formal functional requirements, other than reporting the functions it supports. The calling layer can then check if the functionality provided is sufficient for the goals it wants to achieve. This allows for video hardware beyond framebuffer devices (including "tiling engines" of which VGA text mode is the most familiar, but also serial terminal screens and LCDs up the ranks of the 2D console GPUs)

The common case would be that the driver exposes calls for
- querying output port(s), whether VGA, DVI or something else
- connecting a display engine to a port
- set the parameters of that engine (resolution, etc)
- getting access to video memory (to be able to write pixels in the)
Other calls may include
- VGA related calls
- calls to identify fixed function processors and send commands to them (hardware accelerators, VGA read/write logic)
- calls to identify processors (for GPU programming of any sort) and for compiling code fragments in one or more languages.

Communication is done via message passing, using pipes which are pieces of memory shared between the driver and the client. The driver runs in userspace, although the kernel allows proven safe code to be run in kernel space in case of an IRQ (which can be used to achieve normal microkernel functionality such as sending messages, but it may also be used to implement triple buffering without the need of a task switch)

Drivers may also decide to give direct access to resources to clients, by transferring code to the client and give them direct access to certain I/O ports and memory to give them an even greater speed advantage although potentially at the cost of some security. The multiplexing layer is responsible for implementing the security aspects.

_________________

I don't think it is possible to create a generic driver interface e.g. for 3D-accelerated gfx cards.

So we are to forget about piping OpenGL commands to a driver, no? In fact, OpenGL does define a format of marshalling gl calls. And it regulates extensions so the system is quite future safe (except for some corner cases described in Brendan Paranoia(tm)). All that's left to do is define an communications pipe, and a mechanic to switch application contexts.

I'm hoping that someone else's video drivers are designed in exactly the same way, so that this becomes the common/generic device driver interface for video cards...

In practice, my video drivers are close to a subset of your style video drivers. It should be possible in my OS to define the middle layer as having the interface you described, which means that you can adapt my drivers to your system without losing functionality (and saving some memory by sharing the common code). I'd only need to drop the use of the advantages specific to my OS. (I could settle for that, they don't work yet anyway )

I think that focusing on the actual interface is not the right way compared to the interface design.

and the winning design is the design that's described by the most people.

Well then, lets post two null interfaces and get this over with
*ducks and runs*