UDI, EDI, CDI Which do you use and why

[Kevin]

it might be wise to actually use the scientifically accepted definitions, instead of turning "lock" into something as meaningless as "anything that waits" which only ever serves the purpose of being unable to use the term.

Agreed, it wasn't the best term to use here. It comes from the fact that I like to think in terms of coroutines rather than functions split across multiple callbacks, and that I'm used to an environment where coroutines and callback-based AIO are both used and people are aware that they're equivalent. I shouldn't have said "taking a lock", but to be honest, I'm having a hard time finding a short replacement that includes all forms of letting other requests wait.

I still think that's it's fair to call a request "blocked" when it's sitting in a queue, waiting for another request to complete.

[Brendan]

Hi,

Hi im currently re-working on my module loader to be compatible with other hobby os and I honestly don't know which one to implement i started with cdi but couldnt find enough info on it... So what do you use in your os and why?

I've never had any intention of using any of them.

If you look at UDI, you'll find that the specifications themselves only cover some stuff (e.g. disk drives and network cards) and doesn't cover other stuff (e.g. 2D video, 3D video, GPGPU, sound input/output, GPS, accelerometers, cameras, printers, scanners, force feedback, etc); and the devices it does cover are relatively easy to support without UDI, and the devices it doesn't cover are hard. Of course things get much worse when you try to find any actual drivers (you will only find drivers for simple and obsolete hardware).

Worse; if you look at the UDI specs themselves you'll notice they do "minimal functionality" well; but anything beyond that either doesn't exist or is platform dependent (and therefore non-standard and useless). For example, for storage devices:

• If you want whole disk encryption, you're screwed
• If you want your a sane IO priority scheme, you're screwed
• If you have special synchronisation requirements (e.g. "notify me if sector XYZ is modified") you're screwed
• If you want additional special purpose interfaces (e.g. possibly one designed for emulators, where a disk controller is assigned to a virtual machine such that the guest OS inside the virtual machine has "almost direct hardware access") then you're screwed
• If you want "secure erase" then you're screwed
• If a file system needs to "trim", then you're screwed
• For any/all power management, (e.g. turning hard disks off when they're idle, disabling pre-fetching when you're running on battery, etc) you're screwed

In addition to all of that; you end up with another layer of bloat. Rather than having drivers that support the OS's native interfaces directly, you end up with a bunch of additional "shims" between the OS's native interface/s and UDI.

Finally; if you ever actually do find a driver that supports UDI, you still can't assume it will actually work - you have to do your own "quality assurance" testing to be sure (which means finding/purchasing the appropriate hardware/device); and maintain some sort of blacklist/whitelist (especially when you start considering the possibility of "trojan UDI malware"). You also might not be able to (e.g.) bundle the driver with your OS, because the copyright isn't yours and may be "undesirable" for whatever reason (e.g. proprietary, GPL), and/or because the driver may be owned by a known patent troll like SCO.

The end result is that you need to do a lot of work just to support UDI, and in return for this large amount of work you get a few irrelevant drivers, where implementing the drivers yourself (with or without "cut & paste" from open source drivers) would've been faster/easier, far more flexible and result in a better (less bloated) OS.

CDI and EDI are similar to UDI (worse in many areas and better in some) - the effort needed to support any of them is more hassle than it's worth.


Cheers,

Brendan

[Combuster]

If you want whole disk encryption, you're screwed

If you expect whole-disk encryption to be coded into every driver, you're mentally screwed.

If you want your a sane IO priority scheme, you're screwed

If you want to duplicate your scheduler into every driver, you're mentally screwed.

If you have special synchronisation requirements (e.g. "notify me if sector XYZ is modified") you're screwed

Hey dude, you wrote that sector, why don't you go notify yourself for a change?

If you want additional special purpose interfaces (e.g. possibly one designed for emulators, where a disk controller is assigned to a virtual machine such that the guest OS inside the virtual machine has "almost direct hardware access") then you're screwed

If you don't read specifications, you're screwed.

If you want "secure erase" then you're screwed

If you don't know the SCSI commands to rewrite a specific sector, you're screwed even harder if you do it on your own.

If a file system needs to "trim", then you're screwed

What was the SCSI command for that again?

For any/all power management, (e.g. turning hard disks off when they're idle, disabling pre-fetching when you're running on battery, etc) you're screwed

You can actually turn off devices, and you can define another interface for power management.


Seriously Brendan, I have yet to hear an actual design choice of your os that UDI does not cater for.

[gravaera]

Yo:

I don't normally press issues and pursue an argument like this, but I think it would be very meaningful to prove that UDI does cater to all of these needs quite ably, mostly for the benefit of other readers who might be interested in adopting it. This post is terribly long, but it needed to be this long because Brendan raised a lot of points, each of which needed a decent amount of expansion.

If you look at UDI, you'll find that the specifications themselves only cover some stuff (e.g. disk drives and network cards) and doesn't cover other stuff (e.g. 2D video, 3D video, GPGPU, sound input/output, GPS, accelerometers, cameras, printers, scanners, force feedback, etc); and the devices it does cover are relatively easy to support without UDI, and the devices it doesn't cover are hard. Of course things get much worse when you try to find any actual drivers (you will only find drivers for simple and obsolete hardware).

All of these can easily be supported by any individual; this is the fundamental purpose of Metalanguages. They exist to enable new APIs for device-classes to be created and drafted as needed. If you have a non-ubiquitous device that you wish to deploy, you can define your own custom API, deploy it, and submit your draft API for peer review. Over time, the API will be rounded off by the UDI Project and the community of UDI participants, and when the "Standard" UDI API is agreed upon, you can then use that Metalanguage in your new deployments.

In addition, it is not in any way weird for a driver to export multiple APIs. Many graphics drivers export both OpenGL and D3D. This is nothing novel or unique to UDI. As expected of a framework in its infancy, it will have areas you will need to pioneer. If you don't like pioneering, feel free not to use UDI.

Worse; if you look at the UDI specs themselves you'll notice they do "minimal functionality" well; but anything beyond that either doesn't exist or is platform dependent

This is a false assertion, as I will demonstrate below. By individually tackling each of these bullet-points.

If you want whole disk encryption, you're screwed

A driver that supports full-disk encryption can and will do this regardless of the presence or absence of OS APIs for it. It needs only import the necessary libraries and it will be able to encrypt and decrypt the data. If there is a need to enable the OS to give commands or configuration directives to the driver, in UDI this can be easily and readily supported using the UDI Generic I/O metalanguage.

A custom userspace utility can be created, and this utility can communicate with the driver to control the encryption settings, etc, and it will communicate with the driver over the UDI Generic I/O Metalanguage child-bind channel. A fully compliant UDI OS doesn't need to know about the APIs that a driver exports or what they do; Programs can use them either way, and OS APIs don't need to be a "middle-man". This isn't unique to UDI either; IOCTLs, etc are similar in nature.

If you want your a sane IO priority scheme, you're screwed

Only the kernel has the metadata required to know which processes' requests should be considered high-priority, and usually the I/O scheduler is a feature of the OS. If you wish to do aggressive I/O prioritization, you can fragment requests at the kernel level, and send them to the device in fragmented portions. This will enable you to have a higher chance of applying very fine I/O prioritization to requests. This would be the case whether or not you used UDI, so this is also nothing particularly revealing.

If you have special synchronisation requirements (e.g. "notify me if sector XYZ is modified") you're screwed

This point is fair...but only in so much as this is not a commonly traveled path by contemporary kernels. Usually, you do notification at the file-system level, on the file-level abstraction, and not on the sector-level concretion. This is a very odd feature to want at all.

However, should you want to give applications in your userspace notifications when particular hardware sectors are written to, you can do this very easily in UDI anyway: all requests that go both in and out of a UDI driver will pass through a UDI Metalanguage Library, and this is unavoidable. You can write a custom Metalanguage Library for your kernel that will intercept all IPC for say, disk-I/O, and implement this notification framework out-of-band, without needing to modify any of the drivers. From the documentation for udi_mei_driver_error():

The metalanguage library is not required to check for invalid parameters or other illegal conditions, and should normally not check for invalid parameter values that would (per the metalanguage definition) be expected to be checked in the target driver. Illegal conditions that are detectable only by a portable metalanguage library (not by either the driver or the environment) should be checked for in untrusting metalanguage library implementations.

By implication, Metalanguage Libraries can be used as "interceptors" to do additional, API-specific error checking and security operations, though this is not required. You can then extend this to suit your needs, and provide a UDI Disk Metalanguage library that will intercept disk I/O requests (on the ACK/response side of the asynchronous call) and dispatch messages to the kernel if the sectors involved match those you need notifications for. The kernel can then dispatch notifications to the applications which are listening for such notifications. Note that no driver support or modification of driver code is necessary, and you can add this functionality even to drivers that are distributed only in binary form. That is, you can do it without needing to have the source code for the disk driver, and do it in a generic, portable way too.

I suspect that Brendan wants this feature for security. Perhaps he wants to monitor for changes to the Volume Boot Sector, etc. At any rate, UDI can more than ably, and portably accomplish what he wants.

If you want additional special purpose interfaces (e.g. possibly one designed for emulators, where a disk controller is assigned to a virtual machine such that the guest OS inside the virtual machine has "almost direct hardware access") then you're screwed

This is simply not true, either for UDI, or for any other kernel's disk API. If you want to connect at a low level to a device, you can make use of IOCTLs or that kernel's equivalent. This is not a problem in any modern kernel, tbh, unless I am misunderstanding what you're advocating :O

If you want "secure erase" then you're screwed

When a UDI Storage/Disk Metalanguage is drafted, I will be very astounded if it does not include support for secure erasure, given that this is a very ubiquitous disk driver feature.

If a file system needs to "trim", then you're screwed

I don't know enough about SSDs to comment authoritatively.

For any/all power management, (e.g. turning hard disks off when they're idle, disabling pre-fetching when you're running on battery, etc) you're screwed[/list]

Power management is handled by udi_devmgmt_req(), and in combination with ACPI, I don't see why this is impossible. Out-of-band special commands can be sent to configure the driver using a userspace utility if absolutely necessary. This would not be unusual either. Conventionally today, we see many such drivers: an AMD Catalyst Driver for example, comes with a userspace daemon that augments the kernel with support for its less universal features. This is not novel behaviour.

If a kernel wishes to specify custom APIs for hardware vendors to allow it to control certain aspects of devices, it can specify such an API and try to get hardware vendors to adopt it in their drivers. Then, those kernels which are interested in the custom API can establish child-bind channels on that custom Metalanguage and do whatever they need to do with those custom APIs (enable_prefetch_req()/disable_prefetch_req()). The only problem here is Independent Hardware Vendor adoption of your custom requirement, and the absence of UDI does not improve your prospects here.

In addition to all of that; you end up with another layer of bloat. Rather than having drivers that support the OS's native interfaces directly, you end up with a bunch of additional "shims" between the OS's native interface/s and UDI.

If you implement the UDI environment natively, you will gain native support, and thereby native performance. If you implement the UDI environment as a layer on top of another interface, you will gain second-class-citizen performance, since your UDI support is second-rate. This argument is like saying that "you end up with shims between the OS's native interface and WINE when you use WINE to support the NT API."

Finally; if you ever actually do find a driver that supports UDI, you still can't assume it will actually work - you have to do your own "quality assurance" testing to be sure (which means finding/purchasing the appropriate hardware/device); and maintain some sort of blacklist/whitelist (especially when you start considering the possibility of "trojan UDI malware"). You also might not be able to (e.g.) bundle the driver with your OS, because the copyright isn't yours and may be "undesirable" for whatever reason (e.g. proprietary, GPL), and/or because the driver may be owned by a known patent troll like SCO.

This is not different from any other software that is distributed; drivers are software, like applications. If you want a "trusted driver" framework, you can set about doing this with relative ease. The only barrier is time.

The end result is that you need to do a lot of work just to support UDI, and in return for this large amount of work you get a few irrelevant drivers

The same goes for inventing a custom API. The only difference is that UDI is a very well-thought-through, scalable API, and you will not need to go through design re-iterations with UDI. With your own custom API, due to short-sightedness, the average kernel developer in this community will spend several years butting his head before he eventually, after many re-iterations, settles on something that is adequately well-designed and stable.

All the while, he would have needed to repeatedly update all of his drivers each time he had to redesign. UDI saves him all of this time. It looks like a lot of time spent up-front on implementing UDI, but this is regained in time saved not having to regressively redesign and update dozens of drivers repeatedly, possibly 15, 20, maybe 30+ times over 2-3 of years, and never having a stable driver API.

--Peace out,
gravaera

[gravaera]

Yo:

In addition, I have no intention in engaging in back-and-forth on the merits of UDI. In essence, my argument is that UDI supports anything you need it to, and for those areas where you need outlandish functionality, it supports those too, via Metalanguages.

I understand that there is a gross, terrible aversion to reading specifications on this forum, and that people refuse to read anything, but the ultimate answer to "UDI can't do this" and, "UDI can't do that", is usually going to be, "Yes, UDI can, and probably already does both this, and that. Go and read the specification."

I understand that people don't like to read specifications, though, so topics about UDI will keep popping up every 6 months, and a raging argument will ensue where people who haven't read the specification will keep saying that "UDI doesn't do this", and people who have read the specification will keep saying, "UDI does this, and more."

This will continue until about 8 months from now when my environment is complete, and my kernel is ready for its 0.01 release, and I'll be able to join these arguments and say, "UDI is doing this right as we speak", and link to code.

--Peace out,
gravaera.

[Kevin]

Trying to convince Brendan to use something existing is usually a waste of time, no matter whether it's UDI or anything else.

I don't think anyone can seriously attack UDI for being incomplete when talking about hobby OSes. For me, it's really just the two points I mentioned earlier: The spec is too complex for simple OSes and free drivers are missing.

[AndrewAPrice]

I can't put my finger on it, but the GNU response on UDI (http://www.gnu.org/philosophy/udi.html) sounds a lot like Republican-logic: (no offense here, the writing style and how they walk into conclusions are very similar)

If we imagine a number of operating systems and hardware developers, all cooperating on an equal footing, UDI (if technically feasible) would be a very good idea.

When we apply the idea to the actual world, which contains both free software developers seeking cooperation, and proprietary software developers seeking domination, the consequences are very different. No way of using UDI can benefit the free software movement. If it does anything, it will divide and weaken us.

Given these consequences, it is no surprise that Intel, a supporter of UDI, has started to “look to the Linux community for help with UDI.” How does a rich and self-seeking company approach a cooperating community? By asking for a handout, of course. They have nothing to lose by asking, and we might be caught off guard and say yes.

[Brendan]

Hi,

If you want whole disk encryption, you're screwed

If you expect whole-disk encryption to be coded into every driver, you're mentally screwed.

Who said anything about "every driver"? It might be something that you only use for USB flash, and only because USB flash devices are easily stolen.

If you want your a sane IO priority scheme, you're screwed

If you want to duplicate your scheduler into every driver, you're mentally screwed.

IO priorities have nothing to do with thread priorities and nothing to do with the OS's "thread/process scheduler". Essentially it effects the order that operations are performed by the device (e.g. new higher priority requests may skip ahead of older lower priority requests) and must be in all drivers that allow requests to be queued. This includes virtually all modern disk drivers as the hardware itself accepts queues (e.g. NCQ) to optimise throughput. It also doesn't preclude the option of forcibly cancelling an operation (e.g. a lengthy very low priority disk read) that has already been started by the device itself, when a higher priority request arrives.

If you have special synchronisation requirements (e.g. "notify me if sector XYZ is modified") you're screwed

Hey dude, you wrote that sector, why don't you go notify yourself for a change?

For a simple example, imagine the OS is shutting down; and sends an (asynchronous) write to the disk driver, then kills the power. Do you think it might be nice if the OS was able to know that the write has actually occurred (e.g. been written to the physical disk and not just written to a cache somewhere, and not just written to the hard drive's internal cache)?

If you want additional special purpose interfaces (e.g. possibly one designed for emulators, where a disk controller is assigned to a virtual machine such that the guest OS inside the virtual machine has "almost direct hardware access") then you're screwed

If you don't read specifications, you're screwed.

If you don't read the specifications, or if you do read the specifications, you're screwed.

If you want "secure erase" then you're screwed

If you don't know the SCSI commands to rewrite a specific sector, you're screwed even harder if you do it on your own.

If a file system needs to "trim", then you're screwed

What was the SCSI command for that again?

What is the standard method defined by UDI for doing both "secure erase" and "trim", that works for all devices (including devices that are not SCSI)?

For any/all power management, (e.g. turning hard disks off when they're idle, disabling pre-fetching when you're running on battery, etc) you're screwed

You can actually turn off devices, and you can define another interface for power management.

Most devices have several options (e.g. ranging from "full speed, full disk noise" to "low power, low disk noise"). A simple "on or off" isn't even close to adequate.

You can define your own interface, and as soon as you do that you've completely lost any advantage from using an existing standard - device drivers written for other OSs will not support your own personal interface.

Seriously Brendan, I have yet to hear an actual design choice of your os that UDI does not cater for.

I think you're confusing "hearing" with "understanding" - you've heard several and understood none.


Cheers,

Brendan

[Brendan]

Hi,

If you look at UDI, you'll find that the specifications themselves only cover some stuff (e.g. disk drives and network cards) and doesn't cover other stuff (e.g. 2D video, 3D video, GPGPU, sound input/output, GPS, accelerometers, cameras, printers, scanners, force feedback, etc); and the devices it does cover are relatively easy to support without UDI, and the devices it doesn't cover are hard. Of course things get much worse when you try to find any actual drivers (you will only find drivers for simple and obsolete hardware).

All of these can easily be supported by any individual; this is the fundamental purpose of Metalanguages. They exist to enable new APIs for device-classes to be created and drafted as needed. If you have a non-ubiquitous device that you wish to deploy, you can define your own custom API, deploy it, and submit your draft API for peer review. Over time, the API will be rounded off by the UDI Project and the community of UDI participants, and when the "Standard" UDI API is agreed upon, you can then use that Metalanguage in your new deployments.

The entire point of using UDI is that (in theory) you can just use device drivers written by other people. You can create your own non-standard extensions within the UDI framework, but as soon as you do that you can't use device drivers written by other people (and expect them to support your non-standard extensions), so it's no better than just doing native drivers.

If you want whole disk encryption, you're screwed

A driver that supports full-disk encryption can and will do this regardless of the presence or absence of OS APIs for it. It needs only import the necessary libraries and it will be able to encrypt and decrypt the data. If there is a need to enable the OS to give commands or configuration directives to the driver, in UDI this can be easily and readily supported using the UDI Generic I/O metalanguage.

A custom userspace utility can be created, and this utility can communicate with the driver to control the encryption settings, etc, and it will communicate with the driver over the UDI Generic I/O Metalanguage child-bind channel. A fully compliant UDI OS doesn't need to know about the APIs that a driver exports or what they do; Programs can use them either way, and OS APIs don't need to be a "middle-man". This isn't unique to UDI either; IOCTLs, etc are similar in nature.

So, you admit that UDI is inadequate for this case, and suggest using the UDI framework to create OS specific work-arounds that are no better than native drivers.

If you want your a sane IO priority scheme, you're screwed

Only the kernel has the metadata required to know which processes' requests should be considered high-priority, and usually the I/O scheduler is a feature of the OS. If you wish to do aggressive I/O prioritization, you can fragment requests at the kernel level, and send them to the device in fragmented portions. This will enable you to have a higher chance of applying very fine I/O prioritization to requests. This would be the case whether or not you used UDI, so this is also nothing particularly revealing.

Imagine if there's 256 IO priorities (where 0 is lowest priority and 255 is highest priority); and a single thread asks to read 1 sector at LBA 4 as priority 127 and also asks to read sector at LBA 123456 as priority 128. If the disk drive's heads happen to be in the middle of the disk, you want to read LBA 123456 then LBA 4, because there's a tiny priority difference. If the disk drive's heads happen to be on the cylinder that contains the sector at LBA 4, then you want to read LBA 4 then LBA 12345 because that tiny priority difference isn't enough to justify the additional seek.

Now try explaining how UDI supports this in a standard way; instead of avoiding the issue by proposing half baked work-arounds for UDI's inadequacies.

If you have special synchronisation requirements (e.g. "notify me if sector XYZ is modified") you're screwed

This point is fair...but only in so much as this is not a commonly traveled path by contemporary kernels. Usually, you do notification at the file-system level, on the file-level abstraction, and not on the sector-level concretion. This is a very odd feature to want at all.

Covered in my previous reply to Combuster.

If you want additional special purpose interfaces (e.g. possibly one designed for emulators, where a disk controller is assigned to a virtual machine such that the guest OS inside the virtual machine has "almost direct hardware access") then you're screwed

This is simply not true, either for UDI, or for any other kernel's disk API. If you want to connect at a low level to a device, you can make use of IOCTLs or that kernel's equivalent. This is not a problem in any modern kernel, tbh, unless I am misunderstanding what you're advocating :O

All I'm saying here is that UDI doesn't (and can't) provide standardised OS specific APIs for "more than basic functionality".

For one example, for my OS I want all device drivers to support a special "direct access by emulator" interface (where possible); where the emulator detects reads/writes to the device's registers made by the guest OS and sends messages to the device driver (e.g. saying things like "write the value 123 to the device's IO port <base+4>") where the device driver acts as a middle-man between the emulator and the actual device (to ensure security and to restore the device's state if/when the emulator is terminated). The idea is to allow both real and virtual devices to use the same messaging protocol, and to allow real devices to be assigned to virtual machines.

For another example, for my OS I want all device drivers to track usage statistics for maintenance purposes - how many times has a disk done a "seek", how far has a mouse moved, how many times have keys been pressed on a keyboard, etc. The idea is that you have a distributed system of many computers, where there's a utility that the informs "the maintenance guy" that (e.g.) the mouse on computer 123 has done 12345 meters of movement since it was serviced last and is due for cleaning/service, or the hard disk on computer 33 has done 1 million "seek" operations and should be replaced because that model of hard disk is likely to fail after too many seeks, etc. It's basic preventative maintenance, but means that device drivers need to keep track of a bunch of statistics and also provide "mean time between failure" information.

Can I (or anyone) expect a random UDI driver (written by someone else for some other OS) to support their OS's own special purpose things? Of course not.

For any/all power management, (e.g. turning hard disks off when they're idle, disabling pre-fetching when you're running on battery, etc) you're screwed[/list]

Power management is handled by udi_devmgmt_req(), and in combination with ACPI, I don't see why this is impossible. Out-of-band special commands can be sent to configure the driver using a userspace utility if absolutely necessary. This would not be unusual either. Conventionally today, we see many such drivers: an AMD Catalyst Driver for example, comes with a userspace daemon that augments the kernel with support for its less universal features. This is not novel behaviour.

If a kernel wishes to specify custom APIs for hardware vendors to allow it to control certain aspects of devices, it can specify such an API and try to get hardware vendors to adopt it in their drivers. Then, those kernels which are interested in the custom API can establish child-bind channels on that custom Metalanguage and do whatever they need to do with those custom APIs (enable_prefetch_req()/disable_prefetch_req()). The only problem here is Independent Hardware Vendor adoption of your custom requirement, and the absence of UDI does not improve your prospects here.

None of this is supported by UDI in a standard way. Custom APIs (that work around the inadequacies of UDI in a non-standard way) do not solve the problem.

You are correct - it's no novel behaviour. It's ubiquitous functionality that UDI standards failed to cover. Of course it wasn't ubiquitous functionality 15+ years ago when the UDI specs were being written (so it's not fair to blame the original designers of UDI), it's just that UDI hasn't kept up with modern requirements.

In addition to all of that; you end up with another layer of bloat. Rather than having drivers that support the OS's native interfaces directly, you end up with a bunch of additional "shims" between the OS's native interface/s and UDI.

If you implement the UDI environment natively, you will gain native support, and thereby native performance. If you implement the UDI environment as a layer on top of another interface, you will gain second-class-citizen performance, since your UDI support is second-rate. This argument is like saying that "you end up with shims between the OS's native interface and WINE when you use WINE to support the NT API."

I don't think anyone uses UDI as their OS's native driver interface (and not just because it only provides "basic functionality", but because there's so many differences between different OS designs).

Finally; if you ever actually do find a driver that supports UDI, you still can't assume it will actually work - you have to do your own "quality assurance" testing to be sure (which means finding/purchasing the appropriate hardware/device); and maintain some sort of blacklist/whitelist (especially when you start considering the possibility of "trojan UDI malware"). You also might not be able to (e.g.) bundle the driver with your OS, because the copyright isn't yours and may be "undesirable" for whatever reason (e.g. proprietary, GPL), and/or because the driver may be owned by a known patent troll like SCO.

This is not different from any other software that is distributed; drivers are software, like applications. If you want a "trusted driver" framework, you can set about doing this with relative ease. The only barrier is time.

Unlike applications, drivers are typically considered part of the OS (or, software that's required to "operate a system"). Security is also a much larger issue for device drivers (especially for monolithic kernels, and drivers for devices that use DMA/bus mastering in micro-kernels).

Mostly what I'm saying here is that (after you've implemented support for UDI in your OS) you still can't just google for "UDI drivers" and use whatever you find.

The end result is that you need to do a lot of work just to support UDI, and in return for this large amount of work you get a few irrelevant drivers

The same goes for inventing a custom API. The only difference is that UDI is a very well-thought-through, scalable API, and you will not need to go through design re-iterations with UDI. With your own custom API, due to short-sightedness, the average kernel developer in this community will spend several years butting his head before he eventually, after many re-iterations, settles on something that is adequately well-designed and stable.

All the while, he would have needed to repeatedly update all of his drivers each time he had to redesign. UDI saves him all of this time. It looks like a lot of time spent up-front on implementing UDI, but this is regained in time saved not having to regressively redesign and update dozens of drivers repeatedly, possibly 15, 20, maybe 30+ times over 2-3 of years, and never having a stable driver API.

I'm definitely not against doing adequate research when designing your native driver interfaces; where "adequate research" may include investigating the capabilities of existing driver interfaces.

Both you and Combuster have tried to convince me (in multiple places) that it's possible to work-around UDI's inadequacies with OS specific non-standard extensions. Do you honestly expect me to believe that UDI alone will prevent the need to redesign and update dozens of drivers whenever those "OS specific non-standard extensions" change?


Cheers,

Brendan

[Combuster]

Both you and Combuster have tried to convince me (in multiple places) that it's possible to work-around UDI's inadequacies with OS specific non-standard extensions. Do you honestly expect me to believe that UDI alone will prevent the need to redesign and update dozens of drivers whenever those "OS specific non-standard extensions" change?

And how often did you rewrite your own UDI-free OS already? You could be much meaner than that argument and trick us into doing your upgrade work.

[Owen]

Hi,

If you want whole disk encryption, you're screwed

If you expect whole-disk encryption to be coded into every driver, you're mentally screwed.

Who said anything about "every driver"? It might be something that you only use for USB flash, and only because USB flash devices are easily stolen.

Why would it be a part of the USB Flash driver if the USB Flash device doesn't support encryption? Surely that's a job for a higher layer? (Maybe a disk encryption filter driver...)

If you want your a sane IO priority scheme, you're screwed

If you want to duplicate your scheduler into every driver, you're mentally screwed.

IO priorities have nothing to do with thread priorities and nothing to do with the OS's "thread/process scheduler". Essentially it effects the order that operations are performed by the device (e.g. new higher priority requests may skip ahead of older lower priority requests) and must be in all drivers that allow requests to be queued. This includes virtually all modern disk drivers as the hardware itself accepts queues (e.g. NCQ) to optimise throughput. It also doesn't preclude the option of forcibly cancelling an operation (e.g. a lengthy very low priority disk read) that has already been started by the device itself, when a higher priority request arrives.

Being as SATA doesn't support I/O prioritization, I/O prioritization should be done by the OS before handing off commands to the SATA HBA. UDI has full support for request cancellation.

If you have special synchronisation requirements (e.g. "notify me if sector XYZ is modified") you're screwed

Hey dude, you wrote that sector, why don't you go notify yourself for a change?

For a simple example, imagine the OS is shutting down; and sends an (asynchronous) write to the disk driver, then kills the power. Do you think it might be nice if the OS was able to know that the write has actually occurred (e.g. been written to the physical disk and not just written to a cache somewhere, and not just written to the hard drive's internal cache)?

That's a whole separate issue! Of course the driver gives you completion notifications. What driver interface doesn't?

If you want additional special purpose interfaces (e.g. possibly one designed for emulators, where a disk controller is assigned to a virtual machine such that the guest OS inside the virtual machine has "almost direct hardware access") then you're screwed

If you don't read specifications, you're screwed.

If you don't read the specifications, or if you do read the specifications, you're screwed.

There are two cases:

• The case in which you have an IOMMU, in which case you just hand the guest direct device access
• The case in which you have no IOMMU, in which your emulator scheme has so much trapping overhead you should have instead given the VM, say, a virtio block device.

If you want "secure erase" then you're screwed

If you don't know the SCSI commands to rewrite a specific sector, you're screwed even harder if you do it on your own.

If a file system needs to "trim", then you're screwed

What was the SCSI command for that again?

What is the standard method defined by UDI for doing both "secure erase" and "trim", that works for all devices (including devices that are not SCSI)?

Sure, we can quibble about the absence of support for non-SCSI devices all we want (of course, UDI is no worse in this case than your own driver interface, which doesn't yet define a spec for these things either)

For any/all power management, (e.g. turning hard disks off when they're idle, disabling pre-fetching when you're running on battery, etc) you're screwed

You can actually turn off devices, and you can define another interface for power management.

Most devices have several options (e.g. ranging from "full speed, full disk noise" to "low power, low disk noise"). A simple "on or off" isn't even close to adequate.

You can define your own interface, and as soon as you do that you've completely lost any advantage from using an existing standard - device drivers written for other OSs will not support your own personal interface.

And so maybe they'll use more power. Or maybe you'll collaborate with other people and define a common specification?

Why is it you seem to think that if it isn't covered by any existing spec, then everyone will just run off and do their own thing?

Does the fact that they've chosen to implement an existing specification not imply that, actually, they are interested in working together on this aspect of things?

That maybe their driver interfaces will end up better because they weren't defined in a vacuum?

[Brendan]

Hi,

If you expect whole-disk encryption to be coded into every driver, you're mentally screwed.

Who said anything about "every driver"? It might be something that you only use for USB flash, and only because USB flash devices are easily stolen.

Why would it be a part of the USB Flash driver if the USB Flash device doesn't support encryption? Surely that's a job for a higher layer? (Maybe a disk encryption filter driver...)

What higher level? The next highest level would be file systems, but they cant do whole disk encryption because they don't "see" the whole disk.

IO priorities have nothing to do with thread priorities and nothing to do with the OS's "thread/process scheduler". Essentially it effects the order that operations are performed by the device (e.g. new higher priority requests may skip ahead of older lower priority requests) and must be in all drivers that allow requests to be queued. This includes virtually all modern disk drivers as the hardware itself accepts queues (e.g. NCQ) to optimise throughput. It also doesn't preclude the option of forcibly cancelling an operation (e.g. a lengthy very low priority disk read) that has already been started by the device itself, when a higher priority request arrives.

Being as SATA doesn't support I/O prioritization, I/O prioritization should be done by the OS before handing off commands to the SATA HBA. UDI has full support for request cancellation.

If the underlying device doesn't support any I/O prioritization, the device driver has to do what the hardware lacks so that it can provide the abstraction the rest of the OS relies on.

UDI has request cancellation, but (for the example above) it's more like "request postponing" (e.g. cancel an "in progress" operation and switch to a higher priority operation ASAP, then restart the previously cancelled operation after the higher priority operation completes).

If you want additional special purpose interfaces (e.g. possibly one designed for emulators, where a disk controller is assigned to a virtual machine such that the guest OS inside the virtual machine has "almost direct hardware access") then you're screwed

There are two cases:

• The case in which you have an IOMMU, in which case you just hand the guest direct device access
• The case in which you have no IOMMU, in which your emulator scheme has so much trapping overhead you should have instead given the VM, say, a virtio block device.

If there is an IOMMU and you give the guest direct device access; you're unable to handle timing variations (e.g. if the guest is doing "single step debugging" or something), you can't quickly restore the state of the device before or after (without a full reset), the emulator won't know how to save the state of the device and restore it (if the guest is "frozen"), you can't do things like fault injection, etc. It also won't work at all for most devices (e.g. USB mouse but not the entire USB controller, an individual disk drive but not the entire disk controller, etc). Finally, it doesn't work for distributed systems (e.g. where the device isn't on the same computer as the emulator).

If there is no IOMMU, giving the VM a "virtio block device" adds an additional layer of abstraction and doesn't remove trapping overheads (e.g. "trap -> virtio emulated device -> OS layer/s -> actual device" compared to "trap -> actual device").

What is the standard method defined by UDI for doing both "secure erase" and "trim", that works for all devices (including devices that are not SCSI)?

Sure, we can quibble about the absence of support for non-SCSI devices all we want (of course, UDI is no worse in this case than your own driver interface, which doesn't yet define a spec for these things either)

This is a cheap tactic to avoid the issue. My OS project isn't ready to support device drivers yet (and won't be for several years); so there is no spec at the moment. When I design a spec I will research all the requirements, and you can bet that secure erase and trim will be included; and I'll still have standard support for "secure erase" and "trim" before UDI does.

Most devices have several options (e.g. ranging from "full speed, full disk noise" to "low power, low disk noise"). A simple "on or off" isn't even close to adequate.

You can define your own interface, and as soon as you do that you've completely lost any advantage from using an existing standard - device drivers written for other OSs will not support your own personal interface.

And so maybe they'll use more power. Or maybe you'll collaborate with other people and define a common specification?

Nobody has managed to define a successful common specification yet (even when large companies like SCO, Sun, Intel and IBM collaborate they just ended up with "stillborn" specs); so I doubt a lone hobbyist like me (who doesn't care about common device driver specifications to begin with) will succeed where everyone else failed.

Does the fact that they've chosen to implement an existing specification not imply that, actually, they are interested in working together on this aspect of things?

The original poster (gmoney) asked what people use. I told him what I use and why. I didn't suggest that other people shouldn't be lazy and pin their hopes on a magic bullet that doesn't really exist in a practical/usable form.

That maybe their driver interfaces will end up better because they weren't defined in a vacuum?

There's a first time for everything I guess.


Cheers,

Brendan

[Kevin]

Why would it be a part of the USB Flash driver if the USB Flash device doesn't support encryption? Surely that's a job for a higher layer? (Maybe a disk encryption filter driver...)

What higher level? The next highest level would be file systems, but they cant do whole disk encryption because they don't "see" the whole disk.

That may be true in your OS. Then you're doing it wrong. Other OSes have something like a block layer that provides common driver-independent block device level functionality. Encryption has no business in hardware drivers, except if the device does encryption in hardware.

IO priorities have nothing to do with thread priorities and nothing to do with the OS's "thread/process scheduler".

True, but "scheduler" doesn't imply that. They have a lot to do with a so called "I/O scheduler", which is usually part of that block layer that you're missing. Such an I/O scheduler might make use of information that it gets from the hardware drivers, but again, the hardware driver itself has no business in reordering requests.

For a simple example, imagine the OS is shutting down; and sends an (asynchronous) write to the disk driver, then kills the power. Do you think it might be nice if the OS was able to know that the write has actually occurred (e.g. been written to the physical disk and not just written to a cache somewhere, and not just written to the hard drive's internal cache)?

I have yet to see some disk that generates this kind of event when it writes something from its internal volatile cache to non-volatile storage. I also can't see the use case for it. It's hard to imagine a case where you let the hardware decide whether something is written to disk or not and just want to know whether it happened. Usually, when you're interested in persistent data, you want to force it to happen, right now.

In order to do that, you need to send the write request, wait for its completion, then send a flush request and wait for the completion of that. No silly events involved.

This is a cheap tactic to avoid the issue. My OS project isn't ready to support device drivers yet (and won't be for several years); so there is no spec at the moment. When I design a spec I will research all the requirements, and you can bet that secure erase and trim will be included; and I'll still have standard support for "secure erase" and "trim" before UDI does.

It's not completely unreasonable to use SCSI as a common protocol for all kinds of block devices and have the commands either passed through for real SCSI devices or interpreted by the driver for non-SCSI devices. Especially for a standard like UDI it's useful because SCSI covers pretty much everything that hardware could possibly provide. (CDI is much less strict about compatibility between versions and doesn't have a standard ABI, so it doesn't matter as much there.)

Nobody has managed to define a successful common specification yet (even when large companies like SCO, Sun, Intel and IBM collaborate they just ended up with "stillborn" specs); so I doubt a lone hobbyist like me (who doesn't care about common device driver specifications to begin with) will succeed where everyone else failed.

Define "success". For me using a common interface was a success as soon as I ran the first driver that wasn't primarily written for tyndur. It saved me some work that I could better invest in other parts of the OS.

[Brendan]

Hi,

Why would it be a part of the USB Flash driver if the USB Flash device doesn't support encryption? Surely that's a job for a higher layer? (Maybe a disk encryption filter driver...)

What higher level? The next highest level would be file systems, but they cant do whole disk encryption because they don't "see" the whole disk.

That may be true in your OS. Then you're doing it wrong. Other OSes have something like a block layer that provides common driver-independent block device level functionality.

For my OS; it's a micro-kernel. If one process (e.g. file system code) establishes a connection to another process (e.g. "mounts" a storage device driver) and sends a message saying "I want 1234 bytes at offset 12345678", then that message goes directly to the other process. It is possible to have a middle-man (e.g. something in between the file system code and the storage device driver that does encryption), but this increases the communication overheads (task switching between processes, etc).

Of course my example was not for my OS and was not for any one specific OS. If there are 1000 very different OSs and 5% of those OS decide to put encryption in the storage device driver, then 5% of OSs are screwed.

Encryption has no business in hardware drivers, except if the device does encryption in hardware.

If one specific device's hardware supports encryption, then the OS's standard interface used by all of its device drivers should include (e.g.) a "set_encryption_key()" function; and if one specific device's hardware doesn't support encryption then the OS's standard interface used by all of its device drivers should not include (e.g.) a "set_encryption_key()" function?

IO priorities have nothing to do with thread priorities and nothing to do with the OS's "thread/process scheduler".

True, but "scheduler" doesn't imply that. They have a lot to do with a so called "I/O scheduler", which is usually part of that block layer that you're missing. Such an I/O scheduler might make use of information that it gets from the hardware drivers, but again, the hardware driver itself has no business in reordering requests.

Some OSs may be so stupid and broken that they put I/O scheduling in the block layer (where it's unable to make decisions that take into account device specific things, like where the disk drive's heads happen to be). If there are 1000 very different OSs and 45% of those OS are crap because they've put IO scheduling in some generic block layer where it can't make the best decisions; then 55% of OSs are screwed.

For a simple example, imagine the OS is shutting down; and sends an (asynchronous) write to the disk driver, then kills the power. Do you think it might be nice if the OS was able to know that the write has actually occurred (e.g. been written to the physical disk and not just written to a cache somewhere, and not just written to the hard drive's internal cache)?

I have yet to see some disk that generates this kind of event when it writes something from its internal volatile cache to non-volatile storage. I also can't see the use case for it. It's hard to imagine a case where you let the hardware decide whether something is written to disk or not and just want to know whether it happened. Usually, when you're interested in persistent data, you want to force it to happen, right now.

In order to do that, you need to send the write request, wait for its completion, then send a flush request and wait for the completion of that. No silly events involved.

Assuming it's using asynchronous requests (which UDI does use) how do you wait for the request's completion? Are you saying that the device driver has to send a "request completed" notification back and you wait until you receive this notification?

This is a cheap tactic to avoid the issue. My OS project isn't ready to support device drivers yet (and won't be for several years); so there is no spec at the moment. When I design a spec I will research all the requirements, and you can bet that secure erase and trim will be included; and I'll still have standard support for "secure erase" and "trim" before UDI does.

It's not completely unreasonable to use SCSI as a common protocol for all kinds of block devices and have the commands either passed through for real SCSI devices or interpreted by the driver for non-SCSI devices. Especially for a standard like UDI it's useful because SCSI covers pretty much everything that hardware could possibly provide. (CDI is much less strict about compatibility between versions and doesn't have a standard ABI, so it doesn't matter as much there.)

Do you even know what a device driver interface is? It's supposed to be a clean abstraction that covers a super-set of features that all devices provide. Of course there are many different categories for devices (e.g. storage device, sound, printer, network, video, etc) and therefore (to keep the abstraction for a specific category clean) there's a set of different device driver interfaces (one for each category). SCSI is not a clean abstraction, doesn't cover a super-set of features that all devices (in a specific category) provide and tries to cover many specific categories. It's only usable as a device driver interface if you're lazy and incompetent.

Nobody has managed to define a successful common specification yet (even when large companies like SCO, Sun, Intel and IBM collaborate they just ended up with "stillborn" specs); so I doubt a lone hobbyist like me (who doesn't care about common device driver specifications to begin with) will succeed where everyone else failed.

Define "success". For me using a common interface was a success as soon as I ran the first driver that wasn't primarily written for tyndur. It saved me some work that I could better invest in other parts of the OS.

You did a huge amount of work to support UDI, and after all of that work you've managed to use one driver that provides nothing more than basic functionality? Awesome.


Cheers,

Brendan

[Combuster]

Fortunately, the arguments are getting weaker.

For my OS; it's a micro-kernel. If one process (e.g. file system code) establishes a connection to another process (e.g. "mounts" a storage device driver) and sends a message saying "I want 1234 bytes at offset 12345678", then that message goes directly to the other process. It is possible to have a middle-man (e.g. something in between the file system code and the storage device driver that does encryption), but this increases the communication overheads (task switching between processes, etc).

Even so, if you would include a copy of the same source code that does the encryption in every driver, you end up with bugs and updates equally repeated among all of them. And since you also want the option of doing encryption on a laptop hard drive, you will still put that code in one place and include it in every driver.

Your code to the UDI environment is in one place, and it's included in every driver.

Of course my example was not for my OS and was not for any one specific OS. If there are 1000 very different OSs and 5% of those OS decide to put encryption in the storage device driver, then 5% of OSs are screwed.

Of course they aren't. Any driver without encryption can be turned into a driver with encryption because it's not hardware related at all.

Encryption has no business in hardware drivers, except if the device does encryption in hardware.

If one specific device's hardware supports encryption, then the OS's standard interface used by all of its device drivers should include (e.g.) a "set_encryption_key()" function; and if one specific device's hardware doesn't support encryption then the OS's standard interface used by all of its device drivers should not include (e.g.) a "set_encryption_key()" function?

Did you deliberately put that contradiction in there as an attempt to turn a statement into something completely unrelated, or was that a coincidental interpretation error?

Some OSs may be so stupid and broken that they put I/O scheduling in the block layer (where it's unable to make decisions that take into account device specific things, like where the disk drive's heads happen to be)

If they're stupid enough to suffer from general dementia, then perhaps. But most schedulers will at least make an attempt to remember what the last written block was.

Assuming it's using asynchronous requests (which UDI does use) how do you wait for the request's completion? Are you saying that the device driver has to send a "request completed" notification back and you wait until you receive this notification?

It's actually that simple. The fact that you don't need to wait doesn't mean you are not allowed to wait.

Do you even know what a device driver interface is? It's supposed to be a clean abstraction that covers a super-set of features that all devices provide.

Therefore all video cards must provide an audio interface because it might have an HDMI port? Therefore all sound cards have to provide a storage interface because old soundblasters came with their CD bus controller when ATAPI wasn't a thing yet? Or a simple harddisk has to know proper responses to anything a blu-ray burner might be doing?

This is why in practice, interfaces actually are a subset rather than a superset, and they offer extensions when they are relevant for that device.

SCSI is not a clean abstraction

Are you suggesting that because a bus protocol is too generic, you're not allowed to write drivers for both the host controller, and by extension having no option but not to support it's child devices as well?

----

But I guess this is mostly a futile exercise into actually converting you as opposed to getting silly misunderstandings out of the way. As a wise man posted on IRC earlier today:

haters gonna hate