Should I get on the UDI train?

[JohnnyTheDon]

As far as bytecode goes, you could use LLVM, which would solve the SSE/3DNow/Optimization problem quite nicely. This would add virtually no complexity (you simply distribute drivers as LLVM files, users can then use LLVM to compile to native code tailored to run on their processor) and would still allow you to provide native drivers in architecture-specific binary format.

The operating system could either automate the LLVM to native translation procedure (probably through some high level userspace daemon that monitors drivers for changes and updates their native versions should they change, which would be very easy to write) or completely ignore the issue and just require that the drivers be compiled to native code before the OS even touches them (which could be done very easily by a package manager).

As long as you are using a standard C ABI between the drivers and the OS, it doesn't really matter how exactly they are compiled. I agree that there is no need for a new bytecode system to be built into UDI 2.0 that every OS will need to implement, but there is a definite need, as Love4Boobies stated, for a way to perform processor specific optimization of drivers. If the UDI 2.0 spec simply suggests (or requires) LLVM as a method for compiling and distributing the drivers, the problem is solved.

[Owen]

However: First priority should be defining interfaces to various device and bus types. To the list Gravaera suggested, I would add I²C/SMBus/PMBus and SPI.

This is easily extensible through metalanguages and bus bindings.

Exactly. Lets get to defining them.

I would perhaps also add a standard interface for general purpose IO pins - this is particularly useful in embedded devices.

I don't think I understand. Could you elaborate?

GPIO is just that - basically, an interface for controlling raw IO pins. I'll elaborate in a bit on the wiki

EDIT: Article on the UDI wiki: http://udi.io/wiki/General_Purpose_IO

Finally, we should probably define a way for a driver to directly expose some method of communication ("character device" style?) to userspace, for specialized devices (e.g. chip programmers) which really don't need a common interface (they're special anyway).

If I understand correctly what you mean, this is already possible altough the term userspace should not be used together with UDI.

Basically, I'm proposing a way to expose simple byte streams (and perhaps out of band data ala ioctls) to applciations, rather than other UDI drivers.

[Love4Boobies]

Hi.

My proposal was to ... provide binary compatibility via bytecoded drivers. Although it would make UDI a bit more complicated people should think of this in a similar was as they do about ACPICA: there's a reference implementation that everyone ports.

The difference between ACPICA and UDI is that ACPICA is meant to be provided on many different platforms and many different CPUs with the very same code. That is a bytecoded language. The ACPICA is not abstracted. That is, it does not call on an operating system which has already abstracted everything for it. It is essentially in itself highly machine dependent.

The reason for OSes porting ACPICA is not that it is less of an abstraction, it's because it's a huge amount of work that people would rather not waste any time on. I can't see how Project UDI should be any different in this respect. UDI, just like ACPI is supposed to work on many different platforms and many different types of CPUs.

UDI has a set of OS services it can call on, all of which return the same formatted information. This means that there is no need to bytecode since any platform on which a UDI driver is run will emulate the same behaviour with respect to outputs from service calls. Therefore drivers can be compiled to run natively and expect a certain minimum amount of standard behaviour. This is the whole purpose of the UDI environment which shields the driver from the embedding platform. The environment abstracts all of that form the driver so it can focus on its device.

That's not the point of the bytecode at all. (Already clarified on IRC, anyone who's still confused should re-read my post.)

Therefore, the environment and the UDI source compatibility together provide *more than enough* independence for anyone who needs it, and with a simple re-compile, you can have a driver for your platform, assuming the device will work with the driver.

What about proprietary drivers or drivers?

That's another reason why going so far is almost nonsensical: Devices themselves, unless implemented on a bus which is sufficiently abstracted (such as PCI), are generally tied to a particular chipset. And many chipsets may have the same device tied to different MMIO ranges, without a self enumerating bus to provide the MMIO range for the driver. This means that you will have to have different drivers for each chipset, even for the same device, or #ifdefs for each chipset the device is known to be one, anyway. There are too many cases like this for a bytecoded language to be in any way necessary. In fact, given that this is the common case on most platforms (custom, embedded, etc), the practicality of trying make drivers that abstracted is reduced to almost 0.

MCA alone was available for 3 architectures. PCI and USB are available for many more and there are not the only 2 buses. And even if what you said were true, I can't see why bytecodes wouldn't work... the driver needn't be aware of any ports or MMIO ranges, the environment needs.

Remember, we're not talking about software. This is about drivers. These, no matter how hard you try, will never be fully portable for all time since there are different companies and vendors with different budgets, and different levels of skill diversity in their workers (and thus different levels of compliance and ability even with the implementation of a self-enumerating bus).

What does the budget of the company have to do with the driver interface?

To be perfectly honest, I didn't understand a lot of what you said there, since you seem to be trying to make this too platform specific, i.e.: much ado about nothing. Things like taking advantage of SSE, 3dNow, etc, are really supposed to be up to the embedding OS. All you need is a platform features enumeration API and that that's it. IIRC, the UDI 1.1 even mentions that it does not allow floating point operations.

"UDI shouldn't do fancy stuff because there are architectures out there that can't do it." If that were the case, then UDI itself should be forever banned. Also, I don't know about UDI 1.1 as it does not yet exist but the 1.01 specification says that a FPU can be used when present.

One thing to note is: Even if you use a bytecoded language to distribute the drivers, and then compile them down to native instructions, you will end up having to ask the embedding OS whether or not advanced FPU instruction sets are available before executing code with SSE, etc. Also, even if you run the bytecode as JIT or whatever, you'll STILL have to ask the embedding OS, and in the end, you now have a huge JIT compiler in the environment just to end up having to do the same thing as native, compiled drivers.

What's the problem with asking the OS about the architecture? Of course you ask it. And I have no clue where you got the JIT idea from, I talked about an AOT compiler that is part of the install process.

The rest of the post is a rant so I won't discuss it any further

As far as bytecode goes, you could use LLVM, which would solve the SSE/3DNow/Optimization problem quite nicely. This would add virtually no complexity (you simply distribute drivers as LLVM files, users can then use LLVM to compile to native code tailored to run on their processor) and would still allow you to provide native drivers in architecture-specific binary format.

I've thought about this. I need to check the license, we don't want it to be restrictive.

As long as you are using a standard C ABI between the drivers and the OS, it doesn't really matter how exactly they are compiled. I agree that there is no need for a new bytecode system to be built into UDI 2.0 that every OS will need to implement, but there is a definite need, as Love4Boobies stated, for a way to perform processor specific optimization of drivers. If the UDI 2.0 spec simply suggests (or requires) LLVM as a method for compiling and distributing the drivers, the problem is solved.

Again, we'd be losing proprietary drivers.

[JohnnyTheDon]

I've thought about this. I need to check the license, we don't want it to be restrictive.

Its a BSD-style license, so there shouldn't be any problem.

Again, we'd be losing proprietary drivers.

I don't see why. LLVM is about as source-safe as native assembly (the only possible exception being explicit function blocks), and if proprietary driver writers don't like it they can still distribute native binaries that don't take advantage of non-standard processor features/optimizations.

[thepowersgang]

Exactly.
The way I see it is that UDI 2.0 should have it's API be a superset of UDI 1.1, but suggest that drivers be distributed as LLVM bytecode, that is then AOT compiled into native upon installation (using udisetup)
In this way, it could almost actually be called UDI 1.2 (since native drivers are still supported)

[Solar]

In short, UDI does NOT need a bytecoded distribution form. And also, I am strongly of the opinion that UDI MUST remain in compiled-language source form. I believe that we ALL know that were it not for the open-source, open-standards movement, this community would not exist. Had it not been for those fighting to make sure that things like specifications, etc are released, we would have NO information. Therefore, allowing that much abstraction and making room for pure binary distribution without a need for source distribution for unknown platforms is going to undermine the open-source side of UDI, however small it is.

Without specifications and source, we end up at the mercy of crappy code written by outsourced workers, and crappy buggy code from vendors with no hope of re-compiling and uploading patches to the central repo. There are those who do not see this as a big deal: "Ask the vendors to fix the bugs". Think a bit more, and realize why letting vendors get away with pure binary drivers and nothing more is a bad thing. Think about how hard it would be to have your little crummy "Hello world", even if you do not have some advanced kernel at the moment.

And those who say something like: "What are you talking about? It won't be that bad.". Yes, it will. Have you seen what kinds of CRAP companies like HP produce? A driver for a printer. A printer. And it runs about at least 5 different processes while idle, plus N different processes while operational. No. I will not support anything which will make that kind of mess easier to produce in UDI.

I do not support bytecoded drivers, and again, I assert that if someone wants to move the spec in such a radical direction, they should just go and make their own interface altogether. I support UDI because it clearly provides all needed functionality in it current state, and allows for extensions. That is all that is necessary.

I cannot really express how strongly I disagree.

Mixing up technology and philosophy was always a thing that did chafe me to no end with the GPL crowd. The decision to make some software under an OSS license or not is up to the author, period. Doing anything that would force driver developers to undisclose their sources would make me oppose this "new UDI" instead of advocating it.

Many, many people will make their software OpenSource. That is great, I applaud it, I did it myself on several occasions, and I think it's a good thing. As long as no-one forces me (or others) to do it.

There are many very valid points in favor of not laying open your source, but I don't want to discuss them here. The point is that I don't like to be forced in my decision by some nutcases who think they've got a right to.

</rant>

----

That being said, did you notice that UDI 1.1 does allow for binary driver distribution? Unless I am severely mistaken, UDI drivers are binary compatible for a given CPU architecture. (Indeed, that was why Stallman ranted about UDI.)

As such, I actually agree with you: We don't need a bytecoded UDI 2.0. But I agree that's an opinion shouted down from a helicopter, as I never dug deep enough into the UDI specs to really tell.

[Creature]

I'm not sure where the connection between "building the driver yourself" and "not working wihout problems" is. If building the driver is a problem, there's something wrong already. A binary driver wouldn't be a solution but merely a workaround for that.

I see your point as this may not be a big problem, but this workaround could prove useful. For example, a situation where (many) different compilers are used (besides the fact that most people here use a GCC cross-compiler) that have differences, e.g. Visual C++ Inline Assembly and GCC Inline Assembly, recompiling may require some work "translating" the source. This doesn't necessarily have to take long, I'm just saying it may be useful. Not having to recompile the source code could also allow the source of the driver itself to be closed (and still allowing the binary to be used in a cross-platform way).

As for the audio/video device classes and decode acceleration: My suggestion is that we should engineer our drivers to support the relavent Khronos APIs. That is,

• Accelerated graphics: OpenGL, OpenGL ES 1/2, OpenVG
• Audio: OpenSL ES, OpenMAX
• Video decode: OpenMAX

Agreed, since IMHO these are very good API's that have proven to be good and they are fairly open to engineering them ourselves.

There are many very valid points in favor of not laying open your source, but I don't want to discuss them here. The point is that I don't like to be forced in my decision by some nutcases who think they've got a right to.

I wholeheartedly agree. Open-source can be nice, but there is a difference between wanting to make your sourcecode open-source and being forced to make your sourcecode open-source. If you want to make it open-source and don't like making it closed-source, go ahead and nowone will stop you, but don't prevent others from doing what they want to do (unless it contradicts the license(s), of course).

[gravaera]

Hi:

By all means, different strokes for different folks. However, I am not an advocate for the FSF, since more than one person seems to have gotten that impression. The only place where I believe strongly that open source is *needed*, and is a *must* is in kernel development. My reasons are simple: without the open mentality none of use would be here.

Also, the community is not strong enough, nor is it equipped with enough information to be able to stand strong alone without the influence of free software banner-wavers. If you disagree, that's nice. I can understand a non open source software project of any kind, and I have no bias toward open source in any area other than kernel development. Should anyone decide to probe further, I'll provide a disambiguation of reasoning.

However, the idea that one should decide that my advocacy of open-source in UDI since it is directly attached to kernel development is something to take personally, is slightly irritating, and that one should venture to indirectly try to give me a lecture does not sit well with me.

--Kindest regards,
gravaera.

[Solar]

It's just that I consider drivers to be seperate entities only losely related to the kernel proper (I know that this is orthogonally different from the Linux approach - and not by accident).

I'd even second your notion that kernels should be OpenSource (because there is a level of security by un-obscurity to be achieved there). But I also believe, as I said, that there are excellent reasons to keep your hardware (and corresponding drivers) a black box.

If I have offended, I apologize, as that was not my intention. But I believe it to be one of UDI's strongest points that it does not require undisclosing driver sources in order to provide drivers for all supporting OS's.

[Love4Boobies]

I will try to explain the problem so that people who have not read the specification fully can understand why UDI 2.0 is being dicussed.

At the present time, UDI 1.01's binary package distribution incorporates multiple drivers in an attempt to provide optimized binaries for specific CPU models. The ABI binding for a specific architecture dictates that there should be a generic driver (to be used when an optimized one is not available) and explains how the rest should be packaged. There are two problems with this:

• The ABI binding needs to be revised every couple of months or so due to vendors releasing new CPU models (this is not obvious in the above explanation).
• A driver package shouldn't contain hundreds of drivers; people will simply stop caring about optimized drivers.

I see too aims and 3 possible solutions at the moment and I'd like you guys to tell me what you think and do come with any other proposals.

The aims:

• Provide a unique driver that can be optimized on installation.
• Make it possible for proprietary drivers to be distributed.

The solutions:

• Source code distributions as gravaera proposed on IRC. There are a few problems with this. First, it fails to achieve our second aim. Secondly, we're forcing the user to get a compiler if his environment does not have one (also, note that this compiler has to be able to do the needed optimizations). Less importantly, a parser will slow down installation times.
• Only use binary distributions for generic drivers. Perhaps it's less obvious because the only example I have given was SSE which is particularly useful for multimedia drivers and gives a much smaller advantage to other drivers. Even though we don't care about the 8086 much these days consider that it wasn't able do shift left by more than 1 with immediate values (i.e., "shl ax, 1" was possible while "shl ax, 2" was not). I'm certain this problem exists for many architecture. For instance, if we consider AMD64 as an extension to IA-32, I now have an old CMI8330 sound card who's 32-bit driver does not take advantage of 64-bit registers and certain instructions. Further more, some encodings have even changed (e.g., for INC). Some of you may say "but look at Windows, they don't provide optimized drivers either." Point taken. On the other hand, UDI was concieved with this advantage in mind and I can't see why we shouldn't use it (because we don't like a little change? I see no technical reasons). Windows makes this impossible at the moment.
• Provided bytecoded drivers. This achieves both aims presented above and also raises an interesting point (read below).

Even though I'm not saying we should actually do this (a lot of benchmarks would be needed), there's always the possibility of using a managed bytecode. This would give the advantage that UDI drivers would always be safe on MMU-less architectures. Even the MMU people have something to win. For example, on an x86 we can safely put drivers in kernel mode without worrying that they will write all over memory (as long as the AOT compiler does not contain any bugs of its own) - that will increase the speed of the drivers. There is also an inherent problem with this technique, namely that runtime checks will be needed for certain things like array bounds checking, thus decreasing the speed of the drivers. I don't know whether in the end we will end up with faster or slower drivers (and please don't give me your opinions unless you have some solid benchmarks to present along with them). Another disadvantage is that drivers cannot be written in C (or at least not using the full language).

[Creature]

Another disadvantage is that drivers cannot be written in C (or at least not using the full language).

This may be a rather big disadvantage since there are still plenty of people here that prefer C to write drivers in (which can be slightly faster than e.g. C++ for several reasons).

I don't have a lot of experience with managed bytecode myself, but at the risk of being called a newbie: suppose we accept the managed bytecode idea, we (or whoever wishes to) could always make an "interpreter" open-source (maybe not necessarily for integration in hobby OS', but maybe also for reference implementations). This way, it should be easier for people to have access to this kind of "interpreter", thus being able to run the bytecode drivers.

• Provide a unique driver that can be optimized on installation.
• Make it possible for proprietary drivers to be distributed.

IMHO we should strive for achieving both aims/goals. This however rules out building from source and after having read the posts, binary compatibility may seem like a good idea, but having to optimize for certain architectures or CPU types would just be too much work (unless a generic, less optimized driver would be provided).

[Love4Boobies]

An interpreter would be very, very slow. The idea is that the reference implementation would contain everything that's needed, the AOT compiler and everything. People could just port that in their OS (whether proprietary or open source).

[JohnnyTheDon]

Mixing up technology and philosophy was always a thing that did chafe me to no end with the GPL crowd. The decision to make some software under an OSS license or not is up to the author, period.

I second that. In addition, there are pragmatic reasons for not forcing drivers to be open source. Many manufacturers would not be okay with releasing open source drivers, and being closed to proprietary driver development would make a new UDI almost pointless. We would simply end up having the same problem Linux has with drivers, except to a much greater degree since we will (at least in the beginning) have fewer people working on UDI drivers than Linux drivers.

Even though I'm not saying we should actually do this (a lot of benchmarks would be needed), there's always the possibility of using a managed bytecode. This would give the advantage that UDI drivers would always be safe on MMU-less architectures. Even the MMU people have something to win. For example, on an x86 we can safely put drivers in kernel mode without worrying that they will write all over memory (as long as the AOT compiler does not contain any bugs of its own) - that will increase the speed of the drivers. There is also an inherent problem with this technique, namely that runtime checks will be needed for certain things like array bounds checking, thus decreasing the speed of the drivers. I don't know whether in the end we will end up with faster or slower drivers (and please don't give me your opinions unless you have some solid benchmarks to present along with them). Another disadvantage is that drivers cannot be written in C (or at least not using the full language).

Instead of using a language that many will not know or making modifications to C, which would make UDI 2.0 less accessible (especially to proprietary driver writers who will not want to waste time training people in a new language), why no use CIL as our managed bytecode? This way any .NET language can be used (such as C#) and we do not have to write an entirely new bytecode. As for performance, a reference-counting smart pointer system seems the way to go in the standard UDI implementation, due to the extremely low overhead. When used with AOT compilation, we should get the security desired without taking any real performance hit.

However, IMHO, all of this is a bit unnecessary and will lose us a lot of adopters. Binary drivers with a C ABI, and a suggestion to use LLVM for portability, seems like the best bet for a universal driver framework. If we require implementers and users to do too much, they simply won't use UDI.

[kabure]

Many manufacturers would not be okay with releasing open source drivers...

But why not? What benefits do manufacturers get from hiding source code? This is what I never really understood.

...and being closed to proprietary driver development would make a new UDI almost pointless. We would simply end up having the same problem Linux has with drivers, except to a much greater degree since we will (at least in the beginning) have fewer people working on UDI drivers than Linux drivers.

Thirded. I remember someone arguing that, actually, allowing manufacturers to release closed-source drivers under UDI makes them all that easier to reverse engineer, considering how tightly integrated to the interface UDI drivers are.

[JohnnyTheDon]

Many manufacturers would not be okay with releasing open source drivers...

But why not? What benefits do manufacturers get from hiding source code? This is what I never really understood.

It amounts to them releasing specifications of the internals of their hardware. This is bad for those who either have their own (possibly patented) technologies built into their hardware, or those who are ripping off patents secretly (in particular, this is why GPU manufacturers are iffy about releasing specs). Corporations also tend to have a natural reticence about releasing code that they paid to be created to the public, since they are essentially giving the man-hours away for free.