OSDev.org

The point of moving drivers out of the main OS is so they would be accessible from the firmware and bootloader. The current state of things is to have two or even three drivers for things like disk drives, with the lower-level ones often stuck using legacy or proprietary interfaces. That's exactly the type of wasted effort UDI was designed to avoid, is it not?

Rusky wrote:The point of moving drivers out of the main OS is so they would be accessible from the firmware and bootloader. The current state of things is to have two or even three drivers for things like disk drives, with the lower-level ones often stuck using legacy or proprietary interfaces. That's exactly the type of wasted effort UDI was designed to avoid, is it not?

If they are provided by an auxiliary OS then how would they be accessible from the firmware? As for the boot loader, that would have to be the auxiliary OS.

Why not just build an UDI environment in your ordinary boot loader? The boot loader and OS could easily share drivers; what's the point of this separation? Making them share UDI environments makes no sense because the boot loader and OS don't have the same internals.

It would be nice if the OS could rely on the firmware for its drivers (which is what I assumed was meant by "auxiliary OS"). Downloading e.g. a copy of the Linux kernel would no longer include a giant pile of drivers for every device under the sun, just a UDI environment and some code to grab the drivers already present with the devices on the machine.

However, yes: sharing UDI environments would be silly, and that might make this kind of situation technically infeasible. I don't know enough about UDI to be sure, but binary blobs would probably preclude certain OS designs, even within the same processor architecture, and infuriate certain members here.

Love4Boobies wrote:I'm still waiting for you to explain why this auxiliary OS is useful. You just took the driver interface and power management support out of the main OS and put them in what you call an auxiliary OS.

I've already explained that it's the equivalent of a real-mode OS working in tandem with the BIOS. As soon as you switch into protected mode, you lose direct access to the BIOS, but if you had equivalent services available which run in protected mode you would be able to have a 32/64-bit OS working in tandem with a 32/64-bit BIOS. Collectively the two things would behave as a single OS, and as the main OS gets more advanced it can decouple more and more components from the auxiliary OS and incorporate them more directly into itself in order to use them more efficiently. This is about making hobby OS development easier so that an OS designer can quickly achieve a high level of functionality and be able to go on improving his/her OS by modifying the way it ties in with the auxiliary OS. An advanced version of such an OS will incorporate every aspect of the auxiliary OS directly into itself to the point that there is no longer any separation between the two, and by that point it could be performing on the same level of Linux or Windows if sufficient drivers become available.

Why don't you also create a second such auxiliary OS to provide a validating XML parser and TCP/IP support?

You could have a whole bunch of auxiliary OSes or universal modules capable of being built into a wide range of main OSes. It doesn't really matter whether they're one block or many.

A lot of work needed for no apparent gain, esp. in order to cover all corner cases (e.g., Should the binary blob be compiled for a flat memory model? Should there be a whole bunch of binary blobs for all the modules or should there be a startup compiler with the ability to cache the blobs on a per-machine basis?).

Would anyone seriously want to use anything other than a flat memory model? I was imagining a whole lot of relocatable device drivers in binary form on a storage device and the appropriate ones being loaded in - you wouldn't compile anything after the binary files have been created, and that would all be done in advance. Maybe that isn't practical because of the number of different versions of each driver that you'd have to have on the storage device, so perhaps it would be better to compile them to create versions specific to the machine they're going to be run on, but you'd only have to do that once for each machine as you could then store the results on the internal drive and load them from there every time in future.

It would be less flexible than having the OS developers port ACPICA and udiref or whatever. One can modify the source code so that it can better fit the OS' design but modifying binary bolbs is an entirely different story...[/list]

I'm not fussed about whether ACPICA's built into it or not - it was just a passing thought, based on the idea that you could at an early stage of OS development take control of the power button to prevent the machine being switched off if it's pressed by mistake, which could be useful if you're writing an app for young children where you want to lock them into it and give them no way to reboot the machine. By making ACPICA part of the auxiliary OS you could make such functionality available to a hobby OS creator from the start.

I would now like for you to put together a list of advantages. All you've done so far was to describe how the thing would work; it's time to describe why. No better way to convince people that pro/con lists.

I've given you plenty of reasons above. The main one though is simply to recreate the real mode BIOS services in 32-bit mode form and to add further functionality too, thereby making it easier for people like most of the members of this site to create their own OS and build something powerful in a short time. Most of the power would come from the auxiliary OS, but that isn't any different from old real-mode OSes getting most of their functionality from the BIOS.

Finally, to demonstrate that you're not the kind of guy who thinks a design though, I'll give you a better goal for your project: An open source bare bones OS that implements ACPICA, udiref, and whatever else you think is useful. No more need to relocate anything and thus create startup overhead, no more design compatibility problems, no more flexibility problems, and a much cleaner design. People can just work on top of the building blocks provided by this project.

And then you've just created a highly restrictive community OS project which does everything one way rather than allowing everyone to create their own OS project doing many things their own way while drawing upon the same underlying essential functionality in those areas where everyone's going to do things much the same way anyway. The BIOS doesn't limit the design of a real-mode OS, and the auxiliary OS I'm proposing wouldn't limit the design of the main OSes that run alongside it either. Different projects would then tie components of the auxiliary OS into their own OS in their own way and ultimately end up with an OS which doesn't use the auxiliary OS at all, other than taking direct control over all the UDI drivers within it.

I'm quite sure this is all pie in the sky in that it won't happen, so there isn't a lot of point in discussing it further, but I thought it might be interesting to throw the idea into the ring just to hear what people thought about it, and Oranos in particular (since he's sitting on some secret plan which he isn't keen to reveal to us - I thought he might just drop another clue our way, but he's switched to sending cryptic private messages instead). Anyway, it's probably too late to be worth bothering to start on an auxiliary OS project now as it's likely to be overtaken by events long before it can get anywhere useful, so it should at best be considered an idea that missed its time, and at worst a lame idea from someone who knows next to nothing.

DavidCooper wrote:The BIOS doesn't limit the design of a real-mode OS, and the auxiliary OS I'm proposing wouldn't limit the design of the main OSes that run alongside it either.

Our OS's have vastly different methods of interrupt handling, paging setups, multicore locking, memory layouts, etc. I'm not sure that it's possible. Just think about Windows and Linux, and what sort of aux OS would be required to support the driver models on just those 2 operating systems.

The idea worked for a very basic real mode, single-tasking OS like DOS but as soon as you have time slicing, re-entrancy, multi-core, ..., it's way too hard.

Rusky wrote:However, yes: sharing UDI environments would be silly, and that might make this kind of situation technically infeasible. I don't know enough about UDI to be sure, but binary blobs would probably preclude certain OS designs, even within the same processor architecture, and infuriate certain members here.

Operating systems are complex and the UDI environment would be tied to this complexity so it would be difficult to use the same thing in the boot loader without implementing much of the same functionality. After all, if the boot loader has the same functionality as the OS, what's the purpose of the OS? Fortunately, an UDI environment for something like a boot loader would be really simple to implement.

DavidCooper wrote:

Love4Boobies wrote:I'm still waiting for you to explain why this auxiliary OS is useful. You just took the driver interface and power management support out of the main OS and put them in what you call an auxiliary OS.

I've already explained that it's the equivalent of a real-mode OS working in tandem with the BIOS. As soon as you switch into protected mode, you lose direct access to the BIOS, but if you had equivalent services available which run in protected mode you would be able to have a 32/64-bit OS working in tandem with a 32/64-bit BIOS. Collectively the two things would behave as a single OS, and as the main OS gets more advanced it can decouple more and more components from the auxiliary OS and incorporate them more directly into itself in order to use them more efficiently.

I figured this was what you were looking for but waited for you to make the claim. You see, the BIOS functionality was never really lost; it was merely moved from the firmware into the OS because it's more flexible that way:

• Firmware vendors don't need to implement uber-efficient drivers that take advantage of many-core technologies and whatnot.
• Drivers are typically provided by hardware vendors, not OS developers.
• It's not feasible to get a large EEPROM chip and flash it everytime you install a new device. Don't even get me started on hot plugging.
• Driver interfaces are typically molded after operating systems for efficiency reasons (UDI being an exception).

You're not trying to offer lost functionality. Instead, you're trying to move it out of the OS again into another module you invented. I do understand that you're talking about a pre-installed library for hobby OSes. However, as I already explained before, hobby OS developers can just port whatever components they like (be that udiref, ACPICA, or anything else) with greater flexibility by using the source code---all you want is to pre-package the stuff YOU think is useful in a binary blob, thus creating technical difficulties, as well as increased overhead.

DavidCooper wrote:This is about making hobby OS development easier so that an OS designer can quickly achieve a high level of functionality and be able to go on improving his/her OS by modifying the way it ties in with the auxiliary OS. An advanced version of such an OS will incorporate every aspect of the auxiliary OS directly into itself to the point that there is no longer any separation between the two, and by that point it could be performing on the same level of Linux or Windows if sufficient drivers become available.

You're saying the same thing over and over again. I specifically asked you to stop telling me how things would work and explain to me why they should work like that.

• UDI drivers would offer all the functionality regardless of whether they are implemented in the main or auxiliary OS.
• In the case of things like ACPICA, just because you're pre-packaging it doesn't mean it's any easier to use than the source code. It still takes work to integrate everything and/or properly use all the information it provides. Hence, there are 0 advantages; the disadvantages are covered in many parts of this post (flexbility, incompatibilities, etc.).

DavidCooper wrote:

Why don't you also create a second such auxiliary OS to provide a validating XML parser and TCP/IP support?

You could have a whole bunch of auxiliary OSes or universal modules capable of being built into a wide range of main OSes. It doesn't really matter whether they're one block or many.

Right, my my actual point behind that remark was that there should be 0 extra modules rather than more than 1. That would be done by implementing the auxiliary OS inside the main OS. And that's consistent with what happens in every OS today

DavidCooper wrote:

A lot of work needed for no apparent gain, esp. in order to cover all corner cases (e.g., Should the binary blob be compiled for a flat memory model? Should there be a whole bunch of binary blobs for all the modules or should there be a startup compiler with the ability to cache the blobs on a per-machine basis?).

Would anyone seriously want to use anything other than a flat memory model? I was imagining a whole lot of relocatable device drivers in binary form on a storage device and the appropriate ones being loaded in - you wouldn't compile anything after the binary files have been created, and that would all be done in advance. Maybe that isn't practical because of the number of different versions of each driver that you'd have to have on the storage device, so perhaps it would be better to compile them to create versions specific to the machine they're going to be run on, but you'd only have to do that once for each machine as you could then store the results on the internal drive and load them from there every time in future.

It's irrelevant whether the OS uses segmentation or not. The point is that the auxiliary OS might have restrictions. Anyway, I wasn't expecting answers for these questions; I was just showing some of the many engineering problems to consider and work around---it's likely not feasible to create something that is compatible with all OS designs. I've got plenty more examples.

I would now like for you to put together a list of advantages. All you've done so far was to describe how the thing would work; it's time to describe why. No better way to convince people that pro/con lists.

I've given you plenty of reasons above. The main one though is simply to recreate the real mode BIOS services in 32-bit mode form and to add further functionality too, thereby making it easier for people like most of the members of this site to create their own OS and build something powerful in a short time. Most of the power would come from the auxiliary OS, but that isn't any different from old real-mode OSes getting most of their functionality from the BIOS.

DavidCooper wrote:

Finally, to demonstrate that you're not the kind of guy who thinks a design though, I'll give you a better goal for your project: An open source bare bones OS that implements ACPICA, udiref, and whatever else you think is useful. No more need to relocate anything and thus create startup overhead, no more design compatibility problems, no more flexibility problems, and a much cleaner design. People can just work on top of the building blocks provided by this project.

And then you've just created a highly restrictive community OS project which does everything one way rather than allowing everyone to create their own OS project doing many things their own way while drawing upon the same underlying essential functionality in those areas where everyone's going to do things much the same way anyway.

I'm not trying to create a restrictive community; what I am trying to do is to make the community understand that they need to think about their designs. Often times people seem to mistake "different" for "better" and this seems like a perfect example to me.

You still haven't answered why the bare bones OS wasn't a better design.

DavidCooper wrote:The BIOS doesn't limit the design of a real-mode OS, and the auxiliary OS I'm proposing wouldn't limit the design of the main OSes that run alongside it either.

Really? How would it work with segmentation? How would it work with managed OSes? etc.

DavidCooper wrote:Anyway, it's probably too late to be worth bothering to start on an auxiliary OS project now as it's likely to be overtaken by events long before it can get anywhere useful, so it should at best be considered an idea that missed its time, and at worst a lame idea from someone who knows next to nothing.

Don't take it personally, some ideas are just bad. It's not an offense towards you or anyone else. It's important to have lots of ideas but to also analyze them carefully.

What I would find useful is a common OS-interface for large software-projects like filesystems, ACPCIA, MP3-players, JPEG-viewers that is really well-thought out. The best interface I've seen this far is ACPICAs, even if it has a few issues that could have done better. That would reduce porting-time.

The best way to achieve all that would probably be to create a UNIX and start porting stuff. The downside to creating a unique design is that's it's likely porting will become much more difficult.

A couple of years ago I relied heavily on my Win32 emulation. It was so sofisticated that it could run many of the Borland-tools in their Win32-console versions (compilers, debugger). However, since then I've left Borland completely for OpenWatcom and now I use a completely native API in applications I write. At a time I had advanced plans to provide a full UNIX-environment, and could load ELF formated executables and such, but I never finished that since it took too long to integrate RDOS with GNU/GCC/Newlib.

That is my philosophy: A native API that is designed from scratch with no regards to existing APIs. Then I write emulation wrappers for other environments. If the primary API is good enough, most wrappers are a snap to write (the ACPICA adaption was quite easy to write, as was the JPEG and MP3 adaptions).

Sure, I wasn't trying to suggest that everyone should implement POSIX or whatever. I mentioned it because it's a potential solution to the problem we were discussing---I personally don't like it very much.

Love4Boobies wrote:You're not trying to offer lost functionality. Instead, you're trying to move it out of the OS again into another module you invented. I do understand that you're talking about a pre-installed library for hobby OSes. However, as I already explained before, hobby OS developers can just port whatever components they like (be that udiref, ACPICA, or anything else) with greater flexibility by using the source code---all you want is to pre-package the stuff YOU think is useful in a binary blob, thus creating technical difficulties, as well as increased overhead.

The stuff I think would be useful is the stuff that any respectable OS would need to have in it, so none of it would be superfluous. The big advantage of the auxiliary OS idea is that you wouldn't have to include hundreds of device drivers in the OS you're developing because you'd find all the right ones waiting for you in memory on booting, all put in place by the auxiliary OS and built into a system that already works so that you can get instant functionality out of it without having to do any work. It would be easy for the auxiliary OS to come with thousands of device drivers in it and to load the appropriate ones into memory after checking out the hardware from the inside, but to expect hobby OSes to do that is asking far too much. I think the best way to illustrate the point would be to show you a working auxiliary OS and a simple main OS which works alongside it, so I think I'd better just build one, though it'll have to wait until I've built the intelligent tools needed to do almost all of the work for me. I'll get back to you in a year or two when it's ready. I certainly agree that it wouldn't be the most efficient way to start up an advanced operating system in which every part of the auxiliary OS is immediately going to be rewired directly into the main OS, but it would still only add a fraction of a second to the boot time - the real advantage of the auxiliary OS would be during the development phase of the OS, particularly in a self evolving system like mine where all the work is done in memory and there is never any compilation process, but it would also simplify things considerably for anyone starting out from scratch where they're working with just a few sectors of code written in assembly language.

but to expect hobby OSes to do that is asking far too much.

No David. We're not expecting, or asking them to do that. They have their own ambitions and reasons for an OS.

I think you are being quite biased to your own issues and opinions. I once was an annoying guy who insisted people did me favors for my own provincial concerns. If we spend less time with pleasing one person's low-priority and small-range concerns, it can be far easier to eventually come around and please everyone, including that whiner; and we'll come out with a much more beautiful design.

We are trying to step out of the design fence. To do this, we can't slurp-up every convenient idea there is too quickly. Linus Torvalds made a mistake similar to your own when he decided to create 'Freax' (Linux) based on Unix, with a high degree of similarity. Of course, I don't mean this too technically. My main point is: don't always go with what's originally easiest. If you go the more careful (and harder) way, things will become incredibly easier than possible with the obvious option.

Now to be technical.

The stuff I think would be useful is the stuff that any respectable OS would need to have in it, so none of it would be superfluous.

What exactly is this 'stuff' you think about? Would this 'stuff' taste good? What is a 'respectable OS?' Does that mean a bloated OS? Does that mean a reiteration of already existing features? How boring. How stupid. And.. how abstractly mis-understandable!

David. Because you're so zealous, I win.
And David... I just invoked the Ad hominem spell. I'm now a level 12 Mage.

Edit: I like your name, David.

I might look narrow minded, but this happened to be a quick and "I'm out of time right now" reply. I really want you to respond. I'm sorry, this might be a bit too crude. So next time, I'll try to be more considerate and comprehensive.

Oranos wrote:Linus Torvalds made a mistake similar to your own when he decided to create 'Freax' (Linux) based on Unix, with a high degree of similarity.

Note that Linus Torvalds did not intend to create the next great operating system.

What he was trying to create was a i386 Unix kernel that was not limited my the educational-only Minix license. That was his original goal - no more, no less. As such, it's a bit much to say that Unix compatibility was a "design mistake" if it was, indeed, the design goal.

Hi Oranos,

I'm glad you've posted a little more - you're an interesting character and I was hoping you'd reveal a little more about yourself, just in case you're sitting on some astonishing idea which you might reveal a tantalizing glimpse of, but I suspect you're too clever to make such a mistake.

What exactly is this 'stuff' you think about? Would this 'stuff' taste good? What is a 'respectable OS?' Does that mean a bloated OS? Does that mean a reiteration of already existing features? How boring. How stupid. And.. how abstractly mis-understandable!

Highly unbloated - you install the auxiliary OS once on a machine and it only loads in the device drivers directly necessary for the actual hardware found there. The main OS can be feather light. If you change part of the hardware or plug something new in, the auxiliary OS can find a driver for it without the main OS designer needing to have a clue about such things. If you don't want to use part of the auxiliary OS, you don't have to start that bit up and you can write over it. You could also set it not to load certain things in in the first place if you know you don't want them and merely want to write an app that runs a single app for a specialist purpose. If a new version of the aux OS comes out, everyone can install that on their machines without having to modify their own main OS at all.

With EFI there appears, from what I've been learning in various threads, that some capability is going to disappear, and if you want to get it back you will for example have to support a great number of graphics card drivers in your OS to do hardware scrolling. An auxiliary OS would fix that for every OS that works with it, assuming all the relevant device drivers exist, and if the auxiliary OS was used widely it might get to the point where universal drivers are written by the manufacturers whenever they produce a device that needs one.

Anyway, it isn't going to happen soon enough to matter, so I'm not going to go on discussing it for now, but I will make that demo in the future just to show how elegant the idea could have been.

DavidCooper wrote:

Love4Boobies wrote:I would now like for you to put together a list of advantages. All you've done so far was to describe how the thing would work; it's time to describe why. No better way to convince people that pro/con lists.

I've given you plenty of reasons above. The main one though is simply to recreate the real mode BIOS services in 32-bit mode form and to add further functionality too, thereby making it easier for people like most of the members of this site to create their own OS and build something powerful in a short time. Most of the power would come from the auxiliary OS, but that isn't any different from old real-mode OSes getting most of their functionality from the BIOS.

I believe the original IBM PS/2 range had a protected mode BIOS intended for use with OS/2. Although many PS/2 features made it into mainstream PC design, the new BIOS wasn't one of them. I don't know the reasons why, perhaps IBM imposed licensing restrictions as they'd done with the Microchannel architecture. I agree this is kind of a shame... if it had become a standard, someone wanting to write a new PC operating system today would have a jump start for the initial hardware support. Your system might be limited to VGA only, no networking, no sound (except beeps), but you could get universal support for the basics with much less effort.

I'm quite sure this is all pie in the sky in that it won't happen, so there isn't a lot of point in discussing it further, but I thought it might be interesting to throw the idea into the ring just to hear what people thought about it, and Oranos in particular (since he's sitting on some secret plan which he isn't keen to reveal to us - I thought he might just drop another clue our way, but he's switched to sending cryptic private messages instead). Anyway, it's probably too late to be worth bothering to start on an auxiliary OS project now as it's likely to be overtaken by events long before it can get anywhere useful, so it should at best be considered an idea that missed its time, and at worst a lame idea from someone who knows next to nothing.

I could have some merit, but to re-create a that type of scheme without the motherboard manufacturer's support would be a nontrivial undertaking. The good thing about BIOS routines is that the motherboard manufacturers do them for you. Even so, limiting the scope of some independent "universal driver" project to just the bare basics might make it more manageable than something like UDI, which attempts to cover all possible operating systems and all possible hardware, past, present and future. The trouble with big ideas like that is that often they're either too big and complex to possibly work, or they impose restrictions on one side of the equation. If instead some project focused on just a minimal feature set, not even focusing too much on optimal performance in all cases (in the expectation that more advanced operating systems would bypass them for more flexibility or better performance) it might have a greater chance of success than resurrecting a UDI equivalent idea.

It'd have to deal with some issues of developer motivation though. Hardware manufacturers have a motive to write drivers, they want to sell stuff that works with existing operating systems (mainly Windows, sometimes Linux). Operating system developers have motive to write drivers, because they want their systems to work with existing hardware. Where's the motive for someone (who is not selling their hardware) to write a driver, even a limited one, for someone else to use in their operating system? Seems like some kind of "tragedy of the commons" type scenario.