OS Graphics

[Brendan]

Hi,

OK - that works for games which want "realistic" HDR by some definition. What about games which want to simulate intentionally non-realistic HDR?

Why would a game want to simulate intentionally non-realistic HDR?

Cartoon shading? Psychedelics? The Rolling Stoneds Screensaver?

For cartoon shading create "cartoon shaded textures" (or just use solid colour polygons instead of textures), and if you don't want lighting and shadows use ambient light and no other light sources.

For psychedelics, if "psychedelic textures" isn't enough then I'll just add a "convert colours to psychedelic for this texture" command that people can use for a game's window. I'll also want a few commands like this for other things (the only one I can think of at the moment is a "night vision" effect where intensity gets converted into shades of green). For the final texture (what gets sent to the monitor) I'm going to want hue shifting for colour blind users anyway, so supporting "colour remapping" for other uses isn't going to be a huge problem.

For "Rolling Stoneds" I'm not sure what you mean (and googling didn't help).

I don't see it that way at all. I see it as a huge failure to hide hardware details from application developers, forcing them to all waste time implementing competing graphics engines that all do the same thing that the graphics system should have already done; except that the failure is so huge that the games developers can't cope with all the variations between all the different video cards and end users end up with completely avoidable hassles just trying to figure out if their video card will or won't run a game (assuming the user hasn't given up and got a "fixed hardware" console already). See if you can convince someone writing a normal application (e.g. text editor, IRC client, bitmap image editor, etc) to add a few simple 3D effects and see how quickly they refuse to deal with the mountain of hassles it'd cause them.

The industry, especially for rapidly developing systems, is always making tradeoffs for which percentage of systems to support - typically from the high ends down. Some even have graceful degradation to not use certain graphical features if they would be too demanding on the hardware in question. In practice it ends up being the visual designer wanting to be in charge, and the tech department retaliating with their proverbial drug kickoff clinics.

You're right; but this is an historical artefact that evolved from OSs that sucked (e.g. OS that didn't have 3D APIs forcing developers to bypass the OS and bork with graphics hardware in a much more direct way, and standards that changed too slowly because they were trying to cope with the more direct way that developers grew to expect rather than a higher level abstraction doesn't need to change as often). It isn't how I want things to be.

Bottom line, the developers want to be in charge on what hardware features get to be used, and they typically want to use the latest ones if available, and they're certainly not going to wait for some design committee to spend three years on the next engine standard, and the hardware developers are certainly not going to wait for some design committee to spend three years to get their next "great feature" rolled out to their customers. And the gaming addicts... you get the drill.

Those developers that want to be in charge of what hardware features get to be used can help write video drivers instead; and the developers that want to use 3D in their applications can breathe a sigh of relief.

The whole lesson of this all is that performance gaming is currently just disjunct from desktop stuff. It might change for graphics, but HPC will certainly fill in that particular gap for the time to come.

Performance gaming is currently very disjunct from desktop stuff, which is a symptom of the problem. I want all the "2D desktop stuff" to be 3D - windows/floating toolbars/dialog boxes that have thickness/depth, buttons that are actually raised, text boxes that are actually recessed, shadows that are more than just "baked on" and black smudges glued onto the window edges, etc. I want a GUI with a light source that moves from left of screen to top of screen to right of screen throughout the day, where an office worker can use their mouse pointer like a sun-dial and say "Hey, must be close to lunch time" just from seeing where the mouse pointer's shadow falls. I want to be able to hold down my left mouse mouse button and "alt" key to grab a window and then move the mouse in any direction to spin that window around so that I can see what's written on the back of it. I want windows that shatter when they're closed sending shards of ice bouncing off of other things as they fall towards the bottom of the screen, where some of the shards come to rest on the tops of other windows and icons and melt. On a hot summer's day I want set the GUI's climate control to "5 minute light shower" and watch as the raindrops hit a dialog box and drip of the bottom edge and create small streams of running water that trickles down between my desktop icons. I want to set the GUI to "free flight mode" and see how quickly I can fly laps around my text editor's window, and accidentally crash into the edge of a floating toolbar that was carelessly placed between the back of the text editor's window and the front of the word processor's window. I want to set the GUI's theme to "jungle" and occasionally see a bird fly out of that jungle and land on top of my web browser's window and preen it's feathers, then turn to look at me before flying back into the jungle; leaving behind some bird poo that oozes down my web browser's menus and pools on the small raised edge around a recessed address bar until it dries and slowly fades away.

I want all the "2D desktop stuff" to be 3D.


Cheers,

Brendan

[Brendan]

Hi,

Handling 3D graphics and UIs in an uniform way sounds like a horrible idea, you'll just end up with a system that doesn't handle either very well. Games absolutely need as much control as possible over the GPU to achieve the best results.

See if you can find a game made in 2000 that supports all of the features of video cards made in 2010 and try to explain how all those games managed to achieve the best results. Now try the opposite - find a game made in 2010 that gets the best results from a video card made in 2000.

What you're advocating is a situation where specific games only get the best results from specific video cards (and only if the developers waste huge amounts of time and money to get the best out of those specific video cards).

Programmable shaders were invented to give artists the freedom to innovate, not to be hidden away by a generic solution that applications are forced to use. Artists should be able to program any effect that the hardware is capable of without waiting for it to be implemented in your "ultimate 3D engine to end all 3D engines". Furthermore, figuring out which parts of a texture will be accessed in a drawing operation is impractical for anything but the most trivial rectangle copies. Most clipping (beyond the final screen output) will have to be done by the application itself, but if you provide an automatic back buffer linked to a window, you could help by automatically cropping the clipping rectangle according to the bounding box of the window's visible region.

Artists (and not programmers) write shaders do they?

Let's be more realistic and define 4 groups of people:

• Artists, who create art
• Game developers, who use artists and game engines to create a game
• Game engine developers, who try to sort out the shaders for game developers
• Video device driver writers

For your system, the game engine developers are considered part of the game developers. For my system, the game engine developers are considered part of the video device driver writers. In both cases it's the mostly the same people doing the same things.

The differences are that:

• For your system, if 10 different game developers all want a fancy shader for water then 10 different game engine developers need to write 10 different shaders (for 10 different video cards); while for my system the shaders would be part of the video driver and there'd only be 10 instead of 100 of them.
• For your system, once a game is released development stops and the end user is screwed. For my system the user benefits from any future improvements to the shaders built into their video driver and can upgrade to a completely different video card in 10 years time and still get the latest and greatest shader improvements
• For my system, if the game developer wants something different then they're going to have to wait while the specs and video drivers catch up; while for your system game developers get more flexibility. This was an important benefit when the industry was young and evolving rapidly; but it's not as important now.

On the other hand, other applications usually draw directly to the screen or to a back buffer provided by the system. In this case, a clipping region has already been set by the window manager, so there is no problem. The description you give of "most existing graphics systems" is simply incorrect and is not at all what happens (with normal GUI applications), except possibly in hobby operating systems in early development. The processing of pixels that will not be visible is mainly an issue with badly programmed applications, and games, which are expected to remain in the foreground while in use.

You're saying that my deliberately simplified example didn't cover all the intricate little implementation details, complexities and optimisations (of either approach)? Wow, I would never have realised...

Why would a game want to simulate intentionally non-realistic HDR?

This shows that you don't have experience in developing games. As a game developer you need to be in total control of what appears on the screen. If your particular implementation of HDR hampers gameplay by making the player unable to see something important, I'd like to turn it off. The OS has no business meddling in someone's artistic decisions. You can't possibly foresee what kind of lighting someone will eventually want.

If your game is badly designed and HDR hampers gameplay; shift your light sources (or reduce their intensity) and stop trying to pretend it's my fault.

In one of my applications, I need to colorize some pixels by using some other rendered pixels as a 1D texture index, but only when adjacent pixels differ, and only when the value stored in another rendered texture is greater than the current depth value. I also need to apply fog on some pixels (determined by a stencil bit) by indexing a 1D texture with a value determined by stored depth values, the screen position and stored focal widths (per pixel). This is easy with DirectX shaders, but would your system handle this? I'm guessing not.

As an alternative to your work-arounds for unspecified problems, I offer unspecified solutions and claim that my unspecified solutions are a superior way to solve your unspecified problems.


Cheers,

Brendan

[rdos]

This is the problem your arguments always present. You are arguing in favour of terribly constricted specifications and single-use environments because the product you ship is only used in one way at a time, which is very specific and great for embedded systems but is totally out of the question for everyone else here.

I'm being realistic. Nobody doing a hobby OS project will ever succeed at getting it spread as a super-OS alternative wiping out competitors like Windows or Linux. By being realistic about what to implement you at least have a chance to get something up and working. Brendan's project is so ambitious that he will need 100s of years to get it implemented and tested, and by the time he is done with that, his specifications are obsolete. OTOH, he never will be done as when he has worked a while on the project the ideas for doing a super-OS have changed and he goes back to the specification stage again and starts all over with the boot-loader.

We're trying to build operating systems that have some use in a modern general user environment, and you're arguing against our ideas with references to a design scheme that's several worlds away from what we're doing.

Mine obviously does, and as long as I update it for new standards and chips, it will continue to do so. In fact, I've made a major leap in supporting modern hardware the last few years which means that today I can run on most (x86 based) computers, which was not the case before. Thus, I would rather argue that I'm up-to-date with modern hardware.

In regards to design, I could just as well argue that most of you base your design on outdated ideas from the 70s. My design was in no way influenced by typical OS design because I didn't read all the books on OS development, but rather came up with the solutions myself. Some of them were good while some were bad. Most of the bad one's have been replaced while the good one's remain.

Your product sells to specific users who need an antiquated mono-interface environment? Great. You've succeeded in your niche. The rest of us sell to the general public, and our products must be radically different than yours.

No, it sells because it is a stable product with a small foot-print that can easily be installed, configured and updated.

I also doubt that any of you sell to the general public. Most of you seem to be fiddling with boot-loaders while having grandiose dreams about making the super product that will replace Windows and Linux.

[Antti]

Nobody doing a hobby OS project will ever succeed at getting it spread as a super-OS alternative wiping out competitors like Windows or Linux. By being realistic about what to implement you at least have a chance to get something up and working. Brendan's project is so ambitious that he will need 100s of years to get it implemented and tested, and by the time he is done with that, his specifications are obsolete.

Brendan will probably answer this but at least I am going to disagree with you. I am quite sure that most of the commercial (non-hobby) projects have done things with a more hobby-like approach than Brendan is doing his project. For example, when it comes to the adequate background research of things being designed and implemented. I am quite sure that even if this was a "professional" project made by a company, it would not make it any better at this point. It would not help much to have an army of developers fighting with each other of how to do things. Especially when things are new and never implemented before. Here we can see a clear vision and everything is done in a uniform way following that vision.

As soon as there is something real to show (e.g. simple applications that use new features), it would be much easier to convince people that the fundamental ideas are better than what we have seen so far with other OSs. At that point, it will be easier to get other people involved in. For example, people like me will start making applications for this new system. I am not the only one. It will then start growing exponentially and will not take hundred of years to become something useful. What makes it different when compared to Linux, for example, is that this time everything is really designed from the very beginning and formal specifications of how to do things exist. I hope we will seen some demos (to show things) in the future. After two years maybe? Brendan, does that sound realistic?

[Brendan]

Hi,

This is the problem your arguments always present. You are arguing in favour of terribly constricted specifications and single-use environments because the product you ship is only used in one way at a time, which is very specific and great for embedded systems but is totally out of the question for everyone else here.

I'm being realistic. Nobody doing a hobby OS project will ever succeed at getting it spread as a super-OS alternative wiping out competitors like Windows or Linux. By being realistic about what to implement you at least have a chance to get something up and working. Brendan's project is so ambitious that he will need 100s of years to get it implemented and tested, and by the time he is done with that, his specifications are obsolete. OTOH, he never will be done as when he has worked a while on the project the ideas for doing a super-OS have changed and he goes back to the specification stage again and starts all over with the boot-loader.

I've done it that way already: you spend years getting something "realistic" (limited) working, and after that you end up with having to make a decision - persevere with something that you know is "realistic" (limited) that nobody will ever actually choose to use; or attempt to do something less limited that users might choose to use. This is where we made very different choices - you persevere with "limited" while I attempt "more ambitious".

Please note that I suspect that your "users" come to your company looking for an embedded system of some kind, and your company creates it for them. They choose to use your company (most likely in a "lowest bid for the contract wins" way) but don't get any chance to decide which OS ends up being used. Sadly; nobody else here has the opportunity to force a "limited" OS on victims and then try to pretend that our "users" actually made an informed choice.

I also doubt that any of you sell to the general public. Most of you seem to be fiddling with boot-loaders while having grandiose dreams about making the super product that will replace Windows and Linux.

Most of the people here are learning or just writing a hobby OS because it's fun. They aren't trying to write "a super product that will replace Windows and Linux" at all.

Some of us are trying to write something more serious; but none of them (that I know of) are trying to write "a super product that will replace Windows and Linux" either. For example, my project is about seeing how much better/worse software could be if everything was designed from scratch without any limitations caused by compatibility; and because of this it's much more like a research project than something intended for actual users.

However; if users are able to make an informed choice, then you do need to make sure your OS is better than the other choices for that user's needs (otherwise they'll just choose something else). For example, if your "users" were able to choose between RDOS, QNX, embedded Linux or Windows CE, then you'd have no users at all.


Cheers,

Brendan

[Kazinsal]

Nobody doing a hobby OS project will ever succeed at getting it spread as a super-OS alternative wiping out competitors like Windows or Linux.

Never said that was a goal.

In regards to design, I could just as well argue that most of you base your design on outdated ideas from the 70s. My design was in no way influenced by typical OS design because I didn't read all the books on OS development, but rather came up with the solutions myself. Some of them were good while some were bad. Most of the bad one's have been replaced while the good one's remain.

You sound like Dr. Andrew Tanenbaum, and not in a good way, but in more of a "Linux is Obsolete" flamewar way.

No, it sells because it is a stable product with a small foot-print that can easily be installed, configured and updated.

And works for one task that you choose to support, not whatever the user may want to later do with it, from what I gather.

I also doubt that any of you sell to the general public. Most of you seem to be fiddling with boot-loaders while having grandiose dreams about making the super product that will replace Windows and Linux.

Replace "sell" with "distribute and offer support" and you're a lot less bang-on than you think you are. An example being my motivation to work on my operating system -- a former CS teacher of mine wants to use my project as a tool to teach an intro to systems programming to her students. Each of these students make up a group of people who use the operating system in different ways and may end up recommending it to others. That's a whole diverse group of users in the general public.

The majority of us don't want to replace Windows and Linux -- we want to develop an alternate idea and see what people think about it once they get to try it.

[rdos]

I've done it that way already: you spend years getting something "realistic" (limited) working, and after that you end up with having to make a decision - persevere with something that you know is "realistic" (limited) that nobody will ever actually choose to use; or attempt to do something less limited that users might choose to use. This is where we made very different choices - you persevere with "limited" while I attempt "more ambitious".

That's because my project has 25+ years of development. There never was any intention of providing anything limited. The goal was to write a superior OS that didn't have any of the limitations at that time. But as time goes by other OS alternatives develop, and then you can either decide it is time to start from scratch with something that is superior at that point, or build on what you already have. Obviously, I decided the latter while you seem to prefer the former.

Please note that I suspect that your "users" come to your company looking for an embedded system of some kind, and your company creates it for them. They choose to use your company (most likely in a "lowest bid for the contract wins" way) but don't get any chance to decide which OS ends up being used. Sadly; nobody else here has the opportunity to force a "limited" OS on victims and then try to pretend that our "users" actually made an informed choice.

Not so. Naturally, I've had to sell the idea to managers, and they have agreed. I also had to sell the idea to our collaborators in another company, and they also agreed. They did this because we have a working product that is easy to install and maintain. I wouldn't be able to do this if I didn't have a stable version that had been deployed on a number of sites.

In fact, I managed to teach a person how to build and install the required tools, and let him write a multithreaded application himself that worked in about 6 hours time. In that time, you'd hardly be able to install the development tools for Windows or Linux, and much less expect a user that never had seen the tools to write a multithreaded application with them.

Some of us are trying to write something more serious; but none of them (that I know of) are trying to write "a super product that will replace Windows and Linux" either. For example, my project is about seeing how much better/worse software could be if everything was designed from scratch without any limitations caused by compatibility; and because of this it's much more like a research project than something intended for actual users.

And you think that I didn't do this for RDOS??

[Antti]

The goal was to write a superior OS that didn't have any of the limitations at that time.

When looking at history, year 1994 for example, what was implemented then in RDOS? Windows 95 was not released and Linux was not very mainstream yet. A real Unix was not an option for home PC users. Then it would have been possible to be superior when compared with MS-DOS and Windows 3.1. I am not making fun here. I am just interested in features you had then in RDOS. Are you happy of what you have done or do you feel you missed your opportunity?

[bluemoon]

Woot there are so many replies on this thread. I have tey the read them all but a quick scan I got some idea:

if 10 different game developers all want a fancy shader for water then 10 different game engine developers need to write 10 different shaders (for 10 different video cards); while for my system the shaders would be part of the video driver and there'd only be 10 instead of 100 of them.

For most games you don't need to write 10 shaders all from scratch. You may use 5 "standard shaders" to do general works, and they happen to be open source or well studied subject that it can be implemented easily.
Then you may use 3 from your own library (or company library) from past projects.
Lastly, depends on resources, you may develop 2 from scratch or by tuning existing shaders.
* We assume 10 shaders are needed, you may also consider it as percentage of workload for games that require more shaders.


=========================
I think Brendan's idea may work on some specially considered/targeted scenario (distributed, etc).
There are different ways to handle the distributed problem (provide support in OS level so applications may run transparently; do it like X11; let application do there own client/server protocol; or even web-based, etc)

While his optimization may be un-necessary for usual one-monitor(or dual head card), the idea may indeed fit for NFC-based UI, where the render device may have very limited resource and bandwidth.

We just have different consideration and target environments.

[XanClic]

I quite dislike Brendan's idea of using the GPU for photorealistic rendering only and having the GUI be a special kind of (therefore realistic) 3D application.

First, games are games. If I want reality, I go outside. There are photorealistic games, but there are also quite a few games which use completely unrealistic rendering to enhance gameplay. You, Brendan, seem to want to limit those to a few options you choose to implement (“toon shaders” etc.), but that is indeed a limitation.

Second, when I hear “GUI which is a 3D landscape”, I really got to hold on to my chair. I'm thinking of MS Bob here, at best. Probably worse. I seriously cannot imagine how that should look good in any way, nor how it can be practical to use. MS Bob was at least 2D, so you could quickly reach whatever you wanted to. In 3D, there's a whole new dimension you've got to cross if you want to traverse your applications.

As far as I've understood, you want to implement a whole graphics (and probably also physics) engine as part of your OS? If you don't, at least my first point is invalid and you're free to discard it. If you do… You know there are already such engines out there and people may use them if they want to? For several, non-commercial use is free of charge, some are generally free and sometimes even FOSS. I don't see the point why anyone would prefer an engine provided by the OS over a (probably portable) library. The UDK, for instance, is quite good as far as I've heard and it's also pretty cheap to use (free, if non-commercial, as far as I see it).

[rdos]

The goal was to write a superior OS that didn't have any of the limitations at that time.

When looking at history, year 1994 for example, what was implemented then in RDOS? Windows 95 was not released and Linux was not very mainstream yet. A real Unix was not an option for home PC users. Then it would have been possible to be superior when compared with MS-DOS and Windows 3.1. I am not making fun here. I am just interested in features you had then in RDOS. Are you happy of what you have done or do you feel you missed your opportunity?

The main object was to implement mutilthreading (and multiprocessing) in a DOS compatible environment. That was functional by 1994. However, as DOS became obsolete I started to build a Win32 emulation layer by providing important DLLs (I actually got most of them from another project, but can't remember the name right now). Anyhow, some 5 years later I could run some important Win32 console applications, including many of BCCs tools, and the text mode debugger. Somewhere around 2008 I rediscovered OpenWatcom, and then I implemented a native C libary so I didn't need Win32 emulations anymore. The GUI and widget library also is rather recent.

So, no, I don't think I missed anything, but I did have some luck, like the long-lived IA32 architecture, and getting the right job in the mid 90s.

[Brendan]

Hi,

I quite dislike Brendan's idea of using the GPU for photorealistic rendering only and having the GUI be a special kind of (therefore realistic) 3D application.

First, games are games. If I want reality, I go outside. There are photorealistic games, but there are also quite a few games which use completely unrealistic rendering to enhance gameplay.

Please understand that there's a massive difference between "realistic rendering" and "reality". For example, you can have a game where the player is a yellow sphere with a slit for a mouth who runs (without arms or legs) through corridors filled with pills that float in mid-air (un-effected by gravity) while trying to avoid ghosts; and regardless of how realistic the graphics are for this game, if you turn the computer off and go outside to "reality" you won't see yellow spherical people running around, or pills floating in mid-air, or ghosts. Trying to suggest that "realistic rendering" has anything to do with "reality" is silly.

Now think about how you might do the graphics for that game. You could use 1st person 3D, with detailed textures for walls and floor, complex animated meshes for the player and ghosts (complete with real facial expressions captured from actors, etc), realistic shadows and lighting, special effects, etc; and try to make the graphics look as realistic as possible. Alternatively; you could use 3D with the camera looking down from above, with no textures (e.g. all walls could be "plain blue"), ambient lighting only (no shadows, no specular effects, etc) and no other special effects; and this might look almost as ugly as the original "1980's 2D" version of the game (just with higher resolution). Trying to suggest that a renderer capable of "realistic rendering" can only be used for "realistic rendering" and that you can't just use a subset of the renderer's features is foolish.

You, Brendan, seem to want to limit those to a few options you choose to implement (“toon shaders” etc.), but that is indeed a limitation.

If I can be convinced that failing to shield application/game developers from the hassle/burden of shaders is actually beneficial (and that shaders aren't just something normal application developers don't want to touch and game developers have grown used to); then (for my "list of commands" approach) I'd just add a "load_shader" command (that tells the video card to load a shader that uses some sort of standardised byte-code for shader language, like GLSL) and deal with the consequences (e.g. needing a "shader byte-code to GPU native" compiler in the video driver that I wouldn't have needed otherwise); and I'd still keep using the "list of commands" approach.

Basically; the entire discussion about shaders has nothing to do with "list of commands vs. raw pixels" anyway.

Second, when I hear “GUI which is a 3D landscape”, I really got to hold on to my chair. I'm thinking of MS Bob here, at best. Probably worse. I seriously cannot imagine how that should look good in any way, nor how it can be practical to use. MS Bob was at least 2D, so you could quickly reach whatever you wanted to. In 3D, there's a whole new dimension you've got to cross if you want to traverse your applications.

If someone was giving you a free Ferrari and told you the car is capable of doing 340 km/h, would you tell them that you don't want the free Ferrari because sometimes you want to drive slower? If someone gave you a free OS that's capable of supporting 3D GUIs well, why would you immediately assume that every GUI for that OS must use all the 3D features all the time (and fail to understand that a GUI for that OS could also be made to look as crappy as some monochrome 2D GUI from 3 decades ago)?

As far as I've understood, you want to implement a whole graphics (and probably also physics) engine as part of your OS? If you don't, at least my first point is invalid and you're free to discard it. If you do… You know there are already such engines out there and people may use them if they want to? For several, non-commercial use is free of charge, some are generally free and sometimes even FOSS. I don't see the point why anyone would prefer an engine provided by the OS over a (probably portable) library. The UDK, for instance, is quite good as far as I've heard and it's also pretty cheap to use (free, if non-commercial, as far as I see it).

Rather than thinking about how different pieces of software communicate (e.g. how applications communicate with GUIs, what sort of video driver API to use, etc) and trying to make the interfaces between these pieces better (e.g. more flexible); I should just gather random pieces of code that other people have written for completely different purposes and stitch them together in strange ways (e.g. slap code from the unreal developer's kit into things like text editors, widgets, GUIs, etc), and then hope that the result won't be a hideous abomination?


Cheers,

Brendan

[Brendan]

Hi,

Please note that I suspect that your "users" come to your company looking for an embedded system of some kind, and your company creates it for them. They choose to use your company (most likely in a "lowest bid for the contract wins" way) but don't get any chance to decide which OS ends up being used. Sadly; nobody else here has the opportunity to force a "limited" OS on victims and then try to pretend that our "users" actually made an informed choice.

Not so. Naturally, I've had to sell the idea to managers, and they have agreed. I also had to sell the idea to our collaborators in another company, and they also agreed. They did this because we have a working product that is easy to install and maintain. I wouldn't be able to do this if I didn't have a stable version that had been deployed on a number of sites.

So my guess was quite close - the actual end user's don't get a choice; a long time ago (when the weren't so many alternatives to choose from) your manager chose to go with RDOS; and some time between "a long time ago" and now one other company agreed to go along with the decision your manager made.

The point I'm making here is that it's extremely unfair to expect all of the hobby OS developers here to be in that exact same situation that you were. We can't jump into a time machine and go back 20 years; we can't rub a magic lamp and ask a genie for a manager who will accept our OS project (and fund development) as part of out 3 wishes; and (if we are serious about attracting users to our OS one day) we do have to compete with many existing alternatives and do have to give potential users some reason to choose our OS instead of those alternatives.

Some of us are trying to write something more serious; but none of them (that I know of) are trying to write "a super product that will replace Windows and Linux" either. For example, my project is about seeing how much better/worse software could be if everything was designed from scratch without any limitations caused by compatibility; and because of this it's much more like a research project than something intended for actual users.

And you think that I didn't do this for RDOS??

For an example, (a long long time ago) I looked at the way/s IPC is done in *nix (pipes and sockets and streams and signals and ... yuck). I didn't like it. I looked at lots of different types of messaging, and it looked a lot cleaner to me. I knew I wanted a distributed system and that synchronous messaging wouldn't work too well for normal applications, but I thought it'd be a good idea to use synchronous messaging to reduce overhead for device drivers. I implemented a "sort of monolithic" kernel where device drivers used segmentation and got loaded into the kernel space and communicated with synchronous messaging, and normal applications used asynchronous messaging to talk to each other (and to talk to device drivers). This worked fine, but I didn't like it. Then I tried the micro-kernel approach, with device drivers as processes that use asynchronous messaging. I implemented the kernel, then implemented some drivers, then implemented some applications, etc. I liked it a lot (and it's part of my OS design that I've kept ever since). Did you do something like this for RDOS? I assume you just implemented whatever seemed like a good idea in your first kernel and got stuck with it.

For an example, I had ideas for 2D image file formats rolling around in the back of my head for about 6 months. I came up with several possibilities, found problems with most of them, refined others, etc. One of those possibilities was the idea of using solid and shaded triangles for everything (as they're extremely easy to scale, rotate, render with anti-aliasing, etc). I spent about 3 months writing a prototype/utility to convert bitmap files to my file format (and convert my file format into bitmap files) and experimenting with the idea. The end result was that I discarded the "solid and shaded triangles for everything" plan - converting raw pixel data back into the least number of triangles (while getting good image quality and file sizes) proved too computationally expensive and relatively complex. It wasn't a total loss though - I've formed a new "solid, shaded and textured triangles for everything" plan, where bitmap data doesn't need to be converted but you still get the (scaling, rotating, etc) advantages. I haven't written a formal specification for the file format or implemented a prototype for it yet; but it's not my highest priority at the moment and I will one day. Did you do something like this for RDOS? I assume you just grabbed some open source PNG library and slapped it into your OS.

For an example; so far I've probably had at least 6 different designs for "boot code" (different stages, different responsibilities, different capabilities, different ways for the pieces to communicate, etc); starting from extremely simple "text mode only" with little more than "load kernel and jump" from the early days; leading up to glorious modular pieces of art with full graphical "boot menu" windowing systems (with layers with lighting/shadows, and resolution/colour depth independence, and dithering, and fonts that are scaled to suit the physical size of the monitor), and fault tolerance features (like RAM testing and faulty RAM avoidance), and built-in CPU feature detection and "AP CPU startup", and code to auto-detect the best kernel out of many kernels to actually start. Did you do something like this for RDOS? I assume you just cobbled together a boot loader when you were just starting and haven't been able to make any significant changes to it since.

Do I need to talk about memory management, or schedulers? NUMA optimisations? Executable file formats? How about programming languages (I'm currently researching that - designing languages and writing a crappy tool-chain)?

No; I don't think you've done anything like this for RDOS. To be honest; I don't think anyone here (other than me) has done anything like this for their project.


Cheers,

Brendan

[Combuster]

To be honest; I don't think anyone here (other than me) has done anything like this for their project.

After four kernel rewrites, I'd beg to differ.

[rdos]

For an example, (a long long time ago) I looked at the way/s IPC is done in *nix (pipes and sockets and streams and signals and ... yuck). I didn't like it. I looked at lots of different types of messaging, and it looked a lot cleaner to me. I knew I wanted a distributed system and that synchronous messaging wouldn't work too well for normal applications, but I thought it'd be a good idea to use synchronous messaging to reduce overhead for device drivers. I implemented a "sort of monolithic" kernel where device drivers used segmentation and got loaded into the kernel space and communicated with synchronous messaging, and normal applications used asynchronous messaging to talk to each other (and to talk to device drivers). This worked fine, but I didn't like it. Then I tried the micro-kernel approach, with device drivers as processes that use asynchronous messaging. I implemented the kernel, then implemented some drivers, then implemented some applications, etc. I liked it a lot (and it's part of my OS design that I've kept ever since). Did you do something like this for RDOS? I assume you just implemented whatever seemed like a good idea in your first kernel and got stuck with it.

RDOS has a network transparent messaging API that doesn't run on top of TCP/IP. It uses it's own protocol under IP (SMP/IP), which is registered with IANA. The specification is available on my site: http://www.rdos.net/smp/ . Now please tell me that you are not distributing your system with TCP/IP, but with some efficient message protocol that also is registered with IANA?

As for the OS API itself, it doesn't use the "int" approach typical of the time (because it sucks), and neither does it multiplex the interface into any single interface, but instead uses code-patching in kernel and call-gates on IA32. The kernel isn't linked into one huge binary, rather consists of a number of different files that are combined with a tool into a single image file. This single image file also contain settings and can be proloaded with applications in disk-less systems. None of this was typical of the time, and still is not very common.

Another feature which I regard as essential, but which seems to be unusual, is the ability to use the user level API from kernel. Why would anybody want to invent a separate API for the functions available to use level? Maybe because they haven't got past the crappy "central API handling entry-point" stuff yet?

For an example, I had ideas for 2D image file formats rolling around in the back of my head for about 6 months. I came up with several possibilities, found problems with most of them, refined others, etc. One of those possibilities was the idea of using solid and shaded triangles for everything (as they're extremely easy to scale, rotate, render with anti-aliasing, etc). I spent about 3 months writing a prototype/utility to convert bitmap files to my file format (and convert my file format into bitmap files) and experimenting with the idea. The end result was that I discarded the "solid and shaded triangles for everything" plan - converting raw pixel data back into the least number of triangles (while getting good image quality and file sizes) proved too computationally expensive and relatively complex. It wasn't a total loss though - I've formed a new "solid, shaded and textured triangles for everything" plan, where bitmap data doesn't need to be converted but you still get the (scaling, rotating, etc) advantages. I haven't written a formal specification for the file format or implemented a prototype for it yet; but it's not my highest priority at the moment and I will one day. Did you do something like this for RDOS? I assume you just grabbed some open source PNG library and slapped it into your OS.

The graphic API was constructed from scratch. It was modeled after PicoGUI, but uses no code from that project. The GUI was also written from scratch, as was the widget toolkit. The PNG library naturally was ported as there is no sense in writing that one your own. The font interface was ported from FreeType, as there is no need to reinvent the TrueType interface either.

As for 3D and rendering, I've not yet even considered what to do with that. I (or somebody else) might implement something, but there are no such plans at the moment.

For an example; so far I've probably had at least 6 different designs for "boot code" (different stages, different responsibilities, different capabilities, different ways for the pieces to communicate, etc); starting from extremely simple "text mode only" with little more than "load kernel and jump" from the early days; leading up to glorious modular pieces of art with full graphical "boot menu" windowing systems (with layers with lighting/shadows, and resolution/colour depth independence, and dithering, and fonts that are scaled to suit the physical size of the monitor), and fault tolerance features (like RAM testing and faulty RAM avoidance), and built-in CPU feature detection and "AP CPU startup", and code to auto-detect the best kernel out of many kernels to actually start. Did you do something like this for RDOS? I assume you just cobbled together a boot loader when you were just starting and haven't been able to make any significant changes to it since.

To begin with, certainly. However, there are now boot-loader for both BIOS and GRUB (multiboot). There will probably be for UEFI at some point. The job of the boot-loader is to bring up the system, and so needs nothing more than the absolute minimum in order to work.

What I regard as important instead of fancy boot-stuff is debug ability of the start-up process on real hardware. Thus, you can today put breakpoints almost anywhere in the startup code and dump register content. In addition to that, there is a tool for SMP trouble-shooting as well which can halt the system (automatically on severe faults or manually) to allow inspection of state of CPUs.

In regards to many kernels I feel that is an extremely bad idea. Some functions in the scheduler are different between SMP and single-core, but those are patched in the code at startup so there is no need for multiple schedulers. The kernel also detects things like the availability of PAE paging and then can install either hooks for PAE paging or normal paging. This doesn't require compiler switches or different kernels. Long mode support is loaded as an device-driver. RDOS today could run everything from DOS to 16-bit protected mode (NE), PE (Win32 and native), ELF and long mode applications without any need to configure the kernel or build multiple versions of it.

I'm sure that your design didn't look into debugging the startup-process on real hardware or how to support multiple environments with a single kernel.

Do I need to talk about memory management, or schedulers?

Those have a history of multiple versions. The one's in use today are what I consider adequate.

NUMA optimisations?

That's of no interest for me.

Executable file formats?

As of above, I support many, with the same kernel.

How about programming languages (I'm currently researching that - designing languages and writing a crappy tool-chain)?

BCC, OpenWatcom, GCC.

To be honest; I don't think anyone here (other than me) has done anything like this for their project.

You're wrong.