OSDev.org

Brendan wrote:There's never a need to synchronise user input and physics with the frame rate. Sadly; there are a lot incompetent and/or lazy and/or stupid game developers that have something like a "do { get_user_input(); update_AI(); do_physics(); update_screen(); }" main loop. These people need to be prevented from going near any computer ever again (and are the reason why gamers want "500 frames per second" just to get user input polled more frequently; and the reason why most games fail to use multiple threads effectively).

It's nice to have strong opinions, but don't be so quick to dismiss everyone who follows that formula. Not all game developers are using the same hardware you are.

This is a fairly common discussion when talking about system design with embedded logic. You have two components that must communicate with each other, and you have to decide "how" they will talk to each other (what their "interface" will be.)

We are all pretty familiar with interfaces between hardware and software, and the fact that the interface between them dictates where the majority of the "logic" will be. For instance, a floppy drive controller has an interface that gives the CPU direct control over the state of the drive. It can move the read/write head, start and stop the disk motor, control which drive is being addressed, etc. The advantage to this is that you can do some interesting things like play midi files on your floppy drive.

https://www.youtube.com/watch?v=8kQYRquhcXU

The disadvantage is that it limits the changes you can make on the hardware side. Simply increasing the disk capacity requires the entire OS to be essentially redesigned. Compare this to modern USB storage devices, where the CPU has almost no control over the hardware. It can only make requests, and wait for the device to handle all of the logic, itself.

The same applies to essentially all interfaces between two "logical" entities. They have to agree on who is responsible for doing what, and when. Even when two people work together, they have to work out the "boundaries" concerning who is responsible for what, so that there are no "gaps" where something is missed, and so that effort isn't duplicated on both ends unnecessarily.

The difference is that people can "adjust" those interfaces over time, automatically, to improve overall performance. Hardware has little to no control over its interface to the rest of the system. Software has a little more control, but usually requires upgrades to adapt to changes in its interfaces.

But back to the discussion at hand, choosing the interface between two components is always a trade-off between flexibility and complexity. Since these interfaces usually can't be changed once they have been implemented, the way that this problem is addressed is to layer multiple interfaces together, usually from complex interfaces on the bottom (in this case on the hardware side) to simple interfaces on the top (on the application/user side). The software usually has several options to choose from when working with hardware -- either low level APIs which require the application to handle more "logic", or high level interfaces that are simple for the application to use, but require more "logic" on the OS side or the hardware side.

Neither approach is necessarily better than the other, but there are advantages and disadvantages, depending on the problem you are trying to solve.

I think the best approach would be to include both high-level and low-level facilities in a single interface, and let the application designer decide which elements to use based on what the application actually needs. This is what usually happens over time, anyway, so I think that defining this behavior up front is probably a good idea.

Hi,

Rusky wrote:

Brendan wrote:For movies it's just going to be an application telling the video driver "Play this file" (but I haven't thought about or designed the file format for this yet).

I'm not talking about pre-recorded videos, I'm talking about things like Pixar's render farms that are actually generating those videos.

After my video system has spent many many years evolving, gaining features and gaining rendering quality; and after my OS has support for GPGPU (which would be very beneficial for things that involve massive amounts of processing that have very little to do with the OS's real-time "fixed at 60 frames per second with variable quality" video system); I can't see why my system wouldn't be usable for render farms that generate movies.

Rusky wrote:

Brendan wrote:Games use custom shaders because the graphics API sucks. It's not necessary. It might be "desirable" in some cases, but that doesn't mean it's worth the pain.

It's absolutely necessary. Custom shaders are just as important as custom CPU-side programs. You would call me insane if I proposed only giving access to the CPU through pre-existing code to process pre-existing file formats.

From my perspective; your suggestion is that I:

• Destroy the ability to distribute "video rendering load" across multiple machines;
  despite the fact that distributing load across multiple machines is the primary goal of the project.
• Destroy the ability to support multiple cooperating renders within a single computer (e.g. one renderer using an AMD video card's GPU, one renderer using an Intel on-board GPU and one renderer using CPUs, all working together to generate graphics for a single monitor).
• Destroy the ability to do a real time "fixed frame rate with variable quality" video system; which is something I think is necessary to get acceptable latency from software rendering and therefore necessary for the initial stages of the OS (before native video drivers capable of utilising GPU and/or GPGPU exist); and also something I think is "extremely beneficial" after native video drivers exist.
• Design and implement some form of "shader language", and implement developer tools (IDE, computer, etc) for this, and also implement some sort of "shader emulator" in software so that video can work before native video drivers capable of utilising GPU and/or GPGPU exist; as if my plans aren't ambitious enough already.
• Significantly increase the work necessary for anyone to create a GUI, game, application or widget (it's all 3D).
• Significantly increase the chance of compatibility problems between GUIs, games, applications and widgets (as they all fight to use "shaders" in completely different ways).
• Significantly increase the chance that new software won't work on old systems.
• Significantly increase the chance that old software won't be any better when running on newer/more capable systems.
• Significantly increase the difficulty of supporting multiple monitors (where the same scene is displayed across all monitors).
• Significantly increase the difficulty of recovering gracefully when applications crash (because, who knows what they were doing via. the low level graphics interface at the time or what state the video card might have been left in). Note: This is something that Microsoft has failed to do for about 20 years now.
• Significantly increase the difficulty of true multi-tasking (e.g. having two 3D games running in windows at the same time). Note: This is also something that Microsoft has failed to do for about 20 years now.
• Completely destroy all advantages that make my OS better than existing OSs and able to compete with existing OSs for the video system.

In return to making many massive sacrifices that destroys almost everything I'm trying to achieve, this will:

• Make it possible for irrelevant retards to generate graphics that completely ignores the medium and uses a different (e.g. "drawn with crayons") artistic style for an insignificant number of games.

Rusky wrote:

Brendan wrote:There's never a need to synchronise user input and physics with the frame rate.

There is absolutely a need to synchronize user input and physics with the frame rate. There needs to be a consistent lag time between input and its processing, physics needs to happen at a consistent rate to get consistent results, and rendering needs to happen at a consistent rate to avoid jitter/tearing. Of course these rates don't have to be the same, but they do have to be synchronized. Many games already separate these rates and process input as fast as is reasonable, video at the monitor refresh rate, and physics at a lower, fixed rate for simulation consistency.

There does not need to be a consistent lag time between input and its processing, and this can be "as soon as possible" and not "always slow enough to cover worst case to guarantee constant lag" (and by getting user input once per frame the lag time between input and its processing varies depending on how long until the main loop gets back to checking for user input again, and is therefore not constant).

Things like physics do need to be running at a constant rate and this needs to be synchronised with "wall clock time"; but the granularity of the time keeping can be extremely precise (e.g. nanosecond precision) and should not be crippled (e.g. 1/60th of a second precision). The video system (e.g. renderer, etc) needs to be synchronised with the monitor's vertical sync; but it just uses data from the game to determine what is where at any point in time (as I've described previously) and needn't be synchronised with the game in any other way.

Rusky wrote:

Brendan wrote:(and are the reason why gamers want "500 frames per second" just to get user input polled more frequently; and the reason why most games fail to use multiple threads effectively).

These gamers are wrong. Correctly-written games take into account all the input received since the last frame whether they're running at 15, 20, 30, or 60 frames per second. Doing it any faster than the video framerate is literally impossible to perceive and anyone who claims to do so is experiencing the placebo effect.

I wasn't talking about correctly written games. I was talking about incorrectly written games that have a ""do { get_user_input(); update_AI(); do_physics(); update_screen(); }" main loop.

The difference is extremely easy to perceive in some cases (and not easy to perceive in others). For example; imagine you're playing a game where you've got a rifle and have to shoot a flying duck; the duck is 20 pixels wide on the screen and is flying across the screen from left to right at a speed of 100 pixels per frame (due to "persistence of vision" it looks like the duck is moving but it's actually teleporting from one place on the screen to the next 60 times per second). To hit the duck you need to pull the trigger at exactly the right time - 1/60th of a second too early or 1/60th of a second too late and you miss. If user input is checked once per frame, then it becomes impossible to pull the trigger at the right time.

Rusky wrote:

Brendan wrote:By shifting the renderer into the video driver the renderer gets far more control over what is loaded into VRAM than ever before.

The video driver doesn't have the correct information to utilize that control. It doesn't know the optimal format for the data nor which data is likely to be used next; a renderer that's part of a game does.

Um, what? The video driver knows exactly what needs to be drawn, knows exactly what is in VRAM and what isn't, and knows all the details of the hardware; but "lacks" some missing piece of information that you've failed to mention (like, what the objects its displaying smell like)?

When it's time to render a scene; the format the data is in is whatever format the video driver felt like converting the data into after it loaded the data from files. The data in files (textures, meshes, whatever) may or may not be optimal, but "file format standardisation" is far more important.

Yes; for some things the application may want to provide the data itself (rather than telling the video driver which file contains the data); but that changes nothing - the video driver still converts the data into whatever format it feels like when the data is uploaded to the video driver (and not when a rendering is being done), and "message protocol standardisation" is far more important.

Rusky wrote:

Brendan wrote:Games like (e.g.) Minecraft only need to tell the video driver when a block is placed or removed. Not only is this far simpler for the game but allows the video driver to optimise in ways "generic code for wildly different video hardware" can never hope to do.
...
when games specify what they want (via. a description of their scene) instead of how they want things done, there's no way to scale games up to new hardware without the games specifying what they want??
...
The game just tells the video driver the volume of the liquid, its colour/transparency and how reflective the surface is. Things like "rippling" will be done elsewhere via. control points (part of physics, not rendering).

So you're going to include support specifically for voxel-based games in your API?

For raw data formats; as far as I can tell at this stage; I will want:

• Textures with just "XYZ reflected" for each texel
• Textures with "XYZ reflected" and "XYZ passing through" for each texel
• Textures with "XYZ reflected", and either "surface normal" or "bump height" for each texel
• Textures with "XYZ reflected" and "XYZ passing through"; and either "surface normal" or "bump height" for each texel
• Simple/static meshes with vertices and polygons, where polygons have either "fixed colour", "shaded/interpolated colour for each vertex" or "texture and (u, v) within texture for each vertex)
• Complex/deformable meshes which have control points ("skeleton") and vertices that are effected by those control points (with the same stuff for polygons, etc as simple meshes)
• Cuboids with a 3D grid of "XYZ reflected" for each voxel
• Cuboids with a 3D grid of "object reference" for each coord
• Something to describe a light source (XYZ amplitudes emitted, direction of light emitted, radius of light source)
• Static collections of objects with a list of object references and their location relative to the origin of the collection and their "front" and "top" vectors; that may also include light sources. Note: this is mostly a container object - e.g. so you can have a "car" object that includes "car body", "wheel", "seat" and "engine" sub-objects.

I'm sure there's something I've overlooked, but I'm also sure I'll have a much better idea when I actually start designing the video system.

Rusky wrote:Your standardized scene description format cannot possibly anticipate all the types of information games will want to specify in the future, nor can it possibly provide an optimal format for all the different things games care about specifying today.

Nobody can possibly anticipate all possible things that might or might not be needed in the future when designing anything for any purpose. The only thing that's reasonable to expect is to combine research and foresight in the hope of finding the best solution at the time.

Rusky wrote:Take my water example and explain how that will work- will you design a "Concise Way to Describe Bodies of Water" in all possible situations so that games can scale from "transparent textured plane" to "ripples and reflections using a shader" that takes into account all the possible ways games might want to do that, especially when there's not enough bandwidth to do ripples as a physics process so it must be done in the shader? How will this solution work when you have to specify a "Concise Way to Describe Piles of Dirt" and a "Concise Way to Describe Clouds" and a "Concise Way to Describe Plants" and a "Concise Way to Describe Alien Creatures" and a "Concise Way to Describe Stars" and a "Concise Way to Describe Cars" and a "Concise Way to Describe Spaceships" so that detail can automatically be added when new hardware is available?

No, I won't design "a concise way to describe bodies of water" because that's not general enough. Instead, I will design "a concise way to describe volumes with uniform characteristics" that will be used for things that are solid (e.g. bar of metal) or opaque (e.g. orange juice) or emit light (e.g. lava) or have a reflective surface (e.g. water) or do not (e.g. fog). In fact I will have 2 - one using "simple meshes" and one using "control points". These volumes will act as objects. For example, you might have a huge cuboid using a simple mesh and a uniform "water like material" for an entire ocean, and you might have a small "one meter square and 200 mm high" volume/object using a control points and the same uniform "water like material", and you might duplicate that small volume/object (with the control points) a thousand times over the top of the ocean, so that with a grid of 10*10 control points across the top of that small volume/object you can create waves across a massive ocean by only changing 100 control points once per second.

I will not design "a concise way to describe piles of dirt" (or clouds, or plants, or aliens, or stars, or cars, or whatever else) because it's completely unnecessary as all of these things can be described in a far more general way.

Rusky wrote:On the scaling-up side, we differ in our goals. I care about artistic integrity, and think games and movies should look the way their designers intended them, and that they should have the freedom to control what that look is, whether it's purposefully-blocky, shaded and outlined like a cartoon, or whatever more-subtle variation on reality they care to come up with. Thus, the only thing old games should do on new hardware is run faster and with higher resolution.

You're right, we do have different goals. I don't care about "artistic integrity"; I care about things like complexity and compatibility, and the ability to make full use of all hardware available (in an entire group of computers, even for hardware that didn't exist when software was written) to get the best possible result for the end user (and not "OMG my eyes are bleeding" retro stuff).

Rusky wrote:You don't care about any of that and think the video driver should redesign every game's look on every hardware update for the sake of "realism." I think this should only apply to applications that want it, like CAD tools or scientific imaging tools, that can use a specific library for that purpose, which will only need to be updated once when a new generation of hardware comes out, not again for every driver.

I think you may be getting confused here.

This is a realistic rendering of a realistic scene:


And this is a realistic rendering of an unrealistic scene:


Both of these use realistic rendering; and both of these are examples of what I want.

However...

This is an unrealistic rendering of an unrealistic scene (which I don't want, but I can't actually prevent a game from generating a 2D texture and having a realistically rendered scene with that single texture, ambient lighting and nothing else):


Note: I tried to find an example for "unrealistic rendering of realistic scene" and failed - the closest that I did find was only "extremely bad realistic rendering of realistic scene".

If you use google images to search for "computer aided design screenshot" you'll see a lot of pictures. Some of these pictures use realistic rendering anyway, some use wire frame, and some use "2D lines on 2D plane". For wire frame, you can just connect "source vertexes" with thin black tubes and it'd be mostly the same (a little less, but not enough to care about). For "2D lines on 2D plane" software can just create a texture and say where it wants that texture. There were also some pictures showing realistic rendering on top of a flat surface, where that flat surface was a "2D lines on 2D plane" diagram. Finally there was also a few (much less common) pictures where it had wire frame super-imposed on top of a realistic rendering; and for these you'd need to use the renderer recursively (e.g. generate a texture from "vertices connected by tubes with transparent background", then do "realistic rendering, with previous texture in foreground"). Neither of these cases are a significant problem (a bit less efficient in some cases, but not enough to care about).

Rusky wrote:

Brendan wrote:Often impostors aren't "good enough" and do need to be redrawn, but it's far less often than "every object redrawn every frame".

Please note that this is not something I invented. "Static impostors" have been used in games for a very long time, and (some) game developers are doing auto-generated/dynamic impostors with OpenGL/DirectX.

Imposters are a trick to improve performance when necessary, not the primary way things are drawn in games today. Parallelizing a game across a network by converting everything to imposters is not an improvement, it's throwing everything in the toilet for the sake of forcing your idea to work.

The point is, in the worst case (which happens a lot) the camera will be moving and rotating and changing the perspective on everything visible, so you need to be prepared to handle this case even if it doesn't happen all the time. And if your render state is strewn across the network, you have absolutely no hope of handling this and games will just have imposter artifacts and lowered quality all the time.

I am prepared to handle that case. That case is the sort of thing that my "fixed frame rate, variable quality" approach is designed for - if things are changing so rapidly that the video system can't do everything as "high quality" in enough time; then quality is reduced (lower resolution rendering that gets upscaled, less lights taken into account, "solid colour" used instead of textures, "far clipping plane" much closer to the camera, impostors that are recycled when they're "less good enough", etc). That absolute worst case is that the video driver isn't able to draw anything at all in the time available before a frame is shown and the scene changes so much between frames that this happens for many frames; where the video driver reduces quality all the way down to "user sees nothing more than a screen full of grey". Of course this absolute worst case is something that I hope nobody will ever see in practice.


Cheers,

Brendan

I understand the goal of your project is to make parallelism across devices and networks transparent to applications. I just don't think baking your high level API into the OS is the most efficient way to do that, nor do I think transparent parallelism is the most effective way to take advantage of a disparate collection of hardware.

There's a threshold for when video rendering can be effectively shared between devices and machines. Games fall on the side that makes it a bad idea, but things like render farms make it a good idea. There are also only certain times that "fixed frame rate with variable quality" makes sense- it makes sense for games (and is done today, albeit under user control rather than automatically), but is disastrous for render farms.

The most enabling OS-level API is one that exposes the hardware's capabilities as directly as possible, while still providing cross-hardware compatibility. Successively higher-level APIs should follow the same principle- do the minimum to effectively perform their responsibility, and allow other APIs to provide different trade-offs. I would be very interested to see a higher-level API that does distributed rendering, and what situations it could be made to work in.

This way, applications still get the benefits of a high level API, but they also have the flexibility to use a different API at the same or a different level of abstraction. APIs get better at performance and cross-hardware consistency, because the code that needs multiple implementations is minimized, so there is less surface area for bugs and more shared optimization.

I understand you have a goal of making applications scale well between old and new systems, but this is another goal I simply don't think is a good idea. Trying to improve old applications by changing what APIs do underneath them is a huge burden on backwards compatibility, and I'm not sure even a perfect implementation would be worth the inconsistency for application developers or be able to make a meaningful difference. All the research and foresight in the world can't magically add more information to meshes for things like small surface details, fancier ways of rendering and simulating hair, or complex light interactions for things like skin. All you can do in the present is enable future developers to come up with their own solutions.

I'm also done arguing with you about game engines, which you clearly do not understand. You repeatedly make incorrect assertions and misleading examples about timing. You ignore the ways current rendering engines achieve their performance through domain-specific data format optimization, instead focusing on standard formats, which are much more important for data interchange than for game-specific data that no other application will ever use. You proclaim that generic, best-effort, realism-focused rendering is good enough for everyone, ignoring games that use artistic techniques like cel shading and movies that want precise control over how everything looks.

For example, here is an "unrealistic rendering of an unrealistic scene" that falls under "artistic integrity": http://images.nintendolife.com/screensh ... /large.jpg

Hi,

Rusky wrote:I understand the goal of your project is to make parallelism across devices and networks transparent to applications. I just don't think baking your high level API into the OS is the most efficient way to do that, nor do I think transparent parallelism is the most effective way to take advantage of a disparate collection of hardware.

There's a threshold for when video rendering can be effectively shared between devices and machines. Games fall on the side that makes it a bad idea, but things like render farms make it a good idea. There are also only certain times that "fixed frame rate with variable quality" makes sense- it makes sense for games (and is done today, albeit under user control rather than automatically), but is disastrous for render farms.

The threshold for when video rendering can be effectively shared depends on the network latency of sending the job, the processing time the job takes, and the networking latency of getting the results back. By "stretching" the time available (the "not updated every frame" impostors, using prediction to start things before the last frame is even finished, etc) the real limit is neither latency nor "processing time until deadline"; but becomes available network bandwidth and the amount of remote processing time available.

So, how much data can you shove down a network cable? For older 100 MHz ethernet it's about 12 MiB per second. That's enough to keep a remote "slave renderer" informed about changes in the scene with plenty of bandwidth spare. The problem is getting results back; which depends on what format the results are returned in and how much data is being returned. Raw pixel data for a full 1920*1200 screen is very large/expensive; but doing transformations and lighting remotely for a few objects and sending back "line segments" (where textures are applied locally) is much less expensive.

So; how much can be done remotely? For 100 MHz ethernet and "not too complicated" scene, maybe everything except deciding what should/can be redrawn, a few "extremely important to update ASAP" impostors, applying textures, and doing the whole "HDR XYZ to monitor colour space" thing. For slower networking it'll be less, for gigabit networking it'd be more.

Of course it's impossible to know exactly because it depends on far too many things, including the exact scene (and previous scenes due to the impostor cache), the networking equipment, unrelated networking load, etc. However; it's not about whether its possible to do anything remotely; and it's only about how much can be done remotely.

Don't forget that the opposite is also true - local processing also has limits. Modern Intel CPUs are impressive, but they're not capable of doing "all objects redrawn every frame" high quality real time rendering (complete with dynamic lighting/shadows, etc) at 1920*1200; and if you think GPUs are going to help maybe one day hopefully in the next decade if someone writes native drivers, then you need to have a serious think about (e.g.) a laptop driving a pair of 4K resolution displays.

Note: if you use google to search for "distributed real time renderer" you'll find white-papers/research and youtube videos showing that I'm not the only one that thinks it's a desirable feature, and also showing that it is not only possible but has already been done.

Rusky wrote:The most enabling OS-level API is one that exposes the hardware's capabilities as directly as possible, while still providing cross-hardware compatibility. Successively higher-level APIs should follow the same principle- do the minimum to effectively perform their responsibility, and allow other APIs to provide different trade-offs. I would be very interested to see a higher-level API that does distributed rendering, and what situations it could be made to work in.

The most crippling OS-level API is one that fails to abstract hardware details and pushes the burden of dealing with it onto a "driver driver" (e.g. a rendering engine) that provides a less retarded API that's actually usable. When an application opens a file, do you think the application developer wants to support a bunch of different storage devices, RAID schemes, file systems, etc; just because some moron failed to abstract those details? Why do you think video is any different?

Rusky wrote:This way, applications still get the benefits of a high level API, but they also have the flexibility to use a different API at the same or a different level of abstraction. APIs get better at performance and cross-hardware consistency, because the code that needs multiple implementations is minimized, so there is less surface area for bugs and more shared optimization.

Why stop there? Why not have "raw IO ports" for a device driver interface, then have an "extremely low level" layer of puss on top of that, then have an "slightly less low level" puddle of vomit on top of that, then... We could have a full 10 layers of insanity for every single device. It'll be awesome because someone writing a trivial "Hello world" application could spend 30 years dealing with mind boggling stupidity just to squeeze half a fairy fart worth of extra performance out of it (for the 5 minutes it'll work before any one of those interfaces changes and breaks the app). Yay!

Rusky wrote:I understand you have a goal of making applications scale well between old and new systems, but this is another goal I simply don't think is a good idea. Trying to improve old applications by changing what APIs do underneath them is a huge burden on backwards compatibility, and I'm not sure even a perfect implementation would be worth the inconsistency for application developers or be able to make a meaningful difference. All the research and foresight in the world can't magically add more information to meshes for things like small surface details, fancier ways of rendering and simulating hair, or complex light interactions for things like skin. All you can do in the present is enable future developers to come up with their own solutions.

You're right. You want to ensure the interface doesn't change every 12 months (like DirectX does), and to do that you want to ensure that there can be significant changes in video cards and their video driver that don't effect applications, and to do that you need a high level interface instead of a leaky abstraction.

Rusky wrote:I'm also done arguing with you about game engines, which you clearly do not understand. You repeatedly make incorrect assertions and misleading examples about timing. You ignore the ways current rendering engines achieve their performance through domain-specific data format optimization, instead focusing on standard formats, which are much more important for data interchange than for game-specific data that no other application will ever use. You proclaim that generic, best-effort, realism-focused rendering is good enough for everyone, ignoring games that use artistic techniques like cel shading and movies that want precise control over how everything looks.

You didn't claim any of my examples involving timing were misleading in the numerous previous posts where it would've been relevant rather than petulant whining. I don't ignore the ways current rendering engines achieve their performance or all of the massive disadvantages (that you're ignoring) and want to avoid those massive disadvantages. The entire industry is built on standards (including both DirectX and OpenGL) so complaining about "focusing on standards" is completely retarded. I never claimed that my renderer will be good enough for "everything imaginable" and only claim that it's superior for my project and the things I care about (which does not include "artistic techniques like cel shading" and never will).


Cheers,

Brendan

Brendan wrote:Software (e.g. GUI, applications, games) control how much light objects reflect, the position of the camera, the amount of ambient light, the number and sizes of directional lights; the video driver controls how "brighter than monitor can handle" gets mapped to the intensities that the monitor can handle and (if possible) the monitor's back light. There is nothing else that software can control, and nothing I could possibly do to make you happier.

Here I see two responsibilities - the application and driver ones. The application is responsible for the scene creation and the driver is responsible for the scaling of brightness. But it seems to me your words in previous messages were about much bigger area of driver responsibility. So, I just unable to see your true point.

Brendan wrote:If you've got a display that's actually capable of showing "brighter than the sun" images, then my video system won't need to use its auto-iris for that display. If you don't have a display that's capable of showing "brighter than the sun" images, then don't blame me if "auto-iris" only does the best job possible.

The issue of scaling the input brightness range to some form that is acceptable by a hardware is not opposed by me. Yes, the scaling should be done. But in configurable and extensible manner.

Brendan wrote:Can you provide an example of "applications requirements" that couldn't be done on my video system but actually matter enough to care about?

Provided I have no complete specification of your system it's a bit tricky to infer what it can't manage. And also I just do not know what actually matters enough for you. But I see a lot of algorithms are used in the rendering engines and my impression is your engine just won't be able to incorporate all of them. At least because there are many new algorithms emerging every year and it's impossible to cover them all. And, as it was already said, beside the rendering algorithms there are a lot of other algorithms, that are also very important, like scene descriptions, file formats and even additional hardware supported effects, that also require some support from your very general driver.

Brendan wrote:I'm mostly going to provide:

• A device independent software renderer implemented as a service that uses CPUs (for both real-time rendering and offline rendering). This will also be used as a reference implementation.
• A "video driver template" that contains common code that any video driver would need; that just uses the software renderer service for rendering. The idea is that someone writing a native video driver for a specific video card would start with this, and add/replace pieces with code for the specific video card (rather than starting with nothing). This will also be the "raw framebuffer" generic video driver that's used by the OS.

So, there should be a renderer and a driver. Because the renderer is defined as a service I suppose it should have some interfaces exposed to all it's users. One of such users is supposed to be a driver. But the purpose of a driver is very different in comparison with the purpose of an application, that uses the renderer service (or isn't it?). So, the renderer service should have two very different interfaces (for driver and for applications). Usually, such fork means there is a design flaw in the system. But may be you have some arguments for such design.

SpyderTL

Nice "midi" video

SpyderTL wrote:I think the best approach would be to include both high-level and low-level facilities in a single interface, and let the application designer decide which elements to use based on what the application actually needs. This is what usually happens over time, anyway, so I think that defining this behavior up front is probably a good idea.

Exposing the low level interfaces is dangerous. It's a security threat and the great compatibility issue. The very well known case of the greatest ever seen compatibility problem is the Wintel's creature. Today, it's becoming a new mainstream to create some abstraction layer on top of hardware, like it was done for the Android OS with it's Java development environment, for Windows with it's CLR and .Net platform. Also there is a trend of the simplification of the hardware design, so it is now becomes more easy to implement a different instruction set, a different communication protocol and so on. Together, these trends will produce some merged system with interdependent soft and hard components. And the idea of the "low level" is changed significantly all the time. Today it's not very hard to design your own processor and to run a custom code on it. So, what is low level here?

The main level, as it seems to me, is the level of an idea. All the other stuff is about simple steps of implementation of the idea. The number of steps can be great, but the complexity of every step is low (it's easy to create your processor, your instruction set, your bytecode, your language, your OS, your whatever, even if it takes some time). The Brendan's idea is about efficient use of Brendan's hardware in a manner that Brendan sees viable. The compatibility issue is seemed by Brendan as something more unpleasant than the effect of disallowing to others to touch hardware under Brendan's OS. Also, the idea of a "friendly" OS doesn't allow Brendan to let somebody to behave in uncontrolled manner under his OS.

So, the discussion is actually about the uncontrolled behavior other developers expect from Brendan's OS. But it is Brendan who finally will decide on this issue. For all others this discussion can be useful as an illustration of how important the idea can be. Also, there are some technical details and algorithms available along the way.

Brendan wrote:

Rusky wrote:

Brendan wrote:By shifting the renderer into the video driver the renderer gets far more control over what is loaded into VRAM than ever before.

The video driver doesn't have the correct information to utilize that control. It doesn't know the optimal format for the data nor which data is likely to be used next; a renderer that's part of a game does.

Um, what? The video driver knows exactly what needs to be drawn, knows exactly what is in VRAM and what isn't, and knows all the details of the hardware; but "lacks" some missing piece of information that you've failed to mention (like, what the objects its displaying smell like)?

When it's time to render a scene; the format the data is in is whatever format the video driver felt like converting the data into after it loaded the data from files. The data in files (textures, meshes, whatever) may or may not be optimal, but "file format standardisation" is far more important.

Yes; for some things the application may want to provide the data itself (rather than telling the video driver which file contains the data); but that changes nothing - the video driver still converts the data into whatever format it feels like when the data is uploaded to the video driver (and not when a rendering is being done), and "message protocol standardisation" is far more important.

How does your driver know whether a texture is a suitable target for texture compression using one of DXT1, DXT3, DXT5, ETC1, ETC2, PVRTC or ASTC? Note: Each of these texture formats is lossy and therefore has it's own tradeoffs and some of them will make certain things look terrible. On the other hand, if you don't use them, you'll throw away a bunch of performance (yes, reading from compressed textures is faster than uncompressed) and at a minimum quadruple your memory bandwidth and the size of every texture (from 8bpp to 32bpp).

So, if you're abstracting anything and your driver really has no idea (because lets face it, your driver doesn't contain a human eye which can accurately identify compression aberrations), lets give the hypothetical future situation that your OS is running on my mobile phone. You just quadrupled the size of every texture and massively dropped the texture cache residency. Also, because of this, the memory bus is running at 100% utilization and burning 1W, or 1/3rd of SOC power (yes, that is really how much power the memory interface on a SOC will burn), so now the CPU and GPU are required to down clock due to TDP limitations.

Is that a desirable situation?

Is your driver capable of figuring out the correct order to draw things for a given scene? (note: there is no optimum for every scene. If I'm drawing a race track, I want to draw sequential track segments in closest-to-furthest track order for maximum occlusion. If I'm drawing a corridor shooter, then I want to use the visibility information I pre-baked at map compile time)

Does your driver have any clue whether it can pull tricks like screen-space ambient occlusion, which have no basis in reality but subjectively work? Can it pull tricks like screen-space reflection where suitable to mostly simulate reflections without having to redraw the scene twice?

Brendan wrote:You're right, we do have different goals. I don't care about "artistic integrity"; I care about things like complexity and compatibility, and the ability to make full use of all hardware available (in an entire group of computers, even for hardware that didn't exist when software was written) to get the best possible result for the end user (and not "OMG my eyes are bleeding" retro stuff).

Rusky wrote:You don't care about any of that and think the video driver should redesign every game's look on every hardware update for the sake of "realism." I think this should only apply to applications that want it, like CAD tools or scientific imaging tools, that can use a specific library for that purpose, which will only need to be updated once when a new generation of hardware comes out, not again for every driver.

I think you may be getting confused here.

This is a realistic rendering of a realistic scene:


And this is a realistic rendering of an unrealistic scene:


Both of these use realistic rendering; and both of these are examples of what I want.

However...

This is an unrealistic rendering of an unrealistic scene (which I don't want, but I can't actually prevent a game from generating a 2D texture and having a realistically rendered scene with that single texture, ambient lighting and nothing else):


Note: I tried to find an example for "unrealistic rendering of realistic scene" and failed - the closest that I did find was only "extremely bad realistic rendering of realistic scene".

Unrealistic rendering of realistic scenes:

Hi,

embryo2 wrote:

Brendan wrote:Software (e.g. GUI, applications, games) control how much light objects reflect, the position of the camera, the amount of ambient light, the number and sizes of directional lights; the video driver controls how "brighter than monitor can handle" gets mapped to the intensities that the monitor can handle and (if possible) the monitor's back light. There is nothing else that software can control, and nothing I could possibly do to make you happier.

Here I see two responsibilities - the application and driver ones. The application is responsible for the scene creation and the driver is responsible for the scaling of brightness. But it seems to me your words in previous messages were about much bigger area of driver responsibility. So, I just unable to see your true point.

The point is "separation of concerns". Software (e.g. GUI, applications, games) is responsible for deciding what to display (and has nothing to do with the separate concern of how its displayed); and the video driver is responsible for how it's displayed (and has nothing to do with the separate concern of deciding what to display). The number of lights, their intensities and type is part of "what to display" (and is therefore determined by GUI/applications/games); and the figuring out how to display a wide range of intensities on display hardware that only supports a much more limited range of intensities is part of "how to display" (video driver's responsibility).

However, deciding what to display is not the sole responsibility of one piece alone (e.g. a GUI alone, or an application alone). They need to cooperate. For inter-operability (so that you can have many applications that all work on any GUI) some sort of guidelines are needed to determine which piece is responsible for what. The cleanest way to split things up is to say that applications control one or more "window surfaces", and the GUI controls the position of the windows within a larger 3D space and the position (and intensity and type) of lighting that is applied to those "window surfaces". Of course (in my case) an application would actually control "3D window volumes" and not "2D window surfaces" (it's just easier to imagine it as 2D surfaces because that's what people are used to). An application that (e.g.) includes it's own lighting breaks the guidelines, and an application that (e.g.) tries to inform the video driver how to do its job breaks "separation of concerns".

embryo2 wrote:

Brendan wrote:Can you provide an example of "applications requirements" that couldn't be done on my video system but actually matter enough to care about?

Provided I have no complete specification of your system it's a bit tricky to infer what it can't manage. And also I just do not know what actually matters enough for you. But I see a lot of algorithms are used in the rendering engines and my impression is your engine just won't be able to incorporate all of them. At least because there are many new algorithms emerging every year and it's impossible to cover them all. And, as it was already said, beside the rendering algorithms there are a lot of other algorithms, that are also very important, like scene descriptions, file formats and even additional hardware supported effects, that also require some support from your very general driver.

I don't have complete/detailed specifications of the system either (and I do think it's going to take a few prototypes before I'm satisfied with it enough to finalise the specifications; which always happens when you're doing something that few people have attempted). Mostly at this stage I've got a reasonable idea of the overall design but not the specifics.

For what I want; I need something that can be spread across multiple GPUs and CPUs in multiple computers; which isn't a massive burden for simple applications (calculator, text editor, etc) to use (despite the fact that even simple applications will be 3D) but is also able to support the majority (but not necessarily "all") gaming, and also able to support all display devices that currently exist (from current/common "2D flat monitor" to stereoscopic screens to VR helmets) and likely to cope with future display devices and handle "multiple different display devices at once".

It is possible to cover the majority of (e.g.) new rendering algorithms because the interface is high level. To understand this imagine it's a "calculator service" where applications send a request like "calculate 5*4". In this case it doesn't matter much how the "calculator service" represents number internally, and doesn't matter much if it does "result = 5*4" or "result = 5+5+5+5" or "temp = 5+5; result = temp + temp;" because these details are hidden behind the high level abstraction and don't effect the application using the service. The algorithms that the "calculator service" uses can be completely changed to something radically different and nobody will know or care.

Maybe one day I'll get native video drivers that do rendering (instead of video drivers that use a software renderer) and the renderer will have to be completely different to get the most out of the GPU. Maybe one day we will all be using 1024-core 80x86 CPUs and I'll want to use a completely different approach for software rendering on CPUs (e.g. cast a ray for each pixel). It doesn't matter. The renderer, data formats it uses internally and the algorithms it uses can be radically different because it's all hidden behind a high level abstraction and therefore it doesn't make any difference to any GUI, application, game, widget or whatever (as long as the interface/abstraction remains the same).

embryo2 wrote:

Brendan wrote:I'm mostly going to provide:

• A device independent software renderer implemented as a service that uses CPUs (for both real-time rendering and offline rendering). This will also be used as a reference implementation.
• A "video driver template" that contains common code that any video driver would need; that just uses the software renderer service for rendering. The idea is that someone writing a native video driver for a specific video card would start with this, and add/replace pieces with code for the specific video card (rather than starting with nothing). This will also be the "raw framebuffer" generic video driver that's used by the OS.

So, there should be a renderer and a driver. Because the renderer is defined as a service I suppose it should have some interfaces exposed to all it's users. One of such users is supposed to be a driver. But the purpose of a driver is very different in comparison with the purpose of an application, that uses the renderer service (or isn't it?). So, the renderer service should have two very different interfaces (for driver and for applications). Usually, such fork means there is a design flaw in the system. But may be you have some arguments for such design.

A GUI, games and applications only send "description of scene" (although its more complicated than that - e.g. asking for data to be pre-loaded, then setting up an initial scene, then just describing "what changed when" after that). The GUI, games and applications don't use any renderer themselves. Their "description of scene" is sent to the video driver/s, and each video driver is responsible for ensuring it's rendered and displayed on time (where "ensuring its rendered" means that the video driver ensures that one or more "renderer services" does the rendering; where a native video driver with GPU support may provide its own built-in "renderer service" for its own use and for other drivers to use).

Note: something like a printer driver would also use the same "renderer service/s" (just with a single frame and with an "infinite" deadline). This means that (e.g.) a printer driver might (at least in theory) end up using 3 GPUs and 4 CPUs across the LAN to generate "highest quality possible" XYZ pixel data, and then convert it to CMYK and sent it to the printer.


Cheers,

Brendan

Brendan wrote:When an application opens a file, do you think the application developer wants to support a bunch of different storage devices, RAID schemes, file systems, etc; just because some moron failed to abstract those details? Why do you think video is any different?

I have not ever suggested an API at that low a level. I have talked about APIs at a similar level to the VFS- ones that expose common features of relatively modern graphics hardware - VRAM, command queues, and shaders - in a hardware-agnostic way.

Brendan wrote:You want to ensure the interface doesn't change every 12 months (like DirectX does), and to do that you want to ensure that there can be significant changes in video cards and their video driver that don't effect applications, and to do that you need a high level interface instead of a leaky abstraction.

And somehow, many-years-old games still run. Because DirectX adds new parts of the API for new features, rather than immediately removing old ones. Your high-level API is the leaky one.

Hi,

Owen wrote:

Brendan wrote:When it's time to render a scene; the format the data is in is whatever format the video driver felt like converting the data into after it loaded the data from files. The data in files (textures, meshes, whatever) may or may not be optimal, but "file format standardisation" is far more important.

Yes; for some things the application may want to provide the data itself (rather than telling the video driver which file contains the data); but that changes nothing - the video driver still converts the data into whatever format it feels like when the data is uploaded to the video driver (and not when a rendering is being done), and "message protocol standardisation" is far more important.

How does your driver know whether a texture is a suitable target for texture compression using one of DXT1, DXT3, DXT5, ETC1, ETC2, PVRTC or ASTC? Note: Each of these texture formats is lossy and therefore has it's own tradeoffs and some of them will make certain things look terrible. On the other hand, if you don't use them, you'll throw away a bunch of performance (yes, reading from compressed textures is faster than uncompressed) and at a minimum quadruple your memory bandwidth and the size of every texture (from 8bpp to 32bpp).

Why make one static choice when you're able to make choices dynamically (e.g. depending on the specific video card's capabilities, and whether the texture is being used for something too far away for details to make any difference or for something very close where details are very important, and "usage history" from previous frames, and estimates of how much memory bandwidth you've got to spare, etc)?

Owen wrote:So, if you're abstracting anything and your driver really has no idea (because lets face it, your driver doesn't contain a human eye which can accurately identify compression aberrations), lets give the hypothetical future situation that your OS is running on my mobile phone. You just quadrupled the size of every texture and massively dropped the texture cache residency. Also, because of this, the memory bus is running at 100% utilization and burning 1W, or 1/3rd of SOC power (yes, that is really how much power the memory interface on a SOC will burn), so now the CPU and GPU are required to down clock due to TDP limitations.

If you're running a distributed OS then it's because you've got a network of multiple computers and not a single device (and ironically; about half of the research into "distributed real time rendering" is specifically for mobile phones because they're too under-powered and have to offload some rendering to a remote machine just to get acceptable results). With this in mind I'd be very tempted to just use the mobile phone as a dumb terminal and run its GUI, apps and/or games plus most of its rendering on a server.

Of course if there isn't a suitable computer on the network (e.g. you took the mobile phone with you to lunch and you're out of wifi range) then the video driver will reduce quality to sustain frame rate and you'll probably just be seeing fuzzy grey blobs lurching around the screen for anything that changes often.

Owen wrote:Is that a desirable situation?

It's definitely more desirable than forcing mountains of hassles onto every application developer and forcing compatibility problems on users. Of course this doesn't mean it's a desirable situation, it's just a "least worst" situation.

To put this into perspective; think about a "Hello World" application. For plain text only, a "Hello World" application is about 3 lines of C. For 2D it gets more complicated and it becomes something like (for Windows) the ~80 lines of code shown here. For 3D with OpenGL 1.0 the complexity involved becomes massive (starting with implementing a font engine(!)) and you end up relying on libraries to cope; and for modern OpenGL (with shaders) the complexity involved doesn't get any better. Now imagine what it's going to be like for software developers (especially beginners attempting to learn) when everything is 3D from the smallest little widget all the way up.

Owen wrote:Is your driver capable of figuring out the correct order to draw things for a given scene? (note: there is no optimum for every scene. If I'm drawing a race track, I want to draw sequential track segments in closest-to-furthest track order for maximum occlusion. If I'm drawing a corridor shooter, then I want to use the visibility information I pre-baked at map compile time)

Does your driver have any clue whether it can pull tricks like screen-space ambient occlusion, which have no basis in reality but subjectively work? Can it pull tricks like screen-space reflection where suitable to mostly simulate reflections without having to redraw the scene twice?

Most existing renderers use "draw everything with Z buffer tests". I'm not. I'm using "overlapping triangle depth culling" where nothing is drawn until all the occlusion, lighting, etc is done and there is no Z buffer. Because it's different to existing stuff it has different advantages and different disadvantages; and requires different methods.

The "overlapping triangles" stuff is done in closest-to-furthest order because nothing else makes sense. Lighting is done very differently and screen-space ambient occlusion isn't necessary.

Reflection is an unsolved problem - I haven't found an acceptable method yet (only "fast" methods that are dodgy/fragile/broken and give unacceptable/incorrect results, and "correct" methods that are too expensive unless you're able to throw multiple GPUs at it). My plan here is to use an expensive method that gives correct results, but is effected by the "fixed frame rate, variable quality" rule (where quality is reduced if/when there isn't enough time).

Of course all of this is on the "lower level" side of the video interface's abstraction; and you can have 5 completely different renderers all (internally) using 5 completely different methods, that are all cooperating to generate (different parts of) the same frame. The exact details of any specific renderer are mostly irrelevant for OS design or where GUI/game/applications are concerned. It's not like existing systems where you're intimately tied to a specific renderer with specific techniques; where if you want to change to a radically different renderer (e.g. "cast rays from light sources") you're completely screwed and have to rewrite every game that's ever been written.

Of course (sadly), its far too easy for people who have never invented anything themselves (who only ever repeat ideas that someone else invented) to ignore massive advantages of anything different and only focus on the disadvantages that are far less important.


Cheers,

Brendan

Hi,

Rusky wrote:

Brendan wrote:When an application opens a file, do you think the application developer wants to support a bunch of different storage devices, RAID schemes, file systems, etc; just because some moron failed to abstract those details? Why do you think video is any different?

I have not ever suggested an API at that low a level. I have talked about APIs at a similar level to the VFS- ones that expose common features of relatively modern graphics hardware - VRAM, command queues, and shaders - in a hardware-agnostic way.

When you want to use old hardware (e.g. "fixed function pipeline") the interface becomes broken. When you completely change the way the renderer works (e.g. switch to "cast rays from light sources") the interface becomes broken. When you change the display device (e.g. from "flat 2D screen" to Oculus Rift) the interface becomes broken. When you want to stretch the same scene across 2 or more displays (e.g. with different GPUs from different manufacturers) the interface becomes broken. When you want to take advantage of "N GPUs and N CPU"s to render a frame for one monitor the interface becomes broken.

The only case where it's not deficient is the "single GPU, single monitor, nothing changed significantly since game was written" case. However; even in that case it's "unusably awkward" and developers have to bury it under a usable higher level interface (game engine); it's not able to dynamically adjust for changes in scene complexity and/or load; it's subject to vendor lock-in via. proprietary extensions (e.g. like this); has inherent stability problems (impossible to test everything when games can do anything); etc.

For things like game consoles (where there isn't any variation in hardware or OS, and there isn't multiple screens or multiple GPUs) most (all?) of these disadvantages are irrelevant. However; I'm not writing an OS for game consoles. For the (distributed) OS I'm planning (and the "very varied" 80x86 hardware it's mostly designed for), the disadvantages are completely unacceptable.

Rusky wrote:

Brendan wrote:You want to ensure the interface doesn't change every 12 months (like DirectX does), and to do that you want to ensure that there can be significant changes in video cards and their video driver that don't effect applications, and to do that you need a high level interface instead of a leaky abstraction.

And somehow, many-years-old games still run. Because DirectX adds new parts of the API for new features, rather than immediately removing old ones. Your high-level API is the leaky one.

Let me repeat this again. New games that take advantage of the new features will not work on old systems (without the idiocy of supporting "N different interfaces" in every game for a value of N that increases as time passes); and old games will not take advantage of new features (and will only look the same). These are both problems caused by "too low level, too leaky abstraction". A higher level interface mitigates the majority of these problems; because "how renderer works internally" is decoupled from "how scene is described".


Cheers,

Brendan

So, if I understand correctly, you want old games to improve in quality over time, automatically, and you want new games to play on old PCs, automatically.

If so, that's a pretty tall order.

The only attempt that I'm aware of this idea that ever made it into the real world was VRML, where the 3D scene was defined in a static document, and the "browser" would render the scene as it saw fit.

I also imagine that raytracing software works in the same way, since it's not typically interactive.

Is this roughly what you had in mind? Something like using HTML (or XAML) for your whole user interface, but using VRML instead? The application would just modify the "document" or model, and the OS would take care of making the display match the document/model?

EDIT: This also sounds a lot like Direct3D retained mode, where you built the visual model, and Direct3D handled all of the rendering, live, in a separate thread. (It was deprecated long ago, because no one used it, because they wanted more control over the hardware. Kind of like VRML. )

So, as far as interfaces go, you traditionally have:

VGA
SetPixel(x, y, color)

2D Accelerated Hardware
DrawLine(x1, y1, x2, y2, color)
FillRectangle(x1, y1, x2, y2, color)
CopyRectangle(sourceRectangle, destinationRectangle)

3D Accelerated Hardware
SetTexture(texture)
SetVertexShader(shader)
SetPixelShader(shader)
DrawTriangles(verteces)

Incidentally, this is probably what my interface will look like when it's done.

Compared to this, do you know what your interface is going to look like?

EDIT2: Another comparison would be the difference between a MIDI file, and an MP3 file. Neither file contains raw audio data, but the MP3 file is deterministic, and has exactly one correct output stream. The MIDI file, on the other hand, can be rendered in virtually millions of different ways, depending on the hardware, the software, and the user's preferences.

Over time, the MIDI file may sound better and better, but the MP3 file will always sound the same. Seeing as how MIDI files have been all but forgotten, just like VRML, I'd say that people prefer deterministic systems over non-deterministic systems, so you may be fighting a losing battle. However, I still hope you succeed, just so that the option to go the non-deterministic route is thoroughly explored.

Hi,

SpyderTL wrote:So, if I understand correctly, you want old games to improve in quality over time, automatically, and you want new games to play on old PCs, automatically.

If so, that's a pretty tall order.

The only attempt that I'm aware of this idea that ever made it into the real world was VRML, where the 3D scene was defined in a static document, and the "browser" would render the scene as it saw fit.

I also imagine that raytracing software works in the same way, since it's not typically interactive.

Is this roughly what you had in mind? Something like using HTML (or XAML) for your whole user interface, but using VRML instead? The application would just modify the "document" or model, and the OS would take care of making the display match the document/model?

EDIT: This also sounds a lot like Direct3D retained mode, where you built the visual model, and Direct3D handled all of the rendering, live, in a separate thread. (It was deprecated long ago, because no one used it, because they wanted more control over the hardware. Kind of like VRML. )

It is a "roughly similar" concept to what I have in mind (but the implementation is very different).

SpyderTL wrote:Compared to this, do you know what your interface is going to look like?

It's a "messaging protocol" (sort of like a networking protocol), where a messages consists of a list of one or more commands (and that command's data). Some commands create new things (e.g. light source, texture, mesh, volume, whatever) either directly (using data in the message) or indirectly (using data in a file, where you only send file name); where the data may refer to other things that have been created. Other commands modify existing things (e.g. "at timestamp=1234567, light source #3 has intensity=1234").

SpyderTL wrote:EDIT2: Another comparison would be the difference between a MIDI file, and an MP3 file. Neither file contains raw audio data, but the MP3 file is deterministic, and has exactly one correct output stream. The MIDI file, on the other hand, can be rendered in virtually millions of different ways, depending on the hardware, the software, and the user's preferences.

Over time, the MIDI file may sound better and better, but the MP3 file will always sound the same. Seeing as how MIDI files have been all but forgotten, just like VRML, I'd say that people prefer deterministic systems over non-deterministic systems, so you may be fighting a losing battle. However, I still hope you succeed, just so that the option to go the non-deterministic route is thoroughly explored.

MIDI and MP3 are intended for completely different purposes, and are both still in use for their intended purpose, and are both mostly useless for things they were never intended for. For example, you can't record the sound of a dog barking with MIDI, and can't change an instrument from guitar to flute in an existing MP3.

VRML is for 3D, SVG is for 2D pictures and HTML is for documents. All of these let the renderer do the rendering, and all of them use inefficient "plain text with markup". VRML (and 3D on the web in general) never really took off, but unfortunately SVG and HTML became reasonably popular and now we're stuck with inefficient "plain text with markup".


Cheers,

Brendan

Brendan wrote:It's definitely more desirable than forcing mountains of hassles onto every application developer and forcing compatibility problems on users.
...
you end up relying on libraries to cope

Relying on libraries is a perfectly good way to avoid "forcing mountains of hassles onto every application developer," and libraries in fact allow you to do everything you want without forcing it on applications that don't want it.

Brendan wrote:Most existing renderers use "draw everything with Z buffer tests". I'm not. I'm using "overlapping triangle depth culling" where nothing is drawn until all the occlusion, lighting, etc is done and there is no Z buffer. Because it's different to existing stuff it has different advantages and different disadvantages; and requires different methods.

The "overlapping triangles" stuff is done in closest-to-furthest order because nothing else makes sense. Lighting is done very differently and screen-space ambient occlusion isn't necessary.

Reflection is an unsolved problem - I haven't found an acceptable method yet (only "fast" methods that are dodgy/fragile/broken and give unacceptable/incorrect results, and "correct" methods that are too expensive unless you're able to throw multiple GPUs at it). My plan here is to use an expensive method that gives correct results, but is effected by the "fixed frame rate, variable quality" rule (where quality is reduced if/when there isn't enough time).

More blustering that shows you have no clue how to get good performance and quality out of a renderer.

Brendan wrote:When you want to use old hardware (e.g. "fixed function pipeline")

In other words, never. The most recent fixed-function hardware is at least 10 years old, and even low-end CPUs include powerful programmable graphics hardware. By the time your OS is doing anything with graphics, supporting fixed-function hardware will be as ludicrous an idea as supporting the 8086.

Brendan wrote:New games that take advantage of the new features will not work on old systems and old games will not take advantage of new features.

Let me repeat this again. These are not desirable features, because developers need their applications to behave consistently across hardware so they can test it. Magically upgrading old games is both impossible to do in a way the developers would want and impossible to do without your APIs behavior becoming so inconsistent as to be unusable. Magically downgrading new games is more possible, and in fact is already done to the extent that it is at all reasonable, by letting players adjust quality settings (of course it could be automatic).