Graphics API and GUI

[Brendan]

Hi,

Now you're back to skipping the part where the renderer actually decides what's occluded. You can keep adding new features to the video driver all you want, but the closer it gets to a game engine renderer the less achievable and generally useful it will be. The techniques you describe could all work well for some cases, but they will make things worse in others- and those cases are game-specific, so you're going to end up with an insanely overcomplicated panel of knobs and switches.

How an app/game creates their "dependency graph of nodes" can/will be different for different cases (to do portals or octrees or pre-processed occlusion or whatever); and how a video driver renders that "dependency graph of nodes" is up to the video driver and can be different in different cases (including radically different rendering like ray casting, and possibly even including switching between multiple techniques within a single frame); however..

None of this changes the graphics API, or the format of any data that passes from app/game to video driver for any node. No knobs and switches have been added to the graphics API at all. The graphics API has remained unchanged throughout the entire discussion.


Cheers,

Brendan

[Rusky]

So how does the video driver choose when to do recursive framebuffers, or when to do occlusion queries? These are the kinds of decisions that you claim will be made by the driver, but they are also the exact decisions that need to be made per-game and per-scene. You can of course use heuristics or JIT-style hotspot optimization to raise the quality after a few frames, but that's nowhere near the easiest to implement or the best-looking.

[Brendan]

Hi,

So how does the video driver choose when to do recursive framebuffers, or when to do occlusion queries? These are the kinds of decisions that you claim will be made by the driver, but they are also the exact decisions that need to be made per-game and per-scene. You can of course use heuristics or JIT-style hotspot optimization to raise the quality after a few frames, but that's nowhere near the easiest to implement or the best-looking.

For "recursive framebuffers"; I need more information to determine which part of the (already relatively extensively described) system you're having trouble understanding.

As far as I can tell "occlusion queries" are a solution to a problem that's caused by "graphics API is too low level" (where the application is forced to into doing absurd amounts of micro-managing). Can you think of a significant reason for an application/game to need it for my system (where the application can just include the object in the dependency graph and let the video driver decide if its occluded or not in whatever way the video driver feels like)?

Please note that you are trying to conflate separate issues in an attempt to substantiate false assumptions. The app/game creates the "dependency graph of nodes" and needs per-game decisions to create an efficient graph; but the app/game has the ability to make these decisions without any information from the video driver and the video driver is not involved in these decisions. The video driver renders the "dependency graph of nodes" and needs to make "per node" decisions; but the video driver has the ability to make these decisions without any information from the app/game (beyond the dependency graph itself); and the application/game is not involved in these decisions.


Cheers,

Brendan

[Rusky]

A dependency graph on its own is not enough to render a scene optimally, because it does not make it clear to the renderer which tradeoffs will be worth it- between portals using recursive framebuffers vs portal frustum clipping, occlusion culling using precomputed visibility vs occlusion queries vs antiportals, forward vs deferred lighting, etc. These are all choices to be made by the video driver, even after the application has done its own culling of the dependency graph.

Not only do these tradeoffs depend on the overall structure of the scene, they also depend on the way the scene is used by the game, so the video driver will be reduced to heuristics that will invariably lead to reduced and inconsistent quality.

[Brendan]

Hi,

A dependency graph on its own is not enough to render a scene optimally, because it does not make it clear to the renderer which tradeoffs will be worth it- between portals using recursive framebuffers vs portal frustum clipping, occlusion culling using precomputed visibility vs occlusion queries vs antiportals, forward vs deferred lighting, etc. These are all choices to be made by the video driver, even after the application has done its own culling of the dependency graph.

For recursive framebuffers, portal frustum clipping and occlusion culling using precomputed visibility; I've already explained that these things are NOT choices made by the video driver.

For occlusion queries, I asked if there's any significant reason for my system to care about them and you've failed to provide any reason (significant or otherwise), so I can only assume they're a pointless joke for my system.

Antiportals would be trivial add to my system (e.g. included in a scene by the app/game, and used by the video driver when rendering a scene). In fact, they look so trivial that I don't think I'd need to add support for it at all (e.g. let the app describe the antiportals as plain old "solid colour"/untextured polygons, and let the "intermediate z-buffer occlusion" idea from yesterday use these "antiportal polygons" like any other polygon for occlusion).

Both forward lighting and deferred lighting seem incredibly stupid to me. Instead; I've invented my own method of doing it that I've not seen elsewhere (that works on triangles alone, before rasterisation is done); but I need to implement a prototype and refine the idea. If the outcome of this research isn't favourable I'll just find an alternative.

Not only do these tradeoffs depend on the overall structure of the scene, they also depend on the way the scene is used by the game, so the video driver will be reduced to heuristics that will invariably lead to reduced and inconsistent quality.

As far as I can tell, you're just complaining because it's different. It's easy to imply that (e.g.) it won't be able to do antiportals efficiently when you can't be bothered to say why you think it can't do antiportals efficiently.


Cheers,

Brendan

[Rusky]

Just because your system nominally behaves the same as these techniques does not mean it implements them efficiently, or even at all. Forcing applications to hack them in by editing the dependency graph 1) means you can no longer have a single scene graph, because the application now has to keep its own unculled copy, and 2) is nowhere near as efficient as implementing them correctly, especially for techniques that (should) typically run on the GPU.

I had hoped you would figure this out on your own, but occlusion queries are just an improved, hardware-accelerated version of your "intermediate z-buffer." The point is that the application is the only thing with enough information to decide if it's worth doing as a separate pass. So you can be done fabricating nonsense on that point now.

Forward and deferred lighting also both work on "triangles alone," so please do explain how your magical new lighting system is not just an existing technique with any pre-existing research and improvements stripped out.

[Brendan]

Hi,

Just because your system nominally behaves the same as these techniques does not mean it implements them efficiently, or even at all.

And also doesn't mean the video driver can't optimise them better, or mean that alternative techniques that are different but equivalent or better can't be found. Don't forget that the ideas other people invented (and you only read about afterwards) have been evolving for about 25 years, and I haven't even started research or design properly yet.

Forcing applications to hack them in by editing the dependency graph 1) means you can no longer have a single scene graph, because the application now has to keep its own unculled copy, and 2) is nowhere near as efficient as implementing them correctly, especially for techniques that (should) typically run on the GPU.

1) A single scene graph is useful for a grand total of nothing; as (depending on game) physics and other game logic either needs simpler geometry or something completely different anyway.

2) You've incorrectly assumed they'll be nowhere near as efficient, incorrectly assumed mine won't be implemented correctly, incorrectly assumed that GPUs will still exist by the time it actually matters; and have overlooked the fact that I'm planning distributed rendering where existing approaches are extremely inefficient anyway.

I had hoped you would figure this out on your own, but occlusion queries are just an improved, hardware-accelerated version of your "intermediate z-buffer." The point is that the application is the only thing with enough information to decide if it's worth doing as a separate pass. So you can be done fabricating nonsense on that point now.

Sigh. When the application is responsible for most of the rendering and the video driver/GPU only does what it's told by the application; then the application might need to use occlusion queries. If the video driver does its job itself, the application does not need occlusion queries. In the same way; if I hire a gardener to do all my gardening, I do not need gardening tools myself.

Forward and deferred lighting also both work on "triangles alone," so please do explain how your magical new lighting system is not just an existing technique with any pre-existing research and improvements stripped out.

Wrong. Forward lighting does most of it on fragments; and deferred lighting does it on pixels at the very end of the pipeline (and keeps a lot of information around throughout the pipeline so it can do lighting at the end).

The only thing I could possibly "gain" from describing my lighting technique to you is another 2 pages of you whining because you failed to understand it.


Cheers,

Brendan

[Rusky]

You seem to have the idea that just because someone doesn't personally invent something, it must not be useful to them, which is complete nonsense. After that your post is mostly baseless assertions. However:

Regardless of whether the application or driver is responsible for most of the rendering, occlusion queries (as well as the other techniques we've discussed) are a tool that can drastically improve performance, given that the renderer is set up with enough information. Your scene graph does not give the necessary information to the driver (which has the necessary control over the hardware), and your graphics interface does not give the necessary control to the application (which has the necessary information).

Forward lighting can in fact work exclusively on triangles, and was done that way for a long time. Per-fragment lighting is a later, orthogonal innovation that gets rid of the artifacts from calculating the light only at vertices. The only way, in general, to get rid of these artifacts is to light per-normal, which for curved objects means per-fragment, not per-edge or per-vertex.

[Brendan]

Hi,

You seem to have the idea that just because someone doesn't personally invent something, it must not be useful to them, which is complete nonsense. After that your post is mostly baseless assertions. However:

My problem is not that I don't think existing solutions aren't useful; it's just that often I'm looking for solutions to completely different problems (e.g. getting the performance and/or quality of software rendering good enough so that people who are used to GPU accelerated graphics can at least tolerate it, for a distributed system where the latency between application and video driver can easily be in the "several ms" range).

Note that:

• inventing something new takes a significant amount of effort (researching the problem, finding a potential solution, refining the solution, implementing prototypes, etc)
• implementing an existing/known solution is at least an order of magnitude easier
• critiquing a new invention (e.g. with careful analysis, explanations of the disadvantages, and reasons why the advantages don't justify those disadvantages) is several orders of magnitude easier
• whining about a new invention (without analysis, without explanations and without considering if the advantages justify them) is many orders of magnitude easier.

I do admit that; as someone who typically does invent new approaches; at times I have difficulty respecting the skills and/or opinions of those "many orders of magnitude easier" whiners.

Regardless of whether the application or driver is responsible for most of the rendering, occlusion queries (as well as the other techniques we've discussed) are a tool that can drastically improve performance, given that the renderer is set up with enough information. Your scene graph does not give the necessary information to the driver (which has the necessary control over the hardware), and your graphics interface does not give the necessary control to the application (which has the necessary information).

You've said this multiple times already; but you're still failing to say why. Exactly what is this "necessary information" that the driver would need that the graphics interface does not and can not allow the application to provide; what impact does this have on performance, graphics quality or anything else (are you suggesting it'd cause 2% slower frame rates, or that it makes it impossible to implement, or?); and if you are able to substantiate the perceived disadvantage/s what makes them more significant than all the advantages?

Forward lighting can in fact work exclusively on triangles, and was done that way for a long time. Per-fragment lighting is a later, orthogonal innovation that gets rid of the artifacts from calculating the light only at vertices. The only way, in general, to get rid of these artifacts is to light per-normal, which for curved objects means per-fragment, not per-edge or per-vertex.

Imagine a square room with a point light source in the middle, and 10 animated/moving people dancing around the room. Explain how forward rendering, working exclusively on triangles, does lighting correctly (including the absence of light - the shadows cast by each person onto other people and the walls of the room). As far as I can tell this is impossible and'd you have to do per-fragment or per-pixel lighting.


Cheers,

Brendan

[Rusky]

Exactly what is this "necessary information" that the driver would need that the graphics interface does not and can not allow the application to provide; what impact does this have on performance, graphics quality or anything else (are you suggesting it'd cause 2% slower frame rates, or that it makes it impossible to implement, or?); and if you are able to substantiate the perceived disadvantage/s what makes them more significant than all the advantages?

The "necessary information" has already been described by both me and Ready4Dis. In particular, one example is open, outdoor environments vs closed-in indoor environments. Occlusion culling can be a win for interiors while a loss for exteriors, and the choice of portals, precomputed visibility regions, or occlusion queries depends on exactly how things are laid out.

The application can probably precompute visibility regions with just the dependency graph, but it can't tell the driver whether or not to bother running an occlusion query pass. The driver could try to figure that out using heuristics or by trying it out for a few frames, but like I said, that leads to unacceptable shifts in quality over time as it figures out what it wants to do.

And this is more than just a 2% difference in frame rates. For more demanding scenes, the proper application of these techniques is a make-or-break deal- either the framerate will drop too low or (in your case, if you can even pull it off) the quality will be reduced to untextured, low-poly blobs.

Imagine a square room with a point light source in the middle, and 10 animated/moving people dancing around the room. Explain how forward rendering, working exclusively on triangles, does lighting correctly (including the absence of light - the shadows cast by each person onto other people and the walls of the room). As far as I can tell this is impossible and'd you have to do per-fragment or per-pixel lighting.

Shadow volumes.

[Brendan]

Hi,

Exactly what is this "necessary information" that the driver would need that the graphics interface does not and can not allow the application to provide; what impact does this have on performance, graphics quality or anything else (are you suggesting it'd cause 2% slower frame rates, or that it makes it impossible to implement, or?); and if you are able to substantiate the perceived disadvantage/s what makes them more significant than all the advantages?

The "necessary information" has already been described by both me and Ready4Dis. In particular, one example is open, outdoor environments vs closed-in indoor environments. Occlusion culling can be a win for interiors while a loss for exteriors, and the choice of portals, precomputed visibility regions, or occlusion queries depends on exactly how things are laid out.

Erm. What I'm asking for is more like (e.g.) "To support <feature>, the application has to provide <necessary information>, where that <necessary information> is <description of necessary information>".

All you've done is list the features without describing the data the application provides; and most of these things (all except occlusion queries) have already been discussed and I've already been shown they're easily supported for my system.

Note that as far as I'm concerned occlusion queries are still nothing more than a red herring (completely worthless for my system).

The application can probably precompute visibility regions with just the dependency graph, but it can't tell the driver whether or not to bother running an occlusion query pass. The driver could try to figure that out using heuristics or by trying it out for a few frames, but like I said, that leads to unacceptable shifts in quality over time as it figures out what it wants to do.

Except that for my system, the application has no need to care if an object is actually occluded or not because the application is not responsible for micro-managing the rendering in the first place. The video driver is free to do whatever makes sense for it (which might be conditional rendering if the GPU supports it, or doing the occlusion queries in software for a frame while the GPU is busy doing the previous frame, or using heuristics, or doing ray casting where occlusion queries doesn't make sense in any way at all, or...).

And this is more than just a 2% difference in frame rates. For more demanding scenes, the proper application of these techniques is a make-or-break deal- either the framerate will drop too low or (in your case, if you can even pull it off) the quality will be reduced to untextured, low-poly blobs.

Doing nothing might be make-or-break; but I very much doubt the video driver doing whatever makes sense for it (and not nothing) will make a significant difference at all.

Imagine a square room with a point light source in the middle, and 10 animated/moving people dancing around the room. Explain how forward rendering, working exclusively on triangles, does lighting correctly (including the absence of light - the shadows cast by each person onto other people and the walls of the room). As far as I can tell this is impossible and'd you have to do per-fragment or per-pixel lighting.

Shadow volumes.

Are you serious? You create a shadow volume, use it to create a stencil buffer, and that stencil buffer contains... triangles!?


Cheers,

Brendan

[Rusky]

Okay, we've been over this, but: "To support <not being unacceptably slow or low-quality>, the application has to provide <which culling techniques to use>, where <the usefulness of those techniques> is <dependent on various high-level characteristics of the scene, like whether it is open or maze-like or somewhere in between, or has mostly-static or mostly-dynamic components, or what kinds of motion the camera will be going through>."

most of these things (all except occlusion queries) have already been discussed and I've already been shown they're easily supported for my system.

Except that for my system, the application has no need to care if an object is actually occluded or not because the application is not responsible for micro-managing the rendering in the first place.

So, which is it? Is the application in charge of mangling the dependency graph it hands to the driver- inserting portal framebuffers, culling invisible regions, and otherwise implementing these techniques? Or is the application "not responsible for micro-managing the rendering"?

ray casting where occlusion queries doesn't make sense in any way at all, or...).

Occlusion queries make perfect sense for ray casting. Ray casting still scales with the number of potential objects, and occlusion queries are just one way to cull parts of a scene graph.

Are you serious? You create a shadow volume, use it to create a stencil buffer, and that stencil buffer contains... triangles!?

Perhaps not in the traditional implementation, but that's not the only way to use shadow volumes. Are you telling me your magical triangles-only method doesn't use shadow volumes? Because otherwise it's going to be doing per-fragment stuff.

Besides, you still haven't described how your triangles-only method avoids lighting artifacts on smooth surfaces.

[Brendan]

Hi,

Okay, we've been over this, but: "To support <not being unacceptably slow or low-quality>, the application has to provide <which culling techniques to use>, where <those techniques> are <dependent on various high-level characteristics of the scene, like whether it is open or maze-like or somewhere in between, or has mostly-static or mostly-dynamic components, or what kinds of motion the camera will be going through>.

And as I've already pointed out multiple times (including examples in most cases); an application can provide that information by creating the "dependency graph" a certain way.

most of these things (all except occlusion queries) have already been discussed and I've already been shown they're easily supported for my system.

Except that for my system, the application has no need to care if an object is actually occluded or not because the application is not responsible for micro-managing the rendering in the first place.

So, which is it? Is the application in charge of mangling the dependency graph it hands to the driver- inserting portal framebuffers, culling invisible regions, and otherwise implementing these techniques? Or is the application "not responsible for micro-managing the rendering"?

The application provides the dependency graph in any way it wants (which includes structuring the graph to achieve portals, octrees, pre-processed occlusion, etc); and the video driver renders the dependency graph in any way it wants (which can include doing additional occlusion in any of many ways; and can mean software rendering or GPU, and can include forward rendering or deferred rendering or ray casting or...). Think of it like HTML - the server generates the HTML/CSS however it wants (including describing frames, tables, whatever), and the web browser displays the HTML/CSS however it wants (and the server doesn't constantly pester the web browser for every single part of the DOM).

ray casting where occlusion queries doesn't make sense in any way at all, or...).

Occlusion queries make perfect sense for ray casting. Ray casting still scales with the number of potential objects, and occlusion queries are just one way to cull parts of a scene graph.

Sure; cull all objects that aren't directly visible, that also don't cast visible shadows, that also aren't indirectly seen in any reflection, and also don't cast shadows that are seen in any reflection. The cost of determining that you can't cull anything is going to cripple performance for no gain.

Are you serious? You create a shadow volume, use it to create a stencil buffer, and that stencil buffer contains... triangles!?

Perhaps not in the traditional implementation, but that's not the only way to use shadow volumes.

So now there's a mystical way of using shadow volumes that doesn't involve a stencil buffer, that doesn't seem to be mentioned/described anywhere on the Internet (including in your post). Yay!

Are you telling me your magical triangles-only method doesn't use shadow volumes? Because otherwise it's going to be doing per-fragment stuff.

It doesn't use shadow volumes, doesn't use stencil buffers and doesn't use z buffers.

Besides, you still haven't described how your triangles-only method avoids lighting artifacts on smooth surfaces.

The only cause of artefacts that might effect my lighting system (that I'm aware of); can be solved with smart tessellation (e.g. inserting a new vertex at the point where a specular highlight would be within the triangle, or on the closest edge to a specular highlight if its not within the triangle).


Cheers,

Brendan

[Rusky]

And as I've already pointed out multiple times (including examples in most cases); an application can provide that information by creating the "dependency graph" a certain way.

The application provides the dependency graph in any way it wants (which includes structuring the graph to achieve portals, octrees, pre-processed occlusion, etc);

Gotcha- the application is still in charge of micromanaging the rendering process, just with an interface that's awful at it. Which is exactly what happens when you try to do anything remotely demanding with HTML, until people have to give up and implement that part of their site with Canvas or WebGL.

[Ready4Dis]

I've been following this, maybe I can help shed some light on the disagreement:

Rusky: His implementation isn't anywhere near started, so all the fine details aren't actually completely worked out, just the general concept. He understands that this is a HUGE undertaking an believes that by having the driver writer know the hardware intimately, they can come up with the best methods in most cases depending on the specific hardwares goods/bads. He's trying something new, that may or may not work out any better than existing technologies, but we won't really know until it's done if it is completely acceptable. He may find that he has to implement a very robust game engine similar to what's out there, or he may find out that it's way overkill and only needs a simple scene graph for the driver to figure everything out. Time will tell, and seeing as it's a completely new way of doing things, I can't say for sure it will or will not work, I just know from experience that I had to write specific code paths for specific scene types, but you can find a modern game engine that can do both indoor/outdoor without issues, so maybe he can get it all wrapped up into a neat little package.

Brendan: We are both coming from a background of having worked on graphics engines where we know we can't just do one thing. Whether it's because the API is to low level or the hardware is just to slow and we need to play a bunch of tricks to get the speed we are after, our experience has told us that one size doesn't usually fit all in game programming. I mean, back on my 486 SX I couldn't imagine trying to use floating point math, or render many actual polygons in real time, that's what ray-casting was for . Now things are slightly different, we can throw a million triangles at the GPU and it can have at it. Maybe you have the next big thing, maybe not, but nobody really knows until you have some sort of proof, and based on our experiences thus far, it's hard to imagine a one size fits all approach being elegant.