OSDev.org

Brendan wrote:Hi,

Owen wrote:

Brendan wrote:When it's time to render a scene; the format the data is in is whatever format the video driver felt like converting the data into after it loaded the data from files. The data in files (textures, meshes, whatever) may or may not be optimal, but "file format standardisation" is far more important.

Yes; for some things the application may want to provide the data itself (rather than telling the video driver which file contains the data); but that changes nothing - the video driver still converts the data into whatever format it feels like when the data is uploaded to the video driver (and not when a rendering is being done), and "message protocol standardisation" is far more important.

How does your driver know whether a texture is a suitable target for texture compression using one of DXT1, DXT3, DXT5, ETC1, ETC2, PVRTC or ASTC? Note: Each of these texture formats is lossy and therefore has it's own tradeoffs and some of them will make certain things look terrible. On the other hand, if you don't use them, you'll throw away a bunch of performance (yes, reading from compressed textures is faster than uncompressed) and at a minimum quadruple your memory bandwidth and the size of every texture (from 8bpp to 32bpp).

Why make one static choice when you're able to make choices dynamically (e.g. depending on the specific video card's capabilities, and whether the texture is being used for something too far away for details to make any difference or for something very close where details are very important, and "usage history" from previous frames, and estimates of how much memory bandwidth you've got to spare, etc)?

Brendan wrote:

Owen wrote:So, if you're abstracting anything and your driver really has no idea (because lets face it, your driver doesn't contain a human eye which can accurately identify compression aberrations), lets give the hypothetical future situation that your OS is running on my mobile phone. You just quadrupled the size of every texture and massively dropped the texture cache residency. Also, because of this, the memory bus is running at 100% utilization and burning 1W, or 1/3rd of SOC power (yes, that is really how much power the memory interface on a SOC will burn), so now the CPU and GPU are required to down clock due to TDP limitations.

If you're running a distributed OS then it's because you've got a network of multiple computers and not a single device (and ironically; about half of the research into "distributed real time rendering" is specifically for mobile phones because they're too under-powered and have to offload some rendering to a remote machine just to get acceptable results). With this in mind I'd be very tempted to just use the mobile phone as a dumb terminal and run its GUI, apps and/or games plus most of its rendering on a server.

Of course if there isn't a suitable computer on the network (e.g. you took the mobile phone with you to lunch and you're out of wifi range) then the video driver will reduce quality to sustain frame rate and you'll probably just be seeing fuzzy grey blobs lurching around the screen for anything that changes often.

Brendan wrote:

Owen wrote:Is that a desirable situation?

It's definitely more desirable than forcing mountains of hassles onto every application developer and forcing compatibility problems on users. Of course this doesn't mean it's a desirable situation, it's just a "least worst" situation.

To put this into perspective; think about a "Hello World" application. For plain text only, a "Hello World" application is about 3 lines of C. For 2D it gets more complicated and it becomes something like (for Windows) the ~80 lines of code shown here. For 3D with OpenGL 1.0 the complexity involved becomes massive (starting with implementing a font engine(!)) and you end up relying on libraries to cope; and for modern OpenGL (with shaders) the complexity involved doesn't get any better. Now imagine what it's going to be like for software developers (especially beginners attempting to learn) when everything is 3D from the smallest little widget all the way up.

Brendan wrote:

Owen wrote:Is your driver capable of figuring out the correct order to draw things for a given scene? (note: there is no optimum for every scene. If I'm drawing a race track, I want to draw sequential track segments in closest-to-furthest track order for maximum occlusion. If I'm drawing a corridor shooter, then I want to use the visibility information I pre-baked at map compile time)

Does your driver have any clue whether it can pull tricks like screen-space ambient occlusion, which have no basis in reality but subjectively work? Can it pull tricks like screen-space reflection where suitable to mostly simulate reflections without having to redraw the scene twice?

Most existing renderers use "draw everything with Z buffer tests". I'm not. I'm using "overlapping triangle depth culling" where nothing is drawn until all the occlusion, lighting, etc is done and there is no Z buffer. Because it's different to existing stuff it has different advantages and different disadvantages; and requires different methods.

Brendan wrote: The "overlapping triangles" stuff is done in closest-to-furthest order because nothing else makes sense. Lighting is done very differently and screen-space ambient occlusion isn't necessary.

Reflection is an unsolved problem - I haven't found an acceptable method yet (only "fast" methods that are dodgy/fragile/broken and give unacceptable/incorrect results, and "correct" methods that are too expensive unless you're able to throw multiple GPUs at it). My plan here is to use an expensive method that gives correct results, but is effected by the "fixed frame rate, variable quality" rule (where quality is reduced if/when there isn't enough time).

Brendan wrote:Of course all of this is on the "lower level" side of the video interface's abstraction; and you can have 5 completely different renderers all (internally) using 5 completely different methods, that are all cooperating to generate (different parts of) the same frame. The exact details of any specific renderer are mostly irrelevant for OS design or where GUI/game/applications are concerned. It's not like existing systems where you're intimately tied to a specific renderer with specific techniques; where if you want to change to a radically different renderer (e.g. "cast rays from light sources") you're completely screwed and have to rewrite every game that's ever been written.

Brendan wrote:Of course (sadly), its far too easy for people who have never invented anything themselves (who only ever repeat ideas that someone else invented) to ignore massive advantages of anything different and only focus on the disadvantages that are far less important.

you can't record the sound of a dog barking with MIDI, and can't change an instrument from guitar to flute in an existing MP3.

Owen wrote:

Brendan wrote:Why make one static choice when you're able to make choices dynamically (e.g. depending on the specific video card's capabilities, and whether the texture is being used for something too far away for details to make any difference or for something very close where details are very important, and "usage history" from previous frames, and estimates of how much memory bandwidth you've got to spare, etc)?

And how are you making that dynamic choice without even knowing that the texture compresses reasonably? (Especially considering that your average texture takes around a couple of seconds of CPU time to compress, so you can't exactly try each one and then use metrics to pick)

Owen wrote:

Brendan wrote:If you're running a distributed OS then it's because you've got a network of multiple computers and not a single device (and ironically; about half of the research into "distributed real time rendering" is specifically for mobile phones because they're too under-powered and have to offload some rendering to a remote machine just to get acceptable results). With this in mind I'd be very tempted to just use the mobile phone as a dumb terminal and run its GUI, apps and/or games plus most of its rendering on a server.

Of course if there isn't a suitable computer on the network (e.g. you took the mobile phone with you to lunch and you're out of wifi range) then the video driver will reduce quality to sustain frame rate and you'll probably just be seeing fuzzy grey blobs lurching around the screen for anything that changes often.

I'm pretty sure that in any situation where I'd be playing a game on my phone, I'd not be in range of any of my own machines to use as a render server (because if I was, why am I using my phone?), and given that I'm away from my network I'd rather you didn't use my precious few megabytes of mobile data sending video data back and forth (at unacceptably high latency, anyway).

Owen wrote:Of course, my phone is currently capable of much more than lurching gray blobs.

Owen wrote:

Brendan wrote:It's definitely more desirable than forcing mountains of hassles onto every application developer and forcing compatibility problems on users. Of course this doesn't mean it's a desirable situation, it's just a "least worst" situation.

To put this into perspective; think about a "Hello World" application. For plain text only, a "Hello World" application is about 3 lines of C. For 2D it gets more complicated and it becomes something like (for Windows) the ~80 lines of code shown here. For 3D with OpenGL 1.0 the complexity involved becomes massive (starting with implementing a font engine(!)) and you end up relying on libraries to cope; and for modern OpenGL (with shaders) the complexity involved doesn't get any better. Now imagine what it's going to be like for software developers (especially beginners attempting to learn) when everything is 3D from the smallest little widget all the way up.

Of course, everything is 3D from the smallest little widget up on modern platforms. Have you ever looked at how the Android or iOS UIs are implemented? On top of OpenGL (er, probably Metal in the iOS case these days). Does the developer have to know about this? Not really.

Owen wrote:On the other hand, implementing a UI with a generalized 3D engine sounds like hell (and indeed every 3D engine I've ever worked with comes with a specialized UI library or gives you the option of several to plug in).

Owen wrote:

Brendan wrote:

Owen wrote:Is your driver capable of figuring out the correct order to draw things for a given scene? (note: there is no optimum for every scene. If I'm drawing a race track, I want to draw sequential track segments in closest-to-furthest track order for maximum occlusion. If I'm drawing a corridor shooter, then I want to use the visibility information I pre-baked at map compile time)

Does your driver have any clue whether it can pull tricks like screen-space ambient occlusion, which have no basis in reality but subjectively work? Can it pull tricks like screen-space reflection where suitable to mostly simulate reflections without having to redraw the scene twice?

Most existing renderers use "draw everything with Z buffer tests". I'm not. I'm using "overlapping triangle depth culling" where nothing is drawn until all the occlusion, lighting, etc is done and there is no Z buffer. Because it's different to existing stuff it has different advantages and different disadvantages; and requires different methods.

Your methodology sounds not too different from tile based rendering

Owen wrote:

Brendan wrote:Reflection is an unsolved problem - I haven't found an acceptable method yet (only "fast" methods that are dodgy/fragile/broken and give unacceptable/incorrect results, and "correct" methods that are too expensive unless you're able to throw multiple GPUs at it). My plan here is to use an expensive method that gives correct results, but is effected by the "fixed frame rate, variable quality" rule (where quality is reduced if/when there isn't enough time).

You may call tricks like screen space reflection "broken." I'd say from experience that it's subjectively much better than no reflection at all (of course in some situations it would be and you'd not want it - so in those situations of course you'd want to let the artist turn it off)

Owen wrote:

Brendan wrote:Of course (sadly), its far too easy for people who have never invented anything themselves (who only ever repeat ideas that someone else invented) to ignore massive advantages of anything different and only focus on the disadvantages that are far less important.

Of course (sadly), its far too easy for people who have never actually implemented realtime systems themselves (who only ever theorize new ideas and never implement them) to ignore the massive practical advantages of existing systems and only focus on the disadvantages.

Rusky wrote:

Brendan wrote:It's definitely more desirable than forcing mountains of hassles onto every application developer and forcing compatibility problems on users.
...
you end up relying on libraries to cope

Relying on libraries is a perfectly good way to avoid "forcing mountains of hassles onto every application developer," and libraries in fact allow you to do everything you want without forcing it on applications that don't want it.

Rusky wrote:More blustering that shows you have no clue how to get good performance and quality out of a renderer.

Rusky wrote:

Brendan wrote:When you want to use old hardware (e.g. "fixed function pipeline")

In other words, never. The most recent fixed-function hardware is at least 10 years old, and even low-end CPUs include powerful programmable graphics hardware. By the time your OS is doing anything with graphics, supporting fixed-function hardware will be as ludicrous an idea as supporting the 8086.

Rusky wrote:

Brendan wrote:New games that take advantage of the new features will not work on old systems and old games will not take advantage of new features.

Let me repeat this again. These are not desirable features, because developers need their applications to behave consistently across hardware so they can test it. Magically upgrading old games is both impossible to do in a way the developers would want and impossible to do without your APIs behavior becoming so inconsistent as to be unusable. Magically downgrading new games is more possible, and in fact is already done to the extent that it is at all reasonable, by letting players adjust quality settings (of course it could be automatic).

Brendan wrote:It is possible to cover the majority of (e.g.) new rendering algorithms because the interface is high level. To understand this imagine it's a "calculator service" where applications send a request like "calculate 5*4". In this case it doesn't matter much how the "calculator service" represents number internally, and doesn't matter much if it does "result = 5*4" or "result = 5+5+5+5" or "temp = 5+5; result = temp + temp;" because these details are hidden behind the high level abstraction and don't effect the application using the service. The algorithms that the "calculator service" uses can be completely changed to something radically different and nobody will know or care.

Brendan wrote:A GUI, games and applications only send "description of scene" (although its more complicated than that - e.g. asking for data to be pre-loaded, then setting up an initial scene, then just describing "what changed when" after that). The GUI, games and applications don't use any renderer themselves. Their "description of scene" is sent to the video driver/s, and each video driver is responsible for ensuring it's rendered and displayed on time (where "ensuring its rendered" means that the video driver ensures that one or more "renderer services" does the rendering; where a native video driver with GPU support may provide its own built-in "renderer service" for its own use and for other drivers to use).

embryo2 wrote:

Brendan wrote:It is possible to cover the majority of (e.g.) new rendering algorithms because the interface is high level. To understand this imagine it's a "calculator service" where applications send a request like "calculate 5*4". In this case it doesn't matter much how the "calculator service" represents number internally, and doesn't matter much if it does "result = 5*4" or "result = 5+5+5+5" or "temp = 5+5; result = temp + temp;" because these details are hidden behind the high level abstraction and don't effect the application using the service. The algorithms that the "calculator service" uses can be completely changed to something radically different and nobody will know or care.

I recommend you to split the extensibility part into two pieces - the system and application ones. On the application level there could be a few system variants available. One system is exactly what you are going to implement while all others are some alternatives, that use some lower level gates into the OS internals. For you to control application behavior it is enough to provide a user with only your system variant. But the user should have a choice of another systems with another rules. Here the other variants will come into play. The system level is still higher than OS level, but extends the OS at the lowest possible level. And it's not a new OS, because it still uses some OS services. It's like different Linux distributions, when all security and compatibility issues are completely on the responsibility of the distribution vendor. The failure to provide the low level extensibility (if your OS to succeed) will lead to hacking the source code and creation of something completely unacceptable by you. So, it is better to start caring about the system level extensibility for the actual winner to emerge in the world of competing distributions, at least at the system level controlled by you.

But if you expect that the only your vision will lead the game, then even the application level of extensibility and configurability is useless and can be omitted (with all the bad consequences, that you think are unimportant).

embryo2 wrote:

Brendan wrote:A GUI, games and applications only send "description of scene" (although its more complicated than that - e.g. asking for data to be pre-loaded, then setting up an initial scene, then just describing "what changed when" after that). The GUI, games and applications don't use any renderer themselves. Their "description of scene" is sent to the video driver/s, and each video driver is responsible for ensuring it's rendered and displayed on time (where "ensuring its rendered" means that the video driver ensures that one or more "renderer services" does the rendering; where a native video driver with GPU support may provide its own built-in "renderer service" for its own use and for other drivers to use).

An application provides a "bytecode", a driver sends it to a rendering engine (a compiler?) and receives raw bytes with a color for every pixel, next the driver sends the colors to the hardware. Here the compiler part is hidden from an application developer, but the "bytecode" is perfectly visible and can limit a developer essentially. For example, the HTML has many updates since it's first appearance, next it was extended with CSS and finally the JavaScript was introduced to make the "scene description" acceptable for the real world. So, what do you think about the scene description?

Brendan wrote:The full thing is much more complicated; but I think I've solved the majority of the problems; and I think I can get the equivalent of "infinite super-sampling" out of it, and perfect "focal blur", and perfect motion blur, and perfect dynamic lighting/shadows.

Brendan wrote:How many GPU drivers, shader language compilers and game engines have you written?

Brendan wrote:By shifting complexity into a library it's still there, and now you've got to learn "random library API of the week" on top of that.

Brendan wrote:Yes, technology changes, and by the time I even consider implementing the native GPU support that is required for your brain dead rantings those GPUs will probably be obsolete too (and I'll be extremely glad that I didn't design an API around the way GPUs happen to work today).

Brendan wrote:Sure; you can't convert a web page designed for 1024*768 into Braille or sound (via. speech synthesiser) because it's a higher level representation of the document; and you most definitely can't add fancy graphical effects to a game that was never designed for it.

Rusky wrote:

Brendan wrote:The full thing is much more complicated; but I think I've solved the majority of the problems; and I think I can get the equivalent of "infinite super-sampling" out of it, and perfect "focal blur", and perfect motion blur, and perfect dynamic lighting/shadows.

Would be genuinely interested in more details and/or previous similar systems, perhaps in another thread.

Rusky wrote:

Brendan wrote:How many GPU drivers, shader language compilers and game engines have you written?

Can't speak for Owen, but I have used and worked on several game engines, which entails heavily using (if not writing) drivers and shader compilers. As far as I know, the last game you worked on was in something like C64 BASIC. You certainly don't seem to have much grasp on what modern game engines actually do, as you keep referring to it as "mountains of hassles" for game developers, which it is not- your API sounds exactly like existing game engines, but magically shifting under their feet far more than existing drivers do, and without the workaround of a common game engine library to handle the shifting.

Rusky wrote:

Brendan wrote:By shifting complexity into a library it's still there, and now you've got to learn "random library API of the week" on top of that.

Just because you can make/use a new library doesn't means you have to every week or that it's even a good idea. Granted, this sort of thing is very much a problem in fields like client-side web programming, but I attribute that to the domain rather than the libraries- the "game engine ecosystem" is much more stable.

Rusky wrote:

Brendan wrote:Yes, technology changes, and by the time I even consider implementing the native GPU support that is required for your brain dead rantings those GPUs will probably be obsolete too (and I'll be extremely glad that I didn't design an API around the way GPUs happen to work today).

Unlikely. GPUs have gotten more general-purpose over time, not less. Much like executable formats and system call interfaces have become more stable since the days of DOS when they had to be extremely hardware-specific for performance reasons, GPU APIs and shader languages have also become more generic and uniform. Any changes to GPU interfaces will be similar to changes to CPU interfaces- backwards-compatible additions to their capabilities that can be automatically taken advantage of by new compilers.

Rusky wrote:

Brendan wrote:Sure; you can't convert a web page designed for 1024*768 into Braille or sound (via. speech synthesiser) because it's a higher level representation of the document; and you most definitely can't add fancy graphical effects to a game that was never designed for it.

Way to demonstrate my point. Mods are always designed for one specific game, not included in a driver update that magically improves all your games.

Brendan wrote:If you spent that time (e.g.) building bird houses out of wood, you can point at the bird houses you've made and say "this is what I did". People can see the results of your efforts and understand what you've achieved. Ideas aren't like that.

Brendan wrote:Messages for video and sound go from child to parent, to grandparent, ..., all the way to drivers. Messages for keyboard and mouse follow the same paths but go in the opposite direction (from drivers to driver's child, to driver's grandchild, ...). At each step along these paths messages may be forwarded "as is"; but they can also be modified.

Brendan wrote:You can't take a photo of "concept for highly scalable scheduling algorithm" and show your brother-in-law next time he asks what you've been doing. He doesn't understand how computers work, and you can't explain it.

embryo2 wrote:

Brendan wrote:Messages for video and sound go from child to parent, to grandparent, ..., all the way to drivers. Messages for keyboard and mouse follow the same paths but go in the opposite direction (from drivers to driver's child, to driver's grandchild, ...). At each step along these paths messages may be forwarded "as is"; but they can also be modified.

You confuse the messaging protocol with data representation. For your system to be able to use your messages it should understand them. A protocol usually concerned with moments like "give it" or "take it" where "it" is the data, that should be understood by a recipient (but shouldn't be by the protocol). A recipient usually concerned with the data and shouldn't understand the protocol, that delivers the data.

embryo2 wrote:

Brendan wrote:You can't take a photo of "concept for highly scalable scheduling algorithm" and show your brother-in-law next time he asks what you've been doing. He doesn't understand how computers work, and you can't explain it.

You can create a demonstration. If you are so inclined towards 3D, then the demonstration can look much more attractive than the bird's nest. But the problem here is your ability to finalize your idea. It's better to start making simple, but finished, demonstrations, instead of keeping dreaming about the best ever OS. When there would be some finished products, then the chances for the world to see the best ever OS are greatly increased.

Brendan wrote:I've been researching new ideas for 20 years now. The end result is a reasonable collection of ideas that (if/when implemented at the same time) have the potential to make my OS better than anything that currently exists in multiple areas. This "potential to be better than anything that exists" is the reward for all the work, and all the sacrifices. But none of it is patented, and if you say too much you risk losing your reward, so you only show the tip of the iceberg and don't put too much information in one place.

Brendan wrote:Finally (about 2 years ago) I decided it'd be fun to do something like Minecraft in Java with LWJGL. I was wrong, it wasn't fun, and even simple things (like trying to determine how many CPUs the system has) was painfully frustrating. This lasted about 3 weeks before I remembered how much Java and OpenGL both suck.

Brendan wrote:In theory I could have "wrapper services" (where video driver has lower level interface and wrappers provide higher level interfaces), but that destroys all the benefits of having all the different pieces (widgets, apps, games, GUIs, etc) all using the same standard messaging protocol that everything understands, and therefore destroys the flexibility of the system as whole.

Brendan wrote:For hardware there's a compromise between "efficiency for one specific job" and "flexibility". Rendering got shifted from CPU to hardware accelerators to improve efficiency, then the hardware accelerators got replaced with "graphics processors" to regain the flexibility, now GPUs are getting more "CPU like" and CPUs are getting more "GPU like". Within 10 years I wouldn't necessarily be too surprised if we go full circle and end back to where we started (where "very generic/flexible" CPUs do it all). You only need to look at Xeon Phi ("many core 80x86 with AVX-512") and what Intel has being doing with SIMD in recent generations to realise this is at least a possibility.

Brendan wrote:Yes; a shader mod for one specific game won't work for other games because there isn't a standard representation for data that's used by all games. This problem doesn't apply to what I'm proposing (where there is a standard representation of data used by all games) and is therefore irrelevant.

Now let's try "1 + 1 = 2". By changing the software that GPU executes (regardless of whether that software came from a game or from the video diver), and ensuring that games do use a standard representation for data, improving the video driver can improve the quality of graphics in all games.

Rusky wrote:

Brendan wrote:I've been researching new ideas for 20 years now. The end result is a reasonable collection of ideas that (if/when implemented at the same time) have the potential to make my OS better than anything that currently exists in multiple areas. This "potential to be better than anything that exists" is the reward for all the work, and all the sacrifices. But none of it is patented, and if you say too much you risk losing your reward, so you only show the tip of the iceberg and don't put too much information in one place.

That's unfortunate. It's possible many of these ideas will never see the light of day, and I personally don't think you really need a patent to benefit from them (or that software patents in general should exist). Certainly nobody will benefit from them if they're never shared or implemented. It's up to you, though- I guess consider this some encouragement to build your renderer instead.

Rusky wrote:

Brendan wrote:In theory I could have "wrapper services" (where video driver has lower level interface and wrappers provide higher level interfaces), but that destroys all the benefits of having all the different pieces (widgets, apps, games, GUIs, etc) all using the same standard messaging protocol that everything understands, and therefore destroys the flexibility of the system as whole.

There's no reason all the wrapper services can't speak the same protocols, just like all the many OSes speak TCP/IP. Of course wrapper services also allow for new (versions of) protocols to be created and used alongside the old, if/when necessary.

(widget) => (application1) => (VT_layer) => (driver)

(widget) => (application1) => (GUI) => (VT_layer) => (driver)

(widget) => (application1) => (GUI) => (ice_effect) => (VT_layer) => (driver)

(widget) => (application1) => (GUI) => (ice_effect) => (splitter) => (VT_layer) => (driver)
                                                                  => (recorder) -> VFS

... => (GUI1) =>  (splitter) => (ice_effect) => (GUI2) => (splitter) => (VT_layer) => (driver)
                             =================>                      => (recorder) -> VFS

Rusky wrote:

Brendan wrote:Yes; a shader mod for one specific game won't work for other games because there isn't a standard representation for data that's used by all games. This problem doesn't apply to what I'm proposing (where there is a standard representation of data used by all games) and is therefore irrelevant.

Now let's try "1 + 1 = 2". By changing the software that GPU executes (regardless of whether that software came from a game or from the video diver), and ensuring that games do use a standard representation for data, improving the video driver can improve the quality of graphics in all games.

Let's think more specifically about how you would apply the moral equivalent of a Minecraft mod to all games. Shaders could be done if you standardized the input formats for meshes (trivial) and materials (not too hard either, and modern game engines already seem to be doing this by unifying most of their shaders for Physically Based Rendering). This is essentially your proposal. However, this excludes any other shader styles, which apparently everyone but you cares about.

Rusky wrote:It also excludes any optimizations like Minecraft using tesselation to minimize bandwidth usage, or GPU-side particle systems.

Rusky wrote:Textures and meshes, on the other hand, are completely impossible, since they'll always be stuck at the level of detail they were specified at, and are highly game-specific. This could be mitigated using vector-based formats, but that's not always optimal or even possible.

Brendan wrote:Would you rather there be many KiB of higher level data ("move object A to position X") being shifted around (potentially over network connections), or would you rather have many MiB of low level (e.g. vertex/pixel) data being shifted around (potentially over network connections)? Also think about how you'd implement that "ice effect" process - would you rather be processing low level data or higher level data?

Brendan wrote:For tesselation, I assume you just mean the standardised format for meshes allows a mesh to say "these polygons edges are smooth/curved" and the video driver can break it into as many or as few triangles at it feels like, using whatever method it wants. For GPU-side particle system I don't know if you mean graphics (e.g. "simple object/sphere with location and direction per particle") or physics (that has nothing to do with graphics/rendering).

Rusky wrote:

Brendan wrote:Would you rather there be many KiB of higher level data ("move object A to position X") being shifted around (potentially over network connections), or would you rather have many MiB of low level (e.g. vertex/pixel) data being shifted around (potentially over network connections)? Also think about how you'd implement that "ice effect" process - would you rather be processing low level data or higher level data?

I'm not proposing that every application use a low-level API exclusively- all your examples still work with e.g. the game using a low-level API that the GUI knows how to forward efficiently, just like existing OSes do, with the exception of trying to run a game over a network, which wouldn't work anyway.

Rusky wrote:

Brendan wrote:For tesselation, I assume you just mean the standardised format for meshes allows a mesh to say "these polygons edges are smooth/curved" and the video driver can break it into as many or as few triangles at it feels like, using whatever method it wants. For GPU-side particle system I don't know if you mean graphics (e.g. "simple object/sphere with location and direction per particle") or physics (that has nothing to do with graphics/rendering).

For tesselation in the context of Minecraft, I mean "the game sends only which voxels are filled, and the GPU generates the actual cubes' geometry, for massive bandwidth savings." For GPU-side particle systems I mean "a combination of relatively simple object rendering combined with physics in the same GPU program for zero bandwidth usage."