Graphics API and GUI

[Ready4Dis]

So, I was thinking about my Graphics API that I wanted to implement, and was thinking about how to interface my GUI with my Graphics card. I know in windows they pass the window handle (hwnd) to the graphics card in order for it to get the correct rendering context in 3d (and others). Is the 'best' way to give an app a rendering context that maps to it's window area and use that for rendering? My gui needs to be able to composite multiple rendered objects into a seamless desktop appearance. So, does each app ask the GUI for a rendering context, then the app registers that same rendering context with the graphics driver, that way later on when the GUI is building the screen it can ask the graphics driver for the window/texture/whatever? I guess my biggest confusion is really about how they all fit together. I don't want to pass all my 3d rendering code into the GUI, I'd rather them go straight to the graphics driver. At the same time, do I just make ALL calls go to the graphics driver and only use the 'GUI' as more of a compositor? That would mean each and every window would have to handle it's own button drawing, etc. Or do I make a common set of functions as part of the GUI that 'can' be used for generic things and bypassed for things that are needed? I know there is more than one way to do this, but do you guys have any suggestions, good reading links, etc?

This came up because I was looking into some graphics API's (dx12, vulkan) and decided I'd like to support Vulkan (successor to opengl) in my OS, but am still trying to figure out how exactly it should fit in.

[SpyderTL]

First off, it's entirely up to you how you want to build your OS and its components, so all we can do is offer suggestions and opinions. There's no "right" way to do it. Just a bunch of "wrong" ways

Most people advocate for completely separating your applications, so that they can not possibly have any effect on each other (or, more importantly, the kernel). Following this logic, this means that you can't let applications "share" any resources (like, say, the video card). This leaves two options: a) Have all applications communicate with the OS, and have the OS communicate with the video card, or b) only allow one application to have access to the video card at a time.

Windows started out with option A for applications, and option B for games, but later changed it to option A for both applications and games, because allowing games to have direct access to the hardware was fast, but unstable.

The reality is that you'll need to make this same decision for all system resources, not just the video card. You can decide for each type of resource, or you can just say that no application will ever have direct access to any resource, period. The former may be marginally faster (at run time), but the latter is the only way to guarantee stability.

[Ready4Dis]

Thanks for the response, and I understand the app can't have direct access, it was more about where the sharing and context information happens. I think we differ on the definition of sharing in this case. Whenever multiple processes are using the same resource, I call it sharing a resource. I understand that they don't all get full access to all of the GPU, they go through an API layer that connects to the driver via a system call, but if 10 apps are rendering on screen, then 10 apps are sharing the GPU (in my post).

You have a few layers to worry about. The gpu driver itself, the graphics api (opengl, dx, glide, whatever you're poison), you have your compositor, and you have your apps.

I know nvidias fermi arcitecture has a certain number contexts built into the GPU and it uses a round robin scheduler (that you can modify the time slices of) in order to process each contexts requests. I was just wondering if this is the best method, or would it make more sense that all the context information be built into the driver, or the api layer? I understand this is a decision that I ultimately have to make, I was wondering if anyone could shed some light on ideas or reasons why/why not for different ways of doing things. Does the GPU/driver actually HAVE to know about contexts, or can you just use the API to do all of the context switching/handling, or should everything run through the compositor which can prioritize (which means another layers for calls to go through), then send the GPU commands in the correct order to the correct command buffers?

[SpyderTL]

I would think that you would want to support all three approaches (API, Driver and Hardware), if at all possible. Hardware support would (or should) have the best performance, while API support would have the best compatibility. You would just need a way for the API to determine whether it should "pass-thru" the context information to the driver or handle the rendering, itself. Likewise, the driver may be able to pass through the context to the hardware, if at all possible.

The more "similar" you can make each of these "layers", the easier they should be to implement.

Also, if you want to support multiple video cards, possibly with different drivers, you'll need to manage that at the API level, or maybe in another layer between the API and the Drivers. So you would have multiple applications talking to (hopefully) one API, which, optionally, talks to one "virtual" screen, which talks to multiple drivers, which talk to multiple video cards. And each video card may be communicating with multiple displays.

Putting all of this together may affect performance, though. So you may also need a way to "bypass" one or more of these layers in order to, say, improve performance on a single monitor.

It would also help if you could clarify what you mean by "context". I'm assuming that you ultimately want to support everything between setting a single pixel to a specified color, and drawing 3D primitives using compiled shaders. Also, displaying compressed video by streaming in bytes from a file or network connection poses an interesting challenge to this design, as well.

I'm not sure how helpful this is at a high level. It might help to know what your priorities are at this point, and what features you really want to support.

[Ready4Dis]

By context I mean a process' access to the video card. Each process should be separate, and the video card (hardware or driver) must be aware that multiple processes are all sending commands. It needs to know which one to process at which time. This *could* be done at the API level (aka, opengl, mantel, etc) but the more I think about it, the more I think the driver needs to be aware. I have started working on a mantle software implementation, since Vulkan is based off of mantle, it should be easy to switch this over once they get around to releasing the specs.

The way I am thinking right now is like so:
A windowed app will submit a request to the compositor for a context/window reference. This reference is an actual buffer on the GPU. The app can then render anything it wants into that buffer and the compositor can display it as a texture inside of the 'desktop' or task manager, or on a 3d cube, whatever it wants. Which one should be responsible for rendering the borders ,min/max buttons? If the app does it, I will lose all kinds of continuity. If the compositor does it, it would just overlay that on top afterwards, but the app needs to tell the compositor what kind of stuff it wants drawn (borders, title bar, min/max, etc). I guess that really would be about the extent of it. The compositor would notify the app if their was a window resize, but other than that, they really don't talk a lot (besides keyboard/mouse/joystick inputs).

I don't want a bypass, I want continuity. The API (opengl, directx, etc) on any common OS is not bypasses. The compositor shouldn't be bypassed, as it really doesn't do a lot besides display the buffer to the actual screen. If you are full screen, it can only render your back buffer to the entire screen, it doesn't really make sense to bypass this. Yes, multiple video cards with multiple displays is going to be a tough one, especially if they move a window from one screen to another. All the textures, buffers, etc need to be moved from one GPU to another.

My priorities/features? I am looking for something that uses the hardware for compositing, without being tied to a particular style (aka, the compositor can be replaced without breaking any apps). I plan on using vulkan as my main API, because it will become mainstream and I don't want to support the bloat of opengl and it's many versions. This should make porting apps not to difficult in the long run. The multi-gpu / multi-monitor issue might be something I need to look into further because that may cause some issues. Do I tell the app that it lost it's context and it needs to reload all of it textures, or put this somewhere else where it is smart enough to read all the textures/buffers back from the GPU and move them to the another GPU? Would that need to be part of the API or does the compositor need to do this work (which would mean ALL calls need to go through it for it to know which textures are part of which app, which I'd like to avoid). How do other OS's handle this?

Streaming video does not pose any challenges, app asks compositor for a window buffer, then renders directly into it just like it would in any other way (either via hardware decoding or software decoding).

Thanks for all the thoughts and ideas, that is really what I am looking for in this. Multi gpu/monitor case is a really good thinking point.

[SpyderTL]

Windows handles this by providing two completely separate APIs to the application. One provides access to the window manager, which handles most of the work of communicating with the driver(s) and keeping the screens updated, and allows the application to focus on the functionality. This works well for things like word processors and bitmap editors, because performance is less of a concern than compatibility.

The other API provides direct access to the hardware (or almost direct access), which allows the application more flexibility and support more features, but also requires the application to handle most of the hardware management.

So it's a trade off, and you probably don't want to pick one approach if you want to support both games and applications. Unless you can find a clever way to support both use cases at the same time. But I've yet to see someone pull it off.

My plan is currently to just put both sets of functionality into one API, and offload whatever I can to the driver and the hardware, but that's about as far as I've gotten.

[Ready4Dis]

Yes, I can see how it could be a performance benefit to split the two, but I would prefer not to if possible. I am thinking that I can have the compositor be able to do things like render borders, default buttons, text, etc, which just passes calls to the graphics API. The app could talk directly with the graphics API directly if it wanted to. The compositor just needs to know if there was an update or not and know the reference to the buffer that the app is using. I would prefer not to build my compositor API into the graphics API because I want to be able to write multiple compositors. I want the graphics API to be the standard vulkan API (successor to opengl) that way porting is easy(ish).

Unless you can find a clever way to support both use cases at the same time. But I've yet to see someone pull it off.

I thought this was how most newer OSs and such worked? If both the compositor and the app have the same reference to the draw buffer, why can't they work together instead of against each other? The compositor can make draw calls to the graphics API and so can the application. The application can used the predefined calls in the compositor or it can call the graphics API directly, either way is perfectly acceptable and doesn't interfere with. The app can use the easy way or direct way without having a performance penalty (or at least very minimal penalty).

I am implementing some ideas under windows to test it out, I will probably write my entire compositor,graphics API and a software 3d 'driver' (just an app that runs and uses named pipes for my other apps to communicate with) under windows for for easy testing before I try putting it into my OS. I'm sure i'll find plenty of answers once I start putting the pieces together, I just wanted to see if I could come up with a good starting point and maybe some details before I waste to much time.

Again, I appreciate all the help and point of view!

[Ready4Dis]

Just wanted to clarify one other thing that I was curious of.

"When you run your program, opengl32.dll​ gets loaded and it checks in the Windows registry if there is a true GL driver. If there is, it will load it. For example, ATI's GL driver name starts with atioglxx.dll​ and NVIDIA's GL driver is nvoglv32.dll​. The actual names can change from release versions."

I new each video card vendor couldn't supply their own opengl32.dll otherwise it wouldn't work with multiple gpu's. Each vendor registers their version of the opengl library and opengl32.dll looks for registered vendor libraries in order to pass the function calls along (or when get procedure address is called it can return the correct procedure address). That makes some sense and gives me some ideas.

You can't link the application against the vendor library because you might want to render to both video cards simultaneously, the intermediate library needs to handle calling the correct procedure from the correct library. This seems rather inefficient (another level of indirection). Anybody have an other information on the inter-actions of these separate libraries or how dx 11 or 12 handles this? Or how the mantle.dll handles multiple gpus?

[SpyderTL]

This is why I mentioned above that the more similar your various API layers were to each other, the better your life would be. I'm a C# developer, so I tend to think of the OS in terms of Objects and Interfaces.

To me, the interface provided by your API should be similar or identical to your compositor, and your driver, so that you can a) bypass certain layers as needed, for performance, and b) add additional layers, as needed, in the future to add support for things like remote network terminals or 3D VR headsets without needing to recompile entire applications.

Of course, this is overly simplified, and there's probably a lot more to it than that, but that's my personal goal.

But in any case, the key to mixing hardware from different vendors is at the driver interface level. The OS and the driver have to agree on a contract that the driver will adhere to. It really doesn't matter what that interface is called, or what is in that interface, as long as they both agree. At that point, they are no different than any other DLL. You can even have more than one interface, and let the driver decide which one to use.

[Ready4Dis]

Thanks for the brainstorm, I know I need a consistent interface at some level, but I don't think there is much point in making the compositor and the graphics api the same. I would prefer the compositor has higher level calls that are exposed and the graphics api has the lower level calls exposed. The user is free to choose whichever (or both) to use depending on their needs. I want my graphics API to be vulkan that way it will be easier to port games and such over, and I can do testing under multiple OSs once it finally gets released. I am looking into DX12 as well since it works similar to mantle/vulkan, see how bad it would be to implement some support as it seems very similar in all honestly. It looks like opengl and microsoft basically just looked at mantle and where like, why didn't we think of that, thanks amd.

[Brendan]

Hi,

Thanks for the brainstorm, I know I need a consistent interface at some level, but I don't think there is much point in making the compositor and the graphics api the same. I would prefer the compositor has higher level calls that are exposed and the graphics api has the lower level calls exposed. The user is free to choose whichever (or both) to use depending on their needs. I want my graphics API to be vulkan that way it will be easier to port games and such over, and I can do testing under multiple OSs once it finally gets released. I am looking into DX12 as well since it works similar to mantle/vulkan, see how bad it would be to implement some support as it seems very similar in all honestly. It looks like opengl and microsoft basically just looked at mantle and where like, why didn't we think of that, thanks amd.

Erm...

To get competitive performance in a modern (e.g. resolution independent) graphics system; you mostly must use the GPU (even for just 2D). This means that the video driver (with the assistance of GPU) must do all rendering (and if/when the video driver doesn't support the GPU it still has to do the rendering in software).

For the video driver/GPU to do the rendering it needs to know what you want drawn; so you need to have some sort of higher level description of what to draw. This can be something like a list of OpenGL commands or something else (or something better). It doesn't really matter too much (it's beyond the scope of what I'm trying to say here).

Now; a single massive "higher level description of everything" would be completely insane. You want a to break it down into pieces. You want a graph. More specifically you want a dependency graph.

Basically; you want to break it into "things" (textures, vortex cubes, point clouds and/or whatever - it doesn't matter much which it is and is also beyond the scope of what I'm trying to say); where each "thing" has:

• A higher level description; which could be a list of commands, but could be a file name or something else (or even a UTF-8 string that the video driver can send to a font engine).
• A list of "things" it depends on/uses/includes.
• None or more lower level representations (e.g. pixel data, voxel data, whatever).

This allows basic/intelligent caching behaviours. If the higher level description for texture #9 is changed then the video driver knows it has to update the lower level data for texture #9, but (due to the dependency graph nature of it) also knows that it has to update texture #3 (assuming that texture #3 includes/uses texture #9 in some way). It can even be smarter than that - e.g. if something is occluded in some way (e.g. hidden behind something else, or past the edge of the screen) maybe the video driver doesn't need to re-render the lower level data for the thing and can skip it until/unless it becomes visible at some point in the future.

Now; let's add a basic ownership system to this. What if each "thing" in the dependency graph has an owner, where only the owner can set/modify the thing's higher level description? In this case the GUI might own the root thing (the frame buffer); and texture #9 might be owned by an application; and the higher level description for the root thing might say "put texture #9 at (x, y)" in the root texture". Whenever the application tells the video driver to change the higher level description for texture #9, the video driver automatically knows it has to update the GUI's root thing/frame buffer.

With this sort of a system; a compositor may just be part of the code that creates the higher level description of the GUI's root thing. It's not a separate/independent piece of the system. It doesn't need anything like a special API. There's nothing special about it at all.


Cheers,

Brendan

[Ready4Dis]

Hi,
Erm...

To get competitive performance in a modern (e.g. resolution independent) graphics system; you mostly must use the GPU (even for just 2D). This means that the video driver (with the assistance of GPU) must do all rendering (and if/when the video driver doesn't support the GPU it still has to do the rendering in software).

For the video driver/GPU to do the rendering it needs to know what you want drawn; so you need to have some sort of higher level description of what to draw. This can be something like a list of OpenGL commands or something else (or something better). It doesn't really matter too much (it's beyond the scope of what I'm trying to say here).

Yes, I am planning to implement the vulkan api, which is the api from Khronos that will succeed opengl. I plan to implement software rendering first (as I don't have time to write graphics drivers ATM), then follow it up with maybe an intel integrated GPU driver for one of my systems since they seem to have the best documentation. It does work at a higher level of sorts, but not as basic as opengl, it works straight on the command buffers, command queue's, etc. It hasn't been finalized but is VERY close to AMD's Mantle API (they even used the same function names, just replaced the gr with vk). My question was more about the interaction between all the parts, I mostly understand what each part does, but there is some overlap (and it can be done in multiple ways). I have most of the mantle core library implemented, and part of my software rendering driver done.

Now; a single massive "higher level description of everything" would be completely insane. You want a to break it down into pieces. You want a graph. More specifically you want a dependency graph.

Yes, my compositor will be responsible for storing information about which window is where, which is on top, etc. The graphics API (Vulkan) will take the low level commands which copy textures into graphics memory, send vertex and index buffer arrays, load pixel/vertex shaders ,etc.

This allows basic/intelligent caching behaviours. If the higher level description for texture #9 is changed then the video driver knows it has to update the lower level data for texture #9, but (due to the dependency graph nature of it) also knows that it has to update texture #3 (assuming that texture #3 includes/uses texture #9 in some way). It can even be smarter than that - e.g. if something is occluded in some way (e.g. hidden behind something else, or past the edge of the screen) maybe the video driver doesn't need to re-render the lower level data for the thing and can skip it until/unless it becomes visible at some point in the future.

Hmm.. shouldn't the app be responsible to know if it needs a texture changed (or the compositor if it's a function of it)? I mean, when I did a lot of 3D proramming, I didn't expect the graphics driver to take care of anything, I specifically told it about the data, when it changed, and what to do with it. Maybe I am misunderstanding your view of 'High Level', but to me anything high level the graphics driver should not be aware of (at least not directly). I'm not really sure I would want to put all that into the driver, as the driver really just needs to know how to convert formats and send commands to/through the gpu. I don't disagree there could be a higher level on top of that, but I don't expect the driver itself to know all that.

Now; let's add a basic ownership system to this. What if each "thing" in the dependency graph has an owner, where only the owner can set/modify the thing's higher level description? In this case the GUI might own the root thing (the frame buffer); and texture #9 might be owned by an application; and the higher level description for the root thing might say "put texture #9 at (x, y)" in the root texture". Whenever the application tells the video driver to change the higher level description for texture #9, the video driver automatically knows it has to update the GUI's root thing/frame buffer.

That is pretty much what I was going to use the compositor for. Keep track of each apps buffer and its location. It can then render each buffer in the correct location, in the correct order (on top, etc) and even add its own dressing to it (borders, buttons, etc)

With this sort of a system; a compositor may just be part of the code that creates the higher level description of the GUI's root thing. It's not a separate/independent piece of the system. It doesn't need anything like a special API. There's nothing special about it at all.

We are basically saying a similar thing, except in my case, the compositor isn't a part of the code that creates the description, it is the thing that handles it. I was just wondering where to put things like a standard button? Most windows apps all look the same because they have generic types for most objects. If you leave it up to the app developer, no two apps will look the same. If I expose some standard gui elements via the compositor (or something else, like a standard gui interface api that doesn't have to be part of the compositor), then everyones apps can look the same and if a user wants to change something, it takes effect for everything. I am trying to figure out where each piece of code belongs, how many layers it will be, and how it can all work together and not against each other. I think my focus is going to be getting the Mantle API implemented with a software back end working first so i have a way to test it out.

Thanks for the in depth post, gives me some more to think about.

[Brendan]

Hi,

This allows basic/intelligent caching behaviours. If the higher level description for texture #9 is changed then the video driver knows it has to update the lower level data for texture #9, but (due to the dependency graph nature of it) also knows that it has to update texture #3 (assuming that texture #3 includes/uses texture #9 in some way). It can even be smarter than that - e.g. if something is occluded in some way (e.g. hidden behind something else, or past the edge of the screen) maybe the video driver doesn't need to re-render the lower level data for the thing and can skip it until/unless it becomes visible at some point in the future.

Hmm.. shouldn't the app be responsible to know if it needs a texture changed (or the compositor if it's a function of it)? I mean, when I did a lot of 3D proramming, I didn't expect the graphics driver to take care of anything, I specifically told it about the data, when it changed, and what to do with it.

You'd have to tell video driver when the description of what to draw changes; but if the description of what to draw remains the same there'd be no need to tell the video driver again.

For an example; lets say you're writing a 1st person racing car game; and:

• the description for texture #1234 tells the video driver to use a certain mesh, certain textures, specific camera angle, etc to generate a picture/texture of a tree
• the description for texture #1111 tells the video driver to use many meshes and textures (and camera angle, position of lights, etc) to generate a texture for the car's rear-view mirror; and this includes inserting texture #1234 in multiple places (where the picture/texture of a tree is being used as an impostor).
• the description for texture #1010 tells the video driver to use a few textures and the font engine to generate a texture for the car's dash
• the description for texture #1005 tells the video driver to use a few textures and the font engine to generate a texture for the game's HUD (compass, mini-map, current score, whatever).
• the description for texture #1001 tells the video driver to use many meshes and textures (and camera angle, position of lights, etc) to generate a texture for the player's view out the car's windscreen; and this includes inserting texture #1234 (the tree) in multiple places, inserting texture #1111 (the rear-view mirror), inserting the texture #1010 (the dash), etc.
• the description for texture #1000 tells the video driver to combine texture #10001 (the player's view) with texture #1005 (HUD)

Now imagine that the car is parked in someone's back yard and there's a dog walking around behind the car. This causes the description for texture #1111 to be changed by the game. On seeing this description change the video driver knows it has to update the pixel data for texture #1111 (rear-view mirror), #1001 (player's view from driver's seat) and #1000 (game's final texture).

Let's also assume that:

• the description for texture #200 says to start with texture #1000 in a rectangle in the middle of texture #200 and then draw some borders and other window decorations around it.
• the description for texture #104 tells the video driver to get the texture's pixel data from "/home/Ready4Dis/Desktop/background.bmp"
• the description for texture #0 says to put texture #104 in the background, then put texture #180 (task bar or something?) at the bottom of this texture; then it has a mesh for a cube and says that texture #200 is on the left face of that cube (with other textures for other windows on the other faces of the cube). Note that texture #0 is the "root thing" (frame buffer).

Note that (e.g.) mapping the game's window onto the side of the GUI's "cube of apps" isn't really any different to (e.g.) mapping the "rear-view mirror texture" into the "player's view from driver's seat" texture. In both cases the description just says how/where the child texture gets added into the parent texture (via. a mesh).

Now imagine that the car is parked in someone's back yard and there's a dog walking around behind the car. This causes the description for texture #1111 to be changed by the game. On seeing this description change the video driver knows it has to update the pixel data for texture #1111 (rear-view mirror), #1001 (player's view out the windscreen), #1000 (game's final texture), #200 (game's decorated window) and texture #0 (the screen).

Maybe I am misunderstanding your view of 'High Level', but to me anything high level the graphics driver should not be aware of (at least not directly). I'm not really sure I would want to put all that into the driver, as the driver really just needs to know how to convert formats and send commands to/through the gpu. I don't disagree there could be a higher level on top of that, but I don't expect the driver itself to know all that.

The higher level description may not be high level (e.g. could be stored lists of low level commands that are sent to GPU). Basically you have a low level representation (raw pixel data, voxel data, fragments, point clouds, whatever) and something that tells video driver/GPU how to render/generate that low level stuff (the description).

It's like asking a (human) artist to paint a picture. Without at least some sort of description of what the picture should contain you can't realistically expect anything more than a blank canvas.

With this sort of a system; a compositor may just be part of the code that creates the higher level description of the GUI's root thing. It's not a separate/independent piece of the system. It doesn't need anything like a special API. There's nothing special about it at all.

We are basically saying a similar thing, except in my case, the compositor isn't a part of the code that creates the description, it is the thing that handles it.

So you have:

• a human artist that paints pictures from descriptions (or, a video driver/GPU that does rendering)
• customers that provides descriptions of what they want the artist to paint (or, processes that provide descriptions)
• the artist's manager who pretends he's "handling" something but mostly just increases the fees customers pay the artist without doing anything useful (or, a "compositor")

As far as I'm concerned, when the graphics API that applications use is a crippled piece of ancient crap (e.g. where application diddle with raw pixels) and you want to use modern desktop effects; then you tell the video card/GPU that each application's window is a texture so that you can use the GPU to do those modern desktop effects; and this is a "compositor". When your graphics API isn't a worthless pile of puke and everything is (potentially) using GPU (and nothing is doing idiotic pixel pumping), "compositor" becomes mostly meaningless because it's not doing anything that everything else isn't also doing.

I was just wondering where to put things like a standard button? Most windows apps all look the same because they have generic types for most objects. If you leave it up to the app developer, no two apps will look the same. If I expose some standard gui elements via the compositor (or something else, like a standard gui interface api that doesn't have to be part of the compositor), then everyones apps can look the same and if a user wants to change something, it takes effect for everything.

Most people would just use a shared library for this (but for my OS I'd use a service).


Cheers,

Brendan

[Ready4Dis]

As far as I'm concerned, when the graphics API that applications use is a crippled piece of ancient crap (e.g. where application diddle with raw pixels) and you want to use modern desktop effects; then you tell the video card/GPU that each application's window is a texture so that you can use the GPU to do those modern desktop effects; and this is a "compositor". When your graphics API isn't a worthless pile of puke and everything is (potentially) using GPU (and nothing is doing idiotic pixel pumping), "compositor" becomes mostly meaningless because it's not doing anything that everything else isn't also doing.

This is basically what I was getting at. My thoughts where that compositor just has a list of registers 'textures' (aka, apps) and it can render them in a specific way (layout) for the user interface. It really doesn't do much, the apps render and talk directly with the graphics api, the compositor just uses those textures to display on the screen. Basically when the app is started it registers it's 'screen' (texture) with the compositor and then just renders normally using graphics api commands (sending vertex buffers, textures, etc straight to the video card). The compositor just takes all the registered textures and renders them in a way that makes sense for the user. It *could* be used to dress up the window, or as a middleman if you wanted, but there is nothing forcing you to use it in that manner. The only reason I can think of for using it for common things like rendering buttons, borders, etc is so that all windows can have a similar look and feel to them. This *would* be middle-man, but you are gaining uniformity. If the user wants to select a theme, and you used the 'standard' GUI objects, your app will automagically look like the rest of the OS. If you wrote your own, they obviously wouldn't change. This would (just like now) probably not be used by games at all, just basic apps (calculator, etc).

Yes, you only need to update the things that have changed, but it's not always that easy of course, especially in a game where things are mostly always moving. It makes more sense for a normal application that only updates when the user does something, although there is no need to separate the two. In graphics (game?) programming, you normally just run the game up to it's maximum frame rate, or a predetermined 'good' frame rate. Typically you are moving around and just render frame to frame, you aren't going to check if anything in the game moved, then check if it's potentially visible or partially obstructing something else that has been drawn, you are just going to throw it all at the GPU and let it do it's thing. Now, you will (hopefully) be using some sort of structure to not render the entire world, but you aren't going to check if that dog walking behind the car made the texture for the rear-view window change, as it already requires you to re-render the scene from the review mirror's perspective to check if the dog is even in it's view, in which case, you already rendered the new texture anyways.

Great post, lots of good info. I do agree that mapping the games window onto the side of a GUI cube (or any task switcher, etc) should not be difficult at all. The compositor (or whatever app is responsible) would simply check if that texture was update and render it to the side of the cube. Heck, for something like a task switcher, you could probably even just stop texture updates until it is done instead of trying to run the game on the side of a cube (in which case it might not be the app that has focus anyways).

Like I said though, the graphics API calls do not go through the compositor, they go straight to the graphics library which sends raw commands to the graphics card driver. The compositor is just basically aware of the GUI apps running and renders their windows (texture buffers) to the actual desktop/screen. I think our terminology and such is a bit off, but I believe we are talking about something very similar (unless I am misunderstanding something, which is of course possible).

Again, thanks for the time and effort put into your post, I may have to re-read it again to make sure I got everything!

[Brendan]

Hi,

As far as I'm concerned, when the graphics API that applications use is a crippled piece of ancient crap (e.g. where application diddle with raw pixels) and you want to use modern desktop effects; then you tell the video card/GPU that each application's window is a texture so that you can use the GPU to do those modern desktop effects; and this is a "compositor". When your graphics API isn't a worthless pile of puke and everything is (potentially) using GPU (and nothing is doing idiotic pixel pumping), "compositor" becomes mostly meaningless because it's not doing anything that everything else isn't also doing.

This is basically what I was getting at. My thoughts where that compositor just has a list of registers 'textures' (aka, apps) and it can render them in a specific way (layout) for the user interface. It really doesn't do much, the apps render and talk directly with the graphics api, the compositor just uses those textures to display on the screen. Basically when the app is started it registers it's 'screen' (texture) with the compositor and then just renders normally using graphics api commands (sending vertex buffers, textures, etc straight to the video card).

But...

a) If the video driver manages a dependency graph of "things" (where each thing might be a flat 2D texture, but might be texture with bumps, or a voxel space, or a point cloud, or a set of meshes and textures, or ....); then nothing prevents you from saying (e.g.) "This application's window is a point cloud and not a flat 2D texture".

b) The entire video system involves generating "things" by combining "sub-things" in some way; and because this "combine sub-things" functionality is ubiquitous and used throughout the entire system it's silly to consider "compositor" as a distinct/separate piece that is somehow important in any way at all. It would be more accurate to say that the system is not crap and therefore has no need for a "compositor".

Yes, you only need to update the things that have changed, but it's not always that easy of course, especially in a game where things are mostly always moving.

For 3D games, most things don't change every frame and there's significant opportunities (and techniques) to recycle data to improve performance; including VBOs, distant impostors, portal rendering, overlays (for menu system and/or HUD), etc.


Cheers,

Brendan