Replacing/rewriting x11

[AlexHully]

Hi,

I am aware of Wayland and multiple attempts to get rid of x11 as a window system.

I am digging in its -very large- code.

I think we face a design issue for modern hardware with x11 because, as wayland advocates, letting the application do the connections to the graphic cards may lead to a speed bump.

I would like something more like a two-mode system.

When not in a 3d game / full screen:
1) a window event system, to manage clicks and send to the right app the corresponding callback pointer. But not a server. Shared memory with an event loop.
2) an app with a given and compulsory framework to manage the event like it should (no separate window manager) or send buffers to graphic cards/devices.

When in a 3d game / full screen :
Embed the event system in the game directly for speed reasons and let the app do the work. Same code.

Security:
Raises some questions.
Can you override the framework? not if the kernel/os forces you not to or detects that an action comes from an app.
Is x11 really a good guardian against potential attacks? breaches happened just look at the website.
Do they happen often? No.


This is definitely the full screen event which will choose the strategy.

Right now, let's forget about the fact that few devs would change their habits (it matters, but not right now).
I am just drawing ideas and I am seeking for arguments of why they suck (yes, I happen to love it ) and what improvement you propose.

Bye

[embryo2]

Is x11 really a good guardian against potential attacks?

The really good guardian against anything is you, a developer, who should be able to write a good system. Any available code base can have errors and security breaches. And it is only you who can scrutinize the code and to decide on it's applicability for your purposes. But if you want just an easy answer then yes, some people think that x11 is a good thing. While other people do not agree.

The right way (but not easy) is to explore the overall terrain (high level view) by yourself while asking questions only about a particular areas (like - what's in this hole?). The security question from your post is about high level.

[AlexHully]

Thanks embryo,

I take your point on the security concern.

But my whole idea is about performance implication.
Do you think this design could fit the goal? Are there impossible things that totally discard the idea, at the global level?

[Rusky]

X11 is insecure not just because of implementation bugs, but because of its design, which allows applications to read each other's windows, intercept events, etc. Wayland fixes that so you can have a secure GUI that other apps can't mess with.

[Brendan]

Hi,

I would like something more like a two-mode system.

For 3D games (regardless of whether they're running in a window or full screen), you want to support something like "render to texture". For an example, for a car racing game you might use "render to texture" to dynamically generate distant impostors, make the rear-view mirror work properly, to generate the car's dash (speedo, RPM, etc), and maybe to generate an overlay for a menu system or something. Note that there are 2 cases here - rendering 2D primitives to a texture, and rendering 3D primitives to a texture.

When a 3D game is running in a window, you'd just render the entire thing to a texture; where the window is that texture (plus borders, title bar, etc). When a 3D game is running full screen, you'd just render the entire thing to a texture (the frame buffer).

For 2D games it's virtually identical. The only difference is that you'd only be using "render 2D primitives to texture" and wouldn't be using "render 3D primitives to texture".

For 3D applications (e.g. CAD, etc) there's no sane reason for it to be any different to 3D games. For 2D applications there's no sane reason for it to be any different to 2D games.

Basically; full screen vs. windowed, game vs. application, and 3D vs. 2D are all mostly trivial/superficial differences. You still need the same "render to texture" support for all cases, they all still need keyboard and mouse events, etc.

If you think you need a 2 mode system, then you're mistaken and heading directly towards a design failure.

The reason existing systems tend to be "2 mode" is that they were designed before 3D really existed, and 3D support had to be retro-fitted onto something that was never designed for it (combined with what I'd call momentum - e.g. a desire to avoid significantly changing something that's established/working and relied on by lots of existing software and tools).


Cheers,

Brendan

[AlexHully]

Thanks Brendan,

So yes, we need a way to render as directly as possible, but still in a need to separate properly the different applications on screen and let them temper one with another.

So let's forget the two-mode system. Bad idea.

We still need a simple third party allowing or not the drawing to ensure security and separation.

So that third party would be a few line of code with a bitmap maybe (or anything else, really) and answering the framework which would manage the screen buffering and so on. And this would happen without the server protocol (acks..) as wayland is smartly doing, but also without the xml parsing need.
At the practical level, we can imagine a locked shared memory that the framework would read (1 = ok, 0= not ok).

The new problem is at the level of the non modification of the framework (there will always be a weak point). What if an application decide to override that framework. The OS still has to control that it comes from an authorized entry point, with something like a secret session ID that the app cannot access.

So we have 3 checks for one complete rendering of the screen (one application is modified at a time anyway and the third party keeps track of what is possible, ensuring isolation).

But again, an OS can be modified. There is nothing that can enforce trust.
Anyway, if things can be that easily modified, the dev should compromise him/herself.

Yes it is being paranoid, but we are just discussing design not practicability of use (even if it matters at the utmost point).

As far as adoption is involved, it is to run a controlled hardware anyway, software can be slightly modified, so the world doesn't have to know.

So, what about those 3 checks now ?

[SpyderTL]

For 3D applications (e.g. CAD, etc) there's no sane reason for it to be any different to 3D games. For 2D applications there's no sane reason for it to be any different to 2D games.

From a purely design point of view, I agree. However, in the real world, "applications" and "games" have slightly different "priorities", mainly centered around partial screen updates vs. whole screen updates, and CPU usage.

For example, most games have no need for a font rendering engine, and most applications have no need to "upload" custom data and/or code to the video card.

These differences "may" be significant enough to justify a 2-mode system in order to optimize both use cases, but I've barely scratched the surface on both of these systems, so I don't know how necessary a 2-mode system actually is (or will be). But I'm planning on keeping them lumped together at the API level until I find a compelling reason to separate them.

[kzinti]

You don't need 2 modes. As Brendan explained, it is a legacy thing.

[Ready4Dis]

For 3D applications (e.g. CAD, etc) there's no sane reason for it to be any different to 3D games. For 2D applications there's no sane reason for it to be any different to 2D games.

From a purely design point of view, I agree. However, in the real world, "applications" and "games" have slightly different "priorities", mainly centered around partial screen updates vs. whole screen updates, and CPU usage.

For example, most games have no need for a font rendering engine, and most applications have no need to "upload" custom data and/or code to the video card.

These differences "may" be significant enough to justify a 2-mode system in order to optimize both use cases, but I've barely scratched the surface on both of these systems, so I don't know how necessary a 2-mode system actually is (or will be). But I'm planning on keeping them lumped together at the API level until I find a compelling reason to separate them.

I agree with Brendan, I see no reason to split the two. How do you "optimize" both cases, when both cases require the same thing? They draw stuff to a surface. One might update fast, the other might update slow, who cares? If one is constantly waiting on user input, it shouldn't be using the CPU or GPU... why do you need to optimize a task that is sleeping? If a game is running, it will be using the cpu/gpu consistantly. If it's over-top of other windows, they shouldn't even try to be updating anyways, unless you want to support alpha blending of windows over other windows. If you go full screen, the other apps are background apps and the top one gets priority... just like if it was a window and was on-top. What's the difference?

[Rusky]

However, in the real world, "applications" and "games" have slightly different "priorities", mainly centered around partial screen updates vs. whole screen updates, and CPU usage.

For example, most games have no need for a font rendering engine, and most applications have no need to "upload" custom data and/or code to the video card.

Today, applications are typically rendered very similarly to games- compositing window managers don't do partial screen updates, they just keep each window in a separate framebuffer. Widget toolkits have started to use hardware acceleration to render UI so they do need to upload data to the video card.

[SpyderTL]

Today, applications are typically rendered very similarly to games- compositing window managers don't do partial screen updates, they just keep each window in a separate framebuffer. Widget toolkits have started to use hardware acceleration to render UI so they do need to upload data to the video card.

True, but I was speaking more to the API that is presented to the application/game. You wouldn't expect a word processing application to "pre-load" font data to the video card before rendering characters to the screen, but for games, this is quite common. Forcing games to use the same framework as applications will make life more difficult for one, or the other, or both.

If you are going to (like me) try to create one all-encompassing 2d/3d game/application framework, I think you'll just end up with 1 API with twice as many methods. Which is fine by me, but I just wanted to point out that applications and games have "evolved" to use two different graphics paradigms, due to different needs and concerns. And I haven't seen many (or any) implementations that combine both usages into a single endpoint, so it's hard to say whether that would end up being a "good" solution to both problems, or if it would just make things more difficult for everyone.

I hope to have a single endpoint solution at some point in the next year or two, and if so, I'll have a better feel for whether it was a good idea or not. Likewise, if someone else pulls it off, I'd be interested in seeing it.

EDIT: Now that I think about it, I have seen one framework that (at least, superficially) combines 2d and 3d rendering elements in the same structure. Windows Presentation Foundation / Silverlight. While I don't believe the performance is quite on par with a pure Direct2D / Direct3D solution, I believe it is sufficient for casual games, and, of course, applications. I should probably throw Adobe Flash, and HTML 5 / CSS 3 in there as well, although I'm not as familiar with either of those implementations.

[Brendan]

Hi,

Today, applications are typically rendered very similarly to games- compositing window managers don't do partial screen updates, they just keep each window in a separate framebuffer. Widget toolkits have started to use hardware acceleration to render UI so they do need to upload data to the video card.

True, but I was speaking more to the API that is presented to the application/game. You wouldn't expect a word processing application to "pre-load" font data to the video card before rendering characters to the screen, but for games, this is quite common. Forcing games to use the same framework as applications will make life more difficult for one, or the other, or both.

For games, it's quite common to preload textures (used for fonts) because the API given to games is crap.

Software (games, applications, GUIs, etc) should be able to say (in a description of a texture's contents) that the texture includes one or more UTF-8 strings (using a certain font at a certain size, possibly using a certain layout engine, possibly with bold/italic/colour/whatever markup, etc). The video driver should cooperate with the layout engine and/or font engine to generate the texture (then cache it). There shouldn't be much/any real difference between a texture used for the majority of a wordprocessor's window, or small textures used for a bitmap image editor's menus, or a medium sized texture used for the players inventory in a game like Skyrim.

Of course this doesn't prevent games from doing it themselves (with their own textures, etc) if they actually want to (and aren't just doing it to work-around how hideously lame the graphics API happens to be).

If you are going to (like me) try to create one all-encompassing 2d/3d game/application framework, I think you'll just end up with 1 API with twice as many methods. Which is fine by me, but I just wanted to point out that applications and games have "evolved" to use two different graphics paradigms, due to different needs and concerns. And I haven't seen many (or any) implementations that combine both usages into a single endpoint, so it's hard to say whether that would end up being a "good" solution to both problems, or if it would just make things more difficult for everyone.

I hope to have a single endpoint solution at some point in the next year or two, and if so, I'll have a better feel for whether it was a good idea or not. Likewise, if someone else pulls it off, I'd be interested in seeing it.

Sort of, but not really.

There will always be things that some software uses that other software doesn't; but you can't (e.g.) look at a CAD application and decide that applications will be the only thing that ever uses 3D, and you can't look at (e.g.) a game like dwarf fortress and decide that only games with 3D worlds will use "text only".


Cheers,

Brendan

[Rusky]

True, but I was speaking more to the API that is presented to the application/game. You wouldn't expect a word processing application to "pre-load" font data to the video card before rendering characters to the screen, but for games, this is quite common. Forcing games to use the same framework as applications will make life more difficult for one, or the other, or both.

This is where API layering comes in. You have your API for drawing directly to framebuffers, which will likely be used primarily for games, but you also have APIs on top that do things like drawing text or widgets. Like Brendan said, it's not a big deal if some parts of the API only get used for some applications. The key is to keep the API usable for application developers and maintainable for you- which is much easier with a modern hardware accelerated UI pipeline that can share the lower level API with games.

[AlexHully]

It will be easier to remove a lot of things in x11 and overriding the calls with customized and specific code i guess to match a tiny wayland with no sockets involved at all.

I only have to support few cards.