GUI design from the API perspective

[mystran]

I've been writing a portable GUI toolkit recently (only X11 for now though) and I though I'd share some thoughs I've got from the toolkit developers perspective, trying to deal with GUI APIs...

This is probably quite basic, but I thought I'd share with you what my toolkit needs. Those who are going to develop their first GUI can consider this a checklist.

There are two main with a GUI. One is the ability to display windows, graphics and text. The second is to get events when user does something.

As for drawing graphics, it might be that things like pens, with sophisticated routines for drawing all kinds of things were useful in the past, but I think today, one will get very far from supporting the following primitives: putpixel (with optional alpha channel), line drawing, rentagle (or even polygon) fills, and bitmaps. Personally I find this set needed most, so providing those hardware accelerated is a good idea.

I'd support transparent conversion from 32-bit RGB model to whatever the screen can display. Most applications just want the best that can be displayed and coding support for every kind of colormodel into apps is rather painful.

Since skinned widgets are pretty much a standard by now, hardware accelerated bitmaps and bitmap scaling are very useful. Even without alpha channel they are useful, but many nice theming tricks need alpha channel in bitmaps, so why not?

One thing with bitmap-scaling that is really nice for skinning widgets is being able to specify "margins" that aren't scaled. If you have a button, you don't want to scale it's horizontal edges vertically, or vertical edges horizontally. This way you don't necessarily need to provide the corners and edges in different bitmaps, although it can be done internally. Finally, at least bilinear interpolation is needed to get good looking widgets by scaling bitmaps. Bicubic is nice, but without hardware support it's a bit slow. Tiled bitmaps are also nice, but not nearly as useful as scaling.

For font rendering, I'd say that being able to query ascend/descend/line-height and width of given string for any font is pretty much enough to do decent layout. Since everybody probably uses freetype anyway, I'll just mention that since people are going to want unicode, builtin text-rendering without unicode support is pretty useless. Bidirectional text on the other hand is IMHO better handled in application specific layout code. Text drawing should just make it possible..

This is all nice, and isn't that hard to implement totally in the userspace (after you just have a window) with nothing but a basic putpixel. What's more interesting is the event handling..

X11 (which I've been playing with a lot recently) provides all kinds of events, which can be selected. Most other systems probably have a similar set of events, but here's what I find most important:

Mouse enters/leaves a window, moves in a window, or a mouse button is pressed/released in a window. A reasonable default is that if a button is pressed in one window, all subsequent mouse events before the release of last button go to the same window. Mouse handling isn't that hard after all.

Keyboard on the other hand is another beast. Being able to tell when a window gets/loses keyboard focus is nice (so you know when to blink the cursor and such). As for the actual keypresses/releases there are two things: first one will usually want to be able to get presses/releases of specific keys (like arrows, tab, function keys, space) to implement all kinds of stuff. Personally I much prefer getting the base key and modifier separately (unlike KeySyms in X11) since in some types of applications you want to be able to react to a specific key regardless of the modifier state. Games are one example, but there are others as well.

But this really gets us nowhere, since we want to read text typed by the user, and no, dealing with all that in user applications is NOT a solution. Any half-decent window-system and toolkit provides separate events for that. Those events take care of stuff like keyboard layouts, composing characters (like so called dead-keys) and more sophisticated input methods like those used for CJK text. So you need something like WM_CHAR in Windows, or KeyTyped in Java, which provides you with characters (preferably in unicode again) which the user has typed in. There need not be strict relationship between the press/release events and the character typed events. Any method of inputting text should generate the character events. It makes it MUCH easier to write applications that actually have a chance working with different input styles used in different parts of the world.

[mystran]

Finally, if there's absolutely no way we can get the maximum post length on this board to be at least doubled, can we at least get a more informative error like "Please shorten your message by # characters" so we know what's possible and what must be split into two posts?

[mystran]

Oops, I forgot one important event: Expose. Unless you keep complete backing store for each window, it's nice to know what regions to redraw and when

edit: To keep the list complete, I also have to mention that I also forgot resize events, but then again, everyone would probably implement that anyway

[distantvoices]

hmmm ...

I'd only save what's hidden by the active window. Upon moving the active window, no redrawing should be necessary. *gg*

some hints from practizing that stuff: have your widgets be classes. That's easy to implement once one's buildt the necessary framework, and - have those widget classes initialize themselves. Afterall, the widget knows how it looks like. No need for the windowing system to know this.

(me should split the huge gui service file into several small ones. it's tedious to debug the stuff and search for functions if everything is put in one single source file. Me is a lazy dog. Shame on me ... and maybe it's worth to have some threads inside gui service: one for the redrawing stuff f. ex. (all queues protected with semaphores))

stay safe

[Candy]

BI, that slows realtime drawing to a background window (which you can easily support) because you have to check for each pixel whether it is on the display or on the buffer. You can buffer the entire screen (which with the current default large windows doesn't really matter) without much waste.

[mystran]

Indeed, I would have coded my toolkit in C, but I wanted to use C++ just to get a decent object system. Plus templates are nice for a signals/slots system

Except for having a totally different way of handling layouts, and more simplistic event model, my toolkit is somewhat similar in design to Swing. Component and Container are the two classes that handle most of the stuff. You just ask for the driver to give you a Window, after which you can put components/containers into that window, and other components within containers.. and so on..

As for the event system, each class handles the events that widget is interested in directly. Any basic events are virtual methods of the base component class. Widgets then emit a signal when something interesting happens, which you can connect to any (suitably typed) method of any class that happens to inherit from "Slots" base-class (which is there mainly to handle automatic disconnection).

[Legend]

I think being able to code an app in 32 bit RGBA color format is good, but I think it should be possible to let the app do the drawing in other formats (like one the screen really uses) for speed, too ...

[distantvoices]

@candy: hmmm ... right you are. Thanks for showing this up.

I've forgotten, that I maintain a whole buffer for the whole screen in order to do the window moving stuff properly. So, redrawing a background window is just an operation of redrawing the affected region (currently: after copying stuff and thats plain bad and ugly design). I just need to get rid of several copying-stuff-hither-and-tither operations which are slowing down my system.

@Legend: what kind of drawing? drawing on a canvas: aye, drawing controls: nay, that's the servers task for it knows best how a control looks like. Loading a picture? I'd have the application *load and decode the stuff* but put the decoded image data onto a widget (like canvas or - more specialized - picture control) with a set_content() method. The application shall not directly mess with the picture. Submit location of a buffer and shall the server do the drawing of the picture.

stay safe

[mystran]

I think being able to code an app in 32 bit RGBA color format is good, but I think it should be possible to let the app do the drawing in other formats (like one the screen really uses) for speed, too ...

Definitely. It should be able to use whatever color-system there is, but being able to naively assume basic RGB space is nice.

Actually, with most things I think there should always be the "easy way" and the "proper way", even if the "easy way" was implemented as an emulation layer in whatever interface libraries you have.

[Pype.Clicker]

regarding the keyboard, most of the time what the user sees are *controls*. e.g. It asks for a text box and tell where data should be written when entered. Of course there's a need for catch-all-keystrokes control, but most of the time, you want the GUI to handle backspace, arrows, etc. without bothering the App.

Trouble is that, from time to time, you'd like to prevent the user to type letters in a dial-up box ... A simple regular expression can make the trick.

[mystran]

regarding the keyboard, most of the time what the user sees are *controls*. e.g. It asks for a text box and tell where data should be written when entered. Of course there's a need for catch-all-keystrokes control, but most of the time, you want the GUI to handle backspace, arrows, etc. without bothering the App.

Trouble is that, from time to time, you'd like to prevent the user to type letters in a dial-up box ... A simple regular expression can make the trick.

Actually, while standard controls are nice, I object against the idea that there is a set of common controls that handle stuff like keyboard input, and then there's something like canvas, which only gets raw data, and you have to implement everything yourself.

While in many cases one only wants some standard controls, the need to derive new ones is NOT rare. If it's not possible to reuse functionality and only add what you need to, it makes building new widgets awfully lot of work.

Actually, the whole idea that standard widgets handle most of the messy issues is what I hate in many existing toolkits and systems. Ofcourse text-editing and such is an issue for the standard text-input widget, but typing in characters is DEFINITELY NOT.

Consider that you are an application developer, wanting to develop a new word processor (or WYSIWYG editor or whatever) for the OS, and all input-method support in the OS is implemented on the widget level. If you want to support all kinds of international input methods, you have to duplicate all the effort taken by the OS developers, or you have to build some kludge around the text-input widget, giving it those keyboard events you don't use internally, and reading stuff back from the widget, the putting it to your wysiwyg-structures and drawing to screen. Hardly elegant. What's worse, even if you could derive from the text-widget directly, there's now way you are going to know what events the input methods might need. Some input methods only need keyboard pressed, some need mouse, some need to draw stuff (to give composition feedback) and such.

Whatever the input method logic, it definitely is NOT an issue of any widget, at least not beyond the feedback-drawing part. Honestly, the best thing to do is split the whole text-typing issue from the regular keyboard input mechanism, and give them to the application as two different types of events. There are enough toolkits where it's impossible to write well internatinalized applications where the needs of the application goes beyond what's provided by the basic widgets!

[Pype.Clicker]

hm, sorry if i sounded that evil in my suggestion. Its true that my yet-to-come GUI design massively relies on execution of application-issued scripts at the server.

But indeed i still have to confront it to ultra-complex frames like wysiwyg text editors ... Now, i guess having the choice between raw (keystrokes only) and 'cooked' (with an UTF_CHARACTER event and various events for special characters like CTL_ENTER, CTL_BACKSPACE etc) mode for receiving information at the application is sufficient, no ?

What i was actually suggesting is the option of sending a script that will ease things like single-line textboxes with

Code: Select all

    content {
        textLabel($before) @east;
        BlinkingCusor @east;
        textLabel($after) @east;
    }
    on CTL_BACKSPACE:
        $before=substr($before, 0,length($before)-1);
        emit REDRAW_ME;
    on CTL_LEFTARROW:
        $after=$before[-1] + $after;
        $before=substr($before, 0,length($before)-1);
        emit REDRAW_ME;
    on CTL_RIGHT_ARROW:
        $before=$before+$after[0];
        $after = substr($after,1,length($after));
        emit REDRAW_ME;