Graphics in userspace

[grimpy]

Hi,

I have written a tiny kernel and a driver for my graphics card, as well as a few routines for drawing lines, bitmaps and so on.

I am going to add support for user space processes soon (I only have kernel threads now) and wonder how I should organize access to graphics.

Should I just expose the line, bitmap, etc functions in user space and have them work directly on the framebuffer? Adding system calls for each kind of graphic operation seems rather slow

So basically, the frame buffer would be mapped in each user space and they used a OS graphics library (which I would write) linked in each application. The library does the drawing directly on the framebuffer (or directly uses accelerated functions when available, but on some cards, that requires port i/o access).

The downside I see is that a few megabytes worth of virtual space would be needed from all graphical apps. Maybe there are security issues too (like apps beeing able to look at and potentially interpret output from other apps, but - hey - doing screen captures programmatically is possible in all OS's I've been working on anyway

Ideas and suggestions will be greatly appreciated!

Best regards,

Gri

[Combuster]

There are two common approaches to this:
- you map video memory directly to userspace (linux)
- you buffer calls and send them to the kernel in batches (vista)

[mrkaktus]

Make it better:

Create second buffer that is so big like your screen (framebuffer).
Then you are sharing this second buffer to all apps one by one.
You as a guihost know size and position of each app window you are
giving second buffer to work on. So when app is rewriting its window
in second buffer you are copying only area where should appear apps
window to framebuffer. THis make's you to copy each window one by
one to refresh screen (-) but copying even 20 windows is nothing for
today PC's and you gain nice effect of protection if some app is crashed
(+) because then this app won't destroy your desktop look but only its
window will lok strange .

[Happy Egghead]

Hi, Its still early days but I'm running all my video / graphics in user space.
Basically I create a graphics user task (GUT) and map the physcial frame buffer into that.
As each App starts, it requests a buffer of a certain size (X x Y). This buffer is mapped into the GUT and the App as shared memory.
As a result the GUT contains a physical frame buffer and access to all buffers created by apps (up to about 3Gbytes worth).
Applications can either use the (currently very basic) functions of the GUT or have the option of writing directly to the buffer or buffers they own themselves.
Every time the GUT task is run, it checks all the frame buffers and copies what needs to change into the physical frame buffer. I get some 'tearing' and I think that's because it's not 'double buffered' . Because it currently depends on the GUT task running to update, sometimes the updates don't. Again adding a thread to fire off every 60 to 120hz depending on the refresh rate should sync everything up.

The great advantage of this scheme is that all of the graphics driver is in user space with the exception of the mode switch function (and that currently only supports 1 mode - 320x200). I've been thinking of switching it to a 32-bit vm86 app, that way I can use the I/O protection map to allow access to the video ports and nothing else. This would allow me to move everything into user space and apart from memory allocation make it bulletproof.

My basic aims are to
provide the basic video services, not restricting an App to a particular API,
provide adequete speed for display of information,
make the GUI as secure as possible.

In the future it shouldn't be too hard to add multiple screens just by running multiple GUT tasks. Or perhaps having multiple GUT's each one supporting a different API.
If you haven't noticed there is no support for hardware acceleration mainly due to the fact that there's little documentation. I do have a Cirrus logic and a Tseng ET6000 card so I'll probably support them with their own specific GUT's.

But that's in the future, something called 'a job' keeps keeping me away
Hope all goes well with your OSdev!

[AndrewAPrice]

Create second buffer that is so big like your screen (framebuffer).

That's how Windows does it. Each window has a frame buffer the same resolution as the screen. You can see this for yourself if you stretch a window as big as you can, then drag it off to the side, and stretch it more.

[grimpy]

Thanks for all the suggestions!

Because it currently depends on the GUT task running to update, sometimes the updates don't. Again adding a thread to fire off every 60 to 120hz depending on the refresh rate should sync everything up.

Windows requires the app to invalidate manually, to prevent refresh threads? Doesn't that work out nicely?

[Happy Egghead]

At the moment the Graphics User Task (GUT) just copies the portion of each buffer 'visible' into the physical frame buffer. Letting apps 'hint' that their window has changed felt like too much overhead. If a buffer isn't viewable, it is never drawn, the buffer writing is fully skipped.
In a perfect scenario (at the current 320x200 resolution) at most 64k has to be written into the frame buffer everytime the buffer is updated. This includes the background (black!) and all viewable buffer sections. As a result, assuming linear frame buffering, whenever I decide to add higher resolutions the amount of data will increase (which is why graphics accelerators were invented).

Thinking about it, it should be easy to allow the GUT code to set a flag saying "This buffer area has changed" thus allowing the buffer copying code to redraw updated sections, while other buffers with the flag not set would not be copied as they would be up to date.

One thing I dislike about Windows programming is that you constantly have to tell Windows "I've just done something". The interface should be smart enough to figure out if the buffer 'IT' just wrote to should be updated.
I want to avoid implementing any kind of 'manual' invalidation unless it becomes absolutely necessary.

[Candy]

One thing I dislike about Windows programming is that you constantly have to tell Windows "I've just done something". The interface should be smart enough to figure out if the buffer 'IT' just wrote to should be updated.
I want to avoid implementing any kind of 'manual' invalidation unless it becomes absolutely necessary.

So that's like polling then. The mouse shouldn't need to tell you it's moved, you should figure that out yourself. The network card needn't interrupt your work because it's a friggin 1GBit card with a 10kbit buffer, you can just look at it when you like.

Did you do the maths for say, 5120*1600 ? Dual 30" screen?

[Happy Egghead]

Actually my point was that is appears that when Windows writes into a buffer using the GDI it doesn't seem to understand that it should update that section the next time the screen is put together. I want my code to do this to avoid unecessary messages from the application to the system saying hey "I'm letting you know You did something but were too stupid to update yourself".

However at the moment I haven't implemented the part of my GUI code that gets called by a timer or somesuch to update at a certain FPS, so the screen only gets updated when the GUI task is run. Apps should not have to tell the GUI task if the GUI task has updated a buffer because the GUI task should know!

Devices ARE interrupt driven and messages are passed onto Apps when necessary. The graphics card is one of the few devices that doesn't need polling unless you are using a CGA card and want to avoid the snow or you can somehow snarf the Verticle Retrace Interrupt to do the updates (virtually impossible on older hardware).

As for

5120*1600 ? Dual 30" screen?

my previous post did say that at higher resolutions the simple buffer copying scheme will be slow but only to the areas of the buffers that need updating. So 5 screens (320x200) isn't (5x64k) it will only ever be 64k *Maximum* that needs to be copied into the frame buffer.

The same problems happened with every other OS until the event of graphics accelerators and local bus interfaces. I won't be supporting massive resolutions unless I can support these accelerators and since there's no hope in hell of getting access to the specs or the time to actually write anything like that I'll stick to low-res bitmaps with really bad 8x8 fonts.

[Brynet-Inc]

The same problems happened with every other OS until the event of graphics accelerators and local bus interfaces. I won't be supporting massive resolutions unless I can support these accelerators and since there's no hope in hell of getting access to the specs or the time to actually write anything like that I'll stick to low-res bitmaps with really bad 8x8 fonts.

2D accelerated features could be supported on a select few cards and chipsets, X11 for example contains some (Obfuscated C drivers..) under a fairly decent licence.

A few other projects (If you're using the GPL) could shed some light on a few cards.. I'm thinking Syllable/Haiku and something called directfb.

Ignore this if I don't know what I'm talking about

[Happy Egghead]

Thanks for the tips, at the moment I'm not even thinking about licensing because I get embarassed by my own programming (even if it does occasionally work).
'directfb' looks interesting, I like to trawl through code to see how someone else has solved various problems. The hardest part is wrapping my head around the *nix way of doing things because none of my current code maps to anything like 'standard' system calls. Everything helps.

It's nice to have a good display but if the dang floppy drive still throws errors and refuses to load anything into memory then that's the priority at the moment. When the basics are reliable, I'll start looking at getting some kind of VESA resolution going. That said I have PCI versions of a Tseng, a Cirrus 5430, a tnt 2 and couple of S3's lying around that I can try some very basic 2D stuff on whenever I get around to it.

It's only the last two weeks that I've had any time to think about anything seriously again. I'm getting good at writing letters to myself in the future. Sometimes I remember to read them.

[distantvoices]

@happy egghead:

I think, that you, while having gotten the Knack of some Stuff, have missed to get the point about the gui service - gui client thing.

You know: it is a gui SERVICE, which means, that it is not to do stuff out of it's own but upon request.

So, I'd have the client draw its stuff and then tell the gui service: Hey, lazy hog, update the screen and here is the area to update (which usually contains the client's window or an area within) - simply a sync(win,x1,y1,x2,y2) message and here you go. way better than having the gui service wake up periodically and spending time updating stuff which doesn't need to be updated.

The more, the gui service is completely unaware of what the app is doing to the buffer except of the occasional sync() messages. It's not the gui service's task to know.

Why is this more complicated than doing this having to wait till the gui service wakes up for getting a refreshed screen thing?

Sorry ... I'm just a silly IT-Engineer and do not know what the hell I'm talking about. So you may as well elucidate my NotKNowing.

stay safe.

[Brendan]

Hi,

However at the moment I haven't implemented the part of my GUI code that gets called by a timer or somesuch to update at a certain FPS, so the screen only gets updated when the GUI task is run. Apps should not have to tell the GUI task if the GUI task has updated a buffer because the GUI task should know!

How would the GUT know anything has changed? Would it waste CPU time refreshing the video every N ms when nothing has changed?

Even if you solve this problem, you wouldn't know if the client has finished updating it's buffer or if it's only done some of the changes. This would lead to ugly shearing and tearing effects. For a simple example, imagine if the client draws a large green rectange in one frame, then changes the colour of the rectange to blue for the next frame - the user might not see the green rectangle at all, or they might see a rectangle that's half green and half blue. Now consider an animated ball that bounces around, where the ball might be seen in 2 places at the same time in one frame and be partially invisible in the next.

Lastly, the idea of using a "client buffer" doesn't work well for hardware acceleration. It means the GUT can use a hardware bitblit to transfer the client buffer to the video screen, but that (and perhaps hardware mouse pointers) is probably the only hardware acceleration you can hope to use.

For example, if a client wants to draw a green rectange it has to do it manually, and can't just tell the video driver to draw a green rectange (and let the video driver use hardware acceleration in the video card to draw the rectangle). The client couldn't upload 3 icons into the video card's memory and ask the hardware to blit each icon to 10 different places - it'd have to do that manually in RAM. The client also can't tell the video hardware to draw thousands of 3D textured polygons with lighting and fog - it'd have to do that manually too.

For most (all?) modern video systems the client sends commands that describe a complete frame, and these commands are (eventually) processed by the video driver. For my OS this will mean the client builds a script of commands and sends the entire script as a message.

For OpenGL the client calls OpenGL functions (which could build a list of commands and send those commands to an OpenGL driver when the client calls the "glEnd()" function, or could send each command to the OpenGL driver one at a time - it makes no real difference because nothing is made visible until the client calls the "glEnd()" function).

For X Windows, there's a client/server relationship where the client and server may be on seperate machines. This is done using sockets where the client sends video commands down the socket to the server (and the server sends keyboard/mouse data back). X is shifting to OpenGL for video (where the client sends OpenGL commands down the socket).

AFAIK DirectX (or Direct3D to be more accurate) works the same as OpenGL, although I've never used it. Microsoft also have an interface called GDI which uses a framebuffer, but it dates back to 16-bit windows (before hardware acceleration existed). For an interesting overview of why GDI sucks see this page (most of the problems would apply to any framebuffer interface).

Before designing the video interface for your OS, try to write an OpenGL application. Start with the tutorials on this web site. Once you've got a rough idea of how a decent video system works you'll be in a much better position to design your own video system....


Cheers,

Brendan

[Happy Egghead]

My current 'interface design' is nothing the way I'd like it to be in the future as I really want to get the basics working before focussing fully on an interface that is firendly or even looks good.

My current thought for design is - code to handle the drawing and a task that goes through buffers copying the info that needs to be rewritten into the frame buffer.

The 'graphics code' is called by apps when needed to draw something unless apps write directly into a buffer. This would be the equivalent of GDI/DirectX/OpenGL.
The 'graphics task' is called by the task switching mechanism to perform the physical act of updating the screen.
Both of these are combined into a single 'Graphical User Task'.

The 'graphics code' when called by apps, marks the buffer that it needs to be updated. If there is nothing to be updated the blitter can 'skip the blit'. Thus saving time (the app sends no message and the blit isn't done). When a buffer is created, it can be set to 'always update' allowing applications that write directly to the buffer to do so at the fastest speed they can(subject to the displays refresh speed).

The 'graphics task' should really be split into two with one part coming up with some kind of display list and copying data into the frame buffer, and the second task called by an interrupt (based on a refresh interrupt) to copy the 'double buffer' at 'X'hz into the real frame buffer.


@distantvoices:

Why is this more complicated than doing this having to wait till the gui service wakes up for getting a refreshed screen thing?

Although I can understand the way the 'update now please' message system works, I'm lazy. This box of tricks should be doing the work for me. It's really really hard writing an extra three lines of code to ask the GUI to refresh the window, when with an extra three lines in the 'graphics code' to set a flag in a buffer - I'll never have to do it at all. If a screen area doesn't need to be updated because it can't be seen, it will never be updated - just the bits around it. If a buffer takes up the entire screen area and never gets written to, the 'graphics task' basically checks the list and yields() using no CPU time nor wasting memory bandwidth.

It really doesn't matter at this stage that there isn't a split between 'graphics code' and 'video driver'. After all, everytime I install a new video driver on Windows I also have to reinstall DirectX so that it can figure out what things it can accerate or not. Lumping it all together means a single file guaranteed to work and as long as the data structures in the buffers are compatible, should ease upgrading (hopefully on the fly).

There is a client/server relationship going (that can be bypassed if necessary) but my driver design thoughts came from the 'interrupt'/'sheduled work' type of thing.
Do what you can now (interrupt/function call/graphics API/drawing/moving/filling/copying)
and
Delay what can be delayed (buffer combining/screen updating/sheduled work).

Don't worry about those IT qualifications. I work as a debt collector.

@Brendan too:

How would the GUT know anything has changed? Would it waste CPU time refreshing the video every N ms when nothing has changed?
Even if you solve this problem, you wouldn't know if the client has finished updating it's buffer or if it's only done some of the changes. This would lead to ugly shearing and tearing effects. For a simple example, imagine if the client draws a large green rectange in one frame, then changes the colour of the rectange to blue for the next frame - the user might not see the green rectangle at all, or they might see a rectangle that's half green and half blue. Now consider an animated ball that bounces around, where the ball might be seen in 2 places at the same time in one frame and be partially invisible in the next.

The problems I'm seeing I think are to do with the fact that I don't have that seperate refresh task for 'direct buffer writes' so things get lost unless I draw using the 'graphics code' which ironically acts like a 'please refresh message service' distantvoices and candy were mentioning. The fact that the 'graphics task' doesn't have double buffering results in the horrible shear / dual colour mentioned. I've seen this happen when working with software sprites. The buffering is definitely needed and whenever the time is available to sit and add the code I will.

As for hardware acceleration, I intend to only write for the computers I have (including the emulated ones). At the moment everything is software, but as mentioned the only acceleration useable is 2D. That will be fine for the moment, but there's nothing to prevent the addition of a 3d display list parser to the 'graphics code'. I've recently been trawling through 'Black art of 3d game programming by Andrew LaMoth', it's based in DOS, but develops a software 3D graphics pipeline. It shouldn't be too hard to add some code to parse 'vector buffers' so the graphics task can work on those when it has the time. But again that's all in the future.



Another problem at the moment is that everything is tacked onto the kernel. Once I get the floppy driver fully working, I have to get the graphics task off of disk and into memory properly. At that point it will not only be running at user privileges but also in user memory (below the 0xC0000000) mark. Lot's more bugfinding fun.


I will keep an open mind about other methods, but for the moment I intend to persist with the current design just to see if it will do what I want it to. If it all collapses in a heap, then I'll consider it a learning experience and rip it out. If it does work I'll happily post a screen shot in one of those 'what does your OS look like threads' that turn up every 6 months or so.