Pixel perfect click tracking

[AndrewAPrice]

Currently, everything in my UI toolkit is rectangular, and it's easy to figure out the UI element under the mouse pointer because I can just test if the mouse pointer falls within the bounding box of the widget, e.g.:

Code: Select all

x >= minX && x <= maxX && y >= minY && y <= maxY

I'm interested in doing something less boxy, and potentially using a drawing library such as Cairo or Skia to draw my UI widgets, and giving my buttons curved corner, my tabs slanted edges, etc.

But testing if I'm clicking on an element becomes more complicated. How do people solve this?

Some ideas I've had:

1) Custom math for every widget. e.g. For a button with curved borders, I could do a basic bounding box test, but then if I fall within the 4 corners where it could be curved, do extra math (distance from the quarter-circle's center point to see that we fall within the curve radius.)

2) Don't care about pixel perfection and just use a rectangular bounding box around everything. Might not be noticable with slight curved edges, although round buttons would be obvious.

3) Have a secondary "click target" buffer, where each pixel isn't a color but a widget index (which could be a pointer), and anytime I draw a widget that is clickable, I make sure to draw the widget in the same shape (with anti-aliasing turned off) to the "click target" buffer in a solid color, where the color is the widget ID. This would give me super fast lookup of whatever is under any pixel on the screen (I'd support arbitrary polygons), but at the expense of more memory and slower drawing.

[nullplan]

Most display libraries choose option 2. The only relatively quick solution I would have for pixel perfection is a quadtree, but with rounded corners that might become very deep at those corners. And all for the minuscule chance that the user clicked one pixel next to a rounded corner and did not actually want to click the button. I mean, if there is a different clickable widget in those rounded corners, the UI designer needs some reeducation.

[Ringding]

Option 4)

Similar to option 3, but make the render routine respond to a flag that specifies if it’s supposed to render normally or to the hit buffer.

[Demindiro]

I'd go with 2) but with pixel perfection.

You first do an AABB test to see what the cursor intersects with, which is O(whatever). Then you map the cursor to the widget's space and check if the pixel under the cursor is transparent or not, which is O(1).

[AndrewAPrice]

I'd go with 2) but with pixel perfection.

You first do an AABB test to see what the cursor intersects with, which is O(whatever). Then you map the cursor to the widget's space and check if the pixel under the cursor is transparent or not, which is O(1).

The part about testing if a pixel is transparent would require a buffer per widget (or at least non-overlapping set of widgets) correct?

You've inspired me to do a combination. I'll do an AABB test, but specific widgets are allowed to override this and do a more specific test.

[Demindiro]

The part about testing if a pixel is transparent would require a buffer per widget (or at least non-overlapping set of widgets) correct?

Depends on the type of widget. For e.g. widgets that show a single icon you only need to keep one copy of that icon in memory.

[eekee]

I'm pretty sure X11's shaped-window extension tests against a series of connected lines, making it a variant of option 1. The lists seemed to me to be very long, even for something simple like xeyes which had only 2 ovals, but it was very quick even on a 486. I might be wrong about the connected lines, it might have been loads of rectangles instead. But all this was without alpha blending.

Alpha blending introduces a UI issue: at what alpha threshold do you pass the click through to the underlying widget? If you pick a value based on what you see on your screen, someone else might see it very differently if their screen has a different gamma. Without drivers supporting color profiles for different monitors, this will happen. I don't have a good technical answer to this. I think the best idea is to remember this issue at the art stage: don't make long gradual alpha fades. Or, if your fades are procedurally generated, maybe have an option to change the gamma value used in generating them.

[AndrewAPrice]

Alpha blending introduces a UI issue: at what alpha threshold do you pass the click through to the underlying widget? If you pick a value based on what you see on your screen, someone else might see it very differently if their screen has a different gamma. Without drivers supporting color profiles for different monitors, this will happen. I don't have a good technical answer to this. I think the best idea is to remember this issue at the art stage: don't make long gradual alpha fades. Or, if your fades are procedurally generated, maybe have an option to change the gamma value used in generating them.

I think the answer to this is a design question. Only use standard alpha blending for things like shadows where you can click behind the object. Clickable transparent surfaces should do something a little more advance than basic alpha blending, such as blurring the background. Windows calls this "acrylic":

[eekee]

I think the answer to this is a design question. Only use standard alpha blending for things like shadows where you can click behind the object. Clickable transparent surfaces should do something a little more advance than basic alpha blending, such as blurring the background. Windows calls this "acrylic":

I think you're right. I can't see any other answer than a design question, and this choice of when to blur works very well.

[klange]

Note that there was some discussion about just that topic recently.

While I conflate window alpha transparency with clickability in my compositor, I offer applications the ability to set the threshold opacity value at which pixels become clickable. That same threshold is also used to determine whether blurring should be applied if it's enabled, so Andrew's suggested behavior holds true in Yutani at least for the cases where blurring is enabled at all. Unfortunately, blurring large areas correctly is a rather expensive task in a hobby OS without reasonable access to pixel shaders, so I made it opt-in and limited it to a handful of controlled situations by default...