Flashing/flickering see-through effect when moving windows

[Octacone]

Hello,

I'm working on a compositing window manager and I've been able to put together a primitive (yet effective) dirty rectangle system. In order to avoid unnecessary screen updates.
Each window has its own buffer and they all get composited together. Windows are stored and ordered by the z-index in a linked list.
I still update too much (since I haven't yet figured out how to not render areas that are not visible, but that is another topic).

Now, what happens is, when I move a window over a background everything is fine, but when I move it over other windows, I see unexplained artifacts.
The window being moved moves but I can also see the windows underneath flashing until I stop dragging.
As if I can see the windows being rendered one by one. It is hard to explain.

[Octocontrabass]

You're rendering everything to a single hardware buffer, and your rendering is either not synchronized to vblank or takes longer than vblank. This means partial renders will sometimes be visible. And...

I haven't yet figured out how to not render areas that are not visible

...your partial renders include windows that aren't supposed to be visible.

[Octacone]

You're rendering everything to a single hardware buffer, and your rendering is either not synchronized to vblank or takes longer than vblank. This means partial renders will sometimes be visible. And...

I haven't yet figured out how to not render areas that are not visible

...your partial renders include windows that aren't supposed to be visible.

Yeah, the things I'm seeing are the things that shouldn't even be rendered since they are obscured.

What's the best way to deal with this?
I already have a function that can split two windows and return either 1, 2, 3 or 4 new rectangles, both positive and negative.
Should I just split each window by each other window and each of their rectangles by every other rectangle? That seems super slow.

[Crono]

Should I just split each window by each other window and each of their rectangles by every other rectangle? That seems super slow.

Exactly, its super fast comparing to number of pixels you have to extra draw. There are 2 approaches. 1) Build a grid of all rects with z-order in mind & parse the grid building final rects. This is fastest when no alpha blending involved.2) Clip every single rect against every other rect while doing alpha blending when rects intersect.

[Octacone]

Should I just split each window by each other window and each of their rectangles by every other rectangle? That seems super slow.

Exactly, its super fast comparing to number of pixels you have to extra draw. There are 2 approaches. 1) Build a grid of all rects with z-order in mind & parse the grid building final rects. This is fastest when no alpha blending involved.2) Clip every single rect against every other rect while doing alpha blending when rects intersect.

My plan is to support transparency eventually, so I'll have to go with option 2.
I didn't know doing tons of comparisons was faster than writing to memory. I'm gonna start writing it right away, will report back.

Edit: it was flashing because I was cutting by invisible rects rather than by visible. Which means they would render and you would get to see that for a brief second.
Then I told my function to cut by visible and everything fell apart...I discovered that it had quite a few bugs that I then fixed. Mainly, I wasn't checking whether the two rects actually intersect which would result in a bogus rect being returned and rendered...

Edit 2: I've made it! It is still rendering things that should not be visible (I think?) but it is 100 times faster and no more artifacts (expect tearing lol, can't do nothing about that).