Skip to content

fix(shape): clip descendants to ancestor Bounding regions - #112

Draft
erpalma wants to merge 1 commit into
joske:masterfrom
erpalma:fix/shape-bounding-descendant-clip
Draft

fix(shape): clip descendants to ancestor Bounding regions#112
erpalma wants to merge 1 commit into
joske:masterfrom
erpalma:fix/shape-bounding-descendant-clip

Conversation

@erpalma

@erpalma erpalma commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Why

The SHAPE Bounding region applies to a window and its inferiors. The KMS
scene walk already accumulated ancestor geometry for parent clipping, but it
did not accumulate ancestor Bounding regions. A shaped parent therefore
clipped its own draw while an unshaped descendant could still paint outside
that shape.

This was visible on non-composited Plasma 6.6 on amdgpu/Strix Halo. KWin
shapes the panel frame into a rounded three-rectangle band inside a
3440x40 frame, but the unshaped full-height wrapper child was emitted in
full. Its background painted an opaque white band over the panel margin.
Enabling compositing hid the bug because KWin then sampled the shaped window
as a texture and the cut-out was carried by alpha.

The first extents-based attempt also demonstrated why preserving the exact
region matters: it left the four bounding-box corner pixels painted even
though they were outside the rounded mask.

What changed

  • Thread the accumulated ancestor Bounding region through
    emit_window_subtree as absolute half-open rectangles.
  • Preserve the None versus Some([]) distinction: no shaped ancestor is
    different from an explicitly empty intersection that suppresses the
    subtree.
  • Canonicalize SHAPE regions once at the KMS backend boundary so stored
    regions are sorted, non-overlapping YX bands. This also prevents
    overlapping client rectangles from becoming overlapping SrcOver draws in
    a COW subtree.
  • Intersect nested regions exactly with a YX-band sweep and consume the
    canonical result once per emitted window.
  • Remove the 64-rectangle/bounding-box fallback. Legitimate bitmap masks can
    exceed 64 rectangles, and vertical stripes intersecting horizontal stripes
    can inherently produce the Cartesian product as disjoint output. Replacing
    that exact result with its extents violates SHAPE by painting holes.
  • Keep the common draw path lean: an unshaped window emits directly without
    a candidate Vec, and a shaped leaf with no shaped ancestor does not build
    an unused absolute-region copy.
  • Keep sampled_ids and presentation-damage snapshots once per window even
    when one shape produces multiple draws.

Validation

The original panel case was verified on hardware, including the four
bounding-box corner pixels of the rounded mask. The review follow-ups add
regressions for:

  • shaped ancestor + shaped descendant with negative/local offsets;
  • two nested shaped ancestors and an unshaped grandchild;
  • explicit-empty and empty-intersection propagation;
  • a 65 by 65 vertical/horizontal stripe intersection;
  • COW multi-piece alpha, sampling, and damage-snapshot bookkeeping;
  • backend canonicalization of overlapping and duplicate rectangles.

All repository gates pass:

  • cargo +nightly fmt
  • cargo clippy --all-targets -- -D warnings
  • cargo test --workspace

@joske

joske commented Jul 28, 2026

Copy link
Copy Markdown
Owner

this branch causes GPU faults on my RX580 (your branch separately and also when merged with current master which touches adjacent code)

@joske

joske commented Jul 28, 2026

Copy link
Copy Markdown
Owner

when I log in with restore session on, and systemsettings was open, immediate GPU crash that takes the system down, every single time:

jul 28 16:50:26 silence kernel: amdgpu 0000:01:00.0: GPU fault detected: 147 0x01804802
jul 28 16:50:26 silence kernel: amdgpu 0000:01:00.0:  Process firefox pid 90256 thread firefox:cs0 pid 91069
jul 28 16:50:26 silence kernel: amdgpu 0000:01:00.0:   VM_CONTEXT1_PROTECTION_FAULT_ADDR   0x00100230
jul 28 16:50:26 silence kernel: amdgpu 0000:01:00.0:   VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x0C048002
jul 28 16:50:26 silence kernel: amdgpu 0000:01:00.0: VM fault (0x02, vmid 6, pasid 240) at page 1049136, read from 'TC4' (0x54433400) (72)
jul 28 16:50:26 silence kernel: amdgpu 0000:01:00.0: GPU fault detected: 147 0x01004402
jul 28 16:50:26 silence kernel: amdgpu 0000:01:00.0:  Process firefox pid 90256 thread firefox:cs0 pid 91069
jul 28 16:50:26 silence kernel: amdgpu 0000:01:00.0:   VM_CONTEXT1_PROTECTION_FAULT_ADDR   0x00100220
jul 28 16:50:26 silence kernel: amdgpu 0000:01:00.0:   VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x0E044002
jul 28 16:50:26 silence kernel: amdgpu 0000:01:00.0: VM fault (0x02, vmid 7, pasid 240) at page 1049120, read from 'TC5' (0x54433500) (68)
jul 28 16:50:27 silence tailscaled[1277]: open-conn-track: timeout opening (TCP 100.107.25.13:35072 => 95.216.195.133:80); no associated pe>
jul 28 16:50:28 silence kernel: amdgpu 0000:01:00.0: Dumping IP State
jul 28 16:50:28 silence kernel: amdgpu 0000:01:00.0: Dumping IP State Completed
jul 28 16:50:28 silence kernel: amdgpu 0000:01:00.0: [drm] AMDGPU device coredump file has been created
jul 28 16:50:28 silence kernel: amdgpu 0000:01:00.0: [drm] Check your /sys/class/drm/card2/device/devcoredump/data
jul 28 16:50:28 silence kernel: amdgpu 0000:01:00.0: ring comp_1.1.0 timeout, signaled seq=1873, emitted seq=1877
jul 28 16:50:28 silence kernel: amdgpu 0000:01:00.0:  Process firefox pid 90256 thread firefox:cs0 pid 91069
jul 28 16:50:28 silence kernel: amdgpu 0000:01:00.0: GPU reset begin!. Source:  1
jul 28 16:50:31 silence systemd[1]: dbus-:1.3-org.kde.kded.smart@3.service: Deactivated successfully.
jul 28 16:50:31 silence tailscaled[1277]: open-conn-track: timeout opening (TCP 100.107.25.13:35056 => 95.216.195.133:80); no associated pe>
jul 28 16:50:32 silence tailscaled[1277]: open-conn-track: timeout opening (TCP 100.107.25.13:35072 => 95.216.195.133:80); no associated pe>
jul 28 16:50:32 silence tailscaled[1277]: [RATELIMIT] format("open-conn-track: timeout opening %v; no associated peer node")
jul 28 16:50:32 silence kernel: amdgpu 0000:01:00.0: failed to suspend display audio
jul 28 16:50:33 silence systemd[1]: dbus-:1.3-org.kde.powerdevil.chargethresholdhelper@3.service: Deactivated successfully.
jul 28 16:50:33 silence systemd[1]: dbus-:1.3-org.kde.powerdevil.backlighthelper@3.service: Deactivated successfully.
jul 28 16:50:33 silence systemd[1]: dbus-:1.3-org.kde.powerdevil.discretegpuhelper@3.service: Deactivated successfully.
jul 28 16:50:33 silence kernel: amdgpu: cp is busy, skip halt cp
jul 28 16:50:33 silence kernel: amdgpu: rlc is busy, skip halt rlc
jul 28 16:50:33 silence kernel: amdgpu 0000:01:00.0: BACO reset
jul 28 16:50:33 silence kernel: amdgpu 0000:01:00.0: GPU reset succeeded, trying to resume
jul 28 16:50:33 silence kernel: amdgpu 0000:01:00.0: [drm] PCIE GART of 256M enabled (table at 0x000000F400E80000).
jul 28 16:50:33 silence kernel: amdgpu 0000:01:00.0: VRAM is lost due to GPU reset!
jul 28 16:50:33 silence kernel: amdgpu 0000:01:00.0: [drm] UVD and UVD ENC initialized successfully.
jul 28 16:50:33 silence kernel: amdgpu 0000:01:00.0: [drm] VCE initialized successfully.
jul 28 16:50:33 silence kernel: amdgpu 0000:01:00.0: GPU reset(1) succeeded!
jul 28 16:50:33 silence kernel: amdgpu 0000:01:00.0: [drm] device wedged, but no recovery needed
jul 28 16:50:33 silence gnome-shell[39018]: amdgpu: The CS has cancelled because the context is lost. This context is innocent.
jul 28 16:50:33 silence systemd-coredump[91233]: Process 90025 (plasmashell) of user 1000 terminated abnormally with signal 6/ABRT, process>
jul 28 16:50:33 silence systemd-coredump[91232]: Process 39018 (Xwayland) of user 1000 terminated abnormally with signal 6/ABRT, processing>
jul 28 16:50:33 silence systemd[1]: Started Process Core Dump (PID 91232/UID 0).
jul 28 16:50:33 silence systemd[1]: Started Process Core Dump (PID 91233/UID 0).
jul 28 16:50:33 silence systemd[1]: Started Pass systemd-coredump journal entries to relevant user for potential DrKonqi handling.
jul 28 16:50:33 silence systemd[1]: Started Pass systemd-coredump journal entries to relevant user for potential DrKonqi handling.
jul 28 16:50:33 silence systemd-coredump[91240]: Resource limits disable core dumping for process 90025 (plasmashell).
jul 28 16:50:33 silence systemd-coredump[91240]: [🡕] Process 90025 (plasmashell) of user 1000 terminated abnormally without generating a co>
jul 28 16:50:33 silence systemd[1]: systemd-coredump@13-53250-91233_118492-0.service: Deactivated successfully.
jul 28 16:50:33 silence systemd-coredump[91239]: [🡕] Process 39018 (Xwayland) of user 1000 dumped core.

                                                 Stack trace of thread 39866:

@joske

joske commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Hypothesis: per-frame descriptor-pool exhaustion

MAX_DESCRIPTOR_SETS_PER_FRAME = 1024 (crates/yserver/src/kms/vk/pipeline.rs:119) is not merely advisory — it sizes each per-output descriptor pool, via CompositePoolRing::new(..., MAX_DESCRIPTOR_SETS_PER_FRAME) at crates/yserver/src/kms/render/scene.rs:647. Every emitted CompositeDraw consumes one descriptor set (record_descriptor_allocations(built.scene.draws.len()), scene.rs:~1868).

The 64-rectangle / bounding-box fallback this PR removes was therefore load-bearing for that budget, not only a performance guard: it bounded draws.len() per window. With it gone, one shaped window can emit thousands of draws — this PR's own regression test constructs a 65x65 vertical/horizontal stripe intersection, i.e. 4225 rects from a single window, 4x the entire per-output pool.

Why this reproduces here and not on your setup — three configuration differences:

  1. Compositing is on and KWin sets a bounding shape on the COW covering the whole framebuffer: set_shape_rectangles host_xid=0x103 kind=0 n_rects=1 rects=[RegionRect { x: 0, y: 0, width: 5120, height: 1440 }]. So ancestor_shape == None — documented in the patch as "the common case, no region test runs at all" — never fires. Every composited window takes the region path, every frame.
  2. Two outputs, so the whole scene is walked per output.
  3. Session restore re-redirects every window at once, which is why the fault is immediate on login rather than appearing after a while.

If a frame's descriptor demand exceeds the pool, sets are exhausted or recycled while a prior submission is still in flight; the GPU then dereferences a stale image descriptor and samples a recycled BO. That produces exactly the observed signature — a VM_CONTEXT1 protection fault reading from the texture cache, at a constant GART address, attributed to whichever process owned that memory. Which is why it looks like a client bug and why the named process varies.

Verified: the constant and its role in sizing the pool, one descriptor set per draw, the cap removal, the 4225-rect test, KWin's full-framebuffer COW shape, and the 3/3-vs-clean A/B. Inferred: that descriptor exhaustion specifically is what corrupts. Cheap confirmation would be logging draws.len() per frame per output and checking whether it crosses 1024 shortly before the fault.

Suggested direction

Please don't reinstate the extents fallback — the reasoning for removing it is correct, replacing an exact region with its bounding box paints outside the mask and violates SHAPE. The problem is one draw per rectangle, so either:

  • size the descriptor pool from actual demand (grow/spill when draws.len() exceeds the current pool), and/or
  • stop emitting one draw per rect: batch a window's region into a single draw with scissor rects, or clip via stencil, so draw count stays independent of region complexity.

The second is the more robust fix, since it also removes the per-frame cost of a complex mask rather than just making it affordable.

@erpalma

erpalma commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

This one will require some more work I guess. I should have a Polaris card to try, I just need a spare machine for these tests. Give me a few more days and I'll report back.

@erpalma
erpalma marked this pull request as draft July 30, 2026 05:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants