Skip to content

deko3d: fix two rendering bugs from leftover sticky state - #2

Open
senolfeldmann wants to merge 2 commits into
HueponiK:deko3d-driverfrom
senolfeldmann:deko3d-sticky-state-fixes
Open

deko3d: fix two rendering bugs from leftover sticky state#2
senolfeldmann wants to merge 2 commits into
HueponiK:deko3d-driverfrom
senolfeldmann:deko3d-sticky-state-fixes

Conversation

@senolfeldmann

Copy link
Copy Markdown

Two rendering bugs in the deko3d driver. They share a root cause: the driver leaves
sticky state dirty
-- once by not establishing state it depends on, once by not
restoring state it mutated on a shared resource. Both are small and independent; they
are in one PR because they are the same class of bug in the same file, and they are
split into one commit each.

1. Menu residue in the letterbox bars

Symptom: after opening and closing the menu over 4:3 content, menu fragments stay
visible in the black pillarbox bars.

Cause: in deko3d, dkCmdBufClearColor is not an unconditional surface clear. It is
effectively a fast-clear draw and is therefore bounded by the active scissor (and
gated by the bound color-write mask / blend state). The scissor is sticky queue state:
once set it persists across command buffers and frames until something sets it again.

dk3d_frame() binds the swapchain image and clears it immediately, without resetting
the scissor. The menu and font paths (gfx_display_dk3d_draw, dk3d_font_render_line)
set sub-rectangle scissors while drawing -- Ozone scissors its scrolling item list --
and the last one stays latched. The next frame's clear is clipped to that leftover
rectangle, so the letterbox bars and any region the content blit does not cover keep the
previous frame's contents.

Repro: run 4:3 content, open the menu and scroll a list (so a scissored draw is
last), close the menu -> fragments persist in the bars.

Fix: set viewport and scissor to the full surface immediately before the clear.

Why it is safe: the content blit uses the 2D engine (dkCmdBufBlitImage), which
ignores the 3D viewport/scissor. Every menu/font/OSD draw sets its own viewport and
scissor before drawing. So this change governs only the clear.

2. Menu presented flipped after "Close Content"

Symptom: load a deko3d HW-rendered core, then Close Content -- the whole menu is
presented flipped. The first menu after boot is fine.

Cause: the driver renders with DkDeviceFlags_OriginLowerLeft. To present that
upright, deko3d's dkSwapchainCreate calls
nwindowSetTransform(win, HAL_TRANSFORM_FLIP_V) on the shared default window
(nwindowGetDefault()) and never clears it. dk3d_free() destroys the swapchain and
device but leaves the transform latched on that shared window.

RetroArch hot-swaps video drivers on the same window: when the deko3d HW core unloads,
RA falls back to the GL/EGL driver for the menu, which inherits the leftover
HAL_TRANSFORM_FLIP_V. The window starts at the libnx default transform (0,
documented as "no transformation applied"), which is why this only appears once deko3d
has run.

Repro: boot (menu upright) -> load a deko3d HW core -> Close Content -> menu is
flipped.

Fix: restore the default transform in dk3d_free() after the swapchain is
destroyed.

Notes for reviewers

  • It is deliberate that this lives in the video driver's teardown rather than in generic
    code: deko3d is the component that mutated a shared resource, so it should restore it
    on the way out. (dkSwapchainCreate setting the flip and never clearing it is
    arguably a deko3d-library quirk, but the driver is the right place to contain it.)
  • nwindowSetTransform and the default value 0 are libnx API
    (switch/display/native_window.h); the header documents 0 as the default.
  • The deko3d menu draw path itself is orientation-correct and content-independent
    (Ozone uses a fixed 0-degree MVP, gfx_display_draw_quad pre-flips Y) -- the flip is
    inherited by the GL menu, not produced by deko3d's own menu rendering.

Notes

  • Both fixes are independent; either can be dropped without affecting the other.
  • Happy to split this into two PRs, or to re-target straight at master, if you prefer.

dkCmdBufClearColor is bounded by the active scissor (it is effectively a
fast-clear draw, also gated by the bound color-write mask and blend
state), and in deko3d the scissor is sticky queue state that survives
across frames.

The menu/font draw paths (gfx_display_dk3d_draw, dk3d_font_render_line)
call dkCmdBufSetScissors with sub-rectangles -- e.g. Ozone's item-list
scissor. dk3d_frame then binds the swapchain and clears it WITHOUT first
resetting the scissor, so the per-frame clear only covers whatever
sub-rect the previous frame's last menu/font/OSD draw happened to leave
set. Anything outside it -- the letterbox bars around 4:3 content, and any
region the content blit does not cover -- retains last frame's pixels.
Visible symptom: menu fragments ghost into the black bars after opening
and closing the menu over 4:3 content.

Set the viewport and scissor to the full surface immediately before the
clear. The content blit (2D engine, dkCmdBufBlitImage) ignores both, and
every menu/font draw sets its own viewport+scissor, so this governs only
the clear and changes nothing else.
The driver renders with DkDeviceFlags_OriginLowerLeft, and to present that
upright deko3d's dkSwapchainCreate sets
nwindowSetTransform(win, HAL_TRANSFORM_FLIP_V) on the window. The driver
uses the process-wide default window (nwindowGetDefault()), and dk3d_free
destroys the swapchain but never clears that transform, so the vertical
flip stays latched on the shared NWindow after the driver is gone.

When RetroArch hot-swaps video drivers on the same window -- e.g. a deko3d
HW-rendered core unloads and RA falls back to the GL/EGL driver for the
menu -- the incoming driver inherits HAL_TRANSFORM_FLIP_V and presents its
entire output upside-down. The window starts at the libnx default transform
(0), so the first menu after boot is correct and the flip only appears once
deko3d has run.

Restore the default transform (0) in dk3d_free after destroying the
swapchain, so the next driver to take the window over starts from the same
clean state it had at boot.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant