deko3d: fix two rendering bugs from leftover sticky state - #2
Open
senolfeldmann wants to merge 2 commits into
Open
deko3d: fix two rendering bugs from leftover sticky state#2senolfeldmann wants to merge 2 commits into
senolfeldmann wants to merge 2 commits into
Conversation
dkCmdBufClearColor is bounded by the active scissor (it is effectively a fast-clear draw, also gated by the bound color-write mask and blend state), and in deko3d the scissor is sticky queue state that survives across frames. The menu/font draw paths (gfx_display_dk3d_draw, dk3d_font_render_line) call dkCmdBufSetScissors with sub-rectangles -- e.g. Ozone's item-list scissor. dk3d_frame then binds the swapchain and clears it WITHOUT first resetting the scissor, so the per-frame clear only covers whatever sub-rect the previous frame's last menu/font/OSD draw happened to leave set. Anything outside it -- the letterbox bars around 4:3 content, and any region the content blit does not cover -- retains last frame's pixels. Visible symptom: menu fragments ghost into the black bars after opening and closing the menu over 4:3 content. Set the viewport and scissor to the full surface immediately before the clear. The content blit (2D engine, dkCmdBufBlitImage) ignores both, and every menu/font draw sets its own viewport+scissor, so this governs only the clear and changes nothing else.
The driver renders with DkDeviceFlags_OriginLowerLeft, and to present that upright deko3d's dkSwapchainCreate sets nwindowSetTransform(win, HAL_TRANSFORM_FLIP_V) on the window. The driver uses the process-wide default window (nwindowGetDefault()), and dk3d_free destroys the swapchain but never clears that transform, so the vertical flip stays latched on the shared NWindow after the driver is gone. When RetroArch hot-swaps video drivers on the same window -- e.g. a deko3d HW-rendered core unloads and RA falls back to the GL/EGL driver for the menu -- the incoming driver inherits HAL_TRANSFORM_FLIP_V and presents its entire output upside-down. The window starts at the libnx default transform (0), so the first menu after boot is correct and the flip only appears once deko3d has run. Restore the default transform (0) in dk3d_free after destroying the swapchain, so the next driver to take the window over starts from the same clean state it had at boot.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two rendering bugs in the deko3d driver. They share a root cause: the driver leaves
sticky state dirty -- once by not establishing state it depends on, once by not
restoring state it mutated on a shared resource. Both are small and independent; they
are in one PR because they are the same class of bug in the same file, and they are
split into one commit each.
1. Menu residue in the letterbox bars
Symptom: after opening and closing the menu over 4:3 content, menu fragments stay
visible in the black pillarbox bars.
Cause: in deko3d,
dkCmdBufClearColoris not an unconditional surface clear. It iseffectively a fast-clear draw and is therefore bounded by the active scissor (and
gated by the bound color-write mask / blend state). The scissor is sticky queue state:
once set it persists across command buffers and frames until something sets it again.
dk3d_frame()binds the swapchain image and clears it immediately, without resettingthe scissor. The menu and font paths (
gfx_display_dk3d_draw,dk3d_font_render_line)set sub-rectangle scissors while drawing -- Ozone scissors its scrolling item list --
and the last one stays latched. The next frame's clear is clipped to that leftover
rectangle, so the letterbox bars and any region the content blit does not cover keep the
previous frame's contents.
Repro: run 4:3 content, open the menu and scroll a list (so a scissored draw is
last), close the menu -> fragments persist in the bars.
Fix: set viewport and scissor to the full surface immediately before the clear.
Why it is safe: the content blit uses the 2D engine (
dkCmdBufBlitImage), whichignores the 3D viewport/scissor. Every menu/font/OSD draw sets its own viewport and
scissor before drawing. So this change governs only the clear.
2. Menu presented flipped after "Close Content"
Symptom: load a deko3d HW-rendered core, then Close Content -- the whole menu is
presented flipped. The first menu after boot is fine.
Cause: the driver renders with
DkDeviceFlags_OriginLowerLeft. To present thatupright, deko3d's
dkSwapchainCreatecallsnwindowSetTransform(win, HAL_TRANSFORM_FLIP_V)on the shared default window(
nwindowGetDefault()) and never clears it.dk3d_free()destroys the swapchain anddevice but leaves the transform latched on that shared window.
RetroArch hot-swaps video drivers on the same window: when the deko3d HW core unloads,
RA falls back to the GL/EGL driver for the menu, which inherits the leftover
HAL_TRANSFORM_FLIP_V. The window starts at the libnx default transform (0,documented as "no transformation applied"), which is why this only appears once deko3d
has run.
Repro: boot (menu upright) -> load a deko3d HW core -> Close Content -> menu is
flipped.
Fix: restore the default transform in
dk3d_free()after the swapchain isdestroyed.
Notes for reviewers
code: deko3d is the component that mutated a shared resource, so it should restore it
on the way out. (
dkSwapchainCreatesetting the flip and never clearing it isarguably a deko3d-library quirk, but the driver is the right place to contain it.)
nwindowSetTransformand the default value0are libnx API(
switch/display/native_window.h); the header documents0as the default.(Ozone uses a fixed 0-degree MVP,
gfx_display_draw_quadpre-flips Y) -- the flip isinherited by the GL menu, not produced by deko3d's own menu rendering.
Notes