AtlasEngine: don't render frames that won't be presented

[Teera (teeramusic)]

Summary of the Pull Request

AtlasEngine::Present() currently runs the backend's Render() on every frame and only afterwards, in _present(), decides that a frame with an empty dirty rect must not be presented. Rendering a frame that is then never presented makes some drivers hold on to the intermediate resources indefinitely, so private commit grows without bound for as long as such frames keep coming. This PR skips Render() when the dirty rect is empty, unless the backend requires a continuous redraw.

References and Relevant Issues

Closes #20342

#20346 tried to address the same issue by ignoring UIA output notifications; this PR fixes it in the renderer instead, where the unpresentable frame is produced.

Detailed Description of the Pull Request / Additional comments

Empty-dirty-rect frames are produced continuously in two everyday situations:

• A pane is scrolled up while output keeps arriving below the viewport (the case reported in Massive memory leak if left in a scrolled up position #20342).
• A TUI hides the cursor and keeps repositioning it. Cursor::_RedrawCursor schedules a frame whenever the cursor is "on" (blink phase), regardless of its visibility, while Renderer::_invalidateCurrentCursor only invalidates a visible cursor. So a hidden cursor that moves yields a frame with nothing dirty even when the pane is at the bottom. Claude Code (Ink) is one such TUI, which is why this leak also shows up without scrolling.

The check is placed after _handleSettingsUpdate() / _handleSwapChainUpdate() and before _b->Render(_p). At that point _p.dirtyRectInPx already accounts for everything that can be dirty: invalidated and scrolled-in rows (StartPaint), the cursor's old and new position including a cursor that was just turned off (PaintCursor / EndPaint), and a recreated backend or swap chain (MarkAllAsDirty). The only places the backends extend the dirty rect during Render() are gated on invalidatedRows.contains(y) (glyph overhang in BackendD3D / BackendD2D), and a non-empty invalidatedRows always produces a non-empty dirty rect in StartPaint. The exception is the custom-shader path (_executeCustomShader, also used by experimental.retroTerminalEffect), which sets the full target rect on every Render(). So with a custom shader active, the current code presents a full frame every time one is scheduled even when nothing changed, and this PR skips those. That is safe when the shader doesn't read the time variable (the built-in retro shader declares it but never uses it): its output depends only on the text texture, which hasn't changed, so the last presented frame is already correct. Shaders that do use time have to run every frame; RequiresContinuousRedraw() reports exactly those and they are excluded from the early return. Without a custom shader, the skipped frames are precisely the ones _present() would have dropped. In every case the on-screen result is unchanged.

The simpler variant from the issue thread (unconditionally returning when the dirty rect is empty) was tried first and dropped: it freezes time-based custom shaders.

Validation Steps Performed

• Built on 1.24.11911.0 and used as my default terminal for regular work since (Windows 11 26200, display on an Intel UHD 770 as the primary DXGI adapter). Private commit has stayed flat where the Store build previously grew to tens of GB.
• A/B on the same machine, 15 s samples of WindowsTerminal.exe private commit:
  • Hidden-cursor workload (cursor hidden with CSI ?25l, cursor moved 20x/s with CSI n G, no text, pane at the bottom, unfocused): baseline grew ~120 MB/min linearly; with this change it stayed at 69 MB for the duration.
  • Always-dirty control (cursor hidden, one line rewritten 10x/s): flat on both builds, identical output.

PR Checklist

• Closes Massive memory leak if left in a scrolled up position #20342
• Tests added/passed
• Documentation updated
• Schema updated (if necessary)

---

Per the AI usage policy: this was worked out and written with AI assistance (Claude Code). I have read the complete change and tested the built application as described above.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Dustin L. Howett (DHowett)]

Looks like you made the mistake of not reading the issue you're solving:

#20342 (comment)

dirtyRectInPx is not suitable. Read the thread.

[Leonard Hecker (lhecker)]

To be fair it's hard to spot the absence of rendering haha. My recommendation: presentmon. Check if WindowsTerminal.exe still renders when scrolled up and printing text.