Skip to content

GPU-Engine.md

Latest commit

 

History

History
255 lines (176 loc) · 10.8 KB

GPU-Engine.md

File metadata and controls

255 lines (176 loc) · 10.8 KB
Raw

GPU Engine

Back to Index

WebGPU rendering, texture management, preview output, export capture, and GPU-backed analysis.

Linux/Mesa: GPU-accelerated canvas, worker OffscreenCanvas, and WebGPU paths fail silently on open-source Mesa drivers. Before changing any canvas/GPU code, read Linux / Mesa GPU.

Overview

WebGPUEngine is a thin facade over the rendering subsystems. It owns WebGPU context setup, texture and cache managers, the render loop, the render dispatcher, output windows, and export canvas state.

The engine supports:

  • Main preview rendering
  • Independent preview targets
  • Popup output windows
  • Sliced output / corner-pin rendering
  • RAM preview and scrubbing caches
  • Zero-copy export frames when the export canvas path is available
  • CPU readback as a fallback path
  • Waveform, histogram, and vectorscope GPU panels
  • Native 3D scene and Gaussian-splat composition when their enabled runtime flags apply

Architecture

Core Pieces

  • WebGPUContext handles adapter/device setup and device-loss recovery.
  • RenderLoop runs the RAF loop, idle detection, scrub limiting, and watchdog recovery.
  • RenderDispatcher handles per-frame texture import, compositing, output, and preview fallbacks.
  • RenderTargetManager owns ping/pong compositing buffers plus independent preview buffers.
  • TextureManager and MaskTextureManager manage image/video/mask textures.
  • CacheManager owns scrubbing cache, composite RAM preview cache, and GPU frame cache.
  • ExportCanvasManager owns the export OffscreenCanvas and stacked-alpha mode.

Runtime State

  • The engine keeps a unified Map of target canvases.
  • Preview canvases and output windows are reconfigured after device restore.
  • HMR reuses the engine singleton when possible.

Initialization Failures And Device Recovery

Startup distinguishes missing WebGPU, an unavailable adapter, adapter/device request timeouts, and renderer-resource failures. The preview displays the specific failure instead of describing every failure as incompatible hardware. A timed-out request can still be pending inside the browser, so recovery does not queue additional requests behind it. Late GPU devices are destroyed.

After unexpected device loss, the context drops the lost device after cache cleanup and makes up to three recovery attempts for settled failures. Recovery waits for asynchronous renderer resources before making the preview ready. Timeouts and unsupported runtimes terminate with a visible error rather than an indefinite loading state. Destroy/reinitialize invalidates older recovery work, and retained output canvases are rebound to the replacement device.

Targeted coverage: webgpuContext.test.ts, contextRecoveryWiring.test.ts, and outputCanvasRecovery.test.ts under tests/unit/.

Boot Health And GPU Hazard Recovery

Risky GPU startup records a release-scoped boot attempt in bootHealth.ts. Three interrupted attempts inside 24 hours activate a GPU hazard for that release and route canvas policy through the existing software fallback. An attempt becomes healthy only after the active render mode proves a readable frame; worker submission alone is not treated as pixel proof.

Device loss records the same hazard signal. The visible warning offers a manual Retry GPU action that clears the safety state and reloads the app. Clean pagehide completion and a three-second settled startup prevent normal reloads from being counted as crashes. Runtime owners and the engine singleton remain HMR-safe.

Texture Paths

Video

  • HTMLVideoElement is imported as an external texture when the browser supports it.
  • VideoFrame can also be imported as an external texture.
  • Firefox preview uses copied textures instead of external video import because imported frames can intermittently go black.
  • Android Chromium copies HTML-video preview frames into persistent GPU textures during playback, pause, and seeking because its external video textures can otherwise present intermittent black frames.

Images And Canvas

  • Images and canvases are copied into rgba8unorm GPU textures.
  • Cached image views are reused when possible.

Motion Shapes

  • motion-shape clips render through src/engine/motion/MotionRenderer.ts.
  • Rectangle, ellipse, polygon, and star primitives are drawn with analytic WGSL SDFs into transparent rgba8unorm textures.
  • A bounded uniform layout composites up to 8 ordered color-fill, stroke, linear-gradient, radial-gradient, or texture-fill appearances, with up to 8 stable stops per gradient and six shader blend modes.
  • Replicated motion shapes use a per-shape instance buffer and instanced draws in the same shader path. The shader ceiling is 100,000 instances; effective counts are also constrained by the device, render target, and configured user limit.
  • The resulting texture view is composited through the normal CompositorPipeline, so masks, effects, blend mode, nested comps, preview targets, and export share the same downstream path.

Masks

  • Mask textures are uploaded per layer.
  • A 1x1 white fallback texture is used when no mask is present.

Limitation

  • The zero-copy path only applies when the browser and source type support it. Preview and export still have explicit fallback paths.

Render Loop

Idle And Scrub Control

  • Idle mode starts after about 1 second of no activity.
  • Idle detection is initially suppressed so browser video surfaces can warm up; it is lifted by the first play event or after about 3 seconds.
  • Playback is limited to about 60 fps.
  • Scrubbing is limited to about 60 fps unless a fresh video frame arrives, which bypasses the limiter.

Watchdog

  • A watchdog checks for stalls every 2 seconds.
  • If the render loop stops producing frames for about 3 seconds while it should be active, it wakes the loop or restarts it.

Export And RAM Preview

  • The render loop skips preview rendering while export is active.
  • Export and RAM preview both set flags through ExportCanvasManager.
  • These modes are separate from the normal preview path, but they share the same engine.

Preview Fallback Chain

RenderDispatcher uses a best-effort chain for video preview:

  1. Cached scrubbing frame
  2. Copied fallback frame
  3. Live external texture import
  4. Last known frame or same-clip hold frame

This is why preview can stay stable during seeks while a fresh frame is pending.

Firefox Special Case

  • Firefox uses getCopiedHtmlVideoPreviewFrame() for HTML video preview stability.
  • That path copies the current video frame into a persistent GPU texture and falls back to the previous stable frame when needed.

Output And Export

OutputPipeline

  • The output pipeline uses separate uniform buffers for transparency grid off/on and stacked-alpha export.
  • Full-frame output and compositor passes draw one oversized triangle, avoiding a diagonal seam or missing half-frame when a mobile WebGPU driver drops one primitive from a two-triangle quad.
  • renderToCanvas() catches canvas-context loss and simply skips that target for the frame.
  • Bind-group caches are separate per output mode.

Stacked Alpha

  • Stacked alpha doubles the encoded export height.
  • RGB is rendered in the top half and alpha grayscale in the bottom half.
  • ExportCanvasManager creates the doubled-height OffscreenCanvas when stacked alpha is enabled.

Readback

  • readPixels() is the CPU fallback path.
  • It is used by export fallback paths and preview capture utilities.
  • It is slower than the zero-copy export path and should be treated as a fallback, not the normal route.

Export Canvas

ExportCanvasManager is responsible for export-state flags and the export canvas lifecycle.

Current Behavior

  • initExportCanvas() creates an OffscreenCanvas with WebGPU context.
  • createVideoFrameFromExport() waits for device.queue.onSubmittedWorkDone() before constructing a VideoFrame.
  • If the zero-copy path cannot be created, export falls back to readPixels().
  • cleanupExportCanvas() clears the canvas and stacked-alpha state after export.

Limitation

  • Zero-copy export only works when OffscreenCanvas + WebGPU + VideoFrame creation are available and the export canvas can be configured.

Feature Flags

Runtime flags are exposed on window.__ENGINE_FLAGS__.

  • useFullWebCodecsPlayback and disableHtmlPreviewFallback are synced with the persisted settings toggle.
  • useLiveSlotTrigger swaps slot-grid clicks from editor-open behavior to direct live triggering.
  • useWarmSlotDecks prepares reusable slot-owned live decks for faster layer adoption.
  • use3DLayers and useGaussianSplat are enabled in this branch.
  • workerFirstRenderHost exists for worker-host presentation, but defaults to false; the main-thread render host remains the default.
  • timelineCanvasWorker is enabled for eligible timeline rows; it is separate from the main preview render host.
  • Motion shape/appearance rendering and the Grid Replicator MVP are always part of the native layer path.

Performance Notes

  • ScrubbingCache keeps up to 192 video scrub frames at 30 fps quantization, with a 192 MiB texture budget and a 960-pixel maximum source dimension.
  • Composite RAM preview caches are capped at 900 frames and 512 MB.
  • GPU RAM preview cache is capped at 60 frames.
  • These are hard cache limits from code.

Preview Quality

  • useEngine() scales the active-composition base resolution by the persisted preview-quality multiplier before calling engine.setResolution(...).
  • Full, Half, and Quarter therefore change the GPU render size for engine-backed preview targets.
  • Export output resolution still comes from the composition/export settings, not from preview quality.

Troubleshooting

Problem What To Check
Black preview after reload Video GPU surfaces may not be warm yet. The engine suppresses idle until first play, but browser readiness still matters.
Firefox preview instability HTML video fallback should be using copied textures, not external import.
Export canvas missing Verify WebGPU context creation on OffscreenCanvas and device validity.
Device lost The engine attempts recovery and reconfigures canvases, but a manual reload may still be needed.

Sources

Key implementation files:

  • src/engine/WebGPUEngine.ts
  • src/engine/render/RenderDispatcher.ts
  • src/engine/render/RenderLoop.ts
  • src/engine/render/htmlVideoPreviewFallback.ts
  • src/engine/motion/MotionRenderer.ts
  • src/engine/motion/shaders/motionShapes.wgsl
  • src/engine/analysis/ScopeRenderer.ts
  • src/engine/native3d/NativeSceneRenderer.ts
  • src/engine/pipeline/OutputPipeline.ts
  • src/engine/pipeline/SlicePipeline.ts
  • src/engine/managers/ExportCanvasManager.ts
  • src/engine/texture/ScrubbingCache.ts
  • src/engine/core/RenderTargetManager.ts
  • src/engine/featureFlags.ts
  • src/engine/video/VideoFrameManager.ts
  • src/services/render/renderHostPort.ts