Skip to content

V3: Depth map and parallax on Image layers - #173

Merged
git-chad merged 68 commits into
git-chad/shader-lab-v3-planfrom
git-chad/v3-depth-parallax
Oct 1, 2026
Merged

git-chad merged 68 commits into
git-chad/shader-lab-v3-planfrom
git-chad/v3-depth-parallax

Conversation

@git-chad

@git-chad git-chad commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #172. Roadmap 6.4, stage 1 (feasibility prototype).

What

  • Image layers accept a second grayscale asset as a depth map (depthAssetId; exported to the runtime config as depthAsset). Source → Depth map offers Attach / Replace / Remove. History, duplication, .lab save/reopen (missing depth reported as "Missing depth map"), shader export and MCP summaries carry it.
  • A Depth group appears only with a map attached: Invert Depth, Depth Range, Focus Plane, Motion (Off / Orbit / Sway / Nod / Dolly), Amount, Speed, Camera Offset (keyframable), Edges (Stretch / Transparent), Quality (16 / 32 / 64 layers), Show Depth.
  • MediaPass (editor and runtime) marches depth layers from near to far with five binary refinement steps, inside a TSL Fn; nearer content covers what is behind it. Color and depth are sampled with explicit gradients from the undisplaced UV. Layers without a depth map keep the previous shader.

Known behaviour

Where a near subject uncovers what was behind it, its trailing edge stretches into the gap (DepthFlow does the same without inpainting). Soft depth edges hide it. The depth map is sampled with the photo's UVs and not realigned. Video layers and depth generation are out of scope.

Verification

  • bun run check (3 preexisting warnings).
  • Full bun run test:composition including the new depth-parallax.mjs (77 checks: camera math, editor/runtime passes, leading-edge parallax, occlusion, inversion, edge modes, quality, orbit/dolly over time, detach, parity, project file, history, duplication, missing-depth reopen, export, both pipelines).
  • depth-parallax-ui.mjs against the dev server: attach, gated controls, motion select, remove, undo, actual .lab saves.
  • Manual check: tests/composition/DEPTH-PARALLAX-MANUAL-QA.md. Visual acceptance can now use any photo: Estimate generates the map in the app.

🤖 Generated with Claude Code

Depth generation in the app

  • Source → Depth map → Estimate runs Depth Anything V2 Small (the model DepthFlow uses) through transformers.js in a Web Worker: WebGPU fp32 when an adapter exists, WASM q8 otherwise. Model download from Hugging Face with visible progress and browser caching; the grayscale result (white = near) is resized to the photo and attached as a local asset <name>-depth.png, Invert Depth off. SVG sources fall back to Attach.
  • depth-estimate-ui.mjs runs the real download and inference against the dev server (106 s on SwiftShader including a 10 s download) and checks the attached asset, Invert off and the Show Depth preview.
  • New dependency: @huggingface/transformers 4.3.0. ONNX runtime wasm binaries load from its default CDN.

Scene depth (stage 2)

  • The image pass now marches once into an auxiliary target (RGB = displaced depth, white = near; A = in-bounds) and the editor and runtime pipelines thread it up the stack as scene depth, through groups too. The nearest depth-bearing Image below wins; sources without a map pass it through.
  • Input: Luminance / Depth on ASCII, Halftone, Pattern and Gradient Map: the effect samples depth as if it were the image while still blending over the color. Falls back to luminance without a map.
  • Mask → Shape → Depth with Near / Far / Feather limits any effect by distance. Inert without a map.
  • Zero cost without depth: same shaders, only the bound texture changes. Because TSL shares texture bindings by texture uuid, the source swap is structural and rebuilds the effect node when depth appears or disappears.
  • depth-parallax.mjs now has 103 checks including the aux texture, Input → Depth vs luminance, fallback, depth mask and inversion, groups, runtime pipeline, plus a 1080p timing line ({"plain":32.3,"depthMedium":714.8,"depthHigh":1214.1,"depthLow":433.6} ms median on SwiftShader). Full suite and depth-parallax-ui.mjs pass.

Keyframable masks

  • Mask Center, Size, and for the Depth shape Near and Far as separate tracks, Rotation and Feather get keyframe buttons and appear in the timeline as mask.* parameter tracks (teal). The editor frame builder and the runtime fold the evaluated values back into the mask; audio links can drive them too.
  • Enables the contour sweep: keyframes on Mask Near and Mask Far move a thin depth band through the subject.
  • depth-parallax.mjs (108 checks) renders a keyframed sweep at several times in both pipelines; the UI test creates a mask keyframe from the panel.

@vercel

vercel Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
shader-lab Ready Ready Preview Oct 1, 2026 2:03pm UTC

@greptile-apps

greptile-apps Bot commented Sep 21, 2026

Copy link
Copy Markdown

RetriggerConfidence Score: 2/5

The PR is not yet safe to merge because preview/export motion can diverge, Stretch fails in a supported contain configuration, and missing-depth warnings are cleared after loading the color image.

Summary

This PR adds depth-map attachment and persistence for image layers, parallax controls, matching editor/runtime depth-marching shaders, export support, and composition/UI coverage.

  • Carries depth assets through project files, history, duplication, renderer frames, runtime configs, and MCP summaries.
  • Adds orbit, sway, nod, dolly, edge, quality, inversion, focus, and depth-preview controls.
  • Adds synthetic GPU, pipeline, persistence, export, and UI tests.

Reviews (1) · Last reviewed commit: "Add depth maps and parallax to Image lay..."

Comment thread src/lib/editor/project-file.ts Resolved
Comment thread src/renderer/media-pass.ts Resolved
Comment thread src/renderer/media-pass.ts Resolved
Comment thread src/lib/editor/project-file.ts Resolved
… UI test docs

- GeometryHandles: keep restore in a ref so rerenders during a drag no longer cancel it
- shape-pass (editor and runtime): atan2 returns +-PI on the negative axis when y is zero
- shader-export: add shape to SupportedShaderExportLayerType
- Document shape-layers-ui.mjs in SHAPE-LAYERS-MANUAL-QA.md
…th limit

- restore font size when a text handle resize is cancelled
- rotate the edit textarea around the anchor on both axes
- cap in-place text edits at the Text parameter's 600 characters
…bracket mode

Body: the trail ribbon now accumulates the corner-masked bracket stroke instead of the full outline, in both the app and the React runtime copy.
…ait edge readback on export, hide snapped handles, widen export type

- Painted placement falls back to the undrifted anchor when drift would leave the stroke
- Edge placement lays out nothing until the edge field exists and the export prepare step awaits the pending readback
- Annotation target handles are hidden while Snap to Strongest Edge is active
- Add annotations to SupportedShaderExportLayerType
…dges, depth validation

- Keep the Missing depth map warning when the color image finishes loading
- Drive built-in parallax from the composition timeline time in the editor media pass
- Make Depth Edges Stretch cover exposed areas inside the frame when Contain has transparent borders (editor and runtime media passes)
- Drop depth references on non-image layers or non-image assets during import
@git-chad
git-chad changed the base branch from git-chad/v3-annotations to git-chad/shader-lab-v3-plan October 1, 2026 15:13
@git-chad
git-chad merged commit 6dfbab9 into git-chad/shader-lab-v3-plan Oct 1, 2026
5 checks passed

This branch was successfully deployed

1 active deployment
Preview — 6dfbab9c Deployed Oct 1, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant