Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion admin/app/runs/[runId]/console.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -846,7 +846,7 @@ export default function RunConsole({ runId }: { runId: string }) {
? selected.freshInference === false
? 'Saved inputs were reused without creating a new attempt.'
: 'This stage has not run yet, so there is nothing to look at.'
: 'This attempt produced no files.'
: 'No artifact links recorded for this attempt.'
}
/>

Expand Down
2 changes: 1 addition & 1 deletion admin/components/run-overview.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ function progressTitle(progress: RunProgress): string {
if (progress.finished) return `Run finished: ${progress.finished}`;
if (progress.paused) return 'Run paused';
if (progress.question) return 'Waiting for your answer';
if (progress.stopped.length) return 'Processing stopped';
if (progress.currentStages.length) return `Running ${progress.currentStages.join(', ')}`;
if (progress.stopped.length) return 'Processing stopped';
return 'Waiting for the supervisor';
}

Expand Down
22 changes: 18 additions & 4 deletions docs/audio.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,24 @@ The viewer discovers `audio.json` next to `people.json`, or in the world directo
The timeline duration must match the viewer's people/sequence duration within 0.1 seconds. A track's sample zero occurs at `offsetSeconds` on the clip timeline, so sample time equals clip time minus offset. Trim original audio against the same original-video edit, retaining matching silence and verifying offsets. URLs resolve relative to `audio.json`. Additional provenance fields are allowed. Optional `provenance.attribution`, `provenance.license`, and `provenance.licenseUrl` show a visible audio excerpt credit while the original mix is selected; license links accept only HTTP(S) URLs. The recovered TOS package uses this credit for Blender Foundation's CC BY 3.0 source. Keep media and manifests in gitignored `public/worlds/<clip>-4d/` or `public/clips/audio/`; publish through the existing shared-assets snapshot only when authorized. PCM WAV and MP3 MIME entries already exist. No keys belong in manifests or client code. The attribution strip is currently shown only in original mode; it does not render per-stem credits or remain visible in spatial mode.

The single-person and multiperson packagers probe the supplied local video with `ffprobe` and
write a minimal `audio.json` containing its embedded-audio availability, SHA-256 and the package
playback duration. Silent videos therefore show “No soundtrack available”; videos with an audio
stream keep their original embedded soundtrack. A failed or malformed probe stops packaging
before package files are written instead of guessing that the clip is silent.
write `audio.json` containing soundtrack availability, the submitted source SHA-256 and the
package playback duration. Silent videos therefore show “No soundtrack available”; ordinary
AAC inputs keep their embedded soundtrack. A failed or malformed probe stops packaging before
package files are written instead of guessing that the clip is silent.

H.264 clips with original PCM audio use a separate browser delivery: `media/source.mp4` copies
only the encoded video, and `media/original.wav` copies the original PCM samples. The source
file remains unchanged. `source.hasAudio` is false for this silent delivery video,
`source.videoUrl` selects it, and `original.url` selects the authentic external mix.
`media/provenance.json` binds both output hashes to the submitted source. Packaging compares
every encoded video packet and timestamp, its time base, the PCM sample hash, sample count,
rate, channels and continuous audio timestamps before exposing the files. No encoding,
resampling, normalization or timeline changes occur. Currently this path accepts one
zero-origin H.264 video and one signed little-endian 16-, 24- or 32-bit PCM audio stream;
unsupported PCM layouts fail explicitly. Run previews and completion validate the delivery
hashes while still requiring the original exported source. The standalone printed viewer URL
also selects the delivery MP4. These local checks establish media preservation; browser
playback and audible synchronization still require actual viewer checks.

An identical generated manifest can be reused. An existing manifest with different content,
including authored external audio or reviewed stems, stops packaging before it changes the
Expand Down
8 changes: 8 additions & 0 deletions docs/known-limits.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,18 @@ Approaches already measured to fail.
Do not retry one without a new idea, and add a line here when you measure a new dead end.

- Refining body pose against the footage makes it worse: a single camera leaves a null space.
- Semantic person masks can omit lower bodies against similar-colored floors: the fresh bottle-source clean left black and light trousers in several sampled frames despite a successful worker exit. Tracked video cleaning now unions the existing per-person instance masks and retains their exact source-frame selection; equal FPS and frame count do not imply the same pixels (the observed 30-to-12 FPS tracker and cleaner selected different source frames). Missing, empty, or incomplete track masks and mismatched source/timing metadata block submission. Full mask coverage is not an inpainting-quality pass; inspect the resulting clean for remnants and damaged surroundings. Failure evidence: `.context/evidence/bottle-source-rerun/clean-comparison.jpg`.
- FFmpeg 5.1 rejects a long flat sum of exact-frame selection terms: the 120-frame clean correction failed before inpainting, and the parser failure reproduced in local Debian Bookworm while FFmpeg 8 passed. Balanced expressions preserve the same source indices and pass the full-resolution local Linux reproduction. Keep decoder stderr in failure reports; a successful local decode does not establish remote inpainting quality. Evidence: `.context/evidence/bottle-source-rerun/linux-decoder-check/`.
- Complete tracked person masks can still omit a carried backpack. In the fresh bottle clean, input and output masks matched on all 120 frames, but four inspected backpack regions retained 17–42% unmasked source pixels. A local union with `moved_content_masks` covered those regions but also classified static stair posts and railings as people; its residual-area guard cannot reject semantic person false positives. Do not promote that union as a cleaning fix. Evidence: `.context/evidence/bottle-source-rerun/balanced-clean-review/`.
- Faces below about 64 px of face height are unreadable; per-clip head refits do not help.
- Smoke, fire, and water cannot be reconstructed from one moving camera.
- Clips with too little parallax, or too dark, fail reconstruction.
- A successful Marble video generation can invent major foreground structure despite an accurate prompt. The fresh bottle-source world added closed glazed doors across an open landing and changed the corridor beyond. Eleven local camera views did not recover the source layout; changing scale cannot repair the first-camera view because its translation is zero. Preserve this as a generated-world fidelity failure, not a successful placement or permission to regenerate. Evidence: `.context/evidence/fresh-room-origin-diagnostic-20261018/`.
- An explicitly approved image-conditioned replacement restored that open landing, but source-matched beginning/middle/end views still showed a missing staircase through the right portal and an invented raised wedge at the central column. A good first view is not full-clip layout acceptance. Sparse, ambiguous source/render feature matches did not justify a rigid registration correction. Evidence: `.context/evidence/fresh-room-origin-diagnostic-20261018/image-repair-01/`.
- Counting every generated splat in a camera frustum against frontmost source depth mistakes hidden rear geometry for registration error; duplicating hidden splats can even move the fitted median. The scale diagnostic now compares one generated front-center depth per supported source texel, retains the old distribution separately, and reports overlap counts. Synthetic visible near/far errors still fail the unchanged bands. This center z-buffer is an approximation, not opacity rendering or proof of scene coverage; the image replacement's visible geometry still needs independent review.
- GEN3C as a static fill generator lost to the FLUX panorama fill.
- A dominant low horizontal plane can be a countertop rather than a walking floor. Kitchen source-labelled floor tiles lie about 0.61 body-heights below its selected counter plane; validate surface identity before using camera-height or walker-floor estimates.
- A track with no usable complete-body height does not constrain body scale through vertical silhouette edges alone. In the retained hall-walk shot, head/floor fitting increased visible body area and reduced source overlap despite passing the camera-height check. Tracker-derived crop flags are also fallible: at six seconds its projected feet lie outside the image although the source boots are visible. A visible-width experiment improved overlap but still failed area and contact-noise checks. Preserve the failed verdict and investigate tracking disagreement; a size-check pass does not establish accurate placement.
- Main-thread `toBlob` and `OffscreenCanvas.convertToBlob` took about four seconds per spectator image during native Quest XR sessions. Moving JPEG encoding to a dedicated worker reduced observed encoding latency to roughly 25–40 ms; simulated XR did not reproduce the main-thread stall.

- A source MOV containing H.264 video and PCM audio can stall its browser video clock after Web Audio unlock, despite reporting `paused=false`, `readyState=4` and no media error. The same encoded video in a silent MP4 advanced under the same viewer settings. Packaging now stream-copies compatible video to MP4 and the original PCM to an external WAV; packet timestamps and samples must remain identical. This is a delivery compatibility failure, not an animation or inference failure. Source/media proof: `.context/evidence/bottle-source-rerun/media-compatibility/`.
27 changes: 25 additions & 2 deletions docs/objects.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@ The writer emits `schema: "wander.objects/2"`, plus `clip`, `fps`, `samples`, `s
`timestamps`, `metresPerWorldUnit`, `coordinates`, `people: "people.json"`, and `objects`.
The top-level sample grid comes from `people.json`; it differs from the baked object's frame grid.

New fits and packages also carry `coordinateBasis`, repeated in object `evidence`. It identifies
the scale source and fingerprints the person registration scales, vertical constants, and shared
drift on the source-frame grid. Packaging rejects a fit made against a different basis; re-lift
after changing those corrections.

| Object field | Meaning and current behavior |
|---|---|
| `id`, `label`, `prompt` | Stable object identifier, display name, and segmentation prompt. |
Expand Down Expand Up @@ -122,7 +127,7 @@ Explicit `--object` requests still enable the stage. Without a named request, it

1. Runs `detect_object_flights.py` using camera poses, tracks, masks, and a metre scale.
2. Stops with an empty result if no flights pass. Single-person worlds can record detections,
but cannot package them through this lane without multiperson floor fits and wrist poses.
but this lane needs a `people.json` package and per-track wrist poses to package them.
3. Runs `lift_object_3d.py`, retaining flights that connect tracked wrists at both ends.
4. Runs `describe_object.py` on observed crops; without `OPENAI_API_KEY` it uses `--no-vlm`.
5. Optionally calls `worker/modal_image_to_3d.py`, then `object_appearance.py`. `--no-shape`
Expand All @@ -145,10 +150,28 @@ uv run --locked --group inference scripts/package_objects.py \
--size-m 0.06,0.12,0.06 --shape-provenance 'Proxy; no reconstructed model'
```

This requires `people.json` with `floorFit`, per-track `source-poses.pt`, and a reviewed
This requires `people.json`, per-track `source-poses.pt`, and a reviewed
`wander.object-fit/2` fit. A `/1` fit additionally needs `--thrower` and `--catcher`.
Optional `--model`, `--orientation`, and provenance flags add reviewed appearance/rotation.

Lifting and packaging share two supported coordinate bases. A fresh `package_multiperson.py`
manifest has identity person transforms and no `floorFit`: its scale is
`1.70 / sharedPlacement.bodyHeightUnits`, matching the detector's **avatar stature prior**.
Shared drift and person vertical constants are zero; no floor measurement or metric anchor is
invented. Its objects set `appliesSharedCameraDrift: false`. Sizes, gravity and wrist distances
expressed in metres remain conditional on that assumed stature.
The historical fit flag `driftCorrected: true` and segment space `driftCorrected` describe the
coordinate convention after applying the chosen basis corrections. For this fresh basis those
corrections are zero; these fields do not claim that camera drift was measured or removed.

Legacy `floorFit` packages retain their metre scale, interpolated `sharedCameraDrift`, and
`C_` level `translationY` constants. A present but incomplete or nonfinite floor fit fails instead
of falling back to the stature prior. Person rotation, scale, and horizontal translation overrides
are unsupported; fresh packages also reject vertical translation overrides. Legacy vertical
translation is replaced by the existing level C constants. Fits without `coordinateBasis` remain
accepted only with legacy floor-fit packages; fresh packages require a newly lifted, tagged fit.
Reprojection, observation-count, wrist-contact, and source-clock gates apply unchanged in both bases.

The named `--object ID:MOTION:PROMPT[:ROI]` lane is manual/experimental: it segments an ROI,
uses fixed thrower/catcher indices for flight fitting, and reuses a supplied model when present.
Attached/worldDynamic requests currently omit the packager's required `--fit`; repeated named
Expand Down
Loading
Loading