[Bug]: camera_video publishes scratch-bound artifact URIs as durable, causing silent byte-replacement or hard deletion on sync rewrite

[Sagar-024]

Version or commit

Current main

Environment

Ubuntu 24.04 (WSL2), Python 3.12, x86_64

Minimal reproduction

Using a local-root App with camera_video registered:

1. Run the full pipeline for generation N. The video: artifact URI is published pointing inside scratch/ (e.g., .../scratch/wrist_cam_compressed.mp4).
2. Bump the pipeline version and run a {SYNC, META} reprocess (generation N+1). The canonical is rewritten, and the scratch directory is rmtree'd and rebuilt.
3. Query measurements_latest for the generation N episode's video: artifact URI.

Expected behavior

The durable catalog URI for a published artifact must point to a stable, content-bound location (like run_dir/media/ or run_dir/artifacts/). If the underlying bytes change generations, the URI must either be invalidated, or the digest must change, refusing to serve unverified bytes.

Actual behavior

The publish loop (app.py:2686-2701) accepts the scratch-cached remux as a durable artifact.

• On a {SYNC, META} reprocess, the scratch directory is wiped and rebuilt. The generation N artifact URI now silently resolves to generation N+1's bytes. The server's is_uri_servable check still passes (the file "exists"), so a downstream reviewer gets the wrong pixels with zero errors.
• On a forced sync-only reprocess (where the labels/media stages don't run), the generation N artifact URI resolves to a missing file (404), even though the catalog claims the artifact exists.

Executed proof:
[N] episode_id=0c12703341c7e653
[N] artifact URI: .../scratch/wrist_cam_compressed.mp4 (exists: True, sha=6818cf269ee5)
[N+1] (pipeline bump) artifact URI now resolves to: sha=3d86896cddb6 (identical: False)
[N+2] (sync-only rewrite) artifact file exists: False

Note: The built-in contact sheet, stored under run_dir/media/, survives the shutil.rmtree(scratch_dir) correctly. Only the enrichment publish path fails this boundary.

Additional context

Root cause: The publish loop binds a durable catalog URI to a path inside the scratch directory's lifecycle, and scratch has no content identity of its own. This violates the contract in steps.py:405 which explicitly documents ep.workdir as scratch. Additionally, test_camera_video.py:66 has a flawed assertion (is_relative_to(data_root)) that allowed this to slip through because scratch/ is technically inside the data root.

Why it matters: The published video: artifact is exactly what a browser scrubber or human reviewer opens to trust a curation cut. After any reprocess, the reviewer's video may be a different generation's pixels (silently accepted) or missing. The evidence chain is broken.

Fix direction: In the publish loop, refuse to record artifact URIs under the scratch subtree. Stage such artifacts into run_dir/artifacts/ (like media/) and publish from there. Additionally, bind every published artifact to its own digest so a replaced file is detectable by a reader.

Definition of done:

• Regression test: after a {SYNC,META} reprocess, the generation-N artifact URI hashes to the exact bytes episode N recorded, or is explicitly refused.
• Mutation proof (a): drop the shutil.rmtree(scratch_dir) guard → test must fail on the deleted-file branch.
• Mutation proof (b): skip staging into artifacts/ → test must fail on the silent-replacement branch.
• test_camera_video.py:66 corrected to assert the URI is not under scratch/.

[Sagar-024]

@amansingh-121 did u wanna take it ??? if not i will take it

[amansingh-121]

@Sagar-024 yes i want to take this

[kstonekuan]

Confirmed, and the mechanism is the publish path rather than camera_video: app.py:2737 makes artifact_key the run-relative path when the artifact already sits under run_dir, and LocalStorageRoot.publish (storage.py:402) is then a no-op because source and destination are the same file. So the catalog URI points straight into scratch/.

#539 landed after you filed this and changes the severity. Scratch is now scratch/<episode_id>/ (app.py:2458), so a new canonical gets a new directory and generation N cannot resolve to generation N+1 bytes. What remains is the other half you named: app.py:2424 rmtrees the whole scratch_root on a sync rewrite, so the generation N URI dangles and the catalog claims an artifact that 404s.

Worth re-measuring the wrong-pixels case against current main before fixing, in case it is gone.