fix(camera_video): stage scratch-bound artifacts to durable artifacts/ directory - #548
Closed
amansingh-121 wants to merge 1 commit into
Closed
amansingh-121 wants to merge 1 commit into
amansingh-121 wants to merge 1 commit into
Conversation
…/ directory When camera_video (and other enrichments) produce artifacts in the scratch directory, the publish loop was treating them as durable. On reprocess, scratch is cleaned and these artifacts become stale or missing. Now the publish loop detects if an artifact is under scratch_dir and copies it to run_dir/artifacts/ before publishing, ensuring the URI remains valid across reprocess cycles. Also updates the test assertion in test_camera_video.py to verify that published artifacts are NOT under scratch.
|
👋 Hi @amansingh-121 — thanks for the contribution! To keep starter issues available You already have #542 open, so this one is being closed automatically. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #545.
Problem
When
camera_video(and other enrichments) produce artifacts in thescratch/directory, the publish loop was treating them as durable. The scratch directory is cleaned onSYNCstage, so artifacts published to scratch become stale or missing after a reprocess. This breaks the evidence chain - users opening the artifact URI get wrong pixels or a 404 with no indication of the problem.Root Cause
In
app.py, the artifact publishing logic checked if an artifact was underrun_dirand, if so, used its relative path directly as the artifact key. Sincescratchis insiderun_dir, files in scratch were published without any consideration that they would be deleted on the next run.The comment at
test_camera_video.py:66(assert published_mp4.is_relative_to(data_root)) was flawed because it passed even when the artifact was in scratch (paradoxically, scratch is inside data_root in the test setup).Solution
In the publish loop (
app.py), add a check to detect if an artifact is underscratch_dir. If so:run_dir/artifacts/<step>/locationThis ensures the published URI remains valid across reprocess cycles.
Changes
src/hflow/app.py: Add scratch detection and staging logic before publishingtests/test_camera_video.py: Update assertion to verify artifacts are NOT under scratchTesting
ruff checkandruff format --checkpassfcntlimport and missingffmpeg, but CI coverage is identical to the existing test setupDefinition of Done Met
"scratch" not in published_mp4.parts)