Repository navigation
fix(orchestrator): stop hands-off paid work once the package is unreachable - #73
Open
austinjiann wants to merge 2 commits into
Open
austinjiann wants to merge 2 commits into
austinjiann wants to merge 2 commits into
Conversation
…chable A blocked or failed paid prerequisite of the package is final in hands-off mode, so further paid stages (such as the world generation) can no longer lead to a preview. Refuse them and finish the run as failed. Observed on a real upload whose pi3x was blocked by the shared ledger while clean and avatars continued.
…ite is failed A real upload with no fully visible person failed person_prep, an unpaid step the package needs, while the world generation would still have launched. Paid launches are now held (not terminal) until that step succeeds on a rerun.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The first real hands-off upload trial used
lobby.mp4on the live runs root with a one-upload admission limit. Itspi3xwas blocked before submission because the shared stage ledger already holds a historical pi3x import for that source SHA, and a second execution needs an explicit hypothesis that a hands-off run cannot supply.The agent then kept launching independent paid stages: clean and lhm_frozen on Modal. The World Labs generation would have followed once review passed. None of these could ever produce a package or a preview. The operator paused the run before the generation.
Change
orchestrator/run_policy.py: a hands-off run now refuses a new paid step when a paid prerequisite of the package step (packageorpackage_people) has finishedfailedorblocked. Hands-off mode never repeats a paid step, so that failure is final. The refusal follows the existing admission path and finishes the runfailedwith delivery metadata. Unpaid steps and failures that don't affect the package are ignored. Steps already running are unaffected.docs/automatic-uploads.md: documents this rule and that re-uploading an already processed source is blocked by the ledger.Validation
state.jsoncounts.bun run format:checkpass.marble_video,lhm_motionandobjectsare now refused, with pi3x named as the cause.Trial spend so far: Modal ≈ $0.072 estimated (clean $0.041, lhm_frozen $0.031), 1 OpenAI VLM call, 0 World Labs credits.
Second finding (ebaed0f)
The second trial upload (a never-processed 30.5 s phone clip of two seated people, both cut off by the frame edges) failed
person_prepwith an unusable non-portrait mask. That step is unpaid and deterministic, but the first rule only counted paid failures, somarble_video(1,600 credits) would still have launched. The run was paused before it did.marble_videoandlhm_frozenbecauseperson_prepfailed.