Skip to content

fix(orchestrator): stop hands-off paid work once the package is unreachable - #73

Open
austinjiann wants to merge 2 commits into
mainfrom
codex/hands-off-fail-fast
Open

austinjiann wants to merge 2 commits into
mainfrom
codex/hands-off-fail-fast

Conversation

@austinjiann

@austinjiann austinjiann commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The first real hands-off upload trial used lobby.mp4 on the live runs root with a one-upload admission limit. Its pi3x was blocked before submission because the shared stage ledger already holds a historical pi3x import for that source SHA, and a second execution needs an explicit hypothesis that a hands-off run cannot supply.

The agent then kept launching independent paid stages: clean and lhm_frozen on Modal. The World Labs generation would have followed once review passed. None of these could ever produce a package or a preview. The operator paused the run before the generation.

Change

  • orchestrator/run_policy.py: a hands-off run now refuses a new paid step when a paid prerequisite of the package step (package or package_people) has finished failed or blocked. Hands-off mode never repeats a paid step, so that failure is final. The refusal follows the existing admission path and finishes the run failed with delivery metadata. Unpaid steps and failures that don't affect the package are ignored. Steps already running are unaffected.
  • docs/automatic-uploads.md: documents this rule and that re-uploading an already processed source is blocked by the ledger.
  • No thresholds, ledger rules or retry caps changed.

Validation

  • 2 new tests:
    • blocked or failed prerequisite → no launch, run failed;
    • unpaid or unrelated failures still admit work, and a failed record in state.json counts.
  • Full Python suite: 1,527 passed. Ruff and bun run format:check pass.
  • Read-only check against the real paused lobby run: marble_video, lhm_motion and objects are now refused, with pi3x named as the cause.

Trial spend so far: Modal ≈ $0.072 estimated (clean $0.041, lhm_frozen $0.031), 1 OpenAI VLM call, 0 World Labs credits.

Second finding (ebaed0f)

The second trial upload (a never-processed 30.5 s phone clip of two seated people, both cut off by the frame edges) failed person_prep with an unusable non-portrait mask. That step is unpaid and deterministic, but the first rule only counted paid failures, so marble_video (1,600 credits) would still have launched. The run was paused before it did.

  • Paid launches in a hands-off run are now held, not terminated, while an unpaid prerequisite of the package has a failed latest outcome.
  • Rerunning that unpaid step is still admitted, and a later success lifts the hold.
  • New test added. Full suite: 1,528 passed.
  • A read-only check on the real paused run holds marble_video and lhm_frozen because person_prep failed.

…chable

A blocked or failed paid prerequisite of the package is final in hands-off mode,
so further paid stages (such as the world generation) can no longer lead to a
preview. Refuse them and finish the run as failed. Observed on a real upload
whose pi3x was blocked by the shared ledger while clean and avatars continued.
…ite is failed

A real upload with no fully visible person failed person_prep, an unpaid step
the package needs, while the world generation would still have launched. Paid
launches are now held (not terminal) until that step succeeds on a rerun.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant