Skip to content

fix: keep the FAILED status of a launch that never started - #25

Merged
florian-simvia merged 1 commit into
mainfrom
fix/pending-after-failed-launch
Sep 11, 2026
Merged

florian-simvia merged 1 commit into
mainfrom
fix/pending-after-failed-launch

Conversation

@florian-simvia

Copy link
Copy Markdown
Collaborator

What

registry_transaction now saves in a finally, so a write made before the body
raises is no longer discarded. Five lines of production code.

Why

_launch_local and _launch_slurm record status=FAILED and then raise to
report the failure. registry_transaction only saved when the body completed
normally, so that write was thrown away and the case stayed PENDING forever,
with nothing to tell the user it never ran. Those update_case(..., FAILED)
calls were effectively dead code.

Reproduced before the fix:

with registry_transaction(runs_dir) as registry:
    update_case(registry, "case0001", status="PENDING")
try:
    with registry_transaction(runs_dir) as registry:
        update_case(registry, "case0001", status="FAILED")   # what _launch_local writes
        raise RuntimeError("docker not found")               # then it raises
except RuntimeError:
    pass
load_registry(runs_dir)["case0001"]["status"]   # -> "PENDING"

Triggers in practice: no docker or sbatch on PATH, a solver binary that is
not executable, a rejected sbatch submission.

Call site audit

registry_transaction has ten call sites, so the semantics change was audited
one by one before touching it:

Site Writes before a possible raise? Effect
runner._start_case yes, status=FAILED then raises the fix
runner x3 (pending mark, running snapshot, finalize) no raise in the body unchanged
web_support.kill_case (lookup) raises before writing; later writes are discovered container_id / job_id correct to keep
web_support.kill_case (final), api_case_note one write, nothing raises unchanged
api_case_convergence raises before the write unchanged
api_cleanup_cases loop of writes, OSError possible correct, those cases really were reset
doe.generate_cases writes entries while creating case dirs improvement, see below

generate_cases gains from the change. When prepare aborts part way, the case
directories it already created are not rolled back, so discarding their registry
entries bought no atomicity: it only left the filesystem and the registry out of
step. They now stay in agreement.

How to test

pytest -q   # 383 passed, 3 skipped

Three new tests, two written first and observed failing: registry_transaction
persists a write made before an exception, it still saves normally without one,
and run_cases leaves the case FAILED rather than PENDING when Popen
raises.

Note on real-world verification: forcing the PENDING path end to end means
removing docker from PATH or having Slurm reject a submission, since docker
starts before failing on a bad image. That path is covered by the unit test
where Popen raises, not by a physical reproduction.

Checklist

  • Tests pass locally (pytest -q)
  • No linting errors (ruff check . && ruff format --check .)
  • Frontend builds — not applicable, no frontend file touched
  • frontend/dist/ is up-to-date — not applicable, unchanged

_launch_local and _launch_slurm record status=FAILED and then raise to report
the failure, but registry_transaction only saved when the body completed
normally, so that write was discarded and the case stayed PENDING forever with
no way to tell it never ran.

The save now runs in a finally. Audited every call site: the other nine either
raise before writing anything, or write once with nothing that can raise, or
(cleanup, kill) record discoveries that are correct to keep. generate_cases
gains from it: the case directories it already created on disk were never
rolled back either, so discarding their registry entries only left the two
out of step.
@florian-simvia
florian-simvia merged commit 6a74ffd into main Sep 11, 2026
6 checks passed
@florian-simvia
florian-simvia deleted the fix/pending-after-failed-launch branch September 14, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant