fix: keep the FAILED status of a launch that never started - #25
Merged
Merged
Conversation
_launch_local and _launch_slurm record status=FAILED and then raise to report the failure, but registry_transaction only saved when the body completed normally, so that write was discarded and the case stayed PENDING forever with no way to tell it never ran. The save now runs in a finally. Audited every call site: the other nine either raise before writing anything, or write once with nothing that can raise, or (cleanup, kill) record discoveries that are correct to keep. generate_cases gains from it: the case directories it already created on disk were never rolled back either, so discarding their registry entries only left the two out of step.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
registry_transactionnow saves in afinally, so a write made before the bodyraises is no longer discarded. Five lines of production code.
Why
_launch_localand_launch_slurmrecordstatus=FAILEDand then raise toreport the failure.
registry_transactiononly saved when the body completednormally, so that write was thrown away and the case stayed
PENDINGforever,with nothing to tell the user it never ran. Those
update_case(..., FAILED)calls were effectively dead code.
Reproduced before the fix:
Triggers in practice: no
dockerorsbatchon PATH, a solver binary that isnot executable, a rejected
sbatchsubmission.Call site audit
registry_transactionhas ten call sites, so the semantics change was auditedone by one before touching it:
runner._start_casestatus=FAILEDthen raisesrunnerx3 (pending mark, running snapshot, finalize)web_support.kill_case(lookup)container_id/job_idweb_support.kill_case(final),api_case_noteapi_case_convergenceapi_cleanup_casesOSErrorpossibledoe.generate_casesgenerate_casesgains from the change. Whenprepareaborts part way, the casedirectories it already created are not rolled back, so discarding their registry
entries bought no atomicity: it only left the filesystem and the registry out of
step. They now stay in agreement.
How to test
pytest -q # 383 passed, 3 skippedThree new tests, two written first and observed failing:
registry_transactionpersists a write made before an exception, it still saves normally without one,
and
run_casesleaves the caseFAILEDrather thanPENDINGwhenPopenraises.
Note on real-world verification: forcing the
PENDINGpath end to end meansremoving
dockerfrom PATH or having Slurm reject a submission, since dockerstarts before failing on a bad image. That path is covered by the unit test
where
Popenraises, not by a physical reproduction.Checklist
pytest -q)ruff check . && ruff format --check .)frontend/dist/is up-to-date — not applicable, unchanged