fix: stop reporting crashed cases as RUNNING forever - #24
Merged
Merged
Conversation
A run launched from the web UI that crashed before the solver started stayed RUNNING indefinitely. Two independent causes, either of which masked the other: - is_process_alive treated a zombie as alive. os.kill(pid, 0) succeeds on a process that exited but was never reaped, which is what every run launched by the long-lived server becomes, since run_cases spawns with Popen and never waits. Zombies are now dead, and reaped when we are the parent. This is process management, not solver knowledge, so it stays in runner.py. - CodeSaturneAdapter.detect_outcome now reads the run_status.failed and run_status.exceeded_time_limit markers when the logs give no verdict. A failure during preprocessing writes no run_solver.log at all, only a marker beside it. The names come from code_saturne's own cs_case.py state table; the other markers it writes are progress states. The check lives in the adapter, not in logs.py, because run_status.* is a code_saturne convention.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A run launched from the web UI that crashed before the solver started stayed
RUNNINGindefinitely. Two independent causes, either of which masked the other:is_process_alivetreated a zombie as alive.os.kill(pid, 0)succeeds on aprocess that exited but was never reaped, which is what every run launched by
the long-lived server becomes:
run_casesspawns withPopenand neverwaits. Zombies are now reported as dead, and reaped when we are the parent so
they stop accumulating in the server's process table. Without procfs the
previous behaviour stands.
CodeSaturneAdapter.detect_outcomenow reads therun_status.failedandrun_status.exceeded_time_limitmarkers when the logs give no verdict. Afailure during preprocessing writes no
run_solver.logat all, only a markerbeside it.
Why
Neither defect is visible on its own, which is why this survived so long:
_compute_refresh_result, whichtrusts the log end markers. A successful run launched from the web also
leaves a zombie, but
END OF CALCULATIONstilldetect_outcomeis masked by the `or hfires as soon as the PID dies. From the CLI the parent exits immediately, init
reaps the child, and the case is finalised normally.
Only **web launch + a failure that produces no sol.
The marker names come from code_saturne's own
cs_ maps ninerun_status.files to case statesexceeded_time_limitmean failure;finished,saving,running,preparing,prepared,preprocessingandready` are progress states, and anormal completion removes the marker altogether.
Placement
run_status.*is a code_saturne convention, so thsolvers/code_saturne.py, not inlogs.py. Zombie handling is POSIX processmanagement with no solver knowledge in it, so it stays in
runner.py.How to test