Skip to content

fix: stop reporting crashed cases as RUNNING forever - #24

Merged
florian-simvia merged 1 commit into
mainfrom
fix/stuck-running-after-crash
Sep 11, 2026
Merged

florian-simvia merged 1 commit into
mainfrom
fix/stuck-running-after-crash

Conversation

@florian-simvia

Copy link
Copy Markdown
Collaborator

What

A run launched from the web UI that crashed before the solver started stayed
RUNNING indefinitely. Two independent causes, either of which masked the other:

  • is_process_alive treated a zombie as alive. os.kill(pid, 0) succeeds on a
    process that exited but was never reaped, which is what every run launched by
    the long-lived server becomes: run_cases spawns with Popen and never
    waits. Zombies are now reported as dead, and reaped when we are the parent so
    they stop accumulating in the server's process table. Without procfs the
    previous behaviour stands.
  • CodeSaturneAdapter.detect_outcome now reads the run_status.failed and
    run_status.exceeded_time_limit markers when the logs give no verdict. A
    failure during preprocessing writes no run_solver.log at all, only a marker
    beside it.

Why

Neither defect is visible on its own, which is why this survived so long:

  • The zombie is masked by the second branch of _compute_refresh_result, which
    trusts the log end markers. A successful run launched from the web also
    leaves a zombie, but END OF CALCULATION still
  • The blind detect_outcome is masked by the `or h
    fires as soon as the PID dies. From the CLI the parent exits immediately, init
    reaps the child, and the case is finalised normally.

Only **web launch + a failure that produces no sol.

The marker names come from code_saturne's own cs_ maps nine run_status.files to case statesexceeded_time_limitmean failure;finished, saving, running, preparing, prepared, preprocessingandready` are progress states, and a
normal completion removes the marker altogether.

Placement

run_status.* is a code_saturne convention, so th
solvers/code_saturne.py, not in logs.py. Zombie handling is POSIX process
management with no solver knowledge in it, so it stays in runner.py.

How to test

pytest -q   # 380 passed, 3 skipped

Nine new tests, the two defect ones written first e
non-regression guards: a live process is still ali
failures, and a stale marker from a previous run n
run's log.

End to end, with a campaign whose mesh path does n

csauto serve RUNS      # launch the case from the Status panel

The case now goes RUNNING then FAILED once the con
successive reads of /api/status all returned RUNNING, indefinitely.

Checklist

- [x] Tests pass locally (pytest -q)
- [x] No linting errors (ruff check . && ruff form
- [ ] Frontend builds — not applicable, no fronten
- [ ] frontend/dist/ is up-to-date — not applicable, unchanged

A run launched from the web UI that crashed before the solver started stayed
RUNNING indefinitely. Two independent causes, either of which masked the other:

- is_process_alive treated a zombie as alive. os.kill(pid, 0) succeeds on a
  process that exited but was never reaped, which is what every run launched
  by the long-lived server becomes, since run_cases spawns with Popen and
  never waits. Zombies are now dead, and reaped when we are the parent. This
  is process management, not solver knowledge, so it stays in runner.py.
- CodeSaturneAdapter.detect_outcome now reads the run_status.failed and
  run_status.exceeded_time_limit markers when the logs give no verdict. A
  failure during preprocessing writes no run_solver.log at all, only a marker
  beside it. The names come from code_saturne's own cs_case.py state table;
  the other markers it writes are progress states. The check lives in the
  adapter, not in logs.py, because run_status.* is a code_saturne convention.
@florian-simvia
florian-simvia merged commit 5ec1f14 into main Sep 11, 2026
6 checks passed
@florian-simvia
florian-simvia deleted the fix/stuck-running-after-crash branch September 11, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant