Slice B: runner robustness under failure — v0.10.0 - #168
Merged
Conversation
…ard, no stale process refs Slice B of the crash review — the failure paths that corrupted or wedged a run under exactly the load pytest-fly targets: - _TestRunner._run_single_test guards PytestProcess.start(): CreateProcess raising OSError (WinError 1455 at commit exhaustion) previously killed the worker thread — the dequeued test kept no terminal record and the supervision loop respawned a replacement every second that died the same way, draining the queue into lost tests. Now the test records TERMINATED and the worker backs off (escalating, capped at 30 s, stop-interruptible) before the next dequeue. - Backstop terminal record for an unclean child exit: a test child that dies without writing its result (native crash, os._exit, external kill) left the test "Running" forever, and the stall watchdog cannot flag a dead PID. The worker now checks the child's exit code and routes through mark_test_terminated_if_stale, which never overwrites a real result. - set_number_of_processes refuses a stopping/stopped runner. reconcile_process_count runs every GUI tick and is_running() stays True during wind-down, so a Processes change while a new Run wound down the old one started a second full worker pool running the old run's queue. A pending soft stop still allows resizing (Cancel Stop + resize keeps working). - _TestRunner.process is cleared between tests: force_stop_test matched a finished test's stale name and killed the worker's NEXT test; a hard stop while awaiting admission tree-killed the previous test's dead — possibly recycled — PID. A stop between tests now writes the dequeued test's TERMINATED record instead of killing nothing. - PytestRunner.run() can no longer die before _started_event is set (join(None) hung forever) or during soft-stop finalization; both DB phases are guarded fail-open. - _write_terminated_record consolidates the worker-side TERMINATED writes, fail-open. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEZ9TpT9VwBujBq1S3v2Jd
…ation — v0.10.0 - ProcessMonitor is now a daemon child. As a non-daemon child, any unguarded exception unwinding PytestProcess.run() made multiprocessing's exit handler *join* the monitor — whose loop never ends on its own — so the test process hung forever with no terminal record (the exact wedge the liveness machinery exists to prevent). - Coverage finalization (stop/save/unlink/move) is guarded: in RESUME mode the prior run's .coverage file exists and the aggregator child may have it open, so the unlink can raise WinError 32; a full disk fails coverage.save() the same way. Coverage data for that test is lost, but the result record is still written — previously the raise killed the child first and the test sat "Running" forever. - The live-output read for the final record is guarded the same way. New tests (test_pytest_runner_robustness.py) cover both commits: spawn-failure → TERMINATED + completion, unclean child exit → backstop record, resize-after-stop spawns nothing, process reference cleared between tests, monitor daemon flag. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEZ9TpT9VwBujBq1S3v2Jd
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #168 +/- ##
==========================================
- Coverage 86.08% 85.90% -0.19%
==========================================
Files 74 74
Lines 6168 6224 +56
==========================================
+ Hits 5310 5347 +37
- Misses 858 877 +19 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Slice B of the crash review: harden the runner's failure paths so resource exhaustion, native crashes, and mid-run control actions can no longer lose tests, wedge the run, or double the process load.
PytestProcess.start()raises (WinError 1455 at commit exhaustion)os._exit)mark_test_terminated_if_stale(never overwrites a real result)set_number_of_processesduring wind-down (called every GUI tick)_TestRunner.processbetween testsfinally; a stop between tests writes the dequeued test's TERMINATED recordPytestRunner.run()DB error before_started_event/ during finalizationjoin(None)blocked forever / runner thread died un-finalizedProcessMonitornon-daemonPytestProcess.run()made the exit handler join the never-ending monitor → test process hung foreverdaemon=TrueTests
tests/test_pytest_runner/test_pytest_runner_robustness.py(5 new): spawn-failure → TERMINATED +is_user_complete, unclean child exit → backstop record, resize-after-stop spawns nothing,processcleared between tests, monitor daemon flag.Full suite: 497 passed.
ruffclean;tyunchanged (13 pre-existing diagnostics).Behaviour changes
🤖 Generated with Claude Code
https://claude.ai/code/session_01KEZ9TpT9VwBujBq1S3v2Jd