Repository navigation
Launcher: wait for the fail-stop cleanup before ending; reason = trigger - #68
Merged
Merged
Conversation
…ort the trigger as the reason _fail_stop sets FAILED before the cleanup (so nothing else starts), and the background worker ends on FAILED: its end stopped the cleanup, which runs in the daemon watch thread, halfway. On Windows (slow process stops) an asset killed during a run left the pacer and the scenario link running, with no cleanup in the failure report. Every path that ends on FAILED now waits for the cleanup (up to 60 s). A fail-stop not found by the health check (an asset exit, a watch exception) no longer takes the last periodic check as its status and reason (often UNKNOWN 'OS file lock is busy' while the master runs): the reason names the trigger, the check is kept as last_health_check. Found by the Windows hakoniwa-core-cpp#62 test run (test 4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by the Windows test run for toppers/hakoniwa-core-cpp#62 (test 4: taskkill of the Urban car plant during a run).
Cleanup cut short.
_fail_stopsets FAILED before running the cleanup, so nothing else starts. The background worker'sserve_until_terminatedleaves its loop on FAILED, and_run_background_workerreturns. That ended the process while the cleanup was still running in the daemon watch thread. On Windows, where process stops are slow, the pacer and the scenario link (4 Python processes) kept running,failure_report.cleanupwas null, and the Launcher process was gone. Every path that ends on FAILED now waits for the cleanup (LauncherService.wait_for_failure_cleanup, up to 60 s). These paths are the background worker'sserve_until_terminatedand its exception path, plus the foregroundimmediateandactivate-onlymodes.Misleading reason. The failure report started from the last periodic health report. While the master runs, that report is often UNKNOWN "OS file lock is busy", so an asset exit was reported as
fail-stop: OS file lock is busy. Only the health-check triggers (startup_runtime_integrity_abort,runtime_integrity_abort) now take the health report as their status and reason. For other triggers, the reason names the trigger (for exampleasset urban-car-fleet-plant exited unexpectedly (exit code 1)), and the last check is kept aslast_health_check.Tests: 2 new tests in
tests/test_launcher_fail_stop.py. The worker test fails without the fix. With the fix,PYTHONPATH=src pytest testsgives 136 passed.🤖 Generated with Claude Code