fix(manager): DeviceManager startup cancel 不再殘留 zombie read_loop#137
Open
csp0924 wants to merge 2 commits into
Open
fix(manager): DeviceManager startup cancel 不再殘留 zombie read_loop#137csp0924 wants to merge 2 commits into
csp0924 wants to merge 2 commits into
Conversation
`_on_start` 中途被 cancel 時,finally block 只 rollback `_running = False`, 後續 `mgr.stop()` 命中 `if not self._running: return` early-return,已成功 `start()` 的 standalone device 之 read_loop task 永遠不會被取消 → zombie 殘留。 修法: - _on_start 追蹤已成功 start() 的 standalone device、已 ensure_event_loop_started 的 group device、與已 start 的 group。 - finally rollback 路徑(startup_completed=False)改為先呼叫 `_rollback_partial_startup(...)` 對這些目標反向 stop+disconnect(best-effort, 抓 BaseException 才能在 outer 已 cancel 的情境下仍完成清理),再翻回 `_running = False`。 - 不改 `_on_stop` 早 return 語意,也不改 public API。 涵蓋場景(closed-loop probe H3):N=30(15 fast + 15 slow),cancel at 0.3s ─ 修復前 15 zombie read_loop tasks,修復後 0。
There was a problem hiding this comment.
Pull request overview
This PR addresses a DeviceManager lifecycle bug where cancelling DeviceManager._on_start() mid-startup could leave already-started standalone devices’ background read_loop tasks running indefinitely (“zombie tasks”), because _running was rolled back to False and later stop() became a no-op.
Changes:
- Track partially-started standalone devices, group devices, and groups during
_on_start()to enable cleanup on startup cancellation/failure. - Add
_rollback_partial_startup()to best-effort stop/disconnect devices (and stop groups) when startup doesn’t complete. - Add regression tests reproducing the startup-cancel zombie-task scenario and asserting no task leaks.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
csp_lib/manager/device/manager.py |
Adds partial-startup tracking and rollback cleanup to prevent zombie tasks when startup is cancelled. |
tests/manager/test_device_manager_startup_cancel.py |
New async regression tests for startup cancellation cleanup and task-leak prevention. |
Comments suppressed due to low confidence (2)
csp_lib/manager/device/manager.py:468
_rollback_partial_startup()宣稱 BaseException-safe,但這兩個await asyncio.gather(...)仍是 cancellable await point:在 startup cancel 的 standalone-only 情境下(started_groups為空)它們會成為 rollback 的第一個 await,容易直接被CancelledError打斷而完全跳過 cleanup。建議避免在 rollback 路徑使用未 shield 的 gather(例如改成逐台順序收尾並在每次 await 內捕捉asyncio.CancelledError,或用 shield 包住 gather 並確保 inner cleanup 完成後再返回)。
if started_standalone:
await asyncio.gather(
*(_rollback_standalone(d) for d in started_standalone),
return_exceptions=True,
)
if started_group_devices:
await asyncio.gather(
*(_rollback_group_device(d) for d in started_group_devices),
return_exceptions=True,
)
tests/manager/test_device_manager_startup_cancel.py:202
- 檔尾這個 autouse fixture 目前是 no-op(只
yield),也沒有實際「silence warnings」或「接管 event loop」的行為,且註解描述與實作不一致。建議移除以避免誤導,或改為真正的 warnings/unraisable 捕捉設定(例如用pytest.warns/warnings.filterwarnings/ pytest 的 unraisablehook fixture)並在 docstring 說明其必要性。
# 提供給 pytest 的 fixture 接管 event loop(與其他測試慣例一致)
@pytest.fixture(autouse=True)
def _silence_asyncio_warnings() -> Any:
"""避免測試環境因 unraisable warning 而 noise。"""
yield
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
387
to
398
| finally: | ||
| if not startup_completed: | ||
| # 對已成功 start() 的 standalone device best-effort stop+disconnect, | ||
| # 避免 read_loop task 變 zombie;對已 ensure_event_loop_started 的 | ||
| # group device 反向 ensure_event_loop_stopped + disconnect;對已 start | ||
| # 的 group 也要 stop()。rollback 採 best-effort,個別失敗只 warn。 | ||
| await self._rollback_partial_startup( | ||
| started_standalone, | ||
| started_group_devices, | ||
| started_groups, | ||
| ) | ||
| self._running = False |
Comment on lines
+106
to
+112
| start_task = asyncio.create_task(mgr.start(), name="mgr.start") | ||
| # 等 fast device 完成 connect+start(read_loop 已建立),slow 仍卡 connect | ||
| await asyncio.sleep(0.15) | ||
|
|
||
| # sanity:fast device 應已 start() | ||
| assert all(d.start_count == 1 for d in fast), f"fast 應已 start:{[d.start_count for d in fast]}" | ||
| assert all(d.start_count == 0 for d in slow), "slow 不應 start" |
回應 PR #137 Copilot review: 1. DeviceManager._on_start finally 內的 _rollback_partial_startup 原本仍可能被 caller 二次 cancel 打斷,導致 rollback 未跑完 → zombie read_loop 殘留。改用 asyncio.ensure_future + asyncio.shield 包住 rollback task,並以 current.uncancel() 吸收 cancellation 直到 rollback done; finally 結束後原 CancelledError 自然向上傳播保留取消語意。 2. tests/manager/test_device_manager_startup_cancel.py 原本依賴固定 asyncio.sleep(0.15/0.05) 等待 fast device 啟動 / rollback 完成, CI 排程下容易 flaky。改用既有的 tests.helpers.wait_for_condition 輪詢 start_count / read_loop_alive / mgr.is_running。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
問題
Closed-loop probe 發現:
DeviceManager._on_start啟動中被 cancel 時,已完成start()的 standalone device 的read_looptask 永不被清理。finallyblock 只 rollback_running = False,後續mgr.stop()因if not self._running: returnearly-return(manager.py:393-394),zombie task 永生。Sandbox 證據(
sandbox/device_manager_reconnect_storm_demo.pyH3 scenario):N=30 (15 fast + 15 slow)、cancel @ 0.3s → 15 個read_looptask 在 explicitstop()後仍 alive。改動(Option A:追蹤已啟動 device,rollback 時反向收尾)
csp_lib/manager/device/manager.py::_on_start追蹤已成功啟動的 standalone / group device / group_rollback_partial_startup():在 rollback finally 中反向stop+disconnect(best-effort、BaseException-safe)_on_stopearly-return 語意(沒啟動過的 managerstop()仍然 no-op)驗證
tests/manager/test_device_manager_startup_cancel.py(3 案,含 reproduce + post-fix assertion)