Skip to content

fix(manager): DeviceManager startup cancel 不再殘留 zombie read_loop#137

Open
csp0924 wants to merge 2 commits into
mainfrom
fix/device-manager-startup-cancel-zombies
Open

fix(manager): DeviceManager startup cancel 不再殘留 zombie read_loop#137
csp0924 wants to merge 2 commits into
mainfrom
fix/device-manager-startup-cancel-zombies

Conversation

@csp0924

@csp0924 csp0924 commented May 12, 2026

Copy link
Copy Markdown
Owner

問題

Closed-loop probe 發現:DeviceManager._on_start 啟動中被 cancel 時,已完成 start() 的 standalone device 的 read_loop task 永不被清理。finally block 只 rollback _running = False,後續 mgr.stop()if not self._running: return early-return(manager.py:393-394),zombie task 永生。

Sandbox 證據(sandbox/device_manager_reconnect_storm_demo.py H3 scenario):N=30 (15 fast + 15 slow)、cancel @ 0.3s → 15 個 read_loop task 在 explicit stop() 後仍 alive。

改動(Option A:追蹤已啟動 device,rollback 時反向收尾)

  • csp_lib/manager/device/manager.py::_on_start 追蹤已成功啟動的 standalone / group device / group
  • 新增 _rollback_partial_startup():在 rollback finally 中反向 stop + disconnect(best-effort、BaseException-safe)
  • 保留 _on_stop early-return 語意(沒啟動過的 manager stop() 仍然 no-op)
  • 無 public API 變更(patch bump)

驗證

  • 新增 tests/manager/test_device_manager_startup_cancel.py(3 案,含 reproduce + post-fix assertion)
  • Manager 層 84 tests pass、整體 3680 tests pass、ruff / mypy clean
  • Sandbox H3 scenario 重跑:zombie count 從 15/15 降為 0/0

`_on_start` 中途被 cancel 時,finally block 只 rollback `_running = False`,
後續 `mgr.stop()` 命中 `if not self._running: return` early-return,已成功
`start()` 的 standalone device 之 read_loop task 永遠不會被取消 → zombie 殘留。

修法:

- _on_start 追蹤已成功 start() 的 standalone device、已 ensure_event_loop_started
  的 group device、與已 start 的 group。
- finally rollback 路徑(startup_completed=False)改為先呼叫
  `_rollback_partial_startup(...)` 對這些目標反向 stop+disconnect(best-effort,
  抓 BaseException 才能在 outer 已 cancel 的情境下仍完成清理),再翻回
  `_running = False`。
- 不改 `_on_stop` 早 return 語意,也不改 public API。

涵蓋場景(closed-loop probe H3):N=30(15 fast + 15 slow),cancel at 0.3s ─
修復前 15 zombie read_loop tasks,修復後 0。
Copilot AI review requested due to automatic review settings May 12, 2026 23:01

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a DeviceManager lifecycle bug where cancelling DeviceManager._on_start() mid-startup could leave already-started standalone devices’ background read_loop tasks running indefinitely (“zombie tasks”), because _running was rolled back to False and later stop() became a no-op.

Changes:

  • Track partially-started standalone devices, group devices, and groups during _on_start() to enable cleanup on startup cancellation/failure.
  • Add _rollback_partial_startup() to best-effort stop/disconnect devices (and stop groups) when startup doesn’t complete.
  • Add regression tests reproducing the startup-cancel zombie-task scenario and asserting no task leaks.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
csp_lib/manager/device/manager.py Adds partial-startup tracking and rollback cleanup to prevent zombie tasks when startup is cancelled.
tests/manager/test_device_manager_startup_cancel.py New async regression tests for startup cancellation cleanup and task-leak prevention.
Comments suppressed due to low confidence (2)

csp_lib/manager/device/manager.py:468

  • _rollback_partial_startup() 宣稱 BaseException-safe,但這兩個 await asyncio.gather(...) 仍是 cancellable await point:在 startup cancel 的 standalone-only 情境下(started_groups 為空)它們會成為 rollback 的第一個 await,容易直接被 CancelledError 打斷而完全跳過 cleanup。建議避免在 rollback 路徑使用未 shield 的 gather(例如改成逐台順序收尾並在每次 await 內捕捉 asyncio.CancelledError,或用 shield 包住 gather 並確保 inner cleanup 完成後再返回)。
        if started_standalone:
            await asyncio.gather(
                *(_rollback_standalone(d) for d in started_standalone),
                return_exceptions=True,
            )
        if started_group_devices:
            await asyncio.gather(
                *(_rollback_group_device(d) for d in started_group_devices),
                return_exceptions=True,
            )

tests/manager/test_device_manager_startup_cancel.py:202

  • 檔尾這個 autouse fixture 目前是 no-op(只 yield),也沒有實際「silence warnings」或「接管 event loop」的行為,且註解描述與實作不一致。建議移除以避免誤導,或改為真正的 warnings/unraisable 捕捉設定(例如用 pytest.warns / warnings.filterwarnings / pytest 的 unraisablehook fixture)並在 docstring 說明其必要性。
# 提供給 pytest 的 fixture 接管 event loop(與其他測試慣例一致)
@pytest.fixture(autouse=True)
def _silence_asyncio_warnings() -> Any:
    """避免測試環境因 unraisable warning 而 noise。"""
    yield

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 387 to 398
finally:
if not startup_completed:
# 對已成功 start() 的 standalone device best-effort stop+disconnect,
# 避免 read_loop task 變 zombie;對已 ensure_event_loop_started 的
# group device 反向 ensure_event_loop_stopped + disconnect;對已 start
# 的 group 也要 stop()。rollback 採 best-effort,個別失敗只 warn。
await self._rollback_partial_startup(
started_standalone,
started_group_devices,
started_groups,
)
self._running = False
Comment on lines +106 to +112
start_task = asyncio.create_task(mgr.start(), name="mgr.start")
# 等 fast device 完成 connect+start(read_loop 已建立),slow 仍卡 connect
await asyncio.sleep(0.15)

# sanity:fast device 應已 start()
assert all(d.start_count == 1 for d in fast), f"fast 應已 start:{[d.start_count for d in fast]}"
assert all(d.start_count == 0 for d in slow), "slow 不應 start"
回應 PR #137 Copilot review:

1. DeviceManager._on_start finally 內的 _rollback_partial_startup
   原本仍可能被 caller 二次 cancel 打斷,導致 rollback 未跑完 → zombie
   read_loop 殘留。改用 asyncio.ensure_future + asyncio.shield 包住 rollback
   task,並以 current.uncancel() 吸收 cancellation 直到 rollback done;
   finally 結束後原 CancelledError 自然向上傳播保留取消語意。

2. tests/manager/test_device_manager_startup_cancel.py 原本依賴固定
   asyncio.sleep(0.15/0.05) 等待 fast device 啟動 / rollback 完成,
   CI 排程下容易 flaky。改用既有的 tests.helpers.wait_for_condition
   輪詢 start_count / read_loop_alive / mgr.is_running。
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants