Conversation
Dashboard WS 与 IM 通道的回合处理链路（user_turn → iter_turn_chunks / __call__ → agent.stream → LLM provider）此前没有任何超时：LLM 上游请求 挂起或工具静默挂起时 turn 永久卡住，前端无回帧、session lock 永久持有、 同会话后续消息全部排队。 机制： - ws_hub：active turns 从 set 升级为登记表（agent_id/started_at/ last_progress_at/notified/in_tool/工具段预算），notified 记录防复活 - TurnWatchdog：后台扫描，模型阶段无进展超时（默认 300s）/工具阶段 长阈值（默认 900s）/模型阶段总时长上限（默认 1800s，工具段不计）； 触发后审计留痕 + 推 error/done 帧（IM 推超时提示）+ cancel_stream （harness cancel 可打断阻塞的 __anext__）；notified 宽限后强删僵尸 - ws_channel：chunk 产出打进展点 + tool_call_chunk/tool_result 翻阶段 - processor.__call__：IM 通道同一登记表登记/打点/注销 - ws.py：pre-enqueue 准备阶段 asyncio.wait_for 超时兜底（MCP 加载等 网络 IO 挂住时客户端立即拿到报错） 参数（env）：OCTOP_TURN_STALL_SECONDS / OCTOP_TURN_TOOL_STALL_SECONDS / OCTOP_TURN_MAX_SECONDS / OCTOP_TURN_NOTIFIED_GRACE_SECONDS / OCTOP_TURN_PREPARE_TIMEOUT_SECONDS / OCTOP_TURN_WATCHDOG_INTERVAL_SECONDS / OCTOP_TURN_WATCHDOG_DISABLED
There was a problem hiding this comment.
🟡 Changes recommended
OCTOP_TURN_WATCHDOG_DISABLED=1 逃生阀在当前代码路径未生效（watchdog 仍会被无条件启动），与 PR 描述不一致且会影响运维可控性。
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
该 PR 为 Octop 的 Dashboard WebSocket 与 IM 通道引入“TURN watchdog”机制，用于检测并恢复流式回合在上游 LLM/工具执行静默挂起时造成的永久卡死，从而避免 session lock 长期占用导致同会话后续消息持续排队。
Changes:
- 在
WebSocketHub引入ActiveTurn登记表，并在 WS/IM 两条链路打进度点与工具阶段翻转，供 watchdog 判断“模型无输出卡死”与“工具长时间无 chunk”两类场景。 - 新增
TurnWatchdog后台扫描任务：按“模型阶段 stall / 工具阶段 stall / 模型阶段总时长上限”触发恢复，推送 error/done 并执行cancel_stream，同时提供僵尸记录强删机制与审计写入。 - Dashboard WS 的 pre-enqueue 准备阶段增加
asyncio.wait_for超时兜底；补充单元测试与 Changelog 记录。
File summaries
| File | Description |
|---|---|
| tests/unit/gateway/test_turn_watchdog.py | 新增 watchdog 行为单测：stall 恢复、工具阈值、模型时间上限、notified 宽限强删、IM/WS 通知差异等 |
| src/octop/infra/gateway/ws/ws_hub.py | 将 active turn 从 set 升级为 ActiveTurn 记录表，支持进度与工具阶段跟踪、notified 僵尸防复活 |
| src/octop/infra/gateway/ws/ws_channel.py | Dashboard WS 流式 chunk 路径添加进度/工具阶段标记，接入统一 turn 登记表 |
| src/octop/infra/gateway/ws/turn_watchdog.py | 新增 TurnWatchdog 实现：周期扫描、通知/取消、审计与僵尸清理、环境变量参数化 |
| src/octop/infra/gateway/process/processor.py | IM MessageEvent 流路径接入 turn 登记表，并按 TOOL_START/END 翻转工具阶段、按事件打进度 |
| src/octop/infra/gateway/gateway.py | Gateway 启动/关闭时创建并管理 TurnWatchdog 生命周期 |
| src/octop/api/routers/chat/ws.py | Dashboard WS 的 prepare_dashboard_turn 增加超时兜底与错误帧返回 |
| CHANGELOG.md | 记录新增 TURN watchdog 与 prepare 超时兜底 |
Review details
- Files reviewed: 8/8 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+230
to
+236
| self._turn_watchdog = TurnWatchdog( | ||
| hub=self._ws_hub, | ||
| agent_manager=self._agent_manager, | ||
| audit_repo=self._repos.audit_repo, | ||
| gateway=self, | ||
| ) | ||
| self._turn_watchdog.start() |
Comment on lines
+143
to
+145
| task.cancel() | ||
| with __import__("contextlib").suppress(asyncio.CancelledError): | ||
| await task |
Copilot review feedback (TencentCloud#581): - Gateway.boot() now checks watchdog_disabled() before creating/starting the TurnWatchdog, so OCTOP_TURN_WATCHDOG_DISABLED=1 actually works. - Replace __import__("contextlib") with a top-level import for readability. - Tests: env parsing for watchdog_disabled() + boot honors the hatch.
Author
|
已按 Copilot review 修复，提交 7026268：
测试：test_turn_watchdog.py + test_gateway.py 12 passed。 @copilot-pull-request-reviewer 请 re-review。 |
Author
|
Request Copilot review |
Two connections to the same thread (multi-tab) each open a turn; the first to finish previously dropped the registration while the second was still streaming, so the second lost watchdog coverage and subscribers saw a premature turn_active=false. ActiveTurn now refcounts registrations and mark_turn_idle only clears the record at zero.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题
Dashboard WS 与 IM 通道的回合处理链路（user_turn → iter_turn_chunks /
__call__→ agent.stream → LLM provider）没有任何超时：LLM 上游请求挂起（连接黑洞 / 200 无字节 / 慢流）、或工具执行静默挂起时，turn 永久卡住——前端无回帧、session lock 被永久持有、同会话后续消息全部排队。机制
ws_hub：
_active_turns从set[str]升级为dict[str, ActiveTurn]登记表（agent_id / started_at / last_progress_at / notified / in_tool / 工具段预算）。notified 记录保留在表内（is_turn_active返回 False、snapshot 排除、mark_turn_active拒绝复活）由 harness 收尾 finally 删除，防用户重试撞上僵尸登记。TurnWatchdog（新文件）：后台 task 周期扫描登记表——
OCTOP_TURN_STALL_SECONDS，默认 300s）OCTOP_TURN_TOOL_STALL_SECONDS，默认 900s）——工具执行不产 chunk 是正常态OCTOP_TURN_MAX_SECONDS，默认 1800s，工具段不计入）cancel_stream（harness cancel 可靠，_iter_until_cancelled能打断被阻塞的__anext__）OCTOP_TURN_NOTIFIED_GRACE_SECONDS，默认 120s）后强删僵尸记录ws_channel / processor.call：Dashboard WS 与 IM 通道打进展点（每 chunk/event）+ 工具阶段翻转（tool_call_chunk/TOOL_START ↔ tool_result/TOOL_END），同一登记表两条通道共用。
ws.py：pre-enqueue 准备阶段
asyncio.wait_for超时（OCTOP_TURN_PREPARE_TIMEOUT_SECONDS，默认 60s）——MCP 加载等网络 IO 挂住时客户端立即拿到报错而不是无回帧（watchdog 看不到未入队的 turn）。参数
全部 env 可配：
OCTOP_TURN_STALL_SECONDS/OCTOP_TURN_TOOL_STALL_SECONDS/OCTOP_TURN_MAX_SECONDS/OCTOP_TURN_NOTIFIED_GRACE_SECONDS/OCTOP_TURN_PREPARE_TIMEOUT_SECONDS/OCTOP_TURN_WATCHDOG_INTERVAL_SECONDS/OCTOP_TURN_WATCHDOG_DISABLED=1（逃生阀）测试
tests/unit/gateway/test_turn_watchdog.py7 用例：stall 恢复（error/done+cancel）/健康 turn 不误杀/工具阶段长阈值/模型阶段上限排除工具时间/notified 防复活+超宽限强删/IM turn 恢复推提示/WS turn 不推 IM 提示。