Repository navigation
fix(runtime): recover a live daemon from a damaged locator - #6892
JasonBuildAI wants to merge 4 commits into
Conversation
A damaged locator was collapsed into no runtime, so the next request cold-started a second daemon while doctor reported stopped. Distinguish a missing locator from an unreadable one, retain the diagnostic and recorded PID liveness in the lifecycle projection, and reuse a recorded endpoint only after an authenticated ping proves the same PID, fingerprint and locator path. Signed-off-by: JasonBuildAI <jasonbuildai@gmail.com>
Explain locator_state and recorded_pid_alive so operators can distinguish an absent locator from a damaged one that still authenticates a live runtime. Signed-off-by: JasonBuildAI <jasonbuildai@gmail.com>
JasonBuildAI
left a comment
There was a problem hiding this comment.
Review Verdict: Changes Requested
P2 — exact-head readback is stale after a deep probe can successfully start or reuse the runtime. In loopx/control_plane/effect_runtime.py:1484-1495, the deep path refreshes runtime_identity but uses the pre-probe locator_state and recorded_pid_alive, then forcibly sets diagnostic_code to None. I reproduced this with an isolated real runtime: collect_effect_runtime_readiness(deep=True) returned state=running, locator_state=missing, recorded_pid_alive=None, while the runtime had just published a valid v0 locator with a live PID. The same path also erases runtime_locator_schema_mismatch/runtime_locator_unreadable after a successful deep probe even though the locator remains damaged. The issue requires doctor readback to distinguish and retain these facts; the deep mode does not. Recompute the locator projection after the semantic probe and preserve the locator diagnostic when the locator is still invalid; add deep=True coverage.
P2 — a present but permission-denied locator is still reported as locator_state=missing. locator_state is initialized to missing at line 1351, and the EffectRuntimeHostPermissionError handler at lines 1379-1383 never changes it. An isolated PermissionError probe returned diagnostic_code=runtime_host_permission_denied together with locator_state=missing. On POSIX, a directory at the locator path follows the generic OSError branch in _inspect_info and is also classified as present=False. This re-collapses the missing/invalid distinction that the issue is intended to establish. Classify the locator as invalid when its read is denied or the path exists but is not a readable file, and cover that state in the readiness tests.
P2 delivery blocker — hosted exact-head CI is not green. The open PR head bf3bdad6c29ec46395c0371998abac2ade1e5424 currently has Release Artifacts / build failed. The failure is in personal-workspace-contract.test.mjs (Lark App count does not flash a false zero while loading), which is unrelated to this diff, but several required matrix jobs are still pending and merge readiness is blocked. This is not a defect in the locator change; it is a gate that must be resolved or explicitly dispositioned before merge.
Answers To The Six Review Questions
- Real value: Yes, but narrow. An operator with a corrupt/restored locator gets a silent orphan daemon and a false
stoppedhealth result under the current failure mode. This changes that to authenticated reuse plus a visible diagnostic. Normal users are unaffected; the value is defensive operational recovery, not a common-path feature. - Scope: The change stays within the existing runtime adapter, doctor projection, focused tests, and installation docs. The
_serving_runtime_identityand restart call-site updates are necessary companion paths, not unrelated refactoring. I found no opportunistic changes. - Logic bugs / boundaries: The main authenticated-reuse path is sound: it requires a live recorded PID, loopback endpoint/token, matching fingerprint, and an authenticated
runtime.pingwith PID/path identity. The two projection defects above are the substantive boundary failures. I did not find a duplicate-daemon path in the tested healthy-locator scenario. - Abstraction/design: Proportionate. The local
_RuntimeLocatorRead,_RuntimeLocatorState, and_RuntimeLocatorDiagnosticremain owned byeffect_runtime.py; the existingruntime.pingis reused and no new shared protocol/registry is introduced. The implementation is not over-abstracted for the reported behavior. - Do the tests prove the fix: They prove the headline case strongly: a real daemon is started, only
schema_versionis corrupted, the next RPC reuses the same PID, restart stops it, and malformed JSON/invalid UTF-8/dead PID are classified. They do not cover the two failing projections above (deep=Truerefresh/diagnostic retention and permission-denied state), nor the negative authenticated-ping cases such as wrong token/PID/fingerprint. Those gaps allowed the reproduced defects through. - Checks: I reran the focused suite (
21 passed), integration suite (97 passed), Ruff (All checks passed!), and the exact mypy command (49 diagnostic lines, 7 unique messages, identical on base). The semantic inventory reported only the two local aliases. The full standard canary reproduced14 selected / 4 failures, with 8/8 risk-profile smokes and public-boundary passing; I independently confirmed the four failures: Windows symlink privilege, missing Bash, the basenext_action=Noneassertion, and44 vs 43vocabulary twins (the latter two also fail on base).git diff --checkis clean. The packet's local-test claims are supported; the hosted Release Artifacts build failure above is not covered by the packet.
Lens Checks
- Typed state rules: local locator vocabulary, additive to the existing lifecycle projection; no new shared owner.
- Domain neutrality: runtime-locator wording is appropriately local and not benchmark/product-specific.
- Behavior-change disclosure: docs disclose the main reuse/diagnostic behavior, but the deep-mode stale readback is undisclosed.
- Guidance vs obligation: no new machine-enforced caller obligation.
- Default-off isolation: not applicable; this is always-on diagnostic hardening.
- Authority semantics: no actor-lifecycle, permission, or authority change.
Report the post-probe locator state and liveness instead of the pre-probe snapshot. Keep damaged-locator diagnostics on successful deep probes, classify host permission and generic read failures as invalid rather than absent, and cover both outcomes. Signed-off-by: JasonBuildAI <jasonbuildai@gmail.com>
Propagate a host-permission denial from authenticated locator recovery instead of treating it as no runtime and cold-starting. Inspect locator state and PID liveness even when the Node probe is unavailable, while keeping readiness unavailable. Add regressions for both boundaries. Signed-off-by: JasonBuildAI <jasonbuildai@gmail.com>
|
Reporter here (I filed #6871). Reading the diff, the reuse path looks sound to me: a damaged locator is only adopted after an authenticated One gap before merge: |
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent | model=gpt-6.1-sol | provider=OpenAI | declaration_source=runtime_reported | reasoning_effort=max
结论:REQUEST_CHANGES,评审 head:607a787cef6ea7c23c6b793982b96a079126f063。新增 locator 检查存在一个已独立复现的 P2:损坏 JSON 中的超大 PID 会使 doctor --installation-only 抛异常,无法返回结构化诊断。主恢复路径和既有修复仍分别认可。
动机
受影响的是遇到本机运行时定位文件损坏、并依赖 doctor 判断如何恢复的用户。
旧实现把损坏的定位文件当作不存在,下一次请求会启动第二个 daemon;当前 head 在认证成功时复用原进程,并显示定位文件诊断与 PID 存活信息。
真实 daemon 已证明 schema 损坏后复用原 PID、深度诊断保留损坏状态、重启停止原进程;但超大 PID 会使新增检查抛出异常,doctor 无法返回 JSON。
本次范围是现有本机运行时的发现、认证复用和健康回读;操作系统权限、模型服务与完整安装资格分别保留其验证边界。
改动思路
定位文件诊断、认证探测和 doctor 投影继续留在既有 Python 宿主适配器,复用 TypeScript runtime.ping;无需第二个服务或新的生命周期协议。
当前提案的复用与诊断范围合理,但须先修复损坏数值导致的异常,才能完成这一恢复切片。
定位文件是记录端点、token 和进程标识的私有 JSON,不等同于进程确实服务。提案先区分缺失与损坏,再让损坏但保留端点的候选接受真实 runtime.ping 认证,最后决定复用或交给既有冷启动 owner;doctor 仅显示派生事实。这个范围有实际价值:不修改会继续产生孤儿进程。新类型与小 helper 留在宿主适配器也合理,无须让用户手工启动 daemon 或建立新协议。
具体改动
先读 #6871、完整既有 review 及后续说明,再以修改前的 docs/guides/installing-loopx.md、修订 764b989dce88ef43ade5aba5af4958ee2cf3a821 为已接受基线。IR-1 对应既有自动启动/复用和问题要求的活 daemon 恢复,已实现;IR-2 对应 loopback/token/当前源码身份边界,已实测四条拒绝路径;IR-3 对应可用、真实的 doctor 诊断及问题要求的缺失/损坏区分,当前未满足超大 PID 反例。IR 标签只是本评审的条款索引。
关键代码讲解
_inspect_info:解析 JSON,保留损坏原因与已记录 PID 的存活事实。问题在第 530–534 行:只检查“正整数”,就先调用 OS 存活探测;schema 拒绝在其后。_recover_runtime_info:在原 request/startup owner 内使用既有认证 ping,校验返回 PID、当前 fingerprint 和 locator 路径。真实端点的错误 token、错误 PID、另一运行上下文 fingerprint 和不同 locator 路径均被拒绝采用。collect_effect_runtime_readiness:分别投影 locator 与 runtime,并在 deep probe 后重新读取。doctor 两处 Markdown 显示和安装指南双语说明随之更新;三份测试覆盖 deep、权限拒绝、Node 不可用及正常生命周期。完整六文件 diff 已读,未把旧评审仅有的修复当成整个提案。
对主干的风险
P2:在对损坏 locator 做 PID 探测前安全处理数值范围。 用隔离目录写入非当前 schema、pid=10**30 的合法 JSON,实际基线 API 正常返回,真实 doctor --installation-only 为 exit 0 且输出 JSON;当前 head 的 API 抛 OverflowError: Python int too large to convert to C long,同一 CLI 为 exit 1、回溯且无 JSON。触发链是 _inspect_info → _pid_is_alive → process_is_alive → os.kill,发生在 schema 检查前。请对外部 PID 做安全范围/转换处理,或在本地检查层容纳该数值异常,返回损坏诊断和未知/无效存活事实;增加 API 与实际 CLI 回归,保留正常 live/dead PID 及认证复用覆盖。
独立完整相关矩阵为 221 passed,无 skip。同输入的真实 Node 对照也重现原问题:基线 schema 损坏后产生第二个 PID,原进程仍服务;当前 head 复用原 PID、deep 诊断继续显示 invalid,restart 真正停止原进程。此前 deep 投影与 host-permission 两项修复已有当前证据,四种认证反例也已通过;测试的正面价值不能消除上述新数值缺陷。
风险 premerge 的 6 项 direct 通过,14 项 selected 均执行,完整门仍失败。语义 twin 门在真实 base/head 具有相同完整维护项身份与 44/43 失败,属于既存问题。首次把测试临时目录放在 Git 工作区内导致两项假设失败,原记录保留;移到 Git 外后 catalog e2e 通过,install-local-smoke 仍在 head 与实际基线均遇到原 120 秒超时。两次超时没有足够细节证明同因,保持 unresolved,未认定它是本 PR 新缺陷,也未放宽期限或冒称安装通过。Ruff、diff 与边界扫描通过;按当前策略未查询或等待 CI。
语义与 CI 对齐
_RuntimeLocatorState 与 _RuntimeLocatorDiagnostic 是现有适配器内的两个 Literal 闭集;既有 lifecycle v0 只增加派生显示字段,没有新建共享执行权威。typed 检查仍须覆盖外部数值进入 OS API 的边界:IR-3 的当前义务是损坏状态可诊断,新增路径却丢失结构化结果。最小修复位于该 PID 检查;重跑 locator/readiness 相关测试,并用隔离 locator 对实际 doctor --installation-only 做同一反例。缺少新 vocabulary 注册不是这里的阻塞原因。
我的整体评价
实际认证恢复有用、机制与问题规模相称;保留私有 v0 locator 和既有 reader facade 有当前调用依据。可延后把重复读取合并为一次 inspection,前提是保持诊断与认证语义,这不是额外合并门。长程正常恢复路径为 preserved,但用户体验在损坏数值上是 regression,因此当前切片的交付仍 not_yet_proven。定位文件诊断、认证探测和 doctor 投影继续留在既有 Python 宿主适配器,复用 TypeScript runtime.ping;无需第二个服务或新的生命周期协议。当前提案的复用与诊断范围合理,但须先修复损坏数值导致的异常,才能完成这一恢复切片。 请修复并给出新的精确 head;平台安装、真实模型和长期收益不由本地通过数关闭。
English verdict: REQUEST_CHANGES — 607a787cef6ea7c23c6b793982b96a079126f063. A damaged locator with an oversized positive PID triggers the new pre-schema liveness probe and crashes readiness plus the real installation-only doctor; actual base returns JSON. Bound/contain PID decoding and add API/CLI coverage. 221 tests, real same-PID reuse/deep/restart and four identity-rejection probes pass. Existing twin debt and the unresolved installer timeout remain separate.
| ) | ||
| pid = payload.get("pid") | ||
| recorded_pid_alive = ( | ||
| _pid_is_alive(pid) |
There was a problem hiding this comment.
[P2] Safely validate/bound externally decoded PID or catch numeric conversion overflow locally; preserve invalid/unknown liveness and safe diagnostic; add API and CLI regression.
Observed: The new liveness probe executes before schema validation. OverflowError escapes readiness; actual doctor --installation-only exits1 with a traceback and no JSON. Same input on actual base exits0 with structured output.
Validation: Use a disposable runtime directory, damaged schema and pid=10**30. Assert API and actual installation-only doctor return public-safe diagnostic/unknown liveness without throwing; keep normal live/dead PID and authenticated reuse tests.
Goal And Delivered Outcome
loopx doctorreports the runtime as stopped._read_infocollapsed JSON/schema/fingerprint/field failures intoNone; callers treated that as "no runtime", and doctor had no way to show the live recorded PID.schema_versioncorrupted, before: doctorstopped, diagnosticnull, a new daemon started while the old PID stayed alive and serving. After: doctorrunning,locator_state=invalid,recorded_pid_alive=true, diagnosticruntime_locator_schema_mismatch, and the next RPC reuses the original PID. The real-daemon test intests/control_plane/test_effect_runtime_locator_recovery.pyproves this end to end.loopx-project/loopx:mainat764b989dc.Author Declaration
AI disclosure: this change was drafted with AI assistance at my request, under my direction and review.
The problem and constraints were set by the human requester; this PR is submitted for human review.
Implemented against
mainat764b989dc)._inspect_info,_RuntimeLocatorReadinloopx/control_plane/effect_runtime.pytests/control_plane/test_effect_runtime_locator_recovery.py_RuntimeLocatorDiagnostic; lifecyclediagnostic_codecollect_effect_runtime_readinesscollect_effect_runtime_readiness,render_doctor_markdown_recover_runtime_info,_runtime_info_for_requestruntime.pingmust match PID, fingerprint and locator pathUnicodeError/JSON/type guards in_inspect_infomain, and confirmed the old behavior before changing code. Deliberately left out unrelated refactors and new dependencies. A locator whose token/endpoint are destroyed cannot be authenticated safely; that path cold-starts and retains theruntime_locator_unreadablediagnostic.Scope And Continuation
4b340d9e6): a deep probe now refreshes the locator projection after it may have started the runtime, preserves a damaged-locator diagnostic on success, and reports host-permission or generic locator read failures asinvalidrather thanmissing. Focused tests cover both outcomes.607a787ce): recovery now propagates a host-permission denial instead of treating it as no runtime and cold-starting, and readiness inspects locator state/PID liveness even when the Node probe is unavailable. Regressions cover both boundaries.Validation
607a787cef6ea7c23c6b793982b96a079126f063(all listed checks ran on the final tree).staticpasseduv run --extra test ruff check ...→All checks passed!;git diff --check upstream/main...HEADclean.unitpasseduv run --extra test pytest -q tests/control_plane/test_effect_runtime_locator_recovery.py tests/control_plane/test_effect_runtime_host_permission.py tests/control_plane/test_turn_journal_runtime_readiness.py→45 passed in 54.72s.integrationpassed214 passed, 7 skipped, 3 warnings in 162.31s; skips are the platform-gated cases already present in the suite.real_entrypointpassedeffect_runtime.effect_runtime_result; corrupting onlyschema_versionreportsrunning/invalid/PID alive and reuses the same PID;loopx doctormarkdown readback includes the diagnostic.real_backendpassedregression_paritypassed2before the fix and pass after; malformed JSON and invalid UTF-8 retainruntime_locator_unreadableinstead of looking absent. Follow-up tests prove deep success refreshes absent → valid and preserves damaged → invalid, while host-permission and generic read errors remain visible.regression_paritypassed2before the follow-up (must not cold start;locator_statewasmissing) and pass after; the tests prove host-permission recovery does not launch a replacement and Node-unavailable readiness still reports a valid live locator.staticpassed_RuntimeLocatorState/_RuntimeLocatorDiagnosticaliases for the branch; decision: remain local toeffect_runtime.py, extending the existingloopx_effect_runtime_lifecycle_v0readback; no registered vocabulary or shared contract change.staticfaileduv run --extra test python -m mypy --follow-imports=silent loopx/control_plane/effect_runtime.py loopx/doctor.pyexits 1 with the same 48 errors and 4 unique diagnostic shapes as base764b989dc; this branch adds no new normalized mypy diagnostic.integrationfailedstandard: all direct checks, module ceiling, 8/8 risk-profile smokes and public boundary passed. Four catalog canaries failed and each was reproduced independently on base764b989dc:install-local-smoke(WinError 1314, symlink privilege),codex-cli-packaged-install-smoke(Bash unavailable),loopx-update-smoke(next_action=None), andsemantic-vocabulary-drift-smoke(44 twins vs budget 43).Frontend / Visual Evidence
locator_state,recorded_pid_alive) to the existing runtime lifecycle block; no new screen, hero or CTA.Type of Change
LoopX Area
Technical Direction
locator_stateand its diagnostics stay local to the existing Python Effect runtime adapter; the TypeScript protocol and locator schema are unchanged.Shared-authority RFC fixture impact
Boundary Checklist
.loopx/,.codex/goals/, and liveACTIVE_GOAL_STATE.md).python3 scripts/check_rfc_bilingual.py --base origin/main --head HEADpasses, or no paired RFC changed.none.Signed-off-bytrailer (git commit -s).