fix(gateway): use respawn_executable when restarting after update - #8832
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Root cause fixed at both halves — call sites and the symlink-blind identity predicate — with exec-target validation preserved and the sibling predicate deliberately deferred to #8938. Suggestions
[DESIGN-REVIEWED] 5cf5721 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of All claims verified. The dashboard caller's own comment names the recorded convention ( First-Principles-Verdict: PASS A FIX that lands at cause level: both halves of the ENOENT (unwired sites and a wrong predicate) are fixed, siblings counted, one declared and tracked. What this change shipsIntent: after an auto-update prunes a managed install, the gateway restarts itself instead of staying down — a FIX.
Watch
[FIRST-PRINCIPLES-REVIEWED] 5cf5721 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
UX Review (Fable 5) — ⏭️ skippedRevision |
b760655 to
d37e542
Compare
d37e542 to
ef76748
Compare
|
Disposition: fixed in ef76748 — span=e51cd2330ac2
Legitimate: yes.
|
|
Disposition: rebutted — the
The observation is accurate (4/4 callers now pass the same value), but folding the resolution into
|
ef76748 to
827a058
Compare
|
Disposition: fixed in 827a058 — span=e51cd2330ac2 (2nd finding in this span: round 1 was the resolver's identity rule at the wheel-update restart site; this round is the import ORDER at the provider restart site)
Legitimate: yes.
|
827a058 to
3e88e15
Compare
|
Disposition: fixed in 3e88e15 — span=dd36ca1a6f5b (3rd finding in this PR's GPT sequence, 1st in this span: rounds 1-2 were the resolver's identity rule and the import order, both at
Legitimate: yes, and I reproduced the window on this PR's rebased base rather than taking the trace on faith. Proportional: yes — the fix is the one the finding names, validate the new
|
|
Disposition: fixed in 3e88e15 — description and commit message now declare the
Legitimate: yes, and the premise checks out on measurement, not just on reading
|
|
Disposition: accepted-and-deferred to #8938 (label
Legitimate: yes, and I verified the premise rather than inheriting it. Measured on a real Deferred rather than fixed here, and the reason is blast radius, not effort:
|
|
Disposition: fixed in 3e88e15 — the function is now
Legitimate: yes, and I re-counted before removing it rather than trusting the review. Across
|
After an auto-update replaces the managed install, three restart paths in
GatewayOrchestrator (_restart_after_update, _auto_apply_update,
_auto_apply_wheel_update) called reexec_python_module("kiro_crew", ...)
without `executable=`, so they re-exec'd sys.executable -- an interpreter
inside the old tree the update had just pruned. The exec failed with ENOENT
and the process only recovered through the stale-asset watchdog / service
restart.
Each site now resolves the interpreter the way the dashboard-triggered
restart in handlers/updates.py already does: `exe = await
asyncio.to_thread(respawn_executable)`, passed as `executable=exe`. The
resolver is imported BEFORE the apply step and called after it -- the
module is deliberately off the gateway boot path, so a deferred import
would read a package the migration has already deleted, while the answer
is only correct once the stable link has been repointed.
wheel_engine changes too, and that half is load-bearing rather than
wiring:
- Respawn identity is read from the interpreter's `bin/` DIRECTORY, not
the interpreter file. `python -m venv` writes `bin/python3` as a symlink
to the base interpreter, so resolving the file lands outside every venv
and answered "not managed" for every real cli.sh install -- without this
the three call sites above still get sys.executable and still ENOENT.
- The retired in-data-home venv (`<data home>/venv`, cli.sh's _OLD_VENV)
counts as a respawn identity, so a process it still serves restarts
through the stable link after the migration deletes it.
- When the stable link cannot carry the restart AND sys.executable no
longer exists, the validated legacy tree answers instead of a path that
names nothing. That state is reachable: cli.sh skips the stable-link
repoint when a real directory sits at the stable name and treats a
repoint failure as non-fatal, while the _OLD_VENV delete is gated on the
NEW tree's own `import kiro_crew` check -- so the nested venv goes away
with the link still unusable. A process whose own interpreter still
exists keeps it, so a corrupt link cannot displace a healthy gateway.
Both identity changes also apply to the existing dashboard consumer of
respawn_executable. running_from_managed_venv keeps its own rule; it gates
shadow-build dispatch rather than a restart, and changing it is tracked
separately.
Tests: test_gateway_restart_uses_respawn_executable.py drives each of the
three real methods to its exec line and asserts reexec_python_module
received executable=<sentinel>, plus an AST check pinning the import ahead
of the apply per method. A discovered (not enumerated) check also walks every
reexec_python_module call in gateway.py and requires an explicit
executable=, so a restart path added later cannot reintroduce the defect
without tripping a test. test_wheel_engine.py covers the identity rules and
the fallback, including the two migration shapes that reach the legacy tree
and the two that must NOT (live interpreter present; no usable legacy
tree). Reverting each production hunk fails the matching tests.
Co-authored-by: Kiro Crew <noreply@kiro.dev>
3e88e15 to
5cf5721
Compare
|
Disposition: fixed in 5cf5721 — the fourth-site gap is now closed by a mechanical check rather than by the review rule, though not with the helper as proposed.
Legitimate: yes on the diagnosis, and the concrete worry was correct in a way worth stating plainly — I checked, and the existing On the proposed remedy specifically, a helper is disproportional here, and it would not actually have closed the gap:
|
|
Disposition: accepted-and-deferred to #8953 (label
Legitimate: yes, and it is reachable on a narrower path than the reviewer stated and a broader one. Deferred rather than fixed here, and the honest reason is that the fix is a design call on a subsystem this PR does not touch:
|
|
Disposition: rebutted — the two shapes track two different roles, so converging them would add a parameter to functions that have no injector.
The observation is accurate and the motive is right — one pattern is easier to pin than three. But the asymmetry is not stylistic drift; it follows from which function owns the apply:
|
iamwhatever
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: fix -- a gateway restart after a wheel auto-update exec'd sys.executable, which the same installer re-run had just deleted along with the retired in-data-home venv, so a closed-session gateway was left un-restarted; respawn_executable now recognises that tree as restartable and falls back to a validated in-layout interpreter, and the resolver is imported before provider.apply() so the deferred import cannot fail after the venv is gone. Exec targets stay gated by is_managed_tree, so no new path is trusted with an exec.
chenmingwei23
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: fix(gateway) - resolves stale cached sys.executable at the restart-after-update sites via respawn_executable(), clear single root cause, 4 files (cap 5).
Problem / Motivation
After an auto-update replaces a managed install, the gateway tries to restart itself and dies with
ENOENTinstead of coming back. The three restart-after-update paths inGatewayOrchestrator(_restart_after_update,_auto_apply_update,_auto_apply_wheel_update) execsys.executable, and after the update pruned the old install that path no longer exists.Why it matters
Every operator on a managed (wheel/installer) install who has auto-update on: the update applies, the gateway exits to restart, and nothing comes back up until someone starts it by hand. An unattended host stays down.
What changed (motivation → approach → change)
Symptom: the gateway process is gone after a successful auto-update, with an
os.execvENOENT in the log.Root cause, in two halves. The gateway sites called
platform_compat.reexec_python_module("kiro_crew", sys.argv[1:])with noexecutable=, so they fell back to the cachedsys.executablein the tree the update just removed. But the resolver they should have called could not answer correctly either:respawn_executable()decided "is this a managed venv?" by resolving the interpreter FILE, andpython -m venvwritesbin/python3as a symlink to the base interpreter — so the resolved path landed outside every venv and the predicate said "not managed" for every realcli.shinstall. Wiring the sites up alone would still have ENOENT'd.The change fixes both halves. Each of the three sites resolves the interpreter through
wheel_engine.respawn_executable()(viaasyncio.to_thread, since it stats the filesystem) and passes it asexecutable=, matching what the dashboard update handler already does. The resolver is imported before the apply step and called after it — the module is deliberately off the gateway boot path, so a deferred import would read a package the migration has already deleted, while the answer is only correct once the stable link has been repointed. Inwheel_engine.py, respawn identity is now read from the interpreter'sbin/directory (symlink-correct), the retired in-data-home venv counts as a respawn identity so a process it serves can restart through the stable link after the migration deletes it, and the fallback reaches the validated legacy tree when the stable link is unusable andsys.executableis already gone.This changes behaviour for the existing dashboard caller too (
handlers/updates.py), not just the three new ones: both identity rules widen whatrespawn_executable()answers for symlinked-interpreter and legacy-nested-venv installs.running_from_managed_venv()keeps its own file-resolving rule — it gates shadow-build dispatch rather than a restart, so correcting it moves a different mechanism's behaviour and is tracked in #8938 instead of being folded in here.Tests
test/test_gateway_restart_uses_respawn_executable.py(new) — one test per restart site. Each drives the real method to its exec line with the download/apply/breadcrumb steps stubbed, patchesrespawn_executableto return a sentinel path andreexec_python_moduleto record the call, and asserts the exec receivedexecutable=<sentinel>.TestResolverIsLoadedBeforeTheApplyparsesgateway.pyand pins, per method, exactly onerespawn_executableimport whose line precedes the firstapply/create_subprocess_execcall.test/test_wheel_engine.py— the identity rules and the fallback. Two migration shapes that must reach the legacy tree (a real directory at the stable name, socli.shskips the repoint; and no stable link at all, so the repoint failed non-fatally) and two that must NOT (sys.executablestill present, so a corrupt link cannot displace a healthy gateway; no usable legacy tree, so the answer stayssys.executable).Each production hunk was reverted in place to confirm the matching tests go red. Local run on the rebased tree: 753 passed, 23 skipped across
test_wheel_engine.py,test_gateway_restart_uses_respawn_executable.py,test_update_check_install_aware.py,test_platform_compat.py,test_slack_gateway.py. isort / flake8 7.1.0 / black / mypy (1302 files) all clean.Manual verification
N/A — unit coverage sufficient: the change is which interpreter path is handed to
execv, and the tests assert exactly that argument on every affected site. The one premise that is not unit-testable — thatcli.shcreates venvs with a symlinked interpreter, skips the stable-link repoint when a real directory occupies the stable name, and deletes the nested venv on the new tree's own import check rather than on the repoint — was read offcli.shon this PR's rebased base, and the symlink behaviour was measured on a realpython3 -m venv.Why no screenshot: backend-only change to the gateway restart path; nothing under
website/and no rendered surface.Related Issues
no linked issue: found while tracing a post-update restart failure on a managed install. Follow-up filed as #8938.
Pattern harvest
Rule candidate: review-prompt
Two concretely-checkable shapes, both grep-able:
reexec_python_module(...)/ anos.exec*call with noexecutable=on a code path that runs after an install or update step — the cachedsys.executablemay name a pruned tree.Path(sys.executable).resolve()(oris_managed_tree(Path(sys.executable))) used as a "which venv am I in?" predicate.python -m venvsymlinksbin/python3to the base interpreter, so resolving the FILE escapes the venv; the containment question must be asked ofPath(sys.executable).parent. Grepresolve()applied tosys.executable— the sibling this PR left is running_from_managed_venv() answers False on every real cli.sh install (interpreter symlink resolved past the venv) #8938, found exactly this way.Not generalizable: nothing here is a tuning constant.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)Backport: please include in release/0.6.0.