architect liveness: characterize dead-architect visibility and send behavior (prior claims partly retracted); architect table pid=0; no architect recover command

[amrmelsayed]

Field incident (full report available; summarized with evidence)

On a second workspace, four of five architects died individually over ~two days. Tower — whose daemon ran CONTINUOUSLY the whole window (uptime 150,352s → 316,489s, so this was not a Tower crash) — kept serving all of them as registered. afx status listed an architect with a pid that ps -p showed dead; workspace terminal count dropped 14 → 10 with no deregistration. Worst part: afx send architect:<name> to a dead architect reported [ok] Message delivered. A gate approval relayed to a dead architect is lost with a positive confirmation and no error anywhere — the held-mailbox path exists precisely for this and never engaged because the target was believed live.

Why the gap exists (architecture)

Liveness reconciliation runs at Tower BOOT only. A long-lived Tower never re-checks; rows can only go stale. Compounding detail from the reporter's DB inspection: the architect table stores pid = 0 for every row (unusable for liveness; started_at also stale across restarts), while terminal_sessions carries real pid, shellper_pid, shellper_start_time — correct and sufficient, but nothing validates or prunes those rows during Tower's life. The data needed for liveness is already persisted; it just is not checked after boot.

Fix shape (hypothesis, for the builder to validate)

1. Validate before presenting. afx status / overview: kill(pid, 0) each terminal_sessions row with shellper_start_time compared against the live process's start time (PID-reuse guard — same discipline as tower reconcile: failed reconnect triggers delete + SIGTERM without verifying the shellper is dead (destroyed 53 live sessions in the #1629 incident) #1686's proof-of-death), and present dead rows as STALE, never as live. Presentation-only: this must not delete or signal anything (tower reconcile: failed reconnect triggers delete + SIGTERM without verifying the shellper is dead (destroyed 53 live sessions in the #1629 incident) #1686's invariant stands — destruction requires proof of death AND stays reconcile's job).
2. afx send must not report success to a dead target. Route to the durable mailbox with a dead-target (or similar) hold reason, consistent with existing hold semantics — the sender gets held+reason and the owner escalation path engages, instead of [ok] delivered into a void. Also investigate HOW delivery verification passed against a dead session: if the echo check can pass on a stale ring buffer, that is its own sub-defect of the tower: delivery write edge — verify-or-retry 'delivered', settle-before-write, loud size limit (silent-loss residuals of #1564/#1521) #1573 contract.
3. Periodic or on-read revalidation. Boot-only reconcile is structurally insufficient for a daemon designed to run for weeks. Options for the builder to weigh: lazy revalidation on read (status/overview/send resolve), a slow background sweep with the tower startup can hang unbounded and unlogged in killOrphanedShellpers, tripping the 30s launcher timeout (field report on 3.3.3) #1685 bounding discipline, or both. Any sweep must stay presentation/hold-only per (1).
4. afx workspace recover equivalent for architects. Today it revives only builders. The field recovery that worked, worth building toward and documenting: afx workspace add-architect --name <same-name> then re-init — builder→architect affinity survives (builders keep their Owner), so same-name re-add restores routing with no builder disruption.

Docs note

The afx skill had no entry for recreating/restarting an architect (reporter had to discover workspace add-architect / remove-architect from --help). Being fixed directly in the skill docs alongside this filing; noted here for completeness.

Related: #1714 (registry-vs-substrate consistency check) is the audit-side complement; this issue is the runtime-liveness side.

[amrmelsayed]

Correction from the reporting workspace, plus a codev-side model correction. This reshapes the issue - read before working it.

Retracted by the reporter (inference had been written up as observation):

• The send-to-dead '[ok] delivered' was never observed. Every send in their session went to a live, just-re-added architect. Fix Context Window Optimization - AI Needs Project Knowledge Without Breaking Limits #2 (dead-target hold reason) and the echo-verification sub-question are therefore UNEVIDENCED and must not drive a fix as filed.
• 'Nothing prunes terminal_sessions' is wrong - the dead rows are gone from their DB now.
• Weakened: they cannot show a dead architect listed at any single moment; the 09-19 listing may have shown a then-alive process, and by 09-21 the rows were both dead AND delisted.

Codev-side correction (mine): the body's 'reconcile runs at Tower boot only' was incomplete. The on-read reconnect path in getTerminalsForWorkspace() deletes a row whose reconnect fails AND whose pid is provably dead - the pid-dead delete that PR #1693 deliberately preserved while forbidding live-pid destruction. That is almost certainly the pruning the reporter observed, and it means dead rows are cleaned lazily on the next read that touches them, not only at boot.

What still stands, re-verified by the reporter:

• Four of five architects died individually over ~2 days of CONTINUOUS Tower uptime (150,352s to 316,489s) - the deaths are real; their cause is uncharacterized.
• The architect table stores pid = 0 for every row including seconds-old ones, and carries stale started_at. That table cannot support liveness regardless of anything above.
• The same-name re-add recovery works (affinity survives); docs fixed in e649dc3.

Revised scope for this issue:

1. Characterize first, fix second: on a disposable workspace, kill an architect process and observe end to end (a) how status/overview present it over time before the lazy prune fires, (b) what afx send to it actually returns at each stage. The fixes, if any, follow that evidence - including whether a dead-target hold reason is needed at all.
2. architect table pid=0: populate it honestly or stop pretending it is data (derive from terminal_sessions - single source of truth); its stale started_at suggests the row is write-once. Small, real, evidenced.
3. workspace recover equivalent for architects - stands as filed.
4. Why did four architects die individually over two days? Out of scope here (workspace-side environment), but the characterization run may surface it.

Title updated to match.

[amrmelsayed]

Specimen from the reporting workspace: the four deaths were ONE simultaneous event at a sleep/wake boundary — this re-corrects the premise (including their own earlier 'died individually over two days', which was a sampling artifact).

Their observations, verbatim in substance:

• All four dead architects' terminal logs stop at exactly 2026-09-20 05:07:22 — the same second pmset -g log records DarkWake from Deep Idle [CDNP] ... Using BATT, going FullWake at 05:07:25. The surviving architect logs on.
• No crash reports for node/claude in the window (not a segfault/abort).
• Last painted frame before death: the harness's staged-update banner, then charset-reset and teardown sequences — i.e. an ORDERLY TUI exit, not a hard kill of the child.
• Process-shape discriminator: the survivor runs PPID=1, no controlling TTY (orphaned to launchd ~33 days ago), while a freshly created architect has a live parent. Causation explicitly NOT established by the reporter; the victims' PPIDs are unrecoverable (processes gone, rows lazily pruned).

Codev-side annotations:

1. A candidate mechanism consistent with our architecture: the shellper (the node-pty parent) dies at the DarkWake instant → the PTY master closes → the child TUI sees EOF/SIGHUP and exits cleanly — which would produce exactly the no-crash-report + orderly-teardown fingerprint observed. Under this reading the bug is "shellpers do not survive sleep/wake on battery", which is squarely a codev defect: shellpers are DESIGNED as detached, persistent processes (surviving Tower restarts is a documented property), so sleep/wake survival is within their contract.
2. Open contradiction the repro must resolve: the survivor's PPID=1 means its own shellper died long ago and the child KEPT RUNNING — so shellper death alone does not reliably kill the child. Either the wake event kills both shellper and child through separate paths, or child survival after shellper death depends on state (controlling TTY present or not, EOF handling at that moment). Do not assume the simple chain.
3. Repro shape (endorsing the reporter's, plus one addition): disposable workspace, several architects; BEFORE sleep record per-architect ps -o pid,ppid,tty,lstart for BOTH the shellper and the child, plus shellper socket paths; sleep on battery into Deep Idle; wake; then check shellper liveness and child liveness SEPARATELY. That distinguishes 'shellpers die at wake' from 'children die at wake' and tests the PPID discriminator with the measurement the field specimen lacked.
4. Priority implication if it reproduces: the architect-recover command (fix Scaling Crisis - AI Autonomy Decreases as Projects Grow #3 in the revised scope) graduates from convenience to primary mitigation, and shellper sleep/wake hardening becomes the root fix. Laptops sleep on battery every day; this would not be an exotic event.

The characterization experiment in the revised scope should absorb this repro — it is now the highest-value run on the issue.

[amrmelsayed]

Cross-workspace evidence from the same machine and the same event (one Tower serves every workspace here), plus source verification of the reporter's two facts.

Reporter's new hard facts, accepted: the 05:07:22 event killed 4 of 5 architects and ZERO of 8 builders in their workspace; Tower (the shared parent, up 3d16h spanning the event) survived; orphan-vs-parented does not predict outcome (PPID=1 processes both died and survived). Parent-death cascade and blunt-sweep models are dead.

New facts from this side, same machine, same instant:

1. Tower was passive. tower.log at the wake instant (UTC 2026-09-19T19:07:22) shows exactly: two SSE clients removed (dead), the cloud tunnel websocket closing (code 1006) and reconnecting 5 s later - and NOTHING else. No terminal closes, no kills, no errors. The terminal partial monitor reports the same session count (42) immediately after the instant and for the following minutes. Whatever killed those four processes, it was not a Tower-initiated closure.

2. Another workspace's architects all survived the same event: five architects (created 08-18 and 09-02, i.e. long-lived), every one of them PPID=1 / orphaned to launchd - the same shape as the reporter's survivor. Across both workspaces, every long-orphaned architect survived and only the reporter's recent, current-Tower-era architects died. The characterization run should treat AGE/orphanhood-at-event-time as a candidate axis alongside host.

3. Schema conflation CONFIRMED in source - tower-instances.ts:659-660 passes shellperInfo.pid as BOTH the pid and shellper_pid arguments to saveTerminalSession, so the CHILD pid is never persisted anywhere. The reporter is right that 'record shellper and child separately' cannot be satisfied from the DB; the run needs ps snapshots walking children of the Tower pid at capture time.

4. The hosting-path axis is real in source: architect spawn tries shellper first and falls back to a NON-PERSISTENT bare PTY (a direct Tower child) with only a tower.log WARN - tower-instances.ts:712/765 (main architect) and 1123/1279 (add-architect). The reporter's freshly recreated architect showing PPID=Tower matches the fallback shape. No 'non-persistent' WARN appears in the current tower.log window, which is its own puzzle - the run must record, per architect, WHICH path actually fired (shellper socket present + start_time populated vs bare PTY), not assume from role.

5. The UI-host hypothesis is consistent with (1): the SSE drops at the instant ARE clients (editor windows / dashboards) going away at wake. No close-on-client-disconnect logic was found in the terminal server on first inspection, but absence-of-finding is not absence - if the run shows dead architects were exactly the UI-pane-hosted ones, that code path gets a targeted hunt.

Instrument list for the characterization run, consolidated: per architect and builder BEFORE sleep - ps -o pid,ppid,tty,lstart for shellper AND child (walk Tower's children), shellper socket path presence, spawn path (WARN present or not), terminal HOST (UI pane / dashboard / headless), age and orphan status. Sleep on battery to Deep Idle; wake; re-snapshot; correlate. One run with those columns should separate the surviving hypotheses (host-death, hosting-path, age/orphanhood) in a single pass.

[amrmelsayed]

Weighting correction from the reporter (their fourth, self-caught): the claim 'PPID=1 processes both died and survived, so orphanhood does not predict outcome' is WITHDRAWN in its died-half - the six surviving builders' PPID=1 was observed, but the dead architects' PPID=1 was inferred from their cohort timestamp (one 09-17 09:02:45 batch), never measured, and is now unrecoverable (processes gone, rows pruned; no shellper artifacts under ~/.codev/run were touched in the 05:06-05:09 window - which also independently corroborates the passive-Tower reading above).

Revised hypothesis ranking for the characterization run, per the reporter and concurred with here:

1. Age/orphanhood-at-event-time (up-weighted: cross-workspace, every long-orphaned architect survived and only current-Tower-era architects died - the one axis with positive signal on both sides),
2. Hosting-path fallback (non-persistent bare-PTY architects; confirmed to exist in source, unconfirmed in the field specimens),
3. UI-host death (rests only on role-selectivity, which hypothesis 2 explains equally well).

The instrument list in the previous comment already separates all three in one run; nothing about the experiment changes - only what we expect it to find.