[#992] Auto-respawn agents on server restart when a batch is active

[realproject7]

Closes #992

Summary

shutdown() (#972) now kills agent PTYs cleanly on pm2 restart / local restart, but nothing respawned them on startup — so a project driving an active batch stayed missing (no dev progress, no reviews) until an operator manually agent_control started each of head/dev/re1/re2. This bit the v2.5.0/v2.5.1 upgrades repeatedly.

New respawnActiveBatchAgents(cfg) runs once in the server.listen startup callback (after auto-reseed, before the watchdog starts). For each project with an active batch it respawns the project's agents via the existing spawnAgentPty. Idle projects are left alone — agents still spawn on demand (terminal-connect / batch-start) exactly as before.

"Active batch" is determined from the same source /api/batch-active and the auto-reseed gate already use: getOrComputeBatchProgress(projectId) → isBatchActiveFromProgress(progress). No new batch/pulse logic.

Fail-SAFE direction (inverse of auto-reseed's fail-CLOSED gate): if the batch state can't be proven active — the check throws, or returns null (no repo / no progress payload) — we do nothing and never spawn, so an idle project is never disturbed. Opt out via config.json restart_respawn: { enabled: false } (defaults on).

Dependency-injected (getProgress / isActiveFromProgress / spawnAgentPty / agentSessions / log), exactly like autoReseedOnStartup, so tests need neither gh nor a real pty.

EPIC Alignment

EPIC #967 — folded / isolated (per the batch brief); isolated to startup logic. Enables restart-safe overnight runs so upgrades don't strand in-flight batches. Respects the epic's stated contracts: do NOT auto-spawn idle projects; keep #972's clean shutdown; do not change the batch/pulse model.

Self-Verification

Adversarial diff re-read — the only production change is one new async function + one guarded call in the startup callback + one export. shutdown(), the spawn internals, and the batch/pulse model are untouched.

Kill-list / safety invariants:

• Do NOT auto-spawn idle projects — !active (false OR null) → skip; test 2/2b assert an idle/cleared project spawns 0 agents.
• Fail-safe on unknown state — getProgress throwing or returning null → 0 spawns (tests 3, 4).
• Do NOT change [#S6] shutdown() cleanup + fix quadwork stop PID handling #972 clean shutdown — no edits to shutdown().
• Do NOT alter the batch/pulse model or spawn internals — reuses exported getOrComputeBatchProgress / isBatchActiveFromProgress and calls spawnAgentPty as-is (orchestration only).
• Idempotent — an already-live agent (isPtyAlive(term)) is skipped, not double-spawned (test 6).
• Never blocks boot — the startup call is wrapped in try/catch; the function never throws (per-project + per-agent guards).

Build + test evidence:

• npm test → 64 passed, 0 failed, 2 skipped (pre-existing Jest-style skips).
• New server/restartRespawn.test.js (8 cases): active→4 restored (suppressed lifecycle msg); idle-complete→none; cleared-active→none; unknown(null)→none; throw→none; opt-out→none; idempotent-skip of a live agent; failing-spawn logged & not counted restored; malformed projects ignored. Drives the real isBatchActiveFromProgress via crafted progress payloads.
• node server/pack-smoke.test.js → PASS.
• QUADWORK_SKIP_LISTEN=1 node -e "require('./server/index.js')" → loads OK.

Acceptance criteria (1:1):

• Active-batch project's 4 agents running again after restart, no manual start ✅ (test 1)
• Idle project NOT auto-spawned (unchanged) ✅ (test 2/2b)
• Startup logs which projects/agents were restored ✅ ([respawn] <project>: active batch — restored agents: …)
• No regression to [#S6] shutdown() cleanup + fix quadwork stop PID handling #972 clean shutdown; existing tests pass ✅

Deviations

• Active-batch source, not a shutdown running_agents marker — the issue offered either; the brief directed the batch-progress/active-batch source, which needs no change to shutdown() and reuses the exact predicate the rest of the system trusts.
• Agent set derived from project.agents keys (falling back to the canonical head/re1/re2/dev) so legacy worktree layouts are covered; spawnAgentPty returns {ok:false} for an unknown agent, so a stray key can't crash boot.

[project7-interns]

Verdict: APPROVE

Epic Alignment: PASS

Advances #967's reliability goal and #992's contract: restore active-batch agents after restart while leaving idle projects alone and keeping #972 shutdown/spawn internals unchanged.

Checked (evidence)

• Structural gate: PR body contains filled ## EPIC Alignment, ## Self-Verification, and non-UI ## Deviations.
• Issue/epic context: gh issue view 992 + gh api repos/realproject7/quadwork/issues/967; #992 requires active-batch respawn only and no batch/pulse or #972 shutdown changes.
• Live diff scope: gh api repos/realproject7/quadwork/pulls/995/files → exactly server/index.js (+88) and server/restartRespawn.test.js (+161).
• Active source reuse: server/index.js:2254 uses routes.getOrComputeBatchProgress; server/index.js:2255 uses routes.isBatchActiveFromProgress, matching server/routes.js:3411 and /api/batch-active at server/routes.js:3431.
• Idle/unknown fail-safe: server/index.js:2269-2280; tests cover complete, cleared, null, and throwing progress at server/restartRespawn.test.js:67, :79, :89, :100.
• Active restore/idempotence/logged failures: server/index.js:2286-2307; tests cover four-agent restore at server/restartRespawn.test.js:51, live-session skip at :122, and failed spawn logging at :135.
• Startup placement: server/index.js:2376 runs after auto-reseed and before watchdog; no shutdown path changed.
• Riskiest part of this diff: startup orchestration can accidentally wake idle projects; acceptable because unknown/complete/cleared states all skip and the predicate is the existing lifecycle signal.
• Kill-list: scanned all items — clean.
• CI: gh pr checks 995 → test pass 49s.

Findings

• None.

Decision

Approve at 38c78780503237b14674500fb1376d018d75cc35. The implementation is narrow startup orchestration around existing batch-state and spawn primitives, with focused tests for the idle-project and unknown-state regressions that would violate #992.

[realproject7]

@re2 APPROVE at 38c78780503237b14674500fb1376d018d75cc35

Scope is exactly the two files claimed: server/index.js (new respawnActiveBatchAgents + one guarded startup call + export) and new server/restartRespawn.test.js. Startup-only orchestration; agent spawn internals and #972 shutdown are untouched.

Checked (evidence)

• Idle projects are never disturbed (core Auto-respawn agents on server restart when a batch is active (no manual agent_control needed) #992 invariant). Respawn proceeds only when active === true; if (!active) skips both false (no active batch) and null (unknown). "Active" comes from the SAME source as /api/batch-active and the reseed gate: routes.getOrComputeBatchProgress → routes.isBatchActiveFromProgress, which returns true only when items.length > 0 && !complete (and false on liveActiveBatchCleared, null on missing progress) — server/routes.js. No new batch/pulse logic.
• Fail-SAFE, correct direction. getProgress throwing → skip + log (server/index.js:2274); null/false → skip. An unknowable state never spawns, so an idle project is never disturbed — and since the pre-existing behavior was "operator restarts manually," a missed active project is no regression.
• [#S6] shutdown() cleanup + fix quadwork stop PID handling #972 shutdown untouched. The only edit near shutdown() is the module.exports line; the shutdown body is unchanged (verified by diff). Startup call sits after the auto-reseed try/catch, before butler/watchdog, and is itself wrapped in try/catch so it can never crash boot; it runs inside the listen callback so the HTTP server is already accepting connections.
• Spawn internals untouched / correct reuse. Reuses spawnAgentPty(project.id, agentId, { suppressLifecycleMsg: true }) (server/index.js:625, option honored at :665), which returns {ok,error} and never throws through — the loop handles !ok by logging, and wraps in try/catch anyway.
• Idempotent. Skips an agent whose session is isPtyAlive(existing.term) (server/index.js:750), so a terminal-connect race can't double-spawn. Per-agent keys taken from project.agents (legacy-safe), falling back to the canonical four.
• Opt-out honored via restart_respawn.enabled: false (defaults on).
• Tests + CI. Independently at 38c7878: node server/restartRespawn.test.js → all 8 cases passed (active→4 restored w/ suppressed lifecycle msg; idle-complete/idle-cleared→none; null→fail-safe; throw→skip; opt-out→none; idempotent-skip-live; failing-spawn logged/uncounted; malformed projects ignored), driving the REAL predicate. Full suite 64 passed, 0 failed, 2 skipped; index.js loads. CI gh pr checks 995 → test pass 49s. state OPEN, mergeable MERGEABLE, head unmoved.

No blocking issues. Standing no-merge rule — operator merges.