[#957] Lifecycle-aware cleanup of agent backend temp dirs

[realproject7]

Closes #957

Summary

Claude Code overrides TMPDIR to /tmp/claude-{uid} and never bounds it. On Linux hosts where /tmp is mounted with a per-user quota (usrquota), that dir accumulates until the quota is exhausted — after which every Claude bash call fails silently (exit 1, empty stdout/stderr) because Claude can't write its temp/sandbox-setup files before exec. Near-impossible to diagnose (masquerades as a bwrap/sandbox failure). QuadWork owns the agent lifecycle, so it's the right layer to keep backend temp bounded.

• server/temp-cleanup.js (new) — pure, never-throwing stale sweep. Deletion is strictly age-based (max(mtime, atime, ctime) vs a cutoff, like systemd-tmpfiles): /tmp/claude-{uid} is shared by all of a user's claude agents and per-entry ownership isn't resolvable, so age is the safe signal — anything a live session still touches stays fresh and is spared. Covers claude-{uid} entries + stray gemini-client-error-*.json crash dumps; codex keeps state under ~/.codex (no /tmp sweep needed). Cross-platform: no process.getuid (Windows) → claude sweep skipped; missing dirs → no-op. Config-gated via temp_cleanup.{enabled, max_age_hours} (default enabled, 72h).
• server/index.js — sweeps on agent teardown (stopAgentSession, deferred via setImmediate, stale-only) and on an hourly timer + one at boot. The timer handle is captured in _tempSweepHandle and cleared in shutdown() per the [#S6] shutdown() cleanup + fix quadwork stop PID handling #972 clean-shutdown contract. A reentrancy guard prevents teardown/periodic overlap. Agent spawn path and PTY logic untouched.
• Docs — troubleshooting.md (full silent-exit-1 post-mortem + manual purge + config) and a install-vps.md note.
• Tests — server/temp-cleanup.test.js: stale removed / live-session file preserved / gemini dumps / missing-dir & null-uid no-op / never-throws / settings resolution.

Reimplemented fresh against current main — the old PR #960 bitrotted after #968/#972/#975 reworked index.js. PR #960's module design was used as reference; the index.js integration targets current main (captured-handle + shutdown() clear, which the old PR lacked).

EPIC Alignment

EPIC #967 — folded / isolated (per the batch brief). No sibling contracts to reconcile; this ticket stands alone. Change is confined to a new module + lifecycle wiring and does not alter EPIC-tracked surfaces.

Self-Verification

Adversarial diff re-read — production changes: new temp-cleanup.js (pure), one require, backendTempSweepTick + teardown setImmediate, one captured-handle timer block, one shutdown() clear line. No spawn/PTY code touched.

Kill-list / safety invariants:

• Never delete a live/running agent's temp — deletion gated by newestTimeMs(st) < cutoff; a live session's freshly-touched files test fresh and are kept (asserted: live-session-file preserved, kept=1). Teardown sweep is stale-only and only reaches a session's own temp once that session is gone.
• Respect [#S6] shutdown() cleanup + fix quadwork stop PID handling #972 clean shutdown — _tempSweepHandle captured and cleared in shutdown() alongside the other timers (the old PR [#957] Lifecycle-aware cleanup of agent backend temp dirs #960 used a bare setInterval — fixed here).
• Do NOT touch the agent spawn path or PTY logic — only stopAgentSession (teardown) gained a deferred setImmediate; spawn/PTY untouched.
• Never throws — sweepBackendTemp wraps everything; backendTempSweepTick try/catch/finally; verified against a bogus tmpRoot.
• Cross-platform — null uid (Windows) and missing dirs → clean no-op (asserted).

Build + test evidence:

• npm test → 63 passed, 0 failed, 2 skipped (pre-existing Jest-style skips).
• node server/pack-smoke.test.js → PASS (24 shipped Node files; temp-cleanup.js ships under server/).
• QUADWORK_SKIP_LISTEN=1 node -e "require('./server/index.js')" → loads OK.

Acceptance criteria (1:1):

• Backend temp bounded during multi-day operation ✅ (age-based hourly + teardown sweep)
• No active session's temp deleted mid-use ✅ (conservative 72h + newest-timestamp selection; test)
• Covers claude + gemini; codex documented as N/A (state under ~/.codex) ✅
• Cross-platform safe (Linux primary; macOS/Windows harmless no-op) ✅ (test)
• Runs on teardown AND a periodic timer ✅
• Documented (troubleshooting.md + install-vps.md) ✅

Deviations

• Codex not swept — the issue lists codex as "(+ confirm)". Codex stores state under ~/.codex, not /tmp, so there's nothing to sweep; documented in the module header rather than adding a dead code path.
• Age-based, not PID-ownership — the issue suggests "not-owned-by-a-live-session". Because /tmp/claude-{uid} is shared across all of a user's claude agents, per-entry live-ownership isn't resolvable; the conservative 72h max(mtime,atime,ctime) window is the safe equivalent (a live session keeps touching its files, so they never age out). This matches the issue's own fallback: "conservative age + atime-based selection."

[project7-interns]

Verdict: APPROVE

Epic Alignment: PASS

#957 is folded under #967 as an isolated lifecycle fix; this adds bounded backend-temp cleanup without changing spawn/PTY behavior and clears the new timer in shutdown per #972.

Checked (evidence)

• Structural gate: gh pr view 994 --json body → filled ## EPIC Alignment and ## Self-Verification; non-UI PR.
• Issue/epic context: gh issue view 957 + gh api repos/realproject7/quadwork/issues/967 → lifecycle cleanup for /tmp/claude-{uid}, conservative age-based live-safety, periodic + teardown sweep, docs required.
• Diff scope: gh api repos/realproject7/quadwork/pulls/994/files → server/temp-cleanup.js, server/temp-cleanup.test.js, server/index.js, docs/troubleshooting.md, docs/install-vps.md.
• Lifecycle wiring: teardown sweep at server/index.js:793, boot/hourly timer at server/index.js:2064, shutdown clear at server/index.js:2317; no spawn/PTY hunks in the diff.
• Cleanup behavior: stale predicate and Claude/Gemini sweep at server/temp-cleanup.js:33 and server/temp-cleanup.js:67; settings default/opt-out at server/temp-cleanup.js:116.
• Tests: stale removed + fresh live-session file preserved at server/temp-cleanup.test.js:36; gemini dump coverage at server/temp-cleanup.test.js:57; null uid/bogus root/settings at server/temp-cleanup.test.js:74.
• Riskiest part of this diff: age-only cleanup inside shared /tmp/claude-{uid}; acceptable because it keeps the shared dir itself, deletes only entries older than the conservative cutoff, exposes max age/disable config, and documents the ownership limitation.
• Kill-list: scanned all items — clean.
• CI: gh pr checks 994 → test pass 52s.

Findings

• None.

Decision

Approving at 995d94ec98bbd0dea1d2c364fe80493f625ca53f; the change meets #957's lifecycle cleanup and documentation acceptance criteria with focused tests and clean shutdown integration.

[realproject7]

@re2 APPROVE at 995d94ec98bbd0dea1d2c364fe80493f625ca53f

Fresh reimplementation against current main (old #960 correctly treated as reference only). Scope is exactly the 5 files claimed: server/temp-cleanup.js, server/temp-cleanup.test.js, server/index.js, docs/troubleshooting.md, docs/install-vps.md. Agent spawn / PTY path is untouched.

Checked (evidence)

• Never deletes a live session's temp (core Lifecycle-aware cleanup of agent backend temp dirs (/tmp/claude-{uid}) to prevent /tmp quota exhaustion #957 invariant). Deletion is strictly age-based on newestTimeMs = max(mtime,atime,ctime) (server/temp-cleanup.js:34); removeIfStale keeps anything with newestTimeMs >= cutoffMs (server/temp-cleanup.js:47), so a live agent that keeps touching files under the shared /tmp/claude-{uid} is always spared. The 72h default is generous. The deviation from per-entry "not-owned-by-live-session" is documented (per-entry ownership isn't resolvable on the shared dir) and the age guard satisfies the invariant's intent. Test asserts live-session-file PRESERVED while 100h entries are removed.
• [#S6] shutdown() cleanup + fix quadwork stop PID handling #972 clean shutdown honored. Hourly timer handle captured in _tempSweepHandle (server/index.js:2071) and cleared in shutdown() (server/index.js:2317) alongside the other watchdog timers. server/shutdownCleanup.test.js PASS.
• Teardown sweep is safe. stopAgentSession fires setImmediate(backendTempSweepTick) (server/index.js:797) — deferred (never blocks stop), stale-only, reentrancy-guarded (_tempSweepRunning), and only touches a session's own temp once that session is gone.
• Never-throws. sweepBackendTemp wraps everything in try/catch collecting errors (server/temp-cleanup.js:67); backendTempSweepTick wraps in try/finally. Bogus/missing tmpRoot, null uid (Windows), and readdir failures all degrade to no-op — tested.
• Correct targeting. Sweeps entries under claude-{uid} (dir itself kept), and only gemini-client-error-*.json at the temp root — test confirms a non-gemini old root file is NEVER touched. Codex (~/.codex) correctly out of scope.
• Tests + CI. Independently at 995d94e: node server/temp-cleanup.test.js → all assertions passed; full suite 63 passed, 0 failed, 2 skipped (clean env); index.js loads. CI gh pr checks 994 → test pass 52s. state OPEN, mergeable MERGEABLE, head unmoved. Docs (troubleshooting silent-exit-1 post-mortem + install-vps note) are accurate.

No blocking issues. Standing no-merge rule — operator merges.