fix(heartbeat): recover orphaned runs' own terminal result instead of failing them job_missing (PEN-2421)

[allyblockcast]

Thinking Path

• Paperclip is the open source app people use to manage AI agents for work, and every unit of agent work is a heartbeat run.
• Kubernetes-backed adapters execute runs inside external lifecycle Jobs, and reapOrphanedRuns is the backstop that finalizes runs whose adapter owner died before it could.
• That backstop infers run liveness from presence of the Job object. But agent Jobs set ttlSecondsAfterFinished, so the Job is correctly garbage-collected minutes after a successful agent exits.
• So a run whose owner died is swept after GC, finds no Job, and is recorded failed / job_missing — over work that had already succeeded.
• BLO-18106 (fix(heartbeat): preserve durable missing-job outcomes #791) built the mechanism to preserve a vanished-Job outcome, but scoped its evidence deliberately narrowly to PR-review runs, so every other run class still falls through to job_missing.
• The agent's own terminal verdict does survive, on the shared data PVC, in the same data/run-logs tree this server already owns — and nothing under server/ reads it.
• This pull request consults that artifact before concluding a run was lost, and only an explicit success verdict may rescue one.
• The benefit is that failure counts stop containing successes, and already-delivered work stops being re-done.

Linked Issues or Issue Description

Refs: Penstock PEN-2421 (split out of PEN-2376). Builds directly on BLO-18106 / #791, whose preserveRecordedOutcome mechanism this extends rather than replaces.

Bug report (no BLO issue exists; filing the fields in-PR).

What happened. Three runs on 2026-08-18 reached Complete=true, were TTL-collected 300s later, and were then recorded failed / job_missing:

run	agent	terminal artifact line	Job condition	recorded as	actually
49180a7e	Devops	result/success, is_error:false, 119 turns	Complete=true	job_missing	shipped a merged PR
327b81ce	Engineer	result/success, is_error:false, 74 turns	Complete=true	job_missing	succeeded
e2ba62ab	Ceo	result/success, is_error:false, 24 turns	Complete=true	job_missing	succeeded

kube_job_complete{condition="true"}=1 on all three; kube_job_failed had zero series. Job disappearance matched ttlSecondsAfterFinished to the second.

Expected. A run whose own terminal event reports success is never recorded as a failure. A run with no such evidence still is.

Why it matters beyond the wasted compute. Failure counts contain successes, so anything derived from them — failure rates, incident clustering, error-family timing — is partly measuring the reconciler. And a success marked failed is rescheduled, so already-delivered work is genuinely re-done while its issue keeps looking unfinished.

Steps to reproduce. Start a claude_k8s run; let the agent exit 0; prevent the adapter owner from finalizing (restart the worker); wait for ttlSecondsAfterFinished to collect the Job; run reapOrphanedRuns. Observed failed / job_missing before this change.

Relevant output. External lifecycle Job is missing while heartbeat run is still running

What Changed

• New server/src/services/orphaned-run-terminal-result.ts — locates a run's pod-written <runId>.pod.ndjson artifact (including the isolated/<key>/ layout, whose isolation key is adapter-side state the server does not persist) and returns the structured verdict only from its last {"type":"result"} event. Bounded tail read; never throws; never surfaces transcript content.
• finalizeExternalLifecycleTerminalRun consults it on the missing-Job path and sets preserveRecordedOutcome when the agent reported success, reusing the existing job_missing_recorded_outcome_preserved reason so a rescued run flows into the established corrective handoff.
• Records recoveredFrom: "pod_terminal_result" + the result subtype on the run's externalLifecycleRecovery metadata, and appends a lifecycle run event on every attempt (rescued or not) so the decision is auditable either way.
• Tests: 4 integration cases in heartbeat-process-recovery.test.ts, 12 unit cases for the reader.

Why this does not reopen what #791 deliberately closed

#791's risk note is explicit that the success override is narrow on purpose: generic run-authored comments, documents, and work products must never count as success proof, because treating progress artifacts as success hides genuinely lost runs. This change agrees with that and does not touch it. What it adds is not another progress proxy — it is the agent process's own structured self-report, the same subtype/is_error fields the normal finalize path already trusts via isSuccessfulAdapterResult.

The new check is also strictly additive in control flow: it runs only when the #791 path neither preserved the outcome nor found a review incomplete. No existing finalization decision changes. The neighbouring does not treat generic run artifacts as a completed missing-Job outcome test still passes unmodified, and the new success test deliberately seeds that same generic comment to show it is the verdict — not the artifact — doing the work.

One subtlety worth reviewer attention

The artifact predicate is deliberately not isSuccessfulAdapterResult. That helper accepts (exitCode ?? 0) === 0 as sufficient, and a transcript carries no process exit code — so routing a recovered result through it would score every artifact as a success, including is_error: true. The new predicate requires both halves of the structured verdict (subtype === "success" && is_error !== true), mirroring only that helper's structured clause. A unit test pins the is_error: true and missing-subtype cases.

Verification

All run locally against embedded Postgres on this branch.

• pnpm exec vitest run server/src/__tests__/orphaned-run-terminal-result.test.ts → 12 passed
• pnpm exec vitest run server/src/__tests__/heartbeat-process-recovery.test.ts -t "TTL-collected Job with a surviving terminal result artifact" → 4 passed
• pnpm exec vitest run server/src/__tests__/heartbeat-process-recovery.test.ts (full file, incl. every fix(heartbeat): preserve durable missing-job outcomes #791 PR-review case) → see check run
• pnpm --filter @paperclipai/server typecheck → clean
• pnpm run check:test-undefined-symbols → ok

The two positive regression tests were confirmed to fail against the unfixed code (guard temporarily disabled): both reported expected { …(66) } to match object { Object (status, errorCode) }. The two negative tests (is_error: true, truncated artifact) pass with and without the change by design — they exist to pin the fail-closed contract against a future over-broadening, not to demonstrate this fix.

pnpm run check:tokens fails on this branch, but it fails identically on a stashed-clean master — 470 hits, matching the literal string node across vendor/** and pre-existing server/src/services/heartbeat.ts:1-5. Unrelated to this diff.

Historical impact

Measured from surviving artifacts in one company's namespace (b1d3f3d3…) on the shared run-log PVC. Other tenants' directories exist on the same volume and were deliberately not read, so this is a scoped sample and not a fleet-wide figure.

surviving orphan artifacts		disposition under this change
496	50.3%	truncated / no result event → stays failed
227	23.0%	authoritative success → rescued
165	16.7%	agent reported an error → stays failed
98	9.9%	empty → stays failed
986

The evidence is discriminating rather than a blanket success assumption: 77% of orphans keep their failure verdict. A surviving artifact is itself meaningful — the adapter unlinks it from its cleanup finally, so its presence means the owner never completed cleanup, which is exactly the orphan case.

No backfill is proposed here. Rewriting historical terminal run records is a separate, riskier change than fixing the live path, and re-deriving verdicts for the 227 would need care about runs since superseded. Recommend deciding that separately.

Risks

• Behavioural shift, bounded to one branch. A missing-Job run that previously always failed can now finalize succeeded. It requires an explicit subtype: "success" + is_error !== true event; absent, empty, unreadable, and truncated artifacts all keep job_missing.
• Side-effect direction is the safe one. fix(heartbeat): preserve durable missing-job outcomes #791's concern was a retry duplicating an external side effect. This change suppresses retry rather than causing one, so it cannot duplicate work; the risk it does carry is the opposite — a genuinely-lost run wrongly rescued — which the fail-closed predicate and the 77% stays-failed split are aimed at.
• Filesystem dependency. Adds a stat + bounded read on the missing-Job path only. Every failure mode is caught and degrades to today's behaviour. It reads the same data/run-logs tree getRunLogStore() already resolves, so it introduces no new mount assumption; a deployment that does not share that volume simply always sees no_artifact.
• Adapter coverage. Verified for claude_k8s, which tees the artifact from its Job command. opencode_k8s lives outside this repo; if it writes no equivalent artifact its runs are unaffected (they keep today's behaviour), not broken.
• Not addressed deliberately. The 165 agent-reported-error orphans keep errorCode: "job_missing" rather than a more accurate code, because error codes drive retry policy and re-classifying them belongs in its own change. Likewise the root fix PEN-2421 suggests — having the wrapper POST terminal status so the reconciler is only a backstop — is out of scope here.

Model Used

• Anthropic Claude Opus 5, 1M context (adapter model id claude-opus-5[1m]), extended thinking, with tool use and code execution. Operating as the Penstock CTO agent; code and tests authored and run in-session against embedded Postgres.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above (23 PRs reference job_missing; fix(heartbeat): preserve durable missing-job outcomes #791 is the direct predecessor, fix(recovery): persist job-missing invocation evidence (BLO-18106) #1048/[codex] fix(recovery): persist job-missing invocation evidence #1270 cover invocation evidence, fix(worker): guard process death so crashes stop orphaning agent runs (BLO-19722) #891/fix(heartbeat): crash-time run marking + convergent recovery (BLO-20822) #952 cover worker crash death — none read the terminal artifact)
• I have either (a) linked existing issues OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots — n/a, no UI surface
• I have updated relevant documentation to reflect my changes — behaviour is documented in the module header and at the call site; no external doc describes this path
• I have considered and documented any risks above
• All Paperclip CI gates are green
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups
• I will address all Greptile and reviewer comments before requesting merge

[allyblockcast]

🔗 Paperclip issue: BLO-18106
🔗 Paperclip issue: PEN-2421
🔗 Paperclip issue: PEN-2376

[github-actions]

@ally head 9d22024 has been awaiting review for 84.1h with no review on either surface (pulls/1443/reviews carries no consolidated report for this head, no ## Ally comment either) -- automated sweep (BLO-22892 / BLO-28203), not a human/agent re-ask.

Requested a review from @allyblockcast directly (native GitHub review request, not just this comment) against current head 9d22024.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 9d22024

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

• The recovery is fail-closed: only an explicit structured success result can rescue a missing-Job run, while absent, malformed, truncated, and error artifacts retain the existing job_missing behavior.
• The new path is bounded to missing-Job finalization and preserves the existing PR-review evidence precedence, with focused unit and integration coverage for direct, isolated, error, and truncated artifacts.
• Lifecycle events record both rescued and non-rescued artifact decisions without persisting transcript content.

Recommended Action

1. No Critical or Important issues found; merge when the remaining repository gates are satisfied.

[kkroo]

Reviewed current head 9d22024. Ally reports zero Critical/Important findings; required checks and verify are green. No blocking findings.