[codex] fix(recovery): persist job-missing invocation evidence

[kkroo]

Thinking Path

• Paperclip is the open source app people use to manage AI agents for work.
• Its heartbeat/recovery services reconcile runs executed in external Kubernetes lifecycle Jobs.
• A job_missing result means the Job disappeared after the adapter-invocation phase, so its external work may already have produced a durable side effect such as a GitHub review.
• Current master could reinterpret that terminal state through stale transient metadata and enqueue a retry or recovery continuation, risking a duplicate side effect.
• The App-authored source PR fix(recovery): persist job-missing invocation evidence (BLO-18106) #1048 also contains broader wake/backstop work; this independent successor deliberately keeps only the still-needed job-missing replay-safety slice.
• This pull request persists the invocation observation and treats job_missing as non-retryable across direct retries and recovery continuations.
• The benefit is fail-closed recovery: an ambiguous missing Job is surfaced for intervention instead of automatically replaying potentially non-idempotent work.

Linked Issues or Issue Description

• Refs BLO-18106 — persist and respect missing-Job invocation evidence.
• Related: fix(recovery): persist job-missing invocation evidence (BLO-18106) #1048 is the App-authored source PR. GitHub search found no independent human successor; this draft intentionally excludes its unrelated wake/backstop and k8s_pod_schedule_failed changes.

What Changed

• Preserve job_missing as the terminal error when PR-review evidence is incomplete, while retaining the evidence error as recovery metadata.
• Persist externalLifecycleRecovery.adapterInvocationStarted when the reaper finalizes a missing Job.
• Prevent direct automatic retry, immediate recovery, and continuation recovery from replaying job_missing work.
• Add regression coverage for stale transient metadata, missing-Job reaper paths, and continuation classification; leave k8s_pod_schedule_failed retry behavior unchanged.

Verification

PAPERCLIP_HOME=/tmp/paperclip-pr1048-test-home PGOPTIONS='-c client_min_messages=warning' \
  pnpm --filter @paperclipai/server exec vitest run --no-file-parallelism --silent --reporter=dot \
  src/__tests__/heartbeat-retry-scheduling.test.ts -t 'does not replay job_missing'
# 1 passed, 55 skipped

PAPERCLIP_HOME=/tmp/paperclip-pr1048-test-home PGOPTIONS='-c client_min_messages=warning' \
  pnpm --filter @paperclipai/server exec vitest run --no-file-parallelism --silent --reporter=dot \
  src/__tests__/heartbeat-process-recovery.test.ts -t 'exact-missing|missing-Job|missing Job'
# 7 passed, 167 skipped

PAPERCLIP_HOME=/tmp/paperclip-pr1048-test-home \
  pnpm --filter @paperclipai/server exec vitest run --silent --reporter=dot \
  src/services/recovery/service.job-missing-replay.test.ts
# 1 passed

PAPERCLIP_HOME=/tmp/paperclip-pr1048-test-home pnpm --filter @paperclipai/server typecheck
# passed

git diff --check origin/master...HEAD
# passed

The one intervening master commit was UI-only and did not overlap this slice; the successor was rebased cleanly onto edba5811.

Risks

• Intentional fail-closed behavior: a job_missing run now blocks/requires recovery rather than automatically retrying, which avoids duplicate external side effects at the cost of surfacing ambiguous failures sooner.
• No schema migration or UI change.
• This draft does not import fix(recovery): persist job-missing invocation evidence (BLO-18106) #1048's broader recovery-action/wake-backstop changes, which need independent review.

Model Used

• OpenAI GPT-5 via Codex, with tool use and local code execution.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots (N/A — no UI change)
• I have updated relevant documentation to reflect my changes (N/A — no user-facing documentation is affected)
• I have considered and documented any risks above
• All Paperclip CI gates are green (not yet run for this draft)
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups (not yet run for this draft)
• I will address all Greptile and reviewer comments before requesting merge

[allyblockcast]

🔗 Paperclip issue: BLO-18106

[allyblockcast]

🔗 Paperclip issue: BLO-18106

[kkroo]

Superseded by merged #1207. Current master already preserves job_missing as the terminal outcome, records externalLifecycleRecovery.adapterInvocationStarted, treats job_missing as non-retryable in direct and continuation recovery, and fail-closes the active review stage with durable gate delivery. Rebasing this narrower draft now leaves only duplicate helpers/tests, so closing it avoids another redundant CI and review cycle.