[codex] Detect overdue scheduled retries

[kkroo]

Thinking Path

• Paperclip is the control plane for AI-agent companies, including durable heartbeat scheduling and operational visibility.
• A retry in heartbeat_runs.status = 'scheduled_retry' is intentionally parked until scheduled_retry_at is due.
• The scheduler should promote a due retry or deliberately re-defer it with a later due time.
• Current master has that lifecycle but no metric for a row that remains parked after its due time.
• The app-authored Add overdue-scheduled_retry age gauge and alert (BLO-22094) #1184 correctly identified this observability gap, and its current review required an index for the scrape-time aggregate.
• This independently authored successor preserves that detector on current master without pulling in fix(recovery,observability): stop misclassifying JSON-parse adapter failures, add queued-run-age gauge (BLO-21116) #1019's separate, currently unmerged queued-run gauge.
• The result is an agent-keyed overdue-retry gauge, alert, runbook, and online-safe supporting index.

Linked Issues or Issue Description

Refs: BLO-22094

Supersedes the app-authored implementation in #1184. Its stacked #1019 prerequisite is not on current master; this PR ports only the overdue scheduled-retry functionality that current master already supports.

What Changed

• Add a /metrics refresh that reports the oldest due-but-unpromoted scheduled_retry age per agent and explicitly resets healthy agents to zero.
• Add PaperclipOverdueScheduledRetry, configurable threshold/runbook values, Helm rendering coverage, and an operator runbook.
• Add a guarded partial index migration for status = 'scheduled_retry' lookups, plus migration regression tests for populated and invalid-precreated-index cases.

Verification

• pnpm --filter @paperclipai/db run check:migrations
• pnpm --filter @paperclipai/server exec vitest run src/__tests__/overdue-scheduled-retry-metrics.test.ts --reporter=verbose — 4 passed
• pnpm --filter @paperclipai/db exec vitest run src/heartbeat-runs-overdue-scheduled-retry-index-migration.test.ts --reporter=verbose — 3 passed
• node --test deploy/helm/paperclip/tests/prometheus-rule.test.mjs — 7 passed
• pnpm --filter @paperclipai/server exec tsc --noEmit -p .
• pnpm --filter @paperclipai/db typecheck
• pnpm --filter @paperclipai/server build

Risks

• The new aggregate runs per metrics scrape; the partial (agent_id, scheduled_retry_at) index restricted to status = 'scheduled_retry' prevents historical-table scans. Populated production databases must precreate that index concurrently, as the migration explicitly requires.
• The Helm chart rule remains chart-only on Blockcast while prometheusRule.enabled is false. Landing the live onprem monitoring copy and syncing it are intentionally out of scope for this repository-only successor.
• The 5400-second threshold is a conservative first alerting threshold; operations should revisit it if retry-promotion latency changes materially.

Model Used

OpenAI Codex, GPT-5. Tool-assisted implementation, test execution, and GitHub inspection; the context-window size is not surfaced by this environment.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots — N/A, no UI change
• I have updated relevant documentation to reflect my changes
• I have considered and documented any risks above
• All Paperclip CI gates are green — pending
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups — pending review
• I will address all Greptile and reviewer comments before requesting merge

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[kkroo]

@ally please provide formal App and User reviews for exact head 4e81494.

[github-actions]

@ally head 4e81494 has been awaiting review for 340.8h with no review on either surface (pulls/1210/reviews carries no consolidated report for this head, no ## Ally comment either) -- automated sweep (BLO-22892 / BLO-28203), not a human/agent re-ask.

Requested a review from @allyblockcast directly (native GitHub review request, not just this comment) against current head 4e81494.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 4e81494

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

• The metric refresh derives overdue age from scheduled_retry_at, explicitly resets healthy agents to zero, and keeps the Prometheus series agent-keyed.
• The migration avoids taking a blocking index-build path on populated tables and validates the concurrently precreated index definition before recording the migration.
• Tests cover overdue age, future retries, promoted retries, per-agent oldest selection, and both valid and invalid migration prerequisites.

Recommended Action

1. No Critical or Important issues found; this review is clean for the reviewed head.

[kkroo]

Superseded by changes already landed on current master; no code from this PR should be merged. Closing to remove the stale approved/conflicted entry from the merge queue.\n\n- #970: equivalent authz guard is in master via BLO-22909 / commit fdaa976.\n- #960: intended recovery changes are in master via merged #1489.\n- #1210: intended overdue scheduled-retry changes are in master via merged #1184 and subsequent hardening.