fix(external-runtime): bound reservation telemetry

[kkroo]

Thinking Path

• Paperclip is the control plane that dispatches and monitors autonomous agent work.
• Kubernetes-backed external runtimes use durable per-slot reservations to prevent two Jobs from launching for the same capacity.
• A pool claim probes several slots internally, but callers need telemetry that represents one dispatch attempt and its final result.
• The prior implementation emitted contention and refresh work for every probe, while reconciliation re-emitted launched for an already launched Job.
• This current-master successor of fix(external-runtime): stop amplifying reservation telemetry per slot probe (BLO-23009) #1151 moves claim telemetry to the public claim paths and emits launched only for a state transition.
• It also keeps late Job-UID enrichment idempotent, so receiving a UID after the Job was already launched neither re-books telemetry nor rewrites the unchanged run stamp.
• The benefit is lower metrics-refresh load and reservation telemetry that reports real dispatch and lifecycle events rather than reconciliation or scan depth.

Linked Issues or Issue Description

• Fixes: BLO-23009, the external-runtime reservation telemetry amplification incident.
• Refs fix(external-runtime): stop amplifying reservation telemetry per slot probe (BLO-23009) #1151, the App-authored original; this independent kkroo successor reproduces its intended current-master change and incorporates the outstanding late-UID review finding.
• Related but non-overlapping in-flight work: fix: yield slots during external waits #1195 changes heartbeat wait-yield behavior and does not touch external-runtime-reservations.ts.

What Changed

• Moved reservation claim event/metric refresh work out of the per-slot probe helper and into the public single-slot and pool dispatch outcomes.
• Preserved one event for each pool outcome: a successful claim, an exhausted pool, or an existing same-PR review task.
• Made Job identity recording return its reservation together with an explicit transition flag, so launched telemetry fires only when the state changes.
• Kept late UID enrichment valid without re-emitting launched or rewriting heartbeat_runs.updatedAt when externalRunId is already correct.
• Added focused coverage for exhausted pools, later-slot wins, task-key contention, repeated Job observations, and late UID enrichment.

Verification

• PAPERCLIP_HOME=/tmp/paperclip-1151-successor.Ll7wBb/runtime PAPERCLIP_INSTANCE_ID=pr-1151 pnpm test:k8s-concurrency-chaos passed: 4 test files / 278 tests, followed by 2 webhook concurrency tests.
• pnpm --filter @paperclipai/server typecheck passed.
• git diff --check passed, and the diff contains no pnpm-lock.yaml change.
• The local PR template, linked-issue, and deduplication gates pass against this body.

Risks

• The reserved, contended, and launched counter rates will intentionally fall because they now represent a dispatch attempt or state transition rather than inner probes or steady-state observations.
• Slot selection, reservation lifecycle, API return values, schema, and metric names/labels are unchanged; rollback is a revert of this single commit.
• This is a server Prometheus metric semantic correction, not a shared client telemetry-schema change; packages/shared/src/telemetry/README.md therefore requires no event-contract update.

Model Used

• OpenAI GPT-5 in the Codex agent environment, using reasoning, shell execution, local git, GitHub CLI, and focused test execution. The session does not expose a more specific model identifier or context-window size.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots — N/A, no UI surface
• I have updated relevant documentation to reflect my changes — N/A, server Prometheus semantics only; shared telemetry contract is unchanged
• I have considered and documented any risks above
• All Paperclip CI gates are green — pending the new PR run
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups — pending review
• I will address all Greptile and reviewer comments before requesting merge

[allyblockcast]

🔗 Paperclip issue: BLO-23009

[allyblockcast]

🔗 Paperclip issue: BLO-23009

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 18d18a8

Looks good. The claim path now emits one event and one aggregate refresh per externally visible dispatch attempt, while launch telemetry only records the launching to launched transition.

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

• Pool claims no longer overcount occupied slot probes as contention, and the new tests cover both exhaustion and a later-slot win.
• Repeated Job identity observations, including late UID enrichment, retain the original launch event and avoid unnecessary heartbeat-run timestamp churn.

Recommended Action

1. Merge when the repository's required checks pass.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 18d18a8

Looks good. The claim path now emits one event and one aggregate refresh per externally visible dispatch attempt, while launch telemetry only records the launching to launched transition.

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

• Pool claims no longer overcount occupied slot probes as contention, and the new tests cover both exhaustion and a later-slot win.
• Repeated Job identity observations, including late UID enrichment, retain the original launch event and avoid unnecessary heartbeat-run timestamp churn.

Recommended Action

1. Merge when the repository's required checks pass.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: c9a2c79

Looks good. The claim path now emits one event and one aggregate refresh per externally visible dispatch attempt, while launch telemetry only records the launching to launched transition.

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

• Pool claims no longer overcount occupied slot probes as contention, and the new tests cover both exhaustion and a later-slot win.
• Repeated Job identity observations, including late UID enrichment, retain the original launch event and avoid unnecessary heartbeat-run timestamp churn.

Recommended Action

1. Merge when the repository's required checks pass.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: c9a2c79

Looks good. The claim path now emits one event and one aggregate refresh per externally visible dispatch attempt, while launch telemetry only records the launching to launched transition.

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

• Pool claims no longer overcount occupied slot probes as contention, and the new tests cover both exhaustion and a later-slot win.
• Repeated Job identity observations, including late UID enrichment, retain the original launch event and avoid unnecessary heartbeat-run timestamp churn.

Recommended Action

1. Merge when the repository's required checks pass.