fix(recovery): count an open pull request as an attendance path (PEN-2791) - #1584
Merged
allyblockcast[bot] merged 1 commit intoSep 6, 2026
Merged
Conversation
…2791) The stranded-assigned sweep counted five paths back to life -- a live run, a deferred execution wake, a pending wake interaction, an active monitor, and an unresolved blocker -- and none of them was an external event wake. `service.ts` contained exactly one occurrence of the string `github`, in a comment. That omission was not neutral. It put two controls in direct contradiction. The convergence guard's whole job, against a gate it cannot move, is to stop re-arming and clear `monitorNextCheckAt` (`clearReason: trigger_stalled`), and on an issue with no blockers that column WAS the only durable path. So the guard behaving correctly is exactly what made the row eligible for seizure: an assignee reasoning correctly about when not to poll was the assignee most likely to lose its issue and be unassigned from it. Reported on PEN-2370: a `stranded_assigned_issue` action moved a `critical` security row from `in_progress` to `blocked` and took it from its owner, on an evidence block naming no fault at all -- `latestRunStatus: succeeded`, `latestRunErrorCode: null`, `infraClassCause: false`. It fired on the absence of a counted path, not on anything going wrong, and the owner heartbeated ~2.5 minutes later. The evidence of attendance was already on the row. Since BLO-19566 the webhook persists each PR it wakes on into `issue_work_products`, and at that instant the row carried two `ready_for_review` entries written by the same webhook that had already woken that owner from those PRs earlier the same day. This reads that existing indexed table (`(company_id, issue_id, type)`); no new table, writer or index. Three deliberate narrowings: - Only webhook-written rows count, filtered on metadata source and system source-trust, mirroring the reverse lookup in `routes/github-webhook.ts`. Only a row the webhook wrote is evidence the webhook will fire again; a hand-created PR work product means someone typed a URL. - Only the succeeded-run gate consults it. That is where the sweep reasons from absence of evidence. The failed/nonretryable arms escalate on positive evidence something broke, which an open PR does not refute. - The open/terminal status split lives beside its producer and is a checked partition: the producer's return type is narrowed to it, so adding a fifth PR state without classifying it stops the build rather than silently falling out of the sweep's filter. Bounded on `updatedAt` via a new `openPullRequestAttendanceGraceMs`, because an open PR proves a wake arrives when the PR next MOVES, not on a schedule -- unbounded, an abandoned PR would hold its issue `in_progress` and unattended forever, which is this same defect entered from the other side. Deliberately not `lapsedMonitorGraceMs`: 6h is right for believing an anomaly, wrong for a human-merge wait that routinely spans days. Both graces come off one `loadConfig()`, which shells out to `tailscale` per call. Signed-off-by: Cto <cto@paperclip.blockcast.net>
Author
1 similar comment
Author
Author
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 3e8e6dd
Critical Issues (0)
Important Issues (0)
Suggestions (0)
Strengths
- The change reuses the existing webhook-maintained work-product records instead of introducing a parallel persistence path.
- Provenance filtering and a bounded grace period prevent hand-created or indefinitely stale PR rows from becoming durable attendance claims.
- The status partition and recovery behavior have focused positive and negative-control tests.
Recommended Action
- No Critical or Important changes are required from this review.
- Consider the Suggestions section opportunistically.
13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thinking Path
Started from the report, not the diagnosis. PEN-2791's filer inferred that the strandedness predicate omits external event wakes, and explicitly flagged that they had not read its source. So the first job was to confirm or refute that against the code rather than build on the inference.
Read the predicate and enumerated what it actually counts. Five attendance paths — live run, deferred execution wake, pending wake interaction, active monitor, unresolved
blocksblocker.recovery/service.tscontains exactly one occurrence of the stringgithub, and it is inside a comment. The inference was correct.Found the sharper form: the two controls are not merely unreconciled, one is the other's trigger.
hasActiveMonitorPathgoes false the instantmonitorNextCheckAtis nulled, and nulling it is precisely the convergence guard's job when a gate will not move. On a row with no blockers, that column is the only durable path. So the guard behaving correctly is what makes the row seizable — an assignee reasoning correctly about when not to poll is the assignee most likely to lose its issue.Then looked for what data already exists, before proposing a new one.
issue_pull_requestsis merged-only;pull_request_review_statecovers open PRs but has noissue_id. Neither answers "is this issue waiting on an open PR".issue_work_productsdoes — the webhook has persisted every PR it wakes on since BLO-19566, keyed to the issue and indexed on(company_id, issue_id, type).Checked the reported incident against the live row. PEN-2370 carried two
ready_for_reviewPR work products (Blockcast/paperclip#1583,#1581) written by the same webhook that had already woken that owner from those PRs earlier the same day, with its monitor cleared atmonitorAttemptCount: 8. The evidence of attendance was on the row, in an indexed table, and nothing read it. So this is not "the platform cannot see event-wake eligibility" — it is a read that was never performed.Argued myself into the bound rather than out of it. An open PR proves a wake arrives when the PR next moves, not on a schedule. Unbounded, an abandoned PR would hold its issue
in_progressand unattended forever — the same silent-darkness failure PEN-2791 was filed about, entered from the other side. I raised this against my own proposal on the issue before writing it, and it is why there is a grace and a test pinning it.Rejected the one-line version of that bound. Reusing
lapsedMonitorGraceMs(6h) was tempting and is wrong: it bounds belief in an anomaly, where 6h is generous. An open PR awaiting human merge is the normal resting state of correct work and routinely spans days here, so a 6h bound would re-seize exactly these rows on the second morning.Separated what I measured from what I quoted. The recovery-action record (
fe131799, its evidence keys, the 2m27s gap) is the filer's;/api/issues/{id}/recovery-actionsreturns404from an agent seat, so I could not re-measure it. The code comments say so rather than presenting it as my own measurement.I searched for similar or duplicate prior PRs before opening this one.
Searched
Blockcast/paperclipPRs forstranded,attendance,hasPersistedDurableWaitPathandwork-productacross all states. Closest prior art is #1330 (fix(recovery): bound how long a lapsed monitor counts as a live wake path, BLO-24782, merged), which bounded path 4; this adds a path 4 never had a sibling for. #1247 (BLO-19566) is the PR that introduced the PR work-product rows this reads. Neither duplicates this change, and no open PR toucheshasPersistedDurableWaitPath.Linked issue: PEN-2791 (
https://paperclip.blockcast.net/PEN/issues/PEN-2791). Reported instance: PEN-2370.What Changed
A sixth attendance path — an open, webhook-recorded GitHub PR — is now counted, and bounded.
server/src/services/pull-request-work-products.tsOPEN_/TERMINAL_PULL_REQUEST_WORK_PRODUCT_STATUSES, and the producer's return type narrowed fromIssueWorkProduct["status"]to exactly what it can return, making the split a checked partitionserver/src/config.tsopenPullRequestAttendanceGraceMs— 7d default, 1h–30d,OPEN_PULL_REQUEST_ATTENDANCE_GRACE_MSserver/src/services/recovery/service.tshasOpenPullRequestWakePath, consulted fromhasPersistedDurableWaitPath; both graces now come off oneloadConfig()server/src/__tests__/issue-recovery-actions.test.tsserver/src/__tests__/pull-request-work-products.test.tsNo new table, writer, index or migration — this reads storage the webhook already maintains, on an index that already exists.
Three deliberate narrowings, each with a test that would catch its removal:
routes/github-webhook.ts. Only a row the webhook wrote is evidence the webhook will fire again; a hand-created PR work product means someone typed a URL, which predicts no wake at all.inArrayseveral files away; a second hand-written status list there would drift silently the first time the mapping gains a state. Instead the producer's return type is the partition, so adding a fifth PR state without classifying it fails the build.Verification
Run on the fix commit, embedded Postgres,
--pool=forks --isolate.Typecheck —
pnpm --filter @paperclipai/server typecheck: clean.Full suite for the touched file —
issue-recovery-actions.test.ts: 170 passed (170), i.e. the 165 pre-existing tests plus the 5 new ones.pull-request-work-products.test.ts: 12 passed (12).Mutation-checked, because a passing test is not evidence of a working fix. Disabling only the new disjunct (
if (false && await hasOpenPullRequestWakePath(...))) and re-running:escalated: 1— the row is seized. That is the PEN-2370 outcome reproduced on demand, and it goes green with the disjunct restored. The other four PEN-2791 tests pass in both states by design: they are the negative controls (no PR / merged PR / hand-created PR / stale PR), and each asserts the row does escalate, so they must be insensitive to the fix. The control test in particular establishes that this row shape is genuinely at risk, without which the exemption test would prove nothing.The five sweep tests, in the order they discriminate:
ready_for_reviewPRin_progress, stays assigned, no recovery actionThe seeding builds its rows through the real
buildPullRequestWorkProductFields, not hand-written literals, so if the webhook's metadata source or source-trust constants move, the test moves with them and a stale filter fails loudly instead of silently matching nothing.Risks
blocked, never the reverse. So the failure mode of a bug in this predicate is a row stayingin_progresswith its owner, not a row being taken from one.issue_work_products.updatedAtis the recency signal. That is sound becauseupsertByExternalIdadvances it only when it accepts a strictly-newer PR event — redeliveries and out-of-order webhooks are rejected without touching it, so this cannot be refreshed by webhook noise on a PR that is not really moving. If that upsert's ordering guard were ever relaxed, this bound would weaken with it.Model Used
claude-opus-5 (Claude Code,
1mcontext)