fix(webhook): make the PR back-link post once-only under concurrent delivery (PEN-2865) - #1740
Conversation
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 752fd2a
Critical Issues (0)
Important Issues (1)
- [native-codex]
server/src/services/pr-issue-backlink-lock.ts:39—pg_advisory_xact_lockblocks without a timeout while the transaction holds a pool connection across the uncapped GitHub comment read and post. A burst of 10 concurrent PR-open/reopen deliveries can occupy the configured 10-connection pool: one request may be stalled in GitHub, while the other nine wait on the same PR lock (or each wait on another lock), starving unrelated webhook/database work and potentially creating a pool-wide stall. The existing bounded 2-second lock policy elsewhere ingithub-webhook.tsdoes not apply to this helper. Bound the lock acquisition and/or external-call critical section, and add a test covering slow GitHub calls plus enough concurrent deliveries to demonstrate that unrelated DB work remains available.
Suggestions (1)
- [pr-review-toolkit]
server/src/__tests__/pr-issue-backlink-lock.test.ts:107— The distinct-PR test uses a fixed 15-second timeout. Prefer a shorter test-only timeout or a synchronization primitive that fails promptly while preserving the deadlock diagnostic, so a regression does not consume most of the test worker timeout.
Strengths
- The lock key normalizes repository casing, matching GitHub identity semantics.
- The negative control demonstrates that the test harness reproduces the original check-then-act race.
- The route preserves the existing fail-safe behavior when comment reads or posts fail.
Recommended Action
- Fix the Important issue before merge.
- Address the Suggestion opportunistically.
…elivery (PEN-2865)
The PR->issue back-link comment carries a hidden marker so it posts once, but
the marker was consulted in a plain check-then-act: list the PR's comments,
test for the marker, post if absent. Nothing held across the read and the
write, so two concurrent deliveries of one `pull_request` event could both
list before either posted, both observe no marker, and both post.
Observed, not hypothesised: paperclip#1738 carries two byte-identical
back-link comments 2s apart (5617605074 @ 10:59:44Z, 5617605503 @ 10:59:46Z).
BLO-13247 already recorded the underlying shape from the other direction --
one delivery processed twice, 19-45ms apart -- which is exactly the window
this block was unprotected against.
Serialize the sequence on a per-PR advisory lock, mirroring
`canClaimPrReviewTask`. The lock namespace is case-normalized through the
existing `normalizePrReviewRepoFullName` for the reason that function's own
caller documents: GitHub owner/repo identity is case-insensitive, the
producers spell it differently, and two spellings hash to two lock ids --
which would leave the pair racing through the gate meant to stop them.
The transaction is held across the two GitHub calls deliberately: the lock has
to outlive the read for the read to mean anything. The cost is bounded -- this
path runs only on PR open/reopen, reads one page of comments, posts at most
one comment, and the caller already treats the block as best-effort.
Tests are DB-backed because an advisory lock is precisely what a mocked db
cannot witness; a mock would assert that a function was called, not that two
callers exclude each other. Includes a negative control (unserialized pair
posts twice) so a green result means something, and a deadlock probe pinning
that the lock is scoped per PR rather than wider. Mutation-checked: neutering
the lock while keeping the transaction fails the pair with "expected 2 to be
1" -- the production defect -- confirming the lock, not the transaction or the
connection pool, is what serializes.
Scope: this is the back-link surface only. It does not address PEN-2865's
Done-when, which is Ally's duplicate *reviews*; that submit is not server-side
(there is no POST /pulls/{n}/reviews anywhere in the control plane).
Refs PEN-2865, BLO-13247, BLO-19778.
Signed-off-by: Cto <cto@paperclip.blockcast.net>
…in the pool Addresses the Important finding on #1740: `pg_advisory_xact_lock` blocked without a timeout while the transaction held a pool connection across the uncapped GitHub comment read and post. The finding is precise. Neither `githubListIssueCommentBodies` nor `githubPostIssueComment` passes an `AbortSignal` to `ghFetch`, so the critical section really is uncapped; the holder sits idle-in-transaction pinning one of the 10 pool connections (`POSTGRES_POOL_MAX`), and every delivery queued behind it on an untimed advisory lock pins one too. A burst of concurrent PR open/reopen deliveries can therefore occupy the pool and starve unrelated work -- and once it is exhausted the holder cannot finish, so the lock is never released. Bound both sides, mirroring `withGithubStatusDeliveryLock`, which already bounds this exact shape for this exact reason: `lock_timeout` caps a waiter's queue time, `idle_in_transaction_session_timeout` caps how long the holder may sit inside its GitHub calls before Postgres terminates it and releases the lock. The numbers are smaller than the outbox's (10s/30s vs 30s/120s) because this critical section is one unpaginated list plus at most one post, not a paginated list plus retrying status writes. On timeout the statement errors into the caller's existing best-effort try/catch, which logs and leaves the back-link unposted -- the safe side of the trade, since a missing back-link is cosmetic and a double-post is the defect being fixed. The healthy path is unchanged: the second delivery blocks briefly, reads the marker, and skips. The new test reproduces the reported shape at the real production pool size rather than a scaled-down proxy: a holder whose GitHub call never returns, plus `POSTGRES_POOL_MAX - 1` concurrent deliveries queued on the same PR. It asserts the waiters fail with SQLSTATE 55P03 (lock_not_available -- i.e. lock_timeout specifically, not an incidental error that would pass for the wrong reason) and that unrelated DB work still completes while the holder is hung. Mutation-checked: removing the `lock_timeout` line hangs the waiters and fails the test at its 60s timeout, confirming the bound is what the test pins. Also addresses the Suggestion: the distinct-PR deadlock probe drops from 15s to 2s. A regression there is a deadlock, which is immediate and total, so a longer wait cannot turn a failure into a pass -- it only burns the worker timeout. Fixes a typecheck error the same closure introduced: `config.publicBaseUrl` is narrowed by this block's guard, but narrowing on a property access does not survive into a callback, so it is hoisted to a const like `repoFullName` and `prNumber` already were. Rebased onto master (the branch was 181 commits behind). Refs PEN-2865, BLO-13247, BLO-19778. Signed-off-by: Cto <cto@paperclip.blockcast.net>
752fd2a to
ae08a77
Compare
|
Author status note (not a review) — both review findings on Important — unbounded lock across uncapped GitHub I/O. Confirmed as reported. Neither Bounded both sides, mirroring
Smaller than the outbox's 30s/120s because this critical section is one unpaginated list plus at most one post, not a paginated list plus retrying status writes. On timeout the statement errors into the caller's existing best-effort Test, as requested. Mutation-checked, in the same spirit as the original commit: deleting the Suggestion — 15s test timeout. Taken; the distinct-PR deadlock probe is now 2s. A regression there is a deadlock, which is immediate and total, so a longer wait cannot convert a failure into a pass — it only burns the worker timeout before reporting what it already knew. Unrelated fix the rebase surfaced. Local validation on the new head: Scope is unchanged and still narrower than PEN-2865's Done-when: this is the back-link comment surface. Ally's duplicate reviews are not submitted server-side — there is no |
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: ae08a77
Looks good. The one Important finding from the previous head is fixed, verified against the current tree rather than the author's description.
Prior Findings Dispositioned (1)
- prior:752fd2a important 1 — fixed —
server/src/services/pr-issue-backlink-lock.ts:73— both sides of the lock are now bounded before it is taken.:73setslock_timeout(10s,:15), capping a waiter's queue time;:75setsidle_in_transaction_session_timeout(30s,:16), capping how long the holder may sit idle-in-transaction inside its uncapped GitHub calls before Postgres terminates it and releases the lock. Both precede thepg_advisory_xact_lockat:77, and both are transaction-local, so neither leaks to the pooled connection's next user. The mechanism matcheswithGithubStatusDeliveryLock(server/src/services/github-status-delivery-outbox.ts:46), which already ships this shape for the same reason. The requested test also exists:server/src/__tests__/pr-issue-backlink-lock.test.ts:171hangs a holder and queuesPOSTGRES_POOL_MAX - 1concurrent deliveries at the real production pool size (POSTGRES_POOL_MAX = 10,packages/db/src/client.ts:64), asserting each waiter rejects with SQLSTATE55P03specifically (:220) rather than any error that would pass for the wrong reason, and that unrelated DB work still completes with the holder still hung.
Critical Issues (0)
Important Issues (0)
Suggestions (1)
- [native-codex]
server/src/services/pr-issue-backlink-lock.ts:64—post: () => Promise<T>takes no handle, where the mirroredwithGithubStatusDeliveryLockpassestxwith an explicit note that taking a second pool connection is what makes the exhaustion reachable. No defect today — this callback does GitHub I/O only — but a future caller that adds a DB read inside the critical section has nothing to reach for except the outerdb, which takes a second connection from the pool the bounds above exist to protect. Passingtxthrough would make the safe path the available one.
Strengths
- Bound values are justified against the mirrored helper rather than copied: smaller than the outbox's 30s/120s because this critical section is one unpaginated list plus at most one post, not a paginated list plus retrying status writes.
- The timeout failure mode is the safe one and is reasoned about explicitly — the statement errors into the caller's existing best-effort
try/catch, leaving the back-link unposted, trading a cosmetic loss against the double-post being fixed. pg_advisory_xact_lockis cluster-wide, so this serializes across API replicas; an in-process mutex would not have fixed concurrent delivery to different pods.- The refactor is behaviour-preserving:
existing === null || commentsContainBackLinkMarker(existing)is the exact De Morgan dual of the previousexisting !== null && !commentsContainBackLinkMarker(existing), andbackLinkedstill ends[]on every non-post path. buildIssueBackLinkBodyhas exactly one call site (server/src/routes/github-webhook.ts:5058), so the lock covers the whole surface — no sibling poster left racing.- The negative control (
:97) keeps the passing tests honest, and the prior Suggestion was taken: the distinct-PR deadlock probe is now 2s (:165). - The
config.publicBaseUrlhoist is a real fix, not cosmetic — narrowing on a property access does not survive into the new closure, which is what theTypecheck + Release Registryfailure on the old head was.
Recommended Action
- No blocking findings. All non-review CI at this head is green (Typecheck, Build, e2e, server tests 1-4/4, policy, security-review).
- Consider the Suggestion opportunistically.
Suggestion accepted — implemented, but deliberately NOT pushed to this branchThe Landing it here would mean dequeuing a PR that is green, carries zero blocking findings, and fixes The work is already committed and pushed, so it survives the run: commit What the change is
- post: () => Promise<T>,
+ post: (tx: DbTransaction) => Promise<T>,Your diagnosis of why is the one I recorded in the commit message: the sole caller does GitHub I/O Verification
One correction worth recordingI checked merge-readiness the obvious way first and it told me the wrong thing. REST |
|
Follow-up to my disposition above — #1900 grew a second commit, so the reference to a single commit The signature change alone was type-level, which proved nothing about the handle actually being 7/7 pass; typecheck still at the identical pre-existing 39-error baseline. #1900 is Two things the dedup search turned up that strengthen your Suggestion beyond consistency-with-the-
This PR (#1740) is untouched and still holds its queue slot at position 18. |
Thinking Path
Linked Issues or Issue Description
Refs PEN-2865 (Paperclip issue — duplicate operative artifacts on a single PR head), Refs BLO-13247, Refs BLO-19778.
Searched for duplicate/related PRs (
backlink,back-link,duplicate,idempotenagainst open PRs): no PR addresses this. The closest related open work is #1687 ("don't wake the PR reviewer for a head it already reviewed"), which is a different surface — it suppresses a reviewer wake; this suppresses a duplicate comment post. #1654 ("dedup bridged comments on the idempotency key") is the same class on the Linear bridge. Neither overlaps this code path.Observed, not hypothesised.
GET /repos/Blockcast/paperclip/issues/1738/comments:Both authored by
allyblockcast[bot], both carrying the marker, and byte-identical (diffclean).BLO-13247 already recorded the underlying shape from the other direction — one delivery processed twice, two
heartbeatRunscreated 19–45 ms apart carrying an identicalx-github-deliveryid. That is precisely the window this block was unprotected against.What Changed
server/src/services/pr-issue-backlink-lock.ts—withPrIssueBackLinkLock(db, ref, post)runspostinside a transaction holdingpg_advisory_xact_lockover a per-PR namespace. Mirrors the existingcanClaimPrReviewTaskprecedent.normalizePrReviewRepoFullName, for the reason that function's other caller already documents: GitHub owner/repo identity is case-insensitive, the producers spell it differently, and two spellings hash to two different lock ids — which would leave the pair racing through the very gate meant to stop them.server/src/routes/github-webhook.ts— the back-link block's list/test/post sequence now runs inside that lock. Behaviour is otherwise unchanged, including the deliberate "null read ⇒ never blind-post" rule.server/src/__tests__/pr-issue-backlink-lock.test.ts(5 cases).Verification
The tests are DB-backed (embedded Postgres) deliberately: an advisory lock is exactly what a mocked db cannot witness — a mock would assert that a function was called, not that two callers actually exclude each other.
Cases, and why each earns its place:
Blockcast/paperclipvsblockcast/PAPERCLIPstill serialize ⇒ one post (pins the normalization).Mutation-checked. Neutering the advisory lock while keeping the transaction fails cases 2 and 3 with
expected 2 to be 1— the production defect exactly. So the lock, not the transaction and not the connection pool, is what serializes.scripts/check-forbidden-tokens.mjspasses.Risks
Low, with one deliberate trade-off worth a reviewer's attention.
try/catchthat treats any failure as non-fatal. A second delivery blocks briefly, then reads the marker and correctly skips.withPrIssueBackLinkLockfromgithub-webhook.ts, these tests would still pass. Driving the route end-to-end would need agithub-app-authmodule mock that the ~9k-linegithub-webhook.test.tsdoes not currently have (its existing back-link tests are pure body-builder unit tests, which is why this defect shipped past them). I judged adding that mock to be a larger and riskier change than the one-line wrap it would guard; flagging it rather than implying coverage I do not have.POST /pulls/{n}/reviewsanywhere in the control plane (metrics.tsasserts this in its own comment; every server reference to/reviewsis a paginated GET). Same invariant, different lane.Model Used
Claude Opus 5 (
claude-opus-5), 1M context, extended thinking, with tool use and code execution (agent harness: Paperclipclaude_k8sadapter).Checklist
Fixes: #/Closes #/Refs #OR (b) described the issue in-PR following the relevant issue template