Skip to content

fix(webhook): make the PR back-link post once-only under concurrent delivery (PEN-2865) - #1740

Merged
kkroo merged 2 commits into
masterfrom
fix/pen-2865-backlink-once-only
Sep 18, 2026
Merged

kkroo merged 2 commits into
masterfrom
fix/pen-2865-backlink-once-only

Conversation

@allyblockcast

@allyblockcast allyblockcast Bot commented Sep 10, 2026

Copy link
Copy Markdown

Thinking Path

  • Paperclip is the open source app people use to manage AI agents for work
  • When a pull request opens, the GitHub webhook route posts a back-link comment so a human on the PR can navigate to the Paperclip issue(s) it belongs to
  • That comment is meant to be posted exactly once, and carries a hidden <!-- paperclip-issue-backlink --> marker so a redelivery can detect the earlier post and skip
  • But the marker was consulted in a plain check-then-act — list the PR's comments, test for the marker, post if absent — with nothing held across the read and the write
  • Two concurrent deliveries of one pull_request event therefore both list before either posts, both see no marker, and both post; fix(ally-guard): classify a same-lane duplicate review by body, not by timing #1738 carries two byte-identical back-link comments 2s apart
  • This pull request serializes the read-test-post sequence on a per-PR advisory lock, so the second delivery reads the marker the first one wrote
  • The benefit is that the back-link reaches a PR at most once, and the marker finally does the job it was added to do

Linked Issues or Issue Description

Refs PEN-2865 (Paperclip issue — duplicate operative artifacts on a single PR head), Refs BLO-13247, Refs BLO-19778.

Searched for duplicate/related PRs (backlink, back-link, duplicate, idempoten against open PRs): no PR addresses this. The closest related open work is #1687 ("don't wake the PR reviewer for a head it already reviewed"), which is a different surface — it suppresses a reviewer wake; this suppresses a duplicate comment post. #1654 ("dedup bridged comments on the idempotency key") is the same class on the Linear bridge. Neither overlaps this code path.

Observed, not hypothesised. GET /repos/Blockcast/paperclip/issues/1738/comments:

comment id created bytes
5617605074 2026-09-10T10:59:44Z 284
5617605503 2026-09-10T10:59:46Z 284

Both authored by allyblockcast[bot], both carrying the marker, and byte-identical (diff clean).

BLO-13247 already recorded the underlying shape from the other direction — one delivery processed twice, two heartbeatRuns created 19–45 ms apart carrying an identical x-github-delivery id. That is precisely the window this block was unprotected against.

What Changed

  • New server/src/services/pr-issue-backlink-lock.tswithPrIssueBackLinkLock(db, ref, post) runs post inside a transaction holding pg_advisory_xact_lock over a per-PR namespace. Mirrors the existing canClaimPrReviewTask precedent.
  • Lock namespace is case-normalized through the existing normalizePrReviewRepoFullName, for the reason that function's other caller already documents: GitHub owner/repo identity is case-insensitive, the producers spell it differently, and two spellings hash to two different lock ids — which would leave the pair racing through the very gate meant to stop them.
  • server/src/routes/github-webhook.ts — the back-link block's list/test/post sequence now runs inside that lock. Behaviour is otherwise unchanged, including the deliberate "null read ⇒ never blind-post" rule.
  • New DB-backed test server/src/__tests__/pr-issue-backlink-lock.test.ts (5 cases).

Verification

vitest run src/__tests__/pr-issue-backlink-lock.test.ts
  Test Files  1 passed (1)
       Tests  5 passed (5)

The tests are DB-backed (embedded Postgres) deliberately: an advisory lock is exactly what a mocked db cannot witness — a mock would assert that a function was called, not that two callers actually exclude each other.

Cases, and why each earns its place:

  1. Negative control — the unserialized pair posts twice. Without this, a green result below would prove nothing about whether the harness still reproduces the defect.
  2. Two concurrent deliveries of one PR ⇒ exactly one post.
  3. Blockcast/paperclip vs blockcast/PAPERCLIP still serialize ⇒ one post (pins the normalization).
  4. Two different PRs mutually rendezvous inside their critical sections — if the lock were scoped wider than one PR they would deadlock; a timeout turns that into a named failure rather than a hang.
  5. The lock key is the normalized repo plus the PR number.

Mutation-checked. Neutering the advisory lock while keeping the transaction fails cases 2 and 3 with expected 2 to be 1 — the production defect exactly. So the lock, not the transaction and not the connection pool, is what serializes.

scripts/check-forbidden-tokens.mjs passes.

Risks

Low, with one deliberate trade-off worth a reviewer's attention.

  • A transaction is held across two GitHub API calls. This is intentional: the lock has to outlive the read for the read to mean anything. The cost is bounded — the path runs only on PR open/reopen, reads a single page of comments and posts at most one comment, and the caller already wraps the whole block in a try/catch that treats any failure as non-fatal. A second delivery blocks briefly, then reads the marker and correctly skips.
  • No behavioural change when there is no contention — the single-delivery path is the same read, same test, same post.
  • Coverage boundary, stated plainly: the tests pin the lock, not the call-site wiring. If someone later removed withPrIssueBackLinkLock from github-webhook.ts, these tests would still pass. Driving the route end-to-end would need a github-app-auth module mock that the ~9k-line github-webhook.test.ts does not currently have (its existing back-link tests are pure body-builder unit tests, which is why this defect shipped past them). I judged adding that mock to be a larger and riskier change than the one-line wrap it would guard; flagging it rather than implying coverage I do not have.
  • Scope: this is the back-link surface only. It does not close PEN-2865's Done-when, which is Ally's duplicate reviews. That submit is not server-side — there is no POST /pulls/{n}/reviews anywhere in the control plane (metrics.ts asserts this in its own comment; every server reference to /reviews is a paginated GET). Same invariant, different lane.

Model Used

Claude Opus 5 (claude-opus-5), 1M context, extended thinking, with tool use and code execution (agent harness: Paperclip claude_k8s adapter).

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have searched GitHub for duplicate or related PRs and linked them above
  • I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
  • I have run tests locally and they pass
  • I have added or updated tests where applicable
  • If this change affects the UI, I have included before/after screenshots — n/a, no UI surface
  • I have updated relevant documentation to reflect my changes — the rationale lives in the new module's docblock, next to the code it governs
  • I have considered and documented any risks above
  • All Paperclip CI gates are green — pending first run on this head
  • Greptile is 5/5 with no open P2s, recommendations, or follow-ups — pending review
  • I will address all Greptile and reviewer comments before requesting merge

@allyblockcast

allyblockcast Bot commented Sep 10, 2026

Copy link
Copy Markdown
Author

🔗 Paperclip issue: BLO-19778
🔗 Paperclip issue: PEN-2865
🔗 Paperclip issue: BLO-13247

@allyblockcast allyblockcast Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 752fd2a

Critical Issues (0)

Important Issues (1)

  • [native-codex] server/src/services/pr-issue-backlink-lock.ts:39pg_advisory_xact_lock blocks without a timeout while the transaction holds a pool connection across the uncapped GitHub comment read and post. A burst of 10 concurrent PR-open/reopen deliveries can occupy the configured 10-connection pool: one request may be stalled in GitHub, while the other nine wait on the same PR lock (or each wait on another lock), starving unrelated webhook/database work and potentially creating a pool-wide stall. The existing bounded 2-second lock policy elsewhere in github-webhook.ts does not apply to this helper. Bound the lock acquisition and/or external-call critical section, and add a test covering slow GitHub calls plus enough concurrent deliveries to demonstrate that unrelated DB work remains available.

Suggestions (1)

  • [pr-review-toolkit] server/src/__tests__/pr-issue-backlink-lock.test.ts:107 — The distinct-PR test uses a fixed 15-second timeout. Prefer a shorter test-only timeout or a synchronization primitive that fails promptly while preserving the deadlock diagnostic, so a regression does not consume most of the test worker timeout.

Strengths

  • The lock key normalizes repository casing, matching GitHub identity semantics.
  • The negative control demonstrates that the test harness reproduces the original check-then-act race.
  • The route preserves the existing fail-safe behavior when comment reads or posts fail.

Recommended Action

  1. Fix the Important issue before merge.
  2. Address the Suggestion opportunistically.

Cto added 2 commits September 15, 2026 18:32
…elivery (PEN-2865)

The PR->issue back-link comment carries a hidden marker so it posts once, but
the marker was consulted in a plain check-then-act: list the PR's comments,
test for the marker, post if absent. Nothing held across the read and the
write, so two concurrent deliveries of one `pull_request` event could both
list before either posted, both observe no marker, and both post.

Observed, not hypothesised: paperclip#1738 carries two byte-identical
back-link comments 2s apart (5617605074 @ 10:59:44Z, 5617605503 @ 10:59:46Z).
BLO-13247 already recorded the underlying shape from the other direction --
one delivery processed twice, 19-45ms apart -- which is exactly the window
this block was unprotected against.

Serialize the sequence on a per-PR advisory lock, mirroring
`canClaimPrReviewTask`. The lock namespace is case-normalized through the
existing `normalizePrReviewRepoFullName` for the reason that function's own
caller documents: GitHub owner/repo identity is case-insensitive, the
producers spell it differently, and two spellings hash to two lock ids --
which would leave the pair racing through the gate meant to stop them.

The transaction is held across the two GitHub calls deliberately: the lock has
to outlive the read for the read to mean anything. The cost is bounded -- this
path runs only on PR open/reopen, reads one page of comments, posts at most
one comment, and the caller already treats the block as best-effort.

Tests are DB-backed because an advisory lock is precisely what a mocked db
cannot witness; a mock would assert that a function was called, not that two
callers exclude each other. Includes a negative control (unserialized pair
posts twice) so a green result means something, and a deadlock probe pinning
that the lock is scoped per PR rather than wider. Mutation-checked: neutering
the lock while keeping the transaction fails the pair with "expected 2 to be
1" -- the production defect -- confirming the lock, not the transaction or the
connection pool, is what serializes.

Scope: this is the back-link surface only. It does not address PEN-2865's
Done-when, which is Ally's duplicate *reviews*; that submit is not server-side
(there is no POST /pulls/{n}/reviews anywhere in the control plane).

Refs PEN-2865, BLO-13247, BLO-19778.

Signed-off-by: Cto <cto@paperclip.blockcast.net>
…in the pool

Addresses the Important finding on #1740: `pg_advisory_xact_lock` blocked
without a timeout while the transaction held a pool connection across the
uncapped GitHub comment read and post.

The finding is precise. Neither `githubListIssueCommentBodies` nor
`githubPostIssueComment` passes an `AbortSignal` to `ghFetch`, so the critical
section really is uncapped; the holder sits idle-in-transaction pinning one of
the 10 pool connections (`POSTGRES_POOL_MAX`), and every delivery queued behind
it on an untimed advisory lock pins one too. A burst of concurrent PR
open/reopen deliveries can therefore occupy the pool and starve unrelated work
-- and once it is exhausted the holder cannot finish, so the lock is never
released.

Bound both sides, mirroring `withGithubStatusDeliveryLock`, which already bounds
this exact shape for this exact reason: `lock_timeout` caps a waiter's queue
time, `idle_in_transaction_session_timeout` caps how long the holder may sit
inside its GitHub calls before Postgres terminates it and releases the lock. The
numbers are smaller than the outbox's (10s/30s vs 30s/120s) because this
critical section is one unpaginated list plus at most one post, not a paginated
list plus retrying status writes.

On timeout the statement errors into the caller's existing best-effort
try/catch, which logs and leaves the back-link unposted -- the safe side of the
trade, since a missing back-link is cosmetic and a double-post is the defect
being fixed. The healthy path is unchanged: the second delivery blocks briefly,
reads the marker, and skips.

The new test reproduces the reported shape at the real production pool size
rather than a scaled-down proxy: a holder whose GitHub call never returns, plus
`POSTGRES_POOL_MAX - 1` concurrent deliveries queued on the same PR. It asserts
the waiters fail with SQLSTATE 55P03 (lock_not_available -- i.e. lock_timeout
specifically, not an incidental error that would pass for the wrong reason) and
that unrelated DB work still completes while the holder is hung. Mutation-checked:
removing the `lock_timeout` line hangs the waiters and fails the test at its 60s
timeout, confirming the bound is what the test pins.

Also addresses the Suggestion: the distinct-PR deadlock probe drops from 15s to
2s. A regression there is a deadlock, which is immediate and total, so a longer
wait cannot turn a failure into a pass -- it only burns the worker timeout.

Fixes a typecheck error the same closure introduced: `config.publicBaseUrl` is
narrowed by this block's guard, but narrowing on a property access does not
survive into a callback, so it is hoisted to a const like `repoFullName` and
`prNumber` already were.

Rebased onto master (the branch was 181 commits behind).

Refs PEN-2865, BLO-13247, BLO-19778.

Signed-off-by: Cto <cto@paperclip.blockcast.net>
@allyblockcast
allyblockcast Bot force-pushed the fix/pen-2865-backlink-once-only branch from 752fd2a to ae08a77 Compare September 15, 2026 19:01
@allyblockcast

allyblockcast Bot commented Sep 15, 2026

Copy link
Copy Markdown
Author

Author status note (not a review) — both review findings on 752fd2a7 are addressed, and the branch is rebased onto current master. New head: ae08a77e.

Important — unbounded lock across uncapped GitHub I/O. Confirmed as reported. Neither githubListIssueCommentBodies nor githubPostIssueComment passes an AbortSignal to ghFetch, so the critical section really is uncapped, and POSTGRES_POOL_MAX is 10 — the pool size named in the finding.

Bounded both sides, mirroring withGithubStatusDeliveryLock, which already bounds this same shape for this same reason:

  • lock_timeout (10s) caps a waiter's queue time.
  • idle_in_transaction_session_timeout (30s) caps how long the holder may sit inside its GitHub calls before Postgres terminates it and releases the lock.

Smaller than the outbox's 30s/120s because this critical section is one unpaginated list plus at most one post, not a paginated list plus retrying status writes. On timeout the statement errors into the caller's existing best-effort try/catch, leaving the back-link unposted — the safe side of the trade, since a missing back-link is cosmetic and a double-post is the defect being fixed.

Test, as requested. bounds a waiter's queue time, so a hung GitHub call cannot pin the pool runs at the real production pool size rather than a scaled-down proxy: a holder whose GitHub call never returns, plus POSTGRES_POOL_MAX - 1 concurrent deliveries queued on the same PR. It asserts (a) the waiters reject with SQLSTATE 55P03 (lock_not_available, i.e. lock_timeout specifically — not an incidental error that would pass for the wrong reason), and (b) unrelated DB work still completes while the holder is hung.

Mutation-checked, in the same spirit as the original commit: deleting the lock_timeout line hangs the waiters and fails the test at its 60s timeout. The bound is what the test pins, not the transaction or the pool.

Suggestion — 15s test timeout. Taken; the distinct-PR deadlock probe is now 2s. A regression there is a deadlock, which is immediate and total, so a longer wait cannot convert a failure into a pass — it only burns the worker timeout before reporting what it already knew.

Unrelated fix the rebase surfaced. config.publicBaseUrl is narrowed by the enclosing guard, but narrowing on a property access does not survive into the closure this PR introduced, so it is hoisted to a const exactly as repoFullName and prNumber already were. That was the Typecheck + Release Registry failure on the old head, present since the first push rather than introduced by the rebase.

Local validation on the new head: pnpm typecheck clean, pnpm build clean, and 238 tests pass across pr-issue-backlink-lock + github-webhook. (This repo has no root lint script.)

Scope is unchanged and still narrower than PEN-2865's Done-when: this is the back-link comment surface. Ally's duplicate reviews are not submitted server-side — there is no POST /pulls/{n}/reviews anywhere in the control plane — so that half is tracked separately on the issue.

@allyblockcast allyblockcast Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: ae08a77

Looks good. The one Important finding from the previous head is fixed, verified against the current tree rather than the author's description.

Prior Findings Dispositioned (1)

  • prior:752fd2a important 1 — fixed — server/src/services/pr-issue-backlink-lock.ts:73 — both sides of the lock are now bounded before it is taken. :73 sets lock_timeout (10s, :15), capping a waiter's queue time; :75 sets idle_in_transaction_session_timeout (30s, :16), capping how long the holder may sit idle-in-transaction inside its uncapped GitHub calls before Postgres terminates it and releases the lock. Both precede the pg_advisory_xact_lock at :77, and both are transaction-local, so neither leaks to the pooled connection's next user. The mechanism matches withGithubStatusDeliveryLock (server/src/services/github-status-delivery-outbox.ts:46), which already ships this shape for the same reason. The requested test also exists: server/src/__tests__/pr-issue-backlink-lock.test.ts:171 hangs a holder and queues POSTGRES_POOL_MAX - 1 concurrent deliveries at the real production pool size (POSTGRES_POOL_MAX = 10, packages/db/src/client.ts:64), asserting each waiter rejects with SQLSTATE 55P03 specifically (:220) rather than any error that would pass for the wrong reason, and that unrelated DB work still completes with the holder still hung.

Critical Issues (0)

Important Issues (0)

Suggestions (1)

  • [native-codex] server/src/services/pr-issue-backlink-lock.ts:64post: () => Promise<T> takes no handle, where the mirrored withGithubStatusDeliveryLock passes tx with an explicit note that taking a second pool connection is what makes the exhaustion reachable. No defect today — this callback does GitHub I/O only — but a future caller that adds a DB read inside the critical section has nothing to reach for except the outer db, which takes a second connection from the pool the bounds above exist to protect. Passing tx through would make the safe path the available one.

Strengths

  • Bound values are justified against the mirrored helper rather than copied: smaller than the outbox's 30s/120s because this critical section is one unpaginated list plus at most one post, not a paginated list plus retrying status writes.
  • The timeout failure mode is the safe one and is reasoned about explicitly — the statement errors into the caller's existing best-effort try/catch, leaving the back-link unposted, trading a cosmetic loss against the double-post being fixed.
  • pg_advisory_xact_lock is cluster-wide, so this serializes across API replicas; an in-process mutex would not have fixed concurrent delivery to different pods.
  • The refactor is behaviour-preserving: existing === null || commentsContainBackLinkMarker(existing) is the exact De Morgan dual of the previous existing !== null && !commentsContainBackLinkMarker(existing), and backLinked still ends [] on every non-post path.
  • buildIssueBackLinkBody has exactly one call site (server/src/routes/github-webhook.ts:5058), so the lock covers the whole surface — no sibling poster left racing.
  • The negative control (:97) keeps the passing tests honest, and the prior Suggestion was taken: the distinct-PR deadlock probe is now 2s (:165).
  • The config.publicBaseUrl hoist is a real fix, not cosmetic — narrowing on a property access does not survive into the new closure, which is what the Typecheck + Release Registry failure on the old head was.

Recommended Action

  1. No blocking findings. All non-review CI at this head is green (Typecheck, Build, e2e, server tests 1-4/4, policy, security-review).
  2. Consider the Suggestion opportunistically.

@kkroo
kkroo added this pull request to the merge queue Sep 17, 2026
@allyblockcast

allyblockcast Bot commented Sep 17, 2026

Copy link
Copy Markdown
Author

Suggestion accepted — implemented, but deliberately NOT pushed to this branch

The native-codex Suggestion is correct and I've implemented it. It is not going onto this
head, and the reason is a state that isn't visible in the usual places: this PR is in the merge
queue
(position 18, enqueued 00:19:24Z, isInMergeQueue: true). The push was rejected by the
protected-branch hook:

remote: - A pull request for this branch has been added to a merge queue. Branches that
remote:   are queued for merging cannot be updated. To modify this branch, dequeue the
remote:   associated pull request.

Landing it here would mean dequeuing a PR that is green, carries zero blocking findings, and fixes
a live double-post defect — to add a forward-looking ergonomics change that fixes no defect
today
, by the review's own assessment. That trade is plainly wrong, so the change ships as a
follow-up instead. The review's Recommended Action said "consider opportunistically"; this is the
opportunistic path that doesn't cost the queue slot.

The work is already committed and pushed, so it survives the run: commit 868ba9ce3 on branch
refactor/pen-2865-backlink-lock-tx. I'll open it as a PR against master once this one merges —
it's branched off this head, so it rebases to just the one 13-line commit at that point. Opening it
now would duplicate this PR's entire diff and trigger a redundant review of code already reviewed
here.

What the change is

post now takes the transaction handle, mirroring withGithubStatusDeliveryLock:

-  post: () => Promise<T>,
+  post: (tx: DbTransaction) => Promise<T>,

Your diagnosis of why is the one I recorded in the commit message: the sole caller does GitHub I/O
and no DB work, so it ignores the argument and behaviour is unchanged. The point is the next
caller — with no handle in scope, one that needs a DB read has only the outer db to reach for,
which checks out a second connection while this transaction still holds the first, against a
10-connection pool, under concurrent delivery, with the holder already parked inside uncapped
GitHub calls. That is exactly the exhaustion the two timeouts on this head exist to bound.

Verification

  • pnpm --filter @paperclipai/server typecheckbyte-identical 39-error set before and after
    (diffed the sorted sets). All 39 are pre-existing implicit-any noise in tool-access.ts and the
    shared validators, none in the touched files; CI's Typecheck + Release Registry is green at this
    head, so they're a local install artifact. The widened signature is accepted at the existing
    zero-arg call site with no call-site edit.
  • pr-issue-backlink-lock.test.ts6/6 pass, including the pool-exhaustion proof at the real
    POSTGRES_POOL_MAX.

One correction worth recording

I checked merge-readiness the obvious way first and it told me the wrong thing. REST
pulls/1740 reports mergeable_state: "clean", auto_merge: null, merged: false, and
gh pr view reports mergeStateStatus: "CLEAN"none of which reflect merge-queue membership.
A PR queued behind 17 others presents as freshly mergeable on every one of those fields. Only the
push hook and GraphQL pullRequest.isInMergeQueue / mergeQueueEntry reveal it. Anyone deciding
whether a head is safe to push to should read isInMergeQueue, not mergeable_state.

@allyblockcast

allyblockcast Bot commented Sep 17, 2026

Copy link
Copy Markdown
Author

Follow-up to my disposition above — #1900 grew a second commit, so the reference to a single commit
is now stale.

The signature change alone was type-level, which proved nothing about the handle actually being
the lock-holding transaction. #1900 now also asserts that behaviourally: set_config(..., true) is
transaction-local, so the helper's own lock_timeout / idle_in_transaction_session_timeout are
readable through the real transaction and through nothing else. Non-round values (7777ms /
23456ms) so it cannot pass by matching a server default, with the pooled handle read in the same
test as a control. Negative control on the test itself: swapping post(tx) for the pooled db
fails that test specifically — {lock: '0', idle: '0'} — while the other six still pass.

7/7 pass; typecheck still at the identical pre-existing 39-error baseline. #1900 is e4e5c3a27,
+58/-2 across two files.

Two things the dedup search turned up that strengthen your Suggestion beyond consistency-with-the-
sibling:

This PR (#1740) is untouched and still holds its queue slot at position 18.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 17, 2026
@kkroo
kkroo added this pull request to the merge queue Sep 17, 2026
Merged via the queue into master with commit 67a822e Sep 18, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant