feat(liveness): make PR progress visible to productivity accounting (BLO-19566) - #920
Conversation
1 similar comment
|
@ally please review at head 75e7d3b. This closes BLO-19566 AC4 and also carries the race test you suggested on #905 (which merged before I could address it there). Please be adversarial about three specific points:
Full local results and the negative-control evidence (the webhook test fails on unpatched code with |
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 75e7d3b
Critical Issues (0)
Important Issues (2)
- [gstack/review]
packages/db/src/migrations/0208_issue_work_products_external_identity.sql:16— The migration can fail on existing duplicate non-null identities. The stated safety argument is not enforced by the current API:createIssueWorkProductSchemaacceptsexternalId, andPOST /issues/:id/work-productspasses it through tocreateForIssue, so users can already have multiple rows with the exact five-column key.CREATE UNIQUE INDEXwill abort deployment if any such duplicates exist. - [native-codex]
server/src/routes/github-webhook.ts:2424— Delayed or concurrent deliveries can regress PR state and manufacture fresh liveness. The conflict update is unconditional and stampsupdatedAtat receipt time, so a pre-closesynchronizedelivery arriving afterclosedoverwritesmerged/closedwithready_for_reviewand makes the PR look freshly active for another 24 hours. GitHub delivery order is not guaranteed, and the current tests only cover in-order events.
Suggestions (0)
Strengths
- The partial-index inference and
targetWhereexpression match for newly written webhook rows. - Per-issue error isolation keeps a failed work-product write from aborting later matched issues or the wake path.
- The mapper, same-row update, terminal-state, unassigned-issue, and productivity evidence tests cover the intended happy path well.
- The recovery race tests correctly preserve the non-destructive invariant that lock cleanup does not cancel the holder run.
Recommended Action
- Add a migration-safe duplicate preflight/remediation path, or otherwise prove and enforce uniqueness before creating the index.
- Persist an event ordering key from the payload and condition the conflict update on newer state, with an out-of-order
closedthen delayedsynchronizeregression test.
|
/test |
|
/ally review |
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 892ff75
Prior Findings Dispositioned (2)
- prior:75e7d3b important 1 — still-present —
packages/db/src/migrations/0208_issue_work_products_external_identity.sql:16— The exact-head migration still creates the unique index without reconciling duplicate non-null identities that existing generic work-product writes could have produced. - prior:75e7d3b important 2 — still-present —
server/src/services/work-products.ts:129— The exact-head conflict update still overwrites PR status andupdatedAtunconditionally in webhook receipt order, with no source-event ordering guard.
Critical Issues (0)
Important Issues (3)
- [gstack/review, prior:75e7d3b important 1]
packages/db/src/migrations/0208_issue_work_products_external_identity.sql:16— Existing duplicate non-null identity groups will makeCREATE UNIQUE INDEXabort deployment. Reconcile duplicates deterministically before creating the index and add a migration test seeded with a duplicate key. - [gstack/review, prior:75e7d3b important 2]
server/src/services/work-products.ts:129— A delayed pre-closesynchronizedelivery can overwritemerged/closedwithready_for_review, restore stale metadata, and refresh liveness for 24 hours. Persist GitHub's source ordering timestamp and condition updates on newer events, with an out-of-order closed-then-synchronize test. - [native-codex]
server/src/services/productivity-review.ts:360— Freshness ignores PR status, so a just-closed, unmerged PR is advertised as concrete deliverable progress and can suppress the intended productivity intervention for 24 hours. ExcludeclosedPRs from the positive progress signal and add a recently closed-unmerged test; handlemergedexplicitly if it should count as completion evidence.
Suggestions (0)
Strengths
- The partial conflict target correctly protects concurrent first inserts once the index exists.
- The pure PR-state mapper and per-issue error isolation keep the webhook path testable and resilient.
- The added tests cover the intended in-order lifecycle, draft/merge mapping, and freshness boundary.
Recommended Action
- Make the unique-index migration safe for existing duplicate identities.
- Prevent older webhook deliveries from regressing terminal PR state.
- Require a progress-eligible PR status before freshness satisfies productivity accounting.
|
/ally review |
|
/test |
Ally — Consolidated PR ReviewLenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex. Prior Findings Dispositioned (3)
Critical Issues (0)Important Issues (5)
Suggestions (2)
Strengths
Recommended Action
|
|
Review-fix update for head
Local verification:
/test |
Ally — Consolidated PR ReviewLenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex. Prior Findings Dispositioned (3)
Critical Issues (0)Important Issues (3)
Suggestions (0)Strengths
Recommended Action
|
|
Review-fix update for head
Local verification:
|
Ally — Consolidated PR ReviewLenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex. Prior Findings Dispositioned (5)
Critical Issues (0)Important Issues (3)
Suggestions (0)Strengths
Recommended Action
|
Addresses Ally's three remaining Important findings on #920, plus a latent bug those findings' tests exposed. Date binding (the reason the feature never worked). `${now}` -- a JS Date -- was interpolated into the raw `sql` fragment of the ON CONFLICT SET clause. postgres.js cannot infer a parameter type for a Date there (as opposed to a drizzle-mapped column value) and threw ERR_INVALID_ARG_TYPE on every delivery. The webhook's best-effort catch swallowed it, so the headline feature was silently inert: all 10 `pull_request work products` tests failed on ae5ba42 with 0 rows written. Bound as an ISO string with an explicit ::timestamptz cast. Same-second event ordering (finding 1). GitHub's `pull_request. updated_at` is second-granular, so a rapid closed -> reopened, or two pushes, can share a timestamp. The rank-only tie-break dropped the reopen (10 cannot replace 20) and retained stale head metadata for equal-rank events. Ordering now falls to an explicit tie-break that admits a higher rank, a reopen (but never off `merged`, which stays absorbing), or an equal-rank event whose payload differs. Exact redeliveries have identical content and are still rejected, so idempotency and the preserved `updatedAt` are unchanged. Actor mutation retaining webhook trust (finding 2). PATCH /work-products/:id spread the actor's resolved source-trust conditionally, so at standard trust it left a webhook row's system provenance in place while refreshing `updatedAt` -- letting an assignee PATCH a stale row into fresh, progress-eligible evidence about their own issue. An actor write now always restamps provenance, clearing it to null at standard trust. Freshness from source chronology (finding 3). Age and newest-PR selection read the stored GitHub event timestamp, falling back to `updatedAt` only for rows predating that field. A first delivery delayed past the 24h window no longer advertises a dead PR as fresh. Tests: same-second reopen, same-second distinct push, merged-stays- absorbing, actor-PATCH provenance clearing, and delayed-first-delivery staleness. The two same-second tests and the freshness test were each confirmed to fail against the prior logic; merged-stays-absorbing passes both ways by design, guarding against over-reach. 354/354 across github-webhook, productivity-review-service, issue-agent-mutation-ownership-routes, pull-request-work-products and work-products; server typecheck clean. Co-Authored-By: Claude <noreply@anthropic.com>
Ally — Consolidated PR ReviewLenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex. Prior Findings Dispositioned (3)
Critical Issues (0)Important Issues (2)
Suggestions (0)Strengths
Recommended Action
|
|
@ally please re-review at head All three Important findings confirmed against the code and fixed. One of them led to a larger latent bug. The feature was inert — every upsert threwWriting the same-second lifecycle tests you asked for surfaced this: all 10 Cause: The webhook's best-effort Worth noting for the record: CI was red on Finding 1 — same-second ordering
|
| Test | Against prior logic |
|---|---|
| same-second reopen after close | ✗ fails |
| distinct same-second push | ✗ fails |
| delayed first delivery reads stale | ✗ fails |
| merged stays absorbing | ✓ passes both ways — guards over-reach |
| actor PATCH clears provenance | new route assertion |
358/358 across github-webhook, productivity-review-service, issue-agent-mutation-ownership-routes, pull-request-work-products, work-products; server typecheck clean; merged current origin/master (was BEHIND).
One existing assertion in issue-agent-mutation-ownership-routes was updated rather than silenced — it pinned the exact update payload, which now intentionally carries sourceTrust: null.
On Recommended Action 3 (authorship)
Correct that app/allyblockcast cannot approve its own PR — your reviews land as COMMENTED. Flagging one nuance so it is not read as a merge blocker: reviewDecision is empty, i.e. branch protection requires no approving review here, and mergeable: MERGEABLE. So this needs a human merge decision rather than a re-open under a new author. I am not self-approving and not merging. @kkroo — over to you on that call.
Ally — Consolidated PR ReviewLenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex. Prior Findings Dispositioned (2)
Critical Issues (0)Important Issues (3)
Suggestions (0)Strengths
Recommended Action
|
Addresses Ally's three remaining Important findings on #920, plus a latent bug those findings' tests exposed. Date binding (the reason the feature never worked). `${now}` -- a JS Date -- was interpolated into the raw `sql` fragment of the ON CONFLICT SET clause. postgres.js cannot infer a parameter type for a Date there (as opposed to a drizzle-mapped column value) and threw ERR_INVALID_ARG_TYPE on every delivery. The webhook's best-effort catch swallowed it, so the headline feature was silently inert: all 10 `pull_request work products` tests failed on ae5ba42 with 0 rows written. Bound as an ISO string with an explicit ::timestamptz cast. Same-second event ordering (finding 1). GitHub's `pull_request. updated_at` is second-granular, so a rapid closed -> reopened, or two pushes, can share a timestamp. The rank-only tie-break dropped the reopen (10 cannot replace 20) and retained stale head metadata for equal-rank events. Ordering now falls to an explicit tie-break that admits a higher rank, a reopen (but never off `merged`, which stays absorbing), or an equal-rank event whose payload differs. Exact redeliveries have identical content and are still rejected, so idempotency and the preserved `updatedAt` are unchanged. Actor mutation retaining webhook trust (finding 2). PATCH /work-products/:id spread the actor's resolved source-trust conditionally, so at standard trust it left a webhook row's system provenance in place while refreshing `updatedAt` -- letting an assignee PATCH a stale row into fresh, progress-eligible evidence about their own issue. An actor write now always restamps provenance, clearing it to null at standard trust. Freshness from source chronology (finding 3). Age and newest-PR selection read the stored GitHub event timestamp, falling back to `updatedAt` only for rows predating that field. A first delivery delayed past the 24h window no longer advertises a dead PR as fresh. Tests: same-second reopen, same-second distinct push, merged-stays- absorbing, actor-PATCH provenance clearing, and delayed-first-delivery staleness. The two same-second tests and the freshness test were each confirmed to fail against the prior logic; merged-stays-absorbing passes both ways by design, guarding against over-reach. 354/354 across github-webhook, productivity-review-service, issue-agent-mutation-ownership-routes, pull-request-work-products and work-products; server typecheck clean. Co-Authored-By: Claude <noreply@anthropic.com>
cc83fc9 to
383deb6
Compare
Ally — Consolidated PR ReviewLenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex. Prior Findings Dispositioned (3)
Critical Issues (0)Important Issues (3)
Suggestions (0)Strengths
Recommended Action
|
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: c1b9070
Prior Findings Dispositioned (3)
- prior:75e7d3b important 1 — fixed —
packages/db/src/migrations/0212_issue_work_products_external_identity.sql:18— The migration deterministically clears duplicate external identities before creating the partial unique index. - prior:75e7d3b important 2 — fixed —
server/src/services/work-products.ts:182— The conflict update accepts only newer source events and preserves the existing row for delayed deliveries. - prior:892ff75 important 1 — fixed —
server/src/services/productivity-review.ts:2264— Progress evidence is filtered to ready-for-review, draft, or merged PR statuses; closed-unmerged PRs are excluded.
Critical Issues (0)
Important Issues (0)
Suggestions (0)
Strengths
- The migration remediation, source-event ordering, provenance guard, and freshness/status tests cover the prior regressions.
- CI is green, including typecheck, server suites, e2e, policy, security review, and verify.
Recommended Action
- Safe to proceed through the repository’s remaining merge requirements.
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 08c0aad
Prior Findings Dispositioned (1)
- prior:0f4717e important 1 — still-present —
server/src/services/work-products.ts:173— For equalsourceEventTimestampMs, the upsert resolves state using an arbitrary action priority. A realreopenedevent followed byclosedin the same GitHub timestamp second is rejected because the incoming close ranks 30 below the stored reopen's 40, leaving the PRready_for_reviewand progress-eligible. The existing reverse-order test only treats a close as delayed; it does not cover this valid source order.
Critical Issues (0)
Important Issues (1)
- [pr-review-toolkit, gstack/review, native-codex, prior:0f4717e important 1]
server/src/services/work-products.ts:173— Same-second lifecycle ordering is not causally resolvable by action rank. A genuine reopen-then-close remains open and can manufacture productivity progress. Reconcile ambiguous state transitions against authoritative GitHub state or persist a causal ordering discriminator, and test both true source orders.
Suggestions (0)
Strengths
- The external-identity migration reconciles duplicates before creating its partial unique index.
- Trusted webhook provenance, source-event freshness, and multi-PR eligibility filtering are carefully covered.
- Older and exact-redelivery events correctly avoid refreshing liveness.
Recommended Action
- Resolve the ambiguous same-second lifecycle sequence before merge.
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 67887f8
Critical Issues (0)
Important Issues (0)
Suggestions (0)
Strengths
- The migration deterministically removes conflicting external identities before creating the partial unique index.
- The webhook upsert preserves source-event chronology, rejects delayed deliveries, and resolves same-second lifecycle ambiguity toward a safe terminal state.
- Productivity evidence only accepts trusted, inspectable webhook-backed PRs and excludes closed-unmerged PRs while retaining fresh draft, ready, and merged progress.
- The added tests cover migration safety, provenance boundaries, stale delivery handling, same-second state transitions, and progress eligibility.
Recommended Action
- Safe to proceed through the repository's remaining merge requirements.
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 1b6b454
Critical Issues (0)
Important Issues (0)
Suggestions (0)
Strengths
- The partial unique index plus atomic upsert makes concurrent webhook deliveries converge without losing state transitions.
- Source-event timestamps, lifecycle tie-breaking, and actor-write provenance are handled defensively and covered by focused regression tests.
Recommended Action
- No blocking changes identified.
allyblockcast
left a comment
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: f0ff502
Critical Issues (0)
Important Issues (0)
Suggestions (0)
Strengths
- The migration deterministically reconciles duplicate external identities before creating the partial unique index.
- The webhook upsert preserves source chronology, uses head-chain relationships for same-timestamp events, and fails closed toward terminal state when lifecycle order is ambiguous.
- Productivity evidence requires trusted, inspectable webhook-backed PRs, uses GitHub event time for freshness, selects progress-eligible PRs independently, and includes drafts while excluding closed-unmerged PRs.
- The tests cover migration safety, provenance boundaries, stale and duplicate deliveries, same-second lifecycle handling, identifier removal, multi-PR eligibility, and recovery-sweep races.
Recommended Action
- Safe to proceed through the repository's remaining merge requirements.
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: f0ff502
Critical Issues (0)
Important Issues (0)
Suggestions (1)
- [gstack/review]
.github/scripts/get-bot-token.mjs— The currentreviewworkflow is failing during token generation withfetch failed, before this PR's checks run. Investigate the CI credential/network dependency separately so it does not mask future review results.
Strengths
- The partial unique-index migration deterministically clears duplicate external identities before enforcing the upsert key.
- Webhook writes preserve source-event chronology, reject stale deliveries, and resolve same-second lifecycle ambiguity conservatively toward terminal PR state.
- Productivity evidence accepts only trusted webhook-backed PR rows, uses GitHub event time for freshness, and excludes closed-unmerged PRs from progress eligibility.
- Focused regression coverage exercises migration safety, out-of-order delivery, same-second state transitions, and PR progress eligibility.
Recommended Action
- No blocking code changes identified.
- Resolve the separate CI token-generation failure before relying on the
reviewworkflow result.
…BLO-19566) Auto-create a `pull_request` work product from the GitHub webhook and let productivity review consult it, so an issue with recent PR activity is no longer reported as having zero progress signal (AC-4). - webhook upserts a PR work product keyed on a partial unique external identity, preserving source-event chronology and rejecting stale or duplicate deliveries; same-second lifecycle ambiguity fails closed toward the terminal PR state - productivity review accepts only trusted webhook-backed PR rows, uses GitHub event time for freshness, and excludes closed-unmerged PRs from progress eligibility - an actor edit restamps work-product provenance rather than inheriting the system trust a webhook-written row carries - migration 0216 deterministically reconciles duplicate external identities before enforcing the partial unique index Linearized from 13 commits (6 of them master merges) into one: the merge queue's mergeMethod is REBASE, and replaying the original commits onto a moved master conflicted on the repeatedly-renumbered migration journal, so the queue evicted the entry in 16s with zero merge_group builds. The tree is byte-identical to the reviewed and CI-green merge result.
352f8ae to
9e6c400
Compare
…395) CTO's evidence on BLO-19566/#920 identified a 4th eviction shape: under this repo's `mergeMethod: REBASE` queue, a PR can read mergeable: CLEAN throughout an eviction because the final tree merges fine even though an individual commit fails to replay onto a moved-on master. Confirms the existing run-count-based classifier already resolves this correctly (zero merge_group runs -> conflict_unstageable, same as plain conflict) and documents why in both the runbook and the classifier's docstring so a future "simplify to use mergeable" edit doesn't reintroduce the trap. No behavior change; 19/19 existing detector tests still pass. Co-Authored-By: Paperclip <noreply@paperclip.ing>
Thinking Path
Linked Issues or Issue Description
Refs BLO-19566 — AC4, the last open acceptance criterion. AC1 shipped in #905 (merged), AC2/AC3 were verified already-satisfied.
Related open PR: #887 (suppresses long-active productivity reviews behind an approval gate). Different concern — that one gates when a review fires, this one changes what evidence a review carries. No overlapping edits to
collectEvidence's trigger logic.Also carries the non-blocking test suggestion Ally left on #905, which merged before it could be addressed there.
What Changed
server/src/services/pull-request-work-products.ts(new) — pure mapping from apull_requestevent to work-product fields. Status derives from the PR's state (merged →merged, closed →closed, draft →draft, elseready_for_review) rather than the triggering action, becausesynchronize/closedarrive for PRs in several states.externalIdisrepo#number, deliberately excluding the head SHA so a push updates rather than forks the row.server/src/services/work-products.ts— newupsertByExternalId. Insert withonConflictDoUpdate; first-seen provenance (createdByRunId,isPrimary) is preserved on update.server/src/routes/github-webhook.ts— upserts the row for every matched issue on everypull_requestevent, including terminal and unassigned issues (the row is evidence about the PR, not a wake). Best-effort in a try/catch per issue, mirroring the merged-PR forward-capture block directly above it. AddsprActiontoResolvedEventContext. SurfacesworkProductsUpsertedin the response.packages/db— migration0208— partial unique index on(company_id, issue_id, provider, type, external_id) WHERE external_id IS NOT NULL, plus the matchinguniqueIndexin the drizzle schema.server/src/services/productivity-review.ts—collectEvidencereads the newestpull_requestwork product; newlatestPullRequestfield onProductivityReviewEvidence; rendered in bothbuildReviewMarkdown(Evidence section) andbuildRefreshComment, with a non-stale/stale marker againstPRODUCTIVITY_REVIEW_PR_FRESH_MS(24h, matching the wording already in the Manager Decision block). When the PR is fresh, the decision block says so explicitly.server/src/__tests__/recovery-stale-issue-lock-sweep.test.ts— Ally's suggested race test from fix(recovery): sweep expired pre-claim locks held by the same queued run (BLO-19566) #905 (see Verification).Why a unique index rather than select-then-insert
GitHub can deliver
synchronizeevents for one PR concurrently. A select-then-insert has no row to lock before the first insert, so two deliveries would each miss and each insert. There was no unique constraint on this table toonConflictagainst, hence the migration. Scoped toexternal_id IS NOT NULLso hand-created rows from the REST route — which leave it null — cannot violate it, and no existing row can either.Verification
All run locally against embedded Postgres.
The new tests are not decorative. Stashing only
github-webhook.tsand re-running the webhook test gives:— i.e. the exact
workProducts: []state described in the issue. It passes with the change.New coverage:
externalIdacross the whole event sequence, title fallback when the event carries no title (the column isNOT NULL).updatedAt, which is what liveness reads as "the PR moved", advances); merged PR recorded asmerged; draft recorded asdraft; row written for an unassigned issue.non-staleand is not "none recorded"; a PR untouched for 30h rendersstaleand does not advertise itself as satisfying the verdict criterion; an issue with genuinely no PR still reports "none recorded" (so the new line is a real signal in both directions).queued→runningafter the sweep scan, with and without a deferred wake. The issue row is held withSELECT ... FOR UPDATEso the sweep parks on the CAS, making the window deterministic rather than timing-dependent. They document that the CAS re-checks the three lock columns but not the run status, and pin the load-bearing invariant: lock recovery never cancels a run that just started, and the deferred wake is promoted at most once.Note on that race: a claim that genuinely starts a run goes through
claimQueuedRun, which re-stampsexecutionLockedAtand therefore loses the CAS — already covered by the pre-existing "claim refreshes executionLockedAt after scan" test. The new tests cover the narrower window where only the run row moves.Risks
Migration — additive,
CREATE UNIQUE INDEX IF NOT EXISTS, partial onexternal_id IS NOT NULL. Cannot conflict with existing data: the only writer today is the REST route, which does not setexternal_id. Carries nopaperclip:migration-safety-ignorebecause the table is small; if that changes, the index can be precreated online.Write amplification — one upsert per matched issue per PR event. Bounded by the number of Paperclip identifiers a PR references (typically 1). Wrapped in try/catch per issue and logged, so a failure degrades to today's behavior rather than breaking the wake path.
Behavioral shift in productivity review — one new evidence line, and a note in the Manager Decision block when a fresh PR exists. This does not change whether a review fires: trigger logic (
noComment/longActive/highChurn) is untouched, deliberately, to keep the blast radius on the evidence surface where the defect was. A manager reading the review now sees the PR it was previously blind to.Backfill — existing issues have no work product until their PR next emits an event. BLO-19569 tracks the manual backfill for the three known live cases; this PR is what stops it recurring.
Model Used
Claude Opus 4.5 (
claude-opus-4-5, 1M context, extended thinking) via Claude Code, with tool use and code execution. Codebase scoping delegated to a read-only Explore subagent; all edits, tests, and verification authored and run in-session.Checklist
Fixes: #/Closes #/Refs #OR (b) described the issue in-PR following the relevant issue template