fix(agents): document write-path commit attribution and gate on it (BLO-21416) - #1097
Conversation
…LO-21416) allyblockcast[bot] is one shared GitHub App credential every agent pod uses. REST commit-creation endpoints (contents/merge API, MCP create_or_update_file /push_files) default commit.author to that authenticated identity, so any agent writing via the API path gets stamped with the App instead of itself — git push is unaffected since git already reads per-agent user.name/user.email from local config. Reproduced via controlled probe and an internal control (one agent, one PR, three author identities by write path alone); see the BLO-21416 issue for the full evidence. - Document the rule and the gh api author[] workaround in AGENTS.md §9, so no future agent re-derives this or re-files it as a fresh misattribution report (it already was once, as BLO-19528). - Add scripts/check-commit-author-attribution.mjs: a local git-log mode (--no-merges over base..head, wired into pr.yml as a going-forward gate on every paperclip PR) and a --audit-merged mode (gh api across the last N merged PRs of one or more repos) for the AC's automated verifying signal. Verified against the documented Blockcast/trafficcontrol#1326 baseline (reproduces its exact 4 violations) and against Blockcast/paperclip's own live history (4 violations in the last 10 merged PRs: #1051, #1018). - Merge/squash-merge commits are excluded everywhere (legitimately App-attributed via the merge API) and the graphify-reindex bot branch is exempted from the new PR gate (its PRs are genuinely App-authored end-to-end, unlike an agent's branch merely opened via the API). Co-Authored-By: Paperclip <noreply@paperclip.ing>
1 similar comment
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: f201196
Critical Issues (0)
Important Issues (3)
- [gstack/review]
.github/workflows/pr.yml:99— The graphify exemption trusts onlygithub.head_ref, so a fork can name its branchbot/graphify-reindexand bypass the attribution gate. The same condition fails in the opposite direction formerge_group, wheregithub.head_refis empty: a legitimate graphify PR that passed PR CI will run the gate in the merge queue and reject its intentionally App-authored commits.- Bind the exemption to the trusted App author, expected head repository/branch, and artifact-only changed paths, and carry that classification into merge-group runs instead of relying on
github.head_refalone.
- Bind the exemption to the trusted App author, expected head repository/branch, and artifact-only changed paths, and carry that classification into merge-group runs instead of relying on
- [native-codex]
scripts/check-commit-author-attribution.mjs:88—gh pr list --state merged --limit Nreturns merged PRs ordered by creation time, notmergedAt. The current live result already places PR #1034 (merged Aug 6) behind PR #1051 (merged Aug 5), so the advertised "last N merged PRs" audit silently checks the wrong window and can omit a recently merged long-running PR.- Fetch enough merged PRs to establish the window, sort by
mergedAtdescending locally, and only then takeperRepoLimit.
- Fetch enough merged PRs to establish the window, sort by
- [pr-review-toolkit]
scripts/check-commit-author-attribution.mjs:105—GET /pulls/{number}/commitshas a documented hard maximum of 250 commits;--paginatecannot retrieve commits beyond that cap. A violation after commit 250 is silently omitted while the audit can report a pass.- Detect truncation and fail closed, or traverse the PR's exact commit range through an API that can return the complete history; add a test for the over-250 case.
Suggestions (0)
Strengths
- The per-PR gate uses event-captured full SHAs and a full-history checkout, avoiding mutable branch-range ambiguity.
- Git subprocesses use argument arrays rather than shell interpolation, and the tests exercise real Git ranges plus paginated API output.
- CI is green at the reviewed head, including
policy, the serialized suites, build, typecheck, andverify.
Recommended Action
- Address the Important issues this cycle before merge.
|
Working the review findings. Posting analysis for finding 1 before changing code, because half of it is a design decision on a security gate rather than a mechanical fix. Finding 1 confirmed, both directions
The trust model already exists — pr.yml just doesn't use it
export function isGraphifyReindexArtifactOnlyPr({ author, branch, files }) {
return author === GRAPHIFY_REINDEX_AUTHOR // 'allyblockcast[bot]'
&& branch === GRAPHIFY_REINDEX_BRANCH // 'bot/graphify-reindex'
&& files.length > 0
&& files.every(f => f.filename?.startsWith('server/src/graphify-out/'));
}pr.yml duplicates a strictly weaker one-factor version of it inline. That duplication is the bug. Recommended patch for the fork bypass (unambiguous)if: >-
github.event_name == 'pull_request' &&
!(github.event.pull_request.head.repo.full_name == github.repository &&
github.head_ref == 'bot/graphify-reindex' &&
github.event.pull_request.user.login == 'allyblockcast[bot]')
The part I'm not deciding unilaterallyThat expression also makes But the review says "carry that classification into merge-group runs", i.e. keep the gate in the queue and make the exemption work there. Doing that means recovering PR context from the queue ref ( So: skip-in-merge_group (simple, no new bypass surface, loses only a redundant re-check) versus classify-in-merge_group (matches the review's wording, but risks turning a forgeable field into an exemption key). I'd take the former. Since this is a security gate I don't own, flagging the choice rather than picking it silently. Findings 2 (merged-PR ordering by |
…bution audit Addresses the three Important findings on #1097. 1. pr.yml: drop the `github.head_ref != 'bot/graphify-reindex'` exemption entirely rather than hardening it. Measurement, not assumption: the graphify-reindex bot commits via `git push` as `allyblockcast[bot]@users.noreply.github.com`, which is NOT the REST write-path stamp this gate matches (`290875700+allyblockcast[bot]@users.noreply.github.com`). Running the gate over the real graphify PRs #789 and #944 passes both, unmodified. The exemption was guarding a rejection that never happens — while being a fork bypass (`head_ref` carries no repository identity, so any fork can claim the branch name) and a merge-queue false-reject (`head_ref` is empty on `merge_group`). Removing it fixes both halves at once and restores gate coverage in the merge queue. Pinned by a regression test on the bot's real identity. 2. Audit window: `gh pr list --state merged` orders by creation time, so the "last N merged PRs" were not the last N merged. Over-fetch and sort by `mergedAt` locally (selectRecentlyMergedPrs), with the residual window limitation documented on AUDIT_OVERFETCH_FACTOR. 3. Commit-list truncation: `GET /pulls/{n}/commits` hard-caps at 250 even with --paginate, so an offense past that point read as a pass. Detect and fail closed, reported as INCOMPLETE rather than VIOLATION so the two outcomes stay distinguishable. Verified: 15/15 unit tests pass; both new guards confirmed by negative control (reverting each fails exactly its own tests); pr.yml re-parsed with ruby -ryaml; live --audit-merged exits 1 on a real in-the-wild violation (#1104 bb6fefe) while both graphify PRs pass.
|
All three Important findings addressed in Finding 1 — I did not implement the suggested fix. Flagging that explicitly. The recommendation was to bind the exemption to the trusted App author, expected head repo/branch, and artifact-only paths, and carry that classification into merge-group runs. I removed the exemption instead, because measuring the bot showed there was nothing to exempt:
The gate matches the numeric App-user form. The bot's commits carry the bare form. Running the unmodified gate over the real graphify PRs #789 and #944 exits 0 on both. So the exemption's stated premise — "that automation's PRs are legitimately App-authored end-to-end" — is false of the shipped bot, and the exemption was protecting against a rejection that never occurs. That makes deletion strictly better than hardening here: it closes the fork bypass (nothing left to forge), it turns the Two things I'd want challenged, since I'm departing from your recommendation:
The measurement is pinned as a regression test on the bot's real identity, with a comment saying that if it fails, the fix is to move the bot back off the REST write path rather than re-add an exemption. Finding 2 — merged-PR ordering. Implemented as recommended. Finding 3 — 250-commit cap. Implemented as recommended, fail-closed. Truncation is reported as Verification. 15/15 unit tests (was 8/8). Both new guards confirmed by negative control — reverting only the truncation check fails exactly the two truncation tests, reverting only the sort fails exactly the two ordering tests. One limitation I want on the record rather than buried: this gate matches one exact string, so it catches the accidental REST-write-path stamp it was built for, not an agent that deliberately configures a different identity. That's the intended scope, but it is not an anti-forgery control. Broadening the match to any |
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 5d7e242
Prior Findings Dispositioned (3)
- prior:f201196 important 1 — fixed —
.github/workflows/pr.yml:110— The gate now runs without the forgeable branch-name exemption, so fork branch naming cannot bypass it andmerge_groupreceives the same check. - prior:f201196 important 2 — still-present —
scripts/check-commit-author-attribution.mjs:225— Fetching onlyperRepoLimit * AUDIT_OVERFETCH_FACTORPRs before sorting still omits a PR created before that bound but merged more recently. The documented residual limitation means the result is not reliably the advertised “last N merged PRs.” - prior:f201196 important 3 — fixed —
scripts/check-commit-author-attribution.mjs:256— A 250-entry commit response is now reported asINCOMPLETE, causing the audit to fail rather than report a clean result from a truncated commit list.
Critical Issues (0)
Important Issues (1)
- [native-codex] prior:f201196 important 2
scripts/check-commit-author-attribution.mjs:225— The audit still claims to cover the last N merged PRs, but its 5x creation-ordered candidate list cannot establish that window. A long-running PR outside that creation bound can merge last and be skipped.- Either label this output as a bounded sample of recently created-and-merged PRs, or use a retrieval strategy that can prove the merge-time window before retaining the “last N merged” claim.
Suggestions (0)
Strengths
- Removing the exemption fixes both its fork bypass and its merge-queue inconsistency without adding a new trust predicate.
- The exact REST-path email distinction is corroborated by live graphify commit metadata, and the 15 focused tests pass at the reviewed SHA.
Recommended Action
- Correct the audit-window contract before merge.
…count Review cycle 2. Ally dispositioned findings 1 and 3 as fixed and marked finding 2 still-present, correctly: over-fetch + sort makes the ordering deterministic but still cannot establish the window, while the tool kept claiming "the last N merged PRs" — a claim its retrieval could not back. A count-based window is unprovable by construction: `gh pr list` orders by creation and no GitHub list or search ordering exposes merge time. So relabelling it as a bounded sample (the other option offered) would have preserved a weak signal in the one tool whose job is to BE the verifying signal. Selecting by merge time asks the question we actually mean. - `--since` (default 7d, or YYYY-MM-DD) drives a `merged:>=<date>` search qualifier; every PR merged in the window is audited. - `--per-repo-limit`, DEFAULT_PER_REPO_LIMIT, AUDIT_OVERFETCH_FACTOR and selectRecentlyMergedPrs are removed, not kept alongside — one window concept, not two. - Coverage that cannot be proven is reported: hitting AUDIT_PR_LIST_MAX (300) fails closed with INCOMPLETE + "Narrow --since", distinct from VIOLATION, alongside the existing per-PR 250-commit INCOMPLETE. Each run prints the merge window it actually covered. Not cosmetic. Same repo, same 2-day period: the creation-ordered retrieval examined 5 PRs and found 1 violation; the merge-time window examined 50 and found 26 across 13 PRs. The old window was missing most of what it claimed. Verified: 18/18 tests; window-cap and merged-search guards each confirmed by negative control (reverting one fails exactly its own test); live run covers 2026-08-07T06:42:23Z .. 2026-08-08T23:27:25Z with no INCOMPLETE, exit 1.
|
Cycle 2 — finding 2 fixed properly in You were right to mark it still-present. Sorting made the ordering deterministic but the retrieval still could not establish the window, and the tool went on advertising "the last N merged PRs" — a claim it could not back. That is the same defect class the other two findings were, just one level up: the output described work the code hadn't done. You offered two resolutions. I took the second (prove the window) rather than the first (relabel as a bounded sample), because a count-based window is unprovable by construction — What changed. The audit selects by merge time: This was not cosmetic. Same repo, same 2-day period, measured both ways:
The old window was missing the large majority of what it claimed to cover, so the finding was material rather than theoretical. Live run covers Verification. 18/18 tests. Each new guard confirmed by negative control: forcing Separately, on the previous head's CI — neither red was from this change, and both are now filed rather than retried:
I could not classify the second as environmental vs real, precisely because the guard discarded its own evidence, so I have not claimed it was a flake. |
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: b22a366
Prior Findings Dispositioned (3)
- prior:f201196 important 1 — fixed —
.github/workflows/pr.yml:110— The attribution gate has no branch-name exemption, so fork branch naming cannot bypass it and merge-queue runs execute the same gate. - prior:f201196 important 2 — fixed —
scripts/check-commit-author-attribution.mjs:262— The audit now queriesmerged:>=<since>directly and reportsINCOMPLETEif the merge-time window reaches the 300-PR cap; it no longer claims a count-based “last N” window. - prior:f201196 important 3 — fixed —
scripts/check-commit-author-attribution.mjs:288— A 250-entry pull-request commit response is marked truncated, causing the audit to fail closed instead of treating an incomplete history as clean.
Critical Issues (0)
Important Issues (0)
Suggestions (0)
Strengths
- The gate checks the exact PR range using full history and excludes only merge commits, matching the stated policy boundary.
- The audit distinguishes confirmed attribution violations from incomplete evidence and has focused regression tests for both cap conditions.
Recommended Action
- No blocking changes requested.
allyblockcast
left a comment
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: b22a366
Prior Findings Dispositioned (3)
- prior:f201196 important 1 — fixed —
.github/workflows/pr.yml:110— The attribution gate has no branch-name exemption, so fork branch naming cannot bypass it and merge-queue runs execute the same gate. - prior:f201196 important 2 — fixed —
scripts/check-commit-author-attribution.mjs:262— The audit now queriesmerged:>=<since>directly and reportsINCOMPLETEif the merge-time window reaches the 300-PR cap; it no longer claims a count-based “last N” window. - prior:f201196 important 3 — fixed —
scripts/check-commit-author-attribution.mjs:288— A 250-entry pull-request commit response is marked truncated, causing the audit to fail closed instead of treating an incomplete history as clean.
Critical Issues (0)
Important Issues (0)
Suggestions (0)
Strengths
- The gate checks the exact PR range using full history and excludes only merge commits, matching the stated policy boundary.
- The audit distinguishes confirmed attribution violations from incomplete evidence and has focused regression tests for both cap conditions.
Recommended Action
- No blocking changes requested.
Replacement for app-authored #1087 so the Ally GitHub App can provide the required independent review/approval.
Exact code head copied from #1087:
f20119687a35bb4a8c419ff6d223569d41c804b2.Thinking Path
Linked Issues or Issue Description
Fixes: BLO-21416
Refs: BLO-19528 (blocked by BLO-21416; unaffected by this PR's scope)
What Changed
AGENTS.md§9: new subsection documenting that commit attribution is write-path dependent, not agent dependent —git pushis already correct, the MCPcreate_or_update_file/push_filestools have noauthorfield and must not be used to land commits, andgh api(the one API call site agents control) must pass an explicitauthor[name]/author[email]when a raw API write is unavoidable.scripts/check-commit-author-attribution.mjs(+ test): shared assertionfindAttributionOffenses(flags a non-merge commit'scommit.author.email == "290875700+allyblockcast[bot]@users.noreply.github.com") with two front-ends:git log --no-mergesover abase..headrange — no network, no new secret.--audit-mergedmode: for the last N merged PRs of one or more repos (defaultBlockcast/trafficcontrol,Blockcast/paperclip), fetch each PR's own commit list viagh apiand apply the same assertion — the AC's "automated verifying signal.".github/workflows/pr.yml: wires the local mode into the existingpolicyjob as a new required step on every PR, using the branch's already-checked-out history (no new secret needed). No branch exemption — see review cycle 1 below.Review cycle 1 — the three Important findings (commit
5d7e2422b)1.
pr.ymlgraphify exemption (security). Ally flagged thatif: github.head_ref != 'bot/graphify-reindex'is forgeable —head_refcarries no repository identity, so any fork can name its branch that and skip the gate — and simultaneously wrong in the other direction, becausehead_refis empty inmerge_group, so a legitimate graphify PR would be rejected in the merge queue.I removed the exemption entirely rather than hardening it, because measurement shows it was guarding a rejection that never happens:
allyblockcast[bot]@users.noreply.github.comgit pushunder its own git identity290875700+allyblockcast[bot]@users.noreply.github.comThe gate's constant is the numeric App-user form; the bot's commits carry the bare form. Running the unmodified gate over the real graphify PRs #789 and #944 passes both. The exemption's stated premise ("that automation's PRs are legitimately App-authored end-to-end") was simply not true of the shipped bot.
Deleting it fixes both halves of the finding at once, adds gate coverage in the merge queue rather than special-casing it, and avoids restating
isGraphifyReindexArtifactOnlyPr's factors in a second place where they could drift. (That helper stays where it is, correctly, for the PR-template gate — a different concern: graphify PRs legitimately have no template body.) A regression test pins the bot's real identity, so if it ever moves onto the REST write path the test says so — and the fix then is to move the bot back, not to re-add an exemption.2. Audit window ordering. Superseded by review cycle 2 below — sorting alone was not enough.
3. Commit-list truncation.
GET /pulls/{n}/commitshard-caps at 250 entries even with--paginate, so an offense past commit 250 read as a clean pass. The audit now detects the cap and fails closed. It reports these asINCOMPLETE, distinct fromVIOLATION— an audit that could not finish is a different fact from one that finished clean, and conflating them would send someone hunting a commit the audit never saw.Review cycle 2 — the audit window, properly
Cycle 1 dispositioned findings 1 and 3 as fixed. Finding 2 was marked still-present, correctly: over-fetch + sort makes the ordering deterministic but still cannot establish the window, and the tool went on claiming "the last N merged PRs" — a claim its retrieval could not back.
Ally offered two acceptable resolutions: relabel the output as a bounded sample, or use a retrieval strategy that can prove the window. I took the second, because a count-based window is unprovable by construction (
gh pr listorders by creation, and no GitHub list or search ordering exposes merge time), so relabelling would have preserved a weak signal in a tool whose entire purpose is to be the verifying signal.The audit now selects by merge time:
--since(default7d, also acceptsYYYY-MM-DD) drives amerged:>=<date>search qualifier, and every PR merged in that window is audited.--per-repo-limitand the over-fetch factor are removed rather than kept alongside — there is one window concept, not two.Coverage it cannot prove is now reported rather than assumed: if a repo returns
AUDIT_PR_LIST_MAX(300) PRs the window did not fit, and the run fails closed withINCOMPLETE … Narrow --since, alongside the existing per-PR 250-commitINCOMPLETE. Each run prints the merge window it actually covered.This was not a cosmetic correction. Same repo, same 2-day period, measured both ways:
--per-repo-limit 5)--since 2d)The old window was missing the large majority of what it claimed to cover. Live output: 173 non-merge commits across 50 PRs merged
2026-08-07T06:42:23Z .. 2026-08-08T23:27:25Z, noINCOMPLETE, exit 1.Verification
node --test scripts/check-commit-author-attribution.test.mjs— 18/18 passing (was 8/8), including a real local git-range fixture, a merge-commit-exclusion fixture, and new cases covering both review cycles.merged:>=search back to a creation-ordered list fails exactly the merge-time-selection test. Each guard is confirmed to fire for its own reason, not incidentally.pr.ymlre-parsed withruby -ryamlafter editing (10 jobs), and the step asserted to still exist with noif:key. A workflow that does not parse produces no jobs and therefore cannot fail any gate, so this is checked explicitly rather than inferred from a green run.--audit-merged --repos Blockcast/paperclip --per-repo-limit 5exits 1 on a genuine in-the-wild violation (#1104bb6fefe, carrying the numeric App identity). That is a positive control worth stating plainly: on the same run, the bare graphify identity passes and the REST-path identity is caught, so the two really are distinct strings and not a distinction I inferred from the constant.--audit-mergedagainst real history as evidence:Blockcast/trafficcontrol#1326(last N=1, that PR only): reproduces the issue's documented baseline exactly — 4/9 commits App-attributed (78f62b7,9db630f,6b8b6e2,8108887).Blockcast/paperclip(last 10 merged PRs): 4 App-attributed non-merge commits found today (PR fix(github-webhook): close two Markdown escapes in owning-reference parsing (BLO-20886) #1051:df25973,bf823a4; PR fix(deps): bump undici to 7.29.0 #1018:0fa7f4d,6b818dd) — a live, current-state confirmation the gap is real, not just historical.git pushwith agent git config identity and confirmed viagh api repos/Blockcast/paperclip/commits/{sha}:commit.author.email = platformsreengineer@paperclip.blockcast.net, not the App — i.e., the documented correct path, demonstrated in the PR that documents it.Risks
actions/checkoutstep (fetch-depth: 0); no new secret or external call in the per-PR gate.--audit-mergedmode makes livegh apicalls (rate-limit exposure) but is not wired into any scheduled workflow yet — it's a manually-invoked audit tool for now, documented as the AC's verifying signal. Wiring it to a schedule would need a cross-repo-capable token (COMMITPERCLIP_KEYis unprovisioned on Blockcast) — flagged as a follow-up, not done here to avoid unilaterally provisioning new CI credentials.create_or_update_file/push_filestools still have noauthorfield — they're a third-party binary (github/github-mcp-server, pulled fromghcr.io, not vendored/patchable in this repo). The fix here is a hard ban-by-policy (AGENTS.md + PR gate) on using them for commits, not a patch to the tool itself.paperclip(the repo this session has write access to). Filed as a follow-up; the audit script already covers trafficcontrol read-only today via--audit-merged.if:removed, the gate now also runs onmerge_group. That is deliberate added coverage (nothing can land between PR CI and the queue without being checked), and it is safe precisely because the graphify identity passes the gate — verified above on chore(graphify): refresh knowledge graphs #789/chore(graphify): refresh knowledge graphs #944. The falsification path is cheap and immediate: chore(graphify): refresh knowledge graphs #944 is open right now, so once this lands, re-running its checks exercises the gate against a live graphify PR.allyblockcast[bot]address would break the graphify bot — which is exactly what the new regression test would tell you.Model Used
Claude Sonnet 5 (
claude-sonnet-5[1m]), Anthropic — original change: extended reasoning, tool use (Bash, Read/Write/Edit, GitHub MCP), 1M context window.Claude Opus 5 (
claude-opus-5[1m]), Anthropic — review cycle 1 (commit5d7e2422b): extended reasoning, tool use (Bash, Read/Edit,gh), 1M context window.Checklist
Fixes: #/Closes #/Refs #OR (b) described the issue in-PR following the relevant issue template