From 58c8b071f67d9d597ba4a47494ac86a4fe9979c4 Mon Sep 17 00:00:00 2001 From: Brenton Strickler Date: Thu, 22 Oct 2026 11:47:42 -0400 Subject: [PATCH] ci: bring the Claude review to parity with indextables_spark Three changes that indextables_spark has made or is making to the same workflow, applied here so the two stay in step. - Bump anthropics/claude-code-action from v1.0.244 to v1.0.248 (installs Claude Code 2.1.296). The action's diff between the two commits only raises the bundled Claude Code and Agent SDK versions. In the Claude Code changelog for 2.1.293 to 2.1.296 the one entry touching the reviewer's limits is a 2.1.295 fix to --tools not applying to built-in tools that register after launch, which tightens them; nothing found widens the paths the reviewer can read. The README records this check. - Have Dependabot ignore anthropics/claude-code-action. Each release fixes the CLI that confines the reviewer, so a bump needs that check rather than a weekly grouped update. Ignore rules also stop security updates for it, which the comment says. The review instructions no longer say that dependabot.yml has no ignore list. - Take the review job's credential from a `claude` environment with `deployment: false`. Once the token is stored only there and the environment admits only the default branch, a workflow defined on any other branch cannot read it. Until then the job falls back to the repository-level secret, so nothing changes at merge; GitHub creates the environment, empty, on the first run that references it. test.sh now requires every workflow that uses the credential to start only on pull_request_target or issue_comment and to take it from that environment, and the scripts workflow runs on a change to any workflow. The workflow header and README record what this repository's first runs showed (the workflow runs on Linux and Dependabot-started runs receive the credential) and what they did not (the reviewer's read confinement). The scripts stay byte-identical to indextables_spark's; the test, README and workflow differ only where this repository has no mention workflow and its own history and dependencies. --- .github/dependabot.yml | 12 +++ .github/scripts/claude-review/README.md | 89 +++++++++++++++++---- .github/scripts/claude-review/test.sh | 30 ++++--- .github/workflows/claude-review-scripts.yml | 15 ++-- .github/workflows/claude-review.yml | 87 ++++++++++++-------- 5 files changed, 168 insertions(+), 65 deletions(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 500d40c..b295f0b 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -56,3 +56,15 @@ updates: applies-to: security-updates patterns: - "*" + ignore: + # Bumped by hand, not weekly by Dependabot. Each release of the Claude action + # bundles a fixed Claude Code version, and that CLI is what confines the + # automated reviewer (tool list, permission mode, deny rules). A bump needs a + # check of what changed: the action's diff between the two commits and the + # Claude Code changelog for the bundled versions in between. + # To bump: update the commit SHA and the version comment of the pin in + # .github/workflows/claude-review.yml, and the Claude Code version named in + # the comment above the pin. See .github/scripts/claude-review/README.md. + # Ignore rules also apply to Dependabot security updates, so no security PR + # is raised for this action either; watch its releases and advisories. + - dependency-name: "anthropics/claude-code-action" diff --git a/.github/scripts/claude-review/README.md b/.github/scripts/claude-review/README.md index c51a945..a5955b4 100644 --- a/.github/scripts/claude-review/README.md +++ b/.github/scripts/claude-review/README.md @@ -9,7 +9,7 @@ change the pinned versions. | Job | Step | Script | Holds | |-----|------|--------|-------| | `review` | Fetch the pull request diff | `fetch-diff.sh` | read-only token | -| `review` | Review with Claude | (action) | Claude credential, read-only token | +| `review` | Review with Claude | (action) | Claude credential (from the `claude` environment), read-only token | | `verdict` | Validate and render | `report.sh`, `validate.jq`, `render.jq` | nothing sensitive | | `verdict` | Post or update the comment | `post-comment.sh` | token with `pull-requests: write` | | `verdict` | Enforce the verdict | (inline) | nothing | @@ -69,8 +69,10 @@ comment, and the runner image is `ubuntu-24.04` rather than `ubuntu-latest`. The commit and its version live together on each `uses:` line, for example `actions/checkout@ # v7.0.1`, so there is no second list to keep in -step. Dependabot updates both parts. To bump one by hand, resolve the tag and -replace the commit and the comment wherever the action is used: +step. Dependabot updates both parts for every action except the Claude action, +which `.github/dependabot.yml` ignores so that it is bumped by hand. To bump one +by hand, resolve the tag and replace the commit and the comment wherever the +action is used: ```bash gh api repos/OWNER/REPO/commits/TAG --jq .sha @@ -78,10 +80,58 @@ gh api repos/OWNER/REPO/commits/TAG --jq .sha The Claude action is not an ordinary bump. Each release hard-codes a CLI version, and the CLI is what enforces the tool list, the permission mode and -the deny rules that keep the reviewer away from its own credential. The -reviewer itself reports a change of that pin as `high`, so such a pull request -does not get a green verdict on its own. Before merging one, re-check the -confinement described below. +the deny rules that keep the reviewer away from its own credential. Before +bumping it, check what changed: the action's diff between the two commits and +the Claude Code changelog for the bundled versions in between. Then update the +commit and the version comment in `claude-review.yml`, along with the Claude +Code version named in the comment above the pin. The reviewer itself reports a +change of that pin as `high`, so such a pull request does not get a green +verdict on its own. Before merging one, re-check the confinement described +below. + +Checks done for past bumps of the Claude action: + +- **v1.0.244 to v1.0.248** (Claude Code 2.1.292 to 2.1.296, checked + 2026-10-22). The action's diff between the two commits is four commits that + only raise the bundled Claude Code and Agent SDK versions; 2.1.296 was read + from `src/entrypoints/run.ts` and `base-action/action.yml` at the new commit. + In the Claude Code changelog for 2.1.293 to 2.1.296, the one entry touching + the reviewer's limits is a 2.1.295 fix to `--tools` not applying to built-in + tools that register after launch, which tightens them. 2.1.296 adds an + `allow_large` option to `Read` for files over the usual size limit; it + concerns size, not which paths may be read. Nothing found widens the paths + the reviewer can read. Confinement on Linux remains unobserved, as below. + +## Credential scope + +`CLAUDE_CODE_OAUTH_TOKEN` is used by one job: `review` here. It runs in the +`claude` environment with `deployment: false`, which uses the environment for +its secret and its branch rule and creates no deployment records. + +The environment is meant to be set up as follows (repository settings, not +code): deployment branches limited to the default branch, no required +reviewers, no wait timer, and the token stored there. A workflow defined on +any other branch then cannot read the token, because GitHub matches the branch +rule against the ref a run executes on, and `pull_request_target` executes on +the default branch (as would `issue_comment`, should a mention workflow be +added). + +Two things to keep in mind: + +- **The scope is only real once the repository-level secret of the same name + is deleted.** While it exists, any workflow on any branch can read it, and + this job silently falls back to it. +- If neither copy exists, the review ends as `credential_unavailable` (red). It + does not pass. +- Runs started by Dependabot have been seen to receive repository-level + Actions secrets here. That they also receive the environment's secret is + expected, since the same runs are treated as ordinary Actions runs, but it + can only be observed once the repository-level copy is gone: the first + review Dependabot starts after that either produces a verdict or ends as + `credential_unavailable`. + +`test.sh` fails if any workflow uses the credential outside that environment or +on an event whose definition a pull request controls. ## Read confinement: what has and has not been checked @@ -93,12 +143,23 @@ refused), and the deny list in `settings`. **This was probed with a local CLI (2.1.284) on macOS only.** There, reads, greps and globs outside the two locations were refused, the deny list held even with reads deliberately pre-approved, and pre-approving `Read` was shown to -open every path. **It has not been exercised on a Linux runner**, and not with -the CLI version the pinned action installs (2.1.292). That matters because on -Linux the credential is also present in the reviewer's process environment -under `/proc`, which macOS does not have. `/proc` is covered both by the -working-directory limit and by an explicit deny rule, but neither has been -observed to hold there. +open every path. + +The first runs on the default branch (from 2026-10-14, started by Dependabot +on its grouped pull requests and by a maintainer, `ubuntu-24.04`, CLI 2.1.292) +showed that the workflow works end to end on Linux with the pinned CLI: the +arguments were accepted, every run received the credential (`Secret source: +Actions`), valid verdicts came back and the comments were posted. **They say +nothing about confinement**: the reviewer made no attempt to read outside its +two locations (every run logged zero permission denials), so nothing was there +to be refused. The pin has since moved to a release that installs 2.1.296, +which has not run here yet. + +**Confinement on a Linux runner therefore remains unobserved.** That matters +because on Linux the credential is also present in the reviewer's process +environment under `/proc`, which macOS does not have. `/proc` is covered both +by the working-directory limit and by an explicit deny rule, but neither has +been seen to hold there. Re-check whenever the Claude action pin, `claude_args` or `settings` change: in a private scratch repository with a throwaway credential, run the same action @@ -135,7 +196,7 @@ tool list, schema, pins). Needs `jq`; the checks that parse YAML also need `ruby`. `.github/workflows/claude-review-scripts.yml` runs the same tests on a hosted -runner for every pull request that touches these files. +runner for every pull request that touches these files or any workflow file. The review workflow itself runs only from the default branch, so a pull request that changes it or these scripts is still reviewed by the version already diff --git a/.github/scripts/claude-review/test.sh b/.github/scripts/claude-review/test.sh index 8ad25f1..5c792e2 100644 --- a/.github/scripts/claude-review/test.sh +++ b/.github/scripts/claude-review/test.sh @@ -440,8 +440,11 @@ if command -v ruby > /dev/null 2>&1; then check "workflow: the verdict job tells report.sh whether Dependabot opened the pull request" is "$(wf '.jobs.verdict.steps[] | select(.id == "report") | .env.OPENED_BY_DEPENDABOT')" "\${{ github.event.pull_request.user.login == 'dependabot[bot]' }}" check "workflow: nothing is uploaded" is "$(wf '[.jobs[].steps[] | .uses // empty | select(test("upload-artifact"))] | length')" 0 check "workflow: header no longer suggests reusing the credential as a Dependabot secret" test -z "$(grep -F 'makes this workflow work unchanged' "$workflow")" - check "workflow: header warns against pull-request-defined uses of the credential" grep -qF 'pull_request_review_comment' "$workflow" - check "workflow: header states that confinement is untested on Linux" grep -qF 'been exercised on a Linux runner' "$workflow" + check "workflow: header says the first run does not test the read confinement" grep -qF 'does not test the reviewer'"'"'s read confinement' "$workflow" + check "workflow: header says confinement on a Linux runner remains unobserved" grep -qF 'remains unobserved' "$workflow" + check "workflow: header says the scope depends on deleting the repository-level secret" grep -qF 'repository-level secret of the same name has been' "$workflow" + check "workflow: review job takes the credential from the claude environment, without deployments" is "$(wf '.jobs.review.environment | tojson')" '{"name":"claude","deployment":false}' + check "workflow: verdict job references no environment" is "$(wf '.jobs.verdict | has("environment")')" false # The last step turns the verdict into the job conclusion: only "pass" exits 0. enforce="$(wf '.jobs.verdict.steps[-1].run')" @@ -454,17 +457,24 @@ if command -v ruby > /dev/null 2>&1; then check "workflow: the verdict step is the last step and is not conditional" is "$(wf '.jobs.verdict.steps[-1] | [.name, has("if"), has("continue-on-error")] | join(",")')" 'Enforce the verdict,false,false' check "workflow: only the comment step may fail without failing the job" is "$(wf '[.jobs[].steps[] | select(.["continue-on-error"] == true) | .name] | join(",")')" 'Post or update the pull request comment' - # No workflow defined by a pull request (plain pull_request trigger) may use the credential. - leak="" + # Every workflow that uses the Claude credential, whichever file it is in: + # * may only be started by events that run the default branch's definition + # (a pull request cannot redefine it), and + # * must take the credential, in every job that references it, from the + # `claude` environment, so that the environment's branch rule applies. + wrong_trigger=""; wrong_scope=""; users="" for f in "$here"/../../workflows/*.yml "$here"/../../workflows/*.yaml; do [ -f "$f" ] || continue - if ruby -ryaml -rjson -e 'puts JSON.generate(YAML.load_file(ARGV[0]))' "$f" 2> /dev/null | - jq -e '((.on // .["true"]) | if type == "object" then has("pull_request") elif type == "array" then index("pull_request") != null else . == "pull_request" end) - and (tostring | test("CLAUDE_CODE_OAUTH_TOKEN"))' > /dev/null 2>&1; then - leak="$leak $(basename "$f")" - fi + ruby -ryaml -rjson -e 'puts JSON.generate(YAML.load_file(ARGV[0]))' "$f" > "$T/any.json" 2> /dev/null || { wrong_scope="$wrong_scope $(basename "$f")(unparsed)"; continue; } + jq -e 'tostring | test("CLAUDE_CODE_OAUTH_TOKEN")' "$T/any.json" > /dev/null 2>&1 || continue + users="$users $(basename "$f")" + jq -e '((.on // .["true"]) | if type == "object" then keys elif type == "array" then . else [.] end) + - ["pull_request_target", "issue_comment"] == []' "$T/any.json" > /dev/null 2>&1 || wrong_trigger="$wrong_trigger $(basename "$f")" + jq -e '[.jobs[] | select(tostring | test("CLAUDE_CODE_OAUTH_TOKEN")) | .environment == {"name": "claude", "deployment": false}] | length > 0 and all' "$T/any.json" > /dev/null 2>&1 || wrong_scope="$wrong_scope $(basename "$f")" done - check "workflows: none triggered by pull_request references the Claude credential" test -z "$leak" + check "workflows: the credential is used only by the review workflow" is "$users" ' claude-review.yml' + check "workflows: none that uses the credential can be redefined by a pull request" test -z "$wrong_trigger" + check "workflows: every job that uses the credential takes it from the claude environment" test -z "$wrong_scope" elif [ "${REQUIRE_WORKFLOW_CHECKS:-0}" = "1" ]; then bad "workflow: checks could not run (ruby not available)" else diff --git a/.github/workflows/claude-review-scripts.yml b/.github/workflows/claude-review-scripts.yml index d607695..3f440c1 100644 --- a/.github/workflows/claude-review-scripts.yml +++ b/.github/workflows/claude-review-scripts.yml @@ -1,23 +1,22 @@ name: Claude Review Scripts # Offline tests for the scripts behind claude-review.yml, run on the same -# runner image those scripts run on. Uses no secrets and a read-only token, so -# it is safe as an ordinary pull_request workflow. The review workflow itself -# cannot be exercised from a pull request: it always runs from the default -# branch. +# runner image those scripts run on. They also check every workflow file for +# how it uses the Claude credential, hence the path filter on all workflows. +# Uses no secrets and a read-only token, so it is safe as an ordinary +# pull_request workflow. The review workflow itself cannot be exercised from a +# pull request: it always runs from the default branch. on: pull_request: paths: - .github/scripts/claude-review/** - - .github/workflows/claude-review.yml - - .github/workflows/claude-review-scripts.yml + - .github/workflows/** push: branches: [main] paths: - .github/scripts/claude-review/** - - .github/workflows/claude-review.yml - - .github/workflows/claude-review-scripts.yml + - .github/workflows/** permissions: contents: read diff --git a/.github/workflows/claude-review.yml b/.github/workflows/claude-review.yml index eaab218..fd5bdc2 100644 --- a/.github/workflows/claude-review.yml +++ b/.github/workflows/claude-review.yml @@ -13,7 +13,8 @@ name: Claude Code Review # Title, description, commit messages and release notes are not passed on. # * The `review` job holds the Claude credential and a read-only token. The # reviewer has file-reading tools only: no shell, no network, no GitHub -# write access. +# write access. The job takes the credential from the `claude` environment +# (see "Credential scope" below). # * The `verdict` job holds a token that can comment on pull requests and no # Claude credential. It treats the reviewer's output as untrusted data: # strict validation, then rendering that cannot produce mentions, links or @@ -26,38 +27,53 @@ name: Claude Code Review # permission mode and the deny rules below are enforced. See the README next # to the scripts before changing a pin. # -# Not yet verified on a runner -# ---------------------------- -# The reviewer's read confinement (no --allowedTools, the permission mode, the -# settings deny list) was probed with a local CLI on macOS only. It has not -# been exercised on a Linux runner, where the credential is also present in -# the reviewer's process environment under /proc. Re-check it whenever the -# Claude action pin, claude_args or settings change. +# Observed on a runner, and not +# ----------------------------- +# The first runs on the default branch (from 2026-10-14, started by Dependabot +# on its grouped pull requests and by a maintainer, ubuntu-24.04, Claude Code +# 2.1.292) showed that the workflow runs end to end on Linux with the pinned +# CLI: the arguments were accepted, the credential reached the job, +# schema-valid verdicts came back and the comments were posted. The pin has +# since moved to a release that installs Claude Code 2.1.296. # -# Other uses of the credential -# ---------------------------- -# This is the only workflow in this repository that uses the credential. Do -# not add one that uses it on an event whose definition comes from the pull -# request (`pull_request`, `pull_request_review`, -# `pull_request_review_comment`): for a branch in this repository that -# definition, and its use of the credential, would be under the pull request's -# control. The credential is a repository-level secret, so a workflow pushed -# to any branch by someone with write access can still read it; keeping it in -# an environment restricted to the default branch would close that. +# What they show does not test the reviewer's read confinement (no +# --allowedTools, the permission mode, the settings deny list): the reviewer +# attempted nothing outside its two locations, so nothing was there to be +# refused. Confinement has been probed with a local CLI on macOS only. Whether +# it holds on a Linux runner, where the credential is also present in the +# reviewer's process environment under /proc, remains unobserved. Re-check it +# whenever the Claude action pin, claude_args or settings change. # -# The workflow this one replaces (claude-code-review.yml) also started Claude +# Credential scope +# ---------------- +# The `review` job, the only job in this repository that uses the credential, +# runs in the `claude` environment. When CLAUDE_CODE_OAUTH_TOKEN is stored in +# that environment only, and the environment's deployment-branch rule admits +# only the default branch, a workflow defined on any other branch cannot read +# it: the rule is matched against the ref a run executes on, which for +# pull_request_target is the default branch. `deployment: false` uses the +# environment for its secret and branch rule without creating deployment +# records. The environment must have no required reviewers and no wait timer. +# +# This holds only once the repository-level secret of the same name has been +# deleted. While that copy exists, every workflow in the repository can still +# read it, and this job falls back to it. Any other workflow that needs the +# credential must use the same environment and an event that runs the default +# branch's definition; test.sh checks both. +# +# The workflow this one replaced (claude-code-review.yml) also started Claude # on issue and pull request comments that mention @claude. This one does not. # # Dependabot # ---------- -# Runs started by Dependabot are subject to GitHub's restrictions on secrets. -# If such a run does not receive CLAUDE_CODE_OAUTH_TOKEN, it ends as "did not -# complete (credential_unavailable)"; the "Set up job" log of the run shows -# `Secret source:`. Should that turn out to be the case, the options are a -# dedicated, spend-capped key stored as a Dependabot secret, or a different -# trigger design. Do not put a personal subscription token into Dependabot -# secrets: they are handed to every workflow run that Dependabot triggers, -# including `pull_request` runs whose definition comes from the branch. +# The runs above that Dependabot started received Actions secrets ("Secret +# source: Actions" in their "Set up job" logs), so no Dependabot secret is +# needed. If a later run does not receive the credential, it ends as "did not +# complete (credential_unavailable)". The options then are a dedicated, +# spend-capped key stored as a Dependabot secret, or a different trigger +# design. Do not put a personal subscription token into Dependabot secrets: +# they are handed to every workflow run that Dependabot triggers, including +# `pull_request` runs whose definition comes from the branch. # # Scripts and their offline tests: .github/scripts/claude-review/ @@ -94,6 +110,10 @@ jobs: contains(fromJSON('["dependabot[bot]", "rbrenton", "schenksj", "chrisfauerbach"]'), github.event.pull_request.user.login) runs-on: ubuntu-24.04 timeout-minutes: 30 + # Source of CLAUDE_CODE_OAUTH_TOKEN; see "Credential scope" in the header. + environment: + name: claude + deployment: false permissions: contents: read # check out the default branch pull-requests: read # read the pull request and its diff @@ -126,10 +146,10 @@ jobs: # A step timeout fails the step, so the verdict job still runs and # reports "did not complete". timeout-minutes: 20 - # This release installs Claude Code 2.1.292, which is what enforces + # This release installs Claude Code 2.1.296, which is what enforces # the tool and path limits below. Re-check them on every change of # this pin (see the README next to the scripts). - uses: anthropics/claude-code-action@58985842b834ed26087302ba27d07bc24ca8697a # v1.0.244 + uses: anthropics/claude-code-action@1d6de8cb0c237e7c15e9e1bdf973826ebae490cc # v1.0.248 with: claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} # The job's own read-only token. Passing it also keeps the action @@ -236,10 +256,11 @@ jobs: or action whose version changes, work out the old and new version from the diff, then: - - .github/dependabot.yml has no ignore list here. What is held on - purpose is visible in native/Cargo.toml instead; read the comments - next to the changed line. A change to any of the following is `high`, - even when the step is small: + - .github/dependabot.yml ignores only anthropics/claude-code-action, + which is bumped by hand (see the next rules). For dependencies, what + is held on purpose is visible in native/Cargo.toml instead; read the + comments next to the changed line. A change to any of the following + is `high`, even when the step is small: - the `git` and `rev` of the tantivy and quickwit crates, which come from the project's own forks at a fixed revision, including tantivy-sstable, tantivy-common and the [patch] section. These