Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,3 +56,15 @@ updates:
applies-to: security-updates
patterns:
- "*"
ignore:
# Bumped by hand, not weekly by Dependabot. Each release of the Claude action
# bundles a fixed Claude Code version, and that CLI is what confines the
# automated reviewer (tool list, permission mode, deny rules). A bump needs a
# check of what changed: the action's diff between the two commits and the
# Claude Code changelog for the bundled versions in between.
# To bump: update the commit SHA and the version comment of the pin in
# .github/workflows/claude-review.yml, and the Claude Code version named in
# the comment above the pin. See .github/scripts/claude-review/README.md.
# Ignore rules also apply to Dependabot security updates, so no security PR
# is raised for this action either; watch its releases and advisories.
- dependency-name: "anthropics/claude-code-action"
89 changes: 75 additions & 14 deletions .github/scripts/claude-review/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ change the pinned versions.
| Job | Step | Script | Holds |
|-----|------|--------|-------|
| `review` | Fetch the pull request diff | `fetch-diff.sh` | read-only token |
| `review` | Review with Claude | (action) | Claude credential, read-only token |
| `review` | Review with Claude | (action) | Claude credential (from the `claude` environment), read-only token |
| `verdict` | Validate and render | `report.sh`, `validate.jq`, `render.jq` | nothing sensitive |
| `verdict` | Post or update the comment | `post-comment.sh` | token with `pull-requests: write` |
| `verdict` | Enforce the verdict | (inline) | nothing |
Expand Down Expand Up @@ -69,19 +69,69 @@ comment, and the runner image is `ubuntu-24.04` rather than `ubuntu-latest`.

The commit and its version live together on each `uses:` line, for example
`actions/checkout@<commit> # v7.0.1`, so there is no second list to keep in
step. Dependabot updates both parts. To bump one by hand, resolve the tag and
replace the commit and the comment wherever the action is used:
step. Dependabot updates both parts for every action except the Claude action,
which `.github/dependabot.yml` ignores so that it is bumped by hand. To bump one
by hand, resolve the tag and replace the commit and the comment wherever the
action is used:

```bash
gh api repos/OWNER/REPO/commits/TAG --jq .sha
```

The Claude action is not an ordinary bump. Each release hard-codes a CLI
version, and the CLI is what enforces the tool list, the permission mode and
the deny rules that keep the reviewer away from its own credential. The
reviewer itself reports a change of that pin as `high`, so such a pull request
does not get a green verdict on its own. Before merging one, re-check the
confinement described below.
the deny rules that keep the reviewer away from its own credential. Before
bumping it, check what changed: the action's diff between the two commits and
the Claude Code changelog for the bundled versions in between. Then update the
commit and the version comment in `claude-review.yml`, along with the Claude
Code version named in the comment above the pin. The reviewer itself reports a
change of that pin as `high`, so such a pull request does not get a green
verdict on its own. Before merging one, re-check the confinement described
below.

Checks done for past bumps of the Claude action:

- **v1.0.244 to v1.0.248** (Claude Code 2.1.292 to 2.1.296, checked
2026-10-22). The action's diff between the two commits is four commits that
only raise the bundled Claude Code and Agent SDK versions; 2.1.296 was read
from `src/entrypoints/run.ts` and `base-action/action.yml` at the new commit.
In the Claude Code changelog for 2.1.293 to 2.1.296, the one entry touching
the reviewer's limits is a 2.1.295 fix to `--tools` not applying to built-in
tools that register after launch, which tightens them. 2.1.296 adds an
`allow_large` option to `Read` for files over the usual size limit; it
concerns size, not which paths may be read. Nothing found widens the paths
the reviewer can read. Confinement on Linux remains unobserved, as below.

## Credential scope

`CLAUDE_CODE_OAUTH_TOKEN` is used by one job: `review` here. It runs in the
`claude` environment with `deployment: false`, which uses the environment for
its secret and its branch rule and creates no deployment records.

The environment is meant to be set up as follows (repository settings, not
code): deployment branches limited to the default branch, no required
reviewers, no wait timer, and the token stored there. A workflow defined on
any other branch then cannot read the token, because GitHub matches the branch
rule against the ref a run executes on, and `pull_request_target` executes on
the default branch (as would `issue_comment`, should a mention workflow be
added).

Two things to keep in mind:

- **The scope is only real once the repository-level secret of the same name
is deleted.** While it exists, any workflow on any branch can read it, and
this job silently falls back to it.
- If neither copy exists, the review ends as `credential_unavailable` (red). It
does not pass.
- Runs started by Dependabot have been seen to receive repository-level
Actions secrets here. That they also receive the environment's secret is
expected, since the same runs are treated as ordinary Actions runs, but it
can only be observed once the repository-level copy is gone: the first
review Dependabot starts after that either produces a verdict or ends as
`credential_unavailable`.

`test.sh` fails if any workflow uses the credential outside that environment or
on an event whose definition a pull request controls.

## Read confinement: what has and has not been checked

Expand All @@ -93,12 +143,23 @@ refused), and the deny list in `settings`.
**This was probed with a local CLI (2.1.284) on macOS only.** There, reads,
greps and globs outside the two locations were refused, the deny list held even
with reads deliberately pre-approved, and pre-approving `Read` was shown to
open every path. **It has not been exercised on a Linux runner**, and not with
the CLI version the pinned action installs (2.1.292). That matters because on
Linux the credential is also present in the reviewer's process environment
under `/proc`, which macOS does not have. `/proc` is covered both by the
working-directory limit and by an explicit deny rule, but neither has been
observed to hold there.
open every path.

The first runs on the default branch (from 2026-10-14, started by Dependabot
on its grouped pull requests and by a maintainer, `ubuntu-24.04`, CLI 2.1.292)
showed that the workflow works end to end on Linux with the pinned CLI: the
arguments were accepted, every run received the credential (`Secret source:
Actions`), valid verdicts came back and the comments were posted. **They say
nothing about confinement**: the reviewer made no attempt to read outside its
two locations (every run logged zero permission denials), so nothing was there
to be refused. The pin has since moved to a release that installs 2.1.296,
which has not run here yet.

**Confinement on a Linux runner therefore remains unobserved.** That matters
because on Linux the credential is also present in the reviewer's process
environment under `/proc`, which macOS does not have. `/proc` is covered both
by the working-directory limit and by an explicit deny rule, but neither has
been seen to hold there.

Re-check whenever the Claude action pin, `claude_args` or `settings` change: in
a private scratch repository with a throwaway credential, run the same action
Expand Down Expand Up @@ -135,7 +196,7 @@ tool list, schema, pins). Needs `jq`; the checks that parse YAML also need
`ruby`.

`.github/workflows/claude-review-scripts.yml` runs the same tests on a hosted
runner for every pull request that touches these files.
runner for every pull request that touches these files or any workflow file.

The review workflow itself runs only from the default branch, so a pull request
that changes it or these scripts is still reviewed by the version already
Expand Down
30 changes: 20 additions & 10 deletions .github/scripts/claude-review/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -440,8 +440,11 @@ if command -v ruby > /dev/null 2>&1; then
check "workflow: the verdict job tells report.sh whether Dependabot opened the pull request" is "$(wf '.jobs.verdict.steps[] | select(.id == "report") | .env.OPENED_BY_DEPENDABOT')" "\${{ github.event.pull_request.user.login == 'dependabot[bot]' }}"
check "workflow: nothing is uploaded" is "$(wf '[.jobs[].steps[] | .uses // empty | select(test("upload-artifact"))] | length')" 0
check "workflow: header no longer suggests reusing the credential as a Dependabot secret" test -z "$(grep -F 'makes this workflow work unchanged' "$workflow")"
check "workflow: header warns against pull-request-defined uses of the credential" grep -qF 'pull_request_review_comment' "$workflow"
check "workflow: header states that confinement is untested on Linux" grep -qF 'been exercised on a Linux runner' "$workflow"
check "workflow: header says the first run does not test the read confinement" grep -qF 'does not test the reviewer'"'"'s read confinement' "$workflow"
check "workflow: header says confinement on a Linux runner remains unobserved" grep -qF 'remains unobserved' "$workflow"
check "workflow: header says the scope depends on deleting the repository-level secret" grep -qF 'repository-level secret of the same name has been' "$workflow"
check "workflow: review job takes the credential from the claude environment, without deployments" is "$(wf '.jobs.review.environment | tojson')" '{"name":"claude","deployment":false}'
check "workflow: verdict job references no environment" is "$(wf '.jobs.verdict | has("environment")')" false

# The last step turns the verdict into the job conclusion: only "pass" exits 0.
enforce="$(wf '.jobs.verdict.steps[-1].run')"
Expand All @@ -454,17 +457,24 @@ if command -v ruby > /dev/null 2>&1; then
check "workflow: the verdict step is the last step and is not conditional" is "$(wf '.jobs.verdict.steps[-1] | [.name, has("if"), has("continue-on-error")] | join(",")')" 'Enforce the verdict,false,false'
check "workflow: only the comment step may fail without failing the job" is "$(wf '[.jobs[].steps[] | select(.["continue-on-error"] == true) | .name] | join(",")')" 'Post or update the pull request comment'

# No workflow defined by a pull request (plain pull_request trigger) may use the credential.
leak=""
# Every workflow that uses the Claude credential, whichever file it is in:
# * may only be started by events that run the default branch's definition
# (a pull request cannot redefine it), and
# * must take the credential, in every job that references it, from the
# `claude` environment, so that the environment's branch rule applies.
wrong_trigger=""; wrong_scope=""; users=""
for f in "$here"/../../workflows/*.yml "$here"/../../workflows/*.yaml; do
[ -f "$f" ] || continue
if ruby -ryaml -rjson -e 'puts JSON.generate(YAML.load_file(ARGV[0]))' "$f" 2> /dev/null |
jq -e '((.on // .["true"]) | if type == "object" then has("pull_request") elif type == "array" then index("pull_request") != null else . == "pull_request" end)
and (tostring | test("CLAUDE_CODE_OAUTH_TOKEN"))' > /dev/null 2>&1; then
leak="$leak $(basename "$f")"
fi
ruby -ryaml -rjson -e 'puts JSON.generate(YAML.load_file(ARGV[0]))' "$f" > "$T/any.json" 2> /dev/null || { wrong_scope="$wrong_scope $(basename "$f")(unparsed)"; continue; }
jq -e 'tostring | test("CLAUDE_CODE_OAUTH_TOKEN")' "$T/any.json" > /dev/null 2>&1 || continue
users="$users $(basename "$f")"
jq -e '((.on // .["true"]) | if type == "object" then keys elif type == "array" then . else [.] end)
- ["pull_request_target", "issue_comment"] == []' "$T/any.json" > /dev/null 2>&1 || wrong_trigger="$wrong_trigger $(basename "$f")"
jq -e '[.jobs[] | select(tostring | test("CLAUDE_CODE_OAUTH_TOKEN")) | .environment == {"name": "claude", "deployment": false}] | length > 0 and all' "$T/any.json" > /dev/null 2>&1 || wrong_scope="$wrong_scope $(basename "$f")"
done
check "workflows: none triggered by pull_request references the Claude credential" test -z "$leak"
check "workflows: the credential is used only by the review workflow" is "$users" ' claude-review.yml'
check "workflows: none that uses the credential can be redefined by a pull request" test -z "$wrong_trigger"
check "workflows: every job that uses the credential takes it from the claude environment" test -z "$wrong_scope"
elif [ "${REQUIRE_WORKFLOW_CHECKS:-0}" = "1" ]; then
bad "workflow: checks could not run (ruby not available)"
else
Expand Down
15 changes: 7 additions & 8 deletions .github/workflows/claude-review-scripts.yml
Original file line number Diff line number Diff line change
@@ -1,23 +1,22 @@
name: Claude Review Scripts

# Offline tests for the scripts behind claude-review.yml, run on the same
# runner image those scripts run on. Uses no secrets and a read-only token, so
# it is safe as an ordinary pull_request workflow. The review workflow itself
# cannot be exercised from a pull request: it always runs from the default
# branch.
# runner image those scripts run on. They also check every workflow file for
# how it uses the Claude credential, hence the path filter on all workflows.
# Uses no secrets and a read-only token, so it is safe as an ordinary
# pull_request workflow. The review workflow itself cannot be exercised from a
# pull request: it always runs from the default branch.

on:
pull_request:
paths:
- .github/scripts/claude-review/**
- .github/workflows/claude-review.yml
- .github/workflows/claude-review-scripts.yml
- .github/workflows/**
push:
branches: [main]
paths:
- .github/scripts/claude-review/**
- .github/workflows/claude-review.yml
- .github/workflows/claude-review-scripts.yml
- .github/workflows/**

permissions:
contents: read
Expand Down
Loading
Loading