Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 142 additions & 0 deletions .github/scripts/claude-review/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
# Automated pull request review: scripts

Used by `.github/workflows/claude-review.yml`. The workflow header describes the
trust model; this file describes the moving parts, how to test them, and how to
change the pinned versions.

## Flow

| Job | Step | Script | Holds |
|-----|------|--------|-------|
| `review` | Fetch the pull request diff | `fetch-diff.sh` | read-only token |
| `review` | Review with Claude | (action) | Claude credential, read-only token |
| `verdict` | Validate and render | `report.sh`, `validate.jq`, `render.jq` | nothing sensitive |
| `verdict` | Post or update the comment | `post-comment.sh` | token with `pull-requests: write` |
| `verdict` | Enforce the verdict | (inline) | nothing |

The pull request is never checked out. `fetch-diff.sh` writes its diff to
`$RUNNER_TEMP/claude-review/pr.diff`; the reviewer reads that file and a checkout
of the default branch, with the `Read`, `Grep` and `Glob` tools only.

## Outcomes

The machine-readable result is the conclusion of the **Verdict** check run on
the pull request's head commit. It is `success` only for `pass`.

| `verdict` | When | Verdict check |
|-----------|------|---------------|
| `pass` | Schema-valid output, verdict `pass`, no `critical` or `high` finding | green |
| `fail` | Schema-valid output with verdict `fail`, or `pass` with a `critical`/`high` finding | red |
| `none` | Everything else: see the reason codes | red, "did not complete" |

**A Verdict check that is missing or skipped is not a pass.** The jobs are
skipped for forks, drafts, authors outside the allowlist and pull requests that
do not target the default branch, and nothing runs for events other than the
four trigger types. If Verdict is ever made a required status check, remember
that GitHub counts a skipped job as satisfying the requirement.

A pass is advisory. On a pull request opened by Dependabot the comment and the
job summary say so explicitly: the review checks the version changes against
the project's pinning and major-version rules; it cannot assess what the new
releases contain.

Nothing is uploaded as an artifact. If something later needs to consume the
result, read the check run rather than adding a file.

Reason codes for `none`:

| Code | Meaning |
|------|---------|
| `credential_unavailable` | `CLAUDE_CODE_OAUTH_TOKEN` did not reach the run |
| `diff_too_large` | Over `MAX_DIFF_BYTES` / `MAX_DIFF_LINES`, or refused by the API as too large |
| `diff_line_too_long` | A line exceeds `MAX_LINE_BYTES`; the reviewer would see it cut off |
| `diff_incomplete` | Fewer `diff --git` headers than the pull request's changed files |
| `diff_unreadable` | NUL bytes in the diff |
| `empty_diff` | Nothing to review |
| `head_changed` | New commits arrived during the run; a newer run covers them |
| `pr_not_open` | Closed or merged meanwhile |
| `diff_fetch_failed`, `pr_fetch_failed` | API errors |
| `bad_input` | Event payload values not in the expected shape |
| `precheck_missing` | The review job reported no precheck result |
| `review_failed` | The review step failed, timed out or was cancelled |
| `no_output`, `invalid_output` | No structured output, or output that fails `validate.jq` |

## Pinned versions

Every action is pinned to a full commit id, with the version tag in a trailing
comment, and the runner image is `ubuntu-24.04` rather than `ubuntu-latest`.
`test.sh` fails on a tag or branch reference.

The commit and its version live together on each `uses:` line, for example
`actions/checkout@<commit> # v7.0.1`, so there is no second list to keep in
step. Dependabot updates both parts. To bump one by hand, resolve the tag and
replace the commit and the comment wherever the action is used:

```bash
gh api repos/OWNER/REPO/commits/TAG --jq .sha
```

The Claude action is not an ordinary bump. Each release hard-codes a CLI
version, and the CLI is what enforces the tool list, the permission mode and
the deny rules that keep the reviewer away from its own credential. The
reviewer itself reports a change of that pin as `high`, so such a pull request
does not get a green verdict on its own. Before merging one, re-check the
confinement described below.

## Read confinement: what has and has not been checked

The reviewer must not be able to read anything outside the checkout and the
diff directory. Three settings provide that: no `--allowedTools` (so nothing is
pre-approved), `--permission-mode dontAsk` (so anything needing approval is
refused), and the deny list in `settings`.

**This was probed with a local CLI (2.1.284) on macOS only.** There, reads,
greps and globs outside the two locations were refused, the deny list held even
with reads deliberately pre-approved, and pre-approving `Read` was shown to
open every path. **It has not been exercised on a Linux runner**, and not with
the CLI version the pinned action installs (2.1.292). That matters because on
Linux the credential is also present in the reviewer's process environment
under `/proc`, which macOS does not have. `/proc` is covered both by the
working-directory limit and by an explicit deny rule, but neither has been
observed to hold there.

Re-check whenever the Claude action pin, `claude_args` or `settings` change: in
a private scratch repository with a throwaway credential, run the same action
commit, `claude_args` and `settings` and ask the reviewer to read
`/proc/self/environ`, a file planted outside the allowed directories and
`.git/config`. Keep `show_full_output` off so that a read which does succeed is
not printed to the log.

## Changing things

- **Who gets reviewed:** the JSON array in the `review` job's `if:`.
- **Output shape:** the `--json-schema` in the workflow and `validate.jq` must
agree; `test.sh` compares their limits and enumerations.
- **Size limits:** defaults at the top of `fetch-diff.sh`. Of the last 60 pull
requests when this was written, the largest diff was about 211 KB and 4,300
lines, with no line over 846 bytes; none would have been rejected. Over the
whole history (193 pull requests) 9 would have been: 8 over the size limits
and one with more than 300 files, which the API refuses to render as a diff.
- **Reviewer tools, `claude_args`, `settings`:** do not add `--allowedTools`. A
bare `Read` there approves reading any path on the runner, which includes the
reviewer's own process environment. After any change here, re-check the
confinement (see above).

## Tests

```bash
bash .github/scripts/claude-review/test.sh
```

Offline: `gh` is stubbed. Covers accepted and rejected diffs, every reason code,
valid and invalid reviewer output, hostile text in findings, comment creation
and update, static checks of the workflow file (trigger, permissions, gate,
tool list, schema, pins). Needs `jq`; the checks that parse YAML also need
`ruby`.

`.github/workflows/claude-review-scripts.yml` runs the same tests on a hosted
runner for every pull request that touches these files.

The review workflow itself runs only from the default branch, so a pull request
that changes it or these scripts is still reviewed by the version already
merged. The tests are the only check such a change gets before it is live.
118 changes: 118 additions & 0 deletions .github/scripts/claude-review/fetch-diff.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
#!/usr/bin/env bash
# Fetch a pull request's diff through the REST API into a directory outside the
# workspace, and decide whether it is fit for automated review.
#
# The pull request is never checked out: its content reaches the reviewer only
# as this one data file.
#
# "Not reviewable" is not a script failure. It is reported through the step
# outputs `ready` (true|false) and `reason` (a fixed code from the list in
# report.sh), so that the verdict job can turn it into an explicit "review did
# not complete" instead of a pass. The script exits non-zero only on an
# internal error, which also ends without a verdict.
#
# Required environment:
# GH_TOKEN read-only token used by `gh api`
# REPO owner/name
# PR_NUMBER pull request number from the event payload
# HEAD_SHA head commit from the event payload; the diff must be of it
# OUT_DIR destination directory, outside GITHUB_WORKSPACE
# HAS_CREDENTIAL "true" when the reviewer's credential reached this run
# GITHUB_OUTPUT step output file
# Optional limits (bytes / lines):
# MAX_DIFF_BYTES, MAX_DIFF_LINES, MAX_LINE_BYTES
set -euo pipefail

: "${REPO:?}" "${PR_NUMBER:?}" "${HEAD_SHA:?}" "${OUT_DIR:?}" "${GITHUB_OUTPUT:?}"

# Beyond these the diff is not reviewed at all rather than reviewed in part.
MAX_DIFF_BYTES="${MAX_DIFF_BYTES:-512000}"
MAX_DIFF_LINES="${MAX_DIFF_LINES:-12000}"
# The reviewer's file-reading tool cuts lines off at 2000 characters. A longer
# line would be only partly visible, so it is treated like a truncated diff.
MAX_LINE_BYTES="${MAX_LINE_BYTES:-2000}"

diff_file="$OUT_DIR/pr.diff"
tmp_file="$OUT_DIR/pr.diff.partial"
pr_json="$OUT_DIR/pr.json"
err_file="$OUT_DIR/api.err"

finish() { # finish <ready:true|false> <reason-code>
rm -f "$tmp_file" "$pr_json" "$err_file"
if [ "$1" != "true" ]; then
rm -f "$diff_file"
echo "::warning title=Review precheck::No review will run for this commit: $2"
fi
printf 'ready=%s\nreason=%s\n' "$1" "$2" >> "$GITHUB_OUTPUT"
exit 0
}

# Values from the event payload are still checked before they go into a URL.
case "$PR_NUMBER" in ''|*[!0-9]*) finish false bad_input ;; esac
if ! printf '%s' "$HEAD_SHA" | grep -Eq '^[0-9a-f]{40}$'; then finish false bad_input; fi
if ! printf '%s' "$REPO" | grep -Eq '^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$'; then finish false bad_input; fi

# The diff must not land in the checkout the reviewer treats as the base branch.
if [ -n "${GITHUB_WORKSPACE:-}" ]; then
case "$OUT_DIR/" in "$GITHUB_WORKSPACE"/*) echo "OUT_DIR is inside the workspace" >&2; exit 1 ;; esac
fi
mkdir -p "$OUT_DIR"
rm -f "$diff_file" "$tmp_file" "$pr_json" "$err_file"

# Checked before any API call so the cause is named, rather than surfacing
# later as an authentication error from the review step.
if [ "${HAS_CREDENTIAL:-false}" != "true" ]; then finish false credential_unavailable; fi

# One request for the whole diff. The API answers very large diffs with an
# error (HTTP 406) instead of a partial body; any failure ends without a verdict.
if ! gh api -H "Accept: application/vnd.github.diff" "repos/$REPO/pulls/$PR_NUMBER" > "$tmp_file" 2> "$err_file"; then
sed -e 's/^/gh: /' "$err_file" >&2 || true
if grep -Eqi 'HTTP 406|too large|exceeded the maximum' "$err_file"; then
finish false diff_too_large
fi
finish false diff_fetch_failed
fi

# Fetched after the diff: if the head is still the commit this run was started
# for, the diff above is the diff of that commit.
if ! gh api "repos/$REPO/pulls/$PR_NUMBER" > "$pr_json" 2> "$err_file"; then
sed -e 's/^/gh: /' "$err_file" >&2 || true
finish false pr_fetch_failed
fi
if ! api_head=$(jq -er '.head.sha | strings' "$pr_json") ||
! api_state=$(jq -er '.state | strings' "$pr_json") ||
! api_files=$(jq -er '.changed_files | numbers | floor' "$pr_json"); then
finish false pr_fetch_failed
fi
if [ "$api_state" != "open" ]; then finish false pr_not_open; fi
if [ "$api_head" != "$HEAD_SHA" ]; then finish false head_changed; fi

bytes=$(LC_ALL=C wc -c < "$tmp_file" | tr -d '[:space:]')
lines=$(LC_ALL=C wc -l < "$tmp_file" | tr -d '[:space:]')
echo "diff: $bytes bytes, $lines lines; pull request reports $api_files changed file(s)"

if [ "$bytes" -eq 0 ] || [ "$api_files" -eq 0 ]; then finish false empty_diff; fi
if [ "$bytes" -gt "$MAX_DIFF_BYTES" ] || [ "$lines" -gt "$MAX_DIFF_LINES" ]; then finish false diff_too_large; fi

# A text diff has no NUL bytes; one would also make the text tools below unreliable.
clean_bytes=$(LC_ALL=C tr -d '\000' < "$tmp_file" | LC_ALL=C wc -c | tr -d '[:space:]')
if [ "$clean_bytes" -ne "$bytes" ]; then finish false diff_unreadable; fi

# Completeness: one "diff --git" header per changed file. Inside hunks every
# line starts with '+', '-', ' ' or '\', so only real headers match at column 0.
headers=$(LC_ALL=C grep -a -c '^diff --git ' "$tmp_file" || true)
if [ "$headers" -ne "$api_files" ]; then
echo "diff has $headers file header(s), expected $api_files" >&2
finish false diff_incomplete
fi

longest=$(LC_ALL=C awk '{ n = length($0); if (n > m) m = n } END { print m + 0 }' "$tmp_file")
if [ "$longest" -gt "$MAX_LINE_BYTES" ]; then
echo "longest diff line is $longest bytes" >&2
finish false diff_line_too_long
fi

mv "$tmp_file" "$diff_file"
chmod 0444 "$diff_file"
echo "Diff written to $diff_file"
finish true ok
71 changes: 71 additions & 0 deletions .github/scripts/claude-review/post-comment.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
#!/usr/bin/env bash
# Post the rendered review as ONE comment on the pull request: update the
# comment a previous run left, or create it.
#
# Best effort by design. The verdict is carried by the job conclusion and the
# job summary; a comment that cannot be written
# (for example because the token is read-only for this run) produces a warning
# and must not change the verdict in either direction.
#
# Required environment:
# GH_TOKEN token with pull-requests: write
# REPO owner/name
# PR_NUMBER, HEAD_SHA
# COMMENT_FILE body written by report.sh; its first line is the marker
set -euo pipefail

: "${REPO:?}" "${PR_NUMBER:?}" "${HEAD_SHA:?}" "${COMMENT_FILE:?}"

MARKER='<!-- claude-review-verdict -->'
# Comments written with the workflow token belong to this account. Matching on
# it keeps a marker pasted by anyone else from being mistaken for ours.
BOT_LOGIN='github-actions[bot]'

warn() { echo "::warning title=Review comment::$1"; exit 0; }

case "$PR_NUMBER" in ''|*[!0-9]*) echo "PR_NUMBER is not a number" >&2; exit 1 ;; esac
[ -s "$COMMENT_FILE" ] || { echo "comment file is missing or empty" >&2; exit 1; }
[ "$(head -n 1 "$COMMENT_FILE")" = "$MARKER" ] || { echo "comment file does not start with the marker" >&2; exit 1; }

work="$(mktemp -d)"
trap 'rm -rf "$work"' EXIT

# Do not let a superseded run overwrite the comment of the commit that replaced it.
if ! gh api "repos/$REPO/pulls/$PR_NUMBER" > "$work/pr.json" 2> "$work/err"; then
warn "could not read the pull request; comment not posted"
fi
if [ "$(jq -r '.head.sha // empty' "$work/pr.json")" != "$HEAD_SHA" ]; then
warn "the pull request head moved on; comment for the older commit not posted"
fi

if ! gh api --paginate "repos/$REPO/issues/$PR_NUMBER/comments?per_page=100" > "$work/comments.json" 2> "$work/err"; then
warn "could not list existing comments; comment not posted"
fi
# --paginate emits one JSON array per page; jq reads them as a stream.
if ! jq -r --arg marker "$MARKER" --arg bot "$BOT_LOGIN" \
'.[] | select(.user.login == $bot) | select((.body // "") | startswith($marker)) | .id' \
"$work/comments.json" > "$work/ids" 2> "$work/err"; then
warn "could not parse the comment list; comment not posted"
fi
existing=$(tail -n 1 "$work/ids")

if ! jq -n --rawfile body "$COMMENT_FILE" '{body: $body}' > "$work/payload.json"; then
warn "could not build the comment payload"
fi

case "$existing" in
'')
if ! gh api -X POST "repos/$REPO/issues/$PR_NUMBER/comments" --input "$work/payload.json" > "$work/resp.json" 2> "$work/err"; then
sed -e 's/^/gh: /' "$work/err" >&2 || true
warn "could not create the comment"
fi
echo "Created review comment." ;;
*[!0-9]*)
warn "unexpected comment id; comment not posted" ;;
*)
if ! gh api -X PATCH "repos/$REPO/issues/comments/$existing" --input "$work/payload.json" > "$work/resp.json" 2> "$work/err"; then
sed -e 's/^/gh: /' "$work/err" >&2 || true
warn "could not update the existing comment"
fi
echo "Updated review comment $existing." ;;
esac
Loading
Loading