Skip to content

About

Maintainer-gated pull request review workflow built on PR-Agent

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

208 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

agent-review

A maintainer-gated agentic pull request reviewer for GitHub Actions. Comment an exact command on an open, same-repository PR:

Command Passes
/review Two independent Sonnet passes: bundle first, then callers first
/review deep One Opus pass
/review codex One GPT-6.1 Terra pass at high effort
/review codex deep One GPT-6.1 Sol pass at medium effort

The engine builds a merge-base diff with full changed files, falling back to changed regions when the configured input budget requires it. A change too large for one request is reviewed in deterministic bounded sections instead of being refused: every patch byte reaches every independent pass exactly once, each section can investigate the whole head checkout and the full patch index, and a final cross-section stage follows. The reviewer can read related source. Each invocation reports concrete introduced defects with a head location, trigger, consequence, repair, and confidence. Findings are merged by location and mechanism. Each invocation can return up to six candidates, and every admitted candidate is kept; there is no evidence-challenge veto.

Inference jobs have read-only GitHub permissions. A separate publisher validates the result and transcript-derived receipts, rechecks the captured commits and original command, then publishes resolvable inline threads and a short summary table. Findings on overlapping lines share threads within the per-comment byte bound. Earlier verified success status lines are hidden as outdated; earlier summaries stay visible. A finding whose range is visible in the diff gets a line thread; a confirmed location in a changed file outside the visible diff gets a file-level thread with its real location in the text; anything else stays in the summary with the reason. A summary or completion record that outgrows GitHub's comment limit continues in verified comments and digest-chained evidence pages. Immutable completion records support duplicate detection, incremental coverage, and verified follow-up replies.

When a review runs inference, it can also verify open threads backed by its completion records. One batched pass uses that review's provider, model, read-only tools, and per-pass limits. Only threads whose code changed (GitHub marks them outdated) are checked. Fixed evidence must cover every finding and include each finding's recorded path; evidence only from related files leaves the thread open. Present, unclear, unowned, edited, or unverifiable findings stay open. Silence in a later review never resolves anything. A verification failure leaves threads open without failing the review. Unchanged snapshots skip inference and thread verification.

Resolution remains a model judgment, as with CodeRabbit's documented Auto-Resolve. The pull request author controls both the code the model reads and which threads become outdated, so automatic resolution is a convenience, not a security sign-off; a maintainer should still check resolved security findings before merging. Resolved threads remain in the PR and can be reopened. Once the bot posts a completed verified decision or records a resolution attempt for a root, it never auto-resolves that root again, even at the same head or another review depth. A failed resolve needs a person or a new /resolve command. Model text cannot supply the bot's state markers.

To ask the reviewer about one of its threads, start the reply with /ask followed by the question (case-sensitive; a bare /ask asks it to re-check the finding). Any other reply text does nothing and costs nothing, so ordinary conversation in a thread never starts a paid answer. An admitted /ask reads the head checkout with restricted file tools. It can answer earlier-head threads and resolve an outdated thread when the human reports a fix that the code confirms. When the code appears to disprove a finding, the answer says so and leaves the thread open for a maintainer. Disputed findings that remain valid stay open with an explanation. Membership, event binding, provenance, and publication checks still apply.

To resolve one owned thread explicitly, reply with exactly /resolve (case-sensitive; it must start the reply, and trailing whitespace is allowed). The admitted command skips model inference, acknowledges the request, and asks GitHub to resolve only that thread. It can resolve live or previously reopened threads. An already-resolved thread produces no action. A reply from a non-member, an edited comment, or an unowned root never authorizes the command. Only the pull request author or someone with write access, the people GitHub itself lets resolve a conversation, can have the bot resolve a thread, by /resolve or by a reply reporting a fix; anyone else gets an answer and the thread stays open. Other text, including /resolve with extra words, is not a command. The acknowledgment claims success only after GitHub confirms resolution; ambiguous failures stay visible.

Before a review enters the per-PR queue, the trusted workflow adds an eyes reaction to acknowledge receipt of the exact live command, then posts a queued comment and an Actions link. The reaction does not mean inference has started. The live Actions step log shows each independent pass, section and cross-section consolidation as it starts and is processed. The job summary retains those events after the step finishes. While inference runs, an observer patches that same comment with the pass, section, or consolidation counter, the Actions link, and publication-pending wording. These counters and stages do not certify completed coverage. The queued comment changes to running when admission completes, then becomes the verified completion or incomplete notice. A progress update is best effort and never establishes completed coverage. A caller-side terminal writer reports cancellation even while the review was queued. GitHub runner unavailability or force cancellation can still prevent an update; the Actions run remains the diagnostic source.

The reviewer has no PR line-count limit. Complete changes are packed into bounded sections, while the configured time, turn and dollar allowances still apply to each whole pass. A turn-limit notice identifies max_turns explicitly; an unchanged rerun is not a remedy for an exhausted allowance.

Repository learnings

Reviews and inline answers can use approved, sourced background from the private pipelab-org/agent-review-learnings repository. Map the optional AGENT_REVIEW_LEARNINGS_APP_ID and AGENT_REVIEW_LEARNINGS_APP_KEY secrets for its existing read-only GitHub App. The workflow fetches maintainers.toml and repos/<owner>/<repo>.toml at one default-branch commit, using the caller's exact github.repository as the lookup key.

Only valid, active entries approved by a maintainer and within their review date load. Entries without paths apply repository-wide; path globs select entries for changed files. Missing credentials, fetch failures, missing repository files, invalid data and oversized context all let the review proceed without learnings. A configured credential whose token can't be minted, or whose token revocation can't be confirmed, stops the review before inference instead. Review details reports the count and learnings commit, or a fixed reason for loading none.

Learnings are labeled untrusted background in both provider prompts. They never filter, hide, drop or downgrade findings. A reviewer that disagrees with a learning must explain why its evidence still establishes a defect. Raw private learning context stays out of the public inference artifact; the summary exposes only safe metadata. See the operating contract for bounds and identity rules.

Required operator policy

The central .github/pr-agent.toml retains its filename for compatibility. Each of review_depth.default, review_depth.deep, review_depth.codex, and review_depth.codex-deep requires max_turns and max_budget_usd. The code has no defaults; the shipped policy sets 80 turns, USD 5 per default pass, USD 9 per deep pass, USD 2 per codex pass, and USD 2 per codex-deep pass, so reviews run and incur provider charges once the provider secrets are configured. Removing or invalidating either setting fails before inference with the exact config key to set.

The USD allowance applies per pass, cumulatively over that pass's sections and final cross-section stage, with one monotonic deadline per pass: later sections receive only what earlier ones left. A review's configured allowance is its review pass count multiplied by that limit, plus one pass when thread verification runs. Verification that cannot fit the existing limits is skipped with a fixed notice. These are client-side cost estimates and can overshoot on an in-flight request; provider account limits remain the billing backstop. The 80-turn allowance was increased after an observed large-change turn stop; the Claude budgets of USD 5 and USD 9 per pass are estimates for a pass that fills the 500,000-byte input allowance and will be replaced by measured values. Codex limits are provisional values from a partial evaluation run, as described in .github/pr-agent.toml.

Adoption

  1. Install .github/workflows/agent-review.yaml, using the exact maintained caller template. External callers must pin the reusable workflow to a full merged commit SHA with the # main tracking comment. The example caller provides that form.
  2. Add .github/agent-review.toml with only [review] instructions and [ignore] glob. Profiles cannot change models, tools, budgets, admission, credentials, or publication.
  3. Configure the centrally approved per-depth limits before adopting the engine.
  4. Map ANTHROPIC_API_KEY for Claude and/or OPENAI_API_KEY for Codex as named secrets. Optional ANTHROPIC_API_KEY_2 permits one fallback after an initial HTTP 400 billing refusal with a complete zero-token, zero-cost CLI receipt and no successful model response. Later passes retain the selected key. Other failures stop without switching accounts or providers. Optional ORG_MEMBERS_TOKEN supplies Members:read for private organization membership checks on replies.
  5. Merge the caller and profile to the default branch, then post a command.

Only the designated maintainer's original, unedited command is admitted. Fork and closed PRs are explicitly refused before provider work. Reruns, bots, edited events, and unsupported command arguments are ineligible. Models and repository text have no authority over publication.

Development

To diagnose configured provider credentials without choosing a pull request, manually run Provider connectivity check from Actions on main. It makes a small live request through each selected provider's normal restricted runner and can incur API charges under the existing central limits. Anthropic tests the primary and secondary keys in separate jobs, with fallback disabled, so one working account can't hide a failure on the other. The check prints fixed success or failure diagnostics, never keys or provider response text. It checks access, not the remaining credit balance or review quality. It does not run automatically or retry failed workflows.

CI uses Python 3.12 on Linux x86_64 with hash-pinned dependencies:

pip install --require-hashes -r .github/requirements-runtime.txt -r .github/requirements-test.txt
python -m unittest discover -s scripts -p 'test_*.py'
python -m unittest -v scripts.test_review_engine scripts.test_review_jobs

The unit tests above run offline with fixture credentials and provider transcripts. CI also runs progress-transport-proof on a first attempt: it uploads one synthetic counter through the real progress action and checks that the supervisor starts without the artifact credential and that cancellation stops it. That job does not call a provider or edit a pull request. The argv harness proves argument forwarding and credential separation. A native CLI dry run also checks tool restrictions, scoped reads, skipped PR hooks/MCP, schema completion, and turn/budget stops against a local fixture API. It does not establish real provider behavior or hosted-runner isolation. See the operating contract and vendor evidence.

License

Apache-2.0. See LICENSE.

About

Maintainer-gated pull request review workflow built on PR-Agent

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages