Add PR Agent review, describe, and improve workflow on CI by PeterDaveHello · Pull Request #367 · thlorenz/doctoc

PeterDaveHello · 2026-06-23T18:52:44Z

User description

Generate PR summaries, review feedback, and GitHub suggestion blocks for same-repository PR updates and maintainer-requested reviews.

Allow trusted maintainers to trigger PR Agent on fork PRs with slash commands without switching the workflow to pull_request_target.

Set PR Agent to use gpt-5.4-mini with xhigh reasoning to balance review quality and API cost.

PR Type

Enhancement

Description

Runs parallel describe/review/improve jobs on same-repo PRs
Accepts trusted slash commands on open PRs
Cancels stale reruns with concurrency groups
Pins pr-agent and configures model settings

Diagram Walkthrough

flowchart LR
  A["pull_request events"] -- "same-repo, non-draft" --> B["Parallel describe, review, improve jobs"]
  C["Trusted issue_comment commands"] -- "open PRs" --> D["Manual slash-command run"]
  B -- "uses" --> E["Pinned pr-agent image + model config"]
  D -- "uses" --> E

File Walkthrough

Relevant files

Configuration changes

pr-agent.yml `Add PR Agent workflow automation and commands` .github/workflows/pr-agent.yml Adds `pull_request` and `issue_comment` triggers. Runs `describe`/`review`/`improve` jobs in parallel for same-repo, non-draft PRs. Accepts trusted slash commands on open PRs, including forks, and cancels stale reruns. Pins `pr-agent` and configures model, timeout, and suggestion settings.	+93/-0

Copilot

Pull request overview

Adds a new GitHub Actions workflow to run PR Agent for automated PR review/description/improvement on same-repository PR updates, and to allow trusted maintainers to trigger PR Agent on PRs via slash commands in issue comments.

Changes:

Introduces .github/workflows/pr-agent.yml to run PR Agent on selected pull_request events targeting master.
Adds issue_comment-based triggering for /review, /describe, /improve, and /ask when invoked by trusted author associations.
Configures PR Agent to use gpt-5.4-mini with xhigh reasoning effort and to emit commitable suggestion blocks.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-06-23T19:21:19Z

PR Reviewer Guide 🔍

(Review updated until commit `ce409db`)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 Security concerns Access control: The manual slash-command path accepts `CONTRIBUTOR` comments, so an external user who has previously contributed can invoke the workflow on open PRs and spend the repository's `OPENAI_KEY` and issue/pull-request write permissions. If that is not intended, narrow the whitelist to maintainers/collaborators only.
⚡ Recommended focus areas for review Access control Because this check is on the commenter, not the PR author, any user with `CONTRIBUTOR` association can invoke the workflow on any open PR. That broadens the intended maintainer-only path and can generate paid AI runs and PR review comments. If the intent is maintainer-only access, drop `CONTRIBUTOR` from the allowlist. contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR", "CONTRIBUTOR"]'), github.event.comment.author_association) &&

github-actions · 2026-06-24T22:57:30Z

PR Code Suggestions ✨

No code suggestions found for the PR.

AndrewSouthpaw · 2026-06-24T20:28:05Z

+
+env:
+  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+  OPENAI_KEY: ${{ secrets.OPENAI_KEY }}


where is the OpenAI key coming from? Whose account? This seems like a blocker.

Currently it's mine, that's the reason why I want to keep it using a more affordable model.

AndrewSouthpaw · 2026-06-25T17:50:08Z

+      matrix:
+        include:
+          - tool: describe
+            auto_describe: "true"
+            auto_review: "false"
+            auto_improve: "false"
+          - tool: review
+            auto_describe: "false"
+            auto_review: "true"
+            auto_improve: "false"
+          - tool: improve
+            auto_describe: "false"
+            auto_review: "false"
+            auto_improve: "true"
+


this matrix seems like unnecessary overhead, just run them all together in one?

The goal of doing it this way is to trigger different tasks in parallel, effectively speeding up the entire workflow. This ensures that both contributors and reviewers don't have to spend too much time waiting for automated feedback.

While the implementation will certainly be slightly more complex, I believe it's worth considering given the value of the time saved waiting.

I hear that, and also my experience is that I can give AI a prompt like "look over the PR description, code, and ways to improve the code" and it'll give an answer equivalently fast. I don't think we're saving a worthwhile amount of time. Do you have numbers on the amount of time saved?

AndrewSouthpaw · 2026-06-25T17:52:57Z

+      (
+        contains(fromJSON('["/review", "/describe", "/improve", "/ask"]'), github.event.comment.body) ||
+        startsWith(github.event.comment.body, '/review ') ||
+        startsWith(github.event.comment.body, fromJSON('"/review\n"')) ||
+        startsWith(github.event.comment.body, '/describe ') ||
+        startsWith(github.event.comment.body, fromJSON('"/describe\n"')) ||
+        startsWith(github.event.comment.body, '/improve ') ||
+        startsWith(github.event.comment.body, fromJSON('"/improve\n"')) ||
+        startsWith(github.event.comment.body, '/ask ') ||
+        startsWith(github.event.comment.body, fromJSON('"/ask\n"'))
+      )


seems overkill, let's just support one command like /review

"/improve" should be its core function, otherwise it won't provide code suggestions.

That suggestion is assuming we collapse into a single function; otherwise supporting all the slash commands makes sense.

Run describe, review, and improve as parallel auto jobs so PRs get earlier summaries, feedback, and suggested fixes. Allow trusted contributors to invoke slash commands on open PRs, including forks, while repeatable commands replace stale runs. Suppress persistent review update notices so reruns refresh the review without adding extra bot comments. Set the PR Agent GitHub Action image to 0.38.0-github_action by tag and digest. Use gpt-5.4-mini xhigh for cost balance and cap LLM calls at 10 minutes inside 60-minute jobs.

github-actions · 2026-06-27T18:07:09Z

+      github.event.sender.type != 'Bot' &&
+      github.event.issue.pull_request != null &&
+      github.event.issue.state == 'open' &&
+      contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR", "CONTRIBUTOR"]'), github.event.comment.author_association) &&


Suggestion: Restrict manual slash-command execution to trusted repository roles only. CONTRIBUTOR includes external users who have merely made a prior contribution, so this job can be invoked by people who should not have access to a workflow that carries OPENAI_KEY and write-scoped tokens. If you need broader access, add a separate explicit trust check instead of treating all contributors as trusted. [security, importance: 7]

Suggested change

contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR", "CONTRIBUTOR"]'), github.event.comment.author_association) &&

contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association) &&

github-actions · 2026-06-27T18:07:09Z

+      github.event_name == 'pull_request' &&
+      github.event.sender.type != 'Bot' &&
+      github.event.pull_request.draft == false &&
+      github.event.pull_request.head.repo.full_name == github.repository


Suggestion: Avoid blanket-excluding all bot-authored PR events here. As written, Dependabot, Renovate, and other automation will never get auto review, so the CI workflow silently misses a big class of PRs. If the goal is only to avoid self-triggering, narrow this check to the PR Agent bot account instead of filtering every bot. [general, importance: 4]

Suggested change

github.event_name == 'pull_request' &&

github.event.sender.type != 'Bot' &&

github.event.pull_request.draft == false &&

github.event.pull_request.head.repo.full_name == github.repository

github.event_name == 'pull_request' &&

github.event.pull_request.draft == false &&

github.event.pull_request.head.repo.full_name == github.repository

AndrewSouthpaw

I'm still a "no" for now. AI slop is running rampant, and it's especially bad with lower thinking models. AI review bots are particularly infuriating, because they slowly dole out one quasi-useful bit of feedback at a time, forcing you into a demoralizing endless loop of iterations.

I have to deal with enough slop at my workplace, I don't need to see it here as well. I am not fussed to tell my higher-quality model to review incoming PRs, then take the feedback, QA it, and then publish.

AndrewSouthpaw · 2026-06-28T13:17:12Z

+      (
+        contains(fromJSON('["/review", "/describe", "/improve", "/ask"]'), github.event.comment.body) ||
+        startsWith(github.event.comment.body, '/review ') ||
+        startsWith(github.event.comment.body, fromJSON('"/review\n"')) ||
+        startsWith(github.event.comment.body, '/describe ') ||
+        startsWith(github.event.comment.body, fromJSON('"/describe\n"')) ||
+        startsWith(github.event.comment.body, '/improve ') ||
+        startsWith(github.event.comment.body, fromJSON('"/improve\n"')) ||
+        startsWith(github.event.comment.body, '/ask ') ||
+        startsWith(github.event.comment.body, fromJSON('"/ask\n"'))
+      )


That suggestion is assuming we collapse into a single function; otherwise supporting all the slash commands makes sense.

PeterDaveHello · 2026-06-28T17:56:47Z

Appreciate the continued review, @AndrewSouthpaw.

To clarify cost and ownership first: any OpenAI API usage for this workflow would be covered by me personally using my own official OpenAI API credits, not by the project. If this moves forward, that responsibility can be documented explicitly.

I understand the concern about the quality of AI-generated review output. That is a valid risk, but one that can be managed through scope control, trigger restrictions, model selection, and final maintainer review. At the same time, AI-assisted code review has become a fairly common part of modern development workflows, including GitHub’s own Copilot review capabilities. The models and tools have continued to improve significantly over the past few years, and they can now provide faster first-pass feedback as well as an additional review perspective that may catch issues a single human reviewer might overlook.

AI review is not perfect or authoritative. Like any reviewer, it can be wrong, incomplete, or overly confident, so maintainer judgment remains essential. However, it can serve as a consistent and less fatigue-prone first pass, particularly for repetitive checks, edge cases, documentation consistency, test behavior, and workflow details. This repository has already used GitHub Copilot for code review on multiple PRs in the past, with personal credits, and it provided useful feedback in those areas.

The intent of this PR is to make that pattern more controlled: a maintainer-triggered assistant for PR describe, review, and improve workflows, with clear guardrails around permissions, cost, and ownership. It is not meant to replace human review, require anyone to rely on the output, or allow untrusted actors to trigger paid runs.

Based on the review so far, it seems the remaining concern is with the overall direction rather than the specific workflow details. If there are concrete guardrails that would address the concerns raised here, I’m open to narrowing the PR accordingly. If the objection is to having AI-assisted PR review in the repository at all, it may be better to pause further implementation changes on this PR and first seek broader maintainer alignment on that direction.

PeterDaveHello requested a review from Copilot June 23, 2026 18:52

PeterDaveHello requested a review from AndrewSouthpaw as a code owner June 23, 2026 18:52

Copilot started reviewing on behalf of PeterDaveHello June 23, 2026 18:53 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

Comment thread .github/workflows/pr-agent.yml Outdated

Comment thread .github/workflows/pr-agent.yml Outdated

PeterDaveHello force-pushed the add-pr-agent-review-workflow branch from a2859e3 to 5aa0a47 Compare June 23, 2026 19:12

github-actions Bot added the Review effort 2/5 label Jun 23, 2026

github-actions Bot reviewed Jun 23, 2026

View reviewed changes

Comment thread .github/workflows/pr-agent.yml Outdated

PeterDaveHello force-pushed the add-pr-agent-review-workflow branch from 5aa0a47 to 2cafd74 Compare June 24, 2026 15:47

github-actions Bot reviewed Jun 24, 2026

View reviewed changes

Comment thread .github/workflows/pr-agent.yml Outdated

github-actions Bot added Review effort 3/5 and removed Review effort 2/5 labels Jun 24, 2026

This comment was marked as outdated.

Sign in to view

PeterDaveHello force-pushed the add-pr-agent-review-workflow branch from 2cafd74 to 08f34d6 Compare June 24, 2026 21:20

This comment was marked as outdated.

Sign in to view

github-actions Bot reviewed Jun 24, 2026

View reviewed changes

Comment thread .github/workflows/pr-agent.yml

PeterDaveHello force-pushed the add-pr-agent-review-workflow branch from 08f34d6 to ba49d77 Compare June 24, 2026 21:32

This comment was marked as outdated.

Sign in to view

PeterDaveHello force-pushed the add-pr-agent-review-workflow branch from ba49d77 to 632feac Compare June 24, 2026 22:27

This comment was marked as outdated.

Sign in to view

AndrewSouthpaw requested changes Jun 25, 2026

View reviewed changes

AndrewSouthpaw reviewed Jun 25, 2026

View reviewed changes

Comment thread .github/workflows/pr-agent.yml Outdated

PeterDaveHello force-pushed the add-pr-agent-review-workflow branch from 632feac to 290f4a7 Compare June 27, 2026 15:49

PeterDaveHello requested a review from AndrewSouthpaw June 27, 2026 15:50

PeterDaveHello force-pushed the add-pr-agent-review-workflow branch from 290f4a7 to 023ac21 Compare June 27, 2026 16:01

github-actions Bot added Review effort 2/5 and removed Review effort 3/5 labels Jun 27, 2026

This comment was marked as outdated.

Sign in to view

github-actions Bot reviewed Jun 27, 2026

View reviewed changes

Comment thread .github/workflows/pr-agent.yml

Comment thread .github/workflows/pr-agent.yml

PeterDaveHello force-pushed the add-pr-agent-review-workflow branch from 023ac21 to ce409db Compare June 27, 2026 17:19

github-actions Bot added Review effort 3/5 and removed Review effort 2/5 labels Jun 27, 2026

github-actions Bot reviewed Jun 27, 2026

View reviewed changes

AndrewSouthpaw requested changes Jun 28, 2026

View reviewed changes

	contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR", "CONTRIBUTOR"]'), github.event.comment.author_association) &&
	contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association) &&

Uh oh!

Conversation

PeterDaveHello commented Jun 23, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

User description

PR Type

Description

Diagram Walkthrough

File Walkthrough

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Reviewer Guide 🔍

(Review updated until commit ce409db)

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

github-actions Bot commented Jun 24, 2026

PR Code Suggestions ✨

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PeterDaveHello Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

github-actions Bot Jun 27, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 27, 2026

Choose a reason for hiding this comment

Uh oh!

AndrewSouthpaw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PeterDaveHello commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

PeterDaveHello commented Jun 23, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Jun 23, 2026 •

edited

Loading

(Review updated until commit `ce409db`)

PeterDaveHello Jun 25, 2026 •

edited

Loading

PeterDaveHello commented Jun 28, 2026 •

edited

Loading