Skip to content

fix(running-in-ci): fetch a review's inline comments as part of reading context - #950

Open
tend-agent wants to merge 1 commit into
mainfrom
hourly/review-31552948925
Open

fix(running-in-ci): fetch a review's inline comments as part of reading context#950
tend-agent wants to merge 1 commit into
mainfrom
hourly/review-31552948925

Conversation

@tend-agent

Copy link
Copy Markdown
Collaborator

A maintainer's inline review comment told the bot not to do a thing; seven minutes later the bot published that exact thing on the same thread, because the review body it read gave no sign the inline comment existed.

What happened

On PRQL/prql#6181, kgutwin submitted review 4911815208 (COMMENTED, 00:07:20Z). Its body is one line — "Closing this in favor of #6020 which is a superset of this PR and is more likely to be merged." The substance was in the review's single inline comment, discussion_r3762633057 on prqlc/prqlc/tests/integration/sql.rs:

Do not create a test that has TODO: this is wrong as its description. Tests that encode current, incorrect behavior are never appropriate.

tend-mention run 31549073589 handled the review event and replied at 00:12:37Z — agreeing with the close, then pasting both of the PR's tests so they could be lifted into #6020. One of the two pasted tests is test_aggregate_matching_group_key, carrying the literal // TODO: this is wrong comment the maintainer had just rejected, offered for adoption into another PR. The inline comment itself was never answered.

Root cause

The session log shows the agent read the review and stopped at its body. The prompt named the review ID but not its contents:

A review was submitted on a PR where you previously participated (PR #6181, …, review ID 4911815208). Read the review and full context.

The agent's two context reads were gh pr view 6181 --json title,body,author,state,headRefName,comments,reviews,url and gh api repos/PRQL/prql/pulls/6181/reviews/4911815208. Neither returns inline comments. The second returned exactly the one-line body, which reads as a complete review. There is no call to reviews/4911815208/comments, pulls/6181/comments, or the comment ID anywhere in the log, and no assistant text mentions the objection.

That is the structural part: both obvious ways to read a review omit its inline comments and give no signal that anything is missing. running-in-ci does carry the correct fetch, but only under "Replying to Comments" — framed as how to reply inline. An agent that reads the body, concludes a top-level comment is the right response, and never intends an inline reply has no reason to reach that section. The recipe sits behind the decision it was supposed to inform.

Fix

One subsection in "Read Context", where the agent learns what to read, stating that the inline comments are a separate fetch and giving the command. It duplicates the gh api …/reviews/<id>/comments call already in "Replying to Comments"; that is deliberate — the two are reached at different moments, and the read-time moment is the one that failed here.

Gate assessment

Evidence log: https://gist.github.com/192514ea2c36586f9b7f842a482d62ab

Detected by review-reviewers run https://github.com/max-sixty/tend/actions/runs/31552948925.

…ng context

Neither `gh pr view --json reviews` nor `GET /pulls/<n>/reviews/<id>`
returns a review's inline comments, and neither signals that anything is
missing — so a one-line review body reads as a complete review while the
maintainer's actual instruction sits inline, unread.

The fetch recipe already existed, but only under "Replying to Comments",
which an agent that never intends an inline reply has no reason to open.
Move the knowledge to where the read decision is made.
@tend-agent tend-agent added the claude-behavior Behavioral issues found by review-reviewers label Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-behavior Behavioral issues found by review-reviewers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant