Skip to content

docs: give a docs-only branch a reader test in place of the adversarial pass - #34

Merged
sstriker merged 5 commits into
mainfrom
docs/reader-test
Oct 28, 2026
Merged

sstriker merged 5 commits into
mainfrom
docs/reader-test

Conversation

@sstriker

Copy link
Copy Markdown
Contributor

Summary

Before this change, a branch that touched only docs had no pre-PR pass. This change gives such a branch a reader test in place of the adversarial pass.

A docs-only branch is a branch whose changed paths scripts/docs-only.sh accepts. The reader test has these steps:

  1. The author writes three to five questions that the change must let a reader answer. The author also writes the correct answers, and does not give them to the reader.
  2. The author copies the changed files and the diff into an empty directory.
  3. Two fresh agents start at the same time:
    • The reader reads only that directory and answers the questions.
    • The fact check compares each claim in the change with the code. When the change is to the repository's rules, it also compares the change with AGENTS.md, the commands, the hooks, the scripts and CI.
  4. A wrong or missing answer is a finding against the text. The author changes the text and runs the pass again.
  5. The PR description gives the questions, the results, and what is still open.

The command .claude/commands/reader-test.md gives the full procedure. It also says which revisions need both agents again, which need only the fact check, and which need no pass.

This change also corrects two errors in .claude/commands/adversarial.md. These errors were also in the first version of the new command:

  • The command named a repository that is not this one. The text now says "this repository".
  • The command said to run the pass on a revision before the push. The pass reads origin/<branch>, so it runs after the push and before review is re-requested. AGENTS.md now says this for both passes.

One result of the new definition: a branch that changes only a docs file that is not Markdown, such as docs/design-book.html, now gets the adversarial pass.

Trial

The first trial was PR #33. It ran four rounds. The fact checks found 13 errors in the text: 6, 4, 2 and 1 in the four rounds. The readers found many sentences with two meanings. Details are in #33.

Reader test of this branch

This branch is docs-only, so it got its own reader test. I wrote these questions first:

  1. A branch changes only README.md. What must you do before you open the PR, and in what order?
  2. What may the reader read, and what is kept from it?
  3. The reader answers a question wrongly. What do you do?
  4. You revise the text after the pass. What do you run again?
  5. What must the PR description say about the pass?

Results:

  • Round 1: the reader answered all five questions, but the answer to question 1 depended on a guess about what "docs" means. The fact check found three errors:
    • A reader agent is not fully "cold". Each agent in this repository loads CLAUDE.md, and AGENTS.md through it, before it reads anything. The command now says this, and tells the author to ask questions that AGENTS.md cannot answer.
    • The timing of a pass on a revision was wrong, as described above.
    • The repository name was wrong, as described above.
  • Round 2: all five answers were correct. The fact check found that AGENTS.md did not give the re-run rules, and that AGENTS.md still called the reader "cold". Neither file said which list of paths goes to scripts/docs-only.sh.
  • Round 3: the reader answered all questions correctly in round 2, so only the fact check ran, on the changed lines. It found one gap in three places: the reader also gets the diff, not only the changed files. The reader also loads the user's own memory, as well as AGENTS.md. I added both, and I read the result myself.

These points stay open:

  • Nothing enforces either pass. The table under "Some of these are held for you" lists both prompts, but no hook or CI job runs them.
  • AGENTS.md says "Whatever else lands in .claude/ stays ignored", but .gitignore ignores only two paths under .claude/. This change does not touch that line.

Posted by Claude on behalf of @sstriker

sstriker and others added 5 commits October 27, 2026 09:52
…al pass

A fresh agent reads the changed files alone and answers questions that
the author writes first. A second fresh agent checks each claim
against the code. The first trial, on PR 33, found six errors that a
read of the diff did not find.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bhj4Ck4y5iwZUsjpgQx8A2
The reader test of this branch found these problems. AGENTS.md did not
define a docs-only branch. Both commands said to run a pass before a
push, but a pass reads what is pushed. Both named a repository that
is not this one. A reader agent loads AGENTS.md before it reads the
changed files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bhj4Ck4y5iwZUsjpgQx8A2
The second reader test and fact check of this branch found these
problems. AGENTS.md did not say that a revision can need only the
fact check, or no pass. It called the reader cold. Neither file said
which paths go to scripts/docs-only.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bhj4Ck4y5iwZUsjpgQx8A2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bhj4Ck4y5iwZUsjpgQx8A2
Copilot AI balanced review requested due to automatic review settings October 28, 2026 00:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No blocking or unresolved issues were raised in the review, leaving nothing outstanding to address.

No open issues 🎉

What changed in this PR

This PR extends the repository's pre-PR process so that a docs-only branch gets a "reader test" instead of the adversarial pass, which previously was simply skipped for docs-only branches. A docs-only branch is one whose changed paths scripts/docs-only.sh accepts (Markdown outside crates//taps/). The reader test spawns two fresh subagents at the same pre-PR points as the adversarial pass: one reads only the changed files plus their diff and answers author-written questions, and one fact-checks each claim against the code. The PR also fixes two latent errors in the existing adversarial prompt that were duplicated into the new command.

Changes:

  • Add .claude/commands/reader-test.md defining the reader-test procedure (question writing, two cold agents, grading, and re-run rules).
  • Update AGENTS.md to require the reader test for docs-only branches "at the same points," list the new prompt in the .claude/ table, and clarify that both passes run after the push and before review is (re-)requested.
  • Correct .claude/commands/adversarial.md: replace the hardcoded sstriker/durable-memory with "this repository," and fix the revision-pass timing to "after its push and before review is re-requested."
File Description
.claude/​commands/​reader-test.md New command: full reader-test procedure for docs-only branches, with reader and fact-check prompts and re-run rules.
AGENTS.md Documents the reader test as the docs-only substitute for the adversarial pass, adds the command to the .claude/ table, and corrects both passes' timing.
.claude/​commands/​adversarial.md Fixes the hardcoded repo name and the revision-pass timing.

No blocking issues found. The new command is internally consistent with AGENTS.md and with scripts/docs-only.sh (the referenced git diff --name-only --no-renames origin/main...origin/<branch> | scripts/docs-only.sh invocation matches the script's documented usage and intended semantics), and the two adversarial.md corrections are accurate.

Review effort: Balanced

@sstriker
sstriker merged commit 6182b3e into main Oct 28, 2026
9 checks passed
@sstriker
sstriker deleted the docs/reader-test branch October 28, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants