Skip to content

docs: fix drifted claims and guard them with archtests - #166

Merged
SutuSebastian merged 7 commits into
mainfrom
agent/sta-434-docs-drift
Nov 16, 2026
Merged

SutuSebastian merged 7 commits into
mainfrom
agent/sta-434-docs-drift

Conversation

@SutuSebastian

Copy link
Copy Markdown
Contributor

Fixes STA-434. Part of STA-436 (release 0.7.0).

What

Docs that drifted from the code, each checked against 14aca16d1:

  • AGENTS.md: the hashed prompt set is six files (including prompts/validate.md). The "only prompts_hash lines move" workflow now names the literal pin in internal/provenance/prompts_test.go.
  • CONTRIBUTING.md: the changelog is the root CHANGELOG.md, not a section of docs/manual.md.
  • docs/vision.md: post reaches GitHub, GitLab and Forgejo, not exactly one GitHub review.
  • Docs-site roadmap: the "Shipped today" table lists every CLI verb, adding backtest, index, pack, panel, serve, risk, handoff, ingest, run and pr-wait. The dashboard ships today.
  • docs/benchmark.md and bench/corpus/README.md: the corpus is 50 cases, 32 of them real-PR cases with fixed defects.

Drift guards

internal/archtest/docdrift_test.go compares each document with an independent source:

  • AGENTS.md's hashed set and count against the promptFiles AST;
  • the roadmap's verbs against the CLI synopsis;
  • benchmark.md's counts against the bench/corpus/ tree.

Each guard failed on main, and mutations (dropping a file, a wrong count, a missing or extra verb, a probe case) make it fail.

Evidence

  • make evals 135/135 and whole-repo go test pass, as do golangci-lint (native and GOOS=windows) and bun fmt --check.
  • Cross-family review (Gemini 4 Hydrogen): APPROVE, with every corrected claim verified against the code.
  • No changeset: these are docs and tests only.

…LOG.md

AGENTS.md listed five hashed skill files and named four prompts; the code
hashes six, adding prompts/validate.md. It also promised that a prompt change
moves only prompts_hash golden lines, but internal/provenance/prompts_test.go
pins one literal hash, so the workflow named half the work.

CONTRIBUTING.md pointed at a changelog at the bottom of docs/manual.md, which
has no changelog section: changesets write the root CHANGELOG.md, and the
release body is read back from it.

internal/archtest/docdrift_test.go now holds AGENTS.md's sentence to
provenance.promptFiles: the set, and the spelled count.
The vision said verdict posts "exactly one GitHub review" and that `gh` does
GitHub I/O. `post` has GitLab and Forgejo adapters (internal/core/post), the
README host table marks all three post-capable, and the manual documents the
GitLab request-changes downgrade, so the one-line claim understated the tool.
The "Shipped today" table claimed to be the honest ledger of shipped verbs and
left out backtest, index, pack, panel, serve, risk, handoff, ingest, run, and
pr-wait, all of which the usage synopsis offers and the manual documents. The
Next section still said the dashboard UI follows, but serve ships it today
(inbox, review detail, threads, cost, precision, gate panel) and report --html
exports the same views, so the dashboard is a shipped row now.

TestRoadmapListsEveryShippedVerb reads the table's Commands column and compares
it to the CLI synopsis, so a new verb and a stale row both fail here.
The page called bench/corpus/ "eight seeded bug cases, two seeded clean
controls, and two fake agent outputs per case". That described the seeded half:
the directory holds 50 cases — 8 seeded bug cases, 2 seeded clean controls,
32 real-PR cases, and 8 real clean controls copied from merged fixes — and
only the ten seeded cases carry fixture outputs.

TestBenchmarkCorpusCountMatchesTree reads the case directories and each
manifest's kind, so the total and the real-case count both move with the tree.
The walkers stay under the repo's lint complexity budget.
The page documented only its seeded half. It carries the "Adding a real case"
rule for the 32 real-PR cases and 8 real clean controls checked in beside the
seeded fixtures, so "Seeded benchmark corpus" named half the directory.
@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 8c8070d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 20 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f07ef1ff-b2d4-450e-9c3d-1dbdac3134fd
Commits

Reviewing files that changed from the base of the PR and between 14aca16 and 8c8070d.

Files selected for processing: 7

  • AGENTS.md
  • CONTRIBUTING.md
  • apps/docs/content/04-roadmap/index.mdx
  • bench/corpus/README.md
  • docs/benchmark.md
  • docs/vision.md
  • internal/archtest/docdrift_test.go
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

The opening sentence still called the seeded cases "the template for the
operator's real-PR corpus". The real cases are checked in beside them now, so
the intro says what the directory holds: seeded fixtures for the bench-corpus
eval, plus real-PR cases and clean controls copied from merged fixes.
The doc guards carried a comment on every unexported var, helper and test.
Only the reason hashedPromptFiles reads the list out of Go source survives; the
Commands column is named in the code instead of explained above it.
@SutuSebastian
SutuSebastian merged commit e5ffe9b into main Nov 16, 2026
21 checks passed
@SutuSebastian
SutuSebastian deleted the agent/sta-434-docs-drift branch November 16, 2026 11:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant