Repository navigation
docs: fix drifted claims and guard them with archtests - #166
Conversation
…LOG.md AGENTS.md listed five hashed skill files and named four prompts; the code hashes six, adding prompts/validate.md. It also promised that a prompt change moves only prompts_hash golden lines, but internal/provenance/prompts_test.go pins one literal hash, so the workflow named half the work. CONTRIBUTING.md pointed at a changelog at the bottom of docs/manual.md, which has no changelog section: changesets write the root CHANGELOG.md, and the release body is read back from it. internal/archtest/docdrift_test.go now holds AGENTS.md's sentence to provenance.promptFiles: the set, and the spelled count.
The vision said verdict posts "exactly one GitHub review" and that `gh` does GitHub I/O. `post` has GitLab and Forgejo adapters (internal/core/post), the README host table marks all three post-capable, and the manual documents the GitLab request-changes downgrade, so the one-line claim understated the tool.
The "Shipped today" table claimed to be the honest ledger of shipped verbs and left out backtest, index, pack, panel, serve, risk, handoff, ingest, run, and pr-wait, all of which the usage synopsis offers and the manual documents. The Next section still said the dashboard UI follows, but serve ships it today (inbox, review detail, threads, cost, precision, gate panel) and report --html exports the same views, so the dashboard is a shipped row now. TestRoadmapListsEveryShippedVerb reads the table's Commands column and compares it to the CLI synopsis, so a new verb and a stale row both fail here.
The page called bench/corpus/ "eight seeded bug cases, two seeded clean controls, and two fake agent outputs per case". That described the seeded half: the directory holds 50 cases — 8 seeded bug cases, 2 seeded clean controls, 32 real-PR cases, and 8 real clean controls copied from merged fixes — and only the ten seeded cases carry fixture outputs. TestBenchmarkCorpusCountMatchesTree reads the case directories and each manifest's kind, so the total and the real-case count both move with the tree. The walkers stay under the repo's lint complexity budget.
The page documented only its seeded half. It carries the "Adding a real case" rule for the 32 real-PR cases and 8 real clean controls checked in beside the seeded fixtures, so "Seeded benchmark corpus" named half the directory.
|
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 20 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
The opening sentence still called the seeded cases "the template for the operator's real-PR corpus". The real cases are checked in beside them now, so the intro says what the directory holds: seeded fixtures for the bench-corpus eval, plus real-PR cases and clean controls copied from merged fixes.
The doc guards carried a comment on every unexported var, helper and test. Only the reason hashedPromptFiles reads the list out of Go source survives; the Commands column is named in the code instead of explained above it.
Fixes STA-434. Part of STA-436 (release 0.7.0).
What
Docs that drifted from the code, each checked against
14aca16d1:AGENTS.md: the hashed prompt set is six files (includingprompts/validate.md). The "onlyprompts_hashlines move" workflow now names the literal pin ininternal/provenance/prompts_test.go.CONTRIBUTING.md: the changelog is the rootCHANGELOG.md, not a section ofdocs/manual.md.docs/vision.md:postreaches GitHub, GitLab and Forgejo, not exactly one GitHub review.backtest,index,pack,panel,serve,risk,handoff,ingest,runandpr-wait. The dashboard ships today.docs/benchmark.mdandbench/corpus/README.md: the corpus is 50 cases, 32 of them real-PR cases with fixed defects.Drift guards
internal/archtest/docdrift_test.gocompares each document with an independent source:AGENTS.md's hashed set and count against thepromptFilesAST;benchmark.md's counts against thebench/corpus/tree.Each guard failed on main, and mutations (dropping a file, a wrong count, a missing or extra verb, a probe case) make it fail.
Evidence
make evals135/135 and whole-repogo testpass, as do golangci-lint (native andGOOS=windows) andbun fmt --check.