feat(context): AI-readiness gate, doc-truth rules, and 30% faster scans (re-land #44)

.claude/knowledge/architecture-boundaries.md

@@ -30,3 +30,6 @@ Key architectural decisions, service boundaries, data flow, integration points,
 - **The usage menu makes one outbound call (opt-out).** Running `codeguard` with no args or `help` renders a "What's New" banner (`internal/whatsnew` + `internal/cli/whatsnew.go`) that queries the GitHub releases API for the latest version. It is disabled when `CODEGUARD_NO_UPDATE_CHECK` is truthy OR `$CI` is non-empty (so CI/automated runs make no surprise call), caches the result for 24h under `os.UserCacheDir()/codeguard/update-check.json`, uses the SSRF-guarded `ai/safehttp.Client` with a 1.5s timeout, and fails silent (falls back to stale cache, then to version-only). Changelog bullets come from `CHANGELOG.md` embedded via a **module-root package** (`/changelog.go`, `package codeguard`, imported as `root "github.com/devr-tools/codeguard"`) — `go:embed` cannot reference `../CHANGELOG.md`, so the embed directive must live at the repo root, not in `internal/`.
 - **Lint is enforced in CI.** `make lint` is `go vet`; `make lint-strict` / the CI `golangci-lint-action` step (now blocking — no `continue-on-error`) runs the 16-linter `.golangci.yml`. `golangci-lint run` must report 0 before pushing. The config excludes revive's documentation/naming style rules (`exported`, `package-comments`, `unused-parameter`) and excludes `tests/` from gosec/errcheck/prealloc/bodyclose (intentional fixtures). Cache/identity keys use sha256 (not sha1). Documenting the public `pkg/codeguard` API with doc comments is a known follow-up (revive `exported` is currently excluded rather than satisfied).
 - **Adding a check section touches a fixed wiring list; two are silent-failure traps.** Beyond the obvious (check package with `Run(ctx, env) core.SectionResult`, a `sectionDef` in `runner/checks/registry.go`, a `CheckConfig` bool + rules struct in `core/config_types.go`/`config_rule_types.go`, catalog + fix-template maps merged in `rules/catalog.go`/`catalog_fix_templates.go`, defaults in `config/defaults*.go` + `example*.go`, docs): (1) `runner/support/cache_helpers.go` — add the section to `SectionConfigHashes` AND `sectionConfigFamily`, else the section's per-file cache keys fall back to the all-checks fingerprint (correct but invalidates on any config change); (2) top-level `CheckConfig` bools get NO defaulting in `applyCheckDefaults` — an omitted `checks.<section>` key means **off**, which is how the opt-in `performance` and `supply_chain` sections work. The performance section (v0.9.0) is the reference move: rules migrated from quality with renamed ids (`quality.*` → `performance.*`), no runtime aliasing, `rules.RetiredPerformanceRuleIDs()` powers a `codeguard doctor` waiver migration hint.
+- **PERF: the quality AI checks read through the corpus via `checks/support.Context.ListTargetFiles`/`ReadTargetFile`** (wired in `runner/checks/checks.go`, backed by `runnersupport.ListTargetFiles`/`ReadTargetFile`; `checks/quality/quality_ai_corpus.go` holds the nil-safe fallbacks for unit-test contexts). GOTCHA when routing more reads through the corpus: `corpusRead` caps at 32 MiB (`readCappedFile`) while `os.ReadFile` does not — walk-enumerated files are already under the cap so behavior is identical, but fixed-filename reads that never went through the walk (`go.mod`, root `package.json`, python manifests) deliberately stay direct `os.ReadFile` to preserve semantics. The Go repo-dominant style signals (test framework, error style, naming) are computed in ONE corpus pass (`goRepoStyleProfile` in `quality_ai_target_go.go`); each signal is an independent per-file sum, which is what makes the fold behavior-identical.
+- **GOTCHA: `goFuncEligibleForUnusedCheck` must not consult `fn.Doc`.** The AI dead-code pass historically parsed with `parser.ParseFile(..., 0)`, so `fn.Doc` was always nil and the "go: directive implies external use" exemption NEVER fired. The pass now reuses the shared ParseComments corpus AST (`support.ParseGoSource`), where `fn.Doc` IS populated — keeping that check would have silently exempted unused functions carrying `//go:generate`-style comments and changed findings. It was removed to keep output identical (see the NOTE in `quality_ai_dead_code.go`); activating it is a candidate behavior fix, to be made deliberately with tests.
+- **PERF: clone-detector hashing is not behavior-bearing.** `cloneToken` now stores the original-case source slice plus a per-token FNV-1a hash of the ASCII-lowercased text (tokens are ASCII-only by the regex); window grouping uses a rolling polynomial hash (`cloneWindowIndex`) and equality is verified token-by-token in `sharedCloneLength` (hash short-circuit + ASCII case-fold compare). Because every candidate is value-verified, ANY deterministic hash yields identical clone candidates — collisions only cost time. `tests/checks/quality_clone_differential_test.go` pins findings captured from the pre-rewrite algorithm; don't regenerate its expectations with the current code.

.codeguard/codeguard.yaml

@@ -4,6 +4,7 @@ exclude:
   - .codeguard/cache.json
   - .codeguard/cache.slop-history.json
   - .codeguard/cache.perf-history.json
+  - .codeguard/cache.legibility-history.json
   - .gomodcache/**
   - tests/**/.codeguard/cache.json
   # Detection-precision fixtures: intentionally vulnerable-looking sample
@@ -77,6 +78,10 @@ checks:
         kind: file-size
         path: README.md
         max_bytes: 65536
+  context_rules:
+    # Enforceable AI-readiness gate: warn if the repo_legibility score drops
+    # below 85 (the repo currently scores 100; see docs/checks.md).
+    legibility_warn_threshold: 85
   ci_rules:
     require_workflow_dir: true
     required_workflow_files:

.gitignore

@@ -39,3 +39,4 @@ go.work.sum
 **/.codeguard/cache.slop-history.json
 **/.codeguard/cache.perf-history.json
 **/.codeguard/cache.rule-stats-history.json
+**/.codeguard/cache.legibility-history.json

CLAUDE.md

@@ -6,7 +6,7 @@ agent context — and reports pass/warn/fail findings per rule.
 
 ## Build, test, verify
 
-- `make build` → `dist/codeguard`; `make test`; `make lint` (vet) and `make lint-strict` (golangci-lint, CI-blocking, must be 0 issues)
+- `make build` → `dist/codeguard`; `make test`; `make fmt` (gofmt the tree); `make lint` (vet) and `make lint-strict` (golangci-lint, CI-blocking, must be 0 issues)
 - `make codeguard-ci` — validate + self-scan with `.codeguard/codeguard.yaml`
 - `make check` — the full CI gate (fmt-check, lint, test, codeguard-ci)
 - Scope gofmt to `gofmt -l cmd internal pkg tests changelog.go` (repo root picks up worktrees/module caches)

README.md

@@ -95,13 +95,13 @@ codeguard baseline -config codeguard.yaml -output codeguard-baseline.json
 
 `codeguard rules` prints each rule's level, execution model, language coverage, section, and title. `codeguard explain <rule-id>` includes the same metadata for a single rule.
 
-By default, `codeguard` looks for `codeguard.yaml`, `codeguard.yml`, or `codeguard.json` in the repository root. If those are missing, it also checks `.codeguard/codeguard.yaml`, `.codeguard/codeguard.yml`, and `.codeguard/codeguard.json`.
+By default, `codeguard` looks for `codeguard.yaml`, `codeguard.yml`, or `codeguard.json` in the repository root. If those are missing, it also checks for the same file names inside a `.codeguard/` directory.
 
 If you point `-config` at a directory such as `.codeguard`, `codeguard` will look inside it for `codeguard.*` or `config.*` files.
 
 Text output includes ANSI color and emoji markers by default. Set `NO_COLOR=1` if you want plain terminal output.
 
-If you want a JSON starting point instead, use [examples/codeguard.json](/Users/alex/Documents/GitHub/codeguard/examples/codeguard.json:1).
+If you want a JSON starting point instead, use [examples/codeguard.json](examples/codeguard.json).
 
 ## SDK
 
@@ -129,16 +129,16 @@ func main() {
 
 ## Docs
 
-- [Getting started](/Users/alex/Documents/GitHub/codeguard/docs/getting-started.md:1)
-- [Features](/Users/alex/Documents/GitHub/codeguard/docs/features.md:1)
-- [Security & OWASP](/Users/alex/Documents/GitHub/codeguard/docs/security.md:1)
-- [AI-generated code quality](/Users/alex/Documents/GitHub/codeguard/docs/ai-quality.md:1)
-- [Agent-native features](/Users/alex/Documents/GitHub/codeguard/docs/agent-native.md:1)
-- [Integrations](/Users/alex/Documents/GitHub/codeguard/docs/integrations.md:1)
-- [Hook-pack examples](/Users/alex/Documents/GitHub/codeguard/examples/hooks/README.md:1)
-- [SDK guide](/Users/alex/Documents/GitHub/codeguard/docs/sdk.md:1)
-- [Release automation](/Users/alex/Documents/GitHub/codeguard/docs/release-automation.md:1)
-- [Homebrew packaging](/Users/alex/Documents/GitHub/codeguard/docs/homebrew.md:1)
-- [Checks reference](/Users/alex/Documents/GitHub/codeguard/docs/checks.md:1)
-- [Architecture](/Users/alex/Documents/GitHub/codeguard/docs/architecture.md:1)
-- [Competitive roadmap](/Users/alex/Documents/GitHub/codeguard/docs/competitive-roadmap.md:1)
+- [Getting started](docs/getting-started.md)
+- [Features](docs/features.md)
+- [Security & OWASP](docs/security.md)
+- [AI-generated code quality](docs/ai-quality.md)
+- [Agent-native features](docs/agent-native.md)
+- [Integrations](docs/integrations.md)
+- [Hook-pack examples](examples/hooks/README.md)
+- [SDK guide](docs/sdk.md)
+- [Release automation](docs/release-automation.md)
+- [Homebrew packaging](docs/homebrew.md)
+- [Checks reference](docs/checks.md)
+- [Architecture](docs/architecture.md)
+- [Competitive roadmap](docs/competitive-roadmap.md)

docs/architecture.md

@@ -52,5 +52,6 @@ This keeps the runner tree organized with the same directory-first style as `int
 - waivers and inline suppressions allow time-bounded exceptions
 - baselines suppress known findings so new regressions are the only gate failures
 - cached file findings are keyed by file hash plus config hash so repeat scans skip unchanged files
+- a per-scan file corpus (`internal/codeguard/runner/support/corpus.go`) memoizes the target directory walk, individual file reads, and Go/tree-sitter parses so each file is walked, read, and parsed at most once per scan no matter how many sections inspect it; the quality AI checks share it through the `ListTargetFiles`/`ReadTargetFile`/`ParseGoFile` context hooks rather than re-reading the tree
 - diff mode can scope file findings down to changed lines derived from `git diff`
 - report serialization supports plain text, structured JSON, SARIF, and GitHub workflow annotations

docs/checks.md

@@ -940,10 +940,11 @@ Rules:
 
 Purpose:
 - Agent instruction file presence (CLAUDE.md, AGENTS.md, .cursorrules, .github/copilot-instructions.md)
-- Drift between agent docs / README commands and the actual repository
-- Agent context budget for individual source files
+- Drift between agent docs / README references and the actual repository
+- Canonical dev commands the docs never mention, and markdown links that rot
+- Agent context budget for individual source files and for the agent docs themselves
 - Basename ambiguity that defeats filename-based navigation
-- A `repo_legibility` artifact scoring how legible the repository is to AI agents
+- A `repo_legibility` artifact scoring how legible the repository is to AI agents, with an enforceable score threshold and a persisted per-scan trend
 
 Config keys:
 
@@ -957,8 +958,17 @@ Config keys:
       "detect_readme_drift": true,
       "detect_oversized_files": true,
       "detect_ambiguous_symbols": true,
+      "detect_undocumented_commands": true,
+      "detect_oversized_agent_docs": true,
+      "detect_doc_link_rot": true,
       "max_file_lines": 1500,
-      "ambiguous_symbol_threshold": 4
+      "ambiguous_symbol_threshold": 4,
+      "max_agent_doc_lines": 600,
+      "ambiguous_symbol_ignore": ["index.ts", "__init__.py"],
+      "legibility_warn_threshold": 0,
+      "legibility_fail_threshold": 0,
+      "legibility_history": true,
+      "legibility_history_limit": 100
     }
   }
 }
@@ -969,9 +979,13 @@ When `checks.context` is omitted the family runs in full scans and is skipped in
 Current behavior:
 - `context.agent-docs-missing` warns once at repo level when none of the recognized agent instruction files exist at the target root
 - `context.agent-docs-drift` warns when an agent instruction file references a file or directory path, a `make` target, or an npm/pnpm/yarn `run` script that provably does not exist
-- `context.readme-drift` applies the same resolution to fenced `bash`/`sh`/`shell` blocks in the root README.md: `./`-prefixed paths, make targets, and run scripts that resolve nowhere
+- `context.readme-drift` applies the same full extraction to the root README.md as agent docs get: prose and inline-code references as well as fenced `bash`/`sh`/`shell` blocks — the README is the doc agents and humans read first, so it earns the same truthfulness bar
 - `context.oversized-context-unit` warns when a source file exceeds `context_rules.max_file_lines` (default 1500); the message is framed as agent context cost, distinct from `quality.max-file-lines` maintainability thresholds; generated and vendored files are skipped
-- `context.ambiguous-symbol` warns once per source-file basename shared by at least `context_rules.ambiguous_symbol_threshold` files (default 4), listing up to five locations
+- `context.ambiguous-symbol` warns once per source-file basename shared by at least `context_rules.ambiguous_symbol_threshold` files (default 4), listing up to five locations; conventional basenames in the ignore list (below) never fire
+- `context.undocumented-commands` is the inverse of drift: it warns (up to 10 findings) when a high-signal Makefile target or package.json script — exactly `build`, `check`, `dev`, `fmt`, `lint`, `run`, `start`, `test`; prefixed variants like `fmt-check` are not implied — is mentioned by no agent instruction file and not even the root README. Any plausible mention counts as documentation: a structured reference (inline code, shell fence) or a plain-text `make <name>` / `npm|pnpm|yarn [run] <name>` invocation anywhere in the doc. The rule stays silent when the repo has no agent docs at all (`context.agent-docs-missing` already covers that), and when the Makefile or package.json cannot be parsed reliably (includes, pattern rules, workspaces)
+- `context.oversized-agent-doc` warns when an agent instruction file exceeds `context_rules.max_agent_doc_lines` (default 600): agent docs are loaded into every session verbatim, so an oversized one consumes the context window it exists to save; the README and linked reference docs are free to be long
+- `context.doc-link-rot` warns (up to 20 findings per doc) when a markdown link in an agent instruction file or the root README points at a repository file or directory that does not exist. Relative targets resolve against the doc's own directory and the repo root; absolute `/path` targets resolve against the repo root only (the hosted-viewer convention — an absolute filesystem path baked into a doc is exactly the rot this rule catches). External URLs (any scheme) are never fetched, pure `#anchor` links are skipped, `path.md#anchor` checks only the path part, editor-style `:line` suffixes are stripped, and templated or placeholder targets (`<name>`, `$VAR`, globs, `..` traversals, query strings) are exempt. Markdown links are owned by this rule; the drift rules no longer extract them, so one broken link is never reported twice
+- `context.legibility-threshold` enforces the `repo_legibility` score: because legibility is good-high (the inverse of the slop score), the finding fires when the computed score falls **below** a configured floor — `warn` below `context_rules.legibility_warn_threshold`, `fail` below `context_rules.legibility_fail_threshold` (the fail level takes precedence when both match). `0` (the default) disables each threshold, and when both are set the fail threshold must be less than or equal to the warn threshold. The finding message carries the full component breakdown (e.g. `agent_docs 10/25, readme 10/10, ...`) so the weakest signal is immediately visible
 
 Drift resolution is deliberately conservative — precision over recall. It only flags references it can positively prove broken, and skips:
 - URLs, module/domain paths (`github.com/...`), absolute paths, and `..` traversals
@@ -981,16 +995,37 @@ Drift resolution is deliberately conservative — precision over recall. It only
 - make targets when no root Makefile exists or the Makefile uses `include` or pattern rules
 - npm scripts when there is no root package.json or it declares workspaces
 
+Conventional-basename ignore list:
+
+Basenames imposed by a language or framework convention are expected to repeat, so they neither fire `context.ambiguous-symbol` findings nor count against the `navigability` score component. The default set:
+- JS/TS module entrypoints: `index.ts`, `index.tsx`, `index.js`, `index.jsx`, `index.mjs`, `index.cjs`
+- file-system routers: `route.ts`, `routes.ts`, `page.tsx`, `layout.tsx`
+- Python package markers: `__init__.py`, `__main__.py`
+- Rust module layout: `mod.rs`, `lib.rs`, `main.rs`
+- Go idioms: `main.go`, `doc.go`, `types.go`
+
+Setting `context_rules.ambiguous_symbol_ignore` **replaces** the default list entirely (matching is case-insensitive): re-list the defaults plus your own names to extend it, or set it to `[]` to disable ignoring altogether.
+
 `repo_legibility` artifact:
 
 Every context run publishes one `repo_legibility` artifact per target with a 0-100 score (higher is more legible) and an explainable component breakdown:
-- `agent_docs` (25): any agent instruction file present
+- `agent_docs` (25): agent instruction files must exist *and* have substance — credit is `25 x min(non-blank lines, 10) / 10` measured on the largest agent doc (an empty CLAUDE.md scores 0, full credit from 10 non-blank lines), minus 2 points per unresolvable reference inside the agent docs (capped at 10); the component detail spells out both terms
 - `readme` (10): root README.md present
-- `doc_accuracy` (20): minus 4 points per unresolvable doc/README reference
-- `context_economy` (25): scaled down by the share of source files over the context budget (10% oversized zeroes it)
-- `navigability` (20): scaled down by the share of source files caught in ambiguous basename groups (20% affected zeroes it)
+- `doc_accuracy` (20): scaled by the share of doc/README references that resolve — penalty `round(20 x broken/total)`, so 2 broken of 10 costs 4 points while a doc set that is mostly wrong loses all 20; with no references the component stays at 20
+- `context_economy` (25): scaled by the share of source files over the context budget, ramping linearly to zero at 25% oversized (penalty `round(25 x share x 4)`, capped at 25)
+- `navigability` (20): scaled down by the share of source files caught in ambiguous basename groups (20% affected zeroes it), computed after removing conventional basenames per the ignore list above
+
+The artifact is emitted even when individual rules are toggled off, so the score always reports reality. The `context.legibility-threshold` rule (above) turns the score into a warn/fail gate.
+
+Score history:
+
+The legibility score trend is persisted per target next to the scan cache (`<cache>.legibility-history.<ext>`, e.g. `cache.legibility-history.json`) whenever the cache is enabled; subsequent scans annotate the artifact with `previous_score` and `delta`. `context_rules.legibility_history: false` disables persistence and `context_rules.legibility_history_limit` caps retained entries per target (default 100). Print the recorded trend with:
+
+```bash
+codeguard report -legibility-history [-config path] [-limit n]
+```
 
-The artifact is emitted even when individual rules are toggled off, so the score always reports reality.
+(mirroring `codeguard report -slop-history` and `codeguard report -perf-history`).
 
 ## Output
 

examples/codeguard.json

@@ -137,8 +137,12 @@
       "detect_readme_drift": true,
       "detect_oversized_files": true,
       "detect_ambiguous_symbols": true,
+      "detect_undocumented_commands": true,
+      "detect_oversized_agent_docs": true,
+      "detect_doc_link_rot": true,
       "max_file_lines": 1500,
-      "ambiguous_symbol_threshold": 4
+      "ambiguous_symbol_threshold": 4,
+      "max_agent_doc_lines": 600
     }
   },
   "exclude": ["vendor/**", "**/testdata/**"],