Skip to content

docs: point every citation at the code it names, and gate citation drift - #55

Merged
laithalsaadoon merged 3 commits into
mainfrom
docs/citation-deep-pass
Oct 12, 2026
Merged

laithalsaadoon merged 3 commits into
mainfrom
docs/citation-deep-pass

Conversation

@laithalsaadoon

Copy link
Copy Markdown
Owner

Every path:line citation in docs/ now points at the code its sentence names, the claims on every published page match the code at 4fccb1e (v0.3.0), and a new gate fails the next time code moves out from under a citation.

Why

The site's citation pass (site/src/lib/citation-links.ts) checks only that a cited path exists. A moved line, a range past the end of its file, or a sentence that went stale passed every gate. On main, docs/reference/cli.md cited analyze at app.py:1240 (it's at 1634) and convert at app.py:282 (it's at 675).

Citations

All 2,177 citations on main were read against their target lines.

Outcome Count
Already right 859
Re-pointed by exact text match (the symbol, string or code the sentence quotes) 657
Re-pointed after reading the code 342
Sentence wrong about the code, rewritten 283
Removed with the claim they backed 36
Added in rewritten text 324

The pages carry 2,466 citations after the pass. A second read of 431 of them (the 311 an identifier heuristic flagged, plus 120 drawn at random) found 8 still wrong, and this PR fixes all 8. 119 of the 120 random ones were right on the second read.

Page Citations on main Changed
insights/business-logic.md 412 171
insights/contract-map.md 288 167
insights/impact-analysis.md 210 137
architecture/module-map.md 162 76
behavior/processes.md 153 118
insights/debugging-guide.md 150 117
reference/cli.md 122 106
analysis/dead-code.md 106 70
diagrams/behavioral/sequences.md 77 51
analysis/risk-hotspots.md 75 49
diagrams/architecture/components.md 75 46
behavior/state-machines.md 66 35
insights/tech-debt.md 66 43
diagrams/structural/dependency-graph.md 65 27
architecture/data-flow.md 52 40
reference/public-api.md 50 26
architecture/system-overview.md 48 39

Claims corrected, by page

  • reference/cli.md: every command and flag cites its def and parameter lines. Adds the flags the sections omitted (materialize --lake/--no-lake; embed --install-extension, --prune-orphans, --lake/--no-lake; corpus slim --format), merges a duplicated exit-code paragraph, and completes the exit codes (77, the lake_mismatch, sandbox_refused, extension_missing and lake_unavailable kinds; 127 is retired). The console script is declared in the root pyproject.toml; embed and search call Bedrock only with the default cohere provider; search returns a corpus column; cron has a compact lane.
  • reference/public-api.md: signatures of build_plan, convert_and_audit, materialize, register, run_analyze and scan_source_root; the ConversionOutput, EmbedSettings, main and register descriptions.
  • architecture/module-map.md, system-overview.md: the vendored harbor models are 0.24.0, and no src/ module imports harbor; sections aren't ordered by size, and atif-converter is the largest member; four atif-cli adapter modules import siblings at module scope; atif-cli pins five siblings; the lake and corpus sub-apps; the local EmbeddingGemma 2 provider; only analyze is a dry run by default; five members carry a layers contract; the published wheel builds with hatchling; cyclopts>=5.1.1.
  • architecture/data-flow.md: the artifact write order, and no harbor call at materialize time.
  • behavior/state-machines.md, processes.md: a session whose source vanished is kept and marked source_present: false, never deleted; materialize's write step has no ArtifactProducer or --no-columnar, and the DuckLake sink step is added; the loss report is built after enrichment; status calls preview_pass.
  • diagrams/structural/dependency-graph.md: pydantic comes from atif-converter, pydantic-settings and pyarrow from atif-duck, the sibling pins are ==0.3.0, and zstandard and imagesize are listed.
  • diagrams/architecture/components.md, behavioral/sequences.md: run_analyze is at analyze.py:37; spec_for and classify_structured are methods; the materialize participant is our converter, not harbor; query registers authorship and attaches the lake by default; the lance extension is loaded there, installed only by embed.
  • insights/business-logic.md: drops the rule for a retired harbor drift error; side files are read in place; the embed backfill isn't a dry run by default; EXIT_CODES is the only exit-code table; only the retry queue checks PIPELINE_NAMES; a dozen smaller rules match the code.
  • insights/contract-map.md: stored artifacts are .zst, and the lake loader writes five parquet files; the embedding ports gain the Gemma provider, LakeTextRows, discovery and commit; ConverterPort.convert takes archive_dir and returns six fields; five unlisted Protocols are named.
  • insights/impact-analysis.md: harbor_version is the vendored UPSTREAM_VERSION; the refresh script, not atif_cli.cron, suppresses retries on 78; convert writes with --trajectory-out; gh release create --verify-tag runs in release.yml.
  • insights/debugging-guide.md: an empty scan exits 78 with suspicious_scan; a corrupt watermark logs a warning and re-converts; ATIF_SQL_LOG_LEVEL exists; the exit-code table matches EXIT_CODES.
  • analysis/dead-code.md: re-measured on this tree. Seven public definitions are never used. importlib sits at app.py:129 and gemma_local.py:115, and harbor_adapter.py has neither importlib nor getattr. parse_trajectory_step, read_trajectory_steps and read_edges_uuids are listed as exports nothing imports. DEFAULT_STEPS_CACHE_SIZE is left off, because test_resource_guards.py:177 reads it.
  • analysis/risk-hotspots.md, insights/tech-debt.md: churn, author and scanner counts re-measured (69 bandit B608 sites, 70 ruff S608); retired debt rows removed; FidelityGap has six members; max-args is set by materialize.
  • CONTRACT.md, CONTRACT-V2.md: two reserved corpus slugs; the messages columns; DETACH is refused; --format takes table; the CLI heading names atif-cli as the composing package (atif-analytics also imports atif-models); three cron lanes.
  • README.md, AGENTS.md, CONTRIBUTING.md, RELEASING.md, package READMEs, site/authored/: check has 11 gates; a Codex query needs --agent codex; release examples use 0.3.0, and the failure table quotes the messages the preflight prints; the version-files dry run keeps the tracked CHANGELOG.md; package READMEs list their real commands, views and parameters.
  • Site chrome: every synced page's meta description, the llms.txt summary and the JSON-LD site node said "Claude Code agent trajectories". They now name Codex CLI too, from one constant in site/src/lib/repo.ts.

The guard: mise run docs:citations

scripts/docs_citations.py runs inside docs:gate, on every pre-push, and in docs.yml against the pull request's base.

  • Static: a citation fails when it names a file git doesn't track, a line past the end of its file, or a shorthand with no antecedent. On main it finds 15 past-the-end citations. On this branch it finds none in 2,466.
  • Drift: each citation on a doc line the change leaves alone is mapped through its cited file's diff from the base. Moved lines fail with the new span, and mise run docs:citations:fix rewrites them. Lines that were removed, or rewritten and shifted, fail for a reader. An edit in place prints a note and passes.

Measured before building it:

Measurement Result
False positives on the corrected docs 0 of 2,466 citations (static check)
Replay of the PR #54 range, 5c906c5..4fccb1e 21 failures; the audit had to fix all 21. 14 in-place edits noted, 5 of which needed a fix
Replay over the 29 commits before this PR 19 commits fail, on 991 citations; the new line is named for 934
Runtime on this tree about 2.5 s

The other proposed guard, a check that each cited line holds an identifier from its sentence, was measured and not built. On the corrected docs it could judge 1,203 of 2,465 citations, and 304 of the 311 it flagged were correct (a 25% false-positive rate). Test, behavior and config citations rarely repeat the sentence's identifiers.

Verification

  • mise run check: 2,679 passed, 1 skipped, 1 xfailed; every gate green.
  • mise run docs:gate: docs:citations 0 problems; 20 pages and 1,980 internal links with 0 broken; astro check 0 errors; the dist probes pass, including every Mermaid diagram rendering to SVG.
  • mise run docs:prose: 0 errors. Warnings fell from 269 to 257, and .vale-baseline.json is lowered to match.
  • tests/test_docs_citations.py plants each defect. Disabling the range check or the move detection turns its tests red.

Not verified here

  • risk-hotspots.md's coverage column: it wasn't re-derived, since that needs a coverage run.
  • Dated measurements quoted on the pages: the Gemma timings, the "<50 ms" schema timing, the parity-oracle results and the RSS tables.
  • Error text that DuckDB produces, and incident dates backed only by code comments.

Separately, some code comments disagree with the code, and this PR doesn't touch code:

  • pyproject.toml says 15 SQL sites (there are 70), names search as the max-args owner (it's materialize), and says the root has no [project] table.
  • mise.toml says the VEX ledger ships empty; it has 2 statements.
  • The claude_code_converter.py header says harbor 0.23.0.

…carry

The meta description of every synced page, the llms.txt summary and the
JSON-LD site node said the site covers Claude Code trajectories only, while
the converter also reads Codex CLI rollouts. The description now lives once
in src/lib/repo.ts, in the words site/authored/index.md already uses, and
astro.config.ts and Head.astro import it.
The docs cited code by `path:line`, and the site's citation pass checked only
that each path exists. Of the 2,177 citations on main, 1,318 needed a change:
657 re-pointed by exact text match, 342 re-pointed after reading the code, 283
whose sentence was wrong about the code and is rewritten, and 36 removed with
the claim they backed. 859 were already right. The rewritten pages add 324
citations, and the pages carry 2,466 after the second read.

Claims corrected against the code include the dependency graph's package
attribution (pydantic from atif-converter; pydantic-settings and pyarrow from
atif-duck; sibling pins at ==0.3.0), the dead-code page's importlib sites,
unused exports and dead definitions, the CLI reference's flags and exit
codes, the vendored harbor version (0.24.0), the console script's manifest
(the root pyproject.toml), source-removed sessions being kept rather than
deleted, and the CONTRACT CLI heading. A second read of 431 citations (311 a
heuristic flagged, 120 at random) found 8 still wrong; all are fixed.
scripts/docs_citations.py (`mise run docs:citations`, inside `docs:gate`,
on every pre-push, and in docs.yml against the pull request's base) runs
two checks over docs/ and site/authored/.

The static check fails on a citation naming a file git does not track, a
line past the end of its file, or a shorthand with no antecedent. On main
it finds 15 past-the-end citations; on this tree it finds none.

The drift check maps each citation on a doc line the change leaves alone
through its cited file's diff from the base. Lines that moved fail with
the new span, and `mise run docs:citations:fix` rewrites them. Lines that
were removed, or rewritten and shifted, fail for a reader to re-point. An
edit in place prints a note. Replayed over the 29 commits before this
change, it fails 19 of them on 991 citations and names the new lines for
934. On the 5c906c5..4fccb1e range, all 21 citations it fails were among
those this audit had to fix.

A check that the cited line holds an identifier from its sentence was
measured and not built: on the corrected docs it can judge 1,203 of 2,465
citations, and 304 of the 311 it flags are correct (a 25% false-positive
rate), because a test, behavior or config citation rarely repeats the
sentence's identifiers.
@laithalsaadoon
laithalsaadoon merged commit 3258fcc into main Oct 12, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant