cli-std: <cli> llm content rots — derive fact sections from in-binary sources of truth and gate the rest (llms.txt becomes generated)

Problem

<cli> llm is the agent-facing entry point of every tool, yet its content
is the only fact-bearing surface in the repo with ZERO drift protection:
cli_std::llm::Topic.body is a hand-written &'static str baked at compile
time. Facts restated there (command inventories, API routes, env names,
backup destinations, feature availability) rot silently as the binary
evolves. The 2026-07-23/24 audit wave caught this exact failure class
repeatedly on sibling surfaces: backup docs claiming file/s3-only while
gs:// was unconditional (faberline/axiom#2483), rotation docs claiming restart-needed
while a 15s watcher existed (faberline/axiom#2480), a committed openapi.json stale by two
releases, a Dockerfile ARG stale by five. Every llms.txt is a third
hand-maintained copy of the same facts.

Design: derive facts, gate prose

cli-std: composed topics — extend the topic model so bodies are
sections, and fact-bearing sections are RENDERED AT CALL TIME from the
binary's own sources of truth:
```
pub enum TopicSection {
    Prose(&'static str),                          // narrative only
    Generated { id: &'static str, render: fn() -> String },
}
```
Canonical generators every CLI gets nearly for free:
- command/verb inventory → the clap command tree (already powers --help)
- API topic → the same routes_json()/openapi() the spec verb serves
- backup destinations → a SUPPORTED_SCHEMES const exported by
  service-backup (makes the tape: stale backup-destination docs claim file://+s3:// only — gs:// is unconditional and GKE-proven axiom#2483 class impossible)
- feature availability → cfg! flags; version → CARGO_PKG_VERSION
- standard env names → the same constants the serve path reads
Drift gate — a shared cli-std test helper
(llm_topics_render_and_cross_check) required by the CLI convention:
every Generated section renders non-empty; per-CLI parity asserts where
independent sources exist (llm api section ≡ spec --format routes ≡
served /openapi.json) — the proven
openapi_committed_snapshot_matches_live_generation pattern,
generalized and made mandatory.
Kill the third copy — llms.txt becomes generated output
(<cli> llm --format md > llms.txt) with a committed-snapshot parity
test, never hand-edited.
Convention — CONTRIBUTING's CLI-convention section gains the rule
(inverse of the 2026-07-24 trace-context projection rule): hand-written
llm prose MUST NOT restate machine-derivable facts; facts are Generated
sections or they do not appear.

Rollout

Order	Step
1	cli-std `TopicSection` + render support + the conformance helper (back-compat: existing static topics keep working).
2	lumen (actively being adopted) and tape migrate their fact-bearing topic content + llms.txt generation + parity tests.
3	Remaining CLIs (aw, vat, keep, relay, defer, sift, meter, guard, …) adopt per lane; the CLI convention doc makes it part of "a CLI is not done until".
4	Optional: an `aw health` docs-freshness signal once 1-3 prove the shape.

Refs: faberline/axiom#2483 / faberline/axiom#2480 (the drift class), faberline/axiom#2491 (the projection-rule sibling for capability docs), lumen Agent Task Navigation faberline/axiom#1683 (cli_std::llm::v2 — the typed-protocol precedent showing the lib already evolves this direction).

Design refinement from the owner discussion (2026-07-24):

Audience insight — the content is FOR LLMs, so machine-shaped output is preferable, not a compromise: structured dense blocks (route tables, flag inventories, scheme lists) beat polished prose for the actual consumer. The critical consumer is the agent WITHOUT repo access (an integrator's agent holding only the installed binary) — in-repo agents demonstrably bypass llm and read source directly (this session's audits never invoked it once), but out-of-repo agents have no fallback, which is exactly where staleness poisons.

Two grades of 'machine translation', pick per layer:

Facts: deterministic derivation (the issue body's TopicSection::Generated) — render from the same in-memory structures the binary already trusts (clap tree, openapi(), exported constants). Zero drift by construction; no LLM in the loop; testable. Default for everything fact-bearing.
Narrative: project from the lifecycle-gated semantic TDs, not free LLM translation. The repo ALREADY institutionalizes 'machine translation of source' — aw td create --from-source semantic mirrors, kept honest by td lock/code-check. The llm topics' narrative sections should be a projection of those TD docs (a build-time render step with a parity test), not a fourth hand-written copy and not a free-running build-time LLM summarize (nondeterministic, hallucination-prone, unreviewed). This closes the loop: source → semantic TD (gated) → llm narrative (projected) → llms.txt (generated), with exactly ONE hand-authored layer in the chain.

Build-time mechanism (owner constraint: the installed binary has NO repo access) — everything llm serves must be self-contained in the binary:

Fact sections: already runtime-safe — fn() -> String renders from in-memory structures (clap tree, openapi(), exported constants) that ship inside the binary. No repo needed.
Narrative sections: projected from the semantic TDs at BUILD time and embedded — via build.rs (read TD → distill → OUT_DIR → include!) or direct include_str! of the narrative blocks. This extends the repo's existing 'CLI is the in-binary form of the checked-in fixture' convention (k8s operator render/dockerfile render are already include_str! projections). Same-commit consistency is guaranteed by the td-lock/code-check gates that already bind TD ↔ source at commit time; the freshness/parity tests gate the release tag in CI.
Resulting property: the shipped artifact is SELF-DESCRIBING and version-bound by construction — tape@0.4.10's llm output describes exactly 0.4.10's behavior. Doc/binary version skew becomes structurally impossible, which no external doc site can offer.

Chain: commit-gated TD ↔ source → build-time embed (narrative) + compiled renderers (facts) → CI parity gate → tag → self-contained binary.

Third layer (owner follow-up): the topic TAXONOMY itself rots too — a tool grows a new subsystem and nothing forces a topic to exist for it or the outline to mention it. Design:

Topic registry with declared sources + bidirectional gates. Topics stay hand-curated (grouping/naming is taste and rots slowly) but become a thin declaration table: each TopicSpec lists its sources (clap paths, TD docs) and its sections (Generated fact renderers, Projected narratives). Two conformance gates in cli-std:

No dangling (topic → source): every declared clap path / TD file must exist — a renamed verb or moved TD reddens the build instead of leaving a topic pointing at nothing.
No holes (source → topic): every top-level clap verb and every capability-doc entry must be CLAIMED by at least one topic — a new subsystem landing without llm coverage fails the gate. This is 'keeping up with the times' made structural at the taxonomy layer.

The default outline topic becomes fully generated from the registry (topic index + verb inventory), so it is complete by construction.

Final shape — hand-authored content shrinks to exactly two slow-rotting places, both gated: TD narrative bodies (td-lock/code-check) and topic grouping/naming (bidirectional coverage gates). Facts derived, narrative projected, taxonomy declared-and-covered.

Deprioritized by owner decision (2026-07-24) — for integration specifically, the load-bearing surfaces are already machine-derived and gated: the contract surface is <cli> spec (offline OpenAPI ≡ served ≡ generated clients, snapshot-parity tested; the one route drift found, faberline/axiom#2482, is fixed and gated) and the debugging surface is structured logs + traceparent correlation. llm narrative staleness costs agent onboarding speed, not integration correctness.

Keep this issue as the recorded design (three-layer: derived facts / TD-projected narrative / declared-and-covered taxonomy) for when it becomes load-bearing — the trigger condition is out-of-repo AGENT-DRIVEN CLI usage at scale (agents holding only the installed binary and relying on llm to drive it), not human-adjacent integration against the HTTP contract. No implementation scheduled.

Re-prioritized (owner decision, same day): the trigger condition is already met — an integrator is actively consuming the installed lumen binary (faberline/axiom#2456), so their agents' only self-documentation surface is lumen llm today, not someday.

Execution plan (MVP first, targeting lumen@0.4.25):

Phase 1 — cli-std facts layer (dispatching now): TopicSection::{Prose, Generated} with call-time rendering, back-compat with existing static topics, plus the llm_topics_render_and_cross_check conformance helper.
Phase 2 — lumen adoption (queued behind the in-flight lumen: collections become unreadable after a CONVERGED auto-split — post-cutover fence reports converged while search returns collection not_found (GKE, 2 runs) axiom#2489 fix to avoid worktree contention): migrate lumen's fact-bearing topic content to Generated sections (verb inventory from the clap tree, API topic from the same routes/openapi the spec verb serves, backup destination schemes from service-backup, feature/version from cfg!/CARGO_PKG_VERSION), generate llms.txt from llm --format md with a committed-snapshot parity test.
Phase 3 — taxonomy registry + TD-projected narrative: follows separately; design recorded above.

Ships with lumen@0.4.25 alongside the faberline/axiom#2489 fix so the integrator's next upgrade carries both.

feat(cli-std): call-time-generated llm topic sections with a freshnes…

test(lumen): DX contract command-template flags stay live against the…

Lib-layer work complete (2026-07-24). Phase map:

Phase	Status
P1 `TopicSection::{Prose,Generated}` + `render_sectioned` + `assert_topics_render`	landed `029669f6c2` (shipped lumen@0.4.25)
P2 call-time Generated machinery + freshness helper	landed same commit
Lib enabler: service-backup `SUPPORTED_SCHEMES` (`SchemeInfo` with `cfg!`-derived availability) + `sectioned_topic()`	landed `c7ee43c8ba` — also fixed TWO hand-coded scheme lists inside service-backup itself (`from_uri` error + `fetch_backup_object` bail), the exact faberline/axiom#2483 drift class
P3 taxonomy registry	deferred per the issue's own scoping

Back-compat: service_backup::llm::topic() byte-identical (lumen's spec.rs compiles unchanged); CLIs migrate to sectioned_topic() at their own pace. Tests: service-backup 19/19 (+s3 21/21), cli-std 35/35, clippy -D warnings clean.

Remaining before close: per-CLI adoption (lumen/beam/keep/defer/relay/tape wire sectioned topics + the drift gate; cross-app batch) and the llms.txt-generated piece where applicable.

feat(cli-std): derive service-backup scheme facts from SUPPORTED_SCHE…

feat(cli-std): call-time-generated llm topic sections with a freshnes…

test(lumen): DX contract command-template flags stay live against the…

feat(cli-std): derive service-backup scheme facts from SUPPORTED_SCHE…

fix(tape): derive backup-destination docs from SUPPORTED_SCHEMES — gs…

feat(tape): wire the sectioned backup llm topic into the tape llm verb (

fix(tape): derive backup-destination docs from SUPPORTED_SCHEMES — gs…

feat(tape): wire the sectioned backup llm topic into the tape llm verb (

Goal

Finish the part of #9 that lives in core: the llms.txt snapshot-parity helper and the written convention. The TopicSection model, render_sectioned, and assert_topics_render are already in cli-std (see the lib-layer comment on #9). What is missing is (a) a shared helper that checks a committed llms.txt equals the CLI's own llm --format md output, so the third hand-kept copy cannot drift, and (b) the rule in crates/cli-std/CONTRIBUTING.md that llm prose must not restate machine-derivable facts. Per-CLI adoption in other repos is out of scope here.

Acceptance

crates/cli-std/src/application/llm.rs gains pub fn assert_llms_snapshot_matches<T: RenderableTopic>(project: &str, version: &str, topics: &[T], committed: &str): it renders every topic the same way the md llm output does and panics with a message naming the first differing line (or the first missing/extra topic) when the result differs from committed. Trailing-whitespace-only differences at end of file are ignored.
It is re-exported next to assert_topics_render in crates/cli-std/src/api/llm.rs.
Unit tests in crates/cli-std/src/application/llm/tests.rs cover: a matching snapshot passes; a snapshot with one changed line panics and the message names that line; a snapshot missing a topic panics; works for both &[Topic] and &[SectionedTopic].
crates/cli-std/CONTRIBUTING.md gains a "CLI llm convention" section that states: fact-bearing llm content is a Generated section or does not appear; llms.txt is generated from <cli> llm --format md and checked with assert_llms_snapshot_matches; every CLI calls assert_topics_render. The section links the README capability, not a design tree.
crates/cli-std/README.md ## Capabilities mentions the new helper if it lists assert_topics_render; otherwise unchanged.

Never

Do not edit any llms.txt file, including crates/cli-std/llms.txt.
Do not change the output of render, render_sectioned, Topic, or SectionedTopic (callers in service-backup and downstream CLIs pin them), and do not rename existing public items.
Do not touch other crates, .github/, or the root CONTRIBUTING.md.
Do not add a dependency.
Do not build the topic registry or the TD-projected narrative (deferred in cli-std: <cli> llm content rots — derive fact sections from in-binary sources of truth and gate the rest (llms.txt becomes generated) #9).

Gate

cargo test -p cli-std --lib
cargo test -p service-backup --lib
cargo clippy -p cli-std --all-targets -- -D warnings
cargo fmt --check -p cli-std

Scope of this spec

Phase 1 of this issue is already on main: TopicSection, SectionedTopic, render_sectioned, and assert_topics_render (crates/cli-std/src/application/llm.rs), and service-backup already derives its destinations from SUPPORTED_SCHEMES. This spec covers the part that still lives in this repository: the committed-snapshot parity helper (design point 3) and the convention rule (design point 4). Migrating individual CLIs and generating their llms.txt happens in the downstream repos and is out of scope here.

Goal

Give every CLI a shared, mandatory way to prove its committed llms.txt equals the output of <cli> llm --format md, and write down the rule that hand-written llm prose must not restate machine-derivable facts. Without the helper each CLI has to invent its own parity test, which is how the third copy of the facts rots.

Acceptance

crates/cli-std/src/application/llm.rs gains pub fn assert_snapshot_matches(committed: &str, generated: &str, regen_hint: &str). It compares the two strings after normalizing line endings (\r\n -> \n) and a single trailing newline. On a mismatch it panics with a message that names the first differing line number and includes regen_hint (the command that regenerates the file).
The helper is re-exported from crates/cli-std/src/api/llm.rs next to assert_topics_render, and the module doc of that file mentions it.
Unit tests in crates/cli-std/src/application/llm/tests.rs: equal strings pass; strings that differ only in line endings or one trailing newline pass; a differing line panics with the line number and the hint (use catch_unwind like conformance_helper_catches_empty_generated_section).
crates/cli-std/CONTRIBUTING.md gains a ## CLI llm convention section stating: (1) fact-bearing content is a TopicSection::Generated section or it does not appear; (2) hand-written prose must not restate machine-derivable facts; (3) llms.txt is generated output, checked by assert_snapshot_matches, never hand-edited; (4) each CLI's tests call assert_topics_render and assert_snapshot_matches.
crates/cli-std/README.md evidence line is unchanged except for naming the new helper where llm is listed, if that list names helpers; otherwise leave it alone.

Never

Do not change Topic, SectionedTopic, TopicSection, render, render_sectioned, or the golden output bytes tests.
Do not edit crates/cli-std/llms.txt (cli-std has no llm command to generate it from).
Do not touch any other crate or any downstream CLI.
Do not add a dependency to crates/cli-std/Cargo.toml.
Do not change the existing assert_topics_render signature or messages.

Gate

cargo test --locked -p cli-std
cargo clippy --locked -p cli-std --all-targets -- -D warnings
cargo doc --locked -p cli-std --no-deps

Work submitted in #72; waiting for required CI on the current head.

Navigation Menu

Repository navigation

cli-std: <cli> llm content rots — derive fact sections from in-binary sources of truth and gate the rest (llms.txt becomes generated) #9

Problem

Design: derive facts, gate prose

Rollout

Goal

Acceptance

Never

Gate

Scope of this spec

Goal

Acceptance

Never

Gate

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

cli-std: <cli> llm content rots — derive fact sections from in-binary sources of truth and gate the rest (llms.txt becomes generated) #9

Description

Problem

Design: derive facts, gate prose

Rollout

Activity

chrischeng-c4 commented on Jul 24, 2026

chrischeng-c4 commented on Jul 24, 2026

chrischeng-c4 commented on Jul 24, 2026

chrischeng-c4 commented on Jul 24, 2026

chrischeng-c4 commented on Jul 24, 2026

chrischeng-c4 commented on Jul 24, 2026

chrischeng-c4 commented on Oct 23, 2026

Goal

Acceptance

Never

Gate

chrischeng-c4 commented on Oct 24, 2026

Scope of this spec

Goal

Acceptance

Never

Gate

chrischeng-c4 commented on Oct 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions