diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 87cf761..0f43b0a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -153,15 +153,21 @@ jobs: shell: bash lint-invariants: - name: Shell invariants + name: Repo invariants runs-on: ubuntu-latest steps: + # fetch-depth: 0 for the tags, not the history. The version-claim + # check compares the roadmap's Shipped table against the newest + # release tag, and the default shallow checkout carries no tags -- + # which would leave that half reporting a skip CI never reads. - uses: actions/checkout@v7 + with: + fetch-depth: 0 - # Properties shellcheck has no opinion about, each added because the - # pattern it rejects shipped here and was found by reading rather - # than by any test. - - name: Check shell invariants + # Properties no other tool has an opinion about, each added because + # the pattern it rejects shipped here and was found by reading + # rather than by any test. + - name: Check repo invariants run: ./tests/lint/run-tests.sh preflight-static: diff --git a/CLAUDE.md b/CLAUDE.md index 3e0853b..66cbbd2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -327,7 +327,7 @@ Per-suite requirements: | audit-chain | bash, sha256sum | | docs-index | bash, awk, diff | | cloud-preflight | bash | -| lint | bash, python3 | +| lint | bash, python3; a checkout carrying its tags | | preflight-static | bash, python3; shellcheck if present | | pki | bash, jq, openssl | | pki-migration | bash, jq, python3, openssl | @@ -386,6 +386,14 @@ CI enforces several invariants worth knowing before you push: -rf "$D"; }` returns 1 when the test is false, and bash applies that to the script's exit status from an `EXIT` trap — a successful run reports failure, and an explicit `exit 0` does not save it. +- **The README's version claim must match the roadmap's Shipped table** + (`tests/lint/check_version_claim.py`), and that table must be no older + than the newest git tag. The table is the authority rather than the + tags because a release lands its roadmap row in a PR and is tagged + after it merges — gating on tags would fail that PR for being correct. + The tag half is one-directional for the same reason: ahead is fine, + behind is a release nobody wrote down. This is why the lint job checks + out with `fetch-depth: 0`. - **Trivy findings at HIGH or above fail the build.** Accepted findings go in `.trivyignore.yaml` *with the reason* — an unjustified suppression is indistinguishable from never having run the scanner. diff --git a/README.md b/README.md index 0125dc3..2cb839e 100644 --- a/README.md +++ b/README.md @@ -441,7 +441,7 @@ See [`.github/workflows/ci.yml`](.github/workflows/ci.yml). ## Roadmap -Everything through v0.14 has shipped — see +Everything through v0.18 has shipped — see [Releases](https://github.com/sethbergman/vault-reference-platform/releases) for the log. "Shipped" here means there is a test that fails if the feature breaks, not that the code exists. diff --git a/docs/roadmap.md b/docs/roadmap.md index 746361a..ff3fe2b 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -554,5 +554,40 @@ refused. The assertion failed, correctly, because it greps the specific refusal rather than checking the exit code. Had it only checked that the script exited non-zero, it would have gone green while testing nothing. +A fifth is not about an assertion at all, which is why it survived four +releases. + +The Roadmap section of the README opened "Everything through v0.14 has +shipped". v0.18 had. Four PRs each added a row to the Shipped table at +the top of this file and left that sentence alone — and it is the first +claim a reader meets, in the one section whose entire purpose is to say +what is done and what is not. + +Nothing caught it because nothing was looking. `tests/docs-index` checks +that every file in `docs/` is linked from the generated index and skips +`README.md` by name; markdownlint has opinions about the line and none +about the fact on it. The question above still finds it, but only when +asked of the claim rather than of the test: what would have to break for +that sentence to fail? Nothing could. There was no weak assertion here — +there was no assertion. + +So this is the argument for generating `docs/README.md`, arriving a +second time somewhere too small to generate. One sentence does not earn a +generator, so it is asserted instead. +[`tests/lint/check_version_claim.py`](../tests/lint/check_version_claim.py) +requires the README to name the newest row of the table above, and +requires that row to be no older than the newest git tag. The tag half is +one-directional on purpose: a release lands its roadmap row in a PR and +is tagged after that merges, so gating the README on tags would fail the +PR for being correct, while a table *behind* the tags is a release nobody +wrote down. Six mutations were watched to fail, including the one the +README half cannot see — deleting the v0.18 row while v0.18 is tagged. + +The four earlier entries share a shape: a test existed and was weaker +than it looked. This one is different in a way worth separating, because +the fix is different. The table above was right the whole time; what +drifted was a hand-written summary of it. A weak assertion gets widened. +An unchecked claim gets tied to whatever is already correct. + None of this changes what the table above claims. It changes how much the word "tested" in it is worth, which seemed worth writing down. diff --git a/tests/lint/check_version_claim.py b/tests/lint/check_version_claim.py new file mode 100644 index 0000000..e2eed09 --- /dev/null +++ b/tests/lint/check_version_claim.py @@ -0,0 +1,155 @@ +#!/usr/bin/env python3 +"""Reject a README version claim that has drifted from the roadmap. + +README.md carries one sentence naming the newest shipped release. It is +the first claim a reader meets, and nothing checked it: at v0.18 it still +said v0.14, four releases stale, and had been wrong through four PRs that +each updated the roadmap table beside it. + +This is the failure docs/README.md is generated to avoid -- a +hand-maintained claim that stops being true silently, invisible to +everyone except the reader who relied on it. One sentence is not worth +generating, so it is asserted instead. + +TWO CHECKS, AND WHY THE SOURCE OF TRUTH IS NOT THE TAGS + + 1. The README claim must name the newest row of the roadmap's Shipped + table. + 2. That row must be at least as new as the newest git tag. + +The roadmap table is the authority rather than `git tag` because of the +order releases are cut in: the roadmap row lands in a PR, and the tag is +pushed after it merges. Gating the README on tags would fail that PR for +being correct. Gating the roadmap on tags in one direction only -- table +ahead of tags is fine, table behind them is not -- permits that ordering +and still catches a release that was tagged and never written down. + +Check 2 needs the tags to be present. CI's lint job fetches them +deliberately; see .github/workflows/ci.yml. A checkout with no tags is +reported as a failure rather than skipped, because a skip here is +indistinguishable from a pass and this file exists because something +unchecked went stale. + +Exits non-zero and says which claim disagrees with which. +""" + +import re +import subprocess +import sys + +# "Everything through v0.14 has shipped" -- anchored on the load-bearing +# words rather than the whole sentence, so rewording the prose around it +# is fine and removing the claim is not. +CLAIM = re.compile(r'through (v\d+\.\d+) has shipped') + +# A row of the Shipped table: | v0.18 | Rate limit quotas, ... | +ROW = re.compile(r'^\|\s*(v\d+\.\d+)\s*\|') + +TAG = re.compile(r'^v\d+\.\d+$') + + +def key(version): + """Sort v0.9 below v0.10, which a string compare does not.""" + return tuple(int(part) for part in version.lstrip('v').split('.')) + + +def read(path): + with open(path, encoding='utf-8') as handle: + return handle.read() + + +def shipped_versions(text): + """Every version in the roadmap's `## Shipped` table, that table only.""" + lines = text.split('\n') + try: + start = lines.index('## Shipped') + except ValueError: + return [] + + found = [] + for line in lines[start + 1:]: + if line.startswith('## '): + break + match = ROW.match(line) + if match: + found.append(match.group(1)) + return found + + +def newest_tag(): + """The newest vN.N tag, or None when this is not a git checkout. + + Returns [] rather than None for a checkout that is a repository but + carries no tags -- the caller treats those differently. + """ + try: + out = subprocess.run( + ['git', 'tag', '--list'], + capture_output=True, text=True, check=True, + ).stdout + except (subprocess.CalledProcessError, FileNotFoundError): + return None + + tags = [t for t in out.split('\n') if TAG.match(t.strip())] + return max(tags, key=key) if tags else [] + + +def main(): + failures = [] + + shipped = shipped_versions(read('docs/roadmap.md')) + if not shipped: + print('No `## Shipped` table found in docs/roadmap.md, or no version') + print('rows in it. This check reads that table as the source of truth,') + print('so it cannot run -- fix the table or update this checker.') + return 1 + + newest = max(shipped, key=key) + + claims = CLAIM.findall(read('README.md')) + if not claims: + failures.append( + 'README.md names no shipped version.\n' + ' Expected a sentence matching "through vN.N has shipped".\n' + ' If the wording changed deliberately, update CLAIM in this file\n' + ' -- do not delete the claim, it is what a reader trusts first.' + ) + else: + stale = sorted({c for c in claims if c != newest}, key=key) + if stale: + failures.append( + 'README.md claims %s; docs/roadmap.md ships %s.\n' + ' The roadmap table is the source of truth. Update the README.' + % (', '.join(stale), newest) + ) + + tag = newest_tag() + if tag is None: + print('Not a git checkout; skipping the roadmap-versus-tag check.') + elif tag == []: + failures.append( + 'No vN.N git tags found.\n' + ' This check compares the roadmap table against the newest tag,\n' + ' and a checkout with no tags cannot answer that. Run\n' + ' `git fetch --tags`; CI fetches them with fetch-depth: 0.' + ) + elif key(newest) < key(tag): + failures.append( + '%s is tagged but docs/roadmap.md stops at %s.\n' + ' A release was cut without a Shipped row. Add it.' + % (tag, newest) + ) + + if failures: + print('Version claims disagree:') + for item in failures: + print(' %s' % item) + return 1 + + print('README names %s; roadmap ships %s; newest tag %s.' + % (newest, newest, tag if tag else 'n/a')) + return 0 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/tests/lint/run-tests.sh b/tests/lint/run-tests.sh index 9630d88..10b4735 100755 --- a/tests/lint/run-tests.sh +++ b/tests/lint/run-tests.sh @@ -1,7 +1,6 @@ #!/usr/bin/env bash # -# run-tests.sh — Invariants about shell scripts that shellcheck does not -# enforce +# run-tests.sh — Repo-wide invariants no other tool has an opinion about # # Usage: # ./tests/lint/run-tests.sh @@ -13,7 +12,11 @@ # cheap, they are repo-wide, and they fail the build with an explanation # rather than leaving the next person to rediscover the same thing. # -# Requirements: bash, python3 +# The first was about shell, which is what this suite was named for. +# Prose drifts the same way and the second check is about that, so the +# scope is the invariant rather than the language it is written in. +# +# Requirements: bash, python3, and a checkout carrying its tags set -uo pipefail @@ -55,6 +58,26 @@ else bad "no trap handler returns the result of a bare test" "$OUT" fi +# --------------------------------------------------------------------------- +printf ' +=== The README names the release the roadmap actually ships === +' +# --------------------------------------------------------------------------- +# README.md carries one sentence naming the newest shipped release, and +# nothing checked it: at v0.18 it still said v0.14. Four PRs had updated +# the roadmap table directly beside it and left the sentence alone, which +# is the failure docs/README.md is generated to avoid — a hand-maintained +# claim that stops being true silently. One sentence is not worth +# generating, so it is asserted instead. +# +# The checker also requires the roadmap table to be no older than the +# newest git tag, which is why this suite now wants a checkout with tags. +if OUT="$(python3 "${SCRIPT_DIR}/check_version_claim.py" 2>&1)"; then + ok "README, roadmap and tags agree on the newest release" +else + bad "README, roadmap and tags agree on the newest release" "$OUT" +fi + printf '\n' printf 'passed: %d failed: %d\n' "$PASS" "$FAIL" [[ "$FAIL" -eq 0 ]] || exit 1