Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1 +1,6 @@
**/testdata/** export-ignore

# The assurance goldens and fixtures are compared byte for byte, so they are
# checked out exactly as committed. Without this a Windows checkout rewrites
# their line endings and every golden comparison fails there.
internal/assurance/testdata/** -text
Comment thread
bomly-guy marked this conversation as resolved.
Resolved
11 changes: 8 additions & 3 deletions .github/workflows/assurance-prerequisites.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,12 @@ jobs:
uses: ./.github/workflows/fuzz.yml
with:
ref: ${{ inputs.ref || github.ref }}
fuzztime: 45s
# Thirty targets one after another took over half an hour and made this
# the slowest job in the stage. Four at a time for thirty seconds each
# is the same order of total fuzzing in a fraction of the wall clock;
# the nightly run keeps the long budget.
fuzztime: 30s
jobs: 4
assurance: true
assurance_tag: ${{ inputs.assurance_tag }}

Expand Down Expand Up @@ -98,10 +103,10 @@ jobs:
run: |
set -euo pipefail
exit_code=0
summary="The assurance catalog is valid and every fixture and expected-result file it names still matches its recorded checksum."
summary="The assurance catalog is valid and every fixture and expected-result file it names exists."
if [ "${CATALOG_OUTCOME}" != "success" ]; then
exit_code=1
summary="The assurance catalog is invalid or one of the files it names has changed. Open the validation step for the exact entry."
summary="The assurance catalog is invalid or names a file that is not there. Open the validation step for the exact entry."
fi
checks="$(grep -c '"stage":' docs/assurance/catalog.json || true)"
# --stage and --level are explicit: when the catalog is the thing
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/fuzz.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ on:
type: string
required: false
default: 2m
jobs:
description: How many fuzz targets to run at once.
type: number
required: false
default: 1
assurance:
description: Upload a release assurance check result.
type: boolean
Expand Down Expand Up @@ -62,6 +67,7 @@ jobs:
env:
FUZZ_RESULTS_JSONL: ${{ runner.temp }}/fuzz-details.jsonl
FUZZTIME: ${{ inputs.fuzztime || '2m' }}
FUZZ_JOBS: ${{ inputs.jobs || 1 }}
run: make fuzz FUZZTIME="${FUZZTIME}"

- name: Record the fuzz result
Expand Down
20 changes: 1 addition & 19 deletions .github/workflows/update-smoke-goldens.yml
Original file line number Diff line number Diff line change
Expand Up @@ -319,14 +319,6 @@ jobs:
- name: Check out repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Set up Go
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version-file: go.mod
# No cache writes: this job runs with write permissions, and a
# poisoned cache would reach the jobs that trust it.
cache: false

- name: Download regenerated goldens
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
Expand Down Expand Up @@ -357,16 +349,6 @@ jobs:
--jq '[.jobs[] | select(.name | startswith("Regenerate (")) | select(.conclusion == "failure" or .conclusion == "cancelled" or .conclusion == "timed_out") | .name] | join(", ")')"
echo "failed=${failed}" >> "$GITHUB_OUTPUT"

- name: Refresh the catalog checksums
# Public evidence claims record the checksum of the golden file that
# backs them. Regenerating a golden without refreshing the catalog
# would fail the catalog-valid gate and block the next release.
if: steps.changes.outputs.changed == 'true'
shell: bash
run: |
set -euo pipefail
go run ./internal/assurance/cmd catalog-validate --refresh

- name: Create pull request with updated goldens
if: steps.changes.outputs.changed == 'true'
env:
Expand All @@ -387,7 +369,7 @@ jobs:

git checkout -b "${branch_name}"

git add test/smoke/testdata/golden docs/assurance/catalog.json
git add test/smoke/testdata/golden
git commit -m "test: update smoke golden files"
git push --set-upstream origin "${branch_name}"

Expand Down
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -354,7 +354,7 @@ Smoke tests (`test/smoke/`, `make smoke`) drive the built binary end-to-end agai
- A golden must never carry a host architecture. `make smoke ARGS="-update"` on an arm64 laptop writes a golden CI cannot match, so `normalizeJSON` erases the architecture and `TestGoldensCarryNoHostArchitecture` (untagged, runs in `make test`) fails if one reaches a committed golden. A new architecture-bearing shape means teaching the normalizer, not excepting the guard.
- Register new tests in both slice matrices (`smoke.yml` and exactly one slice in `update-smoke-goldens.yml`); `go test -run` elements are unanchored regexes — use `$` anchors to keep slice ownership exact.
- A new slice also needs an entry in the `smoke` check's `expected_instances` in `docs/assurance/catalog.json` (with its `ecosystems`, which is what puts an ecosystem on the report's coverage list). `TestCatalogSmokeInstancesMatchWorkflowMatrix` fails when the two drift.
- Regenerating goldens invalidates the checksums the catalog's claims pin. `Update Smoke Goldens` runs `catalog-validate --refresh` and commits the catalog with them; do the same when refreshing by hand.
- Regenerating goldens needs no catalog change. A claim names its expected-result file by path only; git already records which bytes a release had, so there is no stored checksum to fall behind. Renaming or deleting a file a claim names does fail `TestRepositoryCatalogIsValid` in `make test`.
- `TestExamplePluginFixtureCompiles` runs in `make test` and must keep compiling against the pinned `bomly-dev/bomly-sdk` release; update the fixture source when the SDK contract changes.

## Release assurance
Expand All @@ -370,7 +370,7 @@ Rules:
- Every check writes one `bomly.assurance-check/v1` document per instance through `go run ./internal/assurance/cmd` (`emit`, `gotest`, `convert`, or `verify-release`) and uploads it as an `assurance-*` artifact. Never hand-write that JSON in a workflow.
- Adding a check means adding a catalog entry **and** emitting its result; a declared check with no result is reported as `missing` and blocks its stage when it is a gate.
- `proves` and `limitations` are mandatory, public, and written in plain language — they are rendered on bomly.dev/assurance.
- Public evidence claims live in the same catalog (`evidence[]`), keep their pinned Git revisions and checksummed artifacts, and name the check that backs them. `make assurance-catalog` re-hashes every file they reference.
- Public evidence claims live in the same catalog (`evidence[]`), pin what this repository's history cannot address (a Git input by revision, a container by digest), name their fixture and expected-result files by path, and name the check that backs them. `make assurance-catalog` confirms every file they name exists. Do not add a checksum of an in-repository file to the catalog: it is a second copy of what git knows, and it can only drift.
- The per-release report (`docs/assurance/reports/<tag>.json`) and `docs/assurance/index.json` are written by the post-release assessment and are the only data source for the public page.

See [`dev-docs/RELEASE_ASSURANCE.md`](dev-docs/RELEASE_ASSURANCE.md) for the contracts and how to add a check.
Expand Down
44 changes: 34 additions & 10 deletions dev-docs/RELEASE_ASSURANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,11 +142,18 @@ check `missing` — that is the intended behavior: gaps are loud.

Evidence claims are the public "we prove X, we do not prove Y" statements that
used to live in `test/evidence/cases.json`. They keep the same rigor: pinned
Git revisions, checksummed fixtures and expected-result files, explicit
reproduction commands, and mandatory limitations. `make assurance-catalog`
verifies every hash a claim names, so a golden file cannot drift away from the
claim it supports, and `--refresh` rewrites them when the goldens are
deliberately regenerated.
Git revisions and container digests, explicit reproduction commands, and
mandatory limitations.

Fixture and expected-result files are named by path, with no checksum. A file
in this repository is already content-addressed by git — a release tag and a
path identify exact bytes, and the report records the commit — so a checksum
stored in the catalog was a second, hand-kept copy of something git knows. It
could only fall behind, and it did on every golden update: two pull requests
that each passed CI left `main` with a catalog that failed its own gate. What
is checked now is what can actually go wrong: `make assurance-catalog`, and
`TestRepositoryCatalogIsValid` in `make test`, fail when a claim names a file
that was renamed or deleted.

A claim must carry a pinned input **and** a committed artifact. That is the line
between the two layers: if a statement would only restate what its check already
Expand Down Expand Up @@ -203,6 +210,25 @@ versions are a contract with another repository.
`TestSchemaVersionsArePinned` fails on any change to these strings, so raising
one is always deliberate.

A version names a shape someone already holds. For the report and index that
starts with the first report committed to `docs/assurance/reports/`: from then
on bomly.dev has mirrored a document of that shape and the rules above apply
without exception. Before any report exists there is nothing to stay
compatible with, and the shape can still be corrected under the same version —
which is how the per-file checksums were removed from report claims before the
first release was assessed.

Nothing checks committed reports against the current shape automatically, and
that is deliberate. A test that did so was written and removed in the same
pull request: each review round found another gap in it, and comparing
published files byte for byte would have failed on a Windows checkout — in the
portable suite, which gates a release. The rule is held by
`TestSchemaVersionsArePinned`, by the report goldens (any change to the shape
shows up as a golden diff in review), and by the reviewer. If a mechanical
guard is wanted later, build it as its own change and compare the report's
JSON schema with the one recorded at the first published release, rather than
comparing committed bytes.

## What the automation needs

- The Bomly Release app needs **Issues: Read and write** on `bomly-cli` for the
Expand All @@ -214,11 +240,9 @@ one is always deliberate.
`assurance-prerequisites.yml` would miss every stage that Auto Version
triggered. Keep that job name stable, or update the lookup in `release.yml`
and `assurance-assessment.yml` with it.
- Regenerating smoke goldens changes files the evidence claims are pinned to.
`Update Smoke Goldens` runs `catalog-validate --refresh` and commits the
catalog alongside them; do the same when refreshing goldens by hand.
That workflow regenerates from the ref it was dispatched on, so the refresh
runs the same commit's tooling against that commit's goldens.
- Regenerating smoke goldens needs no catalog change; claims name their
expected-result files by path only. Do not reintroduce a stored checksum of
an in-repository file.

## Related documents

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,25 @@ actually download.
most prone to drift run before a tag exists.
- Workflow files are no longer pinned by checksum in the catalog. Pinning the
file proved only that the file had not changed; the per-release check result
proves the workflow ran and what it found. Fixture and expected-result files
proves the workflow ran and what it found. *(The rest of this point is
superseded; see the amendment below.)* Fixture and expected-result files
are still checksummed, because a claim about a golden file is only as good as
that file — and regenerating goldens therefore has to refresh the catalog
(`catalog-validate --refresh`, wired into `Update Smoke Goldens`).

> **Amended 2026-10-03:** the catalog no longer stores a checksum for any
> file in this repository, and `--refresh` is gone. The argument above for
> workflow files applies to goldens too: git already addresses a file by
> tag and path, so the stored checksum was a hand-kept copy that could only
> fall behind. It did, the first time the prerequisites stage ran on `main`
> — a golden update and the catalog each passed CI separately and failed
> together, which no earlier check could have caught and a refresh step only
> covers when the right workflow makes the change. A claim now names its
> files by path; what is verified is that they exist. Git inputs stay pinned
> by revision and containers by digest, because this repository's history
> cannot address those. The report keeps its schema version: no report had
> been published when the field was removed, so no reader held the old
> shape. The versions bind from the first published report onward.
- Reports are committed to the default branch rather than attached to the
release, because the exhaustive stage runs after publication and GitHub
releases are immutable once published.
Expand Down
6 changes: 3 additions & 3 deletions docs/ASSURANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ The report uses one vocabulary, and it is worth being precise about it:

Some claims are asserted directly by a check. Others are asserted by comparing
a pinned input — a public repository at a recorded commit, or a checked-in
fixture — against a checksummed result file, and name the check that performed
fixture — against a recorded result file, and name the check that performed
that comparison. Both appear as claims on the page, in the section they belong
to. A claim earns a separate entry only when it adds something the check cannot
say on its own.
Expand All @@ -94,8 +94,8 @@ make assurance-catalog
go run ./internal/assurance/cmd catalog-validate --evidence graph-npm
```

The first command validates the whole catalog, including the checksums of every
expected-result file it names. The second prints one claim with its reproduction
The first command validates the whole catalog and confirms every expected-result
file it names exists. The second prints one claim with its reproduction
command.

## What a verified claim does not mean
Expand Down
Loading
Loading