From 7552aaf0c74f6c36b788d44f227aedc8fa61c50c Mon Sep 17 00:00:00 2001 From: Scott Robinson Date: Thu, 12 Nov 2026 10:39:57 +0000 Subject: [PATCH] ci: rehearse a release without publishing The three publish workflows (release-image, release-dev-image, npm-publish) gain a `dry_run` input. With it the run ends after the gate and build jobs: the tag gates, the licence and migration-freeze checks, both architectures' builds, smoke tests, vulnerability scans and SBOMs run as for a release, and the publish, propagate and attest jobs are skipped, so their environments ask no approval and nothing reaches a registry. A `dry-run-summary` job writes what ran to the step summary. `devtools/release-dry-run` runs the same gate and build steps on a developer's machine for the host's architecture: the licence check and `check-released-through`, then for each of the postgres, mongodb, cassandra and dev images the build with the release's build-args, the architecture check, the scan and SBOM through the release's pinned trivy image, and the image's `ci/smoke-test-*-container.sh`. `--tag` adds the tag gates; `--crates` adds release-crates.yml's verify job (the publish plan, `cargo package --workspace --locked`, `check-crate-packages`). Every step is attempted and the summary lists each outcome. packaging/README.md's release flow gains the rehearsal as its second step. Run on main at 58043741 for all four images: every build, scan, SBOM and smoke test passed except the dev image's scan (libssl3 CVE-2026-84782, fixed in 3.0.22-1~deb12u2, which a rebuild picks up) and the migration-freeze gate (RELEASED_THROUGH is 0, the last migration 10), which the release PR raises. Signed-off-by: Scott Robinson --- .github/workflows/npm-publish.yml | 38 ++- .github/workflows/release-dev-image.yml | 62 ++-- .github/workflows/release-image.yml | 67 +++-- devtools/check-image-sbom | 69 +++++ devtools/check_image_sbom_test.py | 106 +++++++ devtools/release-dry-run | 360 ++++++++++++++++++++++++ packaging/README.md | 26 +- 7 files changed, 678 insertions(+), 50 deletions(-) create mode 100755 devtools/check-image-sbom create mode 100644 devtools/check_image_sbom_test.py create mode 100755 devtools/release-dry-run diff --git a/.github/workflows/npm-publish.yml b/.github/workflows/npm-publish.yml index 43ef92f21..98b7ad3ef 100644 --- a/.github/workflows/npm-publish.yml +++ b/.github/workflows/npm-publish.yml @@ -9,6 +9,11 @@ # # npm dist-tag add @extenddb/dev@ latest (and one per platform pkg) # +# With the `dry_run` input the run ends after the build jobs: the gates and +# the five native builds, smoke tests and launcher suites run as for a +# release, and nothing is published; dry runs have a concurrency group of +# their own (release-image.yml). +# # Shape, and why (mirrors .github/workflows/release-image.yml): # # gate Runs once. Cheap checks that need no build: the dispatch input is @@ -78,6 +83,10 @@ on: tag: description: 'Existing release tag to publish npm candidates for, e.g. v0.1.4' required: true + dry_run: + description: 'Rehearse only: run the gates and the five platform builds with their smoke tests and launcher suites; publish nothing' + type: boolean + default: false # Nothing is committed, so no write scope on the repo is needed. The publish # job adds id-token at job level, which is what both authenticates to npm @@ -87,7 +96,7 @@ permissions: # Two launcher releases must never race. concurrency: - group: npm-publish-extenddb + group: npm-publish-extenddb${{ inputs.dry_run && '-dry' || '' }} cancel-in-progress: false jobs: @@ -200,9 +209,36 @@ jobs: uses: ./.github/workflows/npm-build-matrix.yml with: ref: ${{ needs.gate.outputs.sha }} + # A release stops at the first failing platform; a rehearsal lets all + # five report. + fail-fast: ${{ !inputs.dry_run }} + + # A rehearsal's end: the gates and the builds ran as the release runs them, + # and nothing was published. The publish-side jobs are skipped by their + # `if`, so their environments ask no approval. + dry-run-summary: + needs: [gate, build] + if: ${{ inputs.dry_run }} + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - name: Summarise the rehearsal + run: | + { + echo "### Dry run: gates and builds passed, nothing published" + echo "" + echo "The five platform binaries built, passed their smoke tests and the launcher suite. Dispatch again without the dry run to publish the candidates." + echo "" + echo "| field | value |" + echo "|---|---|" + echo "| tag | \`v${{ needs.gate.outputs.version }}\` |" + echo "| commit | \`${{ needs.gate.outputs.sha }}\` |" + echo "| version (from tag) | \`${{ needs.gate.outputs.version }}\` |" + } >> "$GITHUB_STEP_SUMMARY" publish: needs: [gate, build] + if: ${{ !inputs.dry_run }} runs-on: ubuntu-latest timeout-minutes: 15 # The protected environment requires reviewer approval, and its name is diff --git a/.github/workflows/release-dev-image.yml b/.github/workflows/release-dev-image.yml index 2e89846ab..1d80e18ad 100644 --- a/.github/workflows/release-dev-image.yml +++ b/.github/workflows/release-dev-image.yml @@ -9,7 +9,9 @@ # promote-registry composite action — signature verified at each destination # before any tag is placed, forward-only latest per destination. The attest # job runs beside propagate and attests the build provenance and the SBOMs -# (see release-image.yml for the design). +# (see release-image.yml for the design). With the `dry_run` input the run +# ends after the build jobs and publishes nothing, in a concurrency group of +# its own (release-image.yml). # # Re-entrant on registry state: if the version tag already exists on Docker # Hub WITH a verifying signature, the publish job treats the published, @@ -59,12 +61,16 @@ on: tag: description: 'Existing version tag to release, e.g. v0.1.6' required: true + dry_run: + description: 'Rehearse only: run the gates and both builds with their smoke tests, scans and SBOMs; publish, sign, mirror and attest nothing' + type: boolean + default: false permissions: contents: read concurrency: - group: release-image-extenddb-dev + group: release-image-extenddb-dev${{ inputs.dry_run && '-dry' || '' }} cancel-in-progress: false env: @@ -155,7 +161,9 @@ jobs: build: needs: gate strategy: - fail-fast: true + # A release stops at the first failing architecture; a rehearsal lets + # both report. + fail-fast: ${{ !inputs.dry_run }} matrix: include: - arch: amd64 @@ -268,25 +276,11 @@ jobs: "$TRIVY_IMAGE" image --input /image.tar.gz \ --format cyclonedx --no-progress --skip-version-check \ > "sbom/sbom-${ARCH}.cdx.json" - REQUIRE_CRATES=0 - grep -q 'cargo auditable build' Dockerfile.dev && REQUIRE_CRATES=1 - python3 - "sbom/sbom-${ARCH}.cdx.json" "$REQUIRE_CRATES" <<'PY' - import json, re, sys - bom = json.load(open(sys.argv[1])) - crates = [c for c in bom.get("components", []) if c.get("purl", "").startswith("pkg:cargo/")] - ids = [p.get("value", "") for p in bom.get("metadata", {}).get("component", {}).get("properties", []) - if p.get("name") == "aquasecurity:trivy:ImageID"] - if (bom.get("bomFormat") != "CycloneDX" or not bom.get("components") or len(ids) != 1 - or not re.fullmatch(r"sha256:[0-9a-f]{64}", ids[0])): - print(f"::error::{sys.argv[1]} is not a CycloneDX SBOM of one image") - sys.exit(1) - if not crates: - if sys.argv[2] == "1": - print(f"::error::{sys.argv[1]} lists no crates, but Dockerfile.dev builds with cargo-auditable") - sys.exit(1) - print(f"::warning::{sys.argv[1]} lists no crates: this tag's Dockerfile.dev predates cargo-auditable") - print(f"{len(bom['components'])} components, {len(crates)} of them crates; image {ids[0]}") - PY + # The check is devtools/check-image-sbom at the tagged commit, shared + # with devtools/release-dry-run; a tag from before it has none. + [[ -x devtools/check-image-sbom ]] \ + || { echo "::error::tagged commit has no devtools/check-image-sbom; use a tag created after it was merged"; exit 1; } + devtools/check-image-sbom "sbom/sbom-${ARCH}.cdx.json" Dockerfile.dev - name: Upload the SBOM artifact uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 @@ -303,8 +297,32 @@ jobs: retention-days: 1 compression-level: 0 + # A rehearsal's end: the gates and the builds ran as the release runs them, + # and nothing was published. The publish-side jobs are skipped by their + # `if`, so their environments ask no approval. + dry-run-summary: + needs: [gate, build] + if: ${{ inputs.dry_run }} + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - name: Summarise the rehearsal + run: | + { + echo "### Dry run: gates and builds passed, nothing published" + echo "" + echo "Both architectures of \`${{ env.IMAGE_REPO }}\` built, passed the smoke test and the vulnerability scan, and produced SBOMs. Dispatch again without the dry run to publish." + echo "" + echo "| field | value |" + echo "|---|---|" + echo "| tag | \`v${{ needs.gate.outputs.version }}\` |" + echo "| commit | \`${{ needs.gate.outputs.sha }}\` |" + echo "| version (from tag) | \`${{ needs.gate.outputs.version }}\` |" + } >> "$GITHUB_STEP_SUMMARY" + publish: needs: [gate, build] + if: ${{ !inputs.dry_run }} runs-on: ubuntu-latest timeout-minutes: 30 environment: dockerhub # approval 1: publish + sign (Docker Hub credential lives here) diff --git a/.github/workflows/release-image.yml b/.github/workflows/release-image.yml index 33b193a8a..89195f7cb 100644 --- a/.github/workflows/release-image.yml +++ b/.github/workflows/release-image.yml @@ -8,6 +8,15 @@ # environment approval. Any defect's blast radius stays a tag no user # consumes. # +# With the `dry_run` input the run ends after the build jobs: the gates and +# both builds, smoke tests, scans and SBOMs run as for a release, and nothing +# is published, signed or attested. A release rehearsal dispatches the three +# publish workflows (this one per image, release-dev-image, npm-publish) this +# way against the tag first, then again for real. Dry runs have a +# concurrency group of their own, so a dry run never cancels a pending +# release. devtools/release-dry-run runs the same gate and build steps on +# a developer's machine, for the host's architecture. +# # Shape, and why: # # gate Runs once. Cheap checks that do not need a build, so a bad input @@ -81,6 +90,10 @@ on: - extenddb-mongodb - extenddb-cassandra default: extenddb-postgres + dry_run: + description: 'Rehearse only: run the gates and both builds with their smoke tests, scans and SBOMs; publish, sign, mirror and attest nothing' + type: boolean + default: false # Nothing is committed, so no write scope on the repo is needed. permissions: @@ -89,7 +102,7 @@ permissions: # Two releases must never race. The group names the image, so a future second # backend does not serialise behind this one for no reason. concurrency: - group: release-image-${{ inputs.image }} + group: release-image-${{ inputs.image }}${{ inputs.dry_run && '-dry' || '' }} cancel-in-progress: false env: @@ -252,7 +265,9 @@ jobs: # If one architecture fails, cancel the other: nothing is published either # way, so there is no point paying for the rest of the matrix. strategy: - fail-fast: true + # A release stops at the first failing architecture; a rehearsal lets + # both report. + fail-fast: ${{ !inputs.dry_run }} matrix: include: - arch: amd64 @@ -402,25 +417,11 @@ jobs: "$TRIVY_IMAGE" image --input /image.tar.gz \ --format cyclonedx --no-progress --skip-version-check \ > "sbom/sbom-${ARCH}.cdx.json" - REQUIRE_CRATES=0 - grep -q 'cargo auditable build' "$DOCKERFILE" && REQUIRE_CRATES=1 - python3 - "sbom/sbom-${ARCH}.cdx.json" "$REQUIRE_CRATES" <<'PY' - import json, re, sys - bom = json.load(open(sys.argv[1])) - crates = [c for c in bom.get("components", []) if c.get("purl", "").startswith("pkg:cargo/")] - ids = [p.get("value", "") for p in bom.get("metadata", {}).get("component", {}).get("properties", []) - if p.get("name") == "aquasecurity:trivy:ImageID"] - if (bom.get("bomFormat") != "CycloneDX" or not bom.get("components") or len(ids) != 1 - or not re.fullmatch(r"sha256:[0-9a-f]{64}", ids[0])): - print(f"::error::{sys.argv[1]} is not a CycloneDX SBOM of one image") - sys.exit(1) - if not crates: - if sys.argv[2] == "1": - print(f"::error::{sys.argv[1]} lists no crates, but the Dockerfile builds with cargo-auditable") - sys.exit(1) - print(f"::warning::{sys.argv[1]} lists no crates: this tag's Dockerfile predates cargo-auditable") - print(f"{len(bom['components'])} components, {len(crates)} of them crates; image {ids[0]}") - PY + # The check is devtools/check-image-sbom at the tagged commit, shared + # with devtools/release-dry-run; a tag from before it has none. + [[ -x devtools/check-image-sbom ]] \ + || { echo "::error::tagged commit has no devtools/check-image-sbom; use a tag created after it was merged"; exit 1; } + devtools/check-image-sbom "sbom/sbom-${ARCH}.cdx.json" "$DOCKERFILE" - name: Upload the SBOM artifact uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 @@ -437,8 +438,32 @@ jobs: retention-days: 1 compression-level: 0 # already gzipped + # A rehearsal's end: the gates and the builds ran as the release runs them, + # and nothing was published. The publish-side jobs are skipped by their + # `if`, so their environments ask no approval. + dry-run-summary: + needs: [gate, build] + if: ${{ inputs.dry_run }} + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - name: Summarise the rehearsal + run: | + { + echo "### Dry run: gates and builds passed, nothing published" + echo "" + echo "Both architectures of \`${{ needs.gate.outputs.image_repo }}\` built, passed the smoke test and the vulnerability scan, and produced SBOMs. Dispatch again without the dry run to publish the candidate." + echo "" + echo "| field | value |" + echo "|---|---|" + echo "| tag | \`v${{ needs.gate.outputs.version }}\` |" + echo "| commit | \`${{ needs.gate.outputs.sha }}\` |" + echo "| version (from tag) | \`${{ needs.gate.outputs.version }}\` |" + } >> "$GITHUB_STEP_SUMMARY" + publish-candidate: needs: [gate, build] + if: ${{ !inputs.dry_run }} runs-on: ubuntu-latest timeout-minutes: 45 environment: dockerhub # approval 1: publish + sign (the only job with Docker Hub secrets) diff --git a/devtools/check-image-sbom b/devtools/check-image-sbom new file mode 100755 index 000000000..29befb243 --- /dev/null +++ b/devtools/check-image-sbom @@ -0,0 +1,69 @@ +#!/usr/bin/env python3 +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 +"""Check the CycloneDX SBOM trivy wrote for one container image. + + devtools/check-image-sbom SBOM.cdx.json DOCKERFILE + +The SBOM must be CycloneDX with components and carry exactly one +`aquasecurity:trivy:ImageID` of the sha256 form, so it describes one image. +When DOCKERFILE builds with `cargo auditable build`, the SBOM must list the +crates the binary embeds; otherwise a missing crate list is a warning, since +the image's binary carries no dependency record for trivy to read. + +The release workflows' build jobs (release-image.yml, release-dev-image.yml) +and devtools/release-dry-run run this on the SBOM they generate. Under GitHub +Actions (GITHUB_ACTIONS set) the messages are workflow annotations. + +Exit 0 when the SBOM passes, 1 otherwise, 2 on a usage error. +""" + +import json +import os +import pathlib +import re +import sys + + +def main() -> int: + if len(sys.argv) != 3: + print(__doc__.strip().splitlines()[2].strip(), file=sys.stderr) + return 2 + sbom_path, dockerfile = pathlib.Path(sys.argv[1]), pathlib.Path(sys.argv[2]) + on_actions = os.environ.get("GITHUB_ACTIONS") == "true" + error = "::error::" if on_actions else "error: " + warning = "::warning::" if on_actions else "warning: " + + try: + bom = json.loads(sbom_path.read_text(encoding="utf-8")) + except (OSError, ValueError) as e: + print(f"{error}{sbom_path} is not readable JSON: {e}") + return 1 + components = bom.get("components") or [] + crates = [c for c in components if str(c.get("purl", "")).startswith("pkg:cargo/")] + ids = [ + p.get("value", "") + for p in bom.get("metadata", {}).get("component", {}).get("properties", []) + if p.get("name") == "aquasecurity:trivy:ImageID" + ] + if ( + bom.get("bomFormat") != "CycloneDX" + or not components + or len(ids) != 1 + or not re.fullmatch(r"sha256:[0-9a-f]{64}", ids[0]) + ): + print(f"{error}{sbom_path} is not a CycloneDX SBOM of one image") + return 1 + + require_crates = "cargo auditable build" in dockerfile.read_text(encoding="utf-8") + if not crates: + if require_crates: + print(f"{error}{sbom_path} lists no crates, but {dockerfile} builds with cargo-auditable") + return 1 + print(f"{warning}{sbom_path} lists no crates: {dockerfile} does not build with cargo-auditable") + print(f"{len(components)} components, {len(crates)} of them crates; image {ids[0]}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/devtools/check_image_sbom_test.py b/devtools/check_image_sbom_test.py new file mode 100644 index 000000000..c4286e05f --- /dev/null +++ b/devtools/check_image_sbom_test.py @@ -0,0 +1,106 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 +"""Tests for devtools/check-image-sbom.""" + +import json +import os +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +SCRIPT = Path(__file__).resolve().parent / "check-image-sbom" +IMAGE_ID = "sha256:" + "ab" * 32 + + +def sbom(components, image_ids=(IMAGE_ID,), bom_format="CycloneDX"): + return { + "bomFormat": bom_format, + "metadata": { + "component": { + "properties": [{"name": "aquasecurity:trivy:ImageID", "value": i} for i in image_ids] + } + }, + "components": components, + } + + +DEB = {"purl": "pkg:deb/debian/libc6@2.36"} +CRATE = {"purl": "pkg:cargo/tokio@1.0.0"} + + +class CheckImageSbomTest(unittest.TestCase): + def setUp(self): + self.dir = tempfile.TemporaryDirectory() + self.root = Path(self.dir.name) + + def tearDown(self): + self.dir.cleanup() + + def run_check(self, bom, dockerfile_text, env=None): + sbom_path = self.root / "sbom.cdx.json" + sbom_path.write_text(json.dumps(bom) if isinstance(bom, dict) else bom) + dockerfile = self.root / "Dockerfile" + dockerfile.write_text(dockerfile_text) + full_env = {k: v for k, v in os.environ.items() if k != "GITHUB_ACTIONS"} + full_env.update(env or {}) + return subprocess.run( + [sys.executable, str(SCRIPT), str(sbom_path), str(dockerfile)], + capture_output=True, text=True, env=full_env, + ) + + def test_an_auditable_build_with_crates_passes(self): + r = self.run_check(sbom([DEB, CRATE]), "RUN cargo auditable build --release\n") + self.assertEqual(r.returncode, 0, r.stdout) + self.assertIn("2 components, 1 of them crates; image " + IMAGE_ID, r.stdout) + + def test_an_auditable_build_without_crates_fails(self): + r = self.run_check(sbom([DEB]), "RUN cargo auditable build --release\n") + self.assertEqual(r.returncode, 1) + self.assertIn("lists no crates, but", r.stdout) + self.assertIn("builds with cargo-auditable", r.stdout) + + def test_a_plain_build_without_crates_warns_and_passes(self): + r = self.run_check(sbom([DEB]), "RUN cargo build --release\n") + self.assertEqual(r.returncode, 0, r.stdout) + self.assertIn("warning: ", r.stdout) + self.assertIn("does not build with cargo-auditable", r.stdout) + + def test_a_second_image_id_fails(self): + r = self.run_check(sbom([DEB, CRATE], image_ids=(IMAGE_ID, IMAGE_ID)), "RUN cargo auditable build\n") + self.assertEqual(r.returncode, 1) + self.assertIn("is not a CycloneDX SBOM of one image", r.stdout) + + def test_a_malformed_image_id_fails(self): + r = self.run_check(sbom([DEB], image_ids=("sha256:short",)), "RUN cargo build\n") + self.assertEqual(r.returncode, 1) + + def test_no_components_fails(self): + r = self.run_check(sbom([]), "RUN cargo build\n") + self.assertEqual(r.returncode, 1) + + def test_another_format_fails(self): + r = self.run_check(sbom([DEB], bom_format="SPDX"), "RUN cargo build\n") + self.assertEqual(r.returncode, 1) + + def test_unreadable_json_fails(self): + r = self.run_check("{", "RUN cargo build\n") + self.assertEqual(r.returncode, 1) + self.assertIn("is not readable JSON", r.stdout) + + def test_messages_are_annotations_under_actions(self): + r = self.run_check(sbom([DEB]), "RUN cargo build\n", env={"GITHUB_ACTIONS": "true"}) + self.assertEqual(r.returncode, 0) + self.assertIn("::warning::", r.stdout) + r = self.run_check(sbom([DEB]), "RUN cargo auditable build\n", env={"GITHUB_ACTIONS": "true"}) + self.assertIn("::error::", r.stdout) + + def test_usage_error(self): + r = subprocess.run([sys.executable, str(SCRIPT)], capture_output=True, text=True) + self.assertEqual(r.returncode, 2) + self.assertIn("check-image-sbom SBOM.cdx.json DOCKERFILE", r.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/devtools/release-dry-run b/devtools/release-dry-run new file mode 100755 index 000000000..7f34ec89b --- /dev/null +++ b/devtools/release-dry-run @@ -0,0 +1,360 @@ +#!/usr/bin/env bash +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 +# +# Rehearse the container release workflows locally, with nothing published. +# +# devtools/release-dry-run [--tag vX.Y.Z] [--images LIST|none] [--crates] +# [--skip-scan] [--keep] [--reuse-images] +# [--stop-on-fail] +# +# Runs, on the checked-out commit and for each image, what the `gate` and +# `build` jobs of release-image.yml and release-dev-image.yml run before +# anything is pushed: the licence-notice check, the migration-freeze check +# (devtools/check-released-through), the image build with the release's +# build-args (VERSION from Cargo.toml, VCS_REF the commit, BUILD_DATE the +# commit's timestamp), the architecture check, the vulnerability scan and +# SBOM through the release's pinned trivy image (the pin read from +# release-image.yml) with the release's flags, +# and the image's smoke test (ci/smoke-test-*-container.sh). Publishing, +# signing, mirroring and attestation are not rehearsed here: the signing and +# attestation paths have their own self-test workflows, and the push is the +# release. The working tree must be clean: the release builds a commit, and +# this rehearses the tree as that commit. +# +# `--tag vX.Y.Z` also runs the tag gates on that tag, as the workflows do: +# strict shape, exists on origin, contained in origin's main, and the version +# equals Cargo.toml's; the tag must name the checked-out commit. Without it +# the tag gates are reported as not run. +# +# `--images` names which of postgres, mongodb, cassandra, dev to rehearse +# (default: all four; `none` runs the shared gates alone). `--crates` adds +# release-crates.yml's plan and `verify` steps: the publish plan +# (`devtools/publish-crates --dry-run`), `cargo package --workspace +# --locked`, `devtools/check-crate-packages` and the SHA-256 record of each +# package, all on the toolchain release-crates.yml pins (RUST_TOOLCHAIN, +# installed with rustup if absent); that workflow's other gates need the +# promoted image and are not rehearsed. +# `--skip-scan` skips the vulnerability scan and the SBOM (the trivy image +# and its database need network access). The build runs with `--no-cache`, +# as the release's does, so a rerun installs the base image's current +# package updates where the Dockerfile upgrades them; `--reuse-images` is the +# way to rerun a smoke test without that cost. +# +# Every gate and image is attempted and the summary lists each outcome with +# its log; the exit code is 1 when anything failed. With `--stop-on-fail` +# the run stops at the first failure, as the real gate does. `--keep` leaves +# the built images and the work directory in place, and `--reuse-images` +# skips a build whose image a previous `--keep` run left, so a smoke test +# can be re-run without rebuilding. Images are tagged +# `extenddb-dry-run/:-`, a name no release uses; a +# build moves that tag to the new image, and the run removes only the images +# it built, giving a kept image its tag back. Logs, tarballs and SBOMs go +# under target/release-dry-run/. +# +# Needs: docker (daemon running, with compose v2), aws, python3, cargo, +# cargo-about 0.9.0 (the licence check), rustup for `--crates`, and +# `timeout` (the Cassandra smoke test). PYTHON names the interpreter for the devtools scripts (default +# python3; `--crates` needs 3.11 or later). One architecture only, the +# host's: the release builds linux/amd64 and linux/arm64, each on its own +# runner. + +set -Eeuo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +cd "$ROOT" + +usage() { + # The header comment above, up to the first line of code. + awk 'NR >= 5 && !/^#/ {exit} NR >= 5 {sub(/^# ?/, ""); print}' "$0" +} + +TAG="" +IMAGES="postgres mongodb cassandra dev" +CRATES=0 +SKIP_SCAN=0 +KEEP=0 +REUSE=0 +STOP_ON_FAIL=0 +while [[ $# -gt 0 ]]; do + case "$1" in + --tag) [[ $# -ge 2 ]] || { echo "error: --tag needs a value, e.g. v1.0.0" >&2; exit 2; }; TAG="$2"; shift 2 ;; + --images) [[ $# -ge 2 ]] || { echo "error: --images needs a list, e.g. postgres,dev, or none" >&2; exit 2; }; IMAGES="${2//,/ }"; shift 2 ;; + --crates) CRATES=1; shift ;; + --skip-scan) SKIP_SCAN=1; shift ;; + --keep) KEEP=1; shift ;; + --reuse-images) REUSE=1; shift ;; + --stop-on-fail) STOP_ON_FAIL=1; shift ;; + -h|--help) usage; exit 0 ;; + *) echo "error: unknown argument $1" >&2; usage >&2; exit 2 ;; + esac +done +[[ "$IMAGES" == none ]] && IMAGES="" +for image in $IMAGES; do + case "$image" in + postgres|mongodb|cassandra|dev) ;; + *) echo "error: unknown image '$image' (postgres, mongodb, cassandra, dev, or none)" >&2; exit 2 ;; + esac +done +if [[ -z "$IMAGES" && $CRATES == 0 ]]; then + echo "note: no images and no --crates; only the shared gates run" >&2 +fi + +PYTHON="${PYTHON:-python3}" +# The release's pins, read from the workflows so this script and the release +# use one copy: the trivy image of the build jobs, and the toolchain of the +# crates.io jobs. +pin() { # file key + local value; value=$(sed -n "s/^ *$2: *[\"']\\{0,1\\}\\([^\"' ]*\\)[\"']\\{0,1\\} *$/\\1/p" "$1" | head -n 1) + [[ -n "$value" ]] || { echo "error: no $2 pin in $1" >&2; exit 1; } + printf '%s\n' "$value" +} +TRIVY_IMAGE=$(pin .github/workflows/release-image.yml TRIVY_IMAGE) +RUST_TOOLCHAIN=$(pin .github/workflows/release-crates.yml RUST_TOOLCHAIN) + +# --- Prerequisites ------------------------------------------------------------- +missing=0 +for command in docker aws cargo git timeout "$PYTHON"; do + command -v "$command" >/dev/null 2>&1 || { echo "error: required command not found: $command" >&2; missing=1; } +done +[[ $missing == 0 ]] || exit 1 +docker info >/dev/null 2>&1 || { echo "error: the docker daemon is not reachable" >&2; exit 1; } +if [[ -n "$IMAGES" ]]; then + docker compose version >/dev/null 2>&1 || { echo "error: docker compose v2 is required by the smoke tests" >&2; exit 1; } + cargo about --version >/dev/null 2>&1 || { echo "error: cargo-about is required by the licence check (cargo install --locked --version 0.9.0 --features cli cargo-about)" >&2; exit 1; } +fi +if [[ $CRATES == 1 ]]; then + command -v rustup >/dev/null 2>&1 || { echo "error: --crates needs rustup, for the release's toolchain $RUST_TOOLCHAIN" >&2; exit 1; } +fi + +# --- Work directory and bookkeeping ------------------------------------------ +# Under the repository (gitignored, excluded from the Docker context), so the +# trivy container's bind mount works where /tmp is not shared with the daemon. +mkdir -p target/release-dry-run +WORK="$(mktemp -d "$ROOT/target/release-dry-run/run.XXXXXX")" +RESULTS=() +FAILED=0 +BUILT_IMAGES=() +NAME_WIDTH=24 + +record() { # name outcome seconds detail + RESULTS+=("$(printf '%s\t%s\t%ss\t%s' "$1" "$2" "$3" "$4")") + (( ${#1} > NAME_WIDTH )) && NAME_WIDTH=${#1} + if [[ "$2" == FAIL ]]; then + FAILED=1 + if [[ $STOP_ON_FAIL == 1 ]]; then + summary + exit 1 + fi + fi +} + +# run NAME LOG -- COMMAND...: runs the step in a subshell with errexit, so a +# failing command inside a step function fails the step. The subshell is not +# the condition of an `if` or an `||` list: in those contexts bash ignores +# `set -e`, inside the subshell too, and a step whose first command failed +# would run on and be recorded as PASS. Records PASS or FAIL with the +# duration and the log's path; the log's tail is shown on failure. +run() { + local name="$1" log="$WORK/$2"; shift 2; [[ "$1" == "--" ]] && shift + local start=$SECONDS status + set +e + ( set -e; "$@" ) >"$log" 2>&1 + status=$? + set -e + if [[ $status == 0 ]]; then + record "$name" PASS $((SECONDS - start)) "$log" + else + echo "---- $name failed; last 40 lines of $log:" >&2 + tail -n 40 "$log" >&2 + echo "----" >&2 + record "$name" FAIL $((SECONDS - start)) "$log" + fi +} + +summary() { + echo + echo "release dry run of $SHA (version $VERSION, built $BUILD_DATE), linux/$ARCH, $(rustc --version 2>/dev/null || echo 'rustc unknown')" + printf "%-${NAME_WIDTH}s %-7s %8s %s\n" step outcome time detail + local r n o t d + for r in "${RESULTS[@]}"; do + IFS=$'\t' read -r n o t d <<<"$r" + printf "%-${NAME_WIDTH}s %-7s %8s %s\n" "$n" "$o" "$t" "$d" + done + if [[ $FAILED == 1 ]]; then echo "RESULT: FAIL"; else echo "RESULT: PASS"; fi + echo "logs: $WORK" +} + +# BUILT_IMAGES holds `||` for each +# image this run built. +cleanup() { + if [[ $KEEP == 0 ]]; then + local entry id tag before + for entry in "${BUILT_IMAGES[@]+"${BUILT_IMAGES[@]}"}"; do + IFS='|' read -r id tag before <<<"$entry" + docker image rm -f "$id" >/dev/null 2>&1 || true + [[ "$before" == - ]] || docker tag "$before" "$tag" >/dev/null 2>&1 || true + done + rm -f "$WORK"/image-*.tar.gz + fi +} +trap cleanup EXIT + +# --- Release identity, as the gate derives it ------------------------------- +SHA=$(git rev-parse HEAD) +VERSION=$(grep -m1 '^version' Cargo.toml | sed 's/.*"\(.*\)".*/\1/') +BUILD_DATE=$(TZ=UTC git show -s --date=format-local:'%Y-%m-%dT%H:%M:%SZ' --format=%cd "$SHA") +ARCH=$(docker version --format '{{.Server.Arch}}') +echo "commit $SHA, Cargo.toml version $VERSION, build date $BUILD_DATE, linux/$ARCH" +echo "logs in $WORK" + +# --- Shared gates --------------------------------------------------------------- +clean_tree() { + local dirty; dirty=$(git status --porcelain --untracked-files=no) + [[ -z "$dirty" ]] || { echo "the working tree differs from $SHA:"; echo "$dirty"; return 1; } + echo "the working tree is $SHA" +} +run "gate: working tree clean" gate-clean-tree.log -- clean_tree + +# The tag gates of release-image.yml's gate job, steps 1 to 4. The remote is +# read directly (ls-remote and FETCH_HEAD), so a local tag or branch of the +# same name cannot stand in for it. +tag_gates() { + [[ "$TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]] \ + || { echo "tag '$TAG' is not a strict vMAJOR.MINOR.PATCH"; return 1; } + local remote_tag tag_sha + remote_tag=$(git ls-remote --exit-code --tags origin "refs/tags/${TAG}" "refs/tags/${TAG}^{}") \ + || { echo "tag $TAG does not exist on origin"; return 1; } + # An annotated tag lists its object and its peeled commit; take the commit. + tag_sha=$(awk '$2 ~ /\^\{\}$/ {peeled=$1} $2 !~ /\^\{\}$/ {plain=$1} END {print (peeled != "" ? peeled : plain)}' <<<"$remote_tag") + [[ "$tag_sha" == "$SHA" ]] \ + || { echo "tag $TAG is $tag_sha on origin; the checked-out commit is $SHA"; return 1; } + git fetch --no-tags origin main + git merge-base --is-ancestor "$SHA" FETCH_HEAD \ + || { echo "$TAG ($SHA) is not contained in origin's main"; return 1; } + [[ "${TAG#v}" == "$VERSION" ]] \ + || { echo "tag $TAG implies ${TAG#v} but Cargo.toml says $VERSION"; return 1; } + echo "tag $TAG: strict shape, on origin at $SHA, on main, matches Cargo.toml" +} +if [[ -n "$TAG" ]]; then + run "gate: tag $TAG" gate-tag.log -- tag_gates +else + on_main="whether $SHA is on origin's main was not checked" + if git fetch --no-tags origin main >/dev/null 2>&1; then + if git merge-base --is-ancestor "$SHA" FETCH_HEAD; then on_main="$SHA is on origin's main"; else on_main="$SHA is NOT on origin's main"; fi + fi + record "gate: tag" SKIP 0 "no --tag; the release needs v$VERSION at a commit on main ($on_main)" +fi + +# The migration freeze, shared by every release workflow's gate. +run "gate: migrations frozen" gate-released-through.log -- "$PYTHON" devtools/check-released-through + +# --- Per image ------------------------------------------------------------------- +files_exist() { [[ -f "$1" && -f "$2" ]] || { echo "missing $1 or $2"; return 1; }; } + +build_image() { # dockerfile image + docker build -f "$1" --platform "linux/$ARCH" --no-cache \ + --build-arg "VERSION=$VERSION" --build-arg "VCS_REF=$SHA" --build-arg "BUILD_DATE=$BUILD_DATE" \ + -t "$2" . +} + +arch_check() { # image + local actual; actual=$(docker image inspect "$1" --format '{{.Architecture}}') + [[ "$actual" == "$ARCH" ]] || { echo "expected $ARCH, image reports $actual"; return 1; } + echo "image is $actual" +} + +save_image() { # image tarball + [[ -f "$2" ]] || docker save "$1" | gzip > "$2" + ls -lh "$2" +} + +scan_image() { # image tarball + save_image "$1" "$2" + docker run --rm -v "$2:/image.tar.gz:ro" "$TRIVY_IMAGE" image --input /image.tar.gz \ + --scanners vuln --skip-files usr/local/bin/extenddb --ignore-unfixed --severity HIGH,CRITICAL \ + --db-repository public.ecr.aws/aquasecurity/trivy-db:2 --exit-code 1 --no-progress +} + +# The SBOM step of the build jobs, with the workflow's own validation. +sbom_image() { # image tarball dockerfile sbom + save_image "$1" "$2" + docker run --rm -v "$2:/image.tar.gz:ro" "$TRIVY_IMAGE" image --input /image.tar.gz \ + --format cyclonedx --no-progress --skip-version-check > "$4" + "$PYTHON" devtools/check-image-sbom "$4" "$3" +} + +smoke_image() { # env-var script image + env "$1=$3" "$2" +} + +for image in $IMAGES; do + case "$image" in + postgres) dockerfile=Dockerfile; smoke=ci/smoke-test-container.sh; smoke_env=EXTENDDB_IMAGE; licence=(devtools/generate-software-license-notices --check --backend postgres) ;; + mongodb) dockerfile=Dockerfile.mongodb; smoke=ci/smoke-test-mongodb-container.sh; smoke_env=EXTENDDB_MONGODB_IMAGE; licence=(devtools/generate-software-license-notices --check --backend mongodb) ;; + cassandra) dockerfile=Dockerfile.cassandra; smoke=ci/smoke-test-cassandra-container.sh; smoke_env=EXTENDDB_CASSANDRA_IMAGE; licence=(devtools/generate-software-license-notices --check --backend cassandra) ;; + dev) dockerfile=Dockerfile.dev; smoke=ci/smoke-test-dev-container.sh; smoke_env=EXTENDDB_IMAGE; licence=(devtools/generate-dev-license-notices --check) ;; + esac + # A name no release uses: the candidate is extenddb/extenddb-:sha--. + img="extenddb-dry-run/$image:${SHA:0:12}-$ARCH" + tarball="$WORK/image-$image-$ARCH.tar.gz" + sbom="$WORK/sbom-$image-$ARCH.cdx.json" + + run "$image: gate: Dockerfile and smoke script" "$image-gate-files.log" -- files_exist "$dockerfile" "$smoke" + run "$image: gate: licence notices" "$image-gate-licence.log" -- "${licence[@]}" + + if [[ $REUSE == 1 ]] && docker image inspect "$img" >/dev/null 2>&1; then + record "$image: build ($ARCH)" SKIP 0 "--reuse-images: $img exists and is kept" + else + before=$(docker image inspect --format '{{.Id}}' "$img" 2>/dev/null || echo -) + run "$image: build ($ARCH)" "$image-build.log" -- build_image "$dockerfile" "$img" + after=$(docker image inspect --format '{{.Id}}' "$img" 2>/dev/null || echo -) + [[ "$after" == - || "$after" == "$before" ]] || BUILT_IMAGES+=("$after|$img|$before") + fi + if ! docker image inspect "$img" >/dev/null 2>&1; then + for step in "arch check" "vulnerability scan" SBOM "smoke test"; do record "$image: $step" SKIP 0 "no image"; done + continue + fi + + run "$image: arch check" "$image-arch.log" -- arch_check "$img" + if [[ $SKIP_SCAN == 1 ]]; then + record "$image: vulnerability scan" SKIP 0 "--skip-scan" + record "$image: SBOM" SKIP 0 "--skip-scan" + else + run "$image: vulnerability scan" "$image-scan.log" -- scan_image "$img" "$tarball" + run "$image: SBOM" "$image-sbom.log" -- sbom_image "$img" "$tarball" "$dockerfile" "$sbom" + fi + run "$image: smoke test" "$image-smoke.log" -- smoke_image "$smoke_env" "$smoke" "$img" +done + +# --- crates.io: release-crates.yml's plan (a gate step) and verify job ---------- +if [[ $CRATES == 1 ]]; then + # release-crates.yml runs every cargo on RUST_TOOLCHAIN; a crate that builds + # on a newer toolchain and not on the pin would pass here and fail there. + export RUSTUP_TOOLCHAIN="$RUST_TOOLCHAIN" + toolchain_matches() { + rustup toolchain list | grep -q "^$RUST_TOOLCHAIN" || rustup toolchain install --profile minimal "$RUST_TOOLCHAIN" + local actual; actual=$(rustc --version) + [[ "$actual" == "rustc $RUST_TOOLCHAIN "* ]] || { echo "rustc is '$actual', not the release's $RUST_TOOLCHAIN"; return 1; } + echo "$actual (release-crates.yml pins $RUST_TOOLCHAIN)" + } + run "crates: toolchain $RUST_TOOLCHAIN" crates-toolchain.log -- toolchain_matches + run "crates: gate: publish plan" crates-plan.log -- "$PYTHON" devtools/publish-crates --dry-run --version "$VERSION" + run "crates: cargo package --workspace" crates-package.log -- cargo package --workspace --locked + run "crates: package rules" crates-rules.log -- "$PYTHON" devtools/check-crate-packages + crate_sha256() { + mkdir -p "$WORK/crate-sha256" + "$PYTHON" devtools/publish-crates --list-packages --version "$VERSION" > "$WORK/crate-sha256/packages.txt" + [[ -s "$WORK/crate-sha256/packages.txt" ]] || { echo "devtools/publish-crates lists no package"; return 1; } + printf '%s\n' "$SHA" > "$WORK/crate-sha256/COMMIT" + printf '%s\n' "$RUST_TOOLCHAIN" > "$WORK/crate-sha256/TOOLCHAIN" + local packages=() + while IFS= read -r line; do packages+=("$line"); done < "$WORK/crate-sha256/packages.txt" + (cd target/package && sha256sum -- "${packages[@]}") | tee "$WORK/crate-sha256/SHA256SUMS" + } + run "crates: SHA-256 record" crates-sha256.log -- crate_sha256 +fi + +summary +[[ $FAILED == 0 ]] diff --git a/packaging/README.md b/packaging/README.md index 60f8ffe5a..59b8c238b 100644 --- a/packaging/README.md +++ b/packaging/README.md @@ -112,25 +112,39 @@ deliberate second step): `npm/package.json`. The launcher runs only a binary of its own release. On a tag whose `package.json` lags the workspace version, the launcher tests would fail on every build leg. The publish gate refuses such a tag - before any build. The container release uses the same tag. A maintainer - dispatches the container and npm publish workflows independently against - it. The crates.io workflow runs last, after `promote-image`. For a release + before any build. The container release uses the same tag. For a release that ships a catalog migration, the release commit also raises `RELEASED_THROUGH` to the last migration. That commit holds the checksum lines that the checksum test prints (CONTRIBUTING.md, "Adding a catalog migration"). The gate jobs of the release workflows run `devtools/check-released-through` on the tag and refuse it otherwise. -2. Dispatch the `npm-publish` workflow from `main` with the tag as input. +2. Rehearse first. Before the tag is pushed, run `devtools/release-dry-run` + on the release commit: it runs the image workflows' gate and build steps + on a developer's machine for the host's architecture (`--crates` adds + the crates.io verify steps on the release's pinned toolchain), and a + failure there costs a new commit, not a new version. Then push the tag + and dispatch `npm-publish`, `release-dev-image` and `release-image` (once + per backend image) against it with `dry_run` checked. Each runs its + gates and its builds as for the release (the image workflows: both + architectures, smoke tests, scans and SBOMs; the npm workflow: the five + platform binaries, their smoke tests and the launcher suite) and + publishes nothing. The `protect-release-tags` ruleset blocks any update + or deletion of a `v*` tag, so a rehearsal failure that needs a code + change after the tag is pushed takes the next patch version. +3. A maintainer dispatches the container and npm publish workflows + independently against the tag; the crates.io workflow runs last, after + `promote-image`. Dispatch the `npm-publish` workflow from `main` with the + tag as input. The gate validates shape, existence, ancestry against `main`, and the version match. Five native runners build, smoke test, and run the full launcher suite against their own binary. Only after all five pass does the reviewer-gated `npm` environment release the token, and everything is published with `--provenance` under the `candidate` dist-tag. Nothing reaches `latest`. -3. Verify the candidate from a clean machine: +4. Verify the candidate from a clean machine: `npm install @extenddb/dev@candidate` and run a smoke script in both storage modes. -4. Promote, one dist-tag move per package: +5. Promote, one dist-tag move per package: ``` npm dist-tag add @extenddb/dev@X.Y.Z latest