Repository navigation
ci: rehearse a release without publishing - #973
Conversation
eeabc50 to
5e8ebcb
Compare
LeeroyHannigan
left a comment
There was a problem hiding this comment.
I ran devtools/release-dry-run --keep --crates on the head on an amd64 host, and then its edge cases. I also dispatched the three workflows from this branch with dry_run checked. They ran against v0.1.14, which is on main at version 0.1.14:
- release-image,
extenddb-postgres(run 43422561132): the gate, both builds and the summary passed. The arm64 build first failed to pull the trivy image (ECR Public answeredtoomanyrequests: Rate exceeded) and passed when I re-ran that job.publish-candidate,propagateandattestwere skipped. - npm-publish (run 43422561669): the gate, the five platform builds and the summary passed, and
publishwas skipped. - release-dev-image (run 43422518630): both builds failed the scan on
libssl3CVE-2026-84782, the finding of your local run.publish,propagateandattestwere skipped.
None of the three runs asked for an environment approval.
What matched the description:
- With
dry_runchecked, no job that runs has an environment, anid-token,packages,attestationsorcontents: writepermission, or a registry login. Without the input, each workflow runs the same jobs as onmain. - My run of the script gave your table: the same two failures, and the same SBOM counts for the four images. It took 46 minutes on a loaded host.
- The smoke tests published ports on 127.0.0.1 only.
- The publish plan stops before the first crate, and nothing in the script pushes, uploads, tags or logs in.
--images none, an unknown--imagesvalue,--tagwith no value,--tag v0.1.14,--stop-on-failon a dirty tree and--reuse-images --keepbehaved as the description says.- shellcheck at warning and style level, and actionlint, report nothing in the changed files.
Four items are required. Three are inline. The fourth is the description, where these statements are wrong:
- The dev scan row says of the
libssl3finding: "a rebuild on release day picks the fix up".Dockerfile.devbuilds its runtime stage ongcr.io/distroless/cc-debian12:nonrootpinned by digest, with noRUNstep, so a rebuild keeps the samelibssl3. The pinned image and the currentnonrootbuild both carry 3.0.20-1deb12u2. The dev image passes its scan only after the pin moves to a distroless build with 3.0.22-1deb12u2. - What says that the licence check, both architectures, the scans and the SBOMs run in the three workflows. npm-publish runs the tag and migration gates and the five platform builds, with their smoke tests and launcher suites.
- What says that
propagateandattestare skipped by anifon the input. They have noif. They are skipped because the job they need is skipped. - Why says that there is no
extenddb/extenddb-cassandraon Docker Hub. The repository exists, registered on 4 November, and has no tags. - Testing done says that the workflows and
--tagwere not run. The Review round section reports a--tagrun, and the three runs above now exist.
Four inline items are Minor.
Outside this PR:
- #458 (mine) also changes npm-publish.yml. git merges the two with two
if:keys onpublishand no conflict marker, and actionlint rejects the result. #458'spromotejob also has nodry_runterm. If this PR merges first, I'll fix both in #458. - ECR Public has no
extenddb-cassandrarepository: an anonymous read answersNAME_UNKNOWN. Nothing in the release workflows creates it, so the first Cassandra release would fail inpropagate, after the candidate is pushed and signed. A dry run stops before that job, so it cannot show this.
Once the required items are fixed, I'll approve. The Minor items can go in the same revision.
| run() { | ||
| local name="$1" log="$WORK/$2"; shift 2; [[ "$1" == "--" ]] && shift | ||
| local start=$SECONDS | ||
| if ( set -e; "$@" ) >"$log" 2>&1; then |
There was a problem hiding this comment.
The subshell is the condition of an if, where bash ignores -e. The bash manual, under set -e, says that this also applies to a compound command that sets -e in that context. I copied this function into a test script unchanged. A step whose first command is false runs its later commands and is recorded as PASS, on bash 4.2.46 and 5.2.21. In tag_gates, a failed git fetch --no-tags origin main at line 206 leaves merge-base to read the previous FETCH_HEAD. The gate can then report "on main" for a commit that is not on main. Please run the subshell outside the condition and branch on its status. This form fails the step on both bash versions: set +e; ( set -e; "$@" ) >"$log" 2>&1; status=$?; set -e. ( set -e; "$@" ) || status=$? does not work either, because an || list is also a context where -e is ignored.
There was a problem hiding this comment.
Fixed. I reproduced it on bash 4.2.46 with a false; echo step: the old form recorded PASS and ran the echo. run now uses the form you give: set +e; ( set -e; "$@" ) >"$log" 2>&1; status=$?; set -e, and branches on status. The same probe step is recorded FAIL with the later command not run, and the comment on run says why the subshell must not be an if condition or an || list.
| # --- crates.io: release-crates.yml's plan (a gate step) and verify job ---------- | ||
| if [[ $CRATES == 1 ]]; then | ||
| run "crates: gate: publish plan" crates-plan.log -- "$PYTHON" devtools/publish-crates --dry-run --version "$VERSION" | ||
| run "crates: cargo package --workspace" crates-package.log -- cargo package --workspace --locked |
There was a problem hiding this comment.
cargo package runs on the cargo on PATH. On my host that was rustc 1.99.0, and the TOOLCHAIN file of the SHA-256 record says so. In release-crates.yml, every job that runs cargo, verify and publish included, uses RUST_TOOLCHAIN: "1.97.0". A crate that builds on 1.99 and not on 1.97.0 passes the rehearsal and fails the release. Please read the pin from release-crates.yml and set RUSTUP_TOOLCHAIN to it for the crates steps. The script can then fail when rustc --version does not match the pin. Line 331 can also write the pin into TOOLCHAIN, as the workflow does.
There was a problem hiding this comment.
Fixed. The script reads RUST_TOOLCHAIN from release-crates.yml (one copy), exports RUSTUP_TOOLCHAIN for the crates steps, installs the toolchain with rustup toolchain install --profile minimal if absent, and a new crates: toolchain 1.97.0 step fails unless rustc --version starts with rustc 1.97.0 . The TOOLCHAIN file of the record is the pin, as the workflow writes it. Worth noting: the SHA-256 of every package differs between the 1.97.0 run and this host's earlier 1.98.0 run, so the pin matters for the record, not only for the build.
| migration"). The gate jobs of the release workflows run | ||
| `devtools/check-released-through` on the tag and refuse it otherwise. | ||
| 2. Dispatch the `npm-publish` workflow from `main` with the tag as input. | ||
| 2. Rehearse first: dispatch `npm-publish`, `release-dev-image` and |
There was a problem hiding this comment.
A rehearsal on CI needs the release tag on origin, and the protect-release-tags ruleset blocks any update or deletion of a v* tag. A rehearsal failure that needs a code change therefore takes the next patch version. Please say so in this step, and say to run devtools/release-dry-run on the release commit before the tag is pushed. Step 1 also still says "A maintainer dispatches the container and npm publish workflows independently against it", before this step's rehearsal. That sentence belongs in step 3.
There was a problem hiding this comment.
Fixed. Step 2 now opens with running devtools/release-dry-run on the release commit before the tag is pushed (a failure there costs a commit, not a version), then the tag push and the three dry_run dispatches, and closes with the protect-release-tags rule: a rehearsal failure that needs a code change after the push takes the next patch version. The "A maintainer dispatches the container and npm publish workflows independently against it. The crates.io workflow runs last, after promote-image." sentences moved from step 1 to the head of step 3.
| # both builds, smoke tests, scans and SBOMs run as for a release, and nothing | ||
| # is published, signed or attested. A release rehearsal dispatches the three | ||
| # publish workflows (this one per image, release-dev-image, npm-publish) this | ||
| # way against the tag first, then again for real. A dry run holds the |
There was a problem hiding this comment.
Minor. A dry run and a release of one image share this concurrency group. GitHub keeps one pending run per group, and a newly queued run cancels the pending one. So if a dry run is running and a release is pending, a second dry run cancels the pending release. The project-status-sweep.yml workflow gives its dry runs their own group with ${{ inputs.dry_run && '-dry' || '' }}. With the same suffix here and in release-dev-image.yml and npm-publish.yml, a dry run cannot cancel a pending release. These header sentences could then go.
There was a problem hiding this comment.
Done. The three groups take ${{ inputs.dry_run && '-dry' || '' }}, as project-status-sweep.yml does, and the header sentences about a dry run holding the group are replaced by one saying dry runs have a group of their own.
| files_exist() { [[ -f "$1" && -f "$2" ]] || { echo "missing $1 or $2"; return 1; }; } | ||
|
|
||
| build_image() { # dockerfile image | ||
| docker build -f "$1" --platform "linux/$ARCH" \ |
There was a problem hiding this comment.
Minor. The build uses the host's layer cache. A second run on the same commit replayed every layer from the cache, including the runtime stage's apt-get upgrade. The rebuild took 2 s and gave the same image ID as the run 47 minutes earlier. A rerun days later does not see a Debian security update that the release's build, which has no cache, would install. --no-cache here would remove the difference. Otherwise, the header can say that a rerun reuses the cached layers.
There was a problem hiding this comment.
Done: docker build --no-cache, as the release's build has no cache. The header says so and points at --reuse-images for a cheap rerun. The cost is what you'd expect: the dev build took 548 s here where the cached replay took 2 s.
| if [[ $REUSE == 1 ]] && docker image inspect "$img" >/dev/null 2>&1; then | ||
| record "$image: build ($ARCH)" SKIP 0 "--reuse-images: $img exists and is kept" | ||
| else | ||
| docker image rm -f "$img" >/dev/null 2>&1 || true |
There was a problem hiding this comment.
Minor. Without --reuse-images, this line removes an image a --keep run left. I ran --images postgres --skip-scan after a --keep run. It removed the kept postgres image, rebuilt it from cache and removed it at the end. The header at line 44 and the description say that an image this run did not build is never removed. Comparing the image ID before and after the build would make that true. Or the header can say that a run without --reuse-images replaces the kept image.
There was a problem hiding this comment.
Fixed so the header's statement is true. The pre-build docker image rm is gone. The run records the tag's image id before the build and after it; when they differ it keeps <new id>|<tag>|<previous id>, and the cleanup removes the images it built by id and then retags the previous id, so a kept image gets its tag back. Your reproduction (--images dev --skip-scan after a --keep run) now ends with the kept image under its tag; a run that finds no kept image removes only what it built, as before.
| sbom_image() { # image tarball dockerfile sbom | ||
| save_image "$1" "$2" | ||
| docker run --rm -v "$2:/image.tar.gz:ro" "$TRIVY_IMAGE" image --input /image.tar.gz \ | ||
| --format cyclonedx --no-progress --skip-version-check > "$4" |
There was a problem hiding this comment.
Minor. This SBOM check is now in three places, here and in release-image.yml and release-dev-image.yml, and the trivy image digest is in five. A devtools/check-image-sbom that both workflows and this script call would leave one copy of the check.
There was a problem hiding this comment.
Done. devtools/check-image-sbom SBOM.cdx.json DOCKERFILE is the one copy: both image workflows' SBOM steps call it instead of their inline Python, and so does the script. It prints ::error::/::warning:: annotations when GITHUB_ACTIONS is set and plain text otherwise; devtools/check_image_sbom_test.py (10 cells) runs it through scripts-test.yml. One consequence to know about: the build jobs check out the tagged commit, so a tag from before this PR has no script; the step checks for it and fails with a message saying to use a later tag, as the licence gate does for a missing Dockerfile. The trivy digest is now read from release-image.yml by the script, so it lives in the two workflows only.
The three publish workflows (release-image, release-dev-image, npm-publish) gain a `dry_run` input. With it the run ends after the gate and build jobs: the tag gates, the licence and migration-freeze checks, both architectures' builds, smoke tests, vulnerability scans and SBOMs run as for a release, and the publish, propagate and attest jobs are skipped, so their environments ask no approval and nothing reaches a registry. A `dry-run-summary` job writes what ran to the step summary. `devtools/release-dry-run` runs the same gate and build steps on a developer's machine for the host's architecture: the licence check and `check-released-through`, then for each of the postgres, mongodb, cassandra and dev images the build with the release's build-args, the architecture check, the scan and SBOM through the release's pinned trivy image, and the image's `ci/smoke-test-*-container.sh`. `--tag` adds the tag gates; `--crates` adds release-crates.yml's verify job (the publish plan, `cargo package --workspace --locked`, `check-crate-packages`). Every step is attempted and the summary lists each outcome. packaging/README.md's release flow gains the rehearsal as its second step. Run on main at 5804374 for all four images: every build, scan, SBOM and smoke test passed except the dev image's scan (libssl3 CVE-2026-84782, fixed in 3.0.22-1~deb12u2, which a rebuild picks up) and the migration-freeze gate (RELEASED_THROUGH is 0, the last migration 10), which the release PR raises. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
5e8ebcb to
7552aaf
Compare
|
All four required items are fixed in Thanks for the two notes outside the PR. I've put both on the description as 1.0 checklist items: the Checks on this head: the shared SBOM check against the kept SBOMs (postgres 427/333, cassandra 93/0 with the warning), its 10 unit cells, |
What
A way to rehearse the release without publishing anything, for the 1.0 release and the ones after it.
dry_runinput onrelease-image,release-dev-imageandnpm-publish. With it checked, the run ends after the gate and build jobs, which run as for a release. In the two image workflows: the tag gates (strict shape, exists on origin, onmain, matchesCargo.toml), the licence-notice check,devtools/check-released-through, both architectures' builds on their native runners, their smoke tests, vulnerability scans and SBOMs. Innpm-publish: the tag and migration gates and the five platform builds with their smoke tests and launcher suites. Thepublish/publish-candidatejob carriesif: ${{ !inputs.dry_run }};propagateandattesthave noifof their own and are skipped because the job they need is skipped. So no environment asks for approval and nothing reaches Docker Hub, GHCR, ECR Public, npm or Rekor. Dry runs have a concurrency group of their own (-dry), so a dry run cannot cancel a pending release. Adry-run-summaryjob writes the tag, commit and version to the step summary. Underdry_runthe build matrices run withfail-fast: false, so a rehearsal reports every architecture and platform where a release stops at the first. Dispatching without the input is unchanged.devtools/release-dry-run. The same gate and build steps on a developer's machine, for the host's architecture: the clean-tree, licence andcheck-released-throughgates, then for each of thepostgres,mongodb,cassandraanddevimages thedocker buildwith the release's build-args (VERSIONfromCargo.toml,VCS_REFthe commit,BUILD_DATEthe commit's timestamp,--platformthe host's), the architecture check, the scan and SBOM through the release's pinned trivy image with the release's flags and the release's SBOM validation, and the image'sci/smoke-test-*-container.sh.--tag vX.Y.Zadds the tag gates, read from the remote (ls-remote,FETCH_HEAD) so a local tag or branch cannot stand in;--cratesaddsrelease-crates.yml's publish plan andverifyjob (cargo package --workspace --locked,check-crate-packages, the SHA-256 record of each package);--imagespicks a subset ornone;--keepand--reuse-imageslet a smoke test be re-run without a rebuild, and an image the run did not build is never removed. Images are taggedextenddb-dry-run/<image>:<commit>-<arch>, a name no release uses; logs, tarballs and SBOMs go undertarget/release-dry-run/. Every step runs in a subshell with errexit and the summary lists each outcome with its log; the exit code is 1 when anything failed, or--stop-on-failstops at the first failure as the real gate does.packaging/README.md: the release flow gains the rehearsal as its second step, and the later steps are renumbered.Why
The release is five manual dispatches against one tag, and the Cassandra image has never been released (the
extenddb/extenddb-cassandrarepository on Docker Hub, registered on 4 November, has no tags; v0.1.14 shipped postgres, mongodb, dev and npm). Before 1.0 the team wants to run the release path across all four backends and see what fails, without consuming a version on npm or moving a Docker Hub tag.release-imageis already candidate-first, but it still pushes, signs, mirrors and attests;release-dev-imagepublishes the version tag and moveslatest;npm-publishconsumes the version number. So a throwaway tag is not a rehearsal.The gate and build jobs were already side-effect free; the input only has to stop the run there.
Closes: n/a
Testing done
devtools/release-dry-run --keep --crates(withPYTHONpointing at a 3.12) onmainat 5804374, x86_64, 16 cores:RELEASED_THROUGHis 0 and the last migration is 10 (the release PR raises it; see below)Dockerfile.cassandradoes not usecargo auditable), smoke (178 s)libssl33.0.20-1deb12u2, CVE-2026-84782 HIGH, fixed in 3.0.22-1deb12u2.Dockerfile.dev's runtime stage isgcr.io/distroless/cc-debian12:nonrootpinned by digest with noRUNstep, so a rebuild keeps thislibssl3; the pinned image and the currentnonrootbuild both carry 3.0.20-1deb12u2 (Lee's check). The dev image passes its scan only once the pin moves to a distroless build with 3.0.22-1deb12u2, which is a 1.0 blocker to trackcargo package --workspace --locked(475 s), package rulesThe two failures are the release's real state today, not the script's.
bash -nandshellcheck -S warningare clean. The three workflows parse (yaml.safe_load), each hasdry_runin its inputs, and in each the publish job carriesif: ${{ !inputs.dry_run }}whilepropagateandattestkeep theirneedson it, so they skip with it. Lee dispatched the three workflows from this branch withdry_runagainst v0.1.14 (runs 43422561132, 43422561669, 43422518630): the gates, builds and summaries passed onrelease-imageandnpm-publish,release-dev-image's builds failed the scan on thelibssl3finding above, every publish-side job was skipped and no environment asked for approval.--tag v0.1.14was run locally against a commit it does not name (below).Review round (2026-11-12): two hostile reviews of the first push
Two independent reviews of
eeabc50d, one on the workflows and release safety, one on the script's shell and its fidelity to the workflows. Neither found a way fordry_runto publish, sign, mirror or attest (jobifs,needsskipping, boolean input, environments on skipped jobs). What they found, all fixed in5e8ebcb4:Workflows:
dry_rundescription and the README step promised scans and SBOMs thatnpm-publishhas none of; each workflow's description now says what it runs.${{ inputs.tag }}intorun:, the only raw dispatch input to reach a shell in these files; it printsv${{ needs.gate.outputs.version }}, which the gate validated.fail-fast: ${{ !inputs.dry_run }}on the image build matrices and the npm matrix call, so a rehearsal reports every leg.release-crates,promote-imageandsign-imagehave nodry_run); the dev summary hard-coded the repo instead ofenv.IMAGE_REPO; the headers now say a dry run holds the workflow's concurrency group.Script:
if), so a failedgit fetchor trivy run was reported as a later symptom; each step now runs in( set -e; ... ).--stop-on-failprinted the summary before the failing log's tail; the tail comes first.extenddb/extenddb-<image>:sha-<sha>-<arch>), so a pulled candidate could be retagged, rehearsed as a build and removed; the name is nowextenddb-dry-run/<image>:<commit>-<arch>, and only images this run built are removed.--reuse-imageswithout--keepdeleted the reused image; an unknown--imagesvalue aborted after the gates with no summary; a dirty tree was rehearsed as the commit (a clean-tree gate now fails it);--images ""passed with nothing run (noneis explicit).cargo build) now reports the warning the release would.--cratesomitted--version, the SHA-256 record step and the toolchain note; the tag gate fetched into local refs (a differing local tag was reported as missing on origin) and readorigin/mainthrough the remote's refspec; no--platformondocker build; the work directory under$TMPDIRis not shared with the daemon on some hosts (now undertarget/); prerequisites missed the daemon, compose v2, cargo-about,timeoutand$PYTHON;--tagas the last argument died with an unbound-variable message; the help range was a hard-coded line span.Re-run of the rewritten script on
5e8ebcb4with--reuse-images --keep --crates: the same outcomes as the table above, plus the clean-tree gate and the SHA-256 record passing, and the Cassandra SBOM step now printing the no-crates warning.--tag v0.1.14against this commit fails with "tag v0.1.14 is b5bffa3 on origin; the checked-out commit is 5e8ebcb", the ls-remote path.--stop-on-failon a dirty tree prints thegit statuslines and then the summary.Review round two (2026-11-13): Lee's review of
5e8ebcb4Required, all fixed in this head:
runignored errexit (Blocking): the subshell was the condition of anif, where bash ignoresset -einside it too; a step whose first command failed ran on and was recorded PASS (reproduced on bash 4.2.46 with afalse; echostep). The subshell now runs outside any condition,set +e; ( set -e; "$@" ); status=$?; set -e, and the probe step is recorded FAIL with its later command not run.--cratesused the host's toolchain:release-crates.ymlruns every cargo onRUST_TOOLCHAIN(1.97.0). The script reads that pin from the workflow, exportsRUSTUP_TOOLCHAIN, installs the toolchain if absent, fails the newcrates: toolchainstep whenrustc --versionis not the pin, and writes the pin into the record'sTOOLCHAINfile. The SHA-256 of every package differs between 1.97.0 and this host's 1.98.0cargo package, so the pin matters for the record too.devtools/release-dry-runon the release commit before the tag is pushed (a failure there costs a commit, not a version), and thatprotect-release-tagsblocks any update of av*tag, so a rehearsal failure that needs a code change after the push takes the next patch version. The "a maintainer dispatches the workflows" sentence moved from step 1 to step 3.npm-publishruns and of whypropagateandattestskip, Why's Docker Hub repository, Testing done's "not run").Minor, also fixed:
<group>-dryasproject-status-sweep.ymldoes, so a dry run cannot cancel a pending release; the header sentences about holding the group are gone.docker build --no-cache, as the release's cache-less build, so a rerun installs the base image's current package updates where the Dockerfile upgrades them; the header says so and points at--reuse-imagesfor a cheap rerun. (A--no-cachedev build took 548 s here where the cached replay took 2 s.)--reuse-imagesno longer removes a kept image: the build moves the tag, the run records<new id>|<tag>|<previous id>for the images it built, removes those by id at the end and gives the kept image its tag back. Lee's reproduction (--images dev --skip-scanafter a--keeprun) now leaves the kept image with its tag.devtools/check-image-sbom SBOM DOCKERFILE, that both image workflows' SBOM steps andrelease-dry-runcall (annotations underGITHUB_ACTIONS, plain text otherwise;check_image_sbom_test.py, 10 cells, runs inscripts-test.yml). The build jobs check out the tagged commit, so a tag from before this PR has no script; the step says so and fails. The trivy pin is read fromrelease-image.ymlby the script, leaving the two workflow copies.Not in this PR, from the review: #458 also edits
npm-publish.ymland would merge with twoif:keys onpublish; Lee will fix that in #458 if this merges first. ECR Public has noextenddb-cassandrarepository, so the first Cassandra release would fail inpropagateafter the candidate is pushed and signed; a dry run stops before that job and cannot show it. Both belong on the 1.0 checklist.Checklist
cargo test --workspace): no Rust changedcargo fmt --check): no Rust changedscripts-test.ymlruns only the Python devtools' unit testsADR / RFC: n/a
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.