From 648680c1e5e49c36e8523165394e5cee83824c62 Mon Sep 17 00:00:00 2001 From: Brenton Strickler Date: Thu, 8 Oct 2026 17:06:37 -0400 Subject: [PATCH 1/3] ci(release): build without credentials; publish from a job that runs no project code The publish job imported the signing key and held the Maven Central token while it ran the project's Maven build (twice), its plugins and its dependencies. The native library came from a cache entry that a run on the default branch could have written, and every job had a token that could write to the repository. Split the workflow so that the build and the credentials never share a job: * native and build compile the tagged commit with a read-only token, no environment and no secrets, and hand over unsigned artifacts with digests. Nothing is restored from a cache. tantivy4java is built from the commits pinned in native-pins.txt; the build stops if the tag has moved, if Cargo.lock resolves the quickwit or tantivy fork to another commit, or if the build rewrites Cargo.lock. * assemble accepts only the expected coordinates and file names and writes a manifest of digests. rehearse verifies, signs, bundles and checks with a throwaway key. * publish, the only job in the release environment, verifies the manifest digest reported by assemble, signs with gpg, uploads one bundle per version through the Central Portal API (USER_MANAGED, as before) and attaches the shaded jars to the GitHub Release. It runs the release scripts and tools from the runner image; no Maven, no JVM, nothing from the tagged commit. The workflow is started by workflow_dispatch only, and dry-run is the default. A release is dispatched from main for a tag whose commit is on main, so the workflow file and the scripts that run beside the secrets are the reviewed ones. Pushing a tag no longer starts a release; plan.sh still handles that event if the trigger is added back. Published coordinates, classifiers and file sets are unchanged. The Spark version in the artifact version is now read from pom.xml at the tag instead of being repeated in the build matrix, where it had fallen behind (3.5.8 in the matrix against spark.version 3.5.9). release-scripts.yml runs the offline tests for the scripts and for the shape of release.yml on pull requests, with the runner's gpg. --- .claude/CLAUDE.md | 11 +- .github/scripts/release/README.md | 208 ++++ .github/scripts/release/assemble.sh | 213 ++++ .github/scripts/release/build-artifacts.sh | 123 +++ .github/scripts/release/build-native.sh | 271 ++++++ .github/scripts/release/central-upload.sh | 219 +++++ .github/scripts/release/check-bundles.sh | 90 ++ .github/scripts/release/github-release.sh | 86 ++ .github/scripts/release/lib.sh | 140 +++ .github/scripts/release/native-pins.txt | 22 + .github/scripts/release/plan.sh | 94 ++ .github/scripts/release/sign-bundles.sh | 51 + .github/scripts/release/signing-key.sh | 58 ++ .github/scripts/release/test.sh | 1020 ++++++++++++++++++++ .github/scripts/release/verify-staging.sh | 57 ++ .github/workflows/release-scripts.yml | 56 ++ .github/workflows/release.yml | 501 ++++++++-- 17 files changed, 3133 insertions(+), 87 deletions(-) create mode 100644 .github/scripts/release/README.md create mode 100644 .github/scripts/release/assemble.sh create mode 100644 .github/scripts/release/build-artifacts.sh create mode 100644 .github/scripts/release/build-native.sh create mode 100644 .github/scripts/release/central-upload.sh create mode 100644 .github/scripts/release/check-bundles.sh create mode 100644 .github/scripts/release/github-release.sh create mode 100644 .github/scripts/release/lib.sh create mode 100644 .github/scripts/release/native-pins.txt create mode 100644 .github/scripts/release/plan.sh create mode 100644 .github/scripts/release/sign-bundles.sh create mode 100644 .github/scripts/release/signing-key.sh create mode 100644 .github/scripts/release/test.sh create mode 100644 .github/scripts/release/verify-staging.sh create mode 100644 .github/workflows/release-scripts.yml diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index fae003b9..5faef9bb 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -59,9 +59,14 @@ property-only Maven profiles. Released artifacts encode the Spark version in the - **Always `mvn clean` when switching Spark profiles**: the ANTLR-generated parser sources are not regenerated on profile switch, and 4.9.3-generated code fails at runtime against the 4.13.1 runtime ("Could not deserialize ATN with version 3"). -- **Release**: pushing a `v*` tag runs `.github/workflows/release.yml` — builds tantivy4java - (linux-x86_64) once, then publishes each Spark leg to Maven Central and attaches shaded jars - to a GitHub Release. For local builds of all three legs: +- **Release**: run `.github/workflows/release.yml` from `main` with an existing `v*` tag + (Actions → Release → Run workflow). `dry-run` is ticked by default and publishes nothing; + untick it to publish. Pushing a tag does not start a release. The run builds tantivy4java + (linux-x86_64) from source at the commits pinned in + `.github/scripts/release/native-pins.txt` (add a line there when bumping tantivy4java in + `pom.xml`), builds each Spark leg unsigned, and only then signs and uploads from a separate + job that runs no project code. Runbook: `.github/scripts/release/README.md`. For local + builds of all three legs: `dev/build-release.sh ` (shaded jars land in `dist/`, platform classifier auto-detected for the current machine). diff --git a/.github/scripts/release/README.md b/.github/scripts/release/README.md new file mode 100644 index 00000000..f89d4d17 --- /dev/null +++ b/.github/scripts/release/README.md @@ -0,0 +1,208 @@ +# Release scripts + +The scripts behind `.github/workflows/release.yml`, and the runbook for a +release. + +## What a release publishes + +For a tag `v` (for example `v0.6.0`), one artifact version per Spark +profile, `_spark_`: + +| | | +|---|---| +| Maven Central | `io.indextables:indextables_spark:_spark_`: the pom, the jar, `-sources`, `-javadoc`, `-linux-x86_64-shaded` and `-jar-with-dependencies`, each with a signature and md5, sha1, sha256 and sha512 checksums | +| GitHub Release for the tag | the `-linux-x86_64-shaded` jar of each version | + +`` is the `spark.version` the profile sets in `pom.xml` at the +tagged commit, so the suffix names the Spark version the jar was compiled +against. The file set is the one on Maven Central for `0.6.0-rc2`; the +allow-list is `ARTIFACT_SUFFIXES` in `lib.sh`. + +## Jobs and trust + +``` +plan ─► native ─► build (x3) ─► assemble ─► rehearse ─► publish + └──── run project code ────┘ └── run these scripts only ──┘ + no secrets, read-only token environment: release +``` + +| Job | Runs | Has | +|---|---|---| +| `plan` | `plan.sh` | read-only token | +| `native` | `build-native.sh`: tantivy4java's Maven and cargo build, from pinned commits, no cache | read-only token | +| `build` | `build-artifacts.sh`: this project's Maven build at the tagged commit, with the release profile and signing switched off | read-only token | +| `assemble` | `assemble.sh`: checks the build outputs, lays out what will be uploaded, writes a manifest of digests | read-only token | +| `rehearse` | `verify-staging.sh`, `signing-key.sh generate`, `sign-bundles.sh`, `check-bundles.sh` with a throwaway key | read-only token | +| `publish` | `verify-staging.sh`, `signing-key.sh import`, `sign-bundles.sh`, `check-bundles.sh`, `central-upload.sh`, `github-release.sh` | `release` environment, the four secrets, `contents: write` | + +What executes in `publish`, beside the signing key and the Central token: + +* `actions/checkout` (sparse, this directory, from the commit the workflow + was started on) and `actions/download-artifact`, both pinned to a commit; +* the scripts in this directory, from that same commit; +* `bash`, coreutils, `gpg`, `zip`, `unzip`, `curl`, `jq` and `gh` from the + runner image. + +No Maven, no JVM, no cargo, and nothing from the tagged commit. The files it +signs were produced by jobs that did run project code; it signs exactly the +files in the manifest whose digest `assemble` reported as a job output, and +`assemble` and `verify-staging.sh` accept only the expected coordinates and +file names. A build cannot add another artifact, another version or another +`io.indextables` coordinate to what gets signed. What a build can still do is +put different bytes inside the six expected files: that is what the tagged +commit's review, the pins and the digests in the run summary are for. + +Each secret is passed to the one step that needs it. The key is imported into +a keyring under the runner's temporary directory, and removed before the +first request to Maven Central or GitHub. + +## Native inputs + +`native-pins.txt` lists, per tantivy4java version, the tantivy4java commit +and the quickwit and tantivy fork commits a release may build. The build +fetches that tantivy4java commit by id, refuses to continue if the tag +`v` points anywhere else, and checks the fork commits against the +`Cargo.lock` at that commit. It also fails if the build modifies `Cargo.lock` +(cargo does that when it cannot use the locked versions), if a dependency +points to a path outside the checkout, or if a cargo config overrides +dependency sources. protoc is the same pinned release and digest as in +`scripts/setup.sh`. Rust comes from the runner image and is recorded, not +pinned. + +The resolved commits and tool versions are written to the `native` job +summary, the `assemble` job summary and the release notes. + +When `pom.xml` moves to a new tantivy4java version, add its line in the same +pull request: + +``` +bash .github/scripts/release/build-native.sh print-pin +``` + +The `Release Scripts` workflow fails on a pull request that changes the +version without adding a pin. + +`scripts/setup.sh` is not used by the release. It still clones the quickwit +fork at its default branch for developer and CI builds; current tantivy4java +versions take both forks from `Cargo.lock` and ignore that clone. + +## Settings this depends on + +These are repository settings, not files, and the workflow cannot enforce +them on itself: + +1. Environment `release`: required reviewer set; deployment branches and tags + limited to **`main` only**. With `v*` tags also admitted, a workflow file + on any `v*` tag can ask for this environment (see below). +2. `GPG_PRIVATE_KEY`, `GPG_PASSPHRASE`, `CENTRAL_USERNAME` and + `CENTRAL_PASSWORD` stored as secrets **of that environment**, and the + repository-level secrets of the same names deleted. While a + repository-level copy exists, every workflow in the repository can read it + and none of the separation above applies. +3. `main` protected, so that the workflow file and these scripts change only + through reviewed pull requests. + +## Tag push or dispatch + +The workflow is started by `workflow_dispatch` only. A real release must be +dispatched from `main`, with the tag as input; the tag's commit must be on +`main`. + +* **Dispatch from `main`** (implemented): the workflow file and the scripts + that run beside the secrets are the reviewed ones on `main`. The tag only + selects which source is built, by the unprivileged jobs. Left open: whoever + can dispatch workflows chooses the tag (bounded by "must be on `main`") and + whoever approves the environment is the last check; and the guarantee rests + on settings 1 and 3 above. +* **Tag push** (not enabled): pushing `v*` runs the workflow file and scripts + *of the tagged commit*. Anyone who can push a `v*` tag, on any commit, + merged or not, then decides what the job holding the secrets executes; the + only control left is the reviewer, who approves a run without seeing which + workflow file it uses. To enable it anyway, add `push: tags: ['v*']` under + `on:` in `release.yml` (`plan.sh` already handles the event) and admit `v*` + tags in the environment. + +Creating a GitHub Release (and with it the tag) in the browser therefore does +not publish anything. Do that first if you want hand-written notes, then +dispatch the workflow: it attaches the jars to the existing release and adds +a "Build inputs" block to its notes without touching the rest. + +## Runbook + +### Dry run + +Actions → Release → Run workflow → branch `main`, `tag` = the tag, leave +`dry-run` ticked. Any existing `v*` tag on `main` whose tantivy4java version +is pinned works, including one that is already released (`v0.6.0-rc2`). + +A dry run builds everything, assembles it, and signs, bundles and checks it +with a throwaway key. It does not start the `publish` job: it asks for no +approval, reads no secret, uploads nothing, creates no release. If a run you +meant as a dry run shows a pending approval for `release`, it is not a dry +run: reject it. + +What to look at: + +* **Plan** summary: "Dry run for …", the tag and its commit. +* **native** summary: the tantivy4java, quickwit and tantivy commits. +* **Assemble** summary: three versions with the Spark versions you expect; + the manifest digest; the digests of all files. +* **Rehearse** log: each bundle lists 36 entries and passes. + +A dry run may also be dispatched from a branch, to try a change to the +workflow or these scripts before it is merged. + +### Release + +1. Dispatch from `main` with `dry-run` unticked. +2. Wait for `rehearse` to pass, read the Plan and Assemble summaries, then + approve the `release` environment. Up to here nothing has left the run; + rejecting the approval ends it. +3. `publish` signs, uploads one bundle per version to the Central Portal as + `USER_MANAGED`, waits until each is `VALIDATED`, then creates or updates + the GitHub Release. +4. Open . There is one + deployment per version, named `indextables_spark-`. Press + **Publish** on each. This is the step that cannot be undone. + +### Aborting + +| When | How | What is left behind | +|---|---|---| +| Before approval | Reject the approval, or cancel the run | Nothing | +| While `publish` runs | Cancel the run | The job drops the deployments it uploaded; check the Portal. The GitHub Release may exist with some jars | +| After `publish`, before pressing Publish | **Drop** every deployment in the Portal; delete or edit the GitHub Release by hand | Nothing on Maven Central | +| After pressing Publish | Not possible. Publish the remaining versions, and release a new version if this one is wrong | | + +### When `publish` fails + +`central-upload.sh` handles one version at a time and drops what it uploaded +when anything fails, so a failed job normally leaves nothing in the Portal; +if a drop is refused the log says which deployment to drop by hand. Then use +**Re-run failed jobs**: the re-run signs and uploads the same staged files +(same digests), not a rebuild. + +If some versions were already published by hand and others were not, re-run +the failed job as well: a version that is already on Maven Central with +identical files is skipped, and one with different files stops the run. + +## What the tests cover, and what they cannot + +`bash .github/scripts/release/test.sh` runs offline: `mvn`, `curl` and `gh` +are stubs. It covers the decision logic of every script (including every +refusal), the layout of the bundles, the upload state machine (rejection, +timeouts, drops, re-runs) and the shape of `release.yml` (which job has +secrets, the dry-run guards, pins). On a pull request it runs on the runner +image with the real `gpg`, generating, exporting, importing and signing with +a throwaway key. + +Not covered by any test, and seen for the first time in a real run: + +* the Maven and cargo builds themselves (first seen in a dry run); +* artifacts passing between jobs (first seen in a dry run); +* importing the real key and using the real passphrase; +* the Central Portal accepting the token, the bundle and the signatures; +* `gh` creating or updating the release with the job's token. + +The last three happen in `publish` only. If one of them fails, nothing has +been published: the upload is `USER_MANAGED`, and a failed job drops it. diff --git a/.github/scripts/release/assemble.sh b/.github/scripts/release/assemble.sh new file mode 100644 index 00000000..1de11a95 --- /dev/null +++ b/.github/scripts/release/assemble.sh @@ -0,0 +1,213 @@ +#!/usr/bin/env bash +# Turn the build jobs' outputs into the exact trees that will be signed and +# uploaded, after checking them. Runs no project code: it only reads files. +# +# Everything in IN was produced by jobs that ran the project's build, so it is +# treated as untrusted until checked. What comes out is an allow-list result: +# only the expected coordinates, only the expected file names. +# +# Environment: +# IN directory with the downloaded artifacts: +# native/ build-native.sh output +# artifacts-/ build-artifacts.sh output, per profile +# OUT staging directory to create: +# versions.txt the versions, one per line +# release-notes.md the "Build inputs" block +# bundles// Maven repository layout for that +# version: the six files and their +# md5/sha1/sha256/sha512 (no signatures) +# MANIFEST.sha256 SHA-256 of every file above +# TAG, COMMIT, BASE_VERSION from plan.sh +# RUN_URL link to this workflow run (optional, for the notes) +# +# Step outputs: manifest_sha256 (digest of MANIFEST.sha256), versions. +set -euo pipefail +here="$(cd "$(dirname "$0")" && pwd)" +# shellcheck source=lib.sh +. "$here/lib.sh" + +: "${IN:?}" "${OUT:?}" "${TAG:?}" "${COMMIT:?}" "${BASE_VERSION:?}" +need unzip python3 +[[ "$TAG" =~ $TAG_RE ]] || die "'$TAG' is not a release tag" +[ "v$BASE_VERSION" = "$TAG" ] || die "base version $BASE_VERSION does not belong to tag $TAG" +[[ "$COMMIT" =~ $SHA_RE ]] || die "'$COMMIT' is not a commit id" + +# check_pom : coordinates must be the ones this repository +# publishes, and the metadata Maven Central requires must be present. +check_pom() { + python3 - "$1" "$GROUP_ID" "$ARTIFACT_ID" "$2" << 'PY' || die "$(basename "$1") failed the pom checks (above)" +import sys +import xml.etree.ElementTree as ET + +path, group, artifact, version = sys.argv[1:5] +ns = "{http://maven.apache.org/POM/4.0.0}" +try: + root = ET.parse(path).getroot() +except ET.ParseError as exc: + sys.exit("pom is not well-formed XML: %s" % exc) + +def text(*names): + node = root + for name in names: + node = node.find(ns + name) if node is not None else None + return (node.text or "").strip() if node is not None else "" + +problems = [] +if root.tag != ns + "project": + problems.append("root element is not a Maven ") +if root.find(ns + "parent") is not None: + problems.append("has a ; coordinates must be stated in the pom itself") +for names, want in ((("groupId",), group), (("artifactId",), artifact), (("version",), version)): + got = text(*names) + if got != want: + problems.append("%s is '%s', expected '%s'" % ("/".join(names), got, want)) +if text("packaging") not in ("", "jar"): + problems.append("packaging is '%s', expected jar" % text("packaging")) +required = (("name",), ("description",), ("url",), + ("licenses", "license", "name"), ("licenses", "license", "url"), + ("developers", "developer", "name"), + ("scm", "connection"), ("scm", "url")) +for names in required: + if not text(*names): + problems.append("missing %s (required by Maven Central)" % "/".join(names)) +if problems: + sys.exit("\n".join(problems)) +PY +} + +# --- the native build --------------------------------------------------------- +native="$IN/native" +verify_sums "$native" +ninfo="$native/native-inputs.txt" +t4j_version="$(kv "$ninfo" tantivy4java_version)" +[[ "$t4j_version" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || die "the native artifact does not state a tantivy4java version" +[ "$(kv "$ninfo" cargo_lock_unchanged)" = true ] || die "the native build does not record an unchanged Cargo.lock" +t4j_jar="$native/repository/tantivy4java-$t4j_version-$NATIVE_CLASSIFIER.jar" +[ -f "$t4j_jar" ] || die "the native artifact has no $(basename "$t4j_jar")" +t4j_sha="$(hash_of sha256 "$t4j_jar")" +libs="$(unzip -Z1 "$t4j_jar" | grep -E '\.so$' || true)" +[ -n "$libs" ] || die "$(basename "$t4j_jar") contains no native library (*.so)" + +rm -rf "$OUT" +mkdir -p "$OUT/bundles" +: > "$OUT/versions.txt" + +# --- each Spark profile ------------------------------------------------------- +for profile in $PROFILES; do + d="$IN/artifacts-$profile" + [ -d "$d" ] || die "no build output for $profile" + verify_sums "$d" + info="$d/build-info.txt" + [ "$(kv "$info" profile)" = "$profile" ] || die "artifacts-$profile was not built with profile $profile" + version="$(kv "$info" version)" + check_version "$BASE_VERSION" "$profile" "$version" + [ "${BASE_VERSION}_spark_$(kv "$info" spark_version)" = "$version" ] \ + || die "artifacts-$profile: version $version does not match its recorded Spark version" + + expected="$({ artifact_names "$version" | sed 's|^|files/|'; echo build-info.txt; } | LC_ALL=C sort)" + actual="$(cd "$d" && find . -type f ! -path ./SHA256SUMS | sed 's|^\./||' | LC_ALL=C sort)" + [ "$expected" = "$actual" ] \ + || die "artifacts-$profile does not hold exactly the expected files. Expected: $(echo $expected). Found: $(echo $actual)" + + [ "$(kv "$info" tantivy4java_jar_sha256)" = "$t4j_sha" ] \ + || die "artifacts-$profile was not built with this run's tantivy4java jar" + + p="$ARTIFACT_ID-$version" + check_pom "$d/files/$p.pom" "$version" + for name in $(artifact_names "$version"); do + case "$name" in + *.jar) unzip -tqq "$d/files/$name" > /dev/null 2>&1 || die "$name is not a readable jar" ;; + esac + done + + # The native library inside the published jars must be, byte for byte, the + # one this run built from the pinned commits. + for jar in "$p-$NATIVE_CLASSIFIER-shaded.jar" "$p-jar-with-dependencies.jar"; do + while IFS= read -r lib; do + want="$(unzip -p "$t4j_jar" "$lib" | hash_stream sha256)" + got="$(unzip -p "$d/files/$jar" "$lib" 2> /dev/null | hash_stream sha256 || true)" + [ "$got" = "$want" ] || die "$jar does not contain this run's $lib" + done <<< "$libs" + done + + vd="$OUT/bundles/$version/$GROUP_PATH/$version" + mkdir -p "$vd" + for name in $(artifact_names "$version"); do + cp "$d/files/$name" "$vd/$name" + for algo in $CHECKSUM_ALGOS; do + printf '%s' "$(hash_of "$algo" "$vd/$name")" > "$vd/$name.$algo" + done + done + echo "$version" >> "$OUT/versions.txt" +done + +# --- notes ---------------------------------------------------------------------- +# The values below were written by the native job, which runs build code, and +# they end up in the release notes. Identifiers must have their exact form; +# free-text tool versions are shown only if they are plain text. +SAFE_RE='^[A-Za-z0-9 ._,:/@()+="-]*$' +safe() { + if [ "${#1}" -le 200 ] && [[ "$1" =~ $SAFE_RE ]]; then printf '%s' "$1"; else printf '(not shown: unexpected characters)'; fi +} +[[ "$(kv "$ninfo" tantivy4java_tag)" == "v$t4j_version" ]] || die "the native artifact's tag is not v$t4j_version" +[[ "$(kv "$ninfo" tantivy4java_commit)" =~ $SHA_RE ]] || die "the native artifact does not state a tantivy4java commit" +for key in quickwit_commit tantivy_commit; do + value="$(kv "$ninfo" "$key")" + [ "$value" = "-" ] || [[ "$value" =~ $SHA_RE ]] || die "the native artifact's $key is not a commit id" +done +[[ "$(kv "$ninfo" cargo_lock_sha256)" =~ ^[0-9a-f]{64}$ ]] || die "the native artifact does not state a Cargo.lock digest" +{ + echo "" + echo "### Build inputs" + echo + if [ -n "${RUN_URL:-}" ]; then + echo "Built from commit \`$COMMIT\` (tag \`$TAG\`) by [this workflow run]($RUN_URL)." + else + echo "Built from commit \`$COMMIT\` (tag \`$TAG\`)." + fi + echo "The native library was compiled in that run from the commits below; nothing was restored from a cache." + echo + echo "| Input | Value |" + echo "|---|---|" + echo "| tantivy4java | \`$(kv "$ninfo" tantivy4java_tag)\` at \`$(kv "$ninfo" tantivy4java_commit)\` |" + echo "| quickwit fork | \`$(kv "$ninfo" quickwit_commit)\` |" + echo "| tantivy fork | \`$(kv "$ninfo" tantivy_commit)\` |" + other="$(kv "$ninfo" other_git_dependencies)" + [ -z "$other" ] || echo "| other git dependencies | \`$(safe "$other")\` |" + echo "| Cargo.lock | SHA-256 \`$(kv "$ninfo" cargo_lock_sha256)\`, unchanged by the build |" + for key in rustc cargo protoc java maven runner_image; do + value="$(kv "$ninfo" "$key")" + [ -z "$value" ] || echo "| ${key//_/ } | \`$(safe "$value")\` |" + done + echo + echo "| Version | SHA-256 of \`-$NATIVE_CLASSIFIER-shaded.jar\` |" + echo "|---|---|" + while IFS= read -r version; do + f="$OUT/bundles/$version/$GROUP_PATH/$version/$ARTIFACT_ID-$version-$NATIVE_CLASSIFIER-shaded.jar" + echo "| \`$version\` | \`$(hash_of sha256 "$f")\` |" + done < "$OUT/versions.txt" + echo "" +} > "$OUT/release-notes.md" + +write_sums "$OUT" MANIFEST.sha256 +manifest_sha="$(hash_of sha256 "$OUT/MANIFEST.sha256")" +output manifest_sha256 "$manifest_sha" +output versions "$(tr '\n' ' ' < "$OUT/versions.txt" | sed 's/ $//')" + +{ + echo "## Staged for $TAG" + echo + echo "Manifest SHA-256: \`$manifest_sha\`. The publish job signs and uploads exactly the files this manifest lists, and refuses anything else." + echo + sed -e '/^" { skip = 1 } + !skip { + if ($0 ~ /^[[:space:]]*$/) { blank = blank $0 "\n" } else { printf "%s%s\n", blank, $0; blank = "" } + } + $0 == "" { skip = 0 } +' > "$tmp/notes.md" +if [ -s "$tmp/notes.md" ]; then echo >> "$tmp/notes.md"; fi +cat "$staging/release-notes.md" >> "$tmp/notes.md" +gh release edit "$TAG" --notes-file "$tmp/notes.md" > /dev/null + +url="$(gh release view "$TAG" --json url --jq '.url' 2> /dev/null || true)" +{ + echo "### GitHub Release" + echo + echo "${url:-Release $TAG}: ${#assets[@]} shaded jars attached." +} | summary diff --git a/.github/scripts/release/lib.sh b/.github/scripts/release/lib.sh new file mode 100644 index 00000000..b4983649 --- /dev/null +++ b/.github/scripts/release/lib.sh @@ -0,0 +1,140 @@ +# Shared by the release scripts. Sourced, never executed. +# Works with bash 3.2 or newer and with either GNU or BSD userland, so that +# test.sh can run the same code on a developer machine and on the runner. + +# What is published. Changing any of these changes the published coordinates. +GROUP_ID=io.indextables +ARTIFACT_ID=indextables_spark +GROUP_PATH=io/indextables/indextables_spark +NATIVE_CLASSIFIER=linux-x86_64 + +# One artifact version is published per Spark profile. Must list the same +# profiles, in the same order, as the build matrix in release.yml. +PROFILES="spark-3.5 spark-4.0 spark-4.1" + +# The files that make up one published version, as suffixes of +# -. This is an allow-list: a version directory holding +# any other file is refused. +ARTIFACT_SUFFIXES=".pom .jar -sources.jar -javadoc.jar -${NATIVE_CLASSIFIER}-shaded.jar -jar-with-dependencies.jar" +CHECKSUM_ALGOS="md5 sha1 sha256 sha512" + +# v.. with an optional pre-release part (v0.6.0-rc2). +TAG_RE='^v[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z]+(\.[0-9A-Za-z]+)*)?$' +BASE_RE='^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z]+(\.[0-9A-Za-z]+)*)?$' +SHA_RE='^[0-9a-f]{40}$' + +die() { echo "::error::$*" >&2; exit 1; } +warn() { echo "::warning::$*" >&2; } + +# need : fail unless every command is on PATH. +need() { + local c + for c in "$@"; do + command -v "$c" > /dev/null 2>&1 || die "required command not found: $c" + done +} + +# hash_stream : hex digest of standard input. +hash_stream() { + local out + case "$1" in + md5) + if command -v md5sum > /dev/null 2>&1; then out="$(md5sum)"; else out="$(md5 -q)"; fi ;; + sha1 | sha256 | sha512) + if command -v "${1}sum" > /dev/null 2>&1; then out="$("${1}sum")"; else out="$(shasum -a "${1#sha}")"; fi ;; + *) die "unknown digest algorithm: $1" ;; + esac + out="${out%% *}" + [[ "$out" =~ ^[0-9a-f]+$ ]] || die "could not compute a $1 digest" + printf '%s\n' "$out" +} + +# hash_of +hash_of() { + [ -f "$2" ] || die "no such file: $2" + hash_stream "$1" < "$2" +} + +# output : set a step output (single line) and echo it. +output() { + case "$2" in *$'\n'*) die "refusing to write a multi-line value for output '$1'" ;; esac + if [ -n "${GITHUB_OUTPUT:-}" ]; then printf '%s=%s\n' "$1" "$2" >> "$GITHUB_OUTPUT"; fi + printf 'output: %s=%s\n' "$1" "$2" +} + +# summary: append standard input to the job summary (or print it when there +# is none, as in test.sh). +summary() { + if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then cat >> "$GITHUB_STEP_SUMMARY"; else cat; fi +} + +# spark_line : "3.5" for "spark-3.5". Fails for an unknown profile. +spark_line() { + local p + for p in $PROFILES; do + if [ "$p" = "$1" ]; then printf '%s\n' "${1#spark-}"; return 0; fi + done + die "unknown Spark profile: $1 (expected one of: $PROFILES)" +} + +# check_version : the version must be +# _spark_ where x.y is the profile's Spark line. +check_version() { + local base="$1" line re + line="$(spark_line "$2")" || exit 1 + re="^${base//./\\.}_spark_${line//./\\.}\\.[0-9]+$" + [[ "$3" =~ $re ]] || die "version '$3' is not ${base}_spark_${line}. (profile $2)" +} + +# pom_tantivy4java_version : same extraction as scripts/setup.sh. +pom_tantivy4java_version() { + local v + v="$(grep -A2 'tantivy4java' "$1" | grep '' \ + | sed 's/.*\(.*\)<\/version>.*/\1/' | tr -d '[:space:]' || true)" + [[ "$v" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || die "could not read the tantivy4java version from $1 (got '$v')" + printf '%s\n' "$v" +} + +# kv : value of key=value in a build-info file (first match). +kv() { + sed -n "s/^$2=//p" "$1" | head -n 1 +} + +# artifact_names : the allow-listed file names of one version. +artifact_names() { + local sfx + for sfx in $ARTIFACT_SUFFIXES; do printf '%s-%s%s\n' "$ARTIFACT_ID" "$1" "$sfx"; done +} + +# write_sums [name]: write / (default SHA256SUMS) listing the +# SHA-256 of every other regular file under , in sha256sum format. +write_sums() { + local dir="$1" name="${2:-SHA256SUMS}" + [ -d "$dir" ] || die "no such directory: $dir" + ( + cd "$dir" || exit 1 + find . -type f ! -path "./$name" ! -path "./$name.tmp" | sed 's|^\./||' | LC_ALL=C sort \ + | while IFS= read -r path; do + printf '%s %s\n' "$(hash_of sha256 "$path")" "$path" + done > "$name.tmp" + mv "$name.tmp" "$name" + ) +} + +# verify_sums [name]: / must list exactly the regular files +# under (no more, no fewer, nothing that is not a regular file) and +# every digest must match. The directory may come from another job, so +# nothing in it is used before this passes. +verify_sums() { + local dir="$1" name="${2:-SHA256SUMS}" listed actual sum path + [ -f "$dir/$name" ] || die "$dir has no $name" + [ -z "$(find "$dir" ! -type f ! -type d | head -n 1)" ] \ + || die "$dir contains something that is neither a regular file nor a directory" + listed="$(awk '{ print $2 }' "$dir/$name" | LC_ALL=C sort)" + actual="$(cd "$dir" && find . -type f ! -path "./$name" | sed 's|^\./||' | LC_ALL=C sort)" + [ -n "$actual" ] || die "$dir is empty" + [ "$listed" = "$actual" ] || die "the files in $dir are not exactly the files listed in $name" + while read -r sum path; do + [ "$(hash_of sha256 "$dir/$path")" = "$sum" ] || die "digest mismatch for $path in $dir" + done < "$dir/$name" +} diff --git a/.github/scripts/release/native-pins.txt b/.github/scripts/release/native-pins.txt new file mode 100644 index 00000000..61c92291 --- /dev/null +++ b/.github/scripts/release/native-pins.txt @@ -0,0 +1,22 @@ +# What the release compiles for the tantivy4java native library. +# +# One line per tantivy4java version that pom.xml may name: +# +# +# +# build-native.sh fetches exactly the tantivy4java commit listed here, and +# refuses to build unless +# * the tag v in indextables/tantivy4java still points to it, and +# * the Cargo.lock at that commit resolves indextables/quickwit and +# indextables/tantivy to the two fork commits listed here. +# A fork column is "-" when that tantivy4java version does not use the fork. +# +# When pom.xml moves to a new tantivy4java version, add a line for it in the +# same pull request (keep the old lines: they are needed to dry-run or +# re-run older tags): +# +# bash .github/scripts/release/build-native.sh print-pin +# +# Review the printed commits before committing them; from then on they are +# what a release is allowed to build. +0.34.4 be2d885a5b6d2e7438d88ea7f9818c5086e74a52 3026b228a71bfb6be16f3966ff80c8641834cb21 d491895f1cd54d166c8010469d4625ae81c636f3 diff --git a/.github/scripts/release/plan.sh b/.github/scripts/release/plan.sh new file mode 100644 index 00000000..333f76d5 --- /dev/null +++ b/.github/scripts/release/plan.sh @@ -0,0 +1,94 @@ +#!/usr/bin/env bash +# Decide what this run does, before anything is built. +# +# Reads the triggering event and the dispatch inputs (as strings, from the +# environment), validates the tag, resolves it to a commit and checks that the +# commit is on the default branch. Sets the step outputs: +# +# mode dry-run | publish +# tag the release tag (v0.6.0) +# commit the commit the tag points to; every later job uses this +# base the tag without its "v" (0.6.0) +# prerelease true when the base version has a pre-release part (0.6.0-rc2) +# central true unless a publish run asked to skip Maven Central +# +# Only "publish" lets the publish job start, and only these two cases give it: +# * workflow_dispatch from the default branch with dry-run set to false +# * a pushed tag (not enabled in release.yml; see the README) +# Every other combination is a dry run or an error. +set -euo pipefail +here="$(cd "$(dirname "$0")" && pwd)" +# shellcheck source=lib.sh +. "$here/lib.sh" + +: "${EVENT_NAME:?}" "${REF:?}" "${DEFAULT_BRANCH:?}" +repo_dir="${REPO_DIR:-.}" +need git + +case "$EVENT_NAME" in + workflow_dispatch) + tag="${INPUT_TAG:-}" + case "${INPUT_DRY_RUN:-}" in + true) mode=dry-run ;; + false) + [ "$REF" = "refs/heads/$DEFAULT_BRANCH" ] \ + || die "A release can only be dispatched from $DEFAULT_BRANCH (this run is on $REF). Dry runs may be dispatched from any ref." + mode=publish ;; + *) die "dry-run must be true or false (got '${INPUT_DRY_RUN:-}')" ;; + esac + ;; + push) + [ "${REF_TYPE:-}" = tag ] || die "push runs are only supported for tags (this run is on $REF)" + tag="${REF_NAME:-}" + mode=publish + ;; + *) die "unsupported event: $EVENT_NAME" ;; +esac + +[[ "$tag" =~ $TAG_RE ]] || die "'$tag' is not a release tag (expected v.., optionally with a pre-release part such as -rc1)" +base="${tag#v}" + +commit="$(git -C "$repo_dir" rev-parse --verify --quiet "refs/tags/$tag^{commit}" || true)" +[[ "$commit" =~ $SHA_RE ]] || die "tag $tag does not exist in this repository" + +# The source that gets published must have been merged: the tag's commit has +# to be the tip of the default branch or one of its ancestors. +git -C "$repo_dir" rev-parse --verify --quiet "refs/remotes/origin/$DEFAULT_BRANCH^{commit}" > /dev/null \ + || die "origin/$DEFAULT_BRANCH is not available; the checkout must fetch full history" +git -C "$repo_dir" merge-base --is-ancestor "$commit" "refs/remotes/origin/$DEFAULT_BRANCH" \ + || die "tag $tag ($commit) is not an ancestor of $DEFAULT_BRANCH; only merged commits can be released" + +case "$base" in *-*) prerelease=true ;; *) prerelease=false ;; esac + +central=true +if [ "$mode" = publish ] && [ "${INPUT_SKIP_CENTRAL:-false}" = true ]; then central=false; fi + +output mode "$mode" +output tag "$tag" +output commit "$commit" +output base "$base" +output prerelease "$prerelease" +output central "$central" + +{ + if [ "$mode" = publish ]; then + echo "## Release $tag" + echo + if [ "$central" = true ]; then + echo "This run **publishes**: it uploads to Maven Central and creates or updates the GitHub Release, once the \`release\` environment has been approved." + else + echo "This run creates or updates the **GitHub Release only** (Maven Central was skipped by input), once the \`release\` environment has been approved." + fi + else + echo "## Dry run for $tag" + echo + echo "Nothing is uploaded and no release is created. The publish job is not started, so no approval is requested and no secret is read." + fi + echo + echo "| | |" + echo "|---|---|" + echo "| Tag | \`$tag\` |" + echo "| Commit | \`$commit\` |" + echo "| Versions | \`${base}_spark_\` |" + echo "| Workflow ref | \`$REF\` |" +} | summary diff --git a/.github/scripts/release/sign-bundles.sh b/.github/scripts/release/sign-bundles.sh new file mode 100644 index 00000000..060dba4d --- /dev/null +++ b/.github/scripts/release/sign-bundles.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# Sign every staged file and zip one upload bundle per version. +# +# sign-bundles.sh +# +# Environment: +# GNUPGHOME keyring holding the signing key (see signing-key.sh) +# SIGNING_KEY fingerprint of the key to sign with +# GPG_PASSPHRASE its passphrase (may be empty for an unprotected key) +# +# Writes .asc next to each of the six files of each version (checksum +# files are not signed, as before), then +# /indextables_spark--bundle.zip holding that version's +# Maven repository layout. Run verify-staging.sh first: this script signs +# whatever is in the staging directory. +# +# The same script runs in the rehearse job (throwaway key) and in the publish +# job (release key), so a dry run exercises the exact signing code. +set -euo pipefail +here="$(cd "$(dirname "$0")" && pwd)" +# shellcheck source=lib.sh +. "$here/lib.sh" + +staging="${1:?usage: sign-bundles.sh }" +out="${2:?usage: sign-bundles.sh }" +: "${GNUPGHOME:?}" "${SIGNING_KEY:?}" +export GNUPGHOME +passphrase="${GPG_PASSPHRASE:-}" +need gpg zip +[ -s "$staging/versions.txt" ] || die "$staging lists no versions" + +mkdir -p "$out" +out="$(cd "$out" && pwd)" + +while IFS= read -r version <&4; do + root="$staging/bundles/$version" + [ -d "$root/$GROUP_PATH/$version" ] || die "nothing staged for $version" + for name in $(artifact_names "$version"); do + f="$root/$GROUP_PATH/$version/$name" + [ -f "$f" ] || die "staged file missing: $name" + [ ! -e "$f.asc" ] || die "$name is already signed; sign-bundles.sh needs a clean staging directory" + gpg --batch --yes --no-tty --pinentry-mode loopback --passphrase-fd 3 \ + --local-user "$SIGNING_KEY" --armor --detach-sign --output "$f.asc" "$f" \ + 3<<< "$passphrase" < /dev/null \ + || die "could not sign $name" + done + zip="$out/$ARTIFACT_ID-$version-bundle.zip" + rm -f "$zip" + (cd "$root" && zip -q -r -X "$zip" "${GROUP_PATH%%/*}") + echo "Signed and bundled $version: $zip" +done 4< "$staging/versions.txt" diff --git a/.github/scripts/release/signing-key.sh b/.github/scripts/release/signing-key.sh new file mode 100644 index 00000000..754506e5 --- /dev/null +++ b/.github/scripts/release/signing-key.sh @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +# Manage the keyring used for signing, in a directory of its own. +# +# signing-key.sh import import the private key in GPG_PRIVATE_KEY +# signing-key.sh generate create a throwaway key protected by +# GPG_PASSPHRASE (rehearsal only) +# signing-key.sh remove stop the agent and delete the keyring +# +# GNUPGHOME must name a directory called "release-gnupg"; it is created empty +# by import/generate and deleted by remove. import and generate set the step +# output `fingerprint` to the primary key's fingerprint; the keyring must hold +# exactly one secret key. +set -euo pipefail +here="$(cd "$(dirname "$0")" && pwd)" +# shellcheck source=lib.sh +. "$here/lib.sh" + +: "${GNUPGHOME:?}" +[ "$(basename "$GNUPGHOME")" = release-gnupg ] || die "GNUPGHOME must be a directory named release-gnupg (got $GNUPGHOME)" +export GNUPGHOME + +fresh_home() { + rm -rf "$GNUPGHOME" + (umask 077 && mkdir -p "$GNUPGHOME") +} + +case "${1:-}" in + import) + need gpg + : "${GPG_PRIVATE_KEY:?}" + fresh_home + printf '%s\n' "$GPG_PRIVATE_KEY" | gpg --batch --quiet --no-tty --import \ + || die "could not import the signing key" + ;; + generate) + need gpg + : "${GPG_PASSPHRASE:?}" + fresh_home + gpg --batch --quiet --no-tty --pinentry-mode loopback --passphrase-fd 3 \ + --quick-generate-key "Release rehearsal (throwaway key, never published) " \ + ed25519 sign 1d 3<<< "$GPG_PASSPHRASE" < /dev/null \ + || die "could not generate a throwaway key" + ;; + remove) + if command -v gpgconf > /dev/null 2>&1; then gpgconf --kill all > /dev/null 2>&1 || true; fi + rm -rf "$GNUPGHOME" + echo "Keyring removed." + exit 0 + ;; + *) die "usage: signing-key.sh import|generate|remove" ;; +esac + +fingerprints="$(gpg --batch --no-tty --with-colons --list-secret-keys \ + | awk -F: '$1 == "sec" { primary = 1; next } primary && $1 == "fpr" { print $10; primary = 0 }')" +[ "$(printf '%s' "$fingerprints" | grep -c . || true)" = 1 ] \ + || die "the keyring must hold exactly one secret key" +[[ "$fingerprints" =~ ^[0-9A-F]{40}$ ]] || die "could not read the key's fingerprint" +output fingerprint "$fingerprints" diff --git a/.github/scripts/release/test.sh b/.github/scripts/release/test.sh new file mode 100644 index 00000000..4a1efd6f --- /dev/null +++ b/.github/scripts/release/test.sh @@ -0,0 +1,1020 @@ +#!/usr/bin/env bash +# Offline tests for the release scripts and for the shape of release.yml. +# No network, no credentials, nothing is uploaded: mvn, curl and gh are +# replaced by stubs that serve fixtures and record what they were asked to do. +# +# bash .github/scripts/release/test.sh +# +# Needs bash (3.2 or newer), git, zip, unzip, jq and python3. Optional: +# gpg the signing tests use a real throwaway key in a temporary keyring; +# without gpg they use a stub (REQUIRE_GPG=1 makes that a failure) +# ruby for the checks that parse release.yml (REQUIRE_WORKFLOW_CHECKS=1 +# makes a missing ruby a failure) +# Exits non-zero if any check fails. +set -uo pipefail + +here="$(cd "$(dirname "$0")" && pwd)" +root="$(cd "$here/../../.." && pwd)" +workflow="$root/.github/workflows/release.yml" +T="$(mktemp -d)" +T="$(cd "$T" && pwd -P)" +trap 'rm -rf "$T"' EXIT +# shellcheck source=lib.sh +. "$here/lib.sh" + +pass=0 +fail=0 +ok() { pass=$((pass + 1)); printf 'ok %s\n' "$1"; } +bad() { fail=$((fail + 1)); printf 'FAIL %s\n' "$1"; } +check() { # check + local name="$1"; shift + if "$@" > /dev/null 2>&1; then ok "$name"; else bad "$name"; fi +} +is() { [ "$1" = "$2" ]; } +has() { grep -qF -- "$2" "$1"; } +hasnt() { ! grep -qF -- "$2" "$1"; } +# run : run with outputs captured; step outputs go to .out +run() { + local log="$1"; shift + GITHUB_OUTPUT="$log.out" GITHUB_STEP_SUMMARY="$log.summary" "$@" > "$log" 2>&1 +} +fails_with() { # fails_with : must fail and say + local text="$1" log="$2"; shift 2 + if run "$log" "$@"; then return 1; fi + grep -qF -- "$text" "$log" +} +out() { sed -n "s/^$2=//p" "$1.out" 2> /dev/null | tail -n 1; } + +export GIT_AUTHOR_NAME=test GIT_AUTHOR_EMAIL=test@invalid GIT_COMMITTER_NAME=test GIT_COMMITTER_EMAIL=test@invalid +export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null +unset GITHUB_OUTPUT GITHUB_STEP_SUMMARY GNUPGHOME JAVA_HOME + +SHA_A=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa +SHA_B=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb +SHA_C=cccccccccccccccccccccccccccccccccccccccc +SHA_D=dddddddddddddddddddddddddddddddddddddddd + +# --------------------------------------------------------------------------- +# lib.sh +# --------------------------------------------------------------------------- +printf 'abc' > "$T/abc" +check "lib: md5" is "$(hash_of md5 "$T/abc")" 900150983cd24fb0d6963f7d28e17f72 +check "lib: sha1" is "$(hash_of sha1 "$T/abc")" a9993e364706816aba3e25717850c26c9cd0d89d +check "lib: sha256" is "$(hash_of sha256 "$T/abc")" ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad +check "lib: sha512" is "$(hash_of sha512 "$T/abc")" ddaf35a193617abacc417349ae20413112e6fa4e89a97ea20a9eeee64b55d39a2192992a274fc1a836ba3c23a3feebbd454d4423643ce80e2a9ac94fa54ca49f +check "lib: version for the profile's Spark line is accepted" bash -c ". '$here/lib.sh'; check_version 0.6.0-rc2 spark-4.0 0.6.0-rc2_spark_4.0.3" +check "lib: version for another Spark line is refused" bash -c "! (. '$here/lib.sh'; check_version 0.6.0 spark-4.0 0.6.0_spark_4.1.2)" +check "lib: version for another base is refused" bash -c "! (. '$here/lib.sh'; check_version 0.6.0 spark-4.0 0.6.1_spark_4.0.3)" +check "lib: a longer base does not match by prefix" bash -c "! (. '$here/lib.sh'; check_version 0.6.0 spark-3.5 0.6.0-rc2_spark_3.5.8)" +check "lib: an unknown profile is refused" bash -c "! (. '$here/lib.sh'; check_version 0.6.0 spark-9.9 0.6.0_spark_9.9.1)" +check "lib: the six published files" is "$(artifact_names 1.0.0_spark_3.5.8 | tr '\n' ' ')" \ + "indextables_spark-1.0.0_spark_3.5.8.pom indextables_spark-1.0.0_spark_3.5.8.jar indextables_spark-1.0.0_spark_3.5.8-sources.jar indextables_spark-1.0.0_spark_3.5.8-javadoc.jar indextables_spark-1.0.0_spark_3.5.8-linux-x86_64-shaded.jar indextables_spark-1.0.0_spark_3.5.8-jar-with-dependencies.jar " +mkdir -p "$T/sums/sub" && echo one > "$T/sums/a" && echo two > "$T/sums/sub/b" +(. "$here/lib.sh"; write_sums "$T/sums") +check "lib: verify_sums accepts what write_sums wrote" bash -c ". '$here/lib.sh'; verify_sums '$T/sums'" +echo three > "$T/sums/extra" +check "lib: verify_sums refuses an unlisted file" bash -c "! (. '$here/lib.sh'; verify_sums '$T/sums')" +rm "$T/sums/extra"; echo changed > "$T/sums/a" +check "lib: verify_sums refuses a changed file" bash -c "! (. '$here/lib.sh'; verify_sums '$T/sums')" +echo one > "$T/sums/a"; ln -s a "$T/sums/link" +check "lib: verify_sums refuses a symbolic link" bash -c "! (. '$here/lib.sh'; verify_sums '$T/sums')" + +# --------------------------------------------------------------------------- +# plan.sh +# --------------------------------------------------------------------------- +P="$T/plan" +mkdir -p "$P" +git init -q -b main "$P/upstream" +( + cd "$P/upstream" + echo 1 > f && git add f && git commit -q -m one && git tag v1.2.3 + echo 2 > f && git commit -q -am two && git tag -a v1.3.0-rc1 -m rc + git checkout -q -b side && echo 3 > f && git commit -q -am three && git tag v9.9.9 + git checkout -q main +) +git clone -q "$P/upstream" "$P/repo" +V123="$(git -C "$P/upstream" rev-parse 'v1.2.3^{commit}')" +RC1="$(git -C "$P/upstream" rev-parse 'v1.3.0-rc1^{commit}')" +plan() { # plan EVENT REF REF_TYPE REF_NAME TAG DRY SKIP + run "$1" env EVENT_NAME="$2" REF="$3" REF_TYPE="$4" REF_NAME="$5" INPUT_TAG="$6" INPUT_DRY_RUN="$7" \ + INPUT_SKIP_CENTRAL="$8" DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +} +L="$P/log" +check "plan: dispatch with dry-run ticked is a dry run" plan "$L.1" workflow_dispatch refs/heads/main branch main v1.2.3 true false +check "plan: ... mode dry-run" is "$(out "$L.1" mode)" dry-run +check "plan: ... tag resolved to its commit" is "$(out "$L.1" commit)" "$V123" +check "plan: ... base version" is "$(out "$L.1" base)" 1.2.3 +check "plan: ... not a pre-release" is "$(out "$L.1" prerelease)" false +check "plan: ... summary says nothing is uploaded" has "$L.1.summary" "Nothing is uploaded" +check "plan: dispatch from main with dry-run unticked publishes" plan "$L.2" workflow_dispatch refs/heads/main branch main v1.3.0-rc1 false false +check "plan: ... mode publish" is "$(out "$L.2" mode)" publish +check "plan: ... annotated tag resolved to its commit" is "$(out "$L.2" commit)" "$RC1" +check "plan: ... pre-release" is "$(out "$L.2" prerelease)" true +check "plan: ... Central included" is "$(out "$L.2" central)" true +check "plan: publish from another branch is refused" fails_with "can only be dispatched from main" "$L.3" \ + env EVENT_NAME=workflow_dispatch REF=refs/heads/feature/x INPUT_TAG=v1.2.3 INPUT_DRY_RUN=false DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +check "plan: ... and sets no mode" test -z "$(out "$L.3" mode)" +check "plan: publish from a tag ref is refused" fails_with "can only be dispatched from main" "$L.3b" \ + env EVENT_NAME=workflow_dispatch REF=refs/tags/v1.2.3 INPUT_TAG=v1.2.3 INPUT_DRY_RUN=false DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +check "plan: a dry run may be dispatched from another branch" plan "$L.4" workflow_dispatch refs/heads/feature/x branch feature/x v1.2.3 true false +check "plan: ... mode dry-run" is "$(out "$L.4" mode)" dry-run +for v in "" TRUE False 0 1 yes "true " "false "; do + check "plan: dry-run '$v' is refused" fails_with "dry-run must be true or false" "$L.5" \ + env EVENT_NAME=workflow_dispatch REF=refs/heads/main INPUT_TAG=v1.2.3 INPUT_DRY_RUN="$v" DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +done +check "plan: skip-central in a publish run" plan "$L.6" workflow_dispatch refs/heads/main branch main v1.2.3 false true +check "plan: ... Central skipped" is "$(out "$L.6" central)" false +check "plan: skip-central has no effect in a dry run" plan "$L.7" workflow_dispatch refs/heads/main branch main v1.2.3 true true +check "plan: ... Central not skipped" is "$(out "$L.7" central)" true +check "plan: a pushed tag publishes (when that trigger is enabled)" plan "$L.8" push refs/tags/v1.2.3 tag v1.2.3 "" "" "" +check "plan: ... mode publish" is "$(out "$L.8" mode)" publish +check "plan: a pushed branch is refused" fails_with "only supported for tags" "$L.9" \ + env EVENT_NAME=push REF=refs/heads/main REF_TYPE=branch REF_NAME=main DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +check "plan: ... also when the branch is named like a release tag" bash -c "! EVENT_NAME=push REF=refs/heads/v1.2.3 REF_TYPE=branch REF_NAME=v1.2.3 DEFAULT_BRANCH=main REPO_DIR='$P/repo' GITHUB_OUTPUT='$L.9b.out' bash '$here/plan.sh'" +check "plan: ... and sets no mode" test -z "$(out "$L.9b" mode)" +for e in pull_request pull_request_target schedule release workflow_run ""; do + check "plan: event '$e' is refused" bash -c "! EVENT_NAME='$e' REF=refs/heads/main DEFAULT_BRANCH=main INPUT_TAG=v1.2.3 INPUT_DRY_RUN=false REPO_DIR='$P/repo' bash '$here/plan.sh'" +done +for t in main v1.2 1.2.3 v1.2.3_spark_3.5.8 'v1.2.3;id' 'v1.2.3 ' v1.2.3-rc_1 '../v1.2.3' ''; do + check "plan: tag '$t' is refused" fails_with "is not a release tag" "$L.10" \ + env EVENT_NAME=workflow_dispatch REF=refs/heads/main INPUT_TAG="$t" INPUT_DRY_RUN=true DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +done +check "plan: a tag that does not exist is refused" fails_with "does not exist" "$L.11" \ + env EVENT_NAME=workflow_dispatch REF=refs/heads/main INPUT_TAG=v7.7.7 INPUT_DRY_RUN=true DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +check "plan: a tag that is not on main is refused" fails_with "is not an ancestor of main" "$L.12" \ + env EVENT_NAME=workflow_dispatch REF=refs/heads/main INPUT_TAG=v9.9.9 INPUT_DRY_RUN=false DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +check "plan: ... in a dry run as well" fails_with "is not an ancestor of main" "$L.13" \ + env EVENT_NAME=workflow_dispatch REF=refs/heads/main INPUT_TAG=v9.9.9 INPUT_DRY_RUN=true DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" + +# --------------------------------------------------------------------------- +# Stubs for the build tools. $FAKE holds fixtures and switches. +# --------------------------------------------------------------------------- +mkdir -p "$T/bin" "$T/fx" +FAKE="$T/fake"; mkdir -p "$FAKE"; export FAKE + +# Fixture jars: a tantivy4java jar with a native library, and project jars. +mkdir -p "$T/fx/t4j/native" "$T/fx/t4j/io" +echo "native library built in this run" > "$T/fx/t4j/native/libtantivy4java.so" +echo class > "$T/fx/t4j/io/A.class" +(cd "$T/fx/t4j" && zip -q -r "$T/fx/t4j.jar" .) +mkdir -p "$T/fx/other/native" +echo "some other native library" > "$T/fx/other/native/libtantivy4java.so" +(cd "$T/fx/other" && zip -q -r "$T/fx/other-native.jar" .) +mkdir -p "$T/fx/plain" && echo x > "$T/fx/plain/X.class" +(cd "$T/fx/plain" && zip -q -r "$T/fx/plain.jar" .) +cp "$T/fx/t4j.jar" "$T/fx/fat.jar" && (cd "$T/fx/plain" && zip -q "$T/fx/fat.jar" X.class) +export FX="$T/fx" + +cat > "$T/bin/mvn" << 'STUB' +#!/usr/bin/env bash +# Stub Maven: enough of tantivy4java's `clean install` and of this project's +# help:evaluate, versions:set and `verify` for the release scripts. +set -eu +echo "mvn $*" >> "$FAKE/mvn.log" +repo=""; profile=""; newversion=""; goal="" +for a in "$@"; do + case "$a" in + -Dmaven.repo.local=*) repo="${a#-Dmaven.repo.local=}" ;; + -P*) profile="${a#-P}" ;; + -DnewVersion=*) newversion="${a#-DnewVersion=}" ;; + --version) echo "Apache Maven 3.9.9 (stub)"; exit 0 ;; + *:evaluate) goal=evaluate ;; + *:set) goal=set ;; + install) goal=install ;; + verify) goal=verify ;; + deploy | *:deploy | *:publish | *:sign) echo "stub mvn: refusing $a" >&2; exit 96 ;; + esac +done +case "$goal" in + install) # tantivy4java + v="$(cat "$FAKE/t4j.version")" + d="$repo/io/indextables/tantivy4java/$v" + mkdir -p "$d" + cp "${FAKE_T4J_JAR:-$FX/t4j.jar}" "$d/tantivy4java-$v-linux-x86_64.jar" + cp "${FAKE_T4J_JAR:-$FX/t4j.jar}" "$d/tantivy4java-$v.jar" + echo "" > "$d/tantivy4java-$v.pom" + echo "#stub" > "$d/_remote.repositories" + if [ -f "$FAKE/touch-lock" ]; then echo "# changed by the build" >> native/Cargo.lock; fi + ;; + evaluate) + cat "$FAKE/spark.${profile}" + ;; + set) + echo "$newversion" > .stamped + ;; + verify) + case " $* " in *" -Dgpg.skip=true "*) ;; *) echo "stub mvn: verify without -Dgpg.skip=true" >&2; exit 95 ;; esac + v="$(cat .stamped)"; p="indextables_spark-$v" + mkdir -p target + cp "$FX/plain.jar" "target/$p.jar" + cp "$FX/plain.jar" "target/$p-sources.jar" + [ -f "$FAKE/no-javadoc" ] || cp "$FX/plain.jar" "target/$p-javadoc.jar" + cp "${FAKE_FAT_JAR:-$FX/fat.jar}" "target/$p-linux-x86_64-shaded.jar" + cp "${FAKE_FAT_JAR:-$FX/fat.jar}" "target/$p-linux-x86_64-jar-with-dependencies.jar" + [ ! -f "$FAKE/extra-jar" ] || cp "$FX/plain.jar" "target/$p-tests.jar" + if [ -f "$FAKE/swap-t4j" ]; then + t="$(cat "$FAKE/t4j.version")" + cp "$FX/other-native.jar" "$repo/io/indextables/tantivy4java/$t/tantivy4java-$t-linux-x86_64.jar" + fi + sed -e "s/@VERSION@/$v/" "${FAKE_POM:-$FX/published.pom}" > dependency-reduced-pom.xml + ;; + *) echo "stub mvn: unexpected invocation: $*" >&2; exit 97 ;; +esac +STUB +for tool in java cargo rustc; do + printf '#!/usr/bin/env bash\necho "%s 1.0 (stub)" >&2\necho "%s 1.0 (stub)"\n' "$tool" "$tool" > "$T/bin/$tool" +done +printf '#!/usr/bin/env bash\ncase "$1" in --exists) exit 0 ;; --modversion) echo 3.0.0 ;; esac\n' > "$T/bin/pkg-config" +chmod +x "$T/bin/"* +STUBPATH="$T/bin:$PATH" + +cat > "$T/fx/published.pom" << 'EOF' + + + 4.0.0 + io.indextables + indextables_spark + @VERSION@ + IndexTables4Spark + Fast search file format + https://example.invalid/ + Apache License, Version 2.0https://www.apache.org/licenses/LICENSE-2.0.txt + Contributors + scm:git:git://example.invalid/x.githttps://example.invalid/x + +EOF + +# --------------------------------------------------------------------------- +# build-native.sh +# --------------------------------------------------------------------------- +N="$T/native"; mkdir -p "$N" +QW=1111111111111111111111111111111111111111 +TV=2222222222222222222222222222222222222222 +MR=3333333333333333333333333333333333333333 +new_t4j() { # new_t4j : a tantivy4java-like repository with tag v0.34.4 + rm -rf "$1"; git init -q -b main "$1" + mkdir -p "$1/native" "$1/.cargo" + cat > "$1/native/Cargo.toml" << EOF +[package] +name = "tantivy4java" +[dependencies] +tantivy = { git = "https://github.com/indextables/tantivy", rev = "$TV" } +quickwit-storage = { git = "https://github.com/indextables/quickwit", rev = "$QW" } +EOF + cat > "$1/native/Cargo.lock" << EOF +version = 3 +[[package]] +name = "quickwit-storage" +source = "git+https://github.com/indextables/quickwit?rev=$QW#$QW" +[[package]] +name = "quickwit-proto" +source = "git+https://github.com/indextables/quickwit?rev=$QW#$QW" +[[package]] +name = "tantivy" +source = "git+https://github.com/indextables/tantivy?rev=$TV#$TV" +[[package]] +name = "mrecordlog" +source = "git+https://github.com/quickwit-oss/mrecordlog?rev=3333333#$MR" +[[package]] +name = "serde" +source = "registry+https://github.com/rust-lang/crates.io-index" +EOF + printf '[target.x86_64-unknown-linux-gnu]\nrustflags = ["-C", "target-cpu=x86-64"]\n' > "$1/.cargo/config.toml" + git -C "$1" add -A && git -C "$1" commit -q -m release + git -C "$1" config uploadpack.allowAnySHA1InWant true +} +new_t4j "$N/t4j" +git -C "$N/t4j" tag v0.34.4 +T4J_COMMIT="$(git -C "$N/t4j" rev-parse HEAD)" +printf '\n tantivy4java\n 0.34.4\n\n' > "$N/pom.xml" +printf '# comment\n0.34.4 %s %s %s\n0.1.0 %s - -\n' "$T4J_COMMIT" "$QW" "$TV" "$SHA_A" > "$N/pins.txt" +echo 0.34.4 > "$FAKE/t4j.version" +native() { # native [VAR=value...] + local log="$1" sub="$2"; shift 2 + run "$log" env PATH="$STUBPATH" T4J_REPO="file://$N/t4j" POM="$N/pom.xml" PINS="$N/pins.txt" WORK="$N/work" OUT="$N/out" \ + M2_REPO="$N/m2" "$@" bash "$here/build-native.sh" "$sub" +} +L="$N/log" +check "native: resolve accepts the pinned tag" native "$L.1" resolve +check "native: ... records the tantivy4java commit" has "$N/out/native-inputs.txt" "tantivy4java_commit=$T4J_COMMIT" +check "native: ... records the quickwit fork commit" has "$N/out/native-inputs.txt" "quickwit_commit=$QW" +check "native: ... records the tantivy fork commit" has "$N/out/native-inputs.txt" "tantivy_commit=$TV" +check "native: ... records other git dependencies with their commit" has "$N/out/native-inputs.txt" "other_git_dependencies=https://github.com/quickwit-oss/mrecordlog@$MR" +check "native: ... checked out exactly the pinned commit" is "$(git -C "$N/work/tantivy4java" rev-parse HEAD)" "$T4J_COMMIT" +check "native: print-pin prints the pin line" is "$(PATH="$STUBPATH" T4J_REPO="file://$N/t4j" bash "$here/build-native.sh" print-pin 0.34.4 2> /dev/null)" "0.34.4 $T4J_COMMIT $QW $TV" + +check "native: build needs the toolchain step first" fails_with "toolchain" "$L.2" native "$L.2" build +mkdir -p "$N/work/protoc/bin" "$N/work/protoc/include" && printf '#!/bin/sh\necho "libprotoc 25.5"\n' > "$N/work/protoc/bin/protoc" && chmod +x "$N/work/protoc/bin/protoc" +mkdir -p "$N/m2/io/indextables/tantivy4java/0.34.4" && echo stale > "$N/m2/io/indextables/tantivy4java/0.34.4/tantivy4java-0.34.4-linux-x86_64.jar" +echo stale > "$N/m2/io/indextables/tantivy4java/0.34.4/tantivy4java-0.34.4-left-over.jar" +check "native: build collects the jar" native "$L.3" build +check "native: ... a jar that was there before the build is not reused" has "$L.3" "removing it before the build" +check "native: ... the collected jar is the one this build installed" is "$(hash_of sha256 "$N/out/repository/tantivy4java-0.34.4-linux-x86_64.jar")" "$(hash_of sha256 "$T/fx/t4j.jar")" +check "native: ... nothing that was there before the build is collected" test ! -e "$N/out/repository/tantivy4java-0.34.4-left-over.jar" +check "native: ... Maven bookkeeping files are left out" test ! -e "$N/out/repository/_remote.repositories" +check "native: ... digests cover the output" bash -c ". '$here/lib.sh'; verify_sums '$N/out'" +check "native: ... records an unchanged Cargo.lock" has "$N/out/native-inputs.txt" "cargo_lock_unchanged=true" +check "native: ... records the native library digest" has "$N/out/native-inputs.txt" "native_library=native/libtantivy4java.so sha256 $(hash_of sha256 "$T/fx/t4j/native/libtantivy4java.so")" +check "native: ... summary lists the commits" has "$L.3.summary" "$T4J_COMMIT" +cp -R "$N/out" "$N/out.good" + +touch "$FAKE/touch-lock" +check "native: a build that rewrites Cargo.lock is refused" fails_with "modified native/Cargo.lock" "$L.4" native "$L.4" build +rm "$FAKE/touch-lock" +git -C "$N/work/tantivy4java" checkout -q -- native/Cargo.lock +check "native: a jar without a native library is refused" fails_with "contains no native library" "$L.5" native "$L.5" build FAKE_T4J_JAR="$T/fx/plain.jar" + +printf '0.34.4 %s %s %s\n' "$SHA_A" "$QW" "$TV" > "$N/pins.moved" +check "native: a tag that does not point to the pinned commit is refused" fails_with "points to $T4J_COMMIT, but native-pins.txt pins $SHA_A" "$L.6" native "$L.6" resolve PINS="$N/pins.moved" +printf '0.1.0 %s - -\n' "$SHA_A" > "$N/pins.none" +check "native: a version without a pin is refused" fails_with "exactly one line for tantivy4java 0.34.4" "$L.7" native "$L.7" resolve PINS="$N/pins.none" +printf '0.34.4 %s %s %s\n0.34.4 %s %s %s\n' "$T4J_COMMIT" "$QW" "$TV" "$T4J_COMMIT" "$QW" "$TV" > "$N/pins.twice" +check "native: a version pinned twice is refused" fails_with "exactly one line" "$L.8" native "$L.8" resolve PINS="$N/pins.twice" +printf '0.34.4 %s %s %s\n' "$T4J_COMMIT" "$SHA_B" "$TV" > "$N/pins.qw" +check "native: a quickwit fork commit other than the pinned one is refused" fails_with "resolves the quickwit fork to $QW" "$L.9" native "$L.9" resolve PINS="$N/pins.qw" +printf '0.34.4 %s %s %s\n' "$T4J_COMMIT" "$QW" "$SHA_B" > "$N/pins.tv" +check "native: a tantivy fork commit other than the pinned one is refused" fails_with "resolves the tantivy fork to $TV" "$L.10" native "$L.10" resolve PINS="$N/pins.tv" +printf '0.34.4 %s main %s\n' "$T4J_COMMIT" "$TV" > "$N/pins.branch" +check "native: a pin that is not a commit id is refused" fails_with "not a full commit id" "$L.11" native "$L.11" resolve PINS="$N/pins.branch" + +variant() { # variant : a repository that differs in one way + new_t4j "$N/v-$1"; (cd "$N/v-$1" && eval "$2" && git add -A && git commit -q -m variant && git tag v0.34.4) + printf '0.34.4 %s %s %s\n' "$(git -C "$N/v-$1" rev-parse HEAD)" "$QW" "$TV" > "$N/pins.$1" +} +variant path "echo 'quickwit-common = { path = \"../../quickwit/quickwit-common\" }' >> native/Cargo.toml" +check "native: a dependency on a sibling checkout is refused" fails_with "outside the tantivy4java checkout" "$L.12" native "$L.12" resolve T4J_REPO="file://$N/v-path" PINS="$N/pins.path" +variant patch "printf '[patch.crates-io]\nserde = { path = \"/tmp/serde\" }\n' >> .cargo/config.toml" +check "native: a cargo config that overrides sources is refused" fails_with "overrides dependency sources" "$L.13" native "$L.13" resolve T4J_REPO="file://$N/v-patch" PINS="$N/pins.patch" +variant two "printf '[[package]]\nname = \"quickwit-x\"\nsource = \"git+https://github.com/indextables/quickwit?rev=$SHA_C#$SHA_C\"\n' >> native/Cargo.lock" +check "native: a lock file with two quickwit commits is refused" fails_with "more than one commit" "$L.14" native "$L.14" resolve T4J_REPO="file://$N/v-two" PINS="$N/pins.two" +variant short "printf '[[package]]\nname = \"y\"\nsource = \"git+https://github.com/indextables/quickwit?branch=main\"\n' >> native/Cargo.lock" +check "native: a git dependency without a resolved commit is refused" fails_with "without a resolved commit" "$L.15" native "$L.15" resolve T4J_REPO="file://$N/v-short" PINS="$N/pins.short" +new_t4j "$N/v-annot"; git -C "$N/v-annot" tag -a v0.34.4 -m annotated +printf '0.34.4 %s %s %s\n' "$(git -C "$N/v-annot" rev-parse HEAD)" "$QW" "$TV" > "$N/pins.annot" +check "native: an annotated tag is resolved to its commit" native "$L.16" resolve T4J_REPO="file://$N/v-annot" PINS="$N/pins.annot" + +# toolchain: a download with the wrong digest is never unpacked. +cat > "$T/bin/curl" << 'STUB' +#!/usr/bin/env bash +# Stub curl for the protoc download: writes a small file to the -o target. +o=""; while [ $# -gt 0 ]; do case "$1" in -o) o="$2"; shift 2 ;; *) shift ;; esac; done +echo "not protoc" > "$o" +STUB +chmod +x "$T/bin/curl" +native "$L.0" resolve > /dev/null 2>&1 +check "native: a protoc download with the wrong digest is refused" fails_with "expected e1ed237a17b2e851cf9662cb5ad02b46e70ff8e060e05984725bc4b4228c6b28" "$L.17" native "$L.17" toolchain +check "native: ... and is not unpacked" test ! -e "$N/work/protoc/bin/protoc" +rm "$T/bin/curl" + +check "pins: protoc version is the one scripts/setup.sh uses" is \ + "$(sed -n 's/^PROTOC_VERSION=//p' "$here/build-native.sh")" "$(sed -n 's/^PROTOC_VERSION="\(.*\)"$/\1/p' "$root/scripts/setup.sh")" +check "pins: protoc linux-x86_64 digest is the one scripts/setup.sh uses" is \ + "$(sed -n 's/^PROTOC_SHA256=//p' "$here/build-native.sh")" "$(sed -n 's/^ *linux-x86_64) *echo "\([0-9a-f]*\)".*/\1/p' "$root/scripts/setup.sh")" +pom_t4j="$(pom_tantivy4java_version "$root/pom.xml" 2> /dev/null)" +check "pins: pom.xml's tantivy4java version ($pom_t4j) has exactly one pin" is \ + "$(awk -v v="$pom_t4j" '!/^[[:space:]]*#/ && $1 == v && NF == 4' "$here/native-pins.txt" | grep -c .)" 1 +check "pins: every pin is a full commit id or '-'" test -z \ + "$(grep -v '^[[:space:]]*#' "$here/native-pins.txt" | grep -v -E '^[0-9]+\.[0-9]+\.[0-9]+ [0-9a-f]{40} ([0-9a-f]{40}|-) ([0-9a-f]{40}|-)$')" + +# --------------------------------------------------------------------------- +# build-artifacts.sh +# --------------------------------------------------------------------------- +B="$T/build"; mkdir -p "$B" +echo 3.5.8 > "$FAKE/spark.spark-3.5"; echo 4.0.3 > "$FAKE/spark.spark-4.0"; echo 4.1.2 > "$FAKE/spark.spark-4.1" +artifacts() { # artifacts [VAR=value...] + local log="$1" profile="$2"; shift 2 + rm -rf "$B/src-$profile" "$B/m2-$profile"; mkdir -p "$B/src-$profile"; cp "$N/pom.xml" "$B/src-$profile/pom.xml" + run "$log" env PATH="$STUBPATH" SRC="$B/src-$profile" PROFILE="$profile" BASE_VERSION=1.2.3 NATIVE_DIR="$N/out.good" \ + OUT="$B/out/artifacts-$profile" M2_REPO="$B/m2-$profile" "$@" bash "$here/build-artifacts.sh" +} +L="$B/log" +: > "$FAKE/mvn.log" +check "build: spark-3.5" artifacts "$L.35" spark-3.5 +check "build: spark-4.0" artifacts "$L.40" spark-4.0 +check "build: spark-4.1" artifacts "$L.41" spark-4.1 +check "build: version is _spark_" is "$(out "$L.40" version)" 1.2.3_spark_4.0.3 +check "build: files carry their published names" is "$(ls "$B/out/artifacts-spark-4.0/files" | LC_ALL=C sort | tr '\n' ' ')" "$(artifact_names 1.2.3_spark_4.0.3 | LC_ALL=C sort | tr '\n' ' ')" +check "build: the published pom is the dependency-reduced pom" has "$B/out/artifacts-spark-4.0/files/indextables_spark-1.2.3_spark_4.0.3.pom" "1.2.3_spark_4.0.3" +check "build: digests cover the output" bash -c ". '$here/lib.sh'; verify_sums '$B/out/artifacts-spark-4.0'" +check "build: Maven is never asked to sign, deploy or publish" test -z "$(grep -E '(^| )(deploy|install)( |$)|:deploy|:publish|:sign' "$FAKE/mvn.log")" +check "build: the release profile runs with signing off" has "$FAKE/mvn.log" "-Pspark-4.0,release -DskipTests -Dgpg.skip=true verify" +check "build: plugins invoked by coordinate are pinned" test -z "$(grep -E ':(evaluate|set)' "$FAKE/mvn.log" | grep -v -E '[a-z.-]+:[a-z-]+:[0-9.]+:(evaluate|set)')" +cp -R "$B/out" "$B/out.good" + +mkdir -p "$B/m2-pre/io/indextables/tantivy4java/0.34.4" +check "build: a tantivy4java already in the local repository is refused" fails_with "already in the local Maven repository" "$L.1" \ + env PATH="$STUBPATH" SRC="$B/src-spark-3.5" PROFILE=spark-3.5 BASE_VERSION=1.2.3 NATIVE_DIR="$N/out.good" OUT="$B/out/x" M2_REPO="$B/m2-pre" bash "$here/build-artifacts.sh" +cp -R "$N/out.good" "$B/native.bad" && echo tampered >> "$B/native.bad/repository/tantivy4java-0.34.4-linux-x86_64.jar" +check "build: a native artifact that does not match its digests is refused" fails_with "digest mismatch" "$L.2" artifacts "$L.2" spark-3.5 NATIVE_DIR="$B/native.bad" +echo 4.0.3 > "$FAKE/spark.spark-3.5" +check "build: a Spark version outside the profile's line is refused" fails_with "is not 1.2.3_spark_3.5." "$L.3" artifacts "$L.3" spark-3.5 +echo 3.5.8 > "$FAKE/spark.spark-3.5" +check "build: an unknown profile is refused" fails_with "unknown Spark profile" "$L.4" artifacts "$L.4" spark-5.0 +touch "$FAKE/no-javadoc" +check "build: a missing javadoc jar is refused" fails_with "did not produce" "$L.5" artifacts "$L.5" spark-3.5 +rm "$FAKE/no-javadoc"; touch "$FAKE/extra-jar" +check "build: a jar the scripts do not know is refused" fails_with "which the release scripts do not know" "$L.6" artifacts "$L.6" spark-3.5 +rm "$FAKE/extra-jar"; touch "$FAKE/swap-t4j" +check "build: a tantivy4java jar replaced during the build is refused" fails_with "changed during the build" "$L.7" artifacts "$L.7" spark-3.5 +rm "$FAKE/swap-t4j" +rm -rf "$B/out" && cp -R "$B/out.good" "$B/out" + +# --------------------------------------------------------------------------- +# assemble.sh and verify-staging.sh +# --------------------------------------------------------------------------- +A="$T/assemble"; mkdir -p "$A" +new_in() { rm -rf "$A/in"; mkdir -p "$A/in"; cp -R "$N/out.good" "$A/in/native"; cp -R "$B/out.good/"* "$A/in/"; } +assemble() { # assemble [VAR=value...] + local log="$1"; shift + run "$log" env IN="$A/in" OUT="$A/staging" TAG=v1.2.3 COMMIT="$SHA_D" BASE_VERSION=1.2.3 RUN_URL=https://example.invalid/run/1 "$@" bash "$here/assemble.sh" +} +resum() { (. "$here/lib.sh"; write_sums "$1"); } +L="$A/log" +new_in +check "assemble: accepts the three builds" assemble "$L.1" +check "assemble: versions, in profile order" is "$(out "$L.1" versions)" "1.2.3_spark_3.5.8 1.2.3_spark_4.0.3 1.2.3_spark_4.1.2" +vdir="$A/staging/bundles/1.2.3_spark_4.0.3/io/indextables/indextables_spark/1.2.3_spark_4.0.3" +check "assemble: Maven repository layout, six files and four checksums each" is "$(ls "$vdir" | grep -c .)" 30 +check "assemble: no signatures yet" test -z "$(find "$A/staging" -name '*.asc')" +check "assemble: checksum files hold the bare digest" is "$(cat "$vdir/indextables_spark-1.2.3_spark_4.0.3.pom.sha1")" "$(hash_of sha1 "$vdir/indextables_spark-1.2.3_spark_4.0.3.pom")" +check "assemble: ... without a trailing newline" is "$(wc -c < "$vdir/indextables_spark-1.2.3_spark_4.0.3.pom.md5" | tr -d ' ')" 32 +check "assemble: manifest digest is reported" is "$(out "$L.1" manifest_sha256)" "$(hash_of sha256 "$A/staging/MANIFEST.sha256")" +check "assemble: notes name the tantivy4java commit" has "$A/staging/release-notes.md" "\`v0.34.4\` at \`$T4J_COMMIT\`" +check "assemble: notes name the fork commits" test -n "$(grep -F "$QW" "$A/staging/release-notes.md")" -a -n "$(grep -F "$TV" "$A/staging/release-notes.md")" +check "assemble: notes name the released commit" has "$A/staging/release-notes.md" "$SHA_D" +check "assemble: summary shows the manifest digest" has "$L.1.summary" "$(out "$L.1" manifest_sha256)" +MANIFEST="$(out "$L.1" manifest_sha256)" +rm -rf "$A/staging.good" && cp -R "$A/staging" "$A/staging.good" + +check "assemble: a tag that does not match the base version is refused" fails_with "does not belong to tag" "$L.2" assemble "$L.2" TAG=v1.2.4 +new_in; rm -rf "$A/in/artifacts-spark-4.1" +check "assemble: a missing profile is refused" fails_with "no build output for spark-4.1" "$L.3" assemble "$L.3" +new_in; echo x > "$A/in/artifacts-spark-4.0/files/evil.jar"; resum "$A/in/artifacts-spark-4.0" +check "assemble: an extra file is refused, even with valid digests" fails_with "does not hold exactly the expected files" "$L.4" assemble "$L.4" +new_in; echo x >> "$A/in/artifacts-spark-4.0/files/indextables_spark-1.2.3_spark_4.0.3.jar" +check "assemble: a file changed after its digest was taken is refused" fails_with "digest mismatch" "$L.5" assemble "$L.5" +new_in; sed -i.bak 's/^version=.*/version=1.2.3_spark_4.1.2/' "$A/in/artifacts-spark-4.0/build-info.txt"; rm "$A/in/artifacts-spark-4.0/build-info.txt.bak"; resum "$A/in/artifacts-spark-4.0" +check "assemble: a version for another Spark line is refused" fails_with "is not 1.2.3_spark_4.0." "$L.6" assemble "$L.6" +pom_case() { # pom_case + new_in + local f="$A/in/artifacts-spark-3.5/files/indextables_spark-1.2.3_spark_3.5.8.pom" + sed -i.bak "$2" "$f"; rm "$f.bak"; resum "$A/in/artifacts-spark-3.5" + check "assemble: a pom with $1 is refused" fails_with "$3" "$L.pom" assemble "$L.pom" +} +pom_case "another groupId" 's|io.indextables|io.other|' "groupId is 'io.other'" +pom_case "another artifactId" 's|indextables_spark|tantivy4java|' "artifactId is 'tantivy4java'" +pom_case "another version" 's|1.2.3_spark_3.5.8|9.9.9|' "version is '9.9.9'" +pom_case "no license" 's|.*||' "missing licenses/license/name" +pom_case "no scm" 's|.*||' "missing scm/connection" +pom_case "a parent" 's||x|' "has a " +pom_case "broken XML" 's|||' "not well-formed" +new_in +for f in linux-x86_64-shaded jar-with-dependencies; do cp "$T/fx/other-native.jar" "$A/in/artifacts-spark-4.1/files/indextables_spark-1.2.3_spark_4.1.2-$f.jar"; done +resum "$A/in/artifacts-spark-4.1" +check "assemble: a jar with another native library is refused" fails_with "does not contain this run's native/libtantivy4java.so" "$L.7" assemble "$L.7" +new_in; echo "not a zip" > "$A/in/artifacts-spark-3.5/files/indextables_spark-1.2.3_spark_3.5.8-sources.jar"; resum "$A/in/artifacts-spark-3.5" +check "assemble: a jar that is not a zip file is refused" fails_with "is not a readable jar" "$L.8" assemble "$L.8" +new_in; sed -i.bak 's/^tantivy4java_jar_sha256=.*/tantivy4java_jar_sha256=0000/' "$A/in/artifacts-spark-3.5/build-info.txt"; rm "$A/in/artifacts-spark-3.5/build-info.txt.bak"; resum "$A/in/artifacts-spark-3.5" +check "assemble: a build that used another tantivy4java jar is refused" fails_with "was not built with this run's tantivy4java jar" "$L.9" assemble "$L.9" +new_in; sed -i.bak '/^cargo_lock_unchanged=/d' "$A/in/native/native-inputs.txt"; rm "$A/in/native/native-inputs.txt.bak"; resum "$A/in/native" +check "assemble: a native build without an unchanged Cargo.lock is refused" fails_with "does not record an unchanged Cargo.lock" "$L.10" assemble "$L.10" +new_in; sed -i.bak 's/^tantivy4java_commit=.*/tantivy4java_commit=main/' "$A/in/native/native-inputs.txt"; rm "$A/in/native/native-inputs.txt.bak"; resum "$A/in/native" +check "assemble: a tantivy4java commit that is not a commit id is refused" fails_with "does not state a tantivy4java commit" "$L.11" assemble "$L.11" +# shellcheck disable=SC2016 +new_in; echo 'rustc=rustc 1.0 `x` @everyone' >> "$A/in/native/native-inputs.txt"; resum "$A/in/native" +check "assemble: tool text from the build that is not plain text stays out of the notes" assemble "$L.12" +check "assemble: ... a placeholder is shown instead" has "$A/staging/release-notes.md" "(not shown: unexpected characters)" +check "assemble: ... the text itself is absent" hasnt "$A/staging/release-notes.md" " "$A/s/bundles/1.2.3_spark_3.5.8/io/indextables/indextables_spark/1.2.3_spark_3.5.8/extra.jar" +check "verify-staging: an added file is refused" fails_with "not exactly the files listed" "$L.4" verify "$L.4" +staging_copy; echo x >> "$A/s/bundles/1.2.3_spark_3.5.8/io/indextables/indextables_spark/1.2.3_spark_3.5.8/indextables_spark-1.2.3_spark_3.5.8.jar" +check "verify-staging: a changed file is refused" fails_with "digest mismatch" "$L.5" verify "$L.5" +staging_copy; rm "$A/s/release-notes.md" +check "verify-staging: a removed file is refused" fails_with "not exactly the files listed" "$L.6" verify "$L.6" +staging_copy +check "verify-staging: versions of another tag are refused" fails_with "is not 1.2.4_spark_3.5." "$L.7" verify "$L.7" "$MANIFEST" 1.2.4 +# A self-consistent directory for other coordinates: valid manifest, wrong content. +staging_copy +mkdir -p "$A/s/bundles/1.2.3_spark_3.5.8/io/indextables/tantivy4java/9.9.9" && echo x > "$A/s/bundles/1.2.3_spark_3.5.8/io/indextables/tantivy4java/9.9.9/tantivy4java-9.9.9.jar" +(. "$here/lib.sh"; write_sums "$A/s" MANIFEST.sha256) +check "verify-staging: other coordinates are refused even with a matching manifest" fails_with "lists files other than the expected ones" "$L.8" verify "$L.8" "$(hash_of sha256 "$A/s/MANIFEST.sha256")" + +# --------------------------------------------------------------------------- +# signing-key.sh, sign-bundles.sh, check-bundles.sh +# --------------------------------------------------------------------------- +S="$T/sign"; mkdir -p "$S" +SIGNPATH="$PATH" +REAL_GPG=0 +if command -v gpg > /dev/null 2>&1; then + REAL_GPG=1 + echo "note signing tests use gpg ($(gpg --version | head -n 1)) with a throwaway key" + # gpg's agent socket needs a short path; mktemp directories can be too long. + KEYHOME="$(mktemp -d /tmp/rt.XXXXXX)/release-gnupg" + trap 'rm -rf "$T" "$(dirname "$KEYHOME")"' EXIT +else + if [ "${REQUIRE_GPG:-0}" = 1 ]; then + bad "signing: gpg is required but not available (the checks below ran against a stub)" + else + echo "note gpg not available: signing tests use a stub (set REQUIRE_GPG=1 to make this a failure)" + fi + KEYHOME="$S/release-gnupg" + mkdir -p "$S/bin" + cat > "$S/bin/gpg" << 'STUB' +#!/usr/bin/env bash +# Stub gpg: one key per GNUPGHOME, "signatures" that name the key and the +# file's digest. Refuses a wrong passphrase, like gpg does. +set -eu +h() { if command -v sha256sum > /dev/null 2>&1; then sha256sum "$1" | cut -d' ' -f1; else shasum -a 256 "$1" | cut -d' ' -f1; fi; } +mode=""; pfd=""; user=""; outf=""; files=() +while [ $# -gt 0 ]; do + case "$1" in + --batch | --quiet | --no-tty | --yes | --armor | --with-colons) shift ;; + --pinentry-mode | --status-fd) shift 2 ;; + --passphrase-fd) pfd="$2"; shift 2 ;; + --local-user) user="$2"; shift 2 ;; + --output) outf="$2"; shift 2 ;; + --import) mode=import; shift ;; + --quick-generate-key) mode=generate; shift 5 ;; + --export-secret-keys) mode=export; shift 2 ;; + --list-secret-keys) mode=list; shift ;; + --detach-sign) mode=sign; shift ;; + --verify) mode=verify; shift ;; + -*) echo "stub gpg: unexpected option $1" >&2; exit 97 ;; + *) files+=("$1"); shift ;; + esac +done +pass=""; if [ -n "$pfd" ]; then IFS= read -r pass <&"$pfd" || true; fi +case "$mode" in + import) # the "key" is "STUBKEY " + read -r tag fpr keypass || true + [ "$tag" = STUBKEY ] || { echo "stub gpg: no valid key data" >&2; exit 2; } + echo "$fpr" > "$GNUPGHOME/fpr"; echo "$keypass" > "$GNUPGHOME/pass" ;; + export) echo "STUBKEY $(cat "$GNUPGHOME/fpr") $(cat "$GNUPGHOME/pass")" ;; + generate) echo "0123456789ABCDEF0123456789ABCDEF01234567" > "$GNUPGHOME/fpr"; echo "$pass" > "$GNUPGHOME/pass" ;; + list) [ -f "$GNUPGHOME/fpr" ] || exit 0 + printf 'sec:u:255:22:KEYID:1:::u:::scESC:::+::ed25519:::0:\nfpr:::::::::%s:\nuid:u::::1::X::test::::::::::0:\n' "$(cat "$GNUPGHOME/fpr")" ;; + sign) + [ "$user" = "$(cat "$GNUPGHOME/fpr")" ] || { echo "stub gpg: no such key $user" >&2; exit 2; } + [ "$pass" = "$(cat "$GNUPGHOME/pass")" ] || { echo "stub gpg: bad passphrase" >&2; exit 2; } + printf 'STUBSIG %s %s\n' "$user" "$(h "${files[0]}")" > "$outf" ;; + verify) + read -r tag fpr sum < "${files[0]}" + [ "$tag" = STUBSIG ] && [ "$sum" = "$(h "${files[1]}")" ] || { echo "[GNUPG:] BADSIG"; exit 1; } + echo "[GNUPG:] VALIDSIG $fpr 2026-01-01 0 4 0 22 10 00 $fpr" ;; + *) echo "stub gpg: nothing to do" >&2; exit 97 ;; +esac +STUB + chmod +x "$S/bin/gpg" + SIGNPATH="$S/bin:$PATH" +fi +# With a real gpg the agent remembers a passphrase it has seen; forget it +# before a check that depends on the passphrase being asked for again. +forget() { if [ "$REAL_GPG" = 1 ]; then GNUPGHOME="$KEYHOME" gpgconf --kill gpg-agent > /dev/null 2>&1 || true; fi; } + +L="$S/log" +key() { run "$1" env PATH="$SIGNPATH" GNUPGHOME="$KEYHOME" GPG_PASSPHRASE=rehearsal bash "$here/signing-key.sh" "$2"; } +fresh() { rm -rf "$S/staging" "$S/bundles"; cp -R "$A/staging.good" "$S/staging"; } +sign() { run "$1" env PATH="$SIGNPATH" GNUPGHOME="$KEYHOME" SIGNING_KEY="${2:-$FPR}" GPG_PASSPHRASE="${3-rehearsal}" bash "$here/sign-bundles.sh" "$S/staging" "$S/bundles"; } +chk() { run "$1" env PATH="$SIGNPATH" GNUPGHOME="$KEYHOME" SIGNING_KEY="${2:-$FPR}" BASE_VERSION="${3:-1.2.3}" bash "$here/check-bundles.sh" "$S/bundles" "$S/staging"; } +NOKEY=00000000000000000000000000000000DEADBEEF + +check "signing-key: refuses a keyring directory with another name" fails_with "must be a directory named release-gnupg" "$L.0" \ + env PATH="$SIGNPATH" GNUPGHOME="$S/gnupg" bash "$here/signing-key.sh" remove +check "signing-key: generate creates a throwaway key" key "$L.1" generate +FPR="$(out "$L.1" fingerprint)" +check "signing-key: ... and reports its fingerprint" test -n "$FPR" +check "signing-key: ... in a private directory" test "$(ls -ld "$KEYHOME" | cut -c1-10)" = "drwx------" + +fresh; forget +check "sign: a wrong passphrase fails" fails_with "could not sign" "$L.2" sign "$L.2" "$FPR" wrong-passphrase +fresh +check "sign: an unknown key fails" fails_with "could not sign" "$L.3" sign "$L.3" "$NOKEY" +fresh +check "sign: signs and bundles the three versions" sign "$L.4" +check "sign: one bundle per version" is "$(ls "$S/bundles" | tr '\n' ' ')" "indextables_spark-1.2.3_spark_3.5.8-bundle.zip indextables_spark-1.2.3_spark_4.0.3-bundle.zip indextables_spark-1.2.3_spark_4.1.2-bundle.zip " +# The entry list of a bundle, written out in full: this is the set of files +# that is on Maven Central for 0.6.0-rc2 (36 per version). +want_entries="$(for s in .pom .jar -sources.jar -javadoc.jar -linux-x86_64-shaded.jar -jar-with-dependencies.jar; do + for e in "" .asc .md5 .sha1 .sha256 .sha512; do echo "io/indextables/indextables_spark/1.2.3_spark_4.0.3/indextables_spark-1.2.3_spark_4.0.3$s$e"; done; done | LC_ALL=C sort)" +check "sign: a bundle holds exactly the 36 files Maven Central has for a version" is \ + "$(unzip -Z1 "$S/bundles/indextables_spark-1.2.3_spark_4.0.3-bundle.zip" | grep -v '/$' | LC_ALL=C sort)" "$want_entries" +check "sign: checksum files are not signed" test -z "$(unzip -Z1 "$S/bundles/indextables_spark-1.2.3_spark_4.0.3-bundle.zip" | grep -E '\.(md5|sha1|sha256|sha512)\.asc$')" +check "check: accepts the signed bundles" chk "$L.5" +check "check: ... and says who signed" has "$L.5.summary" "$FPR" +check "sign: a second run on the same directory is refused" fails_with "already signed" "$L.6" sign "$L.6" +check "check: signatures from another key are refused" fails_with "expected $NOKEY" "$L.7" chk "$L.7" "$NOKEY" +check "check: versions of another tag are refused" fails_with "is not 1.2.4_spark_3.5." "$L.8" chk "$L.8" "$FPR" 1.2.4 +z="$S/bundles/indextables_spark-1.2.3_spark_3.5.8-bundle.zip" +zd=io/indextables/indextables_spark/1.2.3_spark_3.5.8 +cp "$z" "$z.keep" +mkdir -p "$S/x/io/indextables/tantivy4java/9.9.9" && echo x > "$S/x/io/indextables/tantivy4java/9.9.9/evil.jar" +(cd "$S/x" && zip -q "$z" io/indextables/tantivy4java/9.9.9/evil.jar) +check "check: a bundle with an extra entry is refused" fails_with "does not hold exactly the expected entries" "$L.9" chk "$L.9" +cp "$z.keep" "$z"; zip -q -d "$z" "$zd/indextables_spark-1.2.3_spark_3.5.8-javadoc.jar.asc" +check "check: a bundle with a missing signature is refused" fails_with "does not hold exactly the expected entries" "$L.10" chk "$L.10" +cp "$z.keep" "$z" +mkdir -p "$S/y/$zd" && echo 0000 > "$S/y/$zd/indextables_spark-1.2.3_spark_3.5.8.jar.sha1" +(cd "$S/y" && zip -q "$z" "$zd/indextables_spark-1.2.3_spark_3.5.8.jar.sha1") +check "check: a wrong checksum is refused" fails_with ".jar.sha1 does not match" "$L.11" chk "$L.11" +# A file replaced after signing, with its checksums recomputed to match: only +# the signature (and the comparison with the staged file) can catch it. +cp "$z.keep" "$z" +mkdir -p "$S/w/$zd" && cp "$T/fx/fat.jar" "$S/w/$zd/indextables_spark-1.2.3_spark_3.5.8.jar" +for algo in $CHECKSUM_ALGOS; do printf '%s' "$(hash_of "$algo" "$S/w/$zd/indextables_spark-1.2.3_spark_3.5.8.jar")" > "$S/w/$zd/indextables_spark-1.2.3_spark_3.5.8.jar.$algo"; done +(cd "$S/w" && zip -q -r "$z" io) +check "check: a file replaced after signing is refused, even with matching checksums" fails_with "the signature of indextables_spark-1.2.3_spark_3.5.8.jar does not verify" "$L.12" chk "$L.12" +cp "$z.keep" "$z"; rm "$z.keep" +staged_pom="$S/staging/bundles/1.2.3_spark_3.5.8/$zd/indextables_spark-1.2.3_spark_3.5.8.pom" +cp "$staged_pom" "$S/pom.keep"; echo "" >> "$staged_pom" +check "check: a bundle that differs from the staging directory is refused" fails_with "in the bundle differs from the staged file" "$L.12b" chk "$L.12b" +cp "$S/pom.keep" "$staged_pom" +check "check: the restored bundles pass again" chk "$L.13" +# Signing writes nothing but signatures into the staging directory: with +# those removed it verifies against the original manifest again. +find "$S/staging" -name '*.asc' -exec rm {} + +check "sign: nothing but signatures is written into the staging directory" run "$L.14" env MANIFEST_SHA256="$MANIFEST" BASE_VERSION=1.2.3 bash "$here/verify-staging.sh" "$S/staging" + +# The publish job's path: import a passphrase-protected private key from the +# environment, then sign with it. The key is the throwaway key, exported. +PATH="$SIGNPATH" GNUPGHOME="$KEYHOME" gpg --batch --no-tty --pinentry-mode loopback --passphrase-fd 3 --armor \ + --export-secret-keys "$FPR" 3<<< rehearsal > "$S/key.asc" 2> /dev/null +check "signing-key: remove deletes the keyring" key "$L.15" remove +check "signing-key: ... it is gone" test ! -e "$KEYHOME" +check "signing-key: import without a key fails" bash -c "! PATH='$SIGNPATH' GNUPGHOME='$KEYHOME' GPG_PRIVATE_KEY= bash '$here/signing-key.sh' import" +check "signing-key: import of something that is not a key fails" fails_with "could not import the signing key" "$L.16" \ + env PATH="$SIGNPATH" GNUPGHOME="$KEYHOME" GPG_PRIVATE_KEY="not a key" bash "$here/signing-key.sh" import +check "signing-key: import reads the private key from the environment" run "$L.17" env PATH="$SIGNPATH" GNUPGHOME="$KEYHOME" GPG_PRIVATE_KEY="$(cat "$S/key.asc")" bash "$here/signing-key.sh" import +check "signing-key: ... and reports the same fingerprint" is "$(out "$L.17" fingerprint)" "$FPR" +check "signing-key: ... the key material is not echoed" test -z "$(grep -E 'BEGIN PGP|STUBKEY' "$L.17" "$L.17.out" "$L.17.summary" 2> /dev/null)" +fresh; forget +check "sign: the imported key does not sign without its passphrase" fails_with "could not sign" "$L.18" sign "$L.18" "$FPR" "" +fresh +check "sign: the imported key signs with its passphrase" sign "$L.19" +check "check: ... and the bundles pass" chk "$L.20" +key "$L.21" remove > /dev/null 2>&1 +rm -f "$S/key.asc" + +# --------------------------------------------------------------------------- +# central-upload.sh, against a stub curl +# --------------------------------------------------------------------------- +C="$T/central"; mkdir -p "$C/bin" +cat > "$C/bin/curl" << 'STUB' +#!/usr/bin/env bash +# Stub curl for the Portal API (https://portal.test/api) and the public +# repository (https://repo.test/maven2). Records " [auth]". +set -u +method=GET; out=""; fmt=""; hdr=""; form=""; url=""; failflag=0 +while [ $# -gt 0 ]; do + case "$1" in + -X) method="$2"; shift 2 ;; + -H) hdr="$2"; shift 2 ;; + -o) out="$2"; shift 2 ;; + -w) fmt="$2"; shift 2 ;; + -F) form="$2"; shift 2 ;; + --connect-timeout | --max-time | --retry) shift 2 ;; + -sS) shift ;; + -fsS) failflag=1; shift ;; + -*) echo "stub curl: unexpected option $1" >&2; exit 97 ;; + *) url="$1"; shift ;; + esac +done +auth="" +case "$hdr" in + @*) grep -q '^Authorization: Bearer ' "${hdr#@}" && auth=" auth:$(sed 's/^Authorization: Bearer //' "${hdr#@}")" ;; + "") ;; + *) echo "stub curl: header passed on the command line: $hdr" >> "$CFAKE/calls.log" ;; +esac +echo "$method $url$auth" >> "$CFAKE/calls.log" +respond() { # respond + if [ -n "$out" ]; then printf '%s' "$2" > "$out"; else printf '%s' "$2"; fi + [ -z "$fmt" ] || printf '%s' "$1" +} +case "$url" in + https://repo.test/maven2/*.pom) + v="$(basename "$(dirname "$url")")" + respond "$(cat "$CFAKE/repo.$v.code" 2> /dev/null || echo 404)" "" ;; + https://repo.test/maven2/*.sha1) + f="$CFAKE/repo.$(basename "$url")" + [ -f "$f" ] || exit 22 + cat "$f" ;; + "https://portal.test/api/upload?name="*"&publishingType=USER_MANAGED") + [ "$method" = POST ] || exit 97 + n="$(grep -c 'POST https://portal.test/api/upload' "$CFAKE/calls.log")" + echo "$form" >> "$CFAKE/forms.log" + if [ -f "$CFAKE/upload.$n.exit" ]; then exit "$(cat "$CFAKE/upload.$n.exit")"; fi + respond "$(cat "$CFAKE/upload.$n.code" 2> /dev/null || echo 201)" "$(cat "$CFAKE/upload.$n.body" 2> /dev/null || echo "00000000-0000-4000-8000-00000000000$n")" ;; + "https://portal.test/api/status?id="*) + [ "$method" = POST ] || exit 97 + id="${url##*id=}"; f="$CFAKE/status.$id" + [ -f "$f" ] || echo VALIDATED > "$f" + st="$(head -n 1 "$f")" + [ "$(grep -c . "$f")" -le 1 ] || { tail -n +2 "$f" > "$f.next" && mv "$f.next" "$f"; } + case "$st" in + HTTP*) respond "${st#HTTP}" "upstream error" ;; + *) respond 200 "{\"deploymentId\":\"$id\",\"deploymentState\":\"$st\",\"errors\":{\"pkg\":[\"reason from the Portal\"]}}" ;; + esac ;; + https://portal.test/api/deployment/*) + id="${url##*/}" + case "$method" in + DELETE) respond "$(cat "$CFAKE/drop.$id.code" 2> /dev/null || echo 204)" "" ;; + *) echo "stub curl: $method on a deployment is the Publish call" >&2; exit 97 ;; + esac ;; + *) echo "stub curl: unexpected request: $method $url" >&2; exit 97 ;; +esac +STUB +chmod +x "$C/bin/curl" +TOKEN="$(printf 'user:pass' | base64 | tr -d '\n')" +new_central() { CFAKE="$C/fake.$1"; rm -rf "$CFAKE" "$C/state"; mkdir -p "$CFAKE"; : > "$CFAKE/calls.log"; export CFAKE; } +central() { # central + local log="$1"; shift + run "$log" env PATH="$C/bin:$PATH" CENTRAL_USERNAME=user CENTRAL_PASSWORD=pass CENTRAL_API_URL=https://portal.test/api \ + CENTRAL_REPO_URL=https://repo.test/maven2 CENTRAL_POLL_SECONDS=0 CENTRAL_WAIT_SECONDS="${WAIT_SECONDS:-5}" bash "$here/central-upload.sh" "$@" +} +up() { central "$1" upload "$S/bundles" "$S/staging" "$C/state"; } +calls() { grep -c -- "$1" "$CFAKE/calls.log" || true; } +ID1=00000000-0000-4000-8000-000000000001; ID2=00000000-0000-4000-8000-000000000002; ID3=00000000-0000-4000-8000-000000000003 +L="$C/log" + +new_central ok; printf 'PENDING\nVALIDATING\nVALIDATED\n' > "$CFAKE/status.$ID1" +check "central: three uploads, each validated" up "$L.1" +check "central: ... three upload calls" is "$(calls 'POST https://portal.test/api/upload')" 3 +check "central: ... as USER_MANAGED, named after the version" has "$CFAKE/calls.log" "POST https://portal.test/api/upload?name=indextables_spark-1.2.3_spark_3.5.8&publishingType=USER_MANAGED" +check "central: ... the bundle is the form field" has "$CFAKE/forms.log" "bundle=@$S/bundles/indextables_spark-1.2.3_spark_4.1.2-bundle.zip;type=application/octet-stream" +check "central: ... one version at a time: validated before the next upload" is \ + "$(grep -n -E 'upload\?name|status\?id' "$CFAKE/calls.log" | sed -e 's/^[0-9]*:POST https:\/\/portal.test\/api\///' -e 's/[?].*//' | tr '\n' ' ')" "upload status status status upload status upload status " +check "central: ... nothing is dropped" is "$(calls '^DELETE')" 0 +check "central: ... the Publish call is never made" is "$(calls '^POST https://portal.test/api/deployment/')" 0 +check "central: ... the token goes to the Portal as a Bearer header" is "$(grep -c "portal.test.* auth:$TOKEN\$" "$CFAKE/calls.log")" "$(grep -c 'portal.test' "$CFAKE/calls.log")" +check "central: ... and never to the public repository" is "$(grep 'repo.test' "$CFAKE/calls.log" | grep -c 'auth:')" 0 +check "central: ... and never on a command line" hasnt "$CFAKE/calls.log" "header passed on the command line" +check "central: ... the encoded token is masked before use" has "$L.1" "::add-mask::$TOKEN" +check "central: ... the summary says it is not public yet" has "$L.1.summary" "Not public yet" +check "central: ... deployments are recorded" is "$(cat "$C/state/deployments.txt" | tr '\n' ' ')" "1.2.3_spark_3.5.8 $ID1 1.2.3_spark_4.0.3 $ID2 1.2.3_spark_4.1.2 $ID3 " +check "central: a second upload from the same job is refused" fails_with "refusing to upload twice" "$L.1b" up "$L.1b" +: > "$CFAKE/calls.log" +check "central: drop removes this run's deployments" central "$L.1c" drop "$C/state" +check "central: ... all three" is "$(calls '^DELETE https://portal.test/api/deployment/')" 3 +check "central: ... and a second drop has nothing left to do" central "$L.1d" drop "$C/state" +check "central: ... (no further calls)" is "$(calls '^DELETE')" 3 + +new_central rejected; echo FAILED > "$CFAKE/status.$ID2" +check "central: a rejected bundle fails the step" fails_with "Maven Central rejected the bundle for 1.2.3_spark_4.0.3" "$L.2" up "$L.2" +check "central: ... the Portal's reasons are shown" has "$L.2" "reason from the Portal" +check "central: ... the third version is not uploaded" is "$(calls 'POST https://portal.test/api/upload')" 2 +check "central: ... both deployments of this run are dropped" is "$(grep '^DELETE' "$CFAKE/calls.log" | sed 's/.*deployment\///' | sed 's/ auth.*//' | tr '\n' ' ')" "$ID1 $ID2 " +check "central: ... nothing is left recorded as waiting" test ! -s "$C/state/deployments.txt" + +new_central refused; echo 401 > "$CFAKE/upload.1.code"; echo "invalid token" > "$CFAKE/upload.1.body" +check "central: a refused upload fails the step" fails_with "was refused: HTTP 401 invalid token" "$L.3" up "$L.3" +check "central: ... nothing else is attempted" is "$(calls 'POST https://portal.test/api/upload')$(calls 'status')$(calls '^DELETE')" 100 + +new_central broken; echo 28 > "$CFAKE/upload.2.exit" +check "central: an upload that does not complete fails the step" fails_with "may or may not have reached the Portal" "$L.4" up "$L.4" +check "central: ... the earlier deployment is dropped" is "$(grep -c "^DELETE https://portal.test/api/deployment/$ID1" "$CFAKE/calls.log")" 1 + +new_central garbled; echo "ok" > "$CFAKE/upload.1.body" +check "central: an upload answer that is not a deployment id fails the step" fails_with "unexpected response" "$L.5" up "$L.5" + +new_central slow; echo VALIDATING > "$CFAKE/status.$ID1"; echo 400 > "$CFAKE/drop.$ID1.code" +WAIT_SECONDS=0 +check "central: a validation that does not finish in time fails the step" fails_with "was not validated within 0 seconds" "$L.6" up "$L.6" +unset WAIT_SECONDS +check "central: ... a drop that is refused is reported, with what to do" has "$L.6" "Could not drop deployment $ID1" +check "central: ... 'do not publish it'" has "$L.6" "do not publish it" + +new_central flaky; printf 'HTTP502\nHTTP502\nVALIDATED\n' > "$CFAKE/status.$ID1" +check "central: a few failed status calls are tolerated" up "$L.7" +new_central down; echo HTTP503 > "$CFAKE/status.$ID1" +check "central: status calls that keep failing fail the step" fails_with "could not be read six times in a row" "$L.8" up "$L.8" + +new_central raced; echo PUBLISHED > "$CFAKE/status.$ID1" +check "central: a deployment published by hand during the run is not an error" up "$L.9" +check "central: ... but is called out" has "$L.9" "someone pressed Publish" + +# Re-run after a partial publish: 3.5.8 is already public with these files. +vd="$S/staging/bundles/1.2.3_spark_3.5.8/io/indextables/indextables_spark/1.2.3_spark_3.5.8" +same_on_central() { echo 200 > "$CFAKE/repo.1.2.3_spark_3.5.8.code"; for n in $(artifact_names 1.2.3_spark_3.5.8); do hash_of sha1 "$vd/$n" > "$CFAKE/repo.$n.sha1"; done; } +new_central partial; same_on_central +check "central: a version already public with identical files is skipped" up "$L.10" +check "central: ... only the other two are uploaded" is "$(grep 'upload?name' "$CFAKE/calls.log" | sed -e 's/.*name=indextables_spark-//' -e 's/&.*//' | tr '\n' ' ')" "1.2.3_spark_4.0.3 1.2.3_spark_4.1.2 " +check "central: ... and the summary says so" has "$L.10.summary" "ALREADY_PUBLISHED" +new_central differs; same_on_central; echo 0000000000000000000000000000000000000000 > "$CFAKE/repo.indextables_spark-1.2.3_spark_3.5.8-javadoc.jar.sha1" +check "central: a version already public with other files stops the release" fails_with "cannot be replaced" "$L.11" up "$L.11" +check "central: ... before anything is uploaded" is "$(calls 'portal.test')" 0 +new_central unknown; echo 503 > "$CFAKE/repo.1.2.3_spark_4.1.2.code" +check "central: an unreadable public repository stops the release" fails_with "could not tell whether 1.2.3_spark_4.1.2 is already on Maven Central" "$L.12" up "$L.12" +check "central: ... before anything is uploaded" is "$(calls 'portal.test')" 0 +new_central nocreds +check "central: missing credentials fail before any upload" bash -c "! PATH='$C/bin:$PATH' CENTRAL_USERNAME= CENTRAL_PASSWORD= CENTRAL_API_URL=https://portal.test/api CENTRAL_REPO_URL=https://repo.test/maven2 bash '$here/central-upload.sh' upload '$S/bundles' '$S/staging' '$C/state'" +check "central: ... (no Portal call)" is "$(calls 'portal.test')" 0 +check "central: the default endpoints are the Portal API and repo1" test -n "$(grep -F 'CENTRAL_API_URL:-https://central.sonatype.com/api/v1/publisher}' "$here/central-upload.sh")" -a -n "$(grep -F 'CENTRAL_REPO_URL:-https://repo1.maven.org/maven2}' "$here/central-upload.sh")" +check "central: AUTOMATIC publishing appears nowhere" test -z "$(grep -l 'AUTOMATIC' "$here"/*.sh | grep -v test.sh)" + +# --------------------------------------------------------------------------- +# github-release.sh, against a stub gh +# --------------------------------------------------------------------------- +G="$T/gh"; mkdir -p "$G/bin" +cat > "$G/bin/gh" << 'STUB' +#!/usr/bin/env bash +# Stub gh: tag lookups and release view/create/upload/edit from $GFAKE. +set -u +echo "gh $*" >> "$GFAKE/calls.log" +case "$1 ${2:-}" in + "api repos/o/r/git/ref/tags/"*) [ -f "$GFAKE/ref.json" ] || exit 1; cat "$GFAKE/ref.json" ;; + "api repos/o/r/git/tags/"*) cat "$GFAKE/tagobj.sha" ;; + "release view") + case "$*" in + *"--json body"*) if [ -f "$GFAKE/body.md" ]; then cat "$GFAKE/body.md"; else cat "$GFAKE/view.err" >&2; exit 1; fi ;; + *"--json url"*) echo "https://example.invalid/releases/$3" ;; + esac ;; + "release upload") ;; + "release edit") shift 3; [ "$1" = --notes-file ] && cp "$2" "$GFAKE/body.md" ;; + "release create") printf "## What's Changed\n* a change\n" > "$GFAKE/body.md" ;; + *) echo "stub gh: unexpected call: $*" >&2; exit 97 ;; +esac +STUB +chmod +x "$G/bin/gh" +new_gh() { GFAKE="$G/fake.$1"; rm -rf "$GFAKE"; mkdir -p "$GFAKE"; : > "$GFAKE/calls.log"; export GFAKE + printf '{"object":{"type":"commit","sha":"%s"}}\n' "$SHA_D" > "$GFAKE/ref.json"; echo "release not found" > "$GFAKE/view.err"; } +release() { # release [VAR=value...] + local log="$1"; shift + run "$log" env PATH="$G/bin:$PATH" GH_TOKEN=t GH_REPO=o/r TAG=v1.2.3 COMMIT="$SHA_D" PRERELEASE=false "$@" bash "$here/github-release.sh" "$A/staging.good" +} +L="$G/log" +new_gh create +check "release: creates the release when there is none" release "$L.1" +check "release: ... for an existing tag, with generated notes" has "$GFAKE/calls.log" "gh release create v1.2.3 --verify-tag --generate-notes $A/staging.good/bundles/" +check "release: ... the generated notes are kept" has "$GFAKE/body.md" "What's Changed" +check "release: ... with the three shaded jars" is "$(grep 'release create' "$GFAKE/calls.log" | tr ' ' '\n' | grep -c -- '-linux-x86_64-shaded.jar$')" 3 +check "release: ... and no other file" is "$(grep 'release create' "$GFAKE/calls.log" | tr ' ' '\n' | grep -c '/bundles/')" 3 +check "release: ... not marked as a pre-release" test -z "$(grep -- '--prerelease' "$GFAKE/calls.log")" +check "release: ... notes carry the build inputs" has "$GFAKE/body.md" "$T4J_COMMIT" +new_gh pre +check "release: a pre-release tag creates a pre-release" release "$L.2" PRERELEASE=true +check "release: ... flag passed" has "$GFAKE/calls.log" "--prerelease" +new_gh existing; printf 'Hand-written notes.\r\n\r\n- item\r\n\r\n' > "$GFAKE/body.md" +check "release: an existing release gets the jars" release "$L.3" +check "release: ... replacing assets of the same name" has "$GFAKE/calls.log" "gh release upload v1.2.3 --clobber" +check "release: ... it is not created again" test -z "$(grep 'release create' "$GFAKE/calls.log")" +check "release: ... its notes are kept" has "$GFAKE/body.md" "Hand-written notes." +check "release: ... and gain the build inputs" has "$GFAKE/body.md" "$T4J_COMMIT" +release "$L.4" > /dev/null 2>&1 +check "release: a re-run replaces the build inputs instead of adding a second block" is "$(grep -c 'release-build-inputs:start' "$GFAKE/body.md")" 1 +check "release: ... and keeps the notes" is "$(grep -c 'Hand-written notes.' "$GFAKE/body.md")" 1 +new_gh moved; printf '{"object":{"type":"commit","sha":"%s"}}\n' "$SHA_A" > "$GFAKE/ref.json" +check "release: a tag that moved since the build is not released" fails_with "now points to $SHA_A" "$L.5" release "$L.5" +check "release: ... no release call is made" test -z "$(grep 'gh release' "$GFAKE/calls.log")" +new_gh annotated; printf '{"object":{"type":"tag","sha":"%s"}}\n' "$SHA_B" > "$GFAKE/ref.json"; echo "$SHA_D" > "$GFAKE/tagobj.sha" +check "release: an annotated tag is followed to its commit" release "$L.6" +new_gh gone; rm "$GFAKE/ref.json" +check "release: a tag that no longer exists is not released" fails_with "does not exist" "$L.7" release "$L.7" +new_gh error; echo "HTTP 502: Bad Gateway" > "$GFAKE/view.err" +check "release: a failed lookup does not fall through to creating a release" fails_with "could not look up release" "$L.8" release "$L.8" +check "release: ... no create call" test -z "$(grep 'release create' "$GFAKE/calls.log")" + +# --------------------------------------------------------------------------- +# The whole chain with the relative paths release.yml passes, from one +# working directory, the way the jobs run it. +# --------------------------------------------------------------------------- +E="$T/chain"; mkdir -p "$E/release/.github/scripts" "$E/src" +cp -R "$here" "$E/release/.github/scripts/release" +cp "$N/pom.xml" "$E/src/pom.xml" +cp "$N/pins.txt" "$E/release/.github/scripts/release/native-pins.txt" +ES=release/.github/scripts/release +chain() { # chain : run in $E with the stubs on PATH + local log="$1"; shift + (cd "$E" && GITHUB_OUTPUT="$log.out" GITHUB_STEP_SUMMARY="$log.summary" PATH="$C/bin:$G/bin:$T/bin:${SIGNPATH}" "$@") > "$log" 2>&1 +} +L="$E/log" +check "chain: native resolve" chain "$L.1" env T4J_REPO="file://$N/t4j" POM=src/pom.xml PINS="$ES/native-pins.txt" WORK=work/native OUT=out/native M2_REPO="$E/m2-native" bash "$ES/build-native.sh" resolve +mkdir -p "$E/work/native/protoc/bin" && cp "$N/work/protoc/bin/protoc" "$E/work/native/protoc/bin/protoc" 2> /dev/null \ + || { printf '#!/bin/sh\necho "libprotoc 25.5"\n' > "$E/work/native/protoc/bin/protoc"; chmod +x "$E/work/native/protoc/bin/protoc"; } +check "chain: native build" chain "$L.2" env T4J_REPO="file://$N/t4j" POM=src/pom.xml PINS="$ES/native-pins.txt" WORK=work/native OUT=out/native M2_REPO="$E/m2-native" bash "$ES/build-native.sh" build +mkdir -p "$E/in" && cp -R "$E/out/native" "$E/in/native" +for profile in $PROFILES; do + rm -rf "$E/src/target" "$E/src/.stamped" "$E/src/dependency-reduced-pom.xml" + check "chain: build $profile" chain "$L.3" env SRC=src PROFILE="$profile" BASE_VERSION=1.2.3 NATIVE_DIR=in/native OUT=out/artifacts M2_REPO="$E/m2-$profile" bash "$ES/build-artifacts.sh" + cp -R "$E/out/artifacts" "$E/in/artifacts-$profile" +done +check "chain: assemble" chain "$L.4" env IN=in OUT=staging TAG=v1.2.3 COMMIT="$SHA_D" BASE_VERSION=1.2.3 bash "$ES/assemble.sh" +CM="$(out "$L.4" manifest_sha256)" +check "chain: verify" chain "$L.5" env MANIFEST_SHA256="$CM" BASE_VERSION=1.2.3 bash "$ES/verify-staging.sh" staging +check "chain: generate a key" chain "$L.6" env GNUPGHOME="$KEYHOME" GPG_PASSPHRASE=rehearsal bash "$ES/signing-key.sh" generate +CK="$(out "$L.6" fingerprint)" +check "chain: sign" chain "$L.7" env GNUPGHOME="$KEYHOME" SIGNING_KEY="$CK" GPG_PASSPHRASE=rehearsal bash "$ES/sign-bundles.sh" staging bundles +check "chain: check" chain "$L.8" env GNUPGHOME="$KEYHOME" SIGNING_KEY="$CK" BASE_VERSION=1.2.3 bash "$ES/check-bundles.sh" bundles staging +check "chain: remove the key" chain "$L.9" env GNUPGHOME="$KEYHOME" bash "$ES/signing-key.sh" remove +new_central chain; rm -rf "$E/state" +check "chain: upload" chain "$L.10" env CENTRAL_USERNAME=user CENTRAL_PASSWORD=pass CENTRAL_API_URL=https://portal.test/api CENTRAL_REPO_URL=https://repo.test/maven2 \ + CENTRAL_POLL_SECONDS=0 bash "$ES/central-upload.sh" upload bundles staging "$E/state" +check "chain: ... three bundles went up" is "$(calls 'POST https://portal.test/api/upload')" 3 +check "chain: ... as form uploads of the relative bundle paths" has "$CFAKE/forms.log" "bundle=@bundles/indextables_spark-1.2.3_spark_4.0.3-bundle.zip;type=application/octet-stream" +new_gh chain +check "chain: release" chain "$L.11" env GH_TOKEN=t GH_REPO=o/r TAG=v1.2.3 COMMIT="$SHA_D" PRERELEASE=false bash "$ES/github-release.sh" staging +check "chain: ... with the staged shaded jars" is "$(grep 'release create' "$GFAKE/calls.log" | tr ' ' '\n' | grep -c '^staging/bundles/.*-linux-x86_64-shaded.jar$')" 3 + +# --------------------------------------------------------------------------- +# release.yml +# --------------------------------------------------------------------------- +if command -v ruby > /dev/null 2>&1 && ruby -ryaml -rjson -e 'puts JSON.generate(YAML.load_file(ARGV[0]))' "$workflow" > "$T/wf.json" 2> /dev/null; then + wf() { jq -r "$1" "$T/wf.json"; } + secret_jobs="$(wf '[.jobs | to_entries[] | select(.value | tostring | test("secrets\\.")) | .key] | join(",")')" + check "workflow: started by workflow_dispatch only" is "$(wf '(.on // .["true"]) | keys | join(",")')" workflow_dispatch + check "workflow: dry-run is a boolean that defaults to true" is "$(wf '(.on // .["true"]).workflow_dispatch.inputs["dry-run"] | [.type, (.default | tostring)] | join(",")')" boolean,true + check "workflow: the tag is a required input" is "$(wf '(.on // .["true"]).workflow_dispatch.inputs.tag.required | tostring')" true + check "workflow: no permissions by default" is "$(wf '.permissions | length')" 0 + check "workflow: jobs" is "$(wf '.jobs | keys | join(",")')" assemble,build,native,plan,publish,rehearse + check "workflow: only publish names an environment" is "$(wf '[.jobs | to_entries[] | select(.value | has("environment")) | .key + "=" + (.value.environment | tostring)] | join(",")')" publish=release + check "workflow: only publish reads secrets" is "$secret_jobs" publish + check "workflow: publish reads exactly the four publishing secrets" is "$(wf '[.jobs.publish | tostring | scan("secrets\\.([A-Z_]+)") | .[0]] | unique | join(",")')" CENTRAL_PASSWORD,CENTRAL_USERNAME,GPG_PASSPHRASE,GPG_PRIVATE_KEY + check "workflow: secrets reach single steps only, never a whole job" is "$(wf '[.env // {}, (.jobs[].env // {})] | tostring | test("secrets\\.")')" false + check "workflow: each secret goes only to the step that needs it" is "$(wf '[.jobs.publish.steps[] | select(.env // {} | tostring | test("secrets\\.")) | .name + ":" + ([.env | to_entries[] | select(.value | tostring | test("secrets\\.")) | .key] | join("+"))] | join(",")')" \ + "Import the signing key:GPG_PRIVATE_KEY,Sign and bundle:GPG_PASSPHRASE,Upload to Maven Central:CENTRAL_USERNAME+CENTRAL_PASSWORD,Drop Maven Central deployments after a failure:CENTRAL_USERNAME+CENTRAL_PASSWORD" + check "workflow: only publish can write" is "$(wf '[.jobs | to_entries[] | select(.value.permissions != {"contents": "read"}) | .key + "=" + (.value.permissions | tostring)] | join(",")')" 'publish={"contents":"write"}' + check "workflow: the job token is used by the GitHub Release step only" is "$(wf '[.jobs[].steps[] | select(tostring | test("github\\.token|GITHUB_TOKEN")) | .name] | join(",")')" "GitHub Release" + cond="$(wf '.jobs.publish.if')" + check "workflow: publish needs the plan's verdict" grep -qF "needs.plan.outputs.mode == 'publish'" <<< "$cond" + check "workflow: ... and, independently, the raw input" grep -qF "!(github.event_name == 'workflow_dispatch' && inputs.dry-run)" <<< "$cond" + check "workflow: ... joined with &&, no ||, no status function that would override a failed need" test -z "$(grep -E '\|\||always\(\)|failure\(\)|cancelled\(\)' <<< "$cond")" + check "workflow: publish runs after assemble and the rehearsal" is "$(wf '.jobs.publish.needs | sort | join(",")')" assemble,plan,rehearse + check "workflow: no other job depends on publish" is "$(wf '[.jobs[] | .needs // [] | if type == "array" then .[] else . end | select(. == "publish")] | length')" 0 + check "workflow: one publish at a time, never cancelled by a newer one" is "$(wf '.jobs.publish.concurrency | [.group, (.["cancel-in-progress"] | tostring)] | join(",")')" release-publish,false + check "workflow: nothing is restored from or saved to a cache" is "$(wf '[.jobs[].steps[] | select(((.uses // "") | test("cache")) or ((.with // {}) | has("cache")))] | length')" 0 + check "workflow: project code is checked out only by native and build" is "$(wf '[.jobs | to_entries[] | select([.value.steps[] | select((.uses // "") | startswith("actions/checkout@")) | .with.ref // ""] | any(. == "${{ needs.plan.outputs.commit }}")) | .key] | sort | join(",")')" build,native + check "workflow: assemble, rehearse and publish check out the release scripts only" is "$(wf '[.jobs | to_entries[] | select(.key == "assemble" or .key == "rehearse" or .key == "publish") | .value.steps[] | select((.uses // "") | startswith("actions/checkout@")) | [.with.ref, .with["sparse-checkout"]] | join(" ")] | unique | join(",")')" '${{ github.workflow_sha }} .github/scripts/release' + check "workflow: no JDK and no Maven in assemble, rehearse or publish" is "$(wf '[.jobs | to_entries[] | select(.key == "assemble" or .key == "rehearse" or .key == "publish") | .value.steps[] | select(((.uses // "") | test("setup-java")) or ((.run // "") | test("mvn|java |cargo")))] | length')" 0 + check "workflow: publish uses two actions, checkout and download-artifact" is "$(wf '[.jobs.publish.steps[] | .uses // empty | sub("@.*"; "")] | unique | join(",")')" actions/checkout,actions/download-artifact + check "workflow: every checkout drops its credentials" is "$(wf '[.jobs[].steps[] | select((.uses // "") | startswith("actions/checkout@")) | .with["persist-credentials"]] | (length == 8 and all(. == false))')" true + check "workflow: every job has a timeout" is "$(wf '[.jobs[] | has("timeout-minutes")] | all')" true + check "workflow: hosted runner, pinned image" is "$(wf '[.jobs[]["runs-on"]] | unique | join(",")')" ubuntu-24.04 + check "workflow: no expression is expanded inside a run script" is "$(wf '[.jobs[].steps[] | .run? // empty | select(test("\\$\\{\\{"))] | length')" 0 + check "workflow: build matrix lists the profiles the scripts expect" is "$(wf '.jobs.build.strategy.matrix.profile | join(" ")')" "$PROFILES" + steps="$(wf '[.jobs.publish.steps[].name] | join("|")')" + check "workflow: publish verifies before it imports, and removes the key before any upload" is "$steps" \ + "Checkout release scripts (workflow commit)|Keep the keyring and upload state outside the workspace|Download staging directory|Verify staging directory|Import the signing key|Sign and bundle|Check bundles|Remove the signing key|Upload to Maven Central|GitHub Release|Drop Maven Central deployments after a failure" + check "workflow: the key is removed even when an earlier step failed" is "$(wf '.jobs.publish.steps[] | select(.name == "Remove the signing key") | .if')" "always()" + check "workflow: rehearse runs the same verify, sign and check commands as publish" is \ + "$(wf '[.jobs.rehearse.steps[] | .run? // empty | select(test("verify-staging|sign-bundles|check-bundles"))] | join("|")')" \ + "$(wf '[.jobs.publish.steps[] | .run? // empty | select(test("verify-staging|sign-bundles|check-bundles"))] | join("|")')" + check "workflow: both take the manifest digest from the assemble job's output" is "$(wf '[.jobs.rehearse, .jobs.publish | .steps[] | select(.name == "Verify staging directory") | .env.MANIFEST_SHA256] | unique | join(",")')" '${{ needs.assemble.outputs.manifest-sha256 }}' + check "workflow: plan gets the inputs as strings, through the environment" is "$(wf '.jobs.plan.steps[] | select(.id == "plan") | .env | [.INPUT_TAG, .INPUT_DRY_RUN, .EVENT_NAME, .REF] | join(" ")')" '${{ inputs.tag }} ${{ inputs.dry-run }} ${{ github.event_name }} ${{ github.ref }}' + check "workflow: no continue-on-error anywhere" is "$(wf '[.. | objects | select(has("continue-on-error"))] | length')" 0 + # Every `run:` step, as the runner would start it (bash -eo pipefail). + # Steps that call a release script must name one that exists; the other + # steps are small enough to execute here. + missing="" + while IFS= read -r script; do + [ -f "$here/$script" ] || missing="$missing $script" + done < <(wf '.jobs[].steps[] | .run? // empty' | grep -o -E '(\$SCRIPTS|\.github/scripts/release)/[a-z-]+\.sh' | sed 's|.*/||' | sort -u) + check "workflow: every script a step calls exists" test -z "$missing" + check "workflow: every run step either calls one release script or is one of the three inline steps" is \ + "$(wf '[.jobs[].steps[] | select(has("run")) | select(.run | test("^bash (\"\\$SCRIPTS|\\.github/scripts/release)/[a-z-]+\\.sh\"?( [a-z-]+| \"\\$CENTRAL_STATE\")*\\n?$") | not) | .name] | join("|")')" \ + "Keep the keyring outside the workspace|Summary|Keep the keyring and upload state outside the workspace" + inline() { # inline [VAR=value...]: run that step's script + local job="$1" name="$2"; shift 2 + wf ".jobs.$job.steps[] | select(.name == \"$name\") | .run" > "$T/inline.sh" + env RUNNER_TEMP="$T/rt" GITHUB_ENV="$T/inline.env" GITHUB_STEP_SUMMARY="$T/inline.summary" "$@" bash --noprofile --norc -eo pipefail "$T/inline.sh" + } + : > "$T/inline.env" + check "workflow: rehearse sets the keyring directory" inline rehearse "Keep the keyring outside the workspace" + check "workflow: ... to a directory signing-key.sh accepts" is "$(cat "$T/inline.env")" "GNUPGHOME=$T/rt/release-gnupg" + : > "$T/inline.env" + check "workflow: publish sets the keyring and state directories" inline publish "Keep the keyring and upload state outside the workspace" + check "workflow: ... both outside the workspace" is "$(tr '\n' ' ' < "$T/inline.env")" "GNUPGHOME=$T/rt/release-gnupg CENTRAL_STATE=$T/rt/central-state " + : > "$T/inline.summary" + check "workflow: rehearse summary step, dry run" inline rehearse Summary MODE=dry-run + check "workflow: ... says nothing was uploaded" has "$T/inline.summary" "nothing was uploaded and no release was created" + : > "$T/inline.summary" + check "workflow: rehearse summary step, publish run" inline rehearse Summary MODE=publish + check "workflow: ... says approval comes next" has "$T/inline.summary" "waits for approval" + check "workflow: ... and does not claim a dry run" hasnt "$T/inline.summary" "Dry run complete" +elif [ "${REQUIRE_WORKFLOW_CHECKS:-0}" = "1" ]; then + bad "workflow: checks could not run (ruby not available, or release.yml does not parse)" +else + echo "skip workflow checks (ruby not available)" +fi + +# Pinned actions and runner image, in this workflow and the one that runs +# these tests. +unpinned="" +for f in "$workflow" "$root/.github/workflows/release-scripts.yml"; do + n=$(grep -c -E '^[[:space:]]*(- )?uses:' "$f") + m=$(grep -c -E '^[[:space:]]*(- )?uses: [A-Za-z0-9._/-]+@[0-9a-f]{40} # v[0-9]+(\.[0-9]+)*$' "$f") + if [ "$n" -eq 0 ] || [ "$n" -ne "$m" ]; then unpinned="$unpinned $(basename "$f")"; fi +done +check "pins: every action is a full commit id with its version tag in a comment" test -z "$unpinned" +check "pins: no floating runner label" test -z "$(grep -h -E 'runs-on:' "$workflow" "$root/.github/workflows/release-scripts.yml" | grep -v -E 'runs-on: ubuntu-24\.04$')" +check "scripts: none enables command tracing" test -z "$(grep -l -E '^[[:space:]]*set -[a-z]*x|xtrace' "$here"/*.sh | grep -v test.sh)" +check "scripts: no installer is piped into a shell" test -z "$(grep -n -E 'curl[^|]*\|[[:space:]]*(ba)?sh' "$here"/*.sh | grep -v test.sh)" + +echo +echo "$pass passed, $fail failed" +[ "$fail" -eq 0 ] diff --git a/.github/scripts/release/verify-staging.sh b/.github/scripts/release/verify-staging.sh new file mode 100644 index 00000000..a07696b6 --- /dev/null +++ b/.github/scripts/release/verify-staging.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# Check a downloaded staging directory before anything in it is signed. +# +# verify-staging.sh +# +# Environment: +# MANIFEST_SHA256 the digest the assemble job reported as a job output +# BASE_VERSION the tag without its "v" +# +# The staging directory travels between jobs as a workflow artifact. Its +# manifest digest travels separately, as a job output, so the two can be +# compared here. After this passes, the directory holds exactly the files the +# assemble job checked and listed: nothing added, removed or changed, and +# every version in it belongs to the tag being released. +set -euo pipefail +here="$(cd "$(dirname "$0")" && pwd)" +# shellcheck source=lib.sh +. "$here/lib.sh" + +staging="${1:?usage: verify-staging.sh }" +: "${MANIFEST_SHA256:?}" "${BASE_VERSION:?}" +[[ "$MANIFEST_SHA256" =~ ^[0-9a-f]{64}$ ]] || die "MANIFEST_SHA256 is not a SHA-256 digest" + +[ -f "$staging/MANIFEST.sha256" ] || die "$staging has no MANIFEST.sha256" +got="$(hash_of sha256 "$staging/MANIFEST.sha256")" +[ "$got" = "$MANIFEST_SHA256" ] \ + || die "the staging manifest has SHA-256 $got, but the assemble job reported $MANIFEST_SHA256" +verify_sums "$staging" MANIFEST.sha256 + +# Independent of what assemble.sh did: one version per Spark profile, each +# belonging to this tag, and the manifest may only list the notes, the +# version list and the allow-listed files of those versions. +[ -s "$staging/versions.txt" ] || die "the staging directory lists no versions" +set -- $PROFILES +while IFS= read -r version; do + [ $# -gt 0 ] || die "the staging directory has more versions than Spark profiles" + check_version "$BASE_VERSION" "$1" "$version" + shift +done < "$staging/versions.txt" +[ $# -eq 0 ] || die "the staging directory has no version for: $*" + +expected="$( + { + echo release-notes.md + echo versions.txt + while IFS= read -r version; do + for name in $(artifact_names "$version"); do + echo "bundles/$version/$GROUP_PATH/$version/$name" + for algo in $CHECKSUM_ALGOS; do echo "bundles/$version/$GROUP_PATH/$version/$name.$algo"; done + done + done < "$staging/versions.txt" + } | LC_ALL=C sort +)" +listed="$(awk '{ print $2 }' "$staging/MANIFEST.sha256" | LC_ALL=C sort)" +[ "$expected" = "$listed" ] || die "the staging manifest lists files other than the expected ones" + +echo "Staging directory verified: manifest $MANIFEST_SHA256, versions: $(tr '\n' ' ' < "$staging/versions.txt")" diff --git a/.github/workflows/release-scripts.yml b/.github/workflows/release-scripts.yml new file mode 100644 index 00000000..09fd7871 --- /dev/null +++ b/.github/workflows/release-scripts.yml @@ -0,0 +1,56 @@ +name: Release Scripts + +# Offline tests for the scripts behind release.yml and for the shape of +# release.yml itself, run on the runner image those scripts run on, with the +# gpg that will do the signing (a throwaway key in a temporary keyring). +# Uses no secrets, no environment and a read-only token, and uploads nothing, +# so it is safe as an ordinary pull_request workflow. The release workflow +# itself cannot be exercised from a pull request; see the dry run in +# .github/scripts/release/README.md. +# +# pom.xml and scripts/setup.sh are in the path filter because the tests check +# them against the pins: the tantivy4java version in pom.xml needs a line in +# native-pins.txt, and the protoc pin must match the one in setup.sh. + +on: + pull_request: + paths: + - .github/scripts/release/** + - .github/workflows/release.yml + - .github/workflows/release-scripts.yml + - pom.xml + - scripts/setup.sh + push: + branches: [main] + paths: + - .github/scripts/release/** + - .github/workflows/release.yml + - .github/workflows/release-scripts.yml + - pom.xml + - scripts/setup.sh + +permissions: + contents: read + +concurrency: + group: release-scripts-${{ github.ref }} + cancel-in-progress: true + +jobs: + test: + name: Offline tests + runs-on: ubuntu-24.04 + timeout-minutes: 10 + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Run the tests + env: + # Fail, rather than fall back or skip, if gpg or the workflow-file + # checks are not available. + REQUIRE_GPG: '1' + REQUIRE_WORKFLOW_CHECKS: '1' + run: bash .github/scripts/release/test.sh diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 3e5bd4e7..2ad93c0a 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,136 +1,469 @@ name: Release -# Tag-triggered release: pushing v (e.g. v0.6.0) builds and publishes -# one artifact per supported Spark version, versioned _spark_ -# (e.g. 0.6.0_spark_4.0.3), each containing the linux-x86_64 tantivy4java -# native library: -# - Maven Central via the `release` profile (GPG + central-publishing plugin) -# - shaded jars attached to a GitHub Release for the tag +# Publishes one artifact per supported Spark version for an existing tag +# v (e.g. v0.6.0). Each is versioned _spark_ +# (e.g. 0.6.0_spark_4.0.3) and contains the linux-x86_64 tantivy4java native +# library: +# - Maven Central: io.indextables:indextables_spark:_spark_ +# (pom, jar, sources, javadoc, linux-x86_64-shaded, jar-with-dependencies) +# - the linux-x86_64-shaded jars attached to the GitHub Release for the tag +# is the spark.version that the profile sets in pom.xml at the tag. +# +# How to run it +# ------------- +# Actions -> Release -> Run workflow, on `main`, with the tag to release. +# `dry-run` is ticked by default: the run then builds, assembles and rehearses +# signing, and stops. Untick it to publish. Pushing a tag does not start a +# release. The runbook, including the first-release checklist and how to +# abort, is in .github/scripts/release/README.md. +# +# Trust model +# ----------- +# * `native` and `build` run the project's build (Maven, its plugins, its +# dependencies, cargo) on the tagged commit. They have a read-only token, no +# environment and no secrets, and restore nothing from a cache: the native +# library is compiled in the run, from the commits pinned in +# native-pins.txt. +# * `assemble`, `rehearse` and `publish` run no project code. They run the +# scripts in .github/scripts/release from the commit this workflow was +# started on, on files the build jobs produced. `assemble` treats those +# files as untrusted: it accepts only the expected coordinates and file +# names and writes a manifest of digests. +# * `publish` is the only job that names the `release` environment, reads a +# secret or holds a token that can write. Beside the signing key and the +# Central token it runs: the pinned checkout and download-artifact actions, +# the release scripts, and gpg, zip, unzip, curl, jq and gh from the runner +# image. No Maven, no JVM, nothing from the tagged commit. It signs and +# uploads exactly the files in the manifest whose digest `assemble` +# reported, and nothing else. +# * A dry run never starts `publish`, so it requests no approval and reads no +# secret. `rehearse` runs the same verification, signing and bundle checks +# as `publish`, with a throwaway key it generates and deletes. +# +# What this depends on outside this file: the `release` environment must +# admit only `main` and require a reviewer, and the four publishing secrets +# must exist in that environment only. See the README. + on: - push: - tags: ['v*'] workflow_dispatch: inputs: tag: description: 'Existing v* tag to release (e.g. v0.6.0)' required: true + type: string + dry-run: + description: 'Dry run: build, assemble and rehearse signing; upload nothing. Untick to publish.' + type: boolean + default: true skip-central: - description: 'Skip Maven Central publish (GitHub Release only)' + description: 'Skip Maven Central (GitHub Release only). No effect in a dry run.' type: boolean default: false + # To start a release from a pushed tag as well, add: + # + # push: + # tags: ['v*'] + # + # plan.sh already handles that event. A pushed tag runs the workflow file + # and the scripts of the tagged commit, not those on main; see "Tag push or + # dispatch" in the README before enabling it. + +permissions: {} -permissions: - contents: write +defaults: + run: + shell: bash env: - RELEASE_TAG: ${{ github.event_name == 'workflow_dispatch' && inputs.tag || github.ref_name }} + SCRIPTS: release/.github/scripts/release jobs: - # tantivy4java is Spark-independent: build the native (Rust) jar once and - # share it across all Spark publish legs. The cache makes re-releases with - # an unchanged tantivy4java version skip the Rust build entirely - # (setup.sh no-ops when the jar is already in ~/.m2). - native: - name: tantivy4java (linux-x86_64) - runs-on: ubuntu-latest + # --------------------------------------------------------------------------- + # Validate the request and resolve the tag to a commit. Every later job uses + # that commit, so a tag that moves during the run changes nothing. + # --------------------------------------------------------------------------- + plan: + name: Plan + runs-on: ubuntu-24.04 + timeout-minutes: 10 + permissions: + contents: read outputs: - t4j-version: ${{ steps.t4j.outputs.version }} + mode: ${{ steps.plan.outputs.mode }} + tag: ${{ steps.plan.outputs.tag }} + commit: ${{ steps.plan.outputs.commit }} + base: ${{ steps.plan.outputs.base }} + prerelease: ${{ steps.plan.outputs.prerelease }} + central: ${{ steps.plan.outputs.central }} + steps: + # Full history: plan.sh checks that the tag's commit is on main. + - name: Checkout (workflow commit, full history) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + + - name: Plan + id: plan + env: + EVENT_NAME: ${{ github.event_name }} + REF: ${{ github.ref }} + REF_TYPE: ${{ github.ref_type }} + REF_NAME: ${{ github.ref_name }} + DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} + INPUT_TAG: ${{ inputs.tag }} + INPUT_DRY_RUN: ${{ inputs.dry-run }} + INPUT_SKIP_CENTRAL: ${{ inputs.skip-central }} + run: bash .github/scripts/release/plan.sh + + # --------------------------------------------------------------------------- + # tantivy4java is Spark-independent: build the native (Rust) jar once, from + # source, and share it with the three build legs. No cache: about 11 minutes + # in every run, which is what each release run has taken so far. + # --------------------------------------------------------------------------- + native: + name: tantivy4java (linux-x86_64, from source) + needs: plan + runs-on: ubuntu-24.04 + timeout-minutes: 45 + permissions: + contents: read + env: + POM: src/pom.xml + PINS: release/.github/scripts/release/native-pins.txt + WORK: work/native + OUT: out/native steps: - - name: Checkout - uses: actions/checkout@v7 + - name: Checkout release scripts (workflow commit) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: - ref: ${{ env.RELEASE_TAG }} + ref: ${{ github.workflow_sha }} + path: release + sparse-checkout: .github/scripts/release + persist-credentials: false + + - name: Checkout source (tagged commit) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ needs.plan.outputs.commit }} + path: src + persist-credentials: false - name: Set up Java 17 - uses: actions/setup-java@v6 + uses: actions/setup-java@de7274f081f381c8f8158605e0321c36c376e2e6 # v6.0.1 with: distribution: 'temurin' java-version: '17' - - name: Extract tantivy4java version from pom.xml - id: t4j - run: | - version=$(grep -A2 'tantivy4java' pom.xml \ - | grep '' | sed -E 's/.*([^<]+)<\/version>.*/\1/' | head -1) - [[ -n "$version" ]] || { echo "could not extract tantivy4java version" >&2; exit 1; } - echo "version=$version" >> "$GITHUB_OUTPUT" + - name: Resolve pinned inputs + run: bash "$SCRIPTS/build-native.sh" resolve - - name: Cache tantivy4java build - id: t4j-cache - uses: actions/cache@v6 - with: - path: ~/.m2/repository/io/indextables/tantivy4java - key: tantivy4java-${{ steps.t4j.outputs.version }}-linux-x86_64 + - name: Prepare toolchain + run: bash "$SCRIPTS/build-native.sh" toolchain - - name: Build tantivy4java (setup.sh) - if: steps.t4j-cache.outputs.cache-hit != 'true' - run: ./scripts/setup.sh + - name: Build tantivy4java + run: bash "$SCRIPTS/build-native.sh" build - - name: Upload tantivy4java jars - uses: actions/upload-artifact@v7 + - name: Upload native build + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 with: - name: tantivy4java-linux-x86_64 - path: ~/.m2/repository/io/indextables/tantivy4java/${{ steps.t4j.outputs.version }}/ + name: native-linux-x86_64 + path: out/native if-no-files-found: error + overwrite: true + retention-days: 7 - publish: - name: Publish ${{ matrix.spark.profile }} - needs: native - runs-on: ubuntu-latest - environment: release + # --------------------------------------------------------------------------- + # Build, unsigned, everything that is published for one Spark profile. + # The profile list must match PROFILES in .github/scripts/release/lib.sh. + # --------------------------------------------------------------------------- + build: + name: Build ${{ matrix.profile }} + needs: [plan, native] + runs-on: ubuntu-24.04 + timeout-minutes: 30 + permissions: + contents: read strategy: fail-fast: false matrix: - spark: - - { profile: spark-3.5, version: 3.5.8 } - - { profile: spark-4.0, version: 4.0.3 } - - { profile: spark-4.1, version: 4.1.2 } + profile: [spark-3.5, spark-4.0, spark-4.1] steps: - - name: Checkout - uses: actions/checkout@v7 + - name: Checkout release scripts (workflow commit) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.workflow_sha }} + path: release + sparse-checkout: .github/scripts/release + persist-credentials: false + + - name: Checkout source (tagged commit) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: - ref: ${{ env.RELEASE_TAG }} + ref: ${{ needs.plan.outputs.commit }} + path: src + persist-credentials: false - - name: Set up Java 17 (with Central credentials + GPG key) - uses: actions/setup-java@v6 + - name: Set up Java 17 + uses: actions/setup-java@de7274f081f381c8f8158605e0321c36c376e2e6 # v6.0.1 with: distribution: 'temurin' java-version: '17' - server-id: central - server-username: CENTRAL_USERNAME - server-password: CENTRAL_PASSWORD - gpg-private-key: ${{ secrets.GPG_PRIVATE_KEY }} - gpg-passphrase: MAVEN_GPG_PASSPHRASE - - name: Download tantivy4java jars - uses: actions/download-artifact@v8 + - name: Download native build + uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + name: native-linux-x86_64 + path: in/native + + - name: Build artifacts + env: + SRC: src + PROFILE: ${{ matrix.profile }} + BASE_VERSION: ${{ needs.plan.outputs.base }} + NATIVE_DIR: in/native + OUT: out/artifacts + run: bash "$SCRIPTS/build-artifacts.sh" + + - name: Upload artifacts + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: artifacts-${{ matrix.profile }} + path: out/artifacts + if-no-files-found: error + compression-level: 0 + overwrite: true + retention-days: 7 + + # --------------------------------------------------------------------------- + # Check what the build jobs produced and lay out exactly what will be signed + # and uploaded. Runs the release scripts only; no project code. + # --------------------------------------------------------------------------- + assemble: + name: Assemble + needs: [plan, native, build] + runs-on: ubuntu-24.04 + timeout-minutes: 20 + permissions: + contents: read + outputs: + manifest-sha256: ${{ steps.assemble.outputs.manifest_sha256 }} + versions: ${{ steps.assemble.outputs.versions }} + steps: + - name: Checkout release scripts (workflow commit) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.workflow_sha }} + path: release + sparse-checkout: .github/scripts/release + persist-credentials: false + + - name: Download native build + uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + name: native-linux-x86_64 + path: in/native + + # One directory per artifact: in/artifacts-/ + - name: Download artifacts + uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + pattern: artifacts-* + path: in + + - name: Assemble + id: assemble + env: + IN: in + OUT: staging + TAG: ${{ needs.plan.outputs.tag }} + COMMIT: ${{ needs.plan.outputs.commit }} + BASE_VERSION: ${{ needs.plan.outputs.base }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: bash "$SCRIPTS/assemble.sh" + + - name: Upload staging directory + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: release-staging + path: staging + if-no-files-found: error + compression-level: 0 + overwrite: true + retention-days: 30 + + # --------------------------------------------------------------------------- + # Everything the publish job does before its first upload, with a throwaway + # key: fetch and verify the staging directory, sign, bundle, check. Runs in + # every run, so a publish run is rehearsed before it asks for approval. + # --------------------------------------------------------------------------- + rehearse: + name: Rehearse signing (throwaway key) + needs: [plan, assemble] + runs-on: ubuntu-24.04 + timeout-minutes: 15 + permissions: + contents: read + env: + BASE_VERSION: ${{ needs.plan.outputs.base }} + # Protects the throwaway key only; it is not a secret. + GPG_PASSPHRASE: rehearsal + steps: + - name: Checkout release scripts (workflow commit) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.workflow_sha }} + path: release + sparse-checkout: .github/scripts/release + persist-credentials: false + + - name: Keep the keyring outside the workspace + run: echo "GNUPGHOME=$RUNNER_TEMP/release-gnupg" >> "$GITHUB_ENV" + + - name: Download staging directory + uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 with: - name: tantivy4java-linux-x86_64 - path: ~/.m2/repository/io/indextables/tantivy4java/${{ needs.native.outputs.t4j-version }}/ + name: release-staging + path: staging + + - name: Verify staging directory + env: + MANIFEST_SHA256: ${{ needs.assemble.outputs.manifest-sha256 }} + run: bash "$SCRIPTS/verify-staging.sh" staging - - name: Stamp release version + - name: Generate a throwaway key + id: key + run: bash "$SCRIPTS/signing-key.sh" generate + + - name: Sign and bundle + env: + SIGNING_KEY: ${{ steps.key.outputs.fingerprint }} + run: bash "$SCRIPTS/sign-bundles.sh" staging bundles + + - name: Check bundles + env: + SIGNING_KEY: ${{ steps.key.outputs.fingerprint }} + run: bash "$SCRIPTS/check-bundles.sh" bundles staging + + - name: Remove the throwaway key + if: always() + run: bash "$SCRIPTS/signing-key.sh" remove + + - name: Summary + env: + MODE: ${{ needs.plan.outputs.mode }} run: | - BASE="${RELEASE_TAG#v}" - VERSION="${BASE}_spark_${{ matrix.spark.version }}" - echo "RELEASE_VERSION=$VERSION" >> "$GITHUB_ENV" - mvn -B versions:set -DnewVersion="$VERSION" -DgenerateBackupPoms=false + { + echo "### Rehearsal passed" + echo + echo "The staging directory was verified, signed with a throwaway key, bundled and checked. Those signatures and bundles are discarded with this job." + if [[ "$MODE" == publish ]]; then + echo + echo "The publish job now waits for approval in the \`release\` environment. Review the Plan and Assemble summaries first." + else + echo + echo "Dry run complete: nothing was uploaded and no release was created." + fi + } >> "$GITHUB_STEP_SUMMARY" - - name: Build shaded jar - run: mvn -B package -P${{ matrix.spark.profile }} -DskipTests + # --------------------------------------------------------------------------- + # The only job with the `release` environment, secrets and a write token. + # Not started in a dry run: both the plan's verdict and the raw input have to + # say "publish". + # + # One publish at a time. A second release waits here; a third replaces the + # second in the queue (GitHub keeps one pending job per group). + # --------------------------------------------------------------------------- + publish: + name: Sign and publish + needs: [plan, assemble, rehearse] + if: ${{ needs.plan.outputs.mode == 'publish' && !(github.event_name == 'workflow_dispatch' && inputs.dry-run) }} + runs-on: ubuntu-24.04 + # Three uploads of about 100 MB, each followed by the Portal's validation. + # In the last release, build, upload and validation together took under + # two minutes per version. The limit covers the scripts' own worst case + # of 15 + 20 minutes per version. + timeout-minutes: 120 + environment: release + concurrency: + group: release-publish + cancel-in-progress: false + permissions: + contents: write + env: + BASE_VERSION: ${{ needs.plan.outputs.base }} + steps: + - name: Checkout release scripts (workflow commit) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.workflow_sha }} + path: release + sparse-checkout: .github/scripts/release + persist-credentials: false - - name: Publish to Maven Central - if: ${{ !(github.event_name == 'workflow_dispatch' && inputs.skip-central) }} - run: mvn -B deploy -P${{ matrix.spark.profile }},release -DskipTests + - name: Keep the keyring and upload state outside the workspace + run: | + echo "GNUPGHOME=$RUNNER_TEMP/release-gnupg" >> "$GITHUB_ENV" + echo "CENTRAL_STATE=$RUNNER_TEMP/central-state" >> "$GITHUB_ENV" + + - name: Download staging directory + uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + name: release-staging + path: staging + + - name: Verify staging directory + env: + MANIFEST_SHA256: ${{ needs.assemble.outputs.manifest-sha256 }} + run: bash "$SCRIPTS/verify-staging.sh" staging + + - name: Import the signing key + id: key + if: needs.plan.outputs.central == 'true' + env: + GPG_PRIVATE_KEY: ${{ secrets.GPG_PRIVATE_KEY }} + run: bash "$SCRIPTS/signing-key.sh" import + + - name: Sign and bundle + if: needs.plan.outputs.central == 'true' + env: + SIGNING_KEY: ${{ steps.key.outputs.fingerprint }} + GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }} + run: bash "$SCRIPTS/sign-bundles.sh" staging bundles + + - name: Check bundles + if: needs.plan.outputs.central == 'true' + env: + SIGNING_KEY: ${{ steps.key.outputs.fingerprint }} + run: bash "$SCRIPTS/check-bundles.sh" bundles staging + + # The key is gone before the first request to Maven Central or GitHub. + - name: Remove the signing key + if: always() + run: bash "$SCRIPTS/signing-key.sh" remove + + - name: Upload to Maven Central + if: needs.plan.outputs.central == 'true' env: CENTRAL_USERNAME: ${{ secrets.CENTRAL_USERNAME }} CENTRAL_PASSWORD: ${{ secrets.CENTRAL_PASSWORD }} - MAVEN_GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }} + run: bash "$SCRIPTS/central-upload.sh" upload bundles staging "$CENTRAL_STATE" - - name: Attach shaded jar to GitHub Release + - name: GitHub Release env: GH_TOKEN: ${{ github.token }} - run: | - # First finishing leg creates the release; the rest just upload. - gh release create "$RELEASE_TAG" --verify-tag --generate-notes 2>/dev/null || true - gh release upload "$RELEASE_TAG" --clobber \ - target/indextables_spark-${RELEASE_VERSION}-linux-x86_64-shaded.jar + GH_REPO: ${{ github.repository }} + TAG: ${{ needs.plan.outputs.tag }} + COMMIT: ${{ needs.plan.outputs.commit }} + PRERELEASE: ${{ needs.plan.outputs.prerelease }} + run: bash "$SCRIPTS/github-release.sh" staging + + # If the job stops after bundles were uploaded (the GitHub Release step + # failed, or the run was cancelled), do not leave validated deployments + # waiting in the Portal: a re-run uploads them again. + - name: Drop Maven Central deployments after a failure + if: ${{ (failure() || cancelled()) && needs.plan.outputs.central == 'true' }} + env: + CENTRAL_USERNAME: ${{ secrets.CENTRAL_USERNAME }} + CENTRAL_PASSWORD: ${{ secrets.CENTRAL_PASSWORD }} + run: bash "$SCRIPTS/central-upload.sh" drop "$CENTRAL_STATE" From 498e0233b8e3d652651dfc4a392646fb8814dc43 Mon Sep 17 00:00:00 2001 From: Brenton Strickler Date: Thu, 8 Oct 2026 18:11:42 -0400 Subject: [PATCH 2/3] ci(release): close the gaps found in review of the release workflow Changes that follow from an independent review of the split release workflow. None of them moves the boundary between the build jobs and the publish job. Deployments left in the Portal. The Portal accepts a drop only for a deployment that is VALIDATED or FAILED. A timeout or a cancellation during validation used to attempt the drop too early, and then forgot the deployment. central-upload.sh now polls a deployment until it settles (bounded) before dropping it, keeps what it could not drop so that the job's last step tries again, and, if something is still left, fails that step and lists each deployment by id and name in the job summary. An upload that was cut off before the Portal answered is reported by name. Deployments are named after the version, the run id and the attempt, and the summary lists their ids as they arrive, with the instruction to publish only those. Jars on an existing GitHub Release. An attached jar is left alone when it is identical to the staged one and is no longer replaced when it differs, unless the new replace-release-assets input is ticked. With skip-central nothing else stopped a rebuild from replacing published jars. Artifact names. The versions still follow spark.version in pom.xml at the tag, but plan now works them out before anything is built, shows them next to the Spark versions last published, and stops a publish run whose Spark versions differ from the last release (or cannot be compared) until the releaser has entered them in the new expected-spark-versions input. The build, assemble, rehearse and publish jobs are held to the versions the plan announced. No redirection in a real run. The Portal, repo1 and repository addresses and the local Maven repository are constants; the variables that used to override them are gone, and the tests use a stub curl and git URL rewriting instead. Three timing settings remain for the tests and stop a script when set without the test flag, which release.yml never sets. print-pin checks where the fork commits it prints can be reached from: it says so for the default branch, warns for another branch or tag, and refuses a commit that is on no branch or tag of the fork. Documentation. The README states the order in which to adopt the workflow (secrets into the environment, environment narrowed to main, dry run and a release candidate taken through Publish, then the repository-level secrets deleted) and why: the existing rc tags carry the previous release.yml with a workflow_dispatch trigger, and repository-level secrets are readable from any workflow on any ref. The rehearsal is rewritten as a real release candidate on a new tag, with what to check at each point. The workflow header no longer reads as a statement about the repository as a whole. --- .claude/CLAUDE.md | 4 +- .github/scripts/release/README.md | 285 +++++++++--- .github/scripts/release/assemble.sh | 6 +- .github/scripts/release/build-artifacts.sh | 9 +- .github/scripts/release/build-native.sh | 39 +- .github/scripts/release/central-upload.sh | 219 ++++++--- .github/scripts/release/github-release.sh | 58 ++- .github/scripts/release/lib.sh | 43 ++ .github/scripts/release/native-pins.txt | 7 +- .github/scripts/release/plan.sh | 142 +++++- .github/scripts/release/test.sh | 500 ++++++++++++++++----- .github/scripts/release/verify-staging.sh | 5 +- .github/workflows/release.yml | 72 ++- 13 files changed, 1134 insertions(+), 255 deletions(-) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 5faef9bb..2eefb9c4 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -61,7 +61,9 @@ property-only Maven profiles. Released artifacts encode the Spark version in the 4.13.1 runtime ("Could not deserialize ATN with version 3"). - **Release**: run `.github/workflows/release.yml` from `main` with an existing `v*` tag (Actions → Release → Run workflow). `dry-run` is ticked by default and publishes nothing; - untick it to publish. Pushing a tag does not start a release. The run builds tantivy4java + untick it to publish. Pushing a tag does not start a release. The artifact names carry + `spark.version` from `pom.xml`; when that differs from the last release, a publish run asks + for the `expected-spark-versions` input (the dry run prints the value). The run builds tantivy4java (linux-x86_64) from source at the commits pinned in `.github/scripts/release/native-pins.txt` (add a line there when bumping tantivy4java in `pom.xml`), builds each Spark leg unsigned, and only then signs and uploads from a separate diff --git a/.github/scripts/release/README.md b/.github/scripts/release/README.md index f89d4d17..0c563924 100644 --- a/.github/scripts/release/README.md +++ b/.github/scripts/release/README.md @@ -1,7 +1,7 @@ # Release scripts -The scripts behind `.github/workflows/release.yml`, and the runbook for a -release. +The scripts behind `.github/workflows/release.yml`, how to adopt that +workflow, and the runbook for a release. ## What a release publishes @@ -13,10 +13,32 @@ profile, `_spark_`: | Maven Central | `io.indextables:indextables_spark:_spark_`: the pom, the jar, `-sources`, `-javadoc`, `-linux-x86_64-shaded` and `-jar-with-dependencies`, each with a signature and md5, sha1, sha256 and sha512 checksums | | GitHub Release for the tag | the `-linux-x86_64-shaded` jar of each version | -`` is the `spark.version` the profile sets in `pom.xml` at the -tagged commit, so the suffix names the Spark version the jar was compiled -against. The file set is the one on Maven Central for `0.6.0-rc2`; the -allow-list is `ARTIFACT_SUFFIXES` in `lib.sh`. +The file set is the one on Maven Central for `0.6.0-rc2`; the allow-list is +`ARTIFACT_SUFFIXES` in `lib.sh`. + +### The Spark version in the name + +`` is the `spark.version` that the profile sets in `pom.xml` at the +tagged commit, so the name says which Spark the jar was compiled against. It +also means that a dependency update which moves Spark from 3.5.9 to 3.5.10 +changes the artifact names of the next release, and anyone who depends on +`…_spark_3.5.9` has to change their build. That must not happen unnoticed: + +* `plan` reads the versions from `pom.xml` at the tag and lists them in the + Plan summary, next to the Spark version last published to Maven Central for + each Spark line. +* A publish run whose Spark versions differ from the last published ones, or + that cannot reach Maven Central to compare, stops in `plan`, before + anything is built. To go ahead, dispatch again with + `expected-spark-versions` set to the Spark versions you expect, for example + `3.5.10 4.0.4 4.1.3` (any order, spaces or commas). The error message and + the Plan summary of a dry run print the exact value. +* Whenever `expected-spark-versions` is given, in a dry run too, it must + match `pom.xml`, or the run stops. +* When nothing changed since the last release the input can stay empty. + +The build jobs check that Maven arrives at the versions `plan` announced, and +`assemble`, `rehearse` and `publish` accept exactly those versions. ## Jobs and trust @@ -28,7 +50,7 @@ plan ─► native ─► build (x3) ─► assemble ─► rehearse ─► publ | Job | Runs | Has | |---|---|---| -| `plan` | `plan.sh` | read-only token | +| `plan` | `plan.sh` (reads `pom.xml` at the tag as data; fetches the public Maven Central metadata) | read-only token | | `native` | `build-native.sh`: tantivy4java's Maven and cargo build, from pinned commits, no cache | read-only token | | `build` | `build-artifacts.sh`: this project's Maven build at the tagged commit, with the release profile and signing switched off | read-only token | | `assemble` | `assemble.sh`: checks the build outputs, lays out what will be uploaded, writes a manifest of digests | read-only token | @@ -52,9 +74,27 @@ file names. A build cannot add another artifact, another version or another put different bytes inside the six expected files: that is what the tagged commit's review, the pins and the digests in the run summary are for. -Each secret is passed to the one step that needs it. The key is imported into -a keyring under the runner's temporary directory, and removed before the -first request to Maven Central or GitHub. +Limits of that description, so that it is not read as more than it is: + +* It is about `release.yml` on `main`. It says nothing about other workflow + files or other refs; see "Adopting this workflow". +* Each secret is passed to the one step that needs it, and the keyring is + deleted before the first request to Maven Central or GitHub. The secret + values themselves stay in the runner's memory for the whole job, as with + any GitHub Actions job; deleting the keyring removes the key from disk and + stops the agent, it does not make the job unable to obtain the key again. +* The scripts reach fixed addresses only (the Portal API, repo1, the three + GitHub repositories). No variable redirects them; three timing settings + exist for `test.sh` and stop the script if they are set without the test + flag, which `release.yml` never sets. +* A re-run ("Re-run failed jobs" or "Re-run all jobs") uses the workflow file + and the scripts of the commit the run was started on, not the current + `main`. A fix to a script only takes effect in a newly dispatched run. +* rustc and cargo, Maven, the JDK patch level and the runner image are + whatever the runner provides on the day. They are recorded in the summaries + and the release notes, not pinned. Pinned: the actions, protoc, the + tantivy4java commit and through its `Cargo.lock` every Rust dependency, and + the Maven plugins that the scripts call by coordinate. ## Native inputs @@ -66,8 +106,7 @@ fetches that tantivy4java commit by id, refuses to continue if the tag (cargo does that when it cannot use the locked versions), if a dependency points to a path outside the checkout, or if a cargo config overrides dependency sources. protoc is the same pinned release and digest as in -`scripts/setup.sh`. Rust comes from the runner image and is recorded, not -pinned. +`scripts/setup.sh`. The resolved commits and tool versions are written to the `native` job summary, the `assemble` job summary and the release notes. @@ -79,28 +118,68 @@ pull request: bash .github/scripts/release/build-native.sh print-pin ``` +`print-pin` prints the line and, on standard error, where each commit can be +reached from. A commit id by itself proves little: GitHub serves a commit +that exists only in somebody's fork through the parent repository's URL. So +`print-pin` clones the quickwit and tantivy forks and + +* says so when a fork commit is on the fork's default branch (the normal + case; both commits pinned for 0.34.4 are the tips of `main`); +* warns, naming the branches or tags, when it is on another branch or tag + only: pin it only if building from that branch is intended; +* refuses to print a pin when it is on no branch or tag of the fork. + +Review that output before committing the line. The release build itself does +not repeat the check; it builds what the reviewed pin file says. + The `Release Scripts` workflow fails on a pull request that changes the -version without adding a pin. +tantivy4java version without adding a pin. `scripts/setup.sh` is not used by the release. It still clones the quickwit fork at its default branch for developer and CI builds; current tantivy4java versions take both forks from `Cargo.lock` and ignore that clone. -## Settings this depends on - -These are repository settings, not files, and the workflow cannot enforce -them on itself: - -1. Environment `release`: required reviewer set; deployment branches and tags - limited to **`main` only**. With `v*` tags also admitted, a workflow file - on any `v*` tag can ask for this environment (see below). -2. `GPG_PRIVATE_KEY`, `GPG_PASSPHRASE`, `CENTRAL_USERNAME` and - `CENTRAL_PASSWORD` stored as secrets **of that environment**, and the - repository-level secrets of the same names deleted. While a - repository-level copy exists, every workflow in the repository can read it - and none of the separation above applies. -3. `main` protected, so that the workflow file and these scripts change only - through reviewed pull requests. +## Adopting this workflow + +Merging the workflow does not by itself protect the key. Two facts decide +that, and both are repository settings: + +* While `GPG_PRIVATE_KEY`, `GPG_PASSPHRASE`, `CENTRAL_USERNAME` and + `CENTRAL_PASSWORD` exist as repository-level secrets, any workflow on any + ref can read them. +* The tags `v0.6.0-rc1` and `v0.6.0-rc2` carry the previous `release.yml`, + which has a `workflow_dispatch` trigger. It can be dispatched on the tag + ref, and then runs the old workflow, with the project build beside the + key. Tags cannot be edited away; only the environment rule and the removal + of the repository-level secrets close this. + +Do the steps in this order. Each one is the owner's or the releaser's to do +in the repository settings; nothing in this directory can do them. + +1. **Add the four secrets to the `release` environment** (Settings → + Environments → release → Environment secrets), with the same names. Leave + the repository-level copies in place for now. +2. **Narrow the environment.** Deployment branches and tags: selected, `main` + only (remove the `v*` tag rule). Required reviewer set. "Allow + administrators to bypass configured protection rules": off. From here on, + a run on a tag ref, including the old workflow on the two rc tags, is + refused by the environment, but it could still read the repository-level + secrets from a job that names no environment, which is why step 4 is not + optional. +3. **Dry run, then the release candidate rehearsal** described in the + runbook below. +4. **Delete the repository-level copies of the four secrets as soon as the + rehearsal has passed.** Not after the next real release: until they are + gone, the separation in this workflow protects nothing. + +One thing the rehearsal cannot show: while both copies exist, a secret that +is missing or mistyped in the environment is silently covered by the +repository-level one. After step 4 such a mistake shows up as a failed key +import or a refused upload, before anything is uploaded. Compare the four +names in the environment by eye before deleting. + +`main` must stay protected, so that the workflow file and these scripts +change only through reviewed pull requests. ## Tag push or dispatch @@ -113,7 +192,7 @@ dispatched from `main`, with the tag as input; the tag's commit must be on selects which source is built, by the unprivileged jobs. Left open: whoever can dispatch workflows chooses the tag (bounded by "must be on `main`") and whoever approves the environment is the last check; and the guarantee rests - on settings 1 and 3 above. + on the settings above. * **Tag push** (not enabled): pushing `v*` runs the workflow file and scripts *of the tagged commit*. Anyone who can push a `v*` tag, on any commit, merged or not, then decides what the job holding the secrets executes; the @@ -143,44 +222,138 @@ run: reject it. What to look at: -* **Plan** summary: "Dry run for …", the tag and its commit. +* **Plan** summary: "Dry run for …", the tag and its commit, the versions, + and whether they differ from the last release. * **native** summary: the tantivy4java, quickwit and tantivy commits. -* **Assemble** summary: three versions with the Spark versions you expect; - the manifest digest; the digests of all files. +* **Assemble** summary: the three versions; the manifest digest; the digests + of all files. For `v0.6.0-rc2` the three `.pom` digests should equal the + published ones (`…//indextables_spark-.pom.sha256` on + repo1): the pom is generated the same way as before. * **Rehearse** log: each bundle lists 36 entries and passes. A dry run may also be dispatched from a branch, to try a change to the workflow or these scripts before it is merged. +### Release candidate rehearsal (the first real use) + +A dry run stops before the key, the Central token, the Portal and the Publish +button. The only way to exercise those is a real release, so make the first +one a release candidate on a new tag, for example `v0.6.0-rc3`, and take it +all the way through Publish. + +It has to be a new tag. An existing one cannot stand in: `0.6.0-rc2` is +already on Maven Central, a rebuild is not byte-identical, and the upload +step stops before uploading anything when it sees that. And it has to +include pressing Publish, not "upload, then drop": dropping leaves a public +GitHub pre-release with jars that are on no repository, and it skips the one +step that no test and no dry run reaches. + +Do it after steps 1 and 2 of "Adopting this workflow". At each point, check: + +1. **Before dispatching.** Create the tag on a commit of `main`. Dry-run it + first; note the `expected-spark-versions` value from the Plan summary if + it says the names changed. +2. **Dispatch from `main`**, `dry-run` unticked (and `expected-spark-versions` + if needed). + * An approval prompt for `release` appears, and only now: the dry run a + moment ago did not ask. A run dispatched from any other branch or from a + tag with `dry-run` unticked fails in `plan` and never asks. + * That is this workflow's own guard. The rule that also stops other + workflow files (the old one on the rc tags) is the environment's; confirm + it by reading it, not by dispatching the old workflow: + `gh api repos/indextables/indextables_spark/environments/release/deployment-branch-policies` + must list `main` and nothing else. + * Read the Plan and Assemble summaries, then approve. +3. **While `publish` runs.** + * The "Import the signing key" step logs `output: fingerprint=…`. It must + be the release key. The signatures on Maven Central for `0.6.0-rc2` name + `A9CD03EDB082ECB61DB66028EDA69278FFE47B19` as their signing key; the log + prints the primary key's fingerprint, which is the same value unless the + key signs with a subkey (`gpg --show-keys --with-subkey-fingerprints` on + the public key shows both). + * The summary lists each upload as it happens, with its deployment id. +4. **After `publish`, before pressing Publish.** Open + . + * Exactly three deployments from this run, named + `indextables_spark--run-`, state VALIDATED. + Their ids are the three in the job summary. A deployment for these + versions with another id or name is from another run: drop it. + * Each holds six files (pom, jar, sources, javadoc, linux-x86_64-shaded, + jar-with-dependencies), each with a signature. + * The GitHub Release for the tag has three jars, and its notes end with a + "Build inputs" block naming the commits. +5. **Press Publish** on each of the three. This cannot be undone. +6. **After Publish** (Maven Central takes a while to show a new version). + Resolve one coordinate from Maven Central and compare it with the run: + + ``` + v=0.6.0-rc3_spark_3.5.9 # one of the three versions + curl -fsSL "https://repo1.maven.org/maven2/io/indextables/indextables_spark/$v/indextables_spark-$v-linux-x86_64-shaded.jar" | sha256sum + ``` + + The digest must be the one in the "Build inputs" block of the release + notes (and in the Assemble summary) for that version, and the jar attached + to the GitHub Release must have it too. +7. **Delete the repository-level secrets** (step 4 of "Adopting this + workflow"). + ### Release -1. Dispatch from `main` with `dry-run` unticked. +1. Dispatch from `main` with `dry-run` unticked; add + `expected-spark-versions` if the Plan summary of the dry run asked for it. 2. Wait for `rehearse` to pass, read the Plan and Assemble summaries, then approve the `release` environment. Up to here nothing has left the run; rejecting the approval ends it. 3. `publish` signs, uploads one bundle per version to the Central Portal as `USER_MANAGED`, waits until each is `VALIDATED`, then creates or updates the GitHub Release. -4. Open . There is one - deployment per version, named `indextables_spark-`. Press - **Publish** on each. This is the step that cannot be undone. +4. Open . Press + **Publish** on the deployments whose ids are in the job summary, on all of + them and on no other. This is the step that cannot be undone. + +`skip-central` makes a publish run leave Maven Central alone and only attach +the jars to the GitHub Release. + +### Jars already on the GitHub Release + +A jar that is already attached to the release is left alone when it is +identical to the one this run built (a re-run of the same run). When it +differs, the GitHub Release step stops and changes nothing: a jar on a +published release may already have been downloaded, and a rebuild is never +byte-identical. This matters most with `skip-central`, where nothing on the +Maven Central side stops a second run for an already released tag. If +replacing the jars is really what you want, dispatch again with +`replace-release-assets` ticked. ### Aborting | When | How | What is left behind | |---|---|---| | Before approval | Reject the approval, or cancel the run | Nothing | -| While `publish` runs | Cancel the run | The job drops the deployments it uploaded; check the Portal. The GitHub Release may exist with some jars | +| While `publish` runs | Cancel the run | The last step drops the deployments the job uploaded and says in the summary if it could not; check the Portal. The GitHub Release may exist with some jars | | After `publish`, before pressing Publish | **Drop** every deployment in the Portal; delete or edit the GitHub Release by hand | Nothing on Maven Central | | After pressing Publish | Not possible. Publish the remaining versions, and release a new version if this one is wrong | | ### When `publish` fails -`central-upload.sh` handles one version at a time and drops what it uploaded -when anything fails, so a failed job normally leaves nothing in the Portal; -if a drop is refused the log says which deployment to drop by hand. Then use -**Re-run failed jobs**: the re-run signs and uploads the same staged files -(same digests), not a rebuild. +`central-upload.sh` handles one version at a time, and when anything fails +(or the run is cancelled) the job drops what it uploaded. The Portal accepts +a drop only once a deployment is VALIDATED or FAILED, so a deployment that is +still being validated is polled first, for up to three minutes, in the +failing step and again in the job's last step. + +If something is still left, the last step fails and the job summary has a +section **"Action needed: deployments left in the Central Portal"**. It +lists each deployment by id and name with what to do: drop it by hand and do +not publish it; or, for an upload that was cut off before the Portal +answered, look for a deployment of that name and drop it if it exists. The +run id and attempt in the name tell this job's deployments from any other. + +Then use **Re-run failed jobs**: the re-run signs and uploads the same staged +files (same digests), not a rebuild, under a new attempt number. It uses the +scripts of the original run; if the failure was a bug in a script, fix it on +`main` and dispatch a new run instead (a new build, so only possible while +none of the versions has been published). If some versions were already published by hand and others were not, re-run the failed job as well: a version that is already on Maven Central with @@ -189,20 +362,26 @@ identical files is skipped, and one with different files stops the run. ## What the tests cover, and what they cannot `bash .github/scripts/release/test.sh` runs offline: `mvn`, `curl` and `gh` -are stubs. It covers the decision logic of every script (including every -refusal), the layout of the bundles, the upload state machine (rejection, -timeouts, drops, re-runs) and the shape of `release.yml` (which job has -secrets, the dry-run guards, pins). On a pull request it runs on the runner -image with the real `gpg`, generating, exporting, importing and signing with -a throwaway key. +are stubs, and git is pointed at fixture repositories. It covers the +decision logic of every script (including every refusal), the layout of the +bundles, the upload state machine (rejection, timeouts and cancellation +during validation, refused drops, stale deployments, re-runs) and the shape +of `release.yml` (which job has secrets, the dry-run guards, pins, that no +test-only setting is set). On a pull request it runs on the runner image +with the real `gpg`, generating, exporting, importing and signing with a +throwaway key. Not covered by any test, and seen for the first time in a real run: * the Maven and cargo builds themselves (first seen in a dry run); * artifacts passing between jobs (first seen in a dry run); * importing the real key and using the real passphrase; -* the Central Portal accepting the token, the bundle and the signatures; -* `gh` creating or updating the release with the job's token. - -The last three happen in `publish` only. If one of them fails, nothing has -been published: the upload is `USER_MANAGED`, and a failed job drops it. +* the Central Portal accepting the token, the bundle and the signatures, and + its answers to status and drop requests (the stub follows the Portal's API + documentation); +* `gh` creating or updating the release with the job's token; +* the Publish button. + +The last four happen in or after `publish` only, which is what the release +candidate rehearsal is for. If one of the first three of them fails, nothing +has been published: the upload is `USER_MANAGED`, and a failed job drops it. diff --git a/.github/scripts/release/assemble.sh b/.github/scripts/release/assemble.sh index 1de11a95..65f9e13b 100644 --- a/.github/scripts/release/assemble.sh +++ b/.github/scripts/release/assemble.sh @@ -17,7 +17,7 @@ # version: the six files and their # md5/sha1/sha256/sha512 (no signatures) # MANIFEST.sha256 SHA-256 of every file above -# TAG, COMMIT, BASE_VERSION from plan.sh +# TAG, COMMIT, BASE_VERSION, EXPECTED_VERSIONS from plan.sh # RUN_URL link to this workflow run (optional, for the notes) # # Step outputs: manifest_sha256 (digest of MANIFEST.sha256), versions. @@ -26,7 +26,7 @@ here="$(cd "$(dirname "$0")" && pwd)" # shellcheck source=lib.sh . "$here/lib.sh" -: "${IN:?}" "${OUT:?}" "${TAG:?}" "${COMMIT:?}" "${BASE_VERSION:?}" +: "${IN:?}" "${OUT:?}" "${TAG:?}" "${COMMIT:?}" "${BASE_VERSION:?}" "${EXPECTED_VERSIONS:?}" need unzip python3 [[ "$TAG" =~ $TAG_RE ]] || die "'$TAG' is not a release tag" [ "v$BASE_VERSION" = "$TAG" ] || die "base version $BASE_VERSION does not belong to tag $TAG" @@ -101,6 +101,8 @@ for profile in $PROFILES; do [ "$(kv "$info" profile)" = "$profile" ] || die "artifacts-$profile was not built with profile $profile" version="$(kv "$info" version)" check_version "$BASE_VERSION" "$profile" "$version" + [ "$version" = "$(profile_version "$EXPECTED_VERSIONS" "$profile")" ] \ + || die "artifacts-$profile holds $version, which is not the version the plan announced for $profile" [ "${BASE_VERSION}_spark_$(kv "$info" spark_version)" = "$version" ] \ || die "artifacts-$profile: version $version does not match its recorded Spark version" diff --git a/.github/scripts/release/build-artifacts.sh b/.github/scripts/release/build-artifacts.sh index 27e0a6c2..5d4d6854 100644 --- a/.github/scripts/release/build-artifacts.sh +++ b/.github/scripts/release/build-artifacts.sh @@ -7,6 +7,7 @@ # SRC checkout of the commit being released # PROFILE spark-3.5 | spark-4.0 | spark-4.1 # BASE_VERSION the tag without its "v" (0.6.0) +# EXPECTED_VERSIONS the versions plan.sh announced, one per profile # NATIVE_DIR the native job's artifact (build-native.sh output) # OUT output directory; becomes the job's artifact: # files/ the six files of this version, under the @@ -17,13 +18,14 @@ # The published version is _spark_, where # spark.version is what the profile sets in pom.xml at the released commit, # so the suffix always names the Spark version the jar was compiled against. +# It must be the version plan.sh worked out from the same pom.xml and showed +# in the Plan summary; if Maven disagrees, the build stops. set -euo pipefail here="$(cd "$(dirname "$0")" && pwd)" # shellcheck source=lib.sh . "$here/lib.sh" -: "${SRC:?}" "${PROFILE:?}" "${BASE_VERSION:?}" "${NATIVE_DIR:?}" "${OUT:?}" -M2_REPO="${M2_REPO:-$HOME/.m2/repository}" +: "${SRC:?}" "${PROFILE:?}" "${BASE_VERSION:?}" "${EXPECTED_VERSIONS:?}" "${NATIVE_DIR:?}" "${OUT:?}" # Pinned here because they are invoked by coordinate, not through pom.xml. VERSIONS_PLUGIN=org.codehaus.mojo:versions-maven-plugin:2.22.0 @@ -56,6 +58,9 @@ spark_version="$(cd "$SRC" && "${mvn_cmd[@]}" -q "-P$PROFILE" "$HELP_PLUGIN:eval -Dexpression=spark.version -DforceStdout | tail -n 1 | tr -d '[:space:]')" version="${BASE_VERSION}_spark_${spark_version}" check_version "$BASE_VERSION" "$PROFILE" "$version" +announced="$(profile_version "$EXPECTED_VERSIONS" "$PROFILE")" +[ "$version" = "$announced" ] \ + || die "Maven derives $version for $PROFILE, but the plan announced $announced" echo "Building $ARTIFACT_ID $version (profile $PROFILE, Spark $spark_version)" (cd "$SRC" && "${mvn_cmd[@]}" "$VERSIONS_PLUGIN:set" "-DnewVersion=$version" -DgenerateBackupPoms=false) diff --git a/.github/scripts/release/build-native.sh b/.github/scripts/release/build-native.sh index 3fedee82..292974f8 100644 --- a/.github/scripts/release/build-native.sh +++ b/.github/scripts/release/build-native.sh @@ -8,7 +8,9 @@ # build-native.sh toolchain install the pinned protoc, check and record # the rest of the toolchain # build-native.sh build build, collect the jars, record digests -# build-native.sh print-pin print a native-pins.txt line for a version +# build-native.sh print-pin print a native-pins.txt line for a version, +# after checking where the fork commits +# it names can be reached from # # Environment (resolve, toolchain, build): # POM pom.xml of the commit being released @@ -25,11 +27,6 @@ here="$(cd "$(dirname "$0")" && pwd)" # shellcheck source=lib.sh . "$here/lib.sh" -T4J_REPO="${T4J_REPO:-https://github.com/indextables/tantivy4java.git}" -QUICKWIT_URL=https://github.com/indextables/quickwit -TANTIVY_URL=https://github.com/indextables/tantivy -M2_REPO="${M2_REPO:-$HOME/.m2/repository}" - # Same release and digest as scripts/setup.sh (test.sh checks that they agree). PROTOC_VERSION=25.5 PROTOC_SHA256=e1ed237a17b2e851cf9662cb5ad02b46e70ff8e060e05984725bc4b4228c6b28 @@ -38,8 +35,8 @@ PROTOC_URL="https://github.com/protocolbuffers/protobuf/releases/download/v${PRO # tag_commit : the commit refs/tags/v points to right now. tag_commit() { local refs commit - refs="$(git ls-remote "$T4J_REPO" "refs/tags/v$1" "refs/tags/v$1^{}")" \ - || die "could not list tags of $T4J_REPO" + refs="$(git ls-remote "$TANTIVY4JAVA_URL" "refs/tags/v$1" "refs/tags/v$1^{}")" \ + || die "could not list tags of $TANTIVY4JAVA_URL" # An annotated tag has a second, peeled line; that is the commit. commit="$(printf '%s\n' "$refs" | awk '$2 ~ /\^\{\}$/ { print $1 }' | head -n 1)" [ -n "$commit" ] || commit="$(printf '%s\n' "$refs" | awk 'NF == 2 { print $1 }' | head -n 1)" @@ -51,7 +48,7 @@ tag_commit() { fetch_commit() { rm -rf "$2" git init -q "$2" - git -C "$2" remote add origin "$T4J_REPO" + git -C "$2" remote add origin "$TANTIVY4JAVA_URL" git -C "$2" fetch -q --depth 1 origin "$1" || die "could not fetch tantivy4java commit $1" git -C "$2" -c advice.detachedHead=false checkout -q --detach FETCH_HEAD [ "$(git -C "$2" rev-parse HEAD)" = "$1" ] || die "fetched commit is not $1" @@ -101,6 +98,27 @@ check_sources() { done } +# fork_reachability : say (on standard error) where a +# fork commit can be reached from. A commit id alone proves little: GitHub +# serves a commit that exists only in someone's fork of a repository through +# the repository's own URL, so a pin must be on a branch or tag of the fork +# itself, and normally on its default branch. +fork_reachability() { + local name="$1" url="$2" commit="$3" dir default refs + [ "$commit" != "-" ] || return 0 + dir="$tmp/$name.git" + git clone -q --bare --filter=tree:0 "$url" "$dir" 2> /dev/null || die "could not clone $url to check commit $commit" + default="$(git -C "$dir" symbolic-ref --short HEAD)" + if git -C "$dir" merge-base --is-ancestor "$commit" "refs/heads/$default" 2> /dev/null; then + echo "$name $commit: reachable from the default branch ($default) of $url" >&2 + return 0 + fi + refs="$(git -C "$dir" for-each-ref --contains "$commit" --format='%(refname:short)' refs/heads refs/tags 2> /dev/null | tr '\n' ' ' || true)" + [ -n "$refs" ] \ + || die "$name commit $commit is not reachable from any branch or tag of $url. It may exist only in a fork of that repository. Do not pin it." + warn "$name $commit is NOT reachable from the default branch ($default) of $url; it is on: $refs. Pin it only if building from that branch is intended." +} + cmd="${1:-}" case "$cmd" in print-pin) @@ -115,6 +133,9 @@ case "$cmd" in sources="$(lock_git_sources "$tmp/src/native/Cargo.lock")" quickwit="$(fork_commit "$sources" "$QUICKWIT_URL")" tantivy="$(fork_commit "$sources" "$TANTIVY_URL")" + echo "tantivy4java $commit: what tag v$version of $TANTIVY4JAVA_URL points to" >&2 + fork_reachability quickwit "$QUICKWIT_URL" "$quickwit" + fork_reachability tantivy "$TANTIVY_URL" "$tantivy" echo "$version $commit $quickwit $tantivy" exit 0 ;; diff --git a/.github/scripts/release/central-upload.sh b/.github/scripts/release/central-upload.sh index 1a1eda16..583458eb 100644 --- a/.github/scripts/release/central-upload.sh +++ b/.github/scripts/release/central-upload.sh @@ -7,9 +7,8 @@ # # Environment: # CENTRAL_USERNAME, CENTRAL_PASSWORD the Portal user token -# CENTRAL_WAIT_SECONDS how long to wait for one validation -# (default 1200) -# CENTRAL_POLL_SECONDS pause between status checks (default 10) +# GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT set by the runner; part of each +# deployment's name # # Publishing type is USER_MANAGED, as it has been so far: an upload that # passes validation waits in the Portal as VALIDATED until a person presses @@ -18,14 +17,25 @@ # abandoned. # # `upload` handles one version at a time: upload, then wait for VALIDATED, -# then the next. If anything goes wrong, it drops every deployment it has -# created in this run and fails, so a failed job leaves nothing waiting in the -# Portal (it says so explicitly if a drop is refused). A version that is -# already on Maven Central is skipped when its files are identical to the -# staged ones (a re-run after a partial publish) and is an error otherwise. +# then the next. Each deployment is named +# indextables_spark--run-, and its id is written to +# the log and the job summary as soon as the Portal returns it, so the +# deployments of this job can be told apart from any other in the Portal. A +# version that is already on Maven Central is skipped when its files are +# identical to the staged ones (a re-run after a partial publish) and is an +# error otherwise. # -# `drop` drops the deployments recorded in the state directory. The workflow -# calls it when a later step of the publish job fails. +# When anything goes wrong, the deployments this job created are dropped. The +# Portal accepts a drop only for a deployment that is VALIDATED or FAILED, so +# a deployment that is still being validated is first polled until it settles +# (for a bounded time). What cannot be dropped stays recorded in the state +# directory. +# +# `drop` does the same for whatever is still recorded there. The workflow +# runs it as the last step whenever the job failed or was cancelled, also +# when the upload step itself was killed in the middle. If a deployment still +# cannot be dropped, or an upload was cut off before the Portal answered, the +# step fails and the job summary says exactly what to drop by hand. # # API: https://central.sonatype.org/publish/publish-portal-api/ set -euo pipefail @@ -33,13 +43,20 @@ here="$(cd "$(dirname "$0")" && pwd)" # shellcheck source=lib.sh . "$here/lib.sh" -API="${CENTRAL_API_URL:-https://central.sonatype.com/api/v1/publisher}" -REPO="${CENTRAL_REPO_URL:-https://repo1.maven.org/maven2}" -PORTAL="https://central.sonatype.com/publishing/deployments" -POLL="${CENTRAL_POLL_SECONDS:-10}" -WAIT="${CENTRAL_WAIT_SECONDS:-1200}" +API="$CENTRAL_API" +POLL="$(test_setting CENTRAL_POLL_SECONDS 10)" +WAIT="$(test_setting CENTRAL_WAIT_SECONDS 1200)" +# How long a drop waits for a deployment that is still being validated. Kept +# well under five minutes: that is what a cancelled job gets to finish its +# remaining steps. +DROP_WAIT="$(test_setting CENTRAL_DROP_WAIT_SECONDS 180)" UUID_RE='^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' +run_id="${GITHUB_RUN_ID:-0}" +run_attempt="${GITHUB_RUN_ATTEMPT:-0}" +[[ "$run_id" =~ ^[0-9]+$ ]] && [[ "$run_attempt" =~ ^[0-9]+$ ]] || die "GITHUB_RUN_ID and GITHUB_RUN_ATTEMPT must be numbers" +RUN_LABEL="run${run_id}-${run_attempt}" + need curl jq base64 auth_file="" @@ -70,30 +87,97 @@ api() { excerpt() { head -c 600 "$1" 2> /dev/null | tr '\n' ' '; } -# drop_all: drop every deployment this run created. Returns non-zero if any -# could not be dropped. +# deployment_state : the deployment's state, or nothing if it could not +# be read. +deployment_state() { + local code + code="$(api POST "$API/status?id=$1" "$state/status.body" --max-time 60)" + if [ "$code" = 200 ]; then jq -r '.deploymentState // empty' "$state/status.body" 2> /dev/null || true; fi +} + +# settle : poll until the deployment is in a state the Portal will not +# move on from by itself (VALIDATED, FAILED, PUBLISHING, PUBLISHED), for at +# most DROP_WAIT seconds. Prints the last state seen, or UNKNOWN. +settle() { + local s deadline + deadline=$(($(date +%s) + DROP_WAIT)) + while :; do + s="$(deployment_state "$1")" + case "$s" in VALIDATED | FAILED | PUBLISHING | PUBLISHED) break ;; esac + [ "$(date +%s)" -lt "$deadline" ] || break + sleep "$POLL" + done + printf '%s\n' "${s:-UNKNOWN}" +} + +# drop_all : drop every deployment still recorded for this +# job. Returns non-zero if anything is left for a person to deal with. With +# "last" (the final step of the job) what is left is written to the summary. drop_all() { - local version id code rc=0 - if [ ! -s "$state/deployments.txt" ]; then - echo "No deployment from this run is waiting in the Portal." - return 0 + local final="$1" version id name s code rc=0 + : > "$state/left.tmp" + if [ -s "$state/deployments.txt" ]; then + while read -r version id name <&5; do + s="$(settle "$id")" + case "$s" in + PUBLISHING | PUBLISHED) + # Not kept for another attempt (it never can be dropped), but + # remembered so that every later call reports it. + echo "$version $id $name $s" >> "$state/published.txt" + continue ;; + esac + code="$(api DELETE "$API/deployment/$id" "$state/drop.body" --max-time 120)" + if [ "$code" = 204 ]; then + echo "Dropped deployment $id ($name)." + echo "$version $id $name" >> "$state/dropped.txt" + else + rc=1 + echo "$version $id $name" >> "$state/keep.tmp" + echo "| \`$version\` | \`$id\` | \`$name\` | $s; the drop was refused (HTTP $code). **Drop it by hand. Do not publish it.** |" >> "$state/left.tmp" + echo "::error::Could not drop deployment $id ($name), state $s: HTTP $code $(excerpt "$state/drop.body")" >&2 + fi + done 5< "$state/deployments.txt" fi - while read -r version id <&5; do - code="$(api DELETE "$API/deployment/$id" "$state/drop.body" --max-time 120)" - if [ "$code" = 204 ]; then - echo "Dropped deployment $id ($version)." - else - rc=1 - echo "::error::Could not drop deployment $id ($version): HTTP $code $(excerpt "$state/drop.body"). Drop it by hand at $PORTAL before releasing again; do not publish it." >&2 - fi - done 5< "$state/deployments.txt" - mv "$state/deployments.txt" "$state/deployments.dropped" + # Keep only what is still in the Portal, so that a later call tries again. + if [ -s "$state/keep.tmp" ]; then mv "$state/keep.tmp" "$state/deployments.txt"; else rm -f "$state/keep.tmp"; : > "$state/deployments.txt"; fi + + if [ -s "$state/published.txt" ]; then + rc=1 + while read -r version id name s <&5; do + echo "| \`$version\` | \`$id\` | \`$name\` | $s: it can no longer be dropped. Someone pressed Publish; this version is or will be on Maven Central although the job failed. |" >> "$state/left.tmp" + echo "::error::Deployment $id ($name) is $s and can no longer be dropped." >&2 + done 5< "$state/published.txt" + fi + + if [ -s "$state/uploading.txt" ]; then + rc=1 + read -r version name < "$state/uploading.txt" + echo "| \`$version\` | unknown | \`$name\` | The upload was cut off before the Portal answered. **If a deployment with this name exists, drop it by hand. Do not publish it.** |" >> "$state/left.tmp" + echo "::error::The upload named $name was cut off before the Portal answered; it may or may not exist there." >&2 + fi + + if [ "$rc" = 0 ]; then + echo "No deployment from this job is left in the Portal." + elif [ "$final" = last ]; then + { + echo "### Action needed: deployments left in the Central Portal" + echo + echo "This job failed and could not clean up everything it uploaded. Open <$CENTRAL_PORTAL> and deal with each line below before releasing again." + echo + echo "| Version | Deployment id | Deployment name | What to do |" + echo "|---|---|---|---|" + cat "$state/left.tmp" + } | summary + else + echo "Some deployments could not be dropped yet (above); the last step of this job tries again." + fi + rm -f "$state/left.tmp" return "$rc" } fail() { echo "::error::$*" >&2 - drop_all || true + drop_all retry-later || true exit 1 } @@ -101,7 +185,7 @@ fail() { # exactly the staged files, 1 when it is not there. Anything else is fatal. already_published() { local version="$1" dir name code remote - dir="$REPO/$GROUP_PATH/$version" + dir="$CENTRAL_REPO/$GROUP_PATH/$version" code="$(curl -sS -o /dev/null -w '%{http_code}' --connect-timeout 30 --max-time 120 \ "$dir/$ARTIFACT_ID-$version.pom")" || code=000 case "$code" in @@ -120,32 +204,32 @@ already_published() { # wait_validated wait_validated() { - local version="$1" id="$2" code deployment_state deadline errors=0 + local version="$1" id="$2" code s deadline errors=0 deadline=$(($(date +%s) + WAIT)) while :; do code="$(api POST "$API/status?id=$id" "$state/status.body" --max-time 60)" - deployment_state="" + s="" if [ "$code" = 200 ]; then - deployment_state="$(jq -r '.deploymentState // empty' "$state/status.body" 2> /dev/null || true)" + s="$(jq -r '.deploymentState // empty' "$state/status.body" 2> /dev/null || true)" errors=0 else errors=$((errors + 1)) [ "$errors" -lt 6 ] || fail "status of deployment $id ($version) could not be read six times in a row (last: HTTP $code $(excerpt "$state/status.body"))" fi - case "$deployment_state" in + case "$s" in VALIDATED) echo "$version: VALIDATED (deployment $id)" return 0 ;; PUBLISHING | PUBLISHED) - warn "$version: deployment $id is already $deployment_state; someone pressed Publish while this job was running" + warn "$version: deployment $id is already $s; someone pressed Publish while this job was running" return 0 ;; FAILED) echo "Validation errors for $version:" >&2 jq '.errors // .' "$state/status.body" >&2 || cat "$state/status.body" >&2 fail "Maven Central rejected the bundle for $version (deployment $id); details above" ;; - *) echo "$version: ${deployment_state:-no status yet} (deployment $id)" ;; + *) echo "$version: ${s:-no status yet} (deployment $id)" ;; esac - [ "$(date +%s)" -lt "$deadline" ] || fail "deployment $id ($version) was not validated within $WAIT seconds (last state: ${deployment_state:-unknown})" + [ "$(date +%s)" -lt "$deadline" ] || fail "deployment $id ($version) was not validated within $WAIT seconds (last state: ${s:-unknown})" sleep "$POLL" done } @@ -154,8 +238,12 @@ cmd="${1:-}" case "$cmd" in drop) state="${2:?usage: central-upload.sh drop }" + if [ ! -d "$state" ]; then + echo "The upload step did not start; there is nothing to drop." + exit 0 + fi auth_setup - drop_all + drop_all last exit $? ;; upload) ;; @@ -167,8 +255,8 @@ staging="${3:?}" state="${4:?}" : "${CENTRAL_USERNAME:?}" "${CENTRAL_PASSWORD:?}" [ -s "$staging/versions.txt" ] || die "$staging lists no versions" +[ ! -e "$state" ] || die "$state already exists: this job has uploaded before; refusing to upload twice" mkdir -p "$state" -[ ! -s "$state/deployments.txt" ] || die "$state already records deployments from this job; refusing to upload twice" : > "$state/deployments.txt" : > "$state/result.txt" @@ -178,42 +266,61 @@ while IFS= read -r version <&4; do [ -f "$bundles/$ARTIFACT_ID-$version-bundle.zip" ] || die "no bundle for $version; nothing was uploaded" if already_published "$version"; then echo "$version is already on Maven Central with identical files; not uploading it again." - echo "$version - ALREADY_PUBLISHED" >> "$state/result.txt" + echo "$version - - ALREADY_PUBLISHED" >> "$state/result.txt" else to_upload="$to_upload $version" fi done 4< "$staging/versions.txt" auth_setup +if [ -n "$to_upload" ]; then + { + echo "### Uploads to the Central Portal" + echo + echo "Written as each upload is accepted, so the ids are here even if the job stops later." + echo + } | summary +fi for version in $to_upload; do zip="$bundles/$ARTIFACT_ID-$version-bundle.zip" - echo "Uploading $(basename "$zip") ..." - code="$(api POST "$API/upload?name=$ARTIFACT_ID-$version&publishingType=USER_MANAGED" "$state/upload.body" \ + name="$ARTIFACT_ID-$version-$RUN_LABEL" + echo "Uploading $(basename "$zip") as $name ..." + # From here until the Portal's answer is recorded, a deployment may exist + # that this job has no id for. The marker tells the drop step to say so. + echo "$version $name" > "$state/uploading.txt" + code="$(api POST "$API/upload?name=$name&publishingType=USER_MANAGED" "$state/upload.body" \ --max-time 900 -F "bundle=@$zip;type=application/octet-stream")" case "$code" in 200 | 201) ;; - 000) fail "the upload of $version did not complete. It may or may not have reached the Portal: look for a deployment named $ARTIFACT_ID-$version at $PORTAL and drop it if it is there." ;; - *) fail "the upload of $version was refused: HTTP $code $(excerpt "$state/upload.body")" ;; + 000) fail "the upload of $version did not complete. It may or may not have reached the Portal: look for a deployment named $name at $CENTRAL_PORTAL and drop it if it is there." ;; + *) + rm -f "$state/uploading.txt" + fail "the upload of $version was refused: HTTP $code $(excerpt "$state/upload.body")" ;; esac id="$(tr -d '[:space:]' < "$state/upload.body")" [[ "$id" =~ $UUID_RE ]] \ - || fail "the upload of $version returned an unexpected response ($(excerpt "$state/upload.body")); look for a deployment named $ARTIFACT_ID-$version at $PORTAL and drop it" - echo "$version $id" >> "$state/deployments.txt" - echo "Uploaded $version as deployment $id" + || fail "the upload of $version returned an unexpected response ($(excerpt "$state/upload.body")); look for a deployment named $name at $CENTRAL_PORTAL and drop it" + echo "$version $id $name" >> "$state/deployments.txt" + rm -f "$state/uploading.txt" + echo "Uploaded $version as deployment $id, named $name" + echo "- Uploaded \`$name\`: deployment \`$id\`" | summary wait_validated "$version" "$id" - echo "$version $id VALIDATED" >> "$state/result.txt" + echo "$version $id $name VALIDATED" >> "$state/result.txt" done { + echo echo "### Maven Central" echo - echo "| Version | Deployment | State |" - echo "|---|---|---|" - while read -r version id deployment_state; do - echo "| \`$version\` | \`$id\` | $deployment_state |" + echo "| Version | Deployment id | Deployment name | State |" + echo "|---|---|---|---|" + while read -r version id name s; do + echo "| \`$version\` | \`$id\` | \`$name\` | $s |" done < "$state/result.txt" echo if [ -n "$to_upload" ]; then - echo "**Not public yet.** Each VALIDATED deployment waits at <$PORTAL> until someone presses **Publish**. Publish all of them, or **Drop** all of them to abandon the release. Publishing cannot be undone." + echo "**Not public yet.** Each VALIDATED deployment waits at <$CENTRAL_PORTAL> until someone presses **Publish**." + echo + echo "Publish **only** the deployments whose id is in this table, and all of them (or **Drop** all of them to abandon the release). A deployment for one of these versions with any other id or name comes from a different run: drop it, do not publish it. Publishing cannot be undone." fi } | summary diff --git a/.github/scripts/release/github-release.sh b/.github/scripts/release/github-release.sh index ca42fcd8..1260ac36 100644 --- a/.github/scripts/release/github-release.sh +++ b/.github/scripts/release/github-release.sh @@ -8,16 +8,24 @@ # GH_TOKEN, GH_REPO token with contents: write, and owner/repository # TAG, COMMIT from plan.sh # PRERELEASE true marks a newly created release as a pre-release +# REPLACE_ASSETS true allows replacing a jar that is already attached +# with different content (the replace-release-assets +# input); anything else refuses # # The tag must still point to COMMIT: the artifacts were built from that # commit, and a tag that moved since the run started is not released. # # Assets are the same as before: one -# indextables_spark--linux-x86_64-shaded.jar per Spark version, -# replacing an asset of the same name if the release already has one. The -# notes (hand-written for an existing release, generated for a new one) are -# kept; only the block between the release-build-inputs markers is added or -# replaced. +# indextables_spark--linux-x86_64-shaded.jar per Spark version. A jar +# that is already attached is left alone when its SHA-256 is that of the +# staged jar (a re-run). When it differs, the release is not touched unless +# REPLACE_ASSETS is true: a jar on a published release may already have been +# downloaded, and a rebuild is never byte-identical. The comparison is made +# for all jars before anything is uploaded. +# +# The notes (hand-written for an existing release, generated for a new one) +# are kept; only the block between the release-build-inputs markers is added +# or replaced. set -euo pipefail here="$(cd "$(dirname "$0")" && pwd)" # shellcheck source=lib.sh @@ -49,9 +57,43 @@ while IFS= read -r version; do assets+=("$f") done < "$staging/versions.txt" -if gh release view "$TAG" --json body --jq '.body' > "$tmp/body.md" 2> "$tmp/view.err"; then - echo "Release $TAG exists; attaching the jars." - gh release upload "$TAG" --clobber "${assets[@]}" +if gh release view "$TAG" --json body,assets > "$tmp/release.json" 2> "$tmp/view.err"; then + echo "Release $TAG exists." + jq -r '.body // ""' "$tmp/release.json" > "$tmp/body.md" + # Sort the jars into: not attached yet, attached and identical, attached + # with other content. Nothing is uploaded until all three are known. + missing=() + differing=() + for f in "${assets[@]}"; do + name="$(basename "$f")" + if [ "$(jq -r --arg n "$name" '[.assets[] | select(.name == $n)] | length' "$tmp/release.json")" = 0 ]; then + missing+=("$f") + continue + fi + attached="$(jq -r --arg n "$name" '.assets[] | select(.name == $n) | .digest // ""' "$tmp/release.json" | head -n 1)" + if [ -z "$attached" ]; then + # No digest recorded for this asset: fetch it and compute one. + rm -rf "$tmp/dl" + mkdir -p "$tmp/dl" + gh release download "$TAG" --pattern "$name" --dir "$tmp/dl" || die "could not download $name from release $TAG to compare it" + attached="sha256:$(hash_of sha256 "$tmp/dl/$name")" + fi + if [ "$attached" = "sha256:$(hash_of sha256 "$f")" ]; then + echo "$name is already attached with identical content." + else + differing+=("$f") + echo "$name is already attached with different content ($attached)." + fi + done + if [ "${#differing[@]}" -gt 0 ]; then + if [ "${REPLACE_ASSETS:-false}" = true ]; then + warn "Replacing ${#differing[@]} jar(s) on release $TAG that had different content, as requested by replace-release-assets" + gh release upload "$TAG" --clobber "${differing[@]}" + else + die "Release $TAG already has ${#differing[@]} of these jars with different content (above). They may have been downloaded already, and a rebuild is never byte-identical. Nothing was changed. If replacing them is intended, dispatch again with replace-release-assets ticked." + fi + fi + if [ "${#missing[@]}" -gt 0 ]; then gh release upload "$TAG" "${missing[@]}"; fi elif grep -qi 'not found' "$tmp/view.err"; then echo "Creating release $TAG." flags=(--verify-tag --generate-notes) diff --git a/.github/scripts/release/lib.sh b/.github/scripts/release/lib.sh index b4983649..0195250c 100644 --- a/.github/scripts/release/lib.sh +++ b/.github/scripts/release/lib.sh @@ -26,6 +26,49 @@ SHA_RE='^[0-9a-f]{40}$' die() { echo "::error::$*" >&2; exit 1; } warn() { echo "::warning::$*" >&2; } +# Where the scripts reach out to. Fixed: there is no variable that redirects +# them. test.sh puts stubs for curl and git redirection in front instead. +CENTRAL_API=https://central.sonatype.com/api/v1/publisher +CENTRAL_REPO=https://repo1.maven.org/maven2 +CENTRAL_PORTAL=https://central.sonatype.com/publishing/deployments +TANTIVY4JAVA_URL=https://github.com/indextables/tantivy4java.git +QUICKWIT_URL=https://github.com/indextables/quickwit +TANTIVY_URL=https://github.com/indextables/tantivy + +# The local Maven repository, where Maven puts it for this user. +M2_REPO="$HOME/.m2/repository" + +# test_setting : the value of a timing setting. Only test.sh +# may change one, by also setting RELEASE_SCRIPTS_TESTING=1, which release.yml +# never does; a setting that is present without it stops the script. +test_setting() { + local value + eval "value=\${$1:-}" + if [ -z "$value" ]; then + printf '%s\n' "$2" + elif [ "${RELEASE_SCRIPTS_TESTING:-}" = 1 ]; then + printf '%s\n' "$value" + else + die "$1 is a test-only setting; it must not be set in a real run" + fi +} + +# profile_version : the entry of a space-separated list +# of versions (one per profile, in PROFILES order) that belongs to . +profile_version() { + local p + # shellcheck disable=SC2086 + set -- "$2" $1 + local want="$1" + shift + for p in $PROFILES; do + [ $# -gt 0 ] || break + if [ "$p" = "$want" ]; then printf '%s\n' "$1"; return 0; fi + shift + done + die "no version for profile $want in the plan's version list" +} + # need : fail unless every command is on PATH. need() { local c diff --git a/.github/scripts/release/native-pins.txt b/.github/scripts/release/native-pins.txt index 61c92291..bf2690a5 100644 --- a/.github/scripts/release/native-pins.txt +++ b/.github/scripts/release/native-pins.txt @@ -17,6 +17,9 @@ # # bash .github/scripts/release/build-native.sh print-pin # -# Review the printed commits before committing them; from then on they are -# what a release is allowed to build. +# print-pin also says, on standard error, whether each fork commit is on the +# fork's default branch, warns when it is only on another branch or tag, and +# refuses when it is on none (such a commit can come from somebody's fork of +# the fork). Review that output and the printed commits before committing +# the line; from then on they are what a release is allowed to build. 0.34.4 be2d885a5b6d2e7438d88ea7f9818c5086e74a52 3026b228a71bfb6be16f3966ff80c8641834cb21 d491895f1cd54d166c8010469d4625ae81c636f3 diff --git a/.github/scripts/release/plan.sh b/.github/scripts/release/plan.sh index 333f76d5..7698850c 100644 --- a/.github/scripts/release/plan.sh +++ b/.github/scripts/release/plan.sh @@ -2,13 +2,16 @@ # Decide what this run does, before anything is built. # # Reads the triggering event and the dispatch inputs (as strings, from the -# environment), validates the tag, resolves it to a commit and checks that the -# commit is on the default branch. Sets the step outputs: +# environment), validates the tag, resolves it to a commit, checks that the +# commit is on the default branch and works out the versions that would be +# published. Sets the step outputs: # # mode dry-run | publish # tag the release tag (v0.6.0) # commit the commit the tag points to; every later job uses this # base the tag without its "v" (0.6.0) +# versions the versions to publish, one per Spark profile, in PROFILES +# order (0.6.0_spark_3.5.9 0.6.0_spark_4.0.4 0.6.0_spark_4.1.3) # prerelease true when the base version has a pre-release part (0.6.0-rc2) # central true unless a publish run asked to skip Maven Central # @@ -16,6 +19,18 @@ # * workflow_dispatch from the default branch with dry-run set to false # * a pushed tag (not enabled in release.yml; see the README) # Every other combination is a dry run or an error. +# +# Versions. _spark_, where spark.version is what each +# profile sets in pom.xml at the tagged commit. That value changes whenever +# the Spark dependency is bumped, and with it the published artifact names. +# So that this never happens unnoticed: +# * the Plan summary lists the exact versions, and for each Spark line the +# version last published to Maven Central; +# * a publish run whose Spark versions differ from the last published ones +# (or that cannot be compared) stops here unless the releaser has typed +# the Spark versions into the expected-spark-versions input; +# * when that input is given it must match what pom.xml says, in any run. +# The build jobs later check that Maven derives the same versions. set -euo pipefail here="$(cd "$(dirname "$0")" && pwd)" # shellcheck source=lib.sh @@ -23,7 +38,7 @@ here="$(cd "$(dirname "$0")" && pwd)" : "${EVENT_NAME:?}" "${REF:?}" "${DEFAULT_BRANCH:?}" repo_dir="${REPO_DIR:-.}" -need git +need git python3 curl case "$EVENT_NAME" in workflow_dispatch) @@ -63,10 +78,116 @@ case "$base" in *-*) prerelease=true ;; *) prerelease=false ;; esac central=true if [ "$mode" = publish ] && [ "${INPUT_SKIP_CENTRAL:-false}" = true ]; then central=false; fi +# --- versions ----------------------------------------------------------------- +tmp="$(mktemp -d)" +trap 'rm -rf "$tmp"' EXIT +git -C "$repo_dir" show "$commit:pom.xml" > "$tmp/pom.xml" 2> /dev/null || die "the tagged commit has no pom.xml" +# pom.xml is read as data; nothing from the tagged commit is executed here. +cat > "$tmp/spark_versions.py" << 'PY' +import sys +import xml.etree.ElementTree as ET + +ns = "{http://maven.apache.org/POM/4.0.0}" +try: + root = ET.parse(sys.argv[1]).getroot() +except ET.ParseError as exc: + sys.exit("pom.xml is not well-formed XML: %s" % exc) + +def child(node, name): + for c in (node if node is not None else []): + if c.tag == ns + name: + return c + return None + +def spark_version(node): + c = child(child(node, "properties"), "spark.version") + return (c.text or "").strip() if c is not None else "" + +top = spark_version(root) +by_profile = {} +for p in (child(root, "profiles") if child(root, "profiles") is not None else []): + pid = child(p, "id") + if pid is not None: + by_profile[(pid.text or "").strip()] = spark_version(p) +out = [] +for name in sys.argv[2:]: + if name not in by_profile: + sys.exit("pom.xml at the tag has no profile %s" % name) + value = by_profile[name] or top + if not value or "$" in value: + sys.exit("pom.xml at the tag does not state spark.version for profile %s" % name) + out.append(value) +print(" ".join(out)) +PY +# shellcheck disable=SC2086 +spark_versions="$(python3 "$tmp/spark_versions.py" "$tmp/pom.xml" $PROFILES)" \ + || die "could not read the Spark versions from pom.xml at $tag (above)" + +versions="" +# shellcheck disable=SC2086 +set -- $spark_versions +for profile in $PROFILES; do + check_version "$base" "$profile" "${base}_spark_$1" + versions="${versions:+$versions }${base}_spark_$1" + shift +done + +# What the releaser expects, if they said so: the same Spark versions, in any +# order, separated by spaces or commas. +expected="$(printf '%s' "${INPUT_EXPECTED_SPARK:-}" | tr ',' ' ' | tr -s ' ' '\n' | grep . | LC_ALL=C sort | tr '\n' ' ' | sed 's/ $//' || true)" +confirmed=false +if [ -n "$expected" ]; then + derived_sorted="$(printf '%s\n' $spark_versions | LC_ALL=C sort | tr '\n' ' ' | sed 's/ $//')" + [ "$expected" = "$derived_sorted" ] \ + || die "expected-spark-versions is '$expected', but pom.xml at $tag gives '$derived_sorted'. Nothing was built. Check which is right before trying again." + confirmed=true +fi + +# Compare with what was last published for each Spark line. +metadata="" +if curl -fsS --connect-timeout 15 --max-time 60 -o "$tmp/metadata.xml" "$CENTRAL_REPO/$GROUP_PATH/maven-metadata.xml" 2> /dev/null; then + metadata="$tmp/metadata.xml" +fi +changed="" +comparison="" +# shellcheck disable=SC2086 +set -- $spark_versions +for profile in $PROFILES; do + line="$(spark_line "$profile")" + if [ -z "$metadata" ]; then + last="unknown" + else + last="$(grep -o -E "[^<]*_spark_${line//./\\.}\.[0-9]+" "$metadata" | tail -n 1 \ + | sed -E 's/.*_spark_([0-9.]+)<.*/\1/' || true)" + last="${last:-none}" + fi + case "$last" in + "$1") note="unchanged" ;; + none) note="first release for Spark $line" ;; + unknown) note="could not be compared"; changed="$changed $line" ;; + *) note="**changed** from $last"; changed="$changed $line" ;; + esac + comparison="$comparison| \`${base}_spark_$1\` | ${last} | $note |"$'\n' + shift +done + +if [ -n "$changed" ] && [ "$confirmed" != true ]; then + if [ -z "$metadata" ]; then + what="The versions last published to Maven Central could not be read, so the artifact names could not be compared with the last release." + else + what="The Spark version in the artifact names differs from the last release for Spark$changed. Consumers that name the full version need the new one." + fi + if [ "$mode" = publish ]; then + die "$what This run would publish: $versions. If that is intended, dispatch again with expected-spark-versions set to: $spark_versions" + fi + warn "$what A publish run will ask for expected-spark-versions: $spark_versions" +fi + output mode "$mode" output tag "$tag" output commit "$commit" output base "$base" +output versions "$versions" output prerelease "$prerelease" output central "$central" @@ -89,6 +210,19 @@ output central "$central" echo "|---|---|" echo "| Tag | \`$tag\` |" echo "| Commit | \`$commit\` |" - echo "| Versions | \`${base}_spark_\` |" echo "| Workflow ref | \`$REF\` |" + echo + echo "### Versions" + echo + echo "| To be published | Spark version last on Maven Central | |" + echo "|---|---|---|" + printf '%s' "$comparison" + echo + if [ "$confirmed" = true ]; then + echo "The Spark versions were confirmed with \`expected-spark-versions\`." + elif [ -n "$changed" ]; then + echo "> **The artifact names are not the same as in the last release.** To publish them, set \`expected-spark-versions\` to \`$spark_versions\`." + else + echo "Same Spark versions as the last release. (\`expected-spark-versions\` would be \`$spark_versions\`.)" + fi } | summary diff --git a/.github/scripts/release/test.sh b/.github/scripts/release/test.sh index 4a1efd6f..c6cb85f7 100644 --- a/.github/scripts/release/test.sh +++ b/.github/scripts/release/test.sh @@ -21,6 +21,7 @@ T="$(cd "$T" && pwd -P)" trap 'rm -rf "$T"' EXIT # shellcheck source=lib.sh . "$here/lib.sh" +SCRIPTS_UNDER_TEST="$(ls "$here"/*.sh | grep -v '/test\.sh$')" pass=0 fail=0 @@ -53,6 +54,12 @@ SHA_A=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa SHA_B=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb SHA_C=cccccccccccccccccccccccccccccccccccccccc SHA_D=dddddddddddddddddddddddddddddddddddddddd +# The versions the plan announces in most tests below. +EV="1.2.3_spark_3.5.8 1.2.3_spark_4.0.3 1.2.3_spark_4.1.2" +# Variables that used to redirect the scripts, or might be expected to. None +# of them may have any effect; they are set here for every test. +export CENTRAL_API_URL=https://override.invalid/api CENTRAL_REPO_URL=https://override.invalid/maven2 +export T4J_REPO=file:///override/invalid M2_REPO=/override/invalid # --------------------------------------------------------------------------- # lib.sh @@ -83,34 +90,113 @@ check "lib: verify_sums refuses a symbolic link" bash -c "! (. '$here/lib.sh'; v # plan.sh # --------------------------------------------------------------------------- P="$T/plan" -mkdir -p "$P" +mkdir -p "$P/bin" +# Stub curl: serves the Maven Central metadata fixture, or fails like an +# unreachable host when there is none. Every plan.sh call below uses it. +cat > "$P/bin/curl" << 'STUB' +#!/usr/bin/env bash +o=""; url="" +while [ $# -gt 0 ]; do case "$1" in -o) o="$2"; shift 2 ;; --connect-timeout | --max-time) shift 2 ;; -*) shift ;; *) url="$1"; shift ;; esac; done +echo "$url" >> "$PFAKE/calls.log" +[ "$url" = "https://repo1.maven.org/maven2/io/indextables/indextables_spark/maven-metadata.xml" ] || exit 97 +[ -f "$PFAKE/metadata.xml" ] || exit 6 +cp "$PFAKE/metadata.xml" "$o" +STUB +chmod +x "$P/bin/curl" +PFAKE="$P/fake"; mkdir -p "$PFAKE"; export PFAKE +central_has() { # central_has ...: what Maven Central has published + { echo ""; for v in "$@"; do echo " $v"; done; echo ""; } > "$PFAKE/metadata.xml" +} +central_has 0.9.0_spark_3.5.3 1.0.0_spark_3.5.8 1.0.0_spark_4.0.3 1.0.0_spark_4.1.2 +plan_pom() { # plan_pom <3.5 version> <4.0 version> <4.1 version> + cat << EOF + + + + + $1 + + + spark-3.5 + spark-4.0$2 + spark-4.1$3 + + +EOF +} git init -q -b main "$P/upstream" ( cd "$P/upstream" - echo 1 > f && git add f && git commit -q -m one && git tag v1.2.3 - echo 2 > f && git commit -q -am two && git tag -a v1.3.0-rc1 -m rc - git checkout -q -b side && echo 3 > f && git commit -q -am three && git tag v9.9.9 + plan_pom 3.5.8 4.0.3 4.1.2 > pom.xml && git add pom.xml && git commit -q -m one && git tag v1.2.3 + plan_pom 3.5.9 4.0.3 4.1.2 > pom.xml && git commit -q -am two && git tag -a v1.3.0-rc1 -m rc + plan_pom 3.5.8 4.1.9 4.1.2 > pom.xml && git commit -q -am wrongline && git tag v1.3.1 + plan_pom 3.5.8 4.0.3 4.1.2-SNAPSHOT > pom.xml && git commit -q -am snapshot && git tag v1.3.2 + echo "" > pom.xml && git commit -q -am noprofiles && git tag v1.4.0 + git rm -q pom.xml && git commit -q -m nopom && git tag v1.5.0 + git checkout -q -b side && echo 3 > f && git add f && git commit -q -m three && git tag v9.9.9 git checkout -q main ) git clone -q "$P/upstream" "$P/repo" V123="$(git -C "$P/upstream" rev-parse 'v1.2.3^{commit}')" RC1="$(git -C "$P/upstream" rev-parse 'v1.3.0-rc1^{commit}')" -plan() { # plan EVENT REF REF_TYPE REF_NAME TAG DRY SKIP +OLDPATH="$PATH" +PATH="$P/bin:$PATH" +plan() { # plan EVENT REF REF_TYPE REF_NAME TAG DRY SKIP [EXPECTED] run "$1" env EVENT_NAME="$2" REF="$3" REF_TYPE="$4" REF_NAME="$5" INPUT_TAG="$6" INPUT_DRY_RUN="$7" \ - INPUT_SKIP_CENTRAL="$8" DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" + INPUT_SKIP_CENTRAL="$8" INPUT_EXPECTED_SPARK="${9:-}" DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" } +dispatch() { plan "$1" workflow_dispatch refs/heads/main branch main "$2" "$3" false "${4:-}"; } # dispatch TAG DRY [EXPECTED] L="$P/log" check "plan: dispatch with dry-run ticked is a dry run" plan "$L.1" workflow_dispatch refs/heads/main branch main v1.2.3 true false check "plan: ... mode dry-run" is "$(out "$L.1" mode)" dry-run check "plan: ... tag resolved to its commit" is "$(out "$L.1" commit)" "$V123" check "plan: ... base version" is "$(out "$L.1" base)" 1.2.3 +check "plan: ... versions from pom.xml at the tag, one per profile" is "$(out "$L.1" versions)" "1.2.3_spark_3.5.8 1.2.3_spark_4.0.3 1.2.3_spark_4.1.2" check "plan: ... not a pre-release" is "$(out "$L.1" prerelease)" false check "plan: ... summary says nothing is uploaded" has "$L.1.summary" "Nothing is uploaded" -check "plan: dispatch from main with dry-run unticked publishes" plan "$L.2" workflow_dispatch refs/heads/main branch main v1.3.0-rc1 false false +check "plan: ... summary lists each version with the one last published" has "$L.1.summary" '| `1.2.3_spark_3.5.8` | 3.5.8 | unchanged |' +check "plan: ... and gives the value for expected-spark-versions" has "$L.1.summary" '`3.5.8 4.0.3 4.1.2`' +check "plan: dispatch from main with dry-run unticked publishes" dispatch "$L.2" v1.2.3 false check "plan: ... mode publish" is "$(out "$L.2" mode)" publish -check "plan: ... annotated tag resolved to its commit" is "$(out "$L.2" commit)" "$RC1" -check "plan: ... pre-release" is "$(out "$L.2" prerelease)" true check "plan: ... Central included" is "$(out "$L.2" central)" true +check "plan: ... only the Maven Central metadata was fetched" is "$(sort -u "$PFAKE/calls.log")" "https://repo1.maven.org/maven2/io/indextables/indextables_spark/maven-metadata.xml" + +# The Spark version in the artifact names changed since the last release. +check "plan: a dry run with changed Spark versions goes ahead" dispatch "$L.20" v1.3.0-rc1 true +check "plan: ... with a warning" has "$L.20" "::warning::The Spark version in the artifact names differs from the last release for Spark 3.5" +check "plan: ... the summary marks the changed line" has "$L.20.summary" '| `1.3.0-rc1_spark_3.5.9` | 3.5.8 | **changed** from 3.5.8 |' +check "plan: ... and leaves the others as unchanged" has "$L.20.summary" '| `1.3.0-rc1_spark_4.0.3` | 4.0.3 | unchanged |' +check "plan: ... and says what a publish run needs" has "$L.20.summary" 'set `expected-spark-versions` to `3.5.9 4.0.3 4.1.2`' +check "plan: publishing changed Spark versions without confirming them is refused" fails_with "dispatch again with expected-spark-versions set to: 3.5.9 4.0.3 4.1.2" "$L.21" dispatch "$L.21" v1.3.0-rc1 false +check "plan: ... and sets no mode" test -z "$(out "$L.21" mode)" +check "plan: ... the message names what would be published" has "$L.21" "1.3.0-rc1_spark_3.5.9 1.3.0-rc1_spark_4.0.3 1.3.0-rc1_spark_4.1.2" +check "plan: publishing them with expected-spark-versions goes ahead" dispatch "$L.22" v1.3.0-rc1 false "3.5.9 4.0.3 4.1.2" +check "plan: ... mode publish" is "$(out "$L.22" mode)" publish +check "plan: ... annotated tag resolved to its commit" is "$(out "$L.22" commit)" "$RC1" +check "plan: ... pre-release" is "$(out "$L.22" prerelease)" true +check "plan: ... versions" is "$(out "$L.22" versions)" "1.3.0-rc1_spark_3.5.9 1.3.0-rc1_spark_4.0.3 1.3.0-rc1_spark_4.1.2" +check "plan: ... the summary says they were confirmed" has "$L.22.summary" "confirmed with" +check "plan: expected-spark-versions in another order, with commas" dispatch "$L.23" v1.3.0-rc1 false "4.1.2, 4.0.3,3.5.9" +check "plan: expected-spark-versions that does not match pom.xml is refused" fails_with "but pom.xml at v1.3.0-rc1 gives '3.5.9 4.0.3 4.1.2'" "$L.24" dispatch "$L.24" v1.3.0-rc1 false "3.5.8 4.0.3 4.1.2" +check "plan: ... in a dry run as well" fails_with "but pom.xml at v1.3.0-rc1 gives" "$L.25" dispatch "$L.25" v1.3.0-rc1 true "3.5.8 4.0.3 4.1.2" +check "plan: ... also when the versions are unchanged" fails_with "but pom.xml at v1.2.3 gives" "$L.26" dispatch "$L.26" v1.2.3 false "3.5.9 4.0.3 4.1.2" +check "plan: an incomplete expected-spark-versions is refused" fails_with "but pom.xml at" "$L.27" dispatch "$L.27" v1.3.0-rc1 false "3.5.9" +central_has 0.9.0_spark_3.5.3 1.0.0_spark_3.5.8 1.0.0_spark_4.0.3 +check "plan: a Spark line that was never published is not a change" dispatch "$L.28" v1.2.3 false +check "plan: ... the summary says it is the first" has "$L.28.summary" "first release for Spark 4.1" +central_has 0.9.0_spark_3.5.3 1.0.0_spark_3.5.8 1.0.0_spark_4.0.3 1.0.0_spark_4.1.2 1.1.0_spark_3.5.9 1.1.0_spark_4.0.3 1.1.0_spark_4.1.2 +check "plan: the comparison is with the version published last" fails_with "expected-spark-versions set to: 3.5.8 4.0.3 4.1.2" "$L.29" dispatch "$L.29" v1.2.3 false +rm "$PFAKE/metadata.xml" +check "plan: publishing is refused when Maven Central cannot be read" fails_with "could not be read" "$L.30" dispatch "$L.30" v1.2.3 false +check "plan: ... unless the versions are confirmed" dispatch "$L.31" v1.2.3 false "3.5.8 4.0.3 4.1.2" +check "plan: ... a dry run goes ahead with a warning" dispatch "$L.32" v1.2.3 true +check "plan: ... (the warning)" has "$L.32" "::warning::The versions last published to Maven Central could not be read" +central_has 0.9.0_spark_3.5.3 1.0.0_spark_3.5.8 1.0.0_spark_4.0.3 1.0.0_spark_4.1.2 +check "plan: a profile whose Spark version is outside its Spark line is refused" fails_with "version '1.3.1_spark_4.1.9' is not 1.3.1_spark_4.0." "$L.33a" dispatch "$L.33a" v1.3.1 true +check "plan: a Spark version that is not a plain release is refused" fails_with "is not 1.3.2_spark_4.1." "$L.33b" dispatch "$L.33b" v1.3.2 true +check "plan: a pom.xml without the Spark profiles is refused" fails_with "pom.xml at the tag has no profile spark-3.5" "$L.33" dispatch "$L.33" v1.4.0 true +check "plan: a commit without pom.xml is refused" fails_with "the tagged commit has no pom.xml" "$L.34" dispatch "$L.34" v1.5.0 true + check "plan: publish from another branch is refused" fails_with "can only be dispatched from main" "$L.3" \ env EVENT_NAME=workflow_dispatch REF=refs/heads/feature/x INPUT_TAG=v1.2.3 INPUT_DRY_RUN=false DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" check "plan: ... and sets no mode" test -z "$(out "$L.3" mode)" @@ -145,6 +231,7 @@ check "plan: a tag that is not on main is refused" fails_with "is not an ancesto env EVENT_NAME=workflow_dispatch REF=refs/heads/main INPUT_TAG=v9.9.9 INPUT_DRY_RUN=false DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" check "plan: ... in a dry run as well" fails_with "is not an ancestor of main" "$L.13" \ env EVENT_NAME=workflow_dispatch REF=refs/heads/main INPUT_TAG=v9.9.9 INPUT_DRY_RUN=true DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +PATH="$OLDPATH" # --------------------------------------------------------------------------- # Stubs for the build tools. $FAKE holds fixtures and switches. @@ -248,9 +335,19 @@ EOF # build-native.sh # --------------------------------------------------------------------------- N="$T/native"; mkdir -p "$N" -QW=1111111111111111111111111111111111111111 -TV=2222222222222222222222222222222222222222 MR=3333333333333333333333333333333333333333 +new_fork() { # new_fork : a fork-like repository, one commit on main and one on a side branch + rm -rf "$1"; git init -q -b main "$1" + (cd "$1" && echo "$1" > f && git add f && git commit -q -m main && git checkout -q -b upgrade && echo side >> f && git commit -q -am side && git checkout -q main) + git -C "$1" config uploadpack.allowAnySHA1InWant true + git -C "$1" config uploadpack.allowFilter true +} +new_fork "$N/quickwit"; new_fork "$N/tantivy" +QW="$(git -C "$N/quickwit" rev-parse main)"; QW_SIDE="$(git -C "$N/quickwit" rev-parse upgrade)" +TV="$(git -C "$N/tantivy" rev-parse main)" +# The scripts always name the real repositories; git is told to fetch them +# from the fixtures instead. +redirect() { echo "GIT_CONFIG_KEY_0=url.file://$1.insteadOf"; } new_t4j() { # new_t4j : a tantivy4java-like repository with tag v0.34.4 rm -rf "$1"; git init -q -b main "$1" mkdir -p "$1/native" "$1/.cargo" @@ -291,8 +388,16 @@ printf '# comment\n0.34.4 %s %s %s\n0.1.0 %s - -\n' "$T4J_COMMIT" "$QW" "$TV" "$ echo 0.34.4 > "$FAKE/t4j.version" native() { # native [VAR=value...] local log="$1" sub="$2"; shift 2 - run "$log" env PATH="$STUBPATH" T4J_REPO="file://$N/t4j" POM="$N/pom.xml" PINS="$N/pins.txt" WORK="$N/work" OUT="$N/out" \ - M2_REPO="$N/m2" "$@" bash "$here/build-native.sh" "$sub" + run "$log" env PATH="$STUBPATH" HOME="$N/home" GIT_CONFIG_COUNT=1 "$(redirect "$N/t4j")" \ + GIT_CONFIG_VALUE_0=https://github.com/indextables/tantivy4java.git \ + POM="$N/pom.xml" PINS="$N/pins.txt" WORK="$N/work" OUT="$N/out" "$@" bash "$here/build-native.sh" "$sub" +} +pin() { # pin : print-pin, with all three repositories redirected + run "$1" env PATH="$STUBPATH" HOME="$N/home" GIT_CONFIG_COUNT=3 "$(redirect "$2")" \ + GIT_CONFIG_VALUE_0=https://github.com/indextables/tantivy4java.git \ + GIT_CONFIG_KEY_1="url.file://$N/quickwit.insteadOf" GIT_CONFIG_VALUE_1=https://github.com/indextables/quickwit \ + GIT_CONFIG_KEY_2="url.file://$N/tantivy.insteadOf" GIT_CONFIG_VALUE_2=https://github.com/indextables/tantivy \ + bash "$here/build-native.sh" print-pin 0.34.4 } L="$N/log" check "native: resolve accepts the pinned tag" native "$L.1" resolve @@ -301,12 +406,18 @@ check "native: ... records the quickwit fork commit" has "$N/out/native-inputs.t check "native: ... records the tantivy fork commit" has "$N/out/native-inputs.txt" "tantivy_commit=$TV" check "native: ... records other git dependencies with their commit" has "$N/out/native-inputs.txt" "other_git_dependencies=https://github.com/quickwit-oss/mrecordlog@$MR" check "native: ... checked out exactly the pinned commit" is "$(git -C "$N/work/tantivy4java" rev-parse HEAD)" "$T4J_COMMIT" -check "native: print-pin prints the pin line" is "$(PATH="$STUBPATH" T4J_REPO="file://$N/t4j" bash "$here/build-native.sh" print-pin 0.34.4 2> /dev/null)" "0.34.4 $T4J_COMMIT $QW $TV" +# shellcheck disable=SC2086 +check "native: no variable redirects the repositories or the local Maven repository" test -z "$(grep -n -E 'T4J_REPO|M2_REPO:-|TANTIVY4JAVA_URL:-|QUICKWIT_URL:-|TANTIVY_URL:-' $SCRIPTS_UNDER_TEST)" +check "native: ... T4J_REPO and M2_REPO in the environment changed nothing" test "$T4J_REPO" = file:///override/invalid -a "$M2_REPO" = /override/invalid -a ! -e /override +check "native: print-pin prints the pin line" pin "$L.p1" "$N/t4j" +check "native: ... (the line)" test "$(grep -c -x "0.34.4 $T4J_COMMIT $QW $TV" "$L.p1")" = 1 +check "native: ... says the quickwit commit is on the fork's default branch" has "$L.p1" "quickwit $QW: reachable from the default branch (main) of https://github.com/indextables/quickwit" +check "native: ... and the tantivy commit" has "$L.p1" "tantivy $TV: reachable from the default branch (main) of https://github.com/indextables/tantivy" check "native: build needs the toolchain step first" fails_with "toolchain" "$L.2" native "$L.2" build mkdir -p "$N/work/protoc/bin" "$N/work/protoc/include" && printf '#!/bin/sh\necho "libprotoc 25.5"\n' > "$N/work/protoc/bin/protoc" && chmod +x "$N/work/protoc/bin/protoc" -mkdir -p "$N/m2/io/indextables/tantivy4java/0.34.4" && echo stale > "$N/m2/io/indextables/tantivy4java/0.34.4/tantivy4java-0.34.4-linux-x86_64.jar" -echo stale > "$N/m2/io/indextables/tantivy4java/0.34.4/tantivy4java-0.34.4-left-over.jar" +mkdir -p "$N/home/.m2/repository/io/indextables/tantivy4java/0.34.4" && echo stale > "$N/home/.m2/repository/io/indextables/tantivy4java/0.34.4/tantivy4java-0.34.4-linux-x86_64.jar" +echo stale > "$N/home/.m2/repository/io/indextables/tantivy4java/0.34.4/tantivy4java-0.34.4-left-over.jar" check "native: build collects the jar" native "$L.3" build check "native: ... a jar that was there before the build is not reused" has "$L.3" "removing it before the build" check "native: ... the collected jar is the one this build installed" is "$(hash_of sha256 "$N/out/repository/tantivy4java-0.34.4-linux-x86_64.jar")" "$(hash_of sha256 "$T/fx/t4j.jar")" @@ -342,16 +453,22 @@ variant() { # variant : a repository that differs in one wa printf '0.34.4 %s %s %s\n' "$(git -C "$N/v-$1" rev-parse HEAD)" "$QW" "$TV" > "$N/pins.$1" } variant path "echo 'quickwit-common = { path = \"../../quickwit/quickwit-common\" }' >> native/Cargo.toml" -check "native: a dependency on a sibling checkout is refused" fails_with "outside the tantivy4java checkout" "$L.12" native "$L.12" resolve T4J_REPO="file://$N/v-path" PINS="$N/pins.path" +check "native: a dependency on a sibling checkout is refused" fails_with "outside the tantivy4java checkout" "$L.12" native "$L.12" resolve "$(redirect "$N/v-path")" PINS="$N/pins.path" variant patch "printf '[patch.crates-io]\nserde = { path = \"/tmp/serde\" }\n' >> .cargo/config.toml" -check "native: a cargo config that overrides sources is refused" fails_with "overrides dependency sources" "$L.13" native "$L.13" resolve T4J_REPO="file://$N/v-patch" PINS="$N/pins.patch" +check "native: a cargo config that overrides sources is refused" fails_with "overrides dependency sources" "$L.13" native "$L.13" resolve "$(redirect "$N/v-patch")" PINS="$N/pins.patch" variant two "printf '[[package]]\nname = \"quickwit-x\"\nsource = \"git+https://github.com/indextables/quickwit?rev=$SHA_C#$SHA_C\"\n' >> native/Cargo.lock" -check "native: a lock file with two quickwit commits is refused" fails_with "more than one commit" "$L.14" native "$L.14" resolve T4J_REPO="file://$N/v-two" PINS="$N/pins.two" +check "native: a lock file with two quickwit commits is refused" fails_with "more than one commit" "$L.14" native "$L.14" resolve "$(redirect "$N/v-two")" PINS="$N/pins.two" variant short "printf '[[package]]\nname = \"y\"\nsource = \"git+https://github.com/indextables/quickwit?branch=main\"\n' >> native/Cargo.lock" -check "native: a git dependency without a resolved commit is refused" fails_with "without a resolved commit" "$L.15" native "$L.15" resolve T4J_REPO="file://$N/v-short" PINS="$N/pins.short" +check "native: a git dependency without a resolved commit is refused" fails_with "without a resolved commit" "$L.15" native "$L.15" resolve "$(redirect "$N/v-short")" PINS="$N/pins.short" +variant side "sed -i.bak 's/$QW/$QW_SIDE/g' native/Cargo.lock native/Cargo.toml && rm native/Cargo.lock.bak native/Cargo.toml.bak" +check "native: print-pin accepts a fork commit on another branch" pin "$L.p2" "$N/v-side" +check "native: ... but says it is not on the default branch, and where it is" has "$L.p2" "quickwit $QW_SIDE is NOT reachable from the default branch (main) of https://github.com/indextables/quickwit; it is on: upgrade" +variant orphan "sed -i.bak 's/$QW/$SHA_C/g' native/Cargo.lock native/Cargo.toml && rm native/Cargo.lock.bak native/Cargo.toml.bak" +check "native: print-pin refuses a fork commit that is on no branch or tag of the fork" fails_with "is not reachable from any branch or tag" "$L.p3" pin "$L.p3" "$N/v-orphan" +check "native: ... and prints no pin line" test -z "$(grep -E '^0\.34\.4 ' "$L.p3")" new_t4j "$N/v-annot"; git -C "$N/v-annot" tag -a v0.34.4 -m annotated printf '0.34.4 %s %s %s\n' "$(git -C "$N/v-annot" rev-parse HEAD)" "$QW" "$TV" > "$N/pins.annot" -check "native: an annotated tag is resolved to its commit" native "$L.16" resolve T4J_REPO="file://$N/v-annot" PINS="$N/pins.annot" +check "native: an annotated tag is resolved to its commit" native "$L.16" resolve "$(redirect "$N/v-annot")" PINS="$N/pins.annot" # toolchain: a download with the wrong digest is never unpacked. cat > "$T/bin/curl" << 'STUB' @@ -383,9 +500,9 @@ B="$T/build"; mkdir -p "$B" echo 3.5.8 > "$FAKE/spark.spark-3.5"; echo 4.0.3 > "$FAKE/spark.spark-4.0"; echo 4.1.2 > "$FAKE/spark.spark-4.1" artifacts() { # artifacts [VAR=value...] local log="$1" profile="$2"; shift 2 - rm -rf "$B/src-$profile" "$B/m2-$profile"; mkdir -p "$B/src-$profile"; cp "$N/pom.xml" "$B/src-$profile/pom.xml" - run "$log" env PATH="$STUBPATH" SRC="$B/src-$profile" PROFILE="$profile" BASE_VERSION=1.2.3 NATIVE_DIR="$N/out.good" \ - OUT="$B/out/artifacts-$profile" M2_REPO="$B/m2-$profile" "$@" bash "$here/build-artifacts.sh" + rm -rf "$B/src-$profile" "$B/home-$profile"; mkdir -p "$B/src-$profile"; cp "$N/pom.xml" "$B/src-$profile/pom.xml" + run "$log" env PATH="$STUBPATH" SRC="$B/src-$profile" PROFILE="$profile" BASE_VERSION=1.2.3 EXPECTED_VERSIONS="$EV" NATIVE_DIR="$N/out.good" \ + OUT="$B/out/artifacts-$profile" HOME="$B/home-$profile" "$@" bash "$here/build-artifacts.sh" } L="$B/log" : > "$FAKE/mvn.log" @@ -401,13 +518,15 @@ check "build: the release profile runs with signing off" has "$FAKE/mvn.log" "-P check "build: plugins invoked by coordinate are pinned" test -z "$(grep -E ':(evaluate|set)' "$FAKE/mvn.log" | grep -v -E '[a-z.-]+:[a-z-]+:[0-9.]+:(evaluate|set)')" cp -R "$B/out" "$B/out.good" -mkdir -p "$B/m2-pre/io/indextables/tantivy4java/0.34.4" +mkdir -p "$B/home-pre/.m2/repository/io/indextables/tantivy4java/0.34.4" check "build: a tantivy4java already in the local repository is refused" fails_with "already in the local Maven repository" "$L.1" \ - env PATH="$STUBPATH" SRC="$B/src-spark-3.5" PROFILE=spark-3.5 BASE_VERSION=1.2.3 NATIVE_DIR="$N/out.good" OUT="$B/out/x" M2_REPO="$B/m2-pre" bash "$here/build-artifacts.sh" + env PATH="$STUBPATH" SRC="$B/src-spark-3.5" PROFILE=spark-3.5 BASE_VERSION=1.2.3 EXPECTED_VERSIONS="$EV" NATIVE_DIR="$N/out.good" OUT="$B/out/x" HOME="$B/home-pre" bash "$here/build-artifacts.sh" cp -R "$N/out.good" "$B/native.bad" && echo tampered >> "$B/native.bad/repository/tantivy4java-0.34.4-linux-x86_64.jar" check "build: a native artifact that does not match its digests is refused" fails_with "digest mismatch" "$L.2" artifacts "$L.2" spark-3.5 NATIVE_DIR="$B/native.bad" echo 4.0.3 > "$FAKE/spark.spark-3.5" check "build: a Spark version outside the profile's line is refused" fails_with "is not 1.2.3_spark_3.5." "$L.3" artifacts "$L.3" spark-3.5 +echo 3.5.9 > "$FAKE/spark.spark-3.5" +check "build: a version other than the one the plan announced is refused" fails_with "Maven derives 1.2.3_spark_3.5.9 for spark-3.5, but the plan announced 1.2.3_spark_3.5.8" "$L.3b" artifacts "$L.3b" spark-3.5 echo 3.5.8 > "$FAKE/spark.spark-3.5" check "build: an unknown profile is refused" fails_with "unknown Spark profile" "$L.4" artifacts "$L.4" spark-5.0 touch "$FAKE/no-javadoc" @@ -426,7 +545,7 @@ A="$T/assemble"; mkdir -p "$A" new_in() { rm -rf "$A/in"; mkdir -p "$A/in"; cp -R "$N/out.good" "$A/in/native"; cp -R "$B/out.good/"* "$A/in/"; } assemble() { # assemble [VAR=value...] local log="$1"; shift - run "$log" env IN="$A/in" OUT="$A/staging" TAG=v1.2.3 COMMIT="$SHA_D" BASE_VERSION=1.2.3 RUN_URL=https://example.invalid/run/1 "$@" bash "$here/assemble.sh" + run "$log" env IN="$A/in" OUT="$A/staging" TAG=v1.2.3 COMMIT="$SHA_D" BASE_VERSION=1.2.3 EXPECTED_VERSIONS="$EV" RUN_URL=https://example.invalid/run/1 "$@" bash "$here/assemble.sh" } resum() { (. "$here/lib.sh"; write_sums "$1"); } L="$A/log" @@ -446,6 +565,7 @@ check "assemble: summary shows the manifest digest" has "$L.1.summary" "$(out "$ MANIFEST="$(out "$L.1" manifest_sha256)" rm -rf "$A/staging.good" && cp -R "$A/staging" "$A/staging.good" +check "assemble: versions other than the ones the plan announced are refused" fails_with "is not the version the plan announced for spark-4.1" "$L.2a" assemble "$L.2a" EXPECTED_VERSIONS="1.2.3_spark_3.5.8 1.2.3_spark_4.0.3 1.2.3_spark_4.1.3" check "assemble: a tag that does not match the base version is refused" fails_with "does not belong to tag" "$L.2" assemble "$L.2" TAG=v1.2.4 new_in; rm -rf "$A/in/artifacts-spark-4.1" check "assemble: a missing profile is refused" fails_with "no build output for spark-4.1" "$L.3" assemble "$L.3" @@ -487,7 +607,7 @@ check "assemble: ... a placeholder is shown instead" has "$A/staging/release-not check "assemble: ... the text itself is absent" hasnt "$A/staging/release-notes.md" "" "$L.7" verify "$L.7" "$MANIFEST" 1.2.4 +check "verify-staging: versions other than the ones the plan announced are refused" fails_with "are not the versions the plan announced" "$L.7b" verify "$L.7b" "$MANIFEST" 1.2.3 "1.2.3_spark_3.5.8 1.2.3_spark_4.0.3 1.2.3_spark_4.1.9" # A self-consistent directory for other coordinates: valid manifest, wrong content. staging_copy mkdir -p "$A/s/bundles/1.2.3_spark_3.5.8/io/indextables/tantivy4java/9.9.9" && echo x > "$A/s/bundles/1.2.3_spark_3.5.8/io/indextables/tantivy4java/9.9.9/tantivy4java-9.9.9.jar" @@ -640,7 +761,7 @@ check "check: the restored bundles pass again" chk "$L.13" # Signing writes nothing but signatures into the staging directory: with # those removed it verifies against the original manifest again. find "$S/staging" -name '*.asc' -exec rm {} + -check "sign: nothing but signatures is written into the staging directory" run "$L.14" env MANIFEST_SHA256="$MANIFEST" BASE_VERSION=1.2.3 bash "$here/verify-staging.sh" "$S/staging" +check "sign: nothing but signatures is written into the staging directory" run "$L.14" env MANIFEST_SHA256="$MANIFEST" BASE_VERSION=1.2.3 EXPECTED_VERSIONS="$EV" bash "$here/verify-staging.sh" "$S/staging" # The publish job's path: import a passphrase-protected private key from the # environment, then sign with it. The key is the throwaway key, exported. @@ -666,12 +787,18 @@ rm -f "$S/key.asc" # central-upload.sh, against a stub curl # --------------------------------------------------------------------------- C="$T/central"; mkdir -p "$C/bin" +PORTAL_API=https://central.sonatype.com/api/v1/publisher +REPO1=https://repo1.maven.org/maven2 cat > "$C/bin/curl" << 'STUB' #!/usr/bin/env bash -# Stub curl for the Portal API (https://portal.test/api) and the public -# repository (https://repo.test/maven2). Records " [auth]". +# Stub curl for the Portal API and the public repository, under their real +# addresses (nothing here reaches the network). Records " +# [auth]". Like the Portal, it accepts a drop only for a deployment that is +# VALIDATED or FAILED. set -u -method=GET; out=""; fmt=""; hdr=""; form=""; url=""; failflag=0 +API=https://central.sonatype.com/api/v1/publisher +REPO=https://repo1.maven.org/maven2 +method=GET; out=""; fmt=""; hdr=""; form=""; url="" while [ $# -gt 0 ]; do case "$1" in -X) method="$2"; shift 2 ;; @@ -680,8 +807,7 @@ while [ $# -gt 0 ]; do -w) fmt="$2"; shift 2 ;; -F) form="$2"; shift 2 ;; --connect-timeout | --max-time | --retry) shift 2 ;; - -sS) shift ;; - -fsS) failflag=1; shift ;; + -sS | -fsS) shift ;; -*) echo "stub curl: unexpected option $1" >&2; exit 97 ;; *) url="$1"; shift ;; esac @@ -697,36 +823,41 @@ respond() { # respond if [ -n "$out" ]; then printf '%s' "$2" > "$out"; else printf '%s' "$2"; fi [ -z "$fmt" ] || printf '%s' "$1" } +state_of() { if [ -f "$CFAKE/status.$1" ]; then head -n 1 "$CFAKE/status.$1"; else echo VALIDATED; fi; } case "$url" in - https://repo.test/maven2/*.pom) + "$REPO"/*.pom) v="$(basename "$(dirname "$url")")" respond "$(cat "$CFAKE/repo.$v.code" 2> /dev/null || echo 404)" "" ;; - https://repo.test/maven2/*.sha1) + "$REPO"/*.sha1) f="$CFAKE/repo.$(basename "$url")" [ -f "$f" ] || exit 22 cat "$f" ;; - "https://portal.test/api/upload?name="*"&publishingType=USER_MANAGED") + "$API/upload?name="*"&publishingType=USER_MANAGED") [ "$method" = POST ] || exit 97 - n="$(grep -c 'POST https://portal.test/api/upload' "$CFAKE/calls.log")" + n="$(grep -c "POST $API/upload" "$CFAKE/calls.log")" echo "$form" >> "$CFAKE/forms.log" if [ -f "$CFAKE/upload.$n.exit" ]; then exit "$(cat "$CFAKE/upload.$n.exit")"; fi respond "$(cat "$CFAKE/upload.$n.code" 2> /dev/null || echo 201)" "$(cat "$CFAKE/upload.$n.body" 2> /dev/null || echo "00000000-0000-4000-8000-00000000000$n")" ;; - "https://portal.test/api/status?id="*) + "$API/status?id="*) [ "$method" = POST ] || exit 97 id="${url##*id=}"; f="$CFAKE/status.$id" - [ -f "$f" ] || echo VALIDATED > "$f" - st="$(head -n 1 "$f")" - [ "$(grep -c . "$f")" -le 1 ] || { tail -n +2 "$f" > "$f.next" && mv "$f.next" "$f"; } + st="$(state_of "$id")" + # A status file is a sequence: each call moves on one line; the last stays. + if [ -f "$f" ] && [ "$(grep -c . "$f")" -gt 1 ]; then tail -n +2 "$f" > "$f.next" && mv "$f.next" "$f"; fi case "$st" in HTTP*) respond "${st#HTTP}" "upstream error" ;; *) respond 200 "{\"deploymentId\":\"$id\",\"deploymentState\":\"$st\",\"errors\":{\"pkg\":[\"reason from the Portal\"]}}" ;; esac ;; - https://portal.test/api/deployment/*) + "$API"/deployment/*) id="${url##*/}" - case "$method" in - DELETE) respond "$(cat "$CFAKE/drop.$id.code" 2> /dev/null || echo 204)" "" ;; - *) echo "stub curl: $method on a deployment is the Publish call" >&2; exit 97 ;; - esac ;; + [ "$method" = DELETE ] || { echo "stub curl: $method on a deployment is the Publish call" >&2; exit 97; } + if [ -f "$CFAKE/drop.$id.code" ]; then respond "$(cat "$CFAKE/drop.$id.code")" "drop failed" + else + case "$(state_of "$id")" in + VALIDATED | FAILED) echo "$id" >> "$CFAKE/dropped.log"; respond 204 "" ;; + *) respond 400 "Can only drop deployments that are in a VALIDATED or FAILED state" ;; + esac + fi ;; *) echo "stub curl: unexpected request: $method $url" >&2; exit 97 ;; esac STUB @@ -735,87 +866,180 @@ TOKEN="$(printf 'user:pass' | base64 | tr -d '\n')" new_central() { CFAKE="$C/fake.$1"; rm -rf "$CFAKE" "$C/state"; mkdir -p "$CFAKE"; : > "$CFAKE/calls.log"; export CFAKE; } central() { # central local log="$1"; shift - run "$log" env PATH="$C/bin:$PATH" CENTRAL_USERNAME=user CENTRAL_PASSWORD=pass CENTRAL_API_URL=https://portal.test/api \ - CENTRAL_REPO_URL=https://repo.test/maven2 CENTRAL_POLL_SECONDS=0 CENTRAL_WAIT_SECONDS="${WAIT_SECONDS:-5}" bash "$here/central-upload.sh" "$@" + run "$log" env PATH="$C/bin:$PATH" CENTRAL_USERNAME=user CENTRAL_PASSWORD=pass GITHUB_RUN_ID=777 GITHUB_RUN_ATTEMPT="${RUN_ATTEMPT:-2}" \ + RELEASE_SCRIPTS_TESTING=1 CENTRAL_POLL_SECONDS=0 CENTRAL_WAIT_SECONDS="${WAIT_SECONDS:-5}" CENTRAL_DROP_WAIT_SECONDS="${DROP_WAIT_SECONDS:-5}" \ + bash "$here/central-upload.sh" "$@" } up() { central "$1" upload "$S/bundles" "$S/staging" "$C/state"; } +drop() { central "$1" drop "$C/state"; } calls() { grep -c -- "$1" "$CFAKE/calls.log" || true; } +dropped() { tr '\n' ' ' < "$CFAKE/dropped.log" 2> /dev/null || true; } ID1=00000000-0000-4000-8000-000000000001; ID2=00000000-0000-4000-8000-000000000002; ID3=00000000-0000-4000-8000-000000000003 +N1=indextables_spark-1.2.3_spark_3.5.8-run777-2; N2=indextables_spark-1.2.3_spark_4.0.3-run777-2; N3=indextables_spark-1.2.3_spark_4.1.2-run777-2 L="$C/log" new_central ok; printf 'PENDING\nVALIDATING\nVALIDATED\n' > "$CFAKE/status.$ID1" check "central: three uploads, each validated" up "$L.1" -check "central: ... three upload calls" is "$(calls 'POST https://portal.test/api/upload')" 3 -check "central: ... as USER_MANAGED, named after the version" has "$CFAKE/calls.log" "POST https://portal.test/api/upload?name=indextables_spark-1.2.3_spark_3.5.8&publishingType=USER_MANAGED" +check "central: ... three upload calls" is "$(calls "POST $PORTAL_API/upload")" 3 +check "central: ... as USER_MANAGED, named after the version, the run and the attempt" has "$CFAKE/calls.log" "POST $PORTAL_API/upload?name=$N1&publishingType=USER_MANAGED" check "central: ... the bundle is the form field" has "$CFAKE/forms.log" "bundle=@$S/bundles/indextables_spark-1.2.3_spark_4.1.2-bundle.zip;type=application/octet-stream" check "central: ... one version at a time: validated before the next upload" is \ - "$(grep -n -E 'upload\?name|status\?id' "$CFAKE/calls.log" | sed -e 's/^[0-9]*:POST https:\/\/portal.test\/api\///' -e 's/[?].*//' | tr '\n' ' ')" "upload status status status upload status upload status " + "$(grep -E 'upload\?name|status\?id' "$CFAKE/calls.log" | sed -e 's/^POST [^ ]*\/publisher\///' -e 's/[?].*//' | tr '\n' ' ')" "upload status status status upload status upload status " check "central: ... nothing is dropped" is "$(calls '^DELETE')" 0 -check "central: ... the Publish call is never made" is "$(calls '^POST https://portal.test/api/deployment/')" 0 -check "central: ... the token goes to the Portal as a Bearer header" is "$(grep -c "portal.test.* auth:$TOKEN\$" "$CFAKE/calls.log")" "$(grep -c 'portal.test' "$CFAKE/calls.log")" -check "central: ... and never to the public repository" is "$(grep 'repo.test' "$CFAKE/calls.log" | grep -c 'auth:')" 0 +check "central: ... the Publish call is never made" is "$(calls "^POST $PORTAL_API/deployment/")" 0 +check "central: ... the token goes to the Portal as a Bearer header" is "$(grep -c "central.sonatype.com.* auth:$TOKEN\$" "$CFAKE/calls.log")" "$(grep -c 'central.sonatype.com' "$CFAKE/calls.log")" +check "central: ... and never to the public repository" is "$(grep 'repo1.maven.org' "$CFAKE/calls.log" | grep -c 'auth:')" 0 check "central: ... and never on a command line" hasnt "$CFAKE/calls.log" "header passed on the command line" +check "central: ... every request went to the Portal API or repo1, whatever CENTRAL_API_URL says" test -z "$(grep -v -E "^[A-Z]+ ($PORTAL_API|$REPO1)/" "$CFAKE/calls.log")" check "central: ... the encoded token is masked before use" has "$L.1" "::add-mask::$TOKEN" check "central: ... the summary says it is not public yet" has "$L.1.summary" "Not public yet" -check "central: ... deployments are recorded" is "$(cat "$C/state/deployments.txt" | tr '\n' ' ')" "1.2.3_spark_3.5.8 $ID1 1.2.3_spark_4.0.3 $ID2 1.2.3_spark_4.1.2 $ID3 " +check "central: ... each deployment id is in the summary as soon as it is known, with its name" has "$L.1.summary" "- Uploaded \`$N2\`: deployment \`$ID2\`" +check "central: ... the final table lists id, name and state" has "$L.1.summary" "| \`1.2.3_spark_4.1.2\` | \`$ID3\` | \`$N3\` | VALIDATED |" +check "central: ... and says to publish only those" has "$L.1.summary" "Publish **only** the deployments whose id is in this table" +check "central: ... deployments are recorded with their names" is "$(tr '\n' ' ' < "$C/state/deployments.txt")" "1.2.3_spark_3.5.8 $ID1 $N1 1.2.3_spark_4.0.3 $ID2 $N2 1.2.3_spark_4.1.2 $ID3 $N3 " check "central: a second upload from the same job is refused" fails_with "refusing to upload twice" "$L.1b" up "$L.1b" -: > "$CFAKE/calls.log" -check "central: drop removes this run's deployments" central "$L.1c" drop "$C/state" -check "central: ... all three" is "$(calls '^DELETE https://portal.test/api/deployment/')" 3 -check "central: ... and a second drop has nothing left to do" central "$L.1d" drop "$C/state" -check "central: ... (no further calls)" is "$(calls '^DELETE')" 3 +check "central: drop (a later step failed) removes this job's deployments" drop "$L.1c" +check "central: ... all three" is "$(dropped)" "$ID1 $ID2 $ID3 " +check "central: ... and a second drop has nothing left to do" drop "$L.1d" +check "central: ... (no further drop)" is "$(calls '^DELETE')" 3 +rm -rf "$C/state" +check "central: drop before the upload step ran has nothing to do" drop "$L.1e" +check "central: ... (no request at all)" is "$(calls '^DELETE')" 3 + +# A deployment left in the Portal by an earlier run, for the same version. +STALE=99999999-0000-4000-8000-999999999999 +new_central stale; echo VALIDATED > "$CFAKE/status.$STALE" +check "central: an earlier run's deployment for the same version does not get in the way" up "$L.s1" +check "central: ... it is never touched" is "$(calls "$STALE")" 0 +check "central: ... this run's deployments carry this run's id and attempt" is "$(grep -c -- '-run777-2&publishingType' "$CFAKE/calls.log")" 3 +check "central: ... the summary tells them apart" has "$L.s1.summary" "with any other id or name comes from a different run: drop it, do not publish it" +check "central: ... dropping after a failure leaves the earlier run's deployment alone" drop "$L.s2" +check "central: ... (dropped: this job's three)" is "$(dropped)" "$ID1 $ID2 $ID3 " +new_central stale2 +RUN_ATTEMPT=3 +check "central: a re-run of the job uploads under another name" up "$L.s3" +unset RUN_ATTEMPT +check "central: ... (attempt 3)" has "$CFAKE/calls.log" "upload?name=indextables_spark-1.2.3_spark_3.5.8-run777-3&" new_central rejected; echo FAILED > "$CFAKE/status.$ID2" check "central: a rejected bundle fails the step" fails_with "Maven Central rejected the bundle for 1.2.3_spark_4.0.3" "$L.2" up "$L.2" check "central: ... the Portal's reasons are shown" has "$L.2" "reason from the Portal" -check "central: ... the third version is not uploaded" is "$(calls 'POST https://portal.test/api/upload')" 2 -check "central: ... both deployments of this run are dropped" is "$(grep '^DELETE' "$CFAKE/calls.log" | sed 's/.*deployment\///' | sed 's/ auth.*//' | tr '\n' ' ')" "$ID1 $ID2 " +check "central: ... the third version is not uploaded" is "$(calls "POST $PORTAL_API/upload")" 2 +check "central: ... both deployments of this job are dropped" is "$(dropped)" "$ID1 $ID2 " check "central: ... nothing is left recorded as waiting" test ! -s "$C/state/deployments.txt" +check "central: ... the last step then has nothing to do" drop "$L.2b" +check "central: ... and raises no alarm" hasnt "$L.2b.summary" "Action needed" new_central refused; echo 401 > "$CFAKE/upload.1.code"; echo "invalid token" > "$CFAKE/upload.1.body" check "central: a refused upload fails the step" fails_with "was refused: HTTP 401 invalid token" "$L.3" up "$L.3" -check "central: ... nothing else is attempted" is "$(calls 'POST https://portal.test/api/upload')$(calls 'status')$(calls '^DELETE')" 100 +check "central: ... nothing else is attempted" is "$(calls "POST $PORTAL_API/upload")$(calls 'status')$(calls '^DELETE')" 100 +check "central: ... nothing is left to drop" drop "$L.3b" new_central broken; echo 28 > "$CFAKE/upload.2.exit" check "central: an upload that does not complete fails the step" fails_with "may or may not have reached the Portal" "$L.4" up "$L.4" -check "central: ... the earlier deployment is dropped" is "$(grep -c "^DELETE https://portal.test/api/deployment/$ID1" "$CFAKE/calls.log")" 1 +check "central: ... the message names the deployment to look for" has "$L.4" "look for a deployment named $N2" +check "central: ... the earlier deployment is dropped" is "$(dropped)" "$ID1 " +check "central: ... the last step fails, because a deployment may exist that has no id" fails_with "was cut off before the Portal answered" "$L.4b" drop "$L.4b" +check "central: ... and puts its name in the summary" has "$L.4b.summary" "| \`1.2.3_spark_4.0.3\` | unknown | \`$N2\` |" new_central garbled; echo "ok" > "$CFAKE/upload.1.body" check "central: an upload answer that is not a deployment id fails the step" fails_with "unexpected response" "$L.5" up "$L.5" +check "central: ... the last step reports the deployment that may exist" fails_with "$N1" "$L.5b" drop "$L.5b" -new_central slow; echo VALIDATING > "$CFAKE/status.$ID1"; echo 400 > "$CFAKE/drop.$ID1.code" +# Validation takes longer than the step waits. The Portal refuses to drop a +# deployment that is still being validated. +new_central slow1; printf 'VALIDATING\nVALIDATING\nVALIDATING\nVALIDATED\n' > "$CFAKE/status.$ID1" WAIT_SECONDS=0 check "central: a validation that does not finish in time fails the step" fails_with "was not validated within 0 seconds" "$L.6" up "$L.6" -unset WAIT_SECONDS -check "central: ... a drop that is refused is reported, with what to do" has "$L.6" "Could not drop deployment $ID1" -check "central: ... 'do not publish it'" has "$L.6" "do not publish it" +check "central: ... the deployment is polled until it can be dropped, then dropped" is "$(dropped)" "$ID1 " +check "central: ... no drop was attempted while it was still validating" is "$(calls '^DELETE')" 1 +check "central: ... nothing is left recorded" test ! -s "$C/state/deployments.txt" +check "central: ... the last step has nothing to do" drop "$L.6b" + +new_central slow2; echo VALIDATING > "$CFAKE/status.$ID1" +DROP_WAIT_SECONDS=0 +check "central: a validation that outlasts the drop wait as well fails the step" fails_with "was not validated within 0 seconds" "$L.7" up "$L.7" +check "central: ... the drop is refused by the Portal" has "$L.7" "Could not drop deployment $ID1 ($N1), state VALIDATING: HTTP 400" +check "central: ... the deployment stays recorded, for the last step" is "$(cat "$C/state/deployments.txt")" "1.2.3_spark_3.5.8 $ID1 $N1" +check "central: ... and the log says the last step tries again" has "$L.7" "the last step of this job tries again" +check "central: the last step, with validation still running, fails" fails_with "Could not drop deployment $ID1" "$L.7b" drop "$L.7b" +check "central: ... the summary says what to drop by hand, by id and name" has "$L.7b.summary" "| \`1.2.3_spark_3.5.8\` | \`$ID1\` | \`$N1\` | VALIDATING; the drop was refused (HTTP 400). **Drop it by hand. Do not publish it.** |" +check "central: ... under a heading that cannot be missed" has "$L.7b.summary" "### Action needed: deployments left in the Central Portal" +check "central: ... and it stays recorded" is "$(cat "$C/state/deployments.txt")" "1.2.3_spark_3.5.8 $ID1 $N1" +echo VALIDATED > "$CFAKE/status.$ID1" +check "central: once validation has finished, the same step drops it" drop "$L.7c" +check "central: ... (dropped)" is "$(dropped)" "$ID1 " +unset DROP_WAIT_SECONDS WAIT_SECONDS + +# The job is cancelled while a deployment is being validated: the upload step +# is killed where it stands, and the last step has to clean up. +new_central cancelled; echo VALIDATING > "$CFAKE/status.$ID1" +env PATH="$C/bin:$PATH" CENTRAL_USERNAME=user CENTRAL_PASSWORD=pass GITHUB_RUN_ID=777 GITHUB_RUN_ATTEMPT=2 RELEASE_SCRIPTS_TESTING=1 \ + CENTRAL_POLL_SECONDS=1 CENTRAL_WAIT_SECONDS=600 bash "$here/central-upload.sh" upload "$S/bundles" "$S/staging" "$C/state" > "$L.8" 2>&1 & +victim=$! +i=0; while [ ! -s "$C/state/deployments.txt" ] && [ "$i" -lt 100 ]; do sleep 0.1; i=$((i + 1)); done +kill -KILL "$victim" 2> /dev/null; wait "$victim" 2> /dev/null +check "central: an upload step killed during validation leaves its deployment recorded" is "$(cat "$C/state/deployments.txt")" "1.2.3_spark_3.5.8 $ID1 $N1" +check "central: ... and nothing dropped yet" is "$(calls '^DELETE')" 0 +printf 'VALIDATING\nVALIDATING\nVALIDATED\n' > "$CFAKE/status.$ID1" +check "central: the last step waits for the validation to finish and drops it" drop "$L.8b" +check "central: ... (dropped)" is "$(dropped)" "$ID1 " +check "central: ... without an alarm" hasnt "$L.8b.summary" "Action needed" + +# The Portal refuses a drop for another reason. +new_central droprefused; echo FAILED > "$CFAKE/status.$ID2"; echo 500 > "$CFAKE/drop.$ID1.code" +check "central: a refused drop does not hide the original failure" fails_with "Maven Central rejected the bundle for 1.2.3_spark_4.0.3" "$L.9" up "$L.9" +check "central: ... the other deployment is still dropped" is "$(dropped)" "$ID2 " +check "central: ... the refused one stays recorded" is "$(cat "$C/state/deployments.txt")" "1.2.3_spark_3.5.8 $ID1 $N1" +check "central: ... the last step tries again and, refused again, fails" fails_with "Could not drop deployment $ID1" "$L.9b" drop "$L.9b" +check "central: ... listing only that one" is "$(grep -c '^| `1.2.3' "$L.9b.summary")" 1 +rm "$CFAKE/drop.$ID1.code" +check "central: ... and succeeds once the Portal accepts the drop" drop "$L.9c" + +# Someone publishes a deployment by hand while the job is failing. +new_central raced2; printf 'VALIDATED\nPUBLISHED\n' > "$CFAKE/status.$ID1"; echo FAILED > "$CFAKE/status.$ID2" +check "central: a deployment published by hand before the job failed cannot be dropped" fails_with "is PUBLISHED and can no longer be dropped" "$L.10" up "$L.10" +check "central: ... no drop is attempted for it" is "$(calls "^DELETE $PORTAL_API/deployment/$ID1")" 0 +check "central: ... the last step says so in the summary" fails_with "can no longer be dropped" "$L.10b" drop "$L.10b" +check "central: ... (the row)" has "$L.10b.summary" "| \`1.2.3_spark_3.5.8\` | \`$ID1\` | \`$N1\` | PUBLISHED: it can no longer be dropped." new_central flaky; printf 'HTTP502\nHTTP502\nVALIDATED\n' > "$CFAKE/status.$ID1" -check "central: a few failed status calls are tolerated" up "$L.7" +check "central: a few failed status calls are tolerated" up "$L.11" new_central down; echo HTTP503 > "$CFAKE/status.$ID1" -check "central: status calls that keep failing fail the step" fails_with "could not be read six times in a row" "$L.8" up "$L.8" +check "central: status calls that keep failing fail the step" fails_with "could not be read six times in a row" "$L.12" up "$L.12" new_central raced; echo PUBLISHED > "$CFAKE/status.$ID1" -check "central: a deployment published by hand during the run is not an error" up "$L.9" -check "central: ... but is called out" has "$L.9" "someone pressed Publish" +check "central: a deployment published by hand during the run is not an error" up "$L.13" +check "central: ... but is called out" has "$L.13" "someone pressed Publish" # Re-run after a partial publish: 3.5.8 is already public with these files. vd="$S/staging/bundles/1.2.3_spark_3.5.8/io/indextables/indextables_spark/1.2.3_spark_3.5.8" same_on_central() { echo 200 > "$CFAKE/repo.1.2.3_spark_3.5.8.code"; for n in $(artifact_names 1.2.3_spark_3.5.8); do hash_of sha1 "$vd/$n" > "$CFAKE/repo.$n.sha1"; done; } new_central partial; same_on_central -check "central: a version already public with identical files is skipped" up "$L.10" -check "central: ... only the other two are uploaded" is "$(grep 'upload?name' "$CFAKE/calls.log" | sed -e 's/.*name=indextables_spark-//' -e 's/&.*//' | tr '\n' ' ')" "1.2.3_spark_4.0.3 1.2.3_spark_4.1.2 " -check "central: ... and the summary says so" has "$L.10.summary" "ALREADY_PUBLISHED" +check "central: a version already public with identical files is skipped" up "$L.14" +check "central: ... only the other two are uploaded" is "$(grep 'upload?name' "$CFAKE/calls.log" | sed -e 's/.*name=indextables_spark-//' -e 's/-run.*//' | tr '\n' ' ')" "1.2.3_spark_4.0.3 1.2.3_spark_4.1.2 " +check "central: ... and the summary says so" has "$L.14.summary" "ALREADY_PUBLISHED" new_central differs; same_on_central; echo 0000000000000000000000000000000000000000 > "$CFAKE/repo.indextables_spark-1.2.3_spark_3.5.8-javadoc.jar.sha1" -check "central: a version already public with other files stops the release" fails_with "cannot be replaced" "$L.11" up "$L.11" -check "central: ... before anything is uploaded" is "$(calls 'portal.test')" 0 +check "central: a version already public with other files stops the release" fails_with "cannot be replaced" "$L.15" up "$L.15" +check "central: ... before anything is uploaded" is "$(calls 'central.sonatype.com')" 0 new_central unknown; echo 503 > "$CFAKE/repo.1.2.3_spark_4.1.2.code" -check "central: an unreadable public repository stops the release" fails_with "could not tell whether 1.2.3_spark_4.1.2 is already on Maven Central" "$L.12" up "$L.12" -check "central: ... before anything is uploaded" is "$(calls 'portal.test')" 0 +check "central: an unreadable public repository stops the release" fails_with "could not tell whether 1.2.3_spark_4.1.2 is already on Maven Central" "$L.16" up "$L.16" +check "central: ... before anything is uploaded" is "$(calls 'central.sonatype.com')" 0 new_central nocreds -check "central: missing credentials fail before any upload" bash -c "! PATH='$C/bin:$PATH' CENTRAL_USERNAME= CENTRAL_PASSWORD= CENTRAL_API_URL=https://portal.test/api CENTRAL_REPO_URL=https://repo.test/maven2 bash '$here/central-upload.sh' upload '$S/bundles' '$S/staging' '$C/state'" -check "central: ... (no Portal call)" is "$(calls 'portal.test')" 0 -check "central: the default endpoints are the Portal API and repo1" test -n "$(grep -F 'CENTRAL_API_URL:-https://central.sonatype.com/api/v1/publisher}' "$here/central-upload.sh")" -a -n "$(grep -F 'CENTRAL_REPO_URL:-https://repo1.maven.org/maven2}' "$here/central-upload.sh")" +check "central: missing credentials fail before any upload" bash -c "! PATH='$C/bin:$PATH' CENTRAL_USERNAME= CENTRAL_PASSWORD= bash '$here/central-upload.sh' upload '$S/bundles' '$S/staging' '$C/state'" +check "central: ... (no request)" is "$(grep -c . "$CFAKE/calls.log")" 0 +new_central knob +check "central: a timing setting without the test flag stops the script" fails_with "CENTRAL_POLL_SECONDS is a test-only setting" "$L.17" \ + env PATH="$C/bin:$PATH" CENTRAL_USERNAME=user CENTRAL_PASSWORD=pass CENTRAL_POLL_SECONDS=0 bash "$here/central-upload.sh" upload "$S/bundles" "$S/staging" "$C/state" +check "central: ... before any request" is "$(grep -c . "$CFAKE/calls.log")" 0 +check "central: ... the same for the drop wait" fails_with "CENTRAL_DROP_WAIT_SECONDS is a test-only setting" "$L.18" \ + env PATH="$C/bin:$PATH" CENTRAL_USERNAME=user CENTRAL_PASSWORD=pass CENTRAL_DROP_WAIT_SECONDS=0 bash "$here/central-upload.sh" drop "$C/state" +check "central: the endpoints are constants" test -n "$(grep -x 'CENTRAL_API=https://central.sonatype.com/api/v1/publisher' "$here/lib.sh")" -a -n "$(grep -x 'CENTRAL_REPO=https://repo1.maven.org/maven2' "$here/lib.sh")" +check "central: no variable redirects them" test -z "$(grep -l -E 'CENTRAL_API_URL|CENTRAL_REPO_URL' "$here"/*.sh | grep -v test.sh)" +# shellcheck disable=SC2086 +check "central: the only settings a test may change are the three timings" is \ + "$(grep -h -o 'test_setting [A-Z_][A-Z_]*' $SCRIPTS_UNDER_TEST | sort -u | tr '\n' ' ')" \ + "test_setting CENTRAL_DROP_WAIT_SECONDS test_setting CENTRAL_POLL_SECONDS test_setting CENTRAL_WAIT_SECONDS " check "central: AUTOMATIC publishing appears nowhere" test -z "$(grep -l 'AUTOMATIC' "$here"/*.sh | grep -v test.sh)" # --------------------------------------------------------------------------- @@ -824,20 +1048,42 @@ check "central: AUTOMATIC publishing appears nowhere" test -z "$(grep -l 'AUTOMA G="$T/gh"; mkdir -p "$G/bin" cat > "$G/bin/gh" << 'STUB' #!/usr/bin/env bash -# Stub gh: tag lookups and release view/create/upload/edit from $GFAKE. +# Stub gh: tag lookups and a release kept in $GFAKE: body.md (its notes; the +# release exists when this file does), assets.json (name and digest of each +# attached file) and files/ (their content, for downloads). set -u echo "gh $*" >> "$GFAKE/calls.log" +h() { if command -v sha256sum > /dev/null 2>&1; then sha256sum "$1" | cut -d' ' -f1; else shasum -a 256 "$1" | cut -d' ' -f1; fi; } +attach() { # attach ...: record the files as assets, replacing same names + local f name + [ -f "$GFAKE/assets.json" ] || echo '[]' > "$GFAKE/assets.json" + mkdir -p "$GFAKE/files" + for f in "$@"; do + case "$f" in -*) continue ;; esac + [ -f "$f" ] || continue + name="$(basename "$f")" + cp "$f" "$GFAKE/files/$name" + jq --arg n "$name" --arg d "sha256:$(h "$f")" '[.[] | select(.name != $n)] + [{name: $n, digest: $d}]' "$GFAKE/assets.json" > "$GFAKE/assets.tmp" \ + && mv "$GFAKE/assets.tmp" "$GFAKE/assets.json" + done +} case "$1 ${2:-}" in "api repos/o/r/git/ref/tags/"*) [ -f "$GFAKE/ref.json" ] || exit 1; cat "$GFAKE/ref.json" ;; "api repos/o/r/git/tags/"*) cat "$GFAKE/tagobj.sha" ;; "release view") + if [ ! -f "$GFAKE/body.md" ]; then cat "$GFAKE/view.err" >&2; exit 1; fi case "$*" in - *"--json body"*) if [ -f "$GFAKE/body.md" ]; then cat "$GFAKE/body.md"; else cat "$GFAKE/view.err" >&2; exit 1; fi ;; + *"--json body,assets"*) + [ -f "$GFAKE/assets.json" ] || echo '[]' > "$GFAKE/assets.json" + jq -n --rawfile body "$GFAKE/body.md" --slurpfile assets "$GFAKE/assets.json" '{body: $body, assets: $assets[0]}' ;; + *"--json body"*) cat "$GFAKE/body.md" ;; *"--json url"*) echo "https://example.invalid/releases/$3" ;; esac ;; - "release upload") ;; + "release upload") shift 3; attach "$@" ;; + "release download") # release download TAG --pattern NAME --dir DIR + cp "$GFAKE/files/$5" "$7/$5" ;; "release edit") shift 3; [ "$1" = --notes-file ] && cp "$2" "$GFAKE/body.md" ;; - "release create") printf "## What's Changed\n* a change\n" > "$GFAKE/body.md" ;; + "release create") printf "## What's Changed\n* a change\n" > "$GFAKE/body.md"; shift 3; attach "$@" ;; *) echo "stub gh: unexpected call: $*" >&2; exit 97 ;; esac STUB @@ -848,6 +1094,15 @@ release() { # release [VAR=value...] local log="$1"; shift run "$log" env PATH="$G/bin:$PATH" GH_TOKEN=t GH_REPO=o/r TAG=v1.2.3 COMMIT="$SHA_D" PRERELEASE=false "$@" bash "$here/github-release.sh" "$A/staging.good" } +uploads() { grep -c 'gh release upload' "$GFAKE/calls.log" || true; } +J35=indextables_spark-1.2.3_spark_3.5.8-linux-x86_64-shaded.jar +staged35="$A/staging.good/bundles/1.2.3_spark_3.5.8/io/indextables/indextables_spark/1.2.3_spark_3.5.8/$J35" +# An earlier build of the same jar: same name, other bytes. +mkdir -p "$G/old" && cp "$T/fx/other-native.jar" "$G/old/$J35" +old_release() { # a published release that already carries an earlier build of the 3.5 jar + new_gh "$1"; printf 'Published notes.\n' > "$GFAKE/body.md" + PATH="$G/bin:$PATH" gh release upload v1.2.3 "$G/old/$J35"; : > "$GFAKE/calls.log" +} L="$G/log" new_gh create check "release: creates the release when there is none" release "$L.1" @@ -861,14 +1116,48 @@ new_gh pre check "release: a pre-release tag creates a pre-release" release "$L.2" PRERELEASE=true check "release: ... flag passed" has "$GFAKE/calls.log" "--prerelease" new_gh existing; printf 'Hand-written notes.\r\n\r\n- item\r\n\r\n' > "$GFAKE/body.md" -check "release: an existing release gets the jars" release "$L.3" -check "release: ... replacing assets of the same name" has "$GFAKE/calls.log" "gh release upload v1.2.3 --clobber" +check "release: an existing release without jars gets them" release "$L.3" +check "release: ... in one upload that replaces nothing" is "$(grep 'gh release upload' "$GFAKE/calls.log" | grep -c -v -- '--clobber')" 1 +check "release: ... all three" is "$(jq -r 'length' "$GFAKE/assets.json")" 3 check "release: ... it is not created again" test -z "$(grep 'release create' "$GFAKE/calls.log")" check "release: ... its notes are kept" has "$GFAKE/body.md" "Hand-written notes." check "release: ... and gain the build inputs" has "$GFAKE/body.md" "$T4J_COMMIT" -release "$L.4" > /dev/null 2>&1 -check "release: a re-run replaces the build inputs instead of adding a second block" is "$(grep -c 'release-build-inputs:start' "$GFAKE/body.md")" 1 +: > "$GFAKE/calls.log" +check "release: a re-run with the same staged jars" release "$L.4" +check "release: ... uploads nothing, the jars are identical" is "$(uploads)" 0 +check "release: ... and says so" has "$L.4" "$J35 is already attached with identical content." +check "release: ... replaces the build inputs instead of adding a second block" is "$(grep -c 'release-build-inputs:start' "$GFAKE/body.md")" 1 check "release: ... and keeps the notes" is "$(grep -c 'Hand-written notes.' "$GFAKE/body.md")" 1 + +# The release already has one of the jars from an earlier build (the +# skip-central case: nothing on the Maven Central side stops the run). +old_release differs +check "release: a jar already attached with other content stops the step" fails_with "already has 1 of these jars with different content" "$L.10" release "$L.10" +check "release: ... it names the jar" has "$L.10" "$J35 is already attached with different content" +check "release: ... and the input that would allow it" has "$L.10" "replace-release-assets" +check "release: ... nothing was uploaded, not even the jars that were missing" is "$(uploads)" 0 +check "release: ... the attached jar is still the earlier one" is "$(jq -r --arg n "$J35" '.[] | select(.name == $n) | .digest' "$GFAKE/assets.json")" "sha256:$(hash_of sha256 "$G/old/$J35")" +check "release: ... the notes were not touched" is "$(cat "$GFAKE/body.md")" "Published notes." +for v in "" false TRUE 1 yes; do + old_release "differs-$v" + check "release: replace-release-assets '$v' does not allow it either" fails_with "with different content" "$L.11" release "$L.11" REPLACE_ASSETS="$v" +done +old_release replace +check "release: with replace-release-assets the jar is replaced" release "$L.12" REPLACE_ASSETS=true +check "release: ... with a warning" has "$L.12" "::warning::Replacing 1 jar(s) on release v1.2.3" +check "release: ... only that jar is uploaded with --clobber" is "$(grep -- '--clobber' "$GFAKE/calls.log" | tr ' ' '\n' | grep -c 'shaded.jar$')" 1 +check "release: ... (it is the 3.5 jar)" test -n "$(grep -- '--clobber' "$GFAKE/calls.log" | grep -F "$J35")" +check "release: ... the other two are uploaded without it" is "$(grep 'gh release upload' "$GFAKE/calls.log" | grep -v -- '--clobber' | tr ' ' '\n' | grep -c 'shaded.jar$')" 2 +check "release: ... the attached jar is now the staged one" is "$(jq -r --arg n "$J35" '.[] | select(.name == $n) | .digest' "$GFAKE/assets.json")" "sha256:$(hash_of sha256 "$staged35")" +# An asset without a recorded digest is downloaded and hashed. +old_release nodigest; jq 'map(del(.digest))' "$GFAKE/assets.json" > "$GFAKE/a.tmp" && mv "$GFAKE/a.tmp" "$GFAKE/assets.json" +check "release: a differing jar without a recorded digest is still caught" fails_with "with different content" "$L.13" release "$L.13" +check "release: ... by downloading it" has "$GFAKE/calls.log" "gh release download v1.2.3 --pattern $J35" +new_gh nodigest2; printf 'Notes.\n' > "$GFAKE/body.md" +PATH="$G/bin:$PATH" gh release upload v1.2.3 "$staged35"; jq 'map(del(.digest))' "$GFAKE/assets.json" > "$GFAKE/a.tmp" && mv "$GFAKE/a.tmp" "$GFAKE/assets.json"; : > "$GFAKE/calls.log" +check "release: an identical jar without a recorded digest is recognised" release "$L.14" +check "release: ... and not uploaded again" test -z "$(grep 'gh release upload' "$GFAKE/calls.log" | grep -F "$J35")" + new_gh moved; printf '{"object":{"type":"commit","sha":"%s"}}\n' "$SHA_A" > "$GFAKE/ref.json" check "release: a tag that moved since the build is not released" fails_with "now points to $SHA_A" "$L.5" release "$L.5" check "release: ... no release call is made" test -z "$(grep 'gh release' "$GFAKE/calls.log")" @@ -894,29 +1183,30 @@ chain() { # chain : run in $E with the stubs on PATH (cd "$E" && GITHUB_OUTPUT="$log.out" GITHUB_STEP_SUMMARY="$log.summary" PATH="$C/bin:$G/bin:$T/bin:${SIGNPATH}" "$@") > "$log" 2>&1 } L="$E/log" -check "chain: native resolve" chain "$L.1" env T4J_REPO="file://$N/t4j" POM=src/pom.xml PINS="$ES/native-pins.txt" WORK=work/native OUT=out/native M2_REPO="$E/m2-native" bash "$ES/build-native.sh" resolve +check "chain: native resolve" chain "$L.1" env HOME="$E/home-native" GIT_CONFIG_COUNT=1 "$(redirect "$N/t4j")" GIT_CONFIG_VALUE_0=https://github.com/indextables/tantivy4java.git POM=src/pom.xml PINS="$ES/native-pins.txt" WORK=work/native OUT=out/native bash "$ES/build-native.sh" resolve mkdir -p "$E/work/native/protoc/bin" && cp "$N/work/protoc/bin/protoc" "$E/work/native/protoc/bin/protoc" 2> /dev/null \ || { printf '#!/bin/sh\necho "libprotoc 25.5"\n' > "$E/work/native/protoc/bin/protoc"; chmod +x "$E/work/native/protoc/bin/protoc"; } -check "chain: native build" chain "$L.2" env T4J_REPO="file://$N/t4j" POM=src/pom.xml PINS="$ES/native-pins.txt" WORK=work/native OUT=out/native M2_REPO="$E/m2-native" bash "$ES/build-native.sh" build +check "chain: native build" chain "$L.2" env HOME="$E/home-native" POM=src/pom.xml PINS="$ES/native-pins.txt" WORK=work/native OUT=out/native bash "$ES/build-native.sh" build mkdir -p "$E/in" && cp -R "$E/out/native" "$E/in/native" for profile in $PROFILES; do rm -rf "$E/src/target" "$E/src/.stamped" "$E/src/dependency-reduced-pom.xml" - check "chain: build $profile" chain "$L.3" env SRC=src PROFILE="$profile" BASE_VERSION=1.2.3 NATIVE_DIR=in/native OUT=out/artifacts M2_REPO="$E/m2-$profile" bash "$ES/build-artifacts.sh" + check "chain: build $profile" chain "$L.3" env SRC=src PROFILE="$profile" BASE_VERSION=1.2.3 EXPECTED_VERSIONS="$EV" NATIVE_DIR=in/native OUT=out/artifacts HOME="$E/home-$profile" bash "$ES/build-artifacts.sh" cp -R "$E/out/artifacts" "$E/in/artifacts-$profile" done -check "chain: assemble" chain "$L.4" env IN=in OUT=staging TAG=v1.2.3 COMMIT="$SHA_D" BASE_VERSION=1.2.3 bash "$ES/assemble.sh" +check "chain: assemble" chain "$L.4" env IN=in OUT=staging TAG=v1.2.3 COMMIT="$SHA_D" BASE_VERSION=1.2.3 EXPECTED_VERSIONS="$EV" bash "$ES/assemble.sh" CM="$(out "$L.4" manifest_sha256)" -check "chain: verify" chain "$L.5" env MANIFEST_SHA256="$CM" BASE_VERSION=1.2.3 bash "$ES/verify-staging.sh" staging +check "chain: verify" chain "$L.5" env MANIFEST_SHA256="$CM" BASE_VERSION=1.2.3 EXPECTED_VERSIONS="$EV" bash "$ES/verify-staging.sh" staging check "chain: generate a key" chain "$L.6" env GNUPGHOME="$KEYHOME" GPG_PASSPHRASE=rehearsal bash "$ES/signing-key.sh" generate CK="$(out "$L.6" fingerprint)" check "chain: sign" chain "$L.7" env GNUPGHOME="$KEYHOME" SIGNING_KEY="$CK" GPG_PASSPHRASE=rehearsal bash "$ES/sign-bundles.sh" staging bundles check "chain: check" chain "$L.8" env GNUPGHOME="$KEYHOME" SIGNING_KEY="$CK" BASE_VERSION=1.2.3 bash "$ES/check-bundles.sh" bundles staging check "chain: remove the key" chain "$L.9" env GNUPGHOME="$KEYHOME" bash "$ES/signing-key.sh" remove new_central chain; rm -rf "$E/state" -check "chain: upload" chain "$L.10" env CENTRAL_USERNAME=user CENTRAL_PASSWORD=pass CENTRAL_API_URL=https://portal.test/api CENTRAL_REPO_URL=https://repo.test/maven2 \ - CENTRAL_POLL_SECONDS=0 bash "$ES/central-upload.sh" upload bundles staging "$E/state" -check "chain: ... three bundles went up" is "$(calls 'POST https://portal.test/api/upload')" 3 +check "chain: upload" chain "$L.10" env CENTRAL_USERNAME=user CENTRAL_PASSWORD=pass GITHUB_RUN_ID=4242 GITHUB_RUN_ATTEMPT=1 RELEASE_SCRIPTS_TESTING=1 CENTRAL_POLL_SECONDS=0 \ + bash "$ES/central-upload.sh" upload bundles staging "$E/state" +check "chain: ... three bundles went up" is "$(calls "POST $PORTAL_API/upload")" 3 check "chain: ... as form uploads of the relative bundle paths" has "$CFAKE/forms.log" "bundle=@bundles/indextables_spark-1.2.3_spark_4.0.3-bundle.zip;type=application/octet-stream" +check "chain: ... named after the runner's run id and attempt" has "$CFAKE/calls.log" "upload?name=indextables_spark-1.2.3_spark_4.0.3-run4242-1&" new_gh chain check "chain: release" chain "$L.11" env GH_TOKEN=t GH_REPO=o/r TAG=v1.2.3 COMMIT="$SHA_D" PRERELEASE=false bash "$ES/github-release.sh" staging check "chain: ... with the staged shaded jars" is "$(grep 'release create' "$GFAKE/calls.log" | tr ' ' '\n' | grep -c '^staging/bundles/.*-linux-x86_64-shaded.jar$')" 3 @@ -965,7 +1255,17 @@ if command -v ruby > /dev/null 2>&1 && ruby -ryaml -rjson -e 'puts JSON.generate "$(wf '[.jobs.rehearse.steps[] | .run? // empty | select(test("verify-staging|sign-bundles|check-bundles"))] | join("|")')" \ "$(wf '[.jobs.publish.steps[] | .run? // empty | select(test("verify-staging|sign-bundles|check-bundles"))] | join("|")')" check "workflow: both take the manifest digest from the assemble job's output" is "$(wf '[.jobs.rehearse, .jobs.publish | .steps[] | select(.name == "Verify staging directory") | .env.MANIFEST_SHA256] | unique | join(",")')" '${{ needs.assemble.outputs.manifest-sha256 }}' - check "workflow: plan gets the inputs as strings, through the environment" is "$(wf '.jobs.plan.steps[] | select(.id == "plan") | .env | [.INPUT_TAG, .INPUT_DRY_RUN, .EVENT_NAME, .REF] | join(" ")')" '${{ inputs.tag }} ${{ inputs.dry-run }} ${{ github.event_name }} ${{ github.ref }}' + check "workflow: plan gets the inputs as strings, through the environment" is "$(wf '.jobs.plan.steps[] | select(.id == "plan") | .env | [.INPUT_TAG, .INPUT_DRY_RUN, .INPUT_EXPECTED_SPARK, .EVENT_NAME, .REF] | join(" ")')" '${{ inputs.tag }} ${{ inputs.dry-run }} ${{ inputs.expected-spark-versions }} ${{ github.event_name }} ${{ github.ref }}' + check "workflow: inputs" is "$(wf '(.on // .["true"]).workflow_dispatch.inputs | keys | join(",")')" dry-run,expected-spark-versions,replace-release-assets,skip-central,tag + check "workflow: expected-spark-versions is optional and empty by default" is "$(wf '(.on // .["true"]).workflow_dispatch.inputs["expected-spark-versions"] | [.type, (.required // false | tostring), .default] | join(",")')" string,false, + check "workflow: replace-release-assets is a boolean that defaults to false" is "$(wf '(.on // .["true"]).workflow_dispatch.inputs["replace-release-assets"] | [.type, (.default | tostring)] | join(",")')" boolean,false + check "workflow: ... and reaches the GitHub Release step only" is "$(wf '[.jobs[].steps[] | select(tostring | test("replace-release-assets")) | .name + ":" + .env.REPLACE_ASSETS] | join(",")')" 'GitHub Release:${{ inputs.replace-release-assets }}' + check "workflow: the plan publishes the versions it worked out" is "$(wf '.jobs.plan.outputs.versions')" '${{ steps.plan.outputs.versions }}' + check "workflow: build, assemble, rehearse and publish are all held to those versions" is "$(wf '[.jobs | to_entries[] | select(tostring | test("EXPECTED_VERSIONS")) | .key] | sort | join(",")')" assemble,build,publish,rehearse + check "workflow: ... taken from the plan job and nowhere else" is "$(wf '[.. | objects | .EXPECTED_VERSIONS? // empty] | unique | join(",")')" '${{ needs.plan.outputs.versions }}' + check "workflow: no test-only setting and no redirection is ever set" test -z "$(grep -n -E 'RELEASE_SCRIPTS_TESTING|CENTRAL_(POLL|WAIT|DROP_WAIT)_SECONDS|CENTRAL_API_URL|CENTRAL_REPO_URL|T4J_REPO|M2_REPO|GIT_CONFIG' "$workflow")" + check "workflow: the last step runs after a failure and after a cancellation" is "$(wf '.jobs.publish.steps[-1] | [.name, .if] | join("|")')" "Drop Maven Central deployments after a failure|\${{ (failure() || cancelled()) && needs.plan.outputs.central == 'true' }}" + check "workflow: header does not claim more than this file can guarantee" test -n "$(grep -F '# * In this file, `publish` is the only job that names the `release`' "$workflow")" -a -n "$(grep -F 'every workflow on' "$workflow")" -a -n "$(grep -F 'v0.6.0-rc1 and v0.6.0-rc2 carry the previous release.yml' "$workflow")" check "workflow: no continue-on-error anywhere" is "$(wf '[.. | objects | select(has("continue-on-error"))] | length')" 0 # Every `run:` step, as the runner would start it (bash -eo pipefail). # Steps that call a release script must name one that exists; the other diff --git a/.github/scripts/release/verify-staging.sh b/.github/scripts/release/verify-staging.sh index a07696b6..88b926a3 100644 --- a/.github/scripts/release/verify-staging.sh +++ b/.github/scripts/release/verify-staging.sh @@ -6,6 +6,7 @@ # Environment: # MANIFEST_SHA256 the digest the assemble job reported as a job output # BASE_VERSION the tag without its "v" +# EXPECTED_VERSIONS the versions plan.sh announced, one per profile # # The staging directory travels between jobs as a workflow artifact. Its # manifest digest travels separately, as a job output, so the two can be @@ -18,7 +19,7 @@ here="$(cd "$(dirname "$0")" && pwd)" . "$here/lib.sh" staging="${1:?usage: verify-staging.sh }" -: "${MANIFEST_SHA256:?}" "${BASE_VERSION:?}" +: "${MANIFEST_SHA256:?}" "${BASE_VERSION:?}" "${EXPECTED_VERSIONS:?}" [[ "$MANIFEST_SHA256" =~ ^[0-9a-f]{64}$ ]] || die "MANIFEST_SHA256 is not a SHA-256 digest" [ -f "$staging/MANIFEST.sha256" ] || die "$staging has no MANIFEST.sha256" @@ -38,6 +39,8 @@ while IFS= read -r version; do shift done < "$staging/versions.txt" [ $# -eq 0 ] || die "the staging directory has no version for: $*" +[ "$(tr '\n' ' ' < "$staging/versions.txt" | sed 's/ $//')" = "$EXPECTED_VERSIONS" ] \ + || die "the staged versions are not the versions the plan announced ($EXPECTED_VERSIONS)" expected="$( { diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 2ad93c0a..db1dec3f 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -8,14 +8,19 @@ name: Release # (pom, jar, sources, javadoc, linux-x86_64-shaded, jar-with-dependencies) # - the linux-x86_64-shaded jars attached to the GitHub Release for the tag # is the spark.version that the profile sets in pom.xml at the tag. +# It changes when the Spark dependency is bumped. The Plan summary shows the +# exact versions next to the ones last published, and a publish run whose +# Spark versions differ from the last release stops in `plan` until the +# releaser has typed them into `expected-spark-versions`. # # How to run it # ------------- # Actions -> Release -> Run workflow, on `main`, with the tag to release. # `dry-run` is ticked by default: the run then builds, assembles and rehearses # signing, and stops. Untick it to publish. Pushing a tag does not start a -# release. The runbook, including the first-release checklist and how to -# abort, is in .github/scripts/release/README.md. +# release. The runbook, including how to adopt this workflow, the release +# candidate rehearsal and how to abort, is in +# .github/scripts/release/README.md. # # Trust model # ----------- @@ -29,20 +34,35 @@ name: Release # started on, on files the build jobs produced. `assemble` treats those # files as untrusted: it accepts only the expected coordinates and file # names and writes a manifest of digests. -# * `publish` is the only job that names the `release` environment, reads a -# secret or holds a token that can write. Beside the signing key and the -# Central token it runs: the pinned checkout and download-artifact actions, -# the release scripts, and gpg, zip, unzip, curl, jq and gh from the runner -# image. No Maven, no JVM, nothing from the tagged commit. It signs and -# uploads exactly the files in the manifest whose digest `assemble` -# reported, and nothing else. +# * In this file, `publish` is the only job that names the `release` +# environment, reads a secret or holds a token that can write. Beside the +# signing key and the Central token it runs: the pinned checkout and +# download-artifact actions, the release scripts, and gpg, zip, unzip, curl, +# jq and gh from the runner image. No Maven, no JVM, nothing from the tagged +# commit. It signs and uploads exactly the files in the manifest whose +# digest `assemble` reported, and nothing else. # * A dry run never starts `publish`, so it requests no approval and reads no # secret. `rehearse` runs the same verification, signing and bundle checks # as `publish`, with a throwaway key it generates and deletes. # -# What this depends on outside this file: the `release` environment must -# admit only `main` and require a reviewer, and the four publishing secrets -# must exist in that environment only. See the README. +# What this file cannot guarantee +# ------------------------------ +# All of the above describes this file on `main`. It says nothing about other +# workflow files or other refs, and two things outside it decide whether the +# separation protects anything: +# +# * While GPG_PRIVATE_KEY, GPG_PASSPHRASE, CENTRAL_USERNAME and +# CENTRAL_PASSWORD exist as repository-level secrets, every workflow on +# every ref can read them, whatever this file does. +# * The tags v0.6.0-rc1 and v0.6.0-rc2 carry the previous release.yml, which +# has a workflow_dispatch trigger. It can be dispatched on the tag ref and +# then runs the old build beside the key. Only an environment that admits +# `main` alone, holding the only copy of the secrets, closes that. +# +# The order in which to get there (secrets into the environment, environment +# narrowed to `main` with administrator bypass off, dry run, release candidate +# rehearsal, repository-level copies deleted) is in the README under +# "Adopting this workflow". on: workflow_dispatch: @@ -55,10 +75,18 @@ on: description: 'Dry run: build, assemble and rehearse signing; upload nothing. Untick to publish.' type: boolean default: true + expected-spark-versions: + description: 'Spark versions expected in the artifact names, e.g. "3.5.9 4.0.4 4.1.3". Needed to publish when they differ from the last release; the dry run prints the value.' + type: string + default: '' skip-central: description: 'Skip Maven Central (GitHub Release only). No effect in a dry run.' type: boolean default: false + replace-release-assets: + description: 'Replace jars already attached to the GitHub Release when their content differs. Unticked: stop instead.' + type: boolean + default: false # To start a release from a pushed tag as well, add: # # push: @@ -79,8 +107,9 @@ env: jobs: # --------------------------------------------------------------------------- - # Validate the request and resolve the tag to a commit. Every later job uses - # that commit, so a tag that moves during the run changes nothing. + # Validate the request, resolve the tag to a commit and work out the versions + # to publish. Every later job uses that commit and must arrive at those + # versions, so a tag that moves during the run changes nothing. # --------------------------------------------------------------------------- plan: name: Plan @@ -93,6 +122,7 @@ jobs: tag: ${{ steps.plan.outputs.tag }} commit: ${{ steps.plan.outputs.commit }} base: ${{ steps.plan.outputs.base }} + versions: ${{ steps.plan.outputs.versions }} prerelease: ${{ steps.plan.outputs.prerelease }} central: ${{ steps.plan.outputs.central }} steps: @@ -113,6 +143,7 @@ jobs: DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} INPUT_TAG: ${{ inputs.tag }} INPUT_DRY_RUN: ${{ inputs.dry-run }} + INPUT_EXPECTED_SPARK: ${{ inputs.expected-spark-versions }} INPUT_SKIP_CENTRAL: ${{ inputs.skip-central }} run: bash .github/scripts/release/plan.sh @@ -221,6 +252,7 @@ jobs: SRC: src PROFILE: ${{ matrix.profile }} BASE_VERSION: ${{ needs.plan.outputs.base }} + EXPECTED_VERSIONS: ${{ needs.plan.outputs.versions }} NATIVE_DIR: in/native OUT: out/artifacts run: bash "$SCRIPTS/build-artifacts.sh" @@ -279,6 +311,7 @@ jobs: TAG: ${{ needs.plan.outputs.tag }} COMMIT: ${{ needs.plan.outputs.commit }} BASE_VERSION: ${{ needs.plan.outputs.base }} + EXPECTED_VERSIONS: ${{ needs.plan.outputs.versions }} RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} run: bash "$SCRIPTS/assemble.sh" @@ -306,6 +339,7 @@ jobs: contents: read env: BASE_VERSION: ${{ needs.plan.outputs.base }} + EXPECTED_VERSIONS: ${{ needs.plan.outputs.versions }} # Protects the throwaway key only; it is not a secret. GPG_PASSPHRASE: rehearsal steps: @@ -392,6 +426,7 @@ jobs: contents: write env: BASE_VERSION: ${{ needs.plan.outputs.base }} + EXPECTED_VERSIONS: ${{ needs.plan.outputs.versions }} steps: - name: Checkout release scripts (workflow commit) uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 @@ -456,11 +491,14 @@ jobs: TAG: ${{ needs.plan.outputs.tag }} COMMIT: ${{ needs.plan.outputs.commit }} PRERELEASE: ${{ needs.plan.outputs.prerelease }} + REPLACE_ASSETS: ${{ inputs.replace-release-assets }} run: bash "$SCRIPTS/github-release.sh" staging - # If the job stops after bundles were uploaded (the GitHub Release step - # failed, or the run was cancelled), do not leave validated deployments - # waiting in the Portal: a re-run uploads them again. + # If the job stops after bundles were uploaded (a step failed, or the run + # was cancelled, also in the middle of the upload step), do not leave + # deployments waiting in the Portal: a re-run uploads them again. A + # deployment that is still being validated is polled until it can be + # dropped; what cannot be dropped is listed in the job summary. - name: Drop Maven Central deployments after a failure if: ${{ (failure() || cancelled()) && needs.plan.outputs.central == 'true' }} env: From 1f40d68dd6ab55d1e9572702f480189bec928ffc Mon Sep 17 00:00:00 2001 From: Brenton Strickler Date: Thu, 22 Oct 2026 11:30:26 -0400 Subject: [PATCH 3/3] ci(release): let keyring removal succeed when no keyring was set up "Remove the signing key" runs in an always() step. When an earlier step failed before GNUPGHOME was set, it failed as well and added a second red step to a failure that had nothing to do with it. signing-key.sh remove now says there is nothing to remove and succeeds when GNUPGHOME is unset or empty, or names a directory that does not exist. When a keyring exists, removal must leave it gone, or the step fails, so the no-op cannot hide a key that was imported and not removed. import and generate still require GNUPGHOME. --- .github/scripts/release/signing-key.sh | 17 +++++++++++++++- .github/scripts/release/test.sh | 27 ++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 1 deletion(-) diff --git a/.github/scripts/release/signing-key.sh b/.github/scripts/release/signing-key.sh index 754506e5..7d437f18 100644 --- a/.github/scripts/release/signing-key.sh +++ b/.github/scripts/release/signing-key.sh @@ -10,11 +10,21 @@ # by import/generate and deleted by remove. import and generate set the step # output `fingerprint` to the primary key's fingerprint; the keyring must hold # exactly one secret key. +# +# remove runs in an always() step, also when the job failed before GNUPGHOME +# was set. With GNUPGHOME unset or empty, or naming a directory that does not +# exist, no keyring was set up and there is nothing to remove: it says so and +# succeeds. Whenever a keyring exists, a removal that does not leave it gone +# fails the step. set -euo pipefail here="$(cd "$(dirname "$0")" && pwd)" # shellcheck source=lib.sh . "$here/lib.sh" +if [ "${1:-}" = remove ] && [ -z "${GNUPGHOME:-}" ]; then + echo "GNUPGHOME is not set: no keyring was set up in this job, nothing to remove." + exit 0 +fi : "${GNUPGHOME:?}" [ "$(basename "$GNUPGHOME")" = release-gnupg ] || die "GNUPGHOME must be a directory named release-gnupg (got $GNUPGHOME)" export GNUPGHOME @@ -42,8 +52,13 @@ case "${1:-}" in || die "could not generate a throwaway key" ;; remove) + if [ ! -e "$GNUPGHOME" ]; then + echo "No keyring at $GNUPGHOME: nothing to remove." + exit 0 + fi if command -v gpgconf > /dev/null 2>&1; then gpgconf --kill all > /dev/null 2>&1 || true; fi - rm -rf "$GNUPGHOME" + rm -rf "$GNUPGHOME" || die "could not remove the keyring at $GNUPGHOME" + [ ! -e "$GNUPGHOME" ] || die "the keyring at $GNUPGHOME is still there after removal" echo "Keyring removed." exit 0 ;; diff --git a/.github/scripts/release/test.sh b/.github/scripts/release/test.sh index c6cb85f7..4d46e1ff 100644 --- a/.github/scripts/release/test.sh +++ b/.github/scripts/release/test.sh @@ -769,6 +769,33 @@ PATH="$SIGNPATH" GNUPGHOME="$KEYHOME" gpg --batch --no-tty --pinentry-mode loopb --export-secret-keys "$FPR" 3<<< rehearsal > "$S/key.asc" 2> /dev/null check "signing-key: remove deletes the keyring" key "$L.15" remove check "signing-key: ... it is gone" test ! -e "$KEYHOME" +check "signing-key: ... and says so" has "$L.15" "Keyring removed." +# remove runs in an always() step, also after a job failed before GNUPGHOME +# was set. +check "signing-key: remove with GNUPGHOME unset succeeds" run "$L.15a" env -u GNUPGHOME PATH="$SIGNPATH" bash "$here/signing-key.sh" remove +check "signing-key: ... and says there was nothing to remove" has "$L.15a" "GNUPGHOME is not set: no keyring was set up in this job, nothing to remove." +check "signing-key: remove with GNUPGHOME empty succeeds" run "$L.15b" env PATH="$SIGNPATH" GNUPGHOME= bash "$here/signing-key.sh" remove +check "signing-key: ... with the same line" has "$L.15b" "GNUPGHOME is not set: no keyring was set up in this job, nothing to remove." +check "signing-key: remove when the keyring directory does not exist succeeds" run "$L.15c" env PATH="$SIGNPATH" GNUPGHOME="$KEYHOME" bash "$here/signing-key.sh" remove +check "signing-key: ... and says there was none" has "$L.15c" "No keyring at $KEYHOME: nothing to remove." +check "signing-key: ... and creates nothing" test ! -e "$KEYHOME" +check "signing-key: the name check still applies when the directory is missing" fails_with "must be a directory named release-gnupg" "$L.15d" \ + env PATH="$SIGNPATH" GNUPGHOME="$S/missing/gnupg" bash "$here/signing-key.sh" remove +check "signing-key: import with GNUPGHOME unset still fails" bash -c "! env -u GNUPGHOME PATH='$SIGNPATH' GPG_PRIVATE_KEY=x bash '$here/signing-key.sh' import" +check "signing-key: generate with GNUPGHOME empty still fails" bash -c "! env PATH='$SIGNPATH' GNUPGHOME= GPG_PASSPHRASE=x bash '$here/signing-key.sh' generate" +if [ "$(id -u)" != 0 ]; then + # A keyring whose contents cannot be deleted (read-only directories) must + # fail the step, not pass as a no-op. + stuck="$S/stuck/release-gnupg" + mkdir -p "$stuck/private-keys-v1.d" && echo key > "$stuck/private-keys-v1.d/k.key" + chmod 555 "$stuck/private-keys-v1.d" "$stuck" + check "signing-key: a keyring that cannot be removed fails the step" fails_with "could not remove the keyring at $stuck" "$L.15e" \ + env PATH="$SIGNPATH" GNUPGHOME="$stuck" bash "$here/signing-key.sh" remove + check "signing-key: ... (it is still there)" test -e "$stuck/private-keys-v1.d/k.key" + chmod 755 "$stuck" "$stuck/private-keys-v1.d"; rm -rf "$S/stuck" +else + echo "skip signing-key: removal failure (running as root, permissions do not apply)" +fi check "signing-key: import without a key fails" bash -c "! PATH='$SIGNPATH' GNUPGHOME='$KEYHOME' GPG_PRIVATE_KEY= bash '$here/signing-key.sh' import" check "signing-key: import of something that is not a key fails" fails_with "could not import the signing key" "$L.16" \ env PATH="$SIGNPATH" GNUPGHOME="$KEYHOME" GPG_PRIVATE_KEY="not a key" bash "$here/signing-key.sh" import