diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index fae003b9..2eefb9c4 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -59,9 +59,16 @@ property-only Maven profiles. Released artifacts encode the Spark version in the - **Always `mvn clean` when switching Spark profiles**: the ANTLR-generated parser sources are not regenerated on profile switch, and 4.9.3-generated code fails at runtime against the 4.13.1 runtime ("Could not deserialize ATN with version 3"). -- **Release**: pushing a `v*` tag runs `.github/workflows/release.yml` — builds tantivy4java - (linux-x86_64) once, then publishes each Spark leg to Maven Central and attaches shaded jars - to a GitHub Release. For local builds of all three legs: +- **Release**: run `.github/workflows/release.yml` from `main` with an existing `v*` tag + (Actions → Release → Run workflow). `dry-run` is ticked by default and publishes nothing; + untick it to publish. Pushing a tag does not start a release. The artifact names carry + `spark.version` from `pom.xml`; when that differs from the last release, a publish run asks + for the `expected-spark-versions` input (the dry run prints the value). The run builds tantivy4java + (linux-x86_64) from source at the commits pinned in + `.github/scripts/release/native-pins.txt` (add a line there when bumping tantivy4java in + `pom.xml`), builds each Spark leg unsigned, and only then signs and uploads from a separate + job that runs no project code. Runbook: `.github/scripts/release/README.md`. For local + builds of all three legs: `dev/build-release.sh ` (shaded jars land in `dist/`, platform classifier auto-detected for the current machine). diff --git a/.github/scripts/release/README.md b/.github/scripts/release/README.md new file mode 100644 index 00000000..0c563924 --- /dev/null +++ b/.github/scripts/release/README.md @@ -0,0 +1,387 @@ +# Release scripts + +The scripts behind `.github/workflows/release.yml`, how to adopt that +workflow, and the runbook for a release. + +## What a release publishes + +For a tag `v` (for example `v0.6.0`), one artifact version per Spark +profile, `_spark_`: + +| | | +|---|---| +| Maven Central | `io.indextables:indextables_spark:_spark_`: the pom, the jar, `-sources`, `-javadoc`, `-linux-x86_64-shaded` and `-jar-with-dependencies`, each with a signature and md5, sha1, sha256 and sha512 checksums | +| GitHub Release for the tag | the `-linux-x86_64-shaded` jar of each version | + +The file set is the one on Maven Central for `0.6.0-rc2`; the allow-list is +`ARTIFACT_SUFFIXES` in `lib.sh`. + +### The Spark version in the name + +`` is the `spark.version` that the profile sets in `pom.xml` at the +tagged commit, so the name says which Spark the jar was compiled against. It +also means that a dependency update which moves Spark from 3.5.9 to 3.5.10 +changes the artifact names of the next release, and anyone who depends on +`…_spark_3.5.9` has to change their build. That must not happen unnoticed: + +* `plan` reads the versions from `pom.xml` at the tag and lists them in the + Plan summary, next to the Spark version last published to Maven Central for + each Spark line. +* A publish run whose Spark versions differ from the last published ones, or + that cannot reach Maven Central to compare, stops in `plan`, before + anything is built. To go ahead, dispatch again with + `expected-spark-versions` set to the Spark versions you expect, for example + `3.5.10 4.0.4 4.1.3` (any order, spaces or commas). The error message and + the Plan summary of a dry run print the exact value. +* Whenever `expected-spark-versions` is given, in a dry run too, it must + match `pom.xml`, or the run stops. +* When nothing changed since the last release the input can stay empty. + +The build jobs check that Maven arrives at the versions `plan` announced, and +`assemble`, `rehearse` and `publish` accept exactly those versions. + +## Jobs and trust + +``` +plan ─► native ─► build (x3) ─► assemble ─► rehearse ─► publish + └──── run project code ────┘ └── run these scripts only ──┘ + no secrets, read-only token environment: release +``` + +| Job | Runs | Has | +|---|---|---| +| `plan` | `plan.sh` (reads `pom.xml` at the tag as data; fetches the public Maven Central metadata) | read-only token | +| `native` | `build-native.sh`: tantivy4java's Maven and cargo build, from pinned commits, no cache | read-only token | +| `build` | `build-artifacts.sh`: this project's Maven build at the tagged commit, with the release profile and signing switched off | read-only token | +| `assemble` | `assemble.sh`: checks the build outputs, lays out what will be uploaded, writes a manifest of digests | read-only token | +| `rehearse` | `verify-staging.sh`, `signing-key.sh generate`, `sign-bundles.sh`, `check-bundles.sh` with a throwaway key | read-only token | +| `publish` | `verify-staging.sh`, `signing-key.sh import`, `sign-bundles.sh`, `check-bundles.sh`, `central-upload.sh`, `github-release.sh` | `release` environment, the four secrets, `contents: write` | + +What executes in `publish`, beside the signing key and the Central token: + +* `actions/checkout` (sparse, this directory, from the commit the workflow + was started on) and `actions/download-artifact`, both pinned to a commit; +* the scripts in this directory, from that same commit; +* `bash`, coreutils, `gpg`, `zip`, `unzip`, `curl`, `jq` and `gh` from the + runner image. + +No Maven, no JVM, no cargo, and nothing from the tagged commit. The files it +signs were produced by jobs that did run project code; it signs exactly the +files in the manifest whose digest `assemble` reported as a job output, and +`assemble` and `verify-staging.sh` accept only the expected coordinates and +file names. A build cannot add another artifact, another version or another +`io.indextables` coordinate to what gets signed. What a build can still do is +put different bytes inside the six expected files: that is what the tagged +commit's review, the pins and the digests in the run summary are for. + +Limits of that description, so that it is not read as more than it is: + +* It is about `release.yml` on `main`. It says nothing about other workflow + files or other refs; see "Adopting this workflow". +* Each secret is passed to the one step that needs it, and the keyring is + deleted before the first request to Maven Central or GitHub. The secret + values themselves stay in the runner's memory for the whole job, as with + any GitHub Actions job; deleting the keyring removes the key from disk and + stops the agent, it does not make the job unable to obtain the key again. +* The scripts reach fixed addresses only (the Portal API, repo1, the three + GitHub repositories). No variable redirects them; three timing settings + exist for `test.sh` and stop the script if they are set without the test + flag, which `release.yml` never sets. +* A re-run ("Re-run failed jobs" or "Re-run all jobs") uses the workflow file + and the scripts of the commit the run was started on, not the current + `main`. A fix to a script only takes effect in a newly dispatched run. +* rustc and cargo, Maven, the JDK patch level and the runner image are + whatever the runner provides on the day. They are recorded in the summaries + and the release notes, not pinned. Pinned: the actions, protoc, the + tantivy4java commit and through its `Cargo.lock` every Rust dependency, and + the Maven plugins that the scripts call by coordinate. + +## Native inputs + +`native-pins.txt` lists, per tantivy4java version, the tantivy4java commit +and the quickwit and tantivy fork commits a release may build. The build +fetches that tantivy4java commit by id, refuses to continue if the tag +`v` points anywhere else, and checks the fork commits against the +`Cargo.lock` at that commit. It also fails if the build modifies `Cargo.lock` +(cargo does that when it cannot use the locked versions), if a dependency +points to a path outside the checkout, or if a cargo config overrides +dependency sources. protoc is the same pinned release and digest as in +`scripts/setup.sh`. + +The resolved commits and tool versions are written to the `native` job +summary, the `assemble` job summary and the release notes. + +When `pom.xml` moves to a new tantivy4java version, add its line in the same +pull request: + +``` +bash .github/scripts/release/build-native.sh print-pin +``` + +`print-pin` prints the line and, on standard error, where each commit can be +reached from. A commit id by itself proves little: GitHub serves a commit +that exists only in somebody's fork through the parent repository's URL. So +`print-pin` clones the quickwit and tantivy forks and + +* says so when a fork commit is on the fork's default branch (the normal + case; both commits pinned for 0.34.4 are the tips of `main`); +* warns, naming the branches or tags, when it is on another branch or tag + only: pin it only if building from that branch is intended; +* refuses to print a pin when it is on no branch or tag of the fork. + +Review that output before committing the line. The release build itself does +not repeat the check; it builds what the reviewed pin file says. + +The `Release Scripts` workflow fails on a pull request that changes the +tantivy4java version without adding a pin. + +`scripts/setup.sh` is not used by the release. It still clones the quickwit +fork at its default branch for developer and CI builds; current tantivy4java +versions take both forks from `Cargo.lock` and ignore that clone. + +## Adopting this workflow + +Merging the workflow does not by itself protect the key. Two facts decide +that, and both are repository settings: + +* While `GPG_PRIVATE_KEY`, `GPG_PASSPHRASE`, `CENTRAL_USERNAME` and + `CENTRAL_PASSWORD` exist as repository-level secrets, any workflow on any + ref can read them. +* The tags `v0.6.0-rc1` and `v0.6.0-rc2` carry the previous `release.yml`, + which has a `workflow_dispatch` trigger. It can be dispatched on the tag + ref, and then runs the old workflow, with the project build beside the + key. Tags cannot be edited away; only the environment rule and the removal + of the repository-level secrets close this. + +Do the steps in this order. Each one is the owner's or the releaser's to do +in the repository settings; nothing in this directory can do them. + +1. **Add the four secrets to the `release` environment** (Settings → + Environments → release → Environment secrets), with the same names. Leave + the repository-level copies in place for now. +2. **Narrow the environment.** Deployment branches and tags: selected, `main` + only (remove the `v*` tag rule). Required reviewer set. "Allow + administrators to bypass configured protection rules": off. From here on, + a run on a tag ref, including the old workflow on the two rc tags, is + refused by the environment, but it could still read the repository-level + secrets from a job that names no environment, which is why step 4 is not + optional. +3. **Dry run, then the release candidate rehearsal** described in the + runbook below. +4. **Delete the repository-level copies of the four secrets as soon as the + rehearsal has passed.** Not after the next real release: until they are + gone, the separation in this workflow protects nothing. + +One thing the rehearsal cannot show: while both copies exist, a secret that +is missing or mistyped in the environment is silently covered by the +repository-level one. After step 4 such a mistake shows up as a failed key +import or a refused upload, before anything is uploaded. Compare the four +names in the environment by eye before deleting. + +`main` must stay protected, so that the workflow file and these scripts +change only through reviewed pull requests. + +## Tag push or dispatch + +The workflow is started by `workflow_dispatch` only. A real release must be +dispatched from `main`, with the tag as input; the tag's commit must be on +`main`. + +* **Dispatch from `main`** (implemented): the workflow file and the scripts + that run beside the secrets are the reviewed ones on `main`. The tag only + selects which source is built, by the unprivileged jobs. Left open: whoever + can dispatch workflows chooses the tag (bounded by "must be on `main`") and + whoever approves the environment is the last check; and the guarantee rests + on the settings above. +* **Tag push** (not enabled): pushing `v*` runs the workflow file and scripts + *of the tagged commit*. Anyone who can push a `v*` tag, on any commit, + merged or not, then decides what the job holding the secrets executes; the + only control left is the reviewer, who approves a run without seeing which + workflow file it uses. To enable it anyway, add `push: tags: ['v*']` under + `on:` in `release.yml` (`plan.sh` already handles the event) and admit `v*` + tags in the environment. + +Creating a GitHub Release (and with it the tag) in the browser therefore does +not publish anything. Do that first if you want hand-written notes, then +dispatch the workflow: it attaches the jars to the existing release and adds +a "Build inputs" block to its notes without touching the rest. + +## Runbook + +### Dry run + +Actions → Release → Run workflow → branch `main`, `tag` = the tag, leave +`dry-run` ticked. Any existing `v*` tag on `main` whose tantivy4java version +is pinned works, including one that is already released (`v0.6.0-rc2`). + +A dry run builds everything, assembles it, and signs, bundles and checks it +with a throwaway key. It does not start the `publish` job: it asks for no +approval, reads no secret, uploads nothing, creates no release. If a run you +meant as a dry run shows a pending approval for `release`, it is not a dry +run: reject it. + +What to look at: + +* **Plan** summary: "Dry run for …", the tag and its commit, the versions, + and whether they differ from the last release. +* **native** summary: the tantivy4java, quickwit and tantivy commits. +* **Assemble** summary: the three versions; the manifest digest; the digests + of all files. For `v0.6.0-rc2` the three `.pom` digests should equal the + published ones (`…//indextables_spark-.pom.sha256` on + repo1): the pom is generated the same way as before. +* **Rehearse** log: each bundle lists 36 entries and passes. + +A dry run may also be dispatched from a branch, to try a change to the +workflow or these scripts before it is merged. + +### Release candidate rehearsal (the first real use) + +A dry run stops before the key, the Central token, the Portal and the Publish +button. The only way to exercise those is a real release, so make the first +one a release candidate on a new tag, for example `v0.6.0-rc3`, and take it +all the way through Publish. + +It has to be a new tag. An existing one cannot stand in: `0.6.0-rc2` is +already on Maven Central, a rebuild is not byte-identical, and the upload +step stops before uploading anything when it sees that. And it has to +include pressing Publish, not "upload, then drop": dropping leaves a public +GitHub pre-release with jars that are on no repository, and it skips the one +step that no test and no dry run reaches. + +Do it after steps 1 and 2 of "Adopting this workflow". At each point, check: + +1. **Before dispatching.** Create the tag on a commit of `main`. Dry-run it + first; note the `expected-spark-versions` value from the Plan summary if + it says the names changed. +2. **Dispatch from `main`**, `dry-run` unticked (and `expected-spark-versions` + if needed). + * An approval prompt for `release` appears, and only now: the dry run a + moment ago did not ask. A run dispatched from any other branch or from a + tag with `dry-run` unticked fails in `plan` and never asks. + * That is this workflow's own guard. The rule that also stops other + workflow files (the old one on the rc tags) is the environment's; confirm + it by reading it, not by dispatching the old workflow: + `gh api repos/indextables/indextables_spark/environments/release/deployment-branch-policies` + must list `main` and nothing else. + * Read the Plan and Assemble summaries, then approve. +3. **While `publish` runs.** + * The "Import the signing key" step logs `output: fingerprint=…`. It must + be the release key. The signatures on Maven Central for `0.6.0-rc2` name + `A9CD03EDB082ECB61DB66028EDA69278FFE47B19` as their signing key; the log + prints the primary key's fingerprint, which is the same value unless the + key signs with a subkey (`gpg --show-keys --with-subkey-fingerprints` on + the public key shows both). + * The summary lists each upload as it happens, with its deployment id. +4. **After `publish`, before pressing Publish.** Open + . + * Exactly three deployments from this run, named + `indextables_spark--run-`, state VALIDATED. + Their ids are the three in the job summary. A deployment for these + versions with another id or name is from another run: drop it. + * Each holds six files (pom, jar, sources, javadoc, linux-x86_64-shaded, + jar-with-dependencies), each with a signature. + * The GitHub Release for the tag has three jars, and its notes end with a + "Build inputs" block naming the commits. +5. **Press Publish** on each of the three. This cannot be undone. +6. **After Publish** (Maven Central takes a while to show a new version). + Resolve one coordinate from Maven Central and compare it with the run: + + ``` + v=0.6.0-rc3_spark_3.5.9 # one of the three versions + curl -fsSL "https://repo1.maven.org/maven2/io/indextables/indextables_spark/$v/indextables_spark-$v-linux-x86_64-shaded.jar" | sha256sum + ``` + + The digest must be the one in the "Build inputs" block of the release + notes (and in the Assemble summary) for that version, and the jar attached + to the GitHub Release must have it too. +7. **Delete the repository-level secrets** (step 4 of "Adopting this + workflow"). + +### Release + +1. Dispatch from `main` with `dry-run` unticked; add + `expected-spark-versions` if the Plan summary of the dry run asked for it. +2. Wait for `rehearse` to pass, read the Plan and Assemble summaries, then + approve the `release` environment. Up to here nothing has left the run; + rejecting the approval ends it. +3. `publish` signs, uploads one bundle per version to the Central Portal as + `USER_MANAGED`, waits until each is `VALIDATED`, then creates or updates + the GitHub Release. +4. Open . Press + **Publish** on the deployments whose ids are in the job summary, on all of + them and on no other. This is the step that cannot be undone. + +`skip-central` makes a publish run leave Maven Central alone and only attach +the jars to the GitHub Release. + +### Jars already on the GitHub Release + +A jar that is already attached to the release is left alone when it is +identical to the one this run built (a re-run of the same run). When it +differs, the GitHub Release step stops and changes nothing: a jar on a +published release may already have been downloaded, and a rebuild is never +byte-identical. This matters most with `skip-central`, where nothing on the +Maven Central side stops a second run for an already released tag. If +replacing the jars is really what you want, dispatch again with +`replace-release-assets` ticked. + +### Aborting + +| When | How | What is left behind | +|---|---|---| +| Before approval | Reject the approval, or cancel the run | Nothing | +| While `publish` runs | Cancel the run | The last step drops the deployments the job uploaded and says in the summary if it could not; check the Portal. The GitHub Release may exist with some jars | +| After `publish`, before pressing Publish | **Drop** every deployment in the Portal; delete or edit the GitHub Release by hand | Nothing on Maven Central | +| After pressing Publish | Not possible. Publish the remaining versions, and release a new version if this one is wrong | | + +### When `publish` fails + +`central-upload.sh` handles one version at a time, and when anything fails +(or the run is cancelled) the job drops what it uploaded. The Portal accepts +a drop only once a deployment is VALIDATED or FAILED, so a deployment that is +still being validated is polled first, for up to three minutes, in the +failing step and again in the job's last step. + +If something is still left, the last step fails and the job summary has a +section **"Action needed: deployments left in the Central Portal"**. It +lists each deployment by id and name with what to do: drop it by hand and do +not publish it; or, for an upload that was cut off before the Portal +answered, look for a deployment of that name and drop it if it exists. The +run id and attempt in the name tell this job's deployments from any other. + +Then use **Re-run failed jobs**: the re-run signs and uploads the same staged +files (same digests), not a rebuild, under a new attempt number. It uses the +scripts of the original run; if the failure was a bug in a script, fix it on +`main` and dispatch a new run instead (a new build, so only possible while +none of the versions has been published). + +If some versions were already published by hand and others were not, re-run +the failed job as well: a version that is already on Maven Central with +identical files is skipped, and one with different files stops the run. + +## What the tests cover, and what they cannot + +`bash .github/scripts/release/test.sh` runs offline: `mvn`, `curl` and `gh` +are stubs, and git is pointed at fixture repositories. It covers the +decision logic of every script (including every refusal), the layout of the +bundles, the upload state machine (rejection, timeouts and cancellation +during validation, refused drops, stale deployments, re-runs) and the shape +of `release.yml` (which job has secrets, the dry-run guards, pins, that no +test-only setting is set). On a pull request it runs on the runner image +with the real `gpg`, generating, exporting, importing and signing with a +throwaway key. + +Not covered by any test, and seen for the first time in a real run: + +* the Maven and cargo builds themselves (first seen in a dry run); +* artifacts passing between jobs (first seen in a dry run); +* importing the real key and using the real passphrase; +* the Central Portal accepting the token, the bundle and the signatures, and + its answers to status and drop requests (the stub follows the Portal's API + documentation); +* `gh` creating or updating the release with the job's token; +* the Publish button. + +The last four happen in or after `publish` only, which is what the release +candidate rehearsal is for. If one of the first three of them fails, nothing +has been published: the upload is `USER_MANAGED`, and a failed job drops it. diff --git a/.github/scripts/release/assemble.sh b/.github/scripts/release/assemble.sh new file mode 100644 index 00000000..65f9e13b --- /dev/null +++ b/.github/scripts/release/assemble.sh @@ -0,0 +1,215 @@ +#!/usr/bin/env bash +# Turn the build jobs' outputs into the exact trees that will be signed and +# uploaded, after checking them. Runs no project code: it only reads files. +# +# Everything in IN was produced by jobs that ran the project's build, so it is +# treated as untrusted until checked. What comes out is an allow-list result: +# only the expected coordinates, only the expected file names. +# +# Environment: +# IN directory with the downloaded artifacts: +# native/ build-native.sh output +# artifacts-/ build-artifacts.sh output, per profile +# OUT staging directory to create: +# versions.txt the versions, one per line +# release-notes.md the "Build inputs" block +# bundles// Maven repository layout for that +# version: the six files and their +# md5/sha1/sha256/sha512 (no signatures) +# MANIFEST.sha256 SHA-256 of every file above +# TAG, COMMIT, BASE_VERSION, EXPECTED_VERSIONS from plan.sh +# RUN_URL link to this workflow run (optional, for the notes) +# +# Step outputs: manifest_sha256 (digest of MANIFEST.sha256), versions. +set -euo pipefail +here="$(cd "$(dirname "$0")" && pwd)" +# shellcheck source=lib.sh +. "$here/lib.sh" + +: "${IN:?}" "${OUT:?}" "${TAG:?}" "${COMMIT:?}" "${BASE_VERSION:?}" "${EXPECTED_VERSIONS:?}" +need unzip python3 +[[ "$TAG" =~ $TAG_RE ]] || die "'$TAG' is not a release tag" +[ "v$BASE_VERSION" = "$TAG" ] || die "base version $BASE_VERSION does not belong to tag $TAG" +[[ "$COMMIT" =~ $SHA_RE ]] || die "'$COMMIT' is not a commit id" + +# check_pom : coordinates must be the ones this repository +# publishes, and the metadata Maven Central requires must be present. +check_pom() { + python3 - "$1" "$GROUP_ID" "$ARTIFACT_ID" "$2" << 'PY' || die "$(basename "$1") failed the pom checks (above)" +import sys +import xml.etree.ElementTree as ET + +path, group, artifact, version = sys.argv[1:5] +ns = "{http://maven.apache.org/POM/4.0.0}" +try: + root = ET.parse(path).getroot() +except ET.ParseError as exc: + sys.exit("pom is not well-formed XML: %s" % exc) + +def text(*names): + node = root + for name in names: + node = node.find(ns + name) if node is not None else None + return (node.text or "").strip() if node is not None else "" + +problems = [] +if root.tag != ns + "project": + problems.append("root element is not a Maven ") +if root.find(ns + "parent") is not None: + problems.append("has a ; coordinates must be stated in the pom itself") +for names, want in ((("groupId",), group), (("artifactId",), artifact), (("version",), version)): + got = text(*names) + if got != want: + problems.append("%s is '%s', expected '%s'" % ("/".join(names), got, want)) +if text("packaging") not in ("", "jar"): + problems.append("packaging is '%s', expected jar" % text("packaging")) +required = (("name",), ("description",), ("url",), + ("licenses", "license", "name"), ("licenses", "license", "url"), + ("developers", "developer", "name"), + ("scm", "connection"), ("scm", "url")) +for names in required: + if not text(*names): + problems.append("missing %s (required by Maven Central)" % "/".join(names)) +if problems: + sys.exit("\n".join(problems)) +PY +} + +# --- the native build --------------------------------------------------------- +native="$IN/native" +verify_sums "$native" +ninfo="$native/native-inputs.txt" +t4j_version="$(kv "$ninfo" tantivy4java_version)" +[[ "$t4j_version" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || die "the native artifact does not state a tantivy4java version" +[ "$(kv "$ninfo" cargo_lock_unchanged)" = true ] || die "the native build does not record an unchanged Cargo.lock" +t4j_jar="$native/repository/tantivy4java-$t4j_version-$NATIVE_CLASSIFIER.jar" +[ -f "$t4j_jar" ] || die "the native artifact has no $(basename "$t4j_jar")" +t4j_sha="$(hash_of sha256 "$t4j_jar")" +libs="$(unzip -Z1 "$t4j_jar" | grep -E '\.so$' || true)" +[ -n "$libs" ] || die "$(basename "$t4j_jar") contains no native library (*.so)" + +rm -rf "$OUT" +mkdir -p "$OUT/bundles" +: > "$OUT/versions.txt" + +# --- each Spark profile ------------------------------------------------------- +for profile in $PROFILES; do + d="$IN/artifacts-$profile" + [ -d "$d" ] || die "no build output for $profile" + verify_sums "$d" + info="$d/build-info.txt" + [ "$(kv "$info" profile)" = "$profile" ] || die "artifacts-$profile was not built with profile $profile" + version="$(kv "$info" version)" + check_version "$BASE_VERSION" "$profile" "$version" + [ "$version" = "$(profile_version "$EXPECTED_VERSIONS" "$profile")" ] \ + || die "artifacts-$profile holds $version, which is not the version the plan announced for $profile" + [ "${BASE_VERSION}_spark_$(kv "$info" spark_version)" = "$version" ] \ + || die "artifacts-$profile: version $version does not match its recorded Spark version" + + expected="$({ artifact_names "$version" | sed 's|^|files/|'; echo build-info.txt; } | LC_ALL=C sort)" + actual="$(cd "$d" && find . -type f ! -path ./SHA256SUMS | sed 's|^\./||' | LC_ALL=C sort)" + [ "$expected" = "$actual" ] \ + || die "artifacts-$profile does not hold exactly the expected files. Expected: $(echo $expected). Found: $(echo $actual)" + + [ "$(kv "$info" tantivy4java_jar_sha256)" = "$t4j_sha" ] \ + || die "artifacts-$profile was not built with this run's tantivy4java jar" + + p="$ARTIFACT_ID-$version" + check_pom "$d/files/$p.pom" "$version" + for name in $(artifact_names "$version"); do + case "$name" in + *.jar) unzip -tqq "$d/files/$name" > /dev/null 2>&1 || die "$name is not a readable jar" ;; + esac + done + + # The native library inside the published jars must be, byte for byte, the + # one this run built from the pinned commits. + for jar in "$p-$NATIVE_CLASSIFIER-shaded.jar" "$p-jar-with-dependencies.jar"; do + while IFS= read -r lib; do + want="$(unzip -p "$t4j_jar" "$lib" | hash_stream sha256)" + got="$(unzip -p "$d/files/$jar" "$lib" 2> /dev/null | hash_stream sha256 || true)" + [ "$got" = "$want" ] || die "$jar does not contain this run's $lib" + done <<< "$libs" + done + + vd="$OUT/bundles/$version/$GROUP_PATH/$version" + mkdir -p "$vd" + for name in $(artifact_names "$version"); do + cp "$d/files/$name" "$vd/$name" + for algo in $CHECKSUM_ALGOS; do + printf '%s' "$(hash_of "$algo" "$vd/$name")" > "$vd/$name.$algo" + done + done + echo "$version" >> "$OUT/versions.txt" +done + +# --- notes ---------------------------------------------------------------------- +# The values below were written by the native job, which runs build code, and +# they end up in the release notes. Identifiers must have their exact form; +# free-text tool versions are shown only if they are plain text. +SAFE_RE='^[A-Za-z0-9 ._,:/@()+="-]*$' +safe() { + if [ "${#1}" -le 200 ] && [[ "$1" =~ $SAFE_RE ]]; then printf '%s' "$1"; else printf '(not shown: unexpected characters)'; fi +} +[[ "$(kv "$ninfo" tantivy4java_tag)" == "v$t4j_version" ]] || die "the native artifact's tag is not v$t4j_version" +[[ "$(kv "$ninfo" tantivy4java_commit)" =~ $SHA_RE ]] || die "the native artifact does not state a tantivy4java commit" +for key in quickwit_commit tantivy_commit; do + value="$(kv "$ninfo" "$key")" + [ "$value" = "-" ] || [[ "$value" =~ $SHA_RE ]] || die "the native artifact's $key is not a commit id" +done +[[ "$(kv "$ninfo" cargo_lock_sha256)" =~ ^[0-9a-f]{64}$ ]] || die "the native artifact does not state a Cargo.lock digest" +{ + echo "" + echo "### Build inputs" + echo + if [ -n "${RUN_URL:-}" ]; then + echo "Built from commit \`$COMMIT\` (tag \`$TAG\`) by [this workflow run]($RUN_URL)." + else + echo "Built from commit \`$COMMIT\` (tag \`$TAG\`)." + fi + echo "The native library was compiled in that run from the commits below; nothing was restored from a cache." + echo + echo "| Input | Value |" + echo "|---|---|" + echo "| tantivy4java | \`$(kv "$ninfo" tantivy4java_tag)\` at \`$(kv "$ninfo" tantivy4java_commit)\` |" + echo "| quickwit fork | \`$(kv "$ninfo" quickwit_commit)\` |" + echo "| tantivy fork | \`$(kv "$ninfo" tantivy_commit)\` |" + other="$(kv "$ninfo" other_git_dependencies)" + [ -z "$other" ] || echo "| other git dependencies | \`$(safe "$other")\` |" + echo "| Cargo.lock | SHA-256 \`$(kv "$ninfo" cargo_lock_sha256)\`, unchanged by the build |" + for key in rustc cargo protoc java maven runner_image; do + value="$(kv "$ninfo" "$key")" + [ -z "$value" ] || echo "| ${key//_/ } | \`$(safe "$value")\` |" + done + echo + echo "| Version | SHA-256 of \`-$NATIVE_CLASSIFIER-shaded.jar\` |" + echo "|---|---|" + while IFS= read -r version; do + f="$OUT/bundles/$version/$GROUP_PATH/$version/$ARTIFACT_ID-$version-$NATIVE_CLASSIFIER-shaded.jar" + echo "| \`$version\` | \`$(hash_of sha256 "$f")\` |" + done < "$OUT/versions.txt" + echo "" +} > "$OUT/release-notes.md" + +write_sums "$OUT" MANIFEST.sha256 +manifest_sha="$(hash_of sha256 "$OUT/MANIFEST.sha256")" +output manifest_sha256 "$manifest_sha" +output versions "$(tr '\n' ' ' < "$OUT/versions.txt" | sed 's/ $//')" + +{ + echo "## Staged for $TAG" + echo + echo "Manifest SHA-256: \`$manifest_sha\`. The publish job signs and uploads exactly the files this manifest lists, and refuses anything else." + echo + sed -e '/^" { skip = 1 } + !skip { + if ($0 ~ /^[[:space:]]*$/) { blank = blank $0 "\n" } else { printf "%s%s\n", blank, $0; blank = "" } + } + $0 == "" { skip = 0 } +' > "$tmp/notes.md" +if [ -s "$tmp/notes.md" ]; then echo >> "$tmp/notes.md"; fi +cat "$staging/release-notes.md" >> "$tmp/notes.md" +gh release edit "$TAG" --notes-file "$tmp/notes.md" > /dev/null + +url="$(gh release view "$TAG" --json url --jq '.url' 2> /dev/null || true)" +{ + echo "### GitHub Release" + echo + echo "${url:-Release $TAG}: ${#assets[@]} shaded jars attached." +} | summary diff --git a/.github/scripts/release/lib.sh b/.github/scripts/release/lib.sh new file mode 100644 index 00000000..0195250c --- /dev/null +++ b/.github/scripts/release/lib.sh @@ -0,0 +1,183 @@ +# Shared by the release scripts. Sourced, never executed. +# Works with bash 3.2 or newer and with either GNU or BSD userland, so that +# test.sh can run the same code on a developer machine and on the runner. + +# What is published. Changing any of these changes the published coordinates. +GROUP_ID=io.indextables +ARTIFACT_ID=indextables_spark +GROUP_PATH=io/indextables/indextables_spark +NATIVE_CLASSIFIER=linux-x86_64 + +# One artifact version is published per Spark profile. Must list the same +# profiles, in the same order, as the build matrix in release.yml. +PROFILES="spark-3.5 spark-4.0 spark-4.1" + +# The files that make up one published version, as suffixes of +# -. This is an allow-list: a version directory holding +# any other file is refused. +ARTIFACT_SUFFIXES=".pom .jar -sources.jar -javadoc.jar -${NATIVE_CLASSIFIER}-shaded.jar -jar-with-dependencies.jar" +CHECKSUM_ALGOS="md5 sha1 sha256 sha512" + +# v.. with an optional pre-release part (v0.6.0-rc2). +TAG_RE='^v[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z]+(\.[0-9A-Za-z]+)*)?$' +BASE_RE='^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z]+(\.[0-9A-Za-z]+)*)?$' +SHA_RE='^[0-9a-f]{40}$' + +die() { echo "::error::$*" >&2; exit 1; } +warn() { echo "::warning::$*" >&2; } + +# Where the scripts reach out to. Fixed: there is no variable that redirects +# them. test.sh puts stubs for curl and git redirection in front instead. +CENTRAL_API=https://central.sonatype.com/api/v1/publisher +CENTRAL_REPO=https://repo1.maven.org/maven2 +CENTRAL_PORTAL=https://central.sonatype.com/publishing/deployments +TANTIVY4JAVA_URL=https://github.com/indextables/tantivy4java.git +QUICKWIT_URL=https://github.com/indextables/quickwit +TANTIVY_URL=https://github.com/indextables/tantivy + +# The local Maven repository, where Maven puts it for this user. +M2_REPO="$HOME/.m2/repository" + +# test_setting : the value of a timing setting. Only test.sh +# may change one, by also setting RELEASE_SCRIPTS_TESTING=1, which release.yml +# never does; a setting that is present without it stops the script. +test_setting() { + local value + eval "value=\${$1:-}" + if [ -z "$value" ]; then + printf '%s\n' "$2" + elif [ "${RELEASE_SCRIPTS_TESTING:-}" = 1 ]; then + printf '%s\n' "$value" + else + die "$1 is a test-only setting; it must not be set in a real run" + fi +} + +# profile_version : the entry of a space-separated list +# of versions (one per profile, in PROFILES order) that belongs to . +profile_version() { + local p + # shellcheck disable=SC2086 + set -- "$2" $1 + local want="$1" + shift + for p in $PROFILES; do + [ $# -gt 0 ] || break + if [ "$p" = "$want" ]; then printf '%s\n' "$1"; return 0; fi + shift + done + die "no version for profile $want in the plan's version list" +} + +# need : fail unless every command is on PATH. +need() { + local c + for c in "$@"; do + command -v "$c" > /dev/null 2>&1 || die "required command not found: $c" + done +} + +# hash_stream : hex digest of standard input. +hash_stream() { + local out + case "$1" in + md5) + if command -v md5sum > /dev/null 2>&1; then out="$(md5sum)"; else out="$(md5 -q)"; fi ;; + sha1 | sha256 | sha512) + if command -v "${1}sum" > /dev/null 2>&1; then out="$("${1}sum")"; else out="$(shasum -a "${1#sha}")"; fi ;; + *) die "unknown digest algorithm: $1" ;; + esac + out="${out%% *}" + [[ "$out" =~ ^[0-9a-f]+$ ]] || die "could not compute a $1 digest" + printf '%s\n' "$out" +} + +# hash_of +hash_of() { + [ -f "$2" ] || die "no such file: $2" + hash_stream "$1" < "$2" +} + +# output : set a step output (single line) and echo it. +output() { + case "$2" in *$'\n'*) die "refusing to write a multi-line value for output '$1'" ;; esac + if [ -n "${GITHUB_OUTPUT:-}" ]; then printf '%s=%s\n' "$1" "$2" >> "$GITHUB_OUTPUT"; fi + printf 'output: %s=%s\n' "$1" "$2" +} + +# summary: append standard input to the job summary (or print it when there +# is none, as in test.sh). +summary() { + if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then cat >> "$GITHUB_STEP_SUMMARY"; else cat; fi +} + +# spark_line : "3.5" for "spark-3.5". Fails for an unknown profile. +spark_line() { + local p + for p in $PROFILES; do + if [ "$p" = "$1" ]; then printf '%s\n' "${1#spark-}"; return 0; fi + done + die "unknown Spark profile: $1 (expected one of: $PROFILES)" +} + +# check_version : the version must be +# _spark_ where x.y is the profile's Spark line. +check_version() { + local base="$1" line re + line="$(spark_line "$2")" || exit 1 + re="^${base//./\\.}_spark_${line//./\\.}\\.[0-9]+$" + [[ "$3" =~ $re ]] || die "version '$3' is not ${base}_spark_${line}. (profile $2)" +} + +# pom_tantivy4java_version : same extraction as scripts/setup.sh. +pom_tantivy4java_version() { + local v + v="$(grep -A2 'tantivy4java' "$1" | grep '' \ + | sed 's/.*\(.*\)<\/version>.*/\1/' | tr -d '[:space:]' || true)" + [[ "$v" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || die "could not read the tantivy4java version from $1 (got '$v')" + printf '%s\n' "$v" +} + +# kv : value of key=value in a build-info file (first match). +kv() { + sed -n "s/^$2=//p" "$1" | head -n 1 +} + +# artifact_names : the allow-listed file names of one version. +artifact_names() { + local sfx + for sfx in $ARTIFACT_SUFFIXES; do printf '%s-%s%s\n' "$ARTIFACT_ID" "$1" "$sfx"; done +} + +# write_sums [name]: write / (default SHA256SUMS) listing the +# SHA-256 of every other regular file under , in sha256sum format. +write_sums() { + local dir="$1" name="${2:-SHA256SUMS}" + [ -d "$dir" ] || die "no such directory: $dir" + ( + cd "$dir" || exit 1 + find . -type f ! -path "./$name" ! -path "./$name.tmp" | sed 's|^\./||' | LC_ALL=C sort \ + | while IFS= read -r path; do + printf '%s %s\n' "$(hash_of sha256 "$path")" "$path" + done > "$name.tmp" + mv "$name.tmp" "$name" + ) +} + +# verify_sums [name]: / must list exactly the regular files +# under (no more, no fewer, nothing that is not a regular file) and +# every digest must match. The directory may come from another job, so +# nothing in it is used before this passes. +verify_sums() { + local dir="$1" name="${2:-SHA256SUMS}" listed actual sum path + [ -f "$dir/$name" ] || die "$dir has no $name" + [ -z "$(find "$dir" ! -type f ! -type d | head -n 1)" ] \ + || die "$dir contains something that is neither a regular file nor a directory" + listed="$(awk '{ print $2 }' "$dir/$name" | LC_ALL=C sort)" + actual="$(cd "$dir" && find . -type f ! -path "./$name" | sed 's|^\./||' | LC_ALL=C sort)" + [ -n "$actual" ] || die "$dir is empty" + [ "$listed" = "$actual" ] || die "the files in $dir are not exactly the files listed in $name" + while read -r sum path; do + [ "$(hash_of sha256 "$dir/$path")" = "$sum" ] || die "digest mismatch for $path in $dir" + done < "$dir/$name" +} diff --git a/.github/scripts/release/native-pins.txt b/.github/scripts/release/native-pins.txt new file mode 100644 index 00000000..bf2690a5 --- /dev/null +++ b/.github/scripts/release/native-pins.txt @@ -0,0 +1,25 @@ +# What the release compiles for the tantivy4java native library. +# +# One line per tantivy4java version that pom.xml may name: +# +# +# +# build-native.sh fetches exactly the tantivy4java commit listed here, and +# refuses to build unless +# * the tag v in indextables/tantivy4java still points to it, and +# * the Cargo.lock at that commit resolves indextables/quickwit and +# indextables/tantivy to the two fork commits listed here. +# A fork column is "-" when that tantivy4java version does not use the fork. +# +# When pom.xml moves to a new tantivy4java version, add a line for it in the +# same pull request (keep the old lines: they are needed to dry-run or +# re-run older tags): +# +# bash .github/scripts/release/build-native.sh print-pin +# +# print-pin also says, on standard error, whether each fork commit is on the +# fork's default branch, warns when it is only on another branch or tag, and +# refuses when it is on none (such a commit can come from somebody's fork of +# the fork). Review that output and the printed commits before committing +# the line; from then on they are what a release is allowed to build. +0.34.4 be2d885a5b6d2e7438d88ea7f9818c5086e74a52 3026b228a71bfb6be16f3966ff80c8641834cb21 d491895f1cd54d166c8010469d4625ae81c636f3 diff --git a/.github/scripts/release/plan.sh b/.github/scripts/release/plan.sh new file mode 100644 index 00000000..7698850c --- /dev/null +++ b/.github/scripts/release/plan.sh @@ -0,0 +1,228 @@ +#!/usr/bin/env bash +# Decide what this run does, before anything is built. +# +# Reads the triggering event and the dispatch inputs (as strings, from the +# environment), validates the tag, resolves it to a commit, checks that the +# commit is on the default branch and works out the versions that would be +# published. Sets the step outputs: +# +# mode dry-run | publish +# tag the release tag (v0.6.0) +# commit the commit the tag points to; every later job uses this +# base the tag without its "v" (0.6.0) +# versions the versions to publish, one per Spark profile, in PROFILES +# order (0.6.0_spark_3.5.9 0.6.0_spark_4.0.4 0.6.0_spark_4.1.3) +# prerelease true when the base version has a pre-release part (0.6.0-rc2) +# central true unless a publish run asked to skip Maven Central +# +# Only "publish" lets the publish job start, and only these two cases give it: +# * workflow_dispatch from the default branch with dry-run set to false +# * a pushed tag (not enabled in release.yml; see the README) +# Every other combination is a dry run or an error. +# +# Versions. _spark_, where spark.version is what each +# profile sets in pom.xml at the tagged commit. That value changes whenever +# the Spark dependency is bumped, and with it the published artifact names. +# So that this never happens unnoticed: +# * the Plan summary lists the exact versions, and for each Spark line the +# version last published to Maven Central; +# * a publish run whose Spark versions differ from the last published ones +# (or that cannot be compared) stops here unless the releaser has typed +# the Spark versions into the expected-spark-versions input; +# * when that input is given it must match what pom.xml says, in any run. +# The build jobs later check that Maven derives the same versions. +set -euo pipefail +here="$(cd "$(dirname "$0")" && pwd)" +# shellcheck source=lib.sh +. "$here/lib.sh" + +: "${EVENT_NAME:?}" "${REF:?}" "${DEFAULT_BRANCH:?}" +repo_dir="${REPO_DIR:-.}" +need git python3 curl + +case "$EVENT_NAME" in + workflow_dispatch) + tag="${INPUT_TAG:-}" + case "${INPUT_DRY_RUN:-}" in + true) mode=dry-run ;; + false) + [ "$REF" = "refs/heads/$DEFAULT_BRANCH" ] \ + || die "A release can only be dispatched from $DEFAULT_BRANCH (this run is on $REF). Dry runs may be dispatched from any ref." + mode=publish ;; + *) die "dry-run must be true or false (got '${INPUT_DRY_RUN:-}')" ;; + esac + ;; + push) + [ "${REF_TYPE:-}" = tag ] || die "push runs are only supported for tags (this run is on $REF)" + tag="${REF_NAME:-}" + mode=publish + ;; + *) die "unsupported event: $EVENT_NAME" ;; +esac + +[[ "$tag" =~ $TAG_RE ]] || die "'$tag' is not a release tag (expected v.., optionally with a pre-release part such as -rc1)" +base="${tag#v}" + +commit="$(git -C "$repo_dir" rev-parse --verify --quiet "refs/tags/$tag^{commit}" || true)" +[[ "$commit" =~ $SHA_RE ]] || die "tag $tag does not exist in this repository" + +# The source that gets published must have been merged: the tag's commit has +# to be the tip of the default branch or one of its ancestors. +git -C "$repo_dir" rev-parse --verify --quiet "refs/remotes/origin/$DEFAULT_BRANCH^{commit}" > /dev/null \ + || die "origin/$DEFAULT_BRANCH is not available; the checkout must fetch full history" +git -C "$repo_dir" merge-base --is-ancestor "$commit" "refs/remotes/origin/$DEFAULT_BRANCH" \ + || die "tag $tag ($commit) is not an ancestor of $DEFAULT_BRANCH; only merged commits can be released" + +case "$base" in *-*) prerelease=true ;; *) prerelease=false ;; esac + +central=true +if [ "$mode" = publish ] && [ "${INPUT_SKIP_CENTRAL:-false}" = true ]; then central=false; fi + +# --- versions ----------------------------------------------------------------- +tmp="$(mktemp -d)" +trap 'rm -rf "$tmp"' EXIT +git -C "$repo_dir" show "$commit:pom.xml" > "$tmp/pom.xml" 2> /dev/null || die "the tagged commit has no pom.xml" +# pom.xml is read as data; nothing from the tagged commit is executed here. +cat > "$tmp/spark_versions.py" << 'PY' +import sys +import xml.etree.ElementTree as ET + +ns = "{http://maven.apache.org/POM/4.0.0}" +try: + root = ET.parse(sys.argv[1]).getroot() +except ET.ParseError as exc: + sys.exit("pom.xml is not well-formed XML: %s" % exc) + +def child(node, name): + for c in (node if node is not None else []): + if c.tag == ns + name: + return c + return None + +def spark_version(node): + c = child(child(node, "properties"), "spark.version") + return (c.text or "").strip() if c is not None else "" + +top = spark_version(root) +by_profile = {} +for p in (child(root, "profiles") if child(root, "profiles") is not None else []): + pid = child(p, "id") + if pid is not None: + by_profile[(pid.text or "").strip()] = spark_version(p) +out = [] +for name in sys.argv[2:]: + if name not in by_profile: + sys.exit("pom.xml at the tag has no profile %s" % name) + value = by_profile[name] or top + if not value or "$" in value: + sys.exit("pom.xml at the tag does not state spark.version for profile %s" % name) + out.append(value) +print(" ".join(out)) +PY +# shellcheck disable=SC2086 +spark_versions="$(python3 "$tmp/spark_versions.py" "$tmp/pom.xml" $PROFILES)" \ + || die "could not read the Spark versions from pom.xml at $tag (above)" + +versions="" +# shellcheck disable=SC2086 +set -- $spark_versions +for profile in $PROFILES; do + check_version "$base" "$profile" "${base}_spark_$1" + versions="${versions:+$versions }${base}_spark_$1" + shift +done + +# What the releaser expects, if they said so: the same Spark versions, in any +# order, separated by spaces or commas. +expected="$(printf '%s' "${INPUT_EXPECTED_SPARK:-}" | tr ',' ' ' | tr -s ' ' '\n' | grep . | LC_ALL=C sort | tr '\n' ' ' | sed 's/ $//' || true)" +confirmed=false +if [ -n "$expected" ]; then + derived_sorted="$(printf '%s\n' $spark_versions | LC_ALL=C sort | tr '\n' ' ' | sed 's/ $//')" + [ "$expected" = "$derived_sorted" ] \ + || die "expected-spark-versions is '$expected', but pom.xml at $tag gives '$derived_sorted'. Nothing was built. Check which is right before trying again." + confirmed=true +fi + +# Compare with what was last published for each Spark line. +metadata="" +if curl -fsS --connect-timeout 15 --max-time 60 -o "$tmp/metadata.xml" "$CENTRAL_REPO/$GROUP_PATH/maven-metadata.xml" 2> /dev/null; then + metadata="$tmp/metadata.xml" +fi +changed="" +comparison="" +# shellcheck disable=SC2086 +set -- $spark_versions +for profile in $PROFILES; do + line="$(spark_line "$profile")" + if [ -z "$metadata" ]; then + last="unknown" + else + last="$(grep -o -E "[^<]*_spark_${line//./\\.}\.[0-9]+" "$metadata" | tail -n 1 \ + | sed -E 's/.*_spark_([0-9.]+)<.*/\1/' || true)" + last="${last:-none}" + fi + case "$last" in + "$1") note="unchanged" ;; + none) note="first release for Spark $line" ;; + unknown) note="could not be compared"; changed="$changed $line" ;; + *) note="**changed** from $last"; changed="$changed $line" ;; + esac + comparison="$comparison| \`${base}_spark_$1\` | ${last} | $note |"$'\n' + shift +done + +if [ -n "$changed" ] && [ "$confirmed" != true ]; then + if [ -z "$metadata" ]; then + what="The versions last published to Maven Central could not be read, so the artifact names could not be compared with the last release." + else + what="The Spark version in the artifact names differs from the last release for Spark$changed. Consumers that name the full version need the new one." + fi + if [ "$mode" = publish ]; then + die "$what This run would publish: $versions. If that is intended, dispatch again with expected-spark-versions set to: $spark_versions" + fi + warn "$what A publish run will ask for expected-spark-versions: $spark_versions" +fi + +output mode "$mode" +output tag "$tag" +output commit "$commit" +output base "$base" +output versions "$versions" +output prerelease "$prerelease" +output central "$central" + +{ + if [ "$mode" = publish ]; then + echo "## Release $tag" + echo + if [ "$central" = true ]; then + echo "This run **publishes**: it uploads to Maven Central and creates or updates the GitHub Release, once the \`release\` environment has been approved." + else + echo "This run creates or updates the **GitHub Release only** (Maven Central was skipped by input), once the \`release\` environment has been approved." + fi + else + echo "## Dry run for $tag" + echo + echo "Nothing is uploaded and no release is created. The publish job is not started, so no approval is requested and no secret is read." + fi + echo + echo "| | |" + echo "|---|---|" + echo "| Tag | \`$tag\` |" + echo "| Commit | \`$commit\` |" + echo "| Workflow ref | \`$REF\` |" + echo + echo "### Versions" + echo + echo "| To be published | Spark version last on Maven Central | |" + echo "|---|---|---|" + printf '%s' "$comparison" + echo + if [ "$confirmed" = true ]; then + echo "The Spark versions were confirmed with \`expected-spark-versions\`." + elif [ -n "$changed" ]; then + echo "> **The artifact names are not the same as in the last release.** To publish them, set \`expected-spark-versions\` to \`$spark_versions\`." + else + echo "Same Spark versions as the last release. (\`expected-spark-versions\` would be \`$spark_versions\`.)" + fi +} | summary diff --git a/.github/scripts/release/sign-bundles.sh b/.github/scripts/release/sign-bundles.sh new file mode 100644 index 00000000..060dba4d --- /dev/null +++ b/.github/scripts/release/sign-bundles.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# Sign every staged file and zip one upload bundle per version. +# +# sign-bundles.sh +# +# Environment: +# GNUPGHOME keyring holding the signing key (see signing-key.sh) +# SIGNING_KEY fingerprint of the key to sign with +# GPG_PASSPHRASE its passphrase (may be empty for an unprotected key) +# +# Writes .asc next to each of the six files of each version (checksum +# files are not signed, as before), then +# /indextables_spark--bundle.zip holding that version's +# Maven repository layout. Run verify-staging.sh first: this script signs +# whatever is in the staging directory. +# +# The same script runs in the rehearse job (throwaway key) and in the publish +# job (release key), so a dry run exercises the exact signing code. +set -euo pipefail +here="$(cd "$(dirname "$0")" && pwd)" +# shellcheck source=lib.sh +. "$here/lib.sh" + +staging="${1:?usage: sign-bundles.sh }" +out="${2:?usage: sign-bundles.sh }" +: "${GNUPGHOME:?}" "${SIGNING_KEY:?}" +export GNUPGHOME +passphrase="${GPG_PASSPHRASE:-}" +need gpg zip +[ -s "$staging/versions.txt" ] || die "$staging lists no versions" + +mkdir -p "$out" +out="$(cd "$out" && pwd)" + +while IFS= read -r version <&4; do + root="$staging/bundles/$version" + [ -d "$root/$GROUP_PATH/$version" ] || die "nothing staged for $version" + for name in $(artifact_names "$version"); do + f="$root/$GROUP_PATH/$version/$name" + [ -f "$f" ] || die "staged file missing: $name" + [ ! -e "$f.asc" ] || die "$name is already signed; sign-bundles.sh needs a clean staging directory" + gpg --batch --yes --no-tty --pinentry-mode loopback --passphrase-fd 3 \ + --local-user "$SIGNING_KEY" --armor --detach-sign --output "$f.asc" "$f" \ + 3<<< "$passphrase" < /dev/null \ + || die "could not sign $name" + done + zip="$out/$ARTIFACT_ID-$version-bundle.zip" + rm -f "$zip" + (cd "$root" && zip -q -r -X "$zip" "${GROUP_PATH%%/*}") + echo "Signed and bundled $version: $zip" +done 4< "$staging/versions.txt" diff --git a/.github/scripts/release/signing-key.sh b/.github/scripts/release/signing-key.sh new file mode 100644 index 00000000..7d437f18 --- /dev/null +++ b/.github/scripts/release/signing-key.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash +# Manage the keyring used for signing, in a directory of its own. +# +# signing-key.sh import import the private key in GPG_PRIVATE_KEY +# signing-key.sh generate create a throwaway key protected by +# GPG_PASSPHRASE (rehearsal only) +# signing-key.sh remove stop the agent and delete the keyring +# +# GNUPGHOME must name a directory called "release-gnupg"; it is created empty +# by import/generate and deleted by remove. import and generate set the step +# output `fingerprint` to the primary key's fingerprint; the keyring must hold +# exactly one secret key. +# +# remove runs in an always() step, also when the job failed before GNUPGHOME +# was set. With GNUPGHOME unset or empty, or naming a directory that does not +# exist, no keyring was set up and there is nothing to remove: it says so and +# succeeds. Whenever a keyring exists, a removal that does not leave it gone +# fails the step. +set -euo pipefail +here="$(cd "$(dirname "$0")" && pwd)" +# shellcheck source=lib.sh +. "$here/lib.sh" + +if [ "${1:-}" = remove ] && [ -z "${GNUPGHOME:-}" ]; then + echo "GNUPGHOME is not set: no keyring was set up in this job, nothing to remove." + exit 0 +fi +: "${GNUPGHOME:?}" +[ "$(basename "$GNUPGHOME")" = release-gnupg ] || die "GNUPGHOME must be a directory named release-gnupg (got $GNUPGHOME)" +export GNUPGHOME + +fresh_home() { + rm -rf "$GNUPGHOME" + (umask 077 && mkdir -p "$GNUPGHOME") +} + +case "${1:-}" in + import) + need gpg + : "${GPG_PRIVATE_KEY:?}" + fresh_home + printf '%s\n' "$GPG_PRIVATE_KEY" | gpg --batch --quiet --no-tty --import \ + || die "could not import the signing key" + ;; + generate) + need gpg + : "${GPG_PASSPHRASE:?}" + fresh_home + gpg --batch --quiet --no-tty --pinentry-mode loopback --passphrase-fd 3 \ + --quick-generate-key "Release rehearsal (throwaway key, never published) " \ + ed25519 sign 1d 3<<< "$GPG_PASSPHRASE" < /dev/null \ + || die "could not generate a throwaway key" + ;; + remove) + if [ ! -e "$GNUPGHOME" ]; then + echo "No keyring at $GNUPGHOME: nothing to remove." + exit 0 + fi + if command -v gpgconf > /dev/null 2>&1; then gpgconf --kill all > /dev/null 2>&1 || true; fi + rm -rf "$GNUPGHOME" || die "could not remove the keyring at $GNUPGHOME" + [ ! -e "$GNUPGHOME" ] || die "the keyring at $GNUPGHOME is still there after removal" + echo "Keyring removed." + exit 0 + ;; + *) die "usage: signing-key.sh import|generate|remove" ;; +esac + +fingerprints="$(gpg --batch --no-tty --with-colons --list-secret-keys \ + | awk -F: '$1 == "sec" { primary = 1; next } primary && $1 == "fpr" { print $10; primary = 0 }')" +[ "$(printf '%s' "$fingerprints" | grep -c . || true)" = 1 ] \ + || die "the keyring must hold exactly one secret key" +[[ "$fingerprints" =~ ^[0-9A-F]{40}$ ]] || die "could not read the key's fingerprint" +output fingerprint "$fingerprints" diff --git a/.github/scripts/release/test.sh b/.github/scripts/release/test.sh new file mode 100644 index 00000000..4d46e1ff --- /dev/null +++ b/.github/scripts/release/test.sh @@ -0,0 +1,1347 @@ +#!/usr/bin/env bash +# Offline tests for the release scripts and for the shape of release.yml. +# No network, no credentials, nothing is uploaded: mvn, curl and gh are +# replaced by stubs that serve fixtures and record what they were asked to do. +# +# bash .github/scripts/release/test.sh +# +# Needs bash (3.2 or newer), git, zip, unzip, jq and python3. Optional: +# gpg the signing tests use a real throwaway key in a temporary keyring; +# without gpg they use a stub (REQUIRE_GPG=1 makes that a failure) +# ruby for the checks that parse release.yml (REQUIRE_WORKFLOW_CHECKS=1 +# makes a missing ruby a failure) +# Exits non-zero if any check fails. +set -uo pipefail + +here="$(cd "$(dirname "$0")" && pwd)" +root="$(cd "$here/../../.." && pwd)" +workflow="$root/.github/workflows/release.yml" +T="$(mktemp -d)" +T="$(cd "$T" && pwd -P)" +trap 'rm -rf "$T"' EXIT +# shellcheck source=lib.sh +. "$here/lib.sh" +SCRIPTS_UNDER_TEST="$(ls "$here"/*.sh | grep -v '/test\.sh$')" + +pass=0 +fail=0 +ok() { pass=$((pass + 1)); printf 'ok %s\n' "$1"; } +bad() { fail=$((fail + 1)); printf 'FAIL %s\n' "$1"; } +check() { # check + local name="$1"; shift + if "$@" > /dev/null 2>&1; then ok "$name"; else bad "$name"; fi +} +is() { [ "$1" = "$2" ]; } +has() { grep -qF -- "$2" "$1"; } +hasnt() { ! grep -qF -- "$2" "$1"; } +# run : run with outputs captured; step outputs go to .out +run() { + local log="$1"; shift + GITHUB_OUTPUT="$log.out" GITHUB_STEP_SUMMARY="$log.summary" "$@" > "$log" 2>&1 +} +fails_with() { # fails_with : must fail and say + local text="$1" log="$2"; shift 2 + if run "$log" "$@"; then return 1; fi + grep -qF -- "$text" "$log" +} +out() { sed -n "s/^$2=//p" "$1.out" 2> /dev/null | tail -n 1; } + +export GIT_AUTHOR_NAME=test GIT_AUTHOR_EMAIL=test@invalid GIT_COMMITTER_NAME=test GIT_COMMITTER_EMAIL=test@invalid +export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null +unset GITHUB_OUTPUT GITHUB_STEP_SUMMARY GNUPGHOME JAVA_HOME + +SHA_A=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa +SHA_B=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb +SHA_C=cccccccccccccccccccccccccccccccccccccccc +SHA_D=dddddddddddddddddddddddddddddddddddddddd +# The versions the plan announces in most tests below. +EV="1.2.3_spark_3.5.8 1.2.3_spark_4.0.3 1.2.3_spark_4.1.2" +# Variables that used to redirect the scripts, or might be expected to. None +# of them may have any effect; they are set here for every test. +export CENTRAL_API_URL=https://override.invalid/api CENTRAL_REPO_URL=https://override.invalid/maven2 +export T4J_REPO=file:///override/invalid M2_REPO=/override/invalid + +# --------------------------------------------------------------------------- +# lib.sh +# --------------------------------------------------------------------------- +printf 'abc' > "$T/abc" +check "lib: md5" is "$(hash_of md5 "$T/abc")" 900150983cd24fb0d6963f7d28e17f72 +check "lib: sha1" is "$(hash_of sha1 "$T/abc")" a9993e364706816aba3e25717850c26c9cd0d89d +check "lib: sha256" is "$(hash_of sha256 "$T/abc")" ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad +check "lib: sha512" is "$(hash_of sha512 "$T/abc")" ddaf35a193617abacc417349ae20413112e6fa4e89a97ea20a9eeee64b55d39a2192992a274fc1a836ba3c23a3feebbd454d4423643ce80e2a9ac94fa54ca49f +check "lib: version for the profile's Spark line is accepted" bash -c ". '$here/lib.sh'; check_version 0.6.0-rc2 spark-4.0 0.6.0-rc2_spark_4.0.3" +check "lib: version for another Spark line is refused" bash -c "! (. '$here/lib.sh'; check_version 0.6.0 spark-4.0 0.6.0_spark_4.1.2)" +check "lib: version for another base is refused" bash -c "! (. '$here/lib.sh'; check_version 0.6.0 spark-4.0 0.6.1_spark_4.0.3)" +check "lib: a longer base does not match by prefix" bash -c "! (. '$here/lib.sh'; check_version 0.6.0 spark-3.5 0.6.0-rc2_spark_3.5.8)" +check "lib: an unknown profile is refused" bash -c "! (. '$here/lib.sh'; check_version 0.6.0 spark-9.9 0.6.0_spark_9.9.1)" +check "lib: the six published files" is "$(artifact_names 1.0.0_spark_3.5.8 | tr '\n' ' ')" \ + "indextables_spark-1.0.0_spark_3.5.8.pom indextables_spark-1.0.0_spark_3.5.8.jar indextables_spark-1.0.0_spark_3.5.8-sources.jar indextables_spark-1.0.0_spark_3.5.8-javadoc.jar indextables_spark-1.0.0_spark_3.5.8-linux-x86_64-shaded.jar indextables_spark-1.0.0_spark_3.5.8-jar-with-dependencies.jar " +mkdir -p "$T/sums/sub" && echo one > "$T/sums/a" && echo two > "$T/sums/sub/b" +(. "$here/lib.sh"; write_sums "$T/sums") +check "lib: verify_sums accepts what write_sums wrote" bash -c ". '$here/lib.sh'; verify_sums '$T/sums'" +echo three > "$T/sums/extra" +check "lib: verify_sums refuses an unlisted file" bash -c "! (. '$here/lib.sh'; verify_sums '$T/sums')" +rm "$T/sums/extra"; echo changed > "$T/sums/a" +check "lib: verify_sums refuses a changed file" bash -c "! (. '$here/lib.sh'; verify_sums '$T/sums')" +echo one > "$T/sums/a"; ln -s a "$T/sums/link" +check "lib: verify_sums refuses a symbolic link" bash -c "! (. '$here/lib.sh'; verify_sums '$T/sums')" + +# --------------------------------------------------------------------------- +# plan.sh +# --------------------------------------------------------------------------- +P="$T/plan" +mkdir -p "$P/bin" +# Stub curl: serves the Maven Central metadata fixture, or fails like an +# unreachable host when there is none. Every plan.sh call below uses it. +cat > "$P/bin/curl" << 'STUB' +#!/usr/bin/env bash +o=""; url="" +while [ $# -gt 0 ]; do case "$1" in -o) o="$2"; shift 2 ;; --connect-timeout | --max-time) shift 2 ;; -*) shift ;; *) url="$1"; shift ;; esac; done +echo "$url" >> "$PFAKE/calls.log" +[ "$url" = "https://repo1.maven.org/maven2/io/indextables/indextables_spark/maven-metadata.xml" ] || exit 97 +[ -f "$PFAKE/metadata.xml" ] || exit 6 +cp "$PFAKE/metadata.xml" "$o" +STUB +chmod +x "$P/bin/curl" +PFAKE="$P/fake"; mkdir -p "$PFAKE"; export PFAKE +central_has() { # central_has ...: what Maven Central has published + { echo ""; for v in "$@"; do echo " $v"; done; echo ""; } > "$PFAKE/metadata.xml" +} +central_has 0.9.0_spark_3.5.3 1.0.0_spark_3.5.8 1.0.0_spark_4.0.3 1.0.0_spark_4.1.2 +plan_pom() { # plan_pom <3.5 version> <4.0 version> <4.1 version> + cat << EOF + + + + + $1 + + + spark-3.5 + spark-4.0$2 + spark-4.1$3 + + +EOF +} +git init -q -b main "$P/upstream" +( + cd "$P/upstream" + plan_pom 3.5.8 4.0.3 4.1.2 > pom.xml && git add pom.xml && git commit -q -m one && git tag v1.2.3 + plan_pom 3.5.9 4.0.3 4.1.2 > pom.xml && git commit -q -am two && git tag -a v1.3.0-rc1 -m rc + plan_pom 3.5.8 4.1.9 4.1.2 > pom.xml && git commit -q -am wrongline && git tag v1.3.1 + plan_pom 3.5.8 4.0.3 4.1.2-SNAPSHOT > pom.xml && git commit -q -am snapshot && git tag v1.3.2 + echo "" > pom.xml && git commit -q -am noprofiles && git tag v1.4.0 + git rm -q pom.xml && git commit -q -m nopom && git tag v1.5.0 + git checkout -q -b side && echo 3 > f && git add f && git commit -q -m three && git tag v9.9.9 + git checkout -q main +) +git clone -q "$P/upstream" "$P/repo" +V123="$(git -C "$P/upstream" rev-parse 'v1.2.3^{commit}')" +RC1="$(git -C "$P/upstream" rev-parse 'v1.3.0-rc1^{commit}')" +OLDPATH="$PATH" +PATH="$P/bin:$PATH" +plan() { # plan EVENT REF REF_TYPE REF_NAME TAG DRY SKIP [EXPECTED] + run "$1" env EVENT_NAME="$2" REF="$3" REF_TYPE="$4" REF_NAME="$5" INPUT_TAG="$6" INPUT_DRY_RUN="$7" \ + INPUT_SKIP_CENTRAL="$8" INPUT_EXPECTED_SPARK="${9:-}" DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +} +dispatch() { plan "$1" workflow_dispatch refs/heads/main branch main "$2" "$3" false "${4:-}"; } # dispatch TAG DRY [EXPECTED] +L="$P/log" +check "plan: dispatch with dry-run ticked is a dry run" plan "$L.1" workflow_dispatch refs/heads/main branch main v1.2.3 true false +check "plan: ... mode dry-run" is "$(out "$L.1" mode)" dry-run +check "plan: ... tag resolved to its commit" is "$(out "$L.1" commit)" "$V123" +check "plan: ... base version" is "$(out "$L.1" base)" 1.2.3 +check "plan: ... versions from pom.xml at the tag, one per profile" is "$(out "$L.1" versions)" "1.2.3_spark_3.5.8 1.2.3_spark_4.0.3 1.2.3_spark_4.1.2" +check "plan: ... not a pre-release" is "$(out "$L.1" prerelease)" false +check "plan: ... summary says nothing is uploaded" has "$L.1.summary" "Nothing is uploaded" +check "plan: ... summary lists each version with the one last published" has "$L.1.summary" '| `1.2.3_spark_3.5.8` | 3.5.8 | unchanged |' +check "plan: ... and gives the value for expected-spark-versions" has "$L.1.summary" '`3.5.8 4.0.3 4.1.2`' +check "plan: dispatch from main with dry-run unticked publishes" dispatch "$L.2" v1.2.3 false +check "plan: ... mode publish" is "$(out "$L.2" mode)" publish +check "plan: ... Central included" is "$(out "$L.2" central)" true +check "plan: ... only the Maven Central metadata was fetched" is "$(sort -u "$PFAKE/calls.log")" "https://repo1.maven.org/maven2/io/indextables/indextables_spark/maven-metadata.xml" + +# The Spark version in the artifact names changed since the last release. +check "plan: a dry run with changed Spark versions goes ahead" dispatch "$L.20" v1.3.0-rc1 true +check "plan: ... with a warning" has "$L.20" "::warning::The Spark version in the artifact names differs from the last release for Spark 3.5" +check "plan: ... the summary marks the changed line" has "$L.20.summary" '| `1.3.0-rc1_spark_3.5.9` | 3.5.8 | **changed** from 3.5.8 |' +check "plan: ... and leaves the others as unchanged" has "$L.20.summary" '| `1.3.0-rc1_spark_4.0.3` | 4.0.3 | unchanged |' +check "plan: ... and says what a publish run needs" has "$L.20.summary" 'set `expected-spark-versions` to `3.5.9 4.0.3 4.1.2`' +check "plan: publishing changed Spark versions without confirming them is refused" fails_with "dispatch again with expected-spark-versions set to: 3.5.9 4.0.3 4.1.2" "$L.21" dispatch "$L.21" v1.3.0-rc1 false +check "plan: ... and sets no mode" test -z "$(out "$L.21" mode)" +check "plan: ... the message names what would be published" has "$L.21" "1.3.0-rc1_spark_3.5.9 1.3.0-rc1_spark_4.0.3 1.3.0-rc1_spark_4.1.2" +check "plan: publishing them with expected-spark-versions goes ahead" dispatch "$L.22" v1.3.0-rc1 false "3.5.9 4.0.3 4.1.2" +check "plan: ... mode publish" is "$(out "$L.22" mode)" publish +check "plan: ... annotated tag resolved to its commit" is "$(out "$L.22" commit)" "$RC1" +check "plan: ... pre-release" is "$(out "$L.22" prerelease)" true +check "plan: ... versions" is "$(out "$L.22" versions)" "1.3.0-rc1_spark_3.5.9 1.3.0-rc1_spark_4.0.3 1.3.0-rc1_spark_4.1.2" +check "plan: ... the summary says they were confirmed" has "$L.22.summary" "confirmed with" +check "plan: expected-spark-versions in another order, with commas" dispatch "$L.23" v1.3.0-rc1 false "4.1.2, 4.0.3,3.5.9" +check "plan: expected-spark-versions that does not match pom.xml is refused" fails_with "but pom.xml at v1.3.0-rc1 gives '3.5.9 4.0.3 4.1.2'" "$L.24" dispatch "$L.24" v1.3.0-rc1 false "3.5.8 4.0.3 4.1.2" +check "plan: ... in a dry run as well" fails_with "but pom.xml at v1.3.0-rc1 gives" "$L.25" dispatch "$L.25" v1.3.0-rc1 true "3.5.8 4.0.3 4.1.2" +check "plan: ... also when the versions are unchanged" fails_with "but pom.xml at v1.2.3 gives" "$L.26" dispatch "$L.26" v1.2.3 false "3.5.9 4.0.3 4.1.2" +check "plan: an incomplete expected-spark-versions is refused" fails_with "but pom.xml at" "$L.27" dispatch "$L.27" v1.3.0-rc1 false "3.5.9" +central_has 0.9.0_spark_3.5.3 1.0.0_spark_3.5.8 1.0.0_spark_4.0.3 +check "plan: a Spark line that was never published is not a change" dispatch "$L.28" v1.2.3 false +check "plan: ... the summary says it is the first" has "$L.28.summary" "first release for Spark 4.1" +central_has 0.9.0_spark_3.5.3 1.0.0_spark_3.5.8 1.0.0_spark_4.0.3 1.0.0_spark_4.1.2 1.1.0_spark_3.5.9 1.1.0_spark_4.0.3 1.1.0_spark_4.1.2 +check "plan: the comparison is with the version published last" fails_with "expected-spark-versions set to: 3.5.8 4.0.3 4.1.2" "$L.29" dispatch "$L.29" v1.2.3 false +rm "$PFAKE/metadata.xml" +check "plan: publishing is refused when Maven Central cannot be read" fails_with "could not be read" "$L.30" dispatch "$L.30" v1.2.3 false +check "plan: ... unless the versions are confirmed" dispatch "$L.31" v1.2.3 false "3.5.8 4.0.3 4.1.2" +check "plan: ... a dry run goes ahead with a warning" dispatch "$L.32" v1.2.3 true +check "plan: ... (the warning)" has "$L.32" "::warning::The versions last published to Maven Central could not be read" +central_has 0.9.0_spark_3.5.3 1.0.0_spark_3.5.8 1.0.0_spark_4.0.3 1.0.0_spark_4.1.2 +check "plan: a profile whose Spark version is outside its Spark line is refused" fails_with "version '1.3.1_spark_4.1.9' is not 1.3.1_spark_4.0." "$L.33a" dispatch "$L.33a" v1.3.1 true +check "plan: a Spark version that is not a plain release is refused" fails_with "is not 1.3.2_spark_4.1." "$L.33b" dispatch "$L.33b" v1.3.2 true +check "plan: a pom.xml without the Spark profiles is refused" fails_with "pom.xml at the tag has no profile spark-3.5" "$L.33" dispatch "$L.33" v1.4.0 true +check "plan: a commit without pom.xml is refused" fails_with "the tagged commit has no pom.xml" "$L.34" dispatch "$L.34" v1.5.0 true + +check "plan: publish from another branch is refused" fails_with "can only be dispatched from main" "$L.3" \ + env EVENT_NAME=workflow_dispatch REF=refs/heads/feature/x INPUT_TAG=v1.2.3 INPUT_DRY_RUN=false DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +check "plan: ... and sets no mode" test -z "$(out "$L.3" mode)" +check "plan: publish from a tag ref is refused" fails_with "can only be dispatched from main" "$L.3b" \ + env EVENT_NAME=workflow_dispatch REF=refs/tags/v1.2.3 INPUT_TAG=v1.2.3 INPUT_DRY_RUN=false DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +check "plan: a dry run may be dispatched from another branch" plan "$L.4" workflow_dispatch refs/heads/feature/x branch feature/x v1.2.3 true false +check "plan: ... mode dry-run" is "$(out "$L.4" mode)" dry-run +for v in "" TRUE False 0 1 yes "true " "false "; do + check "plan: dry-run '$v' is refused" fails_with "dry-run must be true or false" "$L.5" \ + env EVENT_NAME=workflow_dispatch REF=refs/heads/main INPUT_TAG=v1.2.3 INPUT_DRY_RUN="$v" DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +done +check "plan: skip-central in a publish run" plan "$L.6" workflow_dispatch refs/heads/main branch main v1.2.3 false true +check "plan: ... Central skipped" is "$(out "$L.6" central)" false +check "plan: skip-central has no effect in a dry run" plan "$L.7" workflow_dispatch refs/heads/main branch main v1.2.3 true true +check "plan: ... Central not skipped" is "$(out "$L.7" central)" true +check "plan: a pushed tag publishes (when that trigger is enabled)" plan "$L.8" push refs/tags/v1.2.3 tag v1.2.3 "" "" "" +check "plan: ... mode publish" is "$(out "$L.8" mode)" publish +check "plan: a pushed branch is refused" fails_with "only supported for tags" "$L.9" \ + env EVENT_NAME=push REF=refs/heads/main REF_TYPE=branch REF_NAME=main DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +check "plan: ... also when the branch is named like a release tag" bash -c "! EVENT_NAME=push REF=refs/heads/v1.2.3 REF_TYPE=branch REF_NAME=v1.2.3 DEFAULT_BRANCH=main REPO_DIR='$P/repo' GITHUB_OUTPUT='$L.9b.out' bash '$here/plan.sh'" +check "plan: ... and sets no mode" test -z "$(out "$L.9b" mode)" +for e in pull_request pull_request_target schedule release workflow_run ""; do + check "plan: event '$e' is refused" bash -c "! EVENT_NAME='$e' REF=refs/heads/main DEFAULT_BRANCH=main INPUT_TAG=v1.2.3 INPUT_DRY_RUN=false REPO_DIR='$P/repo' bash '$here/plan.sh'" +done +for t in main v1.2 1.2.3 v1.2.3_spark_3.5.8 'v1.2.3;id' 'v1.2.3 ' v1.2.3-rc_1 '../v1.2.3' ''; do + check "plan: tag '$t' is refused" fails_with "is not a release tag" "$L.10" \ + env EVENT_NAME=workflow_dispatch REF=refs/heads/main INPUT_TAG="$t" INPUT_DRY_RUN=true DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +done +check "plan: a tag that does not exist is refused" fails_with "does not exist" "$L.11" \ + env EVENT_NAME=workflow_dispatch REF=refs/heads/main INPUT_TAG=v7.7.7 INPUT_DRY_RUN=true DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +check "plan: a tag that is not on main is refused" fails_with "is not an ancestor of main" "$L.12" \ + env EVENT_NAME=workflow_dispatch REF=refs/heads/main INPUT_TAG=v9.9.9 INPUT_DRY_RUN=false DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +check "plan: ... in a dry run as well" fails_with "is not an ancestor of main" "$L.13" \ + env EVENT_NAME=workflow_dispatch REF=refs/heads/main INPUT_TAG=v9.9.9 INPUT_DRY_RUN=true DEFAULT_BRANCH=main REPO_DIR="$P/repo" bash "$here/plan.sh" +PATH="$OLDPATH" + +# --------------------------------------------------------------------------- +# Stubs for the build tools. $FAKE holds fixtures and switches. +# --------------------------------------------------------------------------- +mkdir -p "$T/bin" "$T/fx" +FAKE="$T/fake"; mkdir -p "$FAKE"; export FAKE + +# Fixture jars: a tantivy4java jar with a native library, and project jars. +mkdir -p "$T/fx/t4j/native" "$T/fx/t4j/io" +echo "native library built in this run" > "$T/fx/t4j/native/libtantivy4java.so" +echo class > "$T/fx/t4j/io/A.class" +(cd "$T/fx/t4j" && zip -q -r "$T/fx/t4j.jar" .) +mkdir -p "$T/fx/other/native" +echo "some other native library" > "$T/fx/other/native/libtantivy4java.so" +(cd "$T/fx/other" && zip -q -r "$T/fx/other-native.jar" .) +mkdir -p "$T/fx/plain" && echo x > "$T/fx/plain/X.class" +(cd "$T/fx/plain" && zip -q -r "$T/fx/plain.jar" .) +cp "$T/fx/t4j.jar" "$T/fx/fat.jar" && (cd "$T/fx/plain" && zip -q "$T/fx/fat.jar" X.class) +export FX="$T/fx" + +cat > "$T/bin/mvn" << 'STUB' +#!/usr/bin/env bash +# Stub Maven: enough of tantivy4java's `clean install` and of this project's +# help:evaluate, versions:set and `verify` for the release scripts. +set -eu +echo "mvn $*" >> "$FAKE/mvn.log" +repo=""; profile=""; newversion=""; goal="" +for a in "$@"; do + case "$a" in + -Dmaven.repo.local=*) repo="${a#-Dmaven.repo.local=}" ;; + -P*) profile="${a#-P}" ;; + -DnewVersion=*) newversion="${a#-DnewVersion=}" ;; + --version) echo "Apache Maven 3.9.9 (stub)"; exit 0 ;; + *:evaluate) goal=evaluate ;; + *:set) goal=set ;; + install) goal=install ;; + verify) goal=verify ;; + deploy | *:deploy | *:publish | *:sign) echo "stub mvn: refusing $a" >&2; exit 96 ;; + esac +done +case "$goal" in + install) # tantivy4java + v="$(cat "$FAKE/t4j.version")" + d="$repo/io/indextables/tantivy4java/$v" + mkdir -p "$d" + cp "${FAKE_T4J_JAR:-$FX/t4j.jar}" "$d/tantivy4java-$v-linux-x86_64.jar" + cp "${FAKE_T4J_JAR:-$FX/t4j.jar}" "$d/tantivy4java-$v.jar" + echo "" > "$d/tantivy4java-$v.pom" + echo "#stub" > "$d/_remote.repositories" + if [ -f "$FAKE/touch-lock" ]; then echo "# changed by the build" >> native/Cargo.lock; fi + ;; + evaluate) + cat "$FAKE/spark.${profile}" + ;; + set) + echo "$newversion" > .stamped + ;; + verify) + case " $* " in *" -Dgpg.skip=true "*) ;; *) echo "stub mvn: verify without -Dgpg.skip=true" >&2; exit 95 ;; esac + v="$(cat .stamped)"; p="indextables_spark-$v" + mkdir -p target + cp "$FX/plain.jar" "target/$p.jar" + cp "$FX/plain.jar" "target/$p-sources.jar" + [ -f "$FAKE/no-javadoc" ] || cp "$FX/plain.jar" "target/$p-javadoc.jar" + cp "${FAKE_FAT_JAR:-$FX/fat.jar}" "target/$p-linux-x86_64-shaded.jar" + cp "${FAKE_FAT_JAR:-$FX/fat.jar}" "target/$p-linux-x86_64-jar-with-dependencies.jar" + [ ! -f "$FAKE/extra-jar" ] || cp "$FX/plain.jar" "target/$p-tests.jar" + if [ -f "$FAKE/swap-t4j" ]; then + t="$(cat "$FAKE/t4j.version")" + cp "$FX/other-native.jar" "$repo/io/indextables/tantivy4java/$t/tantivy4java-$t-linux-x86_64.jar" + fi + sed -e "s/@VERSION@/$v/" "${FAKE_POM:-$FX/published.pom}" > dependency-reduced-pom.xml + ;; + *) echo "stub mvn: unexpected invocation: $*" >&2; exit 97 ;; +esac +STUB +for tool in java cargo rustc; do + printf '#!/usr/bin/env bash\necho "%s 1.0 (stub)" >&2\necho "%s 1.0 (stub)"\n' "$tool" "$tool" > "$T/bin/$tool" +done +printf '#!/usr/bin/env bash\ncase "$1" in --exists) exit 0 ;; --modversion) echo 3.0.0 ;; esac\n' > "$T/bin/pkg-config" +chmod +x "$T/bin/"* +STUBPATH="$T/bin:$PATH" + +cat > "$T/fx/published.pom" << 'EOF' + + + 4.0.0 + io.indextables + indextables_spark + @VERSION@ + IndexTables4Spark + Fast search file format + https://example.invalid/ + Apache License, Version 2.0https://www.apache.org/licenses/LICENSE-2.0.txt + Contributors + scm:git:git://example.invalid/x.githttps://example.invalid/x + +EOF + +# --------------------------------------------------------------------------- +# build-native.sh +# --------------------------------------------------------------------------- +N="$T/native"; mkdir -p "$N" +MR=3333333333333333333333333333333333333333 +new_fork() { # new_fork : a fork-like repository, one commit on main and one on a side branch + rm -rf "$1"; git init -q -b main "$1" + (cd "$1" && echo "$1" > f && git add f && git commit -q -m main && git checkout -q -b upgrade && echo side >> f && git commit -q -am side && git checkout -q main) + git -C "$1" config uploadpack.allowAnySHA1InWant true + git -C "$1" config uploadpack.allowFilter true +} +new_fork "$N/quickwit"; new_fork "$N/tantivy" +QW="$(git -C "$N/quickwit" rev-parse main)"; QW_SIDE="$(git -C "$N/quickwit" rev-parse upgrade)" +TV="$(git -C "$N/tantivy" rev-parse main)" +# The scripts always name the real repositories; git is told to fetch them +# from the fixtures instead. +redirect() { echo "GIT_CONFIG_KEY_0=url.file://$1.insteadOf"; } +new_t4j() { # new_t4j : a tantivy4java-like repository with tag v0.34.4 + rm -rf "$1"; git init -q -b main "$1" + mkdir -p "$1/native" "$1/.cargo" + cat > "$1/native/Cargo.toml" << EOF +[package] +name = "tantivy4java" +[dependencies] +tantivy = { git = "https://github.com/indextables/tantivy", rev = "$TV" } +quickwit-storage = { git = "https://github.com/indextables/quickwit", rev = "$QW" } +EOF + cat > "$1/native/Cargo.lock" << EOF +version = 3 +[[package]] +name = "quickwit-storage" +source = "git+https://github.com/indextables/quickwit?rev=$QW#$QW" +[[package]] +name = "quickwit-proto" +source = "git+https://github.com/indextables/quickwit?rev=$QW#$QW" +[[package]] +name = "tantivy" +source = "git+https://github.com/indextables/tantivy?rev=$TV#$TV" +[[package]] +name = "mrecordlog" +source = "git+https://github.com/quickwit-oss/mrecordlog?rev=3333333#$MR" +[[package]] +name = "serde" +source = "registry+https://github.com/rust-lang/crates.io-index" +EOF + printf '[target.x86_64-unknown-linux-gnu]\nrustflags = ["-C", "target-cpu=x86-64"]\n' > "$1/.cargo/config.toml" + git -C "$1" add -A && git -C "$1" commit -q -m release + git -C "$1" config uploadpack.allowAnySHA1InWant true +} +new_t4j "$N/t4j" +git -C "$N/t4j" tag v0.34.4 +T4J_COMMIT="$(git -C "$N/t4j" rev-parse HEAD)" +printf '\n tantivy4java\n 0.34.4\n\n' > "$N/pom.xml" +printf '# comment\n0.34.4 %s %s %s\n0.1.0 %s - -\n' "$T4J_COMMIT" "$QW" "$TV" "$SHA_A" > "$N/pins.txt" +echo 0.34.4 > "$FAKE/t4j.version" +native() { # native [VAR=value...] + local log="$1" sub="$2"; shift 2 + run "$log" env PATH="$STUBPATH" HOME="$N/home" GIT_CONFIG_COUNT=1 "$(redirect "$N/t4j")" \ + GIT_CONFIG_VALUE_0=https://github.com/indextables/tantivy4java.git \ + POM="$N/pom.xml" PINS="$N/pins.txt" WORK="$N/work" OUT="$N/out" "$@" bash "$here/build-native.sh" "$sub" +} +pin() { # pin : print-pin, with all three repositories redirected + run "$1" env PATH="$STUBPATH" HOME="$N/home" GIT_CONFIG_COUNT=3 "$(redirect "$2")" \ + GIT_CONFIG_VALUE_0=https://github.com/indextables/tantivy4java.git \ + GIT_CONFIG_KEY_1="url.file://$N/quickwit.insteadOf" GIT_CONFIG_VALUE_1=https://github.com/indextables/quickwit \ + GIT_CONFIG_KEY_2="url.file://$N/tantivy.insteadOf" GIT_CONFIG_VALUE_2=https://github.com/indextables/tantivy \ + bash "$here/build-native.sh" print-pin 0.34.4 +} +L="$N/log" +check "native: resolve accepts the pinned tag" native "$L.1" resolve +check "native: ... records the tantivy4java commit" has "$N/out/native-inputs.txt" "tantivy4java_commit=$T4J_COMMIT" +check "native: ... records the quickwit fork commit" has "$N/out/native-inputs.txt" "quickwit_commit=$QW" +check "native: ... records the tantivy fork commit" has "$N/out/native-inputs.txt" "tantivy_commit=$TV" +check "native: ... records other git dependencies with their commit" has "$N/out/native-inputs.txt" "other_git_dependencies=https://github.com/quickwit-oss/mrecordlog@$MR" +check "native: ... checked out exactly the pinned commit" is "$(git -C "$N/work/tantivy4java" rev-parse HEAD)" "$T4J_COMMIT" +# shellcheck disable=SC2086 +check "native: no variable redirects the repositories or the local Maven repository" test -z "$(grep -n -E 'T4J_REPO|M2_REPO:-|TANTIVY4JAVA_URL:-|QUICKWIT_URL:-|TANTIVY_URL:-' $SCRIPTS_UNDER_TEST)" +check "native: ... T4J_REPO and M2_REPO in the environment changed nothing" test "$T4J_REPO" = file:///override/invalid -a "$M2_REPO" = /override/invalid -a ! -e /override +check "native: print-pin prints the pin line" pin "$L.p1" "$N/t4j" +check "native: ... (the line)" test "$(grep -c -x "0.34.4 $T4J_COMMIT $QW $TV" "$L.p1")" = 1 +check "native: ... says the quickwit commit is on the fork's default branch" has "$L.p1" "quickwit $QW: reachable from the default branch (main) of https://github.com/indextables/quickwit" +check "native: ... and the tantivy commit" has "$L.p1" "tantivy $TV: reachable from the default branch (main) of https://github.com/indextables/tantivy" + +check "native: build needs the toolchain step first" fails_with "toolchain" "$L.2" native "$L.2" build +mkdir -p "$N/work/protoc/bin" "$N/work/protoc/include" && printf '#!/bin/sh\necho "libprotoc 25.5"\n' > "$N/work/protoc/bin/protoc" && chmod +x "$N/work/protoc/bin/protoc" +mkdir -p "$N/home/.m2/repository/io/indextables/tantivy4java/0.34.4" && echo stale > "$N/home/.m2/repository/io/indextables/tantivy4java/0.34.4/tantivy4java-0.34.4-linux-x86_64.jar" +echo stale > "$N/home/.m2/repository/io/indextables/tantivy4java/0.34.4/tantivy4java-0.34.4-left-over.jar" +check "native: build collects the jar" native "$L.3" build +check "native: ... a jar that was there before the build is not reused" has "$L.3" "removing it before the build" +check "native: ... the collected jar is the one this build installed" is "$(hash_of sha256 "$N/out/repository/tantivy4java-0.34.4-linux-x86_64.jar")" "$(hash_of sha256 "$T/fx/t4j.jar")" +check "native: ... nothing that was there before the build is collected" test ! -e "$N/out/repository/tantivy4java-0.34.4-left-over.jar" +check "native: ... Maven bookkeeping files are left out" test ! -e "$N/out/repository/_remote.repositories" +check "native: ... digests cover the output" bash -c ". '$here/lib.sh'; verify_sums '$N/out'" +check "native: ... records an unchanged Cargo.lock" has "$N/out/native-inputs.txt" "cargo_lock_unchanged=true" +check "native: ... records the native library digest" has "$N/out/native-inputs.txt" "native_library=native/libtantivy4java.so sha256 $(hash_of sha256 "$T/fx/t4j/native/libtantivy4java.so")" +check "native: ... summary lists the commits" has "$L.3.summary" "$T4J_COMMIT" +cp -R "$N/out" "$N/out.good" + +touch "$FAKE/touch-lock" +check "native: a build that rewrites Cargo.lock is refused" fails_with "modified native/Cargo.lock" "$L.4" native "$L.4" build +rm "$FAKE/touch-lock" +git -C "$N/work/tantivy4java" checkout -q -- native/Cargo.lock +check "native: a jar without a native library is refused" fails_with "contains no native library" "$L.5" native "$L.5" build FAKE_T4J_JAR="$T/fx/plain.jar" + +printf '0.34.4 %s %s %s\n' "$SHA_A" "$QW" "$TV" > "$N/pins.moved" +check "native: a tag that does not point to the pinned commit is refused" fails_with "points to $T4J_COMMIT, but native-pins.txt pins $SHA_A" "$L.6" native "$L.6" resolve PINS="$N/pins.moved" +printf '0.1.0 %s - -\n' "$SHA_A" > "$N/pins.none" +check "native: a version without a pin is refused" fails_with "exactly one line for tantivy4java 0.34.4" "$L.7" native "$L.7" resolve PINS="$N/pins.none" +printf '0.34.4 %s %s %s\n0.34.4 %s %s %s\n' "$T4J_COMMIT" "$QW" "$TV" "$T4J_COMMIT" "$QW" "$TV" > "$N/pins.twice" +check "native: a version pinned twice is refused" fails_with "exactly one line" "$L.8" native "$L.8" resolve PINS="$N/pins.twice" +printf '0.34.4 %s %s %s\n' "$T4J_COMMIT" "$SHA_B" "$TV" > "$N/pins.qw" +check "native: a quickwit fork commit other than the pinned one is refused" fails_with "resolves the quickwit fork to $QW" "$L.9" native "$L.9" resolve PINS="$N/pins.qw" +printf '0.34.4 %s %s %s\n' "$T4J_COMMIT" "$QW" "$SHA_B" > "$N/pins.tv" +check "native: a tantivy fork commit other than the pinned one is refused" fails_with "resolves the tantivy fork to $TV" "$L.10" native "$L.10" resolve PINS="$N/pins.tv" +printf '0.34.4 %s main %s\n' "$T4J_COMMIT" "$TV" > "$N/pins.branch" +check "native: a pin that is not a commit id is refused" fails_with "not a full commit id" "$L.11" native "$L.11" resolve PINS="$N/pins.branch" + +variant() { # variant : a repository that differs in one way + new_t4j "$N/v-$1"; (cd "$N/v-$1" && eval "$2" && git add -A && git commit -q -m variant && git tag v0.34.4) + printf '0.34.4 %s %s %s\n' "$(git -C "$N/v-$1" rev-parse HEAD)" "$QW" "$TV" > "$N/pins.$1" +} +variant path "echo 'quickwit-common = { path = \"../../quickwit/quickwit-common\" }' >> native/Cargo.toml" +check "native: a dependency on a sibling checkout is refused" fails_with "outside the tantivy4java checkout" "$L.12" native "$L.12" resolve "$(redirect "$N/v-path")" PINS="$N/pins.path" +variant patch "printf '[patch.crates-io]\nserde = { path = \"/tmp/serde\" }\n' >> .cargo/config.toml" +check "native: a cargo config that overrides sources is refused" fails_with "overrides dependency sources" "$L.13" native "$L.13" resolve "$(redirect "$N/v-patch")" PINS="$N/pins.patch" +variant two "printf '[[package]]\nname = \"quickwit-x\"\nsource = \"git+https://github.com/indextables/quickwit?rev=$SHA_C#$SHA_C\"\n' >> native/Cargo.lock" +check "native: a lock file with two quickwit commits is refused" fails_with "more than one commit" "$L.14" native "$L.14" resolve "$(redirect "$N/v-two")" PINS="$N/pins.two" +variant short "printf '[[package]]\nname = \"y\"\nsource = \"git+https://github.com/indextables/quickwit?branch=main\"\n' >> native/Cargo.lock" +check "native: a git dependency without a resolved commit is refused" fails_with "without a resolved commit" "$L.15" native "$L.15" resolve "$(redirect "$N/v-short")" PINS="$N/pins.short" +variant side "sed -i.bak 's/$QW/$QW_SIDE/g' native/Cargo.lock native/Cargo.toml && rm native/Cargo.lock.bak native/Cargo.toml.bak" +check "native: print-pin accepts a fork commit on another branch" pin "$L.p2" "$N/v-side" +check "native: ... but says it is not on the default branch, and where it is" has "$L.p2" "quickwit $QW_SIDE is NOT reachable from the default branch (main) of https://github.com/indextables/quickwit; it is on: upgrade" +variant orphan "sed -i.bak 's/$QW/$SHA_C/g' native/Cargo.lock native/Cargo.toml && rm native/Cargo.lock.bak native/Cargo.toml.bak" +check "native: print-pin refuses a fork commit that is on no branch or tag of the fork" fails_with "is not reachable from any branch or tag" "$L.p3" pin "$L.p3" "$N/v-orphan" +check "native: ... and prints no pin line" test -z "$(grep -E '^0\.34\.4 ' "$L.p3")" +new_t4j "$N/v-annot"; git -C "$N/v-annot" tag -a v0.34.4 -m annotated +printf '0.34.4 %s %s %s\n' "$(git -C "$N/v-annot" rev-parse HEAD)" "$QW" "$TV" > "$N/pins.annot" +check "native: an annotated tag is resolved to its commit" native "$L.16" resolve "$(redirect "$N/v-annot")" PINS="$N/pins.annot" + +# toolchain: a download with the wrong digest is never unpacked. +cat > "$T/bin/curl" << 'STUB' +#!/usr/bin/env bash +# Stub curl for the protoc download: writes a small file to the -o target. +o=""; while [ $# -gt 0 ]; do case "$1" in -o) o="$2"; shift 2 ;; *) shift ;; esac; done +echo "not protoc" > "$o" +STUB +chmod +x "$T/bin/curl" +native "$L.0" resolve > /dev/null 2>&1 +check "native: a protoc download with the wrong digest is refused" fails_with "expected e1ed237a17b2e851cf9662cb5ad02b46e70ff8e060e05984725bc4b4228c6b28" "$L.17" native "$L.17" toolchain +check "native: ... and is not unpacked" test ! -e "$N/work/protoc/bin/protoc" +rm "$T/bin/curl" + +check "pins: protoc version is the one scripts/setup.sh uses" is \ + "$(sed -n 's/^PROTOC_VERSION=//p' "$here/build-native.sh")" "$(sed -n 's/^PROTOC_VERSION="\(.*\)"$/\1/p' "$root/scripts/setup.sh")" +check "pins: protoc linux-x86_64 digest is the one scripts/setup.sh uses" is \ + "$(sed -n 's/^PROTOC_SHA256=//p' "$here/build-native.sh")" "$(sed -n 's/^ *linux-x86_64) *echo "\([0-9a-f]*\)".*/\1/p' "$root/scripts/setup.sh")" +pom_t4j="$(pom_tantivy4java_version "$root/pom.xml" 2> /dev/null)" +check "pins: pom.xml's tantivy4java version ($pom_t4j) has exactly one pin" is \ + "$(awk -v v="$pom_t4j" '!/^[[:space:]]*#/ && $1 == v && NF == 4' "$here/native-pins.txt" | grep -c .)" 1 +check "pins: every pin is a full commit id or '-'" test -z \ + "$(grep -v '^[[:space:]]*#' "$here/native-pins.txt" | grep -v -E '^[0-9]+\.[0-9]+\.[0-9]+ [0-9a-f]{40} ([0-9a-f]{40}|-) ([0-9a-f]{40}|-)$')" + +# --------------------------------------------------------------------------- +# build-artifacts.sh +# --------------------------------------------------------------------------- +B="$T/build"; mkdir -p "$B" +echo 3.5.8 > "$FAKE/spark.spark-3.5"; echo 4.0.3 > "$FAKE/spark.spark-4.0"; echo 4.1.2 > "$FAKE/spark.spark-4.1" +artifacts() { # artifacts [VAR=value...] + local log="$1" profile="$2"; shift 2 + rm -rf "$B/src-$profile" "$B/home-$profile"; mkdir -p "$B/src-$profile"; cp "$N/pom.xml" "$B/src-$profile/pom.xml" + run "$log" env PATH="$STUBPATH" SRC="$B/src-$profile" PROFILE="$profile" BASE_VERSION=1.2.3 EXPECTED_VERSIONS="$EV" NATIVE_DIR="$N/out.good" \ + OUT="$B/out/artifacts-$profile" HOME="$B/home-$profile" "$@" bash "$here/build-artifacts.sh" +} +L="$B/log" +: > "$FAKE/mvn.log" +check "build: spark-3.5" artifacts "$L.35" spark-3.5 +check "build: spark-4.0" artifacts "$L.40" spark-4.0 +check "build: spark-4.1" artifacts "$L.41" spark-4.1 +check "build: version is _spark_" is "$(out "$L.40" version)" 1.2.3_spark_4.0.3 +check "build: files carry their published names" is "$(ls "$B/out/artifacts-spark-4.0/files" | LC_ALL=C sort | tr '\n' ' ')" "$(artifact_names 1.2.3_spark_4.0.3 | LC_ALL=C sort | tr '\n' ' ')" +check "build: the published pom is the dependency-reduced pom" has "$B/out/artifacts-spark-4.0/files/indextables_spark-1.2.3_spark_4.0.3.pom" "1.2.3_spark_4.0.3" +check "build: digests cover the output" bash -c ". '$here/lib.sh'; verify_sums '$B/out/artifacts-spark-4.0'" +check "build: Maven is never asked to sign, deploy or publish" test -z "$(grep -E '(^| )(deploy|install)( |$)|:deploy|:publish|:sign' "$FAKE/mvn.log")" +check "build: the release profile runs with signing off" has "$FAKE/mvn.log" "-Pspark-4.0,release -DskipTests -Dgpg.skip=true verify" +check "build: plugins invoked by coordinate are pinned" test -z "$(grep -E ':(evaluate|set)' "$FAKE/mvn.log" | grep -v -E '[a-z.-]+:[a-z-]+:[0-9.]+:(evaluate|set)')" +cp -R "$B/out" "$B/out.good" + +mkdir -p "$B/home-pre/.m2/repository/io/indextables/tantivy4java/0.34.4" +check "build: a tantivy4java already in the local repository is refused" fails_with "already in the local Maven repository" "$L.1" \ + env PATH="$STUBPATH" SRC="$B/src-spark-3.5" PROFILE=spark-3.5 BASE_VERSION=1.2.3 EXPECTED_VERSIONS="$EV" NATIVE_DIR="$N/out.good" OUT="$B/out/x" HOME="$B/home-pre" bash "$here/build-artifacts.sh" +cp -R "$N/out.good" "$B/native.bad" && echo tampered >> "$B/native.bad/repository/tantivy4java-0.34.4-linux-x86_64.jar" +check "build: a native artifact that does not match its digests is refused" fails_with "digest mismatch" "$L.2" artifacts "$L.2" spark-3.5 NATIVE_DIR="$B/native.bad" +echo 4.0.3 > "$FAKE/spark.spark-3.5" +check "build: a Spark version outside the profile's line is refused" fails_with "is not 1.2.3_spark_3.5." "$L.3" artifacts "$L.3" spark-3.5 +echo 3.5.9 > "$FAKE/spark.spark-3.5" +check "build: a version other than the one the plan announced is refused" fails_with "Maven derives 1.2.3_spark_3.5.9 for spark-3.5, but the plan announced 1.2.3_spark_3.5.8" "$L.3b" artifacts "$L.3b" spark-3.5 +echo 3.5.8 > "$FAKE/spark.spark-3.5" +check "build: an unknown profile is refused" fails_with "unknown Spark profile" "$L.4" artifacts "$L.4" spark-5.0 +touch "$FAKE/no-javadoc" +check "build: a missing javadoc jar is refused" fails_with "did not produce" "$L.5" artifacts "$L.5" spark-3.5 +rm "$FAKE/no-javadoc"; touch "$FAKE/extra-jar" +check "build: a jar the scripts do not know is refused" fails_with "which the release scripts do not know" "$L.6" artifacts "$L.6" spark-3.5 +rm "$FAKE/extra-jar"; touch "$FAKE/swap-t4j" +check "build: a tantivy4java jar replaced during the build is refused" fails_with "changed during the build" "$L.7" artifacts "$L.7" spark-3.5 +rm "$FAKE/swap-t4j" +rm -rf "$B/out" && cp -R "$B/out.good" "$B/out" + +# --------------------------------------------------------------------------- +# assemble.sh and verify-staging.sh +# --------------------------------------------------------------------------- +A="$T/assemble"; mkdir -p "$A" +new_in() { rm -rf "$A/in"; mkdir -p "$A/in"; cp -R "$N/out.good" "$A/in/native"; cp -R "$B/out.good/"* "$A/in/"; } +assemble() { # assemble [VAR=value...] + local log="$1"; shift + run "$log" env IN="$A/in" OUT="$A/staging" TAG=v1.2.3 COMMIT="$SHA_D" BASE_VERSION=1.2.3 EXPECTED_VERSIONS="$EV" RUN_URL=https://example.invalid/run/1 "$@" bash "$here/assemble.sh" +} +resum() { (. "$here/lib.sh"; write_sums "$1"); } +L="$A/log" +new_in +check "assemble: accepts the three builds" assemble "$L.1" +check "assemble: versions, in profile order" is "$(out "$L.1" versions)" "1.2.3_spark_3.5.8 1.2.3_spark_4.0.3 1.2.3_spark_4.1.2" +vdir="$A/staging/bundles/1.2.3_spark_4.0.3/io/indextables/indextables_spark/1.2.3_spark_4.0.3" +check "assemble: Maven repository layout, six files and four checksums each" is "$(ls "$vdir" | grep -c .)" 30 +check "assemble: no signatures yet" test -z "$(find "$A/staging" -name '*.asc')" +check "assemble: checksum files hold the bare digest" is "$(cat "$vdir/indextables_spark-1.2.3_spark_4.0.3.pom.sha1")" "$(hash_of sha1 "$vdir/indextables_spark-1.2.3_spark_4.0.3.pom")" +check "assemble: ... without a trailing newline" is "$(wc -c < "$vdir/indextables_spark-1.2.3_spark_4.0.3.pom.md5" | tr -d ' ')" 32 +check "assemble: manifest digest is reported" is "$(out "$L.1" manifest_sha256)" "$(hash_of sha256 "$A/staging/MANIFEST.sha256")" +check "assemble: notes name the tantivy4java commit" has "$A/staging/release-notes.md" "\`v0.34.4\` at \`$T4J_COMMIT\`" +check "assemble: notes name the fork commits" test -n "$(grep -F "$QW" "$A/staging/release-notes.md")" -a -n "$(grep -F "$TV" "$A/staging/release-notes.md")" +check "assemble: notes name the released commit" has "$A/staging/release-notes.md" "$SHA_D" +check "assemble: summary shows the manifest digest" has "$L.1.summary" "$(out "$L.1" manifest_sha256)" +MANIFEST="$(out "$L.1" manifest_sha256)" +rm -rf "$A/staging.good" && cp -R "$A/staging" "$A/staging.good" + +check "assemble: versions other than the ones the plan announced are refused" fails_with "is not the version the plan announced for spark-4.1" "$L.2a" assemble "$L.2a" EXPECTED_VERSIONS="1.2.3_spark_3.5.8 1.2.3_spark_4.0.3 1.2.3_spark_4.1.3" +check "assemble: a tag that does not match the base version is refused" fails_with "does not belong to tag" "$L.2" assemble "$L.2" TAG=v1.2.4 +new_in; rm -rf "$A/in/artifacts-spark-4.1" +check "assemble: a missing profile is refused" fails_with "no build output for spark-4.1" "$L.3" assemble "$L.3" +new_in; echo x > "$A/in/artifacts-spark-4.0/files/evil.jar"; resum "$A/in/artifacts-spark-4.0" +check "assemble: an extra file is refused, even with valid digests" fails_with "does not hold exactly the expected files" "$L.4" assemble "$L.4" +new_in; echo x >> "$A/in/artifacts-spark-4.0/files/indextables_spark-1.2.3_spark_4.0.3.jar" +check "assemble: a file changed after its digest was taken is refused" fails_with "digest mismatch" "$L.5" assemble "$L.5" +new_in; sed -i.bak 's/^version=.*/version=1.2.3_spark_4.1.2/' "$A/in/artifacts-spark-4.0/build-info.txt"; rm "$A/in/artifacts-spark-4.0/build-info.txt.bak"; resum "$A/in/artifacts-spark-4.0" +check "assemble: a version for another Spark line is refused" fails_with "is not 1.2.3_spark_4.0." "$L.6" assemble "$L.6" +pom_case() { # pom_case + new_in + local f="$A/in/artifacts-spark-3.5/files/indextables_spark-1.2.3_spark_3.5.8.pom" + sed -i.bak "$2" "$f"; rm "$f.bak"; resum "$A/in/artifacts-spark-3.5" + check "assemble: a pom with $1 is refused" fails_with "$3" "$L.pom" assemble "$L.pom" +} +pom_case "another groupId" 's|io.indextables|io.other|' "groupId is 'io.other'" +pom_case "another artifactId" 's|indextables_spark|tantivy4java|' "artifactId is 'tantivy4java'" +pom_case "another version" 's|1.2.3_spark_3.5.8|9.9.9|' "version is '9.9.9'" +pom_case "no license" 's|.*||' "missing licenses/license/name" +pom_case "no scm" 's|.*||' "missing scm/connection" +pom_case "a parent" 's||x|' "has a " +pom_case "broken XML" 's|||' "not well-formed" +new_in +for f in linux-x86_64-shaded jar-with-dependencies; do cp "$T/fx/other-native.jar" "$A/in/artifacts-spark-4.1/files/indextables_spark-1.2.3_spark_4.1.2-$f.jar"; done +resum "$A/in/artifacts-spark-4.1" +check "assemble: a jar with another native library is refused" fails_with "does not contain this run's native/libtantivy4java.so" "$L.7" assemble "$L.7" +new_in; echo "not a zip" > "$A/in/artifacts-spark-3.5/files/indextables_spark-1.2.3_spark_3.5.8-sources.jar"; resum "$A/in/artifacts-spark-3.5" +check "assemble: a jar that is not a zip file is refused" fails_with "is not a readable jar" "$L.8" assemble "$L.8" +new_in; sed -i.bak 's/^tantivy4java_jar_sha256=.*/tantivy4java_jar_sha256=0000/' "$A/in/artifacts-spark-3.5/build-info.txt"; rm "$A/in/artifacts-spark-3.5/build-info.txt.bak"; resum "$A/in/artifacts-spark-3.5" +check "assemble: a build that used another tantivy4java jar is refused" fails_with "was not built with this run's tantivy4java jar" "$L.9" assemble "$L.9" +new_in; sed -i.bak '/^cargo_lock_unchanged=/d' "$A/in/native/native-inputs.txt"; rm "$A/in/native/native-inputs.txt.bak"; resum "$A/in/native" +check "assemble: a native build without an unchanged Cargo.lock is refused" fails_with "does not record an unchanged Cargo.lock" "$L.10" assemble "$L.10" +new_in; sed -i.bak 's/^tantivy4java_commit=.*/tantivy4java_commit=main/' "$A/in/native/native-inputs.txt"; rm "$A/in/native/native-inputs.txt.bak"; resum "$A/in/native" +check "assemble: a tantivy4java commit that is not a commit id is refused" fails_with "does not state a tantivy4java commit" "$L.11" assemble "$L.11" +# shellcheck disable=SC2016 +new_in; echo 'rustc=rustc 1.0 `x` @everyone' >> "$A/in/native/native-inputs.txt"; resum "$A/in/native" +check "assemble: tool text from the build that is not plain text stays out of the notes" assemble "$L.12" +check "assemble: ... a placeholder is shown instead" has "$A/staging/release-notes.md" "(not shown: unexpected characters)" +check "assemble: ... the text itself is absent" hasnt "$A/staging/release-notes.md" " "$A/s/bundles/1.2.3_spark_3.5.8/io/indextables/indextables_spark/1.2.3_spark_3.5.8/extra.jar" +check "verify-staging: an added file is refused" fails_with "not exactly the files listed" "$L.4" verify "$L.4" +staging_copy; echo x >> "$A/s/bundles/1.2.3_spark_3.5.8/io/indextables/indextables_spark/1.2.3_spark_3.5.8/indextables_spark-1.2.3_spark_3.5.8.jar" +check "verify-staging: a changed file is refused" fails_with "digest mismatch" "$L.5" verify "$L.5" +staging_copy; rm "$A/s/release-notes.md" +check "verify-staging: a removed file is refused" fails_with "not exactly the files listed" "$L.6" verify "$L.6" +staging_copy +check "verify-staging: versions of another tag are refused" fails_with "is not 1.2.4_spark_3.5." "$L.7" verify "$L.7" "$MANIFEST" 1.2.4 +check "verify-staging: versions other than the ones the plan announced are refused" fails_with "are not the versions the plan announced" "$L.7b" verify "$L.7b" "$MANIFEST" 1.2.3 "1.2.3_spark_3.5.8 1.2.3_spark_4.0.3 1.2.3_spark_4.1.9" +# A self-consistent directory for other coordinates: valid manifest, wrong content. +staging_copy +mkdir -p "$A/s/bundles/1.2.3_spark_3.5.8/io/indextables/tantivy4java/9.9.9" && echo x > "$A/s/bundles/1.2.3_spark_3.5.8/io/indextables/tantivy4java/9.9.9/tantivy4java-9.9.9.jar" +(. "$here/lib.sh"; write_sums "$A/s" MANIFEST.sha256) +check "verify-staging: other coordinates are refused even with a matching manifest" fails_with "lists files other than the expected ones" "$L.8" verify "$L.8" "$(hash_of sha256 "$A/s/MANIFEST.sha256")" + +# --------------------------------------------------------------------------- +# signing-key.sh, sign-bundles.sh, check-bundles.sh +# --------------------------------------------------------------------------- +S="$T/sign"; mkdir -p "$S" +SIGNPATH="$PATH" +REAL_GPG=0 +if command -v gpg > /dev/null 2>&1; then + REAL_GPG=1 + echo "note signing tests use gpg ($(gpg --version | head -n 1)) with a throwaway key" + # gpg's agent socket needs a short path; mktemp directories can be too long. + KEYHOME="$(mktemp -d /tmp/rt.XXXXXX)/release-gnupg" + trap 'rm -rf "$T" "$(dirname "$KEYHOME")"' EXIT +else + if [ "${REQUIRE_GPG:-0}" = 1 ]; then + bad "signing: gpg is required but not available (the checks below ran against a stub)" + else + echo "note gpg not available: signing tests use a stub (set REQUIRE_GPG=1 to make this a failure)" + fi + KEYHOME="$S/release-gnupg" + mkdir -p "$S/bin" + cat > "$S/bin/gpg" << 'STUB' +#!/usr/bin/env bash +# Stub gpg: one key per GNUPGHOME, "signatures" that name the key and the +# file's digest. Refuses a wrong passphrase, like gpg does. +set -eu +h() { if command -v sha256sum > /dev/null 2>&1; then sha256sum "$1" | cut -d' ' -f1; else shasum -a 256 "$1" | cut -d' ' -f1; fi; } +mode=""; pfd=""; user=""; outf=""; files=() +while [ $# -gt 0 ]; do + case "$1" in + --batch | --quiet | --no-tty | --yes | --armor | --with-colons) shift ;; + --pinentry-mode | --status-fd) shift 2 ;; + --passphrase-fd) pfd="$2"; shift 2 ;; + --local-user) user="$2"; shift 2 ;; + --output) outf="$2"; shift 2 ;; + --import) mode=import; shift ;; + --quick-generate-key) mode=generate; shift 5 ;; + --export-secret-keys) mode=export; shift 2 ;; + --list-secret-keys) mode=list; shift ;; + --detach-sign) mode=sign; shift ;; + --verify) mode=verify; shift ;; + -*) echo "stub gpg: unexpected option $1" >&2; exit 97 ;; + *) files+=("$1"); shift ;; + esac +done +pass=""; if [ -n "$pfd" ]; then IFS= read -r pass <&"$pfd" || true; fi +case "$mode" in + import) # the "key" is "STUBKEY " + read -r tag fpr keypass || true + [ "$tag" = STUBKEY ] || { echo "stub gpg: no valid key data" >&2; exit 2; } + echo "$fpr" > "$GNUPGHOME/fpr"; echo "$keypass" > "$GNUPGHOME/pass" ;; + export) echo "STUBKEY $(cat "$GNUPGHOME/fpr") $(cat "$GNUPGHOME/pass")" ;; + generate) echo "0123456789ABCDEF0123456789ABCDEF01234567" > "$GNUPGHOME/fpr"; echo "$pass" > "$GNUPGHOME/pass" ;; + list) [ -f "$GNUPGHOME/fpr" ] || exit 0 + printf 'sec:u:255:22:KEYID:1:::u:::scESC:::+::ed25519:::0:\nfpr:::::::::%s:\nuid:u::::1::X::test::::::::::0:\n' "$(cat "$GNUPGHOME/fpr")" ;; + sign) + [ "$user" = "$(cat "$GNUPGHOME/fpr")" ] || { echo "stub gpg: no such key $user" >&2; exit 2; } + [ "$pass" = "$(cat "$GNUPGHOME/pass")" ] || { echo "stub gpg: bad passphrase" >&2; exit 2; } + printf 'STUBSIG %s %s\n' "$user" "$(h "${files[0]}")" > "$outf" ;; + verify) + read -r tag fpr sum < "${files[0]}" + [ "$tag" = STUBSIG ] && [ "$sum" = "$(h "${files[1]}")" ] || { echo "[GNUPG:] BADSIG"; exit 1; } + echo "[GNUPG:] VALIDSIG $fpr 2026-01-01 0 4 0 22 10 00 $fpr" ;; + *) echo "stub gpg: nothing to do" >&2; exit 97 ;; +esac +STUB + chmod +x "$S/bin/gpg" + SIGNPATH="$S/bin:$PATH" +fi +# With a real gpg the agent remembers a passphrase it has seen; forget it +# before a check that depends on the passphrase being asked for again. +forget() { if [ "$REAL_GPG" = 1 ]; then GNUPGHOME="$KEYHOME" gpgconf --kill gpg-agent > /dev/null 2>&1 || true; fi; } + +L="$S/log" +key() { run "$1" env PATH="$SIGNPATH" GNUPGHOME="$KEYHOME" GPG_PASSPHRASE=rehearsal bash "$here/signing-key.sh" "$2"; } +fresh() { rm -rf "$S/staging" "$S/bundles"; cp -R "$A/staging.good" "$S/staging"; } +sign() { run "$1" env PATH="$SIGNPATH" GNUPGHOME="$KEYHOME" SIGNING_KEY="${2:-$FPR}" GPG_PASSPHRASE="${3-rehearsal}" bash "$here/sign-bundles.sh" "$S/staging" "$S/bundles"; } +chk() { run "$1" env PATH="$SIGNPATH" GNUPGHOME="$KEYHOME" SIGNING_KEY="${2:-$FPR}" BASE_VERSION="${3:-1.2.3}" bash "$here/check-bundles.sh" "$S/bundles" "$S/staging"; } +NOKEY=00000000000000000000000000000000DEADBEEF + +check "signing-key: refuses a keyring directory with another name" fails_with "must be a directory named release-gnupg" "$L.0" \ + env PATH="$SIGNPATH" GNUPGHOME="$S/gnupg" bash "$here/signing-key.sh" remove +check "signing-key: generate creates a throwaway key" key "$L.1" generate +FPR="$(out "$L.1" fingerprint)" +check "signing-key: ... and reports its fingerprint" test -n "$FPR" +check "signing-key: ... in a private directory" test "$(ls -ld "$KEYHOME" | cut -c1-10)" = "drwx------" + +fresh; forget +check "sign: a wrong passphrase fails" fails_with "could not sign" "$L.2" sign "$L.2" "$FPR" wrong-passphrase +fresh +check "sign: an unknown key fails" fails_with "could not sign" "$L.3" sign "$L.3" "$NOKEY" +fresh +check "sign: signs and bundles the three versions" sign "$L.4" +check "sign: one bundle per version" is "$(ls "$S/bundles" | tr '\n' ' ')" "indextables_spark-1.2.3_spark_3.5.8-bundle.zip indextables_spark-1.2.3_spark_4.0.3-bundle.zip indextables_spark-1.2.3_spark_4.1.2-bundle.zip " +# The entry list of a bundle, written out in full: this is the set of files +# that is on Maven Central for 0.6.0-rc2 (36 per version). +want_entries="$(for s in .pom .jar -sources.jar -javadoc.jar -linux-x86_64-shaded.jar -jar-with-dependencies.jar; do + for e in "" .asc .md5 .sha1 .sha256 .sha512; do echo "io/indextables/indextables_spark/1.2.3_spark_4.0.3/indextables_spark-1.2.3_spark_4.0.3$s$e"; done; done | LC_ALL=C sort)" +check "sign: a bundle holds exactly the 36 files Maven Central has for a version" is \ + "$(unzip -Z1 "$S/bundles/indextables_spark-1.2.3_spark_4.0.3-bundle.zip" | grep -v '/$' | LC_ALL=C sort)" "$want_entries" +check "sign: checksum files are not signed" test -z "$(unzip -Z1 "$S/bundles/indextables_spark-1.2.3_spark_4.0.3-bundle.zip" | grep -E '\.(md5|sha1|sha256|sha512)\.asc$')" +check "check: accepts the signed bundles" chk "$L.5" +check "check: ... and says who signed" has "$L.5.summary" "$FPR" +check "sign: a second run on the same directory is refused" fails_with "already signed" "$L.6" sign "$L.6" +check "check: signatures from another key are refused" fails_with "expected $NOKEY" "$L.7" chk "$L.7" "$NOKEY" +check "check: versions of another tag are refused" fails_with "is not 1.2.4_spark_3.5." "$L.8" chk "$L.8" "$FPR" 1.2.4 +z="$S/bundles/indextables_spark-1.2.3_spark_3.5.8-bundle.zip" +zd=io/indextables/indextables_spark/1.2.3_spark_3.5.8 +cp "$z" "$z.keep" +mkdir -p "$S/x/io/indextables/tantivy4java/9.9.9" && echo x > "$S/x/io/indextables/tantivy4java/9.9.9/evil.jar" +(cd "$S/x" && zip -q "$z" io/indextables/tantivy4java/9.9.9/evil.jar) +check "check: a bundle with an extra entry is refused" fails_with "does not hold exactly the expected entries" "$L.9" chk "$L.9" +cp "$z.keep" "$z"; zip -q -d "$z" "$zd/indextables_spark-1.2.3_spark_3.5.8-javadoc.jar.asc" +check "check: a bundle with a missing signature is refused" fails_with "does not hold exactly the expected entries" "$L.10" chk "$L.10" +cp "$z.keep" "$z" +mkdir -p "$S/y/$zd" && echo 0000 > "$S/y/$zd/indextables_spark-1.2.3_spark_3.5.8.jar.sha1" +(cd "$S/y" && zip -q "$z" "$zd/indextables_spark-1.2.3_spark_3.5.8.jar.sha1") +check "check: a wrong checksum is refused" fails_with ".jar.sha1 does not match" "$L.11" chk "$L.11" +# A file replaced after signing, with its checksums recomputed to match: only +# the signature (and the comparison with the staged file) can catch it. +cp "$z.keep" "$z" +mkdir -p "$S/w/$zd" && cp "$T/fx/fat.jar" "$S/w/$zd/indextables_spark-1.2.3_spark_3.5.8.jar" +for algo in $CHECKSUM_ALGOS; do printf '%s' "$(hash_of "$algo" "$S/w/$zd/indextables_spark-1.2.3_spark_3.5.8.jar")" > "$S/w/$zd/indextables_spark-1.2.3_spark_3.5.8.jar.$algo"; done +(cd "$S/w" && zip -q -r "$z" io) +check "check: a file replaced after signing is refused, even with matching checksums" fails_with "the signature of indextables_spark-1.2.3_spark_3.5.8.jar does not verify" "$L.12" chk "$L.12" +cp "$z.keep" "$z"; rm "$z.keep" +staged_pom="$S/staging/bundles/1.2.3_spark_3.5.8/$zd/indextables_spark-1.2.3_spark_3.5.8.pom" +cp "$staged_pom" "$S/pom.keep"; echo "" >> "$staged_pom" +check "check: a bundle that differs from the staging directory is refused" fails_with "in the bundle differs from the staged file" "$L.12b" chk "$L.12b" +cp "$S/pom.keep" "$staged_pom" +check "check: the restored bundles pass again" chk "$L.13" +# Signing writes nothing but signatures into the staging directory: with +# those removed it verifies against the original manifest again. +find "$S/staging" -name '*.asc' -exec rm {} + +check "sign: nothing but signatures is written into the staging directory" run "$L.14" env MANIFEST_SHA256="$MANIFEST" BASE_VERSION=1.2.3 EXPECTED_VERSIONS="$EV" bash "$here/verify-staging.sh" "$S/staging" + +# The publish job's path: import a passphrase-protected private key from the +# environment, then sign with it. The key is the throwaway key, exported. +PATH="$SIGNPATH" GNUPGHOME="$KEYHOME" gpg --batch --no-tty --pinentry-mode loopback --passphrase-fd 3 --armor \ + --export-secret-keys "$FPR" 3<<< rehearsal > "$S/key.asc" 2> /dev/null +check "signing-key: remove deletes the keyring" key "$L.15" remove +check "signing-key: ... it is gone" test ! -e "$KEYHOME" +check "signing-key: ... and says so" has "$L.15" "Keyring removed." +# remove runs in an always() step, also after a job failed before GNUPGHOME +# was set. +check "signing-key: remove with GNUPGHOME unset succeeds" run "$L.15a" env -u GNUPGHOME PATH="$SIGNPATH" bash "$here/signing-key.sh" remove +check "signing-key: ... and says there was nothing to remove" has "$L.15a" "GNUPGHOME is not set: no keyring was set up in this job, nothing to remove." +check "signing-key: remove with GNUPGHOME empty succeeds" run "$L.15b" env PATH="$SIGNPATH" GNUPGHOME= bash "$here/signing-key.sh" remove +check "signing-key: ... with the same line" has "$L.15b" "GNUPGHOME is not set: no keyring was set up in this job, nothing to remove." +check "signing-key: remove when the keyring directory does not exist succeeds" run "$L.15c" env PATH="$SIGNPATH" GNUPGHOME="$KEYHOME" bash "$here/signing-key.sh" remove +check "signing-key: ... and says there was none" has "$L.15c" "No keyring at $KEYHOME: nothing to remove." +check "signing-key: ... and creates nothing" test ! -e "$KEYHOME" +check "signing-key: the name check still applies when the directory is missing" fails_with "must be a directory named release-gnupg" "$L.15d" \ + env PATH="$SIGNPATH" GNUPGHOME="$S/missing/gnupg" bash "$here/signing-key.sh" remove +check "signing-key: import with GNUPGHOME unset still fails" bash -c "! env -u GNUPGHOME PATH='$SIGNPATH' GPG_PRIVATE_KEY=x bash '$here/signing-key.sh' import" +check "signing-key: generate with GNUPGHOME empty still fails" bash -c "! env PATH='$SIGNPATH' GNUPGHOME= GPG_PASSPHRASE=x bash '$here/signing-key.sh' generate" +if [ "$(id -u)" != 0 ]; then + # A keyring whose contents cannot be deleted (read-only directories) must + # fail the step, not pass as a no-op. + stuck="$S/stuck/release-gnupg" + mkdir -p "$stuck/private-keys-v1.d" && echo key > "$stuck/private-keys-v1.d/k.key" + chmod 555 "$stuck/private-keys-v1.d" "$stuck" + check "signing-key: a keyring that cannot be removed fails the step" fails_with "could not remove the keyring at $stuck" "$L.15e" \ + env PATH="$SIGNPATH" GNUPGHOME="$stuck" bash "$here/signing-key.sh" remove + check "signing-key: ... (it is still there)" test -e "$stuck/private-keys-v1.d/k.key" + chmod 755 "$stuck" "$stuck/private-keys-v1.d"; rm -rf "$S/stuck" +else + echo "skip signing-key: removal failure (running as root, permissions do not apply)" +fi +check "signing-key: import without a key fails" bash -c "! PATH='$SIGNPATH' GNUPGHOME='$KEYHOME' GPG_PRIVATE_KEY= bash '$here/signing-key.sh' import" +check "signing-key: import of something that is not a key fails" fails_with "could not import the signing key" "$L.16" \ + env PATH="$SIGNPATH" GNUPGHOME="$KEYHOME" GPG_PRIVATE_KEY="not a key" bash "$here/signing-key.sh" import +check "signing-key: import reads the private key from the environment" run "$L.17" env PATH="$SIGNPATH" GNUPGHOME="$KEYHOME" GPG_PRIVATE_KEY="$(cat "$S/key.asc")" bash "$here/signing-key.sh" import +check "signing-key: ... and reports the same fingerprint" is "$(out "$L.17" fingerprint)" "$FPR" +check "signing-key: ... the key material is not echoed" test -z "$(grep -E 'BEGIN PGP|STUBKEY' "$L.17" "$L.17.out" "$L.17.summary" 2> /dev/null)" +fresh; forget +check "sign: the imported key does not sign without its passphrase" fails_with "could not sign" "$L.18" sign "$L.18" "$FPR" "" +fresh +check "sign: the imported key signs with its passphrase" sign "$L.19" +check "check: ... and the bundles pass" chk "$L.20" +key "$L.21" remove > /dev/null 2>&1 +rm -f "$S/key.asc" + +# --------------------------------------------------------------------------- +# central-upload.sh, against a stub curl +# --------------------------------------------------------------------------- +C="$T/central"; mkdir -p "$C/bin" +PORTAL_API=https://central.sonatype.com/api/v1/publisher +REPO1=https://repo1.maven.org/maven2 +cat > "$C/bin/curl" << 'STUB' +#!/usr/bin/env bash +# Stub curl for the Portal API and the public repository, under their real +# addresses (nothing here reaches the network). Records " +# [auth]". Like the Portal, it accepts a drop only for a deployment that is +# VALIDATED or FAILED. +set -u +API=https://central.sonatype.com/api/v1/publisher +REPO=https://repo1.maven.org/maven2 +method=GET; out=""; fmt=""; hdr=""; form=""; url="" +while [ $# -gt 0 ]; do + case "$1" in + -X) method="$2"; shift 2 ;; + -H) hdr="$2"; shift 2 ;; + -o) out="$2"; shift 2 ;; + -w) fmt="$2"; shift 2 ;; + -F) form="$2"; shift 2 ;; + --connect-timeout | --max-time | --retry) shift 2 ;; + -sS | -fsS) shift ;; + -*) echo "stub curl: unexpected option $1" >&2; exit 97 ;; + *) url="$1"; shift ;; + esac +done +auth="" +case "$hdr" in + @*) grep -q '^Authorization: Bearer ' "${hdr#@}" && auth=" auth:$(sed 's/^Authorization: Bearer //' "${hdr#@}")" ;; + "") ;; + *) echo "stub curl: header passed on the command line: $hdr" >> "$CFAKE/calls.log" ;; +esac +echo "$method $url$auth" >> "$CFAKE/calls.log" +respond() { # respond + if [ -n "$out" ]; then printf '%s' "$2" > "$out"; else printf '%s' "$2"; fi + [ -z "$fmt" ] || printf '%s' "$1" +} +state_of() { if [ -f "$CFAKE/status.$1" ]; then head -n 1 "$CFAKE/status.$1"; else echo VALIDATED; fi; } +case "$url" in + "$REPO"/*.pom) + v="$(basename "$(dirname "$url")")" + respond "$(cat "$CFAKE/repo.$v.code" 2> /dev/null || echo 404)" "" ;; + "$REPO"/*.sha1) + f="$CFAKE/repo.$(basename "$url")" + [ -f "$f" ] || exit 22 + cat "$f" ;; + "$API/upload?name="*"&publishingType=USER_MANAGED") + [ "$method" = POST ] || exit 97 + n="$(grep -c "POST $API/upload" "$CFAKE/calls.log")" + echo "$form" >> "$CFAKE/forms.log" + if [ -f "$CFAKE/upload.$n.exit" ]; then exit "$(cat "$CFAKE/upload.$n.exit")"; fi + respond "$(cat "$CFAKE/upload.$n.code" 2> /dev/null || echo 201)" "$(cat "$CFAKE/upload.$n.body" 2> /dev/null || echo "00000000-0000-4000-8000-00000000000$n")" ;; + "$API/status?id="*) + [ "$method" = POST ] || exit 97 + id="${url##*id=}"; f="$CFAKE/status.$id" + st="$(state_of "$id")" + # A status file is a sequence: each call moves on one line; the last stays. + if [ -f "$f" ] && [ "$(grep -c . "$f")" -gt 1 ]; then tail -n +2 "$f" > "$f.next" && mv "$f.next" "$f"; fi + case "$st" in + HTTP*) respond "${st#HTTP}" "upstream error" ;; + *) respond 200 "{\"deploymentId\":\"$id\",\"deploymentState\":\"$st\",\"errors\":{\"pkg\":[\"reason from the Portal\"]}}" ;; + esac ;; + "$API"/deployment/*) + id="${url##*/}" + [ "$method" = DELETE ] || { echo "stub curl: $method on a deployment is the Publish call" >&2; exit 97; } + if [ -f "$CFAKE/drop.$id.code" ]; then respond "$(cat "$CFAKE/drop.$id.code")" "drop failed" + else + case "$(state_of "$id")" in + VALIDATED | FAILED) echo "$id" >> "$CFAKE/dropped.log"; respond 204 "" ;; + *) respond 400 "Can only drop deployments that are in a VALIDATED or FAILED state" ;; + esac + fi ;; + *) echo "stub curl: unexpected request: $method $url" >&2; exit 97 ;; +esac +STUB +chmod +x "$C/bin/curl" +TOKEN="$(printf 'user:pass' | base64 | tr -d '\n')" +new_central() { CFAKE="$C/fake.$1"; rm -rf "$CFAKE" "$C/state"; mkdir -p "$CFAKE"; : > "$CFAKE/calls.log"; export CFAKE; } +central() { # central + local log="$1"; shift + run "$log" env PATH="$C/bin:$PATH" CENTRAL_USERNAME=user CENTRAL_PASSWORD=pass GITHUB_RUN_ID=777 GITHUB_RUN_ATTEMPT="${RUN_ATTEMPT:-2}" \ + RELEASE_SCRIPTS_TESTING=1 CENTRAL_POLL_SECONDS=0 CENTRAL_WAIT_SECONDS="${WAIT_SECONDS:-5}" CENTRAL_DROP_WAIT_SECONDS="${DROP_WAIT_SECONDS:-5}" \ + bash "$here/central-upload.sh" "$@" +} +up() { central "$1" upload "$S/bundles" "$S/staging" "$C/state"; } +drop() { central "$1" drop "$C/state"; } +calls() { grep -c -- "$1" "$CFAKE/calls.log" || true; } +dropped() { tr '\n' ' ' < "$CFAKE/dropped.log" 2> /dev/null || true; } +ID1=00000000-0000-4000-8000-000000000001; ID2=00000000-0000-4000-8000-000000000002; ID3=00000000-0000-4000-8000-000000000003 +N1=indextables_spark-1.2.3_spark_3.5.8-run777-2; N2=indextables_spark-1.2.3_spark_4.0.3-run777-2; N3=indextables_spark-1.2.3_spark_4.1.2-run777-2 +L="$C/log" + +new_central ok; printf 'PENDING\nVALIDATING\nVALIDATED\n' > "$CFAKE/status.$ID1" +check "central: three uploads, each validated" up "$L.1" +check "central: ... three upload calls" is "$(calls "POST $PORTAL_API/upload")" 3 +check "central: ... as USER_MANAGED, named after the version, the run and the attempt" has "$CFAKE/calls.log" "POST $PORTAL_API/upload?name=$N1&publishingType=USER_MANAGED" +check "central: ... the bundle is the form field" has "$CFAKE/forms.log" "bundle=@$S/bundles/indextables_spark-1.2.3_spark_4.1.2-bundle.zip;type=application/octet-stream" +check "central: ... one version at a time: validated before the next upload" is \ + "$(grep -E 'upload\?name|status\?id' "$CFAKE/calls.log" | sed -e 's/^POST [^ ]*\/publisher\///' -e 's/[?].*//' | tr '\n' ' ')" "upload status status status upload status upload status " +check "central: ... nothing is dropped" is "$(calls '^DELETE')" 0 +check "central: ... the Publish call is never made" is "$(calls "^POST $PORTAL_API/deployment/")" 0 +check "central: ... the token goes to the Portal as a Bearer header" is "$(grep -c "central.sonatype.com.* auth:$TOKEN\$" "$CFAKE/calls.log")" "$(grep -c 'central.sonatype.com' "$CFAKE/calls.log")" +check "central: ... and never to the public repository" is "$(grep 'repo1.maven.org' "$CFAKE/calls.log" | grep -c 'auth:')" 0 +check "central: ... and never on a command line" hasnt "$CFAKE/calls.log" "header passed on the command line" +check "central: ... every request went to the Portal API or repo1, whatever CENTRAL_API_URL says" test -z "$(grep -v -E "^[A-Z]+ ($PORTAL_API|$REPO1)/" "$CFAKE/calls.log")" +check "central: ... the encoded token is masked before use" has "$L.1" "::add-mask::$TOKEN" +check "central: ... the summary says it is not public yet" has "$L.1.summary" "Not public yet" +check "central: ... each deployment id is in the summary as soon as it is known, with its name" has "$L.1.summary" "- Uploaded \`$N2\`: deployment \`$ID2\`" +check "central: ... the final table lists id, name and state" has "$L.1.summary" "| \`1.2.3_spark_4.1.2\` | \`$ID3\` | \`$N3\` | VALIDATED |" +check "central: ... and says to publish only those" has "$L.1.summary" "Publish **only** the deployments whose id is in this table" +check "central: ... deployments are recorded with their names" is "$(tr '\n' ' ' < "$C/state/deployments.txt")" "1.2.3_spark_3.5.8 $ID1 $N1 1.2.3_spark_4.0.3 $ID2 $N2 1.2.3_spark_4.1.2 $ID3 $N3 " +check "central: a second upload from the same job is refused" fails_with "refusing to upload twice" "$L.1b" up "$L.1b" +check "central: drop (a later step failed) removes this job's deployments" drop "$L.1c" +check "central: ... all three" is "$(dropped)" "$ID1 $ID2 $ID3 " +check "central: ... and a second drop has nothing left to do" drop "$L.1d" +check "central: ... (no further drop)" is "$(calls '^DELETE')" 3 +rm -rf "$C/state" +check "central: drop before the upload step ran has nothing to do" drop "$L.1e" +check "central: ... (no request at all)" is "$(calls '^DELETE')" 3 + +# A deployment left in the Portal by an earlier run, for the same version. +STALE=99999999-0000-4000-8000-999999999999 +new_central stale; echo VALIDATED > "$CFAKE/status.$STALE" +check "central: an earlier run's deployment for the same version does not get in the way" up "$L.s1" +check "central: ... it is never touched" is "$(calls "$STALE")" 0 +check "central: ... this run's deployments carry this run's id and attempt" is "$(grep -c -- '-run777-2&publishingType' "$CFAKE/calls.log")" 3 +check "central: ... the summary tells them apart" has "$L.s1.summary" "with any other id or name comes from a different run: drop it, do not publish it" +check "central: ... dropping after a failure leaves the earlier run's deployment alone" drop "$L.s2" +check "central: ... (dropped: this job's three)" is "$(dropped)" "$ID1 $ID2 $ID3 " +new_central stale2 +RUN_ATTEMPT=3 +check "central: a re-run of the job uploads under another name" up "$L.s3" +unset RUN_ATTEMPT +check "central: ... (attempt 3)" has "$CFAKE/calls.log" "upload?name=indextables_spark-1.2.3_spark_3.5.8-run777-3&" + +new_central rejected; echo FAILED > "$CFAKE/status.$ID2" +check "central: a rejected bundle fails the step" fails_with "Maven Central rejected the bundle for 1.2.3_spark_4.0.3" "$L.2" up "$L.2" +check "central: ... the Portal's reasons are shown" has "$L.2" "reason from the Portal" +check "central: ... the third version is not uploaded" is "$(calls "POST $PORTAL_API/upload")" 2 +check "central: ... both deployments of this job are dropped" is "$(dropped)" "$ID1 $ID2 " +check "central: ... nothing is left recorded as waiting" test ! -s "$C/state/deployments.txt" +check "central: ... the last step then has nothing to do" drop "$L.2b" +check "central: ... and raises no alarm" hasnt "$L.2b.summary" "Action needed" + +new_central refused; echo 401 > "$CFAKE/upload.1.code"; echo "invalid token" > "$CFAKE/upload.1.body" +check "central: a refused upload fails the step" fails_with "was refused: HTTP 401 invalid token" "$L.3" up "$L.3" +check "central: ... nothing else is attempted" is "$(calls "POST $PORTAL_API/upload")$(calls 'status')$(calls '^DELETE')" 100 +check "central: ... nothing is left to drop" drop "$L.3b" + +new_central broken; echo 28 > "$CFAKE/upload.2.exit" +check "central: an upload that does not complete fails the step" fails_with "may or may not have reached the Portal" "$L.4" up "$L.4" +check "central: ... the message names the deployment to look for" has "$L.4" "look for a deployment named $N2" +check "central: ... the earlier deployment is dropped" is "$(dropped)" "$ID1 " +check "central: ... the last step fails, because a deployment may exist that has no id" fails_with "was cut off before the Portal answered" "$L.4b" drop "$L.4b" +check "central: ... and puts its name in the summary" has "$L.4b.summary" "| \`1.2.3_spark_4.0.3\` | unknown | \`$N2\` |" + +new_central garbled; echo "ok" > "$CFAKE/upload.1.body" +check "central: an upload answer that is not a deployment id fails the step" fails_with "unexpected response" "$L.5" up "$L.5" +check "central: ... the last step reports the deployment that may exist" fails_with "$N1" "$L.5b" drop "$L.5b" + +# Validation takes longer than the step waits. The Portal refuses to drop a +# deployment that is still being validated. +new_central slow1; printf 'VALIDATING\nVALIDATING\nVALIDATING\nVALIDATED\n' > "$CFAKE/status.$ID1" +WAIT_SECONDS=0 +check "central: a validation that does not finish in time fails the step" fails_with "was not validated within 0 seconds" "$L.6" up "$L.6" +check "central: ... the deployment is polled until it can be dropped, then dropped" is "$(dropped)" "$ID1 " +check "central: ... no drop was attempted while it was still validating" is "$(calls '^DELETE')" 1 +check "central: ... nothing is left recorded" test ! -s "$C/state/deployments.txt" +check "central: ... the last step has nothing to do" drop "$L.6b" + +new_central slow2; echo VALIDATING > "$CFAKE/status.$ID1" +DROP_WAIT_SECONDS=0 +check "central: a validation that outlasts the drop wait as well fails the step" fails_with "was not validated within 0 seconds" "$L.7" up "$L.7" +check "central: ... the drop is refused by the Portal" has "$L.7" "Could not drop deployment $ID1 ($N1), state VALIDATING: HTTP 400" +check "central: ... the deployment stays recorded, for the last step" is "$(cat "$C/state/deployments.txt")" "1.2.3_spark_3.5.8 $ID1 $N1" +check "central: ... and the log says the last step tries again" has "$L.7" "the last step of this job tries again" +check "central: the last step, with validation still running, fails" fails_with "Could not drop deployment $ID1" "$L.7b" drop "$L.7b" +check "central: ... the summary says what to drop by hand, by id and name" has "$L.7b.summary" "| \`1.2.3_spark_3.5.8\` | \`$ID1\` | \`$N1\` | VALIDATING; the drop was refused (HTTP 400). **Drop it by hand. Do not publish it.** |" +check "central: ... under a heading that cannot be missed" has "$L.7b.summary" "### Action needed: deployments left in the Central Portal" +check "central: ... and it stays recorded" is "$(cat "$C/state/deployments.txt")" "1.2.3_spark_3.5.8 $ID1 $N1" +echo VALIDATED > "$CFAKE/status.$ID1" +check "central: once validation has finished, the same step drops it" drop "$L.7c" +check "central: ... (dropped)" is "$(dropped)" "$ID1 " +unset DROP_WAIT_SECONDS WAIT_SECONDS + +# The job is cancelled while a deployment is being validated: the upload step +# is killed where it stands, and the last step has to clean up. +new_central cancelled; echo VALIDATING > "$CFAKE/status.$ID1" +env PATH="$C/bin:$PATH" CENTRAL_USERNAME=user CENTRAL_PASSWORD=pass GITHUB_RUN_ID=777 GITHUB_RUN_ATTEMPT=2 RELEASE_SCRIPTS_TESTING=1 \ + CENTRAL_POLL_SECONDS=1 CENTRAL_WAIT_SECONDS=600 bash "$here/central-upload.sh" upload "$S/bundles" "$S/staging" "$C/state" > "$L.8" 2>&1 & +victim=$! +i=0; while [ ! -s "$C/state/deployments.txt" ] && [ "$i" -lt 100 ]; do sleep 0.1; i=$((i + 1)); done +kill -KILL "$victim" 2> /dev/null; wait "$victim" 2> /dev/null +check "central: an upload step killed during validation leaves its deployment recorded" is "$(cat "$C/state/deployments.txt")" "1.2.3_spark_3.5.8 $ID1 $N1" +check "central: ... and nothing dropped yet" is "$(calls '^DELETE')" 0 +printf 'VALIDATING\nVALIDATING\nVALIDATED\n' > "$CFAKE/status.$ID1" +check "central: the last step waits for the validation to finish and drops it" drop "$L.8b" +check "central: ... (dropped)" is "$(dropped)" "$ID1 " +check "central: ... without an alarm" hasnt "$L.8b.summary" "Action needed" + +# The Portal refuses a drop for another reason. +new_central droprefused; echo FAILED > "$CFAKE/status.$ID2"; echo 500 > "$CFAKE/drop.$ID1.code" +check "central: a refused drop does not hide the original failure" fails_with "Maven Central rejected the bundle for 1.2.3_spark_4.0.3" "$L.9" up "$L.9" +check "central: ... the other deployment is still dropped" is "$(dropped)" "$ID2 " +check "central: ... the refused one stays recorded" is "$(cat "$C/state/deployments.txt")" "1.2.3_spark_3.5.8 $ID1 $N1" +check "central: ... the last step tries again and, refused again, fails" fails_with "Could not drop deployment $ID1" "$L.9b" drop "$L.9b" +check "central: ... listing only that one" is "$(grep -c '^| `1.2.3' "$L.9b.summary")" 1 +rm "$CFAKE/drop.$ID1.code" +check "central: ... and succeeds once the Portal accepts the drop" drop "$L.9c" + +# Someone publishes a deployment by hand while the job is failing. +new_central raced2; printf 'VALIDATED\nPUBLISHED\n' > "$CFAKE/status.$ID1"; echo FAILED > "$CFAKE/status.$ID2" +check "central: a deployment published by hand before the job failed cannot be dropped" fails_with "is PUBLISHED and can no longer be dropped" "$L.10" up "$L.10" +check "central: ... no drop is attempted for it" is "$(calls "^DELETE $PORTAL_API/deployment/$ID1")" 0 +check "central: ... the last step says so in the summary" fails_with "can no longer be dropped" "$L.10b" drop "$L.10b" +check "central: ... (the row)" has "$L.10b.summary" "| \`1.2.3_spark_3.5.8\` | \`$ID1\` | \`$N1\` | PUBLISHED: it can no longer be dropped." + +new_central flaky; printf 'HTTP502\nHTTP502\nVALIDATED\n' > "$CFAKE/status.$ID1" +check "central: a few failed status calls are tolerated" up "$L.11" +new_central down; echo HTTP503 > "$CFAKE/status.$ID1" +check "central: status calls that keep failing fail the step" fails_with "could not be read six times in a row" "$L.12" up "$L.12" + +new_central raced; echo PUBLISHED > "$CFAKE/status.$ID1" +check "central: a deployment published by hand during the run is not an error" up "$L.13" +check "central: ... but is called out" has "$L.13" "someone pressed Publish" + +# Re-run after a partial publish: 3.5.8 is already public with these files. +vd="$S/staging/bundles/1.2.3_spark_3.5.8/io/indextables/indextables_spark/1.2.3_spark_3.5.8" +same_on_central() { echo 200 > "$CFAKE/repo.1.2.3_spark_3.5.8.code"; for n in $(artifact_names 1.2.3_spark_3.5.8); do hash_of sha1 "$vd/$n" > "$CFAKE/repo.$n.sha1"; done; } +new_central partial; same_on_central +check "central: a version already public with identical files is skipped" up "$L.14" +check "central: ... only the other two are uploaded" is "$(grep 'upload?name' "$CFAKE/calls.log" | sed -e 's/.*name=indextables_spark-//' -e 's/-run.*//' | tr '\n' ' ')" "1.2.3_spark_4.0.3 1.2.3_spark_4.1.2 " +check "central: ... and the summary says so" has "$L.14.summary" "ALREADY_PUBLISHED" +new_central differs; same_on_central; echo 0000000000000000000000000000000000000000 > "$CFAKE/repo.indextables_spark-1.2.3_spark_3.5.8-javadoc.jar.sha1" +check "central: a version already public with other files stops the release" fails_with "cannot be replaced" "$L.15" up "$L.15" +check "central: ... before anything is uploaded" is "$(calls 'central.sonatype.com')" 0 +new_central unknown; echo 503 > "$CFAKE/repo.1.2.3_spark_4.1.2.code" +check "central: an unreadable public repository stops the release" fails_with "could not tell whether 1.2.3_spark_4.1.2 is already on Maven Central" "$L.16" up "$L.16" +check "central: ... before anything is uploaded" is "$(calls 'central.sonatype.com')" 0 +new_central nocreds +check "central: missing credentials fail before any upload" bash -c "! PATH='$C/bin:$PATH' CENTRAL_USERNAME= CENTRAL_PASSWORD= bash '$here/central-upload.sh' upload '$S/bundles' '$S/staging' '$C/state'" +check "central: ... (no request)" is "$(grep -c . "$CFAKE/calls.log")" 0 +new_central knob +check "central: a timing setting without the test flag stops the script" fails_with "CENTRAL_POLL_SECONDS is a test-only setting" "$L.17" \ + env PATH="$C/bin:$PATH" CENTRAL_USERNAME=user CENTRAL_PASSWORD=pass CENTRAL_POLL_SECONDS=0 bash "$here/central-upload.sh" upload "$S/bundles" "$S/staging" "$C/state" +check "central: ... before any request" is "$(grep -c . "$CFAKE/calls.log")" 0 +check "central: ... the same for the drop wait" fails_with "CENTRAL_DROP_WAIT_SECONDS is a test-only setting" "$L.18" \ + env PATH="$C/bin:$PATH" CENTRAL_USERNAME=user CENTRAL_PASSWORD=pass CENTRAL_DROP_WAIT_SECONDS=0 bash "$here/central-upload.sh" drop "$C/state" +check "central: the endpoints are constants" test -n "$(grep -x 'CENTRAL_API=https://central.sonatype.com/api/v1/publisher' "$here/lib.sh")" -a -n "$(grep -x 'CENTRAL_REPO=https://repo1.maven.org/maven2' "$here/lib.sh")" +check "central: no variable redirects them" test -z "$(grep -l -E 'CENTRAL_API_URL|CENTRAL_REPO_URL' "$here"/*.sh | grep -v test.sh)" +# shellcheck disable=SC2086 +check "central: the only settings a test may change are the three timings" is \ + "$(grep -h -o 'test_setting [A-Z_][A-Z_]*' $SCRIPTS_UNDER_TEST | sort -u | tr '\n' ' ')" \ + "test_setting CENTRAL_DROP_WAIT_SECONDS test_setting CENTRAL_POLL_SECONDS test_setting CENTRAL_WAIT_SECONDS " +check "central: AUTOMATIC publishing appears nowhere" test -z "$(grep -l 'AUTOMATIC' "$here"/*.sh | grep -v test.sh)" + +# --------------------------------------------------------------------------- +# github-release.sh, against a stub gh +# --------------------------------------------------------------------------- +G="$T/gh"; mkdir -p "$G/bin" +cat > "$G/bin/gh" << 'STUB' +#!/usr/bin/env bash +# Stub gh: tag lookups and a release kept in $GFAKE: body.md (its notes; the +# release exists when this file does), assets.json (name and digest of each +# attached file) and files/ (their content, for downloads). +set -u +echo "gh $*" >> "$GFAKE/calls.log" +h() { if command -v sha256sum > /dev/null 2>&1; then sha256sum "$1" | cut -d' ' -f1; else shasum -a 256 "$1" | cut -d' ' -f1; fi; } +attach() { # attach ...: record the files as assets, replacing same names + local f name + [ -f "$GFAKE/assets.json" ] || echo '[]' > "$GFAKE/assets.json" + mkdir -p "$GFAKE/files" + for f in "$@"; do + case "$f" in -*) continue ;; esac + [ -f "$f" ] || continue + name="$(basename "$f")" + cp "$f" "$GFAKE/files/$name" + jq --arg n "$name" --arg d "sha256:$(h "$f")" '[.[] | select(.name != $n)] + [{name: $n, digest: $d}]' "$GFAKE/assets.json" > "$GFAKE/assets.tmp" \ + && mv "$GFAKE/assets.tmp" "$GFAKE/assets.json" + done +} +case "$1 ${2:-}" in + "api repos/o/r/git/ref/tags/"*) [ -f "$GFAKE/ref.json" ] || exit 1; cat "$GFAKE/ref.json" ;; + "api repos/o/r/git/tags/"*) cat "$GFAKE/tagobj.sha" ;; + "release view") + if [ ! -f "$GFAKE/body.md" ]; then cat "$GFAKE/view.err" >&2; exit 1; fi + case "$*" in + *"--json body,assets"*) + [ -f "$GFAKE/assets.json" ] || echo '[]' > "$GFAKE/assets.json" + jq -n --rawfile body "$GFAKE/body.md" --slurpfile assets "$GFAKE/assets.json" '{body: $body, assets: $assets[0]}' ;; + *"--json body"*) cat "$GFAKE/body.md" ;; + *"--json url"*) echo "https://example.invalid/releases/$3" ;; + esac ;; + "release upload") shift 3; attach "$@" ;; + "release download") # release download TAG --pattern NAME --dir DIR + cp "$GFAKE/files/$5" "$7/$5" ;; + "release edit") shift 3; [ "$1" = --notes-file ] && cp "$2" "$GFAKE/body.md" ;; + "release create") printf "## What's Changed\n* a change\n" > "$GFAKE/body.md"; shift 3; attach "$@" ;; + *) echo "stub gh: unexpected call: $*" >&2; exit 97 ;; +esac +STUB +chmod +x "$G/bin/gh" +new_gh() { GFAKE="$G/fake.$1"; rm -rf "$GFAKE"; mkdir -p "$GFAKE"; : > "$GFAKE/calls.log"; export GFAKE + printf '{"object":{"type":"commit","sha":"%s"}}\n' "$SHA_D" > "$GFAKE/ref.json"; echo "release not found" > "$GFAKE/view.err"; } +release() { # release [VAR=value...] + local log="$1"; shift + run "$log" env PATH="$G/bin:$PATH" GH_TOKEN=t GH_REPO=o/r TAG=v1.2.3 COMMIT="$SHA_D" PRERELEASE=false "$@" bash "$here/github-release.sh" "$A/staging.good" +} +uploads() { grep -c 'gh release upload' "$GFAKE/calls.log" || true; } +J35=indextables_spark-1.2.3_spark_3.5.8-linux-x86_64-shaded.jar +staged35="$A/staging.good/bundles/1.2.3_spark_3.5.8/io/indextables/indextables_spark/1.2.3_spark_3.5.8/$J35" +# An earlier build of the same jar: same name, other bytes. +mkdir -p "$G/old" && cp "$T/fx/other-native.jar" "$G/old/$J35" +old_release() { # a published release that already carries an earlier build of the 3.5 jar + new_gh "$1"; printf 'Published notes.\n' > "$GFAKE/body.md" + PATH="$G/bin:$PATH" gh release upload v1.2.3 "$G/old/$J35"; : > "$GFAKE/calls.log" +} +L="$G/log" +new_gh create +check "release: creates the release when there is none" release "$L.1" +check "release: ... for an existing tag, with generated notes" has "$GFAKE/calls.log" "gh release create v1.2.3 --verify-tag --generate-notes $A/staging.good/bundles/" +check "release: ... the generated notes are kept" has "$GFAKE/body.md" "What's Changed" +check "release: ... with the three shaded jars" is "$(grep 'release create' "$GFAKE/calls.log" | tr ' ' '\n' | grep -c -- '-linux-x86_64-shaded.jar$')" 3 +check "release: ... and no other file" is "$(grep 'release create' "$GFAKE/calls.log" | tr ' ' '\n' | grep -c '/bundles/')" 3 +check "release: ... not marked as a pre-release" test -z "$(grep -- '--prerelease' "$GFAKE/calls.log")" +check "release: ... notes carry the build inputs" has "$GFAKE/body.md" "$T4J_COMMIT" +new_gh pre +check "release: a pre-release tag creates a pre-release" release "$L.2" PRERELEASE=true +check "release: ... flag passed" has "$GFAKE/calls.log" "--prerelease" +new_gh existing; printf 'Hand-written notes.\r\n\r\n- item\r\n\r\n' > "$GFAKE/body.md" +check "release: an existing release without jars gets them" release "$L.3" +check "release: ... in one upload that replaces nothing" is "$(grep 'gh release upload' "$GFAKE/calls.log" | grep -c -v -- '--clobber')" 1 +check "release: ... all three" is "$(jq -r 'length' "$GFAKE/assets.json")" 3 +check "release: ... it is not created again" test -z "$(grep 'release create' "$GFAKE/calls.log")" +check "release: ... its notes are kept" has "$GFAKE/body.md" "Hand-written notes." +check "release: ... and gain the build inputs" has "$GFAKE/body.md" "$T4J_COMMIT" +: > "$GFAKE/calls.log" +check "release: a re-run with the same staged jars" release "$L.4" +check "release: ... uploads nothing, the jars are identical" is "$(uploads)" 0 +check "release: ... and says so" has "$L.4" "$J35 is already attached with identical content." +check "release: ... replaces the build inputs instead of adding a second block" is "$(grep -c 'release-build-inputs:start' "$GFAKE/body.md")" 1 +check "release: ... and keeps the notes" is "$(grep -c 'Hand-written notes.' "$GFAKE/body.md")" 1 + +# The release already has one of the jars from an earlier build (the +# skip-central case: nothing on the Maven Central side stops the run). +old_release differs +check "release: a jar already attached with other content stops the step" fails_with "already has 1 of these jars with different content" "$L.10" release "$L.10" +check "release: ... it names the jar" has "$L.10" "$J35 is already attached with different content" +check "release: ... and the input that would allow it" has "$L.10" "replace-release-assets" +check "release: ... nothing was uploaded, not even the jars that were missing" is "$(uploads)" 0 +check "release: ... the attached jar is still the earlier one" is "$(jq -r --arg n "$J35" '.[] | select(.name == $n) | .digest' "$GFAKE/assets.json")" "sha256:$(hash_of sha256 "$G/old/$J35")" +check "release: ... the notes were not touched" is "$(cat "$GFAKE/body.md")" "Published notes." +for v in "" false TRUE 1 yes; do + old_release "differs-$v" + check "release: replace-release-assets '$v' does not allow it either" fails_with "with different content" "$L.11" release "$L.11" REPLACE_ASSETS="$v" +done +old_release replace +check "release: with replace-release-assets the jar is replaced" release "$L.12" REPLACE_ASSETS=true +check "release: ... with a warning" has "$L.12" "::warning::Replacing 1 jar(s) on release v1.2.3" +check "release: ... only that jar is uploaded with --clobber" is "$(grep -- '--clobber' "$GFAKE/calls.log" | tr ' ' '\n' | grep -c 'shaded.jar$')" 1 +check "release: ... (it is the 3.5 jar)" test -n "$(grep -- '--clobber' "$GFAKE/calls.log" | grep -F "$J35")" +check "release: ... the other two are uploaded without it" is "$(grep 'gh release upload' "$GFAKE/calls.log" | grep -v -- '--clobber' | tr ' ' '\n' | grep -c 'shaded.jar$')" 2 +check "release: ... the attached jar is now the staged one" is "$(jq -r --arg n "$J35" '.[] | select(.name == $n) | .digest' "$GFAKE/assets.json")" "sha256:$(hash_of sha256 "$staged35")" +# An asset without a recorded digest is downloaded and hashed. +old_release nodigest; jq 'map(del(.digest))' "$GFAKE/assets.json" > "$GFAKE/a.tmp" && mv "$GFAKE/a.tmp" "$GFAKE/assets.json" +check "release: a differing jar without a recorded digest is still caught" fails_with "with different content" "$L.13" release "$L.13" +check "release: ... by downloading it" has "$GFAKE/calls.log" "gh release download v1.2.3 --pattern $J35" +new_gh nodigest2; printf 'Notes.\n' > "$GFAKE/body.md" +PATH="$G/bin:$PATH" gh release upload v1.2.3 "$staged35"; jq 'map(del(.digest))' "$GFAKE/assets.json" > "$GFAKE/a.tmp" && mv "$GFAKE/a.tmp" "$GFAKE/assets.json"; : > "$GFAKE/calls.log" +check "release: an identical jar without a recorded digest is recognised" release "$L.14" +check "release: ... and not uploaded again" test -z "$(grep 'gh release upload' "$GFAKE/calls.log" | grep -F "$J35")" + +new_gh moved; printf '{"object":{"type":"commit","sha":"%s"}}\n' "$SHA_A" > "$GFAKE/ref.json" +check "release: a tag that moved since the build is not released" fails_with "now points to $SHA_A" "$L.5" release "$L.5" +check "release: ... no release call is made" test -z "$(grep 'gh release' "$GFAKE/calls.log")" +new_gh annotated; printf '{"object":{"type":"tag","sha":"%s"}}\n' "$SHA_B" > "$GFAKE/ref.json"; echo "$SHA_D" > "$GFAKE/tagobj.sha" +check "release: an annotated tag is followed to its commit" release "$L.6" +new_gh gone; rm "$GFAKE/ref.json" +check "release: a tag that no longer exists is not released" fails_with "does not exist" "$L.7" release "$L.7" +new_gh error; echo "HTTP 502: Bad Gateway" > "$GFAKE/view.err" +check "release: a failed lookup does not fall through to creating a release" fails_with "could not look up release" "$L.8" release "$L.8" +check "release: ... no create call" test -z "$(grep 'release create' "$GFAKE/calls.log")" + +# --------------------------------------------------------------------------- +# The whole chain with the relative paths release.yml passes, from one +# working directory, the way the jobs run it. +# --------------------------------------------------------------------------- +E="$T/chain"; mkdir -p "$E/release/.github/scripts" "$E/src" +cp -R "$here" "$E/release/.github/scripts/release" +cp "$N/pom.xml" "$E/src/pom.xml" +cp "$N/pins.txt" "$E/release/.github/scripts/release/native-pins.txt" +ES=release/.github/scripts/release +chain() { # chain : run in $E with the stubs on PATH + local log="$1"; shift + (cd "$E" && GITHUB_OUTPUT="$log.out" GITHUB_STEP_SUMMARY="$log.summary" PATH="$C/bin:$G/bin:$T/bin:${SIGNPATH}" "$@") > "$log" 2>&1 +} +L="$E/log" +check "chain: native resolve" chain "$L.1" env HOME="$E/home-native" GIT_CONFIG_COUNT=1 "$(redirect "$N/t4j")" GIT_CONFIG_VALUE_0=https://github.com/indextables/tantivy4java.git POM=src/pom.xml PINS="$ES/native-pins.txt" WORK=work/native OUT=out/native bash "$ES/build-native.sh" resolve +mkdir -p "$E/work/native/protoc/bin" && cp "$N/work/protoc/bin/protoc" "$E/work/native/protoc/bin/protoc" 2> /dev/null \ + || { printf '#!/bin/sh\necho "libprotoc 25.5"\n' > "$E/work/native/protoc/bin/protoc"; chmod +x "$E/work/native/protoc/bin/protoc"; } +check "chain: native build" chain "$L.2" env HOME="$E/home-native" POM=src/pom.xml PINS="$ES/native-pins.txt" WORK=work/native OUT=out/native bash "$ES/build-native.sh" build +mkdir -p "$E/in" && cp -R "$E/out/native" "$E/in/native" +for profile in $PROFILES; do + rm -rf "$E/src/target" "$E/src/.stamped" "$E/src/dependency-reduced-pom.xml" + check "chain: build $profile" chain "$L.3" env SRC=src PROFILE="$profile" BASE_VERSION=1.2.3 EXPECTED_VERSIONS="$EV" NATIVE_DIR=in/native OUT=out/artifacts HOME="$E/home-$profile" bash "$ES/build-artifacts.sh" + cp -R "$E/out/artifacts" "$E/in/artifacts-$profile" +done +check "chain: assemble" chain "$L.4" env IN=in OUT=staging TAG=v1.2.3 COMMIT="$SHA_D" BASE_VERSION=1.2.3 EXPECTED_VERSIONS="$EV" bash "$ES/assemble.sh" +CM="$(out "$L.4" manifest_sha256)" +check "chain: verify" chain "$L.5" env MANIFEST_SHA256="$CM" BASE_VERSION=1.2.3 EXPECTED_VERSIONS="$EV" bash "$ES/verify-staging.sh" staging +check "chain: generate a key" chain "$L.6" env GNUPGHOME="$KEYHOME" GPG_PASSPHRASE=rehearsal bash "$ES/signing-key.sh" generate +CK="$(out "$L.6" fingerprint)" +check "chain: sign" chain "$L.7" env GNUPGHOME="$KEYHOME" SIGNING_KEY="$CK" GPG_PASSPHRASE=rehearsal bash "$ES/sign-bundles.sh" staging bundles +check "chain: check" chain "$L.8" env GNUPGHOME="$KEYHOME" SIGNING_KEY="$CK" BASE_VERSION=1.2.3 bash "$ES/check-bundles.sh" bundles staging +check "chain: remove the key" chain "$L.9" env GNUPGHOME="$KEYHOME" bash "$ES/signing-key.sh" remove +new_central chain; rm -rf "$E/state" +check "chain: upload" chain "$L.10" env CENTRAL_USERNAME=user CENTRAL_PASSWORD=pass GITHUB_RUN_ID=4242 GITHUB_RUN_ATTEMPT=1 RELEASE_SCRIPTS_TESTING=1 CENTRAL_POLL_SECONDS=0 \ + bash "$ES/central-upload.sh" upload bundles staging "$E/state" +check "chain: ... three bundles went up" is "$(calls "POST $PORTAL_API/upload")" 3 +check "chain: ... as form uploads of the relative bundle paths" has "$CFAKE/forms.log" "bundle=@bundles/indextables_spark-1.2.3_spark_4.0.3-bundle.zip;type=application/octet-stream" +check "chain: ... named after the runner's run id and attempt" has "$CFAKE/calls.log" "upload?name=indextables_spark-1.2.3_spark_4.0.3-run4242-1&" +new_gh chain +check "chain: release" chain "$L.11" env GH_TOKEN=t GH_REPO=o/r TAG=v1.2.3 COMMIT="$SHA_D" PRERELEASE=false bash "$ES/github-release.sh" staging +check "chain: ... with the staged shaded jars" is "$(grep 'release create' "$GFAKE/calls.log" | tr ' ' '\n' | grep -c '^staging/bundles/.*-linux-x86_64-shaded.jar$')" 3 + +# --------------------------------------------------------------------------- +# release.yml +# --------------------------------------------------------------------------- +if command -v ruby > /dev/null 2>&1 && ruby -ryaml -rjson -e 'puts JSON.generate(YAML.load_file(ARGV[0]))' "$workflow" > "$T/wf.json" 2> /dev/null; then + wf() { jq -r "$1" "$T/wf.json"; } + secret_jobs="$(wf '[.jobs | to_entries[] | select(.value | tostring | test("secrets\\.")) | .key] | join(",")')" + check "workflow: started by workflow_dispatch only" is "$(wf '(.on // .["true"]) | keys | join(",")')" workflow_dispatch + check "workflow: dry-run is a boolean that defaults to true" is "$(wf '(.on // .["true"]).workflow_dispatch.inputs["dry-run"] | [.type, (.default | tostring)] | join(",")')" boolean,true + check "workflow: the tag is a required input" is "$(wf '(.on // .["true"]).workflow_dispatch.inputs.tag.required | tostring')" true + check "workflow: no permissions by default" is "$(wf '.permissions | length')" 0 + check "workflow: jobs" is "$(wf '.jobs | keys | join(",")')" assemble,build,native,plan,publish,rehearse + check "workflow: only publish names an environment" is "$(wf '[.jobs | to_entries[] | select(.value | has("environment")) | .key + "=" + (.value.environment | tostring)] | join(",")')" publish=release + check "workflow: only publish reads secrets" is "$secret_jobs" publish + check "workflow: publish reads exactly the four publishing secrets" is "$(wf '[.jobs.publish | tostring | scan("secrets\\.([A-Z_]+)") | .[0]] | unique | join(",")')" CENTRAL_PASSWORD,CENTRAL_USERNAME,GPG_PASSPHRASE,GPG_PRIVATE_KEY + check "workflow: secrets reach single steps only, never a whole job" is "$(wf '[.env // {}, (.jobs[].env // {})] | tostring | test("secrets\\.")')" false + check "workflow: each secret goes only to the step that needs it" is "$(wf '[.jobs.publish.steps[] | select(.env // {} | tostring | test("secrets\\.")) | .name + ":" + ([.env | to_entries[] | select(.value | tostring | test("secrets\\.")) | .key] | join("+"))] | join(",")')" \ + "Import the signing key:GPG_PRIVATE_KEY,Sign and bundle:GPG_PASSPHRASE,Upload to Maven Central:CENTRAL_USERNAME+CENTRAL_PASSWORD,Drop Maven Central deployments after a failure:CENTRAL_USERNAME+CENTRAL_PASSWORD" + check "workflow: only publish can write" is "$(wf '[.jobs | to_entries[] | select(.value.permissions != {"contents": "read"}) | .key + "=" + (.value.permissions | tostring)] | join(",")')" 'publish={"contents":"write"}' + check "workflow: the job token is used by the GitHub Release step only" is "$(wf '[.jobs[].steps[] | select(tostring | test("github\\.token|GITHUB_TOKEN")) | .name] | join(",")')" "GitHub Release" + cond="$(wf '.jobs.publish.if')" + check "workflow: publish needs the plan's verdict" grep -qF "needs.plan.outputs.mode == 'publish'" <<< "$cond" + check "workflow: ... and, independently, the raw input" grep -qF "!(github.event_name == 'workflow_dispatch' && inputs.dry-run)" <<< "$cond" + check "workflow: ... joined with &&, no ||, no status function that would override a failed need" test -z "$(grep -E '\|\||always\(\)|failure\(\)|cancelled\(\)' <<< "$cond")" + check "workflow: publish runs after assemble and the rehearsal" is "$(wf '.jobs.publish.needs | sort | join(",")')" assemble,plan,rehearse + check "workflow: no other job depends on publish" is "$(wf '[.jobs[] | .needs // [] | if type == "array" then .[] else . end | select(. == "publish")] | length')" 0 + check "workflow: one publish at a time, never cancelled by a newer one" is "$(wf '.jobs.publish.concurrency | [.group, (.["cancel-in-progress"] | tostring)] | join(",")')" release-publish,false + check "workflow: nothing is restored from or saved to a cache" is "$(wf '[.jobs[].steps[] | select(((.uses // "") | test("cache")) or ((.with // {}) | has("cache")))] | length')" 0 + check "workflow: project code is checked out only by native and build" is "$(wf '[.jobs | to_entries[] | select([.value.steps[] | select((.uses // "") | startswith("actions/checkout@")) | .with.ref // ""] | any(. == "${{ needs.plan.outputs.commit }}")) | .key] | sort | join(",")')" build,native + check "workflow: assemble, rehearse and publish check out the release scripts only" is "$(wf '[.jobs | to_entries[] | select(.key == "assemble" or .key == "rehearse" or .key == "publish") | .value.steps[] | select((.uses // "") | startswith("actions/checkout@")) | [.with.ref, .with["sparse-checkout"]] | join(" ")] | unique | join(",")')" '${{ github.workflow_sha }} .github/scripts/release' + check "workflow: no JDK and no Maven in assemble, rehearse or publish" is "$(wf '[.jobs | to_entries[] | select(.key == "assemble" or .key == "rehearse" or .key == "publish") | .value.steps[] | select(((.uses // "") | test("setup-java")) or ((.run // "") | test("mvn|java |cargo")))] | length')" 0 + check "workflow: publish uses two actions, checkout and download-artifact" is "$(wf '[.jobs.publish.steps[] | .uses // empty | sub("@.*"; "")] | unique | join(",")')" actions/checkout,actions/download-artifact + check "workflow: every checkout drops its credentials" is "$(wf '[.jobs[].steps[] | select((.uses // "") | startswith("actions/checkout@")) | .with["persist-credentials"]] | (length == 8 and all(. == false))')" true + check "workflow: every job has a timeout" is "$(wf '[.jobs[] | has("timeout-minutes")] | all')" true + check "workflow: hosted runner, pinned image" is "$(wf '[.jobs[]["runs-on"]] | unique | join(",")')" ubuntu-24.04 + check "workflow: no expression is expanded inside a run script" is "$(wf '[.jobs[].steps[] | .run? // empty | select(test("\\$\\{\\{"))] | length')" 0 + check "workflow: build matrix lists the profiles the scripts expect" is "$(wf '.jobs.build.strategy.matrix.profile | join(" ")')" "$PROFILES" + steps="$(wf '[.jobs.publish.steps[].name] | join("|")')" + check "workflow: publish verifies before it imports, and removes the key before any upload" is "$steps" \ + "Checkout release scripts (workflow commit)|Keep the keyring and upload state outside the workspace|Download staging directory|Verify staging directory|Import the signing key|Sign and bundle|Check bundles|Remove the signing key|Upload to Maven Central|GitHub Release|Drop Maven Central deployments after a failure" + check "workflow: the key is removed even when an earlier step failed" is "$(wf '.jobs.publish.steps[] | select(.name == "Remove the signing key") | .if')" "always()" + check "workflow: rehearse runs the same verify, sign and check commands as publish" is \ + "$(wf '[.jobs.rehearse.steps[] | .run? // empty | select(test("verify-staging|sign-bundles|check-bundles"))] | join("|")')" \ + "$(wf '[.jobs.publish.steps[] | .run? // empty | select(test("verify-staging|sign-bundles|check-bundles"))] | join("|")')" + check "workflow: both take the manifest digest from the assemble job's output" is "$(wf '[.jobs.rehearse, .jobs.publish | .steps[] | select(.name == "Verify staging directory") | .env.MANIFEST_SHA256] | unique | join(",")')" '${{ needs.assemble.outputs.manifest-sha256 }}' + check "workflow: plan gets the inputs as strings, through the environment" is "$(wf '.jobs.plan.steps[] | select(.id == "plan") | .env | [.INPUT_TAG, .INPUT_DRY_RUN, .INPUT_EXPECTED_SPARK, .EVENT_NAME, .REF] | join(" ")')" '${{ inputs.tag }} ${{ inputs.dry-run }} ${{ inputs.expected-spark-versions }} ${{ github.event_name }} ${{ github.ref }}' + check "workflow: inputs" is "$(wf '(.on // .["true"]).workflow_dispatch.inputs | keys | join(",")')" dry-run,expected-spark-versions,replace-release-assets,skip-central,tag + check "workflow: expected-spark-versions is optional and empty by default" is "$(wf '(.on // .["true"]).workflow_dispatch.inputs["expected-spark-versions"] | [.type, (.required // false | tostring), .default] | join(",")')" string,false, + check "workflow: replace-release-assets is a boolean that defaults to false" is "$(wf '(.on // .["true"]).workflow_dispatch.inputs["replace-release-assets"] | [.type, (.default | tostring)] | join(",")')" boolean,false + check "workflow: ... and reaches the GitHub Release step only" is "$(wf '[.jobs[].steps[] | select(tostring | test("replace-release-assets")) | .name + ":" + .env.REPLACE_ASSETS] | join(",")')" 'GitHub Release:${{ inputs.replace-release-assets }}' + check "workflow: the plan publishes the versions it worked out" is "$(wf '.jobs.plan.outputs.versions')" '${{ steps.plan.outputs.versions }}' + check "workflow: build, assemble, rehearse and publish are all held to those versions" is "$(wf '[.jobs | to_entries[] | select(tostring | test("EXPECTED_VERSIONS")) | .key] | sort | join(",")')" assemble,build,publish,rehearse + check "workflow: ... taken from the plan job and nowhere else" is "$(wf '[.. | objects | .EXPECTED_VERSIONS? // empty] | unique | join(",")')" '${{ needs.plan.outputs.versions }}' + check "workflow: no test-only setting and no redirection is ever set" test -z "$(grep -n -E 'RELEASE_SCRIPTS_TESTING|CENTRAL_(POLL|WAIT|DROP_WAIT)_SECONDS|CENTRAL_API_URL|CENTRAL_REPO_URL|T4J_REPO|M2_REPO|GIT_CONFIG' "$workflow")" + check "workflow: the last step runs after a failure and after a cancellation" is "$(wf '.jobs.publish.steps[-1] | [.name, .if] | join("|")')" "Drop Maven Central deployments after a failure|\${{ (failure() || cancelled()) && needs.plan.outputs.central == 'true' }}" + check "workflow: header does not claim more than this file can guarantee" test -n "$(grep -F '# * In this file, `publish` is the only job that names the `release`' "$workflow")" -a -n "$(grep -F 'every workflow on' "$workflow")" -a -n "$(grep -F 'v0.6.0-rc1 and v0.6.0-rc2 carry the previous release.yml' "$workflow")" + check "workflow: no continue-on-error anywhere" is "$(wf '[.. | objects | select(has("continue-on-error"))] | length')" 0 + # Every `run:` step, as the runner would start it (bash -eo pipefail). + # Steps that call a release script must name one that exists; the other + # steps are small enough to execute here. + missing="" + while IFS= read -r script; do + [ -f "$here/$script" ] || missing="$missing $script" + done < <(wf '.jobs[].steps[] | .run? // empty' | grep -o -E '(\$SCRIPTS|\.github/scripts/release)/[a-z-]+\.sh' | sed 's|.*/||' | sort -u) + check "workflow: every script a step calls exists" test -z "$missing" + check "workflow: every run step either calls one release script or is one of the three inline steps" is \ + "$(wf '[.jobs[].steps[] | select(has("run")) | select(.run | test("^bash (\"\\$SCRIPTS|\\.github/scripts/release)/[a-z-]+\\.sh\"?( [a-z-]+| \"\\$CENTRAL_STATE\")*\\n?$") | not) | .name] | join("|")')" \ + "Keep the keyring outside the workspace|Summary|Keep the keyring and upload state outside the workspace" + inline() { # inline [VAR=value...]: run that step's script + local job="$1" name="$2"; shift 2 + wf ".jobs.$job.steps[] | select(.name == \"$name\") | .run" > "$T/inline.sh" + env RUNNER_TEMP="$T/rt" GITHUB_ENV="$T/inline.env" GITHUB_STEP_SUMMARY="$T/inline.summary" "$@" bash --noprofile --norc -eo pipefail "$T/inline.sh" + } + : > "$T/inline.env" + check "workflow: rehearse sets the keyring directory" inline rehearse "Keep the keyring outside the workspace" + check "workflow: ... to a directory signing-key.sh accepts" is "$(cat "$T/inline.env")" "GNUPGHOME=$T/rt/release-gnupg" + : > "$T/inline.env" + check "workflow: publish sets the keyring and state directories" inline publish "Keep the keyring and upload state outside the workspace" + check "workflow: ... both outside the workspace" is "$(tr '\n' ' ' < "$T/inline.env")" "GNUPGHOME=$T/rt/release-gnupg CENTRAL_STATE=$T/rt/central-state " + : > "$T/inline.summary" + check "workflow: rehearse summary step, dry run" inline rehearse Summary MODE=dry-run + check "workflow: ... says nothing was uploaded" has "$T/inline.summary" "nothing was uploaded and no release was created" + : > "$T/inline.summary" + check "workflow: rehearse summary step, publish run" inline rehearse Summary MODE=publish + check "workflow: ... says approval comes next" has "$T/inline.summary" "waits for approval" + check "workflow: ... and does not claim a dry run" hasnt "$T/inline.summary" "Dry run complete" +elif [ "${REQUIRE_WORKFLOW_CHECKS:-0}" = "1" ]; then + bad "workflow: checks could not run (ruby not available, or release.yml does not parse)" +else + echo "skip workflow checks (ruby not available)" +fi + +# Pinned actions and runner image, in this workflow and the one that runs +# these tests. +unpinned="" +for f in "$workflow" "$root/.github/workflows/release-scripts.yml"; do + n=$(grep -c -E '^[[:space:]]*(- )?uses:' "$f") + m=$(grep -c -E '^[[:space:]]*(- )?uses: [A-Za-z0-9._/-]+@[0-9a-f]{40} # v[0-9]+(\.[0-9]+)*$' "$f") + if [ "$n" -eq 0 ] || [ "$n" -ne "$m" ]; then unpinned="$unpinned $(basename "$f")"; fi +done +check "pins: every action is a full commit id with its version tag in a comment" test -z "$unpinned" +check "pins: no floating runner label" test -z "$(grep -h -E 'runs-on:' "$workflow" "$root/.github/workflows/release-scripts.yml" | grep -v -E 'runs-on: ubuntu-24\.04$')" +check "scripts: none enables command tracing" test -z "$(grep -l -E '^[[:space:]]*set -[a-z]*x|xtrace' "$here"/*.sh | grep -v test.sh)" +check "scripts: no installer is piped into a shell" test -z "$(grep -n -E 'curl[^|]*\|[[:space:]]*(ba)?sh' "$here"/*.sh | grep -v test.sh)" + +echo +echo "$pass passed, $fail failed" +[ "$fail" -eq 0 ] diff --git a/.github/scripts/release/verify-staging.sh b/.github/scripts/release/verify-staging.sh new file mode 100644 index 00000000..88b926a3 --- /dev/null +++ b/.github/scripts/release/verify-staging.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# Check a downloaded staging directory before anything in it is signed. +# +# verify-staging.sh +# +# Environment: +# MANIFEST_SHA256 the digest the assemble job reported as a job output +# BASE_VERSION the tag without its "v" +# EXPECTED_VERSIONS the versions plan.sh announced, one per profile +# +# The staging directory travels between jobs as a workflow artifact. Its +# manifest digest travels separately, as a job output, so the two can be +# compared here. After this passes, the directory holds exactly the files the +# assemble job checked and listed: nothing added, removed or changed, and +# every version in it belongs to the tag being released. +set -euo pipefail +here="$(cd "$(dirname "$0")" && pwd)" +# shellcheck source=lib.sh +. "$here/lib.sh" + +staging="${1:?usage: verify-staging.sh }" +: "${MANIFEST_SHA256:?}" "${BASE_VERSION:?}" "${EXPECTED_VERSIONS:?}" +[[ "$MANIFEST_SHA256" =~ ^[0-9a-f]{64}$ ]] || die "MANIFEST_SHA256 is not a SHA-256 digest" + +[ -f "$staging/MANIFEST.sha256" ] || die "$staging has no MANIFEST.sha256" +got="$(hash_of sha256 "$staging/MANIFEST.sha256")" +[ "$got" = "$MANIFEST_SHA256" ] \ + || die "the staging manifest has SHA-256 $got, but the assemble job reported $MANIFEST_SHA256" +verify_sums "$staging" MANIFEST.sha256 + +# Independent of what assemble.sh did: one version per Spark profile, each +# belonging to this tag, and the manifest may only list the notes, the +# version list and the allow-listed files of those versions. +[ -s "$staging/versions.txt" ] || die "the staging directory lists no versions" +set -- $PROFILES +while IFS= read -r version; do + [ $# -gt 0 ] || die "the staging directory has more versions than Spark profiles" + check_version "$BASE_VERSION" "$1" "$version" + shift +done < "$staging/versions.txt" +[ $# -eq 0 ] || die "the staging directory has no version for: $*" +[ "$(tr '\n' ' ' < "$staging/versions.txt" | sed 's/ $//')" = "$EXPECTED_VERSIONS" ] \ + || die "the staged versions are not the versions the plan announced ($EXPECTED_VERSIONS)" + +expected="$( + { + echo release-notes.md + echo versions.txt + while IFS= read -r version; do + for name in $(artifact_names "$version"); do + echo "bundles/$version/$GROUP_PATH/$version/$name" + for algo in $CHECKSUM_ALGOS; do echo "bundles/$version/$GROUP_PATH/$version/$name.$algo"; done + done + done < "$staging/versions.txt" + } | LC_ALL=C sort +)" +listed="$(awk '{ print $2 }' "$staging/MANIFEST.sha256" | LC_ALL=C sort)" +[ "$expected" = "$listed" ] || die "the staging manifest lists files other than the expected ones" + +echo "Staging directory verified: manifest $MANIFEST_SHA256, versions: $(tr '\n' ' ' < "$staging/versions.txt")" diff --git a/.github/workflows/release-scripts.yml b/.github/workflows/release-scripts.yml new file mode 100644 index 00000000..09fd7871 --- /dev/null +++ b/.github/workflows/release-scripts.yml @@ -0,0 +1,56 @@ +name: Release Scripts + +# Offline tests for the scripts behind release.yml and for the shape of +# release.yml itself, run on the runner image those scripts run on, with the +# gpg that will do the signing (a throwaway key in a temporary keyring). +# Uses no secrets, no environment and a read-only token, and uploads nothing, +# so it is safe as an ordinary pull_request workflow. The release workflow +# itself cannot be exercised from a pull request; see the dry run in +# .github/scripts/release/README.md. +# +# pom.xml and scripts/setup.sh are in the path filter because the tests check +# them against the pins: the tantivy4java version in pom.xml needs a line in +# native-pins.txt, and the protoc pin must match the one in setup.sh. + +on: + pull_request: + paths: + - .github/scripts/release/** + - .github/workflows/release.yml + - .github/workflows/release-scripts.yml + - pom.xml + - scripts/setup.sh + push: + branches: [main] + paths: + - .github/scripts/release/** + - .github/workflows/release.yml + - .github/workflows/release-scripts.yml + - pom.xml + - scripts/setup.sh + +permissions: + contents: read + +concurrency: + group: release-scripts-${{ github.ref }} + cancel-in-progress: true + +jobs: + test: + name: Offline tests + runs-on: ubuntu-24.04 + timeout-minutes: 10 + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Run the tests + env: + # Fail, rather than fall back or skip, if gpg or the workflow-file + # checks are not available. + REQUIRE_GPG: '1' + REQUIRE_WORKFLOW_CHECKS: '1' + run: bash .github/scripts/release/test.sh diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 3e5bd4e7..db1dec3f 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,136 +1,507 @@ name: Release -# Tag-triggered release: pushing v (e.g. v0.6.0) builds and publishes -# one artifact per supported Spark version, versioned _spark_ -# (e.g. 0.6.0_spark_4.0.3), each containing the linux-x86_64 tantivy4java -# native library: -# - Maven Central via the `release` profile (GPG + central-publishing plugin) -# - shaded jars attached to a GitHub Release for the tag +# Publishes one artifact per supported Spark version for an existing tag +# v (e.g. v0.6.0). Each is versioned _spark_ +# (e.g. 0.6.0_spark_4.0.3) and contains the linux-x86_64 tantivy4java native +# library: +# - Maven Central: io.indextables:indextables_spark:_spark_ +# (pom, jar, sources, javadoc, linux-x86_64-shaded, jar-with-dependencies) +# - the linux-x86_64-shaded jars attached to the GitHub Release for the tag +# is the spark.version that the profile sets in pom.xml at the tag. +# It changes when the Spark dependency is bumped. The Plan summary shows the +# exact versions next to the ones last published, and a publish run whose +# Spark versions differ from the last release stops in `plan` until the +# releaser has typed them into `expected-spark-versions`. +# +# How to run it +# ------------- +# Actions -> Release -> Run workflow, on `main`, with the tag to release. +# `dry-run` is ticked by default: the run then builds, assembles and rehearses +# signing, and stops. Untick it to publish. Pushing a tag does not start a +# release. The runbook, including how to adopt this workflow, the release +# candidate rehearsal and how to abort, is in +# .github/scripts/release/README.md. +# +# Trust model +# ----------- +# * `native` and `build` run the project's build (Maven, its plugins, its +# dependencies, cargo) on the tagged commit. They have a read-only token, no +# environment and no secrets, and restore nothing from a cache: the native +# library is compiled in the run, from the commits pinned in +# native-pins.txt. +# * `assemble`, `rehearse` and `publish` run no project code. They run the +# scripts in .github/scripts/release from the commit this workflow was +# started on, on files the build jobs produced. `assemble` treats those +# files as untrusted: it accepts only the expected coordinates and file +# names and writes a manifest of digests. +# * In this file, `publish` is the only job that names the `release` +# environment, reads a secret or holds a token that can write. Beside the +# signing key and the Central token it runs: the pinned checkout and +# download-artifact actions, the release scripts, and gpg, zip, unzip, curl, +# jq and gh from the runner image. No Maven, no JVM, nothing from the tagged +# commit. It signs and uploads exactly the files in the manifest whose +# digest `assemble` reported, and nothing else. +# * A dry run never starts `publish`, so it requests no approval and reads no +# secret. `rehearse` runs the same verification, signing and bundle checks +# as `publish`, with a throwaway key it generates and deletes. +# +# What this file cannot guarantee +# ------------------------------ +# All of the above describes this file on `main`. It says nothing about other +# workflow files or other refs, and two things outside it decide whether the +# separation protects anything: +# +# * While GPG_PRIVATE_KEY, GPG_PASSPHRASE, CENTRAL_USERNAME and +# CENTRAL_PASSWORD exist as repository-level secrets, every workflow on +# every ref can read them, whatever this file does. +# * The tags v0.6.0-rc1 and v0.6.0-rc2 carry the previous release.yml, which +# has a workflow_dispatch trigger. It can be dispatched on the tag ref and +# then runs the old build beside the key. Only an environment that admits +# `main` alone, holding the only copy of the secrets, closes that. +# +# The order in which to get there (secrets into the environment, environment +# narrowed to `main` with administrator bypass off, dry run, release candidate +# rehearsal, repository-level copies deleted) is in the README under +# "Adopting this workflow". + on: - push: - tags: ['v*'] workflow_dispatch: inputs: tag: description: 'Existing v* tag to release (e.g. v0.6.0)' required: true + type: string + dry-run: + description: 'Dry run: build, assemble and rehearse signing; upload nothing. Untick to publish.' + type: boolean + default: true + expected-spark-versions: + description: 'Spark versions expected in the artifact names, e.g. "3.5.9 4.0.4 4.1.3". Needed to publish when they differ from the last release; the dry run prints the value.' + type: string + default: '' skip-central: - description: 'Skip Maven Central publish (GitHub Release only)' + description: 'Skip Maven Central (GitHub Release only). No effect in a dry run.' type: boolean default: false + replace-release-assets: + description: 'Replace jars already attached to the GitHub Release when their content differs. Unticked: stop instead.' + type: boolean + default: false + # To start a release from a pushed tag as well, add: + # + # push: + # tags: ['v*'] + # + # plan.sh already handles that event. A pushed tag runs the workflow file + # and the scripts of the tagged commit, not those on main; see "Tag push or + # dispatch" in the README before enabling it. + +permissions: {} -permissions: - contents: write +defaults: + run: + shell: bash env: - RELEASE_TAG: ${{ github.event_name == 'workflow_dispatch' && inputs.tag || github.ref_name }} + SCRIPTS: release/.github/scripts/release jobs: - # tantivy4java is Spark-independent: build the native (Rust) jar once and - # share it across all Spark publish legs. The cache makes re-releases with - # an unchanged tantivy4java version skip the Rust build entirely - # (setup.sh no-ops when the jar is already in ~/.m2). - native: - name: tantivy4java (linux-x86_64) - runs-on: ubuntu-latest + # --------------------------------------------------------------------------- + # Validate the request, resolve the tag to a commit and work out the versions + # to publish. Every later job uses that commit and must arrive at those + # versions, so a tag that moves during the run changes nothing. + # --------------------------------------------------------------------------- + plan: + name: Plan + runs-on: ubuntu-24.04 + timeout-minutes: 10 + permissions: + contents: read outputs: - t4j-version: ${{ steps.t4j.outputs.version }} + mode: ${{ steps.plan.outputs.mode }} + tag: ${{ steps.plan.outputs.tag }} + commit: ${{ steps.plan.outputs.commit }} + base: ${{ steps.plan.outputs.base }} + versions: ${{ steps.plan.outputs.versions }} + prerelease: ${{ steps.plan.outputs.prerelease }} + central: ${{ steps.plan.outputs.central }} steps: - - name: Checkout - uses: actions/checkout@v7 + # Full history: plan.sh checks that the tag's commit is on main. + - name: Checkout (workflow commit, full history) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: - ref: ${{ env.RELEASE_TAG }} + fetch-depth: 0 + persist-credentials: false + + - name: Plan + id: plan + env: + EVENT_NAME: ${{ github.event_name }} + REF: ${{ github.ref }} + REF_TYPE: ${{ github.ref_type }} + REF_NAME: ${{ github.ref_name }} + DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} + INPUT_TAG: ${{ inputs.tag }} + INPUT_DRY_RUN: ${{ inputs.dry-run }} + INPUT_EXPECTED_SPARK: ${{ inputs.expected-spark-versions }} + INPUT_SKIP_CENTRAL: ${{ inputs.skip-central }} + run: bash .github/scripts/release/plan.sh + + # --------------------------------------------------------------------------- + # tantivy4java is Spark-independent: build the native (Rust) jar once, from + # source, and share it with the three build legs. No cache: about 11 minutes + # in every run, which is what each release run has taken so far. + # --------------------------------------------------------------------------- + native: + name: tantivy4java (linux-x86_64, from source) + needs: plan + runs-on: ubuntu-24.04 + timeout-minutes: 45 + permissions: + contents: read + env: + POM: src/pom.xml + PINS: release/.github/scripts/release/native-pins.txt + WORK: work/native + OUT: out/native + steps: + - name: Checkout release scripts (workflow commit) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.workflow_sha }} + path: release + sparse-checkout: .github/scripts/release + persist-credentials: false + + - name: Checkout source (tagged commit) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ needs.plan.outputs.commit }} + path: src + persist-credentials: false - name: Set up Java 17 - uses: actions/setup-java@v6 + uses: actions/setup-java@de7274f081f381c8f8158605e0321c36c376e2e6 # v6.0.1 with: distribution: 'temurin' java-version: '17' - - name: Extract tantivy4java version from pom.xml - id: t4j - run: | - version=$(grep -A2 'tantivy4java' pom.xml \ - | grep '' | sed -E 's/.*([^<]+)<\/version>.*/\1/' | head -1) - [[ -n "$version" ]] || { echo "could not extract tantivy4java version" >&2; exit 1; } - echo "version=$version" >> "$GITHUB_OUTPUT" + - name: Resolve pinned inputs + run: bash "$SCRIPTS/build-native.sh" resolve - - name: Cache tantivy4java build - id: t4j-cache - uses: actions/cache@v6 - with: - path: ~/.m2/repository/io/indextables/tantivy4java - key: tantivy4java-${{ steps.t4j.outputs.version }}-linux-x86_64 + - name: Prepare toolchain + run: bash "$SCRIPTS/build-native.sh" toolchain - - name: Build tantivy4java (setup.sh) - if: steps.t4j-cache.outputs.cache-hit != 'true' - run: ./scripts/setup.sh + - name: Build tantivy4java + run: bash "$SCRIPTS/build-native.sh" build - - name: Upload tantivy4java jars - uses: actions/upload-artifact@v7 + - name: Upload native build + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 with: - name: tantivy4java-linux-x86_64 - path: ~/.m2/repository/io/indextables/tantivy4java/${{ steps.t4j.outputs.version }}/ + name: native-linux-x86_64 + path: out/native if-no-files-found: error + overwrite: true + retention-days: 7 - publish: - name: Publish ${{ matrix.spark.profile }} - needs: native - runs-on: ubuntu-latest - environment: release + # --------------------------------------------------------------------------- + # Build, unsigned, everything that is published for one Spark profile. + # The profile list must match PROFILES in .github/scripts/release/lib.sh. + # --------------------------------------------------------------------------- + build: + name: Build ${{ matrix.profile }} + needs: [plan, native] + runs-on: ubuntu-24.04 + timeout-minutes: 30 + permissions: + contents: read strategy: fail-fast: false matrix: - spark: - - { profile: spark-3.5, version: 3.5.8 } - - { profile: spark-4.0, version: 4.0.3 } - - { profile: spark-4.1, version: 4.1.2 } + profile: [spark-3.5, spark-4.0, spark-4.1] steps: - - name: Checkout - uses: actions/checkout@v7 + - name: Checkout release scripts (workflow commit) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.workflow_sha }} + path: release + sparse-checkout: .github/scripts/release + persist-credentials: false + + - name: Checkout source (tagged commit) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: - ref: ${{ env.RELEASE_TAG }} + ref: ${{ needs.plan.outputs.commit }} + path: src + persist-credentials: false - - name: Set up Java 17 (with Central credentials + GPG key) - uses: actions/setup-java@v6 + - name: Set up Java 17 + uses: actions/setup-java@de7274f081f381c8f8158605e0321c36c376e2e6 # v6.0.1 with: distribution: 'temurin' java-version: '17' - server-id: central - server-username: CENTRAL_USERNAME - server-password: CENTRAL_PASSWORD - gpg-private-key: ${{ secrets.GPG_PRIVATE_KEY }} - gpg-passphrase: MAVEN_GPG_PASSPHRASE - - name: Download tantivy4java jars - uses: actions/download-artifact@v8 + - name: Download native build + uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + name: native-linux-x86_64 + path: in/native + + - name: Build artifacts + env: + SRC: src + PROFILE: ${{ matrix.profile }} + BASE_VERSION: ${{ needs.plan.outputs.base }} + EXPECTED_VERSIONS: ${{ needs.plan.outputs.versions }} + NATIVE_DIR: in/native + OUT: out/artifacts + run: bash "$SCRIPTS/build-artifacts.sh" + + - name: Upload artifacts + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: artifacts-${{ matrix.profile }} + path: out/artifacts + if-no-files-found: error + compression-level: 0 + overwrite: true + retention-days: 7 + + # --------------------------------------------------------------------------- + # Check what the build jobs produced and lay out exactly what will be signed + # and uploaded. Runs the release scripts only; no project code. + # --------------------------------------------------------------------------- + assemble: + name: Assemble + needs: [plan, native, build] + runs-on: ubuntu-24.04 + timeout-minutes: 20 + permissions: + contents: read + outputs: + manifest-sha256: ${{ steps.assemble.outputs.manifest_sha256 }} + versions: ${{ steps.assemble.outputs.versions }} + steps: + - name: Checkout release scripts (workflow commit) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.workflow_sha }} + path: release + sparse-checkout: .github/scripts/release + persist-credentials: false + + - name: Download native build + uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + name: native-linux-x86_64 + path: in/native + + # One directory per artifact: in/artifacts-/ + - name: Download artifacts + uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + pattern: artifacts-* + path: in + + - name: Assemble + id: assemble + env: + IN: in + OUT: staging + TAG: ${{ needs.plan.outputs.tag }} + COMMIT: ${{ needs.plan.outputs.commit }} + BASE_VERSION: ${{ needs.plan.outputs.base }} + EXPECTED_VERSIONS: ${{ needs.plan.outputs.versions }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: bash "$SCRIPTS/assemble.sh" + + - name: Upload staging directory + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 with: - name: tantivy4java-linux-x86_64 - path: ~/.m2/repository/io/indextables/tantivy4java/${{ needs.native.outputs.t4j-version }}/ + name: release-staging + path: staging + if-no-files-found: error + compression-level: 0 + overwrite: true + retention-days: 30 + + # --------------------------------------------------------------------------- + # Everything the publish job does before its first upload, with a throwaway + # key: fetch and verify the staging directory, sign, bundle, check. Runs in + # every run, so a publish run is rehearsed before it asks for approval. + # --------------------------------------------------------------------------- + rehearse: + name: Rehearse signing (throwaway key) + needs: [plan, assemble] + runs-on: ubuntu-24.04 + timeout-minutes: 15 + permissions: + contents: read + env: + BASE_VERSION: ${{ needs.plan.outputs.base }} + EXPECTED_VERSIONS: ${{ needs.plan.outputs.versions }} + # Protects the throwaway key only; it is not a secret. + GPG_PASSPHRASE: rehearsal + steps: + - name: Checkout release scripts (workflow commit) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.workflow_sha }} + path: release + sparse-checkout: .github/scripts/release + persist-credentials: false + + - name: Keep the keyring outside the workspace + run: echo "GNUPGHOME=$RUNNER_TEMP/release-gnupg" >> "$GITHUB_ENV" + + - name: Download staging directory + uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + name: release-staging + path: staging + + - name: Verify staging directory + env: + MANIFEST_SHA256: ${{ needs.assemble.outputs.manifest-sha256 }} + run: bash "$SCRIPTS/verify-staging.sh" staging + + - name: Generate a throwaway key + id: key + run: bash "$SCRIPTS/signing-key.sh" generate + + - name: Sign and bundle + env: + SIGNING_KEY: ${{ steps.key.outputs.fingerprint }} + run: bash "$SCRIPTS/sign-bundles.sh" staging bundles + + - name: Check bundles + env: + SIGNING_KEY: ${{ steps.key.outputs.fingerprint }} + run: bash "$SCRIPTS/check-bundles.sh" bundles staging + + - name: Remove the throwaway key + if: always() + run: bash "$SCRIPTS/signing-key.sh" remove + + - name: Summary + env: + MODE: ${{ needs.plan.outputs.mode }} + run: | + { + echo "### Rehearsal passed" + echo + echo "The staging directory was verified, signed with a throwaway key, bundled and checked. Those signatures and bundles are discarded with this job." + if [[ "$MODE" == publish ]]; then + echo + echo "The publish job now waits for approval in the \`release\` environment. Review the Plan and Assemble summaries first." + else + echo + echo "Dry run complete: nothing was uploaded and no release was created." + fi + } >> "$GITHUB_STEP_SUMMARY" + + # --------------------------------------------------------------------------- + # The only job with the `release` environment, secrets and a write token. + # Not started in a dry run: both the plan's verdict and the raw input have to + # say "publish". + # + # One publish at a time. A second release waits here; a third replaces the + # second in the queue (GitHub keeps one pending job per group). + # --------------------------------------------------------------------------- + publish: + name: Sign and publish + needs: [plan, assemble, rehearse] + if: ${{ needs.plan.outputs.mode == 'publish' && !(github.event_name == 'workflow_dispatch' && inputs.dry-run) }} + runs-on: ubuntu-24.04 + # Three uploads of about 100 MB, each followed by the Portal's validation. + # In the last release, build, upload and validation together took under + # two minutes per version. The limit covers the scripts' own worst case + # of 15 + 20 minutes per version. + timeout-minutes: 120 + environment: release + concurrency: + group: release-publish + cancel-in-progress: false + permissions: + contents: write + env: + BASE_VERSION: ${{ needs.plan.outputs.base }} + EXPECTED_VERSIONS: ${{ needs.plan.outputs.versions }} + steps: + - name: Checkout release scripts (workflow commit) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.workflow_sha }} + path: release + sparse-checkout: .github/scripts/release + persist-credentials: false - - name: Stamp release version + - name: Keep the keyring and upload state outside the workspace run: | - BASE="${RELEASE_TAG#v}" - VERSION="${BASE}_spark_${{ matrix.spark.version }}" - echo "RELEASE_VERSION=$VERSION" >> "$GITHUB_ENV" - mvn -B versions:set -DnewVersion="$VERSION" -DgenerateBackupPoms=false + echo "GNUPGHOME=$RUNNER_TEMP/release-gnupg" >> "$GITHUB_ENV" + echo "CENTRAL_STATE=$RUNNER_TEMP/central-state" >> "$GITHUB_ENV" + + - name: Download staging directory + uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + name: release-staging + path: staging - - name: Build shaded jar - run: mvn -B package -P${{ matrix.spark.profile }} -DskipTests + - name: Verify staging directory + env: + MANIFEST_SHA256: ${{ needs.assemble.outputs.manifest-sha256 }} + run: bash "$SCRIPTS/verify-staging.sh" staging - - name: Publish to Maven Central - if: ${{ !(github.event_name == 'workflow_dispatch' && inputs.skip-central) }} - run: mvn -B deploy -P${{ matrix.spark.profile }},release -DskipTests + - name: Import the signing key + id: key + if: needs.plan.outputs.central == 'true' + env: + GPG_PRIVATE_KEY: ${{ secrets.GPG_PRIVATE_KEY }} + run: bash "$SCRIPTS/signing-key.sh" import + + - name: Sign and bundle + if: needs.plan.outputs.central == 'true' + env: + SIGNING_KEY: ${{ steps.key.outputs.fingerprint }} + GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }} + run: bash "$SCRIPTS/sign-bundles.sh" staging bundles + + - name: Check bundles + if: needs.plan.outputs.central == 'true' + env: + SIGNING_KEY: ${{ steps.key.outputs.fingerprint }} + run: bash "$SCRIPTS/check-bundles.sh" bundles staging + + # The key is gone before the first request to Maven Central or GitHub. + - name: Remove the signing key + if: always() + run: bash "$SCRIPTS/signing-key.sh" remove + + - name: Upload to Maven Central + if: needs.plan.outputs.central == 'true' env: CENTRAL_USERNAME: ${{ secrets.CENTRAL_USERNAME }} CENTRAL_PASSWORD: ${{ secrets.CENTRAL_PASSWORD }} - MAVEN_GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }} + run: bash "$SCRIPTS/central-upload.sh" upload bundles staging "$CENTRAL_STATE" - - name: Attach shaded jar to GitHub Release + - name: GitHub Release env: GH_TOKEN: ${{ github.token }} - run: | - # First finishing leg creates the release; the rest just upload. - gh release create "$RELEASE_TAG" --verify-tag --generate-notes 2>/dev/null || true - gh release upload "$RELEASE_TAG" --clobber \ - target/indextables_spark-${RELEASE_VERSION}-linux-x86_64-shaded.jar + GH_REPO: ${{ github.repository }} + TAG: ${{ needs.plan.outputs.tag }} + COMMIT: ${{ needs.plan.outputs.commit }} + PRERELEASE: ${{ needs.plan.outputs.prerelease }} + REPLACE_ASSETS: ${{ inputs.replace-release-assets }} + run: bash "$SCRIPTS/github-release.sh" staging + + # If the job stops after bundles were uploaded (a step failed, or the run + # was cancelled, also in the middle of the upload step), do not leave + # deployments waiting in the Portal: a re-run uploads them again. A + # deployment that is still being validated is polled until it can be + # dropped; what cannot be dropped is listed in the job summary. + - name: Drop Maven Central deployments after a failure + if: ${{ (failure() || cancelled()) && needs.plan.outputs.central == 'true' }} + env: + CENTRAL_USERNAME: ${{ secrets.CENTRAL_USERNAME }} + CENTRAL_PASSWORD: ${{ secrets.CENTRAL_PASSWORD }} + run: bash "$SCRIPTS/central-upload.sh" drop "$CENTRAL_STATE"