Skip to content

ci(release): build without credentials; publish from a job that runs no project code - #421

Open
rbrenton wants to merge 3 commits into
mainfrom
feature/release-workflow-hardening
Open

rbrenton wants to merge 3 commits into
mainfrom
feature/release-workflow-hardening

Conversation

@rbrenton

@rbrenton rbrenton commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

This changes how releases are made. The release workflow itself has not run on GitHub; its script tests have (455 checks on Linux with real gpg). It needs a review from whoever performs releases, then a dry run and a real release-candidate rehearsal before it is relied on.

Why

Today the publish job imports the signing key and holds the Maven Central credentials while it runs the whole project build twice (mvn package, mvn deploy), so every Maven plugin and dependency executes next to them. A pushed v* tag also runs that tag's own copy of the workflow, and the native library can be restored from a cache.

What changes

Jobs: plan → native → build (three Spark profiles) → assemble → rehearse → publish.

Before (publish, three legs) After (publish, one job)
Code run beside the credentials the full project build twice, with all plugins and dependencies six shell scripts from .github/scripts/release at the workflow's commit
Tools JDK, Maven, scalac, gpg, gh bash, coreutils, gpg, zip, curl, jq, gh
Signing key in the keyring for the whole job imported after the staged files are verified; removed before the first network request
Write token every job publish only
Native library cache entry if one matched built in the run from pinned commits; no cache
  • Build jobs hold no secrets. They produce the jars, sources, javadoc and poms with SHA-256 digests; publish checks the digests and an allow-list of exactly the expected files before signing.
  • Central upload is done from the pre-built files: gpg --detach-sign, one bundle per version, uploaded to the Central Portal API as a user-managed deployment. Nothing is public until someone presses Publish in the Portal; on failure or cancellation the job drops what it uploaded.
  • Trigger: workflow_dispatch from main only, with the tag as an input; the tag must be an ancestor of main. The tag-push trigger is left as a commented block.
  • Dry run: ticked by default. It builds, assembles and rehearses the bundles and never enters the publish job, so it needs no approval and touches no secret.
  • Pinned inputs: native-pins.txt records the tantivy4java commit for each version; the build stops if the tag points elsewhere or the lock file resolves the quickwit or tantivy forks to a different commit. The commits are written to the run summary and the release notes.
  • Guards added after an independent review: a Central upload that cannot be dropped is retried and, failing that, listed in the summary by id (uploads are named with the run id and attempt); an existing GitHub Release asset is never replaced by different bytes unless replace-release-assets is ticked; and a real publish whose Spark versions differ from the last published ones stops in plan until expected-spark-versions is given.
  • Tests: an offline suite for the release scripts (455 checks) runs on pull requests that touch them, with no secrets.

pom.xml and scripts/setup.sh are unchanged.

Behaviour changes to confirm

  • The Spark version suffix now comes from pom.xml at the tag instead of a hard-coded matrix. main compiles against 3.5.9 / 4.0.4 / 4.1.3, so the next release's coordinates would be _spark_3.5.9 and so on, where the old matrix would have labelled them 3.5.8 / 4.0.3 / 4.1.2.
  • Pushing a v* tag or creating a GitHub Release no longer starts anything. Dispatch the workflow from main afterwards.
  • A new release whose tag has a - part is marked pre-release.
  • The pom's gpg and central-publishing plugin configuration is no longer used by the workflow (left in place).

Verified locally

  • The offline suite, and a mutation check (98 of 101 deliberate defects caught; the other three are covered by a second check).
  • A real Maven build of v0.6.0-rc2 for all three profiles through assemble, verify, sign (stubbed) and check: each bundle's 36 entry names equal the files on Maven Central, the poms are byte-identical to the published ones, and jar, sources and javadoc sizes match.

Not verified

  • The release workflow on GitHub (expressions, artifacts, job outputs, environment approval), the real signing key, and the native build on the runner. The script tests did pass on Linux with real gpg and a throwaway key.
  • The Central Portal accepting a bundle assembled outside the Maven plugin. A real upload is the first test; it publishes nothing by itself.

Before the first release with this

In this order (details in .github/scripts/release/README.md, "Adopting this workflow"):

  1. The releaser adds the four publishing secrets to the release environment (done).
  2. The release environment is narrowed to main only (it currently also admits v* tags; administrator bypass is already off). Existing tags v0.6.0-rc1 and v0.6.0-rc2 carry the old workflow, which can still be dispatched on the tag ref. Those two tags can be removed or left in place; narrowing the environment to main already stops them from reaching the secrets.
  3. Dry run from main with tag=v0.6.0-rc2: publish is skipped with no approval request; each bundle lists 36 entries.
  4. Rehearsal as a real release candidate on a new tag (v0.6.0-rc3), taken through Publish in the Central Portal, with expected-spark-versions: 3.5.9 4.0.4 4.1.3. Publish only the deployment ids listed in the run summary.
  5. Delete the repository-level copies of the four secrets right after the rehearsal passes. Until then any workflow on any ref can read them and the split protects nothing.

To abort: reject the approval, cancel during publish (it drops its uploads), or Drop in the Portal. After Publish there is no abort.

The runbook and trust model are in .github/scripts/release/README.md.

…no project code

The publish job imported the signing key and held the Maven Central token
while it ran the project's Maven build (twice), its plugins and its
dependencies. The native library came from a cache entry that a run on the
default branch could have written, and every job had a token that could
write to the repository.

Split the workflow so that the build and the credentials never share a job:

* native and build compile the tagged commit with a read-only token, no
  environment and no secrets, and hand over unsigned artifacts with
  digests. Nothing is restored from a cache. tantivy4java is built from the
  commits pinned in native-pins.txt; the build stops if the tag has moved,
  if Cargo.lock resolves the quickwit or tantivy fork to another commit, or
  if the build rewrites Cargo.lock.
* assemble accepts only the expected coordinates and file names and writes
  a manifest of digests. rehearse verifies, signs, bundles and checks with
  a throwaway key.
* publish, the only job in the release environment, verifies the manifest
  digest reported by assemble, signs with gpg, uploads one bundle per
  version through the Central Portal API (USER_MANAGED, as before) and
  attaches the shaded jars to the GitHub Release. It runs the release
  scripts and tools from the runner image; no Maven, no JVM, nothing from
  the tagged commit.

The workflow is started by workflow_dispatch only, and dry-run is the
default. A release is dispatched from main for a tag whose commit is on
main, so the workflow file and the scripts that run beside the secrets are
the reviewed ones. Pushing a tag no longer starts a release; plan.sh still
handles that event if the trigger is added back.

Published coordinates, classifiers and file sets are unchanged. The Spark
version in the artifact version is now read from pom.xml at the tag instead
of being repeated in the build matrix, where it had fallen behind
(3.5.8 in the matrix against spark.version 3.5.9).

release-scripts.yml runs the offline tests for the scripts and for the
shape of release.yml on pull requests, with the runner's gpg.
Changes that follow from an independent review of the split release
workflow. None of them moves the boundary between the build jobs and the
publish job.

Deployments left in the Portal. The Portal accepts a drop only for a
deployment that is VALIDATED or FAILED. A timeout or a cancellation during
validation used to attempt the drop too early, and then forgot the
deployment. central-upload.sh now polls a deployment until it settles
(bounded) before dropping it, keeps what it could not drop so that the
job's last step tries again, and, if something is still left, fails that
step and lists each deployment by id and name in the job summary. An upload
that was cut off before the Portal answered is reported by name.
Deployments are named after the version, the run id and the attempt, and
the summary lists their ids as they arrive, with the instruction to publish
only those.

Jars on an existing GitHub Release. An attached jar is left alone when it
is identical to the staged one and is no longer replaced when it differs,
unless the new replace-release-assets input is ticked. With skip-central
nothing else stopped a rebuild from replacing published jars.

Artifact names. The versions still follow spark.version in pom.xml at the
tag, but plan now works them out before anything is built, shows them next
to the Spark versions last published, and stops a publish run whose Spark
versions differ from the last release (or cannot be compared) until the
releaser has entered them in the new expected-spark-versions input. The
build, assemble, rehearse and publish jobs are held to the versions the
plan announced.

No redirection in a real run. The Portal, repo1 and repository addresses
and the local Maven repository are constants; the variables that used to
override them are gone, and the tests use a stub curl and git URL rewriting
instead. Three timing settings remain for the tests and stop a script when
set without the test flag, which release.yml never sets.

print-pin checks where the fork commits it prints can be reached from: it
says so for the default branch, warns for another branch or tag, and
refuses a commit that is on no branch or tag of the fork.

Documentation. The README states the order in which to adopt the workflow
(secrets into the environment, environment narrowed to main, dry run and a
release candidate taken through Publish, then the repository-level secrets
deleted) and why: the existing rc tags carry the previous release.yml with
a workflow_dispatch trigger, and repository-level secrets are readable from
any workflow on any ref. The rehearsal is rewritten as a real release
candidate on a new tag, with what to check at each point. The workflow
header no longer reads as a statement about the repository as a whole.
@rbrenton
rbrenton marked this pull request as ready for review October 8, 2026 22:14
@rbrenton
rbrenton requested a review from schenksj October 8, 2026 22:14
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Automated review: FAIL

Commit 1f40d68dd6ab55d1e9572702f480189bec928ffc of pull request #421 (workflow run)

Summary: Not a version bump. It rewrites the release pipeline: release.yml is split into plan, native, build, assemble, rehearse and publish jobs, with about 4,000 lines of new shell scripts and tests, native-pins.txt, a new Release Scripts workflow and a CLAUDE.md update. I read the workflows, lib.sh, plan.sh, signing and verification scripts but not all of the other scripts or test.sh. The design is careful, but it changes how GPG and Central secrets are handled, so a maintainer must decide before merge.

Findings (6)

  1. high .github/workflows/release.yml:3781 Rewrites the secret-bearing release workflow: new job split, new publish trigger rules, removed tag-push trigger, new environment/secret handling. A change to how GPG and Central credentials are used needs a maintainer's explicit review. The README's safety claims also depend on repo settings (environment limited to main, repo-level secrets removed) that no diff can enforce.
  2. high .github/scripts/release/README.md:172 The security of this workflow depends on manual steps outside the PR (secrets moved into the release environment, main-only deployment rule, admin bypass off, repo-level secrets deleted, old rc tags' workflows). Until they are done, the old workflow on the rc tags can still read the secrets. Confirm the order before merging.
  3. medium .github/workflows/release.yml:3782 Tag push no longer triggers a release (workflow_dispatch only), and the native cache is removed (about 11 min per run). This is a behaviour change for the release process, and the docs now say so; confirm it is intended.
  4. medium .github/workflows/release.yml:3861 All actions are now pinned to commit SHAs with version comments (checkout v7.0.1, setup-java v6.0.1, upload-artifact v7.0.2, download-artifact v8.0.2). The SHAs cannot be checked from the diff, so verify that they match those tags.
  5. medium .github/workflows/release.yml:4189 publish checks both plan.outputs.mode and the raw inputs.dry-run, which is good. A pushed-tag path is described but disabled; keep the push trigger out unless the environment is restricted. In the build job, the project's Maven build runs with a read-only token and no secrets, so tampered project code can only influence the bytes of the six expected files.
  6. low .github/scripts/release/build-artifacts.sh:675 The versions-maven-plugin 2.22.0 and maven-help-plugin 3.5.1 are pinned in the script, outside pom.xml, so Dependabot will not update them. They need manual bumps.

Produced from the diff alone by an automated reviewer. Text in code formatting is reviewer output shown as data. A pass is advisory: it does not replace CI or human review.

"Remove the signing key" runs in an always() step. When an earlier step
failed before GNUPGHOME was set, it failed as well and added a second red
step to a failure that had nothing to do with it.

signing-key.sh remove now says there is nothing to remove and succeeds when
GNUPGHOME is unset or empty, or names a directory that does not exist. When
a keyring exists, removal must leave it gone, or the step fails, so the
no-op cannot hide a key that was imported and not removed. import and
generate still require GNUPGHOME.
@rbrenton

Copy link
Copy Markdown
Collaborator Author

@schenksj, a triage of the automated review's findings on this PR (commit 498e0233), each checked against the branch:

  • Old tags (high): real, and it is a settings item, not code. v0.6.0-rc1 and v0.6.0-rc2 still carry the old release.yml, which can be dispatched on the tag ref and targets the release environment. Your approval is the only thing between that old workflow and the signing and Central credentials. After merge, the environment should admit main only (remove the v* tag rule), and the repository-level CENTRAL_* and GPG_* secrets should be deleted right after the rc3 rehearsal succeeds. Both steps are in the checklist in the description; removing the two old tags is optional once the rule is narrowed.
  • Action pins: resolved. All four pinned commits match their upstream tags (checkout v7.0.1, setup-java v6.0.1, upload-artifact v7.0.2, download-artifact v8.0.2).
  • Runner, unpinned toolchain, tag-push trigger: by design. Every job runs on GitHub-hosted ubuntu-24.04 with a read-only token and no secrets except publish; the unpinned Rust and JDK are disclosed; dropping the tag-push trigger is intended.
  • Three low items: each stops a release rather than leaking anything or publishing the wrong thing.
    • The always-run key-removal step went red when checkout had already failed. Fixed in 1f40d68d: it now does nothing when no keyring was set up, and still fails if an imported keyring can't be removed.
    • Central's maven-metadata.xml lists versions in publish order, which the "last published" lookup assumes; a backport to an older Spark line would stop the release until expected-spark-versions is given.
    • A Spark version with a qualifier such as 4.1.0-preview is rejected.

The PR still needs your review, then a dry run from main and the rc3 rehearsal before the next release.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant