Repository navigation
Conversation
…no project code The publish job imported the signing key and held the Maven Central token while it ran the project's Maven build (twice), its plugins and its dependencies. The native library came from a cache entry that a run on the default branch could have written, and every job had a token that could write to the repository. Split the workflow so that the build and the credentials never share a job: * native and build compile the tagged commit with a read-only token, no environment and no secrets, and hand over unsigned artifacts with digests. Nothing is restored from a cache. tantivy4java is built from the commits pinned in native-pins.txt; the build stops if the tag has moved, if Cargo.lock resolves the quickwit or tantivy fork to another commit, or if the build rewrites Cargo.lock. * assemble accepts only the expected coordinates and file names and writes a manifest of digests. rehearse verifies, signs, bundles and checks with a throwaway key. * publish, the only job in the release environment, verifies the manifest digest reported by assemble, signs with gpg, uploads one bundle per version through the Central Portal API (USER_MANAGED, as before) and attaches the shaded jars to the GitHub Release. It runs the release scripts and tools from the runner image; no Maven, no JVM, nothing from the tagged commit. The workflow is started by workflow_dispatch only, and dry-run is the default. A release is dispatched from main for a tag whose commit is on main, so the workflow file and the scripts that run beside the secrets are the reviewed ones. Pushing a tag no longer starts a release; plan.sh still handles that event if the trigger is added back. Published coordinates, classifiers and file sets are unchanged. The Spark version in the artifact version is now read from pom.xml at the tag instead of being repeated in the build matrix, where it had fallen behind (3.5.8 in the matrix against spark.version 3.5.9). release-scripts.yml runs the offline tests for the scripts and for the shape of release.yml on pull requests, with the runner's gpg.
Changes that follow from an independent review of the split release workflow. None of them moves the boundary between the build jobs and the publish job. Deployments left in the Portal. The Portal accepts a drop only for a deployment that is VALIDATED or FAILED. A timeout or a cancellation during validation used to attempt the drop too early, and then forgot the deployment. central-upload.sh now polls a deployment until it settles (bounded) before dropping it, keeps what it could not drop so that the job's last step tries again, and, if something is still left, fails that step and lists each deployment by id and name in the job summary. An upload that was cut off before the Portal answered is reported by name. Deployments are named after the version, the run id and the attempt, and the summary lists their ids as they arrive, with the instruction to publish only those. Jars on an existing GitHub Release. An attached jar is left alone when it is identical to the staged one and is no longer replaced when it differs, unless the new replace-release-assets input is ticked. With skip-central nothing else stopped a rebuild from replacing published jars. Artifact names. The versions still follow spark.version in pom.xml at the tag, but plan now works them out before anything is built, shows them next to the Spark versions last published, and stops a publish run whose Spark versions differ from the last release (or cannot be compared) until the releaser has entered them in the new expected-spark-versions input. The build, assemble, rehearse and publish jobs are held to the versions the plan announced. No redirection in a real run. The Portal, repo1 and repository addresses and the local Maven repository are constants; the variables that used to override them are gone, and the tests use a stub curl and git URL rewriting instead. Three timing settings remain for the tests and stop a script when set without the test flag, which release.yml never sets. print-pin checks where the fork commits it prints can be reached from: it says so for the default branch, warns for another branch or tag, and refuses a commit that is on no branch or tag of the fork. Documentation. The README states the order in which to adopt the workflow (secrets into the environment, environment narrowed to main, dry run and a release candidate taken through Publish, then the repository-level secrets deleted) and why: the existing rc tags carry the previous release.yml with a workflow_dispatch trigger, and repository-level secrets are readable from any workflow on any ref. The rehearsal is rewritten as a real release candidate on a new tag, with what to check at each point. The workflow header no longer reads as a statement about the repository as a whole.
Automated review: FAILCommit Summary: Findings (6)
Produced from the diff alone by an automated reviewer. Text in code formatting is reviewer output shown as data. A pass is advisory: it does not replace CI or human review. |
"Remove the signing key" runs in an always() step. When an earlier step failed before GNUPGHOME was set, it failed as well and added a second red step to a failure that had nothing to do with it. signing-key.sh remove now says there is nothing to remove and succeeds when GNUPGHOME is unset or empty, or names a directory that does not exist. When a keyring exists, removal must leave it gone, or the step fails, so the no-op cannot hide a key that was imported and not removed. import and generate still require GNUPGHOME.
|
@schenksj, a triage of the automated review's findings on this PR (commit
The PR still needs your review, then a dry run from |
This changes how releases are made. The release workflow itself has not run on GitHub; its script tests have (455 checks on Linux with real gpg). It needs a review from whoever performs releases, then a dry run and a real release-candidate rehearsal before it is relied on.
Why
Today the
publishjob imports the signing key and holds the Maven Central credentials while it runs the whole project build twice (mvn package,mvn deploy), so every Maven plugin and dependency executes next to them. A pushedv*tag also runs that tag's own copy of the workflow, and the native library can be restored from a cache.What changes
Jobs:
plan→native→build(three Spark profiles) →assemble→rehearse→publish.publish, three legs)publish, one job).github/scripts/releaseat the workflow's commitpublishonlypublishchecks the digests and an allow-list of exactly the expected files before signing.gpg --detach-sign, one bundle per version, uploaded to the Central Portal API as a user-managed deployment. Nothing is public until someone presses Publish in the Portal; on failure or cancellation the job drops what it uploaded.workflow_dispatchfrommainonly, with the tag as an input; the tag must be an ancestor ofmain. The tag-push trigger is left as a commented block.publishjob, so it needs no approval and touches no secret.native-pins.txtrecords the tantivy4java commit for each version; the build stops if the tag points elsewhere or the lock file resolves the quickwit or tantivy forks to a different commit. The commits are written to the run summary and the release notes.replace-release-assetsis ticked; and a real publish whose Spark versions differ from the last published ones stops inplanuntilexpected-spark-versionsis given.pom.xmlandscripts/setup.share unchanged.Behaviour changes to confirm
pom.xmlat the tag instead of a hard-coded matrix.maincompiles against 3.5.9 / 4.0.4 / 4.1.3, so the next release's coordinates would be_spark_3.5.9and so on, where the old matrix would have labelled them 3.5.8 / 4.0.3 / 4.1.2.v*tag or creating a GitHub Release no longer starts anything. Dispatch the workflow frommainafterwards.-part is marked pre-release.Verified locally
v0.6.0-rc2for all three profiles through assemble, verify, sign (stubbed) and check: each bundle's 36 entry names equal the files on Maven Central, the poms are byte-identical to the published ones, and jar, sources and javadoc sizes match.Not verified
Before the first release with this
In this order (details in
.github/scripts/release/README.md, "Adopting this workflow"):releaseenvironment (done).releaseenvironment is narrowed tomainonly (it currently also admitsv*tags; administrator bypass is already off). Existing tagsv0.6.0-rc1andv0.6.0-rc2carry the old workflow, which can still be dispatched on the tag ref. Those two tags can be removed or left in place; narrowing the environment tomainalready stops them from reaching the secrets.mainwithtag=v0.6.0-rc2:publishis skipped with no approval request; each bundle lists 36 entries.v0.6.0-rc3), taken through Publish in the Central Portal, withexpected-spark-versions: 3.5.9 4.0.4 4.1.3. Publish only the deployment ids listed in the run summary.To abort: reject the approval, cancel during
publish(it drops its uploads), or Drop in the Portal. After Publish there is no abort.The runbook and trust model are in
.github/scripts/release/README.md.