diff --git a/.github/workflows/release-crates.yml b/.github/workflows/release-crates.yml new file mode 100644 index 000000000..e6bfb3304 --- /dev/null +++ b/.github/workflows/release-crates.yml @@ -0,0 +1,577 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 +# +# Publishes the workspace crates of one release to crates.io, at the +# workspace version (docs/adr/0029-crates-io-packaging.md). crates.io has no +# candidate stage and never replaces a version. So this workflow runs last in +# a release, after promote-image. packaging/README.md, "Releasing to +# crates.io", lists the release steps, the repository settings that the +# release depends on, and the steps for a new crate. +# +# This workflow publishes new versions of crates that crates.io has, through +# trusted publishing. Trusted publishing cannot create a crate. So a +# maintainer publishes each new crate with an API token and +# `devtools/publish-crates --new-crates`. That mode never publishes a crate +# that crates.io has. +# +# Shape, and why: +# +# gate Checks that need no build. The run must come from main. The +# input must be a strict `vX.Y.Z` tag that exists on origin. +# Its commit must be in origin/main. Its version must equal the +# workspace version at that commit. The promoted image of that +# version must carry the tagged commit in its revision label. +# The tagged commit must freeze the catalog migrations it ships. +# Then `devtools/publish-crates --dry-run` plans the run, and +# the plan goes to the job summary. The plan fails when a crate +# on crates.io accepts API tokens. A new crate does not fail the +# gate. The gate outputs the number of crates to publish before +# the first new crate, and the name of that new crate. +# +# verify Holds no credentials. `cargo package --workspace --locked` +# compiles every package from its tarball, and +# `devtools/check-crate-packages` checks the packages. The job +# uploads the SHA-256 of the package of each crate that +# crates.io receives, with the tagged commit and the toolchain. +# The build runs the build scripts and procedural macros of the +# dependency graph. That is why no token exists in this job. It +# also runs when the plan has a new crate. The maintainer's +# token run then checks out that commit, and it checks its bytes +# against this list. +# +# publish Runs when the plan has a crate to publish before the first +# new crate. A reviewer approves the `crates-io` environment +# first. The job packages the crates again without compiling +# them, and runs `devtools/check-crate-packages`. It stops +# unless each file has the SHA-256 that verify recorded. Then +# rust-lang/crates-io-auth-action exchanges the job's OIDC token +# for a crates.io token. That token expires after 30 minutes, +# and the action revokes it when the job ends. +# `devtools/publish-crates --no-verify` plans the run again, and +# publishes the crates in dependency order, up to the first new +# crate. After each upload, it compares the uploaded package +# with the list. The job outputs the first new crate of its own +# plan. A token run can publish the gate's new crate while the +# job waits for its approval. The job then does not stop. +# +# index Runs when no new crate stopped the run, so that the run +# published the whole release. The plan of the publish job +# decides that, and the gate's plan when the run skips publish. +# It reads each crate's file in the crates.io sparse index. It +# compares the checksum of the version with the SHA-256 that +# verify recorded. A difference fails the job. +# +# smoke Runs after index. It installs `extenddb` from crates.io with +# an empty cargo home, outside any checkout. It checks the +# version, the commit and the test-hooks lines that +# `extenddb version` prints. +# +# new-crate Runs when the run stopped before a new crate. It writes the +# stop and a pointer to the steps for an API token to the job +# summary, and fails the run. +# +# Every job that runs cargo uses the toolchain in RUST_TOOLCHAIN. For one +# commit and one toolchain, cargo writes the same bytes into each package. +# The publish and index jobs compare those bytes. +# +# Trigger: manual workflow_dispatch only. A run that a pushed tag starts uses +# the workflow file of the tagged commit. That file can be older than the +# reviewed file on main. A dispatch from main runs the current file against +# an existing tag. +# +# The release depends on two repository settings. packaging/README.md, +# "Repository settings for crates.io", gives the commands that read them +# back. A maintainer checks them before adding a trusted publisher. +# 1. The environment crates-io. It has the four required reviewers of the +# other release environments, with prevent self-review, so that the +# dispatcher cannot approve the run. It allows deployments from main +# only, and it holds no secrets. Admin bypass is off. crates.io never +# replaces a version, so even an administrator needs the approval of a +# second maintainer to publish. +# gh api repos/ExtendDB/extenddb/environments/crates-io +# 2. The tag ruleset protect-release-tags. It is active on refs/tags/v*, +# blocks the deletion and the update of those tags, and has no bypass +# actors. So a release tag cannot move to another commit. +# gh api repos/ExtendDB/extenddb/rulesets +# +# Each crate needs a GitHub trusted publisher on crates.io, on the Settings +# tab of the crate, after its first publish. Its fields are repository owner +# ExtendDB, repository name extenddb, workflow filename release-crates.yml +# and environment crates-io. The form labels the last field "Environment +# name (optional)". Fill it in, because a publisher without it gives a token +# to a run of this workflow from any branch. Then select "Require trusted +# publishing for all new versions". crates.io matches the workflow filename +# exactly. A rename of this file stops publishing until every crate's entry +# names the new file. + +name: release-crates + +on: + workflow_dispatch: + inputs: + tag: + description: 'Existing release tag to publish the crates of, e.g. v0.1.15' + required: true + +# No job commits anything, so no job needs to write to the repository. The +# publish job adds id-token at job level. +permissions: + contents: read + +# Two crates.io releases must never run at the same time. +concurrency: + group: release-crates-extenddb + cancel-in-progress: false + +env: + # The publish job compares its packages with the ones verify built. So + # every job that runs cargo uses this toolchain. A maintainer who + # publishes a new crate with an API token uses it too + # (packaging/README.md). Change it in a reviewed pull request of its own. + RUST_TOOLCHAIN: "1.97.0" + +jobs: + gate: + runs-on: ubuntu-latest + timeout-minutes: 15 + outputs: + version: ${{ steps.meta.outputs.version }} + sha: ${{ steps.meta.outputs.sha }} + publish_count: ${{ steps.plan.outputs.publish_count }} + first_new: ${{ steps.plan.outputs.first_new }} + steps: + - name: Gate - the run comes from main + # A dispatch from another branch runs that branch's copy of this + # file. The branch rule of the environment would stop it only at + # publish, after the verify build. + run: | + if [[ "$GITHUB_REF" != refs/heads/main ]]; then + echo "::error::dispatch release-crates from main, not from ${GITHUB_REF}" + exit 1 + fi + + - name: Check out main (the reviewed workflow's own ref) + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + with: + fetch-depth: 0 # tag resolution and the ancestry gate need history + + - name: Gate - the tag must identify released code + id: meta + env: + # Dispatch inputs are untrusted. They reach the shell only through + # the environment, never as text in the script. + RAW_TAG: ${{ inputs.tag }} + run: | + set -euo pipefail + + # 1. The strict shape comes first, so no other input reaches git + # as a ref expression. + if [[ ! "$RAW_TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + echo "::error::input '$RAW_TAG' is not a strict release tag (vMAJOR.MINOR.PATCH)" + exit 1 + fi + VERSION="${RAW_TAG#v}" + + # 2. The tag must exist. This workflow never creates tags. + git fetch --no-tags origin "refs/tags/${RAW_TAG}:refs/tags/${RAW_TAG}" \ + || { echo "::error::tag ${RAW_TAG} does not exist on origin"; exit 1; } + SHA=$(git rev-list -n 1 "refs/tags/${RAW_TAG}") + [[ "$SHA" =~ ^[0-9a-f]{40}$ ]] \ + || { echo "::error::could not resolve ${RAW_TAG} to a full commit SHA"; exit 1; } + + # 3. The tagged commit must be on main, so that a tag on an + # unmerged branch cannot publish code that was not reviewed. + git fetch --no-tags origin main + if ! git merge-base --is-ancestor "$SHA" origin/main; then + echo "::error::${RAW_TAG} (${SHA}) is not contained in origin/main" + exit 1 + fi + + # 4. The tag must equal the workspace version at that commit. + CARGO_VERSION=$(git show "${SHA}:Cargo.toml" \ + | sed -n 's/^version = "\(.*\)"$/\1/p' | head -1) + if [[ "$VERSION" != "$CARGO_VERSION" ]]; then + echo "::error::tag ${RAW_TAG} implies ${VERSION} but the workspace version at ${SHA} is ${CARGO_VERSION}" + exit 1 + fi + + { + echo "version=$VERSION" + echo "sha=$SHA" + } >> "$GITHUB_OUTPUT" + + - name: Gate - the promoted image holds the tagged commit + # promote-image runs before this workflow. The version tag of the + # default image must carry the tagged commit in its revision label. + # The read is anonymous, as in promote-image.yml. A tag that moved + # to another commit fails here, because the image keeps its commit. + env: + VERSION: ${{ steps.meta.outputs.version }} + SHA: ${{ steps.meta.outputs.sha }} + run: | + set -euo pipefail + command -v skopeo >/dev/null || sudo apt-get install -y --no-install-recommends skopeo + IMAGE="docker://docker.io/extenddb/extenddb-postgres:${VERSION}" + if ! INSPECT=$(skopeo inspect --no-creds "$IMAGE"); then + echo "::error::cannot read ${IMAGE}. Run promote-image for ${VERSION} first." + exit 1 + fi + REVISION=$(jq -r '.Labels["org.opencontainers.image.revision"] // ""' <<< "$INSPECT") + if [[ "$REVISION" != "$SHA" ]]; then + echo "::error::${IMAGE} has the revision label '${REVISION}', not the tagged commit ${SHA}" + exit 1 + fi + echo "${IMAGE} has the revision ${REVISION}" + + - name: Check out the tagged commit + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + with: + ref: ${{ steps.meta.outputs.sha }} + persist-credentials: false + + - name: Gate - the tagged commit has the crates.io release scripts + run: | + set -euo pipefail + for script in devtools/check-crate-packages devtools/publish-crates; do + if [[ ! -x "$script" ]]; then + echo "::error::the tagged commit has no ${script}. Use a tag created after release-crates.yml was merged." + exit 1 + fi + done + + - name: Gate - the release freezes the catalog migrations it ships + # The crates carry the catalog migrations of every backend, as the + # images and the npm packages do. So RELEASED_THROUGH must reach the + # last migration id. + run: python3 devtools/check-released-through + + - uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master + with: + toolchain: ${{ env.RUST_TOOLCHAIN }} + + - name: Gate - plan the run + id: plan + # Reads the crates.io sparse index and the crates.io API. The plan + # shows the stop before a new crate. The script writes publish_count + # and first_new to the step outputs. + env: + VERSION: ${{ steps.meta.outputs.version }} + SHA: ${{ steps.meta.outputs.sha }} + run: | + set -euo pipefail + devtools/publish-crates --dry-run --version "$VERSION" --plan-output "$GITHUB_OUTPUT" \ + | tee "$RUNNER_TEMP/plan.txt" + { + printf '### crates.io release v%s\n\n' "$VERSION" + printf '| field | value |\n|---|---|\n' + printf "| tag | \`v%s\` |\n" "$VERSION" + printf "| commit | \`%s\` |\n\n" "$SHA" + printf "\`\`\`\n" + cat "$RUNNER_TEMP/plan.txt" + printf "\`\`\`\n" + } >> "$GITHUB_STEP_SUMMARY" + + verify: + needs: gate + runs-on: ubuntu-latest + timeout-minutes: 60 + permissions: + contents: read + steps: + - name: Check out the tagged commit + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + with: + ref: ${{ needs.gate.outputs.sha }} + # The build runs third-party build scripts, so the repository + # token stays out of .git/config. + persist-credentials: false + + - uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master + with: + toolchain: ${{ env.RUST_TOOLCHAIN }} + + - name: Build every package from its tarball + run: cargo package --workspace --locked + + - name: Check the packages against the crates.io rules + run: devtools/check-crate-packages + + - name: Record the SHA-256 of each crate's package, the commit and the toolchain + # SHA256SUMS has a line for the package of each crate that crates.io + # receives. cargo also packages a member that crates.io does not + # receive. The list leaves that file out, so that index compares only + # the crates of the release. COMMIT and TOOLCHAIN are for the + # maintainer's token run (packaging/README.md, "Publishing new crates + # with an API token"). That run checks out the tag only when the + # tag's commit is COMMIT, and it packages with TOOLCHAIN. + env: + VERSION: ${{ needs.gate.outputs.version }} + SHA: ${{ needs.gate.outputs.sha }} + run: | + set -euo pipefail + mkdir -p "$RUNNER_TEMP/crate-sha256" + devtools/publish-crates --list-packages --version "$VERSION" > "$RUNNER_TEMP/packages.txt" + mapfile -t packages < "$RUNNER_TEMP/packages.txt" + if [[ ${#packages[@]} -eq 0 ]]; then + echo "::error::devtools/publish-crates lists no package" + exit 1 + fi + printf '%s\n' "$SHA" > "$RUNNER_TEMP/crate-sha256/COMMIT" + printf '%s\n' "$RUST_TOOLCHAIN" > "$RUNNER_TEMP/crate-sha256/TOOLCHAIN" + cd target/package + sha256sum -- "${packages[@]}" | tee "$RUNNER_TEMP/crate-sha256/SHA256SUMS" + + - name: Upload the checksums, the commit and the toolchain + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: crate-sha256 + # The three files of the directory, at the root of the artifact. + path: ${{ runner.temp }}/crate-sha256/ + if-no-files-found: error + retention-days: 7 # a failed publish job can be re-run for a week + + publish: + needs: [gate, verify] + # A run with nothing to publish before the first new crate mints no + # token. + if: needs.gate.outputs.publish_count != '0' + runs-on: ubuntu-latest + timeout-minutes: 45 + # Required reviewers with prevent self-review, no admin bypass, and the + # main-only branch rule. Each crate's trusted publisher names this + # environment. So a run outside it cannot get a crates.io token. + environment: crates-io + permissions: + contents: read + id-token: write # the OIDC token that crates.io exchanges for a publish token + outputs: + # The first new crate of this job's own plan, or empty. The script + # reads crates.io again before the first upload, so index and + # new-crate follow this plan and not the gate's. + first_new: ${{ steps.run.outputs.first_new }} + steps: + - name: Check out the tagged commit + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + with: + ref: ${{ needs.gate.outputs.sha }} + persist-credentials: false + + - uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master + with: + toolchain: ${{ env.RUST_TOOLCHAIN }} + + - name: Download the verified checksums + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 + with: + name: crate-sha256 + path: ${{ runner.temp }}/crate-sha256 + + - name: Package without compiling, check, and compare with the verified packages + # `cargo package --no-verify` runs no build script, so the checker + # runs here on files that no dependency code touched. A package with + # other bytes than the one verify compiled stops the release before + # the first upload. + run: | + set -euo pipefail + cargo package --workspace --no-verify --locked + devtools/check-crate-packages + cd target/package + sha256sum --check --strict "$RUNNER_TEMP/crate-sha256/SHA256SUMS" + + - name: Get a crates.io token through trusted publishing + id: auth + uses: rust-lang/crates-io-auth-action@3268eb413133e2960fa4ae0e963ca76380c54cd9 # v1.0.6 + + - name: Publish the crates in dependency order + id: run + env: + CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }} + VERSION: ${{ needs.gate.outputs.version }} + FIRST_NEW: ${{ needs.gate.outputs.first_new }} + # crates.io does not rate-limit trusted publishing per user. Its one + # 429 here is the limit on the versions of one crate in 24 hours, + # and the script stops at it. The cap of 120 s bounds the waits for + # the index and after any other 429. The token expires 30 minutes + # after the step above. + run: | + set -euo pipefail + args=(--no-verify --max-wait 120 --version "$VERSION" + --sha256sums "$RUNNER_TEMP/crate-sha256/SHA256SUMS" + --plan-output "$GITHUB_OUTPUT") + # The gate found a new crate. The run publishes the crates before + # it and stops there, and the new-crate job reports the stop. When + # a token run published that crate in the meantime, the run does + # not stop, and index and smoke run instead. + if [[ -n "$FIRST_NEW" ]]; then + args+=(--expect-new "$FIRST_NEW") + fi + devtools/publish-crates "${args[@]}" + + index: + needs: [gate, verify, publish] + # Only when no new crate stopped the run, so that the run published the + # whole release. The plan of the publish job decides when publish ran. + # The run skips publish when the gate's plan has nothing to publish + # before the first new crate, and the gate's plan decides then. + if: >- + ${{ !cancelled() + && needs.verify.result == 'success' + && ((needs.publish.result == 'success' && needs.publish.outputs.first_new == '') + || (needs.publish.result == 'skipped' && needs.gate.outputs.first_new == '')) }} + runs-on: ubuntu-latest + timeout-minutes: 20 + permissions: {} + steps: + - name: Download the verified checksums + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 + with: + name: crate-sha256 + path: ${{ runner.temp }}/crate-sha256 + + - name: Compare the checksums in the crates.io index with the verified packages + env: + VERSION: ${{ needs.gate.outputs.version }} + run: | + set -euo pipefail + # The path of a crate's file in the sparse index (the Cargo book, + # "Registry Index"). devtools/publish-crates uses the same rule. + index_path() { + local name="${1,,}" + case "${#name}" in + 1|2) echo "${#name}/${name}" ;; + 3) echo "3/${name:0:1}/${name}" ;; + *) echo "${name:0:2}/${name:2:2}/${name}" ;; + esac + } + failed=0 + while read -r sum file; do + name="${file%-"${VERSION}".crate}" + if [[ "$name" == "$file" ]]; then + echo "::error::${file} is not a package of version ${VERSION}" + failed=1 + continue + fi + url="https://index.crates.io/$(index_path "$name")" + # cargo publish waits up to 60 s for each version. A cache in + # front of the index can serve an older file for a short time. + cksum="" + for attempt in $(seq 1 20); do + cksum=$(curl -fsS --max-time 30 "$url" | jq -r --arg v "$VERSION" 'select(.vers == $v) | .cksum' || true) + if [[ -n "$cksum" ]]; then + break + fi + echo "${name} ${VERSION} is not in the index yet (attempt ${attempt} of 20)" + sleep 15 + done + if [[ -z "$cksum" ]]; then + echo "::error::${name} ${VERSION} is not in the crates.io index. See packaging/README.md, \"A failure after the upload\"." + failed=1 + elif [[ "$cksum" != "$sum" ]]; then + echo "::error::${name} ${VERSION}: crates.io has cksum ${cksum}, and the verified package has SHA-256 ${sum}. The version is public and permanent. See packaging/README.md, \"A failure after the upload\"." + failed=1 + else + echo "${name} ${VERSION}: ${cksum}" + fi + done < "$RUNNER_TEMP/crate-sha256/SHA256SUMS" + exit "$failed" + + smoke: + needs: [gate, index] + if: ${{ !cancelled() && needs.index.result == 'success' }} + runs-on: ubuntu-latest + timeout-minutes: 45 + permissions: {} + steps: + - uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master + with: + toolchain: ${{ env.RUST_TOOLCHAIN }} + + - name: Install extenddb from crates.io with an empty cargo home + env: + VERSION: ${{ needs.gate.outputs.version }} + SHA: ${{ needs.gate.outputs.sha }} + run: | + set -euo pipefail + # The job checks out nothing, and it works in a new directory under + # /tmp. So no cargo configuration of a repository or of the + # runner's home directory applies. The new CARGO_HOME holds no + # registry cache, configuration or token. + CARGO_HOME=$(mktemp -d) + export CARGO_HOME + cd "$(mktemp -d)" + # The index job saw every version. Try again only while cargo + # cannot resolve a version, never after a build error. + installed=false + for attempt in $(seq 1 10); do + if cargo install extenddb --version "$VERSION" --locked 2>&1 | tee install.log; then + installed=true + break + fi + if ! grep -qE 'could not find .extenddb. in registry|failed to select a version|no matching package named' install.log; then + break + fi + echo "crates.io does not resolve extenddb ${VERSION} yet (attempt ${attempt} of 10)" + sleep 30 + done + if [[ "$installed" != true ]]; then + echo "::error::cargo install extenddb --version ${VERSION} --locked failed. Rerun this job first. See packaging/README.md, \"A failure after the upload\"." + exit 1 + fi + + "$CARGO_HOME/bin/extenddb" version | tee version.txt + # The binary reads the commit from the .cargo_vcs_info.json that + # cargo publish wrote. So this line shows the tag's commit only + # when cargo packaged the crate from a clean checkout of the tag. + after="See packaging/README.md, \"A failure after the upload\"." + grep -qxF "extenddb ${VERSION}" version.txt \ + || { echo "::error::extenddb version does not report ${VERSION}. ${after}"; exit 1; } + grep -qxF "commit ${SHA}" version.txt \ + || { echo "::error::extenddb version does not report commit ${SHA}. ${after}"; exit 1; } + # The test-hooks lines that `extenddb version` prints: the MongoDB + # test hooks are disabled, and no backup-test-hooks line appears. + # A test-hooks feature of a storage crate adds no line. + grep -qxF "test-hooks disabled" version.txt \ + || { echo "::error::extenddb version does not report test-hooks disabled. ${after}"; exit 1; } + if grep -qF "backup-test-hooks enabled" version.txt; then + echo "::error::extenddb version reports backup-test-hooks enabled. ${after}" + exit 1 + fi + + new-crate: + needs: [gate, verify, publish] + # Only when the run stopped before a new crate. Either the plan of the + # publish job has a new crate, or the run skipped publish and the gate's + # plan has one. The publish job exits 0 at that stop only when the stop + # is at the gate's new crate, so the gate's outputs describe the stop. + if: >- + ${{ !cancelled() + && needs.verify.result == 'success' + && ((needs.publish.result == 'success' && needs.publish.outputs.first_new != '') + || (needs.publish.result == 'skipped' && needs.gate.outputs.first_new != '')) }} + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: {} + steps: + - name: Report the stop before the new crate + env: + VERSION: ${{ needs.gate.outputs.version }} + FIRST_NEW: ${{ needs.gate.outputs.first_new }} + PUBLISH_COUNT: ${{ needs.gate.outputs.publish_count }} + RUN_ID: ${{ github.run_id }} + run: | + set -euo pipefail + crates="crates" + if [[ "$PUBLISH_COUNT" == 1 ]]; then + crates="crate" + fi + { + printf '### Stopped before %s\n\n' "$FIRST_NEW" + printf "This run published %s %s of v%s, and then stopped before \`%s\`.\n" \ + "$PUBLISH_COUNT" "$crates" "$VERSION" "$FIRST_NEW" + printf 'crates.io has no crate of this name, and trusted publishing cannot create a crate.\n' + printf "A maintainer publishes it with an API token and the \`crate-sha256\` artifact of this run,\n" + printf "run ID %s. packaging/README.md, \"Publishing new crates with an API token\", gives the steps.\n" \ + "$RUN_ID" + } >> "$GITHUB_STEP_SUMMARY" + echo "::error::crates.io has no crate named ${FIRST_NEW}. The job summary points at the steps for an API token." + exit 1 diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index b951c035e..5e25601b0 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -111,9 +111,9 @@ jobs: # --no-verify writes the packages without compiling them, in seconds, and # runs no build script. devtools/check-crate-packages then checks the pins, # the LICENSE and NOTICE files, the metadata and crates.io's size limits. - # Compiling each package from its tarball takes minutes. That build belongs - # to the crates.io release workflow (follow-up work), before the upload, and - # not to each PR. + # Compiling each package from its tarball takes minutes. That build runs in + # the verify job of release-crates.yml, before the upload, and not on each + # PR. package: runs-on: extenddb_ubuntu-2404_4-core timeout-minutes: 10 diff --git a/devtools/check-crate-packages b/devtools/check-crate-packages index 8eae2ed5a..52442bb35 100755 --- a/devtools/check-crate-packages +++ b/devtools/check-crate-packages @@ -12,8 +12,13 @@ changes, it also needs `--allow-dirty`. This script needs Python 3.11 or later. It reads the members from `cargo metadata`, so cargo resolves globs, `exclude`, inherited fields and renamed dependencies. It then fails, with one line per problem, unless all of the following hold for each member that -crates.io receives (one without `publish = false`): +crates.io receives (one without a `publish` field, or one whose `publish` +list names crates-io): +- The member's version is the `version` of the root manifest's + `[workspace.package]` table. A release publishes every crate at the + workspace version, and `devtools/publish-crates --version` refuses a + crate of another version. - Each dependency on another workspace crate, normal or build, requires that crate's version exactly (`=X.Y.Z`). A caret requirement would let a fresh install combine crates of two releases. A pin left at the previous @@ -91,8 +96,19 @@ def cargo_metadata(root: Path) -> dict: def is_published(package: dict) -> bool: - """`publish = false` reaches cargo metadata as an empty registry list.""" - return package.get("publish") != [] + """A member that crates.io receives. `publish = false` reaches cargo + metadata as an empty registry list, and a member without the field as + null. devtools/publish-crates uses the same rule.""" + registries = package.get("publish") + return registries is None or "crates-io" in registries + + +def workspace_version(metadata: dict) -> str | None: + """The `version` of the root manifest's [workspace.package] table, or + None when the root manifest sets none.""" + manifest = tomllib.loads((Path(metadata["workspace_root"]) / "Cargo.toml").read_text()) + version = manifest.get("workspace", {}).get("package", {}).get("version") + return version if isinstance(version, str) else None def check(root: Path, packages: Path | None = None) -> list[str]: @@ -103,11 +119,19 @@ def check(root: Path, packages: Path | None = None) -> list[str]: if packages is None: packages = Path(metadata["target_directory"]) / "package" problems: list[str] = [] + release = workspace_version(metadata) + if release is None: + problems.append("the root Cargo.toml sets no [workspace.package] version, so the release version is unknown") for name, package in sorted(members.items()): if not is_published(package): continue manifest = os.path.relpath(package["manifest_path"], root) + if release is not None and package["version"] != release: + problems.append( + f"{manifest}: {name} is version {package['version']}, and the workspace version is {release}. " + f"A release publishes every crate at the workspace version (`version.workspace = true`)." + ) for dependency in package["dependencies"]: target = dependency["name"] if target not in members: diff --git a/devtools/check_crate_packages_test.py b/devtools/check_crate_packages_test.py index 4e12c23ff..a618fc74d 100644 --- a/devtools/check_crate_packages_test.py +++ b/devtools/check_crate_packages_test.py @@ -207,14 +207,36 @@ def test_an_excluded_directory_is_not_a_member(self) -> None: self.write("crates/x/src/lib.rs", "") self.assert_passes() - def test_a_member_with_its_own_version_is_packaged_and_pinned_at_it(self) -> None: + def test_a_member_with_its_own_version_fails_and_its_pins_are_checked_at_it(self) -> None: self.write("Cargo.toml", ROOT_MANIFEST.replace(f'"crates/a", version = "={VERSION}"', '"crates/a", version = "=2.0.0"')) self.member("a", MEMBER_A.replace("version.workspace = true", 'version = "2.0.0"')) (self.packages / f"example-a-{VERSION}.crate").unlink() self.package("example-a", {"Cargo.toml": packaged_manifest("example-a", version="2.0.0")}, version="2.0.0") self.package("example-b", {"Cargo.toml": packaged_manifest("example-b", '\n[dependencies.example-a]\nversion = "=2.0.0"\n')}) + code, output = self.run_check() + self.assertEqual(code, 1, output) + self.assertIn(f"error: crates/a/Cargo.toml: example-a is version 2.0.0, and the workspace version is {VERSION}.", + output) + self.assertIn("1 problem(s) found", output) + + def test_a_root_manifest_without_a_workspace_version_fails(self) -> None: + self.write("Cargo.toml", ROOT_MANIFEST.replace(f'version = "{VERSION}"\nedition', "edition")) + for directory, manifest in (("a", MEMBER_A), ("b", MEMBER_B)): + self.member(directory, manifest.replace("version.workspace = true", f'version = "{VERSION}"')) + self.assert_fails_with("error: the root Cargo.toml sets no [workspace.package] version") + + def test_a_member_published_only_to_another_registry_needs_no_package(self) -> None: + self.member("b", MEMBER_B.replace("edition.workspace = true", 'edition.workspace = true\npublish = ["my-registry"]') + .replace("{ workspace = true }", '{ path = "../a" }')) + (self.packages / f"example-b-{VERSION}.crate").unlink() self.assert_passes() + def test_a_member_whose_publish_list_names_crates_io_is_checked(self) -> None: + self.member("b", MEMBER_B.replace("edition.workspace = true", + 'edition.workspace = true\npublish = ["my-registry", "crates-io"]')) + (self.packages / f"example-b-{VERSION}.crate").unlink() + self.assert_fails_with(f"example-b-{VERSION}.crate: missing") + def test_cargo_target_dir_moves_the_packages(self) -> None: moved = self.root / "elsewhere" for crate in self.packages.iterdir(): diff --git a/devtools/publish-crates b/devtools/publish-crates new file mode 100755 index 000000000..c9b636bf9 --- /dev/null +++ b/devtools/publish-crates @@ -0,0 +1,682 @@ +#!/usr/bin/env python3 +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 +"""Publish the workspace crates to crates.io, one at a time, in dependency order. + + devtools/publish-crates [--new-crates] [--dry-run] [--no-verify] + [--version X.Y.Z] [--sha256sums FILE] + [--expect-new NAME] [--plan-output FILE] + [--max-wait SECONDS] [--root DIR] + devtools/publish-crates --list-packages [--version X.Y.Z] [--root DIR] + +This script needs Python 3.11 or later. It reads the members from +`cargo metadata`. It takes each member that crates.io receives: a member +without a `publish` field, or one whose `publish` list names crates-io. It +orders them so that each crate comes after every workspace crate it depends +on through a normal or build dependency. A cycle stops the run. With +--version, a crate of another version also stops the run. + +The script then reads each crate's file in the crates.io sparse index, and +it plans one of three actions for each crate: + +- skip: the index has the crate's version. crates.io never replaces a + version, so a run that stopped part way continues where it stopped. +- publish: the index has the crate, but not this version. +- new: the index has no file for the crate. + +The script has two modes. The mode --new-crates publishes the first version +of each crate, and the other mode publishes each later version: + +- Without --new-crates, the script publishes the crates of the action + publish. The release-crates workflow runs it with a trusted publishing + token. It stops before the first new crate, because trusted publishing + cannot create a crate. This mode also reads each crate that crates.io has + from the crates.io API. It stops when one of them accepts API tokens. +- With --new-crates, the script publishes the new crates. A maintainer runs + it with an API token. It stops before the first crate of the action + publish. crates.io refuses a new version published with an API token for + a crate that requires trusted publishing, and the release-crates workflow + publishes that crate. + +A run that stops prints the next steps and exits 1. A first publish has only +new crates, so the mode --new-crates publishes all of them. A later release +that adds a crate alternates between the modes. packaging/README.md, +"Releasing to crates.io", describes the steps. + +The script prints the plan and the crate where the run stops. --dry-run +stops there and exits 0. --plan-output FILE appends two lines to FILE, in +the format of GITHUB_OUTPUT. publish_count is the number of crates of the +action publish before the first new crate. first_new is the name of the +first new crate, or empty when the plan has no new crate. With +--expect-new NAME, a stop before the new crate NAME exits 0. The workflow +passes the name that its gate found, and a later job reports the stop. A +token run can publish NAME in the meantime. The run then does not stop, and +it prints one line that says so. NAME must be a crate of the release. + +Otherwise the script runs `cargo publish -p --locked` for each crate +to publish, in order. It adds --no-verify when the caller passes it. Cargo +waits up to 60 seconds until the index has the new version, so the next +crate can resolve it. When that wait times out, cargo warns and exits 0. + +Pass --no-verify only after `cargo package --workspace --locked` built +every package from its tarball without the token. A verifying publish runs +the build scripts and procedural macros of the whole dependency graph while +the token is in the environment. + +--sha256sums FILE takes the SHA-256 list of the verified packages, in the +format of sha256sum. The list must have a line for each crate that the run +publishes or skips, or the run stops before the first upload. The run also +stops there when the index lists the version of a skipped crate with +another checksum. `cargo publish` writes the package that it uploads into +`package/tmp-crate` of cargo's build directory, which `cargo metadata` +names. Only `cargo package` copies the file on into the target directory. +Before each upload, the script deletes the crate's file in `tmp-crate`. +After the upload, it hashes the file that cargo wrote there, and it stops +when the hash differs from the list. --list-packages prints the package +file name of each crate, in publish order, and reads no index. The +release-crates workflow hashes those files. + +crates.io limits how fast one account publishes (src/rate_limiter.rs). Its +defaults allow 5 new crates at once and then one every 10 minutes. They +allow 30 new versions at once and then one a minute. On an answer with +status 429, the script waits until the time in the answer, plus 5 seconds. +Then it runs the publish again, and it prints each wait. crates.io also +limits the versions of one crate in 24 hours, and the script stops at that +answer. When cargo cannot find a crate that this run published, the index +lags, and the script runs the publish again after 30 seconds. --max-wait +caps the total of these waits, not the time of the uploads. The script +stops when the next wait would take the total past the cap. The default of +2 hours covers the first publish of 14 new crates, which takes about 90 +minutes. + +cargo reads the token from CARGO_REGISTRY_TOKEN or from `cargo login`. The +script never prints the token or the environment. + +Exit 0 when the run published its crates, for a dry run, and for +--list-packages. Exit 1 when the run stops before a crate that --expect-new +does not name, and on any error. Exit 2 on a usage error. +""" + +from __future__ import annotations + +import argparse +import email.utils +import hashlib +import http.client +import json +import math +import os +import re +import subprocess +import sys +import time +import urllib.error +import urllib.request +from datetime import datetime +from pathlib import Path + +try: + from datetime import UTC +except ImportError: # Python 3.10 and older + sys.exit("error: devtools/publish-crates needs Python 3.11 or later (for datetime.UTC)") + +INDEX = "https://index.crates.io/" +API = "https://crates.io/api/v1/crates/" +USER_AGENT = "extenddb-publish-crates (https://github.com/ExtendDB/extenddb)" +# The Cargo reference, "Registry Index", "Nonexistent Crates": a registry +# answers 404, 410 or 451 for a crate that it does not have. +NOT_FOUND = (404, 410, 451) +TIMEOUT = 30 +# crates.io's data access policy allows at most one API request a second. +API_INTERVAL = 1 +REGISTRY = "crates-io" +DEFAULT_MAX_WAIT = 2 * 60 * 60 +RETRY_MARGIN = 5 +# The wait after a 429 answer that gives no time. +DEFAULT_RETRY_WAIT = 60 +# The wait before the next try when the index does not list a crate that +# this run published. +INDEX_RETRY_WAIT = 30 + +# cargo reports an answer that holds crates.io's JSON errors as "the remote +# server responded with an error (status 429 Too Many Requests): ", +# and any other answer as "failed to get a 200 OK response, got 429", with +# the headers and the body after it. +TOO_MANY_REQUESTS = re.compile(r"\(status 429\b|\bgot 429\b") +# crates.io's detail says "Please try again after Tue, 10 Nov 2026 14:30:00 +# GMT", and its Retry-After header holds the same date +# (src/util/errors/json.rs). +RETRY_AT = re.compile( + r"(?:try again after|retry-after:)\s*(\w{3}, \d{1,2} \w{3} \d{4} \d{2}:\d{2}:\d{2} GMT)", + re.IGNORECASE, +) +RETRY_SECONDS = re.compile(r"^\s*retry-after:\s*(\d+)\s*$", re.IGNORECASE | re.MULTILINE) +# crates.io's limit on the versions of one crate (src/controllers/krate/publish.rs). +DAILY_LIMIT = re.compile(r"too many versions of this crate in the last 24 hours") +# cargo's resolver errors when the index does not list a dependency yet: a +# crate that it does not know, or a version that it does not have. +MISSING_DEPENDENCY = re.compile( + r"no matching package named `([^`]+)` found" + r"|failed to select a version for the requirement `([^`\s]+) = " +) +# A line of sha256sum output for a file name without blanks, which every +# package file name is. +SHA256_LINE = re.compile(r"([0-9a-f]{64}) [ *](\S+)") + +SKIP = "skip" +PUBLISH = "publish" +NEW = "new" +STATES = {SKIP: "on crates.io", PUBLISH: "new version", NEW: "new crate"} + +TOKEN_STEPS = 'packaging/README.md, "Publishing new crates with an API token"' +AFTER_UPLOAD = 'packaging/README.md, "A failure after the upload"' + +FIRST_PUBLISH = """ +crates.io has no crate named {name}, and trusted publishing cannot create a +crate. A maintainer publishes {name} with an API token of the scope +publish-new, and runs devtools/publish-crates --new-crates for it. Then the +maintainer gives {name} its owners and its trusted publisher, requires +trusted publishing for it, and dispatches release-crates again. +{steps} gives the steps and the commands. +""" + +TOKEN_STOP = """ +crates.io has {name}, and an API token must not publish it. crates.io +refuses a new version published with an API token for a crate that +requires trusted publishing. Revoke the token now, and finish the steps for +each new crate in {steps}. Then dispatch release-crates again with the tag, +to publish {name} and the crates after it. +""" + +TOKEN_DONE = """ +Revoke the token now, and finish the steps for each new crate in {steps}. +Then dispatch release-crates again with the tag, so that its index and smoke +jobs run on the whole release. +""" + + +class Failure(Exception): + """A problem that stops the run, reported on one line.""" + + +def cargo() -> str: + return os.environ.get("CARGO", "cargo") + + +def now() -> datetime: + return datetime.now(UTC) + + +def sleep(seconds: int) -> None: + time.sleep(seconds) + + +def api_pause() -> None: + time.sleep(API_INTERVAL) + + +def count(number: int) -> str: + return f"{number} crate" if number == 1 else f"{number} crates" + + +def package_file(name: str, version: str) -> str: + """The file name of a crate's package, as cargo writes it.""" + return f"{name}-{version}.crate" + + +def cargo_metadata(root: Path) -> dict: + """The workspace as cargo resolves it, without the dependency graph.""" + try: + result = subprocess.run( + [cargo(), "metadata", "--no-deps", "--format-version", "1"], + cwd=root, capture_output=True, text=True, check=False, + ) + except OSError as error: + raise Failure(f"cannot run `{cargo()} metadata` in {root}: {error}") from error + if result.returncode != 0: + raise Failure(f"`cargo metadata` failed in {root}:\n{result.stderr}") + return json.loads(result.stdout) + + +def is_published(package: dict) -> bool: + """A member that crates.io receives. `publish = false` reaches cargo + metadata as an empty list, and a member without the field as null.""" + registries = package.get("publish") + return registries is None or REGISTRY in registries + + +def publish_order(metadata: dict) -> list[dict]: + """The members crates.io receives, each after every workspace crate it + depends on through a normal or build dependency. Crates that are ready + at the same point go in name order.""" + members = {package["name"]: package for package in metadata["packages"]} + published = {name: package for name, package in members.items() if is_published(package)} + waits_for: dict[str, set[str]] = {} + for name, package in published.items(): + waits_for[name] = set() + for dependency in package["dependencies"]: + target = dependency["name"] + if dependency.get("kind") == "dev" or target not in members: + continue + if target not in published: + raise Failure(f"{name} depends on {target}, which crates.io does not receive, so crates.io cannot resolve it") + waits_for[name].add(target) + order: list[dict] = [] + while waits_for: + ready = sorted(name for name, targets in waits_for.items() if not targets) + if not ready: + raise Failure("the workspace crates depend on each other in a cycle: " + " -> ".join(cycle(waits_for))) + order.append(published[ready[0]]) + del waits_for[ready[0]] + for targets in waits_for.values(): + targets.discard(ready[0]) + return order + + +def cycle(waits_for: dict[str, set[str]]) -> list[str]: + """One cycle among crates that all wait for another one of them, as a + path that starts and ends at the same crate.""" + path = [min(waits_for)] + while path.count(path[-1]) < 2: + path.append(min(waits_for[path[-1]])) + return path[path.index(path[-1]):] + + +def check_versions(order: list[dict], version: str | None) -> None: + """Stop when a crate to publish has another version than the release.""" + if version is None: + return + other = [f"{package['name']} {package['version']}" for package in order if package["version"] != version] + if other: + raise Failure(f"the release is {version}, and these crates have another version: {', '.join(other)}") + + +def index_path(name: str) -> str: + """The path of a crate's file in the sparse index (the Cargo book, + "Registry Index", "Index files").""" + name = name.lower() + if len(name) <= 2: + return f"{len(name)}/{name}" + if len(name) == 3: + return f"3/{name[0]}/{name}" + return f"{name[:2]}/{name[2:4]}/{name}" + + +def fetch(url: str) -> bytes | None: + """The body at `url`, or None when the answer says that the crate does + not exist.""" + request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT}) + try: + with urllib.request.urlopen(request, timeout=TIMEOUT) as response: + return response.read() + except urllib.error.HTTPError as error: + if error.code in NOT_FOUND: + return None + raise Failure(f"{url} answered {error.code} {error.reason}") from error + except (OSError, http.client.HTTPException) as error: + raise Failure(f"cannot read {url}: {error!r}") from error + + +def index_versions(name: str) -> dict[str, str] | None: + """The versions of a crate in the crates.io sparse index, yanked ones + included, each with the cksum that the index lists for it. None when + crates.io has no crate of that name. The cksum of a version is the + SHA-256 of the package that crates.io serves.""" + url = INDEX + index_path(name) + body = fetch(url) + if body is None: + return None + try: + entries = [json.loads(line) for line in body.decode().splitlines() if line.strip()] + return {entry["vers"]: entry["cksum"] for entry in entries} + except (ValueError, KeyError, TypeError) as error: + raise Failure(f"{url} holds a line that is not an index entry: {error!r}") from error + + +def trusted_publishing_only(name: str) -> bool: + """crates.io's trustpub_only flag of a crate. When it is true, crates.io + refuses a new version published with an API token.""" + url = API + name + api_pause() + body = fetch(url) + if body is None: + raise Failure(f"{url} has no crate {name}, but the crates.io index lists it") + try: + flag = json.loads(body)["crate"]["trustpub_only"] + except (ValueError, KeyError, TypeError) as error: + raise Failure(f"{url} sent no crate.trustpub_only: {error!r}") from error + return flag is True + + +def plan(order: list[dict], check_trusted_publishing: bool) -> tuple[list[tuple[str, str, str]], dict[str, str]]: + """(name, version, action) for each crate, in publish order, and the + index cksum of each skipped package by its file name. With + `check_trusted_publishing`, stop when a crate that crates.io has accepts + API tokens.""" + steps = [] + skipped = {} + open_to_tokens = [] + for package in order: + name, version = package["name"], package["version"] + versions = index_versions(name) + if versions is None: + action = NEW + else: + action = SKIP if version in versions else PUBLISH + if action == SKIP: + skipped[package_file(name, version)] = versions[version] + if check_trusted_publishing and not trusted_publishing_only(name): + open_to_tokens.append(name) + steps.append((name, version, action)) + if open_to_tokens: + raise Failure( + f"crates.io accepts API tokens for {', '.join(open_to_tokens)}. On the Settings tab of each, select " + f'"Require trusted publishing for all new versions" ({TOKEN_STEPS}).' + ) + return steps, skipped + + +def stop_at(steps: list[tuple[str, str, str]], new_crates: bool) -> int | None: + """The position of the crate where the run stops, or None. Without + --new-crates the run publishes the crates of the action publish, and + with it the new crates. The first crate of the other action stops it.""" + wanted = NEW if new_crates else PUBLISH + for position, (_, _, action) in enumerate(steps): + if action not in (SKIP, wanted): + return position + return None + + +def plan_facts(steps: list[tuple[str, str, str]]) -> tuple[int, str]: + """The number of crates of the action publish before the first new + crate, and the name of that crate ("" when there is none).""" + before = 0 + for name, _, action in steps: + if action == NEW: + return before, name + if action == PUBLISH: + before += 1 + return before, "" + + +def print_plan(steps: list[tuple[str, str, str]], stop: int | None, new_crates: bool) -> None: + mode = "with an API token, new crates only" if new_crates else "with trusted publishing" + print(f"Publish order, {count(len(steps))}, {mode}:") + for position, (name, version, action) in enumerate(steps): + if stop is not None and position > stop: + verdict = "after the stop" + elif position == stop: + verdict = "stop here" + elif action == SKIP: + verdict = "skip" + else: + verdict = "publish" + print(f"{position + 1:3}. {name} {version}: {STATES[action]}, {verdict}") + planned = sum(1 for _, _, action in steps[:stop] if action != SKIP) + if stop is None: + print(f"This run publishes {count(planned)}.") + else: + print(f"This run publishes {count(planned)}, then stops before {steps[stop][0]}.") + + +def write_plan_output(path: Path, steps: list[tuple[str, str, str]]) -> None: + before, first_new = plan_facts(steps) + try: + with path.open("a") as output: + output.write(f"publish_count={before}\nfirst_new={first_new}\n") + except OSError as error: + raise Failure(f"cannot write {path}: {error}") from error + + +def read_sha256sums(path: Path) -> dict[str, str]: + """File name to SHA-256, from a list in the format of sha256sum.""" + try: + text = path.read_text() + except OSError as error: + raise Failure(f"cannot read {path}: {error}") from error + sums = {} + for number, line in enumerate(text.splitlines(), 1): + if not line.strip(): + continue + match = SHA256_LINE.fullmatch(line) + if match is None: + raise Failure(f"{path}:{number} is not a line of sha256sum output") + sums[match.group(2)] = match.group(1) + return sums + + +def check_listed(sums: dict[str, str], path: Path, steps: list[tuple[str, str, str]]) -> None: + """Stop before the first upload unless the list has a line for each + crate of `steps`.""" + missing = [package_file(name, version) for name, version, _ in steps if package_file(name, version) not in sums] + if missing: + raise Failure(f"{path} has no SHA-256 for {', '.join(missing)}, so nothing was published") + + +def check_skipped(sums: dict[str, str], path: Path, skipped: dict[str, str]) -> None: + """Stop before the first upload when the index lists a version of the + release with another checksum than the verified package. crates.io then + serves other bytes for that version, for example after an upload that + the check after the upload stopped.""" + other = [crate for crate, cksum in skipped.items() if cksum != sums[crate]] + if other: + raise Failure( + f"the crates.io index lists {', '.join(other)} with another checksum than {path}, so nothing was " + f"published. crates.io serves other bytes for that version. Follow {AFTER_UPLOAD}." + ) + + +def upload_dir(metadata: dict) -> Path: + """The directory where `cargo publish` writes the package that it + uploads. cargo packages into `package/tmp-crate` of its build directory. + Only `cargo package` copies the file on into the target directory + (cargo's src/cargo/ops/cargo_package/mod.rs).""" + build = metadata.get("build_directory") + if not build: + raise Failure("cargo metadata names no build_directory, so the script cannot find the packages that cargo " + "uploads. Use the toolchain of release-crates.yml.") + return Path(build) / "package" / "tmp-crate" + + +def remove_upload(crate: Path, published: list[str]) -> None: + """Delete the file where `cargo publish` writes the package that it + uploads. The check after the upload then reads only a file that this + publish wrote, and not one that an earlier `cargo package` left.""" + try: + crate.unlink(missing_ok=True) + except OSError as error: + raise Failure(f"cannot delete {crate} before the upload: {error}. {resume(published)}") from error + + +def check_upload(crate: Path, expected: str, published: list[str]) -> None: + """Stop when the package that cargo uploaded is not the verified one.""" + try: + digest = hashlib.sha256(crate.read_bytes()).hexdigest() + except OSError as error: + raise Failure(f"cannot read {crate}, the package that cargo uploaded: {error}. {resume(published)}") from error + if digest != expected: + raise Failure( + f"cargo uploaded {crate.name} with SHA-256 {digest}, and the verified package has {expected}. " + f"crates.io keeps the uploaded bytes. {done(published)} Do not run the script again for " + f"this release. Follow {AFTER_UPLOAD}." + ) + + +def run(command: list[str], root: Path) -> tuple[int, str]: + """Run `command` in `root`, echo its output as it arrives, and return + the exit code and the output.""" + print("$ " + " ".join(command), flush=True) + try: + process = subprocess.Popen(command, cwd=root, stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, text=True) + except OSError as error: + raise Failure(f"cannot run {command[0]}: {error}") from error + lines = [] + with process: + for line in process.stdout: + sys.stdout.write(line) + sys.stdout.flush() + lines.append(line) + return process.returncode, "".join(lines) + + +def retry_wait(output: str, current: datetime) -> int | None: + """The seconds to wait before the next try when `output` reports a 429 + answer, or None for any other failure.""" + if not TOO_MANY_REQUESTS.search(output): + return None + match = RETRY_AT.search(output) + if match: + try: + at = email.utils.parsedate_to_datetime(match.group(1)) + except (TypeError, ValueError): + return DEFAULT_RETRY_WAIT + return max(0, math.ceil((at - current).total_seconds())) + RETRY_MARGIN + match = RETRY_SECONDS.search(output) + if match: + return int(match.group(1)) + RETRY_MARGIN + return DEFAULT_RETRY_WAIT + + +def missing_dependency(output: str, published: list[str]) -> str | None: + """A crate that this run published and that cargo did not find in the + index, or None.""" + for match in MISSING_DEPENDENCY.finditer(output): + name = match.group(1) or match.group(2) + if name in published: + return name + return None + + +def done(published: list[str]) -> str: + return f"Published in this run: {', '.join(published) if published else 'none'}." + + +def resume(published: list[str]) -> str: + return (f"{done(published)} Run the script again to continue, or in CI dispatch release-crates again. " + "It skips each version that crates.io has.") + + +def publish(root: Path, steps: list[tuple[str, str, str]], no_verify: bool, max_wait: int, + sums: dict[str, str] | None, packages: Path | None) -> list[str]: + """Publish each crate of `steps` in order, and return the names of the + crates published.""" + published: list[str] = [] + waited = 0 + for name, version, _ in steps: + command = [cargo(), "publish", "-p", name, "--locked"] + if no_verify: + command.append("--no-verify") + if sums is not None: + remove_upload(packages / package_file(name, version), published) + while True: + code, output = run(command, root) + if code == 0: + break + if DAILY_LIMIT.search(output): + raise Failure(f"crates.io refuses more versions of {name} for 24 hours, its limit for one crate. " + f"{resume(published)}") + wait = retry_wait(output, now()) + reason = "crates.io answered 429" + if wait is None: + missing = missing_dependency(output, published) + if missing is None: + raise Failure(f"`cargo publish` failed for {name} {version} with exit code {code}. {resume(published)}") + wait = INDEX_RETRY_WAIT + reason = f"the crates.io index does not list {missing} yet" + if waited + wait > max_wait: + raise Failure( + f"{reason}, and a wait of {wait} s before {name} {version} takes the total wait to " + f"{waited + wait} s, over --max-wait {max_wait}. {resume(published)}" + ) + waited += wait + print(f"{reason} for {name} {version}: waiting {wait} s, {waited} s of {max_wait} s in total", + flush=True) + sleep(wait) + published.append(name) + if sums is not None: + crate = package_file(name, version) + check_upload(packages / crate, sums[crate], published) + return published + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n\n", 1)[0]) + parser.add_argument("--new-crates", action="store_true", + help="publish only the crates that crates.io does not have, with an API token") + parser.add_argument("--dry-run", action="store_true", help="print the plan and publish nothing") + parser.add_argument("--no-verify", action="store_true", help="pass --no-verify to cargo publish") + parser.add_argument("--version", metavar="X.Y.Z", + help="the release version: stop when a crate to publish has another version") + parser.add_argument("--sha256sums", type=Path, metavar="FILE", + help="the SHA-256 list of the verified packages, in the format of sha256sum") + parser.add_argument("--expect-new", metavar="NAME", + help="the first new crate that the gate found: a stop before it exits 0") + parser.add_argument("--plan-output", type=Path, metavar="FILE", + help="append publish_count and first_new to FILE, in the format of GITHUB_OUTPUT") + parser.add_argument("--list-packages", action="store_true", + help="print the package file name of each crate in publish order, and read no index") + parser.add_argument("--max-wait", type=int, default=DEFAULT_MAX_WAIT, metavar="SECONDS", + help=f"cap on the total of the waits after 429 answers and for the index (default: {DEFAULT_MAX_WAIT})") + parser.add_argument("--root", type=Path, default=Path(__file__).resolve().parent.parent, metavar="DIR", + help="workspace root, or a directory in the workspace (default: this repository)") + args = parser.parse_args(argv) + if args.max_wait < 0: + parser.error("--max-wait must not be negative") + if args.new_crates and args.expect_new: + parser.error("--expect-new applies only without --new-crates") + root = args.root.resolve() + try: + metadata = cargo_metadata(root) + order = publish_order(metadata) + check_versions(order, args.version) + if args.list_packages: + for package in order: + print(package_file(package["name"], package["version"])) + return 0 + sums = read_sha256sums(args.sha256sums) if args.sha256sums else None + steps, skipped = plan(order, check_trusted_publishing=not args.new_crates) + stop = stop_at(steps, args.new_crates) + to_publish = [step for step in steps[:stop] if step[2] != SKIP] + print_plan(steps, stop, args.new_crates) + if args.expect_new and args.expect_new not in {name for name, _, _ in steps}: + raise Failure(f"--expect-new names {args.expect_new}, which is not a crate of this release") + if args.plan_output: + write_plan_output(args.plan_output, steps) + packages = None + if sums is not None: + check_listed(sums, args.sha256sums, [step for step in steps if step[2] == SKIP or step in to_publish]) + check_skipped(sums, args.sha256sums, skipped) + packages = upload_dir(metadata) + if args.dry_run: + print("Dry run: nothing was published.") + return 0 + published = publish(root, to_publish, args.no_verify, args.max_wait, sums, packages) + except Failure as failure: + sys.stdout.flush() + print(f"error: {failure}", file=sys.stderr) + return 1 + skipped = sum(1 for _, _, action in steps[:stop] if action == SKIP) + print(f"Published {count(len(published))}, and skipped {count(skipped)} that crates.io has.") + if stop is None: + if args.new_crates: + print(TOKEN_DONE.format(steps=TOKEN_STEPS)) + elif args.expect_new: + print(f"The gate found {args.expect_new} new, but crates.io has it now, so this run did not stop before it.") + return 0 + name = steps[stop][0] + if args.new_crates: + print(TOKEN_STOP.format(name=name, steps=TOKEN_STEPS)) + return 1 + if args.expect_new == name: + print(f"Stopped before {name}, the new crate that the gate found.") + return 0 + print(FIRST_PUBLISH.format(name=name, steps=TOKEN_STEPS), flush=True) + if args.expect_new: + print(f"error: the gate found {args.expect_new} as the first new crate, and this run stops before {name}", + file=sys.stderr) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/devtools/publish_crates_test.py b/devtools/publish_crates_test.py new file mode 100644 index 000000000..791ba6583 --- /dev/null +++ b/devtools/publish_crates_test.py @@ -0,0 +1,1251 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 +"""Tests of devtools/publish-crates. + +The tests replace the script's calls to `cargo metadata`, to the crates.io +sparse index and API, to `cargo publish`, to the clock and to sleep with +fakes. So they use no network and publish nothing. CargoMetadataTest runs +`cargo metadata` on workspaces in a temporary directory, and RunTest starts +Python child processes. + + python3 -m unittest discover -s devtools -p '*_test.py' +""" +import contextlib +import email.utils +import hashlib +import http.client +import importlib.util +import io +import json +import os +import sys +import tempfile +import unittest +import urllib.error +from datetime import UTC, datetime, timedelta +from importlib.machinery import SourceFileLoader +from pathlib import Path +from unittest import mock + +SCRIPT = Path(__file__).resolve().parent / "publish-crates" +START = datetime(2026, 11, 10, 14, 0, 0, tzinfo=UTC) +TOKEN = "cio-test-token-that-must-not-appear" +INDEX = "https://index.crates.io/" +API = "https://crates.io/api/v1/crates/" + + +def load_script(): + """Import the script, which has no .py suffix, as a module.""" + loader = SourceFileLoader("publish_crates", str(SCRIPT)) + spec = importlib.util.spec_from_loader(loader.name, loader) + module = importlib.util.module_from_spec(spec) + loader.exec_module(module) + return module + + +def member(name: str, *dependencies, version: str = "1.0.0", publish=None) -> dict: + """A workspace member as `cargo metadata --no-deps` lists it. Each + dependency is a name (a normal dependency) or a (name, kind) pair.""" + listed = [] + for dependency in dependencies: + target, kind = dependency if isinstance(dependency, tuple) else (dependency, None) + listed.append({"name": target, "kind": kind}) + return {"name": name, "version": version, "publish": publish, "dependencies": listed} + + +def package_bytes(name: str, version: str) -> bytes: + """The bytes the fake `cargo publish` writes for a package.""" + return f"package {name} {version}\n".encode() + + +def too_many_requests(at: datetime) -> tuple[int, str]: + """cargo's exit code and output for crates.io's 429 answer.""" + return 101, ( + "error: failed to publish to registry at https://crates.io\n\nCaused by:\n" + " the remote server responded with an error (status 429 Too Many Requests): You have " + "published too many new crates in a short period of time. Please try again after " + f"{email.utils.format_datetime(at, usegmt=True)} and see https://crates.io/docs/rate-limits " + "for more details.\n" + ) + + +def code_answer(status: int, *headers: str) -> tuple[int, str]: + """cargo's exit code and output for an answer without crates.io's JSON + errors.""" + lines = "".join(f"\t{header}\n" for header in headers) + return 101, ( + "error: failed to publish to registry at https://crates.io\n\nCaused by:\n" + f" failed to get a 200 OK response, got {status}\nheaders:\n{lines}body:\nslow down\n" + ) + + +# crates.io's answer for its limit on the versions of one crate +# (src/controllers/krate/publish.rs), as cargo prints it. +DAILY_LIMIT_ANSWER = (101, ( + "error: failed to publish to registry at https://crates.io\n\nCaused by:\n" + " the remote server responded with an error (status 429 Too Many Requests): You have " + "published too many versions of this crate in the last 24 hours\n" +)) + + +def missing_crate(name: str) -> tuple[int, str]: + """cargo's answer when the index does not know a dependency.""" + return 101, ( + "error: failed to prepare local package for uploading\n\nCaused by:\n" + f" no matching package named `{name}` found\n location searched: crates.io index\n" + " required by package `b v1.0.0 (/work/crates/b)`\n" + ) + + +def missing_version(name: str, version: str) -> tuple[int, str]: + """cargo's answer when the index does not have a dependency's version.""" + return 101, ( + "error: failed to prepare local package for uploading\n\nCaused by:\n" + f" failed to select a version for the requirement `{name} = \"={version}\"`\n" + " candidate versions found which didn't match: 0.9.0\n location searched: crates.io index\n" + ) + + +class FakeTestCase(unittest.TestCase): + """Runs the script's main with fakes for cargo, the index, the API, the + clock and sleep. Tests fill `members`, `index`, `cksums`, `trustpub` + and `results`. A publish that succeeds adds its version and the cksum + of its bytes to the index, and writes its package where cargo writes + it, as crates.io and cargo do.""" + + def setUp(self) -> None: + self.module = load_script() + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.target = Path(self.tmp.name) / "target" + self.build = Path(self.tmp.name) / "build" + self.metadata_extra: dict = {"build_directory": str(self.build)} + self.members: list[dict] = [] + # Crate name to the versions crates.io has. A missing name is a + # crate crates.io does not have. + self.index: dict[str, list[str]] = {} + # (crate name, version) to the cksum that the index lists, when it + # is not the SHA-256 of package_bytes(). A fake publish records the + # SHA-256 of the bytes it uploads. + self.cksums: dict[tuple[str, str], str] = {} + # Crate name to its trustpub_only flag. A crate in `index` at the + # start requires trusted publishing unless a test says otherwise. A + # crate that a fake publish creates does not. + self.trustpub: dict[str, bool] = {} + # Crate name to the results of its successive publishes: (exit code, + # output) pairs, or functions that return one. A publish with no + # result left succeeds. + self.results: dict[str, list] = {} + # Crate name to the bytes a successful publish writes, when they are + # not package_bytes(). A publish of a crate in no_file writes no file. + self.uploaded: dict[str, bytes] = {} + self.no_file: set[str] = set() + self.commands: list[list[str]] = [] + self.fetched: list[str] = [] + self.api_reads: list[str] = [] + self.slept: list[int] = [] + self.pauses = 0 + self.clock = START + for name, fake in (("cargo_metadata", self.fake_metadata), ("fetch", self.fake_fetch), + ("run", self.fake_run), ("now", lambda: self.clock), ("sleep", self.fake_sleep), + ("api_pause", self.fake_pause)): + patcher = mock.patch.object(self.module, name, fake) + patcher.start() + self.addCleanup(patcher.stop) + environment = mock.patch.dict(os.environ, {"CARGO_REGISTRY_TOKEN": TOKEN}) + environment.start() + self.addCleanup(environment.stop) + os.environ.pop("CARGO", None) + + def fake_metadata(self, root: Path) -> dict: + return {"packages": self.members, "target_directory": str(self.target), **self.metadata_extra} + + def fake_fetch(self, url: str) -> bytes | None: + if url.startswith(API): + name = url[len(API):] + self.api_reads.append(name) + if name not in self.index: + return None + return json.dumps({"crate": {"name": name, "trustpub_only": self.trustpub.get(name, True)}}).encode() + self.fetched.append(url) + name = url.rsplit("/", 1)[1] + if name not in self.index: + return None + return "".join(json.dumps({"name": name, "vers": v, "cksum": self.cksum_of(name, v)}) + "\n" + for v in self.index[name]).encode() + + def cksum_of(self, name: str, version: str) -> str: + return self.cksums.get((name, version), hashlib.sha256(package_bytes(name, version)).hexdigest()) + + def version_of(self, name: str) -> str: + return next(package["version"] for package in self.members if package["name"] == name) + + def fake_run(self, command: list[str], root: Path) -> tuple[int, str]: + self.commands.append(command) + name = command[command.index("-p") + 1] + results = self.results.get(name) + result = results.pop(0) if results else (0, f"Uploaded {name}\n") + if callable(result): + result = result() + if result[0] == 0: + version = self.version_of(name) + if name not in self.index: + self.index[name] = [] + self.trustpub[name] = False + self.index[name].append(version) + uploaded = self.uploaded.get(name, package_bytes(name, version)) + self.cksums[(name, version)] = hashlib.sha256(uploaded).hexdigest() + if name not in self.no_file: + # cargo publish writes the package it uploads here, and only + # cargo package copies it into the target directory. + crate = self.build / "package" / "tmp-crate" / f"{name}-{version}.crate" + crate.parent.mkdir(parents=True, exist_ok=True) + crate.write_bytes(uploaded) + stale = self.target / "package" / crate.name + stale.parent.mkdir(parents=True, exist_ok=True) + stale.write_bytes(package_bytes(name, version)) + return result + + def fake_sleep(self, seconds: int) -> None: + self.slept.append(seconds) + self.clock += timedelta(seconds=seconds) + + def fake_pause(self) -> None: + self.pauses += 1 + + def main(self, *argv: str) -> tuple[int, str, str]: + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + code = self.module.main(list(argv)) + output, errors = out.getvalue(), err.getvalue() + self.assertNotIn(TOKEN, output + errors) + return code, output, errors + + def published(self) -> list[str]: + return [command[command.index("-p") + 1] for command in self.commands] + + def later(self, seconds: float): + """A result: crates.io's 429 answer that asks for a wait of `seconds` + from the time of the publish.""" + return lambda: too_many_requests(self.clock + timedelta(seconds=seconds)) + + def sums_file(self, *wrong: str, leave_out: tuple[str, ...] = ()) -> Path: + """A SHA-256 list of the members' packages. The crates in `wrong` get + another hash, and the crates in `leave_out` get no line.""" + lines = [] + for package in self.members: + name, version = package["name"], package["version"] + if name in leave_out: + continue + digest = "0" * 64 if name in wrong else hashlib.sha256(package_bytes(name, version)).hexdigest() + lines.append(f"{digest} {name}-{version}.crate\n") + path = Path(self.tmp.name) / "SHA256SUMS" + path.write_text("".join(lines)) + return path + + def three_crates(self) -> None: + """a, then b, which depends on a, then c, which depends on b. crates.io + has each crate at 0.9.0.""" + self.members = [member("c", "b"), member("b", "a"), member("a")] + self.index = {"a": ["0.9.0"], "b": ["0.9.0"], "c": ["0.9.0"]} + + def new_crate_between(self) -> None: + """a, b, then the new crate n, then c, which depends on n. crates.io + has a, b and c at 0.9.0.""" + self.members = [member("c", "n"), member("n", "b"), member("b", "a"), member("a")] + self.index = {"a": ["0.9.0"], "b": ["0.9.0"], "c": ["0.9.0"]} + + +class OrderTest(FakeTestCase): + def order(self) -> list[str]: + return [package["name"] for package in self.module.publish_order({"packages": self.members})] + + def test_a_crate_comes_after_each_crate_it_depends_on(self) -> None: + self.members = [member("alpha", ("mid", "build")), member("mid", "zeta"), member("zeta")] + self.assertEqual(self.order(), ["zeta", "mid", "alpha"]) + + def test_crates_that_are_ready_together_go_in_name_order(self) -> None: + self.members = [member("c"), member("b", "a"), member("a")] + self.assertEqual(self.order(), ["a", "b", "c"]) + + def test_a_dev_dependency_does_not_order_the_crates(self) -> None: + self.members = [member("a", ("b", "dev")), member("b", "a")] + self.assertEqual(self.order(), ["a", "b"]) + + def test_a_dependency_outside_the_workspace_is_ignored(self) -> None: + self.members = [member("b", "serde", "a"), member("a", ("tokio", "build"))] + self.assertEqual(self.order(), ["a", "b"]) + + def test_a_member_with_publish_false_is_left_out(self) -> None: + self.members = [member("x", "a", publish=[]), member("a")] + self.assertEqual(self.order(), ["a"]) + + def test_a_member_published_only_to_another_registry_is_left_out(self) -> None: + self.members = [member("x", "a", publish=["my-registry"]), member("a")] + self.assertEqual(self.order(), ["a"]) + + def test_a_member_whose_publish_list_names_crates_io_is_kept(self) -> None: + self.members = [member("x", "a", publish=["my-registry", "crates-io"]), member("a", publish=["crates-io"])] + self.assertEqual(self.order(), ["a", "x"]) + + def test_a_dependency_on_a_member_that_crates_io_does_not_receive_fails(self) -> None: + for publish in ([], ["my-registry"]): + with self.subTest(publish=publish): + self.members = [member("a", "x"), member("x", publish=publish)] + with self.assertRaisesRegex(self.module.Failure, "a depends on x, which crates.io does not receive"): + self.order() + + def test_a_cycle_fails_and_names_the_crates_in_it(self) -> None: + self.members = [member("a", "b"), member("b", "c"), member("c", "a"), member("z")] + with self.assertRaisesRegex(self.module.Failure, "in a cycle: a -> b -> c -> a$"): + self.order() + + def test_the_cycle_named_leaves_out_a_crate_that_only_waits_for_it(self) -> None: + self.members = [member("a", "c"), member("c", "d"), member("d", "c")] + with self.assertRaisesRegex(self.module.Failure, "in a cycle: c -> d -> c$"): + self.order() + + def test_the_cycle_named_follows_the_smallest_name_when_a_crate_waits_for_two(self) -> None: + self.members = [member("a", "c", "b"), member("b", "a"), member("c", "a")] + with self.assertRaisesRegex(self.module.Failure, "in a cycle: a -> b -> a$"): + self.order() + + def test_the_cycle_named_starts_at_the_smallest_name_and_not_the_first_member(self) -> None: + self.members = [member("y", "x"), member("x", "y")] + with self.assertRaisesRegex(self.module.Failure, "in a cycle: x -> y -> x$"): + self.order() + + def test_a_cycle_stops_the_run_before_the_index_is_read(self) -> None: + self.members = [member("a", "b"), member("b", "a")] + code, _, errors = self.main("--dry-run") + self.assertEqual(code, 1) + self.assertIn("error: the workspace crates depend on each other in a cycle: a -> b -> a", errors) + self.assertEqual((self.fetched, self.api_reads, self.commands), ([], [], [])) + + def test_the_plan_lists_the_crates_in_publish_order(self) -> None: + self.three_crates() + code, output, _ = self.main("--dry-run") + self.assertEqual(code, 0, output) + self.assertIn("Publish order, 3 crates, with trusted publishing:\n" + " 1. a 1.0.0: new version, publish\n 2. b 1.0.0: new version, publish\n" + " 3. c 1.0.0: new version, publish\nThis run publishes 3 crates.\n", output) + + +class CargoMetadataTest(unittest.TestCase): + """The script against `cargo metadata` of workspaces in a temporary + directory. Needs cargo, and no network.""" + + def setUp(self) -> None: + self.module = load_script() + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.root = Path(self.tmp.name) + self.write("Cargo.toml", '[workspace]\nresolver = "2"\nmembers = ["crates/*"]\n') + + def write(self, relative: str, text: str) -> None: + path = self.root / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text) + + def crate(self, name: str, extra: str = "") -> None: + self.write(f"crates/{name}/Cargo.toml", + f'[package]\nname = "ex-{name}"\nversion = "0.1.0"\nedition = "2021"\n{extra}') + self.write(f"crates/{name}/src/lib.rs", "") + + def order(self) -> list[str]: + return [package["name"] for package in self.module.publish_order(self.module.cargo_metadata(self.root))] + + def run_main(self, *argv: str, env: dict | None = None) -> tuple[int, str]: + out, err = io.StringIO(), io.StringIO() + with mock.patch.dict(os.environ, env or {}), contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + code = self.module.main(list(argv)) + return code, out.getvalue() + err.getvalue() + + def test_the_order_of_a_workspace_that_cargo_reads(self) -> None: + self.crate("zeta", '\n[dev-dependencies]\nex-alpha = { path = "../alpha" }\n') + self.crate("mid", '\n[dependencies]\nex-zeta = { path = "../zeta", version = "=0.1.0" }\n') + self.crate("alpha", '\n[build-dependencies]\nex-mid = { path = "../mid", version = "=0.1.0" }\n') + self.crate("private", 'publish = false\n\n[dependencies]\nex-zeta = { path = "../zeta" }\n') + self.assertEqual(self.order(), ["ex-zeta", "ex-mid", "ex-alpha"]) + + def test_target_specific_optional_and_renamed_dependencies_order_the_crates(self) -> None: + # Without its dependencies, each crate would go in name order. + self.crate("zeta") + self.crate("yak", "\n[target.'cfg(unix)'.dependencies]\n" + 'ex-zeta = { path = "../zeta", version = "=0.1.0" }\n' + "\n[target.'cfg(windows)'.dependencies]\n" + 'ex-zeta = { path = "../zeta", version = "=0.1.0" }\n') + self.crate("xen", '\n[dependencies]\nex-yak = { path = "../yak", version = "=0.1.0", optional = true }\n') + self.crate("wolf", '\n[dependencies]\nalias = { package = "ex-xen", path = "../xen", version = "=0.1.0" }\n') + self.assertEqual(self.order(), ["ex-zeta", "ex-yak", "ex-xen", "ex-wolf"]) + + def test_publish_lists_of_a_workspace_that_cargo_reads(self) -> None: + self.crate("open") + self.crate("named", 'publish = ["crates-io"]\n') + self.crate("other", 'publish = ["my-registry"]\n') + self.assertEqual(self.order(), ["ex-named", "ex-open"]) + + def test_root_may_be_a_member_directory(self) -> None: + self.crate("zeta") + self.crate("mid", '\n[dependencies]\nex-zeta = { path = "../zeta", version = "=0.1.0" }\n') + code, output = self.run_main("--list-packages", "--root", str(self.root / "crates" / "mid")) + self.assertEqual(code, 0, output) + self.assertEqual(output, "ex-zeta-0.1.0.crate\nex-mid-0.1.0.crate\n") + + def test_a_failing_cargo_metadata_is_reported(self) -> None: + self.write("Cargo.toml", "[workspace\n") + code, output = self.run_main("--dry-run", "--root", str(self.root)) + self.assertEqual(code, 1) + self.assertIn("error: `cargo metadata` failed in", output) + + def test_a_cargo_that_cannot_start_is_reported(self) -> None: + code, output = self.run_main("--dry-run", "--root", str(self.root), env={"CARGO": str(self.root / "no-cargo")}) + self.assertEqual(code, 1) + self.assertIn(f"error: cannot run `{self.root / 'no-cargo'} metadata`", output) + self.assertNotIn("Traceback", output) + + +class IndexTest(unittest.TestCase): + def setUp(self) -> None: + self.module = load_script() + + def test_the_index_path_follows_the_cargo_book(self) -> None: + cases = {"a": "1/a", "ab": "2/ab", "abc": "3/a/abc", "abcd": "ab/cd/abcd", + "Extenddb-Core": "ex/te/extenddb-core"} + for name, path in cases.items(): + with self.subTest(name=name): + self.assertEqual(self.module.index_path(name), path) + + def test_index_versions_reads_each_line_of_the_crate_file(self) -> None: + body = (b'{"name":"example-a","vers":"0.1.0","cksum":"1"}\n' + b'{"name":"example-a","vers":"0.2.0","cksum":"2","yanked":true}\n\n') + with mock.patch.object(self.module, "fetch", return_value=body) as fetch: + self.assertEqual(self.module.index_versions("example-a"), {"0.1.0": "1", "0.2.0": "2"}) + fetch.assert_called_once_with("https://index.crates.io/ex/am/example-a") + + def test_index_versions_is_none_for_a_crate_crates_io_does_not_have(self) -> None: + with mock.patch.object(self.module, "fetch", return_value=None): + self.assertIsNone(self.module.index_versions("example-a")) + + def test_an_index_file_that_is_not_an_index_fails_with_its_url(self) -> None: + bodies = {"not UTF-8": b"\xff\xfe\n", "not JSON": b"\n", "no vers": b'{"name":"example-a"}\n', + "no cksum": b'{"name":"example-a","vers":"0.1.0"}\n', "a JSON list": b'["0.1.0"]\n'} + for label, body in bodies.items(): + with self.subTest(body=label), mock.patch.object(self.module, "fetch", return_value=body): + with self.assertRaisesRegex(self.module.Failure, + "^https://index.crates.io/ex/am/example-a holds a line that is not an index entry"): + self.module.index_versions("example-a") + + def fetch(self, answer, url: str = "https://index.crates.io/ex/am/example-a") -> tuple: + """fetch with urlopen replaced. `answer` is the body to return, an + exception for urlopen to raise, or a function that the body's read + calls. Returns the result and urlopen's call.""" + class Response(io.BytesIO): + def read(self, *args): + return answer() if callable(answer) else super().read(*args) + + def urlopen(request, timeout): + calls.append((request, timeout)) + if isinstance(answer, Exception): + raise answer + return Response(b"" if callable(answer) else answer) + + calls = [] + with mock.patch.object(self.module.urllib.request, "urlopen", urlopen): + result = self.module.fetch(url) + return result, calls[0] + + def test_fetch_sends_a_user_agent_and_a_timeout(self) -> None: + result, (request, timeout) = self.fetch(b"line\n") + self.assertEqual(result, b"line\n") + self.assertEqual(request.full_url, "https://index.crates.io/ex/am/example-a") + self.assertIn("ExtendDB/extenddb", request.get_header("User-agent")) + self.assertEqual(timeout, 30) + + def test_fetch_reads_404_410_and_451_as_no_crate(self) -> None: + for status in (404, 410, 451): + with self.subTest(status=status): + error = urllib.error.HTTPError("https://index.crates.io/x", status, "gone", {}, None) + self.assertIsNone(self.fetch(error)[0]) + + def test_fetch_fails_on_any_other_answer(self) -> None: + for status in (403, 500): + with self.subTest(status=status): + error = urllib.error.HTTPError("https://index.crates.io/x", status, "no", {}, None) + with self.assertRaisesRegex(self.module.Failure, f"answered {status} no"): + self.fetch(error) + + def test_fetch_fails_when_the_index_cannot_be_reached(self) -> None: + with self.assertRaisesRegex(self.module.Failure, "^cannot read https://index.crates.io/ex/am/example-a"): + self.fetch(urllib.error.URLError("no route to host")) + + def test_fetch_fails_with_its_url_on_a_broken_http_answer(self) -> None: + def incomplete(): + raise http.client.IncompleteRead(b"par", 10) + + for label, answer in (("bad status line", http.client.BadStatusLine("HTTP/1.1 9x")), + ("incomplete body", incomplete)): + with self.subTest(answer=label): + with self.assertRaisesRegex(self.module.Failure, "^cannot read https://index.crates.io/ex/am/example-a"): + self.fetch(answer) + + def test_trusted_publishing_only_reads_the_crate_from_the_api(self) -> None: + for flag in (True, False): + with self.subTest(trustpub_only=flag): + body = json.dumps({"crate": {"name": "example-a", "trustpub_only": flag}, "versions": []}).encode() + with mock.patch.object(self.module, "fetch", return_value=body) as fetch, \ + mock.patch.object(self.module, "api_pause") as pause: + self.assertIs(self.module.trusted_publishing_only("example-a"), flag) + fetch.assert_called_once_with("https://crates.io/api/v1/crates/example-a") + pause.assert_called_once_with() + + def test_trusted_publishing_only_needs_the_value_true(self) -> None: + body = json.dumps({"crate": {"name": "example-a", "trustpub_only": "true"}}).encode() + with mock.patch.object(self.module, "fetch", return_value=body), mock.patch.object(self.module, "api_pause"): + self.assertIs(self.module.trusted_publishing_only("example-a"), False) + + def test_trusted_publishing_only_fails_on_an_answer_without_the_flag(self) -> None: + bodies = {"not JSON": b"", "no crate": b'{"errors": []}', "no flag": b'{"crate": {"name": "example-a"}}', + "a JSON list": b"[]"} + for label, body in bodies.items(): + with self.subTest(body=label), mock.patch.object(self.module, "fetch", return_value=body), \ + mock.patch.object(self.module, "api_pause"): + with self.assertRaisesRegex(self.module.Failure, + "^https://crates.io/api/v1/crates/example-a sent no crate.trustpub_only"): + self.module.trusted_publishing_only("example-a") + + def test_trusted_publishing_only_fails_on_a_crate_the_api_does_not_have(self) -> None: + with mock.patch.object(self.module, "fetch", return_value=None), mock.patch.object(self.module, "api_pause"): + with self.assertRaisesRegex(self.module.Failure, "has no crate example-a, but the crates.io index lists it"): + self.module.trusted_publishing_only("example-a") + + +class PlanTest(FakeTestCase): + def test_a_version_on_crates_io_is_skipped(self) -> None: + self.three_crates() + self.index["a"] = ["0.9.0", "1.0.0"] + code, output, _ = self.main() + self.assertEqual(code, 0, output) + self.assertEqual(self.published(), ["b", "c"]) + self.assertIn(" 1. a 1.0.0: on crates.io, skip\n 2. b 1.0.0: new version, publish\n", output) + self.assertIn("Published 2 crates, and skipped 1 crate that crates.io has.", output) + + def test_a_version_below_the_newest_is_skipped(self) -> None: + self.three_crates() + self.index["a"] = ["0.9.0", "1.0.0", "1.1.0"] + code, output, _ = self.main() + self.assertEqual(code, 0, output) + self.assertEqual(self.published(), ["b", "c"]) + + def test_the_plan_reads_each_crate_from_the_index(self) -> None: + self.three_crates() + self.main("--dry-run") + self.assertEqual(self.fetched, [INDEX + "1/a", INDEX + "1/b", INDEX + "1/c"]) + + def test_a_crate_of_another_version_stops_the_run_before_the_index_is_read(self) -> None: + self.members = [member("b", "a", version="1.0.1"), member("a"), member("c", version="2.0.0")] + code, _, errors = self.main("--version", "1.0.0") + self.assertEqual(code, 1) + self.assertIn("error: the release is 1.0.0, and these crates have another version: b 1.0.1, c 2.0.0", errors) + self.assertEqual((self.fetched, self.commands), ([], [])) + + def test_crates_of_the_release_version_pass_the_version_check(self) -> None: + self.three_crates() + code, output, _ = self.main("--version", "1.0.0") + self.assertEqual(code, 0, output) + self.assertEqual(self.published(), ["a", "b", "c"]) + + def test_a_dry_run_prints_the_plan_and_the_stop_and_publishes_nothing(self) -> None: + self.new_crate_between() + self.index["a"].append("1.0.0") + code, output, errors = self.main("--dry-run", "--no-verify") + self.assertEqual(code, 0, output + errors) + self.assertEqual(self.commands, []) + self.assertIn(" 1. a 1.0.0: on crates.io, skip\n 2. b 1.0.0: new version, publish\n" + " 3. n 1.0.0: new crate, stop here\n 4. c 1.0.0: new version, after the stop\n" + "This run publishes 1 crate, then stops before n.\nDry run: nothing was published.\n", output) + self.assertEqual(errors, "") + + def test_a_dry_run_with_new_crates_prints_its_stop(self) -> None: + self.new_crate_between() + self.index["b"].append("1.0.0") + code, output, _ = self.main("--dry-run", "--new-crates") + self.assertEqual(code, 0, output) + self.assertEqual(self.commands, []) + self.assertIn("with an API token, new crates only:\n 1. a 1.0.0: new version, stop here\n" + " 2. b 1.0.0: on crates.io, after the stop\n 3. n 1.0.0: new crate, after the stop\n", output) + self.assertIn("This run publishes 0 crates, then stops before a.\n", output) + + def test_a_dry_run_of_a_first_publish_plans_every_crate(self) -> None: + self.three_crates() + self.index = {} + code, output, _ = self.main("--dry-run", "--new-crates") + self.assertEqual(code, 0, output) + self.assertEqual(self.commands, []) + self.assertIn(" 3. c 1.0.0: new crate, publish\nThis run publishes 3 crates.\n", output) + + def test_the_plan_output_counts_the_crates_before_the_first_new_crate(self) -> None: + self.new_crate_between() + self.index["a"].append("1.0.0") + output_file = Path(self.tmp.name) / "github-output" + output_file.write_text("version=1.0.0\n") + code, output, _ = self.main("--dry-run", "--plan-output", str(output_file)) + self.assertEqual(code, 0, output) + self.assertEqual(output_file.read_text(), "version=1.0.0\npublish_count=1\nfirst_new=n\n") + + def test_the_plan_output_of_a_release_without_a_new_crate(self) -> None: + self.three_crates() + output_file = Path(self.tmp.name) / "github-output" + self.main("--dry-run", "--plan-output", str(output_file)) + self.assertEqual(output_file.read_text(), "publish_count=3\nfirst_new=\n") + + def test_the_plan_output_of_a_first_publish(self) -> None: + self.three_crates() + self.index = {} + output_file = Path(self.tmp.name) / "github-output" + self.main("--dry-run", "--plan-output", str(output_file)) + self.assertEqual(output_file.read_text(), "publish_count=0\nfirst_new=a\n") + + def test_a_plan_output_that_cannot_be_written_fails(self) -> None: + self.three_crates() + code, _, errors = self.main("--dry-run", "--plan-output", str(Path(self.tmp.name) / "missing" / "out")) + self.assertEqual(code, 1) + self.assertIn("error: cannot write", errors) + + def test_a_negative_wait_cap_is_refused(self) -> None: + with self.assertRaises(SystemExit) as caught, contextlib.redirect_stderr(io.StringIO()): + self.module.main(["--max-wait", "-1"]) + self.assertEqual(caught.exception.code, 2) + + def test_expect_new_with_new_crates_is_refused(self) -> None: + err = io.StringIO() + with self.assertRaises(SystemExit) as caught, contextlib.redirect_stderr(err): + self.module.main(["--new-crates", "--expect-new", "n"]) + self.assertEqual(caught.exception.code, 2) + self.assertIn("--expect-new applies only without --new-crates", err.getvalue()) + + +class TrustedPublishingTest(FakeTestCase): + def test_each_crate_on_crates_io_is_read_from_the_api_after_a_pause(self) -> None: + self.new_crate_between() + code, output, _ = self.main("--dry-run") + self.assertEqual(code, 0, output) + self.assertEqual(self.api_reads, ["a", "b", "c"]) + self.assertEqual(self.pauses, 3) + + def test_a_crate_that_accepts_api_tokens_stops_the_run(self) -> None: + self.three_crates() + self.trustpub.update({"a": False, "c": False}) + for argv in (["--dry-run"], []): + with self.subTest(argv=argv): + code, _, errors = self.main(*argv) + self.assertEqual(code, 1) + self.assertEqual(self.commands, []) + self.assertIn('error: crates.io accepts API tokens for a, c. On the Settings tab of each, select ' + '"Require trusted publishing for all new versions"', errors) + + def test_the_mode_new_crates_reads_no_api(self) -> None: + self.new_crate_between() + self.trustpub["a"] = False + code, output, _ = self.main("--dry-run", "--new-crates") + self.assertEqual(code, 0, output) + self.assertEqual((self.api_reads, self.pauses), ([], 0)) + + +class ModeTest(FakeTestCase): + def test_trusted_publishing_stops_before_the_first_new_crate(self) -> None: + self.new_crate_between() + code, output, errors = self.main("--no-verify") + self.assertEqual(code, 1, output) + self.assertEqual(self.published(), ["a", "b"]) + self.assertIn("This run publishes 2 crates, then stops before n.\n", output) + self.assertIn("Published 2 crates, and skipped 0 crates that crates.io has.\n", output) + self.assertIn("crates.io has no crate named n, and trusted publishing cannot create a\ncrate.", output) + self.assertIn("runs devtools/publish-crates --new-crates for it", output) + self.assertIn('packaging/README.md, "Publishing new crates with an API token"', output) + self.assertEqual(errors, "") + + def test_the_summary_counts_only_the_crates_before_the_stop(self) -> None: + self.new_crate_between() + self.index["a"].append("1.0.0") + self.index["c"].append("1.0.0") + code, output, _ = self.main() + self.assertEqual(code, 1, output) + self.assertEqual(self.published(), ["b"]) + self.assertIn(" 4. c 1.0.0: on crates.io, after the stop\n", output) + self.assertIn("Published 1 crate, and skipped 1 crate that crates.io has.\n", output) + + def test_a_stop_before_the_expected_new_crate_exits_0(self) -> None: + self.new_crate_between() + code, output, errors = self.main("--expect-new", "n") + self.assertEqual(code, 0, output + errors) + self.assertEqual(self.published(), ["a", "b"]) + self.assertIn("Stopped before n, the new crate that the gate found.", output) + self.assertNotIn("trusted publishing cannot create", output) + + def test_a_stop_before_another_new_crate_than_expected_exits_1(self) -> None: + self.new_crate_between() + code, output, errors = self.main("--expect-new", "c") + self.assertEqual(code, 1) + self.assertEqual(self.published(), ["a", "b"]) + self.assertIn("crates.io has no crate named n", output) + self.assertIn("error: the gate found c as the first new crate, and this run stops before n", errors) + + def test_expect_new_must_name_a_crate_of_the_release(self) -> None: + self.new_crate_between() + for argv in (["--dry-run"], []): + with self.subTest(argv=argv): + code, output, errors = self.main("--expect-new", "x", *argv) + self.assertEqual(code, 1) + self.assertEqual(self.commands, []) + self.assertIn("error: --expect-new names x, which is not a crate of this release", errors) + self.assertNotIn("Dry run", output) + + def test_a_run_that_does_not_stop_says_why_the_expected_stop_did_not_happen(self) -> None: + # The gate found n new. A token run published n, and its owners + # required trusted publishing, before this run started. + self.new_crate_between() + self.index["n"] = ["1.0.0"] + for argv, published in ((["--sha256sums", str(self.sums_file())], ["a", "b", "c"]), ([], [])): + with self.subTest(argv=argv): + self.commands.clear() + code, output, errors = self.main("--expect-new", "n", *argv) + self.assertEqual(code, 0, output + errors) + self.assertEqual(self.published(), published) + self.assertIn(" 3. n 1.0.0: on crates.io, skip\n", output) + self.assertTrue(output.endswith( + "The gate found n new, but crates.io has it now, so this run did not stop before it.\n"), output) + self.assertNotIn("Stopped before", output) + self.assertEqual(errors, "") + + def test_a_run_without_expect_new_that_does_not_stop_prints_no_line_about_it(self) -> None: + self.three_crates() + code, output, _ = self.main() + self.assertEqual(code, 0, output) + self.assertNotIn("The gate found", output) + + def test_new_crates_stops_before_the_first_crate_that_crates_io_has(self) -> None: + self.members = [member("z", "y"), member("y", "x"), member("x")] + self.index = {"y": ["0.9.0"]} + code, output, _ = self.main("--new-crates", "--no-verify") + self.assertEqual(code, 1, output) + self.assertEqual(self.published(), ["x"]) + self.assertEqual(self.commands[0], ["cargo", "publish", "-p", "x", "--locked", "--no-verify"]) + self.assertIn(" 2. y 1.0.0: new version, stop here\n 3. z 1.0.0: new crate, after the stop\n", output) + self.assertIn("crates.io has y, and an API token must not publish it.", output) + self.assertIn("dispatch release-crates again with the tag", output) + + def test_new_crates_never_publishes_a_crate_that_crates_io_has(self) -> None: + self.members = [member("n", "a"), member("a")] + self.index = {"a": ["0.9.0"]} + code, output, _ = self.main("--new-crates") + self.assertEqual(code, 1, output) + self.assertEqual(self.commands, []) + self.assertIn("This run publishes 0 crates, then stops before a.\n", output) + + def test_new_crates_without_a_stop_exits_0(self) -> None: + self.three_crates() + self.index = {} + code, output, _ = self.main("--new-crates") + self.assertEqual(code, 0, output) + self.assertEqual(self.published(), ["a", "b", "c"]) + self.assertIn("Revoke the token now", output) + self.assertIn("dispatch release-crates again with the tag, so that its index and smoke", output) + + +class SequenceTest(FakeTestCase): + """Whole releases, run by run, against the fake crates.io that each + publish updates.""" + + def run_mode(self, *argv: str) -> tuple[int, list[str]]: + self.commands.clear() + code, output, errors = self.main("--no-verify", *argv) + return code, self.published() + + def test_the_first_publish(self) -> None: + self.three_crates() + self.index = {} + code, output, _ = self.main("--dry-run") + self.assertEqual(code, 0, output) + self.assertIn("This run publishes 0 crates, then stops before a.", output) + self.assertEqual(self.run_mode("--new-crates"), (0, ["a", "b", "c"])) + # Each crate refuses API tokens only after a maintainer sets it. + self.assertEqual(self.run_mode(), (1, [])) + self.trustpub.update(dict.fromkeys("abc", True)) + self.assertEqual(self.run_mode(), (0, [])) + + def test_a_release_that_adds_a_crate(self) -> None: + self.new_crate_between() + self.assertEqual(self.run_mode(), (1, ["a", "b"])) + self.assertEqual(self.run_mode("--new-crates"), (1, ["n"])) + self.trustpub["n"] = True + self.assertEqual(self.run_mode(), (0, ["c"])) + self.assertEqual(self.index, {"a": ["0.9.0", "1.0.0"], "b": ["0.9.0", "1.0.0"], "c": ["0.9.0", "1.0.0"], + "n": ["1.0.0"]}) + + def test_two_new_crates_with_a_crate_between_them(self) -> None: + self.members = [member("c", "n2"), member("n2", "b"), member("b", "n1"), member("n1", "a"), member("a")] + self.index = {"a": ["0.9.0"], "b": ["0.9.0"], "c": ["0.9.0"]} + self.assertEqual(self.run_mode(), (1, ["a"])) + self.assertEqual(self.run_mode("--new-crates"), (1, ["n1"])) + self.trustpub["n1"] = True + self.assertEqual(self.run_mode(), (1, ["b"])) + self.assertEqual(self.run_mode("--new-crates"), (1, ["n2"])) + self.trustpub["n2"] = True + self.assertEqual(self.run_mode(), (0, ["c"])) + + def test_two_new_crates_next_to_each_other(self) -> None: + self.members = [member("c", "n2"), member("n2", "n1"), member("n1", "a"), member("a")] + self.index = {"a": ["0.9.0"], "c": ["0.9.0"]} + self.assertEqual(self.run_mode(), (1, ["a"])) + self.assertEqual(self.run_mode("--new-crates"), (1, ["n1", "n2"])) + self.trustpub.update({"n1": True, "n2": True}) + self.assertEqual(self.run_mode(), (0, ["c"])) + + def test_the_run_after_a_token_run_names_the_crate_that_accepts_tokens(self) -> None: + self.new_crate_between() + self.run_mode() + self.run_mode("--new-crates") + code, output, errors = self.main() + self.assertEqual(code, 1) + self.assertIn("error: crates.io accepts API tokens for n.", errors) + + +class PublishTest(FakeTestCase): + def test_each_crate_is_published_locked_in_order(self) -> None: + self.three_crates() + code, output, _ = self.main() + self.assertEqual(code, 0, output) + self.assertEqual(self.commands, [["cargo", "publish", "-p", name, "--locked"] for name in "abc"]) + # A complete run with trusted publishing prints none of the steps + # for an API token. + for text in ("Revoke the token", "an API token must not publish", "trusted publishing cannot create", + "Publishing new crates with an API token"): + self.assertNotIn(text, output) + + def test_no_verify_reaches_cargo(self) -> None: + self.three_crates() + self.main("--no-verify") + self.assertEqual(self.commands, [["cargo", "publish", "-p", name, "--locked", "--no-verify"] for name in "abc"]) + + def test_the_cargo_variable_names_the_cargo_program(self) -> None: + self.three_crates() + os.environ["CARGO"] = "/opt/rust/bin/cargo" + self.main() + self.assertEqual({command[0] for command in self.commands}, {"/opt/rust/bin/cargo"}) + + def test_a_failed_publish_stops_the_run(self) -> None: + self.three_crates() + self.results["b"] = [(101, "error: failed to prepare local package for uploading\n")] + code, _, errors = self.main() + self.assertEqual(code, 1) + self.assertEqual(self.published(), ["a", "b"]) + self.assertEqual(self.slept, []) + self.assertIn("error: `cargo publish` failed for b 1.0.0 with exit code 101. Published in this run: a. " + "Run the script again to continue, or in CI dispatch release-crates again.", errors) + + def test_a_failed_first_publish_reports_that_none_was_published(self) -> None: + self.three_crates() + self.results["a"] = [(101, "error: crate version `1.0.0` is already uploaded\n")] + _, _, errors = self.main() + self.assertIn("Published in this run: none.", errors) + + def test_uploads_that_match_the_verified_list_pass(self) -> None: + self.three_crates() + code, output, errors = self.main("--sha256sums", str(self.sums_file())) + self.assertEqual(code, 0, output + errors) + self.assertEqual(self.published(), ["a", "b", "c"]) + + def test_an_upload_that_differs_from_the_verified_list_stops_the_run(self) -> None: + self.three_crates() + self.uploaded["b"] = b"other bytes\n" + code, _, errors = self.main("--sha256sums", str(self.sums_file())) + self.assertEqual(code, 1) + self.assertEqual(self.published(), ["a", "b"]) + other = hashlib.sha256(b"other bytes\n").hexdigest() + verified = hashlib.sha256(package_bytes("b", "1.0.0")).hexdigest() + self.assertIn(f"error: cargo uploaded b-1.0.0.crate with SHA-256 {other}, and the verified package has " + f"{verified}. crates.io keeps the uploaded bytes. Published in this run: a, b. Do not run the " + 'script again for this release. Follow packaging/README.md, "A failure after the upload".', errors) + + def test_an_upload_without_a_package_file_stops_the_run(self) -> None: + self.three_crates() + self.no_file.add("a") + code, _, errors = self.main("--sha256sums", str(self.sums_file())) + self.assertEqual(code, 1) + self.assertEqual(self.published(), ["a"]) + self.assertIn(f"error: cannot read {self.build / 'package' / 'tmp-crate' / 'a-1.0.0.crate'}, the package that " + "cargo uploaded", errors) + + def test_the_hash_check_needs_the_build_directory_before_any_upload(self) -> None: + self.three_crates() + self.metadata_extra = {} + code, _, errors = self.main("--sha256sums", str(self.sums_file())) + self.assertEqual(code, 1) + self.assertEqual(self.commands, []) + self.assertIn("error: cargo metadata names no build_directory", errors) + self.commands.clear() + code, output, _ = self.main() + self.assertEqual(code, 0, output) + self.assertEqual(self.published(), ["a", "b", "c"]) + + def test_a_list_without_a_crate_to_publish_stops_the_run_before_any_upload(self) -> None: + self.three_crates() + sums = self.sums_file(leave_out=("b", "c")) + code, _, errors = self.main("--sha256sums", str(sums)) + self.assertEqual(code, 1) + self.assertEqual(self.commands, []) + self.assertIn(f"error: {sums} has no SHA-256 for b-1.0.0.crate, c-1.0.0.crate, so nothing was published", errors) + + def test_a_stale_package_from_an_earlier_cargo_package_does_not_pass_the_check(self) -> None: + # The publish job runs cargo package first, which leaves the verified + # bytes where cargo publish writes its upload. A cargo that wrote its + # upload elsewhere would leave that file in place. + self.three_crates() + stale = self.build / "package" / "tmp-crate" / "a-1.0.0.crate" + stale.parent.mkdir(parents=True) + stale.write_bytes(package_bytes("a", "1.0.0")) + self.no_file.add("a") + code, _, errors = self.main("--sha256sums", str(self.sums_file())) + self.assertEqual(code, 1) + self.assertEqual(self.published(), ["a"]) + self.assertFalse(stale.exists()) + self.assertIn(f"error: cannot read {stale}, the package that cargo uploaded", errors) + + def test_without_a_list_the_upload_file_is_left_alone(self) -> None: + self.three_crates() + kept = self.build / "package" / "tmp-crate" / "a-1.0.0.crate" + kept.parent.mkdir(parents=True) + kept.write_bytes(b"left by cargo package\n") + self.no_file.add("a") + code, output, _ = self.main() + self.assertEqual(code, 0, output) + self.assertEqual(kept.read_bytes(), b"left by cargo package\n") + + def test_an_upload_file_that_cannot_be_deleted_stops_the_run_before_its_upload(self) -> None: + self.three_crates() + blocked = self.build / "package" / "tmp-crate" / "b-1.0.0.crate" + (blocked / "inside").mkdir(parents=True) + code, _, errors = self.main("--sha256sums", str(self.sums_file())) + self.assertEqual(code, 1) + self.assertEqual(self.published(), ["a"]) + self.assertIn(f"error: cannot delete {blocked} before the upload:", errors) + self.assertIn("Published in this run: a.", errors) + + def test_a_list_without_a_crate_that_the_run_skips_stops_the_run_before_any_upload(self) -> None: + self.three_crates() + self.index["a"].append("1.0.0") + sums = self.sums_file(leave_out=("a",)) + code, output, errors = self.main("--sha256sums", str(sums)) + self.assertEqual(code, 1) + self.assertEqual(self.commands, []) + self.assertIn(f"error: {sums} has no SHA-256 for a-1.0.0.crate, so nothing was published", errors) + + def test_a_skipped_crate_whose_index_cksum_matches_the_list_passes(self) -> None: + self.three_crates() + self.index["a"].append("1.0.0") + code, output, errors = self.main("--sha256sums", str(self.sums_file())) + self.assertEqual(code, 0, output + errors) + self.assertEqual(self.published(), ["b", "c"]) + + def test_a_skipped_crate_with_other_bytes_on_crates_io_stops_the_run_before_any_upload(self) -> None: + # An earlier run uploaded a and b, and the check after the upload + # stopped it at b. A new run must not publish c on top of b. + self.three_crates() + self.index["a"].append("1.0.0") + self.index["b"].append("1.0.0") + self.cksums[("b", "1.0.0")] = "f" * 64 + sums = self.sums_file() + for argv in (["--dry-run"], []): + with self.subTest(argv=argv): + code, output, errors = self.main("--sha256sums", str(sums), *argv) + self.assertEqual(code, 1) + self.assertEqual(self.commands, []) + self.assertNotIn("Dry run", output) + self.assertIn(f"error: the crates.io index lists b-1.0.0.crate with another checksum than {sums}, " + "so nothing was published. crates.io serves other bytes for that version. Follow " + 'packaging/README.md, "A failure after the upload".', errors) + + def test_each_skipped_crate_after_the_stop_is_checked_too(self) -> None: + self.new_crate_between() + self.index["c"].append("1.0.0") + self.cksums[("c", "1.0.0")] = "f" * 64 + code, _, errors = self.main("--sha256sums", str(self.sums_file()), "--expect-new", "n") + self.assertEqual(code, 1) + self.assertEqual(self.commands, []) + self.assertIn("lists c-1.0.0.crate with another checksum", errors) + + def test_a_dry_run_with_a_list_without_a_line_stops_before_the_dry_run_output(self) -> None: + self.three_crates() + sums = self.sums_file(leave_out=("b",)) + code, output, errors = self.main("--dry-run", "--sha256sums", str(sums)) + self.assertEqual(code, 1) + self.assertEqual(self.commands, []) + self.assertNotIn("Dry run: nothing was published.", output) + self.assertIn(f"error: {sums} has no SHA-256 for b-1.0.0.crate, so nothing was published", errors) + + def test_a_dry_run_with_a_complete_list_passes(self) -> None: + self.three_crates() + code, output, errors = self.main("--dry-run", "--sha256sums", str(self.sums_file())) + self.assertEqual(code, 0, output + errors) + self.assertEqual(self.commands, []) + self.assertIn("Dry run: nothing was published.", output) + + def test_a_list_that_is_not_sha256sum_output_stops_the_run(self) -> None: + self.three_crates() + sums = Path(self.tmp.name) / "SHA256SUMS" + sums.write_text(f"{'0' * 64} a-1.0.0.crate\n\nnot a line\n") + code, _, errors = self.main("--sha256sums", str(sums)) + self.assertEqual(code, 1) + self.assertIn(f"error: {sums}:3 is not a line of sha256sum output", errors) + self.assertEqual((self.fetched, self.commands), ([], [])) + + def test_a_list_line_with_text_after_the_name_or_upper_case_hex_stops_the_run(self) -> None: + self.three_crates() + digest = hashlib.sha256(package_bytes("a", "1.0.0")).hexdigest() + lines = {"text after the name": f"{digest} a-1.0.0.crate extra", + "a blank after the name": f"{digest} a-1.0.0.crate ", + "upper-case hex": f"{digest.upper()} a-1.0.0.crate"} + for label, line in lines.items(): + with self.subTest(line=label): + sums = Path(self.tmp.name) / "SHA256SUMS" + sums.write_text(line + "\n") + code, _, errors = self.main("--sha256sums", str(sums)) + self.assertEqual(code, 1) + self.assertIn(f"error: {sums}:1 is not a line of sha256sum output", errors) + self.assertEqual((self.fetched, self.commands), ([], [])) + + def test_a_list_in_binary_mode_is_read(self) -> None: + self.three_crates() + lines = "".join(f"{hashlib.sha256(package_bytes(n, '1.0.0')).hexdigest()} *{n}-1.0.0.crate\n" for n in "abc") + sums = Path(self.tmp.name) / "SHA256SUMS" + sums.write_text(lines) + code, output, errors = self.main("--sha256sums", str(sums)) + self.assertEqual(code, 0, output + errors) + + def test_a_list_that_cannot_be_read_stops_the_run(self) -> None: + self.three_crates() + code, _, errors = self.main("--sha256sums", str(Path(self.tmp.name) / "missing")) + self.assertEqual(code, 1) + self.assertIn("error: cannot read", errors) + + +class ListPackagesTest(FakeTestCase): + def test_the_list_names_each_package_in_publish_order(self) -> None: + self.members = [member("c", "b"), member("x", publish=[]), member("b", "a"), member("a"), + member("y", publish=["my-registry"])] + code, output, _ = self.main("--list-packages") + self.assertEqual(code, 0, output) + self.assertEqual(output, "a-1.0.0.crate\nb-1.0.0.crate\nc-1.0.0.crate\n") + self.assertEqual((self.fetched, self.api_reads, self.commands), ([], [], [])) + + def test_the_list_applies_the_version_check(self) -> None: + self.members = [member("a", version="1.0.1")] + code, output, errors = self.main("--list-packages", "--version", "1.0.0") + self.assertEqual(code, 1) + self.assertEqual(output, "") + self.assertIn("these crates have another version: a 1.0.1", errors) + + +class RateLimitTest(FakeTestCase): + def setUp(self) -> None: + super().setUp() + self.three_crates() + + def test_a_429_waits_until_the_time_crates_io_gives(self) -> None: + self.results["b"] = [self.later(600)] + code, output, _ = self.main() + self.assertEqual(code, 0, output) + self.assertEqual(self.slept, [605]) + self.assertEqual(self.published(), ["a", "b", "b", "c"]) + self.assertIn("crates.io answered 429 for b 1.0.0: waiting 605 s, 605 s of 7200 s in total", output) + + def test_a_time_already_past_waits_the_margin_only(self) -> None: + self.results["b"] = [self.later(-30)] + self.main() + self.assertEqual(self.slept, [5]) + + def test_part_of_a_second_rounds_up(self) -> None: + self.clock += timedelta(milliseconds=500) + self.results["b"] = [lambda: too_many_requests(START + timedelta(seconds=60))] + self.main() + self.assertEqual(self.slept, [65]) + + def test_a_retry_after_header_in_seconds(self) -> None: + self.results["b"] = [code_answer(429, "content-type: text/plain", "retry-after: 30")] + self.main() + self.assertEqual(self.slept, [35]) + + def test_a_retry_after_header_with_a_date(self) -> None: + self.results["b"] = [code_answer(429, "retry-after: Tue, 10 Nov 2026 14:02:00 GMT")] + self.main() + self.assertEqual(self.slept, [125]) + + def test_a_429_without_a_time_waits_one_minute(self) -> None: + self.results["b"] = [code_answer(429, "content-type: text/plain")] + self.main() + self.assertEqual(self.slept, [60]) + + def test_a_429_with_a_date_that_does_not_exist_waits_one_minute(self) -> None: + self.results["b"] = [(101, "(status 429 Too Many Requests): Please try again after Sat, 31 Feb 2026 10:00:00 GMT")] + self.main() + self.assertEqual(self.slept, [60]) + + def test_other_answers_are_not_tried_again(self) -> None: + answers = { + "403": (101, "the remote server responded with an error (status 403 Forbidden): New versions of this " + "crate can only be published using Trusted Publishing"), + "500": code_answer(500, "retry-after: 30"), + "429 in a file name": (101, "error: failed to read `/work/crates/b/tests/http-429.json`: no such file\n"), + "429 in a version": (101, "error: crate version `0.429.0` is already uploaded\n"), + } + for label, answer in answers.items(): + with self.subTest(answer=label): + self.commands.clear() + self.index = {"a": ["0.9.0"], "b": ["0.9.0"], "c": ["0.9.0"]} + self.results["b"] = [answer] + code, _, _ = self.main() + self.assertEqual(code, 1) + self.assertEqual(self.published(), ["a", "b"]) + self.assertEqual(self.slept, []) + + def test_the_daily_limit_for_one_crate_stops_the_run_at_once(self) -> None: + self.results["b"] = [DAILY_LIMIT_ANSWER] + code, _, errors = self.main() + self.assertEqual(code, 1) + self.assertEqual(self.slept, []) + self.assertEqual(self.published(), ["a", "b"]) + self.assertIn("error: crates.io refuses more versions of b for 24 hours, its limit for one crate. " + "Published in this run: a. Run the script again to continue", errors) + + def test_the_total_wait_is_capped(self) -> None: + self.results["b"] = [self.later(600), self.later(600)] + code, _, errors = self.main("--max-wait", "1000") + self.assertEqual(code, 1) + self.assertEqual(self.slept, [605]) + self.assertEqual(self.published(), ["a", "b", "b"]) + self.assertIn("error: crates.io answered 429, and a wait of 605 s before b 1.0.0 takes the total wait to " + "1210 s, over --max-wait 1000. Published in this run: a.", errors) + + def test_waits_add_up_across_crates(self) -> None: + self.results = {"a": [self.later(95)], "b": [self.later(95)], "c": [self.later(95)]} + code, _, errors = self.main("--max-wait", "250") + self.assertEqual(code, 1) + self.assertEqual(self.slept, [100, 100]) + self.assertIn("takes the total wait to 300 s, over --max-wait 250. Published in this run: a, b.", errors) + + def test_a_total_wait_equal_to_the_cap_is_allowed(self) -> None: + self.results["b"] = [self.later(300), self.later(300)] + code, output, _ = self.main("--max-wait", "610") + self.assertEqual(code, 0, output) + self.assertEqual(self.slept, [305, 305]) + self.assertIn("waiting 305 s, 610 s of 610 s in total", output) + + def test_a_cap_of_zero_stops_at_the_first_429(self) -> None: + self.results["a"] = [self.later(0)] + code, _, errors = self.main("--max-wait", "0") + self.assertEqual(code, 1) + self.assertEqual(self.slept, []) + self.assertIn("over --max-wait 0. Published in this run: none.", errors) + + def test_the_default_cap_is_two_hours(self) -> None: + for wait, code in ((7195, 0), (7196, 1)): + with self.subTest(wait=wait): + self.commands.clear() + self.slept.clear() + self.index = {"a": ["0.9.0"], "b": ["0.9.0"], "c": ["0.9.0"]} + self.results["b"] = [self.later(wait)] + self.assertEqual(self.main()[0], code) + + def test_a_crate_this_run_published_and_the_index_does_not_list_is_tried_again(self) -> None: + for label, answer in (("a crate", missing_crate("a")), ("a version", missing_version("a", "1.0.0"))): + with self.subTest(missing=label): + self.commands.clear() + self.slept.clear() + self.index = {"a": ["0.9.0"], "b": ["0.9.0"], "c": ["0.9.0"]} + self.results["b"] = [answer] + code, output, _ = self.main() + self.assertEqual(code, 0, output) + self.assertEqual(self.slept, [30]) + self.assertEqual(self.published(), ["a", "b", "b", "c"]) + self.assertIn("the crates.io index does not list a yet for b 1.0.0: waiting 30 s, 30 s of 7200 s " + "in total", output) + + def test_a_missing_crate_that_this_run_did_not_publish_is_not_tried_again(self) -> None: + self.index["a"].append("1.0.0") + self.results["b"] = [missing_crate("a")] + code, _, errors = self.main() + self.assertEqual(code, 1) + self.assertEqual(self.slept, []) + self.assertIn("`cargo publish` failed for b 1.0.0", errors) + + def test_the_waits_for_the_index_count_against_the_cap(self) -> None: + self.results["b"] = [missing_crate("a")] * 5 + code, _, errors = self.main("--max-wait", "100") + self.assertEqual(code, 1) + self.assertEqual(self.slept, [30, 30, 30]) + self.assertIn("error: the crates.io index does not list a yet, and a wait of 30 s before b 1.0.0 takes the " + "total wait to 120 s, over --max-wait 100. Published in this run: a.", errors) + + +class HelpersTest(unittest.TestCase): + def setUp(self) -> None: + self.module = load_script() + + def test_cargo_metadata_reads_the_members_only(self) -> None: + answer = mock.Mock(returncode=0, stdout='{"packages": []}', stderr="") + with mock.patch.dict(os.environ, {"CARGO": "/opt/rust/bin/cargo"}), \ + mock.patch.object(self.module.subprocess, "run", return_value=answer) as run: + self.assertEqual(self.module.cargo_metadata(Path("/work")), {"packages": []}) + self.assertEqual(run.call_args.args[0], ["/opt/rust/bin/cargo", "metadata", "--no-deps", "--format-version", "1"]) + self.assertEqual(run.call_args.kwargs["cwd"], Path("/work")) + + def test_the_clock_is_in_utc(self) -> None: + output = too_many_requests(datetime.now(UTC) + timedelta(seconds=60))[1] + self.assertIn(self.module.retry_wait(output, self.module.now()), range(60, 67)) + + def test_sleep_sleeps(self) -> None: + with mock.patch.object(self.module.time, "sleep") as sleep: + self.module.sleep(3) + sleep.assert_called_once_with(3) + + def test_the_api_pause_is_one_second(self) -> None: + with mock.patch.object(self.module.time, "sleep") as sleep: + self.module.api_pause() + sleep.assert_called_once_with(1) + + def test_counts_of_crates(self) -> None: + self.assertEqual([self.module.count(n) for n in (0, 1, 2)], ["0 crates", "1 crate", "2 crates"]) + + +class RunTest(unittest.TestCase): + """run() with Python child processes in place of cargo.""" + + def setUp(self) -> None: + self.module = load_script() + + def run_child(self, code: str) -> tuple[tuple[int, str], str]: + out = io.StringIO() + command = [sys.executable, "-c", code] + with contextlib.redirect_stdout(out): + result = self.module.run(command, Path.cwd()) + return result, out.getvalue() + + def test_run_echoes_the_output_and_returns_it(self) -> None: + result, echoed = self.run_child("print('one'); print('two'); raise SystemExit(3)") + self.assertEqual(result, (3, "one\ntwo\n")) + self.assertTrue(echoed.startswith(f"$ {sys.executable} -c "), echoed) + self.assertTrue(echoed.endswith("one\ntwo\n"), echoed) + + def test_run_merges_the_error_output(self) -> None: + result, _ = self.run_child("import sys; sys.stderr.write('got 429\\n')") + self.assertEqual(result, (0, "got 429\n")) + + def test_run_reports_a_program_it_cannot_start(self) -> None: + with contextlib.redirect_stdout(io.StringIO()), \ + self.assertRaisesRegex(self.module.Failure, "cannot run /nonexistent/cargo"): + self.module.run(["/nonexistent/cargo", "publish"], Path.cwd()) + + +if __name__ == "__main__": + unittest.main() diff --git a/docs/adr/0029-crates-io-packaging.md b/docs/adr/0029-crates-io-packaging.md index 3db2c36e4..4860386cd 100644 --- a/docs/adr/0029-crates-io-packaging.md +++ b/docs/adr/0029-crates-io-packaging.md @@ -100,11 +100,12 @@ binary crate, and a Cargo feature selects its backend. `devtools/check-crate-packages`. The check fails on a pin that is not exact, or on an internal dev-dependency with a version. It also fails on a missing LICENSE, NOTICE or description, and on a package or manifest field - over crates.io's limits. A pin left at an older version also makes every - cargo command that resolves dependencies fail. -- Compiling each package from its tarball takes minutes, so that build - belongs to the release workflow, before the upload, and not to each pull - request. + over crates.io's limits. A crate whose version is not the workspace + version fails too. A pin left at an older version also makes every cargo + command that resolves dependencies fail. +- Compiling each package from its tarball takes minutes, so that build runs + in the `verify` job of `.github/workflows/release-crates.yml`, before the + upload, and not on each pull request. - A release bump changes the 13 pins together with the workspace version. - Every release publishes all 14 crates, including the crates that did not change. @@ -117,10 +118,17 @@ binary crate, and a Cargo feature selects its backend. from other crates. - The license notices now list the workspace crates under the text of the repository's LICENSE file, with a link to the repository. -- Publishing is follow-up work. It needs a release workflow, and a first - publish with an API token, because trusted publishing cannot create a crate. - Each crate then needs its owners and a trusted publisher entry. Each crate - added to the workspace later needs its own first publish. +- `.github/workflows/release-crates.yml` publishes each release after + `promote-image`, through trusted publishing, with `devtools/publish-crates`. + Trusted publishing cannot create a crate. So a maintainer publishes the + first version of each crate with an API token and + `devtools/publish-crates --new-crates`. That mode publishes only the + crates that crates.io does not have. Each crate then needs its owners and + a trusted publisher entry. +- A release that adds one crate takes three runs. The workflow publishes the + crates before the new crate, and it stops. A maintainer publishes the new + crate with an API token, and a second dispatch publishes the rest. + `packaging/README.md`, "Releasing to crates.io", describes the steps. - A version on crates.io can never be replaced. A yank only stops new lock files from choosing it. diff --git a/packaging/README.md b/packaging/README.md index db0251c22..60f8ffe5a 100644 --- a/packaging/README.md +++ b/packaging/README.md @@ -112,12 +112,13 @@ deliberate second step): `npm/package.json`. The launcher runs only a binary of its own release. On a tag whose `package.json` lags the workspace version, the launcher tests would fail on every build leg. The publish gate refuses such a tag - before any build. This is the same tag the container release consumes; the - two publish workflows are dispatched independently against it. When the - release ships a catalog migration, the same state raises - `RELEASED_THROUGH` to the last migration and pastes the checksum lines the - checksum test prints (CONTRIBUTING.md, "Adding a catalog migration"); the - gate jobs of the two container release workflows run + before any build. The container release uses the same tag. A maintainer + dispatches the container and npm publish workflows independently against + it. The crates.io workflow runs last, after `promote-image`. For a release + that ships a catalog migration, the release commit also raises + `RELEASED_THROUGH` to the last migration. That commit holds the checksum + lines that the checksum test prints (CONTRIBUTING.md, "Adding a catalog + migration"). The gate jobs of the release workflows run `devtools/check-released-through` on the tag and refuse it otherwise. 2. Dispatch the `npm-publish` workflow from `main` with the tag as input. The gate validates shape, existence, ancestry against `main`, and the @@ -144,6 +145,259 @@ automation token scoped to the `@extenddb` scope with publish permission only, and create the GitHub environment `npm` with deployment branch rule `main`, required reviewers, and the token as `NPM_TOKEN`. +## Releasing to crates.io + +The workspace crates publish to crates.io together, at the workspace +version. Each internal dependency requires exactly that version +([ADR-0029](../docs/adr/0029-crates-io-packaging.md)). crates.io has no +candidate stage, and it never replaces a version. So the crates are the +last step of a release, after `promote-image` promotes the images. + +`devtools/publish-crates` publishes the crates in two modes. The token mode +publishes the first version of each crate, and the workflow publishes each +later version: + +- The `release-crates` workflow publishes new versions of the crates that + crates.io has. It uses trusted publishing, so no API token exists. +- A maintainer publishes each new crate with an API token and + `devtools/publish-crates --new-crates`. Trusted publishing cannot create + a crate. This mode never publishes a crate that crates.io has. For a + crate that requires trusted publishing, crates.io refuses a new version + published with an API token. + +### Repository settings for crates.io + +The release depends on two settings of this repository. Check both before +you add a trusted publisher on crates.io. A job that names a missing +environment creates it with no protection rules. + +- The environment `crates-io` has the four required reviewers of the other + release environments, with prevent self-review. So the maintainer who + dispatches a run cannot approve it. The environment allows deployments + from `main` only, and it holds no secrets. Admin bypass is off for + `crates-io`. crates.io never replaces a version, so even an administrator + needs the approval of a second maintainer to publish. + + ```bash + gh api repos/ExtendDB/extenddb/environments/crates-io \ + --jq '{can_admins_bypass, rules: [.protection_rules[] | {type, prevent_self_review, reviewers: [.reviewers[]? | .reviewer.login // .reviewer.slug]}]}' + gh api repos/ExtendDB/extenddb/environments/crates-io/deployment-branch-policies \ + --jq '.branch_policies[] | "\(.type):\(.name)"' + ``` + + The first command prints `can_admins_bypass` false, and a + `required_reviewers` rule with `prevent_self_review` true and the same + reviewers as the environment `npm`. The second command prints only + `branch:main`. +- The tag ruleset `protect-release-tags` is active on `refs/tags/v*`. It + blocks the deletion and the update of those tags, and it has no bypass + actors. So a release tag cannot move to another commit. + + ```bash + gh api repos/ExtendDB/extenddb/rulesets \ + --jq '.[] | select(.name == "protect-release-tags") | {id, target, enforcement}' + gh api repos/ExtendDB/extenddb/rulesets/ \ + --jq '{enforcement, bypass_actors, include: .conditions.ref_name.include, rules: [.rules[].type]}' + ``` + + The second command prints the enforcement `active`, an empty + `bypass_actors` list, the pattern `refs/tags/v*`, and the rules + `deletion` and `update`. + +### Each release + +After `promote-image` promotes the images of the release: + +1. A maintainer dispatches the `release-crates` workflow + (`.github/workflows/release-crates.yml`) from `main`, with the release + tag as input. +2. The `gate` job checks that the run comes from `main`. It checks the tag + as `npm-publish` does. The tag must have the strict shape and exist, and + its commit must be on `main`. Its version must equal the workspace + version. +3. The gate reads `docker.io/extenddb/extenddb-postgres:X.Y.Z` without + credentials. The revision label of that image must be the tagged commit. + The gate also runs `devtools/check-released-through`. +4. Then `devtools/publish-crates --dry-run` plans the run, and the gate + writes the tag, the commit and the plan to the job summary. The plan + fails when a crate on crates.io accepts API tokens. A new crate does not + fail the gate, and the plan shows the stop before it. +5. The `verify` job holds no credentials. It builds every package from its + tarball with `cargo package --workspace --locked`, and it runs + `devtools/check-crate-packages`. It records the SHA-256 of the package of + each crate that crates.io receives, the tagged commit and the toolchain, + in the artifact `crate-sha256`. +6. When the plan has a crate to publish, a reviewer approves the + `crates-io` environment. The `publish` job packages the crates again + without compiling them, and it runs `devtools/check-crate-packages`. It + stops unless each package has the recorded SHA-256. +7. Then the `publish` job receives a crates.io token through trusted + publishing. `devtools/publish-crates --no-verify` plans the run again, + and it publishes the crates in dependency order, up to the first new + crate. It compares each uploaded package with the recorded SHA-256. The + token expires 30 minutes after the job receives it, and the job revokes + it when it ends. +8. When no new crate stopped the run, the `index` job compares the + checksum of each crate's version in the crates.io index with the + recorded SHA-256. Then the `smoke` job runs + `cargo install extenddb --version X.Y.Z --locked` with an empty cargo + home. It checks the version, the commit and `test-hooks disabled` in the + output of `extenddb version`. It also checks that no + `backup-test-hooks` line appears. +9. When the run stopped before a new crate, the `new-crate` job reports + the stop and points at the token steps in the job summary. Then the job + fails the run. + +The plan of the `publish` job decides between steps 8 and 9, because the +script reads crates.io again before its first upload. A token run can +publish the gate's new crate while the job waits for its approval. The job +then publishes the rest of the release, and `index` and `smoke` run. When +the run skips `publish`, the gate's plan decides. + +The script skips each version that crates.io has. So a run that stopped +part way can be dispatched again. + +### A release that adds a crate + +Trusted publishing cannot create a crate. So a release that adds a crate +takes three runs: + +1. The `release-crates` run publishes the crates before the new crate, and + it stops before the new crate. +2. A maintainer publishes the new crate with an API token, as the next + section describes. That run skips each crate that crates.io has at this + version, and it stops before the next crate that needs a new version. +3. A second dispatch of `release-crates` publishes the rest of the release. + Its gate fails until the new crate requires trusted publishing. + +When a crate that crates.io has comes between two new crates, steps 2 and +3 repeat for the second new crate. + +### Publishing new crates with an API token + +The first publish of the workspace uses these steps for all 14 crates. A +later release uses them for each new crate. Both come after +`promote-image` for the release. + +You need Python 3.11 or later, rustup and the GitHub CLI. You also need a +crates.io account, from a sign-in with GitHub, with a verified email +address. crates.io refuses a publish from an account without one. Before +the first publish, an owner of the ExtendDB GitHub organization allows +crates.io's GitHub OAuth app for the organization. A GitHub team can then +own the crates. + +1. Use the `release-crates` run that stopped before the new crate. For the + first publish, dispatch it as in "Each release". Its `verify` job records + the SHA-256 list, the tagged commit and the toolchain in the artifact + `crate-sha256`. Note the ID of the run. The run keeps the artifact for 7 + days. After that, dispatch `release-crates` again for a new artifact. +2. On crates.io, create an API token (Account Settings, API Tokens, New + Token): + - Scopes: `publish-new` only. So the token cannot publish a new version + of a crate that crates.io has. + - Crates: the pattern `extenddb*` for the first publish, or the name of + the new crate for a later one. + - Expiration: "Custom...", with the date of the next day. +3. Fetch the tag into a new directory, build the packages there, and + compare them with the list of the `verify` job. The block fetches the + tag by its full ref and checks it out detached. Before any command from + the new directory runs, the block stops unless the checked-out commit is + the commit in the artifact. So the code of a branch with the name of the + tag never runs. The block takes the toolchain from the artifact, so it + packages with the toolchain of the `verify` job. It accepts only a + toolchain of the form X.Y.Z. No build of dependency code runs in the new + directory. The comparison stops the procedure before any upload when + this machine writes other bytes. + + The block packages with an empty cargo home, so cargo reads no stored + token. Cargo still reads .cargo/config.toml in the new directory and in + each parent directory, which includes ~/.cargo/config.toml when the new + directory is under your home directory. + + ```bash + TAG=vX.Y.Z + RUN_ID= + SUMS="$(mktemp -d)" && SRC="$(mktemp -d)" \ + && gh run download "$RUN_ID" --repo ExtendDB/extenddb --name crate-sha256 --dir "$SUMS" \ + && grep -qxE '[0-9]+\.[0-9]+\.[0-9]+' "$SUMS/TOOLCHAIN" \ + && git init -q "$SRC" && cd "$SRC" \ + && git fetch --depth 1 https://github.com/ExtendDB/extenddb.git "refs/tags/$TAG:refs/tags/$TAG" \ + && git checkout -q --detach "refs/tags/$TAG" \ + && test "$(git rev-parse HEAD)" = "$(cat "$SUMS/COMMIT")" \ + && export RUSTUP_TOOLCHAIN="$(cat "$SUMS/TOOLCHAIN")" \ + && rustup toolchain install "$RUSTUP_TOOLCHAIN" \ + && export CARGO_HOME="$(mktemp -d)" \ + && cargo package --workspace --no-verify --locked \ + && devtools/check-crate-packages \ + && (cd target/package && sha256sum --check --strict "$SUMS/SHA256SUMS") \ + && devtools/publish-crates --new-crates --dry-run --version "${TAG#v}" --sha256sums "$SUMS/SHA256SUMS" \ + && echo "Ready: $TAG is commit $(git rev-parse HEAD), and the packages have the bytes of the verify job." + ``` + +4. Run the next line only after the block prints its last line, in the + same shell. The command is one line, so `read` takes the token from the + keyboard and never from a pasted line. The line asks for the token, + gives it to the publish command only, and then removes it from the + shell. + + ```bash + printf 'crates.io token: ' && read -rs CARGO_REGISTRY_TOKEN && echo && CARGO_REGISTRY_TOKEN="$CARGO_REGISTRY_TOKEN" devtools/publish-crates --new-crates --no-verify --version "${TAG#v}" --sha256sums "$SUMS/SHA256SUMS"; unset CARGO_REGISTRY_TOKEN + ``` + + With `--no-verify`, cargo publishes without a build. So no build script + or procedural macro runs while the token is in the environment. The + script compares each uploaded package with the list. By default, + crates.io accepts 5 new crates at once and then one every 10 minutes. So + the 14 crates of the first publish take about 90 minutes. The script + waits after each 429 answer, up to 2 hours in total. +5. Revoke the token on crates.io at once. The next steps use the browser + session. +6. Check the settings in "Repository settings for crates.io". +7. On the Settings tab of each new crate on crates.io: + - Add a second named owner. That maintainer signed in to crates.io + before, and accepts the invitation. + - Add the team `github:extenddb:extendb-committers`. A team owner can + publish and yank, but cannot change the owners. The maintainer who adds + the team is a member of `extendb-committers` or an owner of the + ExtendDB organization. That maintainer also grants crates.io the GitHub + scope `read:org`. + - Add a GitHub trusted publisher: repository owner `ExtendDB`, + repository name `extenddb`, workflow filename `release-crates.yml`, + and `crates-io` in the field "Environment name (optional)". The form + marks the environment as optional. Fill it in, because a publisher + without it gives a token to a run of `release-crates.yml` from any + branch. + - Select "Require trusted publishing for all new versions", so that + crates.io refuses a new version published with an API token. +8. After the first publish, check that docs.rs built the library crates. +9. Dispatch `release-crates` again with the release tag. The run publishes + the rest of the release, and its `index` and `smoke` jobs check the + whole release. + +### A failure after the upload + +A version on crates.io is permanent. A trusted publishing token can only +publish, so an owner yanks a version. + +- The `index` job fails when the checksum of a version in the crates.io + index differs from the verified SHA-256. crates.io then serves other bytes + than the `verify` job built. An owner yanks every version of the release, + on the page of each crate, or with `cargo yank` and an API token of the + scope `yank`. The next patch release replaces the release. +- When `devtools/publish-crates` stops because an uploaded package differs + from the verified list, the same steps apply. Do not run the script again + for that release. A later run with `--sha256sums` stops before its first + upload, because the index then lists that version with another checksum. +- The `index` job also fails when the index does not list a version of the + release after 20 tries. Rerun the job first, because the index can lag + behind an upload. If the version is still missing, check the log of the + `publish` job for that crate. +- The `smoke` job fails when `cargo install` fails, or when + `extenddb version` prints a wrong line. A rerun helps only when the + install failed, because the crates.io index can lag. A wrong version, + commit or test-hooks line comes from the published bytes, and a rerun + prints the same line. Then decide whether to yank the release. + ## Known limits - A hard-killed caller (SIGKILL, OOM) cannot run its exit handler, so the