Skip to content

build: make the workspace crates packageable for crates.io - #927

Merged
eliseong merged 4 commits into
mainfrom
build/crates-io-packaging
Nov 11, 2026
Merged

eliseong merged 4 commits into
mainfrom
build/crates-io-packaging

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

What

This change makes all 14 workspace crates pass cargo package, the step before a publish to crates.io. It publishes nothing.

  • Each internal entry of [workspace.dependencies] requires the workspace version exactly: version = "=0.1.14".
  • The 14 internal dev-dependencies that used { workspace = true } now use a path and no version. Cargo removes them from the published manifests.
  • The workspace uses resolver = "3".
  • Each crate has a description, a repository and a homepage. The extenddb crate also gets a README for its crates.io page, keywords and categories.
  • Each crate directory links LICENSE and NOTICE to the root files. I regenerated the four license notices, and they now list the workspace crates under the text of our LICENSE file.
  • Cargo.lock moves spin 0.9.8 to 0.9.9 and chacha20 0.10.1 to 0.10.2. crates.io lists both old versions as yanked.
  • The backup-store README and config.rs drop one S3 server from their examples.
  • cdrs-tokio stays on 9.0.x and cassandra-protocol on 4.0.x. cassandra-protocol 4.1.0 moved its public BigInt to num-bigint 0.5, and the Cassandra backend then fails to compile. Without the bounds, a fresh resolution, such as cargo install without --locked, picks 4.1.0.
  • crates/bin/build.rs reads the packaged commit from .cargo_vcs_info.json. A build from crates.io then reports its commit instead of unknown.
  • The new devtools/check-crate-packages checks the pins, the dev-dependencies, the license files and the metadata. It also checks the field rules and size limits that crates.io applies at upload. It reads the workspace through cargo metadata. Its tests are in devtools/check_crate_packages_test.py.
  • The Test workflow gets a package job, and the required test gate needs it.
  • ADR-0029 records the decision. CONTRIBUTING.md describes the dependency rules, and packaging/README.md adds the pins to the release bump.

Why

Because girls just wanna have fun!!

On main, cargo package --workspace stops at the first crate with an internal dependency:

all dependencies must have a version requirement specified when packaging. dependency extenddb-cache does not specify a version

With a version on each internal dependency, it stops at the dev-dependency cycle between extenddb-engine and extenddb-storage-sqlite:

no matching package named extenddb-engine found

crates.io accepts the extenddb binary only after the 13 crates it depends on, so all 14 must package. The crates change their APIs between patch releases. An exact pin stops a fresh install from mixing crates of two releases.

Without --locked, cargo install resolves every dependency again. On Rust 1.88, our minimum version, that chose uuid 1.29.2, which needs Rust 1.89, and the build failed. With resolver = "3", cargo chose uuid 1.26.1 and the build passed.

Testing done

  • cargo package --workspace --no-verify --locked wrote the 14 packages in 5 s, and devtools/check-crate-packages printed ok.
  • From a cold cache, cargo package --workspace --locked built all 14 packages from their tarballs in 305 s with 12 jobs. It passed again on the final tree. The packaged extenddb build recorded the full commit SHA.
  • cargo metadata --locked gave 530 nodes before and after. Only the spin and chacha20 nodes, the edges of their 3 dependents, and the new edge from extenddb-storage-cassandra to cassandra-protocol differ.
  • I copied only Cargo.toml, Cargo.lock and crates/, as the build stage of Dockerfile.dev does. In that copy the LICENSE and NOTICE links point at nothing. The sqlite,dev-mode build passed on Rust 1.88, and cargo gave no warning.
  • devtools/generate-software-license-notices --check for each backend and devtools/generate-dev-license-notices --check pass.
  • python3 -m unittest discover -s devtools -p '*_test.py' runs 35 tests, and all pass. Each of 48 one-line changes to the check script makes a test fail.
  • With the old requirements, cargo update -p cdrs-tokio moved cassandra-protocol to 4.1.0, and cargo check -p extenddb-storage-cassandra failed with E0308 on BigInt. With the bounds, a fresh lock on Rust 1.88 chose 4.0.1, and the check passed. The bounds change one line of Cargo.lock, the new dependency edge.
  • In CI, the run-tests jobs run the workspace tests on Rust 1.88.0 and on stable, and both pass. The Clippy workflow runs clippy with -D warnings on every crate and target of the workspace, and it passes.
  • With -W clippy::pedantic, clippy reports nothing on the lines that this change touches. It still reports warnings in code outside this change. cargo fmt --all -- --check passes.
  • A probe of the .cargo_vcs_info.json reader ran on Rust 1.88. It reads clean and dirty files in pretty and compact form, and it returns None for malformed input.
  • No CI job in this change compiles the packages from their tarballs. That build takes minutes. The crates.io release workflow, which is follow-up work, has to run it before each upload.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: ADR-0029, docs/adr/0029-crates-io-packaging.md in this PR


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

All 14 workspace crates now pass `cargo package`, the step before a
publish to crates.io. This change publishes nothing.

- Each internal entry of `[workspace.dependencies]` requires the
  workspace version exactly, as in `version = "=0.1.14"`. The crates
  change their APIs between patch releases. An exact pin stops a fresh
  install from mixing crates of two releases.
- Internal dev-dependencies use a path and no version, so cargo removes
  them from the published manifests. Packaging then no longer stops at
  the dev-dependency cycle between extenddb-engine and
  extenddb-storage-sqlite.
- The workspace uses `resolver = "3"`. On Rust 1.88, our minimum
  version, resolver 2 picked uuid 1.29.2 for an install without
  `--locked`. That version needs Rust 1.89. Resolver 3 picks uuid
  1.26.1, and the build passes.
- Each crate has a description, a repository and a homepage. The
  extenddb crate also has a README for its crates.io page, keywords and
  categories.
- Each crate directory links LICENSE and NOTICE to the root files. This
  change regenerates the four license notices, and they list the
  workspace crates.
- Cargo.lock moves spin to 0.9.9 and chacha20 to 0.10.2. crates.io lists
  the old versions as yanked.
- cdrs-tokio stays on 9.0.x and cassandra-protocol on 4.0.x.
  cassandra-protocol 4.1.0 uses num-bigint 0.5 for its public BigInt,
  and the Cassandra backend then fails to compile. To hold the bound,
  extenddb-storage-cassandra now depends on cassandra-protocol directly.
- crates/bin/build.rs reads the packaged commit from
  .cargo_vcs_info.json. A build from crates.io then reports its commit
  instead of "unknown".
- The new devtools/check-crate-packages checks the pins, the
  dev-dependencies, the license files and the metadata. It also checks
  the field rules and size limits that crates.io applies at upload. A
  new `package` job in the Test workflow runs it, and the required
  `test` gate needs that job.
- The backup-store README and config.rs drop one S3 server from their
  examples.
- ADR-0029 records the decision. CONTRIBUTING.md describes the
  dependency rules, and packaging/README.md adds the pins to the release
  bump.
@LeeroyHannigan
LeeroyHannigan force-pushed the build/crates-io-packaging branch from 586e3c7 to 0ce13cf Compare November 10, 2026 13:55
@LeeroyHannigan LeeroyHannigan added v1 Needed for the 1.0 release release-blocker Must ship for its milestone's release labels Nov 10, 2026
@LeeroyHannigan LeeroyHannigan added this to the 1.0 milestone Nov 10, 2026
@eliseong eliseong self-assigned this Nov 10, 2026
amrith
amrith previously approved these changes Nov 10, 2026
@LeeroyHannigan
LeeroyHannigan added this pull request to the merge queue Nov 10, 2026
eliseong
eliseong previously approved these changes Nov 10, 2026

@eliseong eliseong left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I built the branch and confirmed the packaging behavior. All 14 crates pass cargo package. The checker prints ok. The 35 devtools tests pass. The cold verify build from the tarballs passes and records the full commit. The cassandra-protocol bound keeps the Cassandra backend on 4.0.x and it compiles. Resolver 3 keeps the install resolution on a toolchain that builds.

Non blocking follow ups, for whenever this checker becomes the gate a release depends on:

  1. The checker prints ok on a manifest that changed after cargo package ran, because it reads the old tarballs. A re-run guard or a .cargo_vcs_info.json SHA check against HEAD would close that.
  2. The checker accepts a few states that crates.io rejects on upload: an invalid SPDX license, a URL field that is not http or https, an unknown category slug, an external dependency pinned to "*", and a NUL character in a field. Either add these, or adjust the body and ADR so they do not claim upload-rule coverage.
  3. The checker rejects a valid exact pin written with a space, "= 0.1.14". Parsing the requirement instead of the string would fix that.

None of these gate the merge. The packaging change is correct and ships nothing.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 10, 2026
On main, #896 changed step 1 of the npm release flow in
packaging/README.md. This branch changed the same step. The merged step
keeps both changes. It lists the 13 pins and the four license notices,
and it says that the tagged commit must carry the matching
npm/package.json.
@LeeroyHannigan
LeeroyHannigan dismissed stale reviews from eliseong and amrith via b0e8bab November 10, 2026 23:48
eliseong
eliseong previously approved these changes Nov 11, 2026
@eliseong
eliseong added this pull request to the merge queue Nov 11, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 11, 2026
@eliseong
eliseong added this pull request to the merge queue Nov 11, 2026
Main gained #692, #897, #880 and #921 since the last merge. The merge had
no conflicts.
…s by path

On main, #880 added extenddb-engine and extenddb-backup-store to the
dev-dependencies of extenddb-storage-cassandra with workspace = true. On
this branch the workspace entries carry the =0.1.14 pin. So cargo keeps
both dev-dependencies in the published manifest, and
devtools/check-crate-packages refuses the package. Both now name only a
path, as the other backends declare them, so cargo drops them from the
published manifest.
@LeeroyHannigan
LeeroyHannigan removed this pull request from the merge queue due to a manual request Nov 11, 2026
@eliseong
eliseong added this pull request to the merge queue Nov 11, 2026
Merged via the queue into main with commit f22e3d3 Nov 11, 2026
71 of 72 checks passed
@eliseong
eliseong deleted the build/crates-io-packaging branch November 11, 2026 03:16
LeeroyHannigan added a commit that referenced this pull request Nov 11, 2026
Main now holds #927, the base of this branch. On main, #896 changed
step 1 of the npm release flow in packaging/README.md, and this branch
changed the same step. The merged step keeps the text of this branch
and adds the four sentences of #896 about the npm/package.json of the
tagged commit.

The merge base of the two branches predates #927. So four files
that #927 added or changed and this branch then changed also
conflicted: devtools/check-crate-packages,
devtools/check_crate_packages_test.py,
docs/adr/0029-crates-io-packaging.md and .github/workflows/test.yml.
Main holds them as #927 wrote them, so the merge keeps the copies of
this branch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-blocker Must ship for its milestone's release v1 Needed for the 1.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants