From 0ce13cf04314b99e40f737a3de6c168de56aa33e Mon Sep 17 00:00:00 2001
From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com>
Date: Tue, 10 Nov 2026 13:55:36 +0000
Subject: [PATCH 1/2] build: make the workspace crates packageable for
crates.io
All 14 workspace crates now pass `cargo package`, the step before a
publish to crates.io. This change publishes nothing.
- Each internal entry of `[workspace.dependencies]` requires the
workspace version exactly, as in `version = "=0.1.14"`. The crates
change their APIs between patch releases. An exact pin stops a fresh
install from mixing crates of two releases.
- Internal dev-dependencies use a path and no version, so cargo removes
them from the published manifests. Packaging then no longer stops at
the dev-dependency cycle between extenddb-engine and
extenddb-storage-sqlite.
- The workspace uses `resolver = "3"`. On Rust 1.88, our minimum
version, resolver 2 picked uuid 1.29.2 for an install without
`--locked`. That version needs Rust 1.89. Resolver 3 picks uuid
1.26.1, and the build passes.
- Each crate has a description, a repository and a homepage. The
extenddb crate also has a README for its crates.io page, keywords and
categories.
- Each crate directory links LICENSE and NOTICE to the root files. This
change regenerates the four license notices, and they list the
workspace crates.
- Cargo.lock moves spin to 0.9.9 and chacha20 to 0.10.2. crates.io lists
the old versions as yanked.
- cdrs-tokio stays on 9.0.x and cassandra-protocol on 4.0.x.
cassandra-protocol 4.1.0 uses num-bigint 0.5 for its public BigInt,
and the Cassandra backend then fails to compile. To hold the bound,
extenddb-storage-cassandra now depends on cassandra-protocol directly.
- crates/bin/build.rs reads the packaged commit from
.cargo_vcs_info.json. A build from crates.io then reports its commit
instead of "unknown".
- The new devtools/check-crate-packages checks the pins, the
dev-dependencies, the license files and the metadata. It also checks
the field rules and size limits that crates.io applies at upload. A
new `package` job in the Test workflow runs it, and the required
`test` gate needs that job.
- The backup-store README and config.rs drop one S3 server from their
examples.
- ADR-0029 records the decision. CONTRIBUTING.md describes the
dependency rules, and packaging/README.md adds the pins to the release
bump.
---
.github/workflows/test.yml | 27 +-
CONTRIBUTING.md | 16 +
Cargo.lock | 13 +-
Cargo.toml | 53 +++-
SOFTWARE-LICENSE-NOTICES-CASSANDRA.html | 232 +++++++++++++-
SOFTWARE-LICENSE-NOTICES-DEV.html | 232 +++++++++++++-
SOFTWARE-LICENSE-NOTICES-MONGODB.html | 232 +++++++++++++-
SOFTWARE-LICENSE-NOTICES.html | 230 +++++++++++++-
crates/app/Cargo.toml | 3 +
crates/app/LICENSE | 1 +
crates/app/NOTICE | 1 +
crates/auth/Cargo.toml | 5 +-
crates/auth/LICENSE | 1 +
crates/auth/NOTICE | 1 +
crates/backup-store/Cargo.toml | 4 +-
crates/backup-store/LICENSE | 1 +
crates/backup-store/NOTICE | 1 +
crates/backup-store/README.md | 2 +-
crates/backup-store/src/config.rs | 2 +-
crates/bin/Cargo.toml | 8 +-
crates/bin/LICENSE | 1 +
crates/bin/NOTICE | 1 +
crates/bin/README.md | 91 ++++++
crates/bin/build.rs | 39 ++-
crates/bin/src/main.rs | 2 +-
crates/cache/Cargo.toml | 4 +-
crates/cache/LICENSE | 1 +
crates/cache/NOTICE | 1 +
crates/config/Cargo.toml | 3 +
crates/config/LICENSE | 1 +
crates/config/NOTICE | 1 +
crates/core/Cargo.toml | 3 +
crates/core/LICENSE | 1 +
crates/core/NOTICE | 1 +
crates/engine/Cargo.toml | 5 +-
crates/engine/LICENSE | 1 +
crates/engine/NOTICE | 1 +
crates/server/Cargo.toml | 9 +-
crates/server/LICENSE | 1 +
crates/server/NOTICE | 1 +
crates/server/src/serve.rs | 4 +-
crates/storage-cassandra/Cargo.toml | 8 +-
crates/storage-cassandra/LICENSE | 1 +
crates/storage-cassandra/NOTICE | 1 +
crates/storage-mongodb/Cargo.toml | 5 +-
crates/storage-mongodb/LICENSE | 1 +
crates/storage-mongodb/NOTICE | 1 +
crates/storage-postgres/Cargo.toml | 9 +-
crates/storage-postgres/LICENSE | 1 +
crates/storage-postgres/NOTICE | 1 +
crates/storage-sqlite/Cargo.toml | 9 +-
crates/storage-sqlite/LICENSE | 1 +
crates/storage-sqlite/NOTICE | 1 +
crates/storage/Cargo.toml | 3 +
crates/storage/LICENSE | 1 +
crates/storage/NOTICE | 1 +
devtools/check-crate-packages | 267 ++++++++++++++++
devtools/check_crate_packages_test.py | 396 ++++++++++++++++++++++++
docs/adr/0029-crates-io-packaging.md | 136 ++++++++
docs/adr/README.md | 1 +
packaging/README.md | 11 +-
61 files changed, 1993 insertions(+), 99 deletions(-)
create mode 120000 crates/app/LICENSE
create mode 120000 crates/app/NOTICE
create mode 120000 crates/auth/LICENSE
create mode 120000 crates/auth/NOTICE
create mode 120000 crates/backup-store/LICENSE
create mode 120000 crates/backup-store/NOTICE
create mode 120000 crates/bin/LICENSE
create mode 120000 crates/bin/NOTICE
create mode 100644 crates/bin/README.md
create mode 120000 crates/cache/LICENSE
create mode 120000 crates/cache/NOTICE
create mode 120000 crates/config/LICENSE
create mode 120000 crates/config/NOTICE
create mode 120000 crates/core/LICENSE
create mode 120000 crates/core/NOTICE
create mode 120000 crates/engine/LICENSE
create mode 120000 crates/engine/NOTICE
create mode 120000 crates/server/LICENSE
create mode 120000 crates/server/NOTICE
create mode 120000 crates/storage-cassandra/LICENSE
create mode 120000 crates/storage-cassandra/NOTICE
create mode 120000 crates/storage-mongodb/LICENSE
create mode 120000 crates/storage-mongodb/NOTICE
create mode 120000 crates/storage-postgres/LICENSE
create mode 120000 crates/storage-postgres/NOTICE
create mode 120000 crates/storage-sqlite/LICENSE
create mode 120000 crates/storage-sqlite/NOTICE
create mode 120000 crates/storage/LICENSE
create mode 120000 crates/storage/NOTICE
create mode 100755 devtools/check-crate-packages
create mode 100644 devtools/check_crate_packages_test.py
create mode 100644 docs/adr/0029-crates-io-packaging.md
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index ef1b68230..b951c035e 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -105,13 +105,36 @@ jobs:
if: failure()
run: cat moto.log || true
+ # crates.io accepts only what `cargo package` can write. Each dependency
+ # needs a version, and the dev-dependency cycle between extenddb-engine and
+ # extenddb-storage-sqlite must stay out of the published manifests.
+ # --no-verify writes the packages without compiling them, in seconds, and
+ # runs no build script. devtools/check-crate-packages then checks the pins,
+ # the LICENSE and NOTICE files, the metadata and crates.io's size limits.
+ # Compiling each package from its tarball takes minutes. That build belongs
+ # to the crates.io release workflow (follow-up work), before the upload, and
+ # not to each PR.
+ package:
+ runs-on: extenddb_ubuntu-2404_4-core
+ timeout-minutes: 10
+ steps:
+ - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
+ with:
+ persist-credentials: false
+ - uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master
+ with:
+ toolchain: stable
+ - run: cargo package --workspace --no-verify --locked
+ - run: devtools/check-crate-packages
+
test:
runs-on: extenddb_ubuntu-2404_4-core
timeout-minutes: 15
- needs: run-tests
+ needs: [run-tests, package]
if: always()
steps:
- run: |
- if [ "${{ needs.run-tests.result }}" != "success" ]; then
+ if [ "${{ needs.run-tests.result }}" != "success" ] \
+ || [ "${{ needs.package.result }}" != "success" ]; then
exit 1
fi
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index e321d02e0..f32f5ec26 100755
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -82,6 +82,22 @@ cargo clippy -- -W clippy::pedantic -W clippy::unwrap_used -W clippy::expect_use
- Address all clippy warnings or add `#[allow(...)]` with a justification comment.
- Avoid `.unwrap()` and `.expect()` in library crates — use `?` or explicit error handling.
+### Dependencies between workspace crates
+
+A crate depends on another workspace crate with `{ workspace = true }`. The
+root `[workspace.dependencies]` pins each workspace crate to the workspace
+version exactly (`=X.Y.Z`), because the crates publish to crates.io together
+([ADR-0029](docs/adr/0029-crates-io-packaging.md)). Declare a dev-dependency
+on a workspace crate by path with no version, for example
+`extenddb-storage = { path = "../storage", features = ["test-util"] }`, so
+that cargo drops it from the published manifest. The `package` job of the
+Test workflow checks both rules. To run the same check locally, on a tree
+with uncommitted changes too (the script needs Python 3.11 or later):
+
+```bash
+cargo package --workspace --no-verify --locked --allow-dirty && devtools/check-crate-packages
+```
+
## When you need an ADR or an RFC
ExtendDB uses two lightweight processes to keep core decisions reviewable:
diff --git a/Cargo.lock b/Cargo.lock
index faabdd021..c052ecd60 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -1016,9 +1016,9 @@ checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
[[package]]
name = "chacha20"
-version = "0.10.1"
+version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81"
+checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06"
dependencies = [
"cfg-if",
"cpufeatures 0.3.0",
@@ -1869,6 +1869,7 @@ dependencies = [
"base64 0.22.1",
"bcrypt",
"bigdecimal",
+ "cassandra-protocol",
"cdrs-tokio",
"chrono",
"crc32fast",
@@ -2018,7 +2019,7 @@ checksum = "da0e4dd2a88388a1f4ccc7c9ce104604dab68d9f408dc34cd45823d5a9069095"
dependencies = [
"futures-core",
"futures-sink",
- "spin 0.9.8",
+ "spin 0.9.9",
]
[[package]]
@@ -2865,7 +2866,7 @@ version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
dependencies = [
- "spin 0.9.8",
+ "spin 0.9.9",
]
[[package]]
@@ -4214,9 +4215,9 @@ dependencies = [
[[package]]
name = "spin"
-version = "0.9.8"
+version = "0.9.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67"
+checksum = "3763264f6b73151db08c50ff20d7d8a0b8796e021cdea7ceedad07b80155fa0e"
dependencies = [
"lock_api",
]
diff --git a/Cargo.toml b/Cargo.toml
index 2ee723c20..9c0c459f0 100755
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,7 +1,12 @@
# Copyright 2026 ExtendDB contributors
# SPDX-License-Identifier: Apache-2.0
[workspace]
-resolver = "2"
+# Resolver 3 prefers dependency versions whose rust-version fits ours. The
+# published crates use it too. (With resolver 2, cargo writes
+# `resolver = "2"` into each of them.) Without `--locked`, `cargo install`
+# then resolves for Rust 1.88 and does not fail on a dependency that needs a
+# newer Rust.
+resolver = "3"
members = [
"crates/core",
"crates/cache",
@@ -24,22 +29,32 @@ version = "0.1.14"
edition = "2024"
rust-version = "1.88"
license = "Apache-2.0"
+repository = "https://github.com/ExtendDB/extenddb"
+homepage = "https://extenddb.org/"
[workspace.dependencies]
-# Internal crates
-extenddb-core = { path = "crates/core" }
-extenddb-cache = { path = "crates/cache" }
-extenddb-backup-store = { path = "crates/backup-store" }
-extenddb-engine = { path = "crates/engine" }
-extenddb-storage = { path = "crates/storage" }
-extenddb-config = { path = "crates/config" }
-extenddb-storage-cassandra = { path = "crates/storage-cassandra" }
-extenddb-storage-mongodb = { path = "crates/storage-mongodb" }
-extenddb-storage-postgres = { path = "crates/storage-postgres" }
-extenddb-storage-sqlite = { path = "crates/storage-sqlite" }
-extenddb-auth = { path = "crates/auth" }
-extenddb-server = { path = "crates/server" }
-extenddb-app = { path = "crates/app" }
+# Internal crates. crates.io needs a version on each of them. Each version is
+# an exact pin to the workspace version, because the crates change their APIs
+# between patch releases. A crate of one release must never resolve against a
+# crate of another. A release bump changes each pin together with `version`
+# above. A pin left at the old version makes every cargo command that
+# resolves dependencies fail, and `devtools/check-crate-packages` fails on a
+# pin that is not exact.
+# Internal dev-dependencies are declared by path in each crate instead, with
+# no version, so that cargo drops them from the published manifests.
+extenddb-core = { path = "crates/core", version = "=0.1.14" }
+extenddb-cache = { path = "crates/cache", version = "=0.1.14" }
+extenddb-backup-store = { path = "crates/backup-store", version = "=0.1.14" }
+extenddb-engine = { path = "crates/engine", version = "=0.1.14" }
+extenddb-storage = { path = "crates/storage", version = "=0.1.14" }
+extenddb-config = { path = "crates/config", version = "=0.1.14" }
+extenddb-storage-cassandra = { path = "crates/storage-cassandra", version = "=0.1.14" }
+extenddb-storage-mongodb = { path = "crates/storage-mongodb", version = "=0.1.14" }
+extenddb-storage-postgres = { path = "crates/storage-postgres", version = "=0.1.14" }
+extenddb-storage-sqlite = { path = "crates/storage-sqlite", version = "=0.1.14" }
+extenddb-auth = { path = "crates/auth", version = "=0.1.14" }
+extenddb-server = { path = "crates/server", version = "=0.1.14" }
+extenddb-app = { path = "crates/app", version = "=0.1.14" }
# Serialization
serde = { version = "1", features = ["derive"] }
@@ -70,7 +85,13 @@ urlencoding = { version = "2.1" }
moka = { version = "0.12", features = ["future"] }
# Database
-cdrs-tokio = { version = "9.0", features = ["derive"] }
+# cdrs-tokio stays on 9.0.x and cassandra-protocol on 4.0.x. cassandra-protocol
+# 4.1.0 moved its public BigInt to num-bigint 0.5, while bigdecimal 0.4 uses
+# num-bigint 0.4. extenddb-storage-cassandra then fails to compile. The pins
+# keep a fresh resolution, such as `cargo install` without `--locked`, on
+# versions that build.
+cdrs-tokio = { version = "~9.0.2", features = ["derive"] }
+cassandra-protocol = "~4.0.1"
sqlx = { version = "0.8", default-features = false, features = ["runtime-tokio", "tls-rustls-aws-lc-rs", "postgres", "json", "time", "uuid", "bigdecimal", "derive"] }
# SQLite's C API, for the online backup API that sqlx does not expose. The
# version is the one sqlx-sqlite depends on: the crate declares
diff --git a/SOFTWARE-LICENSE-NOTICES-CASSANDRA.html b/SOFTWARE-LICENSE-NOTICES-CASSANDRA.html
index 8a5f1b144..73ced0490 100644
--- a/SOFTWARE-LICENSE-NOTICES-CASSANDRA.html
+++ b/SOFTWARE-LICENSE-NOTICES-CASSANDRA.html
@@ -1939,6 +1939,225 @@
Used by:
Copyright [yyyy] [name of copyright owner]
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+
+
+ Apache License 2.0
+ Used by:
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright ©2026 Amazon.com Inc. or Affiliates (“Amazon”).
+
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -6261,7 +6480,7 @@ Used by:
+
+ Apache License 2.0
+ Used by:
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright ©2026 Amazon.com Inc. or Affiliates (“Amazon”).
+
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -7720,17 +7939,6 @@ Used by:
Apache License 2.0
Used by:
The MIT License (MIT)
diff --git a/SOFTWARE-LICENSE-NOTICES-MONGODB.html b/SOFTWARE-LICENSE-NOTICES-MONGODB.html
index 7920a20c8..ac2ff9fe8 100644
--- a/SOFTWARE-LICENSE-NOTICES-MONGODB.html
+++ b/SOFTWARE-LICENSE-NOTICES-MONGODB.html
@@ -1732,6 +1732,225 @@ Used by:
Copyright [yyyy] [name of copyright owner]
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+
+
+ Apache License 2.0
+ Used by:
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright ©2026 Amazon.com Inc. or Affiliates (“Amazon”).
+
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -6061,7 +6280,7 @@ Used by:
block-buffer 0.10.4
block-buffer 0.12.1
blowfish 0.9.1
- chacha20 0.10.1
+ chacha20 0.10.2
cipher 0.4.4
const-oid 0.10.2
cpufeatures 0.2.17
@@ -7538,17 +7757,6 @@ Used by:
Apache License 2.0
Used by:
- - extenddb-app 0.1.14
- - extenddb-auth 0.1.14
- - extenddb-backup-store 0.1.14
- - extenddb 0.1.14
- - extenddb-cache 0.1.14
- - extenddb-config 0.1.14
- - extenddb-core 0.1.14
- - extenddb-engine 0.1.14
- - extenddb-server 0.1.14
- - extenddb-storage 0.1.14
- - extenddb-storage-mongodb 0.1.14
- allocator-api2 0.2.21
- anyhow 1.0.103
- arraydeque 0.5.1
diff --git a/SOFTWARE-LICENSE-NOTICES.html b/SOFTWARE-LICENSE-NOTICES.html
index 9103a0534..1b429178d 100644
--- a/SOFTWARE-LICENSE-NOTICES.html
+++ b/SOFTWARE-LICENSE-NOTICES.html
@@ -1730,6 +1730,225 @@ Used by:
Copyright [yyyy] [name of copyright owner]
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+
+ -
+
Apache License 2.0
+ Used by:
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright ©2026 Amazon.com Inc. or Affiliates (“Amazon”).
+
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
@@ -7302,17 +7521,6 @@ Used by:
Apache License 2.0
Used by:
- - extenddb-app 0.1.14
- - extenddb-auth 0.1.14
- - extenddb-backup-store 0.1.14
- - extenddb 0.1.14
- - extenddb-cache 0.1.14
- - extenddb-config 0.1.14
- - extenddb-core 0.1.14
- - extenddb-engine 0.1.14
- - extenddb-server 0.1.14
- - extenddb-storage 0.1.14
- - extenddb-storage-postgres 0.1.14
- allocator-api2 0.2.21
- anyhow 1.0.103
- arraydeque 0.5.1
diff --git a/crates/app/Cargo.toml b/crates/app/Cargo.toml
index fef29af1e..a31527d90 100644
--- a/crates/app/Cargo.toml
+++ b/crates/app/Cargo.toml
@@ -6,6 +6,9 @@ version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+description = "Command-line interface of the ExtendDB server, shared by every backend build. Internal crate of the extenddb binary, with no API stability promise."
+repository.workspace = true
+homepage.workspace = true
[features]
# Developer mode: plain HTTP on loopback, open authorization (SigV4 still
diff --git a/crates/app/LICENSE b/crates/app/LICENSE
new file mode 120000
index 000000000..30cff7403
--- /dev/null
+++ b/crates/app/LICENSE
@@ -0,0 +1 @@
+../../LICENSE
\ No newline at end of file
diff --git a/crates/app/NOTICE b/crates/app/NOTICE
new file mode 120000
index 000000000..fb376cfaa
--- /dev/null
+++ b/crates/app/NOTICE
@@ -0,0 +1 @@
+../../NOTICE
\ No newline at end of file
diff --git a/crates/auth/Cargo.toml b/crates/auth/Cargo.toml
index a71126ef2..b936b9e52 100755
--- a/crates/auth/Cargo.toml
+++ b/crates/auth/Cargo.toml
@@ -6,6 +6,9 @@ version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+description = "SigV4 authentication and IAM-compatible authorization of the ExtendDB server. Internal crate of the extenddb binary, with no API stability promise."
+repository.workspace = true
+homepage.workspace = true
[dependencies]
extenddb-core = { workspace = true }
@@ -23,7 +26,7 @@ tracing = { workspace = true }
zeroize = { workspace = true }
[dev-dependencies]
-extenddb-cache = { workspace = true, features = ["test-util"] }
+extenddb-cache = { path = "../cache", features = ["test-util"] }
tokio = { workspace = true, features = ["test-util", "macros", "rt-multi-thread"] }
# The SigV4 differential tests (tests/sigv4_differential) sign requests with
# the SDK's signer and generate them from a fixed seed.
diff --git a/crates/auth/LICENSE b/crates/auth/LICENSE
new file mode 120000
index 000000000..30cff7403
--- /dev/null
+++ b/crates/auth/LICENSE
@@ -0,0 +1 @@
+../../LICENSE
\ No newline at end of file
diff --git a/crates/auth/NOTICE b/crates/auth/NOTICE
new file mode 120000
index 000000000..fb376cfaa
--- /dev/null
+++ b/crates/auth/NOTICE
@@ -0,0 +1 @@
+../../NOTICE
\ No newline at end of file
diff --git a/crates/backup-store/Cargo.toml b/crates/backup-store/Cargo.toml
index 12723cb46..73be71b87 100644
--- a/crates/backup-store/Cargo.toml
+++ b/crates/backup-store/Cargo.toml
@@ -6,7 +6,9 @@ version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
-description = "Backup object store abstraction with filesystem and S3 implementations."
+description = "Backup object store of the ExtendDB server, with filesystem and S3 implementations. Internal crate of the extenddb binary, with no API stability promise."
+repository.workspace = true
+homepage.workspace = true
[dependencies]
aws-config = { workspace = true }
diff --git a/crates/backup-store/LICENSE b/crates/backup-store/LICENSE
new file mode 120000
index 000000000..30cff7403
--- /dev/null
+++ b/crates/backup-store/LICENSE
@@ -0,0 +1 @@
+../../LICENSE
\ No newline at end of file
diff --git a/crates/backup-store/NOTICE b/crates/backup-store/NOTICE
new file mode 120000
index 000000000..fb376cfaa
--- /dev/null
+++ b/crates/backup-store/NOTICE
@@ -0,0 +1 @@
+../../NOTICE
\ No newline at end of file
diff --git a/crates/backup-store/README.md b/crates/backup-store/README.md
index ae741ecfa..d1192bb95 100644
--- a/crates/backup-store/README.md
+++ b/crates/backup-store/README.md
@@ -349,7 +349,7 @@ AWS_REGION=us-east-1 \
cargo test -p extenddb-backup-store
```
-Another S3-compatible server (MinIO built from source, Garage, LocalStack)
+Another S3-compatible server (MinIO built from source, or Garage)
should work the same way through the endpoint and credential variables,
but none has been run; see the checksum note in "Behavior". What moto
cannot produce is synthesised in the suite (a 301, a 403 with its error
diff --git a/crates/backup-store/src/config.rs b/crates/backup-store/src/config.rs
index 927329603..59bcc0da9 100644
--- a/crates/backup-store/src/config.rs
+++ b/crates/backup-store/src/config.rs
@@ -22,7 +22,7 @@
//! bucket = "my-extenddb-backups" # required when kind = "s3"; must not be empty
//! prefix = "extenddb/" # optional; at most 512 bytes
//! region = "us-east-1" # optional; SDK chain otherwise
-//! endpoint = "http://127.0.0.1:9000" # optional; moto, Garage, LocalStack, or any S3-compatible endpoint
+//! endpoint = "http://127.0.0.1:9000" # optional; moto, Garage, or any S3-compatible endpoint
//! allow_insecure_endpoint = false # an http:// endpoint must be loopback unless true
//! force_path_style = true # false when absent; an http:// endpoint needs true
//! upload_part_size_bytes = 8388608 # 5 MiB to 5 GiB (S3's part size limits)
diff --git a/crates/bin/Cargo.toml b/crates/bin/Cargo.toml
index df58be389..f1c1148c0 100755
--- a/crates/bin/Cargo.toml
+++ b/crates/bin/Cargo.toml
@@ -6,6 +6,12 @@ version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+description = "A DynamoDB-compatible API server that stores data in PostgreSQL, MongoDB, Cassandra or SQLite."
+repository.workspace = true
+homepage.workspace = true
+readme = "README.md"
+keywords = ["dynamodb", "database", "nosql", "server", "postgresql"]
+categories = ["database-implementations", "command-line-utilities"]
[[bin]]
name = "extenddb"
@@ -45,4 +51,4 @@ extenddb-config = { workspace = true }
sqlx = { workspace = true }
tokio = { workspace = true }
uuid = { workspace = true }
-extenddb-storage = { workspace = true, features = ["test-util"] }
+extenddb-storage = { path = "../storage", features = ["test-util"] }
diff --git a/crates/bin/LICENSE b/crates/bin/LICENSE
new file mode 120000
index 000000000..30cff7403
--- /dev/null
+++ b/crates/bin/LICENSE
@@ -0,0 +1 @@
+../../LICENSE
\ No newline at end of file
diff --git a/crates/bin/NOTICE b/crates/bin/NOTICE
new file mode 120000
index 000000000..fb376cfaa
--- /dev/null
+++ b/crates/bin/NOTICE
@@ -0,0 +1 @@
+../../NOTICE
\ No newline at end of file
diff --git a/crates/bin/README.md b/crates/bin/README.md
new file mode 100644
index 000000000..e1e898e91
--- /dev/null
+++ b/crates/bin/README.md
@@ -0,0 +1,91 @@
+# ExtendDB
+
+> **ExtendDB is an independent open source project managed by Amazon Web Services. It is not Amazon DynamoDB and does
+> not contain any DynamoDB source code.** "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room
+> implementation that speaks the DynamoDB wire protocol. Behavioral differences from the real service are documented
+> in [Differences from DynamoDB](https://github.com/ExtendDB/extenddb/blob/main/docs/differences-from-dynamodb.md).
+
+ExtendDB is a DynamoDB-compatible API server. Any AWS SDK, CLI or tool that works with DynamoDB works with ExtendDB.
+This crate builds the `extenddb` binary.
+
+## Install
+
+You need Rust 1.88 or later and a C compiler. Each build contains one storage backend, which a Cargo feature selects:
+
+| Backend | Command |
+|---|---|
+| PostgreSQL (the default) | `cargo install extenddb --locked` |
+| MongoDB | `cargo install extenddb --locked --no-default-features --features mongodb` |
+| Apache Cassandra (experimental) | `cargo install extenddb --locked --no-default-features --features cassandra` |
+| SQLite dev server | `cargo install extenddb --locked --no-default-features --features sqlite,dev-mode` |
+
+Every build installs a binary named `extenddb`, so you can have one of them installed at a time.
+
+With `--locked`, Cargo builds from the `Cargo.lock` inside the crate, which holds the dependency versions that the
+project tests. Without it, Cargo resolves every dependency again and can choose versions that the project never
+tested.
+
+## Run the dev server
+
+The SQLite dev build needs no database and no configuration file:
+
+```bash
+extenddb serve --foreground
+```
+
+It serves plain HTTP on `127.0.0.1:18443`. It refuses a bind address other than loopback unless the environment sets
+`EXTENDDB_DEV_ALLOW_ANY_BIND=1`, as the dev container image does for its own network.
+
+When it starts with no database, it creates `extenddb.sqlite` and a `backups/` directory in the working directory. It
+also adds an `admin` user to the database and prints the admin password. At startup it prints the AWS example key pair,
+which it always accepts:
+
+```bash
+AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE \
+AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
+aws dynamodb list-tables --region us-east-1 --endpoint-url http://127.0.0.1:18443
+```
+
+The dev build has no TLS, and its authorization is open: a request signed with any key that the server holds can run
+every operation. Use it for local development and tests only, and never store data in it that you need to keep
+private.
+
+Without `--foreground`, `extenddb serve` runs as a daemon and writes its log to syslog. Daemon mode works on Unix
+systems only, so on Windows always pass `--foreground`.
+
+## Run a server build
+
+The PostgreSQL, MongoDB and Cassandra builds each need a running database:
+
+- PostgreSQL 14 or later
+- MongoDB 7.0 or later, configured as a replica set
+- Apache Cassandra 4.1 or later
+
+Run `extenddb init` once to create the catalog, the admin user, a TLS certificate and `extenddb.toml`. Then run
+`extenddb serve`, which listens with TLS on port 18443.
+[Getting started](https://github.com/ExtendDB/extenddb/blob/main/docs/getting-started.md) describes each step and
+each backend's options.
+
+## Other ways to run ExtendDB
+
+- Docker images, with no Rust toolchain needed:
+ [`extenddb/extenddb-postgres`](https://hub.docker.com/r/extenddb/extenddb-postgres),
+ [`extenddb/extenddb-mongodb`](https://hub.docker.com/r/extenddb/extenddb-mongodb) and
+ [`extenddb/extenddb-dev`](https://hub.docker.com/r/extenddb/extenddb-dev).
+- [`@extenddb/dev`](https://www.npmjs.com/package/@extenddb/dev) on npm starts a prebuilt dev server from a Node.js
+ test suite.
+
+The [documentation](https://github.com/ExtendDB/extenddb/tree/main/docs) covers configuration, operations and the API
+surface.
+
+## The library crates
+
+The `extenddb-*` library crates on crates.io are the parts of this binary. Their Rust APIs carry no stability promise,
+and any release can change them. Each release publishes every crate at the same version, and each crate requires its
+ExtendDB dependencies at exactly that version.
+
+## License
+
+ExtendDB is licensed under the Apache License 2.0. See
+[LICENSE](https://github.com/ExtendDB/extenddb/blob/main/LICENSE) and
+[NOTICE](https://github.com/ExtendDB/extenddb/blob/main/NOTICE).
diff --git a/crates/bin/build.rs b/crates/bin/build.rs
index 302d0118b..e973f4375 100755
--- a/crates/bin/build.rs
+++ b/crates/bin/build.rs
@@ -7,6 +7,8 @@
//! time so `extenddb version` can report them without runtime git access.
use std::env;
+use std::fs;
+use std::path::Path;
use std::process::Command;
use std::time::{SystemTime, UNIX_EPOCH};
@@ -14,12 +16,15 @@ fn main() {
println!("cargo:rerun-if-env-changed=EXTENDDB_GIT_HASH");
println!("cargo:rerun-if-env-changed=EXTENDDB_BUILD_TIME");
- // Git commit hash (short). Container builds exclude .git from the context,
- // so the release workflow injects EXTENDDB_GIT_HASH explicitly. Local builds
- // retain the convenient git fallback.
+ // Git commit hash. Container builds exclude .git from the context, so the
+ // release workflow injects EXTENDDB_GIT_HASH (the full SHA) explicitly. A
+ // build of the published crate has no git repository either, so the
+ // commit `cargo package` recorded is read next. Local builds retain the
+ // convenient git fallback (short form).
let git_hash = env::var("EXTENDDB_GIT_HASH")
.ok()
.filter(|value| !value.trim().is_empty())
+ .or_else(packaged_commit)
.unwrap_or_else(|| {
Command::new("git")
.args(["rev-parse", "--short", "HEAD"])
@@ -78,3 +83,31 @@ fn main() {
// file in the package changes, while the env directives above make metadata
// overrides participate in Cargo's normal rebuild tracking.
}
+
+/// The commit of a build from the published crate.
+///
+/// `cargo package` writes `.cargo_vcs_info.json` beside the manifest, with
+/// `"sha1": "<40 hex digits>"` for the packaged commit and `"dirty": true`
+/// when the tree had uncommitted changes. The file exists only in a packaged
+/// crate, so a build in the repository returns `None` here and falls back to
+/// git. A file without a well-formed SHA also returns `None`.
+fn packaged_commit() -> Option {
+ let dir = env::var_os("CARGO_MANIFEST_DIR")?;
+ let text = fs::read_to_string(Path::new(&dir).join(".cargo_vcs_info.json")).ok()?;
+ let after_key = &text[text.find("\"sha1\"")? + "\"sha1\"".len()..];
+ let value = &after_key[after_key.find('"')? + 1..];
+ let sha = value.get(..40)?;
+ if !sha.bytes().all(|b| b.is_ascii_hexdigit()) || !value[40..].starts_with('"') {
+ return None;
+ }
+ let dirty = text.find("\"dirty\"").is_some_and(|at| {
+ let rest = text[at + "\"dirty\"".len()..].trim_start();
+ rest.strip_prefix(':')
+ .is_some_and(|value| value.trim_start().starts_with("true"))
+ });
+ Some(if dirty {
+ format!("{sha}-dirty")
+ } else {
+ sha.to_owned()
+ })
+}
diff --git a/crates/bin/src/main.rs b/crates/bin/src/main.rs
index 15876a052..46b78f315 100755
--- a/crates/bin/src/main.rs
+++ b/crates/bin/src/main.rs
@@ -28,7 +28,7 @@
compile_error!(
"the `postgres`, `mongodb`, `sqlite`, and `cassandra` features are mutually exclusive: \
a thin bin installs exactly one backend (e.g. build the MongoDB binary with \
- `--no-default-features --features mongodb)"
+ `--no-default-features --features mongodb`)"
);
#[cfg(not(any(
diff --git a/crates/cache/Cargo.toml b/crates/cache/Cargo.toml
index 3b180df9d..a4d0db686 100644
--- a/crates/cache/Cargo.toml
+++ b/crates/cache/Cargo.toml
@@ -6,7 +6,9 @@ version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
-description = "In-memory stale-while-revalidate cache primitive used by the auth and storage layers."
+description = "In-memory stale-while-revalidate cache used by the ExtendDB auth and server layers. Internal crate of the extenddb binary, with no API stability promise."
+repository.workspace = true
+homepage.workspace = true
[features]
# `test-util` exposes test-only helpers like `SwrCache::run_pending_tasks`
diff --git a/crates/cache/LICENSE b/crates/cache/LICENSE
new file mode 120000
index 000000000..30cff7403
--- /dev/null
+++ b/crates/cache/LICENSE
@@ -0,0 +1 @@
+../../LICENSE
\ No newline at end of file
diff --git a/crates/cache/NOTICE b/crates/cache/NOTICE
new file mode 120000
index 000000000..fb376cfaa
--- /dev/null
+++ b/crates/cache/NOTICE
@@ -0,0 +1 @@
+../../NOTICE
\ No newline at end of file
diff --git a/crates/config/Cargo.toml b/crates/config/Cargo.toml
index f9f086124..841ffb944 100644
--- a/crates/config/Cargo.toml
+++ b/crates/config/Cargo.toml
@@ -6,6 +6,9 @@ version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+description = "Configuration types and loading of the ExtendDB server. Internal crate of the extenddb binary, with no API stability promise."
+repository.workspace = true
+homepage.workspace = true
[features]
# Developer mode: `is_tls_enabled` reports plain HTTP regardless of the
diff --git a/crates/config/LICENSE b/crates/config/LICENSE
new file mode 120000
index 000000000..30cff7403
--- /dev/null
+++ b/crates/config/LICENSE
@@ -0,0 +1 @@
+../../LICENSE
\ No newline at end of file
diff --git a/crates/config/NOTICE b/crates/config/NOTICE
new file mode 120000
index 000000000..fb376cfaa
--- /dev/null
+++ b/crates/config/NOTICE
@@ -0,0 +1 @@
+../../NOTICE
\ No newline at end of file
diff --git a/crates/core/Cargo.toml b/crates/core/Cargo.toml
index 75495dda6..9b8e71939 100755
--- a/crates/core/Cargo.toml
+++ b/crates/core/Cargo.toml
@@ -6,6 +6,9 @@ version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+description = "Core types, validation, errors and limits of the ExtendDB server. Internal crate of the extenddb binary, with no API stability promise."
+repository.workspace = true
+homepage.workspace = true
[dependencies]
serde = { workspace = true }
diff --git a/crates/core/LICENSE b/crates/core/LICENSE
new file mode 120000
index 000000000..30cff7403
--- /dev/null
+++ b/crates/core/LICENSE
@@ -0,0 +1 @@
+../../LICENSE
\ No newline at end of file
diff --git a/crates/core/NOTICE b/crates/core/NOTICE
new file mode 120000
index 000000000..fb376cfaa
--- /dev/null
+++ b/crates/core/NOTICE
@@ -0,0 +1 @@
+../../NOTICE
\ No newline at end of file
diff --git a/crates/engine/Cargo.toml b/crates/engine/Cargo.toml
index bc33e7ff3..40682b374 100755
--- a/crates/engine/Cargo.toml
+++ b/crates/engine/Cargo.toml
@@ -6,6 +6,9 @@ version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+description = "Operation dispatch, input validation and DynamoDB error mapping of the ExtendDB server. Internal crate of the extenddb binary, with no API stability promise."
+repository.workspace = true
+homepage.workspace = true
[features]
default = ["partiql"]
@@ -44,4 +47,4 @@ futures = { workspace = true }
# `start_paused` for the tests of timers that must not depend on the wall clock.
tokio = { workspace = true, features = ["test-util"] }
# A storage engine for the tests that need an `OperationContext`.
-extenddb-storage-sqlite = { workspace = true }
+extenddb-storage-sqlite = { path = "../storage-sqlite" }
diff --git a/crates/engine/LICENSE b/crates/engine/LICENSE
new file mode 120000
index 000000000..30cff7403
--- /dev/null
+++ b/crates/engine/LICENSE
@@ -0,0 +1 @@
+../../LICENSE
\ No newline at end of file
diff --git a/crates/engine/NOTICE b/crates/engine/NOTICE
new file mode 120000
index 000000000..fb376cfaa
--- /dev/null
+++ b/crates/engine/NOTICE
@@ -0,0 +1 @@
+../../NOTICE
\ No newline at end of file
diff --git a/crates/server/Cargo.toml b/crates/server/Cargo.toml
index 02e5c992b..635ea2130 100755
--- a/crates/server/Cargo.toml
+++ b/crates/server/Cargo.toml
@@ -6,6 +6,9 @@ version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+description = "HTTP layer of the ExtendDB server, which serves the DynamoDB wire protocol. Internal crate of the extenddb binary, with no API stability promise."
+repository.workspace = true
+homepage.workspace = true
[features]
mongodb-test-hooks = []
@@ -60,11 +63,11 @@ syslog-tracing = { workspace = true }
libc = { workspace = true }
[dev-dependencies]
-extenddb-cache = { workspace = true, features = ["test-util"] }
+extenddb-cache = { path = "../cache", features = ["test-util"] }
# The login tests count bcrypt checks with `count_verifications` against an
# in-memory SQLite catalog.
-extenddb-storage = { workspace = true, features = ["test-util"] }
-extenddb-storage-sqlite = { workspace = true }
+extenddb-storage = { path = "../storage", features = ["test-util"] }
+extenddb-storage-sqlite = { path = "../storage-sqlite" }
sqlx = { workspace = true, features = ["sqlite"] }
tempfile = "3"
tokio = { workspace = true, features = ["test-util"] }
diff --git a/crates/server/LICENSE b/crates/server/LICENSE
new file mode 120000
index 000000000..30cff7403
--- /dev/null
+++ b/crates/server/LICENSE
@@ -0,0 +1 @@
+../../LICENSE
\ No newline at end of file
diff --git a/crates/server/NOTICE b/crates/server/NOTICE
new file mode 120000
index 000000000..fb376cfaa
--- /dev/null
+++ b/crates/server/NOTICE
@@ -0,0 +1 @@
+../../NOTICE
\ No newline at end of file
diff --git a/crates/server/src/serve.rs b/crates/server/src/serve.rs
index 8505c0b75..64f015b1b 100644
--- a/crates/server/src/serve.rs
+++ b/crates/server/src/serve.rs
@@ -47,7 +47,9 @@ pub struct BuildInfo {
/// Package version of the deployed binary (e.g. `env!("CARGO_PKG_VERSION")`
/// from the bin crate — not from a library crate, whose version may drift).
pub version: &'static str,
- /// Short git commit hash of the build (e.g. `env!("EXTENDDB_GIT_HASH")`).
+ /// Git commit hash of the build (e.g. `env!("EXTENDDB_GIT_HASH")`): the
+ /// full SHA in release images and in builds from crates.io, the short form
+ /// in local builds.
pub git_hash: &'static str,
/// Build timestamp (e.g. `env!("EXTENDDB_BUILD_TIME")`).
pub build_time: &'static str,
diff --git a/crates/storage-cassandra/Cargo.toml b/crates/storage-cassandra/Cargo.toml
index ad9bb4b7e..fdcebf3dd 100644
--- a/crates/storage-cassandra/Cargo.toml
+++ b/crates/storage-cassandra/Cargo.toml
@@ -6,6 +6,9 @@ version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+description = "Apache Cassandra storage backend of the ExtendDB server (experimental). Internal crate of the extenddb binary, with no API stability promise."
+repository.workspace = true
+homepage.workspace = true
[dependencies]
extenddb-core = { workspace = true }
@@ -27,6 +30,9 @@ bigdecimal = { workspace = true }
crc32fast = { workspace = true }
zeroize = { workspace = true }
cdrs-tokio = { workspace = true }
+# Not used in the code. It holds cdrs-tokio's protocol crate on 4.0.x. The
+# root Cargo.toml gives the reason.
+cassandra-protocol = { workspace = true }
chrono = { workspace = true }
[features]
@@ -39,6 +45,6 @@ test-util = ["extenddb-storage/test-util"]
test-hooks = ["extenddb-storage/test-hooks"]
[dev-dependencies]
-extenddb-storage = { workspace = true, features = ["test-util"] }
+extenddb-storage = { path = "../storage", features = ["test-util"] }
extenddb-storage-cassandra = { path = ".", features = ["test-util"] }
tokio = { workspace = true, features = ["test-util"] }
diff --git a/crates/storage-cassandra/LICENSE b/crates/storage-cassandra/LICENSE
new file mode 120000
index 000000000..30cff7403
--- /dev/null
+++ b/crates/storage-cassandra/LICENSE
@@ -0,0 +1 @@
+../../LICENSE
\ No newline at end of file
diff --git a/crates/storage-cassandra/NOTICE b/crates/storage-cassandra/NOTICE
new file mode 120000
index 000000000..fb376cfaa
--- /dev/null
+++ b/crates/storage-cassandra/NOTICE
@@ -0,0 +1 @@
+../../NOTICE
\ No newline at end of file
diff --git a/crates/storage-mongodb/Cargo.toml b/crates/storage-mongodb/Cargo.toml
index 510805d92..b34203e39 100644
--- a/crates/storage-mongodb/Cargo.toml
+++ b/crates/storage-mongodb/Cargo.toml
@@ -7,6 +7,9 @@ version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+description = "MongoDB storage backend of the ExtendDB server. Internal crate of the extenddb binary, with no API stability promise."
+repository.workspace = true
+homepage.workspace = true
[features]
# Enables deterministic coordination hooks used only by the MongoDB integration
@@ -75,7 +78,7 @@ proptest = "1"
tokio = { workspace = true, features = ["macros", "rt-multi-thread"] }
# The checksum test checks this backend's migration table with the framework's
# test helper.
-extenddb-storage = { workspace = true, features = ["test-util"] }
+extenddb-storage = { path = "../storage", features = ["test-util"] }
# The crate's snapshot tests need `snapshot-test-hooks` and its migration tests
# need `test-util`; a dev-dependency on itself turns both on for test builds
# only.
diff --git a/crates/storage-mongodb/LICENSE b/crates/storage-mongodb/LICENSE
new file mode 120000
index 000000000..30cff7403
--- /dev/null
+++ b/crates/storage-mongodb/LICENSE
@@ -0,0 +1 @@
+../../LICENSE
\ No newline at end of file
diff --git a/crates/storage-mongodb/NOTICE b/crates/storage-mongodb/NOTICE
new file mode 120000
index 000000000..fb376cfaa
--- /dev/null
+++ b/crates/storage-mongodb/NOTICE
@@ -0,0 +1 @@
+../../NOTICE
\ No newline at end of file
diff --git a/crates/storage-postgres/Cargo.toml b/crates/storage-postgres/Cargo.toml
index b2c47b590..fda1a04b6 100755
--- a/crates/storage-postgres/Cargo.toml
+++ b/crates/storage-postgres/Cargo.toml
@@ -6,6 +6,9 @@ version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+description = "PostgreSQL storage backend of the ExtendDB server. Internal crate of the extenddb binary, with no API stability promise."
+repository.workspace = true
+homepage.workspace = true
[features]
# Compiles the snapshot page-size setter the consistency test uses and the
@@ -22,14 +25,14 @@ test-util = ["extenddb-storage/test-util"]
tokio = { workspace = true, features = ["macros", "rt-multi-thread"] }
# The checksum test checks this backend's migration table with the framework's
# test helper.
-extenddb-storage = { workspace = true, features = ["test-util"] }
+extenddb-storage = { path = "../storage", features = ["test-util"] }
# The crate's own tests need `test-hooks` and `test-util`; a dev-dependency on
# itself turns both features on for test builds only.
extenddb-storage-postgres = { path = ".", features = ["test-hooks", "test-util"] }
# The store identity tests run the engine's checks over two engines on one
# scratch catalog and two filesystem stores.
-extenddb-engine = { workspace = true }
-extenddb-backup-store = { workspace = true }
+extenddb-engine = { path = "../engine" }
+extenddb-backup-store = { path = "../backup-store" }
tempfile = "3"
[dependencies]
diff --git a/crates/storage-postgres/LICENSE b/crates/storage-postgres/LICENSE
new file mode 120000
index 000000000..30cff7403
--- /dev/null
+++ b/crates/storage-postgres/LICENSE
@@ -0,0 +1 @@
+../../LICENSE
\ No newline at end of file
diff --git a/crates/storage-postgres/NOTICE b/crates/storage-postgres/NOTICE
new file mode 120000
index 000000000..fb376cfaa
--- /dev/null
+++ b/crates/storage-postgres/NOTICE
@@ -0,0 +1 @@
+../../NOTICE
\ No newline at end of file
diff --git a/crates/storage-sqlite/Cargo.toml b/crates/storage-sqlite/Cargo.toml
index 96a432a1a..e9a6ec2b8 100644
--- a/crates/storage-sqlite/Cargo.toml
+++ b/crates/storage-sqlite/Cargo.toml
@@ -6,6 +6,9 @@ version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+description = "SQLite storage backend of the ExtendDB server. Internal crate of the extenddb binary, with no API stability promise."
+repository.workspace = true
+homepage.workspace = true
[features]
# Compile the in-memory (`:memory:`) database in as the default storage path,
@@ -41,12 +44,12 @@ libc = { workspace = true }
libsqlite3-sys = { workspace = true }
[dev-dependencies]
-extenddb-storage = { workspace = true, features = ["test-util"] }
+extenddb-storage = { path = "../storage", features = ["test-util"] }
# `tests/backup_job_panic.rs` drives the engine's CreateBackup job and
# reconciler against a `SqliteEngine` (the engine depends on `extenddb-storage`,
# not on this crate, so the edge is acyclic).
-extenddb-engine = { workspace = true }
-extenddb-backup-store = { workspace = true }
+extenddb-engine = { path = "../engine" }
+extenddb-backup-store = { path = "../backup-store" }
# The slow-statement test sets sqlx's threshold, a `log` level, and prints
# every tracing event of a child process.
log = "0.4"
diff --git a/crates/storage-sqlite/LICENSE b/crates/storage-sqlite/LICENSE
new file mode 120000
index 000000000..30cff7403
--- /dev/null
+++ b/crates/storage-sqlite/LICENSE
@@ -0,0 +1 @@
+../../LICENSE
\ No newline at end of file
diff --git a/crates/storage-sqlite/NOTICE b/crates/storage-sqlite/NOTICE
new file mode 120000
index 000000000..fb376cfaa
--- /dev/null
+++ b/crates/storage-sqlite/NOTICE
@@ -0,0 +1 @@
+../../NOTICE
\ No newline at end of file
diff --git a/crates/storage/Cargo.toml b/crates/storage/Cargo.toml
index a024143df..f151c3206 100755
--- a/crates/storage/Cargo.toml
+++ b/crates/storage/Cargo.toml
@@ -6,6 +6,9 @@ version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
+description = "Storage traits that every ExtendDB backend implements, with the catalog migration framework and the storage code the backends share. Internal crate of the extenddb binary, with no API stability promise."
+repository.workspace = true
+homepage.workspace = true
[features]
# `test-util` exposes `migrations::check_released`, the checksum test each
diff --git a/crates/storage/LICENSE b/crates/storage/LICENSE
new file mode 120000
index 000000000..30cff7403
--- /dev/null
+++ b/crates/storage/LICENSE
@@ -0,0 +1 @@
+../../LICENSE
\ No newline at end of file
diff --git a/crates/storage/NOTICE b/crates/storage/NOTICE
new file mode 120000
index 000000000..fb376cfaa
--- /dev/null
+++ b/crates/storage/NOTICE
@@ -0,0 +1 @@
+../../NOTICE
\ No newline at end of file
diff --git a/devtools/check-crate-packages b/devtools/check-crate-packages
new file mode 100755
index 000000000..8eae2ed5a
--- /dev/null
+++ b/devtools/check-crate-packages
@@ -0,0 +1,267 @@
+#!/usr/bin/env python3
+# Copyright 2026 ExtendDB contributors
+# SPDX-License-Identifier: Apache-2.0
+"""Check the workspace crates for crates.io before anything is published.
+
+ cargo package --workspace --no-verify --locked
+ devtools/check-crate-packages [--root DIR] [--packages DIR]
+
+The first command writes one `.crate` file per published member into the
+`package` directory of cargo's target directory. On a tree with uncommitted
+changes, it also needs `--allow-dirty`. This script needs Python 3.11 or
+later. It reads the members from `cargo metadata`, so cargo resolves globs,
+`exclude`, inherited fields and renamed dependencies. It then fails, with
+one line per problem, unless all of the following hold for each member that
+crates.io receives (one without `publish = false`):
+
+- Each dependency on another workspace crate, normal or build, requires
+ that crate's version exactly (`=X.Y.Z`). A caret requirement would let a
+ fresh install combine crates of two releases. A pin left at the previous
+ version after a release bump also fails every cargo command that resolves
+ dependencies.
+- Each dev-dependency on a workspace crate has a path and no version, so
+ cargo drops it from the published manifest. With a version, the
+ dev-dependencies between extenddb-engine and extenddb-storage-sqlite form
+ a cycle, and crates.io can publish neither crate first.
+- The member has a package at its version.
+- The package holds LICENSE and NOTICE with the bytes of the root files.
+ Each crate links both from the root, and the Apache License 2.0 requires a
+ redistributor to pass both on.
+- The packaged manifest has a description, a license and a repository. It
+ requires each workspace crate it depends on at exactly that crate's
+ version, and it has no dev-dependency on a workspace crate.
+- The packaged manifest passes the field rules of crates.io's publish
+ handler: a description of at most 1000 bytes, at most 5 keywords of at
+ most 20 bytes in crates.io's keyword format, and at most 5 categories.
+ The category slugs are not checked against crates.io's list.
+- The package is inside the default limits in crates.io's
+ `src/config/publish_limits.rs`. The limits cover the upload size, the
+ unpacked size, the number of tarball entries, the manifest size, and the
+ number of dependencies and features.
+
+Exit 0 when every check passes, 1 otherwise.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+import tarfile
+from pathlib import Path
+
+try:
+ import tomllib
+except ModuleNotFoundError: # Python 3.10 and older
+ sys.exit("error: devtools/check-crate-packages needs Python 3.11 or later (for tomllib)")
+
+# crates.io's PublishLimitsConfig defaults (src/config/publish_limits.rs).
+UPLOAD_SIZE = 10 * 1024 * 1024
+UNPACK_SIZE = 512 * 1024 * 1024
+TARBALL_ENTRIES = 30_000
+METADATA_FILE_SIZE = 1024 * 1024
+DEPENDENCIES = 500
+FEATURES = 300
+
+# crates.io's publish handler (src/controllers/krate/publish.rs, and
+# Keyword::valid_name in crates/crates_io_database/src/models/keyword.rs).
+DESCRIPTION_LENGTH = 1000
+KEYWORDS = 5
+KEYWORD_LENGTH = 20
+CATEGORIES = 5
+
+DEPENDENCY_TABLES = ("dependencies", "build-dependencies", "dev-dependencies")
+
+
+def cargo_metadata(root: Path) -> dict:
+ """The workspace as cargo resolves it, without the dependency graph."""
+ cargo = os.environ.get("CARGO", "cargo")
+ try:
+ result = subprocess.run(
+ [cargo, "metadata", "--no-deps", "--format-version", "1"],
+ cwd=root, capture_output=True, text=True, check=False,
+ )
+ except OSError as error:
+ sys.exit(f"error: cannot run `{cargo} metadata` in {root}: {error}")
+ if result.returncode != 0:
+ sys.exit(f"error: `cargo metadata` failed in {root}:\n{result.stderr}")
+ return json.loads(result.stdout)
+
+
+def is_published(package: dict) -> bool:
+ """`publish = false` reaches cargo metadata as an empty registry list."""
+ return package.get("publish") != []
+
+
+def check(root: Path, packages: Path | None = None) -> list[str]:
+ """Return one message per problem; an empty list means every check passed."""
+ metadata = cargo_metadata(root)
+ members = {package["name"]: package for package in metadata["packages"]}
+ versions = {name: package["version"] for name, package in members.items()}
+ if packages is None:
+ packages = Path(metadata["target_directory"]) / "package"
+ problems: list[str] = []
+
+ for name, package in sorted(members.items()):
+ if not is_published(package):
+ continue
+ manifest = os.path.relpath(package["manifest_path"], root)
+ for dependency in package["dependencies"]:
+ target = dependency["name"]
+ if target not in members:
+ continue
+ key = dependency.get("rename") or target
+ kind = dependency.get("kind") or "normal"
+ if kind == "dev":
+ if dependency.get("path") is None or dependency["req"] != "*":
+ problems.append(
+ f"{manifest}: dev-dependency {key} requires {dependency['req']!r}; declare it "
+ f"by path with no version or workspace key, so cargo drops it from the "
+ f"published manifest"
+ )
+ elif dependency["req"] != f"={versions[target]}":
+ problems.append(
+ f"{manifest}: {kind} dependency {key} requires {dependency['req']!r}; it must be "
+ f'"={versions[target]}" (in the root [workspace.dependencies] for a '
+ f"`workspace = true` entry)"
+ )
+
+ try:
+ root_files = {f: (root / f).read_bytes() for f in ("LICENSE", "NOTICE")}
+ except OSError as error:
+ return problems + [f"cannot read the root LICENSE or NOTICE: {error}"]
+ for name, package in sorted(members.items()):
+ if not is_published(package):
+ continue
+ crate = packages / f"{name}-{package['version']}.crate"
+ if not crate.is_file():
+ problems.append(f"{crate}: missing; run `cargo package --workspace --no-verify --locked` first")
+ continue
+ problems.extend(check_package(crate, name, package["version"], versions, root_files))
+ return problems
+
+
+def dependency_tables(manifest: dict):
+ """Yield (table label, table) for each dependency table of a manifest,
+ including the target-specific ones."""
+ for kind in DEPENDENCY_TABLES:
+ if kind in manifest:
+ yield kind, manifest[kind]
+ for target, tables in manifest.get("target", {}).items():
+ for kind in DEPENDENCY_TABLES:
+ if kind in tables:
+ yield f"target.{target}.{kind}", tables[kind]
+
+
+def check_package(crate: Path, name: str, version: str, versions: dict[str, str], root_files: dict[str, bytes]) -> list[str]:
+ """Check one `.crate` file against the root files and crates.io's limits.
+
+ `versions` maps each workspace crate to its version."""
+ problems: list[str] = []
+ size = crate.stat().st_size
+ if size > UPLOAD_SIZE:
+ problems.append(f"{crate.name}: {size} bytes, over the crates.io upload limit of {UPLOAD_SIZE}")
+ prefix = f"{name}-{version}/"
+ with tarfile.open(crate, "r:gz") as tar:
+ entries = tar.getmembers()
+ if len(entries) > TARBALL_ENTRIES:
+ problems.append(f"{crate.name}: {len(entries)} entries, over the crates.io limit of {TARBALL_ENTRIES}")
+ unpacked = sum(entry.size for entry in entries)
+ if unpacked > UNPACK_SIZE:
+ problems.append(f"{crate.name}: {unpacked} bytes unpacked, over the crates.io limit of {UNPACK_SIZE}")
+ files = {entry.name: entry for entry in entries if entry.isfile()}
+ for filename, expected in root_files.items():
+ entry = files.get(prefix + filename)
+ if entry is None:
+ problems.append(f"{crate.name}: no {filename}; link crates//{filename} to the root file")
+ elif tar.extractfile(entry).read() != expected:
+ problems.append(f"{crate.name}: {filename} differs from the root {filename}")
+ entry = files.get(prefix + "Cargo.toml")
+ if entry is None:
+ return problems + [f"{crate.name}: no Cargo.toml"]
+ if entry.size > METADATA_FILE_SIZE:
+ problems.append(f"{crate.name}: Cargo.toml is {entry.size} bytes, over the crates.io limit of {METADATA_FILE_SIZE}")
+ manifest = tomllib.loads(tar.extractfile(entry).read().decode())
+ package = manifest.get("package", {})
+ for field in ("description", "license", "repository"):
+ if not str(package.get(field, "")).strip():
+ problems.append(f"{crate.name}: the packaged manifest has no {field}")
+ problems.extend(check_metadata(crate.name, package))
+ count = 0
+ for label, table in dependency_tables(manifest):
+ count += len(table)
+ for key, spec in table.items():
+ target = spec.get("package", key) if isinstance(spec, dict) else key
+ if target not in versions:
+ continue
+ if label.endswith("dev-dependencies"):
+ problems.append(
+ f"{crate.name}: [{label}] {key}: cargo kept this dev-dependency in the published "
+ f"manifest; declare it by path with no version"
+ )
+ continue
+ found = spec.get("version") if isinstance(spec, dict) else spec
+ if found != f"={versions[target]}":
+ problems.append(f'{crate.name}: [{label}] {key} requires {found!r}, not "={versions[target]}"')
+ if count > DEPENDENCIES:
+ problems.append(f"{crate.name}: {count} dependencies, over the crates.io limit of {DEPENDENCIES}")
+ features = manifest.get("features", {})
+ if len(features) > FEATURES:
+ problems.append(f"{crate.name}: {len(features)} features, over the crates.io limit of {FEATURES}")
+ for feature, enables in features.items():
+ if len(enables) > FEATURES:
+ problems.append(f"{crate.name}: feature {feature} enables {len(enables)} entries, over the crates.io limit of {FEATURES}")
+ return problems
+
+
+def valid_keyword(keyword: str) -> bool:
+ """crates.io's Keyword::valid_name: an ASCII letter or digit, then ASCII
+ letters, digits, `_`, `-` or `+`."""
+ def allowed(c: str) -> bool:
+ return c.isascii() and (c.isalnum() or c in "_-+")
+ return bool(keyword) and keyword[0].isascii() and keyword[0].isalnum() and all(allowed(c) for c in keyword[1:])
+
+
+def check_metadata(crate: str, package: dict) -> list[str]:
+ """The [package] fields crates.io refuses at upload. Lengths are in bytes,
+ as crates.io counts them. Category slugs are not checked against
+ crates.io's list here."""
+ problems: list[str] = []
+ description = str(package.get("description", ""))
+ if len(description.encode()) > DESCRIPTION_LENGTH:
+ problems.append(f"{crate}: the description is {len(description.encode())} bytes, over the crates.io limit of {DESCRIPTION_LENGTH}")
+ keywords = package.get("keywords", [])
+ if len(keywords) > KEYWORDS:
+ problems.append(f"{crate}: {len(keywords)} keywords, over the crates.io limit of {KEYWORDS}")
+ for keyword in keywords:
+ if len(keyword.encode()) > KEYWORD_LENGTH:
+ problems.append(f"{crate}: keyword {keyword!r} is over the crates.io limit of {KEYWORD_LENGTH} bytes")
+ elif not valid_keyword(keyword):
+ problems.append(f"{crate}: keyword {keyword!r} is not a valid crates.io keyword")
+ categories = package.get("categories", [])
+ if len(categories) > CATEGORIES:
+ problems.append(f"{crate}: {len(categories)} categories, over the crates.io limit of {CATEGORIES}")
+ return problems
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__.split("\n\n", 1)[0])
+ parser.add_argument("--root", type=Path, default=Path(__file__).resolve().parent.parent,
+ help="workspace root (default: this repository)")
+ parser.add_argument("--packages", type=Path,
+ help="directory of .crate files (default: the package directory of cargo's target directory)")
+ args = parser.parse_args()
+ problems = check(args.root.resolve(), args.packages)
+ for problem in problems:
+ print(f"error: {problem}")
+ if problems:
+ print(f"{len(problems)} problem(s) found")
+ return 1
+ print("ok: every published workspace crate is ready for crates.io")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/devtools/check_crate_packages_test.py b/devtools/check_crate_packages_test.py
new file mode 100644
index 000000000..4e12c23ff
--- /dev/null
+++ b/devtools/check_crate_packages_test.py
@@ -0,0 +1,396 @@
+# Copyright 2026 ExtendDB contributors
+# SPDX-License-Identifier: Apache-2.0
+"""Tests of devtools/check-crate-packages.
+
+Each test builds a two-crate cargo workspace and its `.crate` files in a
+temporary directory, then runs the script, which reads the workspace through
+`cargo metadata`. The script passes the complete workspace and names each
+problem a test introduces. The crates.io limits are tested by calling
+`check_package` with the limits lowered.
+
+ python3 -m unittest discover -s devtools -p '*_test.py'
+"""
+import importlib.util
+import io
+import os
+import subprocess
+import sys
+import tarfile
+import tempfile
+import textwrap
+import unittest
+from importlib.machinery import SourceFileLoader
+from pathlib import Path
+from unittest import mock
+
+SCRIPT = Path(__file__).resolve().parent / "check-crate-packages"
+VERSION = "1.2.3"
+
+ROOT_MANIFEST = f"""\
+[workspace]
+resolver = "3"
+members = ["crates/*"]
+
+[workspace.package]
+version = "{VERSION}"
+edition = "2021"
+
+[workspace.dependencies]
+example-a = {{ path = "crates/a", version = "={VERSION}" }}
+example-b = {{ path = "crates/b", version = "={VERSION}" }}
+"""
+
+MEMBER_A = """\
+[package]
+name = "example-a"
+version.workspace = true
+edition.workspace = true
+
+[dev-dependencies]
+example-b = { path = "../b" }
+"""
+
+MEMBER_B = """\
+[package]
+name = "example-b"
+version.workspace = true
+edition.workspace = true
+
+[dependencies]
+example-a = { workspace = true }
+"""
+
+
+def load_script():
+ """Import the script, which has no .py suffix, as a module."""
+ loader = SourceFileLoader("check_crate_packages", str(SCRIPT))
+ spec = importlib.util.spec_from_loader(loader.name, loader)
+ module = importlib.util.module_from_spec(spec)
+ loader.exec_module(module)
+ return module
+
+
+def packaged_manifest(name: str, extra: str = "", version: str = VERSION, **package) -> str:
+ """A manifest in the form `cargo package` writes. `package` overrides or,
+ with None, removes a [package] field; `extra` is appended as is."""
+ fields = {"name": name, "version": version, "description": "d", "license": "Apache-2.0",
+ "repository": "https://example.com"}
+ fields.update(package)
+ lines = ["[package]"] + [f'{k} = "{v}"' for k, v in fields.items() if v is not None]
+ return "\n".join(lines) + "\n" + textwrap.dedent(extra)
+
+
+B_DEPENDENCY = f'\n[dependencies.example-a]\nversion = "={VERSION}"\n'
+
+
+class CheckCratePackagesTest(unittest.TestCase):
+ def setUp(self) -> None:
+ self.tmp = tempfile.TemporaryDirectory()
+ self.root = Path(self.tmp.name)
+ self.packages = self.root / "target" / "package"
+ self.packages.mkdir(parents=True)
+ self.write("Cargo.toml", ROOT_MANIFEST)
+ self.member("a", MEMBER_A)
+ self.member("b", MEMBER_B)
+ self.write("LICENSE", "license text\n")
+ self.write("NOTICE", "notice text\n")
+ self.package("example-a", {"Cargo.toml": packaged_manifest("example-a")})
+ self.package("example-b", {"Cargo.toml": packaged_manifest("example-b", B_DEPENDENCY)})
+
+ def tearDown(self) -> None:
+ self.tmp.cleanup()
+
+ def write(self, relative: str, text: str) -> None:
+ path = self.root / relative
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(text)
+
+ def member(self, directory: str, manifest: str) -> None:
+ self.write(f"crates/{directory}/Cargo.toml", manifest)
+ self.write(f"crates/{directory}/src/lib.rs", "")
+
+ def package(self, name: str, files: dict, licenses: bool = True, version: str = VERSION,
+ into: Path | None = None) -> None:
+ """Write -.crate holding `files`, plus the root LICENSE
+ and NOTICE unless `licenses` is false."""
+ contents = dict(files)
+ if licenses:
+ contents.setdefault("LICENSE", (self.root / "LICENSE").read_bytes())
+ contents.setdefault("NOTICE", (self.root / "NOTICE").read_bytes())
+ directory = into or self.packages
+ directory.mkdir(parents=True, exist_ok=True)
+ with tarfile.open(directory / f"{name}-{version}.crate", "w:gz") as tar:
+ for filename, data in contents.items():
+ raw = data.encode() if isinstance(data, str) else data
+ info = tarfile.TarInfo(f"{name}-{version}/{filename}")
+ info.size = len(raw)
+ tar.addfile(info, io.BytesIO(raw))
+
+ def run_check(self, env: dict | None = None) -> tuple[int, str]:
+ environment = {k: v for k, v in os.environ.items() if k != "CARGO_TARGET_DIR"}
+ environment.update(env or {})
+ result = subprocess.run(
+ [sys.executable, str(SCRIPT), "--root", str(self.root)],
+ capture_output=True, text=True, env=environment, check=False,
+ )
+ return result.returncode, result.stdout + result.stderr
+
+ def assert_passes(self, env: dict | None = None) -> None:
+ code, output = self.run_check(env)
+ self.assertEqual(code, 0, output)
+ self.assertIn("ok: every published workspace crate is ready for crates.io", output)
+
+ def assert_fails_with(self, *fragments: str) -> None:
+ code, output = self.run_check()
+ self.assertEqual(code, 1, output)
+ for fragment in fragments:
+ self.assertIn(fragment, output)
+
+ # The workspace as cargo reads it.
+
+ def test_a_complete_workspace_passes(self) -> None:
+ self.assert_passes()
+
+ def test_a_workspace_pin_that_is_not_exact_fails(self) -> None:
+ self.write("Cargo.toml", ROOT_MANIFEST.replace(f'"crates/a", version = "={VERSION}"', f'"crates/a", version = "{VERSION}"'))
+ self.assert_fails_with(
+ f"crates/b/Cargo.toml: normal dependency example-a requires '^{VERSION}'", f'"={VERSION}"')
+
+ def test_a_workspace_pin_left_at_the_last_release_fails(self) -> None:
+ self.write("Cargo.toml", ROOT_MANIFEST.replace(f'"crates/a", version = "={VERSION}"', '"crates/a", version = "=1.2.2"'))
+ self.assert_fails_with("normal dependency example-a requires '=1.2.2'")
+
+ def test_a_renamed_dependency_is_checked_under_its_package_name(self) -> None:
+ self.member("b", MEMBER_B.replace("example-a = { workspace = true }",
+ f'alias = {{ package = "example-a", path = "../a", version = "{VERSION}" }}'))
+ self.assert_fails_with(f"normal dependency alias requires '^{VERSION}'")
+
+ def test_a_build_dependency_needs_the_exact_pin(self) -> None:
+ self.member("b", MEMBER_B + f'\n[build-dependencies]\nexample-a = {{ path = "../a", version = "{VERSION}" }}\n')
+ self.assert_fails_with(f"build dependency example-a requires '^{VERSION}'")
+
+ def test_an_internal_dev_dependency_through_the_workspace_fails(self) -> None:
+ self.member("a", MEMBER_A.replace('{ path = "../b" }', "{ workspace = true }"))
+ self.assert_fails_with(f"crates/a/Cargo.toml: dev-dependency example-b requires '={VERSION}'")
+
+ def test_an_internal_dev_dependency_with_a_version_fails(self) -> None:
+ self.member("a", MEMBER_A.replace('{ path = "../b" }', f'{{ path = "../b", version = "={VERSION}" }}'))
+ self.assert_fails_with("dev-dependency example-b requires")
+
+ def test_an_internal_dev_dependency_without_a_path_fails(self) -> None:
+ self.member("a", MEMBER_A.replace('{ path = "../b" }', '"*"'))
+ self.assert_fails_with("dev-dependency example-b requires '*'; declare it by path")
+
+ def test_a_target_specific_dev_dependency_is_checked(self) -> None:
+ self.member("a", MEMBER_A + "\n[target.'cfg(unix)'.dev-dependencies]\nexample-b = { workspace = true }\n")
+ self.assert_fails_with(f"dev-dependency example-b requires '={VERSION}'")
+
+ def test_a_missing_package_fails(self) -> None:
+ (self.packages / f"example-b-{VERSION}.crate").unlink()
+ self.assert_fails_with(f"example-b-{VERSION}.crate: missing")
+
+ def test_an_unpublished_member_needs_no_package_and_no_pins(self) -> None:
+ self.member("b", MEMBER_B.replace("edition.workspace = true", "edition.workspace = true\npublish = false")
+ .replace("{ workspace = true }", '{ path = "../a" }'))
+ (self.packages / f"example-b-{VERSION}.crate").unlink()
+ self.assert_passes()
+
+ def test_publish_inherited_from_the_workspace_is_read(self) -> None:
+ self.write("Cargo.toml", ROOT_MANIFEST.replace('edition = "2021"', 'edition = "2021"\npublish = false'))
+ self.member("b", MEMBER_B.replace("edition.workspace = true", "edition.workspace = true\npublish.workspace = true"))
+ (self.packages / f"example-b-{VERSION}.crate").unlink()
+ self.assert_passes()
+
+ def test_an_excluded_directory_is_not_a_member(self) -> None:
+ self.write("Cargo.toml", ROOT_MANIFEST.replace('members = ["crates/*"]', 'members = ["crates/*"]\nexclude = ["crates/x"]'))
+ self.write("crates/x/Cargo.toml", '[package]\nname = "example-x"\nversion = "0.1.0"\nedition = "2021"\n')
+ self.write("crates/x/src/lib.rs", "")
+ self.assert_passes()
+
+ def test_a_member_with_its_own_version_is_packaged_and_pinned_at_it(self) -> None:
+ self.write("Cargo.toml", ROOT_MANIFEST.replace(f'"crates/a", version = "={VERSION}"', '"crates/a", version = "=2.0.0"'))
+ self.member("a", MEMBER_A.replace("version.workspace = true", 'version = "2.0.0"'))
+ (self.packages / f"example-a-{VERSION}.crate").unlink()
+ self.package("example-a", {"Cargo.toml": packaged_manifest("example-a", version="2.0.0")}, version="2.0.0")
+ self.package("example-b", {"Cargo.toml": packaged_manifest("example-b", '\n[dependencies.example-a]\nversion = "=2.0.0"\n')})
+ self.assert_passes()
+
+ def test_cargo_target_dir_moves_the_packages(self) -> None:
+ moved = self.root / "elsewhere"
+ for crate in self.packages.iterdir():
+ moved.joinpath("package").mkdir(parents=True, exist_ok=True)
+ crate.rename(moved / "package" / crate.name)
+ self.assert_fails_with(f"example-a-{VERSION}.crate: missing")
+ self.assert_passes({"CARGO_TARGET_DIR": str(moved)})
+
+ # The packages.
+
+ def test_a_package_without_the_license_files_fails(self) -> None:
+ self.package("example-a", {"Cargo.toml": packaged_manifest("example-a")}, licenses=False)
+ self.assert_fails_with(f"example-a-{VERSION}.crate: no LICENSE", f"example-a-{VERSION}.crate: no NOTICE")
+
+ def test_a_license_file_that_differs_from_the_root_fails(self) -> None:
+ self.package("example-a", {"Cargo.toml": packaged_manifest("example-a"), "LICENSE": "other\n", "NOTICE": "other\n"})
+ self.assert_fails_with(f"example-a-{VERSION}.crate: LICENSE differs from the root LICENSE",
+ f"example-a-{VERSION}.crate: NOTICE differs from the root NOTICE")
+
+ def test_a_package_without_a_manifest_fails(self) -> None:
+ self.package("example-a", {"README.md": "r\n"})
+ self.assert_fails_with(f"example-a-{VERSION}.crate: no Cargo.toml")
+
+ def test_each_required_manifest_field_is_checked(self) -> None:
+ for field in ("description", "license", "repository"):
+ with self.subTest(field=field):
+ self.package("example-a", {"Cargo.toml": packaged_manifest("example-a", **{field: None})})
+ self.assert_fails_with(f"example-a-{VERSION}.crate: the packaged manifest has no {field}")
+
+ def test_each_packaged_dependency_table_needs_the_exact_pin(self) -> None:
+ tables = {
+ "dependencies": f'\n[dependencies.example-a]\nversion = "^{VERSION}"\n',
+ "build-dependencies": f'\n[build-dependencies.example-a]\nversion = "^{VERSION}"\n',
+ "target.cfg(unix).dependencies": f'\n[target."cfg(unix)".dependencies.example-a]\nversion = "^{VERSION}"\n',
+ "dependencies (renamed)": f'\n[dependencies.alias]\npackage = "example-a"\nversion = "^{VERSION}"\n',
+ "dependencies (string form)": f'\n[dependencies]\nexample-a = "^{VERSION}"\n',
+ }
+ for label, extra in tables.items():
+ with self.subTest(table=label):
+ self.package("example-b", {"Cargo.toml": packaged_manifest("example-b", extra)})
+ key = "alias" if "renamed" in label else "example-a"
+ table = label.split(" ")[0]
+ self.assert_fails_with(f"example-b-{VERSION}.crate: [{table}] {key} requires '^{VERSION}'")
+
+ def test_a_dev_dependency_cargo_kept_in_the_package_fails(self) -> None:
+ tables = {
+ "dev-dependencies": '\n[dev-dependencies.example-a]\nversion = "*"\n',
+ "target.cfg(unix).dev-dependencies": f'\n[target."cfg(unix)".dev-dependencies.example-a]\nversion = "={VERSION}"\n',
+ }
+ for label, extra in tables.items():
+ with self.subTest(table=label):
+ self.package("example-b", {"Cargo.toml": packaged_manifest("example-b", B_DEPENDENCY + extra)})
+ self.assert_fails_with(
+ f"example-b-{VERSION}.crate: [{label}] example-a: cargo kept this dev-dependency")
+
+ def test_a_blank_description_fails(self) -> None:
+ self.package("example-a", {"Cargo.toml": packaged_manifest("example-a", description=" ")})
+ self.assert_fails_with(f"example-a-{VERSION}.crate: the packaged manifest has no description")
+
+ def test_a_failing_cargo_metadata_fails(self) -> None:
+ self.write("Cargo.toml", "[workspace\n")
+ self.assert_fails_with("error: `cargo metadata` failed in")
+
+ def test_a_missing_root_fails_with_an_error_line(self) -> None:
+ self.root = self.root / "does-not-exist"
+ code, output = self.run_check()
+ self.assertEqual(code, 1, output)
+ self.assertIn("error: cannot run `cargo metadata`", output)
+ self.assertNotIn("Traceback", output)
+
+
+class LimitsTest(unittest.TestCase):
+ """check_package against crates.io's limits, lowered so a fixture can
+ cross each one."""
+
+ def setUp(self) -> None:
+ self.module = load_script()
+ self.tmp = tempfile.TemporaryDirectory()
+ self.crate = Path(self.tmp.name) / f"example-a-{VERSION}.crate"
+ self.root_files = {"LICENSE": b"l\n", "NOTICE": b"n\n"}
+
+ def tearDown(self) -> None:
+ self.tmp.cleanup()
+
+ def write_crate(self, manifest: str, extra_files: int = 0) -> None:
+ files = {"Cargo.toml": manifest.encode(), **self.root_files}
+ files.update({f"f{i}.rs": b"x" * 100 for i in range(extra_files)})
+ with tarfile.open(self.crate, "w:gz") as tar:
+ for filename, raw in files.items():
+ info = tarfile.TarInfo(f"example-a-{VERSION}/{filename}")
+ info.size = len(raw)
+ tar.addfile(info, io.BytesIO(raw))
+
+ def check(self, **limits) -> list[str]:
+ """check_package on the crate written last, with `limits` patched in."""
+ def run() -> list[str]:
+ return self.module.check_package(self.crate, "example-a", VERSION, {"example-a": VERSION}, self.root_files)
+
+ if not limits:
+ return run()
+ with mock.patch.multiple(self.module, **limits):
+ return run()
+
+ def problems(self, manifest: str, extra_files: int = 0, **limits) -> list[str]:
+ self.write_crate(manifest, extra_files)
+ return self.check(**limits)
+
+ def test_a_package_exactly_at_each_limit_passes_and_one_over_fails(self) -> None:
+ extra = ('\n[dependencies.x]\nversion = "1"\n\n[dev-dependencies.y]\nversion = "1"\n'
+ '\n[features]\none = ["two", "dep:x"]\ntwo = []\n')
+ manifest = packaged_manifest("example-a", extra)
+ self.write_crate(manifest, 3)
+ with tarfile.open(self.crate, "r:gz") as tar:
+ entries = tar.getmembers()
+ measured = {
+ "UPLOAD_SIZE": self.crate.stat().st_size,
+ "TARBALL_ENTRIES": len(entries),
+ "UNPACK_SIZE": sum(entry.size for entry in entries),
+ "METADATA_FILE_SIZE": len(manifest.encode()),
+ "DEPENDENCIES": 2,
+ # Two features, and feature `one` enables two entries.
+ "FEATURES": 2,
+ }
+ for limit, value in measured.items():
+ with self.subTest(limit=limit):
+ self.assertEqual(self.check(**{limit: value}), [])
+ self.assertNotEqual(self.check(**{limit: value - 1}), [])
+
+ def test_the_crates_io_field_rules_are_checked(self) -> None:
+ at_the_limits = {"description": "d" * 1000, "keywords": ["a", "b+c", "d_e", "f-g", "h" * 20],
+ "categories": ["a", "b", "c", "d", "e"]}
+ self.assertEqual(self.module.check_metadata("x.crate", at_the_limits), [])
+ cases = {
+ "the description is 1001 bytes": {"description": "d" * 1001},
+ "6 keywords, over": {"keywords": list("abcdef")},
+ "is over the crates.io limit of 20 bytes": {"keywords": ["k" * 21]},
+ "'-k' is not a valid crates.io keyword": {"keywords": ["-k"]},
+ "'a b' is not a valid crates.io keyword": {"keywords": ["a b"]},
+ "'\u00e9t\u00e9' is not a valid crates.io keyword": {"keywords": ["\u00e9t\u00e9"]},
+ "'caf\u00e9' is not a valid crates.io keyword": {"keywords": ["caf\u00e9"]},
+ "6 categories, over": {"categories": list("abcdef")},
+ }
+ for fragment, package in cases.items():
+ with self.subTest(rule=fragment):
+ problems = self.module.check_metadata("x.crate", package)
+ self.assertTrue(any(fragment in p for p in problems), problems)
+
+ def test_the_field_rules_run_on_each_package(self) -> None:
+ manifest = packaged_manifest("example-a", 'keywords = ["a", "b", "c", "d", "e", "f"]\n')
+ self.assertEqual(self.problems(manifest), [f"example-a-{VERSION}.crate: 6 keywords, over the crates.io limit of 5"])
+
+ def test_a_package_inside_every_limit_passes(self) -> None:
+ self.assertEqual(self.problems(packaged_manifest("example-a")), [])
+
+ def test_each_limit_is_enforced(self) -> None:
+ features = '\n[features]\none = ["two", "three"]\ntwo = []\nthree = []\n'
+ dependencies = '\n[dependencies.x]\nversion = "1"\n\n[dev-dependencies.y]\nversion = "1"\n'
+ cases = {
+ "upload limit": ({"UPLOAD_SIZE": 10}, "", 0),
+ "entries, over the crates.io limit of 2": ({"TARBALL_ENTRIES": 2}, "", 0),
+ "bytes unpacked": ({"UNPACK_SIZE": 500}, "", 10),
+ "Cargo.toml is": ({"METADATA_FILE_SIZE": 20}, "", 0),
+ "2 dependencies, over": ({"DEPENDENCIES": 1}, dependencies, 0),
+ "3 features, over": ({"FEATURES": 2}, features, 0),
+ }
+ for fragment, (limits, extra, files) in cases.items():
+ with self.subTest(limit=fragment):
+ problems = self.problems(packaged_manifest("example-a", extra), files, **limits)
+ self.assertTrue(any(fragment in p for p in problems), problems)
+
+ def test_the_entries_one_feature_enables_are_counted(self) -> None:
+ features = '\n[features]\none = ["two", "dep:x", "x/serde"]\ntwo = []\n'
+ problems = self.problems(packaged_manifest("example-a", features), FEATURES=2)
+ self.assertEqual(
+ problems, [f"example-a-{VERSION}.crate: feature one enables 3 entries, over the crates.io limit of 2"])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/docs/adr/0029-crates-io-packaging.md b/docs/adr/0029-crates-io-packaging.md
new file mode 100644
index 000000000..3db2c36e4
--- /dev/null
+++ b/docs/adr/0029-crates-io-packaging.md
@@ -0,0 +1,136 @@
+# ADR-0029: The workspace crates go to crates.io together, at one version with exact pins
+
+- Status: Accepted
+- Date: 2026-11-10
+- Deciders: @LeeroyHannigan
+
+## Context
+
+Today a user builds ExtendDB from a clone with `cargo build --release`, or
+runs a container image or the npm launcher. Publishing to crates.io adds the
+short form `cargo install extenddb --locked`, an entry in the registry's
+search, and docs.rs pages for the library crates. These facts constrain how
+the workspace can publish:
+
+- crates.io refuses a dependency that is not on crates.io. The one exception
+ is a dev-dependency without a version, which Cargo removes from the
+ published manifest. Cargo drops the path of a path dependency when it
+ packages a crate, so each path dependency also needs a version. The `extenddb` binary
+ depends on the other 13 workspace crates, so publishing the binary means
+ publishing all 14.
+- On main at e681b8bf2, `cargo package --workspace` stopped at its first crate
+ with an internal dependency: "all dependencies must have a version
+ requirement specified when packaging. dependency `extenddb-cache` does not
+ specify a version".
+- `extenddb-engine` and `extenddb-storage-sqlite` are dev-dependencies of each
+ other. With a version on every internal dependency, packaging stopped with
+ "no matching package named `extenddb-engine` found". Each of the two crates
+ must exist on crates.io before the other.
+- ExtendDB patch releases change the Rust APIs between its crates. For a 0.x
+ version, Cargo reads the requirement `0.1.14` as "0.1.14 or any later
+ 0.1.x".
+- A build contains exactly one storage backend. `crates/bin/src/main.rs`
+ rejects two backend features at compile time.
+- Without `--locked`, `cargo install` resolves every dependency again. Measured
+ on v0.1.14 from an empty directory: on Rust 1.88, the minimum supported
+ version, resolution chose `uuid` 1.29.2, which needs Rust 1.89, and the build
+ failed. With `resolver = "3"`, cargo resolved for Rust 1.88, chose `uuid`
+ 1.26.1, and the build passed. With either resolver, the fresh lock changed
+ the version of 173 of the 425 third-party crates that it shares with the
+ tested lock.
+- No package contained LICENSE or NOTICE. Section 4 of the Apache License 2.0
+ requires each redistributor to pass on a copy of the license and the NOTICE
+ attributions. A crate from crates.io would have reached its user without
+ either file.
+
+## Options Considered
+
+1. **Publish only the binary**: crates.io refuses it, because its
+ dependencies are path dependencies.
+2. **One binary crate per backend**, named like the container images:
+ `extenddb-postgres`, `extenddb-mongodb`, `extenddb-cassandra` and
+ `extenddb-dev`. Each installs with one short command, and the four binaries
+ can be installed together. The workspace gets 4 more crates, 18 in all. The rate
+ limit on new crates adds about 40 minutes to the first publish, and each
+ crate needs its own owners and trusted publisher entry. Two database tools
+ with millions of downloads each, diesel_cli and sqlx-cli, publish one
+ binary crate each and select databases with Cargo features.
+3. **One binary crate, `extenddb`, with a Cargo feature that selects the
+ backend**, as the repository builds it today. The crate README gives one
+ install command per backend.
+
+For the version requirements between the workspace crates:
+
+- Caret requirements (`0.1.14`): after the next release exists, a fresh
+ install of an `extenddb` release resolves each library crate to the next
+ release. The binary's own `main.rs` then builds against the next release's
+ `extenddb-app`, and their APIs differ.
+- Exact pins (`=0.1.14`): each release resolves only its own crates.
+
+## Decision
+
+All 14 workspace crates publish together at the workspace version, and each
+internal dependency requires exactly that version. `extenddb` stays the one
+binary crate, and a Cargo feature selects its backend.
+
+## Rationale
+
+- With exact pins, a fresh resolution of any release builds only the crates
+ of that release.
+- One binary crate follows the pattern of the two database tools above, and
+ it adds no crate, owner or trusted publisher entry.
+- Each crate declares its internal dev-dependencies by path, with no version.
+ Cargo drops them from the published manifests, which removes the cycle between
+ `extenddb-engine` and `extenddb-storage-sqlite`. Each crate's tests still
+ build in the repository with the same dependencies.
+- `resolver = "3"` makes a fresh resolution prefer versions that build on
+ Rust 1.88. Install instructions still say `--locked`, because the fresh
+ resolution is not the tested set.
+- Each crate directory links LICENSE and NOTICE to the root files. Cargo packs
+ the content of a linked file, and a link cannot drift from the root file as
+ a copy can.
+- The descriptions of the 13 library crates say that each is internal to the
+ `extenddb` binary, with no API stability promise. The versioning draft in
+ #897 lists the crate APIs under "Not covered", and RFC-0002 keeps the
+ stability policy of the `Storage` trait as its unresolved question 4.
+
+## Consequences
+
+- Each pull request runs `cargo package --workspace --no-verify --locked` and
+ `devtools/check-crate-packages`. The check fails on a pin that is not
+ exact, or on an internal dev-dependency with a version. It also fails on a
+ missing LICENSE, NOTICE or description, and on a package or manifest field
+ over crates.io's limits. A pin left at an older version also makes every
+ cargo command that resolves dependencies fail.
+- Compiling each package from its tarball takes minutes, so that build
+ belongs to the release workflow, before the upload, and not to each pull
+ request.
+- A release bump changes the 13 pins together with the workspace version.
+- Every release publishes all 14 crates, including the crates that did not
+ change.
+- A user can have one backend build of `extenddb` installed at a time.
+- A fresh resolution can choose a dependency release that breaks the build.
+ cassandra-protocol 4.1.0 did this for the Cassandra backend, so it and
+ cdrs-tokio carry upper bounds. `--locked` avoids the problem.
+- The tests of the published library crates build only in the repository.
+ Cargo drops the internal dev-dependencies, and some tests read fixtures
+ from other crates.
+- The license notices now list the workspace crates under the text of the
+ repository's LICENSE file, with a link to the repository.
+- Publishing is follow-up work. It needs a release workflow, and a first
+ publish with an API token, because trusted publishing cannot create a crate.
+ Each crate then needs its owners and a trusted publisher entry. Each crate
+ added to the workspace later needs its own first publish.
+- A version on crates.io can never be replaced. A yank only stops new lock
+ files from choosing it.
+
+---
+
+## License
+
+Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0.
+See [LICENSE](../../LICENSE) for the full text.
+
+This software is provided "as is" without warranty of any kind. ExtendDB is not
+affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is
+a trademark of Amazon.com, Inc.
diff --git a/docs/adr/README.md b/docs/adr/README.md
index 1fe3fe104..7b3963e0a 100644
--- a/docs/adr/README.md
+++ b/docs/adr/README.md
@@ -52,3 +52,4 @@ decision, write a new ADR.
| [0023](0023-catalog-migrations.md) | One list of catalog migrations for every backend, with a ledger in every store | Accepted |
| [0026](0026-credential-crypto-on-aws-lc.md) | Credential cryptography uses AWS-LC through aws-lc-rs | Accepted |
| [0027](0027-cassandra-index-row-version-stamps.md) | Cassandra: index rows carry the base-row version as their write timestamp | Accepted |
+| [0029](0029-crates-io-packaging.md) | The workspace crates go to crates.io together, at one version with exact pins | Accepted |
diff --git a/packaging/README.md b/packaging/README.md
index 85da98149..dc945f86a 100644
--- a/packaging/README.md
+++ b/packaging/README.md
@@ -95,11 +95,12 @@ cargo build --release --locked --no-default-features --features sqlite,dev-mode
Flow (mirrors the container release: candidate first, promotion is a
deliberate second step):
-1. Land the release state on `main` with the workspace version bumped and
- `npm/package.json` bumped to match, then tag that commit `vX.Y.Z` and push
- the tag. This is the same tag the container release consumes; the two
- publish workflows are dispatched independently against it. When the
- release ships a catalog migration, the same state raises
+1. Land the release state on `main`. Bump the workspace version and the 13
+ `=X.Y.Z` pins in the root `[workspace.dependencies]`, regenerate the four
+ license notices, and bump `npm/package.json` to match. Tag that commit
+ `vX.Y.Z` and push the tag. This is the same tag the container release
+ consumes; the two publish workflows are dispatched independently against
+ it. When the release ships a catalog migration, the same state raises
`RELEASED_THROUGH` to the last migration and pastes the checksum lines the
checksum test prints (CONTRIBUTING.md, "Adding a catalog migration"); the
gate jobs of the two container release workflows run
From 89a5759c9bdfaaddeab78256fec73ef8950a9c28 Mon Sep 17 00:00:00 2001
From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com>
Date: Tue, 10 Nov 2026 17:31:38 +0000
Subject: [PATCH 2/2] ci: publish the workspace crates to crates.io
A new workflow, release-crates.yml, publishes the 14 workspace crates
to crates.io for a release tag, with trusted publishing. A new script,
devtools/publish-crates, holds the publish logic for the workflow and
for a maintainer's token run.
- The script plans each crate as skip, publish or new. Skip means that
crates.io has the version. Publish means that the crate exists and
needs the version. New means that crates.io has no crate of that
name. The script orders the crates so that each one comes after the
workspace crates it depends on.
- In workflow mode the script publishes the crates that need a new
version. It stops before the first new crate, because trusted
publishing cannot create a crate.
- In token mode (`--new-crates`) it publishes only new crates and stops
before the first crate that needs a new version. crates.io refuses a
new version published with an API token for a crate that requires
trusted publishing. The first publish runs all 14 crates in token
mode.
- The `gate` job checks the tag, its commit on main, the version and
the revision label of the promoted image. Its dry run gives the
number of crates to publish and the first new crate.
- The `verify` job builds every package from its tarball with no
credentials. It records the SHA-256 of each package, the tag's commit
and the toolchain in an artifact.
- The `publish` job runs in the `crates-io` environment. It packages
again, compares the bytes with that list, and gets a crates.io token
through `rust-lang/crates-io-auth-action`. The script then publishes
each crate and hashes the file that cargo uploaded.
- The `index` job compares the checksums in the crates.io index with
the list. The `smoke` job installs the release in a clean cargo home.
When the release adds a crate, the `new-crate` job fails the run and
points at the token steps.
- The script waits through crates.io's 429 answers up to a cap. It
stops at crates.io's daily version limit for one crate, retries a
short index lag, and never prints the token.
- `devtools/check-crate-packages` also checks that each crate has the
workspace version.
- packaging/README.md describes each release and a release that adds a
crate. It also gives the token procedure, the repository settings
that the release depends on, and the steps after a failed check of
the uploaded bytes. ADR-0029 points at the workflow.
---
.github/workflows/release-crates.yml | 577 ++++++++++++
.github/workflows/test.yml | 6 +-
devtools/check-crate-packages | 30 +-
devtools/check_crate_packages_test.py | 24 +-
devtools/publish-crates | 682 ++++++++++++++
devtools/publish_crates_test.py | 1251 +++++++++++++++++++++++++
docs/adr/0029-crates-io-packaging.md | 26 +-
packaging/README.md | 269 +++++-
8 files changed, 2842 insertions(+), 23 deletions(-)
create mode 100644 .github/workflows/release-crates.yml
create mode 100755 devtools/publish-crates
create mode 100644 devtools/publish_crates_test.py
diff --git a/.github/workflows/release-crates.yml b/.github/workflows/release-crates.yml
new file mode 100644
index 000000000..e6bfb3304
--- /dev/null
+++ b/.github/workflows/release-crates.yml
@@ -0,0 +1,577 @@
+# Copyright 2026 ExtendDB contributors
+# SPDX-License-Identifier: Apache-2.0
+#
+# Publishes the workspace crates of one release to crates.io, at the
+# workspace version (docs/adr/0029-crates-io-packaging.md). crates.io has no
+# candidate stage and never replaces a version. So this workflow runs last in
+# a release, after promote-image. packaging/README.md, "Releasing to
+# crates.io", lists the release steps, the repository settings that the
+# release depends on, and the steps for a new crate.
+#
+# This workflow publishes new versions of crates that crates.io has, through
+# trusted publishing. Trusted publishing cannot create a crate. So a
+# maintainer publishes each new crate with an API token and
+# `devtools/publish-crates --new-crates`. That mode never publishes a crate
+# that crates.io has.
+#
+# Shape, and why:
+#
+# gate Checks that need no build. The run must come from main. The
+# input must be a strict `vX.Y.Z` tag that exists on origin.
+# Its commit must be in origin/main. Its version must equal the
+# workspace version at that commit. The promoted image of that
+# version must carry the tagged commit in its revision label.
+# The tagged commit must freeze the catalog migrations it ships.
+# Then `devtools/publish-crates --dry-run` plans the run, and
+# the plan goes to the job summary. The plan fails when a crate
+# on crates.io accepts API tokens. A new crate does not fail the
+# gate. The gate outputs the number of crates to publish before
+# the first new crate, and the name of that new crate.
+#
+# verify Holds no credentials. `cargo package --workspace --locked`
+# compiles every package from its tarball, and
+# `devtools/check-crate-packages` checks the packages. The job
+# uploads the SHA-256 of the package of each crate that
+# crates.io receives, with the tagged commit and the toolchain.
+# The build runs the build scripts and procedural macros of the
+# dependency graph. That is why no token exists in this job. It
+# also runs when the plan has a new crate. The maintainer's
+# token run then checks out that commit, and it checks its bytes
+# against this list.
+#
+# publish Runs when the plan has a crate to publish before the first
+# new crate. A reviewer approves the `crates-io` environment
+# first. The job packages the crates again without compiling
+# them, and runs `devtools/check-crate-packages`. It stops
+# unless each file has the SHA-256 that verify recorded. Then
+# rust-lang/crates-io-auth-action exchanges the job's OIDC token
+# for a crates.io token. That token expires after 30 minutes,
+# and the action revokes it when the job ends.
+# `devtools/publish-crates --no-verify` plans the run again, and
+# publishes the crates in dependency order, up to the first new
+# crate. After each upload, it compares the uploaded package
+# with the list. The job outputs the first new crate of its own
+# plan. A token run can publish the gate's new crate while the
+# job waits for its approval. The job then does not stop.
+#
+# index Runs when no new crate stopped the run, so that the run
+# published the whole release. The plan of the publish job
+# decides that, and the gate's plan when the run skips publish.
+# It reads each crate's file in the crates.io sparse index. It
+# compares the checksum of the version with the SHA-256 that
+# verify recorded. A difference fails the job.
+#
+# smoke Runs after index. It installs `extenddb` from crates.io with
+# an empty cargo home, outside any checkout. It checks the
+# version, the commit and the test-hooks lines that
+# `extenddb version` prints.
+#
+# new-crate Runs when the run stopped before a new crate. It writes the
+# stop and a pointer to the steps for an API token to the job
+# summary, and fails the run.
+#
+# Every job that runs cargo uses the toolchain in RUST_TOOLCHAIN. For one
+# commit and one toolchain, cargo writes the same bytes into each package.
+# The publish and index jobs compare those bytes.
+#
+# Trigger: manual workflow_dispatch only. A run that a pushed tag starts uses
+# the workflow file of the tagged commit. That file can be older than the
+# reviewed file on main. A dispatch from main runs the current file against
+# an existing tag.
+#
+# The release depends on two repository settings. packaging/README.md,
+# "Repository settings for crates.io", gives the commands that read them
+# back. A maintainer checks them before adding a trusted publisher.
+# 1. The environment crates-io. It has the four required reviewers of the
+# other release environments, with prevent self-review, so that the
+# dispatcher cannot approve the run. It allows deployments from main
+# only, and it holds no secrets. Admin bypass is off. crates.io never
+# replaces a version, so even an administrator needs the approval of a
+# second maintainer to publish.
+# gh api repos/ExtendDB/extenddb/environments/crates-io
+# 2. The tag ruleset protect-release-tags. It is active on refs/tags/v*,
+# blocks the deletion and the update of those tags, and has no bypass
+# actors. So a release tag cannot move to another commit.
+# gh api repos/ExtendDB/extenddb/rulesets
+#
+# Each crate needs a GitHub trusted publisher on crates.io, on the Settings
+# tab of the crate, after its first publish. Its fields are repository owner
+# ExtendDB, repository name extenddb, workflow filename release-crates.yml
+# and environment crates-io. The form labels the last field "Environment
+# name (optional)". Fill it in, because a publisher without it gives a token
+# to a run of this workflow from any branch. Then select "Require trusted
+# publishing for all new versions". crates.io matches the workflow filename
+# exactly. A rename of this file stops publishing until every crate's entry
+# names the new file.
+
+name: release-crates
+
+on:
+ workflow_dispatch:
+ inputs:
+ tag:
+ description: 'Existing release tag to publish the crates of, e.g. v0.1.15'
+ required: true
+
+# No job commits anything, so no job needs to write to the repository. The
+# publish job adds id-token at job level.
+permissions:
+ contents: read
+
+# Two crates.io releases must never run at the same time.
+concurrency:
+ group: release-crates-extenddb
+ cancel-in-progress: false
+
+env:
+ # The publish job compares its packages with the ones verify built. So
+ # every job that runs cargo uses this toolchain. A maintainer who
+ # publishes a new crate with an API token uses it too
+ # (packaging/README.md). Change it in a reviewed pull request of its own.
+ RUST_TOOLCHAIN: "1.97.0"
+
+jobs:
+ gate:
+ runs-on: ubuntu-latest
+ timeout-minutes: 15
+ outputs:
+ version: ${{ steps.meta.outputs.version }}
+ sha: ${{ steps.meta.outputs.sha }}
+ publish_count: ${{ steps.plan.outputs.publish_count }}
+ first_new: ${{ steps.plan.outputs.first_new }}
+ steps:
+ - name: Gate - the run comes from main
+ # A dispatch from another branch runs that branch's copy of this
+ # file. The branch rule of the environment would stop it only at
+ # publish, after the verify build.
+ run: |
+ if [[ "$GITHUB_REF" != refs/heads/main ]]; then
+ echo "::error::dispatch release-crates from main, not from ${GITHUB_REF}"
+ exit 1
+ fi
+
+ - name: Check out main (the reviewed workflow's own ref)
+ uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
+ with:
+ fetch-depth: 0 # tag resolution and the ancestry gate need history
+
+ - name: Gate - the tag must identify released code
+ id: meta
+ env:
+ # Dispatch inputs are untrusted. They reach the shell only through
+ # the environment, never as text in the script.
+ RAW_TAG: ${{ inputs.tag }}
+ run: |
+ set -euo pipefail
+
+ # 1. The strict shape comes first, so no other input reaches git
+ # as a ref expression.
+ if [[ ! "$RAW_TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+ echo "::error::input '$RAW_TAG' is not a strict release tag (vMAJOR.MINOR.PATCH)"
+ exit 1
+ fi
+ VERSION="${RAW_TAG#v}"
+
+ # 2. The tag must exist. This workflow never creates tags.
+ git fetch --no-tags origin "refs/tags/${RAW_TAG}:refs/tags/${RAW_TAG}" \
+ || { echo "::error::tag ${RAW_TAG} does not exist on origin"; exit 1; }
+ SHA=$(git rev-list -n 1 "refs/tags/${RAW_TAG}")
+ [[ "$SHA" =~ ^[0-9a-f]{40}$ ]] \
+ || { echo "::error::could not resolve ${RAW_TAG} to a full commit SHA"; exit 1; }
+
+ # 3. The tagged commit must be on main, so that a tag on an
+ # unmerged branch cannot publish code that was not reviewed.
+ git fetch --no-tags origin main
+ if ! git merge-base --is-ancestor "$SHA" origin/main; then
+ echo "::error::${RAW_TAG} (${SHA}) is not contained in origin/main"
+ exit 1
+ fi
+
+ # 4. The tag must equal the workspace version at that commit.
+ CARGO_VERSION=$(git show "${SHA}:Cargo.toml" \
+ | sed -n 's/^version = "\(.*\)"$/\1/p' | head -1)
+ if [[ "$VERSION" != "$CARGO_VERSION" ]]; then
+ echo "::error::tag ${RAW_TAG} implies ${VERSION} but the workspace version at ${SHA} is ${CARGO_VERSION}"
+ exit 1
+ fi
+
+ {
+ echo "version=$VERSION"
+ echo "sha=$SHA"
+ } >> "$GITHUB_OUTPUT"
+
+ - name: Gate - the promoted image holds the tagged commit
+ # promote-image runs before this workflow. The version tag of the
+ # default image must carry the tagged commit in its revision label.
+ # The read is anonymous, as in promote-image.yml. A tag that moved
+ # to another commit fails here, because the image keeps its commit.
+ env:
+ VERSION: ${{ steps.meta.outputs.version }}
+ SHA: ${{ steps.meta.outputs.sha }}
+ run: |
+ set -euo pipefail
+ command -v skopeo >/dev/null || sudo apt-get install -y --no-install-recommends skopeo
+ IMAGE="docker://docker.io/extenddb/extenddb-postgres:${VERSION}"
+ if ! INSPECT=$(skopeo inspect --no-creds "$IMAGE"); then
+ echo "::error::cannot read ${IMAGE}. Run promote-image for ${VERSION} first."
+ exit 1
+ fi
+ REVISION=$(jq -r '.Labels["org.opencontainers.image.revision"] // ""' <<< "$INSPECT")
+ if [[ "$REVISION" != "$SHA" ]]; then
+ echo "::error::${IMAGE} has the revision label '${REVISION}', not the tagged commit ${SHA}"
+ exit 1
+ fi
+ echo "${IMAGE} has the revision ${REVISION}"
+
+ - name: Check out the tagged commit
+ uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
+ with:
+ ref: ${{ steps.meta.outputs.sha }}
+ persist-credentials: false
+
+ - name: Gate - the tagged commit has the crates.io release scripts
+ run: |
+ set -euo pipefail
+ for script in devtools/check-crate-packages devtools/publish-crates; do
+ if [[ ! -x "$script" ]]; then
+ echo "::error::the tagged commit has no ${script}. Use a tag created after release-crates.yml was merged."
+ exit 1
+ fi
+ done
+
+ - name: Gate - the release freezes the catalog migrations it ships
+ # The crates carry the catalog migrations of every backend, as the
+ # images and the npm packages do. So RELEASED_THROUGH must reach the
+ # last migration id.
+ run: python3 devtools/check-released-through
+
+ - uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master
+ with:
+ toolchain: ${{ env.RUST_TOOLCHAIN }}
+
+ - name: Gate - plan the run
+ id: plan
+ # Reads the crates.io sparse index and the crates.io API. The plan
+ # shows the stop before a new crate. The script writes publish_count
+ # and first_new to the step outputs.
+ env:
+ VERSION: ${{ steps.meta.outputs.version }}
+ SHA: ${{ steps.meta.outputs.sha }}
+ run: |
+ set -euo pipefail
+ devtools/publish-crates --dry-run --version "$VERSION" --plan-output "$GITHUB_OUTPUT" \
+ | tee "$RUNNER_TEMP/plan.txt"
+ {
+ printf '### crates.io release v%s\n\n' "$VERSION"
+ printf '| field | value |\n|---|---|\n'
+ printf "| tag | \`v%s\` |\n" "$VERSION"
+ printf "| commit | \`%s\` |\n\n" "$SHA"
+ printf "\`\`\`\n"
+ cat "$RUNNER_TEMP/plan.txt"
+ printf "\`\`\`\n"
+ } >> "$GITHUB_STEP_SUMMARY"
+
+ verify:
+ needs: gate
+ runs-on: ubuntu-latest
+ timeout-minutes: 60
+ permissions:
+ contents: read
+ steps:
+ - name: Check out the tagged commit
+ uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
+ with:
+ ref: ${{ needs.gate.outputs.sha }}
+ # The build runs third-party build scripts, so the repository
+ # token stays out of .git/config.
+ persist-credentials: false
+
+ - uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master
+ with:
+ toolchain: ${{ env.RUST_TOOLCHAIN }}
+
+ - name: Build every package from its tarball
+ run: cargo package --workspace --locked
+
+ - name: Check the packages against the crates.io rules
+ run: devtools/check-crate-packages
+
+ - name: Record the SHA-256 of each crate's package, the commit and the toolchain
+ # SHA256SUMS has a line for the package of each crate that crates.io
+ # receives. cargo also packages a member that crates.io does not
+ # receive. The list leaves that file out, so that index compares only
+ # the crates of the release. COMMIT and TOOLCHAIN are for the
+ # maintainer's token run (packaging/README.md, "Publishing new crates
+ # with an API token"). That run checks out the tag only when the
+ # tag's commit is COMMIT, and it packages with TOOLCHAIN.
+ env:
+ VERSION: ${{ needs.gate.outputs.version }}
+ SHA: ${{ needs.gate.outputs.sha }}
+ run: |
+ set -euo pipefail
+ mkdir -p "$RUNNER_TEMP/crate-sha256"
+ devtools/publish-crates --list-packages --version "$VERSION" > "$RUNNER_TEMP/packages.txt"
+ mapfile -t packages < "$RUNNER_TEMP/packages.txt"
+ if [[ ${#packages[@]} -eq 0 ]]; then
+ echo "::error::devtools/publish-crates lists no package"
+ exit 1
+ fi
+ printf '%s\n' "$SHA" > "$RUNNER_TEMP/crate-sha256/COMMIT"
+ printf '%s\n' "$RUST_TOOLCHAIN" > "$RUNNER_TEMP/crate-sha256/TOOLCHAIN"
+ cd target/package
+ sha256sum -- "${packages[@]}" | tee "$RUNNER_TEMP/crate-sha256/SHA256SUMS"
+
+ - name: Upload the checksums, the commit and the toolchain
+ uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+ with:
+ name: crate-sha256
+ # The three files of the directory, at the root of the artifact.
+ path: ${{ runner.temp }}/crate-sha256/
+ if-no-files-found: error
+ retention-days: 7 # a failed publish job can be re-run for a week
+
+ publish:
+ needs: [gate, verify]
+ # A run with nothing to publish before the first new crate mints no
+ # token.
+ if: needs.gate.outputs.publish_count != '0'
+ runs-on: ubuntu-latest
+ timeout-minutes: 45
+ # Required reviewers with prevent self-review, no admin bypass, and the
+ # main-only branch rule. Each crate's trusted publisher names this
+ # environment. So a run outside it cannot get a crates.io token.
+ environment: crates-io
+ permissions:
+ contents: read
+ id-token: write # the OIDC token that crates.io exchanges for a publish token
+ outputs:
+ # The first new crate of this job's own plan, or empty. The script
+ # reads crates.io again before the first upload, so index and
+ # new-crate follow this plan and not the gate's.
+ first_new: ${{ steps.run.outputs.first_new }}
+ steps:
+ - name: Check out the tagged commit
+ uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
+ with:
+ ref: ${{ needs.gate.outputs.sha }}
+ persist-credentials: false
+
+ - uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master
+ with:
+ toolchain: ${{ env.RUST_TOOLCHAIN }}
+
+ - name: Download the verified checksums
+ uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
+ with:
+ name: crate-sha256
+ path: ${{ runner.temp }}/crate-sha256
+
+ - name: Package without compiling, check, and compare with the verified packages
+ # `cargo package --no-verify` runs no build script, so the checker
+ # runs here on files that no dependency code touched. A package with
+ # other bytes than the one verify compiled stops the release before
+ # the first upload.
+ run: |
+ set -euo pipefail
+ cargo package --workspace --no-verify --locked
+ devtools/check-crate-packages
+ cd target/package
+ sha256sum --check --strict "$RUNNER_TEMP/crate-sha256/SHA256SUMS"
+
+ - name: Get a crates.io token through trusted publishing
+ id: auth
+ uses: rust-lang/crates-io-auth-action@3268eb413133e2960fa4ae0e963ca76380c54cd9 # v1.0.6
+
+ - name: Publish the crates in dependency order
+ id: run
+ env:
+ CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }}
+ VERSION: ${{ needs.gate.outputs.version }}
+ FIRST_NEW: ${{ needs.gate.outputs.first_new }}
+ # crates.io does not rate-limit trusted publishing per user. Its one
+ # 429 here is the limit on the versions of one crate in 24 hours,
+ # and the script stops at it. The cap of 120 s bounds the waits for
+ # the index and after any other 429. The token expires 30 minutes
+ # after the step above.
+ run: |
+ set -euo pipefail
+ args=(--no-verify --max-wait 120 --version "$VERSION"
+ --sha256sums "$RUNNER_TEMP/crate-sha256/SHA256SUMS"
+ --plan-output "$GITHUB_OUTPUT")
+ # The gate found a new crate. The run publishes the crates before
+ # it and stops there, and the new-crate job reports the stop. When
+ # a token run published that crate in the meantime, the run does
+ # not stop, and index and smoke run instead.
+ if [[ -n "$FIRST_NEW" ]]; then
+ args+=(--expect-new "$FIRST_NEW")
+ fi
+ devtools/publish-crates "${args[@]}"
+
+ index:
+ needs: [gate, verify, publish]
+ # Only when no new crate stopped the run, so that the run published the
+ # whole release. The plan of the publish job decides when publish ran.
+ # The run skips publish when the gate's plan has nothing to publish
+ # before the first new crate, and the gate's plan decides then.
+ if: >-
+ ${{ !cancelled()
+ && needs.verify.result == 'success'
+ && ((needs.publish.result == 'success' && needs.publish.outputs.first_new == '')
+ || (needs.publish.result == 'skipped' && needs.gate.outputs.first_new == '')) }}
+ runs-on: ubuntu-latest
+ timeout-minutes: 20
+ permissions: {}
+ steps:
+ - name: Download the verified checksums
+ uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
+ with:
+ name: crate-sha256
+ path: ${{ runner.temp }}/crate-sha256
+
+ - name: Compare the checksums in the crates.io index with the verified packages
+ env:
+ VERSION: ${{ needs.gate.outputs.version }}
+ run: |
+ set -euo pipefail
+ # The path of a crate's file in the sparse index (the Cargo book,
+ # "Registry Index"). devtools/publish-crates uses the same rule.
+ index_path() {
+ local name="${1,,}"
+ case "${#name}" in
+ 1|2) echo "${#name}/${name}" ;;
+ 3) echo "3/${name:0:1}/${name}" ;;
+ *) echo "${name:0:2}/${name:2:2}/${name}" ;;
+ esac
+ }
+ failed=0
+ while read -r sum file; do
+ name="${file%-"${VERSION}".crate}"
+ if [[ "$name" == "$file" ]]; then
+ echo "::error::${file} is not a package of version ${VERSION}"
+ failed=1
+ continue
+ fi
+ url="https://index.crates.io/$(index_path "$name")"
+ # cargo publish waits up to 60 s for each version. A cache in
+ # front of the index can serve an older file for a short time.
+ cksum=""
+ for attempt in $(seq 1 20); do
+ cksum=$(curl -fsS --max-time 30 "$url" | jq -r --arg v "$VERSION" 'select(.vers == $v) | .cksum' || true)
+ if [[ -n "$cksum" ]]; then
+ break
+ fi
+ echo "${name} ${VERSION} is not in the index yet (attempt ${attempt} of 20)"
+ sleep 15
+ done
+ if [[ -z "$cksum" ]]; then
+ echo "::error::${name} ${VERSION} is not in the crates.io index. See packaging/README.md, \"A failure after the upload\"."
+ failed=1
+ elif [[ "$cksum" != "$sum" ]]; then
+ echo "::error::${name} ${VERSION}: crates.io has cksum ${cksum}, and the verified package has SHA-256 ${sum}. The version is public and permanent. See packaging/README.md, \"A failure after the upload\"."
+ failed=1
+ else
+ echo "${name} ${VERSION}: ${cksum}"
+ fi
+ done < "$RUNNER_TEMP/crate-sha256/SHA256SUMS"
+ exit "$failed"
+
+ smoke:
+ needs: [gate, index]
+ if: ${{ !cancelled() && needs.index.result == 'success' }}
+ runs-on: ubuntu-latest
+ timeout-minutes: 45
+ permissions: {}
+ steps:
+ - uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master
+ with:
+ toolchain: ${{ env.RUST_TOOLCHAIN }}
+
+ - name: Install extenddb from crates.io with an empty cargo home
+ env:
+ VERSION: ${{ needs.gate.outputs.version }}
+ SHA: ${{ needs.gate.outputs.sha }}
+ run: |
+ set -euo pipefail
+ # The job checks out nothing, and it works in a new directory under
+ # /tmp. So no cargo configuration of a repository or of the
+ # runner's home directory applies. The new CARGO_HOME holds no
+ # registry cache, configuration or token.
+ CARGO_HOME=$(mktemp -d)
+ export CARGO_HOME
+ cd "$(mktemp -d)"
+ # The index job saw every version. Try again only while cargo
+ # cannot resolve a version, never after a build error.
+ installed=false
+ for attempt in $(seq 1 10); do
+ if cargo install extenddb --version "$VERSION" --locked 2>&1 | tee install.log; then
+ installed=true
+ break
+ fi
+ if ! grep -qE 'could not find .extenddb. in registry|failed to select a version|no matching package named' install.log; then
+ break
+ fi
+ echo "crates.io does not resolve extenddb ${VERSION} yet (attempt ${attempt} of 10)"
+ sleep 30
+ done
+ if [[ "$installed" != true ]]; then
+ echo "::error::cargo install extenddb --version ${VERSION} --locked failed. Rerun this job first. See packaging/README.md, \"A failure after the upload\"."
+ exit 1
+ fi
+
+ "$CARGO_HOME/bin/extenddb" version | tee version.txt
+ # The binary reads the commit from the .cargo_vcs_info.json that
+ # cargo publish wrote. So this line shows the tag's commit only
+ # when cargo packaged the crate from a clean checkout of the tag.
+ after="See packaging/README.md, \"A failure after the upload\"."
+ grep -qxF "extenddb ${VERSION}" version.txt \
+ || { echo "::error::extenddb version does not report ${VERSION}. ${after}"; exit 1; }
+ grep -qxF "commit ${SHA}" version.txt \
+ || { echo "::error::extenddb version does not report commit ${SHA}. ${after}"; exit 1; }
+ # The test-hooks lines that `extenddb version` prints: the MongoDB
+ # test hooks are disabled, and no backup-test-hooks line appears.
+ # A test-hooks feature of a storage crate adds no line.
+ grep -qxF "test-hooks disabled" version.txt \
+ || { echo "::error::extenddb version does not report test-hooks disabled. ${after}"; exit 1; }
+ if grep -qF "backup-test-hooks enabled" version.txt; then
+ echo "::error::extenddb version reports backup-test-hooks enabled. ${after}"
+ exit 1
+ fi
+
+ new-crate:
+ needs: [gate, verify, publish]
+ # Only when the run stopped before a new crate. Either the plan of the
+ # publish job has a new crate, or the run skipped publish and the gate's
+ # plan has one. The publish job exits 0 at that stop only when the stop
+ # is at the gate's new crate, so the gate's outputs describe the stop.
+ if: >-
+ ${{ !cancelled()
+ && needs.verify.result == 'success'
+ && ((needs.publish.result == 'success' && needs.publish.outputs.first_new != '')
+ || (needs.publish.result == 'skipped' && needs.gate.outputs.first_new != '')) }}
+ runs-on: ubuntu-latest
+ timeout-minutes: 5
+ permissions: {}
+ steps:
+ - name: Report the stop before the new crate
+ env:
+ VERSION: ${{ needs.gate.outputs.version }}
+ FIRST_NEW: ${{ needs.gate.outputs.first_new }}
+ PUBLISH_COUNT: ${{ needs.gate.outputs.publish_count }}
+ RUN_ID: ${{ github.run_id }}
+ run: |
+ set -euo pipefail
+ crates="crates"
+ if [[ "$PUBLISH_COUNT" == 1 ]]; then
+ crates="crate"
+ fi
+ {
+ printf '### Stopped before %s\n\n' "$FIRST_NEW"
+ printf "This run published %s %s of v%s, and then stopped before \`%s\`.\n" \
+ "$PUBLISH_COUNT" "$crates" "$VERSION" "$FIRST_NEW"
+ printf 'crates.io has no crate of this name, and trusted publishing cannot create a crate.\n'
+ printf "A maintainer publishes it with an API token and the \`crate-sha256\` artifact of this run,\n"
+ printf "run ID %s. packaging/README.md, \"Publishing new crates with an API token\", gives the steps.\n" \
+ "$RUN_ID"
+ } >> "$GITHUB_STEP_SUMMARY"
+ echo "::error::crates.io has no crate named ${FIRST_NEW}. The job summary points at the steps for an API token."
+ exit 1
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index b951c035e..5e25601b0 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -111,9 +111,9 @@ jobs:
# --no-verify writes the packages without compiling them, in seconds, and
# runs no build script. devtools/check-crate-packages then checks the pins,
# the LICENSE and NOTICE files, the metadata and crates.io's size limits.
- # Compiling each package from its tarball takes minutes. That build belongs
- # to the crates.io release workflow (follow-up work), before the upload, and
- # not to each PR.
+ # Compiling each package from its tarball takes minutes. That build runs in
+ # the verify job of release-crates.yml, before the upload, and not on each
+ # PR.
package:
runs-on: extenddb_ubuntu-2404_4-core
timeout-minutes: 10
diff --git a/devtools/check-crate-packages b/devtools/check-crate-packages
index 8eae2ed5a..52442bb35 100755
--- a/devtools/check-crate-packages
+++ b/devtools/check-crate-packages
@@ -12,8 +12,13 @@ changes, it also needs `--allow-dirty`. This script needs Python 3.11 or
later. It reads the members from `cargo metadata`, so cargo resolves globs,
`exclude`, inherited fields and renamed dependencies. It then fails, with
one line per problem, unless all of the following hold for each member that
-crates.io receives (one without `publish = false`):
+crates.io receives (one without a `publish` field, or one whose `publish`
+list names crates-io):
+- The member's version is the `version` of the root manifest's
+ `[workspace.package]` table. A release publishes every crate at the
+ workspace version, and `devtools/publish-crates --version` refuses a
+ crate of another version.
- Each dependency on another workspace crate, normal or build, requires
that crate's version exactly (`=X.Y.Z`). A caret requirement would let a
fresh install combine crates of two releases. A pin left at the previous
@@ -91,8 +96,19 @@ def cargo_metadata(root: Path) -> dict:
def is_published(package: dict) -> bool:
- """`publish = false` reaches cargo metadata as an empty registry list."""
- return package.get("publish") != []
+ """A member that crates.io receives. `publish = false` reaches cargo
+ metadata as an empty registry list, and a member without the field as
+ null. devtools/publish-crates uses the same rule."""
+ registries = package.get("publish")
+ return registries is None or "crates-io" in registries
+
+
+def workspace_version(metadata: dict) -> str | None:
+ """The `version` of the root manifest's [workspace.package] table, or
+ None when the root manifest sets none."""
+ manifest = tomllib.loads((Path(metadata["workspace_root"]) / "Cargo.toml").read_text())
+ version = manifest.get("workspace", {}).get("package", {}).get("version")
+ return version if isinstance(version, str) else None
def check(root: Path, packages: Path | None = None) -> list[str]:
@@ -103,11 +119,19 @@ def check(root: Path, packages: Path | None = None) -> list[str]:
if packages is None:
packages = Path(metadata["target_directory"]) / "package"
problems: list[str] = []
+ release = workspace_version(metadata)
+ if release is None:
+ problems.append("the root Cargo.toml sets no [workspace.package] version, so the release version is unknown")
for name, package in sorted(members.items()):
if not is_published(package):
continue
manifest = os.path.relpath(package["manifest_path"], root)
+ if release is not None and package["version"] != release:
+ problems.append(
+ f"{manifest}: {name} is version {package['version']}, and the workspace version is {release}. "
+ f"A release publishes every crate at the workspace version (`version.workspace = true`)."
+ )
for dependency in package["dependencies"]:
target = dependency["name"]
if target not in members:
diff --git a/devtools/check_crate_packages_test.py b/devtools/check_crate_packages_test.py
index 4e12c23ff..a618fc74d 100644
--- a/devtools/check_crate_packages_test.py
+++ b/devtools/check_crate_packages_test.py
@@ -207,14 +207,36 @@ def test_an_excluded_directory_is_not_a_member(self) -> None:
self.write("crates/x/src/lib.rs", "")
self.assert_passes()
- def test_a_member_with_its_own_version_is_packaged_and_pinned_at_it(self) -> None:
+ def test_a_member_with_its_own_version_fails_and_its_pins_are_checked_at_it(self) -> None:
self.write("Cargo.toml", ROOT_MANIFEST.replace(f'"crates/a", version = "={VERSION}"', '"crates/a", version = "=2.0.0"'))
self.member("a", MEMBER_A.replace("version.workspace = true", 'version = "2.0.0"'))
(self.packages / f"example-a-{VERSION}.crate").unlink()
self.package("example-a", {"Cargo.toml": packaged_manifest("example-a", version="2.0.0")}, version="2.0.0")
self.package("example-b", {"Cargo.toml": packaged_manifest("example-b", '\n[dependencies.example-a]\nversion = "=2.0.0"\n')})
+ code, output = self.run_check()
+ self.assertEqual(code, 1, output)
+ self.assertIn(f"error: crates/a/Cargo.toml: example-a is version 2.0.0, and the workspace version is {VERSION}.",
+ output)
+ self.assertIn("1 problem(s) found", output)
+
+ def test_a_root_manifest_without_a_workspace_version_fails(self) -> None:
+ self.write("Cargo.toml", ROOT_MANIFEST.replace(f'version = "{VERSION}"\nedition', "edition"))
+ for directory, manifest in (("a", MEMBER_A), ("b", MEMBER_B)):
+ self.member(directory, manifest.replace("version.workspace = true", f'version = "{VERSION}"'))
+ self.assert_fails_with("error: the root Cargo.toml sets no [workspace.package] version")
+
+ def test_a_member_published_only_to_another_registry_needs_no_package(self) -> None:
+ self.member("b", MEMBER_B.replace("edition.workspace = true", 'edition.workspace = true\npublish = ["my-registry"]')
+ .replace("{ workspace = true }", '{ path = "../a" }'))
+ (self.packages / f"example-b-{VERSION}.crate").unlink()
self.assert_passes()
+ def test_a_member_whose_publish_list_names_crates_io_is_checked(self) -> None:
+ self.member("b", MEMBER_B.replace("edition.workspace = true",
+ 'edition.workspace = true\npublish = ["my-registry", "crates-io"]'))
+ (self.packages / f"example-b-{VERSION}.crate").unlink()
+ self.assert_fails_with(f"example-b-{VERSION}.crate: missing")
+
def test_cargo_target_dir_moves_the_packages(self) -> None:
moved = self.root / "elsewhere"
for crate in self.packages.iterdir():
diff --git a/devtools/publish-crates b/devtools/publish-crates
new file mode 100755
index 000000000..c9b636bf9
--- /dev/null
+++ b/devtools/publish-crates
@@ -0,0 +1,682 @@
+#!/usr/bin/env python3
+# Copyright 2026 ExtendDB contributors
+# SPDX-License-Identifier: Apache-2.0
+"""Publish the workspace crates to crates.io, one at a time, in dependency order.
+
+ devtools/publish-crates [--new-crates] [--dry-run] [--no-verify]
+ [--version X.Y.Z] [--sha256sums FILE]
+ [--expect-new NAME] [--plan-output FILE]
+ [--max-wait SECONDS] [--root DIR]
+ devtools/publish-crates --list-packages [--version X.Y.Z] [--root DIR]
+
+This script needs Python 3.11 or later. It reads the members from
+`cargo metadata`. It takes each member that crates.io receives: a member
+without a `publish` field, or one whose `publish` list names crates-io. It
+orders them so that each crate comes after every workspace crate it depends
+on through a normal or build dependency. A cycle stops the run. With
+--version, a crate of another version also stops the run.
+
+The script then reads each crate's file in the crates.io sparse index, and
+it plans one of three actions for each crate:
+
+- skip: the index has the crate's version. crates.io never replaces a
+ version, so a run that stopped part way continues where it stopped.
+- publish: the index has the crate, but not this version.
+- new: the index has no file for the crate.
+
+The script has two modes. The mode --new-crates publishes the first version
+of each crate, and the other mode publishes each later version:
+
+- Without --new-crates, the script publishes the crates of the action
+ publish. The release-crates workflow runs it with a trusted publishing
+ token. It stops before the first new crate, because trusted publishing
+ cannot create a crate. This mode also reads each crate that crates.io has
+ from the crates.io API. It stops when one of them accepts API tokens.
+- With --new-crates, the script publishes the new crates. A maintainer runs
+ it with an API token. It stops before the first crate of the action
+ publish. crates.io refuses a new version published with an API token for
+ a crate that requires trusted publishing, and the release-crates workflow
+ publishes that crate.
+
+A run that stops prints the next steps and exits 1. A first publish has only
+new crates, so the mode --new-crates publishes all of them. A later release
+that adds a crate alternates between the modes. packaging/README.md,
+"Releasing to crates.io", describes the steps.
+
+The script prints the plan and the crate where the run stops. --dry-run
+stops there and exits 0. --plan-output FILE appends two lines to FILE, in
+the format of GITHUB_OUTPUT. publish_count is the number of crates of the
+action publish before the first new crate. first_new is the name of the
+first new crate, or empty when the plan has no new crate. With
+--expect-new NAME, a stop before the new crate NAME exits 0. The workflow
+passes the name that its gate found, and a later job reports the stop. A
+token run can publish NAME in the meantime. The run then does not stop, and
+it prints one line that says so. NAME must be a crate of the release.
+
+Otherwise the script runs `cargo publish -p --locked` for each crate
+to publish, in order. It adds --no-verify when the caller passes it. Cargo
+waits up to 60 seconds until the index has the new version, so the next
+crate can resolve it. When that wait times out, cargo warns and exits 0.
+
+Pass --no-verify only after `cargo package --workspace --locked` built
+every package from its tarball without the token. A verifying publish runs
+the build scripts and procedural macros of the whole dependency graph while
+the token is in the environment.
+
+--sha256sums FILE takes the SHA-256 list of the verified packages, in the
+format of sha256sum. The list must have a line for each crate that the run
+publishes or skips, or the run stops before the first upload. The run also
+stops there when the index lists the version of a skipped crate with
+another checksum. `cargo publish` writes the package that it uploads into
+`package/tmp-crate` of cargo's build directory, which `cargo metadata`
+names. Only `cargo package` copies the file on into the target directory.
+Before each upload, the script deletes the crate's file in `tmp-crate`.
+After the upload, it hashes the file that cargo wrote there, and it stops
+when the hash differs from the list. --list-packages prints the package
+file name of each crate, in publish order, and reads no index. The
+release-crates workflow hashes those files.
+
+crates.io limits how fast one account publishes (src/rate_limiter.rs). Its
+defaults allow 5 new crates at once and then one every 10 minutes. They
+allow 30 new versions at once and then one a minute. On an answer with
+status 429, the script waits until the time in the answer, plus 5 seconds.
+Then it runs the publish again, and it prints each wait. crates.io also
+limits the versions of one crate in 24 hours, and the script stops at that
+answer. When cargo cannot find a crate that this run published, the index
+lags, and the script runs the publish again after 30 seconds. --max-wait
+caps the total of these waits, not the time of the uploads. The script
+stops when the next wait would take the total past the cap. The default of
+2 hours covers the first publish of 14 new crates, which takes about 90
+minutes.
+
+cargo reads the token from CARGO_REGISTRY_TOKEN or from `cargo login`. The
+script never prints the token or the environment.
+
+Exit 0 when the run published its crates, for a dry run, and for
+--list-packages. Exit 1 when the run stops before a crate that --expect-new
+does not name, and on any error. Exit 2 on a usage error.
+"""
+
+from __future__ import annotations
+
+import argparse
+import email.utils
+import hashlib
+import http.client
+import json
+import math
+import os
+import re
+import subprocess
+import sys
+import time
+import urllib.error
+import urllib.request
+from datetime import datetime
+from pathlib import Path
+
+try:
+ from datetime import UTC
+except ImportError: # Python 3.10 and older
+ sys.exit("error: devtools/publish-crates needs Python 3.11 or later (for datetime.UTC)")
+
+INDEX = "https://index.crates.io/"
+API = "https://crates.io/api/v1/crates/"
+USER_AGENT = "extenddb-publish-crates (https://github.com/ExtendDB/extenddb)"
+# The Cargo reference, "Registry Index", "Nonexistent Crates": a registry
+# answers 404, 410 or 451 for a crate that it does not have.
+NOT_FOUND = (404, 410, 451)
+TIMEOUT = 30
+# crates.io's data access policy allows at most one API request a second.
+API_INTERVAL = 1
+REGISTRY = "crates-io"
+DEFAULT_MAX_WAIT = 2 * 60 * 60
+RETRY_MARGIN = 5
+# The wait after a 429 answer that gives no time.
+DEFAULT_RETRY_WAIT = 60
+# The wait before the next try when the index does not list a crate that
+# this run published.
+INDEX_RETRY_WAIT = 30
+
+# cargo reports an answer that holds crates.io's JSON errors as "the remote
+# server responded with an error (status 429 Too Many Requests): ",
+# and any other answer as "failed to get a 200 OK response, got 429", with
+# the headers and the body after it.
+TOO_MANY_REQUESTS = re.compile(r"\(status 429\b|\bgot 429\b")
+# crates.io's detail says "Please try again after Tue, 10 Nov 2026 14:30:00
+# GMT", and its Retry-After header holds the same date
+# (src/util/errors/json.rs).
+RETRY_AT = re.compile(
+ r"(?:try again after|retry-after:)\s*(\w{3}, \d{1,2} \w{3} \d{4} \d{2}:\d{2}:\d{2} GMT)",
+ re.IGNORECASE,
+)
+RETRY_SECONDS = re.compile(r"^\s*retry-after:\s*(\d+)\s*$", re.IGNORECASE | re.MULTILINE)
+# crates.io's limit on the versions of one crate (src/controllers/krate/publish.rs).
+DAILY_LIMIT = re.compile(r"too many versions of this crate in the last 24 hours")
+# cargo's resolver errors when the index does not list a dependency yet: a
+# crate that it does not know, or a version that it does not have.
+MISSING_DEPENDENCY = re.compile(
+ r"no matching package named `([^`]+)` found"
+ r"|failed to select a version for the requirement `([^`\s]+) = "
+)
+# A line of sha256sum output for a file name without blanks, which every
+# package file name is.
+SHA256_LINE = re.compile(r"([0-9a-f]{64}) [ *](\S+)")
+
+SKIP = "skip"
+PUBLISH = "publish"
+NEW = "new"
+STATES = {SKIP: "on crates.io", PUBLISH: "new version", NEW: "new crate"}
+
+TOKEN_STEPS = 'packaging/README.md, "Publishing new crates with an API token"'
+AFTER_UPLOAD = 'packaging/README.md, "A failure after the upload"'
+
+FIRST_PUBLISH = """
+crates.io has no crate named {name}, and trusted publishing cannot create a
+crate. A maintainer publishes {name} with an API token of the scope
+publish-new, and runs devtools/publish-crates --new-crates for it. Then the
+maintainer gives {name} its owners and its trusted publisher, requires
+trusted publishing for it, and dispatches release-crates again.
+{steps} gives the steps and the commands.
+"""
+
+TOKEN_STOP = """
+crates.io has {name}, and an API token must not publish it. crates.io
+refuses a new version published with an API token for a crate that
+requires trusted publishing. Revoke the token now, and finish the steps for
+each new crate in {steps}. Then dispatch release-crates again with the tag,
+to publish {name} and the crates after it.
+"""
+
+TOKEN_DONE = """
+Revoke the token now, and finish the steps for each new crate in {steps}.
+Then dispatch release-crates again with the tag, so that its index and smoke
+jobs run on the whole release.
+"""
+
+
+class Failure(Exception):
+ """A problem that stops the run, reported on one line."""
+
+
+def cargo() -> str:
+ return os.environ.get("CARGO", "cargo")
+
+
+def now() -> datetime:
+ return datetime.now(UTC)
+
+
+def sleep(seconds: int) -> None:
+ time.sleep(seconds)
+
+
+def api_pause() -> None:
+ time.sleep(API_INTERVAL)
+
+
+def count(number: int) -> str:
+ return f"{number} crate" if number == 1 else f"{number} crates"
+
+
+def package_file(name: str, version: str) -> str:
+ """The file name of a crate's package, as cargo writes it."""
+ return f"{name}-{version}.crate"
+
+
+def cargo_metadata(root: Path) -> dict:
+ """The workspace as cargo resolves it, without the dependency graph."""
+ try:
+ result = subprocess.run(
+ [cargo(), "metadata", "--no-deps", "--format-version", "1"],
+ cwd=root, capture_output=True, text=True, check=False,
+ )
+ except OSError as error:
+ raise Failure(f"cannot run `{cargo()} metadata` in {root}: {error}") from error
+ if result.returncode != 0:
+ raise Failure(f"`cargo metadata` failed in {root}:\n{result.stderr}")
+ return json.loads(result.stdout)
+
+
+def is_published(package: dict) -> bool:
+ """A member that crates.io receives. `publish = false` reaches cargo
+ metadata as an empty list, and a member without the field as null."""
+ registries = package.get("publish")
+ return registries is None or REGISTRY in registries
+
+
+def publish_order(metadata: dict) -> list[dict]:
+ """The members crates.io receives, each after every workspace crate it
+ depends on through a normal or build dependency. Crates that are ready
+ at the same point go in name order."""
+ members = {package["name"]: package for package in metadata["packages"]}
+ published = {name: package for name, package in members.items() if is_published(package)}
+ waits_for: dict[str, set[str]] = {}
+ for name, package in published.items():
+ waits_for[name] = set()
+ for dependency in package["dependencies"]:
+ target = dependency["name"]
+ if dependency.get("kind") == "dev" or target not in members:
+ continue
+ if target not in published:
+ raise Failure(f"{name} depends on {target}, which crates.io does not receive, so crates.io cannot resolve it")
+ waits_for[name].add(target)
+ order: list[dict] = []
+ while waits_for:
+ ready = sorted(name for name, targets in waits_for.items() if not targets)
+ if not ready:
+ raise Failure("the workspace crates depend on each other in a cycle: " + " -> ".join(cycle(waits_for)))
+ order.append(published[ready[0]])
+ del waits_for[ready[0]]
+ for targets in waits_for.values():
+ targets.discard(ready[0])
+ return order
+
+
+def cycle(waits_for: dict[str, set[str]]) -> list[str]:
+ """One cycle among crates that all wait for another one of them, as a
+ path that starts and ends at the same crate."""
+ path = [min(waits_for)]
+ while path.count(path[-1]) < 2:
+ path.append(min(waits_for[path[-1]]))
+ return path[path.index(path[-1]):]
+
+
+def check_versions(order: list[dict], version: str | None) -> None:
+ """Stop when a crate to publish has another version than the release."""
+ if version is None:
+ return
+ other = [f"{package['name']} {package['version']}" for package in order if package["version"] != version]
+ if other:
+ raise Failure(f"the release is {version}, and these crates have another version: {', '.join(other)}")
+
+
+def index_path(name: str) -> str:
+ """The path of a crate's file in the sparse index (the Cargo book,
+ "Registry Index", "Index files")."""
+ name = name.lower()
+ if len(name) <= 2:
+ return f"{len(name)}/{name}"
+ if len(name) == 3:
+ return f"3/{name[0]}/{name}"
+ return f"{name[:2]}/{name[2:4]}/{name}"
+
+
+def fetch(url: str) -> bytes | None:
+ """The body at `url`, or None when the answer says that the crate does
+ not exist."""
+ request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
+ try:
+ with urllib.request.urlopen(request, timeout=TIMEOUT) as response:
+ return response.read()
+ except urllib.error.HTTPError as error:
+ if error.code in NOT_FOUND:
+ return None
+ raise Failure(f"{url} answered {error.code} {error.reason}") from error
+ except (OSError, http.client.HTTPException) as error:
+ raise Failure(f"cannot read {url}: {error!r}") from error
+
+
+def index_versions(name: str) -> dict[str, str] | None:
+ """The versions of a crate in the crates.io sparse index, yanked ones
+ included, each with the cksum that the index lists for it. None when
+ crates.io has no crate of that name. The cksum of a version is the
+ SHA-256 of the package that crates.io serves."""
+ url = INDEX + index_path(name)
+ body = fetch(url)
+ if body is None:
+ return None
+ try:
+ entries = [json.loads(line) for line in body.decode().splitlines() if line.strip()]
+ return {entry["vers"]: entry["cksum"] for entry in entries}
+ except (ValueError, KeyError, TypeError) as error:
+ raise Failure(f"{url} holds a line that is not an index entry: {error!r}") from error
+
+
+def trusted_publishing_only(name: str) -> bool:
+ """crates.io's trustpub_only flag of a crate. When it is true, crates.io
+ refuses a new version published with an API token."""
+ url = API + name
+ api_pause()
+ body = fetch(url)
+ if body is None:
+ raise Failure(f"{url} has no crate {name}, but the crates.io index lists it")
+ try:
+ flag = json.loads(body)["crate"]["trustpub_only"]
+ except (ValueError, KeyError, TypeError) as error:
+ raise Failure(f"{url} sent no crate.trustpub_only: {error!r}") from error
+ return flag is True
+
+
+def plan(order: list[dict], check_trusted_publishing: bool) -> tuple[list[tuple[str, str, str]], dict[str, str]]:
+ """(name, version, action) for each crate, in publish order, and the
+ index cksum of each skipped package by its file name. With
+ `check_trusted_publishing`, stop when a crate that crates.io has accepts
+ API tokens."""
+ steps = []
+ skipped = {}
+ open_to_tokens = []
+ for package in order:
+ name, version = package["name"], package["version"]
+ versions = index_versions(name)
+ if versions is None:
+ action = NEW
+ else:
+ action = SKIP if version in versions else PUBLISH
+ if action == SKIP:
+ skipped[package_file(name, version)] = versions[version]
+ if check_trusted_publishing and not trusted_publishing_only(name):
+ open_to_tokens.append(name)
+ steps.append((name, version, action))
+ if open_to_tokens:
+ raise Failure(
+ f"crates.io accepts API tokens for {', '.join(open_to_tokens)}. On the Settings tab of each, select "
+ f'"Require trusted publishing for all new versions" ({TOKEN_STEPS}).'
+ )
+ return steps, skipped
+
+
+def stop_at(steps: list[tuple[str, str, str]], new_crates: bool) -> int | None:
+ """The position of the crate where the run stops, or None. Without
+ --new-crates the run publishes the crates of the action publish, and
+ with it the new crates. The first crate of the other action stops it."""
+ wanted = NEW if new_crates else PUBLISH
+ for position, (_, _, action) in enumerate(steps):
+ if action not in (SKIP, wanted):
+ return position
+ return None
+
+
+def plan_facts(steps: list[tuple[str, str, str]]) -> tuple[int, str]:
+ """The number of crates of the action publish before the first new
+ crate, and the name of that crate ("" when there is none)."""
+ before = 0
+ for name, _, action in steps:
+ if action == NEW:
+ return before, name
+ if action == PUBLISH:
+ before += 1
+ return before, ""
+
+
+def print_plan(steps: list[tuple[str, str, str]], stop: int | None, new_crates: bool) -> None:
+ mode = "with an API token, new crates only" if new_crates else "with trusted publishing"
+ print(f"Publish order, {count(len(steps))}, {mode}:")
+ for position, (name, version, action) in enumerate(steps):
+ if stop is not None and position > stop:
+ verdict = "after the stop"
+ elif position == stop:
+ verdict = "stop here"
+ elif action == SKIP:
+ verdict = "skip"
+ else:
+ verdict = "publish"
+ print(f"{position + 1:3}. {name} {version}: {STATES[action]}, {verdict}")
+ planned = sum(1 for _, _, action in steps[:stop] if action != SKIP)
+ if stop is None:
+ print(f"This run publishes {count(planned)}.")
+ else:
+ print(f"This run publishes {count(planned)}, then stops before {steps[stop][0]}.")
+
+
+def write_plan_output(path: Path, steps: list[tuple[str, str, str]]) -> None:
+ before, first_new = plan_facts(steps)
+ try:
+ with path.open("a") as output:
+ output.write(f"publish_count={before}\nfirst_new={first_new}\n")
+ except OSError as error:
+ raise Failure(f"cannot write {path}: {error}") from error
+
+
+def read_sha256sums(path: Path) -> dict[str, str]:
+ """File name to SHA-256, from a list in the format of sha256sum."""
+ try:
+ text = path.read_text()
+ except OSError as error:
+ raise Failure(f"cannot read {path}: {error}") from error
+ sums = {}
+ for number, line in enumerate(text.splitlines(), 1):
+ if not line.strip():
+ continue
+ match = SHA256_LINE.fullmatch(line)
+ if match is None:
+ raise Failure(f"{path}:{number} is not a line of sha256sum output")
+ sums[match.group(2)] = match.group(1)
+ return sums
+
+
+def check_listed(sums: dict[str, str], path: Path, steps: list[tuple[str, str, str]]) -> None:
+ """Stop before the first upload unless the list has a line for each
+ crate of `steps`."""
+ missing = [package_file(name, version) for name, version, _ in steps if package_file(name, version) not in sums]
+ if missing:
+ raise Failure(f"{path} has no SHA-256 for {', '.join(missing)}, so nothing was published")
+
+
+def check_skipped(sums: dict[str, str], path: Path, skipped: dict[str, str]) -> None:
+ """Stop before the first upload when the index lists a version of the
+ release with another checksum than the verified package. crates.io then
+ serves other bytes for that version, for example after an upload that
+ the check after the upload stopped."""
+ other = [crate for crate, cksum in skipped.items() if cksum != sums[crate]]
+ if other:
+ raise Failure(
+ f"the crates.io index lists {', '.join(other)} with another checksum than {path}, so nothing was "
+ f"published. crates.io serves other bytes for that version. Follow {AFTER_UPLOAD}."
+ )
+
+
+def upload_dir(metadata: dict) -> Path:
+ """The directory where `cargo publish` writes the package that it
+ uploads. cargo packages into `package/tmp-crate` of its build directory.
+ Only `cargo package` copies the file on into the target directory
+ (cargo's src/cargo/ops/cargo_package/mod.rs)."""
+ build = metadata.get("build_directory")
+ if not build:
+ raise Failure("cargo metadata names no build_directory, so the script cannot find the packages that cargo "
+ "uploads. Use the toolchain of release-crates.yml.")
+ return Path(build) / "package" / "tmp-crate"
+
+
+def remove_upload(crate: Path, published: list[str]) -> None:
+ """Delete the file where `cargo publish` writes the package that it
+ uploads. The check after the upload then reads only a file that this
+ publish wrote, and not one that an earlier `cargo package` left."""
+ try:
+ crate.unlink(missing_ok=True)
+ except OSError as error:
+ raise Failure(f"cannot delete {crate} before the upload: {error}. {resume(published)}") from error
+
+
+def check_upload(crate: Path, expected: str, published: list[str]) -> None:
+ """Stop when the package that cargo uploaded is not the verified one."""
+ try:
+ digest = hashlib.sha256(crate.read_bytes()).hexdigest()
+ except OSError as error:
+ raise Failure(f"cannot read {crate}, the package that cargo uploaded: {error}. {resume(published)}") from error
+ if digest != expected:
+ raise Failure(
+ f"cargo uploaded {crate.name} with SHA-256 {digest}, and the verified package has {expected}. "
+ f"crates.io keeps the uploaded bytes. {done(published)} Do not run the script again for "
+ f"this release. Follow {AFTER_UPLOAD}."
+ )
+
+
+def run(command: list[str], root: Path) -> tuple[int, str]:
+ """Run `command` in `root`, echo its output as it arrives, and return
+ the exit code and the output."""
+ print("$ " + " ".join(command), flush=True)
+ try:
+ process = subprocess.Popen(command, cwd=root, stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT, text=True)
+ except OSError as error:
+ raise Failure(f"cannot run {command[0]}: {error}") from error
+ lines = []
+ with process:
+ for line in process.stdout:
+ sys.stdout.write(line)
+ sys.stdout.flush()
+ lines.append(line)
+ return process.returncode, "".join(lines)
+
+
+def retry_wait(output: str, current: datetime) -> int | None:
+ """The seconds to wait before the next try when `output` reports a 429
+ answer, or None for any other failure."""
+ if not TOO_MANY_REQUESTS.search(output):
+ return None
+ match = RETRY_AT.search(output)
+ if match:
+ try:
+ at = email.utils.parsedate_to_datetime(match.group(1))
+ except (TypeError, ValueError):
+ return DEFAULT_RETRY_WAIT
+ return max(0, math.ceil((at - current).total_seconds())) + RETRY_MARGIN
+ match = RETRY_SECONDS.search(output)
+ if match:
+ return int(match.group(1)) + RETRY_MARGIN
+ return DEFAULT_RETRY_WAIT
+
+
+def missing_dependency(output: str, published: list[str]) -> str | None:
+ """A crate that this run published and that cargo did not find in the
+ index, or None."""
+ for match in MISSING_DEPENDENCY.finditer(output):
+ name = match.group(1) or match.group(2)
+ if name in published:
+ return name
+ return None
+
+
+def done(published: list[str]) -> str:
+ return f"Published in this run: {', '.join(published) if published else 'none'}."
+
+
+def resume(published: list[str]) -> str:
+ return (f"{done(published)} Run the script again to continue, or in CI dispatch release-crates again. "
+ "It skips each version that crates.io has.")
+
+
+def publish(root: Path, steps: list[tuple[str, str, str]], no_verify: bool, max_wait: int,
+ sums: dict[str, str] | None, packages: Path | None) -> list[str]:
+ """Publish each crate of `steps` in order, and return the names of the
+ crates published."""
+ published: list[str] = []
+ waited = 0
+ for name, version, _ in steps:
+ command = [cargo(), "publish", "-p", name, "--locked"]
+ if no_verify:
+ command.append("--no-verify")
+ if sums is not None:
+ remove_upload(packages / package_file(name, version), published)
+ while True:
+ code, output = run(command, root)
+ if code == 0:
+ break
+ if DAILY_LIMIT.search(output):
+ raise Failure(f"crates.io refuses more versions of {name} for 24 hours, its limit for one crate. "
+ f"{resume(published)}")
+ wait = retry_wait(output, now())
+ reason = "crates.io answered 429"
+ if wait is None:
+ missing = missing_dependency(output, published)
+ if missing is None:
+ raise Failure(f"`cargo publish` failed for {name} {version} with exit code {code}. {resume(published)}")
+ wait = INDEX_RETRY_WAIT
+ reason = f"the crates.io index does not list {missing} yet"
+ if waited + wait > max_wait:
+ raise Failure(
+ f"{reason}, and a wait of {wait} s before {name} {version} takes the total wait to "
+ f"{waited + wait} s, over --max-wait {max_wait}. {resume(published)}"
+ )
+ waited += wait
+ print(f"{reason} for {name} {version}: waiting {wait} s, {waited} s of {max_wait} s in total",
+ flush=True)
+ sleep(wait)
+ published.append(name)
+ if sums is not None:
+ crate = package_file(name, version)
+ check_upload(packages / crate, sums[crate], published)
+ return published
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__.split("\n\n", 1)[0])
+ parser.add_argument("--new-crates", action="store_true",
+ help="publish only the crates that crates.io does not have, with an API token")
+ parser.add_argument("--dry-run", action="store_true", help="print the plan and publish nothing")
+ parser.add_argument("--no-verify", action="store_true", help="pass --no-verify to cargo publish")
+ parser.add_argument("--version", metavar="X.Y.Z",
+ help="the release version: stop when a crate to publish has another version")
+ parser.add_argument("--sha256sums", type=Path, metavar="FILE",
+ help="the SHA-256 list of the verified packages, in the format of sha256sum")
+ parser.add_argument("--expect-new", metavar="NAME",
+ help="the first new crate that the gate found: a stop before it exits 0")
+ parser.add_argument("--plan-output", type=Path, metavar="FILE",
+ help="append publish_count and first_new to FILE, in the format of GITHUB_OUTPUT")
+ parser.add_argument("--list-packages", action="store_true",
+ help="print the package file name of each crate in publish order, and read no index")
+ parser.add_argument("--max-wait", type=int, default=DEFAULT_MAX_WAIT, metavar="SECONDS",
+ help=f"cap on the total of the waits after 429 answers and for the index (default: {DEFAULT_MAX_WAIT})")
+ parser.add_argument("--root", type=Path, default=Path(__file__).resolve().parent.parent, metavar="DIR",
+ help="workspace root, or a directory in the workspace (default: this repository)")
+ args = parser.parse_args(argv)
+ if args.max_wait < 0:
+ parser.error("--max-wait must not be negative")
+ if args.new_crates and args.expect_new:
+ parser.error("--expect-new applies only without --new-crates")
+ root = args.root.resolve()
+ try:
+ metadata = cargo_metadata(root)
+ order = publish_order(metadata)
+ check_versions(order, args.version)
+ if args.list_packages:
+ for package in order:
+ print(package_file(package["name"], package["version"]))
+ return 0
+ sums = read_sha256sums(args.sha256sums) if args.sha256sums else None
+ steps, skipped = plan(order, check_trusted_publishing=not args.new_crates)
+ stop = stop_at(steps, args.new_crates)
+ to_publish = [step for step in steps[:stop] if step[2] != SKIP]
+ print_plan(steps, stop, args.new_crates)
+ if args.expect_new and args.expect_new not in {name for name, _, _ in steps}:
+ raise Failure(f"--expect-new names {args.expect_new}, which is not a crate of this release")
+ if args.plan_output:
+ write_plan_output(args.plan_output, steps)
+ packages = None
+ if sums is not None:
+ check_listed(sums, args.sha256sums, [step for step in steps if step[2] == SKIP or step in to_publish])
+ check_skipped(sums, args.sha256sums, skipped)
+ packages = upload_dir(metadata)
+ if args.dry_run:
+ print("Dry run: nothing was published.")
+ return 0
+ published = publish(root, to_publish, args.no_verify, args.max_wait, sums, packages)
+ except Failure as failure:
+ sys.stdout.flush()
+ print(f"error: {failure}", file=sys.stderr)
+ return 1
+ skipped = sum(1 for _, _, action in steps[:stop] if action == SKIP)
+ print(f"Published {count(len(published))}, and skipped {count(skipped)} that crates.io has.")
+ if stop is None:
+ if args.new_crates:
+ print(TOKEN_DONE.format(steps=TOKEN_STEPS))
+ elif args.expect_new:
+ print(f"The gate found {args.expect_new} new, but crates.io has it now, so this run did not stop before it.")
+ return 0
+ name = steps[stop][0]
+ if args.new_crates:
+ print(TOKEN_STOP.format(name=name, steps=TOKEN_STEPS))
+ return 1
+ if args.expect_new == name:
+ print(f"Stopped before {name}, the new crate that the gate found.")
+ return 0
+ print(FIRST_PUBLISH.format(name=name, steps=TOKEN_STEPS), flush=True)
+ if args.expect_new:
+ print(f"error: the gate found {args.expect_new} as the first new crate, and this run stops before {name}",
+ file=sys.stderr)
+ return 1
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/devtools/publish_crates_test.py b/devtools/publish_crates_test.py
new file mode 100644
index 000000000..791ba6583
--- /dev/null
+++ b/devtools/publish_crates_test.py
@@ -0,0 +1,1251 @@
+# Copyright 2026 ExtendDB contributors
+# SPDX-License-Identifier: Apache-2.0
+"""Tests of devtools/publish-crates.
+
+The tests replace the script's calls to `cargo metadata`, to the crates.io
+sparse index and API, to `cargo publish`, to the clock and to sleep with
+fakes. So they use no network and publish nothing. CargoMetadataTest runs
+`cargo metadata` on workspaces in a temporary directory, and RunTest starts
+Python child processes.
+
+ python3 -m unittest discover -s devtools -p '*_test.py'
+"""
+import contextlib
+import email.utils
+import hashlib
+import http.client
+import importlib.util
+import io
+import json
+import os
+import sys
+import tempfile
+import unittest
+import urllib.error
+from datetime import UTC, datetime, timedelta
+from importlib.machinery import SourceFileLoader
+from pathlib import Path
+from unittest import mock
+
+SCRIPT = Path(__file__).resolve().parent / "publish-crates"
+START = datetime(2026, 11, 10, 14, 0, 0, tzinfo=UTC)
+TOKEN = "cio-test-token-that-must-not-appear"
+INDEX = "https://index.crates.io/"
+API = "https://crates.io/api/v1/crates/"
+
+
+def load_script():
+ """Import the script, which has no .py suffix, as a module."""
+ loader = SourceFileLoader("publish_crates", str(SCRIPT))
+ spec = importlib.util.spec_from_loader(loader.name, loader)
+ module = importlib.util.module_from_spec(spec)
+ loader.exec_module(module)
+ return module
+
+
+def member(name: str, *dependencies, version: str = "1.0.0", publish=None) -> dict:
+ """A workspace member as `cargo metadata --no-deps` lists it. Each
+ dependency is a name (a normal dependency) or a (name, kind) pair."""
+ listed = []
+ for dependency in dependencies:
+ target, kind = dependency if isinstance(dependency, tuple) else (dependency, None)
+ listed.append({"name": target, "kind": kind})
+ return {"name": name, "version": version, "publish": publish, "dependencies": listed}
+
+
+def package_bytes(name: str, version: str) -> bytes:
+ """The bytes the fake `cargo publish` writes for a package."""
+ return f"package {name} {version}\n".encode()
+
+
+def too_many_requests(at: datetime) -> tuple[int, str]:
+ """cargo's exit code and output for crates.io's 429 answer."""
+ return 101, (
+ "error: failed to publish to registry at https://crates.io\n\nCaused by:\n"
+ " the remote server responded with an error (status 429 Too Many Requests): You have "
+ "published too many new crates in a short period of time. Please try again after "
+ f"{email.utils.format_datetime(at, usegmt=True)} and see https://crates.io/docs/rate-limits "
+ "for more details.\n"
+ )
+
+
+def code_answer(status: int, *headers: str) -> tuple[int, str]:
+ """cargo's exit code and output for an answer without crates.io's JSON
+ errors."""
+ lines = "".join(f"\t{header}\n" for header in headers)
+ return 101, (
+ "error: failed to publish to registry at https://crates.io\n\nCaused by:\n"
+ f" failed to get a 200 OK response, got {status}\nheaders:\n{lines}body:\nslow down\n"
+ )
+
+
+# crates.io's answer for its limit on the versions of one crate
+# (src/controllers/krate/publish.rs), as cargo prints it.
+DAILY_LIMIT_ANSWER = (101, (
+ "error: failed to publish to registry at https://crates.io\n\nCaused by:\n"
+ " the remote server responded with an error (status 429 Too Many Requests): You have "
+ "published too many versions of this crate in the last 24 hours\n"
+))
+
+
+def missing_crate(name: str) -> tuple[int, str]:
+ """cargo's answer when the index does not know a dependency."""
+ return 101, (
+ "error: failed to prepare local package for uploading\n\nCaused by:\n"
+ f" no matching package named `{name}` found\n location searched: crates.io index\n"
+ " required by package `b v1.0.0 (/work/crates/b)`\n"
+ )
+
+
+def missing_version(name: str, version: str) -> tuple[int, str]:
+ """cargo's answer when the index does not have a dependency's version."""
+ return 101, (
+ "error: failed to prepare local package for uploading\n\nCaused by:\n"
+ f" failed to select a version for the requirement `{name} = \"={version}\"`\n"
+ " candidate versions found which didn't match: 0.9.0\n location searched: crates.io index\n"
+ )
+
+
+class FakeTestCase(unittest.TestCase):
+ """Runs the script's main with fakes for cargo, the index, the API, the
+ clock and sleep. Tests fill `members`, `index`, `cksums`, `trustpub`
+ and `results`. A publish that succeeds adds its version and the cksum
+ of its bytes to the index, and writes its package where cargo writes
+ it, as crates.io and cargo do."""
+
+ def setUp(self) -> None:
+ self.module = load_script()
+ self.tmp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.tmp.cleanup)
+ self.target = Path(self.tmp.name) / "target"
+ self.build = Path(self.tmp.name) / "build"
+ self.metadata_extra: dict = {"build_directory": str(self.build)}
+ self.members: list[dict] = []
+ # Crate name to the versions crates.io has. A missing name is a
+ # crate crates.io does not have.
+ self.index: dict[str, list[str]] = {}
+ # (crate name, version) to the cksum that the index lists, when it
+ # is not the SHA-256 of package_bytes(). A fake publish records the
+ # SHA-256 of the bytes it uploads.
+ self.cksums: dict[tuple[str, str], str] = {}
+ # Crate name to its trustpub_only flag. A crate in `index` at the
+ # start requires trusted publishing unless a test says otherwise. A
+ # crate that a fake publish creates does not.
+ self.trustpub: dict[str, bool] = {}
+ # Crate name to the results of its successive publishes: (exit code,
+ # output) pairs, or functions that return one. A publish with no
+ # result left succeeds.
+ self.results: dict[str, list] = {}
+ # Crate name to the bytes a successful publish writes, when they are
+ # not package_bytes(). A publish of a crate in no_file writes no file.
+ self.uploaded: dict[str, bytes] = {}
+ self.no_file: set[str] = set()
+ self.commands: list[list[str]] = []
+ self.fetched: list[str] = []
+ self.api_reads: list[str] = []
+ self.slept: list[int] = []
+ self.pauses = 0
+ self.clock = START
+ for name, fake in (("cargo_metadata", self.fake_metadata), ("fetch", self.fake_fetch),
+ ("run", self.fake_run), ("now", lambda: self.clock), ("sleep", self.fake_sleep),
+ ("api_pause", self.fake_pause)):
+ patcher = mock.patch.object(self.module, name, fake)
+ patcher.start()
+ self.addCleanup(patcher.stop)
+ environment = mock.patch.dict(os.environ, {"CARGO_REGISTRY_TOKEN": TOKEN})
+ environment.start()
+ self.addCleanup(environment.stop)
+ os.environ.pop("CARGO", None)
+
+ def fake_metadata(self, root: Path) -> dict:
+ return {"packages": self.members, "target_directory": str(self.target), **self.metadata_extra}
+
+ def fake_fetch(self, url: str) -> bytes | None:
+ if url.startswith(API):
+ name = url[len(API):]
+ self.api_reads.append(name)
+ if name not in self.index:
+ return None
+ return json.dumps({"crate": {"name": name, "trustpub_only": self.trustpub.get(name, True)}}).encode()
+ self.fetched.append(url)
+ name = url.rsplit("/", 1)[1]
+ if name not in self.index:
+ return None
+ return "".join(json.dumps({"name": name, "vers": v, "cksum": self.cksum_of(name, v)}) + "\n"
+ for v in self.index[name]).encode()
+
+ def cksum_of(self, name: str, version: str) -> str:
+ return self.cksums.get((name, version), hashlib.sha256(package_bytes(name, version)).hexdigest())
+
+ def version_of(self, name: str) -> str:
+ return next(package["version"] for package in self.members if package["name"] == name)
+
+ def fake_run(self, command: list[str], root: Path) -> tuple[int, str]:
+ self.commands.append(command)
+ name = command[command.index("-p") + 1]
+ results = self.results.get(name)
+ result = results.pop(0) if results else (0, f"Uploaded {name}\n")
+ if callable(result):
+ result = result()
+ if result[0] == 0:
+ version = self.version_of(name)
+ if name not in self.index:
+ self.index[name] = []
+ self.trustpub[name] = False
+ self.index[name].append(version)
+ uploaded = self.uploaded.get(name, package_bytes(name, version))
+ self.cksums[(name, version)] = hashlib.sha256(uploaded).hexdigest()
+ if name not in self.no_file:
+ # cargo publish writes the package it uploads here, and only
+ # cargo package copies it into the target directory.
+ crate = self.build / "package" / "tmp-crate" / f"{name}-{version}.crate"
+ crate.parent.mkdir(parents=True, exist_ok=True)
+ crate.write_bytes(uploaded)
+ stale = self.target / "package" / crate.name
+ stale.parent.mkdir(parents=True, exist_ok=True)
+ stale.write_bytes(package_bytes(name, version))
+ return result
+
+ def fake_sleep(self, seconds: int) -> None:
+ self.slept.append(seconds)
+ self.clock += timedelta(seconds=seconds)
+
+ def fake_pause(self) -> None:
+ self.pauses += 1
+
+ def main(self, *argv: str) -> tuple[int, str, str]:
+ out, err = io.StringIO(), io.StringIO()
+ with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
+ code = self.module.main(list(argv))
+ output, errors = out.getvalue(), err.getvalue()
+ self.assertNotIn(TOKEN, output + errors)
+ return code, output, errors
+
+ def published(self) -> list[str]:
+ return [command[command.index("-p") + 1] for command in self.commands]
+
+ def later(self, seconds: float):
+ """A result: crates.io's 429 answer that asks for a wait of `seconds`
+ from the time of the publish."""
+ return lambda: too_many_requests(self.clock + timedelta(seconds=seconds))
+
+ def sums_file(self, *wrong: str, leave_out: tuple[str, ...] = ()) -> Path:
+ """A SHA-256 list of the members' packages. The crates in `wrong` get
+ another hash, and the crates in `leave_out` get no line."""
+ lines = []
+ for package in self.members:
+ name, version = package["name"], package["version"]
+ if name in leave_out:
+ continue
+ digest = "0" * 64 if name in wrong else hashlib.sha256(package_bytes(name, version)).hexdigest()
+ lines.append(f"{digest} {name}-{version}.crate\n")
+ path = Path(self.tmp.name) / "SHA256SUMS"
+ path.write_text("".join(lines))
+ return path
+
+ def three_crates(self) -> None:
+ """a, then b, which depends on a, then c, which depends on b. crates.io
+ has each crate at 0.9.0."""
+ self.members = [member("c", "b"), member("b", "a"), member("a")]
+ self.index = {"a": ["0.9.0"], "b": ["0.9.0"], "c": ["0.9.0"]}
+
+ def new_crate_between(self) -> None:
+ """a, b, then the new crate n, then c, which depends on n. crates.io
+ has a, b and c at 0.9.0."""
+ self.members = [member("c", "n"), member("n", "b"), member("b", "a"), member("a")]
+ self.index = {"a": ["0.9.0"], "b": ["0.9.0"], "c": ["0.9.0"]}
+
+
+class OrderTest(FakeTestCase):
+ def order(self) -> list[str]:
+ return [package["name"] for package in self.module.publish_order({"packages": self.members})]
+
+ def test_a_crate_comes_after_each_crate_it_depends_on(self) -> None:
+ self.members = [member("alpha", ("mid", "build")), member("mid", "zeta"), member("zeta")]
+ self.assertEqual(self.order(), ["zeta", "mid", "alpha"])
+
+ def test_crates_that_are_ready_together_go_in_name_order(self) -> None:
+ self.members = [member("c"), member("b", "a"), member("a")]
+ self.assertEqual(self.order(), ["a", "b", "c"])
+
+ def test_a_dev_dependency_does_not_order_the_crates(self) -> None:
+ self.members = [member("a", ("b", "dev")), member("b", "a")]
+ self.assertEqual(self.order(), ["a", "b"])
+
+ def test_a_dependency_outside_the_workspace_is_ignored(self) -> None:
+ self.members = [member("b", "serde", "a"), member("a", ("tokio", "build"))]
+ self.assertEqual(self.order(), ["a", "b"])
+
+ def test_a_member_with_publish_false_is_left_out(self) -> None:
+ self.members = [member("x", "a", publish=[]), member("a")]
+ self.assertEqual(self.order(), ["a"])
+
+ def test_a_member_published_only_to_another_registry_is_left_out(self) -> None:
+ self.members = [member("x", "a", publish=["my-registry"]), member("a")]
+ self.assertEqual(self.order(), ["a"])
+
+ def test_a_member_whose_publish_list_names_crates_io_is_kept(self) -> None:
+ self.members = [member("x", "a", publish=["my-registry", "crates-io"]), member("a", publish=["crates-io"])]
+ self.assertEqual(self.order(), ["a", "x"])
+
+ def test_a_dependency_on_a_member_that_crates_io_does_not_receive_fails(self) -> None:
+ for publish in ([], ["my-registry"]):
+ with self.subTest(publish=publish):
+ self.members = [member("a", "x"), member("x", publish=publish)]
+ with self.assertRaisesRegex(self.module.Failure, "a depends on x, which crates.io does not receive"):
+ self.order()
+
+ def test_a_cycle_fails_and_names_the_crates_in_it(self) -> None:
+ self.members = [member("a", "b"), member("b", "c"), member("c", "a"), member("z")]
+ with self.assertRaisesRegex(self.module.Failure, "in a cycle: a -> b -> c -> a$"):
+ self.order()
+
+ def test_the_cycle_named_leaves_out_a_crate_that_only_waits_for_it(self) -> None:
+ self.members = [member("a", "c"), member("c", "d"), member("d", "c")]
+ with self.assertRaisesRegex(self.module.Failure, "in a cycle: c -> d -> c$"):
+ self.order()
+
+ def test_the_cycle_named_follows_the_smallest_name_when_a_crate_waits_for_two(self) -> None:
+ self.members = [member("a", "c", "b"), member("b", "a"), member("c", "a")]
+ with self.assertRaisesRegex(self.module.Failure, "in a cycle: a -> b -> a$"):
+ self.order()
+
+ def test_the_cycle_named_starts_at_the_smallest_name_and_not_the_first_member(self) -> None:
+ self.members = [member("y", "x"), member("x", "y")]
+ with self.assertRaisesRegex(self.module.Failure, "in a cycle: x -> y -> x$"):
+ self.order()
+
+ def test_a_cycle_stops_the_run_before_the_index_is_read(self) -> None:
+ self.members = [member("a", "b"), member("b", "a")]
+ code, _, errors = self.main("--dry-run")
+ self.assertEqual(code, 1)
+ self.assertIn("error: the workspace crates depend on each other in a cycle: a -> b -> a", errors)
+ self.assertEqual((self.fetched, self.api_reads, self.commands), ([], [], []))
+
+ def test_the_plan_lists_the_crates_in_publish_order(self) -> None:
+ self.three_crates()
+ code, output, _ = self.main("--dry-run")
+ self.assertEqual(code, 0, output)
+ self.assertIn("Publish order, 3 crates, with trusted publishing:\n"
+ " 1. a 1.0.0: new version, publish\n 2. b 1.0.0: new version, publish\n"
+ " 3. c 1.0.0: new version, publish\nThis run publishes 3 crates.\n", output)
+
+
+class CargoMetadataTest(unittest.TestCase):
+ """The script against `cargo metadata` of workspaces in a temporary
+ directory. Needs cargo, and no network."""
+
+ def setUp(self) -> None:
+ self.module = load_script()
+ self.tmp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.tmp.cleanup)
+ self.root = Path(self.tmp.name)
+ self.write("Cargo.toml", '[workspace]\nresolver = "2"\nmembers = ["crates/*"]\n')
+
+ def write(self, relative: str, text: str) -> None:
+ path = self.root / relative
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(text)
+
+ def crate(self, name: str, extra: str = "") -> None:
+ self.write(f"crates/{name}/Cargo.toml",
+ f'[package]\nname = "ex-{name}"\nversion = "0.1.0"\nedition = "2021"\n{extra}')
+ self.write(f"crates/{name}/src/lib.rs", "")
+
+ def order(self) -> list[str]:
+ return [package["name"] for package in self.module.publish_order(self.module.cargo_metadata(self.root))]
+
+ def run_main(self, *argv: str, env: dict | None = None) -> tuple[int, str]:
+ out, err = io.StringIO(), io.StringIO()
+ with mock.patch.dict(os.environ, env or {}), contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
+ code = self.module.main(list(argv))
+ return code, out.getvalue() + err.getvalue()
+
+ def test_the_order_of_a_workspace_that_cargo_reads(self) -> None:
+ self.crate("zeta", '\n[dev-dependencies]\nex-alpha = { path = "../alpha" }\n')
+ self.crate("mid", '\n[dependencies]\nex-zeta = { path = "../zeta", version = "=0.1.0" }\n')
+ self.crate("alpha", '\n[build-dependencies]\nex-mid = { path = "../mid", version = "=0.1.0" }\n')
+ self.crate("private", 'publish = false\n\n[dependencies]\nex-zeta = { path = "../zeta" }\n')
+ self.assertEqual(self.order(), ["ex-zeta", "ex-mid", "ex-alpha"])
+
+ def test_target_specific_optional_and_renamed_dependencies_order_the_crates(self) -> None:
+ # Without its dependencies, each crate would go in name order.
+ self.crate("zeta")
+ self.crate("yak", "\n[target.'cfg(unix)'.dependencies]\n"
+ 'ex-zeta = { path = "../zeta", version = "=0.1.0" }\n'
+ "\n[target.'cfg(windows)'.dependencies]\n"
+ 'ex-zeta = { path = "../zeta", version = "=0.1.0" }\n')
+ self.crate("xen", '\n[dependencies]\nex-yak = { path = "../yak", version = "=0.1.0", optional = true }\n')
+ self.crate("wolf", '\n[dependencies]\nalias = { package = "ex-xen", path = "../xen", version = "=0.1.0" }\n')
+ self.assertEqual(self.order(), ["ex-zeta", "ex-yak", "ex-xen", "ex-wolf"])
+
+ def test_publish_lists_of_a_workspace_that_cargo_reads(self) -> None:
+ self.crate("open")
+ self.crate("named", 'publish = ["crates-io"]\n')
+ self.crate("other", 'publish = ["my-registry"]\n')
+ self.assertEqual(self.order(), ["ex-named", "ex-open"])
+
+ def test_root_may_be_a_member_directory(self) -> None:
+ self.crate("zeta")
+ self.crate("mid", '\n[dependencies]\nex-zeta = { path = "../zeta", version = "=0.1.0" }\n')
+ code, output = self.run_main("--list-packages", "--root", str(self.root / "crates" / "mid"))
+ self.assertEqual(code, 0, output)
+ self.assertEqual(output, "ex-zeta-0.1.0.crate\nex-mid-0.1.0.crate\n")
+
+ def test_a_failing_cargo_metadata_is_reported(self) -> None:
+ self.write("Cargo.toml", "[workspace\n")
+ code, output = self.run_main("--dry-run", "--root", str(self.root))
+ self.assertEqual(code, 1)
+ self.assertIn("error: `cargo metadata` failed in", output)
+
+ def test_a_cargo_that_cannot_start_is_reported(self) -> None:
+ code, output = self.run_main("--dry-run", "--root", str(self.root), env={"CARGO": str(self.root / "no-cargo")})
+ self.assertEqual(code, 1)
+ self.assertIn(f"error: cannot run `{self.root / 'no-cargo'} metadata`", output)
+ self.assertNotIn("Traceback", output)
+
+
+class IndexTest(unittest.TestCase):
+ def setUp(self) -> None:
+ self.module = load_script()
+
+ def test_the_index_path_follows_the_cargo_book(self) -> None:
+ cases = {"a": "1/a", "ab": "2/ab", "abc": "3/a/abc", "abcd": "ab/cd/abcd",
+ "Extenddb-Core": "ex/te/extenddb-core"}
+ for name, path in cases.items():
+ with self.subTest(name=name):
+ self.assertEqual(self.module.index_path(name), path)
+
+ def test_index_versions_reads_each_line_of_the_crate_file(self) -> None:
+ body = (b'{"name":"example-a","vers":"0.1.0","cksum":"1"}\n'
+ b'{"name":"example-a","vers":"0.2.0","cksum":"2","yanked":true}\n\n')
+ with mock.patch.object(self.module, "fetch", return_value=body) as fetch:
+ self.assertEqual(self.module.index_versions("example-a"), {"0.1.0": "1", "0.2.0": "2"})
+ fetch.assert_called_once_with("https://index.crates.io/ex/am/example-a")
+
+ def test_index_versions_is_none_for_a_crate_crates_io_does_not_have(self) -> None:
+ with mock.patch.object(self.module, "fetch", return_value=None):
+ self.assertIsNone(self.module.index_versions("example-a"))
+
+ def test_an_index_file_that_is_not_an_index_fails_with_its_url(self) -> None:
+ bodies = {"not UTF-8": b"\xff\xfe\n", "not JSON": b"\n", "no vers": b'{"name":"example-a"}\n',
+ "no cksum": b'{"name":"example-a","vers":"0.1.0"}\n', "a JSON list": b'["0.1.0"]\n'}
+ for label, body in bodies.items():
+ with self.subTest(body=label), mock.patch.object(self.module, "fetch", return_value=body):
+ with self.assertRaisesRegex(self.module.Failure,
+ "^https://index.crates.io/ex/am/example-a holds a line that is not an index entry"):
+ self.module.index_versions("example-a")
+
+ def fetch(self, answer, url: str = "https://index.crates.io/ex/am/example-a") -> tuple:
+ """fetch with urlopen replaced. `answer` is the body to return, an
+ exception for urlopen to raise, or a function that the body's read
+ calls. Returns the result and urlopen's call."""
+ class Response(io.BytesIO):
+ def read(self, *args):
+ return answer() if callable(answer) else super().read(*args)
+
+ def urlopen(request, timeout):
+ calls.append((request, timeout))
+ if isinstance(answer, Exception):
+ raise answer
+ return Response(b"" if callable(answer) else answer)
+
+ calls = []
+ with mock.patch.object(self.module.urllib.request, "urlopen", urlopen):
+ result = self.module.fetch(url)
+ return result, calls[0]
+
+ def test_fetch_sends_a_user_agent_and_a_timeout(self) -> None:
+ result, (request, timeout) = self.fetch(b"line\n")
+ self.assertEqual(result, b"line\n")
+ self.assertEqual(request.full_url, "https://index.crates.io/ex/am/example-a")
+ self.assertIn("ExtendDB/extenddb", request.get_header("User-agent"))
+ self.assertEqual(timeout, 30)
+
+ def test_fetch_reads_404_410_and_451_as_no_crate(self) -> None:
+ for status in (404, 410, 451):
+ with self.subTest(status=status):
+ error = urllib.error.HTTPError("https://index.crates.io/x", status, "gone", {}, None)
+ self.assertIsNone(self.fetch(error)[0])
+
+ def test_fetch_fails_on_any_other_answer(self) -> None:
+ for status in (403, 500):
+ with self.subTest(status=status):
+ error = urllib.error.HTTPError("https://index.crates.io/x", status, "no", {}, None)
+ with self.assertRaisesRegex(self.module.Failure, f"answered {status} no"):
+ self.fetch(error)
+
+ def test_fetch_fails_when_the_index_cannot_be_reached(self) -> None:
+ with self.assertRaisesRegex(self.module.Failure, "^cannot read https://index.crates.io/ex/am/example-a"):
+ self.fetch(urllib.error.URLError("no route to host"))
+
+ def test_fetch_fails_with_its_url_on_a_broken_http_answer(self) -> None:
+ def incomplete():
+ raise http.client.IncompleteRead(b"par", 10)
+
+ for label, answer in (("bad status line", http.client.BadStatusLine("HTTP/1.1 9x")),
+ ("incomplete body", incomplete)):
+ with self.subTest(answer=label):
+ with self.assertRaisesRegex(self.module.Failure, "^cannot read https://index.crates.io/ex/am/example-a"):
+ self.fetch(answer)
+
+ def test_trusted_publishing_only_reads_the_crate_from_the_api(self) -> None:
+ for flag in (True, False):
+ with self.subTest(trustpub_only=flag):
+ body = json.dumps({"crate": {"name": "example-a", "trustpub_only": flag}, "versions": []}).encode()
+ with mock.patch.object(self.module, "fetch", return_value=body) as fetch, \
+ mock.patch.object(self.module, "api_pause") as pause:
+ self.assertIs(self.module.trusted_publishing_only("example-a"), flag)
+ fetch.assert_called_once_with("https://crates.io/api/v1/crates/example-a")
+ pause.assert_called_once_with()
+
+ def test_trusted_publishing_only_needs_the_value_true(self) -> None:
+ body = json.dumps({"crate": {"name": "example-a", "trustpub_only": "true"}}).encode()
+ with mock.patch.object(self.module, "fetch", return_value=body), mock.patch.object(self.module, "api_pause"):
+ self.assertIs(self.module.trusted_publishing_only("example-a"), False)
+
+ def test_trusted_publishing_only_fails_on_an_answer_without_the_flag(self) -> None:
+ bodies = {"not JSON": b"", "no crate": b'{"errors": []}', "no flag": b'{"crate": {"name": "example-a"}}',
+ "a JSON list": b"[]"}
+ for label, body in bodies.items():
+ with self.subTest(body=label), mock.patch.object(self.module, "fetch", return_value=body), \
+ mock.patch.object(self.module, "api_pause"):
+ with self.assertRaisesRegex(self.module.Failure,
+ "^https://crates.io/api/v1/crates/example-a sent no crate.trustpub_only"):
+ self.module.trusted_publishing_only("example-a")
+
+ def test_trusted_publishing_only_fails_on_a_crate_the_api_does_not_have(self) -> None:
+ with mock.patch.object(self.module, "fetch", return_value=None), mock.patch.object(self.module, "api_pause"):
+ with self.assertRaisesRegex(self.module.Failure, "has no crate example-a, but the crates.io index lists it"):
+ self.module.trusted_publishing_only("example-a")
+
+
+class PlanTest(FakeTestCase):
+ def test_a_version_on_crates_io_is_skipped(self) -> None:
+ self.three_crates()
+ self.index["a"] = ["0.9.0", "1.0.0"]
+ code, output, _ = self.main()
+ self.assertEqual(code, 0, output)
+ self.assertEqual(self.published(), ["b", "c"])
+ self.assertIn(" 1. a 1.0.0: on crates.io, skip\n 2. b 1.0.0: new version, publish\n", output)
+ self.assertIn("Published 2 crates, and skipped 1 crate that crates.io has.", output)
+
+ def test_a_version_below_the_newest_is_skipped(self) -> None:
+ self.three_crates()
+ self.index["a"] = ["0.9.0", "1.0.0", "1.1.0"]
+ code, output, _ = self.main()
+ self.assertEqual(code, 0, output)
+ self.assertEqual(self.published(), ["b", "c"])
+
+ def test_the_plan_reads_each_crate_from_the_index(self) -> None:
+ self.three_crates()
+ self.main("--dry-run")
+ self.assertEqual(self.fetched, [INDEX + "1/a", INDEX + "1/b", INDEX + "1/c"])
+
+ def test_a_crate_of_another_version_stops_the_run_before_the_index_is_read(self) -> None:
+ self.members = [member("b", "a", version="1.0.1"), member("a"), member("c", version="2.0.0")]
+ code, _, errors = self.main("--version", "1.0.0")
+ self.assertEqual(code, 1)
+ self.assertIn("error: the release is 1.0.0, and these crates have another version: b 1.0.1, c 2.0.0", errors)
+ self.assertEqual((self.fetched, self.commands), ([], []))
+
+ def test_crates_of_the_release_version_pass_the_version_check(self) -> None:
+ self.three_crates()
+ code, output, _ = self.main("--version", "1.0.0")
+ self.assertEqual(code, 0, output)
+ self.assertEqual(self.published(), ["a", "b", "c"])
+
+ def test_a_dry_run_prints_the_plan_and_the_stop_and_publishes_nothing(self) -> None:
+ self.new_crate_between()
+ self.index["a"].append("1.0.0")
+ code, output, errors = self.main("--dry-run", "--no-verify")
+ self.assertEqual(code, 0, output + errors)
+ self.assertEqual(self.commands, [])
+ self.assertIn(" 1. a 1.0.0: on crates.io, skip\n 2. b 1.0.0: new version, publish\n"
+ " 3. n 1.0.0: new crate, stop here\n 4. c 1.0.0: new version, after the stop\n"
+ "This run publishes 1 crate, then stops before n.\nDry run: nothing was published.\n", output)
+ self.assertEqual(errors, "")
+
+ def test_a_dry_run_with_new_crates_prints_its_stop(self) -> None:
+ self.new_crate_between()
+ self.index["b"].append("1.0.0")
+ code, output, _ = self.main("--dry-run", "--new-crates")
+ self.assertEqual(code, 0, output)
+ self.assertEqual(self.commands, [])
+ self.assertIn("with an API token, new crates only:\n 1. a 1.0.0: new version, stop here\n"
+ " 2. b 1.0.0: on crates.io, after the stop\n 3. n 1.0.0: new crate, after the stop\n", output)
+ self.assertIn("This run publishes 0 crates, then stops before a.\n", output)
+
+ def test_a_dry_run_of_a_first_publish_plans_every_crate(self) -> None:
+ self.three_crates()
+ self.index = {}
+ code, output, _ = self.main("--dry-run", "--new-crates")
+ self.assertEqual(code, 0, output)
+ self.assertEqual(self.commands, [])
+ self.assertIn(" 3. c 1.0.0: new crate, publish\nThis run publishes 3 crates.\n", output)
+
+ def test_the_plan_output_counts_the_crates_before_the_first_new_crate(self) -> None:
+ self.new_crate_between()
+ self.index["a"].append("1.0.0")
+ output_file = Path(self.tmp.name) / "github-output"
+ output_file.write_text("version=1.0.0\n")
+ code, output, _ = self.main("--dry-run", "--plan-output", str(output_file))
+ self.assertEqual(code, 0, output)
+ self.assertEqual(output_file.read_text(), "version=1.0.0\npublish_count=1\nfirst_new=n\n")
+
+ def test_the_plan_output_of_a_release_without_a_new_crate(self) -> None:
+ self.three_crates()
+ output_file = Path(self.tmp.name) / "github-output"
+ self.main("--dry-run", "--plan-output", str(output_file))
+ self.assertEqual(output_file.read_text(), "publish_count=3\nfirst_new=\n")
+
+ def test_the_plan_output_of_a_first_publish(self) -> None:
+ self.three_crates()
+ self.index = {}
+ output_file = Path(self.tmp.name) / "github-output"
+ self.main("--dry-run", "--plan-output", str(output_file))
+ self.assertEqual(output_file.read_text(), "publish_count=0\nfirst_new=a\n")
+
+ def test_a_plan_output_that_cannot_be_written_fails(self) -> None:
+ self.three_crates()
+ code, _, errors = self.main("--dry-run", "--plan-output", str(Path(self.tmp.name) / "missing" / "out"))
+ self.assertEqual(code, 1)
+ self.assertIn("error: cannot write", errors)
+
+ def test_a_negative_wait_cap_is_refused(self) -> None:
+ with self.assertRaises(SystemExit) as caught, contextlib.redirect_stderr(io.StringIO()):
+ self.module.main(["--max-wait", "-1"])
+ self.assertEqual(caught.exception.code, 2)
+
+ def test_expect_new_with_new_crates_is_refused(self) -> None:
+ err = io.StringIO()
+ with self.assertRaises(SystemExit) as caught, contextlib.redirect_stderr(err):
+ self.module.main(["--new-crates", "--expect-new", "n"])
+ self.assertEqual(caught.exception.code, 2)
+ self.assertIn("--expect-new applies only without --new-crates", err.getvalue())
+
+
+class TrustedPublishingTest(FakeTestCase):
+ def test_each_crate_on_crates_io_is_read_from_the_api_after_a_pause(self) -> None:
+ self.new_crate_between()
+ code, output, _ = self.main("--dry-run")
+ self.assertEqual(code, 0, output)
+ self.assertEqual(self.api_reads, ["a", "b", "c"])
+ self.assertEqual(self.pauses, 3)
+
+ def test_a_crate_that_accepts_api_tokens_stops_the_run(self) -> None:
+ self.three_crates()
+ self.trustpub.update({"a": False, "c": False})
+ for argv in (["--dry-run"], []):
+ with self.subTest(argv=argv):
+ code, _, errors = self.main(*argv)
+ self.assertEqual(code, 1)
+ self.assertEqual(self.commands, [])
+ self.assertIn('error: crates.io accepts API tokens for a, c. On the Settings tab of each, select '
+ '"Require trusted publishing for all new versions"', errors)
+
+ def test_the_mode_new_crates_reads_no_api(self) -> None:
+ self.new_crate_between()
+ self.trustpub["a"] = False
+ code, output, _ = self.main("--dry-run", "--new-crates")
+ self.assertEqual(code, 0, output)
+ self.assertEqual((self.api_reads, self.pauses), ([], 0))
+
+
+class ModeTest(FakeTestCase):
+ def test_trusted_publishing_stops_before_the_first_new_crate(self) -> None:
+ self.new_crate_between()
+ code, output, errors = self.main("--no-verify")
+ self.assertEqual(code, 1, output)
+ self.assertEqual(self.published(), ["a", "b"])
+ self.assertIn("This run publishes 2 crates, then stops before n.\n", output)
+ self.assertIn("Published 2 crates, and skipped 0 crates that crates.io has.\n", output)
+ self.assertIn("crates.io has no crate named n, and trusted publishing cannot create a\ncrate.", output)
+ self.assertIn("runs devtools/publish-crates --new-crates for it", output)
+ self.assertIn('packaging/README.md, "Publishing new crates with an API token"', output)
+ self.assertEqual(errors, "")
+
+ def test_the_summary_counts_only_the_crates_before_the_stop(self) -> None:
+ self.new_crate_between()
+ self.index["a"].append("1.0.0")
+ self.index["c"].append("1.0.0")
+ code, output, _ = self.main()
+ self.assertEqual(code, 1, output)
+ self.assertEqual(self.published(), ["b"])
+ self.assertIn(" 4. c 1.0.0: on crates.io, after the stop\n", output)
+ self.assertIn("Published 1 crate, and skipped 1 crate that crates.io has.\n", output)
+
+ def test_a_stop_before_the_expected_new_crate_exits_0(self) -> None:
+ self.new_crate_between()
+ code, output, errors = self.main("--expect-new", "n")
+ self.assertEqual(code, 0, output + errors)
+ self.assertEqual(self.published(), ["a", "b"])
+ self.assertIn("Stopped before n, the new crate that the gate found.", output)
+ self.assertNotIn("trusted publishing cannot create", output)
+
+ def test_a_stop_before_another_new_crate_than_expected_exits_1(self) -> None:
+ self.new_crate_between()
+ code, output, errors = self.main("--expect-new", "c")
+ self.assertEqual(code, 1)
+ self.assertEqual(self.published(), ["a", "b"])
+ self.assertIn("crates.io has no crate named n", output)
+ self.assertIn("error: the gate found c as the first new crate, and this run stops before n", errors)
+
+ def test_expect_new_must_name_a_crate_of_the_release(self) -> None:
+ self.new_crate_between()
+ for argv in (["--dry-run"], []):
+ with self.subTest(argv=argv):
+ code, output, errors = self.main("--expect-new", "x", *argv)
+ self.assertEqual(code, 1)
+ self.assertEqual(self.commands, [])
+ self.assertIn("error: --expect-new names x, which is not a crate of this release", errors)
+ self.assertNotIn("Dry run", output)
+
+ def test_a_run_that_does_not_stop_says_why_the_expected_stop_did_not_happen(self) -> None:
+ # The gate found n new. A token run published n, and its owners
+ # required trusted publishing, before this run started.
+ self.new_crate_between()
+ self.index["n"] = ["1.0.0"]
+ for argv, published in ((["--sha256sums", str(self.sums_file())], ["a", "b", "c"]), ([], [])):
+ with self.subTest(argv=argv):
+ self.commands.clear()
+ code, output, errors = self.main("--expect-new", "n", *argv)
+ self.assertEqual(code, 0, output + errors)
+ self.assertEqual(self.published(), published)
+ self.assertIn(" 3. n 1.0.0: on crates.io, skip\n", output)
+ self.assertTrue(output.endswith(
+ "The gate found n new, but crates.io has it now, so this run did not stop before it.\n"), output)
+ self.assertNotIn("Stopped before", output)
+ self.assertEqual(errors, "")
+
+ def test_a_run_without_expect_new_that_does_not_stop_prints_no_line_about_it(self) -> None:
+ self.three_crates()
+ code, output, _ = self.main()
+ self.assertEqual(code, 0, output)
+ self.assertNotIn("The gate found", output)
+
+ def test_new_crates_stops_before_the_first_crate_that_crates_io_has(self) -> None:
+ self.members = [member("z", "y"), member("y", "x"), member("x")]
+ self.index = {"y": ["0.9.0"]}
+ code, output, _ = self.main("--new-crates", "--no-verify")
+ self.assertEqual(code, 1, output)
+ self.assertEqual(self.published(), ["x"])
+ self.assertEqual(self.commands[0], ["cargo", "publish", "-p", "x", "--locked", "--no-verify"])
+ self.assertIn(" 2. y 1.0.0: new version, stop here\n 3. z 1.0.0: new crate, after the stop\n", output)
+ self.assertIn("crates.io has y, and an API token must not publish it.", output)
+ self.assertIn("dispatch release-crates again with the tag", output)
+
+ def test_new_crates_never_publishes_a_crate_that_crates_io_has(self) -> None:
+ self.members = [member("n", "a"), member("a")]
+ self.index = {"a": ["0.9.0"]}
+ code, output, _ = self.main("--new-crates")
+ self.assertEqual(code, 1, output)
+ self.assertEqual(self.commands, [])
+ self.assertIn("This run publishes 0 crates, then stops before a.\n", output)
+
+ def test_new_crates_without_a_stop_exits_0(self) -> None:
+ self.three_crates()
+ self.index = {}
+ code, output, _ = self.main("--new-crates")
+ self.assertEqual(code, 0, output)
+ self.assertEqual(self.published(), ["a", "b", "c"])
+ self.assertIn("Revoke the token now", output)
+ self.assertIn("dispatch release-crates again with the tag, so that its index and smoke", output)
+
+
+class SequenceTest(FakeTestCase):
+ """Whole releases, run by run, against the fake crates.io that each
+ publish updates."""
+
+ def run_mode(self, *argv: str) -> tuple[int, list[str]]:
+ self.commands.clear()
+ code, output, errors = self.main("--no-verify", *argv)
+ return code, self.published()
+
+ def test_the_first_publish(self) -> None:
+ self.three_crates()
+ self.index = {}
+ code, output, _ = self.main("--dry-run")
+ self.assertEqual(code, 0, output)
+ self.assertIn("This run publishes 0 crates, then stops before a.", output)
+ self.assertEqual(self.run_mode("--new-crates"), (0, ["a", "b", "c"]))
+ # Each crate refuses API tokens only after a maintainer sets it.
+ self.assertEqual(self.run_mode(), (1, []))
+ self.trustpub.update(dict.fromkeys("abc", True))
+ self.assertEqual(self.run_mode(), (0, []))
+
+ def test_a_release_that_adds_a_crate(self) -> None:
+ self.new_crate_between()
+ self.assertEqual(self.run_mode(), (1, ["a", "b"]))
+ self.assertEqual(self.run_mode("--new-crates"), (1, ["n"]))
+ self.trustpub["n"] = True
+ self.assertEqual(self.run_mode(), (0, ["c"]))
+ self.assertEqual(self.index, {"a": ["0.9.0", "1.0.0"], "b": ["0.9.0", "1.0.0"], "c": ["0.9.0", "1.0.0"],
+ "n": ["1.0.0"]})
+
+ def test_two_new_crates_with_a_crate_between_them(self) -> None:
+ self.members = [member("c", "n2"), member("n2", "b"), member("b", "n1"), member("n1", "a"), member("a")]
+ self.index = {"a": ["0.9.0"], "b": ["0.9.0"], "c": ["0.9.0"]}
+ self.assertEqual(self.run_mode(), (1, ["a"]))
+ self.assertEqual(self.run_mode("--new-crates"), (1, ["n1"]))
+ self.trustpub["n1"] = True
+ self.assertEqual(self.run_mode(), (1, ["b"]))
+ self.assertEqual(self.run_mode("--new-crates"), (1, ["n2"]))
+ self.trustpub["n2"] = True
+ self.assertEqual(self.run_mode(), (0, ["c"]))
+
+ def test_two_new_crates_next_to_each_other(self) -> None:
+ self.members = [member("c", "n2"), member("n2", "n1"), member("n1", "a"), member("a")]
+ self.index = {"a": ["0.9.0"], "c": ["0.9.0"]}
+ self.assertEqual(self.run_mode(), (1, ["a"]))
+ self.assertEqual(self.run_mode("--new-crates"), (1, ["n1", "n2"]))
+ self.trustpub.update({"n1": True, "n2": True})
+ self.assertEqual(self.run_mode(), (0, ["c"]))
+
+ def test_the_run_after_a_token_run_names_the_crate_that_accepts_tokens(self) -> None:
+ self.new_crate_between()
+ self.run_mode()
+ self.run_mode("--new-crates")
+ code, output, errors = self.main()
+ self.assertEqual(code, 1)
+ self.assertIn("error: crates.io accepts API tokens for n.", errors)
+
+
+class PublishTest(FakeTestCase):
+ def test_each_crate_is_published_locked_in_order(self) -> None:
+ self.three_crates()
+ code, output, _ = self.main()
+ self.assertEqual(code, 0, output)
+ self.assertEqual(self.commands, [["cargo", "publish", "-p", name, "--locked"] for name in "abc"])
+ # A complete run with trusted publishing prints none of the steps
+ # for an API token.
+ for text in ("Revoke the token", "an API token must not publish", "trusted publishing cannot create",
+ "Publishing new crates with an API token"):
+ self.assertNotIn(text, output)
+
+ def test_no_verify_reaches_cargo(self) -> None:
+ self.three_crates()
+ self.main("--no-verify")
+ self.assertEqual(self.commands, [["cargo", "publish", "-p", name, "--locked", "--no-verify"] for name in "abc"])
+
+ def test_the_cargo_variable_names_the_cargo_program(self) -> None:
+ self.three_crates()
+ os.environ["CARGO"] = "/opt/rust/bin/cargo"
+ self.main()
+ self.assertEqual({command[0] for command in self.commands}, {"/opt/rust/bin/cargo"})
+
+ def test_a_failed_publish_stops_the_run(self) -> None:
+ self.three_crates()
+ self.results["b"] = [(101, "error: failed to prepare local package for uploading\n")]
+ code, _, errors = self.main()
+ self.assertEqual(code, 1)
+ self.assertEqual(self.published(), ["a", "b"])
+ self.assertEqual(self.slept, [])
+ self.assertIn("error: `cargo publish` failed for b 1.0.0 with exit code 101. Published in this run: a. "
+ "Run the script again to continue, or in CI dispatch release-crates again.", errors)
+
+ def test_a_failed_first_publish_reports_that_none_was_published(self) -> None:
+ self.three_crates()
+ self.results["a"] = [(101, "error: crate version `1.0.0` is already uploaded\n")]
+ _, _, errors = self.main()
+ self.assertIn("Published in this run: none.", errors)
+
+ def test_uploads_that_match_the_verified_list_pass(self) -> None:
+ self.three_crates()
+ code, output, errors = self.main("--sha256sums", str(self.sums_file()))
+ self.assertEqual(code, 0, output + errors)
+ self.assertEqual(self.published(), ["a", "b", "c"])
+
+ def test_an_upload_that_differs_from_the_verified_list_stops_the_run(self) -> None:
+ self.three_crates()
+ self.uploaded["b"] = b"other bytes\n"
+ code, _, errors = self.main("--sha256sums", str(self.sums_file()))
+ self.assertEqual(code, 1)
+ self.assertEqual(self.published(), ["a", "b"])
+ other = hashlib.sha256(b"other bytes\n").hexdigest()
+ verified = hashlib.sha256(package_bytes("b", "1.0.0")).hexdigest()
+ self.assertIn(f"error: cargo uploaded b-1.0.0.crate with SHA-256 {other}, and the verified package has "
+ f"{verified}. crates.io keeps the uploaded bytes. Published in this run: a, b. Do not run the "
+ 'script again for this release. Follow packaging/README.md, "A failure after the upload".', errors)
+
+ def test_an_upload_without_a_package_file_stops_the_run(self) -> None:
+ self.three_crates()
+ self.no_file.add("a")
+ code, _, errors = self.main("--sha256sums", str(self.sums_file()))
+ self.assertEqual(code, 1)
+ self.assertEqual(self.published(), ["a"])
+ self.assertIn(f"error: cannot read {self.build / 'package' / 'tmp-crate' / 'a-1.0.0.crate'}, the package that "
+ "cargo uploaded", errors)
+
+ def test_the_hash_check_needs_the_build_directory_before_any_upload(self) -> None:
+ self.three_crates()
+ self.metadata_extra = {}
+ code, _, errors = self.main("--sha256sums", str(self.sums_file()))
+ self.assertEqual(code, 1)
+ self.assertEqual(self.commands, [])
+ self.assertIn("error: cargo metadata names no build_directory", errors)
+ self.commands.clear()
+ code, output, _ = self.main()
+ self.assertEqual(code, 0, output)
+ self.assertEqual(self.published(), ["a", "b", "c"])
+
+ def test_a_list_without_a_crate_to_publish_stops_the_run_before_any_upload(self) -> None:
+ self.three_crates()
+ sums = self.sums_file(leave_out=("b", "c"))
+ code, _, errors = self.main("--sha256sums", str(sums))
+ self.assertEqual(code, 1)
+ self.assertEqual(self.commands, [])
+ self.assertIn(f"error: {sums} has no SHA-256 for b-1.0.0.crate, c-1.0.0.crate, so nothing was published", errors)
+
+ def test_a_stale_package_from_an_earlier_cargo_package_does_not_pass_the_check(self) -> None:
+ # The publish job runs cargo package first, which leaves the verified
+ # bytes where cargo publish writes its upload. A cargo that wrote its
+ # upload elsewhere would leave that file in place.
+ self.three_crates()
+ stale = self.build / "package" / "tmp-crate" / "a-1.0.0.crate"
+ stale.parent.mkdir(parents=True)
+ stale.write_bytes(package_bytes("a", "1.0.0"))
+ self.no_file.add("a")
+ code, _, errors = self.main("--sha256sums", str(self.sums_file()))
+ self.assertEqual(code, 1)
+ self.assertEqual(self.published(), ["a"])
+ self.assertFalse(stale.exists())
+ self.assertIn(f"error: cannot read {stale}, the package that cargo uploaded", errors)
+
+ def test_without_a_list_the_upload_file_is_left_alone(self) -> None:
+ self.three_crates()
+ kept = self.build / "package" / "tmp-crate" / "a-1.0.0.crate"
+ kept.parent.mkdir(parents=True)
+ kept.write_bytes(b"left by cargo package\n")
+ self.no_file.add("a")
+ code, output, _ = self.main()
+ self.assertEqual(code, 0, output)
+ self.assertEqual(kept.read_bytes(), b"left by cargo package\n")
+
+ def test_an_upload_file_that_cannot_be_deleted_stops_the_run_before_its_upload(self) -> None:
+ self.three_crates()
+ blocked = self.build / "package" / "tmp-crate" / "b-1.0.0.crate"
+ (blocked / "inside").mkdir(parents=True)
+ code, _, errors = self.main("--sha256sums", str(self.sums_file()))
+ self.assertEqual(code, 1)
+ self.assertEqual(self.published(), ["a"])
+ self.assertIn(f"error: cannot delete {blocked} before the upload:", errors)
+ self.assertIn("Published in this run: a.", errors)
+
+ def test_a_list_without_a_crate_that_the_run_skips_stops_the_run_before_any_upload(self) -> None:
+ self.three_crates()
+ self.index["a"].append("1.0.0")
+ sums = self.sums_file(leave_out=("a",))
+ code, output, errors = self.main("--sha256sums", str(sums))
+ self.assertEqual(code, 1)
+ self.assertEqual(self.commands, [])
+ self.assertIn(f"error: {sums} has no SHA-256 for a-1.0.0.crate, so nothing was published", errors)
+
+ def test_a_skipped_crate_whose_index_cksum_matches_the_list_passes(self) -> None:
+ self.three_crates()
+ self.index["a"].append("1.0.0")
+ code, output, errors = self.main("--sha256sums", str(self.sums_file()))
+ self.assertEqual(code, 0, output + errors)
+ self.assertEqual(self.published(), ["b", "c"])
+
+ def test_a_skipped_crate_with_other_bytes_on_crates_io_stops_the_run_before_any_upload(self) -> None:
+ # An earlier run uploaded a and b, and the check after the upload
+ # stopped it at b. A new run must not publish c on top of b.
+ self.three_crates()
+ self.index["a"].append("1.0.0")
+ self.index["b"].append("1.0.0")
+ self.cksums[("b", "1.0.0")] = "f" * 64
+ sums = self.sums_file()
+ for argv in (["--dry-run"], []):
+ with self.subTest(argv=argv):
+ code, output, errors = self.main("--sha256sums", str(sums), *argv)
+ self.assertEqual(code, 1)
+ self.assertEqual(self.commands, [])
+ self.assertNotIn("Dry run", output)
+ self.assertIn(f"error: the crates.io index lists b-1.0.0.crate with another checksum than {sums}, "
+ "so nothing was published. crates.io serves other bytes for that version. Follow "
+ 'packaging/README.md, "A failure after the upload".', errors)
+
+ def test_each_skipped_crate_after_the_stop_is_checked_too(self) -> None:
+ self.new_crate_between()
+ self.index["c"].append("1.0.0")
+ self.cksums[("c", "1.0.0")] = "f" * 64
+ code, _, errors = self.main("--sha256sums", str(self.sums_file()), "--expect-new", "n")
+ self.assertEqual(code, 1)
+ self.assertEqual(self.commands, [])
+ self.assertIn("lists c-1.0.0.crate with another checksum", errors)
+
+ def test_a_dry_run_with_a_list_without_a_line_stops_before_the_dry_run_output(self) -> None:
+ self.three_crates()
+ sums = self.sums_file(leave_out=("b",))
+ code, output, errors = self.main("--dry-run", "--sha256sums", str(sums))
+ self.assertEqual(code, 1)
+ self.assertEqual(self.commands, [])
+ self.assertNotIn("Dry run: nothing was published.", output)
+ self.assertIn(f"error: {sums} has no SHA-256 for b-1.0.0.crate, so nothing was published", errors)
+
+ def test_a_dry_run_with_a_complete_list_passes(self) -> None:
+ self.three_crates()
+ code, output, errors = self.main("--dry-run", "--sha256sums", str(self.sums_file()))
+ self.assertEqual(code, 0, output + errors)
+ self.assertEqual(self.commands, [])
+ self.assertIn("Dry run: nothing was published.", output)
+
+ def test_a_list_that_is_not_sha256sum_output_stops_the_run(self) -> None:
+ self.three_crates()
+ sums = Path(self.tmp.name) / "SHA256SUMS"
+ sums.write_text(f"{'0' * 64} a-1.0.0.crate\n\nnot a line\n")
+ code, _, errors = self.main("--sha256sums", str(sums))
+ self.assertEqual(code, 1)
+ self.assertIn(f"error: {sums}:3 is not a line of sha256sum output", errors)
+ self.assertEqual((self.fetched, self.commands), ([], []))
+
+ def test_a_list_line_with_text_after_the_name_or_upper_case_hex_stops_the_run(self) -> None:
+ self.three_crates()
+ digest = hashlib.sha256(package_bytes("a", "1.0.0")).hexdigest()
+ lines = {"text after the name": f"{digest} a-1.0.0.crate extra",
+ "a blank after the name": f"{digest} a-1.0.0.crate ",
+ "upper-case hex": f"{digest.upper()} a-1.0.0.crate"}
+ for label, line in lines.items():
+ with self.subTest(line=label):
+ sums = Path(self.tmp.name) / "SHA256SUMS"
+ sums.write_text(line + "\n")
+ code, _, errors = self.main("--sha256sums", str(sums))
+ self.assertEqual(code, 1)
+ self.assertIn(f"error: {sums}:1 is not a line of sha256sum output", errors)
+ self.assertEqual((self.fetched, self.commands), ([], []))
+
+ def test_a_list_in_binary_mode_is_read(self) -> None:
+ self.three_crates()
+ lines = "".join(f"{hashlib.sha256(package_bytes(n, '1.0.0')).hexdigest()} *{n}-1.0.0.crate\n" for n in "abc")
+ sums = Path(self.tmp.name) / "SHA256SUMS"
+ sums.write_text(lines)
+ code, output, errors = self.main("--sha256sums", str(sums))
+ self.assertEqual(code, 0, output + errors)
+
+ def test_a_list_that_cannot_be_read_stops_the_run(self) -> None:
+ self.three_crates()
+ code, _, errors = self.main("--sha256sums", str(Path(self.tmp.name) / "missing"))
+ self.assertEqual(code, 1)
+ self.assertIn("error: cannot read", errors)
+
+
+class ListPackagesTest(FakeTestCase):
+ def test_the_list_names_each_package_in_publish_order(self) -> None:
+ self.members = [member("c", "b"), member("x", publish=[]), member("b", "a"), member("a"),
+ member("y", publish=["my-registry"])]
+ code, output, _ = self.main("--list-packages")
+ self.assertEqual(code, 0, output)
+ self.assertEqual(output, "a-1.0.0.crate\nb-1.0.0.crate\nc-1.0.0.crate\n")
+ self.assertEqual((self.fetched, self.api_reads, self.commands), ([], [], []))
+
+ def test_the_list_applies_the_version_check(self) -> None:
+ self.members = [member("a", version="1.0.1")]
+ code, output, errors = self.main("--list-packages", "--version", "1.0.0")
+ self.assertEqual(code, 1)
+ self.assertEqual(output, "")
+ self.assertIn("these crates have another version: a 1.0.1", errors)
+
+
+class RateLimitTest(FakeTestCase):
+ def setUp(self) -> None:
+ super().setUp()
+ self.three_crates()
+
+ def test_a_429_waits_until_the_time_crates_io_gives(self) -> None:
+ self.results["b"] = [self.later(600)]
+ code, output, _ = self.main()
+ self.assertEqual(code, 0, output)
+ self.assertEqual(self.slept, [605])
+ self.assertEqual(self.published(), ["a", "b", "b", "c"])
+ self.assertIn("crates.io answered 429 for b 1.0.0: waiting 605 s, 605 s of 7200 s in total", output)
+
+ def test_a_time_already_past_waits_the_margin_only(self) -> None:
+ self.results["b"] = [self.later(-30)]
+ self.main()
+ self.assertEqual(self.slept, [5])
+
+ def test_part_of_a_second_rounds_up(self) -> None:
+ self.clock += timedelta(milliseconds=500)
+ self.results["b"] = [lambda: too_many_requests(START + timedelta(seconds=60))]
+ self.main()
+ self.assertEqual(self.slept, [65])
+
+ def test_a_retry_after_header_in_seconds(self) -> None:
+ self.results["b"] = [code_answer(429, "content-type: text/plain", "retry-after: 30")]
+ self.main()
+ self.assertEqual(self.slept, [35])
+
+ def test_a_retry_after_header_with_a_date(self) -> None:
+ self.results["b"] = [code_answer(429, "retry-after: Tue, 10 Nov 2026 14:02:00 GMT")]
+ self.main()
+ self.assertEqual(self.slept, [125])
+
+ def test_a_429_without_a_time_waits_one_minute(self) -> None:
+ self.results["b"] = [code_answer(429, "content-type: text/plain")]
+ self.main()
+ self.assertEqual(self.slept, [60])
+
+ def test_a_429_with_a_date_that_does_not_exist_waits_one_minute(self) -> None:
+ self.results["b"] = [(101, "(status 429 Too Many Requests): Please try again after Sat, 31 Feb 2026 10:00:00 GMT")]
+ self.main()
+ self.assertEqual(self.slept, [60])
+
+ def test_other_answers_are_not_tried_again(self) -> None:
+ answers = {
+ "403": (101, "the remote server responded with an error (status 403 Forbidden): New versions of this "
+ "crate can only be published using Trusted Publishing"),
+ "500": code_answer(500, "retry-after: 30"),
+ "429 in a file name": (101, "error: failed to read `/work/crates/b/tests/http-429.json`: no such file\n"),
+ "429 in a version": (101, "error: crate version `0.429.0` is already uploaded\n"),
+ }
+ for label, answer in answers.items():
+ with self.subTest(answer=label):
+ self.commands.clear()
+ self.index = {"a": ["0.9.0"], "b": ["0.9.0"], "c": ["0.9.0"]}
+ self.results["b"] = [answer]
+ code, _, _ = self.main()
+ self.assertEqual(code, 1)
+ self.assertEqual(self.published(), ["a", "b"])
+ self.assertEqual(self.slept, [])
+
+ def test_the_daily_limit_for_one_crate_stops_the_run_at_once(self) -> None:
+ self.results["b"] = [DAILY_LIMIT_ANSWER]
+ code, _, errors = self.main()
+ self.assertEqual(code, 1)
+ self.assertEqual(self.slept, [])
+ self.assertEqual(self.published(), ["a", "b"])
+ self.assertIn("error: crates.io refuses more versions of b for 24 hours, its limit for one crate. "
+ "Published in this run: a. Run the script again to continue", errors)
+
+ def test_the_total_wait_is_capped(self) -> None:
+ self.results["b"] = [self.later(600), self.later(600)]
+ code, _, errors = self.main("--max-wait", "1000")
+ self.assertEqual(code, 1)
+ self.assertEqual(self.slept, [605])
+ self.assertEqual(self.published(), ["a", "b", "b"])
+ self.assertIn("error: crates.io answered 429, and a wait of 605 s before b 1.0.0 takes the total wait to "
+ "1210 s, over --max-wait 1000. Published in this run: a.", errors)
+
+ def test_waits_add_up_across_crates(self) -> None:
+ self.results = {"a": [self.later(95)], "b": [self.later(95)], "c": [self.later(95)]}
+ code, _, errors = self.main("--max-wait", "250")
+ self.assertEqual(code, 1)
+ self.assertEqual(self.slept, [100, 100])
+ self.assertIn("takes the total wait to 300 s, over --max-wait 250. Published in this run: a, b.", errors)
+
+ def test_a_total_wait_equal_to_the_cap_is_allowed(self) -> None:
+ self.results["b"] = [self.later(300), self.later(300)]
+ code, output, _ = self.main("--max-wait", "610")
+ self.assertEqual(code, 0, output)
+ self.assertEqual(self.slept, [305, 305])
+ self.assertIn("waiting 305 s, 610 s of 610 s in total", output)
+
+ def test_a_cap_of_zero_stops_at_the_first_429(self) -> None:
+ self.results["a"] = [self.later(0)]
+ code, _, errors = self.main("--max-wait", "0")
+ self.assertEqual(code, 1)
+ self.assertEqual(self.slept, [])
+ self.assertIn("over --max-wait 0. Published in this run: none.", errors)
+
+ def test_the_default_cap_is_two_hours(self) -> None:
+ for wait, code in ((7195, 0), (7196, 1)):
+ with self.subTest(wait=wait):
+ self.commands.clear()
+ self.slept.clear()
+ self.index = {"a": ["0.9.0"], "b": ["0.9.0"], "c": ["0.9.0"]}
+ self.results["b"] = [self.later(wait)]
+ self.assertEqual(self.main()[0], code)
+
+ def test_a_crate_this_run_published_and_the_index_does_not_list_is_tried_again(self) -> None:
+ for label, answer in (("a crate", missing_crate("a")), ("a version", missing_version("a", "1.0.0"))):
+ with self.subTest(missing=label):
+ self.commands.clear()
+ self.slept.clear()
+ self.index = {"a": ["0.9.0"], "b": ["0.9.0"], "c": ["0.9.0"]}
+ self.results["b"] = [answer]
+ code, output, _ = self.main()
+ self.assertEqual(code, 0, output)
+ self.assertEqual(self.slept, [30])
+ self.assertEqual(self.published(), ["a", "b", "b", "c"])
+ self.assertIn("the crates.io index does not list a yet for b 1.0.0: waiting 30 s, 30 s of 7200 s "
+ "in total", output)
+
+ def test_a_missing_crate_that_this_run_did_not_publish_is_not_tried_again(self) -> None:
+ self.index["a"].append("1.0.0")
+ self.results["b"] = [missing_crate("a")]
+ code, _, errors = self.main()
+ self.assertEqual(code, 1)
+ self.assertEqual(self.slept, [])
+ self.assertIn("`cargo publish` failed for b 1.0.0", errors)
+
+ def test_the_waits_for_the_index_count_against_the_cap(self) -> None:
+ self.results["b"] = [missing_crate("a")] * 5
+ code, _, errors = self.main("--max-wait", "100")
+ self.assertEqual(code, 1)
+ self.assertEqual(self.slept, [30, 30, 30])
+ self.assertIn("error: the crates.io index does not list a yet, and a wait of 30 s before b 1.0.0 takes the "
+ "total wait to 120 s, over --max-wait 100. Published in this run: a.", errors)
+
+
+class HelpersTest(unittest.TestCase):
+ def setUp(self) -> None:
+ self.module = load_script()
+
+ def test_cargo_metadata_reads_the_members_only(self) -> None:
+ answer = mock.Mock(returncode=0, stdout='{"packages": []}', stderr="")
+ with mock.patch.dict(os.environ, {"CARGO": "/opt/rust/bin/cargo"}), \
+ mock.patch.object(self.module.subprocess, "run", return_value=answer) as run:
+ self.assertEqual(self.module.cargo_metadata(Path("/work")), {"packages": []})
+ self.assertEqual(run.call_args.args[0], ["/opt/rust/bin/cargo", "metadata", "--no-deps", "--format-version", "1"])
+ self.assertEqual(run.call_args.kwargs["cwd"], Path("/work"))
+
+ def test_the_clock_is_in_utc(self) -> None:
+ output = too_many_requests(datetime.now(UTC) + timedelta(seconds=60))[1]
+ self.assertIn(self.module.retry_wait(output, self.module.now()), range(60, 67))
+
+ def test_sleep_sleeps(self) -> None:
+ with mock.patch.object(self.module.time, "sleep") as sleep:
+ self.module.sleep(3)
+ sleep.assert_called_once_with(3)
+
+ def test_the_api_pause_is_one_second(self) -> None:
+ with mock.patch.object(self.module.time, "sleep") as sleep:
+ self.module.api_pause()
+ sleep.assert_called_once_with(1)
+
+ def test_counts_of_crates(self) -> None:
+ self.assertEqual([self.module.count(n) for n in (0, 1, 2)], ["0 crates", "1 crate", "2 crates"])
+
+
+class RunTest(unittest.TestCase):
+ """run() with Python child processes in place of cargo."""
+
+ def setUp(self) -> None:
+ self.module = load_script()
+
+ def run_child(self, code: str) -> tuple[tuple[int, str], str]:
+ out = io.StringIO()
+ command = [sys.executable, "-c", code]
+ with contextlib.redirect_stdout(out):
+ result = self.module.run(command, Path.cwd())
+ return result, out.getvalue()
+
+ def test_run_echoes_the_output_and_returns_it(self) -> None:
+ result, echoed = self.run_child("print('one'); print('two'); raise SystemExit(3)")
+ self.assertEqual(result, (3, "one\ntwo\n"))
+ self.assertTrue(echoed.startswith(f"$ {sys.executable} -c "), echoed)
+ self.assertTrue(echoed.endswith("one\ntwo\n"), echoed)
+
+ def test_run_merges_the_error_output(self) -> None:
+ result, _ = self.run_child("import sys; sys.stderr.write('got 429\\n')")
+ self.assertEqual(result, (0, "got 429\n"))
+
+ def test_run_reports_a_program_it_cannot_start(self) -> None:
+ with contextlib.redirect_stdout(io.StringIO()), \
+ self.assertRaisesRegex(self.module.Failure, "cannot run /nonexistent/cargo"):
+ self.module.run(["/nonexistent/cargo", "publish"], Path.cwd())
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/docs/adr/0029-crates-io-packaging.md b/docs/adr/0029-crates-io-packaging.md
index 3db2c36e4..4860386cd 100644
--- a/docs/adr/0029-crates-io-packaging.md
+++ b/docs/adr/0029-crates-io-packaging.md
@@ -100,11 +100,12 @@ binary crate, and a Cargo feature selects its backend.
`devtools/check-crate-packages`. The check fails on a pin that is not
exact, or on an internal dev-dependency with a version. It also fails on a
missing LICENSE, NOTICE or description, and on a package or manifest field
- over crates.io's limits. A pin left at an older version also makes every
- cargo command that resolves dependencies fail.
-- Compiling each package from its tarball takes minutes, so that build
- belongs to the release workflow, before the upload, and not to each pull
- request.
+ over crates.io's limits. A crate whose version is not the workspace
+ version fails too. A pin left at an older version also makes every cargo
+ command that resolves dependencies fail.
+- Compiling each package from its tarball takes minutes, so that build runs
+ in the `verify` job of `.github/workflows/release-crates.yml`, before the
+ upload, and not on each pull request.
- A release bump changes the 13 pins together with the workspace version.
- Every release publishes all 14 crates, including the crates that did not
change.
@@ -117,10 +118,17 @@ binary crate, and a Cargo feature selects its backend.
from other crates.
- The license notices now list the workspace crates under the text of the
repository's LICENSE file, with a link to the repository.
-- Publishing is follow-up work. It needs a release workflow, and a first
- publish with an API token, because trusted publishing cannot create a crate.
- Each crate then needs its owners and a trusted publisher entry. Each crate
- added to the workspace later needs its own first publish.
+- `.github/workflows/release-crates.yml` publishes each release after
+ `promote-image`, through trusted publishing, with `devtools/publish-crates`.
+ Trusted publishing cannot create a crate. So a maintainer publishes the
+ first version of each crate with an API token and
+ `devtools/publish-crates --new-crates`. That mode publishes only the
+ crates that crates.io does not have. Each crate then needs its owners and
+ a trusted publisher entry.
+- A release that adds one crate takes three runs. The workflow publishes the
+ crates before the new crate, and it stops. A maintainer publishes the new
+ crate with an API token, and a second dispatch publishes the rest.
+ `packaging/README.md`, "Releasing to crates.io", describes the steps.
- A version on crates.io can never be replaced. A yank only stops new lock
files from choosing it.
diff --git a/packaging/README.md b/packaging/README.md
index dc945f86a..9ff62c2f7 100644
--- a/packaging/README.md
+++ b/packaging/README.md
@@ -98,13 +98,15 @@ deliberate second step):
1. Land the release state on `main`. Bump the workspace version and the 13
`=X.Y.Z` pins in the root `[workspace.dependencies]`, regenerate the four
license notices, and bump `npm/package.json` to match. Tag that commit
- `vX.Y.Z` and push the tag. This is the same tag the container release
- consumes; the two publish workflows are dispatched independently against
- it. When the release ships a catalog migration, the same state raises
- `RELEASED_THROUGH` to the last migration and pastes the checksum lines the
- checksum test prints (CONTRIBUTING.md, "Adding a catalog migration"); the
- gate jobs of the two container release workflows run
- `devtools/check-released-through` on the tag and refuse it otherwise.
+ `vX.Y.Z` and push the tag. The container release uses the same tag. A
+ maintainer dispatches the container and npm publish workflows
+ independently against it. The crates.io workflow runs last, after
+ `promote-image`. For a release that ships a catalog migration, the
+ release commit also raises `RELEASED_THROUGH` to the last migration. That
+ commit holds the checksum lines that the checksum test prints
+ (CONTRIBUTING.md, "Adding a catalog migration"). The gate jobs of the
+ release workflows run `devtools/check-released-through` on the tag and
+ refuse it otherwise.
2. Dispatch the `npm-publish` workflow from `main` with the tag as input.
The gate validates shape, existence, ancestry against `main`, and the
version match. Five native runners build, smoke test, and run the full
@@ -130,6 +132,259 @@ automation token scoped to the `@extenddb` scope with publish permission only,
and create the GitHub environment `npm` with deployment branch rule `main`,
required reviewers, and the token as `NPM_TOKEN`.
+## Releasing to crates.io
+
+The workspace crates publish to crates.io together, at the workspace
+version. Each internal dependency requires exactly that version
+([ADR-0029](../docs/adr/0029-crates-io-packaging.md)). crates.io has no
+candidate stage, and it never replaces a version. So the crates are the
+last step of a release, after `promote-image` promotes the images.
+
+`devtools/publish-crates` publishes the crates in two modes. The token mode
+publishes the first version of each crate, and the workflow publishes each
+later version:
+
+- The `release-crates` workflow publishes new versions of the crates that
+ crates.io has. It uses trusted publishing, so no API token exists.
+- A maintainer publishes each new crate with an API token and
+ `devtools/publish-crates --new-crates`. Trusted publishing cannot create
+ a crate. This mode never publishes a crate that crates.io has. For a
+ crate that requires trusted publishing, crates.io refuses a new version
+ published with an API token.
+
+### Repository settings for crates.io
+
+The release depends on two settings of this repository. Check both before
+you add a trusted publisher on crates.io. A job that names a missing
+environment creates it with no protection rules.
+
+- The environment `crates-io` has the four required reviewers of the other
+ release environments, with prevent self-review. So the maintainer who
+ dispatches a run cannot approve it. The environment allows deployments
+ from `main` only, and it holds no secrets. Admin bypass is off for
+ `crates-io`. crates.io never replaces a version, so even an administrator
+ needs the approval of a second maintainer to publish.
+
+ ```bash
+ gh api repos/ExtendDB/extenddb/environments/crates-io \
+ --jq '{can_admins_bypass, rules: [.protection_rules[] | {type, prevent_self_review, reviewers: [.reviewers[]? | .reviewer.login // .reviewer.slug]}]}'
+ gh api repos/ExtendDB/extenddb/environments/crates-io/deployment-branch-policies \
+ --jq '.branch_policies[] | "\(.type):\(.name)"'
+ ```
+
+ The first command prints `can_admins_bypass` false, and a
+ `required_reviewers` rule with `prevent_self_review` true and the same
+ reviewers as the environment `npm`. The second command prints only
+ `branch:main`.
+- The tag ruleset `protect-release-tags` is active on `refs/tags/v*`. It
+ blocks the deletion and the update of those tags, and it has no bypass
+ actors. So a release tag cannot move to another commit.
+
+ ```bash
+ gh api repos/ExtendDB/extenddb/rulesets \
+ --jq '.[] | select(.name == "protect-release-tags") | {id, target, enforcement}'
+ gh api repos/ExtendDB/extenddb/rulesets/ \
+ --jq '{enforcement, bypass_actors, include: .conditions.ref_name.include, rules: [.rules[].type]}'
+ ```
+
+ The second command prints the enforcement `active`, an empty
+ `bypass_actors` list, the pattern `refs/tags/v*`, and the rules
+ `deletion` and `update`.
+
+### Each release
+
+After `promote-image` promotes the images of the release:
+
+1. A maintainer dispatches the `release-crates` workflow
+ (`.github/workflows/release-crates.yml`) from `main`, with the release
+ tag as input.
+2. The `gate` job checks that the run comes from `main`. It checks the tag
+ as `npm-publish` does. The tag must have the strict shape and exist, and
+ its commit must be on `main`. Its version must equal the workspace
+ version.
+3. The gate reads `docker.io/extenddb/extenddb-postgres:X.Y.Z` without
+ credentials. The revision label of that image must be the tagged commit.
+ The gate also runs `devtools/check-released-through`.
+4. Then `devtools/publish-crates --dry-run` plans the run, and the gate
+ writes the tag, the commit and the plan to the job summary. The plan
+ fails when a crate on crates.io accepts API tokens. A new crate does not
+ fail the gate, and the plan shows the stop before it.
+5. The `verify` job holds no credentials. It builds every package from its
+ tarball with `cargo package --workspace --locked`, and it runs
+ `devtools/check-crate-packages`. It records the SHA-256 of the package of
+ each crate that crates.io receives, the tagged commit and the toolchain,
+ in the artifact `crate-sha256`.
+6. When the plan has a crate to publish, a reviewer approves the
+ `crates-io` environment. The `publish` job packages the crates again
+ without compiling them, and it runs `devtools/check-crate-packages`. It
+ stops unless each package has the recorded SHA-256.
+7. Then the `publish` job receives a crates.io token through trusted
+ publishing. `devtools/publish-crates --no-verify` plans the run again,
+ and it publishes the crates in dependency order, up to the first new
+ crate. It compares each uploaded package with the recorded SHA-256. The
+ token expires 30 minutes after the job receives it, and the job revokes
+ it when it ends.
+8. When no new crate stopped the run, the `index` job compares the
+ checksum of each crate's version in the crates.io index with the
+ recorded SHA-256. Then the `smoke` job runs
+ `cargo install extenddb --version X.Y.Z --locked` with an empty cargo
+ home. It checks the version, the commit and `test-hooks disabled` in the
+ output of `extenddb version`. It also checks that no
+ `backup-test-hooks` line appears.
+9. When the run stopped before a new crate, the `new-crate` job reports
+ the stop and points at the token steps in the job summary. Then the job
+ fails the run.
+
+The plan of the `publish` job decides between steps 8 and 9, because the
+script reads crates.io again before its first upload. A token run can
+publish the gate's new crate while the job waits for its approval. The job
+then publishes the rest of the release, and `index` and `smoke` run. When
+the run skips `publish`, the gate's plan decides.
+
+The script skips each version that crates.io has. So a run that stopped
+part way can be dispatched again.
+
+### A release that adds a crate
+
+Trusted publishing cannot create a crate. So a release that adds a crate
+takes three runs:
+
+1. The `release-crates` run publishes the crates before the new crate, and
+ it stops before the new crate.
+2. A maintainer publishes the new crate with an API token, as the next
+ section describes. That run skips each crate that crates.io has at this
+ version, and it stops before the next crate that needs a new version.
+3. A second dispatch of `release-crates` publishes the rest of the release.
+ Its gate fails until the new crate requires trusted publishing.
+
+When a crate that crates.io has comes between two new crates, steps 2 and
+3 repeat for the second new crate.
+
+### Publishing new crates with an API token
+
+The first publish of the workspace uses these steps for all 14 crates. A
+later release uses them for each new crate. Both come after
+`promote-image` for the release.
+
+You need Python 3.11 or later, rustup and the GitHub CLI. You also need a
+crates.io account, from a sign-in with GitHub, with a verified email
+address. crates.io refuses a publish from an account without one. Before
+the first publish, an owner of the ExtendDB GitHub organization allows
+crates.io's GitHub OAuth app for the organization. A GitHub team can then
+own the crates.
+
+1. Use the `release-crates` run that stopped before the new crate. For the
+ first publish, dispatch it as in "Each release". Its `verify` job records
+ the SHA-256 list, the tagged commit and the toolchain in the artifact
+ `crate-sha256`. Note the ID of the run. The run keeps the artifact for 7
+ days. After that, dispatch `release-crates` again for a new artifact.
+2. On crates.io, create an API token (Account Settings, API Tokens, New
+ Token):
+ - Scopes: `publish-new` only. So the token cannot publish a new version
+ of a crate that crates.io has.
+ - Crates: the pattern `extenddb*` for the first publish, or the name of
+ the new crate for a later one.
+ - Expiration: "Custom...", with the date of the next day.
+3. Fetch the tag into a new directory, build the packages there, and
+ compare them with the list of the `verify` job. The block fetches the
+ tag by its full ref and checks it out detached. Before any command from
+ the new directory runs, the block stops unless the checked-out commit is
+ the commit in the artifact. So the code of a branch with the name of the
+ tag never runs. The block takes the toolchain from the artifact, so it
+ packages with the toolchain of the `verify` job. It accepts only a
+ toolchain of the form X.Y.Z. No build of dependency code runs in the new
+ directory. The comparison stops the procedure before any upload when
+ this machine writes other bytes.
+
+ The block packages with an empty cargo home, so cargo reads no stored
+ token. Cargo still reads .cargo/config.toml in the new directory and in
+ each parent directory, which includes ~/.cargo/config.toml when the new
+ directory is under your home directory.
+
+ ```bash
+ TAG=vX.Y.Z
+ RUN_ID=
+ SUMS="$(mktemp -d)" && SRC="$(mktemp -d)" \
+ && gh run download "$RUN_ID" --repo ExtendDB/extenddb --name crate-sha256 --dir "$SUMS" \
+ && grep -qxE '[0-9]+\.[0-9]+\.[0-9]+' "$SUMS/TOOLCHAIN" \
+ && git init -q "$SRC" && cd "$SRC" \
+ && git fetch --depth 1 https://github.com/ExtendDB/extenddb.git "refs/tags/$TAG:refs/tags/$TAG" \
+ && git checkout -q --detach "refs/tags/$TAG" \
+ && test "$(git rev-parse HEAD)" = "$(cat "$SUMS/COMMIT")" \
+ && export RUSTUP_TOOLCHAIN="$(cat "$SUMS/TOOLCHAIN")" \
+ && rustup toolchain install "$RUSTUP_TOOLCHAIN" \
+ && export CARGO_HOME="$(mktemp -d)" \
+ && cargo package --workspace --no-verify --locked \
+ && devtools/check-crate-packages \
+ && (cd target/package && sha256sum --check --strict "$SUMS/SHA256SUMS") \
+ && devtools/publish-crates --new-crates --dry-run --version "${TAG#v}" --sha256sums "$SUMS/SHA256SUMS" \
+ && echo "Ready: $TAG is commit $(git rev-parse HEAD), and the packages have the bytes of the verify job."
+ ```
+
+4. Run the next line only after the block prints its last line, in the
+ same shell. The command is one line, so `read` takes the token from the
+ keyboard and never from a pasted line. The line asks for the token,
+ gives it to the publish command only, and then removes it from the
+ shell.
+
+ ```bash
+ printf 'crates.io token: ' && read -rs CARGO_REGISTRY_TOKEN && echo && CARGO_REGISTRY_TOKEN="$CARGO_REGISTRY_TOKEN" devtools/publish-crates --new-crates --no-verify --version "${TAG#v}" --sha256sums "$SUMS/SHA256SUMS"; unset CARGO_REGISTRY_TOKEN
+ ```
+
+ With `--no-verify`, cargo publishes without a build. So no build script
+ or procedural macro runs while the token is in the environment. The
+ script compares each uploaded package with the list. By default,
+ crates.io accepts 5 new crates at once and then one every 10 minutes. So
+ the 14 crates of the first publish take about 90 minutes. The script
+ waits after each 429 answer, up to 2 hours in total.
+5. Revoke the token on crates.io at once. The next steps use the browser
+ session.
+6. Check the settings in "Repository settings for crates.io".
+7. On the Settings tab of each new crate on crates.io:
+ - Add a second named owner. That maintainer signed in to crates.io
+ before, and accepts the invitation.
+ - Add the team `github:extenddb:extendb-committers`. A team owner can
+ publish and yank, but cannot change the owners. The maintainer who adds
+ the team is a member of `extendb-committers` or an owner of the
+ ExtendDB organization. That maintainer also grants crates.io the GitHub
+ scope `read:org`.
+ - Add a GitHub trusted publisher: repository owner `ExtendDB`,
+ repository name `extenddb`, workflow filename `release-crates.yml`,
+ and `crates-io` in the field "Environment name (optional)". The form
+ marks the environment as optional. Fill it in, because a publisher
+ without it gives a token to a run of `release-crates.yml` from any
+ branch.
+ - Select "Require trusted publishing for all new versions", so that
+ crates.io refuses a new version published with an API token.
+8. After the first publish, check that docs.rs built the library crates.
+9. Dispatch `release-crates` again with the release tag. The run publishes
+ the rest of the release, and its `index` and `smoke` jobs check the
+ whole release.
+
+### A failure after the upload
+
+A version on crates.io is permanent. A trusted publishing token can only
+publish, so an owner yanks a version.
+
+- The `index` job fails when the checksum of a version in the crates.io
+ index differs from the verified SHA-256. crates.io then serves other bytes
+ than the `verify` job built. An owner yanks every version of the release,
+ on the page of each crate, or with `cargo yank` and an API token of the
+ scope `yank`. The next patch release replaces the release.
+- When `devtools/publish-crates` stops because an uploaded package differs
+ from the verified list, the same steps apply. Do not run the script again
+ for that release. A later run with `--sha256sums` stops before its first
+ upload, because the index then lists that version with another checksum.
+- The `index` job also fails when the index does not list a version of the
+ release after 20 tries. Rerun the job first, because the index can lag
+ behind an upload. If the version is still missing, check the log of the
+ `publish` job for that crate.
+- The `smoke` job fails when `cargo install` fails, or when
+ `extenddb version` prints a wrong line. A rerun helps only when the
+ install failed, because the crates.io index can lag. A wrong version,
+ commit or test-hooks line comes from the published bytes, and a rerun
+ prints the same line. Then decide whether to yank the release.
+
## Known limits
- A hard-killed caller (SIGKILL, OOM) cannot run its exit handler, so the