Skip to content

ci: build each release binary once and run the long pytest jobs as shards - #679

Merged
LeeroyHannigan merged 24 commits into
mainfrom
ci/fast-checks
Nov 9, 2026
Merged

ci: build each release binary once and run the long pytest jobs as shards#679
LeeroyHannigan merged 24 commits into
mainfrom
ci/fast-checks

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Oct 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What

This PR reduces the run time of the integration workflows. It builds each release binary once and runs the long pytest jobs as shards.

  1. One build job per feature set (build-postgres, build-sqlite, build-mongodb, build-cassandra) uploads target/release/extenddb. Each job that ran the same build command downloads it instead. run-integration-dev-mode keeps its own build, because no other job uses its features.
  2. run-integration, run-integration-sqlite and mongodb-pytest run as three shards, and cassandra-pytest runs as five, split by test file. devtools/test-shards assigns the files by the seconds in devtools/test-shard-weights.tsv, and devtools/run-tests --shard K/N runs one shard. The weights of the main and comprehensive suites are each file's seconds on main's push run for 0cfb0d9.
  3. integration, mongodb-integration and cassandra-integration fail unless the shards together collected every test of the whole suite exactly once.
  4. The PostgreSQL storage-level tests run in their own job, run-postgres-storage-tests, beside the Rust integration suite.
  5. cassandra-catalog-checks caches the v0.1.14 binary that it builds from the tag. Every run restores the entry, and only a push to main saves it. On a cache hit, the job skips that build and its Rust toolchain and cache steps.
  6. The rust-cache steps of cassandra-catalog-checks and cassandra-two-nodes save only on pushes to main, as the other rust-cache steps do. Their pull request runs saved entries of their own, which count against the repository's 10 GB cache limit.
  7. With --shard, devtools/run-tests runs the --terraform apply on shard 1 only. Each backend's job runs it once, as on main.

This PR removes or skips no test, and the required check names stay the same.

Why

Main's push run for 2671ee1 on 9 November took 43.1 minutes for integration, 35.4 for cassandra-integration and 41.3 for mongodb-integration. Each figure is the time from the start of the workflow run to the end of its check job. On main, each pytest job builds its own release binary and runs its whole suite.

Testing done

  • actionlint 1.7.12 passes on the three workflows, with the runner label declared and SC2034 ignored. Main's copies give the same 7 SC2034 warnings. The workflows parse as YAML, and bash -n and py_compile pass on the devtools files.
  • CI on 4390d34, which holds main at 2671ee1, passed. Minutes from the start of each workflow run to the end of its check job:
Check Main, push run for 2671ee1 This PR, 4390d34
integration 43.1 29.2
cassandra-integration 35.4 18.1
mongodb-integration 41.3 34.7
  • The test check failed once on tests::panicking_refresh_does_not_leak_inflight_flag in extenddb-cache, a timing test in a crate that this PR does not change. The rerun passed. I found no other failure of that test in the failed jobs of the Test workflow's 724 runs since 26 October.
  • In that run, the completeness check of each summary job passed. For example, run-integration's main suite printed whole 5298, shards 2715 + 1765 + 818 = 5298 OK. For cassandra-pytest, the main suite printed whole 5308, shards 1915 + 703 + 835 + 1031 + 824 = 5308 OK.
  • Shard 1 of each of the four sharded jobs printed ✓ Terraform apply passed. The other shards printed a skip line such as skipped: shard 2/3 (shard 1 runs the apply).
  • The four build jobs and mongodb-rust found no Rust cache, and the build jobs took 6.3 to 9.5 minutes. rust-cache keys its entries by job, and only pushes to main save them. The build jobs can therefore restore a cache only after this PR merges. The v0.1.14 cache missed too, and cassandra-catalog-checks spent 4.5 minutes on that build.

Known issues

  • mongodb-rust sets the time of mongodb-integration. It waits 9.5 minutes for build-mongodb and then runs for 24.9 minutes.
  • Steps outside the sharded suites run on every shard. Each run-integration-sqlite shard repeats about 350 s of backup store and wire tests.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

@LeeroyHannigan LeeroyHannigan mentioned this pull request Oct 30, 2026
Merged
@navareamol
navareamol enabled auto-merge October 30, 2026 21:10
robinnsc
robinnsc previously approved these changes Oct 30, 2026
View reviewed changes
@navareamol
navareamol added this pull request to the merge queue Oct 30, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 30, 2026
@LeeroyHannigan
devtools: add per-file test durations for shard balancing
4737451 
devtools/test-shard-weights.tsv holds the seconds each pytest file and
each tests/rust module took on each backend in the merge group run for
c5fe781 on 30 Oct, read from its job logs. devtools/test-shards uses
them to balance the shards.
@LeeroyHannigan
ci: build each release binary once per feature set
3648e28 
Ten integration jobs each ran their own cargo build --release. Each
distinct feature set now has one build job that uploads
target/release/extenddb as an artifact, and the jobs that test it
download that binary instead of compiling it:

- build-postgres (cargo build --release): run-integration,
  run-rust-integration, run-rust-integration-postgres-novector
- build-sqlite (--no-default-features --features sqlite):
  run-integration-sqlite, run-rust-integration-sqlite
- build-mongodb (--features mongodb-test-hooks): mongodb-pytest,
  mongodb-rust
- build-cassandra (--features cassandra): cassandra-pytest,
  cassandra-rust

The cargo flags and features are unchanged. run-integration-dev-mode
keeps its own build because it is the only job using that feature set.

Jobs that only run pytest no longer install the Rust toolchain or
restore the cargo cache. Jobs that still run cargo test download the
binary after the cache restore, so a cached target/ cannot replace it.
Artifacts do not keep file modes, so each download is followed by
chmod +x.
@LeeroyHannigan
devtools: add test-shards to split pytest suites by file
a4fb235 
devtools/test-shards has three commands:

- collect runs pytest --collect-only -q on the given arguments and
  writes each collected test's file and id, with paths relative to the
  working directory so lists from different rootdirs compare.
- plan assigns every collected file to one of N shards, longest first
  to the least loaded shard, using the seconds in
  devtools/test-shard-weights.tsv. Every shard computes the same
  assignment from the same collection, so each file is in exactly one
  shard.
- check reads the lists all legs of a sharded job wrote and fails
  unless they are shards 1 to N of one N, agree on the whole suite, and
  together ran every test of it exactly once.

The weights only balance the shards; a missing or stale weight cannot
drop a test because check compares the shards with the whole suite.
@LeeroyHannigan
devtools: run one shard of each pytest suite with run-tests --shard
07253c2 
run-tests --shard K/N runs only shard K of the main, comprehensive and
CLI suites. Before each suite, devtools/test-shards prepare collects the
whole suite with the suite's own arguments, picks the shard's files, and
writes the whole and the shard's collected lists under
$EXTENDDB_SHARD_DIR (default discussions/shards) for the summary job.
pytest then loads devtools/pytest_shard.py, which deselects every test
outside the shard's files, so filters, ignores and xdist options apply as
in an unsharded run. A shard with no files of a suite runs nothing for
it rather than calling pytest with no paths.

The existing suite command lines are unchanged; each gains one line that
calls the hook when --shard is given.
@LeeroyHannigan
devtools: return the shard command through SHARD_CMD
be58f0e 
bash 4.2, which some development hosts still run, has no `local -n`, so
shard_suite now takes the suite's command as arguments and leaves the
command to run in SHARD_CMD.
@LeeroyHannigan
ci: run the long pytest jobs as three shards and check them
0f11b75 
run-integration, run-integration-sqlite and mongodb-pytest each run as a
three-leg matrix, every leg calling run-tests with --shard K/3 against its
own server and database, as the single job did. Each leg uploads its
shard lists.

The integration and mongodb-integration summary jobs keep their names and
their result checks, then download the lists and run devtools/test-shards
check. It fails unless the legs of each job are shards 1 to 3 of one
split, agree on the whole suite, and together collected every test of it
exactly once, so a bad split turns the required check red.
@LeeroyHannigan
ci: run the PostgreSQL storage-level tests in their own job
98629a7 
run-rust-integration ran the storage-level tests after the wire-level
Rust suite, 7.1 of its 27.4 minutes in the merge group for #511 on
30 Oct. They drive the storage layer against the job's PostgreSQL and
need no server, so run-postgres-storage-tests runs them with the same
image, environment and cargo command, without waiting for the release
build. The integration summary job requires it.
@LeeroyHannigan
ci: pass the shard through EXTENDDB_TEST_SHARD
b7fbcb7 
run-tests reads its --shard default from EXTENDDB_TEST_SHARD, and the
three sharded jobs set it at job level, so their run lines are unchanged
from main. Each upload of the shard lists moves to the end of its job.
Open pull requests that edit those run lines or add steps before them
merge without conflicts.
@LeeroyHannigan
ci: escape list fields, quiet shard collection, overwrite artifacts
4df7f81 
- devtools/test-shards collect escapes backslashes, tabs and newlines in
  each path and test id, so an id carrying a tab no longer splits a row
  and makes check report a test in two shards.
- prepare captures each collection's output and prints only its summary
  line, or the whole output when the collection fails. Each leg's log
  had carried the collection tree twice, about 4,900 lines for the main
  suite.
- Every upload-artifact step sets overwrite: true, so re-running one
  failed job can upload its artifact under the same name again.
@LeeroyHannigan
ci: save Rust caches only from pushes to main in the new jobs
4a45a51 
build-postgres, build-sqlite, run-postgres-storage-tests, build-mongodb
and build-cassandra are new in this branch, so #680 did not reach their
Swatinem/rust-cache steps. They now set save-if to
github.ref == 'refs/heads/main', as every other rust-cache step outside
launcher.yml does, so pull request and merge group runs of these jobs
restore main's caches and save nothing.
@LeeroyHannigan
Merge main into ci/fast-checks
2f45515 
The three workflow files conflicted. Each one was resolved by taking
main's version of the file and applying this branch's edits to main's
current jobs. The edits use the action SHAs main pins. Every job, needs entry
and if condition from main stays. Every step stays as well, except
the toolchain, cache and build steps that a download of the shared
binary replaces.

integration.yml: build-postgres and build-sqlite are added.
run-integration and run-integration-sqlite run as three shards,
download their binary and upload their shard lists after main's last
step. run-integration-sqlite-s3, which main added, downloads the SQLite
binary instead of building the same one. run-rust-integration,
run-rust-integration-postgres-novector and run-rust-integration-sqlite
download their binary after the cache restore. The storage-level step
moves to run-postgres-storage-tests with main's --tests command, and the
full-history checkout that migrations_tag_blobs needs moves with it. The
integration job also needs run-postgres-storage-tests and checks the
shard lists.

integration-mongodb.yml: build-mongodb is added, and mongodb-pytest runs
as three shards on its binary. mongodb-rust downloads that binary.
mongodb-backup-store and mongodb-catalog, which main added, download it
too instead of building the same one. mongodb-integration checks the
shard lists after main's result check.

integration-cassandra.yml: build-cassandra is added, and
cassandra-pytest and cassandra-rust download its binary.
cassandra-catalog-checks and cassandra-authorizer, which main added,
download it too instead of building the same one.
cassandra-catalog-checks keeps its Rust toolchain for the v0.1.14
build.

devtools/run-tests merged without conflicts.
@LeeroyHannigan
ci: run cassandra-pytest as five shards and check them
87cc396 
cassandra-pytest runs as five legs, each with EXTENDDB_TEST_SHARD set to
K/5 as in mongodb-pytest. Each leg has its own Cassandra service
container and uploads its shard lists. After its result check,
cassandra-integration downloads the lists and runs
devtools/test-shards check, so it fails unless every leg succeeded and
the legs together collected every test of the whole suite exactly once.

devtools/test-shard-weights.tsv gains a cassandra column, which
devtools/test-shards reads for --backend cassandra. It holds the seconds
each file of the main and comprehensive suites took in the
cassandra-pytest job of the push run for d35d860 on 6 Nov, rounded to
whole seconds. A file with no Cassandra measurement has "-", and the
tool weights it by its test count. With these weights the tool models
each leg at 320 to 324 seconds of tests. In that job the two suites took
1,583 seconds.
@LeeroyHannigan
ci: cache the v0.1.14 build in cassandra-catalog-checks
65ede4f 
cassandra-catalog-checks built the v0.1.14 tag on every run. The build
took 310 to 413 seconds in the last four pushes to main. An actions/cache
step, pinned to the commit SHA licenses.yml uses, now keeps the binary
that build writes, keyed on the tag, the cargo features and the runner
OS. On a cache hit the build step is skipped and the check runs the
restored binary from the same path. The check is unchanged.
@LeeroyHannigan
ci: give test_query_start_key_index.py its own Cassandra weight
f7d8142 
The Cassandra column counted the 5.5 s of
tests/python/test_query_start_key_index.py under
test_query_start_key_partition.py, which took 21.7 s by itself in the
same job. Each file now has its own weight. The weights only balance
the shards, so no test moves in or out of a run.
@LeeroyHannigan
ci: let each leg upload the regenerated schema
315d084 
Every leg of run-integration and run-integration-sqlite runs the schema
check. A schema difference fails all three legs, and each one then
uploads schema-postgres or schema-sqlite. upload-artifact refuses a
second artifact of one name in a run, so legs 2 and 3 reported an extra
error. With overwrite: true, each leg uploads the same file without
that error.
@LeeroyHannigan
ci: key the v0.1.14 cache on the build's features and the runner arch
eaef2f1 
The cache key and the build command of the v0.1.14 binary now read the
cargo features from V0114_FEATURES. A change to the features changes the
key, so the job cannot restore a binary built with other features. The
key also names the runner's architecture.

The cache step runs first. On a hit, cassandra-catalog-checks skips the
Rust toolchain and cache steps, because only the v0.1.14 build uses
them.
@LeeroyHannigan LeeroyHannigan changed the title ci: build each release binary once and run the long pytest jobs as three shards ci: build each release binary once and run the long pytest jobs as shards Nov 6, 2026
@LeeroyHannigan
ci: save the v0.1.14 cache only from pushes to main
8f8103b 
cassandra-catalog-checks used actions/cache, which saves the entry from
any run that misses, pull request runs included. The job now restores
the entry with actions/cache/restore in every run, and saves it with
actions/cache/save only on a push to main that missed. No pull request
run writes the entry, the same rule the Rust caches follow.
@LeeroyHannigan
Merge main into ci/fast-checks
e196b29 
integration.yml conflicted in the job-level settings of run-integration
and run-integration-sqlite. Main raised the timeout of both jobs from 40
to 55 minutes and added a comment with each job's measured time. This
branch added needs, the shard matrix and EXTENDDB_TEST_SHARD to the same
settings. Each job now has main's comment and timeout, then this
branch's needs, strategy and env.

The rest of main's workflow changes merged without conflicts. Main added
no job to the three integration workflows. Each job that main changed
already downloads its binary from a shared build job on this branch, so
no build step changes.

integration.yml: run-integration gets the S3 mock step, the S3 endpoint
for the backup store and job suites, and the step that prints the mock's
log. These steps run no cargo command, so the job stays without a Rust
toolchain or cache. run-integration-sqlite-s3 and
run-rust-integration-sqlite get comment changes only.

integration-mongodb.yml: mongodb-pytest and mongodb-rust set
EXTENDDB_TEST_BACKUP_STORE to filesystem. mongodb-rust keeps its Rust
toolchain and cache, because run-mongodb-tests --rust-live runs cargo
test. mongodb-pytest stays without them: run-mongodb-tests and run-tests
run cargo only for their --rust options, and the job passes none of
them.

Main's new matrix-nightly.yml and its devtools/run-mongodb-tests changes
merged without conflicts.
@LeeroyHannigan
ci: save cassandra-catalog-checks' Rust cache only from pushes to main
64faf03 
The Swatinem/rust-cache step of cassandra-catalog-checks had no save-if.
A pull request or merge group run that missed the v0.1.14 cache
therefore saved Rust cache entries of its own. The step now sets save-if
to github.ref == 'refs/heads/main', as the rust-cache steps of
cassandra-storage, build-cassandra, cassandra-rust and
cassandra-production-build do. Pull request and merge group runs
restore main's cache and save nothing.
@LeeroyHannigan
ci: save cassandra-two-nodes' Rust cache only from pushes to main
696c28b 
The rust-cache step of cassandra-two-nodes had no save-if, so each pull
request run saved an entry of its own. Those entries count against the
repository's 10 GB cache limit. When the total passes the limit, GitHub
evicts older entries, main's included. The step now saves only on
pushes to main, as the other rust-cache steps do.
@LeeroyHannigan
Merge main into ci/fast-checks
a209fc5 
Main raised the timeouts of five jobs and added or extended a comment
with each job's measured time. run-integration went from 55 to 70 minutes and
run-integration-sqlite from 55 to 60. run-rust-integration went from 40
to 50 and mongodb-rust from 40 to 55. cassandra-pytest went from 45 to
60. This branch added needs to the same job settings of all five, and the shard
matrix and EXTENDDB_TEST_SHARD to run-integration, run-integration-sqlite
and cassandra-pytest. Each job now has main's comment and timeout, then
this branch's keys.

These were the only conflicts. The timeouts apply to each shard, which
runs part of the suite.

Main added no job and no build step to the three integration workflows,
so no download step changes. The PartiQL test files that main added have
no shard weight. devtools/test-shards weights them by their test count.
@LeeroyHannigan
ci: weight the shards with each test file's time on main
13b1400 
devtools/test-shard-weights.tsv had no row for the 35 PartiQL test
files, which hold 3,133 of the 5,255 tests that the main suite collects.
The planner priced each of their tests at the suite's mean, about 1.2 s
on PostgreSQL, so it modelled them at 62.9 minutes. On main's push run
for 0cfb0d9 they took 8.9 minutes of worker time on PostgreSQL.

The main and comprehensive rows now hold each file's seconds from that
run's run-integration, run-integration-sqlite, mongodb-pytest and
cassandra-pytest jobs. A file's seconds are the gaps between its result
lines, so they include its fixtures. The cli and rust-integration rows
do not change.

With the old weights, the slowest PostgreSQL shard had 9.7 minutes of
that run's measured main-suite time, and the slowest SQLite shard 12.6.
MongoDB's slowest had 9.7 and Cassandra's 7.7. With the new weights the
slowest shards have 7.5, 8.7, 6.4 and 4.4 minutes. Those figures fit the
weights to the run they come from, so a later run can spread less evenly.

The weights only balance the shards. The completeness check does not
read them.
@LeeroyHannigan
Merge main into ci/fast-checks
c1c5b68 
Main (#474) added an "Install Terraform" step after the "Build release"
step of mongodb-pytest and cassandra-pytest, and --terraform to their
test commands. This branch replaced those build steps with a download of
the shared binary and a chmod. These were the only conflicts. Each job
now has the download and the chmod, then main's Terraform step.

The same Terraform changes to run-integration and run-integration-sqlite
merged without conflicts. So did main's changes to devtools/run-tests and
devtools/check-postgres-catalog-lock.
@LeeroyHannigan
ci: run the Terraform apply on shard 1 only
4390d34 
run-integration, run-integration-sqlite, mongodb-pytest and
cassandra-pytest pass --terraform to devtools/run-tests, and each job
runs as shards. The apply is one run against the job's server, not a
suite of test files, so run-tests ran it on every shard. With --shard
K/N, run-tests now runs the apply when K is 1 and prints a skip line on
the other shards. Each backend's job runs the apply once, as on main.
@LeeroyHannigan
LeeroyHannigan merged commit 96a6b6f into main Nov 9, 2026
51 of 54 checks passed
@LeeroyHannigan
LeeroyHannigan deleted the ci/fast-checks branch November 9, 2026 13:58
@LeeroyHannigan LeeroyHannigan mentioned this pull request Nov 9, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@robinnsc robinnsc robinnsc left review comments
@amrith amrith Awaiting requested review from amrith amrith is a code owner
@c33howard c33howard Awaiting requested review from c33howard c33howard is a code owner
@jcshepherd jcshepherd Awaiting requested review from jcshepherd jcshepherd is a code owner
@pdf-amzn pdf-amzn Awaiting requested review from pdf-amzn pdf-amzn is a code owner
@yesyayen yesyayen Awaiting requested review from yesyayen
@eliseong eliseong Awaiting requested review from eliseong eliseong is a code owner
@hitesh96db hitesh96db Awaiting requested review from hitesh96db hitesh96db is a code owner
@navareamol navareamol Awaiting requested review from navareamol navareamol is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@LeeroyHannigan @robinnsc