Repository navigation
ci: shard the CodSpeed simulation benchmarks across two jobs - #91
Merged
Merged
Conversation
The simulation job ran every benchmark one after another under Valgrind, which is single-threaded: about 4m20s of a 6-minute run, the critical path of CI. The tpch train_and_compress benchmarks now run in one job and the tpch parse/decompress_all and clickbench benchmarks in another, at the same time. CodSpeed merges the uploads of one workflow run. A "Benchmark with Codspeed" job still exists, as the required status check, and fails unless both shards succeed. Superseded pull request runs are now cancelled; runs on develop are not. Signed-off-by: Joseph Isaacs <joe.isaacs@live.co.uk>
Contributor
Merging this PR will not alter performance1 warning
Comparing |
Signed-off-by: Joseph Isaacs <joe.isaacs@live.co.uk>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #88, with the commit authored and signed off by Joseph Isaacs so the CLA check can pass.
Why
In the last CI run (run 43905340379) the wall-clock was 5m56s. The critical path was "Benchmark with Codspeed" (5m55s): its "Run benchmarks - Simulation" step took 4m19s, running every benchmark one after another under Valgrind, which uses a single core. Of that, tpch
train_and_compresstook about 2m10s, tpchparseabout 1m35s, tpchdecompress_allabout 10s and clickbench about 10s. The other jobs were shorter: Walltime 3m56s, Build 44s, MSRV 20s.Changes
bench-codspeed-shard), each job running part of the suite withcargo codspeed run '<filter>':tpch\.rs::train_and_compresstpch\.rs::(parse|decompress_all)|clickbench\.rs::CodSpeed merges the uploads from one workflow run. I ran the second filter locally with
cargo codspeed run -m simulation, and it selects exactly the expected benchmarks.Benchmark with Codspeed, because that name is a required status check. Itneeds:the shards, runsif: always(), and fails unless the shards succeeded.concurrency:so a new push to a pull request cancels the older run. Runs ondevelopare never cancelled.Caveats
cargo-codspeedinstall and the benchmark build, about 1 minute in total.Generated by Claude Code