Skip to content

ci: shard the CodSpeed simulation benchmarks across two jobs - #91

Merged
joseph-isaacs merged 2 commits into
developfrom
ji/ci-parallel-jobs-signed
Nov 17, 2026
Merged

joseph-isaacs merged 2 commits into
developfrom
ji/ci-parallel-jobs-signed

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Member

Replaces #88, with the commit authored and signed off by Joseph Isaacs so the CLA check can pass.

Why

In the last CI run (run 43905340379) the wall-clock was 5m56s. The critical path was "Benchmark with Codspeed" (5m55s): its "Run benchmarks - Simulation" step took 4m19s, running every benchmark one after another under Valgrind, which uses a single core. Of that, tpch train_and_compress took about 2m10s, tpch parse about 1m35s, tpch decompress_all about 10s and clickbench about 10s. The other jobs were shorter: Walltime 3m56s, Build 44s, MSRV 20s.

Changes

  • The simulation job is now a two-job matrix (bench-codspeed-shard), each job running part of the suite with cargo codspeed run '<filter>':
    • tpch\.rs::train_and_compress
    • tpch\.rs::(parse|decompress_all)|clickbench\.rs::
      CodSpeed merges the uploads from one workflow run. I ran the second filter locally with cargo codspeed run -m simulation, and it selects exactly the expected benchmarks.
  • A job still carries the name Benchmark with Codspeed, because that name is a required status check. It needs: the shards, runs if: always(), and fails unless the shards succeeded.
  • Added concurrency: so a new push to a pull request cancels the older run. Runs on develop are never cancelled.

Caveats

  • One more c7i.8xlarge runner per run, plus a small ubuntu-latest job for the check. Each shard repeats the toolchain setup, the cargo-codspeed install and the benchmark build, about 1 minute in total.

Generated by Claude Code

The simulation job ran every benchmark one after another under Valgrind,
which is single-threaded: about 4m20s of a 6-minute run, the critical
path of CI. The tpch train_and_compress benchmarks now run in one job and
the tpch parse/decompress_all and clickbench benchmarks in another, at
the same time. CodSpeed merges the uploads of one workflow run.

A "Benchmark with Codspeed" job still exists, as the required status
check, and fails unless both shards succeed. Superseded pull request
runs are now cancelled; runs on develop are not.

Signed-off-by: Joseph Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs joseph-isaacs added the changelog/chore Maintenance, tooling, and housekeeping with no user-facing changes label Nov 17, 2026 — with Claude
@codspeed

codspeed Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

✅ 48 untouched benchmarks


Comparing ji/ci-parallel-jobs-signed (685e76e) with develop (e477cd5)

Open in CodSpeed

Signed-off-by: Joseph Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs enabled auto-merge (squash) November 17, 2026 11:06
@joseph-isaacs
joseph-isaacs merged commit f1de5eb into develop Nov 17, 2026
12 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/ci-parallel-jobs-signed branch November 17, 2026 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore Maintenance, tooling, and housekeeping with no user-facing changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant