Skip to content

ci: shard the CodSpeed simulation benchmarks across two jobs - #88

Closed
joseph-isaacs wants to merge 1 commit into
developfrom
ji/ci-parallel-jobs
Closed

joseph-isaacs wants to merge 1 commit into
developfrom
ji/ci-parallel-jobs

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Member

Why

In the last CI run (run 43905340379) the wall-clock was 5m56s. The critical path was "Benchmark with Codspeed" (5m55s): its "Run benchmarks - Simulation" step took 4m19s, running every benchmark one after another under Valgrind, which uses a single core. Of that, tpch train_and_compress took about 2m10s, tpch parse about 1m35s, tpch decompress_all about 10s and clickbench about 10s. The other jobs were shorter: Walltime 3m56s, Build 44s, MSRV 20s.

Changes

  • The simulation job is now a two-job matrix (bench-codspeed-shard), each job running part of the suite with cargo codspeed run '<filter>':
    • tpch\.rs::train_and_compress
    • tpch\.rs::(parse|decompress_all)|clickbench\.rs::
      CodSpeed merges the uploads from one workflow run. I ran the second filter locally with cargo codspeed run -m simulation, and it selects exactly the expected benchmarks.
  • A job still carries the name Benchmark with Codspeed, because that name is a required status check. It needs: the shards, runs if: always(), and fails unless the shards succeeded.
  • Added concurrency: so a new push to a pull request cancels the older run. Runs on develop are never cancelled.

Caveats

  • One more c7i.8xlarge runner per run, plus a small ubuntu-latest job for the check. Each shard repeats the toolchain setup, the cargo-codspeed install and the benchmark build, about 1 minute in total.

Generated by Claude Code

The simulation job ran every benchmark one after another under Valgrind,
which is single-threaded: about 4m20s of a 6-minute run, the critical
path of CI. The tpch train_and_compress benchmarks now run in one job and
the tpch parse/decompress_all and clickbench benchmarks in another, at
the same time. CodSpeed merges the uploads of one workflow run.

A "Benchmark with Codspeed" job still exists, as the required status
check, and fails unless both shards succeed. Superseded pull request
runs are now cancelled; runs on develop are not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBDYexjeUHTsV3EvkrG
@joseph-isaacs joseph-isaacs added the changelog/chore Maintenance, tooling, and housekeeping with no user-facing changes label Nov 16, 2026 — with Claude
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@codspeed

codspeed Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

✅ 48 untouched benchmarks


Comparing ji/ci-parallel-jobs (7a73be3) with develop (6b973ac)

Open in CodSpeed

Copy link
Copy Markdown
Member Author

Superseded by #91: the same change, with the commit authored and signed off by Joseph Isaacs so the CLA check can pass.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore Maintenance, tooling, and housekeeping with no user-facing changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants