ci: wire up integration-test sharding, off by default - #3443
Conversation
PR SummaryLow Risk Overview Reviewed by Cursor Bugbot for commit 9f09747. Bugbot is set up for automated code reviews on this repo. Configure here. |
❌ 1 Tests Failed:
View the top 1 failed test(s) by shortest run time
View the full list of 5 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
1f9dd1f to
794bdf7
Compare
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 794bdf7. Configure here.
Each compression config runs the whole suite on one runner, where api/templates alone sets the wall clock: 853s of the 856s test step under zstd1, because go test caps every package at -parallel=4. Splitting a config across jobs is the only lever that moves that, so this adds the machinery. scripts/select-tests.sh enumerates the top-level tests from the source tree and bin-packs each package across the shards using recorded per-test times, so every shard holds a slice of every package, a newly added test always lands in exactly one shard, and the shards finish at roughly the same time. update-test-weights.go refreshes the recorded times from a run's JUnit files. SHARDS defaults to 1, so this changes nothing until someone turns the dial. Measured at 4 shards the slowest job drops 20.1m -> 11.3m, but the infra-tests pool only served ~6 concurrent runners, so the extra jobs spent their savings queueing and end-to-end got worse. Raise SHARDS alongside more capacity (#3352), not before.
794bdf7 to
9f09747
Compare
tomassrnka
changed the title
|
Closing — this only pays off with more concurrent runner capacity, and we're not pursuing that (#3352 closed). Measured for the record: sharding cut per-job time 20.1m → 11.3m but raised runner-minutes 60 → 136, and the pool couldn't place the fan-out (13-17 min of staggering), so end-to-end got worse. The win came from cutting work instead — #3462 takes end-to-end 21.5m → 11.1m at 28 runner-minutes with no extra capacity. |
1 participant
Each compression config runs the whole suite on one runner, where
api/templatesalone sets the clock — 853s of the 856s zstd1 test step, becausego testcaps every package at-parallel=4. This adds the machinery to split a config across jobs:scripts/select-tests.shenumerates top-level tests from the source tree and bin-packs each package across the shards by recorded runtime, so every shard holds a slice of every package and a newly added test always lands in exactly one shard.update-test-weights.gorefreshes those times from a run.SHARDSdefaults to 1, so this is a no-op until someone turns the dial. Measured at 4 shards the slowest job drops 20.1m -> 11.3m (44%), butinfra-testsonly served ~6 concurrent runners, so the 12 jobs spent their savings queueing and end-to-end regressed (21.5m -> 26.7m). Reducing to 6 jobs did not buy it back either — the queue delay tracks repo-wide pool saturation, not this workflow. So this should merge/raise only alongside more runners (#3352); staying draft until then. The immediate, capacity-neutral win is split out into #3462.