Repository navigation
fix(bench): include the runner and host identity in comparability #574
Description
Problem
benchmarks/compare.zsh decides comparable from the Zsh version, architecture, sample count and warmup count (lines 77-81 on next at 85214aa). It ignores environment.runner_image and environment.cpu, which run.zsh records in every report (line 276). ADR-0024 decision 5 of z-shell/.github says results are compared only when the environment identity matches, and names the runner or host identity alongside the Zsh version and architecture; decision 6 defines comparable as derived from the identity fields, which include cpu and runner_image.
So two reports from different hosted runner images, or from two local machines with different CPUs but the same Zsh and architecture, are marked comparable, and a hardware difference can be flagged as a code regression. benchmarks/README.md repeats the narrower rule in two places (line 29, and line 33, which also omits the warmup count).
Within one workflow run, which is how CI uses the comparer today, both reports come from the same runner and the fields match, so current CI results are unaffected. The gap bites when reports from different runs or hosts are compared, for example a published per-release result against a new run.
Change
- Include
environment.runner_imageandenvironment.cpuin the baseline-versus-candidate comparability check, so mismatched hosts yield null flags and the "not comparable" diagnostic names them. - Decide whether the control identity check also needs them. The control is the baseline measured again in the same run, so they should always match; including them costs nothing.
- Add harness cases to
tests/benchmark-harness.zsh: a candidate report with a differentrunner_image, and one with a differentcpu, are not comparable and flag nothing. - Update the comparability sentences in
benchmarks/README.md(lines 29 and 33).
This touches the same lines as #560 (variant list in comparability), so it is blocked by #560 and should be implemented on top of it.
Refs #560.
Activity
ss-o commentedon Sep 30, 2026
Project 28 triage pass. Blank values filled on this issue (existing values left unchanged):
Bug.compare.zshignoresrunner_imageandcpuwhen it judges comparability.Low. The issue says current CI results are unaffected.S. The same lines as fix(bench): include the variant list in comparability and control identity #560, plus harness cases and README sentences.P3 - Low. Tooling only, with no current CI effect.Human delivery. Ordinary tooling bug.ss-o commentedon Oct 1, 2026
Project 28 triage pass (2026-10-02): Status
Triage->Blocked, labelstatus:blocked.nextat3fa8433(premise confirmed): benchmarks/compare.zsh:78-81@3fa8433$comparableignores environment.runner_image and environment.cpu; control identity at :66-67 likewise; README.md:29 and :33 repeat the narrower rule (line 33: 'same Zsh version, architecture, and sample count'). Native blocked-by fix(bench): include the variant list in comparability and control identity #560 is already recorded; both edit the same lines, so implement on top of fix(bench): include the variant list in comparability and control identity #560.Todo(or closes, if nothing is left).ss-o commentedon Oct 8, 2026
Blocker cleared: #560 closed when
nextwas promoted tomain(#617, merge commit199bbca).nextat2a097fb: the control identity (benchmarks/compare.zsh:68-70) and$comparable(:84) still compare source revision, Zsh version, architecture, samples, warmups and variants, but notenvironment.runner_imageorenvironment.cpu.Blocked->Todo;status:blockedremoved.nexton top of the fix(bench): include the variant list in comparability and control identity #560 change, with harness cases and the comparability wording in benchmarks/README.md (lines 29 and 33).fix(bench): name unrecorded host fields and test the image control (#574