Repository navigation
Bench: interleaved A/B measurement (CPython vs @compiled) - #92
Merged
Merged
Conversation
run.py interleaved alternates CPython and the compiled extension pair by pair (order flipped each pair) in one time window and reports the median per-pair speedup with its range and the load at every pair. Load hits both sides alike, so the ratio is usable when the machine is never quiet (leader decision 2026-10-03: the official method if no quiet window appears). Refs #41
11 A/B pairs per benchmark at full size, order alternated, under 1-min load 12-36 (the machine had no quiet window all day). Median CPython/compiled speedup: nbody 28.5x (13.6-46.5), spectral-norm 17.9x (1.2-34.0), fannkuch-redux 29.2x (11.1-38.2), wordfreq 1.1x (0.9-1.6); binary-trees not compiled (M2b). Outputs identical in every pair. Refs #41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #41
Feature added
run.py interleaved [--pairs N] [--write]is a new measurement mode. For each benchmark it alternates CPython and the compiled extension A/B, flipping the order every pair, and reports:--write.Any pair whose outputs differ fails the run.
Why
The machine is never quiet during the day (load 5–33). The leader decided (2026-10-03) to use this interleaved A/B method as the official M2 measurement if no quiet window appears, reporting ratios rather than absolute times.
Verified
Smoke run at tiny sizes: all modes run and outputs agree in every pair. At those sizes process startup dominates, so the ratios mean nothing yet.