⚡ Measure test weights at f7a10637 - #865
Merged
Merged
Conversation
`test-weights.json` was provenanced to `084397e1`, which predates the end of Slice B of #854. Nothing measured after it: Slices B, C, D and E each merged without a weights commit, four measurement attempts were cancelled along the way, and the two REPL suites #861 and #864 added — plus the Elicit form suite — had no recorded weight at all. A file without one is charged the heaviest weight the current corpus recorded, so a new test is never treated as free. Three of them were being charged `cli-npm-bin.test.ts`'s 238s each under Deno: 715s of predicted work against 31s of real work. The partition was packing shards against a number that was wrong by eleven minutes per runner. Measured on the runner, at main's exact head, with the provenance that run supplied: the commit, the run URL, the attempt, the runner label and the three runtime versions all come from the environment and none of them is a default. The file is the artifact byte for byte; no millisecond here was typed. Shard counts are unchanged and remain measured rather than chosen. The floors this measurement implies are 14 for Deno, 8 for Node and 5 for Bun, and the installed 15, 10 and 5 each satisfy their own — so nothing here forces a recalibration, which would take its own five consecutive runs on one fixed head.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One commit, one file:
test-weights.json, exactly as the Measure testweights run produced it.
Why
The committed weights were provenanced to
084397e1, which predates the end ofSlice B of #854. Nothing measured after it — Slices B (#859), C (#860), D (#861)
and E (#864) each merged without a weights commit, and four measurement attempts
in between were cancelled. Three suites that landed in those slices had no
recorded weight at all.
That is not free. A file without a recorded weight is charged the heaviest
weight the current corpus recorded, so a new test is never treated as free and a
deleted outlier cannot inflate the fallback forever. With three unmeasured files
the charge was large and wrong in one direction:
cli-npm-bin.test.ts)plan-markdown.test.ts)plan-markdown.test.ts)The partition is longest-processing-time-first, so it was packing shards around
eleven minutes of Deno work that does not exist, and the files it displaced went
somewhere worse.
What changes
Before:
source.commit084397e1, run36655497896.repl-agent-interface.test.ts,repl-agent-journey.test.tsandrepl-forms.test.tsabsent.After:
source.commitf7a10637, run36801798533.All three measured — under Deno,
repl-agent-journey16,939 ms,repl-agent-interface9,505 ms,repl-forms4,868 ms — and every otherfile's weight is this run's.
How it was produced
Dispatched against
mainat exactlyf7a10637, the merged Slice E head, withmain CI already green on it. The workflow prepares the checkout in the usual
order, runs every applicable file alone under each runtime's own runner on
ubuntu-latest, and writes only after every measured file passed.Provenance is supplied, never inferred: the commit, run URL, attempt, runner
label and the three runtime versions arrive in the environment, and a missing one
is a refusal rather than a default. The committed file is the uploaded artifact
byte for byte — verified with
cmpagainst the download, before and afterthe commit. No millisecond here was typed.
What must stay true
ubuntu-latestthroughthe workflow, never on a laptop — which is what the provenance block exists to
make checkable.
14 (Deno), 8 (Node) and 5 (Bun); the installed 15 / 10 / 5 each
satisfy their own, so nothing here forces a recalibration. Raising a count
takes its own five consecutive runs on one fixed head, so
ci.ymlisuntouched.
Scope
Included
test-weights.json, unchanged from the artifact.Intentionally unchanged
ci.ymland the installed shard counts.Risks and limitations
f7a10637, not astanding guarantee. Remeasure after anything that moves the corpus, the
exclusions or a runner command.
(
cli-npm-bin238s → 272s,cli-help20s → 29s,build-npm39s → 43s).That is ordinary runner variance at this scale and is why the partition is
recomputed from the whole file rather than patched per entry.
Scope confirmation
one generated artifact.