Skip to content

⚡ Measure test weights at f7a10637 - #865

Merged
taras merged 1 commit into
mainfrom
agent/measure-weights-f7a10637
Oct 1, 2026
Merged

taras merged 1 commit into
mainfrom
agent/measure-weights-f7a10637

Conversation

@taras

@taras taras commented Oct 1, 2026

Copy link
Copy Markdown
Owner

One commit, one file: test-weights.json, exactly as the Measure test
weights
run produced it.

Why

The committed weights were provenanced to 084397e1, which predates the end of
Slice B of #854. Nothing measured after it — Slices B (#859), C (#860), D (#861)
and E (#864) each merged without a weights commit, and four measurement attempts
in between were cancelled. Three suites that landed in those slices had no
recorded weight at all.

That is not free. A file without a recorded weight is charged the heaviest
weight the current corpus recorded
, so a new test is never treated as free and a
deleted outlier cannot inflate the fallback forever. With three unmeasured files
the charge was large and wrong in one direction:

runtime charged each three files charged actually take overstated by
deno 238,191 ms (cli-npm-bin.test.ts) 714,573 ms 31,312 ms 683 s
node 167,498 ms (plan-markdown.test.ts) 502,494 ms 18,748 ms 484 s
bun 103,624 ms (plan-markdown.test.ts) 310,872 ms 16,252 ms 295 s

The partition is longest-processing-time-first, so it was packing shards around
eleven minutes of Deno work that does not exist, and the files it displaced went
somewhere worse.

What changes

Before:

  • source.commit 084397e1, run 36655497896. repl-agent-interface.test.ts,
    repl-agent-journey.test.ts and repl-forms.test.ts absent.

After:

  • source.commit f7a10637, run
    36801798533.
    All three measured — under Deno, repl-agent-journey 16,939 ms,
    repl-agent-interface 9,505 ms, repl-forms 4,868 ms — and every other
    file's weight is this run's.

How it was produced

Dispatched against main at exactly f7a10637, the merged Slice E head, with
main CI already green on it. The workflow prepares the checkout in the usual
order, runs every applicable file alone under each runtime's own runner on
ubuntu-latest, and writes only after every measured file passed.

Provenance is supplied, never inferred: the commit, run URL, attempt, runner
label and the three runtime versions arrive in the environment, and a missing one
is a refusal rather than a default. The committed file is the uploaded artifact
byte for byte — verified with cmp against the download, before and after
the commit. No millisecond here was typed.

What must stay true

  • Weights describe a machine CI runs on. Measured on ubuntu-latest through
    the workflow, never on a laptop — which is what the provenance block exists to
    make checkable.
  • Shard counts are measured, not chosen. This measurement implies floors of
    14 (Deno), 8 (Node) and 5 (Bun); the installed 15 / 10 / 5 each
    satisfy their own, so nothing here forces a recalibration. Raising a count
    takes its own five consecutive runs on one fixed head, so ci.yml is
    untouched.

Scope

Included

  • test-weights.json, unchanged from the artifact.

Intentionally unchanged

  • ci.yml and the installed shard counts.
  • Every source, test and specification file. This PR changes no behavior.

Risks and limitations

  • Weights drift as the corpus does; this is a snapshot of f7a10637, not a
    standing guarantee. Remeasure after anything that moves the corpus, the
    exclusions or a runner command.
  • Three already-measured files moved by more than four seconds under Deno
    (cli-npm-bin 238s → 272s, cli-help 20s → 29s, build-npm 39s → 43s).
    That is ordinary runner variance at this scale and is why the partition is
    recomputed from the whole file rather than patched per entry.

Scope confirmation

  • Every changed file supports the purpose described above.
  • Unrelated cleanup and formatting changes are excluded.
  • Generated or mechanical changes are clearly identified — the whole diff is
    one generated artifact.
  • The description matches the final diff and test results.

`test-weights.json` was provenanced to `084397e1`, which predates the end of
Slice B of #854. Nothing measured after it: Slices B, C, D and E each merged
without a weights commit, four measurement attempts were cancelled along the
way, and the two REPL suites #861 and #864 added — plus the Elicit form suite —
had no recorded weight at all.

A file without one is charged the heaviest weight the current corpus recorded,
so a new test is never treated as free. Three of them were being charged
`cli-npm-bin.test.ts`'s 238s each under Deno: 715s of predicted work against
31s of real work. The partition was packing shards against a number that was
wrong by eleven minutes per runner.

Measured on the runner, at main's exact head, with the provenance that run
supplied: the commit, the run URL, the attempt, the runner label and the three
runtime versions all come from the environment and none of them is a default.
The file is the artifact byte for byte; no millisecond here was typed.

Shard counts are unchanged and remain measured rather than chosen. The floors
this measurement implies are 14 for Deno, 8 for Node and 5 for Bun, and the
installed 15, 10 and 5 each satisfy their own — so nothing here forces a
recalibration, which would take its own five consecutive runs on one fixed head.
@taras
taras merged commit 40346d9 into main Oct 1, 2026
43 of 44 checks passed
@taras
taras deleted the agent/measure-weights-f7a10637 branch October 1, 2026 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant