ci: move v3 benchmarks into a browser, drop CodSpeed, and measure v3 against v4 - #830
Merged
Merged
Conversation
CodSpeed is removed because it did not work. None of the last twelve merged pull requests changed a file under `packages/js-toolkit/`, yet it failed on five of them and all five merged anyway. On #829 it read fail, fail, pass, fail, pass while that package's content never changed once. A check that fails on an unchanged package is not a signal, and a signal everyone has learned to ignore costs more than no check. happy-dom is removed because the numbers were not comparable to anything. v4's suite runs in Chromium, and a v3-against-v4 figure means nothing until both sides share an environment. The browser is also the honest one on its own terms: v3's `SmartQueue` yields through `setTimeout`, whose clamping — and the `longtask` accounting that goes with it — exists only in a browser. `resolve.conditions` is kept, so the suite still measures `src/` rather than a stale `dist/`. The decorator transform is imported from v4 rather than copied, because `v3-vs-v4.bench.ts` pulls in v4 sources and stage-3 decorators are not lowered by Vite's TypeScript transformer. Files no longer run in parallel: they share the browser's one main thread, and sampled together they measure each other. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011nNdFD3aQhzfdm3EsCSbS9
…profile Mirrors v4's `mount-at-scale.bench.ts` on the v3 side and measures both versions in one browser session, so the rewrite's gains and losses are numbers rather than assumptions. Each version is driven through its own native path — v3's document mutation observer feeding `SmartQueue`, v4's observer feeding its scheduler. That difference is the subject, so nothing normalises it away. What is identical is the definition of done, and it belongs to neither framework: the Nth `mounted()` call, counted by the fixtures themselves, followed by a main thread that has gone quiet. Using v4's `whenDOMSettled()` would have let v4 define away work it defers. Throughput alone cannot say whether v4 gave back what `SmartQueue` was written to buy, so `mount-at-scale.profile.ts` reports the blocking profile too: settle time, time to a quiet thread, total blocking time, longest task and long-task count. It runs under `vitest run` because tinybench reports one number per benchmark and this needs five. **The two versions cannot share a document.** Both publish instances on `el.__base__`, so a live v4 registry walks a removed subtree, finds v3's entries and calls `$destroy()` on them — which throws outright once v3 has left its `'terminated'` marker there. They take the document in turns, v3 first, because v4 installs its mutation processor on first registration and never removes it while v3's registry is a plain map on `globalThis`. Every iteration swaps one page for another instead of appending to a growing one: v3 rescans the whole document on each mutation where v4 walks the inserted subtree, so a document that grew across a cycle would charge v3 for the benchmark's own bookkeeping. CI tracks the at-scale file at one size. The other ten benchmark files stay for local use: they were readable under instruction counting and are not readable on a wall clock, where Chromium's 100 us clamp reports a microsecond body as zero. 5 000 components is not sampled either — v3's queue drains with `Array#shift`, which V8 turns quadratic past ~16 000 entries, so one v3 swap there costs over five seconds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011nNdFD3aQhzfdm3EsCSbS9
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011nNdFD3aQhzfdm3EsCSbS9
Code ReviewRisk: Low — No concrete blocking or behavioural defects were found in the reviewed changes; the benchmark migration is safe to merge aside from the unreviewed lockfile. Moves the v3 benchmark suite from CodSpeed and happy-dom into Chromium, adds a cross-version mount-at-scale benchmark and blocking profile, and configures CI to compare base and head measurements. The fixtures coordinate v3 and v4's native registration paths while using framework-neutral mount and quiet-thread completion criteria. Review usage: 66,726 in (43,504 cached) / 1,495 out tokens — $0.0212 (openrouter/openai/gpt-5.6-luna, thinking: low) Reviewed by @weareikko/code-review v0.9.5 for commit a62fcef. Previous review runsPrevious run archived 2026-08-16T15:49:46ZCode ReviewRisk: Medium — issues that should be addressed before merge. Replaces CodSpeed/happy-dom with Chromium benchmarks, adds cross-version mounting fixtures and a blocking profile, and updates the benchmark configuration and changelog. The v3 workflow tracks only selected paths, but its benchmark imports and measures sources from 1 issue found:
Review usage: 78,074 in (42,658 cached) / 1,419 out tokens — $0.0285 (openrouter/openai/gpt-5.6-luna, thinking: low) Reviewed by @weareikko/code-review v0.9.5 for commit cf64cb4. Previous run archived 2026-08-16T15:19:30ZCode ReviewRisk: Low — no blocking issues; safe to merge aside from nits. Moves v3 benchmarks from CodSpeed and happy-dom into Chromium, adds cross-version mount-at-scale throughput and blocking profiles, and configures CI to compare base and head measurements. It also removes the old CodSpeed workflow and updates the changelog and benchmark dependencies. Review usage: 77,493 in (42,431 cached) / 1,045 out tokens — $0.0272 (openrouter/openai/gpt-5.6-luna, thinking: low) Reviewed by @weareikko/code-review v0.9.5 for commit f70abfd. Previous run archived 2026-08-16T15:17:46ZCode ReviewRisk: Low — no blocking issues; safe to merge aside from nits. Moves v3 benchmarks from CodSpeed and happy-dom to Chromium, adds the shared v3/v4 mount-at-scale suite, and introduces a blocking profile. The workflow tracks only the at-scale benchmark through the bench-diff action, while the profile reports settle, quiet, and long-task metrics locally. Review usage: 57,334 in (19,247 cached) / 775 out tokens — $0.0271 (openrouter/openai/gpt-5.6-luna, thinking: low) Reviewed by @weareikko/code-review v0.9.5 for commit 38b1118. |
Export sizeBundled per export with peer dependencies left external, dynamic imports excluded and the output minified; sizes are gzipped. ✅ No export size changes. Unchanged (388)@studiometa/js-toolkit
@studiometa/js-toolkit-v4
|
The comments predicted a 4-5 ms tail from a clamped `setTimeout(0)`. The measured idle round trip is 0.1-0.7 ms, so the threshold's justification and the caveat about understating the faster version were both wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011nNdFD3aQhzfdm3EsCSbS9
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #830 +/- ##
=======================================
Coverage 97.16% 97.16%
=======================================
Files 170 170
Lines 4133 4133
Branches 1151 1152 +1
=======================================
Hits 4016 4016
Misses 106 106
Partials 11 11
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
v3 mount benchmarksBase and head measured on this runner, alternating over 3 rounds each; every value is the median of the round medians. Running both sides on one machine is what removes cross-machine noise — a cached baseline from another runner would put it back. A move under 25%, or on a benchmark under 5 ms, is not reported as a change: it is inside the measured noise of a shared runner. The base commit has no benchmark suite, so every benchmark below is new. A base that failed to build or measure would have failed this job rather than appearing here.
|
v4 declares both in markup; v3's analogues are decorators wrapped around the component. Comparing them would compare a decorator against an attribute, which is a different unit of work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011nNdFD3aQhzfdm3EsCSbS9
v4 mount benchmarksBase and head measured on this runner, alternating over 3 rounds each; every value is the median of the round medians. Running both sides on one machine is what removes cross-machine noise — a cached baseline from another runner would put it back. A move under 25%, or on a benchmark under 5 ms, is not reported as a change: it is inside the measured noise of a shared runner. No benchmark moved beyond the noise floor. Within noise (18)
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
The cross-version suite imports `packages/v4/src/index.js` and reuses v4's SWC decorator plugin, so a v4-only pull request moves half of what it measures — and the path filter would have skipped it on exactly the changes most likely to regress the comparison. Found by @weareikko/code-review on #830. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011nNdFD3aQhzfdm3EsCSbS9
This was referenced Aug 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CodSpeed is gone, v3's benchmarks run in Chromium like v4's, and the mounting comparison between the two versions is now measured rather than assumed.
Why CodSpeed went
None of the last twelve merged pull requests changed a file under
packages/js-toolkit/, yet CodSpeed failed on five of them and all five merged anyway. On #829 it read fail → fail → pass → fail → pass while that package's content never changed once. A check that fails on an unchanged package is not a signal, and a signal everyone has learned to ignore costs more than no check. Regressions are tracked by.github/actions/bench-diffinstead: base and head measured on one runner, alternating, commented and never blocking.Why the browser
v4's suite already runs in Chromium. A v3-against-v4 figure measured with v3 in happy-dom and v4 in a real browser compares two environments, not two frameworks. The browser is also the honest one on its own terms: v3's
SmartQueueyields throughsetTimeout, whose clamping — and thelongtaskaccounting that goes with it — exists only in a browser.resolve.conditions: ['typescript']is kept, so the suite still measuressrc/.How "done" is defined
The Nth
mounted()call, then a main thread that has gone quiet. The count comes from the fixtures themselves, so it belongs to neither framework — using v4'swhenDOMSettled()would have let v4 define away work it defers. The quiet check awaits a timer probe and a background-task probe together, because v3 queues onsetTimeoutand v4 posts background tasks, and a probe on one queue can overtake pending work on the other. Its idle cost measures at 0.1–0.7 ms, so it is not a meaningful tax on either side.Each version is driven through its own native path — v3's document mutation observer feeding
SmartQueue, v4's observer feeding its scheduler. Nothing normalises that difference away; it is the subject.Every iteration swaps one page for another rather than appending to a growing one. v3 rescans the whole document on each mutation where v4 walks the inserted subtree, so a document that grew across a cycle would charge v3 for the benchmark's own bookkeeping. The cost is that a swap includes a destroy, which the
destroygroups price separately.The measurement
Local, Chromium, median of five swaps per cell.
npm run bench:profile -w @studiometa/js-toolkit.controlis adata-componentname nothing is registered under, with the observer live: the discovery floor. It mounts nothing, so only its quiet time means anything.v4's
inViewandresponsivescenarios have no v3 side. Both are declared in v4's markup (data-mount="in-view",data-option-x:s) and v3 has neither attribute — its analogues are thewithMountWhenInViewandwithResponsiveOptionsdecorators, which are a different unit of work wrapped around the component rather than a cost the registry pays per element. They stay in v4's own file.Read as ratios
Same scenario, same size, same observable definition of done, so these are fair by construction.
What the numbers say
v4 is faster and blocks less. There is no trade-off to report. The case the correction was written to catch — v4 slower end to end while blocking less, or the reverse — does not occur at any size or scenario. v4's total blocking time is zero everywhere v3's is zero, and lower everywhere v3's is not.
v3 wins the discovery floor. For a name nothing is registered under, v3 is 1.3–3.5× cheaper: it iterates its registry and never looks at an unknown name, while v4's observer walks the inserted subtree. That is a real cost v4 pays and v3 does not, and it is the one column where the rewrite went backwards. At 5 000 elements it is 27 ms against 21 ms — small against the 6-second gap in the row below it, but it is the honest counter-entry.
SmartQueuedoes not stop v3 blocking. At 5 000 components v3 produces a single 87–121 ms task on every scenario. The 40 ms budget only stops the queue starting another task past the limit; it cannot chunk a task that is itself long, and v3's registry scan — one queued closure that walks the document and constructs every instance — is exactly such a task. v4 produces no long task at all below the realistic 5 000 case.v3 falls off a cliff between 1 000 and 5 000 components. 65 ms → 6 635 ms is 102× for 5× the work; v4 goes 21 ms → 108 ms, which is linear. The cause looks like
SmartQueue.run(), which drains its task list withArray#shift. V8 abandons the fast path for large arrays, measured standalone on this machine:shiftEach v3
$mount()queues about six closures, so 5 000 components put roughly 30 000 tasks in one queue — squarely in the collapse. This is a v3 bug, not a benchmark artefact, and it is a small fix (Queue.runtakingtasks[i]instead oftasks.shift()). It is not fixed here: this pull request's job was to make the number visible, and changing v3's queue on the same branch would have measured the fix rather than the state of things. Worth its own issue.On a cross-version guard: I recommend against a hard gate
A spec asserting "v4's total blocking time is no worse than v3's" would be poison, for three reasons.
0.00. A threshold on a quantity that is zero until it suddenly is not gives no early warning and flakes at the boundary.packages/v4/src/mount-at-scale.spec.tsassertslongestTask === 0for 500 realistic components and bounds the settle time, with thresholds that do not depend on runner speed and do not mention v3. That is where "v4 must not block" belongs, and it is where a future change would actually be caught.So: keep the cross-version blocking numbers as a reported figure —
mount-at-scale.profile.ts, refreshed on demand — and if v4's blocking behaviour needs defending harder, tighten the absolute budget in v4's own spec rather than tying it to v3. The one v3-relative check I would consider is a very loose sanity floor (v4's longest task never more than 2× v3's), and even that will need retiring the day v3 is dropped, so it buys little.What CI samples
benchmarks-v3.yml, filtered topackages/js-toolkit/**, runsmount-at-scale.bench.tsonly, at size 1 000, three rounds per side. Measured on this pull request: 1 min 33 s end to end, install and Chromium download included — in the same range asbenchmarks-v4.yml.npm run bench). They were readable under instruction counting and are not readable on a wall clock: Chromium clampsperformance.now()to 100 µs, so a microsecond body reports zero.Array#shiftreason above.benchmarks-v4.yml, for the reason the bench-diff README gives: two suites in one job share a base checkout, and only a workflow-levelpaths:filter keeps each off the pull requests that do not concern it. The base of this pull request has nomount-at-scale.bench.ts, so the first run will report an empty base and say so — the action's documented behaviour for the pull request that adds a suite.One thing found along the way
v3 and v4 cannot share a document. Both publish their instances on
el.__base__. A live v4 registry walks a removed subtree, finds v3's entries there and calls$destroy()on them, which throws outright once v3 has left its'terminated'string marker in that map. This is why the two versions take the document in turns here, v3 first — v4 installs its mutation processor on first registration and never removes it, while v3's registry is a plain map onglobalThisthat can be emptied. It also means a page cannot run both versions during a migration unless one of them is left unregistered. Worth knowing before anyone plans an incremental rollout.The sticky comment on this pull request is the action's first run for this suite: the base has no
mount-at-scale.bench.ts, so it reports an empty base and says so, which is its documented behaviour for the pull request that adds a suite. Its numbers also reproduce the ratios above on a different machine — v3 32.4 ms against v4 12.4 ms flat, 106.1 against 59.5 realistic, 24.4 against 3.2 destroy.Verification
npm run lint,npm run lint:types,npm test(113 files, 982 passed),npm run test:v4(79 files, 1 066 passed), and the full browser benchmark suite all green.🤖 Generated with Claude Code
https://claude.ai/code/session_011nNdFD3aQhzfdm3EsCSbS9