diff --git a/.github/actions/bench-diff/README.md b/.github/actions/bench-diff/README.md new file mode 100644 index 000000000..67aca6f65 --- /dev/null +++ b/.github/actions/bench-diff/README.md @@ -0,0 +1,90 @@ +# bench-diff + +Run a [vitest](https://vitest.dev) benchmark suite for the base and the head commit **on one runner**, alternating between them, and upsert a sticky pull-request comment with the difference. + +It is the timing counterpart of [`weareikko/export-size`](https://github.com/weareikko/export-size): same shape — measure head, check the base sha out into a subdirectory of the same runner, measure it with the same action-provided script, diff, comment — with the changes a stopwatch needs that a byte counter does not. + +## Why it works this way + +**Both sides on one runner.** Cross-machine timing noise is the reason services like CodSpeed exist. Measuring the base on the runner that measures the head removes it, with no account, no token and nothing to store. A cached baseline from an earlier `main` run would put that noise straight back, and would be un-interleavable by construction. + +**Alternating, not sequential.** Export-size measures head then base. For byte counts that is fine. For timings it puts every bit of thermal drift and every noisy neighbour on one side of the comparison. This action alternates the sides within a round and flips the order between rounds, so drift lands on both. + +**Median, never mean.** A benchmark's value for a round is tinybench's median; its value for the report is the median across rounds. One GC pause should not become the headline. + +**A measured threshold, and a resolution floor.** Run one commit against itself, read the spread, and set `threshold` from it. Benchmarks whose median falls under `floor` milliseconds are reported but never flagged: Chromium clamps `performance.now()` to 100 µs, so a 1 ms benchmark cannot resolve a percentage. + +**It comments; it does not block — but it fails when it could not measure.** A timing _threshold_ that fails a build on a shared runner is a threshold that gets deleted, so regressions are reported rather than enforced. A broken _measurement_ is the opposite: the job fails, because a comparison that silently did not happen is worse than no comparison. The one state that degrades is a base with no benchmark suite yet, which is a real thing on the pull request that adds one. Absent is not broken: + +| side | every round measured | none measured | some measured | +| ---- | -------------------- | ----------------------------------------- | ------------- | +| head | compare | fail | fail | +| base | compare | empty base, warn, and the comment says so | fail | + +Base `install` and `prepare` are never suppressed either. That distinction is the whole reason counting output is worth doing. + +**It knows nothing about environments.** It takes a directory, a command and an output path, and reads the JSON vitest emits — the same schema whether the benchmark bodies run in Node, in happy-dom or in a real browser over CDP. A second suite is another step with another `id`, not a branch inside the action. Keep it that way. + +## Usage + +A browser suite, which needs a browser downloaded first: + +```yaml +- uses: actions/checkout@v4 +- uses: ./.github/actions/bench-diff + with: + id: v4-mount + title: v4 mount benchmarks + unit: component + working-directory: packages/v4 + prepare: npx playwright install --with-deps chromium + bench: npm exec vitest bench -- --config vitest.bench.config.js --run --outputJson "$BENCH_JSON" + rounds: '3' + threshold: '25' + floor: '5' +``` + +A Node suite, which needs nothing extra — the only difference is the command: + +```yaml +- uses: ./.github/actions/bench-diff + with: + id: v3 + title: v3 benchmarks + working-directory: packages/js-toolkit + bench: npm exec vitest bench -- --config vitest.bench.config.ts --run --outputJson "$BENCH_JSON" +``` + +`bench` must write a `vitest bench --outputJson` file to `$BENCH_JSON`. Anything the suite needs from the environment can be set as `env:` on the step — composite `run` steps inherit it — or written as a prefix assignment on the command itself, which is unambiguous. + +`id` namespaces the sticky comment and the temporary files, so several suites can each keep their own comment on one pull request. Two suites in the _same job_ would share the base checkout; prefer a job or a workflow each, which also lets each one carry its own `paths:` filter. + +The job needs `pull-requests: write` to comment. + +### Inputs + +| input | default | meaning | +| ------------------- | ----------------------------- | ---------------------------------------------------------- | +| `id` | `bench-diff` | Suite identifier; namespaces the comment and temp files. | +| `title` | `Benchmarks` | Heading of the sticky comment. | +| `unit` | `unit` | What a group's count counts, for the per-unit column. | +| `bench` | — | Command writing a bench JSON to `$BENCH_JSON`. | +| `working-directory` | `.` | Where to run it, inside each checkout. | +| `install` | `npm ci --no-audit --no-fund` | Dependency install, at each checkout root. | +| `prepare` | — | Anything else each checkout needs before benchmarking. | +| `rounds` | `3` | Sampling rounds per side. | +| `threshold` | `25` | Percent change reported as a change. | +| `floor` | `5` | Benchmarks under this many ms are reported, never flagged. | +| `comment` | `true` | Upsert the sticky comment. | + +## Reading a group as a per-unit cost + +`bench-report.mjs` divides a benchmark's median by the first integer in its group title. A group named `mount 5000 components, one insertion` therefore reports microseconds per component alongside milliseconds per operation, which is what makes a non-linear curve legible as a number rather than as a shape. Name the unit with `unit:`. A group whose title carries no number simply has no per-unit figure, and the column reads `-`. + +## Locally + +```sh +node .github/actions/bench-diff/bench-report.mjs # one run, as a table +node .github/actions/bench-diff/bench-report.mjs a.json b.json --json out.json # several rounds, aggregated +node .github/actions/bench-diff/bench-comment.mjs base.json head.json # the comment body +``` diff --git a/.github/actions/bench-diff/action.yml b/.github/actions/bench-diff/action.yml new file mode 100644 index 000000000..fac35a787 --- /dev/null +++ b/.github/actions/bench-diff/action.yml @@ -0,0 +1,208 @@ +name: 'Benchmark diff' +description: 'Run a vitest benchmark suite for the base and the head commit on one runner, alternating, and comment the difference.' +author: 'Studio Meta' +branding: + icon: 'activity' + color: 'blue' + +# The action knows nothing about how a suite runs. It takes a directory, a +# command and an output path, and reads the JSON vitest emits — the same +# schema whether the benchmark bodies execute in Node or in a browser. Keep it +# that way: a second suite is added by adding a step, not by adding a branch. + +inputs: + id: + description: 'Identifier for this suite. Namespaces the sticky comment and the temporary files, so several suites can each keep their own comment on one pull request.' + required: false + default: 'bench-diff' + title: + description: 'Heading of the sticky comment.' + required: false + default: 'Benchmarks' + unit: + description: "What a group's count counts, for the per-unit column — `component`, `element`, `node`. Groups whose title carries no number have no per-unit figure." + required: false + default: 'unit' + bench: + description: 'Command producing a `vitest bench --outputJson` file at `$BENCH_JSON`, run in `working-directory` inside each checkout.' + required: true + working-directory: + description: 'Directory to run `bench` in, relative to a checkout root.' + required: false + default: '.' + install: + description: 'Command that installs dependencies, run at the root of each checkout.' + required: false + default: 'npm ci --no-audit --no-fund' + prepare: + description: 'Optional command run after `install` in each checkout, for anything the benchmarks need on disk.' + required: false + default: '' + rounds: + description: 'Sampling rounds per side. Sides alternate within a round, and the order flips between rounds, so thermal drift and runner contention land on both sides equally.' + required: false + default: '3' + threshold: + description: 'Percent change reported as a change. Set it from a measured same-commit noise floor, never from taste.' + required: false + default: '25' + floor: + description: 'Benchmarks with a median under this many milliseconds are reported but never flagged: the timer cannot resolve them.' + required: false + default: '5' + node-version: + description: 'Node.js version.' + required: false + default: '24' + comment: + description: 'Upsert a sticky pull-request comment with the diff.' + required: false + default: 'true' + github-token: + description: 'Token used to upsert the comment (needs `pull-requests: write`).' + required: false + default: ${{ github.token }} + +runs: + using: composite + steps: + - name: Set up Node.js + uses: actions/setup-node@v4 + with: + node-version: ${{ inputs.node-version }} + cache: npm + + - name: Install the pull request + shell: bash + run: | + ${{ inputs.install }} + ${{ inputs.prepare }} + + # The base is measured on this runner, from the same action-provided + # scripts, rather than read from a cache. A cached baseline comes from + # another machine under other contention, which re-imports exactly the + # cross-machine noise this design exists to remove — and a cached number + # cannot be interleaved with anything. + - name: Check out the base branch + if: github.event_name == 'pull_request' + uses: actions/checkout@v4 + with: + ref: ${{ github.event.pull_request.base.sha }} + path: __bench-diff-base + + # No `|| true` here. A base that cannot install or prepare is a broken + # job, and suppressing it would turn a broken baseline into a confident + # "everything is new" comment — a green report of a comparison that never + # happened. Only a base with no benchmark suite may degrade, and that is + # decided further down, from whether it produced any output at all. + - name: Install the base branch + if: github.event_name == 'pull_request' + shell: bash + working-directory: __bench-diff-base + run: | + ${{ inputs.install }} + ${{ inputs.prepare }} + + - name: Run both sides, alternating + shell: bash + env: + BENCH_ID: ${{ inputs.id }} + BENCH_ROUNDS: ${{ inputs.rounds }} + BENCH_WORKDIR: ${{ inputs.working-directory }} + BENCH_HAS_BASE: ${{ github.event_name == 'pull_request' }} + run: | + # `shell: bash` already runs with `-e -o pipefail`; saying so here + # keeps the guarantee visible to whoever edits this next. + set -euo pipefail + run_side() { + local side="$1" round="$2" root="$3" + local out="$RUNNER_TEMP/$BENCH_ID-$side-$round.json" + # `export` on its own line, not a prefix assignment: `$BENCH_JSON` + # inside the command is expanded by this shell before the command + # runs, so a prefix assignment would come too late for it. + ( cd "$root/$BENCH_WORKDIR" && export BENCH_JSON="$out" && ${{ inputs.bench }} ) \ + || echo "::warning::the $side benchmark run failed in round $round" + } + for round in $(seq 1 "$BENCH_ROUNDS"); do + if [ "$BENCH_HAS_BASE" != 'true' ]; then + run_side head "$round" "$GITHUB_WORKSPACE" + elif [ $((round % 2)) -eq 1 ]; then + run_side head "$round" "$GITHUB_WORKSPACE" + run_side base "$round" "$GITHUB_WORKSPACE/__bench-diff-base" + else + run_side base "$round" "$GITHUB_WORKSPACE/__bench-diff-base" + run_side head "$round" "$GITHUB_WORKSPACE" + fi + done + + # Three outcomes per side, and they are not the same thing. + # + # every round measured -> compare + # no round measured -> only the base may do this, and only because + # "the suite does not exist on this commit yet" + # is a real, expected state on the pull request + # that adds it. Degrade to an empty base, and + # say so in the comment. + # some rounds measured -> a suite that ran and then did not is broken, + # not absent. Never degrade; fail. + # + # Counting output is what separates absent from broken without the action + # having to know anything about the command it was given. + - name: Aggregate the rounds + shell: bash + env: + BENCH_ID: ${{ inputs.id }} + BENCH_ROUNDS: ${{ inputs.rounds }} + BENCH_HAS_BASE: ${{ github.event_name == 'pull_request' }} + run: | + set -euo pipefail + shopt -s nullglob + head_rounds=("$RUNNER_TEMP/$BENCH_ID"-head-*.json) + base_rounds=("$RUNNER_TEMP/$BENCH_ID"-base-*.json) + + if [ ${#head_rounds[@]} -ne "$BENCH_ROUNDS" ]; then + echo "::error::the head commit measured ${#head_rounds[@]} of $BENCH_ROUNDS rounds" >&2 + exit 1 + fi + node "$GITHUB_ACTION_PATH/bench-report.mjs" "${head_rounds[@]}" \ + --json "$RUNNER_TEMP/$BENCH_ID-head.json" --markdown + + if [ "$BENCH_HAS_BASE" != 'true' ] || [ ${#base_rounds[@]} -eq 0 ]; then + if [ "$BENCH_HAS_BASE" = 'true' ]; then + echo '::warning::the base commit produced no benchmark output; reporting every benchmark as new' >&2 + fi + echo '[]' > "$RUNNER_TEMP/$BENCH_ID-base.json" + elif [ ${#base_rounds[@]} -ne "$BENCH_ROUNDS" ]; then + echo "::error::the base commit measured ${#base_rounds[@]} of $BENCH_ROUNDS rounds, so its suite is broken rather than absent" >&2 + exit 1 + else + node "$GITHUB_ACTION_PATH/bench-report.mjs" "${base_rounds[@]}" \ + --json "$RUNNER_TEMP/$BENCH_ID-base.json" + fi + + - name: Comment on the pull request + if: inputs.comment == 'true' && github.event_name == 'pull_request' + shell: bash + env: + GH_TOKEN: ${{ inputs.github-token }} + BENCH_ID: ${{ inputs.id }} + run: | + node "$GITHUB_ACTION_PATH/bench-comment.mjs" \ + "$RUNNER_TEMP/$BENCH_ID-base.json" "$RUNNER_TEMP/$BENCH_ID-head.json" \ + --threshold '${{ inputs.threshold }}' \ + --floor '${{ inputs.floor }}' \ + --rounds '${{ inputs.rounds }}' \ + --id '${{ inputs.id }}' \ + --title '${{ inputs.title }}' \ + --unit '${{ inputs.unit }}' > "$RUNNER_TEMP/$BENCH_ID-comment.md" + cat "$RUNNER_TEMP/$BENCH_ID-comment.md" >> "$GITHUB_STEP_SUMMARY" + marker="" + repo='${{ github.repository }}' + pr='${{ github.event.pull_request.number }}' + comment_id=$(gh api "repos/$repo/issues/$pr/comments" --paginate \ + --jq ".[] | select(.body | contains(\"$marker\")) | .id" | head -n1) + if [ -n "$comment_id" ]; then + gh api --method PATCH "repos/$repo/issues/comments/$comment_id" -F body=@"$RUNNER_TEMP/$BENCH_ID-comment.md" >/dev/null + else + gh api --method POST "repos/$repo/issues/$pr/comments" -F body=@"$RUNNER_TEMP/$BENCH_ID-comment.md" >/dev/null + fi diff --git a/.github/actions/bench-diff/bench-comment.mjs b/.github/actions/bench-diff/bench-comment.mjs new file mode 100644 index 000000000..cce5e39c8 --- /dev/null +++ b/.github/actions/bench-diff/bench-comment.mjs @@ -0,0 +1,183 @@ +// Render a pull-request comment diffing two `bench-report.mjs --json` reports +// (base vs head) and print it to stdout. The workflow upserts the output as a +// sticky comment, matched by the marker below. +// +// Timings are not byte counts. Two things follow, and both are visible in the +// comment itself so nobody reads a number as more precise than it is: +// +// * A change under `--threshold` percent is not reported as a change. The +// default is set from a measured same-commit noise floor, not from taste. +// * A benchmark whose median is under `--floor` milliseconds is never +// flagged at all. A sub-millisecond benchmark cannot resolve a 25 % move: +// Chromium clamps `performance.now()` to 100 us, and a Node suite hits +// its own resolution and scheduling limits at a similar scale. +import { readFileSync } from 'node:fs'; + +const args = process.argv.slice(2); +const files = []; +let threshold = 25; +let floor = 5; +let rounds = 0; +// The suite's identity, so several suites can each keep their own sticky +// comment in one repository. Nothing here knows whether a suite runs in Node +// or in a browser: it reads the JSON vitest emits either way. +let id = 'bench-diff'; +let title = 'Benchmarks'; +let unit = 'unit'; + +const flags = { + '--threshold': (value) => { + threshold = Number(value); + }, + '--floor': (value) => { + floor = Number(value); + }, + '--rounds': (value) => { + rounds = Number(value); + }, + '--id': (value) => { + id = value; + }, + '--title': (value) => { + title = value; + }, + '--unit': (value) => { + unit = value; + }, +}; + +for (let index = 0; index < args.length; index += 1) { + const flag = flags[args[index]]; + if (flag) { + index += 1; + flag(args[index]); + } else { + files.push(args[index]); + } +} + +const MARKER = ''; + +const [basePath, headPath] = files; +if (!basePath || !headPath) { + console.error( + 'Usage: node bench-comment.mjs [--threshold 25] [--floor 5] [--rounds 3] [--id name] [--title text] [--unit component]', + ); + process.exit(1); +} + +// No fallback to `[]` on a read or parse error. An absent base is expressed +// by a report that *is* `[]`, written deliberately by the action, so a file +// that cannot be read or parsed means the measurement broke — and turning +// that into an empty base would report a comparison that never happened. Let +// it throw; a non-zero exit fails the step. +function read(path) { + return JSON.parse(readFileSync(path, 'utf8')); +} + +function index(entries) { + const map = new Map(); + for (const entry of entries) { + map.set(entry.group + ' ' + entry.name, entry); + } + return map; +} + +const base = index(read(basePath)); +const head = index(read(headPath)); + +function ms(value) { + return value === undefined ? '-' : value.toFixed(value < 10 ? 2 : 1) + ' ms'; +} + +function per(entry) { + return entry?.perUnit === undefined ? '-' : entry.perUnit.toFixed(2); +} + +const keys = [...new Set([...base.keys(), ...head.keys()])].sort(); +const changed = []; +const steady = []; + +for (const key of keys) { + const b = base.get(key); + const h = head.get(key); + const entry = h ?? b; + const delta = b && h ? ((h.median - b.median) / b.median) * 100 : undefined; + // Below the resolution floor, or too small a move to distinguish from the + // runner: report the value, but never as a change. + const resolvable = b && h && Math.max(b.median, h.median) >= floor; + const significant = resolvable && Math.abs(delta) >= threshold; + const row = + '| ' + + entry.group + + ' | ' + + entry.name + + ' | ' + + ms(b?.median) + + ' | ' + + ms(h?.median) + + ' | ' + + per(h ?? b) + + ' | ' + + (delta === undefined ? 'new' : (delta > 0 ? '+' : '') + delta.toFixed(1) + '%') + + ' |'; + if (significant) { + changed.push({ delta, row }); + } else { + steady.push(row); + } +} + +changed.sort((a, z) => z.delta - a.delta); + +const HEADER = [ + '| Group | Benchmark | Base | Head | us / ' + unit + ' | Change |', + '| :-- | :-- | --: | --: | --: | --: |', +]; + +const body = [MARKER, '## ' + title, '']; +body.push( + 'Base and head measured on this runner, alternating' + + (rounds ? ' over ' + rounds + ' rounds each' : '') + + '; every value is the median of the round medians. Running both sides on one machine is what removes cross-machine noise — a cached baseline from another runner would put it back.', + '', + 'A move under **' + + threshold + + '%**, or on a benchmark under **' + + floor + + ' ms**, is not reported as a change: it is inside the measured noise of a shared runner.', + '', +); + +if (base.size === 0) { + // The base has no benchmark suite — the expected state on the pull request + // that adds one. A base that failed to install, prepare, or measure only + // some of its rounds never reaches here: the action fails the job instead, + // so an empty base always means absent and never means broken. Show every + // row, since none of them is hidden noise; they are the whole story. + body.push( + 'The base commit has no benchmark suite, so every benchmark below is new. A base that failed to build or measure would have failed this job rather than appearing here.', + '', + ...HEADER, + ...steady, + '', + ); +} else { + if (changed.length === 0) { + body.push('No benchmark moved beyond the noise floor.', ''); + } else { + body.push(...HEADER, ...changed.map((entry) => entry.row), ''); + } + if (steady.length > 0) { + body.push( + '
Within noise (' + steady.length + ')', + '', + ...HEADER, + ...steady, + '', + '
', + ); + } +} + +console.log(body.join('\n')); diff --git a/.github/actions/bench-diff/bench-report.mjs b/.github/actions/bench-diff/bench-report.mjs new file mode 100644 index 000000000..8dadf96fe Binary files /dev/null and b/.github/actions/bench-diff/bench-report.mjs differ diff --git a/.github/workflows/benchmarks-v4.yml b/.github/workflows/benchmarks-v4.yml new file mode 100644 index 000000000..ae36b7c8a --- /dev/null +++ b/.github/workflows/benchmarks-v4.yml @@ -0,0 +1,66 @@ +name: benchmarks-v4 + +# v4's benchmarks run in a real Chromium, so they cannot share the v3 job in +# `benchmarks.yml`: CodSpeed's simulation mode instruments the Node process, +# while a browser-mode benchmark body runs in the browser over CDP, so it +# would measure the driver. This job compares the base and the head commit on +# one runner instead, which removes cross-machine noise without an account, a +# token or a stored baseline. +# +# It comments; it never blocks. The hard gate is `src/mount-at-scale.spec.ts` +# in the normal test job, whose thresholds do not depend on runner speed. +# +# Separate file rather than a job in `benchmarks.yml` because only a +# workflow-level `paths:` filter can keep it off pull requests that do not +# touch v4, and `benchmarks.yml` must keep running for v3. + +on: + pull_request: + branches: + - main + paths: + - 'packages/v4/**' + - 'package-lock.json' + - '.github/actions/bench-diff/**' + - '.github/workflows/benchmarks-v4.yml' + +permissions: + contents: read + pull-requests: write + +jobs: + bench_diff: + name: Compare against the base commit + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@v4 + - uses: ./.github/actions/bench-diff + with: + # Identity, not configuration: the action knows nothing about + # browsers or about v4. A second suite — v3's Node and happy-dom + # benchmarks, say — is another step with another `id`, not a branch + # inside the action. + id: v4-mount + title: v4 mount benchmarks + unit: component + working-directory: packages/v4 + # No build: vitest runs the browser suite straight from TypeScript. + prepare: npx playwright install --with-deps chromium + # The at-scale file only. Adding the other three v4 benchmark files + # costs ~55 s per run, so ~5.5 min across six runs, for micro- + # benchmarks a wall-clock diff can barely resolve. + # + # The full size matrix is 100/1 000/5 000. CI samples the two sizes + # above the timer's resolution floor: a 100-component benchmark lands + # near 1 ms, where Chromium's 100 us clamp makes a percentage + # meaningless. It is set on the command rather than as step `env:`, + # which a composite action's steps do not reliably inherit. + bench: V4_BENCH_SIZES=1000,5000 npm exec vitest bench -- --config vitest.bench.config.js --run src/mount-at-scale.bench.ts --outputJson "$BENCH_JSON" + rounds: '3' + # Same-commit noise floor, three interleaved rounds per side: 4.5 % + # median, 12.5 % p90, 33 % worst — and every case above 13 % is a + # benchmark under 2 ms, which `floor` excludes anyway. 25 % leaves + # room for a shared runner being noisier than a developer machine. + threshold: '25' + floor: '5' diff --git a/.gitignore b/.gitignore index 897c5f429..907afb4a8 100644 --- a/.gitignore +++ b/.gitignore @@ -31,6 +31,9 @@ packages/demo/dist/ # Module graph reports written by `npm run measure -- --json ` .measure-*.json + +# Benchmark output written by `npm run bench:v4` +.bench-report.json /packages/docs/.vitepress/cache /packages/docs/.vitepress/dist coverage/ diff --git a/package.json b/package.json index f43063a19..27e60de19 100644 --- a/package.json +++ b/package.json @@ -25,6 +25,7 @@ "test:eslint-plugin": "npm run test -w @studiometa/eslint-plugin-js-toolkit", "bench": "npm run bench -w @studiometa/js-toolkit", "bench:run": "npm run bench:run -w @studiometa/js-toolkit", + "bench:v4": "npm run bench:report -w @studiometa/js-toolkit-v4", "lint": "npm run subpaths:check && npm run lint:static && npm run lint:fmt && npm run lint:types", "lint:static": "oxlint .", "lint:fmt": "oxfmt --check .", diff --git a/packages/v4/package.json b/packages/v4/package.json index ba3c7973f..977781667 100644 --- a/packages/v4/package.json +++ b/packages/v4/package.json @@ -6,7 +6,7 @@ "sideEffects": [ "./dist/responsive-options.js" ], - "description": "v4 prototype — tested in a real browser through Vitest browser mode", + "description": "v4 prototype \u2014 tested in a real browser through Vitest browser mode", "files": [ "dist" ], @@ -767,6 +767,7 @@ "test:watch": "vitest", "test:debug": "vitest --browser.headless=false", "bench": "vitest bench --config vitest.bench.config.js", + "bench:report": "npm run bench -- --run --outputJson .bench-report.json && node ../../.github/actions/bench-diff/bench-report.mjs .bench-report.json", "lint:types": "tsc -p tsconfig.json" }, "dependencies": { diff --git a/packages/v4/scripts/build.js b/packages/v4/scripts/build.js index 1ce81f74b..9dc7fdf06 100644 --- a/packages/v4/scripts/build.js +++ b/packages/v4/scripts/build.js @@ -6,14 +6,16 @@ const pkgRoot = resolve(dirname(new URL(import.meta.url).pathname), '..'); const srcRoot = resolve(pkgRoot, 'src'); const outDir = resolve(pkgRoot, 'dist'); -// Every consumer module under `src/`. Specs, benchmarks and test utilities stay -// source-only. `unbundle` keeps the emitted `dist/` tree one-to-one with the entries. +// Every consumer module under `src/`. Specs, benchmarks, fixtures and test +// utilities stay source-only. `unbundle` keeps the emitted `dist/` tree +// one-to-one with the entries. const entryPoints = glob.sync( [ '**/*.ts', '!**/*.d.ts', '!**/*.spec.ts', '!**/*.bench.ts', + '!**/*.fixtures.ts', '!test-utils.ts', '!**/node_modules/**', ], diff --git a/packages/v4/scripts/check-package.js b/packages/v4/scripts/check-package.js index 193337c21..9d05ee29e 100644 --- a/packages/v4/scripts/check-package.js +++ b/packages/v4/scripts/check-package.js @@ -55,12 +55,13 @@ function assertPackageContent(metadata) { const forbiddenTests = files.filter( (path) => - path.startsWith('dist/test-utils.') || /\.(?:spec|bench)\.(?:js|js\.map|d\.ts)$/.test(path), + path.startsWith('dist/test-utils.') || + /\.(?:spec|bench|fixtures)\.(?:js|js\.map|d\.ts)$/.test(path), ); assert.deepEqual( forbiddenTests, [], - `Test utilities, specs and benchmarks must not be packed:\n${forbiddenTests.join('\n')}`, + `Test utilities, specs, benchmarks and fixtures must not be packed:\n${forbiddenTests.join('\n')}`, ); assert(fileSet.has('dist/index.js'), 'dist/index.js is missing from the package.'); diff --git a/packages/v4/src/mount-at-scale.bench.ts b/packages/v4/src/mount-at-scale.bench.ts new file mode 100644 index 000000000..cc78c09b3 --- /dev/null +++ b/packages/v4/src/mount-at-scale.bench.ts @@ -0,0 +1,159 @@ +/** + * Mounting throughput at page scale. + * + * `mount.bench.ts` measures one instance; this measures a page of them. The + * quantity that matters is not the total but the cost per component, and + * whether it stays flat as the count grows — `.github/actions/bench-diff` + * divides each result by the count in its group title to report it. + * + * Every number here is a settle time, from the DOM write to + * `whenDOMSettled()`. It is not a blocking time: eager mounts are posted one + * background task per element and drained inside the scheduler's 5 ms budget. + * `mount-at-scale.spec.ts` guards the part that does block. + */ +import { bench, describe } from 'vitest'; +import { whenDOMSettled } from './dom-mutations.js'; +import { + buildPool, + clearDocument, + registerScaleComponents, + scenarios, + type ScenarioName, +} from './mount-at-scale.fixtures.js'; + +registerScaleComponents(); + +/** Injected by `vitest.bench.config.js` from `V4_BENCH_SIZES`. */ +declare const __BENCH_SIZES__: number[]; + +/** The page weights the design has to hold up under. */ +const SIZES = __BENCH_SIZES__; + +/** Arrivals for the batching comparison — one morph per chunk. */ +const BATCHES = 10; + +/** + * Sample counts, not sample time: 5 000 realistic components take ~500 ms to + * settle, so tinybench's 500 ms default would stop at its ten-iteration floor + * for the small sizes and at one sample for the large ones. Fixing the count + * keeps every size comparably sampled and the suite bounded. + */ +function samplesFor(size: number) { + return { + time: 0, + iterations: size >= 5000 ? 9 : size >= 1000 ? 13 : 17, + warmupTime: 0, + warmupIterations: 1, + }; +} + +/** + * Insert a prebuilt tree and wait for the registry to finish with it. + * + * Trees are cloned in the per-cycle `setup` hook, never in the timed body: + * vitest exposes no per-iteration hook, so anything the body touches is + * measured. Hosts accumulate across a cycle's iterations, which is harmless — + * the registry scans the inserted subtree only, never the document. + */ +function mountBench(label: string, scenario: ScenarioName, size: number): void { + const options = samplesFor(size); + let pool: HTMLElement[] = []; + let index = 0; + + bench( + label, + async () => { + document.body.append(pool[index]); + index += 1; + await whenDOMSettled(); + }, + { + ...options, + async setup() { + await clearDocument(); + pool = buildPool(scenarios[scenario](size), options.iterations + 1); + index = 0; + }, + // Drop the cycle's trees so the next benchmark does not start under the + // heap pressure this one left behind. + teardown() { + document.body.replaceChildren(); + pool = []; + }, + }, + ); +} + +describe.each(SIZES)('mount %i components, one insertion', (size) => { + mountBench('control — declared but unregistered', 'control', size); + mountBench('flat', 'flat', size); + mountBench('nested 4 deep', 'nested', size); + mountBench('realistic — 5 refs, 3 options, 4 handlers', 'realistic', size); + mountBench('in-view — one controller per element', 'inView', size); + mountBench('responsive option — breakpoint cascade per mount', 'responsive', size); +}); + +describe.each(SIZES)(`mount %i flat components, 1 vs ${BATCHES} insertions`, (size) => { + const options = samplesFor(size); + let pool: HTMLElement[][] = []; + let index = 0; + + mountBench('1 insertion', 'flat', size); + + bench( + `${BATCHES} insertions`, + async () => { + for (const chunk of pool[index]) { + document.body.append(chunk); + // Settling per chunk is what makes them separate observer deliveries. + await whenDOMSettled(); + } + index += 1; + }, + { + ...options, + async setup() { + await clearDocument(); + pool = Array.from({ length: options.iterations + 1 }, () => + buildPool(scenarios.flat(Math.ceil(size / BATCHES)), BATCHES), + ); + index = 0; + }, + teardown() { + document.body.replaceChildren(); + pool = []; + }, + }, + ); +}); + +describe.each(SIZES)('destroy %i flat components, one removal', (size) => { + const options = samplesFor(size); + let pool: HTMLElement[] = []; + let index = 0; + + bench( + 'flat', + async () => { + pool[index].remove(); + index += 1; + await whenDOMSettled(); + }, + { + ...options, + async setup() { + await clearDocument(); + pool = buildPool(scenarios.flat(size), options.iterations + 1); + for (const host of pool) { + document.body.append(host); + } + await whenDOMSettled(); + index = 0; + }, + teardown() { + document.body.replaceChildren(); + pool = []; + }, + }, + ); +}); diff --git a/packages/v4/src/mount-at-scale.fixtures.ts b/packages/v4/src/mount-at-scale.fixtures.ts new file mode 100644 index 000000000..a7f0a413a --- /dev/null +++ b/packages/v4/src/mount-at-scale.fixtures.ts @@ -0,0 +1,145 @@ +/** + * Component shapes and markup for the at-scale mounting measurements. + * + * The throughput benchmarks and the long-task guard specs must describe the + * same page, or a guard would defend a shape nobody measured. They therefore + * share these fixtures instead of restating them. + */ +import { Base, type BaseConfig, type RefEvent } from './Base.js'; +import { whenDOMSettled } from './dom-mutations.js'; +import { registerComponents } from './registry.js'; + +/** Nothing declared: the floor of what mounting one component costs. */ +class ScaleFlat extends Base { + static config: BaseConfig = { name: 'ScaleFlat' }; +} + +/** + * A four-deep chain. Each level declares the next so the whole family + * registers, and no level declares a handler: the comparison against + * `ScaleFlat` is about tree shape alone. + */ +class ScaleDepth4 extends Base { + static config: BaseConfig = { name: 'ScaleDepth4' }; +} + +class ScaleDepth3 extends Base { + static config: BaseConfig = { name: 'ScaleDepth3', components: { ScaleDepth4 } }; +} + +class ScaleDepth2 extends Base { + static config: BaseConfig = { name: 'ScaleDepth2', components: { ScaleDepth3 } }; +} + +class ScaleDepth1 extends Base { + static config: BaseConfig = { name: 'ScaleDepth1', components: { ScaleDepth2 } }; +} + +/** Five refs, three declared options and four handlers — an ordinary component. */ +class ScaleRealistic extends Base<{ + $refs: { trigger: HTMLElement; panel: HTMLElement; items: HTMLElement[]; label: HTMLElement }; + $options: { columns: number; label: string; open: boolean }; +}> { + static config: BaseConfig = { + name: 'ScaleRealistic', + refs: ['trigger', 'panel', 'items[]', 'label', 'close'], + options: { columns: Number, label: String, open: Boolean }, + }; + + count = 0; + + mounted(): void { + // Real components read their options and touch their refs on mount. + this.count = this.$options.columns + this.$refs.items.length; + this.$refs.panel.hidden = !this.$options.open; + } + + onTriggerClick(): void { + this.count += 1; + } + + onItemsPointerdown({ index }: RefEvent): void { + this.count += index; + } + + onLabelMouseover(): void { + this.count += 1; + } + + onCloseClick(): void { + this.count -= 1; + } +} + +/** One responsive option, so every mount walks the breakpoint cascade. */ +class ScaleResponsive extends Base<{ $options: { columns: number } }> { + static config: BaseConfig = { + name: 'ScaleResponsive', + options: { columns: Number }, + }; + + count = 0; + + mounted(): void { + this.count = this.$options.columns; + } +} + +/** Register every at-scale fixture. Idempotent, like `registerComponent()`. */ +export function registerScaleComponents(): void { + registerComponents(ScaleFlat, ScaleDepth1, ScaleRealistic, ScaleResponsive); +} + +/** How many `data-component` elements each scenario's markup holds. */ +export const NESTING_DEPTH = 4; + +const REALISTIC_MARKUP = `
+ +
+ + +
`; + +/** + * Markup builders, keyed by scenario. Each returns exactly `size` + * `data-component` elements, so results divide into a per-component cost that + * is comparable across scenarios. + */ +export const scenarios = { + /** + * Nothing registered under this name: the mutation observer still delivers + * the records and the registry still walks the subtree, so this is the + * discovery floor every other scenario is built on. + */ + control: (size: number) => `
`.repeat(size), + flat: (size: number) => `
`.repeat(size), + nested: (size: number) => + `
`.repeat( + Math.ceil(size / NESTING_DEPTH), + ), + realistic: (size: number) => REALISTIC_MARKUP.repeat(size), + inView: (size: number) => + `
`.repeat(size), + responsive: (size: number) => + `
`.repeat( + size, + ), +} satisfies Record string>; + +export type ScenarioName = keyof typeof scenarios; + +/** + * Clone-ready trees, built before the timer starts. Parsing markup costs more + * than mounting it, so no measurement may include `innerHTML`. + */ +export function buildPool(html: string, count: number): HTMLElement[] { + const template = document.createElement('div'); + template.innerHTML = html; + return Array.from({ length: count }, () => template.cloneNode(true) as HTMLElement); +} + +/** Empty the document and let the registry finish tearing everything down. */ +export async function clearDocument(): Promise { + document.body.replaceChildren(); + await whenDOMSettled(); +} diff --git a/packages/v4/src/mount-at-scale.spec.ts b/packages/v4/src/mount-at-scale.spec.ts new file mode 100644 index 000000000..f366f0683 --- /dev/null +++ b/packages/v4/src/mount-at-scale.spec.ts @@ -0,0 +1,228 @@ +/** + * Blocking guards for mounting at page scale. + * + * `mount-at-scale.bench.ts` reports a number; this file fails the build. + * + * What is guarded, and why these shapes. Eager mounting is already + * time-sliced: `applyMountStrategy()` posts one background task per element + * and the scheduler's 5 ms budget bounds each turn. What is *not* sliced is + * the observer batch itself — `processMutations()` scans an inserted subtree + * and tears down a removed one in one synchronous pass, so the blocking cost + * of a page grows with its DOM node count, not with how heavy its components + * are. That pass is what these guards defend. + * + * Measured in this Chromium, best of three, on a developer machine: + * + * | insertion | settle | longest task | + * | ------------------ | ------ | ------------ | + * | 10 000 flat | 178 ms | none | + * | 12 000 flat | 199 ms | 51 ms | + * | 2 000 realistic | 181 ms | none | + * | 4 000 realistic | 363 ms | 59 ms | + * | remove 10 000 flat | 40 ms | none | + * | remove 15 000 flat | 61 ms | 53 ms | + * + * So the long-task cliff sits near 12 000 inserted DOM nodes and near 13 000 + * removed ones. The guards below stay well under it rather than asserting the + * cliff itself: a faster machine would never reach it, a slower one would + * reach it early, and an assertion that depends on the runner's speed is a + * flake. The one guard that does not depend on machine speed — a ratio — is + * where the tight threshold goes. + * + * Every threshold tolerates noise, because a CI runner is slower and noisier + * than this machine and a guard that flakes gets deleted by the next person, + * which is worse than no guard. Each signal tolerates it in the way that + * suits it: wall time takes the best of several repeats, a long task takes + * the second worst. `mountCost()` explains why they differ. + */ +import { describe, expect, it } from 'vitest'; +import { whenDOMSettled } from './dom-mutations.js'; +import { + buildPool, + clearDocument, + registerScaleComponents, + scenarios, +} from './mount-at-scale.fixtures.js'; + +registerScaleComponents(); + +/** + * Repeats per measurement, so one scheduling hiccup cannot fail the build. + * Must stay at two or more: the long-task reduction below discards the worst + * repeat and reads the one behind it. + */ +const REPEATS = 4; + +/** + * A catastrophe ceiling on wall time, not a regression detector: a shared + * runner's wall clock is too soft for that, and the bench-diff job owns it. + * Roughly 8× the local best, past any plausible runner handicap and still + * short of the order-of-magnitude blow-up this is here to catch. + */ +const SETTLE_CEILING = 400; + +interface MountCost { + /** Wall time from the DOM write to `whenDOMSettled()`. */ + duration: number; + /** The longest `longtask` entry the write produced, or 0 for none. */ + longestTask: number; +} + +/** Watch for long tasks while `work` runs, and report the longest one. */ +async function watchLongTasks(work: () => Promise): Promise { + const durations: number[] = []; + const observer = new PerformanceObserver((list) => { + for (const entry of list.getEntries()) { + durations.push(entry.duration); + } + }); + observer.observe({ entryTypes: ['longtask'] }); + + const start = performance.now(); + await work(); + const duration = performance.now() - start; + + // Long-task entries are queued and delivered on a later task. + await new Promise((resolve) => { + setTimeout(resolve, 0); + }); + for (const entry of observer.takeRecords()) { + durations.push(entry.duration); + } + observer.disconnect(); + + return { duration, longestTask: Math.max(0, ...durations) }; +} + +/** + * Insert one tree and wait for the registry to finish with it. The tree is + * built before the clock starts, so parsing markup is never counted as + * framework work. + */ +async function mountOnce(html: string): Promise { + const [host] = buildPool(html, 1); + const cost = await watchLongTasks(async () => { + document.body.append(host); + await whenDOMSettled(); + }); + await clearDocument(); + return cost; +} + +/** + * Run the measurement `REPEATS` times, after one unmeasured run warms the + * paths, and reduce the repeats — differently for each signal, because they + * are different kinds of number. + * + * **Wall time takes the minimum.** It is a cost, and the least interrupted + * run is the closest any of them gets to the cost of the work itself. Every + * other repeat measured that same work plus something else. + * + * **A long task takes the second worst.** It is a defect, not a cost: any + * occurrence is the finding, so the minimum is exactly wrong — one lucky + * repeat out of four would hide a page that blocks the main thread three + * times out of four. The maximum is wrong in the other direction, because a + * `longtask` entry attributes *any* 50 ms task in the frame, including a GC + * pause or a runner hiccup that has nothing to do with mounting, and one of + * those would fail the build. Discarding only the single worst repeat keeps + * the signal — a page that really blocks does so on every repeat — while + * forgiving one bad sample. One blocked run is noise; two is the finding. + * + * Tolerating noise in *how many repeats blocked*, rather than by allowing + * some blocking, is what keeps the assertion honest: it still reads "no long + * task", never "no long task longer than N". + */ +async function mountCost(html: string): Promise { + await mountOnce(html); + const runs: MountCost[] = []; + for (let index = 0; index < REPEATS; index += 1) { + runs.push(await mountOnce(html)); + } + const blocked = runs.map((run) => run.longestTask).sort((a, b) => a - b); + return { + duration: Math.min(...runs.map((run) => run.duration)), + // The second worst, so the single worst repeat is discarded. + longestTask: blocked[blocked.length - 2], + }; +} + +/** + * Measure two pages against each other and return `b / a`, the ratio of their + * best wall times. + * + * The two sides are measured **alternately**, not one after the other. A + * ratio of two separately-taken measurements absorbs every bit of drift + * between the two blocks: a CI runner that slows down between them reports it + * as a difference between the pages. Nesting read 1.91 that way on + * `ubuntu-latest` while the benchmark suite, which interleaves, put the same + * pair at 1.01 on the same runner class. Alternating puts any drift on both + * sides, which is the same reason `bench-diff` alternates base and head. + */ +async function costRatio(a: string, b: string): Promise { + await mountOnce(a); + await mountOnce(b); + const first: number[] = []; + const second: number[] = []; + for (let index = 0; index < REPEATS; index += 1) { + first.push((await mountOnce(a)).duration); + second.push((await mountOnce(b)).duration); + } + return Math.min(...second) / Math.min(...first); +} + +describe('mounting at page scale', () => { + // 500 realistic components is a heavy real page. Its un-chunked pass costs + // ~7 ms here, so reaching 50 ms needs a machine seven times slower. + it('mounts 500 realistic components without blocking', async () => { + const cost = await mountCost(scenarios.realistic(500)); + expect(cost.longestTask).toBe(0); + expect(cost.duration).toBeLessThan(SETTLE_CEILING); + }, 60000); + + // 2 000 flat components walk about as many nodes, with the same headroom. + it('mounts 2 000 flat components without blocking', async () => { + const cost = await mountCost(scenarios.flat(2000)); + expect(cost.longestTask).toBe(0); + expect(cost.duration).toBeLessThan(SETTLE_CEILING); + }, 60000); + + /** + * The failure this really guards against: a change that makes the scan + * re-query the document per element turns mounting quadratic, and a + * quadratic page only reveals itself at scale. + * + * This guard cannot use two equal sizes — differing sizes are the whole + * question — so it keeps a bias the nesting guard below does not have. + * Taking the best of several repeats favours the shorter side, because a + * 14 ms run slips between interruptions that a 80 ms run absorbs, and + * contention widens the gap: measured at 1.11 to 1.20 per component when + * quiet, and up to 1.57 with all eight cores saturated. An earlier version + * compared 500 against 4 000 and read 2.22 on a CI runner. + * + * So the threshold is placed between the noise and the signal rather than + * just above the noise: five times the components read about 1.6 per + * component at worst when linear, and would read 5 if quadratic. Three + * sits between the two, with room on both sides. + */ + it('costs the same per component at 5 000 as at 1 000', async () => { + const ratio = await costRatio(scenarios.flat(1000), scenarios.flat(5000)); + expect(ratio / 5).toBeLessThan(3); + }, 60000); + + /** + * v4 claims nesting costs nothing, because no parent orchestrates its + * children's mounting. Reintroducing that orchestration means a second + * pass that finds and mounts children from each parent. + * + * Equal sizes on both sides — same component count, same node count, only + * the shape differs — so this one has no short-side bias and can be tight. + * Measured between 0.898 and 1.024 over ten runs, quiet and with every + * core saturated. 5 000 rather than 2 000 so each side is long enough that + * one scheduling gap is a small fraction of it; at 2 000, sequentially, + * this read 1.91 on a CI runner. + */ + it('mounts a four-deep tree for what a flat one costs', async () => { + const ratio = await costRatio(scenarios.flat(5000), scenarios.nested(5000)); + expect(ratio).toBeLessThan(1.8); + }, 60000); +}); diff --git a/packages/v4/vitest.bench.config.js b/packages/v4/vitest.bench.config.js index a1615c080..f2e311977 100644 --- a/packages/v4/vitest.bench.config.js +++ b/packages/v4/vitest.bench.config.js @@ -4,12 +4,27 @@ import { decorators } from './vite-plugin-decorators.js'; /** * Benchmarks answering design questions, run in a real browser because the - * APIs they measure — `matchMedia`, `ResizeObserver`, `AbortSignal` — have - * no meaningful cost in an emulated DOM. Deliberately separate from the - * CodSpeed suite in `packages/tests`, which guards against regressions. + * APIs they measure — `matchMedia`, `ResizeObserver`, `AbortSignal`, the + * mutation observer driving the registry — have no meaningful cost in an + * emulated DOM. + * + * CodSpeed's simulation mode cannot instrument these: its plugin forces + * `pool: "forks"` and measures the Node process, while a browser-mode + * benchmark body runs in Chromium over CDP. Regressions are caught instead by + * `.github/actions/bench-diff`, which runs this suite for the base and the + * head commit on one runner and comments the difference. + * + * `V4_BENCH_SIZES` narrows the at-scale suite — CI compares a subset so the + * job stays inside a few minutes. */ +const sizes = (process.env.V4_BENCH_SIZES ?? '100,1000,5000') + .split(',') + .map((size) => Number(size.trim())) + .filter((size) => Number.isFinite(size) && size > 0); + export default defineConfig({ plugins: [decorators()], + define: { __BENCH_SIZES__: JSON.stringify(sizes) }, test: { include: ['src/**/*.bench.ts'], benchmark: { include: ['src/**/*.bench.ts'] },