Skip to content

bench(gate): widen the url-parse-ipv4 cap to 20.0 for bimodal CI runners - #498

Merged
x-ror merged 2 commits into
masterfrom
bench/url-parse-ipv4-cap
Nov 17, 2026
Merged

x-ror merged 2 commits into
masterfrom
bench/url-parse-ipv4-cap

Conversation

@x-ror

@x-ror x-ror commented Nov 17, 2026

Copy link
Copy Markdown
Owner

Widens the url-parse-ipv4 bench-gate cap from 14.7x to 20.0x, at the owner's request. bench/thresholds.json is on the integrity-protected list, so this change was made with the owner's explicit approval. The PR also corrects the stale "caps proven to fire" claim in bench/micro/url.js. No runtime code changes.

Why

The cap breached on master after #496 (16.06x) and on #497 (15.01x), even though both PRs touched only scripts/ and the mutation manifest.

Across all 28 ubuntu-latest CI runs with a bench-gate row (runs 43938992938 through 44067776896, median of 3 launches per side, Node v26.13.0), the ratio ranged from 10.37x to 16.06x and breached 14.7x three times. Those three breaches are the runs with the lowest Lava time of all 28 (77-83 ms). They breach only because Node was faster still on those runners.

Node's time sits in two clusters:

  • ~12 ms, giving 10.4-11.3x;
  • ~5-7.5 ms, giving a mean of 13.59x with sd 1.06.

So the ratio follows the runner, not Lava. The full figures, with commit references, are in the "//url" note.

The value

The value is 20.0, not the #412 convention of ~1.4x the max (22.5). That convention was set from only 3 samples. With 28 samples, 20.0 is ~1.25x the max and more than 5 sd above the fast-cluster mean, it would have produced 0 false positives across the 28, and it keeps more detection.

regression it still catches
slow-Node runners ~1.8x
fast-Node runners ~1.25-1.7x

Confidence that 20.0 stays green is medium. A false trip needs Node at ~4 ms, which is about 19% faster than the fastest of the 28 runs.

Also

bench/micro/url.js:33 claimed the url caps were "PROVEN TO FIRE". The hand slowdown cited there (ipv4 15.35x) breaches none of the current caps. The comment now says that proof is stale.

Follow-up, not done here: the bench runs 50k iterations, so Node finishes in ~5 ms and launch noise dominates. More iterations might allow a tighter cap. This is untested.

Gates (/pr-gate, verdict SHIP: P0 0, P1 0)

Gate Result
make check / build PASS @ 87688b0
make test / test-lava / test-odin-serial PASS @ 87688b0
make bench-gate PASS @ 87688b0 (11.84x vs 22.5), PASS @ d3fce98 (11.00x vs 20.0); single local runs, not the CI runner population
make check-js PASS @ d3fce98

Reviewers:

  • perf-memory-auditor: neutral. It re-read all 28 CI rows. Its two P2s are fixed in d3fce98: the tighter cap, and two inaccurate sentences in the note.
  • oracle-test-auditor: covered. No recorded mutation depends on a cap value. Its P2, the stale fire-proof comment, is fixed in d3fce98.
  • regression-hunter: clean. The cap is still read correctly by bench/run.mjs, and nothing pins 14.7.
  • quality-docs-auditor: clean. The note meets the CLAUDE.md §1 measurement rule.

Fix-round edits (d3fce98) were made by the conductor directly; they are data and comment text only.

🤖 Generated with Claude Code

x-ror and others added 2 commits November 17, 2026 12:58
Across 28 ubuntu-latest CI bench-gate runs (median of 3 launches per
side, Node v26.13.0) url-parse-ipv4 measured 10.37x-16.06x and breached
the 14.7x cap 3 times with no runtime change: Node's time on that bench
is either ~12 ms or ~5-7 ms depending on the runner, Lava's moves less.
Cap set at ~1.4x the max observed; measurements and the detection cost
are recorded in thresholds.json.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…' fire-proof stale

perf-memory-auditor re-read all 28 CI rows: 20.0 (~1.25x the max, >5 sd
above the fast-runner mean) has no false positive among them and keeps
more detection than 22.5. The note no longer claims no runtime change
between breaches (5228c42 moved the parser) and gives the evidence that
the breaches track the runner instead. bench/micro/url.js no longer
claims the caps are proven to fire: the old hand slowdown breaches none
of the current caps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@x-ror
x-ror merged commit 6622a78 into master Nov 17, 2026
7 of 8 checks passed
@x-ror
x-ror deleted the bench/url-parse-ipv4-cap branch November 17, 2026 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant