perf(jit): V8/Bun roadmap and P0-P4 implementation

benchmarks/README.md

@@ -6,19 +6,21 @@ latencies, process-throughput windows, phase timings, counters, memory/code
 sizes, checksums, build fingerprints, and host provenance.
 
 Current samples identify their timing protocol as
-`protocol.name = "shared-js-fixed-warmup-v2"`. The JSON envelope remains
-`jit-benchmark-v1`, but these timings are **not equivalent to historical v1
-measurements**. The legacy `jit-bench-report` gate reporter explicitly rejects
-v2 samples as an incompatible timing protocol. Do not pass `--report` when
-collecting v2 evidence or interpret that rejection as a runtime test failure.
+`protocol.name = "shared-js-multibatch-v3"`. The JSON envelope remains
+`jit-benchmark-v1`, but these timings are **not equivalent to historical v1 or
+`shared-js-fixed-warmup-v2` measurements** (see
+[Multi-batch timing](#multi-batch-timing-and-readiness-diagnostics)). The legacy
+`jit-bench-report` gate reporter explicitly rejects v2 and v3 samples as an
+incompatible timing protocol. Do not pass `--report` when collecting v2/v3
+evidence or interpret that rejection as a runtime test failure.
 
 Build and collect the complete five-mode raw matrix:
 
 ```sh
 cargo build --release --manifest-path benchmarks/Cargo.toml --bins
 ./target/release/jit-bench compare \
   --modes interpreter,tier1,tier2,automatic,bun \
-  --output benchmarks/results/comparison-v2.json
+  --output benchmarks/results/comparison-v3.json
 ```
 
 Bun uses its default engine configuration: the runner no longer adds `--smol`.
@@ -28,6 +30,13 @@ counters in comparison samples are JSON `null` (N/A). Raw worker output keeps
 zero placeholders in the legacy cumulative counter fields; these are not Bun
 execution-tier measurements.
 
+Bun runs the shared wrapper from a temporary `.mjs` file, not with `bun -e`;
+the file is deleted after the run and its SHA-256 is recorded in
+`protocol.bun_wrapper_sha256` (`protocol.bun_launch = "file"`). The wrapper does
+not embed the script path, so its hash only changes with the driver, the warmup
+count, or `JIT_BENCH_TIMED_BATCHES`. `JIT_BENCH_BUN_LAUNCH=eval` reproduces the
+historical `bun -e` launch as a **control only**; `paired.py` rejects it.
+
 For an external paired baseline/candidate collector, each command below emits
 one worker JSON object. Build both runtime revisions with the exact same
 benchmark harness and scripts, and retain immutable binaries with their hashes:
@@ -43,46 +52,75 @@ External collectors must check checksums, protocol identity, script/driver
 hashes, sampling counts, and native-counter consistency themselves, and record
 engine identities and host/build provenance.
 
-## Fixed-warmup timing and readiness diagnostics
+## Multi-batch timing and readiness diagnostics
 
 Every engine executes one first call, then exactly 64 warmup batches of ten
-calls, then one measured batch of ten calls. The shared `batch-driver.js` uses
-identical workload arguments and stores every result. Promise results complete
-sequentially before the batch ends. Primitive checksums are generated after
-timing, including during warmup, and combine all ten results. The
-`protocol.warmup_batch_ns` array records the 64 consecutive warmup timings.
-
-`elapsed_ns` is the **fixed-warmup batch latency**, not a claim that compilation
-has settled. `protocol.fixed_metrics_before` and `fixed_metrics_after` surround
-that measured batch and exclude checksum execution. Derive native, Tier 2,
-deopt, fallback, compilation, and installation deltas from these snapshots.
+calls, then **K consecutive timed batches** of ten calls (`K = 16` by default;
+`JIT_BENCH_TIMED_BATCHES` overrides it, and `paired.py` passes the same K to
+every engine and variant). The shared `batch-driver.js` uses identical workload
+arguments and stores every result. Promise results complete sequentially before
+the batch ends. Primitive checksums are generated after timing, including during
+warmup, and combine all ten results; every timed batch must reproduce the first
+timed batch's checksum. QuickJS polls the JIT after every warmup and timed
+batch, outside the timed region, exactly as during warmup.
+
+- `elapsed_ns` is the **upper median of the K timed batches** (the harness-wide
+  `ceil((n - 1) / 2)` quantile rule). It is a steady-batch latency after a
+  fixed warmup, not a claim that compilation has settled.
+- `protocol.timed_batch_ns` stores all K raw timings in execution order;
+  `protocol.warmup_batch_ns` stores the 64 warmup timings.
+- `protocol.fixed_batch_ns` is the first timed batch, i.e. the single batch that
+  `shared-js-fixed-warmup-v2` reported as `elapsed_ns`. It is a cold/jitter
+  **diagnostic only**.
+- `protocol.outlier_batches` counts timed batches slower than
+  `protocol.outlier_ratio` (5) times the median. Outliers stay in
+  `timed_batch_ns` and are never silently dropped; `summarize_paired.py` reports
+  their total and the number of affected processes.
+
+Why v3 exists: under `bun -e`, Bun's batch at index 64 (the only batch v2 timed)
+deterministically hit a one-off ~0.1-0.2 ms JSC stall, inflating Bun latency by
+1.2x-32x on two-argument kernels. Running the same wrapper from a file removes
+the stall, and the median of K batches no longer depends on one batch index.
+For example, a noisy smoke of `numeric` on one host measured Bun at about
+0.005 ms per ten calls with the file launch (fixed batch also about 0.005 ms),
+while the `-e` control still showed a 0.13-0.19 ms fixed batch with 2-3 flagged
+outliers and a median of about 0.005 ms. v2 and v3 numbers must not be mixed in
+one comparison; `protocol_check.py` rejects mixed evidence.
+
+`protocol.fixed_metrics_before` and `fixed_metrics_after` surround the first
+timed batch and exclude checksum execution, as in v2.
+`protocol.timed_metrics_before` and `timed_metrics_after` surround the whole
+timed window, **including** the untimed checksum and JIT-poll steps between
+batches. Derive native, Tier 2, deopt, fallback, compilation, and installation
+deltas from these snapshots; `paired.py` stores them as `fixed_deltas` and
+`timed_deltas` and uses the timed window for `compilation_quiet_samples`.
 Counters aggregate the shared driver and workload; they do not identify the
 native tier of an individual workload function. Bun has no equivalent internal
 metrics and reports these snapshots as `null`; interpreter workers have no
 attached JIT and also leave them `null`.
 
-QuickJS still times one outer Rust `Function::call` and, for asynchronous work,
-its Promise completion bridge. Bun times the corresponding JS call and await.
-Both timings include driver array/closure work. The protocol removes the old
-per-workload Rust lookup/checksum/poll loop from the measured region, but does
-not eliminate the outer host-call difference. The driver is part of the
-measured workload; these are not isolated function-body timings.
+QuickJS still times one outer Rust `Function::call` per batch and, for
+asynchronous work, its Promise completion bridge. Bun times the corresponding JS
+call and await. Both timings include driver array/closure work. The protocol
+removes the old per-workload Rust lookup/checksum/poll loop from the measured
+region, but does not eliminate the outer host-call difference. The driver is
+part of the measured workload; these are not isolated function-body timings.
 
-After collecting the fixed-window sample, QuickJS runs the existing bounded
+After collecting the timed batches, QuickJS runs the existing bounded
 JIT readiness and settling checks, then another shared batch. Its timing is
 stored in `protocol.readiness_diagnostic_ns` and `phases.steady_state_ns`.
 Bun has no separately conditioned readiness diagnostic. Do not divide Bun's
-fixed-window latency by QuickJS's post-readiness diagnostic and call that a
+multi-batch latency by QuickJS's post-readiness diagnostic and call that a
 matched steady-state comparison. Legacy cumulative counters include first
-execution, warmup, the fixed sample, and subsequent diagnostics; they cannot
-prove native execution or settled compilation during the fixed window.
+execution, warmup, the timed batches, and subsequent diagnostics; they cannot
+prove native execution or settled compilation during the timed window.
 
 Forced Tier 2 probes can declare `globalThis.tier2ReadyInstalls` to require a
 known publication count in the readiness diagnostic. The direct-call probe
 requires four: two baseline and two optimizing artifacts. An unrelated
 blacklisted function cannot bypass that requirement. Explicit forced-Tier2
 requirements reject unsettled diagnostic startup or new compilation during
-the diagnostic batch. Those checks happen after the fixed-window sample and
+the diagnostic batch. Those checks happen after the timed batches and
 do not make its warmup policy variable. Unsupported probes retain bounded
 fallback/readiness behavior.
 
@@ -161,10 +199,12 @@ fallback-only result as proof that a JIT tier supports the feature.
 | `strings-regexp` | String construction, slicing, RegExp capture and replacement |
 | `arrays-typed` | Packed-array growth/traversal and Int32Array/Float64Array traffic |
 | `objects-polymorphic` | Allocation, property reads/writes, and four stable shapes |
+| `property-polymorphic` | Allocation-free reads at two sites that each observe four stable shapes (one prototype-less); the bounded polymorphic inline cache |
 | `calls-recursion-closures` | Four-deep calls, bounded recursion, and mutable closure capture |
 | `json-codec` | Repeated nested JSON encoding and decoding |
 | `map-set-bigint` | Map and Set mutation/iteration plus bounded BigInt arithmetic |
 | `exceptions-promises-async` | Throw/catch, Promise jobs, async functions, and continuations |
+| `methods-dynamic` | Prototype methods using `this`, `typeof`/`in`/`instanceof` dispatch, object destructuring, `**`, and compound element updates |
 
 Select the matrix with `JIT_BENCH_WORKLOADS` for smoke or publishable runs; the
 sample/warmup/window rules above remain unchanged.

benchmarks/model.rs

@@ -202,9 +202,43 @@ mod tests {
         assert_eq!(p99, 100);
         assert!(ci[0] <= median && median <= ci[1]);
     }
+
+    #[test]
+    fn timed_batch_summary_uses_the_median_and_counts_but_keeps_spikes() {
+        // A single one-off stall (the old fixed-batch artifact) no longer
+        // determines the reported latency, but it stays visible.
+        let mut batches = vec![5_000u64; 16];
+        batches[0] = 187_000;
+        let (median, outliers) = timed_batch_summary(&batches);
+        assert_eq!(median, 5_000);
+        assert_eq!(outliers, 1);
+        // Exactly 5x the median is not an outlier; above it is.
+        batches[0] = 25_000;
+        assert_eq!(timed_batch_summary(&batches).1, 0);
+        batches[0] = 25_001;
+        assert_eq!(timed_batch_summary(&batches).1, 1);
+        // Even K uses the harness-wide upper median.
+        assert_eq!(timed_batch_summary(&[1, 2, 3, 4]), (3, 0));
+        assert_eq!(timed_batch_summary(&[]), (0, 0));
+    }
 }
 
+/// Legacy timing protocol: 64 warmup batches, then one timed batch.
+#[allow(dead_code)]
+pub const PROTOCOL_FIXED_WARMUP_V2: &str = "shared-js-fixed-warmup-v2";
+/// Current timing protocol: 64 warmup batches, then K consecutive timed
+/// batches whose median becomes `elapsed_ns`.
+#[allow(dead_code)]
+pub const PROTOCOL_MULTIBATCH_V3: &str = "shared-js-multibatch-v3";
+/// A timed batch slower than this multiple of the timed-batch median is
+/// counted as an outlier. Outliers are never discarded.
+#[allow(dead_code)]
+pub const OUTLIER_RATIO: u64 = 5;
+
 /// Protocol identity separates these fixed-warmup timings from historical v1 timing.
+///
+/// Fields after `readiness_diagnostic_ns` were added by
+/// `shared-js-multibatch-v3`; they default to empty in v2 evidence.
 #[derive(Clone, Debug, Default, Deserialize, Serialize)]
 #[serde(deny_unknown_fields)]
 pub struct ProtocolEvidence {
@@ -217,4 +251,45 @@ pub struct ProtocolEvidence {
     pub fixed_metrics_before: Option<std::collections::BTreeMap<String, u64>>,
     pub fixed_metrics_after: Option<std::collections::BTreeMap<String, u64>>,
     pub readiness_diagnostic_ns: Option<u64>,
+    /// Number of consecutive timed batches after the warmups (v3).
+    #[serde(default)]
+    pub timed_batches: u32,
+    /// Raw latency of every timed batch, in execution order (v3).
+    #[serde(default)]
+    pub timed_batch_ns: Vec<u64>,
+    /// Diagnostic only: the first timed batch, i.e. the single batch that
+    /// `shared-js-fixed-warmup-v2` reported as `elapsed_ns` (v3).
+    #[serde(default)]
+    pub fixed_batch_ns: Option<u64>,
+    /// Outlier rule: a timed batch above `outlier_ratio` times the median (v3).
+    #[serde(default)]
+    pub outlier_ratio: u64,
+    /// Number of timed batches that exceed the outlier rule (v3).
+    #[serde(default)]
+    pub outlier_batches: u32,
+    /// JIT metric snapshots around the whole timed window. They include the
+    /// untimed checksum and poll steps between timed batches (v3).
+    #[serde(default)]
+    pub timed_metrics_before: Option<std::collections::BTreeMap<String, u64>>,
+    #[serde(default)]
+    pub timed_metrics_after: Option<std::collections::BTreeMap<String, u64>>,
+    /// How Bun loaded its wrapper: `file` (v3 default) or `eval` (control) (v3).
+    #[serde(default)]
+    pub bun_launch: Option<String>,
+    /// SHA-256 of the exact Bun wrapper module that was executed (v3).
+    #[serde(default)]
+    pub bun_wrapper_sha256: Option<String>,
+}
+
+/// Summary of K timed batches: (upper median, outlier count).
+///
+/// The median uses the same upper-quantile rule as the rest of the harness.
+#[allow(dead_code)]
+pub fn timed_batch_summary(batches: &[u64]) -> (u64, u32) {
+    let mut sorted = batches.to_vec();
+    sorted.sort_unstable();
+    let median = quantile(&sorted, 0.5);
+    let threshold = median.saturating_mul(OUTLIER_RATIO);
+    let outliers = batches.iter().filter(|&&ns| ns > threshold).count();
+    (median, u32::try_from(outliers).unwrap_or(u32::MAX))
 }

benchmarks/paired.py

@@ -13,6 +13,7 @@
 from pathlib import Path
 import subprocess
 import time
+from protocol_check import PROTOCOL_V3, timed_batches_from_env, validate_protocol
 
 p = argparse.ArgumentParser()
 p.add_argument('--binaries', required=True, help='JSON map from variant to binary')
@@ -25,6 +26,10 @@
 args = p.parse_args()
 if args.pairs < 1:
     p.error('--pairs must be positive')
+try:
+    timed_batches = timed_batches_from_env()
+except ValueError as error:
+    p.error(str(error))
 if args.cpu is not None and platform.system() != 'Linux':
     p.error('--cpu requires Linux taskset')
 import fcntl
@@ -53,8 +58,8 @@
     bun_version=subprocess.check_output([str(bun), '--version'],text=True).strip(),
     bun_sha256=hashlib.sha256(bun.read_bytes()).hexdigest(), bun_flags=[],
     rustc=subprocess.check_output(['rustc','-Vv'],text=True),
-    protocol='shared-js-fixed-warmup-v2', throughput_windows=0,
-    note='Latency-only paired candidate diagnostic. Fixed 10-call window; counters include shared driver. Readiness diagnostic is separately conditioned.')
+    protocol=PROTOCOL_V3, timed_batches=timed_batches, throughput_windows=0,
+    note='Latency-only paired candidate diagnostic. elapsed_ns is the median of the timed 10-call batches that follow 64 warmups; the first timed batch is the legacy fixed-batch diagnostic. Counters include shared driver. Readiness diagnostic is separately conditioned.')
 (out / 'metadata.json').write_text(json.dumps(metadata,indent=2)+'\n')
 for workload in args.workloads.split(','):
     script = Path(args.scripts) / (workload + '.js')
@@ -68,20 +73,18 @@
             if target.exists():
                 raise RuntimeError(f'Refusing to overwrite existing evidence: {target}')
             command = [bins[variant], 'worker', '--mode', mode, '--script', str(script)]
+            # Every engine and variant receives the same K explicitly.
+            worker_env = dict(os.environ, JIT_BENCH_TIMED_BATCHES=str(timed_batches))
             if args.cpu is not None:
                 command = ['taskset', '-c', args.cpu] + command
             started = time.monotonic()
-            result = subprocess.run(command, capture_output=True, text=True, timeout=180)
+            result = subprocess.run(command, capture_output=True, text=True, timeout=180, env=worker_env)
             if result.returncode:
                 (out / 'failure.json').write_text(json.dumps(dict(command=command, stdout=result.stdout, stderr=result.stderr),indent=2))
                 raise RuntimeError(f'Worker failed: {command}: {result.stderr}')
             value = json.loads(result.stdout)
             protocol = value['protocol']
-            assert protocol['name'] == metadata['protocol']
-            assert protocol['script_sha256'] == metadata['scripts'][workload]
-            assert protocol['warmup_batches'] == len(protocol['warmup_batch_ns']) == 64
-            assert protocol['calls_per_batch'] == 10
-            assert value['elapsed_ns'] > 0
+            validate_protocol(value, metadata, workload, mode)
             if mode != 'bun':
                 assert value['native_exits'] == value['native_entries']
             before, after = protocol['fixed_metrics_before'], protocol['fixed_metrics_after']
@@ -91,12 +94,20 @@
                 assert all(v >= 0 for v in value['fixed_deltas'].values())
                 assert value['fixed_deltas']['native_entries'] == value['fixed_deltas']['native_exits']
                 value['fixed_compilation_quiet'] = (after['pending_worker_jobs'] == before['pending_worker_jobs'] == 0 and after['pending_snapshot_bytes'] == before['pending_snapshot_bytes'] == 0 and value['fixed_deltas']['installed'] == 0 and value['fixed_deltas']['compile_ns'] == 0)
+            before, after = protocol.get('timed_metrics_before'), protocol.get('timed_metrics_after')
+            if before is not None:
+                # Spans every timed batch plus the untimed checksum/poll steps between them.
+                value['timed_deltas'] = {k:after[k]-before[k] for k in cumulative}
+                assert all(v >= 0 for v in value['timed_deltas'].values())
+                assert value['timed_deltas']['native_entries'] == value['timed_deltas']['native_exits']
+                value['timed_compilation_quiet'] = (after['pending_worker_jobs'] == before['pending_worker_jobs'] == 0 and after['pending_snapshot_bytes'] == before['pending_snapshot_bytes'] == 0 and value['timed_deltas']['installed'] == 0 and value['timed_deltas']['compile_ns'] == 0)
             value.update(variant=variant, requested_mode=mode, pair_index=pair, process_wall_seconds=time.monotonic()-started)
             target.write_text(json.dumps(value,indent=2)+'\n')
         print(f'{workload}: pair {pair} complete',flush=True)
     values = [json.loads(path.read_text()) for path in out.glob(f'{workload}-*.json')]
     assert len({v['checksum'] for v in values}) == 1, workload
     assert len({v['protocol']['driver_sha256'] for v in values}) == 1, workload
+    assert len({v['protocol'].get('bun_wrapper_sha256') for v in values if v['requested_mode'] == 'bun'}) <= 1, workload
     (out / 'metadata.json').write_text(json.dumps(metadata,indent=2)+'\n')
 for variant, binary in bins.items():
     assert hashlib.sha256(Path(binary).read_bytes()).hexdigest() == metadata['binary_sha256'][variant]