From cfd5667488da44a3139ea1dbd4475efb4b481f98 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 16:28:00 +0800 Subject: [PATCH 01/54] bench: add shared-js-multibatch-v3 timing protocol Every engine now times K consecutive batches (default 16, JIT_BENCH_TIMED_BATCHES) after the 64 warmups. elapsed_ns is the upper median of those batches, all raw values are kept in protocol.timed_batch_ns, the old single fixed batch is kept as the fixed_batch_ns diagnostic, and batches slower than 5x the median are counted in outlier_batches (never dropped). Bun now runs the wrapper from a temporary .mjs file, recording its SHA-256, instead of bun -e. Under -e, batch index 64 (the only batch v2 timed) hit a one-off JSC stall that inflated Bun latency by 1.2x-32x on two-argument kernels. JIT_BENCH_BUN_LAUNCH=eval keeps -e as a control. The legacy reporter still rejects protocol-tagged samples and names the protocol. paired.py and summarize_paired.py share protocol_check.py, which validates v3 invariants and refuses mixed v2/v3 evidence. Adds docs/roadmap/p0-bench-protocol.md with the remaining P0 work and a JSC late-tier-up caveat for the file launch. Co-Authored-By: Claude Opus 5.5 (1M context) --- benchmarks/README.md | 94 ++++++++---- benchmarks/model.rs | 75 ++++++++++ benchmarks/paired.py | 27 ++-- benchmarks/protocol_check.py | 67 +++++++++ benchmarks/report.rs | 46 +++++- benchmarks/run.rs | 184 +++++++++++++++++++++--- benchmarks/schema/jit-benchmark-v1.json | 22 ++- benchmarks/summarize_paired.py | 19 ++- docs/roadmap/p0-bench-protocol.md | 75 ++++++++++ 9 files changed, 538 insertions(+), 71 deletions(-) create mode 100644 benchmarks/protocol_check.py create mode 100644 docs/roadmap/p0-bench-protocol.md diff --git a/benchmarks/README.md b/benchmarks/README.md index d6dd3e5fc..2ed224e82 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -6,11 +6,13 @@ latencies, process-throughput windows, phase timings, counters, memory/code sizes, checksums, build fingerprints, and host provenance. Current samples identify their timing protocol as -`protocol.name = "shared-js-fixed-warmup-v2"`. The JSON envelope remains -`jit-benchmark-v1`, but these timings are **not equivalent to historical v1 -measurements**. The legacy `jit-bench-report` gate reporter explicitly rejects -v2 samples as an incompatible timing protocol. Do not pass `--report` when -collecting v2 evidence or interpret that rejection as a runtime test failure. +`protocol.name = "shared-js-multibatch-v3"`. The JSON envelope remains +`jit-benchmark-v1`, but these timings are **not equivalent to historical v1 or +`shared-js-fixed-warmup-v2` measurements** (see +[Multi-batch timing](#multi-batch-timing-and-readiness-diagnostics)). The legacy +`jit-bench-report` gate reporter explicitly rejects v2 and v3 samples as an +incompatible timing protocol. Do not pass `--report` when collecting v2/v3 +evidence or interpret that rejection as a runtime test failure. Build and collect the complete five-mode raw matrix: @@ -18,7 +20,7 @@ Build and collect the complete five-mode raw matrix: cargo build --release --manifest-path benchmarks/Cargo.toml --bins ./target/release/jit-bench compare \ --modes interpreter,tier1,tier2,automatic,bun \ - --output benchmarks/results/comparison-v2.json + --output benchmarks/results/comparison-v3.json ``` Bun uses its default engine configuration: the runner no longer adds `--smol`. @@ -28,6 +30,13 @@ counters in comparison samples are JSON `null` (N/A). Raw worker output keeps zero placeholders in the legacy cumulative counter fields; these are not Bun execution-tier measurements. +Bun runs the shared wrapper from a temporary `.mjs` file, not with `bun -e`; +the file is deleted after the run and its SHA-256 is recorded in +`protocol.bun_wrapper_sha256` (`protocol.bun_launch = "file"`). The wrapper does +not embed the script path, so its hash only changes with the driver, the warmup +count, or `JIT_BENCH_TIMED_BATCHES`. `JIT_BENCH_BUN_LAUNCH=eval` reproduces the +historical `bun -e` launch as a **control only**; `paired.py` rejects it. + For an external paired baseline/candidate collector, each command below emits one worker JSON object. Build both runtime revisions with the exact same benchmark harness and scripts, and retain immutable binaries with their hashes: @@ -43,46 +52,75 @@ External collectors must check checksums, protocol identity, script/driver hashes, sampling counts, and native-counter consistency themselves, and record engine identities and host/build provenance. -## Fixed-warmup timing and readiness diagnostics +## Multi-batch timing and readiness diagnostics Every engine executes one first call, then exactly 64 warmup batches of ten -calls, then one measured batch of ten calls. The shared `batch-driver.js` uses -identical workload arguments and stores every result. Promise results complete -sequentially before the batch ends. Primitive checksums are generated after -timing, including during warmup, and combine all ten results. The -`protocol.warmup_batch_ns` array records the 64 consecutive warmup timings. - -`elapsed_ns` is the **fixed-warmup batch latency**, not a claim that compilation -has settled. `protocol.fixed_metrics_before` and `fixed_metrics_after` surround -that measured batch and exclude checksum execution. Derive native, Tier 2, -deopt, fallback, compilation, and installation deltas from these snapshots. +calls, then **K consecutive timed batches** of ten calls (`K = 16` by default; +`JIT_BENCH_TIMED_BATCHES` overrides it, and `paired.py` passes the same K to +every engine and variant). The shared `batch-driver.js` uses identical workload +arguments and stores every result. Promise results complete sequentially before +the batch ends. Primitive checksums are generated after timing, including during +warmup, and combine all ten results; every timed batch must reproduce the first +timed batch's checksum. QuickJS polls the JIT after every warmup and timed +batch, outside the timed region, exactly as during warmup. + +- `elapsed_ns` is the **upper median of the K timed batches** (the harness-wide + `ceil((n - 1) / 2)` quantile rule). It is a steady-batch latency after a + fixed warmup, not a claim that compilation has settled. +- `protocol.timed_batch_ns` stores all K raw timings in execution order; + `protocol.warmup_batch_ns` stores the 64 warmup timings. +- `protocol.fixed_batch_ns` is the first timed batch, i.e. the single batch that + `shared-js-fixed-warmup-v2` reported as `elapsed_ns`. It is a cold/jitter + **diagnostic only**. +- `protocol.outlier_batches` counts timed batches slower than + `protocol.outlier_ratio` (5) times the median. Outliers stay in + `timed_batch_ns` and are never silently dropped; `summarize_paired.py` reports + their total and the number of affected processes. + +Why v3 exists: under `bun -e`, Bun's batch at index 64 (the only batch v2 timed) +deterministically hit a one-off ~0.1-0.2 ms JSC stall, inflating Bun latency by +1.2x-32x on two-argument kernels. Running the same wrapper from a file removes +the stall, and the median of K batches no longer depends on one batch index. +For example, a noisy smoke of `numeric` on one host measured Bun at about +0.005 ms per ten calls with the file launch (fixed batch also about 0.005 ms), +while the `-e` control still showed a 0.13-0.19 ms fixed batch with 2-3 flagged +outliers and a median of about 0.005 ms. v2 and v3 numbers must not be mixed in +one comparison; `protocol_check.py` rejects mixed evidence. + +`protocol.fixed_metrics_before` and `fixed_metrics_after` surround the first +timed batch and exclude checksum execution, as in v2. +`protocol.timed_metrics_before` and `timed_metrics_after` surround the whole +timed window, **including** the untimed checksum and JIT-poll steps between +batches. Derive native, Tier 2, deopt, fallback, compilation, and installation +deltas from these snapshots; `paired.py` stores them as `fixed_deltas` and +`timed_deltas` and uses the timed window for `compilation_quiet_samples`. Counters aggregate the shared driver and workload; they do not identify the native tier of an individual workload function. Bun has no equivalent internal metrics and reports these snapshots as `null`; interpreter workers have no attached JIT and also leave them `null`. -QuickJS still times one outer Rust `Function::call` and, for asynchronous work, -its Promise completion bridge. Bun times the corresponding JS call and await. -Both timings include driver array/closure work. The protocol removes the old -per-workload Rust lookup/checksum/poll loop from the measured region, but does -not eliminate the outer host-call difference. The driver is part of the -measured workload; these are not isolated function-body timings. +QuickJS still times one outer Rust `Function::call` per batch and, for +asynchronous work, its Promise completion bridge. Bun times the corresponding JS +call and await. Both timings include driver array/closure work. The protocol +removes the old per-workload Rust lookup/checksum/poll loop from the measured +region, but does not eliminate the outer host-call difference. The driver is +part of the measured workload; these are not isolated function-body timings. -After collecting the fixed-window sample, QuickJS runs the existing bounded +After collecting the timed batches, QuickJS runs the existing bounded JIT readiness and settling checks, then another shared batch. Its timing is stored in `protocol.readiness_diagnostic_ns` and `phases.steady_state_ns`. Bun has no separately conditioned readiness diagnostic. Do not divide Bun's -fixed-window latency by QuickJS's post-readiness diagnostic and call that a +multi-batch latency by QuickJS's post-readiness diagnostic and call that a matched steady-state comparison. Legacy cumulative counters include first -execution, warmup, the fixed sample, and subsequent diagnostics; they cannot -prove native execution or settled compilation during the fixed window. +execution, warmup, the timed batches, and subsequent diagnostics; they cannot +prove native execution or settled compilation during the timed window. Forced Tier 2 probes can declare `globalThis.tier2ReadyInstalls` to require a known publication count in the readiness diagnostic. The direct-call probe requires four: two baseline and two optimizing artifacts. An unrelated blacklisted function cannot bypass that requirement. Explicit forced-Tier2 requirements reject unsettled diagnostic startup or new compilation during -the diagnostic batch. Those checks happen after the fixed-window sample and +the diagnostic batch. Those checks happen after the timed batches and do not make its warmup policy variable. Unsupported probes retain bounded fallback/readiness behavior. diff --git a/benchmarks/model.rs b/benchmarks/model.rs index e30953d03..9e111cbca 100644 --- a/benchmarks/model.rs +++ b/benchmarks/model.rs @@ -202,9 +202,43 @@ mod tests { assert_eq!(p99, 100); assert!(ci[0] <= median && median <= ci[1]); } + + #[test] + fn timed_batch_summary_uses_the_median_and_counts_but_keeps_spikes() { + // A single one-off stall (the old fixed-batch artifact) no longer + // determines the reported latency, but it stays visible. + let mut batches = vec![5_000u64; 16]; + batches[0] = 187_000; + let (median, outliers) = timed_batch_summary(&batches); + assert_eq!(median, 5_000); + assert_eq!(outliers, 1); + // Exactly 5x the median is not an outlier; above it is. + batches[0] = 25_000; + assert_eq!(timed_batch_summary(&batches).1, 0); + batches[0] = 25_001; + assert_eq!(timed_batch_summary(&batches).1, 1); + // Even K uses the harness-wide upper median. + assert_eq!(timed_batch_summary(&[1, 2, 3, 4]), (3, 0)); + assert_eq!(timed_batch_summary(&[]), (0, 0)); + } } +/// Legacy timing protocol: 64 warmup batches, then one timed batch. +#[allow(dead_code)] +pub const PROTOCOL_FIXED_WARMUP_V2: &str = "shared-js-fixed-warmup-v2"; +/// Current timing protocol: 64 warmup batches, then K consecutive timed +/// batches whose median becomes `elapsed_ns`. +#[allow(dead_code)] +pub const PROTOCOL_MULTIBATCH_V3: &str = "shared-js-multibatch-v3"; +/// A timed batch slower than this multiple of the timed-batch median is +/// counted as an outlier. Outliers are never discarded. +#[allow(dead_code)] +pub const OUTLIER_RATIO: u64 = 5; + /// Protocol identity separates these fixed-warmup timings from historical v1 timing. +/// +/// Fields after `readiness_diagnostic_ns` were added by +/// `shared-js-multibatch-v3`; they default to empty in v2 evidence. #[derive(Clone, Debug, Default, Deserialize, Serialize)] #[serde(deny_unknown_fields)] pub struct ProtocolEvidence { @@ -217,4 +251,45 @@ pub struct ProtocolEvidence { pub fixed_metrics_before: Option>, pub fixed_metrics_after: Option>, pub readiness_diagnostic_ns: Option, + /// Number of consecutive timed batches after the warmups (v3). + #[serde(default)] + pub timed_batches: u32, + /// Raw latency of every timed batch, in execution order (v3). + #[serde(default)] + pub timed_batch_ns: Vec, + /// Diagnostic only: the first timed batch, i.e. the single batch that + /// `shared-js-fixed-warmup-v2` reported as `elapsed_ns` (v3). + #[serde(default)] + pub fixed_batch_ns: Option, + /// Outlier rule: a timed batch above `outlier_ratio` times the median (v3). + #[serde(default)] + pub outlier_ratio: u64, + /// Number of timed batches that exceed the outlier rule (v3). + #[serde(default)] + pub outlier_batches: u32, + /// JIT metric snapshots around the whole timed window. They include the + /// untimed checksum and poll steps between timed batches (v3). + #[serde(default)] + pub timed_metrics_before: Option>, + #[serde(default)] + pub timed_metrics_after: Option>, + /// How Bun loaded its wrapper: `file` (v3 default) or `eval` (control) (v3). + #[serde(default)] + pub bun_launch: Option, + /// SHA-256 of the exact Bun wrapper module that was executed (v3). + #[serde(default)] + pub bun_wrapper_sha256: Option, +} + +/// Summary of K timed batches: (upper median, outlier count). +/// +/// The median uses the same upper-quantile rule as the rest of the harness. +#[allow(dead_code)] +pub fn timed_batch_summary(batches: &[u64]) -> (u64, u32) { + let mut sorted = batches.to_vec(); + sorted.sort_unstable(); + let median = quantile(&sorted, 0.5); + let threshold = median.saturating_mul(OUTLIER_RATIO); + let outliers = batches.iter().filter(|&&ns| ns > threshold).count(); + (median, u32::try_from(outliers).unwrap_or(u32::MAX)) } diff --git a/benchmarks/paired.py b/benchmarks/paired.py index 10d746aa5..9a89bdbb3 100644 --- a/benchmarks/paired.py +++ b/benchmarks/paired.py @@ -13,6 +13,7 @@ from pathlib import Path import subprocess import time +from protocol_check import PROTOCOL_V3, timed_batches_from_env, validate_protocol p = argparse.ArgumentParser() p.add_argument('--binaries', required=True, help='JSON map from variant to binary') @@ -25,6 +26,10 @@ args = p.parse_args() if args.pairs < 1: p.error('--pairs must be positive') +try: + timed_batches = timed_batches_from_env() +except ValueError as error: + p.error(str(error)) if args.cpu is not None and platform.system() != 'Linux': p.error('--cpu requires Linux taskset') import fcntl @@ -53,8 +58,8 @@ bun_version=subprocess.check_output([str(bun), '--version'],text=True).strip(), bun_sha256=hashlib.sha256(bun.read_bytes()).hexdigest(), bun_flags=[], rustc=subprocess.check_output(['rustc','-Vv'],text=True), - protocol='shared-js-fixed-warmup-v2', throughput_windows=0, - note='Latency-only paired candidate diagnostic. Fixed 10-call window; counters include shared driver. Readiness diagnostic is separately conditioned.') + protocol=PROTOCOL_V3, timed_batches=timed_batches, throughput_windows=0, + note='Latency-only paired candidate diagnostic. elapsed_ns is the median of the timed 10-call batches that follow 64 warmups; the first timed batch is the legacy fixed-batch diagnostic. Counters include shared driver. Readiness diagnostic is separately conditioned.') (out / 'metadata.json').write_text(json.dumps(metadata,indent=2)+'\n') for workload in args.workloads.split(','): script = Path(args.scripts) / (workload + '.js') @@ -68,20 +73,18 @@ if target.exists(): raise RuntimeError(f'Refusing to overwrite existing evidence: {target}') command = [bins[variant], 'worker', '--mode', mode, '--script', str(script)] + # Every engine and variant receives the same K explicitly. + worker_env = dict(os.environ, JIT_BENCH_TIMED_BATCHES=str(timed_batches)) if args.cpu is not None: command = ['taskset', '-c', args.cpu] + command started = time.monotonic() - result = subprocess.run(command, capture_output=True, text=True, timeout=180) + result = subprocess.run(command, capture_output=True, text=True, timeout=180, env=worker_env) if result.returncode: (out / 'failure.json').write_text(json.dumps(dict(command=command, stdout=result.stdout, stderr=result.stderr),indent=2)) raise RuntimeError(f'Worker failed: {command}: {result.stderr}') value = json.loads(result.stdout) protocol = value['protocol'] - assert protocol['name'] == metadata['protocol'] - assert protocol['script_sha256'] == metadata['scripts'][workload] - assert protocol['warmup_batches'] == len(protocol['warmup_batch_ns']) == 64 - assert protocol['calls_per_batch'] == 10 - assert value['elapsed_ns'] > 0 + validate_protocol(value, metadata, workload, mode) if mode != 'bun': assert value['native_exits'] == value['native_entries'] before, after = protocol['fixed_metrics_before'], protocol['fixed_metrics_after'] @@ -91,12 +94,20 @@ assert all(v >= 0 for v in value['fixed_deltas'].values()) assert value['fixed_deltas']['native_entries'] == value['fixed_deltas']['native_exits'] value['fixed_compilation_quiet'] = (after['pending_worker_jobs'] == before['pending_worker_jobs'] == 0 and after['pending_snapshot_bytes'] == before['pending_snapshot_bytes'] == 0 and value['fixed_deltas']['installed'] == 0 and value['fixed_deltas']['compile_ns'] == 0) + before, after = protocol.get('timed_metrics_before'), protocol.get('timed_metrics_after') + if before is not None: + # Spans every timed batch plus the untimed checksum/poll steps between them. + value['timed_deltas'] = {k:after[k]-before[k] for k in cumulative} + assert all(v >= 0 for v in value['timed_deltas'].values()) + assert value['timed_deltas']['native_entries'] == value['timed_deltas']['native_exits'] + value['timed_compilation_quiet'] = (after['pending_worker_jobs'] == before['pending_worker_jobs'] == 0 and after['pending_snapshot_bytes'] == before['pending_snapshot_bytes'] == 0 and value['timed_deltas']['installed'] == 0 and value['timed_deltas']['compile_ns'] == 0) value.update(variant=variant, requested_mode=mode, pair_index=pair, process_wall_seconds=time.monotonic()-started) target.write_text(json.dumps(value,indent=2)+'\n') print(f'{workload}: pair {pair} complete',flush=True) values = [json.loads(path.read_text()) for path in out.glob(f'{workload}-*.json')] assert len({v['checksum'] for v in values}) == 1, workload assert len({v['protocol']['driver_sha256'] for v in values}) == 1, workload + assert len({v['protocol'].get('bun_wrapper_sha256') for v in values if v['requested_mode'] == 'bun'}) <= 1, workload (out / 'metadata.json').write_text(json.dumps(metadata,indent=2)+'\n') for variant, binary in bins.items(): assert hashlib.sha256(Path(binary).read_bytes()).hexdigest() == metadata['binary_sha256'][variant] diff --git a/benchmarks/protocol_check.py b/benchmarks/protocol_check.py new file mode 100644 index 000000000..826621c79 --- /dev/null +++ b/benchmarks/protocol_check.py @@ -0,0 +1,67 @@ +"""Shared worker-protocol validation for paired.py and summarize_paired.py. + +`shared-js-fixed-warmup-v2` timed one batch after 64 warmups. It is kept +readable so historical evidence can still be summarized, but it must never be +mixed with `shared-js-multibatch-v3`, where `elapsed_ns` is the upper median +of `timed_batches` consecutive timed batches. +""" +import os + +PROTOCOL_V2 = 'shared-js-fixed-warmup-v2' +PROTOCOL_V3 = 'shared-js-multibatch-v3' +WARMUP_BATCHES = 64 +CALLS_PER_BATCH = 10 +OUTLIER_RATIO = 5 +DEFAULT_TIMED_BATCHES = 16 + + +def timed_batches_from_env(): + raw = os.environ.get('JIT_BENCH_TIMED_BATCHES') + if raw is None: + return DEFAULT_TIMED_BATCHES + if not raw.isdigit() or int(raw) <= 0: + raise ValueError('JIT_BENCH_TIMED_BATCHES must be a positive integer') + return int(raw) + + +def upper_median(values): + ordered = sorted(values) + return ordered[-(-(len(ordered) - 1) // 2)] if ordered else 0 + + +def timed_summary(batches): + median = upper_median(batches) + return median, sum(ns > median * OUTLIER_RATIO for ns in batches) + + +def validate_protocol(sample, metadata, workload, mode): + """Reject incompatible or internally inconsistent worker evidence.""" + protocol = sample['protocol'] + name = protocol['name'] + if name not in (PROTOCOL_V2, PROTOCOL_V3): + raise AssertionError(f'unsupported timing protocol {name!r}') + if name != metadata['protocol']: + raise AssertionError(f'protocol {name!r} mixed into {metadata["protocol"]!r} evidence') + assert protocol['script_sha256'] == metadata['scripts'][workload] + assert protocol['calls_per_batch'] == CALLS_PER_BATCH + assert protocol['warmup_batches'] == len(protocol['warmup_batch_ns']) == WARMUP_BATCHES + assert sample['elapsed_ns'] > 0 + if name == PROTOCOL_V2: + return + timed = protocol['timed_batch_ns'] + expected = metadata.get('timed_batches', DEFAULT_TIMED_BATCHES) + assert protocol['timed_batches'] == len(timed) == expected, (protocol['timed_batches'], len(timed), expected) + assert protocol['fixed_batch_ns'] == timed[0] + assert protocol['outlier_ratio'] == OUTLIER_RATIO + assert (sample['elapsed_ns'], protocol['outlier_batches']) == timed_summary(timed) + if mode == 'bun': + assert protocol['bun_launch'] == 'file', 'publishable Bun evidence must use the file wrapper' + assert len(protocol['bun_wrapper_sha256']) == 64 + + +if __name__ == '__main__': + # Self-test: python3 benchmarks/protocol_check.py + assert upper_median([1, 2, 3, 4]) == 3 and upper_median([1, 2, 3]) == 2 + assert timed_summary([5] * 15 + [187]) == (5, 1) + assert timed_summary([5] * 15 + [25]) == (5, 0) + print('protocol_check self-test ok') diff --git a/benchmarks/report.rs b/benchmarks/report.rs index 52d1550e3..f6361f549 100644 --- a/benchmarks/report.rs +++ b/benchmarks/report.rs @@ -28,14 +28,25 @@ fn real_main() -> Result { } fn validate(data: &BenchmarkFile) -> Result<(), String> { - if data + if let Some(protocol) = data .modes .iter() .flat_map(|m| &m.workloads) .flat_map(|w| &w.samples) - .any(|s| s.protocol.is_some()) + .find_map(|s| s.protocol.as_ref()) { - return Err("incompatible timing protocol: the legacy gate reporter cannot evaluate shared-js-fixed-warmup-v2; use fixed-window metrics and report readiness diagnostics separately".into()); + let name = if protocol.name.is_empty() { + "an unnamed protocol" + } else { + protocol.name.as_str() + }; + return Err(format!( + "incompatible timing protocol: the legacy gate reporter cannot evaluate {name} \ + (it only understands pre-v2 samples without protocol evidence; {} and {} \ + need a protocol-aware paired summary such as summarize_paired.py)", + model::PROTOCOL_FIXED_WARMUP_V2, + model::PROTOCOL_MULTIBATCH_V3, + )); } if data.schema != "jit-benchmark-v1" { return Err("unsupported schema".into()); @@ -709,6 +720,35 @@ mod tests { .contains("incompatible timing protocol")); } + #[test] + fn legacy_gate_reporter_explicitly_rejects_the_multibatch_protocol() { + for name in [ + model::PROTOCOL_FIXED_WARMUP_V2, + model::PROTOCOL_MULTIBATCH_V3, + ] { + let mut data = valid_file(); + data.modes[1].workloads[0].samples[3].protocol = Some(model::ProtocolEvidence { + name: name.into(), + timed_batches: 16, + ..Default::default() + }); + let error = validate(&data).unwrap_err(); + assert!(error.contains("incompatible timing protocol"), "{error}"); + assert!(error.contains(name), "{error}"); + } + } + + #[test] + fn v2_protocol_evidence_still_deserializes_without_v3_fields() { + let v2 = r#"{"name":"shared-js-fixed-warmup-v2","script_sha256":"a","driver_sha256":"b", + "warmup_batches":64,"calls_per_batch":10,"warmup_batch_ns":[1], + "fixed_metrics_before":null,"fixed_metrics_after":null,"readiness_diagnostic_ns":null}"#; + let parsed: model::ProtocolEvidence = serde_json::from_str(v2).unwrap(); + assert_eq!(parsed.name, model::PROTOCOL_FIXED_WARMUP_V2); + assert_eq!(parsed.timed_batches, 0); + assert!(parsed.timed_batch_ns.is_empty() && parsed.fixed_batch_ns.is_none()); + } + #[test] fn validator_rejects_dirty_or_stale_provenance() { let mut data = valid_file(); diff --git a/benchmarks/run.rs b/benchmarks/run.rs index 20fe492ac..271bd6454 100644 --- a/benchmarks/run.rs +++ b/benchmarks/run.rs @@ -17,6 +17,7 @@ use std::{ const BATCH_DRIVER: &str = include_str!("batch-driver.js"); const FIXED_WARMUP_BATCHES: usize = 64; +const DEFAULT_TIMED_BATCHES: usize = 16; const DEFAULT_WARMUPS: usize = 5; const DEFAULT_SAMPLES: usize = 30; @@ -468,12 +469,15 @@ fn worker(mode: &str, script: &str) -> Result<(), String> { let start = Instant::now(); let _first_checksum = timed_batch_count(&context, 1)?; phases.first_eval_ns = ns(start.elapsed()); + let timed_batches = timed_batch_count_config()?; let mut protocol = model::ProtocolEvidence { - name: "shared-js-fixed-warmup-v2".into(), + name: model::PROTOCOL_MULTIBATCH_V3.into(), script_sha256: sha256_file(script), driver_sha256: sha256("batch-driver.js"), warmup_batches: FIXED_WARMUP_BATCHES as u32, calls_per_batch: 10, + timed_batches: timed_batches as u32, + outlier_ratio: model::OUTLIER_RATIO, ..Default::default() }; for _ in 0..FIXED_WARMUP_BATCHES { @@ -483,10 +487,29 @@ fn worker(mode: &str, script: &str) -> Result<(), String> { jit.poll(); } } - let (fixed_elapsed, fixed_checksum, before_fixed, after_fixed) = - timed_batch_recorded(&context, 10, jit.as_ref())?; - protocol.fixed_metrics_before = before_fixed; - protocol.fixed_metrics_after = after_fixed; + // K consecutive timed batches continue the warmup regime exactly: + // checksum and JIT polling stay outside each timed region. The first + // one is the legacy v2 fixed batch, kept as a diagnostic. + protocol.timed_metrics_before = jit.as_ref().map(|j| metric_snapshot(&j.metrics())); + let mut fixed_checksum = String::new(); + for index in 0..timed_batches { + let (elapsed, checksum, before, after) = timed_batch_recorded(&context, 10, jit.as_ref())?; + if index == 0 { + protocol.fixed_batch_ns = Some(elapsed); + protocol.fixed_metrics_before = before; + protocol.fixed_metrics_after = after; + fixed_checksum = checksum; + } else if checksum != fixed_checksum { + return Err(format!("checksum changed at timed batch {index}")); + } + protocol.timed_batch_ns.push(elapsed); + if let Some(jit) = &jit { + jit.poll(); + } + } + protocol.timed_metrics_after = jit.as_ref().map(|j| metric_snapshot(&j.metrics())); + let (timed_median, outliers) = model::timed_batch_summary(&protocol.timed_batch_ns); + protocol.outlier_batches = outliers; let threshold_start = Instant::now(); let threshold_deadline = threshold_start + threshold_timeout(mode); let mut install_poll = 0u64; @@ -584,7 +607,7 @@ fn worker(mode: &str, script: &str) -> Result<(), String> { let abi = AbiInfo::linked().map_err(err)?; phases.total_ns = ns(total.elapsed()); let result = WorkerResult { - elapsed_ns: fixed_elapsed, + elapsed_ns: timed_median, checksum: fixed_checksum, native_entries: metrics.native_entries, native_acquisitions: metrics.native_acquisitions, @@ -731,6 +754,10 @@ fn sampling_config() -> Result { }) } +fn timed_batch_count_config() -> Result { + positive_env("JIT_BENCH_TIMED_BATCHES", DEFAULT_TIMED_BATCHES) +} + fn positive_env(name: &str, default: usize) -> Result { match env::var(name) { Ok(value) => value @@ -953,12 +980,13 @@ fn child(executable: &Path, mode: &str, script: &Path) -> Result Result { - let bun = env::var("JIT_BENCH_BUN").unwrap_or_else(|_| "bun".into()); - let wrapper = format!( +/// Bun wrapper module shared by the file and eval launches. It never embeds +/// the script path, so its SHA-256 identifies the exact timing protocol. +fn bun_wrapper(timed_batches: usize) -> Result { + Ok(format!( r#" import {{ readFileSync }} from 'node:fs'; -(0,eval)(readFileSync(process.argv[1],'utf8')); +(0,eval)(readFileSync(process.argv.at(-1),'utf8')); (0,eval)({driver}); let first=benchmarkBatch(1); if(first&&typeof first.then==='function') await first; benchmarkChecksum(); const warmup_batch_ns=[]; @@ -969,35 +997,103 @@ async function measure() {{ return [elapsed,benchmarkChecksum()]; }} for(let i=0;i<{warmups};i++) warmup_batch_ns.push((await measure())[0]); -const [elapsed_ns,checksum]=await measure(); -console.log(JSON.stringify({{elapsed_ns,checksum,warmup_batch_ns}})); +const timed_batch_ns=[]; +let checksum; +for(let i=0;i<{timed_batches};i++) {{ + const [elapsed,batch_checksum]=await measure(); + if(i===0) checksum=batch_checksum; + else if(batch_checksum!==checksum) throw new Error('checksum changed at timed batch '+i); + timed_batch_ns.push(elapsed); +}} +console.log(JSON.stringify({{checksum,warmup_batch_ns,timed_batch_ns}})); "#, driver = serde_json::to_string(BATCH_DRIVER).map_err(err)?, - warmups = FIXED_WARMUP_BATCHES - ); + warmups = FIXED_WARMUP_BATCHES, + )) +} + +/// A uniquely named wrapper file removed when the Bun child finishes. +struct TempWrapper(PathBuf); +impl TempWrapper { + fn create(contents: &str) -> Result { + use std::sync::atomic::{AtomicU64, Ordering}; + static NEXT: AtomicU64 = AtomicU64::new(0); + let nonce = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_nanos()) + .unwrap_or(0); + let path = env::temp_dir().join(format!( + "jit-bench-bun-{}-{}-{nonce}.mjs", + std::process::id(), + NEXT.fetch_add(1, Ordering::Relaxed) + )); + fs::OpenOptions::new() + .write(true) + .create_new(true) + .open(&path) + .and_then(|mut file| std::io::Write::write_all(&mut file, contents.as_bytes())) + .map_err(|error| format!("cannot write Bun wrapper {}: {error}", path.display()))?; + Ok(Self(path)) + } +} +impl Drop for TempWrapper { + fn drop(&mut self) { + let _ = fs::remove_file(&self.0); + } +} + +/// `file` (default) runs the wrapper from a temporary `.mjs` file. `eval` +/// reproduces the historical `bun -e` launch and is only a control. +fn bun_launch() -> Result<&'static str, String> { + match env::var("JIT_BENCH_BUN_LAUNCH").as_deref() { + Err(env::VarError::NotPresent) | Ok("file") => Ok("file"), + Ok("eval") => Ok("eval"), + Ok(other) => Err(format!( + "JIT_BENCH_BUN_LAUNCH must be file or eval, got {other}" + )), + Err(error) => Err(format!("cannot read JIT_BENCH_BUN_LAUNCH: {error}")), + } +} + +fn bun_child(script: &Path) -> Result { + let bun = env::var("JIT_BENCH_BUN").unwrap_or_else(|_| "bun".into()); + let timed_batches = timed_batch_count_config()?; + let launch = bun_launch()?; + let wrapper = bun_wrapper(timed_batches)?; + let temp = TempWrapper::create(&wrapper)?; + let wrapper_sha256 = sha256_file(&temp.0.to_string_lossy()); let started = Instant::now(); - let output = Command::new(&bun) - .args(["-e", &wrapper]) - .arg(script) - .output() - .map_err(err)?; + let mut cmd = Command::new(&bun); + if launch == "eval" { + cmd.args(["-e", &wrapper]); + } else { + cmd.arg(&temp.0); + } + let output = cmd.arg(script).output().map_err(err)?; + drop(temp); if !output.status.success() { return Err(String::from_utf8_lossy(&output.stderr).into_owned()); } #[derive(Deserialize)] struct BunOut { - elapsed_ns: u64, checksum: String, warmup_batch_ns: Vec, + timed_batch_ns: Vec, } let out: BunOut = serde_json::from_slice(&output.stdout).map_err(err)?; + if out.warmup_batch_ns.len() != FIXED_WARMUP_BATCHES + || out.timed_batch_ns.len() != timed_batches + { + return Err("Bun wrapper returned an incomplete batch sequence".into()); + } + let (elapsed_ns, outlier_batches) = model::timed_batch_summary(&out.timed_batch_ns); let phases = PhaseTiming { - steady_state_ns: out.elapsed_ns, + steady_state_ns: elapsed_ns, total_ns: ns(started.elapsed()), ..Default::default() }; Ok(WorkerResult { - elapsed_ns: out.elapsed_ns, + elapsed_ns, checksum: out.checksum, native_entries: 0, native_acquisitions: 0, @@ -1021,12 +1117,19 @@ console.log(JSON.stringify({{elapsed_ns,checksum,warmup_batch_ns}})); active_ir_bytes: 0, automatic_ready: true, protocol: model::ProtocolEvidence { - name: "shared-js-fixed-warmup-v2".into(), + name: model::PROTOCOL_MULTIBATCH_V3.into(), script_sha256: sha256_file(&script.to_string_lossy()), driver_sha256: sha256("batch-driver.js"), warmup_batches: FIXED_WARMUP_BATCHES as u32, calls_per_batch: 10, warmup_batch_ns: out.warmup_batch_ns, + timed_batches: timed_batches as u32, + fixed_batch_ns: out.timed_batch_ns.first().copied(), + timed_batch_ns: out.timed_batch_ns, + outlier_ratio: model::OUTLIER_RATIO, + outlier_batches, + bun_launch: Some(launch.into()), + bun_wrapper_sha256: Some(wrapper_sha256), ..Default::default() }, phases, @@ -1285,6 +1388,21 @@ mod tests { assert!(validate_sample("tier2", &r, true).is_err()); } #[test] + fn bun_wrapper_is_script_independent_and_times_k_batches() { + let wrapper = bun_wrapper(16).unwrap(); + assert!(wrapper.contains("process.argv.at(-1)")); + assert!(wrapper.contains(&format!("i<{FIXED_WARMUP_BATCHES};"))); + assert!(wrapper.contains("i<16;")); + assert_eq!(wrapper, bun_wrapper(16).unwrap()); + assert_ne!(wrapper, bun_wrapper(8).unwrap()); + let temp = TempWrapper::create(&wrapper).unwrap(); + let path = temp.0.clone(); + assert_eq!(path.extension().and_then(|e| e.to_str()), Some("mjs")); + assert_eq!(fs::read_to_string(&path).unwrap(), wrapper); + drop(temp); + assert!(!path.exists(), "Bun wrapper must be removed after use"); + } + #[test] fn rotation_interleaves_order() { let m = vec!["a".into(), "b".into(), "c".into()]; assert_eq!(rotated(&m, 1).collect::>(), vec!["b", "c", "a"]); @@ -1421,6 +1539,26 @@ mod tests { .join(script); let result = bun_child(&path).unwrap(); validate_sample("bun", &result, true).unwrap(); + let protocol = &result.protocol; + assert_eq!(protocol.name, model::PROTOCOL_MULTIBATCH_V3); + assert_eq!(protocol.warmup_batch_ns.len(), FIXED_WARMUP_BATCHES); + assert_eq!( + protocol.timed_batches as usize, + protocol.timed_batch_ns.len() + ); + assert_eq!( + protocol.fixed_batch_ns, + protocol.timed_batch_ns.first().copied() + ); + assert_eq!( + (result.elapsed_ns, protocol.outlier_batches), + model::timed_batch_summary(&protocol.timed_batch_ns) + ); + assert_eq!(protocol.bun_launch.as_deref(), Some("file")); + assert_eq!( + protocol.bun_wrapper_sha256.as_ref().map(String::len), + Some(64) + ); let sample = evidence_for_mode("bun", 0, result); assert!(sample.native_entries.is_none() && sample.native_exits.is_none()); assert!(sample.tier1_entries.is_none() && sample.tier2_entries.is_none()); diff --git a/benchmarks/schema/jit-benchmark-v1.json b/benchmarks/schema/jit-benchmark-v1.json index 45cbc108b..6129e33a1 100644 --- a/benchmarks/schema/jit-benchmark-v1.json +++ b/benchmarks/schema/jit-benchmark-v1.json @@ -190,7 +190,7 @@ "additionalProperties": false, "required": ["name", "script_sha256", "driver_sha256", "warmup_batches", "calls_per_batch", "warmup_batch_ns", "fixed_metrics_before", "fixed_metrics_after", "readiness_diagnostic_ns"], "properties": { - "name": { "const": "shared-js-fixed-warmup-v2" }, + "name": { "enum": ["shared-js-fixed-warmup-v2", "shared-js-multibatch-v3"] }, "script_sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" }, "driver_sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" }, "warmup_batches": { "const": 64 }, @@ -198,7 +198,25 @@ "warmup_batch_ns": { "type": "array", "minItems": 64, "maxItems": 64, "items": { "type": "integer", "minimum": 0 } }, "fixed_metrics_before": { "type": ["object", "null"], "additionalProperties": { "type": "integer", "minimum": 0 } }, "fixed_metrics_after": { "type": ["object", "null"], "additionalProperties": { "type": "integer", "minimum": 0 } }, - "readiness_diagnostic_ns": { "type": ["integer", "null"], "minimum": 0 } + "readiness_diagnostic_ns": { "type": ["integer", "null"], "minimum": 0 }, + "timed_batches": { "type": "integer", "minimum": 0 }, + "timed_batch_ns": { "type": "array", "items": { "type": "integer", "minimum": 0 } }, + "fixed_batch_ns": { "type": ["integer", "null"], "minimum": 0 }, + "outlier_ratio": { "type": "integer", "minimum": 0 }, + "outlier_batches": { "type": "integer", "minimum": 0 }, + "timed_metrics_before": { "type": ["object", "null"], "additionalProperties": { "type": "integer", "minimum": 0 } }, + "timed_metrics_after": { "type": ["object", "null"], "additionalProperties": { "type": "integer", "minimum": 0 } }, + "bun_launch": { "enum": ["file", "eval", null] }, + "bun_wrapper_sha256": { "type": ["string", "null"], "pattern": "^[0-9a-f]{64}$" } + }, + "if": { "properties": { "name": { "const": "shared-js-multibatch-v3" } } }, + "then": { + "required": ["timed_batches", "timed_batch_ns", "fixed_batch_ns", "outlier_ratio", "outlier_batches"], + "properties": { + "timed_batches": { "minimum": 1 }, + "timed_batch_ns": { "minItems": 1 }, + "outlier_ratio": { "const": 5 } + } } }, "pair_index": { "type": "integer" }, diff --git a/benchmarks/summarize_paired.py b/benchmarks/summarize_paired.py index 96792ef7b..0670ee978 100644 --- a/benchmarks/summarize_paired.py +++ b/benchmarks/summarize_paired.py @@ -5,6 +5,7 @@ from pathlib import Path import random import statistics +from protocol_check import validate_protocol p=argparse.ArgumentParser() p.add_argument('directory') @@ -32,11 +33,7 @@ def interval(values): sample=json.loads((root/f'{workload}-{i}-{variant}-{mode}.json').read_text()) assert (sample['pair_index'],sample['variant'],sample['requested_mode']) == (i,variant,mode) protocol=sample['protocol'] - assert protocol['name'] == metadata['protocol'] - assert protocol['script_sha256'] == metadata['scripts'][workload] - assert protocol['calls_per_batch'] == 10 - assert protocol['warmup_batches'] == len(protocol['warmup_batch_ns']) == 64 - assert sample['elapsed_ns'] > 0 + validate_protocol(sample, metadata, workload, mode) checksums.add(sample['checksum']) drivers.add(protocol['driver_sha256']) if len(checksums) != 1 or len(drivers) != 1: @@ -50,13 +47,21 @@ def interval(values): speed,ci=interval([a['elapsed_ns']/b['elapsed_ns'] for a,b in zip(data[reference],values)]) comparisons['/'.join(reference)]=dict(speed=speed,ci95=ci) deltas=[v.get('fixed_deltas') for v in values] + multibatch=values[0]['protocol'].get('timed_batches',0) > 0 rows.append(dict(workload=workload,variant=variant,mode=mode, + protocol=metadata['protocol'], median_ns=statistics.median(v['elapsed_ns'] for v in values), + # Diagnostics only: the legacy single fixed batch and spikes + # among the timed batches. Spikes are counted, never dropped. + fixed_batch_median_ns=(statistics.median(v['protocol']['fixed_batch_ns'] for v in values) if multibatch else None), + outlier_batches=(sum(v['protocol']['outlier_batches'] for v in values) if multibatch else None), + outlier_processes=(sum(v['protocol']['outlier_batches'] > 0 for v in values) if multibatch else None), comparisons=comparisons, - compilation_quiet_samples=sum(v.get('fixed_compilation_quiet',False) for v in values), + compilation_quiet_samples=sum(v.get('timed_compilation_quiet' if multibatch else 'fixed_compilation_quiet',False) for v in values), native_entry_range=([min(d['native_entries'] for d in deltas),max(d['native_entries'] for d in deltas)] if all(d is not None for d in deltas) else None))) (root/'summary.json').write_text(json.dumps(rows,indent=2)+'\n') for row in rows: old=row['comparisons'][next(iter(metadata['binaries']))+'/automatic'] bun=row['comparisons'][next(iter(metadata['binaries']))+'/bun'] - print(f"{row['workload']:20} {row['variant']:10}/{row['mode']:11} {row['median_ns']/1e6:9.4f} ms; baseline {old['speed']:.3f}x {old['ci95']}; Bun {bun['speed']:.4f}x; quiet {row['compilation_quiet_samples']}/{metadata['pairs']}; entries {row['native_entry_range']}") + spikes='' if row['outlier_batches'] is None else f"; outlier batches {row['outlier_batches']} in {row['outlier_processes']} processes" + print(f"{row['workload']:20} {row['variant']:10}/{row['mode']:11} {row['median_ns']/1e6:9.4f} ms; baseline {old['speed']:.3f}x {old['ci95']}; Bun {bun['speed']:.4f}x; quiet {row['compilation_quiet_samples']}/{metadata['pairs']}; entries {row['native_entry_range']}{spikes}") diff --git a/docs/roadmap/p0-bench-protocol.md b/docs/roadmap/p0-bench-protocol.md new file mode 100644 index 000000000..925abe8f0 --- /dev/null +++ b/docs/roadmap/p0-bench-protocol.md @@ -0,0 +1,75 @@ +# P0 benchmark protocol: `shared-js-multibatch-v3` + +Branch `perf/p0-bench-protocol`, based on `82d3808`. This is a harness-only +change: the runtime, patches, ABI, and README matrix are unchanged. + +## What changed + +- Every engine (interpreter, Tier 1, forced Tier 2, automatic, Bun) runs one + first call, 64 warmup batches, then `K` consecutive timed batches of ten calls + (`K = 16`, `JIT_BENCH_TIMED_BATCHES` overrides it). `elapsed_ns` is the upper + median of the K batches. All raw values are kept in `protocol.timed_batch_ns`. +- `protocol.fixed_batch_ns` keeps the old v2 single-batch value (the first timed + batch) as a diagnostic. +- `protocol.outlier_batches` counts timed batches above 5x the median + (`protocol.outlier_ratio`). Outliers are never dropped. +- Bun runs the wrapper from a temporary `.mjs` file (`protocol.bun_launch = + "file"`, `protocol.bun_wrapper_sha256`). `JIT_BENCH_BUN_LAUNCH=eval` reproduces + `bun -e` as a control only, and `paired.py` rejects it. +- QuickJS records `timed_metrics_before/after` around the whole timed window, in + addition to the v2 `fixed_metrics_before/after` around the first timed batch. +- `jit-bench-report` still rejects all protocol-tagged samples (v2 and v3) and + names the protocol in the error. `paired.py` and `summarize_paired.py` share + `benchmarks/protocol_check.py`, which validates v3 invariants, refuses mixed + v2/v3 evidence, and reports outlier counts and the fixed-batch diagnostic. + +## Noisy diagnostic (loaded 24-core host, 7 alternating processes) + +Bun, `numeric`: the v2 fixed batch was 0.129 ms per ten calls. The v3 median is +0.0045 ms, and the fixed batch under the file launch is also about 0.005 ms. +Under `-e` (control), batches 64 and 68 still stall for 0.13–0.40 ms and are +flagged as outliers, while the median stays at about 0.005 ms. + +## Open issue for the integrator: JSC reaches its top tier later under the file launch + +The raw sequences show that Bun reaches its fastest tier later when started from +a file than when started with `-e`. The wrapper text is identical, and +`bun build --no-bundle` shows no rewrite. Per-process medians, in µs per ten +calls: + +| workload | launch | batches 48–63 | 64–79 (v3 window) | 144+ | +| --- | --- | ---: | ---: | ---: | +| numeric | file | 4.6–11.4 | 4.5–6.3 | 4.2–10.0 | +| numeric | eval | 4.6 | 4.6 | 4.1 | +| call-heavy | file | 16.7–19.0 | 6.3–18.4 | 6.1 | +| call-heavy | eval | 6.3–14.2 | 6.3 | 6.1 | +| scalar-control-flow | file | 20.9–23.8 | 10.4–20.6 | 9.9–19.8 | +| scalar-control-flow | eval | 10.3–10.4 | 10.1–10.2 | 10.0 | +| host-compute | file | 20.9–27.1 | 9.8–20.6 | 9.2–19.4 | +| host-compute | eval | 16.1–20.2 | 9.6–19.5 | 9.3–18.9 | + +With a fixed 64-batch warmup under the file launch, the v3 window can therefore +measure some Bun processes before JSC's top tier lands. For these workloads, v3 +Bun latency can be up to about 3x slower than Bun's eventual steady state. This +is the reverse of the v2 bias (which made Bun look up to about 30x slower on a +single batch), and much smaller. It is the same fixed warmup policy for every +engine, so the comparison remains equal-policy. Still, it is not a +"fully settled Bun" number. Some processes also settle into two different +steady states (host-compute: 9.2 µs or about 19 µs in both launch modes). + +Options before publishing the S0 matrix (not implemented here): + +1. Keep v3 as is, and report `outlier_batches` together with each engine's + `timed_batch_ns` spread. +2. Add a longer common warmup for every engine (for example 256 batches), + which needs a new protocol name because `warmup_batches` is pinned to 64 + in `protocol_check.py` and the schema. +3. Add a settle-based warmup rule, applied identically to all engines. + +## Remaining P0 work (integrator) + +- Re-run the full 30-scenario three-engine matrix under v3 (`paired.py` or + `jit-bench compare`) to establish baseline `S0`. Replace the README numbers and + label the old matrix as historical data affected by the Bun fixed-batch stall. +- Upgrade roadmap section 3 to a paired table with confidence intervals. +- Decide on the JSC late-tier-up issue above. From c52a67ab662ab111acac8e95f9c2c457bf7f9a65 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 16:36:48 +0800 Subject: [PATCH 02/54] perf(jit): make helper-boundary validation O(1) Add QuickJS patch 0021-helper-boundary: - Replace the per-helper linear bytecode scan in qjsjit_valid_pc with a lazily built, bytecode-owned instruction-boundary bitmap that uses the scan's exact decoding rules (scan fallback on allocation failure; freed with the bytecode and counted in JS_ComputeMemoryUsage). - Keep every helper-frame check but inline the pending-exception test and move rejection out of line so the validated fast path stays compact. - JS_JitInlineEnter: drop the redundant call/continuation PC scans and reserve shadow frames from a runtime-owned LIFO arena with heap fallback (heap-only under ASan/MSan). Tests cover mid-instruction and past-end frame PCs in helper validation and LIFO arena accounting for nested inline frames. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p1a-helper-boundary.md | 91 ++++++++ jit/tests/helpers.rs | 17 ++ sys/build_support/patch.rs | 5 +- sys/patches/0021-helper-boundary.patch | 288 +++++++++++++++++++++++++ sys/patches/README.md | 14 ++ sys/tests/inline_recovery.rs | 17 ++ sys/tests/jit_patch.rs | 1 + 7 files changed, 431 insertions(+), 2 deletions(-) create mode 100644 docs/roadmap/p1a-helper-boundary.md create mode 100644 sys/patches/0021-helper-boundary.patch diff --git a/docs/roadmap/p1a-helper-boundary.md b/docs/roadmap/p1a-helper-boundary.md new file mode 100644 index 000000000..b3fb51502 --- /dev/null +++ b/docs/roadmap/p1a-helper-boundary.md @@ -0,0 +1,91 @@ +# P1a: O(1) helper-boundary validation + +Branch `perf/p1a-helper-boundary`, based on `82d3808`. This covers roadmap P1 +items 1, 2 and 5. The C changes are in `sys/patches/0021-helper-boundary.patch`. + +## What changed + +1. **O(1) PC validation (B1).** `qjsjit_valid_pc` used to scan the bytecode + linearly. Its cursor only helped forward queries, so the first helper call + after a loop back-edge rescanned from offset 0. Each `JSFunctionBytecode` + now builds an instruction-boundary bitmap on the first query, and every + later query is a range check plus one bit test. + - The bitmap decodes instructions with exactly the old scan's rules. + Offsets after an undecodable instruction stay clear. + - Bit `byte_code_len` is set only when the decode ends exactly at the end + of the bytecode, so `allow_end` behaves as before. + - The bitmap takes `len/8 + 1` bytes from the runtime allocator. It is + freed with the bytecode and counted in `JS_ComputeMemoryUsage`. + - If the allocation fails, the old cursor scan answers. Nothing raises an + exception. +2. **Cheaper per-helper validation (partial item 2).** Every identity, ABI, + stack and PC check stays. Two changes make the fast path cheaper: + - The pending-exception test is inlined; it was an out-of-line + `JS_HasException` call on every helper. + - `qjsjit_reject_helper` is `no_inline`. The fast path no longer carries + spills or stack-protector setup. +3. **`JS_JitInlineEnter`.** + - The redundant `call_pc`/`continuation_pc` scans are gone. Frame + validation already proves that `caller->pc == call_pc` is an instruction + boundary. The continuation is checked as + `call_pc + size(call opcode) < len`, which makes it the next boundary. + - Shadow frames come from a LIFO arena owned by the runtime. The arena + starts at 32 KiB and grows only while empty, up to the 1 MiB inline + budget plus alignment. It is released in `JS_FreeRuntime`. + - A frame that does not fit, or an arena allocation failure, falls back to + the heap. Declined entries release their storage after the depth + decrement, so depth 0 always resets the arena. + - ASan and MSan builds always use the heap, so use-after-free detection + still works. + +No public ABI, binding layout or ABI version changes. + +## Tests + +- `jit/tests/helpers.rs`: `invalid_identity_map_index_and_slot_are_rejected_before_touching_values` + now also covers a frame PC inside an instruction and a frame PC past the end. + Both must be rejected without touching any values. +- `sys/tests/inline_recovery.rs`: + - Nested inline frames (scenario 16) must be stacked contiguously in the + arena and must unwind it in LIFO order. + - Every scenario must leave depth, bytes and arena use at zero. + - The test also passes with `QUICKJS_INLINE_TEST_SANITIZER=address`. + +## Noisy diagnostic (loaded 24-core host, not publishable) + +Method: 7 alternating runs of `jit-bench worker --mode automatic`. Each run's +value is the median of the last 16 `protocol.warmup_batch_ns`, and the table +shows the median across runs. The baseline is the 82d3808 binary. Speed is +base/new. On every workload, the checksums from base, new and the interpreter +were identical. + +| workload | base ms | new ms | speed | +| --- | ---: | ---: | ---: | +| objects-polymorphic | 23.934 | 13.742 | 1.74x | +| generic-call-fallback | 9.126 | 6.205 | 1.47x | +| property-heavy | 0.0986 | 0.0979 | 1.01x | +| arrays-typed | 2.979 | 2.892 | 1.03x | +| float64array-traversal | 1.102 | 1.053 | 1.05x | + +These match or exceed the roadmap's unsafe "skip PC validation" upper bounds +(1.71x and 1.29x). There are no confidence intervals, so read the results +near 1.0x as ties. + +## Remaining work + +- **Cookie-only per-helper validation (the rest of item 2).** Not done. + - `helpers.rs` requires every helper to reject runtime-id, runtime-API, + helper-ABI-version, stack-capacity, stack-top, generation and cookie + tampering. + - The benchmark binary is built with `test-support`, so moving those checks + behind a CONFIG flag would either break those tests or not be measured. + - After this patch, a gdb-sampled profile shows the remaining validation is + a straight-line set of about 25 L1-hot compares. The next step is to + record a validated-entry epoch at native entry/OSR/resume, and to move the + immutable-field checks under a build flag that the tamper tests enable + explicitly. +- **Stack-map PCs.** Stack maps could carry their compile-time PC so that + helpers compare `frame->pc` exactly. The bitmap already makes this O(1), so + the only remaining gain is removing one load and one bit test. +- **Publishable evidence.** The three-engine README matrix (QuickJS, Bun, + quickjs-jit) is pending and is left to the integrator. diff --git a/jit/tests/helpers.rs b/jit/tests/helpers.rs index 76ce898d2..9b485e4d2 100644 --- a/jit/tests/helpers.rs +++ b/jit/tests/helpers.rs @@ -84,6 +84,8 @@ enum InvalidKind { Slot, ConstantIndex, HelperVersion, + PcInsideInstruction, + PcPastEnd, } #[derive(Clone)] @@ -440,6 +442,7 @@ unsafe extern "C" fn helper_entry(frame: *mut qjs::JSJitExecFrame) -> qjs::JSJit let saved_stack_capacity = (*frame).stack_capacity; let saved_stack_top = (*frame).stack_top; let saved_helper_version = (*frame).entry.helper_abi_version; + let saved_pc = (*frame).pc; let status = match *kind { InvalidKind::RuntimeId => { (*frame).runtime_id ^= 1; @@ -492,6 +495,17 @@ unsafe extern "C" fn helper_entry(frame: *mut qjs::JSJitExecFrame) -> qjs::JSJit (*frame).entry.helper_abi_version ^= 1; api.free.expect("FREE helper")(frame, 0, 0) } + // `return 'constant'` starts with a five-byte atom push, + // so offset 1 is inside an instruction. The boundary + // bitmap must reject it exactly like the linear scan. + InvalidKind::PcInsideInstruction => { + (*frame).pc = (*frame).bytecode_start.add(1); + api.free.expect("FREE helper")(frame, 0, 0) + } + InvalidKind::PcPastEnd => { + (*frame).pc = (*frame).bytecode_start.wrapping_add(1 << 20); + api.free.expect("FREE helper")(frame, 0, 0) + } }; (*frame).rt = saved_runtime; (*frame).ctx = saved_context; @@ -502,6 +516,7 @@ unsafe extern "C" fn helper_entry(frame: *mut qjs::JSJitExecFrame) -> qjs::JSJit (*frame).stack_capacity = saved_stack_capacity; (*frame).stack_top = saved_stack_top; (*frame).entry.helper_abi_version = saved_helper_version; + (*frame).pc = saved_pc; let after = *(*frame).arg_buf; untouched.store( status < 0 && before.u.ptr == after.u.ptr && before.tag == after.tag, @@ -1162,6 +1177,8 @@ fn invalid_identity_map_index_and_slot_are_rejected_before_touching_values() { InvalidKind::Slot, InvalidKind::ConstantIndex, InvalidKind::HelperVersion, + InvalidKind::PcInsideInstruction, + InvalidKind::PcPastEnd, ]; for kind in kinds { let runtime = Runtime::new().unwrap(); diff --git a/sys/build_support/patch.rs b/sys/build_support/patch.rs index 41317e2f2..b58747014 100644 --- a/sys/build_support/patch.rs +++ b/sys/build_support/patch.rs @@ -1,7 +1,7 @@ use std::{fs, io, path::Path}; const QUICKJS_BASELINE_FNV64: u64 = 0x3302_116a_b0fc_269c; -const EXPECTED_PATCHES: [(&str, u64); 19] = [ +const EXPECTED_PATCHES: [(&str, u64); 20] = [ ("0001-rquickjs-jit.patch", 0x18d7_40cc_6943_30cf), ("0002-runtime-feedback.patch", 0xe4c1_ed52_ee5d_d469), ("0003-element-layout.patch", 0xa12c_ccd2_7420_e880), @@ -21,6 +21,7 @@ const EXPECTED_PATCHES: [(&str, u64); 19] = [ ("0018-inline-frame-recovery.patch", 0xb971d61edccde229), ("0019-array-feedback.patch", 0x11c3cb13a8297dd4), ("0020-typed-array-guard.patch", 0x5d10_38a3_8bc9_9821), + ("0021-helper-boundary.patch", 0xdf20_0245_013f_eb89), ]; pub(crate) const BASELINE_FILES: [&str; 19] = [ "api-test.c", @@ -60,7 +61,7 @@ const PATCHED_FILE_FINGERPRINTS: [(&str, u64); 21] = [ ("list.h", 0xb337_70f7_b76d_a3d8), ("quickjs-atom.h", 0x30b4_9116_b6a2_aa99), ("quickjs-c-atomics.h", 0x490b_0f29_f631_3fc0), - ("quickjs.c", 0xc535_b5f1_48e7_9b27), + ("quickjs.c", 0x1a55_429e_146b_3206), ("quickjs-jit.h", 0x0188_04ad_21d2_95a7), ("quickjs-jit-helpers.h", 0xa469_e80f_be47_404c), ("quickjs-opcode.h", 0x3d05_cfdf_5cf7_2930), diff --git a/sys/patches/0021-helper-boundary.patch b/sys/patches/0021-helper-boundary.patch new file mode 100644 index 000000000..fad15f3bc --- /dev/null +++ b/sys/patches/0021-helper-boundary.patch @@ -0,0 +1,288 @@ +diff --git a/quickjs.c b/quickjs.c +--- a/quickjs.c ++++ b/quickjs.c +@@ -379,6 +379,11 @@ + struct JSJitInlineState *jit_inline_top; + size_t jit_inline_bytes; + uint32_t jit_inline_depth; ++ /* LIFO storage for inline shadow frames; retained between calls and ++ * released with the runtime. Heap fallback when absent or full. */ ++ uint8_t *jit_inline_arena; ++ size_t jit_inline_arena_size; ++ size_t jit_inline_arena_used; + uint64_t jit_helper_dup_count; + uint64_t jit_helper_free_count; + #ifdef CONFIG_JIT_TEST_SUPPORT +@@ -437,6 +442,7 @@ + uint32_t value_count, argc, argument_slots; + uint32_t call_base, call_operands, continuation_pc; + uint64_t function_id, generation; ++ size_t arena_mark, arena_bytes; /* arena_bytes == 0: heap allocation */ + bool resumed_exception, values_consumed; + } JSJitInlineState; + #endif +@@ -890,6 +896,10 @@ + uint64_t jit_generation; + uint8_t jit_feedback_disabled; + int jit_validated_pc_offset; /* known instruction boundary, initially zero */ ++ /* Lazily built instruction-boundary bitmap, (byte_code_len + 1) bits. ++ * Bit byte_code_len marks a cleanly decoded end. Owned by this bytecode ++ * and freed with it; NULL until the first helper-boundary query. */ ++ uint8_t *jit_pc_boundary_map; + #endif + } JSFunctionBytecode; + +@@ -3501,6 +3511,10 @@ + JS_FreeValueRT(rt, rt->current_exception); + #ifdef CONFIG_JIT_ABI + JS_FreeValueRT(rt, rt->jit_uncatchable_exception); ++ js_free_rt(rt, rt->jit_inline_arena); ++ rt->jit_inline_arena = NULL; ++ rt->jit_inline_arena_size = 0; ++ rt->jit_inline_arena_used = 0; + #endif + + list_for_each_safe(el, el1, &rt->job_list) { +@@ -8463,6 +8477,12 @@ + if (b->byte_code_buf) { + hp->js_func_code_size += b->byte_code_len; + } ++#ifdef CONFIG_JIT_ABI ++ if (b->jit_pc_boundary_map) { ++ memory_used_count++; ++ js_func_size += (b->byte_code_len >> 3) + 1; ++ } ++#endif + memory_used_count++; + js_func_size += b->source_len + 1; + if (b->pc2line_len) { +@@ -18926,16 +18946,13 @@ + *stack_top = scratch; + } + +-static bool qjsjit_valid_pc(JSFunctionBytecode *b, const uint8_t *pc, +- bool allow_end) ++static bool qjsjit_valid_pc_scan(JSFunctionBytecode *b, const uint8_t *pc, ++ bool allow_end) + { + uintptr_t start = (uintptr_t)b->byte_code_buf; +- uintptr_t end = start + (size_t)b->byte_code_len; + uintptr_t value = (uintptr_t)pc; + int offset = 0; + +- if (value < start || (allow_end ? value > end : value >= end)) +- return false; + /* Published bytecode instruction lengths are immutable. Reuse only a + proven boundary at or before this query; backward branches rescan. */ + if (value - start >= (uintptr_t)b->jit_validated_pc_offset) +@@ -18959,6 +18976,61 @@ + return allow_end && pc == b->byte_code_buf + b->byte_code_len; + } + ++/* Decode the instruction stream once with exactly the scan's rules. Offsets ++ * after an undecodable instruction stay clear, so every query answers as the ++ * linear scan would. Allocation failure leaves the map absent (scan ++ * fallback); it never raises an exception or changes JS-visible state. */ ++static uint8_t *qjsjit_build_pc_boundary_map(JSFunctionBytecode *b) ++{ ++ uint8_t *map; ++ int offset = 0; ++ ++ if (!b->realm || b->byte_code_len < 0) ++ return NULL; ++ map = js_mallocz_rt(b->realm->rt, ((size_t)b->byte_code_len >> 3) + 1); ++ if (!map) ++ return NULL; ++ for (;;) { ++ const JSJitOpcodeInfo *info; ++ uint8_t opcode; ++ ++ /* Mark first: reaching byte_code_len means an exact decode. */ ++ map[(size_t)offset >> 3] |= (uint8_t)(1u << (offset & 7)); ++ if (offset >= b->byte_code_len) ++ break; ++ opcode = b->byte_code_buf[offset]; ++ if (opcode >= countof(qjsjit_opcode_table)) ++ break; ++ info = &qjsjit_opcode_table[opcode]; ++ if (!info->size || info->size > b->byte_code_len - offset) ++ break; ++ offset += info->size; ++ } ++ b->jit_pc_boundary_map = map; ++ return map; ++} ++ ++static inline bool qjsjit_valid_pc(JSFunctionBytecode *b, const uint8_t *pc, ++ bool allow_end) ++{ ++ uintptr_t start = (uintptr_t)b->byte_code_buf; ++ uintptr_t end = start + (size_t)b->byte_code_len; ++ uintptr_t value = (uintptr_t)pc; ++ const uint8_t *map; ++ size_t offset; ++ ++ if (value < start || (allow_end ? value > end : value >= end)) ++ return false; ++ map = b->jit_pc_boundary_map; ++ if (unlikely(!map)) { ++ map = qjsjit_build_pc_boundary_map(b); ++ if (!map) ++ return qjsjit_valid_pc_scan(b, pc, allow_end); ++ } ++ offset = value - start; ++ return (map[offset >> 3] >> (offset & 7)) & 1; ++} ++ + static bool qjsjit_is_uncatchable_sentinel(JSRuntime *rt, JSValueConst value) + { + return JS_VALUE_GET_TAG(value) == JS_TAG_OBJECT && +@@ -19003,8 +19075,10 @@ + * a forged identity/cookie/map/index can never read, duplicate or free the + * value used by a caller to detect accidental touching. + */ +-static JSJitHelperStatus qjsjit_reject_helper(JSJitExecFrame *frame, +- const char *message) ++/* Rejection is cold; keeping it out of line keeps the validated fast path ++ * free of spills and stack-protector setup. */ ++static no_inline JSJitHelperStatus qjsjit_reject_helper(JSJitExecFrame *frame, ++ const char *message) + { + JSValue result = JS_UNDEFINED; + +@@ -19059,8 +19133,10 @@ + return qjsjit_reject_helper(frame, "invalid JIT helper entry"); + if (has_stack_map && stack_map_id >= frame->entry.stack_map_count) + return qjsjit_reject_helper(frame, "invalid JIT helper stack map"); ++ /* Same predicate as JS_HasException(ctx), without an out-of-line call. */ + if (!JS_IsUndefined(frame->result) || +- (validation != QJSJIT_VALIDATE_RESUME && JS_HasException(ctx))) ++ (validation != QJSJIT_VALIDATE_RESUME && ++ !JS_IsUninitialized(rt->current_exception))) + return qjsjit_reject_helper(frame, "invalid JIT helper state"); + + sf = rt->current_stack_frame; +@@ -20113,6 +20189,72 @@ + rt->jit_active_helper_abi_version = view ? view->entry.helper_abi_version : 0; + } + ++#define QJSJIT_INLINE_ARENA_ALIGN 16 ++#define QJSJIT_INLINE_ARENA_MIN_SIZE (32 * 1024) ++ ++#if defined(__SANITIZE_ADDRESS__) ++#define QJSJIT_INLINE_ARENA_ENABLED 0 ++#elif defined(__has_feature) ++#if __has_feature(address_sanitizer) || __has_feature(memory_sanitizer) ++#define QJSJIT_INLINE_ARENA_ENABLED 0 ++#endif ++#endif ++#ifndef QJSJIT_INLINE_ARENA_ENABLED ++#define QJSJIT_INLINE_ARENA_ENABLED 1 ++#endif ++ ++/* Reserve zeroed shadow-frame storage. Frames are strictly nested, so the ++ * arena is a bump stack: a frame's storage never moves while it is live ++ * (captured JSVarRef pointers rely on this). The arena only grows while it ++ * is empty; a frame that does not fit uses the heap. Failure is a pre-effect ++ * miss, exactly like the previous per-call allocation. */ ++static JSJitInlineState *qjsjit_inline_alloc(JSRuntime *rt, size_t bytes) ++{ ++ JSJitInlineState *state; ++ size_t aligned, used, size; ++ ++ aligned = (bytes + QJSJIT_INLINE_ARENA_ALIGN - 1) & ++ ~(size_t)(QJSJIT_INLINE_ARENA_ALIGN - 1); ++ used = rt->jit_inline_arena_used; ++ if (QJSJIT_INLINE_ARENA_ENABLED && aligned >= bytes && ++ aligned <= JS_JIT_INLINE_MAX_BYTES + QJSJIT_INLINE_ARENA_ALIGN) { ++ if (used == 0 && aligned > rt->jit_inline_arena_size) { ++ size = QJSJIT_INLINE_ARENA_MIN_SIZE; ++ while (size < aligned) ++ size <<= 1; ++ js_free_rt(rt, rt->jit_inline_arena); ++ rt->jit_inline_arena_size = 0; ++ rt->jit_inline_arena = js_malloc_rt(rt, size); ++ if (rt->jit_inline_arena) ++ rt->jit_inline_arena_size = size; ++ } ++ if (rt->jit_inline_arena && ++ aligned <= rt->jit_inline_arena_size - used) { ++ state = (JSJitInlineState *)(rt->jit_inline_arena + used); ++ memset(state, 0, bytes); ++ state->arena_mark = used; ++ state->arena_bytes = aligned; ++ rt->jit_inline_arena_used = used + aligned; ++ return state; ++ } ++ } ++ return js_mallocz_rt(rt, bytes); ++} ++ ++/* Callers release after their depth decrement, so depth zero proves that no ++ * reserved frame (published or not) still uses the arena. */ ++static void qjsjit_inline_release(JSRuntime *rt, JSJitInlineState *state) ++{ ++ if (!state->arena_bytes) { ++ js_free_rt(rt, state); ++ return; ++ } ++ if (state->arena_mark + state->arena_bytes == rt->jit_inline_arena_used) ++ rt->jit_inline_arena_used = state->arena_mark; ++ if (rt->jit_inline_depth == 0) ++ rt->jit_inline_arena_used = 0; ++} ++ + /* Free only runtime-owned storage, never a pointer supplied in the mutable + * public view. Interpreter completion has already consumed its local range. */ + static void qjsjit_inline_destroy(JSRuntime *rt, JSJitInlineState *state) +@@ -20132,7 +20274,7 @@ + JS_FreeValueRT(rt, state->sf.cur_func); + JS_FreeValueRT(rt, state->this_value); + JS_FreeValueRT(rt, state->new_target); +- js_free_rt(rt, state); ++ qjsjit_inline_release(rt, state); + } + + /* Native code must never leave a live shadow frame behind. Reject without +@@ -20172,10 +20314,10 @@ + rt = caller->rt; + if (call_kind > JS_JIT_INLINE_CALL_METHOD || argc > UINT16_MAX) + return JS_JIT_HELPER_GUARD_MISS; ++ /* Frame validation proved caller->pc is an instruction boundary; the ++ * continuation is proven below as the next boundary after that call. */ + if (call_pc >= (uint32_t)caller_b->byte_code_len || + continuation_pc >= (uint32_t)caller_b->byte_code_len || +- !qjsjit_valid_pc(caller_b, caller_b->byte_code_buf + call_pc, false) || +- !qjsjit_valid_pc(caller_b, caller_b->byte_code_buf + continuation_pc, false) || + caller->pc != caller_b->byte_code_buf + call_pc) + return qjsjit_reject_helper(caller, "invalid inline call PC"); + opcode = caller_b->byte_code_buf[call_pc]; +@@ -20224,7 +20366,7 @@ + return JS_JIT_HELPER_GUARD_MISS; + /* Unlike js_malloc(ctx), this does not manufacture an OOM exception for + * an optional optimization. Quota/allocation failure is pre-effect miss. */ +- state = js_mallocz_rt(rt, bytes); ++ state = qjsjit_inline_alloc(rt, bytes); + if (!state) + return JS_JIT_HELPER_GUARD_MISS; + /* Every decline happens before this exactly-once entry checkpoint: a +@@ -20306,7 +20448,7 @@ + poll_failed: + rt->jit_inline_depth--; + rt->jit_inline_bytes -= bytes; +- js_free_rt(rt, state); ++ qjsjit_inline_release(rt, state); + return JS_JIT_HELPER_EXCEPTION; + } + +@@ -39998,6 +40140,10 @@ + JS_FreeAtomRT(rt, b->filename); + js_free_rt(rt, b->pc2line_buf); + js_free_rt(rt, b->source); ++#ifdef CONFIG_JIT_ABI ++ js_free_rt(rt, b->jit_pc_boundary_map); ++ b->jit_pc_boundary_map = NULL; ++#endif + + remove_gc_object(&b->header); + if (rt->gc_phase == JS_GC_PHASE_REMOVE_CYCLES && b->header.ref_count != 0) { diff --git a/sys/patches/README.md b/sys/patches/README.md index 931cf9bd3..de4968666 100644 --- a/sys/patches/README.md +++ b/sys/patches/README.md @@ -60,6 +60,20 @@ data pointer before returning `{ data, count, mode }`. The table advertises zero effects and the receiver is passed by pointer so generated calls do not depend on platform-specific aggregate argument classification. +`0021-helper-boundary.patch` makes helper-boundary PC validation O(1). Each +bytecode lazily builds an instruction-boundary bitmap (`byte_code_len / 8 + 1` +bytes, runtime-allocated, freed with the bytecode and reported in +`JS_ComputeMemoryUsage`) using exactly the linear scan's decoding rules; if +the allocation fails the previous validated-PC scan answers instead. Helper +frame validation keeps every identity, ABI and stack check but inlines the +pending-exception test and moves rejection out of line. +`JS_JitInlineEnter` no longer rescans the call and continuation PCs (the frame +PC is already a proven boundary and the continuation is proven as the next +one) and reserves shadow frames from a runtime-owned LIFO arena (grown only +while empty, at most the 1 MiB inline budget plus alignment, released with the +runtime) with heap fallback; sanitizer builds always use the heap. No public +ABI or struct layout visible to the bindings changes. + The build accepts only the patch names and byte digests listed in `build_support/patch.rs`, then verifies the complete patched source manifest. Keeping the baseline and patch separate makes diff --git a/sys/tests/inline_recovery.rs b/sys/tests/inline_recovery.rs index bfa825e03..a2f2678a0 100644 --- a/sys/tests/inline_recovery.rs +++ b/sys/tests/inline_recovery.rs @@ -278,13 +278,26 @@ static JSJitExit native(JSJitExecFrame *root) } else if (scenario == 16) { double value; size_t nested_bytes = root->rt->jit_inline_bytes; + size_t nested_arena = root->rt->jit_inline_arena_used; assert(root->rt->jit_inline_depth == 2 && root->rt->jit_inline_top); assert(&root->rt->jit_inline_top->view == leaf); + /* Nested shadow frames are stacked in the runtime arena. */ + assert(!QJSJIT_INLINE_ARENA_ENABLED || + (root->rt->jit_inline_top->arena_bytes && + root->rt->jit_inline_top->previous->arena_bytes && + root->rt->jit_inline_top->arena_mark == + root->rt->jit_inline_top->previous->arena_mark + + root->rt->jit_inline_top->previous->arena_bytes && + nested_arena == root->rt->jit_inline_top->arena_mark + + root->rt->jit_inline_top->arena_bytes)); leaf->stack_base[0] = JS_NewFloat64(leaf->ctx, 2147483648.0); leaf->stack_top = leaf->stack_base + 1; assert(JS_JitInlineLeave(leaf, callee_snapshot->arg_count + callee_snapshot->local_count) == JS_JIT_HELPER_OK); assert(root->rt->jit_inline_depth == 1 && root->rt->jit_inline_bytes < nested_bytes); assert(root->rt->jit_inline_top && &root->rt->jit_inline_top->view == middle); + assert(!QJSJIT_INLINE_ARENA_ENABLED || + root->rt->jit_inline_arena_used == + root->rt->jit_inline_top->arena_mark + root->rt->jit_inline_top->arena_bytes); assert(root->rt->jit_active_frame == middle && middle->stack_top == middle->stack_base + 1); assert(middle->pc == middle->bytecode_start + find_opcode(middle_snapshot, "call1", 0) + 1); assert(JS_ToFloat64(middle->ctx, &value, middle->stack_base[0]) == 0 && value == 2147483648.0); @@ -292,6 +305,7 @@ static JSJitExit native(JSJitExecFrame *root) middle->stack_base[0] = JS_NewFloat64(middle->ctx, 2147483668.0); assert(JS_JitInlineLeave(middle, middle_snapshot->arg_count + middle_snapshot->local_count) == JS_JIT_HELPER_OK); assert(!root->rt->jit_inline_top && root->rt->jit_inline_depth == 0 && root->rt->jit_inline_bytes == 0); + assert(root->rt->jit_inline_arena_used == 0); assert(root->rt->jit_active_frame == root); } else if (scenario == 17) { unsigned throw_pc = find_opcode(callee_snapshot, "throw", 0); @@ -496,6 +510,9 @@ cleanup: } if (mode == 9) JS_FreeValue(ctx, args[0]); assert(JS_SetJitBackend(rt, NULL, NULL) == JS_JIT_BACKEND_OK); + /* Every shadow frame, published or declined, returned its storage. */ + assert(!rt->jit_inline_top && rt->jit_inline_depth == 0 && + rt->jit_inline_bytes == 0 && rt->jit_inline_arena_used == 0); JS_FreeContext(ctx); JS_FreeRuntime(rt); } diff --git a/sys/tests/jit_patch.rs b/sys/tests/jit_patch.rs index f36861117..5314a2ec7 100644 --- a/sys/tests/jit_patch.rs +++ b/sys/tests/jit_patch.rs @@ -56,6 +56,7 @@ fn copy_patches(destination: &std::path::Path) { "0018-inline-frame-recovery.patch", "0019-array-feedback.patch", "0020-typed-array-guard.patch", + "0021-helper-boundary.patch", ] { fs::copy(source.join(patch), destination.join(patch)).unwrap(); } From 9bcf6f4580837233304437d048b6c6c0f22f83a2 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 16:38:51 +0800 Subject: [PATCH 03/54] perf(jit): link effectful field-writing callees transactionally Extend the target-only linked leaf ABI to bounded, acyclic callees that read and write guarded own data fields of borrowed HeapRef arguments (e.g. generic-call-fallback's incrementAndRecord). Every speculation (Int32 overflow, shape identity/generation, field tag) runs before any write; field stores are buffered in SSA and committed in program order only at return, so status 1 still means an untouched frame and heap and the caller's exact generic-CALL recovery is unchanged. A direct target now always wins over frame inlining at its call PC. Co-Authored-By: Claude Opus 5.5 (1M context) --- jit/src/compiler/baseline.rs | 2 + jit/src/compiler/optimized.rs | 28 +- jit/src/compiler/tagged_call_link.rs | 1023 ++++++++++++++++++++++---- jit/src/runtime/coordinator.rs | 10 +- jit/tests/tagged_call_link.rs | 197 +++++ 5 files changed, 1123 insertions(+), 137 deletions(-) diff --git a/jit/src/compiler/baseline.rs b/jit/src/compiler/baseline.rs index 00e7cfb64..a072841f8 100644 --- a/jit/src/compiler/baseline.rs +++ b/jit/src/compiler/baseline.rs @@ -857,6 +857,7 @@ impl Compiler for BaselineCompiler { &self.isa, request.snapshot(), signature, + Some(request.feedback()), None, ) .ok() @@ -910,6 +911,7 @@ impl Compiler for BaselineCompiler { &self.isa, request.snapshot(), signature, + Some(request.feedback()), Some(control), ) .ok() diff --git a/jit/src/compiler/optimized.rs b/jit/src/compiler/optimized.rs index afc40c273..26ad87311 100644 --- a/jit/src/compiler/optimized.rs +++ b/jit/src/compiler/optimized.rs @@ -4157,6 +4157,7 @@ pub(crate) fn lower_direct_call_machine( isa: &cranelift_codegen::isa::OwnedTargetIsa, function: &VerifiedFunction, signature: &crate::runtime::BoundedSpecializationSignature, + feedback: Option<&crate::runtime::FeedbackSnapshot>, control: Option<&CompileControl>, ) -> Result { use crate::runtime::FeedbackRepresentation; @@ -4175,7 +4176,7 @@ pub(crate) fn lower_direct_call_machine( .contains(&FeedbackRepresentation::HeapRef) { return super::tagged_call_link::lower_target_only_linked_leaf( - isa, function, signature, control, + isa, function, signature, feedback, control, ); } if signature @@ -8814,7 +8815,7 @@ impl Tier2Compiler { let signature = feedback .bounded_specialization(key) .ok_or(CompileFailure::InvalidArtifact)?; - lower_direct_call_machine(&self.isa, function, &signature, None) + lower_direct_call_machine(&self.isa, function, &signature, Some(feedback), None) .map(|code| code.clif().to_owned()) } @@ -8891,9 +8892,10 @@ impl Tier2Compiler { { return Err(CompileFailure::InvalidArtifact); } - let published = lower_direct_call_machine(&self.isa, function, &signature, None)? - .publish() - .map_err(|_| CompileFailure::InvalidArtifact)?; + let published = + lower_direct_call_machine(&self.isa, function, &signature, Some(feedback), None)? + .publish() + .map_err(|_| CompileFailure::InvalidArtifact)?; // The match above proves the exact arity and representation of the // scalar-only ABI before converting the executable entry. let mut output = 0i32; @@ -9375,7 +9377,13 @@ impl Compiler for Tier2Compiler { tier2_stage( key, Tier2CompileStage::DirectLower, - lower_direct_call_machine(&self.isa, request.snapshot(), signature, None), + lower_direct_call_machine( + &self.isa, + request.snapshot(), + signature, + Some(request.feedback()), + None, + ), ) .ok() }); @@ -9597,7 +9605,13 @@ impl Compiler for Tier2Compiler { tier2_stage( key, Tier2CompileStage::DirectLower, - lower_direct_call_machine(&self.isa, request.snapshot(), signature, Some(control)), + lower_direct_call_machine( + &self.isa, + request.snapshot(), + signature, + Some(request.feedback()), + Some(control), + ), ) .ok() }); diff --git a/jit/src/compiler/tagged_call_link.rs b/jit/src/compiler/tagged_call_link.rs index 62873fa68..b8ef519cb 100644 --- a/jit/src/compiler/tagged_call_link.rs +++ b/jit/src/compiler/tagged_call_link.rs @@ -1,17 +1,41 @@ //! Target-only linked leaf entry for heterogeneous, guarded arguments. +//! +//! The entry is a transaction over borrowed arguments. Every speculation +//! (arithmetic overflow, property shape/generation/tag guards) is checked +//! before the first observable write. Guarded own-field stores are buffered in +//! SSA and committed in program order only at `return`, together with the +//! scalar result. Status `1` therefore always means "nothing happened": the +//! caller executes its original generic CALL with an untouched frame and heap. +//! This is the JSC/V8 "exit before the first effect" rule applied to a whole, +//! bounded, acyclic callee, so no callee frame ever needs to be materialized. use crate::{ + abi::PropertyLayout, bytecode::VerifiedFunction, compiler::{CompileControl, CompileFailure}, - ir::{BaselineIr, BinaryOp, IrOp}, - runtime::{BoundedSpecializationSignature, FeedbackRepresentation}, + ir::{eligible_property_observation, BaselineIr, BinaryOp, IrOp, StackOp, UnaryOp}, + runtime::{ + BoundedSpecializationSignature, FeedbackRepresentation, FeedbackSnapshot, ObservedType, + ShapeFeedbackState, ShapeObservation, + }, }; use cranelift_codegen::{ - ir::{types, AbiParam, Function, InstBuilder, MemFlags, Signature, Value}, + ir::{ + condcodes::IntCC, types, AbiParam, Block, Function, InstBuilder, MemFlags, Signature, Value, + }, isa::OwnedTargetIsa, }; -use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext}; +use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Variable}; use rquickjs_core::qjs; +use std::collections::BTreeMap; + +/// Bounds shared with the bounded inline planner. Admission happens before +/// BaselineIr translation or Cranelift block creation. +const MAX_INSTRUCTIONS: usize = 128; +const MAX_SLOTS: usize = 64; +const MAX_RETAINED_BYTES: usize = 16 * 1024; +const MAX_BLOCKS: usize = 16; +const MAX_BUFFERED_STORES: usize = 8; #[derive(Clone, Copy)] struct TypedValue { @@ -19,16 +43,175 @@ struct TypedValue { representation: FeedbackRepresentation, } -/// Build the first, deliberately narrow stage of a JSC-style linked call. +/// One guarded own-field store, buffered until the successful `return`. +struct BufferedStore { + pc: u32, + dirty: Variable, + address: Variable, + payload: Variable, + tag: i32, +} + +fn observed_representation(value: ObservedType) -> Option { + match value { + ObservedType::Int32 => Some(FeedbackRepresentation::Int32), + ObservedType::Float64 => Some(FeedbackRepresentation::Float64), + ObservedType::Bool => Some(FeedbackRepresentation::Bool), + _ => None, + } +} + +fn representation_tag(representation: FeedbackRepresentation) -> Result { + match representation { + FeedbackRepresentation::Int32 => Ok(qjs::JS_TAG_INT), + FeedbackRepresentation::Float64 => Ok(qjs::JS_TAG_FLOAT64), + FeedbackRepresentation::Bool => Ok(qjs::JS_TAG_BOOL), + FeedbackRepresentation::HeapRef => Err(CompileFailure::UnsupportedOpcode), + } +} + +fn scalar_type(representation: FeedbackRepresentation, pointer: types::Type) -> types::Type { + match representation { + FeedbackRepresentation::Int32 | FeedbackRepresentation::Bool => types::I32, + FeedbackRepresentation::Float64 => types::F64, + FeedbackRepresentation::HeapRef => pointer, + } +} + +/// A property site is admitted only for one stable own writable data field +/// holding a representable primitive. Anything else keeps the generic CALL. +fn linked_property( + feedback: Option<&FeedbackSnapshot>, + pc: u32, +) -> Result<(ShapeObservation, FeedbackRepresentation), CompileFailure> { + let site = feedback + .and_then(|feedback| feedback.property_at(pc)) + .ok_or(CompileFailure::UnsupportedOpcode)?; + let [observation] = site.observations() else { + return Err(CompileFailure::UnsupportedOpcode); + }; + if site.state() != ShapeFeedbackState::Monomorphic + || !eligible_property_observation(*observation) + { + return Err(CompileFailure::UnsupportedOpcode); + } + let representation = + observed_representation(observation.value()).ok_or(CompileFailure::UnsupportedOpcode)?; + Ok((*observation, representation)) +} + +/// Guards `object` (an already tag-checked, borrowed object pointer) against +/// the observed shape identity and generation, then the field's current tag. +/// Returns the field address. No memory is written on any path. +fn emit_guarded_field( + builder: &mut FunctionBuilder<'_>, + layout: PropertyLayout, + pointer_type: types::Type, + object: Value, + observation: ShapeObservation, + expected_tag: i32, + miss: Block, +) -> Result { + let offset = observation + .offset() + .checked_mul(16) + .map(i64::from) + .filter(|offset| *offset <= i64::from(i32::MAX) - 8) + .ok_or(CompileFailure::ResourceLimit)?; + let shape = builder.ins().load( + pointer_type, + MemFlags::new(), + object, + layout.object_shape_offset, + ); + let same_shape = + builder + .ins() + .icmp_imm(IntCC::Equal, shape, observation.shape().identity() as i64); + let generation = builder.ins().load( + types::I64, + MemFlags::new(), + shape, + layout.shape_generation_offset, + ); + let same_generation = builder.ins().icmp_imm( + IntCC::Equal, + generation, + observation.shape().generation() as i64, + ); + let shape_ok = builder.ins().band(same_shape, same_generation); + let field = builder.create_block(); + builder.ins().brif(shape_ok, field, &[], miss, &[]); + builder.switch_to_block(field); + let properties = builder.ins().load( + pointer_type, + MemFlags::new(), + object, + layout.object_properties_offset, + ); + let address = builder.ins().iadd_imm(properties, offset); + let tag = builder.ins().load(types::I64, MemFlags::new(), address, 8); + let tag_ok = builder + .ins() + .icmp_imm(IntCC::Equal, tag, i64::from(expected_tag)); + let checked = builder.create_block(); + builder.ins().brif(tag_ok, checked, &[], miss, &[]); + builder.switch_to_block(checked); + Ok(address) +} + +fn emit_int32_arithmetic( + builder: &mut FunctionBuilder<'_>, + operation: BinaryOp, + lhs: Value, + rhs: Value, + miss: Block, +) -> Value { + let (value, overflow) = match operation { + BinaryOp::Add => builder.ins().sadd_overflow(lhs, rhs), + BinaryOp::Sub => builder.ins().ssub_overflow(lhs, rhs), + BinaryOp::Mul => builder.ins().smul_overflow(lhs, rhs), + _ => unreachable!("caller admits only add/sub/mul"), + }; + let must_miss = if operation == BinaryOp::Mul { + let zero = builder.ins().icmp_imm(IntCC::Equal, value, 0); + let lhs_negative = builder.ins().icmp_imm(IntCC::SignedLessThan, lhs, 0); + let rhs_negative = builder.ins().icmp_imm(IntCC::SignedLessThan, rhs, 0); + let opposite_signs = builder.ins().bxor(lhs_negative, rhs_negative); + let negative_zero = builder.ins().band(zero, opposite_signs); + builder.ins().bor(overflow, negative_zero) + } else { + overflow + }; + let ok = builder.create_block(); + builder.ins().brif(must_miss, miss, &[], ok, &[]); + builder.switch_to_block(ok); + value +} + +fn truth(builder: &mut FunctionBuilder<'_>, value: TypedValue) -> Result { + match value.representation { + FeedbackRepresentation::Int32 | FeedbackRepresentation::Bool => { + Ok(builder.ins().icmp_imm(IntCC::NotEqual, value.value, 0)) + } + FeedbackRepresentation::Float64 | FeedbackRepresentation::HeapRef => { + Err(CompileFailure::UnsupportedOpcode) + } + } +} + +/// Build a JSC-style linked call entry over guarded, borrowed arguments. /// /// Target identity and argument-tag guards live at the call site. This entry -/// consumes only those guarded, borrowed values and performs no allocation, -/// ownership transfer, helper call, or frame recovery. Status `1` means the -/// caller must execute the original generic CALL with its untouched frame. +/// performs no allocation, ownership transfer, helper call, or frame recovery. +/// Status `0` stores the scalar result after committing buffered field stores; +/// status `1` means the caller must execute the original generic CALL with its +/// untouched frame, and guarantees that no heap write happened. pub(crate) fn lower_target_only_linked_leaf( isa: &OwnedTargetIsa, function: &VerifiedFunction, signature: &BoundedSpecializationSignature, + feedback: Option<&FeedbackSnapshot>, control: Option<&CompileControl>, ) -> Result { let snapshot = function.snapshot(); @@ -39,14 +222,17 @@ pub(crate) fn lower_target_only_linked_leaf( // Match the existing bounded inline planner. This admission happens before // BaselineIr translation or Cranelift block creation, so rejected input // cannot turn a stable call-link observation into unbounded compilation. - if function.instructions().len() > 128 || slots > 64 || snapshot.retained_bytes() > 16 * 1024 { + if function.instructions().len() > MAX_INSTRUCTIONS + || slots > MAX_SLOTS + || snapshot.retained_bytes() > MAX_RETAINED_BYTES + || function.control_flow_graph().blocks().len() > MAX_BLOCKS + { return Err(CompileFailure::ResourceLimit); } if signature.function().id != snapshot.function_id() || signature.function().generation != snapshot.generation() || signature.arity() != usize::from(snapshot.arg_count()) || !snapshot.exception_map().is_empty() - || function.control_flow_graph().blocks().len() != 1 || !matches!( signature.result(), FeedbackRepresentation::Int32 @@ -64,19 +250,78 @@ pub(crate) fn lower_target_only_linked_leaf( } let ir = BaselineIr::translate(function)?; - if ir.blocks.len() != 1 { + if ir.blocks.is_empty() || ir.blocks.len() > MAX_BLOCKS { return Err(CompileFailure::UnsupportedOpcode); } - let mut abi = Signature::new(isa.default_call_conv()); - abi.params.push(AbiParam::new(isa.pointer_type())); - abi.params - .extend(signature.arguments().iter().map(|representation| { - AbiParam::new(match representation { - FeedbackRepresentation::Int32 | FeedbackRepresentation::Bool => types::I32, - FeedbackRepresentation::Float64 => types::F64, - FeedbackRepresentation::HeapRef => isa.pointer_type(), + // Structural prepass: forward-only control flow with an empty operand + // stack at every block boundary, and no field read after any field store. + // In a forward-only CFG every path orders instructions by PC, so a read + // with a PC above the first store PC is the only way to observe a buffered + // write; rejecting it keeps the deferred-commit transaction exact. + let block_starts = ir + .blocks + .iter() + .map(|block| block.start_pc) + .collect::>(); + if block_starts.windows(2).any(|pair| pair[0] >= pair[1]) { + return Err(CompileFailure::InvalidArtifact); + } + let mut first_store = None::; + let mut store_pcs = Vec::new(); + for block in &ir.blocks { + if block.stack_depth != 0 { + return Err(CompileFailure::UnsupportedOpcode); + } + for instruction in &block.instructions { + match instruction.op { + IrOp::Jump(target) | IrOp::Branch { target, .. } => { + if target <= instruction.pc || block_starts.binary_search(&target).is_err() { + return Err(CompileFailure::UnsupportedOpcode); + } + } + IrOp::SetProperty(_) => { + first_store.get_or_insert(instruction.pc); + store_pcs.push(instruction.pc); + } + IrOp::GetProperty(_) | IrOp::GetPropertyKeep(_) + if first_store.is_some_and(|store| store < instruction.pc) => + { + return Err(CompileFailure::UnsupportedOpcode); + } + _ => {} + } + } + } + if store_pcs.len() > MAX_BUFFERED_STORES { + return Err(CompileFailure::ResourceLimit); + } + let layout = if store_pcs.is_empty() + && !ir.blocks.iter().any(|block| { + block.instructions.iter().any(|instruction| { + matches!( + instruction.op, + IrOp::GetProperty(_) | IrOp::GetPropertyKeep(_) + ) }) - })); + }) { + None + } else { + Some( + crate::abi::AbiInfo::linked() + .map_err(|_| CompileFailure::InvalidArtifact)? + .property_layout(), + ) + }; + let pointer_type = isa.pointer_type(); + + let mut abi = Signature::new(isa.default_call_conv()); + abi.params.push(AbiParam::new(pointer_type)); + abi.params.extend( + signature + .arguments() + .iter() + .map(|representation| AbiParam::new(scalar_type(*representation, pointer_type))), + ); abi.returns.push(AbiParam::new(types::I32)); let mut function_ir = Function::with_name_signature(Default::default(), abi); @@ -88,118 +333,343 @@ pub(crate) fn lower_target_only_linked_leaf( builder.switch_to_block(entry); let output = builder.block_params(entry)[0]; let arguments = builder.block_params(entry)[1..].to_vec(); - let mut stack = Vec::new(); - let mut returned = false; + let miss = builder.create_block(); + builder.set_cold_block(miss); - for instruction in &ir.blocks[0].instructions { - if returned { - return Err(CompileFailure::InvalidArtifact); - } - match &instruction.op { - IrOp::Poll { .. } | IrOp::OsrLabel { .. } | IrOp::Nop => {} - IrOp::Push(tagged) if tagged.tag == i64::from(qjs::JS_TAG_INT) => { - stack.push(TypedValue { - value: builder - .ins() - .iconst(types::I32, tagged.payload as i32 as i64), - representation: FeedbackRepresentation::Int32, - }); - } - IrOp::Push(tagged) if tagged.tag == i64::from(qjs::JS_TAG_FLOAT64) => { - stack.push(TypedValue { - value: builder.ins().f64const(f64::from_bits(tagged.payload)), - representation: FeedbackRepresentation::Float64, - }); - } - IrOp::GetArgument(index) => { - let index = usize::from(*index); - stack.push(TypedValue { - value: *arguments - .get(index) - .ok_or(CompileFailure::InvalidArtifact)?, - representation: *signature - .arguments() - .get(index) - .ok_or(CompileFailure::InvalidArtifact)?, - }); - } - IrOp::Drop => { - stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + let mut stores = Vec::with_capacity(store_pcs.len()); + let mut next_variable = 0u32; + for &pc in &store_pcs { + let (_, representation) = linked_property(feedback, pc)?; + let store = BufferedStore { + pc, + dirty: Variable::from_u32(next_variable), + address: Variable::from_u32(next_variable + 1), + payload: Variable::from_u32(next_variable + 2), + tag: representation_tag(representation)?, + }; + next_variable += 3; + builder.declare_var(store.dirty, types::I8); + builder.declare_var(store.address, pointer_type); + builder.declare_var(store.payload, types::I64); + let clean = builder.ins().iconst(types::I8, 0); + builder.def_var(store.dirty, clean); + let null = builder.ins().iconst(pointer_type, 0); + builder.def_var(store.address, null); + let zero = builder.ins().iconst(types::I64, 0); + builder.def_var(store.payload, zero); + stores.push(store); + } + + let blocks = ir + .blocks + .iter() + .map(|block| (block.start_pc, builder.create_block())) + .collect::>(); + builder.ins().jump(blocks[&ir.blocks[0].start_pc], &[]); + + for (block_index, block) in ir.blocks.iter().enumerate() { + builder.switch_to_block(blocks[&block.start_pc]); + let mut stack = Vec::::new(); + let mut terminated = false; + for instruction in &block.instructions { + if terminated { + return Err(CompileFailure::InvalidArtifact); } - IrOp::Binary(operation @ (BinaryOp::Add | BinaryOp::Sub | BinaryOp::Mul)) => { - let rhs = stack.pop().ok_or(CompileFailure::InvalidArtifact)?; - let lhs = stack.pop().ok_or(CompileFailure::InvalidArtifact)?; - if lhs.representation != rhs.representation - || !matches!( - lhs.representation, - FeedbackRepresentation::Int32 | FeedbackRepresentation::Float64 - ) - { - return Err(CompileFailure::UnsupportedOpcode); + match &instruction.op { + IrOp::Poll { .. } | IrOp::OsrLabel { .. } | IrOp::Nop => {} + IrOp::Push(tagged) if tagged.tag == i64::from(qjs::JS_TAG_INT) => { + stack.push(TypedValue { + value: builder + .ins() + .iconst(types::I32, tagged.payload as i32 as i64), + representation: FeedbackRepresentation::Int32, + }); + } + IrOp::Push(tagged) if tagged.tag == i64::from(qjs::JS_TAG_BOOL) => { + stack.push(TypedValue { + value: builder + .ins() + .iconst(types::I32, i64::from(tagged.payload != 0)), + representation: FeedbackRepresentation::Bool, + }); + } + IrOp::Push(tagged) if tagged.tag == i64::from(qjs::JS_TAG_FLOAT64) => { + stack.push(TypedValue { + value: builder.ins().f64const(f64::from_bits(tagged.payload)), + representation: FeedbackRepresentation::Float64, + }); + } + IrOp::GetArgument(index) => { + let index = usize::from(*index); + stack.push(TypedValue { + value: *arguments + .get(index) + .ok_or(CompileFailure::InvalidArtifact)?, + representation: *signature + .arguments() + .get(index) + .ok_or(CompileFailure::InvalidArtifact)?, + }); + } + IrOp::Drop => { + stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + } + // Typed values are borrowed or immediate: stack shuffles + // carry no ownership and need no DUP/FREE. + IrOp::Stack(StackOp::Dup) => { + let top = *stack.last().ok_or(CompileFailure::InvalidArtifact)?; + stack.push(top); + } + IrOp::Stack(StackOp::Swap) => { + let len = stack.len(); + if len < 2 { + return Err(CompileFailure::InvalidArtifact); + } + stack.swap(len - 1, len - 2); } - let value = match lhs.representation { - FeedbackRepresentation::Int32 => { - let (value, overflow) = match operation { - BinaryOp::Add => builder.ins().sadd_overflow(lhs.value, rhs.value), - BinaryOp::Sub => builder.ins().ssub_overflow(lhs.value, rhs.value), - BinaryOp::Mul => builder.ins().smul_overflow(lhs.value, rhs.value), + IrOp::Binary(operation @ (BinaryOp::Add | BinaryOp::Sub | BinaryOp::Mul)) => { + let rhs = stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + let lhs = stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + if lhs.representation != rhs.representation { + return Err(CompileFailure::UnsupportedOpcode); + } + let value = match lhs.representation { + FeedbackRepresentation::Int32 => emit_int32_arithmetic( + &mut builder, + *operation, + lhs.value, + rhs.value, + miss, + ), + FeedbackRepresentation::Float64 => match operation { + BinaryOp::Add => builder.ins().fadd(lhs.value, rhs.value), + BinaryOp::Sub => builder.ins().fsub(lhs.value, rhs.value), + BinaryOp::Mul => builder.ins().fmul(lhs.value, rhs.value), _ => unreachable!(), - }; - let must_miss = if *operation == BinaryOp::Mul { - use cranelift_codegen::ir::condcodes::IntCC; - let zero = builder.ins().icmp_imm(IntCC::Equal, value, 0); - let lhs_negative = - builder.ins().icmp_imm(IntCC::SignedLessThan, lhs.value, 0); - let rhs_negative = - builder.ins().icmp_imm(IntCC::SignedLessThan, rhs.value, 0); - let opposite_signs = builder.ins().bxor(lhs_negative, rhs_negative); - let negative_zero = builder.ins().band(zero, opposite_signs); - builder.ins().bor(overflow, negative_zero) - } else { - overflow - }; - let ok = builder.create_block(); - let miss = builder.create_block(); - builder.ins().brif(must_miss, miss, &[], ok, &[]); - builder.switch_to_block(miss); - let status = builder.ins().iconst(types::I32, 1); - builder.ins().return_(&[status]); - builder.switch_to_block(ok); - value + }, + FeedbackRepresentation::Bool | FeedbackRepresentation::HeapRef => { + return Err(CompileFailure::UnsupportedOpcode) + } + }; + stack.push(TypedValue { + value, + representation: lhs.representation, + }); + } + IrOp::Binary( + operation @ (BinaryOp::LessThan + | BinaryOp::LessThanOrEqual + | BinaryOp::GreaterThan + | BinaryOp::GreaterThanOrEqual + | BinaryOp::StrictEqual + | BinaryOp::StrictNotEqual), + ) => { + let rhs = stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + let lhs = stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + let equality = + matches!(operation, BinaryOp::StrictEqual | BinaryOp::StrictNotEqual); + let admitted = lhs.representation == rhs.representation + && (lhs.representation == FeedbackRepresentation::Int32 + || (equality + && lhs.representation == FeedbackRepresentation::Bool)); + if !admitted { + return Err(CompileFailure::UnsupportedOpcode); } - FeedbackRepresentation::Float64 => match operation { - BinaryOp::Add => builder.ins().fadd(lhs.value, rhs.value), - BinaryOp::Sub => builder.ins().fsub(lhs.value, rhs.value), - BinaryOp::Mul => builder.ins().fmul(lhs.value, rhs.value), - _ => unreachable!(), - }, - FeedbackRepresentation::Bool | FeedbackRepresentation::HeapRef => { - unreachable!("checked above") + let condition = match operation { + BinaryOp::LessThan => IntCC::SignedLessThan, + BinaryOp::LessThanOrEqual => IntCC::SignedLessThanOrEqual, + BinaryOp::GreaterThan => IntCC::SignedGreaterThan, + BinaryOp::GreaterThanOrEqual => IntCC::SignedGreaterThanOrEqual, + BinaryOp::StrictEqual => IntCC::Equal, + _ => IntCC::NotEqual, + }; + let result = builder.ins().icmp(condition, lhs.value, rhs.value); + stack.push(TypedValue { + value: builder.ins().uextend(types::I32, result), + representation: FeedbackRepresentation::Bool, + }); + } + IrOp::Unary(UnaryOp::LogicalNot) => { + let input = stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + let truthy = truth(&mut builder, input)?; + let falsy = builder.ins().bxor_imm(truthy, 1); + stack.push(TypedValue { + value: builder.ins().uextend(types::I32, falsy), + representation: FeedbackRepresentation::Bool, + }); + } + IrOp::Unary(operation @ (UnaryOp::Increment | UnaryOp::Decrement)) + | IrOp::PostUnary(operation @ (UnaryOp::Increment | UnaryOp::Decrement)) => { + let input = *stack.last().ok_or(CompileFailure::InvalidArtifact)?; + if input.representation != FeedbackRepresentation::Int32 { + return Err(CompileFailure::UnsupportedOpcode); } - }; - stack.push(TypedValue { - value, - representation: lhs.representation, - }); - } - IrOp::Return => { - let result = stack.pop().ok_or(CompileFailure::InvalidArtifact)?; - if !stack.is_empty() || result.representation != signature.result() { - return Err(CompileFailure::UnsupportedOpcode); + let one = builder.ins().iconst(types::I32, 1); + let arithmetic = if *operation == UnaryOp::Increment { + BinaryOp::Add + } else { + BinaryOp::Sub + }; + let value = + emit_int32_arithmetic(&mut builder, arithmetic, input.value, one, miss); + let updated = TypedValue { + value, + representation: FeedbackRepresentation::Int32, + }; + // PostUnary keeps the old numeric value below the update. + if matches!(instruction.op, IrOp::Unary(_)) { + stack.pop(); + } + stack.push(updated); + } + IrOp::GetProperty(_) | IrOp::GetPropertyKeep(_) => { + let receiver = *stack.last().ok_or(CompileFailure::InvalidArtifact)?; + if receiver.representation != FeedbackRepresentation::HeapRef { + return Err(CompileFailure::UnsupportedOpcode); + } + let (observation, representation) = + linked_property(feedback, instruction.pc)?; + let layout = layout.ok_or(CompileFailure::InvalidArtifact)?; + let address = emit_guarded_field( + &mut builder, + layout, + pointer_type, + receiver.value, + observation, + representation_tag(representation)?, + miss, + )?; + let value = builder.ins().load( + scalar_type(representation, pointer_type), + MemFlags::new(), + address, + 0, + ); + if matches!(instruction.op, IrOp::GetProperty(_)) { + stack.pop(); + } + stack.push(TypedValue { + value, + representation, + }); + } + IrOp::SetProperty(_) => { + let value = stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + let receiver = stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + let (observation, representation) = + linked_property(feedback, instruction.pc)?; + if receiver.representation != FeedbackRepresentation::HeapRef + || value.representation != representation + { + return Err(CompileFailure::UnsupportedOpcode); + } + let store = stores + .iter() + .find(|store| store.pc == instruction.pc) + .ok_or(CompileFailure::InvalidArtifact)?; + let layout = layout.ok_or(CompileFailure::InvalidArtifact)?; + // The current tag equals the stored tag, so the old + // value is a primitive: replacing it frees nothing. + let address = emit_guarded_field( + &mut builder, + layout, + pointer_type, + receiver.value, + observation, + store.tag, + miss, + )?; + let payload = match representation { + FeedbackRepresentation::Int32 => { + builder.ins().sextend(types::I64, value.value) + } + FeedbackRepresentation::Bool => { + builder.ins().uextend(types::I64, value.value) + } + FeedbackRepresentation::Float64 => { + builder + .ins() + .bitcast(types::I64, MemFlags::new(), value.value) + } + FeedbackRepresentation::HeapRef => { + return Err(CompileFailure::InvalidArtifact) + } + }; + builder.def_var(store.address, address); + builder.def_var(store.payload, payload); + let dirty = builder.ins().iconst(types::I8, 1); + builder.def_var(store.dirty, dirty); } - builder - .ins() - .store(MemFlags::new(), result.value, output, 0); - let status = builder.ins().iconst(types::I32, 0); - builder.ins().return_(&[status]); - returned = true; + IrOp::Jump(target) => { + builder.ins().jump(blocks[target], &[]); + terminated = true; + } + IrOp::Branch { target, when_true } => { + let condition = stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + if !stack.is_empty() { + return Err(CompileFailure::UnsupportedOpcode); + } + let truthy = truth(&mut builder, condition)?; + let fallthrough = ir + .blocks + .get(block_index + 1) + .map(|next| blocks[&next.start_pc]) + .ok_or(CompileFailure::InvalidArtifact)?; + let (taken, other) = if *when_true { + (blocks[target], fallthrough) + } else { + (fallthrough, blocks[target]) + }; + builder.ins().brif(truthy, taken, &[], other, &[]); + terminated = true; + } + IrOp::Return => { + let result = stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + if !stack.is_empty() || result.representation != signature.result() { + return Err(CompileFailure::UnsupportedOpcode); + } + // Commit in program order: on a forward-only CFG the + // dirty stores of this path are exactly those executed, + // and later same-field stores win as in the interpreter. + for store in stores.iter().filter(|store| store.pc < instruction.pc) { + let dirty = builder.use_var(store.dirty); + let commit = builder.create_block(); + let done = builder.create_block(); + builder.ins().brif(dirty, commit, &[], done, &[]); + builder.switch_to_block(commit); + let address = builder.use_var(store.address); + let payload = builder.use_var(store.payload); + builder.ins().store(MemFlags::new(), payload, address, 0); + let tag = builder.ins().iconst(types::I64, i64::from(store.tag)); + builder.ins().store(MemFlags::new(), tag, address, 8); + builder.ins().jump(done, &[]); + builder.switch_to_block(done); + } + builder + .ins() + .store(MemFlags::new(), result.value, output, 0); + let status = builder.ins().iconst(types::I32, 0); + builder.ins().return_(&[status]); + terminated = true; + } + _ => return Err(CompileFailure::UnsupportedOpcode), } - _ => return Err(CompileFailure::UnsupportedOpcode), + } + if !terminated { + if !stack.is_empty() { + return Err(CompileFailure::UnsupportedOpcode); + } + let next = ir + .blocks + .get(block_index + 1) + .map(|next| blocks[&next.start_pc]) + .ok_or(CompileFailure::UnsupportedOpcode)?; + builder.ins().jump(next, &[]); } } - if !returned { - return Err(CompileFailure::UnsupportedOpcode); - } + builder.switch_to_block(miss); + let status = builder.ins().iconst(types::I32, 1); + builder.ins().return_(&[status]); builder.seal_all_blocks(); builder.finalize(); } @@ -286,7 +756,7 @@ mod tests { cranelift_codegen::settings::builder(), )) .expect("host flags"); - let code = super::lower_target_only_linked_leaf(&isa, &verified, &signature, None) + let code = super::lower_target_only_linked_leaf(&isa, &verified, &signature, None, None) .expect("target-only linked leaf"); assert!( code.clif().contains("(i64, i64, i32) -> i32"), @@ -317,7 +787,7 @@ mod tests { cranelift_codegen::settings::builder(), )) .expect("host flags"); - let code = super::lower_target_only_linked_leaf(&isa, &verified, &signature, None) + let code = super::lower_target_only_linked_leaf(&isa, &verified, &signature, None, None) .expect("target-only linked leaf") .publish() .expect("publish linked leaf"); @@ -351,7 +821,7 @@ mod tests { cranelift_codegen::settings::builder(), )) .expect("host flags"); - let code = super::lower_target_only_linked_leaf(&isa, &verified, &signature, None) + let code = super::lower_target_only_linked_leaf(&isa, &verified, &signature, None, None) .expect("target-only linked leaf") .publish() .expect("publish linked leaf"); @@ -398,7 +868,7 @@ mod tests { cranelift_codegen::settings::builder(), )) .expect("host flags"); - let code = super::lower_target_only_linked_leaf(&isa, &verified, &signature, None) + let code = super::lower_target_only_linked_leaf(&isa, &verified, &signature, None, None) .expect("target-only linked bool leaf") .publish() .expect("publish linked bool leaf"); @@ -431,9 +901,308 @@ mod tests { )) .expect("host flags"); assert_eq!( - super::lower_target_only_linked_leaf(&isa, &verified, &signature, None) + super::lower_target_only_linked_leaf(&isa, &verified, &signature, None, None) .expect_err("oversized target-only leaf"), crate::compiler::CompileFailure::ResourceLimit ); } + mod effectful { + use crate::{ + bytecode::VerifyLimits, + compiler::CompileFailure, + runtime::{ + BinaryFeedbackFlags, FeedbackSnapshot, FeedbackTable, FunctionKey, ObservedType, + PropertyAttributes, PrototypeDependencyToken, ShapeFeedbackTable, ShapeObservation, + ShapeToken, + }, + test_support::SnapshotFixture, + }; + use rquickjs_core::qjs; + + const GENERATION: u64 = 7; + + /// Just enough of a QuickJS object for the guarded field protocol: + /// the shape pointer, the shape generation and the property array. + struct FakeObject { + object: Box<[u64; 64]>, + shape: Box<[u64; 64]>, + properties: Box<[u64; 16]>, + } + + impl FakeObject { + fn new(fields: &[(i32, u64)]) -> Self { + let layout = crate::abi::AbiInfo::linked() + .expect("linked ABI") + .property_layout(); + let mut fake = Self { + object: Box::new([0; 64]), + shape: Box::new([0; 64]), + properties: Box::new([0; 16]), + }; + for (index, (tag, payload)) in fields.iter().enumerate() { + fake.properties[index * 2] = *payload; + fake.properties[index * 2 + 1] = i64::from(*tag) as u64; + } + let shape = fake.shape.as_mut_ptr() as u64; + let properties = fake.properties.as_mut_ptr() as u64; + fake.shape[layout.shape_generation_offset as usize / 8] = GENERATION; + fake.object[layout.object_shape_offset as usize / 8] = shape; + fake.object[layout.object_properties_offset as usize / 8] = properties; + fake + } + fn address(&self) -> usize { + self.object.as_ptr() as usize + } + fn shape_identity(&self) -> u64 { + self.shape.as_ptr() as u64 + } + fn field(&self, index: usize) -> (i32, u64) { + ( + self.properties[index * 2 + 1] as i64 as i32, + self.properties[index * 2], + ) + } + fn set_generation(&mut self, generation: u64) { + let layout = crate::abi::AbiInfo::linked().unwrap().property_layout(); + self.shape[layout.shape_generation_offset as usize / 8] = generation; + } + } + + fn isa() -> cranelift_codegen::isa::OwnedTargetIsa { + cranelift_native::builder() + .expect("host ISA") + .finish(cranelift_codegen::settings::Flags::new( + cranelift_codegen::settings::builder(), + )) + .expect("host flags") + } + + /// Compiles `source` with Int32 arithmetic/return feedback and one + /// monomorphic own-field observation per property site; `offsets` + /// gives each site's field index in bytecode order. + fn compile( + source: &str, + arguments: &[ObservedType], + shape: u64, + offsets: &[u32], + ) -> Result { + let fixture = SnapshotFixture::compile(source); + let verified = fixture + .snapshot() + .verify(VerifyLimits::default()) + .expect("verified effectful leaf"); + let key = FunctionKey::new( + verified.snapshot().function_id(), + verified.snapshot().generation(), + ); + let mut feedback = FeedbackTable::new(32, 2); + let mut shapes = ShapeFeedbackTable::new(4); + let mut property_sites = 0; + for instruction in verified.instructions() { + match instruction.opcode().name() { + "add" | "sub" | "inc" | "post_inc" => { + for _ in 0..32 { + feedback.observe_binary( + key, + instruction.pc(), + ObservedType::Int32, + ObservedType::Int32, + ObservedType::Int32, + BinaryFeedbackFlags::NONE, + ); + } + } + "return" => feedback.observe_return(key, instruction.pc(), ObservedType::Int32), + "get_field" | "get_field2" | "put_field" => { + let offset = offsets[property_sites]; + property_sites += 1; + shapes.observe( + key, + instruction.pc(), + ShapeObservation::new( + ShapeToken::new(shape, GENERATION), + PrototypeDependencyToken::new(0, 0), + offset, + PropertyAttributes::WRITABLE + | PropertyAttributes::ENUMERABLE + | PropertyAttributes::CONFIGURABLE, + ObservedType::Int32, + ), + ); + } + _ => {} + } + } + assert_eq!(property_sites, offsets.len(), "{source}"); + for _ in 0..32 { + feedback.observe_call(key, arguments); + } + let snapshot: FeedbackSnapshot = + feedback.snapshot(93).with_properties(shapes.snapshot(key)); + let signature = snapshot + .bounded_specialization(key) + .expect("bounded heterogeneous signature"); + super::super::lower_target_only_linked_leaf( + &isa(), + &verified, + &signature, + Some(&snapshot), + None, + ) + .map(|code| code.publish().expect("publish effectful leaf")) + } + + const INCREMENT_AND_RECORD: &str = "(function(value, enabled, state){ + state.calls = state.calls + 1; + if (enabled) return value + 1; + return value;})"; + + type Record = extern "C" fn(*mut i32, i32, i32, usize) -> i32; + + fn record_code(state: &FakeObject) -> crate::compiler::baseline::PublishedBaselineCode { + compile( + INCREMENT_AND_RECORD, + &[ + ObservedType::Int32, + ObservedType::Bool, + ObservedType::Object, + ], + state.shape_identity(), + &[0, 0], + ) + .expect("multi-block effectful leaf") + } + + #[test] + fn guarded_field_store_commits_with_the_result_on_every_branch() { + let state = FakeObject::new(&[(qjs::JS_TAG_INT, 5)]); + let code = record_code(&state); + // SAFETY: the signature is (Int32, Bool, HeapRef) -> Int32 and the + // fake object outlives every call. + let record = unsafe { core::mem::transmute::<*const u8, Record>(code.as_ptr()) }; + let mut output = -1; + assert_eq!(record(&mut output, 41, 1, state.address()), 0); + assert_eq!(output, 42); + assert_eq!(state.field(0), (qjs::JS_TAG_INT, 6)); + assert_eq!(record(&mut output, 41, 0, state.address()), 0); + assert_eq!(output, 41); + assert_eq!(state.field(0), (qjs::JS_TAG_INT, 7)); + } + + #[test] + fn late_overflow_misses_before_the_buffered_store_is_committed() { + let state = FakeObject::new(&[(qjs::JS_TAG_INT, 5)]); + let code = record_code(&state); + // SAFETY: see guarded_field_store_commits_with_the_result_on_every_branch. + let record = unsafe { core::mem::transmute::<*const u8, Record>(code.as_ptr()) }; + let mut output = 77; + // `value + 1` overflows after `state.calls` was (virtually) written. + assert_eq!(record(&mut output, i32::MAX, 1, state.address()), 1); + assert_eq!(output, 77, "a miss must not expose a result"); + assert_eq!( + state.field(0), + (qjs::JS_TAG_INT, 5), + "the generic CALL must observe the original field" + ); + // The counter itself overflowing is also a pre-commit miss. + let full = FakeObject::new(&[(qjs::JS_TAG_INT, i32::MAX as u32 as u64)]); + let code = record_code(&full); + // SAFETY: as above. + let record = unsafe { core::mem::transmute::<*const u8, Record>(code.as_ptr()) }; + assert_eq!(record(&mut output, 1, 1, full.address()), 1); + assert_eq!(full.field(0), (qjs::JS_TAG_INT, i32::MAX as u32 as u64)); + assert_eq!(output, 77); + } + + #[test] + fn shape_generation_and_field_tag_guards_miss_without_writing() { + let mut state = FakeObject::new(&[(qjs::JS_TAG_INT, 5)]); + let code = record_code(&state); + // SAFETY: as above. + let record = unsafe { core::mem::transmute::<*const u8, Record>(code.as_ptr()) }; + let mut output = 77; + state.set_generation(GENERATION + 1); + assert_eq!(record(&mut output, 1, 1, state.address()), 1); + assert_eq!(state.field(0), (qjs::JS_TAG_INT, 5)); + state.set_generation(GENERATION); + let other = FakeObject::new(&[(qjs::JS_TAG_INT, 5)]); + assert_eq!(record(&mut output, 1, 1, other.address()), 1); + assert_eq!(other.field(0), (qjs::JS_TAG_INT, 5)); + let double = FakeObject::new(&[(qjs::JS_TAG_FLOAT64, 1.5f64.to_bits())]); + let code = record_code(&double); + // SAFETY: as above. + let record = unsafe { core::mem::transmute::<*const u8, Record>(code.as_ptr()) }; + assert_eq!(record(&mut output, 1, 1, double.address()), 1); + assert_eq!(double.field(0), (qjs::JS_TAG_FLOAT64, 1.5f64.to_bits())); + assert_eq!(output, 77); + } + + #[test] + fn aliased_arguments_commit_stores_in_program_order() { + let source = "(function(value, left, right){ + left.x = value; right.x = value + 1; return value;})"; + let object = FakeObject::new(&[(qjs::JS_TAG_INT, 0)]); + let code = compile( + source, + &[ + ObservedType::Int32, + ObservedType::Object, + ObservedType::Object, + ], + object.shape_identity(), + &[0, 0], + ) + .unwrap(); + // SAFETY: (Int32, HeapRef, HeapRef) -> Int32 over a live fake. + let run = unsafe { + core::mem::transmute::<*const u8, extern "C" fn(*mut i32, i32, usize, usize) -> i32>( + code.as_ptr(), + ) + }; + let mut output = 0; + assert_eq!(run(&mut output, 9, object.address(), object.address()), 0); + assert_eq!(output, 9); + assert_eq!(object.field(0), (qjs::JS_TAG_INT, 10)); + } + + #[test] + fn post_increment_statement_is_a_guarded_read_modify_write() { + let source = "(function(value, state){state.calls++; return value + 1;})"; + let state = FakeObject::new(&[(qjs::JS_TAG_INT, 1), (qjs::JS_TAG_INT, 40)]); + let code = compile( + source, + &[ObservedType::Int32, ObservedType::Object], + state.shape_identity(), + &[1, 1], + ) + .unwrap(); + // SAFETY: (Int32, HeapRef) -> Int32 over a live fake. + let run = unsafe { + core::mem::transmute::<*const u8, extern "C" fn(*mut i32, i32, usize) -> i32>( + code.as_ptr(), + ) + }; + let mut output = 0; + assert_eq!(run(&mut output, 2, state.address()), 0); + assert_eq!(output, 3); + assert_eq!(state.field(0), (qjs::JS_TAG_INT, 1)); + assert_eq!(state.field(1), (qjs::JS_TAG_INT, 41)); + } + + #[test] + fn a_field_read_after_a_store_is_rejected() { + let source = "(function(value, state){state.a = value; return state.b + 1;})"; + let state = FakeObject::new(&[(qjs::JS_TAG_INT, 0)]); + assert_eq!( + compile( + source, + &[ObservedType::Int32, ObservedType::Object], + state.shape_identity(), + &[0, 1], + ) + .err(), + Some(CompileFailure::UnsupportedOpcode) + ); + } + } } diff --git a/jit/src/runtime/coordinator.rs b/jit/src/runtime/coordinator.rs index 475e609fb..df309a38b 100644 --- a/jit/src/runtime/coordinator.rs +++ b/jit/src/runtime/coordinator.rs @@ -1434,9 +1434,13 @@ impl Coordinator { if budget.exhausted() { break; } - if direct_call_targets.iter().any(|target| { - target.pc() == instruction.pc() && target.inline_snapshot().is_some() - }) { + // A linked direct entry is a transaction with an exact + // pre-effect miss to the generic CALL. It needs no shadow + // frame, so it is always preferred over frame inlining. + if direct_call_targets + .iter() + .any(|target| target.pc() == instruction.pc()) + { continue; } if let Some(target) = self.retain_frame_inline_target( diff --git a/jit/tests/tagged_call_link.rs b/jit/tests/tagged_call_link.rs index 0bf68fd79..32a36ff38 100644 --- a/jit/tests/tagged_call_link.rs +++ b/jit/tests/tagged_call_link.rs @@ -219,3 +219,200 @@ fn baseline_target_only_heapref_link_returns_bool_without_a_call_frame() { fn tier2_target_only_heapref_link_returns_bool_without_a_call_frame() { exercise_bool_result(JitTierPolicy::Automatic, true); } + +const EFFECTFUL_SOURCE: &str = " + globalThis.hits=0; + globalThis.state={calls:0}; + function recordLeaf(value,enabled,state){ + state.calls=state.calls+1; + if(enabled)return value+1; + return value; + } + function bumpLeaf(value,state){state.calls++;return value+2} + function recordCaller(target,state,value,enabled,n){ + for(let i=0;i Self { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .tier_policy(policy) + .call_threshold(2) + .loop_threshold(16) + .force_optimized_for_test(force_tier2) + .stress_gc(stress_gc) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context + .with(|ctx| ctx.eval::<(), _>(EFFECTFUL_SOURCE)) + .unwrap(); + Self { + context, + jit, + _runtime: runtime, + } + } + + fn eval(&self, source: &str) -> String { + self.context.with(|ctx| { + ctx.eval::(format!("String({source})")) + .unwrap_or_else(|error| { + let exception = ctx.catch(); + panic!("{source}: {error:?}: {exception:?}") + }) + }) + } + + /// Warm until one complete invocation no longer reaches the generic CALL. + fn wait_for_link(&self, expression: &str, force_tier2: bool) { + let deadline = Instant::now() + Duration::from_secs(60); + loop { + let before = call_count(&self.context); + let entries = self.jit.metrics().native_entries; + self.eval(expression); + self.jit.poll(); + let after = self.jit.metrics(); + if after.pending_worker_jobs == 0 + && after.native_entries > entries + && (!force_tier2 || after.tier2_entries > 0) + && call_count(&self.context) == before + { + return; + } + assert!( + Instant::now() < deadline, + "effectful linked edge not ready for {expression}: {after:?}" + ); + std::thread::sleep(Duration::from_millis(1)); + } + } +} + +fn exercise_effectful(policy: JitTierPolicy, force_tier2: bool, stress_gc: bool) { + let test = Effectful::new(policy, force_tier2, stress_gc); + test.wait_for_link( + "(state.calls=0, recordCaller(recordLeaf,state,7,true,128))", + force_tier2, + ); + // `state.calls++` (get_field2/post_inc) is not admitted by the Tier 2 + // body compiler, so a forced-Tier2 callee publishes no linked entry. In + // production the callee's Baseline artifact provides it. + if !force_tier2 { + test.wait_for_link( + "(state.calls=0, bumpCaller(bumpLeaf,state,7,128))", + force_tier2, + ); + } + + // Stable edge: exact results and effects, with no generic CALL. + let before = call_count(&test.context); + assert_eq!( + test.eval("(state.calls=0, recordCaller(recordLeaf,state,7,true,128))"), + "135" + ); + assert_eq!(test.eval("state.calls"), "128"); + assert_eq!(test.eval("recordCaller(recordLeaf,state,7,false,64)"), "7"); + assert_eq!(test.eval("state.calls"), "192"); + if !force_tier2 { + assert_eq!(test.eval("bumpCaller(bumpLeaf,state,1,10)"), "21"); + assert_eq!(test.eval("state.calls"), "202"); + } + assert_eq!( + call_count(&test.context), + before, + "stable effectful edge still used the generic CALL" + ); + + // Result overflow after the (buffered) store: every call increments once. + assert_eq!( + test.eval("(state.calls=0, recordCaller(recordLeaf,state,2147483646,true,3))"), + "2147483649" + ); + assert_eq!(test.eval("state.calls"), "3"); + // Counter overflow: the field becomes a double through the generic path. + assert_eq!( + test.eval("(state.calls=2147483647, recordCaller(recordLeaf,state,1,true,2))"), + "3" + ); + assert_eq!(test.eval("state.calls"), "2147483649"); + // Field tag miss (double, then string) keeps exact generic semantics. + assert_eq!( + test.eval("(state.calls=1.5, recordCaller(recordLeaf,state,1,true,2))"), + "3" + ); + assert_eq!(test.eval("state.calls"), "3.5"); + assert_eq!( + test.eval("(state.calls='x', recordCaller(recordLeaf,state,1,true,2))"), + "3" + ); + assert_eq!(test.eval("state.calls"), "x11"); + // A different receiver shape misses before any write. + assert_eq!( + test.eval("(globalThis.other={extra:1,calls:5}, recordCaller(recordLeaf,other,1,true,2))"), + "3" + ); + assert_eq!(test.eval("other.calls+':'+state.calls"), "7:x11"); + // The original receiver and representation link again. + assert_eq!( + test.eval("(state.calls=0, recordCaller(recordLeaf,state,1,true,4))"), + "5" + ); + assert_eq!(test.eval("state.calls"), "4"); + // In-place shape mutations: adding a field, then read-only, then accessor. + assert_eq!( + test.eval("(state.added=1, recordCaller(recordLeaf,state,1,true,2))"), + "3" + ); + assert_eq!(test.eval("state.calls"), "6"); + // The generic put_field reports the read-only field; nothing is written. + assert_eq!( + test.eval( + "(Object.defineProperty(state,'calls',{writable:false}), \ + (()=>{try{return recordCaller(recordLeaf,state,1,true,3)}\ + catch(e){return e.constructor.name+':'+state.calls}})())" + ), + "TypeError:6" + ); + assert_eq!( + test.eval( + "(Object.defineProperty(state,'calls',{get(){return 40},set(v){hits+=v}}), \ + recordCaller(recordLeaf,state,1,true,3))" + ), + "4" + ); + assert_eq!(test.eval("hits+':'+state.calls"), "123:40"); + let metrics = test.jit.metrics(); + assert_eq!(metrics.native_entries, metrics.native_exits); +} + +#[test] +fn baseline_effectful_link_commits_field_stores_and_misses_exactly() { + exercise_effectful(JitTierPolicy::BaselineOnly, false, false); +} + +#[test] +fn tier2_effectful_link_commits_field_stores_and_misses_exactly() { + exercise_effectful(JitTierPolicy::Automatic, true, false); +} + +#[test] +fn tier2_effectful_link_is_exact_under_gc_stress() { + exercise_effectful(JitTierPolicy::Automatic, true, true); +} From b8761ea1c706c634d0eb52db1054424d9b4e1d10 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 16:43:12 +0800 Subject: [PATCH 04/54] perf(jit): let Tier 2 callers wait for an in-flight linked callee Linked entries need property feedback, which only the callee's optimizing compile sees. A caller queued in the same scan (or before that install) captured a frame inline of the Baseline body and never re-linked, leaving generic-call-fallback on the ~9 ms JS_JitInlineEnter path in some processes. Queue callees before callers within a scan and defer a frame-inline-ready caller while its callee's optimizing compile is queued, compiling or ready. The wait ends on install or failure. Documents the slice and remaining work in docs/roadmap/p3b-effectful-direct.md. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p3b-effectful-direct.md | 135 +++++++++++++++++++++++++++ jit/src/lib.rs | 39 +++++++- 2 files changed, 173 insertions(+), 1 deletion(-) create mode 100644 docs/roadmap/p3b-effectful-direct.md diff --git a/docs/roadmap/p3b-effectful-direct.md b/docs/roadmap/p3b-effectful-direct.md new file mode 100644 index 000000000..0e7047860 --- /dev/null +++ b/docs/roadmap/p3b-effectful-direct.md @@ -0,0 +1,135 @@ +# P3b: effectful linked direct calls (first slice) + +Branch `perf/p3b-effectful-direct`, based on `82d3808`. Roadmap item: P3 +(native calling convention), effectful compiled-to-compiled ABI. + +## What this slice does + +`lower_target_only_linked_leaf` (`jit/src/compiler/tagged_call_link.rs`) +used to accept only single-block, effect-free Int32/Float64/Bool leaves with at +least one borrowed HeapRef argument. It now also accepts bounded, acyclic +callees that read and write guarded own data fields of borrowed HeapRef +arguments, like `generic-call-fallback`'s `incrementAndRecord`: + +```js +function incrementAndRecord(value, enabled, state) { + state.calls = state.calls + 1; + if (enabled) return value + 1; + return value; +} +``` + +The ABI is unchanged: `(output*, unboxed args...) -> status`. Status `0` +stores the scalar result; status `1` asks the caller to run the original +generic CALL with an untouched frame. No C patch, ABI version or fingerprint +changes. + +### Exactness model: a transaction, not a frame + +The callee is lowered as a transaction over borrowed arguments: + +1. Every speculation happens before the first observable write: Int32 + overflow and `-0`, property shape identity + shape generation, and the + field's current value tag (which also proves the old value is a primitive, + so replacing it frees nothing). +2. Guarded field stores are buffered in Cranelift variables (`dirty`, address, + payload) and committed in program order only at `return`, immediately + before the result is written. +3. Therefore status `1` always means "no effect happened", and the existing + caller-side recovery (Baseline: generic CALL helper; Tier 2: deopt at the + CALL PC) remains exact. No callee frame is ever materialized, so there is no + `JS_JitInlineEnter`, no `calloc`, and no helper/validation call per access. + +Admission (fail-closed, before code generation): + +- at most 128 instructions, 64 frame slots, 16 KiB retained bytes, 16 blocks, + 8 buffered stores; no exception table; +- forward-only control flow (every jump/branch target is a later block start) + and an empty operand stack at every block boundary; +- no field read at a PC after any field store PC (on a forward-only CFG this + is exactly "no path reads a buffered write"; it also makes argument aliasing + safe, and same-field stores still commit last-writer-wins); +- each property site has monomorphic feedback that passes + `eligible_property_observation` (own, writable, non-accessor, no prototype + dependency) with an Int32/Float64/Bool value; a stored value must have the + observed representation; +- supported operations: typed constants, arguments, `drop`, `dup`, `swap`, + Int32/Float64 `add`/`sub`/`mul`, Int32 relational and Int32/Bool strict + equality, `!`, Int32 `inc`/`dec`/`post_inc`/`post_dec`, `get_field`, + `get_field2`, `put_field`, `goto`, `if_true`/`if_false` on Int32/Bool, + `return` of the signature's representation. Everything else rejects the + linked entry, and the call keeps its previous path. + +`lower_direct_call_machine` now receives the callee's `FeedbackSnapshot` +so the linked entry can read property feedback (Baseline and Tier 2 compile +paths, plus the test hooks). + +### Coordinator preference + +`Coordinator` previously skipped frame-inline planning at a call PC only when +the direct target also carried an inline snapshot. A direct target now always +wins over frame inlining at that PC: the linked entry needs no shadow frame, +and the old preference would otherwise route `incrementAndRecord` through +`JS_JitInlineEnter`, as it did before this slice. + +### Tier 2 queueing order (`jit/src/lib.rs`) + +The interpreter records no property feedback, so the callee's Baseline +artifact normally has no linked entry. Only its Tier 2 artifact does. Without +another change, the caller was often queued for Tier 2 in the same scan as the +callee, or before the callee's Tier 2 install. It then captured a frame +inline of the callee's Baseline body and never re-linked. In a noisy check, 2 +of 7 processes stayed on the old ~9 ms path. Two changes fix this: + +- a Tier 2 scan queues callees before their callers (sorted by the number of + ready callees); +- a caller whose loop call site is frame-inline-ready but not yet + direct-ready waits while the callee's optimizing compile is `Queued`, + `Compiling` or `Ready` and the callee has a bounded signature. The wait + ends when that compile installs or fails, so it is bounded. + +After this, 12 of 12 processes linked the edge. + +## Tests + +- Unit (`tagged_call_link::tests::effectful`), against fake object memory + using the linked property layout: commit on both branches; late `value + 1` + overflow and counter overflow miss with the field untouched; shape + identity/generation and field-tag misses write nothing; aliased arguments + commit in program order; `state.calls++` read-modify-write; a read after a + store is rejected. +- Integration (`jit/tests/tagged_call_link.rs`, Baseline, forced Tier 2, and + forced Tier 2 with GC stress): no generic CALL on the stable edge; exact + results and effects for result overflow, counter overflow, Float64 and + string fields, a different receiver shape, a newly added field, a read-only + field (the generic `put_field` throws `TypeError`, nothing is written), and + an accessor replacement. A mutation check (dropping the commit store) makes + all three integration tests fail, so they do exercise the linked entry. + +## Remaining work + +- **`mixed-quotes` / `compareQuotes`**: the comparator is called by + `Array.prototype.sort` from C, not from compiled code, so no linked call edge + exists. `compareQuotes(left, right){return right.score - left.score}` already + fits this ABI (two HeapRef loads, Int32 result). Using it needs a C-side + fast entry: a cached per-function linked entry that `js_array_sort` (and + `map`/`forEach`) can call before `JS_Call`, cleared on artifact retirement. + That is the P1 "C-side tier state" / P3 "fast entry from C" work and needs a + new patch (reserved number `0030`), an ABI minor bump and a retirement + protocol. It was not done in this slice to avoid overlapping the P1 changes + to the same C call path. +- Tier 2 does not compile a callee containing `get_field2` + `post_inc` + (`state.calls++`), so under forced Tier 2 such a callee publishes no linked + entry. In production the callee's Baseline artifact provides it. +- Persistent misses (for example a permanent shape change) cost a Tier 2 caller + one deopt per call until the existing guard-failure demotion moves it to + Baseline, where a miss costs only the guard loads before the generic CALL. + Recompiling the callee's linked entry from newer property feedback would + remove this. +- Not yet admitted: field loads after stores (needs store-to-load forwarding + plus alias checks), heap-valued fields and heap results (needs ownership + transfer), locals, loops (needs polls), nested calls, `this`, Float64 + comparisons, and polymorphic property sites. +- A real native calling convention (P3 proper) with lazily materialized frames + is still required for recursion and general callees. This slice avoids any + frame by keeping the callee transactional. diff --git a/jit/src/lib.rs b/jit/src/lib.rs index d9a15bd96..e98f74fb8 100644 --- a/jit/src/lib.rs +++ b/jit/src/lib.rs @@ -2266,6 +2266,23 @@ impl ProductionBackend { } }) .collect::>(); + let mut ready_for_tier2 = ready_for_tier2; + if ready_for_tier2.len() > 1 { + // Queue callees before their callers in one scan. A caller then + // observes its callee's optimizing compile as in flight (below) + // instead of freezing the callee's Baseline shape into its artifact. + let calls = self.feedback.snapshot(self.clock.max(1)); + let ready = ready_for_tier2 + .iter() + .copied() + .collect::>(); + ready_for_tier2.sort_by_cached_key(|key| { + calls + .call_specializations_for(*key) + .filter(|call| call.callee() != *key && ready.contains(&call.callee())) + .count() + }); + } self.last_scan_installed = self.coordinator.installed_count(); for key in if self.config.tier_policy() == JitTierPolicy::BaselineOnly { Vec::new() @@ -2307,7 +2324,27 @@ impl ProductionBackend { &observed, ); let resolved = self.coordinator.call_target_resolved(call.callee()); - if direct_ready || frame_ready { + if direct_ready { + continue; + } + if frame_ready { + /* A linked entry is published only by an artifact compiled + * with the callee's property feedback, which Baseline + * usually lacks. While that optimizing compile is in + * flight, wait for it rather than capture a frame inline + * of the Baseline body: the caller would never re-link. + * The wait ends when the compile installs or fails. */ + let callee_optimizing = matches!( + self.coordinator + .tier_state(call.callee(), runtime::Tier::Optimizing), + runtime::CompileState::Queued(_) + | runtime::CompileState::Compiling(_) + | runtime::CompileState::Ready(_) + ); + if callee_optimizing && observed.bounded_specialization(call.callee()).is_some() + { + direct_call_pending = true; + } continue; } if !resolved { From d669371a7c38203b700bd2f247082d1d751e4299 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 16:47:58 +0800 Subject: [PATCH 05/54] perf(jit): guard typed-array receivers held in locals Tier 2 only specialized typed-array length/element/store sites whose receiver was an entry argument. float64array-traversal loads both views from buffers.ints/buffers.floats into const locals, so every iteration ran ints.length through the generic GET_PROPERTY bridge and its getter. - ArrayPlan identifies a receiver as an entry argument or as the unique local holding the operand's SSA value (KnownFacts now tracks value identity through agreeing Phis and preserved frame reads); a loop that writes that local is rejected. - Planned typed get_length lowers to the versioned leaf query (exit on miss) instead of the accessor-running bridge. - Lowering keeps up to four guarded metadata tuples keyed by frame slot, carries them across forward joins where every predecessor agrees, and hoists storage-only typed receivers (store destinations) to the loop preheader with poll revalidation. - Numeric constant pushes and guarded to_propkey are certified leaf sites; poll amortization additionally accepts owned-local stores outside every loop and proves frame reads across shape-guarded get_field leaves. Co-Authored-By: Claude Opus 5.5 (1M context) --- jit/src/compiler/optimized.rs | 506 ++++++++++++++++++---- jit/src/compiler/optimized/array_cache.rs | 215 ++++++++- jit/src/ir/facts.rs | 23 +- jit/tests/typed_receivers.rs | 420 ++++++++++++++++++ 4 files changed, 1061 insertions(+), 103 deletions(-) create mode 100644 jit/tests/typed_receivers.rs diff --git a/jit/src/compiler/optimized.rs b/jit/src/compiler/optimized.rs index afc40c273..3d5e02dcc 100644 --- a/jit/src/compiler/optimized.rs +++ b/jit/src/compiler/optimized.rs @@ -739,7 +739,7 @@ mod provenance_merge_tests { } } -#[derive(Clone, Copy)] +#[derive(Clone, Copy, PartialEq)] struct GuardedElementSource { provenance: OptProvenance, block_pc: u32, @@ -756,6 +756,100 @@ struct GuardedElementSource { typed_mode: Option, } +/// Guarded metadata tuples that are live at the current lowering point, keyed +/// by the frame slot (argument or local) that holds each receiver. Every entry +/// is individually valid: operations that can reenter, rebind a frame slot or +/// alter storage clear the whole set, exactly like the former single slot. +/// Leaf-or-exit array operations add or replace only their own receiver. +#[derive(Clone, Copy, Default)] +struct ElementSources { + entries: [Option; array_cache::ArrayPlan::MAX_LIVE_SOURCES], +} + +impl ElementSources { + fn clear(&mut self) { + self.entries = Default::default(); + } + + fn find(&self, provenance: OptProvenance) -> Option { + if !matches!( + provenance, + OptProvenance::Argument(_) | OptProvenance::Local(_) + ) { + return None; + } + self.entries + .iter() + .flatten() + .find(|source| source.provenance == provenance) + .copied() + } + + fn insert(&mut self, source: GuardedElementSource) { + if !matches!( + source.provenance, + OptProvenance::Argument(_) | OptProvenance::Local(_) + ) { + return; + } + if let Some(slot) = self + .entries + .iter_mut() + .find(|entry| entry.is_some_and(|entry| entry.provenance == source.provenance)) + { + *slot = Some(source); + } else if let Some(slot) = self.entries.iter_mut().find(|entry| entry.is_none()) { + *slot = Some(source); + } else { + // Evict the oldest tuple; dropping metadata is always sound. + self.entries.rotate_left(1); + self.entries[self.entries.len() - 1] = Some(source); + } + } + + /// The tuples live on entry to `block_pc`: those live at the end of + /// every CFG predecessor. All predecessors must already be lowered (a + /// loop header's backedge is not), so each tuple is valid on every + /// incoming edge. Its Cranelift values are shared by all predecessors and + /// therefore defined in a block dominating this join. + fn at_block_entry( + block_pc: u32, + predecessors: Option<&Vec>, + exits: &std::collections::BTreeMap, + ) -> Self { + let mut entry = Self::default(); + let Some(predecessors) = predecessors.filter(|incoming| !incoming.is_empty()) else { + return entry; + }; + let Some(states) = predecessors + .iter() + .map(|predecessor| exits.get(predecessor)) + .collect::>>() + else { + return entry; + }; + let same = |left: &GuardedElementSource, right: &GuardedElementSource| { + GuardedElementSource { block_pc, ..*left } + == GuardedElementSource { block_pc, ..*right } + }; + for source in states[0].entries.iter().flatten() { + if states[1..].iter().all(|state| { + state + .entries + .iter() + .flatten() + .any(|other| same(source, other)) + }) { + entry.insert(GuardedElementSource { + block_pc, + ..*source + }); + } + } + entry + } +} + #[derive(Clone, Copy, Default)] enum EntryRepresentation { #[default] @@ -1452,6 +1546,14 @@ fn lower_optimized_machine( matches!(ir.nodes()[id as usize].kind(), crate::ir::OptimizedNodeKind::Bytecode { opcode } if opcode.as_ref() == "to_propkey") }; + // Lowering materializes only numeric constant-pool entries, as immediates + // (any other constant fails compilation). Such a push cannot reenter. + let numeric_constant_push = |id: u32| { + ir.nodes() + .get(id as usize) + .and_then(opt_push_const_index) + .is_some_and(|index| specialization.numeric_constants.contains_key(&index)) + }; let array_loops = crate::ir::LoopAnalysis::analyze(ir, array_work); let assumed_array_sites = ir .nodes() @@ -1472,6 +1574,7 @@ fn lower_optimized_machine( |id| { assumed_array_sites.contains(&id) || guarded_propkey(id) + || numeric_constant_push(id) || ir.nodes().get(id as usize).is_some_and(|node| { matches!( node.kind(), @@ -1514,6 +1617,7 @@ fn lower_optimized_machine( (matches!(opcode.as_ref(), "get_array_el" | "put_array_el") && guarded_sites.contains(&id)) || guarded_propkey(id) + || numeric_constant_push(id) || (matches!( opcode.as_ref(), "add" @@ -1573,6 +1677,7 @@ fn lower_optimized_machine( |id| { guarded_sites.contains(&id) || guarded_propkey(id) + || numeric_constant_push(id) || ir.nodes().get(id as usize).is_some_and(|node| { matches!( node.kind(), @@ -1609,16 +1714,85 @@ fn lower_optimized_machine( } plan }; + // Amortization skips the per-iteration loop-header representation guard, + // so frame writes must normally be numeric-proven. That guard never checks + // an owned (helper-derived heap) local: it accepts any representation and + // every consumer keeps its own checks. A store into an owned local outside + // every natural loop (for example `const ints = buffers.ints` before a + // traversal) therefore neither feeds a guarded loop Phi nor runs between + // two header polls; an in-loop write to it is still rejected below. + let local_store_outside_loops = |id: u32| { + array_loops.is_complete() + && matches!( + ir.scalar_graph().effect_for_node(id), + crate::ir::ScalarHeapEffect::FrameWrite(crate::ir::FrameSlot::Local(local)) + if owned_locals.get(usize::from(local)).copied().unwrap_or(false) + ) + && array_loops.block_for_node(id).is_some_and(|block| { + !array_loops + .loops() + .iter() + .any(|natural_loop| natural_loop.contains_block(block)) + }) + }; // Array feedback only proposes candidates. Poll amortization is enabled // after the concrete plan has proved that every heap site on the native // path is a guarded leaf-or-exit operation. + // Every lowered get_field is a shape-guarded leaf-or-exit access (a + // missing guard fails compilation), as are the certified array, property + // key and constant sites. None of them runs script or writes the frame on + // its continuing path, so an argument or local read after one of them + // keeps its entry-guarded representation. This proof only decides poll + // amortization; lowering keeps using `scalar_numeric`. + let leaf_frame_preserving = |id: u32| { + array_plan.guarded_leaf(id) + || property_plan.access(id).is_some() + || guarded_propkey(id) + || numeric_constant_push(id) + || ir.nodes().get(id as usize).is_some_and(|node| { + (int32_loop + && matches!( + node.kind(), + crate::ir::OptimizedNodeKind::GuardNumeric { .. } + )) + || (matches!(node.kind(), crate::ir::OptimizedNodeKind::Bytecode { opcode } + if opcode.as_ref() == "get_field") + && specialization.properties.contains_key(&node.pc())) + }) + }; + let amortized_numeric = if side_path.is_none() && !int32_loop { + ir.scalar_graph() + .proven_numeric_values_with_preserved_frame_reads( + ir.nodes(), + |index| { + usize::from(index) < usize::from(shape.arguments()) + && matches!( + specialization + .arguments + .get(usize::from(index)) + .copied() + .unwrap_or(specialization.entry), + EntryRepresentation::Int32 + ) + }, + leaf_frame_preserving, + ir.scalar_graph().values().len().saturating_mul(128), + ) + } else { + Vec::new() + }; let amortized_poll = int32_loop || (side_path.is_none() && ir.scalar_graph().permits_amortized_poll_with_guarded_heap( ir.nodes(), - &scalar_numeric, + &amortized_numeric, |id| { - if array_plan.guarded_leaf(id) || property_plan.access(id).is_some() { + if array_plan.guarded_leaf(id) + || property_plan.access(id).is_some() + || numeric_constant_push(id) + || guarded_propkey(id) + || local_store_outside_loops(id) + { return true; } let Some(node) = ir.nodes().get(id as usize) else { @@ -1779,9 +1953,11 @@ fn lower_optimized_machine( .map(|value| (value, alloc())) .collect::>(); let mut stack_provenance = vec![OptProvenance::Unknown; stack_slots]; - let mut guarded_element_source: Option = None; + let mut guarded_element_source: ElementSources; + // Tuples live at the end of each lowered block, for forward joins. + let mut block_exit_sources = std::collections::BTreeMap::::new(); let mut hoisted_element_sources = - std::collections::BTreeMap::::new(); + std::collections::BTreeMap::>::new(); // The scalar-region proof excludes heap effects and requires every // local definition to be primitive (or a lexical sentinel). No local // owns a heap value, so poll/deopt boundaries can publish SSA locals @@ -1920,15 +2096,17 @@ fn lower_optimized_machine( &[], ); for block in ir.blocks() { - guarded_element_source = guarded_element_source.filter(|source| { - source.block_pc == block.start_pc() - || (matches!(source.provenance, OptProvenance::Argument(_)) - && predecessors - .get(&block.start_pc()) - .is_some_and(|incoming| incoming.as_slice() == [source.block_pc])) - }); - if let Some(source) = hoisted_element_sources.get(&block.start_pc()).copied() { - guarded_element_source = Some(source); + guarded_element_source = ElementSources::at_block_entry( + block.start_pc(), + predecessors.get(&block.start_pc()), + &block_exit_sources, + ); + for source in hoisted_element_sources + .get(&block.start_pc()) + .into_iter() + .flatten() + { + guarded_element_source.insert(*source); } let clif_block = blocks[&block.start_pc()]; builder.switch_to_block(clif_block); @@ -2017,11 +2195,18 @@ fn lower_optimized_machine( crate::ir::OptimizedNodeKind::GuardNumeric { mid_loop: true, .. } ) && hoisted_element_sources .contains_key(&block.start_pc()); - if !array_plan.is_empty() - && array_plan.invalidates_before(node.id()) - && !retains_hoisted_metadata - { - guarded_element_source = None; + if !array_plan.is_empty() && array_plan.invalidates_before(node.id()) { + guarded_element_source.clear(); + if retains_hoisted_metadata { + // The poll revalidates exactly the hoisted tuples. + for source in hoisted_element_sources + .get(&block.start_pc()) + .into_iter() + .flatten() + { + guarded_element_source.insert(*source); + } + } } if matches!(node.kind(), crate::ir::OptimizedNodeKind::Bytecode { opcode } if matches!(opcode.as_ref(), "if_false8" | "if_true8" | "if_false" | "if_true" | "goto" | "goto8" | "goto16")) @@ -2079,8 +2264,10 @@ fn lower_optimized_machine( node.pc(), *guard, )?; - if let Some(source) = - hoisted_element_sources.get(&block.start_pc()).copied() + for source in hoisted_element_sources + .get(&block.start_pc()) + .into_iter() + .flatten() { emit_opt_packed_loop_revalidate( builder, @@ -2089,7 +2276,7 @@ fn lower_optimized_machine( depth, node.pc(), *guard, - source, + *source, element_layout, array_query, )?; @@ -2137,8 +2324,10 @@ fn lower_optimized_machine( // amortized cold edge. A typed store can keep its // numeric locals unproven, selecting this path; // cached pointers/length must still be revalidated. - if let Some(source) = - hoisted_element_sources.get(&block.start_pc()).copied() + for source in hoisted_element_sources + .get(&block.start_pc()) + .into_iter() + .flatten() { emit_opt_packed_loop_revalidate( &mut builder, @@ -2147,7 +2336,7 @@ fn lower_optimized_machine( depth, node.pc(), *guard, - source, + *source, element_layout, array_query, )?; @@ -2171,11 +2360,11 @@ fn lower_optimized_machine( if array_plan.is_empty() && node.effect() == crate::ir::OptimizedEffect::Reentrant { - guarded_element_source = None; + guarded_element_source.clear(); } match name { "set_loc_uninitialized" => { - guarded_element_source = None; + guarded_element_source.clear(); let index = opt_u16(node.bytes())?; if owned_locals[index] { emit_opt_free_local_slot(&mut builder, &env, index)?; @@ -2307,7 +2496,7 @@ fn lower_optimized_machine( n if opt_index(n, node.bytes(), "put_loc")?.is_some() || matches!(n, "put_loc_check" | "put_loc_check_init") => { - guarded_element_source = None; + guarded_element_source.clear(); let index = opt_index(n, node.bytes(), "put_loc")? .map_or_else(|| opt_u16(node.bytes()), Ok)?; let source_owned = depth @@ -2555,9 +2744,7 @@ fn lower_optimized_machine( .filter(|candidate| { depth >= 2 && stack_provenance[depth - 2] - == OptProvenance::Argument(usize::from( - candidate.argument, - )) + == opt_array_receiver_provenance(candidate.receiver) }) .map(|candidate| candidate.mode); opt_reject_owned( @@ -2594,6 +2781,39 @@ fn lower_optimized_machine( &stack_provenance, depth.saturating_sub(1)..depth, )?; + // A planned typed length selects the versioned + // leaf query (exit on miss) rather than the + // accessor-running lookup bridge. + let planned_typed = array_plan + .access(node.id()) + .filter(|access| { + access.access == crate::runtime::ArrayAccess::Length + }) + .and_then(|access| { + array_plan.candidates().get(access.candidate) + }) + .filter(|candidate| { + matches!( + candidate.mode, + crate::runtime::ArrayMode::Int32 + | crate::runtime::ArrayMode::Float64 + ) + }); + // The query guards the actual operand, so it + // is valid whichever frame slot supplied it; + // provenance only keys the cached tuple. A + // certified leaf length therefore never falls + // back to the reentrant bridge. + let typed_length = match planned_typed { + Some(candidate) => Some(( + candidate.mode, + node.deopt_guard() + .ok_or(CompileFailure::InvalidArtifact)?, + block.start_pc(), + array_query, + )), + None => None, + }; depth = emit_opt_array_length( &mut builder, &env, @@ -2602,6 +2822,7 @@ fn lower_optimized_machine( node.pc(), element_layout, &mut guarded_element_source, + typed_length, )?; } "put_array_el" => { @@ -2625,7 +2846,7 @@ fn lower_optimized_machine( // A provenance disagreement cannot silently fall back to // an operation stronger than the published effect. if source_provenance - != OptProvenance::Argument(usize::from(candidate.argument)) + != opt_array_receiver_provenance(candidate.receiver) { return Err(CompileFailure::InvalidArtifact); } @@ -3121,7 +3342,7 @@ fn lower_optimized_machine( stack_provenance[index] = OptProvenance::ImmediatePrimitive; } "inc_loc" | "dec_loc" => { - guarded_element_source = None; + guarded_element_source.clear(); let index = opt_u8(node.bytes())?; let (old, _) = opt_prepare_scalar_operation( &mut builder, @@ -3168,7 +3389,7 @@ fn lower_optimized_machine( ); } "add_loc" => { - guarded_element_source = None; + guarded_element_source.clear(); let index = opt_u8(node.bytes())?; let rhs_index = depth .checked_sub(1) @@ -3234,7 +3455,7 @@ fn lower_optimized_machine( } } n if opt_index(n, node.bytes(), "put_arg")?.is_some() => { - guarded_element_source = None; + guarded_element_source.clear(); opt_reject_owned( &stack_provenance, depth.saturating_sub(1)..depth, @@ -3264,7 +3485,7 @@ fn lower_optimized_machine( ); } n if opt_index(n, node.bytes(), "set_arg")?.is_some() => { - guarded_element_source = None; + guarded_element_source.clear(); opt_reject_owned( &stack_provenance, depth.saturating_sub(1)..depth, @@ -3296,7 +3517,7 @@ fn lower_optimized_machine( ); } n if opt_index(n, node.bytes(), "set_loc")?.is_some() => { - guarded_element_source = None; + guarded_element_source.clear(); opt_reject_owned( &stack_provenance, depth.saturating_sub(1)..depth, @@ -3780,6 +4001,7 @@ fn lower_optimized_machine( )?; builder.ins().jump(blocks[&next], &[]); } + block_exit_sources.insert(block.start_pc(), guarded_element_source); } builder.seal_all_blocks(); builder.finalize(); @@ -5585,15 +5807,7 @@ fn emit_opt_packed_loop_revalidate( array_query: usize, ) -> Result<(), CompileFailure> { use cranelift_codegen::ir::{condcodes::IntCC, InstBuilder}; - let OptProvenance::Argument(argument) = expected.provenance else { - return Err(CompileFailure::InvalidArtifact); - }; - let object = opt_use( - builder, - *env.arguments - .get(argument) - .ok_or(CompileFailure::InvalidArtifact)?, - ); + let object = opt_receiver_object(builder, env, expected.provenance)?; let current = if let Some(mode) = expected.typed_mode { emit_opt_typed_metadata_guard( builder, @@ -5634,6 +5848,42 @@ fn emit_opt_packed_loop_revalidate( Ok(()) } +fn opt_array_receiver_provenance(receiver: array_cache::ArrayReceiver) -> OptProvenance { + match receiver { + array_cache::ArrayReceiver::Argument(argument) => { + OptProvenance::Argument(usize::from(argument)) + } + array_cache::ArrayReceiver::Local(local) => OptProvenance::Local(usize::from(local)), + } +} + +/// Constant-pool index of a `push_const`/`push_const8` node. +fn opt_push_const_index(node: &crate::ir::OptimizedNode) -> Option { + let crate::ir::OptimizedNodeKind::Bytecode { opcode } = node.kind() else { + return None; + }; + match opcode.as_ref() { + "push_const8" => node.bytes().get(1).copied().map(u32::from), + "push_const" => Some(u32::from_le_bytes(node.bytes().get(1..5)?.try_into().ok()?)), + _ => None, + } +} + +/// The current borrowed value of the frame slot that keys guarded metadata. +fn opt_receiver_object( + builder: &mut cranelift_frontend::FunctionBuilder<'_>, + env: &OptEnv<'_>, + provenance: OptProvenance, +) -> Result { + let vars = match provenance { + OptProvenance::Argument(argument) => env.arguments.get(argument), + OptProvenance::Local(local) => env.locals.get(local), + _ => None, + } + .ok_or(CompileFailure::InvalidArtifact)?; + Ok(opt_use(builder, *vars)) +} + #[allow(clippy::too_many_arguments)] fn emit_opt_array_loop_hoists( builder: &mut cranelift_frontend::FunctionBuilder<'_>, @@ -5643,7 +5893,7 @@ fn emit_opt_array_loop_hoists( plan: &array_cache::ArrayPlan, preheader: u32, successor: u32, - sources: &mut std::collections::BTreeMap, + sources: &mut std::collections::BTreeMap>, element_layout: crate::abi::ElementLayout, array_query: usize, ir: &OptimizedIr, @@ -5657,13 +5907,8 @@ fn emit_opt_array_loop_hoists( .candidates() .get(hoist.candidate) .ok_or(CompileFailure::InvalidArtifact)?; - let argument = usize::from(candidate.argument); - let object = opt_use( - builder, - *env.arguments - .get(argument) - .ok_or(CompileFailure::InvalidArtifact)?, - ); + let receiver = opt_array_receiver_provenance(candidate.receiver); + let object = opt_receiver_object(builder, env, receiver)?; let guard = ir .blocks() .iter() @@ -5690,7 +5935,7 @@ fn emit_opt_array_loop_hoists( hoist.header, guard, object, - OptProvenance::Argument(argument), + receiver, hoist.header, element_layout, )?, @@ -5703,7 +5948,7 @@ fn emit_opt_array_loop_hoists( hoist.header, guard, object, - OptProvenance::Argument(argument), + receiver, hoist.header, candidate.mode, array_query, @@ -5712,13 +5957,76 @@ fn emit_opt_array_loop_hoists( } crate::runtime::ArrayMode::Generic => return Err(CompileFailure::InvalidArtifact), }; - if sources.insert(hoist.header, source).is_some() { - return Err(CompileFailure::InvalidArtifact); - } + opt_push_hoisted_source(sources, hoist.header, source)?; + } + for hoist in plan + .storage_hoists() + .iter() + .filter(|hoist| hoist.preheader == preheader && hoist.header == successor) + { + let candidate = plan + .candidates() + .get(hoist.candidate) + .ok_or(CompileFailure::InvalidArtifact)?; + let receiver = opt_array_receiver_provenance(candidate.receiver); + let object = opt_receiver_object(builder, env, receiver)?; + let guard = opt_loop_header_guard(ir, hoist.header)?; + let source = emit_opt_typed_metadata_guard( + builder, + env, + provenance, + depth, + hoist.header, + guard, + object, + receiver, + hoist.header, + candidate.mode, + array_query, + false, + )?; + opt_push_hoisted_source(sources, hoist.header, source)?; + } + Ok(()) +} + +fn opt_push_hoisted_source( + sources: &mut std::collections::BTreeMap>, + header: u32, + source: GuardedElementSource, +) -> Result<(), CompileFailure> { + let seeded = sources.entry(header).or_default(); + if seeded.len() >= array_cache::ArrayPlan::MAX_LIVE_SOURCES + || seeded + .iter() + .any(|existing| existing.provenance == source.provenance) + { + return Err(CompileFailure::InvalidArtifact); } + seeded.push(source); Ok(()) } +/// The exact loop-header poll guard that owns every preheader metadata exit. +fn opt_loop_header_guard(ir: &OptimizedIr, header: u32) -> Result { + ir.blocks() + .iter() + .find(|block| block.start_pc() == header) + .and_then(|block| { + block + .nodes() + .iter() + .find_map(|&id| match ir.nodes()[id as usize].kind() { + crate::ir::OptimizedNodeKind::GuardNumeric { + guard, + mid_loop: true, + } => Some(*guard), + _ => None, + }) + }) + .ok_or(CompileFailure::InvalidArtifact) +} + #[allow(clippy::too_many_arguments)] fn emit_opt_array_length( builder: &mut cranelift_frontend::FunctionBuilder<'_>, @@ -5727,7 +6035,8 @@ fn emit_opt_array_length( depth: usize, pc: u32, element_layout: crate::abi::ElementLayout, - guarded_source: &mut Option, + guarded_source: &mut ElementSources, + typed_length: Option<(crate::runtime::ArrayMode, u32, u32, usize)>, ) -> Result { use cranelift_codegen::ir::condcodes::IntCC; use cranelift_codegen::ir::{types, InstBuilder, MemFlags}; @@ -5740,14 +6049,10 @@ fn emit_opt_array_length( .checked_sub(1) .ok_or(CompileFailure::InvalidArtifact)?; let source_provenance = stack_provenance[index]; - if let Some(source) = (*guarded_source).filter(|source| { - source.exact_length - && source.provenance == source_provenance - && matches!( - source_provenance, - OptProvenance::Argument(_) | OptProvenance::Local(_) - ) - }) { + if let Some(source) = guarded_source + .find(source_provenance) + .filter(|source| source.exact_length) + { // This is the V8/JSC-style checked-field path: the preheader guard // established exact logical-length == dense-count and all intervening // effects were admitted as non-reentrant. Preserve the metadata for @@ -5761,6 +6066,34 @@ fn emit_opt_array_length( stack_provenance[index] = OptProvenance::ImmediatePrimitive; return Ok(depth); } + if let Some((mode, guard, block_pc, array_query)) = typed_length { + // The versioned leaf query validates the exact intrinsic getter chain + // and fixed attached backing without running script; a miss resumes + // QuickJS before this `get_length`, whose operand is still borrowed. + let object = opt_use(builder, env.stack[index]); + let source = emit_opt_typed_metadata_guard( + builder, + env, + stack_provenance, + depth, + pc, + guard, + object, + source_provenance, + block_pc, + mode, + array_query, + true, + )?; + let result = OptPair { + payload: builder.ins().sextend(types::I64, source.count), + tag: builder.ins().iconst(types::I64, i64::from(qjs::JS_TAG_INT)), + }; + opt_define(builder, env.stack[index], result); + stack_provenance[index] = OptProvenance::ImmediatePrimitive; + guarded_source.insert(source); + return Ok(depth); + } // A generic lookup borrows its receiver from this existing owning root. // Its output replaces the consumed borrowed stack alias, without creating // an extra receiver owner that would need releasing after an accessor. @@ -5893,7 +6226,7 @@ fn emit_opt_array_length( ); // The logical result is not a dense-storage bounds proof, and the generic // arm may have changed storage through an accessor. - *guarded_source = None; + guarded_source.clear(); Ok(depth) } @@ -5917,7 +6250,7 @@ fn emit_opt_element_get( layout: super::helpers::FrameLayout, element_layout: crate::abi::ElementLayout, block_pc: u32, - guarded_source: &mut Option, + guarded_source: &mut ElementSources, expected_mode: Option, bounds_covered_by_hoist: bool, ) -> Result { @@ -5947,13 +6280,7 @@ fn emit_opt_element_get( .ins() .icmp_imm(IntCC::Equal, key.tag, i64::from(qjs::JS_TAG_INT)); let tags_ok = builder.ins().band(object_ok, key_ok); - let cached = (*guarded_source).filter(|source| { - source.provenance == stack_provenance[object_index] - && matches!( - source.provenance, - OptProvenance::Argument(_) | OptProvenance::Local(_) - ) - }); + let cached = guarded_source.find(stack_provenance[object_index]); let cached_index = cached.map(|_| builder.create_block()); builder .ins() @@ -6319,8 +6646,10 @@ fn emit_opt_element_get( tag: builder.block_params(continuation)[1], }; let params = builder.block_params(continuation); - *guarded_source = cached.or_else(|| { - (expected_mode == Some(crate::runtime::ArrayMode::Packed)).then_some(GuardedElementSource { + // A load neither reenters nor alters storage, so other live tuples stay + // valid. Only an uncached packed load publishes its own metadata. + if cached.is_none() && expected_mode == Some(crate::runtime::ArrayMode::Packed) { + guarded_source.insert(GuardedElementSource { provenance: source_provenance, block_pc, data: params[2], @@ -6328,8 +6657,8 @@ fn emit_opt_element_get( kind: params[4], exact_length: false, typed_mode: None, - }) - }); + }); + } opt_define(builder, stack[object_index], result); stack_provenance[object_index] = OptProvenance::ImmediatePrimitive; stack_provenance[object_index + 1] = OptProvenance::Unknown; @@ -6347,7 +6676,7 @@ fn emit_opt_typed_store( block_pc: u32, mode: crate::runtime::ArrayMode, array_query: usize, - guarded_source: &mut Option, + guarded_source: &mut ElementSources, ) -> Result { use crate::runtime::ArrayMode; use cranelift_codegen::ir::{condcodes::IntCC, types, InstBuilder, MemFlags}; @@ -6373,8 +6702,9 @@ fn emit_opt_typed_store( }; let operands_ok = builder.ins().band(key_int, value_ok); emit_opt_guard_branch(builder, env, provenance, depth, pc, guard, operands_ok)?; - let source = match (*guarded_source) - .filter(|source| source.provenance == provenance[base] && source.typed_mode == Some(mode)) + let source = match guarded_source + .find(provenance[base]) + .filter(|source| source.typed_mode == Some(mode)) { Some(source) => source, None => emit_opt_typed_metadata_guard( @@ -6408,7 +6738,7 @@ fn emit_opt_typed_store( // releases an owner, and fixed nonshared storage cannot change underneath // it. Aliased views observe the write immediately; no element is forwarded. builder.ins().store(MemFlags::new(), scalar, address, 0); - *guarded_source = Some(source); + guarded_source.insert(source); provenance[base..depth].fill(OptProvenance::Unknown); Ok(base) } @@ -6434,7 +6764,7 @@ fn emit_opt_element_put( element_layout: crate::abi::ElementLayout, block_pc: u32, source_provenance: OptProvenance, - guarded_source: &mut Option, + guarded_source: &mut ElementSources, ) -> Result { use cranelift_codegen::ir::condcodes::IntCC; use cranelift_codegen::ir::{types, InstBuilder, MemFlags}; @@ -6732,7 +7062,9 @@ fn emit_opt_element_put( builder.switch_to_block(continuation); let params = builder.block_params(continuation); - *guarded_source = Some(GuardedElementSource { + // In-bounds primitive-over-primitive stores neither grow, reallocate nor + // release; other live tuples (including aliases of this array) stay valid. + guarded_source.insert(GuardedElementSource { provenance: source_provenance, block_pc, count: params[0], diff --git a/jit/src/compiler/optimized/array_cache.rs b/jit/src/compiler/optimized/array_cache.rs index 2f05fb17b..c76730070 100644 --- a/jit/src/compiler/optimized/array_cache.rs +++ b/jit/src/compiler/optimized/array_cache.rs @@ -3,14 +3,28 @@ //! every operation that can reenter, finalize a reference or alter storage. use crate::ir::{ - IntegerRangeAnalysis, KnownFacts, LoopAnalysis, OptimizedIr, ScalarHeapEffect, + FrameSlot, IntegerRangeAnalysis, KnownFacts, LoopAnalysis, OptimizedIr, ScalarHeapEffect, ScalarHeapOperation, ScalarValueId, }; use crate::runtime::{ArrayAccess, ArrayFeedbackSnapshot, ArrayMode}; +/// The frame slot whose current value is the guarded receiver. +/// +/// Lowering keys cached metadata by the provenance of the stack operand, so a +/// receiver is identified by the one frame slot that the operand was read +/// from. An entry argument uses the SSA identity proof. A local receiver +/// (for example `const ints = buffers.ints`) is admitted only when it is the +/// unique frame slot holding the operand's SSA value at every planned access, +/// and when no node of the enclosing natural loop writes that local. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(super) enum ArrayReceiver { + Argument(u16), + Local(u16), +} + #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub(super) struct ArrayCandidate { - pub argument: u16, + pub receiver: ArrayReceiver, pub mode: ArrayMode, /// Typed length requires the metadata query to guard observable lookup. pub needs_length: bool, @@ -48,16 +62,32 @@ pub(super) struct ArrayLoopHoist { pub length_node: u32, } +/// Typed-array storage metadata (mode, fixed attached backing, count and data +/// pointer, but not the observable `length` lookup) guarded once on the +/// loop-entry edge. Accesses keep their per-access index guards; the hoist only +/// removes the repeated leaf query for a loop-invariant receiver, like JSC's +/// hoisted CheckStructure/GetButterfly for typed-array views. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(super) struct ArrayStorageHoist { + pub candidate: usize, + pub preheader: u32, + pub header: u32, +} + #[derive(Debug, Default)] pub(super) struct ArrayPlan { candidates: Vec, accesses: Vec>, hoists: Vec, + storage_hoists: Vec, invalidations: Vec, } impl ArrayPlan { const MAX_CANDIDATES: usize = 16; + /// Metadata tuples lowering may keep live at once. Every loop header seeds + /// at most this many hoisted sources. + pub(super) const MAX_LIVE_SOURCES: usize = 4; #[allow(clippy::too_many_arguments)] pub(super) fn analyze( @@ -96,7 +126,7 @@ impl ArrayPlan { #[derive(Clone, Copy)] struct Proposed { node: u32, - argument: u16, + receiver: ArrayReceiver, mode: ArrayMode, access: ArrayAccess, index: Option, @@ -152,7 +182,11 @@ impl ArrayPlan { if access == ArrayAccess::Length && mode != ArrayMode::Packed && !typed_length_guard { continue; } - let Some(argument) = facts.entry_argument(base) else { + let receiver = if let Some(argument) = facts.entry_argument(base) { + ArrayReceiver::Argument(argument) + } else if let Some(local) = unique_local_receiver(graph, facts, operation, base) { + ArrayReceiver::Local(local) + } else { continue; }; let Some(block) = loops.block_for_node(node.id()) else { @@ -196,7 +230,7 @@ impl ArrayPlan { }; proposed.push(Proposed { node: node.id(), - argument, + receiver, mode, access, index, @@ -213,17 +247,17 @@ impl ArrayPlan { if spend(&mut work, conflict_work).is_none() { return Self::default(); } - let mut conflicting_arguments = Vec::new(); + let mut conflicting_receivers = Vec::new(); for left in &proposed { if proposed .iter() - .any(|right| right.argument == left.argument && right.mode != left.mode) - && !conflicting_arguments.contains(&left.argument) + .any(|right| right.receiver == left.receiver && right.mode != left.mode) + && !conflicting_receivers.contains(&left.receiver) { - conflicting_arguments.push(left.argument); + conflicting_receivers.push(left.receiver); } } - proposed.retain(|access| !conflicting_arguments.contains(&access.argument)); + proposed.retain(|access| !conflicting_receivers.contains(&access.receiver)); // A reentrant operation in the natural loop can replace storage, detach // a buffer, or run arbitrary script. Feedback never authorizes carrying @@ -268,6 +302,15 @@ impl ArrayPlan { { continue; } + // A local receiver is loop-invariant only when no loop node can + // rebind it. The metadata cache is keyed by that frame slot. + if let ArrayReceiver::Local(local) = candidate.receiver { + if graph.effect_for_node(node.id()) + == ScalarHeapEffect::FrameWrite(FrameSlot::Local(local)) + { + return false; + } + } if graph.effect_for_node(node.id()) != ScalarHeapEffect::Reentrant || non_reentrant(node.id()) || proposed_nodes[node.id() as usize] @@ -283,7 +326,7 @@ impl ArrayPlan { let mut accesses: Vec> = vec![None; nodes.len()]; for proposed in proposed { let candidate = if let Some(index) = candidates.iter().position(|candidate| { - candidate.argument == proposed.argument && candidate.mode == proposed.mode + candidate.receiver == proposed.receiver && candidate.mode == proposed.mode }) { index } else { @@ -291,7 +334,7 @@ impl ArrayPlan { return Self::default(); } candidates.push(ArrayCandidate { - argument: proposed.argument, + receiver: proposed.receiver, mode: proposed.mode, needs_length: false, }); @@ -365,7 +408,13 @@ impl ArrayPlan { covered_any = true; } } - if covered_any { + if covered_any + && hoists + .iter() + .filter(|hoist: &&ArrayLoopHoist| hoist.header == natural_loop.header()) + .count() + < Self::MAX_LIVE_SOURCES + { hoists.push(ArrayLoopHoist { candidate, preheader, @@ -376,6 +425,46 @@ impl ArrayPlan { } } } + // A loop-invariant typed receiver without a hoisted length (for + // example the destination of `floats[i] = ...`) still needs only one + // storage query per loop entry. The admission above already rejected + // loops containing a write to the receiver's frame slot or any + // uncertified reentrant node; polls revalidate the tuple. + let mut storage_hoists = Vec::new(); + for (candidate, typed) in candidates.iter().enumerate() { + if !matches!(typed.mode, ArrayMode::Int32 | ArrayMode::Float64) { + continue; + } + for natural_loop in loops.loops() { + let Some(preheader) = natural_loop.preheader() else { + continue; + }; + let header = natural_loop.header(); + if hoists + .iter() + .any(|hoist| hoist.candidate == candidate && hoist.header == header) + { + continue; + } + let used = accesses.iter().flatten().any(|access| { + access.candidate == candidate + && access.loop_header == header + && access.access != ArrayAccess::Length + }); + let live = hoists.iter().filter(|hoist| hoist.header == header).count() + + storage_hoists + .iter() + .filter(|hoist: &&ArrayStorageHoist| hoist.header == header) + .count(); + if used && live < Self::MAX_LIVE_SOURCES { + storage_hoists.push(ArrayStorageHoist { + candidate, + preheader, + header, + }); + } + } + } let mut invalidations = Vec::with_capacity(nodes.len()); for node in nodes { let planned = accesses @@ -405,6 +494,7 @@ impl ArrayPlan { candidates, accesses, hoists, + storage_hoists, invalidations, } } @@ -428,6 +518,9 @@ impl ArrayPlan { pub(super) fn hoists(&self) -> &[ArrayLoopHoist] { &self.hoists } + pub(super) fn storage_hoists(&self) -> &[ArrayStorageHoist] { + &self.storage_hoists + } pub(super) fn invalidates_before(&self, node: u32) -> bool { self.invalidations .get(node as usize) @@ -445,6 +538,36 @@ impl ArrayPlan { } } +/// The single local that holds `base` in the operation's pre-effect frame +/// state. Any other frame slot holding the same SSA value makes the stack +/// operand's lowering provenance ambiguous, so the site is left unplanned. +/// Lowering independently requires the operand provenance to be this local. +fn unique_local_receiver( + graph: &crate::ir::ScalarGraph, + facts: &KnownFacts, + operation: ScalarHeapOperation, + base: ScalarValueId, +) -> Option { + let state = graph.frame_state_for_node(operation.frame_state_node())?; + if state + .arguments + .iter() + .any(|&value| facts.same_value(value, base)) + { + return None; + } + let mut holders = state + .locals + .iter() + .enumerate() + .filter(|(_, &value)| facts.same_value(value, base)); + let (local, _) = holders.next()?; + if holders.next().is_some() { + return None; + } + u16::try_from(local).ok() +} + fn spend(work: &mut usize, cost: usize) -> Option<()> { *work = work.checked_sub(cost)?; Some(()) @@ -587,7 +710,7 @@ mod tests { assert_eq!( plan.candidates(), &[ArrayCandidate { - argument: 0, + receiver: ArrayReceiver::Argument(0), mode: ArrayMode::Packed, needs_length: true }] @@ -752,4 +875,68 @@ mod tests { ); } } + + #[test] + fn loop_invariant_local_typed_receiver_is_a_hoisted_candidate() { + let (ir, feedback) = fixture( + "(function(o){const a=o.a;let s=0;for(let i=0;i Self { match (self, other) { (Self::Unreached, value) | (value, Self::Unreached) => value, - (Self::Argument(a), Self::Argument(b)) if a == b => self, + (a, b) if a == b => a, _ => Self::Unknown, } } @@ -267,7 +270,9 @@ impl KnownFacts { }) .map_or(Identity::Unknown, |source| identities[source.index()]) } - _ => Identity::Unknown, + // Every other producer defines its own identity. An + // unpreserved frame read is a fresh definition as well. + _ => u32::try_from(index).map_or(Identity::Unknown, Identity::Value), }; let next = identities[index].join(next); changed |= identities[index] != next; @@ -306,6 +311,20 @@ impl KnownFacts { } } + /// Both SSA values denote the same definition on every path: an entry + /// argument, or one producer reached only through agreeing Phis and frame + /// reads preserved by the caller's emitted-guard contract. + pub(crate) fn same_value(&self, left: ScalarValueId, right: ScalarValueId) -> bool { + match ( + self.identities.get(left.index()), + self.identities.get(right.index()), + ) { + (Some(Identity::Argument(a)), Some(Identity::Argument(b))) => a == b, + (Some(Identity::Value(a)), Some(Identity::Value(b))) => a == b, + _ => false, + } + } + pub(crate) const fn bytes_per_value() -> usize { core::mem::size_of::() } diff --git a/jit/tests/typed_receivers.rs b/jit/tests/typed_receivers.rs new file mode 100644 index 000000000..4aad4d695 --- /dev/null +++ b/jit/tests/typed_receivers.rs @@ -0,0 +1,420 @@ +//! Typed-array receivers that are not entry arguments. +//! +//! `float64array-traversal` loads both views from an object property into +//! `const` locals before its loop. Tier 2 must guard those local receivers +//! once on the loop-entry edge (revalidating after polls) rather than running +//! the `length` accessor through the generic property bridge per iteration, +//! and every changed receiver must still match the same-version interpreter. +#![cfg(all( + feature = "compiler", + feature = "test-support", + target_endian = "little", + any(target_os = "linux", target_os = "macos"), + any(target_arch = "x86_64", target_arch = "aarch64") +))] + +use rquickjs::{Context, Runtime}; +use rquickjs_jit::bytecode::VerifyLimits; +use rquickjs_jit::compiler::optimized::Tier2Compiler; +use rquickjs_jit::runtime::{ + ArrayAccess, ArrayHazards, ArrayMode, FeedbackTable, FunctionKey, ObservedType, + PropertyAttributes, PrototypeDependencyToken, ShapeFeedbackTable, ShapeObservation, ShapeToken, +}; +use rquickjs_jit::test_support::SnapshotFixture; +use rquickjs_jit::{Jit, JitConfig}; +use std::time::{Duration, Instant}; + +unsafe extern "C" { + fn JS_JitGetHelperCount( + rt: *mut rquickjs_core::qjs::JSRuntime, + helper_id: u32, + count: *mut u64, + ) -> i32; +} + +const TRAVERSAL: &str = "(function convertAndSum(iterations,seed,buffers){const ints=buffers.ints;const floats=buffers.floats;let sum=seed;for(let i=0;i Result { + let fixture = SnapshotFixture::compile(source); + let verified = fixture.snapshot().verify(VerifyLimits::default()).unwrap(); + let key = FunctionKey::new( + verified.snapshot().function_id(), + verified.snapshot().generation(), + ); + let return_pc = verified + .instructions() + .iter() + .find(|instruction| instruction.opcode().name() == "return") + .unwrap() + .pc(); + let names: Vec<_> = verified + .instructions() + .iter() + .map(|instruction| (instruction.pc(), instruction.opcode().name())) + .collect(); + let mut shapes = ShapeFeedbackTable::new(3); + let mut offset = 0; + for &(pc, name) in &names { + if name == "get_field" { + shapes.observe( + key, + pc, + ShapeObservation::new( + ShapeToken::new(0x1000, 7), + PrototypeDependencyToken::new(0, 0), + offset, + PropertyAttributes::WRITABLE, + ObservedType::Object, + ), + ); + offset += 1; + } + } + let mut feedback = FeedbackTable::new(64, 2); + for _ in 0..32 { + feedback.observe_call( + key, + &[ + ObservedType::Int32, + ObservedType::Int32, + ObservedType::Object, + ], + ); + // The first element load reads the Int32 source; later loads read + // back the Float64 destination after its store. + let mut loads = 0; + for &(pc, name) in &names { + match name { + "get_length" => { + feedback.observe_array( + key, + pc, + ArrayAccess::Length, + ArrayMode::Int32, + ArrayHazards::NONE, + ); + } + "get_array_el" => { + let mode = if loads == 0 { + ArrayMode::Int32 + } else { + ArrayMode::Float64 + }; + loads += 1; + feedback.observe_array(key, pc, ArrayAccess::Load, mode, ArrayHazards::NONE); + } + "put_array_el" => { + feedback.observe_array( + key, + pc, + ArrayAccess::Store, + ArrayMode::Float64, + ArrayHazards::NONE, + ); + } + "mul" | "add" => { + feedback.observe_binary( + key, + pc, + ObservedType::Float64, + ObservedType::Float64, + ObservedType::Float64, + Default::default(), + ); + } + "lt" => { + feedback.observe_binary( + key, + pc, + ObservedType::Int32, + ObservedType::Int32, + ObservedType::Bool, + Default::default(), + ); + } + _ => {} + } + } + feedback.observe_return(key, return_pc, ObservedType::Float64); + } + Tier2Compiler::host(id) + .lower_with_feedback_for_test( + &verified, + key, + &feedback.snapshot(id).with_properties(shapes.snapshot(key)), + ) + .map_err(|error| format!("{error:?}; {names:?}")) +} + +/// Leaf metadata queries use a 24-byte `JSJitArrayMetadata` stack slot each. +fn metadata_queries(clif: &str) -> usize { + clif.matches("explicit_slot 24").count() +} + +/// Generic GET_PROPERTY calls pass `JS_ATOM_length` as an immediate operand. +fn generic_length_lookups(clif: &str) -> usize { + let atom = format!("= {}", rquickjs_core::qjs::JS_ATOM_length); + clif.lines() + .filter(|line| line.contains("call_indirect") && line.contains(&atom)) + .count() +} + +#[test] +fn local_typed_receivers_hoist_length_and_storage_out_of_the_loop() { + let clif = lower_traversal(TRAVERSAL, 901).unwrap(); + assert_eq!( + generic_length_lookups(&clif), + 0, + "a guarded local Int32Array length still runs the accessor bridge: {clif}" + ); + // One preheader query per receiver (ints with the intrinsic length flag, + // floats storage-only) plus one revalidation each on the cold poll path. + // No query remains on the hot loop path. + assert_eq!(metadata_queries(&clif), 4, "{clif}"); + assert!( + clif.contains("load.i32") && clif.contains("load.f64") && clif.contains("store"), + "cached typed element accesses missing: {clif}" + ); + // Only the floats store/load keep a bounds check; the ints load is + // covered by the exact hoisted length. + assert!( + !clif.contains("icmp.i32 ult"), + "the range-proven ints load retained a bounds check: {clif}" + ); +} + +#[test] +fn rebound_or_aliased_local_receivers_keep_the_generic_length_path() { + for source in [ + // The receiver local is rebound inside the loop. + "(function f(iterations,seed,buffers){let ints=buffers.ints;const floats=buffers.floats;let sum=seed;for(let i=0;i 0 || metadata_queries(&clif) < 4, + "an unproven local receiver was hoisted: {source}: {clif}" + ); + } + } +} + +fn evaluate_without_jit(source: &str, expression: &str) -> String { + let runtime = Runtime::new().unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| { + ctx.eval::<(), _>(source).unwrap(); + ctx.eval::(expression).unwrap() + }) +} + +const SOURCE: &str = r#" + function convertAndSum(iterations, seed, buffers) { + const ints = buffers.ints; + const floats = buffers.floats; + let sum = seed; + for (let i = 0; i < ints.length; i++) { + floats[i] = ints[i] * 0.25 + 0.5; + sum += floats[i]; + } + return sum; + } + function makeBuffers(n) { + const ints = new Int32Array(n); + for (let i = 0; i < n; i++) ints[i] = (i * 17) & 0xffff; + return { ints: ints, floats: new Float64Array(n) }; + } + globalThis.stable = makeBuffers(64); +"#; + +fn prepared_traversal() -> (Runtime, Jit, Context) { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .call_threshold(2) + .loop_threshold(4) + .force_optimized_for_test(true) + .stress_gc(true) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| ctx.eval::<(), _>(SOURCE)).unwrap(); + let deadline = Instant::now() + Duration::from_secs(60); + loop { + let before = jit.metrics(); + context + .with(|ctx| ctx.eval::("convertAndSum(0, 0, stable)")) + .unwrap(); + let after = jit.metrics(); + if after.tier2_entries > before.tier2_entries + && after.deopts == before.deopts + && after.pending_worker_jobs == 0 + { + return (runtime, jit, context); + } + jit.poll(); + assert!( + Instant::now() < deadline, + "local typed traversal never reached Tier2: {after:?}" + ); + std::thread::sleep(Duration::from_millis(1)); + } +} + +fn helper_count(context: &Context, helper: u32) -> u64 { + let rt = + context.with(|ctx| unsafe { rquickjs_core::qjs::JS_GetRuntime(ctx.as_raw().as_ptr()) }); + let mut count = 0; + assert_eq!(unsafe { JS_JitGetHelperCount(rt, helper, &mut count) }, 0); + count +} + +#[test] +fn production_local_typed_traversal_avoids_the_length_accessor_bridge() { + let (_runtime, jit, context) = prepared_traversal(); + let expected = evaluate_without_jit(SOURCE, "String(convertAndSum(0, 0, stable))"); + let get_property = rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_GET_PROPERTY; + let before_helpers = helper_count(&context, get_property); + let before = jit.metrics(); + for _ in 0..16 { + let actual = context + .with(|ctx| ctx.eval::("String(convertAndSum(0, 0, stable))")) + .unwrap(); + assert_eq!(actual, expected); + } + let after = jit.metrics(); + assert_eq!(after.tier2_entries - before.tier2_entries, 16, "{after:?}"); + assert_eq!(after.deopts, before.deopts, "{after:?}"); + assert_eq!(after.native_entries, after.native_exits); + // 16 calls x 64 iterations previously crossed the generic lookup bridge + // for every `ints.length`. Only the two receiver loads may use a helper. + let lookups = helper_count(&context, get_property) - before_helpers; + assert!( + lookups <= 32, + "per-iteration generic property lookups remain: {lookups}" + ); +} + +#[test] +fn changed_local_receivers_match_the_same_version_interpreter() { + // Each probe changes one premise of the compiled loop: receiver mode, + // extent, own/prototype `length`, attachment, aliasing, the property + // load itself or the seed representation. + const CASES: &str = r#"JSON.stringify([ + convertAndSum(0, 0, makeBuffers(0)), + convertAndSum(0, 0, makeBuffers(3)), + convertAndSum(0, 0, makeBuffers(1000)), + convertAndSum(0, 0, {ints: new Int32Array([1, 2, 3]), floats: new Float64Array(2)}), + convertAndSum(0, 0, {ints: new Float64Array([1.5, 2.5]), floats: new Float64Array(2)}), + convertAndSum(0, 0, {ints: [1, 2, 3], floats: new Float64Array(3)}), + convertAndSum(0, 0, {ints: new Int32Array([7, 9]), floats: new Int32Array(2)}), + convertAndSum(0, 0, {ints: new Int32Array([7, 9]), floats: [0, 0]}), + (() => { const a = new Int32Array([4, 5]); return convertAndSum(0, 0, {ints: a, floats: a}); })(), + (() => { const buffer = new ArrayBuffer(64); const ints = new Int32Array(buffer); for (let i = 0; i < 16; i++) ints[i] = i * 3; return convertAndSum(0, 0, {ints: ints, floats: new Float64Array(buffer)}); })(), + (() => { const b = makeBuffers(4); Object.defineProperty(b.ints, 'length', {value: 2}); return convertAndSum(0, 0, b); })(), + (() => { const b = makeBuffers(4); b.floats.buffer.transfer(); return convertAndSum(0, 0, b); })(), + (() => { const b = makeBuffers(4); b.ints.buffer.transfer(); return convertAndSum(0, 0, b); })(), + (() => { let hits = 0; const b = makeBuffers(4); return [convertAndSum(0, 0, {get ints() { hits++; return b.ints; }, floats: b.floats}), hits]; })(), + convertAndSum(0, 'seed:', makeBuffers(2)), + convertAndSum(0, 0, stable), + (() => { Object.defineProperty(Object.getPrototypeOf(Int32Array.prototype), 'length', {get() { return 2; }, configurable: true}); return convertAndSum(0, 0, makeBuffers(8)); })() + ])"#; + let expected = evaluate_without_jit(SOURCE, CASES); + let (_runtime, jit, context) = prepared_traversal(); + let before = jit.metrics(); + let actual = context.with(|ctx| ctx.eval::(CASES)).unwrap(); + let after = jit.metrics(); + assert_eq!(actual, expected); + assert_eq!(after.native_entries, after.native_exits); + assert!( + after.tier2_entries > before.tier2_entries, + "the probes never entered the prepared Tier2 body: {before:?} -> {after:?}" + ); + assert!( + after.deopts > before.deopts, + "changed receivers must leave the guarded path: {before:?} -> {after:?}" + ); +} + +#[test] +fn local_receiver_loop_revalidates_storage_after_observable_poll() { + for detach_source in [false, true] { + local_receiver_poll_case(detach_source); + } +} + +fn local_receiver_poll_case(detach_source: bool) { + use rquickjs::qjs; + use std::sync::{ + atomic::{AtomicBool, Ordering}, + Arc, + }; + let (runtime, jit, context) = prepared_traversal(); + const N: usize = 1_000_000; + let (raw_ctx, raw_floats, raw_buffer) = context.with(|ctx| { + // `fill` builds the large source in C; each term is 3 * 0.25 + 0.5. + ctx.eval::<(), _>(format!( + "globalThis.big = {{ints: new Int32Array({N}).fill(3), floats: new Float64Array({N})}}; globalThis.detachTarget = {}.buffer;", + if detach_source { "big.ints" } else { "big.floats" } + )) + .unwrap(); + let big: rquickjs::Object = ctx.globals().get("big").unwrap(); + let floats: rquickjs::Object = big.get("floats").unwrap(); + let buffer: rquickjs::Object = ctx.globals().get("detachTarget").unwrap(); + ( + ctx.as_raw().as_ptr() as usize, + unsafe { qjs::JS_VALUE_GET_PTR(floats.as_value().as_raw()) } as usize, + unsafe { qjs::JS_VALUE_GET_PTR(buffer.as_value().as_raw()) } as usize, + ) + }); + let changed = Arc::new(AtomicBool::new(false)); + let observed = changed.clone(); + // Detach once the loop has completed at least one store. The globally + // rooted views stay live; detaching uses the C API without running script. + runtime.set_interrupt_handler(Some(Box::new(move || unsafe { + let ctx = raw_ctx as *mut qjs::JSContext; + let floats = qjs::JS_MKPTR(qjs::JS_TAG_OBJECT, raw_floats as *mut core::ffi::c_void); + let value = qjs::JS_GetPropertyUint32(ctx, floats, 1); + let started = qjs::JS_VALUE_GET_TAG(value) == qjs::JS_TAG_FLOAT64 + && qjs::JS_VALUE_GET_FLOAT64(value) != 0.0; + qjs::JS_FreeValue(ctx, value); + if started && !observed.swap(true, Ordering::SeqCst) { + let buffer = qjs::JS_MKPTR(qjs::JS_TAG_OBJECT, raw_buffer as *mut core::ffi::c_void); + qjs::JS_DetachArrayBuffer(ctx, buffer); + } + false + }))); + let before = jit.metrics(); + let result = context.with(|ctx| ctx.eval::("convertAndSum(0, 0, big)")); + runtime.set_interrupt_handler(None); + let result = result.unwrap(); + assert!( + changed.load(Ordering::SeqCst), + "no poll observed completed stores" + ); + let full = N as f64 * 1.25; + if detach_source { + // A detached source reports length 0, ending the loop early. + assert!( + result.is_finite() && result < full, + "loop used a stale source length after detach: {result}" + ); + } else { + // Every read of a detached destination is undefined. + assert!( + result.is_nan(), + "loop used stale destination storage after detach: {result}" + ); + } + assert!(jit.metrics().tier2_entries > before.tier2_entries); + assert_eq!(jit.metrics().native_entries, jit.metrics().native_exits); +} From 88a728bcca3b3154a9f335eda8b735b407d56456 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 16:48:47 +0800 Subject: [PATCH 06/54] perf(jit): add callback-free native entry for steady functions QuickJS patch 0023 (ABI 1.25) lets the backend grant a bounded budget to the idle pc=0 handle it caches after a DONE exit. While the budget and a shared epoch hold, calls of that function (including C->JS callbacks such as Array.prototype.sort comparators) reuse the handle without call feedback, record_hot, entry_cache_epoch, native_enter or native_exit. Executions are counted in backend-owned shared state and marked JS_JIT_FRAME_FAST_ENTRY; non-DONE exits still report native_exit tagged JS_JIT_EXIT_FAST_UNPAIRED and are never re-cached. Calls that never reach native code no longer evict another function's admissible idle handle. The production backend grants 255 calls only to steady entries: an installed Tier 2 artifact whose profitability trial is decided, or an installed baseline whose optimizing tier is terminally blacklisted. Each renewal is a full call, so hot counts, call/return feedback, timing and maintenance are sampled instead of paid per call. Jit::metrics adds the shared counters, so native entries/exits and Tier 2 entries stay exact. Maintenance now skips re-evaluating deferred Tier 2 candidates while their inputs (feedback and shape-feedback versions, installed count, element loop samples) are unchanged, instead of re-snapshotting feedback on every pass. Shape feedback gains a version that ignores repeats and reordering. Co-Authored-By: Claude Opus 5.5 (1M context) --- core/src/runtime/jit.rs | 33 ++ jit/src/abi.rs | 5 +- jit/src/lib.rs | 230 +++++++++- jit/src/runtime/shape_feedback.rs | 53 +++ jit/tests/abi.rs | 2 +- jit/tests/fast_entry.rs | 399 ++++++++++++++++++ jit/tests/native_boundary.rs | 1 + sys/build_support/patch.rs | 7 +- sys/patches/0023-fast-native-entry.patch | 316 ++++++++++++++ sys/patches/README.md | 15 + sys/src/bindings/aarch64-apple-darwin.rs | 69 ++- sys/src/bindings/aarch64-pc-windows-msvc.rs | 69 ++- sys/src/bindings/aarch64-unknown-linux-gnu.rs | 69 ++- .../bindings/aarch64-unknown-linux-musl.rs | 69 ++- sys/src/bindings/wasm32-wasip1.rs | 69 ++- sys/src/bindings/x86_64-apple-darwin.rs | 69 ++- sys/src/bindings/x86_64-pc-windows-gnu.rs | 69 ++- sys/src/bindings/x86_64-pc-windows-msvc.rs | 69 ++- sys/src/bindings/x86_64-unknown-linux-gnu.rs | 69 ++- sys/src/bindings/x86_64-unknown-linux-musl.rs | 69 ++- sys/tests/jit_patch.rs | 27 +- sys/tests/typed_array_guard.rs | 2 +- 22 files changed, 1746 insertions(+), 34 deletions(-) create mode 100644 jit/tests/fast_entry.rs create mode 100644 sys/patches/0023-fast-native-entry.patch diff --git a/core/src/runtime/jit.rs b/core/src/runtime/jit.rs index 7d35f9a8b..8d4323a37 100644 --- a/core/src/runtime/jit.rs +++ b/core/src/runtime/jit.rs @@ -327,6 +327,23 @@ pub unsafe trait JitBackend: Send + 'static { 0 } + /// Grants callback-free executions to the idle handle QuickJS just cached + /// after a `DONE` exit (ABI 1.25). Leaving `grant.budget` zero keeps every + /// call on the full callback path. A nonzero budget must point + /// `grant.state` at backend-owned state that stays at a fixed address until + /// `runtime_detach`; its epoch follows the [`Self::entry_cache_epoch`] + /// rules. Granted executions skip call feedback, hot probes, the epoch + /// query and `native_enter`, carry `JS_JIT_FRAME_FAST_ENTRY`, and report + /// `native_exit` only for non-`DONE` exits, tagged with + /// `JS_JIT_EXIT_FAST_UNPAIRED`. This query must not reenter JavaScript. + fn entry_fast_grant( + &mut self, + _id: u64, + _generation: u64, + _grant: &mut qjs::JSJitFastEntryGrant, + ) { + } + /// Records the exact boundary immediately before a published native entry. fn native_enter(&mut self, _id: u64, _generation: u64, _pc: u32) {} @@ -429,6 +446,21 @@ unsafe extern "C" fn entry_cache_epoch(opaque: *mut c_void) -> u64 { state.backend.entry_cache_epoch() } +unsafe extern "C" fn entry_fast_grant( + opaque: *mut c_void, + id: u64, + generation: u64, + grant: *mut qjs::JSJitFastEntryGrant, +) { + if grant.is_null() { + return; + } + let state = unsafe { BackendState::from_opaque(opaque) }; + state + .backend + .entry_fast_grant(id, generation, unsafe { &mut *grant }); +} + unsafe extern "C" fn native_enter(opaque: *mut c_void, id: u64, generation: u64, pc: u32) { let state = unsafe { BackendState::from_opaque(opaque) }; state.backend.native_enter(id, generation, pc); @@ -479,6 +511,7 @@ static BACKEND_VTABLE: qjs::JSJitBackendVTable = qjs::JSJitBackendVTable { native_exit: Some(native_exit), record_feedback: Some(record_feedback), entry_cache_epoch: Some(entry_cache_epoch), + entry_fast_grant: Some(entry_fast_grant), }; /// Owns the token for one backend allocation stored by the raw runtime. diff --git a/jit/src/abi.rs b/jit/src/abi.rs index b39c65a23..fd1681829 100644 --- a/jit/src/abi.rs +++ b/jit/src/abi.rs @@ -5,7 +5,7 @@ use core::{fmt, mem}; use rquickjs_core::qjs; pub const ABI_MAJOR: u16 = 1; -pub const ABI_MINOR: u16 = 24; +pub const ABI_MINOR: u16 = 25; pub const SOURCE_REVISION: u64 = 0xfd0a_0210_b7be_0095; pub const OPCODE_FINGERPRINT: u64 = qjs::QJSJIT_GENERATED_OPCODE_FINGERPRINT; @@ -723,6 +723,7 @@ fn backend_vtable_layout_fingerprint() -> u64 { mem::offset_of!(qjs::JSJitBackendVTable, native_exit), mem::offset_of!(qjs::JSJitBackendVTable, record_feedback), mem::offset_of!(qjs::JSJitBackendVTable, entry_cache_epoch), + mem::offset_of!(qjs::JSJitBackendVTable, entry_fast_grant), ] { hash = layout_field(hash, offset, mem::size_of::()); } @@ -1073,7 +1074,7 @@ mod inline_api_tests { fn inline_recovery_requires_the_matching_main_abi_minor() { let native = AbiInfo::query_linked().unwrap(); assert_eq!(ABI_MINOR, native.minor()); - assert_eq!(native.minor(), 24); + assert_eq!(native.minor(), 25); assert!(unsafe { qjs::JS_JitGetInlineAPI(0) }.is_null()); } diff --git a/jit/src/lib.rs b/jit/src/lib.rs index d9a15bd96..0c999ae0d 100644 --- a/jit/src/lib.rs +++ b/jit/src/lib.rs @@ -113,10 +113,70 @@ fn artifact_environment( } } +/// Backend-owned ABI 1.25 fast-entry state shared with QuickJS. +/// +/// QuickJS reads `epoch` and increments the counters with plain stores on +/// the runtime thread while executing callback-free native calls. Rust only +/// loads them, so a reader on another thread may observe a stale count. The +/// allocation outlives the attached backend, which keeps one `Arc` clone. +#[repr(C)] +#[derive(Debug)] +struct FastEntryShared { + struct_size: u32, + reserved: u32, + epoch: AtomicU64, + entries: AtomicU64, + exits: AtomicU64, + optimized_entries: AtomicU64, +} + +const _: () = { + use core::mem::{align_of, offset_of, size_of}; + use rquickjs_core::qjs::JSJitFastEntryState as Raw; + assert!(size_of::() == size_of::()); + assert!(align_of::() >= align_of::()); + assert!(offset_of!(FastEntryShared, epoch) == offset_of!(Raw, epoch)); + assert!(offset_of!(FastEntryShared, entries) == offset_of!(Raw, entries)); + assert!(offset_of!(FastEntryShared, exits) == offset_of!(Raw, exits)); + assert!(offset_of!(FastEntryShared, optimized_entries) == offset_of!(Raw, optimized_entries)); +}; + +impl FastEntryShared { + fn new(epoch: u64) -> Self { + Self { + struct_size: core::mem::size_of::() as u32, + reserved: 0, + epoch: AtomicU64::new(epoch), + entries: AtomicU64::new(0), + exits: AtomicU64::new(0), + optimized_entries: AtomicU64::new(0), + } + } + + /// The counters and epoch are atomics, so QuickJS may update them through + /// this pointer while Rust holds shared references. + /// Unused by targets without the native production backend. + #[allow(dead_code)] + fn as_raw(&self) -> *mut rquickjs_core::qjs::JSJitFastEntryState { + (self as *const Self).cast_mut().cast() + } + + /// Adds callback-free executions to a published snapshot. + fn add_to(&self, metrics: &mut JitMetrics) { + let entries = self.entries.load(Ordering::Relaxed); + let exits = self.exits.load(Ordering::Relaxed); + let optimized = self.optimized_entries.load(Ordering::Relaxed); + metrics.native_entries = metrics.native_entries.saturating_add(entries); + metrics.native_exits = metrics.native_exits.saturating_add(exits); + metrics.tier2_entries = metrics.tier2_entries.saturating_add(optimized); + } +} + /// Owns the guard that keeps a JIT backend attached to a runtime. #[derive(Debug)] pub struct Jit { metrics: Arc>, + fast_entry: Arc, config: JitConfig, _guard: rquickjs_core::runtime::RuntimeJitGuard, #[cfg(all(feature = "test-support", feature = "compiler"))] @@ -265,6 +325,48 @@ impl Jit { let backend = NoopBackend { _config: config.clone(), }; + #[cfg(not(all( + feature = "compiler", + any( + all( + target_os = "macos", + target_endian = "little", + any(target_arch = "x86_64", target_arch = "aarch64") + ), + all( + target_os = "windows", + target_endian = "little", + any(target_arch = "x86_64", target_arch = "aarch64") + ), + all( + target_os = "linux", + target_endian = "little", + any(target_arch = "x86_64", target_arch = "aarch64") + ) + ) + )))] + let fast_entry = Arc::new(FastEntryShared::new(0)); + #[cfg(all( + feature = "compiler", + any( + all( + target_os = "macos", + target_endian = "little", + any(target_arch = "x86_64", target_arch = "aarch64") + ), + all( + target_os = "windows", + target_endian = "little", + any(target_arch = "x86_64", target_arch = "aarch64") + ), + all( + target_os = "linux", + target_endian = "little", + any(target_arch = "x86_64", target_arch = "aarch64") + ) + ) + ))] + let fast_entry = Arc::clone(&backend.fast_entry); let guard = match runtime.attach_jit_backend(backend) { Ok(guard) => guard, Err(error) => { @@ -274,6 +376,7 @@ impl Jit { }; Ok(Self { metrics, + fast_entry, config, _guard: guard, #[cfg(all(feature = "test-support", feature = "compiler"))] @@ -454,10 +557,13 @@ impl Jit { /// Returns the metrics associated with this backend guard. pub fn metrics(&self) -> JitMetrics { - self.metrics + let mut snapshot = self + .metrics .lock() .unwrap_or_else(|p| p.into_inner()) - .clone() + .clone(); + self.fast_entry.add_to(&mut snapshot); + snapshot } /// Performs bounded installation and reclamation work on the runtime thread. @@ -597,6 +703,23 @@ unsafe impl rquickjs_core::runtime::JitBackend for NoopBackend {} #[cfg(all(feature = "compiler", not(target_family = "wasm")))] const HOT_MAINTENANCE_INTERVAL: u32 = 64; +/// Inputs of a deferred Tier-2 candidate decision: feedback lattices, the +/// installed-artifact count (callee resolution and direct-call readiness), and +/// whether the element-loop gate has its eight baseline samples. +#[cfg(all(feature = "compiler", not(target_family = "wasm")))] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +struct Tier2ScanInputs { + feedback: u64, + shapes: u64, + installed: u64, + element_samples: u64, +} + +/// Callback-free executions granted per full call of a steady native entry. +/// Every renewal still runs hot, feedback, timing and maintenance callbacks. +#[cfg(all(feature = "compiler", not(target_family = "wasm")))] +const FAST_ENTRY_BUDGET: u32 = 255; + /// Feedback pc used for return types observed at native `DONE` exits. #[cfg(all(feature = "compiler", not(target_family = "wasm")))] const NATIVE_RETURN_FEEDBACK_PC: u32 = u32::MAX; @@ -665,6 +788,11 @@ struct ProductionBackend { last_scan_feedback_version: u64, last_scan_installed: u64, last_refresh_scan: Option<(u64, u64)>, + /// Tier-2 candidates whose last scan deferred them for missing feedback or + /// an unresolved callee, keyed by the inputs that decision depended on. + /// Rescanning with equal inputs would repeat the same feedback snapshot + /// and reach the same deferral, so maintenance skips it. + tier2_deferred: rustc_hash::FxHashMap, direct_refresh_probes: std::collections::HashMap, queue_reasons: std::collections::HashMap, prequeue_backoff: std::collections::HashMap, @@ -679,6 +807,9 @@ struct ProductionBackend { // retain their own tier in execution_starts across recursive replacement. entry_tiers: rustc_hash::FxHashMap, entry_cache_epoch: u64, + /// ABI 1.25 state granted to cached steady-state entries. Its epoch + /// mirrors `entry_cache_epoch`. + fast_entry: Arc, // C brackets each native invocation with a synchronous enter/exit pair. // Keep active records through retirement and preserve each invocation's tier. execution_starts: Vec<(runtime::FunctionKey, std::time::Instant, runtime::Tier)>, @@ -1326,6 +1457,7 @@ unsafe extern "C" fn production_entry_trampoline( // Capture trusted entry context before native execution can modify the // frame. Rejection must not dereference pointers supplied by a bad exit. let validated_ctx = frame.ctx; + let fast_entry = frame.flags & qjs::JS_JIT_FRAME_FAST_ENTRY != 0; let validated_bytecode_start = frame.bytecode_start; let validated_arg_buf = frame.arg_buf; let validated_var_buf = frame.var_buf; @@ -1373,7 +1505,9 @@ unsafe extern "C" fn production_entry_trampoline( .side_path_entries .fetch_add(1, Ordering::Release); } - if exit.kind == qjs::JSJitExitKind_JS_JIT_EXIT_DONE { + // A granted fast DONE exit reports no native_exit to consume this + // handoff; its steady return type is resampled on the next full call. + if exit.kind == qjs::JSJitExitKind_JS_JIT_EXIT_DONE && !fast_entry { if let Some(observed) = observed_value_type(frame.result) { pin.validation .mark_native_return(pin.key.id, pin.key.generation, observed); @@ -2098,6 +2232,7 @@ impl ProductionBackend { last_scan_feedback_version: u64::MAX, last_scan_installed: u64::MAX, last_refresh_scan: None, + tier2_deferred: rustc_hash::FxHashMap::default(), direct_refresh_probes: std::collections::HashMap::new(), queue_reasons: std::collections::HashMap::new(), prequeue_backoff: std::collections::HashMap::new(), @@ -2110,6 +2245,7 @@ impl ProductionBackend { execution_starts: Vec::new(), entry_tiers: rustc_hash::FxHashMap::default(), entry_cache_epoch: 1, + fast_entry: Arc::new(FastEntryShared::new(1)), execution_profiles: rustc_hash::FxHashMap::default(), profitability_evaluations: 0, profitability_approved: 0, @@ -2171,12 +2307,53 @@ impl ProductionBackend { } } + /// The tier of a cached entry for which per-call callbacks can no longer + /// change any decision: either its optimized artifact is installed and + /// the bounded profitability trial is over (or cannot run without + /// baseline samples), or its baseline artifact is installed and the + /// optimizing tier is terminally blacklisted, so neither hot counts nor + /// baseline timing feed a pending promotion. Hot counts, call feedback + /// and timing are then only sampled on the full call that renews each + /// fast-entry grant. + fn fast_entry_steady(&self, key: runtime::FunctionKey) -> Option { + if self.feedback_disabled.contains(&key) { + return None; + } + let tier = *self.entry_tiers.get(&key)?; + let optimizing = self.coordinator.tier_state(key, runtime::Tier::Optimizing); + let steady = match tier { + runtime::Tier::Optimizing => { + matches!(optimizing, runtime::CompileState::Installed(_)) + && self.execution_profiles.get(&key).is_some_and(|profile| { + profile.tier2_trial_decided || profile.baseline_executions == 0 + }) + } + runtime::Tier::Baseline => { + // Baseline-only refresh probing consumes every hot event. + self.config.tier_policy() != JitTierPolicy::BaselineOnly + && optimizing == runtime::CompileState::Blacklisted + && matches!( + self.coordinator.tier_state(key, runtime::Tier::Baseline), + runtime::CompileState::Installed(_) + ) + && !self.optimizing_requested.contains(&key) + && !self.optimizing_snapshots.contains_key(&key) + } + }; + steady.then_some(tier) + } + fn invalidate_entry_cache(&mut self) { // Epoch zero permanently disables caching after exhaustion: never let // an old handle become admissible again through wraparound. if self.entry_cache_epoch != 0 { self.entry_cache_epoch = self.entry_cache_epoch.checked_add(1).unwrap_or(0); } + // Granted QuickJS handles compare this copy instead of querying + // `entry_cache_epoch`, so publish every change before returning. + self.fast_entry + .epoch + .store(self.entry_cache_epoch, Ordering::Relaxed); } fn maintenance(&mut self) { @@ -2275,6 +2452,18 @@ impl ProductionBackend { let Some(snapshot) = self.optimizing_snapshots.get(&key) else { continue; }; + let scan_inputs = Tier2ScanInputs { + feedback: self.feedback.version(), + shapes: self.shape_feedback.version(), + installed: self.coordinator.installed_count(), + element_samples: self + .execution_profiles + .get(&key) + .map_or(0, |profile| profile.baseline_executions.min(8)), + }; + if self.tier2_deferred.get(&key) == Some(&scan_inputs) { + continue; + } let observed = self .feedback .snapshot(self.clock.max(1)) @@ -2321,6 +2510,7 @@ impl ProductionBackend { * the first Tier2 artifact. Self-recursive and non-specializable * calls keep their existing generic lowering. */ if direct_call_pending { + self.tier2_deferred.insert(key, scan_inputs); continue; } // A stable call link can fund a bounded optimizing trial before @@ -2335,6 +2525,7 @@ impl ProductionBackend { && observed.property_at(instruction.pc()).is_none() }); if property_feedback_pending { + self.tier2_deferred.insert(key, scan_inputs); continue; } /* A native-to-native call through the generic CALL bridge still @@ -2443,6 +2634,7 @@ impl ProductionBackend { }) }); if !observed_element_loop || !completed_numeric_returns { + self.tier2_deferred.insert(key, scan_inputs); continue; } } @@ -2574,6 +2766,7 @@ impl ProductionBackend { { self.optimizing_snapshots.insert(key, snapshot); } else { + self.tier2_deferred.remove(&key); self.tier2_sources.insert(key, snapshot); } } @@ -3277,7 +3470,33 @@ unsafe impl rquickjs_core::runtime::JitBackend for ProductionBackend { .push((key, std::time::Instant::now(), tier)); } + fn entry_fast_grant( + &mut self, + id: u64, + generation: u64, + grant: &mut rquickjs_core::qjs::JSJitFastEntryGrant, + ) { + let key = runtime::FunctionKey::new(id, generation); + if self.entry_cache_epoch == 0 { + return; + } + let Some(tier) = self.fast_entry_steady(key) else { + return; + }; + grant.budget = FAST_ENTRY_BUDGET; + grant.flags = if tier == runtime::Tier::Optimizing { + rquickjs_core::qjs::JS_JIT_FAST_ENTRY_OPTIMIZED + } else { + 0 + }; + grant.state = self.fast_entry.as_raw(); + } + fn native_exit(&mut self, id: u64, generation: u64, pc: u32, exit_kind: u32) { + // A granted execution skipped native_enter; its non-DONE exit still + // reports here for deopt, retry and exception accounting. + let unpaired = exit_kind & rquickjs_core::qjs::JS_JIT_EXIT_FAST_UNPAIRED != 0; + let exit_kind = exit_kind & !rquickjs_core::qjs::JS_JIT_EXIT_FAST_UNPAIRED; self.native_exits = self.native_exits.saturating_add(1); let key = runtime::FunctionKey::new(id, generation); if exit_kind == rquickjs_core::qjs::JSJitExitKind_JS_JIT_EXIT_DONE { @@ -3296,7 +3515,7 @@ unsafe impl rquickjs_core::runtime::JitBackend for ProductionBackend { .execution_starts .last() .copied() - .filter(|(active, _, _)| *active == key) + .filter(|(active, _, _)| !unpaired && *active == key) { self.execution_starts.pop(); let elapsed = start.elapsed().as_nanos().try_into().unwrap_or(u64::MAX); @@ -3419,6 +3638,7 @@ unsafe impl rquickjs_core::runtime::JitBackend for ProductionBackend { self.optimizing_requested.remove(&key); self.optimizing_hotness.remove(&key); self.optimizing_snapshots.remove(&key); + self.tier2_deferred.remove(&key); self.tier2_sources.remove(&key); self.entry_tiers.remove(&key); self.coordinator.retire(key); @@ -3426,6 +3646,8 @@ unsafe impl rquickjs_core::runtime::JitBackend for ProductionBackend { } fn runtime_detach(&mut self) { + self.entry_cache_epoch = 0; + self.fast_entry.epoch.store(0, Ordering::Relaxed); self.workers.shutdown(&mut self.coordinator); self.coordinator.shutdown(); self.maintenance(); diff --git a/jit/src/runtime/shape_feedback.rs b/jit/src/runtime/shape_feedback.rs index a3ac49d57..dea7486fb 100644 --- a/jit/src/runtime/shape_feedback.rs +++ b/jit/src/runtime/shape_feedback.rs @@ -157,6 +157,8 @@ struct ShapeFeedbackKey { pub struct ShapeFeedbackTable { polymorphic_limit: usize, sites: BTreeMap, + /// Advances whenever any site's state or observation list changes. + version: u64, } impl ShapeFeedbackTable { @@ -164,9 +166,18 @@ impl ShapeFeedbackTable { Self { polymorphic_limit: polymorphic_limit.max(1), sites: BTreeMap::new(), + version: 0, } } + /// Monotonic change counter for site states and observation sets. + /// Repeating a recorded observation only refreshes its recency, so + /// steady-state probes do not invalidate consumers that cache decisions + /// derived from snapshots. + pub const fn version(&self) -> u64 { + self.version + } + pub fn observe( &mut self, function: FunctionKey, @@ -183,6 +194,15 @@ impl ShapeFeedbackTable { if site.state == ShapeFeedbackState::Megamorphic { return site.state; } + // Each observation leaves at most one entry per shape identity, so an + // exact match of the newest entry is a repeat that changes nothing, + // and an exact match elsewhere only moves it to the end. + if site.observations.last() == Some(&observation) { + return site.state; + } + if !site.observations.contains(&observation) { + self.version = self.version.wrapping_add(1); + } // A generation change for the same shape identity is invalidation, // not polymorphism. Forget the stale layout so recompilation never // emits a guard for a class/layout version that can no longer match. @@ -277,6 +297,39 @@ mod tests { assert!(table.get(function, 8).unwrap().observations().is_empty()); } + #[test] + fn version_tracks_observation_sets_but_not_repeats_or_recency() { + let mut table = ShapeFeedbackTable::new(3); + let function = FunctionKey::new(4, 1); + let start = table.version(); + table.observe(function, 8, observation(1, 1, 8)); + let first = table.version(); + assert_ne!(first, start); + table.observe(function, 8, observation(1, 1, 8)); + assert_eq!(table.version(), first); + table.observe(function, 8, observation(2, 1, 16)); + let second = table.version(); + assert_ne!(second, first); + // Alternating known shapes only reorders the site. + table.observe(function, 8, observation(1, 1, 8)); + table.observe(function, 8, observation(2, 1, 16)); + assert_eq!(table.version(), second); + assert_eq!( + table.get(function, 8).unwrap().state(), + ShapeFeedbackState::Polymorphic + ); + // A new layout for a known shape identity replaces it. + table.observe(function, 8, observation(1, 1, 24)); + assert_ne!(table.version(), second); + let third = table.version(); + table.observe(function, 8, observation(3, 1, 32)); + table.observe(function, 8, observation(4, 1, 40)); + assert_ne!(table.version(), third); + let megamorphic = table.version(); + table.observe(function, 8, observation(5, 1, 48)); + assert_eq!(table.version(), megamorphic); + } + #[test] fn observation_requires_exact_shape_and_prototype_generations() { let observed = observation(5, 7, 40); diff --git a/jit/tests/abi.rs b/jit/tests/abi.rs index bb3580e33..231b608b0 100644 --- a/jit/tests/abi.rs +++ b/jit/tests/abi.rs @@ -574,7 +574,7 @@ fn helper_abi_is_one_canonical_versioned_table_in_c_bindgen_and_rust() { fn helper_abi_fields_are_append_only_tails() { use rquickjs_core::qjs; - assert_eq!(qjs::QJSJIT_ABI_MINOR, 24); + assert_eq!(qjs::QJSJIT_ABI_MINOR, 25); assert_eq!(qjs::QJSJIT_RUNTIME_API_MAJOR, 1); assert_eq!(qjs::QJSJIT_RUNTIME_API_MINOR, 9); assert_eq!(qjs::QJSJIT_HELPER_ABI_VERSION, 1); diff --git a/jit/tests/fast_entry.rs b/jit/tests/fast_entry.rs new file mode 100644 index 000000000..399c26548 --- /dev/null +++ b/jit/tests/fast_entry.rs @@ -0,0 +1,399 @@ +//! ABI 1.25 callback-free native entry: QuickJS may run a cached, granted +//! handle without per-call backend callbacks, but never past its budget, a +//! changed epoch, suspension, or a non-DONE exit. + +use std::sync::{ + atomic::{AtomicU64, AtomicUsize, Ordering}, + Arc, Mutex, +}; + +use rquickjs::{Context, Function, Runtime}; +use rquickjs_core::{qjs, runtime::JitBackend}; +use rquickjs_jit::{abi::JitExitExt, bytecode::CompileSnapshot}; + +#[derive(Default)] +struct Calls { + acquisitions: AtomicUsize, + releases: AtomicUsize, + enters: AtomicUsize, + exits: Mutex>, + grants: AtomicUsize, + hot_events: AtomicUsize, + call_feedback: AtomicUsize, + executions: AtomicUsize, + fast_executions: AtomicUsize, + /// Execution number (1-based) that returns RETRY instead of DONE. + retry_at: AtomicUsize, +} + +/// Leaked for the process lifetime so native code can reach it without a pin. +fn calls() -> &'static Calls { + Box::leak(Box::default()) +} + +struct Pin { + calls: &'static Calls, +} + +unsafe extern "C" fn native_counted(frame: *mut qjs::JSJitExecFrame) -> qjs::JSJitExit { + let frame = unsafe { &mut *frame }; + let calls = unsafe { &*frame.entry.pin.cast::() }.calls; + let execution = calls.executions.fetch_add(1, Ordering::SeqCst) + 1; + if frame.flags & qjs::JS_JIT_FRAME_FAST_ENTRY != 0 { + calls.fast_executions.fetch_add(1, Ordering::SeqCst); + } + if calls.retry_at.load(Ordering::SeqCst) == execution { + return qjs::JSJitExit::retry_interpreter(); + } + frame.result = qjs::JS_MKVAL(qjs::JS_TAG_INT, 42); + qjs::JSJitExit::done() +} + +struct StateBox(*mut qjs::JSJitFastEntryState); +// The state is only touched on the runtime thread and by the test after the +// calls return. +unsafe impl Send for StateBox {} + +struct FastBackend { + key: (u64, u64), + calls: &'static Calls, + epoch: Arc, + budget: u32, + state: StateBox, + malformed_state: bool, +} + +unsafe impl JitBackend for FastBackend { + fn record_hot(&mut self, _event: &qjs::JSJitHotEvent) -> u32 { + self.calls.hot_events.fetch_add(1, Ordering::SeqCst); + 0 + } + + fn record_feedback(&mut self, event: &qjs::JSJitFeedbackEvent) { + if event.kind == qjs::JSJitFeedbackKind_JS_JIT_FEEDBACK_CALL + && event.flags & qjs::JS_JIT_FEEDBACK_CALL_SITE == 0 + { + self.calls.call_feedback.fetch_add(1, Ordering::SeqCst); + } + } + + fn acquire_entry(&mut self, id: u64, generation: u64, pc: u32) -> qjs::JSJitEntryHandle { + let mut entry: qjs::JSJitEntryHandle = unsafe { std::mem::zeroed() }; + entry.struct_size = std::mem::size_of_val(&entry) as u32; + if (id, generation) == self.key && pc == 0 { + self.calls.acquisitions.fetch_add(1, Ordering::SeqCst); + entry.entry = Some(native_counted); + entry.helper_abi_version = qjs::QJSJIT_HELPER_ABI_VERSION; + entry.pin = Box::into_raw(Box::new(Pin { calls: self.calls })).cast(); + } + entry + } + + fn release_entry(&mut self, entry: qjs::JSJitEntryHandle) { + if !entry.pin.is_null() { + unsafe { drop(Box::from_raw(entry.pin.cast::())) }; + self.calls.releases.fetch_add(1, Ordering::SeqCst); + } + } + + fn entry_cache_epoch(&self) -> u64 { + self.epoch.load(Ordering::SeqCst) + } + + fn entry_fast_grant(&mut self, id: u64, generation: u64, grant: &mut qjs::JSJitFastEntryGrant) { + assert_eq!((id, generation), self.key); + self.calls.grants.fetch_add(1, Ordering::SeqCst); + let state = unsafe { &mut *self.state.0 }; + state.epoch = self.epoch.load(Ordering::SeqCst); + if self.malformed_state { + state.struct_size = 1; + } + grant.budget = self.budget; + grant.flags = qjs::JS_JIT_FAST_ENTRY_OPTIMIZED; + grant.state = self.state.0; + } + + fn native_enter(&mut self, _id: u64, _generation: u64, _pc: u32) { + self.calls.enters.fetch_add(1, Ordering::SeqCst); + } + + fn native_exit(&mut self, _id: u64, _generation: u64, _pc: u32, exit_kind: u32) { + self.calls.exits.lock().unwrap().push(exit_kind); + } +} + +struct Harness { + runtime: Runtime, + context: Context, + calls: &'static Calls, + epoch: Arc, + state: *mut qjs::JSJitFastEntryState, + guard: Option, +} + +impl Harness { + fn new(budget: u32, malformed_state: bool) -> Self { + let runtime = Runtime::new().unwrap(); + let context = Context::full(&runtime).unwrap(); + let captured = context.with(|ctx| { + ctx.eval::<(), _>("globalThis.target = function target(x) { return 1 }") + .unwrap(); + let function = ctx.globals().get::<_, Function>("target").unwrap(); + unsafe { + CompileSnapshot::capture_raw(ctx.as_raw().as_ptr(), function.as_value().as_raw()) + } + .unwrap() + }); + let calls = calls(); + let epoch = Arc::new(AtomicU64::new(1)); + let state = Box::into_raw(Box::new(qjs::JSJitFastEntryState { + struct_size: std::mem::size_of::() as u32, + reserved: 0, + epoch: 1, + entries: 0, + exits: 0, + optimized_entries: 0, + })); + let guard = runtime + .attach_jit_backend(FastBackend { + key: (captured.function_id(), captured.generation()), + calls, + epoch: Arc::clone(&epoch), + budget, + state: StateBox(state), + malformed_state, + }) + .unwrap(); + Self { + runtime, + context, + calls, + epoch, + state, + guard: Some(guard), + } + } + + fn call(&self, expected: i32) { + self.context.with(|ctx| { + let function: Function = ctx.globals().get("target").unwrap(); + assert_eq!(function.call::<_, i32>((7,)).unwrap(), expected); + }); + } + + fn state(&self) -> qjs::JSJitFastEntryState { + unsafe { *self.state } + } + + fn set_epoch(&self, epoch: u64) { + self.epoch.store(epoch, Ordering::SeqCst); + unsafe { (*self.state).epoch = epoch }; + } + + fn enters(&self) -> usize { + self.calls.enters.load(Ordering::SeqCst) + } + + fn exits(&self) -> Vec { + self.calls.exits.lock().unwrap().clone() + } + + fn fast(&self) -> usize { + self.calls.fast_executions.load(Ordering::SeqCst) + } +} + +impl Drop for Harness { + fn drop(&mut self) { + drop(self.guard.take()); + let _ = &self.runtime; + unsafe { drop(Box::from_raw(self.state)) }; + } +} + +const DONE: u32 = qjs::JSJitExitKind_JS_JIT_EXIT_DONE; + +#[test] +fn granted_calls_skip_every_callback_until_the_budget_expires() { + let harness = Harness::new(3, false); + for _ in 0..10 { + harness.call(42); + } + let calls = harness.calls; + // Calls 1, 5 and 9 take the full path (the first acquires); each grants + // three callback-free executions. + assert_eq!(calls.executions.load(Ordering::SeqCst), 10); + assert_eq!(harness.fast(), 7); + assert_eq!(harness.enters(), 3); + assert_eq!(harness.exits(), vec![DONE; 3]); + assert_eq!(calls.acquisitions.load(Ordering::SeqCst), 1); + assert_eq!(calls.grants.load(Ordering::SeqCst), 3); + assert_eq!(calls.hot_events.load(Ordering::SeqCst), 3); + assert_eq!(calls.call_feedback.load(Ordering::SeqCst), 3); + let state = harness.state(); + assert_eq!( + (state.entries, state.exits, state.optimized_entries), + (7, 7, 7) + ); + assert_eq!(calls.releases.load(Ordering::SeqCst), 0); + drop(harness); + assert_eq!(calls.releases.load(Ordering::SeqCst), 1); +} + +#[test] +fn an_epoch_change_stops_granted_calls_before_the_next_execution() { + let harness = Harness::new(100, false); + harness.call(42); + harness.call(42); + assert_eq!((harness.enters(), harness.fast()), (1, 1)); + harness.set_epoch(2); + harness.call(42); + // The stale handle is released and reacquired on the full path. + assert_eq!((harness.enters(), harness.fast()), (2, 1)); + assert_eq!(harness.calls.acquisitions.load(Ordering::SeqCst), 2); + assert_eq!(harness.calls.releases.load(Ordering::SeqCst), 1); + harness.call(42); + assert_eq!((harness.enters(), harness.fast()), (2, 2)); + // Zero disables fast calls outright even for a matching cached epoch. + unsafe { (*harness.state).epoch = 0 }; + harness.call(42); + assert_eq!(harness.fast(), 2); + assert_eq!(harness.enters(), 3); +} + +#[test] +fn a_non_done_fast_exit_reports_an_unpaired_native_exit_and_is_not_cached() { + let harness = Harness::new(100, false); + harness.calls.retry_at.store(3, Ordering::SeqCst); + harness.call(42); + harness.call(42); + // Execution 3 is fast and retries in the interpreter, which returns 1. + harness.call(1); + assert_eq!(harness.fast(), 2); + assert_eq!( + harness.exits(), + vec![ + DONE, + qjs::JSJitExitKind_JS_JIT_EXIT_RETRY_INTERPRETER | qjs::JS_JIT_EXIT_FAST_UNPAIRED + ] + ); + let state = harness.state(); + assert_eq!((state.entries, state.exits), (2, 1)); + assert_eq!(harness.calls.releases.load(Ordering::SeqCst), 1); + harness.call(42); + assert_eq!(harness.calls.acquisitions.load(Ordering::SeqCst), 2); + assert_eq!(harness.enters(), 2); +} + +#[test] +fn suspension_and_malformed_grants_keep_the_full_callback_path() { + let harness = Harness::new(100, false); + harness.call(42); + harness.guard.as_ref().unwrap().suspend().unwrap(); + harness.call(1); + harness.guard.as_ref().unwrap().resume().unwrap(); + harness.call(42); + assert_eq!(harness.fast(), 1); + + let oversized = Harness::new(qjs::JS_JIT_FAST_ENTRY_MAX_BUDGET + 1, false); + let malformed = Harness::new(100, true); + for harness in [&oversized, &malformed] { + for _ in 0..4 { + harness.call(42); + } + assert_eq!(harness.fast(), 0); + assert_eq!(harness.enters(), 4); + assert_eq!(harness.calls.acquisitions.load(Ordering::SeqCst), 1); + } +} + +#[test] +fn calls_without_native_code_keep_another_functions_granted_handle() { + let harness = Harness::new(100, false); + harness.call(42); + harness.context.with(|ctx| { + // `other` never acquires native code; its call must neither run the + // granted handle nor evict it before the nested `target` call. + ctx.eval::<(), _>("globalThis.other = function other() { return target(1) + 1 }") + .unwrap(); + let other: Function = ctx.globals().get("other").unwrap(); + assert_eq!(other.call::<_, i32>(()).unwrap(), 43); + }); + assert_eq!(harness.fast(), 1); + assert_eq!(harness.enters(), 1); + assert_eq!(harness.calls.acquisitions.load(Ordering::SeqCst), 1); + assert_eq!(harness.calls.releases.load(Ordering::SeqCst), 0); +} + +/// Production tiering: a steady Tier-2 sort comparator re-entered from C runs +/// callback-free while metrics stay exact, and a guard failure on a granted +/// call still leaves native code with the interpreter's result. +#[cfg(all( + any(target_os = "linux", target_os = "macos"), + target_endian = "little", + any(target_arch = "x86_64", target_arch = "aarch64") +))] +#[test] +fn steady_sort_comparator_uses_granted_entries_with_exact_metrics() { + use rquickjs_jit::{Jit, JitConfig}; + + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach(&runtime, JitConfig::default()).unwrap(); + let context = Context::full(&runtime).unwrap(); + context + .with(|ctx| { + ctx.eval::<(), _>( + "function compare(left, right) { return left.score - right.score; }\ + function run(count, text) {\ + const items = [];\ + for (let i = 0; i < count; i += 1) {\ + const score = (i * 7919) % 1000;\ + items.push({ score: text ? String(score) : score });\ + }\ + items.sort(compare);\ + return Number(items[0].score) + Number(items[count - 1].score) * 1000;\ + }", + ) + }) + .unwrap(); + let run = |text: bool| { + context.with(|ctx| { + let run: Function = ctx.globals().get("run").unwrap(); + run.call::<_, i32>((1_000, text)).unwrap() + }) + }; + + let mut steady = None; + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(20); + while std::time::Instant::now() < deadline { + jit.poll(); + let before = jit.metrics(); + assert_eq!(run(false), 999_000); + let after = jit.metrics(); + let entries = after.native_entries - before.native_entries; + let acquisitions = after.native_acquisitions - before.native_acquisitions; + let tier2 = after.tier2_entries - before.tier2_entries; + // About 10k comparator calls. The per-call callback path reacquired + // a handle about every 32 calls; a grant renews every 256. + if entries >= 5_000 && tier2 >= 5_000 && acquisitions * 64 < entries { + steady = Some((before, after)); + break; + } + std::thread::sleep(std::time::Duration::from_micros(50)); + } + let (before, after) = steady.unwrap_or_else(|| panic!("never steady: {:?}", jit.metrics())); + assert_eq!(after.native_entries, after.native_exits, "{after:?}"); + assert!( + after.benefit_recordings - before.benefit_recordings + < (after.native_entries - before.native_entries) / 64, + "granted calls must not time every execution: before={before:?}, after={after:?}" + ); + + // String scores leave the numeric Tier-2 path on granted calls; the + // interpreter must finish each comparison with the same answer. + assert_eq!(run(true), 999_000); + assert_eq!(run(false), 999_000); + let last = jit.metrics(); + assert!(last.deopts > after.deopts, "{last:?}"); + assert_eq!(last.native_entries, last.native_exits, "{last:?}"); +} diff --git a/jit/tests/native_boundary.rs b/jit/tests/native_boundary.rs index 2b706a064..4a6eadc31 100644 --- a/jit/tests/native_boundary.rs +++ b/jit/tests/native_boundary.rs @@ -631,6 +631,7 @@ unsafe fn attempt_backend_change( native_exit: None, record_feedback: None, entry_cache_epoch: None, + entry_fast_grant: None, }; unsafe { qjs::JS_SetJitBackend(rt, &replacement, std::ptr::null_mut()) } } diff --git a/sys/build_support/patch.rs b/sys/build_support/patch.rs index 41317e2f2..a3a1e7846 100644 --- a/sys/build_support/patch.rs +++ b/sys/build_support/patch.rs @@ -1,7 +1,7 @@ use std::{fs, io, path::Path}; const QUICKJS_BASELINE_FNV64: u64 = 0x3302_116a_b0fc_269c; -const EXPECTED_PATCHES: [(&str, u64); 19] = [ +const EXPECTED_PATCHES: [(&str, u64); 20] = [ ("0001-rquickjs-jit.patch", 0x18d7_40cc_6943_30cf), ("0002-runtime-feedback.patch", 0xe4c1_ed52_ee5d_d469), ("0003-element-layout.patch", 0xa12c_ccd2_7420_e880), @@ -21,6 +21,7 @@ const EXPECTED_PATCHES: [(&str, u64); 19] = [ ("0018-inline-frame-recovery.patch", 0xb971d61edccde229), ("0019-array-feedback.patch", 0x11c3cb13a8297dd4), ("0020-typed-array-guard.patch", 0x5d10_38a3_8bc9_9821), + ("0023-fast-native-entry.patch", 0x9446_e4a5_9620_e8de), ]; pub(crate) const BASELINE_FILES: [&str; 19] = [ "api-test.c", @@ -60,8 +61,8 @@ const PATCHED_FILE_FINGERPRINTS: [(&str, u64); 21] = [ ("list.h", 0xb337_70f7_b76d_a3d8), ("quickjs-atom.h", 0x30b4_9116_b6a2_aa99), ("quickjs-c-atomics.h", 0x490b_0f29_f631_3fc0), - ("quickjs.c", 0xc535_b5f1_48e7_9b27), - ("quickjs-jit.h", 0x0188_04ad_21d2_95a7), + ("quickjs.c", 0x2d00_8a5b_5f6b_bc14), + ("quickjs-jit.h", 0xb390_03aa_6f10_6176), ("quickjs-jit-helpers.h", 0xa469_e80f_be47_404c), ("quickjs-opcode.h", 0x3d05_cfdf_5cf7_2930), ("quickjs.h", 0x4831_2cde_9c2f_a5ee), diff --git a/sys/patches/0023-fast-native-entry.patch b/sys/patches/0023-fast-native-entry.patch new file mode 100644 index 000000000..1e827f557 --- /dev/null +++ b/sys/patches/0023-fast-native-entry.patch @@ -0,0 +1,316 @@ +diff --git a/quickjs.c b/quickjs.c +--- a/quickjs.c ++++ b/quickjs.c +@@ -365,6 +365,10 @@ + uint64_t jit_cached_entry_generation; + uint64_t jit_cached_entry_epoch; + bool jit_entry_cache_suppressed; ++ /* Callback-free executions left for the cached handle (ABI 1.25). */ ++ uint32_t jit_cached_entry_budget; ++ uint32_t jit_cached_entry_fast_flags; ++ JSJitFastEntryState *jit_fast_state; + uint32_t jit_backend_active_depth; + uint8_t jit_suspended; + uint32_t jit_active_stack_map_count; +@@ -2306,7 +2310,8 @@ + hash = QJSJIT_LAYOUT_FIELD(hash, JSJitBackendVTable, native_enter); + hash = QJSJIT_LAYOUT_FIELD(hash, JSJitBackendVTable, native_exit); + hash = QJSJIT_LAYOUT_FIELD(hash, JSJitBackendVTable, record_feedback); +- return QJSJIT_LAYOUT_FIELD(hash, JSJitBackendVTable, entry_cache_epoch); ++ hash = QJSJIT_LAYOUT_FIELD(hash, JSJitBackendVTable, entry_cache_epoch); ++ return QJSJIT_LAYOUT_FIELD(hash, JSJitBackendVTable, entry_fast_grant); + } + + static uint64_t qjsjit_abi_info_layout(void) +@@ -3169,6 +3174,8 @@ + { + JSJitEntryHandle entry = rt->jit_cached_entry; + memset(&rt->jit_cached_entry, 0, sizeof(rt->jit_cached_entry)); ++ rt->jit_cached_entry_budget = 0; ++ rt->jit_cached_entry_fast_flags = 0; + if (entry.pin) { + bool was_suppressed = rt->jit_entry_cache_suppressed; + rt->jit_entry_cache_suppressed = true; +@@ -3197,6 +3204,7 @@ + old_vtable = rt->jit_backend_vtable; + old_opaque = rt->jit_backend_opaque; + qjsjit_discard_cached_entry(rt); ++ rt->jit_fast_state = NULL; + rt->jit_backend_vtable = NULL; + rt->jit_backend_opaque = NULL; + { +@@ -3220,6 +3228,7 @@ + + rt->jit_backend_vtable = vtable; + rt->jit_backend_opaque = opaque; ++ rt->jit_fast_state = NULL; + return JS_JIT_BACKEND_OK; + } + +@@ -19031,7 +19040,8 @@ + JSObject *function; + JSFunctionBytecode *b; + uint32_t allowed_flags = JS_JIT_FUNCTION_STRICT | +- JS_JIT_FRAME_STRESS_GC | JS_JIT_FRAME_SIDE_PATH_HIT; ++ JS_JIT_FRAME_STRESS_GC | JS_JIT_FRAME_SIDE_PATH_HIT | ++ JS_JIT_FRAME_FAST_ENTRY; + + if (!frame || !frame->rt || !frame->ctx) + return qjsjit_reject_helper(frame, "invalid JIT helper frame"); +@@ -20105,6 +20115,54 @@ + JSValueConst this_obj, JSValueConst new_target, int argc, + JSValueConst *argv, int flags, JSJitInlineState *resume); + ++/* ABI 1.25: an idle cached pc=0 handle with a remaining backend grant runs ++ * without call feedback, hot, epoch, enter or exit callbacks. The shared ++ * epoch replaces the entry_cache_epoch query; any backend state change that ++ * could make the handle inadmissible advances it first. */ ++static inline bool qjsjit_fast_entry_ready(JSRuntime *rt, ++ const JSFunctionBytecode *b) ++{ ++ const JSJitFastEntryState *state = rt->jit_fast_state; ++ return rt->jit_cached_entry_budget != 0 && state && ++ rt->jit_cached_entry.pin && b->jit_function_id != 0 && ++ rt->jit_cached_entry_id == b->jit_function_id && ++ rt->jit_cached_entry_generation == b->jit_generation && ++ !rt->jit_suspended && !b->jit_feedback_disabled && ++ !rt->jit_entry_cache_suppressed && state->epoch != 0 && ++ state->epoch == rt->jit_cached_entry_epoch; ++} ++ ++/* Called with the backend active and the handle already cached. Any malformed ++ * grant leaves the handle on the full callback path. */ ++static void qjsjit_grant_fast_entry(JSRuntime *rt, ++ const JSJitBackendVTable *vtable, ++ void *opaque, uint64_t id, ++ uint64_t generation) ++{ ++ JSJitFastEntryGrant grant; ++ const JSJitFastEntryState *state; ++ ++ rt->jit_cached_entry_budget = 0; ++ rt->jit_cached_entry_fast_flags = 0; ++ if (!vtable->entry_fast_grant || rt->jit_suspended) ++ return; ++ memset(&grant, 0, sizeof(grant)); ++ grant.struct_size = sizeof(grant); ++ vtable->entry_fast_grant(opaque, id, generation, &grant); ++ state = grant.state; ++ if (grant.struct_size != sizeof(grant) || grant.reserved != 0 || ++ grant.budget == 0 || grant.budget > JS_JIT_FAST_ENTRY_MAX_BUDGET || ++ (grant.flags & ~JS_JIT_FAST_ENTRY_OPTIMIZED) != 0 || !state || ++ state->struct_size != sizeof(*state) || state->reserved != 0 || ++ (rt->jit_fast_state && rt->jit_fast_state != state) || ++ !rt->jit_cached_entry.pin || rt->jit_cached_entry_id != id || ++ rt->jit_cached_entry_generation != generation) ++ return; ++ rt->jit_fast_state = grant.state; ++ rt->jit_cached_entry_budget = grant.budget; ++ rt->jit_cached_entry_fast_flags = grant.flags; ++} ++ + static void qjsjit_inline_activate(JSRuntime *rt, JSJitExecFrame *view) + { + rt->jit_active_frame = view; +@@ -20461,6 +20519,7 @@ + size_t alloca_size; + #ifdef CONFIG_JIT_ABI + int jit_call_recorded = 0; ++ bool jit_fast_entry = false; + uint32_t jit_hot_kind = JS_JIT_HOT_CALL, jit_probe_budget = 1; + #endif + +@@ -20625,7 +20684,11 @@ + #endif + + #ifdef CONFIG_JIT_ABI +- if (argc >= 0 && argc <= 8 && qjsjit_feedback_enabled(rt, b)) { ++ /* The grant covers this invocation's call feedback and hot event. */ ++ jit_fast_entry = qjsjit_fast_entry_ready(rt, b); ++ if (jit_fast_entry) { ++ jit_call_recorded = 1; ++ } else if (argc >= 0 && argc <= 8 && qjsjit_feedback_enabled(rt, b)) { + uint32_t call_types[8]; + for (i = 0; i < argc; i++) { + qjsjit_unpoison_slot(&arg_buf[i]); +@@ -20714,11 +20777,32 @@ + const uint8_t *entry_pc = pc; + JSValue *entry_sp = sp; + uint32_t entry_pc_offset = (uint32_t)(pc - b->byte_code_buf); ++ JSJitFastEntryState *fast_state = NULL; ++ uint32_t fast_budget = 0, fast_flags = 0; + + /* acquire_entry is an external callback and may inspect the frame. */ + sf->cur_pc = pc; + rt->jit_backend_active_depth++; +- if (rt->jit_cached_entry.pin) { ++ if (jit_fast_entry && entry_pc_offset == 0 && ++ qjsjit_fast_entry_ready(rt, b)) { ++ /* The active call owns the handle exclusively, as below. */ ++ entry = rt->jit_cached_entry; ++ entry_epoch = rt->jit_cached_entry_epoch; ++ fast_state = rt->jit_fast_state; ++ fast_budget = rt->jit_cached_entry_budget - 1; ++ fast_flags = rt->jit_cached_entry_fast_flags; ++ memset(&rt->jit_cached_entry, 0, sizeof(rt->jit_cached_entry)); ++ rt->jit_cached_entry_budget = 0; ++ rt->jit_cached_entry_fast_flags = 0; ++ } else if (rt->jit_cached_entry.pin && ++ (rt->jit_cached_entry_id != entry_function_id || ++ rt->jit_cached_entry_generation != entry_generation) && ++ rt->jit_fast_state && ++ rt->jit_fast_state->epoch == rt->jit_cached_entry_epoch) { ++ /* Keep another function's still-admissible idle handle: calls ++ * that never reach native code must not evict it. A native call ++ * of this function replaces it after returning. */ ++ } else if (rt->jit_cached_entry.pin) { + entry_epoch = jit_vtable->entry_cache_epoch ? + jit_vtable->entry_cache_epoch(jit_opaque) : 0; + if (entry_pc_offset == 0 && entry_epoch != 0 && +@@ -20727,10 +20811,13 @@ + rt->jit_cached_entry_generation == entry_generation) { + entry = rt->jit_cached_entry; + memset(&rt->jit_cached_entry, 0, sizeof(rt->jit_cached_entry)); ++ rt->jit_cached_entry_budget = 0; ++ rt->jit_cached_entry_fast_flags = 0; + } else { + qjsjit_discard_cached_entry(rt); + } + } ++ jit_fast_entry = fast_state != NULL; + if (!entry.pin) { + entry = jit_vtable->acquire_entry(jit_opaque, entry_function_id, + entry_generation, entry_pc_offset); +@@ -20795,9 +20882,15 @@ + rt->jit_active_frame_cookie = frame.frame_cookie; + rt->jit_active_stack_map_count = entry.stack_map_count; + rt->jit_active_helper_abi_version = entry.helper_abi_version; +- if (jit_vtable->native_enter) ++ if (fast_state) { ++ frame.flags |= JS_JIT_FRAME_FAST_ENTRY; ++ fast_state->entries++; ++ if (fast_flags & JS_JIT_FAST_ENTRY_OPTIMIZED) ++ fast_state->optimized_entries++; ++ } else if (jit_vtable->native_enter) { + jit_vtable->native_enter(jit_opaque, entry_function_id, + entry_generation, entry_pc_offset); ++ } + jit_exit = entry.entry(&frame); + if (qjsjit_inline_abort_root(&frame)) { + qjsjit_reject_exit(ctx, &frame.result, +@@ -20807,10 +20900,15 @@ + jit_exit.resume_pc = frame.pc; + jit_exit.resume_stack_top = frame.stack_top; + } +- if (jit_vtable->native_exit) ++ if (fast_state && jit_exit.kind == JS_JIT_EXIT_DONE) { ++ fast_state->exits++; ++ } else if (jit_vtable->native_exit) { + jit_vtable->native_exit(jit_opaque, entry_function_id, + entry_generation, entry_pc_offset, ++ fast_state ? jit_exit.kind | ++ JS_JIT_EXIT_FAST_UNPAIRED : + jit_exit.kind); ++ } + rt->jit_active_frame = previous_active_frame; + rt->jit_active_frame_cookie = previous_frame_cookie; + rt->jit_active_stack_map_count = previous_stack_map_count; +@@ -20820,12 +20918,21 @@ + b->jit_function_id == entry_function_id && + b->jit_generation == entry_generation && + jit_exit.kind == JS_JIT_EXIT_DONE && +- jit_vtable->entry_cache_epoch(jit_opaque) == entry_epoch) { ++ (fast_state ? fast_state->epoch : ++ jit_vtable->entry_cache_epoch(jit_opaque)) == entry_epoch) { + qjsjit_discard_cached_entry(rt); + rt->jit_cached_entry = entry; + rt->jit_cached_entry_id = entry_function_id; + rt->jit_cached_entry_generation = entry_generation; + rt->jit_cached_entry_epoch = entry_epoch; ++ if (fast_state) { ++ rt->jit_cached_entry_budget = fast_budget; ++ rt->jit_cached_entry_fast_flags = fast_flags; ++ } else { ++ qjsjit_grant_fast_entry(rt, jit_vtable, jit_opaque, ++ entry_function_id, ++ entry_generation); ++ } + } else { + jit_vtable->release_entry(jit_opaque, entry); + } +diff --git a/quickjs-jit.h b/quickjs-jit.h +--- a/quickjs-jit.h ++++ b/quickjs-jit.h +@@ -17,7 +17,7 @@ + #endif + + #define QJSJIT_ABI_MAJOR 1u +-#define QJSJIT_ABI_MINOR 24u ++#define QJSJIT_ABI_MINOR 25u + + /* Frame flags are append-only. Stress collection is a test/debug contract: + * every helper performs an allocating cycle-GC point before and after its +@@ -25,6 +25,9 @@ + #define JS_JIT_FUNCTION_STRICT (1U << 0) + #define JS_JIT_FRAME_STRESS_GC (1U << 1) + #define JS_JIT_FRAME_SIDE_PATH_HIT (1U << 2) ++/* ABI 1.25: set on an execution granted by entry_fast_grant. No ++ * native_enter preceded it, and a DONE exit reports no native_exit. */ ++#define JS_JIT_FRAME_FAST_ENTRY (1U << 3) + + /* Helper slot numbers flatten arguments, locals, and the operand stack, in + * that order. JS_JIT_SLOT_NONE is accepted only as argv for argc == 0. */ +@@ -472,6 +475,36 @@ + uint64_t free_count; + } JSJitHelperCounters; + ++/* ABI 1.25 callback-free native entry. The backend owns the state, which must ++ * stay at a fixed address until runtime_detach returns. QuickJS reads epoch ++ * and updates the counters only on the runtime thread. */ ++typedef struct JSJitFastEntryState { ++ uint32_t struct_size; ++ uint32_t reserved; ++ /* Backend-written. Zero disables fast entry. A nonzero value follows the ++ * entry_cache_epoch rules: it never repeats and changes before any ++ * granted handle becomes inadmissible for another call. */ ++ uint64_t epoch; ++ uint64_t entries; /* executions without native_enter */ ++ uint64_t exits; /* DONE exits without native_exit */ ++ uint64_t optimized_entries; /* entries granted JS_JIT_FAST_ENTRY_OPTIMIZED */ ++} JSJitFastEntryState; ++ ++#define JS_JIT_FAST_ENTRY_OPTIMIZED (1U << 0) ++/* OR-ed into native_exit's exit_kind for a non-DONE exit of a fast ++ * execution: that exit has no matching native_enter. */ ++#define JS_JIT_EXIT_FAST_UNPAIRED 0x80000000u ++/* Upper bound on one grant; QuickJS rejects larger budgets. */ ++#define JS_JIT_FAST_ENTRY_MAX_BUDGET 65536u ++ ++typedef struct JSJitFastEntryGrant { ++ uint32_t struct_size; ++ uint32_t budget; /* following executions that may skip the callbacks */ ++ uint32_t flags; ++ uint32_t reserved; ++ JSJitFastEntryState *state; ++} JSJitFastEntryGrant; ++ + typedef struct JSJitBackendVTable { + uint32_t struct_size; + uint32_t (*record_hot)(void *opaque, const JSJitHotEvent *event); +@@ -499,6 +532,16 @@ + * recursion. native_enter/native_exit still run once per execution. + * The handle and pin must be immutable between executions. */ + uint64_t (*entry_cache_epoch)(void *opaque); ++ /* Optional, requires entry_cache_epoch. Called after a DONE exit whose ++ * pc=0 handle QuickJS caches. A nonzero budget lets that many following ++ * calls of the cached handle (same id, generation and cache epoch) skip ++ * call feedback, record_hot, entry_cache_epoch, native_enter and ++ * native_exit while grant->state->epoch equals the cached epoch. Such ++ * executions carry JS_JIT_FRAME_FAST_ENTRY and are counted in the state; ++ * a non-DONE exit still calls native_exit, with JS_JIT_EXIT_FAST_UNPAIRED. ++ * Afterwards the next call takes the full callback path again. */ ++ void (*entry_fast_grant)(void *opaque, uint64_t id, uint64_t generation, ++ JSJitFastEntryGrant *grant); + } JSJitBackendVTable; + + enum { diff --git a/sys/patches/README.md b/sys/patches/README.md index 931cf9bd3..ddd22a685 100644 --- a/sys/patches/README.md +++ b/sys/patches/README.md @@ -60,6 +60,21 @@ data pointer before returning `{ data, count, mode }`. The table advertises zero effects and the receiver is passed by pointer so generated calls do not depend on platform-specific aggregate argument classification. +`0023-fast-native-entry.patch` adds the ABI 1.25 callback-free native entry. +After a DONE exit whose pc=0 handle QuickJS caches, the optional +`entry_fast_grant` callback may grant a bounded budget (at most 65536) and a +backend-owned `JSJitFastEntryState`. While budget remains and the state epoch +still equals the cached entry epoch, calls of that function reuse the cached +handle without call feedback, `record_hot`, `entry_cache_epoch`, +`native_enter` or `native_exit`; QuickJS counts the executions in the state +and marks their frames `JS_JIT_FRAME_FAST_ENTRY`. A non-DONE exit still calls +`native_exit`, tagged `JS_JIT_EXIT_FAST_UNPAIRED`, and is never re-cached. +Suspension, feedback shutdown, a zero or changed epoch, or a malformed grant +all return to the full callback path. Calls that never reach native code keep +another function's still-admissible idle handle instead of evicting it. The +runtime grows by two 32-bit fields and one pointer; the backend vtable gains +one trailing member. + The build accepts only the patch names and byte digests listed in `build_support/patch.rs`, then verifies the complete patched source manifest. Keeping the baseline and patch separate makes diff --git a/sys/src/bindings/aarch64-apple-darwin.rs b/sys/src/bindings/aarch64-apple-darwin.rs index a64c09ee5..c5e23284e 100644 --- a/sys/src/bindings/aarch64-apple-darwin.rs +++ b/sys/src/bindings/aarch64-apple-darwin.rs @@ -86,10 +86,11 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; +pub const JS_JIT_FRAME_FAST_ENTRY: u32 = 8; pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; @@ -114,6 +115,9 @@ pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const JS_JIT_FAST_ENTRY_OPTIMIZED: u32 = 1; +pub const JS_JIT_EXIT_FAST_UNPAIRED: u32 = 2147483648; +pub const JS_JIT_FAST_ENTRY_MAX_BUDGET: u32 = 65536; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -3644,6 +3648,57 @@ const _: () = { }; #[repr(C)] #[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryState { + pub struct_size: u32, + pub reserved: u32, + pub epoch: u64, + pub entries: u64, + pub exits: u64, + pub optimized_entries: u64, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryState"][::core::mem::size_of::() - 40usize]; + ["Alignment of JSJitFastEntryState"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryState::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryState, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryState::reserved"] + [::core::mem::offset_of!(JSJitFastEntryState, reserved) - 4usize]; + ["Offset of field: JSJitFastEntryState::epoch"] + [::core::mem::offset_of!(JSJitFastEntryState, epoch) - 8usize]; + ["Offset of field: JSJitFastEntryState::entries"] + [::core::mem::offset_of!(JSJitFastEntryState, entries) - 16usize]; + ["Offset of field: JSJitFastEntryState::exits"] + [::core::mem::offset_of!(JSJitFastEntryState, exits) - 24usize]; + ["Offset of field: JSJitFastEntryState::optimized_entries"] + [::core::mem::offset_of!(JSJitFastEntryState, optimized_entries) - 32usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryGrant { + pub struct_size: u32, + pub budget: u32, + pub flags: u32, + pub reserved: u32, + pub state: *mut JSJitFastEntryState, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryGrant"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitFastEntryGrant"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryGrant::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryGrant, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryGrant::budget"] + [::core::mem::offset_of!(JSJitFastEntryGrant, budget) - 4usize]; + ["Offset of field: JSJitFastEntryGrant::flags"] + [::core::mem::offset_of!(JSJitFastEntryGrant, flags) - 8usize]; + ["Offset of field: JSJitFastEntryGrant::reserved"] + [::core::mem::offset_of!(JSJitFastEntryGrant, reserved) - 12usize]; + ["Offset of field: JSJitFastEntryGrant::state"] + [::core::mem::offset_of!(JSJitFastEntryGrant, state) - 16usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] pub struct JSJitBackendVTable { pub struct_size: u32, pub record_hot: ::core::option::Option< @@ -3691,10 +3746,18 @@ pub struct JSJitBackendVTable { >, pub entry_cache_epoch: ::core::option::Option u64>, + pub entry_fast_grant: ::core::option::Option< + unsafe extern "C" fn( + opaque: *mut ::core::ffi::c_void, + id: u64, + generation: u64, + grant: *mut JSJitFastEntryGrant, + ), + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 96usize]; + ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 104usize]; ["Alignment of JSJitBackendVTable"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitBackendVTable::struct_size"] [::core::mem::offset_of!(JSJitBackendVTable, struct_size) - 0usize]; @@ -3720,6 +3783,8 @@ const _: () = { [::core::mem::offset_of!(JSJitBackendVTable, record_feedback) - 80usize]; ["Offset of field: JSJitBackendVTable::entry_cache_epoch"] [::core::mem::offset_of!(JSJitBackendVTable, entry_cache_epoch) - 88usize]; + ["Offset of field: JSJitBackendVTable::entry_fast_grant"] + [::core::mem::offset_of!(JSJitBackendVTable, entry_fast_grant) - 96usize]; }; pub const JS_JIT_BACKEND_OK: _bindgen_ty_6 = 0; pub const JS_JIT_BACKEND_INVALID_ARGUMENT: _bindgen_ty_6 = -1; diff --git a/sys/src/bindings/aarch64-pc-windows-msvc.rs b/sys/src/bindings/aarch64-pc-windows-msvc.rs index a64c09ee5..c5e23284e 100644 --- a/sys/src/bindings/aarch64-pc-windows-msvc.rs +++ b/sys/src/bindings/aarch64-pc-windows-msvc.rs @@ -86,10 +86,11 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; +pub const JS_JIT_FRAME_FAST_ENTRY: u32 = 8; pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; @@ -114,6 +115,9 @@ pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const JS_JIT_FAST_ENTRY_OPTIMIZED: u32 = 1; +pub const JS_JIT_EXIT_FAST_UNPAIRED: u32 = 2147483648; +pub const JS_JIT_FAST_ENTRY_MAX_BUDGET: u32 = 65536; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -3644,6 +3648,57 @@ const _: () = { }; #[repr(C)] #[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryState { + pub struct_size: u32, + pub reserved: u32, + pub epoch: u64, + pub entries: u64, + pub exits: u64, + pub optimized_entries: u64, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryState"][::core::mem::size_of::() - 40usize]; + ["Alignment of JSJitFastEntryState"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryState::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryState, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryState::reserved"] + [::core::mem::offset_of!(JSJitFastEntryState, reserved) - 4usize]; + ["Offset of field: JSJitFastEntryState::epoch"] + [::core::mem::offset_of!(JSJitFastEntryState, epoch) - 8usize]; + ["Offset of field: JSJitFastEntryState::entries"] + [::core::mem::offset_of!(JSJitFastEntryState, entries) - 16usize]; + ["Offset of field: JSJitFastEntryState::exits"] + [::core::mem::offset_of!(JSJitFastEntryState, exits) - 24usize]; + ["Offset of field: JSJitFastEntryState::optimized_entries"] + [::core::mem::offset_of!(JSJitFastEntryState, optimized_entries) - 32usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryGrant { + pub struct_size: u32, + pub budget: u32, + pub flags: u32, + pub reserved: u32, + pub state: *mut JSJitFastEntryState, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryGrant"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitFastEntryGrant"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryGrant::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryGrant, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryGrant::budget"] + [::core::mem::offset_of!(JSJitFastEntryGrant, budget) - 4usize]; + ["Offset of field: JSJitFastEntryGrant::flags"] + [::core::mem::offset_of!(JSJitFastEntryGrant, flags) - 8usize]; + ["Offset of field: JSJitFastEntryGrant::reserved"] + [::core::mem::offset_of!(JSJitFastEntryGrant, reserved) - 12usize]; + ["Offset of field: JSJitFastEntryGrant::state"] + [::core::mem::offset_of!(JSJitFastEntryGrant, state) - 16usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] pub struct JSJitBackendVTable { pub struct_size: u32, pub record_hot: ::core::option::Option< @@ -3691,10 +3746,18 @@ pub struct JSJitBackendVTable { >, pub entry_cache_epoch: ::core::option::Option u64>, + pub entry_fast_grant: ::core::option::Option< + unsafe extern "C" fn( + opaque: *mut ::core::ffi::c_void, + id: u64, + generation: u64, + grant: *mut JSJitFastEntryGrant, + ), + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 96usize]; + ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 104usize]; ["Alignment of JSJitBackendVTable"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitBackendVTable::struct_size"] [::core::mem::offset_of!(JSJitBackendVTable, struct_size) - 0usize]; @@ -3720,6 +3783,8 @@ const _: () = { [::core::mem::offset_of!(JSJitBackendVTable, record_feedback) - 80usize]; ["Offset of field: JSJitBackendVTable::entry_cache_epoch"] [::core::mem::offset_of!(JSJitBackendVTable, entry_cache_epoch) - 88usize]; + ["Offset of field: JSJitBackendVTable::entry_fast_grant"] + [::core::mem::offset_of!(JSJitBackendVTable, entry_fast_grant) - 96usize]; }; pub const JS_JIT_BACKEND_OK: _bindgen_ty_6 = 0; pub const JS_JIT_BACKEND_INVALID_ARGUMENT: _bindgen_ty_6 = -1; diff --git a/sys/src/bindings/aarch64-unknown-linux-gnu.rs b/sys/src/bindings/aarch64-unknown-linux-gnu.rs index a64c09ee5..c5e23284e 100644 --- a/sys/src/bindings/aarch64-unknown-linux-gnu.rs +++ b/sys/src/bindings/aarch64-unknown-linux-gnu.rs @@ -86,10 +86,11 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; +pub const JS_JIT_FRAME_FAST_ENTRY: u32 = 8; pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; @@ -114,6 +115,9 @@ pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const JS_JIT_FAST_ENTRY_OPTIMIZED: u32 = 1; +pub const JS_JIT_EXIT_FAST_UNPAIRED: u32 = 2147483648; +pub const JS_JIT_FAST_ENTRY_MAX_BUDGET: u32 = 65536; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -3644,6 +3648,57 @@ const _: () = { }; #[repr(C)] #[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryState { + pub struct_size: u32, + pub reserved: u32, + pub epoch: u64, + pub entries: u64, + pub exits: u64, + pub optimized_entries: u64, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryState"][::core::mem::size_of::() - 40usize]; + ["Alignment of JSJitFastEntryState"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryState::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryState, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryState::reserved"] + [::core::mem::offset_of!(JSJitFastEntryState, reserved) - 4usize]; + ["Offset of field: JSJitFastEntryState::epoch"] + [::core::mem::offset_of!(JSJitFastEntryState, epoch) - 8usize]; + ["Offset of field: JSJitFastEntryState::entries"] + [::core::mem::offset_of!(JSJitFastEntryState, entries) - 16usize]; + ["Offset of field: JSJitFastEntryState::exits"] + [::core::mem::offset_of!(JSJitFastEntryState, exits) - 24usize]; + ["Offset of field: JSJitFastEntryState::optimized_entries"] + [::core::mem::offset_of!(JSJitFastEntryState, optimized_entries) - 32usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryGrant { + pub struct_size: u32, + pub budget: u32, + pub flags: u32, + pub reserved: u32, + pub state: *mut JSJitFastEntryState, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryGrant"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitFastEntryGrant"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryGrant::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryGrant, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryGrant::budget"] + [::core::mem::offset_of!(JSJitFastEntryGrant, budget) - 4usize]; + ["Offset of field: JSJitFastEntryGrant::flags"] + [::core::mem::offset_of!(JSJitFastEntryGrant, flags) - 8usize]; + ["Offset of field: JSJitFastEntryGrant::reserved"] + [::core::mem::offset_of!(JSJitFastEntryGrant, reserved) - 12usize]; + ["Offset of field: JSJitFastEntryGrant::state"] + [::core::mem::offset_of!(JSJitFastEntryGrant, state) - 16usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] pub struct JSJitBackendVTable { pub struct_size: u32, pub record_hot: ::core::option::Option< @@ -3691,10 +3746,18 @@ pub struct JSJitBackendVTable { >, pub entry_cache_epoch: ::core::option::Option u64>, + pub entry_fast_grant: ::core::option::Option< + unsafe extern "C" fn( + opaque: *mut ::core::ffi::c_void, + id: u64, + generation: u64, + grant: *mut JSJitFastEntryGrant, + ), + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 96usize]; + ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 104usize]; ["Alignment of JSJitBackendVTable"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitBackendVTable::struct_size"] [::core::mem::offset_of!(JSJitBackendVTable, struct_size) - 0usize]; @@ -3720,6 +3783,8 @@ const _: () = { [::core::mem::offset_of!(JSJitBackendVTable, record_feedback) - 80usize]; ["Offset of field: JSJitBackendVTable::entry_cache_epoch"] [::core::mem::offset_of!(JSJitBackendVTable, entry_cache_epoch) - 88usize]; + ["Offset of field: JSJitBackendVTable::entry_fast_grant"] + [::core::mem::offset_of!(JSJitBackendVTable, entry_fast_grant) - 96usize]; }; pub const JS_JIT_BACKEND_OK: _bindgen_ty_6 = 0; pub const JS_JIT_BACKEND_INVALID_ARGUMENT: _bindgen_ty_6 = -1; diff --git a/sys/src/bindings/aarch64-unknown-linux-musl.rs b/sys/src/bindings/aarch64-unknown-linux-musl.rs index a64c09ee5..c5e23284e 100644 --- a/sys/src/bindings/aarch64-unknown-linux-musl.rs +++ b/sys/src/bindings/aarch64-unknown-linux-musl.rs @@ -86,10 +86,11 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; +pub const JS_JIT_FRAME_FAST_ENTRY: u32 = 8; pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; @@ -114,6 +115,9 @@ pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const JS_JIT_FAST_ENTRY_OPTIMIZED: u32 = 1; +pub const JS_JIT_EXIT_FAST_UNPAIRED: u32 = 2147483648; +pub const JS_JIT_FAST_ENTRY_MAX_BUDGET: u32 = 65536; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -3644,6 +3648,57 @@ const _: () = { }; #[repr(C)] #[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryState { + pub struct_size: u32, + pub reserved: u32, + pub epoch: u64, + pub entries: u64, + pub exits: u64, + pub optimized_entries: u64, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryState"][::core::mem::size_of::() - 40usize]; + ["Alignment of JSJitFastEntryState"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryState::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryState, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryState::reserved"] + [::core::mem::offset_of!(JSJitFastEntryState, reserved) - 4usize]; + ["Offset of field: JSJitFastEntryState::epoch"] + [::core::mem::offset_of!(JSJitFastEntryState, epoch) - 8usize]; + ["Offset of field: JSJitFastEntryState::entries"] + [::core::mem::offset_of!(JSJitFastEntryState, entries) - 16usize]; + ["Offset of field: JSJitFastEntryState::exits"] + [::core::mem::offset_of!(JSJitFastEntryState, exits) - 24usize]; + ["Offset of field: JSJitFastEntryState::optimized_entries"] + [::core::mem::offset_of!(JSJitFastEntryState, optimized_entries) - 32usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryGrant { + pub struct_size: u32, + pub budget: u32, + pub flags: u32, + pub reserved: u32, + pub state: *mut JSJitFastEntryState, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryGrant"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitFastEntryGrant"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryGrant::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryGrant, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryGrant::budget"] + [::core::mem::offset_of!(JSJitFastEntryGrant, budget) - 4usize]; + ["Offset of field: JSJitFastEntryGrant::flags"] + [::core::mem::offset_of!(JSJitFastEntryGrant, flags) - 8usize]; + ["Offset of field: JSJitFastEntryGrant::reserved"] + [::core::mem::offset_of!(JSJitFastEntryGrant, reserved) - 12usize]; + ["Offset of field: JSJitFastEntryGrant::state"] + [::core::mem::offset_of!(JSJitFastEntryGrant, state) - 16usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] pub struct JSJitBackendVTable { pub struct_size: u32, pub record_hot: ::core::option::Option< @@ -3691,10 +3746,18 @@ pub struct JSJitBackendVTable { >, pub entry_cache_epoch: ::core::option::Option u64>, + pub entry_fast_grant: ::core::option::Option< + unsafe extern "C" fn( + opaque: *mut ::core::ffi::c_void, + id: u64, + generation: u64, + grant: *mut JSJitFastEntryGrant, + ), + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 96usize]; + ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 104usize]; ["Alignment of JSJitBackendVTable"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitBackendVTable::struct_size"] [::core::mem::offset_of!(JSJitBackendVTable, struct_size) - 0usize]; @@ -3720,6 +3783,8 @@ const _: () = { [::core::mem::offset_of!(JSJitBackendVTable, record_feedback) - 80usize]; ["Offset of field: JSJitBackendVTable::entry_cache_epoch"] [::core::mem::offset_of!(JSJitBackendVTable, entry_cache_epoch) - 88usize]; + ["Offset of field: JSJitBackendVTable::entry_fast_grant"] + [::core::mem::offset_of!(JSJitBackendVTable, entry_fast_grant) - 96usize]; }; pub const JS_JIT_BACKEND_OK: _bindgen_ty_6 = 0; pub const JS_JIT_BACKEND_INVALID_ARGUMENT: _bindgen_ty_6 = -1; diff --git a/sys/src/bindings/wasm32-wasip1.rs b/sys/src/bindings/wasm32-wasip1.rs index a8167404c..ad25a37a7 100644 --- a/sys/src/bindings/wasm32-wasip1.rs +++ b/sys/src/bindings/wasm32-wasip1.rs @@ -87,10 +87,11 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; +pub const JS_JIT_FRAME_FAST_ENTRY: u32 = 8; pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; @@ -115,6 +116,9 @@ pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const JS_JIT_FAST_ENTRY_OPTIMIZED: u32 = 1; +pub const JS_JIT_EXIT_FAST_UNPAIRED: u32 = 2147483648; +pub const JS_JIT_FAST_ENTRY_MAX_BUDGET: u32 = 65536; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -3613,6 +3617,57 @@ const _: () = { }; #[repr(C)] #[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryState { + pub struct_size: u32, + pub reserved: u32, + pub epoch: u64, + pub entries: u64, + pub exits: u64, + pub optimized_entries: u64, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryState"][::core::mem::size_of::() - 40usize]; + ["Alignment of JSJitFastEntryState"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryState::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryState, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryState::reserved"] + [::core::mem::offset_of!(JSJitFastEntryState, reserved) - 4usize]; + ["Offset of field: JSJitFastEntryState::epoch"] + [::core::mem::offset_of!(JSJitFastEntryState, epoch) - 8usize]; + ["Offset of field: JSJitFastEntryState::entries"] + [::core::mem::offset_of!(JSJitFastEntryState, entries) - 16usize]; + ["Offset of field: JSJitFastEntryState::exits"] + [::core::mem::offset_of!(JSJitFastEntryState, exits) - 24usize]; + ["Offset of field: JSJitFastEntryState::optimized_entries"] + [::core::mem::offset_of!(JSJitFastEntryState, optimized_entries) - 32usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryGrant { + pub struct_size: u32, + pub budget: u32, + pub flags: u32, + pub reserved: u32, + pub state: *mut JSJitFastEntryState, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryGrant"][::core::mem::size_of::() - 20usize]; + ["Alignment of JSJitFastEntryGrant"][::core::mem::align_of::() - 4usize]; + ["Offset of field: JSJitFastEntryGrant::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryGrant, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryGrant::budget"] + [::core::mem::offset_of!(JSJitFastEntryGrant, budget) - 4usize]; + ["Offset of field: JSJitFastEntryGrant::flags"] + [::core::mem::offset_of!(JSJitFastEntryGrant, flags) - 8usize]; + ["Offset of field: JSJitFastEntryGrant::reserved"] + [::core::mem::offset_of!(JSJitFastEntryGrant, reserved) - 12usize]; + ["Offset of field: JSJitFastEntryGrant::state"] + [::core::mem::offset_of!(JSJitFastEntryGrant, state) - 16usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] pub struct JSJitBackendVTable { pub struct_size: u32, pub record_hot: ::core::option::Option< @@ -3660,10 +3715,18 @@ pub struct JSJitBackendVTable { >, pub entry_cache_epoch: ::core::option::Option u64>, + pub entry_fast_grant: ::core::option::Option< + unsafe extern "C" fn( + opaque: *mut ::core::ffi::c_void, + id: u64, + generation: u64, + grant: *mut JSJitFastEntryGrant, + ), + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 48usize]; + ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 52usize]; ["Alignment of JSJitBackendVTable"][::core::mem::align_of::() - 4usize]; ["Offset of field: JSJitBackendVTable::struct_size"] [::core::mem::offset_of!(JSJitBackendVTable, struct_size) - 0usize]; @@ -3689,6 +3752,8 @@ const _: () = { [::core::mem::offset_of!(JSJitBackendVTable, record_feedback) - 40usize]; ["Offset of field: JSJitBackendVTable::entry_cache_epoch"] [::core::mem::offset_of!(JSJitBackendVTable, entry_cache_epoch) - 44usize]; + ["Offset of field: JSJitBackendVTable::entry_fast_grant"] + [::core::mem::offset_of!(JSJitBackendVTable, entry_fast_grant) - 48usize]; }; pub const JS_JIT_BACKEND_OK: _bindgen_ty_4 = 0; pub const JS_JIT_BACKEND_INVALID_ARGUMENT: _bindgen_ty_4 = -1; diff --git a/sys/src/bindings/x86_64-apple-darwin.rs b/sys/src/bindings/x86_64-apple-darwin.rs index a64c09ee5..c5e23284e 100644 --- a/sys/src/bindings/x86_64-apple-darwin.rs +++ b/sys/src/bindings/x86_64-apple-darwin.rs @@ -86,10 +86,11 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; +pub const JS_JIT_FRAME_FAST_ENTRY: u32 = 8; pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; @@ -114,6 +115,9 @@ pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const JS_JIT_FAST_ENTRY_OPTIMIZED: u32 = 1; +pub const JS_JIT_EXIT_FAST_UNPAIRED: u32 = 2147483648; +pub const JS_JIT_FAST_ENTRY_MAX_BUDGET: u32 = 65536; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -3644,6 +3648,57 @@ const _: () = { }; #[repr(C)] #[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryState { + pub struct_size: u32, + pub reserved: u32, + pub epoch: u64, + pub entries: u64, + pub exits: u64, + pub optimized_entries: u64, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryState"][::core::mem::size_of::() - 40usize]; + ["Alignment of JSJitFastEntryState"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryState::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryState, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryState::reserved"] + [::core::mem::offset_of!(JSJitFastEntryState, reserved) - 4usize]; + ["Offset of field: JSJitFastEntryState::epoch"] + [::core::mem::offset_of!(JSJitFastEntryState, epoch) - 8usize]; + ["Offset of field: JSJitFastEntryState::entries"] + [::core::mem::offset_of!(JSJitFastEntryState, entries) - 16usize]; + ["Offset of field: JSJitFastEntryState::exits"] + [::core::mem::offset_of!(JSJitFastEntryState, exits) - 24usize]; + ["Offset of field: JSJitFastEntryState::optimized_entries"] + [::core::mem::offset_of!(JSJitFastEntryState, optimized_entries) - 32usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryGrant { + pub struct_size: u32, + pub budget: u32, + pub flags: u32, + pub reserved: u32, + pub state: *mut JSJitFastEntryState, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryGrant"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitFastEntryGrant"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryGrant::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryGrant, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryGrant::budget"] + [::core::mem::offset_of!(JSJitFastEntryGrant, budget) - 4usize]; + ["Offset of field: JSJitFastEntryGrant::flags"] + [::core::mem::offset_of!(JSJitFastEntryGrant, flags) - 8usize]; + ["Offset of field: JSJitFastEntryGrant::reserved"] + [::core::mem::offset_of!(JSJitFastEntryGrant, reserved) - 12usize]; + ["Offset of field: JSJitFastEntryGrant::state"] + [::core::mem::offset_of!(JSJitFastEntryGrant, state) - 16usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] pub struct JSJitBackendVTable { pub struct_size: u32, pub record_hot: ::core::option::Option< @@ -3691,10 +3746,18 @@ pub struct JSJitBackendVTable { >, pub entry_cache_epoch: ::core::option::Option u64>, + pub entry_fast_grant: ::core::option::Option< + unsafe extern "C" fn( + opaque: *mut ::core::ffi::c_void, + id: u64, + generation: u64, + grant: *mut JSJitFastEntryGrant, + ), + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 96usize]; + ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 104usize]; ["Alignment of JSJitBackendVTable"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitBackendVTable::struct_size"] [::core::mem::offset_of!(JSJitBackendVTable, struct_size) - 0usize]; @@ -3720,6 +3783,8 @@ const _: () = { [::core::mem::offset_of!(JSJitBackendVTable, record_feedback) - 80usize]; ["Offset of field: JSJitBackendVTable::entry_cache_epoch"] [::core::mem::offset_of!(JSJitBackendVTable, entry_cache_epoch) - 88usize]; + ["Offset of field: JSJitBackendVTable::entry_fast_grant"] + [::core::mem::offset_of!(JSJitBackendVTable, entry_fast_grant) - 96usize]; }; pub const JS_JIT_BACKEND_OK: _bindgen_ty_6 = 0; pub const JS_JIT_BACKEND_INVALID_ARGUMENT: _bindgen_ty_6 = -1; diff --git a/sys/src/bindings/x86_64-pc-windows-gnu.rs b/sys/src/bindings/x86_64-pc-windows-gnu.rs index a64c09ee5..c5e23284e 100644 --- a/sys/src/bindings/x86_64-pc-windows-gnu.rs +++ b/sys/src/bindings/x86_64-pc-windows-gnu.rs @@ -86,10 +86,11 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; +pub const JS_JIT_FRAME_FAST_ENTRY: u32 = 8; pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; @@ -114,6 +115,9 @@ pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const JS_JIT_FAST_ENTRY_OPTIMIZED: u32 = 1; +pub const JS_JIT_EXIT_FAST_UNPAIRED: u32 = 2147483648; +pub const JS_JIT_FAST_ENTRY_MAX_BUDGET: u32 = 65536; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -3644,6 +3648,57 @@ const _: () = { }; #[repr(C)] #[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryState { + pub struct_size: u32, + pub reserved: u32, + pub epoch: u64, + pub entries: u64, + pub exits: u64, + pub optimized_entries: u64, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryState"][::core::mem::size_of::() - 40usize]; + ["Alignment of JSJitFastEntryState"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryState::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryState, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryState::reserved"] + [::core::mem::offset_of!(JSJitFastEntryState, reserved) - 4usize]; + ["Offset of field: JSJitFastEntryState::epoch"] + [::core::mem::offset_of!(JSJitFastEntryState, epoch) - 8usize]; + ["Offset of field: JSJitFastEntryState::entries"] + [::core::mem::offset_of!(JSJitFastEntryState, entries) - 16usize]; + ["Offset of field: JSJitFastEntryState::exits"] + [::core::mem::offset_of!(JSJitFastEntryState, exits) - 24usize]; + ["Offset of field: JSJitFastEntryState::optimized_entries"] + [::core::mem::offset_of!(JSJitFastEntryState, optimized_entries) - 32usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryGrant { + pub struct_size: u32, + pub budget: u32, + pub flags: u32, + pub reserved: u32, + pub state: *mut JSJitFastEntryState, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryGrant"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitFastEntryGrant"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryGrant::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryGrant, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryGrant::budget"] + [::core::mem::offset_of!(JSJitFastEntryGrant, budget) - 4usize]; + ["Offset of field: JSJitFastEntryGrant::flags"] + [::core::mem::offset_of!(JSJitFastEntryGrant, flags) - 8usize]; + ["Offset of field: JSJitFastEntryGrant::reserved"] + [::core::mem::offset_of!(JSJitFastEntryGrant, reserved) - 12usize]; + ["Offset of field: JSJitFastEntryGrant::state"] + [::core::mem::offset_of!(JSJitFastEntryGrant, state) - 16usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] pub struct JSJitBackendVTable { pub struct_size: u32, pub record_hot: ::core::option::Option< @@ -3691,10 +3746,18 @@ pub struct JSJitBackendVTable { >, pub entry_cache_epoch: ::core::option::Option u64>, + pub entry_fast_grant: ::core::option::Option< + unsafe extern "C" fn( + opaque: *mut ::core::ffi::c_void, + id: u64, + generation: u64, + grant: *mut JSJitFastEntryGrant, + ), + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 96usize]; + ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 104usize]; ["Alignment of JSJitBackendVTable"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitBackendVTable::struct_size"] [::core::mem::offset_of!(JSJitBackendVTable, struct_size) - 0usize]; @@ -3720,6 +3783,8 @@ const _: () = { [::core::mem::offset_of!(JSJitBackendVTable, record_feedback) - 80usize]; ["Offset of field: JSJitBackendVTable::entry_cache_epoch"] [::core::mem::offset_of!(JSJitBackendVTable, entry_cache_epoch) - 88usize]; + ["Offset of field: JSJitBackendVTable::entry_fast_grant"] + [::core::mem::offset_of!(JSJitBackendVTable, entry_fast_grant) - 96usize]; }; pub const JS_JIT_BACKEND_OK: _bindgen_ty_6 = 0; pub const JS_JIT_BACKEND_INVALID_ARGUMENT: _bindgen_ty_6 = -1; diff --git a/sys/src/bindings/x86_64-pc-windows-msvc.rs b/sys/src/bindings/x86_64-pc-windows-msvc.rs index a64c09ee5..c5e23284e 100644 --- a/sys/src/bindings/x86_64-pc-windows-msvc.rs +++ b/sys/src/bindings/x86_64-pc-windows-msvc.rs @@ -86,10 +86,11 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; +pub const JS_JIT_FRAME_FAST_ENTRY: u32 = 8; pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; @@ -114,6 +115,9 @@ pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const JS_JIT_FAST_ENTRY_OPTIMIZED: u32 = 1; +pub const JS_JIT_EXIT_FAST_UNPAIRED: u32 = 2147483648; +pub const JS_JIT_FAST_ENTRY_MAX_BUDGET: u32 = 65536; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -3644,6 +3648,57 @@ const _: () = { }; #[repr(C)] #[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryState { + pub struct_size: u32, + pub reserved: u32, + pub epoch: u64, + pub entries: u64, + pub exits: u64, + pub optimized_entries: u64, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryState"][::core::mem::size_of::() - 40usize]; + ["Alignment of JSJitFastEntryState"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryState::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryState, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryState::reserved"] + [::core::mem::offset_of!(JSJitFastEntryState, reserved) - 4usize]; + ["Offset of field: JSJitFastEntryState::epoch"] + [::core::mem::offset_of!(JSJitFastEntryState, epoch) - 8usize]; + ["Offset of field: JSJitFastEntryState::entries"] + [::core::mem::offset_of!(JSJitFastEntryState, entries) - 16usize]; + ["Offset of field: JSJitFastEntryState::exits"] + [::core::mem::offset_of!(JSJitFastEntryState, exits) - 24usize]; + ["Offset of field: JSJitFastEntryState::optimized_entries"] + [::core::mem::offset_of!(JSJitFastEntryState, optimized_entries) - 32usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryGrant { + pub struct_size: u32, + pub budget: u32, + pub flags: u32, + pub reserved: u32, + pub state: *mut JSJitFastEntryState, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryGrant"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitFastEntryGrant"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryGrant::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryGrant, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryGrant::budget"] + [::core::mem::offset_of!(JSJitFastEntryGrant, budget) - 4usize]; + ["Offset of field: JSJitFastEntryGrant::flags"] + [::core::mem::offset_of!(JSJitFastEntryGrant, flags) - 8usize]; + ["Offset of field: JSJitFastEntryGrant::reserved"] + [::core::mem::offset_of!(JSJitFastEntryGrant, reserved) - 12usize]; + ["Offset of field: JSJitFastEntryGrant::state"] + [::core::mem::offset_of!(JSJitFastEntryGrant, state) - 16usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] pub struct JSJitBackendVTable { pub struct_size: u32, pub record_hot: ::core::option::Option< @@ -3691,10 +3746,18 @@ pub struct JSJitBackendVTable { >, pub entry_cache_epoch: ::core::option::Option u64>, + pub entry_fast_grant: ::core::option::Option< + unsafe extern "C" fn( + opaque: *mut ::core::ffi::c_void, + id: u64, + generation: u64, + grant: *mut JSJitFastEntryGrant, + ), + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 96usize]; + ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 104usize]; ["Alignment of JSJitBackendVTable"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitBackendVTable::struct_size"] [::core::mem::offset_of!(JSJitBackendVTable, struct_size) - 0usize]; @@ -3720,6 +3783,8 @@ const _: () = { [::core::mem::offset_of!(JSJitBackendVTable, record_feedback) - 80usize]; ["Offset of field: JSJitBackendVTable::entry_cache_epoch"] [::core::mem::offset_of!(JSJitBackendVTable, entry_cache_epoch) - 88usize]; + ["Offset of field: JSJitBackendVTable::entry_fast_grant"] + [::core::mem::offset_of!(JSJitBackendVTable, entry_fast_grant) - 96usize]; }; pub const JS_JIT_BACKEND_OK: _bindgen_ty_6 = 0; pub const JS_JIT_BACKEND_INVALID_ARGUMENT: _bindgen_ty_6 = -1; diff --git a/sys/src/bindings/x86_64-unknown-linux-gnu.rs b/sys/src/bindings/x86_64-unknown-linux-gnu.rs index a64c09ee5..c5e23284e 100644 --- a/sys/src/bindings/x86_64-unknown-linux-gnu.rs +++ b/sys/src/bindings/x86_64-unknown-linux-gnu.rs @@ -86,10 +86,11 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; +pub const JS_JIT_FRAME_FAST_ENTRY: u32 = 8; pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; @@ -114,6 +115,9 @@ pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const JS_JIT_FAST_ENTRY_OPTIMIZED: u32 = 1; +pub const JS_JIT_EXIT_FAST_UNPAIRED: u32 = 2147483648; +pub const JS_JIT_FAST_ENTRY_MAX_BUDGET: u32 = 65536; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -3644,6 +3648,57 @@ const _: () = { }; #[repr(C)] #[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryState { + pub struct_size: u32, + pub reserved: u32, + pub epoch: u64, + pub entries: u64, + pub exits: u64, + pub optimized_entries: u64, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryState"][::core::mem::size_of::() - 40usize]; + ["Alignment of JSJitFastEntryState"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryState::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryState, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryState::reserved"] + [::core::mem::offset_of!(JSJitFastEntryState, reserved) - 4usize]; + ["Offset of field: JSJitFastEntryState::epoch"] + [::core::mem::offset_of!(JSJitFastEntryState, epoch) - 8usize]; + ["Offset of field: JSJitFastEntryState::entries"] + [::core::mem::offset_of!(JSJitFastEntryState, entries) - 16usize]; + ["Offset of field: JSJitFastEntryState::exits"] + [::core::mem::offset_of!(JSJitFastEntryState, exits) - 24usize]; + ["Offset of field: JSJitFastEntryState::optimized_entries"] + [::core::mem::offset_of!(JSJitFastEntryState, optimized_entries) - 32usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryGrant { + pub struct_size: u32, + pub budget: u32, + pub flags: u32, + pub reserved: u32, + pub state: *mut JSJitFastEntryState, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryGrant"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitFastEntryGrant"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryGrant::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryGrant, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryGrant::budget"] + [::core::mem::offset_of!(JSJitFastEntryGrant, budget) - 4usize]; + ["Offset of field: JSJitFastEntryGrant::flags"] + [::core::mem::offset_of!(JSJitFastEntryGrant, flags) - 8usize]; + ["Offset of field: JSJitFastEntryGrant::reserved"] + [::core::mem::offset_of!(JSJitFastEntryGrant, reserved) - 12usize]; + ["Offset of field: JSJitFastEntryGrant::state"] + [::core::mem::offset_of!(JSJitFastEntryGrant, state) - 16usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] pub struct JSJitBackendVTable { pub struct_size: u32, pub record_hot: ::core::option::Option< @@ -3691,10 +3746,18 @@ pub struct JSJitBackendVTable { >, pub entry_cache_epoch: ::core::option::Option u64>, + pub entry_fast_grant: ::core::option::Option< + unsafe extern "C" fn( + opaque: *mut ::core::ffi::c_void, + id: u64, + generation: u64, + grant: *mut JSJitFastEntryGrant, + ), + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 96usize]; + ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 104usize]; ["Alignment of JSJitBackendVTable"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitBackendVTable::struct_size"] [::core::mem::offset_of!(JSJitBackendVTable, struct_size) - 0usize]; @@ -3720,6 +3783,8 @@ const _: () = { [::core::mem::offset_of!(JSJitBackendVTable, record_feedback) - 80usize]; ["Offset of field: JSJitBackendVTable::entry_cache_epoch"] [::core::mem::offset_of!(JSJitBackendVTable, entry_cache_epoch) - 88usize]; + ["Offset of field: JSJitBackendVTable::entry_fast_grant"] + [::core::mem::offset_of!(JSJitBackendVTable, entry_fast_grant) - 96usize]; }; pub const JS_JIT_BACKEND_OK: _bindgen_ty_6 = 0; pub const JS_JIT_BACKEND_INVALID_ARGUMENT: _bindgen_ty_6 = -1; diff --git a/sys/src/bindings/x86_64-unknown-linux-musl.rs b/sys/src/bindings/x86_64-unknown-linux-musl.rs index a64c09ee5..c5e23284e 100644 --- a/sys/src/bindings/x86_64-unknown-linux-musl.rs +++ b/sys/src/bindings/x86_64-unknown-linux-musl.rs @@ -86,10 +86,11 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; +pub const JS_JIT_FRAME_FAST_ENTRY: u32 = 8; pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; @@ -114,6 +115,9 @@ pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const JS_JIT_FAST_ENTRY_OPTIMIZED: u32 = 1; +pub const JS_JIT_EXIT_FAST_UNPAIRED: u32 = 2147483648; +pub const JS_JIT_FAST_ENTRY_MAX_BUDGET: u32 = 65536; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -3644,6 +3648,57 @@ const _: () = { }; #[repr(C)] #[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryState { + pub struct_size: u32, + pub reserved: u32, + pub epoch: u64, + pub entries: u64, + pub exits: u64, + pub optimized_entries: u64, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryState"][::core::mem::size_of::() - 40usize]; + ["Alignment of JSJitFastEntryState"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryState::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryState, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryState::reserved"] + [::core::mem::offset_of!(JSJitFastEntryState, reserved) - 4usize]; + ["Offset of field: JSJitFastEntryState::epoch"] + [::core::mem::offset_of!(JSJitFastEntryState, epoch) - 8usize]; + ["Offset of field: JSJitFastEntryState::entries"] + [::core::mem::offset_of!(JSJitFastEntryState, entries) - 16usize]; + ["Offset of field: JSJitFastEntryState::exits"] + [::core::mem::offset_of!(JSJitFastEntryState, exits) - 24usize]; + ["Offset of field: JSJitFastEntryState::optimized_entries"] + [::core::mem::offset_of!(JSJitFastEntryState, optimized_entries) - 32usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitFastEntryGrant { + pub struct_size: u32, + pub budget: u32, + pub flags: u32, + pub reserved: u32, + pub state: *mut JSJitFastEntryState, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitFastEntryGrant"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitFastEntryGrant"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitFastEntryGrant::struct_size"] + [::core::mem::offset_of!(JSJitFastEntryGrant, struct_size) - 0usize]; + ["Offset of field: JSJitFastEntryGrant::budget"] + [::core::mem::offset_of!(JSJitFastEntryGrant, budget) - 4usize]; + ["Offset of field: JSJitFastEntryGrant::flags"] + [::core::mem::offset_of!(JSJitFastEntryGrant, flags) - 8usize]; + ["Offset of field: JSJitFastEntryGrant::reserved"] + [::core::mem::offset_of!(JSJitFastEntryGrant, reserved) - 12usize]; + ["Offset of field: JSJitFastEntryGrant::state"] + [::core::mem::offset_of!(JSJitFastEntryGrant, state) - 16usize]; +}; +#[repr(C)] +#[derive(Debug, Copy, Clone)] pub struct JSJitBackendVTable { pub struct_size: u32, pub record_hot: ::core::option::Option< @@ -3691,10 +3746,18 @@ pub struct JSJitBackendVTable { >, pub entry_cache_epoch: ::core::option::Option u64>, + pub entry_fast_grant: ::core::option::Option< + unsafe extern "C" fn( + opaque: *mut ::core::ffi::c_void, + id: u64, + generation: u64, + grant: *mut JSJitFastEntryGrant, + ), + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 96usize]; + ["Size of JSJitBackendVTable"][::core::mem::size_of::() - 104usize]; ["Alignment of JSJitBackendVTable"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitBackendVTable::struct_size"] [::core::mem::offset_of!(JSJitBackendVTable, struct_size) - 0usize]; @@ -3720,6 +3783,8 @@ const _: () = { [::core::mem::offset_of!(JSJitBackendVTable, record_feedback) - 80usize]; ["Offset of field: JSJitBackendVTable::entry_cache_epoch"] [::core::mem::offset_of!(JSJitBackendVTable, entry_cache_epoch) - 88usize]; + ["Offset of field: JSJitBackendVTable::entry_fast_grant"] + [::core::mem::offset_of!(JSJitBackendVTable, entry_fast_grant) - 96usize]; }; pub const JS_JIT_BACKEND_OK: _bindgen_ty_6 = 0; pub const JS_JIT_BACKEND_INVALID_ARGUMENT: _bindgen_ty_6 = -1; diff --git a/sys/tests/jit_patch.rs b/sys/tests/jit_patch.rs index f36861117..f5d9ad3b4 100644 --- a/sys/tests/jit_patch.rs +++ b/sys/tests/jit_patch.rs @@ -56,6 +56,7 @@ fn copy_patches(destination: &std::path::Path) { "0018-inline-frame-recovery.patch", "0019-array-feedback.patch", "0020-typed-array-guard.patch", + "0023-fast-native-entry.patch", ] { fs::copy(source.join(patch), destination.join(patch)).unwrap(); } @@ -114,7 +115,11 @@ fn pinned_public_quickjs_baseline_applies_cleanly_without_git() { let helper_header = fs::read_to_string(destination.join("quickjs-jit-helpers.h")).unwrap(); assert!(quickjs.contains("JS_GetJitRuntimeId")); assert!(quickjs.contains("JS_JIT_FRAME_SIDE_PATH_HIT")); - assert!(jit_header.contains("#define QJSJIT_ABI_MINOR 24u")); + assert!(jit_header.contains("#define QJSJIT_ABI_MINOR 25u")); + assert!(jit_header.contains("#define JS_JIT_FRAME_FAST_ENTRY (1U << 3)")); + assert!(jit_header.contains("JSJitFastEntryGrant *grant);\n} JSJitBackendVTable;")); + assert!(quickjs.contains("JS_JIT_FRAME_SIDE_PATH_HIT |\n JS_JIT_FRAME_FAST_ENTRY;")); + assert!(quickjs.contains("static inline bool qjsjit_fast_entry_ready(")); assert!(jit_header.contains("JSJitPropertyLayout property_layout;")); assert!(quickjs.contains("uint64_t jit_shape_generation;")); assert!(!quickjs.contains("QJSJIT_SHAPE_HASH")); @@ -255,7 +260,7 @@ fn bundled_jit_bindings_include_materialize_owner_tail() { for target in targets { let binding = fs::read_to_string(bindings.join(target)).unwrap(); assert!( - binding.contains("pub const QJSJIT_ABI_MINOR: u32 = 24;"), + binding.contains("pub const QJSJIT_ABI_MINOR: u32 = 25;"), "{target}" ); assert!( @@ -263,6 +268,12 @@ fn bundled_jit_bindings_include_materialize_owner_tail() { "{target}" ); assert!(binding.contains("pub struct JSJitArrayAPI"), "{target}"); + assert!( + binding.contains("pub struct JSJitFastEntryGrant") + && binding.contains("offset_of!(JSJitBackendVTable, entry_fast_grant) - 96usize") + && binding.contains("size_of::() - 104usize"), + "{target}" + ); assert!( binding.contains("receiver: *const JSValue") && binding.contains("pub fn JS_JitGetArrayAPI"), @@ -362,8 +373,18 @@ fn bundled_wasm_jit_binding_matches_array_abi_1_24() { "pub const JS_JIT_FEEDBACK_ARRAY_IMMUTABLE: u32 = 4096;", "pub const JS_JIT_FEEDBACK_ARRAY_SHARED: u32 = 8192;", "pub const JS_JIT_FEEDBACK_ARRAY_INVALID_BACKING: u32 = 16384;", + // ABI 1.25: callback-free native entry. + "pub const QJSJIT_ABI_MINOR: u32 = 25;", + "pub const JS_JIT_FRAME_FAST_ENTRY: u32 = 8;", + "pub const JS_JIT_FAST_ENTRY_OPTIMIZED: u32 = 1;", + "pub const JS_JIT_EXIT_FAST_UNPAIRED: u32 = 2147483648;", + "pub const JS_JIT_FAST_ENTRY_MAX_BUDGET: u32 = 65536;", + "size_of::() - 40usize", + "size_of::() - 20usize", + "offset_of!(JSJitFastEntryGrant, state) - 16usize", + "size_of::() - 52usize", + "offset_of!(JSJitBackendVTable, entry_fast_grant) - 48usize", // ABI 1.24: typed-array continuing guard. - "pub const QJSJIT_ABI_MINOR: u32 = 24;", "pub const QJSJIT_ARRAY_API_VERSION: u32 = 1;", "pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1;", "JSJitArrayQueryStatus_JS_JIT_ARRAY_QUERY_INVALID: JSJitArrayQueryStatus = -1;", diff --git a/sys/tests/typed_array_guard.rs b/sys/tests/typed_array_guard.rs index 2195d1454..8f485471d 100644 --- a/sys/tests/typed_array_guard.rs +++ b/sys/tests/typed_array_guard.rs @@ -105,7 +105,7 @@ int main(void) void *expected_data; uint32_t expected_count; - assert(QJSJIT_ABI_MAJOR == 1 && QJSJIT_ABI_MINOR == 24); + assert(QJSJIT_ABI_MAJOR == 1 && QJSJIT_ABI_MINOR == 25); assert(QJSJIT_ARRAY_API_VERSION == 1); assert(ctx && api); assert(api->struct_size == sizeof(*api)); From 6a56c2a9470d01ec3f89d225e368d437b63afe55 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 16:49:25 +0800 Subject: [PATCH 07/54] perf(jit): eliminate redundant guards and frame sync in guarded property loops - Guarded property leaves no longer republish operand-stack memory or stack_top; every observer (deopt, ownership helpers, generic bridges) already republishes the live prefix from SSA, and can_emit_access admits only borrowed/primitive operands. - After the final property plan, select checked Int32 updates and recompute numeric facts with the planned guarded leaves (and polls) as preserved frame reads, so loop counters crossing field accesses stay Int32 (integer compare, no Float64 fallback). Overflow and operand checks still deopt. - Cached field values carry their checked constant tag; proven inputs to a cached store skip the input tag check. - emit_opt_guard_branch and the generic if_* truthiness guard omit branches whose condition is statically true from emitted constants. - Canonicalize Tier 2 iconsts into the entry block so Cranelift's identity-based constant-phi pass removes loop-carried constant tags/flags. Noisy diagnostic (P-core pinned, 9 alternating runs, median of last 16 warmup batches): property-heavy 2.47x the 82d3808 JIT speed. Co-Authored-By: Claude Opus 5.5 (1M context) --- jit/src/compiler/optimized.rs | 181 ++++++++++++++++++- jit/src/compiler/optimized/const_canon.rs | 130 +++++++++++++ jit/src/compiler/optimized/property_cache.rs | 107 +++++++---- jit/src/ir/optimized.rs | 14 ++ jit/tests/property_cache.rs | 123 ++++++++++++- 5 files changed, 514 insertions(+), 41 deletions(-) create mode 100644 jit/src/compiler/optimized/const_canon.rs diff --git a/jit/src/compiler/optimized.rs b/jit/src/compiler/optimized.rs index afc40c273..497cb0a6a 100644 --- a/jit/src/compiler/optimized.rs +++ b/jit/src/compiler/optimized.rs @@ -231,6 +231,7 @@ mod array_cache; /// subset and attaches exact guard/deopt metadata. Unsupported semantics reject /// the tier and leave Tier 1 installed. mod call_guards; +mod const_canon; mod element_address; mod frame_inline; mod property_cache; @@ -1399,6 +1400,68 @@ fn lower_optimized_machine( } plan }; + // Guarded property leaves are non-reentrant: each either completes + // without running JavaScript or deoptimizes from its exact pre-access + // frame state. Frame slots read across them therefore keep their + // pre-access identities, so loop-carried Int32 induction variables (for + // example `i++` in a loop that updates object fields) no longer degrade to + // mixed Int32/Float64 updates. Selected updates keep their operand check + // and overflow deopt; the proof only removes redundant checks and the + // Float64 fallback. The premise is the final plan: every preserved site + // must be lowered through the planned guarded leaf or compilation fails. + let property_specialized_ir; + let property_numeric; + let (ir, scalar_numeric) = if !int32_loop + && side_path.is_none() + && !property_plan.candidates().is_empty() + && ir.blocks().iter().any(|block| block.is_loop_header()) + { + let preserved = |id: u32| { + property_plan.access(id).is_some() + || ir.nodes().get(id as usize).is_some_and(|node| { + matches!( + node.kind(), + crate::ir::OptimizedNodeKind::GuardNumeric { .. } + ) + }) + }; + if let Some(control) = control { + control.check_ir_bytes( + ir.scalar_graph() + .allocated_bytes() + .saturating_mul(2) + .saturating_add(ir.nodes().len().saturating_mul(128)), + )?; + } + let mut candidate = ir.clone(); + let work = ir.scalar_graph().values().len().saturating_mul(128); + if candidate.specialize_integer_updates_with_preserved_frame_reads(preserved, work) != 0 { + property_numeric = candidate + .scalar_graph() + .proven_numeric_values_with_preserved_frame_reads( + candidate.nodes(), + |index| { + usize::from(index) < usize::from(shape.arguments()) + && matches!( + specialization + .arguments + .get(usize::from(index)) + .copied() + .unwrap_or(specialization.entry), + EntryRepresentation::Int32 + ) + }, + preserved, + work, + ); + property_specialized_ir = candidate; + (&property_specialized_ir, property_numeric) + } else { + (ir, scalar_numeric) + } + } else { + (ir, scalar_numeric) + }; let array_site = |pc| { specialization .arrays @@ -2491,6 +2554,15 @@ fn lower_optimized_machine( access, node, loop_forwarded_property_sites.contains(&node.id()), + match ir.scalar_graph().heap_operation(node.id()) { + Some(crate::ir::ScalarHeapOperation::PutProperty { + value, + .. + }) if access.store => { + scalar_numeric.get(value.index()).copied().flatten() + } + _ => None, + }, depth, &mut stack_provenance, )?; @@ -3553,9 +3625,15 @@ fn lower_optimized_machine( let allowed = builder.ins().bor(numeric, empty); let truth_block = builder.create_block(); let deopt_block = builder.create_block(); - builder - .ins() - .brif(allowed, truth_block, &[], deopt_block, &[]); + if opt_known_condition(builder.func, allowed, 8) == Some(true) { + // A comparison result: the unreachable + // recovery block is still emitted below. + builder.ins().jump(truth_block, &[]); + } else { + builder + .ins() + .brif(allowed, truth_block, &[], deopt_block, &[]); + } builder.switch_to_block(deopt_block); property_cache.flush(&mut builder); for (index, vars) in arguments.iter().enumerate() { @@ -3784,6 +3862,9 @@ fn lower_optimized_machine( builder.seal_all_blocks(); builder.finalize(); } + // Let Cranelift's identity-based constant-phi pass see loop-invariant + // tags/flags that every edge sets to the same constant. + const_canon::canonicalize_integer_constants(&mut clif, 4096); // Helpers that validate a stack map (CALL, GET_GLOBAL, GET_PROPERTY, ...) // are always invoked with map 0, so an artifact that calls anything must // publish the single helper stack map the runtime checks that id against. @@ -4745,8 +4826,10 @@ fn emit_opt_deopt( /// Branches to a fresh pass block when `condition` holds and otherwise to an /// exact deoptimization exit; the builder is left positioned on the pass -/// block. The returned deopt block may be reused by later checks of the same -/// instruction. +/// block. A condition that the emitted instructions already prove true (for +/// example a tag test of a constant tag carried by a guarded value) emits no +/// branch or exit: Cranelift folds such tests to constants but never removes +/// the resulting constant branches from the hot path. fn emit_opt_guard_branch( builder: &mut cranelift_frontend::FunctionBuilder<'_>, env: &OptEnv<'_>, @@ -4755,8 +4838,11 @@ fn emit_opt_guard_branch( pc: u32, guard: u32, condition: cranelift_codegen::ir::Value, -) -> Result { +) -> Result<(), CompileFailure> { use cranelift_codegen::ir::InstBuilder; + if opt_known_condition(builder.func, condition, 8) == Some(true) { + return Ok(()); + } let pass = builder.create_block(); let deopt = builder.create_block(); // Deoptimization exits are cold: keep them out of the hot fallthrough @@ -4766,7 +4852,88 @@ fn emit_opt_guard_branch( builder.switch_to_block(deopt); emit_opt_deopt(builder, env, provenance, depth, pc, guard)?; builder.switch_to_block(pass); - Ok(deopt) + Ok(()) +} + +/// Evaluates a canonical 0/1 boolean built only from `iconst 0/1`, +/// `icmp_imm eq/ne` of a constant and `band`/`bor` of such booleans. `None` means "not statically known"; +/// callers must then emit the dynamic check. `fuel` bounds the recursion. +fn opt_known_condition( + func: &cranelift_codegen::ir::Function, + value: cranelift_codegen::ir::Value, + fuel: u32, +) -> Option { + use cranelift_codegen::ir::{condcodes::IntCC, InstructionData, Opcode, ValueDef}; + let fuel = fuel.checked_sub(1)?; + let ValueDef::Result(inst, 0) = func.dfg.value_def(func.dfg.resolve_aliases(value)) else { + return None; + }; + let constant = |value: cranelift_codegen::ir::Value| -> Option { + let value = func.dfg.resolve_aliases(value); + let ValueDef::Result(inst, 0) = func.dfg.value_def(value) else { + return None; + }; + match func.dfg.insts[inst] { + InstructionData::UnaryImm { + opcode: Opcode::Iconst, + imm, + } => { + let bits = func.dfg.value_type(value).bits(); + let raw = imm.bits(); + Some(if bits >= 64 { + raw + } else { + raw & ((1i64 << bits) - 1) + }) + } + _ => None, + } + }; + match func.dfg.insts[inst] { + InstructionData::UnaryImm { + opcode: Opcode::Iconst, + .. + } => match constant(value)? { + // Only canonical booleans: band/bor below are bitwise. + 0 => Some(false), + 1 => Some(true), + _ => None, + }, + InstructionData::IntCompareImm { + opcode: Opcode::IcmpImm, + cond, + arg, + imm, + } => { + let bits = func.dfg.value_type(arg).bits(); + let lhs = constant(arg)?; + let rhs = if bits >= 64 { + imm.bits() + } else { + imm.bits() & ((1i64 << bits) - 1) + }; + match cond { + IntCC::Equal => Some(lhs == rhs), + IntCC::NotEqual => Some(lhs != rhs), + _ => None, + } + } + InstructionData::Binary { + opcode: opcode @ (Opcode::Band | Opcode::Bor), + args: [lhs, rhs], + } => { + let lhs = opt_known_condition(func, lhs, fuel); + let rhs = opt_known_condition(func, rhs, fuel); + match (opcode, lhs, rhs) { + (Opcode::Band, Some(false), _) | (Opcode::Band, _, Some(false)) => Some(false), + (Opcode::Band, Some(true), Some(true)) => Some(true), + (Opcode::Bor, Some(true), _) | (Opcode::Bor, _, Some(true)) => Some(true), + (Opcode::Bor, Some(false), Some(false)) => Some(false), + _ => None, + } + } + _ => None, + } } fn opt_tag_is( diff --git a/jit/src/compiler/optimized/const_canon.rs b/jit/src/compiler/optimized/const_canon.rs new file mode 100644 index 000000000..47117abe3 --- /dev/null +++ b/jit/src/compiler/optimized/const_canon.rs @@ -0,0 +1,130 @@ +//! Integer-constant canonicalization before Cranelift optimization. +//! +//! Tier 2 lowering defines tag and flag variables with fresh `iconst` +//! instructions in whichever block happens to write them. A loop-carried +//! representation that is the same constant on every edge therefore reaches +//! the loop header through distinct SSA values, and Cranelift's constant-phi +//! pass (which compares value identity) keeps the block parameter alive. The +//! redundant parameters cost registers, spills and tag tests in hot loops. +//! +//! Replacing every `iconst` with one canonical definition per (type, bits) at +//! the start of the entry block makes those edges pass the identical value, so +//! the existing pass removes the parameters. Constants dominate every use from +//! the entry block, the rewrite only introduces aliases, and Cranelift's +//! e-graph rematerializes constants near their uses, so semantics and +//! instruction selection of immediates are unchanged. + +use std::collections::HashMap; + +use cranelift_codegen::cursor::{Cursor, FuncCursor}; +use cranelift_codegen::ir::{Function, Inst, InstBuilder, InstructionData, Opcode, Type}; + +/// Canonicalizes at most `limit` distinct constants; returns how many +/// redundant `iconst` instructions were replaced. Exceeding the limit leaves +/// the remaining constants untouched (never a correctness issue). +pub(super) fn canonicalize_integer_constants(func: &mut Function, limit: usize) -> usize { + let Some(entry) = func.layout.entry_block() else { + return 0; + }; + let mut constants = Vec::new(); + for block in func.layout.blocks() { + for inst in func.layout.block_insts(block) { + if let InstructionData::UnaryImm { + opcode: Opcode::Iconst, + imm, + } = func.dfg.insts[inst] + { + let result = func.dfg.first_result(inst); + constants.push((inst, func.dfg.value_type(result), imm.bits())); + } + } + } + let mut canonical = HashMap::<(Type, i64), Inst>::new(); + let mut replaced = 0; + for (inst, ty, bits) in constants { + let key = (ty, bits); + let target = match canonical.get(&key) { + Some(&target) => target, + None => { + if canonical.len() == limit { + continue; + } + let mut cursor = FuncCursor::new(func).at_first_insertion_point(entry); + let value = cursor.ins().iconst(ty, bits); + let target = cursor + .func + .dfg + .value_def(value) + .inst() + .expect("iconst defines an instruction result"); + canonical.insert(key, target); + target + } + }; + func.dfg.replace_with_aliases(inst, target); + func.layout.remove_inst(inst); + replaced += 1; + } + replaced +} + +#[cfg(test)] +mod tests { + use super::*; + use cranelift_codegen::ir::{types, AbiParam, Signature}; + use cranelift_codegen::isa::CallConv; + use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext}; + + #[test] + fn identical_loop_edge_constants_become_one_value() { + let mut signature = Signature::new(CallConv::SystemV); + signature.params.push(AbiParam::new(types::I64)); + signature.returns.push(AbiParam::new(types::I64)); + let mut func = Function::with_name_signature(Default::default(), signature); + let mut context = FunctionBuilderContext::new(); + { + let mut builder = FunctionBuilder::new(&mut func, &mut context); + let entry = builder.create_block(); + let header = builder.create_block(); + let exit = builder.create_block(); + builder.append_block_params_for_function_params(entry); + builder.append_block_param(header, types::I64); + builder.switch_to_block(entry); + let n = builder.block_params(entry)[0]; + let zero = builder.ins().iconst(types::I64, 0); + builder.ins().jump(header, &[zero]); + builder.switch_to_block(header); + let tag = builder.block_params(header)[0]; + let again = builder.ins().iconst(types::I64, 0); + let done = builder + .ins() + .icmp_imm(cranelift_codegen::ir::condcodes::IntCC::Equal, n, 0); + builder.ins().brif(done, exit, &[], header, &[again]); + builder.switch_to_block(exit); + builder.ins().return_(&[tag]); + builder.seal_all_blocks(); + builder.finalize(); + } + assert_eq!(canonicalize_integer_constants(&mut func, 16), 2); + let jumps: Vec<_> = func + .layout + .blocks() + .flat_map(|block| func.layout.block_insts(block).collect::>()) + .filter_map(|inst| { + let destinations = func.dfg.insts[inst].branch_destination(&func.dfg.jump_tables); + destinations + .iter() + .flat_map(|call| call.args_slice(&func.dfg.value_lists).to_vec()) + .next() + }) + .map(|value| func.dfg.resolve_aliases(value)) + .collect(); + assert_eq!(jumps.len(), 2); + assert_eq!( + jumps[0], jumps[1], + "both edges must pass one canonical value" + ); + // A limit of zero is a no-op. + assert_eq!(canonicalize_integer_constants(&mut func, 0), 0); + } +} diff --git a/jit/src/compiler/optimized/property_cache.rs b/jit/src/compiler/optimized/property_cache.rs index a8cfbddfd..d17fc0bdb 100644 --- a/jit/src/compiler/optimized/property_cache.rs +++ b/jit/src/compiler/optimized/property_cache.rs @@ -7,12 +7,26 @@ use cranelift_frontend::{FunctionBuilder, Variable}; struct CachedProperty { candidate: PropertyCandidate, + /// The guarded representation. Every definition that can be observed + /// while `valid` (or `dirty`) holds carries exactly this tag. + tag: i32, value: OptVars, data: Variable, valid: Variable, dirty: Variable, } +impl CachedProperty { + /// Read a valid (or dirty) cached value. Its tag is a compile-time + /// constant: loop-carried tag phis would otherwise survive Cranelift's + /// constant-phi removal and keep a redundant tag check in the loop. + fn checked_value(&self, builder: &mut FunctionBuilder<'_>) -> OptPair { + let payload = builder.use_var(self.value.payload); + let tag = builder.ins().iconst(types::I64, i64::from(self.tag)); + OptPair { payload, tag } + } +} + #[derive(Default)] pub(super) struct PropertyCache { entries: Vec, @@ -40,6 +54,7 @@ impl PropertyCache { *next_var = first.checked_add(5).ok_or(CompileFailure::ResourceLimit)?; let entry = CachedProperty { candidate, + tag: primitive_tag(candidate.observation.value())?, value: OptVars { payload: Variable::from_u32(first), tag: Variable::from_u32(first + 1), @@ -83,7 +98,7 @@ impl PropertyCache { builder.ins().brif(dirty, commit, &[], done, &[]); builder.switch_to_block(commit); let data = builder.use_var(entry.data); - let value = opt_use(builder, entry.value); + let value = entry.checked_value(builder); opt_store_at(builder, data, 0, value); let no = builder.ins().iconst(types::I8, 0); builder.def_var(entry.dirty, no); @@ -192,7 +207,17 @@ impl PropertyCache { builder.set_cold_block(checked); builder.ins().brif(tag_ok, checked, &[], deopt, &[]); builder.switch_to_block(checked); - opt_define(builder, entry.value, current); + // The tag was just checked; carry it as a constant so loop-carried + // cache values have a provably fixed representation. + let checked_tag = builder.ins().iconst(types::I64, i64::from(expected_tag)); + opt_define( + builder, + entry.value, + OptPair { + payload: current.payload, + tag: checked_tag, + }, + ); builder.def_var(entry.data, data); let yes = builder.ins().iconst(types::I8, 1); let no = builder.ins().iconst(types::I8, 0); @@ -253,6 +278,7 @@ impl PropertyCache { access: &PropertyAccessPlan, node: &crate::ir::OptimizedNode, loop_forwarded: bool, + proven_input: Option, depth: usize, provenance: &mut [OptProvenance], ) -> Result { @@ -372,7 +398,15 @@ impl PropertyCache { let checked = builder.create_block(); builder.ins().brif(tag_ok, checked, &[], deopt, &[]); builder.switch_to_block(checked); - opt_define(builder, entry.value, current); + let checked_tag = builder.ins().iconst(types::I64, i64::from(expected_tag)); + opt_define( + builder, + entry.value, + OptPair { + payload: current.payload, + tag: checked_tag, + }, + ); builder.def_var(entry.data, data); let yes = builder.ins().iconst(types::I8, 1); builder.def_var(entry.valid, yes); @@ -384,13 +418,34 @@ impl PropertyCache { let input_ok = builder .ins() .icmp_imm(IntCC::Equal, value.tag, i64::from(expected_tag)); - let commit = builder.create_block(); - builder.ins().brif(input_ok, commit, &[], deopt, &[]); - builder.switch_to_block(commit); + // A statically proven input representation (for example a + // checked Int32 arithmetic result, whose lowering always carries + // JS_TAG_INT) needs no branch in the hot path. + let proven = matches!( + (proven_input, expected_tag), + (Some(crate::ir::ScalarNumericMode::Int32), qjs::JS_TAG_INT) + | ( + Some(crate::ir::ScalarNumericMode::Float64), + qjs::JS_TAG_FLOAT64 + ) + ); + if !proven && opt_known_condition(builder.func, input_ok, 8) != Some(true) { + let commit = builder.create_block(); + builder.ins().brif(input_ok, commit, &[], deopt, &[]); + builder.switch_to_block(commit); + } // This is the semantic store. Recovery may publish it from here // onward, but a failing incoming-value guard must retain the prior // completed value instead of writing the failed input. - opt_define(builder, entry.value, value); + let checked_tag = builder.ins().iconst(types::I64, i64::from(expected_tag)); + opt_define( + builder, + entry.value, + OptPair { + payload: value.payload, + tag: checked_tag, + }, + ); let yes = builder.ins().iconst(types::I8, 1); builder.def_var(entry.dirty, yes); } @@ -402,40 +457,25 @@ impl PropertyCache { emit_opt_deopt(builder, env, provenance, depth, node.pc(), guard)?; builder.switch_to_block(continuation); - // Preserve the existing leaf path's stack ownership publication: roots - // are represented by non-owning placeholders and primitives by values. - let undefined = OptPair { - payload: builder.ins().iconst(types::I64, 0), - tag: builder - .ins() - .iconst(types::I64, i64::from(qjs::JS_TAG_UNDEFINED)), - }; - for (index, source) in provenance.iter().take(object_index).enumerate() { - let value = if *source == OptProvenance::ImmediatePrimitive { - opt_use(builder, env.stack[index]) - } else { - undefined - }; - opt_store(builder, env.stack_base, index, value); - } + // No stack memory is written on this leaf path. Every live operand is + // a borrowed argument/local or an immediate primitive (see + // `can_emit_access`), so no owner can be lost, and every observer of + // interpreter stack memory republishes it from SSA first: deopt + // (`emit_opt_deopt`), ownership helpers (`opt_own_stack_for_exit` + // callers) and generic bridges all store the live prefix and set + // `stack_top` themselves. Leaving memory and `stack_top` untouched + // preserves the invariant that `[stack_base, stack_top)` holds only + // values the interpreter may release, which is all an interrupt + // unwind can observe. let new_depth = if access.store { provenance[object_index..depth].fill(OptProvenance::Unknown); object_index } else { - let value = opt_use(builder, entry.value); + let value = entry.checked_value(builder); opt_define(builder, env.stack[object_index], value); provenance[object_index] = OptProvenance::ImmediatePrimitive; - opt_store(builder, env.stack_base, object_index, value); depth }; - opt_set_stack_top( - builder, - env.frame, - env.stack_base, - new_depth, - env.pointer_type, - env.layout, - ); Ok(new_depth) } } @@ -568,6 +608,7 @@ mod tests { let cache = PropertyCache { entries: vec![CachedProperty { candidate: candidate(), + tag: rquickjs_core::qjs::JS_TAG_INT, value: OptVars { payload: Variable::from_u32(0), tag: Variable::from_u32(1), diff --git a/jit/src/ir/optimized.rs b/jit/src/ir/optimized.rs index 635a2cbd3..29ab989bb 100644 --- a/jit/src/ir/optimized.rs +++ b/jit/src/ir/optimized.rs @@ -494,6 +494,20 @@ impl OptimizedIr { pub fn scalar_graph(&self) -> &super::ScalarGraph { &self.scalar_graph } + + /// Select checked Int32 updates whose loop-carried inputs are proven only + /// through frame reads that `preserve` authorizes (for example, planned + /// guarded property leaves). The caller must emit exactly those guarded + /// lowerings, or fail compilation. Returns the number of selected updates; + /// zero leaves the graph untouched. + pub(crate) fn specialize_integer_updates_with_preserved_frame_reads( + &mut self, + preserve: impl Fn(u32) -> bool, + work: usize, + ) -> usize { + self.scalar_graph + .specialize_integer_updates_with_preserved_frame_reads(&self.nodes, preserve, work) + } pub fn nodes(&self) -> &[OptimizedNode] { &self.nodes } diff --git a/jit/tests/property_cache.rs b/jit/tests/property_cache.rs index ead895832..2ea418a3d 100644 --- a/jit/tests/property_cache.rs +++ b/jit/tests/property_cache.rs @@ -201,8 +201,18 @@ fn minimum_native_shape_checks(clif: &str) -> Option { let operation = line.split(" ;").next().unwrap(); operation .strip_prefix("store ") + .or_else(|| operation.strip_prefix("store.i32 ")) .and_then(|store| store.strip_suffix(", v0")) - .and_then(|value| constants.get(value)) + .and_then(|mut value| { + // Constants may be canonicalized into aliases. + for _ in 0..aliases.len() { + let Some(source) = aliases.get(value) else { + break; + }; + value = source; + } + constants.get(value) + }) .copied() == Some(rquickjs::qjs::JSJitExitKind_JS_JIT_EXIT_DONE as i32) }) { @@ -874,3 +884,114 @@ fn interrupt_observes_committed_fields_and_mutation_invalidates_loop_cache() { assert!(jit.metrics().tier2_entries > before.tier2_entries); assert_eq!(jit.metrics().native_entries, jit.metrics().native_exits); } + +#[test] +fn guarded_property_loop_keeps_its_induction_variable_int32() { + // Guarded property leaves cannot reenter JavaScript, so the loop counter + // read across them keeps its Int32 seed: no Float64 fallback for `i++` + // and an integer loop comparison. + let (clif, _) = property_clif( + "(function(n,seed,o){o.x=seed;for(let i=0;i((object, 0, 9)).unwrap(), 10); + }) + }); + let before = jit.metrics(); + context.with(|ctx| { + let f: Function = ctx.globals().get("target").unwrap(); + let object: Object = ctx.globals().get("shared").unwrap(); + // `i++` overflows Int32 after the last iteration: the checked update + // must deoptimize and let the interpreter finish with a double. + object.set("x", 0).unwrap(); + assert_eq!( + f.call::<_, i32>((object.clone(), i32::MAX - 3, i32::MAX)) + .unwrap(), + 4 + ); + assert_eq!(object.get::<_, i32>("x").unwrap(), 4); + object.set("x", 100).unwrap(); + assert_eq!(f.call::<_, i32>((object.clone(), 0, 4)).unwrap(), 105); + // A Float64 seed fails the update's operand check instead of being + // truncated by the Int32 update. + object.set("x", 0).unwrap(); + assert_eq!(f.call::<_, i32>((object.clone(), 0.5, 3.0)).unwrap(), 3); + assert_eq!(object.get::<_, i32>("x").unwrap(), 3); + }); + assert!(jit.metrics().tier2_entries > before.tier2_entries); + assert_eq!(jit.metrics().native_entries, jit.metrics().native_exits); +} + +#[test] +fn terminating_interrupt_after_unpublished_property_reads_unwinds_cleanly() { + use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Arc, + }; + // Property leaves no longer publish their stack operands. A terminating + // interrupt at the loop poll must still unwind only releasable values. + let (runtime, jit, context) = setup( + "globalThis.shared={x:1,y:2};function target(o,n){let s=0; + for(let i=0;i((object, 3)).unwrap(), 9); + }) + }); + let calls = Arc::new(AtomicUsize::new(0)); + let observed = calls.clone(); + runtime.set_interrupt_handler(Some(Box::new(move || { + observed.fetch_add(1, Ordering::SeqCst) >= 2 + }))); + let before = jit.metrics(); + context.with(|ctx| { + let f: Function = ctx.globals().get("target").unwrap(); + let object: Object = ctx.globals().get("shared").unwrap(); + assert!(f.call::<_, i32>((object, 10_000_000)).is_err()); + let _ = ctx.catch(); + }); + runtime.set_interrupt_handler(None); + assert!(calls.load(Ordering::SeqCst) > 2); + context.with(|ctx| { + let f: Function = ctx.globals().get("target").unwrap(); + let object: Object = ctx.globals().get("shared").unwrap(); + object.set("x", 1).unwrap(); + object.set("y", 2).unwrap(); + assert_eq!(f.call::<_, i32>((object.clone(), 3)).unwrap(), 9); + assert_eq!(object.get::<_, i32>("x").unwrap(), 4); + }); + assert!(jit.metrics().tier2_entries > before.tier2_entries); + assert_eq!(jit.metrics().native_entries, jit.metrics().native_exits); + runtime.run_gc(); +} From e24a0661fdbedf50dc34f1f017fe783949e46f9a Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 16:49:25 +0800 Subject: [PATCH 08/54] docs: record p4a guard-elimination slice and remaining work Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p4a-guard-elim.md | 101 +++++++++++++++++++++++++++++++++ 1 file changed, 101 insertions(+) create mode 100644 docs/roadmap/p4a-guard-elim.md diff --git a/docs/roadmap/p4a-guard-elim.md b/docs/roadmap/p4a-guard-elim.md new file mode 100644 index 000000000..e4316ef40 --- /dev/null +++ b/docs/roadmap/p4a-guard-elim.md @@ -0,0 +1,101 @@ +# P4a: guard elimination on guarded property loops (first slice) + +Branch `perf/p4a-guard-elim`, based on `82d3808`. Roadmap item: P4 in +[PERFORMANCE_ROADMAP_V8_BUN.md](../PERFORMANCE_ROADMAP_V8_BUN.md) (redundant +guard elimination, LICM, load/store forwarding on semantic SSA). + +## Starting point + +`#29` already promoted monomorphic primitive own-data fields of an argument +receiver into SSA (`ir/property.rs`, `compiler/optimized/property_cache.rs`): +the shape+generation guard runs once, later accesses are forwarded from the +cached value, stores are sunk until a flush boundary, and the cold amortized +poll publishes, invalidates and revalidates the cache. The `property-heavy` +loop was therefore already free of field loads and shape guards. The remaining +per-iteration cost came from elsewhere in the same loop: + +1. Every guarded property leaf re-published the operand stack into the + interpreter frame (placeholders, primitives and `stack_top`): about 16 + memory stores per `property-heavy` iteration. +2. Frame reads across property operations were treated as reentrant, so the + loop counter `i` degraded to a mixed Int32/Float64 value. `i++` computed + both an Int32 and a Float64 update, and `i < n` converted both operands to + doubles. +3. Cached field tags, and loop-carried tags and flags in general, were block + parameters. Cranelift's identity-based constant-phi pass kept them because + each edge used a different `iconst`, which left tag guards and spills in + the loop. Tag tests folded to constants still left constant `brif`s. + +## What this slice does + +- **Guarded property leaves no longer publish stack memory.** Every observer + of interpreter stack memory (deopt, ownership helpers, generic bridges) + already republishes the live prefix from SSA and sets `stack_top` itself. + `can_emit_access` guarantees that no live operand is an owner. Leaving + memory untouched preserves the invariant that `[stack_base, stack_top)` only + holds values that the interpreter may release, which is all a terminating + interrupt unwind can see. +- **Induction proofs across guarded property leaves.** After the final + property plan is known, lowering specializes Number updates to checked Int32 + updates, using `specialize_integer_updates_with_preserved_frame_reads` with + exactly the planned guarded sites and polls as preserved frame reads. It then + recomputes the numeric facts with the same premise. The IR is cloned and + charged to the IR budget first. Selected updates keep their operand check + and overflow deopt. A planned site that cannot be emitted still fails + compilation. +- **Constant representation of cached fields.** Cached values are defined with + the checked constant tag, and reads and flushes use that constant. + Proven Int32/Float64 inputs to a cached store skip the input tag check. +- **Static guard folding.** `emit_opt_guard_branch` and the generic `if_*` + truthiness guard emit no branch when the condition is provably true from the + emitted constants: `iconst 0/1`, `icmp_imm eq/ne` of a constant, and + `band`/`bor` of such booleans. +- **Constant canonicalization** (`compiler/optimized/const_canon.rs`). Tier 2 + lowering rewrites every `iconst` to a single canonical definition per type + and value in the entry block. Cranelift's constant-phi pass then removes + loop-carried tag and flag parameters, and its e-graph rematerializes the + constants near their uses. + +The resulting `property-heavy` hot loop, after Cranelift optimization, has no +tag guards, no Float64 operations, no stack or field stores, and no constant +branches. What remains is three checked `sadd`/`ssub`, the checked `i++`, an +integer compare and the poll countdown. + +## Evidence + +Tests (all in `jit/tests/property_cache.rs`): +`guarded_property_loop_keeps_its_induction_variable_int32` (structural: no +`fcvt_from_sint`/`fcmp`/`fadd`, integer compare, field-free continuing loop), +`specialized_property_loop_counter_overflow_and_float_seed_deoptimize_exactly` +(the `i++` Int32 overflow and a Float64 seed both deopt exactly), and +`terminating_interrupt_after_unpublished_property_reads_unwinds_cleanly` +(stress GC, a terminating interrupt with primitives below property reads, then +a clean re-entry). The `const_canon` unit test checks that both loop edges pass +one canonical value. The CLIF-walking helper in `property_cache.rs` now +resolves constant aliases. + +Performance: see the branch summary. These are noisy single-host diagnostics, +not publishable matrix data. The integrator must run the three-engine matrix. + +## Remaining work (not in this slice) + +- **Register pressure in property loops.** The loop still spills `i`, `n` and + `toggle` because many argument, local and cache variables stay live across + the loop. Sinking unused argument tags and moving cache data pointers + out of the loop Phis (they are loop-invariant except on the cold poll edge) + should remove the spills. +- **Loop-invariant guard hoisting for non-argument receivers.** Property + candidates are still limited to unmodified arguments (`facts.entry_argument`). + Local and `this` receivers, and receivers loaded from other objects, need + identity facts plus a preheader guard with an exact deopt map. This is also + B5 (`float64array-traversal`). +- **Alias-aware forwarding for distinct receivers with the same atom** + (`mixed-quotes` `right.score - left.score`). Two different arguments with + the same atom flush and invalidate each other; a runtime identity check + (`a != b`) in the preheader could split the paths. `mixed-quotes` is + dominated by per-call bookkeeping (B2), so this slice does not move it. +- **Store sinking across calls** stays a FlushInvalidate boundary. +- **Inherited, pre-existing nondeterminism:** a 20x-longer `property-heavy` + loop sometimes never reaches Tier 2 after an OSR attempt. This happens on + both the `82d3808` baseline and this branch (in 4 of 6 baseline runs), so it + is a tier-up scheduling issue and not related to this change. From e243f15a3c5281bdf924c57560c4bf547728bf40 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 16:50:39 +0800 Subject: [PATCH 09/54] test(jit): cover packed local receivers; document P1-6 slice Adds an interpreter-differential regression for packed-array receivers held in locals and records the noisy float64array-traversal/arrays-typed diagnostic plus remaining work in docs/roadmap. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p1d-typedarray-receivers.md | 138 +++++++++++++++++++++++ jit/tests/typed_receivers.rs | 70 ++++++++++++ 2 files changed, 208 insertions(+) create mode 100644 docs/roadmap/p1d-typedarray-receivers.md diff --git a/docs/roadmap/p1d-typedarray-receivers.md b/docs/roadmap/p1d-typedarray-receivers.md new file mode 100644 index 000000000..d695f9158 --- /dev/null +++ b/docs/roadmap/p1d-typedarray-receivers.md @@ -0,0 +1,138 @@ +# P1 item 6: typed-array receivers held in locals + +Branch `perf/p1d-typedarray-receivers`, based on `82d3808`. This note covers +roadmap item P1-6 (B5 in +[PERFORMANCE_ROADMAP_V8_BUN.md](../PERFORMANCE_ROADMAP_V8_BUN.md)). + +## Problem + +`float64array-traversal` reads both views from `buffers.ints` and +`buffers.floats` into `const` locals. `ArrayPlan` accepted only receivers +whose SSA identity was an entry argument. As a result, the 0020 typed-array +leaf query was never used for this loop. Every iteration ran `ints.length` +through `JS_JitHelperGetProperty` and the intrinsic getter, and ran a leaf +query for each `floats[i]` store. Poll amortization was also disabled, so +the loop called the poll helper on every iteration. + +## What changed + +- **Receiver identity.** A candidate receiver is now + `ArrayReceiver::{Argument, Local}`. A local is accepted only when both of + these hold: + - It is the single frame slot whose SSA value matches the operand at the + access (`KnownFacts::same_value`). Value identity is followed through + Phis that agree and through frame reads kept by the guarded-site + contract. + - No node in the loop writes that local. + + Aliased receivers, receivers also held in an argument, and rebound + receivers are not planned. +- **Typed `get_length`.** A planned Int32/Float64 length is lowered to the + versioned leaf query with `JS_JIT_ARRAY_QUERY_LENGTH`. The query checks + the intrinsic getter chain and a fixed, attached backing store. If the + check fails, the code takes an exact deoptimization exit before the + bytecode runs. It never falls back to the bridge that can run an + accessor. +- **Several live metadata tuples.** Lowering keeps up to four guarded + tuples, keyed by frame slot. The old code kept a single slot. + - Every reentrant or frame-writing boundary still clears all tuples. + - Tuples survive forward joins only when every predecessor ends with the + same tuple. The shared Cranelift values then dominate the join. + - This lets the `floats[i]` store and the following load share one + tuple, while the `ints` tuple stays live for the same iteration. +- **Storage-only hoists.** Consider an Int32/Float64 receiver that is used + in a loop, has no hoisted length, and passes the existing loop admission + (no uncertified reentry and no write to its slot). An example is a store + destination. For such a receiver: + - The preheader runs one storage query, without the length flag. + - The query is revalidated on every poll, both amortized and regular. + - Each access keeps its own index check. +- **Leaf certification.** + - Numeric `push_const`/`push_const8` sites are treated as non-reentrant. + Lowering can only emit immediates for them; any other constant fails + compilation. + - Guarded `to_propkey` sites are also accepted for poll amortization. +- **Poll amortization.** Two narrow extensions: + - A store into an *owned* (helper-derived heap) local outside every + natural loop is accepted. The loop-header representation guard already + treats owned locals as `Any`. + - The numeric proof used only for this decision keeps frame reads across + shape-guarded `get_field` and other certified leaf sites. Lowering still + uses the original proof. + +All exits remain exact deoptimizations at the original bytecode. + +## Tests + +`jit/tests/typed_receivers.rs`: + +- CLIF test for the traversal: + - no `length` bridge call; + - exactly two preheader queries and two cold poll revalidations; + - no bounds check on the range-proven `ints` load. +- Rebound and aliased receivers are not hoisted. +- Production Tier 2 run with stress GC: results match the interpreter, and + `GET_PROPERTY` is no longer called on every iteration. +- 17 changed-receiver probes, compared with the same-version interpreter: + - wrong mode, and packed or plain-array receivers; + - a destination shorter than the source; + - own `length` override and prototype getter override; + - detached source or destination; + - one view over a shared buffer aliasing the other; + - the same object as both receivers; + - an accessor on `buffers`; + - a string seed. +- The source or destination buffer is detached from an interrupt handler + during the native loop. This covers poll revalidation of local receivers. + +`array_cache` unit tests cover: + +- a local candidate with its length hoist; +- rejection of rebound, aliased and argument-aliased locals; +- a storage-only hoist for a store destination. + +## Noisy diagnostic (not publishable evidence) + +This was measured on a shared, loaded 24-core x86_64 host. Each row is 7 +alternating runs of the `82d3808` baseline `jit-bench` and this branch, +using `worker --mode automatic`. Each value is the median over runs of +`median(protocol.warmup_batch_ns[-16:])`, in ms per 10 workload calls. Speed +is baseline / new, so values above 1.00x are faster. Checksums match the +interpreter mode. + +| Workload | Baseline ms | Branch ms | Speed | +| --- | ---: | ---: | ---: | +| float64array-traversal | 1.0922 | 0.1574 | 6.94x | +| arrays-typed | 2.9833 | 2.0123 | 1.48x | +| int32array-traversal (control) | 0.0497 | 0.0495 | tied (bimodal E/P-core samples) | +| float64-dense (control) | 0.1296 | 0.1309 | tied | + +In a 3-run spot check, property-heavy, call-heavy, numeric, scalar-loop, +objects-polymorphic, mixed-quotes, quickjs-int-arith, fibonacci-iterative, +generic-call-entry and scalar-expressions were all within ±7%. Reruns of +quickjs-bitops and host-compute with 7 runs were bimodal ties. The +publishable three-engine matrix, with Bun and confidence intervals, is still +pending and belongs to the integrator. + +## Remaining work + +- float64array-traversal takes about 7.9 ns per iteration against about + 4.1 ns for Bun (roadmap section 3). What is left: + - the `floats` store and the reload after it each keep a bounds check; + - the amortized poll counter; + - the boxed 16-byte `sum` value. + + These belong to P4 bounds-check elimination and store-to-load forwarding. +- arrays-typed is now limited by its `workload` orchestration: `push`, + `new Int32Array(values)` and `toFixed` stay in the interpreter or in + helpers. +- Pre-existing case: `const a = arg; a[i] = v` plans the receiver as + `Argument` because its SSA identity is the entry argument, while the + operand provenance is `Local`. The typed-store provenance check therefore + fails compilation, which is fail-closed. Loads and lengths no longer + depend on provenance matching. Stores could use the same + operand-guarded query. +- Packed-array local receivers go through the same planner path. One + interpreter-differential test covers them: snapshot and live length, + holes, a typed receiver, overflow, an extended length and non-Int32 + elements. None of their CLIF shape is asserted yet. diff --git a/jit/tests/typed_receivers.rs b/jit/tests/typed_receivers.rs index 4aad4d695..1d64aeaac 100644 --- a/jit/tests/typed_receivers.rs +++ b/jit/tests/typed_receivers.rs @@ -418,3 +418,73 @@ fn local_receiver_poll_case(detach_source: bool) { assert!(jit.metrics().tier2_entries > before.tier2_entries); assert_eq!(jit.metrics().native_entries, jit.metrics().native_exits); } + +#[test] +fn packed_local_receivers_match_the_same_version_interpreter() { + const PACKED: &str = r#" + function sumSnapshot(o) { + const values = o.values; + let n = values.length, sum = 0; + for (let i = 0; i < n; i++) sum = (sum + (values[i] | 0)) | 0; + return sum; + } + function sumLive(o) { + const values = o.values; + let sum = 0; + for (let i = 0; i < values.length; i++) sum = (sum + (values[i] | 0)) | 0; + return sum; + } + "#; + const CASES: &str = r#"JSON.stringify([ + sumSnapshot({values: []}), sumLive({values: []}), + sumSnapshot({values: [7]}), sumLive({values: [7]}), + sumSnapshot({values: [1,,3]}), sumLive({values: [1,,3]}), + sumSnapshot({values: new Int32Array([4,5])}), sumLive({values: new Int32Array([4,5])}), + sumSnapshot({values: [2147483647,1]}), sumLive({values: [2147483647,1]}), + (() => { const a = [1,2,3]; a.length = 10; return [sumSnapshot({values: a}), sumLive({values: a})]; })(), + sumSnapshot({values: [1.5, 'x', {}]}), sumLive({values: [1.5, 'x', {}]}) + ])"#; + let expected = evaluate_without_jit(PACKED, CASES); + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .call_threshold(2) + .loop_threshold(4) + .force_optimized_for_test(true) + .stress_gc(true) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| ctx.eval::<(), _>(PACKED)).unwrap(); + let deadline = Instant::now() + Duration::from_secs(60); + loop { + let before = jit.metrics(); + assert_eq!( + context + .with(|ctx| ctx.eval::( + "sumSnapshot({values: [1,2,3,4]}) + sumLive({values: [1,2,3,4]})" + )) + .unwrap(), + 20 + ); + let after = jit.metrics(); + if after.tier2_entries >= before.tier2_entries + 2 && after.pending_worker_jobs == 0 { + break; + } + jit.poll(); + assert!( + Instant::now() < deadline, + "packed local traversals never reached Tier2: {after:?}" + ); + std::thread::sleep(Duration::from_millis(1)); + } + let before = jit.metrics(); + let actual = context.with(|ctx| ctx.eval::(CASES)).unwrap(); + let after = jit.metrics(); + assert_eq!(actual, expected); + assert_eq!(after.native_entries, after.native_exits); + assert!(after.tier2_entries > before.tier2_entries); +} From cb1dbc2ab1c684ea2189a47de2297f9f0053d468 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 16:53:07 +0800 Subject: [PATCH 10/54] feat(jit): admit closures in Tier 1 (fclosure, var refs, close_loc, set_name) Add ABI 1.25 (runtime API 1.10) append-only closure helpers in sys/patches/0025-tier1-closures.patch: FCLOSURE, GET_VAR_REF, PUT_VAR_REF (plain/check modes), CLOSE_LOC and SET_NAME. Each mirrors its interpreter opcode on the validated current frame, including the exact TDZ ReferenceError, set_value ownership and close_lexical_var detaching. Bundled bindings, patch digests, fingerprints and ABI tests are updated. Tier 1 now admits fclosure/fclosure8, get_var_ref0-3/get_var_ref/ get_var_ref_check, put_var_ref0-3/put_var_ref/set_var_ref0-3/set_var_ref/ put_var_ref_check, close_loc and set_name. Attached var refs alias the interpreter's var_buf/arg_buf, so a function that creates closures reloads its native argument/local copies after every helper and poll (a closure may have written them during a call). Functions combining fclosure with put_loc_check_init stay rejected; put_var_ref_check_init and make_var_ref* remain ClosureFrame rejects. Tier 2 and frame inlining keep rejecting closure-creating callees. Adds opcode-case manifest entries for all 21 newly advertised opcodes and tier1_closures regression tests (captured local/argument reload, type-change deopt, stress-GC ownership, per-iteration close_loc, TDZ get/put errors, shared var refs, set_var_ref keep, function names). Co-Authored-By: Claude Opus 5.5 (1M context) --- jit/Cargo.toml | 4 + jit/src/abi.rs | 24 +- jit/src/bytecode/policy.rs | 5 + jit/src/bytecode/policy_audit.rs | 42 ++-- jit/src/compiler/baseline.rs | 209 +++++++++++++++- jit/src/ir/baseline.rs | 74 +++++- jit/src/ir/inlining.rs | 8 + jit/src/ir/mod.rs | 2 +- jit/src/ir/types.rs | 67 ++++- jit/tests/abi.rs | 38 ++- jit/tests/differential.rs | 40 +++ jit/tests/fixtures/opcode-cases.json | 231 +++++++++++++++++ jit/tests/opcodes.rs | 12 +- jit/tests/support/mod.rs | 18 ++ jit/tests/tier1_closures.rs | 204 +++++++++++++++ sys/build.rs | 2 +- sys/build_support/patch.rs | 9 +- sys/patches/0025-tier1-closures.patch | 233 ++++++++++++++++++ sys/patches/README.md | 11 + sys/src/bindings/aarch64-apple-darwin.rs | 109 +++++++- sys/src/bindings/aarch64-pc-windows-msvc.rs | 109 +++++++- sys/src/bindings/aarch64-unknown-linux-gnu.rs | 109 +++++++- .../bindings/aarch64-unknown-linux-musl.rs | 109 +++++++- sys/src/bindings/wasm32-wasip1.rs | 109 +++++++- sys/src/bindings/x86_64-apple-darwin.rs | 109 +++++++- sys/src/bindings/x86_64-pc-windows-gnu.rs | 109 +++++++- sys/src/bindings/x86_64-pc-windows-msvc.rs | 109 +++++++- sys/src/bindings/x86_64-unknown-linux-gnu.rs | 109 +++++++- sys/src/bindings/x86_64-unknown-linux-musl.rs | 109 +++++++- sys/tests/jit_patch.rs | 38 ++- 30 files changed, 2268 insertions(+), 93 deletions(-) create mode 100644 jit/tests/tier1_closures.rs create mode 100644 sys/patches/0025-tier1-closures.patch diff --git a/jit/Cargo.toml b/jit/Cargo.toml index e2b37836d..0f887968f 100644 --- a/jit/Cargo.toml +++ b/jit/Cargo.toml @@ -179,6 +179,10 @@ required-features = ["compiler", "test-support"] name = "tier1_globals" required-features = ["compiler", "test-support"] +[[test]] +name = "tier1_closures" +required-features = ["compiler", "test-support"] + [[test]] name = "tier1_constructors" required-features = ["compiler", "test-support"] diff --git a/jit/src/abi.rs b/jit/src/abi.rs index b39c65a23..3114f5910 100644 --- a/jit/src/abi.rs +++ b/jit/src/abi.rs @@ -5,7 +5,7 @@ use core::{fmt, mem}; use rquickjs_core::qjs; pub const ABI_MAJOR: u16 = 1; -pub const ABI_MINOR: u16 = 24; +pub const ABI_MINOR: u16 = 25; pub const SOURCE_REVISION: u64 = 0xfd0a_0210_b7be_0095; pub const OPCODE_FINGERPRINT: u64 = qjs::QJSJIT_GENERATED_OPCODE_FINGERPRINT; @@ -698,6 +698,26 @@ fn runtime_api_layout_fingerprint() -> u64 { mem::offset_of!(qjs::JSJitRuntimeAPI, unary_arith_slow), mem::size_of::(), ), + ( + mem::offset_of!(qjs::JSJitRuntimeAPI, fclosure), + mem::size_of::(), + ), + ( + mem::offset_of!(qjs::JSJitRuntimeAPI, get_var_ref), + mem::size_of::(), + ), + ( + mem::offset_of!(qjs::JSJitRuntimeAPI, put_var_ref), + mem::size_of::(), + ), + ( + mem::offset_of!(qjs::JSJitRuntimeAPI, close_loc), + mem::size_of::(), + ), + ( + mem::offset_of!(qjs::JSJitRuntimeAPI, set_name), + mem::size_of::(), + ), ] { hash = layout_field(hash, offset, size); } @@ -1073,7 +1093,7 @@ mod inline_api_tests { fn inline_recovery_requires_the_matching_main_abi_minor() { let native = AbiInfo::query_linked().unwrap(); assert_eq!(ABI_MINOR, native.minor()); - assert_eq!(native.minor(), 24); + assert_eq!(native.minor(), 25); assert!(unsafe { qjs::JS_JitGetInlineAPI(0) }.is_null()); } diff --git a/jit/src/bytecode/policy.rs b/jit/src/bytecode/policy.rs index 08bd6b6b9..31c8c3d96 100644 --- a/jit/src/bytecode/policy.rs +++ b/jit/src/bytecode/policy.rs @@ -32,6 +32,11 @@ pub enum HelperId { NewObject, CallConstructor, Regexp, + FClosure, + GetVarRef, + PutVarRef, + CloseLocal, + SetName, } #[derive(Clone, Copy, Debug, Eq, PartialEq)] diff --git a/jit/src/bytecode/policy_audit.rs b/jit/src/bytecode/policy_audit.rs index 05612e6af..290bdbff5 100644 --- a/jit/src/bytecode/policy_audit.rs +++ b/jit/src/bytecode/policy_audit.rs @@ -4,7 +4,7 @@ pub(super) const AUDITED_POLICIES: [AuditedOpcodePolicy; 252] = [ AuditedOpcodePolicy { id: 0, name: "invalid", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 1, name: "push_i32", policy: Tier1Policy::Native }, AuditedOpcodePolicy { id: 2, name: "push_const", policy: Tier1Policy::Helper(HelperId::ResolveConst) }, - AuditedOpcodePolicy { id: 3, name: "fclosure", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 3, name: "fclosure", policy: Tier1Policy::Helper(HelperId::FClosure) }, AuditedOpcodePolicy { id: 4, name: "push_atom_value", policy: Tier1Policy::Helper(HelperId::AtomValue) }, AuditedOpcodePolicy { id: 5, name: "private_symbol", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 6, name: "undefined", policy: Tier1Policy::Native }, @@ -77,7 +77,7 @@ pub(super) const AUDITED_POLICIES: [AuditedOpcodePolicy; 252] = [ AuditedOpcodePolicy { id: 73, name: "get_super_value", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 74, name: "put_super_value", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 75, name: "define_field", policy: Tier1Policy::Helper(HelperId::SetProperty) }, - AuditedOpcodePolicy { id: 76, name: "set_name", policy: Tier1Policy::Reject(FallbackReason::ExtendedFrame) }, + AuditedOpcodePolicy { id: 76, name: "set_name", policy: Tier1Policy::Helper(HelperId::SetName) }, AuditedOpcodePolicy { id: 77, name: "set_name_computed", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 78, name: "set_proto", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 79, name: "set_home_object", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, @@ -94,17 +94,17 @@ pub(super) const AUDITED_POLICIES: [AuditedOpcodePolicy; 252] = [ AuditedOpcodePolicy { id: 90, name: "get_arg", policy: Tier1Policy::Helper(HelperId::Dup) }, AuditedOpcodePolicy { id: 91, name: "put_arg", policy: Tier1Policy::Helper(HelperId::Free) }, AuditedOpcodePolicy { id: 92, name: "set_arg", policy: Tier1Policy::Helper(HelperId::Free) }, - AuditedOpcodePolicy { id: 93, name: "get_var_ref", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, - AuditedOpcodePolicy { id: 94, name: "put_var_ref", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, - AuditedOpcodePolicy { id: 95, name: "set_var_ref", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, + AuditedOpcodePolicy { id: 93, name: "get_var_ref", policy: Tier1Policy::Helper(HelperId::GetVarRef) }, + AuditedOpcodePolicy { id: 94, name: "put_var_ref", policy: Tier1Policy::Helper(HelperId::PutVarRef) }, + AuditedOpcodePolicy { id: 95, name: "set_var_ref", policy: Tier1Policy::Helper(HelperId::PutVarRef) }, AuditedOpcodePolicy { id: 96, name: "set_loc_uninitialized", policy: Tier1Policy::Helper(HelperId::Free) }, AuditedOpcodePolicy { id: 97, name: "get_loc_check", policy: Tier1Policy::Helper(HelperId::Dup) }, AuditedOpcodePolicy { id: 98, name: "put_loc_check", policy: Tier1Policy::Helper(HelperId::Free) }, AuditedOpcodePolicy { id: 99, name: "put_loc_check_init", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, - AuditedOpcodePolicy { id: 100, name: "get_var_ref_check", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, - AuditedOpcodePolicy { id: 101, name: "put_var_ref_check", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, + AuditedOpcodePolicy { id: 100, name: "get_var_ref_check", policy: Tier1Policy::Helper(HelperId::GetVarRef) }, + AuditedOpcodePolicy { id: 101, name: "put_var_ref_check", policy: Tier1Policy::Helper(HelperId::PutVarRef) }, AuditedOpcodePolicy { id: 102, name: "put_var_ref_check_init", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, - AuditedOpcodePolicy { id: 103, name: "close_loc", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 103, name: "close_loc", policy: Tier1Policy::Helper(HelperId::CloseLocal) }, AuditedOpcodePolicy { id: 104, name: "if_false", policy: Tier1Policy::Helper(HelperId::ToBool) }, AuditedOpcodePolicy { id: 105, name: "if_true", policy: Tier1Policy::Helper(HelperId::ToBool) }, AuditedOpcodePolicy { id: 106, name: "goto", policy: Tier1Policy::Native }, @@ -198,7 +198,7 @@ pub(super) const AUDITED_POLICIES: [AuditedOpcodePolicy; 252] = [ AuditedOpcodePolicy { id: 194, name: "push_i8", policy: Tier1Policy::Native }, AuditedOpcodePolicy { id: 195, name: "push_i16", policy: Tier1Policy::Native }, AuditedOpcodePolicy { id: 196, name: "push_const8", policy: Tier1Policy::Helper(HelperId::ResolveConst) }, - AuditedOpcodePolicy { id: 197, name: "fclosure8", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 197, name: "fclosure8", policy: Tier1Policy::Helper(HelperId::FClosure) }, AuditedOpcodePolicy { id: 198, name: "push_empty_string", policy: Tier1Policy::Helper(HelperId::AtomValue) }, AuditedOpcodePolicy { id: 199, name: "get_loc8", policy: Tier1Policy::Helper(HelperId::Dup) }, AuditedOpcodePolicy { id: 200, name: "put_loc8", policy: Tier1Policy::Helper(HelperId::Free) }, @@ -228,18 +228,18 @@ pub(super) const AUDITED_POLICIES: [AuditedOpcodePolicy; 252] = [ AuditedOpcodePolicy { id: 224, name: "set_arg1", policy: Tier1Policy::Helper(HelperId::Free) }, AuditedOpcodePolicy { id: 225, name: "set_arg2", policy: Tier1Policy::Helper(HelperId::Free) }, AuditedOpcodePolicy { id: 226, name: "set_arg3", policy: Tier1Policy::Helper(HelperId::Free) }, - AuditedOpcodePolicy { id: 227, name: "get_var_ref0", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, - AuditedOpcodePolicy { id: 228, name: "get_var_ref1", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, - AuditedOpcodePolicy { id: 229, name: "get_var_ref2", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, - AuditedOpcodePolicy { id: 230, name: "get_var_ref3", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, - AuditedOpcodePolicy { id: 231, name: "put_var_ref0", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, - AuditedOpcodePolicy { id: 232, name: "put_var_ref1", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, - AuditedOpcodePolicy { id: 233, name: "put_var_ref2", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, - AuditedOpcodePolicy { id: 234, name: "put_var_ref3", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, - AuditedOpcodePolicy { id: 235, name: "set_var_ref0", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, - AuditedOpcodePolicy { id: 236, name: "set_var_ref1", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, - AuditedOpcodePolicy { id: 237, name: "set_var_ref2", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, - AuditedOpcodePolicy { id: 238, name: "set_var_ref3", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, + AuditedOpcodePolicy { id: 227, name: "get_var_ref0", policy: Tier1Policy::Helper(HelperId::GetVarRef) }, + AuditedOpcodePolicy { id: 228, name: "get_var_ref1", policy: Tier1Policy::Helper(HelperId::GetVarRef) }, + AuditedOpcodePolicy { id: 229, name: "get_var_ref2", policy: Tier1Policy::Helper(HelperId::GetVarRef) }, + AuditedOpcodePolicy { id: 230, name: "get_var_ref3", policy: Tier1Policy::Helper(HelperId::GetVarRef) }, + AuditedOpcodePolicy { id: 231, name: "put_var_ref0", policy: Tier1Policy::Helper(HelperId::PutVarRef) }, + AuditedOpcodePolicy { id: 232, name: "put_var_ref1", policy: Tier1Policy::Helper(HelperId::PutVarRef) }, + AuditedOpcodePolicy { id: 233, name: "put_var_ref2", policy: Tier1Policy::Helper(HelperId::PutVarRef) }, + AuditedOpcodePolicy { id: 234, name: "put_var_ref3", policy: Tier1Policy::Helper(HelperId::PutVarRef) }, + AuditedOpcodePolicy { id: 235, name: "set_var_ref0", policy: Tier1Policy::Helper(HelperId::PutVarRef) }, + AuditedOpcodePolicy { id: 236, name: "set_var_ref1", policy: Tier1Policy::Helper(HelperId::PutVarRef) }, + AuditedOpcodePolicy { id: 237, name: "set_var_ref2", policy: Tier1Policy::Helper(HelperId::PutVarRef) }, + AuditedOpcodePolicy { id: 238, name: "set_var_ref3", policy: Tier1Policy::Helper(HelperId::PutVarRef) }, AuditedOpcodePolicy { id: 239, name: "get_length", policy: Tier1Policy::Helper(HelperId::GetProperty) }, AuditedOpcodePolicy { id: 240, name: "if_false8", policy: Tier1Policy::Helper(HelperId::ToBool) }, AuditedOpcodePolicy { id: 241, name: "if_true8", policy: Tier1Policy::Helper(HelperId::ToBool) }, diff --git a/jit/src/compiler/baseline.rs b/jit/src/compiler/baseline.rs index 00e7cfb64..67cc9b9a4 100644 --- a/jit/src/compiler/baseline.rs +++ b/jit/src/compiler/baseline.rs @@ -33,7 +33,7 @@ use crate::{ }, ir::{ BaselineIr, BinaryOp, FrameSlot, FrameStateId, FrameStateKind, IrOp, PollKind, StackOp, - TaggedValue, UnaryOp, MAX_HELPER_SCRATCH_SLOTS, + TaggedValue, UnaryOp, VarRefMode, MAX_HELPER_SCRATCH_SLOTS, }, platform::{CodeAllocator, CodeMemoryError, ExecutableCode}, runtime::CompileRequest, @@ -91,9 +91,29 @@ struct HelperLowering<'a> { signatures: &'a [cranelift_codegen::ir::SigRef], pointer_type: cranelift_codegen::ir::Type, layout: FrameLayout, + /// The function creates closures (`fclosure`), so attached var refs may + /// alias its argument and local slots. Any helper that can run JavaScript + /// may then write those slots through the closure, so the native copies + /// are reloaded from the interpreter frame after every helper returns. + reload_captured_slots: bool, } impl HelperLowering<'_> { + /// Refresh the native argument/local copies from the interpreter frame. + /// Every helper call first publishes all of them (`materialize_frame`), + /// so right after it returns the frame is the authoritative owner. + fn reload_captured_slots(&self, builder: &mut FunctionBuilder<'_>) { + if !self.reload_captured_slots { + return; + } + for (index, pair) in self.arguments.iter().copied().enumerate() { + reload_pair(builder, pair, self.arg_buf, index, self.layout); + } + for (index, pair) in self.locals.iter().copied().enumerate() { + reload_pair(builder, pair, self.var_buf, index, self.layout); + } + } + fn invoke( &self, builder: &mut FunctionBuilder<'_>, @@ -123,7 +143,9 @@ impl HelperLowering<'_> { arguments, self.pointer_type, self.layout, - ) + )?; + self.reload_captured_slots(builder); + Ok(()) } fn set_depth( @@ -2041,6 +2063,30 @@ fn analyze_entry_domains(ir: &BaselineIr) -> Result { frame.stack.push(AbstractValue::unknown()); } + IrOp::FClosure(_) => { + frame.stack.push(AbstractValue::known(KnownKind::Other)); + } + // Closure variables may hold any value, including the TDZ + // sentinel for the unchecked form; uses guard at run time. + IrOp::GetVarRef { .. } => { + frame.stack.push(AbstractValue::unknown()); + } + IrOp::PutVarRef { keep, .. } => { + if *keep { + frame.stack.last().ok_or(CompileFailure::InvalidArtifact)?; + } else { + frame.stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + } + } + IrOp::SetName(_) => { + frame.stack.last().ok_or(CompileFailure::InvalidArtifact)?; + } + IrOp::CloseLocal(index) => { + frame + .locals + .get(usize::from(*index)) + .ok_or(CompileFailure::InvalidArtifact)?; + } IrOp::NewObject => { frame.stack.push(AbstractValue::known(KnownKind::Other)); } @@ -2586,6 +2632,7 @@ fn lower_function( signatures: &helper_signatures, pointer_type, layout, + reload_captured_slots: creates_closures(ir), }; macro_rules! invoke_helper { @@ -2674,6 +2721,7 @@ fn lower_function( pointer_type, layout, ); + helper_lowering.reload_captured_slots(builder); if let Some(continuation) = loop_continuation { builder.ins().jump(continuation, &[]); builder.seal_block(continuation); @@ -2871,6 +2919,121 @@ fn lower_function( depth = pattern_index + 1; set_visible_stack_depth(builder, frame, stack_base, depth, layout)?; } + IrOp::FClosure(index) => { + let state = helper_states + .next() + .ok_or(CompileFailure::InvalidArtifact)?; + let output = flat_stack_slot(ir, depth)?; + invoke_helper!( + qjs::JSJitHelperId_JS_JIT_HELPER_FCLOSURE, + state, + depth, + &[output, index] + ); + reload_pair(builder, stack[depth], stack_base, depth, layout); + depth += 1; + set_visible_stack_depth(builder, frame, stack_base, depth, layout)?; + } + IrOp::GetVarRef { index, checked } => { + let state = helper_states + .next() + .ok_or(CompileFailure::InvalidArtifact)?; + let output = flat_stack_slot(ir, depth)?; + let mode = if checked { + qjs::JSJitVarRefMode_JS_JIT_VAR_REF_CHECK + } else { + qjs::JSJitVarRefMode_JS_JIT_VAR_REF_PLAIN + }; + invoke_helper!( + qjs::JSJitHelperId_JS_JIT_HELPER_GET_VAR_REF, + state, + depth, + &[output, u32::from(index), mode] + ); + reload_pair(builder, stack[depth], stack_base, depth, layout); + depth += 1; + set_visible_stack_depth(builder, frame, stack_base, depth, layout)?; + } + IrOp::PutVarRef { index, mode, keep } => { + let top = depth + .checked_sub(1) + .ok_or(CompileFailure::InvalidArtifact)?; + let mode = match mode { + VarRefMode::Plain => qjs::JSJitVarRefMode_JS_JIT_VAR_REF_PLAIN, + VarRefMode::Check => qjs::JSJitVarRefMode_JS_JIT_VAR_REF_CHECK, + VarRefMode::CheckInit => qjs::JSJitVarRefMode_JS_JIT_VAR_REF_CHECK_INIT, + }; + let value_index = if keep { + // `set_var_ref*` keeps the value: store an owned + // duplicate from the scratch slot above the stack. + let dup_state = helper_states + .next() + .ok_or(CompileFailure::InvalidArtifact)?; + let source = use_pair(builder, stack[top]); + lower_dup_if_refcounted( + builder, + &helper_lowering, + dup_state, + depth, + depth, + source, + flat_stack_slot(ir, top)?, + false, + )?; + depth + } else { + top + }; + let state = helper_states + .next() + .ok_or(CompileFailure::InvalidArtifact)?; + let value = flat_stack_slot(ir, value_index)?; + // The helper consumes the value slot on success and on a + // TDZ exception. The exception exit publishes only the + // bytecode-visible stack below the consumed operand. + helper_lowering.invoke( + builder, + qjs::JSJitHelperId_JS_JIT_HELPER_PUT_VAR_REF, + state, + value_index + 1, + value_index, + &[value, u32::from(index), mode], + )?; + clear_pair(builder, stack[value_index], stack_base, value_index, layout)?; + if !keep { + depth = top; + } + set_visible_stack_depth(builder, frame, stack_base, depth, layout)?; + } + IrOp::SetName(atom) => { + let state = helper_states + .next() + .ok_or(CompileFailure::InvalidArtifact)?; + let object = flat_stack_slot( + ir, + depth + .checked_sub(1) + .ok_or(CompileFailure::InvalidArtifact)?, + )?; + // The named object stays borrowed in its stack slot. + invoke_helper!( + qjs::JSJitHelperId_JS_JIT_HELPER_SET_NAME, + state, + depth, + &[object, atom] + ); + } + IrOp::CloseLocal(index) => { + let state = helper_states + .next() + .ok_or(CompileFailure::InvalidArtifact)?; + invoke_helper!( + qjs::JSJitHelperId_JS_JIT_HELPER_CLOSE_LOC, + state, + depth, + &[u32::from(index)] + ); + } IrOp::GetArgument(index) => { let state = helper_states .next() @@ -3626,6 +3789,15 @@ fn lower_function( Ok(()) } +/// Functions that instantiate closures may share argument/local storage with +/// attached var refs (`JSVarRef.pvalue` points into the interpreter frame). +pub(crate) fn creates_closures(ir: &BaselineIr) -> bool { + ir.blocks + .iter() + .flat_map(|block| &block.instructions) + .any(|instruction| matches!(instruction.op, IrOp::FClosure(_))) +} + fn next_helper_state( states: &mut impl Iterator, ) -> Result { @@ -6496,6 +6668,11 @@ mod tests { IrOp::Call { .. } => "call", IrOp::CallConstructor(_) => "call_constructor", IrOp::Regexp => "regexp", + IrOp::FClosure(_) => "fclosure", + IrOp::GetVarRef { .. } => "get_var_ref", + IrOp::PutVarRef { .. } => "put_var_ref", + IrOp::CloseLocal(_) => "close_local", + IrOp::SetName(_) => "set_name", IrOp::GetArgument(_) => "get_argument", IrOp::GetLocal(_) => "get_local", IrOp::GetLocalChecked(_) => "get_local_checked", @@ -6565,6 +6742,32 @@ mod tests { IrOp::CallConstructor(0), ), linear_ir(vec![numeric_push(), numeric_push()], IrOp::Regexp), + linear_ir(Vec::new(), IrOp::FClosure(0)), + linear_ir( + Vec::new(), + IrOp::GetVarRef { + index: 0, + checked: true, + }, + ), + linear_ir( + vec![numeric_push()], + IrOp::PutVarRef { + index: 0, + mode: VarRefMode::Check, + keep: false, + }, + ), + linear_ir( + vec![numeric_push()], + IrOp::PutVarRef { + index: 0, + mode: VarRefMode::Plain, + keep: true, + }, + ), + linear_ir(Vec::new(), IrOp::CloseLocal(0)), + linear_ir(vec![numeric_push()], IrOp::SetName(1)), linear_ir(Vec::new(), IrOp::GetArgument(0)), linear_ir(Vec::new(), IrOp::GetLocal(0)), linear_ir(Vec::new(), IrOp::GetLocalChecked(0)), @@ -6739,7 +6942,7 @@ mod tests { } assert_eq!( seen.len(), - 33, + 38, "every IrOp variant is represented: {seen:?}" ); } diff --git a/jit/src/ir/baseline.rs b/jit/src/ir/baseline.rs index 68e7aa362..181ab78d6 100644 --- a/jit/src/ir/baseline.rs +++ b/jit/src/ir/baseline.rs @@ -11,7 +11,7 @@ use crate::{ use super::{ BinaryOp, FrameSlot, FrameState, FrameStateId, FrameStateKind, FrameStateTable, IrOp, PollKind, - StackOp, TaggedValue, UnaryOp, + StackOp, TaggedValue, UnaryOp, VarRefMode, }; const POLL_INTERVAL: usize = 1_024; @@ -374,6 +374,31 @@ impl BaselineIr { }); } + // Tier 1 proves `put_loc_check_init` statically from the entry + // analysis. Once this frame creates closures, a callee can initialize + // a captured lexical slot (for example a derived constructor's `this` + // through `put_var_ref_check_init`), which that proof cannot observe. + let ops = || { + blocks + .iter() + .flat_map(|block: &IrBlock| &block.instructions) + }; + if ops().any(|instruction| matches!(instruction.op, IrOp::FClosure(_))) + && ops().any(|instruction| { + matches!( + instruction.op, + IrOp::PutLocalChecked { + initialize: true, + .. + } + ) + }) + { + return Err(CompileFailure::Tier1Rejected( + crate::bytecode::FallbackReason::ClosureFrame, + )); + } + Ok(Self { blocks, frame_states: states, @@ -439,6 +464,10 @@ fn operation_may_exit(operation: &IrOp) -> bool { fn operation_helper_call_count(operation: &IrOp) -> usize { match operation { IrOp::ResolveConstant(_) | IrOp::ResolveAtom(_) | IrOp::GetGlobal(_) | IrOp::NewObject => 1, + IrOp::FClosure(_) | IrOp::GetVarRef { .. } | IrOp::CloseLocal(_) | IrOp::SetName(_) => 1, + // `set_var_ref*` first duplicates the kept value into the operand + // slot above the stack, then the store consumes that duplicate. + IrOp::PutVarRef { keep, .. } => 1 + usize::from(*keep), IrOp::NewArrayFrom(count) => 1 + usize::from(*count), IrOp::GetProperty(_) | IrOp::SetProperty(_) => 2, IrOp::DefineProperty(_) => 1, @@ -510,6 +539,7 @@ fn helper_stack_depth( let extra = match operation { IrOp::GetProperty(_) | IrOp::Call { .. } | IrOp::CallConstructor(_) => 2, IrOp::GetPropertyKeep(_) => 1, + IrOp::PutVarRef { keep: true, .. } => 1, IrOp::NewArrayFrom(count) if *count != 0 => 2, IrOp::DefineElement => 2, IrOp::NewArrayFrom(_) => 0, @@ -627,6 +657,14 @@ fn constant_operand(instruction: &Instruction) -> Option { } } +fn closure_operand(instruction: &Instruction) -> Option { + match instruction.opcode().format() { + OperandFormat::Closure => Some(instruction.operand_u16(1)), + OperandFormat::NoneClosure => short_index(instruction), + _ => None, + } +} + fn atom_operand(instruction: &Instruction) -> Option { matches!(instruction.opcode().format(), OperandFormat::Atom).then(|| instruction.operand_u32(1)) } @@ -698,6 +736,40 @@ fn translate_instruction(instruction: &Instruction) -> Result IrOp::CallConstructor(instruction.operand_u16(1)), "regexp" => IrOp::Regexp, + "fclosure" | "fclosure8" => { + IrOp::FClosure(constant_operand(instruction).ok_or(CompileFailure::InvalidArtifact)?) + } + "get_var_ref" | "get_var_ref0" | "get_var_ref1" | "get_var_ref2" | "get_var_ref3" + | "get_var_ref_check" => IrOp::GetVarRef { + index: closure_operand(instruction).ok_or(CompileFailure::InvalidArtifact)?, + checked: name == "get_var_ref_check", + }, + "put_var_ref" + | "put_var_ref0" + | "put_var_ref1" + | "put_var_ref2" + | "put_var_ref3" + | "set_var_ref" + | "set_var_ref0" + | "set_var_ref1" + | "set_var_ref2" + | "set_var_ref3" + | "put_var_ref_check" + | "put_var_ref_check_init" => IrOp::PutVarRef { + index: closure_operand(instruction).ok_or(CompileFailure::InvalidArtifact)?, + mode: match name { + "put_var_ref_check" => VarRefMode::Check, + "put_var_ref_check_init" => VarRefMode::CheckInit, + _ => VarRefMode::Plain, + }, + keep: name.starts_with("set_var_ref"), + }, + "set_name" => { + IrOp::SetName(atom_operand(instruction).ok_or(CompileFailure::InvalidArtifact)?) + } + "close_loc" => { + IrOp::CloseLocal(indexed_operand(instruction).ok_or(CompileFailure::InvalidArtifact)?) + } "get_arg" | "get_arg0" | "get_arg1" | "get_arg2" | "get_arg3" => { IrOp::GetArgument(indexed_operand(instruction).ok_or(CompileFailure::InvalidArtifact)?) } diff --git a/jit/src/ir/inlining.rs b/jit/src/ir/inlining.rs index dfa1d947e..910f05c10 100644 --- a/jit/src/ir/inlining.rs +++ b/jit/src/ir/inlining.rs @@ -178,6 +178,14 @@ impl FrameInlineCallee { || instructions .iter() .any(|instruction| instruction.opcode().name().starts_with("tail_call")) + // A callee that instantiates closures may alias its shadow-frame + // locals through attached var refs; keep it out of frame inlining. + || instructions.iter().any(|instruction| { + matches!( + instruction.opcode().name(), + "fclosure" | "fclosure8" | "close_loc" + ) + }) { return None; } diff --git a/jit/src/ir/mod.rs b/jit/src/ir/mod.rs index 9be6ab40b..8d67bba36 100644 --- a/jit/src/ir/mod.rs +++ b/jit/src/ir/mod.rs @@ -42,6 +42,6 @@ pub use scalar::{ ScalarGraph, ScalarInlineRegion, ScalarInlineStep, ScalarNumericMode, ScalarPhiInput, ScalarValue, ScalarValueId, }; -pub use types::{BinaryOp, IrOp, PollKind, StackOp, TaggedValue, UnaryOp}; +pub use types::{BinaryOp, IrOp, PollKind, StackOp, TaggedValue, UnaryOp, VarRefMode}; pub(crate) use scalar::stack_permutation; diff --git a/jit/src/ir/types.rs b/jit/src/ir/types.rs index 1ab34caf1..6964fc185 100644 --- a/jit/src/ir/types.rs +++ b/jit/src/ir/types.rs @@ -73,10 +73,24 @@ pub enum BinaryOp { StrictNotEqual, } +/// TDZ checking mode for closure-variable stores. Values match the native +/// `JSJitVarRefMode` ABI. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum VarRefMode { + Plain, + Check, + CheckInit, +} + #[derive(Clone, Debug, Eq, PartialEq)] pub enum IrOp { - Poll { state: FrameStateId, kind: PollKind }, - OsrLabel { state: FrameStateId }, + Poll { + state: FrameStateId, + kind: PollKind, + }, + OsrLabel { + state: FrameStateId, + }, Nop, Push(TaggedValue), ResolveConstant(u32), @@ -92,26 +106,63 @@ pub enum IrOp { SetElement, DefineElement, ToPropertyKey, - Call { argc: u16, has_this: bool }, + Call { + argc: u16, + has_this: bool, + }, CallConstructor(u16), Regexp, + /// `fclosure`/`fclosure8`: instantiate the constant-pool bytecode as a + /// closure over the current frame's arguments, locals and var refs. + FClosure(u32), + /// `get_var_ref*`: load a closure variable, optionally with the TDZ check. + GetVarRef { + index: u16, + checked: bool, + }, + /// `put_var_ref*`/`set_var_ref*`: store a closure variable. `keep` leaves + /// the stored value on the operand stack (`set_var_ref*`). + PutVarRef { + index: u16, + mode: VarRefMode, + keep: bool, + }, + /// `close_loc`: detach the var ref capturing a lexical local. + CloseLocal(u16), + /// `set_name`: define the anonymous function/class name of the stack top. + SetName(u32), GetArgument(u16), GetLocal(u16), GetLocalChecked(u16), GetLocalPair, - PutArgument { index: u16, keep: bool }, - PutLocal { index: u16, keep: bool }, - PutLocalChecked { index: u16, initialize: bool }, + PutArgument { + index: u16, + keep: bool, + }, + PutLocal { + index: u16, + keep: bool, + }, + PutLocalChecked { + index: u16, + initialize: bool, + }, SetLocalUninitialized(u16), Drop, Stack(StackOp), Unary(UnaryOp), PostUnary(UnaryOp), - LocalUnary { index: u16, op: UnaryOp }, + LocalUnary { + index: u16, + op: UnaryOp, + }, AddLocal(u16), Binary(BinaryOp), Jump(u32), - Branch { target: u32, when_true: bool }, + Branch { + target: u32, + when_true: bool, + }, Return, ReturnUndefined, } diff --git a/jit/tests/abi.rs b/jit/tests/abi.rs index bb3580e33..6ddbecfe5 100644 --- a/jit/tests/abi.rs +++ b/jit/tests/abi.rs @@ -386,7 +386,7 @@ fn interrupt_runtime_api_is_a_versioned_exec_frame_tail_extension() { use rquickjs_core::qjs; assert_eq!(qjs::QJSJIT_RUNTIME_API_MAJOR, 1); - assert_eq!(qjs::QJSJIT_RUNTIME_API_MINOR, 9); + assert_eq!(qjs::QJSJIT_RUNTIME_API_MINOR, 10); assert_eq!( std::mem::offset_of!(qjs::JSJitExecFrame, runtime_api), std::mem::offset_of!(qjs::JSJitExecFrame, entry) @@ -419,7 +419,12 @@ fn interrupt_runtime_api_is_a_versioned_exec_frame_tail_extension() { std::mem::offset_of!(qjs::JSJitRuntimeAPI, unary_arith_slow), 192 ); - assert_eq!(std::mem::size_of::(), 200); + assert_eq!(std::mem::offset_of!(qjs::JSJitRuntimeAPI, fclosure), 200); + assert_eq!(std::mem::offset_of!(qjs::JSJitRuntimeAPI, get_var_ref), 208); + assert_eq!(std::mem::offset_of!(qjs::JSJitRuntimeAPI, put_var_ref), 216); + assert_eq!(std::mem::offset_of!(qjs::JSJitRuntimeAPI, close_loc), 224); + assert_eq!(std::mem::offset_of!(qjs::JSJitRuntimeAPI, set_name), 232); + assert_eq!(std::mem::size_of::(), 240); assert_eq!(qjs::JS_JIT_HELPER_GUARD_MISS, 1); } @@ -530,6 +535,31 @@ fn helper_abi_is_one_canonical_versioned_table_in_c_bindgen_and_rust() { 2, qjs::JSJitHelperOwnership_JS_JIT_HELPER_OWNED as u8, ), + ( + "FCLOSURE", + 1, + qjs::JSJitHelperOwnership_JS_JIT_HELPER_OWNED as u8, + ), + ( + "GET_VAR_REF", + 1, + qjs::JSJitHelperOwnership_JS_JIT_HELPER_OWNED as u8, + ), + ( + "PUT_VAR_REF", + 1, + qjs::JSJitHelperOwnership_JS_JIT_HELPER_NONE as u8, + ), + ( + "CLOSE_LOC", + 0, + qjs::JSJitHelperOwnership_JS_JIT_HELPER_NONE as u8, + ), + ( + "SET_NAME", + 1, + qjs::JSJitHelperOwnership_JS_JIT_HELPER_NONE as u8, + ), ]; let mut count = 0_u32; @@ -574,9 +604,9 @@ fn helper_abi_is_one_canonical_versioned_table_in_c_bindgen_and_rust() { fn helper_abi_fields_are_append_only_tails() { use rquickjs_core::qjs; - assert_eq!(qjs::QJSJIT_ABI_MINOR, 24); + assert_eq!(qjs::QJSJIT_ABI_MINOR, 25); assert_eq!(qjs::QJSJIT_RUNTIME_API_MAJOR, 1); - assert_eq!(qjs::QJSJIT_RUNTIME_API_MINOR, 9); + assert_eq!(qjs::QJSJIT_RUNTIME_API_MINOR, 10); assert_eq!(qjs::QJSJIT_HELPER_ABI_VERSION, 1); assert_eq!( std::mem::offset_of!(qjs::JSJitEntryHandle, stack_map_count), diff --git a/jit/tests/differential.rs b/jit/tests/differential.rs index 07166009d..a81c8c1d8 100644 --- a/jit/tests/differential.rs +++ b/jit/tests/differential.rs @@ -64,6 +64,11 @@ enum ManifestHelper { Regexp, NewArray, NewObject, + FClosure, + GetVarRef, + PutVarRef, + CloseLocal, + SetName, } impl ManifestHelper { @@ -90,6 +95,11 @@ impl ManifestHelper { Self::Regexp => HelperId::Regexp, Self::NewArray => HelperId::NewArray, Self::NewObject => HelperId::NewObject, + Self::FClosure => HelperId::FClosure, + Self::GetVarRef => HelperId::GetVarRef, + Self::PutVarRef => HelperId::PutVarRef, + Self::CloseLocal => HelperId::CloseLocal, + Self::SetName => HelperId::SetName, } } } @@ -527,6 +537,36 @@ fn every_advertised_helper_family_has_a_real_native_execution_case() { "object", HelperId::NewObject, ), + ( + "function f(a){ const g = () => a + 1; return g() }", + "f(41)", + "fclosure8", + HelperId::FClosure, + ), + ( + "function f(a){ const g = () => a + 1; return g.name + g() }", + "f(41)", + "set_name", + HelperId::SetName, + ), + ( + "var f = (function(){ var c = 40; return function f(a){ return c + a } })()", + "f(2)", + "get_var_ref0", + HelperId::GetVarRef, + ), + ( + "var f = (function(){ var c = 40; return function f(a){ c = a; return 1 } })()", + "f(2)", + "put_var_ref0", + HelperId::PutVarRef, + ), + ( + "function f(n){ const fs = []; for (let i = 0; i < n; i++) { fs.push(() => i) } let s = 0; for (let j = 0; j < n; j++) { const g = fs[j]; s += g() } return s }", + "f(5)", + "close_loc", + HelperId::CloseLocal, + ), ]; for (definition, expression, opcode, helper) in cases { differential(definition, expression) diff --git a/jit/tests/fixtures/opcode-cases.json b/jit/tests/fixtures/opcode-cases.json index 3a6818cfd..775232d79 100644 --- a/jit/tests/fixtures/opcode-cases.json +++ b/jit/tests/fixtures/opcode-cases.json @@ -1431,6 +1431,237 @@ "ownership-gc", "coercion-reentrancy" ] + }, + { + "opcode": "fclosure", + "definition": "var src = 'function f(a){ if (a) {'; for (var i = 0; i < 260; i++) src += '()=>' + i + ';'; src += '} return (()=>42)() }'; (0, eval)(src)", + "expression": "f(0)", + "helper": "FClosure", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "fclosure8", + "definition": "function f(a){ const g = () => a + 1; return g() }", + "expression": "f(41)", + "helper": "FClosure", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "set_name", + "definition": "function f(a){ const g = () => a + 1; return g.name + ':' + g() }", + "expression": "f(41)", + "helper": "SetName", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "get_var_ref0", + "definition": "var f = (function(){ var a = 1, b = 2, c = 3, d = 4, e = 5; return function f(x){ return a + b + c + d + e + x } })()", + "expression": "f(10)", + "helper": "GetVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "get_var_ref1", + "definition": "var f = (function(){ var a = 1, b = 2, c = 3, d = 4, e = 5; return function f(x){ return a + b + c + d + e + x } })()", + "expression": "f(11)", + "helper": "GetVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "get_var_ref2", + "definition": "var f = (function(){ var a = 1, b = 2, c = 3, d = 4, e = 5; return function f(x){ return a + b + c + d + e + x } })()", + "expression": "f(12)", + "helper": "GetVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "get_var_ref3", + "definition": "var f = (function(){ var a = 1, b = 2, c = 3, d = 4, e = 5; return function f(x){ return a + b + c + d + e + x } })()", + "expression": "f(13)", + "helper": "GetVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "get_var_ref", + "definition": "var f = (function(){ var a = 1, b = 2, c = 3, d = 4, e = 5; return function f(x){ return a + b + c + d + e + x } })()", + "expression": "f(14)", + "helper": "GetVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "get_var_ref_check", + "definition": "var f = (function(){ let c = 0; return function f(a){ c = c + a; return c } })()", + "expression": "f(1) + f(2)", + "helper": "GetVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "put_var_ref0", + "definition": "var f = (function(){ var a = 1, b = 2, c = 3, d = 4, e = 5; globalThis.peek = function(){ return [a, b, c, d, e].join() }; return function f(x){ a = x; b = x + 1; c = x + 2; d = x + 3; e = x + 4; return x } })()", + "expression": "f(10) + peek()", + "helper": "PutVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "put_var_ref1", + "definition": "var f = (function(){ var a = 1, b = 2, c = 3, d = 4, e = 5; globalThis.peek = function(){ return [a, b, c, d, e].join() }; return function f(x){ a = x; b = x + 1; c = x + 2; d = x + 3; e = x + 4; return x } })()", + "expression": "f(11) + peek()", + "helper": "PutVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "put_var_ref2", + "definition": "var f = (function(){ var a = 1, b = 2, c = 3, d = 4, e = 5; globalThis.peek = function(){ return [a, b, c, d, e].join() }; return function f(x){ a = x; b = x + 1; c = x + 2; d = x + 3; e = x + 4; return x } })()", + "expression": "f(12) + peek()", + "helper": "PutVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "put_var_ref3", + "definition": "var f = (function(){ var a = 1, b = 2, c = 3, d = 4, e = 5; globalThis.peek = function(){ return [a, b, c, d, e].join() }; return function f(x){ a = x; b = x + 1; c = x + 2; d = x + 3; e = x + 4; return x } })()", + "expression": "f(13) + peek()", + "helper": "PutVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "put_var_ref", + "definition": "var f = (function(){ var a = 1, b = 2, c = 3, d = 4, e = 5; globalThis.peek = function(){ return [a, b, c, d, e].join() }; return function f(x){ a = x; b = x + 1; c = x + 2; d = x + 3; e = x + 4; return x } })()", + "expression": "f(14) + peek()", + "helper": "PutVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "set_var_ref0", + "definition": "var f = (function(){ var a = 1, b = 2, c = 3, d = 4, e = 5; globalThis.peek = function(){ return [a, b, c, d, e].join() }; return function f(x){ return [a = x, b = x + 1, c = x + 2, d = x + 3, e = x + 4] } })()", + "expression": "f(10).join() + '|' + peek()", + "helper": "PutVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "set_var_ref1", + "definition": "var f = (function(){ var a = 1, b = 2, c = 3, d = 4, e = 5; globalThis.peek = function(){ return [a, b, c, d, e].join() }; return function f(x){ return [a = x, b = x + 1, c = x + 2, d = x + 3, e = x + 4] } })()", + "expression": "f(11).join() + '|' + peek()", + "helper": "PutVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "set_var_ref2", + "definition": "var f = (function(){ var a = 1, b = 2, c = 3, d = 4, e = 5; globalThis.peek = function(){ return [a, b, c, d, e].join() }; return function f(x){ return [a = x, b = x + 1, c = x + 2, d = x + 3, e = x + 4] } })()", + "expression": "f(12).join() + '|' + peek()", + "helper": "PutVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "set_var_ref3", + "definition": "var f = (function(){ var a = 1, b = 2, c = 3, d = 4, e = 5; globalThis.peek = function(){ return [a, b, c, d, e].join() }; return function f(x){ return [a = x, b = x + 1, c = x + 2, d = x + 3, e = x + 4] } })()", + "expression": "f(13).join() + '|' + peek()", + "helper": "PutVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "set_var_ref", + "definition": "var f = (function(){ var a = 1, b = 2, c = 3, d = 4, e = 5; globalThis.peek = function(){ return [a, b, c, d, e].join() }; return function f(x){ return [a = x, b = x + 1, c = x + 2, d = x + 3, e = x + 4] } })()", + "expression": "f(14).join() + '|' + peek()", + "helper": "PutVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "put_var_ref_check", + "definition": "var f = (function(){ let c = 0; return function f(a){ c = c + a; return c } })()", + "expression": "f(3) + f(4)", + "helper": "PutVarRef", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "close_loc", + "definition": "function f(n){ const fs = []; for (let i = 0; i < n; i++) { fs.push(() => i) } let s = 0; for (let j = 0; j < n; j++) { const g = fs[j]; s += g() } return s }", + "expression": "f(5)", + "helper": "CloseLocal", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] } ] } diff --git a/jit/tests/opcodes.rs b/jit/tests/opcodes.rs index 77d43fadd..e76bf9cd5 100644 --- a/jit/tests/opcodes.rs +++ b/jit/tests/opcodes.rs @@ -137,8 +137,18 @@ fn unsupported_frame_and_exception_families_are_stable_rejects() { tier1_policy(opcode.id()).unwrap() }; + // Derived-constructor `this` initialization through an arrow and + // with/global reference objects still keep the closure frame rejected. assert_eq!( - by_name("get_var_ref"), + by_name("put_var_ref_check_init"), + Tier1Policy::Reject(FallbackReason::ClosureFrame) + ); + assert_eq!( + by_name("make_var_ref"), + Tier1Policy::Reject(FallbackReason::ClosureFrame) + ); + assert_eq!( + by_name("make_var_ref_ref"), Tier1Policy::Reject(FallbackReason::ClosureFrame) ); assert_eq!( diff --git a/jit/tests/support/mod.rs b/jit/tests/support/mod.rs index bc1ada728..861115902 100644 --- a/jit/tests/support/mod.rs +++ b/jit/tests/support/mod.rs @@ -742,6 +742,11 @@ impl DifferentialRun { rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_TO_PROPKEY } HelperId::GetGlobal => rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_GET_GLOBAL, + HelperId::FClosure => rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_FCLOSURE, + HelperId::GetVarRef => rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_GET_VAR_REF, + HelperId::PutVarRef => rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_PUT_VAR_REF, + HelperId::CloseLocal => rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_CLOSE_LOC, + HelperId::SetName => rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_SET_NAME, }; let expected_opcode = self .expected_opcode @@ -1161,6 +1166,14 @@ unsafe extern "C" fn synthetic_compare( 0 } +unsafe extern "C" fn synthetic_map_in_unavailable( + _frame: *mut rquickjs_core::qjs::JSJitExecFrame, + _stack_map_id: u32, + _input: u32, +) -> i32 { + -1 +} + unsafe extern "C" fn synthetic_map_out_in_unavailable( _frame: *mut rquickjs_core::qjs::JSJitExecFrame, _stack_map_id: u32, @@ -1328,6 +1341,11 @@ static SYNTHETIC_RUNTIME_API: rquickjs_core::qjs::JSJitRuntimeAPI = regexp: Some(synthetic_map_out_two_unavailable), binary_arith_slow: Some(synthetic_map_out_two_op_unavailable), unary_arith_slow: Some(synthetic_map_out_in_op_unavailable), + fclosure: Some(synthetic_map_out_in_unavailable), + get_var_ref: Some(synthetic_map_out_in_op_unavailable), + put_var_ref: Some(synthetic_map_out_in_op_unavailable), + close_loc: Some(synthetic_map_in_unavailable), + set_name: Some(synthetic_map_out_in_unavailable), }; /// Result observed after invoking a generated aggregate-return entry point. diff --git a/jit/tests/tier1_closures.rs b/jit/tests/tier1_closures.rs new file mode 100644 index 000000000..39cc4fba5 --- /dev/null +++ b/jit/tests/tier1_closures.rs @@ -0,0 +1,204 @@ +//! Tier 1 closure coverage: `fclosure`, closure-variable access, `close_loc` +//! and `set_name`. Every program runs once in the interpreter and once with +//! the named function `f` forced into published Tier 1 code; results must be +//! identical, including exceptions, TDZ errors and values that a closure +//! writes into the native frame's captured arguments and locals. +#![cfg(all( + target_endian = "little", + any(target_arch = "x86_64", target_arch = "aarch64"), + any(target_os = "linux", target_os = "macos", target_os = "windows"), + not(all(target_os = "windows", target_arch = "aarch64")) +))] + +use rquickjs::{Context, Runtime}; +use rquickjs_jit::bytecode::{FallbackReason, HelperId}; +use rquickjs_jit::test_support::{assert_tier1_rejected, differential}; +use rquickjs_jit::{Jit, JitConfig, JitTierPolicy}; +use std::time::{Duration, Instant}; + +#[test] +fn closure_writes_to_captured_locals_are_reloaded_after_every_call() { + // `inc` writes the captured `c` through its attached var ref while the + // native frame holds a register copy. The following `c++` must observe it. + differential( + "function f(n){ let c = 0; const inc = () => { c += 2 }; for (let i = 0; i < n; i++) { inc(); c++ } return c }", + "f(10)", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("fclosure8") + .expect_helper(HelperId::FClosure) + .assert_same(); +} + +#[test] +fn closure_writes_to_captured_arguments_are_reloaded() { + differential( + "function f(a){ const set = v => { a = v }; set(5); let r = a + 1; set('x'); return r + ':' + a }", + "f(1)", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("fclosure8") + .assert_same(); +} + +#[test] +fn closure_type_change_takes_the_native_numeric_guard_exit() { + // The closure turns the captured Int32 into a string. Subtraction is a + // native-only numeric operation, so the reloaded value must fail its + // tag guard and deoptimize instead of using the stale register copy. + differential( + "function f(n){ let x = n; const g = () => { x = 'a' + x }; g(); return x - 1 }", + "String(f(3))", + ) + .force_baseline() + .expect_executed_opcode("fclosure8") + .expect_deopt() + .assert_same(); +} + +#[test] +fn captured_heap_values_keep_exact_ownership_under_stress_gc() { + differential( + "function f(n){ let o = { v: 0 }; const g = () => { o = { v: o.v + 1, prev: o } }; for (let i = 0; i < n; i++) g(); return o.v + ':' + o.prev.v }", + "f(20)", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("fclosure8") + .assert_same(); +} + +#[test] +fn per_iteration_bindings_are_closed_at_loop_boundaries() { + differential( + "function f(n){ const fs = []; for (let i = 0; i < n; i++) { fs.push(() => i * 10) } let s = ''; for (let j = 0; j < n; j++) { const g = fs[j]; s += g() + ',' } return s }", + "f(6)", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("close_loc") + .expect_helper(HelperId::CloseLocal) + .assert_same(); +} + +#[test] +fn closure_variable_reads_raise_the_interpreter_tdz_error() { + differential( + "var f = (function(){ if (globalThis) return function f(){ return x }; let x = 1 })()", + "(() => { try { return f() } catch (e) { return e.name + ':' + e.message } })()", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("get_var_ref_check") + .expect_helper(HelperId::GetVarRef) + .assert_same(); +} + +#[test] +fn closure_variable_writes_raise_the_interpreter_tdz_error() { + differential( + "var f = (function(){ if (globalThis) return function f(v){ x = v; return 1 }; let x = 1 })()", + "(() => { try { return f({}) } catch (e) { return e.name + ':' + e.message } })()", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("put_var_ref_check") + .expect_helper(HelperId::PutVarRef) + .assert_same(); +} + +#[test] +fn closure_variables_are_shared_between_closures_and_the_owner() { + differential( + "var peek; var f = (function(){ let c = 1; peek = () => c; return function f(a){ c = c * 2 + a; return c + peek() } })()", + "[f(1), f(2), peek()].join()", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("put_var_ref_check") + .assert_same(); +} + +#[test] +fn set_var_ref_keeps_the_stored_heap_value_on_the_stack() { + differential( + "var peek; var f = (function(){ var c = null; peek = () => c; return function f(a){ return (c = { a: a }).a + peek().a } })()", + "f(21)", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("set_var_ref0") + .assert_same(); +} + +#[test] +fn anonymous_closure_names_match_the_interpreter() { + differential( + "function f(){ const g = () => 1; const h = function(){ return 2 }; return g.name + ',' + h.name + ',' + (g() + h()) }", + "f()", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("set_name") + .expect_helper(HelperId::SetName) + .assert_same(); +} + +#[test] +fn generator_and_async_closures_are_created_but_run_in_the_interpreter() { + differential( + "function f(n){ const gen = function* () { yield n; yield n + 1 }; const it = gen(); const a = it.next().value; const b = it.next().value; return a + b }", + "f(20)", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("fclosure8") + .assert_same(); +} + +#[test] +fn derived_constructor_this_initialization_through_arrows_stays_rejected() { + assert_tier1_rejected( + "var f = (function(){ let r; class A { constructor(){ this.v = 1 } } class C extends A { constructor(){ r = () => super(); r() } } new C(); return r })()", + "(() => { try { f(); return 'no' } catch (e) { return e.name } })()", + FallbackReason::ClosureFrame, + ); +} + +#[test] +fn automatic_tiering_runs_closure_creating_loops_natively() { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .tier_policy(JitTierPolicy::BaselineOnly) + .call_threshold(1) + .loop_threshold(1) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| { + ctx.eval::<(), _>( + "globalThis.kernel = function kernel(n){ let c = 0; const add = x => y => (x + y) | 0; for (let i = 0; i < n; i++) { c = add(c)(i & 31) } return c }", + ) + .unwrap() + }); + let deadline = Instant::now() + Duration::from_secs(10); + let mut result = 0; + while Instant::now() < deadline { + result = context.with(|ctx| ctx.eval::("kernel(200)").unwrap()); + jit.poll(); + if jit.metrics().native_entries > 0 { + break; + } + } + let expected = (0..200).fold(0_i32, |c, i| c.wrapping_add(i & 31)); + assert_eq!(result, expected); + assert!(jit.metrics().native_entries > 0, "{:?}", jit.metrics()); + let after = context.with(|ctx| ctx.eval::("kernel(200)").unwrap()); + assert_eq!(after, expected); +} diff --git a/sys/build.rs b/sys/build.rs index 38375afaa..9c5b68819 100644 --- a/sys/build.rs +++ b/sys/build.rs @@ -228,7 +228,7 @@ QJSJIT_HELPER_LIST(QJSJIT_EMIT) flags, }); } - assert_eq!(helpers.len(), 24, "canonical append-only helper count"); + assert_eq!(helpers.len(), 29, "canonical append-only helper count"); let u32_args = |signature: &str| match signature { "FRAME" => 0, diff --git a/sys/build_support/patch.rs b/sys/build_support/patch.rs index 41317e2f2..fa83a4bb8 100644 --- a/sys/build_support/patch.rs +++ b/sys/build_support/patch.rs @@ -1,7 +1,7 @@ use std::{fs, io, path::Path}; const QUICKJS_BASELINE_FNV64: u64 = 0x3302_116a_b0fc_269c; -const EXPECTED_PATCHES: [(&str, u64); 19] = [ +const EXPECTED_PATCHES: [(&str, u64); 20] = [ ("0001-rquickjs-jit.patch", 0x18d7_40cc_6943_30cf), ("0002-runtime-feedback.patch", 0xe4c1_ed52_ee5d_d469), ("0003-element-layout.patch", 0xa12c_ccd2_7420_e880), @@ -21,6 +21,7 @@ const EXPECTED_PATCHES: [(&str, u64); 19] = [ ("0018-inline-frame-recovery.patch", 0xb971d61edccde229), ("0019-array-feedback.patch", 0x11c3cb13a8297dd4), ("0020-typed-array-guard.patch", 0x5d10_38a3_8bc9_9821), + ("0025-tier1-closures.patch", 0x3894_d14c_2378_dfa2), ]; pub(crate) const BASELINE_FILES: [&str; 19] = [ "api-test.c", @@ -60,9 +61,9 @@ const PATCHED_FILE_FINGERPRINTS: [(&str, u64); 21] = [ ("list.h", 0xb337_70f7_b76d_a3d8), ("quickjs-atom.h", 0x30b4_9116_b6a2_aa99), ("quickjs-c-atomics.h", 0x490b_0f29_f631_3fc0), - ("quickjs.c", 0xc535_b5f1_48e7_9b27), - ("quickjs-jit.h", 0x0188_04ad_21d2_95a7), - ("quickjs-jit-helpers.h", 0xa469_e80f_be47_404c), + ("quickjs.c", 0xbad9_ea86_12ab_0678), + ("quickjs-jit.h", 0xfafa_4511_21a6_ffb4), + ("quickjs-jit-helpers.h", 0x3703_05f2_7534_2073), ("quickjs-opcode.h", 0x3d05_cfdf_5cf7_2930), ("quickjs.h", 0x4831_2cde_9c2f_a5ee), ]; diff --git a/sys/patches/0025-tier1-closures.patch b/sys/patches/0025-tier1-closures.patch new file mode 100644 index 000000000..0e23a9479 --- /dev/null +++ b/sys/patches/0025-tier1-closures.patch @@ -0,0 +1,233 @@ +diff --git a/quickjs-jit-helpers.h b/quickjs-jit-helpers.h +--- a/quickjs-jit-helpers.h ++++ b/quickjs-jit-helpers.h +@@ -66,6 +66,17 @@ + JS_JIT_COMPARE_STRICT_NEQ = 7, + } JSJitCompareOp; + ++/* Closure-variable access modes (ABI 1.25). GET_VAR_REF accepts PLAIN and ++ * CHECK; PUT_VAR_REF accepts all three. They mirror OP_get_var_ref*, ++ * OP_put_var_ref*, including the exact TDZ ReferenceError. For PUT_VAR_REF the ++ * MAP_OUT_IN_OP slots are (value, closure index, mode) and the value slot is ++ * consumed on success and on language exception. */ ++typedef enum JSJitVarRefMode { ++ JS_JIT_VAR_REF_PLAIN = 0, ++ JS_JIT_VAR_REF_CHECK = 1, ++ JS_JIT_VAR_REF_CHECK_INIT = 2, ++} JSJitVarRefMode; ++ + /* id, table field, C adapter, signature, value arity, four ownership entries, + * output ownership, behavioral flags */ + #define QJSJIT_HELPER_LIST(X) \ +@@ -129,7 +140,18 @@ + THROWING | ALLOCATING | REENTRANT | FINALIZING) \ + X(UNARY_ARITH_SLOW, unary_arith_slow, JS_JitHelperUnaryArithSlow, \ + MAP_OUT_IN_OP, 2, OWNED, CONSUMED, NONE, NONE, OWNED, \ +- THROWING | ALLOCATING | REENTRANT | FINALIZING) ++ THROWING | ALLOCATING | REENTRANT | FINALIZING) \ ++ X(FCLOSURE, fclosure, JS_JitHelperFClosure, MAP_OUT_INDEX, 1, OWNED, \ ++ NONE, NONE, NONE, OWNED, THROWING | ALLOCATING) \ ++ X(GET_VAR_REF, get_var_ref, JS_JitHelperGetVarRef, MAP_OUT_IN_OP, 1, \ ++ OWNED, NONE, NONE, NONE, OWNED, THROWING | ALLOCATING) \ ++ X(PUT_VAR_REF, put_var_ref, JS_JitHelperPutVarRef, MAP_OUT_IN_OP, 1, \ ++ CONSUMED, NONE, NONE, NONE, NONE, \ ++ THROWING | ALLOCATING | REENTRANT | FINALIZING) \ ++ X(CLOSE_LOC, close_loc, JS_JitHelperCloseLoc, MAP_IN, 0, NONE, NONE, \ ++ NONE, NONE, NONE, 0) \ ++ X(SET_NAME, set_name, JS_JitHelperSetName, MAP_OUT_INDEX, 1, BORROWED, \ ++ NONE, NONE, NONE, NONE, THROWING | ALLOCATING) + + typedef enum JSJitHelperId { + #define QJSJIT_HELPER_ID(id, field, c_name, signature, value_arity, own0, \ +diff --git a/quickjs-jit.h b/quickjs-jit.h +--- a/quickjs-jit.h ++++ b/quickjs-jit.h +@@ -17,7 +17,7 @@ + #endif + + #define QJSJIT_ABI_MAJOR 1u +-#define QJSJIT_ABI_MINOR 24u ++#define QJSJIT_ABI_MINOR 25u + + /* Frame flags are append-only. Stress collection is a test/debug contract: + * every helper performs an allocating cycle-GC point before and after its +@@ -273,7 +273,7 @@ + * prefix and helper ID zero; every later helper is an append-only v1.1 tail. + */ + #define QJSJIT_RUNTIME_API_MAJOR 1u +-#define QJSJIT_RUNTIME_API_MINOR 9u ++#define QJSJIT_RUNTIME_API_MINOR 10u + + #define QJSJIT_RUNTIME_FIELD_FRAME(field) \ + JSJitHelperStatus (*field)(JSJitExecFrame *frame) +diff --git a/quickjs.c b/quickjs.c +--- a/quickjs.c ++++ b/quickjs.c +@@ -19875,6 +19875,167 @@ + return qjsjit_finish_output(frame, out); + } + ++/* Closure helpers (ABI 1.25) mirror OP_fclosure, OP_get_var_ref*, ++ * OP_put_var_ref* and OP_close_loc on the validated current stack frame. ++ * Attached var refs point into sf->var_buf/sf->arg_buf, so the caller must ++ * publish every local/argument before the call and reload them afterwards. */ ++static JSVarRef *qjsjit_closure_var_ref(JSFunctionBytecode *b, ++ JSStackFrame *sf, uint32_t index) ++{ ++ JSVarRef **var_refs; ++ ++ if (index >= (uint32_t)b->closure_var_count) ++ return NULL; ++ var_refs = JS_VALUE_GET_OBJ(sf->cur_func)->u.func.var_refs; ++ if (!var_refs || !var_refs[index] || !var_refs[index]->pvalue) ++ return NULL; ++ return var_refs[index]; ++} ++ ++JSJitHelperStatus JS_JitHelperFClosure(JSJitExecFrame *frame, ++ uint32_t stack_map_id, ++ uint32_t output, uint32_t index) ++{ ++ JSFunctionBytecode *b; ++ JSStackFrame *sf; ++ JSValue *out, result; ++ ++ if (qjsjit_validate_helper_frame(frame, true, stack_map_id, &b, &sf) < 0) ++ return JS_JIT_HELPER_EXCEPTION; ++ qjsjit_count_helper(frame, JS_JIT_HELPER_FCLOSURE); ++ if (!qjsjit_resolve_slot(frame, b, output, &out) || ++ index >= (uint32_t)b->cpool_count || ++ JS_VALUE_GET_TAG(b->cpool[index]) != JS_TAG_FUNCTION_BYTECODE) ++ return qjsjit_reject_helper(frame, "invalid JIT closure constant"); ++ if (!JS_IsUndefined(*out)) ++ return qjsjit_reject_helper(frame, "occupied JIT closure output"); ++ if (qjsjit_stress_gc(frame) < 0) ++ return JS_JIT_HELPER_EXCEPTION; ++ result = js_closure(frame->ctx, js_dup(b->cpool[index]), ++ JS_VALUE_GET_OBJ(sf->cur_func)->u.func.var_refs, sf); ++ if (JS_IsException(result)) { ++ qjsjit_stress_gc_after_exception(frame); ++ return JS_JIT_HELPER_EXCEPTION; ++ } ++ *out = result; ++ return qjsjit_finish_output(frame, out); ++} ++ ++JSJitHelperStatus JS_JitHelperGetVarRef(JSJitExecFrame *frame, ++ uint32_t stack_map_id, ++ uint32_t output, uint32_t index, ++ uint32_t mode) ++{ ++ JSFunctionBytecode *b; ++ JSStackFrame *sf; ++ JSVarRef *var_ref; ++ JSValue *out, value; ++ ++ if (qjsjit_validate_helper_frame(frame, true, stack_map_id, &b, &sf) < 0) ++ return JS_JIT_HELPER_EXCEPTION; ++ qjsjit_count_helper(frame, JS_JIT_HELPER_GET_VAR_REF); ++ var_ref = qjsjit_closure_var_ref(b, sf, index); ++ if (!qjsjit_resolve_slot(frame, b, output, &out) || !var_ref || ++ (mode != JS_JIT_VAR_REF_PLAIN && mode != JS_JIT_VAR_REF_CHECK)) ++ return qjsjit_reject_helper(frame, "invalid JIT closure variable get"); ++ if (!JS_IsUndefined(*out)) ++ return qjsjit_reject_helper(frame, "occupied JIT closure output"); ++ if (qjsjit_stress_gc(frame) < 0) ++ return JS_JIT_HELPER_EXCEPTION; ++ value = *var_ref->pvalue; ++ if (mode == JS_JIT_VAR_REF_CHECK && unlikely(JS_IsUninitialized(value))) { ++ JS_ThrowReferenceErrorUninitialized2(frame->ctx, b, (int)index, true); ++ qjsjit_stress_gc_after_exception(frame); ++ return JS_JIT_HELPER_EXCEPTION; ++ } ++ *out = js_dup(value); ++ return qjsjit_finish_output(frame, out); ++} ++ ++JSJitHelperStatus JS_JitHelperPutVarRef(JSJitExecFrame *frame, ++ uint32_t stack_map_id, ++ uint32_t input, uint32_t index, ++ uint32_t mode) ++{ ++ JSFunctionBytecode *b; ++ JSStackFrame *sf; ++ JSVarRef *var_ref; ++ JSValue *in, value; ++ bool uninitialized; ++ ++ if (qjsjit_validate_helper_frame(frame, true, stack_map_id, &b, &sf) < 0) ++ return JS_JIT_HELPER_EXCEPTION; ++ qjsjit_count_helper(frame, JS_JIT_HELPER_PUT_VAR_REF); ++ var_ref = qjsjit_closure_var_ref(b, sf, index); ++ if (!qjsjit_resolve_slot(frame, b, input, &in) || !var_ref || ++ mode > JS_JIT_VAR_REF_CHECK_INIT) ++ return qjsjit_reject_helper(frame, "invalid JIT closure variable put"); ++ if (qjsjit_stress_gc(frame) < 0) { ++ qjsjit_consume_slot(frame->ctx, in); ++ return JS_JIT_HELPER_EXCEPTION; ++ } ++ uninitialized = JS_IsUninitialized(*var_ref->pvalue); ++ if ((mode == JS_JIT_VAR_REF_CHECK && unlikely(uninitialized)) || ++ (mode == JS_JIT_VAR_REF_CHECK_INIT && unlikely(!uninitialized))) { ++ JS_ThrowReferenceErrorUninitialized2(frame->ctx, b, (int)index, true); ++ qjsjit_consume_slot(frame->ctx, in); ++ qjsjit_stress_gc_after_exception(frame); ++ return JS_JIT_HELPER_EXCEPTION; ++ } ++ value = *in; ++ *in = JS_UNDEFINED; ++ set_value(frame->ctx, var_ref->pvalue, value); ++ return qjsjit_stress_gc(frame); ++} ++ ++JSJitHelperStatus JS_JitHelperCloseLoc(JSJitExecFrame *frame, ++ uint32_t stack_map_id, uint32_t index) ++{ ++ JSFunctionBytecode *b; ++ JSStackFrame *sf; ++ int var_ref_idx; ++ ++ if (qjsjit_validate_helper_frame(frame, true, stack_map_id, &b, &sf) < 0) ++ return JS_JIT_HELPER_EXCEPTION; ++ qjsjit_count_helper(frame, JS_JIT_HELPER_CLOSE_LOC); ++ if (index >= (uint32_t)b->var_count || !b->vardefs) ++ return qjsjit_reject_helper(frame, "invalid JIT close_loc index"); ++ var_ref_idx = b->vardefs[b->arg_count + index].var_ref_idx; ++ if (var_ref_idx >= sf->var_ref_count || ++ (sf->var_ref_count != 0 && !sf->var_refs)) ++ return qjsjit_reject_helper(frame, "invalid JIT close_loc frame"); ++ if (qjsjit_stress_gc(frame) < 0) ++ return JS_JIT_HELPER_EXCEPTION; ++ close_lexical_var(frame->ctx, b, sf, (int)index); ++ return qjsjit_stress_gc(frame); ++} ++ ++/* OP_set_name names the closure just produced by OP_fclosure (and other ++ * anonymous function/class definitions); the object slot stays borrowed. */ ++JSJitHelperStatus JS_JitHelperSetName(JSJitExecFrame *frame, ++ uint32_t stack_map_id, ++ uint32_t object, uint32_t atom) ++{ ++ JSFunctionBytecode *b; ++ JSStackFrame *sf; ++ JSValue *target; ++ ++ if (qjsjit_validate_helper_frame(frame, true, stack_map_id, &b, &sf) < 0) ++ return JS_JIT_HELPER_EXCEPTION; ++ qjsjit_count_helper(frame, JS_JIT_HELPER_SET_NAME); ++ if (!qjsjit_resolve_slot(frame, b, object, &target) || ++ !qjsjit_valid_atom(frame->rt, (JSAtom)atom)) ++ return qjsjit_reject_helper(frame, "invalid JIT set_name"); ++ if (qjsjit_stress_gc(frame) < 0) ++ return JS_JIT_HELPER_EXCEPTION; ++ if (JS_DefineObjectName(frame->ctx, *target, (JSAtom)atom, ++ JS_PROP_CONFIGURABLE) < 0) { ++ qjsjit_stress_gc_after_exception(frame); ++ return JS_JIT_HELPER_EXCEPTION; ++ } ++ return qjsjit_stress_gc(frame); ++} ++ + JSJitHelperStatus JS_JitHelperCallConstructor( + JSJitExecFrame *frame, uint32_t stack_map_id, uint32_t output, + uint32_t function, uint32_t new_target, uint32_t argv, uint32_t argc) diff --git a/sys/patches/README.md b/sys/patches/README.md index 931cf9bd3..72c7079f8 100644 --- a/sys/patches/README.md +++ b/sys/patches/README.md @@ -60,6 +60,17 @@ data pointer before returning `{ data, count, mode }`. The table advertises zero effects and the receiver is passed by pointer so generated calls do not depend on platform-specific aggregate argument classification. +`0025-tier1-closures.patch` adds ABI 1.25 (runtime API 1.10) Tier 1 closure +helpers as an append-only helper-table tail: `FCLOSURE`, `GET_VAR_REF`, +`PUT_VAR_REF`, `CLOSE_LOC` and `SET_NAME`. Each mirrors its interpreter opcode +on the validated current stack frame (`js_closure`, `*var_refs[idx]->pvalue` +with the exact TDZ `ReferenceError`, `set_value`, `close_lexical_var`, +`JS_DefineObjectName`). `PUT_VAR_REF` consumes its value slot on success and +on exception; `JSJitVarRefMode` selects the plain, checked or +checked-initialization form. Attached var refs alias the interpreter's +argument/local storage, so generated code publishes those slots before every +helper and reloads them afterwards in functions that create closures. + The build accepts only the patch names and byte digests listed in `build_support/patch.rs`, then verifies the complete patched source manifest. Keeping the baseline and patch separate makes diff --git a/sys/src/bindings/aarch64-apple-darwin.rs b/sys/src/bindings/aarch64-apple-darwin.rs index a64c09ee5..6f3c16c7b 100644 --- a/sys/src/bindings/aarch64-apple-darwin.rs +++ b/sys/src/bindings/aarch64-apple-darwin.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2792,6 +2792,10 @@ pub const JSJitCompareOp_JS_JIT_COMPARE_NEQ: JSJitCompareOp = 5; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_EQ: JSJitCompareOp = 6; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_NEQ: JSJitCompareOp = 7; pub type JSJitCompareOp = ::core::ffi::c_uint; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_PLAIN: JSJitVarRefMode = 0; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK: JSJitVarRefMode = 1; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK_INIT: JSJitVarRefMode = 2; +pub type JSJitVarRefMode = ::core::ffi::c_uint; pub const JSJitHelperId_JS_JIT_HELPER_POLL: JSJitHelperId = 0; pub const JSJitHelperId_JS_JIT_HELPER_DUP: JSJitHelperId = 1; pub const JSJitHelperId_JS_JIT_HELPER_FREE: JSJitHelperId = 2; @@ -2816,7 +2820,12 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_FCLOSURE: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GET_VAR_REF: JSJitHelperId = 25; +pub const JSJitHelperId_JS_JIT_HELPER_PUT_VAR_REF: JSJitHelperId = 26; +pub const JSJitHelperId_JS_JIT_HELPER_CLOSE_LOC: JSJitHelperId = 27; +pub const JSJitHelperId_JS_JIT_HELPER_SET_NAME: JSJitHelperId = 28; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 29; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3067,47 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperFClosure( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperGetVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperPutVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperCloseLoc( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperSetName( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3322,51 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub fclosure: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub get_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub put_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub close_loc: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub set_name: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 240usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3422,16 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::fclosure"] + [::core::mem::offset_of!(JSJitRuntimeAPI, fclosure) - 200usize]; + ["Offset of field: JSJitRuntimeAPI::get_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, get_var_ref) - 208usize]; + ["Offset of field: JSJitRuntimeAPI::put_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, put_var_ref) - 216usize]; + ["Offset of field: JSJitRuntimeAPI::close_loc"] + [::core::mem::offset_of!(JSJitRuntimeAPI, close_loc) - 224usize]; + ["Offset of field: JSJitRuntimeAPI::set_name"] + [::core::mem::offset_of!(JSJitRuntimeAPI, set_name) - 232usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/aarch64-pc-windows-msvc.rs b/sys/src/bindings/aarch64-pc-windows-msvc.rs index a64c09ee5..6f3c16c7b 100644 --- a/sys/src/bindings/aarch64-pc-windows-msvc.rs +++ b/sys/src/bindings/aarch64-pc-windows-msvc.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2792,6 +2792,10 @@ pub const JSJitCompareOp_JS_JIT_COMPARE_NEQ: JSJitCompareOp = 5; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_EQ: JSJitCompareOp = 6; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_NEQ: JSJitCompareOp = 7; pub type JSJitCompareOp = ::core::ffi::c_uint; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_PLAIN: JSJitVarRefMode = 0; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK: JSJitVarRefMode = 1; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK_INIT: JSJitVarRefMode = 2; +pub type JSJitVarRefMode = ::core::ffi::c_uint; pub const JSJitHelperId_JS_JIT_HELPER_POLL: JSJitHelperId = 0; pub const JSJitHelperId_JS_JIT_HELPER_DUP: JSJitHelperId = 1; pub const JSJitHelperId_JS_JIT_HELPER_FREE: JSJitHelperId = 2; @@ -2816,7 +2820,12 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_FCLOSURE: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GET_VAR_REF: JSJitHelperId = 25; +pub const JSJitHelperId_JS_JIT_HELPER_PUT_VAR_REF: JSJitHelperId = 26; +pub const JSJitHelperId_JS_JIT_HELPER_CLOSE_LOC: JSJitHelperId = 27; +pub const JSJitHelperId_JS_JIT_HELPER_SET_NAME: JSJitHelperId = 28; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 29; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3067,47 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperFClosure( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperGetVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperPutVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperCloseLoc( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperSetName( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3322,51 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub fclosure: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub get_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub put_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub close_loc: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub set_name: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 240usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3422,16 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::fclosure"] + [::core::mem::offset_of!(JSJitRuntimeAPI, fclosure) - 200usize]; + ["Offset of field: JSJitRuntimeAPI::get_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, get_var_ref) - 208usize]; + ["Offset of field: JSJitRuntimeAPI::put_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, put_var_ref) - 216usize]; + ["Offset of field: JSJitRuntimeAPI::close_loc"] + [::core::mem::offset_of!(JSJitRuntimeAPI, close_loc) - 224usize]; + ["Offset of field: JSJitRuntimeAPI::set_name"] + [::core::mem::offset_of!(JSJitRuntimeAPI, set_name) - 232usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/aarch64-unknown-linux-gnu.rs b/sys/src/bindings/aarch64-unknown-linux-gnu.rs index a64c09ee5..6f3c16c7b 100644 --- a/sys/src/bindings/aarch64-unknown-linux-gnu.rs +++ b/sys/src/bindings/aarch64-unknown-linux-gnu.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2792,6 +2792,10 @@ pub const JSJitCompareOp_JS_JIT_COMPARE_NEQ: JSJitCompareOp = 5; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_EQ: JSJitCompareOp = 6; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_NEQ: JSJitCompareOp = 7; pub type JSJitCompareOp = ::core::ffi::c_uint; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_PLAIN: JSJitVarRefMode = 0; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK: JSJitVarRefMode = 1; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK_INIT: JSJitVarRefMode = 2; +pub type JSJitVarRefMode = ::core::ffi::c_uint; pub const JSJitHelperId_JS_JIT_HELPER_POLL: JSJitHelperId = 0; pub const JSJitHelperId_JS_JIT_HELPER_DUP: JSJitHelperId = 1; pub const JSJitHelperId_JS_JIT_HELPER_FREE: JSJitHelperId = 2; @@ -2816,7 +2820,12 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_FCLOSURE: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GET_VAR_REF: JSJitHelperId = 25; +pub const JSJitHelperId_JS_JIT_HELPER_PUT_VAR_REF: JSJitHelperId = 26; +pub const JSJitHelperId_JS_JIT_HELPER_CLOSE_LOC: JSJitHelperId = 27; +pub const JSJitHelperId_JS_JIT_HELPER_SET_NAME: JSJitHelperId = 28; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 29; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3067,47 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperFClosure( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperGetVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperPutVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperCloseLoc( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperSetName( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3322,51 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub fclosure: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub get_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub put_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub close_loc: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub set_name: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 240usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3422,16 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::fclosure"] + [::core::mem::offset_of!(JSJitRuntimeAPI, fclosure) - 200usize]; + ["Offset of field: JSJitRuntimeAPI::get_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, get_var_ref) - 208usize]; + ["Offset of field: JSJitRuntimeAPI::put_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, put_var_ref) - 216usize]; + ["Offset of field: JSJitRuntimeAPI::close_loc"] + [::core::mem::offset_of!(JSJitRuntimeAPI, close_loc) - 224usize]; + ["Offset of field: JSJitRuntimeAPI::set_name"] + [::core::mem::offset_of!(JSJitRuntimeAPI, set_name) - 232usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/aarch64-unknown-linux-musl.rs b/sys/src/bindings/aarch64-unknown-linux-musl.rs index a64c09ee5..6f3c16c7b 100644 --- a/sys/src/bindings/aarch64-unknown-linux-musl.rs +++ b/sys/src/bindings/aarch64-unknown-linux-musl.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2792,6 +2792,10 @@ pub const JSJitCompareOp_JS_JIT_COMPARE_NEQ: JSJitCompareOp = 5; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_EQ: JSJitCompareOp = 6; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_NEQ: JSJitCompareOp = 7; pub type JSJitCompareOp = ::core::ffi::c_uint; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_PLAIN: JSJitVarRefMode = 0; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK: JSJitVarRefMode = 1; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK_INIT: JSJitVarRefMode = 2; +pub type JSJitVarRefMode = ::core::ffi::c_uint; pub const JSJitHelperId_JS_JIT_HELPER_POLL: JSJitHelperId = 0; pub const JSJitHelperId_JS_JIT_HELPER_DUP: JSJitHelperId = 1; pub const JSJitHelperId_JS_JIT_HELPER_FREE: JSJitHelperId = 2; @@ -2816,7 +2820,12 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_FCLOSURE: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GET_VAR_REF: JSJitHelperId = 25; +pub const JSJitHelperId_JS_JIT_HELPER_PUT_VAR_REF: JSJitHelperId = 26; +pub const JSJitHelperId_JS_JIT_HELPER_CLOSE_LOC: JSJitHelperId = 27; +pub const JSJitHelperId_JS_JIT_HELPER_SET_NAME: JSJitHelperId = 28; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 29; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3067,47 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperFClosure( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperGetVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperPutVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperCloseLoc( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperSetName( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3322,51 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub fclosure: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub get_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub put_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub close_loc: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub set_name: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 240usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3422,16 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::fclosure"] + [::core::mem::offset_of!(JSJitRuntimeAPI, fclosure) - 200usize]; + ["Offset of field: JSJitRuntimeAPI::get_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, get_var_ref) - 208usize]; + ["Offset of field: JSJitRuntimeAPI::put_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, put_var_ref) - 216usize]; + ["Offset of field: JSJitRuntimeAPI::close_loc"] + [::core::mem::offset_of!(JSJitRuntimeAPI, close_loc) - 224usize]; + ["Offset of field: JSJitRuntimeAPI::set_name"] + [::core::mem::offset_of!(JSJitRuntimeAPI, set_name) - 232usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/wasm32-wasip1.rs b/sys/src/bindings/wasm32-wasip1.rs index a8167404c..bd93f7135 100644 --- a/sys/src/bindings/wasm32-wasip1.rs +++ b/sys/src/bindings/wasm32-wasip1.rs @@ -87,7 +87,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -114,7 +114,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2761,6 +2761,10 @@ pub const JSJitCompareOp_JS_JIT_COMPARE_NEQ: JSJitCompareOp = 5; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_EQ: JSJitCompareOp = 6; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_NEQ: JSJitCompareOp = 7; pub type JSJitCompareOp = ::core::ffi::c_uint; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_PLAIN: JSJitVarRefMode = 0; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK: JSJitVarRefMode = 1; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK_INIT: JSJitVarRefMode = 2; +pub type JSJitVarRefMode = ::core::ffi::c_uint; pub const JSJitHelperId_JS_JIT_HELPER_POLL: JSJitHelperId = 0; pub const JSJitHelperId_JS_JIT_HELPER_DUP: JSJitHelperId = 1; pub const JSJitHelperId_JS_JIT_HELPER_FREE: JSJitHelperId = 2; @@ -2785,7 +2789,12 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_FCLOSURE: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GET_VAR_REF: JSJitHelperId = 25; +pub const JSJitHelperId_JS_JIT_HELPER_PUT_VAR_REF: JSJitHelperId = 26; +pub const JSJitHelperId_JS_JIT_HELPER_CLOSE_LOC: JSJitHelperId = 27; +pub const JSJitHelperId_JS_JIT_HELPER_SET_NAME: JSJitHelperId = 28; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 29; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3027,6 +3036,47 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperFClosure( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperGetVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperPutVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperCloseLoc( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperSetName( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3241,10 +3291,51 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub fclosure: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub get_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub put_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub close_loc: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub set_name: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 104usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 124usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 4usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3300,6 +3391,16 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 96usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 100usize]; + ["Offset of field: JSJitRuntimeAPI::fclosure"] + [::core::mem::offset_of!(JSJitRuntimeAPI, fclosure) - 104usize]; + ["Offset of field: JSJitRuntimeAPI::get_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, get_var_ref) - 108usize]; + ["Offset of field: JSJitRuntimeAPI::put_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, put_var_ref) - 112usize]; + ["Offset of field: JSJitRuntimeAPI::close_loc"] + [::core::mem::offset_of!(JSJitRuntimeAPI, close_loc) - 116usize]; + ["Offset of field: JSJitRuntimeAPI::set_name"] + [::core::mem::offset_of!(JSJitRuntimeAPI, set_name) - 120usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/x86_64-apple-darwin.rs b/sys/src/bindings/x86_64-apple-darwin.rs index a64c09ee5..6f3c16c7b 100644 --- a/sys/src/bindings/x86_64-apple-darwin.rs +++ b/sys/src/bindings/x86_64-apple-darwin.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2792,6 +2792,10 @@ pub const JSJitCompareOp_JS_JIT_COMPARE_NEQ: JSJitCompareOp = 5; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_EQ: JSJitCompareOp = 6; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_NEQ: JSJitCompareOp = 7; pub type JSJitCompareOp = ::core::ffi::c_uint; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_PLAIN: JSJitVarRefMode = 0; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK: JSJitVarRefMode = 1; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK_INIT: JSJitVarRefMode = 2; +pub type JSJitVarRefMode = ::core::ffi::c_uint; pub const JSJitHelperId_JS_JIT_HELPER_POLL: JSJitHelperId = 0; pub const JSJitHelperId_JS_JIT_HELPER_DUP: JSJitHelperId = 1; pub const JSJitHelperId_JS_JIT_HELPER_FREE: JSJitHelperId = 2; @@ -2816,7 +2820,12 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_FCLOSURE: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GET_VAR_REF: JSJitHelperId = 25; +pub const JSJitHelperId_JS_JIT_HELPER_PUT_VAR_REF: JSJitHelperId = 26; +pub const JSJitHelperId_JS_JIT_HELPER_CLOSE_LOC: JSJitHelperId = 27; +pub const JSJitHelperId_JS_JIT_HELPER_SET_NAME: JSJitHelperId = 28; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 29; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3067,47 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperFClosure( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperGetVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperPutVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperCloseLoc( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperSetName( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3322,51 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub fclosure: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub get_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub put_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub close_loc: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub set_name: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 240usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3422,16 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::fclosure"] + [::core::mem::offset_of!(JSJitRuntimeAPI, fclosure) - 200usize]; + ["Offset of field: JSJitRuntimeAPI::get_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, get_var_ref) - 208usize]; + ["Offset of field: JSJitRuntimeAPI::put_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, put_var_ref) - 216usize]; + ["Offset of field: JSJitRuntimeAPI::close_loc"] + [::core::mem::offset_of!(JSJitRuntimeAPI, close_loc) - 224usize]; + ["Offset of field: JSJitRuntimeAPI::set_name"] + [::core::mem::offset_of!(JSJitRuntimeAPI, set_name) - 232usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/x86_64-pc-windows-gnu.rs b/sys/src/bindings/x86_64-pc-windows-gnu.rs index a64c09ee5..6f3c16c7b 100644 --- a/sys/src/bindings/x86_64-pc-windows-gnu.rs +++ b/sys/src/bindings/x86_64-pc-windows-gnu.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2792,6 +2792,10 @@ pub const JSJitCompareOp_JS_JIT_COMPARE_NEQ: JSJitCompareOp = 5; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_EQ: JSJitCompareOp = 6; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_NEQ: JSJitCompareOp = 7; pub type JSJitCompareOp = ::core::ffi::c_uint; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_PLAIN: JSJitVarRefMode = 0; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK: JSJitVarRefMode = 1; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK_INIT: JSJitVarRefMode = 2; +pub type JSJitVarRefMode = ::core::ffi::c_uint; pub const JSJitHelperId_JS_JIT_HELPER_POLL: JSJitHelperId = 0; pub const JSJitHelperId_JS_JIT_HELPER_DUP: JSJitHelperId = 1; pub const JSJitHelperId_JS_JIT_HELPER_FREE: JSJitHelperId = 2; @@ -2816,7 +2820,12 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_FCLOSURE: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GET_VAR_REF: JSJitHelperId = 25; +pub const JSJitHelperId_JS_JIT_HELPER_PUT_VAR_REF: JSJitHelperId = 26; +pub const JSJitHelperId_JS_JIT_HELPER_CLOSE_LOC: JSJitHelperId = 27; +pub const JSJitHelperId_JS_JIT_HELPER_SET_NAME: JSJitHelperId = 28; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 29; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3067,47 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperFClosure( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperGetVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperPutVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperCloseLoc( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperSetName( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3322,51 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub fclosure: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub get_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub put_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub close_loc: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub set_name: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 240usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3422,16 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::fclosure"] + [::core::mem::offset_of!(JSJitRuntimeAPI, fclosure) - 200usize]; + ["Offset of field: JSJitRuntimeAPI::get_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, get_var_ref) - 208usize]; + ["Offset of field: JSJitRuntimeAPI::put_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, put_var_ref) - 216usize]; + ["Offset of field: JSJitRuntimeAPI::close_loc"] + [::core::mem::offset_of!(JSJitRuntimeAPI, close_loc) - 224usize]; + ["Offset of field: JSJitRuntimeAPI::set_name"] + [::core::mem::offset_of!(JSJitRuntimeAPI, set_name) - 232usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/x86_64-pc-windows-msvc.rs b/sys/src/bindings/x86_64-pc-windows-msvc.rs index a64c09ee5..6f3c16c7b 100644 --- a/sys/src/bindings/x86_64-pc-windows-msvc.rs +++ b/sys/src/bindings/x86_64-pc-windows-msvc.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2792,6 +2792,10 @@ pub const JSJitCompareOp_JS_JIT_COMPARE_NEQ: JSJitCompareOp = 5; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_EQ: JSJitCompareOp = 6; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_NEQ: JSJitCompareOp = 7; pub type JSJitCompareOp = ::core::ffi::c_uint; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_PLAIN: JSJitVarRefMode = 0; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK: JSJitVarRefMode = 1; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK_INIT: JSJitVarRefMode = 2; +pub type JSJitVarRefMode = ::core::ffi::c_uint; pub const JSJitHelperId_JS_JIT_HELPER_POLL: JSJitHelperId = 0; pub const JSJitHelperId_JS_JIT_HELPER_DUP: JSJitHelperId = 1; pub const JSJitHelperId_JS_JIT_HELPER_FREE: JSJitHelperId = 2; @@ -2816,7 +2820,12 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_FCLOSURE: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GET_VAR_REF: JSJitHelperId = 25; +pub const JSJitHelperId_JS_JIT_HELPER_PUT_VAR_REF: JSJitHelperId = 26; +pub const JSJitHelperId_JS_JIT_HELPER_CLOSE_LOC: JSJitHelperId = 27; +pub const JSJitHelperId_JS_JIT_HELPER_SET_NAME: JSJitHelperId = 28; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 29; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3067,47 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperFClosure( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperGetVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperPutVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperCloseLoc( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperSetName( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3322,51 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub fclosure: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub get_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub put_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub close_loc: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub set_name: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 240usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3422,16 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::fclosure"] + [::core::mem::offset_of!(JSJitRuntimeAPI, fclosure) - 200usize]; + ["Offset of field: JSJitRuntimeAPI::get_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, get_var_ref) - 208usize]; + ["Offset of field: JSJitRuntimeAPI::put_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, put_var_ref) - 216usize]; + ["Offset of field: JSJitRuntimeAPI::close_loc"] + [::core::mem::offset_of!(JSJitRuntimeAPI, close_loc) - 224usize]; + ["Offset of field: JSJitRuntimeAPI::set_name"] + [::core::mem::offset_of!(JSJitRuntimeAPI, set_name) - 232usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/x86_64-unknown-linux-gnu.rs b/sys/src/bindings/x86_64-unknown-linux-gnu.rs index a64c09ee5..6f3c16c7b 100644 --- a/sys/src/bindings/x86_64-unknown-linux-gnu.rs +++ b/sys/src/bindings/x86_64-unknown-linux-gnu.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2792,6 +2792,10 @@ pub const JSJitCompareOp_JS_JIT_COMPARE_NEQ: JSJitCompareOp = 5; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_EQ: JSJitCompareOp = 6; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_NEQ: JSJitCompareOp = 7; pub type JSJitCompareOp = ::core::ffi::c_uint; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_PLAIN: JSJitVarRefMode = 0; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK: JSJitVarRefMode = 1; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK_INIT: JSJitVarRefMode = 2; +pub type JSJitVarRefMode = ::core::ffi::c_uint; pub const JSJitHelperId_JS_JIT_HELPER_POLL: JSJitHelperId = 0; pub const JSJitHelperId_JS_JIT_HELPER_DUP: JSJitHelperId = 1; pub const JSJitHelperId_JS_JIT_HELPER_FREE: JSJitHelperId = 2; @@ -2816,7 +2820,12 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_FCLOSURE: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GET_VAR_REF: JSJitHelperId = 25; +pub const JSJitHelperId_JS_JIT_HELPER_PUT_VAR_REF: JSJitHelperId = 26; +pub const JSJitHelperId_JS_JIT_HELPER_CLOSE_LOC: JSJitHelperId = 27; +pub const JSJitHelperId_JS_JIT_HELPER_SET_NAME: JSJitHelperId = 28; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 29; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3067,47 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperFClosure( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperGetVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperPutVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperCloseLoc( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperSetName( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3322,51 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub fclosure: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub get_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub put_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub close_loc: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub set_name: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 240usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3422,16 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::fclosure"] + [::core::mem::offset_of!(JSJitRuntimeAPI, fclosure) - 200usize]; + ["Offset of field: JSJitRuntimeAPI::get_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, get_var_ref) - 208usize]; + ["Offset of field: JSJitRuntimeAPI::put_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, put_var_ref) - 216usize]; + ["Offset of field: JSJitRuntimeAPI::close_loc"] + [::core::mem::offset_of!(JSJitRuntimeAPI, close_loc) - 224usize]; + ["Offset of field: JSJitRuntimeAPI::set_name"] + [::core::mem::offset_of!(JSJitRuntimeAPI, set_name) - 232usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/x86_64-unknown-linux-musl.rs b/sys/src/bindings/x86_64-unknown-linux-musl.rs index a64c09ee5..6f3c16c7b 100644 --- a/sys/src/bindings/x86_64-unknown-linux-musl.rs +++ b/sys/src/bindings/x86_64-unknown-linux-musl.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2792,6 +2792,10 @@ pub const JSJitCompareOp_JS_JIT_COMPARE_NEQ: JSJitCompareOp = 5; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_EQ: JSJitCompareOp = 6; pub const JSJitCompareOp_JS_JIT_COMPARE_STRICT_NEQ: JSJitCompareOp = 7; pub type JSJitCompareOp = ::core::ffi::c_uint; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_PLAIN: JSJitVarRefMode = 0; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK: JSJitVarRefMode = 1; +pub const JSJitVarRefMode_JS_JIT_VAR_REF_CHECK_INIT: JSJitVarRefMode = 2; +pub type JSJitVarRefMode = ::core::ffi::c_uint; pub const JSJitHelperId_JS_JIT_HELPER_POLL: JSJitHelperId = 0; pub const JSJitHelperId_JS_JIT_HELPER_DUP: JSJitHelperId = 1; pub const JSJitHelperId_JS_JIT_HELPER_FREE: JSJitHelperId = 2; @@ -2816,7 +2820,12 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_FCLOSURE: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GET_VAR_REF: JSJitHelperId = 25; +pub const JSJitHelperId_JS_JIT_HELPER_PUT_VAR_REF: JSJitHelperId = 26; +pub const JSJitHelperId_JS_JIT_HELPER_CLOSE_LOC: JSJitHelperId = 27; +pub const JSJitHelperId_JS_JIT_HELPER_SET_NAME: JSJitHelperId = 28; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 29; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3067,47 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperFClosure( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperGetVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperPutVarRef( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperCloseLoc( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitHelperSetName( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3322,51 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub fclosure: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub get_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub put_var_ref: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + opcode: u32, + ) -> JSJitHelperStatus, + >, + pub close_loc: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + input: u32, + ) -> JSJitHelperStatus, + >, + pub set_name: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + input: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 240usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3422,16 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::fclosure"] + [::core::mem::offset_of!(JSJitRuntimeAPI, fclosure) - 200usize]; + ["Offset of field: JSJitRuntimeAPI::get_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, get_var_ref) - 208usize]; + ["Offset of field: JSJitRuntimeAPI::put_var_ref"] + [::core::mem::offset_of!(JSJitRuntimeAPI, put_var_ref) - 216usize]; + ["Offset of field: JSJitRuntimeAPI::close_loc"] + [::core::mem::offset_of!(JSJitRuntimeAPI, close_loc) - 224usize]; + ["Offset of field: JSJitRuntimeAPI::set_name"] + [::core::mem::offset_of!(JSJitRuntimeAPI, set_name) - 232usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/tests/jit_patch.rs b/sys/tests/jit_patch.rs index f36861117..04d0fd53c 100644 --- a/sys/tests/jit_patch.rs +++ b/sys/tests/jit_patch.rs @@ -56,6 +56,7 @@ fn copy_patches(destination: &std::path::Path) { "0018-inline-frame-recovery.patch", "0019-array-feedback.patch", "0020-typed-array-guard.patch", + "0025-tier1-closures.patch", ] { fs::copy(source.join(patch), destination.join(patch)).unwrap(); } @@ -114,7 +115,7 @@ fn pinned_public_quickjs_baseline_applies_cleanly_without_git() { let helper_header = fs::read_to_string(destination.join("quickjs-jit-helpers.h")).unwrap(); assert!(quickjs.contains("JS_GetJitRuntimeId")); assert!(quickjs.contains("JS_JIT_FRAME_SIDE_PATH_HIT")); - assert!(jit_header.contains("#define QJSJIT_ABI_MINOR 24u")); + assert!(jit_header.contains("#define QJSJIT_ABI_MINOR 25u")); assert!(jit_header.contains("JSJitPropertyLayout property_layout;")); assert!(quickjs.contains("uint64_t jit_shape_generation;")); assert!(!quickjs.contains("QJSJIT_SHAPE_HASH")); @@ -133,7 +134,7 @@ fn pinned_public_quickjs_baseline_applies_cleanly_without_git() { .count(), 3 ); - assert!(jit_header.contains("QJSJIT_RUNTIME_API_MINOR 9u")); + assert!(jit_header.contains("QJSJIT_RUNTIME_API_MINOR 10u")); assert!(jit_header.contains("#define QJSJIT_RUNTIME_FIELD_MAP_OUT_IN_OP(field)")); assert!(helper_header.contains("JS_JIT_HELPER_MATERIALIZED = 2")); assert!(helper_header.contains("JS_JIT_OWNER_SOURCE_ARGUMENT = 0")); @@ -149,9 +150,20 @@ fn pinned_public_quickjs_baseline_applies_cleanly_without_git() { assert!(helper_header.contains("X(ATOM_VALUE, atom_value")); assert!(helper_header.contains("X(BINARY_ARITH_SLOW, binary_arith_slow")); assert!(helper_header.contains("X(UNARY_ARITH_SLOW, unary_arith_slow")); + assert!(helper_header.contains("X(FCLOSURE, fclosure")); + assert!(helper_header.contains("X(GET_VAR_REF, get_var_ref")); + assert!(helper_header.contains("X(PUT_VAR_REF, put_var_ref")); + assert!(helper_header.contains("X(CLOSE_LOC, close_loc")); + assert!(helper_header.contains("X(SET_NAME, set_name")); + assert!(helper_header.contains("JS_JIT_VAR_REF_CHECK_INIT = 2")); assert!(helper_header.contains("#define QJSJIT_DECLARE_MAP_OUT_IN_OP(name)")); assert!(quickjs.contains("JS_JitHelperBinaryArithSlow")); assert!(quickjs.contains("JS_JitHelperUnaryArithSlow")); + assert!(quickjs.contains("JS_JitHelperFClosure")); + assert!(quickjs.contains("JS_JitHelperGetVarRef")); + assert!(quickjs.contains("JS_JitHelperPutVarRef")); + assert!(quickjs.contains("JS_JitHelperCloseLoc")); + assert!(quickjs.contains("JS_JitHelperSetName")); assert!(quickjs.contains("#define QJSJIT_ABI_COUNT_MAP_OUT_IN_OP 6")); assert!(jit_header.contains("JSJitFeedbackEvent")); assert!(destination.join("quickjs-jit-helpers.h").is_file()); @@ -255,7 +267,7 @@ fn bundled_jit_bindings_include_materialize_owner_tail() { for target in targets { let binding = fs::read_to_string(bindings.join(target)).unwrap(); assert!( - binding.contains("pub const QJSJIT_ABI_MINOR: u32 = 24;"), + binding.contains("pub const QJSJIT_ABI_MINOR: u32 = 25;"), "{target}" ); assert!( @@ -299,15 +311,21 @@ fn bundled_jit_bindings_include_materialize_owner_tail() { assert!( binding.contains("JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22") && binding.contains("JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23") - && binding.contains("JS_JIT_HELPER_COUNT: JSJitHelperId = 24"), + && binding.contains("JS_JIT_HELPER_FCLOSURE: JSJitHelperId = 24") + && binding.contains("JS_JIT_HELPER_GET_VAR_REF: JSJitHelperId = 25") + && binding.contains("JS_JIT_HELPER_PUT_VAR_REF: JSJitHelperId = 26") + && binding.contains("JS_JIT_HELPER_CLOSE_LOC: JSJitHelperId = 27") + && binding.contains("JS_JIT_HELPER_SET_NAME: JSJitHelperId = 28") + && binding.contains("JS_JIT_HELPER_COUNT: JSJitHelperId = 29"), "{target}" ); assert!( - binding.contains("pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9;"), + binding.contains("pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10;"), "{target}" ); assert!( - binding.contains("size_of::() - 200usize"), + binding.contains("size_of::() - 240usize") + && binding.contains("pub close_loc: ::core::option::Option"), "{target}" ); } @@ -362,8 +380,14 @@ fn bundled_wasm_jit_binding_matches_array_abi_1_24() { "pub const JS_JIT_FEEDBACK_ARRAY_IMMUTABLE: u32 = 4096;", "pub const JS_JIT_FEEDBACK_ARRAY_SHARED: u32 = 8192;", "pub const JS_JIT_FEEDBACK_ARRAY_INVALID_BACKING: u32 = 16384;", + // ABI 1.25: Tier 1 closure helpers. + "pub const QJSJIT_ABI_MINOR: u32 = 25;", + "pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10;", + "JS_JIT_HELPER_CLOSE_LOC: JSJitHelperId = 27", + "JSJitVarRefMode_JS_JIT_VAR_REF_CHECK_INIT: JSJitVarRefMode = 2;", + "size_of::() - 124usize", + "JS_JIT_HELPER_SET_NAME: JSJitHelperId = 28", // ABI 1.24: typed-array continuing guard. - "pub const QJSJIT_ABI_MINOR: u32 = 24;", "pub const QJSJIT_ARRAY_API_VERSION: u32 = 1;", "pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1;", "JSJitArrayQueryStatus_JS_JIT_ARRAY_QUERY_INVALID: JSJitArrayQueryStatus = -1;", From bfe11e0d77fb092c4ff309fc8ae51f315abb9186 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 16:54:01 +0800 Subject: [PATCH 11/54] perf(jit): bounded polymorphic property inline caches Record up to four own-data shapes per property site (POLYMORPHIC_PROPERTY_LIMIT) and use them in both native tiers: - Tier 1 dispatches inline on the receiver's shape pointer and layout generation (the SHAPE_GUARD predicate) with per-shape slot offsets, dropping the per-site cache and the helper crossing; misses keep the exact GET/SET_PROPERTY helper edge. - Automatic tiering refreshes a baseline once with property feedback when the function is statically outside the Tier-2 vocabulary, so terminal baselines (objects-polymorphic, mixed-quotes workload) get the ICs. - Tier 2 accepts four-shape chains, compares shapes inline for frame-materialized receivers too, lowers megamorphic get/put sites to the owning generic helpers instead of failing (no deopt loop), and rejects shape-guarded receivers that are optimized element loads, which could only deopt on every execution. Co-Authored-By: Claude Opus 5.5 (1M context) --- jit/Cargo.toml | 4 + jit/src/compiler/baseline.rs | 340 +++++++----------- jit/src/compiler/optimized.rs | 381 +++++++++++++------- jit/src/ir/mod.rs | 10 +- jit/src/ir/optimized.rs | 22 ++ jit/src/lib.rs | 80 ++++- jit/src/runtime/mod.rs | 2 +- jit/src/runtime/shape_feedback.rs | 6 + jit/tests/polymorphic_property_ic.rs | 508 +++++++++++++++++++++++++++ jit/tests/property_specialization.rs | 10 +- jit/tests/tier1_calls_properties.rs | 16 +- 11 files changed, 1015 insertions(+), 364 deletions(-) create mode 100644 jit/tests/polymorphic_property_ic.rs diff --git a/jit/Cargo.toml b/jit/Cargo.toml index e2b37836d..7941131bd 100644 --- a/jit/Cargo.toml +++ b/jit/Cargo.toml @@ -111,6 +111,10 @@ required-features = ["compiler", "test-support"] name = "property_specialization" required-features = ["compiler", "test-support"] +[[test]] +name = "polymorphic_property_ic" +required-features = ["compiler", "test-support"] + [[test]] name = "speculative_numeric" required-features = ["compiler", "test-support"] diff --git a/jit/src/compiler/baseline.rs b/jit/src/compiler/baseline.rs index 00e7cfb64..449b0cbfd 100644 --- a/jit/src/compiler/baseline.rs +++ b/jit/src/compiler/baseline.rs @@ -15,7 +15,7 @@ use cranelift_codegen::{ ir::{ condcodes::{FloatCC, IntCC}, types, AbiParam, ArgumentPurpose, Block, Function, InstBuilder, MemFlags, Signature, - SourceLoc, StackSlot, StackSlotData, StackSlotKind, TrapCode, Value, + SourceLoc, TrapCode, Value, }, isa::{unwind::UnwindInfo as CraneliftUnwindInfo, OwnedTargetIsa, TargetIsa}, settings::{self, Configurable}, @@ -199,67 +199,6 @@ impl HelperLowering<'_> { builder.switch_to_block(continuation); Ok(()) } - - fn shape_guard( - &self, - builder: &mut FunctionBuilder<'_>, - state: FrameStateId, - live_depth: usize, - object: u32, - shape: crate::runtime::ShapeToken, - ) -> Result { - let frame_state = self.ir.frame_states.get(state); - let fixed_slots = usize::from(self.ir.argument_count) + usize::from(self.ir.local_count); - let visible_depth = frame_state - .slots - .len() - .checked_sub(fixed_slots) - .ok_or(CompileFailure::InvalidArtifact)?; - materialize_frame( - builder, - self.frame, - self.arg_buf, - self.var_buf, - self.stack_base, - self.arguments, - self.locals, - self.stack, - live_depth, - visible_depth, - frame_state.pc, - self.pointer_type, - self.layout, - )?; - let helper_id = qjs::JSJitHelperId_JS_JIT_HELPER_SHAPE_GUARD as usize; - let helper = builder.ins().load( - self.pointer_type, - MemFlags::new(), - self.runtime_api, - self.layout.helper_offsets[helper_id], - ); - let params = [ - self.frame, - helper_u32( - builder, - u32::try_from(state.index()).map_err(|_| CompileFailure::ResourceLimit)?, - ), - helper_u32(builder, object), - helper_u32(builder, shape.identity() as u32), - helper_u32(builder, (shape.identity() >> 32) as u32), - helper_u32(builder, shape.generation() as u32), - helper_u32(builder, (shape.generation() >> 32) as u32), - ]; - let call = emit_external_call( - builder, - self.signatures[helper_id], - helper, - ¶ms, - self.pointer_type, - Some(self.frame), - None, - ); - Ok(builder.inst_results(call)[0]) - } } /// Cranelift compiler configured for one explicit target ISA. @@ -301,16 +240,19 @@ fn baseline_property_sites( let observations = site.observations(); let safe = site.state() != ShapeFeedbackState::Megamorphic && !observations.is_empty() - && observations.len() <= 3 + && observations.len() <= crate::runtime::POLYMORPHIC_PROPERTY_LIMIT && observations.iter().all(|observation| { - matches!( - observation.value(), - ObservedType::Int32 - | ObservedType::Float64 - | ObservedType::Bool - | ObservedType::Null - | ObservedType::Undefined - ) && observation.prototype().identity() == 0 + observation.shape().identity() != 0 + && observation.shape().generation() != 0 + && matches!( + observation.value(), + ObservedType::Int32 + | ObservedType::Float64 + | ObservedType::Bool + | ObservedType::Null + | ObservedType::Undefined + ) + && observation.prototype().identity() == 0 && observation.prototype().generation() == 0 && !observation .attributes() @@ -2469,21 +2411,6 @@ fn lower_function( } let sret = params[0]; let frame = params[1]; - let property_caches = properties - .iter() - .map(|property| { - let slot = builder.create_sized_stack_slot(StackSlotData::new( - StackSlotKind::ExplicitSlot, - 8, - 0, - )); - (property.pc, slot) - }) - .collect::>(); - for slot in property_caches.values().copied() { - let zero = builder.ins().iconst(types::I64, 0); - builder.ins().stack_store(zero, slot, 0); - } if analysis.retry_before_entry { emit_exit( @@ -2769,7 +2696,6 @@ fn lower_function( properties .iter() .find(|site| site.pc == instruction.pc && !site.store), - property_caches.get(&instruction.pc).copied(), )?, IrOp::GetPropertyKeep(atom) => lower_get_property_keep( builder, @@ -2787,7 +2713,6 @@ fn lower_function( properties .iter() .find(|site| site.pc == instruction.pc && site.store), - property_caches.get(&instruction.pc).copied(), )?, IrOp::DefineProperty(atom) => lower_define_property( builder, @@ -3870,7 +3795,6 @@ fn lower_get_property( depth: usize, atom: u32, property: Option<&BaselinePropertySite>, - property_cache: Option, ) -> Result<(), CompileFailure> { let object_index = depth .checked_sub(1) @@ -3880,72 +3804,31 @@ fn lower_get_property( let object = flat_stack_slot(helpers.ir, object_index)?; let output = flat_stack_slot(helpers.ir, output_index)?; let get_state = next_helper_state(states)?; - if let Some((property, property_cache)) = property.zip(property_cache) { + if let Some(property) = property { let generic = builder.create_block(); let joined = builder.create_block(); builder.append_block_param(joined, types::I64); builder.append_block_param(joined, types::I64); - for (index, observation) in property.observations.iter().copied().enumerate() { - let access = builder.create_block(); - let validate = builder.create_block(); - let next = if index + 1 == property.observations.len() { - generic - } else { - builder.create_block() - }; - let receiver = use_pair(builder, helpers.stack[object_index]); - let object_tag = - builder - .ins() - .icmp_imm(IntCC::Equal, receiver.tag, i64::from(qjs::JS_TAG_OBJECT)); - let cached_check = builder.create_block(); - builder - .ins() - .brif(object_tag, cached_check, &[], validate, &[]); - builder.switch_to_block(cached_check); - let current_shape = builder.ins().load( - helpers.pointer_type, - MemFlags::trusted(), - receiver.payload, - 24, - ); - let cached = builder.ins().stack_load(types::I64, property_cache, 0); - let expected = observation.shape().identity(); - let pointer_ok = builder - .ins() - .icmp_imm(IntCC::Equal, current_shape, expected as i64); - let cache_ok = builder - .ins() - .icmp_imm(IntCC::Equal, cached, expected as i64); - let validated = builder.ins().band(pointer_ok, cache_ok); - builder.ins().brif(validated, access, &[], validate, &[]); - builder.switch_to_block(validate); - let status = - helpers.shape_guard(builder, get_state, depth, object, observation.shape())?; - let matched = - builder - .ins() - .icmp_imm(IntCC::Equal, status, i64::from(qjs::JS_JIT_HELPER_OK)); - let cache = builder.create_block(); - builder.ins().brif(matched, cache, &[], next, &[]); - builder.switch_to_block(cache); - let expected_value = builder.ins().iconst(types::I64, expected as i64); - builder.ins().stack_store(expected_value, property_cache, 0); - builder.ins().jump(access, &[]); + let receiver = use_pair(builder, helpers.stack[object_index]); + let accesses = emit_shape_dispatch( + builder, + helpers.pointer_type, + receiver, + &property.observations, + generic, + )?; + let layout = crate::abi::AbiInfo::linked() + .map_err(|_| CompileFailure::InvalidArtifact)? + .property_layout(); + for (observation, access) in property.observations.iter().copied().zip(accesses) { builder.switch_to_block(access); let props = builder.ins().load( helpers.pointer_type, MemFlags::trusted(), receiver.payload, - 32, + layout.object_properties_offset, ); - let offset = i32::try_from( - usize::try_from(observation.offset()) - .map_err(|_| CompileFailure::ResourceLimit)? - .checked_mul(16) - .ok_or(CompileFailure::ResourceLimit)?, - ) - .map_err(|_| CompileFailure::ResourceLimit)?; + let offset = property_slot_offset(observation)?; let value = Pair { payload: builder .ins() @@ -3963,9 +3846,6 @@ fn lower_get_property( builder.ins().brif(tag_ok, direct, &[], generic, &[]); builder.switch_to_block(direct); builder.ins().jump(joined, &[value.payload, value.tag]); - if index + 1 != property.observations.len() { - builder.switch_to_block(next); - } } builder.switch_to_block(generic); helpers.invoke( @@ -4044,6 +3924,95 @@ fn lower_get_property( helpers.set_depth(builder, depth) } +/// Byte offset of an own-property slot in `JSObject::prop`. +fn property_slot_offset( + observation: crate::runtime::ShapeObservation, +) -> Result { + usize::try_from(observation.offset()) + .ok() + .and_then(|offset| offset.checked_mul(16)) + .and_then(|offset| i32::try_from(offset).ok()) + .filter(|offset| offset.checked_add(8).is_some()) + .ok_or(CompileFailure::ResourceLimit) +} + +/// Inline polymorphic own-property dispatch: one pointer compare per observed +/// shape, then the shape's monotonic layout generation. It is exactly the +/// SHAPE_GUARD helper predicate (the generation is zeroed on any layout +/// change and never reused), without a frame materialization or helper +/// crossing. A non-object receiver, an unknown shape or a stale generation +/// reaches `generic`. Returns one access block per observation, in feedback +/// order; the current block is terminated. +fn emit_shape_dispatch( + builder: &mut FunctionBuilder<'_>, + pointer_type: cranelift_codegen::ir::Type, + receiver: Pair, + observations: &[crate::runtime::ShapeObservation], + generic: Block, +) -> Result, CompileFailure> { + if observations.is_empty() || observations.len() > crate::runtime::POLYMORPHIC_PROPERTY_LIMIT { + return Err(CompileFailure::InvalidArtifact); + } + for (index, observation) in observations.iter().enumerate() { + if observation.shape().identity() == 0 + || observation.shape().generation() == 0 + || observations[..index] + .iter() + .any(|prior| prior.shape().identity() == observation.shape().identity()) + { + return Err(CompileFailure::InvalidArtifact); + } + } + let layout = crate::abi::AbiInfo::linked() + .map_err(|_| CompileFailure::InvalidArtifact)? + .property_layout(); + let is_object = + builder + .ins() + .icmp_imm(IntCC::Equal, receiver.tag, i64::from(qjs::JS_TAG_OBJECT)); + let dispatch = builder.create_block(); + builder.ins().brif(is_object, dispatch, &[], generic, &[]); + builder.switch_to_block(dispatch); + let shape = builder.ins().load( + pointer_type, + MemFlags::trusted(), + receiver.payload, + layout.object_shape_offset, + ); + // A live object's shape is always valid; load its generation once. + let generation = builder.ins().load( + types::I64, + MemFlags::trusted(), + shape, + layout.shape_generation_offset, + ); + let mut accesses = Vec::with_capacity(observations.len()); + for (index, observation) in observations.iter().enumerate() { + let next = if index + 1 == observations.len() { + generic + } else { + builder.create_block() + }; + let same_shape = + builder + .ins() + .icmp_imm(IntCC::Equal, shape, observation.shape().identity() as i64); + let current = builder.ins().icmp_imm( + IntCC::Equal, + generation, + observation.shape().generation() as i64, + ); + let matches = builder.ins().band(same_shape, current); + let access = builder.create_block(); + builder.ins().brif(matches, access, &[], next, &[]); + accesses.push(access); + if index + 1 != observations.len() { + builder.switch_to_block(next); + } + } + Ok(accesses) +} + fn property_value_tag(value: crate::runtime::ObservedType) -> Result { Ok(match value { crate::runtime::ObservedType::Int32 => qjs::JS_TAG_INT, @@ -4097,7 +4066,6 @@ fn lower_set_property( depth: &mut usize, atom: u32, property: Option<&BaselinePropertySite>, - property_cache: Option, ) -> Result<(), CompileFailure> { let object_index = depth .checked_sub(2) @@ -4106,70 +4074,29 @@ fn lower_set_property( let object = flat_stack_slot(helpers.ir, object_index)?; let value = flat_stack_slot(helpers.ir, value_index)?; let set_state = next_helper_state(states)?; - if let Some((property, property_cache)) = property.zip(property_cache) { + if let Some(property) = property { let generic = builder.create_block(); let joined = builder.create_block(); - for (index, observation) in property.observations.iter().copied().enumerate() { - let access = builder.create_block(); - let validate = builder.create_block(); - let next = if index + 1 == property.observations.len() { - generic - } else { - builder.create_block() - }; - let receiver = use_pair(builder, helpers.stack[object_index]); - let object_tag = - builder - .ins() - .icmp_imm(IntCC::Equal, receiver.tag, i64::from(qjs::JS_TAG_OBJECT)); - let cached_check = builder.create_block(); - builder - .ins() - .brif(object_tag, cached_check, &[], validate, &[]); - builder.switch_to_block(cached_check); - let current_shape = builder.ins().load( - helpers.pointer_type, - MemFlags::trusted(), - receiver.payload, - 24, - ); - let cached = builder.ins().stack_load(types::I64, property_cache, 0); - let expected = observation.shape().identity(); - let pointer_ok = builder - .ins() - .icmp_imm(IntCC::Equal, current_shape, expected as i64); - let cache_ok = builder - .ins() - .icmp_imm(IntCC::Equal, cached, expected as i64); - let validated = builder.ins().band(pointer_ok, cache_ok); - builder.ins().brif(validated, access, &[], validate, &[]); - builder.switch_to_block(validate); - let status = - helpers.shape_guard(builder, set_state, *depth, object, observation.shape())?; - let matched = - builder - .ins() - .icmp_imm(IntCC::Equal, status, i64::from(qjs::JS_JIT_HELPER_OK)); - let cache = builder.create_block(); - builder.ins().brif(matched, cache, &[], next, &[]); - builder.switch_to_block(cache); - let expected_value = builder.ins().iconst(types::I64, expected as i64); - builder.ins().stack_store(expected_value, property_cache, 0); - builder.ins().jump(access, &[]); + let receiver = use_pair(builder, helpers.stack[object_index]); + let accesses = emit_shape_dispatch( + builder, + helpers.pointer_type, + receiver, + &property.observations, + generic, + )?; + let layout = crate::abi::AbiInfo::linked() + .map_err(|_| CompileFailure::InvalidArtifact)? + .property_layout(); + for (observation, access) in property.observations.iter().copied().zip(accesses) { builder.switch_to_block(access); let props = builder.ins().load( helpers.pointer_type, MemFlags::trusted(), receiver.payload, - 32, + layout.object_properties_offset, ); - let offset = i32::try_from( - usize::try_from(observation.offset()) - .map_err(|_| CompileFailure::ResourceLimit)? - .checked_mul(16) - .ok_or(CompileFailure::ResourceLimit)?, - ) - .map_err(|_| CompileFailure::ResourceLimit)?; + let offset = property_slot_offset(observation)?; let expected_tag = property_value_tag(observation.value())?; let current_tag = builder @@ -4201,9 +4128,6 @@ fn lower_set_property( helpers.layout, )?; builder.ins().jump(joined, &[]); - if index + 1 != property.observations.len() { - builder.switch_to_block(next); - } } builder.switch_to_block(generic); helpers.invoke( diff --git a/jit/src/compiler/optimized.rs b/jit/src/compiler/optimized.rs index afc40c273..71159babd 100644 --- a/jit/src/compiler/optimized.rs +++ b/jit/src/compiler/optimized.rs @@ -776,6 +776,10 @@ struct NumericSpecialization { float_pcs: std::collections::BTreeSet, calls: std::collections::BTreeMap, properties: std::collections::BTreeMap>, + /// get_field/put_field sites whose feedback is megamorphic. They lower to + /// the owning generic property helper instead of a shape guard, so a + /// shape miss never becomes a deopt/recompile loop. + generic_properties: std::collections::BTreeSet, arrays: Box<[crate::runtime::ArrayFeedbackSnapshot]>, direct_calls: std::collections::BTreeMap, inline_callees: std::collections::BTreeMap, @@ -965,7 +969,7 @@ impl NumericSpecialization { let observations = site.observations(); let safe = site.state() != crate::runtime::ShapeFeedbackState::Megamorphic && !observations.is_empty() - && observations.len() <= 3 + && observations.len() <= crate::runtime::POLYMORPHIC_PROPERTY_LIMIT && observations.iter().all(|observation| { let primitive = matches!( observation.value(), @@ -991,6 +995,17 @@ impl NumericSpecialization { safe.then_some((instruction.pc(), observations.to_vec().into_boxed_slice())) }) .collect(); + let generic_properties = function + .instructions() + .iter() + .filter(|instruction| { + matches!(instruction.opcode().name(), "get_field" | "put_field") + && feedback.property_at(instruction.pc()).is_some_and(|site| { + site.state() == crate::runtime::ShapeFeedbackState::Megamorphic + }) + }) + .map(|instruction| instruction.pc()) + .collect(); let numeric_constants = function .snapshot() .constants() @@ -1061,6 +1076,7 @@ impl NumericSpecialization { int_pcs, calls, properties, + generic_properties, arrays, numeric_constants, ..Self::default() @@ -1076,6 +1092,7 @@ impl NumericSpecialization { int_pcs, calls, properties, + generic_properties, arrays, numeric_constants, ..Self::default() @@ -1130,6 +1147,7 @@ impl NumericSpecialization { float_pcs, calls, properties, + generic_properties, arrays, direct_calls: Default::default(), inline_callees: Default::default(), @@ -1139,6 +1157,35 @@ impl NumericSpecialization { } } +/// Optimized element loads publish only primitives: a heap element +/// deoptimizes before it reaches the operand stack. A shape-guarded property +/// access whose receiver is exactly such an element load therefore deopts on +/// every execution (an object element exits at the load, a primitive one at +/// the receiver guard). Reject the function up front instead of installing an +/// artifact that can only bounce between tiers. Generic (megamorphic) sites +/// accept any receiver and remain admissible. +fn shape_guarded_element_receiver( + ir: &OptimizedIr, + specialization: &NumericSpecialization, +) -> bool { + let graph = ir.scalar_graph(); + ir.nodes().iter().any(|node| { + let base = match graph.heap_operation(node.id()) { + Some( + crate::ir::ScalarHeapOperation::GetProperty { base, .. } + | crate::ir::ScalarHeapOperation::PutProperty { base, .. }, + ) => *base, + _ => return false, + }; + specialization.properties.contains_key(&node.pc()) + && !specialization.generic_properties.contains(&node.pc()) + && matches!( + graph.values().get(base.index()), + Some(crate::ir::ScalarValue::GetElement { .. }) + ) + }) +} + fn lower_optimized_machine( isa: &cranelift_codegen::isa::OwnedTargetIsa, ir: &OptimizedIr, @@ -1166,6 +1213,9 @@ fn lower_optimized_machine( let Some(entry_site) = ir.guard_maps().first() else { return Err(CompileFailure::InvalidArtifact); }; + if shape_guarded_element_receiver(ir, specialization) { + return Err(CompileFailure::UnsupportedOpcode); + } let shape = entry_site.shape(); let int32_loop = matches!(specialization.entry, EntryRepresentation::Int32) && specialization.calls.is_empty() @@ -1730,11 +1780,6 @@ fn lower_optimized_machine( .cloned() .ok_or(CompileFailure::InvalidArtifact)?; let poll_signature = builder.import_signature(poll_signature); - let shape_guard_signature = generated_signatures - .get(qjs::JSJitHelperId_JS_JIT_HELPER_SHAPE_GUARD as usize) - .cloned() - .ok_or(CompileFailure::InvalidArtifact)?; - let shape_guard_signature = builder.import_signature(shape_guard_signature); let helper_signatures = generated_signatures .into_iter() .map(|signature| builder.import_signature(signature)) @@ -2451,15 +2496,32 @@ fn lower_optimized_machine( .ok_or(CompileFailure::InvalidArtifact)?; } "get_field" | "put_field" => { + if name == "put_field" + && specialization.generic_properties.contains(&node.pc()) + { + property_cache.flush(&mut builder); + property_cache.invalidate(&mut builder); + let atom = opt_u32(node.bytes())?; + depth = emit_opt_owned_property_store( + &mut builder, + &env, + &mut stack_provenance, + depth, + node.pc(), + atom, + )?; + continue; + } if name == "get_field" - && specialization.properties.get(&node.pc()).is_some_and( - |observations| { - observations.iter().any(|observation| { - observation.value() - == crate::runtime::ObservedType::Object - }) - }, - ) + && (specialization.generic_properties.contains(&node.pc()) + || specialization.properties.get(&node.pc()).is_some_and( + |observations| { + observations.iter().any(|observation| { + observation.value() + == crate::runtime::ObservedType::Object + }) + }, + )) { property_cache.flush(&mut builder); property_cache.invalidate(&mut builder); @@ -2532,7 +2594,6 @@ fn lower_optimized_machine( property, node.pc(), node.deopt_guard().unwrap_or(entry_site.guard()), - shape_guard_signature, &helper_signatures, pointer_type, layout, @@ -6764,7 +6825,6 @@ fn emit_opt_guarded_property( properties: &[crate::runtime::ShapeObservation], pc: u32, guard: u32, - signature: cranelift_codegen::ir::SigRef, helper_signatures: &[cranelift_codegen::ir::SigRef], pointer_type: cranelift_codegen::ir::Type, layout: super::helpers::FrameLayout, @@ -6828,122 +6888,75 @@ fn emit_opt_guarded_property( layout, )?; } - if properties.is_empty() || properties.len() > 3 { + if properties.is_empty() || properties.len() > crate::runtime::POLYMORPHIC_PROPERTY_LIMIT { return Err(CompileFailure::InvalidArtifact); } - let flat = borrowed_slot - .or_else(|| { - arguments - .len() - .checked_add(locals.len())? - .checked_add(object_index) - }) - .and_then(|slot| u32::try_from(slot).ok()) - .ok_or(CompileFailure::ResourceLimit)?; + for (index, property) in properties.iter().enumerate() { + // Feedback retains one observation per shape identity; a duplicate + // would make the first matching compare select a stale slot. + if property.shape().identity() == 0 + || property.shape().generation() == 0 + || properties[..index] + .iter() + .any(|prior| prior.shape().identity() == property.shape().identity()) + { + return Err(CompileFailure::InvalidArtifact); + } + } let property_layout = crate::abi::AbiInfo::linked() .map_err(|_| CompileFailure::InvalidArtifact)? .property_layout(); - let helper = if borrowed_slot.is_none() { - let api = builder - .ins() - .load(pointer_type, MemFlags::new(), frame, layout.runtime_api); - Some(builder.ins().load( - pointer_type, - MemFlags::new(), - api, - layout.helper_offsets[qjs::JSJitHelperId_JS_JIT_HELPER_SHAPE_GUARD as usize], - )) - } else { - None - }; let deopt = builder.create_block(); - let exception = builder.create_block(); let continuation = builder.create_block(); if !store { builder.append_block_param(continuation, types::I64); builder.append_block_param(continuation, types::I64); } + // Inline polymorphic dispatch for both borrowed and owned receivers: the + // receiver's shape pointer and monotonic layout generation are exactly the + // SHAPE_GUARD helper predicate, so no helper crossing is needed. The owned + // path above has already published the frame for the deopt exit. + let object = opt_use(builder, stack[object_index]); + let is_object = builder + .ins() + .icmp_imm(IntCC::Equal, object.tag, i64::from(qjs::JS_TAG_OBJECT)); + let dispatch = builder.create_block(); + builder.ins().brif(is_object, dispatch, &[], deopt, &[]); + builder.switch_to_block(dispatch); + // Read only the live receiver's shape. The feedback pointer is an integer + // identity, never a pointer we dereference or retain. A live object's + // shape is always valid, so its generation is loaded once for the chain. + let shape = builder.ins().load( + pointer_type, + MemFlags::new(), + object.payload, + property_layout.object_shape_offset, + ); + let live_generation = builder.ins().load( + types::I64, + MemFlags::new(), + shape, + property_layout.shape_generation_offset, + ); for (index, property) in properties.iter().copied().enumerate() { let id = property.shape().identity(); let generation = property.shape().generation(); let access = builder.create_block(); + let check = builder.create_block(); let next = if index + 1 == properties.len() { deopt } else { builder.create_block() }; - if borrowed_slot.is_some() { - if generation == 0 { - return Err(CompileFailure::InvalidArtifact); - } - let object = opt_use(builder, stack[object_index]); - let is_object = - builder - .ins() - .icmp_imm(IntCC::Equal, object.tag, i64::from(qjs::JS_TAG_OBJECT)); - let check_shape = builder.create_block(); - builder.ins().brif(is_object, check_shape, &[], deopt, &[]); - builder.switch_to_block(check_shape); - // Read only the live receiver's shape. The feedback pointer is an - // integer identity, never a pointer we dereference or retain. - let shape = builder.ins().load( - pointer_type, - MemFlags::new(), - object.payload, - property_layout.object_shape_offset, - ); - let same_shape = builder.ins().icmp_imm(IntCC::Equal, shape, id as i64); - let live_generation = builder.ins().load( - types::I64, - MemFlags::new(), - shape, - property_layout.shape_generation_offset, - ); - let same_generation = - builder - .ins() - .icmp_imm(IntCC::Equal, live_generation, generation as i64); - let matches = builder.ins().band(same_shape, same_generation); - builder.ins().brif(matches, access, &[], next, &[]); - } else { - let params = [ - frame, - builder.ins().iconst(types::I32, 0), - builder.ins().iconst(types::I32, i64::from(flat)), - builder.ins().iconst(types::I32, i64::from(id as u32)), - builder - .ins() - .iconst(types::I32, i64::from((id >> 32) as u32)), - builder - .ins() - .iconst(types::I32, i64::from(generation as u32)), - builder - .ins() - .iconst(types::I32, i64::from((generation >> 32) as u32)), - ]; - let call = super::emit_external_call( - builder, - signature, - helper.unwrap(), - ¶ms, - pointer_type, - Some(frame), - None, - ); - let status = builder.inst_results(call)[0]; - let ok = builder + builder.ins().jump(check, &[]); + builder.switch_to_block(check); + let same_shape = builder.ins().icmp_imm(IntCC::Equal, shape, id as i64); + let same_generation = + builder .ins() - .icmp_imm(IntCC::Equal, status, i64::from(qjs::JS_JIT_HELPER_OK)); - let miss_or_exception = builder.create_block(); - builder.ins().brif(ok, access, &[], miss_or_exception, &[]); - builder.switch_to_block(miss_or_exception); - let miss = builder.ins().icmp_imm( - IntCC::Equal, - status, - i64::from(qjs::JS_JIT_HELPER_GUARD_MISS), - ); - builder.ins().brif(miss, next, &[], exception, &[]); - } + .icmp_imm(IntCC::Equal, live_generation, generation as i64); + let matches = builder.ins().band(same_shape, same_generation); + builder.ins().brif(matches, access, &[], next, &[]); builder.switch_to_block(access); let object = opt_use(builder, stack[object_index]); let props = builder.ins().load( @@ -7002,15 +7015,6 @@ fn emit_opt_guarded_property( builder.switch_to_block(next); } } - builder.switch_to_block(exception); - emit_opt_exit( - builder, - sret, - qjs::JSJitExitKind_JS_JIT_EXIT_EXCEPTION, - None, - pointer_type, - 0, - ); builder.switch_to_block(deopt); if borrowed_slot.is_some() { // No state publication or helper crossing occurs on the native hit. @@ -7820,14 +7824,15 @@ fn owned_local_targets( "get_var" | "get_length" => true, "get_field2" => true, "get_field" => { - specialization - .properties - .get(&node.pc()) - .is_some_and(|observations| { - observations.iter().any(|observation| { - observation.value() == crate::runtime::ObservedType::Object + specialization.generic_properties.contains(&node.pc()) + || specialization + .properties + .get(&node.pc()) + .is_some_and(|observations| { + observations.iter().any(|observation| { + observation.value() == crate::runtime::ObservedType::Object + }) }) - }) } n if n.starts_with("call") => { ir.scalar_graph() @@ -8379,6 +8384,128 @@ fn emit_opt_owned_property_replace( Ok(depth) } +/// Executes PutField through the audited generic SET_PROPERTY helper for a +/// megamorphic site. Every live stack value first becomes a real interpreter +/// owner (as for the CALL bridge), so a throwing or reentrant setter sees an +/// exact frame. The helper consumes the value slot; the receiver is released +/// afterwards, leaving the stack below the operands unchanged. +fn emit_opt_owned_property_store( + builder: &mut cranelift_frontend::FunctionBuilder<'_>, + env: &OptEnv<'_>, + provenance: &mut [OptProvenance], + depth: usize, + pc: u32, + atom: u32, +) -> Result { + use cranelift_codegen::ir::{types, InstBuilder, MemFlags}; + use rquickjs_core::qjs; + if env.int32_loop { + return Err(CompileFailure::UnsupportedOpcode); + } + let object_index = depth + .checked_sub(2) + .ok_or(CompileFailure::InvalidArtifact)?; + let value_index = depth - 1; + if depth > env.stack.len() || depth > provenance.len() { + return Err(CompileFailure::InvalidArtifact); + } + let object_slot = opt_flat_stack_slot(env, object_index)?; + let value_slot = opt_flat_stack_slot(env, value_index)?; + for (index, vars) in env.arguments.iter().enumerate() { + let value = opt_use(builder, *vars); + opt_store(builder, env.arg_buf, index, value); + } + for (index, vars) in env.locals.iter().enumerate() { + let value = opt_use(builder, *vars); + opt_store(builder, env.var_buf, index, value); + } + for (index, vars) in env.stack.iter().take(depth).enumerate() { + let value = opt_use(builder, *vars); + opt_store(builder, env.stack_base, index, value); + } + let bytecode = builder.ins().load( + env.pointer_type, + MemFlags::new(), + env.frame, + env.layout.bytecode_start, + ); + let current_pc = builder.ins().iadd_imm(bytecode, i64::from(pc)); + builder + .ins() + .store(MemFlags::new(), current_pc, env.frame, env.layout.pc); + opt_own_stack_for_exit( + builder, + env.frame, + env.sret, + env.stack_base, + depth, + env.arguments.len() + env.locals.len(), + provenance, + env.helper_signatures, + env.pointer_type, + env.layout, + )?; + for slot in provenance.iter_mut().take(depth) { + if matches!(slot, OptProvenance::Argument(_) | OptProvenance::Local(_)) { + *slot = OptProvenance::OwnedSlot; + } + } + opt_set_stack_top( + builder, + env.frame, + env.stack_base, + depth, + env.pointer_type, + env.layout, + ); + emit_opt_helper( + builder, + env.frame, + env.sret, + env.stack_base, + depth, + env.helper_signatures, + qjs::JSJitHelperId_JS_JIT_HELPER_SET_PROPERTY as usize, + &[0, object_slot, atom, value_slot], + env.pointer_type, + env.layout, + )?; + // SET_PROPERTY left UNDEFINED in the consumed value slot; only the + // receiver owner remains above the surviving stack. + emit_opt_helper( + builder, + env.frame, + env.sret, + env.stack_base, + depth, + env.helper_signatures, + qjs::JSJitHelperId_JS_JIT_HELPER_FREE as usize, + &[0, object_slot], + env.pointer_type, + env.layout, + )?; + let undefined = OptPair { + payload: builder.ins().iconst(types::I64, 0), + tag: builder + .ins() + .iconst(types::I64, i64::from(qjs::JS_TAG_UNDEFINED)), + }; + for index in [object_index, value_index] { + opt_store(builder, env.stack_base, index, undefined); + opt_define(builder, env.stack[index], undefined); + provenance[index] = OptProvenance::Unknown; + } + opt_set_stack_top( + builder, + env.frame, + env.stack_base, + object_index, + env.pointer_type, + env.layout, + ); + Ok(object_index) +} + #[allow(clippy::too_many_arguments)] fn emit_opt_helper( builder: &mut cranelift_frontend::FunctionBuilder<'_>, diff --git a/jit/src/ir/mod.rs b/jit/src/ir/mod.rs index 9be6ab40b..dd9284f2c 100644 --- a/jit/src/ir/mod.rs +++ b/jit/src/ir/mod.rs @@ -26,11 +26,11 @@ pub use frame_state::{FrameSlot, FrameState, FrameStateId, FrameStateTable}; pub use heap::{HeapKey, HeapLocation, ScalarHeapEffect, ScalarHeapOperation}; pub(crate) use loops::LoopAnalysis; pub use optimized::{ - DeoptMap, DeoptOwnership, DeoptPhase, DeoptSlot, DeoptValidationError, GuardSite, - Materialization, MaterializedFrame, MaterializedValue, OptimizedBlock, OptimizedEffect, - OptimizedFrameShape, OptimizedIr, OptimizedMetrics, OptimizedNode, OptimizedNodeKind, - OwnedMaterializeError, OwnedMaterializedFrame, OwnedMaterializedValue, - OwnershipTransitionError, SsaValueOwnership, ValueRepresentation, + optimized_vocabulary_admits, DeoptMap, DeoptOwnership, DeoptPhase, DeoptSlot, + DeoptValidationError, GuardSite, Materialization, MaterializedFrame, MaterializedValue, + OptimizedBlock, OptimizedEffect, OptimizedFrameShape, OptimizedIr, OptimizedMetrics, + OptimizedNode, OptimizedNodeKind, OwnedMaterializeError, OwnedMaterializedFrame, + OwnedMaterializedValue, OwnershipTransitionError, SsaValueOwnership, ValueRepresentation, }; pub(crate) use property::{ eligible_property_observation, PropertyAccessPlan, PropertyBoundary, PropertyCandidate, diff --git a/jit/src/ir/optimized.rs b/jit/src/ir/optimized.rs index 635a2cbd3..85e571ab1 100644 --- a/jit/src/ir/optimized.rs +++ b/jit/src/ir/optimized.rs @@ -865,6 +865,28 @@ fn optimized_block_depths( Ok(depths) } +/// Feedback-independent part of Tier-2 admission. `false` means translation +/// rejects this function with `UnsupportedOpcode` for every feedback snapshot +/// (a reachable opcode lies outside the optimized vocabulary), so its +/// baseline artifact is the terminal native tier. +pub fn optimized_vocabulary_admits(function: &VerifiedFunction) -> bool { + let instructions = function.instructions(); + function.control_flow_graph().blocks().iter().all(|block| { + instructions + .get(block.instruction_range()) + .is_some_and(|block| { + block.iter().all(|instruction| { + let name = match instruction.opcode().name() { + "tail_call" => "call", + "tail_call_method" => "call_method", + name => name, + }; + classify_optimized_opcode(name).is_ok() + }) + }) + }) +} + fn classify_optimized_opcode( name: &str, ) -> Result<(ValueRepresentation, OptimizedEffect), CompileFailure> { diff --git a/jit/src/lib.rs b/jit/src/lib.rs index d9a15bd96..c5bb043fd 100644 --- a/jit/src/lib.rs +++ b/jit/src/lib.rs @@ -636,6 +636,8 @@ struct ProductionBackend { optimizing_snapshots: std::collections::HashMap, baseline_property_refreshed: std::collections::HashSet, + // Cached feedback-independent Tier-2 vocabulary admission per function. + optimized_vocabulary: std::collections::HashMap, tier2_sources: std::collections::HashMap, feedback: runtime::FeedbackTable, // CALL feedback consumers finish synchronously and never reenter QuickJS. @@ -2074,10 +2076,11 @@ impl ProductionBackend { optimizing_hotness: rustc_hash::FxHashMap::default(), optimizing_snapshots: std::collections::HashMap::new(), baseline_property_refreshed: std::collections::HashSet::new(), + optimized_vocabulary: std::collections::HashMap::new(), tier2_sources: std::collections::HashMap::new(), feedback: runtime::FeedbackTable::new(feedback_capacity, 3), call_feedback_types: Vec::new(), - shape_feedback: runtime::ShapeFeedbackTable::new(3), + shape_feedback: runtime::ShapeFeedbackTable::new(runtime::POLYMORPHIC_PROPERTY_LIMIT), metrics, cold_metrics_dirty: true, native_entries: 0, @@ -2201,29 +2204,76 @@ impl ProductionBackend { ); } let refresh_inputs = (self.feedback.version(), self.coordinator.installed_count()); - let refresh_scan_due = self.config.tier_policy() == JitTierPolicy::BaselineOnly - && self.last_refresh_scan != Some(refresh_inputs); + let baseline_only = self.config.tier_policy() == JitTierPolicy::BaselineOnly; + let refresh_scan_due = self.last_refresh_scan != Some(refresh_inputs); if refresh_scan_due { self.last_refresh_scan = Some(refresh_inputs); - let refreshes = self - .optimizing_snapshots - .iter() - .filter_map(|(key, snapshot)| { + // Under automatic tiering the baseline artifact is normally a + // stepping stone and keeps its feedback-free helper sites. When + // no feedback can make Tier 2 translate the function, baseline is + // its terminal native tier: refresh it once with the recorded + // (bounded polymorphic) property feedback, exactly as the + // baseline-only policy does. Never race an active Tier-2 job. + let candidates = if baseline_only { + self.optimizing_snapshots + .keys() + .copied() + .collect::>() + } else { + let mut keys = self + .optimizing_snapshots + .keys() + .chain(self.tier2_sources.keys()) + .copied() + .filter(|key| !self.baseline_property_refreshed.contains(key)) + .collect::>(); + keys.sort_unstable_by_key(|key| (key.id, key.generation)); + keys.dedup(); + keys.retain(|key| { + !matches!( + self.coordinator.tier_state(*key, runtime::Tier::Optimizing), + runtime::CompileState::Queued(_) + | runtime::CompileState::Compiling(_) + | runtime::CompileState::Ready(_) + | runtime::CompileState::Installed(_) + ) + }); + let snapshots = &self.optimizing_snapshots; + let sources = &self.tier2_sources; + keys.retain(|key| { + let admitted = *self.optimized_vocabulary.entry(*key).or_insert_with(|| { + snapshots + .get(key) + .or_else(|| sources.get(key)) + .is_none_or(ir::optimized_vocabulary_admits) + }); + !admitted + }); + keys + }; + let refreshes = candidates + .into_iter() + .filter_map(|key| { + let snapshot = self + .optimizing_snapshots + .get(&key) + .or_else(|| self.tier2_sources.get(&key))?; let feedback = self .feedback .snapshot(self.clock.max(1)) - .with_properties(self.shape_feedback.snapshot(*key)); - let call_ready = self - .coordinator - .baseline_direct_refresh_ready(*key, &feedback); - let property_ready = !self.baseline_property_refreshed.contains(key) + .with_properties(self.shape_feedback.snapshot(key)); + let call_ready = baseline_only + && self + .coordinator + .baseline_direct_refresh_ready(key, &feedback); + let property_ready = !self.baseline_property_refreshed.contains(&key) && compiler::baseline::has_baseline_property_sites(snapshot, &feedback) && matches!( - self.coordinator.tier_state(*key, runtime::Tier::Baseline), + self.coordinator.tier_state(key, runtime::Tier::Baseline), runtime::CompileState::Installed(_) ); (call_ready || property_ready) - .then(|| (*key, snapshot.clone(), feedback, property_ready)) + .then(|| (key, snapshot.clone(), feedback, property_ready)) }) .collect::>(); for (key, snapshot, feedback, property_ready) in refreshes { @@ -3420,6 +3470,8 @@ unsafe impl rquickjs_core::runtime::JitBackend for ProductionBackend { self.optimizing_hotness.remove(&key); self.optimizing_snapshots.remove(&key); self.tier2_sources.remove(&key); + self.optimized_vocabulary.remove(&key); + self.baseline_property_refreshed.remove(&key); self.entry_tiers.remove(&key); self.coordinator.retire(key); self.maintenance(); diff --git a/jit/src/runtime/mod.rs b/jit/src/runtime/mod.rs index d40bab251..73c9bf793 100644 --- a/jit/src/runtime/mod.rs +++ b/jit/src/runtime/mod.rs @@ -41,7 +41,7 @@ pub use invalidate::{DependencyError, DependencyGraph, DependencyKey}; pub use osr::{OsrKey, OsrMap}; pub use shape_feedback::{ PropertyAttributes, PrototypeDependencyToken, ShapeFeedbackSite, ShapeFeedbackState, - ShapeFeedbackTable, ShapeObservation, ShapeToken, + ShapeFeedbackTable, ShapeObservation, ShapeToken, POLYMORPHIC_PROPERTY_LIMIT, }; #[cfg(test)] diff --git a/jit/src/runtime/shape_feedback.rs b/jit/src/runtime/shape_feedback.rs index a3ac49d57..6bcbe6c78 100644 --- a/jit/src/runtime/shape_feedback.rs +++ b/jit/src/runtime/shape_feedback.rs @@ -2,6 +2,12 @@ use std::collections::BTreeMap; use super::{FunctionKey, ObservedType}; +/// Maximum number of receiver shapes one property site may record before it +/// becomes megamorphic. Both native tiers emit at most this many inline +/// shape/generation compares per site; a megamorphic site uses the generic +/// property helper instead of guarding (and deoptimizing) on a shape. +pub const POLYMORPHIC_PROPERTY_LIMIT: usize = 4; + #[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)] pub struct ShapeToken { identity: u64, diff --git a/jit/tests/polymorphic_property_ic.rs b/jit/tests/polymorphic_property_ic.rs new file mode 100644 index 000000000..a99f6c84c --- /dev/null +++ b/jit/tests/polymorphic_property_ic.rs @@ -0,0 +1,508 @@ +#![cfg(all( + feature = "compiler", + feature = "test-support", + any(target_os = "linux", target_os = "macos"), + target_endian = "little", + any(target_arch = "x86_64", target_arch = "aarch64") +))] +//! Bounded polymorphic (<= 4 shapes) property inline caches in both native +//! tiers, and the megamorphic generic-helper fallback. + +use std::time::{Duration, Instant}; + +use rquickjs::{Context, Function, Object, Runtime}; +use rquickjs_jit::{ + bytecode::VerifyLimits, + compiler::optimized::Tier2Compiler, + runtime::{ + FeedbackSnapshot, FunctionKey, ObservedType, PropertyAttributes, PrototypeDependencyToken, + ShapeFeedbackState, ShapeFeedbackTable, ShapeObservation, ShapeToken, + POLYMORPHIC_PROPERTY_LIMIT, + }, + test_support::SnapshotFixture, + Jit, JitConfig, JitTierPolicy, +}; + +unsafe extern "C" { + fn JS_JitGetHelperCount( + rt: *mut rquickjs_core::qjs::JSRuntime, + helper_id: u32, + count: *mut u64, + ) -> i32; +} + +fn helper_count(rt: *mut rquickjs_core::qjs::JSRuntime, helper_id: u32) -> u64 { + let mut count = 0; + assert_eq!( + unsafe { JS_JitGetHelperCount(rt, helper_id, &mut count) }, + 0 + ); + count +} + +fn raw_runtime(context: &Context) -> *mut rquickjs_core::qjs::JSRuntime { + context.with(|ctx| unsafe { rquickjs_core::qjs::JS_GetRuntime(ctx.as_raw().as_ptr()) }) +} + +fn observation(identity: u64, offset: u32, value: ObservedType) -> ShapeObservation { + ShapeObservation::new( + ShapeToken::new(identity, 7), + PrototypeDependencyToken::new(0, 0), + offset, + PropertyAttributes::WRITABLE, + value, + ) +} + +fn site_pc(verified: &rquickjs_jit::bytecode::VerifiedFunction, opcode: &str, nth: usize) -> u32 { + verified + .instructions() + .iter() + .filter(|instruction| instruction.opcode().name() == opcode) + .nth(nth) + .unwrap() + .pc() +} + +#[test] +fn shape_feedback_records_four_shapes_before_turning_megamorphic() { + let mut table = ShapeFeedbackTable::new(POLYMORPHIC_PROPERTY_LIMIT); + let key = FunctionKey::new(3, 1); + assert_eq!(POLYMORPHIC_PROPERTY_LIMIT, 4); + for identity in 1..=4u64 { + let state = table.observe( + key, + 9, + observation(identity * 0x1000, identity as u32, ObservedType::Int32), + ); + let expected = if identity == 1 { + ShapeFeedbackState::Monomorphic + } else { + ShapeFeedbackState::Polymorphic + }; + assert_eq!(state, expected); + } + assert_eq!(table.get(key, 9).unwrap().observations().len(), 4); + assert_eq!( + table.observe(key, 9, observation(0x5000, 5, ObservedType::Int32)), + ShapeFeedbackState::Megamorphic + ); + assert!(table.get(key, 9).unwrap().observations().is_empty()); +} + +fn lower( + source: &str, + sites: &[(&str, usize, &[ShapeObservation])], +) -> Result { + let fixture = SnapshotFixture::compile(source); + let verified = fixture.snapshot().verify(VerifyLimits::default()).unwrap(); + let key = FunctionKey::new(1, 1); + let mut table = ShapeFeedbackTable::new(POLYMORPHIC_PROPERTY_LIMIT); + for (opcode, nth, observations) in sites { + let pc = site_pc(&verified, opcode, *nth); + for observation in observations.iter() { + table.observe(key, pc, *observation); + } + } + Tier2Compiler::host(1).lower_with_feedback_for_test( + &verified, + key, + &FeedbackSnapshot::empty(1).with_properties(table.snapshot(key)), + ) +} + +#[test] +fn four_shape_tier2_chain_is_inline_without_helper_calls() { + let four = [ + observation(0x1000, 1, ObservedType::Int32), + observation(0x2000, 3, ObservedType::Int32), + observation(0x3000, 2, ObservedType::Int32), + observation(0x4000, 0, ObservedType::Int32), + ]; + let source = "(function(o){return o.answer})"; + let mono = lower(source, &[("get_field", 0, &four[..1])]).unwrap(); + let poly = lower(source, &[("get_field", 0, &four)]).unwrap(); + assert_eq!( + poly.matches("call_indirect").count(), + mono.matches("call_indirect").count(), + "{poly}" + ); + for identity in [0x1000u64, 0x2000, 0x3000, 0x4000] { + assert!( + poly.contains(&format!(", {identity}\n")) + || poly.contains(&format!(", {identity:#x}\n")), + "missing shape {identity:#x}: {poly}" + ); + } +} + +#[test] +fn megamorphic_tier2_sites_lower_to_generic_helpers_instead_of_guards() { + let five = (1..=5u64) + .map(|identity| observation(identity * 0x1000, identity as u32, ObservedType::Int32)) + .collect::>(); + let get = lower("(function(o){return o.answer})", &[("get_field", 0, &five)]) + .expect("megamorphic get uses the owning helper"); + assert!(get.contains("call_indirect"), "{get}"); + assert!(!get.contains("0x5000") && !get.contains("20480"), "{get}"); + + let put = lower( + "(function(o,v){o.answer=v;return v})", + &[("put_field", 0, &five)], + ) + .expect("megamorphic put uses the owning helper"); + assert!(put.contains("call_indirect"), "{put}"); + assert!(!put.contains("0x5000") && !put.contains("20480"), "{put}"); +} + +#[test] +fn shape_guarded_element_receivers_are_rejected_instead_of_deopting_forever() { + // Optimized element loads only publish primitives, so a shape guard on an + // element result can never hit. Such a function must stay in its current + // tier rather than install an artifact that deopts on every execution. + let source = "(function(a,i){return a[i].answer})"; + let four = [ + observation(0x1000, 1, ObservedType::Int32), + observation(0x2000, 3, ObservedType::Int32), + ]; + assert_eq!( + lower(source, &[("get_field", 0, &four)]).unwrap_err(), + rquickjs_jit::compiler::CompileFailure::UnsupportedOpcode + ); + assert_eq!( + lower(source, &[("get_field", 0, &four[..1])]).unwrap_err(), + rquickjs_jit::compiler::CompileFailure::UnsupportedOpcode + ); + // A generic site accepts any receiver and remains admissible. + let five = (1..=5u64) + .map(|identity| observation(identity * 0x1000, identity as u32, ObservedType::Int32)) + .collect::>(); + assert!(lower(source, &[("get_field", 0, &five)]).is_ok()); +} + +#[test] +fn production_element_receiver_kernel_never_enters_a_deopt_loop() { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .call_threshold(1) + .loop_threshold(1) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context + .with(|ctx| { + ctx.eval::<(), _>(format!( + "{SHAPES}\n\ + function sweep(list,n){{let s=0;for(let i=0;i("sweep(shapes,64)")) + .unwrap(), + 16 * 110 + ); + jit.poll(); + } + let metrics = jit.metrics(); + assert_eq!(metrics.deopts, 0, "{metrics:?}"); + assert_eq!(metrics.tier2_entries, 0, "{metrics:?}"); +} + +fn wait_for(jit: &Jit, mut step: impl FnMut(), done: impl Fn(&rquickjs_jit::JitMetrics) -> bool) { + let deadline = Instant::now() + Duration::from_secs(60); + loop { + step(); + jit.poll(); + if done(&jit.metrics()) { + return; + } + assert!(Instant::now() < deadline, "{:?}", jit.metrics()); + std::thread::sleep(Duration::from_micros(50)); + } +} + +const SHAPES: &str = r#" + globalThis.shapes = [ + { x: 1, y: 10 }, + { y: 20, x: 2, z: 0 }, + { tag: 0, x: 3, y: 30 }, + Object.assign(Object.create(null), { y: 40, w: 0, x: 4 }), + ]; + globalThis.extra = [ + { q: 0, r: 0, x: 5, y: 50 }, + { s: 0, x: 6, y: 60 }, + ]; +"#; + +#[test] +fn production_tier2_four_shape_site_runs_without_deopt_under_stress_gc() { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .call_threshold(64) + .force_optimized_for_test(true) + .stress_gc(true) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context + .with(|ctx| { + ctx.eval::<(), _>(format!( + "{SHAPES}\nfunction bump(o,v){{o.x=o.x+v;o.x=o.x-v;return o.x+o.y}}" + )) + }) + .unwrap(); + let call = |index: usize| { + context.with(|ctx| { + let f: Function = ctx.globals().get("bump").unwrap(); + let shapes: rquickjs::Array = ctx.globals().get("shapes").unwrap(); + let o: Object = shapes.get(index & 3).unwrap(); + let result = f.call::<_, i32>((o, 3)); + assert!(result.is_ok(), "{result:?}; catch={:?}", ctx.catch()); + assert_eq!(result.unwrap(), 11 * ((index & 3) as i32 + 1)); + }) + }; + let mut index = 0; + wait_for( + &jit, + || { + call(index); + index += 1; + }, + |metrics| metrics.tier2_entries > 0, + ); + let rt = raw_runtime(&context); + let before_shape = helper_count( + rt, + rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_SHAPE_GUARD, + ); + let before = jit.metrics(); + for index in 0..1_024 { + call(index); + jit.poll(); + } + let after = jit.metrics(); + assert!(after.tier2_entries > before.tier2_entries, "{after:?}"); + assert_eq!(after.deopts, before.deopts, "{before:?} -> {after:?}"); + assert_eq!( + helper_count( + rt, + rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_SHAPE_GUARD + ), + before_shape + ); +} + +#[test] +fn production_tier2_megamorphic_site_uses_helpers_with_exact_reentrancy() { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .call_threshold(64) + .force_optimized_for_test(true) + .stress_gc(true) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context + .with(|ctx| { + ctx.eval::<(), _>(format!( + "{SHAPES}\n\ + globalThis.all = shapes.concat(extra);\n\ + globalThis.getterCalls = 0;\n\ + globalThis.accessor = {{\n\ + get x() {{ getterCalls++; return 7; }},\n\ + set x(v) {{ if (v === 1000) throw new Error('setter'); this.stored = v; }},\n\ + y: 70\n\ + }};\n\ + function mega(o,v){{o.x=v;return o.x+o.y}}\n\ + function guarded(o,v){{try{{return mega(o,v)}}catch(e){{return -1}}}}" + )) + }) + .unwrap(); + let call = |index: usize| { + context.with(|ctx| { + let f: Function = ctx.globals().get("mega").unwrap(); + let all: rquickjs::Array = ctx.globals().get("all").unwrap(); + let o: Object = all.get(index % 6).unwrap(); + let result = f.call::<_, i32>((o, 1)); + assert!(result.is_ok(), "{result:?}; catch={:?}", ctx.catch()); + assert_eq!(result.unwrap(), 1 + 10 * ((index % 6) as i32 + 1)); + }) + }; + let mut index = 0; + wait_for( + &jit, + || { + call(index); + index += 1; + }, + |metrics| metrics.tier2_entries > 0, + ); + let before = jit.metrics(); + for index in 0..600 { + call(index); + jit.poll(); + } + let after = jit.metrics(); + assert!(after.tier2_entries > before.tier2_entries, "{after:?}"); + assert_eq!( + after.deopts, before.deopts, + "a megamorphic site must not guard or deopt on a shape: {before:?} -> {after:?}" + ); + // Accessor receivers take the reentrant generic path: the getter runs, + // and a throwing setter propagates through the owned exception exit. + let [value, thrown, calls] = context + .with(|ctx| { + ctx.eval::, _>( + "(() => { const a = mega(accessor, 5); const b = guarded(accessor, 1000);\ + return [a, b, getterCalls]; })()", + ) + }) + .unwrap()[..] + else { + panic!("three results"); + }; + assert_eq!((value, thrown), (77, -1)); + assert!(calls >= 1); + assert_eq!( + context + .with(|ctx| ctx.eval::("accessor.stored")) + .unwrap(), + 5 + ); + for index in 0..64 { + call(index); + } + runtime.run_gc(); +} + +#[test] +fn baseline_inline_dispatch_covers_four_shapes_without_helpers() { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .tier_policy(JitTierPolicy::BaselineOnly) + .call_threshold(2) + .loop_threshold(2) + .stress_gc(true) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context + .with(|ctx| { + ctx.eval::<(), _>(format!( + "{SHAPES}\n\ + function sweep(list,n){{let s=0;for(let i=0;i(format!("sweep(shapes,{n})")).unwrap()); + // Each group of four receivers contributes (2+10)+(3+20)+(4+30)+(5+40). + wait_for( + &jit, + || assert_eq!(run(8), 2 * 114), + |metrics| metrics.installed >= 2, + ); + let rt = raw_runtime(&context); + assert_eq!( + unsafe { rquickjs_core::qjs::JS_JitResetHelperCounters(rt) }, + 0 + ); + assert_eq!(run(64), 16 * 114); + for helper in [ + rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_SHAPE_GUARD, + rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_GET_PROPERTY, + rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_SET_PROPERTY, + ] { + assert_eq!(helper_count(rt, helper), 0, "helper {helper} on a hit"); + } + // An unobserved fifth layout misses every compare and takes the exact + // generic helper edge; a mutated observed layout does too. + let fifth = context + .with(|ctx| ctx.eval::("sweep([extra[0],extra[0],extra[0],extra[0]],4)")) + .unwrap(); + assert_eq!(fifth, 4 * 56); + assert!( + helper_count( + rt, + rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_GET_PROPERTY + ) > 0 + ); + let mutated = context + .with(|ctx| ctx.eval::("delete shapes[1].z; sweep(shapes,4)")) + .unwrap(); + assert_eq!(mutated, 114); + assert!(jit.metrics().native_entries > 0); +} + +#[test] +fn automatic_tiering_refreshes_a_terminal_baseline_with_property_ics() { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .call_threshold(2) + .loop_threshold(2) + .stress_gc(true) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + // The object literal is outside the Tier-2 vocabulary, so the baseline + // artifact is this function's terminal native tier. + context + .with(|ctx| { + ctx.eval::<(), _>(format!( + "{SHAPES}\n\ + function sweep(list,n){{const box={{s:0}};for(let i=0;i(format!("sweep(shapes,{n})")).unwrap()); + wait_for( + &jit, + || assert_eq!(run(8), 2 * 110), + |metrics| metrics.installed >= 2 && metrics.pending_worker_jobs == 0, + ); + let rt = raw_runtime(&context); + let settle = Instant::now() + Duration::from_secs(60); + loop { + assert_eq!( + unsafe { rquickjs_core::qjs::JS_JitResetHelperCounters(rt) }, + 0 + ); + assert_eq!(run(64), 16 * 110); + if helper_count( + rt, + rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_GET_PROPERTY, + ) == 0 + { + break; + } + jit.poll(); + assert!(Instant::now() < settle, "{:?}", jit.metrics()); + std::thread::sleep(Duration::from_micros(50)); + } + assert_eq!(jit.metrics().tier2_entries, 0, "{:?}", jit.metrics()); + assert!(jit.metrics().native_entries > 0, "{:?}", jit.metrics()); +} diff --git a/jit/tests/property_specialization.rs b/jit/tests/property_specialization.rs index e87948945..c3f58c57b 100644 --- a/jit/tests/property_specialization.rs +++ b/jit/tests/property_specialization.rs @@ -762,7 +762,7 @@ fn bounded_polymorphic_primitive_store_emits_a_guard_chain_and_raw_stores() { } #[test] -fn megamorphic_property_site_fails_closed_to_the_generic_tier() { +fn megamorphic_property_site_uses_the_generic_helper_instead_of_guards() { let fixture = SnapshotFixture::compile("(function(o){return o.answer})"); let verified = fixture.snapshot().verify(VerifyLimits::default()).unwrap(); let pc = verified @@ -786,13 +786,17 @@ fn megamorphic_property_site_fails_closed_to_the_generic_tier() { ), ); } - assert!(Tier2Compiler::host(1) + // A megamorphic site has no shape contract to guard. It lowers to the + // owning GET_PROPERTY bridge rather than failing the whole function or + // deoptimizing on every unobserved layout. + let clif = Tier2Compiler::host(1) .lower_with_feedback_for_test( &verified, key, &FeedbackSnapshot::empty(1).with_properties(table.snapshot(key)), ) - .is_err()); + .expect("megamorphic site lowers through the generic helper"); + assert!(clif.contains("call_indirect"), "{clif}"); } #[test] diff --git a/jit/tests/tier1_calls_properties.rs b/jit/tests/tier1_calls_properties.rs index 13fbd59d9..2f2a28f37 100644 --- a/jit/tests/tier1_calls_properties.rs +++ b/jit/tests/tier1_calls_properties.rs @@ -328,7 +328,7 @@ fn baseline_only_direct_call_hit_skips_call_helper_and_misses_remain_exact() { } #[test] -fn baseline_property_cache_validates_once_per_site_and_mutation_misses_exactly() { +fn baseline_property_cache_hits_inline_and_mutation_misses_exactly() { let runtime = Runtime::new().unwrap(); let jit = Jit::attach( &runtime, @@ -377,8 +377,8 @@ fn baseline_property_cache_validates_once_per_site_and_mutation_misses_exactly() rt, rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_SHAPE_GUARD ), - 3, - "each get/put site must validate once, not once per loop iteration" + 0, + "the shape/generation compare is inline; hits never cross a helper" ); assert_eq!( helper_count( @@ -401,9 +401,13 @@ fn baseline_property_cache_validates_once_per_site_and_mutation_misses_exactly() assert!( helper_count( rt, - rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_SHAPE_GUARD - ) > 3, - "shape mutation did not force revalidation/miss" + rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_GET_PROPERTY + ) > 0 + && helper_count( + rt, + rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_SET_PROPERTY + ) > 0, + "shape mutation did not take the generic helper miss edge" ); } From f260d817cefc39ddcfe94b7b0c59cc8fb0ddcd2a Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 16:54:01 +0800 Subject: [PATCH 12/54] bench(jit): add property-polymorphic kernel and P4c slice notes Add a non-designated diagnostic workload whose two read sites each see four stable shapes (one prototype-less) on the Tier-2 IC path, and document the first P4c slice, noisy diagnostics and remaining work. Co-Authored-By: Claude Opus 5.5 (1M context) --- benchmarks/README.md | 1 + benchmarks/run.rs | 7 ++ benchmarks/scripts/property-polymorphic.js | 25 ++++++ docs/roadmap/p4c-poly-ic.md | 96 ++++++++++++++++++++++ 4 files changed, 129 insertions(+) create mode 100644 benchmarks/scripts/property-polymorphic.js create mode 100644 docs/roadmap/p4c-poly-ic.md diff --git a/benchmarks/README.md b/benchmarks/README.md index d6dd3e5fc..c1802b0a0 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -161,6 +161,7 @@ fallback-only result as proof that a JIT tier supports the feature. | `strings-regexp` | String construction, slicing, RegExp capture and replacement | | `arrays-typed` | Packed-array growth/traversal and Int32Array/Float64Array traffic | | `objects-polymorphic` | Allocation, property reads/writes, and four stable shapes | +| `property-polymorphic` | Allocation-free reads at two sites that each observe four stable shapes (one prototype-less); the bounded polymorphic inline cache | | `calls-recursion-closures` | Four-deep calls, bounded recursion, and mutable closure capture | | `json-codec` | Repeated nested JSON encoding and decoding | | `map-set-bigint` | Map and Set mutation/iteration plus bounded BigInt arithmetic | diff --git a/benchmarks/run.rs b/benchmarks/run.rs index 20fe492ac..e005dbcbd 100644 --- a/benchmarks/run.rs +++ b/benchmarks/run.rs @@ -222,6 +222,13 @@ const WORKLOADS: &[Workload] = &[ designated: false, file: "objects-polymorphic.js", }, + Workload { + name: "property-polymorphic", + suite: "rquickjs-jit property diagnostics", + group: "objects-polymorphic", + designated: false, + file: "property-polymorphic.js", + }, Workload { name: "calls-recursion-closures", suite: "rquickjs-jit matrix", diff --git a/benchmarks/scripts/property-polymorphic.js b/benchmarks/scripts/property-polymorphic.js new file mode 100644 index 000000000..41d10184f --- /dev/null +++ b/benchmarks/scripts/property-polymorphic.js @@ -0,0 +1,25 @@ +// Focused polymorphic-property kernel: the `point.x`/`point.y` sites each +// observe four stable own-data shapes, one of them prototype-less. The +// receivers are a benchmark argument so allocation stays outside the measured +// function; selecting them by branch (not by array element) keeps the hot +// sites on the optimized tier's bounded polymorphic inline cache. +globalThis.workloadArgument = { + a: { x: 1, y: 10 }, + b: { y: 20, x: 2, z: 5 }, + c: { tag: 7, x: 3, y: 30 }, + d: Object.assign(Object.create(null), { y: 40, w: 9, x: 4 }), +}; + +function workload(iterations, seed, shapes) { + let sum = seed; + let point = shapes.a; + for (let i = 0; i < iterations; i++) { + const k = i & 3; + if (k === 0) point = shapes.a; + else if (k === 1) point = shapes.b; + else if (k === 2) point = shapes.c; + else point = shapes.d; + sum = sum + point.x + point.y; + } + return sum; +} diff --git a/docs/roadmap/p4c-poly-ic.md b/docs/roadmap/p4c-poly-ic.md new file mode 100644 index 000000000..bb1f17e97 --- /dev/null +++ b/docs/roadmap/p4c-poly-ic.md @@ -0,0 +1,96 @@ +# P4c: bounded polymorphic property inline caches (first slice) + +Branch `perf/p4c-poly-ic`, based on `82d3808`. Roadmap item: P4 "Tier 2 +polymorphic property IC" in +[PERFORMANCE_ROADMAP_V8_BUN.md](../PERFORMANCE_ROADMAP_V8_BUN.md). + +## What landed + +- **Feedback**: `POLYMORPHIC_PROPERTY_LIMIT = 4` + (`jit/src/runtime/shape_feedback.rs`). A get/put site records up to four + own-data shapes (prototype-less receivers included, since an own property's + prototype token is 0) before it turns megamorphic. Both tiers read the same + constant. +- **Tier 1 (baseline)**: property sites now dispatch inline on the receiver's + shape pointer and its monotonic layout generation (the exact + `JS_JitHelperShapeGuard` predicate), up to four shapes, each with its own + slot offset. The per-site stack cache and the `SHAPE_GUARD` helper crossing + (frame materialization + O(bytecode) PC validation) are gone. Misses still + take the exact `GET_PROPERTY`/`SET_PROPERTY` helper edge, never a deopt. +- **Automatic tiering, terminal baseline refresh**: when a function contains a + reachable opcode outside the Tier-2 vocabulary + (`ir::optimized_vocabulary_admits`, e.g. `object`/`define_field`/ + `array_from`), Tier 2 can never translate it. Its baseline artifact is then + refreshed once with the recorded property feedback, as the baseline-only + policy already did. This is what gives `objects-polymorphic` and the + `mixed-quotes` workload function property ICs at all under production + automatic tiering. It never races a queued/compiling/installed Tier-2 job. +- **Tier 2**: + - Chains accept four shapes. The shape/generation compare is inline for + both borrowed and frame-materialized receivers (the materialized path used + to call `SHAPE_GUARD` once per observed shape). + - Megamorphic `get_field`/`put_field` sites lower to the owning generic + helper (`GET_PROPERTY` via the audited owned-replace path; new + `emit_opt_owned_property_store` for `SET_PROPERTY` + `FREE`) instead of + failing the whole function. A shape miss therefore cannot become a + deopt/recompile loop. Reentrant getters/setters and a throwing setter are + covered under stress GC. + - A shape-guarded site whose receiver is exactly an optimized element load + is rejected at compile time: optimized `get_array_el` publishes only + primitives, so such an artifact deopts on every execution (and would then + strand the function in the slower baseline). Generic sites stay + admissible. +- **Benchmark**: `benchmarks/scripts/property-polymorphic.js` (registered as a + non-designated `rquickjs-jit property diagnostics` workload) isolates two + four-shape read sites on the Tier-2 path. + +## Tests + +`jit/tests/polymorphic_property_ic.rs` (feedback lattice at four shapes, +Tier-2 CLIF chain without extra helper calls, megamorphic helper lowering, +element-receiver rejection, production Tier 2 four-shape run without deopts, +megamorphic reentrancy/exception run, baseline inline dispatch with zero +`SHAPE_GUARD`/`GET_PROPERTY`/`SET_PROPERTY` on hits, automatic terminal +baseline refresh). Two existing tests were updated to the new contract: +`tier1_calls_properties::baseline_property_cache_hits_inline_and_mutation_misses_exactly` +and `property_specialization::megamorphic_property_site_uses_the_generic_helper_instead_of_guards`. + +## Noisy diagnostics (not publishable evidence) + +Machine shared by ~16 concurrent agents (i7-13700KF, hybrid cores make +samples bimodal). Automatic mode, median over 7–9 alternating runs of +`median(protocol.warmup_batch_ns[-16:])`; speed = 82d3808 / this branch. +Checksums match the interpreter in every run. + +| workload | 82d3808 ms | branch ms | speed | note | +| --- | ---: | ---: | ---: | --- | +| objects-polymorphic | 23.88 | 23.03 | 1.04x | baseline refresh + 4-shape inline IC; still ~0.27x the interpreter (helper-bound, see below) | +| mixed-quotes | 2.60 | 2.65 | 0.98x | tied within noise; cost is the sort-callback entry path (B2) | +| property-polymorphic (new) | 1.88 | 1.95 | 0.96x | 82d3808 rejects the 4-shape site and stays in the interpreter; the branch runs Tier 2 at the monomorphic speed (all-same-shape control: 1.95 ms on both binaries), limited by per-iteration owned `shapes.a` loads | +| property-heavy (control) | 0.099 | 0.098 | 1.01x | unchanged monomorphic Tier-2 path | + +## Remaining work + +1. **Helper-bound baseline**: `objects-polymorphic` stays ~4x slower than the + interpreter because every other operation in its loop (object literal, + `define_field`, `Dup`/`Free` of locals, `call_method`, `get_array_el`) is a + helper call paying the B1 frame validation. The IC removes 5 of ~20 helper + crossings per iteration; P1 (O(1) helper validation) is the lever. +2. **Tier-2 vocabulary**: `object`, `define_field`, `array_from`, + `push_atom_value`, `push_empty_string` keep both target workloads' hot + functions out of Tier 2 entirely (P2). +3. **Object-valued element loads in Tier 2**: `objects[i].x` needs an owned + (or borrowed-from-array) heap element result; today it is rejected up front. +4. **Transition stores**: `put_field` that adds a property never records + feedback (slow path), so Tier 2 queueing waits on it forever + (`property_feedback_pending`). Recording a transition/generic marker needs a + C feedback change (reserved patch number 0031) plus a transition IC. +5. **Deopt-to-baseline trap (pre-existing)**: a Tier-2 artifact that deopts + repeatedly for a non-property reason (e.g. a heap alias copied between + locals) leaves the function in a baseline that can be slower than the + interpreter. Raising the limit to four shapes exposes one more shape count + to this existing behavior (82d3808 shows it with three shapes: an + array-receiver kernel runs at 8.46 ms vs 1.83 ms interpreted). The + element-receiver rejection above removes the most common instance. +6. **Megamorphic probe cache**: megamorphic sites use the generic helper; a + runtime-shared (shape, atom) → slot probe cache would make them cheaper. From 8438fdd8466949efa0379f248f1264dfea60f174 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 16:54:36 +0800 Subject: [PATCH 13/54] docs: record the P1 call-path slice, diagnostics and remaining work Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p1c-call-path.md | 100 ++++++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+) create mode 100644 docs/roadmap/p1c-call-path.md diff --git a/docs/roadmap/p1c-call-path.md b/docs/roadmap/p1c-call-path.md new file mode 100644 index 000000000..87420009c --- /dev/null +++ b/docs/roadmap/p1c-call-path.md @@ -0,0 +1,100 @@ +# P1 item 4: callback-free interpreter-to-native calls (first slice) + +Branch `perf/p1c-call-path`, based on `82d3808`. Roadmap context: +[PERFORMANCE_ROADMAP_V8_BUN.md](../PERFORMANCE_ROADMAP_V8_BUN.md) B2 and P1.4. + +## What changed + +- **QuickJS patch 0023, ABI 1.25.** The backend vtable gains the optional + `entry_fast_grant` callback. After a DONE exit whose pc=0 handle QuickJS + caches, the backend may grant a bounded budget plus a backend-owned + `JSJitFastEntryState`. While the budget lasts and `state->epoch` still + equals the cached entry epoch, calls of that function reuse the cached + handle and skip call feedback, `record_hot`, `entry_cache_epoch`, + `native_enter` and `native_exit`. QuickJS counts these executions in the + shared state and sets `JS_JIT_FRAME_FAST_ENTRY` on their frames. A + non-DONE exit still calls `native_exit`, tagged + `JS_JIT_EXIT_FAST_UNPAIRED`, and is never re-cached. C-to-JS callbacks + (`Array.prototype.sort` comparators, `forEach`/`map` callbacks) use the + same path because they enter through `JS_CallInternal`. +- **The idle cache is no longer evicted by unrelated calls.** A call of a + different function that never reaches native code keeps the other + function's still-admissible handle. A native call replaces it after + returning, as before. +- **Production grant policy.** `ProductionBackend::entry_fast_grant` grants + 255 calls only to steady entries: an installed Tier 2 artifact whose + profitability trial is decided (or cannot run without baseline samples), + or an installed baseline whose optimizing tier is terminally blacklisted + (not under `BaselineOnly`, which probes on every hot event). Each renewal + is a full call, so hot counts, call/return feedback, timing, benefit + accounting and `maintenance_if_due` are sampled once per 256 calls instead + of paid on every call. The shared epoch mirrors `entry_cache_epoch`, so + every existing invalidation (maintenance, deopt, demotion, direct refresh, + detach) also stops granted calls before the next execution. +- **Exact counters.** `Jit::metrics` adds the shared entries, exits and + optimized entries, so `native_entries == native_exits` and `tier2_entries` + stay exact. `native_acquisitions` and `benefit_recordings` now count only + full calls; that is the intended semantic change (acquisitions and timing + are what the fast path removes). +- **Tier 2 candidate scan cache.** Maintenance remembers candidates it + deferred for missing property feedback, an unresolved callee, or an + incomplete numeric/element signature, keyed by the inputs of that + decision (feedback version, a new shape-feedback version, installed + artifact count, element-loop sample gate). It skips rebuilding the + feedback snapshot for them until an input changes. The shape-feedback + version ignores repeated observations and reordering. + +## Verification + +- `jit/tests/fast_entry.rs`: C ABI tests for budget expiry, epoch change, + zero epoch, unpaired non-DONE exits, suspension, malformed grants and + cache retention across interpreted calls; plus a production test where a + steady Tier 2 sort comparator runs granted with exact metrics and still + deoptimizes correctly on string scores. +- `jit_patch` covers the 1.25 header, source and bundled bindings for all + JIT targets (bindings updated by hand, including wasm32 layouts). + +## Noisy diagnostic (not publishable) + +Seven alternating processes per engine on a shared, heavily loaded host; +median over runs of the median of the last 16 warmup batches; checksums +matched the interpreter. Speed is baseline time divided by new time. + +| Workload | 82d3808 ms | branch ms | speed | +| --- | ---: | ---: | ---: | +| mixed-quotes | 2.5036 | 0.6426 | 3.90x | +| generic-call-entry | 0.0709 | 0.0688 | 1.03x | +| call-heavy | 0.1057 | 0.1041 | 1.02x | +| generic-call-fallback | 9.1043 | 8.9973 | 1.01x | +| calls-closures | 3.3230 | 3.3293 | 1.00x | +| fibonacci-recursive | 9.8675 | 9.9872 | 0.99x | +| calls-recursion-closures | 6.4353 | 6.4962 | 0.99x | +| objects-polymorphic | 23.8856 | 23.8650 | 1.00x | +| property-heavy | 0.0989 | 0.0970 | 1.02x | +| numeric | 0.0294 | 0.0300 | 0.98x | +| arrays-typed | 2.9859 | 3.0208 | 0.99x | +| collections | 1.6589 | 1.6539 | 1.00x | + +Only mixed-quotes moves outside the noise band. `generic-call-entry` and +`call-heavy` perform ten native entries per batch (their hot calls are +direct or inlined), and the closure/recursion scenarios never compile, so +they have no steady native entries to grant. + +## Remaining work + +- Grants still require one full call per 256 executions and per maintenance + epoch. A per-bytecode handle cache (instead of the single runtime slot) + would let alternating native callees all stay granted; the slot is still + replaced when two native functions alternate at the same depth. +- Baseline entries that are still Tier 2 candidates keep the full callback + path, including per-call timing and a full maintenance pass about every + 32 calls. Moving hot counting into a C-side decrementing counter (V8's + interrupt budget) would remove that too. +- Maintenance still invalidates the entry epoch on every full pass. Making + invalidation conditional on an actual admissibility change needs an + explicit coordinator state generation for installs, demotions, eviction + and dependency invalidation. +- `calls-closures`, `calls-recursion-closures` and `fibonacci-recursive` + need P2 coverage (closure opcodes) and P3 (native recursion) before this + path can help them. +- The publishable three-engine matrix for this change is pending. From 83425c6323fdab30b9a1715cec7f77397627fcf5 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 16:57:26 +0800 Subject: [PATCH 14/54] docs(roadmap): record P2a closure slice, evidence and remaining work Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p2a-closures.md | 146 +++++++++++++++++++++++++++++++++++ 1 file changed, 146 insertions(+) create mode 100644 docs/roadmap/p2a-closures.md diff --git a/docs/roadmap/p2a-closures.md b/docs/roadmap/p2a-closures.md new file mode 100644 index 000000000..6526c8b2b --- /dev/null +++ b/docs/roadmap/p2a-closures.md @@ -0,0 +1,146 @@ +# P2a closures: Tier 1 coverage slice + +Branch `perf/p2a-closures`, based on `82d3808` (0.12.9). This is the first +slice of roadmap item P2.1 in +[PERFORMANCE_ROADMAP_V8_BUN.md](../PERFORMANCE_ROADMAP_V8_BUN.md): the closure +opcodes are now covered in Tier 1. Tier 2 and native var-ref access are not +part of this slice. + +## What is admitted + +| Opcode family | Tier 1 policy | Lowering | +| --- | --- | --- | +| `fclosure`, `fclosure8` | `Helper(FClosure)` | `JS_JitHelperFClosure` → `js_closure` | +| `get_var_ref0-3`, `get_var_ref`, `get_var_ref_check` | `Helper(GetVarRef)` | `JS_JitHelperGetVarRef` (plain or TDZ-checked) | +| `put_var_ref0-3`, `put_var_ref`, `set_var_ref0-3`, `set_var_ref`, `put_var_ref_check` | `Helper(PutVarRef)` | `JS_JitHelperPutVarRef` (consumes its slot); `set_*` duplicates the value into the scratch slot first | +| `close_loc` | `Helper(CloseLocal)` | `JS_JitHelperCloseLoc` → `close_lexical_var` | +| `set_name` | `Helper(SetName)` | `JS_JitHelperSetName` → `JS_DefineObjectName` | + +`set_name` belongs to roadmap item P2.2, but QuickJS emits it right after +`fclosure` for every anonymous arrow or function expression that is assigned to a +binding, so neither target workload is eligible without it. + +The following remain rejected (`ClosureFrame`): + +- `put_var_ref_check_init`: QuickJS emits it only for a derived constructor's + `this` that an arrow initializes through `super()`, and those arrows also + contain `get_super`, which is still rejected. No Tier 1 program can run it, + so it has no opcode-case manifest entry. +- `make_var_ref`, `make_var_ref_ref`: these only feed + `get_ref_value`/`put_ref_value`, which are still rejected. + +The C side is `sys/patches/0025-tier1-closures.patch`: ABI 1.25 and runtime +API 1.10, with five helpers appended to the end of the helper table (IDs +24-28). + +## Semantic safety argument + +- **Captured slots alias the frame.** An attached `JSVarRef.pvalue` points into + `sf->var_buf` or `sf->arg_buf`. Tier 1 keeps arguments and locals in SSA + variables and publishes all of them before every helper call and poll. In a + function that contains `fclosure`, it also reloads them after every helper + and poll, so a write that a closure made during a call becomes visible to + later native code, and a stale copy never overwrites it at the next publish. + Only `fclosure` can capture this frame's slots: `eval`, `with`, `arguments`, + `make_*_ref` and `define_class` are all still rejected. Native arithmetic + keeps its use-site tag guards, so a closure that changes a slot's type + deoptimizes; the regression test for this passes. +- **TDZ proofs.** Tier 1 proves `get_loc_check`, `put_loc_check` and + `put_loc_check_init` statically. A closure can only move a lexical slot from + uninitialized to initialized (through `put_var_ref_check_init`), never back. + The proofs for `get_loc_check` and `put_loc_check` therefore stay sound: a + slot the analysis believes uninitialized can only make the function retry. + A function that combines `fclosure` with `put_loc_check_init` is rejected + with `ClosureFrame`. +- **Ownership.** FCLOSURE and GET_VAR_REF write an owned value to an empty + output slot. PUT_VAR_REF consumes its input on success and on a TDZ + exception. After a failed PUT_VAR_REF, the exception exit publishes only the + bytecode-visible stack below the consumed operand. SET_NAME borrows. Every + helper runs the stress-GC points. +- **Other tiers.** Tier 2's opcode classifier still rejects all of these + opcodes. Frame inlining rejects callees that contain `fclosure` or + `close_loc`. Scalar or tagged leaf entries and effect-free inlining still + require `closure_count == 0` and a closed operation set. OSR is still refused + when `closure_count != 0`. + +Evidence: + +- `jit/tests/tier1_closures.rs` has 13 tests. +- `opcode-cases.json` has 21 new cases, each executed at its native PC under + stress GC. +- The helper-family cases are in `differential.rs`. +- With the reload turned off, the three reload and type-change tests fail, and + the stress-GC heap test and the per-iteration binding test do not report. + +## Noisy diagnostic (not publishable) + +Setup: + +- Host: i7-13700KF, 24 logical CPUs, shared with about 15 other build and + bench agents. +- Baseline binary: `82d3808` `target/release/jit-bench`. +- New binary: this branch. +- Command: `jit-bench worker --mode automatic --script + benchmarks/scripts/W.js`. +- Sampling: 7 alternating base/new processes. Each process contributes the + median of the last 16 `warmup_batch_ns` values; the table reports the + median over processes. +- Checksums match interpreter mode for every run. + +Speed is base/new, so values above 1.00x mean this branch is faster. + +| Workload | Base ms | New ms | Speed vs base | New vs interpreter | +| --- | ---: | ---: | ---: | ---: | +| calls-closures | 3.304 | 3.275 | 1.01x (tied) | 1.00x | +| calls-recursion-closures | 6.452 | 8.723 | **0.74x** | 0.75x | +| exceptions-promises-async | 3.702 | 3.623 | 1.02x (tied) | 0.65x (unchanged `catch` blocker) | +| call-heavy (control) | 0.060 | 0.062 | 0.97x (bimodal load noise) | 25.6x | + +`calls-closures`: `workload` is now Tier 1 eligible. Its Tier 2 attempts fail +on the closure opcodes, and the function then runs in the interpreter, so the +result is neutral. + +`calls-recursion-closures`: `workload` now runs its loop in Tier 1. Every +iteration crosses the generic helper bridge about 9 times: GET_GLOBAL ×2, DUP, +CALL ×2, and FREE on the callee operands. Each crossing pays the full +`qjsjit_validate_helper_frame_state` check, which took about 19% of samples in +a gdb poor-man's profile. This is not specific to closures. The equivalent +closure-free program below runs its loop in Tier 1 on unmodified `82d3808`, at +**0.78x** the interpreter's speed (automatic 8.50 ms vs interpreter 6.65 ms, +7 pairs). Admitting closures only exposes this target to that existing Tier 1 +call-bridge cost. The fixes are roadmap P1 (O(1) helper validation, cheaper +entry bookkeeping) and P3 (native call convention). Re-measure after those +land. + +```js +function recursiveSum(n) { return n <= 0 ? 0 : n + recursiveSum(n - 1); } +function call4(fn, value) { return fn(value); } +function call3(fn, value) { return call4(fn, value); } +function call2(fn, value) { return call3(fn, value); } +function call1(fn, value) { return call2(fn, value); } +globalThis.state = { captured: 0 }; +function update(value) { return (state.captured = (state.captured + value) | 0); } +function workload(iterations, seed) { + state.captured = seed; + for (let i = 0; i < iterations; i++) call1(update, recursiveSum(i & 7)); + return state.captured; +} +``` + +## Remaining work + +1. **Tier 2 var refs.** Classify `get_var_ref*` as a tagged heap load and + `put_var_ref*` as a heap store with exact effects, lower them through + `emit_opt_owned_helper_push` or a native load, and model a closure-creating + function's captured locals as memory (reload after every reentrant node) + instead of SSA values. Tier 2 has no scalar path for this yet. +2. **Native var-ref access.** Add a versioned layout descriptor for + `JSObject.u.func.var_refs` and `JSVarRef.pvalue` so loads and stores need + no helper, and GET_VAR_REF becomes one guarded dependent load. +3. **Reload only captured slots.** Expose `vardefs[].is_captured` in the + snapshot and reload only captured slots, instead of every argument and + local. +4. `put_var_ref_check_init`, once `get_super` and derived constructors are + admitted. +5. Make the generic call bridge (P1/P3) cheaper before expecting the ≥1.5x + interpreter-speed P2 acceptance on the closure workloads. From fec2b7b6fd5affd68d3aa94d14f796cede62f009 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:00:58 +0800 Subject: [PATCH 15/54] perf(jit): native Tier 1 exception regions and async probe shutdown Admit throw, throw_error, catch, nip_catch, gosub and ret in Tier 1. - Verifier: model catch handler entries (caught value replaces the catch offset; locals joined over the whole try region), gosub return offsets and ret successors resolved to the owning finally block's return points; record the innermost live catch offset per PC. - Tier 1: every helper exception edge inside a try region publishes the exact owned stack and enters a per-handler landing pad that calls JS_JitCatchException (interpreter exception prologue incl. backtrace); a caught exception resumes the handler natively, anything else (uncatchable, invariant failure) exits for exact interpreter unwinding. throw/throw_error use JS_JitThrowValue/JS_JitThrowError. - Patch 0027 (ABI 1.25): the three exception entry points, plus generator/async/eval/with snapshot refusals now disable hot probes locally instead of re-requesting a snapshot at every call/resume/loop poll (the exceptions-promises-async regression). - Tier 2, inlining and direct leaf links reject exception regions up front; OSR validation checks catch offsets by their real tag. - Add the exceptions-sync benchmark scenario and regression tests. Co-Authored-By: Claude Opus 5.5 (1M context) --- benchmarks/run.rs | 9 + benchmarks/scripts/exceptions-sync.js | 28 + jit/src/abi.rs | 4 +- jit/src/bytecode/cfg.rs | 87 ++- jit/src/bytecode/mod.rs | 2 +- jit/src/bytecode/policy_audit.rs | 12 +- jit/src/bytecode/stack.rs | 220 ++++++++ jit/src/bytecode/verify.rs | 49 +- jit/src/code_cache/artifact.rs | 1 + jit/src/compiler/baseline.rs | 496 +++++++++++++++++- jit/src/compiler/optimized.rs | 18 + jit/src/compiler/tagged_call_link.rs | 1 + jit/src/ir/baseline.rs | 132 ++++- jit/src/ir/inlining.rs | 2 + jit/src/ir/mod.rs | 2 +- jit/src/ir/types.rs | 10 + jit/src/lib.rs | 12 +- jit/tests/abi.rs | 2 +- jit/tests/fixtures/opcode-cases.json | 60 +++ jit/tests/opcodes.rs | 10 +- jit/tests/tier1_exceptions.rs | 170 ++++++ jit/tests/verifier.rs | 42 +- sys/build_support/patch.rs | 7 +- .../0027-tier1-exception-regions.patch | 201 +++++++ sys/src/bindings/aarch64-apple-darwin.rs | 22 +- sys/src/bindings/aarch64-pc-windows-msvc.rs | 22 +- sys/src/bindings/aarch64-unknown-linux-gnu.rs | 22 +- .../bindings/aarch64-unknown-linux-musl.rs | 22 +- sys/src/bindings/wasm32-wasip1.rs | 22 +- sys/src/bindings/x86_64-apple-darwin.rs | 22 +- sys/src/bindings/x86_64-pc-windows-gnu.rs | 22 +- sys/src/bindings/x86_64-pc-windows-msvc.rs | 22 +- sys/src/bindings/x86_64-unknown-linux-gnu.rs | 22 +- sys/src/bindings/x86_64-unknown-linux-musl.rs | 22 +- 34 files changed, 1743 insertions(+), 54 deletions(-) create mode 100644 benchmarks/scripts/exceptions-sync.js create mode 100644 jit/tests/tier1_exceptions.rs create mode 100644 sys/patches/0027-tier1-exception-regions.patch diff --git a/benchmarks/run.rs b/benchmarks/run.rs index 20fe492ac..142a7c5f6 100644 --- a/benchmarks/run.rs +++ b/benchmarks/run.rs @@ -250,6 +250,13 @@ const WORKLOADS: &[Workload] = &[ designated: false, file: "exceptions-promises-async.js", }, + Workload { + name: "exceptions-sync", + suite: "rquickjs-jit matrix", + group: "exceptions-sync", + designated: false, + file: "exceptions-sync.js", + }, ]; #[derive(Clone, Debug, Deserialize, Serialize)] @@ -1412,6 +1419,7 @@ mod tests { "scalar-expressions.js", "scalar-control-flow.js", "exceptions-promises-async.js", + "exceptions-sync.js", "packed-array-traversal.js", "int32array-traversal.js", "float64array-traversal.js", @@ -1467,6 +1475,7 @@ mod tests { ("json-codec", "json-codec"), ("map-set-bigint", "map-set-bigint"), ("exceptions-promises-async", "exceptions-promises-async"), + ("exceptions-sync", "exceptions-sync"), ] { let workload = WORKLOADS.iter().find(|w| w.name == name).unwrap(); assert_eq!(workload.group, group); diff --git a/benchmarks/scripts/exceptions-sync.js b/benchmarks/scripts/exceptions-sync.js new file mode 100644 index 000000000..2027c368c --- /dev/null +++ b/benchmarks/scripts/exceptions-sync.js @@ -0,0 +1,28 @@ +// Synchronous exception regions: a primitive throw caught every 16th +// iteration, a finally block on every iteration, and a helper-raised +// TypeError (property read of null) on one of every 256 iterations. +function workload(iterations, seed) { + const table = [{ w: 1 }, { w: 2 }, null, { w: 3 }]; + let caught = 0; + let total = seed | 0; + let finals = 0; + let missing = 0; + for (let i = 0; i < iterations; i++) { + try { + if ((i & 15) === 0) throw i; + total = (total + i) | 0; + } catch (value) { + caught += value; + } finally { + finals++; + } + if ((i & 63) === 1) { + try { + total = (total + table[(i >> 6) & 3].w) | 0; + } catch (error) { + missing++; + } + } + } + return caught + ':' + total + ':' + finals + ':' + missing; +} diff --git a/jit/src/abi.rs b/jit/src/abi.rs index b39c65a23..c763539af 100644 --- a/jit/src/abi.rs +++ b/jit/src/abi.rs @@ -5,7 +5,7 @@ use core::{fmt, mem}; use rquickjs_core::qjs; pub const ABI_MAJOR: u16 = 1; -pub const ABI_MINOR: u16 = 24; +pub const ABI_MINOR: u16 = 25; pub const SOURCE_REVISION: u64 = 0xfd0a_0210_b7be_0095; pub const OPCODE_FINGERPRINT: u64 = qjs::QJSJIT_GENERATED_OPCODE_FINGERPRINT; @@ -1073,7 +1073,7 @@ mod inline_api_tests { fn inline_recovery_requires_the_matching_main_abi_minor() { let native = AbiInfo::query_linked().unwrap(); assert_eq!(ABI_MINOR, native.minor()); - assert_eq!(native.minor(), 24); + assert_eq!(native.minor(), 25); assert!(unsafe { qjs::JS_JitGetInlineAPI(0) }.is_null()); } diff --git a/jit/src/bytecode/cfg.rs b/jit/src/bytecode/cfg.rs index 6c05f1b01..77c13eba6 100644 --- a/jit/src/bytecode/cfg.rs +++ b/jit/src/bytecode/cfg.rs @@ -51,7 +51,18 @@ impl ControlFlowGraph { } fn is_unconditional(instruction: &Instruction) -> bool { - matches!(instruction.opcode().name(), "goto" | "goto8" | "goto16") + // `gosub` transfers to its finally block; control returns to the next + // instruction only through that block's `ret`, never by fallthrough. + matches!( + instruction.opcode().name(), + "goto" | "goto8" | "goto16" | "gosub" + ) +} + +/// `ret` has no static target. Its successors are the return points of the +/// `gosub` sites that enter the finally block containing it. +fn is_subroutine_return(instruction: &Instruction) -> bool { + instruction.opcode().name() == "ret" } fn is_terminal(instruction: &Instruction) -> bool { @@ -69,6 +80,7 @@ fn is_terminal(instruction: &Instruction) -> bool { pub(crate) fn has_valid_exit(instruction: &Instruction) -> bool { is_terminal(instruction) + || is_subroutine_return(instruction) || (instruction.branch_target().is_some() && is_unconditional(instruction)) } @@ -107,7 +119,9 @@ pub(crate) fn build( if next_pc < byte_len { boundaries.insert(next_pc as u32); } - } else if is_terminal(instruction) && next_pc < byte_len { + } else if (is_terminal(instruction) || is_subroutine_return(instruction)) + && next_pc < byte_len + { boundaries.insert(next_pc as u32); } } @@ -146,7 +160,7 @@ pub(crate) fn build( successors.push(next); } } - } else if !is_terminal(tail) { + } else if !is_terminal(tail) && !is_subroutine_return(tail) { if let Some(next) = next { successors.push(next); } @@ -160,14 +174,79 @@ pub(crate) fn build( successors, }); } - let by_pc = blocks + let by_pc: BTreeMap = blocks .iter() .enumerate() .map(|(index, block)| (block.start_pc, index)) .collect(); + link_subroutine_returns(instructions, &mut blocks, &by_pc)?; Ok(ControlFlowGraph { blocks, by_pc, loop_headers, }) } + +/// Resolves every `ret` to the return points of the finally block that owns +/// it. The owner is found by a walk from each `gosub` target in which nested +/// `gosub` sites step over their finally block to their own return point, so +/// a `ret` is attributed only to the finally block that reaches it directly. +/// A `ret` reachable from two different finally entries has no exact static +/// return set and is rejected. +fn link_subroutine_returns( + instructions: &[Instruction], + blocks: &mut [BasicBlock], + by_pc: &BTreeMap, +) -> Result<(), VerifyError> { + let unsupported = |pc| VerifyError::new(pc, VerifyErrorKind::UnsupportedExceptionRegion); + let return_point = |instruction: &Instruction| { + (instruction.pc() as usize) + .checked_add(instruction.size()) + .and_then(|pc| u32::try_from(pc).ok()) + .filter(|pc| by_pc.contains_key(pc)) + .ok_or_else(|| unsupported(instruction.pc())) + }; + let mut return_points: BTreeMap> = BTreeMap::new(); + for block in blocks.iter() { + let tail = &instructions[block.instruction_range.end - 1]; + if tail.opcode().name() == "gosub" { + let target = *block + .successors + .first() + .ok_or_else(|| unsupported(tail.pc()))?; + return_points + .entry(target) + .or_default() + .insert(return_point(tail)?); + } + } + let mut owners: BTreeMap = BTreeMap::new(); + for &entry in return_points.keys() { + let mut visited = BTreeSet::new(); + let mut pending = vec![entry]; + while let Some(pc) = pending.pop() { + if !visited.insert(pc) { + continue; + } + let index = *by_pc.get(&pc).ok_or_else(|| unsupported(pc))?; + let block = &blocks[index]; + let tail = &instructions[block.instruction_range.end - 1]; + match tail.opcode().name() { + "ret" => { + if owners + .insert(index, entry) + .is_some_and(|owner| owner != entry) + { + return Err(unsupported(tail.pc())); + } + } + "gosub" => pending.push(return_point(tail)?), + _ => pending.extend(block.successors.iter().copied()), + } + } + } + for (index, owner) in owners { + blocks[index].successors = return_points[&owner].iter().copied().collect(); + } + Ok(()) +} diff --git a/jit/src/bytecode/mod.rs b/jit/src/bytecode/mod.rs index 7cb5a0abd..162ff2714 100644 --- a/jit/src/bytecode/mod.rs +++ b/jit/src/bytecode/mod.rs @@ -18,7 +18,7 @@ pub use policy::{ audited_opcode_policy_table, tier1_policy, AuditedOpcodePolicy, FallbackReason, HelperId, Tier1Policy, Tier1Rejection, GENERATED_OPCODE_COUNT, GENERATED_OPCODE_FINGERPRINT, }; -pub use stack::SlotKind; +pub use stack::{ExceptionHandler, SlotKind}; pub use verify::{Resource, VerifiedFunction, VerifyError, VerifyErrorKind, VerifyLimits}; /// Numeric opcode constants generated from QuickJS's authoritative macro table. diff --git a/jit/src/bytecode/policy_audit.rs b/jit/src/bytecode/policy_audit.rs index 05612e6af..10ae87465 100644 --- a/jit/src/bytecode/policy_audit.rs +++ b/jit/src/bytecode/policy_audit.rs @@ -49,8 +49,8 @@ pub(super) const AUDITED_POLICIES: [AuditedOpcodePolicy; 252] = [ AuditedOpcodePolicy { id: 45, name: "check_brand", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 46, name: "add_brand", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 47, name: "return_async", policy: Tier1Policy::Reject(FallbackReason::Async) }, - AuditedOpcodePolicy { id: 48, name: "throw", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, - AuditedOpcodePolicy { id: 49, name: "throw_error", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 48, name: "throw", policy: Tier1Policy::Native }, + AuditedOpcodePolicy { id: 49, name: "throw_error", policy: Tier1Policy::Native }, AuditedOpcodePolicy { id: 50, name: "eval", policy: Tier1Policy::Reject(FallbackReason::DirectEval) }, AuditedOpcodePolicy { id: 51, name: "apply_eval", policy: Tier1Policy::Reject(FallbackReason::DirectEval) }, AuditedOpcodePolicy { id: 52, name: "regexp", policy: Tier1Policy::Helper(HelperId::Regexp) }, @@ -108,10 +108,10 @@ pub(super) const AUDITED_POLICIES: [AuditedOpcodePolicy; 252] = [ AuditedOpcodePolicy { id: 104, name: "if_false", policy: Tier1Policy::Helper(HelperId::ToBool) }, AuditedOpcodePolicy { id: 105, name: "if_true", policy: Tier1Policy::Helper(HelperId::ToBool) }, AuditedOpcodePolicy { id: 106, name: "goto", policy: Tier1Policy::Native }, - AuditedOpcodePolicy { id: 107, name: "catch", policy: Tier1Policy::Reject(FallbackReason::ExceptionRegion) }, - AuditedOpcodePolicy { id: 108, name: "gosub", policy: Tier1Policy::Reject(FallbackReason::ExceptionRegion) }, - AuditedOpcodePolicy { id: 109, name: "ret", policy: Tier1Policy::Reject(FallbackReason::ExceptionRegion) }, - AuditedOpcodePolicy { id: 110, name: "nip_catch", policy: Tier1Policy::Reject(FallbackReason::ExceptionRegion) }, + AuditedOpcodePolicy { id: 107, name: "catch", policy: Tier1Policy::Native }, + AuditedOpcodePolicy { id: 108, name: "gosub", policy: Tier1Policy::Native }, + AuditedOpcodePolicy { id: 109, name: "ret", policy: Tier1Policy::Native }, + AuditedOpcodePolicy { id: 110, name: "nip_catch", policy: Tier1Policy::Native }, AuditedOpcodePolicy { id: 111, name: "check_object", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 112, name: "to_object", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 113, name: "to_propkey", policy: Tier1Policy::Helper(HelperId::ToPropertyKey) }, diff --git a/jit/src/bytecode/stack.rs b/jit/src/bytecode/stack.rs index a2582c9ed..5d9801992 100644 --- a/jit/src/bytecode/stack.rs +++ b/jit/src/bytecode/stack.rs @@ -20,6 +20,50 @@ pub enum SlotKind { pub(crate) struct AbstractState { pub(crate) locals: Vec, pub(crate) stack: Vec, + /// Live catch offsets, innermost last: the operand-stack index of each + /// `CatchOffset` slot and its handler (`None` for iterator close offsets, + /// which the interpreter unwinds through without resuming). + pub(crate) catches: Vec<(usize, Option)>, +} + +/// Where an exception raised while executing an instruction resumes. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub struct ExceptionHandler { + catch_index: u16, + handler_pc: Option, +} + +impl ExceptionHandler { + /// Operand-stack index of the innermost live catch offset. + pub const fn catch_index(self) -> u16 { + self.catch_index + } + + /// Handler PC of that catch offset; `None` for an iterator close offset. + pub const fn handler_pc(self) -> Option { + self.handler_pc + } +} + +impl AbstractState { + /// The innermost live catch offset, which the interpreter's exception + /// unwinding reaches first. + fn innermost_catch(&self) -> Option { + self.catches.last().and_then(|(index, handler_pc)| { + Some(ExceptionHandler { + catch_index: u16::try_from(*index).ok()?, + handler_pc: *handler_pc, + }) + }) + } + + /// The innermost catch offset that resumes a handler in this frame. + fn innermost_handler(&self) -> Option<(usize, u32)> { + self.catches + .iter() + .rev() + .find_map(|(index, handler_pc)| handler_pc.map(|pc| (*index, pc))) + } } impl AbstractState { @@ -69,6 +113,8 @@ pub(crate) struct StateProof { pub(crate) before: BTreeMap, pub(crate) after: BTreeMap, pub(crate) visited: BTreeSet, + /// Innermost live catch offset before each instruction inside a region. + pub(crate) handlers: BTreeMap, } fn effective_pop(instruction: &Instruction) -> usize { @@ -203,8 +249,44 @@ fn transfer( } let name = instruction.opcode().name(); + if name == "nip_catch" { + // `catch_offset ... value -> value`: the interpreter releases every + // operand down to the innermost catch offset, whatever its depth. + let value = state.stack.pop().ok_or_else(|| { + VerifyError::new( + instruction.pc(), + VerifyErrorKind::StackUnderflow { + needed: 1, + available: 0, + }, + ) + })?; + let Some((index, _)) = state.catches.pop() else { + return Err(VerifyError::new( + instruction.pc(), + VerifyErrorKind::UnsupportedExceptionRegion, + )); + }; + if state.stack.get(index) != Some(&SlotKind::CatchOffset) { + return Err(VerifyError::new( + instruction.pc(), + VerifyErrorKind::UnsupportedExceptionRegion, + )); + } + state.stack.truncate(index); + state.stack.push(value); + return check_stack_size(snapshot, instruction, state); + } let popped = state.stack.split_off(state.stack.len() - pop); let popped_top = popped.last().copied().unwrap_or(SlotKind::Tagged); + // Any catch offset consumed by this instruction is no longer live. + while state + .catches + .last() + .is_some_and(|(index, _)| *index >= state.stack.len()) + { + state.catches.pop(); + } if name == "get_loc0_loc1" { state.stack.push(state.locals[0]); @@ -233,6 +315,28 @@ fn transfer( state .stack .extend([SlotKind::Tagged, SlotKind::Tagged, SlotKind::CatchOffset]); + state.catches.push((state.stack.len() - 1, None)); + return check_stack_size(snapshot, instruction, state); + } + + if name == "catch" { + let target = instruction + .branch_target() + .and_then(|target| u32::try_from(target).ok()) + .ok_or_else(|| { + VerifyError::new( + instruction.pc(), + VerifyErrorKind::UnsupportedExceptionRegion, + ) + })?; + state.stack.push(SlotKind::CatchOffset); + state.catches.push((state.stack.len() - 1, Some(target))); + return check_stack_size(snapshot, instruction, state); + } + + if name == "gosub" { + // The return address is an Int32 bytecode offset consumed by `ret`. + state.stack.push(SlotKind::Int32); return check_stack_size(snapshot, instruction, state); } @@ -289,6 +393,12 @@ fn merge_state( }, )); } + if expected.catches != actual.catches { + return Err(VerifyError::new( + pc, + VerifyErrorKind::UnsupportedExceptionRegion, + )); + } // Operand-stack representations must agree exactly. Unlike locals, these // are transient values consumed directly by the following bytecode. A // merge is not an operation and therefore cannot silently insert the @@ -355,6 +465,7 @@ pub(crate) fn prove( before: BTreeMap::new(), after: BTreeMap::new(), visited: BTreeSet::new(), + handlers: BTreeMap::new(), }); } let before_points: BTreeSet = snapshot @@ -396,12 +507,14 @@ pub(crate) fn prove( AbstractState { locals: vec![SlotKind::Tagged; snapshot.local_count() as usize], stack: Vec::new(), + catches: Vec::new(), }, ); let mut queue = VecDeque::from([0_u32]); let mut before = BTreeMap::new(); let mut after = BTreeMap::new(); let mut visited = BTreeSet::new(); + let mut handlers = BTreeMap::new(); while let Some(block_pc) = queue.pop_front() { let block = cfg.block(block_pc).expect("CFG successor names a block"); @@ -415,13 +528,83 @@ pub(crate) fn prove( before.insert(instruction.pc(), state.clone()); } visited.insert(instruction.pc()); + if let Some(handler) = state.innermost_catch() { + handlers.insert(instruction.pc(), handler); + } + // Any instruction inside a try region may transfer to its + // handler with the locals it observed on entry. + if let Some((_, handler_pc)) = state.innermost_handler() { + budget.charge(instruction.pc(), state.locals.len())?; + let Some(expected) = block_entries.get_mut(&handler_pc) else { + return Err(VerifyError::new( + instruction.pc(), + VerifyErrorKind::UnsupportedExceptionRegion, + )); + }; + if join_handler_locals(handler_pc, expected, &state.locals)? { + queue.push_back(handler_pc); + } + } transfer(snapshot, instruction, &mut state)?; if after_points.contains(&instruction.pc()) { budget.charge(instruction.pc(), state.cell_count())?; after.insert(instruction.pc(), state.clone()); } } + let tail = &instructions[block.instruction_range()] + .last() + .expect("CFG blocks are non-empty"); + let catch_target = (tail.opcode().name() == "catch") + .then(|| tail.branch_target()) + .flatten() + .and_then(|target| u32::try_from(target).ok()); + // A handler must be distinct from the protected fallthrough, which + // continues with the catch offset itself on the stack. + if catch_target.is_some_and(|target| { + usize::try_from(target).ok() == (tail.pc() as usize).checked_add(tail.size()) + }) { + return Err(VerifyError::new( + tail.pc(), + VerifyErrorKind::UnsupportedExceptionRegion, + )); + } for successor in block.successors() { + if catch_target == Some(*successor) { + // The interpreter resumes a handler with the caught value in + // place of its catch offset and that offset no longer live. + let mut incoming = state.clone(); + let caught = incoming.stack.last_mut().filter(|kind| { + **kind == SlotKind::CatchOffset + && incoming + .catches + .last() + .is_some_and(|(_, handler)| *handler == Some(*successor)) + }); + let Some(caught) = caught else { + return Err(VerifyError::new( + tail.pc(), + VerifyErrorKind::UnsupportedExceptionRegion, + )); + }; + *caught = SlotKind::Tagged; + incoming.catches.pop(); + budget.charge(*successor, incoming.cell_count())?; + if let Some(expected) = block_entries.get_mut(successor) { + if expected.stack != incoming.stack || expected.catches != incoming.catches { + return Err(VerifyError::new( + *successor, + VerifyErrorKind::UnsupportedExceptionRegion, + )); + } + if join_handler_locals(*successor, expected, &incoming.locals)? { + queue.push_back(*successor); + } + } else { + block_entries.insert(*successor, incoming); + queue.push_back(*successor); + } + continue; + } let mut incoming = state.clone(); if cfg.is_loop_header(*successor) { // Establish the conservative frame-domain fixed point before @@ -452,5 +635,42 @@ pub(crate) fn prove( before, after, visited, + handlers, }) } + +/// Joins the locals observed at one exceptional edge into a handler entry. +/// Unlike an ordinary merge, a handler is reached from every instruction of +/// its try region, including those before a block-scoped binding in the +/// region is initialized. That binding is out of scope in the handler, so an +/// initialized/uninitialized disagreement conservatively becomes `Tagged`, +/// which proves nothing about the cell's contents. +fn join_handler_locals( + pc: u32, + expected: &mut AbstractState, + actual: &[SlotKind], +) -> Result { + if expected.locals.len() != actual.len() { + return Err(VerifyError::new( + pc, + VerifyErrorKind::UnsupportedExceptionRegion, + )); + } + let mut changed = false; + for (expected, actual) in expected.locals.iter_mut().zip(actual) { + if expected == actual { + continue; + } + if matches!(*expected, SlotKind::CatchOffset) || matches!(*actual, SlotKind::CatchOffset) { + return Err(VerifyError::new( + pc, + VerifyErrorKind::UnsupportedExceptionRegion, + )); + } + if *expected != SlotKind::Tagged { + *expected = SlotKind::Tagged; + changed = true; + } + } + Ok(changed) +} diff --git a/jit/src/bytecode/verify.rs b/jit/src/bytecode/verify.rs index 24070edf6..a9cb1f3b2 100644 --- a/jit/src/bytecode/verify.rs +++ b/jit/src/bytecode/verify.rs @@ -119,9 +119,37 @@ pub struct VerifiedFunction { instructions: Vec, cfg: ControlFlowGraph, osr_points: Box<[super::OsrPoint]>, + exception_handlers: std::collections::BTreeMap, + unreachable: std::collections::BTreeSet, } impl VerifiedFunction { + /// False for dead instructions the verifier admitted without a proof + /// (only after a finally block's `ret`). + pub fn is_reachable(&self, pc: u32) -> bool { + !self.unreachable.contains(&pc) + } + + /// The innermost live catch offset when the instruction at `pc` starts, + /// i.e. where the interpreter's unwinding of an exception raised by that + /// instruction first stops. `None` outside every try region. + pub fn exception_handler(&self, pc: u32) -> Option { + self.exception_handlers.get(&pc).copied() + } + + /// True when any instruction executes with a live catch offset or the + /// bytecode contains a try/finally transfer. Consumers that cannot model + /// exceptional control flow (inlining, direct leaf calls) must reject it. + pub fn has_exception_regions(&self) -> bool { + !self.exception_handlers.is_empty() + || self.instructions.iter().any(|instruction| { + matches!( + instruction.opcode().name(), + "catch" | "nip_catch" | "gosub" | "ret" | "throw" | "throw_error" + ) + }) + } + pub fn snapshot(&self) -> &CompileSnapshot { &self.snapshot } @@ -387,12 +415,21 @@ pub(crate) fn verify( limits.max_basic_blocks, )?; let proof = stack::prove(&snapshot, &instructions, &cfg, limits.max_work_units)?; - if let Some(instruction) = instructions + // QuickJS emits the enclosing loop's continuation after a finally block's + // `ret` even when every path leaves through the return points. Only + // functions with finally blocks may carry such dead code; it has no proof + // and consumers never translate it. + let has_subroutines = instructions .iter() - .find(|instruction| !proof.visited.contains(&instruction.pc())) - { + .any(|instruction| matches!(instruction.opcode().name(), "gosub" | "ret")); + let unreachable: std::collections::BTreeSet = instructions + .iter() + .map(Instruction::pc) + .filter(|pc| !proof.visited.contains(pc)) + .collect(); + if let Some(pc) = unreachable.first().copied().filter(|_| !has_subroutines) { return Err(VerifyError::new( - instruction.pc(), + pc, VerifyErrorKind::UnreachableInstruction, )); } @@ -408,7 +445,7 @@ pub(crate) fn verify( .blocks() .iter() .map(|block| block.start_pc()) - .filter(|pc| cfg.is_loop_header(*pc)) + .filter(|pc| cfg.is_loop_header(*pc) && !unreachable.contains(pc)) .map(|pc| { let state = proof .before @@ -423,5 +460,7 @@ pub(crate) fn verify( instructions, cfg, osr_points, + exception_handlers: proof.handlers, + unreachable, }) } diff --git a/jit/src/code_cache/artifact.rs b/jit/src/code_cache/artifact.rs index 204607301..fdbaba946 100644 --- a/jit/src/code_cache/artifact.rs +++ b/jit/src/code_cache/artifact.rs @@ -659,6 +659,7 @@ impl CompiledArtifact { if function.instructions().len() <= 128 && snapshot.retained_bytes() <= 16 * 1024 && snapshot.exception_map().is_empty() + && !function.has_exception_regions() && snapshot.function_id() == self.key.function_id && snapshot.generation() == self.key.generation && snapshot.source_revision() == self.key.source_revision diff --git a/jit/src/compiler/baseline.rs b/jit/src/compiler/baseline.rs index 00e7cfb64..96d71801e 100644 --- a/jit/src/compiler/baseline.rs +++ b/jit/src/compiler/baseline.rs @@ -91,6 +91,9 @@ struct HelperLowering<'a> { signatures: &'a [cranelift_codegen::ir::SigRef], pointer_type: cranelift_codegen::ir::Type, layout: FrameLayout, + /// Landing pad of the innermost try region around the instruction being + /// lowered; `None` exits to the interpreter with the pending exception. + exception_target: std::cell::Cell>, } impl HelperLowering<'_> { @@ -123,6 +126,7 @@ impl HelperLowering<'_> { arguments, self.pointer_type, self.layout, + self.exception_target.get(), ) } @@ -2026,9 +2030,66 @@ fn analyze_entry_domains(ir: &BaselineIr) -> Result { + let mut transient = frame.clone(); + *transient + .locals + .get_mut(usize::from(index)) + .ok_or(CompileFailure::InvalidArtifact)? = + AbstractValue::known(KnownKind::Undefined); + successors.push(exceptional_input(&transient, handler)?); + } + IrOp::PutArgument { index, keep: true } => { + let mut transient = frame.clone(); + *transient + .arguments + .get_mut(usize::from(index)) + .ok_or(CompileFailure::InvalidArtifact)? = + AbstractValue::known(KnownKind::Undefined); + successors.push(exceptional_input(&transient, handler)?); + } + _ => {} + } + } match &instruction.op { IrOp::Poll { .. } | IrOp::OsrLabel { state: _ } | IrOp::Nop => {} IrOp::Push(value) => frame.stack.push(AbstractValue::from_tagged(*value)), + IrOp::Catch(_) => frame.stack.push(AbstractValue::known(KnownKind::Other)), + IrOp::NipCatch => { + let value = frame.stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + frame.stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + frame.stack.push(value); + } + IrOp::Throw => { + frame.stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + terminated = true; + } + IrOp::ThrowError { .. } => terminated = true, + IrOp::Gosub { target, .. } => { + frame.stack.push(AbstractValue::known(KnownKind::Number)); + successors.push((*target, frame.clone())); + terminated = true; + } + IrOp::Ret { targets } => { + frame.stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + for target in targets.iter() { + successors.push((*target, frame.clone())); + } + terminated = true; + } // Constant-pool descriptors are pointer-free but intentionally // do not carry the full runtime value. Keep their abstract // domain unknown: the lowering performs the exact tag guard @@ -2321,6 +2382,11 @@ fn analyze_entry_domains(ir: &BaselineIr) -> Result terminated = true, } + if let Some(handler) = handler { + if frame.stack.len() >= usize::from(handler.catch_index) { + successors.push(exceptional_input(&frame, handler)?); + } + } if terminated { break; } @@ -2356,6 +2422,29 @@ fn analyze_entry_domains(ir: &BaselineIr) -> Result Result<(u32, AbstractFrame), CompileFailure> { + let catch_index = usize::from(handler.catch_index); + let mut stack = frame + .stack + .get(..catch_index) + .ok_or(CompileFailure::InvalidArtifact)? + .to_vec(); + stack.push(AbstractValue::unknown()); + Ok(( + handler.handler_pc, + AbstractFrame { + arguments: frame.arguments.clone(), + locals: frame.locals.clone(), + stack, + }, + )) +} + /// `helper_backed` is true for the plain `Unary` form, whose non-numeric /// operands take the exact `UNARY_ARITH_SLOW` helper path instead of /// requiring a numeric entry proof. The post-increment and local forms @@ -2586,7 +2675,16 @@ fn lower_function( signatures: &helper_signatures, pointer_type, layout, + exception_target: std::cell::Cell::new(None), }; + // One landing pad per handler: every exceptional edge of its try region + // publishes the exact owned stack and enters it. + let landing_blocks: BTreeMap = ir + .catch_sites + .keys() + .map(|handler_pc| (*handler_pc, builder.create_block())) + .collect(); + let fixed_slots = usize::from(ir.argument_count) + usize::from(ir.local_count); macro_rules! invoke_helper { ($helper_id:expr, $state:expr, $live_depth:expr, $arguments:expr) => {{ @@ -2622,6 +2720,15 @@ fn lower_function( ) { previous_effectful_op_was_boolean = ir_op_produces_boolean(&instruction.op); } + let exception_target = match ir.exception_handlers.get(&instruction.pc) { + Some(handler) => Some( + *landing_blocks + .get(&handler.handler_pc) + .ok_or(CompileFailure::InvalidArtifact)?, + ), + None => None, + }; + helper_lowering.exception_target.set(exception_target); builder.set_srcloc(SourceLoc::default()); if !matches!(&instruction.op, IrOp::Poll { .. }) { if let Some(state) = instruction.frame_state { @@ -3529,6 +3636,185 @@ fn lower_function( .brif(taken, blocks[&target], &[], blocks[&fallthrough], &[]); terminated = true; } + IrOp::Catch(handler_pc) => { + let offset = constant_pair( + builder, + TaggedValue::new(u64::from(handler_pc), qjs::JS_TAG_CATCH_OFFSET as i64), + ); + define_pair( + builder, + *stack.get(depth).ok_or(CompileFailure::InvalidArtifact)?, + offset, + ); + depth += 1; + } + IrOp::NipCatch => { + // `catch_offset value -> value`; a catch offset owns no + // reference, so moving the value over it releases nothing. + let value_index = depth + .checked_sub(1) + .ok_or(CompileFailure::InvalidArtifact)?; + let catch_index = depth + .checked_sub(2) + .ok_or(CompileFailure::InvalidArtifact)?; + let value = use_pair(builder, stack[value_index]); + define_pair(builder, stack[catch_index], value); + clear_pair(builder, stack[value_index], stack_base, value_index, layout)?; + depth = value_index; + set_visible_stack_depth(builder, frame, stack_base, depth, layout)?; + } + IrOp::Throw | IrOp::ThrowError { .. } => { + let state = helper_states + .next() + .ok_or(CompileFailure::InvalidArtifact)?; + let frame_state = ir.frame_states.get(state); + let visible_depth = frame_state + .slots + .len() + .checked_sub(fixed_slots) + .ok_or(CompileFailure::InvalidArtifact)?; + materialize_frame( + builder, + frame, + arg_buf, + var_buf, + stack_base, + &arguments, + &locals, + &stack, + depth, + visible_depth, + frame_state.pc, + pointer_type, + layout, + )?; + let exception_depth = match instruction.op { + IrOp::ThrowError { atom, kind } => { + emit_exception_api_call( + builder, + frame, + qjs::JS_JitThrowError as *const () as usize, + state, + &[atom, u32::from(kind)], + pointer_type, + )?; + depth + } + _ => { + let value_index = depth + .checked_sub(1) + .ok_or(CompileFailure::InvalidArtifact)?; + emit_exception_api_call( + builder, + frame, + qjs::JS_JitThrowValue as *const () as usize, + state, + &[flat_stack_slot(ir, value_index)?], + pointer_type, + )?; + // The operand is the pending exception now. + clear_pair( + builder, + stack[value_index], + stack_base, + value_index, + layout, + )?; + value_index + } + }; + // Both calls always leave an exception pending. + emit_exception_edge( + builder, + frame, + sret, + stack_base, + exception_depth, + exception_target, + pointer_type, + layout, + )?; + terminated = true; + } + IrOp::Gosub { target, return_pc } => { + let return_offset = constant_pair( + builder, + TaggedValue::new(i64::from(return_pc) as u64, qjs::JS_TAG_INT as i64), + ); + define_pair( + builder, + *stack.get(depth).ok_or(CompileFailure::InvalidArtifact)?, + return_offset, + ); + builder.ins().jump( + *blocks.get(&target).ok_or(CompileFailure::InvalidArtifact)?, + &[], + ); + terminated = true; + } + IrOp::Ret { ref targets } => { + let index = depth + .checked_sub(1) + .ok_or(CompileFailure::InvalidArtifact)?; + let value = use_pair(builder, stack[index]); + clear_pair(builder, stack[index], stack_base, index, layout)?; + let is_int = tag_is(builder, value.tag, qjs::JS_TAG_INT); + let offset = builder.ins().ireduce(types::I32, value.payload); + let unknown = builder.create_block(); + for target in targets.iter() { + let block = *blocks.get(target).ok_or(CompileFailure::InvalidArtifact)?; + let matches = builder.ins().icmp_imm( + IntCC::Equal, + offset, + i64::from( + i32::try_from(*target) + .map_err(|_| CompileFailure::InvalidArtifact)?, + ), + ); + let hit = builder.ins().band(is_int, matches); + let next = builder.create_block(); + builder.ins().brif(hit, block, &[], next, &[]); + builder.seal_block(next); + builder.switch_to_block(next); + } + builder.ins().jump(unknown, &[]); + // Only this function's `gosub` sites push return offsets; + // anything else is resumed by the interpreter's own `ret`. + builder.seal_block(unknown); + builder.switch_to_block(unknown); + builder.set_cold_block(unknown); + define_pair(builder, stack[index], value); + materialize_frame( + builder, + frame, + arg_buf, + var_buf, + stack_base, + &arguments, + &locals, + &stack, + depth, + depth, + instruction.pc, + pointer_type, + layout, + )?; + let bytecode = builder.ins().load( + pointer_type, + MemFlags::new(), + frame, + layout.bytecode_start, + ); + let resume = builder.ins().iadd_imm(bytecode, i64::from(instruction.pc)); + emit_exit( + builder, + sret, + qjs::JSJitExitKind_JS_JIT_EXIT_DEOPT, + Some(resume), + pointer_type, + ); + terminated = true; + } IrOp::Return => { let result_index = depth .checked_sub(1) @@ -3612,6 +3898,63 @@ fn lower_function( } } + // Landing pads run the interpreter's exception prologue on the published + // frame. A caught exception resumes the handler natively with the caught + // value in the catch offset's slot; anything else (uncatchable errors, + // invariant failures) leaves through the ordinary exception exit so the + // interpreter unwinds the exact published frame. + for (handler_pc, landing) in &landing_blocks { + let site = *ir + .catch_sites + .get(handler_pc) + .ok_or(CompileFailure::InvalidArtifact)?; + let catch_index = usize::from(site.catch_index); + let handler = *blocks + .get(handler_pc) + .ok_or(CompileFailure::InvalidArtifact)?; + builder.set_srcloc(SourceLoc::default()); + builder.switch_to_block(*landing); + builder.set_cold_block(*landing); + let status = emit_exception_api_call( + builder, + frame, + qjs::JS_JitCatchException as *const () as usize, + site.state, + &[flat_stack_slot(ir, catch_index)?, *handler_pc], + pointer_type, + )?; + let caught = builder.ins().icmp_imm(IntCC::Equal, status, 0); + let resume = builder.create_block(); + let propagate = builder.create_block(); + builder.ins().brif(caught, resume, &[], propagate, &[]); + builder.seal_block(resume); + builder.seal_block(propagate); + builder.switch_to_block(propagate); + builder.set_cold_block(propagate); + emit_exit( + builder, + sret, + qjs::JSJitExitKind_JS_JIT_EXIT_EXCEPTION, + None, + pointer_type, + ); + builder.switch_to_block(resume); + reload_pair( + builder, + *stack + .get(catch_index) + .ok_or(CompileFailure::InvalidArtifact)?, + stack_base, + catch_index, + layout, + ); + let undefined = constant_pair(builder, TaggedValue::new(0, qjs::JS_TAG_UNDEFINED as i64)); + for pair in stack.iter().skip(catch_index + 1) { + define_pair(builder, *pair, undefined); + } + builder.ins().jump(handler, &[]); + } + builder.set_srcloc(SourceLoc::default()); builder.switch_to_block(retry); emit_exit( @@ -5700,6 +6043,7 @@ fn emit_helper_call( arguments: &[Value], pointer_type: cranelift_codegen::ir::Type, layout: FrameLayout, + exception_target: Option, ) -> Result<(), CompileFailure> { let signature = *signatures .get(helper_id) @@ -5735,18 +6079,49 @@ fn emit_helper_call( builder.seal_block(exception); builder.seal_block(continuation); builder.switch_to_block(exception); - force_visible_stack_depth(builder, frame, stack_base, exception_depth, layout)?; - emit_exit( + emit_exception_edge( builder, + frame, sret, - qjs::JSJitExitKind_JS_JIT_EXIT_EXCEPTION, - None, + stack_base, + exception_depth, + exception_target, pointer_type, - ); + layout, + )?; builder.switch_to_block(continuation); Ok(()) } +/// Publishes the exact operand stack owned at a pending exception, then +/// either enters the innermost try region's landing pad or exits so the +/// interpreter unwinds the frame. +#[allow(clippy::too_many_arguments)] +fn emit_exception_edge( + builder: &mut FunctionBuilder<'_>, + frame: Value, + sret: Value, + stack_base: Value, + exception_depth: usize, + exception_target: Option, + pointer_type: cranelift_codegen::ir::Type, + layout: FrameLayout, +) -> Result<(), CompileFailure> { + force_visible_stack_depth(builder, frame, stack_base, exception_depth, layout)?; + if let Some(landing) = exception_target { + builder.ins().jump(landing, &[]); + } else { + emit_exit( + builder, + sret, + qjs::JSJitExitKind_JS_JIT_EXIT_EXCEPTION, + None, + pointer_type, + ); + } + Ok(()) +} + #[allow(clippy::too_many_arguments)] fn invoke_frame_helper( builder: &mut FunctionBuilder<'_>, @@ -5768,6 +6143,7 @@ fn invoke_frame_helper( helper_arguments: &[u32], pointer_type: cranelift_codegen::ir::Type, layout: FrameLayout, + exception_target: Option, ) -> Result<(), CompileFailure> { let frame_state = ir.frame_states.get(state); let fixed_slots = usize::from(ir.argument_count) + usize::from(ir.local_count); @@ -5815,9 +6191,62 @@ fn invoke_frame_helper( &values, pointer_type, layout, + exception_target, ) } +/// Calls one of the ABI 1.25 exception entry points (`JS_JitThrowValue`, +/// `JS_JitThrowError`, `JS_JitCatchException`). They share the helper +/// convention: frame, stack-map id and u32 operands in, status out. The call +/// carries its frame state's source location so the state maps to exactly +/// this call's return address. +fn emit_exception_api_call( + builder: &mut FunctionBuilder<'_>, + frame: Value, + address: usize, + state: FrameStateId, + operands: &[u32], + pointer_type: cranelift_codegen::ir::Type, +) -> Result { + let mut signature = Signature::new(builder.func.signature.call_conv); + signature.params.push(AbiParam::new(pointer_type)); + signature.params.push(AbiParam::new(types::I32)); + for _ in operands { + signature.params.push(AbiParam::new(types::I32)); + } + signature.returns.push(AbiParam::new(types::I32)); + let signature = builder.import_signature(signature); + let target = builder.ins().iconst( + pointer_type, + i64::try_from(address).map_err(|_| CompileFailure::InvalidArtifact)?, + ); + let mut params = Vec::with_capacity(operands.len() + 2); + params.push(frame); + params.push(helper_u32( + builder, + u32::try_from(state.index()).map_err(|_| CompileFailure::ResourceLimit)?, + )); + params.extend( + operands + .iter() + .copied() + .map(|operand| helper_u32(builder, operand)), + ); + let source_location = frame_state_source_loc(state)?; + builder.set_srcloc(source_location); + let call = emit_external_call( + builder, + signature, + target, + ¶ms, + pointer_type, + Some(frame), + Some(source_location), + ); + builder.set_srcloc(SourceLoc::default()); + Ok(builder.inst_results(call)[0]) +} + fn helper_u32(builder: &mut FunctionBuilder<'_>, value: u32) -> Value { builder.ins().iconst(types::I32, i64::from(value)) } @@ -6471,6 +6900,8 @@ mod tests { max_stack_depth: 16, argument_count: 4, local_count: 4, + exception_handlers: BTreeMap::new(), + catch_sites: BTreeMap::new(), } } @@ -6515,6 +6946,12 @@ mod tests { IrOp::Branch { .. } => "branch", IrOp::Return => "return", IrOp::ReturnUndefined => "return_undefined", + IrOp::Catch(_) => "catch", + IrOp::NipCatch => "nip_catch", + IrOp::Throw => "throw", + IrOp::ThrowError { .. } => "throw_error", + IrOp::Gosub { .. } => "gosub", + IrOp::Ret { .. } => "ret", } } @@ -6604,7 +7041,50 @@ mod tests { linear_ir(vec![numeric_push()], IrOp::AddLocal(0)), linear_ir(vec![numeric_push()], IrOp::Return), linear_ir(Vec::new(), IrOp::ReturnUndefined), + linear_ir(Vec::new(), IrOp::Catch(1)), + linear_ir( + vec![instruction(0, IrOp::Catch(1)), numeric_push()], + IrOp::NipCatch, + ), + linear_ir(vec![numeric_push()], IrOp::Throw), + linear_ir(Vec::new(), IrOp::ThrowError { atom: 1, kind: 2 }), ]; + cases.push(BaselineIr { + blocks: vec![ + IrBlock { + start_pc: 0, + stack_depth: 0, + instructions: vec![instruction( + 0, + IrOp::Gosub { + target: 2, + return_pc: 1, + }, + )], + }, + IrBlock { + start_pc: 1, + stack_depth: 0, + instructions: vec![instruction(1, IrOp::ReturnUndefined)], + }, + IrBlock { + start_pc: 2, + stack_depth: 1, + instructions: vec![instruction( + 2, + IrOp::Ret { + targets: Box::new([1]), + }, + )], + }, + ], + frame_states: FrameStateTable::default(), + max_stack_depth: 1, + argument_count: 0, + local_count: 0, + exception_handlers: BTreeMap::new(), + catch_sites: BTreeMap::new(), + }); for operation in [ StackOp::Nip, StackOp::Nip1, @@ -6687,6 +7167,8 @@ mod tests { max_stack_depth: 0, argument_count: 0, local_count: 0, + exception_handlers: BTreeMap::new(), + catch_sites: BTreeMap::new(), }); cases.push(BaselineIr { blocks: vec![ @@ -6719,6 +7201,8 @@ mod tests { max_stack_depth: 1, argument_count: 0, local_count: 0, + exception_handlers: BTreeMap::new(), + catch_sites: BTreeMap::new(), }); let mut seen = BTreeSet::new(); @@ -6739,7 +7223,7 @@ mod tests { } assert_eq!( seen.len(), - 33, + 39, "every IrOp variant is represented: {seen:?}" ); } diff --git a/jit/src/compiler/optimized.rs b/jit/src/compiler/optimized.rs index afc40c273..7863b16a8 100644 --- a/jit/src/compiler/optimized.rs +++ b/jit/src/compiler/optimized.rs @@ -9262,6 +9262,15 @@ impl Compiler for Tier2Compiler { #[cfg(feature = "test-support")] let _diagnostic_scope = Tier2DiagnosticScope::enter(request.artifact_key().runtime_id); let key = request.key(); + if request.snapshot().has_exception_regions() { + #[cfg(feature = "test-support")] + record_tier2_stage( + key, + Tier2CompileStage::Admission, + Some(CompileFailure::UnsupportedOpcode), + ); + return Err(CompileFailure::UnsupportedOpcode); + } if request.tier() != Tier::Optimizing { #[cfg(feature = "test-support")] record_tier2_stage( @@ -9428,6 +9437,15 @@ impl Compiler for Tier2Compiler { let _diagnostic_scope = Tier2DiagnosticScope::enter(request.artifact_key().runtime_id); let key = request.key(); tier2_stage(key, Tier2CompileStage::Admission, control.check())?; + if request.snapshot().has_exception_regions() { + #[cfg(feature = "test-support")] + record_tier2_stage( + key, + Tier2CompileStage::Admission, + Some(CompileFailure::UnsupportedOpcode), + ); + return Err(CompileFailure::UnsupportedOpcode); + } if request.tier() != Tier::Optimizing { #[cfg(feature = "test-support")] record_tier2_stage( diff --git a/jit/src/compiler/tagged_call_link.rs b/jit/src/compiler/tagged_call_link.rs index 62873fa68..ae53f06a5 100644 --- a/jit/src/compiler/tagged_call_link.rs +++ b/jit/src/compiler/tagged_call_link.rs @@ -46,6 +46,7 @@ pub(crate) fn lower_target_only_linked_leaf( || signature.function().generation != snapshot.generation() || signature.arity() != usize::from(snapshot.arg_count()) || !snapshot.exception_map().is_empty() + || function.has_exception_regions() || function.control_flow_graph().blocks().len() != 1 || !matches!( signature.result(), diff --git a/jit/src/ir/baseline.rs b/jit/src/ir/baseline.rs index 68e7aa362..49ba110b0 100644 --- a/jit/src/ir/baseline.rs +++ b/jit/src/ir/baseline.rs @@ -52,6 +52,23 @@ pub struct IrBlock { pub instructions: Vec, } +/// The handler an exception raised by an instruction resumes in this frame. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub struct IrExceptionHandler { + /// Operand-stack index of the innermost live catch offset. + pub catch_index: u16, + /// First PC of the handler block, entered with the caught value at + /// `catch_index` and stack depth `catch_index + 1`. + pub handler_pc: u32, +} + +/// The CATCH call of one handler's landing pad. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub struct IrCatchSite { + pub catch_index: u16, + pub state: FrameStateId, +} + #[derive(Clone, Debug, Eq, PartialEq)] pub struct BaselineIr { pub blocks: Vec, @@ -59,6 +76,10 @@ pub struct BaselineIr { pub max_stack_depth: u16, pub argument_count: u16, pub local_count: u16, + /// Innermost handler of every instruction PC inside a try region. + pub exception_handlers: BTreeMap, + /// One landing-pad CATCH call per handler PC. + pub catch_sites: BTreeMap, } impl BaselineIr { @@ -85,12 +106,61 @@ impl BaselineIr { let block_depths = block_depths(function)?; let snapshot = function.snapshot(); let mut states = FrameStateTable::default(); + let mut exception_handlers = BTreeMap::new(); + for instruction in function.instructions() { + let Some(handler) = function.exception_handler(instruction.pc()) else { + continue; + }; + // An iterator-close offset is unwound by the interpreter itself; + // native dispatch only resumes ordinary catch handlers. + let handler_pc = handler + .handler_pc() + .ok_or(CompileFailure::UnsupportedOpcode)?; + exception_handlers.insert( + instruction.pc(), + IrExceptionHandler { + catch_index: handler.catch_index(), + handler_pc, + }, + ); + } + let mut catch_sites = BTreeMap::new(); + for handler in exception_handlers.values() { + if catch_sites.contains_key(&handler.handler_pc) { + continue; + } + if block_depths.get(&handler.handler_pc).copied() + != Some(usize::from(handler.catch_index) + 1) + { + return Err(CompileFailure::InvalidArtifact); + } + let state = record_state( + &mut states, + snapshot.arg_count(), + snapshot.local_count(), + usize::from(handler.catch_index) + 1, + FrameStateKind::Helper, + handler.handler_pc, + )?; + catch_sites.insert( + handler.handler_pc, + IrCatchSite { + catch_index: handler.catch_index, + state, + }, + ); + } let mut blocks = Vec::with_capacity(function.control_flow_graph().blocks().len()); let mut max_stack_depth = 0_usize; let mut emitted_since_poll = 0_usize; let loop_bodies = loop_body_blocks(function); for (block_index, block) in function.control_flow_graph().blocks().iter().enumerate() { + // Dead code after a finally block's `ret` has no predecessor and + // no proof; it is never lowered. + if !function.is_reachable(block.start_pc()) { + continue; + } let mut depth = *block_depths .get(&block.start_pc()) .ok_or(CompileFailure::InvalidArtifact)?; @@ -240,7 +310,26 @@ impl BaselineIr { }); } - let op = translate_instruction(instruction)?; + let op = match instruction.opcode().name() { + // The verifier resolved each `ret` to the exact return + // points of its finally block. + "ret" => IrOp::Ret { + targets: block.successors().to_vec().into_boxed_slice(), + }, + // Values between the kept operand and its catch offset + // would need ordered releases; only the direct form is + // lowered. + "nip_catch" + if exception_handlers + .get(&pc) + .map(|handler| usize::from(handler.catch_index) + 2) + == Some(depth) => + { + IrOp::NipCatch + } + "nip_catch" => return Err(CompileFailure::UnsupportedOpcode), + _ => translate_instruction(instruction)?, + }; let depth_before = depth; let helper_call_count = operation_helper_call_count(&op); let frame_state = if operation_may_exit(&op) && helper_call_count == 0 { @@ -259,7 +348,7 @@ impl BaselineIr { if depth < pop { return Err(CompileFailure::InvalidArtifact); } - let next_depth = depth - pop + instruction.opcode().n_push() as usize; + let next_depth = depth - pop + effective_push(instruction); max_stack_depth = max_stack_depth.max(next_depth).max(depth); let helper_depth = helper_stack_depth(&op, depth, next_depth)?; max_stack_depth = max_stack_depth.max(helper_depth); @@ -381,6 +470,8 @@ impl BaselineIr { .map_err(|_| CompileFailure::ResourceLimit)?, argument_count: snapshot.arg_count(), local_count: snapshot.local_count(), + exception_handlers, + catch_sites, }) } } @@ -433,6 +524,7 @@ fn operation_may_exit(operation: &IrOp) -> bool { | IrOp::Branch { .. } | IrOp::Return | IrOp::ReturnUndefined + | IrOp::Ret { .. } ) } @@ -492,6 +584,7 @@ fn operation_helper_call_count(operation: &IrOp) -> usize { | BinaryOp::StrictNotEqual, ) => 1, IrOp::Branch { .. } => 1, + IrOp::Throw | IrOp::ThrowError { .. } => 1, _ => 0, } } @@ -571,6 +664,12 @@ fn effective_pop(instruction: &Instruction) -> usize { } } +/// `gosub` is declared without stack effects in the QuickJS opcode table but +/// pushes the Int32 return offset that its finally block's `ret` consumes. +fn effective_push(instruction: &Instruction) -> usize { + instruction.opcode().n_push() as usize + usize::from(instruction.opcode().name() == "gosub") +} + fn block_depths(function: &VerifiedFunction) -> Result, CompileFailure> { let cfg = function.control_flow_graph(); let mut depths = BTreeMap::from([(0_u32, 0_usize)]); @@ -583,7 +682,7 @@ fn block_depths(function: &VerifiedFunction) -> Result, Com if depth < pop { return Err(CompileFailure::InvalidArtifact); } - depth = depth - pop + instruction.opcode().n_push() as usize; + depth = depth - pop + effective_push(instruction); } for &successor in block.successors() { match depths.get(&successor) { @@ -814,6 +913,33 @@ fn translate_instruction(instruction: &Instruction) -> Result IrOp::Return, "return_undef" => IrOp::ReturnUndefined, + "catch" => IrOp::Catch( + u32::try_from( + instruction + .branch_target() + .ok_or(CompileFailure::InvalidArtifact)?, + ) + .map_err(|_| CompileFailure::InvalidArtifact)?, + ), + "throw" => IrOp::Throw, + "throw_error" => IrOp::ThrowError { + atom: instruction.operand_u32(1), + kind: instruction.operand_u8(5), + }, + "gosub" => IrOp::Gosub { + target: u32::try_from( + instruction + .branch_target() + .ok_or(CompileFailure::InvalidArtifact)?, + ) + .map_err(|_| CompileFailure::InvalidArtifact)?, + return_pc: instruction + .pc() + .checked_add( + u32::try_from(instruction.size()).map_err(|_| CompileFailure::ResourceLimit)?, + ) + .ok_or(CompileFailure::ResourceLimit)?, + }, _ => return Err(CompileFailure::UnsupportedOpcode), }; Ok(operation) diff --git a/jit/src/ir/inlining.rs b/jit/src/ir/inlining.rs index dfa1d947e..35270e02d 100644 --- a/jit/src/ir/inlining.rs +++ b/jit/src/ir/inlining.rs @@ -168,6 +168,7 @@ impl FrameInlineCallee { || snapshot.retained_bytes() > 16 * 1024 || snapshot.closure_count() != 0 || !snapshot.exception_map().is_empty() + || self.body.has_exception_regions() || self.body.control_flow_graph().blocks().len() > 16 || self.body.control_flow_graph().blocks().iter().any(|block| { block @@ -327,6 +328,7 @@ impl InlineCallee { || snapshot.local_count() != 0 || snapshot.closure_count() != 0 || !snapshot.exception_map().is_empty() + || self.body.has_exception_regions() || self.body.instructions().len() > 128 { return None; diff --git a/jit/src/ir/mod.rs b/jit/src/ir/mod.rs index 9be6ab40b..ae78f8e2a 100644 --- a/jit/src/ir/mod.rs +++ b/jit/src/ir/mod.rs @@ -19,7 +19,7 @@ mod types; #[cfg(feature = "test-support")] pub(crate) use baseline::with_execution_trace; pub(crate) use baseline::MAX_HELPER_SCRATCH_SLOTS; -pub use baseline::{BaselineIr, IrBlock, IrInstruction}; +pub use baseline::{BaselineIr, IrBlock, IrCatchSite, IrExceptionHandler, IrInstruction}; pub(crate) use facts::KnownFacts; pub(crate) use frame_state::FrameStateKind; pub use frame_state::{FrameSlot, FrameState, FrameStateId, FrameStateTable}; diff --git a/jit/src/ir/types.rs b/jit/src/ir/types.rs index 1ab34caf1..8e540f127 100644 --- a/jit/src/ir/types.rs +++ b/jit/src/ir/types.rs @@ -73,6 +73,10 @@ pub enum BinaryOp { StrictNotEqual, } +/// Exception regions: `Catch` pushes the catch offset that resumes the +/// handler at its PC, `NipCatch` is the direct `nip_catch` form, `Throw` and +/// `ThrowError` raise, `Gosub` pushes the Int32 return offset and enters a +/// finally block and `Ret` resumes at one of its static return points. #[derive(Clone, Debug, Eq, PartialEq)] pub enum IrOp { Poll { state: FrameStateId, kind: PollKind }, @@ -114,6 +118,12 @@ pub enum IrOp { Branch { target: u32, when_true: bool }, Return, ReturnUndefined, + Catch(u32), + NipCatch, + Throw, + ThrowError { atom: u32, kind: u8 }, + Gosub { target: u32, return_pc: u32 }, + Ret { targets: Box<[u32]> }, } #[derive(Clone, Copy, Debug, Eq, PartialEq)] diff --git a/jit/src/lib.rs b/jit/src/lib.rs index d9a15bd96..076f1d72c 100644 --- a/jit/src/lib.rs +++ b/jit/src/lib.rs @@ -1281,7 +1281,8 @@ fn validate_production_frame( }; let valid = match kind { SlotKind::Tagged => true, - SlotKind::Int32 | SlotKind::CatchOffset => value.tag == i64::from(qjs::JS_TAG_INT), + SlotKind::Int32 => value.tag == i64::from(qjs::JS_TAG_INT), + SlotKind::CatchOffset => value.tag == i64::from(qjs::JS_TAG_CATCH_OFFSET), SlotKind::Float64 => value.tag == i64::from(qjs::JS_TAG_FLOAT64), SlotKind::Uninitialized => value.tag == i64::from(qjs::JS_TAG_UNINITIALIZED), }; @@ -2246,6 +2247,15 @@ impl ProductionBackend { if self.feedback_disabled.contains(key) { return false; } + // Tier 2 has no exceptional control flow; exception regions + // stay on their Tier 1 code instead of failing admission. + if self + .optimizing_snapshots + .get(key) + .is_some_and(bytecode::VerifiedFunction::has_exception_regions) + { + return false; + } if self .profitability_backoff .get(key) diff --git a/jit/tests/abi.rs b/jit/tests/abi.rs index bb3580e33..231b608b0 100644 --- a/jit/tests/abi.rs +++ b/jit/tests/abi.rs @@ -574,7 +574,7 @@ fn helper_abi_is_one_canonical_versioned_table_in_c_bindgen_and_rust() { fn helper_abi_fields_are_append_only_tails() { use rquickjs_core::qjs; - assert_eq!(qjs::QJSJIT_ABI_MINOR, 24); + assert_eq!(qjs::QJSJIT_ABI_MINOR, 25); assert_eq!(qjs::QJSJIT_RUNTIME_API_MAJOR, 1); assert_eq!(qjs::QJSJIT_RUNTIME_API_MINOR, 9); assert_eq!(qjs::QJSJIT_HELPER_ABI_VERSION, 1); diff --git a/jit/tests/fixtures/opcode-cases.json b/jit/tests/fixtures/opcode-cases.json index 3a6818cfd..250f3993e 100644 --- a/jit/tests/fixtures/opcode-cases.json +++ b/jit/tests/fixtures/opcode-cases.json @@ -1431,6 +1431,66 @@ "ownership-gc", "coercion-reentrancy" ] + }, + { + "opcode": "throw", + "definition": "function f(a){ if (a) throw {code: a}; return 'none' }", + "expression": "(()=>{let caught;try{f(7)}catch(e){caught=e.code}return [f(0),caught]})()", + "helper": null, + "dimensions": [ + "normal", + "exception-or-nonthrow" + ] + }, + { + "opcode": "throw_error", + "definition": "function f(a){ const k = a; if (a > 1) k = 2; return k }", + "expression": "(()=>{let caught;try{f(5)}catch(e){caught=e.name+':'+e.message}return [f(1),caught]})()", + "helper": null, + "dimensions": [ + "normal", + "exception-or-nonthrow" + ] + }, + { + "opcode": "catch", + "definition": "function f(o){ let r; try { r = o.x.y } catch (e) { r = e.name } return r }", + "expression": "[f({x:{y:4}}),f({}),f({x:null})]", + "helper": null, + "dimensions": [ + "normal", + "exception-or-nonthrow" + ] + }, + { + "opcode": "nip_catch", + "definition": "function f(g){ try { return g(2) } catch (e) { return 'caught:' + e.name } }", + "expression": "[f(function(v){return v*21}),f(5)]", + "helper": null, + "dimensions": [ + "normal", + "exception-or-nonthrow" + ] + }, + { + "opcode": "gosub", + "definition": "function f(a){ let log = ''; try { if (a) throw a; log += 'body'; } catch (e) { log += 'caught' + e; } finally { log += ',finally'; } return log }", + "expression": "[f(0),f(3)]", + "helper": null, + "dimensions": [ + "normal", + "exception-or-nonthrow" + ] + }, + { + "opcode": "ret", + "definition": "function f(n){ let c = 0; for (let i = 0; i < n; i++) { try { if (i % 3 === 0) continue; c += i; } finally { c += 100; } } return c }", + "expression": "[f(10),f(0)]", + "helper": null, + "dimensions": [ + "normal", + "exception-or-nonthrow" + ] } ] } diff --git a/jit/tests/opcodes.rs b/jit/tests/opcodes.rs index 77d43fadd..2e72f3bed 100644 --- a/jit/tests/opcodes.rs +++ b/jit/tests/opcodes.rs @@ -129,7 +129,7 @@ fn policies_are_semantic_and_categorized() { } #[test] -fn unsupported_frame_and_exception_families_are_stable_rejects() { +fn unsupported_frame_families_reject_and_exception_regions_are_native() { let by_name = |name| { let opcode = linked_opcode_table() .find(|opcode| opcode.name() == name) @@ -145,10 +145,10 @@ fn unsupported_frame_and_exception_families_are_stable_rejects() { by_name("push_this"), Tier1Policy::Reject(FallbackReason::ExtendedFrame) ); - assert_eq!( - by_name("catch"), - Tier1Policy::Reject(FallbackReason::ExceptionRegion) - ); + // Exception regions dispatch natively through the ABI 1.25 entry points. + for native in ["throw", "throw_error", "catch", "nip_catch", "gosub", "ret"] { + assert_eq!(by_name(native), Tier1Policy::Native, "{native}"); + } assert_eq!( by_name("typeof"), Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) diff --git a/jit/tests/tier1_exceptions.rs b/jit/tests/tier1_exceptions.rs new file mode 100644 index 000000000..f241d8674 --- /dev/null +++ b/jit/tests/tier1_exceptions.rs @@ -0,0 +1,170 @@ +//! Tier 1 exception regions: `throw`, `throw_error`, try/catch (`catch`, +//! `nip_catch`) and try/finally (`gosub`, `ret`) execute natively and match +//! the interpreter exactly, including exceptions raised by helpers inside a +//! try region, nested regions, rethrow, and uncatchable interrupts. + +#![cfg(all( + feature = "compiler", + feature = "test-support", + target_endian = "little", + any(target_arch = "x86_64", target_arch = "aarch64"), + any(target_os = "linux", target_os = "macos", target_os = "windows"), + not(all(target_os = "windows", target_arch = "aarch64")) +))] + +use rquickjs_jit::test_support::{differential, forced_baseline}; + +fn same(definition: &str, expression: &str, opcode: &str) { + differential(definition, expression) + .force_baseline() + .expect_executed_opcode(opcode) + .assert_same(); + differential(definition, expression) + .force_baseline() + .expect_executed_opcode(opcode) + .stress_gc() + .assert_same(); +} + +#[test] +fn catch_in_a_hot_loop_resumes_the_handler_natively() { + // `neg` occurs only in the handler: it must execute in native code. + same( + "function f(n){ let caught=0; for(let i=0;i 1) throw {code: a}; return a; }", + "(function(){ try { return f(3); } catch (e) { return 'outer:' + e.code; } })()", + "throw", + ); + same( + "function f(a){ throw new TypeError('bad ' + a); }", + "(function(){ try { f(7); } catch (e) { return e.name + ':' + e.message; } })()", + "throw", + ); +} + +#[test] +fn helper_exceptions_inside_try_reach_the_native_handler() { + // GET_PROPERTY on undefined, CALL of a non-function and a throwing getter. + same( + "function f(o){ let r = 0; try { r = o.x.y; } catch (e) { r = -1; } return r; }", + "[f({x:{y:5}}), f({}), f({x:null})].join()", + "push_minus1", + ); + same( + "function f(g){ try { return g(1); } catch (e) { return 'caught:' + e.name; } }", + "[f(function(v){return v+1}), f(5), f(null)].join()", + "nip_catch", + ); + same( + "function f(o){ let n = 0; for (let i = 0; i < 20; i++) { try { n += o.v; } catch (e) { n -= -e; } } return n; }", + "f({ get v() { if (this.k++ % 3 === 0) throw 7; return 2; }, k: 0 })", + "neg", + ); + // Pending owned operands above the catch offset are released exactly. + same( + "function f(a, o){ let r; try { r = [a, {k: a}, o.x.y]; } catch (e) { r = -1; } return r; }", + "String(f({}, {})) + String(f([1], {x: {y: 2}}).length)", + "push_minus1", + ); +} + +#[test] +fn error_stack_is_captured_at_the_throwing_pc() { + same( + "function f(o){ try { return o.missing.field; } catch (e) { return e.stack; } }", + "f({})", + "catch", + ); + same( + "function f(a){\n try {\n throw new Error('line' + a);\n } catch (e) {\n return e.stack;\n }\n}", + "f(2)", + "throw", + ); +} + +#[test] +fn rethrow_and_nested_regions_match_the_interpreter() { + same( + "function f(a){ let log = ''; try { try { throw a; } catch (e) { log += 'inner' + e; throw e + 1; } } catch (e) { log += ',outer' + e; } return log; }", + "f(1)", + "catch", + ); + same( + "function f(a){ let log = ''; try { try { throw a; } finally { log += 'finally'; } } catch (e) { log += ',caught' + e; } return log; }", + "f(4)", + "gosub", + ); + same( + "function f(a){ let log = ''; try { try { log += 'body'; } finally { log += ',f1'; } log += ',after'; } finally { log += ',f2'; } return log; }", + "f(0)", + "ret", + ); + same( + "function f(a){ try { return a + 1; } finally { a = 100; } }", + "f(41)", + "gosub", + ); + same( + "function f(a){ try { throw a; } finally { return 'override'; } }", + "f(1)", + "throw", + ); + same( + "function f(a){ let log = []; try { log.push('t'); throw new RangeError('r' + a); } catch (e) { log.push(e.message); } finally { log.push('f'); } return log.join(); }", + "f(9)", + "gosub", + ); +} + +#[test] +fn control_transfers_through_finally_blocks() { + same( + "function f(n){ let c = 0; for (let i = 0; i < n; i++) { try { if (i % 3 === 0) continue; c += i; } finally { c += 1000; } } return c; }", + "f(20)", + "ret", + ); + same( + "function f(n){ let c = 0; while (true) { try { c++; if (c > n) break; } finally { c += 10; } } return c; }", + "f(35)", + "gosub", + ); +} + +#[test] +fn throw_error_raises_the_interpreter_error() { + same( + "function f(a){ const k = a; try { k = 2; } catch (e) { return e.name + ':' + e.message; } return 'none'; }", + "f(1)", + "throw_error", + ); + same( + "function f(a){ const k = a; k = 2; }", + "(function(){ try { f(1); } catch (e) { return e.name + ':' + e.message; } })()", + "throw_error", + ); +} + +#[test] +fn interrupts_inside_try_regions_stay_uncatchable() { + forced_baseline( + "function f(n){ let c = 0; for (;;) { try { c++; } catch (e) { c--; } } } f(1)", + ) + .interrupt_after(3) + .assert_uncatchable_interrupt(); + forced_baseline("function f(n){ let c = 0; for (;;) { try { c++; } finally { c--; } } } f(1)") + .interrupt_after(3) + .assert_uncatchable_interrupt(); +} diff --git a/jit/tests/verifier.rs b/jit/tests/verifier.rs index 7616122f8..03636f610 100644 --- a/jit/tests/verifier.rs +++ b/jit/tests/verifier.rs @@ -564,13 +564,43 @@ fn well_formedness_and_tier1_eligibility_are_separate() { } #[test] -fn exception_opcodes_are_well_formed_but_tier1_rejected() { - let verified = snapshot_from_parts(vec![opcode::CATCH, 4, 0, 0, 0, opcode::RETURN], 0, 0, 0, 0) +fn exception_regions_are_tier1_eligible_with_exact_handler_states() { + // catch L7; drop; return_undef; L7: drop; return_undef + let verified = snapshot_from_parts( + vec![ + opcode::CATCH, + 6, + 0, + 0, + 0, + opcode::DROP, + opcode::RETURN_UNDEF, + opcode::DROP, + opcode::RETURN_UNDEF, + ], + 0, + 0, + 0, + 0, + ) + .verify(VerifyLimits::default()) + .unwrap(); + assert!(verified.tier1_eligibility().is_ok()); + let handler = verified + .exception_handler(5) + .expect("drop is inside the region"); + assert_eq!(handler.catch_index(), 0); + assert_eq!(handler.handler_pc(), Some(7)); + assert!(verified.exception_handler(7).is_none()); + assert!(verified.exception_handler(0).is_none()); +} + +#[test] +fn a_catch_whose_handler_is_its_fallthrough_is_rejected() { + let error = snapshot_from_parts(vec![opcode::CATCH, 4, 0, 0, 0, opcode::RETURN], 0, 0, 0, 0) .verify(VerifyLimits::default()) - .unwrap(); - let rejection = verified.tier1_eligibility().unwrap_err(); - assert_eq!(rejection.pc(), 0); - assert_eq!(rejection.reason(), FallbackReason::ExceptionRegion); + .unwrap_err(); + assert_eq!(error.kind(), &VerifyErrorKind::UnsupportedExceptionRegion); } #[test] diff --git a/sys/build_support/patch.rs b/sys/build_support/patch.rs index 41317e2f2..9823306c2 100644 --- a/sys/build_support/patch.rs +++ b/sys/build_support/patch.rs @@ -1,7 +1,7 @@ use std::{fs, io, path::Path}; const QUICKJS_BASELINE_FNV64: u64 = 0x3302_116a_b0fc_269c; -const EXPECTED_PATCHES: [(&str, u64); 19] = [ +const EXPECTED_PATCHES: [(&str, u64); 20] = [ ("0001-rquickjs-jit.patch", 0x18d7_40cc_6943_30cf), ("0002-runtime-feedback.patch", 0xe4c1_ed52_ee5d_d469), ("0003-element-layout.patch", 0xa12c_ccd2_7420_e880), @@ -21,6 +21,7 @@ const EXPECTED_PATCHES: [(&str, u64); 19] = [ ("0018-inline-frame-recovery.patch", 0xb971d61edccde229), ("0019-array-feedback.patch", 0x11c3cb13a8297dd4), ("0020-typed-array-guard.patch", 0x5d10_38a3_8bc9_9821), + ("0027-tier1-exception-regions.patch", 0x28fab1b01554c253), ]; pub(crate) const BASELINE_FILES: [&str; 19] = [ "api-test.c", @@ -60,8 +61,8 @@ const PATCHED_FILE_FINGERPRINTS: [(&str, u64); 21] = [ ("list.h", 0xb337_70f7_b76d_a3d8), ("quickjs-atom.h", 0x30b4_9116_b6a2_aa99), ("quickjs-c-atomics.h", 0x490b_0f29_f631_3fc0), - ("quickjs.c", 0xc535_b5f1_48e7_9b27), - ("quickjs-jit.h", 0x0188_04ad_21d2_95a7), + ("quickjs.c", 0x3e661e5849961ea9), + ("quickjs-jit.h", 0xb748adfe5e91c323), ("quickjs-jit-helpers.h", 0xa469_e80f_be47_404c), ("quickjs-opcode.h", 0x3d05_cfdf_5cf7_2930), ("quickjs.h", 0x4831_2cde_9c2f_a5ee), diff --git a/sys/patches/0027-tier1-exception-regions.patch b/sys/patches/0027-tier1-exception-regions.patch new file mode 100644 index 000000000..b0b463bad --- /dev/null +++ b/sys/patches/0027-tier1-exception-regions.patch @@ -0,0 +1,201 @@ +diff --git a/quickjs-jit.h b/quickjs-jit.h +--- a/quickjs-jit.h ++++ b/quickjs-jit.h +@@ -17,7 +17,7 @@ + #endif + + #define QJSJIT_ABI_MAJOR 1u +-#define QJSJIT_ABI_MINOR 24u ++#define QJSJIT_ABI_MINOR 25u + + /* Frame flags are append-only. Stress collection is a test/debug contract: + * every helper performs an allocating cycle-GC point before and after its +@@ -601,6 +601,28 @@ + * table only after validating the main JIT ABI, table size/version and effects. */ + JS_EXTERN const JSJitInlineAPI *JS_JitGetInlineAPI(uint32_t version); + ++/* Tier 1 exception regions (ABI 1.25). These calls take a validated active ++ * frame and stack-map id. ThrowValue consumes the named operand-stack slot as ++ * the pending exception (OP_throw) and always returns EXCEPTION. ++ * CatchException requires a pending exception. It returns OK only after it ++ * performed the interpreter's exception prologue for this frame (backtrace at ++ * frame->pc), proved the innermost catch offset is catch_slot for handler_pc, ++ * released every operand above it and stored the caught value in catch_slot; ++ * stack_top is then catch_slot + 1. For an uncatchable exception it returns ++ * EXCEPTION with the frame untouched so ordinary unwinding continues. */ ++JS_EXTERN JSJitHelperStatus JS_JitThrowValue(JSJitExecFrame *frame, ++ uint32_t stack_map_id, ++ uint32_t slot); ++/* ThrowError raises OP_throw_error's error for atom/type (JS_THROW_* codes) ++ * and always returns EXCEPTION. */ ++JS_EXTERN JSJitHelperStatus JS_JitThrowError(JSJitExecFrame *frame, ++ uint32_t stack_map_id, ++ uint32_t atom, uint32_t type); ++JS_EXTERN JSJitHelperStatus JS_JitCatchException(JSJitExecFrame *frame, ++ uint32_t stack_map_id, ++ uint32_t catch_slot, ++ uint32_t handler_pc); ++ + #ifdef __cplusplus + } + #endif +diff --git a/quickjs.c b/quickjs.c +--- a/quickjs.c ++++ b/quickjs.c +@@ -20431,6 +20431,128 @@ + return version == QJSJIT_INLINE_RECOVERY_VERSION ? &api : NULL; + } + ++/* Tier 1 exception regions. ThrowValue/ThrowError raise exactly like ++ * OP_throw/OP_throw_error. CatchException replays the interpreter's ++ * `exception:` label for the active root frame up to and including the ++ * innermost catch offset, but only after proving that the unwinding would ++ * stop at the caller-named slot and handler. An uncatchable exception is ++ * returned with the frame untouched, so the ordinary native exception exit ++ * lets the interpreter unwind it; malformed requests raise the usual ++ * uncatchable invariant error. */ ++JSJitHelperStatus JS_JitThrowValue(JSJitExecFrame *frame, ++ uint32_t stack_map_id, uint32_t slot) ++{ ++ JSFunctionBytecode *b; ++ JSStackFrame *sf; ++ JSValue *in, value; ++ ++ if (qjsjit_validate_helper_frame(frame, true, stack_map_id, &b, &sf) < 0) ++ return JS_JIT_HELPER_EXCEPTION; ++ if (!qjsjit_resolve_slot(frame, b, slot, &in) || in < frame->stack_base) ++ return qjsjit_reject_helper(frame, "invalid JIT throw slot"); ++ if (qjsjit_stress_gc(frame) < 0) { ++ qjsjit_consume_slot(frame->ctx, in); ++ return JS_JIT_HELPER_EXCEPTION; ++ } ++ value = *in; ++ *in = JS_UNDEFINED; ++ JS_Throw(frame->ctx, value); ++ qjsjit_stress_gc_after_exception(frame); ++ return JS_JIT_HELPER_EXCEPTION; ++} ++ ++/* OP_throw_error; `type` uses the interpreter's JS_THROW_* numbering. */ ++JSJitHelperStatus JS_JitThrowError(JSJitExecFrame *frame, ++ uint32_t stack_map_id, uint32_t atom, ++ uint32_t type) ++{ ++ JSFunctionBytecode *b; ++ JSStackFrame *sf; ++ JSContext *ctx; ++ ++ if (qjsjit_validate_helper_frame(frame, true, stack_map_id, &b, &sf) < 0) ++ return JS_JIT_HELPER_EXCEPTION; ++ if (!qjsjit_valid_atom(frame->rt, (JSAtom)atom)) ++ return qjsjit_reject_helper(frame, "invalid JIT throw_error atom"); ++ if (qjsjit_stress_gc(frame) < 0) ++ return JS_JIT_HELPER_EXCEPTION; ++ ctx = frame->ctx; ++ if (type == 0) /* JS_THROW_VAR_RO */ ++ JS_ThrowTypeErrorReadOnly(ctx, JS_PROP_THROW, (JSAtom)atom); ++ else if (type == 1) /* JS_THROW_VAR_REDECL */ ++ JS_ThrowSyntaxErrorVarRedeclaration(ctx, (JSAtom)atom); ++ else if (type == 2) /* JS_THROW_VAR_UNINITIALIZED */ ++ JS_ThrowReferenceErrorUninitialized(ctx, (JSAtom)atom); ++ else if (type == 3) /* JS_THROW_ERROR_DELETE_SUPER */ ++ JS_ThrowReferenceError(ctx, "unsupported reference to 'super'"); ++ else if (type == 4) /* JS_THROW_ERROR_ITERATOR_THROW */ ++ JS_ThrowTypeError(ctx, "iterator does not have a throw method"); ++ else ++ JS_ThrowInternalError(ctx, "invalid throw var type %d", (int)type); ++ qjsjit_stress_gc_after_exception(frame); ++ return JS_JIT_HELPER_EXCEPTION; ++} ++ ++JSJitHelperStatus JS_JitCatchException(JSJitExecFrame *frame, ++ uint32_t stack_map_id, ++ uint32_t catch_slot, ++ uint32_t handler_pc) ++{ ++ JSFunctionBytecode *b; ++ JSStackFrame *sf; ++ JSRuntime *rt; ++ JSContext *ctx; ++ JSValue *catch_value, *sp; ++ ++ if (qjsjit_validate_helper_frame_state(frame, true, stack_map_id, &b, &sf, ++ QJSJIT_VALIDATE_RESUME) < 0) ++ return JS_JIT_HELPER_EXCEPTION; ++ rt = frame->rt; ++ ctx = frame->ctx; ++ if (!JS_HasException(ctx)) ++ return qjsjit_reject_helper(frame, "JIT catch without exception"); ++ /* Uncatchable errors bypass every handler; propagate unchanged. */ ++ if (qjsjit_is_uncatchable_sentinel(rt, rt->current_exception) || ++ JS_IsUncatchableError(rt->current_exception)) ++ return JS_JIT_HELPER_EXCEPTION; ++ if (handler_pc == 0 || handler_pc >= (uint32_t)b->byte_code_len || ++ !qjsjit_resolve_slot(frame, b, catch_slot, &catch_value) || ++ catch_value < frame->stack_base || ++ JS_VALUE_GET_TAG(*catch_value) != JS_TAG_CATCH_OFFSET || ++ (uint32_t)JS_VALUE_GET_INT(*catch_value) != handler_pc) ++ return qjsjit_reject_helper(frame, "invalid JIT catch slot"); ++ for (sp = frame->stack_top; sp > catch_value + 1; sp--) { ++ if (JS_VALUE_GET_TAG(sp[-1]) == JS_TAG_CATCH_OFFSET) ++ return qjsjit_reject_helper(frame, "invalid JIT catch nesting"); ++ } ++ /* Exception label prologue, in interpreter order: backtrace first (it ++ * observes this frame's current pc), then catchability. */ ++ if (needs_backtrace(rt->current_exception) || ++ JS_IsUndefined(ctx->error_back_trace)) { ++ sf->cur_pc = (uint8_t *)frame->pc; ++ build_backtrace(ctx, rt->current_exception, JS_UNDEFINED, ++ NULL, 0, 0, 0); ++ } ++ if (!JS_HasException(ctx) || ++ qjsjit_is_uncatchable_sentinel(rt, rt->current_exception) || ++ JS_IsUncatchableError(rt->current_exception)) ++ return JS_JIT_HELPER_EXCEPTION; ++ /* Release operands above the catch offset from the top, like the ++ * interpreter, then replace the offset with the caught value. */ ++ for (sp = frame->stack_top; sp > catch_value + 1;) { ++ JSValue value = *--sp; ++ *sp = JS_UNDEFINED; ++ frame->stack_top = sp; ++ JS_FreeValue(ctx, value); ++ } ++ *catch_value = rt->current_exception; ++ rt->current_exception = JS_UNINITIALIZED; ++ JS_FreeValueRT(rt, ctx->error_back_trace); ++ ctx->error_back_trace = JS_UNDEFINED; ++ frame->stack_top = catch_value + 1; ++ return qjsjit_stress_gc(frame); ++} ++ + static JSValue JS_CallInternal(JSContext *caller_ctx, JSValueConst func_obj, + JSValueConst this_obj, JSValueConst new_target, int argc, + JSValueConst *argv, int flags) +@@ -20691,12 +20813,25 @@ + b->jit_feedback_disabled = 1; + if ((request_snapshot & 1U) && hot_vtable->submit_snapshot) { + JSJitFunctionSnapshot *snapshot = NULL; +- if (JS_JitSnapshotFunction(ctx, func_obj, &snapshot) == +- JS_JIT_SNAPSHOT_OK && snapshot) { ++ /* Generator and async resumes pass their state, not the ++ * function object, as func_obj; classify them by bytecode. */ ++ int snapshot_status = b->func_kind != JS_FUNC_NORMAL ? ++ JS_JIT_SNAPSHOT_ASYNC : ++ JS_JitSnapshotFunction(ctx, func_obj, &snapshot); ++ if (snapshot_status == JS_JIT_SNAPSHOT_OK && snapshot) { + /* Ownership transfers for exactly this callback. */ + rt->jit_backend_active_depth++; + hot_vtable->submit_snapshot(hot_opaque, snapshot); + rt->jit_backend_active_depth--; ++ } else if (snapshot_status == JS_JIT_SNAPSHOT_GENERATOR || ++ snapshot_status == JS_JIT_SNAPSHOT_ASYNC || ++ snapshot_status == JS_JIT_SNAPSHOT_EVAL || ++ snapshot_status == JS_JIT_SNAPSHOT_WITH) { ++ /* These refusals are properties of this immutable bytecode ++ * generation, never reported to the backend. Stop hot and ++ * feedback probes locally instead of re-requesting a ++ * snapshot at every later call, resume and loop poll. */ ++ b->jit_feedback_disabled = 1; + } + } + } diff --git a/sys/src/bindings/aarch64-apple-darwin.rs b/sys/src/bindings/aarch64-apple-darwin.rs index a64c09ee5..72fa1b677 100644 --- a/sys/src/bindings/aarch64-apple-darwin.rs +++ b/sys/src/bindings/aarch64-apple-darwin.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -3900,6 +3900,26 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetInlineAPI(version: u32) -> *const JSJitInlineAPI; } +unsafe extern "C" { + pub fn JS_JitThrowValue(frame: *mut JSJitExecFrame, stack_map_id: u32, slot: u32) + -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitThrowError( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + atom: u32, + type_: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitCatchException( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + catch_slot: u32, + handler_pc: u32, + ) -> JSJitHelperStatus; +} pub const __JS_ATOM_NULL: _bindgen_ty_7 = 0; pub const JS_ATOM_null: _bindgen_ty_7 = 1; pub const JS_ATOM_false: _bindgen_ty_7 = 2; diff --git a/sys/src/bindings/aarch64-pc-windows-msvc.rs b/sys/src/bindings/aarch64-pc-windows-msvc.rs index a64c09ee5..72fa1b677 100644 --- a/sys/src/bindings/aarch64-pc-windows-msvc.rs +++ b/sys/src/bindings/aarch64-pc-windows-msvc.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -3900,6 +3900,26 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetInlineAPI(version: u32) -> *const JSJitInlineAPI; } +unsafe extern "C" { + pub fn JS_JitThrowValue(frame: *mut JSJitExecFrame, stack_map_id: u32, slot: u32) + -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitThrowError( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + atom: u32, + type_: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitCatchException( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + catch_slot: u32, + handler_pc: u32, + ) -> JSJitHelperStatus; +} pub const __JS_ATOM_NULL: _bindgen_ty_7 = 0; pub const JS_ATOM_null: _bindgen_ty_7 = 1; pub const JS_ATOM_false: _bindgen_ty_7 = 2; diff --git a/sys/src/bindings/aarch64-unknown-linux-gnu.rs b/sys/src/bindings/aarch64-unknown-linux-gnu.rs index a64c09ee5..72fa1b677 100644 --- a/sys/src/bindings/aarch64-unknown-linux-gnu.rs +++ b/sys/src/bindings/aarch64-unknown-linux-gnu.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -3900,6 +3900,26 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetInlineAPI(version: u32) -> *const JSJitInlineAPI; } +unsafe extern "C" { + pub fn JS_JitThrowValue(frame: *mut JSJitExecFrame, stack_map_id: u32, slot: u32) + -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitThrowError( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + atom: u32, + type_: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitCatchException( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + catch_slot: u32, + handler_pc: u32, + ) -> JSJitHelperStatus; +} pub const __JS_ATOM_NULL: _bindgen_ty_7 = 0; pub const JS_ATOM_null: _bindgen_ty_7 = 1; pub const JS_ATOM_false: _bindgen_ty_7 = 2; diff --git a/sys/src/bindings/aarch64-unknown-linux-musl.rs b/sys/src/bindings/aarch64-unknown-linux-musl.rs index a64c09ee5..72fa1b677 100644 --- a/sys/src/bindings/aarch64-unknown-linux-musl.rs +++ b/sys/src/bindings/aarch64-unknown-linux-musl.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -3900,6 +3900,26 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetInlineAPI(version: u32) -> *const JSJitInlineAPI; } +unsafe extern "C" { + pub fn JS_JitThrowValue(frame: *mut JSJitExecFrame, stack_map_id: u32, slot: u32) + -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitThrowError( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + atom: u32, + type_: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitCatchException( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + catch_slot: u32, + handler_pc: u32, + ) -> JSJitHelperStatus; +} pub const __JS_ATOM_NULL: _bindgen_ty_7 = 0; pub const JS_ATOM_null: _bindgen_ty_7 = 1; pub const JS_ATOM_false: _bindgen_ty_7 = 2; diff --git a/sys/src/bindings/wasm32-wasip1.rs b/sys/src/bindings/wasm32-wasip1.rs index a8167404c..c48ca8a17 100644 --- a/sys/src/bindings/wasm32-wasip1.rs +++ b/sys/src/bindings/wasm32-wasip1.rs @@ -87,7 +87,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -3869,6 +3869,26 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetInlineAPI(version: u32) -> *const JSJitInlineAPI; } +unsafe extern "C" { + pub fn JS_JitThrowValue(frame: *mut JSJitExecFrame, stack_map_id: u32, slot: u32) + -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitThrowError( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + atom: u32, + type_: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitCatchException( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + catch_slot: u32, + handler_pc: u32, + ) -> JSJitHelperStatus; +} pub const __JS_ATOM_NULL: _bindgen_ty_5 = 0; pub const JS_ATOM_null: _bindgen_ty_5 = 1; pub const JS_ATOM_false: _bindgen_ty_5 = 2; diff --git a/sys/src/bindings/x86_64-apple-darwin.rs b/sys/src/bindings/x86_64-apple-darwin.rs index a64c09ee5..72fa1b677 100644 --- a/sys/src/bindings/x86_64-apple-darwin.rs +++ b/sys/src/bindings/x86_64-apple-darwin.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -3900,6 +3900,26 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetInlineAPI(version: u32) -> *const JSJitInlineAPI; } +unsafe extern "C" { + pub fn JS_JitThrowValue(frame: *mut JSJitExecFrame, stack_map_id: u32, slot: u32) + -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitThrowError( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + atom: u32, + type_: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitCatchException( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + catch_slot: u32, + handler_pc: u32, + ) -> JSJitHelperStatus; +} pub const __JS_ATOM_NULL: _bindgen_ty_7 = 0; pub const JS_ATOM_null: _bindgen_ty_7 = 1; pub const JS_ATOM_false: _bindgen_ty_7 = 2; diff --git a/sys/src/bindings/x86_64-pc-windows-gnu.rs b/sys/src/bindings/x86_64-pc-windows-gnu.rs index a64c09ee5..72fa1b677 100644 --- a/sys/src/bindings/x86_64-pc-windows-gnu.rs +++ b/sys/src/bindings/x86_64-pc-windows-gnu.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -3900,6 +3900,26 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetInlineAPI(version: u32) -> *const JSJitInlineAPI; } +unsafe extern "C" { + pub fn JS_JitThrowValue(frame: *mut JSJitExecFrame, stack_map_id: u32, slot: u32) + -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitThrowError( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + atom: u32, + type_: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitCatchException( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + catch_slot: u32, + handler_pc: u32, + ) -> JSJitHelperStatus; +} pub const __JS_ATOM_NULL: _bindgen_ty_7 = 0; pub const JS_ATOM_null: _bindgen_ty_7 = 1; pub const JS_ATOM_false: _bindgen_ty_7 = 2; diff --git a/sys/src/bindings/x86_64-pc-windows-msvc.rs b/sys/src/bindings/x86_64-pc-windows-msvc.rs index a64c09ee5..72fa1b677 100644 --- a/sys/src/bindings/x86_64-pc-windows-msvc.rs +++ b/sys/src/bindings/x86_64-pc-windows-msvc.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -3900,6 +3900,26 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetInlineAPI(version: u32) -> *const JSJitInlineAPI; } +unsafe extern "C" { + pub fn JS_JitThrowValue(frame: *mut JSJitExecFrame, stack_map_id: u32, slot: u32) + -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitThrowError( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + atom: u32, + type_: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitCatchException( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + catch_slot: u32, + handler_pc: u32, + ) -> JSJitHelperStatus; +} pub const __JS_ATOM_NULL: _bindgen_ty_7 = 0; pub const JS_ATOM_null: _bindgen_ty_7 = 1; pub const JS_ATOM_false: _bindgen_ty_7 = 2; diff --git a/sys/src/bindings/x86_64-unknown-linux-gnu.rs b/sys/src/bindings/x86_64-unknown-linux-gnu.rs index a64c09ee5..72fa1b677 100644 --- a/sys/src/bindings/x86_64-unknown-linux-gnu.rs +++ b/sys/src/bindings/x86_64-unknown-linux-gnu.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -3900,6 +3900,26 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetInlineAPI(version: u32) -> *const JSJitInlineAPI; } +unsafe extern "C" { + pub fn JS_JitThrowValue(frame: *mut JSJitExecFrame, stack_map_id: u32, slot: u32) + -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitThrowError( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + atom: u32, + type_: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitCatchException( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + catch_slot: u32, + handler_pc: u32, + ) -> JSJitHelperStatus; +} pub const __JS_ATOM_NULL: _bindgen_ty_7 = 0; pub const JS_ATOM_null: _bindgen_ty_7 = 1; pub const JS_ATOM_false: _bindgen_ty_7 = 2; diff --git a/sys/src/bindings/x86_64-unknown-linux-musl.rs b/sys/src/bindings/x86_64-unknown-linux-musl.rs index a64c09ee5..72fa1b677 100644 --- a/sys/src/bindings/x86_64-unknown-linux-musl.rs +++ b/sys/src/bindings/x86_64-unknown-linux-musl.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -3900,6 +3900,26 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetInlineAPI(version: u32) -> *const JSJitInlineAPI; } +unsafe extern "C" { + pub fn JS_JitThrowValue(frame: *mut JSJitExecFrame, stack_map_id: u32, slot: u32) + -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitThrowError( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + atom: u32, + type_: u32, + ) -> JSJitHelperStatus; +} +unsafe extern "C" { + pub fn JS_JitCatchException( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + catch_slot: u32, + handler_pc: u32, + ) -> JSJitHelperStatus; +} pub const __JS_ATOM_NULL: _bindgen_ty_7 = 0; pub const JS_ATOM_null: _bindgen_ty_7 = 1; pub const JS_ATOM_false: _bindgen_ty_7 = 2; From 2cc034225fde1f0c80ac1bc0c10e75f0a7d3e0b7 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:04:46 +0800 Subject: [PATCH 16/54] perf(jit): lower for-of/for-in iteration opcodes in Tier 1 Add QuickJS patch 0028 (ABI 1.25, runtime API 1.10): an append-only ITERATOR_OP helper that runs the interpreter's exact stack effect for for_of_start/for_of_next/for_in_start/for_in_next/iterator_close on the materialized frame, plus a versioned non-allocating Array values leaf (JS_JitGetIteratorAPI) that revalidates the live iterator, the exact built-in next method, a fast Array and an in-bounds dense index on every step. Tier 1 now admits the five opcodes; for_of_next tries the leaf first and falls back to the helper, so iterator close on break/throw and all protocol errors keep interpreter semantics (the catch offset stays in the frame for the interpreter unwinder). The verifier keeps enumeration-record slot kinds across for_of_next/for_in_next, joins loop-scoped TDZ locals as Tagged, and OSR validates catch offsets by their real tag. Adds tier1_iteration regression tests, opcode manifest cases, updated bundled bindings for all JIT targets, and a for-of-array scenario. Noisy diagnostic: for-of-array 2.97x the 82d3808 automatic speed; collections/map-set-bigint/strings-json/json-codec statistically unchanged (see docs/roadmap/p2d-iteration.md). Co-Authored-By: Claude Opus 5.5 (1M context) --- benchmarks/run.rs | 7 + benchmarks/scripts/for-of-array.js | 11 + docs/roadmap/p2d-iteration.md | 83 +++++ jit/Cargo.toml | 4 + jit/src/abi.rs | 88 ++++- jit/src/bytecode/policy.rs | 1 + jit/src/bytecode/policy_audit.rs | 10 +- jit/src/bytecode/stack.rs | 24 +- jit/src/compiler/baseline.rs | 225 +++++++++++- jit/src/ir/baseline.rs | 10 +- jit/src/ir/mod.rs | 2 +- jit/src/ir/types.rs | 39 ++ jit/src/lib.rs | 3 +- jit/tests/abi.rs | 16 +- jit/tests/differential.rs | 9 + jit/tests/fixtures/opcode-cases.json | 60 ++++ jit/tests/opcodes.rs | 13 + jit/tests/required_features.rs | 1 + jit/tests/support/mod.rs | 25 +- jit/tests/tier1_iteration.rs | 339 ++++++++++++++++++ sys/build.rs | 2 +- sys/build_support/patch.rs | 9 +- sys/patches/0028-tier1-iteration.patch | 258 +++++++++++++ sys/patches/README.md | 13 + sys/src/bindings/aarch64-apple-darwin.rs | 71 +++- sys/src/bindings/aarch64-pc-windows-msvc.rs | 71 +++- sys/src/bindings/aarch64-unknown-linux-gnu.rs | 71 +++- .../bindings/aarch64-unknown-linux-musl.rs | 71 +++- sys/src/bindings/wasm32-wasip1.rs | 71 +++- sys/src/bindings/x86_64-apple-darwin.rs | 71 +++- sys/src/bindings/x86_64-pc-windows-gnu.rs | 71 +++- sys/src/bindings/x86_64-pc-windows-msvc.rs | 71 +++- sys/src/bindings/x86_64-unknown-linux-gnu.rs | 71 +++- sys/src/bindings/x86_64-unknown-linux-musl.rs | 71 +++- sys/tests/jit_patch.rs | 37 +- sys/tests/typed_array_guard.rs | 2 +- 36 files changed, 1924 insertions(+), 77 deletions(-) create mode 100644 benchmarks/scripts/for-of-array.js create mode 100644 docs/roadmap/p2d-iteration.md create mode 100644 jit/tests/tier1_iteration.rs create mode 100644 sys/patches/0028-tier1-iteration.patch diff --git a/benchmarks/run.rs b/benchmarks/run.rs index 20fe492ac..e792b44f4 100644 --- a/benchmarks/run.rs +++ b/benchmarks/run.rs @@ -152,6 +152,13 @@ const WORKLOADS: &[Workload] = &[ designated: false, file: "collections.js", }, + Workload { + name: "for-of-array", + suite: "rquickjs-jit focused", + group: "compute", + designated: false, + file: "for-of-array.js", + }, Workload { name: "strings-json", suite: "rquickjs-jit", diff --git a/benchmarks/scripts/for-of-array.js b/benchmarks/scripts/for-of-array.js new file mode 100644 index 000000000..a795ca3db --- /dev/null +++ b/benchmarks/scripts/for-of-array.js @@ -0,0 +1,11 @@ +function workload(iterations) { + let values = globalThis.forOfArrayValues; + if (!values || values.length !== iterations) { + values = []; + for (let i = 0; i < iterations; i++) values.push((i * 17) & 1023); + globalThis.forOfArrayValues = values; + } + let sum = 0; + for (const value of values) sum += value; + return String(sum); +} diff --git a/docs/roadmap/p2d-iteration.md b/docs/roadmap/p2d-iteration.md new file mode 100644 index 000000000..26afc9612 --- /dev/null +++ b/docs/roadmap/p2d-iteration.md @@ -0,0 +1,83 @@ +# P2d: Tier 1 iteration opcodes (first slice) + +Branch `perf/p2d-iteration`, based on `82d3808`. Roadmap item P2.5 +(`for_of_*`/`for_in_*`/`iterator_*`) from +[PERFORMANCE_ROADMAP_V8_BUN.md](../PERFORMANCE_ROADMAP_V8_BUN.md). + +## What landed + +- QuickJS patch `0028-tier1-iteration.patch` (ABI 1.25, runtime API 1.10): + - Append-only helper `ITERATOR_OP` (helper ID 24). It runs the interpreter's + own stack effect (`js_for_of_start`, `js_for_of_next`, `js_for_in_start`, + `js_for_in_next`, and the `iterator_close` sequence) directly on the + materialized native frame. Iterator objects, `next` methods and the for-of + catch offset stay C-visible across user iterator code and GC. On an + exception, the helper leaves exactly the stack that the interpreter's + unwinder expects. That includes the catch offset, which closes the + iterator with a throw completion. `iterator_close` reports its exception + at `depth - 2`, as the interpreter does. + - Versioned leaf table `JS_JitGetIteratorAPI(1)` with `array_values_next`. + On every step it revalidates four conditions: the live iterator is an + Array values iterator, its `next` is the exact built-in C function, the + target is a fast `Array`, and the index is below the dense count. When all + four hold, it advances the index and returns a duplicated element. It + performs no allocation, throw, finalization or reentry. Any other state + misses without touching anything. +- Tier 1 admits `for_of_start`, `for_of_next`, `for_in_start`, `for_in_next` + and `iterator_close` (`HelperId::IteratorOp`). `for_of_next` calls the leaf + first and takes the helper only on a miss, which includes completion. A + dense packed-array loop therefore makes three helper calls in total: + start, completion and close. +- Because the leaf revalidates on every step, no pristine + `Array.prototype[Symbol.iterator]` or `%ArrayIteratorPrototype%.next` + watchpoint is needed. A replaced protocol is observed on the next step, and + no deopt path exists to get wrong. +- Verifier: + - `for_of_next`/`for_in_next` keep the enumeration record's slot kinds, + including `CatchOffset`. + - A loop-scoped lexical local now joins `Uninitialized` with + `Tagged`/`Int32`/`Float64` as `Tagged`. Every read of such a local is a + checked `get_loc_check`. + - OSR validation of a `CatchOffset` slot now checks `JS_TAG_CATCH_OFFSET`. + The old code checked `JS_TAG_INT`, which was a latent bug that never ran + because for-of was rejected. +- New scenario `for-of-array` (for-of sum over a cached packed array) in + `benchmarks/run.rs`. The README matrix is not updated; that is left to the + integrator. + +## Deliberately not in this slice + +- `iterator_next`, `iterator_call` and `iterator_get_value_done` are only + emitted in generator (`yield*`) and async (`for await`) functions. Those + functions are already rejected with `Generator`/`Async`, so these opcodes + stay policy rejects. +- `return` inside a for-of emits `nip_catch` (`ExceptionRegion`), so such + functions stay in the interpreter. Supporting it needs static tracking of + catch-offset positions, which fits naturally with P2.3 (try/catch). +- Array destructuring with `...rest`, or with a numeric default (`[a=7]`), + still fails the verifier's exact operand-stack merge (Int32 vs Tagged). + This limitation predates this work. +- Tier 2 rejects any loop header with a non-empty operand stack, so for-of + and for-in loops stay in Tier 1. + +## Remaining work and measured limits + +- `collections` shows no steady-state change. Its for-of loop alone is about + 3x faster (see `for-of-array`), but the same function's `values.push` loop + is slower in Tier 1 than in the interpreter: every Tier 1 generic call pays + the helper-frame validation and call bridge (roadmap B1/B2). In the current + profitability model, the automatic tier then demotes the function to the + interpreter after five rejected trials. That demotion happens to be the + right call today: forcing it off measured `collections` at 0.65x. + `collections` gains once P1 (O(1) helper validation, cached native entry) + or P3 lands, or once Tier 2 accepts stack-carrying loop headers. The + enumeration record could be kept in SSA values with lazy materialization. +- `map-set-bigint` iterates `map.values()`, a Map iterator. Only Array values + iterators have a leaf; Map/Set leaves would follow the same pattern + (`js_map_iterator_next`). +- The JSON and string workloads do not use for-of in their kernels. +- Merge risk: helper IDs and runtime API fields are append-only, and 0028 + hunks use exact line numbers, so parallel ABI patches conflict. Rebase by + regenerating 0028 on top of the merged patch set, renumbering the helper ID + and the `JSJitRuntimeAPI` offset (currently 200, size 208), and updating + the digests in `sys/build_support/patch.rs`. diff --git a/jit/Cargo.toml b/jit/Cargo.toml index e2b37836d..8d2445b51 100644 --- a/jit/Cargo.toml +++ b/jit/Cargo.toml @@ -187,6 +187,10 @@ required-features = ["compiler", "test-support"] name = "tier1_regexp" required-features = ["compiler", "test-support"] +[[test]] +name = "tier1_iteration" +required-features = ["compiler", "test-support"] + [[test]] name = "test262" required-features = ["compiler", "test-support"] diff --git a/jit/src/abi.rs b/jit/src/abi.rs index b39c65a23..31eba6e18 100644 --- a/jit/src/abi.rs +++ b/jit/src/abi.rs @@ -5,7 +5,7 @@ use core::{fmt, mem}; use rquickjs_core::qjs; pub const ABI_MAJOR: u16 = 1; -pub const ABI_MINOR: u16 = 24; +pub const ABI_MINOR: u16 = 25; pub const SOURCE_REVISION: u64 = 0xfd0a_0210_b7be_0095; pub const OPCODE_FINGERPRINT: u64 = qjs::QJSJIT_GENERATED_OPCODE_FINGERPRINT; @@ -30,6 +30,7 @@ pub enum AbiStructure { BackendVTable, InlineApi, ArrayApi, + IteratorApi, } /// Constructors for the authoritative native-to-interpreter exit contract. @@ -115,6 +116,7 @@ pub struct AbiInfo { raw: qjs::JSJitABIInfo, inline_api: qjs::JSJitInlineAPI, array_api: qjs::JSJitArrayAPI, + iterator_api: qjs::JSJitIteratorAPI, } /// Runtime-exported, append-only offsets for element fast paths. QuickJS @@ -198,6 +200,13 @@ impl AbiInfo { &self.array_api } + /// Non-allocating continuing-path leaves for synchronous iteration. A + /// miss never touches state; callers fall back to the exact helper. + #[cfg(feature = "compiler")] + pub(crate) const fn iterator_api(&self) -> &qjs::JSJitIteratorAPI { + &self.iterator_api + } + #[cfg(feature = "compiler")] pub(crate) fn element_layout(&self) -> ElementLayout { let raw = self.raw.element_layout; @@ -240,12 +249,14 @@ impl AbiInfo { // All fields are integers or nullable function pointers. inline_api: unsafe { mem::zeroed() }, array_api: unsafe { mem::zeroed() }, + iterator_api: unsafe { mem::zeroed() }, }; // Do not resolve native inline addresses against a mismatched // main ABI, even when the optional table has a familiar version. info.validate_main()?; info.inline_api = query_inline_api()?; info.array_api = query_array_api()?; + info.iterator_api = query_iterator_api()?; Ok(info) } else { Err(AbiError::QueryFailed(status)) @@ -255,7 +266,8 @@ impl AbiInfo { pub(crate) fn validate(&self) -> Result<(), AbiError> { self.validate_main()?; validate_inline_api(self.inline_api())?; - validate_array_api(&self.array_api) + validate_array_api(&self.array_api)?; + validate_iterator_api(&self.iterator_api) } fn validate_main(&self) -> Result<(), AbiError> { @@ -373,6 +385,9 @@ impl AbiInfo { } AbiMismatch::StructureLayout(AbiStructure::InlineApi) => self.inline_api.version ^= 1, AbiMismatch::StructureLayout(AbiStructure::ArrayApi) => self.array_api.version ^= 1, + AbiMismatch::StructureLayout(AbiStructure::IteratorApi) => { + self.iterator_api.version ^= 1 + } AbiMismatch::BuildFingerprint => self.raw.build_fingerprint ^= 1, } } @@ -698,6 +713,10 @@ fn runtime_api_layout_fingerprint() -> u64 { mem::offset_of!(qjs::JSJitRuntimeAPI, unary_arith_slow), mem::size_of::(), ), + ( + mem::offset_of!(qjs::JSJitRuntimeAPI, iterator_op), + mem::size_of::(), + ), ] { hash = layout_field(hash, offset, size); } @@ -1009,6 +1028,69 @@ fn validate_array_api(api: &qjs::JSJitArrayAPI) -> Result<(), AbiError> { } } +fn query_iterator_api() -> Result { + let pointer = unsafe { qjs::JS_JitGetIteratorAPI(qjs::QJSJIT_ITERATOR_API_VERSION) }; + if pointer.is_null() + || unsafe { core::ptr::addr_of!((*pointer).struct_size).read() } + != mem::size_of::() as u32 + { + return Err(AbiError::Incompatible(AbiMismatch::StructureLayout( + AbiStructure::IteratorApi, + ))); + } + let api = unsafe { pointer.read() }; + validate_iterator_api(&api)?; + Ok(api) +} + +fn validate_iterator_api(api: &qjs::JSJitIteratorAPI) -> Result<(), AbiError> { + if api.struct_size != mem::size_of::() as u32 + || api.version != qjs::QJSJIT_ITERATOR_API_VERSION + || api.effects != 0 + || api.reserved != 0 + || api.array_values_next.is_none() + { + Err(AbiError::Incompatible(AbiMismatch::StructureLayout( + AbiStructure::IteratorApi, + ))) + } else { + Ok(()) + } +} + +#[cfg(test)] +mod iterator_api_tests { + use super::*; + + #[test] + fn iterator_api_is_exactly_versioned_and_complete() { + let pointer = unsafe { qjs::JS_JitGetIteratorAPI(qjs::QJSJIT_ITERATOR_API_VERSION) }; + assert!(!pointer.is_null()); + let valid = unsafe { *pointer }; + assert!(validate_iterator_api(&valid).is_ok()); + assert!(unsafe { qjs::JS_JitGetIteratorAPI(0) }.is_null()); + for field in 0..5 { + let mut invalid = valid; + match field { + 0 => invalid.struct_size -= 1, + 1 => invalid.version += 1, + 2 => invalid.effects = 1, + 3 => invalid.reserved = 1, + _ => invalid.array_values_next = None, + } + assert!(validate_iterator_api(&invalid).is_err(), "field {field}"); + } + let mut info = AbiInfo::linked().unwrap(); + info.iterator_api.effects = 1; + assert_eq!( + info.validate(), + Err(AbiError::Incompatible(AbiMismatch::StructureLayout( + AbiStructure::IteratorApi + ))) + ); + } +} + #[cfg(test)] mod array_api_tests { use super::*; @@ -1073,7 +1155,7 @@ mod inline_api_tests { fn inline_recovery_requires_the_matching_main_abi_minor() { let native = AbiInfo::query_linked().unwrap(); assert_eq!(ABI_MINOR, native.minor()); - assert_eq!(native.minor(), 24); + assert_eq!(native.minor(), 25); assert!(unsafe { qjs::JS_JitGetInlineAPI(0) }.is_null()); } diff --git a/jit/src/bytecode/policy.rs b/jit/src/bytecode/policy.rs index 08bd6b6b9..b9858f7d2 100644 --- a/jit/src/bytecode/policy.rs +++ b/jit/src/bytecode/policy.rs @@ -32,6 +32,7 @@ pub enum HelperId { NewObject, CallConstructor, Regexp, + IteratorOp, } #[derive(Clone, Copy, Debug, Eq, PartialEq)] diff --git a/jit/src/bytecode/policy_audit.rs b/jit/src/bytecode/policy_audit.rs index 05612e6af..09b7d1305 100644 --- a/jit/src/bytecode/policy_audit.rs +++ b/jit/src/bytecode/policy_audit.rs @@ -126,13 +126,13 @@ pub(super) const AUDITED_POLICIES: [AuditedOpcodePolicy; 252] = [ AuditedOpcodePolicy { id: 122, name: "make_arg_ref", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 123, name: "make_var_ref_ref", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, AuditedOpcodePolicy { id: 124, name: "make_var_ref", policy: Tier1Policy::Reject(FallbackReason::ClosureFrame) }, - AuditedOpcodePolicy { id: 125, name: "for_in_start", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, - AuditedOpcodePolicy { id: 126, name: "for_of_start", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 125, name: "for_in_start", policy: Tier1Policy::Helper(HelperId::IteratorOp) }, + AuditedOpcodePolicy { id: 126, name: "for_of_start", policy: Tier1Policy::Helper(HelperId::IteratorOp) }, AuditedOpcodePolicy { id: 127, name: "for_await_of_start", policy: Tier1Policy::Reject(FallbackReason::Async) }, - AuditedOpcodePolicy { id: 128, name: "for_in_next", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, - AuditedOpcodePolicy { id: 129, name: "for_of_next", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 128, name: "for_in_next", policy: Tier1Policy::Helper(HelperId::IteratorOp) }, + AuditedOpcodePolicy { id: 129, name: "for_of_next", policy: Tier1Policy::Helper(HelperId::IteratorOp) }, AuditedOpcodePolicy { id: 130, name: "iterator_get_value_done", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, - AuditedOpcodePolicy { id: 131, name: "iterator_close", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 131, name: "iterator_close", policy: Tier1Policy::Helper(HelperId::IteratorOp) }, AuditedOpcodePolicy { id: 132, name: "iterator_next", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 133, name: "iterator_call", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 134, name: "initial_yield", policy: Tier1Policy::Reject(FallbackReason::Generator) }, diff --git a/jit/src/bytecode/stack.rs b/jit/src/bytecode/stack.rs index a2582c9ed..ebd0e0219 100644 --- a/jit/src/bytecode/stack.rs +++ b/jit/src/bytecode/stack.rs @@ -236,6 +236,16 @@ fn transfer( return check_stack_size(snapshot, instruction, state); } + if matches!(name, "for_of_next" | "for_in_next") { + // The nominal pops are only the window the interpreter reads in + // place: the enumeration record (including its catch offset) and any + // intervening values stay on the stack unchanged, and the opcode + // pushes `value` and the boolean `done` flag above them. + state.stack.extend(popped); + state.stack.extend([SlotKind::Tagged, SlotKind::Tagged]); + return check_stack_size(snapshot, instruction, state); + } + if name == "using_dispose_init" { state.stack.push(SlotKind::Uninitialized); return check_stack_size(snapshot, instruction, state); @@ -323,8 +333,18 @@ fn merge_state( | (SlotKind::Int32 | SlotKind::Float64, SlotKind::Tagged) | (SlotKind::Int32, SlotKind::Float64) | (SlotKind::Float64, SlotKind::Int32) => SlotKind::Tagged, - // Catch offsets and uninitialized cells are verifier-only states, - // not interchangeable JS value representations. + // A lexical binding declared in a loop body (for example the + // `for (const value of values)` binding) is uninitialized on loop + // entry and initialized on the backedge. The interpreter cell + // holds the JS_UNINITIALIZED tag in the first case, which is still + // a tagged value; the join only drops the proof, and every read + // of such a binding is a runtime-checked `get_loc_check`. + (SlotKind::Tagged | SlotKind::Int32 | SlotKind::Float64, SlotKind::Uninitialized) + | (SlotKind::Uninitialized, SlotKind::Tagged | SlotKind::Int32 | SlotKind::Float64) => { + SlotKind::Tagged + } + // Catch offsets are verifier-only states, not interchangeable JS + // value representations. _ => { return Err(VerifyError::new( pc, diff --git a/jit/src/compiler/baseline.rs b/jit/src/compiler/baseline.rs index 00e7cfb64..5cdfbf713 100644 --- a/jit/src/compiler/baseline.rs +++ b/jit/src/compiler/baseline.rs @@ -32,8 +32,8 @@ use crate::{ Relocation, RelocationKind, RelocationTarget, StackMap, UnwindKind, UnwindMetadata, }, ir::{ - BaselineIr, BinaryOp, FrameSlot, FrameStateId, FrameStateKind, IrOp, PollKind, StackOp, - TaggedValue, UnaryOp, MAX_HELPER_SCRATCH_SLOTS, + BaselineIr, BinaryOp, FrameSlot, FrameStateId, FrameStateKind, IrOp, IteratorOp, PollKind, + StackOp, TaggedValue, UnaryOp, MAX_HELPER_SCRATCH_SLOTS, }, platform::{CodeAllocator, CodeMemoryError, ExecutableCode}, runtime::CompileRequest, @@ -2122,6 +2122,34 @@ fn analyze_entry_domains(ir: &BaselineIr) -> Result { + let base = frame + .stack + .len() + .checked_sub(operation.window_below()) + .ok_or(CompileFailure::InvalidArtifact)?; + match operation { + IteratorOp::ForOfStart => { + frame.stack.truncate(base); + frame.stack.extend([ + AbstractValue::unknown(), + AbstractValue::unknown(), + AbstractValue::unknown(), + ]); + } + IteratorOp::ForInStart => { + frame.stack.truncate(base); + frame.stack.push(AbstractValue::unknown()); + } + IteratorOp::ForOfNext(_) | IteratorOp::ForInNext => { + // Completion replaces the iterator with undefined. + frame.stack[base] = AbstractValue::unknown(); + frame.stack.push(AbstractValue::unknown()); + frame.stack.push(AbstractValue::known(KnownKind::Boolean)); + } + IteratorOp::Close => frame.stack.truncate(base), + } + } IrOp::GetArgument(index) => frame.stack.push( frame .arguments @@ -2410,6 +2438,9 @@ fn binary_returns_boolean(operation: BinaryOp) -> bool { fn ir_op_produces_boolean(operation: &IrOp) -> bool { match operation { IrOp::Binary(operation) => binary_returns_boolean(*operation), + // `js_for_of_next`/`js_for_in_next` and the Array values leaf always + // leave an exact JS boolean `done` flag on the stack top. + IrOp::Iterator(IteratorOp::ForOfNext(_) | IteratorOp::ForInNext) => true, _ => false, } } @@ -2572,6 +2603,21 @@ fn lower_function( .into_iter() .map(|signature| builder.import_signature(signature)) .collect::>(); + // Continuing-path Array values leaf. Absent (or an invalid table) means + // every `for_of_next` takes the exact ITERATOR_OP helper. + let array_values_next = if ir.blocks.iter().any(|block| { + block + .instructions + .iter() + .any(|instruction| matches!(instruction.op, IrOp::Iterator(IteratorOp::ForOfNext(_)))) + }) { + crate::abi::AbiInfo::linked() + .ok() + .and_then(|abi| abi.iterator_api().array_values_next) + .map(|leaf| leaf as usize) + } else { + None + }; let helper_lowering = HelperLowering { ir, frame, @@ -2871,6 +2917,20 @@ fn lower_function( depth = pattern_index + 1; set_visible_stack_depth(builder, frame, stack_base, depth, layout)?; } + IrOp::Iterator(operation) => { + let state = helper_states + .next() + .ok_or(CompileFailure::InvalidArtifact)?; + lower_iterator( + builder, + &helper_lowering, + state, + &mut depth, + operation, + array_values_next, + )?; + set_visible_stack_depth(builder, frame, stack_base, depth, layout)?; + } IrOp::GetArgument(index) => { let state = helper_states .next() @@ -5144,6 +5204,166 @@ fn lower_to_property_key( helpers.set_depth(builder, depth) } +/// Synchronous iteration opcodes run the interpreter's own stack effect on +/// the materialized frame through ITERATOR_OP. `for_of_next` first tries the +/// non-allocating Array values leaf, which revalidates the live iterator, its +/// exact built-in `next` method and the dense in-bounds element on every step; +/// any miss (holes past the dense count, non-Array receivers, user iterators, +/// completion) takes the exact helper instead. +fn lower_iterator( + builder: &mut FunctionBuilder<'_>, + helpers: &HelperLowering<'_>, + state: FrameStateId, + depth: &mut usize, + operation: IteratorOp, + array_values_next: Option, +) -> Result<(), CompileFailure> { + let stack = helpers.stack; + let before = *depth; + let base = before + .checked_sub(operation.window_below()) + .ok_or(CompileFailure::InvalidArtifact)?; + let after = before + .checked_add_signed(operation.net_push()) + .ok_or(CompileFailure::InvalidArtifact)?; + if before.max(after) > stack.len() { + return Err(CompileFailure::InvalidArtifact); + } + let (name, operand) = match operation { + IteratorOp::ForOfStart => ("for_of_start", 0), + IteratorOp::ForOfNext(extra) => ("for_of_next", u32::from(extra)), + IteratorOp::ForInStart => ("for_in_start", 0), + IteratorOp::ForInNext => ("for_in_next", 0), + IteratorOp::Close => ("iterator_close", 0), + }; + // The interpreter's `sp` when the operation throws: iterator_close has + // already dropped the catch offset and the next method. + let exception_depth = match operation { + IteratorOp::Close => before - 2, + _ => before, + }; + let arguments = [ + u32::try_from(before).map_err(|_| CompileFailure::ResourceLimit)?, + quickjs_opcode_id(name)?, + operand, + 0, + ]; + + let continuation = match (operation, array_values_next) { + (IteratorOp::ForOfNext(_), Some(leaf)) => { + let continuation = builder.create_block(); + let hit = builder.create_block(); + let miss = builder.create_block(); + let scratch = builder.create_sized_stack_slot(StackSlotData::new( + StackSlotKind::ExplicitSlot, + u32::try_from(3 * mem::size_of::()) + .map_err(|_| CompileFailure::ResourceLimit)?, + 3, + )); + let scratch = builder.ins().stack_addr(helpers.pointer_type, scratch, 0); + let iterator = use_pair(builder, stack[base]); + let next_method = use_pair(builder, stack[base + 1]); + store_jsvalue_slot(builder, scratch, 0, iterator, helpers.layout)?; + store_jsvalue_slot(builder, scratch, 1, next_method, helpers.layout)?; + let mut signature = Signature::new(builder.func.signature.call_conv); + signature.params.extend([ + AbiParam::new(helpers.pointer_type), + AbiParam::new(helpers.pointer_type), + AbiParam::new(helpers.pointer_type), + AbiParam::new(helpers.pointer_type), + ]); + signature.returns.push(AbiParam::new(types::I32)); + let signature = builder.import_signature(signature); + let target = builder.ins().iconst( + helpers.pointer_type, + i64::try_from(leaf).map_err(|_| CompileFailure::ResourceLimit)?, + ); + let ctx = builder.ins().load( + helpers.pointer_type, + MemFlags::new(), + helpers.frame, + helpers.layout.ctx, + ); + let value_bytes = i64::try_from(mem::size_of::()) + .map_err(|_| CompileFailure::ResourceLimit)?; + let next_address = builder.ins().iadd_imm(scratch, value_bytes); + let out_address = builder.ins().iadd_imm(scratch, 2 * value_bytes); + let call = super::emit_external_call( + builder, + signature, + target, + &[ctx, scratch, next_address, out_address], + helpers.pointer_type, + None, + None, + ); + let status = builder.inst_results(call)[0]; + let produced = builder.ins().icmp_imm( + IntCC::Equal, + status, + i64::from(qjs::JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_VALUE), + ); + builder.ins().brif(produced, hit, &[], miss, &[]); + builder.seal_block(hit); + builder.seal_block(miss); + + builder.switch_to_block(hit); + let value = load_jsvalue(builder, scratch, 2, helpers.layout); + define_pair(builder, stack[before], value); + let not_done = constant_pair(builder, TaggedValue::new(0, qjs::JS_TAG_BOOL as i64)); + define_pair(builder, stack[before + 1], not_done); + builder.ins().jump(continuation, &[]); + + builder.switch_to_block(miss); + Some(continuation) + } + _ => None, + }; + + helpers.invoke( + builder, + qjs::JSJitHelperId_JS_JIT_HELPER_ITERATOR_OP, + state, + before, + exception_depth, + &arguments, + )?; + let reload = |builder: &mut FunctionBuilder<'_>, index: usize| { + reload_pair( + builder, + stack[index], + helpers.stack_base, + index, + helpers.layout, + ) + }; + match operation { + IteratorOp::ForOfStart => { + for index in base..after { + reload(builder, index); + } + } + IteratorOp::ForInStart => reload(builder, base), + IteratorOp::ForOfNext(_) | IteratorOp::ForInNext => { + reload(builder, base); + reload(builder, before); + reload(builder, before + 1); + } + IteratorOp::Close => { + for (index, pair) in stack.iter().copied().enumerate().take(before).skip(base) { + clear_pair(builder, pair, helpers.stack_base, index, helpers.layout)?; + } + } + } + if let Some(continuation) = continuation { + builder.ins().jump(continuation, &[]); + builder.seal_block(continuation); + builder.switch_to_block(continuation); + } + *depth = after; + Ok(()) +} + #[allow(clippy::too_many_arguments)] // Call bytecode operands and lowering state are separate by design. fn lower_call( builder: &mut FunctionBuilder<'_>, @@ -6496,6 +6716,7 @@ mod tests { IrOp::Call { .. } => "call", IrOp::CallConstructor(_) => "call_constructor", IrOp::Regexp => "regexp", + IrOp::Iterator(_) => "iterator", IrOp::GetArgument(_) => "get_argument", IrOp::GetLocal(_) => "get_local", IrOp::GetLocalChecked(_) => "get_local_checked", diff --git a/jit/src/ir/baseline.rs b/jit/src/ir/baseline.rs index 68e7aa362..366196622 100644 --- a/jit/src/ir/baseline.rs +++ b/jit/src/ir/baseline.rs @@ -10,8 +10,8 @@ use crate::{ }; use super::{ - BinaryOp, FrameSlot, FrameState, FrameStateId, FrameStateKind, FrameStateTable, IrOp, PollKind, - StackOp, TaggedValue, UnaryOp, + BinaryOp, FrameSlot, FrameState, FrameStateId, FrameStateKind, FrameStateTable, IrOp, + IteratorOp, PollKind, StackOp, TaggedValue, UnaryOp, }; const POLL_INTERVAL: usize = 1_024; @@ -453,6 +453,7 @@ fn operation_helper_call_count(operation: &IrOp) -> usize { IrOp::Call { argc, has_this } => 1 + usize::from(*argc) + 1 + usize::from(*has_this), IrOp::CallConstructor(argc) => 1 + usize::from(*argc) + 2, IrOp::Regexp => 1, + IrOp::Iterator(_) => 1, IrOp::GetArgument(_) | IrOp::GetLocal(_) | IrOp::GetLocalChecked(_) => 1, IrOp::GetLocalPair => 2, IrOp::PutArgument { keep, .. } | IrOp::PutLocal { keep, .. } => 1 + usize::from(*keep), @@ -698,6 +699,11 @@ fn translate_instruction(instruction: &Instruction) -> Result IrOp::CallConstructor(instruction.operand_u16(1)), "regexp" => IrOp::Regexp, + "for_of_start" => IrOp::Iterator(IteratorOp::ForOfStart), + "for_of_next" => IrOp::Iterator(IteratorOp::ForOfNext(instruction.operand_u8(1))), + "for_in_start" => IrOp::Iterator(IteratorOp::ForInStart), + "for_in_next" => IrOp::Iterator(IteratorOp::ForInNext), + "iterator_close" => IrOp::Iterator(IteratorOp::Close), "get_arg" | "get_arg0" | "get_arg1" | "get_arg2" | "get_arg3" => { IrOp::GetArgument(indexed_operand(instruction).ok_or(CompileFailure::InvalidArtifact)?) } diff --git a/jit/src/ir/mod.rs b/jit/src/ir/mod.rs index 9be6ab40b..d9c0dc8c8 100644 --- a/jit/src/ir/mod.rs +++ b/jit/src/ir/mod.rs @@ -42,6 +42,6 @@ pub use scalar::{ ScalarGraph, ScalarInlineRegion, ScalarInlineStep, ScalarNumericMode, ScalarPhiInput, ScalarValue, ScalarValueId, }; -pub use types::{BinaryOp, IrOp, PollKind, StackOp, TaggedValue, UnaryOp}; +pub use types::{BinaryOp, IrOp, IteratorOp, PollKind, StackOp, TaggedValue, UnaryOp}; pub(crate) use scalar::stack_permutation; diff --git a/jit/src/ir/types.rs b/jit/src/ir/types.rs index 1ab34caf1..6035f6bd4 100644 --- a/jit/src/ir/types.rs +++ b/jit/src/ir/types.rs @@ -73,6 +73,44 @@ pub enum BinaryOp { StrictNotEqual, } +/// Synchronous iteration opcodes lowered through the exact interpreter +/// stack-effect helper (`ITERATOR_OP`). +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum IteratorOp { + /// `obj -> iter next catch_offset` + ForOfStart, + /// `iter next catch_offset [objs] -> ... value done`; the operand counts + /// the extra objects between the catch offset and the stack top. + ForOfNext(u8), + /// `obj -> enum_obj` + ForInStart, + /// `enum_obj -> enum_obj value done` + ForInNext, + /// `iter next catch_offset ->` + Close, +} + +impl IteratorOp { + /// Values removed below the stack top (the interpreter reads, and may + /// rewrite, this window in place). + pub const fn window_below(self) -> usize { + match self { + Self::ForOfStart | Self::ForInStart | Self::ForInNext => 1, + Self::ForOfNext(extra) => 3 + extra as usize, + Self::Close => 3, + } + } + + /// Net stack-depth change. + pub const fn net_push(self) -> isize { + match self { + Self::ForOfStart | Self::ForOfNext(_) | Self::ForInNext => 2, + Self::ForInStart => 0, + Self::Close => -3, + } + } +} + #[derive(Clone, Debug, Eq, PartialEq)] pub enum IrOp { Poll { state: FrameStateId, kind: PollKind }, @@ -95,6 +133,7 @@ pub enum IrOp { Call { argc: u16, has_this: bool }, CallConstructor(u16), Regexp, + Iterator(IteratorOp), GetArgument(u16), GetLocal(u16), GetLocalChecked(u16), diff --git a/jit/src/lib.rs b/jit/src/lib.rs index d9a15bd96..58f0125c0 100644 --- a/jit/src/lib.rs +++ b/jit/src/lib.rs @@ -1281,7 +1281,8 @@ fn validate_production_frame( }; let valid = match kind { SlotKind::Tagged => true, - SlotKind::Int32 | SlotKind::CatchOffset => value.tag == i64::from(qjs::JS_TAG_INT), + SlotKind::Int32 => value.tag == i64::from(qjs::JS_TAG_INT), + SlotKind::CatchOffset => value.tag == i64::from(qjs::JS_TAG_CATCH_OFFSET), SlotKind::Float64 => value.tag == i64::from(qjs::JS_TAG_FLOAT64), SlotKind::Uninitialized => value.tag == i64::from(qjs::JS_TAG_UNINITIALIZED), }; diff --git a/jit/tests/abi.rs b/jit/tests/abi.rs index bb3580e33..9a3ecdfe4 100644 --- a/jit/tests/abi.rs +++ b/jit/tests/abi.rs @@ -386,7 +386,7 @@ fn interrupt_runtime_api_is_a_versioned_exec_frame_tail_extension() { use rquickjs_core::qjs; assert_eq!(qjs::QJSJIT_RUNTIME_API_MAJOR, 1); - assert_eq!(qjs::QJSJIT_RUNTIME_API_MINOR, 9); + assert_eq!(qjs::QJSJIT_RUNTIME_API_MINOR, 10); assert_eq!( std::mem::offset_of!(qjs::JSJitExecFrame, runtime_api), std::mem::offset_of!(qjs::JSJitExecFrame, entry) @@ -419,7 +419,8 @@ fn interrupt_runtime_api_is_a_versioned_exec_frame_tail_extension() { std::mem::offset_of!(qjs::JSJitRuntimeAPI, unary_arith_slow), 192 ); - assert_eq!(std::mem::size_of::(), 200); + assert_eq!(std::mem::offset_of!(qjs::JSJitRuntimeAPI, iterator_op), 200); + assert_eq!(std::mem::size_of::(), 208); assert_eq!(qjs::JS_JIT_HELPER_GUARD_MISS, 1); } @@ -530,6 +531,7 @@ fn helper_abi_is_one_canonical_versioned_table_in_c_bindgen_and_rust() { 2, qjs::JSJitHelperOwnership_JS_JIT_HELPER_OWNED as u8, ), + ("ITERATOR_OP", 0, 0), ]; let mut count = 0_u32; @@ -574,9 +576,9 @@ fn helper_abi_is_one_canonical_versioned_table_in_c_bindgen_and_rust() { fn helper_abi_fields_are_append_only_tails() { use rquickjs_core::qjs; - assert_eq!(qjs::QJSJIT_ABI_MINOR, 24); + assert_eq!(qjs::QJSJIT_ABI_MINOR, 25); assert_eq!(qjs::QJSJIT_RUNTIME_API_MAJOR, 1); - assert_eq!(qjs::QJSJIT_RUNTIME_API_MINOR, 9); + assert_eq!(qjs::QJSJIT_RUNTIME_API_MINOR, 10); assert_eq!(qjs::QJSJIT_HELPER_ABI_VERSION, 1); assert_eq!( std::mem::offset_of!(qjs::JSJitEntryHandle, stack_map_count), @@ -766,7 +768,11 @@ fn duplicate_attachment_does_not_replace_the_first_backend() { fn every_abi_mismatch_is_rejected_before_backend_storage() { use rquickjs_jit::test_support::AbiMismatchFixture; - for required in [AbiMismatchFixture::InlineApi, AbiMismatchFixture::ArrayApi] { + for required in [ + AbiMismatchFixture::InlineApi, + AbiMismatchFixture::ArrayApi, + AbiMismatchFixture::IteratorApi, + ] { assert!( AbiMismatchFixture::ALL.contains(&required), "missing integration fixture: {required:?}" diff --git a/jit/tests/differential.rs b/jit/tests/differential.rs index 07166009d..3f6c023ae 100644 --- a/jit/tests/differential.rs +++ b/jit/tests/differential.rs @@ -64,6 +64,7 @@ enum ManifestHelper { Regexp, NewArray, NewObject, + IteratorOp, } impl ManifestHelper { @@ -90,6 +91,7 @@ impl ManifestHelper { Self::Regexp => HelperId::Regexp, Self::NewArray => HelperId::NewArray, Self::NewObject => HelperId::NewObject, + Self::IteratorOp => HelperId::IteratorOp, } } } @@ -149,6 +151,7 @@ fn required_dimensions(case: &OpcodeCase) -> BTreeSet { | ManifestHelper::CallConstructor | ManifestHelper::BinaryArithSlow | ManifestHelper::UnaryArithSlow + | ManifestHelper::IteratorOp ) ) { required.insert(Dimension::CoercionReentrancy); @@ -527,6 +530,12 @@ fn every_advertised_helper_family_has_a_real_native_execution_case() { "object", HelperId::NewObject, ), + ( + "function f(values){ let sum=0; for(const value of values) sum+=value; return sum }", + "f(new Set([20,22]))", + "for_of_next", + HelperId::IteratorOp, + ), ]; for (definition, expression, opcode, helper) in cases { differential(definition, expression) diff --git a/jit/tests/fixtures/opcode-cases.json b/jit/tests/fixtures/opcode-cases.json index 3a6818cfd..bcc158820 100644 --- a/jit/tests/fixtures/opcode-cases.json +++ b/jit/tests/fixtures/opcode-cases.json @@ -1431,6 +1431,66 @@ "ownership-gc", "coercion-reentrancy" ] + }, + { + "opcode": "for_of_start", + "definition": "function f(values){ let sum=0; for(const value of values) sum+=value; return sum }", + "expression": "(()=>{const events=[];return [f([1,2,3]),f(new Set([4,5])),f({[Symbol.iterator](){let i=0;return {next(){events.push('next'+i);return {value:i,done:i++>=3}},return(){events.push('return');return {}}}}}),events,(()=>{try{return f(42)}catch(e){return e.name}})()]})()", + "helper": "IteratorOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc", + "coercion-reentrancy" + ] + }, + { + "opcode": "for_of_next", + "definition": "function f(values){ let out=''; for(const value of values) out+=value; return out }", + "expression": "(()=>{const events=[];return [f(['a','b',{toString(){events.push('str');return 'c'}}]),f('xyz'),f({[Symbol.iterator](){let i=0;return {next(){events.push('next'+i);return {value:i,done:i++>=3}},return(){events.push('return');return {}}}}}),events,(()=>{try{return f({[Symbol.iterator](){return {next(){return 1}}}})}catch(e){return e.name}})()]})()", + "helper": "IteratorOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc", + "coercion-reentrancy" + ] + }, + { + "opcode": "iterator_close", + "definition": "function f(values,stop){ let sum=0; for(const value of values){ if(value===stop) break; sum+=value } return sum }", + "expression": "(()=>{const events=[];return [f([1,2,3],2),f({[Symbol.iterator](){let i=0;return {next(){events.push('next'+i);return {value:i,done:i++>=3}},return(){events.push('return');return {}}}}},1),events,(()=>{try{return f({[Symbol.iterator](){return {next(){return {value:1,done:false}},return(){throw new RangeError('close')}}}},1)}catch(e){return e.name}})()]})()", + "helper": "IteratorOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc", + "coercion-reentrancy" + ] + }, + { + "opcode": "for_in_start", + "definition": "function f(o){ let keys=''; for(const key in o) keys+=key; return keys }", + "expression": "(()=>{const events=[];return [f({a:1,b:2}),f([7,8]),f(null),f(new Proxy({p:1},{ownKeys(t){events.push('keys');return Reflect.ownKeys(t)}})),events,(()=>{try{return f(new Proxy({},{ownKeys(){throw new EvalError('keys')}}))}catch(e){return e.name}})()]})()", + "helper": "IteratorOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc", + "coercion-reentrancy" + ] + }, + { + "opcode": "for_in_next", + "definition": "function f(o,g){ let keys=''; for(const key in o){ keys+=key; g(o) } return keys }", + "expression": "(()=>{const events=[];return [f({a:1,b:2,c:3},o=>{delete o.c;events.push('g')}),f('xy',()=>{}),events,(()=>{try{return f(new Proxy({a:1},{getOwnPropertyDescriptor(){throw new URIError('d')}}),()=>{})}catch(e){return e.name}})()]})()", + "helper": "IteratorOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc", + "coercion-reentrancy" + ] } ] } diff --git a/jit/tests/opcodes.rs b/jit/tests/opcodes.rs index 77d43fadd..c461dd989 100644 --- a/jit/tests/opcodes.rs +++ b/jit/tests/opcodes.rs @@ -113,6 +113,19 @@ fn policies_are_semantic_and_categorized() { Tier1Policy::Helper(HelperId::CallConstructor) ); assert_eq!(by_name("regexp"), Tier1Policy::Helper(HelperId::Regexp)); + for iteration in [ + "for_of_start", + "for_of_next", + "for_in_start", + "for_in_next", + "iterator_close", + ] { + assert_eq!( + by_name(iteration), + Tier1Policy::Helper(HelperId::IteratorOp), + "{iteration}" + ); + } assert_eq!( by_name("eval"), Tier1Policy::Reject(FallbackReason::DirectEval) diff --git a/jit/tests/required_features.rs b/jit/tests/required_features.rs index b5dfd588c..652ab6d46 100644 --- a/jit/tests/required_features.rs +++ b/jit/tests/required_features.rs @@ -43,6 +43,7 @@ fn native_semantics_targets_require_their_execution_features() { "tier1_calls_properties", "tier1_constructors", "tier1_globals", + "tier1_iteration", "tier1_regexp", ] { let target = package["targets"] diff --git a/jit/tests/support/mod.rs b/jit/tests/support/mod.rs index bc1ada728..7d7e07482 100644 --- a/jit/tests/support/mod.rs +++ b/jit/tests/support/mod.rs @@ -508,6 +508,7 @@ pub struct DifferentialRun { expected_opcode: Option, expected_helper: Option, expected_ownership_helper_counts: Option<(u64, u64)>, + expected_helper_count: Option<(u32, u64)>, stress_gc: bool, expect_deopt: bool, } @@ -521,6 +522,7 @@ pub fn differential(definition: &str, expression: &str) -> DifferentialRun { expected_opcode: None, expected_helper: None, expected_ownership_helper_counts: None, + expected_helper_count: None, stress_gc: false, expect_deopt: false, } @@ -583,6 +585,13 @@ impl DifferentialRun { self } + /// Exact number of calls of one runtime helper (by generated helper ID) + /// while evaluating the expression. + pub fn expect_helper_call_count(mut self, helper_id: u32, count: u64) -> Self { + self.expected_helper_count = Some((helper_id, count)); + self + } + /// The native body is expected to hand the frame back to the /// interpreter mid-function (a use-site guard failed) at least once. pub fn expect_deopt(mut self) -> Self { @@ -633,6 +642,15 @@ impl DifferentialRun { ); } + if let Some((helper_id, expected_count)) = self.expected_helper_count { + let mut count = 0_u64; + assert_eq!( + unsafe { rquickjs_core::qjs::JS_JitGetHelperCount(rt, helper_id, &mut count) }, + 0 + ); + assert_eq!(count, expected_count, "helper {helper_id} call count"); + } + let mut trace_len = 0; let mut overflowed = 0; assert_eq!( @@ -742,6 +760,7 @@ impl DifferentialRun { rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_TO_PROPKEY } HelperId::GetGlobal => rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_GET_GLOBAL, + HelperId::IteratorOp => rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_ITERATOR_OP, }; let expected_opcode = self .expected_opcode @@ -1328,6 +1347,7 @@ static SYNTHETIC_RUNTIME_API: rquickjs_core::qjs::JSJitRuntimeAPI = regexp: Some(synthetic_map_out_two_unavailable), binary_arith_slow: Some(synthetic_map_out_two_op_unavailable), unary_arith_slow: Some(synthetic_map_out_in_op_unavailable), + iterator_op: Some(synthetic_map_out_two_op_unavailable), }; /// Result observed after invoking a generated aggregate-return entry point. @@ -1667,11 +1687,12 @@ pub enum AbiMismatchFixture { ElementLayout, InlineApi, ArrayApi, + IteratorApi, BackendVTableLayout, } impl AbiMismatchFixture { - pub const ALL: [Self; 19] = [ + pub const ALL: [Self; 20] = [ Self::SourceRevision, Self::OpcodeFingerprint, Self::ValueLayout, @@ -1690,6 +1711,7 @@ impl AbiMismatchFixture { Self::ElementLayout, Self::InlineApi, Self::ArrayApi, + Self::IteratorApi, Self::BackendVTableLayout, ]; @@ -1715,6 +1737,7 @@ impl AbiMismatchFixture { Self::ElementLayout => AbiMismatch::StructureLayout(AbiStructure::ElementLayout), Self::InlineApi => AbiMismatch::StructureLayout(AbiStructure::InlineApi), Self::ArrayApi => AbiMismatch::StructureLayout(AbiStructure::ArrayApi), + Self::IteratorApi => AbiMismatch::StructureLayout(AbiStructure::IteratorApi), Self::BackendVTableLayout => AbiMismatch::StructureLayout(AbiStructure::BackendVTable), } } diff --git a/jit/tests/tier1_iteration.rs b/jit/tests/tier1_iteration.rs new file mode 100644 index 000000000..4ae733ce6 --- /dev/null +++ b/jit/tests/tier1_iteration.rs @@ -0,0 +1,339 @@ +#![cfg(all( + target_endian = "little", + any(target_arch = "x86_64", target_arch = "aarch64"), + any(target_os = "linux", target_os = "macos", target_os = "windows"), + not(all(target_os = "windows", target_arch = "aarch64")) +))] + +//! Tier 1 lowering of the synchronous iteration opcodes (`for_of_*`, +//! `for_in_*`, `iterator_close`). Every case compares the forced Tier 1 +//! execution against a fresh interpreter, including iterator-close ordering on +//! `break` and on exceptions thrown by the loop body. + +use rquickjs::{Context, Runtime}; +use rquickjs_jit::bytecode::{FallbackReason, HelperId}; +use rquickjs_jit::test_support::{assert_tier1_rejected, differential}; +use rquickjs_jit::{Jit, JitConfig, JitTierPolicy}; +use std::time::{Duration, Instant}; + +const ITERATOR_OP: u32 = rquickjs::qjs::JSJitHelperId_JS_JIT_HELPER_ITERATOR_OP; +const SUM_OF: &str = + "function f(values){let sum=0;for(const value of values)sum+=value;return sum}"; + +#[test] +fn packed_array_for_of_uses_the_array_values_leaf() { + // Dense packed arrays never reach the generic helper for their elements; + // only completion (one `next` past the end) takes ITERATOR_OP. + // for_of_start, the completing for_of_next and iterator_close only. + differential(SUM_OF, "f([1,2,3,4,5,6,7,8,9,10])") + .force_baseline() + .expect_executed_opcode("for_of_next") + .expect_helper_call_count(ITERATOR_OP, 3) + .assert_same(); + differential( + SUM_OF, + "f([0.5,'x',null,undefined,{},[1],1n,Symbol.iterator.description])", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("for_of_next") + .expect_helper_call_count(ITERATOR_OP, 3) + .assert_same(); + differential(SUM_OF, "f(['a','b',{toString(){return 'c'}}])") + .force_baseline() + .stress_gc() + .expect_executed_opcode("for_of_start") + .expect_helper(HelperId::IteratorOp) + .assert_same(); + differential(SUM_OF, "f([])") + .force_baseline() + .expect_executed_opcode("iterator_close") + .assert_same(); +} + +#[test] +fn non_packed_and_non_array_iterables_take_the_exact_helper() { + for expression in [ + // Holes read through the prototype chain. + "(()=>{Array.prototype[1]=40;try{return f([1,,3])}finally{delete Array.prototype[1]}})()", + // `length` above the dense count keeps a fast array. + "(()=>{const a=[1,2];a.length=5;return String(f(a))})()", + "f(new Set([1,2,3]))", + "f(new Map([[1,2]]).values())", + "f('abc')", + "f(new Int32Array([4,5,6]))", + "f((function*(){yield 1;yield 2})())", + "f({[Symbol.iterator](){let i=0;return {next(){return {value:i,done:i++>=3}}}}})", + "(()=>{const a=[1,2,3];a.foo=1;return f(a.entries().next().value)})()", + ] { + differential(SUM_OF, expression) + .force_baseline() + .stress_gc() + .expect_executed_opcode("for_of_next") + .expect_helper(HelperId::IteratorOp) + .assert_same(); + } +} + +#[test] +fn array_mutation_during_iteration_matches_the_builtin_iterator() { + let definition = "function f(values){let out=[];for(const value of values){out.push(value);if(value===2)values.push(9);if(value===9)values.length=0}return out.join(',')}"; + differential(definition, "f([1,2,3])") + .force_baseline() + .expect_executed_opcode("for_of_next") + .assert_same(); + let shrink = "function f(values){let out=[];for(const value of values){out.push(value);values.pop()}return out.join(',')}"; + differential(shrink, "f([1,2,3,4,5])") + .force_baseline() + .expect_executed_opcode("for_of_next") + .assert_same(); +} + +#[test] +fn replaced_array_iteration_protocol_is_observed() { + // A user-installed %ArrayIteratorPrototype%.next is called for every step. + differential( + SUM_OF, + "(()=>{const proto=Object.getPrototypeOf([][Symbol.iterator]());const next=proto.next;let calls=0;proto.next=function(){calls++;return next.call(this)};try{return f([1,2,3])+':'+calls}finally{proto.next=next}})()", + ) + .force_baseline() + .expect_executed_opcode("for_of_next") + .expect_helper(HelperId::IteratorOp) + .assert_same(); + // A replaced Array.prototype[Symbol.iterator] yields a user iterator. + differential( + SUM_OF, + "(()=>{const original=Array.prototype[Symbol.iterator];Array.prototype[Symbol.iterator]=function*(){yield 40;yield 2};try{return f([1,2,3])}finally{Array.prototype[Symbol.iterator]=original}})()", + ) + .force_baseline() + .expect_executed_opcode("for_of_start") + .expect_helper(HelperId::IteratorOp) + .assert_same(); +} + +const LOGGING_ITERABLE: &str = "function iterable(log,limit,failNext){return {[Symbol.iterator](){log.push('iter');let i=0;return {next(){log.push('next'+i);if(failNext&&i===failNext)throw new Error('next');return {value:i,done:i++>=limit}},return(){log.push('return');return {}}}}}}"; + +#[test] +fn break_closes_the_iterator_exactly_once() { + let definition = format!( + "{LOGGING_ITERABLE}\nfunction f(it,stop){{let sum=0;for(const value of it){{if(value===stop)break;sum+=value}}return sum}}" + ); + differential( + &definition, + "(()=>{const log=[];const r=f(iterable(log,5),2);return r+':'+log.join(',')})()", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("iterator_close") + .expect_helper(HelperId::IteratorOp) + .assert_same(); + // Completion does not call `return`. + differential( + &definition, + "(()=>{const log=[];const r=f(iterable(log,3),99);return r+':'+log.join(',')})()", + ) + .force_baseline() + .expect_executed_opcode("iterator_close") + .assert_same(); +} + +#[test] +fn iterator_close_errors_and_non_object_results_propagate() { + let definition = "function f(it){for(const value of it){if(value===1)break}return 'done'}"; + differential( + definition, + "(()=>{try{return f({[Symbol.iterator](){let i=0;return {next(){return {value:i++,done:false}},return(){throw new RangeError('close')}}}})}catch(e){return e.name}})()", + ) + .force_baseline() + .expect_executed_opcode("iterator_close") + .assert_same(); + differential( + definition, + "(()=>{try{return f({[Symbol.iterator](){let i=0;return {next(){return {value:i++,done:false}},return(){return 1}}}})}catch(e){return e.name}})()", + ) + .force_baseline() + .expect_executed_opcode("iterator_close") + .assert_same(); +} + +#[test] +fn body_exceptions_close_the_iterator_with_a_throw_completion() { + let definition = format!( + "{LOGGING_ITERABLE}\nfunction f(it,g){{let sum=0;for(const value of it)sum+=g(value);return sum}}" + ); + differential( + &definition, + "(()=>{const log=[];try{f(iterable(log,5),v=>{if(v===2)throw new TypeError('body');return v})}catch(e){log.push(e.name)}return log.join(',')})()", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("for_of_next") + .assert_same(); + // The throw completion wins over an exception thrown by `return`. + differential( + &definition, + "(()=>{const log=[];const it={[Symbol.iterator](){return {next(){return {value:1,done:false}},return(){log.push('return');throw new RangeError('close')}}}};try{f(it,()=>{throw new TypeError('body')})}catch(e){log.push(e.name)}return log.join(',')})()", + ) + .force_baseline() + .expect_executed_opcode("for_of_next") + .assert_same(); + // Packed arrays: the leaf-produced value throws in the body. + differential( + &definition, + "(()=>{try{return f([1,2,3],v=>{if(v===3)throw new Error('x');return v})}catch(e){return e.message}})()", + ) + .force_baseline() + .expect_executed_opcode("for_of_next") + .assert_same(); +} + +#[test] +fn iterator_protocol_errors_do_not_close_the_iterator() { + let definition = format!( + "{LOGGING_ITERABLE}\nfunction f(it){{let sum=0;for(const value of it)sum+=value;return sum}}" + ); + differential( + &definition, + "(()=>{const log=[];try{f(iterable(log,5,2))}catch(e){log.push(e.message)}return log.join(',')})()", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("for_of_next") + .expect_helper(HelperId::IteratorOp) + .assert_same(); + for expression in [ + "(()=>{try{return f(42)}catch(e){return e.name}})()", + "(()=>{try{return f({[Symbol.iterator](){return 1}})}catch(e){return e.name}})()", + "(()=>{try{return f({[Symbol.iterator](){return {next(){return 1}}}})}catch(e){return e.name}})()", + "(()=>{try{return f({[Symbol.iterator](){return {get next(){throw new SyntaxError('n')}}}})}catch(e){return e.name}})()", + "(()=>{try{return f({[Symbol.iterator](){return {next(){return {get done(){throw new URIError('d')}}}}}})}catch(e){return e.name}})()", + ] { + differential(&definition, expression) + .force_baseline() + .stress_gc() + .expect_executed_opcode("for_of_start") + .assert_same(); + } +} + +#[test] +fn nested_for_of_loops_close_inner_iterators_on_outer_break() { + let definition = format!( + "{LOGGING_ITERABLE}\nfunction f(log){{let out=0;outer:for(const a of iterable(log,3)){{for(const b of iterable(log,3)){{if(a===1&&b===1)break outer;out+=a*10+b}}}}return out}}" + ); + differential( + &definition, + "(()=>{const log=[];return f(log)+':'+log.join(',')})()", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("iterator_close") + .assert_same(); +} + +#[test] +fn for_in_enumerates_like_the_interpreter() { + let definition = + "function f(o,g){let keys=[];for(const key in o){keys.push(key);if(key==='b')g(o)}return keys.join(',')}"; + for expression in [ + "f({a:1,b:2,c:3,d:4},o=>{delete o.c})", + "f({a:1,b:2,c:3},o=>{o.z=1})", + "f([5,6,7],()=>{})", + "f(Object.create({inherited:1},{own:{value:2,enumerable:true}}),()=>{})", + "f(null,()=>{})", + "f(undefined,()=>{})", + "f('xy',()=>{})", + "f(new Proxy({p:1,q:2},{}),()=>{})", + ] { + differential(definition, expression) + .force_baseline() + .stress_gc() + .expect_executed_opcode("for_in_next") + .expect_helper(HelperId::IteratorOp) + .assert_same(); + } + differential( + definition, + "(()=>{try{return f(new Proxy({},{ownKeys(){throw new EvalError('keys')}}),()=>{})}catch(e){return e.name}})()", + ) + .force_baseline() + .expect_executed_opcode("for_in_start") + .assert_same(); +} + +#[test] +fn array_destructuring_and_rest_use_offset_for_of_next() { + // A `...rest` element keeps an Int32 index on the operand stack across + // its loop, which the verifier's exact operand-stack merge rejects; the + // offset form of `for_of_next` is covered by the nested elements here. + let definition = "function f(values){const [a,,b='seven']=values;const [[c,d]=[5,6]]=[];return a+':'+b+':'+c+d}"; + for expression in [ + "f([1,2,3,4,5])", + "f([1])", + "f('hello')", + "f(new Set([9,8]))", + ] { + differential(definition, expression) + .force_baseline() + .stress_gc() + .expect_executed_opcode("for_of_next") + .assert_same(); + } + let close = "function f(it){const [a,b]=it;return a+b}"; + differential( + &format!("{LOGGING_ITERABLE}\n{close}"), + "(()=>{const log=[];return f(iterable(log,9))+':'+log.join(',')})()", + ) + .force_baseline() + .expect_executed_opcode("iterator_close") + .assert_same(); +} + +#[test] +fn return_inside_for_of_stays_in_the_interpreter() { + // `return` from inside a for-of emits `nip_catch`, an exception-region + // opcode Tier 1 does not lower yet. + assert_tier1_rejected( + "function f(values){for(const value of values)if(value>1)return value;return 0}", + "f([1,2,3])", + FallbackReason::ExceptionRegion, + ); +} + +#[test] +fn production_tiering_enters_native_code_for_a_for_of_kernel() { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .tier_policy(JitTierPolicy::BaselineOnly) + .call_threshold(1) + .loop_threshold(1) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| { + ctx.eval::<(), _>( + "globalThis.kernel=function kernel(n){const values=[];let sum=0;for(let i=0;i 0 { + break; + } + } + assert_eq!(result, expected); + assert!(jit.metrics().native_entries > 0, "{:?}", jit.metrics()); +} diff --git a/sys/build.rs b/sys/build.rs index 38375afaa..0f1e4b21f 100644 --- a/sys/build.rs +++ b/sys/build.rs @@ -228,7 +228,7 @@ QJSJIT_HELPER_LIST(QJSJIT_EMIT) flags, }); } - assert_eq!(helpers.len(), 24, "canonical append-only helper count"); + assert_eq!(helpers.len(), 25, "canonical append-only helper count"); let u32_args = |signature: &str| match signature { "FRAME" => 0, diff --git a/sys/build_support/patch.rs b/sys/build_support/patch.rs index 41317e2f2..51ac1a3c5 100644 --- a/sys/build_support/patch.rs +++ b/sys/build_support/patch.rs @@ -1,7 +1,7 @@ use std::{fs, io, path::Path}; const QUICKJS_BASELINE_FNV64: u64 = 0x3302_116a_b0fc_269c; -const EXPECTED_PATCHES: [(&str, u64); 19] = [ +const EXPECTED_PATCHES: [(&str, u64); 20] = [ ("0001-rquickjs-jit.patch", 0x18d7_40cc_6943_30cf), ("0002-runtime-feedback.patch", 0xe4c1_ed52_ee5d_d469), ("0003-element-layout.patch", 0xa12c_ccd2_7420_e880), @@ -21,6 +21,7 @@ const EXPECTED_PATCHES: [(&str, u64); 19] = [ ("0018-inline-frame-recovery.patch", 0xb971d61edccde229), ("0019-array-feedback.patch", 0x11c3cb13a8297dd4), ("0020-typed-array-guard.patch", 0x5d10_38a3_8bc9_9821), + ("0028-tier1-iteration.patch", 0xef33_64c8_cb3b_6448), ]; pub(crate) const BASELINE_FILES: [&str; 19] = [ "api-test.c", @@ -60,9 +61,9 @@ const PATCHED_FILE_FINGERPRINTS: [(&str, u64); 21] = [ ("list.h", 0xb337_70f7_b76d_a3d8), ("quickjs-atom.h", 0x30b4_9116_b6a2_aa99), ("quickjs-c-atomics.h", 0x490b_0f29_f631_3fc0), - ("quickjs.c", 0xc535_b5f1_48e7_9b27), - ("quickjs-jit.h", 0x0188_04ad_21d2_95a7), - ("quickjs-jit-helpers.h", 0xa469_e80f_be47_404c), + ("quickjs.c", 0xb995_a799_0b78_bd7b), + ("quickjs-jit.h", 0x1eb1_418e_159b_8fa7), + ("quickjs-jit-helpers.h", 0xceeb_da36_0acf_7633), ("quickjs-opcode.h", 0x3d05_cfdf_5cf7_2930), ("quickjs.h", 0x4831_2cde_9c2f_a5ee), ]; diff --git a/sys/patches/0028-tier1-iteration.patch b/sys/patches/0028-tier1-iteration.patch new file mode 100644 index 000000000..c19aabb69 --- /dev/null +++ b/sys/patches/0028-tier1-iteration.patch @@ -0,0 +1,258 @@ +diff --git a/quickjs-jit-helpers.h b/quickjs-jit-helpers.h +--- a/quickjs-jit-helpers.h ++++ b/quickjs-jit-helpers.h +@@ -129,6 +129,9 @@ + THROWING | ALLOCATING | REENTRANT | FINALIZING) \ + X(UNARY_ARITH_SLOW, unary_arith_slow, JS_JitHelperUnaryArithSlow, \ + MAP_OUT_IN_OP, 2, OWNED, CONSUMED, NONE, NONE, OWNED, \ ++ THROWING | ALLOCATING | REENTRANT | FINALIZING) \ ++ X(ITERATOR_OP, iterator_op, JS_JitHelperIteratorOp, MAP_OUT_TWO_OP, 0, \ ++ NONE, NONE, NONE, NONE, NONE, \ + THROWING | ALLOCATING | REENTRANT | FINALIZING) + + typedef enum JSJitHelperId { +diff --git a/quickjs-jit.h b/quickjs-jit.h +--- a/quickjs-jit.h ++++ b/quickjs-jit.h +@@ -17,7 +17,7 @@ + #endif + + #define QJSJIT_ABI_MAJOR 1u +-#define QJSJIT_ABI_MINOR 24u ++#define QJSJIT_ABI_MINOR 25u + + /* Frame flags are append-only. Stress collection is a test/debug contract: + * every helper performs an allocating cycle-GC point before and after its +@@ -131,6 +131,37 @@ + + JS_EXTERN const JSJitArrayAPI *JS_JitGetArrayAPI(uint32_t version); + ++/* Versioned continuing-path leaf for `for_of_next` over the built-in Array ++ * values iterator. It performs no allocation, throw, finalization or VM ++ * reentry. VALUE is returned only when the live iterator is an Array values ++ * iterator whose `next` method is the exact built-in, the iterated object is ++ * a fast Array and the current index is inside its dense storage; it then ++ * advances the iterator index and stores an owned (duplicated) element in ++ * *out, exactly as the built-in `next` would. Every other state is a MISS ++ * that leaves the iterator and *out untouched so the caller can run the ++ * generic interpreter operation. */ ++#define QJSJIT_ITERATOR_API_VERSION 1u ++ ++typedef enum JSJitIteratorQueryStatus { ++ JS_JIT_ITERATOR_QUERY_INVALID = -1, ++ JS_JIT_ITERATOR_QUERY_MISS = 0, ++ JS_JIT_ITERATOR_QUERY_VALUE = 1, ++} JSJitIteratorQueryStatus; ++ ++typedef int JSJitArrayValuesNextFunc(JSContext *ctx, const JSValue *iterator, ++ const JSValue *next_method, ++ JSValue *out); ++ ++typedef struct JSJitIteratorAPI { ++ uint32_t struct_size; ++ uint32_t version; ++ uint32_t effects; ++ uint32_t reserved; ++ JSJitArrayValuesNextFunc *array_values_next; ++} JSJitIteratorAPI; ++ ++JS_EXTERN const JSJitIteratorAPI *JS_JitGetIteratorAPI(uint32_t version); ++ + #define JS_JIT_FEEDBACK_ARRAY_STORE (1U << 6) + #define JS_JIT_FEEDBACK_ARRAY_LENGTH (1U << 7) + #define JS_JIT_FEEDBACK_ARRAY_EXOTIC (1U << 8) +@@ -273,7 +304,7 @@ + * prefix and helper ID zero; every later helper is an append-only v1.1 tail. + */ + #define QJSJIT_RUNTIME_API_MAJOR 1u +-#define QJSJIT_RUNTIME_API_MINOR 9u ++#define QJSJIT_RUNTIME_API_MINOR 10u + + #define QJSJIT_RUNTIME_FIELD_FRAME(field) \ + JSJitHelperStatus (*field)(JSJitExecFrame *frame) +diff --git a/quickjs.c b/quickjs.c +--- a/quickjs.c ++++ b/quickjs.c +@@ -20098,6 +20098,124 @@ + return qjsjit_finish_output(frame, out); + } + ++/* Exact interpreter stack effects for the synchronous iteration opcodes. ++ * `depth` is the bytecode-visible stack depth before the opcode. Native code ++ * materializes the complete frame, including the opcode's push window (which ++ * must still be JS_UNDEFINED), and reloads the affected slots afterwards. The ++ * interpreter routines run directly on the materialized stack, so every owner ++ * stays C-visible across user iterator code, allocation and cycle collection. ++ * On a language exception every slot at or above the opcode's interpreter ++ * exception depth is JS_UNDEFINED and every slot below it is exactly what the ++ * interpreter's unwinder observes, including the for-of catch offset that ++ * closes the iterator with a throw completion. */ ++JSJitHelperStatus JS_JitHelperIteratorOp(JSJitExecFrame *frame, ++ uint32_t stack_map_id, uint32_t depth, ++ uint32_t opcode, uint32_t operand, ++ uint32_t reserved) ++{ ++ JSFunctionBytecode *b; ++ JSStackFrame *sf; ++ JSContext *ctx; ++ JSValue *sp; ++ uint32_t visible, pops, window, exception_depth, success_depth, i; ++ int status; ++ ++ if (qjsjit_validate_helper_frame(frame, true, stack_map_id, &b, &sf) < 0) ++ return JS_JIT_HELPER_EXCEPTION; ++ qjsjit_count_helper(frame, JS_JIT_HELPER_ITERATOR_OP); ++ ctx = frame->ctx; ++ switch (opcode) { ++ case OP_for_of_start: ++ pops = 1; ++ window = 2; ++ break; ++ case OP_for_of_next: ++ if (operand > UINT8_MAX) ++ return qjsjit_reject_helper(frame, "invalid JIT for-of offset"); ++ pops = 3 + operand; ++ window = 2; ++ break; ++ case OP_for_in_start: ++ pops = 1; ++ window = 0; ++ break; ++ case OP_for_in_next: ++ pops = 1; ++ window = 2; ++ break; ++ case OP_iterator_close: ++ pops = 3; ++ window = 0; ++ break; ++ default: ++ return qjsjit_reject_helper(frame, "invalid JIT iterator opcode"); ++ } ++ if (opcode != OP_for_of_next && operand != 0) ++ return qjsjit_reject_helper(frame, "invalid JIT iterator operand"); ++ visible = (uint32_t)(frame->stack_top - frame->stack_base); ++ if (reserved != 0 || depth < pops || depth > b->stack_size || ++ visible > b->stack_size || depth + window > visible) ++ return qjsjit_reject_helper(frame, "invalid JIT iterator stack"); ++ sp = frame->stack_base + depth; ++ for (i = 0; i < window; i++) { ++ if (!JS_IsUndefined(sp[i])) ++ return qjsjit_reject_helper(frame, "occupied JIT iterator output"); ++ } ++ if (opcode == OP_iterator_close) { ++ /* iter_obj next catch_offset -> ; the catch offset is dropped and the ++ * next method released before closing, exactly as the interpreter ++ * does, so neither is live if closing throws. */ ++ exception_depth = depth - 2; ++ success_depth = depth - 3; ++ JS_FreeValue(ctx, sp[-1]); ++ sp[-1] = JS_UNDEFINED; ++ JS_FreeValue(ctx, sp[-2]); ++ sp[-2] = JS_UNDEFINED; ++ } else { ++ exception_depth = depth; ++ success_depth = depth + window; ++ } ++ if (qjsjit_stress_gc(frame) < 0) ++ return JS_JIT_HELPER_EXCEPTION; ++ switch (opcode) { ++ case OP_for_of_start: ++ status = js_for_of_start(ctx, sp, false); ++ if (status == 0) ++ sp[1] = JS_NewCatchOffset(ctx, 0); ++ break; ++ case OP_for_of_next: ++ status = js_for_of_next(ctx, sp, -3 - (int)operand); ++ break; ++ case OP_for_in_start: ++ status = js_for_in_start(ctx, sp); ++ break; ++ case OP_for_in_next: ++ status = js_for_in_next(ctx, sp); ++ break; ++ default: ++ status = 0; ++ if (!JS_IsUndefined(sp[-3])) { ++ status = JS_IteratorClose(ctx, sp[-3], false); ++ if (status == 0) ++ JS_FreeValue(ctx, sp[-3]); ++ } ++ if (status == 0) ++ sp[-3] = JS_UNDEFINED; ++ break; ++ } ++ if (status < 0) { ++ qjsjit_stress_gc_after_exception(frame); ++ return JS_JIT_HELPER_EXCEPTION; ++ } ++ if (qjsjit_stress_gc(frame) == JS_JIT_HELPER_OK) ++ return JS_JIT_HELPER_OK; ++ /* An allocation failure after a completed operation reports the pending ++ * exception at the interpreter's exception depth for this opcode. */ ++ for (i = exception_depth; i < success_depth; i++) ++ qjsjit_consume_slot(ctx, frame->stack_base + i); ++ return JS_JIT_HELPER_EXCEPTION; ++} ++ + #endif + + #ifdef CONFIG_JIT_ABI +@@ -62243,6 +62361,57 @@ + }; + return version == QJSJIT_ARRAY_API_VERSION ? &api : NULL; + } ++ ++/* A leaf: no allocation, property lookup, exception, finalization, callback ++ * or VM reentry. The only effects on a VALUE result are the ones the built-in ++ * %ArrayIteratorPrototype%.next performs for an in-bounds dense element: the ++ * iterator index advances and the element is duplicated into *out. Every ++ * other state misses before touching anything. */ ++static int qjsjit_array_values_next(JSContext *ctx, const JSValue *iterator, ++ const JSValue *next_method, JSValue *out) ++{ ++ JSObject *iterator_object, *method, *array; ++ JSArrayIteratorData *it; ++ uint32_t idx; ++ ++ if (!ctx || !iterator || !next_method || !out) ++ return JS_JIT_ITERATOR_QUERY_INVALID; ++ if (JS_VALUE_GET_TAG(*iterator) != JS_TAG_OBJECT || ++ JS_VALUE_GET_TAG(*next_method) != JS_TAG_OBJECT) ++ return JS_JIT_ITERATOR_QUERY_MISS; ++ iterator_object = JS_VALUE_GET_OBJ(*iterator); ++ method = JS_VALUE_GET_OBJ(*next_method); ++ if (iterator_object->class_id != JS_CLASS_ARRAY_ITERATOR || ++ method->class_id != JS_CLASS_C_FUNCTION || ++ method->u.cfunc.cproto != JS_CFUNC_iterator_next || ++ method->u.cfunc.c_function.iterator_next != js_array_iterator_next) ++ return JS_JIT_ITERATOR_QUERY_MISS; ++ it = iterator_object->u.array_iterator_data; ++ if (!it || it->kind != JS_ITERATOR_KIND_VALUE || ++ JS_VALUE_GET_TAG(it->obj) != JS_TAG_OBJECT) ++ return JS_JIT_ITERATOR_QUERY_MISS; ++ array = JS_VALUE_GET_OBJ(it->obj); ++ /* A fast Array's own `length` data property is never below its dense ++ * count, so an in-bounds dense index is exactly the built-in's ++ * `idx < length` element read, with no prototype or getter lookup. */ ++ if (array->class_id != JS_CLASS_ARRAY || !array->fast_array) ++ return JS_JIT_ITERATOR_QUERY_MISS; ++ idx = it->idx; ++ if (idx >= array->u.array.count) ++ return JS_JIT_ITERATOR_QUERY_MISS; ++ it->idx = idx + 1; ++ *out = js_dup(array->u.array.u.values[idx]); ++ return JS_JIT_ITERATOR_QUERY_VALUE; ++} ++ ++const JSJitIteratorAPI *JS_JitGetIteratorAPI(uint32_t version) ++{ ++ static const JSJitIteratorAPI api = { ++ sizeof(JSJitIteratorAPI), QJSJIT_ITERATOR_API_VERSION, 0, 0, ++ qjsjit_array_values_next, ++ }; ++ return version == QJSJIT_ITERATOR_API_VERSION ? &api : NULL; ++} + #endif + + static JSValue js_typed_array_get_buffer(JSContext *ctx, JSValueConst this_val) diff --git a/sys/patches/README.md b/sys/patches/README.md index 931cf9bd3..1768fbcb7 100644 --- a/sys/patches/README.md +++ b/sys/patches/README.md @@ -60,6 +60,19 @@ data pointer before returning `{ data, count, mode }`. The table advertises zero effects and the receiver is passed by pointer so generated calls do not depend on platform-specific aggregate argument classification. +`0028-tier1-iteration.patch` adds ABI 1.25 (runtime API 1.10). The +append-only `ITERATOR_OP` helper runs the interpreter's own stack effect for +`for_of_start`, `for_of_next`, `for_in_start`, `for_in_next` and +`iterator_close` on the materialized native frame, so iterator objects, next +methods and for-of catch offsets stay C-visible across user iterator code, and +an exception leaves exactly the stack the interpreter unwinder expects (which +closes the iterator with a throw completion). The versioned +`JS_JitGetIteratorAPI` table adds a leaf for `for_of_next`: it revalidates a +built-in Array values iterator, its exact built-in `next` method, a fast Array +target and an in-bounds dense index on every step, then advances the iterator +and returns a duplicated element. It performs no allocation, throw, +finalization or reentry; every other state misses without touching anything. + The build accepts only the patch names and byte digests listed in `build_support/patch.rs`, then verifies the complete patched source manifest. Keeping the baseline and patch separate makes diff --git a/sys/src/bindings/aarch64-apple-darwin.rs b/sys/src/bindings/aarch64-apple-darwin.rs index a64c09ee5..2fb1aee81 100644 --- a/sys/src/bindings/aarch64-apple-darwin.rs +++ b/sys/src/bindings/aarch64-apple-darwin.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,7 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_ITERATOR_API_VERSION: u32 = 1; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -113,7 +114,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2275,6 +2276,45 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_INVALID: JSJitIteratorQueryStatus = -1; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_MISS: JSJitIteratorQueryStatus = 0; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_VALUE: JSJitIteratorQueryStatus = 1; +pub type JSJitIteratorQueryStatus = ::core::ffi::c_int; +pub type JSJitArrayValuesNextFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + iterator: *const JSValue, + next_method: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitIteratorAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub reserved: u32, + pub array_values_next: JSJitArrayValuesNextFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitIteratorAPI"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitIteratorAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitIteratorAPI::struct_size"] + [::core::mem::offset_of!(JSJitIteratorAPI, struct_size) - 0usize]; + ["Offset of field: JSJitIteratorAPI::version"] + [::core::mem::offset_of!(JSJitIteratorAPI, version) - 4usize]; + ["Offset of field: JSJitIteratorAPI::effects"] + [::core::mem::offset_of!(JSJitIteratorAPI, effects) - 8usize]; + ["Offset of field: JSJitIteratorAPI::reserved"] + [::core::mem::offset_of!(JSJitIteratorAPI, reserved) - 12usize]; + ["Offset of field: JSJitIteratorAPI::array_values_next"] + [::core::mem::offset_of!(JSJitIteratorAPI, array_values_next) - 16usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetIteratorAPI(version: u32) -> *const JSJitIteratorAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { @@ -2816,7 +2856,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_ITERATOR_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3099,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperIteratorOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3323,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub iterator_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3392,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::iterator_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, iterator_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/aarch64-pc-windows-msvc.rs b/sys/src/bindings/aarch64-pc-windows-msvc.rs index a64c09ee5..2fb1aee81 100644 --- a/sys/src/bindings/aarch64-pc-windows-msvc.rs +++ b/sys/src/bindings/aarch64-pc-windows-msvc.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,7 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_ITERATOR_API_VERSION: u32 = 1; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -113,7 +114,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2275,6 +2276,45 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_INVALID: JSJitIteratorQueryStatus = -1; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_MISS: JSJitIteratorQueryStatus = 0; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_VALUE: JSJitIteratorQueryStatus = 1; +pub type JSJitIteratorQueryStatus = ::core::ffi::c_int; +pub type JSJitArrayValuesNextFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + iterator: *const JSValue, + next_method: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitIteratorAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub reserved: u32, + pub array_values_next: JSJitArrayValuesNextFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitIteratorAPI"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitIteratorAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitIteratorAPI::struct_size"] + [::core::mem::offset_of!(JSJitIteratorAPI, struct_size) - 0usize]; + ["Offset of field: JSJitIteratorAPI::version"] + [::core::mem::offset_of!(JSJitIteratorAPI, version) - 4usize]; + ["Offset of field: JSJitIteratorAPI::effects"] + [::core::mem::offset_of!(JSJitIteratorAPI, effects) - 8usize]; + ["Offset of field: JSJitIteratorAPI::reserved"] + [::core::mem::offset_of!(JSJitIteratorAPI, reserved) - 12usize]; + ["Offset of field: JSJitIteratorAPI::array_values_next"] + [::core::mem::offset_of!(JSJitIteratorAPI, array_values_next) - 16usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetIteratorAPI(version: u32) -> *const JSJitIteratorAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { @@ -2816,7 +2856,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_ITERATOR_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3099,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperIteratorOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3323,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub iterator_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3392,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::iterator_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, iterator_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/aarch64-unknown-linux-gnu.rs b/sys/src/bindings/aarch64-unknown-linux-gnu.rs index a64c09ee5..2fb1aee81 100644 --- a/sys/src/bindings/aarch64-unknown-linux-gnu.rs +++ b/sys/src/bindings/aarch64-unknown-linux-gnu.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,7 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_ITERATOR_API_VERSION: u32 = 1; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -113,7 +114,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2275,6 +2276,45 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_INVALID: JSJitIteratorQueryStatus = -1; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_MISS: JSJitIteratorQueryStatus = 0; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_VALUE: JSJitIteratorQueryStatus = 1; +pub type JSJitIteratorQueryStatus = ::core::ffi::c_int; +pub type JSJitArrayValuesNextFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + iterator: *const JSValue, + next_method: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitIteratorAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub reserved: u32, + pub array_values_next: JSJitArrayValuesNextFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitIteratorAPI"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitIteratorAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitIteratorAPI::struct_size"] + [::core::mem::offset_of!(JSJitIteratorAPI, struct_size) - 0usize]; + ["Offset of field: JSJitIteratorAPI::version"] + [::core::mem::offset_of!(JSJitIteratorAPI, version) - 4usize]; + ["Offset of field: JSJitIteratorAPI::effects"] + [::core::mem::offset_of!(JSJitIteratorAPI, effects) - 8usize]; + ["Offset of field: JSJitIteratorAPI::reserved"] + [::core::mem::offset_of!(JSJitIteratorAPI, reserved) - 12usize]; + ["Offset of field: JSJitIteratorAPI::array_values_next"] + [::core::mem::offset_of!(JSJitIteratorAPI, array_values_next) - 16usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetIteratorAPI(version: u32) -> *const JSJitIteratorAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { @@ -2816,7 +2856,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_ITERATOR_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3099,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperIteratorOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3323,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub iterator_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3392,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::iterator_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, iterator_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/aarch64-unknown-linux-musl.rs b/sys/src/bindings/aarch64-unknown-linux-musl.rs index a64c09ee5..2fb1aee81 100644 --- a/sys/src/bindings/aarch64-unknown-linux-musl.rs +++ b/sys/src/bindings/aarch64-unknown-linux-musl.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,7 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_ITERATOR_API_VERSION: u32 = 1; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -113,7 +114,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2275,6 +2276,45 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_INVALID: JSJitIteratorQueryStatus = -1; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_MISS: JSJitIteratorQueryStatus = 0; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_VALUE: JSJitIteratorQueryStatus = 1; +pub type JSJitIteratorQueryStatus = ::core::ffi::c_int; +pub type JSJitArrayValuesNextFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + iterator: *const JSValue, + next_method: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitIteratorAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub reserved: u32, + pub array_values_next: JSJitArrayValuesNextFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitIteratorAPI"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitIteratorAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitIteratorAPI::struct_size"] + [::core::mem::offset_of!(JSJitIteratorAPI, struct_size) - 0usize]; + ["Offset of field: JSJitIteratorAPI::version"] + [::core::mem::offset_of!(JSJitIteratorAPI, version) - 4usize]; + ["Offset of field: JSJitIteratorAPI::effects"] + [::core::mem::offset_of!(JSJitIteratorAPI, effects) - 8usize]; + ["Offset of field: JSJitIteratorAPI::reserved"] + [::core::mem::offset_of!(JSJitIteratorAPI, reserved) - 12usize]; + ["Offset of field: JSJitIteratorAPI::array_values_next"] + [::core::mem::offset_of!(JSJitIteratorAPI, array_values_next) - 16usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetIteratorAPI(version: u32) -> *const JSJitIteratorAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { @@ -2816,7 +2856,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_ITERATOR_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3099,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperIteratorOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3323,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub iterator_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3392,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::iterator_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, iterator_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/wasm32-wasip1.rs b/sys/src/bindings/wasm32-wasip1.rs index a8167404c..34f9359ff 100644 --- a/sys/src/bindings/wasm32-wasip1.rs +++ b/sys/src/bindings/wasm32-wasip1.rs @@ -87,7 +87,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -95,6 +95,7 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_ITERATOR_API_VERSION: u32 = 1; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -114,7 +115,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2244,6 +2245,45 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_INVALID: JSJitIteratorQueryStatus = -1; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_MISS: JSJitIteratorQueryStatus = 0; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_VALUE: JSJitIteratorQueryStatus = 1; +pub type JSJitIteratorQueryStatus = ::core::ffi::c_int; +pub type JSJitArrayValuesNextFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + iterator: *const JSValue, + next_method: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitIteratorAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub reserved: u32, + pub array_values_next: JSJitArrayValuesNextFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitIteratorAPI"][::core::mem::size_of::() - 20usize]; + ["Alignment of JSJitIteratorAPI"][::core::mem::align_of::() - 4usize]; + ["Offset of field: JSJitIteratorAPI::struct_size"] + [::core::mem::offset_of!(JSJitIteratorAPI, struct_size) - 0usize]; + ["Offset of field: JSJitIteratorAPI::version"] + [::core::mem::offset_of!(JSJitIteratorAPI, version) - 4usize]; + ["Offset of field: JSJitIteratorAPI::effects"] + [::core::mem::offset_of!(JSJitIteratorAPI, effects) - 8usize]; + ["Offset of field: JSJitIteratorAPI::reserved"] + [::core::mem::offset_of!(JSJitIteratorAPI, reserved) - 12usize]; + ["Offset of field: JSJitIteratorAPI::array_values_next"] + [::core::mem::offset_of!(JSJitIteratorAPI, array_values_next) - 16usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetIteratorAPI(version: u32) -> *const JSJitIteratorAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { @@ -2785,7 +2825,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_ITERATOR_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3027,6 +3068,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperIteratorOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3241,10 +3292,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub iterator_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 104usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 108usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 4usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3300,6 +3361,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 96usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 100usize]; + ["Offset of field: JSJitRuntimeAPI::iterator_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, iterator_op) - 104usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/x86_64-apple-darwin.rs b/sys/src/bindings/x86_64-apple-darwin.rs index a64c09ee5..2fb1aee81 100644 --- a/sys/src/bindings/x86_64-apple-darwin.rs +++ b/sys/src/bindings/x86_64-apple-darwin.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,7 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_ITERATOR_API_VERSION: u32 = 1; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -113,7 +114,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2275,6 +2276,45 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_INVALID: JSJitIteratorQueryStatus = -1; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_MISS: JSJitIteratorQueryStatus = 0; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_VALUE: JSJitIteratorQueryStatus = 1; +pub type JSJitIteratorQueryStatus = ::core::ffi::c_int; +pub type JSJitArrayValuesNextFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + iterator: *const JSValue, + next_method: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitIteratorAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub reserved: u32, + pub array_values_next: JSJitArrayValuesNextFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitIteratorAPI"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitIteratorAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitIteratorAPI::struct_size"] + [::core::mem::offset_of!(JSJitIteratorAPI, struct_size) - 0usize]; + ["Offset of field: JSJitIteratorAPI::version"] + [::core::mem::offset_of!(JSJitIteratorAPI, version) - 4usize]; + ["Offset of field: JSJitIteratorAPI::effects"] + [::core::mem::offset_of!(JSJitIteratorAPI, effects) - 8usize]; + ["Offset of field: JSJitIteratorAPI::reserved"] + [::core::mem::offset_of!(JSJitIteratorAPI, reserved) - 12usize]; + ["Offset of field: JSJitIteratorAPI::array_values_next"] + [::core::mem::offset_of!(JSJitIteratorAPI, array_values_next) - 16usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetIteratorAPI(version: u32) -> *const JSJitIteratorAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { @@ -2816,7 +2856,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_ITERATOR_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3099,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperIteratorOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3323,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub iterator_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3392,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::iterator_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, iterator_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/x86_64-pc-windows-gnu.rs b/sys/src/bindings/x86_64-pc-windows-gnu.rs index a64c09ee5..2fb1aee81 100644 --- a/sys/src/bindings/x86_64-pc-windows-gnu.rs +++ b/sys/src/bindings/x86_64-pc-windows-gnu.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,7 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_ITERATOR_API_VERSION: u32 = 1; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -113,7 +114,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2275,6 +2276,45 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_INVALID: JSJitIteratorQueryStatus = -1; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_MISS: JSJitIteratorQueryStatus = 0; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_VALUE: JSJitIteratorQueryStatus = 1; +pub type JSJitIteratorQueryStatus = ::core::ffi::c_int; +pub type JSJitArrayValuesNextFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + iterator: *const JSValue, + next_method: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitIteratorAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub reserved: u32, + pub array_values_next: JSJitArrayValuesNextFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitIteratorAPI"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitIteratorAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitIteratorAPI::struct_size"] + [::core::mem::offset_of!(JSJitIteratorAPI, struct_size) - 0usize]; + ["Offset of field: JSJitIteratorAPI::version"] + [::core::mem::offset_of!(JSJitIteratorAPI, version) - 4usize]; + ["Offset of field: JSJitIteratorAPI::effects"] + [::core::mem::offset_of!(JSJitIteratorAPI, effects) - 8usize]; + ["Offset of field: JSJitIteratorAPI::reserved"] + [::core::mem::offset_of!(JSJitIteratorAPI, reserved) - 12usize]; + ["Offset of field: JSJitIteratorAPI::array_values_next"] + [::core::mem::offset_of!(JSJitIteratorAPI, array_values_next) - 16usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetIteratorAPI(version: u32) -> *const JSJitIteratorAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { @@ -2816,7 +2856,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_ITERATOR_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3099,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperIteratorOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3323,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub iterator_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3392,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::iterator_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, iterator_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/x86_64-pc-windows-msvc.rs b/sys/src/bindings/x86_64-pc-windows-msvc.rs index a64c09ee5..2fb1aee81 100644 --- a/sys/src/bindings/x86_64-pc-windows-msvc.rs +++ b/sys/src/bindings/x86_64-pc-windows-msvc.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,7 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_ITERATOR_API_VERSION: u32 = 1; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -113,7 +114,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2275,6 +2276,45 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_INVALID: JSJitIteratorQueryStatus = -1; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_MISS: JSJitIteratorQueryStatus = 0; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_VALUE: JSJitIteratorQueryStatus = 1; +pub type JSJitIteratorQueryStatus = ::core::ffi::c_int; +pub type JSJitArrayValuesNextFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + iterator: *const JSValue, + next_method: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitIteratorAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub reserved: u32, + pub array_values_next: JSJitArrayValuesNextFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitIteratorAPI"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitIteratorAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitIteratorAPI::struct_size"] + [::core::mem::offset_of!(JSJitIteratorAPI, struct_size) - 0usize]; + ["Offset of field: JSJitIteratorAPI::version"] + [::core::mem::offset_of!(JSJitIteratorAPI, version) - 4usize]; + ["Offset of field: JSJitIteratorAPI::effects"] + [::core::mem::offset_of!(JSJitIteratorAPI, effects) - 8usize]; + ["Offset of field: JSJitIteratorAPI::reserved"] + [::core::mem::offset_of!(JSJitIteratorAPI, reserved) - 12usize]; + ["Offset of field: JSJitIteratorAPI::array_values_next"] + [::core::mem::offset_of!(JSJitIteratorAPI, array_values_next) - 16usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetIteratorAPI(version: u32) -> *const JSJitIteratorAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { @@ -2816,7 +2856,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_ITERATOR_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3099,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperIteratorOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3323,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub iterator_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3392,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::iterator_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, iterator_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/x86_64-unknown-linux-gnu.rs b/sys/src/bindings/x86_64-unknown-linux-gnu.rs index a64c09ee5..2fb1aee81 100644 --- a/sys/src/bindings/x86_64-unknown-linux-gnu.rs +++ b/sys/src/bindings/x86_64-unknown-linux-gnu.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,7 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_ITERATOR_API_VERSION: u32 = 1; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -113,7 +114,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2275,6 +2276,45 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_INVALID: JSJitIteratorQueryStatus = -1; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_MISS: JSJitIteratorQueryStatus = 0; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_VALUE: JSJitIteratorQueryStatus = 1; +pub type JSJitIteratorQueryStatus = ::core::ffi::c_int; +pub type JSJitArrayValuesNextFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + iterator: *const JSValue, + next_method: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitIteratorAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub reserved: u32, + pub array_values_next: JSJitArrayValuesNextFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitIteratorAPI"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitIteratorAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitIteratorAPI::struct_size"] + [::core::mem::offset_of!(JSJitIteratorAPI, struct_size) - 0usize]; + ["Offset of field: JSJitIteratorAPI::version"] + [::core::mem::offset_of!(JSJitIteratorAPI, version) - 4usize]; + ["Offset of field: JSJitIteratorAPI::effects"] + [::core::mem::offset_of!(JSJitIteratorAPI, effects) - 8usize]; + ["Offset of field: JSJitIteratorAPI::reserved"] + [::core::mem::offset_of!(JSJitIteratorAPI, reserved) - 12usize]; + ["Offset of field: JSJitIteratorAPI::array_values_next"] + [::core::mem::offset_of!(JSJitIteratorAPI, array_values_next) - 16usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetIteratorAPI(version: u32) -> *const JSJitIteratorAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { @@ -2816,7 +2856,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_ITERATOR_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3099,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperIteratorOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3323,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub iterator_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3392,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::iterator_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, iterator_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/x86_64-unknown-linux-musl.rs b/sys/src/bindings/x86_64-unknown-linux-musl.rs index a64c09ee5..2fb1aee81 100644 --- a/sys/src/bindings/x86_64-unknown-linux-musl.rs +++ b/sys/src/bindings/x86_64-unknown-linux-musl.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,7 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_ITERATOR_API_VERSION: u32 = 1; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -113,7 +114,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2275,6 +2276,45 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_INVALID: JSJitIteratorQueryStatus = -1; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_MISS: JSJitIteratorQueryStatus = 0; +pub const JSJitIteratorQueryStatus_JS_JIT_ITERATOR_QUERY_VALUE: JSJitIteratorQueryStatus = 1; +pub type JSJitIteratorQueryStatus = ::core::ffi::c_int; +pub type JSJitArrayValuesNextFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + iterator: *const JSValue, + next_method: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitIteratorAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub reserved: u32, + pub array_values_next: JSJitArrayValuesNextFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitIteratorAPI"][::core::mem::size_of::() - 24usize]; + ["Alignment of JSJitIteratorAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitIteratorAPI::struct_size"] + [::core::mem::offset_of!(JSJitIteratorAPI, struct_size) - 0usize]; + ["Offset of field: JSJitIteratorAPI::version"] + [::core::mem::offset_of!(JSJitIteratorAPI, version) - 4usize]; + ["Offset of field: JSJitIteratorAPI::effects"] + [::core::mem::offset_of!(JSJitIteratorAPI, effects) - 8usize]; + ["Offset of field: JSJitIteratorAPI::reserved"] + [::core::mem::offset_of!(JSJitIteratorAPI, reserved) - 12usize]; + ["Offset of field: JSJitIteratorAPI::array_values_next"] + [::core::mem::offset_of!(JSJitIteratorAPI, array_values_next) - 16usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetIteratorAPI(version: u32) -> *const JSJitIteratorAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { @@ -2816,7 +2856,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_ITERATOR_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3099,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperIteratorOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3323,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub iterator_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3392,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::iterator_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, iterator_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/tests/jit_patch.rs b/sys/tests/jit_patch.rs index f36861117..f0264747b 100644 --- a/sys/tests/jit_patch.rs +++ b/sys/tests/jit_patch.rs @@ -56,6 +56,7 @@ fn copy_patches(destination: &std::path::Path) { "0018-inline-frame-recovery.patch", "0019-array-feedback.patch", "0020-typed-array-guard.patch", + "0028-tier1-iteration.patch", ] { fs::copy(source.join(patch), destination.join(patch)).unwrap(); } @@ -114,7 +115,10 @@ fn pinned_public_quickjs_baseline_applies_cleanly_without_git() { let helper_header = fs::read_to_string(destination.join("quickjs-jit-helpers.h")).unwrap(); assert!(quickjs.contains("JS_GetJitRuntimeId")); assert!(quickjs.contains("JS_JIT_FRAME_SIDE_PATH_HIT")); - assert!(jit_header.contains("#define QJSJIT_ABI_MINOR 24u")); + assert!(jit_header.contains("#define QJSJIT_ABI_MINOR 25u")); + assert!(jit_header.contains("JS_JitGetIteratorAPI(uint32_t version)")); + assert!(quickjs.contains("JS_JitHelperIteratorOp")); + assert!(quickjs.contains("qjsjit_array_values_next")); assert!(jit_header.contains("JSJitPropertyLayout property_layout;")); assert!(quickjs.contains("uint64_t jit_shape_generation;")); assert!(!quickjs.contains("QJSJIT_SHAPE_HASH")); @@ -133,7 +137,7 @@ fn pinned_public_quickjs_baseline_applies_cleanly_without_git() { .count(), 3 ); - assert!(jit_header.contains("QJSJIT_RUNTIME_API_MINOR 9u")); + assert!(jit_header.contains("QJSJIT_RUNTIME_API_MINOR 10u")); assert!(jit_header.contains("#define QJSJIT_RUNTIME_FIELD_MAP_OUT_IN_OP(field)")); assert!(helper_header.contains("JS_JIT_HELPER_MATERIALIZED = 2")); assert!(helper_header.contains("JS_JIT_OWNER_SOURCE_ARGUMENT = 0")); @@ -149,6 +153,7 @@ fn pinned_public_quickjs_baseline_applies_cleanly_without_git() { assert!(helper_header.contains("X(ATOM_VALUE, atom_value")); assert!(helper_header.contains("X(BINARY_ARITH_SLOW, binary_arith_slow")); assert!(helper_header.contains("X(UNARY_ARITH_SLOW, unary_arith_slow")); + assert!(helper_header.contains("X(ITERATOR_OP, iterator_op")); assert!(helper_header.contains("#define QJSJIT_DECLARE_MAP_OUT_IN_OP(name)")); assert!(quickjs.contains("JS_JitHelperBinaryArithSlow")); assert!(quickjs.contains("JS_JitHelperUnaryArithSlow")); @@ -255,7 +260,7 @@ fn bundled_jit_bindings_include_materialize_owner_tail() { for target in targets { let binding = fs::read_to_string(bindings.join(target)).unwrap(); assert!( - binding.contains("pub const QJSJIT_ABI_MINOR: u32 = 24;"), + binding.contains("pub const QJSJIT_ABI_MINOR: u32 = 25;"), "{target}" ); assert!( @@ -293,28 +298,33 @@ fn bundled_jit_bindings_include_materialize_owner_tail() { && binding.contains("pub regexp: ::core::option::Option") && binding.contains("pub atom_value: ::core::option::Option") && binding.contains("pub binary_arith_slow: ::core::option::Option") - && binding.contains("pub unary_arith_slow: ::core::option::Option"), + && binding.contains("pub unary_arith_slow: ::core::option::Option") + && binding.contains("pub iterator_op: ::core::option::Option"), "{target}" ); assert!( binding.contains("JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22") && binding.contains("JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23") - && binding.contains("JS_JIT_HELPER_COUNT: JSJitHelperId = 24"), + && binding.contains("JS_JIT_HELPER_ITERATOR_OP: JSJitHelperId = 24") + && binding.contains("JS_JIT_HELPER_COUNT: JSJitHelperId = 25"), "{target}" ); assert!( - binding.contains("pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9;"), + binding.contains("pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10;"), "{target}" ); assert!( - binding.contains("size_of::() - 200usize"), + binding.contains("size_of::() - 208usize") + && binding.contains( + "pub fn JS_JitGetIteratorAPI(version: u32) -> *const JSJitIteratorAPI;" + ), "{target}" ); } } #[test] -fn bundled_wasm_jit_binding_matches_array_abi_1_24() { +fn bundled_wasm_jit_binding_matches_iterator_abi_1_25() { let binding = fs::read_to_string( PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("src/bindings/wasm32-wasip1.rs"), ) @@ -363,7 +373,6 @@ fn bundled_wasm_jit_binding_matches_array_abi_1_24() { "pub const JS_JIT_FEEDBACK_ARRAY_SHARED: u32 = 8192;", "pub const JS_JIT_FEEDBACK_ARRAY_INVALID_BACKING: u32 = 16384;", // ABI 1.24: typed-array continuing guard. - "pub const QJSJIT_ABI_MINOR: u32 = 24;", "pub const QJSJIT_ARRAY_API_VERSION: u32 = 1;", "pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1;", "JSJitArrayQueryStatus_JS_JIT_ARRAY_QUERY_INVALID: JSJitArrayQueryStatus = -1;", @@ -374,6 +383,16 @@ fn bundled_wasm_jit_binding_matches_array_abi_1_24() { "receiver: *const JSValue", "pub struct JSJitArrayAPI", "pub fn JS_JitGetArrayAPI(", + // ABI 1.25: Tier 1 iteration helper and Array values leaf. + "pub const QJSJIT_ABI_MINOR: u32 = 25;", + "pub const QJSJIT_ITERATOR_API_VERSION: u32 = 1;", + "JSJitHelperId_JS_JIT_HELPER_ITERATOR_OP: JSJitHelperId = 24;", + "pub fn JS_JitHelperIteratorOp(", + "pub struct JSJitIteratorAPI", + "pub fn JS_JitGetIteratorAPI(", + "size_of::() - 20usize", + "offset_of!(JSJitIteratorAPI, array_values_next) - 16usize", + "offset_of!(JSJitRuntimeAPI, iterator_op) - 104usize", // CONFIG_JIT_TEST_SUPPORT declarations retained by the CI generator. "pub fn JS_JitGetHelperCount(", "pub fn JS_JitSetExecutionTrace(", diff --git a/sys/tests/typed_array_guard.rs b/sys/tests/typed_array_guard.rs index 2195d1454..8f485471d 100644 --- a/sys/tests/typed_array_guard.rs +++ b/sys/tests/typed_array_guard.rs @@ -105,7 +105,7 @@ int main(void) void *expected_data; uint32_t expected_count; - assert(QJSJIT_ABI_MAJOR == 1 && QJSJIT_ABI_MINOR == 24); + assert(QJSJIT_ABI_MAJOR == 1 && QJSJIT_ABI_MINOR == 25); assert(QJSJIT_ARRAY_API_VERSION == 1); assert(ctx && api); assert(api->struct_size == sizeof(*api)); From 12c4b629e5ed8ae84eab201f965a2afb510a8510 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:06:40 +0800 Subject: [PATCH 17/54] perf(jit): add Tier 1 object-literal and Array push leaf fast paths Add the ABI 1.25 versioned JSJitObjectAPI (patch 0032). Each leaf either completes the interpreter's exact effect or misses without an observable effect; none throws, runs user code or reenters the VM, so generated code keeps the generic helper sequence as the fallback. - literal: build {a: v0, ...} from the existing final hashed shape, found directly from the (Object.prototype, atoms, C_W_E) hash chain, with non-throwing allocation and the JS_NewObjectFromShape GC trigger. - retain_shape: keep generically built literal shapes in the bounded, GC-traced 0012 shape cache so later executions hit. - array_method / array_push / array_push_method: exact get_field2 lookup of Array.prototype data properties on fast arrays and one-argument built-in push (stack check, fast-array path, non-throwing growth). - Fix the upstream js_array_push fast path that appended at `length` when a fast array kept count < length, leaving uninitialized slots (the unmodified interpreter crashes on it). Tier 1 fuses complete pure object literals and `x.push(v)` statements with pure operands into one leaf call each, and tries the lookup/push leaves before GET_PROPERTY/CALL on Array receivers. Noisy diagnostic (loaded shared host, 9 alternating runs, automatic): objects-polymorphic 1.21x the 82d3808 speed; mixed-quotes and collections statistically tied. Remaining work and a pre-existing define_field set-semantics bug are recorded in docs/roadmap/p4e-alloc-fastpath.md. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p4e-alloc-fastpath.md | 118 +++ jit/src/abi.rs | 103 ++- jit/src/compiler/baseline.rs | 740 +++++++++++++++++- jit/tests/abi.rs | 8 +- jit/tests/object_fast_paths.rs | 397 ++++++++++ jit/tests/support/mod.rs | 5 +- sys/build_support/patch.rs | 7 +- sys/patches/0032-object-fast-paths.patch | 356 +++++++++ sys/patches/README.md | 17 + sys/src/bindings/aarch64-apple-darwin.rs | 87 +- sys/src/bindings/aarch64-pc-windows-msvc.rs | 87 +- sys/src/bindings/aarch64-unknown-linux-gnu.rs | 87 +- .../bindings/aarch64-unknown-linux-musl.rs | 87 +- sys/src/bindings/wasm32-wasip1.rs | 87 +- sys/src/bindings/x86_64-apple-darwin.rs | 87 +- sys/src/bindings/x86_64-pc-windows-gnu.rs | 87 +- sys/src/bindings/x86_64-pc-windows-msvc.rs | 87 +- sys/src/bindings/x86_64-unknown-linux-gnu.rs | 87 +- sys/src/bindings/x86_64-unknown-linux-musl.rs | 87 +- sys/tests/jit_patch.rs | 30 +- 20 files changed, 2624 insertions(+), 27 deletions(-) create mode 100644 docs/roadmap/p4e-alloc-fastpath.md create mode 100644 jit/tests/object_fast_paths.rs create mode 100644 sys/patches/0032-object-fast-paths.patch diff --git a/docs/roadmap/p4e-alloc-fastpath.md b/docs/roadmap/p4e-alloc-fastpath.md new file mode 100644 index 000000000..9c92aa124 --- /dev/null +++ b/docs/roadmap/p4e-alloc-fastpath.md @@ -0,0 +1,118 @@ +# P4e: object allocation and `Array.prototype.push` fast paths (first slice) + +Branch `perf/p4e-alloc-fastpath`, based on `82d3808`. Roadmap item: P4 +"对象分配快路径" in [PERFORMANCE_ROADMAP_V8_BUN.md](../PERFORMANCE_ROADMAP_V8_BUN.md). + +## What landed + +- `sys/patches/0032-object-fast-paths.patch` (ABI 1.25): a versioned + `JSJitObjectAPI` table of leaf functions. Every leaf either completes the + interpreter's exact effect or misses with no observable effect; none throws, + calls user code or reenters the VM, so generated code always keeps the + generic helper sequence as its fallback. + - `literal(ctx, atoms, values, count, out)`: builds `{a: v0, ...}` + (OP_object + one OP_define_field per atom, `JS_PROP_C_W_E`). The final + hashed shape is a pure function of `(Object.prototype, atom list)`, so it + is looked up directly in the runtime shape hash (no intermediate + transition shapes, which QuickJS mutates in place and does not retain). + Allocation uses the non-throwing `js_malloc_rt`; a miss (no shape, duplicate + atom, uninitialized value, OOM) has no effect. It keeps the + `JS_NewObjectFromShape` GC trigger point, so GC accounting is unchanged. + - `retain_shape(ctx, object)`: after a literal is built generically, keeps + its final shape in the bounded, GC-traced per-context JIT shape cache from + 0012, so literals whose receivers are later extended (`o = {x, kind}; + o.y = ...`) still find their shape next time. + - `array_method(ctx, receiver, atom, out)`: exact `get_field2` lookup on a + fast Array when it resolves to a plain data property of the realm's + `Array.prototype` (no own property, no exotic handler, getter or autoinit). + - `array_push(ctx, fn, receiver, value, out)` and + `array_push_method(ctx, receiver, atom, value, out)`: a one-argument call + of the built-in `Array.prototype.push` (identity: C function, + `js_array_push`, magic 0, same runtime realm), mirroring + `js_call_c_function`'s stack-overflow check and `js_array_push`'s + fast-array path; element-buffer growth uses `expand_fast_array`'s policy + with non-throwing `js_realloc_rt`. + - Upstream fix: `js_array_push`'s fast path appended at `length` even when + a fast array kept `count < length` (after `arr.length = n` grew it), + leaving uninitialized element slots. The unmodified 82d3808 interpreter + crashes (`SIGSEGV`) on `a=[1,2,3]; a.length=6; a.push(7,8)` in a loop + (`jit-bench worker --mode interpreter` on such a script dumps core). The + fast path now requires `length == count`; otherwise the generic path runs. +- Tier 1 lowering (`jit/src/compiler/baseline.rs`): + - Complete object literals whose values are side-effect-free reads + (constants, arguments, locals) are fused into one `literal` leaf call; the + unchanged `object`/`define_field` sequence is the fallback and retains the + resulting shape. Prefixes of longer literals are not fused (their shape is + never alive), checked by a bounded stack-effect scan. + - `x.push(v)` statements with side-effect-free receiver and argument reads + are fused into one `array_push_method` leaf call: no DUP/FREE, + GET_PROPERTY or CALL helpers on a hit. + - Other `get_field2` sites on fast Arrays try `array_method` before the + GET_PROPERTY helper, and other one-argument method calls on Arrays try + `array_push` before the CALL helper (all inputs stay borrowed, so the + interpreter-order cleanup is shared by both paths). + +Tests: `jit/tests/object_fast_paths.rs` (interpreter differential checks, +stress GC, every push miss shape: frozen/sealed/read-only length, own `push`, +subclass, holey and grown-length arrays, array-likes, patched or accessor +`Array.prototype.push`, `unshift` aliasing, TDZ bindings, non-array receivers; +helper-count proofs that hits skip NEW_OBJECT/SET_PROPERTY/GET_PROPERTY/CALL), +plus ABI table validation tests in `jit/src/abi.rs`. + +## Noisy diagnostic (not publishable evidence) + +Shared, loaded host (i7-13700KF, load average about 11–40 from 16 +concurrent agents, no CPU pinning). `jit-bench worker --mode automatic`, +alternating the 82d3808 binary (baseline) and this branch. Each value is the +median over processes of `median(protocol.warmup_batch_ns[-16:])`, in ms per +10 workload calls. Speed = baseline / new (above 1.00x is faster). Checksums +are identical across both binaries and the interpreter. There are no +confidence intervals; the three-engine README matrix still has to be +measured by the integrator. + +| Workload | Runs | Baseline ms | New ms | Speed | +| --- | ---: | ---: | ---: | ---: | +| objects-polymorphic | 9 | 24.151 | 19.951 | 1.21x | +| mixed-quotes | 9 | 2.520 | 2.501 | 1.01x (tied, noise) | +| collections | 9 | 1.651 | 1.645 | 1.00x (tied, noise; not native) | +| property-heavy (control) | 5 | 0.1030 | 0.0996 | 1.03x (bimodal samples) | +| call-heavy (control) | 5 | 0.1136 | 0.0604 | inconclusive: samples are bimodal (0.06 and 0.115 ms in both binaries) | +| generic-call-fallback (control) | 5 | 9.123 | 9.089 | 1.00x | +| arrays-typed (control) | 5 | 2.991 | 3.032 | 0.99x | +| calls-closures (control) | 5 | 3.379 | 3.255 | 1.04x (bimodal samples) | + +Remaining helper calls per `objects-polymorphic` iteration on this branch +(Tier 1, `JS_JitGetHelperCount`): NEW_OBJECT 0, SET_PROPERTY 2.0 (`o.y =`, +`o.x +=` and the `Object.create(null)` case), GET_PROPERTY 5.25, GET_ELEMENT +2, DUP 7, FREE 8.5, CALL 0.25 (`Object.create`). Every literal and every +`objects.push(object)` is handled by a leaf. + +## Remaining work + +1. **Pre-existing bug, not fixed here:** Tier 1 lowers `define_field` (and + `define_array_el`) with *set* semantics (`JS_JIT_HELPER_SET_PROPERTY` / + `SET_ELEMENT`). An inherited setter on `Object.prototype` runs and no own + property is created for literals that are not fused (for example + `{x: i, y: f()}`). Reproduced on this branch's generic path, which is + unchanged from 82d3808; recorded as the ignored test + `generic_define_field_ignores_inherited_setters`. The fix needs an exact + define helper (new helper-table entry, ABI bump) or a non-throwing define + leaf with a deopt fallback. +2. **Tier 2:** `object`/`define_field` are not admitted by the optimized + tier, so functions building literals stay in Tier 1. Admitting them needs + an allocation node in the semantic SSA (fresh object: no aliasing with + existing heap locations), ownership/materialization for deopt, and then + scalar replacement/allocation sinking. +3. **Non-simple push arguments** (for example `quotes.push({index, score})` in + `mixed-quotes`) still use the separate `array_method` + `array_push` + leaves plus DUP/FREE helpers; fusing a literal value into the push leaf + needs an owned-temporary release path. +4. **Empty literals without live empty objects** (`const e = {}` that is + dropped) miss because no hashed empty shape survives; creating the shape + non-throwingly in the leaf would remove the NEW_OBJECT helper there too. +5. `collections` is unaffected: its workload is not natively compiled + (`for_of` is rejected by the Tier 1 policy; roadmap P2). +6. The remaining cost in `objects-polymorphic` is DUP/FREE helpers (≈7 and + ≈8.5 per iteration) and polymorphic property reads (≈5 GET_PROPERTY per + iteration); those are roadmap P1 (inline Dup/Free, O(1) helper checks) and + P4 polymorphic property ICs. diff --git a/jit/src/abi.rs b/jit/src/abi.rs index b39c65a23..a25576713 100644 --- a/jit/src/abi.rs +++ b/jit/src/abi.rs @@ -5,7 +5,7 @@ use core::{fmt, mem}; use rquickjs_core::qjs; pub const ABI_MAJOR: u16 = 1; -pub const ABI_MINOR: u16 = 24; +pub const ABI_MINOR: u16 = 25; pub const SOURCE_REVISION: u64 = 0xfd0a_0210_b7be_0095; pub const OPCODE_FINGERPRINT: u64 = qjs::QJSJIT_GENERATED_OPCODE_FINGERPRINT; @@ -30,6 +30,7 @@ pub enum AbiStructure { BackendVTable, InlineApi, ArrayApi, + ObjectApi, } /// Constructors for the authoritative native-to-interpreter exit contract. @@ -115,6 +116,7 @@ pub struct AbiInfo { raw: qjs::JSJitABIInfo, inline_api: qjs::JSJitInlineAPI, array_api: qjs::JSJitArrayAPI, + object_api: qjs::JSJitObjectAPI, } /// Runtime-exported, append-only offsets for element fast paths. QuickJS @@ -198,6 +200,14 @@ impl AbiInfo { &self.array_api } + /// Non-throwing, non-reentrant allocation and array-method fast paths. + /// Each entry completes the exact interpreter effect or misses without an + /// observable effect; callers keep their generic helper as the fallback. + #[cfg(feature = "compiler")] + pub(crate) const fn object_api(&self) -> &qjs::JSJitObjectAPI { + &self.object_api + } + #[cfg(feature = "compiler")] pub(crate) fn element_layout(&self) -> ElementLayout { let raw = self.raw.element_layout; @@ -240,12 +250,14 @@ impl AbiInfo { // All fields are integers or nullable function pointers. inline_api: unsafe { mem::zeroed() }, array_api: unsafe { mem::zeroed() }, + object_api: unsafe { mem::zeroed() }, }; // Do not resolve native inline addresses against a mismatched // main ABI, even when the optional table has a familiar version. info.validate_main()?; info.inline_api = query_inline_api()?; info.array_api = query_array_api()?; + info.object_api = query_object_api()?; Ok(info) } else { Err(AbiError::QueryFailed(status)) @@ -255,7 +267,8 @@ impl AbiInfo { pub(crate) fn validate(&self) -> Result<(), AbiError> { self.validate_main()?; validate_inline_api(self.inline_api())?; - validate_array_api(&self.array_api) + validate_array_api(&self.array_api)?; + validate_object_api(&self.object_api) } fn validate_main(&self) -> Result<(), AbiError> { @@ -373,6 +386,7 @@ impl AbiInfo { } AbiMismatch::StructureLayout(AbiStructure::InlineApi) => self.inline_api.version ^= 1, AbiMismatch::StructureLayout(AbiStructure::ArrayApi) => self.array_api.version ^= 1, + AbiMismatch::StructureLayout(AbiStructure::ObjectApi) => self.object_api.version ^= 1, AbiMismatch::BuildFingerprint => self.raw.build_fingerprint ^= 1, } } @@ -1009,6 +1023,89 @@ fn validate_array_api(api: &qjs::JSJitArrayAPI) -> Result<(), AbiError> { } } +fn query_object_api() -> Result { + let pointer = unsafe { qjs::JS_JitGetObjectAPI(qjs::QJSJIT_OBJECT_API_VERSION) }; + if pointer.is_null() + || unsafe { core::ptr::addr_of!((*pointer).struct_size).read() } + != mem::size_of::() as u32 + { + return Err(AbiError::Incompatible(AbiMismatch::StructureLayout( + AbiStructure::ObjectApi, + ))); + } + let api = unsafe { pointer.read() }; + validate_object_api(&api)?; + Ok(api) +} + +/// The object table may allocate and trigger QuickJS' allocation-time cycle +/// collection (C finalizers only). Any other advertised effect, such as a +/// throw or VM reentry, is incompatible with the leaf call sites. +fn validate_object_api(api: &qjs::JSJitObjectAPI) -> Result<(), AbiError> { + let permitted = qjs::JS_JIT_OBJECT_EFFECT_ALLOCATE | qjs::JS_JIT_OBJECT_EFFECT_COLLECT; + if api.struct_size != mem::size_of::() as u32 + || api.version != qjs::QJSJIT_OBJECT_API_VERSION + || api.effects & !permitted != 0 + || api.max_literal_fields != qjs::JS_JIT_OBJECT_LITERAL_MAX_FIELDS + || api.literal.is_none() + || api.retain_shape.is_none() + || api.array_method.is_none() + || api.array_push.is_none() + || api.array_push_method.is_none() + { + Err(AbiError::Incompatible(AbiMismatch::StructureLayout( + AbiStructure::ObjectApi, + ))) + } else { + Ok(()) + } +} + +#[cfg(test)] +mod object_api_tests { + use super::*; + + fn linked_table() -> qjs::JSJitObjectAPI { + let pointer = unsafe { qjs::JS_JitGetObjectAPI(qjs::QJSJIT_OBJECT_API_VERSION) }; + assert!(!pointer.is_null()); + unsafe { *pointer } + } + + #[test] + fn object_api_is_exactly_versioned_bounded_and_complete() { + let valid = linked_table(); + assert!(validate_object_api(&valid).is_ok()); + assert!(unsafe { qjs::JS_JitGetObjectAPI(0) }.is_null()); + for field in 0..9 { + let mut invalid = valid; + match field { + 0 => invalid.struct_size -= 1, + 1 => invalid.version += 1, + 2 => invalid.effects |= 1 << 8, + 3 => invalid.max_literal_fields += 1, + 4 => invalid.literal = None, + 5 => invalid.retain_shape = None, + 6 => invalid.array_method = None, + 7 => invalid.array_push_method = None, + _ => invalid.array_push = None, + } + assert!(validate_object_api(&invalid).is_err(), "field {field}"); + } + } + + #[test] + fn object_table_validation_is_part_of_the_main_abi_contract() { + let mut info = AbiInfo::linked().unwrap(); + info.object_api.array_push = None; + assert_eq!( + info.validate(), + Err(AbiError::Incompatible(AbiMismatch::StructureLayout( + AbiStructure::ObjectApi + ))) + ); + } +} + #[cfg(test)] mod array_api_tests { use super::*; @@ -1073,7 +1170,7 @@ mod inline_api_tests { fn inline_recovery_requires_the_matching_main_abi_minor() { let native = AbiInfo::query_linked().unwrap(); assert_eq!(ABI_MINOR, native.minor()); - assert_eq!(native.minor(), 24); + assert_eq!(native.minor(), 25); assert!(unsafe { qjs::JS_JitGetInlineAPI(0) }.is_null()); } diff --git a/jit/src/compiler/baseline.rs b/jit/src/compiler/baseline.rs index 00e7cfb64..816e5cc01 100644 --- a/jit/src/compiler/baseline.rs +++ b/jit/src/compiler/baseline.rs @@ -91,6 +91,496 @@ struct HelperLowering<'a> { signatures: &'a [cranelift_codegen::ir::SigRef], pointer_type: cranelift_codegen::ir::Type, layout: FrameLayout, + element_layout: crate::abi::ElementLayout, + object_fast: Option, +} + +/// Addresses of the versioned object/array leaves (`JSJitObjectAPI`). Every +/// leaf either completes the interpreter's exact effect or misses without an +/// observable effect, never throws and never reenters the VM, so generated +/// code keeps the generic helper sequence as the fallback of every hit test. +#[derive(Clone, Copy, Debug)] +struct ObjectFastPaths { + literal: usize, + retain_shape: usize, + array_method: usize, + array_push: usize, + array_push_method: usize, + max_literal_fields: usize, +} + +impl ObjectFastPaths { + fn from_api(api: &qjs::JSJitObjectAPI) -> Option { + Some(Self { + literal: api.literal? as usize, + retain_shape: api.retain_shape? as usize, + array_method: api.array_method? as usize, + array_push: api.array_push? as usize, + array_push_method: api.array_push_method? as usize, + max_literal_fields: usize::try_from(api.max_literal_fields).ok()?, + }) + } +} + +/// A straight-line instruction range with a leaf fast path. The hit path +/// jumps to `join`; the unchanged instructions `start..=end` are lowered in +/// the fallback block, which then jumps to `join` as well. +#[derive(Clone, Copy, Debug)] +struct FusedRegion { + end: usize, + join: Block, + /// Stack slot of a literal built on the fallback path whose final shape + /// is retained so later executions hit the literal leaf. + retain_literal: Option, +} + +fn literal_operand(op: &IrOp, argument_count: usize, local_count: usize) -> Option { + match *op { + IrOp::Push(constant) if constant.tag >= 0 => Some(LiteralValue::Constant(constant)), + IrOp::GetArgument(index) if usize::from(index) < argument_count => { + Some(LiteralValue::Argument(index)) + } + IrOp::GetLocal(index) | IrOp::GetLocalChecked(index) + if usize::from(index) < local_count => + { + Some(LiteralValue::Local(index)) + } + _ => None, + } +} + +fn literal_operand_pair( + builder: &mut FunctionBuilder<'_>, + helpers: &HelperLowering<'_>, + value: LiteralValue, +) -> Result { + Ok(match value { + LiteralValue::Constant(constant) => constant_pair(builder, constant), + LiteralValue::Argument(argument) => use_pair( + builder, + *helpers + .arguments + .get(usize::from(argument)) + .ok_or(CompileFailure::InvalidArtifact)?, + ), + LiteralValue::Local(local) => use_pair( + builder, + *helpers + .locals + .get(usize::from(local)) + .ok_or(CompileFailure::InvalidArtifact)?, + ), + }) +} + +/// `receiver.method(value)` where the receiver and the single argument are +/// side-effect-free reads: `(receiver, atom, value, end)`. +fn array_push_statement_pattern( + instructions: &[crate::ir::IrInstruction], + start: usize, + argument_count: usize, + local_count: usize, +) -> Option<(LiteralValue, u32, LiteralValue, usize)> { + let receiver = match literal_operand(&instructions.get(start)?.op, argument_count, local_count)? + { + LiteralValue::Constant(_) => return None, + receiver => receiver, + }; + let IrOp::GetPropertyKeep(atom) = instructions.get(start + 1)?.op else { + return None; + }; + let value = literal_operand( + &instructions.get(start + 2)?.op, + argument_count, + local_count, + )?; + matches!( + instructions.get(start + 3)?.op, + IrOp::Call { + argc: 1, + has_this: true + } + ) + .then_some((receiver, atom, value, start + 3)) +} + +fn emit_fused_fast_path( + builder: &mut FunctionBuilder<'_>, + helpers: &HelperLowering<'_>, + fast: ObjectFastPaths, + instructions: &[crate::ir::IrInstruction], + start: usize, + depth: usize, +) -> Result, CompileFailure> { + let argument_count = helpers.arguments.len(); + let local_count = helpers.locals.len(); + if let Some((atoms, values, end)) = object_literal_pattern( + instructions, + start, + argument_count, + local_count, + fast.max_literal_fields, + ) { + if depth + usize::from(!values.is_empty()) >= helpers.stack.len() { + return Ok(None); + } + let join = emit_object_literal_fast_path(builder, helpers, fast, &atoms, &values, depth)?; + return Ok(Some(FusedRegion { + end, + join, + retain_literal: Some(depth), + })); + } + if let Some((receiver, atom, value, end)) = + array_push_statement_pattern(instructions, start, argument_count, local_count) + { + if depth + 2 >= helpers.stack.len() { + return Ok(None); + } + let join = emit_array_push_statement_fast_path( + builder, helpers, fast, receiver, atom, value, depth, + )?; + return Ok(Some(FusedRegion { + end, + join, + retain_literal: None, + })); + } + Ok(None) +} + +/// Whole-statement `array.push(value)`: one leaf performs the exact +/// `get_field2` lookup and the built-in push without materializing the +/// receiver, method or argument references (no DUP/FREE helpers). +#[allow(clippy::too_many_arguments)] +fn emit_array_push_statement_fast_path( + builder: &mut FunctionBuilder<'_>, + helpers: &HelperLowering<'_>, + fast: ObjectFastPaths, + receiver: LiteralValue, + atom: u32, + value: LiteralValue, + depth: usize, +) -> Result { + let pointer_type = helpers.pointer_type; + let receiver = literal_operand_pair(builder, helpers, receiver)?; + let value = literal_operand_pair(builder, helpers, value)?; + let call = builder.create_block(); + let hit = builder.create_block(); + let fallback = builder.create_block(); + let join = builder.create_block(); + let is_object = tag_is(builder, receiver.tag, qjs::JS_TAG_OBJECT); + let class_check = builder.create_block(); + builder + .ins() + .brif(is_object, class_check, &[], fallback, &[]); + builder.seal_block(class_check); + builder.switch_to_block(class_check); + let class = builder.ins().load( + types::I16, + MemFlags::trusted(), + receiver.payload, + helpers.element_layout.object_class_id_offset, + ); + let class = builder.ins().uextend(types::I64, class); + let is_array = + builder + .ins() + .icmp_imm(IntCC::Equal, class, helpers.element_layout.array_class_id); + builder.ins().brif(is_array, call, &[], fallback, &[]); + builder.seal_block(call); + builder.switch_to_block(call); + let value_size = + i32::try_from(mem::size_of::()).map_err(|_| CompileFailure::ResourceLimit)?; + let scratch = object_scratch(builder, 3); + for (index, pair) in [receiver, value].into_iter().enumerate() { + let offset = + value_size * i32::try_from(index).map_err(|_| CompileFailure::ResourceLimit)?; + builder.ins().stack_store(pair.payload, scratch, offset); + builder + .ins() + .stack_store(pair.tag, scratch, offset + helpers.layout.value_tag); + } + let ctx = builder.ins().load( + pointer_type, + MemFlags::new(), + helpers.frame, + helpers.layout.ctx, + ); + let receiver_address = builder.ins().stack_addr(pointer_type, scratch, 0); + let value_address = builder.ins().stack_addr(pointer_type, scratch, value_size); + let out_address = builder + .ins() + .stack_addr(pointer_type, scratch, 2 * value_size); + let atom = builder.ins().iconst(types::I32, i64::from(atom)); + let status = emit_object_leaf_call( + builder, + pointer_type, + fast.array_push_method, + &[ctx, receiver_address, atom, value_address, out_address], + ); + let succeeded = builder.ins().icmp_imm( + IntCC::Equal, + status, + i64::from(qjs::JSJitObjectStatus_JS_JIT_OBJECT_OK), + ); + builder.ins().brif(succeeded, hit, &[], fallback, &[]); + builder.seal_block(hit); + builder.seal_block(fallback); + builder.switch_to_block(hit); + let length = Pair { + payload: builder + .ins() + .stack_load(types::I64, scratch, 2 * value_size), + tag: builder.ins().stack_load( + types::I64, + scratch, + 2 * value_size + helpers.layout.value_tag, + ), + }; + define_pair(builder, helpers.stack[depth], length); + let undefined = constant_pair(builder, TaggedValue::new(0, qjs::JS_TAG_UNDEFINED as i64)); + define_pair(builder, helpers.stack[depth + 1], undefined); + define_pair(builder, helpers.stack[depth + 2], undefined); + builder.ins().jump(join, &[]); + builder.switch_to_block(fallback); + Ok(join) +} + +/// After the generic literal sequence, retain the object's final shape. +fn emit_retain_literal_shape( + builder: &mut FunctionBuilder<'_>, + helpers: &HelperLowering<'_>, + fast: ObjectFastPaths, + object_index: usize, +) -> Result<(), CompileFailure> { + let pointer_type = helpers.pointer_type; + let object = use_pair(builder, helpers.stack[object_index]); + let scratch = object_scratch(builder, 1); + builder.ins().stack_store(object.payload, scratch, 0); + builder + .ins() + .stack_store(object.tag, scratch, helpers.layout.value_tag); + let ctx = builder.ins().load( + pointer_type, + MemFlags::new(), + helpers.frame, + helpers.layout.ctx, + ); + let object_address = builder.ins().stack_addr(pointer_type, scratch, 0); + emit_object_leaf_call( + builder, + pointer_type, + fast.retain_shape, + &[ctx, object_address], + ); + Ok(()) +} + +/// One side-effect-free value of a fused object literal. Each operand is read +/// in place and duplicated by the leaf, exactly like the bytecode push would. +#[derive(Clone, Copy, Debug)] +enum LiteralValue { + Constant(TaggedValue), + Argument(u16), + Local(u16), +} + +/// `object` followed by `(pure push, define_field)*`, all inside one block. +/// Returns the atoms/values and the index of the last fused instruction. +fn object_literal_pattern( + instructions: &[crate::ir::IrInstruction], + start: usize, + argument_count: usize, + local_count: usize, + max_fields: usize, +) -> Option<(Vec, Vec, usize)> { + if !matches!(instructions.get(start)?.op, IrOp::NewObject) { + return None; + } + let mut atoms = Vec::new(); + let mut values = Vec::new(); + let mut end = start; + while atoms.len() < max_fields { + let (Some(value), Some(define)) = (instructions.get(end + 1), instructions.get(end + 2)) + else { + break; + }; + let IrOp::DefineProperty(atom) = define.op else { + break; + }; + let Some(value) = literal_operand(&value.op, argument_count, local_count) else { + break; + }; + if atom == 0 || atoms.contains(&atom) { + break; + } + atoms.push(atom); + values.push(value); + end += 2; + } + // QuickJS keeps only the final hashed shape of a literal alive (the + // transition shapes are mutated in place), so a prefix of a longer + // literal would miss on every execution. Fuse complete literals only. + literal_is_complete(instructions, end).then_some((atoms, values, end)) +} + +/// Bytecode stack effect `(pops, pushes)` of the Tier 1 operations that may +/// follow a literal. `None` stops the completeness scan conservatively. +fn literal_scan_stack_effect(op: &IrOp) -> Option<(usize, usize)> { + Some(match *op { + IrOp::Nop => (0, 0), + IrOp::Push(_) + | IrOp::ResolveConstant(_) + | IrOp::ResolveAtom(_) + | IrOp::GetGlobal(_) + | IrOp::NewObject + | IrOp::GetArgument(_) + | IrOp::GetLocal(_) + | IrOp::GetLocalChecked(_) => (0, 1), + IrOp::GetLocalPair => (0, 2), + IrOp::NewArrayFrom(count) => (usize::from(count), 1), + IrOp::GetProperty(_) | IrOp::ToPropertyKey | IrOp::Unary(_) => (1, 1), + IrOp::GetPropertyKeep(_) => (1, 2), + IrOp::SetProperty(_) => (2, 0), + IrOp::DefineProperty(_) | IrOp::GetElement | IrOp::Binary(_) => (2, 1), + IrOp::SetElement => (3, 0), + IrOp::DefineElement => (3, 1), + IrOp::Call { argc, has_this } => (usize::from(argc) + 1 + usize::from(has_this), 1), + IrOp::PutArgument { keep, .. } | IrOp::PutLocal { keep, .. } => (1, usize::from(keep)), + IrOp::Drop => (1, 0), + _ => return None, + }) +} + +/// True when the object produced by the literal ending at `end` is consumed +/// in this block before another definition could target it. +fn literal_is_complete(instructions: &[crate::ir::IrInstruction], end: usize) -> bool { + // Depths are relative to the literal object's slot, which sits at 0. + let mut depth = 1_usize; + for instruction in instructions.iter().skip(end + 1).take(64) { + match instruction.op { + IrOp::DefineProperty(_) if depth == 2 => return false, + IrOp::DefineElement if depth == 3 => return false, + _ => {} + } + let Some((pops, pushes)) = literal_scan_stack_effect(&instruction.op) else { + return true; + }; + if pops >= depth { + return true; + } + depth = depth - pops + pushes; + } + false +} + +fn object_scratch(builder: &mut FunctionBuilder<'_>, values: usize) -> StackSlot { + let bytes = values.max(1) * mem::size_of::(); + builder.create_sized_stack_slot(StackSlotData::new( + StackSlotKind::ExplicitSlot, + u32::try_from(bytes).expect("bounded scratch"), + 3, + )) +} + +fn emit_object_leaf_call( + builder: &mut FunctionBuilder<'_>, + pointer_type: cranelift_codegen::ir::Type, + target: usize, + params: &[Value], +) -> Value { + let mut signature = Signature::new(builder.func.signature.call_conv); + for param in params { + signature + .params + .push(AbiParam::new(builder.func.dfg.value_type(*param))); + } + signature.returns.push(AbiParam::new(types::I32)); + let signature = builder.import_signature(signature); + let target = builder.ins().iconst(pointer_type, target as i64); + let call = emit_external_call(builder, signature, target, params, pointer_type, None, None); + builder.inst_results(call)[0] +} + +/// Emit the literal leaf before the generic `object`/`define_field` sequence. +/// On a hit the fused result is installed and control jumps to the returned +/// join block; on a miss the builder is left in the fallback block, where the +/// caller lowers the unchanged instruction sequence and then jumps to join. +#[allow(clippy::too_many_arguments)] +fn emit_object_literal_fast_path( + builder: &mut FunctionBuilder<'_>, + helpers: &HelperLowering<'_>, + fast: ObjectFastPaths, + atoms: &[u32], + values: &[LiteralValue], + depth: usize, +) -> Result { + let pointer_type = helpers.pointer_type; + let value_slot = object_scratch(builder, values.len()); + let out_slot = object_scratch(builder, 1); + let atom_slot = builder.create_sized_stack_slot(StackSlotData::new( + StackSlotKind::ExplicitSlot, + u32::try_from(atoms.len().max(1) * 4).map_err(|_| CompileFailure::ResourceLimit)?, + 2, + )); + for (index, (atom, value)) in atoms.iter().zip(values).enumerate() { + let pair = literal_operand_pair(builder, helpers, *value)?; + let offset = i32::try_from(index * mem::size_of::()) + .map_err(|_| CompileFailure::ResourceLimit)?; + builder.ins().stack_store(pair.payload, value_slot, offset); + builder + .ins() + .stack_store(pair.tag, value_slot, offset + helpers.layout.value_tag); + let atom = builder.ins().iconst(types::I32, i64::from(*atom)); + let atom_offset = i32::try_from(index * 4).map_err(|_| CompileFailure::ResourceLimit)?; + builder.ins().stack_store(atom, atom_slot, atom_offset); + } + let ctx = builder.ins().load( + pointer_type, + MemFlags::new(), + helpers.frame, + helpers.layout.ctx, + ); + let atoms_address = builder.ins().stack_addr(pointer_type, atom_slot, 0); + let values_address = builder.ins().stack_addr(pointer_type, value_slot, 0); + let count = builder.ins().iconst( + types::I32, + i64::try_from(atoms.len()).map_err(|_| CompileFailure::ResourceLimit)?, + ); + let out_address = builder.ins().stack_addr(pointer_type, out_slot, 0); + let status = emit_object_leaf_call( + builder, + pointer_type, + fast.literal, + &[ctx, atoms_address, values_address, count, out_address], + ); + let hit = builder.create_block(); + let fallback = builder.create_block(); + let join = builder.create_block(); + let succeeded = builder.ins().icmp_imm( + IntCC::Equal, + status, + i64::from(qjs::JSJitObjectStatus_JS_JIT_OBJECT_OK), + ); + builder.ins().brif(succeeded, hit, &[], fallback, &[]); + builder.seal_block(hit); + builder.seal_block(fallback); + builder.switch_to_block(hit); + let object = Pair { + payload: builder.ins().stack_load(types::I64, out_slot, 0), + tag: builder + .ins() + .stack_load(types::I64, out_slot, helpers.layout.value_tag), + }; + define_pair(builder, helpers.stack[depth], object); + if !values.is_empty() { + let undefined = constant_pair(builder, TaggedValue::new(0, qjs::JS_TAG_UNDEFINED as i64)); + define_pair(builder, helpers.stack[depth + 1], undefined); + } + builder.ins().jump(join, &[]); + builder.switch_to_block(fallback); + Ok(join) } impl HelperLowering<'_> { @@ -552,9 +1042,9 @@ impl BaselineCompiler { let layout = FrameLayout::validated( u8::try_from(pointer_type.bytes()).map_err(|_| CompileFailure::InvalidArtifact)?, )?; - let element_layout = crate::abi::AbiInfo::linked() - .map_err(|_| CompileFailure::InvalidArtifact)? - .element_layout(); + let abi = crate::abi::AbiInfo::linked().map_err(|_| CompileFailure::InvalidArtifact)?; + let element_layout = abi.element_layout(); + let object_fast = ObjectFastPaths::from_api(abi.object_api()); let ir = match policy { CompilePolicy::AdvertisedOnly => BaselineIr::translate(function)?, #[cfg(feature = "test-support")] @@ -599,6 +1089,7 @@ impl BaselineCompiler { GuardExit::Retry, direct_calls, properties, + object_fast, )?; builder.seal_all_blocks(); builder.finalize(); @@ -2441,6 +2932,7 @@ fn lower_function( _guard_exit: GuardExit, direct_calls: &[BaselineDirectCallSite], properties: &[BaselinePropertySite], + object_fast: Option, ) -> Result<(), CompileFailure> { let pointer_type = isa.pointer_type(); let blocks: BTreeMap = ir @@ -2586,6 +3078,8 @@ fn lower_function( signatures: &helper_signatures, pointer_type, layout, + element_layout, + object_fast, }; macro_rules! invoke_helper { @@ -2613,7 +3107,20 @@ fn lower_function( let mut terminated = false; let mut entered_osr_continuation = false; let mut previous_effectful_op_was_boolean = false; - for instruction in &block.instructions { + let mut fused: Option = None; + for (instruction_index, instruction) in block.instructions.iter().enumerate() { + if fused.is_none() { + if let Some(fast) = object_fast { + fused = emit_fused_fast_path( + builder, + &helper_lowering, + fast, + &block.instructions, + instruction_index, + depth, + )?; + } + } let mut helper_states = instruction.helper_states.iter().copied(); let prior_op_was_boolean = previous_effectful_op_was_boolean; if !matches!( @@ -3599,10 +4106,27 @@ fn lower_function( if helper_states.next().is_some() { return Err(CompileFailure::InvalidArtifact); } + if let Some(region) = fused { + if instruction_index == region.end { + if terminated { + return Err(CompileFailure::InvalidArtifact); + } + if let (Some(object_index), Some(fast)) = (region.retain_literal, object_fast) { + emit_retain_literal_shape(builder, &helper_lowering, fast, object_index)?; + } + builder.ins().jump(region.join, &[]); + builder.seal_block(region.join); + builder.switch_to_block(region.join); + fused = None; + } + } if terminated { break; } } + if fused.is_some() { + return Err(CompileFailure::InvalidArtifact); + } if !terminated { if let Some(next) = ir.blocks.get(block_index + 1) { builder.ins().jump(blocks[&next.start_pc], &[]); @@ -4071,10 +4595,18 @@ fn lower_get_property_keep( .ok_or(CompileFailure::InvalidArtifact)?; let object = flat_stack_slot(helpers.ir, object_index)?; let output = flat_stack_slot(helpers.ir, *depth)?; + let state = next_helper_state(states)?; + // `receiver.method` on a fast Array whose lookup resolves to a plain data + // property of the realm's Array.prototype (for example `push`): the leaf + // returns the duplicated value without running the generic lookup helper. + let joined = helpers + .object_fast + .map(|fast| emit_array_method_fast_path(builder, helpers, fast, object_index, *depth, atom)) + .transpose()?; helpers.invoke( builder, qjs::JSJitHelperId_JS_JIT_HELPER_GET_PROPERTY, - next_helper_state(states)?, + state, *depth, *depth, &[output, object, atom], @@ -4086,10 +4618,189 @@ fn lower_get_property_keep( *depth, helpers.layout, ); + if let Some(joined) = joined { + builder.ins().jump(joined, &[]); + builder.seal_block(joined); + builder.switch_to_block(joined); + } *depth += 1; helpers.set_depth(builder, *depth) } +/// Inline fast-Array class test followed by the `array_method` leaf. Leaves +/// the builder in the generic block and returns the join block. +fn emit_array_method_fast_path( + builder: &mut FunctionBuilder<'_>, + helpers: &HelperLowering<'_>, + fast: ObjectFastPaths, + object_index: usize, + output_index: usize, + atom: u32, +) -> Result { + let pointer_type = helpers.pointer_type; + let receiver = use_pair(builder, helpers.stack[object_index]); + let class_check = builder.create_block(); + let call = builder.create_block(); + let hit = builder.create_block(); + let generic = builder.create_block(); + let joined = builder.create_block(); + let is_object = tag_is(builder, receiver.tag, qjs::JS_TAG_OBJECT); + builder + .ins() + .brif(is_object, class_check, &[], generic, &[]); + builder.seal_block(class_check); + builder.switch_to_block(class_check); + let class = builder.ins().load( + types::I16, + MemFlags::trusted(), + receiver.payload, + helpers.element_layout.object_class_id_offset, + ); + let class = builder.ins().uextend(types::I64, class); + let is_array = + builder + .ins() + .icmp_imm(IntCC::Equal, class, helpers.element_layout.array_class_id); + builder.ins().brif(is_array, call, &[], generic, &[]); + builder.seal_block(call); + builder.switch_to_block(call); + let scratch = object_scratch(builder, 2); + builder.ins().stack_store(receiver.payload, scratch, 0); + builder + .ins() + .stack_store(receiver.tag, scratch, helpers.layout.value_tag); + let ctx = builder.ins().load( + pointer_type, + MemFlags::new(), + helpers.frame, + helpers.layout.ctx, + ); + let receiver_address = builder.ins().stack_addr(pointer_type, scratch, 0); + let out_offset = + i32::try_from(mem::size_of::()).map_err(|_| CompileFailure::ResourceLimit)?; + let out_address = builder.ins().stack_addr(pointer_type, scratch, out_offset); + let atom_value = builder.ins().iconst(types::I32, i64::from(atom)); + let status = emit_object_leaf_call( + builder, + pointer_type, + fast.array_method, + &[ctx, receiver_address, atom_value, out_address], + ); + let succeeded = builder.ins().icmp_imm( + IntCC::Equal, + status, + i64::from(qjs::JSJitObjectStatus_JS_JIT_OBJECT_OK), + ); + builder.ins().brif(succeeded, hit, &[], generic, &[]); + builder.seal_block(hit); + builder.seal_block(generic); + builder.switch_to_block(hit); + let method = Pair { + payload: builder.ins().stack_load(types::I64, scratch, out_offset), + tag: builder + .ins() + .stack_load(types::I64, scratch, out_offset + helpers.layout.value_tag), + }; + define_pair(builder, helpers.stack[output_index], method); + builder.ins().jump(joined, &[]); + builder.switch_to_block(generic); + Ok(joined) +} + +/// `Array.prototype.push` with one argument: the leaf mirrors +/// `js_array_push`'s fast-array path after verifying the exact built-in +/// function, and misses (generic CALL) on anything else. All inputs remain +/// borrowed, so the caller's interpreter-order cleanup is shared by both paths. +#[allow(clippy::too_many_arguments)] +fn emit_array_push_fast_path( + builder: &mut FunctionBuilder<'_>, + helpers: &HelperLowering<'_>, + fast: ObjectFastPaths, + function_index: usize, + this_index: usize, + argument_index: usize, + output_index: usize, +) -> Result { + let pointer_type = helpers.pointer_type; + let function = use_pair(builder, helpers.stack[function_index]); + let receiver = use_pair(builder, helpers.stack[this_index]); + let argument = use_pair(builder, helpers.stack[argument_index]); + let call = builder.create_block(); + let hit = builder.create_block(); + let generic = builder.create_block(); + let joined = builder.create_block(); + let function_is_object = tag_is(builder, function.tag, qjs::JS_TAG_OBJECT); + let receiver_is_object = tag_is(builder, receiver.tag, qjs::JS_TAG_OBJECT); + let objects = builder.ins().band(function_is_object, receiver_is_object); + let class_check = builder.create_block(); + builder.ins().brif(objects, class_check, &[], generic, &[]); + builder.seal_block(class_check); + builder.switch_to_block(class_check); + // Only Array receivers can hit; other method calls go straight to CALL. + let class = builder.ins().load( + types::I16, + MemFlags::trusted(), + receiver.payload, + helpers.element_layout.object_class_id_offset, + ); + let class = builder.ins().uextend(types::I64, class); + let is_array = + builder + .ins() + .icmp_imm(IntCC::Equal, class, helpers.element_layout.array_class_id); + builder.ins().brif(is_array, call, &[], generic, &[]); + builder.seal_block(call); + builder.switch_to_block(call); + let value_size = mem::size_of::(); + let scratch = object_scratch(builder, 4); + for (index, pair) in [function, receiver, argument].into_iter().enumerate() { + let offset = + i32::try_from(index * value_size).map_err(|_| CompileFailure::ResourceLimit)?; + builder.ins().stack_store(pair.payload, scratch, offset); + builder + .ins() + .stack_store(pair.tag, scratch, offset + helpers.layout.value_tag); + } + let ctx = builder.ins().load( + pointer_type, + MemFlags::new(), + helpers.frame, + helpers.layout.ctx, + ); + let mut addresses = Vec::with_capacity(4); + for index in 0..4 { + let offset = + i32::try_from(index * value_size).map_err(|_| CompileFailure::ResourceLimit)?; + addresses.push(builder.ins().stack_addr(pointer_type, scratch, offset)); + } + let status = emit_object_leaf_call( + builder, + pointer_type, + fast.array_push, + &[ctx, addresses[0], addresses[1], addresses[2], addresses[3]], + ); + let succeeded = builder.ins().icmp_imm( + IntCC::Equal, + status, + i64::from(qjs::JSJitObjectStatus_JS_JIT_OBJECT_OK), + ); + builder.ins().brif(succeeded, hit, &[], generic, &[]); + builder.seal_block(hit); + builder.seal_block(generic); + builder.switch_to_block(hit); + let out_offset = i32::try_from(3 * value_size).map_err(|_| CompileFailure::ResourceLimit)?; + let length = Pair { + payload: builder.ins().stack_load(types::I64, scratch, out_offset), + tag: builder + .ins() + .stack_load(types::I64, scratch, out_offset + helpers.layout.value_tag), + }; + define_pair(builder, helpers.stack[output_index], length); + builder.ins().jump(joined, &[]); + builder.switch_to_block(generic); + Ok(joined) +} + fn lower_set_property( builder: &mut FunctionBuilder<'_>, helpers: &HelperLowering<'_>, @@ -5347,6 +6058,20 @@ fn lower_call( builder.switch_to_block(joined); direct_hit = Some(builder.block_params(joined)[0]); } else { + let push_joined = match helpers.object_fast { + Some(fast) if has_this && !is_constructor && argc == 1 => { + Some(emit_array_push_fast_path( + builder, + helpers, + fast, + function_index, + this_index, + argv_index, + output_index, + )?) + } + _ => None, + }; helpers.invoke( builder, if is_constructor { @@ -5372,6 +6097,11 @@ fn lower_call( output_index, helpers.layout, ); + if let Some(joined) = push_joined { + builder.ins().jump(joined, &[]); + builder.seal_block(joined); + builder.switch_to_block(joined); + } } // CALL borrows every input. The bytecode stack effect is separate and is // implemented in QuickJS interpreter order, clearing primitive inputs diff --git a/jit/tests/abi.rs b/jit/tests/abi.rs index bb3580e33..294adcf5a 100644 --- a/jit/tests/abi.rs +++ b/jit/tests/abi.rs @@ -574,7 +574,7 @@ fn helper_abi_is_one_canonical_versioned_table_in_c_bindgen_and_rust() { fn helper_abi_fields_are_append_only_tails() { use rquickjs_core::qjs; - assert_eq!(qjs::QJSJIT_ABI_MINOR, 24); + assert_eq!(qjs::QJSJIT_ABI_MINOR, 25); assert_eq!(qjs::QJSJIT_RUNTIME_API_MAJOR, 1); assert_eq!(qjs::QJSJIT_RUNTIME_API_MINOR, 9); assert_eq!(qjs::QJSJIT_HELPER_ABI_VERSION, 1); @@ -766,7 +766,11 @@ fn duplicate_attachment_does_not_replace_the_first_backend() { fn every_abi_mismatch_is_rejected_before_backend_storage() { use rquickjs_jit::test_support::AbiMismatchFixture; - for required in [AbiMismatchFixture::InlineApi, AbiMismatchFixture::ArrayApi] { + for required in [ + AbiMismatchFixture::InlineApi, + AbiMismatchFixture::ArrayApi, + AbiMismatchFixture::ObjectApi, + ] { assert!( AbiMismatchFixture::ALL.contains(&required), "missing integration fixture: {required:?}" diff --git a/jit/tests/object_fast_paths.rs b/jit/tests/object_fast_paths.rs new file mode 100644 index 000000000..2f0b15dc3 --- /dev/null +++ b/jit/tests/object_fast_paths.rs @@ -0,0 +1,397 @@ +#![cfg(all( + feature = "compiler", + target_endian = "little", + any(target_arch = "x86_64", target_arch = "aarch64"), + any(target_os = "linux", target_os = "macos", target_os = "windows"), + not(all(target_os = "windows", target_arch = "aarch64")) +))] +//! Tier 1 object-literal and `Array.prototype.push` fast paths (ABI 1.25 +//! `JSJitObjectAPI`). Every scenario is checked against the interpreter; the +//! helper counters prove that hits skip the generic helpers while misses keep +//! the exact helper semantics. + +use rquickjs::{Context, Function, Runtime}; +use rquickjs_jit::{Jit, JitConfig, JitTierPolicy}; +use std::time::{Duration, Instant}; + +unsafe extern "C" { + fn JS_JitGetHelperCount( + rt: *mut rquickjs_core::qjs::JSRuntime, + helper_id: u32, + count: *mut u64, + ) -> i32; +} + +fn helper_count(rt: *mut rquickjs_core::qjs::JSRuntime, helper_id: u32) -> u64 { + let mut count = 0; + assert_eq!( + unsafe { JS_JitGetHelperCount(rt, helper_id, &mut count) }, + 0 + ); + count +} + +fn interpreted(source: &str, expression: &str) -> String { + let runtime = Runtime::new().unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| { + ctx.eval::<(), _>(source).unwrap(); + ctx.eval::(expression).unwrap() + }) +} + +struct Native { + _jit: Jit, + context: Context, + rt: *mut rquickjs_core::qjs::JSRuntime, +} + +impl Native { + fn start(source: &str, expression: &str, stress_gc: bool) -> (Self, String) { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .tier_policy(JitTierPolicy::BaselineOnly) + .call_threshold(1) + .loop_threshold(1) + .stress_gc(stress_gc) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| ctx.eval::<(), _>(source)).unwrap(); + context + .with(|ctx| { + ctx.eval::<(), _>(format!( + "globalThis.__objectFastInvoke=function(){{return {expression}}}" + )) + }) + .unwrap(); + let native = Self { + _jit: jit, + rt: context + .with(|ctx| unsafe { rquickjs_core::qjs::JS_GetRuntime(ctx.as_raw().as_ptr()) }), + context, + }; + let deadline = Instant::now() + Duration::from_secs(60); + let mut result = native.invoke(); + loop { + native._jit.poll(); + let metrics = native._jit.metrics(); + if metrics.native_entries > 0 && metrics.pending_worker_jobs == 0 { + break; + } + assert!(Instant::now() < deadline, "{metrics:?}"); + std::thread::sleep(Duration::from_millis(1)); + result = native.invoke(); + } + (native, result) + } + + fn invoke(&self) -> String { + self.context.with(|ctx| { + let invoke: Function<'_> = ctx.globals().get("__objectFastInvoke").unwrap(); + let result = invoke.call::<_, String>(()); + assert!(result.is_ok(), "{result:?}: {:?}", ctx.catch()); + result.unwrap() + }) + } + + fn helper(&self, id: u32) -> u64 { + helper_count(self.rt, id) + } +} + +fn assert_differential(source: &str, expression: &str, stress_gc: bool) -> Native { + let expected = interpreted(source, expression); + let (native, first) = Native::start(source, expression, stress_gc); + assert_eq!(first, expected, "warmup result"); + for _ in 0..8 { + assert_eq!(native.invoke(), expected); + native._jit.poll(); + } + assert!(native._jit.metrics().native_entries > 0); + native +} + +const LITERALS: &str = r#" +function make(i, s, o) { + const plain = {a: i, b: s, c: true, d: null, e: undefined, f: -1, g: o}; + const numeric = {0: i, 1: s}; + const dup = {k: i, k: s}; + const empty = {}; + return [plain, numeric, dup, empty]; +} +function run() { + const keep = []; + let text = ''; + for (let i = 0; i < 300; i++) { + const shared = {tag: i}; + const made = make(i, 'v' + (i & 3), shared); + keep.push(made); + text = Object.keys(made[0]).join(',') + '|' + JSON.stringify(made[0]) + '|' + + JSON.stringify(made[1]) + '|' + JSON.stringify(made[2]) + '|' + + Object.keys(made[3]).length; + } + const first = keep[3][0]; + first.z = 1; + delete first.a; + return text + '|' + keep.length + '|' + JSON.stringify(first) + '|' + + JSON.stringify(keep[4][0]) + '|' + (keep[5][0].g === keep[5][0].g) + '|' + + Object.isExtensible(keep[6][0]) + '|' + + JSON.stringify(Object.getOwnPropertyDescriptor(keep[7][0], 'a')); +} +"#; + +#[test] +fn object_literals_match_the_interpreter() { + assert_differential(LITERALS, "run()", false); +} + +#[test] +fn complete_object_literals_skip_allocation_and_define_helpers() { + let source = r#" +function make(i, s, o) { + const point = {x: i, y: s, owner: o, flag: false}; + const pair = {0: i, 1: s}; + return [point, pair]; +} +function run() { + const keep = []; + for (let i = 0; i < 200; i++) keep.push(make(i, 'v' + (i & 7), keep)); + return JSON.stringify(keep[123][1]) + ':' + keep[199][0].x + keep[5][0].y + + (keep[9][0].owner === keep) + keep[3][0].flag; +} +"#; + let native = assert_differential(source, "run()", false); + let new_object = native.helper(rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_NEW_OBJECT); + let set_property = native.helper(rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_SET_PROPERTY); + native.invoke(); + // Once each final shape exists, `make` builds both literals in the leaf. + assert_eq!( + native.helper(rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_NEW_OBJECT) - new_object, + 0 + ); + // Only the two `array_from` element stores of `[point, pair]` remain. + assert_eq!( + native.helper(rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_SET_PROPERTY) - set_property, + 400 + ); +} + +#[test] +fn object_literals_survive_stress_gc() { + assert_differential(LITERALS, "run()", true); +} + +#[test] +fn object_literal_defines_own_fields_despite_prototype_accessors() { + let source = r#" +let hits = 0; +Object.defineProperty(Object.prototype, 'x', { + set(v) { hits++; }, get() { return 'proto'; }, configurable: true +}); +function lit(i) { return {x: i, y: 2}; } +function run() { + let text = ''; + for (let i = 0; i < 200; i++) { + const o = lit(i); + text = o.x + ':' + Object.getOwnPropertyNames(o).join(',') + ':' + o.hasOwnProperty('x'); + } + return text + ':' + hits; +} +"#; + assert_differential(source, "run()", false); +} + +#[test] +fn object_literal_uses_the_current_object_prototype() { + let source = r#" +function lit(i) { return {p: i}; } +function run() { + let text = ''; + for (let i = 0; i < 100; i++) text = String(lit(i).marker); + Object.prototype.marker = 'installed'; + for (let i = 0; i < 100; i++) text += ':' + lit(i).marker; + delete Object.prototype.marker; + return text + ':' + (Object.getPrototypeOf(lit(1)) === Object.prototype); +} +"#; + assert_differential(source, "run()", false); +} + +const PUSHES: &str = r#" +function pushAll(array, count, base) { + let length = 0; + for (let i = 0; i < count; i++) length = array.push(base + i); + return length; +} +function pushObjects(array, count) { + for (let i = 0; i < count; i++) array.push({index: i, score: i * 3}); + return array.length; +} +function attempt(array, count) { + try { return String(pushAll(array, count, 0)); } catch (e) { return e.constructor.name; } +} +function run() { + const out = []; + const grow = []; + out.push(pushAll(grow, 1000, 1), grow[0], grow[999], grow.length); + const objects = []; + out.push(pushObjects(objects, 100), objects[50].score); + out.push(attempt(Object.freeze([1]), 1)); + const sealed = Object.seal([1, 2]); + out.push(attempt(sealed, 1), sealed.length); + const readonly = [1, 2]; + Object.defineProperty(readonly, 'length', {writable: false}); + out.push(attempt(readonly, 1), readonly.length); + const own = []; + own.push = function (v) { return 'own:' + v; }; + out.push(pushAll(own, 2, 5), own.length); + class Sub extends Array { push(v) { return super.push(v * 10); } } + const sub = new Sub(); + out.push(pushAll(sub, 3, 1), JSON.stringify(sub)); + const sparse = [1, 2, 3]; + sparse.length = 6; + out.push(pushAll(sparse, 2, 7), JSON.stringify(sparse)); + const holey = [1, , 3]; + out.push(pushAll(holey, 1, 9), JSON.stringify(holey)); + const arrayLike = {length: 1, push: Array.prototype.push}; + out.push(pushAll(arrayLike, 2, 4), JSON.stringify(arrayLike)); + const saved = Array.prototype.push; + let intercepted = 0; + Array.prototype.push = function (v) { intercepted++; return saved.call(this, -v); }; + const patched = []; + out.push(pushAll(patched, 3, 1), JSON.stringify(patched), intercepted); + Array.prototype.push = saved; + const unshift = []; + unshift.push = Array.prototype.unshift; + out.push(pushAll(unshift, 3, 1), JSON.stringify(unshift)); + Object.defineProperty(Array.prototype, 'push', { + get() { intercepted += 100; return saved; }, configurable: true + }); + const accessor = []; + out.push(pushAll(accessor, 2, 1), intercepted); + Object.defineProperty(Array.prototype, 'push', { + value: saved, writable: true, enumerable: false, configurable: true + }); + return JSON.stringify(out); +} +"#; + +#[test] +fn array_push_matches_the_interpreter_on_hits_and_every_miss_shape() { + assert_differential(PUSHES, "run()", false); +} + +#[test] +fn array_push_survives_stress_gc() { + assert_differential(PUSHES, "run()", true); +} + +#[test] +fn packed_array_push_skips_the_lookup_and_call_helpers() { + let source = r#" +function fill(count) { + const values = []; + for (let i = 0; i < count; i++) values.push((i * 17) & 1023); + let sum = 0; + for (let i = 0; i < values.length; i++) sum += values[i]; + return sum; +} +"#; + let native = assert_differential(source, "String(fill(500))", false); + let call = native.helper(rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_CALL); + let get = native.helper(rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_GET_PROPERTY); + native.invoke(); + // Only the native invocation wrapper's `fill(...)` and `String(...)` calls + // use the generic CALL helper; the 500 pushes and their `push` lookups + // are handled by the leaves. + assert_eq!( + native.helper(rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_CALL) - call, + 2 + ); + let lookups = native.helper(rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_GET_PROPERTY) - get; + // `values.length` (501 reads) still uses the generic lookup helper. + assert!(lookups <= 501, "{lookups}"); +} + +#[test] +fn polymorphic_object_workload_matches_the_interpreter() { + let source = r#" +function workload(iterations, seed) { + const objects = []; + for (let i = 0; i < iterations; i++) { + let object; + switch (i & 3) { + case 0: object = { x: i, y: seed, kind: 0 }; break; + case 1: object = { y: seed, x: i, extra: 1, kind: 1 }; break; + case 2: object = { x: i, kind: 2 }; object.y = seed; break; + default: object = Object.create(null); object.kind = 3; object.x = i; object.y = seed; + } + object.x += object.kind; + objects.push(object); + } + let sum = 0; + for (let i = 0; i < objects.length; i++) sum += objects[i].x + objects[i].y; + return sum; +} +"#; + assert_differential(source, "String(workload(2000, 7))", true); +} + +#[test] +fn push_statements_on_non_arrays_and_uninitialized_bindings_stay_exact() { + let source = r#" +function drive(target, value) { + let out = 0; + for (let i = 0; i < 50; i++) out = target.push(value); + return out; +} +function lexical(flag) { + if (flag) return early.push(1); + let early = []; + early.push(2); + return early.length; +} +function run() { + const out = []; + const counter = {n: 0, push(v) { this.n += v; return this.n; }}; + out.push(drive(counter, 2), drive([], 'x'), drive(new Array(3), 1)); + try { lexical(true); } catch (e) { out.push(e.constructor.name); } + out.push(lexical(false)); + try { drive(null, 1); } catch (e) { out.push(e.constructor.name); } + return JSON.stringify(out); +} +"#; + assert_differential(source, "run()", true); +} + +/// Pre-existing on 82d3808: the generic Tier 1 lowering implements +/// `define_field` with set semantics (`JS_JIT_HELPER_SET_PROPERTY`), so an +/// inherited setter runs and no own property is created. Only literals that +/// the new leaf builds are exact; this non-fusable literal still diverges. +/// Fixing it needs an exact define helper (see docs/roadmap/p4e-alloc-fastpath.md). +#[test] +#[ignore = "pre-existing Tier 1 define_field set-semantics bug"] +fn generic_define_field_ignores_inherited_setters() { + let source = r#" +let hits = 0; +Object.defineProperty(Object.prototype, 'x', { + set(v) { hits++; }, get() { return 'proto'; }, configurable: true +}); +function id(v) { return v; } +function lit(i) { return {x: i, y: id(2)}; } +function run() { + let text = ''; + for (let i = 0; i < 200; i++) { + const o = lit(i); + text = o.x + ':' + Object.getOwnPropertyNames(o).join(','); + } + return text + ':' + hits; +} +"#; + assert_differential(source, "run()", false); +} diff --git a/jit/tests/support/mod.rs b/jit/tests/support/mod.rs index bc1ada728..534fbd0cd 100644 --- a/jit/tests/support/mod.rs +++ b/jit/tests/support/mod.rs @@ -1667,11 +1667,12 @@ pub enum AbiMismatchFixture { ElementLayout, InlineApi, ArrayApi, + ObjectApi, BackendVTableLayout, } impl AbiMismatchFixture { - pub const ALL: [Self; 19] = [ + pub const ALL: [Self; 20] = [ Self::SourceRevision, Self::OpcodeFingerprint, Self::ValueLayout, @@ -1690,6 +1691,7 @@ impl AbiMismatchFixture { Self::ElementLayout, Self::InlineApi, Self::ArrayApi, + Self::ObjectApi, Self::BackendVTableLayout, ]; @@ -1715,6 +1717,7 @@ impl AbiMismatchFixture { Self::ElementLayout => AbiMismatch::StructureLayout(AbiStructure::ElementLayout), Self::InlineApi => AbiMismatch::StructureLayout(AbiStructure::InlineApi), Self::ArrayApi => AbiMismatch::StructureLayout(AbiStructure::ArrayApi), + Self::ObjectApi => AbiMismatch::StructureLayout(AbiStructure::ObjectApi), Self::BackendVTableLayout => AbiMismatch::StructureLayout(AbiStructure::BackendVTable), } } diff --git a/sys/build_support/patch.rs b/sys/build_support/patch.rs index 41317e2f2..307d9afd1 100644 --- a/sys/build_support/patch.rs +++ b/sys/build_support/patch.rs @@ -1,7 +1,7 @@ use std::{fs, io, path::Path}; const QUICKJS_BASELINE_FNV64: u64 = 0x3302_116a_b0fc_269c; -const EXPECTED_PATCHES: [(&str, u64); 19] = [ +const EXPECTED_PATCHES: [(&str, u64); 20] = [ ("0001-rquickjs-jit.patch", 0x18d7_40cc_6943_30cf), ("0002-runtime-feedback.patch", 0xe4c1_ed52_ee5d_d469), ("0003-element-layout.patch", 0xa12c_ccd2_7420_e880), @@ -21,6 +21,7 @@ const EXPECTED_PATCHES: [(&str, u64); 19] = [ ("0018-inline-frame-recovery.patch", 0xb971d61edccde229), ("0019-array-feedback.patch", 0x11c3cb13a8297dd4), ("0020-typed-array-guard.patch", 0x5d10_38a3_8bc9_9821), + ("0032-object-fast-paths.patch", 0x4453_a265_1fe9_be85), ]; pub(crate) const BASELINE_FILES: [&str; 19] = [ "api-test.c", @@ -60,8 +61,8 @@ const PATCHED_FILE_FINGERPRINTS: [(&str, u64); 21] = [ ("list.h", 0xb337_70f7_b76d_a3d8), ("quickjs-atom.h", 0x30b4_9116_b6a2_aa99), ("quickjs-c-atomics.h", 0x490b_0f29_f631_3fc0), - ("quickjs.c", 0xc535_b5f1_48e7_9b27), - ("quickjs-jit.h", 0x0188_04ad_21d2_95a7), + ("quickjs.c", 0x8897_fc33_ebd6_d570), + ("quickjs-jit.h", 0x58b7_1630_5510_6aa0), ("quickjs-jit-helpers.h", 0xa469_e80f_be47_404c), ("quickjs-opcode.h", 0x3d05_cfdf_5cf7_2930), ("quickjs.h", 0x4831_2cde_9c2f_a5ee), diff --git a/sys/patches/0032-object-fast-paths.patch b/sys/patches/0032-object-fast-paths.patch new file mode 100644 index 000000000..4f60698ed --- /dev/null +++ b/sys/patches/0032-object-fast-paths.patch @@ -0,0 +1,356 @@ +diff --git a/quickjs-jit.h b/quickjs-jit.h +--- a/quickjs-jit.h ++++ b/quickjs-jit.h +@@ -17,7 +17,7 @@ + #endif + + #define QJSJIT_ABI_MAJOR 1u +-#define QJSJIT_ABI_MINOR 24u ++#define QJSJIT_ABI_MINOR 25u + + /* Frame flags are append-only. Stress collection is a test/debug contract: + * every helper performs an allocating cycle-GC point before and after its +@@ -130,6 +130,53 @@ + } JSJitArrayAPI; + + JS_EXTERN const JSJitArrayAPI *JS_JitGetArrayAPI(uint32_t version); ++ ++/* Versioned allocation and array-method fast paths (ABI 1.25). Every entry ++ * either completes the exact interpreter effect or misses with no observable ++ * effect; it never throws, calls user code or reenters the VM. `literal` may ++ * run the same allocation-triggered cycle collection as OP_object (C ++ * finalizers only) and allocates without throwing. `retain_shape` keeps a ++ * literal's final hashed shape in the bounded per-context JIT shape cache. ++ * The array entries only duplicate references and, for push, may grow the ++ * element buffer without throwing. Inputs are borrowed; a hit returns an owned ++ * value in `out`, a miss leaves `out` undefined. */ ++#define QJSJIT_OBJECT_API_VERSION 1u ++#define JS_JIT_OBJECT_LITERAL_MAX_FIELDS 16u ++#define JS_JIT_OBJECT_EFFECT_ALLOCATE (1U << 0) ++#define JS_JIT_OBJECT_EFFECT_COLLECT (1U << 1) ++ ++typedef enum JSJitObjectStatus { ++ JS_JIT_OBJECT_INVALID = -1, ++ JS_JIT_OBJECT_MISS = 0, ++ JS_JIT_OBJECT_OK = 1, ++} JSJitObjectStatus; ++ ++typedef int JSJitObjectLiteralFunc(JSContext *ctx, const uint32_t *atoms, ++ const JSValue *values, uint32_t count, ++ JSValue *out); ++typedef int JSJitObjectRetainShapeFunc(JSContext *ctx, const JSValue *object); ++typedef int JSJitArrayMethodFunc(JSContext *ctx, const JSValue *receiver, ++ uint32_t atom, JSValue *out); ++typedef int JSJitArrayPushFunc(JSContext *ctx, const JSValue *function, ++ const JSValue *receiver, const JSValue *value, ++ JSValue *out); ++typedef int JSJitArrayPushMethodFunc(JSContext *ctx, const JSValue *receiver, ++ uint32_t atom, const JSValue *value, ++ JSValue *out); ++ ++typedef struct JSJitObjectAPI { ++ uint32_t struct_size; ++ uint32_t version; ++ uint32_t effects; ++ uint32_t max_literal_fields; ++ JSJitObjectLiteralFunc *literal; ++ JSJitObjectRetainShapeFunc *retain_shape; ++ JSJitArrayMethodFunc *array_method; ++ JSJitArrayPushFunc *array_push; ++ JSJitArrayPushMethodFunc *array_push_method; ++} JSJitObjectAPI; ++ ++JS_EXTERN const JSJitObjectAPI *JS_JitGetObjectAPI(uint32_t version); + + #define JS_JIT_FEEDBACK_ARRAY_STORE (1U << 6) + #define JS_JIT_FEEDBACK_ARRAY_LENGTH (1U << 7) +diff --git a/quickjs.c b/quickjs.c +--- a/quickjs.c ++++ b/quickjs.c +@@ -46928,7 +46928,11 @@ + (p->shape->prop->flags & JS_PROP_WRITABLE))) { + array_len = JS_VALUE_GET_INT(p->prop[0].u.value); + new_len = array_len + argc; +- if (likely(new_len >= array_len && new_len <= (uint32_t)INT32_MAX)) { /* no overflow and within fast-array bounds */ ++ /* A fast array may keep count < length after `length` grows ++ (set_array_length); appending at `length` would leave ++ uninitialized holes, so take the generic path. */ ++ if (likely(new_len >= array_len && new_len <= (uint32_t)INT32_MAX && ++ array_len == p->u.array.count)) { /* no overflow and within fast-array bounds */ + if (unlikely(new_len > p->u.array.u1.size)) { + if (expand_fast_array(ctx, p, new_len)) + return JS_EXCEPTION; +@@ -46972,6 +46976,273 @@ + JS_FreeValue(ctx, obj); + return JS_EXCEPTION; + } ++ ++#ifdef CONFIG_JIT_ABI ++/* Native allocation and array-method fast paths. Each entry either performs ++ * exactly the interpreter's effect or returns a miss after touching nothing ++ * observable, so generated code can always fall back to its generic helpers. ++ * None of them throws, calls user code or reenters the VM. */ ++ ++/* Object literal `{a: v0, b: v1, ...}` built by OP_object followed by one ++ * OP_define_field per atom. The final hashed shape is a pure function of the ++ * realm's Object.prototype and the ordered (atom, JS_PROP_C_W_E) list, so it ++ * is found directly in the runtime shape hash without the intermediate ++ * transition shapes (which QuickJS mutates in place and does not retain). ++ * Structurally equal hashed shapes are semantically indistinguishable. A ++ * missing shape, duplicate atom, uninitialized value or allocation failure ++ * misses without throwing; the generic path then reproduces every effect. */ ++static int qjsjit_object_literal(JSContext *ctx, const uint32_t *atoms, ++ const JSValue *values, uint32_t count, ++ JSValue *out) ++{ ++ JSRuntime *rt; ++ JSObject *proto, *p; ++ JSShape *sh; ++ JSProperty *props; ++ uint32_t h, i, j; ++ ++ if (!ctx || !out || count > JS_JIT_OBJECT_LITERAL_MAX_FIELDS || ++ (count != 0 && (!atoms || !values))) ++ return JS_JIT_OBJECT_INVALID; ++ *out = JS_UNDEFINED; ++ rt = ctx->rt; ++ for (i = 0; i < count; i++) { ++ if (!qjsjit_valid_atom(rt, atoms[i])) ++ return JS_JIT_OBJECT_INVALID; ++ if (JS_VALUE_GET_TAG(values[i]) == JS_TAG_UNINITIALIZED) ++ return JS_JIT_OBJECT_MISS; ++ for (j = 0; j < i; j++) { ++ if (atoms[j] == atoms[i]) ++ return JS_JIT_OBJECT_MISS; ++ } ++ } ++ if (JS_VALUE_GET_TAG(ctx->class_proto[JS_CLASS_OBJECT]) != JS_TAG_OBJECT) ++ return JS_JIT_OBJECT_MISS; ++ /* Same collection point as JS_NewObjectFromShape(). Look the shape up ++ afterwards: a collection can free unreferenced hashed shapes. */ ++ js_trigger_gc(rt, sizeof(JSObject)); ++ proto = JS_VALUE_GET_OBJ(ctx->class_proto[JS_CLASS_OBJECT]); ++ h = shape_initial_hash(proto); ++ for (i = 0; i < count; i++) ++ h = shape_hash(shape_hash(h, atoms[i]), JS_PROP_C_W_E); ++ for (sh = rt->shape_hash[get_shape_hash(h, rt->shape_hash_bits)]; ++ sh != NULL; sh = sh->shape_hash_next) { ++ if (sh->hash != h || sh->proto != proto || !sh->is_hashed || ++ sh->prop_count != count || sh->deleted_prop_count != 0 || ++ sh->prop_size < count) ++ continue; ++ for (i = 0; i < count; i++) { ++ if (sh->prop[i].atom != atoms[i] || ++ sh->prop[i].flags != JS_PROP_C_W_E) ++ break; ++ } ++ if (i == count) ++ break; ++ } ++ if (!sh || sh->prop_size == 0) ++ return JS_JIT_OBJECT_MISS; ++ p = js_malloc_rt(rt, sizeof(JSObject)); ++ if (unlikely(!p)) ++ return JS_JIT_OBJECT_MISS; ++ props = js_malloc_rt(rt, sizeof(JSProperty) * sh->prop_size); ++ if (unlikely(!props)) { ++ js_free_rt(rt, p); ++ return JS_JIT_OBJECT_MISS; ++ } ++ /* Mirrors JS_NewObjectFromShape() for JS_CLASS_OBJECT. */ ++ p->class_id = JS_CLASS_OBJECT; ++ p->extensible = true; ++ p->free_mark = 0; ++ p->is_exotic = 0; ++ p->fast_array = 0; ++ p->is_constructor = 0; ++ p->is_uncatchable_error = 0; ++ p->tmp_mark = 0; ++ p->is_HTMLDDA = 0; ++ p->is_prototype = 0; ++ p->first_weak_ref = NULL; ++ p->u.opaque = NULL; ++ p->shape = js_dup_shape(sh); ++ p->prop = props; ++ for (i = 0; i < count; i++) ++ props[i].u.value = js_dup(values[i]); ++ p->header.ref_count = 1; ++ add_gc_object(rt, &p->header, JS_GC_OBJ_TYPE_JS_OBJECT); ++ *out = JS_MKPTR(JS_TAG_OBJECT, p); ++ return JS_JIT_OBJECT_OK; ++} ++ ++/* Keep the final shape of a literal built on the generic path alive in the ++ * bounded, GC-traced per-context JIT shape cache (0012), so the next ++ * `literal` lookup finds it even when every receiver is later extended. */ ++static int qjsjit_object_retain_shape(JSContext *ctx, const JSValue *object) ++{ ++ JSObject *p; ++ ++ if (!ctx || !object) ++ return JS_JIT_OBJECT_INVALID; ++ if (JS_VALUE_GET_TAG(*object) != JS_TAG_OBJECT) ++ return JS_JIT_OBJECT_MISS; ++ p = JS_VALUE_GET_OBJ(*object); ++ if (p->class_id != JS_CLASS_OBJECT || !p->shape->is_hashed) ++ return JS_JIT_OBJECT_MISS; ++ qjsjit_remember_shape(ctx, p); ++ return JS_JIT_OBJECT_OK; ++} ++ ++/* Exact OP_get_field2 lookup on a fast Array when it resolves, without ++ * exotic handlers, getters or auto-initialization, to an own data property ++ * of the realm's Array.prototype. Returns the borrowed property value. */ ++static JSValue *qjsjit_array_method_slot(JSContext *ctx, JSValueConst receiver, ++ uint32_t atom) ++{ ++ JSObject *p, *proto; ++ JSProperty *pr; ++ JSShapeProperty *prs; ++ ++ if (JS_VALUE_GET_TAG(receiver) != JS_TAG_OBJECT || ++ __JS_AtomIsTaggedInt(atom) || ++ JS_VALUE_GET_TAG(ctx->class_proto[JS_CLASS_ARRAY]) != JS_TAG_OBJECT) ++ return NULL; ++ p = JS_VALUE_GET_OBJ(receiver); ++ if (p->class_id != JS_CLASS_ARRAY || !p->fast_array) ++ return NULL; ++ if (find_own_property(&pr, p, atom)) ++ return NULL; ++ proto = p->shape->proto; ++ if (!proto || proto != JS_VALUE_GET_OBJ(ctx->class_proto[JS_CLASS_ARRAY])) ++ return NULL; ++ prs = find_own_property(&pr, proto, atom); ++ if (!prs || (prs->flags & JS_PROP_TMASK)) ++ return NULL; ++ return &pr->u.value; ++} ++ ++static int qjsjit_array_method(JSContext *ctx, const JSValue *receiver, ++ uint32_t atom, JSValue *out) ++{ ++ JSValue *method; ++ ++ if (!ctx || !receiver || !out || !qjsjit_valid_atom(ctx->rt, atom)) ++ return JS_JIT_OBJECT_INVALID; ++ *out = JS_UNDEFINED; ++ method = qjsjit_array_method_slot(ctx, *receiver, atom); ++ if (!method) ++ return JS_JIT_OBJECT_MISS; ++ *out = js_dup(*method); ++ return JS_JIT_OBJECT_OK; ++} ++ ++/* A one-argument call of the built-in Array.prototype.push. Mirrors the ++ * js_call_c_function() stack check and js_array_push()'s fast-array path; ++ * growth uses the same policy as expand_fast_array() without throwing. */ ++static int qjsjit_array_push_one(JSContext *ctx, JSValueConst function, ++ JSValueConst receiver, JSValueConst value, ++ JSValue *out) ++{ ++ JSRuntime *rt = ctx->rt; ++ JSContext *realm; ++ JSObject *f, *p; ++ JSValue *values; ++ uint32_t array_len, new_len, old_size, new_size; ++ size_t bytes, usable; ++ ++ if (JS_VALUE_GET_TAG(function) != JS_TAG_OBJECT || ++ JS_VALUE_GET_TAG(receiver) != JS_TAG_OBJECT || ++ JS_VALUE_GET_TAG(value) == JS_TAG_UNINITIALIZED) ++ return JS_JIT_OBJECT_MISS; ++ f = JS_VALUE_GET_OBJ(function); ++ if (f->class_id != JS_CLASS_C_FUNCTION || ++ f->u.cfunc.cproto != JS_CFUNC_generic_magic || ++ f->u.cfunc.c_function.generic_magic != js_array_push || ++ f->u.cfunc.magic != 0) ++ return JS_JIT_OBJECT_MISS; ++ realm = f->u.cfunc.realm; ++ if (!realm || realm->rt != rt || ++ js_check_stack_overflow(rt, sizeof(JSValue) * f->u.cfunc.length)) ++ return JS_JIT_OBJECT_MISS; ++ p = JS_VALUE_GET_OBJ(receiver); ++ if (p->class_id != JS_CLASS_ARRAY || !p->fast_array || !p->extensible || ++ JS_VALUE_GET_TAG(realm->class_proto[JS_CLASS_ARRAY]) != JS_TAG_OBJECT || ++ p->shape->proto != JS_VALUE_GET_OBJ(realm->class_proto[JS_CLASS_ARRAY]) || ++ !realm->std_array_prototype) ++ return JS_JIT_OBJECT_MISS; ++ if (JS_VALUE_GET_TAG(p->prop[0].u.value) != JS_TAG_INT || ++ !(p->shape->prop->flags & JS_PROP_WRITABLE)) ++ return JS_JIT_OBJECT_MISS; ++ array_len = JS_VALUE_GET_INT(p->prop[0].u.value); ++ if (array_len != p->u.array.count || array_len >= (uint32_t)INT32_MAX) ++ return JS_JIT_OBJECT_MISS; ++ new_len = array_len + 1; ++ if (new_len > p->u.array.u1.size) { ++ old_size = p->u.array.u1.size; ++ new_size = old_size + old_size / 2; ++ if (new_size < old_size) ++ return JS_JIT_OBJECT_MISS; ++ new_size = max_uint32(new_len, new_size); ++ if ((size_t)new_size > SIZE_MAX / sizeof(JSValue)) ++ return JS_JIT_OBJECT_MISS; ++ bytes = sizeof(JSValue) * new_size; ++ values = js_realloc_rt(rt, p->u.array.u.values, bytes); ++ if (!values) ++ return JS_JIT_OBJECT_MISS; ++ usable = js_malloc_usable_size_rt(rt, values); ++ if (usable > bytes) ++ new_size += (usable - bytes) / sizeof(JSValue); ++ p->u.array.u.values = values; ++ p->u.array.u1.size = new_size; ++ } ++ p->u.array.u.values[array_len] = js_dup(value); ++ p->u.array.count = new_len; ++ p->prop[0].u.value = js_uint32(new_len); ++ *out = js_int32(new_len); ++ return JS_JIT_OBJECT_OK; ++} ++ ++static int qjsjit_array_push(JSContext *ctx, const JSValue *function, ++ const JSValue *receiver, const JSValue *value, ++ JSValue *out) ++{ ++ if (!ctx || !function || !receiver || !value || !out) ++ return JS_JIT_OBJECT_INVALID; ++ *out = JS_UNDEFINED; ++ return qjsjit_array_push_one(ctx, *function, *receiver, *value, out); ++} ++ ++/* The whole `receiver.(value)` statement when the method resolves to ++ * the built-in push: the lookup and call above without materializing the ++ * method, receiver or argument references. */ ++static int qjsjit_array_push_method(JSContext *ctx, const JSValue *receiver, ++ uint32_t atom, const JSValue *value, ++ JSValue *out) ++{ ++ JSValue *method; ++ ++ if (!ctx || !receiver || !value || !out || ++ !qjsjit_valid_atom(ctx->rt, atom)) ++ return JS_JIT_OBJECT_INVALID; ++ *out = JS_UNDEFINED; ++ method = qjsjit_array_method_slot(ctx, *receiver, atom); ++ if (!method) ++ return JS_JIT_OBJECT_MISS; ++ return qjsjit_array_push_one(ctx, *method, *receiver, *value, out); ++} ++ ++const JSJitObjectAPI *JS_JitGetObjectAPI(uint32_t version) ++{ ++ static const JSJitObjectAPI api = { ++ sizeof(JSJitObjectAPI), QJSJIT_OBJECT_API_VERSION, ++ JS_JIT_OBJECT_EFFECT_ALLOCATE | JS_JIT_OBJECT_EFFECT_COLLECT, ++ JS_JIT_OBJECT_LITERAL_MAX_FIELDS, ++ qjsjit_object_literal, ++ qjsjit_object_retain_shape, ++ qjsjit_array_method, ++ qjsjit_array_push, ++ qjsjit_array_push_method, ++ }; ++ return version == QJSJIT_OBJECT_API_VERSION ? &api : NULL; ++} ++#endif + + static JSValue js_array_reverse(JSContext *ctx, JSValueConst this_val, + int argc, JSValueConst *argv) diff --git a/sys/patches/README.md b/sys/patches/README.md index 931cf9bd3..6a7df35b9 100644 --- a/sys/patches/README.md +++ b/sys/patches/README.md @@ -60,6 +60,23 @@ data pointer before returning `{ data, count, mode }`. The table advertises zero effects and the receiver is passed by pointer so generated calls do not depend on platform-specific aggregate argument classification. +`0032-object-fast-paths.patch` adds the ABI 1.25 versioned object table +(`JS_JitGetObjectAPI`). Its leaves either complete the interpreter's exact +effect or miss without an observable effect; none throws, calls user code or +reenters the VM, so native code keeps the generic helpers as the fallback. +`literal` builds `{a: v0, ...}` (OP_object plus one OP_define_field per atom) +from the existing final hashed shape, found directly from the +`(Object.prototype, atom, JS_PROP_C_W_E)` hash chain, with non-throwing +allocation and the same allocation-time GC trigger as `JS_NewObjectFromShape`. +`retain_shape` keeps a generically built literal's final shape in the bounded +0012 shape cache. `array_method` performs the exact `get_field2` lookup of an +`Array.prototype` data property on fast arrays; `array_push` and +`array_push_method` mirror a one-argument call of the built-in +`Array.prototype.push` (stack check, fast-array path, non-throwing growth). +The patch also fixes an upstream `js_array_push` fast path that appended at +`length` when a fast array kept `count < length` after `length` grew, leaving +uninitialized element slots (a crash in the unmodified interpreter). + The build accepts only the patch names and byte digests listed in `build_support/patch.rs`, then verifies the complete patched source manifest. Keeping the baseline and patch separate makes diff --git a/sys/src/bindings/aarch64-apple-darwin.rs b/sys/src/bindings/aarch64-apple-darwin.rs index a64c09ee5..d9c084062 100644 --- a/sys/src/bindings/aarch64-apple-darwin.rs +++ b/sys/src/bindings/aarch64-apple-darwin.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,10 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_OBJECT_API_VERSION: u32 = 1; +pub const JS_JIT_OBJECT_LITERAL_MAX_FIELDS: u32 = 16; +pub const JS_JIT_OBJECT_EFFECT_ALLOCATE: u32 = 1; +pub const JS_JIT_OBJECT_EFFECT_COLLECT: u32 = 2; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -2275,6 +2279,87 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitObjectStatus_JS_JIT_OBJECT_INVALID: JSJitObjectStatus = -1; +pub const JSJitObjectStatus_JS_JIT_OBJECT_MISS: JSJitObjectStatus = 0; +pub const JSJitObjectStatus_JS_JIT_OBJECT_OK: JSJitObjectStatus = 1; +pub type JSJitObjectStatus = ::core::ffi::c_int; +pub type JSJitObjectLiteralFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + atoms: *const u32, + values: *const JSValue, + count: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitObjectRetainShapeFunc = ::core::option::Option< + unsafe extern "C" fn(ctx: *mut JSContext, object: *const JSValue) -> ::core::ffi::c_int, +>; +pub type JSJitArrayMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + function: *const JSValue, + receiver: *const JSValue, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitObjectAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub max_literal_fields: u32, + pub literal: JSJitObjectLiteralFunc, + pub retain_shape: JSJitObjectRetainShapeFunc, + pub array_method: JSJitArrayMethodFunc, + pub array_push: JSJitArrayPushFunc, + pub array_push_method: JSJitArrayPushMethodFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitObjectAPI"][::core::mem::size_of::() - 56usize]; + ["Alignment of JSJitObjectAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitObjectAPI::struct_size"] + [::core::mem::offset_of!(JSJitObjectAPI, struct_size) - 0usize]; + ["Offset of field: JSJitObjectAPI::version"] + [::core::mem::offset_of!(JSJitObjectAPI, version) - 4usize]; + ["Offset of field: JSJitObjectAPI::effects"] + [::core::mem::offset_of!(JSJitObjectAPI, effects) - 8usize]; + ["Offset of field: JSJitObjectAPI::max_literal_fields"] + [::core::mem::offset_of!(JSJitObjectAPI, max_literal_fields) - 12usize]; + ["Offset of field: JSJitObjectAPI::literal"] + [::core::mem::offset_of!(JSJitObjectAPI, literal) - 16usize]; + ["Offset of field: JSJitObjectAPI::retain_shape"] + [::core::mem::offset_of!(JSJitObjectAPI, retain_shape) - 24usize]; + ["Offset of field: JSJitObjectAPI::array_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_method) - 32usize]; + ["Offset of field: JSJitObjectAPI::array_push"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push) - 40usize]; + ["Offset of field: JSJitObjectAPI::array_push_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push_method) - 48usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetObjectAPI(version: u32) -> *const JSJitObjectAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { diff --git a/sys/src/bindings/aarch64-pc-windows-msvc.rs b/sys/src/bindings/aarch64-pc-windows-msvc.rs index a64c09ee5..d9c084062 100644 --- a/sys/src/bindings/aarch64-pc-windows-msvc.rs +++ b/sys/src/bindings/aarch64-pc-windows-msvc.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,10 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_OBJECT_API_VERSION: u32 = 1; +pub const JS_JIT_OBJECT_LITERAL_MAX_FIELDS: u32 = 16; +pub const JS_JIT_OBJECT_EFFECT_ALLOCATE: u32 = 1; +pub const JS_JIT_OBJECT_EFFECT_COLLECT: u32 = 2; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -2275,6 +2279,87 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitObjectStatus_JS_JIT_OBJECT_INVALID: JSJitObjectStatus = -1; +pub const JSJitObjectStatus_JS_JIT_OBJECT_MISS: JSJitObjectStatus = 0; +pub const JSJitObjectStatus_JS_JIT_OBJECT_OK: JSJitObjectStatus = 1; +pub type JSJitObjectStatus = ::core::ffi::c_int; +pub type JSJitObjectLiteralFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + atoms: *const u32, + values: *const JSValue, + count: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitObjectRetainShapeFunc = ::core::option::Option< + unsafe extern "C" fn(ctx: *mut JSContext, object: *const JSValue) -> ::core::ffi::c_int, +>; +pub type JSJitArrayMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + function: *const JSValue, + receiver: *const JSValue, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitObjectAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub max_literal_fields: u32, + pub literal: JSJitObjectLiteralFunc, + pub retain_shape: JSJitObjectRetainShapeFunc, + pub array_method: JSJitArrayMethodFunc, + pub array_push: JSJitArrayPushFunc, + pub array_push_method: JSJitArrayPushMethodFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitObjectAPI"][::core::mem::size_of::() - 56usize]; + ["Alignment of JSJitObjectAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitObjectAPI::struct_size"] + [::core::mem::offset_of!(JSJitObjectAPI, struct_size) - 0usize]; + ["Offset of field: JSJitObjectAPI::version"] + [::core::mem::offset_of!(JSJitObjectAPI, version) - 4usize]; + ["Offset of field: JSJitObjectAPI::effects"] + [::core::mem::offset_of!(JSJitObjectAPI, effects) - 8usize]; + ["Offset of field: JSJitObjectAPI::max_literal_fields"] + [::core::mem::offset_of!(JSJitObjectAPI, max_literal_fields) - 12usize]; + ["Offset of field: JSJitObjectAPI::literal"] + [::core::mem::offset_of!(JSJitObjectAPI, literal) - 16usize]; + ["Offset of field: JSJitObjectAPI::retain_shape"] + [::core::mem::offset_of!(JSJitObjectAPI, retain_shape) - 24usize]; + ["Offset of field: JSJitObjectAPI::array_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_method) - 32usize]; + ["Offset of field: JSJitObjectAPI::array_push"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push) - 40usize]; + ["Offset of field: JSJitObjectAPI::array_push_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push_method) - 48usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetObjectAPI(version: u32) -> *const JSJitObjectAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { diff --git a/sys/src/bindings/aarch64-unknown-linux-gnu.rs b/sys/src/bindings/aarch64-unknown-linux-gnu.rs index a64c09ee5..d9c084062 100644 --- a/sys/src/bindings/aarch64-unknown-linux-gnu.rs +++ b/sys/src/bindings/aarch64-unknown-linux-gnu.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,10 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_OBJECT_API_VERSION: u32 = 1; +pub const JS_JIT_OBJECT_LITERAL_MAX_FIELDS: u32 = 16; +pub const JS_JIT_OBJECT_EFFECT_ALLOCATE: u32 = 1; +pub const JS_JIT_OBJECT_EFFECT_COLLECT: u32 = 2; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -2275,6 +2279,87 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitObjectStatus_JS_JIT_OBJECT_INVALID: JSJitObjectStatus = -1; +pub const JSJitObjectStatus_JS_JIT_OBJECT_MISS: JSJitObjectStatus = 0; +pub const JSJitObjectStatus_JS_JIT_OBJECT_OK: JSJitObjectStatus = 1; +pub type JSJitObjectStatus = ::core::ffi::c_int; +pub type JSJitObjectLiteralFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + atoms: *const u32, + values: *const JSValue, + count: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitObjectRetainShapeFunc = ::core::option::Option< + unsafe extern "C" fn(ctx: *mut JSContext, object: *const JSValue) -> ::core::ffi::c_int, +>; +pub type JSJitArrayMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + function: *const JSValue, + receiver: *const JSValue, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitObjectAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub max_literal_fields: u32, + pub literal: JSJitObjectLiteralFunc, + pub retain_shape: JSJitObjectRetainShapeFunc, + pub array_method: JSJitArrayMethodFunc, + pub array_push: JSJitArrayPushFunc, + pub array_push_method: JSJitArrayPushMethodFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitObjectAPI"][::core::mem::size_of::() - 56usize]; + ["Alignment of JSJitObjectAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitObjectAPI::struct_size"] + [::core::mem::offset_of!(JSJitObjectAPI, struct_size) - 0usize]; + ["Offset of field: JSJitObjectAPI::version"] + [::core::mem::offset_of!(JSJitObjectAPI, version) - 4usize]; + ["Offset of field: JSJitObjectAPI::effects"] + [::core::mem::offset_of!(JSJitObjectAPI, effects) - 8usize]; + ["Offset of field: JSJitObjectAPI::max_literal_fields"] + [::core::mem::offset_of!(JSJitObjectAPI, max_literal_fields) - 12usize]; + ["Offset of field: JSJitObjectAPI::literal"] + [::core::mem::offset_of!(JSJitObjectAPI, literal) - 16usize]; + ["Offset of field: JSJitObjectAPI::retain_shape"] + [::core::mem::offset_of!(JSJitObjectAPI, retain_shape) - 24usize]; + ["Offset of field: JSJitObjectAPI::array_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_method) - 32usize]; + ["Offset of field: JSJitObjectAPI::array_push"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push) - 40usize]; + ["Offset of field: JSJitObjectAPI::array_push_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push_method) - 48usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetObjectAPI(version: u32) -> *const JSJitObjectAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { diff --git a/sys/src/bindings/aarch64-unknown-linux-musl.rs b/sys/src/bindings/aarch64-unknown-linux-musl.rs index a64c09ee5..d9c084062 100644 --- a/sys/src/bindings/aarch64-unknown-linux-musl.rs +++ b/sys/src/bindings/aarch64-unknown-linux-musl.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,10 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_OBJECT_API_VERSION: u32 = 1; +pub const JS_JIT_OBJECT_LITERAL_MAX_FIELDS: u32 = 16; +pub const JS_JIT_OBJECT_EFFECT_ALLOCATE: u32 = 1; +pub const JS_JIT_OBJECT_EFFECT_COLLECT: u32 = 2; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -2275,6 +2279,87 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitObjectStatus_JS_JIT_OBJECT_INVALID: JSJitObjectStatus = -1; +pub const JSJitObjectStatus_JS_JIT_OBJECT_MISS: JSJitObjectStatus = 0; +pub const JSJitObjectStatus_JS_JIT_OBJECT_OK: JSJitObjectStatus = 1; +pub type JSJitObjectStatus = ::core::ffi::c_int; +pub type JSJitObjectLiteralFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + atoms: *const u32, + values: *const JSValue, + count: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitObjectRetainShapeFunc = ::core::option::Option< + unsafe extern "C" fn(ctx: *mut JSContext, object: *const JSValue) -> ::core::ffi::c_int, +>; +pub type JSJitArrayMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + function: *const JSValue, + receiver: *const JSValue, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitObjectAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub max_literal_fields: u32, + pub literal: JSJitObjectLiteralFunc, + pub retain_shape: JSJitObjectRetainShapeFunc, + pub array_method: JSJitArrayMethodFunc, + pub array_push: JSJitArrayPushFunc, + pub array_push_method: JSJitArrayPushMethodFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitObjectAPI"][::core::mem::size_of::() - 56usize]; + ["Alignment of JSJitObjectAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitObjectAPI::struct_size"] + [::core::mem::offset_of!(JSJitObjectAPI, struct_size) - 0usize]; + ["Offset of field: JSJitObjectAPI::version"] + [::core::mem::offset_of!(JSJitObjectAPI, version) - 4usize]; + ["Offset of field: JSJitObjectAPI::effects"] + [::core::mem::offset_of!(JSJitObjectAPI, effects) - 8usize]; + ["Offset of field: JSJitObjectAPI::max_literal_fields"] + [::core::mem::offset_of!(JSJitObjectAPI, max_literal_fields) - 12usize]; + ["Offset of field: JSJitObjectAPI::literal"] + [::core::mem::offset_of!(JSJitObjectAPI, literal) - 16usize]; + ["Offset of field: JSJitObjectAPI::retain_shape"] + [::core::mem::offset_of!(JSJitObjectAPI, retain_shape) - 24usize]; + ["Offset of field: JSJitObjectAPI::array_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_method) - 32usize]; + ["Offset of field: JSJitObjectAPI::array_push"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push) - 40usize]; + ["Offset of field: JSJitObjectAPI::array_push_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push_method) - 48usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetObjectAPI(version: u32) -> *const JSJitObjectAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { diff --git a/sys/src/bindings/wasm32-wasip1.rs b/sys/src/bindings/wasm32-wasip1.rs index a8167404c..fe58fe31f 100644 --- a/sys/src/bindings/wasm32-wasip1.rs +++ b/sys/src/bindings/wasm32-wasip1.rs @@ -87,7 +87,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -95,6 +95,10 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_OBJECT_API_VERSION: u32 = 1; +pub const JS_JIT_OBJECT_LITERAL_MAX_FIELDS: u32 = 16; +pub const JS_JIT_OBJECT_EFFECT_ALLOCATE: u32 = 1; +pub const JS_JIT_OBJECT_EFFECT_COLLECT: u32 = 2; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -2244,6 +2248,87 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitObjectStatus_JS_JIT_OBJECT_INVALID: JSJitObjectStatus = -1; +pub const JSJitObjectStatus_JS_JIT_OBJECT_MISS: JSJitObjectStatus = 0; +pub const JSJitObjectStatus_JS_JIT_OBJECT_OK: JSJitObjectStatus = 1; +pub type JSJitObjectStatus = ::core::ffi::c_int; +pub type JSJitObjectLiteralFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + atoms: *const u32, + values: *const JSValue, + count: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitObjectRetainShapeFunc = ::core::option::Option< + unsafe extern "C" fn(ctx: *mut JSContext, object: *const JSValue) -> ::core::ffi::c_int, +>; +pub type JSJitArrayMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + function: *const JSValue, + receiver: *const JSValue, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitObjectAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub max_literal_fields: u32, + pub literal: JSJitObjectLiteralFunc, + pub retain_shape: JSJitObjectRetainShapeFunc, + pub array_method: JSJitArrayMethodFunc, + pub array_push: JSJitArrayPushFunc, + pub array_push_method: JSJitArrayPushMethodFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitObjectAPI"][::core::mem::size_of::() - 36usize]; + ["Alignment of JSJitObjectAPI"][::core::mem::align_of::() - 4usize]; + ["Offset of field: JSJitObjectAPI::struct_size"] + [::core::mem::offset_of!(JSJitObjectAPI, struct_size) - 0usize]; + ["Offset of field: JSJitObjectAPI::version"] + [::core::mem::offset_of!(JSJitObjectAPI, version) - 4usize]; + ["Offset of field: JSJitObjectAPI::effects"] + [::core::mem::offset_of!(JSJitObjectAPI, effects) - 8usize]; + ["Offset of field: JSJitObjectAPI::max_literal_fields"] + [::core::mem::offset_of!(JSJitObjectAPI, max_literal_fields) - 12usize]; + ["Offset of field: JSJitObjectAPI::literal"] + [::core::mem::offset_of!(JSJitObjectAPI, literal) - 16usize]; + ["Offset of field: JSJitObjectAPI::retain_shape"] + [::core::mem::offset_of!(JSJitObjectAPI, retain_shape) - 20usize]; + ["Offset of field: JSJitObjectAPI::array_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_method) - 24usize]; + ["Offset of field: JSJitObjectAPI::array_push"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push) - 28usize]; + ["Offset of field: JSJitObjectAPI::array_push_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push_method) - 32usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetObjectAPI(version: u32) -> *const JSJitObjectAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { diff --git a/sys/src/bindings/x86_64-apple-darwin.rs b/sys/src/bindings/x86_64-apple-darwin.rs index a64c09ee5..d9c084062 100644 --- a/sys/src/bindings/x86_64-apple-darwin.rs +++ b/sys/src/bindings/x86_64-apple-darwin.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,10 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_OBJECT_API_VERSION: u32 = 1; +pub const JS_JIT_OBJECT_LITERAL_MAX_FIELDS: u32 = 16; +pub const JS_JIT_OBJECT_EFFECT_ALLOCATE: u32 = 1; +pub const JS_JIT_OBJECT_EFFECT_COLLECT: u32 = 2; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -2275,6 +2279,87 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitObjectStatus_JS_JIT_OBJECT_INVALID: JSJitObjectStatus = -1; +pub const JSJitObjectStatus_JS_JIT_OBJECT_MISS: JSJitObjectStatus = 0; +pub const JSJitObjectStatus_JS_JIT_OBJECT_OK: JSJitObjectStatus = 1; +pub type JSJitObjectStatus = ::core::ffi::c_int; +pub type JSJitObjectLiteralFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + atoms: *const u32, + values: *const JSValue, + count: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitObjectRetainShapeFunc = ::core::option::Option< + unsafe extern "C" fn(ctx: *mut JSContext, object: *const JSValue) -> ::core::ffi::c_int, +>; +pub type JSJitArrayMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + function: *const JSValue, + receiver: *const JSValue, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitObjectAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub max_literal_fields: u32, + pub literal: JSJitObjectLiteralFunc, + pub retain_shape: JSJitObjectRetainShapeFunc, + pub array_method: JSJitArrayMethodFunc, + pub array_push: JSJitArrayPushFunc, + pub array_push_method: JSJitArrayPushMethodFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitObjectAPI"][::core::mem::size_of::() - 56usize]; + ["Alignment of JSJitObjectAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitObjectAPI::struct_size"] + [::core::mem::offset_of!(JSJitObjectAPI, struct_size) - 0usize]; + ["Offset of field: JSJitObjectAPI::version"] + [::core::mem::offset_of!(JSJitObjectAPI, version) - 4usize]; + ["Offset of field: JSJitObjectAPI::effects"] + [::core::mem::offset_of!(JSJitObjectAPI, effects) - 8usize]; + ["Offset of field: JSJitObjectAPI::max_literal_fields"] + [::core::mem::offset_of!(JSJitObjectAPI, max_literal_fields) - 12usize]; + ["Offset of field: JSJitObjectAPI::literal"] + [::core::mem::offset_of!(JSJitObjectAPI, literal) - 16usize]; + ["Offset of field: JSJitObjectAPI::retain_shape"] + [::core::mem::offset_of!(JSJitObjectAPI, retain_shape) - 24usize]; + ["Offset of field: JSJitObjectAPI::array_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_method) - 32usize]; + ["Offset of field: JSJitObjectAPI::array_push"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push) - 40usize]; + ["Offset of field: JSJitObjectAPI::array_push_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push_method) - 48usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetObjectAPI(version: u32) -> *const JSJitObjectAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { diff --git a/sys/src/bindings/x86_64-pc-windows-gnu.rs b/sys/src/bindings/x86_64-pc-windows-gnu.rs index a64c09ee5..d9c084062 100644 --- a/sys/src/bindings/x86_64-pc-windows-gnu.rs +++ b/sys/src/bindings/x86_64-pc-windows-gnu.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,10 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_OBJECT_API_VERSION: u32 = 1; +pub const JS_JIT_OBJECT_LITERAL_MAX_FIELDS: u32 = 16; +pub const JS_JIT_OBJECT_EFFECT_ALLOCATE: u32 = 1; +pub const JS_JIT_OBJECT_EFFECT_COLLECT: u32 = 2; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -2275,6 +2279,87 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitObjectStatus_JS_JIT_OBJECT_INVALID: JSJitObjectStatus = -1; +pub const JSJitObjectStatus_JS_JIT_OBJECT_MISS: JSJitObjectStatus = 0; +pub const JSJitObjectStatus_JS_JIT_OBJECT_OK: JSJitObjectStatus = 1; +pub type JSJitObjectStatus = ::core::ffi::c_int; +pub type JSJitObjectLiteralFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + atoms: *const u32, + values: *const JSValue, + count: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitObjectRetainShapeFunc = ::core::option::Option< + unsafe extern "C" fn(ctx: *mut JSContext, object: *const JSValue) -> ::core::ffi::c_int, +>; +pub type JSJitArrayMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + function: *const JSValue, + receiver: *const JSValue, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitObjectAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub max_literal_fields: u32, + pub literal: JSJitObjectLiteralFunc, + pub retain_shape: JSJitObjectRetainShapeFunc, + pub array_method: JSJitArrayMethodFunc, + pub array_push: JSJitArrayPushFunc, + pub array_push_method: JSJitArrayPushMethodFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitObjectAPI"][::core::mem::size_of::() - 56usize]; + ["Alignment of JSJitObjectAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitObjectAPI::struct_size"] + [::core::mem::offset_of!(JSJitObjectAPI, struct_size) - 0usize]; + ["Offset of field: JSJitObjectAPI::version"] + [::core::mem::offset_of!(JSJitObjectAPI, version) - 4usize]; + ["Offset of field: JSJitObjectAPI::effects"] + [::core::mem::offset_of!(JSJitObjectAPI, effects) - 8usize]; + ["Offset of field: JSJitObjectAPI::max_literal_fields"] + [::core::mem::offset_of!(JSJitObjectAPI, max_literal_fields) - 12usize]; + ["Offset of field: JSJitObjectAPI::literal"] + [::core::mem::offset_of!(JSJitObjectAPI, literal) - 16usize]; + ["Offset of field: JSJitObjectAPI::retain_shape"] + [::core::mem::offset_of!(JSJitObjectAPI, retain_shape) - 24usize]; + ["Offset of field: JSJitObjectAPI::array_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_method) - 32usize]; + ["Offset of field: JSJitObjectAPI::array_push"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push) - 40usize]; + ["Offset of field: JSJitObjectAPI::array_push_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push_method) - 48usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetObjectAPI(version: u32) -> *const JSJitObjectAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { diff --git a/sys/src/bindings/x86_64-pc-windows-msvc.rs b/sys/src/bindings/x86_64-pc-windows-msvc.rs index a64c09ee5..d9c084062 100644 --- a/sys/src/bindings/x86_64-pc-windows-msvc.rs +++ b/sys/src/bindings/x86_64-pc-windows-msvc.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,10 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_OBJECT_API_VERSION: u32 = 1; +pub const JS_JIT_OBJECT_LITERAL_MAX_FIELDS: u32 = 16; +pub const JS_JIT_OBJECT_EFFECT_ALLOCATE: u32 = 1; +pub const JS_JIT_OBJECT_EFFECT_COLLECT: u32 = 2; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -2275,6 +2279,87 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitObjectStatus_JS_JIT_OBJECT_INVALID: JSJitObjectStatus = -1; +pub const JSJitObjectStatus_JS_JIT_OBJECT_MISS: JSJitObjectStatus = 0; +pub const JSJitObjectStatus_JS_JIT_OBJECT_OK: JSJitObjectStatus = 1; +pub type JSJitObjectStatus = ::core::ffi::c_int; +pub type JSJitObjectLiteralFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + atoms: *const u32, + values: *const JSValue, + count: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitObjectRetainShapeFunc = ::core::option::Option< + unsafe extern "C" fn(ctx: *mut JSContext, object: *const JSValue) -> ::core::ffi::c_int, +>; +pub type JSJitArrayMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + function: *const JSValue, + receiver: *const JSValue, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitObjectAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub max_literal_fields: u32, + pub literal: JSJitObjectLiteralFunc, + pub retain_shape: JSJitObjectRetainShapeFunc, + pub array_method: JSJitArrayMethodFunc, + pub array_push: JSJitArrayPushFunc, + pub array_push_method: JSJitArrayPushMethodFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitObjectAPI"][::core::mem::size_of::() - 56usize]; + ["Alignment of JSJitObjectAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitObjectAPI::struct_size"] + [::core::mem::offset_of!(JSJitObjectAPI, struct_size) - 0usize]; + ["Offset of field: JSJitObjectAPI::version"] + [::core::mem::offset_of!(JSJitObjectAPI, version) - 4usize]; + ["Offset of field: JSJitObjectAPI::effects"] + [::core::mem::offset_of!(JSJitObjectAPI, effects) - 8usize]; + ["Offset of field: JSJitObjectAPI::max_literal_fields"] + [::core::mem::offset_of!(JSJitObjectAPI, max_literal_fields) - 12usize]; + ["Offset of field: JSJitObjectAPI::literal"] + [::core::mem::offset_of!(JSJitObjectAPI, literal) - 16usize]; + ["Offset of field: JSJitObjectAPI::retain_shape"] + [::core::mem::offset_of!(JSJitObjectAPI, retain_shape) - 24usize]; + ["Offset of field: JSJitObjectAPI::array_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_method) - 32usize]; + ["Offset of field: JSJitObjectAPI::array_push"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push) - 40usize]; + ["Offset of field: JSJitObjectAPI::array_push_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push_method) - 48usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetObjectAPI(version: u32) -> *const JSJitObjectAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { diff --git a/sys/src/bindings/x86_64-unknown-linux-gnu.rs b/sys/src/bindings/x86_64-unknown-linux-gnu.rs index a64c09ee5..d9c084062 100644 --- a/sys/src/bindings/x86_64-unknown-linux-gnu.rs +++ b/sys/src/bindings/x86_64-unknown-linux-gnu.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,10 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_OBJECT_API_VERSION: u32 = 1; +pub const JS_JIT_OBJECT_LITERAL_MAX_FIELDS: u32 = 16; +pub const JS_JIT_OBJECT_EFFECT_ALLOCATE: u32 = 1; +pub const JS_JIT_OBJECT_EFFECT_COLLECT: u32 = 2; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -2275,6 +2279,87 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitObjectStatus_JS_JIT_OBJECT_INVALID: JSJitObjectStatus = -1; +pub const JSJitObjectStatus_JS_JIT_OBJECT_MISS: JSJitObjectStatus = 0; +pub const JSJitObjectStatus_JS_JIT_OBJECT_OK: JSJitObjectStatus = 1; +pub type JSJitObjectStatus = ::core::ffi::c_int; +pub type JSJitObjectLiteralFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + atoms: *const u32, + values: *const JSValue, + count: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitObjectRetainShapeFunc = ::core::option::Option< + unsafe extern "C" fn(ctx: *mut JSContext, object: *const JSValue) -> ::core::ffi::c_int, +>; +pub type JSJitArrayMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + function: *const JSValue, + receiver: *const JSValue, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitObjectAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub max_literal_fields: u32, + pub literal: JSJitObjectLiteralFunc, + pub retain_shape: JSJitObjectRetainShapeFunc, + pub array_method: JSJitArrayMethodFunc, + pub array_push: JSJitArrayPushFunc, + pub array_push_method: JSJitArrayPushMethodFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitObjectAPI"][::core::mem::size_of::() - 56usize]; + ["Alignment of JSJitObjectAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitObjectAPI::struct_size"] + [::core::mem::offset_of!(JSJitObjectAPI, struct_size) - 0usize]; + ["Offset of field: JSJitObjectAPI::version"] + [::core::mem::offset_of!(JSJitObjectAPI, version) - 4usize]; + ["Offset of field: JSJitObjectAPI::effects"] + [::core::mem::offset_of!(JSJitObjectAPI, effects) - 8usize]; + ["Offset of field: JSJitObjectAPI::max_literal_fields"] + [::core::mem::offset_of!(JSJitObjectAPI, max_literal_fields) - 12usize]; + ["Offset of field: JSJitObjectAPI::literal"] + [::core::mem::offset_of!(JSJitObjectAPI, literal) - 16usize]; + ["Offset of field: JSJitObjectAPI::retain_shape"] + [::core::mem::offset_of!(JSJitObjectAPI, retain_shape) - 24usize]; + ["Offset of field: JSJitObjectAPI::array_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_method) - 32usize]; + ["Offset of field: JSJitObjectAPI::array_push"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push) - 40usize]; + ["Offset of field: JSJitObjectAPI::array_push_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push_method) - 48usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetObjectAPI(version: u32) -> *const JSJitObjectAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { diff --git a/sys/src/bindings/x86_64-unknown-linux-musl.rs b/sys/src/bindings/x86_64-unknown-linux-musl.rs index a64c09ee5..d9c084062 100644 --- a/sys/src/bindings/x86_64-unknown-linux-musl.rs +++ b/sys/src/bindings/x86_64-unknown-linux-musl.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -94,6 +94,10 @@ pub const JS_JIT_SLOT_NONE: u32 = 4294967295; pub const JS_JIT_HELPER_SCRATCH_SLOTS: u32 = 2; pub const QJSJIT_ARRAY_API_VERSION: u32 = 1; pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1; +pub const QJSJIT_OBJECT_API_VERSION: u32 = 1; +pub const JS_JIT_OBJECT_LITERAL_MAX_FIELDS: u32 = 16; +pub const JS_JIT_OBJECT_EFFECT_ALLOCATE: u32 = 1; +pub const JS_JIT_OBJECT_EFFECT_COLLECT: u32 = 2; pub const JS_JIT_FEEDBACK_ARRAY_STORE: u32 = 64; pub const JS_JIT_FEEDBACK_ARRAY_LENGTH: u32 = 128; pub const JS_JIT_FEEDBACK_ARRAY_EXOTIC: u32 = 256; @@ -2275,6 +2279,87 @@ const _: () = { unsafe extern "C" { pub fn JS_JitGetArrayAPI(version: u32) -> *const JSJitArrayAPI; } +pub const JSJitObjectStatus_JS_JIT_OBJECT_INVALID: JSJitObjectStatus = -1; +pub const JSJitObjectStatus_JS_JIT_OBJECT_MISS: JSJitObjectStatus = 0; +pub const JSJitObjectStatus_JS_JIT_OBJECT_OK: JSJitObjectStatus = 1; +pub type JSJitObjectStatus = ::core::ffi::c_int; +pub type JSJitObjectLiteralFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + atoms: *const u32, + values: *const JSValue, + count: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitObjectRetainShapeFunc = ::core::option::Option< + unsafe extern "C" fn(ctx: *mut JSContext, object: *const JSValue) -> ::core::ffi::c_int, +>; +pub type JSJitArrayMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + function: *const JSValue, + receiver: *const JSValue, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +pub type JSJitArrayPushMethodFunc = ::core::option::Option< + unsafe extern "C" fn( + ctx: *mut JSContext, + receiver: *const JSValue, + atom: u32, + value: *const JSValue, + out: *mut JSValue, + ) -> ::core::ffi::c_int, +>; +#[repr(C)] +#[derive(Debug, Copy, Clone)] +pub struct JSJitObjectAPI { + pub struct_size: u32, + pub version: u32, + pub effects: u32, + pub max_literal_fields: u32, + pub literal: JSJitObjectLiteralFunc, + pub retain_shape: JSJitObjectRetainShapeFunc, + pub array_method: JSJitArrayMethodFunc, + pub array_push: JSJitArrayPushFunc, + pub array_push_method: JSJitArrayPushMethodFunc, +} +#[allow(clippy::unnecessary_operation, clippy::identity_op)] +const _: () = { + ["Size of JSJitObjectAPI"][::core::mem::size_of::() - 56usize]; + ["Alignment of JSJitObjectAPI"][::core::mem::align_of::() - 8usize]; + ["Offset of field: JSJitObjectAPI::struct_size"] + [::core::mem::offset_of!(JSJitObjectAPI, struct_size) - 0usize]; + ["Offset of field: JSJitObjectAPI::version"] + [::core::mem::offset_of!(JSJitObjectAPI, version) - 4usize]; + ["Offset of field: JSJitObjectAPI::effects"] + [::core::mem::offset_of!(JSJitObjectAPI, effects) - 8usize]; + ["Offset of field: JSJitObjectAPI::max_literal_fields"] + [::core::mem::offset_of!(JSJitObjectAPI, max_literal_fields) - 12usize]; + ["Offset of field: JSJitObjectAPI::literal"] + [::core::mem::offset_of!(JSJitObjectAPI, literal) - 16usize]; + ["Offset of field: JSJitObjectAPI::retain_shape"] + [::core::mem::offset_of!(JSJitObjectAPI, retain_shape) - 24usize]; + ["Offset of field: JSJitObjectAPI::array_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_method) - 32usize]; + ["Offset of field: JSJitObjectAPI::array_push"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push) - 40usize]; + ["Offset of field: JSJitObjectAPI::array_push_method"] + [::core::mem::offset_of!(JSJitObjectAPI, array_push_method) - 48usize]; +}; +unsafe extern "C" { + pub fn JS_JitGetObjectAPI(version: u32) -> *const JSJitObjectAPI; +} #[repr(C)] #[derive(Debug, Copy, Clone)] pub struct JSJitFeedbackEvent { diff --git a/sys/tests/jit_patch.rs b/sys/tests/jit_patch.rs index f36861117..3d1a06a97 100644 --- a/sys/tests/jit_patch.rs +++ b/sys/tests/jit_patch.rs @@ -56,6 +56,7 @@ fn copy_patches(destination: &std::path::Path) { "0018-inline-frame-recovery.patch", "0019-array-feedback.patch", "0020-typed-array-guard.patch", + "0032-object-fast-paths.patch", ] { fs::copy(source.join(patch), destination.join(patch)).unwrap(); } @@ -114,7 +115,7 @@ fn pinned_public_quickjs_baseline_applies_cleanly_without_git() { let helper_header = fs::read_to_string(destination.join("quickjs-jit-helpers.h")).unwrap(); assert!(quickjs.contains("JS_GetJitRuntimeId")); assert!(quickjs.contains("JS_JIT_FRAME_SIDE_PATH_HIT")); - assert!(jit_header.contains("#define QJSJIT_ABI_MINOR 24u")); + assert!(jit_header.contains("#define QJSJIT_ABI_MINOR 25u")); assert!(jit_header.contains("JSJitPropertyLayout property_layout;")); assert!(quickjs.contains("uint64_t jit_shape_generation;")); assert!(!quickjs.contains("QJSJIT_SHAPE_HASH")); @@ -154,6 +155,9 @@ fn pinned_public_quickjs_baseline_applies_cleanly_without_git() { assert!(quickjs.contains("JS_JitHelperUnaryArithSlow")); assert!(quickjs.contains("#define QJSJIT_ABI_COUNT_MAP_OUT_IN_OP 6")); assert!(jit_header.contains("JSJitFeedbackEvent")); + assert!(jit_header.contains("JS_EXTERN const JSJitObjectAPI *JS_JitGetObjectAPI")); + assert!(quickjs.contains("static int qjsjit_object_literal(")); + assert!(quickjs.contains("static int qjsjit_array_push(")); assert!(destination.join("quickjs-jit-helpers.h").is_file()); fs::remove_dir_all(destination).unwrap(); } @@ -255,7 +259,7 @@ fn bundled_jit_bindings_include_materialize_owner_tail() { for target in targets { let binding = fs::read_to_string(bindings.join(target)).unwrap(); assert!( - binding.contains("pub const QJSJIT_ABI_MINOR: u32 = 24;"), + binding.contains("pub const QJSJIT_ABI_MINOR: u32 = 25;"), "{target}" ); assert!( @@ -263,6 +267,13 @@ fn bundled_jit_bindings_include_materialize_owner_tail() { "{target}" ); assert!(binding.contains("pub struct JSJitArrayAPI"), "{target}"); + assert!( + binding.contains("pub struct JSJitObjectAPI") + && binding.contains("pub fn JS_JitGetObjectAPI") + && binding.contains("pub const QJSJIT_OBJECT_API_VERSION: u32 = 1;") + && binding.contains("size_of::() - 56usize"), + "{target}" + ); assert!( binding.contains("receiver: *const JSValue") && binding.contains("pub fn JS_JitGetArrayAPI"), @@ -363,7 +374,7 @@ fn bundled_wasm_jit_binding_matches_array_abi_1_24() { "pub const JS_JIT_FEEDBACK_ARRAY_SHARED: u32 = 8192;", "pub const JS_JIT_FEEDBACK_ARRAY_INVALID_BACKING: u32 = 16384;", // ABI 1.24: typed-array continuing guard. - "pub const QJSJIT_ABI_MINOR: u32 = 24;", + "pub const QJSJIT_ABI_MINOR: u32 = 25;", "pub const QJSJIT_ARRAY_API_VERSION: u32 = 1;", "pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1;", "JSJitArrayQueryStatus_JS_JIT_ARRAY_QUERY_INVALID: JSJitArrayQueryStatus = -1;", @@ -393,6 +404,19 @@ fn bundled_wasm_jit_binding_matches_array_abi_1_24() { "offset_of!(JSJitArrayAPI, effects) - 8usize", "offset_of!(JSJitArrayAPI, reserved) - 12usize", "offset_of!(JSJitArrayAPI, query) - 16usize", + // ABI 1.25: object allocation and array-method fast paths. + "pub const QJSJIT_OBJECT_API_VERSION: u32 = 1;", + "pub const JS_JIT_OBJECT_LITERAL_MAX_FIELDS: u32 = 16;", + "JSJitObjectStatus_JS_JIT_OBJECT_MISS: JSJitObjectStatus = 0;", + "pub struct JSJitObjectAPI", + "pub fn JS_JitGetObjectAPI(", + "size_of::() - 36usize", + "align_of::() - 4usize", + "offset_of!(JSJitObjectAPI, literal) - 16usize", + "offset_of!(JSJitObjectAPI, retain_shape) - 20usize", + "offset_of!(JSJitObjectAPI, array_method) - 24usize", + "offset_of!(JSJitObjectAPI, array_push) - 28usize", + "offset_of!(JSJitObjectAPI, array_push_method) - 32usize", ] { assert!( binding.contains(declaration), From 743605da1c35623cd483c456d849ac8cfac59ac6 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:07:21 +0800 Subject: [PATCH 18/54] perf(jit): tighten Tier 2 scalar loop code Reduce per-iteration overhead in Tier 2 loops without changing semantics: - keep the amortized poll countdown in an SSA variable instead of a stack slot; - amortize the header poll of each pure natural loop (same permits-amortized-poll proof, restricted to the loop's nodes) even when calls elsewhere rule out whole-function amortization; locals, numeric guards and revalidation still run every iteration; - share one definition per integer constant so loop-invariant tags stop becoming block parameters, then fold branches on constant conditions (after Cranelift constant-phi removal) and drop the dead exits; - forward scalar bitcast round trips on Float64 payloads; - reload frame buffers and the exit record in exit blocks instead of keeping prologue values live across loops; on x86-64 System V pass the exit record as a plain pointer and return it explicitly, which keeps the ABI contract while ending the sret live range at entry; - lower `x >>> c` with a non-zero masked constant count as Int32. RelocatableCode::clif() keeps the lowering output; a test-support-only QJSJIT_DUMP_CODE directory dumps final CLIF and machine code. Co-Authored-By: Claude Opus 5.5 (1M context) --- jit/src/compiler/baseline.rs | 633 +++++++++++++++++++++++++++++++++- jit/src/compiler/optimized.rs | 394 +++++++++++++++++---- jit/src/ir/scalar.rs | 13 +- jit/tests/optimized.rs | 56 ++- jit/tests/property_cache.rs | 36 +- jit/tests/scalar_codegen.rs | 221 ++++++++++++ 6 files changed, 1277 insertions(+), 76 deletions(-) create mode 100644 jit/tests/scalar_codegen.rs diff --git a/jit/src/compiler/baseline.rs b/jit/src/compiler/baseline.rs index 00e7cfb64..ced8b1fea 100644 --- a/jit/src/compiler/baseline.rs +++ b/jit/src/compiler/baseline.rs @@ -771,6 +771,16 @@ impl BaselineCompiler { machine_disassembly: compiled.vcode.clone(), osr_codes: Vec::new(), }; + #[cfg(feature = "test-support")] + dump_code_for_diagnostics( + if osr_start.is_some() { + "tier1-osr" + } else { + "tier1" + }, + &code.clif, + code.machine_disassembly(), + ); if osr_start.is_none() && matches!(policy, CompilePolicy::AdvertisedOnly) { for point in function.osr_points() { let Some(map) = @@ -953,13 +963,65 @@ pub(crate) fn finalize_optimized_machine( control: Option<&CompileControl>, requires_helper_stack_map: bool, ) -> Result { + finalize_optimized_machine_with_lowered_text( + isa, + clif, + control, + requires_helper_stack_map, + None, + ) +} + +/// As [`finalize_optimized_machine`]. `lowered_text` is the builder's CLIF +/// before any post-lowering cleanup; it becomes the artifact's `clif()` text so +/// lowering-structure evidence is independent of the machine-level cleanups +/// (constant canonicalization, constant-branch folding, exit +/// rematerialization) that only change register pressure and dead code. +pub(crate) fn finalize_optimized_machine_with_lowered_text( + isa: &OwnedTargetIsa, + clif: Function, + control: Option<&CompileControl>, + requires_helper_stack_map: bool, + lowered_text: Option, +) -> Result { + let clif_text = lowered_text.unwrap_or_else(|| clif.display().to_string()); if let Some(control) = control { control.check()?; - control.check_ir_bytes(clif.display().to_string().len())?; + control.check_ir_bytes(clif_text.len())?; } - let clif_text = clif.display().to_string(); + let mut clif = clif; + canonicalize_integer_constants(&mut clif); + fold_constant_branches(&mut clif); let function_parameters = clif.params.clone(); let mut context = Context::for_function(clif); + // With shared constants, loop-invariant tags and values become constant + // block parameters. Remove those first so guards on them fold to jumps + // too; `compile` below reruns the complete Cranelift pipeline. + context.compute_cfg(); + context.compute_domtree(); + context + .eliminate_unreachable_code(&**isa) + .map_err(|_| CompileFailure::InvalidArtifact)?; + context + .remove_constant_phis(&**isa) + .map_err(|_| CompileFailure::InvalidArtifact)?; + context.func.dfg.resolve_all_aliases(); + fold_bitcast_round_trips(&mut context.func); + fold_constant_branches(&mut context.func); + // Folding can remove the only helper call (for example one guarded by a + // proven check); an artifact without calls publishes no helper map. + let requires_helper_stack_map = requires_helper_stack_map + && context.func.layout.blocks().any(|block| { + context + .func + .layout + .block_insts(block) + .any(|inst| context.func.dfg.insts[inst].opcode().is_call()) + }); + #[cfg(feature = "test-support")] + let final_text = diagnostics_directory() + .is_some() + .then(|| context.func.display().to_string()); context.set_disasm(cfg!(feature = "test-support")); let compiled = context .compile(&**isa, &mut ControlPlane::default()) @@ -1020,6 +1082,14 @@ pub(crate) fn finalize_optimized_machine( } else { (Vec::new(), Vec::new()) }; + #[cfg(feature = "test-support")] + if let Some(final_text) = final_text { + dump_code_for_diagnostics( + "tier2", + &final_text, + compiled.vcode.as_deref().unwrap_or(""), + ); + } Ok(RelocatableCode { bytes, relocations, @@ -1036,6 +1106,332 @@ pub(crate) fn finalize_optimized_machine( }) } +/// Rewrites every `iconst` to one canonical definition per (type, value) at +/// the top of the entry block. +/// +/// The optimizing lowering materializes tag and payload constants where they +/// are needed, so SSA construction turns loop-invariant constants such as the +/// Int32 tag into block parameters whose incoming values are distinct `iconst` +/// instructions. Cranelift's constant-phi removal compares value identity and +/// runs before GVN, so those parameters otherwise survive into register +/// allocation and cost a register (and moves) per loop-carried slot. Constants +/// are pure and the entry block dominates every block, so sharing one +/// definition preserves semantics exactly. +pub(crate) fn canonicalize_integer_constants(function: &mut Function) { + use cranelift_codegen::cursor::{Cursor, FuncCursor}; + use cranelift_codegen::ir::{InstructionData, Opcode}; + let Some(entry) = function.layout.entry_block() else { + return; + }; + function.dfg.resolve_all_aliases(); + let mut constants = Vec::new(); + for block in function.layout.blocks() { + for inst in function.layout.block_insts(block) { + if let InstructionData::UnaryImm { + opcode: Opcode::Iconst, + imm, + } = function.dfg.insts[inst] + { + let value = function.dfg.first_result(inst); + constants.push((inst, value, function.dfg.value_type(value), imm.bits())); + } + } + } + if constants.len() < 2 { + return; + } + let mut canonical = std::collections::HashMap::new(); + let mut cursor = FuncCursor::new(function); + cursor.goto_first_insertion_point(entry); + for &(_, _, ty, bits) in &constants { + canonical + .entry((ty, bits)) + .or_insert_with(|| cursor.ins().iconst(ty, bits)); + } + let mut replacements = std::collections::HashMap::new(); + for &(inst, value, ty, bits) in &constants { + cursor.func.layout.remove_inst(inst); + replacements.insert(value, canonical[&(ty, bits)]); + } + let function = cursor.func; + let blocks = function.layout.blocks().collect::>(); + for block in blocks { + let mut next = function.layout.first_inst(block); + while let Some(inst) = next { + next = function.layout.next_inst(inst); + function.dfg.map_inst_values(inst, |value| { + replacements.get(&value).copied().unwrap_or(value) + }); + } + } +} + +/// Forwards `bitcast.T (bitcast.U x)` to `x` when `x` already has type `T`. +/// +/// The optimizing lowering keeps Float64 payloads in 64-bit integer frame +/// variables, so each arithmetic step converts its operands back to `f64` +/// and its result to `i64`. Without this rewrite every chained Float64 +/// operation pays a GPR/XMM move pair on its dependency chain. Only scalar +/// bitcasts are forwarded; a round trip through another type of the same +/// width is the identity on the bits. +pub(crate) fn fold_bitcast_round_trips(function: &mut Function) { + use cranelift_codegen::ir::{InstructionData, Opcode, ValueDef}; + let mut replacements = std::collections::HashMap::new(); + for block in function.layout.blocks() { + for inst in function.layout.block_insts(block) { + let InstructionData::LoadNoOffset { + opcode: Opcode::Bitcast, + arg, + .. + } = function.dfg.insts[inst] + else { + continue; + }; + let ValueDef::Result(source, 0) = function.dfg.value_def(arg) else { + continue; + }; + let InstructionData::LoadNoOffset { + opcode: Opcode::Bitcast, + arg: original, + .. + } = function.dfg.insts[source] + else { + continue; + }; + let result = function.dfg.first_result(inst); + let ty = function.dfg.value_type(result); + let middle = function.dfg.value_type(arg); + if ty.is_vector() || middle.is_vector() || function.dfg.value_type(original) != ty { + continue; + } + replacements.insert(result, original); + } + } + if replacements.is_empty() { + return; + } + let blocks = function.layout.blocks().collect::>(); + for block in blocks { + let mut next = function.layout.first_inst(block); + while let Some(inst) = next { + next = function.layout.next_inst(inst); + function.dfg.map_inst_values(inst, |value| { + replacements.get(&value).copied().unwrap_or(value) + }); + } + } +} + +/// Evaluates `value` when it is a compile-time integer built only from +/// constants, integer width conversions, bitwise operations and integer +/// comparisons. Results are normalized to the value's type width. +fn constant_integer_value( + function: &Function, + value: cranelift_codegen::ir::Value, + memo: &mut std::collections::HashMap>, + depth: u32, +) -> Option { + use cranelift_codegen::ir::{condcodes::IntCC, InstructionData, Opcode, ValueDef}; + if let Some(&known) = memo.get(&value) { + return known; + } + if depth > 64 { + return None; + } + let ty = function.dfg.value_type(value); + if !ty.is_int() || ty.bits() > 64 { + return None; + } + let bits = ty.bits(); + let wrap = |raw: i64| -> i64 { + if bits == 64 { + raw + } else { + let shift = 64 - bits; + (raw << shift) >> shift + } + }; + let ValueDef::Result(inst, 0) = function.dfg.value_def(value) else { + return None; + }; + let mut operand = |value| constant_integer_value(function, value, memo, depth + 1); + let compare = |cond: IntCC, lhs: i64, rhs: i64, width: u32| -> i64 { + let unsigned = |raw: i64| -> u64 { + if width == 64 { + raw as u64 + } else { + (raw as u64) & ((1u64 << width) - 1) + } + }; + let result = match cond { + IntCC::Equal => lhs == rhs, + IntCC::NotEqual => lhs != rhs, + IntCC::SignedLessThan => lhs < rhs, + IntCC::SignedGreaterThanOrEqual => lhs >= rhs, + IntCC::SignedGreaterThan => lhs > rhs, + IntCC::SignedLessThanOrEqual => lhs <= rhs, + IntCC::UnsignedLessThan => unsigned(lhs) < unsigned(rhs), + IntCC::UnsignedGreaterThanOrEqual => unsigned(lhs) >= unsigned(rhs), + IntCC::UnsignedGreaterThan => unsigned(lhs) > unsigned(rhs), + IntCC::UnsignedLessThanOrEqual => unsigned(lhs) <= unsigned(rhs), + }; + i64::from(result) + }; + let result = match function.dfg.insts[inst] { + InstructionData::UnaryImm { + opcode: Opcode::Iconst, + imm, + } => Some(wrap(imm.bits())), + InstructionData::Unary { opcode, arg } => { + let arg_ty = function.dfg.value_type(arg); + let raw = operand(arg); + match (opcode, raw) { + (Opcode::Ireduce | Opcode::Sextend, Some(raw)) => Some(wrap(raw)), + (Opcode::Uextend, Some(raw)) if arg_ty.bits() < 64 => { + Some(wrap(raw & ((1i64 << arg_ty.bits()) - 1))) + } + (Opcode::Bnot, Some(raw)) => Some(wrap(!raw)), + _ => None, + } + } + InstructionData::Binary { opcode, args } => { + match (opcode, operand(args[0]), operand(args[1])) { + (Opcode::Band, Some(lhs), Some(rhs)) => Some(wrap(lhs & rhs)), + (Opcode::Bor, Some(lhs), Some(rhs)) => Some(wrap(lhs | rhs)), + (Opcode::Bxor, Some(lhs), Some(rhs)) => Some(wrap(lhs ^ rhs)), + // An and with a known zero, or an or with a known all-ones + // operand, is constant whatever the other operand is. + (Opcode::Band, Some(0), _) | (Opcode::Band, _, Some(0)) => Some(0), + (Opcode::Bor, Some(-1), _) | (Opcode::Bor, _, Some(-1)) => Some(wrap(-1)), + _ => None, + } + } + InstructionData::BinaryImm64 { opcode, arg, imm } => { + let imm = wrap(imm.bits()); + match (opcode, operand(arg)) { + (Opcode::BandImm, Some(raw)) => Some(wrap(raw & imm)), + (Opcode::BorImm, Some(raw)) => Some(wrap(raw | imm)), + (Opcode::BxorImm, Some(raw)) => Some(wrap(raw ^ imm)), + _ => None, + } + } + InstructionData::IntCompare { cond, args, .. } => { + let width = function.dfg.value_type(args[0]).bits(); + match (operand(args[0]), operand(args[1])) { + (Some(lhs), Some(rhs)) if width <= 64 => Some(compare(cond, lhs, rhs, width)), + _ => None, + } + } + InstructionData::IntCompareImm { cond, arg, imm, .. } => { + let width = function.dfg.value_type(arg).bits(); + let rhs = if width == 64 { + imm.bits() + } else { + let shift = 64 - width; + (imm.bits() << shift) >> shift + }; + operand(arg) + .filter(|_| width <= 64) + .map(|lhs| compare(cond, lhs, rhs, width)) + } + _ => None, + }; + memo.insert(value, result); + result +} + +/// Replaces `brif` on a compile-time constant condition with a jump. +/// +/// Guards whose checks the specializer has already proven (for example a tag +/// comparison against the canonical Int32 tag constant) reach Cranelift as +/// branches on folded constants, which its mid-end does not turn into jumps. +/// The dead deopt blocks then keep values live for register allocation and +/// the hot path executes a materialize/test/branch per proven guard. The +/// untaken successor is unreachable and is removed by Cranelift's +/// unreachable-code elimination. +pub(crate) fn fold_constant_branches(function: &mut Function) { + use cranelift_codegen::ir::{InstBuilder, InstructionData, Opcode}; + let mut memo = std::collections::HashMap::new(); + let mut folds = Vec::new(); + for block in function.layout.blocks() { + let Some(inst) = function.layout.last_inst(block) else { + continue; + }; + let InstructionData::Brif { + opcode: Opcode::Brif, + arg, + blocks, + } = function.dfg.insts[inst] + else { + continue; + }; + let Some(condition) = constant_integer_value(function, arg, &mut memo, 0) else { + continue; + }; + let taken = blocks[usize::from(condition == 0)]; + let target = taken.block(&function.dfg.value_lists); + let arguments = taken.args_slice(&function.dfg.value_lists).to_vec(); + folds.push((inst, target, arguments)); + } + if folds.is_empty() { + return; + } + for (inst, target, arguments) in folds { + function.dfg.replace(inst).jump(target, &arguments); + } + // Drop the blocks the folded branches made unreachable, so callers that + // inspect the finalized IR (for example to decide whether a helper call + // remains) see exactly the code that will be emitted. + let Some(entry) = function.layout.entry_block() else { + return; + }; + let mut reachable = std::collections::HashSet::new(); + let mut worklist = vec![entry]; + while let Some(block) = worklist.pop() { + if !reachable.insert(block) { + continue; + } + for inst in function.layout.block_insts(block) { + for successor in function.dfg.insts[inst].branch_destination(&function.dfg.jump_tables) + { + worklist.push(successor.block(&function.dfg.value_lists)); + } + } + } + let unreachable = function + .layout + .blocks() + .filter(|block| !reachable.contains(block)) + .collect::>(); + for block in unreachable { + while let Some(inst) = function.layout.first_inst(block) { + function.layout.remove_inst(inst); + } + function.layout.remove_block(block); + } +} + +/// Test-support diagnostic: when `QJSJIT_DUMP_CODE` names a directory, write +/// each finalized function's Cranelift IR and machine disassembly there so +/// benchmark kernels can be inspected under production tiering. +#[cfg(feature = "test-support")] +fn diagnostics_directory() -> Option { + std::env::var_os("QJSJIT_DUMP_CODE") +} + +#[cfg(feature = "test-support")] +fn dump_code_for_diagnostics(kind: &str, clif: &str, machine: &str) { + use std::sync::atomic::{AtomicU64, Ordering}; + static SEQUENCE: AtomicU64 = AtomicU64::new(0); + let Some(directory) = diagnostics_directory() else { + return; + }; + let sequence = SEQUENCE.fetch_add(1, Ordering::Relaxed); + let path = std::path::Path::new(&directory) + .join(format!("{}-{sequence:04}-{kind}.txt", std::process::id())); + let _ = std::fs::write(path, format!("{clif}\n;; machine code\n{machine}")); +} + pub(crate) fn artifact_from_relocatable( request: CompileRequest, code: RelocatableCode, @@ -6744,3 +7140,236 @@ mod tests { ); } } + +#[cfg(test)] +mod post_lowering_tests { + use super::{canonicalize_integer_constants, fold_bitcast_round_trips, fold_constant_branches}; + use cranelift_codegen::ir::{ + condcodes::IntCC, types, AbiParam, Function, InstBuilder, Signature, + }; + use cranelift_codegen::isa::CallConv; + use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext, Variable}; + + fn verify(function: &Function) { + cranelift_codegen::verify_function( + function, + &cranelift_codegen::settings::Flags::new(cranelift_codegen::settings::builder()), + ) + .unwrap_or_else(|errors| panic!("{errors}: {}", function.display())); + } + + fn signature(params: &[types::Type], returns: &[types::Type]) -> Signature { + let mut signature = Signature::new(CallConv::SystemV); + signature + .params + .extend(params.iter().map(|&ty| AbiParam::new(ty))); + signature + .returns + .extend(returns.iter().map(|&ty| AbiParam::new(ty))); + signature + } + + /// A loop whose `tag` variable is redefined to a fresh `iconst 0` on the + /// backedge, as the lowering does for Int32 tags. + #[test] + fn loop_invariant_constants_share_one_definition() { + let mut function = Function::with_name_signature( + Default::default(), + signature(&[types::I64], &[types::I64]), + ); + let mut context = FunctionBuilderContext::new(); + let mut builder = FunctionBuilder::new(&mut function, &mut context); + let entry = builder.create_block(); + let header = builder.create_block(); + let body = builder.create_block(); + let exit = builder.create_block(); + builder.append_block_params_for_function_params(entry); + let counter = Variable::from_u32(0); + let tag = Variable::from_u32(1); + builder.declare_var(counter, types::I64); + builder.declare_var(tag, types::I64); + builder.switch_to_block(entry); + let x = builder.block_params(entry)[0]; + builder.def_var(counter, x); + let zero = builder.ins().iconst(types::I64, 0); + builder.def_var(tag, zero); + builder.ins().jump(header, &[]); + builder.switch_to_block(header); + let current = builder.use_var(counter); + let done = builder.ins().icmp_imm(IntCC::Equal, current, 0); + builder.ins().brif(done, exit, &[], body, &[]); + builder.switch_to_block(body); + let next = builder.ins().iadd_imm(current, -1); + builder.def_var(counter, next); + let again = builder.ins().iconst(types::I64, 0); + builder.def_var(tag, again); + builder.ins().jump(header, &[]); + builder.switch_to_block(exit); + let result = builder.use_var(tag); + builder.ins().return_(&[result]); + builder.seal_all_blocks(); + builder.finalize(); + + canonicalize_integer_constants(&mut function); + verify(&function); + let text = function.display().to_string(); + assert_eq!(text.matches("iconst.i64 0").count(), 1, "{text}"); + // Every edge into the header now passes the same zero, which is what + // lets Cranelift's constant-phi removal delete the tag parameter. + let jumps = text + .lines() + .filter(|line| line.trim_start().starts_with("jump block1")) + .collect::>(); + assert_eq!(jumps.len(), 2, "{text}"); + let zero = text + .lines() + .find_map(|line| { + let line = line.trim(); + line.ends_with("iconst.i64 0") + .then(|| line.split_whitespace().next().unwrap().to_owned()) + }) + .unwrap(); + assert!( + jumps.iter().all(|jump| jump.contains(&format!("{zero})"))), + "{text}" + ); + } + + #[test] + fn constant_conditions_become_jumps_and_drop_dead_exits() { + let mut function = Function::with_name_signature( + Default::default(), + signature(&[types::I64], &[types::I64]), + ); + let mut context = FunctionBuilderContext::new(); + let mut builder = FunctionBuilder::new(&mut function, &mut context); + let entry = builder.create_block(); + let pass = builder.create_block(); + let deopt = builder.create_block(); + builder.append_block_params_for_function_params(entry); + builder.append_block_param(pass, types::I64); + builder.switch_to_block(entry); + let x = builder.block_params(entry)[0]; + let tag = builder.ins().iconst(types::I64, 0); + let int = builder.ins().icmp_imm(IntCC::Equal, tag, 0); + let float = builder.ins().icmp_imm(IntCC::Equal, tag, 8); + let one = builder.ins().iconst(types::I8, 1); + let numeric = builder.ins().bor(int, float); + let numeric = builder.ins().band(one, numeric); + builder.ins().brif(numeric, pass, &[x], deopt, &[]); + builder.switch_to_block(pass); + let value = builder.block_params(pass)[0]; + builder.ins().return_(&[value]); + builder.switch_to_block(deopt); + let marker = builder.ins().iconst(types::I64, -1); + builder.ins().return_(&[marker]); + builder.seal_all_blocks(); + builder.finalize(); + + canonicalize_integer_constants(&mut function); + fold_constant_branches(&mut function); + verify(&function); + let folded = function.display().to_string(); + assert!(!folded.contains("brif"), "{folded}"); + assert!(folded.contains("jump block1(v0)"), "{folded}"); + assert!(!folded.contains("block2"), "dead exit removed: {folded}"); + } + + #[test] + fn runtime_conditions_are_left_alone() { + let mut function = Function::with_name_signature( + Default::default(), + signature(&[types::I64], &[types::I64]), + ); + let mut context = FunctionBuilderContext::new(); + let mut builder = FunctionBuilder::new(&mut function, &mut context); + let entry = builder.create_block(); + let left = builder.create_block(); + let right = builder.create_block(); + builder.append_block_params_for_function_params(entry); + builder.switch_to_block(entry); + let x = builder.block_params(entry)[0]; + let small = builder.ins().icmp_imm(IntCC::UnsignedLessThan, x, 8); + let zero = builder.ins().iconst(types::I8, 0); + // `small | 0` still depends on x; `x & 0` would not. + let condition = builder.ins().bor(small, zero); + builder.ins().brif(condition, left, &[], right, &[]); + builder.switch_to_block(left); + builder.ins().return_(&[x]); + builder.switch_to_block(right); + let zero = builder.ins().iconst(types::I64, 0); + builder.ins().return_(&[zero]); + builder.seal_all_blocks(); + builder.finalize(); + fold_constant_branches(&mut function); + verify(&function); + assert!(function.display().to_string().contains("brif")); + } + + #[test] + fn narrow_comparisons_fold_with_their_type_width() { + let mut function = + Function::with_name_signature(Default::default(), signature(&[], &[types::I32])); + let mut context = FunctionBuilderContext::new(); + let mut builder = FunctionBuilder::new(&mut function, &mut context); + let entry = builder.create_block(); + let taken = builder.create_block(); + let other = builder.create_block(); + builder.switch_to_block(entry); + // -1 as i32 is below zero signed but the largest value unsigned, and + // its zero extension is positive. + let minus_one = builder.ins().iconst(types::I32, -1); + let wide = builder.ins().uextend(types::I64, minus_one); + let unsigned_big = builder + .ins() + .icmp_imm(IntCC::UnsignedGreaterThan, minus_one, 7); + let signed_negative = builder.ins().icmp_imm(IntCC::SignedLessThan, minus_one, 0); + let widened_positive = builder.ins().icmp_imm(IntCC::SignedGreaterThan, wide, 0); + let both = builder.ins().band(unsigned_big, widened_positive); + let all = builder.ins().band(both, signed_negative); + builder.ins().brif(all, taken, &[], other, &[]); + builder.switch_to_block(taken); + let one = builder.ins().iconst(types::I32, 1); + builder.ins().return_(&[one]); + builder.switch_to_block(other); + let two = builder.ins().iconst(types::I32, 2); + builder.ins().return_(&[two]); + builder.seal_all_blocks(); + builder.finalize(); + fold_constant_branches(&mut function); + verify(&function); + let folded = function.display().to_string(); + assert!(folded.contains("jump block1"), "{folded}"); + assert!(!folded.contains("block2"), "{folded}"); + } + + #[test] + fn float_payload_round_trips_forward_the_original_value() { + use cranelift_codegen::ir::MemFlags; + let mut function = Function::with_name_signature( + Default::default(), + signature(&[types::F64, types::I64], &[types::F64, types::I64]), + ); + let mut context = FunctionBuilderContext::new(); + let mut builder = FunctionBuilder::new(&mut function, &mut context); + let entry = builder.create_block(); + builder.append_block_params_for_function_params(entry); + builder.switch_to_block(entry); + let float = builder.block_params(entry)[0]; + let bits = builder.block_params(entry)[1]; + let payload = builder.ins().bitcast(types::I64, MemFlags::new(), float); + let back = builder.ins().bitcast(types::F64, MemFlags::new(), payload); + let sum = builder.ins().fadd(back, back); + // An integer payload viewed as f64 and back is also the identity. + let viewed = builder.ins().bitcast(types::F64, MemFlags::new(), bits); + let raw = builder.ins().bitcast(types::I64, MemFlags::new(), viewed); + builder.ins().return_(&[sum, raw]); + builder.seal_all_blocks(); + builder.finalize(); + fold_bitcast_round_trips(&mut function); + verify(&function); + let text = function.display().to_string(); + assert!(text.contains("fadd v0, v0"), "{text}"); + assert!(text.contains("return v4, v1"), "{text}"); + } +} diff --git a/jit/src/compiler/optimized.rs b/jit/src/compiler/optimized.rs index afc40c273..d617d5184 100644 --- a/jit/src/compiler/optimized.rs +++ b/jit/src/compiler/optimized.rs @@ -1612,47 +1612,67 @@ fn lower_optimized_machine( // Array feedback only proposes candidates. Poll amortization is enabled // after the concrete plan has proved that every heap site on the native // path is a guarded leaf-or-exit operation. + let guarded_heap = |id: u32| { + if array_plan.guarded_leaf(id) || property_plan.access(id).is_some() { + return true; + } + let Some(node) = ir.nodes().get(id as usize) else { + return false; + }; + let crate::ir::OptimizedNodeKind::Bytecode { opcode } = node.kind() else { + return false; + }; + if matches!(opcode.as_ref(), "get_field" | "put_field") + && specialization.properties.contains_key(&node.pc()) + { + return true; + } + (matches!( + opcode.as_ref(), + "add" | "sub" | "mul" | "div" | "or" | "and" | "xor" | "shl" | "sar" | "shr" + ) && ir.scalar_graph().binary_operation(id).is_some()) + || (matches!( + opcode.as_ref(), + "or" | "and" | "xor" | "shl" | "sar" | "shr" + ) && ir.scalar_graph().bitwise_operation(id).is_some()) + || (matches!(opcode.as_ref(), "lt" | "lte" | "gt" | "gte") + && ir.scalar_graph().comparison(id).is_some()) + }; let amortized_poll = int32_loop || (side_path.is_none() && ir.scalar_graph().permits_amortized_poll_with_guarded_heap( ir.nodes(), &scalar_numeric, - |id| { - if array_plan.guarded_leaf(id) || property_plan.access(id).is_some() { - return true; - } - let Some(node) = ir.nodes().get(id as usize) else { - return false; - }; - let crate::ir::OptimizedNodeKind::Bytecode { opcode } = node.kind() else { - return false; - }; - if matches!(opcode.as_ref(), "get_field" | "put_field") - && specialization.properties.contains_key(&node.pc()) - { - return true; - } - (matches!( - opcode.as_ref(), - "add" - | "sub" - | "mul" - | "div" - | "or" - | "and" - | "xor" - | "shl" - | "sar" - | "shr" - ) && ir.scalar_graph().binary_operation(id).is_some()) - || (matches!( - opcode.as_ref(), - "or" | "and" | "xor" | "shl" | "sar" | "shr" - ) && ir.scalar_graph().bitwise_operation(id).is_some()) - || (matches!(opcode.as_ref(), "lt" | "lte" | "gt" | "gte") - && ir.scalar_graph().comparison(id).is_some()) - }, + guarded_heap, )); + // A function whose other code (for example a call before or after the + // loops) rules out whole-function amortization can still count down the + // poll of an individual loop whose own nodes pass the same proof. Unlike + // the whole-function mode, frame locals keep their immediate stores and + // the header's numeric guard, hoisted-call and property revalidation all + // still run on every iteration; only the runtime poll call is amortized, + // exactly as it already is for raw Int32 loops. + let amortized_loop_headers = + if amortized_poll || side_path.is_some() || !array_loops.is_complete() { + std::collections::BTreeSet::new() + } else { + array_loops + .loops() + .iter() + .filter(|loop_| { + ir.scalar_graph().permits_amortized_poll_for( + ir.blocks() + .iter() + .filter(|block| loop_.contains_block(block.start_pc())) + .flat_map(|block| block.nodes().iter()) + .filter_map(|&id| ir.nodes().get(id as usize)), + &scalar_numeric, + guarded_heap, + ) + }) + .map(|loop_| loop_.header()) + .collect::>() + }; let loop_forwarded_property_sites = ir .nodes() .iter() @@ -1722,6 +1742,7 @@ fn lower_optimized_machine( signature.params.push(AbiParam::new(pointer_type)); let mut clif = Function::with_name_signature(Default::default(), signature); let mut context = FunctionBuilderContext::new(); + let frame_buffers; { let mut builder = FunctionBuilder::new(&mut clif, &mut context); let generated_signatures = super::helpers::generated_signatures(&**isa)?; @@ -1755,6 +1776,15 @@ fn lower_optimized_machine( let stack_base = builder .ins() .load(pointer_type, flags, frame, layout.stack_base); + frame_buffers = FrameBufferLoads { + sret, + frame, + loads: [ + (arg_buf, layout.arg_buf), + (var_buf, layout.var_buf), + (stack_base, layout.stack_base), + ], + }; let mut next_var = 0u32; let mut alloc = || { let pair = OptVars { @@ -1824,16 +1854,13 @@ fn lower_optimized_machine( builder.declare_var(vars.payload, payload_type); builder.declare_var(vars.tag, types::I64); } - let poll_budget = - builder.create_sized_stack_slot(cranelift_codegen::ir::StackSlotData::new( - cranelift_codegen::ir::StackSlotKind::ExplicitSlot, - 8, - 3, - )); - let initial_poll_budget = builder.ins().iconst(types::I64, 64); - builder - .ins() - .stack_store(initial_poll_budget, poll_budget, 0); + // The amortized poll countdown is an SSA variable so the register + // allocator can keep it in a register across the loop instead of a + // load/decrement/store through a stack slot on every iteration. + let poll_budget = Variable::from_u32(next_var); + builder.declare_var(poll_budget, types::I32); + let initial_poll_budget = builder.ins().iconst(types::I32, 64); + builder.def_var(poll_budget, initial_poll_budget); for (index, vars) in arguments.iter().enumerate() { let mut pair = opt_load(&mut builder, arg_buf, index); if int32_loop { @@ -2106,15 +2133,28 @@ fn lower_optimized_machine( }, )?; } else if *mid_loop { - emit_opt_poll( - &mut builder, - frame, - sret, - poll_signature, - pointer_type, - layout, - node.pc(), - ); + if amortized_loop_headers.contains(&block.start_pc()) { + emit_opt_countdown_poll( + &mut builder, + frame, + sret, + poll_signature, + pointer_type, + layout, + node.pc(), + poll_budget, + ); + } else { + emit_opt_poll( + &mut builder, + frame, + sret, + poll_signature, + pointer_type, + layout, + node.pc(), + ); + } let pass = builder.create_block(); emit_opt_numeric_guard( &mut builder, @@ -3784,6 +3824,18 @@ fn lower_optimized_machine( builder.seal_all_blocks(); builder.finalize(); } + let lowered_text = clif.display().to_string(); + if ir.blocks().iter().any(|block| block.is_loop_header()) { + let explicit_sret_return = isa.triple().architecture + == target_lexicon::Architecture::X86_64 + && isa.default_call_conv() == cranelift_codegen::isa::CallConv::SystemV; + rematerialize_invariants_in_exits( + &mut clif, + &frame_buffers, + pointer_type, + explicit_sret_return, + ); + } // Helpers that validate a stack map (CALL, GET_GLOBAL, GET_PROPERTY, ...) // are always invoked with map 0, so an artifact that calls anything must // publish the single helper stack map the runtime checks that id against. @@ -3792,7 +3844,164 @@ fn lower_optimized_machine( .block_insts(block) .any(|inst| clif.dfg.insts[inst].opcode().is_call()) }); - super::baseline::finalize_optimized_machine(isa, clif, control, calls_helpers) + super::baseline::finalize_optimized_machine_with_lowered_text( + isa, + clif, + control, + calls_helpers, + Some(lowered_text), + ) +} + +/// The activation's invariant pointers: the `sret` exit record, the root +/// `JSJitExecFrame`, and its buffer pointers loaded once in the prologue. +struct FrameBufferLoads { + sret: cranelift_codegen::ir::Value, + frame: cranelift_codegen::ir::Value, + loads: [(cranelift_codegen::ir::Value, i32); 3], +} + +/// Keeps activation-invariant pointers out of loop register pressure. +/// +/// Exit blocks (`return`: deopt, interrupt and normal exits) reload the root +/// frame's `arg_buf`/`var_buf`/`stack_base` from the frame and `sret` from a +/// prologue stash slot, instead of keeping the prologue values live across +/// every loop that can exit. +/// +/// All four values are fixed for the whole activation: `sret` is an entry +/// parameter, `JSJitExecFrame` buffers are initialized before entry and no +/// helper rewrites them (the lowering itself already reuses the prologue +/// values after arbitrary helper calls). A reload therefore yields exactly the +/// prologue value. Exits run at most once per activation; only functions with +/// a loop are rewritten, since the stash costs one store per entry. +/// +/// Exit blocks are outside every loop, so Cranelift cannot hoist the reload +/// back into a loop preheader. The stash is read through a distinct +/// `stack_addr` (slot offset 8, load offset -8) so GVN cannot merge it with +/// the prologue store's address and keep that address live instead. +/// +/// With `explicit_sret_return` (x86-64 System V only), the exit record is +/// additionally passed as an ordinary pointer argument and every exit returns +/// the reloaded pointer explicitly. The machine-level contract is unchanged: +/// the caller's hidden pointer still arrives in the first integer argument +/// register and is returned in `rax`, as the ABI requires for +/// `JSJitExit (*)(JSJitExecFrame *)`. Cranelift's `StructReturn` purpose +/// would instead keep the incoming parameter live until every return. +fn rematerialize_invariants_in_exits( + function: &mut cranelift_codegen::ir::Function, + invariants: &FrameBufferLoads, + pointer_type: cranelift_codegen::ir::Type, + explicit_sret_return: bool, +) { + use cranelift_codegen::cursor::{Cursor, FuncCursor}; + use cranelift_codegen::ir::{ + AbiParam, ArgumentPurpose, InstBuilder, MemFlags, Opcode, StackSlotData, StackSlotKind, + }; + let Some(entry) = function.layout.entry_block() else { + return; + }; + let explicit_sret_return = explicit_sret_return + && function.signature.returns.is_empty() + && function + .signature + .params + .first() + .is_some_and(|param| param.purpose == ArgumentPurpose::StructReturn) + && function.dfg.block_params(entry).first() == Some(&invariants.sret) + && function.layout.blocks().all(|block| { + function.layout.block_insts(block).all(|inst| { + function.dfg.insts[inst].opcode() != Opcode::Return + || function.dfg.inst_args(inst).is_empty() + }) + }); + function.dfg.resolve_all_aliases(); + let exits = function + .layout + .blocks() + .filter(|&block| { + block != entry + && function + .layout + .last_inst(block) + .is_some_and(|inst| function.dfg.insts[inst].opcode() == Opcode::Return) + }) + .collect::>(); + let pointer_bytes = pointer_type.bytes(); + let Ok(stash_offset) = i32::try_from(pointer_bytes) else { + return; + }; + let mut sret_slot = None; + for block in exits { + let insts = function.layout.block_insts(block).collect::>(); + let uses = |value| { + insts + .iter() + .any(|&inst| function.dfg.inst_values(inst).any(|arg| arg == value)) + }; + let buffers = invariants.loads.map(|(value, _)| uses(value)); + let sret_used = explicit_sret_return || uses(invariants.sret); + if !sret_used && !buffers.iter().any(|&used| used) { + continue; + } + let slot = if sret_used { + Some(*sret_slot.get_or_insert_with(|| { + let align = u8::try_from(pointer_bytes.trailing_zeros()).unwrap_or(3); + let slot = function.create_sized_stack_slot(StackSlotData::new( + StackSlotKind::ExplicitSlot, + pointer_bytes * 2, + align, + )); + let mut cursor = FuncCursor::new(function); + cursor.goto_first_insertion_point(entry); + cursor.ins().stack_store(invariants.sret, slot, 0); + slot + })) + } else { + None + }; + let mut cursor = FuncCursor::new(function); + cursor.goto_first_insertion_point(block); + let sret = slot.map(|slot| { + let address = cursor.ins().stack_addr(pointer_type, slot, stash_offset); + cursor + .ins() + .load(pointer_type, MemFlags::trusted(), address, -stash_offset) + }); + let mut replacements = vec![(invariants.sret, sret)]; + for (index, &(original, offset)) in invariants.loads.iter().enumerate() { + let reload = buffers[index].then(|| { + cursor + .ins() + .load(pointer_type, MemFlags::new(), invariants.frame, offset) + }); + replacements.push((original, reload)); + } + for &inst in &insts { + function.dfg.map_inst_values(inst, |value| { + replacements + .iter() + .find_map(|&(original, replacement)| { + (original == value).then_some(replacement).flatten() + }) + .unwrap_or(value) + }); + } + if let (true, Some(sret), Some(&last)) = (explicit_sret_return, sret, insts.last()) { + function.dfg.replace(last).return_(&[sret]); + } + } + if explicit_sret_return { + let entry_returns = function + .layout + .block_insts(entry) + .filter(|&inst| function.dfg.insts[inst].opcode() == Opcode::Return) + .collect::>(); + for inst in entry_returns { + function.dfg.replace(inst).return_(&[invariants.sret]); + } + function.signature.params[0] = AbiParam::new(pointer_type); + function.signature.returns.push(AbiParam::new(pointer_type)); + } } /// Emit an admitted effect-free region. Every failure reconstructs the @@ -5337,7 +5546,12 @@ fn emit_opt_guarded_int_binary( } } }; - let result = if operation == ScalarBitwiseOp::Shr { + // `x >>> c` with a constant count whose low five bits are non-zero is + // below 2^31, so it is always an Int32 and needs neither a guard nor a + // Float64 alternative. + let shr_always_int32 = operation == ScalarBitwiseOp::Shr + && opt_constant(builder, ri).is_some_and(|count| count & 31 != 0); + let result = if operation == ScalarBitwiseOp::Shr && !shr_always_int32 { let fits_int32 = builder .ins() .icmp_imm(IntCC::SignedGreaterThanOrEqual, value, 0); @@ -5368,6 +5582,32 @@ fn emit_opt_guarded_int_binary( Ok(output + 1) } +/// The integer constant `value` is defined as, looking through the integer +/// width conversions the lowering inserts between payloads and operands. +fn opt_constant( + builder: &cranelift_frontend::FunctionBuilder<'_>, + value: cranelift_codegen::ir::Value, +) -> Option { + use cranelift_codegen::ir::{InstructionData, Opcode, ValueDef}; + let mut value = builder.func.dfg.resolve_aliases(value); + loop { + let ValueDef::Result(inst, 0) = builder.func.dfg.value_def(value) else { + return None; + }; + match builder.func.dfg.insts[inst] { + InstructionData::UnaryImm { + opcode: Opcode::Iconst, + imm, + } => return Some(imm.bits()), + InstructionData::Unary { + opcode: Opcode::Ireduce | Opcode::Sextend | Opcode::Uextend, + arg, + } => value = builder.func.dfg.resolve_aliases(arg), + _ => return None, + } + } +} + #[allow(clippy::too_many_arguments)] fn emit_opt_packed_metadata_guard( builder: &mut cranelift_frontend::FunctionBuilder<'_>, @@ -8680,6 +8920,38 @@ fn emit_opt_poll( builder.switch_to_block(continuation); } +/// Calls the runtime poll once every 64 executions of this point. The frame +/// is already synchronized here (callers use it only where the regular poll +/// would run), so only the call frequency changes. The poll block keeps its +/// layout position: moving calls changes which call return publishes the +/// artifact's helper stack map. +#[allow(clippy::too_many_arguments)] // Poll ABI parameters mirror the generated helper signature. +fn emit_opt_countdown_poll( + builder: &mut cranelift_frontend::FunctionBuilder<'_>, + frame: cranelift_codegen::ir::Value, + sret: cranelift_codegen::ir::Value, + signature: cranelift_codegen::ir::SigRef, + pointer_type: cranelift_codegen::ir::Type, + layout: super::helpers::FrameLayout, + pc: u32, + budget: cranelift_frontend::Variable, +) { + use cranelift_codegen::ir::{condcodes::IntCC, types, InstBuilder}; + let remaining = builder.use_var(budget); + let remaining = builder.ins().iadd_imm(remaining, -1); + builder.def_var(budget, remaining); + let due = builder.ins().icmp_imm(IntCC::Equal, remaining, 0); + let poll = builder.create_block(); + let continuation = builder.create_block(); + builder.ins().brif(due, poll, &[], continuation, &[]); + builder.switch_to_block(poll); + let reset = builder.ins().iconst(types::I32, 64); + builder.def_var(budget, reset); + emit_opt_poll(builder, frame, sret, signature, pointer_type, layout, pc); + builder.ins().jump(continuation, &[]); + builder.switch_to_block(continuation); +} + #[allow(clippy::too_many_arguments)] // Poll ABI parameters mirror the generated helper signature. fn emit_opt_amortized_poll( builder: &mut cranelift_frontend::FunctionBuilder<'_>, @@ -8689,15 +8961,15 @@ fn emit_opt_amortized_poll( pointer_type: cranelift_codegen::ir::Type, layout: super::helpers::FrameLayout, pc: u32, - budget: cranelift_codegen::ir::StackSlot, + budget: cranelift_frontend::Variable, cold_poll: bool, before_poll: impl FnOnce(&mut cranelift_frontend::FunctionBuilder<'_>), after_poll: impl FnOnce(&mut cranelift_frontend::FunctionBuilder<'_>) -> Result<(), CompileFailure>, ) -> Result<(), CompileFailure> { use cranelift_codegen::ir::{condcodes::IntCC, types, InstBuilder}; - let remaining = builder.ins().stack_load(types::I64, budget, 0); + let remaining = builder.use_var(budget); let remaining = builder.ins().iadd_imm(remaining, -1); - builder.ins().stack_store(remaining, budget, 0); + builder.def_var(budget, remaining); let due = builder.ins().icmp_imm(IntCC::Equal, remaining, 0); let poll = builder.create_block(); // Preserve the established raw-i32 loop layout. The new mixed scalar @@ -8711,8 +8983,8 @@ fn emit_opt_amortized_poll( before_poll(builder); emit_opt_poll(builder, frame, sret, signature, pointer_type, layout, pc); after_poll(builder)?; - let reset = builder.ins().iconst(types::I64, 64); - builder.ins().stack_store(reset, budget, 0); + let reset = builder.ins().iconst(types::I32, 64); + builder.def_var(budget, reset); builder.ins().jump(continuation, &[]); builder.switch_to_block(continuation); Ok(()) diff --git a/jit/src/ir/scalar.rs b/jit/src/ir/scalar.rs index 83532ffac..e663b1442 100644 --- a/jit/src/ir/scalar.rs +++ b/jit/src/ir/scalar.rs @@ -1027,7 +1027,18 @@ impl ScalarGraph { numeric: &[Option], guarded_heap: impl Fn(u32) -> bool, ) -> bool { - nodes.iter().all(|node| { + self.permits_amortized_poll_for(nodes.iter(), numeric, guarded_heap) + } + + /// As [`Self::permits_amortized_poll_with_guarded_heap`], restricted to + /// the given nodes (for example the members of one natural loop). + pub(crate) fn permits_amortized_poll_for<'a>( + &self, + mut nodes: impl Iterator, + numeric: &[Option], + guarded_heap: impl Fn(u32) -> bool, + ) -> bool { + nodes.all(|node| { // Legacy optimized effects classify several property/element // opcodes as FrameWrite because their result replaces operands. // A concrete guarded leaf-or-exit lowering is nevertheless a diff --git a/jit/tests/optimized.rs b/jit/tests/optimized.rs index 3b8d4beba..9f72098e4 100644 --- a/jit/tests/optimized.rs +++ b/jit/tests/optimized.rs @@ -3211,7 +3211,24 @@ fn guard_specific_float_side_path_changes_machine_guard_and_preserves_profile() generic, specialized, "side path must alter the selected guard block" ); - assert!(specialized.matches("brif").count() > generic.matches("brif").count()); + // Proven checks fold away in both versions, so count the side path's + // own evidence: the guard block that records a side-path hit in the + // frame flags before continuing on the alternate representation. + let side_path_hits = |clif: &str| { + clif.lines() + .filter(|line| { + line.contains("bor_imm") + && line.trim_end().ends_with(&format!( + ", {}", + rquickjs_core::qjs::JS_JIT_FRAME_SIDE_PATH_HIT + )) + }) + .count() + }; + assert!( + side_path_hits(&specialized) > side_path_hits(&generic), + "{specialized}" + ); } // --------------------------------------------------------------------------- @@ -3265,16 +3282,40 @@ fn bitops_loop_lowers_to_native_shift_and_xor_instructions() { 0, 0, ); - for operation in ["ishl", "ushr", "bxor", "fcvt_from_uint"] { + for operation in ["ishl", "ushr", "bxor"] { assert!( straight.contains(operation), "missing {operation}: {straight}" ); } + // A constant count with non-zero low five bits keeps `>>>` below 2^31, + // so the result is an Int32 without a Float64 alternative. + assert!( + !straight.contains("fcvt_from_uint"), + "constant unsigned shifts are always Int32: {straight}" + ); assert!( !straight.contains("call_indirect"), "bit operations must not call a helper: {straight}" ); + // A count whose low five bits are zero shifts by nothing: `-1 >>> 32` + // is 4294967295 and must keep its Float64 representation. + let unmasked = raw_clif( + vec![ + rquickjs_core::qjs::QJS_JIT_OP_PUSH_I8, + 255, + rquickjs_core::qjs::QJS_JIT_OP_PUSH_I8, + 32, + rquickjs_core::qjs::QJS_JIT_OP_SHR, + rquickjs_core::qjs::QJS_JIT_OP_RETURN, + ], + 0, + 0, + ); + assert!( + unmasked.contains("fcvt_from_uint"), + "a zero effective count may produce a Float64: {unmasked}" + ); } #[test] @@ -4115,6 +4156,17 @@ mod m2_core_opcodes { ]), float(4_294_967_295.0) ); + // A constant count of 32 masks to zero: still a Float64 result. + assert_eq!( + value(&[ + qjs::QJS_JIT_OP_PUSH_MINUS1, + opcode::PUSH_I8, + 32, + qjs::QJS_JIT_OP_SHR, + opcode::RETURN + ]), + float(4_294_967_295.0) + ); assert_eq!( value(&[ qjs::QJS_JIT_OP_PUSH_MINUS1, diff --git a/jit/tests/property_cache.rs b/jit/tests/property_cache.rs index ead895832..5af4eb043 100644 --- a/jit/tests/property_cache.rs +++ b/jit/tests/property_cache.rs @@ -397,16 +397,29 @@ fn guarded_property_loop_uses_a_cold_amortized_poll() { line.contains("iadd_imm") && line.ends_with(", -1") }) .expect("guarded scalar/property leaves must have a poll countdown"); - let load = lines[decrement - 1].trim(); - let store = lines[decrement + 1].trim(); - let slot = load - .split_whitespace() - .last() - .filter(|slot| slot.starts_with("ss")) - .expect("countdown must load a function-local stack slot"); + // The countdown is an SSA value carried by a loop-header block + // parameter, so it stays in a register instead of a stack-slot + // load/decrement/store on every iteration. + let operand = lines[decrement] + .split(" ;") + .next() + .and_then(|line| line.split("iadd_imm").nth(1)) + .and_then(|rest| rest.split(',').next()) + .map(str::trim) + .expect("countdown decrement operand"); + let header = lines[..decrement] + .iter() + .rev() + .find(|line| line.trim_start().starts_with("block")) + .expect("countdown decrement inside a block"); + assert!( + header.contains(&format!("{operand}: i32")), + "countdown must be a loop-header block parameter, not memory: {header}; {}", + lines[decrement] + ); assert!( - load.contains("stack_load.i64") && store.contains("stack_store") && store.ends_with(slot), - "countdown decrement must be bracketed by an explicit stack-slot load/store: {load}; {store}" + !clif.contains("stack_load") && !clif.contains("stack_store"), + "countdown must not round-trip through a stack slot: {clif}" ); assert_eq!( @@ -426,7 +439,10 @@ fn continuing_property_loop_check_rejects_an_injected_field_access() { assert_eq!(continuing_property_loop_is_field_free(&clif), Ok(())); let countdown = clif .lines() - .find(|line| line.contains("stack_load.i64")) + .find(|line| { + let line = line.split(" ;").next().unwrap_or(line).trim(); + line.contains("iadd_imm") && line.ends_with(", -1") + }) .expect("fixture must contain the loop countdown"); let mutated = clif.replacen( countdown, diff --git a/jit/tests/scalar_codegen.rs b/jit/tests/scalar_codegen.rs new file mode 100644 index 000000000..664db4f77 --- /dev/null +++ b/jit/tests/scalar_codegen.rs @@ -0,0 +1,221 @@ +//! Scalar-loop code quality: per-loop poll amortization, exit +//! rematerialization, constant-branch folding and constant unsigned shifts. +//! Generated-code assertions establish the shape; runtime tests establish +//! that the cheaper code keeps exact interpreter semantics, deopts and +//! interrupt service. +#![cfg(all( + feature = "compiler", + feature = "test-support", + target_endian = "little", + any(target_os = "linux", target_os = "macos"), + any(target_arch = "x86_64", target_arch = "aarch64") +))] + +use rquickjs::{Context, Function, Runtime}; +use rquickjs_jit::{ + bytecode::VerifyLimits, compiler::optimized::Tier2Compiler, test_support::SnapshotFixture, Jit, + JitConfig, +}; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Arc, +}; +use std::time::{Duration, Instant}; + +/// Calls before and after the loops rule out whole-function poll +/// amortization; the loop nest itself is pure Int32 arithmetic. +const INT_ARITH: &str = "function arith(iterations){\ + let total=0;\ + const outer=Math.max(1,iterations>>7);\ + for(let j=0;j String { + let outer = (iterations >> 7).max(1); + let sum: i64 = (0..1000).map(|i| i * i).sum(); + (outer * sum).to_string() +} + +fn tier2_clif(source: &str) -> String { + let fixture = SnapshotFixture::compile(source); + let verified = fixture.snapshot().verify(VerifyLimits::default()).unwrap(); + Tier2Compiler::host(91) + .lower_for_test(&verified, 91) + .unwrap_or_else(|error| panic!("{source} did not lower: {error:?}")) +} + +fn countdowns(clif: &str) -> usize { + clif.lines() + .filter(|line| { + let line = line.split(" ;").next().unwrap_or(line).trim(); + line.contains("iadd_imm") && line.ends_with(", -1") + }) + .count() +} + +#[test] +fn a_pure_loop_counts_down_its_poll_even_when_the_function_calls() { + let clif = tier2_clif(&format!( + "({})", + INT_ARITH.replace("function arith", "function") + )); + assert!( + clif.contains("call_indirect"), + "fixture must keep generic calls outside the loops: {clif}" + ); + // Both the outer and the inner loop are call-free, so each header polls + // through its own countdown instead of calling the runtime every trip. + assert!(countdowns(&clif) >= 2, "{clif}"); +} + +#[test] +fn a_loop_containing_a_call_keeps_its_per_iteration_poll() { + let clif = tier2_clif("(function(n){let s=0;for(let i=0;i Warm { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .call_threshold(2) + .loop_threshold(16) + .force_optimized_for_test(true) + .stress_gc(true) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| ctx.eval::<(), _>(source)).unwrap(); + let deadline = Instant::now() + Duration::from_secs(20); + loop { + let before = jit.metrics(); + let result = context.with(|ctx| { + let function: Function = ctx.globals().get(name).unwrap(); + function + .call::<_, rquickjs::Value>((argument,)) + .map(|value| { + value + .as_string() + .map(|string| string.to_string().unwrap()) + .or_else(|| value.as_number().map(|number| number.to_string())) + .unwrap() + }) + }); + assert_eq!(result.unwrap(), expected); + jit.poll(); + if jit.metrics().tier2_entries > before.tier2_entries { + break; + } + assert!( + Instant::now() < deadline, + "{name} never entered Tier 2: {:?}", + jit.metrics() + ); + std::thread::sleep(Duration::from_micros(200)); + } + Warm { + runtime, + jit, + context, + } +} + +#[test] +fn per_loop_amortized_poll_keeps_results_and_interrupt_service() { + let warm = warm_tier2(INT_ARITH, "arith", 1 << 12, &expected_arith(1 << 12)); + // Steady state: exact results across trip counts, all in Tier 2. + for iterations in [0, 127, 128, 5000, 1 << 14] { + let before = warm.jit.metrics(); + let actual = warm.context.with(|ctx| { + let function: Function = ctx.globals().get("arith").unwrap(); + function.call::<_, String>((iterations,)).unwrap() + }); + assert_eq!(actual, expected_arith(i64::from(iterations))); + assert!(warm.jit.metrics().tier2_entries > before.tier2_entries); + } + // A (practically) unbounded loop nest must still reach the runtime poll + // and unwind through the uncatchable interrupt. + let interrupts = Arc::new(AtomicUsize::new(0)); + let seen = Arc::clone(&interrupts); + warm.runtime.set_interrupt_handler(Some(Box::new(move || { + seen.fetch_add(1, Ordering::SeqCst) >= 2 + }))); + let before = warm.jit.metrics(); + warm.context.with(|ctx| { + let function: Function = ctx.globals().get("arith").unwrap(); + assert!(function.call::<_, String>((i32::MAX,)).is_err()); + drop(ctx.catch()); + }); + warm.runtime.set_interrupt_handler(None); + let after = warm.jit.metrics(); + assert!(after.tier2_entries > before.tier2_entries); + assert!(interrupts.load(Ordering::SeqCst) >= 3); + assert_eq!(after.native_entries, after.native_exits); + let again = warm.context.with(|ctx| { + let function: Function = ctx.globals().get("arith").unwrap(); + function.call::<_, String>((300,)).unwrap() + }); + assert_eq!(again, expected_arith(300)); +} + +#[test] +fn scalar_loop_overflow_deopt_restores_the_exact_frame() { + // The overflow exit reloads the frame buffers and exit record instead of + // keeping the prologue values live; the interpreter must resume with the + // exact pre-overflow sum and induction variable. + let warm = warm_tier2( + "function sum(n){let s=0;for(let i=0;i((100_000,)).unwrap() + }); + assert_eq!(overflowing, 4_999_950_000.0); + let after = warm.jit.metrics(); + assert!(after.tier2_entries > before.tier2_entries); + assert!(after.deopts > before.deopts, "{after:?}"); + assert_eq!(after.native_entries, after.native_exits); +} + +#[test] +fn constant_unsigned_shift_stays_int32_in_a_mixed_loop() { + // `v >>> 3` of a negative Int32 is a large positive Int32; the xor must + // consume it exactly without a Float64 detour. + let source = "function bits(n){let v=-123456789;\ + for(let i=0;i>>3)^i)|0;\ + return v}"; + let expected = |n: i32| { + let mut v: i32 = -123_456_789; + for i in 0..n { + v = v.wrapping_shl(5) ^ ((v as u32 >> 3) as i32) ^ i; + } + v + }; + let warm = warm_tier2(source, "bits", 300, &expected(300).to_string()); + for n in [0, 1, 7, 1000, 4097] { + let before = warm.jit.metrics(); + let actual = warm.context.with(|ctx| { + let function: Function = ctx.globals().get("bits").unwrap(); + function.call::<_, i32>((n,)).unwrap() + }); + assert_eq!(actual, expected(n)); + assert!(warm.jit.metrics().tier2_entries > before.tier2_entries); + } + assert_eq!( + warm.jit.metrics().native_entries, + warm.jit.metrics().native_exits + ); +} From 4acb26ad4f77d6193562e15258eb719c1fa32beb Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:07:21 +0800 Subject: [PATCH 19/54] docs: record P4d scalar codegen slice and noisy diagnostics Co-Authored-By: Claude Opus 5.5 (1M context) --- ...calar-codegen-noisy-diagnostic-x86_64.json | 610 ++++++++++++++++++ docs/roadmap/p4d-scalar-codegen.md | 191 ++++++ 2 files changed, 801 insertions(+) create mode 100644 benchmarks/results/p4d-scalar-codegen-noisy-diagnostic-x86_64.json create mode 100644 docs/roadmap/p4d-scalar-codegen.md diff --git a/benchmarks/results/p4d-scalar-codegen-noisy-diagnostic-x86_64.json b/benchmarks/results/p4d-scalar-codegen-noisy-diagnostic-x86_64.json new file mode 100644 index 000000000..f746f18e3 --- /dev/null +++ b/benchmarks/results/p4d-scalar-codegen-noisy-diagnostic-x86_64.json @@ -0,0 +1,610 @@ +{ + "description": "Noisy diagnostic, not publishable. jit-bench worker --mode automatic, alternating baseline 82d3808 (/home/jason/work/quickjs-jit/target/release/jit-bench) and perf/p4d-scalar-codegen per run; value per run = median(protocol.warmup_batch_ns[-16:]) in ns per batch of 10 workload calls. Host i7-13700KF shared with ~16 concurrent agents; hybrid P/E cores make runs bimodal. fast_mode = median of runs <= 1.4x the per-engine minimum. checksums_match compares both engines against the interpreter-mode checksum of the new binary.", + "targets": { + "numeric": { + "base_ns": [ + 54494.5, + 29215.0, + 34412.5, + 29503.5, + 31949.0, + 32261.5, + 29444.5, + 55794.0, + 54762.0, + 29313.5, + 42845.5 + ], + "new_ns": [ + 49389.5, + 52401.5, + 37153.5, + 21503.5, + 23211.5, + 21416.5, + 21375.0, + 21390.5, + 21486.0, + 21359.0, + 21320.5 + ], + "median_speed": 1.5015126128641907, + "fast_mode_speed": 1.3784427780503188, + "checksums_match": true + }, + "scalar-loop": { + "base_ns": [ + 29187.0, + 29600.0, + 29272.5, + 55925.5, + 29431.5, + 29263.5, + 30753.0, + 29127.5, + 29207.0, + 29431.0, + 55535.0 + ], + "new_ns": [ + 20652.0, + 49473.5, + 21461.5, + 21291.5, + 21454.0, + 21383.0, + 21422.5, + 21447.5, + 21499.0, + 50205.5, + 21298.5 + ], + "median_speed": 1.3722345261685511, + "fast_mode_speed": 1.3664371571945384, + "checksums_match": true + }, + "scalar-control-flow": { + "base_ns": [ + 54308.0, + 54458.0, + 54627.0, + 54600.0, + 54353.0, + 103877.5, + 54548.5, + 54277.5, + 54939.0, + 55237.5, + 55044.0 + ], + "new_ns": [ + 43934.5, + 49981.0, + 43947.0, + 43737.0, + 43768.5, + 43919.5, + 43723.5, + 44319.5, + 43922.0, + 43736.0, + 43844.0 + ], + "median_speed": 1.2431835517253156, + "fast_mode_speed": 1.242597251790207, + "checksums_match": true + }, + "scalar-expressions": { + "base_ns": [ + 50451.0, + 50391.0, + 50464.5, + 50451.0, + 50562.5, + 50430.0, + 50672.5, + 50643.0, + 50685.5, + 50440.5, + 50604.5 + ], + "new_ns": [ + 45820.5, + 45495.5, + 45484.0, + 45681.0, + 45572.5, + 45641.5, + 45603.0, + 45589.5, + 45924.0, + 45425.5, + 77217.5 + ], + "median_speed": 1.1066048286296954, + "fast_mode_speed": 1.1067686487375608, + "checksums_match": true + }, + "quickjs-bitops": { + "base_ns": [ + 62113.5, + 111476.5, + 77017.0, + 103516.5, + 61908.0, + 61901.0, + 65440.5, + 62794.0, + 63674.0, + 101951.0, + 61894.0 + ], + "new_ns": [ + 63716.5, + 31625.5, + 64177.5, + 31730.0, + 59846.0, + 62208.5, + 31998.5, + 64153.0, + 61983.0, + 31866.0, + 31547.0 + ], + "median_speed": 1.063964174715102, + "fast_mode_speed": 1.9682871099905452, + "checksums_match": true + }, + "quickjs-int-arith": { + "base_ns": [ + 1378596.0, + 1363117.5, + 1323394.5, + 1337757.0, + 1330072.0, + 1355545.0, + 1327279.5, + 1328156.0, + 1357338.5, + 1370443.5, + 1330557.5 + ], + "new_ns": [ + 331377.5, + 336021.0, + 326964.0, + 327726.5, + 330250.5, + 333516.0, + 328931.5, + 328968.5, + 325912.0, + 327642.5, + 327632.5 + ], + "median_speed": 4.0669774709932005, + "fast_mode_speed": 4.0669774709932005, + "checksums_match": true + }, + "fibonacci-iterative": { + "base_ns": [ + 986966.0, + 997385.5, + 1008717.0, + 1003569.5, + 980623.0, + 995214.5, + 989059.0, + 991964.0, + 1002596.0, + 1003265.0, + 986143.5 + ], + "new_ns": [ + 743178.0, + 742836.0, + 744703.0, + 747322.0, + 746036.5, + 744285.0, + 751882.0, + 744536.5, + 744561.0, + 746252.5, + 743135.5 + ], + "median_speed": 1.3366460236300317, + "fast_mode_speed": 1.3366460236300317, + "checksums_match": true + }, + "host-compute": { + "base_ns": [ + 68686.5, + 71117.5, + 71365.0, + 71299.5, + 70783.0, + 128140.5, + 139753.5, + 72315.5, + 70552.0, + 68864.0, + 72185.0 + ], + "new_ns": [ + 51547.0, + 50883.5, + 51380.0, + 51000.5, + 51628.5, + 57548.0, + 50849.0, + 50939.5, + 50996.5, + 52080.5, + 52412.0 + ], + "median_speed": 1.3876897625535227, + "fast_mode_speed": 1.3841475282210978, + "checksums_match": true + }, + "float64-dense": { + "base_ns": [ + 128755.5, + 129131.5, + 130562.0, + 129489.5, + 128452.0, + 129830.5, + 128900.0, + 128758.5, + 129125.5, + 131718.5, + 128510.0 + ], + "new_ns": [ + 101013.0, + 101455.5, + 101705.0, + 101200.0, + 100684.5, + 100508.0, + 99297.5, + 101186.5, + 100890.5, + 101116.0, + 101120.5 + ], + "median_speed": 1.2770036393844693, + "fast_mode_speed": 1.2770036393844693, + "checksums_match": true + } + }, + "controls": { + "property-heavy": { + "base_ns": [ + 98624.5, + 98886.0, + 103049.5, + 98892.5, + 98562.0, + 99010.5, + 148450.5, + 98658.0, + 98688.5 + ], + "new_ns": [ + 150656.5, + 96908.0, + 96232.5, + 96172.0, + 96474.0, + 96609.0, + 96370.5, + 150852.0, + 96324.0 + ], + "median_speed": 1.025001554823061, + "fast_mode_speed": 1.0250776949377662, + "checksums_match": true + }, + "int32array-traversal": { + "base_ns": [ + 49468.5, + 54987.0, + 49512.0, + 49578.5, + 49753.5, + 49437.5, + 49803.0, + 49412.5, + 49450.5 + ], + "new_ns": [ + 37405.0, + 66709.0, + 37436.0, + 37661.0, + 37417.5, + 67173.0, + 37536.0, + 37246.5, + 37479.0 + ], + "median_speed": 1.3210597934843513, + "fast_mode_speed": 1.3225771984186345, + "checksums_match": true + }, + "packed-array-traversal": { + "base_ns": [ + 48088.5, + 48567.0, + 48250.0, + 48017.5, + 47968.0, + 48144.0, + 48286.5, + 110812.0, + 111423.0 + ], + "new_ns": [ + 39759.5, + 39900.0, + 40148.5, + 40145.5, + 39922.5, + 40087.5, + 77187.0, + 77181.0, + 39937.0 + ], + "median_speed": 1.2036170876208294, + "fast_mode_speed": 1.2054986603901143, + "checksums_match": true + }, + "call-heavy": { + "base_ns": [ + 59794.5, + 105266.5, + 59370.5, + 60694.5, + 59662.5, + 59807.0, + 59407.5, + 59835.5, + 59556.5 + ], + "new_ns": [ + 41770.0, + 41635.5, + 41732.0, + 42405.0, + 44070.5, + 41310.0, + 41300.0, + 41555.0, + 41408.0 + ], + "median_speed": 1.4361422343913246, + "fast_mode_speed": 1.4345570486724069, + "checksums_match": true + }, + "float64array-traversal": { + "base_ns": [ + 1096087.5, + 1109460.0, + 1090380.5, + 1092694.0, + 1067363.0, + 1102924.0, + 1122704.0, + 1090801.5, + 1096589.0 + ], + "new_ns": [ + 1027350.5, + 1063833.5, + 1038933.5, + 1057446.5, + 1539293.0, + 1100576.0, + 1063299.5, + 1046204.0, + 1762201.5 + ], + "median_speed": 1.030836090866214, + "fast_mode_speed": 1.0365418014055559, + "checksums_match": true + }, + "arrays-typed": { + "base_ns": [ + 2974803.5, + 2966092.0, + 3011055.0, + 2992880.0, + 2985622.5, + 2958535.0, + 2977477.5, + 2992062.5, + 2967241.5 + ], + "new_ns": [ + 2956533.5, + 2942832.0, + 2955104.5, + 2925640.0, + 2925449.5, + 2986125.5, + 2948524.5, + 2942493.5, + 2980814.5 + ], + "median_speed": 1.009819487679346, + "fast_mode_speed": 1.009819487679346, + "checksums_match": true + }, + "generic-call-entry": { + "base_ns": [ + 32314.5, + 32176.5, + 32586.0, + 32660.5, + 70743.0, + 75214.5, + 71521.5, + 32213.0, + 32318.0 + ], + "new_ns": [ + 62168.0, + 32154.5, + 61640.0, + 61980.0, + 62243.0, + 61283.5, + 30006.5, + 30219.5, + 29970.0 + ], + "median_speed": 0.5317255052338721, + "fast_mode_speed": 1.0731660744528941, + "checksums_match": true + }, + "generic-call-fallback": { + "base_ns": [ + 9521545.5, + 9378445.0, + 9127386.5, + 9122762.5, + 9218784.0, + 9240363.0, + 9079020.0, + 9476228.5, + 9156024.5 + ], + "new_ns": [ + 9196247.0, + 9122232.0, + 9113840.0, + 9036587.0, + 9041973.0, + 9045365.5, + 9163495.0, + 9040979.5, + 9141391.0 + ], + "median_speed": 1.011514795080888, + "fast_mode_speed": 1.011514795080888, + "checksums_match": true + }, + "objects-polymorphic": { + "base_ns": [ + 24034730.0, + 24430464.5, + 24493621.5, + 23874224.5, + 24172948.5, + 23833836.5, + 24169369.5, + 24356837.0, + 33140320.0 + ], + "new_ns": [ + 23614740.0, + 23748308.0, + 24633324.5, + 23783489.5, + 23636898.0, + 23705968.5, + 25485659.0, + 32314781.5, + 24646027.0 + ], + "median_speed": 1.0163751832967993, + "fast_mode_speed": 1.0163751832967993, + "checksums_match": true + }, + "mixed-quotes": { + "base_ns": [ + 2547404.5, + 2528437.0, + 2502408.5, + 4013325.5, + 2518825.0, + 2506579.0, + 2682975.5, + 2556969.5, + 2535039.0 + ], + "new_ns": [ + 2597051.5, + 2560822.5, + 2529385.5, + 2614593.0, + 2714129.0, + 2647784.0, + 2654474.5, + 2638569.0, + 2550716.0 + ], + "median_speed": 0.9695730846062848, + "fast_mode_speed": 0.96831055540958, + "checksums_match": true + }, + "fibonacci-recursive": { + "base_ns": [ + 10044544.0, + 9849620.5, + 9888162.0, + 12593757.0, + 9855305.0, + 9839368.0, + 12742754.5, + 9847309.5, + 9857063.0 + ], + "new_ns": [ + 10160929.5, + 9899014.0, + 9847764.0, + 9856289.5, + 9834067.5, + 9875858.5, + 9842308.5, + 9839606.0, + 9834596.0 + ], + "median_speed": 1.000944275268985, + "fast_mode_speed": 1.000944275268985, + "checksums_match": true + }, + "calls-closures": { + "base_ns": [ + 3313062.5, + 3310130.0, + 3319059.0, + 3423131.0, + 3547198.0, + 4475368.5, + 4997349.0, + 3315583.5, + 3335263.0 + ], + "new_ns": [ + 3312646.5, + 3708894.5, + 3362192.5, + 3360146.5, + 3399654.5, + 3311014.5, + 3525387.0, + 3484235.5, + 5269595.0 + ], + "median_speed": 0.981059398830087, + "fast_mode_speed": 0.9840982796564312, + "checksums_match": true + } + } +} \ No newline at end of file diff --git a/docs/roadmap/p4d-scalar-codegen.md b/docs/roadmap/p4d-scalar-codegen.md new file mode 100644 index 000000000..03f07990d --- /dev/null +++ b/docs/roadmap/p4d-scalar-codegen.md @@ -0,0 +1,191 @@ +# P4d: scalar loop code quality (first slice) + +Branch `perf/p4d-scalar-codegen`, based on `82d3808`. Roadmap item: P4, +"降低 poll 和溢出检查的密度" plus frame syncs, tag checks and Float64/Int32 +representation churn in scalar kernels +([roadmap](../PERFORMANCE_ROADMAP_V8_BUN.md)). + +This slice changes only Tier 2 (optimizing) code generation. No QuickJS C +patch, ABI version, bytecode fingerprint or README matrix changes. + +## How the code was inspected + +Benchmark binaries are built with `test-support`, which already enables +Cranelift disassembly. A new test-support-only diagnostic writes every +finalized function's CLIF and machine code: + +```sh +QJSJIT_DUMP_CODE=/some/dir ./target/release/jit-bench worker --mode automatic \ + --script benchmarks/scripts/scalar-loop.js +``` + +Files are named `--tier1|tier1-osr|tier2.txt`. Tier 2 files contain +the CLIF after the post-lowering cleanups below, followed by the VCode +disassembly. Without the variable the hook does nothing, and it does not exist +in builds without `test-support`. + +## Findings on `82d3808` + +`for (let i = z; i < n; i++) sum = sum + i` ran a Tier 2 loop of about 27 +instructions per iteration, including 2 loads and 4 stores: + +- the amortized poll countdown lived in a stack slot (load, decrement, store + every iteration, plus a register holding the slot address); +- every Int32 tag was a fresh `iconst` per block, so SSA construction turned + loop-invariant tags into block parameters (four constant-zero phis in this + loop). Cranelift's constant-phi removal compares value identity and runs + before GVN, so they survived into register allocation; +- `arg_buf`, `var_buf` and `stack_base` loaded in the prologue were kept live + across the loop only for the deopt/exit blocks; +- the `sret` exit-record pointer was kept live (and spilled/reloaded inside + the loop) because Cranelift's `StructReturn` parameter must be returned in + `rax` at every `return`; +- guards the specializer had already proven compiled to `movl $1; test; jnz` + because Cranelift does not fold `brif` on constants; +- `x >>> 3` always produced a Float64 alternative plus a select, and Float64 + arithmetic round-tripped every intermediate through a GPR (`bitcast` pairs); +- functions with any call (for example `quickjs-int-arith`: `Math.max` before + the loops and `String` after them) failed the whole-function amortized-poll + proof, so every iteration of their pure inner loops called the poll helper + through the full helper-frame validation. + +## Changes + +All rewrites are semantics-preserving CLIF transforms or narrower proofs; +failures fall back to the previous code shape. + +1. **Poll countdown in a register.** The amortized poll budget is a Cranelift + `Variable` instead of a stack slot (`emit_opt_amortized_poll`). +2. **Per-loop poll amortization.** When whole-function amortization is + rejected, each natural loop whose own nodes pass the same + `permits_amortized_poll_with_guarded_heap` proof (no reentrant or unknown + calls, no unguarded heap operations, primitive frame writes) counts down its + header poll (`emit_opt_countdown_poll`). Unlike whole-function mode, frame + locals keep their immediate stores, and the header numeric guard, + hoisted-call revalidation and packed-array revalidation still run on every + iteration: only the poll call frequency changes, exactly as it already does + for raw Int32 loops. The poll block keeps its layout position (moving call + sites changes which call return publishes the helper stack map). +3. **Canonical integer constants** (`canonicalize_integer_constants`): every + `iconst` is replaced by one definition per (type, value) at the top of the + entry block, so constant phis collapse. +4. **Constant-branch folding** (`fold_constant_branches`): after constant-phi + removal, a `brif` whose condition evaluates to a constant (constants, width + conversions, bitwise ops, integer compares) becomes a `jump`, and the dead + successors are removed. If folding removes every call, the artifact no + longer publishes a helper stack map. +5. **Bitcast round trips** (`fold_bitcast_round_trips`): `bitcast.T(bitcast.U + x)` with `x: T` forwards `x` (scalar types only). +6. **Exit rematerialization** (`rematerialize_invariants_in_exits`, functions + with loops only): blocks ending in `return` reload the root frame's buffer + pointers from the frame, and `sret` from a prologue stash slot. On x86-64 + System V the exit record pointer is passed as an ordinary pointer argument + and each exit returns the reloaded pointer explicitly; the machine-level + contract of `JSJitExit (*)(JSJitExecFrame *)` (hidden pointer in `rdi`, + returned in `rax`) is unchanged. Other targets keep `StructReturn`. +7. **Constant unsigned shifts**: `x >>> c` with `c & 31 != 0` is always below + 2^31 and lowers directly to an Int32 (no Float64 alternative). + +`RelocatableCode::clif()` still returns the lowering output (before items 3-6) +so existing CLIF-structure tests keep describing the lowering; the dump shows +the final CLIF. + +After the change the same scalar loop is 14 instructions per iteration with no +memory operations: + +```text +lea -1(%rcx),%ecx; test; jz poll ; countdown +cmp %r15d,%ebx; jl body ; i < n +mov %r12,%r8; add %ebx,%r8d; seto; test; jnz deopt ; checked sum + i +lea 1(%rbx),%ebx; mov %r8,%r12; jmp header ; i++ (range-proven) +``` + +## Noisy diagnostic measurements + +Not publishable evidence: the i7-13700KF was shared with about 16 concurrent +agents, there is no Bun column, and runs were not pinned. Each run alternates +the `82d3808` baseline binary and this branch +(`jit-bench worker --mode automatic --script benchmarks/scripts/W.js`); a +run's value is the median of its last 16 warmup batches (ms per 10 workload +calls), and the table reports the median over runs. Hybrid P/E cores make runs +bimodal, so the "P-core mode" columns repeat the medians over runs within 1.4x +of each engine's fastest run. Speed is baseline time / new time (above 1.00x is +faster). Checksums of both binaries matched the interpreter in every run. Raw +per-run values: +[`p4d-scalar-codegen-noisy-diagnostic-x86_64.json`](../../benchmarks/results/p4d-scalar-codegen-noisy-diagnostic-x86_64.json). + +| Workload | Role | Base ms | New ms | Speed | Base ms (P-core mode) | New ms (P-core mode) | Speed (P-core mode) | Runs | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| numeric | target | 0.0323 | 0.0215 | 1.50x | 0.0295 | 0.0214 | 1.38x | 11 | +| scalar-loop | target | 0.0294 | 0.0214 | 1.37x | 0.0293 | 0.0214 | 1.37x | 11 | +| scalar-control-flow | target | 0.0546 | 0.0439 | 1.24x | 0.0546 | 0.0439 | 1.24x | 11 | +| scalar-expressions | target | 0.0505 | 0.0456 | 1.11x | 0.0505 | 0.0456 | 1.11x | 11 | +| quickjs-bitops | target | 0.0637 | 0.0598 | 1.06x | 0.0625 | 0.0317 | 1.97x | 11 | +| quickjs-int-arith | target | 1.3378 | 0.3289 | 4.07x | 1.3378 | 0.3289 | 4.07x | 11 | +| fibonacci-iterative | target | 0.9952 | 0.7446 | 1.34x | 0.9952 | 0.7446 | 1.34x | 11 | +| host-compute | target | 0.0713 | 0.0514 | 1.39x | 0.0711 | 0.0514 | 1.38x | 11 | +| float64-dense | target | 0.1291 | 0.1011 | 1.28x | 0.1291 | 0.1011 | 1.28x | 11 | +| property-heavy | control | 0.0989 | 0.0965 | 1.03x | 0.0988 | 0.0964 | 1.03x | 9 | +| int32array-traversal | control | 0.0495 | 0.0375 | 1.32x | 0.0495 | 0.0374 | 1.32x | 9 | +| packed-array-traversal | control | 0.0483 | 0.0401 | 1.20x | 0.0481 | 0.0399 | 1.21x | 9 | +| call-heavy | control | 0.0598 | 0.0416 | 1.44x | 0.0597 | 0.0416 | 1.43x | 9 | +| float64array-traversal | control | 1.0961 | 1.0633 | 1.03x | 1.0961 | 1.0574 | 1.04x | 9 | +| arrays-typed | control | 2.9775 | 2.9485 | 1.01x | 2.9775 | 2.9485 | 1.01x | 9 | +| generic-call-entry | control | 0.0326 | 0.0613 | 0.53x | 0.0323 | 0.0301 | 1.07x | 9 | +| generic-call-fallback | control | 9.2188 | 9.1138 | 1.01x | 9.2188 | 9.1138 | 1.01x | 9 | +| objects-polymorphic | control | 24.1729 | 23.7835 | 1.02x | 24.1729 | 23.7835 | 1.02x | 9 | +| mixed-quotes | control | 2.5350 | 2.6146 | 0.97x | 2.5317 | 2.6146 | 0.97x | 9 | +| fibonacci-recursive | control | 9.8571 | 9.8478 | 1.00x | 9.8571 | 9.8478 | 1.00x | 9 | +| calls-closures | control | 3.3353 | 3.3997 | 0.98x | 3.3272 | 3.3809 | 0.98x | 9 | + +Notes: + +- `quickjs-bitops` and `generic-call-entry` medians are dominated by core + placement. Pinned to E-core 20, `quickjs-bitops` measured about 0.101 ms + (base) and 0.061 ms (new), 1.66x; on an uncontended P-core, 0.062 ms and + 0.032 ms. In `generic-call-entry`, this branch was faster within each mode + (about 0.030 ms vs 0.032 ms, and 0.062 ms vs 0.071 ms). More of its runs + landed on slow cores, which pulls the overall median to 0.53x. +- A 13-run rerun of `mixed-quotes` (2.537 ms vs 2.547 ms, 1.00x) and + `calls-closures` (3.359 ms vs 3.246 ms, 1.03x) found both statistically + tied. These controls do not execute rewritten loops. + +## Tests + +- `jit/tests/scalar_codegen.rs` (new): per-loop countdown appears in a + function with calls outside the loop and not in a loop containing a call; + exact results across trip counts, uncatchable interrupt service in an + effectively unbounded amortized loop nest, exact overflow deopt of a scalar + loop (rematerialized exit), and constant `>>>` in a Tier 2 bit-mixing loop. +- `compiler::baseline::post_lowering_tests`: constant sharing, constant-branch + folding (including narrow unsigned/signed compares and runtime conditions + left alone), and bitcast forwarding, each verified with the Cranelift + verifier. +- `m2_core_opcodes::shifts_mask_the_count_and_unsigned_shift_renormalizes` + now also covers `-1 >>> 32` (a constant count masking to zero keeps its + Float64 result). +- Updated evidence tests: the property-loop countdown test now requires an + SSA countdown (block parameter, no stack slot); the bitops CLIF test + requires no `fcvt_from_uint` for constant counts and keeps it for a + zero-masked count; the float side-path test counts the side-path-hit flag + write instead of raw `brif` counts (proven guards now fold in both + versions). + +## Remaining work + +- Fold constant branches that only become constant after Cranelift's egraph + (for example in `float64-dense`); this needs a second optimize pass or a + custom GVN before folding. +- Overflow checks still lower to `seto; test; jnz` because Cranelift does not + fuse `sadd_overflow` with `brif`. Accumulators such as `sum + i` need the + check; range facts already drop it for bounded induction variables. +- Values needed only by deopt state (for example the per-iteration `next` + local and its tag in `fibonacci-iterative`) are spilled every iteration. + Recomputing them at the exit from other live values would remove the stores. +- The root frame pointer still moves between an argument register and a + callee-saved register around the cold poll call; a preserve-all poll + trampoline would avoid that but is not available in Cranelift 0.116. +- Extend per-loop amortization to deferred local stores (currently only the + poll call is amortized per loop), with a flush at loop exit. +- Publishable three-engine measurements (the numbers below are noisy + diagnostics on a loaded machine). From 505169d4fa4b2bcba9c02f444d18eb709f1ee871 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:09:23 +0800 Subject: [PATCH 20/54] test(jit): production, OSR and async-probe coverage for exception regions - Production tiering runs a hot try/catch/finally loop natively and an OSR entry lands on a loop header inside a try region. - Refused async/generator snapshots are no longer re-requested. - sys patch tests cover patch 0027 and the ABI 1.25 bindings. - Document patch 0027 and the p2c-exceptions slice, noisy diagnostics and remaining work. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p2c-exceptions.md | 81 ++++++++++++++++++++++++++++++++++ jit/tests/tier1_exceptions.rs | 77 ++++++++++++++++++++++++++++++++ sys/patches/README.md | 13 ++++++ sys/tests/jit_patch.rs | 21 +++++++-- 4 files changed, 188 insertions(+), 4 deletions(-) create mode 100644 docs/roadmap/p2c-exceptions.md diff --git a/docs/roadmap/p2c-exceptions.md b/docs/roadmap/p2c-exceptions.md new file mode 100644 index 000000000..e94cda889 --- /dev/null +++ b/docs/roadmap/p2c-exceptions.md @@ -0,0 +1,81 @@ +# P2c: exception regions in Tier 1 + +Branch `perf/p2c-exceptions`, based on `82d3808`. Roadmap item P2.3 in +[the V8/Bun roadmap](../PERFORMANCE_ROADMAP_V8_BUN.md). + +## What landed + +- `throw`, `throw_error`, `catch`, `nip_catch`, `gosub` and `ret` are Tier 1 + `Native` opcodes (opcode manifest cases added). +- Verifier: a `catch` handler is entered with the caught value in place of its + catch offset and with locals joined over every instruction of the try + region (initialized/uninitialized disagreements widen to `Tagged`). `gosub` + pushes an Int32 return offset and has only its finally block as successor; + each `ret` gets the return points of the `gosub` sites of the finally block + that reaches it (a `ret` owned by two finally entries is rejected). Dead code + after a finally block's `ret` is tolerated only in functions with + subroutines and is never lowered. The innermost live catch offset is + recorded for every PC. +- Native dispatch: every helper exception edge inside a try region publishes + the exact owned operand stack (as the ordinary exception exit does) and + enters a per-handler landing pad. The pad calls `JS_JitCatchException`, which + replays the interpreter's exception label (backtrace at the throwing PC, + catchability) and, after proving the innermost catch offset is the expected + slot/handler, releases the operands above it and stores the caught value. + A caught exception resumes the handler natively; uncatchable errors, + interrupts and invariant failures leave through the normal exception exit + so the interpreter unwinds the published frame exactly. +- `JS_JitThrowValue` / `JS_JitThrowError` implement `throw` / `throw_error`. +- The entry-domain analysis merges every exceptional edge (before, after and + mid-store states) into handler inputs, so `get_loc_check` proofs stay sound. +- Tier 2, bounded inlining, tagged direct links and inline snapshot retention + reject functions with exception regions up front; production maintenance no + longer offers them to Tier 2. +- OSR validation now checks `CatchOffset` slots against + `JS_TAG_CATCH_OFFSET` (it compared them with the Int32 tag), so loop headers + inside try regions can be entered. +- Patch 0027 (ABI 1.25) also fixes the `exceptions-promises-async` + regression: generator/async resumes pass their state as `func_obj`, the + snapshot was refused without the backend ever learning why, and maintenance + re-armed the request, so every call, resume and loop poll paid a `record_hot` + round trip plus maintenance (2,030 snapshot requests per worker run; now 7). + Refused generator/async/eval/`with` bytecode now disables its probes locally. +- New benchmark scenario `exceptions-sync` (pure JS, Bun-compatible): a caught + primitive throw every 16th iteration, a finally block on every iteration and + a helper-raised TypeError every 256th iteration. + +## Noisy diagnostic (not publishable evidence) + +Median over 7 alternating worker runs of `median(warmup_batch_ns[-16:])`, +automatic tiering, on the shared 24-core host while ~16 agents were building +and testing. Baseline is the `82d3808` `jit-bench`; speed is baseline time / +new time. Checksums matched the interpreter in every run. + +| workload | 82d3808 ms | p2c ms | speed | +| --- | ---: | ---: | ---: | +| exceptions-sync (new) | 2.9537 | 1.6643 | 1.78x | +| exceptions-promises-async | 3.7634 | 2.4500 | 1.54x | +| adversarial | 0.9545 | 0.9501 | 1.00x (tied) | +| scalar-loop (control) | 0.0293 | 0.0290 | 1.01x (tied) | +| call-heavy (control) | 0.0599 | 0.0605 | 0.99x (tied) | +| mixed-quotes (control) | 2.5143 | 2.5564 | 0.98x (tied) | + +`exceptions-promises-async` is now at interpreter speed (its `workload` and +`asyncStep` are async and stay interpreted; the gain is the removed probe +overhead). `adversarial` has no exception region; its Tier 1 code is rejected +by the profitability gate, which this item does not change. The publishable +three-engine matrix, including `exceptions-sync`, is left to the integrator. + +## Remaining work + +- Tier 2 exception regions (landing pads with deopt-map ownership, handler + blocks in the scalar graph). Tier 2 currently rejects these functions. +- `nip_catch` with operands between the kept value and its catch offset (only + the direct form is lowered; others reject the compile). +- For-of/for-in iterator close offsets (`catch` offset 0) inside try regions: + rejected until the iteration opcodes are admitted (P2.5). +- Every native catch still performs the interpreter's backtrace build for + primitive throws (`ctx->error_back_trace`), which dominates throw-heavy + loops; skipping it when it is provably unobservable is a runtime-library + change, not a JIT one. +- Helper-frame validation (P1 B1) is paid twice per native throw/catch. diff --git a/jit/tests/tier1_exceptions.rs b/jit/tests/tier1_exceptions.rs index f241d8674..a4518b945 100644 --- a/jit/tests/tier1_exceptions.rs +++ b/jit/tests/tier1_exceptions.rs @@ -12,7 +12,9 @@ not(all(target_os = "windows", target_arch = "aarch64")) ))] +use rquickjs::{Context, Runtime}; use rquickjs_jit::test_support::{differential, forced_baseline}; +use rquickjs_jit::{Jit, JitConfig, JitMetrics}; fn same(definition: &str, expression: &str, opcode: &str) { differential(definition, expression) @@ -157,6 +159,81 @@ fn throw_error_raises_the_interpreter_error() { ); } +/// Runs `source` then evaluates `expression` `rounds` times with and without +/// the production JIT, returning both results and the JIT metrics. +fn production(source: &str, expression: &str, rounds: usize) -> (String, String, JitMetrics) { + let run = |jit: bool| { + let runtime = Runtime::new().unwrap(); + let attached = jit.then(|| Jit::attach(&runtime, JitConfig::default()).unwrap()); + let context = Context::full(&runtime).unwrap(); + let mut result = String::new(); + context.with(|ctx| ctx.eval::<(), _>(source).unwrap()); + for _ in 0..rounds { + result = context.with(|ctx| ctx.eval::(expression).unwrap()); + while runtime.is_job_pending() { + runtime.execute_pending_job().unwrap(); + } + if let Some(jit) = &attached { + jit.poll(); + } + } + ( + result, + attached.map(|jit| jit.metrics()).unwrap_or_default(), + ) + }; + let (interpreted, _) = run(false); + let (compiled, metrics) = run(true); + (interpreted, compiled, metrics) +} + +#[test] +fn production_tiering_runs_hot_exception_loops_natively() { + let (interpreted, compiled, metrics) = production( + "function f(n){ let caught=0, finals=0; for(let i=0;i 0, "{metrics:?}"); + assert_eq!(metrics.native_retries, 0, "{metrics:?}"); + assert_eq!(metrics.invalid_artifacts, 0, "{metrics:?}"); +} + +#[test] +fn osr_enters_a_loop_header_inside_a_try_region() { + // The first invocation is long enough to enter the loop through OSR with + // the catch offset live on the interpreter operand stack. + let (interpreted, compiled, metrics) = production( + "function f(n){ let s = 0; try { for (let i = 0; i < n; i++) { s = (s + i) | 0; if (i === n - 3) throw s; } } catch (e) { return 'caught:' + e; } return 'done:' + s; }", + "f(3000000)", + 1, + ); + assert_eq!(compiled, interpreted); + assert!(metrics.osr_entries >= 1, "{metrics:?}"); + assert_eq!(metrics.native_retries, 0, "{metrics:?}"); +} + +#[test] +fn async_functions_stop_requesting_refused_snapshots() { + // Async and generator frames can never compile. Their refusal is local to + // the bytecode, so hot probes must stop instead of re-requesting a + // snapshot at every call, resume and loop poll. + let (interpreted, compiled, metrics) = production( + "async function step(v){ return (await Promise.resolve(v + 1)) * 3; }\n\ + async function g(n){ let s = 0; for (let i = 0; i < n; i++) { try { if ((i & 15) === 0) throw i; } catch (e) { s += e; } } for (let i = 0; i < 8; i++) s = await step(s & 0xffff); globalThis.out = String(s); }\n\ + function* gen(n){ for (let i = 0; i < n; i++) yield i; }\n\ + function h(){ let t = 0; for (const v of gen(500)) t += v; return t; }", + "(g(2000), String(h()))", + 60, + ); + assert_eq!(compiled, interpreted); + assert!( + metrics.snapshot_requests <= 8, + "refused async/generator snapshots were re-requested: {metrics:?}" + ); +} + #[test] fn interrupts_inside_try_regions_stay_uncatchable() { forced_baseline( diff --git a/sys/patches/README.md b/sys/patches/README.md index 931cf9bd3..13c5e88dd 100644 --- a/sys/patches/README.md +++ b/sys/patches/README.md @@ -60,6 +60,19 @@ data pointer before returning `{ data, count, mode }`. The table advertises zero effects and the receiver is passed by pointer so generated calls do not depend on platform-specific aggregate argument classification. +`0027-tier1-exception-regions.patch` adds the ABI 1.25 Tier 1 exception entry +points. `JS_JitThrowValue` and `JS_JitThrowError` raise exactly like +`OP_throw` and `OP_throw_error`. `JS_JitCatchException` replays the +interpreter's `exception:` label for the active root frame (backtrace at the +throwing PC, then catchability) and, only after proving that unwinding would +stop at the named catch-offset slot and handler PC, releases the operands +above it and stores the caught value in that slot. Uncatchable exceptions are +returned with the frame untouched so the ordinary native exception exit lets +the interpreter unwind them. The patch also stops hot and feedback probes for +generator, async, eval and `with` bytecode when the backend requests a +snapshot the runtime must refuse; previously every later call, async resume +and loop poll re-requested it. + The build accepts only the patch names and byte digests listed in `build_support/patch.rs`, then verifies the complete patched source manifest. Keeping the baseline and patch separate makes diff --git a/sys/tests/jit_patch.rs b/sys/tests/jit_patch.rs index f36861117..16f418308 100644 --- a/sys/tests/jit_patch.rs +++ b/sys/tests/jit_patch.rs @@ -56,6 +56,7 @@ fn copy_patches(destination: &std::path::Path) { "0018-inline-frame-recovery.patch", "0019-array-feedback.patch", "0020-typed-array-guard.patch", + "0027-tier1-exception-regions.patch", ] { fs::copy(source.join(patch), destination.join(patch)).unwrap(); } @@ -114,7 +115,12 @@ fn pinned_public_quickjs_baseline_applies_cleanly_without_git() { let helper_header = fs::read_to_string(destination.join("quickjs-jit-helpers.h")).unwrap(); assert!(quickjs.contains("JS_GetJitRuntimeId")); assert!(quickjs.contains("JS_JIT_FRAME_SIDE_PATH_HIT")); - assert!(jit_header.contains("#define QJSJIT_ABI_MINOR 24u")); + assert!(jit_header.contains("#define QJSJIT_ABI_MINOR 25u")); + assert!(jit_header.contains("JS_JitCatchException")); + assert!(quickjs.contains("JSJitHelperStatus JS_JitThrowValue(")); + assert!(quickjs.contains("JSJitHelperStatus JS_JitThrowError(")); + assert!(quickjs.contains("JSJitHelperStatus JS_JitCatchException(")); + assert!(quickjs.contains("b->func_kind != JS_FUNC_NORMAL ?")); assert!(jit_header.contains("JSJitPropertyLayout property_layout;")); assert!(quickjs.contains("uint64_t jit_shape_generation;")); assert!(!quickjs.contains("QJSJIT_SHAPE_HASH")); @@ -255,7 +261,10 @@ fn bundled_jit_bindings_include_materialize_owner_tail() { for target in targets { let binding = fs::read_to_string(bindings.join(target)).unwrap(); assert!( - binding.contains("pub const QJSJIT_ABI_MINOR: u32 = 24;"), + binding.contains("pub const QJSJIT_ABI_MINOR: u32 = 25;") + && binding.contains("pub fn JS_JitThrowValue(") + && binding.contains("pub fn JS_JitThrowError(") + && binding.contains("pub fn JS_JitCatchException("), "{target}" ); assert!( @@ -314,7 +323,7 @@ fn bundled_jit_bindings_include_materialize_owner_tail() { } #[test] -fn bundled_wasm_jit_binding_matches_array_abi_1_24() { +fn bundled_wasm_jit_binding_matches_exception_abi_1_25() { let binding = fs::read_to_string( PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("src/bindings/wasm32-wasip1.rs"), ) @@ -362,8 +371,12 @@ fn bundled_wasm_jit_binding_matches_array_abi_1_24() { "pub const JS_JIT_FEEDBACK_ARRAY_IMMUTABLE: u32 = 4096;", "pub const JS_JIT_FEEDBACK_ARRAY_SHARED: u32 = 8192;", "pub const JS_JIT_FEEDBACK_ARRAY_INVALID_BACKING: u32 = 16384;", + // ABI 1.25: Tier 1 exception regions. + "pub fn JS_JitThrowValue(", + "pub fn JS_JitThrowError(", + "pub fn JS_JitCatchException(", // ABI 1.24: typed-array continuing guard. - "pub const QJSJIT_ABI_MINOR: u32 = 24;", + "pub const QJSJIT_ABI_MINOR: u32 = 25;", "pub const QJSJIT_ARRAY_API_VERSION: u32 = 1;", "pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1;", "JSJitArrayQueryStatus_JS_JIT_ARRAY_QUERY_INVALID: JSJitArrayQueryStatus = -1;", From c17019324dec1d99f61ed68cb571b0b797ea0544 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:10:46 +0800 Subject: [PATCH 21/54] perf(jit): carry typed/packed loop metadata and fold literal tags in Tier 2 Bounds-check elimination and metadata hoisting for packed Array, Int32Array and Float64Array loops: - Fold literal-constant guard conditions (iconst tags through icmp/band/ bor/extends) so representation checks on hoisted Int32 lengths, loaded typed elements and proven induction variables emit no branch; compare literal-Int32 operands with integer icmp; branch directly on literal Bool/Int32 conditions. - Emit only the statically known storage kind for cached element loads and drop the redundant receiver tag check on the cached path. - Seal IR blocks whose predecessors are all lowered and local guard diamonds, and restate literal tags after the amortized poll join, so variable reads keep dominating literal definitions. - Track guarded element metadata per frame-slot provenance instead of one slot, carry loop-hoisted sources across in-loop diamonds when every incoming edge still holds the identical hoisted source, and hoist storage-only metadata for typed candidates without a length read (polls still revalidate data/count; bounds checks stay). - Treat numeric push_const sites as non-reentrant pure literals for array plans, amortized polls and representation proofs. - Restate a hoisted, never-rebound argument receiver's object tag at the loop header. - De-race automatic_call_heavy_promotes_the_direct_edge_caller by waiting for the leaf's five paced profitability retries. Co-Authored-By: Claude Opus 5.5 (1M context) --- jit/src/compiler/optimized.rs | 719 +++++++++++++++++++--- jit/src/compiler/optimized/array_cache.rs | 77 ++- jit/src/ir/scalar.rs | 57 +- jit/tests/array_bce.rs | 263 ++++++++ jit/tests/optimized.rs | 6 + 5 files changed, 1036 insertions(+), 86 deletions(-) create mode 100644 jit/tests/array_bce.rs diff --git a/jit/src/compiler/optimized.rs b/jit/src/compiler/optimized.rs index afc40c273..00067b4f3 100644 --- a/jit/src/compiler/optimized.rs +++ b/jit/src/compiler/optimized.rs @@ -754,6 +754,60 @@ struct GuardedElementSource { /// `exact_length` separately records whether it guarded the intrinsic /// lookup. Both facts are revalidated after polls. typed_mode: Option, + /// Compile-time value of `kind` when every path that established this + /// source produced one storage kind (0 packed, 1 Int32, 2 Float64). + /// Cached accesses then emit only that kind's load instead of a runtime + /// dispatch on `kind`. + static_kind: Option, +} + +/// Guarded element metadata currently valid on the lowering path, at most one +/// entry per frame-slot provenance. Every consumer that may run a helper, +/// reenter, or rebind a frame slot clears the whole set; element loads and +/// typed stores are non-reentrant leaves that keep the other entries. +#[derive(Clone, Default)] +struct ElementSources { + entries: Vec, +} + +impl GuardedElementSource { + /// The same guarded metadata: identical provenance, SSA values and facts. + fn same(&self, other: &Self) -> bool { + self.provenance == other.provenance + && self.block_pc == other.block_pc + && self.data == other.data + && self.count == other.count + && self.kind == other.kind + && self.exact_length == other.exact_length + && self.typed_mode == other.typed_mode + && self.static_kind == other.static_kind + } +} + +impl ElementSources { + fn clear(&mut self) { + self.entries.clear(); + } + + fn retain(&mut self, keep: impl FnMut(&GuardedElementSource) -> bool) { + self.entries.retain(keep); + } + + fn insert(&mut self, source: GuardedElementSource) { + self.entries + .retain(|existing| existing.provenance != source.provenance); + self.entries.push(source); + } + + /// Removes and returns the source for `provenance`, for a consumer that + /// takes the historical single-slot `&mut Option` interface. + fn take(&mut self, provenance: OptProvenance) -> Option { + let index = self + .entries + .iter() + .position(|source| source.provenance == provenance)?; + Some(self.entries.remove(index)) + } } #[derive(Clone, Copy, Default)] @@ -1172,7 +1226,7 @@ fn lower_optimized_machine( && ir.scalar_graph().frame_inlined_calls() == 0 && ir.blocks().iter().any(|block| block.is_loop_header()); let scalar_numeric = if side_path.is_none() { - ir.scalar_graph().proven_numeric_values( + ir.scalar_graph().proven_numeric_values_with_constants( ir.nodes(), |index| { usize::from(index) < usize::from(shape.arguments()) @@ -1186,6 +1240,26 @@ fn lower_optimized_machine( ) }, int32_loop, + // Tier 2 lowers `push_const*` only for numeric constant-pool + // entries, as literal Int32/Float64 immediates, and otherwise + // rejects the function. + |id| { + let node = ir.nodes().get(id as usize)?; + let crate::ir::OptimizedNodeKind::Bytecode { opcode } = node.kind() else { + return None; + }; + let index = match opcode.as_ref() { + "push_const8" => u32::from(*node.bytes().get(1)?), + "push_const" => u32::from_le_bytes(node.bytes().get(1..5)?.try_into().ok()?), + _ => return None, + }; + let constant = specialization.numeric_constants.get(&index)?; + match i32::try_from(constant.tag).ok()? { + qjs::JS_TAG_INT => Some(crate::ir::ScalarNumericMode::Int32), + qjs::JS_TAG_FLOAT64 => Some(crate::ir::ScalarNumericMode::Float64), + _ => None, + } + }, ir.scalar_graph().values().len().saturating_mul(128), ) } else { @@ -1452,6 +1526,27 @@ fn lower_optimized_machine( matches!(ir.nodes()[id as usize].kind(), crate::ir::OptimizedNodeKind::Bytecode { opcode } if opcode.as_ref() == "to_propkey") }; + // Tier 2 lowers `push_const*` only for numeric constant-pool entries (as + // literal immediates) and rejects every other constant at compile time, + // so an admitted site can neither reenter nor touch the heap. + let numeric_push_const = |id: u32| { + let Some(node) = ir.nodes().get(id as usize) else { + return false; + }; + let crate::ir::OptimizedNodeKind::Bytecode { opcode } = node.kind() else { + return false; + }; + let index = match opcode.as_ref() { + "push_const8" => node.bytes().get(1).map(|&index| u32::from(index)), + "push_const" => node + .bytes() + .get(1..5) + .and_then(|bytes| bytes.try_into().ok()) + .map(u32::from_le_bytes), + _ => None, + }; + index.is_some_and(|index| specialization.numeric_constants.contains_key(&index)) + }; let array_loops = crate::ir::LoopAnalysis::analyze(ir, array_work); let assumed_array_sites = ir .nodes() @@ -1472,6 +1567,7 @@ fn lower_optimized_machine( |id| { assumed_array_sites.contains(&id) || guarded_propkey(id) + || numeric_push_const(id) || ir.nodes().get(id as usize).is_some_and(|node| { matches!( node.kind(), @@ -1514,6 +1610,7 @@ fn lower_optimized_machine( (matches!(opcode.as_ref(), "get_array_el" | "put_array_el") && guarded_sites.contains(&id)) || guarded_propkey(id) + || numeric_push_const(id) || (matches!( opcode.as_ref(), "add" @@ -1573,6 +1670,7 @@ fn lower_optimized_machine( |id| { guarded_sites.contains(&id) || guarded_propkey(id) + || numeric_push_const(id) || ir.nodes().get(id as usize).is_some_and(|node| { matches!( node.kind(), @@ -1609,6 +1707,38 @@ fn lower_optimized_machine( } plan }; + // Hoisted receivers whose argument slot no node of the loop rebinds. Only + // these may have their object tag restated at the loop header. + let stable_hoisted_receivers = array_plan + .hoists() + .iter() + .filter_map(|hoist| { + let argument = array_plan.candidates().get(hoist.candidate)?.argument; + let natural_loop = array_loops + .loops() + .iter() + .find(|natural_loop| natural_loop.header() == hoist.header)?; + let rebinds = ir.nodes().iter().any(|node| { + if !array_loops + .block_for_node(node.id()) + .is_some_and(|block| natural_loop.contains_block(block)) + { + return false; + } + let slot = crate::ir::FrameSlot::Argument(argument); + ir.scalar_graph().effect_for_node(node.id()) + == crate::ir::ScalarHeapEffect::FrameWrite(slot) + || ir + .scalar_graph() + .frame_definitions_for_node(node.id()) + .iter() + .any(|(defined, _)| *defined == slot) + || matches!(node.kind(), crate::ir::OptimizedNodeKind::Bytecode { opcode } + if opcode.starts_with("put_arg") || opcode.starts_with("set_arg")) + }); + (!rebinds).then_some((hoist.header, usize::from(argument))) + }) + .collect::>(); // Array feedback only proposes candidates. Poll amortization is enabled // after the concrete plan has proved that every heap site on the native // path is a guarded leaf-or-exit operation. @@ -1632,6 +1762,9 @@ fn lower_optimized_machine( { return true; } + if numeric_push_const(id) || guarded_propkey(id) { + return true; + } (matches!( opcode.as_ref(), "add" @@ -1779,9 +1912,13 @@ fn lower_optimized_machine( .map(|value| (value, alloc())) .collect::>(); let mut stack_provenance = vec![OptProvenance::Unknown; stack_slots]; - let mut guarded_element_source: Option = None; + let mut guarded_element_source = ElementSources::default(); let mut hoisted_element_sources = - std::collections::BTreeMap::::new(); + std::collections::BTreeMap::>::new(); + // Element sources still valid after each lowered IR block's final node + // (hence on every outgoing edge). + let mut exit_element_sources = + std::collections::BTreeMap::>::new(); // The scalar-region proof excludes heap effects and requires every // local definition to be primitive (or a lexical sentinel). No local // owns a heap value, so poll/deopt boundaries can publish SSA locals @@ -1920,18 +2057,93 @@ fn lower_optimized_machine( &[], ); for block in ir.blocks() { - guarded_element_source = guarded_element_source.filter(|source| { + guarded_element_source.retain(|source| { source.block_pc == block.start_pc() || (matches!(source.provenance, OptProvenance::Argument(_)) && predecessors .get(&block.start_pc()) .is_some_and(|incoming| incoming.as_slice() == [source.block_pc])) }); - if let Some(source) = hoisted_element_sources.get(&block.start_pc()).copied() { - guarded_element_source = Some(source); + for source in hoisted_element_sources + .get(&block.start_pc()) + .into_iter() + .flatten() + { + guarded_element_source.insert(*source); + } + let hoisted_receivers = hoisted_element_sources + .get(&block.start_pc()) + .into_iter() + .flatten() + .filter_map(|source| match source.provenance { + OptProvenance::Argument(argument) + if stable_hoisted_receivers.contains(&(block.start_pc(), argument)) => + { + Some(argument) + } + _ => None, + }) + .collect::>(); + // A loop-hoisted source reaching this block's entry along every + // incoming edge (all already lowered) stays valid here: each edge + // carries the identical preheader-defined values, and the preheader + // dominates every block reached only through such edges. This + // carries loop metadata across in-loop diamonds such as the + // nullish-destination check of `a[i] = v`. + if let Some(incoming) = predecessors.get(&block.start_pc()).filter(|incoming| { + !incoming.is_empty() + && incoming + .iter() + .all(|&predecessor| predecessor < block.start_pc()) + }) { + let inherited = exit_element_sources + .get(&incoming[0]) + .into_iter() + .flatten() + .filter(|source| { + hoisted_element_sources + .values() + .flatten() + .any(|hoisted| hoisted.same(source)) + && incoming[1..].iter().all(|predecessor| { + exit_element_sources + .get(predecessor) + .is_some_and(|exits| exits.iter().any(|exit| exit.same(source))) + }) + }) + .copied() + .collect::>(); + for source in inherited { + guarded_element_source.insert(source); + } } let clif_block = blocks[&block.start_pc()]; builder.switch_to_block(clif_block); + // IR blocks are lowered in bytecode order and are entered only by + // IR edges (the function entry edge targets pc 0 and is emitted + // before this loop). When every predecessor precedes this block, + // no later edge can target it, so it can be sealed now. Loop + // headers keep their backedges open until `seal_all_blocks`. + if predecessors.get(&block.start_pc()).is_none_or(|incoming| { + incoming + .iter() + .all(|&predecessor| predecessor < block.start_pc()) + }) { + builder.seal_block(clif_block); + } + // Each such hoisted receiver is an argument slot no loop node + // rebinds, and the dominating preheader guard proved it holds an + // object. Its variable therefore holds that object on every edge + // into the header. + for &argument in &hoisted_receivers { + let vars = *arguments + .get(argument) + .ok_or(CompileFailure::InvalidArtifact)?; + let tag = builder + .ins() + .iconst(types::I64, i64::from(qjs::JS_TAG_OBJECT)); + builder.def_var(vars.tag, tag); + } let mut depth = usize::from(block.stack_depth()); stack_provenance.fill(OptProvenance::Unknown); let entry_provenance = cfg_entry_provenance @@ -2021,7 +2233,7 @@ fn lower_optimized_machine( && array_plan.invalidates_before(node.id()) && !retains_hoisted_metadata { - guarded_element_source = None; + guarded_element_source.clear(); } if matches!(node.kind(), crate::ir::OptimizedNodeKind::Bytecode { opcode } if matches!(opcode.as_ref(), "if_false8" | "if_true8" | "if_false" | "if_true" | "goto" | "goto8" | "goto16")) @@ -2031,6 +2243,22 @@ fn lower_optimized_machine( match node.kind() { crate::ir::OptimizedNodeKind::GuardNumeric { guard, mid_loop } => { if *mid_loop && amortized_poll { + // The amortized poll's cold path never redefines + // a frame variable (it only publishes, polls and + // revalidates heap metadata), so each variable + // leaves the join holding its pre-poll value. + // Re-state literal tags after the join; otherwise + // they hide behind a merge parameter and every + // later representation check stays in the loop. + let literal_tags = arguments + .iter() + .chain(&locals) + .chain(stack.iter().take(depth)) + .filter_map(|vars| { + let tag = builder.use_var(vars.tag); + opt_known_const(&builder, tag).map(|known| (*vars, known)) + }) + .collect::>(); emit_opt_amortized_poll( &mut builder, frame, @@ -2079,8 +2307,10 @@ fn lower_optimized_machine( node.pc(), *guard, )?; - if let Some(source) = - hoisted_element_sources.get(&block.start_pc()).copied() + for &source in hoisted_element_sources + .get(&block.start_pc()) + .into_iter() + .flatten() { emit_opt_packed_loop_revalidate( builder, @@ -2105,6 +2335,10 @@ fn lower_optimized_machine( Ok(()) }, )?; + for (vars, known) in literal_tags { + let tag = builder.ins().iconst(types::I64, known); + builder.def_var(vars.tag, tag); + } } else if *mid_loop { emit_opt_poll( &mut builder, @@ -2137,8 +2371,10 @@ fn lower_optimized_machine( // amortized cold edge. A typed store can keep its // numeric locals unproven, selecting this path; // cached pointers/length must still be revalidated. - if let Some(source) = - hoisted_element_sources.get(&block.start_pc()).copied() + for &source in hoisted_element_sources + .get(&block.start_pc()) + .into_iter() + .flatten() { emit_opt_packed_loop_revalidate( &mut builder, @@ -2171,11 +2407,11 @@ fn lower_optimized_machine( if array_plan.is_empty() && node.effect() == crate::ir::OptimizedEffect::Reentrant { - guarded_element_source = None; + guarded_element_source.clear(); } match name { "set_loc_uninitialized" => { - guarded_element_source = None; + guarded_element_source.clear(); let index = opt_u16(node.bytes())?; if owned_locals[index] { emit_opt_free_local_slot(&mut builder, &env, index)?; @@ -2307,7 +2543,7 @@ fn lower_optimized_machine( n if opt_index(n, node.bytes(), "put_loc")?.is_some() || matches!(n, "put_loc_check" | "put_loc_check_init") => { - guarded_element_source = None; + guarded_element_source.clear(); let index = opt_index(n, node.bytes(), "put_loc")? .map_or_else(|| opt_u16(node.bytes()), Ok)?; let source_owned = depth @@ -2564,6 +2800,11 @@ fn lower_optimized_machine( &stack_provenance, depth.saturating_sub(2)..depth, )?; + let mut element_source = guarded_element_source.take( + stack_provenance[depth + .checked_sub(2) + .ok_or(CompileFailure::InvalidArtifact)?], + ); depth = emit_opt_element_get( &mut builder, frame, @@ -2583,17 +2824,27 @@ fn lower_optimized_machine( layout, element_layout, block.start_pc(), - &mut guarded_element_source, + &mut element_source, expected_mode, planned_access .is_some_and(|access| access.bounds_covered_by_hoist), )?; + // A guarded load deopts on every miss and + // never reenters: other sources stay valid. + if let Some(source) = element_source { + guarded_element_source.insert(source); + } } "get_length" => { opt_reject_owned( &stack_provenance, depth.saturating_sub(1)..depth, )?; + let mut element_source = guarded_element_source.take( + stack_provenance[depth + .checked_sub(1) + .ok_or(CompileFailure::InvalidArtifact)?], + ); depth = emit_opt_array_length( &mut builder, &env, @@ -2601,8 +2852,15 @@ fn lower_optimized_machine( depth, node.pc(), element_layout, - &mut guarded_element_source, + &mut element_source, )?; + // Only the cached exact-length path keeps its + // source; the generic lookup may run a getter, + // so it invalidates every source. + match element_source { + Some(source) => guarded_element_source.insert(source), + None => guarded_element_source.clear(), + } } "put_array_el" => { opt_reject_owned( @@ -2629,6 +2887,8 @@ fn lower_optimized_machine( { return Err(CompileFailure::InvalidArtifact); } + let mut element_source = + guarded_element_source.take(source_provenance); depth = emit_opt_typed_store( &mut builder, &env, @@ -2640,10 +2900,17 @@ fn lower_optimized_machine( block.start_pc(), candidate.mode, array_query, - &mut guarded_element_source, + &mut element_source, )?; + // A fixed-backing typed store neither + // reallocates nor reenters. + if let Some(source) = element_source { + guarded_element_source.insert(source); + } continue; } + let mut element_source = + guarded_element_source.take(source_provenance); depth = emit_opt_element_put( &mut builder, frame, @@ -2664,8 +2931,14 @@ fn lower_optimized_machine( element_layout, block.start_pc(), source_provenance, - &mut guarded_element_source, + &mut element_source, )?; + // Conservatively drop every other source + // around a generic element store. + guarded_element_source.clear(); + if let Some(source) = element_source { + guarded_element_source.insert(source); + } } "call" | "call0" | "call1" | "call2" | "call3" | "call_method" => { // Native callees (Math.max, String, ...) never @@ -2959,6 +3232,12 @@ fn lower_optimized_machine( } builder.ins().brif(failure, deopt, &[], pass, &[result]); } + // Every edge into this local diamond is + // declared; sealing keeps later reads of + // unchanged variables (and their literal + // tags) direct. + builder.seal_block(deopt); + builder.seal_block(pass); builder.switch_to_block(deopt); emit_opt_deopt( &mut builder, @@ -3121,7 +3400,7 @@ fn lower_optimized_machine( stack_provenance[index] = OptProvenance::ImmediatePrimitive; } "inc_loc" | "dec_loc" => { - guarded_element_source = None; + guarded_element_source.clear(); let index = opt_u8(node.bytes())?; let (old, _) = opt_prepare_scalar_operation( &mut builder, @@ -3168,7 +3447,7 @@ fn lower_optimized_machine( ); } "add_loc" => { - guarded_element_source = None; + guarded_element_source.clear(); let index = opt_u8(node.bytes())?; let rhs_index = depth .checked_sub(1) @@ -3234,7 +3513,7 @@ fn lower_optimized_machine( } } n if opt_index(n, node.bytes(), "put_arg")?.is_some() => { - guarded_element_source = None; + guarded_element_source.clear(); opt_reject_owned( &stack_provenance, depth.saturating_sub(1)..depth, @@ -3264,7 +3543,7 @@ fn lower_optimized_machine( ); } n if opt_index(n, node.bytes(), "set_arg")?.is_some() => { - guarded_element_source = None; + guarded_element_source.clear(); opt_reject_owned( &stack_provenance, depth.saturating_sub(1)..depth, @@ -3296,7 +3575,7 @@ fn lower_optimized_machine( ); } n if opt_index(n, node.bytes(), "set_loc")?.is_some() => { - guarded_element_source = None; + guarded_element_source.clear(); opt_reject_owned( &stack_provenance, depth.saturating_sub(1)..depth, @@ -3395,11 +3674,22 @@ fn lower_optimized_machine( .comparison(*node_id) .ok_or(CompileFailure::InvalidArtifact)?; use crate::ir::ScalarCompareOp; + // A literal Int32 tag (for example a + // hoisted exact array length or a checked + // Int32 result) is as strong as the graph + // proof for this one lowering. let integer_compare = int32_loop - || [lhs, rhs].iter().all(|value| { - scalar_numeric.get(value.index()).copied().flatten() - == Some(crate::ir::ScalarNumericMode::Int32) - }); + || [(lhs, lhs_pair), (rhs, rhs_pair)].iter().all( + |(value, pair)| { + scalar_numeric.get(value.index()).copied().flatten() + == Some(crate::ir::ScalarNumericMode::Int32) + || opt_tag_known( + &builder, + pair.tag, + qjs::JS_TAG_INT, + ) + }, + ); let value = if integer_compare { let cc = match operation { ScalarCompareOp::LessThan => IntCC::SignedLessThan, @@ -3504,16 +3794,12 @@ fn lower_optimized_machine( condition.payload, 0, ); - let target = *blocks - .get( - &node - .branch_target() - .ok_or(CompileFailure::InvalidArtifact)?, - ) - .ok_or(CompileFailure::InvalidArtifact)?; - let fallthrough = *blocks - .get(&next_block_pc(ir, block.start_pc())?) - .ok_or(CompileFailure::InvalidArtifact)?; + let target = opt_ir_edge(&blocks, block, node.branch_target())?; + let fallthrough = opt_ir_edge( + &blocks, + block, + Some(next_block_pc(ir, block.start_pc())?), + )?; if name.starts_with("if_false") { builder.ins().brif(truth, fallthrough, &[], target, &[]); } else { @@ -3522,6 +3808,40 @@ fn lower_optimized_machine( terminated = true; continue; } + if opt_tag_known(&builder, condition.tag, qjs::JS_TAG_BOOL) + || opt_tag_known(&builder, condition.tag, qjs::JS_TAG_INT) + { + // A literal Bool/Int32 tag: ToBoolean is + // exactly `int32 payload != 0`, and the + // representation deopt edge is unreachable. + let integer = + builder.ins().ireduce(types::I32, condition.payload); + let truth = builder.ins().icmp_imm(IntCC::NotEqual, integer, 0); + let target = opt_ir_edge(&blocks, block, node.branch_target())?; + let fallthrough = opt_ir_edge( + &blocks, + block, + Some(next_block_pc(ir, block.start_pc())?), + )?; + if let Some(known) = opt_known_const(&builder, truth) { + // Folded from literals: only one edge + // is ever taken. The other IR block + // simply loses this predecessor. + let taken = if (known != 0) == name.starts_with("if_false") + { + fallthrough + } else { + target + }; + builder.ins().jump(taken, &[]); + } else if name.starts_with("if_false") { + builder.ins().brif(truth, fallthrough, &[], target, &[]); + } else { + builder.ins().brif(truth, target, &[], fallthrough, &[]); + } + terminated = true; + continue; + } let is_int = builder.ins().icmp_imm( IntCC::Equal, condition.tag, @@ -3617,16 +3937,12 @@ fn lower_optimized_machine( builder.ins().select(is_float, float_truth, integer_truth); let false_value = builder.ins().iconst(types::I8, 0); let truth = builder.ins().select(empty, false_value, numeric_truth); - let target = *blocks - .get( - &node - .branch_target() - .ok_or(CompileFailure::InvalidArtifact)?, - ) - .ok_or(CompileFailure::InvalidArtifact)?; - let fallthrough = *blocks - .get(&next_block_pc(ir, block.start_pc())?) - .ok_or(CompileFailure::InvalidArtifact)?; + let target = opt_ir_edge(&blocks, block, node.branch_target())?; + let fallthrough = opt_ir_edge( + &blocks, + block, + Some(next_block_pc(ir, block.start_pc())?), + )?; if name.starts_with("if_false") { builder.ins().brif(truth, fallthrough, &[], target, &[]); } else { @@ -3635,13 +3951,7 @@ fn lower_optimized_machine( terminated = true; } "goto" | "goto8" | "goto16" => { - let target = *blocks - .get( - &node - .branch_target() - .ok_or(CompileFailure::InvalidArtifact)?, - ) - .ok_or(CompileFailure::InvalidArtifact)?; + let target = opt_ir_edge(&blocks, block, node.branch_target())?; builder.ins().jump(target, &[]); terminated = true; } @@ -3778,8 +4088,11 @@ fn lower_optimized_machine( array_query, ir, )?; - builder.ins().jump(blocks[&next], &[]); + builder + .ins() + .jump(opt_ir_edge(&blocks, block, Some(next))?, &[]); } + exit_element_sources.insert(block.start_pc(), guarded_element_source.entries.clone()); } builder.seal_all_blocks(); builder.finalize(); @@ -4745,8 +5058,8 @@ fn emit_opt_deopt( /// Branches to a fresh pass block when `condition` holds and otherwise to an /// exact deoptimization exit; the builder is left positioned on the pass -/// block. The returned deopt block may be reused by later checks of the same -/// instruction. +/// block. Both new blocks are sealed immediately, so no later branch may +/// target the exit; a condition that is a known-true constant emits nothing. fn emit_opt_guard_branch( builder: &mut cranelift_frontend::FunctionBuilder<'_>, env: &OptEnv<'_>, @@ -4755,18 +5068,136 @@ fn emit_opt_guard_branch( pc: u32, guard: u32, condition: cranelift_codegen::ir::Value, -) -> Result { +) -> Result<(), CompileFailure> { use cranelift_codegen::ir::InstBuilder; + // A condition folded from literal constants holds on every execution: + // the deoptimization edge is unreachable, so emit no branch at all. + if opt_known_const(builder, condition).is_some_and(|known| known != 0) { + return Ok(()); + } let pass = builder.create_block(); let deopt = builder.create_block(); // Deoptimization exits are cold: keep them out of the hot fallthrough // path so a guarded loop body stays straight-line machine code. builder.set_cold_block(deopt); builder.ins().brif(condition, pass, &[], deopt, &[]); + // Both blocks are fresh and this branch is their only predecessor. + // Sealing lets variable reads resolve to dominating definitions + // (including literal tags) instead of placeholder block parameters. + builder.seal_block(deopt); + builder.seal_block(pass); builder.switch_to_block(deopt); emit_opt_deopt(builder, env, provenance, depth, pc, guard)?; builder.switch_to_block(pass); - Ok(deopt) + Ok(()) +} + +/// Exact compile-time value of an integer SSA value already emitted into the +/// current function: an `iconst`, or an `icmp_imm`/`icmp` equality, `band`, +/// `bor`, `uextend`, `sextend` or `ireduce` of such values. Every other +/// definition (block parameters, loads, calls) is unknown. Because this folds only instructions whose operands are literal +/// constants, a condition reported as known holds on every execution, so a +/// guard on it can be omitted without changing any taken path. Results are +/// sign-extended from the value's type width. +fn opt_known_const( + builder: &cranelift_frontend::FunctionBuilder<'_>, + value: cranelift_codegen::ir::Value, +) -> Option { + fn fold( + dfg: &cranelift_codegen::ir::DataFlowGraph, + value: cranelift_codegen::ir::Value, + depth: u8, + ) -> Option { + use cranelift_codegen::ir::{condcodes::IntCC, InstructionData, Opcode, ValueDef}; + let bits = dfg.value_type(value).bits(); + if !(1..=64).contains(&bits) || !dfg.value_type(value).is_int() { + return None; + } + let normalize = |raw: i64| -> i64 { + if bits >= 64 { + raw + } else { + let shift = 64 - bits; + (raw << shift) >> shift + } + }; + let ValueDef::Result(inst, 0) = dfg.value_def(value) else { + return None; + }; + match dfg.insts[inst] { + InstructionData::UnaryImm { + opcode: Opcode::Iconst, + imm, + } => Some(normalize(imm.bits())), + InstructionData::IntCompareImm { + opcode: Opcode::IcmpImm, + cond, + arg, + imm, + } if depth > 0 => { + let arg_bits = dfg.value_type(arg).bits(); + let lhs = fold(dfg, arg, depth - 1)?; + let rhs = if arg_bits >= 64 { + imm.bits() + } else { + let shift = 64 - arg_bits; + (imm.bits() << shift) >> shift + }; + match cond { + IntCC::Equal => Some(i64::from(lhs == rhs)), + IntCC::NotEqual => Some(i64::from(lhs != rhs)), + _ => None, + } + } + InstructionData::Unary { + opcode: Opcode::Uextend, + arg, + } if depth > 0 => { + let arg_bits = dfg.value_type(arg).bits(); + let known = fold(dfg, arg, depth - 1)?; + Some(if arg_bits >= 64 { + known + } else { + known & ((1_i64 << arg_bits) - 1) + }) + } + InstructionData::Unary { + opcode: Opcode::Sextend | Opcode::Ireduce, + arg, + } if depth > 0 => fold(dfg, arg, depth - 1).map(normalize), + InstructionData::IntCompare { + opcode: Opcode::Icmp, + cond: cond @ (IntCC::Equal | IntCC::NotEqual), + args: [lhs, rhs], + } if depth > 0 => { + let equal = fold(dfg, lhs, depth - 1)? == fold(dfg, rhs, depth - 1)?; + Some(i64::from(if cond == IntCC::Equal { equal } else { !equal })) + } + InstructionData::Binary { + opcode: opcode @ (Opcode::Band | Opcode::Bor), + args: [lhs, rhs], + } if depth > 0 => { + let lhs = fold(dfg, lhs, depth - 1)?; + let rhs = fold(dfg, rhs, depth - 1)?; + Some(normalize(if opcode == Opcode::Band { + lhs & rhs + } else { + lhs | rhs + })) + } + _ => None, + } + } + fold(&builder.func.dfg, value, 8) +} + +/// True when `pair.tag` is a compile-time constant equal to `expected`. +fn opt_tag_known( + builder: &cranelift_frontend::FunctionBuilder<'_>, + tag: cranelift_codegen::ir::Value, + expected: i32, +) -> bool { + opt_known_const(builder, tag) == Some(i64::from(expected)) } fn opt_tag_is( @@ -5238,6 +5669,11 @@ fn emit_opt_guarded_propkey( .checked_sub(1) .ok_or(CompileFailure::InvalidArtifact)?; let value = opt_use(builder, stack[index]); + if opt_tag_known(builder, value.tag, qjs::JS_TAG_INT) { + // A literal Int32 key is already canonical: ToPropertyKey is the + // identity and the exit is unreachable. + return Ok(depth); + } let int = builder .ins() .icmp_imm(IntCC::Equal, value.tag, i64::from(qjs::JS_TAG_INT)); @@ -5252,6 +5688,8 @@ fn emit_opt_guarded_propkey( let continuation = builder.create_block(); let deopt = builder.create_block(); builder.ins().brif(valid, continuation, &[], deopt, &[]); + builder.seal_block(continuation); + builder.seal_block(deopt); builder.switch_to_block(deopt); for (slot, vars) in arguments.iter().enumerate() { let current = opt_use(builder, *vars); @@ -5453,6 +5891,7 @@ fn emit_opt_packed_metadata_guard( kind: builder.ins().iconst(types::I8, 0), exact_length: true, typed_mode: None, + static_kind: Some(0), }) } @@ -5569,6 +6008,7 @@ fn emit_opt_typed_metadata_guard( kind: builder.ins().iconst(types::I8, kind), exact_length: needs_length, typed_mode: Some(mode), + static_kind: Some(kind), }) } @@ -5643,7 +6083,7 @@ fn emit_opt_array_loop_hoists( plan: &array_cache::ArrayPlan, preheader: u32, successor: u32, - sources: &mut std::collections::BTreeMap, + sources: &mut std::collections::BTreeMap>, element_layout: crate::abi::ElementLayout, array_query: usize, ir: &OptimizedIr, @@ -5707,14 +6147,21 @@ fn emit_opt_array_loop_hoists( hoist.header, candidate.mode, array_query, - true, + // Only a hoist that owns the loop's length read queries + // (and so certifies) the intrinsic length lookup. + hoist.length.is_some(), )? } crate::runtime::ArrayMode::Generic => return Err(CompileFailure::InvalidArtifact), }; - if sources.insert(hoist.header, source).is_some() { + let hoisted = sources.entry(hoist.header).or_default(); + if hoisted + .iter() + .any(|existing| existing.provenance == source.provenance) + { return Err(CompileFailure::InvalidArtifact); } + hoisted.push(source); } Ok(()) } @@ -5954,21 +6401,109 @@ fn emit_opt_element_get( OptProvenance::Argument(_) | OptProvenance::Local(_) ) }); + let expected_kind = expected_mode.map(|mode| match mode { + crate::runtime::ArrayMode::Packed => 0, + crate::runtime::ArrayMode::Int32 => 1, + crate::runtime::ArrayMode::Float64 => 2, + // A generic site is never admitted by ArrayPlan. Keep this + // fail-closed if that invariant changes. + crate::runtime::ArrayMode::Generic => -1, + }); + // A cached source whose storage kind is a compile-time constant needs no + // runtime kind dispatch. Its receiver was tag/class/storage guarded when + // the source was established for this exact frame slot, and every frame + // write or unadmitted effect since then invalidated it, so only the key + // representation (unless it is a literal Int32 tag), the bounds check + // (unless range analysis covered it) and the per-element packed value + // check remain. A statically mismatched expected mode keeps the dynamic + // path below, whose kind guard deopts. + let static_cached = cached.and_then(|source| { + let kind = source.static_kind?; + ((0..=2).contains(&kind) && expected_kind.is_none_or(|expected| expected == kind)) + .then_some((source, kind)) + }); + let mut static_result = None; + if let Some((source, kind)) = static_cached { + // A literal Int32 key needs no representation exit. + if !opt_tag_known(builder, key.tag, qjs::JS_TAG_INT) { + let key_checked = builder.create_block(); + builder.ins().brif(key_ok, key_checked, &[], deopt, &[]); + builder.seal_block(key_checked); + builder.switch_to_block(key_checked); + } + let index = builder.ins().ireduce(types::I32, key.payload); + if !bounds_covered_by_hoist { + let in_bounds = builder + .ins() + .icmp(IntCC::UnsignedLessThan, index, source.count); + let bounded = builder.create_block(); + builder.ins().brif(in_bounds, bounded, &[], deopt, &[]); + builder.seal_block(bounded); + builder.switch_to_block(bounded); + } + let result = match kind { + 0 => { + let address = element_address::emit(builder, source.data, index, 16, pointer_type); + let payload = builder.ins().load(types::I64, MemFlags::new(), address, 0); + let tag = + builder + .ins() + .load(types::I64, MemFlags::new(), address, layout.value_tag); + let primitive = builder + .ins() + .icmp_imm(IntCC::SignedGreaterThanOrEqual, tag, 0); + let loaded = builder.create_block(); + builder.ins().brif(primitive, loaded, &[], deopt, &[]); + builder.seal_block(loaded); + builder.switch_to_block(loaded); + OptPair { payload, tag } + } + 1 => { + let address = element_address::emit(builder, source.data, index, 4, pointer_type); + let value = builder.ins().load(types::I32, MemFlags::new(), address, 0); + OptPair { + payload: builder.ins().sextend(types::I64, value), + tag: builder.ins().iconst(types::I64, i64::from(qjs::JS_TAG_INT)), + } + } + _ => { + let address = element_address::emit(builder, source.data, index, 8, pointer_type); + let value = builder.ins().load(types::F64, MemFlags::new(), address, 0); + OptPair { + payload: builder.ins().bitcast(types::I64, MemFlags::new(), value), + tag: builder + .ins() + .iconst(types::I64, i64::from(qjs::JS_TAG_FLOAT64)), + } + } + }; + builder.ins().jump( + continuation, + &[ + result.payload, + result.tag, + source.data, + source.count, + source.kind, + ], + ); + // The continuation's only predecessor is this block, so these values + // dominate it; using them directly keeps literal tags visible to the + // consumers' representation checks. + static_result = Some(result); + } + let retained_source = cached; + let cached = cached.filter(|_| static_cached.is_none()); let cached_index = cached.map(|_| builder.create_block()); - builder - .ins() - .brif(tags_ok, cached_index.unwrap_or(direct), &[], deopt, &[]); + if static_cached.is_none() { + builder + .ins() + .brif(tags_ok, cached_index.unwrap_or(direct), &[], deopt, &[]); + } if let (Some(source), Some(cached_index)) = (cached, cached_index) { builder.switch_to_block(cached_index); - if let Some(expected_kind) = expected_mode.map(|mode| match mode { - crate::runtime::ArrayMode::Packed => 0, - crate::runtime::ArrayMode::Int32 => 1, - crate::runtime::ArrayMode::Float64 => 2, - // A generic site is never admitted by ArrayPlan. Keep this - // fail-closed if that invariant changes. - crate::runtime::ArrayMode::Generic => -1, - }) { + if let Some(expected_kind) = expected_kind { let mode_ok = builder .ins() .icmp_imm(IntCC::Equal, source.kind, expected_kind); @@ -6046,7 +6581,7 @@ fn emit_opt_element_get( ); } - if cached.is_none() { + if cached.is_none() && static_cached.is_none() { let classify = builder.create_block(); let packed = builder.create_block(); let int32 = builder.create_block(); @@ -6271,6 +6806,10 @@ fn emit_opt_element_get( .jump(continuation, &[value, tag, data, count, cached_kind]); } + // All edges into the shared exit and the continuation are declared once + // the arms above are emitted; nothing below branches to either again. + builder.seal_block(deopt); + builder.seal_block(continuation); builder.switch_to_block(deopt); for (index, vars) in arguments.iter().enumerate() { let value = opt_use(builder, *vars); @@ -6314,12 +6853,12 @@ fn emit_opt_element_get( ); builder.switch_to_block(continuation); - let result = OptPair { + let result = static_result.unwrap_or_else(|| OptPair { payload: builder.block_params(continuation)[0], tag: builder.block_params(continuation)[1], - }; + }); let params = builder.block_params(continuation); - *guarded_source = cached.or_else(|| { + *guarded_source = retained_source.or_else(|| { (expected_mode == Some(crate::runtime::ArrayMode::Packed)).then_some(GuardedElementSource { provenance: source_provenance, block_pc, @@ -6328,6 +6867,8 @@ fn emit_opt_element_get( kind: params[4], exact_length: false, typed_mode: None, + // Only the packed arm reaches the continuation for this mode. + static_kind: Some(0), }) }); opt_define(builder, stack[object_index], result); @@ -6740,6 +7281,7 @@ fn emit_opt_element_put( kind: params[2], exact_length: false, typed_mode: None, + static_kind: None, }); for provenance in &mut stack_provenance[object_index..depth] { *provenance = OptProvenance::Unknown; @@ -7316,6 +7858,24 @@ fn opt_own_stack_for_exit( } Ok(()) } +/// The CLIF block of an IR successor edge. Blocks are sealed as soon as all of +/// their IR predecessors have been lowered, so lowering may branch to an IR +/// block only along an edge the IR's successor lists already declare. +fn opt_ir_edge( + blocks: &std::collections::BTreeMap, + block: &crate::ir::OptimizedBlock, + target: Option, +) -> Result { + let target = target.ok_or(CompileFailure::InvalidArtifact)?; + if !block.successors().contains(&target) { + return Err(CompileFailure::InvalidArtifact); + } + blocks + .get(&target) + .copied() + .ok_or(CompileFailure::InvalidArtifact) +} + fn next_block_pc(ir: &OptimizedIr, pc: u32) -> Result { ir.blocks() .iter() @@ -8707,6 +9267,7 @@ fn emit_opt_amortized_poll( } let continuation = builder.create_block(); builder.ins().brif(due, poll, &[], continuation, &[]); + builder.seal_block(poll); builder.switch_to_block(poll); before_poll(builder); emit_opt_poll(builder, frame, sret, signature, pointer_type, layout, pc); @@ -8714,6 +9275,10 @@ fn emit_opt_amortized_poll( let reset = builder.ins().iconst(types::I64, 64); builder.ins().stack_store(reset, budget, 0); builder.ins().jump(continuation, &[]); + // Both edges into the fresh join are now declared. Sealing it lets + // values the poll path did not redefine (such as literal Int32 tags of + // proven induction variables) stay visible to the loop body. + builder.seal_block(continuation); builder.switch_to_block(continuation); Ok(()) } diff --git a/jit/src/compiler/optimized/array_cache.rs b/jit/src/compiler/optimized/array_cache.rs index 2f05fb17b..6b5856bf8 100644 --- a/jit/src/compiler/optimized/array_cache.rs +++ b/jit/src/compiler/optimized/array_cache.rs @@ -39,13 +39,21 @@ pub(super) struct ArrayAccessPlan { /// Packed arrays require `logical_length == dense_count` in this guard. That /// equality makes the SSA `bound` used by range analysis an exact storage /// bound and rejects sparse/extended arrays before entering native loop code. +/// +/// A typed-array candidate with element accesses but no `length` read in the +/// loop may still hoist its storage guard (`length == None`): fixed, +/// attached, nonshared backing cannot change without one of the reentrant +/// effects the plan already excludes from the loop, and every poll +/// revalidates `data`/`count`. Such a hoist removes the per-iteration +/// metadata query but never a bounds check. #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub(super) struct ArrayLoopHoist { pub candidate: usize, pub preheader: u32, pub header: u32, - pub bound: ScalarValueId, - pub length_node: u32, + /// The single observable length read whose exact result bounds the + /// covered loads: `(length_node, bound)`. + pub length: Option<(u32, ScalarValueId)>, } #[derive(Debug, Default)] @@ -327,7 +335,7 @@ impl ArrayPlan { work, ); let mut hoists = Vec::new(); - for candidate in 0..candidates.len() { + for (candidate, candidate_info) in candidates.iter().enumerate() { for natural_loop in loops.loops() { let Some(preheader) = natural_loop.preheader() else { continue; @@ -340,6 +348,23 @@ impl ArrayPlan { .then_some((node as u32, access.result?)) }); let Some((length_node, bound)) = lengths.next() else { + // No length read: a typed candidate may still carry its + // storage metadata across iterations; bounds checks stay. + let typed = + matches!(candidate_info.mode, ArrayMode::Int32 | ArrayMode::Float64); + let accessed = accesses.iter().flatten().any(|access| { + access.candidate == candidate + && access.loop_header == natural_loop.header() + && access.access != ArrayAccess::Length + }); + if typed && accessed { + hoists.push(ArrayLoopHoist { + candidate, + preheader, + header: natural_loop.header(), + length: None, + }); + } continue; }; // More than one observable length read does not identify one @@ -370,8 +395,7 @@ impl ArrayPlan { candidate, preheader, header: natural_loop.header(), - bound, - length_node, + length: Some((length_node, bound)), }); } } @@ -422,7 +446,10 @@ impl ArrayPlan { pub(super) fn guarded_leaf(&self, node: u32) -> bool { self.access(node).is_some_and(|access| match access.access { ArrayAccess::Load | ArrayAccess::Store => true, - ArrayAccess::Length => self.hoists.iter().any(|hoist| hoist.length_node == node), + ArrayAccess::Length => self + .hoists + .iter() + .any(|hoist| hoist.length.is_some_and(|(length, _)| length == node)), }) } pub(super) fn hoists(&self) -> &[ArrayLoopHoist] { @@ -687,9 +714,10 @@ mod tests { panic!("canonical packed traversal must have one legal metadata hoist: {plan:#?}"); }; assert_ne!(hoist.preheader, hoist.header); - let length = plan.access(hoist.length_node).unwrap(); + let (length_node, bound) = hoist.length.unwrap(); + let length = plan.access(length_node).unwrap(); assert_eq!(length.access, ArrayAccess::Length); - assert_eq!(length.result, Some(hoist.bound)); + assert_eq!(length.result, Some(bound)); let covered = ir .nodes() .iter() @@ -752,4 +780,37 @@ mod tests { ); } } + + #[test] + fn typed_loop_without_length_hoists_metadata_but_keeps_every_bounds_check() { + for mode in [ArrayMode::Int32, ArrayMode::Float64, ArrayMode::Packed] { + let (ir, feedback) = fixture( + "(function(a,n){let s=0;for(let i=0;i>(); + assert_eq!(loads.len(), 1, "{mode:?}: {plan:#?}"); + assert!( + loads[0].requires_index_guard && !loads[0].bounds_covered_by_hoist, + "an unrelated loop bound can never delete a bounds check: {mode:?}" + ); + if mode == ArrayMode::Packed { + // A packed guard would demand logical length == dense count, + // which an element-only loop never needed. + assert!(plan.hoists().is_empty(), "{plan:#?}"); + } else { + let [hoist] = plan.hoists() else { + panic!("typed element loop must hoist its storage metadata: {plan:#?}"); + }; + assert_eq!(hoist.length, None); + assert_ne!(hoist.preheader, hoist.header); + } + } + } } diff --git a/jit/src/ir/scalar.rs b/jit/src/ir/scalar.rs index 83532ffac..c4fe4f8fb 100644 --- a/jit/src/ir/scalar.rs +++ b/jit/src/ir/scalar.rs @@ -714,6 +714,7 @@ impl ScalarGraph { /// Representation facts valid after the caller-supplied entry guards. /// Poll aliases are enabled only when lowering preserves register bindings /// across its poll; arbitrary reentrant frame reads remain unknown. + #[cfg_attr(not(test), allow(dead_code))] pub(crate) fn proven_numeric_values( &self, nodes: &[OptimizedNode], @@ -721,7 +722,40 @@ impl ScalarGraph { preserve_poll_bindings: bool, work: usize, ) -> Vec> { - self.proven_numeric_values_with_preserved_frame_reads( + self.proven_numeric_values_with_constants( + nodes, + guarded_argument, + preserve_poll_bindings, + |_| None, + work, + ) + } + + /// As [`Self::proven_numeric_values`], additionally seeding the opaque + /// value pushed by node `constant(node)`: the caller certifies that its + /// lowering pushes a literal of exactly that representation (for example + /// a numeric constant-pool entry) or rejects compilation. + pub(crate) fn proven_numeric_values_with_constants( + &self, + nodes: &[OptimizedNode], + guarded_argument: impl Fn(u16) -> bool, + preserve_poll_bindings: bool, + constant: impl Fn(u32) -> Option, + work: usize, + ) -> Vec> { + let mut seeds = vec![None; self.values.len()]; + for node in nodes { + let Some(mode) = constant(node.id()) else { + continue; + }; + let Some(value) = self.value_for_node(node.id()) else { + continue; + }; + if matches!(self.values.get(value.index()), Some(ScalarValue::Opaque)) { + seeds[value.index()] = Some(mode); + } + } + self.proven_numeric_values_seeded( nodes, guarded_argument, |node| { @@ -730,6 +764,7 @@ impl ScalarGraph { matches!(node.kind(), OptimizedNodeKind::GuardNumeric { .. }) }) }, + &seeds, work, ) } @@ -739,11 +774,25 @@ impl ScalarGraph { /// revalidated poll, but never an arbitrary reentrant slow path. It only /// recovers the pre-operation identity of each frame slot: heap results do /// not become numeric, and arguments still require their own entry guards. + #[cfg_attr(not(test), allow(dead_code))] pub(crate) fn proven_numeric_values_with_preserved_frame_reads( &self, nodes: &[OptimizedNode], guarded_argument: impl Fn(u16) -> bool, preserve: impl Fn(u32) -> bool, + work: usize, + ) -> Vec> { + self.proven_numeric_values_seeded(nodes, guarded_argument, preserve, &[], work) + } + + /// `seeds[value]` gives the representation of an opaque value whose + /// producer the caller certified (see `proven_numeric_values_with_constants`). + fn proven_numeric_values_seeded( + &self, + nodes: &[OptimizedNode], + guarded_argument: impl Fn(u16) -> bool, + preserve: impl Fn(u32) -> bool, + seeds: &[Option], mut work: usize, ) -> Vec> { // Zero is not-yet-reached evidence, not an unknown JS type. Union is @@ -793,6 +842,12 @@ impl ScalarGraph { ScalarValue::FrameRead { node, slot } => self .preserved_frame_read_source(nodes, *node, *slot, &preserve) .map_or(OTHER, |source| types[source.index()]), + ScalarValue::Opaque => match seeds.get(index).copied().flatten() { + Some(ScalarNumericMode::Int32) => INT, + Some(ScalarNumericMode::Float64) => FLOAT, + Some(ScalarNumericMode::Number) => INT | FLOAT, + None => OTHER, + }, _ => OTHER, }; let next = next | types[index]; diff --git a/jit/tests/array_bce.rs b/jit/tests/array_bce.rs new file mode 100644 index 000000000..17fb71d1c --- /dev/null +++ b/jit/tests/array_bce.rs @@ -0,0 +1,263 @@ +//! Loop-hoisted array metadata, bounds-check elimination and literal-tag +//! folding in Tier 2 must match the same-version QuickJS interpreter. +#![cfg(all( + feature = "compiler", + feature = "test-support", + target_endian = "little", + any(target_os = "linux", target_os = "macos"), + any(target_arch = "x86_64", target_arch = "aarch64") +))] + +use rquickjs::{Context, Runtime}; +use rquickjs_jit::{Jit, JitConfig}; +use std::time::{Duration, Instant}; + +fn evaluate_without_jit(source: &str, expression: &str) -> String { + let runtime = Runtime::new().unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| { + ctx.eval::<(), _>(source).unwrap(); + ctx.eval::(expression).unwrap() + }) +} + +/// Evaluates `warmup` until it enters a published Tier 2 body. +fn prepared(source: &str, warmup: &str) -> (Runtime, Jit, Context) { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .call_threshold(2) + .loop_threshold(4) + .force_optimized_for_test(true) + .stress_gc(true) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| ctx.eval::<(), _>(source)).unwrap(); + let deadline = Instant::now() + Duration::from_secs(60); + loop { + let before = jit.metrics(); + context.with(|ctx| ctx.eval::<(), _>(warmup)).unwrap(); + jit.poll(); + let after = jit.metrics(); + if after.tier2_entries > before.tier2_entries + && after.deopts == before.deopts + && after.pending_worker_jobs == 0 + { + return (runtime, jit, context); + } + assert!( + Instant::now() < deadline, + "{warmup} never reached a stable Tier2 body: {after:?}" + ); + std::thread::sleep(Duration::from_millis(1)); + } +} + +const CONVERT: &str = r#" + function convertAndSum(ints, floats) { + let mixed = 0.0; + for (let i = 0; i < ints.length; i++) { + floats[i] = ints[i] * 0.25 + 0.5; + mixed += floats[i]; + } + return mixed; + } + function copyScaled(src, dst, n) { + let last = -1; + for (let i = 0; i < n; i++) { + dst[i] = src[i] * 2; + last = dst[i]; + } + return last; + } +"#; + +#[test] +fn two_typed_arrays_share_one_loop_and_match_the_interpreter() { + // Both receivers carry loop-hoisted metadata at once (one with the exact + // length bound, one metadata-only) across the `a[i] = v` nullish-check + // diamond. Aliased views observe every store, short destinations and + // out-of-range counts leave the guarded path, and zero-trip loops still + // return exactly what QuickJS returns. + const CASES: &str = r#"JSON.stringify([ + convertAndSum(new Int32Array([1, 2, 3, 4]), new Float64Array(4)), + convertAndSum(new Int32Array(0), new Float64Array(0)), + convertAndSum(new Int32Array([8, 9, 10]), new Float64Array(2)), + (() => { const buffer = new ArrayBuffer(64); + const ints = new Int32Array(buffer, 0, 4); + const floats = new Float64Array(buffer, 0, 4); + ints.set([7, 11, 13, 17]); + return [convertAndSum(ints, floats), Array.from(ints)]; })(), + copyScaled(new Int32Array([3, 4, 5]), new Float64Array(3), 3), + copyScaled(new Int32Array([3, 4, 5]), new Float64Array(3), 5), + copyScaled(new Int32Array([3, 4, 5]), new Int32Array(2), 3), + copyScaled(new Int32Array(0), new Float64Array(0), 0), + copyScaled([1, 2, 3], new Float64Array(3), 3) + ])"#; + let expected = evaluate_without_jit(CONVERT, CASES); + let (_runtime, jit, context) = prepared( + CONVERT, + "convertAndSum(new Int32Array([1,2,3,4,5,6,7,8]), new Float64Array(8));\ + copyScaled(new Int32Array([1,2,3,4,5,6,7,8]), new Float64Array(8), 8)", + ); + let before = jit.metrics(); + let actual = context.with(|ctx| ctx.eval::(CASES)).unwrap(); + let after = jit.metrics(); + assert_eq!(actual, expected); + assert_eq!(after.native_entries, after.native_exits); + assert!( + after.tier2_entries > before.tier2_entries, + "cases never entered Tier2: {before:?} -> {after:?}" + ); + assert!( + after.deopts > before.deopts, + "short destinations and wrong modes must leave the guarded path: {before:?} -> {after:?}" + ); +} + +#[test] +fn metadata_only_typed_hoist_revalidates_detach_after_observable_poll() { + use rquickjs::qjs; + use std::sync::{ + atomic::{AtomicBool, Ordering}, + Arc, + }; + let (runtime, jit, context) = prepared( + CONVERT, + "copyScaled(new Int32Array([1,2,3,4,5,6,7,8]), new Float64Array(8), 8)", + ); + let (raw_ctx, raw_destination, raw_buffer) = context.with(|ctx| { + ctx.eval::<(), _>( + "globalThis.src=new Int32Array(1000000).fill(3);\ + globalThis.dst=new Float64Array(1000000);globalThis.buffer=dst.buffer", + ) + .unwrap(); + let destination: rquickjs::Object = ctx.globals().get("dst").unwrap(); + let buffer: rquickjs::Object = ctx.globals().get("buffer").unwrap(); + ( + ctx.as_raw().as_ptr() as usize, + unsafe { qjs::JS_VALUE_GET_PTR(destination.as_value().as_raw()) } as usize, + unsafe { qjs::JS_VALUE_GET_PTR(buffer.as_value().as_raw()) } as usize, + ) + }); + let detached = Arc::new(AtomicBool::new(false)); + let observed = detached.clone(); + // Detach the destination only after native stores are visible. The + // interrupt runs at a loop poll, which must revalidate the hoisted + // data/count before any later iteration writes through them. + runtime.set_interrupt_handler(Some(Box::new(move || unsafe { + let ctx = raw_ctx as *mut qjs::JSContext; + let destination = qjs::JS_MKPTR( + qjs::JS_TAG_OBJECT, + raw_destination as *mut core::ffi::c_void, + ); + let value = qjs::JS_GetPropertyUint32(ctx, destination, 1); + let tag = qjs::JS_VALUE_GET_TAG(value); + let started = (tag == qjs::JS_TAG_INT && qjs::JS_VALUE_GET_INT(value) == 6) + || (tag == qjs::JS_TAG_FLOAT64 && qjs::JS_VALUE_GET_FLOAT64(value) == 6.0); + qjs::JS_FreeValue(ctx, value); + if started && !observed.swap(true, Ordering::SeqCst) { + let buffer = qjs::JS_MKPTR(qjs::JS_TAG_OBJECT, raw_buffer as *mut core::ffi::c_void); + qjs::JS_DetachArrayBuffer(ctx, buffer); + } + false + }))); + let before = jit.metrics(); + let result = context.with(|ctx| { + ctx.eval::("copyScaled(src, dst, src.length) === undefined && dst.length === 0") + }); + runtime.set_interrupt_handler(None); + assert!( + detached.load(Ordering::SeqCst), + "no poll observed completed stores" + ); + assert!( + result.unwrap(), + "a store or load used stale hoisted metadata after detach" + ); + let after = jit.metrics(); + assert!(after.tier2_entries > before.tier2_entries); + assert_eq!(after.native_entries, after.native_exits); +} + +#[test] +fn rebound_argument_keeps_its_value_when_an_alias_receiver_is_hoisted() { + // The traversal reads through a local alias of the entry argument while + // the loop rebinds the argument itself. The argument's representation + // must come from the rebinding, never from the receiver guard. + const SOURCE: &str = r#" + function rebind(a) { + let b = a, s = 0; + for (let i = 0; i < b.length; i++) { s = (s + b[i]) | 0; a = i; } + return a * 1000 + s; + } + function keep(a) { + let s = 0; + for (let i = 0; i < a.length; i++) s = (s + a[i]) | 0; + return a.length * 1000 + s; + } + "#; + const CASES: &str = r#"JSON.stringify([ + rebind(new Int32Array([1, 2, 3])), rebind(new Int32Array(0)), + rebind([4, 5, 6, 7]), rebind([]), + keep(new Int32Array([1, 2, 3])), keep([9, 8]), keep(new Float64Array([0.5])) + ])"#; + let expected = evaluate_without_jit(SOURCE, CASES); + let (_runtime, jit, context) = prepared( + SOURCE, + "rebind(new Int32Array([1,2,3,4,5,6,7,8]));keep(new Int32Array([1,2,3,4,5,6,7,8]))", + ); + let before = jit.metrics(); + let actual = context.with(|ctx| ctx.eval::(CASES)).unwrap(); + let after = jit.metrics(); + assert_eq!(actual, expected); + assert_eq!(after.native_entries, after.native_exits); + assert!(after.tier2_entries > before.tier2_entries); +} + +#[test] +fn literal_tag_folding_keeps_exact_numeric_semantics() { + // Numeric constant-pool literals, literal Bool/Int32 branch conditions, + // hoisted Int32 lengths and checked overflow must all behave exactly as + // in QuickJS, including the Float64 overflow and NaN comparisons. + const SOURCE: &str = r#" + function literals(values, seed) { + let acc = seed, flag = true; + for (let i = 0; i < values.length; i++) { + acc = acc + values[i] * 0.5 + 1.25; + if (flag) acc = acc - 0.25; + flag = i < 2; + } + return acc; + } + function overflow(values) { + let sum = 2147483000; + for (let i = 0; i < values.length; i++) sum = sum + values[i]; + return sum; + } + "#; + const CASES: &str = r#"JSON.stringify([ + literals(new Int32Array([1, 2, 3, 4]), 0), + literals(new Float64Array([0.5, NaN, 2]), 1), + literals([1, 2.5, -3], 0.75), + literals(new Int32Array(0), 7), + overflow(new Int32Array([600, 100])), + overflow([1, 2, 3]), + overflow(new Float64Array([1e300, 1e300])) + ])"#; + let expected = evaluate_without_jit(SOURCE, CASES); + let (_runtime, jit, context) = prepared( + SOURCE, + "literals(new Int32Array([1,2,3,4,5,6,7,8]),0);overflow(new Int32Array([1,2,3,4,5,6,7,8]))", + ); + let before = jit.metrics(); + let actual = context.with(|ctx| ctx.eval::(CASES)).unwrap(); + let after = jit.metrics(); + assert_eq!(actual, expected); + assert_eq!(after.native_entries, after.native_exits); + assert!(after.tier2_entries > before.tier2_entries); +} diff --git a/jit/tests/optimized.rs b/jit/tests/optimized.rs index 3b8d4beba..e7e7305ad 100644 --- a/jit/tests/optimized.rs +++ b/jit/tests/optimized.rs @@ -2783,7 +2783,13 @@ fn automatic_call_heavy_promotes_the_direct_edge_caller() { assert_eq!(result, 7_000); jit.poll(); let after = jit.metrics(); + // The leaf's five baseline profitability retries are paced by + // the runtime clock, while the Tier1-rejection blacklist and the + // caller's first Tier2 entry can both happen earlier; faster + // native loops finish that race sooner. Wait for the leaf's + // retries to settle so the exact count below checks the caller. if after.blacklisted > 0 + && after.profitability_rejected >= 5 && after.pending_worker_jobs == 0 && after.pending_snapshot_bytes == 0 && after.native_entries - before.native_entries == 1 From 44097e73af011ecf991aa9b596e62598d5618a3d Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:13:19 +0800 Subject: [PATCH 22/54] sys: pin the refcount header layout for inline DUP/FREE (patch 0022) Add sys/patches/0022-inline-refcount.patch. Static assertions require every payload a JS_VALUE_HAS_REF_COUNT value can address (objects, bytecode, strings, ropes, symbols, BigInts, modules) to start with the C int count, and the ABI value-layout fingerprint now also hashes its offset and width. The Rust side pins REF_COUNT_OFFSET/REF_COUNT_BYTES into the same fingerprint, so a QuickJS build without the patch fails ABI validation and the JIT stays closed. No ABI table or structure changes; ABI minor and bindings unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) --- jit/src/abi.rs | 12 +++++++- sys/build_support/patch.rs | 5 ++-- sys/patches/0022-inline-refcount.patch | 39 ++++++++++++++++++++++++++ sys/patches/README.md | 10 +++++++ sys/tests/jit_patch.rs | 1 + 5 files changed, 64 insertions(+), 3 deletions(-) create mode 100644 sys/patches/0022-inline-refcount.patch diff --git a/jit/src/abi.rs b/jit/src/abi.rs index b39c65a23..52b83364a 100644 --- a/jit/src/abi.rs +++ b/jit/src/abi.rs @@ -912,11 +912,21 @@ fn element_layout_fingerprint() -> u64 { hash } +/// Byte offset of `JSRefCountHeader::ref_count` in every payload addressed by +/// a JSValue with `JS_VALUE_HAS_REF_COUNT`. Generated code updates the count +/// in place; the linked QuickJS pins this value in its value-layout +/// fingerprint (`0022-inline-refcount.patch`), so a mismatch fails closed. +pub(crate) const REF_COUNT_OFFSET: i32 = 0; +/// Width in bytes of the C `int` reference count. +pub(crate) const REF_COUNT_BYTES: u64 = 4; + fn value_layout_fingerprint() -> u64 { let mut hash = layout_start::(); hash = hash_u64(hash, (qjs::JS_TAG_FIRST as i64) as u64); hash = hash_u64(hash, (qjs::JS_TAG_FLOAT64 as i64) as u64); - hash_u64(hash, u64::from(cfg!(target_pointer_width = "32"))) + hash = hash_u64(hash, u64::from(cfg!(target_pointer_width = "32"))); + hash = hash_u64(hash, REF_COUNT_OFFSET as u64); + hash_u64(hash, REF_COUNT_BYTES) } fn expected_build_fingerprint() -> u64 { diff --git a/sys/build_support/patch.rs b/sys/build_support/patch.rs index 41317e2f2..37eadd3da 100644 --- a/sys/build_support/patch.rs +++ b/sys/build_support/patch.rs @@ -1,7 +1,7 @@ use std::{fs, io, path::Path}; const QUICKJS_BASELINE_FNV64: u64 = 0x3302_116a_b0fc_269c; -const EXPECTED_PATCHES: [(&str, u64); 19] = [ +const EXPECTED_PATCHES: [(&str, u64); 20] = [ ("0001-rquickjs-jit.patch", 0x18d7_40cc_6943_30cf), ("0002-runtime-feedback.patch", 0xe4c1_ed52_ee5d_d469), ("0003-element-layout.patch", 0xa12c_ccd2_7420_e880), @@ -21,6 +21,7 @@ const EXPECTED_PATCHES: [(&str, u64); 19] = [ ("0018-inline-frame-recovery.patch", 0xb971d61edccde229), ("0019-array-feedback.patch", 0x11c3cb13a8297dd4), ("0020-typed-array-guard.patch", 0x5d10_38a3_8bc9_9821), + ("0022-inline-refcount.patch", 0xd0ee_ecf9_5982_f27e), ]; pub(crate) const BASELINE_FILES: [&str; 19] = [ "api-test.c", @@ -60,7 +61,7 @@ const PATCHED_FILE_FINGERPRINTS: [(&str, u64); 21] = [ ("list.h", 0xb337_70f7_b76d_a3d8), ("quickjs-atom.h", 0x30b4_9116_b6a2_aa99), ("quickjs-c-atomics.h", 0x490b_0f29_f631_3fc0), - ("quickjs.c", 0xc535_b5f1_48e7_9b27), + ("quickjs.c", 0x5005_e994_a5f8_5c7c), ("quickjs-jit.h", 0x0188_04ad_21d2_95a7), ("quickjs-jit-helpers.h", 0xa469_e80f_be47_404c), ("quickjs-opcode.h", 0x3d05_cfdf_5cf7_2930), diff --git a/sys/patches/0022-inline-refcount.patch b/sys/patches/0022-inline-refcount.patch new file mode 100644 index 000000000..269a9c6d2 --- /dev/null +++ b/sys/patches/0022-inline-refcount.patch @@ -0,0 +1,39 @@ +diff --git a/quickjs.c b/quickjs.c +--- a/quickjs.c ++++ b/quickjs.c +@@ -2432,6 +2432,26 @@ + out->float64_array_class_id = JS_CLASS_FLOAT64_ARRAY; + } + ++/* Generated code increments and decrements the reference count of every ++ * JS_VALUE_HAS_REF_COUNT payload in place and calls JS_JitHelperFree only ++ * for the last reference. Every such payload must start with the C int count; ++ * the value-layout fingerprint pins its offset and width. */ ++_Static_assert(offsetof(JSRefCountHeader, ref_count) == 0, ++ "JIT inline refcount offset"); ++_Static_assert(sizeof(((JSRefCountHeader *)0)->ref_count) == sizeof(int32_t), ++ "JIT inline refcount width"); ++_Static_assert(offsetof(JSGCObjectHeader, ref_count) == 0, ++ "JIT inline GC refcount offset"); ++_Static_assert(offsetof(JSObject, header) == 0, "JIT inline object refcount"); ++_Static_assert(offsetof(JSFunctionBytecode, header) == 0, ++ "JIT inline bytecode refcount"); ++_Static_assert(offsetof(JSString, header) == 0, "JIT inline string refcount"); ++_Static_assert(offsetof(JSStringRope, header) == 0, "JIT inline rope refcount"); ++_Static_assert(offsetof(JSBigInt, header) == 0, "JIT inline BigInt refcount"); ++_Static_assert(offsetof(JSModuleDef, header) == 0, "JIT inline module refcount"); ++_Static_assert(JS_TAG_FIRST == JS_TAG_BIG_INT && JS_TAG_OBJECT == -1, ++ "JIT inline refcount tag interval"); ++ + static uint64_t qjsjit_value_layout(void) + { + uint64_t hash = qjsjit_layout_start(sizeof(JSValue), _Alignof(JSValue)); +@@ -2442,6 +2462,8 @@ + #else + hash = qjsjit_hash_u64(hash, 0); + #endif ++ hash = qjsjit_hash_u64(hash, offsetof(JSRefCountHeader, ref_count)); ++ hash = qjsjit_hash_u64(hash, sizeof(((JSRefCountHeader *)0)->ref_count)); + return hash; + } + diff --git a/sys/patches/README.md b/sys/patches/README.md index 931cf9bd3..ca4a4e896 100644 --- a/sys/patches/README.md +++ b/sys/patches/README.md @@ -60,6 +60,16 @@ data pointer before returning `{ data, count, mode }`. The table advertises zero effects and the receiver is passed by pointer so generated calls do not depend on platform-specific aggregate argument classification. +`0022-inline-refcount.patch` pins the reference-count header that generated +code now updates in place. Static assertions require every payload a +`JS_VALUE_HAS_REF_COUNT` value can address (objects, bytecode, strings, ropes, +symbols, BigInts and modules) to start with the C `int` count, and the ABI +value-layout fingerprint additionally hashes its offset and width. Native code +increments the count for DUP, decrements it for FREE while other references +remain, and still calls `JS_JitHelperFree` for the last reference and in +stress-GC frames. No ABI table or structure changes, so the ABI minor version +is unchanged; a library without this patch fails the fingerprint check. + The build accepts only the patch names and byte digests listed in `build_support/patch.rs`, then verifies the complete patched source manifest. Keeping the baseline and patch separate makes diff --git a/sys/tests/jit_patch.rs b/sys/tests/jit_patch.rs index f36861117..9d614a360 100644 --- a/sys/tests/jit_patch.rs +++ b/sys/tests/jit_patch.rs @@ -56,6 +56,7 @@ fn copy_patches(destination: &std::path::Path) { "0018-inline-frame-recovery.patch", "0019-array-feedback.patch", "0020-typed-array-guard.patch", + "0022-inline-refcount.patch", ] { fs::copy(source.join(patch), destination.join(patch)).unwrap(); } From 582c7e2a60c8bd4847d65628b224ddbe5b1097e6 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:13:19 +0800 Subject: [PATCH 23/54] perf(jit): inline DUP/FREE reference counting in Tier 1 and Tier 2 Generated code now increments ref_count for DUP and decrements it for FREE while other references remain; JS_JitHelperFree still runs for the last reference (finalizers, GC lists, reentrancy) and every stress-GC frame keeps the helpers and their collection points. - Tier 1: argument/local/stack dups, put/drop/nip/is_undefined frees, property receivers and consumed generic CALL operands. Inline releases write undefined to the consumed slot; CALL/GET_PROPERTY scratch outputs are cleared after their SSA reload because a following FREE no longer rematerializes the frame (exits reject owned scratch). - Tier 2: owned stack/local frees, owned-slot dup, shadow-frame dup/free and owner materialization on helper-bound paths (MATERIALIZE_OWNER is a js_dup); deopt/exception bridges keep the helper. The direct-call signature deopt block is cold and helper continuations inherit coldness. - Tests: new differential/ownership suite (tiers x stress GC, helper-count evidence that last references still reach FREE); helper-count expectations updated where shared references are now released inline. Noisy diagnostic vs 82d3808 (automatic tiering, P-cores, 9 paired runs): objects-polymorphic 1.23x, generic-call-fallback 1.38x, arrays-typed 1.06x, call-heavy 1.35x, float64array-traversal 1.27x, property-heavy 1.00x, mixed-quotes 0.98x. Not publishable evidence. Co-Authored-By: Claude Opus 5.5 (1M context) --- jit/src/compiler/baseline.rs | 299 +++++++++++++++--- jit/src/compiler/mod.rs | 2 + jit/src/compiler/optimized.rs | 334 +++++++++++++++++++-- jit/src/compiler/optimized/frame_inline.rs | 132 ++++++-- jit/src/compiler/refcount.rs | 243 +++++++++++++++ jit/tests/frame_inline.rs | 7 +- jit/tests/generic_call_cleanup.rs | 14 +- jit/tests/inline_refcount.rs | 311 +++++++++++++++++++ jit/tests/property_specialization.rs | 9 +- 9 files changed, 1252 insertions(+), 99 deletions(-) create mode 100644 jit/src/compiler/refcount.rs create mode 100644 jit/tests/inline_refcount.rs diff --git a/jit/src/compiler/baseline.rs b/jit/src/compiler/baseline.rs index 00e7cfb64..34a84457b 100644 --- a/jit/src/compiler/baseline.rs +++ b/jit/src/compiler/baseline.rs @@ -2885,6 +2885,7 @@ fn lower_function( source, flat_argument_slot(index), false, + false, )?; depth += 1; set_visible_stack_depth(builder, frame, stack_base, depth, layout)?; @@ -2903,6 +2904,7 @@ fn lower_function( source, flat_local_slot(ir, index), false, + false, )?; depth += 1; set_visible_stack_depth(builder, frame, stack_base, depth, layout)?; @@ -2921,6 +2923,7 @@ fn lower_function( source, flat_local_slot(ir, index), true, + false, )?; depth += 1; set_visible_stack_depth(builder, frame, stack_base, depth, layout)?; @@ -2931,20 +2934,18 @@ fn lower_function( .next() .ok_or(CompileFailure::InvalidArtifact)?; let output_index = depth + usize::from(local_index); - let output = flat_stack_slot(ir, output_index)?; - invoke_helper!( - qjs::JSJitHelperId_JS_JIT_HELPER_DUP, + let source = use_pair(builder, locals[usize::from(local_index)]); + lower_dup_if_refcounted( + builder, + &helper_lowering, state, output_index, - &[output, flat_local_slot(ir, local_index)] - ); - reload_pair( - builder, - stack[output_index], - stack_base, output_index, - layout, - ); + source, + flat_local_slot(ir, local_index), + false, + true, + )?; } depth += 2; set_visible_stack_depth(builder, frame, stack_base, depth, layout)?; @@ -3157,22 +3158,18 @@ fn lower_function( let state = helper_states .next() .ok_or(CompileFailure::InvalidArtifact)?; - invoke_helper!( - qjs::JSJitHelperId_JS_JIT_HELPER_FREE, + let value = use_pair(builder, stack[start]); + lower_free_always_in_stress( + builder, + &helper_lowering, state, depth, - &[flat_stack_slot(ir, start)?] - ); - for (offset, variables) in stack - .get(start..depth) - .ok_or(CompileFailure::InvalidArtifact)? - .iter() - .copied() - .enumerate() - { - let index = start + offset; - reload_pair(builder, variables, stack_base, index, layout); - } + value, + flat_stack_slot(ir, start)?, + stack[start], + stack_base, + start, + )?; apply_stack_operation(builder, &stack, &mut depth, operation); } StackOp::Dup @@ -3211,6 +3208,7 @@ fn lower_function( source_pair, flat_stack_slot(ir, start + source)?, false, + false, )?; } debug_assert_eq!( @@ -3260,13 +3258,17 @@ fn lower_function( UnaryOp::IsNull => tag_is(builder, value.tag, qjs::JS_TAG_NULL), _ => unreachable!(), }; - invoke_helper!( - qjs::JSJitHelperId_JS_JIT_HELPER_FREE, + lower_free_always_in_stress( + builder, + &helper_lowering, state, depth, - &[slot] - ); - reload_pair(builder, stack[index], stack_base, index, layout); + value, + slot, + stack[index], + stack_base, + index, + )?; let boolean = pair_from_bool(builder, result); define_pair(builder, stack[index], boolean); } else if matches!(operation, UnaryOp::Plus | UnaryOp::LogicalNot) { @@ -3642,17 +3644,21 @@ fn lower_dup_if_refcounted( source: Pair, source_slot: u32, checked: bool, + stress_primitives: bool, ) -> Result<(), CompileFailure> { let output_slot = flat_stack_slot(helpers.ir, output_index)?; - let mut needs_helper = builder.ins().icmp_imm(IntCC::SignedLessThan, source.tag, 0); - if checked { - let uninitialized = tag_is(builder, source.tag, qjs::JS_TAG_UNINITIALIZED); - needs_helper = builder.ins().bor(needs_helper, uninitialized); - } let primitive = builder.create_block(); let slow = builder.create_block(); let continuation = builder.create_block(); - builder.ins().brif(needs_helper, slow, &[], primitive, &[]); + emit_inline_dup_dispatch( + builder, + helpers, + source, + checked, + stress_primitives, + primitive, + slow, + ); builder.seal_block(primitive); builder.switch_to_block(primitive); define_pair(builder, helpers.stack[output_index], source); @@ -3680,6 +3686,54 @@ fn lower_dup_if_refcounted( Ok(()) } +/// Branches to `primitive` (after an inline `ref_count++` for heap values) +/// when DUP needs no helper, else to `slow`. Checked locals send +/// `JS_TAG_UNINITIALIZED` to the helper exactly as before. +fn emit_inline_dup_dispatch( + builder: &mut FunctionBuilder<'_>, + helpers: &HelperLowering<'_>, + source: Pair, + checked: bool, + stress_primitives: bool, + primitive: Block, + slow: Block, +) { + if checked { + let uninitialized = tag_is(builder, source.tag, qjs::JS_TAG_UNINITIALIZED); + let initialized = builder.create_block(); + builder + .ins() + .brif(uninitialized, slow, &[], initialized, &[]); + builder.seal_block(initialized); + builder.switch_to_block(initialized); + } + let refcounted = super::refcount::emit_has_ref_count(builder, source.tag); + let duplicate = builder.create_block(); + if stress_primitives { + let copy = builder.create_block(); + builder.ins().brif(refcounted, duplicate, &[], copy, &[]); + builder.seal_block(copy); + builder.switch_to_block(copy); + let no_stress = + super::refcount::emit_no_stress(builder, helpers.frame, helpers.layout.flags); + builder.ins().brif(no_stress, primitive, &[], slow, &[]); + } else { + builder + .ins() + .brif(refcounted, duplicate, &[], primitive, &[]); + } + builder.seal_block(duplicate); + builder.switch_to_block(duplicate); + super::refcount::emit_dup_refcounted( + builder, + helpers.frame, + helpers.layout.flags, + source.payload, + primitive, + slow, + ); +} + #[allow(clippy::too_many_arguments)] fn lower_dup_local_if_refcounted( builder: &mut FunctionBuilder<'_>, @@ -3693,11 +3747,10 @@ fn lower_dup_local_if_refcounted( output_base: Value, output_index: usize, ) -> Result<(), CompileFailure> { - let needs_helper = builder.ins().icmp_imm(IntCC::SignedLessThan, source.tag, 0); let primitive = builder.create_block(); let slow = builder.create_block(); let continuation = builder.create_block(); - builder.ins().brif(needs_helper, slow, &[], primitive, &[]); + emit_inline_dup_dispatch(builder, helpers, source, false, false, primitive, slow); builder.seal_block(primitive); builder.switch_to_block(primitive); define_pair(builder, output, source); @@ -3731,10 +3784,104 @@ fn lower_free_if_refcounted( backing_base: Value, backing_index: usize, ) -> Result<(), CompileFailure> { - let refcounted = builder.ins().icmp_imm(IntCC::SignedLessThan, value.tag, 0); + lower_free( + builder, + helpers, + state, + live_depth, + value, + value_slot, + variables, + backing_base, + backing_index, + false, + ) +} + +/// FREE for sites that historically invoked the helper for every value: +/// outside stress GC primitives need no work, but stress-GC frames keep the +/// helper (and its collection points) for primitives too. +#[allow(clippy::too_many_arguments)] +fn lower_free_always_in_stress( + builder: &mut FunctionBuilder<'_>, + helpers: &HelperLowering<'_>, + state: FrameStateId, + live_depth: usize, + value: Pair, + value_slot: u32, + variables: PairVars, + backing_base: Value, + backing_index: usize, +) -> Result<(), CompileFailure> { + lower_free( + builder, + helpers, + state, + live_depth, + value, + value_slot, + variables, + backing_base, + backing_index, + true, + ) +} + +#[allow(clippy::too_many_arguments)] +fn lower_free( + builder: &mut FunctionBuilder<'_>, + helpers: &HelperLowering<'_>, + state: FrameStateId, + live_depth: usize, + value: Pair, + value_slot: u32, + variables: PairVars, + backing_base: Value, + backing_index: usize, + stress_primitives: bool, +) -> Result<(), CompileFailure> { + let refcounted = super::refcount::emit_has_ref_count(builder, value.tag); + let release = builder.create_block(); + let released = builder.create_block(); let slow = builder.create_block(); let continuation = builder.create_block(); - builder.ins().brif(refcounted, slow, &[], continuation, &[]); + if stress_primitives { + let primitive = builder.create_block(); + builder.ins().brif(refcounted, release, &[], primitive, &[]); + builder.seal_block(primitive); + builder.switch_to_block(primitive); + let no_stress = + super::refcount::emit_no_stress(builder, helpers.frame, helpers.layout.flags); + builder.ins().brif(no_stress, continuation, &[], slow, &[]); + } else { + builder + .ins() + .brif(refcounted, release, &[], continuation, &[]); + } + builder.seal_block(release); + builder.switch_to_block(release); + super::refcount::emit_release_refcounted( + builder, + helpers.frame, + helpers.layout.flags, + value.payload, + released, + slow, + ); + builder.seal_block(released); + builder.switch_to_block(released); + // Match the helper's post-state: the consumed slot holds undefined, in + // the SSA variables and in its backing frame memory. + let undefined = constant_pair(builder, TaggedValue::new(0, qjs::JS_TAG_UNDEFINED as i64)); + define_pair(builder, variables, undefined); + store_jsvalue_slot( + builder, + backing_base, + backing_index, + undefined, + helpers.layout, + )?; + builder.ins().jump(continuation, &[]); builder.seal_block(slow); builder.switch_to_block(slow); helpers.invoke( @@ -3983,6 +4130,7 @@ fn lower_get_property( output_index, helpers.layout, ); + clear_reloaded_output(builder, helpers, output_index)?; let value = use_pair(builder, helpers.stack[output_index]); builder.ins().jump(joined, &[value.payload, value.tag]); builder.switch_to_block(joined); @@ -4011,6 +4159,7 @@ fn lower_get_property( output_index, helpers.layout, ); + clear_reloaded_output(builder, helpers, output_index)?; } move_stack_pair( builder, @@ -5342,6 +5491,7 @@ fn lower_call( output_index, helpers.layout, ); + clear_reloaded_output(builder, helpers, output_index)?; let miss = builder.ins().iconst(types::I8, 0); builder.ins().jump(joined, &[miss]); builder.switch_to_block(joined); @@ -5372,6 +5522,7 @@ fn lower_call( output_index, helpers.layout, ); + clear_reloaded_output(builder, helpers, output_index)?; } // CALL borrows every input. The bytecode stack effect is separate and is // implemented in QuickJS interpreter order, clearing primitive inputs @@ -5411,13 +5562,19 @@ fn lower_call( // precisely one duplicate, so dropping that duplicate can never run a // finalizer and is the exact non-finalizing JS_FreeValue fast path. let function = use_pair(builder, helpers.stack[displaced_index]); - let ref_count = builder - .ins() - .load(types::I32, MemFlags::trusted(), function.payload, 0); + let ref_count = builder.ins().load( + types::I32, + MemFlags::trusted(), + function.payload, + crate::abi::REF_COUNT_OFFSET, + ); let ref_count = builder.ins().iadd_imm(ref_count, -1); - builder - .ins() - .store(MemFlags::trusted(), ref_count, function.payload, 0); + builder.ins().store( + MemFlags::trusted(), + ref_count, + function.payload, + crate::abi::REF_COUNT_OFFSET, + ); clear_pair( builder, helpers.stack[displaced_index], @@ -5525,6 +5682,36 @@ fn lower_call_input_free( )?; builder.ins().jump(continuation, &[]); builder.switch_to_block(free); + // Heap owners, and every operand in stress mode, reach this block. A + // shared heap reference is released inline; the last one and stress mode + // keep the exact FREE helper. + let refcounted = super::refcount::emit_has_ref_count(builder, input.tag); + let release = builder.create_block(); + let released = builder.create_block(); + let helper = builder.create_block(); + builder.ins().brif(refcounted, release, &[], helper, &[]); + builder.switch_to_block(release); + super::refcount::emit_release_refcounted( + builder, + helpers.frame, + helpers.layout.flags, + input.payload, + released, + helper, + ); + builder.switch_to_block(released); + // The slot may be a scratch owner that an earlier helper published; the + // FREE helper would leave undefined there, and exits require it. + let undefined = constant_pair(builder, TaggedValue::new(0, qjs::JS_TAG_UNDEFINED as i64)); + store_jsvalue_slot( + builder, + helpers.stack_base, + index, + undefined, + helpers.layout, + )?; + builder.ins().jump(clear, &[]); + builder.switch_to_block(helper); let slot = flat_stack_slot(helpers.ir, index)?; helpers.invoke( builder, @@ -5866,6 +6053,26 @@ fn move_stack_pair( clear_pair(builder, stack[source], base, source, layout) } +/// After a helper publishes an owned output in frame memory and the value is +/// reloaded into SSA, the SSA variable is the only owner the lowering tracks. +/// A later inline FREE no longer rematerializes the frame, so clear the +/// published copy: CALL/GET_PROPERTY outputs live in scratch slots, which must +/// hold undefined at every exit. +fn clear_reloaded_output( + builder: &mut FunctionBuilder<'_>, + helpers: &HelperLowering<'_>, + index: usize, +) -> Result<(), CompileFailure> { + let undefined = constant_pair(builder, TaggedValue::new(0, qjs::JS_TAG_UNDEFINED as i64)); + store_jsvalue_slot( + builder, + helpers.stack_base, + index, + undefined, + helpers.layout, + ) +} + fn clear_pair( builder: &mut FunctionBuilder<'_>, variables: PairVars, diff --git a/jit/src/compiler/mod.rs b/jit/src/compiler/mod.rs index 0a339d41e..7854225b3 100644 --- a/jit/src/compiler/mod.rs +++ b/jit/src/compiler/mod.rs @@ -59,6 +59,8 @@ mod helpers; #[cfg(all(feature = "compiler", not(target_family = "wasm")))] pub mod optimized; #[cfg(all(feature = "compiler", not(target_family = "wasm")))] +mod refcount; +#[cfg(all(feature = "compiler", not(target_family = "wasm")))] mod tagged_call_link; #[cfg(all(feature = "compiler", not(target_family = "wasm")))] diff --git a/jit/src/compiler/optimized.rs b/jit/src/compiler/optimized.rs index afc40c273..2b247b3ce 100644 --- a/jit/src/compiler/optimized.rs +++ b/jit/src/compiler/optimized.rs @@ -3340,14 +3340,14 @@ fn lower_optimized_machine( property_cache.flush(&mut builder); property_cache.invalidate(&mut builder); let source = opt_flat_stack_slot(&env, start)?; - depth = emit_opt_owned_helper_push( + depth = emit_opt_owned_dup( &mut builder, &env, &mut stack_provenance, + start, depth, node.pc(), - qjs::JSJitHelperId_JS_JIT_HELPER_DUP as usize, - &[source], + source, )?; continue; } @@ -3694,7 +3694,7 @@ fn lower_optimized_machine( ); let pc = builder.ins().iadd_imm(start, i64::from(node.pc())); builder.ins().store(MemFlags::new(), pc, frame, layout.pc); - opt_own_stack_for_exit( + opt_own_stack_for_helper( &mut builder, frame, sret, @@ -5796,7 +5796,7 @@ fn emit_opt_array_length( builder .ins() .store(MemFlags::new(), current_pc, env.frame, env.layout.pc); - opt_own_stack_for_exit( + opt_own_stack_for_helper( builder, env.frame, env.sret, @@ -6815,7 +6815,7 @@ fn emit_opt_guarded_property( builder .ins() .store(MemFlags::new(), current_pc, frame, layout.pc); - opt_own_stack_for_exit( + opt_own_stack_for_helper( builder, frame, sret, @@ -7162,7 +7162,6 @@ fn opt_release_materialized_aliases( pointer_type: cranelift_codegen::ir::Type, layout: super::helpers::FrameLayout, ) -> Result<(), CompileFailure> { - use rquickjs_core::qjs; for index in range { if !matches!( provenance.get(index), @@ -7174,15 +7173,16 @@ fn opt_release_materialized_aliases( .checked_add(index) .and_then(|slot| u32::try_from(slot).ok()) .ok_or(CompileFailure::ResourceLimit)?; - emit_opt_helper( + emit_opt_free_slot( builder, frame, sret, stack_base, exception_depth, signatures, - qjs::JSJitHelperId_JS_JIT_HELPER_FREE as usize, - &[0, slot], + stack_base, + index, + slot, pointer_type, layout, )?; @@ -7203,21 +7203,21 @@ fn opt_release_owned_stack( pointer_type: cranelift_codegen::ir::Type, layout: super::helpers::FrameLayout, ) -> Result<(), CompileFailure> { - use rquickjs_core::qjs; for index in start..end { let slot = flat_stack_base .checked_add(index) .and_then(|slot| u32::try_from(slot).ok()) .ok_or(CompileFailure::ResourceLimit)?; - emit_opt_helper( + emit_opt_free_slot( builder, frame, sret, stack_base, end, signatures, - qjs::JSJitHelperId_JS_JIT_HELPER_FREE as usize, - &[0, slot], + stack_base, + index, + slot, pointer_type, layout, )?; @@ -7226,8 +7226,70 @@ fn opt_release_owned_stack( Ok(()) } +/// Turns borrowed argument/local aliases on the operand stack into real +/// interpreter owners before a deopt or exception exit, using the audited +/// MATERIALIZE_OWNER helper (these exits are rare). #[allow(clippy::too_many_arguments)] fn opt_own_stack_for_exit( + builder: &mut cranelift_frontend::FunctionBuilder<'_>, + frame: cranelift_codegen::ir::Value, + sret: cranelift_codegen::ir::Value, + stack_base: cranelift_codegen::ir::Value, + depth: usize, + flat_stack_base: usize, + provenance: &[OptProvenance], + signatures: &[cranelift_codegen::ir::SigRef], + pointer_type: cranelift_codegen::ir::Type, + layout: super::helpers::FrameLayout, +) -> Result<(), CompileFailure> { + opt_own_stack( + builder, + frame, + sret, + stack_base, + depth, + flat_stack_base, + provenance, + signatures, + pointer_type, + layout, + false, + ) +} + +/// The same ownership transition on paths that continue into a helper or +/// runtime call and may run every iteration: outside stress GC each owner is +/// taken with an inline `js_dup` instead of a MATERIALIZE_OWNER helper call. +#[allow(clippy::too_many_arguments)] +fn opt_own_stack_for_helper( + builder: &mut cranelift_frontend::FunctionBuilder<'_>, + frame: cranelift_codegen::ir::Value, + sret: cranelift_codegen::ir::Value, + stack_base: cranelift_codegen::ir::Value, + depth: usize, + flat_stack_base: usize, + provenance: &[OptProvenance], + signatures: &[cranelift_codegen::ir::SigRef], + pointer_type: cranelift_codegen::ir::Type, + layout: super::helpers::FrameLayout, +) -> Result<(), CompileFailure> { + opt_own_stack( + builder, + frame, + sret, + stack_base, + depth, + flat_stack_base, + provenance, + signatures, + pointer_type, + layout, + true, + ) +} + +#[allow(clippy::too_many_arguments)] +fn opt_own_stack( builder: &mut cranelift_frontend::FunctionBuilder<'_>, frame: cranelift_codegen::ir::Value, sret: cranelift_codegen::ir::Value, @@ -7238,6 +7300,7 @@ fn opt_own_stack_for_exit( signatures: &[cranelift_codegen::ir::SigRef], pointer_type: cranelift_codegen::ir::Type, layout: super::helpers::FrameLayout, + inline_owner: bool, ) -> Result<(), CompileFailure> { use cranelift_codegen::ir::{condcodes::IntCC, types, InstBuilder, MemFlags}; const MATERIALIZE_OWNER_HELPER: usize = @@ -7253,6 +7316,9 @@ fn opt_own_stack_for_exit( .iconst(types::I64, i64::from(rquickjs_core::qjs::JS_TAG_UNDEFINED)), }; opt_set_stack_top(builder, frame, stack_base, 0, pointer_type, layout); + let cold = builder + .current_block() + .is_some_and(|block| builder.func.layout.is_cold(block)); for index in 0..depth { if matches!( provenance.get(index), @@ -7267,6 +7333,43 @@ fn opt_own_stack_for_exit( _ => return Err(CompileFailure::InvalidArtifact), }; opt_store(builder, stack_base, index, undefined); + let materialized = inline_owner.then(|| builder.create_block()); + if let Some(materialized) = materialized { + // Outside stress GC the owner is an exact inline `js_dup` of the + // published argument/local slot; stress GC keeps the helper and + // its collection points out of line. + let inline = builder.create_block(); + let helper_block = builder.create_block(); + let heap = builder.create_block(); + let copied = builder.create_block(); + builder.set_cold_block(helper_block); + let no_stress = super::refcount::emit_no_stress(builder, frame, layout.flags); + builder + .ins() + .brif(no_stress, inline, &[], helper_block, &[]); + builder.switch_to_block(inline); + let source_buffer = builder.ins().load( + pointer_type, + MemFlags::new(), + frame, + if source_kind == SOURCE_ARGUMENT { + layout.arg_buf + } else { + layout.var_buf + }, + ); + let source = opt_load(builder, source_buffer, source_index); + let refcounted = super::refcount::emit_has_ref_count(builder, source.tag); + builder.ins().brif(refcounted, heap, &[], copied, &[]); + builder.switch_to_block(heap); + super::refcount::emit_increment(builder, source.payload); + builder.ins().jump(copied, &[]); + builder.switch_to_block(copied); + opt_store(builder, stack_base, index, source); + opt_set_stack_top(builder, frame, stack_base, index + 1, pointer_type, layout); + builder.ins().jump(materialized, &[]); + builder.switch_to_block(helper_block); + } let signature = *signatures .get(MATERIALIZE_OWNER_HELPER) .ok_or(CompileFailure::InvalidArtifact)?; @@ -7302,6 +7405,12 @@ fn opt_own_stack_for_exit( let ok = builder.ins().icmp_imm(IntCC::Equal, status, MATERIALIZED); let continuation = builder.create_block(); let exception = builder.create_block(); + // The helper call is out of line whenever the bridge itself is (a + // cold deopt path) or the inline owner path makes it stress-only. + if inline_owner || cold { + builder.set_cold_block(continuation); + builder.set_cold_block(exception); + } builder.ins().brif(ok, continuation, &[], exception, &[]); builder.switch_to_block(exception); emit_opt_exit( @@ -7313,6 +7422,10 @@ fn opt_own_stack_for_exit( 0, ); builder.switch_to_block(continuation); + if let Some(materialized) = materialized { + builder.ins().jump(materialized, &[]); + builder.switch_to_block(materialized); + } } Ok(()) } @@ -7422,6 +7535,7 @@ fn emit_opt_specialized_call( let signature = builder.create_block(); let invoke = builder.create_block(); let deopt = builder.create_block(); + builder.set_cold_block(deopt); super::emit_guarded_direct_callee_identity( builder, function.tag, @@ -7616,7 +7730,7 @@ fn emit_opt_specialized_call( .and_then(|value| u32::try_from(value).ok()) .ok_or(CompileFailure::ResourceLimit) }; - opt_own_stack_for_exit( + opt_own_stack_for_helper( builder, frame, sret, @@ -7703,6 +7817,24 @@ fn emit_opt_specialized_call( opt_define(builder, stack[index], undefined); builder.ins().jump(cleaned, &[]); builder.switch_to_block(release); + // Shared heap owners drop one reference inline and then take the + // primitive path's exact post-state; the last reference and every + // stress-mode operand keep the FREE helper. + let refcounted = super::refcount::emit_has_ref_count(builder, consumed.tag); + let heap = builder.create_block(); + let helper = builder.create_block(); + builder.set_cold_block(helper); + builder.ins().brif(refcounted, heap, &[], helper, &[]); + builder.switch_to_block(heap); + super::refcount::emit_release_refcounted( + builder, + frame, + layout.flags, + consumed.payload, + primitive, + helper, + ); + builder.switch_to_block(helper); emit_opt_helper( builder, frame, @@ -8177,15 +8309,16 @@ fn emit_opt_free_stack_slot( env.pointer_type, env.layout, ); - emit_opt_helper( + emit_opt_free_slot( builder, env.frame, env.sret, env.stack_base, index + 1, env.helper_signatures, - rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_FREE as usize, - &[0, opt_flat_stack_slot(env, index)?], + env.stack_base, + index, + opt_flat_stack_slot(env, index)?, env.pointer_type, env.layout, ) @@ -8202,15 +8335,16 @@ fn emit_opt_free_local_slot( } let pair = opt_use(builder, env.locals[index]); opt_store(builder, env.var_buf, index, pair); - emit_opt_helper( + emit_opt_free_slot( builder, env.frame, env.sret, env.stack_base, 0, env.helper_signatures, - rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_FREE as usize, - &[0, opt_flat_local_slot(env, index)?], + env.var_buf, + index, + opt_flat_local_slot(env, index)?, env.pointer_type, env.layout, ) @@ -8267,7 +8401,7 @@ fn emit_opt_owned_helper_push( builder .ins() .store(MemFlags::new(), current_pc, env.frame, env.layout.pc); - opt_own_stack_for_exit( + opt_own_stack_for_helper( builder, env.frame, env.sret, @@ -8314,6 +8448,88 @@ fn emit_opt_owned_helper_push( Ok(depth + 1) } +/// DUP of an interpreter-owned stack slot. Outside stress GC the second +/// reference is taken inline (`ref_count++` for heap values) and published in +/// its own owning slot. When no borrowed argument/local alias is live below +/// the operand, the helper path would not change any provenance, so stress-GC +/// frames branch to the exact DUP helper (with its collection points) and both +/// paths join with identical ownership. Otherwise the helper is used always. +fn emit_opt_owned_dup( + builder: &mut cranelift_frontend::FunctionBuilder<'_>, + env: &OptEnv<'_>, + provenance: &mut [OptProvenance], + source_index: usize, + depth: usize, + pc: u32, + source_slot: u32, +) -> Result { + use cranelift_codegen::ir::{types, InstBuilder}; + let has_borrowed_alias = provenance[..depth] + .iter() + .any(|slot| matches!(slot, OptProvenance::Argument(_) | OptProvenance::Local(_))); + if env.int32_loop || has_borrowed_alias || depth >= env.stack.len() { + return emit_opt_owned_helper_push( + builder, + env, + provenance, + depth, + pc, + rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_DUP as usize, + &[source_slot], + ); + } + let inline = builder.create_block(); + let stress = builder.create_block(); + let joined = builder.create_block(); + builder.set_cold_block(stress); + let no_stress = super::refcount::emit_no_stress(builder, env.frame, env.layout.flags); + builder.ins().brif(no_stress, inline, &[], stress, &[]); + + builder.switch_to_block(stress); + let mut helper_provenance = provenance.to_vec(); + let helper_depth = emit_opt_owned_helper_push( + builder, + env, + &mut helper_provenance, + depth, + pc, + rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_DUP as usize, + &[source_slot], + )?; + builder.ins().jump(joined, &[]); + + builder.switch_to_block(inline); + let value = opt_use(builder, env.stack[source_index]); + if builder.func.dfg.value_type(value.payload) == types::I64 { + let heap = builder.create_block(); + let copied = builder.create_block(); + let refcounted = super::refcount::emit_has_ref_count(builder, value.tag); + builder.ins().brif(refcounted, heap, &[], copied, &[]); + builder.switch_to_block(heap); + super::refcount::emit_increment(builder, value.payload); + builder.ins().jump(copied, &[]); + builder.switch_to_block(copied); + } + opt_define(builder, env.stack[depth], value); + opt_store(builder, env.stack_base, depth, value); + opt_set_stack_top( + builder, + env.frame, + env.stack_base, + depth + 1, + env.pointer_type, + env.layout, + ); + builder.ins().jump(joined, &[]); + builder.switch_to_block(joined); + + provenance[depth] = OptProvenance::OwnedSlot; + if helper_depth != depth + 1 || helper_provenance != provenance { + return Err(CompileFailure::InvalidArtifact); + } + Ok(depth + 1) +} + /// Executes GetField through the audited owning helper when IC feedback says /// the result is a heap object. The helper keeps the receiver, so release it /// after the owned result has been published, then move that result into the @@ -8343,15 +8559,16 @@ fn emit_opt_owned_property_replace( &[receiver_slot, atom], )?; debug_assert_eq!(result_depth, depth + 1); - emit_opt_helper( + emit_opt_free_slot( builder, env.frame, env.sret, env.stack_base, result_depth, env.helper_signatures, - qjs::JSJitHelperId_JS_JIT_HELPER_FREE as usize, - &[0, receiver_slot], + env.stack_base, + receiver, + receiver_slot, env.pointer_type, env.layout, )?; @@ -8450,6 +8667,73 @@ fn emit_opt_helper( Ok(()) } +/// `JS_JitHelperFree` of the owner stored at `base[index]` (flat helper slot +/// `flat_slot`). Primitives and shared heap references are released inline +/// with the helper's exact post-state (an undefined slot); the last heap +/// reference and stress-GC frames keep the helper call. +#[allow(clippy::too_many_arguments)] +fn emit_opt_free_slot( + builder: &mut cranelift_frontend::FunctionBuilder<'_>, + frame: cranelift_codegen::ir::Value, + sret: cranelift_codegen::ir::Value, + stack_base: cranelift_codegen::ir::Value, + exception_depth: usize, + signatures: &[cranelift_codegen::ir::SigRef], + base: cranelift_codegen::ir::Value, + index: usize, + flat_slot: u32, + pointer_type: cranelift_codegen::ir::Type, + layout: super::helpers::FrameLayout, +) -> Result<(), CompileFailure> { + use cranelift_codegen::ir::{types, InstBuilder}; + let value = opt_load(builder, base, index); + let heap = builder.create_block(); + let primitive = builder.create_block(); + let clear = builder.create_block(); + let helper = builder.create_block(); + let done = builder.create_block(); + builder.set_cold_block(helper); + let refcounted = super::refcount::emit_has_ref_count(builder, value.tag); + builder.ins().brif(refcounted, heap, &[], primitive, &[]); + builder.switch_to_block(primitive); + let no_stress = super::refcount::emit_no_stress(builder, frame, layout.flags); + builder.ins().brif(no_stress, clear, &[], helper, &[]); + builder.switch_to_block(heap); + super::refcount::emit_release_refcounted( + builder, + frame, + layout.flags, + value.payload, + clear, + helper, + ); + builder.switch_to_block(clear); + let undefined = OptPair { + payload: builder.ins().iconst(types::I64, 0), + tag: builder + .ins() + .iconst(types::I64, i64::from(rquickjs_core::qjs::JS_TAG_UNDEFINED)), + }; + opt_store(builder, base, index, undefined); + builder.ins().jump(done, &[]); + builder.switch_to_block(helper); + emit_opt_helper( + builder, + frame, + sret, + stack_base, + exception_depth, + signatures, + rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_FREE as usize, + &[0, flat_slot], + pointer_type, + layout, + )?; + builder.ins().jump(done, &[]); + builder.switch_to_block(done); + Ok(()) +} + fn opt_set_stack_top( builder: &mut cranelift_frontend::FunctionBuilder<'_>, frame: cranelift_codegen::ir::Value, diff --git a/jit/src/compiler/optimized/frame_inline.rs b/jit/src/compiler/optimized/frame_inline.rs index c0d5fb019..883e63960 100644 --- a/jit/src/compiler/optimized/frame_inline.rs +++ b/jit/src/compiler/optimized/frame_inline.rs @@ -288,6 +288,23 @@ impl InlineFrame<'_, '_> { builder.switch_to_block(success); Ok(()) } + /// Memory location of a flat helper slot in this shadow frame. + fn slot_address(&self, slot: u32) -> (Value, usize) { + let slot = slot as usize; + let arguments = usize::from(self.region.body.argument_count); + let locals = usize::from(self.region.body.local_count); + if slot < arguments { + (self.arguments, slot) + } else if slot < arguments + locals { + (self.locals, slot - arguments) + } else { + (self.stack, slot - arguments - locals) + } + } + /// Exact `JS_JitHelperFree`: the helper's pc and depth are published + /// first, then primitives and shared heap references are released inline + /// (leaving undefined, as the helper does). The last heap reference and + /// stress-GC frames still call the helper. fn free( &self, builder: &mut FunctionBuilder<'_>, @@ -296,6 +313,36 @@ impl InlineFrame<'_, '_> { live: usize, exception_depth: usize, ) -> Result<(), CompileFailure> { + set_pc(builder, self.root, self.frame, pc); + self.depth(builder, live); + let (base, index) = self.slot_address(slot); + let value = opt_load(builder, base, index); + let heap = builder.create_block(); + let primitive = builder.create_block(); + let clear = builder.create_block(); + let helper = builder.create_block(); + let done = builder.create_block(); + builder.set_cold_block(helper); + let refcounted = super::super::refcount::emit_has_ref_count(builder, value.tag); + builder.ins().brif(refcounted, heap, &[], primitive, &[]); + builder.switch_to_block(primitive); + let no_stress = + super::super::refcount::emit_no_stress(builder, self.frame, self.root.layout.flags); + builder.ins().brif(no_stress, clear, &[], helper, &[]); + builder.switch_to_block(heap); + super::super::refcount::emit_release_refcounted( + builder, + self.frame, + self.root.layout.flags, + value.payload, + clear, + helper, + ); + builder.switch_to_block(clear); + let empty = undefined(builder); + opt_store(builder, base, index, empty); + builder.ins().jump(done, &[]); + builder.switch_to_block(helper); self.helper( builder, qjs::JSJitHelperId_JS_JIT_HELPER_FREE, @@ -303,7 +350,64 @@ impl InlineFrame<'_, '_> { pc, live, exception_depth, - ) + )?; + builder.ins().jump(done, &[]); + builder.switch_to_block(done); + Ok(()) + } + /// Exact `JS_JitHelperDup` into an unoccupied slot: primitives are copied + /// and heap values gain one reference inline; stress-GC frames keep the + /// helper for every value. + fn dup( + &self, + builder: &mut FunctionBuilder<'_>, + output: u32, + input: u32, + pc: u32, + live: usize, + exception_depth: usize, + ) -> Result<(), CompileFailure> { + set_pc(builder, self.root, self.frame, pc); + self.depth(builder, live); + let (input_base, input_index) = self.slot_address(input); + let (output_base, output_index) = self.slot_address(output); + let value = opt_load(builder, input_base, input_index); + let heap = builder.create_block(); + let copy = builder.create_block(); + let helper = builder.create_block(); + let done = builder.create_block(); + builder.set_cold_block(helper); + let primitive = builder.create_block(); + let refcounted = super::super::refcount::emit_has_ref_count(builder, value.tag); + builder.ins().brif(refcounted, heap, &[], primitive, &[]); + builder.switch_to_block(primitive); + let no_stress = + super::super::refcount::emit_no_stress(builder, self.frame, self.root.layout.flags); + builder.ins().brif(no_stress, copy, &[], helper, &[]); + builder.switch_to_block(heap); + super::super::refcount::emit_dup_refcounted( + builder, + self.frame, + self.root.layout.flags, + value.payload, + copy, + helper, + ); + builder.switch_to_block(copy); + opt_store(builder, output_base, output_index, value); + builder.ins().jump(done, &[]); + builder.switch_to_block(helper); + self.helper( + builder, + qjs::JSJitHelperId_JS_JIT_HELPER_DUP, + &[0, output, input], + pc, + live, + exception_depth, + )?; + builder.ins().jump(done, &[]); + builder.switch_to_block(done); + Ok(()) } fn check(&self, builder: &mut FunctionBuilder<'_>, pc: u32) -> Result<(), CompileFailure> { let status = runtime_call( @@ -432,7 +536,7 @@ pub(super) fn emit( opt_store(builder, env.stack_base, index, pair); } set_pc(builder, env, env.frame, node.pc()); - opt_own_stack_for_exit( + opt_own_stack_for_helper( builder, env.frame, env.sret, @@ -749,14 +853,7 @@ fn emit_body( state.resume(builder, instruction.pc, false)?; builder.switch_to_block(valid); } - state.helper( - builder, - qjs::JSJitHelperId_JS_JIT_HELPER_DUP, - &[0, state.flat(depth)?, input], - next, - depth + 1, - depth, - )?; + state.dup(builder, state.flat(depth)?, input, next, depth + 1, depth)?; } IrOp::PutArgument { index, keep } | IrOp::PutLocal { index, keep } => { let (buffer, slot) = if matches!(instruction.op, IrOp::PutArgument { .. }) { @@ -769,14 +866,7 @@ fn emit_body( }; state.free(builder, slot, next, depth, depth)?; if keep { - state.helper( - builder, - qjs::JSJitHelperId_JS_JIT_HELPER_DUP, - &[0, slot, state.flat(depth - 1)?], - next, - depth, - depth, - )?; + state.dup(builder, slot, state.flat(depth - 1)?, next, depth, depth)?; } else { let pair = opt_load(builder, state.stack, depth - 1); opt_store(builder, buffer, usize::from(index), pair); @@ -818,10 +908,10 @@ fn emit_body( opt_store(builder, state.locals, usize::from(index), empty); } IrOp::Drop => state.free(builder, state.flat(depth - 1)?, next, depth, depth)?, - IrOp::Stack(StackOp::Dup) => state.helper( + IrOp::Stack(StackOp::Dup) => state.dup( builder, - qjs::JSJitHelperId_JS_JIT_HELPER_DUP, - &[0, state.flat(depth)?, state.flat(depth - 1)?], + state.flat(depth)?, + state.flat(depth - 1)?, next, depth + 1, depth, diff --git a/jit/src/compiler/refcount.rs b/jit/src/compiler/refcount.rs new file mode 100644 index 000000000..afeb684fc --- /dev/null +++ b/jit/src/compiler/refcount.rs @@ -0,0 +1,243 @@ +//! Inline QuickJS reference-count fast paths. +//! +//! `JS_JitHelperDup` and `JS_JitHelperFree` are exact `js_dup` and +//! `JS_FreeValue` operations behind a full helper-frame validation. Generated +//! code performs the non-finalizing part of both operations directly: +//! +//! * DUP of a refcounted value increments `ref_count`; it can never allocate, +//! finalize, throw or reenter. +//! * FREE of a refcounted value whose `ref_count` is greater than one only +//! decrements it. When the count would reach zero the helper still runs, so +//! finalizers, GC list maintenance and reentrancy stay in C. +//! +//! Frames with `JS_JIT_FRAME_STRESS_GC` keep the helpers, because collecting at +//! every helper boundary belongs to the stress contract. The header offset is +//! pinned by the ABI value-layout fingerprint (see +//! `sys/patches/0022-inline-refcount.patch`), which is validated before any +//! artifact is compiled or entered. + +use cranelift_codegen::ir::{condcodes::IntCC, types, Block, InstBuilder, MemFlags, Value}; +use cranelift_frontend::FunctionBuilder; +use rquickjs_core::qjs; + +/// Offset of the in-place count, pinned by the ABI value-layout fingerprint. +use crate::abi::REF_COUNT_OFFSET; +const _: () = assert!(crate::abi::REF_COUNT_BYTES == 4); + +/// Exact `JS_VALUE_HAS_REF_COUNT`: QuickJS compares the tag as `unsigned`. +pub(super) fn emit_has_ref_count(builder: &mut FunctionBuilder<'_>, tag: Value) -> Value { + let tag = if builder.func.dfg.value_type(tag) == types::I32 { + tag + } else { + builder.ins().ireduce(types::I32, tag) + }; + builder.ins().icmp_imm( + IntCC::UnsignedGreaterThanOrEqual, + tag, + i64::from(qjs::JS_TAG_FIRST), + ) +} + +/// True unless the frame requests a GC at every helper boundary. +pub(super) fn emit_no_stress(builder: &mut FunctionBuilder<'_>, frame: Value, flags: i32) -> Value { + let flags = builder + .ins() + .load(types::I32, MemFlags::new(), frame, flags); + let stress = builder + .ins() + .band_imm(flags, i64::from(qjs::JS_JIT_FRAME_STRESS_GC)); + builder.ins().icmp_imm(IntCC::Equal, stress, 0) +} + +/// Duplicates `payload`, which the caller proved refcounted, unless the frame +/// is in stress mode. Continues in `done` after the increment, or in `helper` +/// with no side effect. +pub(super) fn emit_dup_refcounted( + builder: &mut FunctionBuilder<'_>, + frame: Value, + flags: i32, + payload: Value, + done: Block, + helper: Block, +) { + let no_stress = emit_no_stress(builder, frame, flags); + let increment = builder.create_block(); + builder.ins().brif(no_stress, increment, &[], helper, &[]); + builder.seal_block(increment); + builder.switch_to_block(increment); + emit_increment(builder, payload); + builder.ins().jump(done, &[]); +} + +/// Unconditional `ref_count++` of a payload known to be refcounted. +pub(super) fn emit_increment(builder: &mut FunctionBuilder<'_>, payload: Value) { + let count = builder + .ins() + .load(types::I32, MemFlags::trusted(), payload, REF_COUNT_OFFSET); + let count = builder.ins().iadd_imm(count, 1); + builder + .ins() + .store(MemFlags::trusted(), count, payload, REF_COUNT_OFFSET); +} + +/// Releases `payload`, which the caller proved refcounted, without a helper +/// when the reference is not the last one. Continues in `released` after the +/// decrement, or in `helper` with the count untouched (last reference or +/// stress mode) so the helper performs the exact `JS_FreeValue`. +pub(super) fn emit_release_refcounted( + builder: &mut FunctionBuilder<'_>, + frame: Value, + flags: i32, + payload: Value, + released: Block, + helper: Block, +) { + let no_stress = emit_no_stress(builder, frame, flags); + let check = builder.create_block(); + builder.ins().brif(no_stress, check, &[], helper, &[]); + builder.seal_block(check); + builder.switch_to_block(check); + let count = builder + .ins() + .load(types::I32, MemFlags::trusted(), payload, REF_COUNT_OFFSET); + let shared = builder.ins().icmp_imm(IntCC::SignedGreaterThan, count, 1); + let decrement = builder.create_block(); + builder.ins().brif(shared, decrement, &[], helper, &[]); + builder.seal_block(decrement); + builder.switch_to_block(decrement); + let count = builder.ins().iadd_imm(count, -1); + builder + .ins() + .store(MemFlags::trusted(), count, payload, REF_COUNT_OFFSET); + builder.ins().jump(released, &[]); +} + +#[cfg(test)] +mod tests { + use super::*; + use cranelift_codegen::{ + ir::{AbiParam, Function, Signature}, + settings, + }; + use cranelift_frontend::FunctionBuilderContext; + + /// fn(flags: *const u32, tag: i64, payload: *mut i32) -> i32 + /// returns 1 for the inline path and 0 for the helper path. + fn compile(release: bool) -> super::super::baseline::PublishedBaselineCode { + let isa = cranelift_native::builder() + .unwrap() + .finish(settings::Flags::new(settings::builder())) + .unwrap(); + let mut signature = Signature::new(isa.default_call_conv()); + signature.params.push(AbiParam::new(isa.pointer_type())); + signature.params.push(AbiParam::new(types::I64)); + signature.params.push(AbiParam::new(isa.pointer_type())); + signature.returns.push(AbiParam::new(types::I32)); + let mut function = Function::with_name_signature(Default::default(), signature); + let mut context = FunctionBuilderContext::new(); + { + let mut builder = FunctionBuilder::new(&mut function, &mut context); + let entry = builder.create_block(); + let refcounted = builder.create_block(); + let inline = builder.create_block(); + let helper = builder.create_block(); + let primitive = builder.create_block(); + builder.append_block_params_for_function_params(entry); + builder.switch_to_block(entry); + let parameters = builder.block_params(entry).to_vec(); + let has = emit_has_ref_count(&mut builder, parameters[1]); + builder.ins().brif(has, refcounted, &[], primitive, &[]); + builder.switch_to_block(refcounted); + if release { + emit_release_refcounted( + &mut builder, + parameters[0], + 0, + parameters[2], + inline, + helper, + ); + } else { + emit_dup_refcounted( + &mut builder, + parameters[0], + 0, + parameters[2], + inline, + helper, + ); + } + builder.switch_to_block(inline); + let one = builder.ins().iconst(types::I32, 1); + builder.ins().return_(&[one]); + builder.switch_to_block(helper); + let zero = builder.ins().iconst(types::I32, 0); + builder.ins().return_(&[zero]); + builder.switch_to_block(primitive); + let two = builder.ins().iconst(types::I32, 2); + builder.ins().return_(&[two]); + builder.seal_all_blocks(); + builder.finalize(); + } + super::super::baseline::finalize_optimized_machine(&isa, function, None, false) + .unwrap() + .publish() + .unwrap() + } + + type Probe = unsafe extern "C" fn(*const u32, i64, *mut i32) -> i32; + + #[test] + fn dup_increments_only_refcounted_payloads_outside_stress() { + let code = compile(false); + // SAFETY: the generated function has exactly this ABI and only touches + // the supplied flag word and count. + let probe: Probe = unsafe { core::mem::transmute(code.as_ptr()) }; + let flags = 0_u32; + let mut count = 1_i32; + for tag in [qjs::JS_TAG_OBJECT, qjs::JS_TAG_STRING, qjs::JS_TAG_BIG_INT] { + assert_eq!(unsafe { probe(&flags, i64::from(tag), &mut count) }, 1); + } + assert_eq!(count, 4); + for tag in [ + qjs::JS_TAG_INT, + qjs::JS_TAG_UNDEFINED, + qjs::JS_TAG_FLOAT64, + -10, + ] { + assert_eq!(unsafe { probe(&flags, i64::from(tag), &mut count) }, 2); + } + let stress = qjs::JS_JIT_FRAME_STRESS_GC; + assert_eq!( + unsafe { probe(&stress, i64::from(qjs::JS_TAG_OBJECT), &mut count) }, + 0 + ); + assert_eq!(count, 4); + } + + #[test] + fn release_defers_last_reference_and_stress_to_helper() { + let code = compile(true); + // SAFETY: see dup_increments_only_refcounted_payloads_outside_stress. + let probe: Probe = unsafe { core::mem::transmute(code.as_ptr()) }; + let flags = 0_u32; + let mut count = 3_i32; + let object = i64::from(qjs::JS_TAG_OBJECT); + assert_eq!(unsafe { probe(&flags, object, &mut count) }, 1); + assert_eq!(count, 2); + assert_eq!(unsafe { probe(&flags, object, &mut count) }, 1); + assert_eq!(count, 1); + // The last reference must reach JS_FreeValue untouched. + assert_eq!(unsafe { probe(&flags, object, &mut count) }, 0); + assert_eq!(count, 1); + count = 5; + let stress = qjs::JS_JIT_FRAME_STRESS_GC; + assert_eq!(unsafe { probe(&stress, object, &mut count) }, 0); + assert_eq!(count, 5); + assert_eq!( + unsafe { probe(&flags, i64::from(qjs::JS_TAG_INT), &mut count) }, + 2 + ); + assert_eq!(count, 5); + } +} diff --git a/jit/tests/frame_inline.rs b/jit/tests/frame_inline.rs index 9443ecaca..5c5ad2b67 100644 --- a/jit/tests/frame_inline.rs +++ b/jit/tests/frame_inline.rs @@ -380,7 +380,12 @@ fn duplicated_frame_inline_result_preserves_owners_through_gc_and_deopt() { "heap truthiness must resume with both owners: {after:?}" ); assert_eq!(test.count(qjs::JSJitHelperId_JS_JIT_HELPER_CALL), calls); - assert!(test.count(qjs::JSJitHelperId_JS_JIT_HELPER_DUP) > duplicates); + // Outside stress GC the second owner is taken with an inline + // `ref_count++`; the surviving marker above is the ownership + // evidence. Stress GC keeps the helper and its collection points. + if stress { + assert!(test.count(qjs::JSJitHelperId_JS_JIT_HELPER_DUP) > duplicates); + } assert_eq!(after.native_entries, after.native_exits); assert_eq!(after.compile_failures, 0, "stress={stress}: {after:?}"); } diff --git a/jit/tests/generic_call_cleanup.rs b/jit/tests/generic_call_cleanup.rs index 0a0fc6729..f128dbb33 100644 --- a/jit/tests/generic_call_cleanup.rs +++ b/jit/tests/generic_call_cleanup.rs @@ -128,13 +128,17 @@ fn exercise_cleanup(force_tier2: bool, stress_gc: bool) { } // Four consumed operands: the function and object own references; Int32 // and Bool need only slot clearing, except in helper-observable stress GC. - // Tier2 also frees two local/return slots outside CALL consumption; those - // transitions are unchanged by this optimization. - let expected_call_frees = if stress_gc { 4 } else { 2 }; + // The function and object are also held by globals, so their consumed + // references are never the last ones: generated code decrements them + // inline and only a final release (or stress GC) reaches FREE. Tier2 also + // frees two local/return slots outside CALL consumption; those are + // primitives or shared owners here and likewise stay inline without + // stress GC. + let expected_call_frees = if stress_gc { 4 } else { 0 }; assert_eq!( helper_count(qjs::JSJitHelperId_JS_JIT_HELPER_FREE) - frees, - expected_call_frees + if force_tier2 { 2 } else { 0 }, - "primitive call operands still invoke FREE: {trace:?}" + expected_call_frees + if force_tier2 && stress_gc { 2 } else { 0 }, + "shared or primitive call operands still invoke FREE: {trace:?}" ); if !force_tier2 { assert_eq!( diff --git a/jit/tests/inline_refcount.rs b/jit/tests/inline_refcount.rs new file mode 100644 index 000000000..a21fb66ca --- /dev/null +++ b/jit/tests/inline_refcount.rs @@ -0,0 +1,311 @@ +//! Inline DUP/FREE reference-count fast paths must keep interpreter results, +//! ownership and finalization exact in every tier and in stress-GC mode. +#![cfg(all( + feature = "compiler", + feature = "test-support", + any(target_os = "linux", target_os = "macos"), + target_endian = "little", + any(target_arch = "x86_64", target_arch = "aarch64") +))] + +use rquickjs::{CatchResultExt, Context, Runtime}; +use rquickjs_core::qjs; +use rquickjs_jit::{Jit, JitConfig, JitTierPolicy}; + +unsafe extern "C" { + fn JS_JitGetHelperCount(rt: *mut qjs::JSRuntime, helper: u32, count: *mut u64) -> i32; +} + +#[derive(Clone, Copy, Debug)] +enum Mode { + Interpreter, + Tier1, + Tier2, + Automatic, +} + +const DRIVER: &str = r#" +var results = []; +function batch(count) { + results = new Array(count); + function callWorkload() { return workload(200, 3); } + function next(index) { + for (let i = index; i < count; i++) { + const result = callWorkload(); + results[i] = result; + } + } + return next(0); +} +function checksum() { + return results.map(function(value) { + if (typeof value === 'number') return 'number:' + value; + if (typeof value === 'string') return 'string:' + value; + return typeof value + ':' + String(value); + }).join('|'); +} +"#; + +/// Runs `batch(4)` repeatedly and checks every round against `expected` +/// (workloads keep no state across calls). JIT modes keep going until native +/// code has run for at least ten rounds, so background compilation latency on +/// a loaded host cannot turn the comparison into an interpreter-only one. +fn run(mode: Mode, stress: bool, source: &str, expected: Option<&str>) -> String { + let runtime = Runtime::new().unwrap(); + let jit = match mode { + Mode::Interpreter => None, + _ => { + let mut builder = JitConfig::builder().stress_gc(stress); + builder = match mode { + Mode::Tier1 => builder + .call_threshold(2) + .loop_threshold(1) + .tier_policy(JitTierPolicy::BaselineOnly), + Mode::Tier2 => builder + .call_threshold(1) + .loop_threshold(1) + .tier_policy(JitTierPolicy::Optimize) + .force_optimized_for_test(true), + _ => builder, + }; + Some(Jit::attach(&runtime, builder.build().unwrap()).unwrap()) + } + }; + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| { + ctx.eval::<(), _>(source).catch(&ctx).unwrap(); + ctx.eval::<(), _>(DRIVER).catch(&ctx).unwrap(); + }); + let round = || { + context.with(|ctx| { + ctx.eval::<(), _>("batch(4)") + .catch(&ctx) + .unwrap_or_else(|error| panic!("{mode:?} stress={stress}: {error}")); + ctx.eval::("checksum()").catch(&ctx).unwrap() + }) + }; + let first = round(); + let Some(jit) = jit else { + return first; + }; + assert_eq!(Some(first.as_str()), expected, "{mode:?} stress={stress}"); + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(60); + let mut native_rounds = 0; + let mut rounds = 1; + // Production tiering may legitimately demote an unprofitable function to + // the interpreter, so only forced tiers must keep running native code. + let required_native_rounds = if matches!(mode, Mode::Automatic) { + 0 + } else { + 10 + }; + while rounds < 40 || native_rounds < required_native_rounds { + let before = jit.metrics().native_entries; + assert_eq!(Some(round().as_str()), expected, "{mode:?} stress={stress}"); + rounds += 1; + if jit.metrics().native_entries > before { + native_rounds += 1; + } + jit.poll(); + assert!( + std::time::Instant::now() < deadline, + "{mode:?} stress={stress} never ran native code: {:?}", + jit.metrics() + ); + if native_rounds < required_native_rounds { + std::thread::sleep(std::time::Duration::from_millis(1)); + } + } + context.with(|ctx| ctx.run_gc()); + first +} + +fn assert_all_tiers(source: &str) { + let expected = run(Mode::Interpreter, false, source, None); + for mode in [Mode::Tier1, Mode::Tier2, Mode::Automatic] { + for stress in [false, true] { + run(mode, stress, source, Some(&expected)); + } + } +} + +#[test] +fn scalar_loop_under_benchmark_driver() { + assert_all_tiers( + "function workload(n, seed) { let s = 0; for (let i = 0; i < n; i++) s += i; return s + seed; }", + ); +} + +#[test] +fn shared_object_references_survive_repeated_dup_and_free() { + assert_all_tiers( + r#" + const shared = { x: 1, nested: { y: 2 } }; + function workload(n, seed) { + let sum = 0; + for (let i = 0; i < n; i++) { + let a = shared; + let b = a; + let c = b.nested; + a = null; + sum += b.x + c.y; + } + return sum + seed; + } + "#, + ); +} + +#[test] +fn last_reference_is_finalized_exactly_once() { + assert_all_tiers( + r#" + function workload(n, seed) { + const registry = []; + let live = 0; + for (let i = 0; i < n; i++) { + let object = { i, text: 'v' + i }; + let alias = object; + object = undefined; + live += alias.i; + alias = undefined; + if ((i & 31) === 0) registry.push({ i }); + if (registry.length > 8) registry.length = 0; + } + return live + registry.length + seed; + } + "#, + ); +} + +#[test] +fn polymorphic_objects_and_strings() { + assert_all_tiers( + r#" + function workload(iterations, seed) { + const objects = []; + for (let i = 0; i < iterations; i++) { + let object; + switch (i & 3) { + case 0: object = { x: i, y: seed, kind: 0 }; break; + case 1: object = { y: seed, x: i, extra: 'e' + i, kind: 1 }; break; + case 2: object = { x: i, kind: 2 }; object.y = seed; break; + default: object = Object.create(null); object.kind = 3; object.x = i; object.y = seed; + } + object.x += object.kind; + objects.push(object); + } + let sum = 0; + for (let i = 0; i < objects.length; i++) sum += objects[i].x + objects[i].y; + return sum; + } + "#, + ); +} + +#[test] +fn generic_calls_release_consumed_operands() { + assert_all_tiers( + r#" + const box = { v: 7 }; + function id(value) { return value; } + function pick(a, b) { return a.v > b.v ? a : b; } + function workload(n, seed) { + let best = box; + let s = ''; + for (let i = 0; i < n; i++) { + const candidate = { v: (i * 7) % 13 }; + best = pick(id(best), candidate); + s = id(s.length > 8 ? '' : s + 'x'); + } + return best.v + s.length + seed; + } + "#, + ); +} + +/// Outside stress GC, a shared heap reference is duplicated and released +/// without DUP/FREE helpers, while the last reference still reaches the FREE +/// helper (the only path that may finalize). Stress GC keeps every helper. +#[test] +fn helpers_run_only_for_last_references_or_stress() { + for stress in [false, true] { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .call_threshold(2) + .loop_threshold(1) + .tier_policy(JitTierPolicy::BaselineOnly) + .stress_gc(stress) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| { + ctx.eval::<(), _>( + r#" + const shared = { v: 1 }; + function sharedOnly(n) { + let sum = 0; + for (let i = 0; i < n; i++) { let a = shared; let b = a; sum += b.v; a = b = undefined; } + return sum; + } + function lastOnly(n) { + let sum = 0; + for (let i = 0; i < n; i++) { let a = { v: i }; sum += a.v; a = undefined; } + return sum; + } + "#, + ) + .catch(&ctx) + .unwrap(); + }); + let rt = context.with(|ctx| unsafe { qjs::JS_GetRuntime(ctx.as_raw().as_ptr()) }); + let count = |helper| { + let mut value = 0; + assert_eq!(unsafe { JS_JitGetHelperCount(rt, helper, &mut value) }, 0); + value + }; + let call = + |source: &str| context.with(|ctx| ctx.eval::(source).catch(&ctx).unwrap()); + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(60); + loop { + let before = jit.metrics().native_entries; + assert_eq!(call("sharedOnly(64)"), 64); + assert_eq!(call("lastOnly(64)"), 2016); + jit.poll(); + let metrics = jit.metrics(); + if metrics.native_entries >= before + 2 && metrics.pending_worker_jobs == 0 { + break; + } + assert!( + std::time::Instant::now() < deadline, + "never native: {metrics:?}" + ); + std::thread::sleep(std::time::Duration::from_millis(1)); + } + let dup = qjs::JSJitHelperId_JS_JIT_HELPER_DUP; + let free = qjs::JSJitHelperId_JS_JIT_HELPER_FREE; + let entries = jit.metrics().native_entries; + let (dups, frees) = (count(dup), count(free)); + assert_eq!(call("sharedOnly(64)"), 64); + assert_eq!(jit.metrics().native_entries, entries + 1); + if stress { + assert!(count(dup) > dups, "stress GC must keep DUP helpers"); + assert!(count(free) > frees, "stress GC must keep FREE helpers"); + } else { + assert_eq!(count(dup), dups, "shared DUP reached the helper"); + assert_eq!(count(free), frees, "shared FREE reached the helper"); + } + let frees = count(free); + assert_eq!(call("lastOnly(64)"), 2016); + assert_eq!(jit.metrics().native_entries, entries + 2); + assert!( + count(free) >= frees + 64, + "every last reference must reach the FREE helper" + ); + runtime.run_gc(); + } +} diff --git a/jit/tests/property_specialization.rs b/jit/tests/property_specialization.rs index e87948945..38d5fa96b 100644 --- a/jit/tests/property_specialization.rs +++ b/jit/tests/property_specialization.rs @@ -437,7 +437,14 @@ fn guarded_property_path(clif: &str, shape: i64, store_offset: Option) -> R .then_some(value) }) .collect(); - if roots.len() != 2 { + // Cold owner-materialization paths may reload a root buffer pointer from + // the frame (inline `js_dup` of an argument/local), so every such load is + // a root; both buffers must be present. + if !root_loads.iter().all(|address| { + definitions + .values() + .any(|definition| definition.starts_with("load") && definition.ends_with(address)) + }) { return Err("missing argument/local root buffers".into()); } let mut edges = BTreeMap::<&str, Vec<&str>>::new(); From ac528303e3dcc7c5922b1f2c6af69969fec3ddd4 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:13:19 +0800 Subject: [PATCH 24/54] docs: record P1.3 inline refcount slice and remaining work Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p1b-inline-refcount.md | 141 ++++++++++++++++++++++++++++ 1 file changed, 141 insertions(+) create mode 100644 docs/roadmap/p1b-inline-refcount.md diff --git a/docs/roadmap/p1b-inline-refcount.md b/docs/roadmap/p1b-inline-refcount.md new file mode 100644 index 000000000..5835397f1 --- /dev/null +++ b/docs/roadmap/p1b-inline-refcount.md @@ -0,0 +1,141 @@ +# P1 item 3: inline DUP/FREE reference counting + +Branch `perf/p1b-inline-refcount`, based on `82d3808` (0.12.9). This note +covers roadmap item P1.3 in +[PERFORMANCE_ROADMAP_V8_BUN.md](../PERFORMANCE_ROADMAP_V8_BUN.md): generated +code updates QuickJS reference counts directly instead of calling +`JS_JitHelperDup` / `JS_JitHelperFree` for every ownership transition. + +## What changed + +- `jit/src/compiler/refcount.rs` holds the shared Cranelift emitters: + - `JS_VALUE_HAS_REF_COUNT` compares the tag as `unsigned`, exactly as C + does. + - DUP is `ref_count++`. + - FREE is `ref_count--` only while the count is greater than one. + - Frame-flag checks keep stress GC on the helpers. +- The FREE helper still runs for the last reference, so finalizers, GC list + maintenance, and reentrancy stay in C. It also still runs for every value in + `JS_JIT_FRAME_STRESS_GC` frames, which keeps the stress collection points. + Sites that used to call the helper for primitives keep doing so in stress + mode (`lower_free_always_in_stress`, the `GetLocalPair` DUP, and frame-inline + DUP/FREE). +- Tier 1 (`baseline.rs`): + - These operations duplicate inline: get/put argument and local, the checked + local, `dup`/`dup1..3`/`insert*`, and `get_loc0_loc1`. + - These release inline: put, `drop`, `nip`, the `is_undefined`/`is_null` + family, property receivers, and consumed generic CALL operands. + - An inline release writes undefined into the consumed slot's frame memory, + as the helper does. CALL and GET_PROPERTY outputs are cleared from scratch + memory after the helper result is reloaded into SSA + (`clear_reloaded_output`). Before this change, the following FREE helper's + frame materialization cleared them implicitly. Exits reject non-undefined + scratch slots, so without the clear the exit fails with "owned JIT scratch + at exit". +- Tier 2 (`optimized.rs`, `optimized/frame_inline.rs`): + - FREE of owned stack and local slots goes through `emit_opt_free_slot`. + This covers consumed operands of specialized CALLs and the receiver of + owned GetField. + - `dup` of an `OwnedSlot` goes through `emit_opt_owned_dup`. It is inline + when no borrowed alias is live, and stress mode keeps the old + helper-and-materialize path with identical provenance. + - Shadow-frame DUP/FREE are inline. + - Borrowed-alias owner materialization + (`JS_JitHelperMaterializeOwner`, which is a `js_dup`) is inline on paths + that continue into helper or runtime calls (`opt_own_stack_for_helper`). + Deopt and exception bridges keep the helper (`opt_own_stack_for_exit`): + inlining them measurably perturbed hot-loop code + (`int32array-traversal` was 0.94x). + - Out-of-line placement: + - The direct-call signature deopt block is now a cold block. + - The helper continuation of owner materialization inherits the caller's + coldness. Stress-only blocks are cold. +- ABI and layout: `sys/patches/0022-inline-refcount.patch` adds static + assertions that every refcounted payload starts with the C `int` count. + These cover objects, bytecode, strings, ropes, symbols, BigInt, and modules. + The patch also hashes the count's offset and width into the value-layout + fingerprint. `jit/src/abi.rs` pins `REF_COUNT_OFFSET = 0` and + `REF_COUNT_BYTES = 4` in the same fingerprint, so a library without the patch + fails ABI validation and the JIT stays closed. No ABI table or struct changed + and the ABI minor version is unchanged. Bindings are unchanged. + +## Helper counter semantics + +`JS_JitGetHelperCounters` and `JS_JitGetHelperCount` now count only the +helper calls that still happen: last references, stress GC, and cold bridges. +Tests were updated on purpose: + +- `generic_call_cleanup`: shared operands no longer reach FREE outside stress + mode. +- `frame_inline`: the duplicated owner uses the helper only under stress. The + surviving marker after GC remains the ownership evidence. +- `property_specialization`: an inline owner reload of `arg_buf`/`var_buf` + counts as a root load. + +`jit/tests/inline_refcount.rs` adds differential checks against the +interpreter in Tier 1, Tier 2, and automatic mode, each with and without +stress GC. Forced tiers must keep running native code. The file also asserts +that shared DUP/FREE skip the helpers while every last reference still +reaches FREE. + +## Verification (this branch) + +- `cargo test --release -p quickjs-jit-runtime --features compiler,test-support --tests`: + 791 passed, 1 failed, 1 ignored. The single failure is + `mixed_direct_calls::tier2_mixed_edge_eliminates_helper_and_preserves_guard_misses`. + That flake depends on wall-clock profitability and also fails on unmodified + `82d3808` under the shared machine's load (5 of 12 runs failed there). + `optimized::automatic_call_heavy_promotes_the_direct_edge_caller` is a + similar timing flake, with 2 of 12 runs failing on `82d3808`. An earlier + full run on this branch passed all 792 tests. +- The `jit_patch` test in `quickjs-jit-sys` passes 13 of 13. Clippy with + `-D warnings` is clean, both with and without the compiler features. + +## Noisy diagnostic (not publishable) + +- Protocol: `jit-bench worker --mode automatic`, pinned to P-cores + (`taskset -c 0-15`) because E-core placement makes samples bimodal. +- 9 alternating baseline and candidate processes per workload. Each value is + the median over processes of `median(warmup_batch_ns[-16:])`. +- Baseline: `/home/jason/work/quickjs-jit/target/release/jit-bench` built at + `82d3808`. +- Host: i7-13700KF, shared with about 15 other agents. +- Checksums matched interpreter mode in every run. +- Speed is baseline time divided by new time. Values above 1.00x are faster. + +| Workload | 82d3808 (ms) | Branch (ms) | Speed | +| --- | ---: | ---: | ---: | +| objects-polymorphic | 23.878 | 19.342 | 1.23x | +| generic-call-fallback | 9.119 | 6.611 | 1.38x | +| mixed-quotes | 2.492 | 2.535 | 0.98x | +| arrays-typed | 2.980 | 2.810 | 1.06x | +| property-heavy | 0.099 | 0.099 | 1.00x | +| call-heavy | 0.060 | 0.044 | 1.35x | +| numeric | 0.029 | 0.029 | 1.01x | +| scalar-loop | 0.029 | 0.029 | 1.01x | +| int32array-traversal | 0.050 | 0.050 | 1.00x | +| packed-array-traversal | 0.048 | 0.048 | 1.01x | +| float64array-traversal | 1.091 | 0.858 | 1.27x | +| calls-closures | 3.304 | 3.364 | 0.98x | +| fibonacci-iterative | 0.997 | 0.989 | 1.01x | + +The two results near 0.98x are mixed-quotes and calls-closures. Both spend +most of their time in the interpreter and in C: sort-callback bookkeeping and +closure opcodes that are still rejected. Interpreter mode shows a similar +build-to-build spread: mixed-quotes 0.99x and calls-closures 1.02x. Treat them +as unresolved until the paired three-engine matrix runs. The `call-heavy` gain +comes mostly from the cold direct-call deopt layout, not from reference +counting. + +## Remaining work + +- `lower_define_element` (Tier 1) still calls the DUP helper twice per + element definition. +- Inlining owner materialization on deopt bridges is possible only once + those bridges are laid out cold everywhere. That needs every deopt caller to + mark its fail block cold; `emit_opt_deopt`, the element get/put guards, and + the property-key guards currently do not. +- An owned `dup` with a live borrowed alias below it still uses the helper + path with full frame materialization. +- The publishable three-engine matrix and README update are left to the + integrator. From f4c2647f816b59a2bf1bfd66c6873933c1efd5ae Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:15:02 +0800 Subject: [PATCH 25/54] perf(jit): skip unread backtraces when natively catching primitives The interpreter's exception label builds a backtrace for every caught primitive or non-Error value only to store it in ctx->error_back_trace, which the catch then releases unread. JS_JitCatchException now skips that build when it cannot run user code (Error.prepareStackTrace unset, numeric Error.stackTraceLimit) and cannot define `stack` on the caught value. A regression test checks both hooks still fire exactly as in the interpreter for natively caught throws. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p2c-exceptions.md | 12 ++++--- jit/tests/tier1_exceptions.rs | 17 ++++++++++ sys/build_support/patch.rs | 4 +-- .../0027-tier1-exception-regions.patch | 31 ++++++++++++++++--- sys/patches/README.md | 9 ++++-- 5 files changed, 60 insertions(+), 13 deletions(-) diff --git a/docs/roadmap/p2c-exceptions.md b/docs/roadmap/p2c-exceptions.md index e94cda889..0a1d6fda3 100644 --- a/docs/roadmap/p2c-exceptions.md +++ b/docs/roadmap/p2c-exceptions.md @@ -26,6 +26,12 @@ Branch `perf/p2c-exceptions`, based on `82d3808`. Roadmap item P2.3 in interrupts and invariant failures leave through the normal exception exit so the interpreter unwinds the published frame exactly. - `JS_JitThrowValue` / `JS_JitThrowError` implement `throw` / `throw_error`. +- The interpreter builds a backtrace string for every caught primitive or + non-Error throw only to store it in `ctx->error_back_trace`, which the catch + releases unread. The native catch skips that build when it cannot run user + code (`Error.prepareStackTrace` unset, numeric `Error.stackTraceLimit`) and + cannot define `stack` on the caught value; a regression test checks both + hooks still fire exactly as in the interpreter. - The entry-domain analysis merges every exceptional edge (before, after and mid-store states) into handler inputs, so `get_loc_check` proofs stay sound. - Tier 2, bounded inlining, tagged direct links and inline snapshot retention @@ -74,8 +80,6 @@ three-engine matrix, including `exceptions-sync`, is left to the integrator. the direct form is lowered; others reject the compile). - For-of/for-in iterator close offsets (`catch` offset 0) inside try regions: rejected until the iteration opcodes are admitted (P2.5). -- Every native catch still performs the interpreter's backtrace build for - primitive throws (`ctx->error_back_trace`), which dominates throw-heavy - loops; skipping it when it is provably unobservable is a runtime-library - change, not a JIT one. - Helper-frame validation (P1 B1) is paid twice per native throw/catch. +- The interpreter could skip the same unread backtrace build; that is a + runtime-library change outside this item. diff --git a/jit/tests/tier1_exceptions.rs b/jit/tests/tier1_exceptions.rs index a4518b945..3917d60f7 100644 --- a/jit/tests/tier1_exceptions.rs +++ b/jit/tests/tier1_exceptions.rs @@ -97,6 +97,23 @@ fn error_stack_is_captured_at_the_throwing_pc() { ); } +#[test] +fn observable_backtrace_hooks_run_for_natively_caught_primitives() { + // The interpreter builds a backtrace for every caught primitive. It is + // skipped natively only when building it cannot run user code; these + // hooks make it observable and must fire exactly as in the interpreter. + same( + "var calls = 0; Error.prepareStackTrace = function(e, s) { calls++; return 'trace'; };\nfunction f(n){ let c = 0; for (let i = 0; i < n; i++) { try { if (i % 4 === 0) throw i; } catch (v) { c -= -v; } } return c + ':' + calls; }", + "f(40)", + "neg", + ); + same( + "var calls = 0; Error.stackTraceLimit = { valueOf() { calls++; return 3; } };\nfunction f(n){ let c = 0; for (let i = 0; i < n; i++) { try { if (i % 4 === 0) throw {i}; } catch (v) { c -= -v.i; } } return c + ':' + calls; }", + "f(40)", + "neg", + ); +} + #[test] fn rethrow_and_nested_regions_match_the_interpreter() { same( diff --git a/sys/build_support/patch.rs b/sys/build_support/patch.rs index 9823306c2..57edd4624 100644 --- a/sys/build_support/patch.rs +++ b/sys/build_support/patch.rs @@ -21,7 +21,7 @@ const EXPECTED_PATCHES: [(&str, u64); 20] = [ ("0018-inline-frame-recovery.patch", 0xb971d61edccde229), ("0019-array-feedback.patch", 0x11c3cb13a8297dd4), ("0020-typed-array-guard.patch", 0x5d10_38a3_8bc9_9821), - ("0027-tier1-exception-regions.patch", 0x28fab1b01554c253), + ("0027-tier1-exception-regions.patch", 0x63478d4d077fdced), ]; pub(crate) const BASELINE_FILES: [&str; 19] = [ "api-test.c", @@ -61,7 +61,7 @@ const PATCHED_FILE_FINGERPRINTS: [(&str, u64); 21] = [ ("list.h", 0xb337_70f7_b76d_a3d8), ("quickjs-atom.h", 0x30b4_9116_b6a2_aa99), ("quickjs-c-atomics.h", 0x490b_0f29_f631_3fc0), - ("quickjs.c", 0x3e661e5849961ea9), + ("quickjs.c", 0x02f230042e7bc62d), ("quickjs-jit.h", 0xb748adfe5e91c323), ("quickjs-jit-helpers.h", 0xa469_e80f_be47_404c), ("quickjs-opcode.h", 0x3d05_cfdf_5cf7_2930), diff --git a/sys/patches/0027-tier1-exception-regions.patch b/sys/patches/0027-tier1-exception-regions.patch index b0b463bad..244517036 100644 --- a/sys/patches/0027-tier1-exception-regions.patch +++ b/sys/patches/0027-tier1-exception-regions.patch @@ -42,7 +42,7 @@ diff --git a/quickjs-jit.h b/quickjs-jit.h diff --git a/quickjs.c b/quickjs.c --- a/quickjs.c +++ b/quickjs.c -@@ -20431,6 +20431,128 @@ +@@ -20431,6 +20431,151 @@ return version == QJSJIT_INLINE_RECOVERY_VERSION ? &api : NULL; } @@ -76,6 +76,24 @@ diff --git a/quickjs.c b/quickjs.c + return JS_JIT_HELPER_EXCEPTION; +} + ++/* True when the interpreter's exception label would build a backtrace only ++ * to store it in ctx->error_back_trace, which catching releases unread: ++ * building it can neither call Error.prepareStackTrace nor convert a ++ * non-numeric Error.stackTraceLimit, and cannot define `stack` on `exc`. */ ++static bool qjsjit_backtrace_is_discarded(JSContext *ctx, JSValueConst exc) ++{ ++ int limit_tag = JS_VALUE_GET_TAG(ctx->error_stack_trace_limit); ++ ++ if (ctx->rt->in_build_stack_trace) ++ return false; ++ if (can_add_backtrace(exc)) ++ return false; ++ if (limit_tag != JS_TAG_INT && limit_tag != JS_TAG_FLOAT64) ++ return false; ++ return JS_IsNull(ctx->error_ctor) || ++ !JS_IsFunction(ctx, ctx->error_prepare_stack); ++} ++ +/* OP_throw_error; `type` uses the interpreter's JS_THROW_* numbering. */ +JSJitHelperStatus JS_JitThrowError(JSJitExecFrame *frame, + uint32_t stack_map_id, uint32_t atom, @@ -141,9 +159,14 @@ diff --git a/quickjs.c b/quickjs.c + return qjsjit_reject_helper(frame, "invalid JIT catch nesting"); + } + /* Exception label prologue, in interpreter order: backtrace first (it -+ * observes this frame's current pc), then catchability. */ ++ * observes this frame's current pc), then catchability. A backtrace that ++ * would only populate ctx->error_back_trace is released again below ++ * without ever being read; skip building it when doing so cannot run ++ * user code (no Error.prepareStackTrace, numeric Error.stackTraceLimit) ++ * and cannot define a `stack` property on the caught value. */ + if (needs_backtrace(rt->current_exception) || -+ JS_IsUndefined(ctx->error_back_trace)) { ++ (JS_IsUndefined(ctx->error_back_trace) && ++ !qjsjit_backtrace_is_discarded(ctx, rt->current_exception))) { + sf->cur_pc = (uint8_t *)frame->pc; + build_backtrace(ctx, rt->current_exception, JS_UNDEFINED, + NULL, 0, 0, 0); @@ -171,7 +194,7 @@ diff --git a/quickjs.c b/quickjs.c static JSValue JS_CallInternal(JSContext *caller_ctx, JSValueConst func_obj, JSValueConst this_obj, JSValueConst new_target, int argc, JSValueConst *argv, int flags) -@@ -20691,12 +20813,25 @@ +@@ -20691,12 +20836,25 @@ b->jit_feedback_disabled = 1; if ((request_snapshot & 1U) && hot_vtable->submit_snapshot) { JSJitFunctionSnapshot *snapshot = NULL; diff --git a/sys/patches/README.md b/sys/patches/README.md index 13c5e88dd..8395f594b 100644 --- a/sys/patches/README.md +++ b/sys/patches/README.md @@ -66,9 +66,12 @@ points. `JS_JitThrowValue` and `JS_JitThrowError` raise exactly like interpreter's `exception:` label for the active root frame (backtrace at the throwing PC, then catchability) and, only after proving that unwinding would stop at the named catch-offset slot and handler PC, releases the operands -above it and stores the caught value in that slot. Uncatchable exceptions are -returned with the frame untouched so the ordinary native exception exit lets -the interpreter unwind them. The patch also stops hot and feedback probes for +above it and stores the caught value in that slot. A backtrace that would +only populate `ctx->error_back_trace` (released unread by the catch) is not +built when building it cannot run user code (`Error.prepareStackTrace`, a +non-numeric `Error.stackTraceLimit`) or define `stack` on the caught value. +Uncatchable exceptions are returned with the frame untouched so the ordinary +native exception exit lets the interpreter unwind them. The patch also stops hot and feedback probes for generator, async, eval and `with` bytecode when the backend requests a snapshot the runtime must refuse; previously every later call, async resume and loop poll re-requested it. From 8c542989744d661f604f8fd180a6fc4cb26b79d6 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:17:42 +0800 Subject: [PATCH 26/54] bench: add typed-convert-traversal diagnostic and P4b status note The arrays-typed scenario buries its two-receiver typed loop under interpreter-only push/construction work. typed-convert-traversal passes a stable Int32Array and Float64Array as arguments to one callee so the multi-source, metadata-only-hoist path is measured directly. Registered in the array traversal diagnostics suite with a hand-checked QuickJS checksum and the Bun checksum-parity list. docs/roadmap/p4b-bce.md records the slice, noisy diagnostics and the remaining work (non-argument receivers, poll countdown register, wrapping add under |0, cross-length range facts). Co-Authored-By: Claude Opus 5.5 (1M context) --- benchmarks/run.rs | 14 ++ benchmarks/scripts/typed-convert-traversal.js | 23 ++++ docs/roadmap/p4b-bce.md | 126 ++++++++++++++++++ 3 files changed, 163 insertions(+) create mode 100644 benchmarks/scripts/typed-convert-traversal.js create mode 100644 docs/roadmap/p4b-bce.md diff --git a/benchmarks/run.rs b/benchmarks/run.rs index 20fe492ac..07b41eb5d 100644 --- a/benchmarks/run.rs +++ b/benchmarks/run.rs @@ -215,6 +215,13 @@ const WORKLOADS: &[Workload] = &[ designated: false, file: "float64array-traversal.js", }, + Workload { + name: "typed-convert-traversal", + suite: "rquickjs-jit array traversal diagnostics", + group: "arrays-typed", + designated: false, + file: "typed-convert-traversal.js", + }, Workload { name: "objects-polymorphic", suite: "rquickjs-jit matrix", @@ -1415,6 +1422,7 @@ mod tests { "packed-array-traversal.js", "int32array-traversal.js", "float64array-traversal.js", + "typed-convert-traversal.js", ] { let path = Path::new(env!("CARGO_MANIFEST_DIR")) .join("scripts") @@ -1499,6 +1507,11 @@ mod tests { "arrays-typed", "float64array-traversal.js", ), + ( + "typed-convert-traversal", + "arrays-typed", + "typed-convert-traversal.js", + ), ] { let workload = WORKLOADS .iter() @@ -1523,6 +1536,7 @@ mod tests { ("packed-array-traversal.js", "number:41803450c0000000"), ("int32array-traversal.js", "number:41803450c0000000"), ("float64array-traversal.js", "number:416034cdc0000000"), + ("typed-convert-traversal.js", "number:416034cdc0000000"), ] { let runtime = Runtime::new().unwrap(); let context = Context::full(&runtime).unwrap(); diff --git a/benchmarks/scripts/typed-convert-traversal.js b/benchmarks/scripts/typed-convert-traversal.js new file mode 100644 index 000000000..14a4cbc57 --- /dev/null +++ b/benchmarks/scripts/typed-convert-traversal.js @@ -0,0 +1,23 @@ +// Construct and populate both stable views before the harness starts timing. +// Unlike float64array-traversal, both typed arrays reach the hot loop as +// arguments of one callee, and only the source's length bounds the loop: the +// destination is a second, simultaneously live typed receiver. +const typedConvertInts = new Int32Array(2000); +const typedConvertFloats = new Float64Array(2000); +for (let i = 0; i < typedConvertInts.length; i++) { + typedConvertInts[i] = (i * 17) & 0xffff; +} +globalThis.workloadArgument = typedConvertInts; + +function convertAndSum(ints, floats) { + let mixed = 0.0; + for (let i = 0; i < ints.length; i++) { + floats[i] = ints[i] * 0.25 + 0.5; + mixed += floats[i]; + } + return mixed; +} + +function workload(iterations, seed, ints) { + return convertAndSum(ints, typedConvertFloats) + seed; +} diff --git a/docs/roadmap/p4b-bce.md b/docs/roadmap/p4b-bce.md new file mode 100644 index 000000000..9bb55ecdb --- /dev/null +++ b/docs/roadmap/p4b-bce.md @@ -0,0 +1,126 @@ +# P4b: array bounds-check elimination and loop metadata (first slice) + +Branch `perf/p4b-bce`, based on `82d3808`. Roadmap item: P4 bounds-check +elimination and length/data hoisting for packed Array, Int32Array and +Float64Array loops ([roadmap](../PERFORMANCE_ROADMAP_V8_BUN.md), section 6). + +## What 82d3808 already had + +`ArrayPlan` + `IntegerRangeAnalysis` already proved `0 <= i < length` for +canonical induction loops whose single `length` read is guarded in the +preheader (packed `logical_length == dense_count`, typed intrinsic-getter +query), deleted the per-access comparison, and revalidated data/count after +every poll. The remaining per-iteration cost in the target loops was not the +bounds compare itself but everything around it. + +## What this slice changes (Tier 2 only, fail-closed) + +1. **Literal-tag folding.** `opt_known_const` folds already-emitted literal + IR (`iconst` through `icmp`/`band`/`bor`/extends). Guards whose condition + folds to true emit no branch; `lt/lte/gt/gte` use an integer compare when + both operand tags are literal Int32 (for example the hoisted exact length); + `if_*` on a literal Bool/Int32 condition branches on the payload, or jumps + directly when the truth itself is literal; `to_propkey` on a literal Int32 + key is the identity. +2. **Static storage kind.** Cached element sources record their kind when it + is statically known; cached loads emit only that kind's load and skip the + redundant receiver-tag check (the source already guarded that frame slot). +3. **Block sealing.** IR blocks whose predecessors are all lowered, guard + diamonds and the amortized-poll join are sealed immediately, and literal + tags are restated after the poll join (the cold path never redefines a + frame variable), so literal definitions stay visible instead of hiding + behind merge parameters. `opt_ir_edge` rejects any branch to an IR block + that is not a declared IR successor, which keeps early sealing sound. +4. **Multiple live sources.** Guarded element metadata is tracked per frame + slot provenance (`ElementSources`). Loads, cached lengths and typed stores + keep the other sources; generic lengths, generic element stores, frame + writes and every plan invalidation clear all of them. +5. **Metadata-only typed hoists.** A typed candidate accessed in a loop with no + `length` read hoists its storage query to the preheader (`length: None`). + Bounds checks stay; polls revalidate data/count exactly like length hoists. + Packed candidates never get such a hoist (their guard would demand + `length == dense_count`, which an element-only loop never needed). +6. **Hoisted sources across in-loop diamonds.** A block inherits a hoisted + source only when every (already lowered) predecessor's exit state holds the + identical source; the preheader dominates every such block. This carries + metadata across the `a[i] = v` nullish-destination diamond. +7. **Numeric constants.** `push_const*` of a numeric constant-pool entry is + lowered only as a literal, so it is non-reentrant for array plans and the + amortized-poll proof, preserves frame identities, and seeds representation + proofs (`proven_numeric_values_with_constants`). This is what admits + `x * 0.25 + 0.5` loops to the array plan at all. +8. **Receiver tag at the header.** A hoisted argument receiver that no loop + node rebinds has its object tag restated at the loop header. + +## Tests + +- `jit/tests/array_bce.rs`: two typed receivers in one loop (exact-length and + metadata-only hoists, aliased views over one buffer, short destinations, + wrong modes, zero-trip loops); a detach at an observable poll with a + metadata-only hoist (the test segfaults if revalidation is removed); + an argument rebound inside a loop that reads through a hoisted alias; + literal folding with Float64 overflow and NaN. +- `array_cache` unit test: element-only typed loops hoist metadata but keep + every bounds check; packed element-only loops do not hoist. +- `automatic_call_heavy_promotes_the_direct_edge_caller` now waits for the + leaf's five clock-paced profitability retries before asserting their exact + count. That race already failed on 82d3808 (3/60 local runs); the faster + native caller loop made it more frequent (11/40) until de-raced. A second, + pre-existing race remains in the same test: the post-warmup + `tier2_entries == 11` assertion occasionally sees the leaf's generic entries + (for example `6008`). With the de-raced test it fails at the same rate on + both revisions (82d3808: 1/60, this branch: 2/60, alternating runs). + +## New diagnostic scenario + +`typed-convert-traversal` (arrays-typed group, diagnostics suite): the +`convertAndSum` kernel with both typed arrays passed as arguments to one +callee, so the float64 destination is a second live receiver with no length +read. It isolates the path that `arrays-typed` covers but buries under its +interpreter-only `push`/construction work. It needs a Bun row in the next +publishable matrix. + +## Noisy diagnostics (not publishable) + +Loaded 24-core host (16 agents), `taskset -c 0-15`, alternating baseline +(`82d3808` `target/release/jit-bench`) and branch binaries, median over runs +of the median of the last 16 warmup batches, µs per 10 calls, automatic mode. +Checksums match the interpreter for every run. Speed = baseline / branch. + +| Workload | runs | 82d3808 | branch | speed | +| --- | ---: | ---: | ---: | ---: | +| int32array-traversal | 7 | 49.77 | 31.00 | 1.61x | +| packed-array-traversal | 7 | 48.15 | 31.09 | 1.55x | +| arrays-typed | 7 | 2973.81 | 1890.80 | 1.57x | +| float64array-traversal | 7 | 1091.95 | 1069.62 | 1.02x | +| typed-convert-traversal (new) | 7 | 1069.40 | 58.83 | 18.2x | +| call-heavy (control) | 3 | 60.24 | 43.87 | 1.37x | +| numeric (control) | 3 | 29.19 | 26.55 | 1.10x | +| scalar-loop (control) | 3 | 29.26 | 26.46 | 1.11x | +| property-heavy (control) | 3 | 98.21 | 100.22 | 0.98x | +| other scalar/call controls | 3 | — | — | 0.99x–1.02x | + +`property-heavy` stays about 2% slower even with every new lowering change +disabled through temporary switches, so it is attributed to host-binary layout +noise, not to generated code; the publishable matrix must confirm it. + +## Remaining work + +- `float64array-traversal` is unchanged: its receivers come from + `buffers.ints`/`buffers.floats` locals, and `ArrayPlan` still admits only + entry-argument identities (roadmap P1 item 6). Once non-argument receivers + are admitted, items 4–6 above apply to it directly. +- `arrays-typed` is now dominated by its interpreter-only orchestration + (`values.push`, `new Int32Array(values)`, `toFixed`, concatenation), not by + the typed traversals. +- The amortized poll countdown lives in a stack slot; keeping it in an SSA + variable measured roughly a further 1.1x on the traversals (int32 loop to + about 27 µs) but conflicts with the structural property-cache tests that + pin the stack-slot form. It belongs with the poll-density item. +- `(sum + a[i]) | 0` still performs a checked add; a wrapping add when the + sum only feeds `| 0` would remove the overflow exit. +- Bounds checks are still per access for loops bounded by anything other than + the receiver's own single `length` read (for example `i < n` or a second + array's length); a range fact relating two lengths needs a guarded + `n <= length` check in the preheader. +- Register pressure spills hoisted `data`/`count` in the packed loop. From a72694cb5eea7ab3b2b69fca7bbbd66ac0201f37 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:22:17 +0800 Subject: [PATCH 27/54] feat(jit): admit common rejected opcodes through an exact GENERIC_OP helper Add patch 0026 (ABI 1.25, runtime API 1.10) with one append-only helper, GENERIC_OP (ID 24, MAP_OUT_TWO_OP), that runs the interpreter's own C routines for push_this, special_object (arguments, this function, new.target, home object, variable object, null prototype), get_var_undef, delete_var, put_var, typeof, typeof_is_undefined, typeof_is_function, to_object, to_propkey2, in, instanceof and delete. Root native entries publish a stack-allocated receiver/new.target/argv context that is saved and restored around the pinned entry; push_this and special_object reject any other frame. Mapped (sloppy) arguments and import.meta stay interpreter-only through an operand-sensitive Tier 1 rejection. Tier 1 lowers the new IrOp::Generic family with the same slot/ownership contract as the other exact slow paths, adds native fast paths for typeof_is_* on non-object primitives and for to_propkey2 with ready keys, routes pow through the existing BINARY_ARITH_SLOW helper, and admits the reachable stack shuffles (dup1, dup2, perm4, swap2, rot3l, rot3r). Unreachable shuffles, set_name*, and the global declaration family stay rejected. Every admitted opcode has an opcode-cases.json entry, and tests/tier1_generic_ops.rs covers exceptions, strict/sloppy receivers, proxies, getters, coercion and stress GC against the interpreter. Co-Authored-By: Claude Opus 5.5 (1M context) --- jit/Cargo.toml | 4 + jit/src/abi.rs | 8 +- jit/src/bytecode/mod.rs | 3 + jit/src/bytecode/policy.rs | 1 + jit/src/bytecode/policy_audit.rs | 40 +- jit/src/bytecode/verify.rs | 34 +- jit/src/compiler/baseline.rs | 267 ++++++++++++- jit/src/ir/baseline.rs | 34 +- jit/src/ir/mod.rs | 2 +- jit/src/ir/types.rs | 67 ++++ jit/tests/abi.rs | 14 +- jit/tests/differential.rs | 6 +- jit/tests/fixtures/opcode-cases.json | 217 +++++++++++ jit/tests/gpui_shell_surface.rs | 9 +- jit/tests/opcodes.rs | 129 +++++-- jit/tests/osr.rs | 10 +- jit/tests/support/mod.rs | 2 + jit/tests/tier1_generic_ops.rs | 298 ++++++++++++++ sys/build.rs | 2 +- sys/build_support/patch.rs | 9 +- sys/patches/0026-tier1-generic-ops.patch | 362 ++++++++++++++++++ sys/patches/README.md | 14 + sys/src/bindings/aarch64-apple-darwin.rs | 31 +- sys/src/bindings/aarch64-pc-windows-msvc.rs | 31 +- sys/src/bindings/aarch64-unknown-linux-gnu.rs | 31 +- .../bindings/aarch64-unknown-linux-musl.rs | 31 +- sys/src/bindings/wasm32-wasip1.rs | 31 +- sys/src/bindings/x86_64-apple-darwin.rs | 31 +- sys/src/bindings/x86_64-pc-windows-gnu.rs | 31 +- sys/src/bindings/x86_64-pc-windows-msvc.rs | 31 +- sys/src/bindings/x86_64-unknown-linux-gnu.rs | 31 +- sys/src/bindings/x86_64-unknown-linux-musl.rs | 31 +- sys/tests/jit_patch.rs | 26 +- sys/tests/typed_array_guard.rs | 2 +- 34 files changed, 1747 insertions(+), 123 deletions(-) create mode 100644 jit/tests/tier1_generic_ops.rs create mode 100644 sys/patches/0026-tier1-generic-ops.patch diff --git a/jit/Cargo.toml b/jit/Cargo.toml index e2b37836d..97291692c 100644 --- a/jit/Cargo.toml +++ b/jit/Cargo.toml @@ -187,6 +187,10 @@ required-features = ["compiler", "test-support"] name = "tier1_regexp" required-features = ["compiler", "test-support"] +[[test]] +name = "tier1_generic_ops" +required-features = ["compiler", "test-support"] + [[test]] name = "test262" required-features = ["compiler", "test-support"] diff --git a/jit/src/abi.rs b/jit/src/abi.rs index b39c65a23..d00fb42d3 100644 --- a/jit/src/abi.rs +++ b/jit/src/abi.rs @@ -5,7 +5,7 @@ use core::{fmt, mem}; use rquickjs_core::qjs; pub const ABI_MAJOR: u16 = 1; -pub const ABI_MINOR: u16 = 24; +pub const ABI_MINOR: u16 = 25; pub const SOURCE_REVISION: u64 = 0xfd0a_0210_b7be_0095; pub const OPCODE_FINGERPRINT: u64 = qjs::QJSJIT_GENERATED_OPCODE_FINGERPRINT; @@ -698,6 +698,10 @@ fn runtime_api_layout_fingerprint() -> u64 { mem::offset_of!(qjs::JSJitRuntimeAPI, unary_arith_slow), mem::size_of::(), ), + ( + mem::offset_of!(qjs::JSJitRuntimeAPI, generic_op), + mem::size_of::(), + ), ] { hash = layout_field(hash, offset, size); } @@ -1073,7 +1077,7 @@ mod inline_api_tests { fn inline_recovery_requires_the_matching_main_abi_minor() { let native = AbiInfo::query_linked().unwrap(); assert_eq!(ABI_MINOR, native.minor()); - assert_eq!(native.minor(), 24); + assert_eq!(native.minor(), 25); assert!(unsafe { qjs::JS_JitGetInlineAPI(0) }.is_null()); } diff --git a/jit/src/bytecode/mod.rs b/jit/src/bytecode/mod.rs index 7cb5a0abd..cd0011744 100644 --- a/jit/src/bytecode/mod.rs +++ b/jit/src/bytecode/mod.rs @@ -20,6 +20,9 @@ pub use policy::{ }; pub use stack::SlotKind; pub use verify::{Resource, VerifiedFunction, VerifyError, VerifyErrorKind, VerifyLimits}; +pub(crate) use verify::{ + SPECIAL_OBJECT_IMPORT_META, SPECIAL_OBJECT_MAPPED_ARGUMENTS, SPECIAL_OBJECT_MAX, +}; /// Numeric opcode constants generated from QuickJS's authoritative macro table. pub mod opcode { diff --git a/jit/src/bytecode/policy.rs b/jit/src/bytecode/policy.rs index 08bd6b6b9..12564ab77 100644 --- a/jit/src/bytecode/policy.rs +++ b/jit/src/bytecode/policy.rs @@ -32,6 +32,7 @@ pub enum HelperId { NewObject, CallConstructor, Regexp, + GenericOp, } #[derive(Clone, Copy, Debug, Eq, PartialEq)] diff --git a/jit/src/bytecode/policy_audit.rs b/jit/src/bytecode/policy_audit.rs index 05612e6af..041aece5f 100644 --- a/jit/src/bytecode/policy_audit.rs +++ b/jit/src/bytecode/policy_audit.rs @@ -9,29 +9,29 @@ pub(super) const AUDITED_POLICIES: [AuditedOpcodePolicy; 252] = [ AuditedOpcodePolicy { id: 5, name: "private_symbol", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 6, name: "undefined", policy: Tier1Policy::Native }, AuditedOpcodePolicy { id: 7, name: "null", policy: Tier1Policy::Native }, - AuditedOpcodePolicy { id: 8, name: "push_this", policy: Tier1Policy::Reject(FallbackReason::ExtendedFrame) }, + AuditedOpcodePolicy { id: 8, name: "push_this", policy: Tier1Policy::Helper(HelperId::GenericOp) }, AuditedOpcodePolicy { id: 9, name: "push_false", policy: Tier1Policy::Native }, AuditedOpcodePolicy { id: 10, name: "push_true", policy: Tier1Policy::Native }, AuditedOpcodePolicy { id: 11, name: "object", policy: Tier1Policy::Helper(HelperId::NewObject) }, - AuditedOpcodePolicy { id: 12, name: "special_object", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 12, name: "special_object", policy: Tier1Policy::Helper(HelperId::GenericOp) }, AuditedOpcodePolicy { id: 13, name: "rest", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 14, name: "drop", policy: Tier1Policy::Helper(HelperId::Free) }, AuditedOpcodePolicy { id: 15, name: "nip", policy: Tier1Policy::Helper(HelperId::Free) }, AuditedOpcodePolicy { id: 16, name: "nip1", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 17, name: "dup", policy: Tier1Policy::Helper(HelperId::Dup) }, - AuditedOpcodePolicy { id: 18, name: "dup1", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, - AuditedOpcodePolicy { id: 19, name: "dup2", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 18, name: "dup1", policy: Tier1Policy::Helper(HelperId::Dup) }, + AuditedOpcodePolicy { id: 19, name: "dup2", policy: Tier1Policy::Helper(HelperId::Dup) }, AuditedOpcodePolicy { id: 20, name: "dup3", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 21, name: "insert2", policy: Tier1Policy::Helper(HelperId::Dup) }, AuditedOpcodePolicy { id: 22, name: "insert3", policy: Tier1Policy::Helper(HelperId::Dup) }, AuditedOpcodePolicy { id: 23, name: "insert4", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 24, name: "perm3", policy: Tier1Policy::Native }, - AuditedOpcodePolicy { id: 25, name: "perm4", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 25, name: "perm4", policy: Tier1Policy::Native }, AuditedOpcodePolicy { id: 26, name: "perm5", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 27, name: "swap", policy: Tier1Policy::Native }, - AuditedOpcodePolicy { id: 28, name: "swap2", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, - AuditedOpcodePolicy { id: 29, name: "rot3l", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, - AuditedOpcodePolicy { id: 30, name: "rot3r", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 28, name: "swap2", policy: Tier1Policy::Native }, + AuditedOpcodePolicy { id: 29, name: "rot3l", policy: Tier1Policy::Native }, + AuditedOpcodePolicy { id: 30, name: "rot3r", policy: Tier1Policy::Native }, AuditedOpcodePolicy { id: 31, name: "rot4l", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 32, name: "rot5l", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 33, name: "call_constructor", policy: Tier1Policy::Helper(HelperId::CallConstructor) }, @@ -56,9 +56,9 @@ pub(super) const AUDITED_POLICIES: [AuditedOpcodePolicy; 252] = [ AuditedOpcodePolicy { id: 52, name: "regexp", policy: Tier1Policy::Helper(HelperId::Regexp) }, AuditedOpcodePolicy { id: 53, name: "get_super", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 54, name: "import", policy: Tier1Policy::Reject(FallbackReason::DynamicImport) }, - AuditedOpcodePolicy { id: 55, name: "get_var_undef", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 55, name: "get_var_undef", policy: Tier1Policy::Helper(HelperId::GenericOp) }, AuditedOpcodePolicy { id: 56, name: "get_var", policy: Tier1Policy::Helper(HelperId::GetGlobal) }, - AuditedOpcodePolicy { id: 57, name: "put_var", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 57, name: "put_var", policy: Tier1Policy::Helper(HelperId::GenericOp) }, AuditedOpcodePolicy { id: 58, name: "put_var_init", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 59, name: "get_ref_value", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, AuditedOpcodePolicy { id: 60, name: "put_ref_value", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, @@ -113,9 +113,9 @@ pub(super) const AUDITED_POLICIES: [AuditedOpcodePolicy; 252] = [ AuditedOpcodePolicy { id: 109, name: "ret", policy: Tier1Policy::Reject(FallbackReason::ExceptionRegion) }, AuditedOpcodePolicy { id: 110, name: "nip_catch", policy: Tier1Policy::Reject(FallbackReason::ExceptionRegion) }, AuditedOpcodePolicy { id: 111, name: "check_object", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, - AuditedOpcodePolicy { id: 112, name: "to_object", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 112, name: "to_object", policy: Tier1Policy::Helper(HelperId::GenericOp) }, AuditedOpcodePolicy { id: 113, name: "to_propkey", policy: Tier1Policy::Helper(HelperId::ToPropertyKey) }, - AuditedOpcodePolicy { id: 114, name: "to_propkey2", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 114, name: "to_propkey2", policy: Tier1Policy::Helper(HelperId::GenericOp) }, AuditedOpcodePolicy { id: 115, name: "with_get_var", policy: Tier1Policy::Reject(FallbackReason::WithScope) }, AuditedOpcodePolicy { id: 116, name: "with_put_var", policy: Tier1Policy::Reject(FallbackReason::WithScope) }, AuditedOpcodePolicy { id: 117, name: "with_delete_var", policy: Tier1Policy::Reject(FallbackReason::WithScope) }, @@ -151,9 +151,9 @@ pub(super) const AUDITED_POLICIES: [AuditedOpcodePolicy; 252] = [ AuditedOpcodePolicy { id: 147, name: "add_loc", policy: Tier1Policy::Native }, AuditedOpcodePolicy { id: 148, name: "not", policy: Tier1Policy::Helper(HelperId::UnaryArithSlow) }, AuditedOpcodePolicy { id: 149, name: "lnot", policy: Tier1Policy::Helper(HelperId::ToBool) }, - AuditedOpcodePolicy { id: 150, name: "typeof", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, - AuditedOpcodePolicy { id: 151, name: "delete", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, - AuditedOpcodePolicy { id: 152, name: "delete_var", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 150, name: "typeof", policy: Tier1Policy::Helper(HelperId::GenericOp) }, + AuditedOpcodePolicy { id: 151, name: "delete", policy: Tier1Policy::Helper(HelperId::GenericOp) }, + AuditedOpcodePolicy { id: 152, name: "delete_var", policy: Tier1Policy::Helper(HelperId::GenericOp) }, AuditedOpcodePolicy { id: 153, name: "mul", policy: Tier1Policy::Native }, AuditedOpcodePolicy { id: 154, name: "div", policy: Tier1Policy::Native }, AuditedOpcodePolicy { id: 155, name: "mod", policy: Tier1Policy::Helper(HelperId::BinaryArithSlow) }, @@ -165,13 +165,13 @@ pub(super) const AUDITED_POLICIES: [AuditedOpcodePolicy; 252] = [ AuditedOpcodePolicy { id: 161, name: "and", policy: Tier1Policy::Native }, AuditedOpcodePolicy { id: 162, name: "xor", policy: Tier1Policy::Native }, AuditedOpcodePolicy { id: 163, name: "or", policy: Tier1Policy::Native }, - AuditedOpcodePolicy { id: 164, name: "pow", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 164, name: "pow", policy: Tier1Policy::Helper(HelperId::BinaryArithSlow) }, AuditedOpcodePolicy { id: 165, name: "lt", policy: Tier1Policy::Helper(HelperId::CompareSlow) }, AuditedOpcodePolicy { id: 166, name: "lte", policy: Tier1Policy::Helper(HelperId::CompareSlow) }, AuditedOpcodePolicy { id: 167, name: "gt", policy: Tier1Policy::Helper(HelperId::CompareSlow) }, AuditedOpcodePolicy { id: 168, name: "gte", policy: Tier1Policy::Helper(HelperId::CompareSlow) }, - AuditedOpcodePolicy { id: 169, name: "instanceof", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, - AuditedOpcodePolicy { id: 170, name: "in", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 169, name: "instanceof", policy: Tier1Policy::Helper(HelperId::GenericOp) }, + AuditedOpcodePolicy { id: 170, name: "in", policy: Tier1Policy::Helper(HelperId::GenericOp) }, AuditedOpcodePolicy { id: 171, name: "eq", policy: Tier1Policy::Helper(HelperId::CompareSlow) }, AuditedOpcodePolicy { id: 172, name: "neq", policy: Tier1Policy::Helper(HelperId::CompareSlow) }, AuditedOpcodePolicy { id: 173, name: "strict_eq", policy: Tier1Policy::Helper(HelperId::CompareSlow) }, @@ -251,6 +251,6 @@ pub(super) const AUDITED_POLICIES: [AuditedOpcodePolicy; 252] = [ AuditedOpcodePolicy { id: 247, name: "call3", policy: Tier1Policy::Helper(HelperId::Call) }, AuditedOpcodePolicy { id: 248, name: "is_undefined", policy: Tier1Policy::Helper(HelperId::Free) }, AuditedOpcodePolicy { id: 249, name: "is_null", policy: Tier1Policy::Helper(HelperId::Free) }, - AuditedOpcodePolicy { id: 250, name: "typeof_is_undefined", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, - AuditedOpcodePolicy { id: 251, name: "typeof_is_function", policy: Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) }, + AuditedOpcodePolicy { id: 250, name: "typeof_is_undefined", policy: Tier1Policy::Helper(HelperId::GenericOp) }, + AuditedOpcodePolicy { id: 251, name: "typeof_is_function", policy: Tier1Policy::Helper(HelperId::GenericOp) }, ]; diff --git a/jit/src/bytecode/verify.rs b/jit/src/bytecode/verify.rs index 24070edf6..e61992d6d 100644 --- a/jit/src/bytecode/verify.rs +++ b/jit/src/bytecode/verify.rs @@ -148,10 +148,13 @@ impl VerifiedFunction { } let mut unsupported = None; for instruction in &self.instructions { - if let super::Tier1Policy::Reject(reason) = - super::tier1_policy(instruction.opcode().id()) - .expect("verified opcode belongs to the generated table") - { + let policy = super::tier1_policy(instruction.opcode().id()) + .expect("verified opcode belongs to the generated table"); + let policy = match special_object_rejection(instruction) { + Some(reason) => super::Tier1Policy::Reject(reason), + None => policy, + }; + if let super::Tier1Policy::Reject(reason) = policy { let rejection = super::Tier1Rejection::new(instruction.pc(), reason); if reason == super::FallbackReason::UnsupportedOpcode { unsupported.get_or_insert(rejection); @@ -164,6 +167,29 @@ impl VerifiedFunction { } } +/// QuickJS `OP_SPECIAL_OBJECT_MAPPED_ARGUMENTS`: sloppy-mode `arguments` +/// aliases argument slots through variable references, which native code +/// keeps in registers, so it remains an extended-frame rejection. +pub(crate) const SPECIAL_OBJECT_MAPPED_ARGUMENTS: u8 = 1; +/// QuickJS `OP_SPECIAL_OBJECT_IMPORT_META`: module state is interpreter-only. +pub(crate) const SPECIAL_OBJECT_IMPORT_META: u8 = 6; +/// QuickJS `OP_SPECIAL_OBJECT_NULL_PROTO`, the largest kind this ABI knows. +pub(crate) const SPECIAL_OBJECT_MAX: u8 = 7; + +/// Operand-sensitive rejections for opcodes whose policy is otherwise a +/// helper: only `special_object` kinds that copy frame state are admitted. +fn special_object_rejection(instruction: &Instruction) -> Option { + if instruction.opcode().name() != "special_object" { + return None; + } + match instruction.operand_u8(1) { + SPECIAL_OBJECT_MAPPED_ARGUMENTS => Some(super::FallbackReason::ExtendedFrame), + SPECIAL_OBJECT_IMPORT_META => Some(super::FallbackReason::UnsupportedOpcode), + kind if kind > SPECIAL_OBJECT_MAX => Some(super::FallbackReason::UnsupportedOpcode), + _ => None, + } +} + fn decode_error_pc(error: &DecodeError) -> u32 { match error { DecodeError::UnknownOpcode { pc, .. } diff --git a/jit/src/compiler/baseline.rs b/jit/src/compiler/baseline.rs index 00e7cfb64..e50a7ff30 100644 --- a/jit/src/compiler/baseline.rs +++ b/jit/src/compiler/baseline.rs @@ -32,8 +32,8 @@ use crate::{ Relocation, RelocationKind, RelocationTarget, StackMap, UnwindKind, UnwindMetadata, }, ir::{ - BaselineIr, BinaryOp, FrameSlot, FrameStateId, FrameStateKind, IrOp, PollKind, StackOp, - TaggedValue, UnaryOp, MAX_HELPER_SCRATCH_SLOTS, + BaselineIr, BinaryOp, FrameSlot, FrameStateId, FrameStateKind, GenericOp, IrOp, PollKind, + StackOp, TaggedValue, UnaryOp, MAX_HELPER_SCRATCH_SLOTS, }, platform::{CodeAllocator, CodeMemoryError, ExecutableCode}, runtime::CompileRequest, @@ -2122,6 +2122,30 @@ fn analyze_entry_domains(ir: &BaselineIr) -> Result { + let (pop, push) = operation.stack_effect(); + let start = frame + .stack + .len() + .checked_sub(pop) + .ok_or(CompileFailure::InvalidArtifact)?; + if *operation == GenericOp::ToPropertyKey2 { + // The object stays in place; only the key changes. + frame.stack.truncate(start + 1); + frame.stack.push(AbstractValue::unknown()); + } else { + frame.stack.truncate(start); + for _ in 0..push { + frame.stack.push(if operation.produces_boolean() { + AbstractValue::known(KnownKind::Boolean) + } else if *operation == GenericOp::ToObject { + AbstractValue::known(KnownKind::Other) + } else { + AbstractValue::unknown() + }); + } + } + } IrOp::GetArgument(index) => frame.stack.push( frame .arguments @@ -2410,6 +2434,7 @@ fn binary_returns_boolean(operation: BinaryOp) -> bool { fn ir_op_produces_boolean(operation: &IrOp) -> bool { match operation { IrOp::Binary(operation) => binary_returns_boolean(*operation), + IrOp::Generic(operation) => operation.produces_boolean(), _ => false, } } @@ -2871,6 +2896,13 @@ fn lower_function( depth = pattern_index + 1; set_visible_stack_depth(builder, frame, stack_base, depth, layout)?; } + IrOp::Generic(operation) => lower_generic( + builder, + &helper_lowering, + &mut helper_states, + &mut depth, + operation, + )?, IrOp::GetArgument(index) => { let state = helper_states .next() @@ -3632,6 +3664,236 @@ fn next_helper_state( states.next().ok_or(CompileFailure::InvalidArtifact) } +/// `GENERIC_OP` packs the QuickJS opcode in bits 0..7 and the bytecode's +/// 8-bit immediate in bits 8..15. +fn generic_operation(name: &str, immediate: u8) -> Result { + Ok(quickjs_opcode_id(name)? | (u32::from(immediate) << 8)) +} + +/// Lowers an opcode whose semantics are executed exactly by `GENERIC_OP` +/// (or `BINARY_ARITH_SLOW` for `pow`). Every operand stays in its frame slot +/// across the helper; the helper clears consumed slots on success and on a +/// language exception, so the exception depth is the pre-instruction depth. +fn lower_generic( + builder: &mut FunctionBuilder<'_>, + helpers: &HelperLowering<'_>, + states: &mut impl Iterator, + depth: &mut usize, + operation: GenericOp, +) -> Result<(), CompileFailure> { + const NONE: u32 = qjs::JS_JIT_SLOT_NONE; + let live = *depth; + let (pop, push) = operation.stack_effect(); + let base = live + .checked_sub(pop) + .ok_or(CompileFailure::InvalidArtifact)?; + let slot = |index: usize| flat_stack_slot(helpers.ir, index); + let state = next_helper_state(states)?; + let generic = qjs::JSJitHelperId_JS_JIT_HELPER_GENERIC_OP; + match operation { + GenericOp::PushThis + | GenericOp::SpecialObject(_) + | GenericOp::GetVarUndef(_) + | GenericOp::DeleteVar(_) => { + let (name, immediate, atom) = match operation { + GenericOp::PushThis => ("push_this", 0, NONE), + GenericOp::SpecialObject(kind) => ("special_object", kind, NONE), + GenericOp::GetVarUndef(atom) => ("get_var_undef", 0, atom), + GenericOp::DeleteVar(atom) => ("delete_var", 0, atom), + _ => unreachable!(), + }; + let output = slot(live)?; + helpers.invoke( + builder, + generic, + state, + live, + live, + &[output, NONE, atom, generic_operation(name, immediate)?], + )?; + reload_pair( + builder, + helpers.stack[live], + helpers.stack_base, + live, + helpers.layout, + ); + } + GenericOp::PutVar(atom) => { + helpers.invoke( + builder, + generic, + state, + live, + live, + &[NONE, slot(base)?, atom, generic_operation("put_var", 0)?], + )?; + clear_pair( + builder, + helpers.stack[base], + helpers.stack_base, + base, + helpers.layout, + )?; + } + GenericOp::ToPropertyKey2 => { + // The interpreter leaves an object-coercible receiver and an + // Int32/string/symbol key untouched; everything else (including + // the TypeError) takes the exact helper. + let object = use_pair(builder, helpers.stack[base]); + let key = use_pair(builder, helpers.stack[base + 1]); + let undefined = tag_is(builder, object.tag, qjs::JS_TAG_UNDEFINED); + let null = tag_is(builder, object.tag, qjs::JS_TAG_NULL); + let not_coercible = builder.ins().bor(undefined, null); + let int_key = tag_is(builder, key.tag, qjs::JS_TAG_INT); + let string_key = tag_is(builder, key.tag, qjs::JS_TAG_STRING); + let symbol_key = tag_is(builder, key.tag, qjs::JS_TAG_SYMBOL); + let key_ready = builder.ins().bor(int_key, string_key); + let key_ready = builder.ins().bor(key_ready, symbol_key); + let slow_needed = builder.ins().bxor_imm(key_ready, 1); + let slow_needed = builder.ins().bor(slow_needed, not_coercible); + let slow = builder.create_block(); + let continuation = builder.create_block(); + builder + .ins() + .brif(slow_needed, slow, &[], continuation, &[]); + builder.seal_block(slow); + builder.switch_to_block(slow); + builder.set_cold_block(slow); + helpers.invoke( + builder, + generic, + state, + live, + live, + &[ + slot(base + 1)?, + slot(base)?, + NONE, + generic_operation("to_propkey2", 0)?, + ], + )?; + reload_pair( + builder, + helpers.stack[base + 1], + helpers.stack_base, + base + 1, + helpers.layout, + ); + builder.ins().jump(continuation, &[]); + builder.seal_block(continuation); + builder.switch_to_block(continuation); + } + GenericOp::TypeOf + | GenericOp::TypeOfIsUndefined + | GenericOp::TypeOfIsFunction + | GenericOp::ToObject => { + let name = match operation { + GenericOp::TypeOf => "typeof", + GenericOp::TypeOfIsUndefined => "typeof_is_undefined", + GenericOp::TypeOfIsFunction => "typeof_is_function", + _ => "to_object", + }; + let input = slot(base)?; + let arguments = [input, input, NONE, generic_operation(name, 0)?]; + if matches!( + operation, + GenericOp::TypeOfIsUndefined | GenericOp::TypeOfIsFunction + ) { + // Non-negative tags are neither objects nor reference counted: + // their `typeof` never names a function and only `undefined` + // names itself, so the answer needs no helper or release. + let value = use_pair(builder, helpers.stack[base]); + let primitive = + builder + .ins() + .icmp_imm(IntCC::SignedGreaterThanOrEqual, value.tag, 0); + let fast = builder.create_block(); + let slow = builder.create_block(); + let continuation = builder.create_block(); + builder.ins().brif(primitive, fast, &[], slow, &[]); + builder.seal_block(fast); + builder.switch_to_block(fast); + let result = if operation == GenericOp::TypeOfIsUndefined { + tag_is(builder, value.tag, qjs::JS_TAG_UNDEFINED) + } else { + builder.ins().iconst(types::I8, 0) + }; + let result = pair_from_bool(builder, result); + define_pair(builder, helpers.stack[base], result); + builder.ins().jump(continuation, &[]); + builder.seal_block(slow); + builder.switch_to_block(slow); + builder.set_cold_block(slow); + helpers.invoke(builder, generic, state, live, live, &arguments)?; + reload_pair( + builder, + helpers.stack[base], + helpers.stack_base, + base, + helpers.layout, + ); + builder.ins().jump(continuation, &[]); + builder.seal_block(continuation); + builder.switch_to_block(continuation); + } else { + helpers.invoke(builder, generic, state, live, live, &arguments)?; + reload_pair( + builder, + helpers.stack[base], + helpers.stack_base, + base, + helpers.layout, + ); + } + } + GenericOp::In | GenericOp::InstanceOf | GenericOp::Delete | GenericOp::Pow => { + let left = slot(base)?; + let right = slot(base + 1)?; + if operation == GenericOp::Pow { + helpers.invoke( + builder, + qjs::JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW, + state, + live, + live, + &[left, left, right, quickjs_opcode_id("pow")?], + )?; + } else { + let name = match operation { + GenericOp::In => "in", + GenericOp::InstanceOf => "instanceof", + _ => "delete", + }; + helpers.invoke( + builder, + generic, + state, + live, + live, + &[left, left, right, generic_operation(name, 0)?], + )?; + } + reload_pair( + builder, + helpers.stack[base], + helpers.stack_base, + base, + helpers.layout, + ); + clear_pair( + builder, + helpers.stack[base + 1], + helpers.stack_base, + base + 1, + helpers.layout, + )?; + } + } + *depth = base + push; + helpers.set_depth(builder, *depth) +} + #[allow(clippy::too_many_arguments)] fn lower_dup_if_refcounted( builder: &mut FunctionBuilder<'_>, @@ -6496,6 +6758,7 @@ mod tests { IrOp::Call { .. } => "call", IrOp::CallConstructor(_) => "call_constructor", IrOp::Regexp => "regexp", + IrOp::Generic(_) => "generic", IrOp::GetArgument(_) => "get_argument", IrOp::GetLocal(_) => "get_local", IrOp::GetLocalChecked(_) => "get_local_checked", diff --git a/jit/src/ir/baseline.rs b/jit/src/ir/baseline.rs index 68e7aa362..943ef79e8 100644 --- a/jit/src/ir/baseline.rs +++ b/jit/src/ir/baseline.rs @@ -10,8 +10,8 @@ use crate::{ }; use super::{ - BinaryOp, FrameSlot, FrameState, FrameStateId, FrameStateKind, FrameStateTable, IrOp, PollKind, - StackOp, TaggedValue, UnaryOp, + BinaryOp, FrameSlot, FrameState, FrameStateId, FrameStateKind, FrameStateTable, GenericOp, + IrOp, PollKind, StackOp, TaggedValue, UnaryOp, }; const POLL_INTERVAL: usize = 1_024; @@ -453,6 +453,7 @@ fn operation_helper_call_count(operation: &IrOp) -> usize { IrOp::Call { argc, has_this } => 1 + usize::from(*argc) + 1 + usize::from(*has_this), IrOp::CallConstructor(argc) => 1 + usize::from(*argc) + 2, IrOp::Regexp => 1, + IrOp::Generic(_) => 1, IrOp::GetArgument(_) | IrOp::GetLocal(_) | IrOp::GetLocalChecked(_) => 1, IrOp::GetLocalPair => 2, IrOp::PutArgument { keep, .. } | IrOp::PutLocal { keep, .. } => 1 + usize::from(*keep), @@ -698,6 +699,35 @@ fn translate_instruction(instruction: &Instruction) -> Result IrOp::CallConstructor(instruction.operand_u16(1)), "regexp" => IrOp::Regexp, + "push_this" => IrOp::Generic(GenericOp::PushThis), + "special_object" => match instruction.operand_u8(1) { + crate::bytecode::SPECIAL_OBJECT_MAPPED_ARGUMENTS + | crate::bytecode::SPECIAL_OBJECT_IMPORT_META => { + return Err(CompileFailure::UnsupportedOpcode) + } + kind if kind > crate::bytecode::SPECIAL_OBJECT_MAX => { + return Err(CompileFailure::UnsupportedOpcode) + } + kind => IrOp::Generic(GenericOp::SpecialObject(kind)), + }, + "get_var_undef" => IrOp::Generic(GenericOp::GetVarUndef( + atom_operand(instruction).ok_or(CompileFailure::InvalidArtifact)?, + )), + "delete_var" => IrOp::Generic(GenericOp::DeleteVar( + atom_operand(instruction).ok_or(CompileFailure::InvalidArtifact)?, + )), + "put_var" => IrOp::Generic(GenericOp::PutVar( + atom_operand(instruction).ok_or(CompileFailure::InvalidArtifact)?, + )), + "typeof" => IrOp::Generic(GenericOp::TypeOf), + "typeof_is_undefined" => IrOp::Generic(GenericOp::TypeOfIsUndefined), + "typeof_is_function" => IrOp::Generic(GenericOp::TypeOfIsFunction), + "to_object" => IrOp::Generic(GenericOp::ToObject), + "to_propkey2" => IrOp::Generic(GenericOp::ToPropertyKey2), + "in" => IrOp::Generic(GenericOp::In), + "instanceof" => IrOp::Generic(GenericOp::InstanceOf), + "delete" => IrOp::Generic(GenericOp::Delete), + "pow" => IrOp::Generic(GenericOp::Pow), "get_arg" | "get_arg0" | "get_arg1" | "get_arg2" | "get_arg3" => { IrOp::GetArgument(indexed_operand(instruction).ok_or(CompileFailure::InvalidArtifact)?) } diff --git a/jit/src/ir/mod.rs b/jit/src/ir/mod.rs index 9be6ab40b..b01b762e1 100644 --- a/jit/src/ir/mod.rs +++ b/jit/src/ir/mod.rs @@ -42,6 +42,6 @@ pub use scalar::{ ScalarGraph, ScalarInlineRegion, ScalarInlineStep, ScalarNumericMode, ScalarPhiInput, ScalarValue, ScalarValueId, }; -pub use types::{BinaryOp, IrOp, PollKind, StackOp, TaggedValue, UnaryOp}; +pub use types::{BinaryOp, GenericOp, IrOp, PollKind, StackOp, TaggedValue, UnaryOp}; pub(crate) use scalar::stack_permutation; diff --git a/jit/src/ir/types.rs b/jit/src/ir/types.rs index 1ab34caf1..ea69c0bd7 100644 --- a/jit/src/ir/types.rs +++ b/jit/src/ir/types.rs @@ -73,6 +73,72 @@ pub enum BinaryOp { StrictNotEqual, } +/// Opcodes executed with exact interpreter semantics by the `GENERIC_OP` +/// helper (or `BINARY_ARITH_SLOW` for `pow`). Atoms and 8-bit immediates are +/// the verified bytecode operands. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum GenericOp { + /// `push_this`: pushes the (possibly boxed) receiver. + PushThis, + /// `special_object`: pushes the frame object selected by the immediate. + SpecialObject(u8), + /// `get_var_undef`: pushes a global or `undefined` when it is missing. + GetVarUndef(u32), + /// `delete_var`: pushes the boolean result of deleting a global binding. + DeleteVar(u32), + /// `put_var`: pops a value into a global binding. + PutVar(u32), + /// `typeof`: replaces the top value with its type string. + TypeOf, + /// `typeof_is_undefined`: replaces the top value with a boolean. + TypeOfIsUndefined, + /// `typeof_is_function`: replaces the top value with a boolean. + TypeOfIsFunction, + /// `to_object`: converts the top value to an object in place. + ToObject, + /// `to_propkey2`: checks that `sp[-2]` is object-coercible and converts + /// the key at `sp[-1]` to a property key in place. + ToPropertyKey2, + /// `in`: `key in object`, producing a boolean. + In, + /// `instanceof`: `value instanceof constructor`, producing a boolean. + InstanceOf, + /// `delete`: `delete object[key]`, producing a boolean. + Delete, + /// `pow`: exponentiation through the exact arithmetic slow path. + Pow, +} + +impl GenericOp { + /// Values consumed from the operand stack and values pushed back. + pub const fn stack_effect(self) -> (usize, usize) { + match self { + Self::PushThis | Self::SpecialObject(_) | Self::GetVarUndef(_) | Self::DeleteVar(_) => { + (0, 1) + } + Self::PutVar(_) => (1, 0), + Self::TypeOf | Self::TypeOfIsUndefined | Self::TypeOfIsFunction | Self::ToObject => { + (1, 1) + } + Self::ToPropertyKey2 => (2, 2), + Self::In | Self::InstanceOf | Self::Delete | Self::Pow => (2, 1), + } + } + + /// Whether the single result is always a JS boolean. + pub const fn produces_boolean(self) -> bool { + matches!( + self, + Self::TypeOfIsUndefined + | Self::TypeOfIsFunction + | Self::In + | Self::InstanceOf + | Self::Delete + | Self::DeleteVar(_) + ) + } +} + #[derive(Clone, Debug, Eq, PartialEq)] pub enum IrOp { Poll { state: FrameStateId, kind: PollKind }, @@ -95,6 +161,7 @@ pub enum IrOp { Call { argc: u16, has_this: bool }, CallConstructor(u16), Regexp, + Generic(GenericOp), GetArgument(u16), GetLocal(u16), GetLocalChecked(u16), diff --git a/jit/tests/abi.rs b/jit/tests/abi.rs index bb3580e33..c9bc84f40 100644 --- a/jit/tests/abi.rs +++ b/jit/tests/abi.rs @@ -386,7 +386,7 @@ fn interrupt_runtime_api_is_a_versioned_exec_frame_tail_extension() { use rquickjs_core::qjs; assert_eq!(qjs::QJSJIT_RUNTIME_API_MAJOR, 1); - assert_eq!(qjs::QJSJIT_RUNTIME_API_MINOR, 9); + assert_eq!(qjs::QJSJIT_RUNTIME_API_MINOR, 10); assert_eq!( std::mem::offset_of!(qjs::JSJitExecFrame, runtime_api), std::mem::offset_of!(qjs::JSJitExecFrame, entry) @@ -419,7 +419,8 @@ fn interrupt_runtime_api_is_a_versioned_exec_frame_tail_extension() { std::mem::offset_of!(qjs::JSJitRuntimeAPI, unary_arith_slow), 192 ); - assert_eq!(std::mem::size_of::(), 200); + assert_eq!(std::mem::offset_of!(qjs::JSJitRuntimeAPI, generic_op), 200); + assert_eq!(std::mem::size_of::(), 208); assert_eq!(qjs::JS_JIT_HELPER_GUARD_MISS, 1); } @@ -530,6 +531,11 @@ fn helper_abi_is_one_canonical_versioned_table_in_c_bindgen_and_rust() { 2, qjs::JSJitHelperOwnership_JS_JIT_HELPER_OWNED as u8, ), + ( + "GENERIC_OP", + 3, + qjs::JSJitHelperOwnership_JS_JIT_HELPER_OWNED as u8, + ), ]; let mut count = 0_u32; @@ -574,9 +580,9 @@ fn helper_abi_is_one_canonical_versioned_table_in_c_bindgen_and_rust() { fn helper_abi_fields_are_append_only_tails() { use rquickjs_core::qjs; - assert_eq!(qjs::QJSJIT_ABI_MINOR, 24); + assert_eq!(qjs::QJSJIT_ABI_MINOR, 25); assert_eq!(qjs::QJSJIT_RUNTIME_API_MAJOR, 1); - assert_eq!(qjs::QJSJIT_RUNTIME_API_MINOR, 9); + assert_eq!(qjs::QJSJIT_RUNTIME_API_MINOR, 10); assert_eq!(qjs::QJSJIT_HELPER_ABI_VERSION, 1); assert_eq!( std::mem::offset_of!(qjs::JSJitEntryHandle, stack_map_count), diff --git a/jit/tests/differential.rs b/jit/tests/differential.rs index 07166009d..55cc13d9d 100644 --- a/jit/tests/differential.rs +++ b/jit/tests/differential.rs @@ -64,6 +64,7 @@ enum ManifestHelper { Regexp, NewArray, NewObject, + GenericOp, } impl ManifestHelper { @@ -90,6 +91,7 @@ impl ManifestHelper { Self::Regexp => HelperId::Regexp, Self::NewArray => HelperId::NewArray, Self::NewObject => HelperId::NewObject, + Self::GenericOp => HelperId::GenericOp, } } } @@ -250,8 +252,8 @@ fn manifest_dimension_schema_is_closed_and_required_dimensions_are_mechanical() #[test] fn rejected_programs_have_exact_fallback_and_interpreter_semantics() { assert_tier1_rejected( - "function f(a){ return typeof a }", - "f(86)", + "function f(a){ return a ? 1n : 2n }", + "String(f(86))", FallbackReason::UnsupportedOpcode, ); } diff --git a/jit/tests/fixtures/opcode-cases.json b/jit/tests/fixtures/opcode-cases.json index 3a6818cfd..a449bcd3d 100644 --- a/jit/tests/fixtures/opcode-cases.json +++ b/jit/tests/fixtures/opcode-cases.json @@ -1431,6 +1431,223 @@ "ownership-gc", "coercion-reentrancy" ] + }, + { + "opcode": "push_this", + "definition": "function f(o){ return this.x + o }", + "expression": "[f.call({x:40},2), f.call(7,1), f.call(undefined,1), f.call('s',1), (()=>{try{return f.call({get x(){throw new RangeError('x')}},1)}catch(e){return e.name}})()]", + "helper": "GenericOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "special_object", + "definition": "function f(){ 'use strict'; return arguments.length + ':' + arguments[0] }", + "expression": "[f(40,2,3), f(), f('a')]", + "helper": "GenericOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "get_var_undef", + "definition": "function f(){ return typeof missingGlobal + ':' + typeof Math }", + "expression": "f()", + "helper": "GenericOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "put_var", + "definition": "globalThis.counterGlobal = 0; function f(v){ counterGlobal = v; return counterGlobal }", + "expression": "[f(1), f('x'), f({a:1}).a, globalThis.counterGlobal.a]", + "helper": "GenericOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "delete_var", + "definition": "globalThis.deletableGlobal = 1; function f(){ return delete deletableGlobal }", + "expression": "[f(), typeof deletableGlobal, f()]", + "helper": "GenericOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "typeof", + "definition": "function f(x){ return typeof x }", + "expression": "[f(1), f(1.5), f('s'), f(null), f(undefined), f({}), f(f), f(1n), f(Symbol()), f(true), f(class {})]", + "helper": "GenericOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "typeof_is_undefined", + "definition": "function f(x){ return typeof x === 'undefined' }", + "expression": "[f(), f(undefined), f(1), f(null), f({}), f(''), f('x'.repeat(3))]", + "helper": "GenericOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "typeof_is_function", + "definition": "function f(x){ return typeof x === 'function' }", + "expression": "[f(f), f(1), f({}), f(class {}), f(new Proxy(function(){}, {})), f(new Proxy({}, {})), f('s')]", + "helper": "GenericOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "in", + "definition": "function f(k,o){ return k in o }", + "expression": "[f('a',{a:1}), f('b',{a:1}), f(0,[1]), f(Symbol.iterator,[]), (()=>{try{return f('a',1)}catch(e){return e.name}})(), (()=>{try{return f({toString(){throw new RangeError('k')}},{})}catch(e){return e.name}})()]", + "helper": "GenericOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "instanceof", + "definition": "function f(a,b){ return a instanceof b }", + "expression": "[f([],Array), f({},Array), f(1,Number), (()=>{try{return f(1,1)}catch(e){return e.name}})(), (()=>{try{return f({}, {[Symbol.hasInstance](){throw new RangeError('h')}})}catch(e){return e.name}})()]", + "helper": "GenericOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "delete", + "definition": "function f(o,k){ return delete o[k] }", + "expression": "[f({a:1},'a'), f(Object.freeze({a:1}),'a'), f([1,2],0), f({}, 'missing'), (()=>{try{return f(null,'a')}catch(e){return e.name}})()]", + "helper": "GenericOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "to_object", + "definition": "function f(o){ const {a, b} = o; return a + ':' + b }", + "expression": "[f({a:1,b:2}), f('xy'), f(5), (()=>{try{return f(null)}catch(e){return e.name}})()]", + "helper": "GenericOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "to_propkey2", + "definition": "function f(o,k){ o[k] += 1; return o[k] }", + "expression": "[f({a:1},'a'), f([5],0), f({}, {toString(){return 'z'}}), (()=>{try{return f(null,'a')}catch(e){return e.name}})(), (()=>{try{return f({}, {toString(){throw new RangeError('k')}})}catch(e){return e.name}})()]", + "helper": "GenericOp", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "pow", + "definition": "function f(a,b){ return a ** b }", + "expression": "[f(2,10), f(2,0.5), f(-8,1/3), String(f(2n,3n)), f('3',2), f({valueOf(){return 3}},2), (()=>{try{return f(2n,1)}catch(e){return e.name}})(), (()=>{try{return f({valueOf(){throw new RangeError('v')}},2)}catch(e){return e.name}})()]", + "helper": "BinaryArithSlow", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc", + "coercion-reentrancy" + ] + }, + { + "opcode": "dup1", + "definition": "function f(s,k){ let x; ({[k]: x} = s); return x }", + "expression": "[f({a:1},'a'), f({},'b'), f([7,8],1)]", + "helper": "Dup", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "dup2", + "definition": "function f(o,k){ o[k] *= 2; return o[k] }", + "expression": "[f({a:21},'a'), f([5],0), f({m:1.5}, 'm')]", + "helper": "Dup", + "dimensions": [ + "normal", + "exception-or-nonthrow", + "ownership-gc" + ] + }, + { + "opcode": "perm4", + "definition": "function f(a,i){ let v = a[i]++; return v + ':' + a[i] }", + "expression": "[f([1,2],0), f({x:5},'x'), f([0,0,0,-1], 3)]", + "helper": null, + "dimensions": [ + "normal", + "exception-or-nonthrow" + ] + }, + { + "opcode": "rot3r", + "definition": "function f(s,k,o){ ({[k]: o.p} = s); return o.p }", + "expression": "[f({a:1},'a',{}), f({},'b',{p:3})]", + "helper": null, + "dimensions": [ + "normal", + "exception-or-nonthrow" + ] + }, + { + "opcode": "swap2", + "definition": "function f(s,k,a,i){ ({[k]: a[i]} = s); return a[i] }", + "expression": "[f({a:1},'a',[],0), f({b:2},'b',{},'x')]", + "helper": null, + "dimensions": [ + "normal", + "exception-or-nonthrow" + ] + }, + { + "opcode": "rot3l", + "definition": "function f(s,a,i){ ({p: a[i]} = s); return a[i] }", + "expression": "[f({p:4},[],1), f({},{q:1},'q')]", + "helper": null, + "dimensions": [ + "normal", + "exception-or-nonthrow" + ] } ] } diff --git a/jit/tests/gpui_shell_surface.rs b/jit/tests/gpui_shell_surface.rs index 623b5a881..80a7f411a 100644 --- a/jit/tests/gpui_shell_surface.rs +++ b/jit/tests/gpui_shell_surface.rs @@ -196,9 +196,10 @@ fn deterministic_tier1_rejection_stops_after_one_snapshot_without_queueing() { context .with(|ctx| { ctx.eval::<(), _>( - // `typeof` has no Tier 1 lowering, so this continuation is a - // deterministic policy reject. - "globalThis.__terminalHost = value => JSON.stringify({ value, kind: typeof value });", + // BigInt literals (`push_bigint_i32`) have no Tier 1 + // lowering, so this continuation is a deterministic policy + // reject. + "globalThis.__terminalHost = value => JSON.stringify({ value, kind: String(1n) });", ) }) .unwrap(); @@ -208,7 +209,7 @@ fn deterministic_tier1_rejection_stops_after_one_snapshot_without_queueing() { let function: Function = ctx.globals().get("__terminalHost").unwrap(); assert_eq!( function.call::<_, String>((42,)).unwrap(), - r#"{"value":42,"kind":"number"}"# + r#"{"value":42,"kind":"1"}"# ); }); runtime.jit().poll(); diff --git a/jit/tests/opcodes.rs b/jit/tests/opcodes.rs index 77d43fadd..90fe614da 100644 --- a/jit/tests/opcodes.rs +++ b/jit/tests/opcodes.rs @@ -7,13 +7,13 @@ use std::collections::BTreeSet; #[test] fn synthetic_function_identity_cannot_bypass_the_closed_policy() { - // `typeof` has no Tier 1 lowering and stays a policy reject; the synthetic - // ID-zero function must not bypass that row. + // `get_super` has no Tier 1 lowering and stays a policy reject; the + // synthetic ID-zero function must not bypass that row. let mut bytecode = vec![ rquickjs_jit::bytecode::opcode::PUSH_UNDEFINED, linked_opcode_table() - .find(|opcode| opcode.name() == "typeof") - .expect("linked typeof opcode") + .find(|opcode| opcode.name() == "get_super") + .expect("linked get_super opcode") .id(), ]; bytecode.push(rquickjs_jit::bytecode::opcode::RETURN); @@ -141,20 +141,115 @@ fn unsupported_frame_and_exception_families_are_stable_rejects() { by_name("get_var_ref"), Tier1Policy::Reject(FallbackReason::ClosureFrame) ); - assert_eq!( - by_name("push_this"), - Tier1Policy::Reject(FallbackReason::ExtendedFrame) - ); assert_eq!( by_name("catch"), Tier1Policy::Reject(FallbackReason::ExceptionRegion) ); assert_eq!( - by_name("typeof"), + by_name("get_super"), Tier1Policy::Reject(FallbackReason::UnsupportedOpcode) ); } +#[test] +fn generic_opcode_family_is_advertised_through_the_exact_helper() { + let by_name = |name| { + let opcode = linked_opcode_table() + .find(|opcode| opcode.name() == name) + .unwrap(); + tier1_policy(opcode.id()).unwrap() + }; + + for generic in [ + "push_this", + "special_object", + "get_var_undef", + "put_var", + "typeof", + "typeof_is_undefined", + "typeof_is_function", + "in", + "instanceof", + "delete", + "delete_var", + "to_object", + "to_propkey2", + ] { + assert_eq!( + by_name(generic), + Tier1Policy::Helper(HelperId::GenericOp), + "{generic}" + ); + } + assert_eq!( + by_name("pow"), + Tier1Policy::Helper(HelperId::BinaryArithSlow) + ); + for dup in ["dup1", "dup2"] { + assert_eq!(by_name(dup), Tier1Policy::Helper(HelperId::Dup), "{dup}"); + } + for native in ["perm4", "swap2", "rot3l", "rot3r"] { + assert_eq!(by_name(native), Tier1Policy::Native, "{native}"); + } + + // These remain closed until their producers can reach Tier 1: `nop` and + // `nip1` are never emitted after label resolution, `dup3`/`insert4`/ + // `perm5`/`rot4l`/`rot5l` only follow `super` references or async + // iteration, `set_name`/`set_name_computed` only name fresh closures or + // classes, and the global declaration family only occurs in script/eval + // code, which is never a Tier 1 function. + for rejected in [ + "nop", + "nip1", + "dup3", + "insert4", + "perm5", + "rot4l", + "rot5l", + "set_name_computed", + "put_var_init", + "define_var", + "check_define_var", + "define_func", + ] { + assert_eq!( + by_name(rejected), + Tier1Policy::Reject(FallbackReason::UnsupportedOpcode), + "{rejected}" + ); + } + assert_eq!( + by_name("set_name"), + Tier1Policy::Reject(FallbackReason::ExtendedFrame) + ); +} + +#[test] +fn aliasing_and_module_special_objects_remain_operand_rejections() { + let special = linked_opcode_table() + .find(|opcode| opcode.name() == "special_object") + .unwrap() + .id(); + // QuickJS OP_SPECIAL_OBJECT_{ARGUMENTS, MAPPED_ARGUMENTS, IMPORT_META}. + for (kind, expected) in [ + (0_u8, None), + (1, Some(FallbackReason::ExtendedFrame)), + (6, Some(FallbackReason::UnsupportedOpcode)), + (8, Some(FallbackReason::UnsupportedOpcode)), + ] { + let bytecode = vec![special, kind, rquickjs_jit::bytecode::opcode::RETURN]; + let verified = rquickjs_jit::test_support::verified_bytecode(bytecode, 0, 0); + assert_eq!( + verified + .tier1_eligibility() + .err() + .map(|rejection| rejection.reason()), + expected, + "special_object kind {kind}" + ); + } +} + #[test] fn m2_core_opcodes_are_advertised_with_their_lowering_family() { let by_name = |name| { @@ -239,20 +334,4 @@ fn m2_core_opcodes_are_advertised_with_their_lowering_family() { by_name("tail_call_method"), Tier1Policy::Helper(HelperId::Call) ); - - // Stack shuffles that ordinary Tier 1 eligible source cannot reach stay - // rejected: `nop`/`nip1` are never emitted by the QuickJS compiler, - // `dup1`/`swap2`/`rot3r`/`rot3l` need destructuring or for-in lvalues - // (`to_object`, `for_in_start`), `dup2`/`perm4` need `to_propkey2`, and - // `dup3`/`insert4`/`perm5`/`rot4l`/`rot5l` need `super` lvalues. - for rejected in [ - "nop", "nip1", "dup1", "dup2", "dup3", "insert4", "perm4", "perm5", "swap2", "rot3l", - "rot3r", "rot4l", "rot5l", - ] { - assert_eq!( - by_name(rejected), - Tier1Policy::Reject(FallbackReason::UnsupportedOpcode), - "{rejected}" - ); - } } diff --git a/jit/tests/osr.rs b/jit/tests/osr.rs index b7e77f35a..76f4c0a7a 100644 --- a/jit/tests/osr.rs +++ b/jit/tests/osr.rs @@ -516,7 +516,7 @@ fn first_invocation_osr_executes_helper_with_side_effect_gc_and_reentry() { #[test] fn production_ineligible_loop_is_rejected_once_before_queueing() { let failure_fixture = - SnapshotFixture::compile("(function f(n,z){let i=z;while(i("function f(n,z){let i=z;while(i( + "function f(n,z){let i=z;while(i rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_GET_GLOBAL, + HelperId::GenericOp => rquickjs_core::qjs::JSJitHelperId_JS_JIT_HELPER_GENERIC_OP, }; let expected_opcode = self .expected_opcode @@ -1328,6 +1329,7 @@ static SYNTHETIC_RUNTIME_API: rquickjs_core::qjs::JSJitRuntimeAPI = regexp: Some(synthetic_map_out_two_unavailable), binary_arith_slow: Some(synthetic_map_out_two_op_unavailable), unary_arith_slow: Some(synthetic_map_out_in_op_unavailable), + generic_op: Some(synthetic_map_out_two_op_unavailable), }; /// Result observed after invoking a generated aggregate-return entry point. diff --git a/jit/tests/tier1_generic_ops.rs b/jit/tests/tier1_generic_ops.rs new file mode 100644 index 000000000..13364888f --- /dev/null +++ b/jit/tests/tier1_generic_ops.rs @@ -0,0 +1,298 @@ +#![cfg(all( + target_endian = "little", + any(target_arch = "x86_64", target_arch = "aarch64"), + any(target_os = "linux", target_os = "macos", target_os = "windows"), + not(all(target_os = "windows", target_arch = "aarch64")) +))] + +//! Exact-semantics coverage for the `GENERIC_OP` helper family and the stack +//! shuffles admitted with it. Every run compares a forced Tier 1 execution +//! against the interpreter, including language exceptions, and executes the +//! helper under stress GC. + +use rquickjs_jit::bytecode::{FallbackReason, HelperId}; +use rquickjs_jit::test_support::{assert_tier1_rejected, differential}; + +const TRY: &str = "(()=>{try{return ("; +const CATCH: &str = ")}catch(e){return e.name+':'+e.message}})()"; + +fn guarded(expression: &str) -> String { + format!("{TRY}{expression}{CATCH}") +} + +fn generic(definition: &str, expression: &str, opcode: &str) { + differential(definition, expression) + .force_baseline() + .stress_gc() + .expect_executed_opcode(opcode) + .expect_helper(HelperId::GenericOp) + .assert_same(); +} + +#[test] +fn push_this_boxes_sloppy_primitives_and_keeps_strict_receivers() { + generic( + "function f(o){ return typeof this + ':' + this.x + ':' + o }", + "[f.call({x:40},2), f.call(7,1), f.call(undefined,1), f.call(null,1), f.call('s',1), f.call(true,1)]", + "push_this", + ); + generic( + "function f(o){ 'use strict'; return typeof this + ':' + o }", + "[f.call({x:40},2), f.call(7,1), f.call(undefined,1), f.call(null,1), f.call('s',1)]", + "push_this", + ); + generic( + "function f(o){ return this.x + o }", + &format!( + "[{}, {}]", + guarded("f.call({get x(){throw new RangeError('getter')}},1)"), + guarded("f.call({x:1},2)") + ), + "push_this", + ); +} + +#[test] +fn push_this_observes_method_constructor_and_recursive_receivers() { + generic( + "function f(n){ if(n>0) return this.tag + f.call({tag:n}, n-1); return this.tag }", + "f.call({tag:'root'}, 3)", + "push_this", + ); + generic( + "function f(v){ this.v = v; return this }", + "[new f(4).v, new f('x').v, f.call({}, 5).v]", + "push_this", + ); +} + +#[test] +fn unmapped_arguments_new_target_this_function_and_null_proto_objects() { + generic( + "function f(){ 'use strict'; let a = arguments; a[0] = 9; return a.length + ':' + a[0] + ':' + Array.isArray(a) }", + "[f(), f(1), f(1,2,3), f(...Array(20).fill(0))]", + "special_object", + ); + generic( + "function f(a, b){ 'use strict'; arguments[0] = 'changed'; return a + ':' + arguments[0] + ':' + arguments.length }", + "[f(1), f(1,2), f()]", + "special_object", + ); + generic( + "function f(){ return new.target === undefined ? 'call' : new.target.name }", + "[f(), new f() instanceof f, Reflect.construct(f, [], Array) instanceof Array]", + "special_object", + ); + generic( + "var f = function g(n){ return n > 0 ? g(n - 1) + 1 : typeof g }", + "[f(0), f(3)]", + "special_object", + ); + generic( + "function f(v){ return {__proto__: null, v} }", + "[Object.getPrototypeOf(f(1)), f(2).v, 'toString' in f(3)]", + "special_object", + ); +} + +#[test] +fn sloppy_mapped_arguments_stay_in_the_interpreter() { + assert_tier1_rejected( + "function f(a){ arguments[0] = 2; return a }", + "f(1)", + FallbackReason::ExtendedFrame, + ); +} + +#[test] +fn global_reads_writes_and_deletes_match_the_interpreter() { + generic( + "function f(){ return typeof missingGlobal + ':' + typeof Math + ':' + typeof f }", + "f()", + "get_var_undef", + ); + generic( + "globalThis.writableGlobal = 0; function f(v){ writableGlobal = v; return writableGlobal }", + "[f(1), f('x'), f({a:1}).a, writableGlobal.a]", + "put_var", + ); + generic( + "Object.defineProperty(globalThis, 'frozenGlobal', {value: 1, writable: false}); \ + function f(v){ 'use strict'; frozenGlobal = v; return frozenGlobal }", + &guarded("f(2)"), + "put_var", + ); + generic( + "function f(v){ 'use strict'; undeclaredStrictGlobal = v; return v }", + &guarded("f(2)"), + "put_var", + ); + generic( + "globalThis.deletable = 1; Object.defineProperty(globalThis, 'pinned', {value: 1}); \ + function f(){ return [delete deletable, delete pinned, delete neverDefined] }", + "[f(), typeof deletable, typeof pinned]", + "delete_var", + ); +} + +#[test] +fn typeof_family_matches_every_tag() { + let values = "[1, 1.5, -0, NaN, 's', '', null, undefined, {}, [], f, class {}, 1n, 2n**80n, Symbol(), true, new Proxy(function(){}, {}), new Proxy({}, {})]"; + generic( + "function f(x){ return typeof x }", + &format!("{values}.map(f)"), + "typeof", + ); + generic( + "function f(x){ return typeof x === 'undefined' }", + &format!("{values}.map(f)"), + "typeof_is_undefined", + ); + generic( + "function f(x){ return typeof x === 'function' }", + &format!("{values}.map(f)"), + "typeof_is_function", + ); + generic( + "function f(x){ let n = 0; for (let i = 0; i < 64; i++) { if (typeof x === 'undefined') n++; else n += 2 } return n }", + "[f(), f(1), f('s'), f({})]", + "typeof_is_undefined", + ); +} + +#[test] +fn in_instanceof_and_delete_preserve_exceptions() { + generic( + "function f(k,o){ return k in o }", + &format!( + "[f('a',{{a:1}}), f('b',{{a:1}}), f(0,[1]), f(1,[1]), f(Symbol.iterator,[]), f('x', new Proxy({{}}, {{has(){{return true}}}})), {}, {}, {}]", + guarded("f('a',1)"), + guarded("f({toString(){throw new RangeError('key')}},{})"), + guarded("f('a', new Proxy({}, {has(){throw new RangeError('trap')}}))"), + ), + "in", + ); + generic( + "function f(a,b){ return a instanceof b }", + &format!( + "[f([],Array), f({{}},Array), f(1,Number), f(Object.create(null),Object), f(new (class A {{}}), Object), {}, {}, {}]", + guarded("f(1,1)"), + guarded("f({}, {[Symbol.hasInstance](){throw new RangeError('has')}})"), + guarded("f({}, () => 1)"), + ), + "instanceof", + ); + generic( + "function f(o,k){ return delete o[k] }", + &format!( + "[f({{a:1}},'a'), f(Object.freeze({{a:1}}),'a'), f([1,2],0), f({{}}, 'missing'), f('str', 0), {}]", + guarded("f(null,'a')"), + ), + "delete", + ); + generic( + "function f(o,k){ 'use strict'; return delete o[k] }", + &format!( + "[f({{a:1}},'a'), {}, {}]", + guarded("f(Object.freeze({a:1}),'a')"), + guarded("f(new Proxy({}, {deleteProperty(){throw new RangeError('trap')}}),'a')"), + ), + "delete", + ); +} + +#[test] +fn object_destructuring_and_compound_element_updates() { + generic( + "function f(o){ const {a, b} = o; return a + ':' + b }", + &format!( + "[f({{a:1,b:2}}), f('xy'), f(5), f(Object.create({{a:'proto'}})), {}, {}]", + guarded("f(null)"), + guarded("f(undefined)"), + ), + "to_object", + ); + generic( + "function f(o,k){ o[k] += 1; return o[k] }", + &format!( + "[f({{a:1}},'a'), f([5],0), f({{}}, {{toString(){{return 'z'}}}}), f({{}}, Symbol.iterator), f([1], 0.5), {}, {}, {}]", + guarded("f(null,'a')"), + guarded("f(undefined,{toString(){throw new RangeError('never')}})"), + guarded("f({}, {toString(){throw new RangeError('key')}})"), + ), + "to_propkey2", + ); +} + +#[test] +fn exponent_uses_the_exact_arithmetic_slow_path() { + differential( + "function f(a,b){ return a ** b }", + &format!( + "[f(2,10), f(2,0.5), f(-8,1/3), f(2,-1074), String(f(2n,3n)), f('3',2), f({{valueOf(){{return 3}}}},2), {}, {}]", + guarded("f(2n,1)"), + guarded("f({valueOf(){throw new RangeError('v')}},2)"), + ), + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("pow") + .expect_helper(HelperId::BinaryArithSlow) + .assert_same(); +} + +#[test] +fn destructuring_stack_shuffles_run_natively() { + for (definition, expression, opcode) in [ + ( + "function f(s,k){ let x; ({[k]: x} = s); return x }", + "[f({a:1},'a'), f({},'b'), f([7,8],1)]", + "dup1", + ), + ( + "function f(o,k){ o[k] *= 2; return o[k] }", + "[f({a:21},'a'), f([5],0), f({m:1.5}, 'm')]", + "dup2", + ), + ( + "function f(a,i){ let v = a[i]++; return v + ':' + a[i] }", + "[f([1,2],0), f({x:5},'x'), f([0,0,0,-1], 3)]", + "perm4", + ), + ( + "function f(s,k,o){ ({[k]: o.p} = s); return o.p }", + "[f({a:1},'a',{}), f({},'b',{p:3})]", + "rot3r", + ), + ( + "function f(s,k,a,i){ ({[k]: a[i]} = s); return a[i] }", + "[f({a:1},'a',[],0), f({b:2},'b',{},'x')]", + "swap2", + ), + ( + "function f(s,a,i){ ({p: a[i]} = s); return a[i] }", + "[f({p:4},[],1), f({},{q:1},'q')]", + "rot3l", + ), + ] { + differential(definition, expression) + .force_baseline() + .stress_gc() + .expect_executed_opcode(opcode) + .assert_same(); + } +} + +#[test] +fn method_heavy_loop_with_this_typeof_and_in_stays_native() { + generic( + "function f(items){ let total = 0; for (let i = 0; i < items.length; i++) { \ + const item = items[i]; \ + if (typeof item === 'undefined') continue; \ + if ('w' in item && item instanceof Object) total += item.w ** 2; \ + else if (typeof item.v === 'number') total += item.v; \ + } if (this === undefined) return -total; return total }", + "[f.call({}, [{w:2},{v:3},undefined,{v:'x'},{w:1}]), f([{w:3}])]", + "typeof_is_undefined", + ); +} diff --git a/sys/build.rs b/sys/build.rs index 38375afaa..0f1e4b21f 100644 --- a/sys/build.rs +++ b/sys/build.rs @@ -228,7 +228,7 @@ QJSJIT_HELPER_LIST(QJSJIT_EMIT) flags, }); } - assert_eq!(helpers.len(), 24, "canonical append-only helper count"); + assert_eq!(helpers.len(), 25, "canonical append-only helper count"); let u32_args = |signature: &str| match signature { "FRAME" => 0, diff --git a/sys/build_support/patch.rs b/sys/build_support/patch.rs index 41317e2f2..79a5169d3 100644 --- a/sys/build_support/patch.rs +++ b/sys/build_support/patch.rs @@ -1,7 +1,7 @@ use std::{fs, io, path::Path}; const QUICKJS_BASELINE_FNV64: u64 = 0x3302_116a_b0fc_269c; -const EXPECTED_PATCHES: [(&str, u64); 19] = [ +const EXPECTED_PATCHES: [(&str, u64); 20] = [ ("0001-rquickjs-jit.patch", 0x18d7_40cc_6943_30cf), ("0002-runtime-feedback.patch", 0xe4c1_ed52_ee5d_d469), ("0003-element-layout.patch", 0xa12c_ccd2_7420_e880), @@ -21,6 +21,7 @@ const EXPECTED_PATCHES: [(&str, u64); 19] = [ ("0018-inline-frame-recovery.patch", 0xb971d61edccde229), ("0019-array-feedback.patch", 0x11c3cb13a8297dd4), ("0020-typed-array-guard.patch", 0x5d10_38a3_8bc9_9821), + ("0026-tier1-generic-ops.patch", 0x1933_9cb3_c0c7_5b48), ]; pub(crate) const BASELINE_FILES: [&str; 19] = [ "api-test.c", @@ -60,9 +61,9 @@ const PATCHED_FILE_FINGERPRINTS: [(&str, u64); 21] = [ ("list.h", 0xb337_70f7_b76d_a3d8), ("quickjs-atom.h", 0x30b4_9116_b6a2_aa99), ("quickjs-c-atomics.h", 0x490b_0f29_f631_3fc0), - ("quickjs.c", 0xc535_b5f1_48e7_9b27), - ("quickjs-jit.h", 0x0188_04ad_21d2_95a7), - ("quickjs-jit-helpers.h", 0xa469_e80f_be47_404c), + ("quickjs.c", 0xc5d3_8fc4_ebba_a0b4), + ("quickjs-jit.h", 0xfafa_4511_21a6_ffb4), + ("quickjs-jit-helpers.h", 0x6963_5583_a4a4_700f), ("quickjs-opcode.h", 0x3d05_cfdf_5cf7_2930), ("quickjs.h", 0x4831_2cde_9c2f_a5ee), ]; diff --git a/sys/patches/0026-tier1-generic-ops.patch b/sys/patches/0026-tier1-generic-ops.patch new file mode 100644 index 000000000..73506c53f --- /dev/null +++ b/sys/patches/0026-tier1-generic-ops.patch @@ -0,0 +1,362 @@ +diff --git a/quickjs-jit-helpers.h b/quickjs-jit-helpers.h +--- a/quickjs-jit-helpers.h ++++ b/quickjs-jit-helpers.h +@@ -129,6 +129,9 @@ + THROWING | ALLOCATING | REENTRANT | FINALIZING) \ + X(UNARY_ARITH_SLOW, unary_arith_slow, JS_JitHelperUnaryArithSlow, \ + MAP_OUT_IN_OP, 2, OWNED, CONSUMED, NONE, NONE, OWNED, \ ++ THROWING | ALLOCATING | REENTRANT | FINALIZING) \ ++ X(GENERIC_OP, generic_op, JS_JitHelperGenericOp, MAP_OUT_TWO_OP, 3, \ ++ OWNED, CONSUMED, CONSUMED, NONE, OWNED, \ + THROWING | ALLOCATING | REENTRANT | FINALIZING) + + typedef enum JSJitHelperId { +diff --git a/quickjs-jit.h b/quickjs-jit.h +--- a/quickjs-jit.h ++++ b/quickjs-jit.h +@@ -17,7 +17,7 @@ + #endif + + #define QJSJIT_ABI_MAJOR 1u +-#define QJSJIT_ABI_MINOR 24u ++#define QJSJIT_ABI_MINOR 25u + + /* Frame flags are append-only. Stress collection is a test/debug contract: + * every helper performs an allocating cycle-GC point before and after its +@@ -273,7 +273,7 @@ + * prefix and helper ID zero; every later helper is an append-only v1.1 tail. + */ + #define QJSJIT_RUNTIME_API_MAJOR 1u +-#define QJSJIT_RUNTIME_API_MINOR 9u ++#define QJSJIT_RUNTIME_API_MINOR 10u + + #define QJSJIT_RUNTIME_FIELD_FRAME(field) \ + JSJitHelperStatus (*field)(JSJitExecFrame *frame) +diff --git a/quickjs.c b/quickjs.c +--- a/quickjs.c ++++ b/quickjs.c +@@ -376,6 +376,8 @@ + uint64_t jit_next_frame_cookie; + uint64_t jit_active_frame_cookie; + JSJitExecFrame *jit_active_frame; ++ /* Receiver/argument context of the innermost root native entry. */ ++ struct QJSJitRootCall *jit_active_root; + struct JSJitInlineState *jit_inline_top; + size_t jit_inline_bytes; + uint32_t jit_inline_depth; +@@ -20098,6 +20100,285 @@ + return qjsjit_finish_output(frame, out); + } + ++/* Private call context of one root native entry (never an inline view). ++ * JS_CallInternal publishes it for exactly one pinned entry call so that ++ * GENERIC_OP observes the interpreter's receiver, new.target and caller ++ * arguments. Nothing here is retained after the entry returns. */ ++typedef struct QJSJitRootCall { ++ struct QJSJitRootCall *previous; ++ const JSJitExecFrame *frame; ++ JSValueConst this_obj; ++ JSValueConst new_target; ++ JSValueConst *argv; ++ int argc; ++} QJSJitRootCall; ++ ++static void qjsjit_generic_release(JSContext *ctx, JSValue *out, ++ JSValue *lhs, JSValue *rhs) ++{ ++ if (out) ++ qjsjit_consume_slot(ctx, out); ++ if (lhs && lhs != out) ++ qjsjit_consume_slot(ctx, lhs); ++ if (rhs && rhs != out && rhs != lhs) ++ qjsjit_consume_slot(ctx, rhs); ++} ++ ++/* Exact interpreter semantics for opcodes without a dedicated native ++ * lowering. operation packs the QuickJS opcode in bits 0..7 and its 8-bit ++ * immediate (the special_object kind) in bits 8..15; bits 16..31 are zero. ++ * Slots per opcode, JS_JIT_SLOT_NONE otherwise: ++ * push_this, special_object output (unoccupied) ++ * get_var_undef, delete_var output (unoccupied), right = atom ++ * put_var left (consumed), right = atom ++ * typeof, typeof_is_undefined, ++ * typeof_is_function, to_object output == left (replaced in place) ++ * to_propkey2 output = key (replaced in place), ++ * left = output - 1 = object (kept) ++ * in, instanceof, delete output == left, right == left + 1 ++ * push_this and special_object read the root entry's call context and are ++ * rejected in inline views. MAPPED_ARGUMENTS aliases frame slots and ++ * IMPORT_META needs module state; both remain interpreter-only. On a ++ * language exception every named slot is released and cleared, so callers ++ * may keep them inside the exception depth. */ ++JSJitHelperStatus JS_JitHelperGenericOp(JSJitExecFrame *frame, ++ uint32_t stack_map_id, ++ uint32_t output, uint32_t left, ++ uint32_t right, uint32_t operation) ++{ ++ JSFunctionBytecode *b; ++ JSStackFrame *sf; ++ JSContext *ctx; ++ QJSJitRootCall *root = NULL; ++ JSValue *out = NULL, *lhs = NULL, *rhs = NULL; ++ JSValue result = JS_UNDEFINED; ++ JSAtom atom = JS_ATOM_NULL; ++ uint32_t opcode = operation & 0xffu, imm = (operation >> 8) & 0xffu; ++ bool uses_out = false, uses_left = false, uses_right = false; ++ bool uses_atom = false, fresh_out = false; ++ uint32_t tag; ++ int ret = 0; ++ ++ if (qjsjit_validate_helper_frame(frame, true, stack_map_id, &b, &sf) < 0) ++ return JS_JIT_HELPER_EXCEPTION; ++ qjsjit_count_helper(frame, JS_JIT_HELPER_GENERIC_OP); ++ if ((operation >> 16) != 0 || (opcode != OP_special_object && imm != 0)) ++ return qjsjit_reject_helper(frame, "invalid JIT generic operation"); ++ switch (opcode) { ++ case OP_push_this: ++ case OP_special_object: ++ uses_out = fresh_out = true; ++ break; ++ case OP_get_var_undef: ++ case OP_delete_var: ++ uses_out = fresh_out = uses_atom = true; ++ break; ++ case OP_put_var: ++ uses_left = uses_atom = true; ++ break; ++ case OP_typeof: ++ case OP_typeof_is_undefined: ++ case OP_typeof_is_function: ++ case OP_to_object: ++ case OP_to_propkey2: ++ uses_out = uses_left = true; ++ break; ++ case OP_in: ++ case OP_instanceof: ++ case OP_delete: ++ uses_out = uses_left = uses_right = true; ++ break; ++ default: ++ return qjsjit_reject_helper(frame, "invalid JIT generic opcode"); ++ } ++ if ((uses_out ? !qjsjit_resolve_slot(frame, b, output, &out) ++ : output != JS_JIT_SLOT_NONE) || ++ (uses_left ? !qjsjit_resolve_slot(frame, b, left, &lhs) ++ : left != JS_JIT_SLOT_NONE)) ++ return qjsjit_reject_helper(frame, "invalid JIT generic slots"); ++ if (uses_atom) { ++ atom = (JSAtom)right; ++ if (!qjsjit_valid_atom(frame->rt, atom)) ++ return qjsjit_reject_helper(frame, "invalid JIT generic atom"); ++ } else if (uses_right ? !qjsjit_resolve_slot(frame, b, right, &rhs) ++ : right != JS_JIT_SLOT_NONE) { ++ return qjsjit_reject_helper(frame, "invalid JIT generic slots"); ++ } ++ if (fresh_out && !JS_IsUndefined(*out)) ++ return qjsjit_reject_helper(frame, "occupied JIT generic output"); ++ switch (opcode) { ++ case OP_typeof: ++ case OP_typeof_is_undefined: ++ case OP_typeof_is_function: ++ case OP_to_object: ++ if (lhs != out) ++ return qjsjit_reject_helper(frame, "invalid JIT generic slots"); ++ break; ++ case OP_to_propkey2: ++ if (lhs + 1 != out) ++ return qjsjit_reject_helper(frame, "invalid JIT generic slots"); ++ break; ++ case OP_in: ++ case OP_instanceof: ++ case OP_delete: ++ if (lhs != out || rhs != lhs + 1) ++ return qjsjit_reject_helper(frame, "invalid JIT generic slots"); ++ break; ++ case OP_push_this: ++ case OP_special_object: ++ root = frame->rt->jit_active_root; ++ if (!root || root->frame != frame) ++ return qjsjit_reject_helper(frame, ++ "JIT generic operation outside a root frame"); ++ if (opcode == OP_special_object && ++ imm != OP_SPECIAL_OBJECT_ARGUMENTS && ++ imm != OP_SPECIAL_OBJECT_THIS_FUNC && ++ imm != OP_SPECIAL_OBJECT_NEW_TARGET && ++ imm != OP_SPECIAL_OBJECT_HOME_OBJECT && ++ imm != OP_SPECIAL_OBJECT_VAR_OBJECT && ++ imm != OP_SPECIAL_OBJECT_NULL_PROTO) ++ return qjsjit_reject_helper(frame, "invalid JIT special object"); ++ break; ++ default: ++ break; ++ } ++ ctx = frame->ctx; ++ if (qjsjit_stress_gc(frame) < 0) { ++ qjsjit_generic_release(ctx, fresh_out ? NULL : out, lhs, rhs); ++ return JS_JIT_HELPER_EXCEPTION; ++ } ++ ++ switch (opcode) { ++ case OP_push_this: ++ /* The interpreter's normal_this path for strict code and objects. */ ++ tag = JS_VALUE_GET_TAG(root->this_obj); ++ if (b->is_strict_mode || tag == JS_TAG_OBJECT) ++ result = js_dup(root->this_obj); ++ else if (tag == JS_TAG_NULL || tag == JS_TAG_UNDEFINED) ++ result = js_dup(ctx->global_obj); ++ else ++ result = JS_ToObject(ctx, root->this_obj); ++ ret = JS_IsException(result) ? -1 : 0; ++ break; ++ case OP_special_object: ++ switch (imm) { ++ case OP_SPECIAL_OBJECT_ARGUMENTS: ++ result = js_build_arguments(ctx, root->argc, root->argv); ++ break; ++ case OP_SPECIAL_OBJECT_THIS_FUNC: ++ result = js_dup(sf->cur_func); ++ break; ++ case OP_SPECIAL_OBJECT_NEW_TARGET: ++ result = js_dup(root->new_target); ++ break; ++ case OP_SPECIAL_OBJECT_HOME_OBJECT: ++ { ++ JSObject *home = ++ JS_VALUE_GET_OBJ(sf->cur_func)->u.func.home_object; ++ result = home ? js_dup(JS_MKPTR(JS_TAG_OBJECT, home)) ++ : JS_UNDEFINED; ++ } ++ break; ++ case OP_SPECIAL_OBJECT_VAR_OBJECT: ++ result = JS_NewObjectProto(ctx, JS_NULL); ++ break; ++ default: ++ result = JS_NewObjectProtoClass(ctx, JS_NULL, JS_CLASS_OBJECT); ++ break; ++ } ++ ret = JS_IsException(result) ? -1 : 0; ++ break; ++ case OP_get_var_undef: ++ result = JS_GetGlobalVar(ctx, atom, false); ++ ret = JS_IsException(result) ? -1 : 0; ++ break; ++ case OP_delete_var: ++ ret = JS_DeleteGlobalVar(ctx, atom); ++ if (ret >= 0) { ++ result = js_bool(ret); ++ ret = 0; ++ } ++ break; ++ case OP_put_var: ++ /* JS_SetGlobalVar consumes the value on every path. */ ++ ret = JS_SetGlobalVar(ctx, atom, *lhs, 0); ++ *lhs = JS_UNDEFINED; ++ break; ++ case OP_typeof: ++ atom = js_operator_typeof(ctx, *lhs); ++ qjsjit_consume_slot(ctx, lhs); ++ result = JS_AtomToString(ctx, atom); ++ ret = JS_IsException(result) ? -1 : 0; ++ break; ++ case OP_typeof_is_undefined: ++ case OP_typeof_is_function: ++ atom = js_operator_typeof(ctx, *lhs); ++ qjsjit_consume_slot(ctx, lhs); ++ result = js_bool(atom == (opcode == OP_typeof_is_undefined ? ++ JS_ATOM_undefined : JS_ATOM_function)); ++ break; ++ case OP_to_object: ++ if (JS_VALUE_GET_TAG(*lhs) == JS_TAG_OBJECT) ++ break; ++ result = JS_ToObject(ctx, *lhs); ++ if (JS_IsException(result)) { ++ ret = -1; ++ break; ++ } ++ JS_FreeValue(ctx, *lhs); ++ *lhs = result; ++ break; ++ case OP_to_propkey2: ++ /* The object-coercible check precedes key conversion. */ ++ if (JS_IsUndefined(*lhs) || JS_IsNull(*lhs)) { ++ JS_ThrowTypeError(ctx, "value has no property"); ++ ret = -1; ++ break; ++ } ++ switch (JS_VALUE_GET_TAG(*out)) { ++ case JS_TAG_INT: ++ case JS_TAG_STRING: ++ case JS_TAG_SYMBOL: ++ break; ++ default: ++ result = JS_ToPropertyKey(ctx, *out); ++ if (JS_IsException(result)) { ++ ret = -1; ++ break; ++ } ++ JS_FreeValue(ctx, *out); ++ *out = result; ++ break; ++ } ++ break; ++ case OP_in: ++ ret = js_operator_in(ctx, rhs + 1); ++ break; ++ case OP_instanceof: ++ ret = js_operator_instanceof(ctx, rhs + 1); ++ break; ++ default: ++ ret = js_operator_delete(ctx, rhs + 1); ++ break; ++ } ++ if (ret < 0) { ++ qjsjit_generic_release(ctx, fresh_out ? NULL : out, lhs, rhs); ++ qjsjit_stress_gc_after_exception(frame); ++ return JS_JIT_HELPER_EXCEPTION; ++ } ++ if (rhs) { ++ /* The operator consumed both operands and stored its boolean in the ++ * left slot; the right slot only holds a stale copy. */ ++ *rhs = JS_UNDEFINED; ++ } ++ if (fresh_out || opcode == OP_typeof || opcode == OP_typeof_is_undefined || ++ opcode == OP_typeof_is_function) ++ *out = result; ++ if (out) ++ return qjsjit_finish_output(frame, out); ++ return qjsjit_stress_gc(frame); ++} ++ + #endif + + #ifdef CONFIG_JIT_ABI +@@ -20746,6 +21027,7 @@ + rt->jit_backend_active_depth--; + } else { + JSJitExecFrame frame; ++ QJSJitRootCall root_call; + JSJitExit jit_exit; + const uint8_t *resume_pc; + JSValue *resume_sp; +@@ -20795,6 +21077,13 @@ + rt->jit_active_frame_cookie = frame.frame_cookie; + rt->jit_active_stack_map_count = entry.stack_map_count; + rt->jit_active_helper_abi_version = entry.helper_abi_version; ++ root_call.previous = rt->jit_active_root; ++ root_call.frame = &frame; ++ root_call.this_obj = this_obj; ++ root_call.new_target = new_target; ++ root_call.argv = argv; ++ root_call.argc = argc; ++ rt->jit_active_root = &root_call; + if (jit_vtable->native_enter) + jit_vtable->native_enter(jit_opaque, entry_function_id, + entry_generation, entry_pc_offset); +@@ -20815,6 +21104,7 @@ + rt->jit_active_frame_cookie = previous_frame_cookie; + rt->jit_active_stack_map_count = previous_stack_map_count; + rt->jit_active_helper_abi_version = previous_helper_abi_version; ++ rt->jit_active_root = root_call.previous; + if (!rt->jit_entry_cache_suppressed && + entry_pc_offset == 0 && entry_epoch != 0 && + b->jit_function_id == entry_function_id && diff --git a/sys/patches/README.md b/sys/patches/README.md index 931cf9bd3..2d784d892 100644 --- a/sys/patches/README.md +++ b/sys/patches/README.md @@ -60,6 +60,20 @@ data pointer before returning `{ data, count, mode }`. The table advertises zero effects and the receiver is passed by pointer so generated calls do not depend on platform-specific aggregate argument classification. +`0026-tier1-generic-ops.patch` adds ABI 1.25 / runtime API 1.10 helper +`GENERIC_OP` (append-only helper ID 24, `MAP_OUT_TWO_OP`). It executes +`push_this`, `special_object` (arguments, this function, `new.target`, home +object, variable object and null-prototype objects), `get_var_undef`, +`delete_var`, `put_var`, the `typeof` family, `to_object`, `to_propkey2`, +`in`, `instanceof` and `delete` with the interpreter's own C routines. The +operation word packs the opcode and its 8-bit immediate; atoms travel in the +right operand and are validated. Each root native entry publishes a private +stack-allocated call context (receiver, `new.target`, caller `argc`/`argv`) that +is saved and restored around the pinned entry call; `push_this` and +`special_object` reject any frame that is not that root (for example an inline +view). Mapped (sloppy) `arguments` and `import.meta` stay interpreter-only. +On a language exception every named slot is released and cleared. + The build accepts only the patch names and byte digests listed in `build_support/patch.rs`, then verifies the complete patched source manifest. Keeping the baseline and patch separate makes diff --git a/sys/src/bindings/aarch64-apple-darwin.rs b/sys/src/bindings/aarch64-apple-darwin.rs index a64c09ee5..c2f27c3d3 100644 --- a/sys/src/bindings/aarch64-apple-darwin.rs +++ b/sys/src/bindings/aarch64-apple-darwin.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2816,7 +2816,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GENERIC_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3059,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperGenericOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3283,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub generic_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3352,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::generic_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, generic_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/aarch64-pc-windows-msvc.rs b/sys/src/bindings/aarch64-pc-windows-msvc.rs index a64c09ee5..c2f27c3d3 100644 --- a/sys/src/bindings/aarch64-pc-windows-msvc.rs +++ b/sys/src/bindings/aarch64-pc-windows-msvc.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2816,7 +2816,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GENERIC_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3059,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperGenericOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3283,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub generic_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3352,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::generic_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, generic_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/aarch64-unknown-linux-gnu.rs b/sys/src/bindings/aarch64-unknown-linux-gnu.rs index a64c09ee5..c2f27c3d3 100644 --- a/sys/src/bindings/aarch64-unknown-linux-gnu.rs +++ b/sys/src/bindings/aarch64-unknown-linux-gnu.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2816,7 +2816,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GENERIC_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3059,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperGenericOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3283,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub generic_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3352,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::generic_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, generic_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/aarch64-unknown-linux-musl.rs b/sys/src/bindings/aarch64-unknown-linux-musl.rs index a64c09ee5..c2f27c3d3 100644 --- a/sys/src/bindings/aarch64-unknown-linux-musl.rs +++ b/sys/src/bindings/aarch64-unknown-linux-musl.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2816,7 +2816,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GENERIC_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3059,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperGenericOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3283,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub generic_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3352,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::generic_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, generic_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/wasm32-wasip1.rs b/sys/src/bindings/wasm32-wasip1.rs index a8167404c..dfe851f41 100644 --- a/sys/src/bindings/wasm32-wasip1.rs +++ b/sys/src/bindings/wasm32-wasip1.rs @@ -87,7 +87,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -114,7 +114,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2785,7 +2785,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GENERIC_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3027,6 +3028,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperGenericOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3241,10 +3252,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub generic_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 104usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 108usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 4usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3300,6 +3321,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 96usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 100usize]; + ["Offset of field: JSJitRuntimeAPI::generic_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, generic_op) - 104usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/x86_64-apple-darwin.rs b/sys/src/bindings/x86_64-apple-darwin.rs index a64c09ee5..c2f27c3d3 100644 --- a/sys/src/bindings/x86_64-apple-darwin.rs +++ b/sys/src/bindings/x86_64-apple-darwin.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2816,7 +2816,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GENERIC_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3059,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperGenericOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3283,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub generic_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3352,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::generic_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, generic_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/x86_64-pc-windows-gnu.rs b/sys/src/bindings/x86_64-pc-windows-gnu.rs index a64c09ee5..c2f27c3d3 100644 --- a/sys/src/bindings/x86_64-pc-windows-gnu.rs +++ b/sys/src/bindings/x86_64-pc-windows-gnu.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2816,7 +2816,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GENERIC_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3059,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperGenericOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3283,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub generic_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3352,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::generic_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, generic_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/x86_64-pc-windows-msvc.rs b/sys/src/bindings/x86_64-pc-windows-msvc.rs index a64c09ee5..c2f27c3d3 100644 --- a/sys/src/bindings/x86_64-pc-windows-msvc.rs +++ b/sys/src/bindings/x86_64-pc-windows-msvc.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2816,7 +2816,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GENERIC_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3059,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperGenericOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3283,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub generic_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3352,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::generic_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, generic_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/x86_64-unknown-linux-gnu.rs b/sys/src/bindings/x86_64-unknown-linux-gnu.rs index a64c09ee5..c2f27c3d3 100644 --- a/sys/src/bindings/x86_64-unknown-linux-gnu.rs +++ b/sys/src/bindings/x86_64-unknown-linux-gnu.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2816,7 +2816,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GENERIC_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3059,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperGenericOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3283,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub generic_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3352,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::generic_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, generic_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/src/bindings/x86_64-unknown-linux-musl.rs b/sys/src/bindings/x86_64-unknown-linux-musl.rs index a64c09ee5..c2f27c3d3 100644 --- a/sys/src/bindings/x86_64-unknown-linux-musl.rs +++ b/sys/src/bindings/x86_64-unknown-linux-musl.rs @@ -86,7 +86,7 @@ pub const JS_DEF_ALIAS: u32 = 9; pub const JS_DEF_PROP_SYMBOL: u32 = 10; pub const JS_DEF_PROP_BOOL: u32 = 11; pub const QJSJIT_ABI_MAJOR: u32 = 1; -pub const QJSJIT_ABI_MINOR: u32 = 24; +pub const QJSJIT_ABI_MINOR: u32 = 25; pub const JS_JIT_FUNCTION_STRICT: u32 = 1; pub const JS_JIT_FRAME_STRESS_GC: u32 = 2; pub const JS_JIT_FRAME_SIDE_PATH_HIT: u32 = 4; @@ -113,7 +113,7 @@ pub const QJSJIT_HELPER_ABI_VERSION: u32 = 1; pub const QJSJIT_HELPER_MAX_ABI_TYPES: u32 = 8; pub const QJSJIT_HELPER_MAX_VALUES: u32 = 4; pub const QJSJIT_RUNTIME_API_MAJOR: u32 = 1; -pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9; +pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10; pub const QJSJIT_INLINE_RECOVERY_VERSION: u32 = 1; pub const JS_JIT_INLINE_CALL: u32 = 0; pub const JS_JIT_INLINE_CALL_METHOD: u32 = 1; @@ -2816,7 +2816,8 @@ pub const JSJitHelperId_JS_JIT_HELPER_REGEXP: JSJitHelperId = 20; pub const JSJitHelperId_JS_JIT_HELPER_ATOM_VALUE: JSJitHelperId = 21; pub const JSJitHelperId_JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22; pub const JSJitHelperId_JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23; -pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_GENERIC_OP: JSJitHelperId = 24; +pub const JSJitHelperId_JS_JIT_HELPER_COUNT: JSJitHelperId = 25; pub type JSJitHelperId = ::core::ffi::c_uint; #[repr(C)] #[derive(Debug, Copy, Clone)] @@ -3058,6 +3059,16 @@ unsafe extern "C" { opcode: u32, ) -> JSJitHelperStatus; } +unsafe extern "C" { + pub fn JS_JitHelperGenericOp( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus; +} unsafe extern "C" { pub fn JS_JitGetHelperTable(count: *mut u32, fingerprint: *mut u64) -> *const JSJitHelperInfo; } @@ -3272,10 +3283,20 @@ pub struct JSJitRuntimeAPI { opcode: u32, ) -> JSJitHelperStatus, >, + pub generic_op: ::core::option::Option< + unsafe extern "C" fn( + frame: *mut JSJitExecFrame, + stack_map_id: u32, + output: u32, + left: u32, + right: u32, + operation: u32, + ) -> JSJitHelperStatus, + >, } #[allow(clippy::unnecessary_operation, clippy::identity_op)] const _: () = { - ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 200usize]; + ["Size of JSJitRuntimeAPI"][::core::mem::size_of::() - 208usize]; ["Alignment of JSJitRuntimeAPI"][::core::mem::align_of::() - 8usize]; ["Offset of field: JSJitRuntimeAPI::struct_size"] [::core::mem::offset_of!(JSJitRuntimeAPI, struct_size) - 0usize]; @@ -3331,6 +3352,8 @@ const _: () = { [::core::mem::offset_of!(JSJitRuntimeAPI, binary_arith_slow) - 184usize]; ["Offset of field: JSJitRuntimeAPI::unary_arith_slow"] [::core::mem::offset_of!(JSJitRuntimeAPI, unary_arith_slow) - 192usize]; + ["Offset of field: JSJitRuntimeAPI::generic_op"] + [::core::mem::offset_of!(JSJitRuntimeAPI, generic_op) - 200usize]; }; #[repr(C)] #[derive(Debug, Copy, Clone)] diff --git a/sys/tests/jit_patch.rs b/sys/tests/jit_patch.rs index f36861117..0963791a1 100644 --- a/sys/tests/jit_patch.rs +++ b/sys/tests/jit_patch.rs @@ -56,6 +56,7 @@ fn copy_patches(destination: &std::path::Path) { "0018-inline-frame-recovery.patch", "0019-array-feedback.patch", "0020-typed-array-guard.patch", + "0026-tier1-generic-ops.patch", ] { fs::copy(source.join(patch), destination.join(patch)).unwrap(); } @@ -114,7 +115,7 @@ fn pinned_public_quickjs_baseline_applies_cleanly_without_git() { let helper_header = fs::read_to_string(destination.join("quickjs-jit-helpers.h")).unwrap(); assert!(quickjs.contains("JS_GetJitRuntimeId")); assert!(quickjs.contains("JS_JIT_FRAME_SIDE_PATH_HIT")); - assert!(jit_header.contains("#define QJSJIT_ABI_MINOR 24u")); + assert!(jit_header.contains("#define QJSJIT_ABI_MINOR 25u")); assert!(jit_header.contains("JSJitPropertyLayout property_layout;")); assert!(quickjs.contains("uint64_t jit_shape_generation;")); assert!(!quickjs.contains("QJSJIT_SHAPE_HASH")); @@ -133,7 +134,10 @@ fn pinned_public_quickjs_baseline_applies_cleanly_without_git() { .count(), 3 ); - assert!(jit_header.contains("QJSJIT_RUNTIME_API_MINOR 9u")); + assert!(jit_header.contains("QJSJIT_RUNTIME_API_MINOR 10u")); + assert!(helper_header.contains("X(GENERIC_OP, generic_op, JS_JitHelperGenericOp")); + assert!(quickjs.contains("JSJitHelperStatus JS_JitHelperGenericOp(")); + assert!(quickjs.contains("rt->jit_active_root = root_call.previous;")); assert!(jit_header.contains("#define QJSJIT_RUNTIME_FIELD_MAP_OUT_IN_OP(field)")); assert!(helper_header.contains("JS_JIT_HELPER_MATERIALIZED = 2")); assert!(helper_header.contains("JS_JIT_OWNER_SOURCE_ARGUMENT = 0")); @@ -255,7 +259,7 @@ fn bundled_jit_bindings_include_materialize_owner_tail() { for target in targets { let binding = fs::read_to_string(bindings.join(target)).unwrap(); assert!( - binding.contains("pub const QJSJIT_ABI_MINOR: u32 = 24;"), + binding.contains("pub const QJSJIT_ABI_MINOR: u32 = 25;"), "{target}" ); assert!( @@ -299,15 +303,18 @@ fn bundled_jit_bindings_include_materialize_owner_tail() { assert!( binding.contains("JS_JIT_HELPER_BINARY_ARITH_SLOW: JSJitHelperId = 22") && binding.contains("JS_JIT_HELPER_UNARY_ARITH_SLOW: JSJitHelperId = 23") - && binding.contains("JS_JIT_HELPER_COUNT: JSJitHelperId = 24"), + && binding.contains("JS_JIT_HELPER_GENERIC_OP: JSJitHelperId = 24") + && binding.contains("pub generic_op: ::core::option::Option") + && binding.contains("pub fn JS_JitHelperGenericOp(") + && binding.contains("JS_JIT_HELPER_COUNT: JSJitHelperId = 25"), "{target}" ); assert!( - binding.contains("pub const QJSJIT_RUNTIME_API_MINOR: u32 = 9;"), + binding.contains("pub const QJSJIT_RUNTIME_API_MINOR: u32 = 10;"), "{target}" ); assert!( - binding.contains("size_of::() - 200usize"), + binding.contains("size_of::() - 208usize"), "{target}" ); } @@ -363,7 +370,12 @@ fn bundled_wasm_jit_binding_matches_array_abi_1_24() { "pub const JS_JIT_FEEDBACK_ARRAY_SHARED: u32 = 8192;", "pub const JS_JIT_FEEDBACK_ARRAY_INVALID_BACKING: u32 = 16384;", // ABI 1.24: typed-array continuing guard. - "pub const QJSJIT_ABI_MINOR: u32 = 24;", + "pub const QJSJIT_ABI_MINOR: u32 = 25;", + // ABI 1.25: exact generic-opcode helper. + "JS_JIT_HELPER_GENERIC_OP: JSJitHelperId = 24;", + "pub fn JS_JitHelperGenericOp(", + "size_of::() - 108usize", + "offset_of!(JSJitRuntimeAPI, generic_op) - 104usize", "pub const QJSJIT_ARRAY_API_VERSION: u32 = 1;", "pub const JS_JIT_ARRAY_QUERY_LENGTH: u32 = 1;", "JSJitArrayQueryStatus_JS_JIT_ARRAY_QUERY_INVALID: JSJitArrayQueryStatus = -1;", diff --git a/sys/tests/typed_array_guard.rs b/sys/tests/typed_array_guard.rs index 2195d1454..8f485471d 100644 --- a/sys/tests/typed_array_guard.rs +++ b/sys/tests/typed_array_guard.rs @@ -105,7 +105,7 @@ int main(void) void *expected_data; uint32_t expected_count; - assert(QJSJIT_ABI_MAJOR == 1 && QJSJIT_ABI_MINOR == 24); + assert(QJSJIT_ABI_MAJOR == 1 && QJSJIT_ABI_MINOR == 25); assert(QJSJIT_ARRAY_API_VERSION == 1); assert(ctx && api); assert(api->struct_size == sizeof(*api)); From 354a86c75cbecd740302b6b1e5ea19262e562c38 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:24:48 +0800 Subject: [PATCH 28/54] perf(jit): keep the Tier 2 candidate scan off the exception check Maintenance runs every 32 calls in callback-heavy code, so checking every optimizing snapshot for exception regions measurably slowed mixed-quotes (noisy diagnostic: 0.98x of 82d3808). Compute the flag once at verification and consult it only for candidates that are otherwise ready for Tier 2. Co-Authored-By: Claude Opus 5.5 (1M context) --- jit/src/bytecode/verify.rs | 18 +++++++++++------- jit/src/lib.rs | 15 ++++++--------- 2 files changed, 17 insertions(+), 16 deletions(-) diff --git a/jit/src/bytecode/verify.rs b/jit/src/bytecode/verify.rs index a9cb1f3b2..9d690e415 100644 --- a/jit/src/bytecode/verify.rs +++ b/jit/src/bytecode/verify.rs @@ -121,6 +121,7 @@ pub struct VerifiedFunction { osr_points: Box<[super::OsrPoint]>, exception_handlers: std::collections::BTreeMap, unreachable: std::collections::BTreeSet, + exception_regions: bool, } impl VerifiedFunction { @@ -140,14 +141,9 @@ impl VerifiedFunction { /// True when any instruction executes with a live catch offset or the /// bytecode contains a try/finally transfer. Consumers that cannot model /// exceptional control flow (inlining, direct leaf calls) must reject it. + /// Computed once at verification: maintenance consults it on every scan. pub fn has_exception_regions(&self) -> bool { - !self.exception_handlers.is_empty() - || self.instructions.iter().any(|instruction| { - matches!( - instruction.opcode().name(), - "catch" | "nip_catch" | "gosub" | "ret" | "throw" | "throw_error" - ) - }) + self.exception_regions } pub fn snapshot(&self) -> &CompileSnapshot { @@ -455,6 +451,13 @@ pub(crate) fn verify( }) .collect::, VerifyError>>()? .into_boxed_slice(); + let exception_regions = !proof.handlers.is_empty() + || instructions.iter().any(|instruction| { + matches!( + instruction.opcode().name(), + "catch" | "nip_catch" | "gosub" | "ret" | "throw" | "throw_error" + ) + }); Ok(VerifiedFunction { snapshot, instructions, @@ -462,5 +465,6 @@ pub(crate) fn verify( osr_points, exception_handlers: proof.handlers, unreachable, + exception_regions, }) } diff --git a/jit/src/lib.rs b/jit/src/lib.rs index 076f1d72c..ce7472b3a 100644 --- a/jit/src/lib.rs +++ b/jit/src/lib.rs @@ -2247,15 +2247,6 @@ impl ProductionBackend { if self.feedback_disabled.contains(key) { return false; } - // Tier 2 has no exceptional control flow; exception regions - // stay on their Tier 1 code instead of failing admission. - if self - .optimizing_snapshots - .get(key) - .is_some_and(bytecode::VerifiedFunction::has_exception_regions) - { - return false; - } if self .profitability_backoff .get(key) @@ -2285,6 +2276,12 @@ impl ProductionBackend { let Some(snapshot) = self.optimizing_snapshots.get(&key) else { continue; }; + // Tier 2 has no exceptional control flow; exception regions stay + // on their Tier 1 code instead of failing admission. Checked only + // for ready candidates so the per-maintenance scan stays as is. + if snapshot.has_exception_regions() { + continue; + } let observed = self .feedback .snapshot(self.clock.max(1)) From d5b40f4f293d2d2c568da7165e2564882878674a Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:28:21 +0800 Subject: [PATCH 29/54] docs(jit): record p2c-exceptions diagnostics and verification Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p2c-exceptions.md | 54 ++++++++++++++++++++++++---------- 1 file changed, 39 insertions(+), 15 deletions(-) diff --git a/docs/roadmap/p2c-exceptions.md b/docs/roadmap/p2c-exceptions.md index 0a1d6fda3..db0d90d4c 100644 --- a/docs/roadmap/p2c-exceptions.md +++ b/docs/roadmap/p2c-exceptions.md @@ -53,25 +53,49 @@ Branch `perf/p2c-exceptions`, based on `82d3808`. Roadmap item P2.3 in ## Noisy diagnostic (not publishable evidence) Median over 7 alternating worker runs of `median(warmup_batch_ns[-16:])`, -automatic tiering, on the shared 24-core host while ~16 agents were building -and testing. Baseline is the `82d3808` `jit-bench`; speed is baseline time / -new time. Checksums matched the interpreter in every run. +automatic tiering, measured at `354a86c` on the shared 24-core host while ~16 +agents were building and testing. Baseline is the `82d3808` `jit-bench`; +speed is baseline time / new time (above 1.00x is faster). Checksums matched +the interpreter in every run. The interpreter column is one same-host run. -| workload | 82d3808 ms | p2c ms | speed | -| --- | ---: | ---: | ---: | -| exceptions-sync (new) | 2.9537 | 1.6643 | 1.78x | -| exceptions-promises-async | 3.7634 | 2.4500 | 1.54x | -| adversarial | 0.9545 | 0.9501 | 1.00x (tied) | -| scalar-loop (control) | 0.0293 | 0.0290 | 1.01x (tied) | -| call-heavy (control) | 0.0599 | 0.0605 | 0.99x (tied) | -| mixed-quotes (control) | 2.5143 | 2.5564 | 0.98x (tied) | +| workload | 82d3808 ms | p2c ms | speed | interpreter ms | +| --- | ---: | ---: | ---: | ---: | +| exceptions-sync (new) | 2.9318 | 0.8765 | 3.35x | 2.9442 | +| exceptions-promises-async | 3.6864 | 2.4385 | 1.51x | 2.3155 | +| adversarial | 0.9528 | 0.9526 | 1.00x | 0.9347 | +| mixed-quotes (control) | 2.4885 | 2.5132 | 0.99x | 2.7690 | +| property-heavy (control) | 0.0981 | 0.0968 | 1.01x | 1.0673 | +| generic-call-entry (control) | 0.0323 | 0.0323 | 1.00x | 0.9689 | +| scalar-loop (control) | 0.0540 | 0.0300 | inconclusive | 0.5405 | +| call-heavy (control) | 0.0853 | 0.0598 | inconclusive | 1.2823 | -`exceptions-promises-async` is now at interpreter speed (its `workload` and -`asyncStep` are async and stay interpreted; the gain is the removed probe -overhead). `adversarial` has no exception region; its Tier 1 code is rejected -by the profitability gate, which this item does not change. The publishable +The scalar-loop and call-heavy baseline medians were inflated by host load: +earlier rounds measured 0.0291/0.0290 ms and 0.0595/0.0608 ms, and an A/A run +of the same `82d3808` binary against itself spread call-heavy between 0.95x +and 1.77x. Neither control executes code this item changes. An earlier +candidate checked every Tier 2 candidate snapshot for exception regions on +each maintenance pass and measured mixed-quotes at 0.98x in three rounds; the +check now runs once at verification and only for ready candidates (0.999x on +11 runs, 0.99x above). + +`exceptions-sync` uses Tier 1 with native handlers (3.35x the previous JIT, +about 3.4x the interpreter). `exceptions-promises-async` is now at interpreter +speed: its `workload` and `asyncStep` are async and stay interpreted, and the +gain is the removed probe overhead (2,030 to 7 snapshot requests per worker +run). `adversarial` has no exception region; its Tier 1 code is rejected by +the profitability gate, which this item does not change. The publishable three-engine matrix, including `exceptions-sync`, is left to the integrator. +## Verification + +- `cargo test --release -p quickjs-jit-runtime --features compiler,test-support --tests`: + 797 passed, 0 failed, 1 ignored at `354a86c`. +- `cargo test -p quickjs-jit-sys --test jit_patch`: 13 passed. +- `cargo clippy -p quickjs-jit-runtime --all-targets --features compiler,test-support -- -D warnings`: clean. +- `mixed_direct_calls::tier2_mixed_edge_eliminates_helper_and_preserves_guard_misses` + failed once in an earlier full run on this branch. It is timing-dependent and + also fails on an unmodified `82d3808` build on this host (3 of 19 runs). + ## Remaining work - Tier 2 exception regions (landing pads with deopt-map ownership, handler From 7053984d0161daf8435b711824af9bbc510390d7 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:29:10 +0800 Subject: [PATCH 30/54] docs: design the P3a native-to-native call convention Describe the target JIT->JIT call ABI (monomorphic call-target IC with identity, generation and publication epoch; native callee frames; lazy JSStackFrame materialization on deopt, exception, GC stack walk and backtrace; interaction with the stack walker, interrupts, stack-overflow checks and GC roots) and the first fail-closed slice implemented on this branch, with noisy diagnostics and remaining work. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p3a-native-calls.md | 254 +++++++++++++++++++++++++++++++ 1 file changed, 254 insertions(+) create mode 100644 docs/roadmap/p3a-native-calls.md diff --git a/docs/roadmap/p3a-native-calls.md b/docs/roadmap/p3a-native-calls.md new file mode 100644 index 000000000..c71bbd263 --- /dev/null +++ b/docs/roadmap/p3a-native-calls.md @@ -0,0 +1,254 @@ +# P3a 原生调用约定：设计与第一切片 + +分支 `perf/p3a-native-calls`，基于 `82d3808`。对应 +[性能路线图](../PERFORMANCE_ROADMAP_V8_BUN.md) 的 B4/P3。本文先给出完整的 +JIT→JIT 原生调用约定设计，再说明本分支实际落地的第一个 fail-closed 切片、 +它刻意没有做的部分，以及后续工作。 + +## 1. 问题 + +目前 compiled→compiled 调用只有两条快速路径： + +1. 受限的 Int32/Bool **无副作用叶函数**直调（`lower_direct_call_machine`、 + `lower_direct_bool_leaf`）：`(out*, 标量参数...) -> status`，status 非零时 + 调用方在 CALL 字节码处 deopt，由解释器重新执行这次调用。 +2. 有界的 effect-free / frame 内联。 + +其余调用都走 `JS_JitHelperCall → JS_Call → JS_CallInternal → 回调 → +原生入口`，每次调用都要经过 helper 帧校验（B1）和 JIT 簿记回调（B2）。 +递归函数因此被 `maintenance` 的 `generic_call_without_loop` 规则留在 +解释器里；强制 Tier 2 时 `fibonacci-recursive` 比解释器慢约 7.8x。 + +## 2. 完整设计（目标状态） + +### 2.1 调用 ABI + +每个可原生调用的 Tier 2 artifact 额外发布一个 **native call entry**： + +```text +status = entry(nc: *mut JSJitNativeCallContext, out: *mut Scalar, args...) +``` + +- `nc` 是按"最外层原生调用"分配在调用方栈上的上下文：栈下限、剩余中断 + 预算、状态标志，以及（后续切片）惰性帧链的头指针。所有嵌套原生调用 + 共享同一个 `nc`，因此一次原生调用链只需要一次进入/退出簿记。 +- 参数按 feedback 签名拆箱：Int32/Bool 为 `i32`，Float64 为 `f64`，借用的 + 堆引用为对象指针（调用方保证在调用期间持有根）。结果写入 `out`。 +- `status`：`0` 成功；`1` 重试（被调方保证尚未产生任何可观察副作用，调用方 + 在 CALL 处 deopt，由解释器按原语义重新执行）；`2` 异常已挂在 + `ctx->rt->current_exception` 上，调用方按异常出口展开（后续切片）。 + +### 2.2 调用点 IC + +调用点使用单态 target IC： + +- **identity**：函数对象指针（被 feedback 根住）和 `JSObject::u.func.function_bytecode`； +- **generation**：被调方 `FunctionKey` 的 generation，写入调用方 artifact 的依赖； +- **epoch**：被调方 artifact 的发布 pin。调用方持有被调方 native entry 的 + `PublishedBaselineCode` 克隆，被调方被替换或淘汰时，调用方 artifact 随依赖 + 一起失效，旧代码在所有 pin 释放前保持可执行。 + +未命中 IC 时走原来的 generic CALL 路径（或在 CALL 处 deopt），绝不猜测。 + +### 2.3 原生帧与惰性 `JSStackFrame` 物化 + +原生 callee 不压 `JSStackFrame`。它的帧由 Cranelift 管理，并为每个可能 +离开原生代码的点（helper 调用、异常、deopt）记录一个 frame-state：函数身份、 +字节码 pc、参数/局部/操作数栈到机器位置的映射。只有下列事件才需要把原生 +帧物化成真实的 `JSStackFrame`： + +| 事件 | 处理 | +| --- | --- | +| deopt | 从最内层原生帧开始，按 frame-state 为链上每一层重建解释器帧（与 `0018-inline-frame-recovery` 的影子帧同构），然后在最内层 pc 恢复解释器。 | +| 异常 | 构造 `Error.stack` 之前，C 侧 backtrace 遍历器先询问 `nc` 链上的原生帧，按 frame-state 输出函数名和 pc；没有 try/catch 的原生帧直接返回 status 2 展开。 | +| GC 栈扫描 | QuickJS 使用引用计数加环检测；原生帧只持有借用引用（调用方的根保证存活）或被 frame-state 标注为拥有的值。拥有值在 helper 调用前溢出到 `nc` 管理的影子槽，`mark_children` 通过运行时上的原生帧链把它们当作根。 | +| backtrace / 调试 | 同异常。 | + +在做到帧物化之前，原生 callee 必须满足"失败时无副作用"，这正是第一切片 +的约束（见第 3 节）。 + +### 2.4 与 C 栈遍历器、中断和栈溢出的交互 + +- **栈遍历器**：`rt->current_stack_frame` 链不包含原生帧；目标状态下运行时 + 新增 `jit_native_frames` 链头，遍历器（backtrace、`Function.caller` 类、 + GC）在遇到 JIT 入口帧时插入原生帧。第一切片的原生帧从不对外可见：没有 + helper、没有异常、没有分配，因此无需遍历。 +- **中断**：解释器在每次 `JS_CallInternal` 入口递减 + `ctx->interrupt_counter`。原生入口同样按调用递减 `nc->budget`（从 + `ctx->interrupt_counter` 读入，退出时写回）。预算耗尽时：没有安装 + interrupt handler 就只重置预算（与 `__js_poll_interrupts` 相同）；安装了 + handler 则调用 handler，返回 0 时继续原生执行，返回非零时把 + `ctx->interrupt_counter` 置 0 并返回 status 1，由解释器重试这次调用、在 + 它自己的 poll 点抛出不可捕获的 `interrupted`（异常与 backtrace 与解释器 + 完全一致；handler 会多被调用一次，这只对宿主可见）。 +- **栈溢出**：解释器用 `js_check_stack_overflow(rt, alloca_size)` 比较栈指针 + 和 `rt->stack_limit`。原生入口在序言里做同样的比较，并且按"解释器帧的 + 保守开销"累计虚拟栈深度（见 3.4），保证原生递归不会在解释器会抛 + `RangeError` 的深度之后才失败。失败返回 status 1，由解释器以真实帧 + 重新执行并在同样的检查处抛出 `RangeError`。为避免每一层解释器帧重复一次 + 原生递归（二次复杂度），失败时记录一个线程局部的栈水位；在更深的位置 + 不再尝试原生调用，回到更浅的位置后自动清除。 +- **GC 根**：原生参数只有标量和借用引用；借用引用的根由最外层调用方持有 + （参数/局部或 owned 栈槽）。第一切片不传堆参数。 + +## 3. 第一切片（本分支） + +### 3.1 范围 + +- **native call entry**（`jit/src/compiler/native_call.rs`）：Tier 2 编译一个 + 函数时，如果它的 `bounded_specialization` 签名是 Int32/Float64 参数、 + Int32/Float64 结果，且函数体只包含下列字节码，就在同一个 artifact 里 + 额外发布一个 native entry： + - `get_arg*`、整数常量（`push_*`）、常量池里的 Int32/Float64 + （`push_const`/`push_const8`）； + - Int32 的 `add`/`sub`/`mul`/`inc`/`dec`（溢出和 `-0` 失败），Float64 的 + `add`/`sub`/`mul`/`div`/`inc`/`dec`（精确 IEEE-754，Int32 操作数先精确 + 扩宽），比较 `lt`/`lte`/`gt`/`gte`/`eq`/`neq`/`strict_eq`/`strict_neq` + （Float64 用有序比较，NaN 结果与解释器一致）、`lnot`、`drop`、`nop`； + - 前向 `if_true`/`if_false`/`goto`（汇合点的操作数栈按 Cranelift block + 参数传递，类型必须一致），`return`； + - 对自身的调用：`get_var ` + `call0..3`/`call`/`tail_call`，实参个数 + 等于形参个数，所有 `get_var` 必须是同一个 atom。 + + 任何其他字节码、异常表、闭包变量、局部变量、回边（循环）、Bool/堆引用 + 签名都让 native entry 不发布（fail-closed），函数照旧编译。 +- **Tier 2 调用点**（`emit_opt_native_call`）：函数自己的自递归调用点，以及 + 调用另一个已发布 native entry 的单态调用点（coordinator 在排队时把 + 被调方的 native entry 作为 `NativeCallTarget` 交给请求，并保持其发布 pin）， + 按以下顺序执行：保护 callee 对象身份和字节码身份 → 保护实参 tag 与签名 + 一致 → `qjsjit_native_call_begin`（Rust，包一层线程局部栈水位后调用补丁 + 0029 的 `JS_JitNativeCallBegin`）→ 调用 native entry → `qjsjit_native_call_end` + → 释放 `get_var` 得到的函数引用（`FREE` helper）。任一步失败都在 CALL 处 + deopt，由解释器重新执行这次调用（与现有叶直调的失败路径相同）。 + 有 native target 的调用点不再参与 pure/frame 内联（内联一层递归函数只会 + 让内层调用走 generic bridge）。 +- **分层策略**（`jit/src/lib.rs` 的 `maintenance`）： + - 自递归调用点被新路径接纳的函数，不再被 `generic_call_without_loop` 挡在 + 解释器；调用一个已就绪 native entry 的无循环函数同样豁免。 + - 进入原生代码仍有几百纳秒的簿记开销（B2），约等于十次解释器调用， + 所以只接纳"每次外部进入平均至少 16 次调用"的递归：baseline 在 + `native_enter` 统计外部进入次数（父执行不是同一函数），与 baseline + 执行次数相比。数据不足时保持 baseline 等待（最多等到 64 次外部进入）， + 之后仍然太浅就按原规则拒绝。`calls-recursion-closures` 的 + `recursiveSum(i & 7)` 因此继续留在解释器，没有退步。 + - 这类函数获得一次有界 Tier 2 试验；它们的 Tier 2 单次调用覆盖整棵 + 递归树，不能与逐次调用的 baseline 样本比较，所以不参加 + `classify_tier2_trial` 的单次时间比较。deopt 统计和已有的 side-exit + 预算仍然有效。 + - 调用方在单态被调方的 native entry 发布之前保持 baseline（与等待标量 + 直调 entry 相同），避免把调用点永久编译成 generic CALL。 + +### 3.2 为什么"重试"是精确的 + +原生 entry 的函数体没有任何副作用：不调用 helper（中断轮询除外，见 3.4）、 +不分配、不读写堆。自调用的 `get_var ` 在原生代码里**不执行**，而是在 +进入原生调用链前由 `JS_JitNativeCallBegin` 一次性证明： + +1. 被调对象是 `JS_CLASS_BYTECODE_FUNCTION`、`func_kind == JS_FUNC_NORMAL`， + 其 realm 就是调用方的 context（调用点已经保护了对象和字节码身份）； +2. 按 `JS_GetGlobalVar` 的查找顺序，名字解析为普通数据属性 + （`global_var_obj` 中的词法绑定，或 `global_obj` 自有的 + `JS_PROP_NORMAL` 属性），值就是这个函数对象本身。未初始化的词法绑定 + （值不是该对象）、访问器、auto-init、var-ref 都拒绝。 + +链中不会运行任何 JS 代码，所以这个绑定在整条链执行期间不变，每一次内部 +`get_var` 都会得到同一个值、不会调用 getter、不会抛异常。任何检查失败 +（Int32 溢出、`-0`、栈深、中断请求）都只丢弃纯计算，然后在最外层 CALL 处 +deopt，由解释器重新执行，结果、异常类型、消息和 backtrace 都来自解释器。 + +测试（`jit/tests/native_calls.rs`）覆盖：自动分层和强制 Tier 2 下的 +native 递归（稳态无 generic CALL helper、每次求值只有一次 QuickJS 入口）； +深处 Int32 溢出、非 Int32 实参、`-0`；运行中重绑定全局函数、改成访问器 +（getter 调用次数与解释器一致）、被全局 `let` 遮蔽；栈耗尽抛出 +`RangeError` 且不出现二次复杂度、解释器会抛错的深度原生代码也抛错； +interrupt handler 被原生链轮询并能中断；单态调用方链接到 native entry、 +重绑定后回退；浅递归被拒绝；Float64 递归与签名保护。 + +### 3.3 发布与生命周期 + +- native entry 与主代码在同一个 artifact 中编译：`publish_relocatable` + 先发布 native entry，再发布主代码，主代码里的 `func_addr` 符号重定位 + （`qjsjit.native_call_entry`）解析为 native entry 地址；主代码（及其 OSR + 代码）的发布分配持有 native entry 的发布 pin，释放顺序是先主代码后 entry。 +- native entry 调用自身使用 colocated 用户名（`NATIVE_SELF_NAMESPACE`）， + 最终重定位解析为自身基址（x86_64 `X86CallPCRel4`，aarch64 `Arm64Call`）。 +- 跨函数调用方把被调方 native entry 的发布 pin 放入 + `direct_call_dependencies`，与现有叶直调一致。native entry 只依赖字节码， + 被调方 artifact 被替换后旧 entry 仍然正确，调用点的身份保护负责选择。 +- native entry 有 unwind 信息，但没有 stack map 或 frame state；它从不在 + helper、GC 或异常中出现在栈遍历器面前。 + +### 3.4 中断、栈深度和 GC + +- **中断**：`Begin` 从 `ctx->interrupt_counter` 读入预算（没有 handler 时 + 为 `INT32_MAX`，与 `__js_poll_interrupts` 只重置计数器等价）。每次原生 + 调用递减一次（对应 `JS_CallInternal` 入口的 poll）。预算耗尽时调用 + `JS_JitNativeCallPoll`：handler 返回 0 就重置预算继续；返回非零则标记 + `INTERRUPT_DUE` 并让整条链重试，`End` 把 `ctx->interrupt_counter` 置 0， + 重试的 CALL 进入 `JS_CallInternal` 时立即 poll，由解释器抛出不可捕获的 + `interrupted`（异常和 backtrace 与解释器一致；handler 因此多被调用一次， + 这只对宿主可见）。 +- **栈深度**：native entry 的第三个参数是"虚拟栈指针"，最外层取调用点的 + 真实栈指针，每层减去 `frame_charge`：`16 × (参数 + 局部 + 栈 + 4)` 加上 + **本构建实测**的解释器单层开销（`interpreter_frame_bytes`：在一个无 JIT 的 + 私有 runtime 里，比较宿主探针在递归深度 0 和 64 时的栈指针，再加 25% + 余量；x86_64 release 实测约 1984 字节/层，取 2480）。检查条件等价于 + 解释器的 `js_check_stack_overflow`（`vsp - charge < rt->stack_limit`）。 + 因为每层至少按解释器开销计费，原生链总在解释器会抛 `RangeError` 的深度 + 之前失败，然后由解释器以真实帧复现。失败时 `End` 在线程局部记录最外层 + 调用点的栈地址；在这个地址及更深处不再尝试原生链（否则每一层解释器帧都 + 会重新跑一次接近栈限的原生递归，变成二次复杂度），回到更浅处自动清除。 +- **GC**：链中没有分配，也没有堆值；借用的函数对象由调用方 owned 栈槽和 + 全局绑定共同持有。 + +### 3.5 本切片不做（后续工作） + +- 堆引用参数/结果、Bool 参数/结果、局部变量、循环、helper 调用、属性访问、 + 闭包变量、`this`、`arguments`； +- 原生 entry 内部调用**其他**函数（需要对每个全局名做同样的绑定证明，或者 + 依赖 2.3 的帧物化）； +- status 2（原生帧内的异常）和惰性帧物化、栈遍历器集成、GC 根扫描——这是 + 放开"失败时无副作用"约束的前提； +- Tier 1 调用方和 C→JS 回调（`sort`/`map`）使用 native entry； +- 用 P1 的 C 侧入口缓存降低 QuickJS→native 的进入成本后，再降低 + "每次外部进入至少 16 次调用"的门槛（`calls-recursion-closures`）。 + +## 4. 噪声诊断（非发布数据） + +主机 i7-13700KF，24 核 / 31 GB，同时运行约 16 个 agent；未绑核，P/E 核 +导致明显的双峰分布。每个场景交替运行基线（`82d3808` 的 +`target/release/jit-bench`）和本分支二进制，取每次运行最后 16 个预热批次的 +中位数，再取各次运行的中位数；速度 = 基线 / 新版本（>1 更快）。单位 +ms / 10 次 workload 调用。这不是 AGENTS.md 要求的发布级配对证据，README +矩阵未修改。 + +| 场景（automatic） | 运行次数 | 基线 ms | 本分支 ms | 速度 | 备注 | +| --- | ---: | ---: | ---: | ---: | --- | +| fibonacci-recursive | 9 | 9.8984 | 0.3892 | 25.4x | 同一二进制解释器 9.82 ms → 本分支约 25x 解释器（目标 ≥ 5x） | +| calls-recursion-closures | 9 | 9.0645 | 6.4681 | 1.40x（噪声） | 两边各次最小值 6.44 / 6.45，实为持平；`recursiveSum(i&7)` 被浅递归门槛拒绝，留在解释器 | +| call-heavy | 9 | 0.1140 | 0.1118 | 1.02x | 持平（已走叶直调，本切片不改变） | +| mixed-quotes | 7 | 2.9061 | 2.6447 | 1.10x（噪声） | 双峰，持平 | +| generic-call-entry | 7 | 0.0576 | 0.0548 | 1.05x（噪声） | 持平 | +| quickjs-fibonacci | 7 | 0.5326 | 0.3709 | 1.44x（噪声） | 两边最小值 0.370 / 0.359，持平 | +| numeric | 7 | 0.0298 | 0.0293 | 1.02x | 持平 | +| property-heavy | 7 | 0.0988 | 0.0985 | 1.00x | 持平 | + +强制 Tier 2 诊断：`fibonacci-recursive` 从 74.6 ms 降到 0.368 ms（此前比 +解释器慢 7.6x，现在快约 27x）；`calls-recursion-closures` 从 85.7 ms 降到 +44.2 ms（仍慢于解释器 6.4 ms，闭包路径不在本切片范围内）。 + +全部 30 个场景在 interpreter / tier1 / automatic / tier2 四种模式下的 +checksum 与解释器一致。按第 2 节的稳态口径，Bun 的 `fibonacci-recursive` +约 0.274 ms，本分支约为 Bun 速度的 0.7x（噪声诊断，待集成者的三引擎矩阵 +确认）。 + +## 5. 已知的既有问题（在 `82d3808` 上同样复现） + +- 不可捕获的 `interrupted` 异常展开 Tier 2 帧后，如果宿主没有调用 + `ctx.catch()` 清除异常，下一次求值中的任何 JIT helper 都会因 + `JS_HasException` 而抛出 "invalid JIT helper state"（解释器不受影响）。 + 本分支的中断测试在中断后清除异常。 +- 强制 Tier 2 的 `call-heavy` 和 `generic-call-entry` 多数运行约 5.6 ms + （偶尔 0.06 ms），基线二进制同样如此。 +- frame 内联一个递归被调方时，调用方每次求值 deopt 一次；有 native target + 的调用点现在不再内联，因此不受影响。 From 963a6bf0619b9651473b9e7a3330735e03330b27 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:29:10 +0800 Subject: [PATCH 31/54] sys: add patch 0029 with native-call chain support functions JS_JitNativeCallBegin proves once per native call chain that a callee's global self reference resolves, exactly like OP_get_var in the callee realm, to a plain data binding holding the callee, and loads the stack limit and interrupt budget into a caller-owned context. JS_JitNativeCallPoll is __js_poll_interrupts without throwing, and JS_JitNativeCallEnd publishes the chain's interrupt accounting. The functions are appended to quickjs.c outside the versioned helper table, so the runtime API/helper ABI version and the bundled bindings are unchanged. The patched quickjs.c fingerprint is updated. Co-Authored-By: Claude Opus 5.5 (1M context) --- sys/build_support/patch.rs | 5 +- sys/patches/0029-native-call-convention.patch | 125 ++++++++++++++++++ sys/patches/README.md | 13 ++ sys/tests/jit_patch.rs | 1 + 4 files changed, 142 insertions(+), 2 deletions(-) create mode 100644 sys/patches/0029-native-call-convention.patch diff --git a/sys/build_support/patch.rs b/sys/build_support/patch.rs index 41317e2f2..53dcddac6 100644 --- a/sys/build_support/patch.rs +++ b/sys/build_support/patch.rs @@ -1,7 +1,7 @@ use std::{fs, io, path::Path}; const QUICKJS_BASELINE_FNV64: u64 = 0x3302_116a_b0fc_269c; -const EXPECTED_PATCHES: [(&str, u64); 19] = [ +const EXPECTED_PATCHES: [(&str, u64); 20] = [ ("0001-rquickjs-jit.patch", 0x18d7_40cc_6943_30cf), ("0002-runtime-feedback.patch", 0xe4c1_ed52_ee5d_d469), ("0003-element-layout.patch", 0xa12c_ccd2_7420_e880), @@ -21,6 +21,7 @@ const EXPECTED_PATCHES: [(&str, u64); 19] = [ ("0018-inline-frame-recovery.patch", 0xb971d61edccde229), ("0019-array-feedback.patch", 0x11c3cb13a8297dd4), ("0020-typed-array-guard.patch", 0x5d10_38a3_8bc9_9821), + ("0029-native-call-convention.patch", 0x6bbd_3477_da28_d2f8), ]; pub(crate) const BASELINE_FILES: [&str; 19] = [ "api-test.c", @@ -60,7 +61,7 @@ const PATCHED_FILE_FINGERPRINTS: [(&str, u64); 21] = [ ("list.h", 0xb337_70f7_b76d_a3d8), ("quickjs-atom.h", 0x30b4_9116_b6a2_aa99), ("quickjs-c-atomics.h", 0x490b_0f29_f631_3fc0), - ("quickjs.c", 0xc535_b5f1_48e7_9b27), + ("quickjs.c", 0x3d9e_5139_db1c_b47b), ("quickjs-jit.h", 0x0188_04ad_21d2_95a7), ("quickjs-jit-helpers.h", 0xa469_e80f_be47_404c), ("quickjs-opcode.h", 0x3d05_cfdf_5cf7_2930), diff --git a/sys/patches/0029-native-call-convention.patch b/sys/patches/0029-native-call-convention.patch new file mode 100644 index 000000000..64cbb4f52 --- /dev/null +++ b/sys/patches/0029-native-call-convention.patch @@ -0,0 +1,125 @@ +diff --git a/quickjs.c b/quickjs.c +--- a/quickjs.c ++++ b/quickjs.c +@@ -67202,3 +67202,121 @@ + #undef malloc + #undef free + #undef realloc ++ ++#ifdef CONFIG_JIT_ABI ++/* Native-call convention (P3a). A native call entry is generated code that ++ never runs JavaScript, a JIT helper or an allocator: it only computes on ++ unboxed scalars and calls itself. One context lives on the stack of the ++ outermost compiled caller and is shared by the whole native call chain. ++ Every failure makes the outermost caller re-execute the CALL in the ++ interpreter, which is exact because nothing observable happened. */ ++typedef struct JSJitNativeCallContext { ++ uintptr_t stack_limit; /* rt->stack_limit, zero when unlimited */ ++ int32_t budget; /* remaining interrupt polls in this chain */ ++ uint32_t flags; /* JS_JIT_NATIVE_CALL_* */ ++ JSContext *ctx; ++} JSJitNativeCallContext; ++ ++#define JS_JIT_NATIVE_CALL_STACK_EXHAUSTED (1u << 0) ++#define JS_JIT_NATIVE_CALL_INTERRUPT_DUE (1u << 1) ++#define JS_JIT_NATIVE_CALL_BUDGET_LIMITED (1u << 2) ++ ++int JS_JitNativeCallBegin(JSContext *ctx, JSObject *p, JSAtom atom, ++ JSJitNativeCallContext *nc); ++int JS_JitNativeCallPoll(JSJitNativeCallContext *nc); ++void JS_JitNativeCallEnd(JSJitNativeCallContext *nc); ++size_t JS_JitNativeCallContextLayout(size_t *offsets, size_t count); ++ ++/* Proves, once per outermost native call chain, that the callee's self ++ reference `atom` resolves exactly like OP_get_var in the callee realm to a ++ plain data binding holding `p`. No JavaScript or host code runs inside the ++ chain (the interrupt handler is only polled through JS_JitNativeCallPoll), ++ so every OP_get_var of `atom` in the chain yields the same function ++ without invoking a getter or throwing. Returns 0 when admitted. */ ++int JS_JitNativeCallBegin(JSContext *ctx, JSObject *p, JSAtom atom, ++ JSJitNativeCallContext *nc) ++{ ++ JSRuntime *rt; ++ JSFunctionBytecode *b; ++ JSShapeProperty *prs; ++ JSProperty *pr; ++ ++ if (!ctx || !p || !nc || p->class_id != JS_CLASS_BYTECODE_FUNCTION) ++ return -1; ++ b = p->u.func.function_bytecode; ++ /* The chain shares the caller's interrupt counter and global lookup. */ ++ if (!b || b->realm != ctx || b->func_kind != JS_FUNC_NORMAL) ++ return -1; ++ prs = find_own_property(&pr, JS_VALUE_GET_OBJ(ctx->global_var_obj), atom); ++ if (!prs) { ++ prs = find_own_property(&pr, JS_VALUE_GET_OBJ(ctx->global_obj), atom); ++ if (!prs) ++ return -1; ++ } ++ /* Accessors, auto-init and var-ref slots can run code or change; an ++ uninitialized lexical binding is not the function object below. */ ++ if ((prs->flags & JS_PROP_TMASK) != JS_PROP_NORMAL) ++ return -1; ++ if (JS_VALUE_GET_TAG(pr->u.value) != JS_TAG_OBJECT || ++ JS_VALUE_GET_OBJ(pr->u.value) != p) ++ return -1; ++ rt = ctx->rt; ++ nc->ctx = ctx; ++ nc->stack_limit = rt->stack_limit; ++ nc->flags = 0; ++ if (rt->interrupt_handler) { ++ nc->flags |= JS_JIT_NATIVE_CALL_BUDGET_LIMITED; ++ nc->budget = ctx->interrupt_counter; ++ } else { ++ /* __js_poll_interrupts only resets the counter without a handler. */ ++ nc->budget = INT32_MAX; ++ } ++ return 0; ++} ++ ++/* Called by a native entry after its interrupt budget reached zero. This is ++ __js_poll_interrupts without throwing: an interrupt request makes the chain ++ retry in the interpreter, whose own poll then throws the uncatchable error ++ with an exact backtrace. Returns non-zero to request that retry. */ ++int JS_JitNativeCallPoll(JSJitNativeCallContext *nc) ++{ ++ JSRuntime *rt = nc->ctx->rt; ++ ++ if (!(nc->flags & JS_JIT_NATIVE_CALL_BUDGET_LIMITED)) { ++ nc->budget = INT32_MAX; ++ return 0; ++ } ++ nc->budget = JS_INTERRUPT_COUNTER_INIT; ++ if (rt->interrupt_handler && ++ rt->interrupt_handler(rt, rt->interrupt_opaque)) { ++ nc->flags |= JS_JIT_NATIVE_CALL_INTERRUPT_DUE; ++ nc->budget = 0; ++ return 1; ++ } ++ return 0; ++} ++ ++/* Publishes the chain's interrupt accounting back to the context. A due ++ interrupt leaves the counter exhausted so the retried CALL polls first. */ ++void JS_JitNativeCallEnd(JSJitNativeCallContext *nc) ++{ ++ JSContext *ctx = nc->ctx; ++ ++ if (nc->flags & JS_JIT_NATIVE_CALL_INTERRUPT_DUE) ++ ctx->interrupt_counter = 0; ++ else if (nc->flags & JS_JIT_NATIVE_CALL_BUDGET_LIMITED) ++ ctx->interrupt_counter = nc->budget > 0 ? nc->budget : 0; ++} ++ ++/* Layout check for generated code: size followed by field offsets. */ ++size_t JS_JitNativeCallContextLayout(size_t *offsets, size_t count) ++{ ++ if (offsets && count >= 4) { ++ offsets[0] = offsetof(JSJitNativeCallContext, stack_limit); ++ offsets[1] = offsetof(JSJitNativeCallContext, budget); ++ offsets[2] = offsetof(JSJitNativeCallContext, flags); ++ offsets[3] = offsetof(JSJitNativeCallContext, ctx); ++ } ++ return sizeof(JSJitNativeCallContext); ++} ++#endif /* CONFIG_JIT_ABI */ diff --git a/sys/patches/README.md b/sys/patches/README.md index 931cf9bd3..d0ef56223 100644 --- a/sys/patches/README.md +++ b/sys/patches/README.md @@ -60,6 +60,19 @@ data pointer before returning `{ data, count, mode }`. The table advertises zero effects and the receiver is passed by pointer so generated calls do not depend on platform-specific aggregate argument classification. +`0029-native-call-convention.patch` adds the P3a native-call support +functions used by pure self-recursive Tier 2 native entries. They are plain +exported functions appended to `quickjs.c`, outside the versioned helper table, +so the runtime API/helper ABI version and the bundled bindings are unchanged. +`JS_JitNativeCallBegin` proves once per native call chain that the callee's +global self reference resolves, as `OP_get_var` would in the callee realm, to a +plain data binding holding exactly the callee, and loads the stack limit and +interrupt budget into a caller-owned `JSJitNativeCallContext`. +`JS_JitNativeCallPoll` is `__js_poll_interrupts` without throwing (an +interrupt makes the chain retry in the interpreter), and `JS_JitNativeCallEnd` +publishes the chain's interrupt accounting. `JS_JitNativeCallContextLayout` +lets the compiler verify the context layout before generating code. + The build accepts only the patch names and byte digests listed in `build_support/patch.rs`, then verifies the complete patched source manifest. Keeping the baseline and patch separate makes diff --git a/sys/tests/jit_patch.rs b/sys/tests/jit_patch.rs index f36861117..80cd9bdce 100644 --- a/sys/tests/jit_patch.rs +++ b/sys/tests/jit_patch.rs @@ -56,6 +56,7 @@ fn copy_patches(destination: &std::path::Path) { "0018-inline-frame-recovery.patch", "0019-array-feedback.patch", "0020-typed-array-guard.patch", + "0029-native-call-convention.patch", ] { fs::copy(source.join(patch), destination.join(patch)).unwrap(); } From 3286bf61f4750724a1c47bc3e42759f4db702e1d Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:29:10 +0800 Subject: [PATCH 32/54] perf(jit): run pure self-recursion as native-to-native calls Tier 2 now publishes a native call entry beside the main body of a pure, self-recursive Int32/Float64 function: (nc, out, vsp, scalars...) -> status. The body only computes on unboxed scalars and calls itself through a colocated self relocation; get_var of its own name is proven once per chain by JS_JitNativeCallBegin instead of being evaluated. Every failure (Int32 overflow, -0, stack depth, interrupt request) discards pure work and deoptimizes the outermost CALL, so the interpreter re-executes it exactly. Self-recursive call sites and monomorphic callers of a published entry guard callee identity and argument tags, then call the entry directly. Stack depth is charged per level with the interpreter's measured frame cost, so native recursion fails before the interpreter would throw RangeError; a thread-local watermark prevents quadratic retries. Interrupt budgets mirror JS_CallInternal's per-call poll. Admitted functions are exempt from generic_call_without_loop when their observed recursion averages at least 16 calls per outside entry, and their collapsed-tree Tier 2 timings are not compared against per-call baseline samples. fibonacci-recursive (automatic) goes from interpreter speed to about 25x the interpreter in noisy diagnostics. Co-Authored-By: Claude Opus 5.5 (1M context) --- jit/src/code_cache/artifact.rs | 61 +- jit/src/compiler/baseline.rs | 68 ++- jit/src/compiler/mod.rs | 2 + jit/src/compiler/native_call.rs | 1017 +++++++++++++++++++++++++++++++ jit/src/compiler/optimized.rs | 438 +++++++++++++ jit/src/lib.rs | 167 ++++- jit/src/runtime/coordinator.rs | 79 +++ jit/src/runtime/mod.rs | 2 +- jit/tests/native_calls.rs | 465 ++++++++++++++ 9 files changed, 2278 insertions(+), 21 deletions(-) create mode 100644 jit/src/compiler/native_call.rs create mode 100644 jit/tests/native_calls.rs diff --git a/jit/src/code_cache/artifact.rs b/jit/src/code_cache/artifact.rs index 204607301..a4dcc8600 100644 --- a/jit/src/code_cache/artifact.rs +++ b/jit/src/code_cache/artifact.rs @@ -348,6 +348,8 @@ pub struct OptimizedArtifactMetadata { inlined_calls: u64, side_path_profile: Option, direct_call_signature: Option, + #[cfg(not(target_family = "wasm"))] + native_call_plan: Option, } #[cfg(feature = "compiler")] @@ -368,6 +370,8 @@ impl OptimizedArtifactMetadata { inlined_calls: 0, side_path_profile: None, direct_call_signature: None, + #[cfg(not(target_family = "wasm"))] + native_call_plan: None, } } /// Number of statically expanded call sites, not runtime call executions. @@ -412,6 +416,19 @@ impl OptimizedArtifactMetadata { ) -> Option<&crate::runtime::BoundedSpecializationSignature> { self.direct_call_signature.as_ref() } + #[cfg(not(target_family = "wasm"))] + pub(crate) fn with_native_call_plan( + mut self, + plan: crate::compiler::native_call::NativeCallPlan, + ) -> Self { + self.native_call_plan = Some(plan); + self + } + /// The pure self-recursive native entry published with this artifact. + #[cfg(not(target_family = "wasm"))] + pub const fn native_call_plan(&self) -> Option<&crate::compiler::native_call::NativeCallPlan> { + self.native_call_plan.as_ref() + } } impl ArtifactDependency { @@ -479,6 +496,10 @@ pub struct CompiledArtifact { #[cfg(all(feature = "compiler", not(target_family = "wasm")))] direct_call_published: Option, #[cfg(all(feature = "compiler", not(target_family = "wasm")))] + native_call_relocatable: Option>, + #[cfg(all(feature = "compiler", not(target_family = "wasm")))] + native_call_published: Option, + #[cfg(all(feature = "compiler", not(target_family = "wasm")))] direct_call_dependencies: Box<[crate::compiler::baseline::PublishedBaselineCode]>, #[cfg(all(feature = "compiler", not(target_family = "wasm")))] inline_snapshot: Option, @@ -526,6 +547,10 @@ impl CompiledArtifact { #[cfg(all(feature = "compiler", not(target_family = "wasm")))] direct_call_published: None, #[cfg(all(feature = "compiler", not(target_family = "wasm")))] + native_call_relocatable: None, + #[cfg(all(feature = "compiler", not(target_family = "wasm")))] + native_call_published: None, + #[cfg(all(feature = "compiler", not(target_family = "wasm")))] direct_call_dependencies: Box::new([]), #[cfg(all(feature = "compiler", not(target_family = "wasm")))] inline_snapshot: None, @@ -695,10 +720,33 @@ impl CompiledArtifact { self } + #[cfg(all(feature = "compiler", not(target_family = "wasm")))] + pub(crate) fn with_native_call_relocatable( + mut self, + code: crate::compiler::baseline::RelocatableCode, + ) -> Self { + self.native_call_relocatable = Some(Box::new(code)); + self + } + #[cfg(all(feature = "compiler", not(target_family = "wasm")))] pub(crate) fn publish_relocatable(&mut self) -> Result<(), crate::platform::CodeMemoryError> { + // The native entry publishes first: the main body embeds its address + // and retains its publication for the body's whole executable life. + if let Some(code) = self.native_call_relocatable.take() { + self.native_call_published = Some(code.publish()?); + } if let Some(code) = self.relocatable.take() { - self.published = Some(code.publish()?); + self.published = Some(match &self.native_call_published { + Some(native) => code.publish_with( + &[( + crate::compiler::native_call::NATIVE_ENTRY_SYMBOL, + native.as_ptr() as usize as u64, + )], + vec![native.clone()], + )?, + None => code.publish()?, + }); } if let Some(code) = self.direct_call_relocatable.take() { self.direct_call_published = Some(code.publish()?); @@ -711,6 +759,13 @@ impl CompiledArtifact { self.published.as_ref() } + #[cfg(all(feature = "compiler", not(target_family = "wasm")))] + pub fn native_call_published( + &self, + ) -> Option<&crate::compiler::baseline::PublishedBaselineCode> { + self.native_call_published.as_ref() + } + #[cfg(all(feature = "compiler", not(target_family = "wasm")))] pub fn direct_call_published( &self, @@ -761,6 +816,10 @@ impl CompiledArtifact { #[cfg(all(feature = "compiler", not(target_family = "wasm")))] direct_call_published: None, #[cfg(all(feature = "compiler", not(target_family = "wasm")))] + native_call_relocatable: None, + #[cfg(all(feature = "compiler", not(target_family = "wasm")))] + native_call_published: None, + #[cfg(all(feature = "compiler", not(target_family = "wasm")))] direct_call_dependencies: Box::new([]), #[cfg(all(feature = "compiler", not(target_family = "wasm")))] inline_snapshot: None, diff --git a/jit/src/compiler/baseline.rs b/jit/src/compiler/baseline.rs index 00e7cfb64..279679979 100644 --- a/jit/src/compiler/baseline.rs +++ b/jit/src/compiler/baseline.rs @@ -947,6 +947,29 @@ impl Compiler for BaselineCompiler { /// Finalizes Cranelift IR produced by the independent optimizing builder. This /// owns only target encoding/unwind packaging; it does not translate or lower /// baseline IR. +/// Native-call entries call themselves through a colocated user name, and a +/// Tier 2 body names the native entry published in its own artifact. Both are +/// resolved at publication; every other external name keeps its symbol text. +fn native_call_relocation_target( + name: &cranelift_codegen::ir::ExternalName, + parameters: &cranelift_codegen::ir::function::FunctionParameters, +) -> Option { + let cranelift_codegen::ir::ExternalName::User(reference) = name else { + return None; + }; + let user = parameters.user_named_funcs().get(*reference)?; + if user.index != 0 { + return None; + } + match user.namespace { + super::native_call::NATIVE_SELF_NAMESPACE => Some(RelocationTarget::FunctionOffset(0)), + super::native_call::NATIVE_ENTRY_NAMESPACE => Some(RelocationTarget::Symbol( + super::native_call::NATIVE_ENTRY_SYMBOL.into(), + )), + _ => None, + } +} + pub(crate) fn finalize_optimized_machine( isa: &OwnedTargetIsa, clif: Function, @@ -981,11 +1004,15 @@ pub(crate) fn finalize_optimized_machine( .map(|reloc| { let target = match &reloc.target { FinalizedRelocTarget::Func(offset) => RelocationTarget::FunctionOffset(*offset), - FinalizedRelocTarget::ExternalName(name) => RelocationTarget::Symbol( - name.display(Some(&function_parameters)) - .to_string() - .into_boxed_str(), - ), + FinalizedRelocTarget::ExternalName(name) => { + native_call_relocation_target(name, &function_parameters).unwrap_or_else(|| { + RelocationTarget::Symbol( + name.display(Some(&function_parameters)) + .to_string() + .into_boxed_str(), + ) + }) + } }; Relocation::with_target( reloc.offset, @@ -1148,6 +1175,17 @@ impl RelocatableCode { } pub fn publish(self) -> Result { + self.publish_with(&[], Vec::new()) + } + + /// Publishes with named relocation symbols resolved to absolute + /// addresses. `retained` publications stay executable at least as long + /// as this code, which may embed their addresses. + pub(crate) fn publish_with( + self, + symbols: &[(&str, u64)], + retained: Vec, + ) -> Result { if !self.host_publishable || self.target != Triple::host() { return Err(CodeMemoryError::TargetIsaMismatch); } @@ -1165,7 +1203,10 @@ impl RelocatableCode { base.checked_add(u64::from(*offset)) } RelocationTarget::Absolute(value) => Some(*value), - RelocationTarget::Symbol(_) => None, + RelocationTarget::Symbol(name) => symbols + .iter() + .find(|(symbol, _)| *symbol == &**name) + .map(|(_, address)| *address), }) .map_err(|_| CodeMemoryError::UnresolvedRelocationTarget)?; resolved.push(resolved_relocation); @@ -1177,16 +1218,22 @@ impl RelocatableCode { let osr_codes = self .osr_codes .into_iter() - .map(|(map, code)| Ok((map, code.publish()?))) + .map(|(map, code)| Ok((map, code.publish_with(symbols, retained.clone())?))) .collect::, CodeMemoryError>>()?; - Ok(PublishedBaselineCode::new( + let mut published = PublishedBaselineCode::new( executable, unwind_registration, self.unwind_metadata, self.stack_maps, self.frame_states, osr_codes, - )) + ); + if !retained.is_empty() { + Arc::get_mut(&mut published.allocation) + .expect("new publication is uniquely owned") + .retained = retained.into_boxed_slice(); + } + Ok(published) } } @@ -1204,6 +1251,8 @@ struct PublishedBaselineAllocation { stack_maps: Box<[StackMap]>, frame_states: Box<[ArtifactFrameState]>, osr_codes: Box<[(crate::runtime::OsrMap, PublishedBaselineCode)]>, + /// Publications whose absolute addresses this code embeds. + retained: Box<[PublishedBaselineCode]>, #[cfg(feature = "test-support")] lifetime_events: Arc>>, } @@ -1225,6 +1274,7 @@ impl PublishedBaselineCode { stack_maps: stack_maps.into_boxed_slice(), frame_states: frame_states.into_boxed_slice(), osr_codes: osr_codes.into_boxed_slice(), + retained: Box::new([]), #[cfg(feature = "test-support")] lifetime_events: Arc::new(Mutex::new(Vec::new())), }), diff --git a/jit/src/compiler/mod.rs b/jit/src/compiler/mod.rs index 0a339d41e..1e0f21bd0 100644 --- a/jit/src/compiler/mod.rs +++ b/jit/src/compiler/mod.rs @@ -57,6 +57,8 @@ mod call_cleanup; #[cfg(all(feature = "compiler", not(target_family = "wasm")))] mod helpers; #[cfg(all(feature = "compiler", not(target_family = "wasm")))] +pub mod native_call; +#[cfg(all(feature = "compiler", not(target_family = "wasm")))] pub mod optimized; #[cfg(all(feature = "compiler", not(target_family = "wasm")))] mod tagged_call_link; diff --git a/jit/src/compiler/native_call.rs b/jit/src/compiler/native_call.rs new file mode 100644 index 000000000..896e273d1 --- /dev/null +++ b/jit/src/compiler/native_call.rs @@ -0,0 +1,1017 @@ +//! P3a native-to-native call convention, first fail-closed slice. +//! +//! A *native call entry* is a secondary machine entry published beside a +//! Tier 2 artifact for a pure, self-recursive scalar function (see +//! `docs/roadmap/p3a-native-calls.md`). Its ABI is +//! +//! ```text +//! status = entry(nc: *mut NativeCallContext, out: *mut Scalar, vsp: usize, args: Scalar...) +//! ``` +//! +//! where each `Scalar` is `i32` (Int32) or `f64` (Float64) as fixed by the +//! function's bounded feedback signature. Status zero stores the result. Any +//! other status means the chain performed no observable effect and the +//! outermost compiled caller must re-execute its CALL in the interpreter. The body never calls a helper, +//! allocates, reads or writes the heap, or materializes a `JSStackFrame`; its +//! only calls are to itself. The self reference (`get_var `) is not +//! evaluated natively: [`qjsjit_native_call_begin`] proves once per chain that +//! the binding is a plain data property holding the callee in its realm. + +use crate::bytecode::VerifiedFunction; +use crate::runtime::{BoundedSpecializationSignature, FeedbackRepresentation}; +use cranelift_codegen::ir::condcodes::{FloatCC, IntCC}; +use cranelift_codegen::ir::{ + types, AbiParam, Block, ExtFuncData, ExternalName, Function, InstBuilder, MemFlags, Signature, + StackSlotData, StackSlotKind, UserExternalName, Value, +}; +use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext}; +use std::cell::Cell; +use std::collections::BTreeMap; + +use super::{CompileControl, CompileFailure}; + +/// User external-name namespace of a native entry's call to itself. Final +/// relocation resolves it to the entry's own base address. +pub(crate) const NATIVE_SELF_NAMESPACE: u32 = 0x5033_a001; +/// User external-name namespace of a Tier 2 body's reference to the native +/// entry published in the same artifact. +pub(crate) const NATIVE_ENTRY_NAMESPACE: u32 = 0x5033_a002; +/// Relocation symbol for [`NATIVE_ENTRY_NAMESPACE`]. +pub(crate) const NATIVE_ENTRY_SYMBOL: &str = "qjsjit.native_call_entry"; + +/// Status returned by a native entry after a successful return. +pub(crate) const NATIVE_CALL_OK: i64 = 0; +/// Status requesting interpreter re-execution of the outermost CALL. +pub(crate) const NATIVE_CALL_RETRY: i64 = 1; + +const FLAG_STACK_EXHAUSTED: u32 = 1 << 0; + +/// Per-level charge used when the interpreter frame cannot be measured. It +/// is far above any measured build (about 2 KiB for x86_64 release), so the +/// fallback only shortens native recursion. +const INTERPRETER_FRAME_FALLBACK: u32 = 32 * 1024; +/// Lower bound on the per-level charge; it also covers the native entry's +/// own frame, which is much smaller. +const INTERPRETER_FRAME_MINIMUM: u32 = 1024; +/// Upper bound on generated self-recursive bodies. +const MAX_NATIVE_CALL_INSTRUCTIONS: usize = 256; + +/// Layout-checked mirror of `JSJitNativeCallContext` (patch 0029). +#[repr(C)] +#[derive(Debug)] +pub(crate) struct NativeCallContext { + stack_limit: usize, + budget: i32, + flags: u32, + ctx: *mut rquickjs_core::qjs::JSContext, +} + +const OFFSET_STACK_LIMIT: i32 = 0; +const OFFSET_BUDGET: i32 = 8; +const OFFSET_FLAGS: i32 = 12; +const CONTEXT_SIZE: u32 = 24; + +unsafe extern "C" { + fn JS_JitNativeCallBegin( + ctx: *mut rquickjs_core::qjs::JSContext, + function: *mut core::ffi::c_void, + atom: u32, + nc: *mut NativeCallContext, + ) -> core::ffi::c_int; + fn JS_JitNativeCallPoll(nc: *mut NativeCallContext) -> core::ffi::c_int; + fn JS_JitNativeCallEnd(nc: *mut NativeCallContext); + fn JS_JitNativeCallContextLayout(offsets: *mut usize, count: usize) -> usize; +} + +static INTERPRETER_FRAME_BYTES: std::sync::OnceLock = std::sync::OnceLock::new(); + +/// C stack bytes one interpreter call level of a minimal function uses in +/// this build (`JS_CallInternal`'s frame plus its value allocation), with a +/// 25% margin. Native entries charge it per level on top of their own value +/// slots, so a native chain fails before the depth at which the interpreter +/// would throw `RangeError`, whatever the C compiler and flags produced. +pub fn interpreter_frame_bytes() -> u32 { + *INTERPRETER_FRAME_BYTES.get_or_init(|| { + measure_interpreter_frame() + .map_or(INTERPRETER_FRAME_FALLBACK, |bytes| { + bytes.saturating_add(bytes / 4) + }) + .max(INTERPRETER_FRAME_MINIMUM) + }) +} + +/// Measures the distance between the C stack pointers seen by a host probe +/// called at recursion depths 0 and 64 of an interpreted JS function, in a +/// private runtime without a JIT. +fn measure_interpreter_frame() -> Option { + use rquickjs_core::{Context, Function, Runtime}; + const LEVELS: f64 = 64.0; + let runtime = Runtime::new().ok()?; + let context = Context::full(&runtime).ok()?; + context.with(|ctx| { + let probe = Function::new(ctx.clone(), || -> f64 { + let marker = 0u8; + core::hint::black_box(&marker) as *const u8 as usize as f64 + }) + .ok()?; + ctx.globals().set("__qjsjit_stack_probe", probe).ok()?; + ctx.eval::<(), _>( + "function __qjsjit_depth(n){return n<=0?__qjsjit_stack_probe():__qjsjit_depth(n-1);}", + ) + .ok()?; + let shallow: f64 = ctx.eval("__qjsjit_depth(0)").ok()?; + let deep: f64 = ctx.eval("__qjsjit_depth(64)").ok()?; + let bytes = (shallow - deep) / LEVELS; + (bytes.is_finite() && bytes > 0.0 && bytes < f64::from(INTERPRETER_FRAME_FALLBACK)) + .then(|| bytes.ceil() as u32) + }) +} + +thread_local! { + /// Stack address of the outermost caller whose chain ran out of stack. + /// While the thread executes at or below it, native chains are not + /// attempted, so exhausted recursion continues on real interpreter + /// frames instead of retrying a native chain at every level. + static EXHAUSTED_FLOOR: Cell = const { Cell::new(0) }; +} + +/// True when the C context layout matches the offsets generated code uses. +pub(crate) fn context_layout_matches() -> bool { + let mut offsets = [0usize; 4]; + let size = unsafe { JS_JitNativeCallContextLayout(offsets.as_mut_ptr(), offsets.len()) }; + size == CONTEXT_SIZE as usize + && size == core::mem::size_of::() + && offsets + == [ + OFFSET_STACK_LIMIT as usize, + OFFSET_BUDGET as usize, + OFFSET_FLAGS as usize, + 16, + ] +} + +/// Admits one outermost native chain. `sp` is the caller's stack pointer. +/// Returns zero when the chain may run. +/// +/// # Safety +/// `ctx` must be the live context of the calling frame, `function` a live +/// object guarded by the caller, and `nc` writable for the whole chain. +pub(crate) unsafe extern "C" fn qjsjit_native_call_begin( + ctx: *mut rquickjs_core::qjs::JSContext, + function: *mut core::ffi::c_void, + atom: u32, + nc: *mut NativeCallContext, + sp: usize, +) -> i32 { + let floor = EXHAUSTED_FLOOR.with(Cell::get); + if floor != 0 { + if sp <= floor { + return 1; + } + EXHAUSTED_FLOOR.with(|cell| cell.set(0)); + } + if unsafe { JS_JitNativeCallBegin(ctx, function, atom, nc) } != 0 { + 1 + } else { + 0 + } +} + +/// Publishes interrupt accounting and records stack exhaustion. +/// +/// # Safety +/// `nc` must have been admitted by [`qjsjit_native_call_begin`]. +pub(crate) unsafe extern "C" fn qjsjit_native_call_end(nc: *mut NativeCallContext, sp: usize) { + unsafe { JS_JitNativeCallEnd(nc) }; + if unsafe { (*nc).flags } & FLAG_STACK_EXHAUSTED != 0 { + EXHAUSTED_FLOOR.with(|cell| cell.set(sp)); + } +} + +/// Validated description of a function's native entry. +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct NativeCallPlan { + atom: u32, + arguments: Box<[FeedbackRepresentation]>, + result: FeedbackRepresentation, + frame_charge: u32, +} + +impl NativeCallPlan { + pub const fn atom(&self) -> u32 { + self.atom + } + pub fn arity(&self) -> usize { + self.arguments.len() + } + /// Unboxed representation of each argument (Int32 or Float64). + pub fn arguments(&self) -> &[FeedbackRepresentation] { + &self.arguments + } + /// Unboxed representation of the result (Int32 or Float64). + pub const fn result(&self) -> FeedbackRepresentation { + self.result + } + pub const fn frame_charge(&self) -> u32 { + self.frame_charge + } +} + +/// Returns the native-entry plan when the function is a pure, self-recursive +/// Int32/Float64 function under `signature`. This builds (and discards) the +/// same CLIF the lowering publishes, so admission and lowering cannot disagree. +pub fn plan( + function: &VerifiedFunction, + signature: &BoundedSpecializationSignature, +) -> Option { + if !context_layout_matches() { + return None; + } + // Admission only builds the CLIF; the host ABI does not affect which + // bytecode shapes the lowering accepts. + build( + types::I64, + cranelift_codegen::isa::CallConv::SystemV, + function, + signature, + ) + .ok() + .map(|(plan, _)| plan) +} + +/// Lowers the native entry. Fails closed for every unsupported shape. +pub(crate) fn lower( + isa: &cranelift_codegen::isa::OwnedTargetIsa, + function: &VerifiedFunction, + signature: &BoundedSpecializationSignature, + control: Option<&CompileControl>, +) -> Result<(NativeCallPlan, super::baseline::RelocatableCode), CompileFailure> { + if !context_layout_matches() { + return Err(CompileFailure::InvalidArtifact); + } + let (plan, clif) = build( + isa.pointer_type(), + isa.default_call_conv(), + function, + signature, + )?; + let code = super::baseline::finalize_optimized_machine(isa, clif, control, false)?; + Ok((plan, code)) +} + +/// Machine type of an unboxed native-call scalar. +pub(crate) fn scalar_type(representation: FeedbackRepresentation) -> Option { + match representation { + FeedbackRepresentation::Int32 => Some(types::I32), + FeedbackRepresentation::Float64 => Some(types::F64), + FeedbackRepresentation::Bool | FeedbackRepresentation::HeapRef => None, + } +} + +/// The native entry signature: `(nc, out, vsp, scalar args...) -> i32`. +pub(crate) fn entry_signature( + pointer_type: types::Type, + call_conv: cranelift_codegen::isa::CallConv, + arguments: &[FeedbackRepresentation], +) -> Result { + let mut abi = Signature::new(call_conv); + abi.params.push(AbiParam::new(pointer_type)); + abi.params.push(AbiParam::new(pointer_type)); + abi.params.push(AbiParam::new(pointer_type)); + for argument in arguments { + abi.params.push(AbiParam::new( + scalar_type(*argument).ok_or(CompileFailure::UnsupportedOpcode)?, + )); + } + abi.returns.push(AbiParam::new(types::I32)); + Ok(abi) +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +enum Kind { + Int32, + Float64, + Bool, + SelfFunction, +} + +#[derive(Clone, Copy)] +enum Slot { + Int32(Value), + Float64(Value), + Bool(Value), + SelfFunction, +} + +impl Slot { + const fn kind(self) -> Kind { + match self { + Self::Int32(_) => Kind::Int32, + Self::Float64(_) => Kind::Float64, + Self::Bool(_) => Kind::Bool, + Self::SelfFunction => Kind::SelfFunction, + } + } + + const fn value(self) -> Option { + match self { + Self::Int32(value) | Self::Float64(value) | Self::Bool(value) => Some(value), + Self::SelfFunction => None, + } + } + + fn from_representation( + representation: FeedbackRepresentation, + value: Value, + ) -> Result { + match representation { + FeedbackRepresentation::Int32 => Ok(Self::Int32(value)), + FeedbackRepresentation::Float64 => Ok(Self::Float64(value)), + FeedbackRepresentation::Bool | FeedbackRepresentation::HeapRef => { + Err(CompileFailure::UnsupportedOpcode) + } + } + } +} + +/// A JavaScript number as a machine value of the requested representation. +/// Int32 widens exactly to Float64; Float64 never narrows. +fn number_as( + builder: &mut FunctionBuilder<'_>, + slot: Slot, + representation: FeedbackRepresentation, +) -> Result { + match (slot, representation) { + (Slot::Int32(value), FeedbackRepresentation::Int32) + | (Slot::Float64(value), FeedbackRepresentation::Float64) => Ok(value), + (Slot::Int32(value), FeedbackRepresentation::Float64) => { + Ok(builder.ins().fcvt_from_sint(types::F64, value)) + } + _ => Err(CompileFailure::UnsupportedOpcode), + } +} + +/// Both operands as Float64 when either is Float64; `None` when both Int32. +fn float_operands( + builder: &mut FunctionBuilder<'_>, + lhs: Slot, + rhs: Slot, +) -> Result, CompileFailure> { + match (lhs, rhs) { + (Slot::Int32(_), Slot::Int32(_)) => Ok(None), + (Slot::Int32(_) | Slot::Float64(_), Slot::Int32(_) | Slot::Float64(_)) => Ok(Some(( + number_as(builder, lhs, FeedbackRepresentation::Float64)?, + number_as(builder, rhs, FeedbackRepresentation::Float64)?, + ))), + _ => Err(CompileFailure::UnsupportedOpcode), + } +} + +struct BlockState { + block: Block, + kinds: Option>, +} + +fn int_constant(name: &str, bytes: &[u8]) -> Result, CompileFailure> { + Ok(match name { + "push_minus1" => Some(-1), + "push_0" | "push_1" | "push_2" | "push_3" | "push_4" | "push_5" | "push_6" | "push_7" => { + Some(i64::from(name.as_bytes()[5] - b'0')) + } + "push_i8" => Some(i64::from( + *bytes.get(1).ok_or(CompileFailure::InvalidArtifact)? as i8, + )), + "push_i16" => Some(i64::from(i16::from_le_bytes( + bytes + .get(1..3) + .ok_or(CompileFailure::InvalidArtifact)? + .try_into() + .map_err(|_| CompileFailure::InvalidArtifact)?, + ))), + "push_i32" => Some(i64::from(i32::from_le_bytes( + bytes + .get(1..5) + .ok_or(CompileFailure::InvalidArtifact)? + .try_into() + .map_err(|_| CompileFailure::InvalidArtifact)?, + ))), + _ => None, + }) +} + +fn argument_index(name: &str, bytes: &[u8]) -> Result, CompileFailure> { + Ok(match name { + "get_arg0" => Some(0), + "get_arg1" => Some(1), + "get_arg2" => Some(2), + "get_arg3" => Some(3), + "get_arg" => Some(usize::from(u16::from_le_bytes( + bytes + .get(1..3) + .ok_or(CompileFailure::InvalidArtifact)? + .try_into() + .map_err(|_| CompileFailure::InvalidArtifact)?, + ))), + _ => None, + }) +} + +/// Explicit argument count of a call opcode, or `None` for other opcodes. +fn call_argc(name: &str, bytes: &[u8]) -> Result, CompileFailure> { + let u16_operand = || -> Result { + Ok(usize::from(u16::from_le_bytes( + bytes + .get(1..3) + .ok_or(CompileFailure::InvalidArtifact)? + .try_into() + .map_err(|_| CompileFailure::InvalidArtifact)?, + ))) + }; + Ok(match name { + "call0" => Some((0, false)), + "call1" => Some((1, false)), + "call2" => Some((2, false)), + "call3" => Some((3, false)), + "call" => Some((u16_operand()?, false)), + "tail_call" => Some((u16_operand()?, true)), + _ => None, + }) +} + +fn emit_retry(builder: &mut FunctionBuilder<'_>) { + let status = builder.ins().iconst(types::I32, NATIVE_CALL_RETRY); + builder.ins().return_(&[status]); +} + +/// Continues in a fresh block when `failed` is non-zero; otherwise retries. +fn emit_fail_if(builder: &mut FunctionBuilder<'_>, failed: Value) { + let fail = builder.create_block(); + let ok = builder.create_block(); + builder.ins().brif(failed, fail, &[], ok, &[]); + builder.switch_to_block(fail); + emit_retry(builder); + builder.switch_to_block(ok); +} + +fn jump_args(stack: &[Slot]) -> Vec { + stack.iter().filter_map(|slot| slot.value()).collect() +} + +/// Records the operand-stack kinds of a forward edge's target, creating its +/// block parameters on first use and rejecting inconsistent joins. +fn bind_edge( + builder: &mut FunctionBuilder<'_>, + blocks: &mut BTreeMap, + current_pc: u32, + target_pc: u32, + stack: &[Slot], +) -> Result<(Block, Vec), CompileFailure> { + // Only forward edges: every predecessor is lowered before its target, + // so each target's stack kinds are known when it is reached. + if target_pc <= current_pc { + return Err(CompileFailure::UnsupportedOpcode); + } + let state = blocks + .get_mut(&target_pc) + .ok_or(CompileFailure::InvalidArtifact)?; + let kinds = stack.iter().map(|slot| slot.kind()).collect::>(); + match &state.kinds { + Some(existing) if *existing != kinds => return Err(CompileFailure::UnsupportedOpcode), + Some(_) => {} + None => { + for kind in &kinds { + match kind { + Kind::Int32 => { + builder.append_block_param(state.block, types::I32); + } + Kind::Float64 => { + builder.append_block_param(state.block, types::F64); + } + Kind::Bool => { + builder.append_block_param(state.block, types::I8); + } + Kind::SelfFunction => {} + } + } + state.kinds = Some(kinds); + } + } + Ok((state.block, jump_args(stack))) +} + +fn build( + pointer_type: types::Type, + call_conv: cranelift_codegen::isa::CallConv, + function: &VerifiedFunction, + signature: &BoundedSpecializationSignature, +) -> Result<(NativeCallPlan, Function), CompileFailure> { + let snapshot = function.snapshot(); + if signature.function().id != snapshot.function_id() + || signature.function().generation != snapshot.generation() + || signature.arity() != usize::from(snapshot.arg_count()) + || scalar_type(signature.result()).is_none() + || signature + .arguments() + .iter() + .any(|argument| scalar_type(*argument).is_none()) + || !snapshot.exception_map().is_empty() + || snapshot.closure_count() != 0 + || function.instructions().len() > MAX_NATIVE_CALL_INSTRUCTIONS + { + return Err(CompileFailure::UnsupportedOpcode); + } + let arity = signature.arity(); + let parameters = signature.arguments(); + let result_representation = signature.result(); + let result_type = + scalar_type(result_representation).ok_or(CompileFailure::UnsupportedOpcode)?; + let value_slots = u32::from(snapshot.arg_count()) + .saturating_add(u32::from(snapshot.local_count())) + .saturating_add(u32::from(snapshot.stack_size())) + .saturating_add(4); + let frame_charge = value_slots + .saturating_mul(16) + .saturating_add(interpreter_frame_bytes()); + + let mut clif = Function::with_name_signature( + Default::default(), + entry_signature(pointer_type, call_conv, parameters)?, + ); + let mut context = FunctionBuilderContext::new(); + let mut atom = None; + let mut self_calls = 0usize; + { + let mut builder = FunctionBuilder::new(&mut clif, &mut context); + let self_signature = + builder.import_signature(entry_signature(pointer_type, call_conv, parameters)?); + let self_name = builder + .func + .declare_imported_user_function(UserExternalName::new(NATIVE_SELF_NAMESPACE, 0)); + let self_ref = builder.import_function(ExtFuncData { + name: ExternalName::user(self_name), + signature: self_signature, + colocated: true, + }); + let mut poll_signature = Signature::new(call_conv); + poll_signature.params.push(AbiParam::new(pointer_type)); + poll_signature.returns.push(AbiParam::new(types::I32)); + let poll_signature = builder.import_signature(poll_signature); + + let cfg = function.control_flow_graph(); + let mut blocks = cfg + .blocks() + .iter() + .map(|block| { + ( + block.start_pc(), + BlockState { + block: builder.create_block(), + kinds: None, + }, + ) + }) + .collect::>(); + let prologue = builder.create_block(); + builder.append_block_params_for_function_params(prologue); + builder.switch_to_block(prologue); + let params = builder.block_params(prologue).to_vec(); + let nc = params[0]; + let out = params[1]; + let incoming_vsp = params[2]; + let arguments = params[3..].to_vec(); + + // Interpreter-equivalent stack check: charge a conservative + // interpreter frame per level against the caller's stack limit. + let limit = builder + .ins() + .load(pointer_type, MemFlags::trusted(), nc, OFFSET_STACK_LIMIT); + let charged_limit = builder.ins().iadd_imm(limit, i64::from(frame_charge)); + let exhausted = builder + .ins() + .icmp(IntCC::UnsignedLessThan, incoming_vsp, charged_limit); + let stack_fail = builder.create_block(); + let stack_ok = builder.create_block(); + builder + .ins() + .brif(exhausted, stack_fail, &[], stack_ok, &[]); + builder.switch_to_block(stack_fail); + let flags = builder + .ins() + .load(types::I32, MemFlags::trusted(), nc, OFFSET_FLAGS); + let flags = builder + .ins() + .bor_imm(flags, i64::from(FLAG_STACK_EXHAUSTED)); + builder + .ins() + .store(MemFlags::trusted(), flags, nc, OFFSET_FLAGS); + emit_retry(&mut builder); + builder.switch_to_block(stack_ok); + let vsp = builder + .ins() + .iadd_imm(incoming_vsp, -i64::from(frame_charge)); + + // Interrupt accounting mirrors JS_CallInternal's per-call poll. + let budget = builder + .ins() + .load(types::I32, MemFlags::trusted(), nc, OFFSET_BUDGET); + let budget = builder.ins().iadd_imm(budget, -1); + builder + .ins() + .store(MemFlags::trusted(), budget, nc, OFFSET_BUDGET); + let due = builder + .ins() + .icmp_imm(IntCC::SignedLessThanOrEqual, budget, 0); + let poll = builder.create_block(); + let body = blocks.get(&0).ok_or(CompileFailure::InvalidArtifact)?.block; + builder.ins().brif(due, poll, &[], body, &[]); + builder.switch_to_block(poll); + let poll_target = builder.ins().iconst( + pointer_type, + JS_JitNativeCallPoll as *const () as usize as i64, + ); + let call = super::emit_external_call( + &mut builder, + poll_signature, + poll_target, + &[nc], + pointer_type, + None, + None, + ); + let interrupted = builder.inst_results(call)[0]; + let interrupted = builder.ins().icmp_imm(IntCC::NotEqual, interrupted, 0); + emit_fail_if(&mut builder, interrupted); + builder.ins().jump(body, &[]); + blocks + .get_mut(&0) + .ok_or(CompileFailure::InvalidArtifact)? + .kinds = Some(Vec::new()); + + for block in cfg.blocks() { + let state = &blocks[&block.start_pc()]; + let Some(kinds) = state.kinds.clone() else { + // Unreachable from the entry through forward edges. + if block.start_pc() == 0 { + return Err(CompileFailure::InvalidArtifact); + } + continue; + }; + let machine_block = state.block; + builder.switch_to_block(machine_block); + let block_params = builder.block_params(machine_block).to_vec(); + let mut next_param = block_params.into_iter(); + let mut stack = Vec::with_capacity(kinds.len()); + for kind in kinds { + stack.push(match kind { + Kind::Int32 => { + Slot::Int32(next_param.next().ok_or(CompileFailure::InvalidArtifact)?) + } + Kind::Float64 => { + Slot::Float64(next_param.next().ok_or(CompileFailure::InvalidArtifact)?) + } + Kind::Bool => { + Slot::Bool(next_param.next().ok_or(CompileFailure::InvalidArtifact)?) + } + Kind::SelfFunction => Slot::SelfFunction, + }); + } + let mut terminated = false; + for instruction in &function.instructions()[block.instruction_range()] { + if terminated { + return Err(CompileFailure::InvalidArtifact); + } + let name = instruction.opcode().name(); + let bytes = instruction.bytes(); + if let Some(value) = int_constant(name, bytes)? { + stack.push(Slot::Int32(builder.ins().iconst(types::I32, value))); + continue; + } + if let Some(index) = argument_index(name, bytes)? { + let value = *arguments + .get(index) + .ok_or(CompileFailure::UnsupportedOpcode)?; + stack.push(Slot::from_representation(parameters[index], value)?); + continue; + } + if let Some((argc, tail)) = call_argc(name, bytes)? { + if argc != arity { + return Err(CompileFailure::UnsupportedOpcode); + } + let base = stack + .len() + .checked_sub(argc + 1) + .ok_or(CompileFailure::InvalidArtifact)?; + if !matches!(stack[base], Slot::SelfFunction) { + return Err(CompileFailure::UnsupportedOpcode); + } + let mut call_arguments = Vec::with_capacity(argc + 3); + let result_slot = builder.create_sized_stack_slot(StackSlotData::new( + StackSlotKind::ExplicitSlot, + 8, + 3, + )); + let result_address = builder.ins().stack_addr(pointer_type, result_slot, 0); + call_arguments.push(nc); + call_arguments.push(result_address); + call_arguments.push(vsp); + let passed = stack.split_off(base + 1); + for (slot, parameter) in passed.into_iter().zip(parameters) { + let value = number_as(&mut builder, slot, *parameter)?; + call_arguments.push(value); + } + stack.truncate(base); + let call = builder.ins().call(self_ref, &call_arguments); + let status = builder.inst_results(call)[0]; + let failed = builder + .ins() + .icmp_imm(IntCC::NotEqual, status, NATIVE_CALL_OK); + let propagate = builder.create_block(); + let resume = builder.create_block(); + builder.ins().brif(failed, propagate, &[], resume, &[]); + builder.switch_to_block(propagate); + builder.ins().return_(&[status]); + builder.switch_to_block(resume); + let result = + builder + .ins() + .load(result_type, MemFlags::trusted(), result_address, 0); + self_calls += 1; + if tail { + if !stack.is_empty() { + return Err(CompileFailure::UnsupportedOpcode); + } + builder.ins().store(MemFlags::trusted(), result, out, 0); + let ok = builder.ins().iconst(types::I32, NATIVE_CALL_OK); + builder.ins().return_(&[ok]); + terminated = true; + } else { + stack.push(Slot::from_representation(result_representation, result)?); + } + continue; + } + match name { + "nop" => {} + "push_const" | "push_const8" => { + let index = if name == "push_const8" { + u32::from(*bytes.get(1).ok_or(CompileFailure::InvalidArtifact)?) + } else { + u32::from_le_bytes( + bytes + .get(1..5) + .ok_or(CompileFailure::InvalidArtifact)? + .try_into() + .map_err(|_| CompileFailure::InvalidArtifact)?, + ) + }; + let constant = snapshot + .constants() + .iter() + .find(|constant| constant.index() == index) + .ok_or(CompileFailure::UnsupportedOpcode)?; + stack.push(match constant.tag() { + rquickjs_core::qjs::JS_TAG_INT => Slot::Int32( + builder + .ins() + .iconst(types::I32, i64::from(constant.payload() as i32)), + ), + rquickjs_core::qjs::JS_TAG_FLOAT64 => Slot::Float64( + builder.ins().f64const(f64::from_bits(constant.payload())), + ), + _ => return Err(CompileFailure::UnsupportedOpcode), + }); + } + "get_var" => { + let operand = u32::from_le_bytes( + bytes + .get(1..5) + .ok_or(CompileFailure::InvalidArtifact)? + .try_into() + .map_err(|_| CompileFailure::InvalidArtifact)?, + ); + if *atom.get_or_insert(operand) != operand { + return Err(CompileFailure::UnsupportedOpcode); + } + stack.push(Slot::SelfFunction); + } + "drop" => match stack.pop() { + Some(Slot::Int32(_) | Slot::Float64(_) | Slot::Bool(_)) => {} + _ => return Err(CompileFailure::UnsupportedOpcode), + }, + "add" | "sub" | "mul" | "div" => { + let (Some(rhs), Some(lhs)) = (stack.pop(), stack.pop()) else { + return Err(CompileFailure::InvalidArtifact); + }; + // Every Float64 operation is exact IEEE-754 double + // arithmetic, as in the interpreter; division always + // produces a JavaScript number, so it uses doubles. + let floats = if name == "div" { + Some(( + number_as(&mut builder, lhs, FeedbackRepresentation::Float64)?, + number_as(&mut builder, rhs, FeedbackRepresentation::Float64)?, + )) + } else { + float_operands(&mut builder, lhs, rhs)? + }; + if let Some((lhs, rhs)) = floats { + let value = match name { + "add" => builder.ins().fadd(lhs, rhs), + "sub" => builder.ins().fsub(lhs, rhs), + "mul" => builder.ins().fmul(lhs, rhs), + _ => builder.ins().fdiv(lhs, rhs), + }; + stack.push(Slot::Float64(value)); + continue; + } + let (Slot::Int32(lhs), Slot::Int32(rhs)) = (lhs, rhs) else { + return Err(CompileFailure::UnsupportedOpcode); + }; + let (result, overflow) = match name { + "add" => builder.ins().sadd_overflow(lhs, rhs), + "sub" => builder.ins().ssub_overflow(lhs, rhs), + _ => builder.ins().smul_overflow(lhs, rhs), + }; + emit_fail_if(&mut builder, overflow); + if name == "mul" { + // `0 * -n` is -0, which Int32 cannot represent. + let zero = builder.ins().icmp_imm(IntCC::Equal, result, 0); + let signs = builder.ins().bor(lhs, rhs); + let negative = builder.ins().icmp_imm(IntCC::SignedLessThan, signs, 0); + let negative_zero = builder.ins().band(zero, negative); + emit_fail_if(&mut builder, negative_zero); + } + stack.push(Slot::Int32(result)); + } + "inc" | "dec" => { + let value = match stack.pop() { + Some(Slot::Int32(value)) => value, + Some(Slot::Float64(value)) => { + let one = builder.ins().f64const(1.0); + let result = if name == "inc" { + builder.ins().fadd(value, one) + } else { + builder.ins().fsub(value, one) + }; + stack.push(Slot::Float64(result)); + continue; + } + _ => return Err(CompileFailure::UnsupportedOpcode), + }; + let one = builder.ins().iconst(types::I32, 1); + let (result, overflow) = if name == "inc" { + builder.ins().sadd_overflow(value, one) + } else { + builder.ins().ssub_overflow(value, one) + }; + emit_fail_if(&mut builder, overflow); + stack.push(Slot::Int32(result)); + } + "lt" | "lte" | "gt" | "gte" | "eq" | "neq" | "strict_eq" | "strict_neq" => { + let (Some(rhs), Some(lhs)) = (stack.pop(), stack.pop()) else { + return Err(CompileFailure::InvalidArtifact); + }; + if let Some((lhs, rhs)) = float_operands(&mut builder, lhs, rhs)? { + // Ordered comparisons are false for NaN; `!=` + // is unordered-or-not-equal, true for NaN. + let condition = match name { + "lt" => FloatCC::LessThan, + "lte" => FloatCC::LessThanOrEqual, + "gt" => FloatCC::GreaterThan, + "gte" => FloatCC::GreaterThanOrEqual, + "eq" | "strict_eq" => FloatCC::Equal, + _ => FloatCC::NotEqual, + }; + stack.push(Slot::Bool(builder.ins().fcmp(condition, lhs, rhs))); + continue; + } + let (Slot::Int32(lhs), Slot::Int32(rhs)) = (lhs, rhs) else { + return Err(CompileFailure::UnsupportedOpcode); + }; + let condition = match name { + "lt" => IntCC::SignedLessThan, + "lte" => IntCC::SignedLessThanOrEqual, + "gt" => IntCC::SignedGreaterThan, + "gte" => IntCC::SignedGreaterThanOrEqual, + "eq" | "strict_eq" => IntCC::Equal, + _ => IntCC::NotEqual, + }; + stack.push(Slot::Bool(builder.ins().icmp(condition, lhs, rhs))); + } + "lnot" => { + let truth = match stack.pop() { + Some(Slot::Bool(value)) => { + builder.ins().icmp_imm(IntCC::NotEqual, value, 0) + } + Some(Slot::Int32(value)) => { + builder.ins().icmp_imm(IntCC::NotEqual, value, 0) + } + _ => return Err(CompileFailure::UnsupportedOpcode), + }; + let negated = builder.ins().bxor_imm(truth, 1); + stack.push(Slot::Bool(negated)); + } + "if_false" | "if_true" | "if_false8" | "if_true8" => { + let condition = match stack.pop() { + Some(Slot::Bool(value) | Slot::Int32(value)) => value, + _ => return Err(CompileFailure::UnsupportedOpcode), + }; + let target_pc = u32::try_from( + instruction + .branch_target() + .ok_or(CompileFailure::InvalidArtifact)?, + ) + .map_err(|_| CompileFailure::InvalidArtifact)?; + let (target, target_args) = bind_edge( + &mut builder, + &mut blocks, + block.start_pc(), + target_pc, + &stack, + )?; + let (fallthrough, fallthrough_args) = bind_edge( + &mut builder, + &mut blocks, + block.start_pc(), + block.end_pc(), + &stack, + )?; + if name.starts_with("if_false") { + builder.ins().brif( + condition, + fallthrough, + &fallthrough_args, + target, + &target_args, + ); + } else { + builder.ins().brif( + condition, + target, + &target_args, + fallthrough, + &fallthrough_args, + ); + } + terminated = true; + } + "goto" | "goto8" | "goto16" => { + let target_pc = u32::try_from( + instruction + .branch_target() + .ok_or(CompileFailure::InvalidArtifact)?, + ) + .map_err(|_| CompileFailure::InvalidArtifact)?; + let (target, target_args) = bind_edge( + &mut builder, + &mut blocks, + block.start_pc(), + target_pc, + &stack, + )?; + builder.ins().jump(target, &target_args); + terminated = true; + } + "return" => { + let result = stack.pop().ok_or(CompileFailure::InvalidArtifact)?; + let result = number_as(&mut builder, result, result_representation)?; + if !stack.is_empty() { + return Err(CompileFailure::UnsupportedOpcode); + } + builder.ins().store(MemFlags::trusted(), result, out, 0); + let ok = builder.ins().iconst(types::I32, NATIVE_CALL_OK); + builder.ins().return_(&[ok]); + terminated = true; + } + _ => return Err(CompileFailure::UnsupportedOpcode), + } + } + if !terminated { + if block.successors().len() != 1 || block.successors()[0] != block.end_pc() { + return Err(CompileFailure::UnsupportedOpcode); + } + let (target, target_args) = bind_edge( + &mut builder, + &mut blocks, + block.start_pc(), + block.end_pc(), + &stack, + )?; + builder.ins().jump(target, &target_args); + } + } + builder.seal_all_blocks(); + builder.finalize(); + } + let atom = atom.ok_or(CompileFailure::UnsupportedOpcode)?; + if self_calls == 0 { + return Err(CompileFailure::UnsupportedOpcode); + } + Ok(( + NativeCallPlan { + atom, + arguments: parameters.into(), + result: result_representation, + frame_charge, + }, + clif, + )) +} diff --git a/jit/src/compiler/optimized.rs b/jit/src/compiler/optimized.rs index afc40c273..3f467f8ed 100644 --- a/jit/src/compiler/optimized.rs +++ b/jit/src/compiler/optimized.rs @@ -778,6 +778,7 @@ struct NumericSpecialization { properties: std::collections::BTreeMap>, arrays: Box<[crate::runtime::ArrayFeedbackSnapshot]>, direct_calls: std::collections::BTreeMap, + native_calls: std::collections::BTreeMap, inline_callees: std::collections::BTreeMap, frame_inline_callees: std::collections::BTreeMap, numeric_constants: std::collections::BTreeMap, @@ -789,6 +790,19 @@ struct DirectCallSite { entry: usize, } +/// A monomorphic call site lowered through the P3a native-call convention. +#[derive(Clone)] +struct NativeCallSite { + /// Global name whose binding `JS_JitNativeCallBegin` proves. + atom: u32, + arguments: Box<[crate::runtime::FeedbackRepresentation]>, + result: crate::runtime::FeedbackRepresentation, + callee_identity: u64, + callee_bytecode_identity: u64, + /// `None` names the native entry published in this same artifact. + entry: Option, +} + impl NumericSpecialization { fn retain_frame_inline_target( target: &crate::runtime::FrameInlineTarget, @@ -817,6 +831,11 @@ impl NumericSpecialization { return; } for target in request.direct_call_targets() { + // A published native entry runs the callee's whole recursion + // natively; inlining one level would call it generically. + if request.native_call_target(target.pc()).is_some() { + continue; + } if let Some(body) = target.inline_snapshot().and_then(|snapshot| { snapshot .verify(crate::bytecode::VerifyLimits::default()) @@ -833,6 +852,9 @@ impl NumericSpecialization { } } for target in request.frame_inline_targets() { + if request.native_call_target(target.pc()).is_some() { + continue; + } if let Some(callee) = Self::retain_frame_inline_target(target) { self.frame_inline_callees.insert(target.pc(), callee); } @@ -1132,6 +1154,7 @@ impl NumericSpecialization { properties, arrays, direct_calls: Default::default(), + native_calls: Default::default(), inline_callees: Default::default(), frame_inline_callees: Default::default(), numeric_constants, @@ -2812,6 +2835,7 @@ fn lower_optimized_machine( pointer_type, layout, specialization.direct_calls.get(&node.pc()), + specialization.native_calls.get(&node.pc()), call_guard, call.is_some_and(|call| { call.result() @@ -7345,6 +7369,7 @@ fn emit_opt_specialized_call( pointer_type: cranelift_codegen::ir::Type, layout: super::helpers::FrameLayout, direct: Option<&DirectCallSite>, + native: Option<&NativeCallSite>, guard: u32, scalar_result: bool, diagnostic_key: Option, @@ -7393,6 +7418,38 @@ fn emit_opt_specialized_call( } return Err(CompileFailure::ResourceLimit); } + if let Some(native) = native.filter(|native| { + !has_this + && native.arguments.len() == argc + && native.callee_identity != 0 + && native.callee_bytecode_identity != 0 + && matches!( + stack_provenance[function_index], + OptProvenance::Argument(_) | OptProvenance::Local(_) | OptProvenance::OwnedSlot + ) + }) { + return emit_opt_native_call( + builder, + frame, + sret, + arg_buf, + var_buf, + stack_base, + arguments, + locals, + stack, + stack_provenance, + depth, + base, + argc, + pc, + signatures, + pointer_type, + layout, + native, + guard, + ); + } if let Some(direct) = direct.filter(|direct| { !has_this && direct.call.arity() == argc @@ -7766,6 +7823,283 @@ fn emit_opt_specialized_call( Ok(base + 1) } +/// Lowers a monomorphic CALL through the P3a native-call convention: guard +/// the callee identity and Int32 argument tags, prove the callee's global +/// self binding once (`JS_JitNativeCallBegin`), then run the whole pure +/// native chain. Every refusal or chain failure deoptimizes before the CALL, +/// so the interpreter re-executes it with no effect lost or duplicated. +#[allow(clippy::too_many_arguments)] // Mirrors the specialized-call lowering state. +fn emit_opt_native_call( + builder: &mut cranelift_frontend::FunctionBuilder<'_>, + frame: cranelift_codegen::ir::Value, + sret: cranelift_codegen::ir::Value, + arg_buf: cranelift_codegen::ir::Value, + var_buf: cranelift_codegen::ir::Value, + stack_base: cranelift_codegen::ir::Value, + arguments: &[OptVars], + locals: &[OptVars], + stack: &[OptVars], + stack_provenance: &mut [OptProvenance], + depth: usize, + base: usize, + argc: usize, + pc: u32, + signatures: &[cranelift_codegen::ir::SigRef], + pointer_type: cranelift_codegen::ir::Type, + layout: super::helpers::FrameLayout, + native: &NativeCallSite, + guard: u32, +) -> Result { + use cranelift_codegen::ir::condcodes::IntCC; + use cranelift_codegen::ir::{ + types, AbiParam, ExtFuncData, ExternalName, InstBuilder, MemFlags, Signature, + StackSlotData, StackSlotKind, UserExternalName, + }; + use rquickjs_core::qjs; + + let function_index = base; + let argv_index = base + 1; + let function = opt_use(builder, stack[function_index]); + let arguments_ok = builder.create_block(); + let invoke = builder.create_block(); + let deopt = builder.create_block(); + super::emit_guarded_direct_callee_identity( + builder, + function.tag, + function.payload, + super::DirectCalleeIdentity { + object: native.callee_identity, + bytecode: native.callee_bytecode_identity, + }, + pointer_type, + arguments_ok, + deopt, + ); + builder.switch_to_block(arguments_ok); + let mut matches = builder.ins().iconst(types::I8, 1); + for (index, representation) in native.arguments.iter().enumerate() { + let value = opt_use(builder, stack[argv_index + index]); + let tag = match representation { + crate::runtime::FeedbackRepresentation::Int32 => qjs::JS_TAG_INT, + crate::runtime::FeedbackRepresentation::Float64 => qjs::JS_TAG_FLOAT64, + _ => return Err(CompileFailure::InvalidArtifact), + }; + let typed = builder + .ins() + .icmp_imm(IntCC::Equal, value.tag, i64::from(tag)); + matches = builder.ins().band(matches, typed); + } + builder.ins().brif(matches, invoke, &[], deopt, &[]); + + builder.switch_to_block(deopt); + for (index, vars) in arguments.iter().enumerate() { + let value = opt_use(builder, *vars); + opt_store(builder, arg_buf, index, value); + } + for (index, vars) in locals.iter().enumerate() { + let value = opt_use(builder, *vars); + opt_store(builder, var_buf, index, value); + } + for (index, vars) in stack.iter().take(depth).enumerate() { + let value = opt_use(builder, *vars); + opt_store(builder, stack_base, index, value); + } + opt_set_stack_top(builder, frame, stack_base, 0, pointer_type, layout); + let start = builder + .ins() + .load(pointer_type, MemFlags::new(), frame, layout.bytecode_start); + let resume = builder.ins().iadd_imm(start, i64::from(pc)); + builder + .ins() + .store(MemFlags::new(), resume, frame, layout.pc); + opt_own_stack_for_exit( + builder, + frame, + sret, + stack_base, + depth, + arguments.len() + locals.len(), + stack_provenance, + signatures, + pointer_type, + layout, + )?; + emit_opt_exit( + builder, + sret, + qjs::JSJitExitKind_JS_JIT_EXIT_DEOPT, + Some(resume), + pointer_type, + guard, + ); + + builder.switch_to_block(invoke); + let call_conv = builder.func.signature.call_conv; + let context_slot = + builder.create_sized_stack_slot(StackSlotData::new(StackSlotKind::ExplicitSlot, 32, 3)); + let context = builder.ins().stack_addr(pointer_type, context_slot, 0); + let output_slot = + builder.create_sized_stack_slot(StackSlotData::new(StackSlotKind::ExplicitSlot, 8, 3)); + let output = builder.ins().stack_addr(pointer_type, output_slot, 0); + let stack_pointer = builder.ins().get_stack_pointer(pointer_type); + let ctx = builder + .ins() + .load(pointer_type, MemFlags::new(), frame, layout.ctx); + let mut begin_signature = Signature::new(call_conv); + for ty in [ + pointer_type, + pointer_type, + types::I32, + pointer_type, + pointer_type, + ] { + begin_signature.params.push(AbiParam::new(ty)); + } + begin_signature.returns.push(AbiParam::new(types::I32)); + let begin_signature = builder.import_signature(begin_signature); + let begin = builder.ins().iconst( + pointer_type, + super::native_call::qjsjit_native_call_begin as *const () as usize as i64, + ); + let atom = builder + .ins() + .iconst(types::I32, i64::from(native.atom as i32)); + let call = super::emit_external_call( + builder, + begin_signature, + begin, + &[ctx, function.payload, atom, context, stack_pointer], + pointer_type, + Some(frame), + None, + ); + let refused = builder.inst_results(call)[0]; + let enter = builder.create_block(); + builder.ins().brif(refused, deopt, &[], enter, &[]); + + builder.switch_to_block(enter); + let entry_signature = builder.import_signature(super::native_call::entry_signature( + pointer_type, + call_conv, + &native.arguments, + )?); + let target = match native.entry { + Some(entry) => builder.ins().iconst(pointer_type, entry as i64), + None => { + let name = builder + .func + .declare_imported_user_function(UserExternalName::new( + super::native_call::NATIVE_ENTRY_NAMESPACE, + 0, + )); + let entry = builder.import_function(ExtFuncData { + name: ExternalName::user(name), + signature: entry_signature, + colocated: false, + }); + builder.ins().func_addr(pointer_type, entry) + } + }; + let mut params = Vec::with_capacity(argc + 3); + params.push(context); + params.push(output); + params.push(stack_pointer); + for (index, representation) in native.arguments.iter().enumerate() { + let value = opt_use(builder, stack[argv_index + index]); + params.push(match representation { + crate::runtime::FeedbackRepresentation::Float64 => { + builder + .ins() + .bitcast(types::F64, MemFlags::new(), value.payload) + } + _ => builder.ins().ireduce(types::I32, value.payload), + }); + } + let call = super::emit_external_call( + builder, + entry_signature, + target, + ¶ms, + pointer_type, + None, + None, + ); + let status = builder.inst_results(call)[0]; + let mut end_signature = Signature::new(call_conv); + end_signature.params.push(AbiParam::new(pointer_type)); + end_signature.params.push(AbiParam::new(pointer_type)); + let end_signature = builder.import_signature(end_signature); + let end = builder.ins().iconst( + pointer_type, + super::native_call::qjsjit_native_call_end as *const () as usize as i64, + ); + super::emit_external_call( + builder, + end_signature, + end, + &[context, stack_pointer], + pointer_type, + Some(frame), + None, + ); + let done = builder.create_block(); + builder.ins().brif(status, deopt, &[], done, &[]); + + builder.switch_to_block(done); + let result = match native.result { + crate::runtime::FeedbackRepresentation::Int32 => { + let raw = builder + .ins() + .load(types::I32, MemFlags::trusted(), output, 0); + OptPair { + payload: builder.ins().sextend(types::I64, raw), + tag: builder.ins().iconst(types::I64, i64::from(qjs::JS_TAG_INT)), + } + } + crate::runtime::FeedbackRepresentation::Float64 => OptPair { + payload: builder + .ins() + .load(types::I64, MemFlags::trusted(), output, 0), + tag: builder + .ins() + .iconst(types::I64, i64::from(qjs::JS_TAG_FLOAT64)), + }, + _ => return Err(CompileFailure::InvalidArtifact), + }; + if stack_provenance[function_index] == OptProvenance::OwnedSlot { + // The binding proven by Begin still holds the callee, so this only + // drops the global lookup's extra reference. + opt_store(builder, stack_base, function_index, function); + opt_set_stack_top( + builder, + frame, + stack_base, + function_index + 1, + pointer_type, + layout, + ); + let flat_slot = (arguments.len() + locals.len()) + .checked_add(function_index) + .and_then(|slot| u32::try_from(slot).ok()) + .ok_or(CompileFailure::ResourceLimit)?; + emit_opt_helper( + builder, + frame, + sret, + stack_base, + function_index + 1, + signatures, + qjs::JSJitHelperId_JS_JIT_HELPER_FREE as usize, + &[0, flat_slot], + pointer_type, + layout, + )?; + } + opt_define(builder, stack[base], result); + stack_provenance[base] = OptProvenance::ImmediatePrimitive; + Ok(base + 1) +} + /// Locals that ever receive a value owned by its interpreter stack slot /// (helper results: global lookups, kept-receiver property loads and /// non-specialized call results). Stores into such a local must first @@ -9230,6 +9564,68 @@ fn tier2_stage( result } +/// Routes monomorphic calls to callees that published a P3a native entry, +/// and compiles this artifact's own native entry when the function is a pure +/// self-recursive Int32 function, routing its self-call sites to it. Other +/// sites keep their existing lowering. +fn prepare_native_calls( + isa: &cranelift_codegen::isa::OwnedTargetIsa, + request: &CompileRequest, + specialization: &mut NumericSpecialization, + control: Option<&CompileControl>, + direct_dependencies: &mut Vec, +) -> Option<( + super::native_call::NativeCallPlan, + super::baseline::RelocatableCode, +)> { + if request.side_path_profile().is_some() { + return None; + } + let key = request.key(); + for instruction in request.snapshot().instructions() { + if let Some(target) = request.native_call_target(instruction.pc()) { + direct_dependencies.push(target.publication()); + specialization.native_calls.insert( + instruction.pc(), + NativeCallSite { + atom: target.plan().atom(), + arguments: target.plan().arguments().into(), + result: target.plan().result(), + callee_identity: target.link().callee_identity(), + callee_bytecode_identity: target.link().callee_bytecode_identity(), + entry: Some(target.entry() as usize), + }, + ); + } + } + let signature = request.feedback().bounded_specialization(key)?; + let (plan, code) = + super::native_call::lower(isa, request.snapshot(), &signature, control).ok()?; + for instruction in request.snapshot().instructions() { + if !is_call_site(instruction.opcode().name()) { + continue; + } + let Some(link) = request.feedback().call_link_at(key, instruction.pc()) else { + continue; + }; + if link.callee() != key { + continue; + } + specialization.native_calls.insert( + instruction.pc(), + NativeCallSite { + atom: plan.atom(), + arguments: plan.arguments().into(), + result: plan.result(), + callee_identity: link.callee_identity(), + callee_bytecode_identity: link.callee_bytecode_identity(), + entry: None, + }, + ); + } + Some((plan, code)) +} + /// Call opcodes whose call-site feedback the runtime records at the /// instruction pc; tail calls are lowered as the equivalent call plus return. fn is_call_site(name: &str) -> bool { @@ -9345,6 +9741,13 @@ impl Compiler for Tier2Compiler { } #[cfg(feature = "test-support")] record_tier2_stage(key, Tier2CompileStage::DirectDependencies, None); + let native_entry = prepare_native_calls( + &self.isa, + &request, + &mut specialization, + None, + &mut direct_dependencies, + ); let code = match lower_optimized_machine(&self.isa, &ir, None, profile, &specialization) { Err(CompileFailure::InvalidArtifact) if ir.scalar_graph().frame_inlined_calls() != 0 => @@ -9414,6 +9817,20 @@ impl Compiler for Tier2Compiler { .with_optimized_metadata(optimized) .with_direct_call_relocatable(direct_code); } + if let Some((plan, native_code)) = native_entry { + let optimized = tier2_stage( + key, + Tier2CompileStage::DirectMetadata, + artifact + .optimized_metadata() + .cloned() + .ok_or(CompileFailure::InvalidArtifact), + )? + .with_native_call_plan(plan); + artifact = artifact + .with_optimized_metadata(optimized) + .with_native_call_relocatable(native_code); + } #[cfg(feature = "test-support")] record_tier2_stage(key, Tier2CompileStage::Complete, None); Ok(artifact.with_direct_call_dependencies(direct_dependencies)) @@ -9544,6 +9961,13 @@ impl Compiler for Tier2Compiler { } #[cfg(feature = "test-support")] record_tier2_stage(key, Tier2CompileStage::DirectDependencies, None); + let native_entry = prepare_native_calls( + &self.isa, + &request, + &mut specialization, + Some(control), + &mut direct_dependencies, + ); let code = match lower_optimized_machine( &self.isa, &ir, @@ -9637,6 +10061,20 @@ impl Compiler for Tier2Compiler { .with_optimized_metadata(optimized) .with_direct_call_relocatable(direct_code); } + if let Some((plan, native_code)) = native_entry { + let optimized = tier2_stage( + key, + Tier2CompileStage::DirectMetadata, + artifact + .optimized_metadata() + .cloned() + .ok_or(CompileFailure::InvalidArtifact), + )? + .with_native_call_plan(plan); + artifact = artifact + .with_optimized_metadata(optimized) + .with_native_call_relocatable(native_code); + } #[cfg(feature = "test-support")] record_tier2_stage(key, Tier2CompileStage::Complete, None); Ok(artifact.with_direct_call_dependencies(direct_dependencies)) diff --git a/jit/src/lib.rs b/jit/src/lib.rs index d9a15bd96..10dd48614 100644 --- a/jit/src/lib.rs +++ b/jit/src/lib.rs @@ -148,6 +148,10 @@ impl Jit { } return Err(error.into()); } + // Measure the interpreter's per-call stack cost once per process, + // outside JIT callbacks, before any native call entry is compiled. + #[cfg(all(feature = "compiler", not(target_family = "wasm")))] + let _ = compiler::native_call::interpreter_frame_bytes(); let metrics = Arc::new(Mutex::new(JitMetrics::disabled())); #[cfg(all( @@ -691,6 +695,19 @@ struct ProductionBackend { /// not a terminal function blacklist: stable feedback may still justify /// one of the coordinator's bounded optimizing-tier trials. profitability_blacklisted: rustc_hash::FxHashSet, + /// Functions whose Tier 2 artifact runs self recursion as native calls. + /// One optimized entry then covers a whole recursion tree, so its + /// invocation time is not comparable with per-call baseline samples. + native_recursive: rustc_hash::FxHashSet, + /// Last native-entry admission answer per function and signature. + native_call_plans: rustc_hash::FxHashMap< + runtime::FunctionKey, + ( + Box<[runtime::FeedbackRepresentation]>, + runtime::FeedbackRepresentation, + bool, + ), + >, /// Immutable generations for which neither tier can ever produce code. /// `record_hot` reports these to QuickJS so it can turn off all probes and /// feedback at the bytecode object, avoiding a permanent C -> Rust tax. @@ -757,6 +774,10 @@ struct ProductionProfile { helper_calls: u64, baseline_executions: u64, baseline_ns: u64, + /// Baseline entries not nested in an active execution of the same + /// function: `baseline_executions / baseline_outermost` estimates how many + /// calls one outside entry performs through self recursion. + baseline_outermost: u64, optimized_executions: u64, optimized_ns: u64, /// Fastest optimized invocation observed so far; meaningful only while @@ -783,6 +804,17 @@ impl ProductionProfile { } } +/// Minimum observed self-recursive calls per outside entry before a pure +/// recursive function is admitted to the native-call convention. +#[cfg(all(feature = "compiler", not(target_family = "wasm")))] +const NATIVE_RECURSION_MIN_CALLS_PER_ENTRY: u64 = 16; +/// Baseline executions observed before that ratio can admit a function. +#[cfg(all(feature = "compiler", not(target_family = "wasm")))] +const NATIVE_RECURSION_PROFILE_EXECUTIONS: u64 = 64; +/// Outside entries observed before a shallow ratio rejects a function. +#[cfg(all(feature = "compiler", not(target_family = "wasm")))] +const NATIVE_RECURSION_PROFILE_ENTRIES: u64 = 64; + /// Finish the bounded Tier-2 profitability trial once. Production JITs patch /// an IC/tier state after classification; repeating wide average comparisons /// on every successful optimized exit would turn policy into hot-path tax. @@ -2116,6 +2148,8 @@ impl ProductionBackend { profitability_rejected: 0, profitability_backoff: std::collections::HashMap::new(), profitability_blacklisted: rustc_hash::FxHashSet::default(), + native_recursive: rustc_hash::FxHashSet::default(), + native_call_plans: rustc_hash::FxHashMap::default(), feedback_disabled: rustc_hash::FxHashSet::default(), benefit_recordings: 0, measured_benefit_ns: 0, @@ -2307,7 +2341,24 @@ impl ProductionBackend { &observed, ); let resolved = self.coordinator.call_target_resolved(call.callee()); - if direct_ready || frame_ready { + let native_ready = self.coordinator.native_call_ready(call.callee()); + if direct_ready || frame_ready || native_ready { + continue; + } + // An admitted pure recursive callee publishes its native + // entry with its optimizing artifact; wait for it as for a + // compiling scalar callee. + if self.native_recursive.contains(&call.callee()) + && matches!( + self.coordinator + .tier_state(call.callee(), runtime::Tier::Optimizing), + runtime::CompileState::Cold + | runtime::CompileState::Queued(_) + | runtime::CompileState::Compiling(_) + | runtime::CompileState::Ready(_) + ) + { + direct_call_pending = true; continue; } if !resolved { @@ -2350,6 +2401,74 @@ impl ProductionBackend { .control_flow_graph() .is_loop_header(block.start_pc()) }); + /* A pure self-recursive Int32 function publishes a native call + * entry (P3a): its monomorphic self calls run as direct native + * calls without the generic bridge, so they no longer need the + * loop amortization below. */ + let native_self_call = |pc: u32| { + observed + .call_link_at(key, pc) + .is_some_and(|link| link.callee() == key) + }; + let native_call_shape = !has_loop + && snapshot + .instructions() + .iter() + .any(|instruction| native_self_call(instruction.pc())) + && observed + .bounded_specialization(key) + .is_some_and(|signature| { + // Admission builds the entry's CLIF; reuse the answer + // while the function waits for its baseline profile. + // The feedback epoch changes every pass; the plan only + // depends on the representations. + match self.native_call_plans.get(&key) { + Some((arguments, result, admitted)) + if **arguments == *signature.arguments() + && *result == signature.result() => + { + *admitted + } + _ => { + let admitted = + compiler::native_call::plan(snapshot, &signature).is_some(); + self.native_call_plans.insert( + key, + (signature.arguments().into(), signature.result(), admitted), + ); + admitted + } + } + }); + // Entering native code from QuickJS still costs several hundred + // nanoseconds of bookkeeping (B2), about ten interpreter calls. + // Admit only recursion whose observed trees amortize it; while the + // installed baseline is still collecting that profile, wait. + let (baseline_executions, baseline_outermost) = + self.execution_profiles.get(&key).map_or((0, 0), |profile| { + (profile.baseline_executions, profile.baseline_outermost) + }); + let native_call_admitted = native_call_shape + && baseline_executions >= NATIVE_RECURSION_PROFILE_EXECUTIONS + && baseline_outermost != 0 + && baseline_executions + >= baseline_outermost.saturating_mul(NATIVE_RECURSION_MIN_CALLS_PER_ENTRY); + // Trees already running in the interpreter when the baseline was + // installed make early samples look shallow; reject only after + // enough outside entries have been observed. + if native_call_shape + && !native_call_admitted + && baseline_outermost < NATIVE_RECURSION_PROFILE_ENTRIES + && matches!( + self.coordinator.tier_state(key, runtime::Tier::Baseline), + runtime::CompileState::Installed(_) + ) + { + continue; + } + if native_call_admitted { + self.native_recursive.insert(key); + } let generic_call_without_loop = !has_loop && snapshot.instructions().iter().any(|instruction| { matches!( @@ -2368,6 +2487,13 @@ impl ProductionBackend { .is_some_and(|call| { call.callee() != key && self.coordinator.direct_call_ready(&call) }) + && !(native_call_admitted && native_self_call(instruction.pc())) + && !observed + .call_link_at(key, instruction.pc()) + .is_some_and(|link| { + link.callee() != key + && self.coordinator.native_call_ready(link.callee()) + }) }); #[cfg(feature = "test-support")] let forced_call_only = self.config.force_optimized(); @@ -2455,15 +2581,16 @@ impl ProductionBackend { // compiled call/property/element fast path, so rejecting it here // would make the optimized path permanently unreachable. The // measured Tier-2 window below still demotes a losing artifact. - let stable_ic_candidate = snapshot.instructions().iter().any(|instruction| { - observed - .call_specialization_at(key, instruction.pc()) - .is_some_and(|call| call.callee() != key) - || observed.property_at(instruction.pc()).is_some() - || observed - .array_at(key, instruction.pc()) - .is_some_and(|site| site.can_specialize()) - }); + let stable_ic_candidate = native_call_admitted + || snapshot.instructions().iter().any(|instruction| { + observed + .call_specialization_at(key, instruction.pc()) + .is_some_and(|call| call.callee() != key) + || observed.property_at(instruction.pc()).is_some() + || observed + .array_at(key, instruction.pc()) + .is_some_and(|site| site.can_specialize()) + }); /* A rejected baseline does not predict Tier2 profitability: the * baseline can lose to dispatch/callback overhead while a stable * unboxed loop wins by orders of magnitude. Give such a function @@ -3273,6 +3400,19 @@ unsafe impl rquickjs_core::runtime::JitBackend for ProductionBackend { if pc != 0 { self.osr_attempts = self.osr_attempts.saturating_add(1); } + // Counted at entry: a recursion tree's root is still active while + // maintenance samples its completed nested executions. Profiles are + // created by the first exit; entry never allocates. + if tier == runtime::Tier::Baseline + && self + .execution_starts + .last() + .is_none_or(|(parent, _, _)| *parent != key) + { + if let Some(profile) = self.execution_profiles.get_mut(&key) { + profile.baseline_outermost = profile.baseline_outermost.saturating_add(1); + } + } self.execution_starts .push((key, std::time::Instant::now(), tier)); } @@ -3304,6 +3444,11 @@ unsafe impl rquickjs_core::runtime::JitBackend for ProductionBackend { let profile = self.execution_profiles.entry(key).or_default(); if optimized { profile.record_optimized(elapsed); + if self.native_recursive.contains(&key) { + // Keep the trial, but never classify a collapsed + // recursion tree against per-call baseline samples. + profile.tier2_trial_decided = true; + } // Like V8/JSC tier-down decisions, use a bounded observation // window and a material margin instead of reacting to one // noisy invocation. Cross-multiply so the decision remains @@ -3413,6 +3558,8 @@ unsafe impl rquickjs_core::runtime::JitBackend for ProductionBackend { fn function_retire(&mut self, id: u64, generation: u64) { let key = runtime::FunctionKey::new(id, generation); self.requested.remove(&key); + self.native_recursive.remove(&key); + self.native_call_plans.remove(&key); self.queue_reasons.remove(&key); self.prequeue_backoff.remove(&key); self.hotness.remove(&key); diff --git a/jit/src/runtime/coordinator.rs b/jit/src/runtime/coordinator.rs index 475e609fb..58f592884 100644 --- a/jit/src/runtime/coordinator.rs +++ b/jit/src/runtime/coordinator.rs @@ -129,6 +129,33 @@ impl DirectCallTarget { } } +/// A monomorphic call edge whose callee artifact publishes a P3a native call +/// entry. The caller guards the callee identity and proves its global self +/// binding at run time; the publication pin keeps the entry executable. +#[cfg(all(feature = "compiler", not(target_family = "wasm")))] +#[derive(Clone, Debug)] +pub struct NativeCallTarget { + link: super::CallLinkStatus, + plan: crate::compiler::native_call::NativeCallPlan, + published: crate::compiler::baseline::PublishedBaselineCode, +} + +#[cfg(all(feature = "compiler", not(target_family = "wasm")))] +impl NativeCallTarget { + pub const fn link(&self) -> &super::CallLinkStatus { + &self.link + } + pub const fn plan(&self) -> &crate::compiler::native_call::NativeCallPlan { + &self.plan + } + pub fn entry(&self) -> *const u8 { + self.published.as_ptr() + } + pub(crate) fn publication(&self) -> crate::compiler::baseline::PublishedBaselineCode { + self.published.clone() + } +} + /// Copied callee body retained for tagged frame inlining, independently of a /// scalar direct-call entry. Target guards and dependency registration remain /// the consuming compiler's responsibility. @@ -509,6 +536,8 @@ pub struct CompileRequest { direct_call_targets: Arc<[DirectCallTarget]>, #[cfg(all(feature = "compiler", not(target_family = "wasm")))] frame_inline_targets: Arc<[FrameInlineTarget]>, + #[cfg(all(feature = "compiler", not(target_family = "wasm")))] + native_call_targets: Arc<[NativeCallTarget]>, } impl CompileRequest { @@ -601,6 +630,15 @@ impl CompileRequest { pub fn frame_inline_targets(&self) -> &[FrameInlineTarget] { &self.frame_inline_targets } + /// The native-call target of a monomorphic call site, if its callee + /// published a P3a native entry when this request was queued. + #[cfg(all(feature = "compiler", not(target_family = "wasm")))] + pub fn native_call_target(&self, pc: u32) -> Option<&NativeCallTarget> { + let status = self.feedback.call_link_at(self.key, pc)?; + self.native_call_targets + .iter() + .find(|target| target.link == status) + } #[cfg(all(feature = "compiler", not(target_family = "wasm")))] pub fn frame_inline_target(&self, pc: u32) -> Option<&FrameInlineTarget> { let status = self.feedback.call_link_at(self.key, pc)?; @@ -1455,6 +1493,32 @@ impl Coordinator { Arc::from([]) }; #[cfg(all(feature = "compiler", not(target_family = "wasm")))] + let native_call_targets: Arc<[NativeCallTarget]> = + if tier == Tier::Optimizing && side_path_profile.is_none() { + snapshot + .instructions() + .iter() + .filter_map(|instruction| { + let link = feedback.call_link_at(key, instruction.pc())?; + if link.callee() == key { + return None; + } + let pin = self.pin(link.callee(), Tier::Optimizing)?; + let artifact = pin.artifact(); + let plan = artifact.optimized_metadata()?.native_call_plan()?.clone(); + let published = artifact.native_call_published()?.clone(); + Some(NativeCallTarget { + link, + plan, + published, + }) + }) + .collect::>() + .into() + } else { + Arc::from([]) + }; + #[cfg(all(feature = "compiler", not(target_family = "wasm")))] let artifact_key = if frame_inline_targets.is_empty() { artifact_key } else { @@ -1484,6 +1548,8 @@ impl Coordinator { direct_call_targets, #[cfg(all(feature = "compiler", not(target_family = "wasm")))] frame_inline_targets, + #[cfg(all(feature = "compiler", not(target_family = "wasm")))] + native_call_targets, }); if let Some(signature) = side_path_signature { let versions = self @@ -1987,6 +2053,19 @@ impl Coordinator { .admission_probe(kind, caller_shape) } + /// True when the callee's installed optimizing artifact publishes a P3a + /// native call entry that a monomorphic caller can link to. + #[cfg(all(feature = "compiler", not(target_family = "wasm")))] + pub fn native_call_ready(&mut self, callee: FunctionKey) -> bool { + self.pin(callee, Tier::Optimizing).is_some_and(|pin| { + pin.artifact().native_call_published().is_some() + && pin + .artifact() + .optimized_metadata() + .is_some_and(|metadata| metadata.native_call_plan().is_some()) + }) + } + /// A resolved target cannot become frame-inlineable without a generation /// or artifact replacement, both of which change the normal readiness /// inputs and retry the caller independently. diff --git a/jit/src/runtime/mod.rs b/jit/src/runtime/mod.rs index d40bab251..7d10fda45 100644 --- a/jit/src/runtime/mod.rs +++ b/jit/src/runtime/mod.rs @@ -24,7 +24,7 @@ pub use coordinator::{ SideExitAction, SidePathProfile, Tier, DEFAULT_COMPLETION_DRAIN_BUDGET, }; #[cfg(all(feature = "compiler", not(target_family = "wasm")))] -pub use coordinator::{DirectCallTarget, FrameInlineTarget}; +pub use coordinator::{DirectCallTarget, FrameInlineTarget, NativeCallTarget}; pub use feedback::{ BinaryFeedbackFlags, BinaryFeedbackSnapshot, BoundedSpecializationSignature, BranchFeedbackSnapshot, CallFeedbackSnapshot, CallLinkStatus, CallSignatureFeedbackSnapshot, diff --git a/jit/tests/native_calls.rs b/jit/tests/native_calls.rs new file mode 100644 index 000000000..0686e1a5e --- /dev/null +++ b/jit/tests/native_calls.rs @@ -0,0 +1,465 @@ +//! P3a native-call convention: pure self-recursive Int32 functions run their +//! recursion as direct native calls, and every failure re-executes the CALL +//! in the interpreter with identical results, exceptions and effects. +#![cfg(all( + feature = "compiler", + feature = "test-support", + target_endian = "little", + any(target_os = "linux", target_os = "macos"), + any(target_arch = "x86_64", target_arch = "aarch64") +))] + +use rquickjs::{Context, Runtime}; +use rquickjs_core::qjs; +use rquickjs_jit::bytecode::VerifyLimits; +use rquickjs_jit::runtime::{FeedbackTable, FunctionKey, ObservedType}; +use rquickjs_jit::test_support::SnapshotFixture; +use rquickjs_jit::{Jit, JitConfig, JitTierPolicy}; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +unsafe extern "C" { + fn JS_JitGetHelperCount(rt: *mut qjs::JSRuntime, helper: u32, count: *mut u64) -> i32; +} + +const FIBONACCI: &str = "function fib(n){if(n<2)return n;return fib(n-1)+fib(n-2);}"; + +fn generic_calls(context: &Context) -> u64 { + context.with(|ctx| { + let rt = unsafe { qjs::JS_GetRuntime(ctx.as_raw().as_ptr()) }; + let mut count = 0; + assert_eq!( + unsafe { JS_JitGetHelperCount(rt, qjs::JSJitHelperId_JS_JIT_HELPER_CALL, &mut count) }, + 0 + ); + count + }) +} + +struct Harness { + _runtime: Runtime, + jit: Jit, + context: Context, +} + +impl Harness { + fn new(source: &str, force_tier2: bool) -> Self { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .tier_policy(JitTierPolicy::Automatic) + .call_threshold(2) + .loop_threshold(16) + .force_optimized_for_test(force_tier2) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| ctx.eval::<(), _>(source)).unwrap(); + Self { + _runtime: runtime, + jit, + context, + } + } + + fn eval_string(&self, expression: &str) -> String { + self.context.with(|ctx| { + ctx.eval::(format!("String({expression})")) + .unwrap_or_else(|error| { + let exception = ctx.catch(); + panic!("{expression} threw {error}: {exception:?}") + }) + }) + } + + /// Runs `expression` until its self-recursive calls stop using the + /// generic CALL bridge: one Tier 2 entry per evaluation and no helper + /// CALL, so the whole recursion runs as native calls. + fn warm_native(&self, expression: &str, expected: &str) { + let deadline = Instant::now() + Duration::from_secs(60); + loop { + let calls = generic_calls(&self.context); + let before = self.jit.metrics(); + assert_eq!(self.eval_string(expression), expected); + let after = self.jit.metrics(); + self.jit.poll(); + if after.tier2_entries > before.tier2_entries + && after.pending_worker_jobs == 0 + && generic_calls(&self.context) == calls + && after.deopts == before.deopts + { + return; + } + assert!( + Instant::now() < deadline, + "native recursion never engaged for {expression}: {:?}", + self.jit.metrics() + ); + std::thread::sleep(Duration::from_millis(1)); + } + } +} + +#[test] +fn pure_self_recursion_publishes_a_helper_free_native_entry() { + let fixture = SnapshotFixture::compile(&format!("{FIBONACCI} fib")); + let verified = fixture.snapshot().verify(VerifyLimits::default()).unwrap(); + let key = FunctionKey::new( + verified.snapshot().function_id(), + verified.snapshot().generation(), + ); + let mut feedback = FeedbackTable::new(64, 2); + feedback.observe_call(key, &[ObservedType::Int32]); + for instruction in verified.instructions() { + match instruction.opcode().name() { + "add" | "sub" => { + feedback.observe_binary( + key, + instruction.pc(), + ObservedType::Int32, + ObservedType::Int32, + ObservedType::Int32, + Default::default(), + ); + } + "return" => feedback.observe_return(key, instruction.pc(), ObservedType::Int32), + _ => {} + } + } + let signature = feedback + .snapshot(7) + .bounded_specialization(key) + .expect("Int32 signature"); + let plan = rquickjs_jit::compiler::native_call::plan(&verified, &signature) + .expect("fibonacci is a pure self-recursive Int32 function"); + assert_eq!(plan.arity(), 1); + // At least one measured interpreter level plus the value slots. + assert!( + plan.frame_charge() >= rquickjs_jit::compiler::native_call::interpreter_frame_bytes() + 16 + ); + + // Anything that could run code, allocate or observe the frame is refused. + for rejected in [ + "function f(n){if(n<2)return n;return f(n-1)+g(n-2);} f", + "function f(n){if(n<2)return n;return f(n-1)+this.x;} f", + "function f(n){for(let i=0;i{const old=fib;fib=function(n){return 100};const r=old(10);fib=old;return r})()", + "200", + ), + ], + ); + check_after_warmup( + "globalThis.fib=function(n){if(n<2)return n;return fib(n-1)+fib(n-2);};", + &[("fib(12)", "144")], + &[ + // An accessor binding runs code on every lookup; the native path + // must not skip those getter calls. + ( + "(()=>{const impl=fib;let hits=0;delete globalThis.fib;\ + Object.defineProperty(globalThis,'fib',{configurable:true,get(){hits++;return impl}});\ + const r=impl(6);delete globalThis.fib;globalThis.fib=impl;return r+':'+hits})()", + "8:24", + ), + ], + ); +} + +#[test] +fn native_recursion_observes_a_shadowing_lexical_global() { + for force in [false, true] { + let harness = Harness::new( + "globalThis.fib=function(n){if(n<2)return n;return fib(n-1)+fib(n-2);};", + force, + ); + harness.warm_native("fib(12)", "144"); + harness + .context + .with(|ctx| { + ctx.eval::<(), _>("globalThis.impl=fib;")?; + ctx.eval::<(), _>("let fib=function(n){return 1};") + }) + .unwrap(); + assert_eq!(harness.eval_string("impl(6)"), "2"); + assert_eq!(harness.eval_string("impl(12)"), "2"); + let metrics = harness.jit.metrics(); + assert_eq!(metrics.native_entries, metrics.native_exits); + } +} + +#[test] +fn native_recursion_stack_exhaustion_throws_the_interpreter_range_error() { + let source = "function depth(n){return n<=0?0:1+depth(n-1);}"; + let harness = Harness::new(source, false); + harness.warm_native("depth(40)", "40"); + let started = Instant::now(); + assert_eq!( + harness.eval_string( + "(()=>{try{return depth(1e6)}catch(e){return e instanceof RangeError}})()" + ), + "true" + ); + // A retry per interpreter level would make exhaustion quadratic. + assert!(started.elapsed() < Duration::from_secs(10)); + assert_eq!(harness.eval_string("depth(100)"), "100"); + let metrics = harness.jit.metrics(); + assert_eq!(metrics.native_entries, metrics.native_exits); + // The interpreter's own maximum depth is still reachable. + let interpreter = Runtime::new().unwrap(); + let context = Context::full(&interpreter).unwrap(); + let limit: i32 = context.with(|ctx| { + ctx.eval(format!( + "{source} let lo=1,hi=1<<20;while(lo>1;try{{depth(m);lo=m}}catch(e){{hi=m-1}}}} lo" + )) + .unwrap() + }); + // Warm native code never reports a result where the interpreter throws. + assert_eq!( + harness.eval_string(&format!( + "(()=>{{try{{return depth({})}}catch(e){{return e.name}}}})()", + limit + 64 + )), + "RangeError" + ); +} + +#[test] +fn native_recursion_polls_the_interrupt_handler() { + let harness = Harness::new(FIBONACCI, false); + harness.warm_native("fib(15)", "610"); + let polls = Arc::new(AtomicUsize::new(0)); + let interrupt = Arc::new(AtomicBool::new(false)); + harness._runtime.set_interrupt_handler(Some(Box::new({ + let polls = Arc::clone(&polls); + let interrupt = Arc::clone(&interrupt); + move || { + polls.fetch_add(1, Ordering::SeqCst); + interrupt.load(Ordering::SeqCst) + } + }))); + // A handler that never interrupts must not change the result. + assert_eq!(harness.eval_string("fib(22)"), "17711"); + assert!( + polls.load(Ordering::SeqCst) > 0, + "native recursion never polled" + ); + interrupt.store(true, Ordering::SeqCst); + let result = harness.context.with(|ctx| { + ctx.eval::("fib(25)").map_err(|error| { + // Clear the uncatchable error so later evaluations start clean. + let exception = ctx.catch(); + format!("{error}: {exception:?}") + }) + }); + let error = result.expect_err("interrupt was not delivered"); + assert!(error.contains("interrupted"), "{error}"); + interrupt.store(false, Ordering::SeqCst); + harness._runtime.set_interrupt_handler(None); + assert_eq!(harness.eval_string("fib(20)"), "6765"); + let metrics = harness.jit.metrics(); + assert_eq!(metrics.native_entries, metrics.native_exits); +} + +#[test] +fn monomorphic_callers_link_to_a_published_native_entry() { + let source = format!( + "{FIBONACCI}\nfunction caller(n){{let s=0;for(let i=0;i {:?}", + before, + after + ); + std::thread::sleep(Duration::from_millis(1)); + } + // A rebinding makes the callee identity guard miss; the CALL is + // re-executed by the interpreter against the new binding. + assert_eq!( + harness.eval_string( + "(()=>{const old=fib;fib=function(n){return 2};const r=caller(3);fib=old;return r})()" + ), + "6" + ); + assert_eq!(harness.eval_string("caller(2)"), "288"); + let metrics = harness.jit.metrics(); + assert_eq!(metrics.native_entries, metrics.native_exits); + } +} + +#[test] +fn shallow_recursion_stays_out_of_native_code() { + // Four calls per outside entry cannot amortize QuickJS's native entry. + let harness = Harness::new( + "function shallow(n){return n<=0?0:n+shallow(n-1);}\n\ + function drive(k){let s=0;for(let i=0;i Date: Sun, 27 Sep 2026 17:22:25 +0800 Subject: [PATCH 33/54] perf(jit): settle Tier 1 functions that Tier 2 can never translate Tier 2 translates whole functions, so a generation containing an opcode without an optimized classification can never compile there. The optimizing scan nevertheless revisited such candidates at every maintenance, rebuilding a full feedback snapshot whenever a property, direct-call or numeric gate deferred the trial (such candidates stayed in Tier 1 forever), and otherwise queued Tier 2 attempts that could only fail until the optimizing tier was blacklisted and the generation's probes were disabled, leaving it in the interpreter. Admitting more opcodes to Tier 1 makes this common for methods and host-style helpers (a new method-heavy kernel ran 5.4x slower, most of it in maintenance). After a 16-execution baseline warmup, settle an untranslatable candidate once: keep the call-only interpreter demotion and its metric, and otherwise move straight to the terminal state (baseline demoted, probes off) that the failed bounded trials used to reach. Before the warmup completes the deferral is a map lookup. Forced Tier 2 test runs keep the old path. Co-Authored-By: Claude Opus 5.5 (1M context) --- jit/Cargo.toml | 4 ++ jit/src/ir/mod.rs | 1 + jit/src/ir/optimized.rs | 12 ++++ jit/src/lib.rs | 101 ++++++++++++++++++++++++++++- jit/tests/untranslatable_settle.rs | 86 ++++++++++++++++++++++++ 5 files changed, 203 insertions(+), 1 deletion(-) create mode 100644 jit/tests/untranslatable_settle.rs diff --git a/jit/Cargo.toml b/jit/Cargo.toml index 97291692c..5b80e145b 100644 --- a/jit/Cargo.toml +++ b/jit/Cargo.toml @@ -191,6 +191,10 @@ required-features = ["compiler", "test-support"] name = "tier1_generic_ops" required-features = ["compiler", "test-support"] +[[test]] +name = "untranslatable_settle" +required-features = ["compiler", "test-support"] + [[test]] name = "test262" required-features = ["compiler", "test-support"] diff --git a/jit/src/ir/mod.rs b/jit/src/ir/mod.rs index b01b762e1..786c1f9a6 100644 --- a/jit/src/ir/mod.rs +++ b/jit/src/ir/mod.rs @@ -44,4 +44,5 @@ pub use scalar::{ }; pub use types::{BinaryOp, GenericOp, IrOp, PollKind, StackOp, TaggedValue, UnaryOp}; +pub(crate) use optimized::optimized_opcodes_supported; pub(crate) use scalar::stack_permutation; diff --git a/jit/src/ir/optimized.rs b/jit/src/ir/optimized.rs index 635a2cbd3..e2c042a1c 100644 --- a/jit/src/ir/optimized.rs +++ b/jit/src/ir/optimized.rs @@ -865,6 +865,18 @@ fn optimized_block_depths( Ok(depths) } +/// Whether every instruction has an optimized-IR classification. Tier 2 +/// translates the whole function, so a single unclassified opcode (exact +/// generic helpers, literal allocation, ...) makes every attempt fail. +pub(crate) fn optimized_opcodes_supported(function: &VerifiedFunction) -> bool { + function.instructions().iter().all(|instruction| { + matches!( + instruction.opcode().name(), + "tail_call" | "tail_call_method" + ) || classify_optimized_opcode(instruction.opcode().name()).is_ok() + }) +} + fn classify_optimized_opcode( name: &str, ) -> Result<(ValueRepresentation, OptimizedEffect), CompileFailure> { diff --git a/jit/src/lib.rs b/jit/src/lib.rs index d9a15bd96..01157699c 100644 --- a/jit/src/lib.rs +++ b/jit/src/lib.rs @@ -597,6 +597,11 @@ unsafe impl rquickjs_core::runtime::JitBackend for NoopBackend {} #[cfg(all(feature = "compiler", not(target_family = "wasm")))] const HOT_MAINTENANCE_INTERVAL: u32 = 64; +/// Baseline executions an untranslatable generation serves before it is +/// settled into the interpreter (see `settle_untranslatable_tier2_candidate`). +#[cfg(all(feature = "compiler", not(target_family = "wasm")))] +const UNTRANSLATABLE_SETTLE_EXECUTIONS: u64 = 16; + /// Feedback pc used for return types observed at native `DONE` exits. #[cfg(all(feature = "compiler", not(target_family = "wasm")))] const NATIVE_RETURN_FEEDBACK_PC: u32 = u32::MAX; @@ -691,6 +696,9 @@ struct ProductionBackend { /// not a terminal function blacklist: stable feedback may still justify /// one of the coordinator's bounded optimizing-tier trials. profitability_blacklisted: rustc_hash::FxHashSet, + /// Generations whose bytecode Tier 2 can never translate and whose + /// baseline decision is settled; they leave the optimizing scan. + tier2_untranslatable: rustc_hash::FxHashSet, /// Immutable generations for which neither tier can ever produce code. /// `record_hot` reports these to QuickJS so it can turn off all probes and /// feedback at the bytecode object, avoiding a permanent C -> Rust tax. @@ -2116,6 +2124,7 @@ impl ProductionBackend { profitability_rejected: 0, profitability_backoff: std::collections::HashMap::new(), profitability_blacklisted: rustc_hash::FxHashSet::default(), + tier2_untranslatable: rustc_hash::FxHashSet::default(), feedback_disabled: rustc_hash::FxHashSet::default(), benefit_recordings: 0, measured_benefit_ns: 0, @@ -2179,6 +2188,87 @@ impl ProductionBackend { } } + /// Tier 2 translates the whole function, so a generation containing an + /// opcode without an optimized classification can never compile there. + /// The optimizing scan would otherwise revisit it at every maintenance, + /// rebuilding feedback snapshots whenever a property/call/numeric gate + /// defers the trial. Every other route ends in the interpreter: call-only + /// functions are demoted directly, five measured rejections demote the + /// baseline, and an approved trial fails its bounded Tier 2 attempts until + /// the optimizing tier is blacklisted, which disables the generation's + /// probes. Reach that terminal state at the first scan, without the + /// failing compilations or the slower baseline in between. The call-only + /// rule still runs first so its metric keeps its meaning. + fn settle_untranslatable_tier2_candidate(&mut self, key: runtime::FunctionKey) { + let Some(snapshot) = self.optimizing_snapshots.get(&key) else { + return; + }; + if self.profitability_blacklisted.contains(&key) { + self.retire_untranslatable_to_interpreter(key); + return; + } + // Like the bounded Tier 2 retries this replaces, let the installed + // baseline serve a short warmup (and any OSR it enabled) first. The + // deferral is a map lookup; no feedback snapshot is built. + if self + .execution_profiles + .get(&key) + .is_none_or(|profile| profile.baseline_executions < UNTRANSLATABLE_SETTLE_EXECUTIONS) + { + return; + } + let has_loop = snapshot.control_flow_graph().blocks().iter().any(|block| { + snapshot + .control_flow_graph() + .is_loop_header(block.start_pc()) + }); + let call_pcs = snapshot + .instructions() + .iter() + .filter(|instruction| { + matches!( + instruction.opcode().name(), + "call" + | "call0" + | "call1" + | "call2" + | "call3" + | "call_method" + | "tail_call" + | "tail_call_method" + | "call_constructor" + ) + }) + .map(|instruction| instruction.pc()) + .collect::>(); + if !has_loop && !call_pcs.is_empty() { + let observed = self.feedback.snapshot(self.clock.max(1)); + let generic_call = call_pcs.iter().any(|pc| { + !observed + .call_specialization_at(key, *pc) + .is_some_and(|call| { + call.callee() != key && self.coordinator.direct_call_ready(&call) + }) + }); + if generic_call { + self.cold_metrics_dirty = true; + self.generic_call_rejections = self.generic_call_rejections.saturating_add(1); + self.retire_untranslatable_to_interpreter(key); + return; + } + } + self.retire_untranslatable_to_interpreter(key); + } + + fn retire_untranslatable_to_interpreter(&mut self, key: runtime::FunctionKey) { + self.optimizing_snapshots.remove(&key); + self.feedback_disabled.insert(key); + self.tier2_untranslatable.insert(key); + if self.profitability_blacklisted.insert(key) { + self.coordinator.demote_baseline_to_interpreter(key); + } + } + fn maintenance(&mut self) { // Maintenance owns installation, demotion, feedback-epoch updates, // and reclamation. Invalidate before touching any of those states. @@ -2243,7 +2333,7 @@ impl ProductionBackend { .keys() .copied() .filter(|key| { - if self.feedback_disabled.contains(key) { + if self.feedback_disabled.contains(key) || self.tier2_untranslatable.contains(key) { return false; } if self @@ -2275,6 +2365,14 @@ impl ProductionBackend { let Some(snapshot) = self.optimizing_snapshots.get(&key) else { continue; }; + #[cfg(feature = "test-support")] + let forced_trial = self.config.force_optimized(); + #[cfg(not(feature = "test-support"))] + let forced_trial = false; + if !forced_trial && !ir::optimized_opcodes_supported(snapshot) { + self.settle_untranslatable_tier2_candidate(key); + continue; + } let observed = self .feedback .snapshot(self.clock.max(1)) @@ -3419,6 +3517,7 @@ unsafe impl rquickjs_core::runtime::JitBackend for ProductionBackend { self.optimizing_requested.remove(&key); self.optimizing_hotness.remove(&key); self.optimizing_snapshots.remove(&key); + self.tier2_untranslatable.remove(&key); self.tier2_sources.remove(&key); self.entry_tiers.remove(&key); self.coordinator.retire(key); diff --git a/jit/tests/untranslatable_settle.rs b/jit/tests/untranslatable_settle.rs new file mode 100644 index 000000000..1c3b3e6c8 --- /dev/null +++ b/jit/tests/untranslatable_settle.rs @@ -0,0 +1,86 @@ +#![cfg(all( + any(target_os = "linux", target_os = "macos"), + target_endian = "little", + any(target_arch = "x86_64", target_arch = "aarch64") +))] + +//! Automatic tiering for Tier 1 functions that Tier 2 can never translate. + +fn automatic_runtime() -> (rquickjs::Runtime, rquickjs_jit::Jit, rquickjs::Context) { + let runtime = rquickjs::Runtime::new().unwrap(); + let jit = rquickjs_jit::Jit::attach( + &runtime, + rquickjs_jit::JitConfig::builder() + .call_threshold(1) + .loop_threshold(1) + .build() + .unwrap(), + ) + .unwrap(); + let context = rquickjs::Context::full(&runtime).unwrap(); + (runtime, jit, context) +} + +#[test] +fn untranslatable_loop_settles_its_baseline_decision_once() { + // `typeof` has no optimized classification, so Tier 2 can never compile + // this function. Its baseline decision must be taken a bounded number of + // times instead of rebuilding feedback snapshots at every maintenance. + let (_runtime, jit, context) = automatic_runtime(); + context.with(|ctx| { + ctx.eval::<(), _>( + "globalThis.kinds = function(items) { let n = 0; \ + for (let i = 0; i < items.length; i++) { if (typeof items[i] === 'string') n++; } \ + return n; }; \ + globalThis.items = ['a', 1, 'b', {}, 'c'];", + ) + .unwrap(); + }); + let mut evaluations = Vec::new(); + let mut entries = Vec::new(); + for round in 0..4_000 { + let value = context.with(|ctx| ctx.eval::("kinds(items)").unwrap()); + assert_eq!(value, 3); + jit.poll(); + if round % 1_000 == 999 { + let metrics = jit.metrics(); + evaluations.push(metrics.profitability_evaluations); + entries.push(metrics.native_entries); + } + } + let metrics = jit.metrics(); + assert!(metrics.native_entries > 0, "{metrics:?}"); + assert_eq!(metrics.tier2_entries, 0, "{metrics:?}"); + assert_eq!(metrics.unsupported_opcode_failures, 0, "{metrics:?}"); + assert!( + evaluations.windows(2).all(|pair| pair[0] == pair[1]) && evaluations[0] <= 5, + "untranslatable candidate kept re-evaluating: {evaluations:?} {metrics:?}" + ); + // The settled generation runs in the interpreter with its probes off, + // exactly where a failed bounded Tier 2 trial used to leave it. + assert!( + entries.windows(2).all(|pair| pair[0] == pair[1]), + "settled generation kept entering native code: {entries:?} {metrics:?}" + ); +} + +#[test] +fn untranslatable_call_only_method_returns_to_the_interpreter() { + let (_runtime, jit, context) = automatic_runtime(); + context.with(|ctx| { + ctx.eval::<(), _>( + "function Box(v) { this.v = v; } \ + Box.prototype.twice = function() { return this.add(this.v); }; \ + Box.prototype.add = function(x) { return this.v + x; }; \ + globalThis.box = new Box(21);", + ) + .unwrap(); + }); + for _ in 0..4_000 { + let value = context.with(|ctx| ctx.eval::("box.twice()").unwrap()); + assert_eq!(value, 42); + jit.poll(); + } + let metrics = jit.metrics(); + assert!(metrics.generic_call_rejections > 0, "{metrics:?}"); +} From f2f1e2a230d33059e92cff9c81163e5be016409d Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:40:00 +0800 Subject: [PATCH 34/54] fix(jit): keep lexical TDZ exact in Tier 2 after the uninitialized join The verifier now joins Uninitialized with value kinds for loop-scoped and switch fall-through lexical bindings. Tier 1 keeps its runtime TDZ check, but Tier 2 lowers get_loc_check/put_loc_check as plain frame accesses, so such functions returned undefined (or wrote) where the interpreter throws ReferenceError. Add ir::optimized::prove_lexical_checks, a forward may-be-uninitialized dataflow over the verified CFG. Tier 2 rejects a function when a checked access may observe JS_UNINITIALIZED, and never admits put_loc_check_init. Tier 2 set_loc_uninitialized now stores the exact JS_TAG_UNINITIALIZED cell so deopt/exit frames match the interpreter. Add Tier 2 TDZ regressions (switch fall-through read/write, loop variant, straight-line TDZ) and iteration coverage for mid-loop storage transitions, arguments/subclass/array-like iterators, interrupts in the leaf loop, and OSR entry into for-of/for-in loops. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p2d-iteration.md | 30 +++++++++ jit/Cargo.toml | 4 ++ jit/src/bytecode/mod.rs | 1 + jit/src/bytecode/stack.rs | 7 +- jit/src/compiler/optimized.rs | 14 +++- jit/src/ir/optimized.rs | 103 ++++++++++++++++++++++++++++++ jit/tests/tier1_iteration.rs | 82 +++++++++++++++++++++++- jit/tests/tier2_tdz.rs | 117 ++++++++++++++++++++++++++++++++++ 8 files changed, 353 insertions(+), 5 deletions(-) create mode 100644 jit/tests/tier2_tdz.rs diff --git a/docs/roadmap/p2d-iteration.md b/docs/roadmap/p2d-iteration.md index 26afc9612..8597d573c 100644 --- a/docs/roadmap/p2d-iteration.md +++ b/docs/roadmap/p2d-iteration.md @@ -45,6 +45,36 @@ Branch `perf/p2d-iteration`, based on `82d3808`. Roadmap item P2.5 `benchmarks/run.rs`. The README matrix is not updated; that is left to the integrator. +## Review fixes (branch `perf/p2d-iteration-r1`) + +- Tier 2 TDZ regression (blocking). The widened `Uninitialized` join + admitted functions such as `switch(k){case 0: let y=1; case 1: return y}`, + and Tier 2 lowers `get_loc_check`/`put_loc_check` as plain frame accesses. + After Tier 2 warmup, `f(1)` returned `undefined` instead of throwing + ReferenceError. The straight-line case `if(k) return y; let y=...` was + also lowered without a check. The fix is `ir::optimized::prove_lexical_checks`, + a forward may-be-uninitialized dataflow over the verified CFG. Frame + entry is initialized (QuickJS sets every local to `undefined`), and only + `set_loc_uninitialized` introduces TDZ. Tier 2 now rejects a function when + a checked access may see an uninitialized binding, and rejects + `put_loc_check_init` everywhere (Tier 1 also rejects it). Tier 2 + `set_loc_uninitialized` now stores the exact `JS_TAG_UNINITIALIZED` cell + instead of `undefined`, so deopt/exit frames stay exact. Tier 1 keeps its + runtime checks, and the verifier join is unchanged. Regression tests are in + `jit/tests/tier2_tdz.rs`. +- The added coverage in `jit/tests/tier1_iteration.rs` tests fast-to-slow + storage transitions, `delete`, accessor installation, freeze and + truncation in the middle of a leaf loop. It also tests `arguments`, Array + subclasses (plain and overriding `Symbol.iterator`), and + `Array.prototype.values/keys.call(...)` iterators. Further tests cover an + uncatchable interrupt inside a leaf for-of loop, and OSR entry on the first + invocation into both a for-of loop (catch offset on the stack) and a for-in + loop (`osr_entries >= 1`). +- Recorded design deviation: the roadmap asked for a pristine-prototype guard + with deopt. This slice instead revalidates the iterator class, the exact + `next` C function, fast-array storage and bounds on every leaf step, and + every `for_of_start` goes through the exact helper. + ## Deliberately not in this slice - `iterator_next`, `iterator_call` and `iterator_get_value_done` are only diff --git a/jit/Cargo.toml b/jit/Cargo.toml index 8d2445b51..af088a41f 100644 --- a/jit/Cargo.toml +++ b/jit/Cargo.toml @@ -191,6 +191,10 @@ required-features = ["compiler", "test-support"] name = "tier1_iteration" required-features = ["compiler", "test-support"] +[[test]] +name = "tier2_tdz" +required-features = ["compiler", "test-support"] + [[test]] name = "test262" required-features = ["compiler", "test-support"] diff --git a/jit/src/bytecode/mod.rs b/jit/src/bytecode/mod.rs index 7cb5a0abd..cecfda62e 100644 --- a/jit/src/bytecode/mod.rs +++ b/jit/src/bytecode/mod.rs @@ -18,6 +18,7 @@ pub use policy::{ audited_opcode_policy_table, tier1_policy, AuditedOpcodePolicy, FallbackReason, HelperId, Tier1Policy, Tier1Rejection, GENERATED_OPCODE_COUNT, GENERATED_OPCODE_FINGERPRINT, }; +pub(crate) use stack::local_index; pub use stack::SlotKind; pub use verify::{Resource, VerifiedFunction, VerifyError, VerifyErrorKind, VerifyLimits}; diff --git a/jit/src/bytecode/stack.rs b/jit/src/bytecode/stack.rs index ebd0e0219..96c771bc1 100644 --- a/jit/src/bytecode/stack.rs +++ b/jit/src/bytecode/stack.rs @@ -88,7 +88,7 @@ fn effective_pop(instruction: &Instruction) -> usize { } } -fn local_index(instruction: &Instruction) -> Option { +pub(crate) fn local_index(instruction: &Instruction) -> Option { match instruction.opcode().format() { OperandFormat::Local => Some(instruction.operand_u16(1) as usize), OperandFormat::Local8 => Some(instruction.operand_u8(1) as usize), @@ -338,7 +338,10 @@ fn merge_state( // entry and initialized on the backedge. The interpreter cell // holds the JS_UNINITIALIZED tag in the first case, which is still // a tagged value; the join only drops the proof, and every read - // of such a binding is a runtime-checked `get_loc_check`. + // of such a binding is a `get_loc_check`. Tier 1 keeps that + // runtime check; Tier 2 lowers it as a plain load and therefore + // separately proves definite initialization before admitting it + // (`ir::optimized::prove_lexical_checks`). (SlotKind::Tagged | SlotKind::Int32 | SlotKind::Float64, SlotKind::Uninitialized) | (SlotKind::Uninitialized, SlotKind::Tagged | SlotKind::Int32 | SlotKind::Float64) => { SlotKind::Tagged diff --git a/jit/src/compiler/optimized.rs b/jit/src/compiler/optimized.rs index afc40c273..11cdf0f5c 100644 --- a/jit/src/compiler/optimized.rs +++ b/jit/src/compiler/optimized.rs @@ -2180,9 +2180,19 @@ fn lower_optimized_machine( if owned_locals[index] { emit_opt_free_local_slot(&mut builder, &env, index)?; } - opt_define(&mut builder, locals[index], undefined); + // Store the exact interpreter cell so a later + // deopt/exit resumes with the binding still in + // its TDZ. Checked reads of it are never + // admitted (see `prove_lexical_checks`). + let uninitialized = OptPair { + payload: builder.ins().iconst(payload_type, 0), + tag: builder + .ins() + .iconst(types::I64, i64::from(qjs::JS_TAG_UNINITIALIZED)), + }; + opt_define(&mut builder, locals[index], uninitialized); if !int32_loop && !defer_scalar_locals { - opt_store(&mut builder, var_buf, index, undefined); + opt_store(&mut builder, var_buf, index, uninitialized); } } "push_i8" => { diff --git a/jit/src/ir/optimized.rs b/jit/src/ir/optimized.rs index 635a2cbd3..7bc304452 100644 --- a/jit/src/ir/optimized.rs +++ b/jit/src/ir/optimized.rs @@ -249,6 +249,9 @@ impl OptimizedIr { }) { return Err(CompileFailure::UnsupportedOpcode); } + // Tier 2 lowers lexical checks as plain frame accesses, so it may only + // admit them where the binding is initialized on every path. + prove_lexical_checks(function)?; let make_guard = |pc: u32, mid_loop: bool, next_guard: &mut u32, @@ -865,6 +868,106 @@ fn optimized_block_depths( Ok(depths) } +/// Local may hold an initialized JS value on some path reaching this point. +const LEXICAL_MAY_BE_INITIALIZED: u8 = 1; +/// Local may hold `JS_UNINITIALIZED` on some path reaching this point. +const LEXICAL_MAY_BE_UNINITIALIZED: u8 = 2; + +/// Proves that every `get_loc_check`/`put_loc_check` reads a local that is +/// initialized on every incoming path, and rejects `put_loc_check_init`. +/// +/// The bytecode verifier deliberately joins "uninitialized" with a JS value +/// kind into a plain tagged cell (a lexical binding declared in a loop body or +/// reached by switch fall-through), because Tier 1 keeps the runtime TDZ +/// check. The Tier 2 lowering has no TDZ check, so without this proof the +/// interpreter's ReferenceError would be silently replaced by a load. +/// +/// QuickJS initializes every frame local to `undefined` on entry and only +/// `set_loc_uninitialized` stores `JS_UNINITIALIZED`. Nested closures cannot +/// uninitialize a frame binding, so "initialized on every path" is stable +/// across calls; the converse is not (an arrow `super()` initializes `this`), +/// which is why `put_loc_check_init` is never admitted here. +fn prove_lexical_checks(function: &VerifiedFunction) -> Result<(), CompileFailure> { + use std::collections::{BTreeMap, VecDeque}; + let cfg = function.control_flow_graph(); + let local_count = usize::from(function.snapshot().local_count()); + let instructions = function.instructions(); + let touches_lexical = instructions.iter().any(|instruction| { + matches!( + instruction.opcode().name(), + "get_loc_check" | "put_loc_check" | "put_loc_check_init" | "set_loc_uninitialized" + ) + }); + if !touches_lexical { + return Ok(()); + } + let mut entry: BTreeMap> = + BTreeMap::from([(0u32, vec![LEXICAL_MAY_BE_INITIALIZED; local_count])]); + let mut queue = VecDeque::from([0u32]); + // A block is queued once on discovery and once per join that adds a bit, + // so the fixpoint is bounded by blocks * (2 * locals + 1). + let mut budget = cfg + .blocks() + .len() + .saturating_mul(local_count.saturating_mul(2).saturating_add(1)) + .saturating_add(1); + while let Some(pc) = queue.pop_front() { + budget = budget.checked_sub(1).ok_or(CompileFailure::ResourceLimit)?; + let block = cfg.block(pc).ok_or(CompileFailure::InvalidArtifact)?; + let mut state = entry + .get(&pc) + .cloned() + .ok_or(CompileFailure::InvalidArtifact)?; + for instruction in &instructions[block.instruction_range()] { + let name = instruction.opcode().name(); + let Some(index) = crate::bytecode::local_index(instruction) else { + continue; + }; + let slot = state + .get_mut(index) + .ok_or(CompileFailure::InvalidArtifact)?; + match name { + "set_loc_uninitialized" => *slot = LEXICAL_MAY_BE_UNINITIALIZED, + "get_loc_check" | "put_loc_check" => { + if *slot & LEXICAL_MAY_BE_UNINITIALIZED != 0 { + return Err(CompileFailure::UnsupportedOpcode); + } + *slot = LEXICAL_MAY_BE_INITIALIZED; + } + "put_loc_check_init" => return Err(CompileFailure::UnsupportedOpcode), + _ if name.starts_with("put_loc") + || name.starts_with("set_loc") + || matches!(name, "inc_loc" | "dec_loc" | "add_loc") => + { + *slot = LEXICAL_MAY_BE_INITIALIZED; + } + // Plain reads and `close_loc` leave the cell unchanged. + _ => {} + } + } + for successor in block.successors() { + match entry.get_mut(successor) { + Some(existing) => { + let mut changed = false; + for (joined, incoming) in existing.iter_mut().zip(&state) { + let next = *joined | *incoming; + changed |= next != *joined; + *joined = next; + } + if changed { + queue.push_back(*successor); + } + } + None => { + entry.insert(*successor, state.clone()); + queue.push_back(*successor); + } + } + } + } + Ok(()) +} + fn classify_optimized_opcode( name: &str, ) -> Result<(ValueRepresentation, OptimizedEffect), CompileFailure> { diff --git a/jit/tests/tier1_iteration.rs b/jit/tests/tier1_iteration.rs index 4ae733ce6..f0d48632e 100644 --- a/jit/tests/tier1_iteration.rs +++ b/jit/tests/tier1_iteration.rs @@ -12,7 +12,7 @@ use rquickjs::{Context, Runtime}; use rquickjs_jit::bytecode::{FallbackReason, HelperId}; -use rquickjs_jit::test_support::{assert_tier1_rejected, differential}; +use rquickjs_jit::test_support::{assert_tier1_rejected, differential, forced_baseline}; use rquickjs_jit::{Jit, JitConfig, JitTierPolicy}; use std::time::{Duration, Instant}; @@ -337,3 +337,83 @@ fn production_tiering_enters_native_code_for_a_for_of_kernel() { assert_eq!(result, expected); assert!(jit.metrics().native_entries > 0, "{:?}", jit.metrics()); } + +#[test] +fn leaf_path_storage_transitions_and_array_likes_match_the_interpreter() { + // Fast-to-slow storage transitions inside the body must leave the leaf + // (the per-step fast_array/count revalidation) without skipping or + // repeating an element. + let log = "function f(values,mutate){let out=[];for(const value of values){out.push(String(value));mutate(values,out.length)}return out.join(',')}"; + for expression in [ + "f([1,2,3,4],(a,n)=>{if(n===2)a[10]=7})", + "f([1,2,3,4],(a,n)=>{if(n===1)delete a[2]})", + "f([1,2,3,4],(a,n)=>{if(n===2)Object.defineProperty(a,3,{get(){return 'g'}})})", + "f([1,2,3,4],(a,n)=>{if(n===2)Object.freeze(a)})", + "f([1,2,3,4],(a,n)=>{if(n===1)a.length=2})", + ] { + differential(log, expression) + .force_baseline() + .stress_gc() + .expect_executed_opcode("for_of_next") + .assert_same(); + } + // Array-likes and subclasses: each either hits the Array values leaf or + // takes the exact helper, and matches the interpreter. + for expression in [ + "(function(){return f(arguments)})(1,2,3)", + "(()=>{class A extends Array{};const a=new A();a.push(4,5,6);return f(a)})()", + "(()=>{class A extends Array{*[Symbol.iterator](){yield 40;yield 2}};return f(A.from([1,2,3]))})()", + "f({[Symbol.iterator](){return Array.prototype.values.call({length:3,0:1,1:2,2:3})}})", + "f({[Symbol.iterator](){return Array.prototype.values.call([7,8,9])}})", + "f({[Symbol.iterator](){return Array.prototype.keys.call([7,8,9])}})", + ] { + differential(SUM_OF, expression) + .force_baseline() + .stress_gc() + .expect_executed_opcode("for_of_next") + .assert_same(); + } +} + +#[test] +fn interrupt_inside_a_leaf_for_of_loop_is_uncatchable() { + forced_baseline( + "function f(values){let sum=0;for(const value of values)sum+=value;return sum} f(new Array(200000).fill(1))", + ) + .interrupt_after(3) + .assert_uncatchable_interrupt(); +} + +fn osr_first_invocation(source: &str, expected: f64) { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .tier_policy(JitTierPolicy::BaselineOnly) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + let value = context + .with(|ctx| ctx.eval::(source)) + .unwrap_or_else(|error| panic!("{error:?}; {:?}", jit.metrics())); + assert_eq!(value, expected, "{:?}", jit.metrics()); + assert!(jit.metrics().osr_entries >= 1, "{:?}", jit.metrics()); +} + +#[test] +fn long_first_for_of_invocation_osr_enters_with_the_catch_offset_on_the_stack() { + osr_first_invocation( + "const values=new Array(4000000).fill(3);function f(values){let sum=0;for(const value of values)sum+=value;return sum} f(values)", + 12_000_000.0, + ); +} + +#[test] +fn long_first_for_in_invocation_osr_enters_with_the_enumerator_on_the_stack() { + osr_first_invocation( + "const object={};for(let i=0;i<300000;i++)object['k'+i]=1;function f(object){let sum=0;for(const key in object)sum+=object[key];return sum} f(object)", + 300_000.0, + ); +} diff --git a/jit/tests/tier2_tdz.rs b/jit/tests/tier2_tdz.rs new file mode 100644 index 000000000..c5d6baba7 --- /dev/null +++ b/jit/tests/tier2_tdz.rs @@ -0,0 +1,117 @@ +//! Tier 2 must preserve lexical TDZ semantics. +//! +//! The verifier joins a lexical binding that is uninitialized on one incoming +//! edge and initialized on another into a plain tagged cell. Tier 2 lowers +//! `get_loc_check`/`put_loc_check` as plain frame accesses, so it may only +//! admit them where every path proves the binding initialized; otherwise the +//! interpreter's ReferenceError would be silently lost. +#![cfg(all( + feature = "compiler", + feature = "test-support", + target_endian = "little", + any(target_os = "linux", target_os = "macos"), + any(target_arch = "x86_64", target_arch = "aarch64") +))] + +use rquickjs::{Context, Runtime}; +use rquickjs_jit::{Jit, JitConfig}; +use std::time::{Duration, Instant}; + +fn interpreter_result(definition: &str, probe: &str) -> String { + let runtime = Runtime::new().unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| { + ctx.eval::<(), _>(definition).unwrap(); + ctx.eval::(probe).unwrap() + }) +} + +/// Warms `warm` under forced Tier 2 until the function enters Tier 2 or the +/// bounded warmup is exhausted, then evaluates `probe` and compares it with +/// the plain interpreter. Before the fix every case entered Tier 2 within a +/// few calls and the probes returned a value instead of the ReferenceError. +fn tier2_case(definition: &str, warm: &str, probe: &str) { + let expected = interpreter_result(definition, probe); + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .call_threshold(1) + .loop_threshold(1) + .force_optimized_for_test(true) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| ctx.eval::<(), _>(definition)).unwrap(); + let deadline = Instant::now() + Duration::from_secs(5); + for _ in 0..300 { + context.with(|ctx| ctx.eval::<(), _>(warm)).unwrap(); + jit.poll(); + if jit.metrics().tier2_entries > 0 || Instant::now() > deadline { + break; + } + std::thread::sleep(Duration::from_millis(1)); + } + for _ in 0..4 { + let actual = context.with(|ctx| ctx.eval::(probe)).unwrap(); + assert_eq!( + actual, + expected, + "TDZ result differs from QuickJS for {definition} / {probe}: {:?}", + jit.metrics() + ); + } +} + +const PROBE_WRAP: &str = "(function(){try{return 'value:'+String(PROBE)}catch(e){return e.constructor.name+':'+e.message}})()"; + +fn probe(expression: &str) -> String { + PROBE_WRAP.replace("PROBE", expression) +} + +#[test] +fn switch_fallthrough_tdz_read_throws_after_tier2_warmup() { + tier2_case( + "function f(k){switch(k){case 0: let y=1; case 1: return y}}", + "f(0)", + &probe("f(1)"), + ); +} + +#[test] +fn switch_fallthrough_tdz_read_in_loop_throws_after_tier2_warmup() { + tier2_case( + "function f(n,k){let s=0;for(let i=0;i Date: Sun, 27 Sep 2026 17:42:55 +0800 Subject: [PATCH 35/54] fix(jit): keep closure-creating functions out of automatic Tier 1 Review round 1 for P2a closures. Admitting fclosure let automatic tiering promote `workload` in calls-recursion-closures to the helper-bridge Tier 1, running at 0.74x the baseline and interpreter speed. Tier 2 rejects the closure opcodes, so that function could never tier up further. Under JitTierPolicy::Automatic, submit_snapshot now rejects functions that contain fclosure/fclosure8 (ClosureFrame) and leaves them in the interpreter, the same as 82d3808. Explicit BaselineOnly/Optimize policies and forced test tiers still compile them. Noisy diagnostic after the gate: calls-recursion-closures 0.99x (tied), calls-closures 1.04x, exceptions-promises-async 1.00x, call-heavy 0.99x; all checksums match the interpreter. Patch 0025: FCLOSURE, GET_VAR_REF and SET_NAME are now flagged REENTRANT, like every other allocating helper. The patch digest and the quickjs-jit-helpers.h fingerprint are re-pinned; the ABI version and the bindings are unchanged. New regression tests: the automatic gate compared with BaselineOnly; closure writes during an accessor and valueOf (GET_PROPERTY, ADD_SLOW, COMPARE_SLOW); a PUT_VAR_REF TDZ error unwinding live operands to an interpreter caller's catch; OSR into a closure-creating loop whose closure changes the captured local's type. The roadmap note now describes the gate and gives the correct TDZ and ClosureFrame wording. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p2a-closures.md | 79 ++++++++++++++-- jit/src/lib.rs | 38 ++++++++ jit/tests/tier1_closures.rs | 128 +++++++++++++++++++++++++- sys/build_support/patch.rs | 4 +- sys/patches/0025-tier1-closures.patch | 6 +- 5 files changed, 239 insertions(+), 16 deletions(-) diff --git a/docs/roadmap/p2a-closures.md b/docs/roadmap/p2a-closures.md index 6526c8b2b..8f69df201 100644 --- a/docs/roadmap/p2a-closures.md +++ b/docs/roadmap/p2a-closures.md @@ -31,7 +31,27 @@ The following remain rejected (`ClosureFrame`): The C side is `sys/patches/0025-tier1-closures.patch`: ABI 1.25 and runtime API 1.10, with five helpers appended to the end of the helper table (IDs -24-28). +24-28). FCLOSURE, GET_VAR_REF and SET_NAME carry `THROWING | ALLOCATING | +REENTRANT`, like every other allocating helper. CLOSE_LOC carries no flags, +like DUP: outside the test-only stress-GC points it neither allocates, throws +nor frees. + +## Automatic tiering gate (review round 1) + +Production automatic tiering (`JitTierPolicy::Automatic`) does **not** +compile a function that contains `fclosure`/`fclosure8`. In `submit_snapshot`, +such a function gets a Tier 1 `ClosureFrame` rejection and stays in the +interpreter (`automatic_closure_creation_unprofitable` in `jit/src/lib.rs`). +Tier 2 rejects the closure opcodes, so a closure-creating function would stay +in the helper-bridge Tier 1 for good, and that tier measured at 0.74x the +interpreter's speed on `calls-recursion-closures`. Explicit policies +(`BaselineOnly`, `Optimize`) and forced test tiers still compile these +functions, so the lowering stays exercised. Functions that only read or write +var refs (the created closures themselves) remain eligible, and the existing +profitability trial and blacklist still govern them. On `82d3808` every +`fclosure` function was already rejected, so the gate cannot regress any +function that was compiled before. Remove the gate once P1/P3 make the bridge +cheaper, or once Tier 2 admits var refs. ## Semantic safety argument @@ -45,18 +65,23 @@ API 1.10, with five helpers appended to the end of the helper table (IDs `make_*_ref` and `define_class` are all still rejected. Native arithmetic keeps its use-site tag guards, so a closure that changes a slot's type deoptimizes; the regression test for this passes. -- **TDZ proofs.** Tier 1 proves `get_loc_check`, `put_loc_check` and - `put_loc_check_init` statically. A closure can only move a lexical slot from - uninitialized to initialized (through `put_var_ref_check_init`), never back. - The proofs for `get_loc_check` and `put_loc_check` therefore stay sound: a - slot the analysis believes uninitialized can only make the function retry. - A function that combines `fclosure` with `put_loc_check_init` is rejected - with `ClosureFrame`. +- **TDZ checks.** `get_loc_check` is still checked at run time + (`lower_dup_if_refcounted(checked = true)`). `put_loc_check` relies on the + static initialization proof. A closure can only move a lexical slot from + uninitialized to initialized (through `put_var_ref_check_init`), never back, + so that proof stays sound. `put_loc_check_init` is rejected by the policy + table (`UnsupportedOpcode`) before translation. The `ClosureFrame` + rejection for `fclosure` + `put_loc_check_init` in + `BaselineIr::translate_with_policy` is therefore defense in depth. The + production path cannot reach it; only `translate_implemented_for_test` can. - **Ownership.** FCLOSURE and GET_VAR_REF write an owned value to an empty output slot. PUT_VAR_REF consumes its input on success and on a TDZ exception. After a failed PUT_VAR_REF, the exception exit publishes only the bytecode-visible stack below the consumed operand. SET_NAME borrows. Every - helper runs the stress-GC points. + helper runs the stress-GC points. A PUT_VAR_REF *invariant* reject (invalid + index or slot) does not consume its input. SET_PROPERTY and the other + consuming helpers behave the same way. The error is uncatchable and cannot + happen for compiler-validated operands. - **Other tiers.** Tier 2's opcode classifier still rejects all of these opcodes. Frame inlining rejects callees that contain `fclosure` or `close_loc`. Scalar or tagged leaf entries and effect-free inlining still @@ -65,7 +90,14 @@ API 1.10, with five helpers appended to the end of the helper table (IDs Evidence: -- `jit/tests/tier1_closures.rs` has 13 tests. +- `jit/tests/tier1_closures.rs` has 17 tests. Round 1 added these: + closure writes during GET_PROPERTY (an accessor) and ADD_SLOW/COMPARE_SLOW + (`valueOf`), each changing the slot's type; a PUT_VAR_REF TDZ error with + live operands that unwinds to an interpreter caller's `catch`; OSR into a + closure-creating loop whose closure changes the captured local from Int32 + to Float64 to string; and the automatic-tiering gate, contrasted with + `BaselineOnly`. With the reload disabled, the accessor/valueOf test and the + OSR test fail. With the gate disabled, the gate test fails. - `opcode-cases.json` has 21 new cases, each executed at its native PC under stress GC. - The helper-family cases are in `differential.rs`. @@ -89,6 +121,26 @@ Setup: Speed is base/new, so values above 1.00x mean this branch is faster. +### After the automatic-tiering gate (round 1, current) + +Same protocol and host, with 9 alternating base/new processes and 3 +interpreter-mode processes. Checksums are identical in the baseline, +branch and interpreter runs. These are noisy diagnostics with no confidence +intervals, and the publishable three-engine README matrix is still pending. + +| Workload | Base ms | New ms | Speed vs base | New vs interpreter | +| --- | ---: | ---: | ---: | ---: | +| calls-closures | 3.309 | 3.193 | 1.04x (within noise) | 1.04x | +| calls-recursion-closures | 6.453 | 6.498 | 0.99x (tied) | 0.99x | +| exceptions-promises-async | 3.670 | 3.656 | 1.00x (tied) | 0.64x (unchanged `catch` blocker) | +| call-heavy (control) | 0.060 | 0.061 | 0.99x (tied) | 21.1x | + +A separate 15-pair run of `calls-recursion-closures` gave 6.615 ms against +6.514 ms, 1.02x (tied). The 0.74x regression below is gone: `workload` stays +in the interpreter, as it does on `82d3808`. + +### Before the gate (original slice, superseded) + | Workload | Base ms | New ms | Speed vs base | New vs interpreter | | --- | ---: | ---: | ---: | ---: | | calls-closures | 3.304 | 3.275 | 1.01x (tied) | 1.00x | @@ -144,3 +196,10 @@ function workload(iterations, seed) { admitted. 5. Make the generic call bridge (P1/P3) cheaper before expecting the ≥1.5x interpreter-speed P2 acceptance on the closure workloads. +6. Remove the automatic-tiering gate for closure-creating functions once + items 1 or 5 make Tier 1 (or Tier 2) faster than the interpreter on + `calls-recursion-closures`. Re-measure with the full three-engine matrix + before removing it. +7. Publishable evidence: the integrator must refresh the README + QuickJS/Bun/quickjs-jit matrix with confidence intervals for this branch. + Per the task rules, this branch does not edit README.md. diff --git a/jit/src/lib.rs b/jit/src/lib.rs index d9a15bd96..9a9c49ab4 100644 --- a/jit/src/lib.rs +++ b/jit/src/lib.rs @@ -783,6 +783,33 @@ impl ProductionProfile { } } +/// Automatic-tiering profitability gate for closure-creating functions. +/// +/// Tier 1 lowers `fclosure`, var-ref access and `close_loc` through the +/// generic helper bridge, and a function that creates closures must also +/// reload every argument and local after each helper. Tier 2 rejects these +/// opcodes, so such a function would stay in Tier 1 for its lifetime with +/// no optimizing follow-up. Measured on `calls-recursion-closures`, that +/// Tier 1 loop runs at about 0.74x the interpreter's speed. Until the +/// bridge (roadmap P1/P3) or Tier 2 var refs make it profitable, automatic +/// tiering leaves closure-creating functions in the interpreter. Explicit +/// tier policies (`BaselineOnly`, `Optimize`) and forced test tiers still +/// compile them, so the lowering stays covered. +fn automatic_closure_creation_unprofitable( + config: &JitConfig, + verified: &bytecode::VerifiedFunction, +) -> bool { + #[cfg(feature = "test-support")] + if config.force_optimized() { + return false; + } + config.tier_policy() == JitTierPolicy::Automatic + && verified + .instructions() + .iter() + .any(|instruction| matches!(instruction.opcode().name(), "fclosure" | "fclosure8")) +} + /// Finish the bounded Tier-2 profitability trial once. Production JITs patch /// an IC/tier state after classification; repeating wide average comparisons /// on every successful optimized exit would turn policy into hot-path tax. @@ -3044,6 +3071,17 @@ unsafe impl rquickjs_core::runtime::JitBackend for ProductionBackend { self.maintenance(); return; } + if automatic_closure_creation_unprofitable(&self.config, &verified) { + self.coordinator.reject_tier1( + key, + runtime::Tier::Baseline, + bytecode::FallbackReason::ClosureFrame, + ); + self.feedback_disabled.insert(key); + self.clear_failed_request(key); + self.maintenance(); + return; + } self.ensure_artifact_environment(); let adaptive = runtime::AdaptiveInputs { bytecode_bytes: verified diff --git a/jit/tests/tier1_closures.rs b/jit/tests/tier1_closures.rs index 39cc4fba5..00b408a4b 100644 --- a/jit/tests/tier1_closures.rs +++ b/jit/tests/tier1_closures.rs @@ -168,7 +168,7 @@ fn derived_constructor_this_initialization_through_arrows_stays_rejected() { } #[test] -fn automatic_tiering_runs_closure_creating_loops_natively() { +fn baseline_only_tiering_runs_closure_creating_loops_natively() { let runtime = Runtime::new().unwrap(); let jit = Jit::attach( &runtime, @@ -202,3 +202,129 @@ fn automatic_tiering_runs_closure_creating_loops_natively() { let after = context.with(|ctx| ctx.eval::("kernel(200)").unwrap()); assert_eq!(after, expected); } + +#[test] +fn closure_writes_during_getters_and_value_of_are_reloaded() { + // The closures write the captured `c` from inside GET_PROPERTY (an + // accessor), ADD_SLOW and COMPARE_SLOW (`valueOf`), not only from CALL. + // Each write also changes the slot's type, so the native copy must be + // reloaded after every one of those helpers. + differential( + "function f(n){ let c = 0; const o = {}; Object.defineProperty(o, 'p', { get: () => { c = c + 0.5; return 1 } }); const v = { valueOf: () => { c = 'v' + c; return 2 } }; let s = 0; for (let i = 0; i < n; i++) { s += o.p; c++ } const t = s + v; const lt = s < v; c += '!'; return t + ':' + lt + ':' + c }", + "f(6)", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("get_field") + .expect_helper(HelperId::GetProperty) + .assert_same(); +} + +#[test] +fn closure_tdz_write_error_unwinds_live_operands_to_the_callers_handler() { + // `x = v` raises the TDZ ReferenceError from PUT_VAR_REF while the array, + // its `concat` method and `v` are live operands. The native frame must + // release exactly those values and let the interpreter caller catch the + // error; stress GC detects a missing or doubled release. + differential( + "var f = (function(){ if (globalThis) return function f(v){ const r = [1, 2]; return r.concat(v, (x = v, 3)).length }; let x = 1 })()", + "(() => { const out = []; for (let i = 0; i < 3; i++) { try { out.push(f({ i })) } catch (e) { out.push(e.name + ':' + e.message) } } return out.join('|') })()", + ) + .force_baseline() + .stress_gc() + .expect_executed_opcode("put_var_ref_check") + .expect_helper(HelperId::PutVarRef) + .assert_same(); +} + +fn run_until_native(jit: &Jit, context: &Context, source: &str) -> (i32, u64) { + let deadline = Instant::now() + Duration::from_secs(10); + let mut result = 0; + while Instant::now() < deadline { + result = context.with(|ctx| ctx.eval::(source).unwrap()); + jit.poll(); + if jit.metrics().native_entries > 0 { + break; + } + } + (result, jit.metrics().native_entries) +} + +#[test] +fn automatic_tiering_keeps_closure_creating_functions_in_the_interpreter() { + // Tier 2 rejects closure opcodes, so a closure-creating function would be + // stuck in the helper-bridge Tier 1, which measured slower than the + // interpreter on `calls-recursion-closures`. Production automatic tiering + // therefore leaves it in the interpreter, while an explicit Tier 1 policy + // still compiles it. The closure is never called, so no other function can + // account for a native entry. + let kernel = "globalThis.kernel = function kernel(n){ let s = 0; let keep = null; for (let i = 0; i < n; i++) { keep = () => s; s = (s + i) | 0 } return keep === null ? -1 : s }"; + let expected = (0..200).fold(0_i32, |s, i| s.wrapping_add(i)); + for (policy, native) in [ + (JitTierPolicy::Automatic, false), + (JitTierPolicy::BaselineOnly, true), + ] { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .tier_policy(policy) + .call_threshold(1) + .loop_threshold(1) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| ctx.eval::<(), _>(kernel).unwrap()); + let (result, entries) = if native { + run_until_native(&jit, &context, "kernel(200)") + } else { + let mut result = 0; + for _ in 0..200 { + result = context.with(|ctx| ctx.eval::("kernel(200)").unwrap()); + jit.poll(); + } + (result, jit.metrics().native_entries) + }; + assert_eq!(result, expected, "{policy:?}"); + let metrics = jit.metrics(); + if native { + assert!(entries > 0, "{metrics:?}"); + } else { + assert_eq!(entries, 0, "{metrics:?}"); + assert_eq!(metrics.osr_entries, 0, "{metrics:?}"); + assert!(metrics.tier1_rejections >= 1, "{metrics:?}"); + } + } +} + +#[test] +fn osr_into_a_closure_creating_loop_observes_closure_type_changes() { + // A long first invocation enters the Tier 1 loop through OSR. The + // function creates closures but captures nothing itself + // (`closure_count == 0`), so OSR is allowed. Late in the loop the closure + // turns the captured Int32 into a Float64 and then into a string; the OSR + // code must reload the slot after the call and keep the interpreter's + // result. + let source = "function f(n){ let x = 0; const bump = v => { x = v }; let s = 0; for (let i = 0; i < n; i++) { if (i === n - 1000) bump(x + 0.5); else if (i === n - 10) bump('s'); s = (s + i) | 0; x = x + 1 } return s + ':' + x } f(3000000)"; + let expected = { + let runtime = Runtime::new().unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| ctx.eval::(source).unwrap()) + }; + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .tier_policy(JitTierPolicy::BaselineOnly) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + let actual = context.with(|ctx| ctx.eval::(source).unwrap()); + assert_eq!(actual, expected); + assert!(expected.ends_with("s1111111111"), "{expected}"); + assert!(jit.metrics().osr_entries >= 1, "{:?}", jit.metrics()); +} diff --git a/sys/build_support/patch.rs b/sys/build_support/patch.rs index fa83a4bb8..9ddf774d8 100644 --- a/sys/build_support/patch.rs +++ b/sys/build_support/patch.rs @@ -21,7 +21,7 @@ const EXPECTED_PATCHES: [(&str, u64); 20] = [ ("0018-inline-frame-recovery.patch", 0xb971d61edccde229), ("0019-array-feedback.patch", 0x11c3cb13a8297dd4), ("0020-typed-array-guard.patch", 0x5d10_38a3_8bc9_9821), - ("0025-tier1-closures.patch", 0x3894_d14c_2378_dfa2), + ("0025-tier1-closures.patch", 0xe667_12ea_93d3_2533), ]; pub(crate) const BASELINE_FILES: [&str; 19] = [ "api-test.c", @@ -63,7 +63,7 @@ const PATCHED_FILE_FINGERPRINTS: [(&str, u64); 21] = [ ("quickjs-c-atomics.h", 0x490b_0f29_f631_3fc0), ("quickjs.c", 0xbad9_ea86_12ab_0678), ("quickjs-jit.h", 0xfafa_4511_21a6_ffb4), - ("quickjs-jit-helpers.h", 0x3703_05f2_7534_2073), + ("quickjs-jit-helpers.h", 0xffda_7a03_38b3_32ae), ("quickjs-opcode.h", 0x3d05_cfdf_5cf7_2930), ("quickjs.h", 0x4831_2cde_9c2f_a5ee), ]; diff --git a/sys/patches/0025-tier1-closures.patch b/sys/patches/0025-tier1-closures.patch index 0e23a9479..bf76f1651 100644 --- a/sys/patches/0025-tier1-closures.patch +++ b/sys/patches/0025-tier1-closures.patch @@ -26,16 +26,16 @@ diff --git a/quickjs-jit-helpers.h b/quickjs-jit-helpers.h - THROWING | ALLOCATING | REENTRANT | FINALIZING) + THROWING | ALLOCATING | REENTRANT | FINALIZING) \ + X(FCLOSURE, fclosure, JS_JitHelperFClosure, MAP_OUT_INDEX, 1, OWNED, \ -+ NONE, NONE, NONE, OWNED, THROWING | ALLOCATING) \ ++ NONE, NONE, NONE, OWNED, THROWING | ALLOCATING | REENTRANT) \ + X(GET_VAR_REF, get_var_ref, JS_JitHelperGetVarRef, MAP_OUT_IN_OP, 1, \ -+ OWNED, NONE, NONE, NONE, OWNED, THROWING | ALLOCATING) \ ++ OWNED, NONE, NONE, NONE, OWNED, THROWING | ALLOCATING | REENTRANT) \ + X(PUT_VAR_REF, put_var_ref, JS_JitHelperPutVarRef, MAP_OUT_IN_OP, 1, \ + CONSUMED, NONE, NONE, NONE, NONE, \ + THROWING | ALLOCATING | REENTRANT | FINALIZING) \ + X(CLOSE_LOC, close_loc, JS_JitHelperCloseLoc, MAP_IN, 0, NONE, NONE, \ + NONE, NONE, NONE, 0) \ + X(SET_NAME, set_name, JS_JitHelperSetName, MAP_OUT_INDEX, 1, BORROWED, \ -+ NONE, NONE, NONE, NONE, THROWING | ALLOCATING) ++ NONE, NONE, NONE, NONE, THROWING | ALLOCATING | REENTRANT) typedef enum JSJitHelperId { #define QJSJIT_HELPER_ID(id, field, c_name, signature, value_arity, own0, \ From a43c209177baef2073c65917eac5ebfe910d237e Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:51:59 +0800 Subject: [PATCH 36/54] refactor(sys): publish the GENERIC_OP root context out of line Move the root call context (receiver, new.target, caller argv/argc) into a no_inline wrapper around the native entry call and keep its runtime pointer at the end of the JIT fields. JS_CallInternal now only swaps the indirect entry call for the wrapper, so the interpreter loop's frame and the offsets of existing runtime fields are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) --- sys/build_support/patch.rs | 4 +- sys/patches/0026-tier1-generic-ops.patch | 75 +++++++++++++----------- 2 files changed, 42 insertions(+), 37 deletions(-) diff --git a/sys/build_support/patch.rs b/sys/build_support/patch.rs index 79a5169d3..2ba5feccd 100644 --- a/sys/build_support/patch.rs +++ b/sys/build_support/patch.rs @@ -21,7 +21,7 @@ const EXPECTED_PATCHES: [(&str, u64); 20] = [ ("0018-inline-frame-recovery.patch", 0xb971d61edccde229), ("0019-array-feedback.patch", 0x11c3cb13a8297dd4), ("0020-typed-array-guard.patch", 0x5d10_38a3_8bc9_9821), - ("0026-tier1-generic-ops.patch", 0x1933_9cb3_c0c7_5b48), + ("0026-tier1-generic-ops.patch", 0x6199_e6f2_84d4_3349), ]; pub(crate) const BASELINE_FILES: [&str; 19] = [ "api-test.c", @@ -61,7 +61,7 @@ const PATCHED_FILE_FINGERPRINTS: [(&str, u64); 21] = [ ("list.h", 0xb337_70f7_b76d_a3d8), ("quickjs-atom.h", 0x30b4_9116_b6a2_aa99), ("quickjs-c-atomics.h", 0x490b_0f29_f631_3fc0), - ("quickjs.c", 0xc5d3_8fc4_ebba_a0b4), + ("quickjs.c", 0xb06b_067c_74ac_34b2), ("quickjs-jit.h", 0xfafa_4511_21a6_ffb4), ("quickjs-jit-helpers.h", 0x6963_5583_a4a4_700f), ("quickjs-opcode.h", 0x3d05_cfdf_5cf7_2930), diff --git a/sys/patches/0026-tier1-generic-ops.patch b/sys/patches/0026-tier1-generic-ops.patch index 73506c53f..42835ab1e 100644 --- a/sys/patches/0026-tier1-generic-ops.patch +++ b/sys/patches/0026-tier1-generic-ops.patch @@ -35,16 +35,16 @@ diff --git a/quickjs-jit.h b/quickjs-jit.h diff --git a/quickjs.c b/quickjs.c --- a/quickjs.c +++ b/quickjs.c -@@ -376,6 +376,8 @@ - uint64_t jit_next_frame_cookie; - uint64_t jit_active_frame_cookie; - JSJitExecFrame *jit_active_frame; +@@ -393,6 +393,8 @@ + #endif + /* Runtime-owned root; boundary throws always duplicate this value. */ + JSValue jit_uncatchable_exception; + /* Receiver/argument context of the innermost root native entry. */ + struct QJSJitRootCall *jit_active_root; - struct JSJitInlineState *jit_inline_top; - size_t jit_inline_bytes; - uint32_t jit_inline_depth; -@@ -20098,6 +20100,285 @@ + #endif + }; + +@@ -20098,6 +20100,310 @@ return qjsjit_finish_output(frame, out); } @@ -326,37 +326,42 @@ diff --git a/quickjs.c b/quickjs.c + return qjsjit_finish_output(frame, out); + return qjsjit_stress_gc(frame); +} ++ ++/* Publish the root call context for exactly one pinned native entry. Kept ++ * out of line so the interpreter loop's frame and code layout do not change; ++ * the context lives on this wrapper's stack for the duration of the call. */ ++static no_inline JSJitExit qjsjit_call_root_entry(JSRuntime *rt, ++ JSJitExecFrame *frame, ++ JSValueConst this_obj, ++ JSValueConst new_target, ++ int argc, ++ JSValueConst *argv) ++{ ++ QJSJitRootCall root_call; ++ JSJitExit exit; ++ ++ root_call.previous = rt->jit_active_root; ++ root_call.frame = frame; ++ root_call.this_obj = this_obj; ++ root_call.new_target = new_target; ++ root_call.argv = argv; ++ root_call.argc = argc; ++ rt->jit_active_root = &root_call; ++ exit = frame->entry.entry(frame); ++ rt->jit_active_root = root_call.previous; ++ return exit; ++} + #endif #ifdef CONFIG_JIT_ABI -@@ -20746,6 +21027,7 @@ - rt->jit_backend_active_depth--; - } else { - JSJitExecFrame frame; -+ QJSJitRootCall root_call; - JSJitExit jit_exit; - const uint8_t *resume_pc; - JSValue *resume_sp; -@@ -20795,6 +21077,13 @@ - rt->jit_active_frame_cookie = frame.frame_cookie; - rt->jit_active_stack_map_count = entry.stack_map_count; - rt->jit_active_helper_abi_version = entry.helper_abi_version; -+ root_call.previous = rt->jit_active_root; -+ root_call.frame = &frame; -+ root_call.this_obj = this_obj; -+ root_call.new_target = new_target; -+ root_call.argv = argv; -+ root_call.argc = argc; -+ rt->jit_active_root = &root_call; +@@ -20798,7 +21104,8 @@ if (jit_vtable->native_enter) jit_vtable->native_enter(jit_opaque, entry_function_id, entry_generation, entry_pc_offset); -@@ -20815,6 +21104,7 @@ - rt->jit_active_frame_cookie = previous_frame_cookie; - rt->jit_active_stack_map_count = previous_stack_map_count; - rt->jit_active_helper_abi_version = previous_helper_abi_version; -+ rt->jit_active_root = root_call.previous; - if (!rt->jit_entry_cache_suppressed && - entry_pc_offset == 0 && entry_epoch != 0 && - b->jit_function_id == entry_function_id && +- jit_exit = entry.entry(&frame); ++ jit_exit = qjsjit_call_root_entry(rt, &frame, this_obj, new_target, ++ argc, argv); + if (qjsjit_inline_abort_root(&frame)) { + qjsjit_reject_exit(ctx, &frame.result, + "unconsumed inline activation at native exit"); From bbd1a9a57a7db8d5cbe4083328b43144d6b87764 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:54:25 +0800 Subject: [PATCH 37/54] fix(jit): revalidate outer hoisted tuples at inner loop polls Review fixes for P1-6 (typed-array receivers held in locals): - Every loop poll that reaches the runtime now revalidates the hoisted length/storage tuples of every loop that encloses it, not just its own header's. An outer header re-seeds its preheader tuple on each backedge, and the inner loop consumes almost every amortized countdown expiry. Before this, an interrupt handler that detached an outer receiver's buffer at an inner poll caused reads and writes through the freed backing store. The argument-receiver length hoist had the same flaw at 82d3808. Admission rejects any safepoint other than a loop poll. Lowering fails closed when an enclosing header is not seeded as planned. - Storage-only hoists need an access that dominates every loop latch, so a conditionally used receiver is no longer guarded on every loop entry. - The MAX_LIVE_SOURCES cap is checked before any access is marked as covered by a hoist. - An Object-observed get_field runs through the generic bridge. It counts as frame-preserving for poll amortization only when the function has no closure, special_object or direct eval. - Pre-existing at 82d3808: the entry and loop-header numeric-guard exits did not republish the stack top. After an owned get_field published a deeper top, the runtime rejected the exit as invalid deopt metadata, an uncatchable error. The exit now republishes an empty stack from a cold block. Lowering rejects a loop poll at a non-empty depth. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p1d-typedarray-receivers.md | 65 ++++- jit/src/compiler/optimized.rs | 96 +++++-- jit/src/compiler/optimized/array_cache.rs | 213 ++++++++++++++- jit/tests/typed_receivers.rs | 307 ++++++++++++++++++++++ 4 files changed, 651 insertions(+), 30 deletions(-) diff --git a/docs/roadmap/p1d-typedarray-receivers.md b/docs/roadmap/p1d-typedarray-receivers.md index d695f9158..e952b5df9 100644 --- a/docs/roadmap/p1d-typedarray-receivers.md +++ b/docs/roadmap/p1d-typedarray-receivers.md @@ -44,9 +44,23 @@ the loop called the poll helper on every iteration. in a loop, has no hoisted length, and passes the existing loop admission (no uncertified reentry and no write to its slot). An example is a store destination. For such a receiver: + - Some access of the receiver must dominate every latch of the loop, so + a conditionally used receiver (`if (c) out[i] = v`) is not guarded on + loop entry, where a mismatch would deopt on every entry. - The preheader runs one storage query, without the length flag. - - The query is revalidated on every poll, both amortized and regular. - Each access keeps its own index check. +- **Poll revalidation of hoisted tuples (review fix r1).** Every loop poll + that reaches the runtime revalidates the hoisted tuples (length and + storage) of *every* loop that contains it, not only those of its own + header. An outer header re-seeds its preheader tuple on each backedge + and, with amortized polling, an inner loop consumes nearly every + countdown expiry. Before this fix, an interrupt handler that detached an + outer receiver's buffer at an inner poll let the outer loop keep reading + and writing the freed backing store. Loop admission now also rejects any + safepoint other than a loop poll, and lowering fails closed when an + enclosing header's tuples are not seeded exactly as planned. The length + hoist of an argument receiver had the same flaw at `82d3808`, and this + fix covers it too. - **Leaf certification.** - Numeric `push_const`/`push_const8` sites are treated as non-reentrant. Lowering can only emit immediates for them; any other constant fails @@ -57,7 +71,11 @@ the loop called the poll helper on every iteration. natural loop is accepted. The loop-header representation guard already treats owned locals as `Any`. - The numeric proof used only for this decision keeps frame reads across - shape-guarded `get_field` and other certified leaf sites. Lowering still + shape-guarded `get_field` and other certified leaf sites. An + Object-observed `get_field` goes through the generic bridge, which can + run a getter. It counts as frame-preserving only when the function + contains no `fclosure`, `special_object` or direct `eval`: the only ways + script could write this frame. Lowering still uses the original proof. All exits remain exact deoptimizations at the original bytecode. @@ -91,6 +109,30 @@ All exits remain exact deoptimizations at the original bytecode. - rejection of rebound, aliased and argument-aliased locals; - a storage-only hoist for a store destination. +## Review fixes (r1) + +- The hoist-cap check now runs before any access is marked covered. When a + loop exceeds `MAX_LIVE_SOURCES` receivers, the extra receivers keep every + index guard; the rest of the plan is not discarded. +- A pre-existing bug, reproduced at `82d3808`, is fixed. The numeric guard + at the loop header and at entry exited without republishing the stack top. + After `const out = o.out` had published a one-slot top through the owned + property bridge, a failing header guard (for example on a `null` local) + produced an exit that the runtime rejected as "invalid native + deoptimization metadata". That is an uncatchable error for the script. + The guard exit now republishes an empty stack. Lowering rejects a loop + poll at a non-empty depth. +- New tests, in `typed_receivers.rs` unless noted: + - `outer_loop_hoists_are_revalidated_by_inner_loop_polls`: the + reviewer's store/UAF case with a detached destination or source, plus + the argument-receiver length hoist. + - `conditional_and_wide_receiver_loops_match_the_interpreter`: no + deopts on entry for an unused conditional destination, and more than + four receivers. + - `loop_header_guard_exit_republishes_an_empty_stack`. + - `array_cache` unit tests for enclosing-poll revalidation, conditional + storage hoists and the live-tuple cap. + ## Noisy diagnostic (not publishable evidence) This was measured on a shared, loaded 24-core x86_64 host. Each row is 7 @@ -114,6 +156,25 @@ quickjs-bitops and host-compute with 7 runs were bimodal ties. The publishable three-engine matrix, with Bun and confidence intervals, is still pending and belongs to the integrator. +### After the r1 review fixes + +The r1 run used the same method, with 9 rotated-order rounds. `prev` is +the implementer's commit `e243f15`. + +| Workload | 82d3808 ms | e243f15 ms | r1 ms | Speed vs 82d3808 | Speed vs e243f15 | +| --- | ---: | ---: | ---: | ---: | ---: | +| float64array-traversal | 1.0940 | 0.1642 | 0.1690 | 6.47x | 0.97x | +| arrays-typed | 2.9781 | 1.9978 | 1.9953 | 1.49x | 1.00x | +| int32array-traversal (control) | 0.0498 | 0.0497 | 0.0500 | 1.00x | 0.99x | +| float64-dense (control) | 0.1291 | 0.1305 | 0.1288 | 1.00x | 1.01x | + +float64array-traversal is sensitive to code layout. Across r1 builds that +differ only in cold deopt blocks, it measured between 0.93x and 1.19x the +speed of e243f15. One early build made the entry and header numeric-guard +exit use the entry `stack_base` SSA value. That build was about 7% slower +on int32array-traversal, because it kept the value live across the loop. +The exit now reloads the base from the frame and is marked cold. + ## Remaining work - float64array-traversal takes about 7.9 ns per iteration against about diff --git a/jit/src/compiler/optimized.rs b/jit/src/compiler/optimized.rs index 3d5e02dcc..8e116d4fa 100644 --- a/jit/src/compiler/optimized.rs +++ b/jit/src/compiler/optimized.rs @@ -1738,12 +1738,25 @@ fn lower_optimized_machine( // Array feedback only proposes candidates. Poll amortization is enabled // after the concrete plan has proved that every heap site on the native // path is a guarded leaf-or-exit operation. - // Every lowered get_field is a shape-guarded leaf-or-exit access (a - // missing guard fails compilation), as are the certified array, property - // key and constant sites. None of them runs script or writes the frame on - // its continuing path, so an argument or local read after one of them - // keeps its entry-guarded representation. This proof only decides poll + // The certified array, property key and constant sites, and primitive + // get_field sites (shape-guarded leaf-or-exit accesses; a missing guard + // fails compilation) run no script on their continuing path. + // Object-observed get_field sites are different: they lower through the + // generic GET_PROPERTY bridge, which can run a getter. Script can only + // write this frame's arguments or locals through a closure, a mapped + // `arguments` object or direct eval, so those sites preserve the frame + // only while the function contains none of the opcodes that create them + // (Tier 1 policy already rejects them; this is checked again here). + // An argument or local read after a frame-preserving site keeps its + // entry-guarded representation. This proof only decides poll // amortization; lowering keeps using `scalar_numeric`. + let script_cannot_write_frame = !ir.nodes().iter().any(|node| { + matches!(node.kind(), crate::ir::OptimizedNodeKind::Bytecode { opcode } + if matches!( + opcode.as_ref(), + "fclosure" | "fclosure8" | "special_object" | "eval" | "apply_eval" + )) + }); let leaf_frame_preserving = |id: u32| { array_plan.guarded_leaf(id) || property_plan.access(id).is_some() @@ -1757,7 +1770,15 @@ fn lower_optimized_machine( )) || (matches!(node.kind(), crate::ir::OptimizedNodeKind::Bytecode { opcode } if opcode.as_ref() == "get_field") - && specialization.properties.contains_key(&node.pc())) + && specialization + .properties + .get(&node.pc()) + .is_some_and(|observations| { + script_cannot_write_frame + || observations.iter().all(|observation| { + observation.value() != crate::runtime::ObservedType::Object + }) + })) }) }; let amortized_numeric = if side_path.is_none() && !int32_loop { @@ -2215,6 +2236,25 @@ fn lower_optimized_machine( } match node.kind() { crate::ir::OptimizedNodeKind::GuardNumeric { guard, mid_loop } => { + if *mid_loop && depth != 0 { + // The poll's numeric-guard exit republishes an + // empty operand stack only. + return Err(CompileFailure::UnsupportedOpcode); + } + // A poll that reaches the runtime can run the interrupt + // handler, which may detach or replace storage of any + // enclosing loop's hoisted receiver. Revalidate every + // enclosing header's tuples, not only this block's: + // an outer header re-seeds its tuples on each backedge. + let poll_revalidations = if *mid_loop { + opt_poll_revalidations( + &array_plan, + node.id(), + &hoisted_element_sources, + )? + } else { + Vec::new() + }; if *mid_loop && amortized_poll { emit_opt_amortized_poll( &mut builder, @@ -2264,11 +2304,7 @@ fn lower_optimized_machine( node.pc(), *guard, )?; - for source in hoisted_element_sources - .get(&block.start_pc()) - .into_iter() - .flatten() - { + for source in &poll_revalidations { emit_opt_packed_loop_revalidate( builder, &env, @@ -2324,11 +2360,7 @@ fn lower_optimized_machine( // amortized cold edge. A typed store can keep its // numeric locals unproven, selecting this path; // cached pointers/length must still be revalidated. - for source in hoisted_element_sources - .get(&block.start_pc()) - .into_iter() - .flatten() - { + for source in &poll_revalidations { emit_opt_packed_loop_revalidate( &mut builder, &env, @@ -5990,6 +6022,26 @@ fn emit_opt_array_loop_hoists( Ok(()) } +/// Every hoisted tuple that poll `node` must revalidate, from all loops that +/// contain it. Fails closed when an enclosing header's tuples are not seeded +/// exactly as planned (for example when lowering order would let a poll run +/// before the enclosing preheader emitted its guards). +fn opt_poll_revalidations( + plan: &array_cache::ArrayPlan, + node: u32, + sources: &std::collections::BTreeMap>, +) -> Result, CompileFailure> { + let mut revalidations = Vec::new(); + for &header in plan.revalidating_headers(node) { + let seeded = sources.get(&header).map_or(&[][..], Vec::as_slice); + if seeded.len() != plan.hoisted_sources_at(header) { + return Err(CompileFailure::InvalidArtifact); + } + revalidations.extend_from_slice(seeded); + } + Ok(revalidations) +} + fn opt_push_hoisted_source( sources: &mut std::collections::BTreeMap>, header: u32, @@ -8917,6 +8969,7 @@ fn emit_opt_numeric_guard( alternate_seen = builder.ins().bor(alternate_seen, float); } let deopt = builder.create_block(); + builder.set_cold_block(deopt); if side_path.is_some_and(|profile| profile.observed() == crate::runtime::ObservedType::Float64) { let side_check = builder.create_block(); @@ -8943,6 +8996,17 @@ fn emit_opt_numeric_guard( builder.ins().brif(numeric, pass, &[], deopt, &[]); } builder.switch_to_block(deopt); + // Entry and loop-header guards resume with an empty operand stack (the + // lowering rejects a loop poll at a non-empty depth). An earlier helper + // call may have published a deeper stack top (for example the owned + // `get_field` that initialized a receiver local before the loop); the + // resume shape is validated against it, so republish the empty top. + // Reload the base here: keeping the entry SSA value live into this cold + // exit would extend it across the whole loop body. + let stack_base = builder + .ins() + .load(pointer_type, MemFlags::new(), frame, layout.stack_base); + opt_set_stack_top(builder, frame, stack_base, 0, pointer_type, layout); let start = builder .ins() .load(pointer_type, MemFlags::new(), frame, layout.bytecode_start); diff --git a/jit/src/compiler/optimized/array_cache.rs b/jit/src/compiler/optimized/array_cache.rs index c76730070..14c83a992 100644 --- a/jit/src/compiler/optimized/array_cache.rs +++ b/jit/src/compiler/optimized/array_cache.rs @@ -80,6 +80,8 @@ pub(super) struct ArrayPlan { accesses: Vec>, hoists: Vec, storage_hoists: Vec, + /// Loop headers whose hoisted tuples a poll node must revalidate. + poll_revalidations: Vec<(u32, Vec)>, invalidations: Vec, } @@ -311,6 +313,18 @@ impl ArrayPlan { return false; } } + // Lowering revalidates hoisted tuples only at loop polls + // (every such poll of an enclosing loop, see + // `revalidating_headers`). Any other safepoint could observe + // an interrupt without that revalidation. + if graph.effect_for_node(node.id()) == ScalarHeapEffect::Safepoint + && !matches!( + node.kind(), + crate::ir::OptimizedNodeKind::GuardNumeric { mid_loop: true, .. } + ) + { + return false; + } if graph.effect_for_node(node.id()) != ScalarHeapEffect::Reentrant || non_reentrant(node.id()) || proposed_nodes[node.id() as usize] @@ -390,6 +404,16 @@ impl ArrayPlan { if lengths.next().is_some() { continue; } + // Check the live-tuple cap before marking any access as + // covered: a capped loop keeps every per-access index guard. + if hoists + .iter() + .filter(|hoist: &&ArrayLoopHoist| hoist.header == natural_loop.header()) + .count() + >= Self::MAX_LIVE_SOURCES + { + continue; + } let mut covered_any = false; for (node, access) in accesses.iter_mut().enumerate() { let Some(access) = access.as_mut() else { @@ -408,13 +432,7 @@ impl ArrayPlan { covered_any = true; } } - if covered_any - && hoists - .iter() - .filter(|hoist: &&ArrayLoopHoist| hoist.header == natural_loop.header()) - .count() - < Self::MAX_LIVE_SOURCES - { + if covered_any { hoists.push(ArrayLoopHoist { candidate, preheader, @@ -446,11 +464,31 @@ impl ArrayPlan { { continue; } - let used = accesses.iter().flatten().any(|access| { - access.candidate == candidate - && access.loop_header == header - && access.access != ArrayAccess::Length - }); + // Guarding the receiver on the loop-entry edge is profitable + // only when every completed iteration performs one of its + // accesses anyway. A conditionally used receiver (for example + // `if (out) out[i] = v` with `out === null`) would otherwise + // deopt on every loop entry although the loop never touches it. + let latches: Vec = ir + .blocks() + .iter() + .filter(|block| { + natural_loop.contains_block(block.start_pc()) + && block.successors().contains(&header) + }) + .map(|block| block.start_pc()) + .collect(); + let used = !latches.is_empty() + && accesses.iter().enumerate().any(|(node, access)| { + access.as_ref().is_some_and(|access| { + access.candidate == candidate + && access.loop_header == header + && access.access != ArrayAccess::Length + && loops.block_for_node(node as u32).is_some_and(|block| { + latches.iter().all(|&latch| loops.dominates(block, latch)) + }) + }) + }); let live = hoists.iter().filter(|hoist| hoist.header == header).count() + storage_hoists .iter() @@ -465,6 +503,36 @@ impl ArrayPlan { } } } + // Every loop poll that can run the interrupt handler must revalidate + // the hoisted tuples of each loop enclosing it, not only those of its + // own header: an inner loop's poll can detach an outer receiver's + // buffer, and the outer header re-seeds its preheader tuple on every + // backedge while its own amortized countdown rarely expires. + let mut poll_revalidations = Vec::new(); + if !hoists.is_empty() || !storage_hoists.is_empty() { + for node in nodes { + if graph.effect_for_node(node.id()) != ScalarHeapEffect::Safepoint { + continue; + } + let Some(block) = loops.block_for_node(node.id()) else { + continue; + }; + let mut headers = Vec::new(); + for natural_loop in loops.loops() { + let header = natural_loop.header(); + if natural_loop.contains_block(block) + && (hoists.iter().any(|hoist| hoist.header == header) + || storage_hoists.iter().any(|hoist| hoist.header == header)) + && !headers.contains(&header) + { + headers.push(header); + } + } + if !headers.is_empty() { + poll_revalidations.push((node.id(), headers)); + } + } + } let mut invalidations = Vec::with_capacity(nodes.len()); for node in nodes { let planned = accesses @@ -495,6 +563,7 @@ impl ArrayPlan { accesses, hoists, storage_hoists, + poll_revalidations, invalidations, } } @@ -521,6 +590,27 @@ impl ArrayPlan { pub(super) fn storage_hoists(&self) -> &[ArrayStorageHoist] { &self.storage_hoists } + /// Headers of every loop containing poll `node` that owns hoisted + /// metadata. The poll's observable (runtime-calling) path must revalidate + /// all of their tuples before native code continues. + pub(super) fn revalidating_headers(&self, node: u32) -> &[u32] { + self.poll_revalidations + .iter() + .find(|(poll, _)| *poll == node) + .map_or(&[], |(_, headers)| headers.as_slice()) + } + /// Hoisted tuples (length and storage-only) seeded at `header`. + pub(super) fn hoisted_sources_at(&self, header: u32) -> usize { + self.hoists + .iter() + .filter(|hoist| hoist.header == header) + .count() + + self + .storage_hoists + .iter() + .filter(|hoist| hoist.header == header) + .count() + } pub(super) fn invalidates_before(&self, node: u32) -> bool { self.invalidations .get(node as usize) @@ -916,6 +1006,105 @@ mod tests { } } + #[test] + fn inner_loop_polls_revalidate_outer_hoisted_tuples() { + let (ir, feedback) = fixture( + "(function(o,m){const f=o.f;let t=0;for(let i=0;i = ir + .nodes() + .iter() + .filter(|node| { + matches!( + node.kind(), + OptimizedNodeKind::GuardNumeric { mid_loop: true, .. } + ) + }) + .collect(); + assert_eq!(polls.len(), 2, "outer and inner loop polls expected"); + // Both the outer header poll and the nested inner poll must + // revalidate the outer receiver's tuple. + for poll in polls { + assert!( + plan.revalidating_headers(poll.id()).contains(&hoist.header), + "poll {} does not revalidate the outer tuple: {plan:#?}", + poll.id() + ); + } + assert_eq!(plan.hoisted_sources_at(hoist.header), 1); + } + + #[test] + fn conditionally_used_typed_receiver_gets_no_storage_hoist() { + let (ir, feedback) = fixture( + "(function(o,n){const f=o.f;for(let i=0;i = plan + .hoists() + .iter() + .map(|hoist| hoist.candidate) + .chain(plan.storage_hoists().iter().map(|hoist| hoist.candidate)) + .collect(); + let mut unhoisted = 0; + for node in ir.nodes() { + let Some(access) = plan.access(node.id()) else { + continue; + }; + if access.bounds_covered_by_hoist { + assert!( + plan.hoists() + .iter() + .any(|hoist| hoist.candidate == access.candidate), + "{plan:#?}" + ); + } + if access.access != ArrayAccess::Length && !hoisted.contains(&access.candidate) { + assert!(access.requires_index_guard, "{plan:#?}"); + unhoisted += 1; + } + } + assert!( + unhoisted > 0, + "six receivers exceed the live-tuple cap: {plan:#?}" + ); + } + #[test] fn store_only_typed_receiver_gets_one_storage_hoist() { let (ir, feedback) = fixture( diff --git a/jit/tests/typed_receivers.rs b/jit/tests/typed_receivers.rs index 1d64aeaac..acc9a22ec 100644 --- a/jit/tests/typed_receivers.rs +++ b/jit/tests/typed_receivers.rs @@ -488,3 +488,310 @@ fn packed_local_receivers_match_the_same_version_interpreter() { assert_eq!(after.native_entries, after.native_exits); assert!(after.tier2_entries > before.tier2_entries); } + +const NESTED: &str = r#" + function storeNested(n, m, buffers) { + const ints = buffers.ints; + const floats = buffers.floats; + let s = 0, t = 0; + for (let i = 0; i < ints.length; i++) { + floats[i] = ints[i] * 0.25 + 0.5; + s += floats[i]; + for (let j = 0; j < m; j++) t = (t + 1) | 0; + } + return s; + } + function sumNested(f, m) { + let s = 0, t = 0; + for (let i = 0; i < f.length; i++) { + s += f[i]; + for (let j = 0; j < m; j++) t = (t + 1) | 0; + } + return s; + } + function warmBuffers(n) { + return {ints: new Int32Array(n).fill(4), floats: new Float64Array(n).fill(1.25)}; + } +"#; + +/// A hoisted outer-loop tuple must be revalidated by every poll that can run +/// the interrupt handler, including the poll of a nested inner loop. With an +/// amortized countdown the inner loop consumes nearly every expiry, so a +/// revalidation only at the outer header would miss a detach performed there. +#[test] +fn outer_loop_hoists_are_revalidated_by_inner_loop_polls() { + for (warm, call, target, detached_destination) in [ + ( + "storeNested(0, 2, warmBuffers(16))", + "storeNested(0, 2000, big)", + "big.floats.buffer", + true, + ), + ( + "storeNested(0, 2, warmBuffers(16))", + "storeNested(0, 2000, big)", + "big.ints.buffer", + false, + ), + ( + "sumNested(warmBuffers(16).floats, 2)", + "sumNested(big.floats, 2000)", + "big.floats.buffer", + false, + ), + ] { + nested_detach_case(warm, call, target, detached_destination); + } +} + +fn nested_detach_case(warm: &str, call: &str, target: &str, detached_destination: bool) { + use rquickjs::qjs; + use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Arc, + }; + const N: usize = 4000; + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .call_threshold(2) + .loop_threshold(4) + .force_optimized_for_test(true) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| ctx.eval::<(), _>(NESTED)).unwrap(); + let deadline = Instant::now() + Duration::from_secs(60); + loop { + let before = jit.metrics(); + context.with(|ctx| ctx.eval::(warm)).unwrap(); + let after = jit.metrics(); + if after.tier2_entries > before.tier2_entries + && after.deopts == before.deopts + && after.pending_worker_jobs == 0 + { + break; + } + jit.poll(); + assert!( + Instant::now() < deadline, + "{call}: nested traversal never reached Tier2: {after:?}" + ); + std::thread::sleep(Duration::from_millis(1)); + } + let (raw_ctx, raw_buffer) = context.with(|ctx| { + ctx.eval::<(), _>(format!( + "globalThis.big = {{ints: new Int32Array({N}).fill(4), floats: new Float64Array({N}).fill(1.25)}}; globalThis.detachTarget = {target};" + )) + .unwrap(); + let buffer: rquickjs::Object = ctx.globals().get("detachTarget").unwrap(); + ( + ctx.as_raw().as_ptr() as usize, + unsafe { qjs::JS_VALUE_GET_PTR(buffer.as_value().as_raw()) } as usize, + ) + }); + let calls = Arc::new(AtomicUsize::new(0)); + let observed = calls.clone(); + // Detach on the third observable poll, well inside the outer loop. The + // globally rooted buffer stays live; detaching runs no script. + runtime.set_interrupt_handler(Some(Box::new(move || unsafe { + if observed.fetch_add(1, Ordering::SeqCst) == 2 { + let ctx = raw_ctx as *mut qjs::JSContext; + let buffer = qjs::JS_MKPTR(qjs::JS_TAG_OBJECT, raw_buffer as *mut core::ffi::c_void); + qjs::JS_DetachArrayBuffer(ctx, buffer); + } + false + }))); + let before = jit.metrics(); + let result = context.with(|ctx| ctx.eval::(call)); + runtime.set_interrupt_handler(None); + let result = result.unwrap(); + let after = jit.metrics(); + assert!( + calls.load(Ordering::SeqCst) > 2, + "{call}: the interrupt handler never detached" + ); + assert!( + after.tier2_entries > before.tier2_entries, + "{call}: {after:?}" + ); + assert_eq!(after.native_entries, after.native_exits); + let full = N as f64 * 1.25; + if detached_destination { + // Every read of a detached destination is undefined. + assert!( + result.is_nan(), + "{call}: stale destination storage used after detach: {result}" + ); + } else { + // A detached source reports length 0, ending the outer loop early. + assert!( + result.is_finite() && result < full, + "{call}: stale source length used after detach: {result}" + ); + } + assert!( + after.deopts > before.deopts, + "{call}: native code never observed the detach: {after:?}" + ); +} + +const CONDITIONAL_AND_WIDE: &str = r#" + function maybeStore(n, o) { + const src = o.src; + const out = o.out; + let s = 0; + for (let i = 0; i < src.length; i++) { + const v = src[i] * 0.5; + if (n > 0) out[i] = v; + s += v; + } + return s; + } + function fiveReceivers(n, o) { + const a = o.a, b = o.b, c = o.c, d = o.d, e = o.e; + let s = 0; + for (let i = 0; i < a.length; i++) { + s += a[i] + b[i] + c[i] + d[i]; + e[i] = s; + } + return s; + } + function views(n) { + const f = (k) => new Float64Array(n).fill(k); + return {src: new Int32Array(n).fill(3), out: new Float64Array(n), + a: f(1), b: f(2), c: f(3), d: f(4), e: new Float64Array(n)}; + } +"#; + +/// Storage hoists guard their receiver on the loop-entry edge only when every +/// iteration touches it, so an unused conditional destination stays native. More +/// than `MAX_LIVE_SOURCES` receivers keep exact per-access guards. +#[test] +fn conditional_and_wide_receiver_loops_match_the_interpreter() { + const CASES: &str = r#"JSON.stringify([ + maybeStore(1, views(8)), maybeStore(0, {src: views(8).src, out: null}), + maybeStore(1, {src: views(8).src, out: new Float64Array(3)}), + maybeStore(1, {src: views(8).src, out: new Int32Array(8)}), + fiveReceivers(0, views(8)), + (() => { const v = views(8); v.c = new Float64Array(2); return fiveReceivers(0, v); })(), + (() => { const v = views(8); v.e = v.a; return fiveReceivers(0, v); })() + ])"#; + let expected = evaluate_without_jit(CONDITIONAL_AND_WIDE, CASES); + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .call_threshold(2) + .loop_threshold(4) + .force_optimized_for_test(true) + .stress_gc(true) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context + .with(|ctx| ctx.eval::<(), _>(CONDITIONAL_AND_WIDE)) + .unwrap(); + context + .with(|ctx| ctx.eval::<(), _>("globalThis.W = views(64)")) + .unwrap(); + for warm in ["maybeStore(1, W)", "fiveReceivers(0, W)"] { + let deadline = Instant::now() + Duration::from_secs(60); + loop { + let before = jit.metrics(); + context.with(|ctx| ctx.eval::(warm)).unwrap(); + let after = jit.metrics(); + if after.tier2_entries > before.tier2_entries + && after.deopts == before.deopts + && after.pending_worker_jobs == 0 + { + break; + } + jit.poll(); + assert!( + Instant::now() < deadline, + "{warm} never reached Tier2: {after:?}" + ); + std::thread::sleep(Duration::from_millis(1)); + } + } + // A destination that is not a typed array but is never stored to (the + // store is conditional) must not deopt on every loop entry. + context + .with(|ctx| ctx.eval::<(), _>("globalThis.plainOut = {src: views(64).src, out: {}}")) + .unwrap(); + let before = jit.metrics(); + for _ in 0..16 { + context + .with(|ctx| ctx.eval::("maybeStore(0, plainOut)")) + .unwrap(); + } + let after = jit.metrics(); + assert_eq!(after.tier2_entries - before.tier2_entries, 16, "{after:?}"); + assert_eq!(after.deopts, before.deopts, "{after:?}"); + let actual = context.with(|ctx| ctx.eval::(CASES)).unwrap(); + assert_eq!(actual, expected); + let after = jit.metrics(); + assert_eq!(after.native_entries, after.native_exits); +} + +/// Pre-existing at 82d3808: `const out = o.out` publishes a one-slot stack +/// top through the owned property bridge. When the loop-header numeric guard +/// then rejected the non-numeric `null` local, its exit resumed at the header +/// (stack depth 0) with that stale top, and the runtime rejected the exit as +/// "invalid native deoptimization metadata" (an uncatchable error). +#[test] +fn loop_header_guard_exit_republishes_an_empty_stack() { + const SOURCE: &str = r#" + function nullLocal(n, o) { + const src = o.src; + const out = o.out; + let s = 0; + for (let i = 0; i < src.length; i++) { + const v = src[i] * 0.5; + if (out) out[i] = v; + s += v; + } + return s; + } + globalThis.nullOut = {src: new Int32Array(64).fill(3), out: null}; + "#; + let expected = evaluate_without_jit(SOURCE, "String(nullLocal(0, nullOut))"); + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .call_threshold(2) + .loop_threshold(4) + .force_optimized_for_test(true) + .stress_gc(true) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| ctx.eval::<(), _>(SOURCE)).unwrap(); + let deadline = Instant::now() + Duration::from_secs(60); + let mut entered = 0; + while entered < 8 { + let before = jit.metrics(); + let actual = context + .with(|ctx| ctx.eval::("String(nullLocal(0, nullOut))")) + .unwrap(); + assert_eq!(actual, expected); + let after = jit.metrics(); + entered += after.tier2_entries - before.tier2_entries; + assert_eq!(after.native_entries, after.native_exits); + jit.poll(); + assert!( + Instant::now() < deadline, + "nullLocal never reached Tier2: {after:?}" + ); + std::thread::sleep(Duration::from_millis(1)); + } +} From 00285d858a54c7df72d3f8ead9c53f3a91ad3cfa Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 17:59:21 +0800 Subject: [PATCH 38/54] bench: add methods-dynamic workload and record the P2b slice methods-dynamic exercises prototype methods using `this`, typeof/in/ instanceof dispatch, object destructuring, `**` and compound element updates, the paths admitted by GENERIC_OP that no matrix scenario covered. It is registered as a non-designated matrix workload with its suites.lock hash; its checksum matches QuickJS and Bun. docs/roadmap/p2b-misc-ops.md lists what is admitted and what still falls back and why, the maintenance interaction found while measuring, noisy diagnostic numbers, and the remaining work. The README matrix is left to the integrator's publishable three-engine run. Co-Authored-By: Claude Opus 5.5 (1M context) --- benchmarks/README.md | 1 + benchmarks/run.rs | 8 ++ benchmarks/scripts/methods-dynamic.js | 24 +++++ benchmarks/suites.lock | 9 ++ docs/roadmap/p2b-misc-ops.md | 141 ++++++++++++++++++++++++++ 5 files changed, 183 insertions(+) create mode 100644 benchmarks/scripts/methods-dynamic.js create mode 100644 docs/roadmap/p2b-misc-ops.md diff --git a/benchmarks/README.md b/benchmarks/README.md index d6dd3e5fc..f07d3072f 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -165,6 +165,7 @@ fallback-only result as proof that a JIT tier supports the feature. | `json-codec` | Repeated nested JSON encoding and decoding | | `map-set-bigint` | Map and Set mutation/iteration plus bounded BigInt arithmetic | | `exceptions-promises-async` | Throw/catch, Promise jobs, async functions, and continuations | +| `methods-dynamic` | Prototype methods using `this`, `typeof`/`in`/`instanceof` dispatch, object destructuring, `**`, and compound element updates | Select the matrix with `JIT_BENCH_WORKLOADS` for smoke or publishable runs; the sample/warmup/window rules above remain unchanged. diff --git a/benchmarks/run.rs b/benchmarks/run.rs index 20fe492ac..b1e025738 100644 --- a/benchmarks/run.rs +++ b/benchmarks/run.rs @@ -250,6 +250,13 @@ const WORKLOADS: &[Workload] = &[ designated: false, file: "exceptions-promises-async.js", }, + Workload { + name: "methods-dynamic", + suite: "rquickjs-jit matrix", + group: "methods-dynamic", + designated: false, + file: "methods-dynamic.js", + }, ]; #[derive(Clone, Debug, Deserialize, Serialize)] @@ -1467,6 +1474,7 @@ mod tests { ("json-codec", "json-codec"), ("map-set-bigint", "map-set-bigint"), ("exceptions-promises-async", "exceptions-promises-async"), + ("methods-dynamic", "methods-dynamic"), ] { let workload = WORKLOADS.iter().find(|w| w.name == name).unwrap(); assert_eq!(workload.group, group); diff --git a/benchmarks/scripts/methods-dynamic.js b/benchmarks/scripts/methods-dynamic.js new file mode 100644 index 000000000..ea95134e3 --- /dev/null +++ b/benchmarks/scripts/methods-dynamic.js @@ -0,0 +1,24 @@ +function Vec(x, y) { this.x = x; this.y = y; } +Vec.prototype.dot = function (other) { return this.x * other.x + this.y * other.y; }; +Vec.prototype.scale = function (k) { return new Vec(this.x * k, this.y * k); }; +function describe(value) { + if (typeof value === 'undefined') return 0; + if (typeof value === 'function') return 1; + if (value instanceof Vec) return 2; + if (typeof value !== 'object') return 3; + if ('kind' in value) return 4 + value.kind; + return 7; +} +function tally(counts, key) { counts[key] += 1; } +function workload(iterations, seed) { + const values = [undefined, describe, new Vec(1, 2), { kind: 1 }, { other: 1 }, 'text', { kind: 2 }]; + const counts = [0, 0, 0, 0, 0, 0, 0, 0]; + let sum = 0; + for (let i = 0; i < iterations; i++) { + const v = new Vec(i & 7, seed + 1); + const { x, y } = v.scale(2); + sum += v.dot({ x, y }) + (x ** 2) % 5; + tally(counts, describe(values[i % values.length])); + } + return sum + ':' + counts.join(','); +} diff --git a/benchmarks/suites.lock b/benchmarks/suites.lock index 8a1f6cadb..4ae0ae758 100644 --- a/benchmarks/suites.lock +++ b/benchmarks/suites.lock @@ -186,3 +186,12 @@ subdirectory = "benchmarks/scripts/generic-call-fallback.js" license = "workspace" upstream_sha256 = "not-applicable-local-workload" adapted_sha256 = "af1c8b8d3eb97968c1924927bdb5744c1678a2c9d82dec6f8b3f772451f12596" + +[[suite]] +name = "rquickjs-jit-matrix-methods-dynamic" +repository = "local" +commit = "local" +subdirectory = "benchmarks/scripts/methods-dynamic.js" +license = "workspace" +upstream_sha256 = "not-applicable-local-workload" +adapted_sha256 = "f358b173509f6aaa9f4f74b7d39b1e23391af24222829507e7da8a6347a4734d" diff --git a/docs/roadmap/p2b-misc-ops.md b/docs/roadmap/p2b-misc-ops.md new file mode 100644 index 000000000..91965a350 --- /dev/null +++ b/docs/roadmap/p2b-misc-ops.md @@ -0,0 +1,141 @@ +# P2b: common rejected opcodes (first slice) + +Branch `perf/p2b-misc-ops`, based on `82d3808`. Roadmap item: P2 coverage for +`push_this`, `special_object`, global variable access, the `typeof` family, +`in`, `instanceof`, `pow`, `delete`, `to_object` and the stack shuffles. See +[the roadmap](../PERFORMANCE_ROADMAP_V8_BUN.md), sections 4 (B3) and 6 (P2). + +## What Tier 1 now admits + +| Opcode(s) | Lowering | +| --- | --- | +| `push_this`, `special_object` (arguments, this function, `new.target`, home object, variable object, null prototype) | `GENERIC_OP` helper, reading the root entry's call context | +| `get_var_undef`, `delete_var`, `put_var` | `GENERIC_OP` helper with a validated atom | +| `typeof` | `GENERIC_OP` helper | +| `typeof_is_undefined`, `typeof_is_function` | native answer for every non-negative tag (not an object, not reference counted); `GENERIC_OP` otherwise | +| `to_object` | `GENERIC_OP` helper (object destructuring) | +| `to_propkey2` | native no-op when the receiver is object-coercible and the key is Int32/string/symbol; `GENERIC_OP` otherwise (compound element updates, computed destructuring) | +| `in`, `instanceof`, `delete` | `GENERIC_OP` helper; the boolean result feeds branches without `ToBool` | +| `pow` | existing exact `BINARY_ARITH_SLOW` helper, which already accepted `OP_pow` | +| `dup1`, `dup2` | existing `DUP` lowering | +| `perm4`, `swap2`, `rot3l`, `rot3r` | existing native stack permutations | + +`GENERIC_OP` is one append-only helper: ID 24, ABI 1.25, runtime API 1.10, +patch `0026-tier1-generic-ops.patch`. Each opcode runs the same C routine the +interpreter uses, under the same slot and ownership contract as the other +exact slow-path helpers. Operands stay in C-visible frame slots. Consumed slots +are cleared on success and on a language exception, and the exception depth is +the pre-instruction depth. A no_inline wrapper around each root native entry +publishes the receiver, `new.target` and the caller's `argc`/`argv`. +`push_this` and `special_object` reject any other frame, such as an inline +view. + +Tier 2 is unchanged. Its translator has no classification for these opcodes, +so functions that contain them stay at Tier 1; the stack shuffles were already +classified. The frame-inline and effect-free inliners still resume at, or +reject, these instructions. + +## Still rejected, and why + +| Opcode(s) | Reason | +| --- | --- | +| `special_object` MAPPED_ARGUMENTS (`ExtendedFrame`) | Sloppy `arguments` aliases argument slots through variable references, but Tier 1 keeps arguments in registers. This needs a frame-aliasing model, shared with the closure work. | +| `special_object` IMPORT_META | Needs module-only state. | +| `set_name`, `set_name_computed` | They only name a fresh closure or class, so they cannot be reached until `fclosure` and `define_class` are admitted. Adding them to `GENERIC_OP` takes a few lines: `JS_DefineObjectName[Computed]`, with the value kept in place. | +| `put_var_init`, `define_var`, `check_define_var`, `define_func` | Only occur in script or eval code, which is never compiled as a Tier 1 function. The differential manifest cannot execute them natively. | +| `nop`, `nip1` | Never emitted after label resolution. | +| `dup3`, `insert4`, `perm5`, `rot4l`, `rot5l` | Only follow `super` references or async iteration. | + +Each admitted opcode has an `opcode-cases.json` entry. The entry covers the +interpreter differential, execution at the native PC, the exception path and +stress GC. The focused tests in `jit/tests/tier1_generic_ops.rs` add more +cases. + +## Runtime interaction: untranslatable Tier 1 functions + +Admitting these opcodes turns many small methods and host-style helpers into +Tier 1 functions that Tier 2 can never translate. At `82d3808` such a function +followed one of two paths: + +* It stayed in Tier 1 forever when a property, direct-call or numeric gate + deferred its Tier 2 trial, and it rebuilt a full feedback snapshot at every + maintenance. `objects-polymorphic` is on this path; `object` has no Tier 2 + classification. +* Otherwise it queued Tier 2 attempts that could only fail. When the + optimizing tier was blacklisted, `record_hot` disabled the generation's + probes and the function returned to the interpreter. `json-codec` is on this + path. + +With the new opcodes, the new `methods-dynamic` kernel took the first path and +ran 5.4x slower than `82d3808`, where its functions never entered native code. +Most samples were in maintenance feedback snapshots, followed by helper-PC +validation (B1). + +`ProductionBackend::maintenance` now settles such a candidate once it has +served 16 baseline executions. Before that point, deferring it costs one map +lookup. It keeps the call-only interpreter demotion and that demotion's +metric. Otherwise it moves directly to the terminal state that the failed +bounded trials used to reach: baseline demoted, probes off. Forced Tier 2 test +runs keep the old path. As a result, steady-state automatic tiering runs these +functions in the interpreter, exactly as at `82d3808` or better. The new +coverage matters for warmup and OSR, for forced Tier 1, and for later Tier 2 +work. A real gain from Tier 1 alone depends on P1: O(1) helper-PC validation +(B1) and cheaper per-call bookkeeping (B2), followed by a baseline-versus- +interpreter measurement. + +## Noisy diagnostics (not publishable evidence) + +The host is shared by 16 agents on 24 cores. Each value is the median over +alternating runs of `median(protocol.warmup_batch_ns[-16:])` from +`jit-bench worker --mode automatic`. The baseline binary was built at +`82d3808`; the branch binary was built at `a43c209`. Speed is baseline time +divided by branch time, so values above 1.00x are faster. Every run's checksum +matched the interpreter. For `methods-dynamic`, Bun's checksum also matched +QuickJS. + +| Workload | Runs | 82d3808 ms | branch ms | Speed | +| --- | ---: | ---: | ---: | ---: | +| objects-polymorphic | 7 | 24.222 | 6.697 | 3.62x (now at interpreter speed, 6.18 ms) | +| methods-dynamic (new) | 7 | 13.006 | 13.215 | 0.98x | +| adversarial | 7 | 0.954 | 0.949 | 1.01x | +| strings-json | 7 | 2.002 | 1.998 | 1.00x | +| strings-regexp | 7 | 19.888 | 20.158 | 0.99x | +| json-codec | 7 | 78.838 | 78.893 | 1.00x | +| mixed-quotes | 7 | 2.487 | 2.525 | 0.98x | +| generic-call-fallback | 7 | 9.196 | 9.033 | 1.02x | +| map-set-bigint | 7 | 17.228 | 17.703 | 0.97x | +| property-heavy | 7 | 0.0988 | 0.0979 | 1.01x | +| call-heavy | 7 | 0.0600 | 0.0599 | 1.00x | + +`map-set-bigint` never enters native code. In pure interpreter mode, with no +JIT attached, it measured 0.98x across 9 runs, while `calls-closures`, +`fibonacci-recursive`, `collections` and `objects-polymorphic` measured +0.99x to 1.00x in the same mode. This looks like C code layout rather than a +JIT effect, and the published matrix should confirm it. A 7-run pass over the +other 20 matrix scenarios, using the binary built just before the +out-of-line wrapper, measured between 0.99x and 1.02x. `numeric` was bimodal +in both binaries (0.029 ms or 0.055 ms per batch), as it was at `82d3808`. +Bun runs `methods-dynamic` in 0.150 ms, against 12.85 ms for the QuickJS +interpreter. + +Progression of `methods-dynamic` during development: + +| Variant | Speed vs 82d3808 | +| --- | ---: | +| Opcodes admitted, maintenance unchanged | 0.18x | +| Plus settling with the measured profitability trial kept | 0.26x to 0.40x (bimodal); `json-codec` fell to 0.84x | +| Final (terminal settle after 16 baseline executions) | 0.98x | + +## Remaining work + +1. Land P1 (O(1) helper-PC validation and C-side tier state), then replace the + terminal settle with a measured baseline-versus-interpreter decision, so + profitable Tier 1-only code can stay native. +2. Mapped `arguments`, once frame slots can be aliased (closure frames). +3. `set_name` and `set_name_computed`, together with `fclosure` and + `define_class`. +4. Script-level compilation, or OSR in global code, for the global + declaration family. +5. Tier 2 classifications for `typeof_is_*`, `in`, `instanceof`, + `to_propkey2`, `to_object` and `push_this`, so hot loops with these checks + can reach the optimizer. From bd5c287906618b68c38a3ad9984c1756e05a3cd1 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 18:18:49 +0800 Subject: [PATCH 39/54] fix(jit): settle only functions that use newly admitted Tier 1 opcodes The untranslatable settle sent every Tier 1 function that Tier 2 cannot classify back to the interpreter after 16 baseline executions. That also demoted loops that were already Tier 1 functions at 82d3808 (string and object literals, `new`, regexp literals), so a plain numeric loop ending in `'r' + sum` ran at about 0.77x the 82d3808 speed. Settle a candidate only when it contains an opcode this slice newly admitted to Tier 1 (SETTLED_GENERIC_TIER1_OPCODES); such a function ran in the interpreter at 82d3808. Every other function keeps the unchanged optimizing scan. Fold the duplicated call-only retire branches into one. Add a regression test that untranslatable loops without the new opcodes keep entering native code, plus GENERIC_OP coverage for put_var through a re-entrant, throwing global setter under stress GC and for unmapped `arguments` in sloppy functions with non-simple parameters. Co-Authored-By: Claude Opus 5.5 (1M context) --- jit/src/lib.rs | 93 ++++++++++++++++++++---------- jit/tests/tier1_generic_ops.rs | 40 +++++++++++++ jit/tests/untranslatable_settle.rs | 56 ++++++++++++++++++ 3 files changed, 159 insertions(+), 30 deletions(-) diff --git a/jit/src/lib.rs b/jit/src/lib.rs index 01157699c..c4c48399f 100644 --- a/jit/src/lib.rs +++ b/jit/src/lib.rs @@ -602,6 +602,47 @@ const HOT_MAINTENANCE_INTERVAL: u32 = 64; #[cfg(all(feature = "compiler", not(target_family = "wasm")))] const UNTRANSLATABLE_SETTLE_EXECUTIONS: u64 = 16; +/// Opcodes that Tier 1 admitted only through the P2b slice (82d3808 rejected +/// each of them). A function containing one ran in the interpreter at +/// 82d3808; the untranslatable settle returns it there. Every other function +/// keeps the unchanged optimizing scan, so Tier 1-only code that already ran +/// natively (string/object literals, `new`, ...) keeps its baseline. +#[cfg(all(feature = "compiler", not(target_family = "wasm")))] +const SETTLED_GENERIC_TIER1_OPCODES: &[&str] = &[ + "push_this", + "special_object", + "dup1", + "dup2", + "perm4", + "swap2", + "rot3l", + "rot3r", + "get_var_undef", + "put_var", + "to_object", + "to_propkey2", + "typeof", + "delete", + "delete_var", + "pow", + "instanceof", + "in", + "typeof_is_undefined", + "typeof_is_function", +]; + +/// Whether the optimizing scan settles this generation into the interpreter +/// instead of evaluating a Tier 2 trial: Tier 2 can never translate it and +/// it contains an opcode from `SETTLED_GENERIC_TIER1_OPCODES`. +#[cfg(all(feature = "compiler", not(target_family = "wasm")))] +fn settles_untranslatable_candidate(snapshot: &bytecode::VerifiedFunction) -> bool { + snapshot + .instructions() + .iter() + .any(|instruction| SETTLED_GENERIC_TIER1_OPCODES.contains(&instruction.opcode().name())) + && !ir::optimized_opcodes_supported(snapshot) +} + /// Feedback pc used for return types observed at native `DONE` exits. #[cfg(all(feature = "compiler", not(target_family = "wasm")))] const NATIVE_RETURN_FEEDBACK_PC: u32 = u32::MAX; @@ -2188,17 +2229,17 @@ impl ProductionBackend { } } - /// Tier 2 translates the whole function, so a generation containing an - /// opcode without an optimized classification can never compile there. - /// The optimizing scan would otherwise revisit it at every maintenance, - /// rebuilding feedback snapshots whenever a property/call/numeric gate - /// defers the trial. Every other route ends in the interpreter: call-only - /// functions are demoted directly, five measured rejections demote the - /// baseline, and an approved trial fails its bounded Tier 2 attempts until - /// the optimizing tier is blacklisted, which disables the generation's - /// probes. Reach that terminal state at the first scan, without the - /// failing compilations or the slower baseline in between. The call-only - /// rule still runs first so its metric keeps its meaning. + /// Settles a generation that Tier 2 can never translate and whose Tier 1 + /// code exists only because of the opcodes in + /// `SETTLED_GENERIC_TIER1_OPCODES` (see `settles_untranslatable_candidate`). + /// Such a function ran in the interpreter at 82d3808, and its baseline is + /// dominated by exact slow-path helper calls: the optimizing scan would + /// otherwise revisit it at every maintenance, rebuilding feedback + /// snapshots whenever a property/call/numeric gate defers the trial, and + /// the new `methods-dynamic` kernel ran 5.4x slower that way. After a short + /// baseline warmup it returns to the interpreter with its probes off, the + /// state it had at 82d3808. The call-only rule runs first so that its + /// metric keeps its meaning. fn settle_untranslatable_tier2_candidate(&mut self, key: runtime::FunctionKey) { let Some(snapshot) = self.optimizing_snapshots.get(&key) else { return; @@ -2207,9 +2248,9 @@ impl ProductionBackend { self.retire_untranslatable_to_interpreter(key); return; } - // Like the bounded Tier 2 retries this replaces, let the installed - // baseline serve a short warmup (and any OSR it enabled) first. The - // deferral is a map lookup; no feedback snapshot is built. + // Let the installed baseline serve a short warmup (and any OSR it + // enabled) first. The deferral is a map lookup; no feedback snapshot + // is built. if self .execution_profiles .get(&key) @@ -2222,10 +2263,11 @@ impl ProductionBackend { .control_flow_graph() .is_loop_header(block.start_pc()) }); - let call_pcs = snapshot - .instructions() - .iter() - .filter(|instruction| { + // Both outcomes retire to the interpreter; only the call-only + // demotion metric differs. + if !has_loop { + let observed = self.feedback.snapshot(self.clock.max(1)); + let generic_call = snapshot.instructions().iter().any(|instruction| { matches!( instruction.opcode().name(), "call" @@ -2237,15 +2279,8 @@ impl ProductionBackend { | "tail_call" | "tail_call_method" | "call_constructor" - ) - }) - .map(|instruction| instruction.pc()) - .collect::>(); - if !has_loop && !call_pcs.is_empty() { - let observed = self.feedback.snapshot(self.clock.max(1)); - let generic_call = call_pcs.iter().any(|pc| { - !observed - .call_specialization_at(key, *pc) + ) && !observed + .call_specialization_at(key, instruction.pc()) .is_some_and(|call| { call.callee() != key && self.coordinator.direct_call_ready(&call) }) @@ -2253,8 +2288,6 @@ impl ProductionBackend { if generic_call { self.cold_metrics_dirty = true; self.generic_call_rejections = self.generic_call_rejections.saturating_add(1); - self.retire_untranslatable_to_interpreter(key); - return; } } self.retire_untranslatable_to_interpreter(key); @@ -2369,7 +2402,7 @@ impl ProductionBackend { let forced_trial = self.config.force_optimized(); #[cfg(not(feature = "test-support"))] let forced_trial = false; - if !forced_trial && !ir::optimized_opcodes_supported(snapshot) { + if !forced_trial && settles_untranslatable_candidate(snapshot) { self.settle_untranslatable_tier2_candidate(key); continue; } diff --git a/jit/tests/tier1_generic_ops.rs b/jit/tests/tier1_generic_ops.rs index 13364888f..2f2197bfb 100644 --- a/jit/tests/tier1_generic_ops.rs +++ b/jit/tests/tier1_generic_ops.rs @@ -135,6 +135,46 @@ fn global_reads_writes_and_deletes_match_the_interpreter() { ); } +#[test] +fn put_var_through_a_reentrant_global_setter() { + // The setter re-enters JS, allocates enough to trigger GC under stress + // mode, and throws on odd values while the stored value is still owned + // by the helper's operand slot. + generic( + "globalThis.setterLog = []; \ + Object.defineProperty(globalThis, 'accessorGlobal', { configurable: true, \ + get(){ return setterLog.length }, \ + set(v){ const junk = []; for (let i = 0; i < 64; i++) junk.push({i, s: 'x' + i}); \ + if (typeof v === 'number' && v % 2) throw new TypeError('odd ' + v); \ + setterLog.push(v && v.tag ? v.tag : v); } }); \ + function f(v){ accessorGlobal = v; return accessorGlobal }", + &format!( + "[{}, {}, {}, {}, setterLog.join('|')]", + guarded("f(2)"), + guarded("f(3)"), + guarded("f({tag:'obj'})"), + guarded("f('s')") + ), + "put_var", + ); +} + +#[test] +fn sloppy_non_simple_parameters_use_unmapped_arguments() { + // Sloppy functions with default or destructured parameters get an + // unmapped arguments object, which Tier 1 admits. + generic( + "function f(a = 'd'){ arguments[0] = 'changed'; return a + ':' + arguments.length + ':' + arguments[0] }", + "[f(), f(5), f(5, 6), f(undefined, 7)]", + "special_object", + ); + generic( + "function f({x}){ return x + ':' + arguments.length + ':' + typeof arguments[0] }", + &format!("[f({{x:1}}), f({{x:2}}, 3), {}]", guarded("f()")), + "special_object", + ); +} + #[test] fn typeof_family_matches_every_tag() { let values = "[1, 1.5, -0, NaN, 's', '', null, undefined, {}, [], f, class {}, 1n, 2n**80n, Symbol(), true, new Proxy(function(){}, {}), new Proxy({}, {})]"; diff --git a/jit/tests/untranslatable_settle.rs b/jit/tests/untranslatable_settle.rs index 1c3b3e6c8..0002da249 100644 --- a/jit/tests/untranslatable_settle.rs +++ b/jit/tests/untranslatable_settle.rs @@ -84,3 +84,59 @@ fn untranslatable_call_only_method_returns_to_the_interpreter() { let metrics = jit.metrics(); assert!(metrics.generic_call_rejections > 0, "{metrics:?}"); } + +/// Runs `call` repeatedly under automatic tiering and returns the +/// `native_entries` metric sampled after each 1,000-round window. +fn native_entry_windows(source: &str, call: &str, expected: &str) -> Vec { + let (_runtime, jit, context) = automatic_runtime(); + context.with(|ctx| ctx.eval::<(), _>(source).unwrap()); + let mut entries = Vec::new(); + for round in 0..4_000 { + let value = context.with(|ctx| ctx.eval::(call).unwrap()); + assert_eq!(value, expected); + jit.poll(); + if round % 1_000 == 999 { + entries.push(jit.metrics().native_entries); + } + } + entries +} + +#[test] +fn untranslatable_loop_without_newly_admitted_opcodes_keeps_its_baseline() { + // Regression: the settle must not demote Tier 1 code that already ran + // natively at 82d3808. String literals (`push_atom_value`), object + // literals (`object`) and `new` (`call_constructor`) have no Tier 2 + // classification, but these loops were Tier 1 functions before the + // GENERIC_OP opcodes were admitted and must keep entering native code. + let cases = [ + ( + "globalThis.f = function(n, z) { let sum = z; \ + for (let i = z; i < n; i++) sum = sum + i * 0.5; return 'r' + sum; };", + "f(64, 0)", + "r1008", + ), + ( + "globalThis.f = function(n) { let sum = 0; \ + for (let i = 0; i < n; i++) { const o = { v: i }; sum = sum + o.v; } \ + return '' + sum; };", + "f(64)", + "2016", + ), + ( + "function P(v) { this.v = v; } \ + globalThis.f = function(n) { let sum = 0; \ + for (let i = 0; i < n; i++) { const p = new P(i); sum = sum + p.v; } \ + return '' + sum; };", + "f(64)", + "2016", + ), + ]; + for (source, call, expected) in cases { + let entries = native_entry_windows(source, call, expected); + assert!( + entries.windows(2).all(|pair| pair[1] > pair[0]), + "Tier 1 function stopped entering native code: {source} {entries:?}" + ); + } +} From 0960ad86538d011f737ba4ed74b000c2fca7a745 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 18:21:11 +0800 Subject: [PATCH 40/54] fix(jit): keep frame inlining at sites Tier 2 cannot link directly The planner skipped frame inlining at every call PC with a direct target. The Tier 2 caller only takes a linked entry for a receiver-free call whose function operand and HeapRef arguments come from frame reads, so an effectful callee reached by global name, closure variable, method, or with a HeapRef argument from a global fell back to the generic CALL. Predict Tier 2 usability conservatively from the call's basic block (tier2_direct_call_site_usable) and skip frame inlining only there, or when pure inlining owns the site. Gate the scheduler's wait on an in-flight callee compile on the same prediction. Add production-tiering regression tests for each site shape. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p3b-effectful-direct.md | 34 ++++++-- jit/src/bytecode/mod.rs | 2 + jit/src/bytecode/stack.rs | 2 +- jit/src/lib.rs | 6 +- jit/src/runtime/coordinator.rs | 114 +++++++++++++++++++++++++-- jit/src/runtime/mod.rs | 2 + jit/tests/tagged_call_link.rs | 97 +++++++++++++++++++++++ 7 files changed, 243 insertions(+), 14 deletions(-) diff --git a/docs/roadmap/p3b-effectful-direct.md b/docs/roadmap/p3b-effectful-direct.md index 0e7047860..4e0445849 100644 --- a/docs/roadmap/p3b-effectful-direct.md +++ b/docs/roadmap/p3b-effectful-direct.md @@ -67,10 +67,23 @@ paths, plus the test hooks). ### Coordinator preference `Coordinator` previously skipped frame-inline planning at a call PC only when -the direct target also carried an inline snapshot. A direct target now always -wins over frame inlining at that PC: the linked entry needs no shadow frame, -and the old preference would otherwise route `incrementAndRecord` through -`JS_JitInlineEnter`, as it did before this slice. +the direct target also carried an inline snapshot. It now also skips it when +the Tier 2 caller can actually take the linked entry there: the linked entry +needs no shadow frame, and the old preference would otherwise route +`incrementAndRecord` through `JS_JitInlineEnter`, as it did before this slice. + +`emit_opt_specialized_call` takes a linked entry only for a receiver-free +`call` whose function operand and every HeapRef argument come from frame +reads. `tier2_direct_call_site_usable` (coordinator.rs) predicts that +conservatively by replaying the call's own basic block: only `get_arg*`, +`get_loc*` and `get_loc_check` pushes that no later frame write, closure +variable write or nested call could have changed count as frame reads. Every +other site (callee by global name, closure variable, `obj.fn(...)`, or a HeapRef +argument read from a global) keeps frame inlining; an earlier revision of this +branch sent those sites to the generic CALL (review r1, regression tests +`production_effectful_callee_*` in `jit/tests/tagged_call_link.rs`). The +prediction can under-approximate, which only keeps the pre-linking frame +inline. ### Tier 2 queueing order (`jit/src/lib.rs`) @@ -86,7 +99,9 @@ of 7 processes stayed on the old ~9 ms path. Two changes fix this: - a caller whose loop call site is frame-inline-ready but not yet direct-ready waits while the callee's optimizing compile is `Queued`, `Compiling` or `Ready` and the callee has a bounded signature. The wait - ends when that compile installs or fails, so it is bounded. + ends when that compile installs or fails, so it is bounded. The wait also + requires `tier2_direct_call_site_usable`, because other sites are + frame-inlined anyway. After this, 12 of 12 processes linked the edge. @@ -130,6 +145,15 @@ After this, 12 of 12 processes linked the edge. plus alias checks), heap-valued fields and heap results (needs ownership transfer), locals, loops (needs polls), nested calls, `this`, Float64 comparisons, and polymorphic property sites. +- Known divergence (shared with the earlier effect-free leaf): the linked leaf + skips the interpreter's call-entry stack-overflow and interrupt checks. An + effectful leaf can therefore commit its field writes where the interpreter + would have thrown `RangeError` at call entry. The leaf is bounded and + loop-free, so the window is one call deep. +- The scheduler wait still covers callees that can never publish a linked + entry (they use `this`, locals, loops or calls); gating it on the linked-leaf + admission prepass would avoid delaying those callers. Transitive chains + (A -> B -> leaf) and mutual recursion under the wait are not tested. - A real native calling convention (P3 proper) with lazily materialized frames is still required for recursion and general callees. This slice avoids any frame by keeping the callee transactional. diff --git a/jit/src/bytecode/mod.rs b/jit/src/bytecode/mod.rs index 7cb5a0abd..68c3959bc 100644 --- a/jit/src/bytecode/mod.rs +++ b/jit/src/bytecode/mod.rs @@ -18,6 +18,8 @@ pub use policy::{ audited_opcode_policy_table, tier1_policy, AuditedOpcodePolicy, FallbackReason, HelperId, Tier1Policy, Tier1Rejection, GENERATED_OPCODE_COUNT, GENERATED_OPCODE_FINGERPRINT, }; +#[cfg(all(feature = "compiler", not(target_family = "wasm")))] +pub(crate) use stack::effective_pop; pub use stack::SlotKind; pub use verify::{Resource, VerifiedFunction, VerifyError, VerifyErrorKind, VerifyLimits}; diff --git a/jit/src/bytecode/stack.rs b/jit/src/bytecode/stack.rs index a2582c9ed..c6b14e941 100644 --- a/jit/src/bytecode/stack.rs +++ b/jit/src/bytecode/stack.rs @@ -71,7 +71,7 @@ pub(crate) struct StateProof { pub(crate) visited: BTreeSet, } -fn effective_pop(instruction: &Instruction) -> usize { +pub(crate) fn effective_pop(instruction: &Instruction) -> usize { let base = instruction.opcode().n_pop() as usize; match instruction.opcode().format() { OperandFormat::NPop | OperandFormat::NPopU16 => { diff --git a/jit/src/lib.rs b/jit/src/lib.rs index e98f74fb8..a7b3652c3 100644 --- a/jit/src/lib.rs +++ b/jit/src/lib.rs @@ -2341,7 +2341,11 @@ impl ProductionBackend { | runtime::CompileState::Compiling(_) | runtime::CompileState::Ready(_) ); - if callee_optimizing && observed.bounded_specialization(call.callee()).is_some() + // Only a site the Tier 2 caller can link is worth the + // wait; any other site is frame-inlined regardless. + if callee_optimizing + && observed.bounded_specialization(call.callee()).is_some() + && runtime::tier2_direct_call_site_usable(snapshot, instruction.pc(), &call) { direct_call_pending = true; } diff --git a/jit/src/runtime/coordinator.rs b/jit/src/runtime/coordinator.rs index df309a38b..c66517116 100644 --- a/jit/src/runtime/coordinator.rs +++ b/jit/src/runtime/coordinator.rs @@ -340,6 +340,99 @@ fn artifact_matches_direct_call(artifact: &CompiledArtifact, call: &CallSpeciali }) } +/// Conservatively predicts whether a Tier 2 caller can take the linked direct +/// entry at `pc`. `emit_opt_specialized_call` uses it only for a receiver-free +/// call whose function operand and every HeapRef argument are frame reads +/// (`get_arg*` / `get_loc*` / `get_loc_check`). Any other site (callee by +/// global name, closure variable, method receiver, or a HeapRef argument from a +/// global or an expression) falls back to the generic CALL there, so the +/// planner must keep frame inlining it. +/// +/// The prediction replays only the call's own basic block: values from +/// earlier blocks, rearranged by stack shuffles, or read before any later +/// frame write or nested call are unknown. It may under-approximate (the site +/// then stays frame-inlined, the pre-linking behavior) but never admits a site +/// whose operands the block does not visibly read from frame slots. +#[cfg(all(feature = "compiler", not(target_family = "wasm")))] +pub(crate) fn tier2_direct_call_site_usable( + body: &VerifiedFunction, + pc: u32, + call: &CallSpecializationKey, +) -> bool { + let instructions = body.instructions(); + let Some(index) = instructions + .iter() + .position(|instruction| instruction.pc() == pc) + else { + return false; + }; + let instruction = &instructions[index]; + if !matches!( + instruction.opcode().name(), + "call" | "call0" | "call1" | "call2" | "call3" + ) { + return false; + } + let argc = crate::bytecode::effective_pop(instruction).saturating_sub(1); + if argc != call.arguments().len() { + return false; + } + let Some(block) = body + .control_flow_graph() + .blocks() + .iter() + .find(|block| block.instruction_range().contains(&index)) + else { + return false; + }; + // `true` marks a value pushed by a frame read that is still current. + let mut stack: Vec = Vec::new(); + for instruction in &instructions[block.instruction_range().start..index] { + let name = instruction.opcode().name(); + let frame_read = name != "get_loc0_loc1" + && (name.starts_with("get_arg") + || name.starts_with("get_loc") + && (name == "get_loc_check" + || name + .strip_prefix("get_loc") + .is_some_and(|rest| rest.bytes().all(|byte| byte.is_ascii_digit())))); + if frame_read { + stack.push(true); + continue; + } + // A frame write, closure-variable write, or nested call may change a + // slot after it was read; forget every earlier frame read. + if name.contains("_loc") + || name.contains("_arg") + || name.contains("var_ref") + || name.starts_with("call") + || name.starts_with("tail_call") + || name.starts_with("apply") + || name.starts_with("eval") + { + stack.iter_mut().for_each(|known| *known = false); + } + let pop = crate::bytecode::effective_pop(instruction); + stack.truncate(stack.len().saturating_sub(pop)); + stack.extend(std::iter::repeat_n( + false, + usize::from(instruction.opcode().n_push()), + )); + } + let Some(function_index) = stack.len().checked_sub(argc + 1) else { + return false; + }; + stack[function_index] + && call + .arguments() + .iter() + .enumerate() + .all(|(argument, representation)| { + *representation != FeedbackRepresentation::HeapRef + || stack[function_index + 1 + argument] + }) +} + impl CompiledCallTarget { pub const fn identity(&self) -> ArtifactVersionIdentity { self.identity @@ -1434,13 +1527,20 @@ impl Coordinator { if budget.exhausted() { break; } - // A linked direct entry is a transaction with an exact - // pre-effect miss to the generic CALL. It needs no shadow - // frame, so it is always preferred over frame inlining. - if direct_call_targets - .iter() - .any(|target| target.pc() == instruction.pc()) - { + // Pure inlining owns its site. A linked direct entry is a + // transaction with an exact pre-effect miss to the generic + // CALL and needs no shadow frame, so it is preferred over + // frame inlining, but only where the Tier 2 caller can + // take it. Elsewhere the site would reach the generic CALL. + if direct_call_targets.iter().any(|target| { + target.pc() == instruction.pc() + && (target.inline_snapshot().is_some() + || tier2_direct_call_site_usable( + &snapshot, + instruction.pc(), + target.call(), + )) + }) { continue; } if let Some(target) = self.retain_frame_inline_target( diff --git a/jit/src/runtime/mod.rs b/jit/src/runtime/mod.rs index d40bab251..ea69fffe6 100644 --- a/jit/src/runtime/mod.rs +++ b/jit/src/runtime/mod.rs @@ -15,6 +15,8 @@ pub use array_feedback::{ ArrayAccess, ArrayFeedbackSnapshot, ArrayFeedbackTable, ArrayHazards, ArrayMode, }; pub use background::{BackgroundCompiler, BackgroundCompilerError}; +#[cfg(all(feature = "compiler", not(target_family = "wasm")))] +pub(crate) use coordinator::tier2_direct_call_site_usable; #[cfg(feature = "test-support")] pub use coordinator::CompletionDisposition; pub use coordinator::{ diff --git a/jit/tests/tagged_call_link.rs b/jit/tests/tagged_call_link.rs index 32a36ff38..24cae7e9b 100644 --- a/jit/tests/tagged_call_link.rs +++ b/jit/tests/tagged_call_link.rs @@ -416,3 +416,100 @@ fn tier2_effectful_link_commits_field_stores_and_misses_exactly() { fn tier2_effectful_link_is_exact_under_gc_stress() { exercise_effectful(JitTierPolicy::Automatic, true, true); } + +/// Production tiering. `setup` defines `caller`; `run` calls it. Warm until +/// Tier 2 is entered and one complete run makes no generic CALL. For sites +/// the Tier 2 caller cannot link (callee by global name, closure variable, +/// method receiver, or a HeapRef argument read from a global) this proves the +/// effectful callee stays frame-inlined instead of reaching the generic CALL. +fn exercise_production_site(setup: &str, run: &str, expected: &str) { + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach(&runtime, JitConfig::builder().build().unwrap()).unwrap(); + let context = Context::full(&runtime).unwrap(); + let source = format!( + "function leaf(v,e,s){{s.calls=s.calls+1; if(e) return v+1; return v}} + globalThis.st={{calls:0}}; + {setup}" + ); + context.with(|ctx| ctx.eval::<(), _>(source)).unwrap(); + let eval = |source: &str| -> String { + context.with(|ctx| { + ctx.eval::(format!("String({source})")) + .unwrap_or_else(|error| { + let exception = ctx.catch(); + panic!("{source}: {error:?}: {exception:?}") + }) + }) + }; + let deadline = Instant::now() + Duration::from_secs(60); + loop { + let before = call_count(&context); + assert_eq!(eval(run), expected); + let calls = call_count(&context) - before; + jit.poll(); + let metrics = jit.metrics(); + if metrics.tier2_entries > 0 && metrics.pending_worker_jobs == 0 && calls == 0 { + break; + } + assert!( + Instant::now() < deadline, + "{run}: {calls} generic CALLs per run after warmup: {metrics:?}" + ); + } + // Still exact after the edge is compiled: effects and results. + assert_eq!(eval(run), expected); + assert_eq!(eval("st.calls"), "2000"); + let metrics = jit.metrics(); + assert_eq!(metrics.native_entries, metrics.native_exits); +} + +#[test] +fn production_effectful_callee_by_global_name_avoids_generic_call() { + exercise_production_site( + "function caller(n,s){s.calls=0; let v=0; \ + for(let i=0;i Date: Sun, 27 Sep 2026 18:26:22 +0800 Subject: [PATCH 41/54] fix(jit): keep bounds checks when a load's receiver provenance differs from its hoist A hoist-covered get_array_el skipped its bounds check even when the operand's provenance did not name the planned receiver (expected_mode dropped). The cached tuple for the operand's slot then belonged to another receiver. Only skip the check when the planned receiver matches. Add an interpreter differential test for the argument-aliased local shape and record the verification in the roadmap note. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p1d-typedarray-receivers.md | 22 ++++++++++ jit/src/compiler/optimized.rs | 10 ++++- jit/tests/typed_receivers.rs | 52 ++++++++++++++++++++++++ 3 files changed, 82 insertions(+), 2 deletions(-) diff --git a/docs/roadmap/p1d-typedarray-receivers.md b/docs/roadmap/p1d-typedarray-receivers.md index e952b5df9..01f745aac 100644 --- a/docs/roadmap/p1d-typedarray-receivers.md +++ b/docs/roadmap/p1d-typedarray-receivers.md @@ -175,6 +175,28 @@ exit use the entry `stack_base` SSA value. That build was about 7% slower on int32array-traversal, because it kept the value live across the loop. The exit now reloads the base from the frame and is marked cold. +### r1 follow-up (fail-closed bounds hoist use) + +- A `get_array_el` skips its bounds check for a hoist only when the operand's + provenance names the planned receiver, which is when `expected_mode` is + kept. When the provenances differ, a cached tuple for the operand's slot + belongs to another receiver, so the load keeps its own bounds check. Test: + `argument_aliased_local_receiver_loads_match_the_interpreter`. Today Tier 2 + rejects the `const a = arg` shape as an unsupported opcode and fails + closed, so the test checks interpreter equality in whichever tier runs it. +- `outer_loop_hoists_are_revalidated_by_inner_loop_polls` was checked + against a build that revalidated only the innermost loop's tuples. That + build fails the test, so the test covers the blocking review defect. +- Noisy diagnostic after this change, 7 alternating runs against the `82d3808` + `jit-bench`: float64array-traversal 1.0985 ms to 0.1693 ms, 6.49x the + baseline speed; arrays-typed 2.9716 ms to 2.0081 ms, 1.48x; the + int32array-traversal control 0.0497 ms to 0.0501 ms, 0.99x, which is a tie. + Checksums match the interpreter mode. +- `mixed_direct_calls::tier2_mixed_edge_eliminates_helper_and_preserves_guard_misses` + is flaky under load. It failed 6 of 30 runs on an `82d3808` build and 5 of + 30 on this branch. The failure is timing dependent and predates this + branch. + ## Remaining work - float64array-traversal takes about 7.9 ns per iteration against about diff --git a/jit/src/compiler/optimized.rs b/jit/src/compiler/optimized.rs index 8e116d4fa..c7738a5f4 100644 --- a/jit/src/compiler/optimized.rs +++ b/jit/src/compiler/optimized.rs @@ -2804,8 +2804,14 @@ fn lower_optimized_machine( block.start_pc(), &mut guarded_element_source, expected_mode, - planned_access - .is_some_and(|access| access.bounds_covered_by_hoist), + // The hoist proved bounds for the planned + // receiver only. When the operand's + // provenance does not name that receiver, + // any cached tuple belongs to another slot: + // keep the per-access bounds check. + expected_mode.is_some() + && planned_access + .is_some_and(|access| access.bounds_covered_by_hoist), )?; } "get_length" => { diff --git a/jit/tests/typed_receivers.rs b/jit/tests/typed_receivers.rs index acc9a22ec..222ed7d74 100644 --- a/jit/tests/typed_receivers.rs +++ b/jit/tests/typed_receivers.rs @@ -795,3 +795,55 @@ fn loop_header_guard_exit_republishes_an_empty_stack() { std::thread::sleep(Duration::from_millis(1)); } } + +/// `const a = arg` keeps the entry argument's SSA identity, so the planner +/// can name `Argument` as the receiver while the operand provenance is +/// `Local`. A hoist-covered load must then keep its own bounds check (the +/// cached tuple, if any, is keyed by another slot). Out-of-range reads and +/// shrunk receivers must match the interpreter in whichever tier runs them. +#[test] +fn argument_aliased_local_receiver_loads_match_the_interpreter() { + const SOURCE: &str = r#" + function aliasSum(seed, values) { + const a = values; + let sum = seed; + for (let i = 0; i < values.length; i++) sum = (sum + a[i]) | 0; + return sum; + } + globalThis.A = new Int32Array(64).fill(3); + "#; + const CASES: &str = r#"JSON.stringify([ + aliasSum(0, A), aliasSum(0.5, A), aliasSum(1, new Int32Array(3).fill(5)), + aliasSum(0, new Float64Array([1.5, 2.5])), aliasSum(1, [1, 2, 3]), + (() => { const b = new Int32Array(8).fill(2); Object.defineProperty(b, 'length', {value: 12}); return aliasSum(0, b); })(), + (() => { const b = new Int32Array(8).fill(1); b.buffer.transfer(); return aliasSum(0, b); })() + ])"#; + let expected = evaluate_without_jit(SOURCE, CASES); + let runtime = Runtime::new().unwrap(); + let jit = Jit::attach( + &runtime, + JitConfig::builder() + .call_threshold(2) + .loop_threshold(4) + .force_optimized_for_test(true) + .stress_gc(true) + .build() + .unwrap(), + ) + .unwrap(); + let context = Context::full(&runtime).unwrap(); + context.with(|ctx| ctx.eval::<(), _>(SOURCE)).unwrap(); + // Today this shape is rejected by Tier 2 (fail-closed) and runs in a + // lower tier; warm it without requiring a Tier 2 entry so the probes + // stay exact whichever tier ends up executing them. + for _ in 0..64 { + context + .with(|ctx| ctx.eval::("aliasSum(0, A)")) + .unwrap(); + jit.poll(); + } + let actual = context.with(|ctx| ctx.eval::(CASES)).unwrap(); + assert_eq!(actual, expected); + let after = jit.metrics(); + assert_eq!(after.native_entries, after.native_exits); +} From 2558b062a4fde43abecff631b957edf3c19a02ef Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 18:27:47 +0800 Subject: [PATCH 42/54] docs: correct the P2b settle scope and refresh its diagnostics Record that the untranslatable settle now applies only to functions with newly admitted Tier 1 opcodes, withdraw the 3.62x objects-polymorphic claim that came from the over-broad settle, and add noisy diagnostics for string literal, object literal and `new` loops next to the earlier table. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p2b-misc-ops.md | 139 +++++++++++++++++++++++++---------- 1 file changed, 99 insertions(+), 40 deletions(-) diff --git a/docs/roadmap/p2b-misc-ops.md b/docs/roadmap/p2b-misc-ops.md index 91965a350..415ebb63b 100644 --- a/docs/roadmap/p2b-misc-ops.md +++ b/docs/roadmap/p2b-misc-ops.md @@ -54,48 +54,100 @@ cases. ## Runtime interaction: untranslatable Tier 1 functions Admitting these opcodes turns many small methods and host-style helpers into -Tier 1 functions that Tier 2 can never translate. At `82d3808` such a function -followed one of two paths: - -* It stayed in Tier 1 forever when a property, direct-call or numeric gate - deferred its Tier 2 trial, and it rebuilt a full feedback snapshot at every - maintenance. `objects-polymorphic` is on this path; `object` has no Tier 2 - classification. -* Otherwise it queued Tier 2 attempts that could only fail. When the - optimizing tier was blacklisted, `record_hot` disabled the generation's - probes and the function returned to the interpreter. `json-codec` is on this - path. - -With the new opcodes, the new `methods-dynamic` kernel took the first path and -ran 5.4x slower than `82d3808`, where its functions never entered native code. -Most samples were in maintenance feedback snapshots, followed by helper-PC -validation (B1). - -`ProductionBackend::maintenance` now settles such a candidate once it has -served 16 baseline executions. Before that point, deferring it costs one map -lookup. It keeps the call-only interpreter demotion and that demotion's -metric. Otherwise it moves directly to the terminal state that the failed -bounded trials used to reach: baseline demoted, probes off. Forced Tier 2 test -runs keep the old path. As a result, steady-state automatic tiering runs these -functions in the interpreter, exactly as at `82d3808` or better. The new -coverage matters for warmup and OSR, for forced Tier 1, and for later Tier 2 -work. A real gain from Tier 1 alone depends on P1: O(1) helper-PC validation -(B1) and cheaper per-call bookkeeping (B2), followed by a baseline-versus- -interpreter measurement. +Tier 1 functions that Tier 2 can never translate, because Tier 2 has no +classification for the new opcodes. At `82d3808` those functions never +entered native code, since Tier 1 rejected them. Left on the unchanged +optimizing scan, the new `methods-dynamic` kernel ran 5.4x slower than +`82d3808`: a property, direct-call or numeric gate kept deferring its Tier 2 +trial, so maintenance rebuilt a full feedback snapshot every time. Most +samples were in those snapshots, followed by helper-PC validation (B1). + +`ProductionBackend::maintenance` therefore settles a candidate only when both +of these hold: + +* Tier 2 cannot classify at least one of its opcodes + (`ir::optimized_opcodes_supported`). +* It contains at least one opcode that this slice newly admitted to Tier 1 + (`SETTLED_GENERIC_TIER1_OPCODES` in `jit/src/lib.rs`: the `GENERIC_OP` + family, `pow`, `dup1`, `dup2`, `perm4`, `swap2`, `rot3l` and `rot3r`). + +Such a candidate is settled once it has served 16 baseline executions. +Before that point, deferring it costs one map lookup. The call-only rule +still bumps its metric first. The candidate then goes back to the +interpreter with its probes off, which is where it ran at `82d3808`. Forced +Tier 2 test runs keep the old path. + +Every other function keeps the unchanged `82d3808` scan. This includes +untranslatable functions that were already Tier 1 functions at `82d3808`, +such as loops with string literals (`push_atom_value`), object literals +(`object`), `new` (`call_constructor`) or regexp literals. The first version +of this slice settled every untranslatable function. That demoted such loops +from Tier 1 to the interpreter and made a plain numeric loop ending in +`'r' + sum` about 0.77x the speed of `82d3808`. The regression test +`untranslatable_loop_without_newly_admitted_opcodes_keeps_its_baseline` in +`jit/tests/untranslatable_settle.rs` now guards this. + +As a result, steady-state automatic tiering runs functions that use the new +opcodes in the interpreter, as at `82d3808`. The new coverage helps warmup +and OSR, forced Tier 1, and later Tier 2 work, but gives no steady-state +native benefit under automatic tiering yet. `objects-polymorphic` gets no +opcodes from this slice, so it is back on its `82d3808` path, which is Tier 1 +code that is slower than the interpreter. The earlier 3.62x result on that +workload came only from the over-broad settle. ## Noisy diagnostics (not publishable evidence) The host is shared by 16 agents on 24 cores. Each value is the median over alternating runs of `median(protocol.warmup_batch_ns[-16:])` from -`jit-bench worker --mode automatic`. The baseline binary was built at -`82d3808`; the branch binary was built at `a43c209`. Speed is baseline time -divided by branch time, so values above 1.00x are faster. Every run's checksum -matched the interpreter. For `methods-dynamic`, Bun's checksum also matched -QuickJS. +`jit-bench worker --mode automatic`. Speed is baseline time divided by branch +time, so values above 1.00x are faster. Every run's checksum matched the +interpreter. For `methods-dynamic`, Bun's checksum also matched QuickJS. +These are not the three-engine README matrix, which the integrator runs. + +### After the narrowed settle (`bd5c287`) + +The baseline binary was built at `82d3808`. The `tier1-*` rows are +diagnostic-only scripts, each a single `workload(n, z)` function (not part of +the matrix): + +* `tier1-string-literal`: `for (...) sum = sum + i * 0.5; return 'r' + sum` +* `tier1-object-literal`: the same loop, allocating `{ v: i, w: z }` +* `tier1-new`: the same loop, allocating `new P(i)` + +They cover the Tier 1-only opcode families that the first settle wrongly +demoted. The interpreter column comes from one run of the branch binary with +`--mode interpreter`. + +| Workload | Runs | 82d3808 ms | branch ms | Speed | Interpreter ms | +| --- | ---: | ---: | ---: | ---: | ---: | +| tier1-string-literal | 7 | 0.4535 | 0.4514 | 1.00x | 0.604 | +| tier1-object-literal | 7 | 2.076 | 2.019 | 1.03x | 2.208 | +| tier1-new | 7 | 2.408 | 2.418 | 1.00x | 2.440 | +| methods-dynamic (new) | 7 | 12.99 | 12.91 | 1.01x | 12.96 | +| objects-polymorphic | 7 | 24.02 | 24.07 | 1.00x | 6.22 | +| adversarial | 7 | 0.953 | 0.950 | 1.00x | 0.938 | +| strings-json | 7 | 2.007 | 1.993 | 1.01x | 1.899 | +| json-codec | 7 | 79.67 | 79.05 | 1.01x | 77.21 | +| mixed-quotes | 7 | 2.517 | 2.529 | 1.00x | 2.762 | +| property-heavy | 7 | 0.0987 | 0.0984 | 1.00x | 1.082 | +| generic-call-fallback | 7 | 9.142 | 9.141 | 1.00x | 1.281 | + +With the first, over-broad settle (`a43c209`), the reviewer measured +`tier1-string-literal` at 0.586 to 0.592 ms per batch, or about 0.77x the +speed of `82d3808`. The narrowed settle brings it back to parity: the +function stays in Tier 1 and keeps entering native code. `objects-polymorphic` +is back at its `82d3808` time, which is slower than the interpreter. That is +a problem that already existed at `82d3808` and is tracked under remaining +work. + +### First slice (`a43c209`, historical) + +The baseline binary was built at `82d3808`, and the branch binary was built +at `a43c209`, before the narrowed settle. | Workload | Runs | 82d3808 ms | branch ms | Speed | | --- | ---: | ---: | ---: | ---: | -| objects-polymorphic | 7 | 24.222 | 6.697 | 3.62x (now at interpreter speed, 6.18 ms) | +| objects-polymorphic | 7 | 24.222 | 6.697 | 3.62x (over-broad settle; reverted) | | methods-dynamic (new) | 7 | 13.006 | 13.215 | 0.98x | | adversarial | 7 | 0.954 | 0.949 | 1.01x | | strings-json | 7 | 2.002 | 1.998 | 1.00x | @@ -124,18 +176,25 @@ Progression of `methods-dynamic` during development: | --- | ---: | | Opcodes admitted, maintenance unchanged | 0.18x | | Plus settling with the measured profitability trial kept | 0.26x to 0.40x (bimodal); `json-codec` fell to 0.84x | -| Final (terminal settle after 16 baseline executions) | 0.98x | +| Terminal settle after 16 baseline executions, every untranslatable function (`a43c209`) | 0.98x | +| Terminal settle only with newly admitted opcodes (`bd5c287`) | 1.01x | ## Remaining work 1. Land P1 (O(1) helper-PC validation and C-side tier state), then replace the terminal settle with a measured baseline-versus-interpreter decision, so - profitable Tier 1-only code can stay native. -2. Mapped `arguments`, once frame slots can be aliased (closure frames). -3. `set_name` and `set_name_computed`, together with `fclosure` and + profitable Tier 1-only code with the new opcodes can stay native. The same + decision could also stop the pre-existing untranslatable Tier 1 functions + (for example `objects-polymorphic`) from rebuilding feedback snapshots at + every maintenance, without demoting profitable loops. +2. Add tests for `special_object` HOME_OBJECT and VAR_OBJECT once `super` and + direct `eval` reach Tier 1. Until then no Tier 1 function can reach those + admitted cases, and the C helper matches the interpreter line for line. +3. Mapped `arguments`, once frame slots can be aliased (closure frames). +4. `set_name` and `set_name_computed`, together with `fclosure` and `define_class`. -4. Script-level compilation, or OSR in global code, for the global +5. Script-level compilation, or OSR in global code, for the global declaration family. -5. Tier 2 classifications for `typeof_is_*`, `in`, `instanceof`, +6. Tier 2 classifications for `typeof_is_*`, `in`, `instanceof`, `to_propkey2`, `to_object` and `push_this`, so hot loops with these checks can reach the optimizer. From 01619bedfd9a967d601ee9abd57f40ee722b9cae Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 18:36:38 +0800 Subject: [PATCH 43/54] fix(jit): make metadata-only typed hoists exit-free The metadata-only typed-array hoist ran its storage query in the loop preheader and side-exited on a miss, charged to the header's numeric guard. A receiver of another kind whose access never ran (a skipped conditional access, a zero-trip loop) therefore deoptimized on every call and demoted the function, which recompiled with the same plan. A metadata-only hoist covers no bounds check, so its miss now publishes an empty view (count 0, null data) through a cold join instead of exiting. Every consumer's retained bounds check then exits at the access that actually runs, poll revalidation is skipped while the count is 0, and the receiver's object tag is no longer restated at the header for such a source. Also require an exact-length (hoist-guarded) source before a range-covered load skips its bounds check, so a load-established source never inherits a proof whose preheader guard was not emitted. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p4b-bce.md | 30 +++++ jit/src/compiler/optimized.rs | 87 ++++++++++++- jit/src/compiler/optimized/array_cache.rs | 5 +- jit/tests/array_bce.rs | 144 ++++++++++++++++++++++ 4 files changed, 260 insertions(+), 6 deletions(-) diff --git a/docs/roadmap/p4b-bce.md b/docs/roadmap/p4b-bce.md index 9bb55ecdb..29e34b59c 100644 --- a/docs/roadmap/p4b-bce.md +++ b/docs/roadmap/p4b-bce.md @@ -40,6 +40,12 @@ bounds compare itself but everything around it. Bounds checks stay; polls revalidate data/count exactly like length hoists. Packed candidates never get such a hoist (their guard would demand `length == dense_count`, which an element-only loop never needed). + The preheader runs even when no access does (zero trips, a conditional + access), so this query never side-exits (review round 1): a miss publishes + an empty view (`count = 0`, `data = null`) through a cold join, every + consumer's retained bounds check then exits at the access that actually + runs, poll revalidation is skipped while `count == 0`, and the receiver's + object tag is not restated at the header. 6. **Hoisted sources across in-loop diamonds.** A block inherits a hoisted source only when every (already lowered) predecessor's exit state holds the identical source; the preheader dominates every such block. This carries @@ -104,6 +110,30 @@ Checksums match the interpreter for every run. Speed = baseline / branch. disabled through temporary switches, so it is attributed to host-binary layout noise, not to generated code; the publishable matrix must confirm it. +## Review round 1 (`perf/p4b-bce-r1`) + +- Blocking: the metadata-only hoist side-exited in the preheader under the + header's numeric guard id, so a wrong-kind receiver whose access never ran + deopted on every call and demoted the function (reproduced: 10 deopts and + a demotion for 50 `cond([9, 9], 4, false)` calls). Fixed as described in + item 5; `metadata_only_typed_hoist_never_deopts_for_an_access_that_does_not_run` + covers skipped conditional, zero-trip and poll-crossing loops, plus exact + results when the access does run with a wrong receiver. +- Hardening: a range-covered load skips its bounds check only against a + source whose count was guarded equal to the length (`exact_length`, which + only a preheader hoist guard establishes). A load- or store-established + source keeps the check even if the planned hoist was not emitted (for + example a preheader ending in an explicit `goto`). + `covered_loads_keep_bounds_checks_without_an_exact_length_source` checks + four loop shapes against a stretched packed Array. +- An exit-free `select` form of the miss path measured about 0.81x the speed + of the guarded form on `typed-convert-traversal`; the cold-join form + measured 0.99x (tied) against the pre-fix branch, so the join is used. +- `tier2_mixed_edge_eliminates_helper_and_preserves_guard_misses` + (`mixed_direct_calls`) is flaky under load on both revisions: with 6 + concurrent copies, 9/60 failures on the `82d3808` sources and 11/60 on this + branch. + ## Remaining work - `float64array-traversal` is unchanged: its receivers come from diff --git a/jit/src/compiler/optimized.rs b/jit/src/compiler/optimized.rs index 00067b4f3..8a2058bde 100644 --- a/jit/src/compiler/optimized.rs +++ b/jit/src/compiler/optimized.rs @@ -759,6 +759,15 @@ struct GuardedElementSource { /// Cached accesses then emit only that kind's load instead of a runtime /// dispatch on `kind`. static_kind: Option, + /// Established by a metadata-only loop hoist that does not side-exit. + /// Such a hoist runs in the preheader even when the loop body never + /// reaches the access (zero trips, a conditional access), so a failed + /// storage query must not deopt there. Instead `count` is 0 and `data` + /// null, and every consumer's retained bounds check exits at the access + /// itself, exactly where the unhoisted guard would have. Consumers must + /// therefore never skip the bounds check for such a source (it never has + /// `exact_length`), and poll revalidation is skipped while `count` is 0. + unguarded: bool, } /// Guarded element metadata currently valid on the lowering path, at most one @@ -781,6 +790,7 @@ impl GuardedElementSource { && self.exact_length == other.exact_length && self.typed_mode == other.typed_mode && self.static_kind == other.static_kind + && self.unguarded == other.unguarded } } @@ -2075,6 +2085,8 @@ fn lower_optimized_machine( .get(&block.start_pc()) .into_iter() .flatten() + // An unguarded hoist never proved its receiver is an object. + .filter(|source| !source.unguarded) .filter_map(|source| match source.provenance { OptProvenance::Argument(argument) if stable_hoisted_receivers.contains(&(block.start_pc(), argument)) => @@ -5892,6 +5904,7 @@ fn emit_opt_packed_metadata_guard( exact_length: true, typed_mode: None, static_kind: Some(0), + unguarded: false, }) } @@ -5909,6 +5922,7 @@ fn emit_opt_typed_metadata_guard( mode: crate::runtime::ArrayMode, array_query: usize, needs_length: bool, + side_exit: bool, ) -> Result { use cranelift_codegen::ir::{ condcodes::IntCC, types, AbiParam, InstBuilder, MemFlags, Signature, StackSlotData, @@ -5990,25 +6004,52 @@ fn emit_opt_typed_metadata_guard( status, i64::from(qjs::JSJitArrayQueryStatus_JS_JIT_ARRAY_QUERY_OK), ); - emit_opt_guard_branch(builder, env, provenance, depth, pc, guard, accepted)?; + if side_exit { + emit_opt_guard_branch(builder, env, provenance, depth, pc, guard, accepted)?; + } let live_mode = builder.ins().load(types::I32, MemFlags::new(), metadata, 4); let mode_matches = builder .ins() .icmp(IntCC::Equal, live_mode, expected_mode_value); - emit_opt_guard_branch(builder, env, provenance, depth, pc, guard, mode_matches)?; - let count = builder.ins().load(types::I32, MemFlags::new(), metadata, 8); - let data = builder + if side_exit { + emit_opt_guard_branch(builder, env, provenance, depth, pc, guard, mode_matches)?; + } + let mut count = builder.ins().load(types::I32, MemFlags::new(), metadata, 8); + let mut data = builder .ins() .load(env.pointer_type, MemFlags::new(), metadata, 16); + if !side_exit { + // A missed query publishes an empty view instead of exiting: every + // consumer's bounds check then exits at the access that actually + // runs. The slot is a live stack slot, so the loads discarded on the + // miss path read valid (if stale) memory. + let usable = builder.ins().band(accepted, mode_matches); + let miss = builder.create_block(); + let join = builder.create_block(); + builder.append_block_param(join, types::I32); + builder.append_block_param(join, env.pointer_type); + builder.set_cold_block(miss); + builder.ins().brif(usable, join, &[count, data], miss, &[]); + builder.seal_block(miss); + builder.switch_to_block(miss); + let empty_count = builder.ins().iconst(types::I32, 0); + let null_data = builder.ins().iconst(env.pointer_type, 0); + builder.ins().jump(join, &[empty_count, null_data]); + builder.seal_block(join); + builder.switch_to_block(join); + count = builder.block_params(join)[0]; + data = builder.block_params(join)[1]; + } Ok(GuardedElementSource { provenance: source_provenance, block_pc, data, count, kind: builder.ins().iconst(types::I8, kind), - exact_length: needs_length, + exact_length: needs_length && side_exit, typed_mode: Some(mode), static_kind: Some(kind), + unguarded: !side_exit, }) } @@ -6028,6 +6069,20 @@ fn emit_opt_packed_loop_revalidate( let OptProvenance::Argument(argument) = expected.provenance else { return Err(CompileFailure::InvalidArtifact); }; + // An unguarded source whose query missed (or found an empty view) holds + // `count == 0`: every consumer's bounds check exits before touching + // `data`, whatever the receiver has become, so there is nothing to + // revalidate. Requiring the current state here would reintroduce the + // speculative exit the unguarded hoist exists to avoid. + let skip = expected.unguarded.then(|| { + let empty = builder.ins().icmp_imm(IntCC::Equal, expected.count, 0); + let check = builder.create_block(); + let done = builder.create_block(); + builder.ins().brif(empty, done, &[], check, &[]); + builder.seal_block(check); + builder.switch_to_block(check); + done + }); let object = opt_use( builder, *env.arguments @@ -6048,6 +6103,7 @@ fn emit_opt_packed_loop_revalidate( mode, array_query, expected.exact_length, + true, )? } else { emit_opt_packed_metadata_guard( @@ -6071,6 +6127,11 @@ fn emit_opt_packed_loop_revalidate( .icmp(IntCC::Equal, current.data, expected.data); let unchanged = builder.ins().band(same_count, same_data); emit_opt_guard_branch(builder, env, provenance, depth, pc, guard, unchanged)?; + if let Some(done) = skip { + builder.ins().jump(done, &[]); + builder.seal_block(done); + builder.switch_to_block(done); + } Ok(()) } @@ -6150,6 +6211,10 @@ fn emit_opt_array_loop_hoists( // Only a hoist that owns the loop's length read queries // (and so certifies) the intrinsic length lookup. hoist.length.is_some(), + // A metadata-only hoist covers no bounds check, so it + // need not (and, being speculative for zero-trip loops + // and conditional accesses, must not) side-exit here. + hoist.length.is_some(), )? } crate::runtime::ArrayMode::Generic => return Err(CompileFailure::InvalidArtifact), @@ -6409,6 +6474,15 @@ fn emit_opt_element_get( // fail-closed if that invariant changes. crate::runtime::ArrayMode::Generic => -1, }); + // Range analysis proved `index < bound` for the loop's single length + // read; that deletes this check only against a count guarded equal to + // that length. Only a hoist guard establishes `exact_length` (packed + // logical length == dense count, or the intrinsic typed getter), so a + // source established by an earlier load or store, or by an unguarded + // metadata-only hoist, keeps its bounds check even when the planned + // hoist that the proof relied on was not emitted on this path. + let bounds_covered_by_hoist = + bounds_covered_by_hoist && cached.is_some_and(|source| source.exact_length); // A cached source whose storage kind is a compile-time constant needs no // runtime kind dispatch. Its receiver was tag/class/storage guarded when // the source was established for this exact frame slot, and every frame @@ -6869,6 +6943,7 @@ fn emit_opt_element_get( typed_mode: None, // Only the packed arm reaches the continuation for this mode. static_kind: Some(0), + unguarded: false, }) }); opt_define(builder, stack[object_index], result); @@ -6931,6 +7006,7 @@ fn emit_opt_typed_store( mode, array_query, false, + true, )?, }; let index = builder.ins().ireduce(types::I32, key.payload); @@ -7282,6 +7358,7 @@ fn emit_opt_element_put( exact_length: false, typed_mode: None, static_kind: None, + unguarded: false, }); for provenance in &mut stack_provenance[object_index..depth] { *provenance = OptProvenance::Unknown; diff --git a/jit/src/compiler/optimized/array_cache.rs b/jit/src/compiler/optimized/array_cache.rs index 6b5856bf8..158b09d87 100644 --- a/jit/src/compiler/optimized/array_cache.rs +++ b/jit/src/compiler/optimized/array_cache.rs @@ -45,7 +45,10 @@ pub(super) struct ArrayAccessPlan { /// attached, nonshared backing cannot change without one of the reentrant /// effects the plan already excludes from the loop, and every poll /// revalidates `data`/`count`. Such a hoist removes the per-iteration -/// metadata query but never a bounds check. +/// metadata query but never a bounds check. Because the preheader runs even +/// when no access does (zero trips, a conditional access), its query never +/// side-exits: a miss publishes an empty view, so the retained bounds check +/// exits at the access that actually runs, exactly like the unhoisted guard. #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub(super) struct ArrayLoopHoist { pub candidate: usize, diff --git a/jit/tests/array_bce.rs b/jit/tests/array_bce.rs index 17fb71d1c..cb799c684 100644 --- a/jit/tests/array_bce.rs +++ b/jit/tests/array_bce.rs @@ -261,3 +261,147 @@ fn literal_tag_folding_keeps_exact_numeric_semantics() { assert_eq!(after.native_entries, after.native_exits); assert!(after.tier2_entries > before.tier2_entries); } + +const SKIPPED: &str = r#" + function cond(a, n, use) { + let s = 0; + for (let i = 0; i < n; i++) { + if (use) s = (s + a[i]) | 0; + else s = (s + i) | 0; + } + return s; + } + function zero(a, n) { + let s = 0; + for (let i = 0; i < n; i++) s = (s + a[i]) | 0; + return s; + } +"#; + +/// Runs `call` `times` times in separate evaluations and returns the metric +/// deltas `(deopts, tier2_entries, optimized_demotions)` plus the results. +fn repeated(jit: &Jit, context: &Context, call: &str, times: usize) -> ((u64, u64, u64), String) { + let before = jit.metrics(); + let mut results = Vec::with_capacity(times); + for _ in 0..times { + results.push(context.with(|ctx| ctx.eval::(call)).unwrap()); + jit.poll(); + } + let after = jit.metrics(); + assert_eq!(after.native_entries, after.native_exits); + ( + ( + after.deopts - before.deopts, + after.tier2_entries - before.tier2_entries, + after.optimized_demotions - before.optimized_demotions, + ), + results.join(","), + ) +} + +#[test] +fn metadata_only_typed_hoist_never_deopts_for_an_access_that_does_not_run() { + // A typed element access with no `length` read in its loop hoists its + // storage query to the preheader. That query runs even when the access + // never does (a skipped conditional, a zero-trip loop), so a receiver of + // another kind must not side-exit there: the unhoisted per-access guard + // would never have run, and a preheader exit would deopt on every call + // and demote the function. Long loops also cross amortized polls, whose + // revalidation must accept the empty view such a miss publishes. + let (_runtime, jit, context) = prepared( + SKIPPED, + "cond(new Int32Array([1,2,3,4,5,6,7,8]), 8, true);\ + zero(new Int32Array([1,2,3,4,5,6,7,8]), 8)", + ); + for call in [ + "String(cond([9, 9], 4, false))", + "String(cond({ 0: 1 }, 3, false))", + "String(cond(new Float64Array([0.5]), 60000, false))", + "String(zero([9, 9], 0))", + "String(zero(new Float64Array(3), 0))", + ] { + let expected = evaluate_without_jit(SKIPPED, call); + let ((deopts, entries, demotions), actual) = repeated(&jit, &context, call, 50); + assert_eq!(actual, vec![expected; 50].join(","), "{call}"); + assert_eq!(deopts, 0, "{call}: a skipped access must not deopt"); + assert_eq!(demotions, 0, "{call}: a skipped access must not demote"); + assert!( + entries >= 50, + "{call}: every call stays in Tier 2 ({entries})" + ); + } + + // An access that does run with the wrong receiver still exits at the + // access itself and matches the interpreter exactly, as do primitive + // receivers (which the entry specialization already rejects). + const CASES: &str = r#"JSON.stringify([ + cond(undefined, 3, false), + zero(undefined, 0), + zero(7, 0), + cond([1, 2, 3, 4], 4, true), + cond(new Float64Array([1.5, 2.5]), 2, true), + cond(new Int32Array([5, 6]), 4, true), + cond(new Int32Array(0), 3, true), + zero([1, 2, 3], 3), + zero(new Int32Array([4, 5, 6]), 5), + zero(new Int32Array([4, 5, 6]), 3), + cond(new Int32Array(100000).fill(3), 100000, true) + ])"#; + let expected = evaluate_without_jit(SKIPPED, CASES); + let actual = context.with(|ctx| ctx.eval::(CASES)).unwrap(); + assert_eq!(actual, expected); + assert_eq!(jit.metrics().native_entries, jit.metrics().native_exits); +} + +#[test] +fn covered_loads_keep_bounds_checks_without_an_exact_length_source() { + // Range analysis may cover `a[i + 1]` by `i + 1 < a.length`, but that + // only deletes the check against a count guarded equal to that length. + // Whatever the loop shape (and so whether the preheader hoist was + // emitted), a source established by the earlier `a[i]` load must not + // skip the check: a packed Array whose logical length exceeds its dense + // count would otherwise read past its storage. + const SOURCE: &str = r#" + function pairsFor(a) { + let s = 0; + for (let i = 0; i + 1 < a.length; i++) s = (s + a[i] + a[i + 1]) | 0; + return s; + } + function pairsWhile(a) { + let s = 0, i = 0; + while (i + 1 < a.length) { s = (s + a[i] + a[i + 1]) | 0; i++; } + return s; + } + function pairsBreak(a) { + let s = 0, i = 0; + for (;;) { + if (!(i + 1 < a.length)) break; + s = (s + a[i] + a[i + 1]) | 0; + i++; + } + return s; + } + function pairsDo(a) { + let s = 0, i = 0; + if (a.length < 2) return 0; + do { s = (s + a[i] + a[i + 1]) | 0; i++; } while (i + 1 < a.length); + return s; + } + function all(a) { + return [pairsFor(a), pairsWhile(a), pairsBreak(a), pairsDo(a)]; + } + function stretched() { const a = [1, 2, 3]; a.length = 10; return a; } + "#; + const CASES: &str = r#"JSON.stringify([ + all([1, 2, 3, 4]), all([]), all([5]), all(stretched()), + all(new Int32Array([1, 2, 3])), all([1, 2.5, 3]), all([1, , 3]) + ])"#; + let expected = evaluate_without_jit(SOURCE, CASES); + let (_runtime, jit, context) = prepared(SOURCE, "all([1,2,3,4,5,6,7,8])"); + let before = jit.metrics(); + let actual = context.with(|ctx| ctx.eval::(CASES)).unwrap(); + let after = jit.metrics(); + assert_eq!(actual, expected); + assert_eq!(after.native_entries, after.native_exits); + assert!(after.tier2_entries > before.tier2_entries); +} From cec8f5566e185d3be9a65a66cc9a0115ff323c0f Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 18:38:05 +0800 Subject: [PATCH 44/54] fix(jit): compile try regions whose handlers no exception edge reaches QuickJS wraps each catch body in an inner region whose only instruction is its rethrow. When nothing in it can raise, its landing pad and handler are unreachable CLIF, Cranelift removes them with their Poll/Marker safepoints, and the frame-state publisher failed the whole Tier 1 compile as InvalidArtifact (e.g. `try { return o.x.y; } catch (e) { return -1; }`). Drop a frame state only when its source location occurs in no block the CLIF control flow graph reaches from entry; missing states in reachable code still fail closed. Also: - forced-baseline harness advertises required_stack_map_count(), as production does, so dropped states no longer make valid ids uncatchable invariant errors (unblocks rethrow-from-handler coverage); - a FREE of a local/argument slot inside a try region exits to the interpreter on failure instead of resuming the native handler with a stale SSA alias of the consumed slot (stress-GC OOM only); - patch 0027: JS_JitCatchException runs its stress collection before mutating the frame, so a failure leaves the catch offset for the interpreter's own unwinding (digest and quickjs.c fingerprint updated; no ABI change). Co-Authored-By: Claude Opus 5.5 (1M context) --- jit/src/compiler/baseline.rs | 54 ++++++++++++- jit/tests/support/mod.rs | 4 +- jit/tests/tier1_exceptions.rs | 78 ++++++++++++++++++- sys/build_support/patch.rs | 4 +- .../0027-tier1-exception-regions.patch | 10 ++- 5 files changed, 140 insertions(+), 10 deletions(-) diff --git a/jit/src/compiler/baseline.rs b/jit/src/compiler/baseline.rs index 96d71801e..7fb7075c3 100644 --- a/jit/src/compiler/baseline.rs +++ b/jit/src/compiler/baseline.rs @@ -616,6 +616,7 @@ impl BaselineCompiler { control.check_ir_bytes(clif_text.len())?; } let function_parameters = clif.params.clone(); + let reachable_locations = reachable_source_locations(&clif); let mut context = Context::for_function(clif); context.set_disasm(cfg!(feature = "test-support")); let compiled = context @@ -713,6 +714,15 @@ impl BaselineCompiler { if osr_start.is_some() && matching_ranges.is_empty() { return None; } + // A state lowered only into blocks the CLIF control + // flow graph cannot reach (for example the handler of + // a try region with no exceptional edge) is removed + // with its block; no machine code can observe it. + if matching_ranges.is_empty() + && !reachable_locations.contains(&source_location) + { + return None; + } return Some(Err(CompileFailure::InvalidArtifact)); }; let (location_kind, code_offset) = match state.kind { @@ -1824,6 +1834,34 @@ fn frame_state_source_loc(state: FrameStateId) -> Result BTreeSet { + let mut locations = BTreeSet::new(); + let Some(entry) = function.layout.entry_block() else { + return locations; + }; + let cfg = cranelift_codegen::flowgraph::ControlFlowGraph::with_function(function); + let mut visited = BTreeSet::from([entry]); + let mut worklist = vec![entry]; + while let Some(block) = worklist.pop() { + for instruction in function.layout.block_insts(block) { + let location = function.srcloc(instruction); + if !location.is_default() { + locations.insert(location.bits()); + } + } + for successor in cfg.succ_iter(block) { + if visited.insert(successor) { + worklist.push(successor); + } + } + } + locations +} + #[derive(Clone, Copy, Debug, Eq, Ord, PartialEq, PartialOrd)] enum EntryRoot { Argument(u16), @@ -4080,14 +4118,26 @@ fn lower_free_if_refcounted( builder.ins().brif(refcounted, slow, &[], continuation, &[]); builder.seal_block(slow); builder.switch_to_block(slow); - helpers.invoke( + // FREE consumes its slot before it can fail (stress collection only). + // A local or argument slot is then undefined in the frame while its SSA + // variable still names the released value; a native handler would resume + // with that stale alias. Such an edge exits instead, and the interpreter + // unwinds from the frame itself. Operand slots are safe: the landing pad + // resets every operand above the catch offset. + let exception_target = helpers.exception_target.get(); + if backing_base != helpers.stack_base { + helpers.exception_target.set(None); + } + let result = helpers.invoke( builder, qjs::JSJitHelperId_JS_JIT_HELPER_FREE, state, live_depth, live_depth, &[value_slot], - )?; + ); + helpers.exception_target.set(exception_target); + result?; reload_pair( builder, variables, diff --git a/jit/tests/support/mod.rs b/jit/tests/support/mod.rs index bc1ada728..55333aa85 100644 --- a/jit/tests/support/mod.rs +++ b/jit/tests/support/mod.rs @@ -361,7 +361,9 @@ unsafe impl JitBackend for ForcedBaselineBackend { stress_gc: self.stress_gc, })) .cast(), - stack_map_count: u32::try_from(self.code.stack_maps().len()).unwrap_or(u32::MAX), + // Stack-map ids are frame-state source locations; states Cranelift + // removed leave gaps, so the count is the production bound. + stack_map_count: self.code.required_stack_map_count(), helper_abi_version: rquickjs_core::qjs::QJSJIT_HELPER_ABI_VERSION, } } diff --git a/jit/tests/tier1_exceptions.rs b/jit/tests/tier1_exceptions.rs index 3917d60f7..ea1e6b497 100644 --- a/jit/tests/tier1_exceptions.rs +++ b/jit/tests/tier1_exceptions.rs @@ -14,7 +14,7 @@ use rquickjs::{Context, Runtime}; use rquickjs_jit::test_support::{differential, forced_baseline}; -use rquickjs_jit::{Jit, JitConfig, JitMetrics}; +use rquickjs_jit::{Jit, JitConfig, JitMetrics, JitTierPolicy}; fn same(definition: &str, expression: &str, opcode: &str) { differential(definition, expression) @@ -179,9 +179,18 @@ fn throw_error_raises_the_interpreter_error() { /// Runs `source` then evaluates `expression` `rounds` times with and without /// the production JIT, returning both results and the JIT metrics. fn production(source: &str, expression: &str, rounds: usize) -> (String, String, JitMetrics) { + production_with(JitConfig::default(), source, expression, rounds) +} + +fn production_with( + config: JitConfig, + source: &str, + expression: &str, + rounds: usize, +) -> (String, String, JitMetrics) { let run = |jit: bool| { let runtime = Runtime::new().unwrap(); - let attached = jit.then(|| Jit::attach(&runtime, JitConfig::default()).unwrap()); + let attached = jit.then(|| Jit::attach(&runtime, config.clone()).unwrap()); let context = Context::full(&runtime).unwrap(); let mut result = String::new(); context.with(|ctx| ctx.eval::<(), _>(source).unwrap()); @@ -262,3 +271,68 @@ fn interrupts_inside_try_regions_stay_uncatchable() { .interrupt_after(3) .assert_uncatchable_interrupt(); } + +#[test] +fn handlers_without_helper_edges_still_compile_natively() { + // QuickJS wraps a catch body in an inner region whose only instruction + // is its own rethrow. Nothing in that region can raise, so its landing + // pad and handler are unreachable native code and Cranelift removes + // their safepoints; the function must still compile. + same( + "function f(o){ try { return o.x.y; } catch (e) { return -1; } }", + "[f({x:{y:5}}), f({}), f({x:null})].join()", + "catch", + ); + same( + "function f(a){ let r = 0; try { r = a + 1; } catch (e) { r = -2; } finally { r *= 3; } return r; }", + "f(4)", + "gosub", + ); +} + +#[test] +fn refcounted_local_stores_inside_try_regions_keep_exact_ownership() { + // Every store releases the previous heap value through FREE inside the + // region; the handler then reads and overwrites the same locals. + same( + "function f(n, o){ let a = {v: 0}, b = [n]; for (let i = 0; i < n; i++) { try { a = {v: i}; b = [a, i]; if (i % 3 === 0) a = o.missing.x; } catch (e) { b = [b, a.v]; a = {v: -i}; } } return a.v + ':' + b.length; }", + "f(12, {})", + "catch", + ); +} + +#[test] +fn throws_from_handlers_without_an_enclosing_region_leave_exactly() { + same( + "function f(n){ try { throw 'a'; } catch (e) { throw e + n; } }", + "(function(){ try { f(3); } catch (e) { return 'outer:' + e; } })()", + "throw", + ); + // Recursive catch and rethrow across native frames. + same( + "function f(n){ if (n <= 0) throw 'bottom'; try { return f(n - 1) + 1; } catch (e) { throw e + n; } }", + "(function(){ try { return f(6); } catch (e) { return 'outer:' + e; } })()", + "throw", + ); +} + +#[test] +fn production_tiering_installs_common_try_catch_shapes() { + // Tier 1 only: every compile outcome counted here is a Tier 1 outcome. + let (interpreted, compiled, metrics) = production_with( + JitConfig::builder() + .tier_policy(JitTierPolicy::BaselineOnly) + .build() + .unwrap(), + "function f(n){ if (n <= 0) return 0; try { return f(n - 1) + 1; } catch (e) { return -1; } }\n\ + function g(o){ try { return o.x.y; } catch (e) { return -1; } }\n\ + function run(){ let t = 0; for (let i = 0; i < 400; i++) { t += f(i & 31); t += g((i & 7) ? {x:{y:i}} : {}); } return String(t); }", + "run()", + 40, + ); + assert_eq!(compiled, interpreted); + // f, g and run all install; none fails as an invalid artifact. + assert!(metrics.installed >= 3, "{metrics:?}"); + assert!(metrics.native_entries > 0, "{metrics:?}"); + assert_eq!(metrics.invalid_artifacts, 0, "{metrics:?}"); +} diff --git a/sys/build_support/patch.rs b/sys/build_support/patch.rs index 57edd4624..a1fa2b80b 100644 --- a/sys/build_support/patch.rs +++ b/sys/build_support/patch.rs @@ -21,7 +21,7 @@ const EXPECTED_PATCHES: [(&str, u64); 20] = [ ("0018-inline-frame-recovery.patch", 0xb971d61edccde229), ("0019-array-feedback.patch", 0x11c3cb13a8297dd4), ("0020-typed-array-guard.patch", 0x5d10_38a3_8bc9_9821), - ("0027-tier1-exception-regions.patch", 0x63478d4d077fdced), + ("0027-tier1-exception-regions.patch", 0x1f7e786816733e28), ]; pub(crate) const BASELINE_FILES: [&str; 19] = [ "api-test.c", @@ -61,7 +61,7 @@ const PATCHED_FILE_FINGERPRINTS: [(&str, u64); 21] = [ ("list.h", 0xb337_70f7_b76d_a3d8), ("quickjs-atom.h", 0x30b4_9116_b6a2_aa99), ("quickjs-c-atomics.h", 0x490b_0f29_f631_3fc0), - ("quickjs.c", 0x02f230042e7bc62d), + ("quickjs.c", 0x3def7470ab187c8d), ("quickjs-jit.h", 0xb748adfe5e91c323), ("quickjs-jit-helpers.h", 0xa469_e80f_be47_404c), ("quickjs-opcode.h", 0x3d05_cfdf_5cf7_2930), diff --git a/sys/patches/0027-tier1-exception-regions.patch b/sys/patches/0027-tier1-exception-regions.patch index 244517036..c89756e63 100644 --- a/sys/patches/0027-tier1-exception-regions.patch +++ b/sys/patches/0027-tier1-exception-regions.patch @@ -42,7 +42,7 @@ diff --git a/quickjs-jit.h b/quickjs-jit.h diff --git a/quickjs.c b/quickjs.c --- a/quickjs.c +++ b/quickjs.c -@@ -20431,6 +20431,151 @@ +@@ -20431,6 +20431,155 @@ return version == QJSJIT_INLINE_RECOVERY_VERSION ? &api : NULL; } @@ -175,6 +175,10 @@ diff --git a/quickjs.c b/quickjs.c + qjsjit_is_uncatchable_sentinel(rt, rt->current_exception) || + JS_IsUncatchableError(rt->current_exception)) + return JS_JIT_HELPER_EXCEPTION; ++ /* Stress collection runs before any frame mutation: a failure leaves the ++ * catch offset in place for the interpreter's own unwinding. */ ++ if (qjsjit_stress_gc(frame) < 0) ++ return JS_JIT_HELPER_EXCEPTION; + /* Release operands above the catch offset from the top, like the + * interpreter, then replace the offset with the caught value. */ + for (sp = frame->stack_top; sp > catch_value + 1;) { @@ -188,13 +192,13 @@ diff --git a/quickjs.c b/quickjs.c + JS_FreeValueRT(rt, ctx->error_back_trace); + ctx->error_back_trace = JS_UNDEFINED; + frame->stack_top = catch_value + 1; -+ return qjsjit_stress_gc(frame); ++ return JS_JIT_HELPER_OK; +} + static JSValue JS_CallInternal(JSContext *caller_ctx, JSValueConst func_obj, JSValueConst this_obj, JSValueConst new_target, int argc, JSValueConst *argv, int flags) -@@ -20691,12 +20836,25 @@ +@@ -20691,12 +20840,25 @@ b->jit_feedback_disabled = 1; if ((request_snapshot & 1U) && hot_vtable->submit_snapshot) { JSJitFunctionSnapshot *snapshot = NULL; From c0aab8a18d0be97cc53fcfa4c3f47e5601421133 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 18:38:05 +0800 Subject: [PATCH 45/54] docs(jit): record p2c-exceptions r1 review fixes and diagnostics Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p2c-exceptions-r1.md | 91 +++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 docs/roadmap/p2c-exceptions-r1.md diff --git a/docs/roadmap/p2c-exceptions-r1.md b/docs/roadmap/p2c-exceptions-r1.md new file mode 100644 index 000000000..f566ebb15 --- /dev/null +++ b/docs/roadmap/p2c-exceptions-r1.md @@ -0,0 +1,91 @@ +# P2c exception regions: review fixes (r1) + +Branch `perf/p2c-exceptions-r1`, based on `perf/p2c-exceptions` (`d5b40f4`). +This note follows [p2c-exceptions.md](p2c-exceptions.md). + +## Fixed + +- **Blocking: handlers with no exceptional edge failed Tier 1 as + `InvalidArtifact`.** QuickJS wraps every catch body in an inner + `catch Lx ... Lx: throw` region. When nothing in that region can raise + (for example `catch (e) { return -1; }`), its landing pad and handler are + unreachable CLIF. Cranelift removes them together with their `Poll` and + `Marker` safepoints, and the frame-state publisher rejected the missing + states. The compiler now walks the CLIF control flow graph from the entry + block before code generation. A frame state is dropped only when its + source location occurs in no reachable block. This is the same graph + Cranelift's unreachable-code elimination uses, so the rule accepts only + states that no emitted code can reference. A missing state in a reachable + block is still `InvalidArtifact`. +- **Test harness stack-map bound.** The forced-baseline harness advertised + `stack_maps().len()` as the stack-map count, while production uses + `required_stack_map_count()` (maximum source location + 1). Once states + are dropped, the harness rejected valid ids as an uncatchable + "invalid JIT helper stack map". The harness now uses the production bound. + That unblocks forced-baseline coverage of rethrow from a handler with no + enclosing region, and of recursive catch-and-rethrow. +- **Stress-GC stale local alias on a FREE edge.** `JS_JitHelperFree` consumes + its slot before its stress collection can fail. For a local or argument + store inside a try region, that edge now exits to the interpreter instead + of entering the native landing pad. The interpreter unwinds from the frame, + whose slot already holds `undefined`. Before this change, the native handler + resumed with the SSA variable still naming the released value. Operand-slot + FREEs keep native dispatch because the landing pad resets every operand + above the catch offset. +- **Stress-GC catch after mutation.** In patch 0027, `JS_JitCatchException` + now runs its stress collection before it releases operands or overwrites + the catch offset. A failure now leaves the frame untouched, and the + interpreter's own exception label handles it. Before, the frame had already + been rewritten, so unwinding skipped the handler. The patch digest and the + patched `quickjs.c` fingerprint are updated. There is no ABI or header + change. + +## New regression tests (`jit/tests/tier1_exceptions.rs`) + +- `handlers_without_helper_edges_still_compile_natively`: failed with + "forced baseline compilation failed: InvalidArtifact" before the fix. +- `production_tiering_installs_common_try_catch_shapes`: Tier 1 only. The + reviewer's recursive `f` plus `o.x.y` shapes must install (`installed >= 3`) + with `invalid_artifacts == 0`. It failed before the fix. +- `throws_from_handlers_without_an_enclosing_region_leave_exactly`: failed on + the harness stack-map bound before that fix. +- `refcounted_local_stores_inside_try_regions_keep_exact_ownership`: FREE of + heap locals inside a region, with and without stress GC. This test does not + force an allocation failure, and no deterministic OOM-injection hook exists. + +## Remaining + +- Under default tiering, a caller with no try region that calls + try/catch functions from a loop (the `run` driver in the production test) + is offered to Tier 2. Tier 2 then refuses it at admission ("no stable + feedback"), which is counted as `InvalidArtifact` (4 attempts with + backoff). The refusal is in Tier 2 candidate selection, not in exception + lowering. It is fail-closed, and the caller keeps its Tier 1 code. A clean + fix would stop offering such callers, or count the refusal as a rejection + rather than an invalid artifact. +- Skipping the discarded backtrace can avoid an OOM that the interpreter + could raise inside `build_backtrace`. This only matters under allocation + failure. +- Coverage gaps carried over from r0: indirect `nip_catch` forms, labeled + break across two finally blocks, some `let`-in-try shapes (all fail closed + with `UnsupportedOpcode`), and Tier 2 for functions with exception regions. +- The publishable three-engine README matrix is still pending, for the + integrator. + +## Noisy diagnostic (not publishable evidence) + +These numbers come from the shared, loaded 24-core host. Each value is the +median over 7 alternating worker runs of `median(warmup_batch_ns[-16:])`, +automatic tiering. Speed is `82d3808` time / r1 time, and above 1.00x is +faster. Checksums matched interpreter mode in every run. + +| workload | 82d3808 ms | r1 ms | speed | interpreter ms | +| --- | ---: | ---: | ---: | ---: | +| exceptions-sync | 2.9201 | 0.8108 | 3.60x | 2.9582 | +| exceptions-promises-async | 3.6806 | 2.4597 | 1.50x | 2.4182 | +| adversarial | 0.9522 | 0.9512 | 1.00x | 0.9371 | + +In this run, exceptions-promises-async measured about 2% slower than the +interpreter (0.98x the interpreter speed). There is no confidence interval +for these numbers, and the gap is within host noise. Adversarial is +unchanged. From 57a065b57724267331515f563e4300e509f119f4 Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 18:39:12 +0800 Subject: [PATCH 46/54] fix(jit): deliver native-chain interrupts exactly once and unstick the stack floor Review fixes for the P3a native-call slice: - A handler `true` observed by JS_JitNativeCallPoll is now recorded in rt->jit_interrupt_pending; the retried CALL's __js_poll_interrupts throws `interrupted` without asking the handler again, so one-shot interrupt handlers are no longer lost. Begin refuses while pending. - The stack-exhaustion floor moves into the runtime (patch 0029). It is cleared by the retry's own stack overflow error, by a chain at or above it, or after a refusal budget bounded by the remaining stack, so one caught RangeError no longer disables native calls for a stack height. - Non-self native call sites add the callee FunctionKey as an artifact dependency, so a retired callee invalidates the linked caller before its raw identity guard can see reused addresses. - Patch 0029 digest and patched quickjs.c fingerprint re-pinned; the new JSRuntime fields are internal (JSRuntime is opaque in the bindings). Regression tests: one-shot interrupt, resume after caught RangeError, tail self-calls, Float64 NaN/-0 comparisons, Int32 lnot, aliased callee, replaced callee, and callee release on every native-site edge. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/roadmap/p3a-native-calls.md | 105 ++++++- jit/src/compiler/native_call.rs | 33 +-- jit/src/compiler/optimized.rs | 20 ++ jit/tests/native_calls.rs | 260 ++++++++++++++++++ sys/build_support/patch.rs | 4 +- sys/patches/0029-native-call-convention.patch | 106 ++++++- sys/patches/README.md | 17 +- 7 files changed, 492 insertions(+), 53 deletions(-) diff --git a/docs/roadmap/p3a-native-calls.md b/docs/roadmap/p3a-native-calls.md index c71bbd263..5f9bfb0a3 100644 --- a/docs/roadmap/p3a-native-calls.md +++ b/docs/roadmap/p3a-native-calls.md @@ -50,6 +50,13 @@ status = entry(nc: *mut JSJitNativeCallContext, out: *mut Scalar, args...) 未命中 IC 时走原来的 generic CALL 路径（或在 CALL 处 deopt），绝不猜测。 +第一切片的实现：运行时守卫只比较对象指针和字节码指针（没有独立的 +generation/epoch 比较字）。generation 通过依赖实现：每个链接到其他函数 +native entry 的调用点都把被调方 `FunctionKey` 加入调用方 artifact 的 +`ArtifactDependency`，被调方被回收（retire）时调用方 artifact 随之失效， +因此被释放的地址被另一个同名函数复用时不会命中旧的原始指针守卫（ABA）。 +epoch 就是上面的发布 pin。 + ### 2.3 原生帧与惰性 `JSStackFrame` 物化 原生 callee 不压 `JSStackFrame`。它的帧由 Cranelift 管理，并为每个可能 @@ -77,17 +84,22 @@ status = entry(nc: *mut JSJitNativeCallContext, out: *mut Scalar, args...) `ctx->interrupt_counter`。原生入口同样按调用递减 `nc->budget`（从 `ctx->interrupt_counter` 读入，退出时写回）。预算耗尽时：没有安装 interrupt handler 就只重置预算（与 `__js_poll_interrupts` 相同）；安装了 - handler 则调用 handler，返回 0 时继续原生执行，返回非零时把 - `ctx->interrupt_counter` 置 0 并返回 status 1，由解释器重试这次调用、在 - 它自己的 poll 点抛出不可捕获的 `interrupted`（异常与 backtrace 与解释器 - 完全一致；handler 会多被调用一次，这只对宿主可见）。 + handler 则调用 handler，返回 0 时继续原生执行，返回非零时在运行时上记录 + `rt->jit_interrupt_pending`、把 `ctx->interrupt_counter` 置 0 并返回 + status 1。解释器重试这次调用，它的下一个 poll（重试 CALL 的入口 poll） + 看到 pending 后直接抛出不可捕获的 `interrupted`，**不会再次调用 handler**， + 所以 handler 的一次 `true` 不会丢失（只返回一次 `true` 的 handler 也能 + 中断）。pending 期间 `Begin` 拒绝新链。 - **栈溢出**：解释器用 `js_check_stack_overflow(rt, alloca_size)` 比较栈指针 和 `rt->stack_limit`。原生入口在序言里做同样的比较，并且按"解释器帧的 保守开销"累计虚拟栈深度（见 3.4），保证原生递归不会在解释器会抛 `RangeError` 的深度之后才失败。失败返回 status 1，由解释器以真实帧 重新执行并在同样的检查处抛出 `RangeError`。为避免每一层解释器帧重复一次 - 原生递归（二次复杂度），失败时记录一个线程局部的栈水位；在更深的位置 - 不再尝试原生调用，回到更浅的位置后自动清除。 + 原生递归（二次复杂度），失败时在运行时上记录栈水位 + `rt->jit_native_floor`；严格更深的位置不再尝试原生调用。水位在以下任一 + 情况清除：重试到达真实栈限并抛出 stack overflow `RangeError` + （`JS_ThrowStackOverflow`）；某条链在水位或更浅处开始；或者拒绝次数超过 + 按剩余栈估计的上限（`(sp - stack_limit) / 256 + 64`）。 - **GC 根**：原生参数只有标量和借用引用；借用引用的根由最外层调用方持有 （参数/局部或 owned 栈槽）。第一切片不传堆参数。 @@ -173,7 +185,8 @@ interrupt handler 被原生链轮询并能中断；单态调用方链接到 nati - native entry 调用自身使用 colocated 用户名（`NATIVE_SELF_NAMESPACE`）， 最终重定位解析为自身基址（x86_64 `X86CallPCRel4`，aarch64 `Arm64Call`）。 - 跨函数调用方把被调方 native entry 的发布 pin 放入 - `direct_call_dependencies`，与现有叶直调一致。native entry 只依赖字节码， + `direct_call_dependencies`，与现有叶直调一致；同时把被调方 + `FunctionKey` 加入 artifact 依赖（r1），被调方 retire 时调用方失效。native entry 只依赖字节码， 被调方 artifact 被替换后旧 entry 仍然正确，调用点的身份保护负责选择。 - native entry 有 unwind 信息，但没有 stack map 或 frame state；它从不在 helper、GC 或异常中出现在栈遍历器面前。 @@ -185,9 +198,15 @@ interrupt handler 被原生链轮询并能中断；单态调用方链接到 nati 调用递减一次（对应 `JS_CallInternal` 入口的 poll）。预算耗尽时调用 `JS_JitNativeCallPoll`：handler 返回 0 就重置预算继续；返回非零则标记 `INTERRUPT_DUE` 并让整条链重试，`End` 把 `ctx->interrupt_counter` 置 0， - 重试的 CALL 进入 `JS_CallInternal` 时立即 poll，由解释器抛出不可捕获的 - `interrupted`（异常和 backtrace 与解释器一致；handler 因此多被调用一次， - 这只对宿主可见）。 + 并设置 `rt->jit_interrupt_pending`。重试的 CALL 进入 `JS_CallInternal` + 时立即 poll，`__js_poll_interrupts` 看到 pending 就清除它并抛出不可捕获 + 的 `interrupted`，不再调用 handler。r1 之前重试会再次调用 handler， + 只返回一次 `true` 的 handler 的中断因此丢失（审查发现，已有回归测试）。 + 仍然存在的差异只涉及 handler 的调用时机：原生链每次调用轮询一次，而 + 解释器还在每个 `goto`/`if_true`/`if_false` 上轮询；因其他原因 + （溢出、`-0`、栈）重试的链会让解释器重新消耗预算，handler 可能比纯解释器 + 多被调用（返回 0 的调用）。这些只影响中断延迟和 handler 调用次数，不会 + 丢失或重复一次中断。 - **栈深度**：native entry 的第三个参数是"虚拟栈指针"，最外层取调用点的 真实栈指针，每层减去 `frame_charge`：`16 × (参数 + 局部 + 栈 + 4)` 加上 **本构建实测**的解释器单层开销（`interpreter_frame_bytes`：在一个无 JIT 的 @@ -195,9 +214,14 @@ interrupt handler 被原生链轮询并能中断；单态调用方链接到 nati 余量；x86_64 release 实测约 1984 字节/层，取 2480）。检查条件等价于 解释器的 `js_check_stack_overflow`（`vsp - charge < rt->stack_limit`）。 因为每层至少按解释器开销计费，原生链总在解释器会抛 `RangeError` 的深度 - 之前失败，然后由解释器以真实帧复现。失败时 `End` 在线程局部记录最外层 - 调用点的栈地址；在这个地址及更深处不再尝试原生链（否则每一层解释器帧都 - 会重新跑一次接近栈限的原生递归，变成二次复杂度），回到更浅处自动清除。 + 之前失败，然后由解释器以真实帧复现。失败时 `End` 在运行时 + （`rt->jit_native_floor`）记录最外层调用点的栈地址；严格更深处不再尝试 + 原生链（否则每一层解释器帧都会重新跑一次接近栈限的原生递归，变成二次 + 复杂度）。重试抛出 stack overflow `RangeError`、同一高度或更浅处开始新链、 + 或拒绝次数超过剩余栈对应的层数上限时清除，因此一次被捕获的 + `RangeError` 不会让同一调用高度永久失去原生调用（r1 之前的线程局部水位 + 只在严格更浅处清除，而同一个包装函数换了执行层级后帧高度会变化，导致 + 永久拒绝并反复 deopt，最终被降级）。 - **GC**：链中没有分配，也没有堆值；借用的函数对象由调用方 owned 栈槽和 全局绑定共同持有。 @@ -252,3 +276,58 @@ checksum 与解释器一致。按第 2 节的稳态口径，Bun 的 `fibonacci-r （偶尔 0.06 ms），基线二进制同样如此。 - frame 内联一个递归被调方时，调用方每次求值 deopt 一次；有 native target 的调用点现在不再内联，因此不受影响。 + +## 6. 审查修复（`perf/p3a-native-calls-r1`） + +阻塞项： + +1. **中断丢失**（已修复）。patch 0029 在 `JSRuntime` 上新增 + `jit_interrupt_pending`：`JS_JitNativeCallPoll` 在 handler 返回非零时设置它， + `__js_poll_interrupts` 先检查它并直接抛出 `interrupted`，`Begin` 在 + pending 时拒绝。回归测试 + `native_recursion_delivers_a_one_shot_interrupt`（handler 只返回一次 + `true`，自动分层和强制 Tier 2 都必须得到 `interrupted`）；修复前失败。 +2. **README 三引擎矩阵与发布级证据**：按本次工作流的环境规则（16 个 agent + 共享机器，"do NOT run the full benchmark matrix and do NOT edit the README + matrix"），发布级 QuickJS / Bun / quickjs-jit 矩阵由集成者在合并前统一 + 测量并写入 README。本分支只提供第 4 节和最终报告里的噪声诊断， + 因此在集成者刷新矩阵之前，这个优化**不算完成**（AGENTS.md）。 + +非阻塞项中已修复的： + +- 粘性栈水位：改为运行时字段，并增加 stack overflow 清除和拒绝预算 + （回归测试 `native_recursion_resumes_after_a_caught_range_error`，修复前 + 失败）。 +- 非自调用 native 调用点缺少被调方依赖（ABA）：已加入 + `ArtifactDependency`（测试 `a_replaced_callee_invalidates_its_linked_caller` + 是尽力而为的回归测试，地址复用无法被确定性地构造）。 +- 设计文档与实现的 IC 描述不一致：2.2 已更新。 + +新增测试覆盖：tail call 自调用（`native_tail_self_calls_return_the_callee_result`）、 +Float64 比较中的 NaN / `-0`（`native_float64_comparisons_match_the_interpreter`）、 +Int32 `lnot`（`native_int32_lnot_matches_the_interpreter`）、调用方通过别名 +（参数）调用被调方而全局名已重绑定（`a_linked_caller_follows_an_aliased_callee_binding`）、 +get_var 取得的被调方在成功和各 deopt 边上的引用计数（用 `WeakRef` 检查被 +替换的函数确实被回收，`native_call_sites_release_the_global_callee_on_every_edge`）。 + +仍未处理（记录为后续工作）： + +- 进入 `native_recursive` 后 `tier2_trial_decided` 被强制为 true：每次都重试 + 的链（例如 Int32 反馈但实际溢出）不会被度量或降级。应改为按链重试率 + （native fallback / Tier 2 entry）降级。 +- 调用方在 native_recursive 被调方处于 Cold/Queued/Compiling/Ready 时一直 + 等待；被调方如果因回退永远停留在 Cold，调用方就不会升级。应给等待加上 + 次数或时间上限。维护逻辑也把 `native_call_ready` 的调用点都算作已覆盖， + 但 `obj.fib(n)` 等带 `this` 的形状仍走 generic call。 +- 三元表达式把 Float64 值留在操作数栈上再调用自身（例如 + `(x>1.5?0.5:0.75)+cmp(x-1.125)`）时 Tier 2 编译返回 `InvalidArtifact`， + 失败是关闭的（留在低层级），但这个形状没有得到 native 调用。 +- 第 3.5 节列出的范围（Bool / 堆参数、链内调用其他函数、惰性帧物化）以及 + `calls-recursion-closures` 的 `recursiveSum` 和 `call-heavy` 目标仍然未完成。 + +r1 噪声诊断（同一台负载机器，交替运行 7 次，最后 16 个预热批次中位数的 +中位数，automatic；速度 = `82d3808` / r1，>1 更快；非发布数据）： +`fibonacci-recursive` 9.839 → 0.360 ms（27.3x；同一基线二进制的解释器 +9.830 ms）；`calls-recursion-closures` 6.426 → 6.436 ms（1.00x，持平）； +`call-heavy` 0.0598 → 0.0603 ms（0.99x，持平）；`quickjs-fibonacci` +0.365 → 0.364 ms（1.00x，持平）。四个场景的 checksum 都与解释器一致。 diff --git a/jit/src/compiler/native_call.rs b/jit/src/compiler/native_call.rs index 896e273d1..dfb8336ff 100644 --- a/jit/src/compiler/native_call.rs +++ b/jit/src/compiler/native_call.rs @@ -25,7 +25,6 @@ use cranelift_codegen::ir::{ StackSlotData, StackSlotKind, UserExternalName, Value, }; use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext}; -use std::cell::Cell; use std::collections::BTreeMap; use super::{CompileControl, CompileFailure}; @@ -77,9 +76,10 @@ unsafe extern "C" { function: *mut core::ffi::c_void, atom: u32, nc: *mut NativeCallContext, + sp: usize, ) -> core::ffi::c_int; fn JS_JitNativeCallPoll(nc: *mut NativeCallContext) -> core::ffi::c_int; - fn JS_JitNativeCallEnd(nc: *mut NativeCallContext); + fn JS_JitNativeCallEnd(nc: *mut NativeCallContext, sp: usize); fn JS_JitNativeCallContextLayout(offsets: *mut usize, count: usize) -> usize; } @@ -127,14 +127,6 @@ fn measure_interpreter_frame() -> Option { }) } -thread_local! { - /// Stack address of the outermost caller whose chain ran out of stack. - /// While the thread executes at or below it, native chains are not - /// attempted, so exhausted recursion continues on real interpreter - /// frames instead of retrying a native chain at every level. - static EXHAUSTED_FLOOR: Cell = const { Cell::new(0) }; -} - /// True when the C context layout matches the offsets generated code uses. pub(crate) fn context_layout_matches() -> bool { let mut offsets = [0usize; 4]; @@ -153,6 +145,13 @@ pub(crate) fn context_layout_matches() -> bool { /// Admits one outermost native chain. `sp` is the caller's stack pointer. /// Returns zero when the chain may run. /// +/// While the interpreter retries a chain that ran out of stack, chains +/// started below that chain's caller are refused, so the doomed recursion is +/// not retried natively at every level. The runtime-owned floor is cleared by +/// the retry's own stack overflow error, by a chain at or above the floor, or +/// after a refusal budget bounded by the remaining stack (patch 0029), so one +/// caught `RangeError` never disables native calls for a stack height. +/// /// # Safety /// `ctx` must be the live context of the calling frame, `function` a live /// object guarded by the caller, and `nc` writable for the whole chain. @@ -163,14 +162,7 @@ pub(crate) unsafe extern "C" fn qjsjit_native_call_begin( nc: *mut NativeCallContext, sp: usize, ) -> i32 { - let floor = EXHAUSTED_FLOOR.with(Cell::get); - if floor != 0 { - if sp <= floor { - return 1; - } - EXHAUSTED_FLOOR.with(|cell| cell.set(0)); - } - if unsafe { JS_JitNativeCallBegin(ctx, function, atom, nc) } != 0 { + if unsafe { JS_JitNativeCallBegin(ctx, function, atom, nc, sp) } != 0 { 1 } else { 0 @@ -182,10 +174,7 @@ pub(crate) unsafe extern "C" fn qjsjit_native_call_begin( /// # Safety /// `nc` must have been admitted by [`qjsjit_native_call_begin`]. pub(crate) unsafe extern "C" fn qjsjit_native_call_end(nc: *mut NativeCallContext, sp: usize) { - unsafe { JS_JitNativeCallEnd(nc) }; - if unsafe { (*nc).flags } & FLAG_STACK_EXHAUSTED != 0 { - EXHAUSTED_FLOOR.with(|cell| cell.set(sp)); - } + unsafe { JS_JitNativeCallEnd(nc, sp) }; } /// Validated description of a function's native entry. diff --git a/jit/src/compiler/optimized.rs b/jit/src/compiler/optimized.rs index 3f467f8ed..2836aa174 100644 --- a/jit/src/compiler/optimized.rs +++ b/jit/src/compiler/optimized.rs @@ -9568,12 +9568,17 @@ fn tier2_stage( /// and compiles this artifact's own native entry when the function is a pure /// self-recursive Int32 function, routing its self-call sites to it. Other /// sites keep their existing lowering. +/// +/// Every linked callee is added to `callee_dependencies`: the site guards raw +/// object and bytecode addresses, so retiring the callee must invalidate this +/// artifact before a different function can reuse those addresses (ABA). fn prepare_native_calls( isa: &cranelift_codegen::isa::OwnedTargetIsa, request: &CompileRequest, specialization: &mut NumericSpecialization, control: Option<&CompileControl>, direct_dependencies: &mut Vec, + callee_dependencies: &mut Vec, ) -> Option<( super::native_call::NativeCallPlan, super::baseline::RelocatableCode, @@ -9585,6 +9590,7 @@ fn prepare_native_calls( for instruction in request.snapshot().instructions() { if let Some(target) = request.native_call_target(instruction.pc()) { direct_dependencies.push(target.publication()); + callee_dependencies.push(target.link().callee()); specialization.native_calls.insert( instruction.pc(), NativeCallSite { @@ -9741,12 +9747,14 @@ impl Compiler for Tier2Compiler { } #[cfg(feature = "test-support")] record_tier2_stage(key, Tier2CompileStage::DirectDependencies, None); + let mut native_callee_dependencies = Vec::new(); let native_entry = prepare_native_calls( &self.isa, &request, &mut specialization, None, &mut direct_dependencies, + &mut native_callee_dependencies, ); let code = match lower_optimized_machine(&self.isa, &ir, None, profile, &specialization) { Err(CompileFailure::InvalidArtifact) @@ -9789,6 +9797,11 @@ impl Compiler for Tier2Compiler { .values() .map(|call| crate::code_cache::ArtifactDependency::new(call.callee())), ); + dependencies.extend( + native_callee_dependencies + .into_iter() + .map(crate::code_cache::ArtifactDependency::new), + ); if ir.scalar_graph().frame_inlined_calls() != 0 { dependencies.extend( frame_inline_dependencies @@ -9961,12 +9974,14 @@ impl Compiler for Tier2Compiler { } #[cfg(feature = "test-support")] record_tier2_stage(key, Tier2CompileStage::DirectDependencies, None); + let mut native_callee_dependencies = Vec::new(); let native_entry = prepare_native_calls( &self.isa, &request, &mut specialization, Some(control), &mut direct_dependencies, + &mut native_callee_dependencies, ); let code = match lower_optimized_machine( &self.isa, @@ -10033,6 +10048,11 @@ impl Compiler for Tier2Compiler { .values() .map(|call| crate::code_cache::ArtifactDependency::new(call.callee())), ); + dependencies.extend( + native_callee_dependencies + .into_iter() + .map(crate::code_cache::ArtifactDependency::new), + ); if ir.scalar_graph().frame_inlined_calls() != 0 { dependencies.extend( frame_inline_dependencies diff --git a/jit/tests/native_calls.rs b/jit/tests/native_calls.rs index 0686e1a5e..23eb2b4b6 100644 --- a/jit/tests/native_calls.rs +++ b/jit/tests/native_calls.rs @@ -463,3 +463,263 @@ fn float64_self_recursion_runs_natively_and_guards_its_signature() { ], ); } + +/// Evaluates `expression` and returns the error text, clearing the pending +/// (possibly uncatchable) exception so later evaluations start clean. +fn eval_error(harness: &Harness, expression: &str) -> Result { + harness.context.with(|ctx| { + ctx.eval::(expression).map_err(|error| { + let exception = ctx.catch(); + format!("{error}: {exception:?}") + }) + }) +} + +#[test] +fn native_recursion_delivers_a_one_shot_interrupt() { + // A host handler that answers `true` exactly once: the chain's poll + // consumes that answer, so the retried CALL must deliver it instead of + // asking the handler again (which would now answer `false`). + for force in [false, true] { + let harness = Harness::new(FIBONACCI, force); + harness.warm_native("fib(15)", "610"); + let armed = Arc::new(AtomicBool::new(true)); + let calls = Arc::new(AtomicUsize::new(0)); + harness._runtime.set_interrupt_handler(Some(Box::new({ + let armed = Arc::clone(&armed); + let calls = Arc::clone(&calls); + move || { + calls.fetch_add(1, Ordering::SeqCst); + armed.swap(false, Ordering::SeqCst) + } + }))); + let error = eval_error(&harness, "fib(27)") + .expect_err("a one-shot interrupt was lost (force_tier2={force})"); + assert!(error.contains("interrupted"), "{error}"); + assert!(!armed.load(Ordering::SeqCst)); + assert!(calls.load(Ordering::SeqCst) > 0); + // The delivered interrupt is not delivered a second time. + assert_eq!(harness.eval_string("fib(20)"), "6765"); + harness._runtime.set_interrupt_handler(None); + assert_eq!(harness.eval_string("fib(20)"), "6765"); + let metrics = harness.jit.metrics(); + assert_eq!(metrics.native_entries, metrics.native_exits); + } +} + +/// Like [`check_after_warmup`], with every expectation computed by a plain +/// interpreter runtime without a JIT. +fn check_against_interpreter(source: &str, warm: &[&str], probes: &[&str]) { + let interpreter = Runtime::new().unwrap(); + let context = Context::full(&interpreter).unwrap(); + let expect = |expression: &str| -> String { + context.with(|ctx| { + ctx.eval::<(), _>(source).unwrap(); + ctx.eval::(format!("String({expression})")) + .unwrap() + }) + }; + let warm: Vec<(&str, String)> = warm.iter().map(|w| (*w, expect(w))).collect(); + let probes: Vec<(&str, String)> = probes.iter().map(|p| (*p, expect(p))).collect(); + let warm: Vec<(&str, &str)> = warm.iter().map(|(e, r)| (*e, r.as_str())).collect(); + let probes: Vec<(&str, &str)> = probes.iter().map(|(e, r)| (*e, r.as_str())).collect(); + check_after_warmup(source, &warm, &probes); +} + +#[test] +fn native_tail_self_calls_return_the_callee_result() { + check_against_interpreter( + "function count(n,acc){if(n<=0)return acc;return count(n-1,acc+1);}\n\ + function down(a,b){return a<=b?a:down(a-b,b);}", + &["count(30,0)", "down(1000,7)"], + &[ + "count(100,5)", + // Overflow of the tail-call argument deep in the chain. + "count(20,2147483640)", + "down(1071,462)", + "down(10,10)", + "down(-2147483648,1)", + ], + ); +} + +#[test] +fn native_recursion_resumes_after_a_caught_range_error() { + // Both chains start from exactly the same stack height: the same + // wrapper, called from the same global code. + let source = "function depth(n){return n<=0?0:1+depth(n-1);}\n\ + function run(n){try{return depth(n)}catch(e){return -1}}"; + let harness = Harness::new(source, false); + harness.warm_native("run(200)", "200"); + assert_eq!(harness.eval_string("run(1e6)"), "-1"); + // A later chain from that height runs natively again instead of being + // refused (and deoptimized) at every call. + harness.warm_native("run(300)", "300"); + let metrics = harness.jit.metrics(); + assert_eq!(metrics.native_entries, metrics.native_exits); +} + +#[test] +fn native_float64_comparisons_match_the_interpreter() { + check_against_interpreter( + // Results stay non-integral (0.1 plus quarters) so the baseline tier + // never re-tags them as Int32. + "function cmp(x){if(x>0.5){if(x>1.5)return 0.5+cmp(x-1.125);return 0.75+cmp(x-1.125);}return 0.1;}", + &["cmp(20.3)"], + &[ + // NaN fails every ordered comparison and ends the recursion. + "cmp(NaN)", + "cmp(5.5+NaN)", + "Object.is(cmp(-0),0.1)", + "cmp(2)", + "cmp(1.5)", + "cmp(0.5)", + ], + ); +} + +#[test] +fn native_int32_lnot_matches_the_interpreter() { + check_against_interpreter( + "function inv(n){if(!n)return 0;return 1+inv(n-1);}", + &["inv(30)"], + &[ + // `!n` on Int32 zero and non-zero, and on non-Int32 arguments. + "inv(0)", + "inv(-0)", + "inv(3)", + "inv(true)", + ], + ); +} + +#[test] +fn a_linked_caller_follows_an_aliased_callee_binding() { + // The caller reaches the callee through a parameter, not through the + // callee's own global name, so the chain must still prove that name. + let source = format!( + "{FIBONACCI}\nfunction caller(n,g){{let s=0;for(let i=0;i{const old=fib;fib=function(n){return 1};const r=caller(2,old);fib=old;return r})()" + ), + "4" + ); + assert_eq!(harness.eval_string("caller(2,fib)"), "288"); + let metrics = harness.jit.metrics(); + assert_eq!(metrics.native_entries, metrics.native_exits); + } +} + +#[test] +fn a_replaced_callee_invalidates_its_linked_caller() { + // The caller guards raw callee identities. When the callee is freed and a + // different function with the same name takes its place, possibly at the + // same addresses, the caller must not run the old native body. + let source = format!( + "{FIBONACCI}\nfunction caller(n){{let s=0;for(let i=0;i i64 { + if n < 2 { + n + k + } else { + reference(n - 1, k) + reference(n - 2, k) + } + } + for force in [false, true] { + let harness = Harness::new(&source, force); + harness.warm_native("fib(15)", "610"); + for _ in 0..64 { + assert_eq!(harness.eval_string("caller(8)"), "1152"); + harness.jit.poll(); + } + for round in 0..16i64 { + harness + .context + .with(|ctx| { + ctx.eval::<(), _>(format!( + "fib=(0,eval)('(function fib(n){{if(n<2)return n+{round};return fib(n-1)+fib(n-2);}})');" + )) + }) + .unwrap(); + harness._runtime.run_gc(); + harness.jit.poll(); + assert_eq!( + harness.eval_string("caller(3)"), + (reference(12, round) * 3).to_string(), + "round {round} (force_tier2={force})" + ); + } + let metrics = harness.jit.metrics(); + assert_eq!(metrics.native_entries, metrics.native_exits); + } +} + +#[test] +fn native_call_sites_release_the_global_callee_on_every_edge() { + // `acc` is loaded by get_var (an owned stack slot). The native site + // frees that reference after success and hands it to the interpreter on + // every deopt edge; a leak keeps the replaced callee alive and an extra + // release frees it early. + let source = "function acc(n,k){return n<=0?k:1+acc(n-1,k);}\n\ + function caller(n,x,k){let s=0;for(let i=0;i{const old=acc;acc=function(n,k){return k};const r=caller(2,5,7);acc=old;return r})()" + ), + "14" + ); + assert_eq!(harness.eval_string("caller(4,20,1)"), "84"); + assert_eq!( + harness.eval_string( + "(globalThis.weak=new WeakRef(acc),acc=function(n,k){return 0},caller(3,20,1))" + ), + "0" + ); + harness._runtime.run_gc(); + harness.jit.poll(); + harness._runtime.run_gc(); + assert_eq!( + harness.eval_string("weak.deref()===undefined"), + "true", + "the replaced callee leaked (force_tier2={force})" + ); + let metrics = harness.jit.metrics(); + assert_eq!(metrics.native_entries, metrics.native_exits); + } +} diff --git a/sys/build_support/patch.rs b/sys/build_support/patch.rs index 53dcddac6..1d62c7694 100644 --- a/sys/build_support/patch.rs +++ b/sys/build_support/patch.rs @@ -21,7 +21,7 @@ const EXPECTED_PATCHES: [(&str, u64); 20] = [ ("0018-inline-frame-recovery.patch", 0xb971d61edccde229), ("0019-array-feedback.patch", 0x11c3cb13a8297dd4), ("0020-typed-array-guard.patch", 0x5d10_38a3_8bc9_9821), - ("0029-native-call-convention.patch", 0x6bbd_3477_da28_d2f8), + ("0029-native-call-convention.patch", 0x1061_fde9_800b_1e1f), ]; pub(crate) const BASELINE_FILES: [&str; 19] = [ "api-test.c", @@ -61,7 +61,7 @@ const PATCHED_FILE_FINGERPRINTS: [(&str, u64); 21] = [ ("list.h", 0xb337_70f7_b76d_a3d8), ("quickjs-atom.h", 0x30b4_9116_b6a2_aa99), ("quickjs-c-atomics.h", 0x490b_0f29_f631_3fc0), - ("quickjs.c", 0x3d9e_5139_db1c_b47b), + ("quickjs.c", 0x46d0_2a68_77d0_2884), ("quickjs-jit.h", 0x0188_04ad_21d2_95a7), ("quickjs-jit-helpers.h", 0xa469_e80f_be47_404c), ("quickjs-opcode.h", 0x3d05_cfdf_5cf7_2930), diff --git a/sys/patches/0029-native-call-convention.patch b/sys/patches/0029-native-call-convention.patch index 64cbb4f52..2dd99346a 100644 --- a/sys/patches/0029-native-call-convention.patch +++ b/sys/patches/0029-native-call-convention.patch @@ -1,7 +1,53 @@ diff --git a/quickjs.c b/quickjs.c --- a/quickjs.c +++ b/quickjs.c -@@ -67202,3 +67202,121 @@ +@@ -316,6 +316,20 @@ + + JSInterruptHandler *interrupt_handler; + void *interrupt_opaque; ++#ifdef CONFIG_JIT_ABI ++ /* An interrupt the handler requested inside a JIT native call chain. The ++ chain retries its CALL in the interpreter, whose next poll delivers it ++ without asking the handler again. */ ++ bool jit_interrupt_pending; ++ /* Stack address of the outermost caller of a native call chain that ran ++ out of stack, zero when none. Chains below it are refused while the ++ interpreter retries that chain, so the doomed recursion is not retried ++ natively at every level. Cleared by the next stack overflow error ++ (the retry reached the real limit), by a chain at or above it, or ++ after jit_native_floor_refusals refusals. */ ++ uintptr_t jit_native_floor; ++ uint32_t jit_native_floor_refusals; ++#endif + + JSPromiseHook *promise_hook; + void *promise_hook_opaque; +@@ -9453,6 +9467,10 @@ + + static JSValue JS_ThrowStackOverflow(JSContext *ctx) + { ++#ifdef CONFIG_JIT_ABI ++ /* A retried exhausted native chain has reached the real limit. */ ++ ctx->rt->jit_native_floor = 0; ++#endif + return JS_ThrowRangeError(ctx, "Maximum call stack size exceeded"); + } + +@@ -9538,6 +9556,13 @@ + { + JSRuntime *rt = ctx->rt; + ctx->interrupt_counter = JS_INTERRUPT_COUNTER_INIT; ++#ifdef CONFIG_JIT_ABI ++ if (unlikely(rt->jit_interrupt_pending)) { ++ rt->jit_interrupt_pending = false; ++ JS_ThrowInterrupted(ctx); ++ return -1; ++ } ++#endif + if (rt->interrupt_handler) { + if (rt->interrupt_handler(rt, rt->interrupt_opaque)) { + JS_ThrowInterrupted(ctx); +@@ -67202,3 +67227,157 @@ #undef malloc #undef free #undef realloc @@ -12,7 +58,9 @@ diff --git a/quickjs.c b/quickjs.c + unboxed scalars and calls itself. One context lives on the stack of the + outermost compiled caller and is shared by the whole native call chain. + Every failure makes the outermost caller re-execute the CALL in the -+ interpreter, which is exact because nothing observable happened. */ ++ interpreter, which is exact because nothing observable happened except ++ interrupt-handler polls; a handler request is kept pending in the runtime ++ so the retried CALL delivers it instead of polling the handler again. */ +typedef struct JSJitNativeCallContext { + uintptr_t stack_limit; /* rt->stack_limit, zero when unlimited */ + int32_t budget; /* remaining interrupt polls in this chain */ @@ -24,10 +72,15 @@ diff --git a/quickjs.c b/quickjs.c +#define JS_JIT_NATIVE_CALL_INTERRUPT_DUE (1u << 1) +#define JS_JIT_NATIVE_CALL_BUDGET_LIMITED (1u << 2) + ++/* Minimum C stack bytes per interpreter level assumed when bounding the ++ refusals charged to an exhausted chain's retry. */ ++#define JS_JIT_NATIVE_FLOOR_LEVEL_BYTES 256 ++#define JS_JIT_NATIVE_FLOOR_SLACK 64 ++ +int JS_JitNativeCallBegin(JSContext *ctx, JSObject *p, JSAtom atom, -+ JSJitNativeCallContext *nc); ++ JSJitNativeCallContext *nc, uintptr_t sp); +int JS_JitNativeCallPoll(JSJitNativeCallContext *nc); -+void JS_JitNativeCallEnd(JSJitNativeCallContext *nc); ++void JS_JitNativeCallEnd(JSJitNativeCallContext *nc, uintptr_t sp); +size_t JS_JitNativeCallContextLayout(size_t *offsets, size_t count); + +/* Proves, once per outermost native call chain, that the callee's self @@ -35,9 +88,10 @@ diff --git a/quickjs.c b/quickjs.c + plain data binding holding `p`. No JavaScript or host code runs inside the + chain (the interrupt handler is only polled through JS_JitNativeCallPoll), + so every OP_get_var of `atom` in the chain yields the same function -+ without invoking a getter or throwing. Returns 0 when admitted. */ ++ without invoking a getter or throwing. `sp` is the caller's stack ++ pointer. Returns 0 when admitted. */ +int JS_JitNativeCallBegin(JSContext *ctx, JSObject *p, JSAtom atom, -+ JSJitNativeCallContext *nc) ++ JSJitNativeCallContext *nc, uintptr_t sp) +{ + JSRuntime *rt; + JSFunctionBytecode *b; @@ -46,10 +100,21 @@ diff --git a/quickjs.c b/quickjs.c + + if (!ctx || !p || !nc || p->class_id != JS_CLASS_BYTECODE_FUNCTION) + return -1; ++ rt = ctx->rt; ++ if (rt->jit_native_floor) { ++ if (sp < rt->jit_native_floor && rt->jit_native_floor_refusals) { ++ rt->jit_native_floor_refusals--; ++ return -1; ++ } ++ rt->jit_native_floor = 0; ++ } + b = p->u.func.function_bytecode; + /* The chain shares the caller's interrupt counter and global lookup. */ + if (!b || b->realm != ctx || b->func_kind != JS_FUNC_NORMAL) + return -1; ++ /* An undelivered interrupt must reach the interpreter's next poll. */ ++ if (rt->jit_interrupt_pending) ++ return -1; + prs = find_own_property(&pr, JS_VALUE_GET_OBJ(ctx->global_var_obj), atom); + if (!prs) { + prs = find_own_property(&pr, JS_VALUE_GET_OBJ(ctx->global_obj), atom); @@ -63,7 +128,6 @@ diff --git a/quickjs.c b/quickjs.c + if (JS_VALUE_GET_TAG(pr->u.value) != JS_TAG_OBJECT || + JS_VALUE_GET_OBJ(pr->u.value) != p) + return -1; -+ rt = ctx->rt; + nc->ctx = ctx; + nc->stack_limit = rt->stack_limit; + nc->flags = 0; @@ -78,9 +142,12 @@ diff --git a/quickjs.c b/quickjs.c +} + +/* Called by a native entry after its interrupt budget reached zero. This is -+ __js_poll_interrupts without throwing: an interrupt request makes the chain -+ retry in the interpreter, whose own poll then throws the uncatchable error -+ with an exact backtrace. Returns non-zero to request that retry. */ ++ __js_poll_interrupts without throwing: an interrupt request is recorded in ++ rt->jit_interrupt_pending and makes the chain retry in the interpreter, ++ whose next poll (the retried CALL's entry poll, since End exhausts the ++ counter) throws the uncatchable error without calling the handler again, ++ so a handler's single `true` is never lost. Returns non-zero to request ++ that retry. */ +int JS_JitNativeCallPoll(JSJitNativeCallContext *nc) +{ + JSRuntime *rt = nc->ctx->rt; @@ -92,6 +159,7 @@ diff --git a/quickjs.c b/quickjs.c + nc->budget = JS_INTERRUPT_COUNTER_INIT; + if (rt->interrupt_handler && + rt->interrupt_handler(rt, rt->interrupt_opaque)) { ++ rt->jit_interrupt_pending = true; + nc->flags |= JS_JIT_NATIVE_CALL_INTERRUPT_DUE; + nc->budget = 0; + return 1; @@ -100,10 +168,24 @@ diff --git a/quickjs.c b/quickjs.c +} + +/* Publishes the chain's interrupt accounting back to the context. A due -+ interrupt leaves the counter exhausted so the retried CALL polls first. */ -+void JS_JitNativeCallEnd(JSJitNativeCallContext *nc) ++ interrupt leaves the counter exhausted so the retried CALL polls first and ++ delivers the pending interrupt. A chain that ran out of stack records its ++ caller's stack pointer `sp` as the runtime's native floor. */ ++void JS_JitNativeCallEnd(JSJitNativeCallContext *nc, uintptr_t sp) +{ + JSContext *ctx = nc->ctx; ++ JSRuntime *rt = ctx->rt; ++ ++ if ((nc->flags & JS_JIT_NATIVE_CALL_STACK_EXHAUSTED) && ++ nc->stack_limit && sp > nc->stack_limit) { ++ /* The retry has at most one interpreter level per LEVEL_BYTES of ++ the remaining stack; refusals beyond that are stale. */ ++ uintptr_t levels = (sp - nc->stack_limit) / ++ JS_JIT_NATIVE_FLOOR_LEVEL_BYTES + JS_JIT_NATIVE_FLOOR_SLACK; ++ rt->jit_native_floor = sp; ++ rt->jit_native_floor_refusals = ++ levels > UINT32_MAX ? UINT32_MAX : (uint32_t)levels; ++ } + + if (nc->flags & JS_JIT_NATIVE_CALL_INTERRUPT_DUE) + ctx->interrupt_counter = 0; diff --git a/sys/patches/README.md b/sys/patches/README.md index d0ef56223..dbf0a5243 100644 --- a/sys/patches/README.md +++ b/sys/patches/README.md @@ -68,10 +68,19 @@ so the runtime API/helper ABI version and the bundled bindings are unchanged. global self reference resolves, as `OP_get_var` would in the callee realm, to a plain data binding holding exactly the callee, and loads the stack limit and interrupt budget into a caller-owned `JSJitNativeCallContext`. -`JS_JitNativeCallPoll` is `__js_poll_interrupts` without throwing (an -interrupt makes the chain retry in the interpreter), and `JS_JitNativeCallEnd` -publishes the chain's interrupt accounting. `JS_JitNativeCallContextLayout` -lets the compiler verify the context layout before generating code. +`JS_JitNativeCallPoll` is `__js_poll_interrupts` without throwing: an +interrupt request is recorded in the new internal `JSRuntime` field +`jit_interrupt_pending` and makes the chain retry in the interpreter, whose +next `__js_poll_interrupts` delivers it without calling the handler again, so +a handler's single `true` is never lost. `JS_JitNativeCallEnd` publishes the +chain's interrupt accounting and, after stack exhaustion, records the +caller's stack pointer in `jit_native_floor`; `Begin` refuses chains strictly +below it while the interpreter retries, and the floor clears on the next +stack overflow error, on a chain at or above it, or after a refusal budget +derived from the remaining stack. `JS_JitNativeCallContextLayout` lets the +compiler verify the context layout before generating code. The new +`JSRuntime` fields are internal (the bundled bindings keep `JSRuntime` +opaque), so neither the ABI version nor the bindings change. The build accepts only the patch names and byte digests listed in `build_support/patch.rs`, then verifies the complete patched source manifest. From cb28355b4a8dd4ec14bc62e9eb0b0cf8c53a9c7a Mon Sep 17 00:00:00 2001 From: Jason Lee Date: Sun, 27 Sep 2026 20:12:33 +0800 Subject: [PATCH 47/54] docs(perf): add V8/Bun gap attribution and P0-P5 roadmap Attribute the remaining quickjs-jit gap to Bun/V8 on 82d3808 and plan staged, acceptance-gated work: - The Bun worker (`bun -e`) times only batch 64, which hits a one-off JSC stall; two-argument scenarios are inflated 1.2x-32x, so the README "faster than Bun" scalar rows are 0.15x-0.51x in steady state. - Diagnostic steady-state three-engine table (median of the last 16 warmup batches); every scenario is slower than Bun. - gdb signal-sampling profiles: linear PC rescans in helper frame validation (63% self time in objects-polymorphic), per-call JIT bookkeeping, 124/255 rejected opcodes, no general native call ABI. - Capability map against V8 and JSC and phased roadmap P0-P5. Diagnostics only; no runtime change and no README matrix change. Co-Authored-By: Claude Opus 5.5 (1M context) --- ...dmap-steady-diagnostic-82d3808-x86_64.json | 422 +++++++++++++++++ docs/PERFORMANCE_NEXT.md | 6 + docs/PERFORMANCE_ROADMAP_V8_BUN.md | 433 ++++++++++++++++++ 3 files changed, 861 insertions(+) create mode 100644 benchmarks/results/roadmap-steady-diagnostic-82d3808-x86_64.json create mode 100644 docs/PERFORMANCE_ROADMAP_V8_BUN.md diff --git a/benchmarks/results/roadmap-steady-diagnostic-82d3808-x86_64.json b/benchmarks/results/roadmap-steady-diagnostic-82d3808-x86_64.json new file mode 100644 index 000000000..346416c11 --- /dev/null +++ b/benchmarks/results/roadmap-steady-diagnostic-82d3808-x86_64.json @@ -0,0 +1,422 @@ +{ + "adversarial": { + "interpreter": [ + 0.936002, + 0.933591 + ], + "automatic": [ + 0.952012, + 0.9524565 + ], + "bun": [ + 0.123311, + 0.0355765 + ] + }, + "arrays-typed": { + "interpreter": [ + 4.134539, + 4.137498 + ], + "automatic": [ + 2.969289, + 2.9720165 + ], + "bun": [ + 0.13876, + 0.0446725 + ] + }, + "call-heavy": { + "interpreter": [ + 1.2778, + 1.2788855 + ], + "automatic": [ + 0.059685, + 0.059414 + ], + "bun": [ + 0.006336, + 0.006371 + ] + }, + "calls-closures": { + "interpreter": [ + 3.347925, + 3.3355985 + ], + "automatic": [ + 3.324853, + 3.3101155 + ], + "bun": [ + 0.184617, + 0.0823825 + ] + }, + "calls-recursion-closures": { + "interpreter": [ + 6.428335, + 6.405529 + ], + "automatic": [ + 6.415237, + 6.406498 + ], + "bun": [ + 0.206022, + 0.1252145 + ] + }, + "collections": { + "interpreter": [ + 1.631005, + 1.6327025 + ], + "automatic": [ + 1.638505, + 1.644423 + ], + "bun": [ + 0.135146, + 0.045662 + ] + }, + "exceptions-promises-async": { + "interpreter": [ + 2.332285, + 2.331115 + ], + "automatic": [ + 3.67627, + 3.6856845 + ], + "bun": [ + 0.282677, + 0.198623 + ] + }, + "fibonacci-iterative": { + "interpreter": [ + 35.46101, + 35.5026675 + ], + "automatic": [ + 1.003959, + 0.9942645 + ], + "bun": [ + 0.941968, + 0.159185 + ] + }, + "fibonacci-recursive": { + "interpreter": [ + 9.794635, + 9.7966025 + ], + "automatic": [ + 9.829058, + 9.8253595 + ], + "bun": [ + 0.539598, + 0.274126 + ] + }, + "float64-dense": { + "interpreter": [ + 2.297226, + 2.2795935 + ], + "automatic": [ + 0.128599, + 0.1291535 + ], + "bun": [ + 0.147671, + 0.0656965 + ] + }, + "float64array-traversal": { + "interpreter": [ + 1.532843, + 1.536856 + ], + "automatic": [ + 1.122251, + 1.1065345 + ], + "bun": [ + 0.00822, + 0.0082595 + ] + }, + "generic-call-entry": { + "interpreter": [ + 0.961134, + 0.961305 + ], + "automatic": [ + 0.032348, + 0.0321885 + ], + "bun": [ + 0.00493, + 0.0049685 + ] + }, + "generic-call-fallback": { + "interpreter": [ + 1.277341, + 1.280864 + ], + "automatic": [ + 9.117751, + 9.1241035 + ], + "bun": [ + 0.008123, + 0.008128 + ] + }, + "host-compute": { + "interpreter": [ + 2.164716, + 2.1640915 + ], + "automatic": [ + 0.068316, + 0.0714985 + ], + "bun": [ + 0.23893, + 0.016875 + ] + }, + "int32array-traversal": { + "interpreter": [ + 1.027674, + 1.034962 + ], + "automatic": [ + 0.049606, + 0.0496065 + ], + "bun": [ + 0.005769, + 0.0057285 + ] + }, + "json-codec": { + "interpreter": [ + 76.982995, + 77.0292145 + ], + "automatic": [ + 79.027859, + 78.986806 + ], + "bun": [ + 7.698369, + 7.5309665 + ] + }, + "map-set-bigint": { + "interpreter": [ + 16.504971, + 16.4096075 + ], + "automatic": [ + 17.758799, + 17.5385175 + ], + "bun": [ + 0.966065, + 0.7951905 + ] + }, + "mixed-quotes": { + "interpreter": [ + 2.76319, + 2.7641745 + ], + "automatic": [ + 2.47999, + 2.4808655 + ], + "bun": [ + 0.145391, + 0.0597135 + ] + }, + "numeric": { + "interpreter": [ + 0.531989, + 0.538418 + ], + "automatic": [ + 0.029369, + 0.029281 + ], + "bun": [ + 0.130152, + 0.0045095 + ] + }, + "objects-polymorphic": { + "interpreter": [ + 6.182188, + 6.189503 + ], + "automatic": [ + 23.811075, + 23.82931 + ], + "bun": [ + 0.198658, + 0.106358 + ] + }, + "packed-array-traversal": { + "interpreter": [ + 0.823972, + 0.826235 + ], + "automatic": [ + 0.047952, + 0.048104 + ], + "bun": [ + 0.004985, + 0.0049295 + ] + }, + "property-heavy": { + "interpreter": [ + 1.080826, + 1.066427 + ], + "automatic": [ + 0.097924, + 0.0984835 + ], + "bun": [ + 0.007811, + 0.007854 + ] + }, + "quickjs-bitops": { + "interpreter": [ + 1.193064, + 1.1930495 + ], + "automatic": [ + 0.061564, + 0.061514 + ], + "bun": [ + 0.147257, + 0.014029 + ] + }, + "quickjs-fibonacci": { + "interpreter": [ + 0.946607, + 0.9467165 + ], + "automatic": [ + 0.351308, + 0.3657225 + ], + "bun": [ + 0.143341, + 0.004444 + ] + }, + "quickjs-int-arith": { + "interpreter": [ + 5.801455, + 5.79549 + ], + "automatic": [ + 1.325363, + 1.3430865 + ], + "bun": [ + 0.131311, + 0.0476325 + ] + }, + "scalar-control-flow": { + "interpreter": [ + 2.131625, + 2.133814 + ], + "automatic": [ + 0.05449, + 0.0544175 + ], + "bun": [ + 0.090601, + 0.0103175 + ] + }, + "scalar-expressions": { + "interpreter": [ + 2.159235, + 2.1598165 + ], + "automatic": [ + 0.050452, + 0.050305 + ], + "bun": [ + 0.090121, + 0.009258 + ] + }, + "scalar-loop": { + "interpreter": [ + 0.54558, + 0.537524 + ], + "automatic": [ + 0.0293, + 0.0291975 + ], + "bun": [ + 0.132566, + 0.0044785 + ] + }, + "strings-json": { + "interpreter": [ + 1.926643, + 1.9159105 + ], + "automatic": [ + 2.001462, + 2.000367 + ], + "bun": [ + 0.165444, + 0.069683 + ] + }, + "strings-regexp": { + "interpreter": [ + 19.291352, + 19.2642415 + ], + "automatic": [ + 19.921122, + 19.89564 + ], + "bun": [ + 1.294154, + 0.973643 + ] + } +} \ No newline at end of file diff --git a/docs/PERFORMANCE_NEXT.md b/docs/PERFORMANCE_NEXT.md index c894ec736..6314914ee 100644 --- a/docs/PERFORMANCE_NEXT.md +++ b/docs/PERFORMANCE_NEXT.md @@ -1,5 +1,11 @@ # 下一阶段性能目标：以 Bun 为差距参照 +> **2026-09-27 补充：** [对齐 V8/Bun 路线图](PERFORMANCE_ROADMAP_V8_BUN.md) +> 发现 Bun worker（`bun -e`）的唯一计时批次会稳定遇到一次 JSC 侧停顿。 +> 双参数场景的 Bun 延迟因此被放大 1.2x–32x，本文及 README 中"快于 +> Bun"的标量场景读数都不成立。在协议修正（路线图 P0）之前，不要再 +> 引用这些读数设定 Bun 目标。 + 更新：2026-09-10。下一阶段以 **quickjs-jit automatic 相对 Bun 默认配置的性能差距**选择优化点和定义阶段目标。旧版 JIT 是回归基线，QuickJS 解释器是判断原生执行是否有收益的基线；二者都不能代替 Bun 这一目标参照。 以下优先级最初以运行时 `07535b1` 制定。本轮调用/属性/数组实现与验证见 diff --git a/docs/PERFORMANCE_ROADMAP_V8_BUN.md b/docs/PERFORMANCE_ROADMAP_V8_BUN.md new file mode 100644 index 000000000..49a307a98 --- /dev/null +++ b/docs/PERFORMANCE_ROADMAP_V8_BUN.md @@ -0,0 +1,433 @@ +# 对齐 V8 / Bun 的 JIT 性能路线图（2026-09-27 分析） + +本文件接续 [M3](M3.md)、[下一阶段目标](PERFORMANCE_NEXT.md)、 +[边界优化记录](BUN_BOUNDARIES_20260909.md) 和 +[semantic SSA 进展](SEMANTIC_SSA_PROGRESS.md)，基于 `82d3808`（0.12.9） +做一次**差距归因**，并给出分阶段、可验收的规划。 + +本文件**不包含任何运行时优化**，README 矩阵保持不变。文中所有实验 +数值都是单机诊断（i7-13700KF，CPU 8，`powersave`，Bun 1.4.0，3–7 个 +进程取中位数），不是 AGENTS.md 要求的发布级配对证据，不能当作优化 +收益引用；每个阶段落地时仍需完整三引擎矩阵。 + +## 1. 结论摘要 + +1. **测量协议有系统偏差，当前矩阵高估了 quickjs-jit 相对 Bun 的位置。** + Bun worker 用 `bun -e` 执行，第 64 个批次（恰好是协议唯一计时的 + 那一批）会稳定出现一次 JSC 侧的一次性停顿（约 0.1–0.2 ms）。 + 所有无 `workloadArgument` 的双参数场景的 Bun 延迟都被放大 + 1.2x–32x。README 中 7 个"快于 Bun"的场景，按稳态批次口径**全部 + 慢于 Bun**（见第 2 节）。先修协议，否则后续所有"对齐 Bun"的 + 目标都建立在错误基线上。 +2. **按稳态口径，quickjs-jit 在全部 30 个场景都慢于 Bun。** 最好的 + 标量循环约为 Bun 的 0.16x–0.5x；调用、属性、数组约 0.1x；闭包、 + 递归、集合、异常、多态对象约 0.01x–0.05x；最差的 + `generic-call-fallback` 为 0.0009x，而且只有解释器的 0.14x。 +3. **最大的单点损耗是 JIT 自身的防御性开销，不是缺少优化。** + - `qjsjit_validate_helper_frame_state`：每次 helper 调用都做 + ~30 项身份/ABI 校验，外加 `qjsjit_valid_pc` 的**线性字节码扫描** + （回边之后从字节码开头重扫）。它在 `objects-polymorphic` 中占 + 63% 自身时间，在真实的 `generic-call-fallback` 中占 61%。 + 把 PC 校验变成 O(1) 的诊断实验分别带来 **1.71x** 和 **1.29x**， + 其余场景持平，checksum 一致。 + - 每次调用的簿记：`record_hot`/`acquire_entry`/`native_enter`/ + `native_exit` 四次跨 C/Rust 回调、两次 `Instant::now()`、哈希表 + 查找、`record_benefit`，以及每 32 次调用一次的全量 `maintenance`。 + `mixed-quotes` 的排序回调中 `maintenance` 占 37.5%。 +4. **覆盖面比 V8 窄一个数量级。** 约 255 个 opcode 中有 124 个被 + policy 拒绝，包括全部闭包变量访问、`push_this`、`throw`、`typeof`、 + `in`/`instanceof`、`for_of`/`for_in`、`fclosure`、try/catch、 + `arguments`。含其中任一 opcode 的函数整体停在解释器。V8 Sparkplug + 和 JSC Baseline 覆盖全部字节码。 +5. **没有通用的原生到原生调用约定。** 除了受限的 Int32/Bool 叶函数 + 直调和有界内联，所有调用都经过 + `JS_JitHelperCall → JS_Call → JS_CallInternal → 回调 → 原生入口`。 + 递归因此被 policy 禁止进入原生层；强制进入 Tier 2 后 + `fibonacci-recursive` 反而比解释器慢 7.8x。 +6. 在这些之后，才轮到 V8 TurboFan/Maglev、JSC DFG/FTL 级别的优化 + 编译能力：冗余 guard 与检查消除、LICM、load/store 转发、边界检查 + 消除、多态内联缓存、逃逸分析和分配下沉、通用内联。 + `property-heavy`、`call-heavy`、`int32array-traversal` 已经在原生层， + 但每次迭代仍比 Bun 慢约 10x，差距就在这里。 + +## 2. 测量协议缺陷与修正 + +`benchmarks/run.rs` 的 `bun_child` 用 `bun -e