diff --git a/.claude/agents/gate-runner.md b/.claude/agents/gate-runner.md index f221079a..52094447 100644 --- a/.claude/agents/gate-runner.md +++ b/.claude/agents/gate-runner.md @@ -27,8 +27,8 @@ five that CI does not run. The brief's `phase:` picks one. -- **`compile`** — `make check`; `make check-js` if any JS changed; `make - check-md` if any `.md` changed; `make check-actions` if a workflow changed; +- **`compile`** — `make check`; `make check-js` if any JS changed; + `make check-md` if any `.md` changed; `make check-actions` if a workflow changed; `make build`. Fast, and the conductor waits on it: reviewers are not spawned on code that does not compile. Stop at the first failure of `make check` or `make build`. @@ -54,8 +54,8 @@ Before the first target, compare the toolchain with what CI pins — `node --version` against `.node-version` (and `node-version:` in `ci.yml`), `odin version` if the brief names one. On a mismatch: -- Look for the pinned version before saying it is absent: `ls /opt`, `type -a - node`, the version manager on the box. "Not on `PATH`" is not "not +- Look for the pinned version before saying it is absent: `ls /opt`, `type -a node`, + the version manager on the box. "Not on `PATH`" is not "not installed". - Found → run every gate with it first on `PATH` and report `ENV MISMATCH: default node , pinned — ran with from `. diff --git a/.claude/agents/oracle-test-auditor.md b/.claude/agents/oracle-test-auditor.md index 043aae40..6ac29bad 100644 --- a/.claude/agents/oracle-test-auditor.md +++ b/.claude/agents/oracle-test-auditor.md @@ -1,6 +1,6 @@ --- name: oracle-test-auditor -description: Judges a diff against its oracle (the project's first ranking criterion) and whether its tests would catch a revert — real Node 22+/24 for the Node profile, recorded Chrome output for the Window profile; oracle cases, Odin tests for what no oracle can see, recorded mutations, and gates wired into CI. Use on every code change under review; it is the owner of criterion 1 and of the test-coverage gate, including any moved `known-lava-gaps.txt`, deleted or skipped test, or declared deviation. +description: Judges a diff against its oracle (the project's first ranking criterion) and whether its tests would catch a revert — real Node 26+ (the `.nvmrc` floor) for the Node profile, recorded Chrome output for the Window profile; oracle cases, Odin tests for what no oracle can see, recorded mutations, and gates wired into CI. Use on every code change under review; it is the owner of criterion 1 and of the test-coverage gate, including any moved `known-lava-gaps.txt`, deleted or skipped test, or declared deviation. tools: Read, Grep, Glob, Bash model: opus --- @@ -23,11 +23,11 @@ or tooling change), say so, skip Part 1, and do Part 2 only. ## Step 0 — classify every touched surface -| Profile | What lives there | Oracle | -| ------- | ---------------- | ------ | -| **Node** | `process`, `require`, `Buffer`, `node:*` modules, the `lava` CLI | real `node` 22+/24, run now | +| Profile | What lives there | Oracle | +| ---------- | -------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | +| **Node** | `process`, `require`, `Buffer`, `node:*` modules, the `lava` CLI | real `node` 26+ (`.nvmrc`), run now | | **Window** | anything a page can reach: DOM, `window`/`document`, page `fetch`/XHR, `location`, storage, events | recorded Chrome output (roadmap stage 0); real Chrome when available | -| **Shared** | Web API code both profiles load (`url.js`, `encoding.js`, streams, timers, `fetch` core) | **both** — each profile's oracle on its own entry point | +| **Shared** | Web API code both profiles load (`url.js`, `encoding.js`, streams, timers, `fetch` core) | **both** — each profile's oracle on its own entry point | How Window differs, so you do not grade it against Node by habit: @@ -46,8 +46,8 @@ risk in `docs/LARVA.md`, not a finding — unless the diff makes them worse. ## Node profile method -1. Pin the oracle: `NODE="${NODE_BIN:-node}"; "$NODE" -v`. CI runs 24; a major - outside 22/24 makes the result advisory. Report the version you ran. +1. Pin the oracle: `NODE="${NODE_BIN:-node}"; "$NODE" -v`. CI runs the `.nvmrc` pin; a major + below the `.nvmrc` floor makes the result advisory. Report the version you ran. 2. Write a probe and run it under both: ```sh @@ -194,15 +194,15 @@ name the missing tests concretely. deny. 10. **Would a mutation fail it?** For every test the diff adds or changes, name - the mutation it should not survive — the line to delete or invert — and say - whether it actually would. This is where decorative tests are caught, and - presence of a test is not evidence: in #320 two new tests with confident - comments both survived the mutation they claimed to pin. Ask specifically - whether the assertion would still hold with the code under test removed, and - whether it observes state it can actually reach (allocations made under - `runtime.default_context()` are invisible to a caller's tracking allocator; - the runner does not fail on leaks — `ODIN_TEST_FAIL_ON_BAD_MEMORY` is false; - `/proc/self/fd` is noise under the default thread-per-core runner). + the mutation it should not survive — the line to delete or invert — and say + whether it actually would. This is where decorative tests are caught, and + presence of a test is not evidence: in #320 two new tests with confident + comments both survived the mutation they claimed to pin. Ask specifically + whether the assertion would still hold with the code under test removed, and + whether it observes state it can actually reach (allocations made under + `runtime.default_context()` are invisible to a caller's tracking allocator; + the runner does not fail on leaks — `ODIN_TEST_FAIL_ON_BAD_MEMORY` is false; + `/proc/self/fd` is noise under the default thread-per-core runner). 11. **Contract comments** (`CLAUDE.md` §4/§5) on new user-visible surfaces: is the `Node:` line backed by a real probe, and does `Deviates:` name the test that pins it? A contract asserting behavior no test holds is a finding. diff --git a/.claude/agents/quality-docs-auditor.md b/.claude/agents/quality-docs-auditor.md index 030533bd..b3008772 100644 --- a/.claude/agents/quality-docs-auditor.md +++ b/.claude/agents/quality-docs-auditor.md @@ -43,7 +43,7 @@ Ask of each structural change: _is there a version of this with zero new concept ## Lens 3 — altitude and size - **Layer**: engine/string/view/host ABI → `pkg/jsc`; Node Buffer codecs → - `pkg/runtime/buffer*.odin`; HTTP → `http.odin` + `js/internal/http.js`; loop + `pkg/runtime/buffer*.odin`; HTTP → `http.odin` + `js/node/http.js`; loop mechanics → `pkg/runtime/eventloop`; VM pooling and page/VM lifecycle → `vm_pool.odin` and the page layer, not sprinkled through `eval`. Logic in the wrong package is a P1 even if it works — `environment.odin` accumulating a diff --git a/.claude/agents/regression-hunter.md b/.claude/agents/regression-hunter.md index ba3a44c5..5f026082 100644 --- a/.claude/agents/regression-hunter.md +++ b/.claude/agents/regression-hunter.md @@ -43,7 +43,7 @@ to ship, because the diff shows what arrived and hides what left. refactor — a dual path that outlives its migration becomes permanent. - **JS ↔ Odin contract drift**: shared constants, array index layouts, property names, encoding assumptions must move together - (`pkg/runtime/http.odin` ↔ `js/internal/http.js` parse layouts are the classic + (`pkg/runtime/http.odin` ↔ `js/node/http.js` parse layouts are the classic case; `ORDER_*` in `dns.odin` ↔ `dns.js` is another). - A registration/inject path that misses a platform branch or a backend (proactor vs readiness; io_uring vs epoll). diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a6ac577c..783dc6b3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -58,9 +58,9 @@ jobs: sudo apt-get update sudo apt-get install -y libjavascriptcoregtk-6.0-dev libsqlite3-dev libssl-dev pkg-config - - uses: actions/setup-node@v4 + - uses: actions/setup-node@v7 with: - node-version: '24' + node-version-file: .nvmrc # Bun runs the JS gates (`bun fmt`, `bun run js:*`) and installs the dev # dependencies. The version comes from package.json's packageManager so CI @@ -78,6 +78,7 @@ jobs: with: token: ${{ secrets.GITHUB_TOKEN }} llvm-version: '18' + release: 'dev-2026-11a' - name: Type-check if: needs.scope.outputs.scope == 'code' || needs.scope.outputs.scope == 'full' @@ -115,9 +116,9 @@ jobs: sudo apt-get update sudo apt-get install -y libjavascriptcoregtk-6.0-dev libsqlite3-dev libssl-dev pkg-config - - uses: actions/setup-node@v4 + - uses: actions/setup-node@v7 with: - node-version: '24' + node-version-file: .nvmrc # Bun runs the JS gates (`bun fmt`, `bun run js:*`) and installs the dev # dependencies. The version comes from package.json's packageManager so CI @@ -134,6 +135,7 @@ jobs: with: token: ${{ secrets.GITHUB_TOKEN }} llvm-version: '18' + release: 'dev-2026-11a' - name: Build run: make build @@ -216,12 +218,10 @@ jobs: - name: Pages bench smoke run: make test-bench-pages-smoke - # Benchmarks: print the node-vs-Lava ratio table in the job log. Report-only for - # now (never fails) because the bench/thresholds.json caps are uncalibrated - # starters — once the ratios here are observed and the caps tightened, flip this - # to `make bench-gate` to enforce regressions. See bench/README.md. - - name: Benchmarks (report-only) - run: make bench + # Benchmarks: enforce the per-benchmark lava/node ratio caps in + # bench/thresholds.json (median of 3 launches per side via --gate, #362). + - name: Benchmark gate + run: make bench-gate # MUTATION GATE. Breaks the code each recorded test claims to pin and requires # the test to go red. It exists because three tests in #321 passed for the @@ -255,9 +255,9 @@ jobs: sudo apt-get update sudo apt-get install -y libjavascriptcoregtk-6.0-dev libsqlite3-dev libssl-dev pkg-config - - uses: actions/setup-node@v4 + - uses: actions/setup-node@v7 with: - node-version: '24' + node-version-file: .nvmrc # Bun runs the JS gates (`bun fmt`, `bun run js:*`) and installs the dev # dependencies. The version comes from package.json's packageManager so CI @@ -274,6 +274,7 @@ jobs: with: token: ${{ secrets.GITHUB_TOKEN }} llvm-version: '18' + release: 'dev-2026-11a' - name: Test (mutation gate) run: make test-mutation SHARD=${{ matrix.shard }}/3 ${{ needs.scope.outputs.scope != 'full' && 'CHANGED=1' || '' }} diff --git a/.nvmrc b/.nvmrc new file mode 100644 index 00000000..1dda03dd --- /dev/null +++ b/.nvmrc @@ -0,0 +1 @@ +26.13.0 diff --git a/CLAUDE.md b/CLAUDE.md index e2162681..cd689984 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -21,7 +21,8 @@ Layering (never blur it): 1. **Oracle conformance.** For the browser (`Window`/DOM/network) layer, recorded Chrome output is the oracle ([docs/LARVA.md](docs/LARVA.md)); for the base - runtime (`Node` profile), Node 22+/24 is the oracle. The observable surface — + runtime (`Node` profile), Node 26+ is the oracle (the `.nvmrc` floor; 28 is the + planned final pin). The observable surface — names, argument coercion order, defaults, thrown error `code`/`name`/`message`, event order, return types — matches the oracle, not our reading of the docs. Deviation is allowed **only** when it buys measured speed or memory control @@ -87,8 +88,8 @@ Use the `odin-sdk-scout` agent for this search — it reads the real SDK source. ```sh make check # Odin type-check (incl. windows/darwin cross-target front-end) -make check-js # bun fmt --check + orphan-JS + node:test over scripts/ + primordials ratchet -make check-md # markdownlint over the repo's docs (make fix-md auto-fixes) +make check-js # bun fmt --check + JS module graph + node:test over scripts/ + primordials ratchet +make check-md # bun fmt + markdownlint over the repo's docs (make fix-md auto-fixes) make check-actions # actionlint over .github/workflows make build # ./scripts/build.sh → bin/lava make test # Odin unit tests + oracle suites @@ -178,7 +179,7 @@ the comment block verbatim, so this is the doc source, not decoration. // and the caller must fall back to JSObjectMakeFunctionWithCallback. // Node: // setTimeout/setInterval report `.length` 2, every other global 1 — verified -// against node 24, not read off the docs. +// against node 26, not read off the docs. // Deviates: // The C-API fallback reports 0 and is not repairable through the public API. // Pinned by tests/node-compat/cases/56-native-function-arity.js. @@ -198,16 +199,26 @@ Rules that keep this from becoming ceremony: - Internal helpers with no Node counterpart keep the plain why-comment. Do not retrofit existing code; this applies to new and changed surfaces. -## 5. Embedded JS conventions (`pkg/runtime/js`, incl. the top-level preludes) +## 5. Embedded JS conventions (`pkg/runtime/js`, incl. the bootstrap preludes) -- Route through `require('primordials')` — internal modules run alongside user +- **The directory decides visibility.** `js/node/` holds every module user code may + `require`, at the path of its specifier (`node/stream/web.js`); `js/internal/` + holds what only builtins may require (`require('internal/mime')`); `js/bootstrap/` + holds the loader, microtasks, `esm.js` and the console prelude, which are not + modules. A new file is embedded and wired by existing in the right directory — + there is no file list to edit. `scripts/check-js-graph.mjs` fails on a non-literal + or unresolvable `require`, an unreachable file, an unembedded directory, or a + top-level require cycle (a `require` inside a function is the sanctioned late + binding). + +- Route through `require('internal/primordials')` — internal modules run alongside user code that can poison prototypes. `make check-primordials` is a **ratchet**: a hardened file (baseline 0) rejects any new pollutable call. `UPDATE=1` only to _lower_ a baseline. - One exception, and do not add a second: `js/internal/esm.js` is evaluated + One exception, and do not add a second: `js/bootstrap/esm.js` is evaluated standalone and deliberately kept off the module table, so user code cannot reach the transform — which also denies it a `require`. It takes the table as a factory - argument instead (`loader.js` publishes it on the resolver, `globals.odin` passes + argument instead (`bootstrap/loader.js` publishes it on the resolver, `globals.odin` passes it in and fails closed without it). The exception is affordable there because the file emits executed source and had to be hardened regardless. - The ratchet parses with acorn and counts **four** classes, each baselined @@ -218,18 +229,53 @@ Rules that keep this from becoming ceremony: getter such as `TypedArrayPrototypeGetBuffer`), and `global` (a replaceable global read live instead of captured at module-eval, which the loader runs before user code). + **`Function.prototype.call/apply/bind` are locked** realm-wide before user code + (`lockIntrinsics`, run by `bootstrap/loader.js`; #450): non-configurable + accessors, so `Function.prototype.call = x` throws in both modes while + `f.call = x` on an ordinary function still defines an own property, as in node. + That is a §1 deviation, pinned by `cmd/lava/function_call_lock_test.odin`. What + it buys: `Function.prototype.call = x` can no longer steer every `.call` at once. + The held functions are locked too (#494): `call`/`apply`/`bind` and every + method a primordials wrapper closes over (the set the `callerN`/fixed-arity + factories record, so a newly wrapped method is covered automatically) are + non-extensible, so neither an own `call` nor a re-pointed prototype on them + reaches a wrapper's `fn.call` or events.js's `apply.call(listener, …)`. Also a + §1 deviation — those functions answer as under `node --frozen-intrinsics` + (isExtensible false; own property, defineProperty and setPrototypeOf throw) but + are not frozen — pinned by the same test. Intrinsics no wrapper holds keep + node's extensibility. `call.call(fn, …)` routing was measured and rejected: + JSC inlines only `fn.call` (caller1 ~1 ns vs ~55 ns, Linux arm64), costing + `new URL` 2.64x and `URLSearchParams.get` 3.81x end to end. + What it does NOT buy: `.call` on a function user code can HOLD outside that + set — every caller-supplied callback (own `call`, re-pointed prototype), every + intrinsic captured raw OUTSIDE primordials and invoked with `.call` + (`toString.call` in structured_clone.js, `u8Fill.call` in buffer.js), and every + live prototype read (`Array.prototype.slice.call(…)`), which plain replacement + already steers. Route those through a primordials wrapper. + The ratchet therefore counts every `.call`/`.apply` (`invoke`) and `.bind` + (`method`) except on ONE receiver: a strict function declaration the file never + lets go of — exactly one binding of the name in the file, never written, + referenced only as `g(…)` or `g.call/apply/bind(…)` inside its declaring scope, + no direct `eval` in the file (`capturedCallees`, `scripts/lib/primordials-detect.mjs`). + A module-eval capture of an intrinsic or a primordials export is NOT exempt, by + design: a false negative there is a hole. **Module-eval capture is pristine only in a module the loader instantiates BEFORE user - code** — the eager list in `loader.js`, plus `stream`/`stdio` via `installStdio`. A LAZY + code** — the eager list in `bootstrap/loader.js`, plus `stream`/`stdio` via `installStdio`. A LAZY module's factory runs on the user's first `require()`, so a capture there is a live read - in disguise: `var StringG = String` at the top of `net.js` scores 0 and is still - steerable. Take the handle from the loader instead (`require('primordials')`, - `require.pristineBuffer`). The ratchet cannot see this — it exempts every module-eval - capture, which is why #333 shipped. Computed and destructured forms count the same as the dot + in disguise — `var StringG = String` at the top of `net.js` is still steerable. Take the + handle from the loader instead (`require('internal/primordials')`, + `require.pristineBuffer`). The ratchet exempts module-eval reads only in that eager set, + which it derives from the loader's `req()` roots closed over top-level requires + (`scripts/lib/js-graph.mjs`); elsewhere they count as `global`. It used to exempt every + module-eval capture, which is why #333 shipped. Computed and destructured forms count the same as the dot form, so `view['buffer']` is not a way to lower a number. - A false positive takes `// primordials-ok` on the line. On a line carrying candidates from **more than one class** the bare marker suppresses nothing — name it, `// primordials-ok: method` (comma-separated list allowed). An - unrecognized class name suppresses nothing, so a typo fails loud. + unrecognized class name suppresses nothing, so a typo fails loud. Either form may + end in a reason after ONE separator, an em dash — + `// primordials-ok: invoke — receiver is our own closure`; `--`, `-` or an + empty reason suppress nothing. - `UPDATE=1` only _lowers_ a baseline. Raising one needs `make check-primordials UPDATE=1 RAISE=--allow-raise` and a stated reason (a newly scanned file, or a new class). A corrupt baseline fails hard rather than @@ -264,19 +310,33 @@ Rules that keep this from becoming ceremony: readable rather than reassuring. It earned that caveat: `encoding.js` stood at 0 while `units.buffer` went through the live `%TypedArray%.prototype.buffer` getter, which is why the accessor class exists. And a high number is not - automatically work: `primordials.js` reads 9 in `invoke` **by construction**, - because every `callerN` wrapper is a `fn.call` — its `global` column is 1, - since the module-eval rule already exempts the capture table. + automatically work: `primordials.js`'s `global` column is 0 because the + module-eval rule exempts the capture table. Its 9 `invoke` sites, though, are + real: every `callerN` wrapper is a `fn.call` on a captured intrinsic, which the + lock above does not protect (an own `call` on the intrinsic steers it). - Pick the primordial by **arity**. `callerN` (`ArrayPrototypePush`) carries an `arguments` switch and belongs at cold call sites; per-element loops use the fixed-arity wrappers (`ArrayPrototypePush1`/`Push2`) or plain indexed writes into a preallocated null-prototype array. Getting this wrong cost 1.43x on the TextDecoder JS path. +- New and rewritten code writes constructors as `class` (`class Blob`, + `class File extends Blob`), whatever node does internally. **Exception, and do + not add to it without a probe:** constructors that npm code inherits from with + the pre-ES6 pattern `function S() { Base.call(this) }` + `util.inherits(S, Base)` + stay function constructors. That call throws on a class, so a class there breaks + working packages, not just a message. Today these are `EventEmitter`, `Stream`, + `Readable`, `Writable`, `Duplex`, `Transform`, `PassThrough`, and `Buffer` + (callable without `new`). For every other class, the TypeError text for a call + without `new` is JSC's own wording: an engine trait, declared with `@deviates`, + pinned by a Lava-only assertion, and never printed by an oracle case. Method + enumerability stays explicit wherever it is observable today. - Nothing transient lands on `globalThis`; natives arrive as the factory's fourth argument. - Errors follow the **profile** ([docs/LARVA.md](docs/LARVA.md) decision 1). Node profile: Node-shaped coded errors (`ERR_INVALID_ARG_TYPE`, `ERR_OUT_OF_RANGE`) - with Node's exact message template. Window profile (anything a page can reach): + with Node's exact message template, built through `require('internal/errors')` + (`ERR_*`, and `nodeError`/`nativeError` for node's JS/C++ shapes), never a local + copy. Window profile (anything a page can reach): plain `TypeError`/`RangeError`/`DOMException` with Chrome's message and no `code`, byte results as plain `Uint8Array`, and no `process`/`require`/`Buffer` reachable. Shared Web API code (`url.js`, `encoding.js`, …) takes its error @@ -298,7 +358,7 @@ Rules that keep this from becoming ceremony: * @returns {string} * @throws {TypeError} ERR_INVALID_ARG_TYPE — options is a non-null scalar. * @node Options are validated BEFORE the input is converted; `decode(5, 5)` - * reports the options (node 24, verified). + * reports the options (node 26, verified). * @deviates none */ ``` diff --git a/Makefile b/Makefile index a97ebab7..5a412992 100644 --- a/Makefile +++ b/Makefile @@ -25,7 +25,7 @@ endif SOURCE ?= console.log('hello from Lava') FILE ?= -.PHONY: test-stdio help bootstrap-windows-deps build-sqlite-windows build run eval check check-cli check-runtime check-js fix-js check-md fix-md check-actions check-primordials test-scripts test-property test-mutation check-jsc check-native native-deps test test-all test-lava test-lava-nohostfn test-odin-serial api-surface bun-buffer-tests test-compat test-compat-lava test-compat-lava-strict test-odin test-eventloop-odin test-runtime-odin test-sqlite-odin test-sqlite-node test-sqlite-lava test-fs-node test-fs-lava test-eventloop-node test-eventloop-lava test-fetch-smoke test-net-smoke test-http-smoke test-https-smoke test-multicore-smoke test-zerocopy-smoke test-bench-pages-smoke bench bench-gate bench-http bench-pages fmt clean +.PHONY: gates test-stdio help bootstrap-windows-deps build-sqlite-windows build run eval check check-cli check-runtime check-js fix-js check-md fix-md check-actions check-primordials test-scripts test-property test-mutation check-jsc check-native native-deps test test-all test-lava test-lava-nohostfn test-odin-serial api-surface bun-buffer-tests test-compat test-compat-lava test-compat-lava-strict test-odin test-eventloop-odin test-runtime-odin test-sqlite-odin test-sqlite-node test-sqlite-lava test-fs-node test-fs-lava test-eventloop-node test-eventloop-lava test-fetch-smoke test-net-smoke test-http-smoke test-https-smoke test-multicore-smoke test-zerocopy-smoke bench bench-gate bench-http fmt clean test-bench-pages-smoke bench-pages help: @printf '%s\n' 'Lava commands' @@ -43,9 +43,9 @@ help: @printf '%s\n' ' make check Type-check Odin packages (incl. windows/darwin front-ends)' @printf '%s\n' ' make check-cli Type-check cmd/lava only' @printf '%s\n' ' make check-runtime Type-check runtime packages only' - @printf '%s\n' ' make check-js Vite+ fmt/lint + orphan-JS + scripts tests + primordials + global-replace' + @printf '%s\n' ' make check-js Vite+ fmt/lint + JS module graph + scripts tests + primordials + global-replace' @printf '%s\n' ' make fix-js Auto-fix JS formatting/lint' - @printf '%s\n' ' make check-md Markdown lint (markdownlint-cli2)' + @printf '%s\n' ' make check-md Markdown fmt + lint' @printf '%s\n' ' make fix-md Auto-fix markdown lint' @printf '%s\n' ' make check-actions actionlint over .github/workflows' @printf '%s\n' ' make check-primordials Prototype-pollution ratchet (UPDATE=1 to lower; RAISE=--allow-raise)' @@ -73,6 +73,7 @@ help: @printf '%s\n' ' make test-property Differential property tests (PROPERTY_RUNS=N)' @printf '%s\n' ' make test-scripts node:test over scripts/ only (also inside check-js)' @printf '%s\n' ' make test-mutation Mutation gate (CHANGED=1: only what the diff affects; FILTER=name)' + @printf '%s\n' ' make gates Run routed gates for the current diff (stops on first failure)' @printf '%s\n' ' make bun-buffer-tests Ported Buffer cases node-vs-Lava' @printf '%s\n' '' @printf '%s\n' 'Smokes (bind local ports)' @@ -92,6 +93,7 @@ help: @printf '%s\n' ' SKIP_KNOWN_LAVA_GAPS=1 Skip paths listed in the matching known-lava-gaps.txt' @printf '%s\n' ' NODE_BIN=/path/to/node Override the Node oracle binary' @printf '%s\n' ' LAVA_BIN=/path/to/lava Override the Lava binary under test' + @printf '%s\n' ' CASE_TIMEOUT_SEC=30 Per-case wall-clock timeout in seconds (default 30)' bootstrap-windows-deps: $(RUNSCRIPT) ./scripts/bootstrap-windows-deps.sh @@ -150,7 +152,7 @@ check-js: fix-js: bun run js:fix -# Markdown lint over the repo's own docs (config + rationale for the two disabled rules: +# Markdown format (bun fmt, bunfig.toml) + lint over the repo's own docs (lint config + rationale for the two disabled rules: # .markdownlint-cli2.jsonc). Catches MD040 and friends locally instead of in review. check-md: bun run md:check @@ -178,6 +180,12 @@ check-primordials: test-scripts: node --test 'scripts/**/*.test.mjs' +# Honest gate runner (#360): routes the current diff through runtime/gates/route-gates.mjs, +# runs each gate without piping into tail, stops on the first failure, and prints a +# summary table with real exit codes. Pass PATHS="file1 file2" or ARGS="--all" to override. +gates: + node scripts/run-gates.mjs $(ARGS) $(PATHS) + # MUTATION GATE. Applies each entry in tests/mutation-manifest.json to production # source and requires the named test to go RED. CLAUDE.md §6 has always demanded # this by hand; doing it by hand depends on remembering to, and three tests in #321 diff --git a/README.md b/README.md index f5890b39..6ef6b16c 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ for how the runtime pieces fit together. ## Requirements -- Odin +- Odin (CI pins `dev-2026-11a` with LLVM 18) - JavaScriptCore development headers and libraries - SQLite development headers and libraries (for `node:sqlite`) - OpenSSL development headers and libraries (for `fetch` HTTPS/TLS). The roadmap @@ -40,7 +40,7 @@ fetch JavaScriptCore/WebKit and the SQLite amalgamation into the ignored `.deps/ cache. `scripts/build-sqlite-windows.sh` then compiles SQLite into `build/sqlite3.lib`; WebKit is linked from `.deps/webkit/bun-webkit/lib`. -Node 24+ and Bun are needed for the JavaScript gates — run `bun install` once. +Node 26+ (`26.13.0` pinned in [`.nvmrc`](.nvmrc); oracle suites refuse an older major) and Bun are needed for the JavaScript gates — run `bun install` once. They are dev-only (`markdownlint-cli2`, `acorn`/`acorn-walk` for the prototype-pollution ratchet, `fast-check` for `make test-property`); `bin/lava` links none of them. `make check-js` is an always-gate and now fails without them. diff --git a/agents/prompts/odin-feature.md b/agents/prompts/odin-feature.md index 4e71b58c..58c1bf39 100644 --- a/agents/prompts/odin-feature.md +++ b/agents/prompts/odin-feature.md @@ -24,7 +24,7 @@ Read `CLAUDE.md` and `docs/LARVA.md` first. Reuse index: Decide which profile the surface belongs to (`docs/LARVA.md` decision 1) — it picks the oracle for every later phase: -- **Node** (`process`, `require`, `Buffer`, `node:*`) → real `node` 22+/24. +- **Node** (`process`, `require`, `Buffer`, `node:*`) → real `node` 26+ (`.nvmrc`). - **Window** (anything a page can reach) → recorded Chrome output. No `process`, `require` or `Buffer`; plain `TypeError`/`DOMException` with Chrome's messages and no `code`; plain `Uint8Array`. diff --git a/agents/prompts/pr-gate-reference/gates.md b/agents/prompts/pr-gate-reference/gates.md index 2aeafb74..fdf0ab8b 100644 --- a/agents/prompts/pr-gate-reference/gates.md +++ b/agents/prompts/pr-gate-reference/gates.md @@ -7,11 +7,12 @@ disagree with this table, they win — and fix this table. every routed smoke, on both I/O backends. Nothing else runs them: `/pr-gate` is invoked by a human, locally, and `--review-only` (which executes no `make` target and reads CI's conclusions instead) is for reviewing from a machine -without the toolchain. **Five** routed targets are **not** in CI — `make bun-buffer-tests`, -`make api-surface`, `make test-compat-lava-strict`, `make bench-gate`, and -`make bench-http` — so a diff that routes to one of those needs a local run, or a -new CI step if it should be enforced. (`make bench` is in CI but **report-only** -and cannot fail.) (`make test-scripts` runs in CI inside `make check-js`; +without the toolchain. **Four** routed targets are **not** in CI — `make bun-buffer-tests`, +`make api-surface`, `make test-compat-lava-strict`, and `make bench-http` — +so a diff that routes to one of those needs a local run, or a new CI step if +it should be enforced. (`make bench-gate` runs in CI on Linux `ubuntu-latest` +and fails on cap breaches; `make bench` is the local report-only form.) +(`make test-scripts` runs in CI inside `make check-js`; `make test-property` is its own CI step since batching took it from 64s to ~1s; `make test-mutation` is its own CI step because it rebuilds `bin/lava` once per embedded-JS mutation.) @@ -40,44 +41,44 @@ disagree with this file, they win — fix this file. `make fmt` (`odin strip-semicolon`) before committing Odin. `make check-js` whenever any `.js`/`.mjs`/`.cjs` under `pkg/runtime/js`, `tests`, -`scripts`, or `bench` changed — it runs format, lint, orphan-JS detection, and the +`scripts`, or `bench` changed — it runs format, lint, the JS module-graph check (`check-js-graph`), and the primordials ratchet. -`make check-md` whenever any `.md` changed — markdownlint over the repo's own docs -(config and the rationale for its two disabled rules: `.markdownlint-cli2.jsonc`). +`make check-md` whenever any `.md` changed — `bun fmt --check` plus markdownlint over the +repo's own docs (lint config and the rationale for its two disabled rules: `.markdownlint-cli2.jsonc`). `make check-actions` whenever any `.github/workflows/*.yml` changed — actionlint catches what a YAML parse cannot: a context used where it is not available, a bad `needs`, shellcheck errors inside `run:`. ## By path -| Changed path | Additionally required | -| --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `pkg/runtime/buffer*.odin`, `typed_array.odin`, `pkg/jsc/**` | `make test-odin`, `make bun-buffer-tests`, `make api-surface`, `make bench` | -| `pkg/runtime/http.odin`, `js/internal/http.js`, `picohttpparser/**` | `make test-http-smoke` (runs both proactor and readiness backends), `make bench-http` | -| `pkg/runtime/net.odin`, `net_other.odin`, `js/internal/net.js` | `make test-net-smoke`, `make test-zerocopy-smoke` | -| `pkg/runtime/tls*.odin`, `tls_server.odin`, `js/internal/https.js` | `make test-https-smoke` (drives both backends internally) | -| `pkg/runtime/fetch*.odin`, `js/internal/fetch.js` | `make test-fetch-smoke` | -| `pkg/runtime/eventloop/**` | `make test-eventloop-odin`, `make test-eventloop-lava` | -| `pkg/runtime/fs*.odin`, `os_*.odin` | `make test-fs-lava` | -| `pkg/std/sqlite/**`, `sqlite.odin`, `js/internal/sqlite.js` | `make test-sqlite-odin`, `make test-sqlite-lava` | -| `pkg/runtime/dns.odin`, `fetch_dns.odin`, `js/internal/dns*.js` | `make test-compat-lava`, `make test-fetch-smoke` | -| `pkg/runtime/workers*.odin` | `make test-multicore-smoke` | -| `pkg/runtime/stdio*.odin`, `js/internal/stdio.js`, `tests/stdio/**` | `make test-stdio` — pins the non-blocking-fd retry loop that `console.log`, `lava eval` and `process.stdout.write` all share, plus the lifecycle/backpressure deviations in node's PIPE shape (the oracle harness only ever gives a case a FILE) | -| `js/internal/fs.js` | `make test-fs-lava` — the JS layer owns every observable `node:fs` read semantic since #330 (Buffer re-tag, encoding decode, error taxonomy, path validation). It routes only to the generic `js/**.js` row otherwise, so a change there passes `check-js` and `test-compat-lava` while breaking every `run-fs-oracle.sh` case | -| `js/internal/stream.js` | `make test-http-smoke`, `make test-net-smoke`, `make test-stdio` — nothing else routes here, and this is the base layer `net`/`http`/`stdio` compose on. Added after #326: a change to the `write()` accept set passed `check-js` and `test-compat-lava` while stalling `pipe()` forever, because a DataView made `writableLength` NaN and `'drain'` could never fire. The suites that would have caught it were not on anyone's list | -| `require.odin`, `module_resolution.odin`, `js/internal/{loader,esm}.js` | `make test-compat-lava-strict`; for `loader.js` ALSO `make test-fs-lava`, `make test-http-smoke`, `make test-net-smoke`, `make test-https-smoke`, `make test-stdio` — since #333 it hands `require.pristineBuffer` to every lazy module, so a wrong snapshot breaks framing in suites `test-compat-lava-strict` never runs | -| `crypto.odin`, `js/internal/crypto.js` | `make test-odin`, `make test-compat-lava`, `make api-surface` | -| `pkg/runtime/js/**/*.js` (any under the scan root, `console.js` included; use `**/*` not bare `**.js`) | `make check-js`, `make check-primordials`, `make test-compat-lava` | -| `js/internal/url.js` | `make bench` — `bench/micro/url.js` exists since 2026-07-30; before it, a +6% to +20% `new URL` regression passed every gate and was found by hand in review | -| `js/internal/{encoding,url,buffer}.js`, `pkg/runtime/buffer*.odin` | `make test-property` — differential property tests (fast-check generates the corpus; hand-picked oracle cases missed edge cases here twice, and the batched suite found a utf-16le divergence at 5000 inputs). `PROPERTY_RUNS=N` for a deeper local sweep | -| `scripts/**` | `make test-scripts` — node:test over the build tooling | -| any file named in `tests/mutation-manifest.json` as a `source` | `make test-mutation CHANGED=1` — re-applies the recorded breaks this diff can affect (changed `source`, changed gate file, new or edited entry; all of them when the runner, `Makefile` or `build.sh` changed) and requires the named test to go red. The full `make test-mutation` is the release-time form: `CHANGED=1` cannot see a shared helper a gate imports. Changing one of these files without running it means the recorded mutation may no longer describe the code, which the runner reports as STALE rather than skipping | -| `pkg/runtime/globals.odin`, `runtime.odin`, `errors.odin` | `make test-odin`, `make test-compat-lava` | -| `pkg/runtime/vm_pool.odin`, VM/context binding in `pkg/jsc/bindings_*.odin`, `cmd/lava/vm_pool_test.odin` | `make test-odin`, `make test-odin-serial` (the timerfd/fd census only means something on one runner thread), `make test-lava-nohostfn`, and a `bench/jsc-vm` run (scenarios A vs C at least) when the change could move per-page cost — the numbers in `docs/LARVA.md` decision 2 and the `vm_pool.odin` header come from it, and nothing in CI re-measures them | -| `bench/jsc-vm/**` | `odin build bench/jsc-vm` — no Makefile target or CI step builds it, so a broken bench is otherwise invisible | -| `bench/**`, `bench/thresholds.json` | `make bench-gate` — and a stated reason if a cap was loosened | -| `Makefile`, `scripts/**`, `.github/workflows/**` | `__ALL__` fallback via `route-gates.mjs` — run L0+L1 (and L2 if mutation/CI surface touched), end to end | -| `pkg/runtime/*_test.odin` / eventloop Odin tests | `make test-runtime-odin` when present in Makefile — not every Odin unit path is covered by `make test-odin` alone | +| Changed path | Additionally required | +| -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `pkg/runtime/buffer*.odin`, `typed_array.odin`, `pkg/jsc/**` | `make test-odin`, `make bun-buffer-tests`, `make api-surface`, `make bench-gate` | +| `pkg/runtime/http.odin`, `js/node/http.js`, `js/internal/http/**`, `picohttpparser/**` | `make test-http-smoke` (runs both proactor and readiness backends), `make bench-http` | +| `pkg/runtime/net.odin`, `net_other.odin`, `js/node/net.js` | `make test-net-smoke`, `make test-zerocopy-smoke` | +| `pkg/runtime/tls*.odin`, `tls_server.odin`, `js/node/https.js` | `make test-https-smoke` (drives both backends internally) | +| `pkg/runtime/fetch*.odin`, `js/internal/fetch.js`, `js/internal/fetch/*.js` | `make test-fetch-smoke` | +| `pkg/runtime/eventloop/**` | `make test-eventloop-odin`, `make test-eventloop-lava` | +| `pkg/runtime/fs*.odin`, `os_*.odin` | `make test-fs-lava` | +| `pkg/std/sqlite/**`, `sqlite.odin`, `js/node/sqlite.js` | `make test-sqlite-odin`, `make test-sqlite-lava` | +| `pkg/runtime/dns.odin`, `fetch_dns.odin`, `js/node/dns.js`, `js/node/dns/promises.js` | `make test-compat-lava`, `make test-fetch-smoke` | +| `pkg/runtime/workers*.odin` | `make test-multicore-smoke` | +| `pkg/runtime/stdio*.odin`, `js/internal/stdio.js`, `tests/stdio/**` | `make test-stdio` — pins the non-blocking-fd retry loop that `console.log`, `lava eval` and `process.stdout.write` all share, plus the lifecycle/backpressure deviations in node's PIPE shape (the oracle harness only ever gives a case a FILE) | +| `js/node/fs.js` | `make test-fs-lava` — the JS layer owns every observable `node:fs` read semantic since #330 (Buffer re-tag, encoding decode, error taxonomy, path validation). It routes only to the generic `js/**.js` row otherwise, so a change there passes `check-js` and `test-compat-lava` while breaking every `run-fs-oracle.sh` case | +| `js/node/stream.js` | `make test-http-smoke`, `make test-net-smoke`, `make test-stdio` — nothing else routes here, and this is the base layer `net`/`http`/`stdio` compose on. Added after #326: a change to the `write()` accept set passed `check-js` and `test-compat-lava` while stalling `pipe()` forever, because a DataView made `writableLength` NaN and `'drain'` could never fire. The suites that would have caught it were not on anyone's list | +| `require.odin`, `module_resolution.odin`, `js/bootstrap/{loader,esm}.js` | `make test-compat-lava-strict`; for `loader.js` ALSO `make test-fs-lava`, `make test-http-smoke`, `make test-net-smoke`, `make test-https-smoke`, `make test-stdio` — since #333 it hands `require.pristineBuffer` to every lazy module, so a wrong snapshot breaks framing in suites `test-compat-lava-strict` never runs | +| `crypto.odin`, `js/node/crypto.js`, `js/internal/crypto/*.js` | `make test-odin`, `make test-compat-lava`, `make api-surface` | +| `pkg/runtime/js/**/*.js` (any under the scan root — `bootstrap/`, `node/`, `internal/`; use `**/*` not bare `**.js`) | `make check-js`, `make check-primordials`, `make test-compat-lava` | +| `js/{node,internal}/url.js`, `js/internal/{idna,punycode}.js` | `make bench-gate` — `bench/micro/url.js` exists since 2026-07-30; before it, a +6% to +20% `new URL` regression passed every gate and was found by hand in review | +| `js/internal/{encoding,buffer,url,idna,punycode}.js`, `js/node/{url,buffer}.js`, `pkg/runtime/buffer*.odin` | `make test-property` — differential property tests (fast-check generates the corpus; hand-picked oracle cases missed edge cases here twice, and the batched suite found a utf-16le divergence at 5000 inputs). `PROPERTY_RUNS=N` for a deeper local sweep | +| `scripts/**` | `make test-scripts` — node:test over the build tooling | +| any file named in `tests/mutation-manifest.json` as a `source` | `make test-mutation CHANGED=1` — re-applies the recorded breaks this diff can affect (changed `source`, changed gate file, new or edited entry; all of them when the runner, `Makefile` or `build.sh` changed) and requires the named test to go red. The full `make test-mutation` is the release-time form: `CHANGED=1` cannot see a shared helper a gate imports. Changing one of these files without running it means the recorded mutation may no longer describe the code, which the runner reports as STALE rather than skipping | +| `pkg/runtime/globals.odin`, `runtime.odin`, `errors.odin` | `make test-odin`, `make test-compat-lava` | +| `pkg/runtime/vm_pool.odin`, VM/context binding in `pkg/jsc/bindings_*.odin`, `cmd/lava/vm_pool_test.odin` | `make test-odin`, `make test-odin-serial` (the timerfd/fd census only means something on one runner thread), `make test-lava-nohostfn`, and a `bench/jsc-vm` run (scenarios A vs C at least) when the change could move per-page cost — the numbers in `docs/LARVA.md` decision 2 and the `vm_pool.odin` header come from it, and nothing in CI re-measures them | +| `bench/jsc-vm/**` | `odin build bench/jsc-vm` — no Makefile target or CI step builds it, so a broken bench is otherwise invisible | +| `bench/**`, `bench/thresholds.json` | `make bench-gate` — and a stated reason if a cap was loosened | +| `Makefile`, `scripts/**`, `.github/workflows/**` | `__ALL__` fallback via `route-gates.mjs` — run L0+L1 (and L2 if mutation/CI surface touched), end to end | +| `pkg/runtime/*_test.odin` / eventloop Odin tests | `make test-runtime-odin` when present in Makefile — not every Odin unit path is covered by `make test-odin` alone | ## Perf claims @@ -86,7 +87,7 @@ carry numbers: `make bench` (ratio table), `make bench-http` (throughput, latenc memory per idle connection), or a profile. JSC sampling profiler: `JSC_useSamplingProfiler=1` plus a dump directory. -`make bench` is report-only in CI today. `make bench-gate` enforces +`make bench-gate` runs in CI on Linux (`ubuntu-latest`) and enforces `bench/thresholds.json`; run it locally when touching a floor. ## Backends diff --git a/agents/prompts/pr-gate-reference/scoring.md b/agents/prompts/pr-gate-reference/scoring.md index cb22bdab..918656ae 100644 --- a/agents/prompts/pr-gate-reference/scoring.md +++ b/agents/prompts/pr-gate-reference/scoring.md @@ -7,7 +7,7 @@ it is _mergeable_. A change can be fast and still not merge. ### 1. Oracle conformance (primary) -The oracle depends on the profile (`docs/LARVA.md` decision 1): real Node 22+/24 +The oracle depends on the profile (`docs/LARVA.md` decision 1): real Node 26+ (`.nvmrc`) for the Node profile, recorded Chrome output for the Window profile, both for shared Web API code. A diff touching several profiles is graded by its worst. diff --git a/agents/specialists/gate-runner.md b/agents/specialists/gate-runner.md index f221079a..52094447 100644 --- a/agents/specialists/gate-runner.md +++ b/agents/specialists/gate-runner.md @@ -27,8 +27,8 @@ five that CI does not run. The brief's `phase:` picks one. -- **`compile`** — `make check`; `make check-js` if any JS changed; `make - check-md` if any `.md` changed; `make check-actions` if a workflow changed; +- **`compile`** — `make check`; `make check-js` if any JS changed; + `make check-md` if any `.md` changed; `make check-actions` if a workflow changed; `make build`. Fast, and the conductor waits on it: reviewers are not spawned on code that does not compile. Stop at the first failure of `make check` or `make build`. @@ -54,8 +54,8 @@ Before the first target, compare the toolchain with what CI pins — `node --version` against `.node-version` (and `node-version:` in `ci.yml`), `odin version` if the brief names one. On a mismatch: -- Look for the pinned version before saying it is absent: `ls /opt`, `type -a - node`, the version manager on the box. "Not on `PATH`" is not "not +- Look for the pinned version before saying it is absent: `ls /opt`, `type -a node`, + the version manager on the box. "Not on `PATH`" is not "not installed". - Found → run every gate with it first on `PATH` and report `ENV MISMATCH: default node , pinned — ran with from `. diff --git a/agents/specialists/oracle-test-auditor.md b/agents/specialists/oracle-test-auditor.md index 043aae40..6ac29bad 100644 --- a/agents/specialists/oracle-test-auditor.md +++ b/agents/specialists/oracle-test-auditor.md @@ -1,6 +1,6 @@ --- name: oracle-test-auditor -description: Judges a diff against its oracle (the project's first ranking criterion) and whether its tests would catch a revert — real Node 22+/24 for the Node profile, recorded Chrome output for the Window profile; oracle cases, Odin tests for what no oracle can see, recorded mutations, and gates wired into CI. Use on every code change under review; it is the owner of criterion 1 and of the test-coverage gate, including any moved `known-lava-gaps.txt`, deleted or skipped test, or declared deviation. +description: Judges a diff against its oracle (the project's first ranking criterion) and whether its tests would catch a revert — real Node 26+ (the `.nvmrc` floor) for the Node profile, recorded Chrome output for the Window profile; oracle cases, Odin tests for what no oracle can see, recorded mutations, and gates wired into CI. Use on every code change under review; it is the owner of criterion 1 and of the test-coverage gate, including any moved `known-lava-gaps.txt`, deleted or skipped test, or declared deviation. tools: Read, Grep, Glob, Bash model: opus --- @@ -23,11 +23,11 @@ or tooling change), say so, skip Part 1, and do Part 2 only. ## Step 0 — classify every touched surface -| Profile | What lives there | Oracle | -| ------- | ---------------- | ------ | -| **Node** | `process`, `require`, `Buffer`, `node:*` modules, the `lava` CLI | real `node` 22+/24, run now | +| Profile | What lives there | Oracle | +| ---------- | -------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | +| **Node** | `process`, `require`, `Buffer`, `node:*` modules, the `lava` CLI | real `node` 26+ (`.nvmrc`), run now | | **Window** | anything a page can reach: DOM, `window`/`document`, page `fetch`/XHR, `location`, storage, events | recorded Chrome output (roadmap stage 0); real Chrome when available | -| **Shared** | Web API code both profiles load (`url.js`, `encoding.js`, streams, timers, `fetch` core) | **both** — each profile's oracle on its own entry point | +| **Shared** | Web API code both profiles load (`url.js`, `encoding.js`, streams, timers, `fetch` core) | **both** — each profile's oracle on its own entry point | How Window differs, so you do not grade it against Node by habit: @@ -46,8 +46,8 @@ risk in `docs/LARVA.md`, not a finding — unless the diff makes them worse. ## Node profile method -1. Pin the oracle: `NODE="${NODE_BIN:-node}"; "$NODE" -v`. CI runs 24; a major - outside 22/24 makes the result advisory. Report the version you ran. +1. Pin the oracle: `NODE="${NODE_BIN:-node}"; "$NODE" -v`. CI runs the `.nvmrc` pin; a major + below the `.nvmrc` floor makes the result advisory. Report the version you ran. 2. Write a probe and run it under both: ```sh @@ -194,15 +194,15 @@ name the missing tests concretely. deny. 10. **Would a mutation fail it?** For every test the diff adds or changes, name - the mutation it should not survive — the line to delete or invert — and say - whether it actually would. This is where decorative tests are caught, and - presence of a test is not evidence: in #320 two new tests with confident - comments both survived the mutation they claimed to pin. Ask specifically - whether the assertion would still hold with the code under test removed, and - whether it observes state it can actually reach (allocations made under - `runtime.default_context()` are invisible to a caller's tracking allocator; - the runner does not fail on leaks — `ODIN_TEST_FAIL_ON_BAD_MEMORY` is false; - `/proc/self/fd` is noise under the default thread-per-core runner). + the mutation it should not survive — the line to delete or invert — and say + whether it actually would. This is where decorative tests are caught, and + presence of a test is not evidence: in #320 two new tests with confident + comments both survived the mutation they claimed to pin. Ask specifically + whether the assertion would still hold with the code under test removed, and + whether it observes state it can actually reach (allocations made under + `runtime.default_context()` are invisible to a caller's tracking allocator; + the runner does not fail on leaks — `ODIN_TEST_FAIL_ON_BAD_MEMORY` is false; + `/proc/self/fd` is noise under the default thread-per-core runner). 11. **Contract comments** (`CLAUDE.md` §4/§5) on new user-visible surfaces: is the `Node:` line backed by a real probe, and does `Deviates:` name the test that pins it? A contract asserting behavior no test holds is a finding. diff --git a/agents/specialists/quality-docs-auditor.md b/agents/specialists/quality-docs-auditor.md index 030533bd..b3008772 100644 --- a/agents/specialists/quality-docs-auditor.md +++ b/agents/specialists/quality-docs-auditor.md @@ -43,7 +43,7 @@ Ask of each structural change: _is there a version of this with zero new concept ## Lens 3 — altitude and size - **Layer**: engine/string/view/host ABI → `pkg/jsc`; Node Buffer codecs → - `pkg/runtime/buffer*.odin`; HTTP → `http.odin` + `js/internal/http.js`; loop + `pkg/runtime/buffer*.odin`; HTTP → `http.odin` + `js/node/http.js`; loop mechanics → `pkg/runtime/eventloop`; VM pooling and page/VM lifecycle → `vm_pool.odin` and the page layer, not sprinkled through `eval`. Logic in the wrong package is a P1 even if it works — `environment.odin` accumulating a diff --git a/agents/specialists/regression-hunter.md b/agents/specialists/regression-hunter.md index ba3a44c5..5f026082 100644 --- a/agents/specialists/regression-hunter.md +++ b/agents/specialists/regression-hunter.md @@ -43,7 +43,7 @@ to ship, because the diff shows what arrived and hides what left. refactor — a dual path that outlives its migration becomes permanent. - **JS ↔ Odin contract drift**: shared constants, array index layouts, property names, encoding assumptions must move together - (`pkg/runtime/http.odin` ↔ `js/internal/http.js` parse layouts are the classic + (`pkg/runtime/http.odin` ↔ `js/node/http.js` parse layouts are the classic case; `ORDER_*` in `dns.odin` ↔ `dns.js` is another). - A registration/inject path that misses a platform branch or a backend (proactor vs readiness; io_uring vs epoll). diff --git a/bench/README.md b/bench/README.md index 81666ea5..e5ebeacb 100644 --- a/bench/README.md +++ b/bench/README.md @@ -38,18 +38,18 @@ so the harness itself can be smoke-tested anywhere Node is present. ## The gate, and calibrating it -`thresholds.json` ships with **generous starter caps**: they catch only catastrophic -regressions and are not meant to flake. CI currently runs `make bench` (report-only) so -the ratio table shows in the job log without ever failing the build. +CI runs `make bench-gate` on Linux (`ubuntu-latest`) and fails the build when any +benchmark's median-of-3 `lava/node` ratio exceeds its cap in `thresholds.json`. +`make bench` runs a single launch per side as a local report-only view (never fails). -To turn the gate on: +When adding or tuning benchmarks: -1. Run `make bench` (locally or read a CI run's table) to observe Lava's real ratios. -2. Tighten each cap in `thresholds.json` toward its observed ratio, leaving headroom for - CI noise (a cap of ~1.5–2× the observed ratio is a reasonable start). -3. Flip the CI step from `make bench` to `make bench-gate` (`.github/workflows/ci.yml`). +1. Run `make bench` / `make bench-gate` (or read a CI run's table) to observe Lava's real ratios. +2. Adjust caps in `thresholds.json` with headroom for CI noise and document the + measurement method, commit, and hardware/CI context in the comment keys. -A benchmark with no entry in `thresholds.json` is always report-only. +A benchmark with no entry in `thresholds.json` (and not listed in `report_only`) +fails `make bench-gate` so new benchmarks cannot land unmonitored. ## Pages per thread (`make bench-pages`) @@ -57,14 +57,14 @@ A benchmark with no entry in `thresholds.json` is always report-only. page runner (#4) and reports, per N, the median of 5 launches (one process per N per launch, so a "cold" VM really has run nothing): -| metric | meaning | -| --- | --- | -| `cold_ms` | mean `page_runner_open` of the first N pages, each on a fresh VM: Node-profile runtime install plus a one-timer script | -| `warm_ms` | the same for the next N pages on the same VMs (JSC CodeCache warm) | -| `rss_growth_kb_per_live_page` | whole-process VmRSS growth with N pages just opened, divided by N; includes JSC's one-time process init | -| `rss_growth_kb_per_vm` | whole-process VmRSS growth after `TOTAL` pages of churn, divided by VMs created (see below) | -| `pages_per_s` | sustained, N pages in flight, `TOTAL` pages in all | -| `*_failed` | pages of each round that did not finish `.Ok`; any makes the launch exit 1 | +| metric | meaning | +| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------- | +| `cold_ms` | mean `page_runner_open` of the first N pages, each on a fresh VM: Node-profile runtime install plus a one-timer script | +| `warm_ms` | the same for the next N pages on the same VMs (JSC CodeCache warm) | +| `rss_growth_kb_per_live_page` | whole-process VmRSS growth with N pages just opened, divided by N; includes JSC's one-time process init | +| `rss_growth_kb_per_vm` | whole-process VmRSS growth after `TOTAL` pages of churn, divided by VMs created (see below) | +| `pages_per_s` | sustained, N pages in flight, `TOTAL` pages in all | +| `*_failed` | pages of each round that did not finish `.Ok`; any makes the launch exit 1 | `rss_growth_kb_per_vm` is an upper bound, not the steady-state cost of an idle pooled VM: it lumps together JSC's process init, each VM's heap as left uncollected @@ -79,12 +79,12 @@ bench-pages` (`-o:speed`), every page with `cpu_budget` = 30 s of thread CPU (`BENCH_LIMITS`), `TOTAL` = 1000 pages per launch, medians of 5 launches per N, interleaved (each round runs N = 1, 10, 50, 100 once): -| N | cold_ms | warm_ms | rss_growth_kb_per_live_page | rss_growth_kb_per_vm | pages_per_s | -| --- | --- | --- | --- | --- | --- | -| 1 | 28.95 | 9.88 | 23808 | 59216 | 111 | -| 10 | 23.49 | 10.6 | 6233 | 26254 | 109 | -| 50 | 23.93 | 10.17 | 4659 | 19701 | 113 | -| 100 | 24.44 | 10.94 | 4463 | 18688 | 107 | +| N | cold_ms | warm_ms | rss_growth_kb_per_live_page | rss_growth_kb_per_vm | pages_per_s | +| --- | ------- | ------- | --------------------------- | -------------------- | ----------- | +| 1 | 28.95 | 9.88 | 23808 | 59216 | 111 | +| 10 | 23.49 | 10.6 | 6233 | 26254 | 109 | +| 50 | 23.93 | 10.17 | 4659 | 19701 | 113 | +| 100 | 24.44 | 10.94 | 4463 | 18688 | 107 | Read: a page's start-up is dominated by the runtime install, cold or warm, so throughput is CPU-bound and flat in N on one thread. diff --git a/bench/micro/url.js b/bench/micro/url.js index 48da4fba..0f57bc43 100644 --- a/bench/micro/url.js +++ b/bench/micro/url.js @@ -30,11 +30,15 @@ // before being benched — a ratio between two runtimes doing different work means // nothing. // -// THE CAPS WERE PROVEN TO FIRE, by hand, and are NOT wired into the mutation gate. -// Slowing basicURLParse's shared entry with ~45 redundant stripChars passes moved -// special 11.34x -> 17.42x, ipv4 -> 15.35x and octal -> 11.68x, all three breaching -// and each named individually, while urlsearchparams-* stayed put — so the caps -// fire and they name the right bench. (The first attempt discarded the redundant +// THE CAPS WERE ONCE PROVEN TO FIRE, by hand, and are NOT wired into the mutation +// gate. Slowing basicURLParse's shared entry with ~45 redundant stripChars passes +// moved special 11.34x -> 17.42x, ipv4 -> 15.35x and octal -> 11.68x, all three +// breaching the caps of that time and each named individually, while +// urlsearchparams-* stayed put. That proof is STALE: the caps have since been +// widened for CI runner noise (bench/thresholds.json "//url": special 28.0, +// ipv4 20.0, octal 13.5), and that same slowdown would breach none of them. No +// current url-* cap has been shown to fire; re-prove one with a slowdown large +// enough to clear it before trusting it as a regression detector. (The first attempt discarded the redundant // result and JIT dead-code elimination removed it: MORE work measured FASTER, 40 // passes 16.70x vs 120 passes 13.90x. A mutation that can be optimised away proves // nothing.) diff --git a/bench/thresholds.json b/bench/thresholds.json index 2f189668..6a168ebe 100644 --- a/bench/thresholds.json +++ b/bench/thresholds.json @@ -8,7 +8,7 @@ "json-stringify": 2.0, "buffer-to-hex": 8.0, "buffer-to-base64": 9.0, - "buffer-from-utf8": 5.0, + "buffer-from-utf8": 9.5, "buffer-to-utf8": 6.0, "buffer-to-hex-tiny": 4.0, "buffer-from-hex-tiny": 13.0, @@ -22,22 +22,23 @@ "fs-read-utf8-1k": 4.0, "fs-read-utf8-64k": 8.0, "fs-read-bin-1k": 3.0, - "//encoding": "Added 2026-07-27 with bench/micro/encoding.js. The absolute ratios are high because the non-fastpath decoders do their per-byte work in JS against Node's fully native ICU/C++ codecs (since 2026-07-28 the accumulator's string-building does end in one native utf16le crossing, but the per-byte loop is still JS); the caps catch a REGRESSION, they do not claim parity. Caps are ~1.4x the typical ratio on a quiet box. decode-win1252 has NO cap on purpose (report-only): its measured run-to-run spread reaches 2.26x under load, larger than the ~1.43x effect a cap would exist to catch, so no value separates signal from noise - a light that fires on one clean run in four and misses the regression 40% of the time trains reviewers to re-run. decode-utf8-stream keeps 17.0: widening it to 19.0 was measured to cost detection (100% -> 66-91%) without removing the false positives it was meant to fix (22% still trip 19.0 under heavy load), i.e. a middle position worse than either end. On a breach, re-run and compare MEDIANS of >=3 launches before believing it: median-of-3 cuts the loaded-box false-positive rate from ~24% to ~16% with detection unchanged at 100%. Since 2026-07-28 the gate does this itself - `--gate` takes the median of 3 launches per side (bench/run.mjs, collectMedian); report-only runs still take one, since they fail nothing. Recalibrated 2026-07-28 after the Uint16Array-accumulator/native-utf16le rewrite: fatal/utf16le/chunked were capped against the old pure-JS chunked path (26/35/23) and had become loose detectors; new caps are ~1.4x the fresh idle-box median-of-3 (11.5/9.8/6.7 -> 16.5/14.5/9.5). decode-win1252-tiny is report-only like decode-win1252: it exists to make a tiny-input regression visible (measured 0.98x - the native crossing wins even at 8 bytes), and its fixed-cost path is too noisy to gate.", + "//encoding": "Added 2026-07-27 with bench/micro/encoding.js. The absolute ratios are high because the non-fastpath decoders do their per-byte work in JS against Node's fully native ICU/C++ codecs (since 2026-07-28 the accumulator's string-building does end in one native utf16le crossing, but the per-byte loop is still JS); the caps catch a REGRESSION, they do not claim parity. Caps are ~1.4x the typical ratio on a quiet box. decode-win1252 has NO cap on purpose (report-only): its measured run-to-run spread reaches 2.26x under load, larger than the ~1.43x effect a cap would exist to catch, so no value separates signal from noise - a light that fires on one clean run in four and misses the regression 40% of the time trains reviewers to re-run. decode-utf8-stream keeps 17.0: widening it to 19.0 was measured to cost detection (100% -> 66-91%) without removing the false positives it was meant to fix (22% still trip 19.0 under heavy load), i.e. a middle position worse than either end. On a breach, re-run and compare MEDIANS of >=3 launches before believing it: median-of-3 cuts the loaded-box false-positive rate from ~24% to ~16% with detection unchanged at 100%. Since 2026-07-28 the gate does this itself - `--gate` takes the median of 3 launches per side (bench/run.mjs, collectMedian); report-only runs still take one, since they fail nothing. Recalibrated 2026-07-28 after the Uint16Array-accumulator/native-utf16le rewrite (fatal/utf16le/chunked -> 16.5/14.5/9.5). Recalibrated 2026-11-15 (#362): decode-utf16le widened 14.5 -> 17.5 because across 8 `make bench-gate` runs (median of 3 launches per side) recorded in #362 it measured 11.07x, 14.01x, 14.44x, 14.63x, 15.22x, 15.61x plus 2 passes (~14.3x median, breaching 14.5x in 3 of 8 runs; 4-core ubuntu-latest CI 1-launch runs 43719701306/43725488015/43742288199/43743582297 measured 4.61x, 9.29x, 9.02x, 8.94x under Node v24.22.2 vs bin/lava -o:speed). decode-win1252-tiny is report-only like decode-win1252: it exists to make a tiny-input regression visible (measured 0.98x - the native crossing wins even at 8 bytes), and its fixed-cost path is too noisy to gate.", "decode-utf8-fastpath": 6.5, "decode-utf8-fatal": 16.5, - "decode-utf16le": 14.5, + "decode-utf16le": 17.5, "decode-chunked": 9.5, "decode-utf8-stream": 17.0, - "encode-utf8": 3.5, - "encode-into": 3.5, - "new-textdecoder": 14.0, - "//url": "Added 2026-07-30 with bench/micro/url.js, calibrated that day. Nothing measured URL before, which is why a +6% to +20% `new URL` regression shipped and was caught by hand in review rather than by a gate. Caps are ~1.4x the median ratio, matching the encoding convention: measured unpinned (the mode run-bench.sh uses), median-of-3 per round, 5 rounds -> special 11.34x (spread +-4.9%), ipv4 10.51x (+-5.2%), octal 8.17x (+-3.2%), relative 5.64x (+-1.5%), sp-parse 39.70x (+-1.0%), sp-get 0.63x (+-4.8%). READ THE CAPS AS WHAT THEY ARE: 1.4x headroom means a ~40% regression trips them, NOT the 10% one this file was added to catch. A 10% self-regression is resolvable — the same numbers show +-1 to +-5% spread — but not by a ratio-vs-node cap, which is a different mechanism (A/B against the merge-base build); tracked separately rather than faked here. url-relative is capped despite an earlier note calling it too noisy: that +-18% was raw per-launch lava ms on a PINNED box, and the gated statistic (median-of-3 ratio, unpinned) is +-1.5%, the tightest of the six. Do not re-pin to reduce variance; run.mjs already records that pinning shifts the level 25-40% and would invalidate every cap, and this set reproduces it (special 11.34x unpinned vs ~8.5x pinned).", - "url-parse-special": 16.0, - "url-parse-ipv4": 14.7, - "url-parse-octal": 11.4, - "url-relative": 7.9, - "//urlsearchparams": "sp-parse at ~40x is the worst ratio in this directory (next worst: buffer-from-latin1 and decode-utf8-stream at 17.0), and it is entirely CONSTRUCTION: measured separately, node 3.4ms vs lava 149ms for the parse, against node 0.5ms vs lava 0.6ms for a lookup on a prebuilt object. The two were conflated at first and the combined number read as a slow `get`, which it is not. urlsearchparams-get is capped BELOW 1.0 on purpose — lava wins there unpinned — and 0.9 still leaves the documented 1.4x headroom over 0.63x. It is the one cap that flips with pinning (1.52x pinned), so a pinned run will trip it; that is a symptom of running off-spec, not a regression.", + "//node26": "Recalibrated 2026-11-15 (#412) when the oracle moved from Node 24.22.2 to 26.13.0. The denominator moved, not Lava: on the same ubuntu-latest 4-core CI, Node's wall time fell 108.9 -> 35.0 ms (encode-utf8), 64.0 -> 24.2 (encode-into), 11.4 -> 5.3 (buffer-from-utf8), 7.1 -> 6.1 (new-textdecoder) - master run 43757454473 vs #412 run 43759974818 attempt 1, while unrelated Lava rows (url-*, require-*) drifted ~1.2x on runner noise. Measured across 3 `make bench-gate` attempts of run 43759974818 (median of 3 launches per side, Node v26.13.0 vs bin/lava -o:speed): buffer-from-utf8 6.77x, 6.87x, 4.87x; encode-utf8 6.68x, 6.30x, 4.65x; encode-into 8.30x, 8.30x, 7.17x; new-textdecoder 14.09x, 13.93x, 14.29x. Caps are ~1.4x the max observed (5.0 -> 9.5, 3.5 -> 9.5, 3.5 -> 11.5, 14.0 -> 20.0). Tighten when Lava's UTF-8 encode path catches up.", + "encode-utf8": 9.5, + "encode-into": 11.5, + "new-textdecoder": 20.0, + "//url": "Added 2026-07-30 with bench/micro/url.js. Recalibrated 2026-11-15 (#362, #403) before enabling `make bench-gate` in CI, after url.js primordials hardening (118c8b5, 5466d59, 2f43897). Measured across 4 ubuntu-latest 4-core CI single-launch runs (runs 43719701306, 43725488015, 43742288199, 43743582297; Node v24.22.2 vs bin/lava -o:speed): url-parse-special 15.24x, 16.84x, 17.20x, 18.45x; url-parse-octal 7.00x, 8.45x, 8.62x, 8.67x; url-relative 6.64x, 7.12x, 7.19x, 7.28x; plus 2 shared-container runs in #403 (median-of-3 `make bench-gate` at cf850ca and 1-launch `make bench` at 49bc489, Node 24.22.1 vs bin/lava -o:speed): special 23.53x/26.69x, relative 9.99x/11.16x, octal 11.47x. Caps give headroom above both environments (special 16.0 -> 28.0, octal 11.4 -> 13.5, relative 7.9 -> 12.0). Do not re-pin to reduce variance; run.mjs records that pinning shifts the level 25-40%. Recalibrated 2026-11-17: url-parse-ipv4 widened 14.7 -> 20.0. Measured across all 28 ubuntu-latest 4-core CI runs with a bench-gate row between runs 43938992938 and 44067776896 (median of 3 launches per side, Node v26.13.0 vs bin/lava -o:speed): 10.37x-16.06x, breaching 14.7x in 3 (15.22x at aea9da8, 15.01x and 16.06x at f9eb1b8/ac9e124). The parser this bench runs did move between those breaches (5228c42 moved it into internal/url), but the breaches track the runner, not Lava: the 3 breaching runs are the 3 with the LOWEST Lava times of all 28 (77-83 ms of a 77-134 ms range), and they are breaches only because Node ran faster still. Node's time clusters at ~11.7-12.1 ms (ratio 10.4-11.3x) or ~4.9-7.5 ms (17 runs under 10.5 ms: mean 13.59x, sd 1.06), with 2 runs between (8.98 ms 13.33x, 10.05 ms 12.79x). 20.0 is ~1.25x the max observed and >5 sd above the fast-cluster mean; it would have had 0 false positives across the 28, while #412's ~1.4x-of-max (22.5) was set from only 3 samples and would give up detection here. Tripping 20.0 without a Lava change needs Node at ~4 ms against Lava's fastest ~77 ms, ~19% faster than the fastest Node of the 28 (4.89 ms): not observed, so medium confidence. The cost: the gate trips on a ~1.8x regression on slow-Node runners (~11x) and ~1.25-1.7x on fast ones (~12-16x). bench/micro/url.js runs 50k iterations, so Node finishes in ~5 ms and launch noise dominates; more iterations may allow a tighter cap (untested).", + "url-parse-special": 28.0, + "url-parse-ipv4": 20.0, + "url-parse-octal": 13.5, + "url-relative": 12.0, + "//urlsearchparams": "sp-parse at ~26-40x is construction cost; lookup is fast (0.62x-0.90x in the 4 ubuntu-latest 4-core CI runs listed above, 0.98x median-of-3 in #403, 1.00x in #336 after url.js primordials hardening). Widened urlsearchparams-get 0.9 -> 1.5 (#336, #362) so CI and unpinned dev runs do not flake at 0.9x-1.0x while still catching multi-fold lookup regressions.", "urlsearchparams-parse": 55.6, - "urlsearchparams-get": 0.9 + "urlsearchparams-get": 1.5 } } diff --git a/bun.lock b/bun.lock index 0ed2f49f..7de5952e 100644 --- a/bun.lock +++ b/bun.lock @@ -35,7 +35,7 @@ "actionlint": ["actionlint@2.0.6", "", {}, "sha512-tNx8f48yJNSLXTIygGntu5dSZlIZblDt8sR7pIs7EEmmb2PkEF87d+3UKZV2GUgCuN9Awj7k4ZPrwdAxFWEqgw=="], - "ansi-regex": ["ansi-regex@6.6.0", "", {}, "sha512-XZ5apbzreCbXSKwPjJstu8R0zOi247w3XYwWxtwBXgpZrCyc1Ka7p9/xY0A9/diMewmHbEKz+FIraWaEWjlN5g=="], + "ansi-regex": ["ansi-regex@6.2.2", "", {}, "sha512-Bq3SmSpyFHaWjPk8If9yc6svM8c56dB5BAtW4Qbw5jHTwwXXcTLoRMkpDJp6VL0XzlWaCHTXrkFURMYmD0sLqg=="], "argparse": ["argparse@3.0.4", "", {}, "sha512-fwWK3Zq22l4XJJjx5mcBHGikIk+hJqO+32dPm0Jur7+rtG5x1scOjSfdT9ZKeWHPpsya/l0HHK4tDImBkTrtvA=="], @@ -59,21 +59,21 @@ "entities": ["entities@8.1.0", "", {}, "sha512-kxL7msIffSuh9aaFAMD7rxAIuTRMAHMeBtgHW2yUdWw732ZNh4MehkF2gdjvtdmikkaIP9bFDDJOPlsvm7avrA=="], - "fast-check": ["fast-check@4.10.2", "", { "dependencies": { "pure-rand": "^8.0.0" } }, "sha512-iK2f+YrcmoeGqk6fA0ea2bptcu/itMIm4NfEozq6N25+aG6h7s5HZbB/k1aV7b5w5sFLMCbbtRUsTVR+BgC3xw=="], + "fast-check": ["fast-check@4.10.3", "", { "dependencies": { "pure-rand": "^8.0.0" } }, "sha512-SBaU6nhh+sXxb05r2IHAKfxnR/1Pu6MqtHYaewM1BY6V5xzOxOk+4yoWO7WfeeyZG1iArFE6/DO/7iLYMUuOJg=="], "fast-glob": ["fast-glob@3.3.3", "", { "dependencies": { "@nodelib/fs.stat": "^2.0.2", "@nodelib/fs.walk": "^1.2.3", "glob-parent": "^5.1.2", "merge2": "^1.3.0", "micromatch": "^4.0.8" } }, "sha512-7MptL8U0cqcFdzIzwOTHoilX9x5BrNqye7Z/LuC7kCMRio1EMSyqRK3BEAUD7sXRq4iT4AzTVuZdhgQ2TCvYLg=="], - "fastq": ["fastq@1.20.3", "", { "dependencies": { "reusify": "^1.0.4" } }, "sha512-XKv5nnLs6nLF71NgiKJLIZFLkPyIEuOselLG7ujZnGrRfQK8HpvY+WqKhAJUAdLomwVHErVS4LfxFlPq0/FTAw=="], + "fastq": ["fastq@1.20.1", "", { "dependencies": { "reusify": "^1.0.4" } }, "sha512-GGToxJ/w1x32s/D2EKND7kTil4n8OVk/9mycTc4VDza13lOvpUZTGX3mFSCtV9ksdGBVzvsyAVLM6mHFThxXxw=="], "fill-range": ["fill-range@7.1.1", "", { "dependencies": { "to-regex-range": "^5.0.1" } }, "sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg=="], - "get-east-asian-width": ["get-east-asian-width@1.7.0", "", {}, "sha512-XjH1AECxf0giL2V1aU8vKyRR2ppRUb5c0EvT7zuJTokQ74bNo52zOtghqdWIqrhUD79fo3x0WfKZdOqxF6LG1Q=="], + "get-east-asian-width": ["get-east-asian-width@1.6.0", "", {}, "sha512-QRbvDIbx6YklUe6RxeTeleMR0yv3cYH6PsPZHcnVn7xv7zO1BHN8r0XETu8n6Ye3Q+ahtSarc3WgtNWmehIBfA=="], "glob-parent": ["glob-parent@5.1.2", "", { "dependencies": { "is-glob": "^4.0.1" } }, "sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow=="], "globby": ["globby@16.2.4", "", { "dependencies": { "@sindresorhus/merge-streams": "^4.0.0", "fast-glob": "^3.3.3", "ignore": "^7.0.5", "is-path-inside": "^4.0.0", "micromatch": "^4.0.8", "slash": "^5.1.0", "unicorn-magic": "^0.4.0" } }, "sha512-c8B/VNLmxRcmqqenRA9t+9IyOjf9+V6lTxPaUJLqOCONdQkWZ0ETYgX0qbtJqPsgCNusT9MZ5Jeidw8Eb9tn2g=="], - "ignore": ["ignore@7.0.14", "", {}, "sha512-wCBiZ77u1tuWde6G/nfOtLaCTYcFDU5yCzARzK10y+rLhzYoL7a7yEemcopcc2Mzow7gZOIjnV2ym1NYmdiJOQ=="], + "ignore": ["ignore@7.0.6", "", {}, "sha512-BAg6QkE8W+TuQLrrw0Ugr7HegXduRuuj8/ti2kSOc+jz1dmx8/WNcjr6XGnq5YpDWxFwwaavqD0+jIUOKelTsw=="], "is-alphabetical": ["is-alphabetical@2.0.1", "", {}, "sha512-FWyyY60MeTNyeSRpkM2Iry0G9hpr7/9kD40mD/cGQEuilcZYS4okz8SN2Q6rLCJ8gbCt6fN+rC+6tMGS99LaxQ=="], @@ -131,7 +131,7 @@ "micromark-factory-label": ["micromark-factory-label@2.0.1", "", { "dependencies": { "devlop": "^1.0.0", "micromark-util-character": "^2.0.0", "micromark-util-symbol": "^2.0.0", "micromark-util-types": "^2.0.0" } }, "sha512-VFMekyQExqIW7xIChcXn4ok29YE3rnuyveW3wZQWWqF4Nv9Wk5rgJ99KzPvHjkmPXF93FXIbBp6YdW3t71/7Vg=="], - "micromark-factory-space": ["micromark-factory-space@2.1.0", "", { "dependencies": { "micromark-util-character": "^2.0.0", "micromark-util-types": "^2.0.0" } }, "sha512-fS8hnLIjnjvdQIj39Geug8wWsR0HrYZ43KShKxNfwT7t2LOHo/LbWZEzEaSOjwtjdCsjoE6syHhonEzW0zv0+Q=="], + "micromark-factory-space": ["micromark-factory-space@2.0.1", "", { "dependencies": { "micromark-util-character": "^2.0.0", "micromark-util-types": "^2.0.0" } }, "sha512-zRkxjtBxxLd2Sc0d+fbnEunsTj46SWXgXciZmHq0kDYGnck/ZSGj9/wULTV95uoeYiK5hRXP2mJ98Uo4cq/LQg=="], "micromark-factory-title": ["micromark-factory-title@2.0.1", "", { "dependencies": { "micromark-factory-space": "^2.0.0", "micromark-util-character": "^2.0.0", "micromark-util-symbol": "^2.0.0", "micromark-util-types": "^2.0.0" } }, "sha512-5bZ+3CjhAd9eChYTHsjy6TGxpOFSKgKKJPJxr293jTbfry2KDoWkhBb6TcPVB4NmzaPhMs1Frm9AZH7OD4Cjzw=="], @@ -197,6 +197,48 @@ "uc.micro": ["uc.micro@3.0.0", "", {}, "sha512-U3PppEkleoTnIfi8BozMx3yju3qc/L6SwqWo2Sw+54PX+PX0q9I+r1Um5HCmqD7n9VDX5/v3vQH/AjA6deDdtw=="], - "unicorn-magic": ["unicorn-magic@0.4.1", "", {}, "sha512-lzlXPoVB0Uy/FvTaUgX39RbixYiSKoc3JSoSf01+0c2B6FR3qdYIPN7Qjo/AV7n6sqLCk0509cT3LRj1oSZ1+A=="], + "unicorn-magic": ["unicorn-magic@0.4.0", "", {}, "sha512-wH590V9VNgYH9g3lH9wWjTrUoKsjLF6sGLjhR4sH1LWpLmCOH0Zf7PukhDA8BiS7KHe4oPNkcTHqYkj7SOGUOw=="], + + "micromark/micromark-core-commonmark": ["micromark-core-commonmark@2.0.3", "", { "dependencies": { "decode-named-character-reference": "^1.0.0", "devlop": "^1.0.0", "micromark-factory-destination": "^2.0.0", "micromark-factory-label": "^2.0.0", "micromark-factory-space": "^2.0.0", "micromark-factory-title": "^2.0.0", "micromark-factory-whitespace": "^2.0.0", "micromark-util-character": "^2.0.0", "micromark-util-chunked": "^2.0.0", "micromark-util-classify-character": "^2.0.0", "micromark-util-html-tag-name": "^2.0.0", "micromark-util-normalize-identifier": "^2.0.0", "micromark-util-resolve-all": "^2.0.0", "micromark-util-subtokenize": "^2.0.0", "micromark-util-symbol": "^2.0.0", "micromark-util-types": "^2.0.0" } }, "sha512-RDBrHEMSxVFLg6xvnXmb1Ayr2WzLAWjeSATAoxwKYJV94TeNavgoIdA0a9ytzDSVzBy2YKFK+emCPOEibLeCrg=="], + + "micromark/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-core-commonmark/micromark-factory-space": ["micromark-factory-space@2.1.0", "", { "dependencies": { "micromark-util-character": "^2.0.0", "micromark-util-types": "^2.0.0" } }, "sha512-fS8hnLIjnjvdQIj39Geug8wWsR0HrYZ43KShKxNfwT7t2LOHo/LbWZEzEaSOjwtjdCsjoE6syHhonEzW0zv0+Q=="], + + "micromark-extension-directive/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-extension-gfm-autolink-literal/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-extension-gfm-footnote/micromark-core-commonmark": ["micromark-core-commonmark@2.0.3", "", { "dependencies": { "decode-named-character-reference": "^1.0.0", "devlop": "^1.0.0", "micromark-factory-destination": "^2.0.0", "micromark-factory-label": "^2.0.0", "micromark-factory-space": "^2.0.0", "micromark-factory-title": "^2.0.0", "micromark-factory-whitespace": "^2.0.0", "micromark-util-character": "^2.0.0", "micromark-util-chunked": "^2.0.0", "micromark-util-classify-character": "^2.0.0", "micromark-util-html-tag-name": "^2.0.0", "micromark-util-normalize-identifier": "^2.0.0", "micromark-util-resolve-all": "^2.0.0", "micromark-util-subtokenize": "^2.0.0", "micromark-util-symbol": "^2.0.0", "micromark-util-types": "^2.0.0" } }, "sha512-RDBrHEMSxVFLg6xvnXmb1Ayr2WzLAWjeSATAoxwKYJV94TeNavgoIdA0a9ytzDSVzBy2YKFK+emCPOEibLeCrg=="], + + "micromark-extension-gfm-footnote/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-extension-gfm-table/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-extension-math/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-factory-destination/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-factory-label/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-factory-space/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-factory-title/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-factory-whitespace/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-util-character/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-util-classify-character/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-util-combine-extensions/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-util-edit-map/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-util-resolve-all/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-util-subtokenize/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], + + "micromark-core-commonmark/micromark-factory-space/micromark-util-types": ["micromark-util-types@2.0.2", "", {}, "sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA=="], } } diff --git a/bunfig.toml b/bunfig.toml index 148ae9c8..2237beee 100644 --- a/bunfig.toml +++ b/bunfig.toml @@ -1,10 +1,19 @@ -# `bun fmt` (make check-js / fix-js). Replaces vite-plus's `vp fmt` (oxfmt); the -# style is the one the tree was already formatted in: oxfmt's 100-column width, -# semicolons, single quotes. There is no separate linter: `bun fmt --check` is the -# style gate, and the correctness checks are the repo's own (primordials ratchet, -# orphan-JS, global-replace) plus the oracle suites. +# `bun fmt` (make check-js / fix-js; markdown via make check-md / fix-md). Replaces +# vite-plus's `vp fmt` (oxfmt); the style is the one the tree was already formatted in: +# oxfmt's 100-column width, semicolons, single quotes. There is no separate JS linter: +# `bun fmt --check` is the style gate, and the correctness checks are the repo's own +# (primordials ratchet, JS module graph, global-replace) plus the oracle suites. [fmt] printWidth = 100 semi = true singleQuote = true -ignorePatterns = ["bin/", "build/", "node_modules/", "reports/", "tests/vendor/"] +ignorePatterns = [ + "bin/", + "build/", + "node_modules/", + "reports/", + "tests/vendor/", + ".deps/", + # Vendored upstream Node docs: not ours to reformat (markdownlint ignores them too). + "reference/node-doc-api/", +] diff --git a/cmd/lava/atob_invalid_character_test.odin b/cmd/lava/atob_invalid_character_test.odin new file mode 100644 index 00000000..5e364091 --- /dev/null +++ b/cmd/lava/atob_invalid_character_test.odin @@ -0,0 +1,54 @@ +#+build linux, darwin +package main + +import "core:testing" +import lava "lava:pkg/runtime" +import eventloop "lava:pkg/runtime/eventloop" + +// Pins the shape of the error atob/btoa throw for a bad character, which is a +// deliberate deviation the node-compat oracle cannot express: Node throws a +// DOMException, Lava has no DOMException (the same gap structured_clone.js's +// DataCloneError documents), so it throws a plain Error carrying the +// DOMException's name, legacy numeric code and Node's exact message. +// tests/node-compat/cases/03-buffer.js compares name/code/message against node; +// this test pins the part node cannot oracle — that the value IS an Error and +// that there is no DOMException for it to be an instance of. When Lava grows a +// DOMException this test is what has to change, deliberately. +ATOB_INVALID_CHARACTER_SCRIPT :: ` +'use strict'; +function check(label, fn, message) { + let e; + try { + fn(); + } catch (err) { + e = err; + } + const got = e === undefined ? 'no throw' : + 'name=' + e.name + ' code=' + typeof e.code + ':' + e.code + ' message=' + e.message + + ' isError=' + (e instanceof Error); + const want = 'name=InvalidCharacterError code=number:5 message=' + message + ' isError=true'; + if (got !== want) throw new Error(label + ': got ' + got + ', want ' + want); +} +check('atob', () => atob('!'), 'Invalid character'); +check('atob-length', () => atob('Z'), 'The string to be decoded is not correctly encoded.'); +check('btoa', () => btoa('Ā'), 'Invalid character'); +if (typeof DOMException !== 'undefined') throw new Error('DOMException now exists: revisit atob/btoa'); +console.log('atob-invalid-character ok'); +` + +@(test) +atob_invalid_character_shape :: proc(t: ^testing.T) { + loop := eventloop.init() + // eval consumes (destroys) the loop on every path; do not destroy it here. + result := lava.eval(ATOB_INVALID_CHARACTER_SCRIPT, "", &loop, false) + defer lava.result_destroy(&result) + + testing.expectf( + t, + result.status == .Ok, + "eval did not complete cleanly: status=%v message=%q", + result.status, + result.message, + ) + testing.expectf(t, result.exit_code == 0, "eval exit code=%d (want 0)", result.exit_code) +} diff --git a/cmd/lava/blob_class_test.odin b/cmd/lava/blob_class_test.odin new file mode 100644 index 00000000..67b71f98 --- /dev/null +++ b/cmd/lava/blob_class_test.odin @@ -0,0 +1,110 @@ +#+build linux, darwin +package main + +import "core:testing" +import lava "lava:pkg/runtime" +import eventloop "lava:pkg/runtime/eventloop" + +// Pins what the Blob/File class conversion deliberately leaves different from node, +// which the oracle case (tests/node-compat/cases/77-blob-file-module.js) cannot express +// because the two runtimes answer differently by construction: +// +// - Calling either without `new` throws the ENGINE's class-constructor TypeError. +// JSC: `Cannot call a class constructor Blob without |new|`; node (V8): +// `Class constructor Blob cannot be invoked without 'new'`. A class constructor +// cannot customise it, so the text is an engine trait; the oracle case compares +// only the error's class. If this message changes, JSC changed — or Blob stopped +// being a class, which is the regression this guards. +// - Blob.prototype's methods stay ENUMERABLE, as they were while Blob was a function +// with an assigned prototype. Class methods are not, so internal/blob.js re-marks +// them; node's enumerable set differs (size/type/textStream in, toString out). +BLOB_CLASS_SCRIPT :: ` +'use strict'; +const { Blob, File } = require('node:buffer'); +function noNew(C) { + try { + C([], 'f'); + return 'no throw'; + } catch (e) { + return e.name + ': ' + e.message; + } +} +const want = { + blob: 'TypeError: Cannot call a class constructor Blob without |new|', + file: 'TypeError: Cannot call a class constructor File without |new|', + keys: 'arrayBuffer,bytes,stream,text,slice,toString', +}; +const got = { + blob: noNew(Blob), + file: noNew(File), + keys: Object.keys(Blob.prototype).join(','), +}; +for (const k of Object.keys(want)) { + if (got[k] !== want[k]) throw new Error(k + ': got ' + got[k] + ', want ' + want[k]); +} +console.log('blob-class ok'); +` + +@(test) +blob_class_engine_traits :: proc(t: ^testing.T) { + loop := eventloop.init() + // eval consumes (destroys) the loop on every path; do not destroy it here. + result := lava.eval(BLOB_CLASS_SCRIPT, "", &loop, false) + defer lava.result_destroy(&result) + + testing.expectf( + t, + result.status == .Ok, + "eval did not complete cleanly: status=%v message=%q", + result.status, + result.message, + ) + testing.expectf(t, result.exit_code == 0, "eval exit code=%d (want 0)", result.exit_code) +} + +// A view part's window must come from the engine's slots, not the configurable +// %TypedArray% / DataView accessors. Read live, a forged byteOffset/byteLength selects +// any window of the backing store — across a shared allocUnsafe pool, a neighbouring +// Buffer's bytes (the gadget internal/encoding.js's toBytes already closed). Lava-only: +// node's own Blob reads these accessors off every view part (node 26.13.0, verified with +// spy getters), so node lands on the forged window and an oracle would diverge by +// construction. Deviates from node in Lava's favour; documented at blobPartToChunks. +BLOB_VIEW_WINDOW_SCRIPT :: ` +'use strict'; +const TA = Object.getPrototypeOf(Uint8Array.prototype); +const saved = []; +function forge(owner, key, value) { + saved.push([owner, key, Object.getOwnPropertyDescriptor(owner, key)]); + Object.defineProperty(owner, key, { configurable: true, get() { return value; } }); +} +const backing = new Uint8Array([0x53, 0x45, 0x43, 0x52, 0x45, 0x54, 0x61, 0x62]); // SECRETab +const view = backing.subarray(6, 8); // "ab" +const dv = new DataView(backing.buffer, 6, 2); // "ab" +forge(TA, 'byteOffset', 0); +forge(TA, 'byteLength', 6); +forge(DataView.prototype, 'byteOffset', 0); +forge(DataView.prototype, 'byteLength', 6); +const b = new Blob([view, dv]); +for (const [o, k, d] of saved) Object.defineProperty(o, k, d); +b.text().then(function (s) { + if (s !== 'abab') throw new Error('view window came from forged accessors: ' + JSON.stringify(s)); + console.log('blob-view-window ok'); +}); +` + +@(test) +blob_view_window_ignores_forged_accessors :: proc(t: ^testing.T) { + loop := eventloop.init() + // eval consumes (destroys) the loop on every path; do not destroy it here. + result := lava.eval(BLOB_VIEW_WINDOW_SCRIPT, "", &loop, false) + defer lava.result_destroy(&result) + + testing.expectf( + t, + result.status == .Ok, + "eval did not complete cleanly: status=%v message=%q", + result.status, + result.message, + ) + testing.expectf(t, result.exit_code == 0, "eval exit code=%d (want 0)", result.exit_code) +} diff --git a/cmd/lava/encoding_pollution_test.odin b/cmd/lava/encoding_pollution_test.odin index 7d3ffb1c..d9c443a0 100644 --- a/cmd/lava/encoding_pollution_test.odin +++ b/cmd/lava/encoding_pollution_test.odin @@ -22,7 +22,6 @@ ENCODING_POLLUTION_SCRIPT :: ` const realToLowerCase = String.prototype.toLowerCase; const realTrim = String.prototype.trim; const realFromCharCode = String.fromCharCode; -const realApply = Function.prototype.apply; const realPush = Array.prototype.push; const realBufferFrom = Buffer.from; const realBufferToString = Buffer.prototype.toString; @@ -45,7 +44,8 @@ function record(name, fn) { String.prototype.toLowerCase = function () { return 'utf-8'; }; String.prototype.trim = function () { return 'utf-8'; }; String.fromCharCode = function () { return 'PWNED'; }; -Function.prototype.apply = function () { return 'PWNED'; }; +// Function.prototype.apply is no longer poisoned here: since #450 it is locked +// before user code, so the assignment throws (pinned by function_call_lock_test). Array.prototype.push = function () { return 0; }; Buffer.from = function () { return new Uint8Array([0x50]); }; Buffer.prototype.toString = function () { return 'PWNED'; }; @@ -85,7 +85,6 @@ record('encode', function () { String.prototype.toLowerCase = realToLowerCase; String.prototype.trim = realTrim; String.fromCharCode = realFromCharCode; -Function.prototype.apply = realApply; Array.prototype.push = realPush; Buffer.from = realBufferFrom; Buffer.prototype.toString = realBufferToString; diff --git a/cmd/lava/errors_test.odin b/cmd/lava/errors_test.odin index c98bd841..4f7cb795 100644 --- a/cmd/lava/errors_test.odin +++ b/cmd/lava/errors_test.odin @@ -74,7 +74,7 @@ node_error_factory :: proc(t: ^testing.T) { // node_error_received_types pins determine_received_type's "Received …" rendering // (via err_invalid_arg_type's message) across every value kind — including BigInt, // the primitive JSValueGetType's enum omits — so it matches the JS-side -// inspectReceived (js/internal/crypto.js). A bare context suffices: only the +// inspectReceived (js/node/crypto.js). A bare context suffices: only the // message text is asserted here, which does not depend on captured intrinsics. @(test) node_error_received_types :: proc(t: ^testing.T) { diff --git a/cmd/lava/function_call_lock_test.odin b/cmd/lava/function_call_lock_test.odin new file mode 100644 index 00000000..6869df4c --- /dev/null +++ b/cmd/lava/function_call_lock_test.odin @@ -0,0 +1,218 @@ +#+build linux, darwin +package main + +import "core:testing" +import lava "lava:pkg/runtime" +import eventloop "lava:pkg/runtime/eventloop" + +// Pins a deliberate §1 deviation where Lava is STRICTER than node: +// Function.prototype.call/apply/bind are locked realm-wide before user code runs +// (primordials.js lockIntrinsics, called from bootstrap/loader.js). node lets a +// script replace them; Lava throws. That is a §1 deviation, so no oracle case can +// express it — node prints "assigned" where Lava prints "THREW" — and it lives here. +// +// The shape is node's own `--frozen-intrinsics` (node 26, verified with +// scratch probe.js): an accessor whose setter throws on Function.prototype itself +// in BOTH modes, and defines an own data property on any other receiver, so +// `f.call = x` on an ordinary function behaves exactly as it does in plain node. +// A non-writable DATA property would be cheaper to write and is wrong in two ways +// the checks below each catch: sloppy `Function.prototype.call = x` would no-op +// silently instead of throwing, and `f.call = x` on an ordinary function would +// throw in strict code (the override mistake) where node succeeds. +// +// Every check is recorded and compared at the end rather than thrown at first +// failure, so a mutation's red output names the check it broke (the manifest's +// expect_detail matches on that name). +FUNCTION_CALL_LOCK_SCRIPT :: ` +const FP = Function.prototype; +const real = { call: FP.call, apply: FP.apply, bind: FP.bind }; +const results = []; +function record(name, fn) { + let out; + try { + out = String(fn()); + } catch (e) { + out = 'THREW:' + e.name; + } + results[results.length] = name + '=' + out; +} + +// Sloppy assignment on Function.prototype: a plain data lock no-ops here silently, +// which is exactly the outcome this lock exists to rule out. +for (const k of ['call', 'apply', 'bind']) { + record('sloppy-' + k, function () { + FP[k] = function () { return 'POISON'; }; + return 'assigned'; + }); + record('strict-' + k, function () { + 'use strict'; + FP[k] = function () { return 'POISON'; }; + return 'assigned'; + }); + record('define-' + k, function () { + Object.defineProperty(FP, k, { value: function () { return 'POISON'; } }); + return 'defined'; + }); + record('intact-' + k, function () { return FP[k] === real[k]; }); +} +record('names', function () { + return [FP.call.name, FP.call.length, FP.apply.name, FP.apply.length, FP.bind.name, FP.bind.length].join(','); +}); + +// The override mistake: assigning to an ORDINARY function's inherited call/apply/ +// bind must create an own data property, as in node, and must not leak to others. +record('own-strict', function () { + 'use strict'; + function f() {} + f.call = function () { return 'own'; }; + const d = Object.getOwnPropertyDescriptor(f, 'call'); + return [f.call(), d.writable, d.enumerable, d.configurable, (function g() {}).call === real.call].join(','); +}); +record('own-sloppy', function () { + function f() {} + f.apply = function () { return 'own'; }; + return [f.apply(), Object.prototype.hasOwnProperty.call(f, 'apply')].join(','); +}); + +// #450's table: every builtin that used to break under a replaced .call, run +// after the replacement attempts above so it proves the lock held. +record('usp', function () { return new URLSearchParams('a=1').get('a'); }); +record('format', function () { return require('util').format('%s', 'x'); }); +record('buffer', function () { return Buffer.from('hi').toString('hex'); }); +record('url', function () { return new URL('http://a/b').pathname; }); +record('decode', function () { return new TextDecoder().decode(new Uint8Array([104, 105])); }); +record('decode16', function () { + return new TextDecoder('utf-16le').decode(new Uint8Array([0x41, 0, 0x42, 0])); +}); +// Vectors H1-H4 and Q2 of tests/node-compat/cases/55-encoding-pollution.js, which +// poisoned Function.prototype.apply (String.fromCharCode.apply once rewrote a +// decoded host to http://a/). node accepts that assignment and Lava now refuses it, +// so they moved here; the expected values are node 26's output for the same calls. +record('h1', function () { return new URL('http://%C3%BCber.example/').href; }); +record('h2', function () { + return new TextDecoder('utf-8', { fatal: true }).decode(new Uint8Array([0xc3, 0xbc, 0x62, 0x65, 0x72])) === 'über'; +}); +record('h3', function () { + return new TextDecoder('utf-16le').decode(new Uint8Array([0x3d, 0xd8, 0x00, 0xde, 0x41, 0x00])) === '😀A'; +}); +record('h4', function () { return Array.from(new TextEncoder().encode('ü€😀')).join(','); }); +record('q2', function () { + const big = new Uint8Array(9000); + for (let i = 0; i < big.length; i++) big[i] = 0x41 + (i % 26); + const s = new TextDecoder('windows-1252').decode(big); + return s.length + ':' + s.charCodeAt(0) + ':' + s.charCodeAt(8192) + ':' + s.charCodeAt(8999); +}); + +// #494: the captured intrinsics themselves. Locking the three slots on +// Function.prototype does not stop a script from giving the HELD function an own +// 'call': 'Function.prototype.apply.call = evil' steered every +// 'apply.call(listener, …)' in events.js (node runs the listener). lockIntrinsics() +// now makes call/apply/bind non-extensible, so neither an own property nor a +// re-pointed prototype can change what a read off them finds. +// The same hole on every OTHER method primordials wraps: 'Array.prototype.push.call +// = evil' steered the callerN 'fn.call(t, …)' wrappers. lockIntrinsics() makes +// exactly the wrapped set non-extensible too (the wrappers keep 'fn.call', which is +// the only spelling JSC inlines); a method nobody wraps (Array.prototype.at) keeps +// node's default behavior. Buffer.prototype.toString is wrapped by the loader AFTER +// the lock (pristineBuffer), so it pins the lock-on-wrap path. +// The deviation rows (ext … setproto) are node --frozen-intrinsics' answers (node +// 26.13, probed); default node answers true / length|name / assigned / assigned / +// assigned / 1 / assigned / set. +const evil = function () { return 'HIJACK'; }; +record('ext', function () { + return [Object.isExtensible(real.call), Object.isExtensible(real.apply), Object.isExtensible(real.bind)].join(','); +}); +record('keys', function () { + return [real.call, real.apply, real.bind].map(function (f) { return Object.getOwnPropertyNames(f).join('|'); }).join(','); +}); +// Through lockMethod's setter, which defines an own property: on a non-extensible +// receiver that throws in sloppy code too, as under --frozen-intrinsics. +record('own-call-sloppy', function () { FP.apply.call = evil; return 'assigned'; }); +record('own-apply-strict', function () { 'use strict'; FP.call.apply = evil; return 'assigned'; }); +record('own-bind-strict', function () { 'use strict'; FP.bind.bind = evil; return 'assigned'; }); +// A key with no inherited setter: plain [[Set]] on a non-extensible object. +record('new-key-sloppy', function () { FP.bind.x = 1; return FP.bind.x; }); +record('new-key-strict', function () { 'use strict'; FP.bind.x = 1; return 'assigned'; }); +// Re-pointing the prototype is the other way to change what 'apply.call' reads. +record('setproto', function () { + Object.setPrototypeOf(real.apply, { __proto__: FP, call: evil }); + return 'set'; +}); +record('wrapped-ext', function () { + return [Object.isExtensible(Array.prototype.push), Object.isExtensible(String.prototype.charCodeAt), + Object.isExtensible(Buffer.prototype.toString), Object.isExtensible(Array.prototype.at)].join(','); +}); +record('push-own-sloppy', function () { Array.prototype.push.call = evil; return 'assigned'; }); +record('push-fresh-sloppy', function () { Array.prototype.push.x = 1; return Array.prototype.push.x; }); +record('at-own', function () { + Array.prototype.at.call = evil; + return Object.getOwnPropertyDescriptor(Array.prototype.at, 'call').value === evil; +}); +// The #494 probe: every attempt is made (and may throw on Lava), then the wrappers +// must still run the real push/charCodeAt. +record('push-hijack', function () { + let hit = 0; + const bump = function () { hit++; return 0; }; + try { Array.prototype.push.call = bump; } catch (e) {} + try { Object.setPrototypeOf(String.prototype.charCodeAt, { __proto__: FP, call: bump }); } catch (e) {} + const u = new URLSearchParams('a=1&b=2'); + u.append('c', '3'); + return [hit, u.toString(), new URL('http://a/b?c').href].join(' '); +}); +record('emit-hijack', function () { + const EE = require('events'); + const e = new EE(); + const got = []; + e.on('x', function (a) { got[got.length] = 'listener:' + a; }); + e.emit('x', 1); + return got.join(','); +}); +const want = [ + 'sloppy-call=THREW:TypeError', 'strict-call=THREW:TypeError', 'define-call=THREW:TypeError', 'intact-call=true', + 'sloppy-apply=THREW:TypeError', 'strict-apply=THREW:TypeError', 'define-apply=THREW:TypeError', 'intact-apply=true', + 'sloppy-bind=THREW:TypeError', 'strict-bind=THREW:TypeError', 'define-bind=THREW:TypeError', 'intact-bind=true', + 'names=call,1,apply,2,bind,1', + 'own-strict=own,true,true,true,true', + 'own-sloppy=own,true', + 'usp=1', 'format=x', 'buffer=6869', 'url=/b', 'decode=hi', 'decode16=AB', + 'h1=http://xn--ber-goa.example/', 'h2=true', 'h3=true', 'h4=195,188,226,130,172,240,159,152,128', + 'q2=9000:65:67:68', + 'ext=false,false,false', 'keys=length|name,length|name,length|name', + 'own-call-sloppy=THREW:TypeError', 'own-apply-strict=THREW:TypeError', 'own-bind-strict=THREW:TypeError', + 'new-key-sloppy=undefined', 'new-key-strict=THREW:TypeError', 'setproto=THREW:TypeError', + 'wrapped-ext=false,false,false,true', 'push-own-sloppy=THREW:TypeError', 'push-fresh-sloppy=undefined', 'at-own=true', + 'push-hijack=0 a=1&b=2&c=3 http://a/b?c', + 'emit-hijack=listener:1', +]; +const bad = []; +for (let i = 0; i < want.length; i++) { + if (results[i] !== want[i]) bad[bad.length] = 'want ' + want[i] + ' got ' + results[i]; +} +if (bad.length !== 0) throw new Error(bad.join('; ')); + +// Moved from tests/runtime/eventloop/cases/11-scheduler-hardening.js, which used to +// poison call/apply before scheduling: nextTick must still forward its arguments +// and queueMicrotask must still run. The exit code carries a miss. +let hits = 0; +process.nextTick(function (a, b) { if (a + b === 'xy') hits += 1; }, 'x', 'y'); +queueMicrotask(function () { hits += 10; }); +setTimeout(function () { if (hits !== 11) process.exitCode = 3; }, 0); +console.log('function-call-lock ok'); +` + +@(test) +function_call_lock :: proc(t: ^testing.T) { + loop := eventloop.init() + // eval consumes (destroys) the loop on every path; do not destroy it here. + result := lava.eval(FUNCTION_CALL_LOCK_SCRIPT, "", &loop, false) + defer lava.result_destroy(&result) + + testing.expectf( + t, + result.status == .Ok, + "eval did not complete cleanly: status=%v message=%q", + result.status, + result.message, + ) + testing.expectf(t, result.exit_code == 0, "eval exit code=%d (want 0)", result.exit_code) +} diff --git a/cmd/lava/lazy_builtins_test.odin b/cmd/lava/lazy_builtins_test.odin new file mode 100644 index 00000000..f038cdef --- /dev/null +++ b/cmd/lava/lazy_builtins_test.odin @@ -0,0 +1,118 @@ +#+build linux, darwin +package main + +import "core:strings" +import "core:testing" +import jsc "lava:pkg/jsc" +import lava "lava:pkg/runtime" + +// Pins compile-on-first-require for the embedded builtin factories +// (install_internal_modules / builtin_compile_cb, globals.odin): context init +// compiles only what the loader's eager list pulls in, a lazy builtin's source is +// evaluated by its first require and by nothing else, and the loader's instance +// cache keeps a second require from compiling it again. Lava-only: how many +// JSEvaluateScript calls happen before user code is not something Node exposes. +// Runtime_State.builtin_compiles is the observable; it counts successful compiles. +// +// Built by hand rather than through lava.eval so the state outlives each script and +// the counter can be read between them. Scoped linux+darwin, where JSC links. +@(test) +builtin_factories_compile_on_first_require :: proc(t: ^testing.T) { + jsc.lava_jsc_init() + gclass := lava.make_global_class() + gctx := jsc.JSGlobalContextCreate(gclass) + testing.expect(t, gctx != nil, "could not create a JSC global context") + if gctx == nil { + jsc.JSClassRelease(gclass) + return + } + defer jsc.JSClassRelease(gclass) + defer jsc.JSGlobalContextRelease(gctx) + ctx := cast(jsc.JSContextRef)gctx + + state := lava.new_runtime_state(nil) + jsc.JSObjectSetPrivate(jsc.JSContextGetGlobalObject(ctx), cast(rawptr)state) + defer lava.destroy_runtime_state(ctx, state) + lava.capture_error_intrinsics(ctx, state) + jsc.prime_view_probe(ctx) + lava.setup_module_environment(ctx, "", nil) + defer free_all(context.temp_allocator) + + embedded := 0 + for dir in lava.EMBEDDED_JS_DIRS { + for file in dir.files { + if strings.has_suffix(file.name, ".js") do embedded += 1 + } + } + + // The eager closure (primordials, buffer, stream/web, fetch, …, plus stdio and + // stream from installStdio) is well under the whole table. Eager compilation + // compiles every file and fails the upper bound. + at_init := state.builtin_compiles + testing.expectf(t, at_init > 0, "nothing compiled at init — the eager list did not run") + testing.expectf( + t, + at_init < embedded, + "init compiled %d of %d embedded modules — factories are still compiled eagerly", + at_init, + embedded, + ) + + // node:os is outside the eager closure and its requires (primordials, internal/errors, + // buffer through require.pristineBuffer) are eager except internal/os/constants, which + // only os requires, so its first require compiles exactly two factories: its own and + // the constants table. + testing.expect(t, run_bool(ctx, "typeof require('node:os').platform() === 'string'"), "node:os did not load") + testing.expect_value(t, state.builtin_compiles, at_init + 2) + + // Cached by the loader: a second require compiles nothing. + testing.expect(t, run_bool(ctx, "require('os') === require('node:os')"), "node:os was re-instantiated") + testing.expect_value(t, state.builtin_compiles, at_init + 2) + + // Ids outside the embedded set stay unknown, compile nothing, and never reach + // an inherited property or a path outside js/. + UNKNOWN :: `(function () { + var ids = ['node:constructor', 'node:__proto__', 'node:', 'node:../globals', 'node:internal/primordials', 'node:node/os']; + for (var i = 0; i < ids.length; i++) { + try { require(ids[i]); return false; } catch (e) { if (e.code !== 'MODULE_NOT_FOUND') return false; } + } + return true; +})()` + testing.expect(t, run_bool(ctx, UNKNOWN), "an id outside the embedded set resolved") + testing.expect_value(t, state.builtin_compiles, at_init + 2) + + // The ratchet's static eager set (scripts/lib/js-graph.mjs eagerModules) against what + // init really compiled. Static ⊆ runtime is the property that matters: a module the + // ratchet calls eager but that is actually lazy has its module-eval captures exempted + // while they run after user code (#333). Each public id must already be compiled — + // its require compiles nothing; internal ids are not reachable from user code, so they + // are covered by the count: an equal-sized set with every node/ member present leaves + // a wrong internal member only if init also gained an unlisted one at the same time. + // ponytail: count-based for internal/ ids; record compiled ids on Runtime_State if + // that coincidence ever matters. check-js-graph.test.mjs keeps this list current. + for id in STATIC_EAGER { + if !strings.has_prefix(id, "node/") do continue + before := state.builtin_compiles + src, _ := strings.concatenate({"require('node:", id[len("node/"):], "') != null"}, context.temp_allocator) + testing.expectf(t, run_bool(ctx, src), "%s did not load", id) + testing.expectf(t, state.builtin_compiles == before, "%s is listed eager but compiled on first require", id) + } + // After the loop, so a lazy module wrongly listed is reported by name first. + testing.expect_value(t, at_init, len(STATIC_EAGER)) +} + +// The static eager set, sorted. Must equal eagerModules() — asserted by +// scripts/check-js-graph.test.mjs, which parses this literal. +@(private = "file") +STATIC_EAGER :: [?]string{"internal/abort", "internal/blob", "internal/buffer", "internal/crypto/hash", "internal/crypto/hkdf", "internal/crypto/pbkdf2", "internal/crypto/random", "internal/crypto/scrypt", "internal/crypto/util", "internal/encoding", "internal/errors", "internal/fetch", "internal/fetch/body", "internal/fetch/headers", "internal/fetch/request", "internal/fetch/response", "internal/file", "internal/idna", "internal/primordials", "internal/punycode", "internal/stdio", "internal/streams/add-abort-signal", "internal/streams/destroy", "internal/streams/duplex", "internal/streams/end-of-stream", "internal/streams/legacy", "internal/streams/passthrough", "internal/streams/pipeline", "internal/streams/readable", "internal/streams/transform", "internal/streams/utils", "internal/streams/writable", "internal/structured_clone", "internal/url", "internal/webstreams/queuingstrategies", "internal/webstreams/readablestream", "internal/webstreams/transformstream", "internal/webstreams/util", "internal/webstreams/writablestream", "node/buffer", "node/crypto", "node/events", "node/stream", "node/stream/web", "node/string_decoder", "node/url"} + +@(private = "file") +run_bool :: proc(ctx: jsc.JSContextRef, src: string) -> bool { + c_src := strings.clone_to_cstring(src, context.temp_allocator) + script := jsc.JSStringCreateWithUTF8CString(c_src) + defer jsc.JSStringRelease(script) + exception: jsc.JSValueRef + value := jsc.JSEvaluateScript(ctx, script, nil, nil, 1, &exception) + if exception != nil do return false + return jsc.JSValueToBoolean(ctx, value) +} diff --git a/cmd/lava/primordials_test.odin b/cmd/lava/primordials_test.odin index b74483d0..c46cfda0 100644 --- a/cmd/lava/primordials_test.odin +++ b/cmd/lava/primordials_test.odin @@ -91,6 +91,147 @@ primordials_pollution_immunity :: proc(t: ^testing.T) { testing.expectf(t, result.exit_code == 0, "eval exit code=%d (want 0)", result.exit_code) } +// The events-module invariants the post-#485 rewrite added on top of the test above, +// one row each so a mutation's red output names what it broke (expect_detail matches +// the row). Lava-only for the same reason: node fails on-queue (its FixedQueue slots +// are index writes into a holey Array, so an Array.prototype index setter sees them). +// The other four rows also hold on node 26.13 (probed with this script) and were red +// on the pre-rewrite events.js: +// setter0/1 emit()'s snapshot was built by index writes into [], so an +// Array.prototype[0|1] setter threw or silently DROPPED a listener. +// promise-statics once()/on() settled through the live Promise.reject/resolve, +// which a script can replace. +// options-proto on() without options read signal off Object.prototype. +// on-queue events.on() queued through push/shift into an Array. +EVENTS_POLLUTION_SCRIPT :: ` +'use strict'; +const EventEmitter = require('node:events'); +const results = []; +function record(name, fn) { + let out; + try { + out = String(fn()); + } catch (e) { + out = 'THREW:' + e.name; + } + results[results.length] = name + '=' + out; +} + +// emit()'s snapshot of a listener array is an array literal, so an index accessor +// on Array.prototype never sees it. Index 0 and 1 are the slots the copy writes +// first; a copy built by index writes into [] hands every listener to the setter. +for (const idx of ['0', '1']) { + record('setter' + idx, function () { + let stolen = 0; + Object.defineProperty(Array.prototype, idx, { + configurable: true, + set: function () { stolen++; }, + get: function () { return undefined; }, + }); + try { + let calls = 0; + const ee = new EventEmitter(); + ee.on('e', function () { calls++; }); + ee.on('e', function () { calls++; }); + ee.emit('e'); + return calls + ',' + stolen + ',' + ee.listeners('e').length + ',' + ee.rawListeners('e').length; + } finally { + delete Array.prototype[idx]; + } + }); +} + +// once() and events.on() reject/resolve through captured Promise statics, not the +// live Promise.reject / Promise.resolve a script can replace. +record('promise-statics', function () { + const realResolve = Promise.resolve; + const realReject = Promise.reject; + Promise.resolve = function () { throw new Error('Promise.resolve leaked'); }; + Promise.reject = function () { throw new Error('Promise.reject leaked'); }; + try { + const a = EventEmitter.once(new EventEmitter(), 'x', 5); + const it = EventEmitter.on(new EventEmitter(), 'x'); + const b = it.return(); + a.then(null, function () {}); + return (a instanceof Promise) + ',' + (b instanceof Promise); + } finally { + Promise.resolve = realResolve; + Promise.reject = realReject; + } +}); + +// events.on() with no options must not read 'signal' off Object.prototype. +record('options-proto', function () { + Object.prototype.signal = 5; + try { + const it = EventEmitter.on(new EventEmitter(), 'x'); + return typeof it.next; + } finally { + delete Object.prototype.signal; + } +}); + +// events.on()'s queues are not Arrays: neither a replaced push/shift nor an index +// setter on Array.prototype can drop or read a queued event. +const asyncChecks = (async function () { + const realPush = Array.prototype.push; + const realShift = Array.prototype.shift; + let stolen = 0; + Array.prototype.push = function () { throw new Error('push leaked'); }; + Array.prototype.shift = function () { throw new Error('shift leaked'); }; + Object.defineProperty(Array.prototype, '0', { + configurable: true, + set: function () { stolen++; }, + get: function () { return undefined; }, + }); + let got; + try { + const ee = new EventEmitter(); + const it = EventEmitter.on(ee, 'v'); + ee.emit('v', 'a'); + ee.emit('v', 'b'); + const first = await it.next(); + const second = await it.next(); + await it.return(); + got = first.value + '|' + second.value + ',' + stolen; + } catch (e) { + got = 'THREW:' + e.name; + } finally { + Array.prototype.push = realPush; + Array.prototype.shift = realShift; + delete Array.prototype[0]; + } + results[results.length] = 'on-queue=' + got; +})(); + +asyncChecks.then(function () { + const want = ['setter0=2,0,2,2', 'setter1=2,0,2,2', 'promise-statics=true,true', 'options-proto=function', 'on-queue=a|b,0']; + const bad = []; + for (let i = 0; i < want.length; i++) { + if (results[i] !== want[i]) bad[bad.length] = 'want ' + want[i] + ' got ' + results[i]; + } + if (bad.length !== 0) throw new Error(bad.join('; ')); + console.log('events-pollution ok'); +}); +` + +@(test) +events_pollution_immunity :: proc(t: ^testing.T) { + loop := eventloop.init() + // eval consumes (destroys) the loop on every path; do not destroy it here. + result := lava.eval(EVENTS_POLLUTION_SCRIPT, "", &loop, false) + defer lava.result_destroy(&result) + + testing.expectf( + t, + result.status == .Ok, + "eval did not complete cleanly: status=%v message=%q", + result.status, + result.message, + ) + testing.expectf(t, result.exit_code == 0, "eval exit code=%d (want 0)", result.exit_code) +} + // primordials is an internal-only module: internal factories require() it, but it // must not leak through the public resolver (it would shadow a user package named // "primordials" and wrongly answer require('node:primordials')). This evals user diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index e78a499f..f8414e4c 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -11,7 +11,7 @@ Lava is a Node-compatible JavaScript runtime built on **JavaScriptCore (JSC)** as the execution engine and **Odin** as the systems language for everything around it: the engine FFI, the event loop, the I/O transports, and the native half of -the standard library. Compatibility targets modern Node (22+/24 behavior) rather +the standard library. Compatibility targets modern Node (26+ behavior, floor pinned in `.nvmrc`) rather than legacy APIs. Two design decisions shape the whole system: @@ -47,8 +47,9 @@ Two design decisions shape the whole system: │ module_resolution native resolver (file/dir/node_modules/main) │ │ fs / buffer / crypto / dns / sqlite / fetch* native primitives │ │ │ -│ js/internal/*.js embedded spec surface (Buffer, fetch, streams, …) │ -│ js/console.js console implemented in JS over two write prims │ +│ js/node/**.js public builtins; path == specifier (stream/web) │ +│ js/internal/*.js helpers only builtins may require (primordials, …) │ +│ js/bootstrap/*.js loader, microtasks, esm transform, console prelude │ └───────────────┬───────────────────────────────────┬──────────────────┘ │ │ ┌───────────────▼─────────────┐ ┌────────────────▼─────────────────┐ @@ -121,7 +122,7 @@ Thin, per-platform `foreign` declarations of JSC's C API. Highlights: failure the same way — by setting `exception^`, never by its return value — with two audited exceptions recorded at the dispatcher comment in `host_natives.odin`: `net.write` returns a backpressure bool (not a failure - signal), and `crypto.randomFill` returns the view it filled, which `crypto.js` + signal), and `crypto.randomFill` returns the view it filled, which `internal/crypto/random.js` identity-checks as its second fail-closed layer. - The host-call path itself is reached by `dlsym`ing a C++ mangled symbol @@ -134,9 +135,28 @@ Thin, per-platform `foreign` declarations of JSC's C API. Highlights: ### 3.3 Standard library: native + embedded JS -Built-ins are JS factories `#load`-embedded at compile time (the `EMBEDDED_*` -constants in `globals.odin`) and instantiated by `loader.js`. Each factory -receives `(require, module, exports, +Built-ins are JS factories embedded at compile time and instantiated by +`js/bootstrap/loader.js`. A factory's source is evaluated on its module's first +require, through a native `compile(id)` (`builtin_compile_cb`) the loader keeps in its +closure, not at context init: only the eager closure (13 of 39 files) is compiled before +user code, which took `bench/micro/noop.js` from 16.85 to 14.63 ms (median of 50 +interleaved spawn-to-exit runs per arm, darwin arm64, `-o:speed`). The directory +layout under `pkg/runtime/js` is the module table — there is no second list of files: + +| Directory | Holds | Module id / visibility | +| ------------ | ---------------------------------------- | -------------------------------------------------------------------- | +| `node/` | every module user code may `require` | `node/`; file path == specifier (`node/stream/web.js`) | +| `internal/` | helpers (`primordials`, `fetch`, `mime`) | `internal/`; only the factories' own `require` reaches them | +| `bootstrap/` | loader, microtasks, `esm.js`, console | not modules — evaluated standalone by Odin through explicit `#load`s | + +`globals.odin` embeds each module directory whole with `#load_directory` +(`EMBEDDED_JS_DIRS`, one row per directory because the directive is not recursive) and +derives the id and the JSC source name from directory + basename: stack frames read +`node:buffer`, `node:stream/web`, `node:internal/encoding`. The public resolver answers +`node/` ids only, so the directory a file lives in decides whether user code can reach +it; `scripts/check-js-graph.mjs` (`make check-js`) fails on an unembedded directory, an +unresolvable or non-literal `require`, an unreachable file, or a top-level require +cycle. Each factory receives `(require, module, exports, native)`; the native fourth argument carries Odin-backed bindings (crypto, buffer, fetch, sqlite, dns, fs, net, http(s), os, tty, stdio — twelve today, see `install_internal_modules`) so **nothing transient lands on `globalThis`** @@ -153,12 +173,92 @@ attaches them to the `req` it passes each factory, so an internal module reads t It exists because a LAZY module's own `require('buffer').Buffer` runs on the user's first require of that module — after user code, and therefore steerable (#333). `fs`, `http`, `https`, `net` and `os` take it; `stream` and `stdio` do not need it because `installStdio` -instantiates them during bootstrap. - -The JS layer is substantial and high quality: `url.js` (2071 LOC, WHATWG URL), -`streams.js` (2027, Web Streams), `buffer.js` (1646), `fetch.js` (1054), -`crypto.js` (917). Errors use Node-shaped coded errors (`ERR_INVALID_ARG_TYPE`, -`ERR_OUT_OF_RANGE`, …). Native byte ops are gated by a size threshold +instantiates them during bootstrap — and with them every `internal/streams/*` file `stream` +requires at its top level. `internal/webstreams/adapters` is the exception: `Readable`/ +`Writable.fromWeb`/`toWeb` require it on first call (it requires `readable`/`writable` back +at its own top level, so the late edge is what keeps the graph acyclic), as node does. +`internal/blob` is eager yet takes it too, at call time: `buffer` requires it from its own +factory, before the table exists, and a top-level `require('buffer')` back would be a cycle. + +The JS layer is substantial and high quality: URL (~2400 across `node/url.js`, +`internal/url.js` and `internal/{idna,punycode}.js`, WHATWG URL), +Web Streams (~2100 across `internal/webstreams/`), buffer (~1530 across `node/buffer.js` and +`internal/buffer.js`; `Blob`, `File` and the `blob:` URL registry live in `internal/blob.js` +and `internal/file.js`, as in node's `lib/internal`), fetch +(~1160 across `internal/fetch.js` and `internal/fetch/`), crypto (~1070 across `node/crypto.js` +and `internal/crypto/`). `node/stream.js` only assembles the public module; the classic +streams live in `internal/streams/` (`legacy`, `readable`, `writable`, `duplex`, +`transform`, `passthrough`, `destroy`, `add-abort-signal`, `end-of-stream`, `pipeline`, +`utils`) and the Web Streams bridges in `internal/webstreams/adapters.js`, the files node +keeps them in, so their frames read `node:internal/streams/readable`. `node/http.js` is +likewise only the public entry (and `createServer`): the server lives in `internal/http/` +(`common`, `incoming`, `outgoing`, `server`), split the way node splits `_http_common`, +`_http_incoming`, `_http_outgoing` and `_http_server`. Unlike streams the names differ from +node's — node publishes `require('_http_server')` and friends with a parser/agent surface +Lava does not have, so a public file under that name would answer with the wrong keys — +and frames read `node:internal/http/server` where node's read `node:_http_server`. Odin +keys the http bindings by `node/http` only, so `loader.js` aliases that one object to +`internal/http/outgoing` and `internal/http/server`. `node/stream/web.js` similarly +only assembles exports and installs the globals; the implementation is in +`internal/webstreams/` (`util`, `queuingstrategies`, `readablestream` — with tee, async +iteration and pipeTo, as in node — `writablestream`, `transformstream`), all eager through +`stream/web` and acyclic: `readablestream` requires `writablestream`, `transformstream` +requires both, nothing requires back. Fetch follows undici's `lib/web/fetch` seams: +`internal/fetch/headers`, `body` (the Body mixin plus the byte and transport-stream +helpers), `request` and `response` (each requiring `headers` and `body`), while +`internal/fetch` keeps the driver — body collection, redirects, the hand-off to +`native.request` — because the loader keys natives by module id, and installs the globals. +All five are eager through `internal/fetch`. Crypto follows node's `lib/internal/crypto` +seams, so its frames read `node:internal/crypto/pbkdf2` exactly as node's do: +`internal/crypto/util` (validators, byte coercion, the digest table and `getHashes`), +`hash` (`Hash`, `Hmac`, one-shot `hash`), `random`, `pbkdf2`, `hkdf` and `scrypt`, each +requiring only `util`. `node/crypto` keeps `createHash`/`createHmac` and `timingSafeEqual` +where node's `lib/crypto.js` has them, assembles the exports in their old order and +installs `globalThis.crypto`; the five files that call into Odin get the `node/crypto` +bindings through the same `loader.js` aliasing as http. All seven are eager through the one +`req('crypto')` root. Util follows node's `lib/internal/util` seams the same way: +`internal/util/inspect` (`inspect`, `format`/`formatWithOptions`, +`stripVTControlCharacters`, the `inspect.colors` table), `internal/util/comparisons` +(`isDeepStrictEqual`), `internal/util/debuglog` (requiring `inspect` for its formatting) +and `internal/util` (`promisify`, `deprecate`). `node/util` keeps `callbackify`, +`inherits`, `styleText` and `toUSVString` where node's `lib/util.js` has them, plus +`aborted`, and assembles the exports in their old order. Util calls no native, so there +is no alias, and every file stays lazy behind `require('util')`. Buffer takes node's one +seam: `internal/buffer` holds the encoding codecs (`normalizeEncoding`, `strToBytes`, +`bytesToString`) and `addBufferPrototypeMethods`, which installs the fixed-width +`read*`/`write*` accessors and the per-encoding `*Slice`/`*Write` aliases, so a bounds +error frames `node:internal/buffer` as node's does. `node/buffer` keeps the constructor, +the statics, the pool, the `lib/buffer.js` prototype methods and `atob`/`btoa`, calls the +installer where those methods were always assigned (the prototype's key order is +unchanged), and captures the codecs once at module-eval, so a `toString` pays the same +closure call as before. `internal/buffer` decodes through `node/buffer`'s bindings via the +same `loader.js` aliasing, and is eager through `req('buffer')`. `os` moves only its +data: `internal/os/constants` holds the signal, errno, priority and dlopen tables, which +node reads from its C++ constants binding rather than `lib/os.js`. The tables throw +nothing, so no frame moves; they call no native, so there is no alias, and the file stays +lazy behind `require('os')`. URL takes node's `lib/internal/url` seam: `internal/url` holds +the URL Standard parser, `URL`, `URLSearchParams`, `fileURLToPath`, `urlToHttpOptions` and +`domainToASCII`/`domainToUnicode`, so an invalid URL or a bad `fileURLToPath` argument frames +`node:internal/url` as node's does. `node/url` keeps `pathToFileURL` and `format`, which +node's `lib/url.js` defines and frames in `node:url`, installs the `URL`/`URLSearchParams` +globals (after `buffer`, keeping its object-URL statics) and assembles the exports in their +old order, borrowing the record slot, the path serializer and the Punycode decoder for +`format`. Url calls no native, so there is no alias; `internal/url` is eager through +`req('url')`, which keeps its module-eval captures pristine. The IDNA step takes node's +`lib/internal/idna` seam: `internal/idna` (separator fold, UTS-46 approximation) and the +`internal/punycode` codec under it return failure values instead of throwing, so the host +parser and every throw it feeds stay in `internal/url` and no frame moves; both are eager +through `internal/url`'s top-level require, and `internal/url` captures their functions at +module-eval, so the parse path gains no call layer. `internal/punycode` is not node's public +`punycode` (`node/punycode.js`, the userland punycode.js surface). Errors use Node-shaped +coded errors (`ERR_INVALID_ARG_TYPE`, +`ERR_OUT_OF_RANGE`, …), built in one place: `js/internal/errors.js` (eager, hardened) +holds node's `determineSpecificType`, `ERR_INVALID_ARG_TYPE`/`_VALUE`, `ERR_OUT_OF_RANGE`, +`ERR_MISSING_ARGS`, `ERR_UNKNOWN_ENCODING`, and two constructors for node's two shapes — +`nodeError` (node's JS layer: `String(e)` is `TypeError [ERR_X]: msg`) and `nativeError` +(node's C++ layer: `TypeError: msg`). A builtin requires it rather than keeping a local +copy; per-module message templates call `nodeError`/`nativeError` with their own text. +Native byte ops are gated by a size threshold so small inputs avoid FFI overhead — except that the threshold this sentence used to name (`NATIVE_BYTEOP_MIN`) does not exist in the tree: `bytesToString` dispatches to a native codec at every size. Removed rather than left as folklore steering review. @@ -169,18 +269,19 @@ native codec at every size. Removed rather than left as folklore steering review cache → JS internal builtins (`require_builtin`) → filesystem resolution. `node:fs` used to sit between the last two as a natively-assembled object, the one built-in with no JS layer; it is now an ordinary internal builtin - (`js/internal/fs.js` over `make_fs_bindings`). Nothing may rebuild it in + (`js/node/fs.js` over `make_fs_bindings`). Nothing may rebuild it in `native_require_cb`: `require_builtin` runs first, so a second construction would be dead code silently disagreeing with the live one. Circular requires terminate via a pre-registered partial-exports entry (`__lava_precache`), and a module that throws while loading is _removed_ from the cache so a later require re-runs it (Node parity). - `.mjs`/static `import`/`export` are handled by a **source transform** - (`esm.js`) that rewrites ESM onto the CommonJS `require`, not by JSC's native + (`js/bootstrap/esm.js`) that rewrites ESM onto the CommonJS `require`, not by JSC's native module records (the classic C API only runs script-goal source). This is pragmatic and works, with documented divergences (named imports from CJS). - Its output is _executed source_, which is why it is the one internal module the - runtime hands `primordials` as a factory argument: it is deliberately not - registered as a requireable module (so user code cannot reach the transform), + Its output is _executed source_, which is why it is the one internal file the + runtime hands `primordials` as a factory argument: it lives in `bootstrap/`, which + no `#load_directory` reads, so it is never a requireable module (user code cannot + reach the transform), and that also denies it a `require`, so `loader.js` exposes the already-built table on the resolver object and `globals.odin` passes it in — failing closed if it is absent, since a forged match group here is code injection into the module @@ -292,8 +393,9 @@ base Error with an overridden name only when the constructor is unreachable. `make_js_error` / `make_js_named_error` and the typed `ERR_*` helpers (`err_out_of_range`, `err_invalid_arg_type`, …) all route through it; the ad-hoc "base Error + manual `name`/`code` patch" sites (process.exit, sqlite, module -resolution) were migrated. The JS layer still sets `err.code` per module; growing -the `ERR_*` helper list lets native sites adopt the same codes as they need them. +resolution) were migrated. The JS layer builds its coded errors through +`js/internal/errors.js` (§3.3); growing the `ERR_*` helper list lets native sites adopt +the same codes as they need them. ### 4.5 Primordials in the JS layer @@ -310,7 +412,7 @@ routing a per-code-unit codec loop through it cost 1.43x — hence into a null-prototype array where the loop is the inner loop. The loader **eager-loads it first**, before any other internal module and before user code, so the captured references are -pristine; modules consume it via `require('primordials')` and get the cached table +pristine; modules consume it via `require('internal/primordials')` and get the cached table (one exception — `esm.js`, which has no `require`; see §3.4). This is the JS-layer analog of the native error-intrinsic capture (§4.4). @@ -338,6 +440,30 @@ pollutable site, each baselined separately in | `accessor` | `view.buffer` | a configurable prototype getter | | `global` | `String(x)` | a replaceable global, read live | +`Function.prototype.call/apply/bind` themselves are **locked** before user code: +the loader calls `lockIntrinsics()` right after the primordials table is captured, +turning each into a non-configurable accessor in the shape of node's +`--frozen-intrinsics` (assignment on `Function.prototype` throws in both modes; +`f.call = x` on an ordinary function still defines an own property). That is a +deliberate node deviation (#450), pinned by `cmd/lava/function_call_lock_test.odin`. +It does **not** make `.call` safe on a captured INTRINSIC: user code holds the same +object, so `Function.prototype.apply.call = evil` defines an own `call` that +`apply.call(listener, …)` in `events.js` then runs, and `Array.prototype.push.call = evil` +reaches the `callerN` wrappers' `fn.call` (both probed on `bin/lava`; node runs +neither). So the `invoke` class counts every `.call`/`.apply` (and `method` every +`.bind`) except on one receiver: a strict function declaration the file never lets +go of — one binding of the name in the file, never written, referenced only as +`g(…)` or `g.call/apply/bind(…)` inside its declaring scope, and no direct `eval` +(`capturedCallees` in `primordials-detect.mjs`). A suppression marker may carry its +reason after an em dash: `// primordials-ok: invoke — why`. + +A `global` read at module-eval is exempt only in a module instantiated before user +code — the loader's `req()` roots closed over top-level requires, computed by +`scripts/lib/js-graph.mjs` (the parser the graph rules in `lib/js-graph-check.mjs` +also use). In a lazy module the factory runs on the user's first `require()`, so its +capture counts (#333). +`cmd/lava/lazy_builtins_test.odin` checks that set against what context init compiles. + `make check-primordials` (part of `make check-js`) fails on any per-class increase, and `--update` refuses to raise a floor without `--allow-raise`. The detector self-tests against known-positive and known-negative fixtures on every @@ -348,8 +474,8 @@ is deliberately stronger than Node. A baseline of 0 in a class means "no **counted** site of that class remains", not "immune" — the ratchet is a floor, not a proof. Per-class counts are what make -that readable: `events.js` is at 0 `method` and 0 `accessor` while still carrying -`invoke` and `global` sites, which a single total hid. Do not restate the numbers +that readable: `events.js` is at 0 `method`, `accessor` and `global` while still +carrying `invoke` sites, which a single total hid. Do not restate the numbers here — `pollution-baseline.json` is the source and `make check-primordials` prints the live split; the counts copied into prose went stale twice. `.call`/`.apply` and live globals used to be the author's problem and are now gated classes; @@ -381,10 +507,10 @@ member expression in the source to see. So the explicit half is gated and only t assimilation half is blind, which is why the guidance is a code convention rather than a counter: never `await` a caller-supplied value directly. -`primordials` is internal-only: the loader serves it to internal factories but -hides it from the public resolver native `require()` consults, so it neither -shadows a user package named `primordials` nor answers `require('node:primordials')` -(which Node rejects). The _specifier_ is what is hidden — the table itself also hangs -off that resolver object as `publicReq.primordials`, which is how `esm.js` receives it -(§3.4); the resolver is reachable only from Odin and never reaches user code. Gating -the other internal helper modules the same way is a future follow-up. +`primordials` is internal-only by living in `js/internal/`: the public resolver native +`require()` consults answers `node/` ids only, so it neither shadows a user package +named `primordials` nor answers `require('node:primordials')` (which Node rejects). The +same holds for every file under `internal/`. The _specifier_ is what is hidden — the +table itself also hangs off that resolver object as `publicReq.primordials`, which is +how `esm.js` receives it (§3.4); the resolver is reachable only from Odin and never +reaches user code. diff --git a/docs/LARVA.md b/docs/LARVA.md index 3cddbf3f..816297e7 100644 --- a/docs/LARVA.md +++ b/docs/LARVA.md @@ -218,15 +218,15 @@ core has fake geometry. Sensor-script and captcha sites stay on Chrome either wa ## Roadmap -| # | Stage | Done when | Status | Issues | -| --- | --- | --- | --- | --- | -| 0 | **Oracle** — record what Chrome returns (HTML, status, cookies) for 100–200 URLs from real JS traffic | Every later stage is measured against it | Not started | [#2](https://github.com/x-ror/larva/issues/2) | -| 1 | **Multi-VM core** — per-page `Runtime_State`, VM pool per thread, execution limits, Window profile skeleton, start-up cost measured | N pages run concurrently in one process with isolated VMs; a runaway page is killed alone | In progress | [#3](https://github.com/x-ror/larva/issues/3), [#4](https://github.com/x-ror/larva/issues/4), [#5](https://github.com/x-ror/larva/issues/5) | -| 2 | **Network** — BoringSSL, HTTP/2 (nghttp2), per-session connection pool and TLS sessions, CONNECT proxies, cookie jar, redirects, gzip/brotli/zstd, Chrome client profiles | ClientHello and H2 fingerprints match the claimed Chrome version | Not started | [#6](https://github.com/x-ror/larva/issues/6), [#7](https://github.com/x-ror/larva/issues/7), [#8](https://github.com/x-ror/larva/issues/8), [#9](https://github.com/x-ror/larva/issues/9) | -| 3 | **HTML + DOM** — streaming parser (pause/resume at `