Merge kernel/shen-42 (incl. Shen/Scheme mappings) into main

BENCHMARKS.md

@@ -10,21 +10,34 @@ ratio, not the absolute. Harnesses live in `crates/shen-rust/benches/`,
 `scripts/cross-port-bench.sh` runs the upstream Shen kernel test suite (134
 tests) through both ports on the same machine, interleaved.
 
+**S41.2 (2026-06-10, paired interleaved min-of-5)** — historical. Do not read
+these 3.0s figures as S42.
+
 | Port | `--kernel-tests` | vs shen-cl |
 |---|---:|---:|
 | shen-cl (SBCL interpreted) | ≈ 1.0 s | 1× |
 | **shen-rust (release, bare)** | ≈ 3.0 s | **~3.0×** |
 | shen-rust + warm tc-cache (`SHEN_RUST_TC_CACHE`) | ≈ 1.0 s | **~1× (parity / ahead)** |
 
-(Paired interleaved min-of-5, 2026-06-10. The 2026-06-09 snapshot read
-≈4.3 s vs ≈1.3 s = ~3.3×; the 2026-06-10 profiling round — thin-LTO build
-flags, the split-TLS heap, the direct-mapped intern cache — bought ~18%
-cumulative, and shen-cl also ran faster on this box that day. The *ratio* is
-the stable claim.) Down from ~17× at first conformance (see `PERFORMANCE.md`
-for the path). The remaining bare gap is the boxed-`Value` +
-interpreted-dispatch model, not a single hot spot — each remaining local
-lever measures ≤ ~8%. The tc-cache row is verdict memoization (off by
-default), not raw speed.
+(The 2026-06-09 snapshot read ≈4.3 s vs ≈1.3 s = ~3.3×; the 2026-06-10
+profiling round — thin-LTO build flags, the split-TLS heap, the direct-mapped
+intern cache — bought ~18% cumulative, and shen-cl also ran faster on this box
+that day. The *ratio* is the stable S41.2 claim.) Down from ~17× at first
+conformance (see `PERFORMANCE.md` for the path). The remaining bare gap is the
+boxed-`Value` + interpreted-dispatch model, not a single hot spot — each
+remaining local lever measures ≤ ~8%. The tc-cache row is verdict memoization
+(off by default), not raw speed.
+
+**S42 (2026-09-01 retime, core AOT restored; `*version*` 42).** Same suite,
+release `shen-rust --kernel-tests`, `/usr/bin/time -p` wall, min-of-3 on this
+box. PR #21 (`42c2f16`) was ~8s with `core::install` skipped; June S41.2 was
+~3s. This run is **not** 3s.
+
+| Kernel / config | `--kernel-tests` wall | 134/0 | notes |
+|---|---:|:---:|---|
+| S41.2 (2026-06-10, paired min-of-5) | ≈ 3.0 s | yes | historical; vs shen-cl ≈1.0 s |
+| S42 PR #21 (`42c2f16`, core AOT skipped) | ≈ 8 s | yes | first S42 vendor |
+| **S42 (2026-09-01, core AOT restored)** | **≈ 7.9 s** (7.89 / 8.39 / 9.43) | **yes** | same ~8s band as PR #21, not the 3s S41.2 path |
 
 ## Cross-port: the wider field (rust vs cl vs lua)
 
@@ -166,6 +179,9 @@ Gate 7 in `scripts/gates.sh`; Gate 8 repeats it in a debug build, where the
 heap-reentrancy sentinel (`value.rs` split-TLS note) is live; Gate 9 repeats
 the debug run with `SHEN_RUST_GC` at a small trigger floor, so full
 mark/sweep cycles run under the sentinel plus the debug poison-on-sweep.
+S42 release wall on this box (2026-09-01, core AOT restored) is **≈ 7.9 s**
+min-of-3 — quote against PR #21 ~8s and June S41.2 ~3s; do not claim 3s for
+this kernel.
 
 ## Methodology
 

DEMO.md

@@ -17,8 +17,9 @@ Re-verify the whole tour at any time with
 
 ## Build and conformance
 
-One workspace, plain cargo. The build AOT-compiles the entire vendored
-ShenOSKernel-41.2 (21 KLambda files) to Rust via `crates/klcompile`.
+One workspace, plain cargo. The build AOT-compiles the vendored **Shen S42**
+kernel (15 canonical KLambda files plus port extensions; provenance
+`s42-pristine-20260825`) to Rust via `crates/klcompile`.
 
 ```bash
 cargo build --release --quiet --bin shen-rust && echo build ok
@@ -53,7 +54,7 @@ printf '(define fact 0 -> 1 N -> (* N (fact (- N 1))))\n(fact 20)\n(map (* 2) [1
 ```output
 shen-rust booting kernel… ready.
 
-Shen 41.2, ©2021–2026 Mark Tarver  (shen-rust 0.1.0, Rust)
+Shen 42, ©2021–2026 Mark Tarver  (shen-rust 0.1.0, Rust)
 
 (0-) (fn fact)
 (0-) 2432902008176640000
@@ -183,10 +184,12 @@ RESULT: PASS
 ## Performance and verification
 
 Measured against the reference `shen-cl` (SBCL) port, paired and interleaved
-on the same machine: ~3× off bare on a one-shot kernel-tests run (down from
-17× at first conformance), at parity with a warm typecheck cache, and ahead on
-served workloads (VM ~2.3×, AOT overlay ~3× over that). Every number — and
-every experiment that *failed* — is documented in `PERFORMANCE.md` and
+on the same machine: **S41.2** was ~3× off bare on a one-shot kernel-tests run
+(down from 17× at first conformance), at parity with a warm typecheck cache,
+and ahead on served workloads (VM ~2.3×, AOT overlay ~3× over that). S42 PR #21
+was ~8s with core AOT skipped; a 2026-09-01 retime with core restored is ≈ 7.9 s
+(min-of-3, 134/0) — those 3.0s numbers are not S42. Every number —
+and every experiment that *failed* — is documented in `PERFORMANCE.md` and
 `BENCHMARKS.md`, reproducible from `scripts/` and `benches/`.
 
 The repo holds itself to ten CI gates (`scripts/gates.sh`): fmt+clippy,

PERFORMANCE.md

@@ -3,12 +3,23 @@
 ## Current state
 
 On the full `--kernel-tests` suite, against the reference `shen-cl` (SBCL)
-interpreter, `shen-rust` is **~3.0× slower** (≈3.0s vs ≈1.0s wall, paired
+interpreter, `shen-rust` on **S41.2** was **~3.0× slower** (≈3.0s vs ≈1.0s wall, paired
 interleaved min-of-5, 2026-06-10) — down from **~17×** at first conformance.
 With a warm tc-cache (verdict memoization, off by default) it runs **at
 parity**. Measure head-to-head with `scripts/cross-port-bench.sh`
 (interleaved; the machine has ~5–12% thermal variance, so trust min-of-N,
-not single runs).
+not single runs). Those 3.0s figures are **S41.2 history**, not S42.
+
+**S42 (2026-09-01 retime).** Kernel is **Shen S42**, `*version*` 42, provenance
+`s42-pristine-20260825`. PR #21 (`42c2f16`) reported `--kernel-tests` ≈ **~8s**
+(134/0) with **core AOT skipped**: the first S42 `core.rs` codegen was empty, so
+`9f81b64` omitted `core::install` (41.2 had installed core). `839cca1`
+regenerated `core.rs` (~11k lines, `pub fn install`) but left the skip.
+`core.kl` has ~60 defuns (`shen->kl`, `define`, `linearise`, …); AOT install is
+restored. This machine's release retime (`./target/release/shen-rust --kernel-tests`,
+`/usr/bin/time -p` wall, min-of-3, 2026-09-01): **≈ 7.9 s** (7.89 / 8.39 / 9.43 s;
+internal suite `run time` ≈ 6.4 s), **passed: 134, failed: 0**. That is the same
+band as PR #21's ~8s, **not** the June S41.2 ≈3s path — do not quote 3s for S42.
 
 The bytecode VM is **~2.3× faster than the tree-walker on warm / served
 workloads** (`scripts/warm-bench.sh`), which is why it ships behind `--served`
@@ -65,6 +76,57 @@ The living, detailed record is in `design/`:
    work didn't vanish — it CSE'd into `eval_in` and the call sites — but the
    per-step call/return overhead and cold-blob bloat did.
 
+## Shen/Scheme-style compiler mappings + kernel overrides
+
+Shen/Scheme (and the old SBCL port that reused its compiler) is fast not only
+because Chez/SBCL are fast, but because the KL→host compiler exploits KLambda
+properties and because well-placed overrides replace the kernel's portable
+bodies. The kernel is deliberately inefficient on hot functions so a new port
+can boot at all; once the port is up, those bodies are the cheapest wins.
+
+Landed (134/0, kernel AOT regenerated):
+
+1. **Native overrides** (after AOT install, dual-registered into `aot_direct`)
+   for the portable bodies Shen/Scheme also replaces: `hash` (must be live
+   *before* `declarations.kl` fills `*property-vector*` — same algorithm at
+   populate and lookup; 0-guard so bucket 0 stays the vector length),
+   `integer?` (kernel is recursive subtraction via `magless`/`integer-test?`),
+   `symbol?` / `variable?` / `shen.analyse-symbol?` (kernel walks `str` through
+   `alpha?`/`alphanums?` under `trap-error`), `@p`/`tuple?`/`fst`/`snd`,
+   `vector`/`<-vector`/`vector->`/`limit`, `empty?`/`not`/`boolean?`, reader
+   leaves (`hdstr`, `shen.digit?`, `shen.byte->digit`, case predicates).
+2. **klcompile mappings** that Chez already does: `(intern "foo")` →
+   `intern_static`; `(value unbound-sym)` → `rt::global_value`; `(set unbound-sym
+   V)` → `rt::set_global`; `(trap-error (value X) (lambda E H))` → `value/or`
+   and the same for `<-address` (Shen/Scheme `emit-trap-error-optimize`); more
+   primitives inlined (`empty?`, `cn`, `hdstr`/`tlstr`, `string->n`/`n->string`,
+   `<-address`/`address->`, `not`, `boolean?`, `fail`, `intern`). Kernel
+   `symbol?` is **not** inlined to `is_sym()` — that was the KL primitive, not
+   the kernel predicate.
+
+Follow-up (`af86b09`, Astra review): trap-error rewrite is gated on unused `E`
++ non-raising operands; `intern` of `"true"`/`"false"` is Bool on the primitive
+*and* `rt::intern`; `hash` agrees with `shen_eq`; freeze-continuation inlining
+refuses escaping `lambda`/`freeze`; same-module seal is kernel-AOT only;
+native `get`/`put` distinguish uninit vs assoc-miss. Tests in
+`tests/scheme_mapping_soundness.rs`.
+
+**Paired A/B** (this machine, 2026-09-20, interleaved min-of-5, both 134/0,
+`kernel/shen-42` `c8f5cc9` vs `perf/shen-scheme-kl-overrides` `af86b09`):
+
+| | wall (s) | suite `run time` (s) |
+|---|---|---|
+| base | 19.29, 11.59, 13.87, 15.81, 13.19 (**min 11.59**) | 13.40, 8.17, 7.71, 11.88, 8.84 (**min 7.71**) |
+| new | 10.42, 7.29, 14.07, 10.79, 10.27 (**min 7.29**) | 7.46, 5.22, 9.65, 6.56, 7.78 (**min 5.22**) |
+
+Min-of-5 wall **11.59 → 7.29 (~1.6×)**; suite **7.71 → 5.22 (~1.5×)**. NEW won
+wall in 4/5 pairs (pair 3 lost: 13.87 vs 14.07). Variance is large (~2× spread
+within each arm) — treat the factor as directional, not a scoreboard number.
+No ablation of individual mappings.
+
+Still on the table: `yields-boolean?`; dedicated `@p` type; remaining `t-star`
+heap freezes; cross-module seal (needs a documented immutable kernel boundary).
+
 ## Why the remaining ~3.0× is structural
 
 Two execution-engine bets — the bytecode VM and the Cranelift closure-JIT —

README.md

@@ -7,8 +7,9 @@ with native [AWS Cedar](https://www.cedarpolicy.com/) authorization integration.
 
 Shen is a functional language with an integrated logic engine and an optional,
 very expressive type system (a sequent-calculus theorem prover). `shen-rust`
-boots the upstream **ShenOSKernel-41.2** and passes its full conformance suite —
-**134 / 134 kernel tests**, in every execution mode.
+boots the upstream **Shen S42** kernel (`*version*` 42, provenance
+`s42-pristine-20260825`; see `kernel/klambda/PROVENANCE.md`) and passes its
+full conformance suite — **134 / 134 kernel tests**, in every execution mode.
 
 The name follows the Shen-port convention (`shen-cl`, `shen-go`, `shen-ocaml`):
 the engine is `shen-rust`. The name `shen-cedar` is reused for the Shen **+**
@@ -135,7 +136,7 @@ crates/klcompile/           build-time KL → Rust AOT compiler for the kernel
 crates/shengen-rust/        Shen sequent-calc specs → Rust guard types (backpressure)
 bin/shen-rust/              the REPL / CLI (`--served`, `--kernel-tests`)
 examples/shen-cedar-authz/  Shen + Cedar integration (gate / verify / generate)
-kernel/                     vendored ShenOSKernel-41.2 (klambda + conformance tests)
+kernel/                     vendored Shen S42 (klambda + conformance tests)
 specs/                      backpressure specs in Shen sequent-calculus syntax
 design/                     architecture + performance design notes
 scripts/                    gates.sh (CI), benches, cross-port + warm benchmarks
@@ -144,7 +145,12 @@ scripts/                    gates.sh (CI), benches, cross-port + warm benchmarks
 ## Performance
 
 The reference target is the upstream `shen-cl` (SBCL) port. Two metrics, two
-answers (paired interleaved runs, 2026-06-10, Apple M-series):
+answers. The one-shot table below is **S41.2** (paired interleaved, 2026-06-10,
+Apple M-series) — do not read it as S42. PR #21's S42 vendor (~8s
+`--kernel-tests`, 134/0) ran with core AOT skipped. A 2026-09-01 release retime
+with core AOT restored is **≈ 7.9 s wall** (min-of-3: 7.89 / 8.39 / 9.43 s,
+134/0) — the same ~8s band as PR #21, **not** the 3s S41.2 path. See
+`PERFORMANCE.md`.
 
 **One-shot** (`--kernel-tests`, boot + load + run + exit):
 

STATUS.md

@@ -3,7 +3,7 @@
 ## Current state
 
 A working port of the Shen language to Rust. It boots Mark Tarver's
-**Shen S41.2 (2026-07-11 refresh)** kernel (see
+**Shen S42** kernel (`*version*` 42, provenance `s42-pristine-20260825`; see
 `kernel/klambda/PROVENANCE.md`) and passes the full conformance suite — **134 / 134**
 kernel tests (`scripts/kernel-tests.sh`) — in every execution mode. All gates
 green (`scripts/gates.sh`): shengen-codegen, fmt+clippy, build, test
@@ -46,12 +46,14 @@ What's in the tree today:
   types; `specs/core.shen`.
 
 Performance vs the reference `shen-cl` (SBCL) on one-shot `--kernel-tests`:
-**~3.0× bare** (≈3.0 s vs ≈1.0 s, paired 2026-06-10; down from ~17× at first
-conformance), **at parity with warm tc-cache** (≈1.0 s, off by default). The
-remaining bare gap is structural — the boxed-`Value` + interpreted-dispatch
-model, not a single hot spot. On served workloads the story inverts: VM
-~2.3× warm, AOT overlay ~3.1× over that on spec code. Full story in
-`PERFORMANCE.md`, `BENCHMARKS.md`, and `design/perf-*.md`.
+**S41.2 ~3.0× bare** (≈3.0 s vs ≈1.0 s, paired 2026-06-10; down from ~17× at
+first conformance), **at parity with warm tc-cache** (≈1.0 s, off by default).
+Those 3.0s numbers are S41.2, not S42 — PR #21's S42 run was ~8s with core AOT
+skipped; a 2026-09-01 release retime with core AOT restored is ≈ 7.9 s wall
+(min-of-3, 134/0), the same ~8s band, not 3s (see `PERFORMANCE.md`). The remaining bare gap is structural — the
+boxed-`Value` + interpreted-dispatch model, not a single hot spot. On served
+workloads the story inverts: VM ~2.3× warm, AOT overlay ~3.1× over that on spec
+code. Full story in `PERFORMANCE.md`, `BENCHMARKS.md`, and `design/perf-*.md`.
 
 ## Milestones
 
@@ -105,6 +107,15 @@ model, not a single hot spot. On served workloads the story inverts: VM
 - **Rename** — the engine port `shen-cedar` → **`shen-rust`** (the name
   `shen-cedar` now denotes the Shen+Cedar examples). History before that commit
   says "shen-cedar".
+- **S42 kernel** (2026-08-25/27) — vendor Mark Tarver Shen 42.0
+  (`s42-pristine-20260825`; PR #21 squash `42c2f16`). `(value *version*)` is
+  `"42"`. First S42 `core.rs` codegen was empty, so `9f81b64` skipped
+  `core::install`; `839cca1` regenerated `core.rs` (~11k lines, `pub fn
+  install`) but left the skip. `core.kl` has ~60 defuns (`shen->kl`, `define`,
+  `linearise`, …); 41.2 installed core, and AOT install is restored. Release
+  `--kernel-tests` retime 2026-09-01: ≈ 7.9 s wall (min-of-3), 134/0 — vs PR #21
+  ~8s and vs June S41.2 ~3s (do not quote 3s for S42). GC, VM, overlay, and
+  Cedar are unchanged.
 
 ## Known limitations
 

bin/shen-rust/src/main.rs

@@ -1,6 +1,6 @@
 //! shen-rust REPL.
 //!
-//! Boots the ShenOSKernel-41.2 then routes each user line through the
+//! Boots the Shen 42 kernel then routes each user line through the
 //! kernel's own `eval` (which in turn does `(eval-kl (shen.shen->kl …))`).
 //! Input is read one balanced s-expression at a time — partial lines are
 //! buffered until the parentheses balance.