Performance roadmap after Chez-first study: current-kernel benchmarks, checker qualification, typed numeric regions and cache semantics #98

Problem and intended outcome

Turn the Chez-first investigation into measured improvements to Shen-Lua's real applications, using the existing LuaJIT runtime and native logic engine. Establish a reliable current-kernel baseline first, then qualify the exact upstream checker change and prototype explicit typed numerical storage. Preserve ordinary Shen semantics and publish negative results when an optimization does not help.

Source reviewed: c6c3a2280f6276e8b2b89b231257f50776c583ca (live main, refreshed on 2026-10-22). The earlier review was against e3dec8b; this issue explicitly incorporates the changes merged since then. This is a source-based roadmap: no new Shen-Lua performance run or whole-port qualification was performed for this issue.

Related open tracking: #72 (native coverage/whole-query fallback) and #73 (guard/materialization boundary costs). Use those for their existing work rather than creating duplicate implementation tickets.

Already implemented — build on these

#96 reports a substantial improvement on capcov's b-typed workload. Those are that PR's reported measurements, not measurements repeated by this review. Use current main as the baseline for subsequent work.

1. Repair and pin the performance harness

Current source evidence

• bench/typecheck_time.lua and bench/typecheck_alloc.lua call chdir("../cl-source/ShenOSKernel-42/tests") without checking its result.
• bench/golden_typecheck.lua explicitly requires that external Shen 42 location.
• The vendored kernel provenance is community Shen 43.1, commit 715824a52b55e9c81a9501df8548ff3edd8411f5, archive SHA-256 900eedef44f440af78f9395ed06e689ce1604546fc309f5bc7c8bc84cc651920.

An unchecked directory change and an unpinned external fixture can produce a failed or mismatched benchmark. Historical handoff counts, inference totals and timings are not current qualification.

Work and acceptance

• Replace the external 42 paths with repository-owned, authenticated fixtures; fail explicitly on missing inputs.
• Record source SHA, kernel/archive identity, actual loaded fixture hashes, Lua/LuaJIT version, engine switches, compiler settings and cache configuration.
• Separate bootstrap, source read/lower/compile, first typecheck and persistent queries; reset or delta inference counters per query.
• Include authentic interpreter/parser and application typechecks, successful and rejected judgments, nested hypotheses and multiple datatype alternatives.
• Retain every paired process sample and validate complete results; do not report only a best/minimum time.
• Keep normal GC-on timing separate from allocation diagnostics. Heap occupancy with GC enabled is not cumulative allocation.
• Make the corrected commands reproducible from a clean checkout with private cache directories.

This is the first implementation slice, because it supplies the baseline for the remaining lanes.

2. Qualify the exact upstream dereference-once kernel with the current native adapter

Kernel identity matters

The Chez study used upstream shen-sources commit ec16a9fa4fc2e87e2aa336fb1ae30e8ac2266394, merged through Shen-Language/shen-sources#117. Shen-Lua still vendors the 43.1 release commit above. Both report version 43.1; that does not make their checker sources identical.

Current main already tests a dereference-hoist fixture through #97, but its fixture provenance identifies an earlier 5596445 source change. Source comparison with the authenticated ec16a9f kernel shows a concrete distinction: ec16a9f's shen.by-hypothesis and shen.search-user-datatypes wrappers first reject an empty hypothesis/datatype list, before doing the full dereference. The fixture wrappers dereference unconditionally. Their -h helpers match after gensym normalization.

Therefore do not treat the existing fixture as qualification of the exact merged kernel, or silently label it as that release. This review did not execute the ec16a9f kernel on current Lua.

Work and acceptance

• Authenticate the exact full kernel build and compare all official KL files, not just a copied t-star.kl. A published build of unmodified ec16a9f is available as ShenOSKernel-43.1-ec16a9f.tar.gz, SHA-256 060066a8f84d9a2f7d41cb3b37b986a8d4653279752b739b5063cd648d9baacd.
• Add/qualify the exact merged-kernel fixture and update provenance explicitly. Keep the release kernel as a matched control.
• Exercise derived driver closure and SHEN_TYPECHECK_NATIVE=require; record actual native routing versus fallback. A passing verdict via silent/unexpected legacy execution is not evidence of native support.
• Review handported operations before accepting new fingerprints. Regenerating the known table alone is not semantic qualification.
• Cover empty hypotheses/datatypes, accepted/rejected nested terms, failed earlier alternatives, binding/backtracking, cut, spy/legacy fallback, missing helpers and runtime errors with observable effects.
• Compare inferred results and observable effects with the kernel's KL path; explain any inference-count differences.
• Exercise cache off/cold/hit, source loading and standalone bundle loading, then run the authenticated community suite and port tests in the relevant engine configurations.

Residual conformance risk

#97 explicitly notes that prims.lua overrides some Prolog operations for both native and legacy paths. Falling back to the legacy checker does not protect against a semantic mismatch in a shared override. Extend the conformance work to define what a mismatched shared primitive must do; do not assume the fallback is an independent implementation of those operations.

For performance, separate legacy repeated traversal from remaining native boundary costs. #96 already removes eligible equality materialization, and native datatype dispatch already avoids the legacy full-term copy boundary. Use #73 to measure remaining materialization, guards and conversions before another specialization.

3. Prototype explicit typed numerical storage and coarse operations

Motivation and existing hooks

The historical Chez experiment's complete 100×100 SOR solve showed a paired CPU improvement of about 9.157× and much lower allocation through explicit managed binary64 storage. It ran on a busy host, its direct typed control was the same implementation, and the figures predate the Chez rebase/refactor. They are motivation for a Lua experiment, not a projected Lua speedup. See the historical experiment report and current Chez integration boundary.

Shen-Lua's generic vectors remain tables. Interop preserves otherwise unsupported values as opaque boxes, and declared Shen-facing bridges already have tests. Build on those facilities.

Work and acceptance

• Add an opt-in, checked owner for LuaJIT double[] storage, leaving ordinary Shen numbers/vectors unchanged.
• Specify explicit binary64 conversion, finite-value policy, bounded dimensions and exact indices, aliasing, lifetime/revocation and import/export behavior.
• Define PUC-Lua behavior: a qualified fallback or an explicit unsupported result.
• Keep the numerical operation separate from generic storage. Start with one batched application operation and preserve resident buffers across calls.
• Validate arguments before mutation; distinguish argument rejection from a numerical execution failure and document whether partial writes remain.
• Exercise typed callers, invalid types, closed aliases and execution-failure controls.
• Compare complete application output and stopping conditions against an independent/reference path, including conversion, initialization, reductions, export and cleanup.
• Compare current generic-table execution with the checked buffer path in matched fresh processes, JIT on/off where relevant, and retained repeated requests.

LuaJIT's number/table/trace representation differs from Chez's generic vectors. Do not copy the one-cell reduction scratch or assume the same boxing/allocation problem without measurement. A second runtime or an automatic hot-swap dispatcher is not required for this experiment.

4. Expand cache-semantic coverage before broader replay

The FASL key includes load history and live datatype/macro names. Existing boot-cache state fingerprints already test significant reconstructed state and a real typecheck. Extend that foundation.

• Test same-name macro/datatype replacement, destroy, typed-bridge registration, changed dependencies and failed loads.
• Keep fix(fasl): replay a nested (load) once on a warm hit (#92; stacked on #89) #94's nested-load exactly-once checks in the corpus; extend them for any new cache behavior.
• Compare metadata, arities, type signatures, output/effects and errors across off/cold/warm configurations.
• Decide explicitly whether interactive same-name mutation invalidates a cached artifact or is excluded by its documented contract.
• If broader replay is justified, fingerprint the state that affects compilation, rather than assuming equal names imply equal definitions.
• Preserve cache-format/version invalidation and source/flag fingerprints for any new compiler or runtime feature.

This is a coverage/invalidation investigation, not an assertion that a new cache bug has already been reproduced.

Measurement and completion gates

• Alternate baseline/candidate order across multiple fresh processes; identify the process as the statistical unit and retain raw samples, medians, paired ranges and wins.
• Keep startup, compilation, cache miss/hit, first call and persistent complete requests separate. Include conversion/ownership costs and do not substitute a kernel microbenchmark for an application.
• Record engine routing, inference/result identities and fallback reasons. Keep profiling/counters/trace dumps outside accepted timing cohorts.
• Use private cache state and bounded processes; bind source/kernel/runtime/artifact hashes before and after. Record host contention; busy-host timing remains exploratory.
• Require focused semantic regressions and the applicable full port/community kernel suites before adopting a performance change.
• Preserve rejected experiments. The Chez compiler arity shortcut was rejected after insufficient application benefit; synthetic-only gains are not an adoption criterion.

Suggested implementation slices

1. Harness repair and current-main baseline.
2. Exact ec16a9f fixture/kernel qualification, conformance policy and engine-mode matrix; coordinate with Track native Prolog/typechecker coverage and whole-query fallback #72/Measure native logic-engine guard/materialization boundary costs #73.
3. Optional typed numerical buffer plus one complete application.
4. Cache mutation coverage, followed by a narrowly measured change if the evidence supports it.

Done means: reproducible source-bound evidence for each adopted change, complete result/behavior checks, explicit supported modes and honest remaining gates. No general backend or universal speedup claim is intended.