diff --git a/.claude/board/AGENT_LOG.md b/.claude/board/AGENT_LOG.md index 0689b9f6..73b7505e 100644 --- a/.claude/board/AGENT_LOG.md +++ b/.claude/board/AGENT_LOG.md @@ -1,3 +1,15 @@ +## 2026-06-14 — edge-codec flavors: all three implemented, class-selectable, measured (ICC/Pearson/Cronbach/Spearman) + +**Main thread (Opus 4.8 1M), branch `claude/wonderful-hawking-lodtql` (ndarray + lance-graph).** Operator (relaying a second session's 32×4-bit edge proposal + the key insight *"you could literally combine deterministic↔residue"*): *"implement all and allow the class>schema inheritance mapping to choose which flavor … measure all options and validate/invalidate ICC Pearson Cronbach alpha … using the JC crate and ndarray/crates/hpc/pillars."* Resolved the second session's false dichotomy: the deterministic part (nearest-centroid palette index, recomputed via AMX `matmul_i8_to_i32`) is the EdgeBlock byte unchanged; the residue is a value-slab 4-bit plane — so all flavors are *interpretations* of the locked 16-byte block, none changes `NODE_ROW_STRIDE`. + +**Shipped:** +- **ndarray** (`d3b608f`): `hpc::reliability` (Pearson/Spearman/Cronbach α/ICC(2,1) + `FidelityReport`; the JC-consumable measurement layer — jc/pillar had only private Spearman, missing the other three) and `hpc::edge_codec` (Codebook k-means, `CoarseResidueCodec`, `ProductQuantizer`, `reconstruct_coarse`). Harness `examples/edge_codec_compare`. 16 unit + 9 doctests; lib clippy `-D warnings` clean. +- **lance-graph contract**: `EdgeCodecFlavor` enum (`CoarseOnly`/`CoarseResidue`/`Pq32x4`) + defaulted `ClassView::edge_codec_flavor` selector (non-breaking). +3 tests, 609 lib green, clippy clean. LATEST_STATE contract inventory updated (same commit). + +**Measured (AMX host):** CoarseResidue dominates agreement (ICC 0.97–0.99, ρ 0.98, α 0.99 across blob+continuous); Pq32x4 preserves rank (ρ 0.60–0.67) but not absolute distance (ICC 0.11–0.29 — the Pearson-vs-ICC contrast working as designed); CoarseOnly collapses on continuous (ICC 0.003). AMX assign 100% vs scalar, 24–28 GMAC/s. + +**Deferred/flagged:** turbovec PQ4 *throughput* path blocked on #493 P2 (turbovec `ndarray-simd` feature removed in `7fa217c`; polyfill fns gone). Fidelity is kernel-independent, so throughput-only follow-up. Per-class flavor STORAGE (override `edge_codec_flavor` from a class config) = follow-up in `lance-graph-ontology`. Also fixed bgz17 SIMD gather OOB (P1 from #493, commit `6d48ced`). + ## 2026-06-13 — turbovec ⇄ ndarray integration: fork-wired + ndarray::simd polyfill GEMM + measured AMX-vs-LUT **Main thread (Opus 4.8 1M) + 1 Opus general-purpose agent (bgz-tensor synergy map).** User: "create a crate in lance-graph for turbovec and check synergies; route SIMD through ndarray::simd (simd.rs→simd_amx/avx512/ops/soa); the polyfill does the work, ndarray ships AMX via byte-asm dispatch; pin rust 1.95." Cross-repo, branch `claude/wonderful-hawking-lodtql` in all three repos. diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 7bdff330..bd8fe084 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,19 @@ +## 2026-06-14 — entropy-ladder-spo-rung-v1 (Staunen↔Wisdom entropy coordinate unifies SPO rungs + NARS reliability + edge-codec; R1 shipped, R2–R6 roadmap) + +**Status:** PLATEAU — R1 (foundation) SHIPPED; R2–R6 planned (probe-gated where marked). **Plan file:** `.claude/plans/entropy-ladder-spo-rung-v1.md`. +**Owns:** 7 deliverables D-EL-1..6 + D-EL-COCA. +- D-EL-1: entropy-ladder foundation — `ndarray::hpc::{reliability, edge_codec, entropy_ladder}` + `EdgeCodecFlavor` selector + bgz17 OOB fix (SHIPPED; ρ=−0.78 reliability proxy; ICC/Pearson/Cronbach/Spearman; edge-codec fidelity table) +- D-EL-2: `entropy_class` → CausalEdge64 spare bits [63:61] (version-gated, field-isolation tests) — next +- D-EL-3: CAM-PQ AMX centroid assignment (assignment-as-GEMM + 2×2/4×4 tiled grid) — queued +- D-EL-4: HHTL+helix basin attraction (probe-gated, +15% recall vs HHTL-alone) +- D-EL-5: Markov SPO rung-ladder → episodic context/basins/supporting edges (probe-gated) +- D-EL-6: energy axis / particle↔wave (gated on Mailbox-SoA map) +- D-EL-COCA: superposition 2/3 pruning (needs cluster-identity layer; I-VSA-IDENTITIES design-gate) +**Key decisions:** read the 3×palette256 SPO already in CausalEdge64 (NO re-quant); Pearl 2³ mask = the 8 SPO iterations; flavors/classes are interpretation not layout (no NODE_ROW_STRIDE / no ENVELOPE_LAYOUT_VERSION bump); COCA pruning must bundle identities not PQ codes (I-VSA-IDENTITIES). +**Repos:** ndarray (`d3b608f`, `83be7c3`) + lance-graph (`6d48ced`, `920671d`), branch `claude/wonderful-hawking-lodtql`. PRs: this plateau pair. + +--- + ## 2026-06-09 — polyglot-container-query-membrane-v1 (Node Container answers Cypher + SurrealQL AST + DataFusion UDF over one HHTL address space; mailbox = a normal cold path) **Status:** RESEARCH MAP + PLAN. Grounded by two parallel sweeps (lance-graph + surrealdb fork) with main-thread spot-verification; one agent claim caught false (SoaEnvelope has ZERO real impls — identity N3 stands live). **Plan file:** `.claude/plans/polyglot-container-query-membrane-v1.md`. diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 5d08e121..273c30eb 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -512,3 +512,13 @@ PR sequence: #360 → #361 → post-#360 substrate-sweep (this PR). > **2026-06-01 — PR-in-flight (autoattended)** (D-EW64-3/4): `lance_graph_contract::episodic_edges` gains `EpisodicEdges64::{coldest, contains, promote_into}` + the `DemotionSink` trait. `coldest()` = the eviction victim (symmetric to `strongest()`); `contains()` = family-discriminating membership; `promote_into(e, sink)` = `promote` routing the evicted (coldest) edge to a `DemotionSink` — the hot→cold connectome exit. `DemotionSink` impls (surreal/LanceDB-LIVE "wingman", `E-SUBSTRATE-IS-THE-SCHEDULER`) are deferred + GATED on OQ-11.6. Zero-dep; contract lib 545 green; default clippy clean; `episodic_edges.rs` pedantic+nursery clean. > **2026-06-01 — Shipped (autoattended, 5-agent council)** (D-ATOM-4/RawEdge): `contract::counterfactual` wired into `lib.rs` (was orphaned); `RawEdge(i8)` mantissa-only **structural** impl of `EpisodicEdge` (`size_of==1` — a u64 newtype could read plasticity 50–52); `deposit_counterfactual` v2 filled (−6 on split). Closes the counterfactual seam (NOT the prefetch loop). +3 latent scaffold fixes. 550 contract lib green, clippy clean. The council REFUTED the prior "compose `Heel.plasticity` × MRU" ① resolution (`E-BASIN-NOT-EDGE-PLASTICITY`): coarse strength = MRU slot-order (shipped); per-edge Hebbian = per-plane `PlasticityState` (gated). + +## 2026-06-14 — Append: `EdgeCodecFlavor` selector + ndarray edge-codec/reliability layer (branch `claude/wonderful-hawking-lodtql`) + +(Per APPEND-ONLY rule: new top-of-inventory entry. Branch work; records the contract type so a new session does not re-derive it.) + +### Current Contract Inventory — new entry + +- **`canonical_node::EdgeCodecFlavor`** (NEW; re-exported from `lib.rs`). Per-class selector for how a node's 16-byte `EdgeBlock` (+ optional value-slab residue) is *interpreted* — `CoarseOnly` (1 B palette index, the canon zero-fallback default), `CoarseResidue` (1 + ⌈D/2⌉ B, value-slab signed-4-bit residue), `Pq32x4` (16 B = 32×4-bit product code, the turbovec PQ model). `bytes_per_vector(dim)` + `is_layout_preserving()` (always `true`). **Iron invariant:** the flavor is interpretation, NOT layout — every variant leaves `NODE_ROW_STRIDE = 512` untouched, so adoption needs no `ENVELOPE_LAYOUT_VERSION` bump (canon "registry-resolved via `classid → ClassView`"). Default `CoarseOnly` matches the all-zero bootstrap. Selection surface: new defaulted `ClassView::edge_codec_flavor(&self, ClassId) -> EdgeCodecFlavor` (non-breaking — `RegistryClassView` inherits the default; per-class override is the follow-up wiring in `lance-graph-ontology`). +3 tests; contract lib 609 green; clippy `-D warnings` clean. +- **Encode/measure kernels live in `ndarray` (the hardware layer), not the contract:** `ndarray::hpc::edge_codec` (Codebook k-means, `CoarseResidueCodec`, `ProductQuantizer`, `reconstruct_coarse`) + `ndarray::hpc::reliability` (Pearson r, Spearman ρ, Cronbach α, ICC(2,1), `FidelityReport`). Harness `examples/edge_codec_compare` measures all flavors × {blob, continuous} regimes. **Measured:** CoarseResidue dominates agreement (ICC 0.97–0.99, ρ 0.98, α 0.99); Pq32x4 keeps rank (ρ 0.60–0.67) but not absolute distance (ICC 0.11–0.29); CoarseOnly collapses on continuous (ICC 0.003); AMX `matmul_i8_to_i32` assign 100% vs scalar, 24–28 GMAC/s. ndarray commit `d3b608f`. +- **Deferred (flagged):** turbovec PQ4 *throughput* path (the FastScan nibble-LUT for the Pq32x4 flavor) blocked on the **#493 P2** build break — `lance-graph-turbovec` requests the `ndarray-simd` turbovec feature that was REMOVED (turbovec commit `7fa217c`); the polyfill fns are gone. turbovec's API is end-to-end (owns encode+scan), so it is a *PQ4 flavor*, not a residue-nibble-scan primitive. Fidelity (what ICC/Pearson/α measure) is independent of the fast kernel, so this is throughput-only follow-up. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index feeeb8da..1ecbbcb0 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -1,3 +1,19 @@ +## entropy-ladder-spo-rung-v1 — Staunen↔Wisdom entropy coordinate unifies SPO rungs + NARS reliability (R1 shipped; R2–R6 roadmap) + +Plan path: `.claude/plans/entropy-ladder-spo-rung-v1.md`. Foundation: `ndarray::hpc::{reliability, edge_codec, entropy_ladder}`. Selector: `lance-graph-contract::EdgeCodecFlavor`. + +| D-id | Title | Crate(s) / repo | Risk | Status | PR / Evidence | +|---|---|---|---|---|---| +| D-EL-1 | Entropy-ladder foundation (reliability + edge_codec + entropy_ladder + EdgeCodecFlavor + bgz17 fix) | ndarray + lance-graph-contract + bgz17 | LOW | **In PR** | `d3b608f`,`83be7c3`,`920671d`,`6d48ced`; ρ=−0.78; ICC 0.97–0.99 | +| D-EL-2 | `entropy_class` → CausalEdge64 spare bits [63:61] | `causal-edge` | MED | **Queued** | version-gated + field-isolation (I-LEGACY-API-FEATURE-GATED) | +| D-EL-3 | CAM-PQ AMX centroid assignment (GEMM + 2×2/4×4 grid) | `ndarray` | MED | **Queued** | bit-exact + GMAC/s probe | +| D-EL-4 | HHTL+helix basin attraction | `lance-graph` + `helix` | MED | **Probe queued** | +15% recall vs HHTL-alone gate | +| D-EL-5 | Markov SPO rung-ladder → episodic context | `deepnsm` / `lance-graph` | MED | **Probe queued** | prune-without-recall-loss gate | +| D-EL-6 | Energy axis / particle↔wave | `lance-graph` MailboxSoA | MED | **Blocked** | gated on Mailbox-SoA map | +| D-EL-COCA | Superposition 2/3 pruning (cluster-identity layer) | `deepnsm` | HIGH | **Design** | I-VSA-IDENTITIES design-gate | + +--- + ## singleton-to-snapshot-nudge-v1 — every shared-mutable singleton → per-owner MailboxSoA + Arc-swap snapshot (7 deliverables; codebooks left as-is) Plan path: `.claude/plans/singleton-to-snapshot-nudge-v1.md`. Companions: `bindspace-singleton-to-mailbox-soa-v1` (BindSpace dissolution), `cycle-coherent-soa-snapshot-v1` (snapshot mechanism). Debt: TD-UNBUNDLE-FROM-1. diff --git a/.claude/plans/entropy-ladder-spo-rung-v1.md b/.claude/plans/entropy-ladder-spo-rung-v1.md new file mode 100644 index 00000000..ffee95b5 --- /dev/null +++ b/.claude/plans/entropy-ladder-spo-rung-v1.md @@ -0,0 +1,105 @@ +# Entropy-Ladder SPO Rung Decomposition — v1 + +> **Status:** PLATEAU — R1 (foundation) SHIPPED; R2–R6 planned (probe-gated where marked). +> **Confidence:** R1 measured (ρ=−0.78 reliability-proxy); R2/R3 unblocked; R4/R5 probe-first; R6 gated on the Mailbox-SoA map. +> **Date:** 2026-06-14 +> **Branch:** `claude/wonderful-hawking-lodtql` (ndarray + lance-graph) +> **Owner:** main thread (Opus) + +## The Click — one coordinate unifies the stack + +**A fact's position on the cognitive ladder IS an entropy level.** The same +scalar orders SPO triples, NARS truth, and the linguistic ladder: + +``` + Staunen ── high entropy ── raw stimulus, not yet crystallized ── SYNTAX + │ │ + (semantics sits between) (meaning) + │ │ + Wisdom ── low entropy ── crystalline knowledge / settled fact ─ PRAGMATICS +``` + +`entropy = 1 − c·|2f − 1|` over a `CausalEdge64`'s NARS `(f, c)` — one minus the +decisiveness of the NARS expectation. Validated as a **reliability proxy**: +ρ(entropy, empirical prediction accuracy) = **−0.78** over a synthetic NARS +population (the more crystalline the edge, the more reliably its belief matches +fresh reality), grounded via `ndarray::hpc::reliability`. + +## Architecture decisions (locked this session) + +1. **No re-quantization.** The SPO rungs read the **3×palette-256 indices already + inside `CausalEdge64`** (`s_idx`/`p_idx`/`o_idx`) — "exact enough" (operator). + `cam_pq` is NOT used to re-encode SPO for the ladder. +2. **Pearl 2³ mask = the 8 SPO iterations.** The existing causal mask at + `CausalEdge64` [42:40] enumerates the 8 `(S,P,O)` subsets; `decompose_spo` + zeroes inactive components and emits an HHTL-routable `basin_key`. +3. **Flavors / classes are INTERPRETATION, not layout.** No `NODE_ROW_STRIDE` + change, no `ENVELOPE_LAYOUT_VERSION` bump (canon "registry-resolved via + `classid → ClassView`"). +4. **deterministic↔residue.** Coarse = nearest-centroid palette index + (recomputed via AMX `matmul_i8_to_i32`); residue = signed-4-bit correction. +5. **I-VSA-IDENTITIES correction.** COCA "codebook superposition for 2/3 pruning" + must bundle cluster **identity fingerprints**, NOT the CAM-PQ codes + (superposing PQ codes is register-loss). Pruning needs a cluster-identity + layer that does not exist yet → design before build. +6. **Placement.** `ndarray` = hardware/math (codecs, reliability, entropy); + `lance-graph-contract` = the selector + the edge type; `causal-edge` = the + bit-storage; `lance-graph` = thinking (orchestration, basins, context). + +## The Plateau (delivered) — R1 + +| Artifact | Repo / path | Commit | +|---|---|---| +| `hpc::reliability` (Pearson, Spearman, Cronbach α, ICC(2,1), `FidelityReport`) | ndarray `src/hpc/reliability.rs` | `d3b608f` | +| `hpc::edge_codec` (Codebook k-means, `CoarseResidueCodec`, `ProductQuantizer`, `reconstruct_coarse`) | ndarray `src/hpc/edge_codec.rs` | `d3b608f` | +| `examples/edge_codec_compare` (measure all flavors × regimes) | ndarray | `d3b608f` | +| `hpc::entropy_ladder` (`nars_entropy`, `EntropyRung`, `Quadrant`, `PEARL_SUBSETS`, `decompose_spo`, `entropy_class`) | ndarray `src/hpc/entropy_ladder.rs` | `83be7c3` | +| `examples/entropy_ladder_probe` (rung/quadrant partition + SPO decomposition) | ndarray | `83be7c3` | +| `EdgeCodecFlavor` + `ClassView::edge_codec_flavor` selector | lance-graph `lance-graph-contract` | `920671d` | +| bgz17 SIMD gather OOB guard (P1) | lance-graph `crates/bgz17/src/simd.rs` | `6d48ced` | + +**Measured:** +- Edge-codec fidelity: CoarseResidue dominates agreement (ICC 0.97–0.99, ρ 0.98, + α 0.99); Pq32x4 preserves rank (ρ 0.60–0.67) but not absolute distance + (ICC 0.11–0.29); CoarseOnly collapses on continuous data (ICC 0.003). AMX + assign 100% vs scalar, 24–28 GMAC/s. +- Entropy ladder: ρ(entropy, accuracy) = −0.78; balanced rung/quadrant partitions; + crystalline SPO → H=0.107 (Pragmatics/Wisdom), ambiguous S-only → H=1.000 + (Syntax/Staunen). + +Tests: ndarray 28 new unit + 14 doctests; contract +3 (609 lib green); bgz17 +1. +All clippy `-D warnings` clean. + +## Roadmap — R2..R6 + COCA + +| Rung | What | Plugs into | Dependency / gate | +|---|---|---|---| +| **R1** | NARS f/c → validity/reliability + entropy coordinate | `nars_entropy` + `reliability` | ✅ **SHIPPED** (ρ=−0.78) | +| **R2** | Store the entropy/reliability class where it belongs | `entropy_class` → `CausalEdge64` spare bits [63:61] | `causal-edge` (zero-dep bit ops), **version-gated** + field-isolation tests (I-LEGACY-API-FEATURE-GATED) | +| **R3** | CAM-PQ AMX centroid assignment (the "scanning" win) | `matmul_i8_to_i32`; 2×2/4×4 tiled centroid grid | independent of the ladder; `cam_pq.rs:precompute_distances` → AMX variant + bit-exact/GMAC-s probe | +| **R4** | HHTL + helix basin attraction | `decompose_spo.basin_key` + `helix` residue | 🔬 **probe**: residue-Δ from basin centroid predicts re-access > HHTL-tier alone (+15% recall gate) | +| **R5** | Markov SPO rung-ladder → episodic context/basins/supporting edges | `EntropyRung` over a `CausalEdge64` stream | 🔬 **probe**: ladder prunes without recall loss; builds on R1+R2 | +| **R6** | Energy axis / particle↔wave | real `MailboxSoA.energy` → `Quadrant`'s energy input | ⏳ gated on the Mailbox-SoA map (in flight) | +| **COCA** | superposition 2/3 pruning between 2⁸ SPO | cluster-identity FP layer (NEW) | 🛑 **I-VSA-IDENTITIES**: bundle identities not PQ codes — design first | + +## Probe specs (the gates) + +- **R4 helix-basin** — synthetic/real episodic workload; for nodes in the same + HHTL basin, does a small Fisher-2z helix-residue Δ from the basin centroid + predict re-access better than HHTL-tier membership alone? Promote to FINDING + and wire only if ≥ +15% recall@k; else archive ("HHTL alone sufficient"). +- **R5 Markov rung-ladder** — build word→clause→discourse centroid-SPO levels; + measure whether checking the low-entropy (Pragmatics) rung first prunes ≥ X% + of candidates with < Y% recall loss vs flat scan. Entropy band = the prune key. + +## D-ids + +- **D-EL-1** (R1) — entropy-ladder foundation + reliability + edge-codec — **Shipped** +- **D-EL-2** (R2) — entropy class in CausalEdge64 spare bits — **Queued** (next) +- **D-EL-3** (R3) — CAM-PQ AMX assignment — **Queued** +- **D-EL-4** (R4) — helix-basin attraction — **Probe queued** +- **D-EL-5** (R5) — Markov SPO rung-ladder context — **Probe queued** +- **D-EL-6** (R6) — energy axis / particle↔wave — **Blocked** (SoA map) +- **D-EL-COCA** — superposition pruning (cluster-identity layer) — **Design** + +https://claude.ai/code/session_01D2WSmezQBNC3bUdHuGfGmo diff --git a/crates/bgz17/src/simd.rs b/crates/bgz17/src/simd.rs index fd735a27..ab996746 100644 --- a/crates/bgz17/src/simd.rs +++ b/crates/bgz17/src/simd.rs @@ -93,6 +93,19 @@ unsafe fn avx2_batch(dm_data: &[u16], k: usize, query: u8, candidates: &[u8], ou use core::arch::x86_64::*; let row_offset = query as usize * k; + + // The i32 gather (scale=2 on the u16 base) loads the target u16 AND the + // following u16 into each lane's high half (masked off below). For the last + // valid index of `dm_data` that high half is out of bounds, so when the + // row's worst-case over-read (`row_offset + k`, candidate k-1's high half) + // is not strictly inside the backing slice, route the whole batch through + // the scalar path — which only ever touches `row_offset + c`. This affects + // at most the final row of a tight k×k matrix (≤1/k of queries). + if row_offset + k >= dm_data.len() { + scalar_batch(dm_data, k, query, candidates, out); + return; + } + let row_ptr = dm_data.as_ptr().add(row_offset); let n = candidates.len(); @@ -160,6 +173,19 @@ unsafe fn avx512_batch(dm_data: &[u16], k: usize, query: u8, candidates: &[u8], use core::arch::x86_64::*; let row_offset = query as usize * k; + + // The i32 gather (scale=2 on the u16 base) loads the target u16 AND the + // following u16 into each lane's high half (masked off below). For the last + // valid index of `dm_data` that high half is out of bounds, so when the + // row's worst-case over-read (`row_offset + k`, candidate k-1's high half) + // is not strictly inside the backing slice, route the whole batch through + // the scalar path — which only ever touches `row_offset + c`. This affects + // at most the final row of a tight k×k matrix (≤1/k of queries). + if row_offset + k >= dm_data.len() { + scalar_batch(dm_data, k, query, candidates, out); + return; + } + let row_ptr = dm_data.as_ptr().add(row_offset); let n = candidates.len(); @@ -366,6 +392,39 @@ mod tests { assert_eq!(out[0], 0); } + #[test] + fn test_batch_last_row_full_width() { + // Regression: the SIMD gather (i32 gather, scale=2 over the u16 matrix) + // over-reads the u16 following each lane's target. On the very last row + // (query == k-1) with the last candidate (k-1), that over-read lands one + // u16 past `dm.data` — a memory-safety bug. k=64 makes 64 candidates an + // exact multiple of both the AVX-512 (16) and AVX2 (8) widths, so the + // last candidate is processed by the gather path, not the scalar tail. + // The boundary guard must route this row to scalar and still be correct. + let pal = make_palette(64); + let dm = DistanceMatrix::build(&pal); + assert_eq!( + dm.data.len(), + dm.k * dm.k, + "tight k×k matrix (no trailing slack)" + ); + + let query = (dm.k - 1) as u8; // last row + let candidates: Vec = (0..dm.k as u8).collect(); // includes k-1 + let mut batch_out = vec![0u16; dm.k]; + + batch_palette_distance(&dm.data, dm.k, query, &candidates, &mut batch_out); + + for (i, &cand) in candidates.iter().enumerate() { + let expected = dm.distance(query, cand); + assert_eq!( + batch_out[i], expected, + "last-row mismatch at candidate {}: batch={} scalar={}", + cand, batch_out[i], expected + ); + } + } + #[test] fn test_detect_simd() { let level = detect_simd(); diff --git a/crates/lance-graph-contract/src/canonical_node.rs b/crates/lance-graph-contract/src/canonical_node.rs index 359ca433..ad2911c7 100644 --- a/crates/lance-graph-contract/src/canonical_node.rs +++ b/crates/lance-graph-contract/src/canonical_node.rs @@ -45,29 +45,43 @@ impl NodeGuid { /// Panics (incl. const-eval) when `family` or `identity` exceed 24 bits — the /// silent-truncation footgun: distinct u32 inputs would otherwise collapse /// to the same stored key. - pub const fn new(classid: u32, heel: u16, hip: u16, twig: u16, family: u32, identity: u32) -> Self { + pub const fn new( + classid: u32, + heel: u16, + hip: u16, + twig: u16, + family: u32, + identity: u32, + ) -> Self { assert!(family <= 0x00FF_FFFF, "family must fit in 24 bits"); assert!(identity <= 0x00FF_FFFF, "identity must fit in 24 bits"); let c = classid.to_le_bytes(); let h = heel.to_le_bytes(); let p = hip.to_le_bytes(); let t = twig.to_le_bytes(); - let f = family.to_le_bytes(); // low 3 bytes + let f = family.to_le_bytes(); // low 3 bytes let i = identity.to_le_bytes(); // low 3 bytes Self([ c[0], c[1], c[2], c[3], // 0..4 classid - h[0], h[1], // 4..6 HEEL - p[0], p[1], // 6..8 HIP - t[0], t[1], // 8..10 TWIG - f[0], f[1], f[2], // 10..13 family - i[0], i[1], i[2], // 13..16 identity + h[0], h[1], // 4..6 HEEL + p[0], p[1], // 6..8 HIP + t[0], t[1], // 8..10 TWIG + f[0], f[1], f[2], // 10..13 family + i[0], i[1], i[2], // 13..16 identity ]) } /// Default-class, default-basin node: only `identity` discriminates. /// This is the bootstrap address while classid and family are zero. pub const fn local(identity: u32) -> Self { - Self::new(Self::CLASSID_DEFAULT, 0, 0, 0, Self::FAMILY_DEFAULT, identity) + Self::new( + Self::CLASSID_DEFAULT, + 0, + 0, + 0, + Self::FAMILY_DEFAULT, + identity, + ) } #[inline] @@ -171,6 +185,60 @@ pub struct EdgeBlock { pub out_family: [u8; 4], } +/// Which edge-codec flavor a class uses to *read* its node's edge block. +/// +/// The flavor is an INTERPRETATION of the canonical 16-byte [`EdgeBlock`] (plus +/// an optional value-slab residue), selected per class via +/// [`ClassView::edge_codec_flavor`](crate::class_view::ClassView::edge_codec_flavor) +/// — never a change to [`NodeRow`]'s 512-byte layout. Every variant leaves +/// [`NODE_ROW_STRIDE`] untouched (the canon "registry-resolved via +/// `classid → ClassView`" rule), so adopting a flavor needs NO +/// `ENVELOPE_LAYOUT_VERSION` bump. +/// +/// Encode/reconstruct kernels live in `ndarray::hpc::edge_codec`; per-flavor +/// fidelity is measured by `ndarray::hpc::reliability` (see the +/// `edge_codec_compare` example — CoarseResidue dominates on agreement, Pq32x4 +/// preserves rank but not absolute distance). Default is [`CoarseOnly`], the +/// zero-fallback reading that matches the canon all-zero bootstrap default. +/// +/// [`CoarseOnly`]: EdgeCodecFlavor::CoarseOnly +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)] +#[repr(u8)] +pub enum EdgeCodecFlavor { + /// 1 byte/vector: each edge byte is a palette/centroid index — the + /// [`EdgeBlock`] read literally. The canon zero-fallback default. + #[default] + CoarseOnly = 0, + /// 1 + ⌈D/2⌉ bytes: coarse index + a per-dimension signed-4-bit residue + /// carried in the reserved value slab. Highest fidelity / agreement. + CoarseResidue = 1, + /// 16 bytes: the edge block read as 32 × 4-bit product-quantizer codes (the + /// turbovec PQ model). Preserves neighbour *rank* better than absolute + /// distance (low ICC, decent Spearman). + Pq32x4 = 2, +} + +impl EdgeCodecFlavor { + /// Per-vector byte cost for dimensionality `dim` (D even for the residue's + /// nibble packing; `⌈D/2⌉` is used so odd D rounds up). + #[inline] + pub const fn bytes_per_vector(self, dim: usize) -> usize { + match self { + EdgeCodecFlavor::CoarseOnly => 1, + EdgeCodecFlavor::CoarseResidue => 1 + dim.div_ceil(2), + EdgeCodecFlavor::Pq32x4 => 16, + } + } + + /// Every flavor re-interprets the SAME 512-byte node row — none changes + /// [`NODE_ROW_STRIDE`], so no flavor requires a layout-version bump. This is + /// the canon invariant, encoded so a regression test can assert it. + #[inline] + pub const fn is_layout_preserving(self) -> bool { + true + } +} + /// One node = 4096 bit = 512 byte: key(16) | edges(16) | value(480). /// /// The 480-byte value is deferred — energy/meta/qualia/entity_type, materialized @@ -353,6 +421,37 @@ mod tests { assert_eq!(core::mem::size_of_val(&e), 16); } + #[test] + fn edge_codec_flavor_default_is_coarse_only() { + // Zero-fallback default: the all-zero reading is the canon bootstrap. + assert_eq!(EdgeCodecFlavor::default(), EdgeCodecFlavor::CoarseOnly); + assert_eq!(EdgeCodecFlavor::CoarseOnly as u8, 0); + } + + #[test] + fn edge_codec_flavor_byte_costs() { + // D = 128: coarse 1 B, residue 1 + 64 = 65 B, PQ fixed 16 B. + assert_eq!(EdgeCodecFlavor::CoarseOnly.bytes_per_vector(128), 1); + assert_eq!(EdgeCodecFlavor::CoarseResidue.bytes_per_vector(128), 65); + assert_eq!(EdgeCodecFlavor::Pq32x4.bytes_per_vector(128), 16); + // Odd D rounds the residue nibble count up. + assert_eq!(EdgeCodecFlavor::CoarseResidue.bytes_per_vector(7), 1 + 4); + } + + #[test] + fn every_flavor_preserves_node_layout() { + // The canon invariant: a flavor is an interpretation, never a stride + // change — so no flavor forces an ENVELOPE_LAYOUT_VERSION bump. + for f in [ + EdgeCodecFlavor::CoarseOnly, + EdgeCodecFlavor::CoarseResidue, + EdgeCodecFlavor::Pq32x4, + ] { + assert!(f.is_layout_preserving()); + } + assert_eq!(NODE_ROW_STRIDE, core::mem::size_of::()); + } + #[test] fn uniqueness_guard_is_noop_outside_bootstrap() { // family != 0 ⇒ no longer the bootstrap address: the guard is a no-op @@ -414,10 +513,7 @@ mod tests { #[test] fn node_row_column_table_sums_to_row_stride() { - let total: usize = NODE_ROW_COLUMNS - .iter() - .map(|c| c.col_bytes_per_row()) - .sum(); + let total: usize = NODE_ROW_COLUMNS.iter().map(|c| c.col_bytes_per_row()).sum(); assert_eq!(total, NODE_ROW_STRIDE); assert_eq!(NODE_ROW_STRIDE, core::mem::size_of::()); } diff --git a/crates/lance-graph-contract/src/class_view.rs b/crates/lance-graph-contract/src/class_view.rs index 4fc8b846..635f4701 100644 --- a/crates/lance-graph-contract/src/class_view.rs +++ b/crates/lance-graph-contract/src/class_view.rs @@ -204,6 +204,20 @@ pub trait ClassView { }) .collect() } + + /// Which edge-codec flavor this class reads its node edge block with. + /// + /// Default is + /// [`EdgeCodecFlavor::CoarseOnly`](crate::canonical_node::EdgeCodecFlavor::CoarseOnly) + /// — the canon zero-fallback reading (each edge byte is a palette index). An + /// implementor overrides this to let a class opt into residue or PQ fidelity. + /// This is *selection only*: every flavor shares the SAME byte layout, so the + /// choice never changes `NODE_ROW_STRIDE` (canon "registry-resolved via + /// `classid → ClassView`", never a stride change). + #[inline] + fn edge_codec_flavor(&self, _class: ClassId) -> crate::canonical_node::EdgeCodecFlavor { + crate::canonical_node::EdgeCodecFlavor::CoarseOnly + } } /// One populated field to render — the late-resolved `label` + its `predicate` key. diff --git a/crates/lance-graph-contract/src/lib.rs b/crates/lance-graph-contract/src/lib.rs index 8f333ed0..ddfab9d2 100644 --- a/crates/lance-graph-contract/src/lib.rs +++ b/crates/lance-graph-contract/src/lib.rs @@ -106,7 +106,7 @@ pub mod world_map; pub mod world_model; // Re-exports for the most commonly used collapse_gate types. -pub use canonical_node::{EdgeBlock, NodeGuid, NodeRow}; +pub use canonical_node::{EdgeBlock, EdgeCodecFlavor, NodeGuid, NodeRow}; pub use class_view::{ClassId, ClassProjection, ClassView, FieldMask, RenderRow}; pub use collapse_gate::{GateDecision, MailboxId, MergeMode}; pub use episodic_edges::{EdgeRef, EpisodicEdges64};