diff --git a/.claude/board/AGENT_LOG.md b/.claude/board/AGENT_LOG.md index 98638cdf..a0150f69 100644 --- a/.claude/board/AGENT_LOG.md +++ b/.claude/board/AGENT_LOG.md @@ -1,3 +1,75 @@ +## [Main thread / Opus] PR #436 review response — 5 CodeRabbit findings (public-API hardening) + +**Branch:** claude/jolly-cori-clnf9 | **Files:** `crates/lance-graph-arm-discovery/src/{aerial/codebook.rs, aerial/extract.rs, translator.rs, encode.rs}` + `.claude/knowledge/aerial-arm-ruff-spo-codegen-synergies.md`. + +**Cargo:** **37/37** (33 + 4 regression tests) + clippy `-D warnings` clean on BOTH default and `--features ndarray-simd`. + +**Outcome:** DONE. PR #436 had 11 CodeRabbit review comments across 2 rounds. Verified each against CURRENT code (post de-float + SIMD); 5 still valid, 6 outdated/resolved. **Fixed (all quick-win public-API guards + a `#[should_panic]`/behavior test each):** (1) `MatrixDistance::code` validates `(feature,category)` bounds so an invalid item fails fast instead of aliasing another feature's block; (2) `extract_rules` honors `max_antecedent==0` (returns no rules) instead of forcing singletons; (3) `arm_to_truth_u8` asserts `k>0` (k=0 made any cooccur>0 dogmatically `confidence=1.0`); (4) `Dataset::new` rejects rows with `category ≥ cardinality` (was silently undercounted); (5) added `text` language tag to a fenced block (MD040). **Skipped (with reason):** autoencoder `forward/mean_loss/train` guards + `mod.rs` hidden_dim invariant + Cargo.toml bgz-tensor comment → all target code DELETED/rewritten by the de-float (8681cdf); STATUS_BOARD "Shipped (branch)" taxonomy nit → established convention used by every D-ARM row (changing one row would be inconsistent). No PR replies posted (frugal); fixes self-document via the commit. PR #436 updated. + +--- + +## [Main thread / Opus] Splat-codebook ↔ aerial ↔ Wikidata wiring — jc resolves both aerial seams + +**Branch:** claude/jolly-cori-clnf9 | **Files:** ADDED `.claude/knowledge/splat-codebook-aerial-wikidata-compression.md`; EPIPHANIES (E-ARM-JC-RESOLVES-BOTH-SEAMS); STATUS_BOARD (D-ARM-7 engine pointer, D-ARM-13 → 33/33 + SIMD clause, new D-ARM-14); `aerial/codebook.rs` oracle doc names jc + the float boundary; AGENT_LOG. + +**Outcome:** ANALYSIS + doc (no new code — the seam already exists). User direction: wire the 10000² Gaussian-splat BLASGraph top-k as the aerial oracle for OWL/DOLCE+ SPO HHTL class/basin discovery → deterministic Wikidata compression, adjacent to jc (Jirak-Cartan) EWA-sandwich splat. Grounded it against real workspace artifacts (consult-before-guess): `crates/jc` is "Jirak-Cartan" ("candar"≈Cartan), holding `ewa_sandwich{,_3d}` (Pillars 9/9b: splat Σ-push-forward for `ndarray::hpc::splat3d`), `sigma_codebook_probe` (**the ρ=0.9973 source** — 256-codebook R²≥0.99 viability), `pflug` (Pillar 10: CAM-PQ/HHTL Lε-faithful), `jirak` (Pillar 5: weak-dep Berry-Esseen). FINDING (E-ARM-JC-RESOLVES-BOTH-SEAMS): aerial's two open seams — the production `CodebookDistance` oracle AND the D-ARM-7 Jirak floor — **both resolve to jc**. jc PROVES the codebook (builds + certifies the frozen `[u32;dim²]` table offline, float OK); aerial USES it online (integer) to discover the `wikidata-hhtl-load.md` skeleton (P279/P31 DAG + basins, DOLCE as axis template). Float boundary = CAM-PQ doctrine end-to-end: build offline (float), address online (integer); the runtime stays float-free. No new aerial dependency — pass the table through the existing `MatrixDistance` seam. Confirmed user's target-cpu point (AVX-512/AMX need native/x86-64-v4) — already in the SIMD commit. PR #436 updated. + +--- + +## [Main thread / Opus] D-ARM-13 SIMD seam — bitset SoA + ndarray::simd::U64x8 (AND+popcount) + +**Branch:** claude/jolly-cori-clnf9 | **Files:** `crates/lance-graph-arm-discovery/` — ADDED `bitset.rs` (`RowMasks` row-bitset SoA), `simd.rs` (`popcount`/`and_popcount`, scalar default + `ndarray-simd` feature); rewired `aerial/extract.rs` (probe counts via `RowMasks`, not AoS rescan); `lib.rs`, `Cargo.toml` (`ndarray-simd` feature + optional ndarray path dep, `default-features=false` + `std`), `README.md`. + +**Cargo:** DEFAULT (scalar, zero-dep) → **33/33**, clippy `-D warnings` clean. `--features ndarray-simd` → **33/33**, clippy clean (ndarray builds here as a path dep with `std`; `ndarray-rand` NOT pulled). + +**Outcome:** DONE. User directive: "use ndarray crate::simd::*". The data-confirmation count loop is the `faiss-homology` "SIMD batch-AND over the SoA facet column" workload. Transposed the window into one `u64` bitset per item (`RowMasks`), so every candidate count is `AND` + popcount over `&[u64]`. Per `ndarray-vertical-simd-alien-magic.md` (MANDATORY), the primitive routes through `ndarray::simd::U64x8` (`from_slice`/`&`/`popcnt`/`to_array`) — zero raw intrinsics, zero `cfg(target_arch)` in this crate; scalar `u64::count_ones` is the default so the crate stays std-only/verifiable. **target-cpu caveat** (per user): the real AVX-512 VPOPCNTQ / AMX kernels need `-C target-cpu=native` or `x86-64-v4`; otherwise it is ndarray's correct-but-scalar polyfill. The palette256 `CodebookDistance` oracle is SIMD on the consumer side (`bgz17::batch_palette_distance` / BLASGraph splat top-k). PR #436 updated. + +--- + +## [Main thread / Opus] D-ARM-13 de-float — autoencoder → deterministic codebook-probe (palette256) + +**Branch:** claude/jolly-cori-clnf9 | **Files:** +- `crates/lance-graph-arm-discovery/` — DELETED `aerial/{autoencoder,rng}.rs`; ADDED `aerial/codebook.rs` (`CodebookDistance` trait + `MatrixDistance`); rewrote `aerial/{extract,mod}.rs` (codebook probe), `rule.rs` (integer counts + ppm gates), `translator.rs` (`TruthU8` + f32 edge), `encode.rs` (integer-only, dropped one-hot/`bin`/f32 helpers), `lib.rs`, `README.md`, `Cargo.toml` (dropped `aerial` feature) +- `.claude/board/`: EPIPHANIES (E-ARM-PROBE-IS-CODEBOOK-TOPK), STATUS_BOARD (D-ARM-13 row), AGENT_LOG + +**Cargo:** `cargo test --manifest-path …` → **28/28**; `cargo clippy … -D warnings` → clean; float audit → **zero f32 in `aerial/` discovery path**. + +**Outcome:** DONE. User directive: "neither cam_pq nor any crate uses (or should) float … all is deterministic [a,b] codebook distance, ρ=0.9973 spearman." Conceded — the v1 transcode's `f32` denoising autoencoder was a substrate regression. Replaced it with an integer **codebook-probe** backend: Aerial+'s reconstruction probe is mechanically a nearest-neighbour query, which the **palette256 distance table** answers exactly at ρ=0.9973 vs cosine. The oracle is injected via a zero-dep `CodebookDistance` trait (real impl = `bgz17::PaletteDistanceTable` / BLASGraph splat top-k / HDR-popcount, consumer-side; `MatrixDistance` in tests) so the crate stays standalone. Discovery path is now all integers (codebook distance `u32`, evidence counts `u32`, ppm gates); truth is `TruthU8` (= CausalEdge64 `confidence_u8` + i4 mantissa); the only residual f32 is the `TruthValue`/`Triple` serialization edge (those downstream contracts are themselves f32). Structural payoff: float was the only nondeterminism, so removing it makes the probe bitwise-deterministic ⇒ it joins the deterministic trunk; the nondeterminism firewall and D-ARM-9 (Python-IPC isolation) are moot; the seeded-reproducibility caveat closes. See EPIPHANIES E-ARM-PROBE-IS-CODEBOOK-TOPK. PR #436 updated. + +--- + +## [Main thread / Opus + 3 savant agents] D-ARM-13 brutal review (council) + honesty revisions + +**Branch:** claude/jolly-cori-clnf9 | **Agents:** 3 background Opus savants (brutally-honest-tester, iron-rule-savant, dto-soa-savant) — the Stage-D ratification ensemble applied to the ARM code. **Files:** +- `.claude/board/reviews/aerial-d-arm-13-{council-verdict,iron-rule-savant}.md` (council consolidation + 1 persisted review; the other 2 agents were write-denied and returned inline, captured in the consolidation) +- `crates/lance-graph-arm-discovery/src/{rule,ndjson,translator,lib}.rs` + `README.md` — honesty revisions +- `.claude/board/ISSUES.md` (ARM-JIRAK-FLOOR), `.claude/board/TECH_DEBT.md` (TD-ARM-CARRIER-FORK) + +**Cargo:** tester independently re-ran `cargo test` (35/35) + `--no-default-features` (17/17) + clippy `-D warnings` (clean); main thread re-verified 35/35 + clippy clean after the doc edits. + +**Verdict:** **LAND-with-revision** (2 LAND-with-revision + 1 HOLD-on-P1; zero P0). Code sound; all revisions are prose/honesty/tech-debt, no logic change. + +**Outcome:** DONE. Convened a 3-savant brutal review (the same family as the plan's ratification gate). Converged findings, all addressed: (1) the "loads through the *same* loader / byte-compatible" claim was split-true — `lance_graph::parse_triples` accepts `implies`, but `ruff_spo_triplet::from_ndjson` rejects it (closed vocab) until D-ARM-SYN-1 — downgraded to "shape-compatible" with the precise caveat across `ndjson`/`translator`/`lib`/`README`; (2) the `rule::passes` doc claimed a Jirak floor that doesn't exist (D-ARM-7 Queued, `jirak` appears 0×) — made honest + filed ISSUE ARM-JIRAK-FLOOR (hard prerequisite before wiring to a live SpoStore / D-ARM-5); (3) contract-homing drift — local `CandidateRule` (`n:u32`) disagrees with planned D-ARM-2 (`WindowMetadata`), "shape identical" promise was already false — corrected docs + filed TD-ARM-CARRIER-FORK with the `pub use`-when-D-ARM-2-lands path (firewall forbids depending on `lance-graph`, NOT on zero-dep `lance-graph-contract`); (4) "bit-identical weights" → intra-platform reproducibility footnote. The tester **refuted** the suspected `fmt_f32` drift (0 mismatches vs serde_json across [0,1]). Iron-rule confirmed `arm_to_nars` is a single-observation constructor that round-trips into `TruthValue::revision` (w=m) with no rival kernel, and `expectation` is byte-identical to `spo::truth` — I-SUBSTRATE-MARKOV/I-VSA-IDENTITIES/I-LEGACY-API all yield. Determinism firewall structurally intact. PR #436 updated. + +--- + +## [Main thread / Opus] Aerial+ Rust transcode (D-ARM-13) + ruff DTO/SPO/codegen synergy map + +**Branch:** claude/jolly-cori-clnf9 | **Files:** +- `crates/lance-graph-arm-discovery/` (NEW standalone crate, ~1.2K LOC + tests) — `aerial::{rng, autoencoder, extract}`, `translator`, `ndjson`, `rule`, `encode`, `lib`, `README` +- `Cargo.toml` (root) — added crate to `exclude` (standalone pattern, like bgz17/deepnsm) +- `.claude/knowledge/aerial-arm-ruff-spo-codegen-synergies.md` (NEW) — the synergy map +- `.claude/board/STATUS_BOARD.md` — D-ARM-13 + D-ARM-SYN-1/2/3 rows +- `.claude/board/EPIPHANIES.md` — predicate-vocabulary-gap finding (prepend) + +**Cargo:** `cargo test --manifest-path crates/lance-graph-arm-discovery/Cargo.toml` → **35/35 pass**; `cargo clippy … -- -D warnings` → clean. Main-thread only (no agents spawned, per session-stability rule). Big workspace NOT built (crate is excluded/zero-dep, verified independently). + +**D-ids:** D-ARM-13 (**Shipped on branch**); D-ARM-SYN-1/2/3 (**Queued**, council-gated). + +**Outcome:** DONE. Transcoded **Aerial+** (Karabulut 2025, 2504.19354v1) to zero-dep Rust — the autoencoder leg the plan §14 had explicitly deferred to Python; the user's directive ("transcode aerial rule mining to rust") supersedes that deferral. Faithful port: one-hot encoding → under-complete **denoising autoencoder** (per-feature softmax + cross-entropy, hand-written backprop, seeded SplitMix64 for reproducibility) → **Algorithm 1** reconstruction-probe rule extraction (mark antecedent, uniform elsewhere, forward, τ_a antecedent test + τ_c consequent test) → support/confidence confirmed on data → `CandidateRule`. Tests prove the AE learns a cross-feature dependency and Algorithm 1 recovers a planted rule while rejecting an independent feature. Translator `arm_to_nars` maps `(support, confidence, n) → NARS (f, c)` verbatim per paper §2/§3.3; `ndjson` emits the exact `{"s","p","o","f","c"}` line shape the SPO store loader reads. **Synergy finding:** the Aerial leg is the *runtime-data* frontend of a three-frontend/one-substrate/two-codegen bracket whose substrate (`ruff_spo_triplet::Triple`) and codegen (`ruff_python_codegen` ∥ `op_emitter.rs`) legs already exist in the ruff fork; `ruff_python_dto_check` is the *static-AST* sibling frontend. Key gap surfaced: `ruff_spo_triplet::Predicate` is a closed vocabulary with **no implication/association predicate**, so loading ARM rules through that ndjson path needs `Implies` added there first (D-ARM-SYN-1, deliberate ontology change → council-gated). Determinism boundary preserved: the nondeterministic AE stays a seeded *fan-in proposer*, out of the deterministic compile path, output gated by Stage D. PR to follow. + +--- + ## [Main thread / Opus 4.7] streaming-arm-nars-discovery-v1 — integration plan + handover + #434 corrections (the upstream proposer leg) **Branch:** claude/activate-lance-graph-att-k2pHI (rebased onto origin/main post PR #434 merge) | **Files:** diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index c74b93c4..8426d570 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,50 @@ +## 2026-05-30 — E-ARM-JC-RESOLVES-BOTH-SEAMS — aerial's two open seams (the distance oracle AND the D-ARM-7 Jirak floor) both resolve to `crates/jc`; jc PROVES the splat codebook, aerial USES it to discover the DOLCE skeleton that compresses Wikidata + +**Status:** FINDING (architecture; seams concrete, end-to-end pipeline is CONJECTURE). User framing: "gaussian-splat spatial blasgraph top-k 10000×10000 … for OWL/DOLCE+ SPO HHTL classes and basin via aerial+ to deterministically compress Wikidata … adjacent to JC Jirak[-Cartan] with EWA-sandwich gaussian splat." + +The de-float (E-ARM-PROBE-IS-CODEBOOK-TOPK) left aerial with two open seams: a production `CodebookDistance` oracle, and the unimplemented D-ARM-7 Jirak significance floor (ISSUE ARM-JIRAK-FLOOR). **Both resolve to the same crate — `crates/jc` (Jirak-Cartan)** — and the grounding is already in the workspace, not invented: + +- **Oracle ← jc.** `jc::ewa_sandwich{,_3d}` (Pillars 9/9b) certify the Gaussian-splat Σ-push-forward `J·W·Σ·Wᵀ·Jᵀ` for `ndarray::hpc::splat3d` — i.e. the 10000² BLASGraph spatial top-k that *builds* the codebook is correct. `jc::sigma_codebook_probe` is **literally where ρ=0.9973 comes from**: it measures a 256-codebook capturing the Σ-distribution at R²≥0.99 in log-Euclidean space. `jc::pflug` (Pillar 10) certifies the CAM-PQ/HHTL tree quantization is Lε-faithful (compression is faithful, not lossy-by-surprise). The frozen `[u32; dim²]` table feeds aerial's existing `MatrixDistance`/`CodebookDistance` seam — no new aerial dependency. +- **Jirak floor ← jc.** `jc::jirak` (Pillar 5) is the weak-dependence Berry-Esseen engine: classical IID is wrong here (I-NOISE-FLOOR-JIRAK), the correct rate is `n^(p/2-1)`. D-ARM-7's significance gate derives its threshold from it. + +**The float boundary is the punchline.** float lives ONLY in jc's offline build+certify (k-means, SPD/EWA math, Berry-Esseen sup-error); it runs once, emits a frozen integer artifact, and is never on aerial's online path. That is the CAM-PQ doctrine end-to-end — *build the codebook offline (float OK), address it online with integer codes* — and it makes the whole "deterministically compress Wikidata" claim float-free at runtime. Downstream: aerial discovers the OWL/DOLCE+ SPO HHTL classes + basins (the `specs/wikidata-hhtl-load.md` skeleton: P279/P31 DAG + basin assignment, DOLCE as the axis template), the Jirak floor decides which rules are significant enough to persist, and codebook-HHTL is the 16ⁿ bucket router. Full map: `.claude/knowledge/splat-codebook-aerial-wikidata-compression.md`. Cross-ref: jc pillars 5/9/9b/10, `wikidata-hhtl-load.md`, `ogit-owl-dolce-ontology-compartments.md`, `3DGS-HHTL-datalake-traversal-plan.md`, E-ARM-PROBE-IS-CODEBOOK-TOPK. + +--- + +## 2026-05-30 — E-ARM-PROBE-IS-CODEBOOK-TOPK — Aerial+'s reconstruction probe IS a codebook top-k; the autoencoder was a float approximation of a lookup the substrate already does exactly; de-floating it dissolves the determinism firewall + +**Status:** FINDING (de-float shipped on `claude/jolly-cori-clnf9`, 28/28 tests, zero f32 in the discovery path). User-directed: "neither cam_pq nor any other crate uses (or should) float … all is deterministic [a,b] codebook distance, ρ=0.9973 spearman." + +The first D-ARM-13 transcode reproduced Aerial+ literally — an `f32` denoising autoencoder. That was a substrate regression: this stack addresses by **exact codebook CAM, never float similarity** (`faiss-homology-cam-pq.md`; `I-VSA-IDENTITIES` — CAM-PQ codes are for *search*, integer, never superposed as float). The autoencoder's reconstruction probe ("mark the antecedent, read off the high-probability consequents") is, mechanically, a **nearest-neighbour query** — and the **palette256 distance table** answers that *exactly and in integers* at ρ=0.9973 vs cosine. So the float net was a slow, seed-dependent approximation of a lookup the substrate already performs. + +The de-float (codebook-probe backend): +- autoencoder (`f32` weights) → injected integer `CodebookDistance` oracle (palette256; zero-dep trait, real impl is `bgz17::PaletteDistanceTable` / BLASGraph splat top-k / HDR-popcount); +- reconstruction probe → codebook top-k from the antecedent within `θ` (integer); +- softmax ranking → integer distance ranking; support/confidence → integer counts + ppm cross-multiply gates; +- `(f,c)` `f32` → `TruthU8` (= CausalEdge64 `confidence_u8` + i4 mantissa); seeded RNG → deleted. + +**The payoff is structural, not cosmetic.** Float was the *only* source of nondeterminism, so removing it dissolves three things at once: (1) the "seeded ≠ bitwise-deterministic" caveat closes — the probe is bitwise-identical on every target; (2) Aerial is no longer a *nondeterministic fan-in* that must hide behind the ratification gate / out of the compile path — it joins the **deterministic trunk** beside pair-stats (D-ARM-3); (3) **D-ARM-9** (Python-IPC to isolate nondeterminism) is fully moot — there is no nondeterminism to isolate. The ratification council still governs *promotion to the SPO store*, but the firewall is now about ratification, not float. General rule this crystallises: **a "learned" similarity that the codebook reproduces at ρ≈1.0 should never be reached for as float — the codebook IS the learned-once model, frozen and integer.** Code: `crates/lance-graph-arm-discovery/src/aerial/{codebook,extract,mod}.rs`. Cross-ref: `faiss-homology-cam-pq.md`, `cognitive-risc-classes.md` (CAM-not-ANN), `I-VSA-IDENTITIES`, the prior float transcode's review (`.claude/board/reviews/`). + +--- + +## 2026-05-30 — E-DISCOVERY-CODEGEN-BRACKET-1 (realised) — the Aerial+ transcode is the runtime-data frontend of a bracket whose substrate + codegen legs ALREADY EXIST in the ruff fork; the ruff SPO predicate vocabulary is the only missing seam + +**Status:** FINDING (type-level, grounded in source read 2026-05-30) + CONJECTURE (the three D-ARM-SYN wiring deliverables). Author-stated; the three wirings are council-gated. + +The two-paper bracket (`streaming-arm-nars-discovery-v1.md`: Aerial+ discovery upstream, Abreu M2M codegen downstream, SPO+NARS middle) is not a future aspiration — **its substrate and both codegen legs are already implemented in `adaworldapi/ruff`** and `lance-graph-ontology`. Transcoding Aerial+ to Rust (D-ARM-13) made this concrete: + +1. **One substrate, three frontends.** `ruff_spo_triplet::Triple { s, p, o, f, c }` *"mirrors `lance_graph::graph::spo::odoo_ontology::OntologyTriple` field-for-field"* and its ndjson is *"exactly what `parse_triples` reads."* Three proposer frontends converge on it: `ruff_python_dto_check` (static Python AST → `Extracted`/Authoritative), `ruff_ruby_spo` (Rails scaffold), and **`lance-graph-arm-discovery` (Aerial+, runtime data → `ArmDiscovered`)**. The first two are bounded by the literal artifact; only the ARM leg surfaces co-correlations that exist solely in runtime rows. + +2. **Two codegen legs, one thesis.** `op_emitter.rs` (ratified SoA → deterministic Rust dispatch) and `ruff_python_codegen::round_trip` (AST → deterministic Python) are the same externalise-interpretation thesis the Abreu paper validates, in two target languages. Both sit downstream of the ratification firewall. + +3. **The truth scale is already shared.** Aerial's `arm_to_nars` produces the exact `(f, c)` that `TruthValue::new` and `ruff_spo_triplet::Provenance::truth()` carry, and `NarsTruth::expectation()` reimplements `TruthValue::expectation` so one `TruthGate` covers mined and extracted facts alike. + +4. **The one missing seam (the actionable finding).** `ruff_spo_triplet::Predicate` is a **closed vocabulary** (`rdf:type, has_function, emitted_by, depends_on, reads_field, raises, traverses_relation`) and `from_ndjson` **hard-rejects** anything else — and **none of them is an implication/association relation.** An `X → Y` ARM rule therefore cannot flow through that loader until `Implies`/`CoOccursWith` is added (a *deliberate* ontology change per that crate's own doc). This is D-ARM-SYN-1; it gates SYN-2 (the `CandidateRule → ModelGraph` adapter) and SYN-3 (the `ArmDiscovered` truth calibration below the codegen gate). + +**Determinism boundary (unchanged, reaffirmed):** Aerial+ is the only nondeterministic node in the bracket. The transcode keeps it standalone, seeded (`aerial::Rng`), behind the `aerial` feature, and emitting a `CandidateRule` *proposal* — never a committed triple. Promotion is the council's job. Full map: `.claude/knowledge/aerial-arm-ruff-spo-codegen-synergies.md`. Code: `crates/lance-graph-arm-discovery/`. Cross-ref: `E-INTERPRET-NOT-STORE-1`, `E-SOA-IS-THE-ONLY`, `I-NOISE-FLOOR-JIRAK`, Karabulut 2025 §2/§3.3, Abreu 2025 §4. + +--- + ## 2026-05-29 — E-SOA-IS-THE-ONLY — there is ONE SoA, never transformed; mailbox SoA mutation IS the hot path; Libet −550 ms anchors the Rubicon kanban in surrealkv-on-lance; SPO-W witness is a *pointer* via the belief-state arc **Status:** CONJECTURE / design (user-stated 2026-05-29, post-PR-#433). Records five layered rulings; details in `bindspace-singleton-to-mailbox-soa-v1.md` §11.1–§11.5. Author-stated; not council-gated. diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index 62572fb2..3256889e 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -72,6 +72,10 @@ gets buried. ## Open Issues +## 2026-05-30 — [ARM-JIRAK-FLOOR] Aerial+ proposer (D-ARM-13) ships without the mandatory Jirak Stage-A floor + +**Status: OPEN.** Surfaced by the 3-savant brutal review of D-ARM-13 (iron-rule-savant #1 finding, brutally-honest-tester P1). The transcoded Aerial+ proposer (`crates/lance-graph-arm-discovery`) gates rule emission only on classical `min_support`/`min_confidence` (`extract.rs` → `rule::CandidateRule::passes`). `I-NOISE-FLOOR-JIRAK` and `streaming-arm-nars-discovery-v1.md` §4 (line 395 "This is not optional") + §11.1 declare the Jirak weak-dependence significance floor **mandatory at Stage A** — but `jirak` exists nowhere in the crate and **D-ARM-7 (the Jirak module) is Queued**. Consequence: with `c = m/(m+k)` saturating as `m = support×n` grows, a thin-but-frequent spurious rule at a 200K window becomes a high-confidence candidate → "substrate calcifies on noise." **Hard prerequisite:** D-ARM-7 MUST land before this proposer is wired into D-ARM-5 (the first stage where `(f,c)` meets a live `SpoStore` + `TruthValue::revision`). Documented honestly in `rule.rs::passes` doc + the synergy doc §4. Resolve by: implement D-ARM-7 and route `extract_rules` emission through `jirak_significance_threshold` BEFORE the classical floor. + ## 2026-04-20 — [E-MEMB-1] Python↔Rust slice layouts are incompatible at the 10 kD membrane **Status:** Open diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 2666f583..a93c9740 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -472,3 +472,11 @@ PR sequence: #360 → #361 → post-#360 substrate-sweep (this PR). | PR | Merged | Title | What it added | |---|---|---|---| | **#427** | 2026-05-28 | feat(mailbox-soa): bindspace→mailbox migration wave A1-A4 (thoughtspace columns + transitional routing + WitnessTable + plan §10) | First implementation pulse of `bindspace-singleton-to-mailbox-soa-v1` (PR #418 plan). **A1** (`1df12eca`, +103): 4 thoughtspace columns on `MailboxSoA` (`edges`/`qualia`/`meta`/`entity_type`) + 8 row accessors + zero-init in `new()` + reset in `reset_row()`. **A2** (`61b641d5`, +42): transitional `mailboxes: HashMap` + `with_mailbox()` builder + `mailbox()` accessor on `ShaderDriver` — sibling-shape, additive, singleton untouched. **A3** (`ef848a34`, +187): new `WitnessTable` + `WitnessEntry{ mailbox_ref, spo_fact_ref }` primitive in `lance-graph-contract::witness_table` (zero-dep, 3 unit tests, `const fn new`, `get`/`set` bounds-checked). **A4** (`0f448730`, +36): plan §10 "2026-05-28 architectural refinements" appended — 7 ratified findings (SoA-Lance ≠ cascade; cascade is not an index space; 64k-256k mailbox envelope ~360 MB - 1.4 GB RAM-resident; W-slot = per-cohort witness table not corpus pointer; cascade granularities = CPU/cache boundaries 64/256/4096/16384; `simd_soa.rs` introspects per-SoA shape; SoA invariant spawn → commit, two egress modes external/internal) + 2 surviving OQs (OQ-MBX-8 `persisted_row` vs Lance native versioning; OQ-MBX-15′ container scoping granularity). Codex P1 follow-on `f541b280`: widen `WitnessEntry.mailbox_ref` u16 → u32 + correct `Option` size doc. **457 contract+driver tests passing**, zero new behavioural code outside the columns/builder/primitive. Singleton `Arc` NOT removed (sibling pattern); cutover in a downstream slice (D-MBX-3/4). Merge commit `84296118`. Author session — this governance row is the post-merge close-out; per-deliverable AGENT_LOG entries D-MBX-A1..A4 already prepended at branch HEAD pre-merge. | + +## 2026-05-30 — Append: new standalone crate `lance-graph-arm-discovery` (Aerial+ transcode, D-ARM-13) on branch `claude/jolly-cori-clnf9` + +(Per APPEND-ONLY rule: dated annotation augmenting the "Current Contract Inventory" snapshot above. Branch work, not yet merged — recorded so a new session does not re-derive the crate.) + +### Current Contract Inventory — new entry + +- **`crates/lance-graph-arm-discovery`** (NEW, **excluded** standalone zero-dep crate; build via `cargo test --manifest-path crates/lance-graph-arm-discovery/Cargo.toml`). The **Aerial+** Rust transcode (Karabulut 2025, 2504.19354v1) — the upstream runtime-data proposer leg of `streaming-arm-nars-discovery-v1`. Public surface: `encode::{FeatureSpec, Dataset}`, `rule::{Item, CandidateRule, Proposer}`, `translator::{arm_to_nars, NarsTruth, CandidateTriple, FeedProjector, DebugProjector, NARS_PERSONALITY_K}`, `ndjson::to_ndjson`, and (feature `aerial`, default-on) `aerial::{Rng, AerialAutoencoder, AerialParams, AerialProposer, extract_rules, ExtractParams}`. 35/35 tests, clippy `-D warnings` clean. Emits the `{"s","p","o","f","c"}` ndjson the SPO store loader reads; `(f,c)` == `TruthValue::new(f,c)` == `ruff_spo_triplet::Triple{f,c}`. Determinism boundary: nondeterministic AE is seeded + feature-gated + emits *proposals* only. Synergy map: `.claude/knowledge/aerial-arm-ruff-spo-codegen-synergies.md`. Status board: D-ARM-13 (Shipped on branch) + D-ARM-SYN-1/2/3 (Queued). **Not** in `lance-graph-contract` yet — `rule`/`translator` are the local seam until D-ARM-1/2 land the shared carriers. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index aeed9336..3954f13e 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -622,17 +622,22 @@ The missing UPSTREAM discovery leg. Today's proposers (curated L-docs + AST-extr | D-id | Title | Crate | Lines | Conf | Status | Notes | |---|---|---|---|---|---|---| | D-ARM-1 | `ProvenanceTier::{Curated,Extracted,ArmDiscovered,Ratified,Conjecture}` enum + ordering | lance-graph-contract | 50 | HIGH | **Queued** | blocks all other D-ARM-*; additive | -| D-ARM-2 | `Proposer` trait + `CandidateRule` carrier + `WindowMetadata` | lance-graph-contract | 100 | HIGH | **Queued** | blocks D-ARM-3, D-ARM-9 | +| D-ARM-2 | `Proposer` trait + `CandidateRule` carrier + `WindowMetadata` | lance-graph-contract | 100 | HIGH | **Queued** | blocks D-ARM-3, D-ARM-9. D-ARM-13 shipped **local mirrors** (`rule::{CandidateRule, Proposer, Item}`) ahead of this — see **TD-ARM-CARRIER-FORK**: re-export via `pub use` when this lands (firewall allows path-dep on zero-dep contract). Field set diverges — local carries bare `n: u32`, this plans `WindowMetadata`; reconcile (recommend `n: u32`) so the shape matches. | | D-ARM-3 | Pair-stats proposer (default trunk, deterministic, k² pair counters per window) | lance-graph-arm-discovery::proposer::pair_stats | 400 | HIGH | **Queued** | depends on D-ARM-1/2/7; blocks D-ARM-12 | -| D-ARM-4 | ARM-truth → NARS-truth translator + Odoo `FeedProjector` impl | lance-graph-arm-discovery::translator | 200 | HIGH | **Queued** | depends on D-ARM-1/2 | +| D-ARM-4 | ARM-truth → NARS-truth translator + Odoo `FeedProjector` impl | lance-graph-arm-discovery::translator | 200 | HIGH | **Partially shipped (branch)** | The translator substance landed early inside D-ARM-13: `translator::{arm_to_nars, NarsTruth, CandidateTriple, FeedProjector}` (verbatim paper §2/§3.3 mapping, 35/35 tests). REMAINING: the real Odoo `FeedProjector` (currently a `DebugProjector` stub emitting `implies`) + contract homing on D-ARM-1/2. Depends on D-ARM-1/2. | | D-ARM-5 | Hypothesis test: SpoStore round-trip, NARS revision, contradiction commit per The Click | lance-graph-arm-discovery::hypothesis | 350 | MED | **Queued** | depends on D-ARM-4; verifies `spo::truth::Contradiction` primitive exists | | D-ARM-6 | `RatificationQueue` ring buffer + corrections-to-#434 spec PR (`discovery_arc D=8`, `discovery_origin u8`) | lance-graph-arm-discovery::queue + #434 spec follow-up | 200 + spec | MED | **Queued** | depends on PR #434 D-MBX-A3 landing | -| D-ARM-7 | Jirak-2016 weak-dependence significance thresholds (mandatory Stage A floor) | lance-graph-arm-discovery::jirak | 150 | HIGH | **Queued** | blocks D-ARM-3; cites I-NOISE-FLOOR-JIRAK | +| D-ARM-7 | Jirak-2016 weak-dependence significance thresholds (mandatory Stage A floor) | lance-graph-arm-discovery::jirak | 150 | HIGH | **Queued — HARD PREREQUISITE** | blocks D-ARM-3; cites I-NOISE-FLOOR-JIRAK. **ISSUE ARM-JIRAK-FLOOR (2026-05-30, 3-savant review):** D-ARM-13 ships the Aerial proposer with NO Jirak floor (classical `min_support`/`min_confidence` only). MUST land before D-ARM-5 wires the proposer to a live `SpoStore`, else the substrate calcifies on thin-but-frequent noise (plan §11.1). **ENGINE EXISTS:** `jc::jirak` (Jirak-Cartan Pillar 5) is the weak-dependence Berry-Esseen rate (`n^(p/2-1)`); this deliverable is the *gate function* (rule → significant?) that derives its threshold from it — NOT a from-scratch Jirak impl. See E-ARM-JC-RESOLVES-BOTH-SEAMS + `splat-codebook-aerial-wikidata-compression.md`. | | D-ARM-8 | `Feed` + `FeedProjector` + window-size config + Odoo `account.move` projector example | lance-graph-arm-discovery::feed | 250 | MED | **Queued** | depends on D-ARM-2 | -| D-ARM-9 | Aerial+ IPC client (feature-gated `arm-aerial`, NDJSON over Unix socket) | lance-graph-arm-discovery::proposer::aerial_ipc | 200 | MED | **Queued** | optional; depends on D-ARM-2 | +| D-ARM-9 | Aerial+ IPC client (feature-gated `arm-aerial`, NDJSON over Unix socket) | lance-graph-arm-discovery::proposer::aerial_ipc | 200 | MED | **Superseded by D-ARM-13** | The native in-process Aerial+ transcode (D-ARM-13, branch `claude/jolly-cori-clnf9`) replaces the need for the Python IPC client. The determinism-boundary rationale the IPC was designed for (keep the nondeterministic autoencoder out of the Rust path) is now met in-process via seed (`aerial::Rng`) + `aerial` feature gate + workspace `exclude`. Keep this row ONLY if a Python-only Aerial variant is later required; otherwise close as Abandoned-by-replacement. | | D-ARM-10 | `op_emitter::bucket_corpus` ratification filter (`confidence ≥ Ratified`) + 2 tests | lance-graph-ontology::op_emitter | 30 | HIGH | **Queued** | depends on D-ARM-1 | | D-ARM-11 | `style_recipe.rs` rule 8 — ArmDiscovered backing adds `DAtom::Compute` weight 2 (provisional) | lance-graph-ontology::style_recipe | 80 | MED | **Queued** | depends on D-ARM-1 | | D-ARM-12 | End-to-end pipeline test + bench (synthetic Odoo feed → all 5 stages → council micro-batch) | lance-graph-arm-discovery::tests + benches | 400 | MED | **Queued** | depends on Waves 1-6; informs OQ-ARM-2 + OQ-ARM-7 | +| D-ARM-13 | **Aerial+ Rust transcode — deterministic codebook-probe backend** (float-free). The paper's `f32` denoising autoencoder is REPLACED by an integer `CodebookDistance` oracle (palette256 distance, ρ=0.9973 vs cosine): the reconstruction probe is a codebook top-k, not a softmax over float weights. Integer evidence counts + ppm gates + `TruthU8` (= CausalEdge64 wire). `AerialProposer` impl of `Proposer`. Count loop is a row-bitset SoA (`RowMasks`) → AND+popcount, routed through `ndarray::simd::U64x8` under the `ndarray-simd` feature. | lance-graph-arm-discovery::aerial | ~1.1K | HIGH | **Shipped (branch)** | branch `claude/jolly-cori-clnf9`; standalone zero-dep crate (excluded); **33/33** tests + clippy `-D warnings` clean on BOTH default (scalar) and `--features ndarray-simd`; **zero f32 in the discovery path** (audit), float only at the `TruthValue`/`Triple` serialization edge. Bitwise-deterministic ⇒ joins the trunk; the nondeterminism firewall + D-ARM-9 IPC rationale are moot. SIMD target-cpu caveat: real AVX-512/AMX kernels need `-C target-cpu=native`/`x86-64-v4`. v1 (autoencoder) superseded per the user's no-float directive. | +| D-ARM-14 | **Splat-codebook oracle + Wikidata skeleton discovery** — wire the certified jc splat codebook into aerial as the `CodebookDistance` oracle, discover OWL/DOLCE+ SPO HHTL classes + basins, drive the `wikidata-hhtl-load.md` deterministic compression (skeleton + basins + CAM-dedup + thin rows). | lance-graph-arm-discovery::aerial + crates/jc + wikidata loader | ~? | MED | **Queued (architecture)** | CONJECTURE pipeline; seams concrete (`CodebookDistance` ← jc `[u32;dim²]` table built by `ewa_sandwich`+`sigma_codebook_probe` ρ=0.9973+`pflug` Lε; Jirak floor ← `jc::jirak`). Float lives OFFLINE in jc only; aerial online path integer. No new aerial dep (pass the frozen table in). Gated on D-ARM-7. Full map: `splat-codebook-aerial-wikidata-compression.md`; finding E-ARM-JC-RESOLVES-BOTH-SEAMS. | +| D-ARM-SYN-1 | Add `Implies`/`CoOccursWith` to `ruff_spo_triplet::Predicate` closed vocabulary (+ `Provenance` tier) so ARM rules load through the same `parse_triples` ndjson path as the static extractor | ruff/ruff_spo_triplet | 40 | MED | **Queued** | council-gated (deliberate ontology change); blocks SYN-2; see `.claude/knowledge/aerial-arm-ruff-spo-codegen-synergies.md` §1 | +| D-ARM-SYN-2 | `CandidateRule → ruff_spo_triplet::ModelGraph` adapter so the Aerial runtime-data leg joins the `ruff_python_dto_check` static-AST leg in one graph before `expand()` | lance-graph-arm-discovery + ruff_spo_triplet | 120 | MED | **Queued** | depends on SYN-1; synergy doc §2 | +| D-ARM-SYN-3 | Calibrate `ProvenanceTier::ArmDiscovered` `(f,c)` below the `op_emitter` ratification gate + below static `Inferred (0.85,0.75)` so un-ratified ARM truth is council-visible but codegen-filtered | lance-graph-contract + lance-graph-ontology::op_emitter | 30 | MED | **Queued** | depends on D-ARM-1 + SYN-1; synergy doc §3/§4 | --- diff --git a/.claude/board/TECH_DEBT.md b/.claude/board/TECH_DEBT.md index a9d985ac..bbe29c5c 100644 --- a/.claude/board/TECH_DEBT.md +++ b/.claude/board/TECH_DEBT.md @@ -13,6 +13,30 @@ --- +### TD-ARM-CARRIER-FORK (D-ARM-13 / streaming-arm-nars-discovery-v1) + +**Status: Open.** Surfaced by the 3-savant brutal review of D-ARM-13 +(dto-soa-savant Finding 3, the load-bearing one). `crates/lance-graph-arm-discovery` +defines **local** `rule::{CandidateRule, Proposer, Item}` + `translator::NarsTruth` +because the planned contract homes — `lance-graph-contract::{CandidateRule, +Proposer, ProvenanceTier}` (D-ARM-1/D-ARM-2) — are still **Queued** and do not +exist yet (verified: empty grep in the contract crate). Two debts: (a) the local +`CandidateRule` carries a bare `n: u32` while D-ARM-2 specs `WindowMetadata` — +the field sets already disagree, so the "re-export, shape identical" promise is +not yet true (doc now states this honestly, `rule.rs` module + struct docs); +(b) `rule`/`translator` are re-exported at `lib.rs` top level, so if D-ARM-2 +stays Queued, downstream may import the excluded crate's types and make them the +de-facto canonical — the ThinkingStyle "contract exists to unify, nobody depends +on it" CRITICAL pattern (`TYPE_DUPLICATION_MAP.md`). **Pay by:** when D-ARM-1/2 +land, `path`-dep the zero-dep `lance-graph-contract` (the determinism firewall +forbids depending on `lance-graph`, NOT on the contract crate) and convert the +locals to `pub use` re-exports; freeze the D-ARM-2 field set (recommend bare +`n: u32`, not `WindowMetadata`) so the shape genuinely matches. Secondary P2 +test debt (brutally-honest-tester): the `max_antecedent ≥ 2` recovery path is +enumerated but untested; reproducibility tests assert bit-exact f32 (intra-target +only). Both non-blocking; add coverage when D-ARM-3 (pair-stats) lands the +multi-antecedent path for real. + ### TD-RESONANCEDTO-DUP-1 (bindspace-singleton-to-mailbox-soa-v1) - **Severity:** P3 (name collision; two distinct `ResonanceDto` structs under the same name) diff --git a/.claude/board/reviews/aerial-d-arm-13-council-verdict.md b/.claude/board/reviews/aerial-d-arm-13-council-verdict.md new file mode 100644 index 00000000..86bdfad0 --- /dev/null +++ b/.claude/board/reviews/aerial-d-arm-13-council-verdict.md @@ -0,0 +1,42 @@ +# Council verdict — D-ARM-13 (Aerial+ Rust transcode) + +> 3-savant brutal review convened 2026-05-30 on branch `claude/jolly-cori-clnf9` +> (the Stage-D ratification ensemble applied to the ARM *code*). Background +> Opus agents; two were write-denied at the tool level and returned their +> reviews inline (captured below); iron-rule-savant persisted its own file +> (`aerial-d-arm-13-iron-rule-savant.md`). + +## Consolidated verdict: **LAND-with-revision** + +| Savant | Verdict | Headline | +| --- | --- | --- | +| brutally-honest-tester | **HOLD pending fixes** (0 P0, 2 P1) | "loads through the same loader" is split-true; Jirak floor missing | +| iron-rule-savant | **LAND-with-revision** | code yields (doesn't violate) every iron rule; the Jirak *doc-comment* writes a cheque the code can't cash | +| dto-soa-savant | **LAND-with-revision** | no parallel store / no fifth column; contract-homing drift is the real risk | + +**No P0. Code is sound** — independently re-verified by the tester: `cargo test` +35/35, `--no-default-features` 17/17, clippy `-D warnings` clean, `#![forbid(unsafe_code)]`, +all 9 modules registered, `exclude` standalone done right (AP3/AP4/AP6/AP7 clean). +Every revision below is **prose / honesty / tech-debt**, not logic. + +## Converged action ledger (all applied this commit unless noted) + +| # | Finding (who) | Severity | Fix | Done | +| --- | --- | --- | --- | --- | +| 1 | "loads through the *same* loader / byte-compatible" overstates — `lance_graph::parse_triples` **accepts** `implies`, but `ruff_spo_triplet::from_ndjson` **rejects** it (closed vocab) until D-ARM-SYN-1 (tester P1, dto-soa F2, iron-rule) | P1 | Downgrade to "shape-compatible" + precise loader caveat in `ndjson.rs`, `translator.rs`, `lib.rs`, `README.md` | ✅ | +| 2 | Jirak floor doc-comment claims an upstream gate that doesn't exist; D-ARM-7 Queued (iron-rule #1, tester P1) | P1 | Make `rule::passes` doc honest ("not implemented; MUST NOT wire to live SpoStore until D-ARM-7"); file **ISSUE ARM-JIRAK-FLOOR** | ✅ | +| 3 | Contract-homing drift — local `CandidateRule` disagrees with D-ARM-2 (`n:u32` vs `WindowMetadata`); "shape identical" promise already false (dto-soa F3) | P1 | Honest `rule.rs` module+struct docs; file **TD-ARM-CARRIER-FORK** w/ the `pub use`-when-D-ARM-2-lands path | ✅ | +| 4 | "bit-identical weights" overclaims portability (iron-rule secondary, tester P2) | P2 | Footnote: intra-platform reproducibility, not bitwise-portable determinism (`lib.rs`, `README`) | ✅ | +| 5 | `max_antecedent ≥ 2` recovery untested; reproducibility asserts bit-exact f32 (tester P2) | P2 | Logged in TD-ARM-CARRIER-FORK; add coverage when D-ARM-3 lands the real multi-antecedent path | ⏶ logged | +| — | **Refuted:** `fmt_f32` drift vs `serde_json` — tester ran the full [0,1] grid: **0 mismatches** (Rust `{}` already shortest-round-trip; `1`→`1.0` is exactly what it patches) | — | no action | ✅ refuted | + +## Cross-agent agreements (high confidence) + +- **D-ARM-SYN-1 is the one genuine blocker** for the "same ruff loader" story: `ruff_spo_triplet::Predicate` is closed and `from_str`/`from_ndjson` hard-reject `implies`. All three flagged it; correctly council-gated (not silently patched). +- **D-ARM-7 (Jirak) is a doctrinal — not compile — dependency.** Enforce D-ARM-7 *before* any D-ARM-5 that consumes this proposer (D-ARM-5 = where `(f,c)` first meets live `revision` + `SpoStore`). +- **The determinism firewall is structurally intact** (excluded, std-only, seeded, proposal-only output). `arm_to_nars`'s `c = m/(m+k)` round-trips into the canonical `TruthValue::revision` as evidence `w = m` exactly; `expectation` is byte-identical to `spo::truth::TruthValue::expectation` — no rival revision kernel (iron-rule, I-SUBSTRATE-MARKOV check). +- **`NarsTruth` is a transport handle, not a parallel truth store** — the proof is it carries *no* revision algebra (dto-soa F1). + +## Adjacency (council consensus) + +D-ARM-13 = the **D-ARM-9 Aerial leg pulled in-process** (supersedes the plan §14 Python-IPC deferral per user directive). Hardest tie: **D-ARM-7 (Jirak)**. Pre-stages **D-ARM-1/D-ARM-2** (contract carriers — re-export when they land). Partially lands **D-ARM-4** (`translator`, now ahead of its own spec). Feeds **D-ARM-5** (hypothesis test) and the **D-ARM-10/11** codegen gates. Spawns **D-ARM-SYN-1/2/3**. Realises **E-DISCOVERY-CODEGEN-BRACKET-1** and **E-INTERPRET-NOT-STORE-1** at the shape level — qualified: truth scale genuinely unified, triple shape unified-by-convention, contract home not yet unified. diff --git a/.claude/board/reviews/aerial-d-arm-13-iron-rule-savant.md b/.claude/board/reviews/aerial-d-arm-13-iron-rule-savant.md new file mode 100644 index 00000000..20bec684 --- /dev/null +++ b/.claude/board/reviews/aerial-d-arm-13-iron-rule-savant.md @@ -0,0 +1,121 @@ +# IRON_RULE_SAVANT review — D-ARM-13 (Aerial+ Rust transcode) + +> **Subject:** `crates/lance-graph-arm-discovery/` on branch `claude/jolly-cori-clnf9` +> **Council question:** does this candidate violate an iron rule or doctrinal invariant — especially **I-NOISE-FLOOR-JIRAK** by claiming significance the n-bound doesn't support? +> **Reviewer lens:** substrate-veto angle. Any VIOLATES is an automatic REJECT for the council. +> **Reviewed:** 2026-05-30. Read-only doctrine pass; no cargo, no edits. + +--- + +## Verdict: **LAND-with-revision** + +No iron rule is *violated* by the code as written. The crate is honest about what it is (an upstream, seeded, feature-gated *proposer*) and it does **not** make any statistical-significance claim in code — so I-NOISE-FLOOR-JIRAK has nothing to bite on yet. But the crate ships a **doc-comment promissory note** (`rule.rs:70-71`) that asserts a Jirak floor exists "upstream of this one," and **that gate does not exist anywhere in the crate or its callers**. That is not a violation today; it is a *primed* violation — the moment this proposer is wired to a live `SpoStore` (D-ARM-5) without D-ARM-7 landing first, plan §11.1's named failure (substrate calcifies on noise) fires. The revision required before LAND is narrow and is spelled out in Finding 1. + +This is **not a HOLD**, because: +- the crate is `exclude`d from the workspace (root `Cargo.toml:46`, under `exclude`, confirmed absent from `members`), +- it is std-only / zero-dep (no external `use` beyond `crate::`/`super::`/`std::`), +- its output type is `CandidateRule` / `CandidateTriple` — a *proposal*, never a committed triple, with no `SpoStore` write path in this crate, + +so nothing it produces can reach the deterministic compile path or the SPO store *from within this PR*. The firewall is structurally intact at the crate boundary. The HOLD would only be warranted if this crate itself performed the Stage-C revision; it does not (Stage C lives in `lance-graph`, D-ARM-5, still Queued). + +--- + +## Per-iron-rule findings + +### I-NOISE-FLOOR-JIRAK — **YIELDS (with a mandatory pre-wire condition)** — the big one + +**What the rule demands:** any noise-floor or statistical-significance claim on this substrate must cite Jirak 2016 weak-dependence rates, never classical IID Berry-Esseen (`CLAUDE.md:311-343`). The plan elevates this to MANDATORY at Stage A: §4 "This is not optional," §2.1 termination criterion `n × support ≥ JIRAK_MIN_EVIDENCE`, §11.1 risk, §15 invariant table. + +**What the code actually does — the gates, concretely:** + +- `rule.rs:73-75` `CandidateRule::passes(min_support, min_confidence)` gates ONLY on the two classical ARM floors: `self.support >= min_support && self.confidence >= min_confidence`. No Jirak term. +- `extract.rs:172` is the single emission gate in the Aerial proposer: `if rule.passes(params.min_support, params.min_confidence) { out.push(rule); }`. Classical-only. +- `extract.rs:78` the apriori pre-prune also uses `params.min_support` only. +- `ExtractParams` (`extract.rs:46-56`) carries `min_support: 0.01`, `min_confidence: 0.5` — the plan §2.1 classical defaults — and **no** `jirak_*` field. +- `aerial/mod.rs:103-115` `AerialProposer::mine()` / `next_batch()` returns `extract_rules(...)` directly. No post-filter. +- **Grep confirms:** the token `jirak` appears **zero times** in `crates/lance-graph-arm-discovery/src/`. There is no `jirak` module, no `JIRAK_MIN_EVIDENCE` const, no significance-deviation test against the independence null. + +**Is this a VIOLATES?** No — and the distinction is load-bearing for the council. I-NOISE-FLOOR-JIRAK is triggered by *making a significance claim* ("N σ above noise floor," a calibrated threshold presented as principled). **This crate makes no such claim.** `support`/`confidence` are honest descriptive statistics measured on the window (`encode.rs:165-184`), and `passes()` is explicitly documented as the *classical* gate. The doc-comment even names the rule and says the Jirak floor is separate. A descriptive statistic with no significance assertion does not invoke Berry-Esseen, so there is nothing to mis-cite. **YIELDS.** + +**BUT — the doc-comment writes a cheque the codebase can't cash.** `rule.rs:68-71` states verbatim: + +> "The Jirak-bound significance floor (`I-NOISE-FLOOR-JIRAK`) is a *separate*, stricter gate applied **upstream of this one** — see the module-level docs of the proposer." + +Following that pointer: the proposer module docs (`aerial/mod.rs:17-24`) describe the determinism rationale and say output "is gated by the downstream ratification council" — they do **not** describe, implement, or reference any Jirak upstream gate. The synergy doc (`aerial-arm-ruff-spo-codegen-synergies.md:171, §4`) repeats the same claim ("the `I-NOISE-FLOOR-JIRAK` floor (D-ARM-7, separate gate) keeps the thin tail from ever being emitted"). **D-ARM-7 is Queued, not Shipped** (`STATUS_BOARD.md:630`). So the "upstream gate" the comment promises is a *future deliverable that does not yet exist*. The code therefore documents a safety property it does not possess. + +This is the precise risk plan §11.1 names: "Stage A emits rules at high rates that scrape past `MIN_CONFIDENCE` but below the Jirak floor; Stage C's revise weights them in; the substrate calcifies on weak signal." With `min_confidence` defaulting to 0.5 (`extract.rs:53`) and the NARS confidence mapping `c = m/(m+k)` saturating toward 1 as `m = support×n` grows (`translator.rs:57-58`), a thin-but-frequent spurious correlation at a 200K window produces a *high-c* candidate — exactly the "weak signal the revise cannot down-weight fast enough" case. The Jirak floor is the only thing in the design that stops it, and it is absent. + +**Required revision before LAND (any one of these closes it):** +1. **Preferred:** weaken the `rule.rs:70-71` doc-comment from a present-tense assertion ("is a separate gate applied upstream") to a forward reference ("**will be** gated upstream by D-ARM-7 `jirak`; until that lands, `passes()` is the *only* gate and this proposer MUST NOT be wired to a live `SpoStore`"). Make the absence honest. Cheapest; no code. +2. Add the `jirak` floor field to `ExtractParams` now (even if the default is a permissive `0.0` placeholder that cites D-ARM-7 as the source of the real bound), so the gate *site* exists in `extract.rs:172` and D-ARM-7 only has to supply the number. Removes the "primed violation" entirely. +3. Add a crate-level `// DOCTRINE:` note + a `#[test]` canary that asserts the proposer is not exported for SPO ingestion until a Jirak gate is present (a compile-time tripwire mirroring plan §11.1's canary). + +**Council bottom line on the big one:** the *finding the council asked me to test* — "does this claim significance the n-bound doesn't support?" — **the code does not; the doc-comment claims a floor that doesn't exist.** Land only after the comment is made honest (or the gate site is stubbed). Do **not** let D-ARM-5 (the live-`SpoStore` round-trip) reference this proposer until D-ARM-7 lands. The exclusion + proposal-only output is what downgrades this from HOLD to LAND-with-revision. + +### I-SUBSTRATE-MARKOV — **YIELDS** (verified against the canonical revision) + +**What I checked:** does the truth mapping silently diverge from `lance_graph::graph::spo::truth::TruthValue::revision` (`truth.rs:57-72`) and thereby break the Chapman-Kolmogorov/NARS revision invariant? + +**The two functions are different operations and must not be confused:** +- The crate's `arm_to_nars` (`translator.rs:53-63`) is a **single-observation constructor**: it turns one ARM rule's raw stats into an *initial* `(f, c)`. `f = confidence` (P(Y|X)); `c = m/(m+k)`, `m = support×n`, `k=1.0`. +- The canonical `TruthValue::revision` (`truth.rs:58-72`) is a **two-belief combinator**: `w_i = c_i/(1-c_i)`, `w = w1+w2`, `f = (w1·f1+w2·f2)/w`, `c = w/(w+k)`, `k=1.0`. + +**The crate does NOT reimplement revision** — and that is correct. It stops at producing the `(f,c)` that a *downstream* Stage C (D-ARM-5, in `lance-graph`, not this crate) feeds into the canonical `revision`. So there is no parallel/divergent revision kernel here to violate the semigroup property. `translator.rs:27-30` is explicit about this intent ("Deliberately *not* a re-implementation of the SPO store's `TruthValue`... honours `E-SOA-IS-THE-ONLY`"). + +**Algebraic consistency check (the subtle part):** for the downstream `revision` to be coherent, the crate's `c` must be the inverse of revision's confidence→evidence map. Revision recovers evidence as `w = c/(1-c)`. The crate sets `c = m/(m+k)` ⇒ `c/(1-c) = m/k = m` (at `k=1`). So a candidate carrying `c` round-trips into `revision` as evidence weight `w = m = support×n` — i.e. the crate's "evidential mass" IS exactly the NARS evidence count the revision arithmetic expects. **They agree by construction.** No divergence. + +**The one place the crate touches the SPO truth algebra directly** is `NarsTruth::expectation` (`translator.rs:43-45`): `c·(f−0.5)+0.5`. This is **byte-identical** to `TruthValue::expectation` (`truth.rs:48-50`) and is asserted by `translator.rs:200-207`. Same `TruthGate` will gate an ARM-mined rule and a structurally-extracted triple. Correct. + +Bundle math (vsa_bind / vsa_bundle / d=16384 / concentration-of-measure) is entirely untouched — this crate never enters that path. **YIELDS.** + +### I-VSA-IDENTITIES — **YIELDS** (clean separation; no register touched) + +ARM operates strictly on identity-typed atoms and never superposes content: +- `Item { feature: u32, category: u32 }` (`rule.rs:17-23`) is a pure identity pair — small ints, `Eq + Hash + Ord`, the Test-0 "natural ID/enum" register the rule blesses. +- `CandidateTriple { s, p, o, f, c }` (`translator.rs:86-98`) is `(s,p,o)` identity IRIs + scalar truth — never a fingerprint. +- The autoencoder's `Vec` one-hot input (`encode.rs:90-98`) and decoder probabilities (`autoencoder.rs:96-124`) are **a learned compressor's internal activations**, not a VSA carrier. They are never `vsa_bundle`d, never written to a `BindSpace` column, never persisted. They live and die inside one `extract_rules` call. +- **No `Vsa16kF32` / `Binary16K` / CAM-PQ / palette-codebook / quantized register appears anywhere in the crate** (no such import; std-only). `arm_to_nars` and the ndjson emitter touch only `f32` scalars + `String` IRIs. + +The four tests are respected by virtue of the crate using the register (HashMap/enum-style `Item`) for exact-match work rather than reaching for VSA at all. There is no superposition of content here to destroy. **YIELDS.** + +### I-LEGACY-API-FEATURE-GATED — **YIELDS (NA-leaning)** + +No v1/v2 layout reclaim is in play. The crate defines no packed bitfield accessor (`pack`/`unpack`/`with_*`/`set_*`) over a versioned layout; `CausalEdge64` is not touched. The `aerial` feature (`Cargo.toml:34-36`, `lib.rs:63-64`) gates the *autoencoder module*, not a v1 accessor — and it is gated cleanly: `#[cfg(feature = "aerial")] pub mod aerial;` with the carriers (`rule`/`translator`/`ndjson`/`encode`) compiling feature-free, so a deployment can take the truth/triple contract without the AE. That is the *correct* application of feature-gating doctrine (isolate the heavy/nondeterministic path), not the anti-pattern (silent semantic divergence under a flag). No AP1 alias. **YIELDS.** + +--- + +## Determinism boundary — enforced or merely documented? + +**Structurally enforced at the crate boundary, but "seeded ⇒ reproducible" is NOT "deterministic" and the docs overclaim by one word.** + +What IS enforced (genuine, not vibes): +- The crate is in `exclude` (root `Cargo.toml:46`), confirmed **absent** from `members`. The nondeterministic AE cannot enter the `lance-graph` compile graph — verified, not asserted. +- Output is `CandidateRule` — a proposal. There is **no `SpoStore` write, no codegen call, no ratification-bypass** path in the crate. The only sink is `ndjson::to_ndjson` (a `String`), which a *downstream* loader must choose to ingest. Nothing nondeterministic crosses into compile output from here. +- The AE is seeded through one `Rng` stream (`aerial/mod.rs:84`, `rng.rs` SplitMix64) covering weight init, denoising mask, and epoch shuffle. `reproducible_from_seed` (`mod.rs:194-205`) and `training_is_reproducible_from_seed` (`autoencoder.rs:337-347`) assert bit-identical rules / weights for same seed+data+build. + +Where the claim is too strong — **"seeded ⇒ reproducible" ≠ "deterministic across platforms":** +- The forward/backward pass is single-threaded fixed-order f32 accumulation (`autoencoder.rs:100-117, 199-232`) — good, that removes *reduction-order* nondeterminism **within one binary**. +- But the AE uses f32 **transcendental / libm** functions whose last-ULP results are **not guaranteed identical across platforms, libm versions, or `-ffast-math`-style codegen**: `tanh` (`autoencoder.rs:106`), `exp` (`autoencoder.rs:246`), `ln` (`autoencoder.rs:263`; `rng.rs:51`), `cos` + `sqrt` (`rng.rs:51-52`). A one-ULP difference in `tanh` early in training, amplified over 500 epochs of SGD, can flip a borderline `p > τ_c` reconstruction-probe decision (`extract.rs:155`) and thus *add or drop a rule* on a different machine. So same-seed reproducibility holds **on the same platform/build**, which is all the tests check and all the audit story needs — but the lib-doc phrase "Same seed, same data, and same hyper-parameters give **bit-identical** weights and identical rules" (`lib.rs:39-42`) is true only intra-platform. It should say "bit-identical *on a fixed platform/toolchain*." + +Why this is acceptable (not a HOLD): the determinism that the doctrine *requires* is determinism of the **compile path downstream of the ratification firewall** — `op_emitter` Rust codegen, `ruff_python_codegen`. This crate sits **upstream** of that firewall (synergy doc §5 table; plan §0 Stage-D firewall). A proposer is *allowed* to be nondeterministic precisely because the council + hypothesis-test re-derive and ratify before anything compiles. Platform-variance in the proposer changes *which candidates get proposed*, not *what compiles* — and the council is the gate that absorbs that. So "seeded for auditability, fenced behind exclusion + proposal-only output" is the correct posture. The only fix needed is the one-word doc honesty (`lib.rs:41` "bit-identical" → "bit-identical on a fixed platform"). + +**Determinism verdict:** boundary is real and enforced by construction; the reproducibility *claim* is mildly overstated (cross-platform f32 transcendental variance) and should be footnoted. Folds into the LAND-with-revision doc pass. + +--- + +## Adjacency / epiphany (which D-ARM-* this most depends on or blocks) + +D-ARM-13 is the **transcode of D-ARM-9's Aerial leg pulled in-process** (the plan §14 had deferred the autoencoder to Python-over-IPC; the user directive superseded that — see `AGENT_LOG` top entry + `STATUS_BOARD.md:636`). Its hardest dependency is **D-ARM-7 (Jirak)**: not a compile dependency, but a *doctrinal* one — D-ARM-13's own doc-comments (`rule.rs:70`, synergy §4) promise the Jirak floor that only D-ARM-7 supplies, and plan §11.1 makes that floor the sole defense against the calcify-on-noise failure. **D-ARM-13 should be treated as blocked-for-SPO-ingestion-purposes by D-ARM-7**, even though it ships green standalone. It also pre-stages **D-ARM-1/D-ARM-2** (the contract carriers): `rule.rs:8-9, 84` and `translator.rs` are deliberate *local mirrors* of the planned `lance-graph-contract::{CandidateRule, Proposer}` — when D-ARM-1/2 land, this crate should re-export them rather than keep the duplicates, or the AP-style "two CandidateRule shapes" drift the plan warns against (§3.2 seam) sets in. It most directly **blocks/feeds D-ARM-5** (hypothesis-test): D-ARM-5 is where this proposer's `(f,c)` actually meets `TruthValue::revision` and the live `SpoStore` — and D-ARM-5 is exactly the wave where the missing Jirak gate becomes load-bearing, so **the sequencing constraint is firm: D-ARM-7 before D-ARM-5-consuming-this-proposer.** The epiphany I see, and endorse: the transcode correctly realizes `E-DISCOVERY-CODEGEN-BRACKET-1` (Aerial = the runtime-data frontend of the three-frontend/one-substrate/two-codegen bracket; synergy doc §0) and respects `E-INTERPRET-NOT-STORE-1` (ARM rules are one interpretation projection, emitted as ndjson, never the canonical store). The genuinely useful *new* finding surfaced here is the **predicate-vocabulary gap** (D-ARM-SYN-1): `ruff_spo_triplet::Predicate` is a closed set with no `Implies`/`CoOccursWith`, so the very ndjson this crate emits (`DebugProjector` → `"implies"`, `translator.rs:131-135`) cannot yet load through the canonical `parse_triples` loader — a real, deliberate ontology decision the council must take before the bracket closes end-to-end. That gap is correctly flagged as council-gated rather than silently patched, which is itself doctrine-respecting. + +--- + +## Summary table + +| Rule | Verdict | One-line rationale | +|---|---|---| +| I-NOISE-FLOOR-JIRAK | YIELDS* | code makes no significance claim; but `rule.rs:70-71` doc asserts an upstream Jirak gate that does not exist (D-ARM-7 Queued) — make the comment honest before LAND, and never wire to live SpoStore (D-ARM-5) until D-ARM-7 lands | +| I-SUBSTRATE-MARKOV | YIELDS | `arm_to_nars` is a single-obs constructor, not a rival revision; `c=m/(m+k)` round-trips into canonical `TruthValue::revision` as `w=m` exactly; `expectation` is byte-identical; bundle math untouched | +| I-VSA-IDENTITIES | YIELDS | operates on identity `Item{feature,category}` + `(s,p,o)` IRIs; AE activations are a compressor's internals, never bundled/persisted; no Vsa/CAM-PQ/quantized register touched | +| I-LEGACY-API-FEATURE-GATED | YIELDS | no v1/v2 layout reclaim; `aerial` feature gates the module cleanly (carriers compile feature-free), not a silently-divergent v1 accessor | + +\* the `*` is the LAND-condition: one honest-doc edit (plus the `lib.rs:41` "bit-identical" → "bit-identical on a fixed platform" footnote) and a firm D-ARM-7-before-D-ARM-5 sequencing note. No code logic change required; the firewall (exclusion + proposal-only output) holds today. diff --git a/.claude/knowledge/aerial-arm-ruff-spo-codegen-synergies.md b/.claude/knowledge/aerial-arm-ruff-spo-codegen-synergies.md new file mode 100644 index 00000000..d5378ed5 --- /dev/null +++ b/.claude/knowledge/aerial-arm-ruff-spo-codegen-synergies.md @@ -0,0 +1,205 @@ +# Aerial+ ARM-discovery ↔ ruff DTO / SPO / codegen — synergy map + +> **READ BY:** integration-lead, truth-architect, dto-soa-savant, prior-art-savant, +> any agent touching `lance-graph-arm-discovery`, `ruff_spo_triplet`, +> `ruff_python_dto_check`, `ruff_python_codegen`, or `op_emitter.rs`. +> +> **Status:** FINDING (type-level, grounded in source read 2026-05-30) + +> CONJECTURE (the three proposed wiring deliverables, council-gated). +> +> **Anchors:** `streaming-arm-nars-discovery-v1.md` (the plan this implements +> the Aerial leg of), `E-DISCOVERY-CODEGEN-BRACKET-1` (candidate epiphany), +> `E-SOA-IS-THE-ONLY`, `I-NOISE-FLOOR-JIRAK`. Papers: Karabulut 2025 +> (Aerial+, 2504.19354v1), Abreu 2025 (ontology M2M, 2511.13661v1). + +--- + +## 0. One sentence + +The Aerial+ Rust transcode (`crates/lance-graph-arm-discovery`) is the +**runtime-data proposer** of a three-frontend / one-substrate / two-codegen +bracket whose substrate and codegen legs **already exist in the `ruff` fork** — +so Aerial's job is to emit into contracts the ruff crates already define, not +to invent new ones. + +```text + PROPOSER FRONTENDS SUBSTRATE CODEGEN LEGS + ┌───────────────────────┐ ┌────────────────────┐ + │ ruff_python_dto_check │ static AST ┐ │ op_emitter.rs │ Rust + │ (Python source → IR) │ │ ruff_spo_triplet │ (ratified SoA → │ dispatch + ├───────────────────────┤ ├──► Triple{s,p,o,f,c} ─┤ RECIPE_* + Ops) │ + │ ruff_ruby_spo (scaffold)│ static AST │ = NARS truth ├────────────────────┤ + ├───────────────────────┤ │ = SPO ndjson │ ruff_python_codegen │ Python + │ lance-graph-arm- │ RUNTIME ┘ loader contract │ (AST → source, │ source + │ discovery (Aerial+) │ DATA ◄── THIS WORK │ round_trip) │ + └───────────────────────┘ └────────────────────┘ + ▲ determinism firewall = ratification gate ▲ + (Aerial is nondeterministic → stays UPSTREAM of the gate) +``` + +This is the literal realisation of the two-paper bracket: Aerial+ supplies the +upstream discovery leg, the Abreu M2M paper validates the downstream codegen +leg, and `ruff_spo_triplet`'s `Triple{s,p,o,f,c}` is the SPO+NARS invariant +middle both converge on. + +--- + +## 1. Synergy A — `ruff_spo_triplet` IS the truth/triple contract Aerial emits into + +**FINDING.** `ruff_spo_triplet::triple::Triple` is `{ s, p, o, f: f32, c: f32 }` +and its doc says it *"mirrors `lance_graph::graph::spo::odoo_ontology::OntologyTriple` +field-for-field so the ndjson this crate writes loads into that store with no +transform."* `ruff_spo_triplet::ndjson::to_ndjson` emits one +`{"s","p","o","f","c"}` object per line — *"exactly what +`lance_graph::graph::spo::odoo_ontology::parse_triples` reads."* + +Our transcode targets this contract directly: + +| Aerial (this crate) | ruff_spo_triplet | lance_graph SPO | +| --- | --- | --- | +| `translator::CandidateTriple{s,p,o,f,c}` | `triple::Triple{s,p,o,f,c}` | `odoo_ontology::OntologyTriple` | +| `ndjson::to_ndjson` → `{"s","p","o","f","c"}\n` | `ndjson::to_ndjson` (same shape) | `parse_triples` (the loader) | +| `arm_to_nars` → `(f, c)` | `Provenance::truth()` → `(f, c)` | `TruthValue::new(f, c)` | + +The truth scales line up exactly — `NarsTruth::expectation()` reimplements +`TruthValue::expectation` (`c·(f−0.5)+0.5`) so an Aerial-mined rule and a +ruff-extracted structural triple are gated by the **same** `TruthGate`. Test +`aerial::tests::mined_rules_serialise_to_spo_ndjson` proves a mined rule round- +trips into that line shape end-to-end. + +**FINDING — the gap (the actionable bit).** `ruff_spo_triplet::Predicate` is a +**closed vocabulary**: `{rdf:type, has_function, emitted_by, depends_on, +reads_field, raises, traverses_relation}`, and `ndjson::from_ndjson` +**hard-rejects** any predicate outside it. None of these is an *association / +implication* relation. So a mined `X → Y` rule cannot today flow through the +ruff loader: its natural predicate (`implies` / `co_occurs_with`) is not in the +set. This is by design — the doc says adding a predicate is *"a deliberate +ontology change."* The Aerial `DebugProjector` emits `"implies"` precisely to +surface this seam. + +→ **Deliverable D-ARM-SYN-1 (CONJECTURE, council-gated):** add `Implies` +(and/or `CoOccursWith`) to `ruff_spo_triplet::Predicate` with a new +`Provenance` tier (see §4) so ARM-discovered rules load through the *same* +`parse_triples` path as the static extractor. Touches: one enum variant + one +`as_str`/`from_str` arm + one `default_provenance` arm + the ndjson closed- +vocab test. Low LOC, deliberate ontology decision → must pass the council +(`dto-soa-savant` + `prior-art-savant`). + +--- + +## 2. Synergy B — `ruff_python_dto_check` and Aerial are the two ends of the proposer spectrum + +`ruff_python_dto_check` harvests DTO/route/handler/model facts from **static +Python source** via `ruff_python_parser` + `ruff_python_ast` into a JSON +`Bundle`, and (with `ruff_ruby_spo`) fills the shared `ruff_spo_triplet::ModelGraph` +IR. It is the Rust-native successor to the plan's `tools/odoo-blueprint-extractor` +AST walk — the **`Extracted` / "Authoritative"** proposer leg. + +**FINDING — complementary, not redundant.** The dto_check extractor is *bounded +by what the source literally states* (it reads `@api.depends`, compute bodies, +`raise`s — see `ir.rs` cheat-sheet). It structurally **cannot** surface a +co-correlation that only exists in runtime rows — e.g. "invoices route through +Fiscal Position A when partner-country = B AND product-category = C", implicit +in years of `account.move` but absent from `account_fiscal_position.py`. That +is exactly the gap Aerial fills: the **`ArmDiscovered` runtime-data leg**. The +two are the same shape (`ModelGraph` → `Triple`), different evidence source: + +| Leg | Source | Crate | Provenance tier | Truth | +| --- | --- | --- | --- | --- | +| Curated | L-docs prose | hand | `Curated` | high f, high c | +| Extracted (static AST) | Python/Ruby source| `ruff_python_dto_check` | `Structural`/`Authoritative`/`Inferred` | tiered fixed | +| **ArmDiscovered (runtime)** | **parquet rows** | **`lance-graph-arm-discovery`** | **`ArmDiscovered`** | **data-derived `(f,c)`** | + +→ **Deliverable D-ARM-SYN-2 (CONJECTURE):** a `CandidateRule → ModelGraph` +adapter (or a parallel `arm:`-namespaced `ModelGraph`) so Aerial output joins +the dto_check output in one graph before `expand()`. The IR is "intentionally +dumb owned data" (per `ir.rs`), so the adapter is pure mapping — the antecedent +items become a synthetic subject IRI, the consequent an object IRI, the rule a +`Implies` edge. Gated on D-ARM-SYN-1. + +--- + +## 3. Synergy C — `op_emitter.rs` and `ruff_python_codegen` are the two codegen legs of one bracket + +`op_emitter.rs` (lance-graph-ontology) turns ratified typed-SoA recipes into +**deterministic Rust** (`RECIPE_*` consts + per-kind `Op` structs + static +slices, `include!`-ed at build time). `ruff_python_codegen::round_trip` +turns a Python AST back into **deterministic Python source** (`Generator` + +`Stylist`). Same thesis, two target languages: + +**FINDING.** Both are *deterministic, externalised-interpretation* codegen — +the exact doctrine the Abreu M2M paper independently validates ("externalize +mapping knowledge into ontologies + rules, not code"). Both consume an already- +ratified symbolic artifact; neither does discovery. They sit **downstream of +the determinism firewall.** + +**FINDING — the firewall is load-bearing for Aerial specifically.** Aerial+'s +autoencoder is nondeterministic in general (random init, denoising mask, float +reduction order). The transcode makes it *seedable* (`aerial::Rng`, +`reproducible_from_seed` test) so it is auditable, but it is still a **fan-in +proposer, never the trunk** (`streaming-arm-nars-discovery-v1.md` §2.1: pair- +stats is the deterministic default trunk). Therefore: + +- Aerial output is `CandidateRule` (a *proposal*), never a committed triple. +- Promotion to a triple `op_emitter` will compile is the **council's** job + (Stage D), via the `op_emitter::bucket_corpus` ratification filter + (D-ARM-10: `confidence ≥ Ratified`). An ArmDiscovered candidate that has not + passed the council never reaches either codegen leg. + +→ **Deliverable D-ARM-SYN-3 (CONJECTURE):** when D-ARM-SYN-1's `ArmDiscovered` +provenance tier lands, calibrate its `(f, c)` so that **un-ratified** ARM truth +sits *below* `op_emitter`'s ratification gate and *below* the static +`Inferred` tier `(0.85, 0.75)` — i.e. ARM rules are visible to the council but +filtered out of codegen until ratified. This makes the firewall a truth-gate, +not a separate code path. + +--- + +## 4. The truth-tier alignment (the number that has to be right) + +`ruff_spo_triplet::Provenance::truth()` fixes three tiers: +`Structural (1.0, 1.0)`, `Authoritative (0.95, 0.90)`, `Inferred (0.85, 0.75)`. +Aerial's truth is **not** a fixed tier — it is computed per rule by +`arm_to_nars`: + +- `f = ARM confidence = P(Y|X)` (clamped `[0,1]`) +- `c = m / (m + k)`, `m = support × n` (evidential mass), `k` = NAL-9 personality (default 1.0) + +So a well-evidenced, high-confidence ARM rule can *reach* Authoritative-tier +expectation, while a thin-evidence rule lands far below Inferred. That is the +desired behaviour: ARM truth is **continuous and evidence-driven**, and the +`I-NOISE-FLOOR-JIRAK` floor (D-ARM-7, separate gate) keeps the thin tail from +ever being emitted. The proposed `ArmDiscovered` tier is therefore best modelled +not as a fixed `(f,c)` pair but as "the `arm_to_nars` output, Jirak-floored, +council-gated for codegen." + +--- + +## 5. Determinism boundary — who sits where + +| Component | Det? | Side of firewall | Crate | +| --- | --- | --- | --- | +| pair-stats proposer (D-ARM-3) | yes | upstream (trunk) | lance-graph-arm-discovery (future) | +| **Aerial+ proposer (this work)** | seeded| **upstream (fan-in)** | **lance-graph-arm-discovery** | +| `ruff_python_dto_check` extractor | yes | upstream | ruff | +| translator `arm_to_nars` | yes | upstream | lance-graph-arm-discovery | +| ratification council (Stage D) | n/d→d| **THE firewall** | session A2A | +| `op_emitter` Rust codegen | yes | downstream | lance-graph-ontology | +| `ruff_python_codegen` round_trip | yes | downstream | ruff | + +The single rule: **nothing nondeterministic crosses the council.** Aerial is +the only nondeterministic node in the whole bracket, which is exactly why the +transcode keeps it standalone, seeded, and behind the `aerial` feature. + +--- + +## 6. Cross-refs + +- Code: `crates/lance-graph-arm-discovery/` (this work) — `aerial::{autoencoder, extract}`, `translator`, `ndjson`, `rule`, `encode`. +- `ruff/crates/ruff_spo_triplet/{triple.rs, ir.rs, ndjson.rs}` — the contract. +- `ruff/crates/ruff_python_dto_check/` — the static-AST sibling proposer. +- `ruff/crates/ruff_python_codegen/lib.rs` — the Python codegen sibling. +- `crates/lance-graph-ontology/src/odoo_blueprint/op_emitter.rs` — the Rust codegen leg + the ratification-filter touchpoint (D-ARM-10). +- `crates/lance-graph/src/graph/spo/truth.rs` — `TruthValue` (the truth the (f,c) feeds). +- Plan: `.claude/plans/streaming-arm-nars-discovery-v1.md` (D-ARM-9 Aerial leg; this is its transcode). +- Papers: Karabulut 2025 §2/§3.3 (truth + Algorithm 1); Abreu 2025 §4 (externalise-interpretation). diff --git a/.claude/knowledge/splat-codebook-aerial-wikidata-compression.md b/.claude/knowledge/splat-codebook-aerial-wikidata-compression.md new file mode 100644 index 00000000..f97cb521 --- /dev/null +++ b/.claude/knowledge/splat-codebook-aerial-wikidata-compression.md @@ -0,0 +1,111 @@ + + +# KNOWLEDGE: Splat codebook → aerial discovery → deterministic Wikidata compression + +## READ BY: +- Any worker wiring `lance-graph-arm-discovery::aerial::CodebookDistance` to a production oracle +- Any worker touching `crates/jc` (Jirak-Cartan) pillars 5/9/9b/10 in an ARM/compression context +- Any worker on the Wikidata-HHTL load (`specs/wikidata-hhtl-load.md`) skeleton/basin pass +- `truth-architect`, `integration-lead`, `palette-engineer` + +## P0 TRIGGERS: +- About to give `aerial` a real distance oracle → the oracle is jc's certified palette256 table, read this +- About to implement the D-ARM-7 Jirak significance floor → it derives from `jc::jirak`, read this +- About to add a float similarity to the ARM/discovery path → STOP, the float lives OFFLINE in jc, not here + +--- + +## The convergence + +`lance-graph-arm-discovery` (the Aerial+ transcode) shipped with two open seams: + +1. a **production `CodebookDistance` oracle** (the de-float replaced the autoencoder with an injected integer distance — E-ARM-PROBE-IS-CODEBOOK-TOPK), and +2. the **D-ARM-7 Jirak significance floor** (the still-unimplemented gate above the classical ARM support/confidence floors — ISSUE ARM-JIRAK-FLOOR). + +**Both resolve to the same crate: `crates/jc` (Jirak-Cartan).** jc *proves the +codebook is sound*; aerial *uses it to discover the ontology skeleton* that +drives deterministic Wikidata compression. The user's framing — +"gaussian-splat spatial blasgraph top-k 10000×10000 … for OWL/DOLCE+ SPO HHTL +classes and basin via aerial+ to deterministically compress Wikidata … adjacent +to JC Jirak[-Cartan] with EWA-sandwich gaussian splat" — names this end to end. + +```text + OFFLINE (build + certify the codebook — f64 is allowed here) + ─────────────────────────────────────────────────────────────────────── + ndarray::hpc::splat3d jc::ewa_sandwich{,_3d} (Pillars 9/9b) + Gaussian-splat spatial top-k → Σ-push-forward J·W·Σ·Wᵀ·Jᵀ certified + (10000×10000 BLASGraph; jc::sigma_codebook_probe: 256-codebook + jc splat_* graph algos: captures the Σ-distribution at + Louvain / triangle / LPA / ρ=0.9973 (R² ≥ 0.99, log-Euclidean) + Jaccard-Adamic-Adar) jc::pflug (Pillar 10): CAM-PQ tree + │ quantization is Lε-faithful ⇒ the + ▼ HHTL cascade preserves FreeEnergy + 256×256 integer [a,b] distance table ──────────────────────────────┐ + │ + ONLINE (use the frozen table — integer ONLY) │ + ───────────────────────────────────────────────────────────────────│─── + aerial::CodebookDistance ◄── MatrixDistance::new(spec, table) ◄─────┘ + │ (the codebook probe: nearest consequents within θ) + ▼ + aerial discovers OWL/DOLCE+ SPO HHTL classes + basins from Wikidata triples + │ gated by: classical support/confidence (ppm) + │ AND D-ARM-7 Jirak floor ◄── derives its rate from jc::jirak + ▼ (n^(p/2-1), weak dependence) + skeleton (P279/P31 class DAG + basin assignment) → codebook-HHTL compression + (specs/wikidata-hhtl-load.md: skeleton + basins + CAM-dedup + thin rows) +``` + +## The two seams, precisely + +| Seam | aerial side | jc side | +|---|---|---| +| Similarity oracle | `aerial::CodebookDistance::distance(a,b) -> u32` (integer, frozen). `MatrixDistance` already consumes a `[u32; dim²]` table. | jc builds + certifies that table: `ewa_sandwich` (splat Σ-push-forward correct) + `sigma_codebook_probe` (256-codebook viable at ρ=0.9973). | +| Significance floor (D-ARM-7) | `CandidateRule::passes()` gates classical support/confidence; the Jirak floor is the stricter, unimplemented gate above it. | `jc::jirak` supplies the correct rate (`n^(p/2-1)` for weakly-dependent data) — classical IID Berry-Esseen is wrong here (I-NOISE-FLOOR-JIRAK). The floor threshold derives from this rate. | + +## The float boundary (why this is doctrine-clean) + +The de-float removed float from aerial's **online** path. This pipeline keeps +that invariant: **float lives only in jc's OFFLINE certification** — k-means in +log-Euclidean space (`sigma_codebook_probe`), the SPD/covariance EWA math +(`ewa_sandwich`), the Berry-Esseen sup-error (`jirak`). All of that runs once, +produces a frozen integer artifact (the 256×256 `[a,b]` table + a derived +significance threshold), and is never on aerial's hot path. This is exactly the +CAM-PQ doctrine (`faiss-homology-cam-pq.md`, I-VSA-IDENTITIES): **build the +codebook offline (float OK), address it online with integer codes.** jc is the +"build + prove"; aerial is the "use". + +## jc pillar → role in this pipeline + +| jc module | Pillar | Role here | +|---|---|---| +| `ewa_sandwich`, `ewa_sandwich_3d` | 9, 9b | Certifies the Gaussian-splat Σ-push-forward (`ndarray::hpc::splat3d`) — the splat top-k that *builds* the codebook is correct. | +| `sigma_codebook_probe` | (probe) | The ρ=0.9973 / R²≥0.99 viability measurement — *why* a 256-codebook may replace a float similarity. | +| `pflug` | 10 | Pflug-Pichler nested-distance Lipschitz — the CAM-PQ/HHTL tree quantization preserves FreeEnergy within Lε, so codebook-HHTL compression is faithful, not lossy-by-surprise. | +| `jirak` | 5 | The weak-dependence Berry-Esseen rate — the engine the D-ARM-7 significance floor derives from. | + +## Wikidata / DOLCE mapping (specs/wikidata-hhtl-load.md, ogit-owl-dolce) + +- **Skeleton (Pass 1):** P279 (subClassOf) DAG + P31 (instance-of) → classes + basins (HHTL levels). aerial's discovered `(X → Y)` rules over the class/property items ARE candidate skeleton edges; the Jirak floor decides which are significant enough to persist. +- **DOLCE as axis template:** Endurant/Perdurant/Quality/Abstract (≈ Object/Process/Quality/Region) define WHICH HHTL axes exist; Wikidata properties fill WHAT occurs. aerial discovers the fill against the DOLCE scaffold. +- **Compression is structural:** classes + masks + refs + CAM-deduped shapes, not a gzip trick (~120GB → ~38GB). aerial supplies the class/basin structure; codebook-HHTL supplies the bucket router (16ⁿ nibble addressing). + +## Status & open questions + +- **CONJECTURE** (architecture): the splat→aerial→Wikidata wiring as a whole. The *seams* are concrete (the `CodebookDistance` trait + `jc::jirak`); the end-to-end pipeline is not yet built. +- **Hard prerequisite:** D-ARM-7 (the Jirak floor) must land before aerial promotes any rule to a live skeleton — ISSUE ARM-JIRAK-FLOOR. jc::jirak is the engine; the *gate function* (rule → significant?) is the deliverable. +- **No new aerial dependency needed:** jc emits a frozen integer table; aerial consumes it through the existing `MatrixDistance`/`CodebookDistance` seam. Do NOT make aerial depend on jc — keep the float-free standalone posture; pass the certified table in. +- **Open (from wikidata-hhtl-load):** P279 fan-out is wildly uneven (2…4000 children) — whether it rebalances onto a clean 16ⁿ tree or forces adaptive fan-out is MEASURABLE; measure on a real P279 subtree before fixing the HHTL base. +- **target-cpu:** the splat/SIMD kernels (AVX-512 VPOPCNTQ, Intel AMX) need `-C target-cpu=native` or `x86-64-v4`; otherwise ndarray's correct-but-scalar polyfill runs. + +## Cross-references + +- `crates/lance-graph-arm-discovery/` — the aerial consumer; `aerial::CodebookDistance` is the seam. +- `crates/jc/src/{ewa_sandwich,ewa_sandwich_3d,sigma_codebook_probe,pflug,jirak}.rs` — the certifier. +- `.claude/specs/wikidata-hhtl-load.md` — the compression pipeline. +- `.claude/knowledge/ogit-owl-dolce-ontology-compartments.md` — the DOLCE scaffold. +- `.claude/plans/3DGS-HHTL-datalake-traversal-plan.md` — the splat-HHTL traversal. +- `.claude/knowledge/faiss-homology-cam-pq.md`, CLAUDE.md `I-VSA-IDENTITIES`, `I-NOISE-FLOOR-JIRAK` — the float/codebook + Jirak iron rules. +- `EPIPHANIES.md` E-ARM-PROBE-IS-CODEBOOK-TOPK (the de-float) + the entry that points here. diff --git a/Cargo.toml b/Cargo.toml index d0cd8a57..0a30d429 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -39,6 +39,11 @@ exclude = [ "crates/learning", "crates/jc", "crates/sigker", + # Aerial+ ARM-discovery transcode — standalone zero-dep proposer crate, + # verified via `cargo test --manifest-path crates/lance-graph-arm-discovery/Cargo.toml`. + # Kept out of the workspace so the nondeterministic autoencoder never + # enters the deterministic lance-graph compile path (determinism boundary). + "crates/lance-graph-arm-discovery", ] resolver = "2" diff --git a/crates/lance-graph-arm-discovery/Cargo.lock b/crates/lance-graph-arm-discovery/Cargo.lock new file mode 100644 index 00000000..1bcdfbbd --- /dev/null +++ b/crates/lance-graph-arm-discovery/Cargo.lock @@ -0,0 +1,170 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "arrayref" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" + +[[package]] +name = "arrayvec" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "blake3" +version = "1.8.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0aa83c34e62843d924f905e0f5c866eb1dd6545fc4d719e803d9ba6030371fce" +dependencies = [ + "arrayref", + "arrayvec", + "cc", + "cfg-if", + "constant_time_eq", + "cpufeatures", +] + +[[package]] +name = "cc" +version = "1.2.63" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "556e016178bb5662a08681bbe0f00f8e17631781a4dfc8c45e466e4b185ec27f" +dependencies = [ + "find-msvc-tools", + "shlex", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "constant_time_eq" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" + +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + +[[package]] +name = "find-msvc-tools" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" + +[[package]] +name = "lance-graph-arm-discovery" +version = "0.1.0" +dependencies = [ + "ndarray", +] + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "matrixmultiply" +version = "0.3.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a06de3016e9fae57a36fd14dba131fccf49f74b40b7fbdb472f96e361ec71a08" +dependencies = [ + "autocfg", + "rawpointer", +] + +[[package]] +name = "ndarray" +version = "0.17.2" +dependencies = [ + "blake3", + "matrixmultiply", + "num-complex", + "num-integer", + "num-traits", + "paste", + "portable-atomic", + "portable-atomic-util", + "rawpointer", +] + +[[package]] +name = "num-complex" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-integer" +version = "0.1.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", +] + +[[package]] +name = "paste" +version = "1.0.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" + +[[package]] +name = "portable-atomic" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" + +[[package]] +name = "portable-atomic-util" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" +dependencies = [ + "portable-atomic", +] + +[[package]] +name = "rawpointer" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "60a357793950651c4ed0f3f52338f53b2f809f32d83a07f72909fa13e4c6c1e3" + +[[package]] +name = "shlex" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" diff --git a/crates/lance-graph-arm-discovery/Cargo.toml b/crates/lance-graph-arm-discovery/Cargo.toml new file mode 100644 index 00000000..6937fbd9 --- /dev/null +++ b/crates/lance-graph-arm-discovery/Cargo.toml @@ -0,0 +1,45 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright The Lance Authors + +[package] +name = "lance-graph-arm-discovery" +version = "0.1.0" +edition = "2021" +description = "Deterministic codebook-probe association-rule discovery (Aerial+ transcode) → NARS-truth SPO candidates" +license = "Apache-2.0" + +# Standalone, zero-dependency crate by default — EXCLUDED from the workspace +# (see root Cargo.toml `exclude`), like bgz17 / deepnsm / bgz-tensor. Verified +# with the DEFAULT (scalar, std-only) feature set: +# +# cargo test --manifest-path crates/lance-graph-arm-discovery/Cargo.toml +# +# The discovery path is float-free and bitwise-deterministic (integer codebook +# distance, integer evidence counts, AND+popcount over row bitsets). + +[features] +default = [] +# Route the bitset AND+popcount primitive (`simd.rs`) through +# `ndarray::simd::U64x8` per `ndarray-vertical-simd-alien-magic.md`. Verified +# here: `cargo test/clippy --features ndarray-simd` → 33/33, clippy clean +# (ndarray builds as a path dep with default-features=false + std; the +# `ndarray-rand` workspace member is NOT pulled). +# +# TARGET-CPU: the U64x8 polyfill is CORRECT on any target but only emits the +# real vector kernels (AVX-512 VPOPCNTQ for popcnt, AMX for the matmul/VNNI +# paths) when built for native silicon. Build with +# RUSTFLAGS="-C target-cpu=native" (this machine), or +# RUSTFLAGS="-C target-cpu=x86-64-v4" (AVX-512 baseline) +# Without it the popcnt path is the scalar fused-loop fallback — same result, +# no SIMD speedup. The default (scalar) feature does not pull ndarray at all. +ndarray-simd = ["dep:ndarray"] + +[dependencies.ndarray] +path = "../../../ndarray" +optional = true +default-features = false +# `std` gates `ndarray::simd` (the `U64x8` polyfill); that is all we need. +features = ["std"] + +[lib] +doctest = true diff --git a/crates/lance-graph-arm-discovery/README.md b/crates/lance-graph-arm-discovery/README.md new file mode 100644 index 00000000..534812ac --- /dev/null +++ b/crates/lance-graph-arm-discovery/README.md @@ -0,0 +1,98 @@ + + +# lance-graph-arm-discovery + +Rust transcode of **Aerial+** — neurosymbolic association-rule mining +(Karabulut, Groth, Degeler, arXiv 2504.19354v1) — with the paper's `f32` +autoencoder **replaced by an integer codebook distance oracle**. The +**upstream proposer leg** of `streaming-arm-nars-discovery-v1.md`: it mines +`(X → Y)` rules from tabular runtime data and lifts each into a NARS-truth SPO +candidate emitted in the `{s,p,o,f,c}` shape the SPO store loader reads. + +```text + rows ─► codebook items ─► codebook probe (nearest consequents within θ) + (integer CodebookDistance oracle = palette256) + ─► confirm on data: integer support/confidence counts + ─► CandidateRule (integer evidence) + ─► arm_to_truth_u8 → TruthU8 (= CausalEdge64 wire) + ─► {s,p,o,f,c} ndjson ─► SPO store loader +``` + +## Float-free by design + +The autoencoder was a slow, seed-dependent, `f32` way to approximate a +nearest-neighbour query — the very thing this substrate answers *exactly* and +in integers via the **palette256 distance table** (`[a,b] → u32`, ρ=0.9973 vs +cosine). So Aerial+'s reconstruction probe becomes a **codebook top-k**: + +| Aerial+ float piece | Deterministic replacement | +| --- | --- | +| denoising autoencoder (`f32` weights) | frozen `CodebookDistance` oracle (palette256) | +| reconstruction probe (forward pass) | codebook top-k from the antecedent, within `θ` | +| softmax ranking | integer distance ranking | +| support/confidence (`f32`) | integer counts; ppm cross-multiply gates | +| `(f,c)` `f32` truth | `TruthU8` = `confidence_u8` + i4 mantissa | +| seeded RNG | **deleted** — codebook lookup is bitwise-exact | + +The only `f32` left is the **serialization edge** (`arm_to_nars`, +`CandidateTriple`, `ndjson`), present solely because the downstream +`spo::truth::TruthValue` and `ruff_spo_triplet::Triple` are themselves `f32`. +Nothing in the discovery path consumes it. + +The similarity oracle is a **zero-dep trait** (`aerial::CodebookDistance`), so +the real distance table — `bgz17::PaletteDistanceTable`, the BLASGraph +Gaussian-splat top-k, or an HDR-popcount Hamming primitive — lives on the +consumer side and this crate stays standalone. `aerial::MatrixDistance` is the +in-crate reference impl. + +## No determinism firewall needed + +The codebook probe is **bitwise-deterministic by construction** (same data + +oracle + `θ` ⇒ identical rules, every target). It can sit in the deterministic +trunk beside pair-stats (D-ARM-3); the ratification council still governs +*promotion to the SPO store*, but no longer because of any nondeterminism here. +The Python-IPC isolation rationale (D-ARM-9) is fully moot. + +## Layout + +| Module | Stage | Role | +| --- | --- | --- | +| `encode` | — | integer `FeatureSpec` + `Dataset` + `count_matching` | +| `aerial::codebook` | A | `CodebookDistance` trait + `MatrixDistance` reference impl | +| `aerial::extract` | A | codebook-probe rule extraction (Algorithm 1, integer) | +| `aerial` (`AerialProposer`) | A | `Proposer` impl over a data window + oracle | +| `translator` | B | `arm_to_truth_u8` (`TruthU8`) + `f32` edge + `FeedProjector` | +| `ndjson` | emit | `{s,p,o,f,c}` lines = the SPO-loader contract | +| `rule` | — | `Item`, `CandidateRule` (integer counts, ppm gates), `Proposer` | + +## Build & test + +```bash +# Default: scalar, std-only, zero-dep (independently verifiable) +cargo test --manifest-path crates/lance-graph-arm-discovery/Cargo.toml +cargo clippy --manifest-path crates/lance-graph-arm-discovery/Cargo.toml --all-targets -- -D warnings + +# SIMD: route the bitset AND+popcount through ndarray::simd::U64x8 +cargo test --manifest-path crates/lance-graph-arm-discovery/Cargo.toml --features ndarray-simd +``` + +The count loop transposes the window into row bitsets (`bitset::RowMasks`) so +every candidate count is an `AND` + popcount over `&[u64]` (`simd.rs`). The +default path is scalar `u64::count_ones`; the `ndarray-simd` feature swaps in +`ndarray::simd::U64x8` (zero raw intrinsics here — the polyfill owns dispatch). + +**target-cpu:** the `U64x8` path is correct everywhere but only emits the real +vector kernels (AVX-512 VPOPCNTQ, AMX) when built for native silicon — +`RUSTFLAGS="-C target-cpu=native"` or `-C target-cpu=x86-64-v4`. Otherwise it is +ndarray's scalar fused-loop fallback (same result, no speedup). + +## Status + +Implements the Aerial+ leg of the plan as a deterministic codebook backend +(D-ARM-13). Still open: D-ARM-7 Jirak floor (ISSUE ARM-JIRAK-FLOOR — hard +prerequisite before wiring to a live `SpoStore`), the `lance-graph-contract` +carriers (D-ARM-1/2, TD-ARM-CARRIER-FORK), and the real palette256 oracle wiring +(consumer-side `CodebookDistance` impl over `bgz17::PaletteDistanceTable`). diff --git a/crates/lance-graph-arm-discovery/src/aerial/codebook.rs b/crates/lance-graph-arm-discovery/src/aerial/codebook.rs new file mode 100644 index 00000000..c5083c79 --- /dev/null +++ b/crates/lance-graph-arm-discovery/src/aerial/codebook.rs @@ -0,0 +1,152 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! The deterministic similarity oracle — `CodebookDistance`. +//! +//! This is the float-free replacement for Aerial+'s autoencoder. The paper's +//! reconstruction probe ("mark the antecedent, read off the high-probability +//! consequents") is, mechanically, a **nearest-neighbour query in the learned +//! co-occurrence space** — and this substrate already answers that exactly, +//! in integers, via the **palette256 distance table** (`[a,b] → u32`, ρ=0.9973 +//! vs cosine). So the probe ranks consequents by integer codebook distance, +//! never by a softmax over float weights. +//! +//! The oracle is a **trait**, not a concrete table, so this crate stays +//! zero-dep and standalone. The production table is **built and certified +//! offline by `crates/jc` (Jirak-Cartan)** — `ewa_sandwich` certifies the +//! Gaussian-splat Σ-push-forward (the 10000² BLASGraph spatial top-k that +//! constructs it), and `sigma_codebook_probe` measures the 256-codebook viable +//! at ρ=0.9973. That offline build may use float (k-means, SPD math); the +//! frozen `[u32; dim²]` table it emits is consumed here as integer through +//! [`MatrixDistance`]. Float to BUILD the codebook, integer to USE it — the +//! CAM-PQ doctrine. See `.claude/knowledge/splat-codebook-aerial-wikidata-compression.md`. +//! [`MatrixDistance`] is also the in-crate reference impl used by tests. +//! +//! Invariant (per `faiss-homology-cam-pq.md`): this distance is **discovery / +//! shape-family only**, never identity addressing. Identity is the exact CAM +//! hash over canonicalized `(s,p,o)`. Same triples, two indexes, never swapped. + +use crate::encode::FeatureSpec; +use crate::rule::Item; + +/// A deterministic, integer distance between two codebook items. Lower means +/// more associated (nearer). No float, no seed — bitwise-identical on every +/// target. +pub trait CodebookDistance { + /// Distance between two items' codebook codes. Lower = nearer. + fn distance(&self, a: Item, b: Item) -> u32; +} + +/// Reference [`CodebookDistance`] over a flat `dim × dim` `u32` table keyed by +/// the [`FeatureSpec`] slot of each item. Zero-dep; the in-crate stand-in for +/// `bgz17::PaletteDistanceTable`. +#[derive(Debug, Clone)] +pub struct MatrixDistance { + dim: usize, + offsets: Vec, + table: Vec, +} + +impl MatrixDistance { + /// Build from a spec and a flat `dim × dim` row-major distance table. + /// + /// # Panics + /// If `table.len() != spec.dim()²`. + #[must_use] + pub fn new(spec: &FeatureSpec, table: Vec) -> Self { + let dim = spec.dim(); + assert_eq!(table.len(), dim * dim, "distance table must be dim × dim"); + let offsets: Vec = (0..spec.num_features()) + .map(|f| spec.block(f).0) + .collect(); + Self { dim, offsets, table } + } + + /// The flat code (slot index) of an item. Validates bounds so an invalid + /// `(feature, category)` fails fast rather than aliasing another feature's + /// block and returning a real-but-wrong distance. + #[must_use] + fn code(&self, it: Item) -> usize { + let feature = it.feature as usize; + let start = *self + .offsets + .get(feature) + .expect("item feature out of range for distance table"); + let end = self.offsets.get(feature + 1).copied().unwrap_or(self.dim); + let category = it.category as usize; + assert!( + category < end - start, + "item category {category} out of range for feature {feature} block" + ); + start + category + } +} + +impl CodebookDistance for MatrixDistance { + fn distance(&self, a: Item, b: Item) -> u32 { + self.table[self.code(a) * self.dim + self.code(b)] + } +} + +/// Aggregate the distance from a (multi-item) antecedent to a candidate +/// consequent item: the **nearest** antecedent item wins (min). For a +/// single-item antecedent this is just `distance(a, item)`. +#[must_use] +pub fn antecedent_distance( + oracle: &dyn CodebookDistance, + antecedent: &[Item], + item: Item, +) -> u32 { + antecedent + .iter() + .map(|&a| oracle.distance(a, item)) + .min() + .unwrap_or(u32::MAX) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn matrix_distance_is_slot_addressed_and_symmetric_if_built_so() { + let spec = FeatureSpec::new(vec![2, 2]); // dim 4 + // 4×4 table; (f0,c0)=slot0 near (f1,c0)=slot2 (dist 1), far from slot3. + #[rustfmt::skip] + let table = vec![ + 0, 5, 1, 9, + 5, 0, 9, 1, + 1, 9, 0, 5, + 9, 1, 5, 0, + ]; + let d = MatrixDistance::new(&spec, table); + assert_eq!(d.distance(Item::new(0, 0), Item::new(1, 0)), 1); + assert_eq!(d.distance(Item::new(0, 0), Item::new(1, 1)), 9); + assert_eq!(d.distance(Item::new(0, 0), Item::new(0, 0)), 0); + } + + #[test] + fn antecedent_distance_takes_the_nearest_item() { + let spec = FeatureSpec::new(vec![2, 2]); + #[rustfmt::skip] + let table = vec![ + 0, 5, 1, 9, + 5, 0, 9, 1, + 1, 9, 0, 5, + 9, 1, 5, 0, + ]; + let d = MatrixDistance::new(&spec, table); + // antecedent {(f0,c0)=slot0, (f0,c1)=slot1} to (f1,c0)=slot2: min(1, 9) = 1 + let ant = [Item::new(0, 0), Item::new(0, 1)]; + assert_eq!(antecedent_distance(&d, &ant, Item::new(1, 0)), 1); + } + + #[test] + #[should_panic(expected = "out of range")] + fn code_rejects_out_of_range_category() { + let spec = FeatureSpec::new(vec![2, 2]); + let d = MatrixDistance::new(&spec, vec![0u32; 16]); + // category 5 ≥ feature-0 block width 2 — must fail, not alias feature 1. + let _ = d.distance(Item::new(0, 5), Item::new(1, 0)); + } +} diff --git a/crates/lance-graph-arm-discovery/src/aerial/extract.rs b/crates/lance-graph-arm-discovery/src/aerial/extract.rs new file mode 100644 index 00000000..56a629be --- /dev/null +++ b/crates/lance-graph-arm-discovery/src/aerial/extract.rs @@ -0,0 +1,334 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! Aerial+ rule extraction — the **codebook-probe** backend (float-free). +//! +//! Aerial+'s reconstruction probe (paper §3.3) marks an antecedent, leaves +//! the rest "unknown", and reads off the consequents the network reconstructs +//! with high probability. Mechanically that is: *which other items are +//! nearest to the antecedent in the learned co-occurrence space?* This backend +//! answers it deterministically with an integer [`CodebookDistance`] oracle +//! (the palette256 table) instead of a float autoencoder: +//! +//! 1. **probe** — for every feature not in the antecedent, take the category +//! whose codebook distance to the antecedent is minimal; if it is within +//! `theta`, propose it as a consequent (the τ_c analogue, integer); +//! 2. **confirm** — measure `support`/`confidence` on the actual data as +//! integer counts and gate by the classical ARM floors (ppm). +//! +//! Double gate, exactly as the paper: the codebook *proposes* (near pairs), +//! the data *confirms* (co-occurrence). An independent feature whose nearest +//! category clears `theta` is dropped at the confidence floor because +//! `P(Y|X) ≈ P(Y)` for independent `Y`. No softmax, no seed, no `f32`. + +use crate::aerial::codebook::{antecedent_distance, CodebookDistance}; +use crate::bitset::RowMasks; +use crate::encode::Dataset; +use crate::rule::{CandidateRule, Item}; + +/// Thresholds + bounds for [`extract_rules`]. All integer. +#[derive(Debug, Clone, Copy)] +pub struct ExtractParams { + /// Maximum codebook distance for a category to be a candidate consequent + /// (the τ_c analogue, in the oracle's integer units — e.g. palette256 + /// distance). `u32::MAX` = no codebook prune (always take the nearest + /// category, let the data gates decide). Tune to your distance scale. + pub theta: u32, + /// Maximum antecedent size `|X|`. + pub max_antecedent: usize, + /// Classical minimum support floor, parts-per-million. + pub min_support_ppm: u32, + /// Classical minimum confidence floor, parts-per-million. + pub min_confidence_ppm: u32, +} + +impl Default for ExtractParams { + fn default() -> Self { + Self { + theta: u32::MAX, + max_antecedent: 2, + min_support_ppm: 10_000, // 1% + min_confidence_ppm: 500_000, // 50% + } + } +} + +/// Run the codebook probe against a distance oracle and the data, returning +/// deterministically-sorted candidate rules. +#[must_use] +pub fn extract_rules( + oracle: &dyn CodebookDistance, + data: &Dataset, + params: &ExtractParams, +) -> Vec { + let spec = &data.spec; + let n = data.len() as u32; + if n == 0 { + return Vec::new(); + } + + // Transpose the window into row bitsets once; every count below is then an + // AND + popcount (the `ndarray::simd` seam), not an AoS rescan. + let masks = RowMasks::build(data); + + // Frequent single items (support floor) — the apriori prune that bounds + // the antecedent search, grouped by feature. Integer ppm. + let mut frequent_by_feature: Vec> = vec![Vec::new(); spec.num_features()]; + for (f, bucket) in frequent_by_feature.iter_mut().enumerate() { + for cat in 0..spec.cardinality(f) { + let item = Item::new(f as u32, cat); + let count = masks.support_count(item); + let support_ppm = ((count as u64 * crate::rule::PPM) / n as u64) as u32; + if support_ppm >= params.min_support_ppm { + bucket.push(item); + } + } + } + let candidate_features: Vec = (0..spec.num_features()) + .filter(|&f| !frequent_by_feature[f].is_empty()) + .collect(); + + let mut rules: Vec = Vec::new(); + if params.max_antecedent == 0 { + return rules; // caller asked for no antecedents — honor it, don't force singletons + } + for size in 1..=params.max_antecedent { + for feature_combo in feature_combinations(&candidate_features, size) { + for antecedent in item_product(&feature_combo, &frequent_by_feature) { + probe(oracle, data, &masks, &antecedent, &feature_combo, params, &mut rules); + } + } + } + + rules.sort_by(|a, b| { + a.antecedent + .cmp(&b.antecedent) + .then_with(|| a.consequent.cmp(&b.consequent)) + }); + rules.dedup_by(|a, b| a.antecedent == b.antecedent && a.consequent == b.consequent); + rules +} + +/// Probe one antecedent: for each other feature, take the codebook-nearest +/// category; if within `theta`, confirm on data and emit if it clears the +/// ARM floors. +#[allow(clippy::too_many_arguments)] +fn probe( + oracle: &dyn CodebookDistance, + data: &Dataset, + masks: &RowMasks, + antecedent: &[Item], + antecedent_features: &[usize], + params: &ExtractParams, + out: &mut Vec, +) { + let spec = &data.spec; + let n = data.len() as u32; + let antecedent_count = masks.and_count(antecedent); + if antecedent_count == 0 { + return; + } + + for g in 0..spec.num_features() { + if antecedent_features.contains(&g) { + continue; + } + // Codebook-nearest category of feature g to the antecedent. + let (mut best_cat, mut best_dist) = (0u32, u32::MAX); + for cat in 0..spec.cardinality(g) { + let d = antecedent_distance(oracle, antecedent, Item::new(g as u32, cat)); + if d < best_dist { + best_dist = d; + best_cat = cat; + } + } + if best_dist > params.theta { + continue; + } + + let consequent = vec![Item::new(g as u32, best_cat)]; + let mut both = antecedent.to_vec(); + both.extend_from_slice(&consequent); + let cooccur = masks.and_count(&both); + + let rule = CandidateRule { + antecedent: antecedent.to_vec(), + consequent, + cooccur, + antecedent_count, + window: n, + }; + if rule.passes(params.min_support_ppm, params.min_confidence_ppm) { + out.push(rule); + } + } +} + +/// All size-`k` combinations of the given feature indices (ascending, +/// deterministic). +fn feature_combinations(features: &[usize], k: usize) -> Vec> { + let mut out = Vec::new(); + let mut combo = Vec::with_capacity(k); + fn recurse( + features: &[usize], + start: usize, + k: usize, + combo: &mut Vec, + out: &mut Vec>, + ) { + if combo.len() == k { + out.push(combo.clone()); + return; + } + for i in start..features.len() { + combo.push(features[i]); + recurse(features, i + 1, k, combo, out); + combo.pop(); + } + } + recurse(features, 0, k, &mut combo, &mut out); + out +} + +/// Cartesian product of the frequent items of each feature in `combo`, +/// yielding one antecedent (sorted by feature) per element. +fn item_product(combo: &[usize], frequent_by_feature: &[Vec]) -> Vec> { + let mut result: Vec> = vec![Vec::new()]; + for &f in combo { + let items = &frequent_by_feature[f]; + let mut next = Vec::with_capacity(result.len() * items.len()); + for prefix in &result { + for &item in items { + let mut ext = prefix.clone(); + ext.push(item); + next.push(ext); + } + } + result = next; + } + result +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::aerial::codebook::MatrixDistance; + use crate::encode::FeatureSpec; + + /// 3 features × 2 cats. Codebook places (f*,c0) near each other and + /// (f*,c1) near each other; feature 2 is far from features 0/1 (independent + /// in codebook space). Data plants f0 == f1 with a little noise; f2 random. + fn fixture(noise_every: usize) -> (Dataset, MatrixDistance) { + let spec = FeatureSpec::new(vec![2, 2, 2]); // dim 6 + // slots: f0c0=0 f0c1=1 | f1c0=2 f1c1=3 | f2c0=4 f2c1=5 + // near = 1, mid = 5, far = 50. + let near = 1u32; + let mid = 5u32; + let far = 50u32; + let mut table = vec![0u32; 36]; + let set = |t: &mut Vec, a: usize, b: usize, v: u32| { + t[a * 6 + b] = v; + t[b * 6 + a] = v; + }; + // f0c0(0) ~ f1c0(2) near; f0c1(1) ~ f1c1(3) near; cross pairs mid. + set(&mut table, 0, 2, near); + set(&mut table, 1, 3, near); + set(&mut table, 0, 3, mid); + set(&mut table, 1, 2, mid); + set(&mut table, 0, 1, mid); + set(&mut table, 2, 3, mid); + // feature 2 (slots 4,5) far from everything. + for a in 0..4 { + set(&mut table, a, 4, far); + set(&mut table, a, 5, far); + } + set(&mut table, 4, 5, mid); + + let rows: Vec> = (0..600) + .map(|i| { + let a = (i % 2) as u32; + let b = if noise_every != 0 && i % noise_every == 0 { 1 - a } else { a }; + let c = ((i / 2) % 2) as u32; + vec![a, b, c] + }) + .collect(); + (Dataset::new(spec.clone(), rows), MatrixDistance::new(&spec, table)) + } + + #[test] + fn recovers_planted_rule_and_rejects_independent_feature() { + let (data, dist) = fixture(20); // 5% label noise on f1 + let params = ExtractParams { + theta: 2, // codebook prune: only "near" (dist ≤ 2) consequents + max_antecedent: 1, + min_support_ppm: 50_000, // 5% + min_confidence_ppm: 700_000, // 70% + }; + let rules = extract_rules(&dist, &data, ¶ms); + + // planted f0=0 ⇒ f1=0 (codebook-near AND data-confirmed) + let has = rules.iter().any(|r| { + r.antecedent == vec![Item::new(0, 0)] && r.consequent == vec![Item::new(1, 0)] + }); + assert!(has, "planted f0=0 ⇒ f1=0 not recovered: {rules:#?}"); + + // feature 2 is codebook-far → pruned by theta → never a consequent of f0 + let spurious = rules.iter().any(|r| { + r.antecedent == vec![Item::new(0, 0)] && r.consequent[0].feature == 2 + }); + assert!(!spurious, "independent feature 2 leaked: {rules:#?}"); + + let rule = rules + .iter() + .find(|r| r.antecedent == vec![Item::new(0, 0)] && r.consequent == vec![Item::new(1, 0)]) + .unwrap(); + assert!(rule.confidence_ppm() >= 700_000, "confidence below floor"); + } + + #[test] + fn fully_deterministic_no_seed() { + let (data, dist) = fixture(20); + let p = ExtractParams { theta: 2, max_antecedent: 1, min_support_ppm: 50_000, min_confidence_ppm: 700_000 }; + let a = extract_rules(&dist, &data, &p); + let b = extract_rules(&dist, &data, &p); + assert_eq!(a, b, "codebook probe is bitwise-deterministic by construction"); + } + + #[test] + fn theta_prune_blocks_far_consequents() { + // With a generous theta the far feature is *tested* (then data-gated); + // with a tight theta it is pruned before the data even sees it. + let (data, dist) = fixture(0); // no noise: f1==f0 exactly + let tight = ExtractParams { theta: 2, max_antecedent: 1, min_support_ppm: 50_000, min_confidence_ppm: 600_000 }; + let rules = extract_rules(&dist, &data, &tight); + assert!(rules.iter().all(|r| r.consequent[0].feature != 2)); + } + + #[test] + fn feature_combinations_are_correct() { + assert_eq!( + feature_combinations(&[0, 1, 2], 2), + vec![vec![0, 1], vec![0, 2], vec![1, 2]] + ); + } + + #[test] + fn empty_dataset_yields_no_rules() { + let spec = FeatureSpec::new(vec![2, 2]); + let data = Dataset::new(spec.clone(), Vec::new()); + let dist = MatrixDistance::new(&spec, vec![0u32; 16]); + assert!(extract_rules(&dist, &data, &ExtractParams::default()).is_empty()); + } + + #[test] + fn max_antecedent_zero_yields_no_rules() { + let (data, dist) = fixture(0); + let p = ExtractParams { + theta: 2, + max_antecedent: 0, + min_support_ppm: 0, + min_confidence_ppm: 0, + }; + assert!(extract_rules(&dist, &data, &p).is_empty()); + } +} diff --git a/crates/lance-graph-arm-discovery/src/aerial/mod.rs b/crates/lance-graph-arm-discovery/src/aerial/mod.rs new file mode 100644 index 00000000..69ef577a --- /dev/null +++ b/crates/lance-graph-arm-discovery/src/aerial/mod.rs @@ -0,0 +1,158 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! Aerial+ rule discovery — the **deterministic codebook-probe** backend. +//! +//! Transcode of Karabulut, Groth, Degeler, *Neurosymbolic Association Rule +//! Mining from Tabular Data* (arXiv 2504.19354v1), with the paper's float +//! autoencoder **replaced by an integer codebook distance oracle**. The +//! reconstruction probe is a nearest-neighbour query; this substrate answers +//! it exactly via the palette256 distance table (ρ=0.9973 vs cosine), so no +//! `f32`, no SGD, no seed enter the proposer. +//! +//! - [`codebook::CodebookDistance`] — the injected integer similarity oracle +//! (palette256 / BLASGraph splat / HDR-popcount; `MatrixDistance` in tests). +//! - [`extract::extract_rules`] — Algorithm 1 as a codebook top-k + data +//! confirmation. +//! - [`AerialProposer`] — wires them into the [`crate::rule::Proposer`] +//! contract. +//! +//! # No determinism firewall needed +//! +//! The autoencoder transcode was a *nondeterministic fan-in* that had to stay +//! behind the ratification gate and out of the compile path. The codebook +//! probe is **bitwise-deterministic by construction** — same data + same +//! oracle + same `theta` ⇒ identical rules on every target. It can sit in the +//! deterministic trunk beside pair-stats (D-ARM-3); the ratification gate +//! still governs *promotion to the SPO store*, but no longer because of any +//! nondeterminism here. + +pub mod codebook; +pub mod extract; + +pub use codebook::{antecedent_distance, CodebookDistance, MatrixDistance}; +pub use extract::{extract_rules, ExtractParams}; + +use crate::encode::Dataset; +use crate::rule::{CandidateRule, Proposer}; + +/// Configuration for an [`AerialProposer`] — the extraction thresholds. +/// (No autoencoder hyper-parameters and no seed: the backend is deterministic.) +pub type AerialParams = ExtractParams; + +/// An Aerial+ codebook-probe proposer over one data window + a distance oracle. +#[derive(Debug)] +pub struct AerialProposer { + data: Dataset, + oracle: D, + params: AerialParams, + drained: bool, +} + +impl AerialProposer { + /// Build a proposer over a window and an injected codebook oracle. + /// No training step — the oracle is the frozen co-occurrence model. + #[must_use] + pub fn new(data: Dataset, oracle: D, params: AerialParams) -> Self { + Self { + data, + oracle, + params, + drained: false, + } + } + + /// Run the codebook probe once, regardless of the drained flag. + #[must_use] + pub fn mine(&self) -> Vec { + extract_rules(&self.oracle, &self.data, &self.params) + } +} + +impl Proposer for AerialProposer { + fn next_batch(&mut self) -> Vec { + if self.drained { + return Vec::new(); + } + self.drained = true; + self.mine() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::encode::FeatureSpec; + use crate::ndjson::to_ndjson; + use crate::rule::Item; + use crate::translator::{CandidateTriple, DebugProjector, NARS_PERSONALITY_K}; + + fn fixture() -> (Dataset, MatrixDistance) { + let spec = FeatureSpec::new(vec![2, 2, 2]); + // Initialise every off-diagonal pair to "far" (50), diagonal to 0, then + // mark only the near pairs — so an unset cell can never read as nearest. + let mut table = vec![50u32; 36]; + for d in 0..6 { + table[d * 6 + d] = 0; + } + let set = |a: usize, b: usize, v: u32, t: &mut Vec| { + t[a * 6 + b] = v; + t[b * 6 + a] = v; + }; + // f0c0(0)~f1c0(2) near; f0c1(1)~f1c1(3) near. f2 (4,5) stays far. + set(0, 2, 1, &mut table); + set(1, 3, 1, &mut table); + let rows = (0..600) + .map(|i| { + let a = (i % 2) as u32; + let c = ((i / 2) % 2) as u32; + vec![a, a, c] + }) + .collect(); + (Dataset::new(spec.clone(), rows), MatrixDistance::new(&spec, table)) + } + + #[test] + fn proposer_yields_then_drains() { + let (data, dist) = fixture(); + let params = AerialParams { + theta: 2, + max_antecedent: 1, + min_support_ppm: 50_000, + min_confidence_ppm: 700_000, + }; + let mut p = AerialProposer::new(data, dist, params); + assert!(!p.next_batch().is_empty()); + assert!(p.next_batch().is_empty()); + } + + #[test] + fn mined_rules_serialise_to_spo_ndjson() { + let (data, dist) = fixture(); + let params = AerialParams { theta: 2, max_antecedent: 1, min_support_ppm: 50_000, min_confidence_ppm: 700_000 }; + let p = AerialProposer::new(data, dist, params); + let rules = p.mine(); + assert!(!rules.is_empty()); + let proj = DebugProjector::default(); + let triples: Vec = rules + .iter() + .map(|r| CandidateTriple::from_rule(r, &proj, NARS_PERSONALITY_K)) + .collect(); + let ndjson = to_ndjson(&triples); + for line in ndjson.lines() { + assert!(line.starts_with("{\"s\":\""), "line: {line}"); + assert!(line.contains("\"p\":\"implies\"")); + } + } + + #[test] + fn reproducible_no_seed_needed() { + let (d1, dist1) = fixture(); + let (d2, dist2) = fixture(); + let p = AerialParams { theta: 2, max_antecedent: 1, min_support_ppm: 50_000, min_confidence_ppm: 700_000 }; + let r1 = AerialProposer::new(d1, dist1, p).mine(); + let r2 = AerialProposer::new(d2, dist2, p).mine(); + assert_eq!(r1, r2); + let _ = Item::new(0, 0); + } +} diff --git a/crates/lance-graph-arm-discovery/src/bitset.rs b/crates/lance-graph-arm-discovery/src/bitset.rs new file mode 100644 index 00000000..43c61bdb --- /dev/null +++ b/crates/lance-graph-arm-discovery/src/bitset.rs @@ -0,0 +1,150 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! Row-bitset SoA — the columnar form the codebook probe counts over. +//! +//! `count_matching` on the AoS `Dataset` is an `O(rows × items)` scan. The +//! probe calls it once per candidate, so the window is rescanned thousands of +//! times. [`RowMasks`] transposes the window once into one `u64` bitset per +//! `(feature, category)` item (bit `r` set iff row `r` has that category); +//! then every candidate's support/co-occurrence count is an `AND` + popcount +//! over `&[u64]` — the [`crate::simd`] primitive that routes through +//! `ndarray::simd::U64x8` under the `ndarray-simd` feature. + +use crate::encode::Dataset; +use crate::rule::Item; +use crate::simd::{and_popcount, popcount}; + +/// One `u64` bitset per item slot, over the window's rows. +#[derive(Debug, Clone)] +pub struct RowMasks { + /// Rows in the window. + n: usize, + /// `u64` words per mask = `ceil(n / 64)`. + words: usize, + /// Per-feature code offset (`offsets[feature] + category` = item slot). + offsets: Vec, + /// Flat `dim × words` storage; mask of slot `s` is `[s·words .. (s+1)·words)`. + masks: Vec, +} + +impl RowMasks { + /// Transpose a dataset into row bitsets (one pass over the window). + #[must_use] + pub fn build(data: &Dataset) -> Self { + let spec = &data.spec; + let dim = spec.dim(); + let n = data.len(); + let words = n.div_ceil(64).max(1); + let offsets: Vec = (0..spec.num_features()).map(|f| spec.block(f).0).collect(); + let mut masks = vec![0u64; dim * words]; + for (r, row) in data.rows.iter().enumerate() { + let w = r / 64; + let bit = 1u64 << (r % 64); + for (f, &cat) in row.iter().enumerate() { + let slot = offsets[f] + cat as usize; + masks[slot * words + w] |= bit; + } + } + Self { + n, + words, + offsets, + masks, + } + } + + /// Window size `n`. + #[must_use] + pub fn len(&self) -> usize { + self.n + } + + /// Whether the window is empty. + #[must_use] + pub fn is_empty(&self) -> bool { + self.n == 0 + } + + /// The bitset of an item. + #[must_use] + fn mask(&self, item: Item) -> &[u64] { + let slot = self.offsets[item.feature as usize] + item.category as usize; + &self.masks[slot * self.words..(slot + 1) * self.words] + } + + /// `|{rows ⊇ {item}}|` — single-item support count. + #[must_use] + pub fn support_count(&self, item: Item) -> u32 { + popcount(self.mask(item)) + } + + /// `|{rows ⊇ items}|` — conjunction count (the evidence primitive). + /// The 1- and 2-item cases (the probe's hot path: a 1–2 item antecedent + /// plus one consequent) avoid any allocation. + #[must_use] + pub fn and_count(&self, items: &[Item]) -> u32 { + match items { + [] => self.n as u32, + [a] => self.support_count(*a), + [a, b] => and_popcount(self.mask(*a), self.mask(*b)), + [a, rest @ ..] => { + let mut acc = self.mask(*a).to_vec(); + for &it in rest { + for (x, &y) in acc.iter_mut().zip(self.mask(it)) { + *x &= y; + } + } + popcount(&acc) + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::encode::FeatureSpec; + + fn data() -> Dataset { + // rows: (0,0),(0,0),(0,1),(1,1) + let spec = FeatureSpec::new(vec![2, 2]); + Dataset::new(spec, vec![vec![0, 0], vec![0, 0], vec![0, 1], vec![1, 1]]) + } + + #[test] + fn masks_match_aos_count() { + let d = data(); + let m = RowMasks::build(&d); + // single-item: feature0=0 in 3 rows + assert_eq!(m.support_count(Item::new(0, 0)), 3); + assert_eq!(m.support_count(Item::new(0, 0)), d.count_matching(&[Item::new(0, 0)])); + // conjunction f0=0 ∧ f1=0 → 2 rows + let both = [Item::new(0, 0), Item::new(1, 0)]; + assert_eq!(m.and_count(&both), 2); + assert_eq!(m.and_count(&both), d.count_matching(&both)); + // empty conjunction = n + assert_eq!(m.and_count(&[]), 4); + } + + #[test] + fn three_item_conjunction_folds() { + let spec = FeatureSpec::new(vec![2, 2, 2]); + let d = Dataset::new(spec, vec![vec![0, 0, 0], vec![0, 0, 1], vec![0, 0, 0]]); + let m = RowMasks::build(&d); + let items = [Item::new(0, 0), Item::new(1, 0), Item::new(2, 0)]; + assert_eq!(m.and_count(&items), 2); + assert_eq!(m.and_count(&items), d.count_matching(&items)); + } + + #[test] + fn spans_multiple_words() { + // 130 rows → 3 u64 words per mask; every row has feature0=0. + let spec = FeatureSpec::new(vec![1, 2]); + let rows: Vec> = (0..130).map(|i| vec![0, (i % 2) as u32]).collect(); + let d = Dataset::new(spec, rows); + let m = RowMasks::build(&d); + assert_eq!(m.support_count(Item::new(0, 0)), 130); + assert_eq!(m.support_count(Item::new(1, 0)), 65); + } +} diff --git a/crates/lance-graph-arm-discovery/src/encode.rs b/crates/lance-graph-arm-discovery/src/encode.rs new file mode 100644 index 00000000..2ba674b0 --- /dev/null +++ b/crates/lance-graph-arm-discovery/src/encode.rs @@ -0,0 +1,178 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! Codebook items + the data window — **integer only**. +//! +//! Each row is a list of category indices, one per feature (a tuple of +//! codebook codes). There is no one-hot float vector and no embedding: the +//! codebook-probe backend ([`crate::aerial`]) reasons over category indices +//! and integer co-occurrence counts, and the similarity it needs comes from +//! the injected [`crate::aerial::CodebookDistance`] oracle. Numeric columns +//! must be discretised into category indices *before* they reach this layer +//! (that quantisation belongs to the CAM/palette codebook, not here). + +use crate::rule::Item; + +/// The schema for a window: the ordered features and how many categories each +/// has. Block offsets are derived once and cached. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct FeatureSpec { + cardinalities: Vec, + /// Prefix-sum offsets: `offsets[i]` is the first code slot of feature `i`; + /// the last entry is the total code space size. + offsets: Vec, +} + +impl FeatureSpec { + /// Build a spec from per-feature cardinalities. + /// + /// # Panics + /// If any cardinality is zero. + #[must_use] + pub fn new(cardinalities: Vec) -> Self { + assert!( + cardinalities.iter().all(|&k| k > 0), + "every feature needs ≥1 category" + ); + let mut offsets = Vec::with_capacity(cardinalities.len() + 1); + let mut acc = 0usize; + offsets.push(0); + for &k in &cardinalities { + acc += k as usize; + offsets.push(acc); + } + Self { + cardinalities, + offsets, + } + } + + /// Total code space `Σ kᵢ` (the side length of a per-item distance table). + #[must_use] + pub fn dim(&self) -> usize { + *self.offsets.last().expect("offsets always has ≥1 entry") + } + + /// Number of features (columns). + #[must_use] + pub fn num_features(&self) -> usize { + self.cardinalities.len() + } + + /// Category count for a feature. + #[must_use] + pub fn cardinality(&self, feature: usize) -> u32 { + self.cardinalities[feature] + } + + /// The `[start, end)` code range owned by a feature's block. + #[must_use] + pub fn block(&self, feature: usize) -> (usize, usize) { + (self.offsets[feature], self.offsets[feature + 1]) + } + + /// The absolute code slot for an `(feature, category)` item. + #[must_use] + pub fn slot(&self, item: Item) -> usize { + self.offsets[item.feature as usize] + item.category as usize + } +} + +/// A window of rows in category-index form, sharing one [`FeatureSpec`]. +/// `rows.len()` is the `n` the evidence counts are measured over. +#[derive(Debug, Clone)] +pub struct Dataset { + /// The shared schema. + pub spec: FeatureSpec, + /// One `Vec` per row; each is one category index per feature. + pub rows: Vec>, +} + +impl Dataset { + /// Wrap rows in a dataset, validating arity against the spec. + /// + /// # Panics + /// If any row's length differs from the spec's feature count. + #[must_use] + pub fn new(spec: FeatureSpec, rows: Vec>) -> Self { + for (row_idx, r) in rows.iter().enumerate() { + assert_eq!(r.len(), spec.num_features(), "row arity mismatch"); + for (feature_idx, &category) in r.iter().enumerate() { + assert!( + category < spec.cardinality(feature_idx), + "row {row_idx} feature {feature_idx} category {category} out of range \ + (cardinality {})", + spec.cardinality(feature_idx) + ); + } + } + Self { spec, rows } + } + + /// Window size `n`. + #[must_use] + pub fn len(&self) -> usize { + self.rows.len() + } + + /// Whether the window is empty. + #[must_use] + pub fn is_empty(&self) -> bool { + self.rows.is_empty() + } + + /// Count of rows containing every item in `items` (an AND over items). + /// The one and only evidence primitive — integer, exact. + #[must_use] + pub fn count_matching(&self, items: &[Item]) -> u32 { + self.rows + .iter() + .filter(|row| { + items + .iter() + .all(|it| row[it.feature as usize] == it.category) + }) + .count() as u32 + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn offsets_and_dim() { + let spec = FeatureSpec::new(vec![2, 3, 1]); + assert_eq!(spec.dim(), 6); + assert_eq!(spec.num_features(), 3); + assert_eq!(spec.block(0), (0, 2)); + assert_eq!(spec.block(1), (2, 5)); + assert_eq!(spec.block(2), (5, 6)); + } + + #[test] + fn slot_addressing() { + let spec = FeatureSpec::new(vec![2, 3]); + assert_eq!(spec.slot(Item::new(0, 1)), 1); + assert_eq!(spec.slot(Item::new(1, 2)), 4); + } + + #[test] + fn count_matching_is_an_and_over_items() { + let spec = FeatureSpec::new(vec![2, 2]); + // rows: (0,0),(0,0),(0,1),(1,1) + let data = Dataset::new(spec, vec![vec![0, 0], vec![0, 0], vec![0, 1], vec![1, 1]]); + assert_eq!(data.count_matching(&[Item::new(0, 0)]), 3); + assert_eq!(data.count_matching(&[Item::new(0, 0), Item::new(1, 0)]), 2); + assert_eq!(data.count_matching(&[Item::new(0, 5)]), 0); // absent category + assert_eq!(data.len(), 4); + } + + #[test] + #[should_panic(expected = "out of range")] + fn out_of_range_category_is_rejected_at_construction() { + let spec = FeatureSpec::new(vec![2, 2]); + // feature 1 category 2 ≥ cardinality 2 — would silently undercount. + let _ = Dataset::new(spec, vec![vec![0, 2]]); + } +} diff --git a/crates/lance-graph-arm-discovery/src/lib.rs b/crates/lance-graph-arm-discovery/src/lib.rs new file mode 100644 index 00000000..96f8545f --- /dev/null +++ b/crates/lance-graph-arm-discovery/src/lib.rs @@ -0,0 +1,80 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! `lance-graph-arm-discovery` — the **upstream proposer** leg of the +//! streaming association-rule discovery → NARS revision → ratified SPO → +//! deterministic codegen pipeline (`streaming-arm-nars-discovery-v1.md`). +//! +//! A Rust transcoding of **Aerial+** (Karabulut, Groth, Degeler — +//! *Neurosymbolic Association Rule Mining from Tabular Data*, arXiv +//! 2504.19354v1) — with the paper's `f32` autoencoder **replaced by an +//! integer codebook distance oracle**. Aerial+'s reconstruction probe is a +//! nearest-neighbour query, and this substrate answers it exactly via the +//! palette256 distance table (`[a,b] → u32`, ρ=0.9973 vs cosine). So the +//! discovery path is **float-free and bitwise-deterministic** — no autoencoder, +//! no SGD, no seed. +//! +//! ```text +//! tabular rows ─► one-hot codebook items +//! ─► codebook probe: nearest consequents within θ (Stage A) +//! (integer CodebookDistance oracle = palette256) +//! ─► confirm on data: integer support/confidence counts +//! ─► CandidateRule (integer evidence) +//! ─► arm_to_truth_u8: (cooccur, antecedent_count) → TruthU8 (Stage B) +//! (= CausalEdge64 confidence_u8 + i4 mantissa) +//! ─► {s,p,o,f,c} ndjson ─► SPO store loader +//! ``` +//! +//! # Float discipline (why this exists) +//! +//! This substrate addresses by exact codebook CAM, never by float similarity +//! (`faiss-homology-cam-pq.md`: similarity lives in the discovery layer as an +//! *integer codebook distance*, never as float, never in addressing). The +//! discovery path here is therefore all integers: codebook distance (`u32`), +//! evidence counts (`u32`), thresholds in parts-per-million. `f32` appears +//! only at the [`translator::arm_to_nars`] / [`ndjson`] edge, because the +//! downstream `spo::truth::TruthValue` and `ruff_spo_triplet::Triple` are +//! themselves `f32`; the canonical truth is the quantised +//! [`translator::TruthU8`]. +//! +//! # No nondeterminism to fence +//! +//! The codebook probe is deterministic by construction (same data + oracle + +//! `theta` ⇒ identical rules, every target). Unlike the autoencoder it +//! replaced, it does not need to hide behind the ratification gate or stay out +//! of the compile path — it can join the deterministic trunk beside +//! pair-stats (D-ARM-3). The ratification council still governs *promotion to +//! the SPO store*, but no longer because of any nondeterminism here. + +//! # SIMD seam +//! +//! The data-confirmation count loop transposes the window into row bitsets +//! ([`bitset::RowMasks`]) so every candidate's count is an `AND` + popcount +//! over `&[u64]` ([`simd`]). The default path is scalar (`u64::count_ones`), +//! keeping the crate std-only and independently verifiable; the optional +//! `ndarray-simd` feature routes that primitive through `ndarray::simd::U64x8` +//! per `.claude/knowledge/ndarray-vertical-simd-alien-magic.md` (zero raw +//! intrinsics in this crate). The palette256 [`aerial::CodebookDistance`] +//! oracle is similarly SIMD on the consumer side (`bgz17::batch_palette_distance`). + +#![forbid(unsafe_code)] + +pub mod aerial; +pub mod bitset; +pub mod encode; +pub mod ndjson; +pub mod rule; +pub mod simd; +pub mod translator; + +pub use aerial::{ + antecedent_distance, extract_rules, AerialParams, AerialProposer, CodebookDistance, + ExtractParams, MatrixDistance, +}; +pub use bitset::RowMasks; +pub use encode::{Dataset, FeatureSpec}; +pub use rule::{CandidateRule, Item, Proposer, PPM}; +pub use translator::{ + arm_to_nars, arm_to_truth_u8, CandidateTriple, FeedProjector, NarsTruth, TruthU8, + NARS_PERSONALITY_K, +}; diff --git a/crates/lance-graph-arm-discovery/src/ndjson.rs b/crates/lance-graph-arm-discovery/src/ndjson.rs new file mode 100644 index 00000000..74a16b86 --- /dev/null +++ b/crates/lance-graph-arm-discovery/src/ndjson.rs @@ -0,0 +1,147 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! Newline-delimited JSON emit for [`CandidateTriple`]. +//! +//! The on-the-wire shape is **shape-compatible** (same `{"s","p","o","f","c"}` +//! object per line, field order `s, p, o, f, c`, trailing newline) with +//! `ruff_spo_triplet::ndjson` and with what +//! `lance_graph::graph::spo::odoo_ontology::parse_triples` reads. +//! +//! **Loader caveat (verified by review):** the lance-graph `parse_triples` +//! loader does *not* check the predicate and **accepts** the output as-is. But +//! `ruff_spo_triplet::from_ndjson` enforces a *closed predicate vocabulary* +//! and **rejects** the `implies` predicate this crate emits — so output flows +//! into the SPO store via `parse_triples` today, but NOT through the `ruff` +//! loader until `Implies` is added to that vocabulary (D-ARM-SYN-1). "Same +//! loader as the static ruff extractor" is the *goal*, gated on SYN-1; it is +//! not true on this branch. +//! +//! This is a zero-dependency hand emitter (no `serde_json`) so the crate +//! stays std-only and offline-buildable. It produces the same logical shape; +//! the downstream loader parses it with `serde_json::from_str`. + +use crate::translator::CandidateTriple; + +/// Serialise candidate triples to ndjson (one object per line, trailing +/// newline). Order is preserved as given. +#[must_use] +pub fn to_ndjson(triples: &[CandidateTriple]) -> String { + let mut out = String::new(); + for t in triples { + out.push('{'); + push_str_field(&mut out, "s", &t.s, true); + push_str_field(&mut out, "p", &t.p, false); + push_str_field(&mut out, "o", &t.o, false); + push_num_field(&mut out, "f", t.f, false); + push_num_field(&mut out, "c", t.c, false); + out.push('}'); + out.push('\n'); + } + out +} + +fn push_str_field(out: &mut String, key: &str, value: &str, first: bool) { + if !first { + out.push(','); + } + out.push('"'); + out.push_str(key); + out.push_str("\":\""); + escape_into(out, value); + out.push('"'); +} + +fn push_num_field(out: &mut String, key: &str, value: f32, first: bool) { + if !first { + out.push(','); + } + out.push('"'); + out.push_str(key); + out.push_str("\":"); + out.push_str(&fmt_f32(value)); +} + +/// Format an `f32` as a JSON-valid number, always with a fractional part so +/// the shape mirrors `serde_json`'s float rendering (`1` → `1.0`). +/// Non-finite values are coerced to `0.0` (defensive; truths are clamped to +/// `[0, 1]` upstream, so this never fires in practice). +fn fmt_f32(value: f32) -> String { + if !value.is_finite() { + return "0.0".to_string(); + } + let s = format!("{value}"); + if s.contains('.') || s.contains('e') || s.contains('E') { + s + } else { + format!("{s}.0") + } +} + +/// Minimal JSON string escaping for the characters that can appear in IRIs. +fn escape_into(out: &mut String, value: &str) { + for ch in value.chars() { + match ch { + '"' => out.push_str("\\\""), + '\\' => out.push_str("\\\\"), + '\n' => out.push_str("\\n"), + '\r' => out.push_str("\\r"), + '\t' => out.push_str("\\t"), + c if (c as u32) < 0x20 => out.push_str(&format!("\\u{:04x}", c as u32)), + c => out.push(c), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn triple(s: &str, p: &str, o: &str, f: f32, c: f32) -> CandidateTriple { + CandidateTriple { + s: s.to_string(), + p: p.to_string(), + o: o.to_string(), + f, + c, + } + } + + #[test] + fn line_shape_matches_spo_loader_contract() { + let t = triple("arm:feat0=cat1", "implies", "arm:feat1=cat0", 0.9, 0.95); + let text = to_ndjson(&[t]); + // exact field order s, p, o, f, c — the shape parse_triples reads + assert_eq!( + text, + "{\"s\":\"arm:feat0=cat1\",\"p\":\"implies\",\"o\":\"arm:feat1=cat0\",\"f\":0.9,\"c\":0.95}\n" + ); + } + + #[test] + fn integral_truth_renders_with_fraction() { + // 1.0 must render as "1.0", not "1" — mirrors serde_json float output. + let t = triple("a", "implies", "b", 1.0, 1.0); + let text = to_ndjson(&[t]); + assert!(text.contains("\"f\":1.0"), "got: {text}"); + assert!(text.contains("\"c\":1.0"), "got: {text}"); + } + + #[test] + fn each_triple_is_one_line_with_trailing_newline() { + let text = to_ndjson(&[ + triple("a", "implies", "b", 0.5, 0.5), + triple("c", "implies", "d", 0.6, 0.6), + ]); + assert_eq!(text.lines().count(), 2); + assert!(text.ends_with('\n')); + } + + #[test] + fn quotes_and_backslashes_are_escaped() { + let t = triple("a\"x", "implies", "b\\y", 0.5, 0.5); + let text = to_ndjson(&[t]); + assert!(text.contains("a\\\"x"), "quote escaped: {text}"); + assert!(text.contains("b\\\\y"), "backslash escaped: {text}"); + } +} diff --git a/crates/lance-graph-arm-discovery/src/rule.rs b/crates/lance-graph-arm-discovery/src/rule.rs new file mode 100644 index 00000000..c3733dc2 --- /dev/null +++ b/crates/lance-graph-arm-discovery/src/rule.rs @@ -0,0 +1,179 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! The proposer output carrier — `CandidateRule` and the `Proposer` trait. +//! +//! **Float-free.** A rule carries integer evidence counts, not `f32` +//! statistics. Support and confidence are rationals (`count / count`); they +//! are compared as integers (parts-per-million, cross-multiplied) so the +//! discovery decision path never touches a float. `f32` appears only at the +//! explicit edge accessors that feed the downstream `f32` contracts +//! (`spo::truth::TruthValue`, `ruff_spo_triplet::Triple`), and the canonical +//! truth wire is the quantized `CausalEdge64` form — see +//! [`crate::translator::arm_to_truth_u8`]. +//! +//! This is the shape every Stage-A proposer (the codebook-probe Aerial+ +//! backend, the pair-stats trunk) emits. It is the **local mirror** of the +//! planned `lance-graph-contract::CandidateRule` (D-ARM-2) — *not yet +//! field-frozen*; adopt by `pub use` re-export when the contract carrier +//! lands (the determinism firewall forbids depending on `lance-graph`, NOT on +//! the zero-dep `lance-graph-contract`). Until then this is the migration +//! seam, not the canonical type (TD-ARM-CARRIER-FORK). + +/// One `(feature, category)` atom — the unit an antecedent / consequent is +/// built from. `feature` indexes a [`crate::FeatureSpec`] column; `category` +/// indexes a value within it. Both are small integers — a codebook code, not +/// a vector. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct Item { + /// Feature (column) index into the [`crate::FeatureSpec`]. + pub feature: u32, + /// Category (value) index within the feature. + pub category: u32, +} + +impl Item { + /// Construct an item. + #[must_use] + pub const fn new(feature: u32, category: u32) -> Self { + Self { feature, category } + } +} + +/// Fixed-point scale for support/confidence comparison (parts per million). +/// A threshold of "1%" is `10_000`; "50%" is `500_000`. Integer throughout. +pub const PPM: u64 = 1_000_000; + +/// A mined association rule `antecedent → consequent` with **integer** +/// ARM evidence. +/// +/// Mirrors the planned `lance-graph-contract::CandidateRule` (D-ARM-2) but +/// does not yet match it field-for-field (D-ARM-2 pairs the rule with +/// `WindowMetadata`; this carries a bare `window`). The classical ARM +/// quantities (paper §2) are derived from the three counts: +/// +/// - `support = cooccur / window` +/// - `confidence = cooccur / antecedent_count` (= `P(consequent | antecedent)`) +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct CandidateRule { + /// Left-hand side items (all must hold). Sorted, at most one per feature. + pub antecedent: Vec, + /// Right-hand side items. The codebook probe emits exactly one; the `Vec` + /// keeps shape-parity with the planned contract. + pub consequent: Vec, + /// `|{rows ⊇ antecedent ∪ consequent}|` — the co-occurrence count, and the + /// NARS evidential mass `m`. + pub cooccur: u32, + /// `|{rows ⊇ antecedent}|` — the denominator of confidence. + pub antecedent_count: u32, + /// Window size `n` the counts were measured over. + pub window: u32, +} + +impl CandidateRule { + /// Evidential mass `m = cooccur` — the count of rows supporting the rule. + /// Drives the NARS confidence term `c = m / (m + k)`. Integer. + #[must_use] + pub fn evidence(&self) -> u32 { + self.cooccur + } + + /// Support in parts-per-million: `cooccur · 1e6 / window`. Integer. + #[must_use] + pub fn support_ppm(&self) -> u32 { + if self.window == 0 { + return 0; + } + ((self.cooccur as u64 * PPM) / self.window as u64) as u32 + } + + /// Confidence in parts-per-million: `cooccur · 1e6 / antecedent_count`. + /// Integer. Returns 0 when the antecedent never occurs. + #[must_use] + pub fn confidence_ppm(&self) -> u32 { + if self.antecedent_count == 0 { + return 0; + } + ((self.cooccur as u64 * PPM) / self.antecedent_count as u64) as u32 + } + + /// The classical ARM gate, compared entirely in integers (ppm). + /// The Jirak-bound significance floor (`I-NOISE-FLOOR-JIRAK`, D-ARM-7) is a + /// separate, stricter, still-unimplemented gate — see ISSUE + /// ARM-JIRAK-FLOOR; do not wire this proposer to a live `SpoStore` until + /// D-ARM-7 lands. + #[must_use] + pub fn passes(&self, min_support_ppm: u32, min_confidence_ppm: u32) -> bool { + self.support_ppm() >= min_support_ppm && self.confidence_ppm() >= min_confidence_ppm + } + + /// `f32` support — **edge convenience only**, for the downstream `f32` + /// `TruthValue`/`Triple` contracts. Not used in any decision. + #[must_use] + pub fn support_f32(&self) -> f32 { + self.support_ppm() as f32 / PPM as f32 + } + + /// `f32` confidence — edge convenience only (see [`Self::support_f32`]). + #[must_use] + pub fn confidence_f32(&self) -> f32 { + self.confidence_ppm() as f32 / PPM as f32 + } +} + +/// A Stage-A proposer: something that emits a batch of candidate rules. +/// Mirrors the planned `lance-graph-contract::Proposer` (D-ARM-2). +pub trait Proposer { + /// Produce the next batch of candidate rules. An empty `Vec` signals the + /// proposer is exhausted for now. + fn next_batch(&mut self) -> Vec; +} + +#[cfg(test)] +mod tests { + use super::*; + + fn rule(cooccur: u32, antecedent_count: u32, window: u32) -> CandidateRule { + CandidateRule { + antecedent: vec![Item::new(0, 1)], + consequent: vec![Item::new(1, 0)], + cooccur, + antecedent_count, + window, + } + } + + #[test] + fn evidence_is_the_cooccur_count() { + assert_eq!(rule(250, 300, 1000).evidence(), 250); + } + + #[test] + fn support_and_confidence_are_integer_ppm() { + let r = rule(250, 300, 1000); + assert_eq!(r.support_ppm(), 250_000); // 25% + assert_eq!(r.confidence_ppm(), 833_333); // 250/300 ≈ 0.8333 + } + + #[test] + fn passes_gates_on_both_floors_in_ppm() { + let r = rule(100, 125, 2000); // support 5%, confidence 80% + assert!(r.passes(10_000, 500_000)); + assert!(!r.passes(100_000, 500_000), "support floor not met"); + assert!(!r.passes(10_000, 900_000), "confidence floor not met"); + } + + #[test] + fn zero_antecedent_is_zero_confidence_not_a_panic() { + let r = rule(0, 0, 1000); + assert_eq!(r.confidence_ppm(), 0); + assert!(!r.passes(0, 1)); + } + + #[test] + fn item_ordering_is_feature_then_category() { + let mut items = vec![Item::new(1, 0), Item::new(0, 2), Item::new(0, 1)]; + items.sort(); + assert_eq!(items, vec![Item::new(0, 1), Item::new(0, 2), Item::new(1, 0)]); + } +} diff --git a/crates/lance-graph-arm-discovery/src/simd.rs b/crates/lance-graph-arm-discovery/src/simd.rs new file mode 100644 index 00000000..445d7a55 --- /dev/null +++ b/crates/lance-graph-arm-discovery/src/simd.rs @@ -0,0 +1,98 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! Bitset population-count primitives — the SIMD seam. +//! +//! The data-confirmation hot loop (counting rows that satisfy a conjunction of +//! items) is the `faiss-homology` "SIMD batch-AND over the SoA facet column" +//! workload. With the row-bitset SoA ([`crate::bitset::RowMasks`]) it reduces +//! to `AND` + popcount over `&[u64]`. +//! +//! Per `.claude/knowledge/ndarray-vertical-simd-alien-magic.md` (MANDATORY for +//! consumer SIMD), this routes through **`ndarray::simd::U64x8`** — zero raw +//! intrinsics, zero `cfg(target_arch)`, zero feature detection in this crate; +//! the polyfill owns dispatch. The scalar path is the default (so the crate +//! stays std-only and independently verifiable); the `ndarray-simd` feature +//! swaps in the vectorised path. + +/// Population count of a bitset (`Σ count_ones`). +#[cfg(not(feature = "ndarray-simd"))] +#[must_use] +pub fn popcount(words: &[u64]) -> u32 { + words.iter().map(|w| w.count_ones()).sum() +} + +/// Population count of `a AND b` (the co-occurrence count of two item masks). +#[cfg(not(feature = "ndarray-simd"))] +#[must_use] +pub fn and_popcount(a: &[u64], b: &[u64]) -> u32 { + a.iter().zip(b).map(|(x, y)| (x & y).count_ones()).sum() +} + +// ── ndarray::simd vectorised path (feature `ndarray-simd`) ────────────────── +// +// Uses ONLY `ndarray::simd::U64x8` (`from_slice` + `&` (BitAnd) + `popcnt` + +// `to_array`) — the canonical W1b consumer surface, zero raw intrinsics. +// Verified: `cargo test/clippy --features ndarray-simd` → 33/33, clippy clean. +// The real AVX-512 VPOPCNTQ / AMX kernels need `-C target-cpu=native` or +// `x86-64-v4`; otherwise this is ndarray's scalar polyfill (correct, not +// accelerated). + +/// Population count of a bitset, via `ndarray::simd::U64x8::popcnt`. +#[cfg(feature = "ndarray-simd")] +#[must_use] +pub fn popcount(words: &[u64]) -> u32 { + use ndarray::simd::U64x8; + let chunks = words.len() / 8; + let mut acc = 0u64; + for i in 0..chunks { + acc += U64x8::from_slice(&words[i * 8..i * 8 + 8]) + .popcnt() + .to_array() + .iter() + .sum::(); + } + for &w in &words[chunks * 8..] { + acc += u64::from(w.count_ones()); + } + acc as u32 +} + +/// Population count of `a AND b`, via `ndarray::simd::U64x8`. +#[cfg(feature = "ndarray-simd")] +#[must_use] +pub fn and_popcount(a: &[u64], b: &[u64]) -> u32 { + use ndarray::simd::U64x8; + let chunks = a.len() / 8; + let mut acc = 0u64; + for i in 0..chunks { + let va = U64x8::from_slice(&a[i * 8..i * 8 + 8]); + let vb = U64x8::from_slice(&b[i * 8..i * 8 + 8]); + acc += (va & vb).popcnt().to_array().iter().sum::(); + } + for i in (chunks * 8)..a.len() { + acc += u64::from((a[i] & b[i]).count_ones()); + } + acc as u32 +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn popcount_counts_set_bits() { + assert_eq!(popcount(&[0u64]), 0); + assert_eq!(popcount(&[u64::MAX]), 64); + assert_eq!(popcount(&[0b1011, 0b11]), 5); // 3 + 2 set bits + // exercise the 8-word vector chunk + tail + assert_eq!(popcount(&[u64::MAX; 9]), 64 * 9); + } + + #[test] + fn and_popcount_is_conjunction_count() { + assert_eq!(and_popcount(&[0b1100], &[0b1010]), 1); // only bit 3 shared + assert_eq!(and_popcount(&[u64::MAX; 9], &[u64::MAX; 9]), 64 * 9); + assert_eq!(and_popcount(&[u64::MAX; 9], &[0u64; 9]), 0); + } +} diff --git a/crates/lance-graph-arm-discovery/src/translator.rs b/crates/lance-graph-arm-discovery/src/translator.rs new file mode 100644 index 00000000..1ffe803d --- /dev/null +++ b/crates/lance-graph-arm-discovery/src/translator.rs @@ -0,0 +1,241 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright The Lance Authors + +//! Stage B — translate a [`CandidateRule`]'s integer evidence into NARS truth +//! + an SPO triple carrier. +//! +//! The mapping is verbatim from Aerial+ §2/§3.3, computed in **integers**: +//! +//! - ARM **confidence** = `cooccur / antecedent_count` → NARS **frequency** +//! - ARM **evidence** `m = cooccur` → NARS **confidence** `c = m / (m + k)` +//! +//! The canonical truth is [`TruthU8`] — `frequency`/`confidence` as `u8`, +//! which is exactly the `CausalEdge64` wire (`confidence_u8` + i4 mantissa), +//! float-free. [`NarsTruth`] / [`arm_to_nars`] are a thin `f32` **edge** that +//! exists only because the downstream `spo::truth::TruthValue` and +//! `ruff_spo_triplet::Triple` are themselves `f32`; nothing in the discovery +//! path consumes the `f32` form. + +use crate::rule::{CandidateRule, Item}; + +/// NAL-9 default personality constant (integer). Larger `k` ⇒ more evidence +/// needed before confidence approaches saturation. +pub const NARS_PERSONALITY_K: u32 = 1; + +/// Quantised NARS truth — the canonical, float-free wire form (mirrors the +/// `CausalEdge64` `confidence_u8` + i4 mantissa fields). `255` = 1.0. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct TruthU8 { + /// NARS frequency, `0..=255` (255 = 1.0). + pub frequency: u8, + /// NARS confidence, `0..=255`. + pub confidence: u8, +} + +/// Translate a candidate rule's integer evidence into quantised NARS truth. +/// `k` is the NARS personality constant ([`NARS_PERSONALITY_K`] default). +#[must_use] +pub fn arm_to_truth_u8(rule: &CandidateRule, k: u32) -> TruthU8 { + assert!( + k > 0, + "NARS personality constant k must be > 0; k=0 makes any cooccur>0 dogmatic (confidence=1.0)" + ); + // frequency = P(Y|X) = cooccur / antecedent_count + let frequency = if rule.antecedent_count == 0 { + 128 // unknown ≈ 0.5 + } else { + ((rule.cooccur as u64 * 255) / rule.antecedent_count as u64).min(255) as u8 + }; + // confidence = m / (m + k), m = cooccur (integer evidential mass); k>0 ⇒ denom>0 + let m = rule.cooccur as u64; + let confidence = ((m * 255) / (m + k as u64)) as u8; + TruthU8 { + frequency, + confidence, + } +} + +/// An `f32` NARS truth — **edge convenience only** (see module docs). Derived +/// from [`TruthU8`]; never read inside the discovery path. +#[derive(Debug, Clone, Copy, PartialEq)] +pub struct NarsTruth { + /// NARS frequency `f ∈ [0, 1]`. + pub frequency: f32, + /// NARS confidence `c ∈ [0, 1)`. + pub confidence: f32, +} + +impl NarsTruth { + /// NARS expectation `e = c·(f − 0.5) + 0.5` — matches + /// `spo::truth::TruthValue::expectation`. + #[must_use] + pub fn expectation(&self) -> f32 { + self.confidence * (self.frequency - 0.5) + 0.5 + } +} + +/// `f32` edge of [`arm_to_truth_u8`], for the downstream `f32` `TruthValue` / +/// `Triple` contracts. +#[must_use] +pub fn arm_to_nars(rule: &CandidateRule, k: u32) -> NarsTruth { + let t = arm_to_truth_u8(rule, k); + NarsTruth { + frequency: t.frequency as f32 / 255.0, + confidence: t.confidence as f32 / 255.0, + } +} + +/// Projects rule items into SPO IRIs for a particular feed/domain. +pub trait FeedProjector { + /// IRI for the subject built from the antecedent items. + fn subject(&self, antecedent: &[Item]) -> String; + /// The predicate IRI for a discovered implication. + fn predicate(&self) -> String; + /// IRI for the object built from the consequent items. + fn object(&self, consequent: &[Item]) -> String; +} + +/// One SPO triple with NARS truth — shape-compatible with +/// `ruff_spo_triplet::Triple` and the SPO store's ndjson loader (same +/// `{s,p,o,f,c}` fields; see [`crate::ndjson`] for the predicate-vocabulary +/// caveat that gates the `ruff` loader on D-ARM-SYN-1). The `f`/`c` floats are +/// the *serialization format* of that downstream contract, derived from the +/// canonical integer [`TruthU8`] at the wire edge. +#[derive(Debug, Clone, PartialEq)] +pub struct CandidateTriple { + /// Subject IRI. + pub s: String, + /// Predicate IRI. + pub p: String, + /// Object IRI. + pub o: String, + /// NARS frequency (serialization edge). + pub f: f32, + /// NARS confidence (serialization edge). + pub c: f32, +} + +impl CandidateTriple { + /// Build a triple from a candidate rule, a projector, and a NARS `k`. + #[must_use] + pub fn from_rule(rule: &CandidateRule, projector: &dyn FeedProjector, k: u32) -> Self { + let truth = arm_to_nars(rule, k); + Self { + s: projector.subject(&rule.antecedent), + p: projector.predicate(), + o: projector.object(&rule.consequent), + f: truth.frequency, + c: truth.confidence, + } + } +} + +/// A minimal projector rendering items as `feat=cat`, joined with `&`. +#[derive(Debug, Clone)] +pub struct DebugProjector { + /// Predicate IRI to stamp on every emitted implication. + pub predicate: String, +} + +impl Default for DebugProjector { + fn default() -> Self { + Self { + // Not in `ruff_spo_triplet`'s current closed vocabulary — see the + // synergy doc: the `ruff` loader needs an `implies` predicate added + // (D-ARM-SYN-1) before ARM rules flow through it. + predicate: "implies".to_string(), + } + } +} + +fn render_items(items: &[Item]) -> String { + let mut parts: Vec = items.to_vec(); + parts.sort(); + parts + .iter() + .map(|it| format!("feat{}=cat{}", it.feature, it.category)) + .collect::>() + .join("&") +} + +impl FeedProjector for DebugProjector { + fn subject(&self, antecedent: &[Item]) -> String { + format!("arm:{}", render_items(antecedent)) + } + fn predicate(&self) -> String { + self.predicate.clone() + } + fn object(&self, consequent: &[Item]) -> String { + format!("arm:{}", render_items(consequent)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn rule(cooccur: u32, antecedent_count: u32, window: u32) -> CandidateRule { + CandidateRule { + antecedent: vec![Item::new(0, 1)], + consequent: vec![Item::new(1, 0)], + cooccur, + antecedent_count, + window, + } + } + + #[test] + fn frequency_is_quantised_arm_confidence() { + // confidence = 249/300 ≈ 0.83 → 0.83×255 ≈ 211 + let t = arm_to_truth_u8(&rule(249, 300, 1000), NARS_PERSONALITY_K); + assert_eq!(t.frequency, ((249u64 * 255) / 300) as u8); + } + + #[test] + fn confidence_grows_with_evidence_integer() { + // m=200, k=1 → 200·255/201 = 253 + let strong = arm_to_truth_u8(&rule(200, 240, 1000), NARS_PERSONALITY_K); + // m=2, k=1 → 2·255/3 = 170 + let weak = arm_to_truth_u8(&rule(2, 3, 10), NARS_PERSONALITY_K); + assert!(strong.confidence > weak.confidence); + assert_eq!(strong.confidence, ((200u64 * 255) / 201) as u8); + assert_eq!(weak.confidence, ((2u64 * 255) / 3) as u8); + } + + #[test] + fn larger_k_demands_more_evidence_integer() { + let k1 = arm_to_truth_u8(&rule(5, 10, 100), 1).confidence; + let k10 = arm_to_truth_u8(&rule(5, 10, 100), 10).confidence; + assert!(k10 < k1); + } + + #[test] + fn zero_antecedent_is_unknown_frequency() { + let t = arm_to_truth_u8(&rule(0, 0, 1000), NARS_PERSONALITY_K); + assert_eq!(t.frequency, 128); + assert_eq!(t.confidence, 0); + } + + #[test] + fn f32_edge_is_derived_from_u8() { + let r = rule(100, 125, 400); // m=100; conf_u8 = 100·255/101 = 252 + let t8 = arm_to_truth_u8(&r, NARS_PERSONALITY_K); + let tf = arm_to_nars(&r, NARS_PERSONALITY_K); + assert!((tf.confidence - t8.confidence as f32 / 255.0).abs() < 1e-6); + assert!((tf.frequency - t8.frequency as f32 / 255.0).abs() < 1e-6); + } + + #[test] + fn triple_projection() { + let t = CandidateTriple::from_rule(&rule(90, 100, 400), &DebugProjector::default(), NARS_PERSONALITY_K); + assert_eq!(t.s, "arm:feat0=cat1"); + assert_eq!(t.p, "implies"); + assert_eq!(t.o, "arm:feat1=cat0"); + } + + #[test] + #[should_panic(expected = "k must be > 0")] + fn k_zero_is_rejected_not_made_dogmatic() { + let _ = arm_to_truth_u8(&rule(5, 10, 100), 0); + } +}