From 9f2f8783d82b114d53466b1cc531be8e42d20fba Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 17 Jun 2026 20:58:50 +0000 Subject: [PATCH 1/5] board: post-merge hygiene for #521 (PR_ARC + LATEST_STATE) Records the merged #521 (lance-graph-contract C++ codegen target MethodSig + UniCharSet content store) per the Mandatory Board-Hygiene Rule's post-merge step. PR_ARC_INVENTORY prepend (Added/Locked/Deferred/Docs/Confidence) + LATEST_STATE narrative entry + "Recently Shipped PRs" table row. Captures the PROBE-OGAR-ADAPTER-UNICHARSET FINDING: the full transcode pipeline (ruff ruff_cpp_spo harvest -> reassemble -> ruff_cpp_codegen -> these contract types) produces a UniCharSet byte-identical 112/112 to the libtesseract oracle on real eng data, proving the core-first transcode doctrine end-to-end. Pairs with ruff #20. Merge commit 620bd8e. Co-Authored-By: Claude https://claude.ai/code/session_016b33swuXE23hKtqxsHu9p1 --- .claude/board/LATEST_STATE.md | 5 ++++- .claude/board/PR_ARC_INVENTORY.md | 29 +++++++++++++++++++++++++++++ 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index e6d27aa3..9db941c2 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -11,7 +11,9 @@ --- > **2026-06-17 — IN PR (branch `claude/odoo-spo-fk-target-deep-reads`)** — Odoo SPO corpus enrichment (odoo-rs `UPSTREAM_WISHLIST` P1 + coupled P0). The corpus `crates/lance-graph/src/graph/spo/odoo_ontology.spo.ndjson` now carries **two new predicate families** (was 7 predicates: `depends_on / emitted_by / has_function / raises / rdf:type / reads_field / traverses_relation`): **`target`** (618) + **`inverse_name`** (102) — the relational comodel/inverse keyed by the relation IRI, ruff#18 sibling-triple shape `(odoo:account_move.line_ids, target, "account.move.line")`; and **+736 deep `reads_field`** (so `reads_field` 2 095 → 2 831) — each `@api.depends('rel.leaf', …)` resolved through the new target map and lifted onto the field's emitting method as a transitive read. Corpus 22 245 → **23 701** triples. New stdlib-only generator `tools/odoo-blueprint-extractor/odoo_blueprint_extractor/spo_enrich.py` (+14 unit tests) reads `/home/user/odoo/addons` (the same source the ORM extractor parses) to build the `(model, field) → (comodel, inverse)` map; additive, deterministic, idempotent. `odoo_ontology.rs` doc + tests updated (count 23 701, histogram incl. new predicates, 2 new enrichment tests); `action_emitter`/`spo` unaffected (function count 3 328 unchanged). **Cross-repo finding (verified, not faked):** the deep reads make the cross-model recompute-ordering edge `account_move_line._compute_amount_residual → account_move._compute_amount` *visible* to `od_ontology::RecomputeDag` (baseline: 0 cross-model compute edges → enriched: 27), delivering the wishlist's P0 ask — but the audit's MISSED-1 is a unidirectional *ordering edge*, NOT a cycle, so odoo-rs's `slice_2_compute_subset_no_cross_model_cycle` no-cycle assertion legitimately still holds (the "circularity" is semantic, not a `reads_field`↔`emitted_by` back-edge). The corpus's original generator (`emit_ontology2.py`/`methods.parquet`) is absent from the tree — only its output is committed; enrichment runs at the correct additive stage over the shipped corpus + present source. See `EPIPHANIES.md` E-ODOO-FK-DEEP-READS. - +> +> **2026-06-17 — MERGED #521** (lance-graph-contract: C++ codegen target `MethodSig` + `UniCharSet` content store): **+940/-4 across 8 files, additive to `lance-graph-contract` only** (zero `NodeRow`/`ValueTenant`/`ValueSchema`/stride/`ENVELOPE_LAYOUT_VERSION` impact). The Core-side of the Tesseract C++→Rust transcode. **`codegen_manifest`** — `MethodSig` (the `&'static`-backed, `const`-constructible method-signature type the generated Rust names; the method-axis sibling of `ClassView`'s field projection, distinct because `ClassView::FieldRef` is `String`-backed and can't be `const`) + `ClassMethods` + `methods_for`. **`unicharset`** — `UniCharSet` (deepnsm::Vocabulary-shaped id↔unichar bijection), `.unicharset` parser, `id_to_unichar`/`unichar_to_id`/`dump`, **zero leptonica** (pure text, never touches `Pix`). **PROBE-OGAR-ADAPTER-UNICHARSET → FINDING:** the pipeline (ruff `ruff_cpp_spo` harvest → `reassemble` → `ruff_cpp_codegen` → these types) produces a `UniCharSet` **byte-identical 112/112** to the C++ libtesseract oracle on real `eng` data — the core-first transcode doctrine is now empirically proven end-to-end (doctrine flipped CONJECTURE→FINDING). The sole id-0 diff was the `NULL`→space convention (`unicharset.cpp:882`), fixed + locked by `null_token_maps_to_space` (codex P1 flagged it independently; resolved + thread closed). Pairs with **ruff #20** (harvester + codegen side, merged same day). 644 contract lib green; clippy `-D warnings` + fmt clean. Branch `claude/happy-hamilton-0azlw4`, merge `620bd8e`. **Next honest increment:** wire `UniCharSet` lookups through `classid → ClassView → UnifiedStep` (the `classid → &UniCharSet` resolver), per-leaf one-`diff` parity. +> > **2026-06-17 — IN PR (branch `claude/bindspace-mailbox-soa-w3-w4a`)** — W3+W4a atomic read/write shim, the first behaviour-touching step of the BindSpace→MailboxSoA migration. New `cognitive-shader-driver::backing` module (`pub(crate)`): `BackingStore<'a>` (read) + `BackingStoreWrite<'a>` (write) — an enum with a `Singleton(&BindSpace)` arm (live default) and a `#[cfg(feature = "mailbox-thoughtspace")] Mailbox(&MailboxSoA<1024>)` arm. New Cargo feature **`mailbox-thoughtspace`** — **default-OFF, NOT in `lab`**; production stays singleton-read+write until W7. `driver.run()` keeps ONE body: all six dispatch reads (meta_prefilter / qualia17d / content_row / edge / entity_type / len) re-pointed through a `self.backing()` selector (`const DEFAULT_MAILBOX: MailboxId = 0`, `debug_assert!(mailboxes.len() <= 1)`, singleton fallback when no mailbox registered); `ontology()` stays on the singleton (re-home is W4b). Gates: **W2 differential** (`tests/w2_differential.rs`, 4 tests) asserts the WHOLE `ShaderCrystal` bit-identical (`f32::to_bits()`) across both arms incl. a non-zero-window case + non-vacuity; firewall CI lint (`tests/firewall.rs`) bars the two `CausalEdge64` twins (`ndarray::hpc::causal_diff` / `thinking_engine::layered`) from `src/`; field-isolation matrix + cycle-drop footprint (~6 KB/row vs ~71.6 KB) in `mailbox_soa.rs`. `unbind_busdto` C5 downgrade: cycle-plane index recovery feature-gated OUT under `mailbox-thoughtspace` (cycle plane never migrated — D-DIST-5), headline survives via `qualia[9]`; singleton build keeps bit-exact recovery. Tests: default **97 lib + 2 firewall + 2 e2e**; feature-on **98 lib + 2 firewall + 2 e2e + 4 w2**; clippy `--all-targets` (both cfgs) + fmt clean on touched files. **Pre-existing P0 surfaced (NOT introduced, NOT fixed here):** the `with-engine` build does not compile on `main`/HEAD (`engine_bridge.rs:259` uses `QUALIA_DIMS` without importing it); the busdto round-trip tests have never run, and (separately) the D-CSV-5b i4-qualia cutover breaks the `codebook_index` round-trip (stored in i4 `qualia[9]`, ±7 range, cannot hold a u16). Left untouched to keep scope to W3+W4a — flagged for operator. Plan: `.claude/plans/bindspace-mailbox-soa-w3-w4a-impl-v1.md`. > > **2026-06-16 — MERGED #512** (perturbation-sim review fixes + **core-first transcode doctrine**): +591/-5 across 11 files. **Code fixes (review of #511):** `examples/calibrate.rs` divide-by-zero guard on degenerate grid; `src/hhtl.rs::basin_lambda2` `assert_eq!(keys.len(), grid.n, …)` precondition (silent corruption→loud panic); `TECH_DEBT.md` MD018 reflow. **Doctrine (the structural delivery):** new mandatory-read `core-first-transcode-doctrine.md` (218 LOC) + 3 new agent cards (`core-first-architect`, `core-gap-auditor`, `adapter-shaper`) + `BOOT.md`/`README.md` wires + EPIPHANIES entry + CLAUDE.md (+21 LOC, doctrine wire-up — NEW content unread by this session). Likely directly aligned with the ontology-first stance the operator locked on odoo-rs. Branch `claude/happy-hamilton-0azlw4`, merge `1e23c410`. 75 lib tests + clippy + fmt clean. @@ -50,6 +52,7 @@ | PR | Merged | Title | What it added | |---|---|---|---| +| **#521** | 2026-06-17 | lance-graph-contract: C++ codegen target (`MethodSig`) + `UniCharSet` content store | **+940/-4 across 8 files, additive to `lance-graph-contract` only.** The Core-side of the Tesseract C++→Rust transcode. **`codegen_manifest`** `MethodSig` (`&'static`/`const`-constructible method-signature type the generated Rust names; method-axis sibling of `ClassView`'s field projection) + `ClassMethods` + `methods_for`. **`unicharset`** `UniCharSet` (deepnsm::Vocabulary-shaped id↔unichar bijection) + `.unicharset` parser + `dump()` — **zero leptonica** (pure text). **PROBE-OGAR-ADAPTER-UNICHARSET → FINDING:** the full pipeline produces a `UniCharSet` **byte-identical 112/112** to the libtesseract oracle on real `eng` data; core-first doctrine proven end-to-end. `NULL`→space (`unicharset.cpp:882`) was the sole id-0 diff, locked by `null_token_maps_to_space` (codex P1 independently flagged + resolved). Pairs with **ruff #20**. 644 contract lib green; clippy `-D warnings` + fmt clean. Branch `claude/happy-hamilton-0azlw4`, merge `620bd8e`. | | **#512** | 2026-06-16 | perturbation-sim review fixes + **core-first transcode doctrine** + 3 new agent cards | **+591/-5 across 11 files**. Fixes from #511 review: `examples/calibrate.rs` divide-by-zero guard on degenerate grid; `src/hhtl.rs::basin_lambda2` `assert_eq!(keys.len(), grid.n, …)` precondition (silent-corruption → loud panic); `TECH_DEBT.md` MD018 reflow. **The structural delivery:** new mandatory-read `core-first-transcode-doctrine.md` (218 LOC) + 3 specialist agents (`core-first-architect`, `core-gap-auditor`, `adapter-shaper`) + EPIPHANIES entry + CLAUDE.md (+21 LOC) wire-up. Likely aligned with the ontology-first / codegen-as-cut-tail doctrine the operator just locked on odoo-rs. 75 lib tests + clippy `-D warnings` + fmt clean. Branch `claude/happy-hamilton-0azlw4`, merge `1e23c410`. | | **#513** | 2026-06-16 | perturbation-sim: inertia §0 promotion gate + CAKES/CHAODA + witness standing-wave + H ingest | **+1009/-2 across 10 files**, disjoint from #512 by design. **§0 promotion gate** for `inertia_buffer`: `GuardrailVerdict::RatifiedReuse` — takes `ResidueEdge INERTIA_SLOT = 5`, reuses existing tenant, invents no axis → **passes by reuse, not waiver**. **Probe 1 CAKES + CHAODA-lite** over HHTL basins (`CHAODA_FLAG=0.75` mirrors ndarray::clam; example flags brittle block 1.1.0 score 1.000). **Probe 2 witness arc as standing wave** — Parseval proves `particle == wave` to 0.00e0; `witness_from_spectrum` is the O(N)-per-arc read-many amortization. **Probe 3** per-bus inertia (H) ingest path. Does NOT touch `canonical_node`. Branch `claude/perturbation-sim-inertia-clam`, merge `8a3e335b`. | | **#511** | 2026-06-16 | perturbation-sim: substrate calibration (study as ground truth) + calibrated SoA member spec | **+886/-0 additive across 9 files** — `examples/calibrate.rs` (new, 318 LOC, ICC/Spearman/Pearson/Cronbach battery), `src/columns.rs` (new, 177 LOC, **spec only**), `src/hhtl.rs` (new, 175 LOC), `examples/hhtl_grid.rs` (new, 81 LOC), `CLAM_CHAODA_FRAMING.md` (new, 75 LOC). Calibrates the SoA value tenants against perturbation-sim's deterministic study as ground truth: **all 5 contingency factors certify by VALUE at 2-bit linear** (ICC ≥ 0.96), the §10 "statistics survive the encoding" claim **CONFIRMED**; α preserved within Δ ≤ 0.02 at ≥4-bit; read ≥6-bit for cross-axis orthogonality. The **one additive member** named: **`inertia_buffer`** — orthogonal to topology per PR #509's `Spearman(λ₂, buffer) ≈ 0`, spec only, promotion gated by §0 guardrail. Self-correction: two prior guesses on `d_lambda2`'s ICC=0 (heavy-tail / near-constant) **retracted** — it was a variance-guard underflow at ~1e-7. Significance per Jirak `n^(p/2−1)`. Does **NOT** touch `canonical_node` (operator-locked). Branch `claude/perturbation-sim-calibrate-soa`, merge commit `c3dddfc9`. | diff --git a/.claude/board/PR_ARC_INVENTORY.md b/.claude/board/PR_ARC_INVENTORY.md index 8bb14b95..8c4b307e 100644 --- a/.claude/board/PR_ARC_INVENTORY.md +++ b/.claude/board/PR_ARC_INVENTORY.md @@ -35,6 +35,35 @@ --- +## #521 lance-graph-contract: C++ codegen target (`MethodSig`) + `UniCharSet` content store + +**Status:** MERGED 2026-06-17 20:53 UTC (merge commit `620bd8e`), branch `claude/happy-hamilton-0azlw4`. **+940/-4 across 8 files, 9 commits.** The Core-side of the Tesseract C++→Rust transcode — the contract types ruff's `ruff_cpp_codegen` targets, plus the byte-parity probe's Rust side. **Additive to `lance-graph-contract` only** (container-architect ADDITIVE-CONFIRMED: zero `NodeRow`/`ValueTenant`/`ValueSchema`/stride/`ENVELOPE_LAYOUT_VERSION` impact). Pairs with **ruff #20** (the harvester + codegen that produces what these types consume; merged 2026-06-17 19:38 into ruff `main`). + +**Added (code):** +- **`codegen_manifest` module** — `MethodSig`: the `&'static`-backed, **`const`-constructible** method-signature type the generated Rust names (the method-axis sibling of `ClassView`'s field projection). `ClassMethods` + `methods_for(registry, classid)`: the registry entry + zero-fallback lookup (classid bound OGAR-side; the data is generated downstream, no runtime registry stored in the contract). +- **`unicharset` module** — `UniCharSet` (`deepnsm::Vocabulary`-shaped: `reverse: Vec` id→unichar + `lookup: HashMap` unichar→id), `load_from_str`/`load_from_file`/`id_to_unichar`/`unichar_to_id`/`size`/`dump`, typed `UniCharSetError`. The Rust side of `PROBE-OGAR-ADAPTER-UNICHARSET`: pure text parsing, **zero leptonica** (the unicharset path never touches `Pix`). +- **`examples/unicharset_dump.rs`** — renders the oracle-shape `"\t\n"` table so byte-parity is a single `diff`. + +**Locked:** +- **PROBE-OGAR-ADAPTER-UNICHARSET → FINDING.** The core-first transcode doctrine is now **empirically proven end-to-end**: a real Tesseract adapter produced through the pipeline (`ruff_cpp_spo` harvest → `reassemble` → `ruff_cpp_codegen` → these contract types) is **byte-identical (112/112)** to the C++ libtesseract `UNICHARSET::id_to_unichar` oracle on real trained `eng` data. Doctrine flipped CONJECTURE→FINDING in `core-first-transcode-doctrine.md`. +- **`NULL`→space convention** (`unicharset.cpp:882`): the loader special-cases the file token `"NULL"` to the runtime space unichar `" "`. This was the sole id-0 diff the probe surfaced (was 111/112). Locked by the `null_token_maps_to_space` unit test. (codex P1 on #521 flagged the same edge case independently; resolved + thread closed.) +- **`MethodSig` must be `&'static`/`const`-constructible** — because `ClassView::FieldRef` is `String`-backed and cannot appear in a `const`, the method axis needed a distinct `&'static`-backed type. That is why this is a new type, not a `ClassView` extension. + +**Deferred:** +- Wiring the proven `UniCharSet` lookups through `classid → ClassView → UnifiedStep` (the `classid → &UniCharSet` `LazyLock` resolver) — the honest next increment. +- Per-leaf-method one-`diff` parity checks as each future Tesseract method body is transcoded. +- Fragment / `CleanupString` unichar normalization (a separate, later adapter leaf; this PR is the `old_style_included_ == true` plain-table scope). + +**Docs:** +- `.claude/plans/transcode-extend-core-probe-v1.md` — full 5-consolidate + 3-brutal council record + the C-FIRST D → emitter → EXTEND-CORE arc + the BYTE-PARITY ACHIEVED section. +- `.claude/board/EPIPHANIES.md` — `E-CPP-PARITY-1` (the byte-parity FINDING). +- `.claude/knowledge/core-first-transcode-doctrine.md` — falsifier gate flipped CONJECTURE→FINDING (RAN GREEN 2026-06-17, 112/112). +- LATEST_STATE Contract Inventory: `D-CPP-CODEGEN-1` + `D-UNICHARSET-1` (in-PR). + +**Confidence (2026-06-17):** working. 644 contract lib tests green; clippy `-D warnings` + fmt clean. Byte-parity 112/112 re-verified post-rebase (onto main after #522 odoo-extractor). codex P1 (NULL→space) resolved + thread closed; CodeRabbit summary clean. + +--- + ## #512 perturbation-sim: degenerate-grid + key-cardinality guards + core-first-transcode doctrine (review #511 + #513) **Status:** MERGED 2026-06-16 20:33 UTC (merge commit `1e23c410`), branch `claude/happy-hamilton-0azlw4`. **+591/-5 across 11 files** — addresses the open review findings from #511 (which merged before they resolved) AND introduces the **core-first-transcode** doctrine + 3 new agent cards. From 0312ab233508b5992bad656c0c537bb9256a6114 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 17 Jun 2026 21:43:05 +0000 Subject: [PATCH 2/5] =?UTF-8?q?lance-graph-contract:=20UNICHAR=20UTF-8=20c?= =?UTF-8?q?odec=20=E2=80=94=20second=20byte-parity=20adapter?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Transcode of Tesseract's `ccutil/unichar.cpp` (the UTF-8 layer UNICHARSET sits on top of) as a pure-Rust, zero-leptonica adapter — the second leaf through the harvest -> reassemble -> codegen pipeline after UniCharSet, proving PROBE-OGAR-ADAPTER-UNICHARSET generalizes beyond one class. New `unichar` module: - `utf8_step(lead) -> u8`: const-fn transcription of the 256-entry lead-byte table (unichar.cpp:143). 1/2/3/4 for legal leads, 0 for continuation bytes and 0xF8.. . - `utf8_to_utf32(bytes) -> Option>`: mirrors UNICHAR::UTF8ToUTF32 (unichar.cpp:220) — lead-byte validation only, None on illegal lead, the offset-decode of first_uni (unichar.cpp:105) inlined. Byte-parity: `examples/unichar_dump.rs` vs a libtesseract UNICHAR oracle is 268/268 identical — all 256 utf8_step values (EXHAUSTIVE) + 12 utf8_to_utf32 corpus rows. Why a faithful transcode and not core::str: Tesseract maps 0xC0/0xC1 to step 2 and decodes the overlong NUL `C0 80` to [0]; core::str::from_utf8 rejects both. A native-UTF-8 shortcut would silently diverge from the oracle. The `from_utf8_rejects_what_tesseract_accepts` test pins the gap. Additive, zero-dep, pure text. +8 tests; 653 contract lib green; clippy --all-targets -D warnings + fmt clean. Board: LATEST_STATE D-UNICHAR-1 + EPIPHANIES E-CPP-PARITY-2. Co-Authored-By: Claude https://claude.ai/code/session_016b33swuXE23hKtqxsHu9p1 --- .claude/board/EPIPHANIES.md | 7 + .claude/board/LATEST_STATE.md | 2 + .../examples/unichar_dump.rs | 64 ++++++ crates/lance-graph-contract/src/lib.rs | 1 + crates/lance-graph-contract/src/unichar.rs | 193 ++++++++++++++++++ 5 files changed, 267 insertions(+) create mode 100644 crates/lance-graph-contract/examples/unichar_dump.rs create mode 100644 crates/lance-graph-contract/src/unichar.rs diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index fd18cb1c..3dcd947a 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -78,6 +78,13 @@ Two code objects share the term **"witness arc"** but operate on **different val **Regression discipline (the "one NaN less + something to check against"):** three new tests drive the precursor regime (near-zero bridge, fully disconnected graph, degenerate ring) and assert NaN-freedom on every output + sentinel-correctness on fragmentation; one parity test (asymmetric weighted path, genuinely separated `λ₂`) pins that the gate **does not perturb healthy-grid numbers** — it gates ONLY the singular path. 91 → 95 `perturbation-sim` tests, clippy `-D warnings` clean. **Adjacent (NOT in B1 scope — a different mechanism):** `acflow.rs` (the AC Newton–Raphson power-flow path) carries its own division landmines — `Cx::recip` divides by `|z|²` (NaN on a zero-impedance line `r=x=0`) and the Jacobian partials `dp_dv`/`dq_dv` divide by bus voltage `v[i]` (a divergent NR iteration could drive `v→0`). These are **voltage-collapse**-regime sites, not spectral-fragmentation, and are gated differently (the `solve_linear` partial-pivot `best < 1e-13 → return false` already protects the linear solve, and a non-converged solve returns `converged: false`). Flagged here for a future AC-flow NaN pass; B1 deliberately scoped to the Laplacian-spectral path the task named. +## 2026-06-17 — E-CPP-PARITY-2 — the transcode generalizes: a SECOND Tesseract adapter (UNICHAR) is byte-identical to libtesseract, and proves why "just use core::str" fails + +**Status:** FINDING (in-env; exhaustive on the step table). `lance_graph_contract::unichar::{utf8_step, utf8_to_utf32}` (transcoded from `ccutil/unichar.cpp`) matches the C++ `UNICHAR` libtesseract oracle **268/268 byte-identical** — ALL 256 `utf8_step` lead-byte values (exhaustive, not sampled) plus 12 `utf8_to_utf32` corpus rows. This is the SECOND adapter through the pipeline after `UniCharSet` (E-CPP-PARITY-1), so `PROBE-OGAR-ADAPTER-UNICHARSET` is no longer a single data point — the core-first transcode generalizes across distinct classes (a content-store tier AND a pure codec). + +**The sharper finding — why a faithful transcode is mandatory, not optional.** Tesseract's `utf8_step` table maps `0xC0`/`0xC1` (overlong 2-byte leads) to step 2, and `UTF8ToUTF32` decodes the overlong NUL `C0 80` to codepoint `0` (it validates only the lead byte, never re-checks continuations). Rust's native `core::str::from_utf8` REJECTS both as invalid UTF-8. So the "obvious" shortcut — delegate UTF-8 to the standard library — would have DIVERGED from the oracle on real inputs. The behaviour-preservation rule (transcode the algorithm, do not "fix" it) is not pedantry here; it is the difference between byte-parity and silent corruption. The `from_utf8_rejects_what_tesseract_accepts` test pins the gap so a future "simplification" cannot quietly erase it. + +**Consequence.** The doctrine falsifier (flipped CONJECTURE→FINDING by E-CPP-PARITY-1) now has a second confirming adapter, with an *exhaustive* proof on its core table. The remaining Tesseract→pure-Rust OCR transcode is leaf-by-leaf adapters, each carrying its own one-`diff` parity check; the method itself is validated. Branch work (not yet a PR); contract module `D-UNICHAR-1`. ## 2026-06-17 — E-ODOO-EXTRACT-DEEP-READS — the consumer-side recompute-DAG probe surfaces a corpus enrichment ask, not a ClassView gap diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 9db941c2..173782e3 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -87,6 +87,8 @@ ## Current Contract Inventory (lance-graph-contract) +> **2026-06-17 — ADDED (D-UNICHAR-1, SECOND byte-parity adapter)**: `lance_graph_contract::unichar::{utf8_step, utf8_to_utf32}` — the Tesseract `UNICHAR` UTF-8 codec that `UNICHARSET` sits on top of (`ccutil/unichar.cpp`). `utf8_step(lead) -> u8` is a `const fn` transcription of Tesseract's 256-entry lead-byte table (1/2/3/4 for legal leads, 0 for continuation bytes `0x80..=0xBF` + `0xF8..`); `utf8_to_utf32(bytes) -> Option>` mirrors `UNICHAR::UTF8ToUTF32` (lead-byte validation only, `None` on an illegal lead). **The second adapter through the transcode pipeline, byte-parity proven**: `examples/unichar_dump.rs` vs a libtesseract `UNICHAR` oracle is **268/268 identical** (256 EXHAUSTIVE `utf8_step` lead-byte values + 12 `utf8_to_utf32` corpus rows). Faithful-transcode note (the point of the exercise): Tesseract maps `0xC0`/`0xC1` to step 2 and decodes the overlong NUL `C0 80` to `[0]`; `core::str::from_utf8` REJECTS both, so a native-UTF-8 shortcut would silently diverge — mirroring the exact table is mandatory (`from_utf8_rejects_what_tesseract_accepts` test pins it). Additive, zero-dep, pure text (no leptonica). +8 tests + the `unichar_dump` example; 653 contract lib green; clippy `--all-targets -D warnings` clean. Sibling of D-UNICHARSET-1, same `PROBE-OGAR-ADAPTER-UNICHARSET` falsifier family (E-CPP-PARITY-2). + > **2026-06-17 — ADDED (D-UNICHARSET-1, byte-parity probe Rust side)**: `lance_graph_contract::unicharset::{UniCharSet, UniCharSetError}` — the Tesseract `UNICHARSET` content-store tier (the Core-First doctrine's variable-length classid-keyed registry, `deepnsm::Vocabulary`-shaped: `reverse: Vec` id→unichar + `lookup: HashMap` unichar→id). `load_from_str`/`load_from_file` parse the `.unicharset` text format (line 1 = count, then the first whitespace token per line = unichar, id = position; property columns ignored — the `old_style_included_` plain-table scope); `id_to_unichar`/`unichar_to_id` are the two adapter leaves; `dump()` renders the `\t` table matching the C++ oracle. **The Rust side of `PROBE-OGAR-ADAPTER-UNICHARSET`** — pure text parsing, ZERO leptonica (the unicharset path never touches `Pix`), so it builds + unit-tests in-env; byte-parity is one `diff` against a libtesseract oracle harness on a leptonica-installed box (steps in `examples/unicharset_dump.rs`). Additive (a sibling content-store module, zero `NodeRow`/tenant impact). +4 tests (format parse, bijection round-trip, oracle-shape dump, typed errors) + the `unicharset_dump` example; 644 contract lib green; clippy `-D warnings` clean. Plan: `transcode-extend-core-probe-v1.md` (the deferred Option A content-store tier, now built for the probe). The classid→`&UniCharSet` `LazyLock` resolver remains the wiring follow-up. > **2026-06-17 — ADDED (D-CPP-CODEGEN-1, C-FIRST step 2 compile target)**: `lance_graph_contract::codegen_manifest::{MethodSig, ClassMethods, methods_for}` — the Core-side target of the C++ method-resolution manifest emitted by `ruff_cpp_codegen` (the Tesseract AST-DLL pipeline's stage 2). `MethodSig` is the dispatch-relevant signature in a **`const`-constructible** shape (all fields `&'static`: `name`, `params: &'static [&'static str]`, `ret`, `is_const`, `is_static`, `overrides`) — the method-axis sibling of `class_view::ClassView`'s field projection, deliberately NOT `String`-backed (a generated `const X: &[MethodSig] = &[MethodSig { .. }]` must compile; `FieldRef` is `String`-backed and cannot). `ClassMethods{classid, methods}` is the registry ENTRY the generated code emits (classid bound OGAR-side, never minted here); `methods_for(registry, classid) -> &'static [MethodSig]` is the pure lookup with zero-fallback (unregistered classid → empty slice). **Additive** (container-architect ADDITIVE-CONFIRMED): a sibling module, zero `NodeRow`/`ValueTenant`/`ValueSchema`/stride/`ENVELOPE_LAYOUT_VERSION` impact; the runtime `classid→methods` registry DATA lives downstream (generated in the consumer repo), not here. Body-shaping flags (pure-virtual/constexpr/noexcept/operator/requires) are out of scope (they drive body generation, not the signature manifest). The 8-agent step-2 council's deferred-runtime-registry resolution. +2 tests (const-constructibility proof + zero-fallback lookup); 640 contract lib green; clippy `-D warnings` clean. Plan: `.claude/plans/transcode-extend-core-probe-v1.md` (C step 2). Consumer: `ruff_cpp_codegen::render` (AdaWorldAPI/ruff) names this type in emit-text-only output. diff --git a/crates/lance-graph-contract/examples/unichar_dump.rs b/crates/lance-graph-contract/examples/unichar_dump.rs new file mode 100644 index 00000000..2969dd10 --- /dev/null +++ b/crates/lance-graph-contract/examples/unichar_dump.rs @@ -0,0 +1,64 @@ +//! Dump the `UNICHAR` byte-parity TSV — the Rust side of the `unichar` half of +//! the Tesseract transcode probe (`PROBE-OGAR-ADAPTER-UNICHARSET` sibling). +//! +//! Emits the exact shape a small libtesseract oracle prints: all 256 +//! `utf8_step` values, then `utf8_to_utf32` over a curated hex corpus. +//! +//! ```sh +//! # C++ oracle (links -lleptonica only to satisfy the linker; never calls it): +//! # g++ -std=c++17 -I/include oracle.cpp -ltesseract -lleptonica -o oracle +//! # ./oracle cases.hex > /tmp/oracle.tsv +//! cargo run -p lance-graph-contract --example unichar_dump > /tmp/rust.tsv +//! diff /tmp/oracle.tsv /tmp/rust.tsv # byte-identical => parity holds +//! ``` + +#![allow( + clippy::print_stdout, + reason = "a dump CLI example writes to stdout by design" +)] + +use lance_graph_contract::unichar::{utf8_step, utf8_to_utf32}; + +/// Curated corpus (hex-encoded bytes), identical to the oracle's input: valid +/// 1/2/3/4-byte chars, multi-char strings, lone illegal leads, and the overlong +/// NUL quirk. Hex keeps the exact bytes independent of source encoding. +const CASES: &[&str] = &[ + "41", // A + "c3a9", // é U+00E9 + "e4b8ad", // 中 U+4E2D + "f09f9880", // 😀 U+1F600 + "414243", // ABC + "48c3a9", // Hé + "e4b8ade69687", // 中文 + "80", // lone continuation -> ILLEGAL + "bf", // lone continuation -> ILLEGAL + "f8", // 5-byte form -> ILLEGAL + "ff", // 0xFF -> ILLEGAL + "c080", // overlong NUL -> [0] +]; + +fn hex_to_bytes(hex: &str) -> Vec { + (0..hex.len() / 2) + .map(|i| u8::from_str_radix(&hex[2 * i..2 * i + 2], 16).expect("valid hex pair")) + .collect() +} + +fn main() { + // Section 1: exhaustive utf8_step over all 256 lead bytes. + for b in 0u16..256 { + println!("STEP\t{b}\t{}", utf8_step(b as u8)); + } + // Section 2: utf8_to_utf32 over the corpus. + for &hex in CASES { + let bytes = hex_to_bytes(hex); + let decoded = match utf8_to_utf32(&bytes) { + None => "ILLEGAL".to_string(), + Some(codepoints) => codepoints + .iter() + .map(ToString::to_string) + .collect::>() + .join(","), + }; + println!("U32\t{hex}\t{decoded}"); + } +} diff --git a/crates/lance-graph-contract/src/lib.rs b/crates/lance-graph-contract/src/lib.rs index ac9ee71a..b94b36b4 100644 --- a/crates/lance-graph-contract/src/lib.rs +++ b/crates/lance-graph-contract/src/lib.rs @@ -101,6 +101,7 @@ pub mod soa_view; pub mod splat; pub mod tax; pub mod thinking; +pub mod unichar; pub mod unicharset; pub mod view_angle; pub mod vsa; diff --git a/crates/lance-graph-contract/src/unichar.rs b/crates/lance-graph-contract/src/unichar.rs new file mode 100644 index 00000000..3a4c7e5a --- /dev/null +++ b/crates/lance-graph-contract/src/unichar.rs @@ -0,0 +1,193 @@ +//! `UNICHAR` UTF-8 codec — the second byte-parity adapter of the Tesseract +//! C++→Rust transcode (the sibling of `unicharset`, this time on +//! `ccutil/unichar.cpp`). +//! +//! Tesseract's `UNICHAR` is the hand-rolled UTF-8 layer that `UNICHARSET` sits +//! on top of. Two pure functions carry the whole bytes↔codepoint substrate: +//! [`utf8_step`] (how many bytes the first character occupies, read from a +//! 256-entry lead-byte table) and [`utf8_to_utf32`] (decode a byte string to +//! its Unicode codepoints). Both are pure text — zero leptonica, zero `Pix` — +//! so this is a second zero-C-dependency byte-parity surface, exactly like +//! [`crate::unicharset`]. +//! +//! # Why a faithful transcode, not [`core::str`] +//! +//! Tesseract's `utf8_step` table maps `0xC0`/`0xC1` (the overlong 2-byte leads) +//! to **step 2**, and [`utf8_to_utf32`] decodes the overlong NUL `C0 80` to +//! codepoint `0` — Rust's native [`core::str::from_utf8`] *rejects* both as +//! invalid UTF-8. A "just call `from_utf8`" shortcut would therefore DIVERGE +//! from the C++ oracle on real inputs. Reproducing the exact lead-byte table, +//! quirks and all, is the core-first doctrine and the behaviour-preservation +//! rule made concrete: a transcode mirrors the algorithm, it does not "fix" it. +//! The [`tests::from_utf8_rejects_what_tesseract_accepts`] test pins this gap. +//! +//! # Byte-parity surface (`PROBE-OGAR-ADAPTER-UNICHARSET` sibling) +//! +//! The `unichar_dump` example renders the exact TSV shape a small libtesseract +//! oracle prints — all 256 `utf8_step` values plus `utf8_to_utf32` over a +//! curated hex corpus — so byte-parity is a single `diff`. The exhaustive +//! 256-byte step table makes that half of the proof *complete*, not sampled. +//! +//! Mirrors `ccutil/unichar.cpp`: `utf8_step` (the 256-table, lines 143-156), +//! `first_uni` (the offset decode, lines 105-131), and `UTF8ToUTF32` +//! (lines 220-234 — lead-byte validation only; continuation bytes are NOT +//! re-checked, which is what lets `C0 80` through). + +/// The number of bytes the first UTF-8 character of a buffer occupies, keyed on +/// its lead byte: `1`/`2`/`3`/`4` for a legal lead, `0` for an illegal one +/// (a continuation byte `0x80..=0xBF`, or `0xF8..=0xFF`). +/// +/// This is a `const fn` transcription of Tesseract's 256-entry `utf8_bytes` +/// table (`unichar.cpp:143`). The exhaustive 256-byte byte-parity probe proves +/// it equals the C++ table value-for-value. +/// +/// Note the faithfully-preserved quirk: `0xC0`/`0xC1` (the overlong 2-byte +/// leads) map to `2`, not `0` — Tesseract does not reject overlong forms here, +/// so neither does this transcode. +#[must_use] +pub const fn utf8_step(lead: u8) -> u8 { + match lead { + 0x00..=0x7F => 1, // ASCII + 0x80..=0xBF => 0, // continuation byte — illegal as a lead + 0xC0..=0xDF => 2, // 2-byte lead (incl. overlong 0xC0/0xC1, faithfully) + 0xE0..=0xEF => 3, // 3-byte lead + 0xF0..=0xF7 => 4, // 4-byte lead + 0xF8..=0xFF => 0, // illegal + } +} + +/// UCS-4 offsets subtracted after accumulating the UTF-8 bytes, indexed by the +/// character's byte length (`unichar.cpp:106`). Indices 0/1 are `0`; the rest +/// cancel the accumulated lead-byte and continuation marker bits. +const UTF8_OFFSETS: [i32; 5] = [0, 0, 0x3080, 0xE2080, 0x3C8_2080]; + +/// The Unicode codepoint of the first character in `bytes`, mirroring +/// `UNICHAR::first_uni` (`unichar.cpp:105`). `bytes` must be non-empty and +/// begin with a legal lead whose continuation bytes are present — the only +/// caller, [`utf8_to_utf32`], guarantees the lead is legal before calling. +/// If the buffer is shorter than the lead implies (a truncated trailing +/// character) this decodes only the bytes present rather than reading past the +/// slice; that case is out of the documented byte-parity scope. +fn first_uni(bytes: &[u8]) -> i32 { + let len = utf8_step(bytes[0]) as usize; + let mut uni: i32 = 0; + for (i, &b) in bytes.iter().take(len).enumerate() { + uni += i32::from(b); + if i + 1 < len { + uni <<= 6; + } + } + uni - UTF8_OFFSETS[len.min(4)] +} + +/// Decode a UTF-8 byte string to its Unicode codepoints, mirroring +/// `UNICHAR::UTF8ToUTF32` (`unichar.cpp:220`). +/// +/// Returns `None` if any character's **lead byte** is illegal (the C++ +/// "return an empty vector" path). Like the C++, continuation bytes are NOT +/// re-validated, so the overlong NUL `C0 80` decodes to `[0]` rather than being +/// rejected (see the module docs). Empty input is `Some(vec![])` — distinct +/// from the `None` illegal case, which the C++ conflates as an empty vector +/// (this is the one place the Rust surface is strictly clearer; the corpus +/// avoids empty input so the byte-parity diff is unaffected). +#[must_use] +pub fn utf8_to_utf32(bytes: &[u8]) -> Option> { + let mut out = Vec::new(); + let mut i = 0; + while i < bytes.len() { + let step = utf8_step(bytes[i]) as usize; + if step == 0 { + return None; // illegal lead + } + out.push(first_uni(&bytes[i..])); + i += step; + } + Some(out) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The `const fn` ranges reproduce the documented 256-table shape: 128 ones, + /// 72 zeros (64 continuation + 8 `0xF8..`), 32 twos, 16 threes, 8 fours. + #[test] + fn step_table_value_histogram() { + let mut counts = [0usize; 5]; + for b in 0u16..=255 { + counts[utf8_step(b as u8) as usize] += 1; + } + assert_eq!(counts, [72, 128, 32, 16, 8]); + } + + /// The faithfully-preserved quirks: overlong 2-byte leads are "valid" + /// (step 2), continuation bytes and `0xF8..` are illegal (step 0). + #[test] + fn step_quirks_match_tesseract() { + assert_eq!(utf8_step(0xC0), 2, "overlong lead kept as step 2"); + assert_eq!(utf8_step(0xC1), 2, "overlong lead kept as step 2"); + assert_eq!(utf8_step(0x80), 0, "continuation byte illegal as lead"); + assert_eq!(utf8_step(0xBF), 0, "continuation byte illegal as lead"); + assert_eq!(utf8_step(0xF8), 0, "5-byte form illegal"); + assert_eq!(utf8_step(0xFF), 0, "0xFF illegal"); + assert_eq!(utf8_step(b'A'), 1); + assert_eq!(utf8_step(0xE4), 3); + assert_eq!(utf8_step(0xF0), 4); + } + + #[test] + fn decodes_one_to_four_byte_chars() { + assert_eq!(utf8_to_utf32(b"A"), Some(vec![65])); + assert_eq!(utf8_to_utf32(&[0xC3, 0xA9]), Some(vec![0xE9])); // é U+00E9 + assert_eq!(utf8_to_utf32(&[0xE4, 0xB8, 0xAD]), Some(vec![0x4E2D])); // 中 + assert_eq!( + utf8_to_utf32(&[0xF0, 0x9F, 0x98, 0x80]), + Some(vec![0x1_F600]) // 😀 U+1F600 = 128512 + ); + } + + #[test] + fn decodes_multi_char_strings() { + assert_eq!(utf8_to_utf32(b"ABC"), Some(vec![65, 66, 67])); + // "中文" = U+4E2D U+6587 + assert_eq!( + utf8_to_utf32(&[0xE4, 0xB8, 0xAD, 0xE6, 0x96, 0x87]), + Some(vec![0x4E2D, 0x6587]) + ); + } + + #[test] + fn illegal_lead_byte_is_none() { + assert_eq!(utf8_to_utf32(&[0x80]), None); + assert_eq!(utf8_to_utf32(&[0xFF]), None); + assert_eq!(utf8_to_utf32(&[b'A', 0xF8]), None); // legal then illegal + } + + /// The overlong NUL `C0 80` is accepted and decodes to `0` — the defining + /// quirk the byte-parity probe confirmed against the C++ oracle. + #[test] + fn overlong_nul_decodes_to_zero() { + assert_eq!(utf8_to_utf32(&[0xC0, 0x80]), Some(vec![0])); + } + + #[test] + fn empty_input_is_some_empty() { + assert_eq!(utf8_to_utf32(b""), Some(vec![])); + } + + /// The whole reason this is a transcode and not a `from_utf8` call: Rust's + /// native decoder rejects exactly the inputs Tesseract's hand-rolled table + /// accepts. If these ever agreed, the faithful transcode would be moot. + #[test] + #[expect( + invalid_from_utf8, + reason = "the point of this test IS that the literal is invalid UTF-8 that std rejects but Tesseract accepts" + )] + fn from_utf8_rejects_what_tesseract_accepts() { + // overlong NUL: std rejects, Tesseract decodes to [0] + assert!(core::str::from_utf8(&[0xC0, 0x80]).is_err()); + assert_eq!(utf8_to_utf32(&[0xC0, 0x80]), Some(vec![0])); + // 0xC0 lead: std rejects as invalid, Tesseract treats as a 2-byte lead + assert_eq!(utf8_step(0xC0), 2); + } +} From b1483a2ec19109221d1741d92a44625bbdcfc514 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 18 Jun 2026 07:01:29 +0000 Subject: [PATCH 3/5] =?UTF-8?q?lance-graph-contract:=20UniCharSet=20keysto?= =?UTF-8?q?ne=20=E2=80=94=20classid=20->=20ClassView=20->=20adapter?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wires the proven UniCharSet adapter (E-CPP-PARITY-1, byte-parity 112/112) through the OGAR Core's three movable parts — steps 2-3 of PROBE-OGAR-ADAPTER-UNICHARSET, which prior work left as "mechanical wiring, conjectured." This proves the core-first transcode doctrine END-TO-END for the unicharset class, not just for one leaf's bytes. New `unicharset_adapter` module: - `UniCharSetStore` trait: the classid-keyed content-store tier (consumer- provided, dependency-inverted like ClassView). The adapter holds NO state; the variable-length bijection rides this tier (I-VSA-IDENTITIES). - `UniCharCall` (DO-in) / `UniCharOut` (DO-out, zero-copy borrow) / `DispatchError`. - `invoke_unicharset(registry, store, classid, call)` — the keystone: 1. ClassView composition gate: codegen_manifest::methods_for(registry, classid) must list the method (the harvested has_function manifest), else MethodNotComposed (zero-fallback: unconfigured classid composes nothing). 2. content-store tier: UniCharSetStore::unicharset(classid). 3. adapter leaf: UniCharSet::{id_to_unichar, unichar_to_id}. Byte-parity is inherited from UniCharSet; the keystone proves the dispatch path is faithful (the NULL->space edge survives it), the gate works, and there is NO Core gap (the doctrine's iron guard holds with zero strain). Not routed through the heavy OrchestrationBridge cross-subsystem router; this is the adapter- invocation primitive a UnifiedStep calls. Additive, zero-dep. +5 tests; clippy --all-targets -D warnings + fmt clean. Board: LATEST_STATE D-UNICHARSET-KEYSTONE; EPIPHANIES E-CPP-KEYSTONE-1; core-first-transcode-doctrine.md steps 2-3 marked wired. Co-Authored-By: Claude https://claude.ai/code/session_016b33swuXE23hKtqxsHu9p1 --- .claude/board/EPIPHANIES.md | 7 + .claude/board/LATEST_STATE.md | 2 + .../core-first-transcode-doctrine.md | 10 + crates/lance-graph-contract/src/lib.rs | 1 + .../src/unicharset_adapter.rs | 338 ++++++++++++++++++ 5 files changed, 358 insertions(+) create mode 100644 crates/lance-graph-contract/src/unicharset_adapter.rs diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 3dcd947a..3f82a015 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -56,6 +56,13 @@ Two code objects share the term **"witness arc"** but operate on **different val **Why they must NOT be welded under a `WitnessArcEvaluator` trait:** (a) no inner-product/transform structure exists over opaque identity tuples — superposing them is the register-loss `I-VSA-IDENTITIES` forbids; (b) a trait with one math impl in an *excluded* crate and zero call-sites is AP6 speculative generality; (c) the AGI-as-glove doctrine bars a new trait/bridge ("new capability = new column, not a new layer"); (d) the trait would force the **zero-dep** contract crate to gain a `perturbation-sim` dependency / `f64` field semantics (baton CATCH-CRITICAL, dep-direction inversion). The genuine relationship is a **pipeline seam** (W-slot arc *addresses*, numeric column *evaluates*) — a consumer-side free function over a borrowed `&[f64]` column, never a unifying trait. **Secondary correction (truth-architect + brutally-honest-tester):** the `witness.rs` doc overclaimed per-arc cost as `q·O(N)`; as implemented `witness_from_spectrum` re-transforms each arc (`O(N log N)`), so the honest total is `O(N log N) + q·O(N log N)`, narrowing to `q·O(N)` only for precomputed/structured arc spectra. The amortized quantity is the **field** transform, not the arc dot. Doc-string corrected; no measured speed claim is made. Cross-references added in both files; seam recorded as `TD-WITNESS-EVAL-WIRING-1`, wiring gated as D-MBX-A3 (on D-MBX-A2 + OQ-11.2 + a §0 dep-direction ruling). +## 2026-06-18 — E-CPP-KEYSTONE-1 — the core-first transcode is proven END-TO-END: classid → ClassView → content store → adapter dispatches with no Core gap + +**Status:** FINDING (in-contract, tested). `lance_graph_contract::unicharset_adapter::invoke_unicharset` wires the proven `UniCharSet` adapter (E-CPP-PARITY-1, 112/112) through the OGAR Core's three movable parts — completing steps 2–3 of `PROBE-OGAR-ADAPTER-UNICHARSET` (the doctrine's falsifier), which step 1 (E-CPP-PARITY-1) and the generalization (E-CPP-PARITY-2, UNICHAR) had left as "mechanical wiring, conjectured." + +**What the keystone proves.** The doctrine claimed the OGAR Core makes the transcode clean: identity = classid, composition = `classid → ClassView` (the harvested `has_function` manifest), state = classid-keyed content-store tier, invocation = thin DO-in/DO-out. The keystone is the first place all three meet: `invoke_unicharset` gates on `methods_for(registry, classid)` (composition), resolves `UniCharSetStore::unicharset(classid)` (content tier), and routes to the proven leaf — and the `NULL`→space byte-parity edge survives the whole dispatch path. **The iron guard held with zero strain:** no adapter-state-leak (the adapter holds nothing; the bijection rides the content tier), no Core gap (no new tenant / ClassView capability needed), no parallel object model. The doctrine's central worry ("the adapter needs state the Core can't carry") is now refuted not just for the leaf (parity) but for the composed dispatch. + +**Consequence.** Core-first transcode is no longer "proven for one leaf's bytes" — it is proven END-TO-END for the unicharset class: a classid composes its adapters via the manifest, dispatch reaches them through the content tier, output is byte-faithful. The remaining transcode is repetition of a validated pattern (each new leaf = its own one-`diff` parity + a manifest entry), not new architecture. The composition gate also yields the zero-fallback for free (an unconfigured classid composes nothing → a typed refusal, never a panic). Contract module `D-UNICHARSET-KEYSTONE`. Branch work, not yet a PR. ## 2026-06-17 — E-ODOO-SPO-INHERITS-VALKIND — spo_enrich extended with inherits_from (P1b/ruff#19) + validation_kind (P2/ruff#21); corpus regen pending Odoo source diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 173782e3..a4f1d8c4 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -87,6 +87,8 @@ ## Current Contract Inventory (lance-graph-contract) +> **2026-06-18 — ADDED (D-UNICHARSET-KEYSTONE, classid → ClassView → adapter wiring)**: `lance_graph_contract::unicharset_adapter::{UniCharSetStore, UniCharCall, UniCharOut, DispatchError, invoke_unicharset}` — steps 2–3 of `PROBE-OGAR-ADAPTER-UNICHARSET`, the keystone composing the proven `UniCharSet` adapter through the OGAR Core's three movable parts. `invoke_unicharset(registry, store, classid, call)`: (1) **ClassView composition gate** — `codegen_manifest::methods_for(registry, classid)` must list the call's method (the harvested `has_function` manifest), else `MethodNotComposed` (zero-fallback: an unconfigured classid composes nothing); (2) **content-store tier** — `UniCharSetStore::unicharset(classid)`, a consumer-provided trait (dependency-inverted like `ClassView`/`PlannerContract`; the adapter holds NO state — `I-VSA-IDENTITIES`); (3) **adapter leaf** — routes to `UniCharSet::{id_to_unichar, unichar_to_id}`. DO-in (`UniCharCall`) / DO-out (`UniCharOut`, zero-copy borrow). **Byte-parity inherited** from `UniCharSet` (112/112); the keystone proves the dispatch path is faithful (the `NULL`→space edge survives it), the gate works, and there is **no Core gap** (the doctrine's iron guard holds — the variable-length bijection rides the content tier cleanly). NOT routed through the heavy `OrchestrationBridge` (cross-subsystem router); this is the adapter-invocation primitive a `UnifiedStep` calls. Additive, zero-dep. +5 tests; clippy `--all-targets -D warnings` + fmt clean. Completes the core-first doctrine END-TO-END for the unicharset leaf (`E-CPP-KEYSTONE-1`). + > **2026-06-17 — ADDED (D-UNICHAR-1, SECOND byte-parity adapter)**: `lance_graph_contract::unichar::{utf8_step, utf8_to_utf32}` — the Tesseract `UNICHAR` UTF-8 codec that `UNICHARSET` sits on top of (`ccutil/unichar.cpp`). `utf8_step(lead) -> u8` is a `const fn` transcription of Tesseract's 256-entry lead-byte table (1/2/3/4 for legal leads, 0 for continuation bytes `0x80..=0xBF` + `0xF8..`); `utf8_to_utf32(bytes) -> Option>` mirrors `UNICHAR::UTF8ToUTF32` (lead-byte validation only, `None` on an illegal lead). **The second adapter through the transcode pipeline, byte-parity proven**: `examples/unichar_dump.rs` vs a libtesseract `UNICHAR` oracle is **268/268 identical** (256 EXHAUSTIVE `utf8_step` lead-byte values + 12 `utf8_to_utf32` corpus rows). Faithful-transcode note (the point of the exercise): Tesseract maps `0xC0`/`0xC1` to step 2 and decodes the overlong NUL `C0 80` to `[0]`; `core::str::from_utf8` REJECTS both, so a native-UTF-8 shortcut would silently diverge — mirroring the exact table is mandatory (`from_utf8_rejects_what_tesseract_accepts` test pins it). Additive, zero-dep, pure text (no leptonica). +8 tests + the `unichar_dump` example; 653 contract lib green; clippy `--all-targets -D warnings` clean. Sibling of D-UNICHARSET-1, same `PROBE-OGAR-ADAPTER-UNICHARSET` falsifier family (E-CPP-PARITY-2). > **2026-06-17 — ADDED (D-UNICHARSET-1, byte-parity probe Rust side)**: `lance_graph_contract::unicharset::{UniCharSet, UniCharSetError}` — the Tesseract `UNICHARSET` content-store tier (the Core-First doctrine's variable-length classid-keyed registry, `deepnsm::Vocabulary`-shaped: `reverse: Vec` id→unichar + `lookup: HashMap` unichar→id). `load_from_str`/`load_from_file` parse the `.unicharset` text format (line 1 = count, then the first whitespace token per line = unichar, id = position; property columns ignored — the `old_style_included_` plain-table scope); `id_to_unichar`/`unichar_to_id` are the two adapter leaves; `dump()` renders the `\t` table matching the C++ oracle. **The Rust side of `PROBE-OGAR-ADAPTER-UNICHARSET`** — pure text parsing, ZERO leptonica (the unicharset path never touches `Pix`), so it builds + unit-tests in-env; byte-parity is one `diff` against a libtesseract oracle harness on a leptonica-installed box (steps in `examples/unicharset_dump.rs`). Additive (a sibling content-store module, zero `NodeRow`/tenant impact). +4 tests (format parse, bijection round-trip, oracle-shape dump, typed errors) + the `unicharset_dump` example; 644 contract lib green; clippy `-D warnings` clean. Plan: `transcode-extend-core-probe-v1.md` (the deferred Option A content-store tier, now built for the probe). The classid→`&UniCharSet` `LazyLock` resolver remains the wiring follow-up. diff --git a/.claude/knowledge/core-first-transcode-doctrine.md b/.claude/knowledge/core-first-transcode-doctrine.md index d5f788f4..609f692d 100644 --- a/.claude/knowledge/core-first-transcode-doctrine.md +++ b/.claude/knowledge/core-first-transcode-doctrine.md @@ -196,6 +196,16 @@ is no longer a conjecture. Leptonica is an *install*, not a transcode — it is a link dep of the C++ oracle, never in the Rust path (the unicharset path never touches `Pix`). +**Update (2026-06-18) — steps 2–3 wired (keystone landed).** The "mechanical +wiring" is done: `lance_graph_contract::unicharset_adapter::invoke_unicharset` +composes the adapter via `codegen_manifest::methods_for` (the ClassView +composition gate over the harvested `has_function` manifest) + a classid-keyed +`UniCharSetStore` content tier, and routes to the proven leaf. The `NULL`→space +parity edge survives the full dispatch; the iron guard held with zero strain — +no adapter-state-leak, no Core gap, no parallel object model. The doctrine is now +proven END-TO-END for the unicharset class, not just for the leaf's bytes. See +`EPIPHANIES.md` E-CPP-KEYSTONE-1 + LATEST_STATE `D-UNICHARSET-KEYSTONE`. + ## Anti-patterns this doctrine exists to catch - **Residue-Core** — treating the Core as "the parts we couldn't codegen" diff --git a/crates/lance-graph-contract/src/lib.rs b/crates/lance-graph-contract/src/lib.rs index b94b36b4..3ec51157 100644 --- a/crates/lance-graph-contract/src/lib.rs +++ b/crates/lance-graph-contract/src/lib.rs @@ -103,6 +103,7 @@ pub mod tax; pub mod thinking; pub mod unichar; pub mod unicharset; +pub mod unicharset_adapter; pub mod view_angle; pub mod vsa; pub mod witness_table; diff --git a/crates/lance-graph-contract/src/unicharset_adapter.rs b/crates/lance-graph-contract/src/unicharset_adapter.rs new file mode 100644 index 00000000..73e08e7d --- /dev/null +++ b/crates/lance-graph-contract/src/unicharset_adapter.rs @@ -0,0 +1,338 @@ +//! The UniCharSet **keystone** — `classid → ClassView → content store → adapter`, +//! steps 2–3 of `PROBE-OGAR-ADAPTER-UNICHARSET`. +//! +//! [`crate::unicharset`] proved step 1: the `UniCharSet` content adapter is +//! byte-identical to libtesseract (112/112). This module is the wiring the +//! core-first doctrine calls "mechanical once the adapter is proven": it +//! composes that adapter through the OGAR Core's three movable parts and shows a +//! dispatch reaches the proven leaf without the adapter carrying any state. +//! +//! | Core movable part | Here | +//! |---|---| +//! | identity = `classid` | the `u32` classid parameter (bound OGAR-side, never minted here) | +//! | composition = `classid → ClassView` | [`methods_for`] over the harvested `has_function` manifest gates which methods a class composes | +//! | state = classid-keyed content tier | [`UniCharSetStore`] resolves `classid → &UniCharSet` (consumer-provided; the adapter holds NO state — `I-VSA-IDENTITIES` content-store tier) | +//! | invocation | [`invoke_unicharset`] — the thin DO-in ([`UniCharCall`]) / DO-out ([`UniCharOut`]) shape a [`crate::orchestration::UnifiedStep`] would call | +//! +//! # Why this is "the keystone, not another adapter" +//! +//! It is the first place all three axes meet: the composition gate +//! ([`methods_for`]), the content-store tier ([`UniCharSetStore`]), and the +//! proven leaf ([`UniCharSet`]). Byte-parity is **inherited** from `UniCharSet` +//! (already green vs the C++ oracle); what this proves is that the dispatch path +//! is faithful (the parity edge survives it) and that the ClassView composition +//! gate works — i.e. the doctrine's iron guard holds (no adapter-state-leak, no +//! Core gap: the variable-length bijection rides the content tier cleanly). +//! +//! The full [`crate::orchestration::OrchestrationBridge`] is the *cross-subsystem* +//! router (Crew / Ladybug / LanceGraph / …); routing one in-class method call +//! through it would be over-wiring. [`invoke_unicharset`] is the adapter- +//! invocation primitive that the broader `UnifiedStep` orchestration dispatches. + +use crate::codegen_manifest::{methods_for, ClassMethods}; +use crate::unicharset::UniCharSet; + +/// The classid-keyed **content-store tier**: resolve a `classid` to its loaded +/// [`UniCharSet`]. Implemented by the consumer (e.g. a `HashMap` +/// loaded from `.unicharset` files); the contract owns only this vocabulary +/// (dependency inversion, exactly like [`crate::class_view::ClassView`] and +/// [`crate::plan::PlannerContract`]). +/// +/// The content NEVER lives on the node row or the adapter — a variable-length +/// bijection is not fixed-width SoA state; it rides this tier, keyed by the +/// identity (`I-VSA-IDENTITIES`; the core-first doctrine's "content-store tier, +/// `deepnsm::Vocabulary`-shaped"). +pub trait UniCharSetStore { + /// The [`UniCharSet`] bound to `classid`, or `None` if no store is bound. + fn unicharset(&self, classid: u32) -> Option<&UniCharSet>; +} + +/// A typed call into the UniCharSet adapter set — the **DO-in**. Each variant's +/// [`method_name`](UniCharCall::method_name) is the C++ method the harvested +/// `has_function` manifest must list for the class to compose it. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum UniCharCall<'a> { + /// `const char *id_to_unichar(UNICHAR_ID id) const` — the id→unichar read. + IdToUnichar(u32), + /// `UNICHAR_ID unichar_to_id(const char *repr) const` — the inverse lookup. + UnicharToId(&'a str), +} + +impl UniCharCall<'_> { + /// The C++ method name this call dispatches to — the key the ClassView method + /// manifest must contain for the call to be composed. These match the + /// `UNICHARSET` member names the `ruff_cpp_spo` harvest emits. + #[must_use] + pub const fn method_name(&self) -> &'static str { + match self { + Self::IdToUnichar(_) => "id_to_unichar", + Self::UnicharToId(_) => "unichar_to_id", + } + } +} + +/// The adapter's typed result — the **DO-out**. Borrows the unichar string from +/// the content store (zero-copy; the store outlives the call). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum UniCharOut<'a> { + /// The unichar at an id (`None` = out of range), from [`UniCharCall::IdToUnichar`]. + Unichar(Option<&'a str>), + /// The id of a unichar (`None` = absent, the C++ `INVALID_UNICHAR_ID` + /// sentinel), from [`UniCharCall::UnicharToId`]. + Id(Option), +} + +/// Why a keystone dispatch could not run (a typed refusal, never a panic). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum DispatchError { + /// The classid's ClassView does not compose the called method — the harvested + /// `has_function` manifest ([`methods_for`]) has no such [`MethodSig`](crate::codegen_manifest::MethodSig). + /// This is the **composition gate** firing; an unconfigured classid composes + /// nothing (zero-fallback), so it always lands here. + MethodNotComposed { + /// The classid whose manifest lacked the method. + classid: u32, + /// The C++ method name that was not composed. + method: &'static str, + }, + /// The method is composed, but no [`UniCharSet`] is bound to this classid in + /// the content-store tier. + NoContentStore { + /// The classid with no bound content store. + classid: u32, + }, +} + +impl core::fmt::Display for DispatchError { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + match self { + Self::MethodNotComposed { classid, method } => write!( + f, + "classid {classid:#010x} does not compose method `{method}`" + ), + Self::NoContentStore { classid } => { + write!( + f, + "no UniCharSet content store bound to classid {classid:#010x}" + ) + } + } + } +} + +impl std::error::Error for DispatchError {} + +/// **The keystone.** Invoke a `UniCharSet` adapter through the +/// `classid → ClassView` method manifest and the classid-keyed content store — +/// steps 2–3 of `PROBE-OGAR-ADAPTER-UNICHARSET`. +/// +/// 1. **ClassView composition gate**: [`methods_for(registry, classid)`](methods_for) +/// must contain a method whose name is the call's +/// [`method_name`](UniCharCall::method_name) — i.e. the harvested +/// `has_function` manifest says this class composes this adapter. Otherwise +/// [`DispatchError::MethodNotComposed`] (an unconfigured classid composes +/// nothing — the zero-fallback ladder). +/// 2. **Content-store tier**: [`store.unicharset(classid)`](UniCharSetStore::unicharset) +/// supplies the loaded bijection; the adapter holds no state of its own. +/// 3. **Adapter leaf**: route to [`UniCharSet::id_to_unichar`] / +/// [`UniCharSet::unichar_to_id`]. +/// +/// Byte-parity is inherited from [`UniCharSet`] (112/112 vs libtesseract); this +/// proves the dispatch path is faithful and the composition gate holds. +/// +/// # Errors +/// +/// [`DispatchError::MethodNotComposed`] if the class's manifest does not list the +/// called method; [`DispatchError::NoContentStore`] if no `UniCharSet` is bound +/// to the classid. +pub fn invoke_unicharset<'a, S: UniCharSetStore + ?Sized>( + registry: &[ClassMethods], + store: &'a S, + classid: u32, + call: &UniCharCall<'_>, +) -> Result, DispatchError> { + let method = call.method_name(); + // 1. ClassView composition gate (the harvested has_function manifest). + if !methods_for(registry, classid) + .iter() + .any(|m| m.name == method) + { + return Err(DispatchError::MethodNotComposed { classid, method }); + } + // 2. Content-store tier — state lives here, never on the adapter. + let unicharset = store + .unicharset(classid) + .ok_or(DispatchError::NoContentStore { classid })?; + // 3. The proven adapter leaf. + Ok(match *call { + UniCharCall::IdToUnichar(id) => UniCharOut::Unichar(unicharset.id_to_unichar(id)), + UniCharCall::UnicharToId(repr) => UniCharOut::Id(unicharset.unichar_to_id(repr)), + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::codegen_manifest::MethodSig; + use std::collections::HashMap; + + /// A test OGAR classid for the Tesseract `UNICHARSET` class (bound OGAR-side; + /// the keystone never mints it). + const TESS_UNICHARSET: u32 = 0x0001_0001; + + /// The ClassView method manifest the `ruff_cpp_spo` harvest emits for + /// `UNICHARSET` — the two byte-parity-proven leaves. + const UNICHARSET_METHODS: &[MethodSig] = &[ + MethodSig { + name: "id_to_unichar", + params: &["UNICHAR_ID"], + ret: Some("const char *"), + is_const: true, + is_static: false, + overrides: None, + }, + MethodSig { + name: "unichar_to_id", + params: &["const char *"], + ret: Some("UNICHAR_ID"), + is_const: true, + is_static: false, + overrides: None, + }, + ]; + + const REGISTRY: &[ClassMethods] = &[ClassMethods { + classid: TESS_UNICHARSET, + methods: UNICHARSET_METHODS, + }]; + + /// An in-memory content store — what a consumer builds from `.unicharset` + /// files. The contract owns the trait; the consumer owns the answers. + struct MemStore(HashMap); + + impl UniCharSetStore for MemStore { + fn unicharset(&self, classid: u32) -> Option<&UniCharSet> { + self.0.get(&classid) + } + } + + fn store_with_null_sample() -> MemStore { + // id 0 = NULL→space (the byte-parity edge), id 1 = "a". + let u = UniCharSet::load_from_str("2\nNULL 0 Common 0\na 3 0 a Left a a\n").expect("valid"); + let mut m = HashMap::new(); + m.insert(TESS_UNICHARSET, u); + MemStore(m) + } + + /// The full keystone path: classid → ClassView (methods_for) → content store + /// → adapter leaf. The `NULL`→space parity edge survives the whole dispatch. + #[test] + fn keystone_dispatches_with_inherited_byte_parity() { + let store = store_with_null_sample(); + assert_eq!( + invoke_unicharset( + REGISTRY, + &store, + TESS_UNICHARSET, + &UniCharCall::IdToUnichar(0) + ), + Ok(UniCharOut::Unichar(Some(" "))), + "the NULL->space edge survives classid -> ClassView -> store -> adapter" + ); + assert_eq!( + invoke_unicharset( + REGISTRY, + &store, + TESS_UNICHARSET, + &UniCharCall::UnicharToId(" ") + ), + Ok(UniCharOut::Id(Some(0))) + ); + assert_eq!( + invoke_unicharset( + REGISTRY, + &store, + TESS_UNICHARSET, + &UniCharCall::IdToUnichar(1) + ), + Ok(UniCharOut::Unichar(Some("a"))) + ); + assert_eq!( + invoke_unicharset( + REGISTRY, + &store, + TESS_UNICHARSET, + &UniCharCall::IdToUnichar(99) + ), + Ok(UniCharOut::Unichar(None)), + "out-of-range id is a clean None, not a panic" + ); + } + + /// The composition gate: a class whose manifest composes only `id_to_unichar` + /// refuses `unichar_to_id`, even though the content store could answer it. + #[test] + fn keystone_classview_gate_rejects_uncomposed_method() { + const ID_ONLY: &[ClassMethods] = &[ClassMethods { + classid: TESS_UNICHARSET, + methods: &[MethodSig { + name: "id_to_unichar", + params: &["UNICHAR_ID"], + ret: Some("const char *"), + is_const: true, + is_static: false, + overrides: None, + }], + }]; + let store = store_with_null_sample(); + assert_eq!( + invoke_unicharset( + ID_ONLY, + &store, + TESS_UNICHARSET, + &UniCharCall::UnicharToId(" ") + ), + Err(DispatchError::MethodNotComposed { + classid: TESS_UNICHARSET, + method: "unichar_to_id", + }) + ); + } + + /// Zero-fallback: an unregistered classid composes nothing, so the gate + /// refuses before the store is even consulted. + #[test] + fn keystone_unregistered_classid_composes_nothing() { + let store = store_with_null_sample(); + assert_eq!( + invoke_unicharset(REGISTRY, &store, 0xDEAD_BEEF, &UniCharCall::IdToUnichar(0)), + Err(DispatchError::MethodNotComposed { + classid: 0xDEAD_BEEF, + method: "id_to_unichar", + }) + ); + } + + /// A composed method with no bound content store is a typed error, not a panic. + #[test] + fn keystone_missing_content_store_is_typed_error() { + let empty = MemStore(HashMap::new()); + assert_eq!( + invoke_unicharset( + REGISTRY, + &empty, + TESS_UNICHARSET, + &UniCharCall::IdToUnichar(0) + ), + Err(DispatchError::NoContentStore { + classid: TESS_UNICHARSET, + }) + ); + } + + #[test] + fn call_method_names_match_the_harvest() { + assert_eq!(UniCharCall::IdToUnichar(0).method_name(), "id_to_unichar"); + assert_eq!(UniCharCall::UnicharToId("x").method_name(), "unichar_to_id"); + } +} From 7834828a72484f5470147bc6880c54ac5b927cdd Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 18 Jun 2026 07:33:04 +0000 Subject: [PATCH 4/5] lance-graph-ontology: wire OGAR GUID -> ClassView (class_id_for_guid) Closes the OGAR -> lance-graph-ontology gap an audit this session found: NiblePath::from_guid_prefix (the canon GUID->NiblePath fold) and the registry's entity_type <-> NiblePath bijection were BOTH built with ZERO callers -- the two halves of the bridge forged but never chained. OntologyRegistry::class_id_for_guid(&NodeGuid) -> Option lays the join: from_guid_prefix(guid)? -> entity_type_of(path). A node row carrying a classid now resolves its ontology class (entity_type / ClassId), which RegistryClassView already turns into the class shape (fields/labels/template/DOLCE). No new predicate, no new type -- a method composing two existing surfaces (aligns with E-ODOO-CORE-FIRST-STRUCTURAL: Core-side resolution, not an SPO bolt-on). Round-trip test pins the classid_lo <-> entity_type consistency the audit flagged: register_class_path(t, from_guid_prefix(g)) => class_id_for_guid(g) == Some(t); zero-fallback (unbound GUID -> None); lossy-fold refusal (high classid u16 -> None). 16 ontology tests green; registry.rs clippy-clean + fmt clean. Board: LATEST_STATE wiring entry; EPIPHANIES E-OGAR-ONTOLOGY-WIRED-1; TECH_DEBT TD-ONTOLOGY-LINT (pre-existing crate clippy debt, present on main, un-CI-gated). Co-Authored-By: Claude https://claude.ai/code/session_016b33swuXE23hKtqxsHu9p1 --- .claude/board/EPIPHANIES.md | 7 +++ .claude/board/LATEST_STATE.md | 2 + .claude/board/TECH_DEBT.md | 4 ++ crates/lance-graph-ontology/src/registry.rs | 62 +++++++++++++++++++++ 4 files changed, 75 insertions(+) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 3f82a015..cd12c4b9 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -7,6 +7,13 @@ **Honest scope.** The curated-only manifest resolves a *small* set today — most curated models inherit *uncurated* mixins (`mail.activity.mixin`, `sequence.mixin`), so few overrides have both ends present. Not fabricated: the resolver is total and the breadth grows as mixins are curated OR the SPO manifest is supplied. This is the same Curated-leg / Extracted-leg convergence the structural correction established, now for the derived relation. **Wishlist status:** all six items are now home-correct — P0/P1/P1b/P2/P3 as typed-Core facts + Extracted-leg breadth (#523/#526/#527/#530/#532), and `virtually_overrides` as this computed ClassView relation. None of them is a standalone flat-ndjson harvest predicate; the predicate-bolt-on cadence is fully retired. Cross-ref: `mro.rs` module doc; E-ODOO-CORE-FIRST-STRUCTURAL; the #531 `RegistryClassView` (the contract-side ClassView this resolution feeds). +## 2026-06-18 — E-OGAR-ONTOLOGY-WIRED-1 — the OGAR→ontology link is closed: NodeGuid → ClassView now resolves (audit found the gap, this lays the join) + +**Status:** FINDING (in-crate, tested). The OGAR→lance-graph-ontology audit this session found `NiblePath::from_guid_prefix` (the canon GUID→NiblePath fold) and `OntologyRegistry`'s `entity_type ↔ NiblePath` bijection BOTH built with **ZERO callers** — the two halves of the bridge forged, never chained. `OntologyRegistry::class_id_for_guid(&NodeGuid) -> Option` lays the join: `from_guid_prefix(guid)? → entity_type_of(path)`. A node row carrying a `classid` now resolves its ontology class (`entity_type`/`ClassId`), which `RegistryClassView` already turns into the class shape (fields/labels/template/DOLCE). + +**Aligned with E-ODOO-CORE-FIRST-STRUCTURAL (below).** That entry says structural facts belong in the *typed OGAR Core*, not bolted-on SPO predicates. This wiring is the Core-side resolution it implies — the GUID→ClassView path makes the typed Core reachable. It adds NO predicate and NO new type, just a method composing two existing surfaces. + +**Consistency invariant pinned.** The audit flagged a latent hazard: `from_guid_prefix` *derives* a path from the GUID's low-u16 classid; `register_class_path` *stores* an externally-supplied path — nothing guaranteed they coincide. The round-trip test (`class_id_for_guid_wires_ogar_guid_to_classview`) pins it: `register_class_path(t, from_guid_prefix(guid))` ⟹ `class_id_for_guid(guid) == Some(t)`; zero-fallback (unbound GUID → None); lossy fold refused upstream (high classid u16 → None, no silent collision). With this, all three "classid → X" axes are reachable from a GUID: read-mode (`classid_read_mode`, via ocr.rs), methods (`codegen_manifest::methods_for`, via the unicharset keystone E-CPP-KEYSTONE-1), and now ontology-shape. Branch work. ## 2026-06-18 — E-ODOO-CORE-FIRST-STRUCTURAL — structural Odoo facts belong in the typed OGAR Core, NOT the SPO harvest; the predicate-bolt-on cadence was drift diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index a4f1d8c4..42df4f73 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -10,6 +10,8 @@ --- +> **2026-06-18 — branch work** — **OGAR → lance-graph-ontology wiring closed.** `OntologyRegistry::class_id_for_guid(&NodeGuid) -> Option` composes the canon GUID→NiblePath fold (`contract::hhtl::NiblePath::from_guid_prefix`) with the registry's `NiblePath ↔ entity_type` bijection — the single missing join an audit this session surfaced (both halves were built with **ZERO callers**). A node carrying a classid now resolves its ontology class → `RegistryClassView` (fields/labels/template/DOLCE). Round-trip test pins the `classid_lo ↔ entity_type` consistency the audit flagged; zero-fallback (unbound → None) + lossy-fold refusal (high classid u16 → None). Completes the third "classid → X" axis reachable from a GUID (read-mode ✅ ocr.rs, methods ✅ unicharset keystone, ontology-shape ✅ now); aligns with `E-ODOO-CORE-FIRST-STRUCTURAL` (Core-side resolution, no new predicate/type). 16 ontology tests green; `registry.rs` clippy-clean + fmt clean. EPIPHANIES `E-OGAR-ONTOLOGY-WIRED-1`. Pre-existing `lance-graph-ontology` clippy debt noted (`TD-ONTOLOGY-LINT`). +> > **2026-06-17 — IN PR (branch `claude/odoo-spo-fk-target-deep-reads`)** — Odoo SPO corpus enrichment (odoo-rs `UPSTREAM_WISHLIST` P1 + coupled P0). The corpus `crates/lance-graph/src/graph/spo/odoo_ontology.spo.ndjson` now carries **two new predicate families** (was 7 predicates: `depends_on / emitted_by / has_function / raises / rdf:type / reads_field / traverses_relation`): **`target`** (618) + **`inverse_name`** (102) — the relational comodel/inverse keyed by the relation IRI, ruff#18 sibling-triple shape `(odoo:account_move.line_ids, target, "account.move.line")`; and **+736 deep `reads_field`** (so `reads_field` 2 095 → 2 831) — each `@api.depends('rel.leaf', …)` resolved through the new target map and lifted onto the field's emitting method as a transitive read. Corpus 22 245 → **23 701** triples. New stdlib-only generator `tools/odoo-blueprint-extractor/odoo_blueprint_extractor/spo_enrich.py` (+14 unit tests) reads `/home/user/odoo/addons` (the same source the ORM extractor parses) to build the `(model, field) → (comodel, inverse)` map; additive, deterministic, idempotent. `odoo_ontology.rs` doc + tests updated (count 23 701, histogram incl. new predicates, 2 new enrichment tests); `action_emitter`/`spo` unaffected (function count 3 328 unchanged). **Cross-repo finding (verified, not faked):** the deep reads make the cross-model recompute-ordering edge `account_move_line._compute_amount_residual → account_move._compute_amount` *visible* to `od_ontology::RecomputeDag` (baseline: 0 cross-model compute edges → enriched: 27), delivering the wishlist's P0 ask — but the audit's MISSED-1 is a unidirectional *ordering edge*, NOT a cycle, so odoo-rs's `slice_2_compute_subset_no_cross_model_cycle` no-cycle assertion legitimately still holds (the "circularity" is semantic, not a `reads_field`↔`emitted_by` back-edge). The corpus's original generator (`emit_ontology2.py`/`methods.parquet`) is absent from the tree — only its output is committed; enrichment runs at the correct additive stage over the shipped corpus + present source. See `EPIPHANIES.md` E-ODOO-FK-DEEP-READS. > > **2026-06-17 — MERGED #521** (lance-graph-contract: C++ codegen target `MethodSig` + `UniCharSet` content store): **+940/-4 across 8 files, additive to `lance-graph-contract` only** (zero `NodeRow`/`ValueTenant`/`ValueSchema`/stride/`ENVELOPE_LAYOUT_VERSION` impact). The Core-side of the Tesseract C++→Rust transcode. **`codegen_manifest`** — `MethodSig` (the `&'static`-backed, `const`-constructible method-signature type the generated Rust names; the method-axis sibling of `ClassView`'s field projection, distinct because `ClassView::FieldRef` is `String`-backed and can't be `const`) + `ClassMethods` + `methods_for`. **`unicharset`** — `UniCharSet` (deepnsm::Vocabulary-shaped id↔unichar bijection), `.unicharset` parser, `id_to_unichar`/`unichar_to_id`/`dump`, **zero leptonica** (pure text, never touches `Pix`). **PROBE-OGAR-ADAPTER-UNICHARSET → FINDING:** the pipeline (ruff `ruff_cpp_spo` harvest → `reassemble` → `ruff_cpp_codegen` → these types) produces a `UniCharSet` **byte-identical 112/112** to the C++ libtesseract oracle on real `eng` data — the core-first transcode doctrine is now empirically proven end-to-end (doctrine flipped CONJECTURE→FINDING). The sole id-0 diff was the `NULL`→space convention (`unicharset.cpp:882`), fixed + locked by `null_token_maps_to_space` (codex P1 flagged it independently; resolved + thread closed). Pairs with **ruff #20** (harvester + codegen side, merged same day). 644 contract lib green; clippy `-D warnings` + fmt clean. Branch `claude/happy-hamilton-0azlw4`, merge `620bd8e`. **Next honest increment:** wire `UniCharSet` lookups through `classid → ClassView → UnifiedStep` (the `classid → &UniCharSet` resolver), per-leaf one-`diff` parity. diff --git a/.claude/board/TECH_DEBT.md b/.claude/board/TECH_DEBT.md index 40d44f66..e0e9fd69 100644 --- a/.claude/board/TECH_DEBT.md +++ b/.claude/board/TECH_DEBT.md @@ -15,6 +15,10 @@ ## Open Debt +### TD-ONTOLOGY-LINT — `lance-graph-ontology` pre-existing clippy (12) on toolchain 1.95 (2026-06-18) + +`cargo clippy -p lance-graph-ontology -- -D warnings` exits 101 with 12 errors on the pinned 1.95 toolchain — all **pre-existing on `main`** (e.g. `odoo_blueprint/op_emitter.rs:182` is byte-identical on `origin/main`), in `hydrators/owl.rs` (2), `odoo_blueprint/op_emitter.rs` (1), `ttl_parse.rs` (3), + others. Mostly mechanical (`iter_cloned_collect` → `.to_vec()`, etc.). The crate is not in the CI clippy sweep ("CI tests 4 of ~30 crates"), so the debt accumulated un-gated. Surfaced while wiring `class_id_for_guid` (E-OGAR-ONTOLOGY-WIRED-1; `registry.rs` itself is clippy-clean + fmt-clean). Fix is a focused lint pass, out of scope for the wiring increment. Same class as `TD-CAUSAL-EDGE-LINT`. + ### TD-CI-LINUX-BUILD-DEBUGINFO0 — `linux-build` was the last job missing the link-cliff mitigation; debuginfo=0 + mold extended to it (2026-06-18) **Paid (this commit).** `build.yml`'s `linux-build (stable)` job linked the full diff --git a/crates/lance-graph-ontology/src/registry.rs b/crates/lance-graph-ontology/src/registry.rs index 5ebc64a0..0bc1da1c 100644 --- a/crates/lance-graph-ontology/src/registry.rs +++ b/crates/lance-graph-ontology/src/registry.rs @@ -25,6 +25,8 @@ use crate::proposal::{ }; use crate::semantic_types::SemanticTypeMap; use crate::ttl_parse::{parse_ttl_directory_with_provenance, ttl_root_checksum}; +use lance_graph_contract::canonical_node::NodeGuid; +use lance_graph_contract::class_view::ClassId; use lance_graph_contract::hhtl::NiblePath; use lance_graph_contract::property::{Marking, SemanticType}; use std::collections::HashMap; @@ -420,6 +422,29 @@ impl OntologyRegistry { self.inner.read().unwrap().type_by_path.get(&path).copied() } + /// **OGAR → ClassView wiring.** Resolve an OGAR node's [`NodeGuid`] to its + /// ontology class (`entity_type` / [`ClassId`]) — the single link that lets a + /// node row carrying a `classid` reach its class shape through + /// [`RegistryClassView`](crate::class_resolver::RegistryClassView), which keys + /// on the returned `entity_type`. + /// + /// Composes the two halves that already existed but were never chained: the + /// canon GUID→NiblePath fold ([`NiblePath::from_guid_prefix`] — root-first + /// `classid_lo · HEEL · HIP · TWIG`) and this registry's `NiblePath → + /// entity_type` bijection ([`Self::entity_type_of`]). + /// + /// Returns `None` when the GUID's high `classid` u16 is nonzero (the fold + /// refuses the lossy case — no silent collision) OR when no class is + /// registered at the folded path (zero-fallback: an unbound GUID resolves to + /// no class, never a panic). The `classid_lo ↔ entity_type` correspondence is + /// the registrar's invariant: `register_class_path(t, + /// NiblePath::from_guid_prefix(guid))` makes `class_id_for_guid(guid) == + /// Some(t)` (see `class_id_for_guid_wires_ogar_guid_to_classview`). + pub fn class_id_for_guid(&self, guid: &NodeGuid) -> Option { + let path = NiblePath::from_guid_prefix(guid)?; + self.entity_type_of(path) + } + /// Export the registry to an OGIT-shaped TTL fragment for the named /// namespace. Used by the Lance ↔ OGIT round-trip and for fork PRs /// that promote schema-scanner suggestions back into the canonical @@ -873,6 +898,43 @@ mod tests { .expect("re-registering the same pair is idempotent"); } + #[test] + fn class_id_for_guid_wires_ogar_guid_to_classview() { + let reg = OntologyRegistry::new_in_memory(); + let a = reg + .append_mapping(proposal("ogit.Tess:UniCharSet")) + .unwrap(); + let t = a.schema_ptr.entity_type_id(); + + // An OGAR node GUID (classid in the low u16 + the HHT cascade). The + // registrar pairs the class with the GUID-DERIVED path — that is the + // `classid_lo ↔ entity_type` consistency the wiring relies on. + let g = NodeGuid::new(0x0000_ABCD, 0x1234, 0x5678, 0x9ABC, 1, 2); + let path = NiblePath::from_guid_prefix(&g).expect("classid_lo only ⇒ Some"); + reg.register_class_path(t, path).unwrap(); + + // The wiring: GUID → from_guid_prefix → entity_type_of → ClassId. + assert_eq!(reg.class_id_for_guid(&g), Some(t)); + // Equal to the explicit composition (documents exactly what it does). + assert_eq!( + reg.class_id_for_guid(&g), + reg.entity_type_of(NiblePath::from_guid_prefix(&g).unwrap()), + ); + + // Zero-fallback: a GUID whose folded path was never registered → None. + let unbound = NodeGuid::new(0x0000_1111, 0, 0, 0, 0, 0); + assert_eq!(reg.class_id_for_guid(&unbound), None); + + // The fold refuses a high-classid-u16 GUID upstream → None (no silent + // collision onto a folded path). + let high = NodeGuid::new(0xDEAD_BEEF, 0, 0, 0, 0, 0); + assert_eq!( + reg.class_id_for_guid(&high), + None, + "high classid u16 ⇒ lossy fold refused, so no class resolves" + ); + } + #[test] fn class_path_bijection_conflicts_are_rejected() { let reg = OntologyRegistry::new_in_memory(); From 5ed5a1fa5c3ab0b81fafade4441ffc620d5aa1e6 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 18 Jun 2026 07:45:21 +0000 Subject: [PATCH 5/5] unichar: reject truncated trailing multibyte sequences (codex P2 on #534) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit utf8_to_utf32 only checked the lead byte was legal; for a truncated trailing multibyte lead (e.g. [0xC3], or a 3-byte lead with 2 bytes present) it still called first_uni on the short slice, where take(len) decodes from the partial bytes and the offset subtraction fabricates a codepoint ([0xC3] -> Some([64])). The C++ UTF8ToUTF32 reads past its buffer here (UB on length-delimited input); this length-delimited decoder now rejects it (i + step > len -> None) instead of fabricating. Byte-parity unchanged (268/268 vs the libtesseract oracle — the corpus has no truncated cases). +1 test (truncated_trailing_multibyte_is_rejected); docs on utf8_to_utf32 + first_uni updated. clippy --all-targets -D warnings + fmt clean. Co-Authored-By: Claude https://claude.ai/code/session_016b33swuXE23hKtqxsHu9p1 --- crates/lance-graph-contract/src/unichar.rs | 48 ++++++++++++++++------ 1 file changed, 36 insertions(+), 12 deletions(-) diff --git a/crates/lance-graph-contract/src/unichar.rs b/crates/lance-graph-contract/src/unichar.rs index 3a4c7e5a..c08c22d9 100644 --- a/crates/lance-graph-contract/src/unichar.rs +++ b/crates/lance-graph-contract/src/unichar.rs @@ -62,12 +62,11 @@ pub const fn utf8_step(lead: u8) -> u8 { const UTF8_OFFSETS: [i32; 5] = [0, 0, 0x3080, 0xE2080, 0x3C8_2080]; /// The Unicode codepoint of the first character in `bytes`, mirroring -/// `UNICHAR::first_uni` (`unichar.cpp:105`). `bytes` must be non-empty and -/// begin with a legal lead whose continuation bytes are present — the only -/// caller, [`utf8_to_utf32`], guarantees the lead is legal before calling. -/// If the buffer is shorter than the lead implies (a truncated trailing -/// character) this decodes only the bytes present rather than reading past the -/// slice; that case is out of the documented byte-parity scope. +/// `UNICHAR::first_uni` (`unichar.cpp:105`). The only caller, [`utf8_to_utf32`], +/// guarantees `bytes` is non-empty, begins with a legal lead, AND contains all +/// `utf8_step(bytes[0])` bytes of that character (it rejects truncated trailing +/// sequences *before* calling). The `.take(len)` is therefore defensive — it can +/// never read past the slice even if that guarantee were ever violated. fn first_uni(bytes: &[u8]) -> i32 { let len = utf8_step(bytes[0]) as usize; let mut uni: i32 = 0; @@ -84,12 +83,15 @@ fn first_uni(bytes: &[u8]) -> i32 { /// `UNICHAR::UTF8ToUTF32` (`unichar.cpp:220`). /// /// Returns `None` if any character's **lead byte** is illegal (the C++ -/// "return an empty vector" path). Like the C++, continuation bytes are NOT -/// re-validated, so the overlong NUL `C0 80` decodes to `[0]` rather than being -/// rejected (see the module docs). Empty input is `Some(vec![])` — distinct -/// from the `None` illegal case, which the C++ conflates as an empty vector -/// (this is the one place the Rust surface is strictly clearer; the corpus -/// avoids empty input so the byte-parity diff is unaffected). +/// "return an empty vector" path) OR if the input ends mid-character (a trailing +/// multibyte lead whose continuation bytes are not all present). The C++ reads +/// past its buffer in that truncation case (UB on a length-delimited slice); +/// this length-delimited decoder rejects it instead of fabricating a codepoint +/// from the partial bytes. Like the C++, continuation bytes that ARE present are +/// not re-validated, so the overlong NUL `C0 80` decodes to `[0]` rather than +/// being rejected (see the module docs). Empty input is `Some(vec![])` — +/// distinct from the `None` illegal case, which the C++ conflates as an empty +/// vector (the corpus avoids empty input so the byte-parity diff is unaffected). #[must_use] pub fn utf8_to_utf32(bytes: &[u8]) -> Option> { let mut out = Vec::new(); @@ -99,6 +101,9 @@ pub fn utf8_to_utf32(bytes: &[u8]) -> Option> { if step == 0 { return None; // illegal lead } + if i + step > bytes.len() { + return None; // truncated trailing multibyte sequence + } out.push(first_uni(&bytes[i..])); i += step; } @@ -163,6 +168,25 @@ mod tests { assert_eq!(utf8_to_utf32(&[b'A', 0xF8]), None); // legal then illegal } + /// A truncated trailing multibyte sequence (legal lead, but its continuation + /// bytes are not all present) is rejected — NOT decoded into a fabricated + /// codepoint. The C++ reads past its buffer here (UB); this length-delimited + /// decoder returns `None`. (Codex P2 on PR #534.) + #[test] + fn truncated_trailing_multibyte_is_rejected() { + assert_eq!(utf8_to_utf32(&[0xC3]), None, "2-byte lead, 1 byte present"); + assert_eq!(utf8_to_utf32(&[0xE4, 0xB8]), None, "3-byte lead, 2 present"); + assert_eq!( + utf8_to_utf32(&[0xF0, 0x9F, 0x98]), + None, + "4-byte lead, 3 present" + ); + // Truncation after a valid char is rejected too (the whole decode fails). + assert_eq!(utf8_to_utf32(&[b'a', 0xE4, 0xB8]), None); + // A COMPLETE multibyte char is still accepted (regression guard). + assert_eq!(utf8_to_utf32(&[0xC3, 0xA9]), Some(vec![0xE9])); + } + /// The overlong NUL `C0 80` is accepted and decodes to `0` — the defining /// quirk the byte-parity probe confirmed against the C++ oracle. #[test]