From a2c2d09b52f99d56b1cd91c057944333303aa1de Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 13 Jun 2026 19:19:07 +0000 Subject: [PATCH 1/4] feat(lance-graph-turbovec): TurboQuant ANN index bridged onto the spine New excluded standalone crate wrapping the AdaWorldAPI turbovec fork (Google TurboQuant, arXiv 2504.19874). TurboVec exposes a Kernel::{NativeLut, PolyfillGemm} A/B switch over one index: the native hand-written nibble-LUT ADC, and the ndarray::simd::matmul_i8_to_i32 int8 GEMM (AMX-ready via ndarray's runtime dispatch). path-deps both the turbovec + ndarray forks; kept out of the main graph so turbovec's faer/statrs tree never enters the deterministic lance-graph compile path. KNOWLEDGE.md is the synergy map vs the bgz-tensor primitives the request named -- HDR popcount stacking early-exit (stacked.rs vedic cascade), Belichtungsmesser sigma confidence thresholds (belichtungsmesser.rs), preheating vs palette256 ranking (WeightPalette / prepare()) -- plus the measured finding that the polyfill GEMM is 11.4x slower than native: AMX is the wrong tool for this index because a gather is not a matmul. Placement verdict: the index belongs on the spine (lance-graph), the kernel math belongs in ndarray (which already owns clam/cam_pq/cascade/amx_matmul). The promising synergy is a Belichtungsmesser sigma-gate on the LUT scan, not AMX. Board hygiene (same commit): EPIPHANIES E-TURBOVEC-AMX-WRONG-TOOL-1, AGENT_LOG run entry, LATEST_STATE shipped entry, root Cargo.toml exclude. https://claude.ai/code/session_01D2WSmezQBNC3bUdHuGfGmo --- .claude/board/AGENT_LOG.md | 13 + .claude/board/EPIPHANIES.md | 44 + .claude/board/LATEST_STATE.md | 2 + Cargo.toml | 7 + crates/lance-graph-turbovec/Cargo.lock | 1188 ++++++++++++++++++++++ crates/lance-graph-turbovec/Cargo.toml | 37 + crates/lance-graph-turbovec/KNOWLEDGE.md | 178 ++++ crates/lance-graph-turbovec/src/lib.rs | 218 ++++ 8 files changed, 1687 insertions(+) create mode 100644 crates/lance-graph-turbovec/Cargo.lock create mode 100644 crates/lance-graph-turbovec/Cargo.toml create mode 100644 crates/lance-graph-turbovec/KNOWLEDGE.md create mode 100644 crates/lance-graph-turbovec/src/lib.rs diff --git a/.claude/board/AGENT_LOG.md b/.claude/board/AGENT_LOG.md index e281827c5..267745258 100644 --- a/.claude/board/AGENT_LOG.md +++ b/.claude/board/AGENT_LOG.md @@ -1,3 +1,16 @@ +## 2026-06-13 — turbovec ⇄ ndarray integration: fork-wired + ndarray::simd polyfill GEMM + measured AMX-vs-LUT + +**Main thread (Opus 4.8 1M) + 1 Opus general-purpose agent (bgz-tensor synergy map).** User: "create a crate in lance-graph for turbovec and check synergies; route SIMD through ndarray::simd (simd.rs→simd_amx/avx512/ops/soa); the polyfill does the work, ndarray ships AMX via byte-asm dispatch; pin rust 1.95." Cross-repo, branch `claude/wonderful-hawking-lodtql` in all three repos. + +**Shipped:** +- **turbovec** (the AdaWorldAPI fork of Google TurboQuant, arXiv 2504.19874): re-pointed `ndarray = "0.17"` (crates.io) → the AdaWorldAPI fork (`path = ../../ndarray`, `default-features=false, features=["std"]`) — P0 forks-only; the fork IS rust-ndarray 0.17.2 + HPC/SIMD so the array API is unchanged AND `ndarray::simd` is reachable. `blas` made opt-in (build.rs gates the OpenBLAS link on `CARGO_FEATURE_BLAS`; default uses pure-Rust matrixmultiply for the one encode `.dot()`). Added `rust-toolchain.toml` = 1.95.0. New `src/search_polyfill.rs` (feature `ndarray-simd`): TurboQuant scoring as a batched int8 GEMM `Q·X̂ᵀ` via `ndarray::simd::matmul_i8_to_i32` — zero raw intrinsics; ndarray picks AMX tile / VPDPBUSD / AVX-VNNI / scalar. `FORCE_SCALAR_FALLBACK` exposed under new `bench-internals` feature. `examples/kernel_speed.rs` (native vs polyfill vs scalar + recall). 2 polyfill tests green. +- **ndarray**: re-exported `hpc::amx_matmul::{matmul_i8_to_i32, amx_available}` through `simd.rs` (std-gated) so the AMX int8-GEMM ladder is reachable via the canonical `ndarray::simd::*` consumer surface (W1a). Additive; no behaviour change. +- **lance-graph**: new excluded standalone crate `crates/lance-graph-turbovec` (path-deps both forks) — `TurboVec` bridge with a `Kernel::{NativeLut, PolyfillGemm}` A/B switch + lazy reconstruction cache + `polyfill_backend()` report; 2 tests green. `KNOWLEDGE.md` = full synergy map. Root Cargo.toml `exclude` updated. EPIPHANIES E-TURBOVEC-AMX-WRONG-TOOL-1 + this entry + LATEST_STATE. + +**Measured (AVX-512+VNNI host, no AMX tiles; n=20k dim=512 k=10 4-bit):** native LUT-ADC 76 µs/q (recall 0.785) ; polyfill GEMM 867 µs/q (recall 0.764) ; scalar 6 267 µs/q. **polyfill 11.4× slower than native** → TurboQuant deliberately trades the matmul away (LUT gather, not dot), so AMX accelerates the op it removed. Native LUT stays the production kernel; polyfill retained as AMX-ready baseline. Placement verdict: index → spine (lance-graph), kernel-math → ndarray (already owns clam/cam_pq/cascade/amx_matmul). The promising synergy is a Belichtungsmesser σ-gate on the LUT scan, NOT AMX. + +**Verification:** `cargo build --lib -p turbovec` (fork-wired) green; `cargo test -p turbovec --features ndarray-simd search_polyfill` 2/2 green; `cargo test --manifest-path crates/lance-graph-turbovec/Cargo.toml` green; benchmark ran. Pre-existing upstream turbovec dead-code warning (`avx2_block_epilogue`) silenced minimally. Commits: one per repo on the branch. + ## 2026-06-11 — tombstone commit: emission artifacts removed per PR #477 follow-up **Main thread (Fable, session splat3d-cpu-simd-renderer).** Executed the PR #477 documented follow-up (the "what does NOT exist" table → source reality): removed `CollapseGateEmission` from `lance-graph-contract::collapse_gate` (+ lib.rs re-export; `MailboxId`/`MergeMode`/`GateDecision` survive), removed `MailboxSoA::emit()`, renamed `last_emission_cycle` → `last_active_cycle`, added in-place `consume_firing(row)` successor (same threshold + same-cycle-idempotency semantics, no carrier object), reworded 4 stale doc references (kanban/episodic_edges/witness_tombstone/mailbox_soa header), superseded the CLAUDE.md Baton-scoping block, fixed cycle-coherent-soa-snapshot-v1 D-SOA-SNAP-1/2 to generic `SnapshotProvider::Column` (closes #477 CodeRabbit Critical — contract stays zero-dep), closed TD-COLLAPSE-GATE-SMALLVEC-1 as moot. Verified #477 codex P2 (`verify_layout` ColumnOutOfBounds) already fixed on main with regression test. Tests: contract 594 (−8 emission, +2 gate/merge), driver 85 (emit tests → consume tests, +1 OOB), clippy clean, workspace check clean. Commit: in PR. diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index cdf145c32..3a3f949a4 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,47 @@ +## 2026-06-13 — E-TURBOVEC-AMX-WRONG-TOOL-1 — AMX accelerates the operation TurboQuant deliberately removed + +**Status:** FINDING (benchmarked; AVX-512+VNNI host, `amx_available=false`). +**Confidence:** High — measured, with a mechanistic explanation that holds across the tier ladder. + +**The finding.** turbovec (Google TurboQuant, arXiv 2504.19874) was brought +onto the spine as `crates/lance-graph-turbovec` (excluded standalone, path-deps +the AdaWorldAPI turbovec + ndarray forks). Its scan was *also* expressed as a +batched int8 GEMM through `ndarray::simd::matmul_i8_to_i32` (the polyfill that +ships AMX `TDPBUSD` → AVX-512 VPDPBUSD → AVX-VNNI → scalar). Measured +(`n=20 000, dim=512, k=10, 4-bit`): + +| kernel | ns/query | recall@10 | +|---|---|---| +| native nibble-LUT ADC (AVX-512BW) | 76 073 | 0.785 | +| polyfill int8 GEMM (VPDPBUSD-zmm) | 866 899 | 0.764 | +| scalar reference | 6 267 279 | — | + +The polyfill GEMM is **11.4× slower** than the native LUT, and native is 82× +faster than scalar. **Mechanism:** TurboQuant's design *trades the matmul away* +— LUT-ADC is an O(1) table gather per coordinate; the GEMM does the full +`dim`-length dot per (query,vector) pair. AMX is a tile *matrix-multiply* unit, +so it accelerates exactly the operation TurboQuant removed. The AMX tile (256 +MAC/instr, ~4× VNNI) would bring the polyfill from 11.4× → ~3× slower — still a +loss. **A gather is not a matmul; no tile engine makes it one.** + +**Consequences.** +- Keep the native LUT kernel as turbovec's production path. The polyfill is + retained only as (a) proof the index is `ndarray::simd`-clean / AMX-ready and + (b) a measured baseline. AMX is the right tool only where the workload is + genuinely matmul-shaped (e.g. an exact-rerank LEAF over a tiny survivor set). +- Generalises the I-VSA-IDENTITIES register lesson to *kernels*: match the SIMD + primitive to the algorithm's operation, not to peak MAC/instr. "Ship AMX via + dispatch" is correct *plumbing* (the polyfill does ship it), but plumbing + doesn't make the wrong-shaped op fast. +- The genuinely promising turbovec⇄bgz-tensor wiring is NOT AMX: it is a + Belichtungsmesser σ-gated block reject on the LUT scan (turbovec has only a + heap-min prune, no statistical threshold). See + `crates/lance-graph-turbovec/KNOWLEDGE.md` §3B. + +Cross-ref: `crates/lance-graph-turbovec/KNOWLEDGE.md` (full synergy map + +reproduce); `ndarray::hpc::amx_matmul::matmul_i8_to_i32` (the 4-tier ladder); +I-NOISE-FLOOR-JIRAK (the σ-threshold path inherits the Jirak obligation). + ## 2026-06-12 — E-OUTER-BOUNDARY-IS-ORM-1 — there is only one boundary, and it is ontology-mediated **Status:** FINDING (PR #487 tombstone commit makes this source-true; OGAR class + `SoaEnvelope` + Lance columnar I/O is the realized triangle). diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 7e6b27af2..5d08e121d 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -10,6 +10,8 @@ --- +> **2026-06-13 — shipped (autoattended, cross-repo)** (turbovec ⇄ ndarray): new excluded standalone crate **`crates/lance-graph-turbovec`** — Google TurboQuant (arXiv 2504.19874, the AdaWorldAPI `turbovec` fork) bridged onto the spine. `TurboVec` wraps `turbovec::TurboQuantIndex` with a `Kernel::{NativeLut, PolyfillGemm}` A/B switch. **Cross-repo (branch `claude/wonderful-hawking-lodtql` in turbovec + ndarray + lance-graph):** turbovec re-pointed from crates.io `ndarray 0.17` → the AdaWorldAPI fork (path, P0 forks-only; `blas` opt-in so default builds BLAS-free; `rust-toolchain.toml` = 1.95.0); new `turbovec::search_polyfill` (feature `ndarray-simd`) expresses scoring as a batched int8 GEMM via **`ndarray::simd::matmul_i8_to_i32`** (re-exported through `simd.rs` — AMX `TDPBUSD` tile → AVX-512 VPDPBUSD → AVX-VNNI → scalar, dispatched inside ndarray, zero intrinsics in turbovec). **Measured finding (E-TURBOVEC-AMX-WRONG-TOOL-1):** the polyfill GEMM is 11.4× SLOWER than the native nibble-LUT (TurboQuant trades the matmul away → AMX accelerates the op it removed); native LUT stays production, polyfill is the AMX-ready baseline. Placement: index → spine, kernel-math → ndarray (already owns clam/cam_pq/cascade/amx_matmul). Synergy map (HDR popcount stacking early-exit, Belichtungsmesser σ thresholds, preheating vs palette256) in `crates/lance-graph-turbovec/KNOWLEDGE.md`. Tests green in all three repos; benchmark via `examples/kernel_speed.rs`. NOT a merged PR yet (branch work). +> > **2026-06-03 — hardened (follow-up after #460)** (D-HELIX-1 wiring): `crates/helix` now takes **ndarray as a MANDATORY, non-optional git dependency** (`git = AdaWorldAPI/ndarray @ master`), replacing the optional `path` dep + `ndarray-hpc` feature. Why: (1) codex P2 — an optional *path* dep still forces Cargo to read the local sibling manifest at resolution, so a clean checkout failed before feature selection; (2) directive "ndarray is mandatory for lance-graph". `simd.rs` always uses `ndarray::simd` (no scalar fallback); the self-contained fork → no import cycle. 63 unit + 6 doctests green; clippy/fmt clean. See E-HELIX-NDARRAY-MANDATORY. > > **2026-06-03 — shipped (autoattended)** (D-HELIX-1): new standalone crate `crates/helix` — the golden-spiral **Place/Residue** codec from the user's `KNOWLEDGE.md`. HHTL = deterministic PLACE; helix = orthogonal RESIDUE. Pipeline: equal-area `√u` hemisphere placement (`HemispherePoint`) → stride-4-over-17 `CurveRuler` coupling → Fisher-Z/arctanh `Similarity` alignment → EULER_GAMMA hand-off → 256-palette `RollingFloor` quantise (occupancy-drift + version stamp) → 3-byte `ResidueEdge` endpoint pair; metric-safe L1 via 256×256 `DistanceLut` (`distance_adaptive`) + non-metric byte-Hamming `distance_heuristic`. `prove()` closes the 2-D discrepancy Open Item (companion to `jc::weyl`). Zero-dep default (`edition 2021`, empty `[workspace]`, root `exclude`); optional `ndarray-hpc` feature routes batch Fisher-Z through `ndarray::simd::simd_ln_f32`. **61 unit + 6 doctests green** on BOTH feature configs; clippy -D warnings + fmt clean. ~80% overlaps existing CERTIFIED primitives by design (clean-room, user-directed) — see `crates/helix/KNOWLEDGE.md` § Overlap & Consolidation + E-HELIX-OVERLAP + TD-HELIX-OVERLAP-1. Branch claude/gallant-rubin-Y9pQd. diff --git a/Cargo.toml b/Cargo.toml index b63ab4b58..f6f98657c 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -60,6 +60,13 @@ exclude = [ # machinery to a single-tier unigram pipeline (see crate README). # Verified via `cargo test --manifest-path crates/quasicryth-research/Cargo.toml`. "crates/quasicryth-research", + # TurboQuant ANN index (Google arXiv 2504.19874) bridged onto the spine — + # standalone, path-deps the AdaWorldAPI turbovec + ndarray forks. Kept out + # of the main graph so turbovec's faer/statrs tree never enters the + # deterministic lance-graph compile path. Both scoring kernels (native + # nibble-LUT ADC + ndarray::simd::matmul_i8_to_i32 polyfill GEMM) compiled. + # Verify via `cargo test --manifest-path crates/lance-graph-turbovec/Cargo.toml`. + "crates/lance-graph-turbovec", ] resolver = "2" diff --git a/crates/lance-graph-turbovec/Cargo.lock b/crates/lance-graph-turbovec/Cargo.lock new file mode 100644 index 000000000..22cd0cffa --- /dev/null +++ b/crates/lance-graph-turbovec/Cargo.lock @@ -0,0 +1,1188 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "approx" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cab112f0a86d568ea0e627cc1d6be74a1e9cd55214684db5561995f6dad897c6" +dependencies = [ + "num-traits", +] + +[[package]] +name = "arrayref" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" + +[[package]] +name = "arrayvec" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "bitflags" +version = "2.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" + +[[package]] +name = "blake3" +version = "1.8.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0aa83c34e62843d924f905e0f5c866eb1dd6545fc4d719e803d9ba6030371fce" +dependencies = [ + "arrayref", + "arrayvec", + "cc", + "cfg-if", + "constant_time_eq", + "cpufeatures 0.3.0", +] + +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + +[[package]] +name = "bytemuck" +version = "1.25.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8efb64bd706a16a1bdde310ae86b351e4d21550d98d056f22f8a7f7a2183fec" +dependencies = [ + "bytemuck_derive", +] + +[[package]] +name = "bytemuck_derive" +version = "1.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9abbd1bc6865053c427f7198e6af43bfdedc55ab791faed4fbd361d789575ff" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "cc" +version = "1.2.64" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dad887fd958be91b5098c0248def011f4523ab786cd411be668777e55063501f" +dependencies = [ + "find-msvc-tools", + "shlex", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "coe-rs" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e8f1e641542c07631228b1e0dc04b69ae3c1d58ef65d5691a439711d805c698" + +[[package]] +name = "constant_time_eq" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" + +[[package]] +name = "cpufeatures" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +dependencies = [ + "libc", +] + +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + +[[package]] +name = "crossbeam-deque" +version = "0.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9dd111b7b7f7d55b72c0a6ae361660ee5853c9af73f70c3c2ef6858b950e2e51" +dependencies = [ + "crossbeam-epoch", + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-epoch" +version = "0.9.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e" +dependencies = [ + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-utils" +version = "0.8.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" + +[[package]] +name = "crunchy" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "dbgf" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6ca96b45ca70b8045e0462f191bd209fcb3c3bfe8dbfb1257ada54c4dd59169" + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer", + "crypto-common", +] + +[[package]] +name = "dyn-stack" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf6fa63092e3ca9f602f6500fddd05502412b748c4c4682938565b44eb9e0066" +dependencies = [ + "bytemuck", +] + +[[package]] +name = "dyn-stack" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1c4713e43e2886ba72b8271aa66c93d722116acf7a75555cce11dcde84388fe8" +dependencies = [ + "bytemuck", + "dyn-stack-macros", +] + +[[package]] +name = "dyn-stack-macros" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e1d926b4d407d372f141f93bb444696142c29d32962ccbd3531117cf3aa0bfa9" + +[[package]] +name = "either" +version = "1.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e" + +[[package]] +name = "enum-as-inner" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a1e6a265c649f3f5979b601d26f1d05ada116434c87741c9493cb56218f76cbc" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "equator" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c35da53b5a021d2484a7cc49b2ac7f2d840f8236a286f84202369bd338d761ea" +dependencies = [ + "equator-macro 0.2.1", +] + +[[package]] +name = "equator" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4711b213838dfee0117e3be6ac926007d7f433d7bbe33595975d4190cb07e6fc" +dependencies = [ + "equator-macro 0.4.2", +] + +[[package]] +name = "equator-macro" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3bf679796c0322556351f287a51b49e48f7c4986e727b5dd78c972d30e2e16cc" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "equator-macro" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "44f23cf4b44bfce11a86ace86f8a73ffdec849c9fd00a386a53d278bd9e81fb3" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "faer" +version = "0.20.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2b19b8c3570ea226e507fe3dbae2aa9d7e0f16676abd35ea3adeeb9f90f7b5d" +dependencies = [ + "bytemuck", + "coe-rs", + "dbgf", + "dyn-stack 0.11.0", + "equator 0.4.2", + "faer-entity", + "gemm", + "generativity", + "libm", + "matrixcompare", + "matrixcompare-core", + "nano-gemm", + "npyz", + "num-complex", + "num-traits", + "paste", + "rand", + "rand_distr", + "rayon", + "reborrow", + "serde", +] + +[[package]] +name = "faer-entity" +version = "0.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "072f96f1bc8b2b30dfc26c086baeadb63aae08019b1ed84721809b9fd2006685" +dependencies = [ + "bytemuck", + "coe-rs", + "libm", + "num-complex", + "num-traits", + "pulp 0.18.22", + "reborrow", +] + +[[package]] +name = "find-msvc-tools" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" + +[[package]] +name = "gemm" +version = "0.18.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ab96b703d31950f1aeddded248bc95543c9efc7ac9c4a21fda8703a83ee35451" +dependencies = [ + "dyn-stack 0.13.2", + "gemm-c32", + "gemm-c64", + "gemm-common", + "gemm-f16", + "gemm-f32", + "gemm-f64", + "num-complex", + "num-traits", + "paste", + "raw-cpuid", + "seq-macro", +] + +[[package]] +name = "gemm-c32" +version = "0.18.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f6db9fd9f40421d00eea9dd0770045a5603b8d684654816637732463f4073847" +dependencies = [ + "dyn-stack 0.13.2", + "gemm-common", + "num-complex", + "num-traits", + "paste", + "raw-cpuid", + "seq-macro", +] + +[[package]] +name = "gemm-c64" +version = "0.18.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfcad8a3d35a43758330b635d02edad980c1e143dc2f21e6fd25f9e4eada8edf" +dependencies = [ + "dyn-stack 0.13.2", + "gemm-common", + "num-complex", + "num-traits", + "paste", + "raw-cpuid", + "seq-macro", +] + +[[package]] +name = "gemm-common" +version = "0.18.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a352d4a69cbe938b9e2a9cb7a3a63b7e72f9349174a2752a558a8a563510d0f3" +dependencies = [ + "bytemuck", + "dyn-stack 0.13.2", + "half", + "libm", + "num-complex", + "num-traits", + "once_cell", + "paste", + "pulp 0.21.5", + "raw-cpuid", + "rayon", + "seq-macro", + "sysctl", +] + +[[package]] +name = "gemm-f16" +version = "0.18.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cff95ae3259432f3c3410eaa919033cd03791d81cebd18018393dc147952e109" +dependencies = [ + "dyn-stack 0.13.2", + "gemm-common", + "gemm-f32", + "half", + "num-complex", + "num-traits", + "paste", + "raw-cpuid", + "rayon", + "seq-macro", +] + +[[package]] +name = "gemm-f32" +version = "0.18.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bc8d3d4385393304f407392f754cd2dc4b315d05063f62cf09f47b58de276864" +dependencies = [ + "dyn-stack 0.13.2", + "gemm-common", + "num-complex", + "num-traits", + "paste", + "raw-cpuid", + "seq-macro", +] + +[[package]] +name = "gemm-f64" +version = "0.18.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35b2a4f76ce4b8b16eadc11ccf2e083252d8237c1b589558a49b0183545015bd" +dependencies = [ + "dyn-stack 0.13.2", + "gemm-common", + "num-complex", + "num-traits", + "paste", + "raw-cpuid", + "seq-macro", +] + +[[package]] +name = "generativity" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2c81fb5260e37854d09d5c87183309fd8c555b75289427884b25660bc87a85e" + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "libc", + "wasi", +] + +[[package]] +name = "half" +version = "2.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b" +dependencies = [ + "bytemuck", + "cfg-if", + "crunchy", + "num-traits", + "zerocopy", +] + +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "lance-graph-turbovec" +version = "0.1.0" +dependencies = [ + "ndarray", + "turbovec", +] + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "libm" +version = "0.2.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" + +[[package]] +name = "matrixcompare" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "37832ba820e47c93d66b4360198dccb004b43c74abc3ac1ce1fed54e65a80445" +dependencies = [ + "matrixcompare-core", + "num-traits", +] + +[[package]] +name = "matrixcompare-core" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b0bdabb30db18805d5290b3da7ceaccbddba795620b86c02145d688e04900a73" + +[[package]] +name = "matrixmultiply" +version = "0.3.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a06de3016e9fae57a36fd14dba131fccf49f74b40b7fbdb472f96e361ec71a08" +dependencies = [ + "autocfg", + "rawpointer", +] + +[[package]] +name = "memchr" +version = "2.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" + +[[package]] +name = "nalgebra" +version = "0.32.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b5c17de023a86f59ed79891b2e5d5a94c705dbe904a5b5c9c952ea6221b03e4" +dependencies = [ + "approx", + "matrixmultiply", + "nalgebra-macros", + "num-complex", + "num-rational", + "num-traits", + "rand", + "rand_distr", + "simba", + "typenum", +] + +[[package]] +name = "nalgebra-macros" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "254a5372af8fc138e36684761d3c0cdb758a4410e938babcff1c860ce14ddbfc" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "nano-gemm" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb5ba2bea1c00e53de11f6ab5bd0761ba87dc0045d63b0c87ee471d2d3061376" +dependencies = [ + "equator 0.2.2", + "nano-gemm-c32", + "nano-gemm-c64", + "nano-gemm-codegen", + "nano-gemm-core", + "nano-gemm-f32", + "nano-gemm-f64", + "num-complex", +] + +[[package]] +name = "nano-gemm-c32" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a40449e57a5713464c3a1208c4c3301c8d29ee1344711822cf022bc91373a91b" +dependencies = [ + "nano-gemm-codegen", + "nano-gemm-core", + "num-complex", +] + +[[package]] +name = "nano-gemm-c64" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "743a6e6211358fba85d1009616751e4107da86f4c95b24e684ce85f25c25b3bf" +dependencies = [ + "nano-gemm-codegen", + "nano-gemm-core", + "num-complex", +] + +[[package]] +name = "nano-gemm-codegen" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "963bf7c7110d55430169dc74c67096375491ed580cd2ef84842550ac72e781fa" + +[[package]] +name = "nano-gemm-core" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fe3fc4f83ae8861bad79dc3c016bd6b0220da5f9de302e07d3112d16efc24aa6" + +[[package]] +name = "nano-gemm-f32" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4e3681b7ce35658f79da94b7f62c60a005e29c373c7111ed070e3bf64546a8bb" +dependencies = [ + "nano-gemm-codegen", + "nano-gemm-core", +] + +[[package]] +name = "nano-gemm-f64" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bc1e619ed04d801809e1f63e61b669d380c4119e8b0cdd6ed184c6b111f046d8" +dependencies = [ + "nano-gemm-codegen", + "nano-gemm-core", +] + +[[package]] +name = "ndarray" +version = "0.17.2" +dependencies = [ + "blake3", + "matrixmultiply", + "num-complex", + "num-integer", + "num-traits", + "paste", + "portable-atomic", + "portable-atomic-util", + "rawpointer", +] + +[[package]] +name = "npyz" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f0e759e014e630f90af745101b614f761306ddc541681e546649068e25ec1b9" +dependencies = [ + "byteorder", + "num-bigint", + "py_literal", +] + +[[package]] +name = "num-bigint" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a5e44f723f1133c9deac646763579fdb3ac745e418f2a7af9cd0c431da1f20b9" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] +name = "num-complex" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" +dependencies = [ + "bytemuck", + "num-traits", + "rand", +] + +[[package]] +name = "num-integer" +version = "0.1.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-rational" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", + "libm", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "ordered-float" +version = "4.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7bb71e1b3fa6ca1c61f383464aaf2bb0e2f8e772a1f01d486832464de363b951" +dependencies = [ + "num-traits", +] + +[[package]] +name = "paste" +version = "1.0.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" + +[[package]] +name = "pest" +version = "2.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e0848c601009d37dfa3430c4666e147e49cdcf1b92ecd3e63657d8a5f19da662" +dependencies = [ + "memchr", + "ucd-trie", +] + +[[package]] +name = "pest_derive" +version = "2.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11f486f1ea21e6c10ed15d5a7c77165d0ee443402f0780849d1768e7d9d6fe77" +dependencies = [ + "pest", + "pest_generator", +] + +[[package]] +name = "pest_generator" +version = "2.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8040c4647b13b210a963c1ed407c1ff4fdfa01c31d6d2a098218702e6664f94f" +dependencies = [ + "pest", + "pest_meta", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "pest_meta" +version = "2.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89815c69d36021a140146f26659a81d6c2afa33d216d736dd4be5381a7362220" +dependencies = [ + "pest", + "sha2", +] + +[[package]] +name = "portable-atomic" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" + +[[package]] +name = "portable-atomic-util" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" +dependencies = [ + "portable-atomic", +] + +[[package]] +name = "ppv-lite86" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" +dependencies = [ + "zerocopy", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "pulp" +version = "0.18.22" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a0a01a0dc67cf4558d279f0c25b0962bd08fc6dec0137699eae304103e882fe6" +dependencies = [ + "bytemuck", + "libm", + "num-complex", + "reborrow", +] + +[[package]] +name = "pulp" +version = "0.21.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96b86df24f0a7ddd5e4b95c94fc9ed8a98f1ca94d3b01bdce2824097e7835907" +dependencies = [ + "bytemuck", + "cfg-if", + "libm", + "num-complex", + "reborrow", + "version_check", +] + +[[package]] +name = "py_literal" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "102df7a3d46db9d3891f178dcc826dc270a6746277a9ae6436f8d29fd490a8e1" +dependencies = [ + "num-bigint", + "num-complex", + "num-traits", + "pest", + "pest_derive", +] + +[[package]] +name = "quote" +version = "1.0.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rand" +version = "0.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5ca0ecfa931c29007047d1bc58e623ab12e5590e8c7cc53200d5202b69266d8a" +dependencies = [ + "libc", + "rand_chacha", + "rand_core", +] + +[[package]] +name = "rand_chacha" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" +dependencies = [ + "ppv-lite86", + "rand_core", +] + +[[package]] +name = "rand_core" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" +dependencies = [ + "getrandom", +] + +[[package]] +name = "rand_distr" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32cb0b9bc82b0a0876c2dd994a7e7a2683d3e7390ca40e6886785ef0c7e3ee31" +dependencies = [ + "num-traits", + "rand", +] + +[[package]] +name = "raw-cpuid" +version = "11.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "498cd0dc59d73224351ee52a95fee0f1a617a2eae0e7d9d720cc622c73a54186" +dependencies = [ + "bitflags", +] + +[[package]] +name = "rawpointer" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "60a357793950651c4ed0f3f52338f53b2f809f32d83a07f72909fa13e4c6c1e3" + +[[package]] +name = "rayon" +version = "1.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fb39b166781f92d482534ef4b4b1b2568f42613b53e5b6c160e24cfbfa30926d" +dependencies = [ + "either", + "rayon-core", +] + +[[package]] +name = "rayon-core" +version = "1.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "22e18b0f0062d30d4230b2e85ff77fdfe4326feb054b9783a3460d8435c8ab91" +dependencies = [ + "crossbeam-deque", + "crossbeam-utils", +] + +[[package]] +name = "reborrow" +version = "0.5.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "03251193000f4bd3b042892be858ee50e8b3719f2b08e5833ac4353724632430" + +[[package]] +name = "safe_arch" +version = "0.7.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96b02de82ddbe1b636e6170c21be622223aea188ef2e139be0a5b219ec215323" +dependencies = [ + "bytemuck", +] + +[[package]] +name = "same-file" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" +dependencies = [ + "winapi-util", +] + +[[package]] +name = "seq-macro" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "sha2" +version = "0.10.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "digest", +] + +[[package]] +name = "shlex" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" + +[[package]] +name = "simba" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "061507c94fc6ab4ba1c9a0305018408e312e17c041eb63bef8aa726fa33aceae" +dependencies = [ + "approx", + "num-complex", + "num-traits", + "paste", + "wide", +] + +[[package]] +name = "statrs" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f697a07e4606a0a25c044de247e583a330dbb1731d11bc7350b81f48ad567255" +dependencies = [ + "approx", + "nalgebra", + "num-traits", + "rand", +] + +[[package]] +name = "syn" +version = "2.0.117" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "sysctl" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "01198a2debb237c62b6826ec7081082d951f46dbb64b0e8c7649a452230d1dfc" +dependencies = [ + "bitflags", + "byteorder", + "enum-as-inner", + "libc", + "thiserror", + "walkdir", +] + +[[package]] +name = "thiserror" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "turbovec" +version = "0.9.0" +dependencies = [ + "faer", + "ndarray", + "ordered-float", + "rand", + "rand_chacha", + "rand_distr", + "rayon", + "statrs", +] + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "ucd-trie" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2896d95c02a80c6d6a5d6e953d479f5ddf2dfdb6a244441010e373ac0fb88971" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "walkdir" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" +dependencies = [ + "same-file", + "winapi-util", +] + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "wide" +version = "0.7.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ce5da8ecb62bcd8ec8b7ea19f69a51275e91299be594ea5cc6ef7819e16cd03" +dependencies = [ + "bytemuck", + "safe_arch", +] + +[[package]] +name = "winapi-util" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" +dependencies = [ + "windows-sys", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "zerocopy" +version = "0.8.52" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce1022995ff5ff5d841ad7d994facc23098cd40152f2c1d11cd607c6f530653f" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.52" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ae7f38b72ec2a254e2b87ef277cf2cd4fb97cbebf944faa6f33354da0867930" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] diff --git a/crates/lance-graph-turbovec/Cargo.toml b/crates/lance-graph-turbovec/Cargo.toml new file mode 100644 index 000000000..a9422453e --- /dev/null +++ b/crates/lance-graph-turbovec/Cargo.toml @@ -0,0 +1,37 @@ +[package] +name = "lance-graph-turbovec" +version = "0.1.0" +edition = "2021" +license = "Apache-2.0" +publish = false +description = """ +lance-graph-turbovec: the TurboQuant data-oblivious scalar-quantization ANN +index (Google Research, arXiv 2504.19874) wired into the lance-graph spine. + +A thin bridge over the AdaWorldAPI `turbovec` fork that (1) pins turbovec to +the AdaWorldAPI `ndarray` fork (P0: forks only) so its scan can reach +`ndarray::simd`, and (2) exposes BOTH scoring kernels side by side — the +native nibble-LUT ADC and the `ndarray::simd::matmul_i8_to_i32` polyfill GEMM +(AMX TDPBUSD tile -> AVX-512 VPDPBUSD -> AVX-VNNI -> scalar, dispatched +inside ndarray) — so the lance-graph layer can A/B them. See KNOWLEDGE.md for +the synergy map (HDR popcount stacking early-exit, Belichtungsmesser sigma +thresholds, preheating vs palette256 ranking) and the measured speed delta. +""" + +# Both deps are LOCAL PATHS (not git): this crate must build against the +# in-tree branch changes to BOTH siblings — the turbovec ndarray-fork wiring +# and the `ndarray::simd::matmul_i8_to_i32` re-export that the polyfill needs. +# Standalone (own [workspace] + root `exclude`), so turbovec's faer/statrs +# tree never enters the deterministic lance-graph main compile graph; verify +# via `cargo test --manifest-path crates/lance-graph-turbovec/Cargo.toml`. +[dependencies] +# turbovec with the `ndarray-simd` polyfill path compiled in (the native +# kernel is always present); turbovec itself path-deps the ndarray fork. +turbovec = { path = "../../../turbovec/turbovec", default-features = false, features = ["ndarray-simd"] } +# The fork, for `ndarray::simd::amx_available()` in the backend report. Same +# canonical path turbovec resolves, so cargo unifies to one ndarray package. +ndarray = { path = "../../../ndarray", default-features = false, features = ["std"] } + +[dev-dependencies] + +[workspace] diff --git a/crates/lance-graph-turbovec/KNOWLEDGE.md b/crates/lance-graph-turbovec/KNOWLEDGE.md new file mode 100644 index 000000000..d63a0e0d9 --- /dev/null +++ b/crates/lance-graph-turbovec/KNOWLEDGE.md @@ -0,0 +1,178 @@ +# turbovec ⇄ ndarray/lance-graph — synergy map & measured findings + +> **READ BY:** anyone touching `lance-graph-turbovec`, the ndarray AMX/VNNI +> int8 GEMM (`hpc::amx_matmul`), or the bgz-tensor cascade primitives. +> **Status:** FINDING (benchmarked 2026-06-13 on AVX-512+VNNI, no AMX tiles). +> **Branch:** `claude/wonderful-hawking-lodtql` (turbovec + ndarray + lance-graph). + +## 0. What turbovec is, and where it goes + +`turbovec` = Google Research **TurboQuant** (arXiv 2504.19874, ICLR 2026): a +data-oblivious scalar-quantization ANN index. Pipeline: normalize → random +rotation (faer QR) → per-coord TQ+ calibration → Lloyd-Max 2/3/4-bit → +bit-pack → SIMD nibble-LUT scan (FAISS-FastScan-style `pshufb`/`vqtbl`, +32-vector blocks, u16 accumulators, in-register top-k prune + allowlist +early-exit). + +**Placement (answers "does it belong in ndarray?"): NO — it belongs on the +lance-graph spine.** It is a *search index* (IO, id-map, deletes, filtered +search), not a hardware primitive. It sits beside `bgz17` / `deepnsm` / +`bgz-tensor` as an `exclude`d standalone codec/search crate. What belongs in +**ndarray** is the *kernel math* — and ndarray already owns that substrate: +`hpc::clam` + `clam_search` (CLAM neighborhood), `hpc::cam_pq` (PQ-ADC), +`hpc::cascade` (HDR Welford banding), `hpc::amx_matmul` (AMX/VNNI int8 GEMM), +`simd::*` (the typed-lane polyfill). Integration rule: **index here, every +wide op borrowed from `ndarray::simd`.** + +P0 wiring: turbovec's `ndarray = "0.17"` (crates.io) is re-pointed to the +AdaWorldAPI fork (`path`), which *is* rust-ndarray 0.17.2 + HPC/SIMD — so the +core array API is unchanged AND `ndarray::simd` becomes reachable. The `blas` +feature is made opt-in (default uses the pure-Rust `matrixmultiply` fallback +for the one encode-time `.dot()`), so no OpenBLAS link is needed. Rust pinned +to 1.95 (`turbovec/rust-toolchain.toml`) to match the fork. + +## 1. The polyfill / AMX architecture (the "ship AMX via dispatch" path) + +turbovec writes **zero raw intrinsics** in the polyfill path. Scoring is +re-expressed as a batched int8 GEMM and handed to ndarray: + +``` +S[nq×n] = Q_i8[nq×dim] · X̂ᵀ_i8[dim×n] via ndarray::simd::matmul_i8_to_i32 +``` + +`matmul_i8_to_i32` (re-exported through `simd.rs`; impl in `hpc::amx_matmul`) +runs a 4-tier ladder, **bit-identical across tiers**, selected at runtime: + +| tier | instr | MAC/instr | when | +|---|---|---|---| +| 1 | AMX `TDPBUSD` tile (byte-asm) | 16 384 | `amx_available()` ∧ 16/16/64-aligned (Sapphire Rapids+) | +| 2 | AVX-512 `VPDPBUSD` zmm | 64 | `avx512vnni` | +| 3 | AVX-VNNI `VPDPBUSD` ymm | 32 | `avxvnni` (Arrow/Meteor Lake) | +| 4 | scalar i32 reference | 1 | otherwise | + +So a turbovec built on a Sapphire-Rapids host **ships AMX for free** — no code +change, no `#[cfg]` in turbovec. ndarray "knows how to enable AMX via byte +asm" (`ldtilecfg` / `TDPBUSD` encodings + `amx_available()` CPUID/XCR0/prctl +gating live in `hpc::amx_matmul` + `simd_amx`). The carrier for the tiles is +`hpc::blocked_grid::AmxInt8Grid = BlockedGrid` (the 16×64 TDPBUSD +half-square — the "gridlake SoA tiles"). `simd_soa::MultiLaneColumn` is the +layout-only 64B-aligned SoA byte carrier; `simd_ops` is the f32 slice layer. + +## 2. THE HEADLINE FINDING (measured) + +`n=20 000, dim=512, nq=256, k=10, 4-bit`, AVX-512+VNNI host (`amx_available = +false`), `cargo run --release --example kernel_speed --features +ndarray-simd,bench-internals` in turbovec: + +| kernel | ns/query | recall@10 | DB memory | +|---|---|---|---| +| native LUT-ADC (AVX-512BW) | **76 073** | 0.785 | 5 000 KB (4-bit) | +| polyfill GEMM (VPDPBUSD-zmm) | 866 899 | 0.764 | 10 000 KB (i8) | +| scalar reference | 6 267 279 | — | — | + +- native vs scalar: **82.4× faster** (the hand-written nibble-LUT kernel earns + its keep). +- polyfill GEMM vs native: **11.4× slower**. + +**Why, and the architectural lesson:** TurboQuant's design *deliberately +trades the matmul away*. LUT-ADC is an O(1) table gather per coordinate; the +GEMM does the full `dim`-length dot per (query,vector) pair. **AMX accelerates +exactly the operation TurboQuant removed.** Even VPDPBUSD (64 MAC/instr) +can't close the algorithmic gap; the AMX tile (256 MAC/instr, ~4× VNNI) would +bring the polyfill from 11.4× → ~3× slower — still a loss. A gather is not a +matmul, and no tile engine makes it one. + +**Decision: keep the native LUT kernel as the production path.** AMX is the +wrong tool for *this* index. The polyfill is retained because (a) it proves +the index is `ndarray::simd`-clean and AMX-ready should the workload ever +become matmul-shaped (e.g. an exact-rerank LEAF over a tiny survivor set — +see §3C), and (b) it is the honest measured baseline. Caveat on the 11.4×: +`matmul_i8_to_i32` re-allocates the u8 LHS + colsum-bias buffers per call; +a production AMX rerank would amortise those — but the work-ratio conclusion +(full dot ≫ LUT hit) dominates regardless. + +## 3. Synergy map — the three primitives the request named + +### (A) HDR popcount stacking early-exit +- **bgz-tensor home:** `stacked.rs` (`StackedBF16x4`, `SearchKey17`, + `vedic_cascade` 3-stage L1 thresholds), `stacked_n.rs`, `hdr_belichtung.rs` + (`PaletteCascade` over `ndarray::hpc::cascade`). NB the "popcount" name is + **vestigial** — `stacked_n.rs:158` deleted sign-Hamming as "not a valid + metric," replaced by palette-L1; true Hamming survives only in + `turboquant_kv.rs`. +- **turbovec analogue:** its early-exit is the per-block `block_has_allowed` + mask skip + the in-register `_mm256_cmp_ps(scores, heap_min)` threshold + prune (`avx2_post_flush_heap_update`). Same *shape* as the vedic cascade's + staged reject, but turbovec's thresholds are the running top-k min, not a + pre-calibrated band. +- **Synergy:** wrap turbovec's nibble-LUT scan as the **TWIG/LEAF** of + bgz-tensor's HEEL→HIP→TWIG→LEAF cascade — a `SearchKey17`/palette HEEL + rejects ~85% before turbovec ever runs. turbovec contributes the fast exact + ADC at the leaf; bgz-tensor contributes the coarse pre-reject it lacks. + +### (B) Belichtungsmesser statistical confidence-interval thresholds +- **bgz-tensor home:** `belichtungsmesser.rs` — `calibrate(&[u32])` computes + μ + σ of pairwise L1 distances → 12 half-σ bands → `classify`/`three_stroke` + σ-relative reject. Online σ + recalibration (`ShiftAlert`) delegated to + `ndarray::hpc::cascade`. **No Jirak grounding** — bands are hand-set indices + (open obligation per I-NOISE-FLOOR-JIRAK; any "N σ above noise floor" claim + on these palette-correlated distances needs a Jirak-2016 weak-dependence + bound, not classical Berry-Esseen). +- **turbovec gap → synergy:** turbovec has **no statistical threshold** — it + keeps a fixed top-k heap and prunes by the current heap-min only. A + `Belichtungsmesser` calibrated on a query's score distribution could supply + a **σ-gated block reject** (skip a 32-vector block whose best-possible score + is below μ+kσ) *in addition to* the heap-min prune — turning fixed-k search + into confidence-bounded early termination. This is the cleanest, lowest-risk + turbovec⇄bgz-tensor wiring and the one most worth prototyping next. + +### (C) preheating vs palette256 ranking / attention headers +- **palette256 ranking (bgz-tensor):** `WeightPalette::build(rows, k≤256)` — + 256-archetype CLAM codebook by *furthest-point sampling* (NOT k-means; + metric-safe radius bound) + L1 ranking; `Codebook4096` is the 2-level 6+6 + variant; `AttentionTable`/`CompiledHead` freeze all k² pairs into a u16 LUT + ("attention headers"). Ranking = nearest-archetype-by-L1. +- **preheating:** **absent in bgz-tensor** (tables are built once, eagerly; no + lazy/warm path) — but turbovec *has* it: `TurboQuantIndex::prepare()` eagerly + fills the `OnceLock` caches (rotation matrix, Lloyd-Max centroids, blocked + SIMD layout) so the first query pays no init cost. The polyfill mirrors this + with the lazily-built i8 reconstruction (`reconstruct_db_i8_transposed`, + cached in `TurboVec`'s `OnceLock`). +- **Synergy / contrast:** turbovec's `prepare()` is the *codec/layout* warmup + (data→SIMD-ready); palette256 ranking is a *coarse quantizer* (data→256 + archetypes). They compose: a palette256 HEEL ranks a query to a handful of + archetype buckets, then turbovec — **preheated** — runs the exact ADC only + within those buckets. "Preheating vs palette256" is not either/or: preheat + the leaf scanner, palette-rank the coarse router. + +## 4. Overlap / convergence (prior-art guard) + +turbovec's scoring is ~80% conceptual overlap with two things already present: +- `bgz-tensor::turboquant_kv.rs` (`TurboQuantEntry`/`TurboQuantKvCache`) — + literally "TurboQuant": gain-shape → i4 → sign-fingerprint → Hamming-cascade + → cosine rerank. The KV-cache cousin; uses sign-Hamming (the metric + `stacked_n` deleted — a `truth-architect`/`iron-rule-savant` call, not free). +- `ndarray::hpc::cam_pq::PackedDatabase::distance_batch_avx512` — already a + SIMD-LUT PQ-ADC scan with a 99%-rejection HEEL/branch/full cascade. + +**This crate does NOT re-implement those** — it brings the *upstream Google +index* in verbatim (the published recall/speed numbers are the value) and adds +only the bridge + the `ndarray::simd` polyfill. If a future session wants ONE +unified ADC, the convergence target is `cam_pq` (it already has the cascade); +turbovec's contribution there would be TQ+ calibration + the data-oblivious +rotation, not a third scan kernel. + +## 5. Reproduce + +```bash +# native vs polyfill vs scalar + recall (this host = VNNI): +cd /home/user/turbovec && cargo run --release --example kernel_speed \ + --features ndarray-simd,bench-internals +# on a Sapphire Rapids / AMX host, prepend RUSTFLAGS="-C target-cpu=native" +# and matmul_i8_to_i32 lights up the TDPBUSD tile path (amx_available → true). + +# bridge tests: +cargo test --manifest-path crates/lance-graph-turbovec/Cargo.toml +# polyfill recall + reconstruct tests: +cd /home/user/turbovec && cargo test -p turbovec --features ndarray-simd search_polyfill +``` diff --git a/crates/lance-graph-turbovec/src/lib.rs b/crates/lance-graph-turbovec/src/lib.rs new file mode 100644 index 000000000..9b556a0bd --- /dev/null +++ b/crates/lance-graph-turbovec/src/lib.rs @@ -0,0 +1,218 @@ +//! # lance-graph-turbovec — TurboQuant ANN index on the lance-graph spine +//! +//! [`turbovec`](https://arxiv.org/abs/2504.19874) is Google Research's +//! **TurboQuant**: a *data-oblivious* scalar-quantization index for vector +//! search — normalize → random rotation → per-coordinate (TQ+) calibration → +//! Lloyd-Max 2/3/4-bit quantization → bit-pack → SIMD nibble-LUT scan. No +//! training, no rebuilds; a 10 M-doc f32 corpus (31 GB) fits in ~4 GB. +//! +//! This crate is the **thin bridge** that fits turbovec into the workspace. +//! +//! ## Placement — why lance-graph, not ndarray +//! +//! turbovec is a *search index* (IO, id-map, deletes, filtered search), not a +//! hardware SIMD primitive. Per the stack split it belongs on the **spine** +//! (lance-graph), a sibling to `bgz17` / `deepnsm` / `bgz-tensor` — the +//! standalone codec/search crates — and is `exclude`d from the main workspace +//! exactly like them. What belongs in **ndarray** is the *kernel*: and indeed +//! ndarray already owns the relevant ANN substrate — `hpc::clam` / +//! `clam_search` (CLAM neighborhood), `hpc::cam_pq` (PQ-ADC), `hpc::cascade` +//! (HDR), and `hpc::amx_matmul` (the AMX/VNNI int8 GEMM). So the integration +//! rule is: **the index lives here; every wide op is borrowed from +//! `ndarray::simd`.** +//! +//! ## Two kernels, one index — the [`Kernel`] switch +//! +//! - [`Kernel::NativeLut`] — turbovec's hand-written nibble-LUT ADC +//! (`pshufb`/`vqtbl` gather, 32-vector blocks, runtime AVX-512BW / AVX2 / +//! NEON / scalar). The fast path; lowest memory (2–4 bits/dim). +//! - [`Kernel::PolyfillGemm`] — TurboQuant scoring re-expressed as a batched +//! int8 GEMM `Q·X̂ᵀ` routed through [`ndarray::simd::matmul_i8_to_i32`]. +//! turbovec writes **zero** raw intrinsics; ndarray picks the backend: +//! **AMX `TDPBUSD` tile (byte-asm, 16 384 MAC/instr, Sapphire Rapids+) → +//! AVX-512 VPDPBUSD → AVX-VNNI → scalar**, all bit-identical. This is the +//! "ship AMX through dispatch" path — it lights up AMX on capable silicon +//! with no code change. +//! +//! ### The headline synergy finding (measured) +//! +//! On an AVX-512+VNNI host (no AMX tiles), `n=20 000, dim=512, 4-bit`: +//! +//! | kernel | ns/query | recall@10 | DB memory | +//! |---|---|---|---| +//! | `NativeLut` (AVX-512BW) | **76 073** | 0.785 | 5 000 KB (4-bit) | +//! | `PolyfillGemm` (VPDPBUSD-zmm) | 866 899 | 0.764 | 10 000 KB (i8) | +//! | scalar reference | 6 267 279 | — | — | +//! +//! The polyfill GEMM is **11.4× slower** than the native LUT — **because +//! TurboQuant's whole design trades the matmul away.** LUT-ADC is an O(1) +//! table gather per coordinate; the GEMM does the full `dim`-length dot per +//! (query, vector) pair. AMX accelerates *exactly the operation TurboQuant +//! removed*, so even VPDPBUSD (and, extrapolating, the 4×-wider AMX tile) +//! cannot close the algorithmic gap. **Conclusion: keep the native LUT +//! kernel; AMX is the wrong tool for this index.** The polyfill is retained +//! as a measured baseline and as proof the index is `ndarray::simd`-clean. +//! +//! See `KNOWLEDGE.md` for the full synergy map against the bgz-tensor +//! primitives (HDR popcount stacking early-exit, Belichtungsmesser sigma +//! confidence thresholds, preheating vs palette256 ranking). + +pub use turbovec::{AddError, ConstructError, IdMapIndex, SearchResults, TurboQuantIndex}; + +use std::sync::OnceLock; +use turbovec::TurboQuantIndex as Inner; + +/// Which scoring kernel [`TurboVec::search`] dispatches. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Kernel { + /// turbovec's native nibble-LUT ADC (hand-written AVX-512BW / AVX2 / NEON + /// / scalar). Fast path, lowest memory. + NativeLut, + /// `ndarray::simd::matmul_i8_to_i32` int8-GEMM scoring. SIMD/AMX backend + /// chosen inside ndarray; zero intrinsics in turbovec. + PolyfillGemm, +} + +/// lance-graph-facing bridge over [`turbovec::TurboQuantIndex`]. +/// +/// Adds (1) the [`Kernel`] A/B switch, (2) a lazily-built reconstruction +/// cache for the polyfill GEMM (the analogue of turbovec's native blocked +/// cache), and (3) a [`TurboVec::polyfill_backend`] report of which +/// `ndarray::simd` int8 tier this host dispatches. +pub struct TurboVec { + inner: Inner, + /// Transposed i8 reconstruction `X̂ᵀ` for the polyfill GEMM, built once + /// on first polyfill search and reset on `add` (interior mutability so + /// `search` stays `&self`, matching turbovec's concurrent-search design). + db_i8_t: OnceLock>, +} + +impl TurboVec { + /// Construct with a fixed dimensionality. `bit_width ∈ {2,3,4}`, + /// `dim` a positive multiple of 8. + pub fn new(dim: usize, bit_width: usize) -> Result { + Ok(Self { + inner: Inner::new(dim, bit_width)?, + db_i8_t: OnceLock::new(), + }) + } + + /// Add a flat `n * dim` batch of f32 vectors. Invalidates the polyfill + /// reconstruction cache. + pub fn add(&mut self, vectors: &[f32]) { + self.inner.add(vectors); + self.db_i8_t = OnceLock::new(); + } + + /// Top-`k` search via the chosen [`Kernel`]. Safe to call from multiple + /// threads (`&self`); the polyfill path lazily materialises its + /// reconstruction on first use. + pub fn search(&self, queries: &[f32], k: usize, kernel: Kernel) -> SearchResults { + match kernel { + Kernel::NativeLut => self.inner.search(queries, k), + Kernel::PolyfillGemm => { + let db = self + .db_i8_t + .get_or_init(|| self.inner.reconstruct_db_i8_transposed()); + self.inner.search_polyfill_with_db(queries, k, db) + } + } + } + + /// Number of indexed vectors. + pub fn len(&self) -> usize { + self.inner.len() + } + + /// Whether the index holds zero vectors. + pub fn is_empty(&self) -> bool { + self.inner.is_empty() + } + + /// Borrow the underlying [`turbovec::TurboQuantIndex`] (write/load, ids, + /// filtered search, deletes — the full upstream surface). + pub fn inner(&self) -> &Inner { + &self.inner + } + + /// Which `ndarray::simd` int8-GEMM tier the [`Kernel::PolyfillGemm`] path + /// dispatches on THIS host. `"amx-tdpbusd-tile"` on Sapphire-Rapids-class + /// silicon (OS-enabled tiles), otherwise the AVX-512/AVX-VNNI/scalar + /// ladder inside `matmul_i8_to_i32` selects at call time. + pub fn polyfill_backend(&self) -> &'static str { + if ndarray::simd::amx_available() { + "amx-tdpbusd-tile" + } else { + "avx512-vnni / avx-vnni / scalar (runtime-selected by ndarray)" + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn unit_vectors(n: usize, dim: usize, seed: u64) -> Vec { + let mut s = seed.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut out = vec![0.0f32; n * dim]; + for row in out.chunks_mut(dim) { + let mut norm = 0.0f64; + for x in row.iter_mut() { + s = s + .wrapping_mul(6_364_136_223_846_793_005) + .wrapping_add(1_442_695_040_888_963_407); + let v = ((s >> 33) as f64 / (1u64 << 31) as f64) - 1.0; + *x = v as f32; + norm += v * v; + } + let inv = 1.0 / (norm.sqrt() + 1e-9); + for x in row.iter_mut() { + *x = (*x as f64 * inv) as f32; + } + } + out + } + + #[test] + fn both_kernels_agree_on_majority_of_topk() { + let (dim, n, nq, k) = (64usize, 800usize, 6usize, 10usize); + let db = unit_vectors(n, dim, 3); + let queries = unit_vectors(nq, dim, 77); + let mut tv = TurboVec::new(dim, 4).unwrap(); + tv.add(&db); + assert_eq!(tv.len(), n); + + let native = tv.search(&queries, k, Kernel::NativeLut); + let poly = tv.search(&queries, k, Kernel::PolyfillGemm); + assert_eq!(native.nq, nq); + assert_eq!(poly.nq, nq); + + // The two kernels are different approximations of the same inner + // product; on clean data they should agree on a solid majority of + // each query's top-k. + let mut overlap = 0usize; + for qi in 0..nq { + let a: std::collections::BTreeSet = native.indices[qi * k..(qi + 1) * k] + .iter() + .copied() + .collect(); + for &i in &poly.indices[qi * k..(qi + 1) * k] { + if a.contains(&i) { + overlap += 1; + } + } + } + let agree = overlap as f64 / (nq * k) as f64; + assert!( + agree >= 0.5, + "native/polyfill top-k agreement {agree:.3} below 0.5" + ); + } + + #[test] + fn backend_report_is_nonempty() { + let tv = TurboVec::new(8, 2).unwrap(); + assert!(!tv.polyfill_backend().is_empty()); + assert!(tv.is_empty()); + } +} From cfb530b438265a34cc7040e115eaa9312ad32fa4 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 13 Jun 2026 22:44:42 +0000 Subject: [PATCH 2/4] fix(bgz17): real AVX-512 arm for batch_palette_distance (was silently scalar on v4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit detect_simd() returned SimdLevel::Avx512 but batch_palette_distance had only an Avx2 match arm, so AVX-512 hosts fell through to scalar_batch. Added avx512_batch — a 16-wide _mm512_i32gather_epi32::<2> mirror of avx2_batch, low-u16 masked, bit-identical to scalar_batch. Kept bgz17 0-dependency (option b; routing through ndarray::simd is the Phase-3 'move bgz17 into the workspace' change, not a bugfix). 126 tests pass (incl test_batch_matches_scalar exercising the AVX-512 arm here). https://claude.ai/code/session_01D2WSmezQBNC3bUdHuGfGmo --- crates/bgz17/src/simd.rs | 78 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) diff --git a/crates/bgz17/src/simd.rs b/crates/bgz17/src/simd.rs index 53cc709bd..fd735a27a 100644 --- a/crates/bgz17/src/simd.rs +++ b/crates/bgz17/src/simd.rs @@ -55,6 +55,11 @@ pub fn batch_palette_distance( let level = detect_simd(); match level { + #[cfg(target_arch = "x86_64")] + SimdLevel::Avx512 => { + // Safety: detect_simd() confirmed AVX-512F is available. + unsafe { avx512_batch(dm_data, k, query, candidates, out) }; + } #[cfg(target_arch = "x86_64")] SimdLevel::Avx2 => { // Safety: detect_simd() confirmed AVX2 is available. @@ -138,6 +143,79 @@ unsafe fn avx2_batch(dm_data: &[u16], k: usize, query: u8, candidates: &[u8], ou } } +/// AVX-512 gather batch lookup: process 16 lookups at a time using _mm512_i32gather_epi32. +/// +/// Widened analogue of `avx2_batch`. The distance matrix stores u16 values; we +/// gather i32 words from the u16 base pointer with byte-scale 2 (each u16 is 2 +/// bytes), then mask off the high u16 of each lane. The low u16 of each gathered +/// i32 is exactly `dm[query][candidate]`, so the result is identical to +/// `scalar_batch` (and to `avx2_batch`). The 16-wide remainder falls back to +/// scalar, matching the AVX2 path's tail handling. +/// +/// # Safety +/// Caller must ensure AVX-512F is available (checked via `is_x86_feature_detected!("avx512f")`). +#[cfg(target_arch = "x86_64")] +#[target_feature(enable = "avx512f")] +unsafe fn avx512_batch(dm_data: &[u16], k: usize, query: u8, candidates: &[u8], out: &mut [u16]) { + use core::arch::x86_64::*; + + let row_offset = query as usize * k; + let row_ptr = dm_data.as_ptr().add(row_offset); + let n = candidates.len(); + + // Process 16 candidates at a time + let chunks = n / 16; + let remainder = n % 16; + + for chunk in 0..chunks { + let base = chunk * 16; + + // Build index vector: candidate indices as i32 (lane 0 = candidates[base]). + let indices = _mm512_set_epi32( + candidates[base + 15] as i32, + candidates[base + 14] as i32, + candidates[base + 13] as i32, + candidates[base + 12] as i32, + candidates[base + 11] as i32, + candidates[base + 10] as i32, + candidates[base + 9] as i32, + candidates[base + 8] as i32, + candidates[base + 7] as i32, + candidates[base + 6] as i32, + candidates[base + 5] as i32, + candidates[base + 4] as i32, + candidates[base + 3] as i32, + candidates[base + 2] as i32, + candidates[base + 1] as i32, + candidates[base] as i32, + ); + + // Gather u16 values via i32 gather on the u16 array. With scale=2 on the + // u16 base pointer, lane j reads the i32 at byte offset candidates[..]*2, + // i.e. the target u16 (low half) plus the next u16 (high half). Identical + // trick to avx2_batch, widened to 16 lanes. + let gathered = _mm512_i32gather_epi32::<2>(indices, row_ptr as *const i32); + + // Mask to extract only the low u16 from each i32 lane. + let mask = _mm512_set1_epi32(0x0000FFFF); + let masked = _mm512_and_si512(gathered, mask); + + // Extract and store individually (no direct i32→u16 pack across 16 lanes). + let mut tmp = [0i32; 16]; + _mm512_storeu_si512(tmp.as_mut_ptr() as *mut __m512i, masked); + + for i in 0..16 { + out[base + i] = tmp[i] as u16; + } + } + + // Scalar fallback for remaining elements + let tail_start = chunks * 16; + for i in 0..remainder { + out[tail_start + i] = dm_data[row_offset + candidates[tail_start + i] as usize]; + } +} + /// Batch SPO distance: combined S+P+O distance for multiple candidates. /// /// For each candidate i: From 0eaaa86c520931bd26812b722ce5747174a8ecdd Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 06:40:34 +0000 Subject: [PATCH 3/4] refactor(blasgraph): route Hamming/popcount through ndarray::hpc::bitwise MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Under `ndarray-hpc`, `dispatch_hamming`/`dispatch_popcount` and the typed `hamming_distance_dispatch` now call `ndarray::hpc::bitwise::{hamming_distance_raw, popcount_raw}` (the canonical VPOPCNTDQ → AVX-512BW → AVX2 → scalar dispatch), per the "all SIMD from ndarray" doctrine. The hand-rolled in-crate intrinsics survive only as the `#[cfg(not(feature = "ndarray-hpc"))]` fallback for minimal / non-x86 builds (CI, wasm, embedded). Mirrors the episodic.rs pattern. Validated: `cargo test -p lance-graph --lib blasgraph` → 194 passed, 0 failed (protoc installed to unblock the lance-encoding build script). https://claude.ai/code/session_01D2WSmezQBNC3bUdHuGfGmo --- .../src/graph/blasgraph/ndarray_bridge.rs | 88 +++++++++------ .../lance-graph/src/graph/blasgraph/types.rs | 106 +++++------------- 2 files changed, 83 insertions(+), 111 deletions(-) diff --git a/crates/lance-graph/src/graph/blasgraph/ndarray_bridge.rs b/crates/lance-graph/src/graph/blasgraph/ndarray_bridge.rs index 00fdd7719..767f9c1bd 100644 --- a/crates/lance-graph/src/graph/blasgraph/ndarray_bridge.rs +++ b/crates/lance-graph/src/graph/blasgraph/ndarray_bridge.rs @@ -128,60 +128,80 @@ impl From<&ndarray::hpc::fingerprint::Fingerprint<256>> for BitVec { } // --------------------------------------------------------------------------- -// SIMD dispatch — 4-tier fallback matching ndarray's bitwise.rs pattern +// SIMD dispatch — routed through ndarray under `ndarray-hpc`, else 4-tier +// in-crate fallback. Per the "all SIMD from ndarray" doctrine the canonical +// SIMD dispatch lives in `ndarray::hpc::bitwise`; the hand-rolled intrinsics +// below survive only as the `#[cfg(not(feature = "ndarray-hpc"))]` fallback +// for minimal / non-x86 builds (CI, wasm, embedded). // --------------------------------------------------------------------------- /// SIMD-dispatched Hamming distance between two byte slices. /// -/// Computes `popcount(a XOR b)` using the best available instruction set: -/// -/// 1. **VPOPCNTDQ** (AVX-512 VPOPCNTDQ) — 512-bit popcount in one instruction -/// 2. **AVX-512BW** — 512-bit XOR + byte-level popcount via shuffle LUT -/// 3. **AVX2** — 256-bit XOR + byte-level popcount via shuffle LUT -/// 4. **Scalar** — word-by-word `count_ones()` +/// Computes `popcount(a XOR b)`. Under `ndarray-hpc` this routes through +/// `ndarray::hpc::bitwise::hamming_distance_raw` (VPOPCNTDQ → AVX-512BW → +/// AVX2 → scalar). Without the feature it uses the in-crate 4-tier fallback +/// (VPOPCNTDQ → AVX-512BW → AVX2 → scalar). /// /// Both slices must have the same length. Panics otherwise. pub fn dispatch_hamming(a: &[u8], b: &[u8]) -> u64 { assert_eq!(a.len(), b.len(), "hamming: slices must have equal length"); - #[cfg(target_arch = "x86_64")] + #[cfg(feature = "ndarray-hpc")] { - if is_x86_feature_detected!("avx512vpopcntdq") && is_x86_feature_detected!("avx512f") { - // SAFETY: feature detection guarantees VPOPCNTDQ is available. - return unsafe { hamming_avx512_vpopcntdq(a, b) }; - } - if is_x86_feature_detected!("avx512bw") && is_x86_feature_detected!("avx512f") { - // SAFETY: feature detection guarantees AVX-512BW is available. - return unsafe { hamming_avx512bw(a, b) }; - } - if is_x86_feature_detected!("avx2") { - // SAFETY: feature detection guarantees AVX2 is available. - return unsafe { hamming_avx2(a, b) }; - } + // Lengths are equal (asserted above), so ndarray's `min(len)` is exact. + ndarray::hpc::bitwise::hamming_distance_raw(a, b) } - hamming_scalar(a, b) + #[cfg(not(feature = "ndarray-hpc"))] + { + #[cfg(target_arch = "x86_64")] + { + if is_x86_feature_detected!("avx512vpopcntdq") && is_x86_feature_detected!("avx512f") { + // SAFETY: feature detection guarantees VPOPCNTDQ is available. + return unsafe { hamming_avx512_vpopcntdq(a, b) }; + } + if is_x86_feature_detected!("avx512bw") && is_x86_feature_detected!("avx512f") { + // SAFETY: feature detection guarantees AVX-512BW is available. + return unsafe { hamming_avx512bw(a, b) }; + } + if is_x86_feature_detected!("avx2") { + // SAFETY: feature detection guarantees AVX2 is available. + return unsafe { hamming_avx2(a, b) }; + } + } + + hamming_scalar(a, b) + } } /// SIMD-dispatched population count over a byte slice. /// -/// Uses the same 4-tier fallback as `dispatch_hamming`: -/// VPOPCNTDQ -> AVX-512BW -> AVX2 -> scalar. +/// Under `ndarray-hpc` this routes through `ndarray::hpc::bitwise:: +/// popcount_raw`. Without the feature it uses the same in-crate 4-tier +/// fallback as `dispatch_hamming` (VPOPCNTDQ → AVX-512BW → AVX2 → scalar). pub fn dispatch_popcount(a: &[u8]) -> u64 { - #[cfg(target_arch = "x86_64")] + #[cfg(feature = "ndarray-hpc")] { - if is_x86_feature_detected!("avx512vpopcntdq") && is_x86_feature_detected!("avx512f") { - return unsafe { popcount_avx512_vpopcntdq(a) }; - } - if is_x86_feature_detected!("avx512bw") && is_x86_feature_detected!("avx512f") { - return unsafe { popcount_avx512bw(a) }; - } - if is_x86_feature_detected!("avx2") { - return unsafe { popcount_avx2(a) }; - } + ndarray::hpc::bitwise::popcount_raw(a) } - popcount_scalar(a) + #[cfg(not(feature = "ndarray-hpc"))] + { + #[cfg(target_arch = "x86_64")] + { + if is_x86_feature_detected!("avx512vpopcntdq") && is_x86_feature_detected!("avx512f") { + return unsafe { popcount_avx512_vpopcntdq(a) }; + } + if is_x86_feature_detected!("avx512bw") && is_x86_feature_detected!("avx512f") { + return unsafe { popcount_avx512bw(a) }; + } + if is_x86_feature_detected!("avx2") { + return unsafe { popcount_avx2(a) }; + } + } + + popcount_scalar(a) + } } // --------------------------------------------------------------------------- diff --git a/crates/lance-graph/src/graph/blasgraph/types.rs b/crates/lance-graph/src/graph/blasgraph/types.rs index ebe188fdd..454d36e0d 100644 --- a/crates/lance-graph/src/graph/blasgraph/types.rs +++ b/crates/lance-graph/src/graph/blasgraph/types.rs @@ -420,10 +420,16 @@ pub enum SelectOp { // ─── SIMD-dispatched Hamming distance ───────────────────────────────── // -// Dispatch chain: AVX-512 VPOPCNTDQ → AVX2 → scalar. -// Uses `std::arch` intrinsics only, no external crate. +// Per the "all SIMD from ndarray" doctrine, the SIMD dispatch lives in +// `ndarray::hpc::bitwise` (VPOPCNTDQ → AVX-512BW → AVX2 → scalar) and is +// routed in under the `ndarray-hpc` feature. The hand-rolled scalar path +// below is the `#[cfg(not(feature = "ndarray-hpc"))]` fallback so minimal / +// non-x86 builds (CI, wasm, embedded) keep working without the dep. /// Scalar fallback: portable popcount via `count_ones()`. +/// +/// Used as the `#[cfg(not(feature = "ndarray-hpc"))]` Hamming path and by the +/// in-crate parity tests. fn hamming_distance_scalar(a: &[u64; VECTOR_WORDS], b: &[u64; VECTOR_WORDS]) -> u32 { let mut dist = 0u32; for i in 0..VECTOR_WORDS { @@ -432,85 +438,31 @@ fn hamming_distance_scalar(a: &[u64; VECTOR_WORDS], b: &[u64; VECTOR_WORDS]) -> dist } -/// AVX2 implementation: processes 4 × u64 = 256 bits per iteration. -/// Uses the Harley-Seal popcount algorithm on 256-bit XOR results. -#[cfg(target_arch = "x86_64")] -#[target_feature(enable = "avx2")] -unsafe fn hamming_distance_avx2(a: &[u64; VECTOR_WORDS], b: &[u64; VECTOR_WORDS]) -> u32 { - use std::arch::x86_64::*; - - // Lookup table for 4-bit popcount - let lookup = _mm256_setr_epi8( - 0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, 3, 4, 0, 1, 1, 2, 1, 2, 2, 3, 1, 2, 2, 3, 2, 3, - 3, 4, - ); - let low_mask = _mm256_set1_epi8(0x0f); - let mut total = _mm256_setzero_si256(); - - let a_ptr = a.as_ptr() as *const __m256i; - let b_ptr = b.as_ptr() as *const __m256i; - let n_vecs = VECTOR_WORDS / 4; // 256 / 4 = 64 iterations - - for i in 0..n_vecs { - let va = _mm256_loadu_si256(a_ptr.add(i)); - let vb = _mm256_loadu_si256(b_ptr.add(i)); - let xor = _mm256_xor_si256(va, vb); - - // Popcount via lookup table (Mula et al.) - let lo = _mm256_and_si256(xor, low_mask); - let hi = _mm256_and_si256(_mm256_srli_epi16(xor, 4), low_mask); - let popcnt_lo = _mm256_shuffle_epi8(lookup, lo); - let popcnt_hi = _mm256_shuffle_epi8(lookup, hi); - let popcnt = _mm256_add_epi8(popcnt_lo, popcnt_hi); - - // Horizontal sum within bytes → u64 sums via sad - let sad = _mm256_sad_epu8(popcnt, _mm256_setzero_si256()); - total = _mm256_add_epi64(total, sad); - } - - // Extract and sum the 4 u64 lanes - let lo128 = _mm256_castsi256_si128(total); - let hi128 = _mm256_extracti128_si256(total, 1); - let sum128 = _mm_add_epi64(lo128, hi128); - let upper = _mm_unpackhi_epi64(sum128, sum128); - let final_sum = _mm_add_epi64(sum128, upper); - _mm_cvtsi128_si64(final_sum) as u32 -} - -/// AVX-512 VPOPCNTDQ implementation: processes 8 × u64 = 512 bits per iteration. -#[cfg(target_arch = "x86_64")] -#[target_feature(enable = "avx512f,avx512vpopcntdq")] -unsafe fn hamming_distance_avx512(a: &[u64; VECTOR_WORDS], b: &[u64; VECTOR_WORDS]) -> u32 { - use std::arch::x86_64::*; - - let mut total = _mm512_setzero_si512(); - let a_ptr = a.as_ptr() as *const __m512i; - let b_ptr = b.as_ptr() as *const __m512i; - let n_vecs = VECTOR_WORDS / 8; // 256 / 8 = 32 iterations - - for i in 0..n_vecs { - let va = _mm512_loadu_si512(a_ptr.add(i)); - let vb = _mm512_loadu_si512(b_ptr.add(i)); - let xor = _mm512_xor_si512(va, vb); - let popcnt = _mm512_popcnt_epi64(xor); - total = _mm512_add_epi64(total, popcnt); +/// Runtime-dispatched Hamming distance. +/// +/// Under `ndarray-hpc` this routes through `ndarray::hpc::bitwise:: +/// hamming_distance_raw` (the canonical SIMD dispatch shared with the rest of +/// the Ada stack). Without the feature it falls back to the in-crate scalar +/// path. Both views reinterpret the same `[u64; VECTOR_WORDS]` backing store +/// as native-endian bytes; Hamming distance is a bit count and is therefore +/// invariant under the (consistent) byte layout on both operands. +fn hamming_distance_dispatch(a: &[u64; VECTOR_WORDS], b: &[u64; VECTOR_WORDS]) -> u32 { + #[cfg(feature = "ndarray-hpc")] + { + const BYTE_LEN: usize = VECTOR_WORDS * 8; + // SAFETY: `[u64; VECTOR_WORDS]` is plain-old-data with no padding; a + // `&[u8]` view of the same `BYTE_LEN` bytes is always valid (u8 has + // alignment 1). Same layout on both operands → bit count is exact. + let a_bytes = unsafe { std::slice::from_raw_parts(a.as_ptr() as *const u8, BYTE_LEN) }; + let b_bytes = unsafe { std::slice::from_raw_parts(b.as_ptr() as *const u8, BYTE_LEN) }; + // Max distance is VECTOR_BITS (16384), well within u32. + ndarray::hpc::bitwise::hamming_distance_raw(a_bytes, b_bytes) as u32 } - _mm512_reduce_add_epi64(total) as u32 -} - -/// Runtime-dispatched Hamming distance using best available SIMD. -fn hamming_distance_dispatch(a: &[u64; VECTOR_WORDS], b: &[u64; VECTOR_WORDS]) -> u32 { - #[cfg(target_arch = "x86_64")] + #[cfg(not(feature = "ndarray-hpc"))] { - if is_x86_feature_detected!("avx512f") && is_x86_feature_detected!("avx512vpopcntdq") { - return unsafe { hamming_distance_avx512(a, b) }; - } - if is_x86_feature_detected!("avx2") { - return unsafe { hamming_distance_avx2(a, b) }; - } + hamming_distance_scalar(a, b) } - hamming_distance_scalar(a, b) } #[cfg(test)] From 31d77572186ad0c0c33ba72c6bcc6efa095a42a3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 07:37:47 +0000 Subject: [PATCH 4/4] fix(blasgraph): silence dead_code for scalar Hamming fallback under ndarray-hpc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit After routing `hamming_distance_dispatch` through `ndarray::hpc::bitwise`, the in-crate `hamming_distance_scalar` is reached only from the `not(ndarray-hpc)` fallback and `#[cfg(test)]` parity tests — so a non-test build with the default `ndarray-hpc` feature warns it's unused. `#[cfg_attr(feature = "ndarray-hpc", allow(dead_code))]` keeps the lib build warning-free while leaving it live for the fallback and tests. https://claude.ai/code/session_01D2WSmezQBNC3bUdHuGfGmo --- crates/lance-graph/src/graph/blasgraph/types.rs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/crates/lance-graph/src/graph/blasgraph/types.rs b/crates/lance-graph/src/graph/blasgraph/types.rs index 454d36e0d..391128241 100644 --- a/crates/lance-graph/src/graph/blasgraph/types.rs +++ b/crates/lance-graph/src/graph/blasgraph/types.rs @@ -429,7 +429,10 @@ pub enum SelectOp { /// Scalar fallback: portable popcount via `count_ones()`. /// /// Used as the `#[cfg(not(feature = "ndarray-hpc"))]` Hamming path and by the -/// in-crate parity tests. +/// in-crate parity tests. Under `ndarray-hpc` (the default) the dispatch routes +/// through `ndarray::hpc::bitwise`, so this is reachable only from `#[cfg(test)]` +/// then — `allow(dead_code)` keeps the non-test lib build warning-free. +#[cfg_attr(feature = "ndarray-hpc", allow(dead_code))] fn hamming_distance_scalar(a: &[u64; VECTOR_WORDS], b: &[u64; VECTOR_WORDS]) -> u32 { let mut dist = 0u32; for i in 0..VECTOR_WORDS {