perf(vector/index): multi-vector / sparse vector layout reservation in segment header

[mosuka]

Round-3 perf push sub-issue (tracked under umbrella #535).

[S] Multi-vector / sparse vector layout reservation in segment header

• Where: laurus/src/vector/index/format.rs:46-86 (no
  multi-vector or sparse-vector slot reserved in the segment header).
• Current behaviour: quant_kind is a u16 with only Scalar8Bit
  and PQ implemented; no slot reserved for ColBERT-style
  multi-vector-per-doc or sparse-vector indexes.
• Why it might be a bottleneck / risk: when we eventually want
  multi-vector ranking (ColBERT, late interaction) or sparse-vector
  retrieval (SPLADE, BM25-style sparse), retrofitting will require
  another LVS format break.
• Suggested direction: reserve quant_kind ranges (e.g. 100–199
  for multi-vector, 200–299 for sparse), and reserve 8 more bytes in
  the LVS header for a future vector_kind discriminator (Single /
  Multi / Sparse). Pair with Add parallel index #2 to bake into LVS2.
• Risk / scope: zero cost today, future-proofs the format.

---

Cross-cutting themes

• Data structures stuck in "owned-tuple" form: (u64, String, Vector) is the everywhere-tuple, leading to repeated heap
  allocations, repeated string clones, and HashMap indirections
  throughout build, write, merge, search, and NRT paths. Migrating to
  (u32 ordinal, u16 field_id, &[u8] payload) enables CSR / columnar
  layouts (Add parallel search #1, Add parallel index #2, Fix fuzzy query example #8) and drops a large slab of pointer chasing.
• No format-level versioning beyond quant_kind: the current LVS1
  header is a thin shim around the writer-specific preamble. Several
  proposed wins (CSR adjacency, column layout, field dictionary,
  shared PQ codebook, half-precision) require an LVS2 break — they
  should land together rather than incurring N migrations.
• Build vs commit are not separated: finalize() rebuilds the
  graph in-process and write() serialises it in one shot, with no
  incremental fsync, no out-of-core path, no resumable build. This
  caps the index size at RAM and creates a long crash-recovery tail.
• k-means is hand-rolled and serial: both IVF and PQ training go
  through bespoke loops that don't use rayon and don't use SIMD even
  though distance.rs already has SIMD primitives. Reusing a single
  parallel k-means kernel would benefit five callsites.
• Deletion is bolted-on, not first-class: HNSW deletion drops the
  whole graph for rebuild; AHashSet<u64> per segment scales linearly
  in bytes. A tombstone-with-recycle + Roaring deletion bitmap pair
  brings laurus into line with hnswlib + Lucene + Qdrant on a single
  feature axis.
• Active-segment / NRT is unquantized: even though Stage 1 mandates
  SQ in persisted segments, the in-memory writer state stays f32 and
  the NRT path scans f32 brute force. Reconciling NRT with the
  quantized pool unifies the hot path across active / managed
  segments.
• u64 dominance: doc-IDs, neighbour IDs, sums, lengths — almost
  everything is u64. Half of them only need u32. Combined with the CSR
  migration this halves a sizeable fraction of the index RAM.

---

ID: VI-25 — see ~/.claude/tasks/laurus/20260523_perf_round3_audit/task_list.md for the full Round-3 issue list.