Feature request: table scoping for `optimize` (--table / --exclude-table)

[FlorianSeemann]

Feature request: table scoping for optimize (--table / --exclude-table)

Problem

omnigraph optimize visits every node + edge table. When one table is operationally
un-compactable — in our case a blob-bearing File table (~151 GB of document blobs; since the
Lance 9.0.0-beta.15 bump removed the blob-compaction skip, optimize rewrites it) — the whole
maintenance sweep becomes unrunnable inside any reasonable operator window:

• every scheduled run dies at its timeout mid-File-rewrite, restarting that table's compaction
  from scratch on the next tick, and
• crucially, index-coverage maintenance for every other table dies with it. In production
  that meant fragments appended to a 751k-row vector table (Chunk, 3072-dim @embed +
  @index ANN) were never folded into the index. Reads stayed correct via the planner's
  brute-force flat-scan union branch, but each semantic query then pulled ~10.5 GB of raw
  vectors from object storage (~60 s per query; two concurrent queries OOM'd a 24 GB host).

The blob rewrite is also a storage-headroom hazard: compaction keeps old versions reachable
until cleanup, so an unscoped run transiently near-doubles the largest table in the bucket.

Proposal

Let the operator scope the sweep:

omnigraph optimize --exclude-table File s3://…            # everything except the blob table
omnigraph optimize --table Chunk --table ChunkOfArtifact s3://…   # just these two

• Names resolve against the catalog as node:<Type> / edge:<Type> or a bare type name
  matching either; an unknown name errors (fail closed — a typo must not silently change the
  sweep), as does a scope that selects nothing.
• Scoped runs keep the exact same per-table machinery (write gates, recovery sidecar, manifest
  publish); they just visit fewer tables.
• The internal __manifest compaction still runs regardless of scope (cheap, in-place,
  content-preserving — and skipping it would let per-write metadata scans grow whenever the
  operator is forced to run scoped).

Patch

We are running this in production as a minimal patch on top of v0.8.1:
https://github.com/FlorianSeemann/omnigraph/tree/p9/v0.8.1-optimize-table-scope
(OptimizeScope { include, exclude } + optimize_tables_scoped in
db/omnigraph/optimize.rs, CLI flags, and four tests in tests/maintenance.rs including a
scoped-reindex coverage test). Happy to port it onto main's reworked optimize (the graph-wide
envelope) as a PR if you want the feature — the selection logic is independent of the publish
model.

[FlorianSeemann]

Follow-up — while running the scoped optimize in production we found a second, more severe problem that the scoping only exposed: create_vector_index hardcodes VectorIndexParams::ivf_flat(1, MetricType::L2) (table_store.rs). Every vector index the engine builds has a degenerate partition count, and since optimize_indices delta-folds never re-partition, the index keeps that count forever as the table grows.

Consequence at our scale: 852k × 3072-dim vectors behind 8 partitions meant every ANN query read the entire ~10.5 GB raw-vector index payload from object storage (~60 s per query on a LAN, minutes over WAN; two concurrent queries OOM'd a 24 GB host). The index was functionally a brute-force scan — full coverage, zero pruning.

Fix we're running (same patch branch, v0.8.1-p9.3):

1. create_vector_index sizes partitions as clamp(sqrt(non-null vectors), 1, 4096) (never above the vector count — k-means can't train more centroids than vectors).
2. optimize --retrain-vector-index (composes with --table): rebuilds every existing, trainable vector index through the current sizing with replace=true — a deliberate, scoped operator action, since delta folds can never fix a stale partition count.

Suggest treating the hardcoded ivf_flat(1) as a standalone bug even if the scoping feature isn't taken — happy to split either into its own issue/PR.

[FlorianSeemann]

One more from the same production incident — third bug, and the one that mattered most for query latency: the engine never sets a probe budget on nearest() scans (exec/query.rs calls scanner.nearest(prop, q, k) and nothing else). Lance's Rust scanner defaults to minimum_nprobes: 1, maximum_nprobes: None, which on lance @ f24e42c probes every IVF partition — a full read of the raw-vector index payload per query. Measured on the same 852k × 3072-dim table over object storage: nprobes=20 → 7.5 s, engine defaults → 289 s. This is also why fixing the partition count alone didn't move engine-served latency.

Patch (same branch, v0.8.1-p9.4): set scanner.nprobes(N) after nearest, default 20 (pylance's long-standing default), overridable via OMNIGRAPH_ANN_NPROBES. Applies to both plain nearest() and the rrf vector leg. Cold no-join query over WAN went 4:38 → 11.7 s.

Summary of the three related fixes on the branch: sqrt(N) IVF partition sizing (was ivf_flat(1)), optimize --retrain-vector-index, and the nprobes pin. Together: production semantic query went from ~4 min + OOM-under-concurrency to seconds.