Skip to content

feat(cli): create index --train-pq-codebook trains codebooks at creation (#920 sub-item 3) - #939

Merged
mosuka merged 1 commit into
mainfrom
feat/920-create-train-pq-codebook
Aug 4, 2026
Merged

feat(cli): create index --train-pq-codebook trains codebooks at creation (#920 sub-item 3)#939
mosuka merged 1 commit into
mainfrom
feat/920-create-train-pq-codebook

Conversation

@mosuka

@mosuka mosuka commented Aug 4, 2026

Copy link
Copy Markdown
Owner

Summary

Changes

  • train.rs: JSONL collection loop extracted into pub(crate) collect_vectors_from_jsonl (behavior-preserving; the 7 existing train tests are the regression gate).
  • cli.rs / main.rs: CreateResource::Index gains train_pq_codebook: Option<PathBuf>.
  • create.rs::run_index:
    • Resolves the effective schema first, replicating init_index's recovery rule (an existing schema.toml without store/ wins over the argument/wizard schema); all paths now go through create_index_from_schema.
    • Pre-creation validation: a missing JSONL or zero eligible fields bails before anything is created — no half-initialized index.
    • After creation, reopens the index and trains every eligible field (Hnsw + ProductQuantization + pq_codebook_path, sorted by field name since Schema::fields is a HashMap) with output=None, so the persisted schema and trained file agree by construction.
  • Docs (EN+JA): laurus-cli/commands.md create-index section (usage, argument row, one-step example); vector_indexing.md shared-codebook section.

Tests

4 new in create.rs (CLI total 17):

  • create_with_train_flag_makes_first_commit_succeed — the direct hazard-resolution regression (.pqcb exists, reopened first commit succeeds).
  • create_without_train_flag_leaves_first_commit_failingcontrol pinning the hazard itself: without the flag the same schema's first commit fails with the pq_codebook_path hard-error. Together these form a RED/GREEN pair attributing the fix causally to the flag.
  • create_with_train_flag_rejects_schema_without_pq_field — bails and nothing is created.
  • create_with_train_flag_trains_every_pq_field — multiple PQ fields all trained.

Verification

  • cargo fmt --check / cargo clippy --all-targets -- -D warnings: clean on stable and 1.97.0.
  • cargo test -p laurus-cli --bin laurus: 17 passed (+4). cargo test -p laurus --lib: 1273 passed (library crate untouched).
  • markdownlint EN+JA: 0 errors; both mdBooks build.

…ion (refs #920)

Add a --train-pq-codebook <jsonl> flag to `laurus create index` that
trains the shared PQ codebook for every eligible HNSW field
(ProductQuantization + pq_codebook_path, field-name-sorted) immediately
after creation, so the very first commit can already encode against it —
removing the create → train → ingest ordering the #918 failure policy
otherwise leaves the user to manage manually.

Validation (JSONL exists, at least one eligible field) runs before
anything is created, so a bad invocation never leaves a half-initialized
index. The effective schema follows init_index's recovery rule (an
existing schema.toml without store/ wins over the argument/wizard
schema). The JSONL collection loop is extracted from `train pq-codebook`
into a shared helper; sub-item 2 (FastScan) of #920 remains open.
@mosuka
mosuka merged commit 6b1c59f into main Aug 4, 2026
14 checks passed
@mosuka
mosuka deleted the feat/920-create-train-pq-codebook branch August 4, 2026 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant