Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docs/ja/src/concepts/indexing/vector_indexing.md
Original file line number Diff line number Diff line change
Expand Up @@ -486,7 +486,10 @@ laurus train pq-codebook --field embedding --input vectors.jsonl --update-schema
```

(`--input` の代わりに `--from-index` でインデックスにコミット済みの
ベクトルを直接サンプリングすることもできます、Issue #920)、または
ベクトルを直接サンプリングすることも、`laurus create index
--train-pq-codebook <jsonl>` で学習をインデックス作成に畳み込んで
train-before-first-commit の順序ハザードを完全に解消することもできます
— いずれも Issue #920)、または
プログラムから
[`Engine::train_pq_codebook`](https://docs.rs/laurus/latest/laurus/struct.Engine.html)
(`engine.train_pq_codebook("embedding", &vectors, None)`。from-index
Expand Down
10 changes: 9 additions & 1 deletion docs/ja/src/laurus-cli/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,15 @@ laurus --index-dir /var/data/my_index --format json search "title:rust"
新しいインデックスを作成します。`--schema` が指定された場合はその TOML ファイルを使用し、省略された場合は対話型スキーマウィザードが起動します。

```bash
laurus create index [--schema <FILE>]
laurus create index [--schema <FILE>] [--train-pq-codebook <JSONL>]
```

**引数:**

| フラグ | 必須 | 説明 |
| :--- | :--- | :--- |
| `--schema <FILE>` | いいえ | インデックススキーマを定義する TOML ファイルのパス。省略時はインデックスディレクトリに既存の `schema.toml` があればそれを使用し、なければ対話型ウィザードが起動します。 |
| `--train-pq-codebook <JSONL>` | いいえ | インデックス作成の一部として共有 PQ codebook を学習します(Issue #920)。`ProductQuantization` + `pq_codebook_path` を設定したすべての HNSW フィールドを、作成直後にこの JSONL ファイル(`put docs` / `add docs` と同じ形式、事前計算済み `Vector` 値)から学習します。最初の commit がすぐに codebook でエンコードできるため、create → `train pq-codebook` → ingest の順序を手動で守る必要がなくなります。ファイル不在または対象フィールドなしの場合は、何も作成する前にエラーになります。 |

**スキーマファイルの形式:**

Expand Down Expand Up @@ -65,6 +66,13 @@ laurus --index-dir ./my_index create index
# Field name: title
# ...
# Index created at ./my_index.

# 作成と共有 PQ codebook の学習を1ステップで(Issue #920)
laurus --index-dir ./my_index create index --schema schema.toml \
--train-pq-codebook train.jsonl
# Index created at ./my_index.
# Training PQ codebook for field 'embedding' on 300 vectors...
# Trained codebook 'embedding.pqcb' (m = 4, k = 256, sub_dim = 8, dimension = 32) from 300 vectors.
```

> **注意:** `schema.toml` と `store/` の両方が存在する場合はエラーが返されます。再作成するにはインデックスディレクトリを削除してください。`schema.toml` のみ存在する場合(作成が中断された場合など)は、`--schema` なしで `create index` を実行すると既存スキーマからストレージが復旧されます。
Expand Down
5 changes: 4 additions & 1 deletion docs/src/concepts/indexing/vector_indexing.md
Original file line number Diff line number Diff line change
Expand Up @@ -509,7 +509,10 @@ laurus train pq-codebook --field embedding --input vectors.jsonl --update-schema
```

(or `--from-index` in place of `--input` to sample vectors already
committed to the index, Issue #920), or programmatically via
committed to the index; or fold training into index creation with
`laurus create index --train-pq-codebook <jsonl>`, which removes the
train-before-first-commit ordering hazard entirely — both Issue #920),
or programmatically via
[`Engine::train_pq_codebook`](https://docs.rs/laurus/latest/laurus/struct.Engine.html)
(`engine.train_pq_codebook("embedding", &vectors, None)`; pair with
`engine.sample_committed_vectors("embedding", Some(n))` for the
Expand Down
10 changes: 9 additions & 1 deletion docs/src/laurus-cli/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,15 @@ laurus --index-dir /var/data/my_index --format json search "title:rust"
Create a new index. If `--schema` is given, uses that TOML file; otherwise launches the interactive schema wizard.

```bash
laurus create index [--schema <FILE>]
laurus create index [--schema <FILE>] [--train-pq-codebook <JSONL>]
```

**Arguments:**

| Flag | Required | Description |
| :--- | :--- | :--- |
| `--schema <FILE>` | No | Path to a TOML file defining the index schema. When omitted, the command checks if a `schema.toml` already exists in the index directory and uses it; otherwise the interactive wizard is launched. |
| `--train-pq-codebook <JSONL>` | No | Train shared PQ codebooks as part of creation (Issue #920). Every HNSW field configuring `ProductQuantization` + `pq_codebook_path` is trained from this JSONL file (the `put docs` / `add docs` shape with pre-computed `Vector` values) immediately after the index is created, so the very first commit can already encode against the codebook — removing the create → `train pq-codebook` → ingest ordering the failure policy otherwise requires you to manage manually. Errors before creating anything if the file is missing or no field is eligible. |

**Schema file format:**

Expand Down Expand Up @@ -65,6 +66,13 @@ laurus --index-dir ./my_index create index
# Field name: title
# ...
# Index created at ./my_index.

# Create and train the shared PQ codebook in one step (Issue #920)
laurus --index-dir ./my_index create index --schema schema.toml \
--train-pq-codebook train.jsonl
# Index created at ./my_index.
# Training PQ codebook for field 'embedding' on 300 vectors...
# Trained codebook 'embedding.pqcb' (m = 4, k = 256, sub_dim = 8, dimension = 32) from 300 vectors.
```

> **Note:** If both `schema.toml` and `store/` already exist, an error is returned. Delete the index directory to recreate. If only `schema.toml` exists (e.g. after an interrupted creation), running `create index` without `--schema` recovers the index by creating the missing storage from the existing schema.
Expand Down
13 changes: 12 additions & 1 deletion laurus-cli/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -67,12 +67,23 @@ pub struct CreateCommand {
#[derive(Subcommand)]
pub enum CreateResource {
/// Create a new index. If --schema is given, uses that TOML file;
/// otherwise launches the interactive schema wizard.
/// otherwise launches the interactive schema wizard. With
/// --train-pq-codebook, shared PQ codebooks are trained as part of
/// creation (Issue #920), removing the train-before-first-commit
/// ordering hazard for fields that configure pq_codebook_path.
Index {
/// Path to an existing schema TOML file. When omitted, the
/// interactive schema wizard is launched instead.
#[arg(long)]
schema: Option<PathBuf>,
/// Path to a JSONL training file (the `put docs` / `add docs`
/// shape; pre-computed Vector values). When given, every HNSW
/// field that configures ProductQuantization + pq_codebook_path
/// gets its shared codebook trained from this file immediately
/// after creation, so the very first commit can already encode
/// against it.
#[arg(long)]
train_pq_codebook: Option<PathBuf>,
},
/// Interactively generate a schema TOML file.
Schema {
Expand Down
Loading