Description
With ingest v2, quickwit index ingest --force appears to force-commit every submitted batch instead of only the final batch.
The CLI help describes --force as:
Force a commit after the last document is sent, and wait for all documents to be committed and available for search before exiting.
In practice, an ingest using an 8 MB batch limit published one split per approximately 8 MB request. This generated many tiny splits and continuous merge activity while the CLI was still ingesting.
Steps to reproduce
- Start a single-node Quickwit 0.9.0-rc1 instance with ingest v2 enabled.
- Create an index with a normal stable-log merge policy.
- Generate 131,072 NDJSON documents of approximately 4 KB each (approximately 525 MB uncompressed in total).
- Ingest them with:
quickwit index ingest \
--index migration-test \
--batch-size-limit 8MB \
--force \
--commit-timeout 15m \
-y
Observed behavior
Each approximately 8 MB input batch triggered a published split containing roughly 1,995 documents:
publish-new-splits num_splits=1 num_docs=1995 split_size_bytes=54998
checkpoint_delta: ...00000000000000027943..00000000000000029939
publish-new-splits num_splits=1 num_docs=1995 split_size_bytes=55029
checkpoint_delta: ...00000000000000029939..00000000000000031935
For 131,072 documents, this corresponds to roughly 66 forced commits/splits before concurrent merges. The merge pipeline continuously compacted the small splits, and the final published state contained three merged splits.
The behavior was reproduced both with a native build and the official quickwit/quickwit:v0.9.0-rc1 image.
Expected behavior
Non-final batches should use the normal commit mode. Only the last request should use a forced commit, after which the CLI should wait until all submitted documents are searchable.
This should avoid unnecessary split creation, uploads, metastore operations, and immediate merge work.
Environment
Quickwit 0.9.0
commit: bfcc15e
official image: quickwit/quickwit:v0.9.0-rc1
platform: aarch64-unknown-linux-gnu
embedded node config version: 0.8
The documents were ingested successfully and search results were correct; this report concerns split/commit behavior and the resulting write amplification.
Description
With ingest v2,
quickwit index ingest --forceappears to force-commit every submitted batch instead of only the final batch.The CLI help describes
--forceas:In practice, an ingest using an 8 MB batch limit published one split per approximately 8 MB request. This generated many tiny splits and continuous merge activity while the CLI was still ingesting.
Steps to reproduce
Observed behavior
Each approximately 8 MB input batch triggered a published split containing roughly 1,995 documents:
For 131,072 documents, this corresponds to roughly 66 forced commits/splits before concurrent merges. The merge pipeline continuously compacted the small splits, and the final published state contained three merged splits.
The behavior was reproduced both with a native build and the official
quickwit/quickwit:v0.9.0-rc1image.Expected behavior
Non-final batches should use the normal commit mode. Only the last request should use a forced commit, after which the CLI should wait until all submitted documents are searchable.
This should avoid unnecessary split creation, uploads, metastore operations, and immediate merge work.
Environment
The documents were ingested successfully and search results were correct; this report concerns split/commit behavior and the resulting write amplification.