Skip to content

fix(config): apply --service override before validation#6594

Draft
hushy wants to merge 5 commits into
mainfrom
lucas.verdonk/searcher-disk-usage-validation
Draft

fix(config): apply --service override before validation#6594
hushy wants to merge 5 commits into
mainfrom
lucas.verdonk/searcher-disk-usage-validation

Conversation

@hushy

@hushy hushy commented Jul 12, 2026

Copy link
Copy Markdown
Collaborator

A searcher node started with --service searcher could log a misleading "data dir volume too small" warning at startup, comparing its volume against the indexer split-store and ingest queue disk budgets. A searcher does not use those budgets, so the warning was wrong and misleading.

There were two related design issues:

  1. the CLI --service override was applied after load_node_config() had already run service-dependent validation;
  2. the disk-capacity check lived in config validation even though it inspects runtime host state.

What changed

  • thread the typed service override from the CLI through node-config loading;
  • apply it to the resolved service set before validation, keeping the effective services in NodeConfig;
  • keep CLI precedence over both QW_ENABLED_SERVICES and the config file, matching the previous post-load behavior;
  • move the disk-capacity warning from quickwit-config to quickwit-serve startup;
  • measure the data directory's backing disk once and compare it with the aggregate budgets of the effective enabled services;
  • include data_dir, backing device, canonical mount_point, and volume_size in the warning;
  • leave config loading unchanged when no CLI override is supplied;
  • cover both CLI-to-config forwarding and CLI precedence over an invalid
    QW_ENABLED_SERVICES value with dedicated regression tests.

The budgets remain aggregated because all enabled services share the data-dir volume: budgets that fit individually can still exceed the disk together. A searcher without a configured split cache has zero expected disk usage and does not warn.

Integration-test follow-up

Applying overrides before validation exposed an invalid setup in the CLI integration-test helper. It passed QuickwitService::supported_services(), which includes the standalone compactor, while its config left enable_standalone_compactors disabled. That only appeared to work previously because the override was applied after validation.

The helper now uses the normal default all-in-one service set, which intentionally excludes the standalone compactor. A code comment documents why supported_services() must not be used there.

How was this PR tested?

  • cargo test -p quickwit-common fs
  • cargo test -p quickwit-config (104 passed)
  • cargo test -p quickwit-cli test_run_command_forwards_service_override_before_validation
  • cargo test -p quickwit-serve disk_usage (2 passed)
  • cargo test -p quickwit-cli --test cli --features=metrics -- --test-threads=1 (18 passed, 1 ignored)
  • cargo clippy -p quickwit-common -p quickwit-config -p quickwit-serve -p quickwit-cli --all-targets -- -D warnings
  • cargo +nightly fmt --all -- --check
  • bash scripts/check_log_format.sh
  • git diff --check

A searcher node started with `--service searcher` could log a misleading
"data dir volume too small" warning at startup, comparing its volume
against the indexer split-store and ingest queue disk budgets. A searcher
does not use those budgets, so the warning was wrong and misleading.

The cause was validation order: the CLI `--service` override was applied
after `load_node_config()` had already run validation, so the disk-usage
check ran against the default (all-services) set and counted indexer/ingest
budgets. This threads the service override through `load_node_config_with_env`
into `build_and_validate`, applying it where `enabled_services` is resolved,
before validation. The override takes precedence over the config file and
env vars, matching the previous post-load behavior.

Co-Authored-By: Claude <noreply@anthropic.com>
@hushy
hushy force-pushed the lucas.verdonk/searcher-disk-usage-validation branch from 756a09e to f8fb91a Compare July 12, 2026 13:25
@guilload

Copy link
Copy Markdown
Member

I think a better approach would be to move these disk size checks in the init of each individual services:

  1. measure disk size once on start up in quickwit_serve
  2. each service (compactor, indexer, searcher) does check in its routine and warn if necessary

The warning should also be improved to specify on which device the size estimation is based.

@hushy hushy closed this Jul 15, 2026
@hushy hushy reopened this Jul 15, 2026
@PSeitz-dd

Copy link
Copy Markdown
Contributor

I ran into a similar issue for the readonly metastore searcher config flag. config validation fails because it ignores the service override (fix is pretty much the same)

2027c9e

@hushy hushy changed the title fix(config): apply --service override before disk-usage validation fix(config): apply --service override before validation Jul 19, 2026
@hushy
hushy force-pushed the lucas.verdonk/searcher-disk-usage-validation branch from 4a660de to 0fc03e1 Compare July 19, 2026 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants