Graceful shutdown can take ~70s draining an empty replacement ingest shard

[fmassot]

Description

Graceful shutdown of a single-node Quickwit 0.9.0-rc1 instance can spend approximately 70 seconds draining and deleting an empty replacement ingest shard while repeatedly reporting that no indexing capacity is available.

All submitted documents had already been committed and verified through REST search before shutdown.

Setup

• Single node running all default services
• File-backed metastore and local split storage
• Index migrated from Quickwit 0.8.2 to 0.9.0-rc1
• 524,288 legacy documents plus 131,072 documents ingested after the upgrade
• The final search returned exactly 655,360 documents with no split errors

Steps to reproduce

1. Start Quickwit 0.8.2 and ingest data into an index.
2. Stop 0.8.2 cleanly.
3. Start 0.9.0-rc1 on the same data directory.
4. Ingest additional data using ingest v2 and wait for quickwit index ingest --force to complete.
5. Verify the expected document count through the REST search API.
6. Send SIGINT to initiate graceful shutdown.

Observed behavior

Shutdown began normally:

Graceful shutdown initiated. Waiting for ingested data to be indexed.
quickwit_ingest::ingest_v2::ingester: retiring ingester

The completed shard reached EOF and was deleted, but the control plane assigned a new replacement shard to the retiring indexing pipeline. Shutdown then repeatedly logged:

no ready indexer available, falling back to retiring indexers for shard draining
no indexing capacity available, cannot schedule an indexing plan

The plan diff continued to contain the replacement _ingest-source shard on the retiring node. Eventually Quickwit published an empty checkpoint for that shard, deleted it, and completed:

publish-new-splits num_splits=0 num_docs=0 split_size_bytes=0
successfully decommissioned ingester in 70.30s
REST server successfully shut down
quickwit successfully terminated

No data was lost and the process exited successfully. A second run using the official RC1 container completed graceful shutdown in approximately 22 seconds, so the 70-second path appears timing-sensitive rather than deterministic.

Expected behavior

Once the node is retiring and all accepted data has been indexed, graceful shutdown should not allocate a replacement empty shard that must itself be scheduled and drained.

If the longer drain is an intentional bounded path, the scheduler should avoid repeated alarming no indexing capacity available warnings for an expected single-node shutdown.

Environment

Quickwit 0.9.0
commit: bfcc15e
release feature set (DataFusion disabled)
single-node, file-backed metastore

The issue was first observed with a native build from the exact RC1 commit. The migration and data-integrity checks also passed using the official quickwit/quickwit:v0.9.0-rc1 image.