control-plane: recover from a non-converging indexing scheduler (stuck "reapply last plan" loop)

[aanufriev-celonis]

Summary

The control plane can get stuck indefinitely in a non-converging indexing-scheduler loop, stalling ingestion for the affected shards until manual restart of indexers.

This issue tracks the staged work to make the loop observable and then self-recovering.

Background

control_running_plan compares what indexers report running with the last applied plan:

• Different nodes -> recompute the plan from the model (rebuild_plan).
• Same nodes, different tasks -> re-send the same plan, assuming an indexer just has not applied it yet.

That assumption breaks when the plan cannot be applied. It assigns a shard that no longer exists (our case) in the indexer's WAL (a phantom Open shard after a crash/restart onto a fresh volume). The indexer can never run that task, so has_same_tasks() stays false forever and the loop re-sends the same plan every ~30s. Reapplying cannot fix a plan that is wrong, it only re-sends it over and over again.

Today this is also invisible: the loop logs the same info line each cycle, so a one-cycle propagation lag is indistinguishable from a cluster stuck for hours, and there is nothing to alert on.

Fix Plan

Stage 1. Observability - #6585

Stage 2. Recovery - #6602 (includes #6585 but can be merged separately, on top)

[fulmicoton]

@aanufriev-celonis is this fixing a problem you observed in production?

When you talk about a non-converging indexing-scheduler loop, do you mean:

• the scheduled plan keeps changing despite a successful application of the plan (there was a bug in the scheduler logic that has been fixed... this is not what this is about is it?)
• the impossibility to apply a scheduling plan

Can you detail if this is something you actually observed, what happened?

[aanufriev-celonis]

hi @fulmicoton

This one is a real, recurring production problem. We hit it across many clusters, and it is currently our largest source of ingest no_shards errors.

In short, it is your second case: the impossibility to apply a scheduling plan.

The node set is stable, so we are on the has_same_tasks() == false branch (same nodes, different tasks). The last applied plan assigns a shard that the target indexer cannot run, so the loop re-sends the identical plan forever. The plan never changes and it just can never converge.

---

What we observe:

• Indexers log no open shard found on ingester continuously for the affected indexes (they log a lot, 500K logs per hour).
• The control plane sits in the reapply branch, logging the same running tasks and last applied tasks differ: reapply last plan line indefinitely.
• We have seen it persist from several hours up to a couple of days. The only thing that clears it is a manual rolling restart of the indexers, which forces the plan to be rebuilt against a fresh state.
• quickwit_ingest_reset_shards_operations_total stays 0, and there are no shard-reset / kill-switch loops during the stuck period, it is genuinely a stuck, stable non-convergence.