Skip to content

fix(cli,operator): a release deploy runs its own backend, and an old one fails fast - #683

Merged
bmscomp merged 1 commit into
mainfrom
fix/deploy-quorum-image
Nov 16, 2026
Merged

bmscomp merged 1 commit into
mainfrom
fix/deploy-quorum-image

Conversation

@bmscomp

@bmscomp bmscomp commented Nov 16, 2026

Copy link
Copy Markdown
Owner

Why

kates deploy from kates v1.41.1's published charts failed on kind after a 10-minute wait:

Kates Backend: the Kates resource kates/kates is not Ready (NoLeader: the quorum has no leader): exit status 1: error: timed out waiting for the condition on kates/kates

On kind, deploy runs a local kates:native-local whenever Docker has one, whatever its age, even from a release's published charts. This one was built on 2026-10-25, before the quorum. Its /api/health has no quorum, so the operator never sees a leader and reports NoLeader indefinitely, and deploy waits out the full kubectl wait --timeout=10m. Switching the same Kates resource to the published 1.41.1-native image gave three Ready members and a leader in 59 s.

Reproducing it live turned up two more hangs:

  • A roll onto an image older than the quorum waits forever on the first member it moves (RollingOut: the next member is replaced once kates-0 is not ready or has not caught up). Such a member never catches up.
  • Rolling back stalls for the same reason. The member to be replaced is the one blocking the roll, whether it crash-loops or runs no quorum.

CI could not see any of this. The operator E2E builds Kates from the same commit and loads it under the published tag, so the runner never has a local build, and backend and operator always match. The kindBackendImageArgs tests predate published charts and cover the checkout case only.

What changes

  • A release's published charts run the release's backend. kindBackendImageArgs skips local builds when chartsPublished. A checkout deploy (--checkout, KATES_CHECKOUT, or a dev build) still prefers them.
  • The client tells an old backend from a quorum that is off. QuorumView.predates is set when /api/health has no quorum at all. A new backend whose quorum is unconfigured or still starting still answers enabled: false without predates, so it is never mistaken for an old one.
  • The operator says when no election or wait can help. When a member on the current spec runs such a backend, Ready is False with reason NoQuorum, the message names the image, and the roll stops there. Moving more members onto that image would cost the quorum its majority. NoLeader stays for a quorum that is still electing.
  • The roll replaces a stale follower that runs no quorum first, under a leader. That covers a member that is crashing or older than the quorum. It holds no vote, so replacing it costs nothing, and waiting for it to catch up never ends. Healthy stale voters still wait for it, and the leader still goes last.
  • Deploy stops on NoQuorum. waitKatesResource polls the Ready condition instead of running kubectl wait. It returns at once on NoQuorum, and counts a condition only when its observedGeneration is the resource's generation. kubectl wait also accepted a Ready left from before the applied spec.

Tests

  • New: TestKindBackendImageArgsUsesThePublishedImageWithPublishedCharts, TestAKatesResourceWithoutAQuorumStopsTheWaitAtOnce, TestOnlyTheCurrentGenerationsConditionCounts, only_a_health_with_no_quorum_at_all_predates_it, only_a_current_member_older_than_the_quorum_is_without_one, and a_stale_follower_that_runs_no_quorum_is_replaced_first_under_a_leader.
  • Live, on a 3-node kind cluster (panda), with this branch's operator. A busybox stand-in serving /api/health without quorum played the old backend; the real native-local crash-loops there, because that cluster's database already has V27 and V28.
    • Moving a member onto it gave NoQuorum: kates-0 runs kates-old-backend:1, which is older than the quorum, … within 10 s. The roll stopped, and the other two members kept 1.41.1 and their leader.
    • The CLI's Ready jsonpath, run against that live resource, returns 4|4|False|NoQuorum|….
    • Rolling back to 1.41.1-native stalled under the operator's previous build (RollingOut … kates-0 … has not caught up). With the roll change, it completed in 77 s: three members on 1.41.1, leader kates-1. Earlier, with the real native-local crash-looping, the same stall had needed a manual kubectl delete pod.
  • Local, from cli/: go test ./..., go vet ./..., golangci-lint run --new-from-rev=<merge-base> (0 issues), govulncheck ./..., scripts/check-cli-style.sh, scripts/check-cli-compat.sh.
  • Local, from operator/: cargo fmt --check, cargo clippy --all-targets --all-features -D warnings, and cargo test --workspace --all-features (43 + 12 + 6 + 1). cargo deny isn't installed here, and no dependency changed.
  • Local: backend ./mvnw test, 1,944 tests, 0 failures (CI's Build & Test runs on cli/**).

Not done

  • Fail-fast on NoQuorum needs the operator from this PR. Against operator 0.3.0, deploy still times out on NoLeader.
  • A checkout deploy with a stale local build still runs it. It now fails within a reconcile, naming the image, instead of after 10 minutes.
  • No E2E pairs an operator with an older backend. The live run above was by hand.

…one fails fast

On kind, kates deploy ran a local kates:native-local whenever Docker had
one, even from a release's published charts. One built before the quorum
never answers with a quorum, so the operator reported NoLeader and deploy
waited out its 10 minutes.

- A release's published charts run the published backend on kind too; a
  checkout deploy still prefers a local build.
- The client tells a backend older than the quorum (its health has no
  quorum at all) from one whose quorum is off or not started yet.
- A member of the current spec on such a backend sets Ready=False,
  reason NoQuorum, naming the image, and stops the roll: it never leads
  or catches up, so the roll would wait on it forever or move the
  majority onto it. NoLeader stays for a quorum that is still electing.
- Under a leader, the roll replaces a stale follower that runs no quorum
  (crashing, or older than it) first: it holds no vote, and waiting for
  it to catch up stalled every rollback from such an image.
- Deploy polls the Ready condition, stops at once on NoQuorum, and counts
  only a condition of the resource's current generation.
@bmscomp
bmscomp merged commit 18d71a6 into main Nov 16, 2026
65 checks passed
@bmscomp
bmscomp deleted the fix/deploy-quorum-image branch November 16, 2026 19:39
bmscomp added a commit that referenced this pull request Nov 16, 2026
Moves every pin to 1.41.2, the tag this release publishes: the kates
chart's appVersion and image tags (0.20.3), kates/k8s/deployment.yaml,
the kates-tester pins in strimzi-operator 0.5.6, kafka-cluster 1.1.6,
connect-cluster 2.2.6, mirror-maker2 0.12.6 and kates-chaos 2.6.3, the
Litmus experiment pins, server.json, the registry docs, the runbook and
recipes, the version matrix in appendix D, and kates-platform 0.10.3.

kates-operator 0.3.1 (chart and appVersion, and the README's install,
verify and pull commands) carries #683's operator side: NoQuorum for a
backend older than the quorum, and a roll that replaces a stale member
running no quorum first. The CronJob example pins kates-tester 1.41.2.

The release ships #683: on kind, a release's published charts run the
published backend instead of a stale local kates:native-local, and
deploy stops at once on NoQuorum. Nothing under kates/src or tester/
moved since v1.41.0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant