Repository navigation
fix(cli,operator): a release deploy runs its own backend, and an old one fails fast - #683
Merged
Merged
Conversation
…one fails fast On kind, kates deploy ran a local kates:native-local whenever Docker had one, even from a release's published charts. One built before the quorum never answers with a quorum, so the operator reported NoLeader and deploy waited out its 10 minutes. - A release's published charts run the published backend on kind too; a checkout deploy still prefers a local build. - The client tells a backend older than the quorum (its health has no quorum at all) from one whose quorum is off or not started yet. - A member of the current spec on such a backend sets Ready=False, reason NoQuorum, naming the image, and stops the roll: it never leads or catches up, so the roll would wait on it forever or move the majority onto it. NoLeader stays for a quorum that is still electing. - Under a leader, the roll replaces a stale follower that runs no quorum (crashing, or older than it) first: it holds no vote, and waiting for it to catch up stalled every rollback from such an image. - Deploy polls the Ready condition, stops at once on NoQuorum, and counts only a condition of the resource's current generation.
bmscomp
added a commit
that referenced
this pull request
Nov 16, 2026
Moves every pin to 1.41.2, the tag this release publishes: the kates chart's appVersion and image tags (0.20.3), kates/k8s/deployment.yaml, the kates-tester pins in strimzi-operator 0.5.6, kafka-cluster 1.1.6, connect-cluster 2.2.6, mirror-maker2 0.12.6 and kates-chaos 2.6.3, the Litmus experiment pins, server.json, the registry docs, the runbook and recipes, the version matrix in appendix D, and kates-platform 0.10.3. kates-operator 0.3.1 (chart and appVersion, and the README's install, verify and pull commands) carries #683's operator side: NoQuorum for a backend older than the quorum, and a roll that replaces a stale member running no quorum first. The CronJob example pins kates-tester 1.41.2. The release ships #683: on kind, a release's published charts run the published backend instead of a stale local kates:native-local, and deploy stops at once on NoQuorum. Nothing under kates/src or tester/ moved since v1.41.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
kates deployfrom kates v1.41.1's published charts failed on kind after a 10-minute wait:On kind, deploy runs a local
kates:native-localwhenever Docker has one, whatever its age, even from a release's published charts. This one was built on 2026-10-25, before the quorum. Its/api/healthhas noquorum, so the operator never sees a leader and reportsNoLeaderindefinitely, and deploy waits out the fullkubectl wait --timeout=10m. Switching the sameKatesresource to the published1.41.1-nativeimage gave three Ready members and a leader in 59 s.Reproducing it live turned up two more hangs:
RollingOut: the next member is replaced once kates-0 is not ready or has not caught up). Such a member never catches up.CI could not see any of this. The operator E2E builds Kates from the same commit and loads it under the published tag, so the runner never has a local build, and backend and operator always match. The
kindBackendImageArgstests predate published charts and cover the checkout case only.What changes
kindBackendImageArgsskips local builds whenchartsPublished. A checkout deploy (--checkout,KATES_CHECKOUT, or a dev build) still prefers them.QuorumView.predatesis set when/api/healthhas noquorumat all. A new backend whose quorum is unconfigured or still starting still answersenabled: falsewithoutpredates, so it is never mistaken for an old one.ReadyisFalsewith reasonNoQuorum, the message names the image, and the roll stops there. Moving more members onto that image would cost the quorum its majority.NoLeaderstays for a quorum that is still electing.NoQuorum.waitKatesResourcepolls the Ready condition instead of runningkubectl wait. It returns at once onNoQuorum, and counts a condition only when itsobservedGenerationis the resource's generation.kubectl waitalso accepted aReadyleft from before the applied spec.Tests
TestKindBackendImageArgsUsesThePublishedImageWithPublishedCharts,TestAKatesResourceWithoutAQuorumStopsTheWaitAtOnce,TestOnlyTheCurrentGenerationsConditionCounts,only_a_health_with_no_quorum_at_all_predates_it,only_a_current_member_older_than_the_quorum_is_without_one, anda_stale_follower_that_runs_no_quorum_is_replaced_first_under_a_leader.panda), with this branch's operator. A busybox stand-in serving/api/healthwithoutquorumplayed the old backend; the realnative-localcrash-loops there, because that cluster's database already has V27 and V28.NoQuorum: kates-0 runs kates-old-backend:1, which is older than the quorum, …within 10 s. The roll stopped, and the other two members kept 1.41.1 and their leader.4|4|False|NoQuorum|….1.41.1-nativestalled under the operator's previous build (RollingOut … kates-0 … has not caught up). With the roll change, it completed in 77 s: three members on 1.41.1, leaderkates-1. Earlier, with the realnative-localcrash-looping, the same stall had needed a manualkubectl delete pod.cli/:go test ./...,go vet ./...,golangci-lint run --new-from-rev=<merge-base>(0 issues),govulncheck ./...,scripts/check-cli-style.sh,scripts/check-cli-compat.sh.operator/:cargo fmt --check,cargo clippy --all-targets --all-features -D warnings, andcargo test --workspace --all-features(43 + 12 + 6 + 1).cargo denyisn't installed here, and no dependency changed../mvnw test, 1,944 tests, 0 failures (CI's Build & Test runs oncli/**).Not done
NoQuorumneeds the operator from this PR. Against operator 0.3.0, deploy still times out onNoLeader.