Repository navigation
fix(scripts,cli): wait for KafkaUser Secrets to settle before they are copied - #685
Merged
Merged
Conversation
…pied Strimzi's User Operator reads a user's Secret from its informer cache. The reconcile its own status update triggers can run before the cache holds the Secret the first reconcile created: it issues a new password, sets it in Kafka and overwrites the Secret (create, 409, update). A copy taken at the first Ready, as e2e-stack.sh and integration.yml do, can hold the password that was just replaced, and every Kafka call then fails authentication. That failed CI · Operator twice on main at f566a19, the tree #684's PR run passed with: both runs copied within a second of Ready. After the Ready wait, deploy-kafka-generic.sh now waits until the user Secrets' resource versions hold for 10 s (60 s at most).
The kafka-users wait now ends with the same settle as deploy-kafka-generic.sh: the KafkaUser Secrets' resource versions must hold for 10 s (a minute at most). Kates, Connect, Kafka UI and MirrorMaker 2 copy their Secrets after it, so none copies a password the User Operator replaces a moment later.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
CI · Operator's E2E failed twice on main at f566a19: run 43946278936, attempts 1 and 2. Every Kates run ended
FAILED, and the Helm-installed backend loggedAuthentication failedagainstkrafterfor minutes. #684's PR run had passed on the same tree.Strimzi 1.2.0's User Operator reads the user's Secret from its informer cache (
UserControllerLoop:secretInformer.get(...)). The reconcile its own status update triggers can run before that cache holds the Secret the first reconcile created. That reconcile sees no Secret, generates a new password, sets it in Kafka, and overwrites the Secret:internalCreategets a 409 and falls back toupdate(). A copy taken right at the firstReadycan hold the password that was just replaced.e2e-stack.sh(andintegration.yml) copykates-backendright afterdeploy-kafka-generic.shreturns:Waiting for user secrets→ReadyReadyci/kafka-auth-diagWhat changes
Readywait,deploy-kafka-generic.shwaits until the KafkaUser Secrets' resource versions hold for 10 s, 60 s at most. Every workflow that deploys Kafka, andoperator/e2e-stack.sh, goes through it.kates deploy'skafka-userswait ends with the same settle (waitUserSecretsSettled). Kates, Connect, Kafka UI and MirrorMaker 2 depend on it and copy their Secrets afterwards.When the Secrets are already stable, either one costs 10 s.
Tests
shellcheck -x -S warning -e SC2317,SC2034,SC2155 scripts/*.sh kates-ci.sh, the lint job's commandkafkanamespace (kindpanda): it settles after one interval.TestUserSecretsAreCopiedOnlyOnceTheyHoldandTestUserSecretsThatNeverHoldStopTheWaitAfterAMinute.cli/:go test ./...,go vet ./...,golangci-lint run --new-from-rev=<merge-base>(0 issues),govulncheck ./...,scripts/check-cli-style.sh,scripts/check-cli-compat.sh.Not done
kafka-userswait times out with users still not Ready, a consumer that waits for its own Secret (Connect) copies it as soon as it appears. That race remains on that already-degraded path.