Skip to content

fix(scripts,cli): wait for KafkaUser Secrets to settle before they are copied - #685

Merged
bmscomp merged 2 commits into
mainfrom
fix/kafka-user-secret-settle
Nov 16, 2026
Merged

bmscomp merged 2 commits into
mainfrom
fix/kafka-user-secret-settle

Conversation

@bmscomp

@bmscomp bmscomp commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Why

CI · Operator's E2E failed twice on main at f566a19: run 43946278936, attempts 1 and 2. Every Kates run ended FAILED, and the Helm-installed backend logged Authentication failed against krafter for minutes. #684's PR run had passed on the same tree.

Strimzi 1.2.0's User Operator reads the user's Secret from its informer cache (UserControllerLoop: secretInformer.get(...)). The reconcile its own status update triggers can run before that cache holds the Secret the first reconcile created. That reconcile sees no Secret, generates a new password, sets it in Kafka, and overwrites the Secret: internalCreate gets a 409 and falls back to update(). A copy taken right at the first Ready can hold the password that was just replaced.

e2e-stack.sh (and integration.yml) copy kates-backend right after deploy-kafka-generic.sh returns:

run Waiting for user secrets → Ready copy after Ready E2E
PR #684 0.37 s, already Ready 0.7 s pass
main 18d71a6 0.36 s, already Ready 0.7 s pass
main f566a19, attempt 1 2.3 s, became Ready during the wait 0.6 s auth fails
main f566a19, attempt 2 1.2 s, became Ready during the wait 0.8 s auth fails
diagnostic, ci/kafka-auth-diag the status update came 7 s before the wait 0.8 s pass; both copies hash the same, the Secret was written once, SCRAM login with it works

What changes

  • Scripts. After the Ready wait, deploy-kafka-generic.sh waits until the KafkaUser Secrets' resource versions hold for 10 s, 60 s at most. Every workflow that deploys Kafka, and operator/e2e-stack.sh, goes through it.
  • CLI. kates deploy's kafka-users wait ends with the same settle (waitUserSecretsSettled). Kates, Connect, Kafka UI and MirrorMaker 2 depend on it and copy their Secrets afterwards.

When the Secrets are already stable, either one costs 10 s.

Tests

  • shellcheck -x -S warning -e SC2317,SC2034,SC2155 scripts/*.sh kates-ci.sh, the lint job's command
  • The loop run against a live cluster's kafka namespace (kind panda): it settles after one interval.
  • New: TestUserSecretsAreCopiedOnlyOnceTheyHold and TestUserSecretsThatNeverHoldStopTheWaitAfterAMinute.
  • Local, from cli/: go test ./..., go vet ./..., golangci-lint run --new-from-rev=<merge-base> (0 issues), govulncheck ./..., scripts/check-cli-style.sh, scripts/check-cli-compat.sh.
  • The E2E itself is in this PR's CI · Operator and CI · Integration runs.

Not done

  • If the kafka-users wait times out with users still not Ready, a consumer that waits for its own Secret (Connect) copies it as soon as it appears. That race remains on that already-degraded path.

…pied

Strimzi's User Operator reads a user's Secret from its informer cache. The
reconcile its own status update triggers can run before the cache holds the
Secret the first reconcile created: it issues a new password, sets it in
Kafka and overwrites the Secret (create, 409, update). A copy taken at the
first Ready, as e2e-stack.sh and integration.yml do, can hold the password
that was just replaced, and every Kafka call then fails authentication.

That failed CI · Operator twice on main at f566a19, the tree #684's PR run
passed with: both runs copied within a second of Ready. After the Ready
wait, deploy-kafka-generic.sh now waits until the user Secrets' resource
versions hold for 10 s (60 s at most).
The kafka-users wait now ends with the same settle as
deploy-kafka-generic.sh: the KafkaUser Secrets' resource versions must hold
for 10 s (a minute at most). Kates, Connect, Kafka UI and MirrorMaker 2
copy their Secrets after it, so none copies a password the User Operator
replaces a moment later.
@bmscomp bmscomp changed the title fix(scripts): wait for KafkaUser Secrets to settle before they are copied fix(scripts,cli): wait for KafkaUser Secrets to settle before they are copied Nov 16, 2026
@bmscomp
bmscomp merged commit be64ba1 into main Nov 16, 2026
66 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant