Repository navigation
feat(chaos): MEMORY_STRESS, DISK_FILL and NODE_DRAIN on the kubernetes provider - #670
Merged
Merged
Conversation
…s provider The kubernetes chaos provider refused these three types, so a cluster without Litmus could not run them. Now only NETWORK_LATENCY and DNS_ERROR need Litmus. - MEMORY_STRESS: the unprivileged stress ephemeral container, holding memoryMb (stress --vm 1 --vm-keep). - DISK_FILL: an ephemeral container mounts the broker's data volume (first PVC, else first data* volume) and writes one file as the broker's user until the volume is fillPercentage full. dd and sleep run in the background so a SIGTERM removes the file; a container that can write nowhere fails the step. - NODE_DRAIN: cordon, evict through the Eviction API (PodDisruptionBudgets honoured, refusals retried), uncordon when chaosDurationSec ends, even on failure. DaemonSet and static pods stay; an already cordoned node is refused. The node is picked as on litmus-crd (DrainTargets, moved out of LitmusChaosProvider), never the Kates API's own. - Orphan recovery uncordons a node a drain left cordoned (kates.io/cordoned label, cordoned-at annotation, fault marks). - Chart 0.20.0: rbac.directChaos also grants patch on nodes and create on pods/eviction. kates-platform 0.10.0 packs it.
…Kates chart's does operator/backend-role-check.sh holds the operator's backend ClusterRole to the kates chart's: with rbac.backendDirectChaos it now also grants patch on nodes and create on pods/eviction.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On the
kuberneteschaos provider,MEMORY_STRESS,DISK_FILLandNODE_DRAINfailed withnot supported by kubernetes provider, so a cluster without LitmusChaos couldn't run them. All three can be done with the Kubernetes API alone. With this PR onlyNETWORK_LATENCYandDNS_ERRORstill need Litmus: they need privileged helpers in the pod's network namespace.What changes
MEMORY_STRESS: the same unprivileged stress ephemeral container asCPU_STRESS, holdingmemoryMb(stress --vm 1 --vm-bytes <memoryMb>M --vm-keep).DISK_FILL: an ephemeral container mounts the broker's data volume (the first PersistentVolumeClaim the broker container mounts, else its firstdata*volume, as Strimzi names them) and writes onekates-disk-fill-*file as the broker's user until the volume isfillPercentagefull. The file goes in the volume root, or in the log directory when the root isn't writable. It is removed at the end of the duration or on SIGTERM. A container that can write nowhere exits 1 and fails the step.NODE_DRAIN: cordons the node, evicts its pods through the Eviction API (PodDisruptionBudgets honoured, refusals retried every 5 s), and uncordons it whenchaosDurationSecends, even when the step fails. DaemonSet and static pods stay. A pod still not evicted at the end fails the step. A node that is already cordoned is refused, so Kates never undoes someone else's cordon.litmus-crd: the picking code moved fromLitmusChaosProviderintoDrainTargets, unchanged.TARGET_NODEandNODE_LABELwork on both providers, and the Kates API's own node is never drained.kates.io/cordonedlabel, and judges it by the cordon's time and the fault marks.kates0.20.0:rbac.directChaosalso grantspatchonnodesandcreateonpods/eviction. The default role is unchanged.kates-platform0.10.0 packs it. The operator chart's backend role (rbac.backendDirectChaos) grants the same two rules, asoperator/backend-role-check.shrequires.NODE_LABELwarning no longer says Litmus picks the node. Four MCP caveat refs this change moved are re-pointed.Behaviour that differs from Litmus
MEMORY_STRESS(likeCPU_STRESS) presses on the pod and its node, not on the broker container's own limits aspod-memory-hogdoes.DISK_FILLfills the broker's log volume. Litmusdisk-fillfills the pod's ephemeral storage.kates-disk-fill-*file on the volume.Found while testing
The first fill script ran
ddin the foreground. sh runs a trap only once a foreground command ends, so a SIGTERM during a long fill did nothing, the container was killed, and a multi-GB file stayed on the volume.ddandsleepnow run in the background and the script waits on them.Tested locally
./mvnw verify -Dquarkus.grpc.codegen.skip=true -Dquarkus.http.test-port=0: 1,921 unit tests (3 skipped) and 101 ITs on PostgreSQL.spotless:checkpasses.NODE_LABELdrain avoiding it; orphan recovery uncordoning only stray Kates cordons.polinux/stress:1.0.4as uid 1001 on a tmpfs: fills to the target, removes the file at the end and on a SIGTERM duringdd, falls back to a writable subdirectory, and exits 1 when nothing is writable.go test ./...,go vet,golangci-lint --new-from-rev=origin/main, govulncheck,check-cli-style.sh,check-cli-compat.sh.operator/backend-role-check.sh,helm lint, yamllint,chart_bumps.py,helm templatewith and withoutrbac.directChaos.check-book-style.sh,gen-version-matrix.sh --check,gen-chart-table.sh --check,gen-book-index.py --check, contrast and mermaid checks.Not covered
node-maintenanceresilience scenario stays refused, as before.