Skip to content

feat(chaos): MEMORY_STRESS, DISK_FILL and NODE_DRAIN on the kubernetes provider - #670

Merged
bmscomp merged 2 commits into
mainfrom
feat/kubernetes-chaos-faults
Nov 15, 2026
Merged

bmscomp merged 2 commits into
mainfrom
feat/kubernetes-chaos-faults

Conversation

@bmscomp

@bmscomp bmscomp commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Why

On the kubernetes chaos provider, MEMORY_STRESS, DISK_FILL and NODE_DRAIN failed with not supported by kubernetes provider, so a cluster without LitmusChaos couldn't run them. All three can be done with the Kubernetes API alone. With this PR only NETWORK_LATENCY and DNS_ERROR still need Litmus: they need privileged helpers in the pod's network namespace.

What changes

  • MEMORY_STRESS: the same unprivileged stress ephemeral container as CPU_STRESS, holding memoryMb (stress --vm 1 --vm-bytes <memoryMb>M --vm-keep).
  • DISK_FILL: an ephemeral container mounts the broker's data volume (the first PersistentVolumeClaim the broker container mounts, else its first data* volume, as Strimzi names them) and writes one kates-disk-fill-* file as the broker's user until the volume is fillPercentage full. The file goes in the volume root, or in the log directory when the root isn't writable. It is removed at the end of the duration or on SIGTERM. A container that can write nowhere exits 1 and fails the step.
  • NODE_DRAIN: cordons the node, evicts its pods through the Eviction API (PodDisruptionBudgets honoured, refusals retried every 5 s), and uncordons it when chaosDurationSec ends, even when the step fails. DaemonSet and static pods stay. A pod still not evicted at the end fails the step. A node that is already cordoned is refused, so Kates never undoes someone else's cordon.
  • Node choice is shared with litmus-crd: the picking code moved from LitmusChaosProvider into DrainTargets, unchanged. TARGET_NODE and NODE_LABEL work on both providers, and the Kates API's own node is never drained.
  • Orphan recovery uncordons a node a drain left cordoned when the Kates API died mid-drain. It finds it by the kates.io/cordoned label, and judges it by the cordon's time and the fault marks.
  • Chart kates 0.20.0: rbac.directChaos also grants patch on nodes and create on pods/eviction. The default role is unchanged. kates-platform 0.10.0 packs it. The operator chart's backend role (rbac.backendDirectChaos) grants the same two rules, as operator/backend-role-check.sh requires.
  • Docs: the book's provider table and the passages that said these types need Litmus. The safety guard's NODE_LABEL warning no longer says Litmus picks the node. Four MCP caveat refs this change moved are re-pointed.

Behaviour that differs from Litmus

  • The stress container is the pod's, not the broker container's, so MEMORY_STRESS (like CPU_STRESS) presses on the pod and its node, not on the broker container's own limits as pod-memory-hog does.
  • DISK_FILL fills the broker's log volume. Litmus disk-fill fills the pod's ephemeral storage.
  • A broker pod killed outright, without SIGTERM, leaves its kates-disk-fill-* file on the volume.

Found while testing

The first fill script ran dd in the foreground. sh runs a trap only once a foreground command ends, so a SIGTERM during a long fill did nothing, the container was killed, and a multi-GB file stayed on the volume. dd and sleep now run in the background and the script waits on them.

Tested locally

  • ./mvnw verify -Dquarkus.grpc.codegen.skip=true -Dquarkus.http.test-port=0: 1,921 unit tests (3 skipped) and 101 ITs on PostgreSQL. spotless:check passes.
  • New tests: memory stress command; disk-fill mount, user and script; disk-fill failing on a container that exited 1; data-volume choice; drain evicting everything but DaemonSet pods and uncordoning; a PodDisruptionBudget refusal retried, then failing, with the node still uncordoned; an already cordoned node refused; the Kates API's node refused; a NODE_LABEL drain avoiding it; orphan recovery uncordoning only stray Kates cordons.
  • The fill script in polinux/stress:1.0.4 as uid 1001 on a tmpfs: fills to the target, removes the file at the end and on a SIGTERM during dd, falls back to a writable subdirectory, and exits 1 when nothing is writable.
  • CLI: go test ./..., go vet, golangci-lint --new-from-rev=origin/main, govulncheck, check-cli-style.sh, check-cli-compat.sh.
  • Charts: operator/backend-role-check.sh, helm lint, yamllint, chart_bumps.py, helm template with and without rbac.directChaos.
  • Book: check-book-style.sh, gen-version-matrix.sh --check, gen-chart-table.sh --check, gen-book-index.py --check, contrast and mermaid checks.

Not covered

  • No run on a live cluster: the drain and the fill haven't run against Kind or a real Strimzi broker.
  • Book PDF render and the Kyverno check weren't run locally.
  • The node-maintenance resilience scenario stays refused, as before.

…s provider

The kubernetes chaos provider refused these three types, so a cluster
without Litmus could not run them. Now only NETWORK_LATENCY and
DNS_ERROR need Litmus.

- MEMORY_STRESS: the unprivileged stress ephemeral container, holding
  memoryMb (stress --vm 1 --vm-keep).
- DISK_FILL: an ephemeral container mounts the broker's data volume
  (first PVC, else first data* volume) and writes one file as the
  broker's user until the volume is fillPercentage full. dd and sleep run
  in the background so a SIGTERM removes the file; a container that can
  write nowhere fails the step.
- NODE_DRAIN: cordon, evict through the Eviction API (PodDisruptionBudgets
  honoured, refusals retried), uncordon when chaosDurationSec ends, even
  on failure. DaemonSet and static pods stay; an already cordoned node is
  refused. The node is picked as on litmus-crd (DrainTargets, moved out of
  LitmusChaosProvider), never the Kates API's own.
- Orphan recovery uncordons a node a drain left cordoned (kates.io/cordoned
  label, cordoned-at annotation, fault marks).
- Chart 0.20.0: rbac.directChaos also grants patch on nodes and create on
  pods/eviction. kates-platform 0.10.0 packs it.
…Kates chart's does

operator/backend-role-check.sh holds the operator's backend ClusterRole to
the kates chart's: with rbac.backendDirectChaos it now also grants patch on
nodes and create on pods/eviction.
@bmscomp
bmscomp merged commit 2535e99 into main Nov 15, 2026
74 of 76 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant