From 6c8afc93e58994b314ab7940a362d9571c7971a5 Mon Sep 17 00:00:00 2001 From: Said Boudjelda Date: Sun, 15 Nov 2026 17:38:31 +0100 Subject: [PATCH 1/2] feat(chaos): MEMORY_STRESS, DISK_FILL and NODE_DRAIN on the kubernetes provider The kubernetes chaos provider refused these three types, so a cluster without Litmus could not run them. Now only NETWORK_LATENCY and DNS_ERROR need Litmus. - MEMORY_STRESS: the unprivileged stress ephemeral container, holding memoryMb (stress --vm 1 --vm-keep). - DISK_FILL: an ephemeral container mounts the broker's data volume (first PVC, else first data* volume) and writes one file as the broker's user until the volume is fillPercentage full. dd and sleep run in the background so a SIGTERM removes the file; a container that can write nowhere fails the step. - NODE_DRAIN: cordon, evict through the Eviction API (PodDisruptionBudgets honoured, refusals retried), uncordon when chaosDurationSec ends, even on failure. DaemonSet and static pods stay; an already cordoned node is refused. The node is picked as on litmus-crd (DrainTargets, moved out of LitmusChaosProvider), never the Kates API's own. - Orphan recovery uncordons a node a drain left cordoned (kates.io/cordoned label, cordoned-at annotation, fault marks). - Chart 0.20.0: rbac.directChaos also grants patch on nodes and create on pods/eviction. kates-platform 0.10.0 packs it. --- charts/kates-platform/Chart.yaml | 6 +- charts/kates/Chart.yaml | 6 +- charts/kates/README.md | 6 +- charts/kates/templates/rbac.yaml | 12 +- cli/cmd/mcp_tools_chaos.go | 4 +- cli/cmd/mcp_tools_runs_activity.go | 4 +- cli/examples/README.md | 2 +- docs/book/07-chaos-practice.md | 33 +- docs/book/appendix-d-versions.md | 4 +- docs/book/appendix-index.md | 2 +- .../com/bmscomp/kates/chaos/DrainTargets.java | 162 +++++++ .../kates/chaos/KubernetesChaosProvider.java | 357 ++++++++++++++ .../kates/chaos/LitmusChaosProvider.java | 105 +---- .../DisruptionOrphanReconciler.java | 61 ++- .../disruption/DisruptionSafetyGuard.java | 2 +- .../kates/resilience/ResilienceResource.java | 7 +- .../chaos/KubernetesChaosProviderTest.java | 441 ++++++++++++++++++ .../DisruptionOrphanReconcilerTest.java | 56 +++ .../disruption/DisruptionSafetyGuardTest.java | 2 +- 19 files changed, 1134 insertions(+), 138 deletions(-) create mode 100644 kates/src/main/java/com/bmscomp/kates/chaos/DrainTargets.java diff --git a/charts/kates-platform/Chart.yaml b/charts/kates-platform/Chart.yaml index a2f517d2..74f8c3b1 100644 --- a/charts/kates-platform/Chart.yaml +++ b/charts/kates-platform/Chart.yaml @@ -3,8 +3,8 @@ name: kates-platform description: Umbrella chart for the full Kates platform — Kafka, Kates, and supporting infrastructure icon: https://raw.githubusercontent.com/bmscomp/kates/main/docs/assets/charts/kates-platform.svg type: application -# Patch: the umbrella packs kates 0.19.5 (the CNPG JDBC URL's 30 s socket timeout). -version: 0.9.5 +# Minor: the umbrella packs kates 0.20.0 (rbac.directChaos grants what NODE_DRAIN needs). +version: 0.10.0 appVersion: "1.0.0" keywords: - kafka @@ -41,6 +41,8 @@ dependencies: condition: connect-cluster.enabled annotations: artifacthub.io/changes: | + - kind: changed + description: "Packs kates 0.20.0: rbac.directChaos also grants patch on nodes and create on pods/eviction, for NODE_DRAIN on the kubernetes chaos provider. The default role is unchanged." - kind: changed description: "Packs kates 0.19.5: with cnpg.enabled, the JDBC URL's socketTimeout is 30 s, the backend's own. Templates are otherwise unchanged." - kind: changed diff --git a/charts/kates/Chart.yaml b/charts/kates/Chart.yaml index 7bc62855..0fee4566 100644 --- a/charts/kates/Chart.yaml +++ b/charts/kates/Chart.yaml @@ -2,8 +2,8 @@ apiVersion: v2 name: kates description: Kates — Kafka Advanced Testing & Engineering Suite type: application -# Patch: with cnpg.enabled, the JDBC URL's socket timeout is 30 s, as the backend's own. -version: 0.19.5 +# Minor: rbac.directChaos grants patch on nodes and create on pods/eviction, for NODE_DRAIN. +version: 0.20.0 # The non-blocking health endpoints this chart's probes now point at come from # a BUILD-TIME property (quarkus.messaging.health.enabled=false), so they only # exist in the image built from this branch. Keep appVersion and @@ -27,6 +27,8 @@ annotations: artifacthub.io/category: integration-delivery artifacthub.io/license: Apache-2.0 artifacthub.io/changes: | + - kind: added + description: "rbac.directChaos also grants patch on nodes and create on pods/eviction: on the kubernetes chaos provider, NODE_DRAIN cordons a node, evicts its pods and uncordons it. MEMORY_STRESS and DISK_FILL run there too, through pods/ephemeralcontainers, which it already granted. They need a Kates image that has them; the default role is unchanged" - kind: changed description: "With cnpg.enabled, the JDBC URL's socketTimeout is 30 s instead of 60 s, the backend's own: a request against an unreachable database is answered 503 within about 30 s. Templates are otherwise unchanged." - kind: changed diff --git a/charts/kates/README.md b/charts/kates/README.md index 8b832825..8f97a581 100644 --- a/charts/kates/README.md +++ b/charts/kates/README.md @@ -156,11 +156,13 @@ With the default chaos provider, `litmus-crd`, Kates creates and watches `ChaosE | Resource | Verbs | Used by | |----------|-------|---------| | `apps` `statefulsets/scale` | `get`, `update` | On a Kafka that Strimzi does not manage: `SCALE_DOWN`, its rollback and orphan recovery set replicas | -| `pods/ephemeralcontainers` | `update` | `CPU_STRESS` and `IO_STRESS` add a stress container to the target pod | +| `pods/ephemeralcontainers` | `update` | `CPU_STRESS`, `MEMORY_STRESS` and `IO_STRESS` add a stress container to the target pod; `DISK_FILL` adds one that fills its data volume | +| `nodes` | `patch` | `NODE_DRAIN` cordons the node and uncordons it when the duration ends; orphan recovery uncordons a node a drain left cordoned | +| `pods/eviction` | `create` | `NODE_DRAIN` evicts the pods on the node, in every namespace, through the Eviction API | `NETWORK_PARTITION` needs none of these, on either backend: Kates creates its deny-all NetworkPolicy itself, and cleanup, rollback and startup orphan recovery delete it, so the default role grants `create`, `delete` and `deletecollection` on `networkpolicies`. `ROLLING_RESTART` needs none of them either, and neither does `SCALE_DOWN` on a Strimzi cluster. On a Strimzi cluster `ROLLING_RESTART` annotates the Kafka pods for the Strimzi Cluster Operator to roll, and on a plain StatefulSet, such as `charts/legacy-kafka`, it restarts the StatefulSet. `SCALE_DOWN` lowers `spec.replicas` of a KafkaNodePool. Both run on either backend, so the default role grants `patch` on pods, on StatefulSets and on `kafkanodepools`. On a Kafka that Strimzi does not manage, `SCALE_DOWN` also patches the StatefulSet, to record its original replica count. -These rules are cluster-wide, like the rest of the role, and `pods/ephemeralcontainers` lets Kates start a container in any pod. Leave them off unless you use the direct backend. It is selected with `chaos.provider`: +These rules are cluster-wide, like the rest of the role, `pods/ephemeralcontainers` lets Kates start a container in any pod, and `nodes` `patch` with `pods/eviction` lets it drain any node. Leave them off unless you use the direct backend. It is selected with `chaos.provider`: ```yaml chaos: diff --git a/charts/kates/templates/rbac.yaml b/charts/kates/templates/rbac.yaml index ea27662f..69c65e27 100644 --- a/charts/kates/templates/rbac.yaml +++ b/charts/kates/templates/rbac.yaml @@ -66,8 +66,10 @@ rules: every experiment as its own litmus-admin service account, so a Litmus install needs none of them. On a Kafka that Strimzi does not manage, SCALE_DOWN sets replicas through the StatefulSet's scale subresource (its - PATCH on the StatefulSet is granted above, for ROLLING_RESTART); CPU_STRESS - and IO_STRESS add a container through pods/ephemeralcontainers. + PATCH on the StatefulSet is granted above, for ROLLING_RESTART); CPU_STRESS, + MEMORY_STRESS, IO_STRESS and DISK_FILL add a container through + pods/ephemeralcontainers; NODE_DRAIN cordons and uncordons a node (patch) + and evicts its pods in every namespace through pods/eviction. */}} - apiGroups: ["apps"] resources: ["statefulsets/scale"] @@ -75,6 +77,12 @@ rules: - apiGroups: [""] resources: ["pods/ephemeralcontainers"] verbs: ["update"] + - apiGroups: [""] + resources: ["nodes"] + verbs: ["patch"] + - apiGroups: [""] + resources: ["pods/eviction"] + verbs: ["create"] {{- end }} {{- with .Values.rbac.extraRules }} {{- toYaml . | nindent 2 }} diff --git a/cli/cmd/mcp_tools_chaos.go b/cli/cmd/mcp_tools_chaos.go index c573c967..c53668f3 100644 --- a/cli/cmd/mcp_tools_chaos.go +++ b/cli/cmd/mcp_tools_chaos.go @@ -280,7 +280,7 @@ var mcpCaveatsChaos = []mcpCaveat{ "CompletableFuture.delayedExecutor(spec.delayBeforeSec(), TimeUnit.SECONDS, executor.get());", "long startNanos = System.nanoTime();", "buildChaosEngine(spec, engineName, experimentName)", "onInject.accept(startNanos);", ".resource(engine)"), - mcpAnchoredRef(mcpJava+"chaos/KubernetesChaosProvider.java:170-182", + mcpAnchoredRef(mcpJava+"chaos/KubernetesChaosProvider.java:178-190", "Thread.sleep(spec.delayBeforeSec() * 1000L);", "onInject.accept(startNanos);", "applyDisruption(spec, engineName, marks);"), mcpAnchoredRef(mcpJava+"chaos/PodTargets.java:58,69-75", @@ -316,7 +316,7 @@ var mcpCaveatsChaos = []mcpCaveat{ "starts while another runs a plan marks that plan's report too, until the plan ends and stores its " + "outcome.", Refs: []string{ - mcpJava + "disruption/DisruptionOrphanReconciler.java:71-75,80-109,116-166,334-368", + mcpJava + "disruption/DisruptionOrphanReconciler.java:71-75,80-109,116-168,387-421", mcpJava + "disruption/DisruptionReportRepository.java:35-60,93-102", mcpJava + "disruption/DisruptionPersistence.java:26-37", mcpJava + "disruption/DisruptionLauncher.java:114-121,129-144", diff --git a/cli/cmd/mcp_tools_runs_activity.go b/cli/cmd/mcp_tools_runs_activity.go index 748ed5ac..40b58ec1 100644 --- a/cli/cmd/mcp_tools_runs_activity.go +++ b/cli/cmd/mcp_tools_runs_activity.go @@ -324,7 +324,7 @@ func mcpActivityTestsSince(ctx context.Context, call *mcpCall, since time.Time, // are injected for part of its run, or one whose backend process stopped // before it ended and has not started again to mark it INTERRUPTED // (DisruptionLauncher.java:95-123, DisruptionReportRepository.java:35-43, -// DisruptionOrphanReconciler.java:359-393). +// DisruptionOrphanReconciler.java:412-446). func mcpActivityDisruptionsSince(ctx context.Context, call *mcpCall, since time.Time, page mcpActivityPage) (mcpActivityDisruptions, error) { out := mcpActivityDisruptions{StoredAsRunning: []mcpActivityDisruptionRow{}, Since: []mcpActivityDisruptionRow{}} for p := 0; p < mcpActivityMaxPages; p++ { @@ -486,7 +486,7 @@ var mcpCaveatsRunsActivity = []mcpCaveat{ mcpJava + "disruption/DisruptionLauncher.java:114-144", mcpJava + "disruption/DisruptionPersistence.java:16-37", mcpJava + "disruption/DisruptionReportRepository.java:20-43", - mcpJava + "disruption/DisruptionOrphanReconciler.java:334-368", + mcpJava + "disruption/DisruptionOrphanReconciler.java:387-421", mcpJava + "disruption/DisruptionTemplateResource.java:70-72", mcpJava + "disruption/DisruptionScheduler.java:112-124", mcpJava + "disruption/DisruptionReportEntity.java:39-47", diff --git a/cli/examples/README.md b/cli/examples/README.md index a45b41b6..62c88ca1 100644 --- a/cli/examples/README.md +++ b/cli/examples/README.md @@ -158,7 +158,7 @@ that sets a field its run doesn't read, or that the API would refuse. | `networkLatencyMs` | int | Added latency ms (`NETWORK_LATENCY`) | | `targetTopic` | string | A disruption plan aims a fault at the leader of this topic's `targetPartition`. A resilience run doesn't, but a `DNS_ERROR` on `litmus-crd` takes it as `TARGET_HOSTNAMES`. It has to be a Kafka topic name: letters, digits, `.`, `_` or `-` | | `targetPartition` | int | Partition of `targetTopic`, in a disruption plan only | -| `envOverrides` | map | Env vars for the Litmus experiment (`litmus-crd` only). Only `TARGET_NODE` or `NODE_LABEL`, which pick the node a `NODE_DRAIN` drains instead of the pod `targetLabel` picks; any other variable is refused | +| `envOverrides` | map | Only `TARGET_NODE` or `NODE_LABEL`, which pick the node a `NODE_DRAIN` drains instead of the pod `targetLabel` picks, on either chaos provider; any other variable is refused | ### `probes` field reference diff --git a/docs/book/07-chaos-practice.md b/docs/book/07-chaos-practice.md index 1129af66..e0e4ddf7 100644 --- a/docs/book/07-chaos-practice.md +++ b/docs/book/07-chaos-practice.md @@ -81,11 +81,11 @@ The table compares the four values. A stock install uses `litmus-crd`, the first | Provider | Needs | Fault types it runs | |-----------|----------------|-------------------------------------------| | `litmus-crd` | LitmusChaos, with each type's experiment installed | Every type through a Litmus experiment, except `POD_DELETE`, `NETWORK_PARTITION`, `ROLLING_RESTART` and `SCALE_DOWN`, which use the Kubernetes API | -| `kubernetes` | Nothing beyond Kates | `POD_KILL`, `POD_DELETE`, `LEADER_ELECTION`, `NETWORK_PARTITION`, `CPU_STRESS`, `IO_STRESS`, `ROLLING_RESTART`, `SCALE_DOWN` | +| `kubernetes` | Nothing beyond Kates | `POD_KILL`, `POD_DELETE`, `LEADER_ELECTION`, `NETWORK_PARTITION`, `CPU_STRESS`, `MEMORY_STRESS`, `IO_STRESS`, `DISK_FILL`, `NODE_DRAIN`, `ROLLING_RESTART`, `SCALE_DOWN` | | `hybrid` | Nothing: it uses LitmusChaos when the Litmus [CRDs](appendix-a-glossary.md#gl-crd) exist | Those of the provider it picks | | `noop` | Nothing | None: every fault comes back `Skipped` | -So `NETWORK_LATENCY`, `MEMORY_STRESS`, `DNS_ERROR`, `DISK_FILL` and `NODE_DRAIN` need LitmusChaos, and [The Hybrid Provider](#the-hybrid-provider) shows how `hybrid` picks. +So only `NETWORK_LATENCY` and `DNS_ERROR` need LitmusChaos, and [The Hybrid Provider](#the-hybrid-provider) shows how `hybrid` picks. On a stock install you already have LitmusChaos: `kates deploy`, and so `make all`, installs the `kates-chaos` chart unless you pass `--with-chaos=false`. The chart installs nine experiments in its target namespaces, `kafka` by default: `pod-delete`, `pod-cpu-hog`, `pod-memory-hog`, `pod-network-partition`, `pod-io-stress`, `pod-dns-error`, `disk-fill`, `pod-network-latency` and `node-drain`; Kates doesn't use `pod-network-partition`. A step whose experiment isn't installed in the target namespace fails at once, with the missing experiment named, before any ChaosEngine exists. @@ -93,7 +93,7 @@ When Kates picks the provider, it checks that the provider can work, and falls b Kates picks the provider once per Kates API process and keeps it until the pod restarts. After you install LitmusChaos, change the provider or clear a fallback, restart the Kates API Deployment; on the lab that's `kubectl rollout restart deploy/kates -n kates`. -To switch providers, set `KATES_CHAOS_PROVIDER` through the `kates` chart's `extraEnv`. The direct Kubernetes provider makes its own writes, so it also needs `rbac.directChaos` for `CPU_STRESS`, `IO_STRESS` and the `SCALE_DOWN` of a StatefulSet: +To switch providers, set `KATES_CHAOS_PROVIDER` through the `kates` chart's `extraEnv`. The direct Kubernetes provider makes its own writes, so it also needs `rbac.directChaos` for `CPU_STRESS`, `MEMORY_STRESS`, `IO_STRESS`, `DISK_FILL`, `NODE_DRAIN` and the `SCALE_DOWN` of a StatefulSet: ```yaml extraEnv: @@ -103,7 +103,7 @@ rbac: directChaos: true ``` -Leave `rbac.directChaos` off with `litmus-crd`. Litmus runs its experiments as its own `litmus-admin` service account, and the extra rules are cluster-wide: they let Kates scale any StatefulSet and start a container in any pod. +Leave `rbac.directChaos` off with `litmus-crd`. Litmus runs its experiments as its own `litmus-admin` service account, and the extra rules are cluster-wide: they let Kates scale any StatefulSet, start a container in any pod and drain any node. Kates doesn't check a type against the provider before the fault. `kates disruption types` lists every type whatever the provider, and neither the safety guard nor the dry run looks, so an unsupported type fails only when its step reaches the fault. On `kubernetes`, the step's `Failure` line in `kates disruption status` reads `DisruptionType not supported by kubernetes provider`. On `litmus-crd`, Kates creates the ChaosEngine even when its experiment isn't installed, and waits up to `chaosDurationSec` plus two minutes for a result. @@ -111,7 +111,7 @@ Each Litmus fault leaves a ChaosEngine named `kates-- ### The `kubernetes` Provider {#direct-kubernetes-api} -The `kubernetes` provider implements these disruptions against the Kubernetes API directly. It needs no extra tooling, but `CPU_STRESS`, `IO_STRESS` and the `SCALE_DOWN` of a StatefulSet need `rbac.directChaos=true` on the `kates` chart: +The `kubernetes` provider implements these disruptions against the Kubernetes API directly. It needs no extra tooling, but `CPU_STRESS`, `MEMORY_STRESS`, `IO_STRESS`, `DISK_FILL`, `NODE_DRAIN` and the `SCALE_DOWN` of a StatefulSet need `rbac.directChaos=true` on the `kates` chart: | Type | Implementation | Effect | |------|---------------|--------| @@ -122,13 +122,20 @@ The `kubernetes` provider implements these disruptions against the Kubernetes AP | `SCALE_DOWN` | Lower each selected [KafkaNodePool](appendix-a-glossary.md#gl-kafkanodepool)'s `spec.replicas` by one and wait for the operator to remove a broker; see [Scaling Down a Node Pool](#scaling-down-a-node-pool) | One broker fewer per node pool, until rollback or the end of the run puts it back | | `NETWORK_PARTITION` | Create a [NetworkPolicy](appendix-a-glossary.md#gl-networkpolicy) with no allow rules that selects the target pod | Adds a NetworkPolicy with no allow rules to the pod, removed after `chaosDurationSec` | | `CPU_STRESS` | A non-root stress ephemeral container, `chaos.stressImage`, injected into the pod | Saturates CPU on the broker pod | +| `MEMORY_STRESS` | The same container, holding `memoryMb` | Presses on the memory of the broker pod and its node | | `IO_STRESS` | The same container, with I/O workers | Injects disk I/O pressure on broker storage | +| `DISK_FILL` | An ephemeral container that mounts the pod's data volume and writes one file, as the broker's user, until the volume is `fillPercentage` full | Fills the broker's log volume, removed after `chaosDurationSec` | +| `NODE_DRAIN` | Cordon the node, evict its pods through the Eviction API, uncordon it after `chaosDurationSec` | Every pod on the node moves or waits, as in a node or [zone](appendix-a-glossary.md#gl-zone) failure | + +Three of these don't do what their Litmus experiments do. A stress container is the pod's, not the broker container's, so `MEMORY_STRESS` and `CPU_STRESS` press on the pod and its node, not on the broker container's own limits as `pod-memory-hog` and `pod-cpu-hog` do. `DISK_FILL` fills the volume that holds the broker's logs: the first `PersistentVolumeClaim` the broker container mounts, or else its first volume named `data*`, as Strimzi names them. Litmus `disk-fill` fills the pod's ephemeral storage instead. The file is named `kates-disk-fill-*`, goes in the volume's root or, when the broker's user can't write there, in the broker's log directory, and is removed when the duration ends or the pod stops. A broker pod killed outright, without SIGTERM, leaves it on the volume. A container that can write nowhere on the volume fails the step. + +`NODE_DRAIN` evicts every pod on the node in every namespace, as `kubectl drain` does, except DaemonSet and static pods. An eviction a PodDisruptionBudget refuses is tried again every five seconds. A pod still not evicted when `chaosDurationSec` ends fails the step, and the node is uncordoned all the same. Kates refuses a node already cordoned, because uncordoning it at the end would undo someone else's cordon. It labels the node it cordons `kates.io/cordoned`, and when the Kates API restarts mid-drain, orphan recovery uncordons it. The provider injects each fault once, and doesn't try it again when a call fails. The API server may have carried out the call and lost only its answer. A second try would then act on what the first changed: it would kill another pod picked at random, or scale a StatefulSet down from the count the first try left. The step fails instead. A partition that fails or is interrupted removes its NetworkPolicies all the same, as at the end of its duration, and a failed node pool scale-down gives the pools their replicas back. Removing a partition is the one call the provider retries, up to three more times, because doing it again removes nothing more. ### The `litmus-crd` Provider {#litmuschaos-integration} -On the default `litmus-crd` provider, Kates maps every disruption type but four to a Litmus experiment (for example, `POD_KILL` and `LEADER_ELECTION` both map to `pod-delete`). Four types never go through Litmus. `NETWORK_PARTITION` doesn't, because `pod-network-partition` ignores the pods Kates picks and fails without an `appinfo` label, which Kates can't give it for one pod. `ROLLING_RESTART` doesn't, because no Litmus experiment does a rolling restart. `SCALE_DOWN` doesn't, because `pod-delete` kills a broker that its [StrimziPodSet](appendix-a-glossary.md#gl-strimzipodset) brings straight back. And `POD_DELETE` doesn't, because `pod-delete` never deletes a pod with `gracePeriodSec`. It deletes the pod at once with the `FORCE=true` that pods of a StrimziPodSet need, and with the pod's own grace period otherwise. The `litmus-crd` provider hands all four to the `kubernetes` provider, so they run the same way on both. Five types are only available through Litmus: +On the default `litmus-crd` provider, Kates maps every disruption type but four to a Litmus experiment (for example, `POD_KILL` and `LEADER_ELECTION` both map to `pod-delete`). Four types never go through Litmus. `NETWORK_PARTITION` doesn't, because `pod-network-partition` ignores the pods Kates picks and fails without an `appinfo` label, which Kates can't give it for one pod. `ROLLING_RESTART` doesn't, because no Litmus experiment does a rolling restart. `SCALE_DOWN` doesn't, because `pod-delete` kills a broker that its [StrimziPodSet](appendix-a-glossary.md#gl-strimzipodset) brings straight back. And `POD_DELETE` doesn't, because `pod-delete` never deletes a pod with `gracePeriodSec`. It deletes the pod at once with the `FORCE=true` that pods of a StrimziPodSet need, and with the pod's own grace period otherwise. The `litmus-crd` provider hands all four to the `kubernetes` provider, so they run the same way on both. Five types run as these Litmus experiments. `NETWORK_LATENCY` and `DNS_ERROR` run nowhere else. `MEMORY_STRESS`, `DISK_FILL` and `NODE_DRAIN` also run on `kubernetes`, which does them differently, as [The `kubernetes` Provider](#direct-kubernetes-api) says: | Type | Litmus Experiment | Effect | |------|-------------------|--------| @@ -147,12 +154,12 @@ The diagram shows the one decision the hybrid provider makes, and when it makes ```mermaid %%| label: fig-practice-hybrid-provider %%| fig-cap: "With `kates.chaos.provider=hybrid`, Kates looks for the LitmusChaos CRDs once, and sends every fault to the provider it picked." -%%| fig-alt: "Flowchart. A fault from the Kates API goes to the hybrid provider, which asks once whether the Litmus CRDs are installed. If Litmus is detected, it uses the litmus-crd provider, which runs Litmus experiments through the LitmusChaos CRDs and hands POD_DELETE, ROLLING_RESTART and SCALE_DOWN to the kubernetes provider. If Litmus is not found, it uses the kubernetes provider, which calls the Kubernetes API directly for the eight types it implements." +%%| fig-alt: "Flowchart. A fault from the Kates API goes to the hybrid provider, which asks once whether the Litmus CRDs are installed. If Litmus is detected, it uses the litmus-crd provider, which runs Litmus experiments through the LitmusChaos CRDs and hands POD_DELETE, ROLLING_RESTART and SCALE_DOWN to the kubernetes provider. If Litmus is not found, it uses the kubernetes provider, which calls the Kubernetes API directly for the eleven types it implements." graph TD DO[Kates API
a fault to inject] --> HCP[hybrid
once: are Litmus CRDs installed?] HCP -->|Litmus detected| LCP[litmus-crd
Litmus experiments] - HCP -->|Litmus not found| KCP[kubernetes
eight types, directly] + HCP -->|Litmus not found| KCP[kubernetes
eleven types, directly] LCP --> LIT[LitmusChaos CRDs] LCP -->|POD_DELETE,
ROLLING_RESTART,
SCALE_DOWN| KCP @@ -182,7 +189,7 @@ A pod-level fault hits one pod unless told otherwise. The first of these that ap Both chaos providers resolve the pods the same way: `kubernetes` applies the fault to each of them, and `litmus-crd` passes them to the experiment as a comma-separated `TARGET_PODS` list. A selector that matches no pod fails the step with `No pods found matching label selector` instead of doing nothing. -A `NODE_DRAIN` drains the node that runs the pod these rules pick. On `litmus-crd`, Kates passes that node to the `node-drain` experiment as `TARGET_NODE`, with no `TARGET_PODS`; the `kubernetes` provider doesn't run `NODE_DRAIN`. The experiment drains one node, so with `targetAll` every pod the selector matches has to run on the same node. On the [`panda`](appendix-a-glossary.md#gl-panda) [Kind](appendix-a-glossary.md#gl-kind) cluster, a zone's broker pods all run on the node named after the zone. The step fails without draining anything when the pods run on more than one node, when the pod is gone, or when it isn't on a node yet. +A `NODE_DRAIN` drains the node that runs the pod these rules pick. On `litmus-crd`, Kates passes that node to the `node-drain` experiment as `TARGET_NODE`, with no `TARGET_PODS`; the `kubernetes` provider drains that node itself. The experiment drains one node, so with `targetAll` every pod the selector matches has to run on the same node. On the [`panda`](appendix-a-glossary.md#gl-panda) [Kind](appendix-a-glossary.md#gl-kind) cluster, a zone's broker pods all run on the node named after the zone. The step fails without draining anything when the pods run on more than one node, when the pod is gone, or when it isn't on a node yet. To drain a node you choose, name it in `envOverrides.TARGET_NODE`, or give `envOverrides.NODE_LABEL` for Litmus to pick a node with that label. Kates then picks no node itself. The drain evicts every pod on the node, not only the one picked, in every namespace, so the other Kafka pods there go down with it. @@ -407,7 +414,7 @@ steps: ### storage-pressure -Fills broker disk to 90% with the LitmusChaos `disk-fill` experiment, to observe the cluster under storage pressure. The `kubernetes` chaos provider can't run `DISK_FILL`, so the playbook needs `litmus-crd` (or `hybrid`) and the `disk-fill` experiment the `kates-chaos` chart installs. +Fills broker disk to 90%, to observe the cluster under storage pressure. On `litmus-crd` it runs the `disk-fill` experiment the `kates-chaos` chart installs. On `kubernetes` it fills each target broker's data volume itself, and needs `rbac.directChaos`. ```yaml name: storage-pressure @@ -566,7 +573,7 @@ The count covers the whole plan, not one step at a time: Kates adds up the disti The label matches the KRaft controllers as well, so Kates counts as brokers only the pods Strimzi labels `strimzi.io/broker-role=true`: a node with both roles is a broker, and a dedicated controller is not. On the default cluster, with brokers 0–2 and controllers 3–5, a plan that takes down all three brokers is refused. A pod without the role label, such as Kafka not run by Strimzi, counts as a broker. -A step's affected brokers are the broker pods its fault hits, chosen by the rules in [Targeting Pods](#targeting-pods): a `targetAll` step counts every broker its selector matches, and a random pick counts as one. A `ROLLING_RESTART` step also counts as one, because the Cluster Operator takes its brokers down one at a time. A `SCALE_DOWN` step counts the broker each node pool it selects loses. A step in the Kates API's namespace counts none, but a pod named with `targetPod` counts as a broker wherever it runs, unless it's a dedicated KRaft controller. A `NODE_DRAIN` step counts every broker on the node it drains, whatever namespace its pod is in. For a random pick, that's the node among the candidates' that runs the most brokers, and the same goes for a node Litmus picks by `envOverrides.NODE_LABEL`. Both leave out the node the Kates API runs on, which no drain takes, and a step that could drain only that node is refused. The dry run lists every pod a step hits, controllers included, and warns when a step would hit none of the Kafka pods or when `targetBrokerId` names no broker. For a `NODE_DRAIN`, it lists every Kafka pod on the node and names the node. +A step's affected brokers are the broker pods its fault hits, chosen by the rules in [Targeting Pods](#targeting-pods): a `targetAll` step counts every broker its selector matches, and a random pick counts as one. A `ROLLING_RESTART` step also counts as one, because the Cluster Operator takes its brokers down one at a time. A `SCALE_DOWN` step counts the broker each node pool it selects loses. A step in the Kates API's namespace counts none, but a pod named with `targetPod` counts as a broker wherever it runs, unless it's a dedicated KRaft controller. A `NODE_DRAIN` step counts every broker on the node it drains, whatever namespace its pod is in. For a random pick, that's the node among the candidates' that runs the most brokers, and the same goes for a node picked among those `envOverrides.NODE_LABEL` matches. Both leave out the node the Kates API runs on, which no drain takes, and a step that could drain only that node is refused. The dry run lists every pod a step hits, controllers included, and warns when a step would hit none of the Kafka pods or when `targetBrokerId` names no broker. For a `NODE_DRAIN`, it lists every Kafka pod on the node and names the node. Kates runs the plan anyway, with a warning in `validationWarnings`, in three cases: @@ -610,7 +617,7 @@ Four rules go beyond the table. Each applies to some faults only, and the last c The other types may set a `chaosDurationSec` of `0`, because a deleted pod comes back by itself, and a `ROLLING_RESTART` or `SCALE_DOWN` with `0` doesn't wait. A `cmdProbe` is the type a probe without one gets. A resilience run's top-level `probes` may hold one, since Kates runs those in a broker pod, and Kates sends Litmus no probe at all. -The duration ceiling matters most on `kubernetes`, where a CPU or IO stress runs in an ephemeral container that Kubernetes can't remove: its duration is all that stops it. A `ROLLING_RESTART`'s `chaosDurationSec` is its wait for the Cluster Operator to roll every pod, so on a cluster whose roll takes over an hour, raise `kates.chaos.limits.max-duration-sec`. +The duration ceiling matters most on `kubernetes`, where a stress or a disk fill runs in an ephemeral container that Kubernetes can't remove: its duration is all that stops it. A `ROLLING_RESTART`'s `chaosDurationSec` is its wait for the Cluster Operator to roll every pod, so on a cluster whose roll takes over an hour, raise `kates.chaos.limits.max-duration-sec`. The same ceiling bounds the waits around a fault, each from 0: a plan step's `steadyStateSec` and `observationWindowSec`, a compound run's `timeoutSec` and `delayBetweenSec`, and a resilience run's `steadyStateSec`, `maxRecoveryWaitSec` and each top-level probe's `timeoutSec`. Kates holds the disruption lease through every one of them and nothing ends one early, so an `observationWindowSec` of `300000`, a typo for `300`, would keep every other disruption out for three and a half days. @@ -629,7 +636,7 @@ What rollback does depends on the fault. The table lists what it undoes, and wha |-------|--------------------|-------------------------------| | `NETWORK_PARTITION` | Deletes every NetworkPolicy labeled `managed-by=kates` in the step's namespace, the label Kates puts on its policies | Kates deletes its policies when `chaosDurationSec` ends, or as soon as the partition fails | | `SCALE_DOWN` | Gives every KafkaNodePool and StatefulSet in the step's namespace that carries `kates.io/original-replicas` its replicas back | The end of the run: when a plan, compound run or resilience run ends, Kates gives back every replica a `SCALE_DOWN` of that run took, before another disruption may start. Orphan recovery does it when the Kates API restarts mid-run | -| Every other type | Nothing, although the step report still says it rolled back | The StrimziPodSet recreates a deleted pod; a stress container on `kubernetes` stops after `chaosDurationSec`; Litmus gets `chaosDurationSec` as its duration | +| Every other type | Nothing, although the step report still says it rolled back | The StrimziPodSet recreates a deleted pod; a stress container on `kubernetes` stops after `chaosDurationSec`, a disk fill removes its file then, and a drain uncordons its node, or orphan recovery does when the Kates API restarted mid-drain; Litmus gets `chaosDurationSec` as its duration | So rollback matters for two faults. For a partition, it deletes the policies the `kubernetes` provider made. On `litmus-crd`, the default, Kates creates no NetworkPolicy itself, and rollback deletes a Litmus policy only if it carries the `managed-by=kates` label. A successful `SCALE_DOWN` always fails its recovery check, because the removed broker never comes back. With `requireRecovery` and `autoRollback` on, the step gives the broker back when the recovery timeout runs out. diff --git a/docs/book/appendix-d-versions.md b/docs/book/appendix-d-versions.md index 140842e4..328b356b 100644 --- a/docs/book/appendix-d-versions.md +++ b/docs/book/appendix-d-versions.md @@ -22,8 +22,8 @@ Every version the platform pins, in one place. This table is generated from the | `kafka-common` chart | 0.1.3 (library) | `charts/kafka-common/Chart.yaml` | | `kafka-ui` chart | 0.3.4 (app v1.5.0) | `charts/kafka-ui/Chart.yaml` | | `kates-chaos` chart | 2.6.0 (app 3.28.0) | `charts/kates-chaos/Chart.yaml` | -| `kates-platform` chart | 0.9.5 (app 1.0.0) | `charts/kates-platform/Chart.yaml` | -| `kates` chart | 0.19.5 (app 1.40.0) | `charts/kates/Chart.yaml` | +| `kates-platform` chart | 0.10.0 (app 1.0.0) | `charts/kates-platform/Chart.yaml` | +| `kates` chart | 0.20.0 (app 1.40.0) | `charts/kates/Chart.yaml` | | `legacy-kafka` chart | 0.2.2 (app 3.9.1) | `charts/legacy-kafka/Chart.yaml` | | `mirror-maker2` chart | 0.12.3 (app 4.3.1) | `charts/mirror-maker2/Chart.yaml` | | `monitoring` chart | 1.9.0 (app 82.4.3) | `charts/monitoring/Chart.yaml` | diff --git a/docs/book/appendix-index.md b/docs/book/appendix-index.md index bcbc3b6a..6aca047c 100644 --- a/docs/book/appendix-index.md +++ b/docs/book/appendix-index.md @@ -598,6 +598,6 @@ Every term the book teaches, with the sections that teach it: the ones that use ## Z [Zone]{#idx-zone} -: [The Cluster Under Test](03-cluster.md): [Physical Topology](03-cluster.md#physical-topology); [Chaos Engineering Theory](06-chaos-theory.md): [Core Principles](06-chaos-theory.md#core-principles); [Chaos Engineering in Practice](07-chaos-practice.md): [Built-In Playbooks](07-chaos-practice.md#built-in-playbooks); [Kafka Deployment Engineering](15-kafka-deployment.md): [KafkaNodePool Configuration](15-kafka-deployment.md#kafkanodepool-configuration); [Compliance Checks](compliance.md): [Placing Pods Across Zones and Nodes](compliance.md#placing-pods-across-zones-and-nodes), [Installing Kates for Compliance Checks](compliance.md#installing-for-compliance); [Operating Kafka Connect](operating-kafka-connect.md): [Spreading Workers Across Zones](operating-kafka-connect.md#multi-az-deployment-strategy); [Compliance Language Reference](appendix-compliance-language.md): [Brokers and Controllers](appendix-compliance-language.md#brokers-and-controllers) +: [The Cluster Under Test](03-cluster.md): [Physical Topology](03-cluster.md#physical-topology); [Chaos Engineering Theory](06-chaos-theory.md): [Core Principles](06-chaos-theory.md#core-principles); [Chaos Engineering in Practice](07-chaos-practice.md): [Disruption Types](07-chaos-practice.md#disruption-types), [Built-In Playbooks](07-chaos-practice.md#built-in-playbooks); [Kafka Deployment Engineering](15-kafka-deployment.md): [KafkaNodePool Configuration](15-kafka-deployment.md#kafkanodepool-configuration); [Compliance Checks](compliance.md): [Placing Pods Across Zones and Nodes](compliance.md#placing-pods-across-zones-and-nodes); [Operating Kafka Connect](operating-kafka-connect.md): [Spreading Workers Across Zones](operating-kafka-connect.md#multi-az-deployment-strategy); [Compliance Language Reference](appendix-compliance-language.md): [Brokers and Controllers](appendix-compliance-language.md#brokers-and-controllers) ::: diff --git a/kates/src/main/java/com/bmscomp/kates/chaos/DrainTargets.java b/kates/src/main/java/com/bmscomp/kates/chaos/DrainTargets.java new file mode 100644 index 00000000..ca0f3635 --- /dev/null +++ b/kates/src/main/java/com/bmscomp/kates/chaos/DrainTargets.java @@ -0,0 +1,162 @@ +package com.bmscomp.kates.chaos; + +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.TreeSet; +import java.util.concurrent.ThreadLocalRandom; +import java.util.function.Predicate; + +import io.fabric8.kubernetes.api.model.Node; +import io.fabric8.kubernetes.client.KubernetesClient; +import org.jboss.logging.Logger; + +/** + * The node a NODE_DRAIN drains, picked the same way on both chaos providers: + * the node of the pods {@link PodTargets} picks, or the one envOverrides + * names (TARGET_NODE) or matches (NODE_LABEL). Never the node the Kates API + * runs on ({@link KatesNode}). + */ +final class DrainTargets { + + private static final Logger LOG = Logger.getLogger(DrainTargets.class); + + private DrainTargets() {} + + /** + * The node the kubernetes provider drains. TARGET_NODE names it; NODE_LABEL + * has Kates pick one of the nodes it matches at random, as Litmus would; + * otherwise it is the node of the pods the step picks. + * + * @throws IllegalStateException when the drain can take no node but the + * Kates API's, NODE_LABEL matches no node, or {@link #nodeOfPods} fails + */ + static String node(KubernetesClient client, FaultSpec spec, Optional kates) { + Map overrides = spec.envOverrides() != null ? spec.envOverrides() : Map.of(); + String targetNode = overrides.get("TARGET_NODE"); + if (targetNode != null && !targetNode.isBlank()) { + if (kates.isPresent() && targetNode.equals(kates.get())) { + throw new IllegalStateException(KatesNode.refusal("envOverrides.TARGET_NODE names", targetNode)); + } + return targetNode; + } + String nodeLabel = overrides.get("NODE_LABEL"); + if (nodeLabel != null && !nodeLabel.isBlank()) { + List matching = client.nodes().list().getItems().stream() + .filter(drainedByOverride(spec)) + .map(n -> n.getMetadata().getName()) + .sorted() + .toList(); + List others = matching.stream() + .filter(n -> kates.isEmpty() || !n.equals(kates.get())) + .toList(); + if (others.isEmpty()) { + throw new IllegalStateException( + matching.isEmpty() + ? "NODE_DRAIN: envOverrides.NODE_LABEL '" + nodeLabel + "' matches no node" + : KatesNode.refusal( + "envOverrides.NODE_LABEL '" + nodeLabel + "' matches only", kates.get())); + } + String node = others.get(ThreadLocalRandom.current().nextInt(others.size())); + LOG.info("NODE_DRAIN: node " + node + ", one envOverrides.NODE_LABEL matches"); + return node; + } + return nodeOfPods(client, spec, kates); + } + + /** + * The node a drain drains: the one the pods it picks run on + * ({@link PodTargets}), so it hits the pod its targetPod, targetBrokerId + * or selector picks, as every other fault does. node-drain, given no + * TARGET_NODE, drains the node of a random pod in any namespace, the + * control plane's among them, and Kates used to give it none. It drains + * one node, so the pods a targetAll drain picks must all run on the same + * one. It never drains {@code kates}, the node the Kates API runs on: a + * random pick chooses among the pods on other nodes. + * + * @throws IllegalStateException when a pod is gone or not on a node yet, + * the pods run on more than one node, or their node is the Kates + * API's; the fault fails before anything is drained + */ + static String nodeOfPods(KubernetesClient client, FaultSpec spec, Optional kates) { + List pods = kates.isPresent() && PodTargets.mode(spec) == PodTargets.Mode.ONE_RANDOM + ? onePodAwayFrom(client, spec, kates.get()) + : PodTargets.resolve(client, spec); + Set nodes = new TreeSet<>(); + for (String name : pods) { + var pod = client.pods() + .inNamespace(spec.targetNamespace()) + .withName(name) + .get(); + if (pod == null) { + throw new IllegalStateException("NODE_DRAIN: pod " + name + " was not found in namespace '" + + spec.targetNamespace() + "', so there is no node to drain"); + } + String node = pod.getSpec() != null ? pod.getSpec().getNodeName() : null; + if (node == null || node.isBlank()) { + throw new IllegalStateException( + "NODE_DRAIN: pod " + name + " is not on a node yet, so there is no node to drain"); + } + nodes.add(node); + } + if (nodes.size() > 1) { + throw new IllegalStateException("NODE_DRAIN: the pods targetAll picks run on " + nodes.size() + " nodes (" + + String.join(", ", nodes) + "), and node-drain drains one. Narrow targetLabel to the pods of" + + " one node, or name the node in envOverrides.TARGET_NODE"); + } + String node = nodes.iterator().next(); + if (kates.isPresent() && node.equals(kates.get())) { + throw new IllegalStateException( + KatesNode.refusal(KatesNode.picked(pods), node) + ". Aim the drain at a pod on another node"); + } + LOG.info("NODE_DRAIN: TARGET_NODE " + node + ", the node of " + String.join(", ", pods)); + return node; + } + + /** + * The pod a random drain takes: one of those targetLabel matches that + * don't run on {@code kates}, the node the Kates API runs on. + * + * @throws IllegalStateException when every pod it matches runs there, or + * it matches none + */ + private static List onePodAwayFrom(KubernetesClient client, FaultSpec spec, String kates) { + ParsedLabelSelector selector = ParsedLabelSelector.parse(spec.targetLabel()); + var matching = client.pods() + .inNamespace(spec.targetNamespace()) + .withLabelSelector(selector.toString()) + .list() + .getItems(); + if (matching.isEmpty()) { + return PodTargets.resolve(client, spec); + } + var elsewhere = matching.stream() + .filter(p -> p.getSpec() == null || !kates.equals(p.getSpec().getNodeName())) + .toList(); + if (elsewhere.isEmpty()) { + throw new IllegalStateException(KatesNode.refusal("every pod targetLabel matches runs on", kates)); + } + return PodTargets.select(spec, elsewhere); + } + + /** + * The nodes an override may have node-drain drain: the one TARGET_NODE + * names, or else those NODE_LABEL matches. Null when both are empty, and + * node-drain then drains the node of a random pod in any namespace, which + * no node is safe from. + */ + static Predicate drainedByOverride(FaultSpec spec) { + String targetNode = spec.envOverrides().get("TARGET_NODE"); + if (targetNode != null && !targetNode.isBlank()) { + return n -> targetNode.equals(n.getMetadata().getName()); + } + String nodeLabel = spec.envOverrides().get("NODE_LABEL"); + if (nodeLabel != null && !nodeLabel.isBlank()) { + ParsedLabelSelector selector = ParsedLabelSelector.parse(nodeLabel); + return n -> selector.matches( + n.getMetadata().getLabels() != null ? n.getMetadata().getLabels() : Map.of()); + } + return null; + } +} diff --git a/kates/src/main/java/com/bmscomp/kates/chaos/KubernetesChaosProvider.java b/kates/src/main/java/com/bmscomp/kates/chaos/KubernetesChaosProvider.java index 27a9719d..25536749 100644 --- a/kates/src/main/java/com/bmscomp/kates/chaos/KubernetesChaosProvider.java +++ b/kates/src/main/java/com/bmscomp/kates/chaos/KubernetesChaosProvider.java @@ -3,8 +3,10 @@ import java.time.Instant; import java.util.LinkedHashMap; import java.util.LinkedHashSet; +import java.util.List; import java.util.Map; import java.util.Objects; +import java.util.Optional; import java.util.Set; import java.util.concurrent.CompletableFuture; import java.util.function.LongConsumer; @@ -14,9 +16,12 @@ import io.fabric8.kubernetes.api.model.EphemeralContainer; import io.fabric8.kubernetes.api.model.EphemeralContainerBuilder; +import io.fabric8.kubernetes.api.model.Node; +import io.fabric8.kubernetes.api.model.NodeBuilder; import io.fabric8.kubernetes.api.model.OwnerReference; import io.fabric8.kubernetes.api.model.Pod; import io.fabric8.kubernetes.api.model.PodBuilder; +import io.fabric8.kubernetes.api.model.VolumeMount; import io.fabric8.kubernetes.api.model.apps.StatefulSet; import io.fabric8.kubernetes.api.model.apps.StatefulSetBuilder; import io.fabric8.kubernetes.api.model.networking.v1.*; @@ -144,6 +149,9 @@ private void applyDisruption(FaultSpec spec, String engineName, Map executePodKill(spec); case CPU_STRESS -> executeCpuStress(spec, engineName); case IO_STRESS -> executeIoStress(spec, engineName); + case MEMORY_STRESS -> executeMemoryStress(spec, engineName); + case DISK_FILL -> executeDiskFill(spec, engineName); + case NODE_DRAIN -> executeNodeDrain(spec, marks); default -> throw new UnsupportedOperationException( "DisruptionType " + spec.disruptionType() + " not supported by kubernetes provider"); @@ -568,6 +576,344 @@ private void executeIoStress(FaultSpec spec, String engineName) throws Interrupt Thread.sleep(Math.max(0, spec.chaosDurationSec()) * 1000L); } + /** + * As {@link #executeCpuStress}, with one worker that holds {@code memoryMb}. + * The container is the pod's, not the broker container's, so it presses on + * the pod's and the node's memory, not the broker container's own limit. + */ + private void executeMemoryStress(FaultSpec spec, String engineName) throws InterruptedException { + for (String podName : PodTargets.resolve(client, spec)) { + LOG.info("MEMORY_STRESS: injecting stress ephemeral container into " + podName); + injectEphemeralContainer( + spec.targetNamespace(), + podName, + stressContainerName("chaos-memory-stress", engineName), + stressImage, + "stress", + "--vm", + "1", + "--vm-bytes", + spec.memoryMb() + "M", + "--vm-keep", + "--timeout", + spec.chaosDurationSec() + "s"); + } + Thread.sleep(Math.max(0, spec.chaosDurationSec()) * 1000L); + } + + /** Where the DISK_FILL container mounts the broker's data volume. */ + static final String FILL_MOUNT = "/kates-fill"; + + /** + * Fills each target pod's data volume to {@code fillPercentage} for + * {@code chaosDurationSec}, from an ephemeral container that mounts it and + * writes one file, as the broker's own user, into the first directory there + * it may write: the volume's root, or the broker's log directory, where + * Kafka ignores a plain file. The file goes when the container exits, + * at the end of the duration or when the pod stops; a pod killed outright + * leaves it, named kates-disk-fill-*, on the volume. + */ + private void executeDiskFill(FaultSpec spec, String engineName) throws InterruptedException { + String id = engineName.substring(engineName.lastIndexOf('-') + 1); + String containerName = "chaos-disk-fill-" + id; + List pods = PodTargets.resolve(client, spec); + for (String podName : pods) { + Pod pod = client.pods() + .inNamespace(spec.targetNamespace()) + .withName(podName) + .get(); + if (pod == null) { + throw new IllegalStateException("Pod not found: " + podName); + } + String volume = dataVolume(pod); + LOG.info("DISK_FILL: filling volume " + volume + " of " + podName + " to " + spec.fillPercentage() + "%"); + injectEphemeralContainer( + spec.targetNamespace(), + podName, + containerName, + stressImage, + new io.fabric8.kubernetes.api.model.VolumeMountBuilder() + .withName(volume) + .withMountPath(FILL_MOUNT) + .build(), + "sh", + "-c", + fillScript(spec.fillPercentage(), spec.chaosDurationSec(), id)); + } + Thread.sleep(Math.max(0, spec.chaosDurationSec()) * 1000L); + for (String podName : pods) { + Integer exit = ephemeralExitCode(spec.targetNamespace(), podName, containerName); + if (exit != null && exit != 0) { + throw new IllegalStateException("DISK_FILL: the fill container in " + podName + " exited " + exit + + ", so the volume was not filled; kubectl logs " + podName + " -c " + containerName + + " says why"); + } + } + } + + /** + * The shell the DISK_FILL container runs (busybox sh): it writes up to + * {@code percent} of the volume, holds it until {@code durationSec} has + * passed since it started, and removes the file on exit or SIGTERM. dd and + * sleep run in the background, waited on, because sh runs a trap only once + * a foreground command ends: a SIGTERM during a long dd otherwise came to + * nothing, the kubelet killed the container, and the file stayed. + */ + static String fillScript(int percent, int durationSec, String id) { + return "end=$(($(date +%s)+" + durationSec + ")); d=; for c in " + FILL_MOUNT + " " + FILL_MOUNT + "/*/; do" + + " if [ -d \"$c\" ] && [ -w \"$c\" ]; then d=$c; break; fi; done;" + + " [ -n \"$d\" ] || { echo 'no directory on the volume this user may write' >&2; exit 1; };" + + " f=${d%/}/kates-disk-fill-" + id + "; p=;" + + " trap '[ -n \"$p\" ] && kill $p 2>/dev/null; rm -f \"$f\"' EXIT; trap 'exit 0' TERM INT;" + + " set -- $(df -Pk " + FILL_MOUNT + " | tail -n 1); want=$((($2*" + percent + "/100-$3)/1024));" + + " if [ \"$want\" -gt 0 ]; then dd if=/dev/zero of=\"$f\" bs=1048576 count=\"$want\" 2>/dev/null &" + + " p=$!; wait $p; p=; fi; left=$((end-$(date +%s)));" + + " if [ \"$left\" -gt 0 ]; then sleep \"$left\" & p=$!; wait $p; p=; fi; exit 0"; + } + + /** + * The volume DISK_FILL fills: the first one the pod's first container + * mounts that a PersistentVolumeClaim backs, else the first it mounts + * named data*, as Strimzi names a broker's (data-0 for JBOD and + * ephemeral storage). + * + * @throws IllegalStateException when the pod has neither + */ + static String dataVolume(Pod pod) { + var mounted = pod.getSpec().getContainers().getFirst().getVolumeMounts().stream() + .map(VolumeMount::getName) + .toList(); + var volumes = pod.getSpec().getVolumes().stream() + .filter(v -> mounted.contains(v.getName())) + .toList(); + return volumes.stream() + .filter(v -> v.getPersistentVolumeClaim() != null) + .findFirst() + .or(() -> volumes.stream() + .filter(v -> v.getName().startsWith("data")) + .findFirst()) + .map(io.fabric8.kubernetes.api.model.Volume::getName) + .orElseThrow(() -> new IllegalStateException( + "DISK_FILL: pod " + pod.getMetadata().getName() + + " mounts no PersistentVolumeClaim and no volume named data*, so there is no data volume" + + " to fill")); + } + + /** The exit code of a terminated ephemeral container, or null while it runs or when it can't be read. */ + private Integer ephemeralExitCode(String namespace, String podName, String containerName) { + try { + Pod pod = client.pods().inNamespace(namespace).withName(podName).get(); + if (pod == null || pod.getStatus() == null || pod.getStatus().getEphemeralContainerStatuses() == null) { + return null; + } + return pod.getStatus().getEphemeralContainerStatuses().stream() + .filter(s -> containerName.equals(s.getName())) + .map(s -> s.getState() != null && s.getState().getTerminated() != null + ? s.getState().getTerminated().getExitCode() + : null) + .filter(Objects::nonNull) + .findFirst() + .orElse(null); + } catch (KubernetesClientException e) { + LOG.debug("DISK_FILL: could not read " + podName + "'s containers", e); + return null; + } + } + + /** Label on a node a NODE_DRAIN cordoned, which orphan recovery looks for. */ + public static final String CORDONED_LABEL = "kates.io/cordoned"; + + /** Epoch millis of the cordon, which orphan recovery compares with its start. */ + public static final String CORDONED_AT_ANNOTATION = "kates.io/cordoned-at"; + + /** How often NODE_DRAIN tries again to evict the pods a PodDisruptionBudget held back. */ + long drainPollIntervalMs = 5_000; + + /** The node the Kates API runs on, which a drain never takes; null in tests. */ + @Inject + KatesNode katesNode; + + /** + * Drains a node as kubectl drain does, for {@code chaosDurationSec}: cordons + * it, evicts its pods through the Eviction API, which honours their + * PodDisruptionBudgets, and tries again every few seconds for those a + * budget holds back; then uncordons it, whatever happened, as the partition + * removes its policy. DaemonSet and static pods stay. The node is picked + * as on litmus-crd ({@link DrainTargets}), never the Kates API's. A node + * already cordoned is refused: uncordoning it at the end would undo + * someone else's cordon. + * + * @throws IllegalStateException when a pod was still not evicted when the + * duration ended; the node is uncordoned all the same + */ + private void executeNodeDrain(FaultSpec spec, Map marks) throws InterruptedException { + Optional kates = katesNode != null ? katesNode.name() : Optional.empty(); + String node = DrainTargets.node(client, spec, kates); + Node current = client.nodes().withName(node).get(); + if (current == null) { + throw new IllegalStateException("NODE_DRAIN: node " + node + " was not found"); + } + if (current.getSpec() != null && Boolean.TRUE.equals(current.getSpec().getUnschedulable())) { + throw new IllegalStateException("NODE_DRAIN: node " + node + " is already cordoned, and Kates leaves a" + + " cordon it did not make alone"); + } + long deadline = System.nanoTime() + Math.max(0, spec.chaosDurationSec()) * 1_000_000_000L; + LOG.info("NODE_DRAIN: cordoning " + node); + client.nodes() + .withName(node) + .edit(n -> new NodeBuilder(n) + .editMetadata() + .addToLabels(CORDONED_LABEL, "true") + .addToAnnotations(marks) + .addToAnnotations(CORDONED_AT_ANNOTATION, String.valueOf(System.currentTimeMillis())) + .endMetadata() + .editOrNewSpec() + .withUnschedulable(true) + .endSpec() + .build()); + Exception failure = null; + try { + Set pending = evictAll(node, deadline); + long remainingMs = (deadline - System.nanoTime()) / 1_000_000L; + if (remainingMs > 0) { + Thread.sleep(remainingMs); + } + if (!pending.isEmpty()) { + throw new IllegalStateException("NODE_DRAIN: " + pending.size() + " pod(s) on " + node + + " were not evicted within chaosDurationSec=" + spec.chaosDurationSec() + "s: " + pending + + ". A PodDisruptionBudget refused their eviction"); + } + } catch (RuntimeException | InterruptedException e) { + failure = e; + throw e; + } finally { + removeCordon(node, failure); + } + } + + /** + * Evicts every pod on {@code node} a drain evicts, trying again those a + * PodDisruptionBudget holds back until {@code deadline}; the namespace/name + * of those still not evicted then. + */ + private Set evictAll(String node, long deadline) throws InterruptedException { + Map pending = new LinkedHashMap<>(); + // The field selector spares the API server; the filter is what decides. + for (Pod pod : client.pods() + .inAnyNamespace() + .withField("spec.nodeName", node) + .list() + .getItems()) { + if (node.equals(pod.getSpec().getNodeName()) && isEvictable(pod)) { + pending.put( + pod.getMetadata().getNamespace() + "/" + + pod.getMetadata().getName(), + pod); + } + } + LOG.info("NODE_DRAIN: evicting " + pending.keySet() + " from " + node); + while (true) { + pending.values().removeIf(this::evict); + long remainingMs = (deadline - System.nanoTime()) / 1_000_000L; + if (pending.isEmpty() || remainingMs <= 0) { + return pending.keySet(); + } + Thread.sleep(Math.min(drainPollIntervalMs, remainingMs)); + } + } + + /** As kubectl drain: not a DaemonSet's pod, a static pod, one already going, or one that has ended. */ + static boolean isEvictable(Pod pod) { + var meta = pod.getMetadata(); + if (meta.getDeletionTimestamp() != null + || (meta.getAnnotations() != null + && meta.getAnnotations().containsKey("kubernetes.io/config.mirror"))) { + return false; + } + if (meta.getOwnerReferences() != null + && meta.getOwnerReferences().stream().anyMatch(o -> "DaemonSet".equals(o.getKind()))) { + return false; + } + String phase = pod.getStatus() != null ? pod.getStatus().getPhase() : null; + return !"Succeeded".equals(phase) && !"Failed".equals(phase); + } + + /** Whether the pod is evicted or gone; false while a PodDisruptionBudget holds it back. */ + private boolean evict(Pod pod) { + String namespace = pod.getMetadata().getNamespace(); + String name = pod.getMetadata().getName(); + try { + boolean evicted = + client.pods().inNamespace(namespace).withName(name).evict(); + if (evicted) { + LOG.info("NODE_DRAIN: evicted " + namespace + "/" + name); + } + return evicted; + } catch (KubernetesClientException e) { + if (e.getCode() == 404) { + return true; + } + throw e; + } + } + + /** + * Uncordons the node, trying three times: a node left cordoned takes no new + * pod until someone uncordons it. When that fails, the fault fails saying + * how, with its own failure, if any, suppressed; orphan recovery uncordons + * it once its deadline has passed. + */ + private void removeCordon(String node, Exception failure) { + KubernetesClientException last = null; + for (int attempt = 0; attempt < 3; attempt++) { + try { + uncordon(client, node); + LOG.info("NODE_DRAIN: uncordoned " + node); + return; + } catch (KubernetesClientException e) { + last = e; + if (attempt == 2) { + break; + } + try { + Thread.sleep(2_000); + } catch (InterruptedException ie) { + Thread.currentThread().interrupt(); + break; + } + } + } + IllegalStateException notRemoved = new IllegalStateException( + "NODE_DRAIN: could not uncordon node " + node + + ", so it takes no new pod. Uncordon it with kubectl uncordon " + node + ". " + + (last != null ? last.getMessage() : "interrupted"), + last); + if (failure != null) { + notRemoved.addSuppressed(failure); + } + if (failure instanceof InterruptedException) { + Thread.currentThread().interrupt(); + } + throw notRemoved; + } + + /** Uncordons a node a NODE_DRAIN cordoned and takes Kates's marks off it; also orphan recovery's undo. */ + public static void uncordon(KubernetesClient client, String node) { + client.nodes() + .withName(node) + .edit(n -> new NodeBuilder(n) + .editMetadata() + .removeFromLabels(CORDONED_LABEL) + .removeFromAnnotations(CORDONED_AT_ANNOTATION) + .removeFromAnnotations(FaultMarks.TERM) + .removeFromAnnotations(FaultMarks.DEADLINE) + .endMetadata() + .editOrNewSpec() + .withUnschedulable(null) + .endSpec() + .build()); + } + /** * An ephemeral container is never removed, and a pod refuses a second one * of the same name, so each fault names its own: the prefix and the engine @@ -592,6 +938,16 @@ static String stressContainerName(String prefix, String engineName) { private void injectEphemeralContainer( String namespace, String podName, String containerName, String image, String... command) { + injectEphemeralContainer(namespace, podName, containerName, image, null, command); + } + + private void injectEphemeralContainer( + String namespace, + String podName, + String containerName, + String image, + VolumeMount mount, + String... command) { var podResource = client.pods().inNamespace(namespace).withName(podName); var pod = podResource.get(); if (pod == null) { @@ -624,6 +980,7 @@ private void injectEphemeralContainer( .withName(containerName) .withImage(image) .withCommand(command) + .withVolumeMounts(mount == null ? List.of() : List.of(mount)) .withNewSecurityContext() .withRunAsNonRoot(true) .withRunAsUser(uid == null ? 65534L : uid) diff --git a/kates/src/main/java/com/bmscomp/kates/chaos/LitmusChaosProvider.java b/kates/src/main/java/com/bmscomp/kates/chaos/LitmusChaosProvider.java index fe728120..8155acfc 100644 --- a/kates/src/main/java/com/bmscomp/kates/chaos/LitmusChaosProvider.java +++ b/kates/src/main/java/com/bmscomp/kates/chaos/LitmusChaosProvider.java @@ -357,13 +357,15 @@ ChaosEngine buildChaosEngine(FaultSpec spec, String engineName, String experimen // node the drain doesn't take. if (EXPERIMENT_MAP.get(DisruptionType.NODE_DRAIN).equals(experimentName)) { Optional kates = katesNode != null ? katesNode.name() : Optional.empty(); - String node = overridesTheNode(spec) ? nodeInsteadOfOverride(spec, kates) : targetNode(spec, kates); + String node = overridesTheNode(spec) + ? nodeInsteadOfOverride(spec, kates) + : DrainTargets.nodeOfPods(client, spec, kates); if (node != null) { envVars.removeIf(e -> "TARGET_NODE".equals(e.name)); envVars.add(new ChaosEngineSpec.EnvVar("TARGET_NODE", node)); } java.util.function.Predicate drained = - node != null ? n -> node.equals(n.getMetadata().getName()) : drainedByOverride(spec); + node != null ? n -> node.equals(n.getMetadata().getName()) : DrainTargets.drainedByOverride(spec); if (drained != null) { Map away = litmusPodsAwayFrom(drained); components.nodeSelector = away; @@ -399,82 +401,6 @@ private static boolean overridesTheNode(FaultSpec spec) { || spec.envOverrides().containsKey("NODE_LABEL")); } - /** - * The node a drain drains: the one the pods it picks run on - * ({@link PodTargets}), so it hits the pod its targetPod, targetBrokerId - * or selector picks, as every other fault does. node-drain, given no - * TARGET_NODE, drains the node of a random pod in any namespace, the - * control plane's among them, and Kates used to give it none. It drains - * one node, so the pods a targetAll drain picks must all run on the same - * one. It never drains {@code kates}, the node the Kates API runs on: a - * random pick chooses among the pods on other nodes. - * - * @throws IllegalStateException when a pod is gone or not on a node yet, - * the pods run on more than one node, or their node is the Kates - * API's; the fault fails before the ChaosEngine exists, so nothing is - * drained - */ - private String targetNode(FaultSpec spec, Optional kates) { - List pods = kates.isPresent() && PodTargets.mode(spec) == PodTargets.Mode.ONE_RANDOM - ? onePodAwayFrom(spec, kates.get()) - : PodTargets.resolve(client, spec); - Set nodes = new TreeSet<>(); - for (String name : pods) { - var pod = client.pods() - .inNamespace(spec.targetNamespace()) - .withName(name) - .get(); - if (pod == null) { - throw new IllegalStateException("NODE_DRAIN: pod " + name + " was not found in namespace '" - + spec.targetNamespace() + "', so there is no node to drain"); - } - String node = pod.getSpec() != null ? pod.getSpec().getNodeName() : null; - if (node == null || node.isBlank()) { - throw new IllegalStateException( - "NODE_DRAIN: pod " + name + " is not on a node yet, so there is no node to drain"); - } - nodes.add(node); - } - if (nodes.size() > 1) { - throw new IllegalStateException("NODE_DRAIN: the pods targetAll picks run on " + nodes.size() + " nodes (" - + String.join(", ", nodes) + "), and node-drain drains one. Narrow targetLabel to the pods of" - + " one node, or name the node in envOverrides.TARGET_NODE"); - } - String node = nodes.iterator().next(); - if (kates.isPresent() && node.equals(kates.get())) { - throw new IllegalStateException( - KatesNode.refusal(KatesNode.picked(pods), node) + ". Aim the drain at a pod on another node"); - } - LOG.info("NODE_DRAIN: TARGET_NODE " + node + ", the node of " + String.join(", ", pods)); - return node; - } - - /** - * The pod a random drain takes: one of those targetLabel matches that - * don't run on {@code kates}, the node the Kates API runs on. - * - * @throws IllegalStateException when every pod it matches runs there, or - * it matches none - */ - private List onePodAwayFrom(FaultSpec spec, String kates) { - ParsedLabelSelector selector = ParsedLabelSelector.parse(spec.targetLabel()); - var matching = client.pods() - .inNamespace(spec.targetNamespace()) - .withLabelSelector(selector.toString()) - .list() - .getItems(); - if (matching.isEmpty()) { - return PodTargets.resolve(client, spec); - } - var elsewhere = matching.stream() - .filter(p -> p.getSpec() == null || !kates.equals(p.getSpec().getNodeName())) - .toList(); - if (elsewhere.isEmpty()) { - throw new IllegalStateException(KatesNode.refusal("every pod targetLabel matches runs on", kates)); - } - return PodTargets.select(spec, elsewhere); - } - /** * The node Kates names in place of an override that could drain the node * the Kates API runs on, or null when the override is left to pick. A @@ -500,7 +426,7 @@ private String nodeInsteadOfOverride(FaultSpec spec, Optional kates) { if (nodeLabel == null || nodeLabel.isBlank()) { return null; } - var matched = drainedByOverride(spec); + var matched = DrainTargets.drainedByOverride(spec); List matching = client.nodes().list().getItems().stream() .filter(matched) .map(n -> n.getMetadata().getName()) @@ -521,27 +447,6 @@ private String nodeInsteadOfOverride(FaultSpec spec, Optional kates) { return node; } - /** - * The nodes an override may have node-drain drain: the one TARGET_NODE - * names, or else those NODE_LABEL matches. Null when both are empty, and - * node-drain then drains the node of a random pod in any namespace, which - * no node is safe from. - */ - private static java.util.function.Predicate drainedByOverride( - FaultSpec spec) { - String targetNode = spec.envOverrides().get("TARGET_NODE"); - if (targetNode != null && !targetNode.isBlank()) { - return n -> targetNode.equals(n.getMetadata().getName()); - } - String nodeLabel = spec.envOverrides().get("NODE_LABEL"); - if (nodeLabel != null && !nodeLabel.isBlank()) { - ParsedLabelSelector selector = ParsedLabelSelector.parse(nodeLabel); - return n -> selector.matches( - n.getMetadata().getLabels() != null ? n.getMetadata().getLabels() : Map.of()); - } - return null; - } - /** The node label Litmus's pods are pinned by, which the kubelet sets on every node. */ private static final String HOSTNAME_LABEL = "kubernetes.io/hostname"; diff --git a/kates/src/main/java/com/bmscomp/kates/disruption/DisruptionOrphanReconciler.java b/kates/src/main/java/com/bmscomp/kates/disruption/DisruptionOrphanReconciler.java index dff03e58..bb9ff1b0 100644 --- a/kates/src/main/java/com/bmscomp/kates/disruption/DisruptionOrphanReconciler.java +++ b/kates/src/main/java/com/bmscomp/kates/disruption/DisruptionOrphanReconciler.java @@ -142,6 +142,7 @@ private synchronized void reconcile(Instant startedAt) { int policies = reconcileNetworkPolicies(startedAt); int engines = reconcileChaosEngines(startedAt); ScaleDownSnapshots.Restored scaled = reconcileScaleDowns(startedAt); + int cordons = reconcileCordons(startedAt); if (orchestrator != null) { try { // After the faults: a database that hangs must not hold a partition (J4 review). @@ -153,11 +154,12 @@ private synchronized void reconcile(Instant startedAt) { } // A report says INTERRUPTED only once its faults are out (s08b reported it with one in). int reports = undoFailed ? 0 : markInterruptedReports(startedAt != null ? startedAt : since); - if (policies + engines + scaled.total() + reports > 0) { + if (policies + engines + scaled.total() + cordons + reports > 0) { LOG.warnf( "Orphan recovery: removed %d NetworkPolicy(ies) and %d ChaosEngine(s), restored %d" - + " KafkaNodePool(s) and %d StatefulSet(s), marked %d report(s) INTERRUPTED", - policies, engines, scaled.nodePools(), scaled.statefulSets(), reports); + + " KafkaNodePool(s) and %d StatefulSet(s), uncordoned %d node(s), marked %d" + + " report(s) INTERRUPTED", + policies, engines, scaled.nodePools(), scaled.statefulSets(), cordons, reports); } } catch (Exception e) { // Never block startup on cleanup — the API must come up either way. @@ -198,7 +200,8 @@ boolean isStray(ObjectMeta meta, Long ageCutoffMs, Instant now) { /** * Deletes the Kates ChaosEngines no live fault owns, in every namespace, as Litmus cleanup does - * in-process. A node-drain's cordon is Litmus' to revert: Kates may not patch nodes. + * in-process. Litmus node-drain's cordon is Litmus' to revert; {@link #reconcileCordons} undoes the + * kubernetes provider's. */ int reconcileChaosEngines(Instant startedAt) { Long cutoff = startedAt == null ? null : cutoff(startedAt); @@ -308,6 +311,56 @@ ScaleDownSnapshots.Restored reconcileScaleDowns(Instant startedAt) { meta -> isStray(meta, null, now) || (cutoff != null && isScaleDownOlderThan(meta, cutoff))); } + /** + * Uncordons the nodes a kubernetes-provider NODE_DRAIN cordoned and no + * live fault owns: past their deadline, of an older term, or, at start or + * takeover, cordoned before it. Judged by the cordon's own time, not the + * node's age. + */ + int reconcileCordons(Instant startedAt) { + Long cutoff = startedAt == null ? null : cutoff(startedAt); + Instant now = Instant.now(); + List nodes; + try { + nodes = kubeClient + .nodes() + .withLabel(KubernetesChaosProvider.CORDONED_LABEL) + .list() + .getItems(); + } catch (Exception e) { + LOG.debugf("Could not list the nodes Kates cordoned: %s", e.getMessage()); + undoFailed = true; + return 0; + } + int uncordoned = 0; + for (var node : nodes) { + ObjectMeta meta = node.getMetadata(); + if (!isStray(meta, null, now) && !(cutoff != null && isCordonedBefore(meta, cutoff))) { + continue; + } + try { + KubernetesChaosProvider.uncordon(kubeClient, meta.getName()); + LOG.warnf("Orphan recovery: uncordoned node %s, which a NODE_DRAIN left cordoned", meta.getName()); + uncordoned++; + } catch (Exception e) { + undoFailed = true; + LOG.warnf("Could not uncordon node %s: %s", meta.getName(), e.getMessage()); + } + } + return uncordoned; + } + + private static boolean isCordonedBefore(ObjectMeta meta, long cutoff) { + String stamp = meta.getAnnotations() != null + ? meta.getAnnotations().get(KubernetesChaosProvider.CORDONED_AT_ANNOTATION) + : null; + try { + return stamp != null && Long.parseLong(stamp) < cutoff; + } catch (NumberFormatException e) { + return false; + } + } + private long cutoff(Instant startedAt) { // creationTimestamp has whole seconds: a policy made in this process's // first second can be stamped before startedAt itself. diff --git a/kates/src/main/java/com/bmscomp/kates/disruption/DisruptionSafetyGuard.java b/kates/src/main/java/com/bmscomp/kates/disruption/DisruptionSafetyGuard.java index ef97b5c7..e4872578 100644 --- a/kates/src/main/java/com/bmscomp/kates/disruption/DisruptionSafetyGuard.java +++ b/kates/src/main/java/com/bmscomp/kates/disruption/DisruptionSafetyGuard.java @@ -638,7 +638,7 @@ Drain drain(FaultSpec spec, List kafkaPods) { drainable.size() < nodes.size() ? "NODE_DRAIN drains a node Kates picks among those envOverrides.NODE_LABEL '" + nodeLabel + "' matches, never " + kates.get() + ", which the Kates API runs on" - : "NODE_DRAIN drains a node Litmus picks by envOverrides.NODE_LABEL '" + nodeLabel + "'"); + : "NODE_DRAIN drains a node envOverrides.NODE_LABEL '" + nodeLabel + "' matches"); } if (overrides.containsKey(TARGET_NODE) || overrides.containsKey(NODE_LABEL)) { return worstOf( diff --git a/kates/src/main/java/com/bmscomp/kates/resilience/ResilienceResource.java b/kates/src/main/java/com/bmscomp/kates/resilience/ResilienceResource.java index 033d9018..393ac350 100644 --- a/kates/src/main/java/com/bmscomp/kates/resilience/ResilienceResource.java +++ b/kates/src/main/java/com/bmscomp/kates/resilience/ResilienceResource.java @@ -362,9 +362,10 @@ public Response runScenario( } // A scenario can't name the node to drain, and Kates sets no // TARGET_NODE itself: Litmus node-drain, given none, drains the node - // of a random pod in any namespace, and the kubernetes provider has no - // NODE_DRAIN. The scenario stays listed, and is refused until Kates - // can aim a drain at a broker's node. + // of a random pod in any namespace, and on the kubernetes provider an + // untargeted drain takes a random pod's node. The scenario stays + // listed, and is refused until Kates can aim a drain at a broker's + // node. if (scenario.disruptionType() == com.bmscomp.kates.chaos.DisruptionType.NODE_DRAIN) { return Response.status(400) .entity(ApiError.of( diff --git a/kates/src/test/java/com/bmscomp/kates/chaos/KubernetesChaosProviderTest.java b/kates/src/test/java/com/bmscomp/kates/chaos/KubernetesChaosProviderTest.java index 5e3a99bb..a7b93a2f 100644 --- a/kates/src/test/java/com/bmscomp/kates/chaos/KubernetesChaosProviderTest.java +++ b/kates/src/test/java/com/bmscomp/kates/chaos/KubernetesChaosProviderTest.java @@ -1123,4 +1123,445 @@ void aFaultIsMarkedWithTheTermItWasConfirmedIn() throws Exception { assertMarked(policy.getMetadata().getAnnotations(), "7"); } + + // ---- MEMORY_STRESS, DISK_FILL and NODE_DRAIN on the Kubernetes API ---- + + @Test + void memoryStressHoldsMemoryMbInAStressContainer() throws Exception { + Pod pod = stressTarget("broker-6", "kafka7"); + String subresource = "/api/v1/namespaces/default/pods/broker-6/ephemeralcontainers"; + server.expect().put().withPath(subresource).andReturn(200, pod).once(); + FaultSpec spec = FaultSpec.builder("test-memory-stress") + .targetNamespace("default") + .targetLabel("app=kafka7") + .disruptionType(DisruptionType.MEMORY_STRESS) + .memoryMb(256) + .chaosDurationSec(1) + .build(); + + assertTrue(provider.triggerFault(spec).get(5, TimeUnit.SECONDS).isPass()); + + Pod sent = client.getKubernetesSerialization() + .unmarshal(server.getLastRequest().getUtf8Body(), Pod.class); + EphemeralContainer stress = sent.getSpec().getEphemeralContainers().get(0); + assertTrue(stress.getName().startsWith("chaos-memory-stress-"), stress.getName()); + assertEquals( + List.of("stress", "--vm", "1", "--vm-bytes", "256M", "--vm-keep", "--timeout", "1s"), + stress.getCommand()); + assertRestricted(stress, 65534L, 65534L); + } + + private Pod broker(String name, String app, io.fabric8.kubernetes.api.model.Volume... volumes) { + var container = new io.fabric8.kubernetes.api.model.ContainerBuilder() + .withName("kafka") + .withImage("kafka"); + for (var v : volumes) { + container + .addNewVolumeMount() + .withName(v.getName()) + .withMountPath("/var/lib/kafka/" + v.getName()) + .endVolumeMount(); + } + Pod pod = new PodBuilder() + .withNewMetadata() + .withName(name) + .withNamespace("default") + .addToLabels("app", app) + .endMetadata() + .withNewSpec() + .withNewSecurityContext() + .withRunAsUser(1001L) + .withRunAsGroup(1001L) + .endSecurityContext() + .withContainers(container.build()) + .withVolumes(volumes) + .endSpec() + .build(); + client.pods().inNamespace("default").resource(pod).create(); + return pod; + } + + private static io.fabric8.kubernetes.api.model.Volume pvc(String name) { + return new io.fabric8.kubernetes.api.model.VolumeBuilder() + .withName(name) + .withNewPersistentVolumeClaim() + .withClaimName(name + "-claim") + .endPersistentVolumeClaim() + .build(); + } + + private static io.fabric8.kubernetes.api.model.Volume emptyDir(String name) { + return new io.fabric8.kubernetes.api.model.VolumeBuilder() + .withName(name) + .withNewEmptyDir() + .endEmptyDir() + .build(); + } + + @Test + void diskFillMountsTheBrokersDataVolumeAndFillsItAsTheBroker() throws Exception { + Pod pod = broker("broker-7", "kafka8", emptyDir("strimzi-tmp"), pvc("data-0")); + String subresource = "/api/v1/namespaces/default/pods/broker-7/ephemeralcontainers"; + server.expect().put().withPath(subresource).andReturn(200, pod).once(); + FaultSpec spec = FaultSpec.builder("test-disk-fill") + .targetNamespace("default") + .targetLabel("app=kafka8") + .disruptionType(DisruptionType.DISK_FILL) + .fillPercentage(90) + .chaosDurationSec(1) + .build(); + + ChaosOutcome outcome = provider.triggerFault(spec).get(5, TimeUnit.SECONDS); + + assertTrue(outcome.isPass(), outcome.failureReason()); + RecordedRequest write = server.takeRequest(0, TimeUnit.SECONDS); + while (write != null && !subresource.equals(write.getPath())) { + write = server.takeRequest(0, TimeUnit.SECONDS); + } + assertNotNull(write, "the fill container was added"); + Pod sent = client.getKubernetesSerialization().unmarshal(write.getUtf8Body(), Pod.class); + EphemeralContainer fill = sent.getSpec().getEphemeralContainers().get(0); + assertTrue(fill.getName().startsWith("chaos-disk-fill-"), fill.getName()); + assertEquals(1, fill.getVolumeMounts().size()); + assertEquals("data-0", fill.getVolumeMounts().get(0).getName()); + assertEquals( + KubernetesChaosProvider.FILL_MOUNT, + fill.getVolumeMounts().get(0).getMountPath()); + assertEquals(List.of("sh", "-c"), fill.getCommand().subList(0, 2)); + assertTrue( + fill.getCommand().get(2).contains("*90/100"), fill.getCommand().get(2)); + assertRestricted(fill, 1001L, 1001L); + } + + @Test + void diskFillFailsWhenTheFillContainerExitedWithAnError() throws Exception { + Pod pod = broker("broker-8", "kafka9", pvc("data-0")); + server.expect() + .put() + .withPath("/api/v1/namespaces/default/pods/broker-8/ephemeralcontainers") + .andReturn(200, pod) + .once(); + FaultSpec spec = FaultSpec.builder("test-disk-fill-fails") + .targetNamespace("default") + .targetPod("broker-8") + .disruptionType(DisruptionType.DISK_FILL) + .chaosDurationSec(2) + .build(); + + var future = provider.triggerFault(spec); + // While the fault holds the volume, the container exits 1, as when it may write nowhere. + RecordedRequest write = null; + for (int i = 0; i < 50 && write == null; i++) { + RecordedRequest r = server.takeRequest(100, TimeUnit.MILLISECONDS); + if (r != null && r.getPath().endsWith("/ephemeralcontainers")) { + write = r; + } + } + assertNotNull(write); + String container = client.getKubernetesSerialization() + .unmarshal(write.getUtf8Body(), Pod.class) + .getSpec() + .getEphemeralContainers() + .get(0) + .getName(); + client.pods() + .inNamespace("default") + .withName("broker-8") + .editStatus(p -> new PodBuilder(p) + .withNewStatus() + .addNewEphemeralContainerStatus() + .withName(container) + .withNewState() + .withNewTerminated() + .withExitCode(1) + .endTerminated() + .endState() + .endEphemeralContainerStatus() + .endStatus() + .build()); + + ChaosOutcome outcome = future.get(10, TimeUnit.SECONDS); + + assertFalse(outcome.isPass()); + assertTrue(outcome.failureReason().contains("exited 1"), outcome.failureReason()); + } + + @Test + void diskFillPicksAClaimThenAVolumeNamedDataAndRefusesAPodWithNeither() { + assertEquals("data-1", KubernetesChaosProvider.dataVolume(podWith(emptyDir("data-0"), pvc("data-1")))); + assertEquals( + "data-0", KubernetesChaosProvider.dataVolume(podWith(emptyDir("strimzi-tmp"), emptyDir("data-0")))); + IllegalStateException e = assertThrows( + IllegalStateException.class, () -> KubernetesChaosProvider.dataVolume(podWith(emptyDir("tmp")))); + assertTrue(e.getMessage().contains("no data volume"), e.getMessage()); + } + + private static Pod podWith(io.fabric8.kubernetes.api.model.Volume... volumes) { + var container = new io.fabric8.kubernetes.api.model.ContainerBuilder().withName("kafka"); + for (var v : volumes) { + container + .addNewVolumeMount() + .withName(v.getName()) + .withMountPath("/" + v.getName()) + .endVolumeMount(); + } + return new PodBuilder() + .withNewMetadata() + .withName("broker") + .endMetadata() + .withNewSpec() + .withContainers(container.build()) + .withVolumes(volumes) + .endSpec() + .build(); + } + + private void node(String name, boolean unschedulable) { + client.nodes() + .resource(new io.fabric8.kubernetes.api.model.NodeBuilder() + .withNewMetadata() + .withName(name) + .endMetadata() + .withNewSpec() + .withUnschedulable(unschedulable ? true : null) + .endSpec() + .build()) + .create(); + } + + private void podOn(String namespace, String name, String node, String app, String ownerKind) { + var meta = new io.fabric8.kubernetes.api.model.ObjectMetaBuilder() + .withName(name) + .withNamespace(namespace) + .addToLabels("app", app); + if (ownerKind != null) { + meta.addNewOwnerReference() + .withKind(ownerKind) + .withName(name + "-owner") + .withApiVersion("apps/v1") + .withUid(name + "-uid") + .endOwnerReference(); + } + client.pods() + .inNamespace(namespace) + .resource(new PodBuilder() + .withMetadata(meta.build()) + .withNewSpec() + .withNodeName(node) + .addNewContainer() + .withName("c") + .withImage("c") + .endContainer() + .endSpec() + .build()) + .create(); + } + + /** The CRUD mock filters by no spec.nodeName field selector, which the API server does: answer it here. */ + private void podsOnNodeListed(String node) { + var pods = client.pods().inAnyNamespace().list().getItems().stream() + .filter(p -> node.equals(p.getSpec().getNodeName())) + .toList(); + server.expect() + .get() + .withPath("/api/v1/pods?fieldSelector=spec.nodeName%3D" + node) + .andReturn( + 200, + new io.fabric8.kubernetes.api.model.PodListBuilder() + .withItems(pods) + .build()) + .always(); + } + + private static String eviction(String namespace, String pod) { + return "/api/v1/namespaces/" + namespace + "/pods/" + pod + "/eviction"; + } + + private List requestPaths() throws InterruptedException { + List paths = new ArrayList<>(); + RecordedRequest r; + while ((r = server.takeRequest(0, TimeUnit.SECONDS)) != null) { + paths.add(r.getMethod() + " " + r.getPath()); + } + return paths; + } + + @Test + void nodeDrainCordonsEvictsEveryPodButDaemonSetsAndUncordons() throws Exception { + node("node-a", false); + node("node-b", false); + podOn("kafka", "broker-0", "node-a", "drain1", "StrimziPodSet"); + podOn("other", "web-0", "node-a", "web", "ReplicaSet"); + podOn("kube-system", "proxy-a", "node-a", "proxy", "DaemonSet"); + podOn("kafka", "broker-1", "node-b", "drain1", "StrimziPodSet"); + podsOnNodeListed("node-a"); + server.expect() + .post() + .withPath(eviction("kafka", "broker-0")) + .andReturn(201, "{}") + .once(); + server.expect() + .post() + .withPath(eviction("other", "web-0")) + .andReturn(201, "{}") + .once(); + FaultSpec spec = FaultSpec.builder("drain") + .targetNamespace("kafka") + .targetPod("broker-0") + .disruptionType(DisruptionType.NODE_DRAIN) + .chaosDurationSec(1) + .build(); + + ChaosOutcome outcome = provider.triggerFault(spec).get(10, TimeUnit.SECONDS); + + assertTrue(outcome.isPass(), outcome.failureReason()); + List paths = requestPaths(); + assertTrue(paths.contains("POST " + eviction("kafka", "broker-0")), paths.toString()); + assertTrue(paths.contains("POST " + eviction("other", "web-0")), paths.toString()); + assertFalse(paths.stream().anyMatch(p -> p.contains("proxy-a/eviction")), "a DaemonSet's pod stays"); + assertFalse(paths.stream().anyMatch(p -> p.contains("broker-1/eviction")), "another node's pod stays"); + int cordon = paths.indexOf("PATCH /api/v1/nodes/node-a"); + assertTrue(cordon >= 0 && cordon < paths.indexOf("POST " + eviction("kafka", "broker-0")), paths.toString()); + var after = client.nodes().withName("node-a").get(); + assertNull(after.getSpec().getUnschedulable(), "uncordoned"); + assertFalse(after.getMetadata().getLabels() != null + && after.getMetadata().getLabels().containsKey(KubernetesChaosProvider.CORDONED_LABEL)); + } + + @Test + void nodeDrainTriesABudgetsRefusalAgainAndFailsWhenTheDurationEnds() throws Exception { + node("node-c", false); + podOn("kafka", "broker-2", "node-c", "drain2", "StrimziPodSet"); + podsOnNodeListed("node-c"); + server.expect() + .post() + .withPath(eviction("kafka", "broker-2")) + .andReturn(429, "{\"kind\":\"Status\",\"code\":429}") + .always(); + provider.drainPollIntervalMs = 100; + FaultSpec spec = FaultSpec.builder("drain") + .targetNamespace("kafka") + .targetPod("broker-2") + .disruptionType(DisruptionType.NODE_DRAIN) + .chaosDurationSec(1) + .build(); + + ChaosOutcome outcome; + // The client retries a 429 itself, request-retry-backoff-limit times (2 in the application, + // 10 in the mock's): with none, each refusal comes back at once and the provider tries again. + try (KubernetesClient noRetry = new io.fabric8.kubernetes.client.KubernetesClientBuilder() + .withConfig(new io.fabric8.kubernetes.client.ConfigBuilder(client.getConfiguration()) + .withRequestRetryBackoffLimit(0) + .build()) + .build()) { + provider.client = noRetry; + outcome = provider.triggerFault(spec).get(10, TimeUnit.SECONDS); + } + + assertFalse(outcome.isPass()); + assertTrue(outcome.failureReason().contains("kafka/broker-2"), outcome.failureReason()); + assertTrue(outcome.failureReason().contains("PodDisruptionBudget"), outcome.failureReason()); + long tries = requestPaths().stream() + .filter(p -> p.equals("POST " + eviction("kafka", "broker-2"))) + .count(); + assertTrue(tries > 2, "tried " + tries + " times"); + assertNull(client.nodes().withName("node-c").get().getSpec().getUnschedulable(), "uncordoned all the same"); + } + + @Test + void nodeDrainLeavesANodeSomeoneElseCordonedAlone() throws Exception { + node("node-d", true); + podOn("kafka", "broker-3", "node-d", "drain3", "StrimziPodSet"); + FaultSpec spec = FaultSpec.builder("drain") + .targetNamespace("kafka") + .targetPod("broker-3") + .disruptionType(DisruptionType.NODE_DRAIN) + .chaosDurationSec(1) + .build(); + + ChaosOutcome outcome = provider.triggerFault(spec).get(10, TimeUnit.SECONDS); + + assertFalse(outcome.isPass()); + assertTrue(outcome.failureReason().contains("already cordoned"), outcome.failureReason()); + assertEquals( + Boolean.TRUE, client.nodes().withName("node-d").get().getSpec().getUnschedulable()); + assertFalse(requestPaths().stream().anyMatch(p -> p.startsWith("PATCH ")), "nothing was written"); + } + + @Test + void nodeDrainNeverDrainsTheKatesApisNode() throws Exception { + node("node-e", false); + provider.katesNode = new KatesNode() { + @Override + public java.util.Optional name() { + return java.util.Optional.of("node-e"); + } + }; + FaultSpec spec = FaultSpec.builder("drain") + .targetNamespace("kafka") + .disruptionType(DisruptionType.NODE_DRAIN) + .envOverrides(java.util.Map.of("TARGET_NODE", "node-e")) + .chaosDurationSec(1) + .build(); + + ChaosOutcome outcome = provider.triggerFault(spec).get(10, TimeUnit.SECONDS); + + assertFalse(outcome.isPass()); + assertTrue(outcome.failureReason().contains("the node the Kates API runs on"), outcome.failureReason()); + assertNull(client.nodes().withName("node-e").get().getSpec().getUnschedulable()); + } + + @Test + void nodeDrainByNodeLabelTakesAMatchingNodeOtherThanTheKatesApis() throws Exception { + for (String name : List.of("zone-a-1", "zone-a-2", "zone-b-1")) { + client.nodes() + .resource(new io.fabric8.kubernetes.api.model.NodeBuilder() + .withNewMetadata() + .withName(name) + .addToLabels("zone", name.substring(0, 6)) + .endMetadata() + .build()) + .create(); + } + provider.katesNode = new KatesNode() { + @Override + public java.util.Optional name() { + return java.util.Optional.of("zone-a-1"); + } + }; + FaultSpec spec = FaultSpec.builder("drain") + .targetNamespace("kafka") + .disruptionType(DisruptionType.NODE_DRAIN) + .envOverrides(java.util.Map.of("NODE_LABEL", "zone=zone-a")) + .chaosDurationSec(1) + .build(); + + for (int i = 0; i < 5; i++) { + assertEquals("zone-a-2", DrainTargets.node(client, spec, provider.katesNode.name())); + } + assertTrue(provider.triggerFault(spec).get(10, TimeUnit.SECONDS).isPass()); + assertTrue(requestPaths().contains("PATCH /api/v1/nodes/zone-a-2")); + } + + @Test + void isEvictableSkipsWhatKubectlDrainSkips() { + Pod plain = + new PodBuilder().withNewMetadata().withName("p").endMetadata().build(); + assertTrue(KubernetesChaosProvider.isEvictable(plain)); + assertFalse(KubernetesChaosProvider.isEvictable(new PodBuilder(plain) + .editMetadata() + .addToAnnotations("kubernetes.io/config.mirror", "x") + .endMetadata() + .build())); + assertFalse(KubernetesChaosProvider.isEvictable(new PodBuilder(plain) + .editMetadata() + .withDeletionTimestamp("2026-11-15T00:00:00Z") + .endMetadata() + .build())); + assertFalse(KubernetesChaosProvider.isEvictable(new PodBuilder(plain) + .withNewStatus() + .withPhase("Succeeded") + .endStatus() + .build())); + } } diff --git a/kates/src/test/java/com/bmscomp/kates/disruption/DisruptionOrphanReconcilerTest.java b/kates/src/test/java/com/bmscomp/kates/disruption/DisruptionOrphanReconcilerTest.java index 8e7d1db4..e4d962cd 100644 --- a/kates/src/test/java/com/bmscomp/kates/disruption/DisruptionOrphanReconcilerTest.java +++ b/kates/src/test/java/com/bmscomp/kates/disruption/DisruptionOrphanReconcilerTest.java @@ -84,6 +84,62 @@ void aPolicyStampedInTheSecondTheProcessStartedIsKept() { assertNotNull(get("kafka", "kates-netpol-krafter-brokers-sigma-4")); } + @Test + void uncordonsANodeADrainLeftCordonedAndLeavesTheOthers() { + Instant startedAt = Instant.now(); + cordoned("node-a", startedAt.minusSeconds(120), Instant.now().plusSeconds(3600)); + // A drain this process started. + cordoned("node-b", startedAt.plusSeconds(1), Instant.now().plusSeconds(3600)); + // Past its deadline, found by a later pass. + cordoned("node-c", startedAt.plusSeconds(1), Instant.now().minusSeconds(1)); + // Cordoned by someone else. + client.nodes() + .resource(new io.fabric8.kubernetes.api.model.NodeBuilder() + .withNewMetadata() + .withName("node-d") + .endMetadata() + .withNewSpec() + .withUnschedulable(true) + .endSpec() + .build()) + .create(); + + assertEquals(2, reconciler().reconcileCordons(startedAt)); + + assertNull(client.nodes().withName("node-a").get().getSpec().getUnschedulable()); + assertTrue( + client.nodes() + .withName("node-a") + .get() + .getMetadata() + .getLabels() + .isEmpty(), + "label taken off"); + assertEquals( + Boolean.TRUE, client.nodes().withName("node-b").get().getSpec().getUnschedulable()); + assertNull(client.nodes().withName("node-c").get().getSpec().getUnschedulable()); + assertEquals( + Boolean.TRUE, client.nodes().withName("node-d").get().getSpec().getUnschedulable()); + } + + private void cordoned(String name, Instant at, Instant deadline) { + client.nodes() + .resource(new io.fabric8.kubernetes.api.model.NodeBuilder() + .withNewMetadata() + .withName(name) + .addToLabels(com.bmscomp.kates.chaos.KubernetesChaosProvider.CORDONED_LABEL, "true") + .addToAnnotations( + com.bmscomp.kates.chaos.KubernetesChaosProvider.CORDONED_AT_ANNOTATION, + String.valueOf(at.toEpochMilli())) + .addToAnnotations(com.bmscomp.kates.chaos.FaultMarks.DEADLINE, deadline.toString()) + .endMetadata() + .withNewSpec() + .withUnschedulable(true) + .endSpec() + .build()) + .create(); + } + private DisruptionOrphanReconciler reconciler() { DisruptionOrphanReconciler reconciler = new DisruptionOrphanReconciler(); reconciler.kubeClient = client; diff --git a/kates/src/test/java/com/bmscomp/kates/disruption/DisruptionSafetyGuardTest.java b/kates/src/test/java/com/bmscomp/kates/disruption/DisruptionSafetyGuardTest.java index 8d826e61..dd62e2d4 100644 --- a/kates/src/test/java/com/bmscomp/kates/disruption/DisruptionSafetyGuardTest.java +++ b/kates/src/test/java/com/bmscomp/kates/disruption/DisruptionSafetyGuardTest.java @@ -1037,7 +1037,7 @@ void aNodeLabelOverrideCountsTheWorstNodeItMatches() { List.of("Plan would affect 2 brokers but maxAffectedBrokers=1"), guard.validatePlan(plan(1, byLabel, kill(0))).errors()); assertEquals( - List.of("NODE_DRAIN drains a node Litmus picks by envOverrides.NODE_LABEL 'drainable=true'; the count" + List.of("NODE_DRAIN drains a node envOverrides.NODE_LABEL 'drainable=true' matches; the count" + " takes the worst case, node gamma, which runs 1 broker"), guard.dryRun(plan(-1, byLabel)).steps().getFirst().warnings()); } From 3203324f35894ad8c24d7c4430dd59ac15163c1d Mon Sep 17 00:00:00 2001 From: Said Boudjelda Date: Sun, 15 Nov 2026 17:43:33 +0100 Subject: [PATCH 2/2] fix(operator): the backend role grants what NODE_DRAIN needs, as the Kates chart's does operator/backend-role-check.sh holds the operator's backend ClusterRole to the kates chart's: with rbac.backendDirectChaos it now also grants patch on nodes and create on pods/eviction. --- operator/chart/templates/backend-rbac.yaml | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/operator/chart/templates/backend-rbac.yaml b/operator/chart/templates/backend-rbac.yaml index 742604d2..95641414 100644 --- a/operator/chart/templates/backend-rbac.yaml +++ b/operator/chart/templates/backend-rbac.yaml @@ -72,8 +72,10 @@ rules: every experiment as its own litmus-admin service account, so a Litmus install needs none of them. On a Kafka that Strimzi does not manage, SCALE_DOWN sets replicas through the StatefulSet's scale subresource (its - PATCH on the StatefulSet is granted above, for ROLLING_RESTART); CPU_STRESS - and IO_STRESS add a container through pods/ephemeralcontainers. + PATCH on the StatefulSet is granted above, for ROLLING_RESTART); CPU_STRESS, + MEMORY_STRESS, IO_STRESS and DISK_FILL add a container through + pods/ephemeralcontainers; NODE_DRAIN cordons and uncordons a node (patch) + and evicts its pods in every namespace through pods/eviction. */}} - apiGroups: ["apps"] resources: ["statefulsets/scale"] @@ -81,6 +83,12 @@ rules: - apiGroups: [""] resources: ["pods/ephemeralcontainers"] verbs: ["update"] + - apiGroups: [""] + resources: ["nodes"] + verbs: ["patch"] + - apiGroups: [""] + resources: ["pods/eviction"] + verbs: ["create"] {{- end }} --- apiVersion: rbac.authorization.k8s.io/v1