From 37a1a9a8f60f22244299f2b8605af807828a5ba9 Mon Sep 17 00:00:00 2001 From: Said Boudjelda Date: Tue, 17 Nov 2026 01:58:48 +0100 Subject: [PATCH] docs: explain the Kates operator mode, and that standalone is unchanged A new book chapter, Running Kates Under the Kates Operator, after the Deployment Guide: the two modes side by side, what stays the same in the default standalone mode (the quorum is off unless the operator configures peers; the drain switch and the health quorum block are additions), the one change for early operator users (KatesTest and KafkaCompliance), deploying with --kates-mode operator, the Kates resource's conditions, the quorum and its failover, drained upgrades, a lost node, a lost majority, a Helm install, switching modes and the limits. Outputs are from panda. Tutorial 16 runs the quorum on panda: deploy, read the resource, use the same CLI, kill the leader, drain, lose a node, switch back. The Deployment Guide, Upgrade Playbook, Part V page, glossary, concept registry, book README, tutorials 1 and 8, the tutorials index and the README point to both. --- README.md | 8 +- docs/book/12-deployment.md | 4 +- docs/book/18-upgrade-playbook.md | 2 + docs/book/CONCEPTS.md | 1 + docs/book/README.md | 4 +- docs/book/_quarto.yml | 1 + docs/book/appendix-a-glossary.md | 4 + docs/book/appendix-index.md | 2 +- docs/book/kates-operator.md | 245 +++++++++++++++ docs/book/part-deployment-operations.md | 3 +- docs/tutorials/01-getting-started.md | 2 + docs/tutorials/08-deploy-and-detect.md | 1 + .../16-running-kates-under-the-operator.md | 285 ++++++++++++++++++ docs/tutorials/README.md | 7 +- 14 files changed, 560 insertions(+), 9 deletions(-) create mode 100644 docs/book/kates-operator.md create mode 100644 docs/tutorials/16-running-kates-under-the-operator.md diff --git a/README.md b/README.md index 0da668ff3..e2b3b37e6 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,7 @@ Performance tests work against any Kafka cluster the Kates API can reach; fault - **Security audit.** `kates security audit` grades broker configuration A–F. Related commands map the checks to CIS, SOC 2 and PCI DSS, save a baseline and report drift from it. - **CI gates.** `kates gate --min-grade B` runs a load test and exits non-zero below the performance grade, `kates test apply -f --wait` fails when a scenario violates its SLA gates, and `kates security gate` does the same for the security grade. Finished tests can notify webhooks (`kates webhook add`). - **A multi-zone lab on one machine.** A three-node Kind cluster (`panda`) whose nodes (`alpha`, `sigma`, `gamma`) sit in separate zones. On it, `kates deploy` gives each zone its own StorageClass and broker pool and turns on rack awareness, so a full zone failure can be rehearsed locally. +- **A Kates API that survives a lost node.** The default install runs the Kates API standalone, as one pod, and keeps working as it always has. `kates deploy --kates-mode operator --kates-replicas 3` runs it under the Kates operator instead: a quorum of three or five behind the same Service and key, whose leader is replaced within seconds, and which drains live runs before an upgrade. See [Running Kates Under the Kates Operator](docs/book/kates-operator.md). - **Metrics and dashboards.** The Kates API exposes Prometheus metrics, the `monitoring` chart adds twelve Grafana dashboards for Kafka, Kates and chaos runs, and the charts ship Prometheus alert rules. The Kates API also exports OTLP traces, though the default install has no Jaeger to receive them; [Observability](docs/book/09-observability.md) covers the dashboards, the alerts and tracing. ## Why Kates @@ -147,7 +148,9 @@ With the default isolated topology: | Kafka UI | `kafka-ui` | `kafka` | | LitmusChaos | `chaos` | `litmus` | -Kafka Connect, MirrorMaker 2, Kyverno and the secret manager are opt-in (`--with-kafka-connect`, `--with-mirror-maker2`, `--with-kyverno`, `--with-secret-manager`). `--topology single` puts everything except the operators in one namespace, `kates-stack`. After a failure, fix the cause and run `kates deploy` again: it skips or upgrades the components already installed. +Kafka Connect, MirrorMaker 2, Kyverno and the secret manager are opt-in (`--with-kafka-connect`, `--with-mirror-maker2`, `--with-kyverno`, `--with-secret-manager`). `--topology single` puts everything except the operators in one namespace, `kates-stack`. + +The table shows the default, standalone mode: the `kates` chart runs the Kates API as one pod, and every install made before the operator mode keeps working unchanged. With `--kates-mode operator --kates-replicas 3`, deploy installs the Kates operator (release `kates-operator` in `kates-operator`) and a PostgreSQL for Kates (`kates-db`) instead, and the operator runs the Kates API as a quorum of three. Its leader takes the writes, a new one is elected within seconds when it dies, a member whose node is gone is rebuilt elsewhere, and `mode: Drain` lets an upgrade wait for live runs. The Service `kates` and the Secret `kates-api-key` are the same in both modes, so every command below works with either. [Running Kates Under the Kates Operator](docs/book/kates-operator.md) explains the mode, and [Tutorial 16](docs/tutorials/16-running-kates-under-the-operator.md) walks through it. After a failure, fix the cause and run `kates deploy` again: it skips or upgrades the components already installed. ### Reach the services @@ -260,11 +263,12 @@ The CLI and its full-screen views talk to the backend over REST. The backend als | Where | What | |:--|:--| | [The book](https://bmscomp.github.io/kates/) ([source](docs/book/README.md)) | Concepts, the CLI, the REST and gRPC APIs, deployment, security, upgrades and troubleshooting | -| [Tutorials](docs/tutorials/README.md) | Step-by-step labs: a first test, every test type, chaos, CI integration, Kafka Connect, MirrorMaker 2 migrations, dashboards | +| [Tutorials](docs/tutorials/README.md) | Step-by-step labs: a first test, every test type, chaos, CI integration, Kafka Connect, MirrorMaker 2 migrations, dashboards, the Kates operator | | [CLI reference](docs/book/10-cli-reference.md) | Commands, flags and contexts | | [REST API](docs/book/11-api-reference.md) · [gRPC API](docs/book/16-grpc-api.md) | Endpoints, authentication, exports | | [Chaos in practice](docs/book/07-chaos-practice.md) | Disruption types, providers, playbooks, guardrails and grading | | [Deployment](docs/book/12-deployment.md) · [Installing Kafka with Helm](docs/book/20-installation-guide.md) | Topologies, sizing, cloud overlays, chart-by-chart installs | +| [Kates operator](docs/book/kates-operator.md) · [its chart](operator/chart/README.md) | The Kates API as a quorum: deploy, conditions, failover, drained upgrades; standalone stays the default | | [Troubleshooting Index](docs/book/appendix-b-troubleshooting.md) | Symptoms mapped to causes and chapters | | [Version matrix](docs/book/appendix-d-versions.md) | Every pinned version (Kafka, Strimzi, Quarkus, Java, Go, charts), generated and checked in CI | | [Local development](docs/local-development.md) | Makefile targets, image management, working behind a proxy | diff --git a/docs/book/12-deployment.md b/docs/book/12-deployment.md index 70a055054..27f8bb21b 100644 --- a/docs/book/12-deployment.md +++ b/docs/book/12-deployment.md @@ -147,7 +147,7 @@ The old PersistentVolumeClaim stays until you delete it. kates deploy --kates-mode operator --kates-replicas 3 ``` -Either way the Service is `kates` in the Kates namespace, and the API key is in Secret `kates-api-key`, so `kates ports`, `kates health` and the rest work the same. Deploy refuses to install one mode over the other: uninstall first (`kates clean`), or keep the mode you have. `kates clean` removes the `Kates` resource while the operator still runs, then the operator. The operator doesn't yet support the `INTEGRATION_CDC` test. +[Running Kates Under the Kates Operator](kates-operator.md) explains the operator mode in full: the quorum, its conditions, draining before an upgrade and what stays the same in the standalone mode. Either way the Service is `kates` in the Kates namespace, and the API key is in Secret `kates-api-key`, so `kates ports`, `kates health` and the rest work the same. Deploy refuses to install one mode over the other: uninstall first (`kates clean`), or keep the mode you have. `kates clean` removes the `Kates` resource while the operator still runs, then the operator. The operator doesn't yet support the `INTEGRATION_CDC` test. For Kafka broker storage, always use persistent volumes — even in development. Kafka's log retention depends on data being durable, and losing broker data mid-test invalidates results. @@ -1382,4 +1382,4 @@ This deletes the Kind cluster and all associated resources. - `make all` is `kates deploy` with a topology prompt; the per-component targets build the same stack piece by piece. - The JVM image measures with a low-pause collector; the native image starts fast but pauses, so it suits CI, not measurement. -Your stack is running, so check that it is the cluster you meant to build: [Compliance Checks](compliance.md) writes that architecture down as a file and holds `krafter` to it. +Your stack is running with the Kates API as one standalone pod. If a lost node or an upgrade must not take it down, [Running Kates Under the Kates Operator](kates-operator.md) runs it as a quorum behind the same Service. diff --git a/docs/book/18-upgrade-playbook.md b/docs/book/18-upgrade-playbook.md index bec0189c2..d32b18912 100644 --- a/docs/book/18-upgrade-playbook.md +++ b/docs/book/18-upgrade-playbook.md @@ -304,6 +304,8 @@ The `kates` chart installs the Kates API, the service in the cluster that runs y Use this procedure to move the Kates API to a new release, after Kafka. It needs the published chart of that release, or on Kind the repository's `charts/kates`, and rights to upgrade the `kates` release in the `kates` namespace. When the new pods don't become ready, read their log with `kubectl logs deployment/kates -n kates`, and roll back with Step 4. +This procedure is for the standalone mode, the default. Under the Kates operator there is no `kates` release to upgrade: you change the `Kates` resource's version, and under `mode: Drain` no live run is lost, as [Upgrading Without Losing a Run](kates-operator.md#upgrading-without-losing-a-run) shows. + ### Step 1 — Find the Chart for the New Version A chart version and the Kates API version it ships differ, so read the pair from the chart itself. The [Version & Compatibility Matrix](appendix-d-versions.md) lists the pair for this release: diff --git a/docs/book/CONCEPTS.md b/docs/book/CONCEPTS.md index df5199b69..1bb294f5d 100644 --- a/docs/book/CONCEPTS.md +++ b/docs/book/CONCEPTS.md @@ -85,6 +85,7 @@ Part V is home to the platform around the cluster: the operator, the charts, sec | Operator, custom resource, CRD and reconciliation | [How the Operator Works](deploying-strimzi-operator.md#how-the-operator-works), in [Deploying the Strimzi Operator](deploying-strimzi-operator.md) | The Strimzi Cluster Operator watches custom resources such as `Kafka` and `KafkaNodePool` and changes the cluster's pods, Secrets and other objects until they match, so while it's down, changes to those resources wait. | `gl-operator`; also `gl-custom-resource`, `gl-crd`, `gl-reconciliation` | | Installing and upgrading the operator | [Upgrading the Operator](deploying-strimzi-operator.md#upgrading-the-operator), in [Deploying the Strimzi Operator](deploying-strimzi-operator.md) | The Strimzi operator is its own Helm release, `strimzi-operator` in the `strimzi-operator` namespace, installed and upgraded separately from the Kafka clusters it manages. | `gl-strimzi` | | Which Kates image runs where, and whose GC shows in P99 | planned: JVM or Native Image (Phase C), in [Deployment Guide](12-deployment.md) | `kates deploy` runs the Kates API as a native image on every cluster, which uses GraalVM's Serial GC, and a plain `helm install` the JVM image with generational ZGC, so the P99 and Max of a run `kates deploy` set up include Serial GC pauses. | `gl-graalvm-native-image`; also `gl-virtual-thread` | +| The Kates operator, its quorum and the standalone mode | [Two Modes, One API](kates-operator.md#two-modes-one-api), in [Running Kates Under the Kates Operator](kates-operator.md) | `kates deploy --kates-mode operator` runs the Kates API from a `Kates` resource, as one member or a quorum of three or five whose leader takes the writes behind the same `kates` Service, while the standalone `kates` chart stays the default and unchanged. | `gl-kates-operator` | | Single versus isolated namespace topology | [Single-Namespace vs Multi-Namespace](12-deployment.md#single-namespace-vs-multi-namespace), in [Deployment Guide](12-deployment.md) | `kates deploy --topology isolated`, the default, gives Kafka, the Kates API and the chaos tooling their own namespaces (`kafka`, `kates`, `litmus`), `--topology single` puts them in one (`kates-stack` by default), and the operator goes to `strimzi-operator` either way. | `gl-isolated-topology`; also `gl-single-namespace-topology` | | PrometheusRule alerts and their runbooks | [PrometheusRule Alerts](20-installation-guide.md#prometheusrule-alerts), in [Installing Kafka with the kafka-cluster Helm Chart](20-installation-guide.md) | The `kafka-cluster` chart renders its alerts as a PrometheusRule where the Prometheus Operator's API exists, and each rule's `runbook_url` points at its entry in `docs/kafka-cluster-runbook.md`. | `gl-prometheusrule` | | Topics and users as code | [Topics & Users Reference](20-installation-guide.md#topics--users-reference), in [Installing Kafka with the kafka-cluster Helm Chart](20-installation-guide.md) | The `kafka-cluster` chart turns each entry under `topics.items` and `users.items` into a Strimzi `KafkaTopic` or `KafkaUser` resource, which the Entity Operator applies to the cluster. | `gl-kafkatopic`; also `gl-kafkauser` | diff --git a/docs/book/README.md b/docs/book/README.md index 6fd3cab34..2f0232091 100644 --- a/docs/book/README.md +++ b/docs/book/README.md @@ -47,6 +47,7 @@ Reading order is defined by [`_quarto.yml`](_quarto.yml) — filenames are stabl | [Installing Kafka with the kafka-cluster Helm Chart](20-installation-guide.md) | Step-by-step Kafka deployment with prerequisites and verification | | [Kafka Deployment Engineering](15-kafka-deployment.md) | The engineering rationale: Strimzi, KRaft, broker tuning, and operations | | [Deployment Guide](12-deployment.md) | Deploying the Kates stack: topology decisions, sizing, and cloud guidance | +| [Running Kates Under the Kates Operator](kates-operator.md) | The Kates API as a quorum under the operator: failover, drained upgrades, lost nodes; standalone stays the default | | [Compliance Checks](compliance.md) | A reference architecture as a file, and checks that hold the live cluster to it | | [Security & Compliance](17-security.md) | Authentication, authorization, certificates, network policies, and Kyverno | | [Multi-Tenancy](19-multi-tenancy.md) | Topic naming, service onboarding, quotas, and tenant isolation | @@ -119,7 +120,8 @@ Don't read this book cover-to-cover. Pick a reading path based on what you need: ### 🚀 "I want to deploy Kates" 1. [Installing Kafka with the kafka-cluster Helm Chart](20-installation-guide.md) — full step-by-step with prerequisites (~30 min) 2. [Deployment Guide](12-deployment.md) — architecture decisions, resource sizing, cloud deployment (~15 min) -3. [The Cluster Under Test](03-cluster.md) — understanding what you just deployed (~10 min) +3. [Running Kates Under the Kates Operator](kates-operator.md) — optional: the Kates API as a quorum that survives a lost node or an upgrade (~20 min) +4. [The Cluster Under Test](03-cluster.md) — understanding what you just deployed (~10 min) ### 💥 "I want to run chaos experiments" 1. [Chaos Engineering Theory](06-chaos-theory.md) — principles and methodology (~10 min) diff --git a/docs/book/_quarto.yml b/docs/book/_quarto.yml index 645133375..b592c1261 100644 --- a/docs/book/_quarto.yml +++ b/docs/book/_quarto.yml @@ -93,6 +93,7 @@ book: - 20-installation-guide.md - 15-kafka-deployment.md - 12-deployment.md + - kates-operator.md - compliance.md - 17-security.md - 19-multi-tenancy.md diff --git a/docs/book/appendix-a-glossary.md b/docs/book/appendix-a-glossary.md index df876bede..d0b90ff50 100644 --- a/docs/book/appendix-a-glossary.md +++ b/docs/book/appendix-a-glossary.md @@ -342,6 +342,10 @@ The Kafka Advanced Testing & Engineering Suite, in two halves: the `kates` CLI o The Kates server: the Quarkus application that serves the REST and gRPC APIs, runs benchmark backends and faults, and stores results. The `kates` chart deploys it as the `kates` Deployment and Service, serving REST and gRPC on port 8080, in the `kates` namespace under the isolated topology; `make ports` forwards it to `localhost:30083`. See [Architecture & Design](02-architecture.md#high-level-architecture). +### Kates Operator {#gl-kates-operator} + +A Kubernetes operator: a controller that runs an application from a custom resource describing it. In Kates, it runs the Kates API from a `Kates` resource, as one member or as a quorum of three or five whose leader takes the writes, and drains live runs before an upgrade; `kates deploy --kates-mode operator` installs it, and the standalone `kates` chart stays the default. See [Running Kates Under the Kates Operator](kates-operator.md). + ### Kind {#gl-kind} Kubernetes IN Docker: a tool that runs a Kubernetes cluster in Docker containers, one container per node. `make cluster` creates the `panda` cluster with it. See [Deployment Guide](12-deployment.md#kubernetes-cluster). diff --git a/docs/book/appendix-index.md b/docs/book/appendix-index.md index b8cefe2ef..1f3e06abe 100644 --- a/docs/book/appendix-index.md +++ b/docs/book/appendix-index.md @@ -419,7 +419,7 @@ Every term the book teaches, with the sections that teach it: the ones that use ## Q [Quorum]{#idx-quorum} -: [The Cluster Under Test](03-cluster.md): [Kafka You Need for This Book](03-cluster.md#kafka-you-need-for-this-book), [Replication Configuration](03-cluster.md#replication-configuration); [Observability & Monitoring](09-observability.md): [The Dashboard Set](09-observability.md#the-dashboard-set); [Kafka Deployment Engineering](15-kafka-deployment.md): [Cluster Architecture](15-kafka-deployment.md#cluster-architecture); [Compliance Checks](compliance.md): [Writing Constraints](compliance.md#writing-constraints), [Placing Pods Across Zones and Nodes](compliance.md#placing-pods-across-zones-and-nodes); [CLI Reference](10-cli-reference.md): [Commands](10-cli-reference.md#commands); [REST API Reference](11-api-reference.md): [Endpoints](11-api-reference.md#endpoints) +: [Kafka Deployment Engineering](15-kafka-deployment.md): [Cluster Architecture](15-kafka-deployment.md#cluster-architecture); [Running Kates Under the Kates Operator](kates-operator.md): [Standalone Stays the Default](kates-operator.md#standalone-stays-the-default), [Reading the Kates Resource](kates-operator.md#reading-the-kates-resource), [The Quorum](kates-operator.md#the-quorum), [Limits](kates-operator.md#limits); [Compliance Checks](compliance.md): [Writing Constraints](compliance.md#writing-constraints), [Placing Pods Across Zones and Nodes](compliance.md#placing-pods-across-zones-and-nodes); [REST API Reference](11-api-reference.md): [Endpoints](11-api-reference.md#endpoints) [Quota]{#idx-quota} : [Kafka Deployment Engineering](15-kafka-deployment.md): [Broker Settings and Why](15-kafka-deployment.md#broker-settings-and-why); [Multi-Tenancy](19-multi-tenancy.md): [How Quotas Work](19-multi-tenancy.md#how-quotas-work), [Monitoring a Tenant](19-multi-tenancy.md#monitoring-a-tenant) diff --git a/docs/book/kates-operator.md b/docs/book/kates-operator.md new file mode 100644 index 000000000..f4819c8be --- /dev/null +++ b/docs/book/kates-operator.md @@ -0,0 +1,245 @@ +# Running Kates Under the Kates Operator + +A standalone install runs the [Kates API](appendix-a-glossary.md#gl-kates-api) as one pod. If the node under that pod dies, every run and fault it had live is lost, and nothing answers until Kubernetes schedules the pod again. An upgrade has the same effect: the rollout replaces the pod whether or not a run is halfway through. For a lab that is fine. For a cluster where a nightly suite or a Game Day depends on Kates, it is a single point of failure. + +The [Kates operator](appendix-a-glossary.md#gl-kates-operator) removes it. You describe the Kates API as a `Kates` resource, and the operator runs it as one member or as a quorum of three or five. The quorum keeps a leader when members are lost, an upgrade waits for live runs to finish, and a member whose node is gone is rebuilt on another node. You keep the same CLI, the same Service and the same API key. + +This chapter shows how to choose between the two modes, deploy under the operator, read what it reports and get through a node loss or an upgrade. It also shows what stays exactly as it was when you keep the standalone mode. + +- Choose between the standalone and operator modes +- Deploy Kates under the operator with `kates deploy` +- Read the `Kates` resource's conditions and the quorum's leader +- Upgrade without losing a live run, and recover from a lost node or majority + +## Two Modes, One API + +`kates deploy` installs the Kates API in one of two modes, chosen with `--kates-mode` or in the wizard right after the components. The table compares what each one runs. The last two rows are why you would pick one over the other: + +| | `standalone` (default) | `operator` | +|---|---|---| +| What runs the Kates API | The `kates` chart: a Deployment of one pod | The Kates operator, from a `Kates` resource: a StatefulSet | +| PostgreSQL | The chart's own | The `kates-db` release, or your own with `--kates-db-secret` | +| Members | One | One, or three or five: a quorum whose leader takes the writes | +| A lost node | Live runs are lost until the pod is rescheduled | A follower takes over as leader within seconds | +| Upgrades | A rollout that replaces the pod at once | Followers first, the leader last, and under `Drain` only once live runs end | +| Needs cluster admin | No (`--adminless` works) | Yes: the operator brings a CRD and a ClusterRole | + +Everything you use the Kates API through is the same in both modes. The Service is `kates` in the Kates namespace, on ports 8080 and 9000, and the API key is in the Secret `kates-api-key`. So `kates ports`, `kates health`, `kates test apply`, the dashboards, the MCP server and every tutorial work unchanged in either mode. + +## Standalone Stays the Default + +The operator mode is an addition. Nothing in it changes a standalone install, and a standalone install is what you get unless you ask for the operator. The table lists what that means in practice: + +| Question | Answer | +|---|---| +| What does `kates deploy` install with no new flag? | The standalone `kates` chart | +| Does a running standalone install change on upgrade? | No: the Kates API runs its quorum only when the operator configures peers | +| Do older scenario files, scripts and CI jobs still work? | Yes: the REST, gRPC and CLI interfaces are the same in both modes | +| Does `--adminless` still work? | Yes, with the standalone mode; the operator mode refuses it | +| Is anything new in the Kates API itself? | Two additions: a drain switch (`PUT /api/drain`) and a `quorum` block in `/api/health` | + +The quorum's code ships in every Kates API image, but it stays off unless the setting that lists the members is present. Only the operator sets it, so a standalone pod's `/api/health` shows `"quorum":{"enabled":false}` and serves every request itself, as one standalone pod. The drain switch is new and needs the `admin` scope. Nothing calls it unless you or the operator do, so an install that never drains sees no difference. + +`kates deploy` also guards the boundary between the modes. It refuses to install one mode over the other, because both would claim the Service `kates`. The flags that only make sense under the operator (`--kates-replicas` above 1 and `--kates-db-secret`) are refused with the standalone mode, before anything reaches the cluster. + +::: {.callout-important title="The one change for earlier operator users"} +The operator runs the `Kates` kind only. If you ran an early operator with `KatesTest` or `KafkaCompliance` resources, move them to a `CronJob` that runs the CLI before you upgrade the operator, as [Upgrading to 0.3.0](../../operator/chart/README.md#upgrading-to-030) shows. A standalone install never had those kinds and needs nothing. +::: + +The usual upgrade notes still apply to a standalone install, whatever the operator does. The [Upgrade Playbook](18-upgrade-playbook.md) lists them per component, such as the bundled PostgreSQL's change of image. + +## Deploying Under the Operator + +One flag switches the mode, and a second asks for a quorum. With three members the quorum keeps a leader when one is lost, and with five when two are: + +```bash +kates deploy --kates-mode operator --kates-replicas 3 +``` + +Deploy then installs three things in place of the `kates` chart, in this order: + +1. The Kates operator, as the [Helm release](appendix-a-glossary.md#gl-helm-release) `kates-operator` in the namespace `kates-operator` (`--operator-ns` changes it). It watches the Kates namespace and may bind the Kates API's cluster role there, for faults, topology and compliance reads. +2. A PostgreSQL for Kates, the release `kates-db` of the repository's `cdc-postgresql` chart, open to the Kates pods alone. With `--kates-db-secret `, deploy skips it and uses an existing Secret with the keys `url` (`jdbc:postgresql://…`), `username` and `password`. +3. The `Kates` resource `kates`, with its API key, its Kafka connection to [`krafter`](appendix-a-glossary.md#gl-krafter), the Kafka cluster under test, and the image version this CLI pins. Deploy then waits up to 10 minutes for the resource to report `Ready`. + +Deploy stops at once rather than waiting out the 10 minutes when the operator reports a member that can never join, as [Reading the Kates Resource](#reading-the-kates-resource) explains. The number of members is fixed once the resource exists. To change it, remove Kates and deploy again. + +The wizard (`kates deploy -i`) asks the same questions after the components: standalone or operator, then the number of members, the operator's namespace and an existing database. + +## Reading the Kates Resource + +The operator reports everything on the `Kates` resource, so `kubectl get` is the first place to look. On [`panda`](appendix-a-glossary.md#gl-panda), the book's three-node Kind cluster, after `kates deploy --kates-mode operator --kates-replicas 3`: + +```bash +kubectl get kates -n kates +``` + +Output: + +```text +NAME VERSION REPLICAS LEADER READY MODE DRAINED AGE +kates 1.41.2-native 3 kates-1 True Active 7m11s +``` + +Read it left to right: + +1. `VERSION` is the version every member runs. During an upgrade it shows the old version until the last member is replaced. +2. `LEADER` is the member that takes the writes, here `kates-1`. +3. `READY` is the operator's verdict: every member is up and, with a quorum, a leader is elected. +4. `MODE` is `Active` or `Drain`, and `DRAINED` fills in only under `Drain`. + +When `READY` is `False`, the condition's reason says why. These are the reasons you can meet, with what each one asks of you: + +| Condition and reason | What it means | What to do | +|---|---|---| +| `Ready=False`, `NoLeader` | The quorum is still electing | Wait: an election takes seconds | +| `Ready=False`, `NoQuorum` | A member runs an image older than the quorum, so it can never lead or catch up | Set `spec.version` to an image with the quorum | +| `Ready=False`, `MemberLost` | A member's node is gone and the operator is rebuilding it elsewhere | Wait for it to catch up | +| `Ready=False`, `NameTaken` | A Service, ServiceAccount or StatefulSet of that name exists and the operator did not create it | Rename one of them | +| `Ready=False`, `QuorumSecretTaken` | Another `Kates` in the namespace already runs a quorum | Use one quorum per namespace | +| `Drained=False`, `KeyUnreadable` | The operator cannot read the API key from `kates-api-key` | Put the key in that Secret | +| `Drained=False`, `DrainUnsupported` | The Kates API is older than the drain switch | Upgrade it under `Active` | + +To print every condition with its reason and message: + +```bash +kubectl get kates kates -n kates -o jsonpath=\ +'{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}' +``` + +## The Quorum + +The members of a quorum elect a leader among themselves with Raft, the consensus algorithm KRaft also uses [@ongaro2014search]. They keep only who leads in their quorum. Runs, results and schedules stay in PostgreSQL, so a member loses nothing it needs when its volume goes. Only the leader takes writes: a follower answers a write with `503` and a `Retry-After` header, and serves reads. + +You never pick a member yourself. The Service `kates` sends traffic to the leader alone, through the pod label `kates.io/quorum-role=leader`, which the operator moves to a new leader within about five seconds. The label shows which pod leads: + +```bash +kubectl get pods -n kates -L kates.io/quorum-role +``` + +Output: + +```text +NAME READY STATUS RESTARTS AGE QUORUM-ROLE +kates-0 1/1 Running 0 7m10s +kates-1 1/1 Running 0 7m10s leader +kates-2 1/1 Running 0 7m10s +kates-db-0 1/1 Running 0 9m26s +``` + +Each member also answers its own health check, whose `quorum` block names the leader and the Raft term. It is the same `/api/health` the CLI reads, so you can see it through the API server's proxy without a port-forward: + +```bash +kubectl get --raw \ + /api/v1/namespaces/kates/services/http:kates:8080/proxy/api/health | jq .quorum +``` + +Output: + +```text +{ + "enabled": true, + "state": "RUNNING", + "memberId": "kates-1", + "leader": true, + "leaderId": "kates-1", + "term": 1, + "appliedIndex": 0 +} +``` + +When the leader dies, the others elect a new one within seconds. Before the new leader takes a write, it fails the runs the dead leader had live and removes the faults it left behind, so no fault outlives the member that injected it. A run that was live on the dead leader is reported failed, not lost: run it again. + +The members reach each other over mutual TLS on port 9300, each through a Service of its own name (`kates-0`, `kates-1` and so on). The operator makes their CA and certificates and renews each certificate at two-thirds of its year-long life. A `PodDisruptionBudget` lets a node drain take one member at a time. + +::: {.callout-tip title="Try it: does a run survive the leader?"} +Start a long run, delete the leader's pod, and watch `LEADER` change: `kubectl delete pod -n kates kates-1`, then `kubectl get kates -n kates -w`. The run the old leader had live ends as failed; a new run started after the election succeeds through the same `kates` Service. +::: + +## Upgrading Without Losing a Run + +An upgrade replaces one member at a time, followers first and the leader last. The operator replaces the next member only once every member is ready, knows the leader and holds an entry of its log, so a roll never leaves the quorum without a majority. A plain version change does that under `Active`, but runs live on the leader when its turn comes are lost with it. + +To lose none, set `mode: Drain` with the new version. The three commands below drain, wait for the new version and lift the drain: + +```bash +kubectl patch kates kates -n kates --type merge \ + -p '{"spec":{"mode":"Drain","version":"-native"}}' +kubectl wait --for=jsonpath='{.status.version}'=-native \ + kates/kates -n kates --timeout=30m +kubectl patch kates kates -n kates --type merge -p '{"spec":{"mode":"Active"}}' +``` + +Under `Drain`, the operator calls the drain switch on every member. New runs, faults and resilience runs are answered `503`, schedules stop firing, and live work goes on to its end. The `Drained` condition lists what is still live and turns `True` once nothing is. Only then is a member on the old version replaced, and the new member is drained as soon as it answers. + +The operator never sets the mode back to `Active`. You do, with the last command, and new runs are taken again from then on. Running `kates deploy --kates-mode operator` from a newer CLI also upgrades: it applies the version that CLI pins and leaves `mode` as you set it. + +## When a Node Goes + +The members prefer one node each, so a lost node takes one member, not the majority. On storage tied to a node, such as Kind's `local-path` class, a member whose node is gone cannot start anywhere else: its volume is on that node. The operator handles it in four steps: + +1. The member's pod waits `Pending`, and `Ready` is `False` with reason `MemberLost`. +2. After two minutes, the operator deletes the member's claim and its pod. It does this only while the other members follow a leader and have caught up. +3. The StatefulSet makes both again, on another node. +4. The new member starts with an empty log, catches up, and votes again. `Ready` turns `True`. + +The operator rebuilds one member at a time, because a member that is catching up does not vote. A node you drain and leave unschedulable for two minutes is handled the same way. The quorum keeps serving throughout, since the remaining members still hold the majority. + +## Losing the Majority + +With two of three members lost, or three of five, there is no leader, and the operator replaces nothing. Nothing on the members' volumes matters beyond terms and votes, so you start the quorum over. Delete every member's claim, then the pods: + +```bash +kubectl -n kates delete pvc -l app.kubernetes.io/instance=kates --wait=false +kubectl -n kates delete pod -l app.kubernetes.io/instance=kates +``` + +The new quorum counts terms from the start again. Your runs and results are in PostgreSQL and are all still there. A fault the old quorum left behind is removed when its deadline passes, not straight away. + +## Installing With Helm Instead + +`kates deploy` is the shortest path, but the operator is an ordinary chart, published as `oci://ghcr.io/bmscomp/charts/kates-operator` and signed with cosign. With the chart installed, a minimal `Kates` resource names a version, a database Secret and the Kafka bootstrap address: + +```yaml +apiVersion: kates.io/v1alpha1 +kind: Kates +metadata: + name: main +spec: + version: -native + replicas: 3 # 1 (default), 3 or 5; fixed once created + database: + secretRef: + name: kates-db # keys: url (jdbc:postgresql://...), username, password + kafka: + bootstrapServers: krafter-kafka-bootstrap.kafka.svc:9092 +``` + +The chart's README, [kates-operator](../../operator/chart/README.md), covers what this chapter leaves out. That includes the four RBAC modes (one namespace, several, every namespace, or nothing cluster-scoped), which namespaces get the Kates API's cluster role, SASL to Kafka and every field of the resource. + +## Switching Modes + +You switch modes by removing Kates and deploying again, because `kates deploy` refuses to install one mode over the other. `kates clean` removes either mode. Under the operator, it deletes the `Kates` resource while the operator still runs, so the operator removes what it made, then removes the operator. To leave the rest of the stack in place, remove only Kates: `helm uninstall kates -n kates` for the standalone mode, or, for the operator mode, `kubectl delete kates kates -n kates` and the Secret `kates-api-key`, which the `kates` chart then makes again with a new key. + +The two modes keep their run history in different databases: the `kates` chart's own PostgreSQL, or `kates-db`. A switch therefore starts with an empty history unless you carry it over. To keep it, dump the old database before you remove it and restore the dump into the new one, as [A Database That Survives a Node](12-deployment.md#a-database-that-survives-a-node) does for a move to CloudNativePG. + +## Limits + +The operator mode has these limits: + +- The number of members is fixed when the resource is created. The operator doesn't scale a quorum. +- The operator doesn't recover a quorum that lost its majority: you start it over, as [Losing the Majority](#losing-the-majority) shows. +- The operator doesn't support the INTEGRATION_CDC test yet. It has no counterpart to the chart's `rbac.cdcSecretNamespaces` or `rbac.extraRules`. +- The operator doesn't create a highly available PostgreSQL. The quorum protects the Kates API, not its database. +- The operator doesn't notice a changed database Secret. Delete the pods to pick it up. +- The operator mode needs cluster admin, so it doesn't work with `--adminless`. +- One `Kates` resource per namespace can run a quorum. + +## Summary + +- The standalone mode is the default and is unchanged: older installs, files and jobs keep working. +- The operator mode runs the Kates API as one member or a quorum, behind the same Service and key. +- A quorum keeps a leader through a lost member, and rebuilds a member whose node is gone. +- Draining before an upgrade lets live runs finish before any member is replaced. + +The operator keeps the Kates API standing, but it says nothing about whether `krafter` is the cluster you meant to build: [Compliance Checks](compliance.md) writes that architecture down as a file and holds `krafter` to it. diff --git a/docs/book/part-deployment-operations.md b/docs/book/part-deployment-operations.md index b38363abf..e3c4ead36 100644 --- a/docs/book/part-deployment-operations.md +++ b/docs/book/part-deployment-operations.md @@ -6,12 +6,13 @@ You don't read this Part straight through, and Parts II to IV don't need any of ## Build It -Read these when you stand up a cluster. The operator comes first, because without its CRDs a `Kafka` resource is inert. Installing Kafka is the step-by-step walkthrough, and Kafka Deployment Engineering gives the reason behind each of its numbers. The Deployment Guide then installs the Kates stack beside the cluster, and Compliance Checks holds what you built to the architecture you meant to build. +Read these when you stand up a cluster. The operator comes first, because without its CRDs a `Kafka` resource is inert. Installing Kafka is the step-by-step walkthrough, and Kafka Deployment Engineering gives the reason behind each of its numbers. The Deployment Guide then installs the Kates stack beside the cluster, standalone by default or under the Kates operator, which the next chapter explains, and Compliance Checks holds what you built to the architecture you meant to build. - [Deploying the Strimzi Operator](deploying-strimzi-operator.md): how do you install the operator that every Kafka cluster here depends on, take over one installed from the upstream chart, and upgrade or roll it back? About 30 minutes. - [Installing Kafka with the kafka-cluster Helm Chart](20-installation-guide.md): how do you deploy and verify a Kafka cluster with the chart, step by step? About 70 minutes. - [Kafka Deployment Engineering](15-kafka-deployment.md): why is the `krafter` Kafka cluster built the way it is, from node pools to listeners and certificates? About 25 minutes. - [Deployment Guide](12-deployment.md): how do you size and deploy the Kates stack, on Kind or on EKS, GKE or AKS? About 30 minutes. +- [Running Kates Under the Kates Operator](kates-operator.md): how do you keep the Kates API standing through a lost node or an upgrade, and what stays the same if you don't? About 20 minutes. - [Compliance Checks](compliance.md): does the cluster you run still match the architecture you meant to build? About 25 minutes. To practise the install, [Tutorial 8 — Deploy, Detect & Clean](../tutorials/08-deploy-and-detect.md) checks a cluster with `kates detect`, deploys the stack with the interactive `kates deploy -i`, and removes it with `kates clean`. diff --git a/docs/tutorials/01-getting-started.md b/docs/tutorials/01-getting-started.md index 2f47149e9..29a097dd3 100644 --- a/docs/tutorials/01-getting-started.md +++ b/docs/tutorials/01-getting-started.md @@ -29,6 +29,8 @@ kates deploy --topology isolated With no cluster reachable, `kates deploy` offers to create [`panda`](../book/appendix-a-glossary.md#gl-panda), a three-zone Kind cluster, and switches kubectl to it. Accept, and it then installs Strimzi, Kafka, monitoring, LitmusChaos and the Kates API. On a fresh machine this takes 5–10 minutes, most of it pulling images. When a cluster is already reachable, it deploys into kubectl's current context instead, so check `kubectl config current-context` first. +This installs the Kates API standalone, as one pod: the default, which every tutorial uses. To run it as a quorum that survives a lost node or an upgrade, `kates deploy --kates-mode operator --kates-replicas 3` installs it under the Kates operator instead, behind the same Service and API key; [Tutorial 16](16-running-kates-under-the-operator.md) walks through it. + `--topology isolated` puts each component in its own namespace (`kafka`, `kates`, `monitoring`, `litmus`), which the rest of this tutorial uses. `--topology single` puts the whole stack in one `kates-stack` namespace instead, which is fine for a scratch cluster but means every `-n ` below needs changing. > [!TIP] diff --git a/docs/tutorials/08-deploy-and-detect.md b/docs/tutorials/08-deploy-and-detect.md index 325b9d9b4..fe17d91b2 100644 --- a/docs/tutorials/08-deploy-and-detect.md +++ b/docs/tutorials/08-deploy-and-detect.md @@ -224,3 +224,4 @@ kates clean --yes - [Tutorial 1 — Getting Started](01-getting-started.md) — Run your first performance test - [Tutorial 3 — Chaos Engineering](03-chaos-engineering.md) — Inject faults during tests - [Tutorial 6 — CI/CD Integration](06-cicd-integration.md) — Add quality gates to your pipeline +- [Tutorial 16 — Running Kates Under the Kates Operator](16-running-kates-under-the-operator.md) — Answer **How should Kates be installed?** with Operator, and run the Kates API as a quorum diff --git a/docs/tutorials/16-running-kates-under-the-operator.md b/docs/tutorials/16-running-kates-under-the-operator.md new file mode 100644 index 000000000..4df45b463 --- /dev/null +++ b/docs/tutorials/16-running-kates-under-the-operator.md @@ -0,0 +1,285 @@ +# Tutorial 16: Running Kates Under the Kates Operator + +This tutorial runs the Kates API on `panda` as a quorum of three members under +the Kates operator. You then take its members away one at a time and watch it +keep answering. You will see the leader move, drain the API before an upgrade, +and rebuild a member whose node is unavailable, all through the same `kates` CLI +you used in Tutorial 1. + +**Level:** Intermediate · **Duration:** 30 min · **Prerequisites:** Tutorial 1 + +> [!NOTE] +> The standalone mode, which every other tutorial uses, is still the default +> and is unchanged: `kates deploy` with no new flag installs the `kates` +> chart as before. Nothing in this tutorial is needed to keep an existing +> install working. [Running Kates Under the Kates Operator](../book/kates-operator.md) +> in the book explains both modes in full. + +## What You'll Learn + +- Deploy Kates under the operator with three members +- Read the `Kates` resource, its leader and its conditions +- Use the same CLI, Service and API key as in the standalone mode +- Watch a new leader take over when the old one dies +- Drain the Kates API so an upgrade loses no live run +- Watch the operator rebuild a member whose node is unavailable + +## What You Will End Up With + +The operator runs three members, and the `kates` Service sends your traffic to +whichever one leads: + +```text + kates CLI ──► localhost:8080 ──► Service kates ──► the leader (kates.io/quorum-role=leader) + │ + kates-0 ◄──── Raft over mTLS ────┼────► kates-2 + (alpha) port 9300 │ (gamma) + kates-1 + (sigma) + │ + ▼ + PostgreSQL kates-db: runs, results, schedules +``` + +## Prerequisites + +You need what Tutorial 1 set up: the tools, the CLI and `panda`, the +three-node Kind cluster. You also need cluster admin, because the operator +brings a CRD and a ClusterRole, and `jq` for one step. + +If Kates is already installed standalone, remove only the Kates API: deploy +refuses to install one mode over the other, since both would claim the +Service `kates`. Kafka and the rest of the stack stay as they are. + +```bash +helm uninstall kates -n kates +``` + +The standalone run history is in the chart's own PostgreSQL, which that +command removes. To keep it, dump it first, as +[Switching Modes](../book/kates-operator.md#switching-modes) explains. + +## Step 1: Deploy Under the Operator + +Ask for the operator mode and three members. On a cluster where the rest of +the stack is already installed, deploy keeps those components, upgrading any +that are out of date, and adds what this mode needs: + +```bash +kates deploy --kates-mode operator --kates-replicas 3 +``` + +Deploy installs the Kates operator in the namespace `kates-operator`, a +PostgreSQL for Kates as the release `kates-db`, and the `Kates` resource +`kates`. It then waits until the operator reports the resource `Ready`, which +means three members are up and one of them leads. + +With `kates deploy -i`, the wizard asks the same thing after the components: +**How should Kates be installed?**, then the number of members. + +## Step 2: Read the Kates Resource + +Everything the operator knows is on the `Kates` resource: + +```bash +kubectl get kates -n kates +``` + +Output (on `panda`): + +```text +NAME VERSION REPLICAS LEADER READY MODE DRAINED AGE +kates 1.41.2-native 3 kates-1 True Active 7m11s +``` + +`LEADER` is the member that takes the writes. The pod label shows the same +thing, and `-o wide` shows that each member has a node of its own: + +```bash +kubectl get pods -n kates -L kates.io/quorum-role -o wide +``` + +Output (trimmed to the columns that matter here): + +```text +NAME READY STATUS NODE QUORUM-ROLE +kates-0 1/1 Running alpha +kates-1 1/1 Running sigma leader +kates-2 1/1 Running gamma +kates-db-0 1/1 Running gamma +``` + +Each member also reports the quorum in its health check: + +```bash +kubectl get --raw \ + /api/v1/namespaces/kates/services/http:kates:8080/proxy/api/health | jq .quorum +``` + +Output: + +```text +{ + "enabled": true, + "state": "RUNNING", + "memberId": "kates-1", + "leader": true, + "leaderId": "kates-1", + "term": 1, + "appliedIndex": 0 +} +``` + +The request went through the Service `kates`, so the leader answered it. On a +standalone install the same request shows `"enabled": false`. + +## Step 3: Use It as You Always Do + +The Service is still `kates` and the key is still in the Secret +`kates-api-key`, so the commands from Tutorial 1 work unchanged: + +```bash +kates ports +kates health +kates test create --type LOAD --records 100000 --wait +``` + +Nothing in your scenario files, scripts or CI jobs needs to know which mode +runs the Kates API. + +## Step 4: Kill the Leader + +Start a longer run in one terminal, so the leader has work live when it dies: + +```bash +kates test create --type LOAD --records 5000000 +``` + +In a second terminal, watch the resource: + +```bash +kubectl get kates -n kates -w +``` + +In a third, delete the leader's pod. Use the name `LEADER` showed, `kates-1` +here: + +```bash +kubectl delete pod -n kates kates-1 +``` + +What to look for: + +1. `LEADER` changes to another member within seconds, and `READY` may show + `False` with reason `NoLeader` while the election runs. +2. The Service follows the label `kates.io/quorum-role=leader`, which the + operator moves to the new leader within about five seconds. `kates health` + answers again through the same `localhost:8080`. +3. The run the old leader had live is reported failed: `kates test list` shows + it. The new leader fails it before taking any write, and removes any fault + it had injected, so nothing is left behind. +4. The deleted pod comes back as a follower, and `READY` returns to `True`. + +Run the same test again: it succeeds on the new leader. + +## Step 5: Drain Before an Upgrade + +An upgrade replaces the members one at a time, followers first and the +leader last. Under `mode: Drain` the operator first lets live work finish, so +no run is lost. You can try the drain without changing the version. Set the +mode: + +```bash +kubectl patch kates kates -n kates --type merge -p '{"spec":{"mode":"Drain"}}' +``` + +Then try to start a run: + +```bash +kates test create --type LOAD --records 100000 +``` + +The Kates API refuses it with `503`: under a drain, new runs, faults and +resilience runs are refused, schedules stop firing, and live work goes on to +its end. The `DRAINED` column turns `True` once nothing is live. For a real +upgrade you would set the new version in the same patch, +`{"spec":{"mode":"Drain","version":"-native"}}`, and wait for +`VERSION` to show it. + +The operator never lifts the drain itself. Lift it: + +```bash +kubectl patch kates kates -n kates --type merge -p '{"spec":{"mode":"Active"}}' +``` + +New runs are taken again. + +## Step 6: Lose a Node + +On `panda` each member's volume lives on its node, so a member whose node is +unavailable can't start anywhere else. Make the leader's node unschedulable, +then delete the leader's pod, so it can't come back there. Use the node and pod +from Step 2: + +```bash +kubectl cordon sigma +kubectl delete pod -n kates kates-1 +``` + +Watch the resource and the pods: + +```bash +kubectl get kates,pods -n kates -w +``` + +What to look for: + +1. A new leader is elected among the two members left, as in Step 4. The + quorum keeps serving, because two of three is still a majority. +2. The deleted member waits `Pending`, and `READY` shows `False` with reason + `MemberLost`. +3. About two minutes later the operator deletes that member's volume claim and + pod. The StatefulSet makes both again on another node. +4. The new member starts with an empty log, catches up and votes again, and + `READY` returns to `True`. Nothing was lost: the quorum holds only who + leads, and every run and result is in PostgreSQL. + +Make the node schedulable again: + +```bash +kubectl uncordon sigma +``` + +The rebuilt member stays on the node it moved to until its volume is +replaced again. + +## Step 7: Go Back to Standalone, or Clean Up + +To go back to the standalone mode, delete the `Kates` resource. The operator +removes the members, their Services and their volumes. Also delete the API key +Secret, which deploy made for the operator mode and which the `kates` chart +makes again with a new key. Then deploy without the flag, and run +`kates ports` again so the CLI context picks up the new key: + +```bash +kubectl delete kates kates -n kates +kubectl delete secret kates-api-key -n kates +kates deploy +kates ports +``` + +The operator and `kates-db` stay installed and unused. Remove them with +`helm uninstall kates-operator -n kates-operator` and +`helm uninstall kates-db -n kates`. To remove the whole stack instead, +`kates clean` handles either mode: it deletes the `Kates` resource while the +operator still runs, then the operator. + +## What's Next? + +- [Running Kates Under the Kates Operator](../book/kates-operator.md): the + conditions, losing the majority, installing with Helm, and the limits. +- [The operator chart's README](../../operator/chart/README.md): its RBAC modes + and every field of the `Kates` resource. +- [Tutorial 3: Chaos Engineering with Kates](03-chaos-engineering.md): runs + faults against Kafka. Under the operator, a fault a dead leader injected is + removed by the next leader. diff --git a/docs/tutorials/README.md b/docs/tutorials/README.md index 9048644f9..f59a57926 100644 --- a/docs/tutorials/README.md +++ b/docs/tutorials/README.md @@ -24,6 +24,7 @@ The table gives each tutorial's level, a rough duration, and the tutorials it bu | 13 | [Using the Grafana Dashboards](13-using-the-dashboards.md) | Intermediate | 45 min | Tutorial 1 | | 14 | [Using Kates from an AI Agent](14-using-kates-from-an-ai-agent.md) | Intermediate | 20 min | Tutorial 1 | | 15 | [Authoring & Debugging YAML Suites in VS Code](15-ide-authoring-and-lsp.md) | Intermediate | 20 min | Tutorial 1 | +| 16 | [Running Kates Under the Kates Operator](16-running-kates-under-the-operator.md) | Intermediate | 30 min | Tutorial 1 | Tutorials 10–12 rehearse a migration on Kind, where nothing costs anything if it goes wrong. When you are ready to migrate a real cluster, the chart's own @@ -40,7 +41,7 @@ Tutorial 1 (Getting Started) ├── Tutorial 5 (Observability) ──── Tutorial 13 (Grafana Dashboards) ├── Tutorial 6 (CI/CD) ← requires Tutorial 2 ├── Tutorial 7 (Security) - ├── Tutorial 8 (Deploy & Detect) + ├── Tutorial 8 (Deploy & Detect) ──── Tutorial 16 (Kates operator: quorum, drain, lost node) ├── Tutorial 9 (Kafka Connect) ├── Tutorial 14 (AI agent over MCP, read-only) ├── Tutorial 15 (IDE authoring & kates lsp) @@ -51,7 +52,9 @@ Tutorial 1 (Getting Started) - **Performance track:** Tutorials 1 → 2 → 5 — learn test types, then understand the metrics. - **Observability track:** Tutorials 1 → 5 → 13 — the CLI's own views of a run first, then the twelve Grafana boards that watch the cluster around it. - **Chaos track:** Tutorials 1 → 2 → 3 → 4 — build up to chaos engineering and data integrity. -- **Operations track:** Tutorials 1 → 8 → 7 — deployment, security, and lifecycle management. +- **Operations track:** Tutorials 1 → 8 → 7 → 16 — deployment, security, lifecycle management, then the Kates API as a quorum under the Kates operator. + +Every tutorial runs against the default standalone install. Tutorial 16 switches to the operator mode, and its last step switches back; the commands of the other tutorials are the same in both modes. ## Prerequisites