From f0005469b7fb2d77822fbfd2169621e7342a85d5 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Fri, 17 Apr 2026 16:00:35 +1200 Subject: [PATCH 01/23] 100: Add sidecar injection webhook proposal Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 444 +++++++++++++++++++++ 1 file changed, 444 insertions(+) create mode 100644 proposals/100-sidecar-injection-webhook.md diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md new file mode 100644 index 0000000..51fe2a4 --- /dev/null +++ b/proposals/100-sidecar-injection-webhook.md @@ -0,0 +1,444 @@ +# Proposal 016 — Sidecar Injection Webhook + +A Kubernetes mutating admission webhook that automatically injects a Kroxylicious proxy sidecar into application pods. The sidecar intercepts Kafka traffic on localhost, allowing filters to be applied transparently without changes to the application. + +## Current situation + +Kroxylicious is deployed either standalone or via the operator as a shared proxy tier, fronting one or more Kafka clusters with ingress networking. Applications connect to the proxy over the network. + +There is no mechanism for running Kroxylicious as a per-pod sidecar. Users who want per-pod proxying must manually construct the sidecar container spec, generate proxy configuration, and manage the lifecycle themselves. + +The proxy already has properties that make it suitable for sidecar use: it can bind to localhost, and runs as non-root with no special capabilities. + +## Motivation + +A sidecar model is useful when: + +- The application should connect to Kafka via `localhost` rather than through a shared proxy tier. +- Per-pod filter configuration is needed (e.g. different encryption keys per tenant). +- The organisation prefers a service mesh-style deployment where each pod carries its own proxy. + +Manual sidecar construction is error-prone and creates a maintenance burden. A webhook automates injection, enforces a consistent security posture, and gives the webhook administrator control over what runs in the sidecar. + +## Proposal + +### Trust model + +The webhook operates under a strict two-party trust model: + +- **Webhook administrator**: controls what gets injected — the proxy image, upstream Kafka address, filter definitions, security context. These are never overridable by the app owner. +- **Application pod owner**: can opt out of injection, and may override specific settings (bootstrap port, node ID range, resource requests) if the admin explicitly delegates those annotations. + +The webhook always overwrites the `kroxylicious.io/proxy-config` annotation on the pod, regardless of any value the app owner may have set. Non-delegated annotations in the `kroxylicious.io/` namespace are silently ignored with a logged warning. + +### Injection decision + +Injection is opt-in at the namespace level and opt-out at the pod level, following the Istio/Linkerd convention: + +| Mechanism | Key | Effect | +|-----------|-----|--------| +| Namespace label | `kroxylicious.io/sidecar-injection: enabled` | Webhook intercepts pod creates in this namespace | +| Pod label | `kroxylicious.io/inject-sidecar: "false"` | Pod is excluded via `objectSelector` — never reaches the webhook | + +The `MutatingWebhookConfiguration` uses `namespaceSelector` to scope interception and `objectSelector` to exclude opted-out pods. The webhook itself is idempotent: if a container named `kroxylicious-proxy` already exists, injection is skipped. + +The webhook always returns `allowed: true`, even on internal errors. Errors are logged; the pod is admitted unmodified. This fail-open policy (`failurePolicy: Ignore`) ensures a broken webhook never blocks workloads. + +### CRD: `KroxyliciousSidecarConfig` + +A namespaced CRD (group `kroxylicious.io`, version `v1alpha1`) defines the sidecar configuration. The webhook admin creates one per namespace. The following edge cases are handled: + +1. **No config in namespace**: the pod is admitted without injection (debug log only). This is the common case for namespaces where the admin has enabled the namespace label but not yet created a config. +2. **Multiple configs in namespace**: the pod is admitted without injection (warning logged). The pod can select a specific config via the `kroxylicious.io/sidecar-config` annotation; without this annotation the webhook cannot choose and skips injection. +3. **Config is invalid in a way the webhook can detect** (e.g. malformed delegated annotation values, plugin image without a digest): the webhook logs a warning and admits the pod without injection. Consistent with fail-open semantics. +4. **Config is invalid in a way only the proxy can detect** (e.g. unreachable upstream Kafka, wrong TLS trust anchor, non-existent filter type): the webhook injects the sidecar normally. The proxy will fail its startup probe and the pod will not become ready, surfacing the problem via standard Kubernetes health-check mechanisms. + +```yaml +apiVersion: kroxylicious.io/v1alpha1 +kind: KroxyliciousSidecarConfig +metadata: + name: my-config +spec: + upstreamBootstrapServers: kafka-prod.internal:9092 + bootstrapPort: 19092 # default, configurable + nodeIdRange: + startInclusive: 0 + endInclusive: 2 + managementPort: 9190 + proxyImage: quay.io/kroxylicious/proxy:0.21.0 # optional override + setBootstrapEnvVar: true # sets KAFKA_BOOTSTRAP_SERVERS on app containers + filterDefinitions: + - name: my-filter + type: io.example.MyFilterFactory + config: { ... } + upstreamTls: + trustAnchorSecretRef: + name: kafka-ca + key: ca.crt + plugins: + - name: my-plugin + image: + reference: registry.example.com/my-filter:v1.0@sha256:abc123 + pullPolicy: IfNotPresent + delegatedAnnotations: + - kroxylicious.io/sidecar-bootstrap-port + - kroxylicious.io/sidecar-node-id-range +``` + +**Why a CRD, not a ConfigMap?** Schema validation by the API server, RBAC separation (admin creates, app owners can't modify), status conditions for observability, consistency with the existing Kroxylicious Kubernetes API. + +**Why not reuse the operator's CRDs?** The operator CRDs model a shared proxy deployment with ingress networking, multi-cluster support, and cross-resource references. The sidecar use case is fundamentally simpler — one virtual cluster, localhost binding, no ingress. Coupling them would constrain both models. + +### Config injection + +The webhook generates proxy configuration YAML from the `KroxyliciousSidecarConfig` spec, using the same `Configuration` model from `kroxylicious-runtime`. The generated config is stored in a pod annotation (`kroxylicious.io/proxy-config`) and projected into the sidecar container via a `downwardAPI` volume. + +This avoids creating per-pod ConfigMaps, which would require additional RBAC, lifecycle management for orphaned ConfigMaps, and unique name generation. The annotation approach is self-contained within the pod. + +A typical sidecar config is a few hundred bytes, well within the ~256KB practical annotation size limit. + +### Port allocation + +| Port | Purpose | Bind address | +|------|---------|-------------| +| 19092 | Kafka bootstrap | `localhost` | +| 19093+ | Per-broker ports (one per node ID) | `localhost` | +| 9190 | Management (`/livez`, `/metrics`) | `0.0.0.0` | + +Bootstrap defaults to 19092 rather than 9092 to avoid clashing with Kafka client libraries' default port. The webhook sets `KAFKA_BOOTSTRAP_SERVERS=localhost:19092` on application containers (configurable, can be disabled). + +The management endpoint binds to `0.0.0.0` because kubelet HTTP probes target the pod IP, not loopback. This means the application container can also reach `/livez` and `/metrics`, but neither endpoint exposes sensitive data. + +### Native sidecar containers + +On Kubernetes 1.28+, the webhook injects the proxy as a native sidecar — an init container with `restartPolicy: Always`. This gives proper startup ordering (proxy starts before the application) and shutdown ordering (proxy stops after the application). On older clusters, the webhook falls back to injecting into `spec.containers`. + +The webhook detects the cluster's Kubernetes version at startup and chooses the injection strategy accordingly. + +### Sidecar container spec + +The injected sidecar follows the same patterns as `ProxyDeploymentDependentResource` in the operator: + +- `securityContext`: `allowPrivilegeEscalation: false`, `capabilities: drop ALL`, `readOnlyRootFilesystem: true` +- Probes: `startupProbe` (30 x 2s), `livenessProbe`, `readinessProbe` — all HTTP GET `/livez` on port 9190 +- `terminationMessagePolicy: FallbackToLogsOnError` + +The security context is never weakened. If the pod already has a stricter security context, it is preserved. + +### Upstream TLS + +When `spec.upstreamTls.trustAnchorSecretRef` is set, the webhook adds a volume mounting the referenced Secret into the sidecar and configures the proxy to use it as a PEM trust store. The Secret must exist in the pod's namespace. + +### Delegated annotations + +The `delegatedAnnotations` field in `KroxyliciousSidecarConfig` lists which annotations the app owner may set to override sidecar parameters: + +| Annotation | Effect | +|-----------|--------| +| `kroxylicious.io/sidecar-bootstrap-port` | Override bootstrap port | +| `kroxylicious.io/sidecar-node-id-range` | Override node ID range (e.g. `"0-5"`) | +| `kroxylicious.io/sidecar-resources-cpu` | Override CPU request/limit | +| `kroxylicious.io/sidecar-resources-memory` | Override memory request/limit | +| `kroxylicious.io/sidecar-plugin-images` | Additional plugin images (JSON array) | + +Delegation is opt-in per annotation. By default nothing is delegated. Upstream Kafka address, filter definitions, proxy image, and security context are never delegatable. + +### Configuration drift detection + +The webhook stamps each injected pod with metadata annotations: + +| Annotation | Value | +|-----------|-------| +| `kroxylicious.io/config-generation` | The `metadata.generation` of the `KroxyliciousSidecarConfig` at injection time | +| `kroxylicious.io/injection-timestamp` | ISO-8601 timestamp of when the sidecar was injected | + +Because the webhook only mutates pods at creation time, configuration changes to `KroxyliciousSidecarConfig` do not propagate to running pods. This matches how Istio and Linkerd handle sidecar injection. Users must restart pods to pick up new configuration. + +The generation stamp allows operators to identify stale pods: + +``` +kubectl get pods -n my-ns -o json | jq '[.items[] | + select(.metadata.annotations["kroxylicious.io/config-generation"] != null) | + {name: .metadata.name, generation: .metadata.annotations["kroxylicious.io/config-generation"]}]' +``` + +In a future iteration, a reconciler could watch for pods with outdated generations and surface an `UpToDate` condition on the `KroxyliciousSidecarConfig` status, giving operators visibility into configuration drift without requiring manual queries. + +### Third-party plugin support + +#### The problem + +Users will want to run third-party Kroxylicious plugins (custom filters, KMS providers) in the sidecar. The proxy discovers plugins via `ServiceLoader` from the classpath. Built-in plugins live in `libs/`. Third-party plugin JARs must be delivered separately. + +OCI image volumes (KEP-4639) allow mounting an OCI image as a read-only volume in a pod. This is the cleanest delivery mechanism: plugin authors package their JARs in a `FROM scratch` image, and the webhook mounts it into the sidecar at a known path. + +#### Solution + +The proxy startup script already scans `/opt/kroxylicious/classpath-plugins/*/` for subdirectories containing JARs and adds them to the classpath. The webhook mounts each plugin's OCI image at `/opt/kroxylicious/classpath-plugins//`. + +For each plugin in `spec.plugins`, the webhook adds: + +1. An OCI image volume referencing the plugin image. +2. A read-only volume mount on the sidecar container. + +```yaml +volumes: + - name: plugin-my-filter + image: + reference: registry.example.com/my-filter:v1.0@sha256:abc123 + pullPolicy: IfNotPresent +``` + +```yaml +volumeMounts: + - name: plugin-my-filter + mountPath: /opt/kroxylicious/classpath-plugins/my-filter + readOnly: true +``` + +ServiceLoader discovers the plugin implementations from the combined classpath. Multiple plugin images can be mounted simultaneously, each at its own subdirectory. + +#### Plugin image convention + +Plugin images should be built `FROM scratch` with JARs at the image root: + +```dockerfile +FROM scratch +COPY target/my-filter.jar /my-filter.jar +COPY target/dependency/*.jar / +``` + +#### Flat classpath limitations + +All plugin JARs share the proxy's flat classpath. There is no classloader isolation. If two plugins bundle different versions of the same library, the one the classloader finds first wins — silently, without error. + +Jackson is the concrete concern. The proxy ships Jackson and uses it for filter config deserialization. A plugin bundling an incompatible Jackson version could cause silent serialization differences. Other proxy-provided libraries (Netty, Kafka clients, SLF4J, Micrometer) carry the same risk. + +**Mitigations:** + +- **Document the constraint**: plugin images should not bundle libraries the proxy already provides. Plugin authors should treat the proxy's dependencies as `provided` scope. Publishing the proxy's transitive dependency closure as a Maven BOM would make this mechanical. +- **Shade transitive dependencies**: plugin authors should shade (relocate) any transitive dependency that might conflict. + +Classloader isolation (a classloader per plugin directory, similar to what application servers do) would eliminate this problem but is a significant architectural change. It should be treated as a known future requirement, not a hypothetical. + +#### Kubernetes version requirements + +| Feature | K8s version | OpenShift version | Status | +|---------|-------------|-------------------|--------| +| OCI image volumes (alpha) | 1.31+ | 4.18+ | Feature gate `ImageVolume` must be enabled | +| OCI image volumes (beta) | 1.33+ | 4.20+ | Feature gate `ImageVolume` must be enabled | +| OCI image volumes (default on) | 1.35+ | 4.22+ | Enabled by default | + +**Container runtime support**: OpenShift uses CRI-O exclusively, which supports OCI image volumes from v1.31+ (matching the Kubernetes version). containerd support is maturing (alpha in v2.1.0) but is not relevant to OpenShift deployments. For non-OpenShift clusters using containerd, the init-container fallback (below) is the practical path until containerd support stabilises. + +#### Init-container fallback + +For clusters without OCI image volume support, the webhook supports an init-container fallback: + +1. An init container per plugin image copies JARs to an `emptyDir` volume. +2. The `emptyDir` is mounted at `/opt/kroxylicious/classpath-plugins/` on the sidecar with `readOnly: true`. + +This works on any Kubernetes version but adds startup latency and uses writable storage (though the mount itself is read-only on the sidecar). + +The webhook auto-detects OCI image volume support via the Kubernetes API server version. + +#### Security analysis: admin-controlled plugin images + +When the admin specifies plugin images in `KroxyliciousSidecarConfig.spec.plugins`: + +- The admin trusts the plugin image publisher. +- The app owner has no control over which images are mounted. +- OCI image volumes are read-only and `noexec` by design. +- Plugin JARs run on the proxy's classpath with the proxy's JVM permissions (non-root, no capabilities, read-only root filesystem). + +**Remaining risks:** + +- **Supply chain**: a compromised plugin image contains malicious code with access to Kafka traffic and mounted credentials. Mitigate with image signing and digest-pinned references. +- **Dependency conflicts**: as described above under flat classpath limitations. + +#### Security analysis: delegated plugin image selection + +If the admin delegates plugin image selection to app owners (via the `kroxylicious.io/sidecar-plugin-images` annotation), the risks escalate: + +| Risk | Severity | Description | +|------|----------|-------------| +| Arbitrary code on proxy classpath | Critical | App owner specifies an image containing malicious JARs. The proxy has access to upstream Kafka credentials, TLS certs, and all Kafka traffic. | +| Registry credential leakage | High | OCI image volumes reuse the pod's `imagePullSecrets` and node-level credentials. An attacker-controlled registry receives pull requests carrying bearer tokens. | +| Image tag mutation | High | A tag (not a digest) can be replaced with malicious content between pulls. | +| Resource exhaustion | Medium | Large OCI images consume node disk. No per-volume size limits exist. | + +**Mitigations:** + +1. **Registry allow-list**: `allowedPluginRegistries` in `KroxyliciousSidecarConfig`. The webhook rejects delegated image references not matching an allowed prefix. +2. **Require digest pinning**: delegated image references without `@sha256:` are rejected. +3. **Audit logging**: all plugin image references are logged at INFO, with warnings for delegated images. + +Even with these mitigations, the fundamental trust grant is unchanged: a digest-pinned image from an allowed registry still runs arbitrary code in the proxy JVM. The mitigations reduce supply-chain risk but do not constrain what the code does once loaded. Delegating plugin image selection is equivalent to allowing the app owner to run arbitrary code with the proxy's identity. + +**Delegation is disabled by default.** When enabled, both `allowedPluginRegistries` and digest pinning are enforced. + +#### Future direction: PluginRegistry CRD + +A cleaner model for pre-approved plugins would be a cluster-scoped `PluginRegistry` CRD where the admin defines approved plugin images with namespace-level scoping: + +```yaml +apiVersion: kroxylicious.io/v1alpha1 +kind: PluginRegistry +metadata: + name: approved-filters +spec: + plugins: + - name: record-encryption + image: quay.io/kroxylicious/record-encryption@sha256:abc123 + allowedNamespaces: ["prod-*", "staging"] +``` + +This separates the "which plugins are trusted" question from the per-namespace sidecar config, avoids JSON-in-annotation, and makes the approval surface auditable via standard Kubernetes RBAC. It is a better long-term model than annotation-based delegation, but is out of scope for the initial implementation. + +### Upstream cluster selection + +In many deployments the admin manages multiple Kafka clusters (e.g. production, staging) and the app owner needs to choose which one their pod connects to. Rather than creating a separate `KroxyliciousSidecarConfig` per cluster, the admin defines an allow-list of named upstream clusters: + +```yaml +spec: + allowedUpstreamClusters: + - name: production + bootstrapServers: kafka-prod.internal:9092 + - name: staging + bootstrapServers: kafka-staging.internal:9092 +``` + +The app owner selects a cluster by annotation: + +```yaml +kroxylicious.io/sidecar-upstream-cluster: staging +``` + +The admin retains control over which clusters are reachable. The app owner cannot specify an arbitrary bootstrap address — only names from the allow-list are accepted. If the annotation names a cluster not in the list, or is absent when multiple clusters are defined, injection is skipped with a warning. + +When `allowedUpstreamClusters` is not set, the existing `upstreamBootstrapServers` field is used directly and there is no cluster selection. + +This is the lowest-risk form of delegation — the app owner chooses a network destination from an admin-controlled set — and is likely the highest-demand delegation feature for app teams. It is included in the initial implementation. + +### Future delegation + +The delegated annotations mechanism (bootstrap port, node ID range, resource requests, plugin images) described above provides a general-purpose extension point for further delegation. These are ordered roughly by blast radius: + +1. **Port and resource overrides** — app owner adjusts operational parameters. Low risk. +2. **Filter configuration** — app owner adjusts parameters on admin-selected filters. Medium risk: bounded by the filter's config surface. +3. **Plugin image selection** — app owner chooses what code runs in the proxy JVM. High risk: arbitrary code execution (see security analysis above). + +All delegation beyond upstream cluster selection requires the admin to explicitly list the delegated annotations. Nothing is delegated by default. + +### Bypass prevention + +When the proxy is used as a policy enforcement point (e.g. record-level encryption, audit logging), applications that bypass the sidecar bypass the policy. This will be flagged in any compliance audit. The webhook needs to support configurations that make bypass difficult. + +#### Baseline: `KAFKA_BOOTSTRAP_SERVERS` + +The webhook sets `KAFKA_BOOTSTRAP_SERVERS=localhost:19092` on application containers. This is sufficient to prevent *accidental* bypass — the application connects to the sidecar by default. Bypassing requires deliberately hardcoding the upstream Kafka address. + +#### Defence-in-depth: NetworkPolicy + upstream authentication + +A stronger enforcement model combines two mechanisms: + +1. **NetworkPolicy restricting pod egress** — the admin creates a `NetworkPolicy` in the application namespace that limits egress to only the upstream Kafka cluster IPs/ports defined in `allowedUpstreamClusters`. This prevents the application from connecting to arbitrary external services. Because NetworkPolicy operates at the pod level, both the sidecar and the application container are subject to the same egress rules — but this is acceptable, because the sidecar only needs to reach the allowed clusters. + +2. **Upstream Kafka authentication** — the upstream Kafka cluster requires authentication (mTLS client certificates or SASL credentials). The sidecar holds the credentials; the application does not. Even if the application connects directly to the Kafka broker's address, the broker rejects the unauthenticated connection. + +Together these mean the application can only reach the allowed Kafka clusters (NetworkPolicy) and cannot authenticate to them without going through the sidecar. + +This model requires: + +- The upstream Kafka cluster enforces authentication (not optional/unauthenticated). +- The CRD supports upstream client authentication configuration. The current `upstreamTls` field handles server certificate validation; client certificate or SASL credential references would need to be added. + +#### Credential isolation limits + +Mounting the credential Secret only into the sidecar container is not a hard security boundary. An app owner with standard namespace-level RBAC can extract the credentials through several paths: + +- **Secret read access**: app owners typically have `get` on Secrets in their namespace (needed for their own application secrets). They can `kubectl get secret -o yaml`. +- **Container exec**: `kubectl exec -c kroxylicious-proxy` gives access to the mounted credential files. +- **Predictable volume names**: if the app owner knows the volume name the webhook will create, they can add a `volumeMount` in their own container spec referencing it. The webhook adds the volume before API server validation, so this passes. + +Container filesystem isolation raises the bar but does not create a trust boundary against a determined app owner. The real enforcement boundary is RBAC. + +**Mitigations:** + +- **Restrict Secret access by name**: the webhook uses a consistent naming convention for credential Secrets (e.g. `kroxylicious-upstream-*`). The admin can write RBAC rules that grant the app owner `get` on Secrets *except* those matching this pattern. Kubernetes RBAC supports `resourceNames` on deny, though operationally this means enumerating allowed Secrets rather than denying specific ones. +- **Restrict exec into the sidecar**: a `ValidatingAdmissionPolicy` can deny `pods/exec` subresource requests targeting the `kroxylicious-proxy` container. +- **Short-lived credentials**: if the upstream Kafka cluster supports OAuth/OIDC token-based authentication, the sidecar can use a projected ServiceAccount token with a short lifetime. Extracting a token is still possible but the window of use is limited. + +None of these fully close the gap. Within a single pod, Kubernetes does not offer strong credential isolation between containers. This is a fundamental platform limitation, not specific to this design. + +For deployments where policy bypass is an audit-critical concern, the strongest posture is: NetworkPolicy restricting egress + upstream Kafka requiring authentication + RBAC preventing app owners from reading proxy credential Secrets or exec-ing into the sidecar container. This is operationally achievable but requires deliberate RBAC design by the cluster admin. + +#### Alternative: iptables redirection + +The Istio model — an init container with `NET_ADMIN` that sets up iptables rules to redirect Kafka-port traffic to the sidecar, excluding the proxy process by UID — would enforce bypass prevention at the network level without requiring credential isolation at all. However, it requires granting `NET_ADMIN` to the init container, conflicting with the security posture of dropping all capabilities. + +### Webhook deployment + +The webhook is packaged as a container image and deployed as a single-replica `Deployment` in a dedicated `kroxylicious-webhook` namespace. Install manifests are provided for: + +- Namespace, ServiceAccount, ClusterRole, ClusterRoleBinding +- Deployment (port 8443) +- Service (port 443 -> 8443) +- MutatingWebhookConfiguration +- cert-manager Certificate (optional) + +**TLS**: Kubernetes requires HTTPS for admission webhooks. The primary path uses cert-manager with a self-signed issuer. A manual alternative (admin provides cert/key Secret) is documented. The webhook watches cert files for rotation and reloads the SSLContext. + +**RBAC**: The webhook needs only `get`, `list`, `watch` on `KroxyliciousSidecarConfig` resources and `get`, `list`, `watch` on namespaces. No ConfigMap or Secret creation permissions are needed. + +**HTTP server**: Uses the JDK built-in `HttpsServer` (same pattern as `OperatorMain.java`), serving `POST /mutate` and `GET /livez`. No additional HTTP framework dependencies. + +### Independence from the operator + +The webhook operates independently of the operator. It does not depend on the operator being deployed, does not use JOSDK, and does not reference operator CRDs. + +The only shared dependencies are: + +- `kroxylicious-kubernetes-api` — for the CRD Java types +- `kroxylicious-runtime` — for the proxy `Configuration` model, used to generate valid proxy config YAML + +## Affected/not affected projects + +| Project | Affected | Nature of change | +|---------|----------|-----------------| +| `kroxylicious-kubernetes-web-hook` | Yes | New module | +| `kroxylicious-kubernetes-api` | Yes | New CRD: `KroxyliciousSidecarConfig` | +| `kroxylicious-app` | Already merged | `classpath-plugins/` directory scanning | +| `kroxylicious-operator` | No | | +| `kroxylicious-runtime` | No | Used as a dependency, not modified | +| `kroxylicious-filters` | No | | + +## Compatibility + +This is a new feature with no backwards compatibility concerns. + +The `KroxyliciousSidecarConfig` CRD uses `v1alpha1`, signalling that the API may change without notice in future releases. + +The webhook can be deployed alongside the operator without conflict — they watch different CRDs and do not interact. + +## Rejected alternatives + +### ConfigMap instead of CRD for sidecar configuration + +A ConfigMap is simpler to create but lacks schema validation, gives no status reporting, and cannot be distinguished from other ConfigMaps by RBAC policy. The CRD provides all of these and is consistent with the project's existing Kubernetes API patterns. + +### Per-pod ConfigMap for proxy configuration + +Creating a ConfigMap per pod avoids the annotation size limit but introduces lifecycle management (orphaned ConfigMaps), requires `create`/`delete` RBAC for the webhook, and requires unique name generation. The annotation + downwardAPI approach is self-contained. + +### Reuse of operator CRDs (`KafkaProxy`, `VirtualKafkaCluster`) + +The operator CRDs model multi-cluster, multi-ingress proxy deployments. The sidecar use case is a single virtual cluster on localhost. Coupling them would constrain both APIs and prevent deploying the webhook independently of the operator. + +### Classloader-per-plugin isolation + +A custom classloader per plugin directory would eliminate dependency conflicts between plugins. This is architecturally significant (the proxy currently assumes a flat classpath via `ServiceLoader.load()`) and is deferred as a future enhancement. The flat classpath with documented constraints is the right tactical choice for the initial implementation. + +### `KROXYLICIOUS_CLASSPATH` environment variable for plugins + +The proxy already supports a `KROXYLICIOUS_CLASSPATH` env var. However, this is a single classpath string and cannot accommodate multiple independently-mounted plugin directories. The `classpath-plugins/` subdirectory scanning is more natural for volume-per-plugin mounting. From 646141241f0effb93a32e08d8b4da6395fd190f0 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 27 Apr 2026 16:30:50 +1200 Subject: [PATCH 02/23] Review comments + water-down trust model commitments Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 139 ++++++--------------- 1 file changed, 35 insertions(+), 104 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 51fe2a4..588c235 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -24,12 +24,19 @@ Manual sidecar construction is error-prone and creates a maintenance burden. A w ### Trust model -The webhook operates under a strict two-party trust model: +The webhook is design to eventually operate under a strict two-party trust model: - **Webhook administrator**: controls what gets injected — the proxy image, upstream Kafka address, filter definitions, security context. These are never overridable by the app owner. - **Application pod owner**: can opt out of injection, and may override specific settings (bootstrap port, node ID range, resource requests) if the admin explicitly delegates those annotations. -The webhook always overwrites the `kroxylicious.io/proxy-config` annotation on the pod, regardless of any value the app owner may have set. Non-delegated annotations in the `kroxylicious.io/` namespace are silently ignored with a logged warning. +Making the boundary between the webhook administrator and the application pod owner a _reliable_ trust boundary will require further development than is specified in this proposal. +However pod annotations in the `kroxylicious.io/` namespace form the basic building blocks for a flexible but reliable trust boundary. +* Some annotations are always set by the webhook. +For example, the proxy obtains its configuration via a `kroxylicious.io/proxy-config` pod annotation which is projected into the sidecar container via a `downwardAPI` volume. +The webhook always overwrites that annotation (`kroxylicious.io/proxy-config`) on the pod, regardless of any value the app owner may have set. +* The administrator can delegate some annotations to be specifed by/overridden the application owner. When specified by the application owner they will not be overwritten by the webhook. Examples defined in this proposal are `kroxylicious.io/sidecar-resources-cpu` and `kroxylicious.io/sidecar-resources-memory` (see below). +* Annotations which the administrator has **not** delegated will have their effect overridden by the webhook based on the Administrator-controlled `KroxyliciousSidecarConfig` resource. A warning will be logged when such overriding is necessary. + ### Injection decision @@ -42,7 +49,17 @@ Injection is opt-in at the namespace level and opt-out at the pod level, followi The `MutatingWebhookConfiguration` uses `namespaceSelector` to scope interception and `objectSelector` to exclude opted-out pods. The webhook itself is idempotent: if a container named `kroxylicious-proxy` already exists, injection is skipped. -The webhook always returns `allowed: true`, even on internal errors. Errors are logged; the pod is admitted unmodified. This fail-open policy (`failurePolicy: Ignore`) ensures a broken webhook never blocks workloads. +The failure policy of the webhook will be configurable. +It will default to fail closed (`failurePolicy: Fail`), which is safe, but sacrifices availability of the Kubernetes control plane to admit workloads in cases where the webhook experiences internal errors. +When configured to fail open and the webhook experiences an internal errors, it will log the error and return `allowed: true`; the pod will be admitted unmodified. + +#### Bypass prevention + +The webhook sets `KAFKA_BOOTSTRAP_SERVERS` to point at the sidecar, but nothing prevents an application from connecting directly to the upstream Kafka cluster. Kubernetes `NetworkPolicy` cannot help here: it operates at the pod level, so a policy blocking egress to Kafka would also block the sidecar's upstream connection. + +The Istio model — an init container with `NET_ADMIN` that sets up iptables rules to redirect Kafka-port traffic to the sidecar, excluding the proxy process by UID — would enforce this, but requires granting `NET_ADMIN` to the init container, conflicting with the security posture of dropping all capabilities. + +In practice, bypassing the sidecar requires the application to deliberately hardcode the real Kafka address. An app owner determined to bypass can also opt out of injection entirely via pod labels. The enforcement boundary is RBAC on who can create pods in the namespace, not network controls within the pod. If the threat model requires enforcement against a hostile app owner, iptables redirection could be added as an opt-in capability in a future iteration. ### CRD: `KroxyliciousSidecarConfig` @@ -60,11 +77,11 @@ metadata: name: my-config spec: upstreamBootstrapServers: kafka-prod.internal:9092 - bootstrapPort: 19092 # default, configurable + bootstrapPort: 9092 # default, configurable nodeIdRange: startInclusive: 0 endInclusive: 2 - managementPort: 9190 + managementPort: 9082 # default, configurable proxyImage: quay.io/kroxylicious/proxy:0.21.0 # optional override setBootstrapEnvVar: true # sets KAFKA_BOOTSTRAP_SERVERS on app containers filterDefinitions: @@ -81,8 +98,8 @@ spec: reference: registry.example.com/my-filter:v1.0@sha256:abc123 pullPolicy: IfNotPresent delegatedAnnotations: - - kroxylicious.io/sidecar-bootstrap-port - - kroxylicious.io/sidecar-node-id-range + - kroxylicious.io/sidecar-resources-cpu + - kroxylicious.io/sidecar-resources-memory ``` **Why a CRD, not a ConfigMap?** Schema validation by the API server, RBAC separation (admin creates, app owners can't modify), status conditions for observability, consistency with the existing Kroxylicious Kubernetes API. @@ -101,11 +118,11 @@ A typical sidecar config is a few hundred bytes, well within the ~256KB practica | Port | Purpose | Bind address | |------|---------|-------------| -| 19092 | Kafka bootstrap | `localhost` | -| 19093+ | Per-broker ports (one per node ID) | `localhost` | -| 9190 | Management (`/livez`, `/metrics`) | `0.0.0.0` | +| 9092 | Kafka bootstrap | `localhost` | +| 9093+ | Per-broker ports (one per node ID) | `localhost` | +| 9082 | Management (`/livez`, `/metrics`) | `0.0.0.0` | -Bootstrap defaults to 19092 rather than 9092 to avoid clashing with Kafka client libraries' default port. The webhook sets `KAFKA_BOOTSTRAP_SERVERS=localhost:19092` on application containers (configurable, can be disabled). +The webhook sets `KAFKA_BOOTSTRAP_SERVERS=localhost:9092` on application containers (configurable, can be disabled). The management endpoint binds to `0.0.0.0` because kubelet HTTP probes target the pod IP, not loopback. This means the application container can also reach `/livez` and `/metrics`, but neither endpoint exposes sensitive data. @@ -131,17 +148,15 @@ When `spec.upstreamTls.trustAnchorSecretRef` is set, the webhook adds a volume m ### Delegated annotations -The `delegatedAnnotations` field in `KroxyliciousSidecarConfig` lists which annotations the app owner may set to override sidecar parameters: +The `delegatedAnnotations` field in `KroxyliciousSidecarConfig` lists which annotations the app owner may set to override sidecar parameters. +Initially it will support: | Annotation | Effect | |-----------|--------| -| `kroxylicious.io/sidecar-bootstrap-port` | Override bootstrap port | -| `kroxylicious.io/sidecar-node-id-range` | Override node ID range (e.g. `"0-5"`) | -| `kroxylicious.io/sidecar-resources-cpu` | Override CPU request/limit | -| `kroxylicious.io/sidecar-resources-memory` | Override memory request/limit | -| `kroxylicious.io/sidecar-plugin-images` | Additional plugin images (JSON array) | +| `kroxylicious.io/sidecar-resources-cpu` | Override CPU request/limit of the sidecar container | +| `kroxylicious.io/sidecar-resources-memory` | Override memory request/limit of the sidecar container | -Delegation is opt-in per annotation. By default nothing is delegated. Upstream Kafka address, filter definitions, proxy image, and security context are never delegatable. +Delegation is opt-in per annotation. By default nothing is delegated. ### Configuration drift detection @@ -198,6 +213,8 @@ volumeMounts: ServiceLoader discovers the plugin implementations from the combined classpath. Multiple plugin images can be mounted simultaneously, each at its own subdirectory. +A known and accepted risk of supporting OCI image mounting while the proxy only uses a flat classpath for plugin loading is that the ordering of Jars on that classpath is poorly defined. The longer term solution for that is plugin classloader isolation, which is out of scope for this proposal. + #### Plugin image convention Plugin images should be built `FROM scratch` with JARs at the image root: @@ -256,45 +273,6 @@ When the admin specifies plugin images in `KroxyliciousSidecarConfig.spec.plugin - **Supply chain**: a compromised plugin image contains malicious code with access to Kafka traffic and mounted credentials. Mitigate with image signing and digest-pinned references. - **Dependency conflicts**: as described above under flat classpath limitations. -#### Security analysis: delegated plugin image selection - -If the admin delegates plugin image selection to app owners (via the `kroxylicious.io/sidecar-plugin-images` annotation), the risks escalate: - -| Risk | Severity | Description | -|------|----------|-------------| -| Arbitrary code on proxy classpath | Critical | App owner specifies an image containing malicious JARs. The proxy has access to upstream Kafka credentials, TLS certs, and all Kafka traffic. | -| Registry credential leakage | High | OCI image volumes reuse the pod's `imagePullSecrets` and node-level credentials. An attacker-controlled registry receives pull requests carrying bearer tokens. | -| Image tag mutation | High | A tag (not a digest) can be replaced with malicious content between pulls. | -| Resource exhaustion | Medium | Large OCI images consume node disk. No per-volume size limits exist. | - -**Mitigations:** - -1. **Registry allow-list**: `allowedPluginRegistries` in `KroxyliciousSidecarConfig`. The webhook rejects delegated image references not matching an allowed prefix. -2. **Require digest pinning**: delegated image references without `@sha256:` are rejected. -3. **Audit logging**: all plugin image references are logged at INFO, with warnings for delegated images. - -Even with these mitigations, the fundamental trust grant is unchanged: a digest-pinned image from an allowed registry still runs arbitrary code in the proxy JVM. The mitigations reduce supply-chain risk but do not constrain what the code does once loaded. Delegating plugin image selection is equivalent to allowing the app owner to run arbitrary code with the proxy's identity. - -**Delegation is disabled by default.** When enabled, both `allowedPluginRegistries` and digest pinning are enforced. - -#### Future direction: PluginRegistry CRD - -A cleaner model for pre-approved plugins would be a cluster-scoped `PluginRegistry` CRD where the admin defines approved plugin images with namespace-level scoping: - -```yaml -apiVersion: kroxylicious.io/v1alpha1 -kind: PluginRegistry -metadata: - name: approved-filters -spec: - plugins: - - name: record-encryption - image: quay.io/kroxylicious/record-encryption@sha256:abc123 - allowedNamespaces: ["prod-*", "staging"] -``` - -This separates the "which plugins are trusted" question from the per-namespace sidecar config, avoids JSON-in-annotation, and makes the approval surface auditable via standard Kubernetes RBAC. It is a better long-term model than annotation-based delegation, but is out of scope for the initial implementation. - ### Upstream cluster selection In many deployments the admin manages multiple Kafka clusters (e.g. production, staging) and the app owner needs to choose which one their pod connects to. Rather than creating a separate `KroxyliciousSidecarConfig` per cluster, the admin defines an allow-list of named upstream clusters: @@ -330,53 +308,6 @@ The delegated annotations mechanism (bootstrap port, node ID range, resource req All delegation beyond upstream cluster selection requires the admin to explicitly list the delegated annotations. Nothing is delegated by default. -### Bypass prevention - -When the proxy is used as a policy enforcement point (e.g. record-level encryption, audit logging), applications that bypass the sidecar bypass the policy. This will be flagged in any compliance audit. The webhook needs to support configurations that make bypass difficult. - -#### Baseline: `KAFKA_BOOTSTRAP_SERVERS` - -The webhook sets `KAFKA_BOOTSTRAP_SERVERS=localhost:19092` on application containers. This is sufficient to prevent *accidental* bypass — the application connects to the sidecar by default. Bypassing requires deliberately hardcoding the upstream Kafka address. - -#### Defence-in-depth: NetworkPolicy + upstream authentication - -A stronger enforcement model combines two mechanisms: - -1. **NetworkPolicy restricting pod egress** — the admin creates a `NetworkPolicy` in the application namespace that limits egress to only the upstream Kafka cluster IPs/ports defined in `allowedUpstreamClusters`. This prevents the application from connecting to arbitrary external services. Because NetworkPolicy operates at the pod level, both the sidecar and the application container are subject to the same egress rules — but this is acceptable, because the sidecar only needs to reach the allowed clusters. - -2. **Upstream Kafka authentication** — the upstream Kafka cluster requires authentication (mTLS client certificates or SASL credentials). The sidecar holds the credentials; the application does not. Even if the application connects directly to the Kafka broker's address, the broker rejects the unauthenticated connection. - -Together these mean the application can only reach the allowed Kafka clusters (NetworkPolicy) and cannot authenticate to them without going through the sidecar. - -This model requires: - -- The upstream Kafka cluster enforces authentication (not optional/unauthenticated). -- The CRD supports upstream client authentication configuration. The current `upstreamTls` field handles server certificate validation; client certificate or SASL credential references would need to be added. - -#### Credential isolation limits - -Mounting the credential Secret only into the sidecar container is not a hard security boundary. An app owner with standard namespace-level RBAC can extract the credentials through several paths: - -- **Secret read access**: app owners typically have `get` on Secrets in their namespace (needed for their own application secrets). They can `kubectl get secret -o yaml`. -- **Container exec**: `kubectl exec -c kroxylicious-proxy` gives access to the mounted credential files. -- **Predictable volume names**: if the app owner knows the volume name the webhook will create, they can add a `volumeMount` in their own container spec referencing it. The webhook adds the volume before API server validation, so this passes. - -Container filesystem isolation raises the bar but does not create a trust boundary against a determined app owner. The real enforcement boundary is RBAC. - -**Mitigations:** - -- **Restrict Secret access by name**: the webhook uses a consistent naming convention for credential Secrets (e.g. `kroxylicious-upstream-*`). The admin can write RBAC rules that grant the app owner `get` on Secrets *except* those matching this pattern. Kubernetes RBAC supports `resourceNames` on deny, though operationally this means enumerating allowed Secrets rather than denying specific ones. -- **Restrict exec into the sidecar**: a `ValidatingAdmissionPolicy` can deny `pods/exec` subresource requests targeting the `kroxylicious-proxy` container. -- **Short-lived credentials**: if the upstream Kafka cluster supports OAuth/OIDC token-based authentication, the sidecar can use a projected ServiceAccount token with a short lifetime. Extracting a token is still possible but the window of use is limited. - -None of these fully close the gap. Within a single pod, Kubernetes does not offer strong credential isolation between containers. This is a fundamental platform limitation, not specific to this design. - -For deployments where policy bypass is an audit-critical concern, the strongest posture is: NetworkPolicy restricting egress + upstream Kafka requiring authentication + RBAC preventing app owners from reading proxy credential Secrets or exec-ing into the sidecar container. This is operationally achievable but requires deliberate RBAC design by the cluster admin. - -#### Alternative: iptables redirection - -The Istio model — an init container with `NET_ADMIN` that sets up iptables rules to redirect Kafka-port traffic to the sidecar, excluding the proxy process by UID — would enforce bypass prevention at the network level without requiring credential isolation at all. However, it requires granting `NET_ADMIN` to the init container, conflicting with the security posture of dropping all capabilities. - ### Webhook deployment The webhook is packaged as a container image and deployed as a single-replica `Deployment` in a dedicated `kroxylicious-webhook` namespace. Install manifests are provided for: From 8c48d93a84d8df42331470b95e457b7df9047b8b Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 5 May 2026 11:24:46 +1200 Subject: [PATCH 03/23] Adopt sidecar.kroxylicious.io/ annotation prefix Use Istio-style subdomain prefix (sidecar.kroxylicious.io/) instead of kroxylicious.io/sidecar-* for all sidecar webhook annotations and labels. Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 36 +++++++++++----------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 588c235..5113c98 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -30,12 +30,12 @@ The webhook is design to eventually operate under a strict two-party trust model - **Application pod owner**: can opt out of injection, and may override specific settings (bootstrap port, node ID range, resource requests) if the admin explicitly delegates those annotations. Making the boundary between the webhook administrator and the application pod owner a _reliable_ trust boundary will require further development than is specified in this proposal. -However pod annotations in the `kroxylicious.io/` namespace form the basic building blocks for a flexible but reliable trust boundary. +However pod annotations in the `sidecar.kroxylicious.io/` namespace form the basic building blocks for a flexible but reliable trust boundary. * Some annotations are always set by the webhook. -For example, the proxy obtains its configuration via a `kroxylicious.io/proxy-config` pod annotation which is projected into the sidecar container via a `downwardAPI` volume. -The webhook always overwrites that annotation (`kroxylicious.io/proxy-config`) on the pod, regardless of any value the app owner may have set. -* The administrator can delegate some annotations to be specifed by/overridden the application owner. When specified by the application owner they will not be overwritten by the webhook. Examples defined in this proposal are `kroxylicious.io/sidecar-resources-cpu` and `kroxylicious.io/sidecar-resources-memory` (see below). -* Annotations which the administrator has **not** delegated will have their effect overridden by the webhook based on the Administrator-controlled `KroxyliciousSidecarConfig` resource. A warning will be logged when such overriding is necessary. +For example, the proxy obtains its configuration via a `sidecar.kroxylicious.io/proxy-config` pod annotation which is projected into the sidecar container via a `downwardAPI` volume. +The webhook always overwrites that annotation (`sidecar.kroxylicious.io/proxy-config`) on the pod, regardless of any value the app owner may have set. +* The administrator can delegate some annotations to be specified by/overridden by the application owner. When specified by the application owner they will not be overwritten by the webhook. Examples defined in this proposal are `sidecar.kroxylicious.io/resources-cpu` and `sidecar.kroxylicious.io/resources-memory` (see below). +* Annotations which the administrator has **not** delegated will have their effect overridden by the webhook based on the Administrator-controlled `KroxyliciousSidecarConfig` resource. A warning will be logged when such overriding is necessary. ### Injection decision @@ -44,8 +44,8 @@ Injection is opt-in at the namespace level and opt-out at the pod level, followi | Mechanism | Key | Effect | |-----------|-----|--------| -| Namespace label | `kroxylicious.io/sidecar-injection: enabled` | Webhook intercepts pod creates in this namespace | -| Pod label | `kroxylicious.io/inject-sidecar: "false"` | Pod is excluded via `objectSelector` — never reaches the webhook | +| Namespace label | `sidecar.kroxylicious.io/injection: enabled` | Webhook intercepts pod creates in this namespace | +| Pod label | `sidecar.kroxylicious.io/inject: "false"` | Pod is excluded via `objectSelector` — never reaches the webhook | The `MutatingWebhookConfiguration` uses `namespaceSelector` to scope interception and `objectSelector` to exclude opted-out pods. The webhook itself is idempotent: if a container named `kroxylicious-proxy` already exists, injection is skipped. @@ -66,7 +66,7 @@ In practice, bypassing the sidecar requires the application to deliberately hard A namespaced CRD (group `kroxylicious.io`, version `v1alpha1`) defines the sidecar configuration. The webhook admin creates one per namespace. The following edge cases are handled: 1. **No config in namespace**: the pod is admitted without injection (debug log only). This is the common case for namespaces where the admin has enabled the namespace label but not yet created a config. -2. **Multiple configs in namespace**: the pod is admitted without injection (warning logged). The pod can select a specific config via the `kroxylicious.io/sidecar-config` annotation; without this annotation the webhook cannot choose and skips injection. +2. **Multiple configs in namespace**: the pod is admitted without injection (warning logged). The pod can select a specific config via the `sidecar.kroxylicious.io/config` annotation; without this annotation the webhook cannot choose and skips injection. 3. **Config is invalid in a way the webhook can detect** (e.g. malformed delegated annotation values, plugin image without a digest): the webhook logs a warning and admits the pod without injection. Consistent with fail-open semantics. 4. **Config is invalid in a way only the proxy can detect** (e.g. unreachable upstream Kafka, wrong TLS trust anchor, non-existent filter type): the webhook injects the sidecar normally. The proxy will fail its startup probe and the pod will not become ready, surfacing the problem via standard Kubernetes health-check mechanisms. @@ -98,8 +98,8 @@ spec: reference: registry.example.com/my-filter:v1.0@sha256:abc123 pullPolicy: IfNotPresent delegatedAnnotations: - - kroxylicious.io/sidecar-resources-cpu - - kroxylicious.io/sidecar-resources-memory + - sidecar.kroxylicious.io/resources-cpu + - sidecar.kroxylicious.io/resources-memory ``` **Why a CRD, not a ConfigMap?** Schema validation by the API server, RBAC separation (admin creates, app owners can't modify), status conditions for observability, consistency with the existing Kroxylicious Kubernetes API. @@ -108,7 +108,7 @@ spec: ### Config injection -The webhook generates proxy configuration YAML from the `KroxyliciousSidecarConfig` spec, using the same `Configuration` model from `kroxylicious-runtime`. The generated config is stored in a pod annotation (`kroxylicious.io/proxy-config`) and projected into the sidecar container via a `downwardAPI` volume. +The webhook generates proxy configuration YAML from the `KroxyliciousSidecarConfig` spec, using the same `Configuration` model from `kroxylicious-runtime`. The generated config is stored in a pod annotation (`sidecar.kroxylicious.io/proxy-config`) and projected into the sidecar container via a `downwardAPI` volume. This avoids creating per-pod ConfigMaps, which would require additional RBAC, lifecycle management for orphaned ConfigMaps, and unique name generation. The annotation approach is self-contained within the pod. @@ -153,8 +153,8 @@ Initially it will support: | Annotation | Effect | |-----------|--------| -| `kroxylicious.io/sidecar-resources-cpu` | Override CPU request/limit of the sidecar container | -| `kroxylicious.io/sidecar-resources-memory` | Override memory request/limit of the sidecar container | +| `sidecar.kroxylicious.io/resources-cpu` | Override CPU request/limit of the sidecar container | +| `sidecar.kroxylicious.io/resources-memory` | Override memory request/limit of the sidecar container | Delegation is opt-in per annotation. By default nothing is delegated. @@ -164,8 +164,8 @@ The webhook stamps each injected pod with metadata annotations: | Annotation | Value | |-----------|-------| -| `kroxylicious.io/config-generation` | The `metadata.generation` of the `KroxyliciousSidecarConfig` at injection time | -| `kroxylicious.io/injection-timestamp` | ISO-8601 timestamp of when the sidecar was injected | +| `sidecar.kroxylicious.io/config-generation` | The `metadata.generation` of the `KroxyliciousSidecarConfig` at injection time | +| `sidecar.kroxylicious.io/injection-timestamp` | ISO-8601 timestamp of when the sidecar was injected | Because the webhook only mutates pods at creation time, configuration changes to `KroxyliciousSidecarConfig` do not propagate to running pods. This matches how Istio and Linkerd handle sidecar injection. Users must restart pods to pick up new configuration. @@ -173,8 +173,8 @@ The generation stamp allows operators to identify stale pods: ``` kubectl get pods -n my-ns -o json | jq '[.items[] | - select(.metadata.annotations["kroxylicious.io/config-generation"] != null) | - {name: .metadata.name, generation: .metadata.annotations["kroxylicious.io/config-generation"]}]' + select(.metadata.annotations["sidecar.kroxylicious.io/config-generation"] != null) | + {name: .metadata.name, generation: .metadata.annotations["sidecar.kroxylicious.io/config-generation"]}]' ``` In a future iteration, a reconciler could watch for pods with outdated generations and surface an `UpToDate` condition on the `KroxyliciousSidecarConfig` status, giving operators visibility into configuration drift without requiring manual queries. @@ -289,7 +289,7 @@ spec: The app owner selects a cluster by annotation: ```yaml -kroxylicious.io/sidecar-upstream-cluster: staging +sidecar.kroxylicious.io/upstream-cluster: staging ``` The admin retains control over which clusters are reachable. The app owner cannot specify an arbitrary bootstrap address — only names from the allow-list are accepted. If the annotation names a cluster not in the list, or is absent when multiple clusters are defined, injection is skipped with a warning. From fe32aa46b23689e3ee152b02baf0cda461067242 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 5 May 2026 11:26:13 +1200 Subject: [PATCH 04/23] Use 'target' terminology instead of 'upstream' Align the design document with the implementation, which uses targetBootstrapServers, targetClusterTls, etc. Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 26 +++++++++++----------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 5113c98..226e1e2 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -26,7 +26,7 @@ Manual sidecar construction is error-prone and creates a maintenance burden. A w The webhook is design to eventually operate under a strict two-party trust model: -- **Webhook administrator**: controls what gets injected — the proxy image, upstream Kafka address, filter definitions, security context. These are never overridable by the app owner. +- **Webhook administrator**: controls what gets injected — the proxy image, target Kafka address, filter definitions, security context. These are never overridable by the app owner. - **Application pod owner**: can opt out of injection, and may override specific settings (bootstrap port, node ID range, resource requests) if the admin explicitly delegates those annotations. Making the boundary between the webhook administrator and the application pod owner a _reliable_ trust boundary will require further development than is specified in this proposal. @@ -55,7 +55,7 @@ When configured to fail open and the webhook experiences an internal errors, it #### Bypass prevention -The webhook sets `KAFKA_BOOTSTRAP_SERVERS` to point at the sidecar, but nothing prevents an application from connecting directly to the upstream Kafka cluster. Kubernetes `NetworkPolicy` cannot help here: it operates at the pod level, so a policy blocking egress to Kafka would also block the sidecar's upstream connection. +The webhook sets `KAFKA_BOOTSTRAP_SERVERS` to point at the sidecar, but nothing prevents an application from connecting directly to the target Kafka cluster. Kubernetes `NetworkPolicy` cannot help here: it operates at the pod level, so a policy blocking egress to Kafka would also block the sidecar's connection to the target cluster. The Istio model — an init container with `NET_ADMIN` that sets up iptables rules to redirect Kafka-port traffic to the sidecar, excluding the proxy process by UID — would enforce this, but requires granting `NET_ADMIN` to the init container, conflicting with the security posture of dropping all capabilities. @@ -68,7 +68,7 @@ A namespaced CRD (group `kroxylicious.io`, version `v1alpha1`) defines the sidec 1. **No config in namespace**: the pod is admitted without injection (debug log only). This is the common case for namespaces where the admin has enabled the namespace label but not yet created a config. 2. **Multiple configs in namespace**: the pod is admitted without injection (warning logged). The pod can select a specific config via the `sidecar.kroxylicious.io/config` annotation; without this annotation the webhook cannot choose and skips injection. 3. **Config is invalid in a way the webhook can detect** (e.g. malformed delegated annotation values, plugin image without a digest): the webhook logs a warning and admits the pod without injection. Consistent with fail-open semantics. -4. **Config is invalid in a way only the proxy can detect** (e.g. unreachable upstream Kafka, wrong TLS trust anchor, non-existent filter type): the webhook injects the sidecar normally. The proxy will fail its startup probe and the pod will not become ready, surfacing the problem via standard Kubernetes health-check mechanisms. +4. **Config is invalid in a way only the proxy can detect** (e.g. unreachable target Kafka cluster, wrong TLS trust anchor, non-existent filter type): the webhook injects the sidecar normally. The proxy will fail its startup probe and the pod will not become ready, surfacing the problem via standard Kubernetes health-check mechanisms. ```yaml apiVersion: kroxylicious.io/v1alpha1 @@ -76,7 +76,7 @@ kind: KroxyliciousSidecarConfig metadata: name: my-config spec: - upstreamBootstrapServers: kafka-prod.internal:9092 + targetBootstrapServers: kafka-prod.internal:9092 bootstrapPort: 9092 # default, configurable nodeIdRange: startInclusive: 0 @@ -88,7 +88,7 @@ spec: - name: my-filter type: io.example.MyFilterFactory config: { ... } - upstreamTls: + targetClusterTls: trustAnchorSecretRef: name: kafka-ca key: ca.crt @@ -142,9 +142,9 @@ The injected sidecar follows the same patterns as `ProxyDeploymentDependentResou The security context is never weakened. If the pod already has a stricter security context, it is preserved. -### Upstream TLS +### Target cluster TLS -When `spec.upstreamTls.trustAnchorSecretRef` is set, the webhook adds a volume mounting the referenced Secret into the sidecar and configures the proxy to use it as a PEM trust store. The Secret must exist in the pod's namespace. +When `spec.targetClusterTls.trustAnchorSecretRef` is set, the webhook adds a volume mounting the referenced Secret into the sidecar and configures the proxy to use it as a PEM trust store. The Secret must exist in the pod's namespace. ### Delegated annotations @@ -273,13 +273,13 @@ When the admin specifies plugin images in `KroxyliciousSidecarConfig.spec.plugin - **Supply chain**: a compromised plugin image contains malicious code with access to Kafka traffic and mounted credentials. Mitigate with image signing and digest-pinned references. - **Dependency conflicts**: as described above under flat classpath limitations. -### Upstream cluster selection +### Target cluster selection -In many deployments the admin manages multiple Kafka clusters (e.g. production, staging) and the app owner needs to choose which one their pod connects to. Rather than creating a separate `KroxyliciousSidecarConfig` per cluster, the admin defines an allow-list of named upstream clusters: +In many deployments the admin manages multiple Kafka clusters (e.g. production, staging) and the app owner needs to choose which one their pod connects to. Rather than creating a separate `KroxyliciousSidecarConfig` per cluster, the admin defines an allow-list of named target clusters: ```yaml spec: - allowedUpstreamClusters: + allowedTargetClusters: - name: production bootstrapServers: kafka-prod.internal:9092 - name: staging @@ -289,12 +289,12 @@ spec: The app owner selects a cluster by annotation: ```yaml -sidecar.kroxylicious.io/upstream-cluster: staging +sidecar.kroxylicious.io/target-cluster: staging ``` The admin retains control over which clusters are reachable. The app owner cannot specify an arbitrary bootstrap address — only names from the allow-list are accepted. If the annotation names a cluster not in the list, or is absent when multiple clusters are defined, injection is skipped with a warning. -When `allowedUpstreamClusters` is not set, the existing `upstreamBootstrapServers` field is used directly and there is no cluster selection. +When `allowedTargetClusters` is not set, the existing `targetBootstrapServers` field is used directly and there is no cluster selection. This is the lowest-risk form of delegation — the app owner chooses a network destination from an admin-controlled set — and is likely the highest-demand delegation feature for app teams. It is included in the initial implementation. @@ -306,7 +306,7 @@ The delegated annotations mechanism (bootstrap port, node ID range, resource req 2. **Filter configuration** — app owner adjusts parameters on admin-selected filters. Medium risk: bounded by the filter's config surface. 3. **Plugin image selection** — app owner chooses what code runs in the proxy JVM. High risk: arbitrary code execution (see security analysis above). -All delegation beyond upstream cluster selection requires the admin to explicitly list the delegated annotations. Nothing is delegated by default. +All delegation beyond target cluster selection requires the admin to explicitly list the delegated annotations. Nothing is delegated by default. ### Webhook deployment From 0b1bd891c88da5921843e27de0fc896dd47e72b0 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 5 May 2026 11:46:25 +1200 Subject: [PATCH 05/23] Replace sidecar-status with config-generation annotation Use sidecar.kroxylicious.io/config-generation (recording the KroxyliciousSidecarConfig's metadata.generation) instead of a simple status flag. Serves as both idempotency guard and drift detection mechanism. Drop injection-timestamp. Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 24 ++++++++++++++-------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 226e1e2..f9db8bd 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -83,6 +83,13 @@ spec: endInclusive: 2 managementPort: 9082 # default, configurable proxyImage: quay.io/kroxylicious/proxy:0.21.0 # optional override + resources: # default resource requests/limits for the sidecar + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 500m + memory: 256Mi setBootstrapEnvVar: true # sets KAFKA_BOOTSTRAP_SERVERS on app containers filterDefinitions: - name: my-filter @@ -148,24 +155,23 @@ When `spec.targetClusterTls.trustAnchorSecretRef` is set, the webhook adds a vol ### Delegated annotations -The `delegatedAnnotations` field in `KroxyliciousSidecarConfig` lists which annotations the app owner may set to override sidecar parameters. +The `delegatedAnnotations` field in `KroxyliciousSidecarConfig` lists which annotations the app owner may set on their pod to override sidecar parameters. The admin sets defaults via the CRD spec (e.g. `spec.resources`); delegation allows the app owner to override those defaults for a specific pod. + Initially it will support: | Annotation | Effect | |-----------|--------| -| `sidecar.kroxylicious.io/resources-cpu` | Override CPU request/limit of the sidecar container | -| `sidecar.kroxylicious.io/resources-memory` | Override memory request/limit of the sidecar container | +| `sidecar.kroxylicious.io/resources-cpu` | Override the CPU request/limit from `spec.resources` | +| `sidecar.kroxylicious.io/resources-memory` | Override the memory request/limit from `spec.resources` | -Delegation is opt-in per annotation. By default nothing is delegated. +Delegation is opt-in per annotation. By default nothing is delegated. ### Configuration drift detection -The webhook stamps each injected pod with metadata annotations: +The webhook stamps each injected pod with a `sidecar.kroxylicious.io/config-generation` annotation recording the `metadata.generation` of the `KroxyliciousSidecarConfig` at injection time. This annotation serves two purposes: -| Annotation | Value | -|-----------|-------| -| `sidecar.kroxylicious.io/config-generation` | The `metadata.generation` of the `KroxyliciousSidecarConfig` at injection time | -| `sidecar.kroxylicious.io/injection-timestamp` | ISO-8601 timestamp of when the sidecar was injected | +1. **Idempotency guard**: its presence indicates that the sidecar has already been injected, preventing re-injection when the webhook is reinvoked. +2. **Drift detection**: its value can be compared (equality only) with the current generation of the `KroxyliciousSidecarConfig` to identify pods running stale configuration. Because the webhook only mutates pods at creation time, configuration changes to `KroxyliciousSidecarConfig` do not propagate to running pods. This matches how Istio and Linkerd handle sidecar injection. Users must restart pods to pick up new configuration. From a407e8ecb6c84752001e745dce3a49b97e47614d Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 5 May 2026 11:53:01 +1200 Subject: [PATCH 06/23] Add SASL out-of-scope note and introduce proxy-config annotation Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index f9db8bd..0b38240 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -32,8 +32,9 @@ The webhook is design to eventually operate under a strict two-party trust model Making the boundary between the webhook administrator and the application pod owner a _reliable_ trust boundary will require further development than is specified in this proposal. However pod annotations in the `sidecar.kroxylicious.io/` namespace form the basic building blocks for a flexible but reliable trust boundary. * Some annotations are always set by the webhook. -For example, the proxy obtains its configuration via a `sidecar.kroxylicious.io/proxy-config` pod annotation which is projected into the sidecar container via a `downwardAPI` volume. -The webhook always overwrites that annotation (`sidecar.kroxylicious.io/proxy-config`) on the pod, regardless of any value the app owner may have set. +For example, the webhook generates proxy configuration YAML from the `KroxyliciousSidecarConfig` and stores it in a `sidecar.kroxylicious.io/proxy-config` pod annotation (see [Config injection](#config-injection)). +This annotation is projected into the sidecar container as a file via a `downwardAPI` volume. +The webhook always overwrites `sidecar.kroxylicious.io/proxy-config` on the pod, regardless of any value the app owner may have set. * The administrator can delegate some annotations to be specified by/overridden by the application owner. When specified by the application owner they will not be overwritten by the webhook. Examples defined in this proposal are `sidecar.kroxylicious.io/resources-cpu` and `sidecar.kroxylicious.io/resources-memory` (see below). * Annotations which the administrator has **not** delegated will have their effect overridden by the webhook based on the Administrator-controlled `KroxyliciousSidecarConfig` resource. A warning will be logged when such overriding is necessary. @@ -61,6 +62,8 @@ The Istio model — an init container with `NET_ADMIN` that sets up iptables rul In practice, bypassing the sidecar requires the application to deliberately hardcode the real Kafka address. An app owner determined to bypass can also opt out of injection entirely via pod labels. The enforcement boundary is RBAC on who can create pods in the namespace, not network controls within the pod. If the threat model requires enforcement against a hostile app owner, iptables redirection could be added as an opt-in capability in a future iteration. +SASL handling (e.g. rejecting downstream SASL handshakes or requiring proxy-initiated authentication to the target cluster) is out of scope for the alpha. The proxy passes SASL frames through unmodified. + ### CRD: `KroxyliciousSidecarConfig` A namespaced CRD (group `kroxylicious.io`, version `v1alpha1`) defines the sidecar configuration. The webhook admin creates one per namespace. The following edge cases are handled: From 8e4acfc7946d54eb3c403775031c520ed530c361 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 5 May 2026 11:54:48 +1200 Subject: [PATCH 07/23] Document feature gate detection via FEATURE_GATES env var Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 0b38240..9ac42a5 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -138,9 +138,9 @@ The management endpoint binds to `0.0.0.0` because kubelet HTTP probes target th ### Native sidecar containers -On Kubernetes 1.28+, the webhook injects the proxy as a native sidecar — an init container with `restartPolicy: Always`. This gives proper startup ordering (proxy starts before the application) and shutdown ordering (proxy stops after the application). On older clusters, the webhook falls back to injecting into `spec.containers`. +On Kubernetes 1.29+ (where the `SidecarContainers` feature gate is enabled by default), the webhook injects the proxy as a native sidecar — an init container with `restartPolicy: Always`. This gives proper startup ordering (proxy starts before the application) and shutdown ordering (proxy stops after the application). On older clusters, the webhook falls back to injecting into `spec.containers`. -The webhook detects the cluster's Kubernetes version at startup and chooses the injection strategy accordingly. +The webhook detects the cluster's Kubernetes version at startup and chooses the injection strategy accordingly. For clusters running alpha versions of features (e.g. native sidecars on 1.28, OCI image volumes on 1.31-1.32), the deployer can set the `FEATURE_GATES` environment variable on the webhook (e.g. `FEATURE_GATES=SidecarContainers=true,ImageVolume=true`) to override the version-based defaults. ### Sidecar container spec From 93152447b7db36f5acedd5c61f3553a7c9315060 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 5 May 2026 12:01:21 +1200 Subject: [PATCH 08/23] Add CRD status API and fix module naming in design Document the KroxyliciousSidecarConfig status subresource (Ready condition, observedGeneration) to match the implementation. Update module names and shared dependencies to reflect actual repo structure. Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 40 +++++++++++++++++++--- 1 file changed, 36 insertions(+), 4 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 9ac42a5..8a1f9c2 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -116,6 +116,39 @@ spec: **Why not reuse the operator's CRDs?** The operator CRDs model a shared proxy deployment with ingress networking, multi-cluster support, and cross-resource references. The sidecar use case is fundamentally simpler — one virtual cluster, localhost binding, no ingress. Coupling them would constrain both models. +#### Status + +The CRD has a `status` subresource with the following fields: + +| Field | Type | Description | +|-------|------|-------------| +| `observedGeneration` | `int64` | The `metadata.generation` most recently observed by the webhook | +| `conditions` | `[]Condition` | Standard Kubernetes conditions (see below) | + +The webhook maintains a single condition type: + +| Condition | Meaning | +|-----------|---------| +| `Ready` | The webhook has observed and accepted this configuration. `observedGeneration` on the condition tracks which generation was acknowledged. | + +The webhook sets `Ready=True` (reason `Accepted`) when it first observes the config via its informer. The condition is only updated when the generation changes, avoiding unnecessary status writes. If the status update fails (e.g. conflict), the webhook logs a warning but continues to inject pods normally — status is informational, not load-bearing. + +Example status: + +```yaml +status: + observedGeneration: 3 + conditions: + - type: Ready + status: "True" + reason: Accepted + message: "" + lastTransitionTime: "2025-01-15T10:30:00Z" + observedGeneration: 3 +``` + +This gives operators visibility into whether the webhook has picked up the latest configuration, complementing the per-pod `sidecar.kroxylicious.io/config-generation` annotation for drift detection. + ### Config injection The webhook generates proxy configuration YAML from the `KroxyliciousSidecarConfig` spec, using the same `Configuration` model from `kroxylicious-runtime`. The generated config is stored in a pod annotation (`sidecar.kroxylicious.io/proxy-config`) and projected into the sidecar container via a `downwardAPI` volume. @@ -337,17 +370,16 @@ The webhook is packaged as a container image and deployed as a single-replica `D The webhook operates independently of the operator. It does not depend on the operator being deployed, does not use JOSDK, and does not reference operator CRDs. -The only shared dependencies are: +The only shared dependency is: -- `kroxylicious-kubernetes-api` — for the CRD Java types - `kroxylicious-runtime` — for the proxy `Configuration` model, used to generate valid proxy config YAML ## Affected/not affected projects | Project | Affected | Nature of change | |---------|----------|-----------------| -| `kroxylicious-kubernetes-web-hook` | Yes | New module | -| `kroxylicious-kubernetes-api` | Yes | New CRD: `KroxyliciousSidecarConfig` | +| `kroxylicious-kubernetes/kroxylicious-admission` | Yes | New module: webhook implementation | +| `kroxylicious-kubernetes/kroxylicious-admission-api` | Yes | New module: `KroxyliciousSidecarConfig` CRD | | `kroxylicious-app` | Already merged | `classpath-plugins/` directory scanning | | `kroxylicious-operator` | No | | | `kroxylicious-runtime` | No | Used as a dependency, not modified | From 0f0d7506c32a4b6ee66c3005ce1c1f6b79d9833e Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 5 May 2026 12:01:56 +1200 Subject: [PATCH 09/23] Fix sidecar container spec: security context, probes Document pod-level securityContext (runAsNonRoot, seccompProfile: RuntimeDefault). Fix probe port from 9190 to management port (9082). Add actual probe parameters from implementation. Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 8a1f9c2..081af55 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -179,11 +179,13 @@ The webhook detects the cluster's Kubernetes version at startup and chooses the The injected sidecar follows the same patterns as `ProxyDeploymentDependentResource` in the operator: -- `securityContext`: `allowPrivilegeEscalation: false`, `capabilities: drop ALL`, `readOnlyRootFilesystem: true` -- Probes: `startupProbe` (30 x 2s), `livenessProbe`, `readinessProbe` — all HTTP GET `/livez` on port 9190 +- Container-level `securityContext`: `allowPrivilegeEscalation: false`, `capabilities: drop ALL`, `readOnlyRootFilesystem: true` +- Probes: `startupProbe` (initialDelay 5s, period 2s, failure threshold 30), `livenessProbe` (initialDelay 30s, period 10s, failure threshold 3), `readinessProbe` (initialDelay 5s, period 2s, failure threshold 5) — all HTTP GET `/livez` on the management port (default 9082) - `terminationMessagePolicy: FallbackToLogsOnError` -The security context is never weakened. If the pod already has a stricter security context, it is preserved. +If the pod has no pod-level `securityContext`, the webhook sets `runAsNonRoot: true` and `seccompProfile: RuntimeDefault`. If a pod-level security context is already present, it is left unchanged — the webhook does not attempt to merge or strengthen it. + +The container-level security context is never weakened. If the pod already has a stricter security context, it is preserved. ### Target cluster TLS From 69b6221f4c6f4374378957f84df1b6929ff0551c Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 5 May 2026 12:37:23 +1200 Subject: [PATCH 10/23] Revise status failure handling, drop pod-level securityContext Status update failures should be retried, not silently swallowed. Remove pod-level securityContext from webhook; admins should use PodSecurity admission policies instead to avoid webhook ordering conflicts. Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 081af55..d8cec1a 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -131,7 +131,7 @@ The webhook maintains a single condition type: |-----------|---------| | `Ready` | The webhook has observed and accepted this configuration. `observedGeneration` on the condition tracks which generation was acknowledged. | -The webhook sets `Ready=True` (reason `Accepted`) when it first observes the config via its informer. The condition is only updated when the generation changes, avoiding unnecessary status writes. If the status update fails (e.g. conflict), the webhook logs a warning but continues to inject pods normally — status is informational, not load-bearing. +The webhook sets `Ready=True` (reason `Accepted`) when it first observes the config via its informer. The condition is only updated when the generation changes, avoiding unnecessary status writes. Status update failures (e.g. conflicts) should be retried; persistent failures should be surfaced via logging so that operators can investigate. A status update failure does not block pod admission, but a `KroxyliciousSidecarConfig` with a stale or missing `Ready` condition indicates a problem that needs attention. Example status: @@ -147,7 +147,7 @@ status: observedGeneration: 3 ``` -This gives operators visibility into whether the webhook has picked up the latest configuration, complementing the per-pod `sidecar.kroxylicious.io/config-generation` annotation for drift detection. +This gives operators visibility into whether the webhook has picked up the latest configuration, complementing the per-pod `sidecar.kroxylicious.io/config-generation` annotation for drift detection. The user documentation should describe how these mechanisms work together and how operators are expected to use them to reason about the state of their sidecar fleet. ### Config injection @@ -183,7 +183,7 @@ The injected sidecar follows the same patterns as `ProxyDeploymentDependentResou - Probes: `startupProbe` (initialDelay 5s, period 2s, failure threshold 30), `livenessProbe` (initialDelay 30s, period 10s, failure threshold 3), `readinessProbe` (initialDelay 5s, period 2s, failure threshold 5) — all HTTP GET `/livez` on the management port (default 9082) - `terminationMessagePolicy: FallbackToLogsOnError` -If the pod has no pod-level `securityContext`, the webhook sets `runAsNonRoot: true` and `seccompProfile: RuntimeDefault`. If a pod-level security context is already present, it is left unchanged — the webhook does not attempt to merge or strengthen it. +The webhook does not set a pod-level `securityContext`. Pod-level security policies (e.g. `runAsNonRoot`, `seccompProfile`) are the responsibility of the cluster admin via Kubernetes `PodSecurity` admission (`pod-security.kubernetes.io/enforce: restricted` namespace label) or equivalent policy enforcement. Setting pod-level security context from a mutating webhook risks ordering conflicts with other webhooks. The container-level security context is never weakened. If the pod already has a stricter security context, it is preserved. From c8be00caa22f40db0ff9f7433e945732b0aa8e2a Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 5 May 2026 14:21:28 +1200 Subject: [PATCH 11/23] Remove app-owner plugin image delegation from design Per review consensus, delegated plugin image selection is too high risk (arbitrary code execution in the proxy JVM). Admin-specified plugins via spec.plugins remain unaffected. Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index d8cec1a..b4a24f5 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -70,7 +70,7 @@ A namespaced CRD (group `kroxylicious.io`, version `v1alpha1`) defines the sidec 1. **No config in namespace**: the pod is admitted without injection (debug log only). This is the common case for namespaces where the admin has enabled the namespace label but not yet created a config. 2. **Multiple configs in namespace**: the pod is admitted without injection (warning logged). The pod can select a specific config via the `sidecar.kroxylicious.io/config` annotation; without this annotation the webhook cannot choose and skips injection. -3. **Config is invalid in a way the webhook can detect** (e.g. malformed delegated annotation values, plugin image without a digest): the webhook logs a warning and admits the pod without injection. Consistent with fail-open semantics. +3. **Config is invalid in a way the webhook can detect** (e.g. malformed delegated annotation values): the webhook logs a warning and admits the pod without injection. Consistent with fail-open semantics. 4. **Config is invalid in a way only the proxy can detect** (e.g. unreachable target Kafka cluster, wrong TLS trust anchor, non-existent filter type): the webhook injects the sidecar normally. The proxy will fail its startup probe and the pod will not become ready, surfacing the problem via standard Kubernetes health-check mechanisms. ```yaml @@ -344,11 +344,10 @@ This is the lowest-risk form of delegation — the app owner chooses a network d ### Future delegation -The delegated annotations mechanism (bootstrap port, node ID range, resource requests, plugin images) described above provides a general-purpose extension point for further delegation. These are ordered roughly by blast radius: +The delegated annotations mechanism (bootstrap port, node ID range, resource requests) described above provides a general-purpose extension point for further delegation. These are ordered roughly by blast radius: 1. **Port and resource overrides** — app owner adjusts operational parameters. Low risk. 2. **Filter configuration** — app owner adjusts parameters on admin-selected filters. Medium risk: bounded by the filter's config surface. -3. **Plugin image selection** — app owner chooses what code runs in the proxy JVM. High risk: arbitrary code execution (see security analysis above). All delegation beyond target cluster selection requires the admin to explicitly list the delegated annotations. Nothing is delegated by default. From 020007fe5d7f0263809e2f5dfae9c07a51e4834a Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 5 May 2026 15:16:48 +1200 Subject: [PATCH 12/23] Restructure CRD for virtualClusters, remove delegation Move targetBootstrapServers, bootstrapPort, nodeIdRange, and targetClusterTls under a virtualClusters list. Remove delegated annotations section. Rewrite target cluster selection as a virtual clusters section. Update trust model, port allocation table, and future delegation to reflect the simplified alpha scope. Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 103 +++++++-------------- 1 file changed, 35 insertions(+), 68 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index b4a24f5..b3d5a7a 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -24,19 +24,19 @@ Manual sidecar construction is error-prone and creates a maintenance burden. A w ### Trust model -The webhook is design to eventually operate under a strict two-party trust model: +The webhook operates under a two-party trust model: -- **Webhook administrator**: controls what gets injected — the proxy image, target Kafka address, filter definitions, security context. These are never overridable by the app owner. -- **Application pod owner**: can opt out of injection, and may override specific settings (bootstrap port, node ID range, resource requests) if the admin explicitly delegates those annotations. +- **Webhook administrator**: controls what gets injected — the proxy image, target Kafka address, filter definitions, security context. These are not overridable by the app owner. +- **Application pod owner**: can opt out of injection via pod labels, and can select a specific `KroxyliciousSidecarConfig` by name via the `sidecar.kroxylicious.io/config` annotation. -Making the boundary between the webhook administrator and the application pod owner a _reliable_ trust boundary will require further development than is specified in this proposal. -However pod annotations in the `sidecar.kroxylicious.io/` namespace form the basic building blocks for a flexible but reliable trust boundary. +Pod annotations in the `sidecar.kroxylicious.io/` namespace form the building blocks for this trust boundary: * Some annotations are always set by the webhook. For example, the webhook generates proxy configuration YAML from the `KroxyliciousSidecarConfig` and stores it in a `sidecar.kroxylicious.io/proxy-config` pod annotation (see [Config injection](#config-injection)). This annotation is projected into the sidecar container as a file via a `downwardAPI` volume. The webhook always overwrites `sidecar.kroxylicious.io/proxy-config` on the pod, regardless of any value the app owner may have set. -* The administrator can delegate some annotations to be specified by/overridden by the application owner. When specified by the application owner they will not be overwritten by the webhook. Examples defined in this proposal are `sidecar.kroxylicious.io/resources-cpu` and `sidecar.kroxylicious.io/resources-memory` (see below). -* Annotations which the administrator has **not** delegated will have their effect overridden by the webhook based on the Administrator-controlled `KroxyliciousSidecarConfig` resource. A warning will be logged when such overriding is necessary. +* The `sidecar.kroxylicious.io/config` annotation allows the app owner to select which `KroxyliciousSidecarConfig` applies when multiple exist in the namespace. + +Annotation-based delegation of operational parameters (resource overrides, filter configuration) is not included in this proposal but could be added in a future iteration (see [Future delegation](#future-delegation)). ### Injection decision @@ -70,7 +70,7 @@ A namespaced CRD (group `kroxylicious.io`, version `v1alpha1`) defines the sidec 1. **No config in namespace**: the pod is admitted without injection (debug log only). This is the common case for namespaces where the admin has enabled the namespace label but not yet created a config. 2. **Multiple configs in namespace**: the pod is admitted without injection (warning logged). The pod can select a specific config via the `sidecar.kroxylicious.io/config` annotation; without this annotation the webhook cannot choose and skips injection. -3. **Config is invalid in a way the webhook can detect** (e.g. malformed delegated annotation values): the webhook logs a warning and admits the pod without injection. Consistent with fail-open semantics. +3. **Config is invalid in a way the webhook can detect** (e.g. missing required fields): the webhook logs a warning and admits the pod without injection. Consistent with fail-open semantics. 4. **Config is invalid in a way only the proxy can detect** (e.g. unreachable target Kafka cluster, wrong TLS trust anchor, non-existent filter type): the webhook injects the sidecar normally. The proxy will fail its startup probe and the pod will not become ready, surfacing the problem via standard Kubernetes health-check mechanisms. ```yaml @@ -79,42 +79,43 @@ kind: KroxyliciousSidecarConfig metadata: name: my-config spec: - targetBootstrapServers: kafka-prod.internal:9092 - bootstrapPort: 9092 # default, configurable - nodeIdRange: - startInclusive: 0 - endInclusive: 2 - managementPort: 9082 # default, configurable + virtualClusters: + - name: my-cluster + targetBootstrapServers: kafka-prod.internal:9092 + bootstrapPort: 9092 # default, configurable + nodeIdRange: + startInclusive: 0 + endInclusive: 2 + targetClusterTls: + trustAnchorSecretRef: + name: kafka-ca + key: ca.crt + managementPort: 9082 # default, configurable proxyImage: quay.io/kroxylicious/proxy:0.21.0 # optional override - resources: # default resource requests/limits for the sidecar + resources: # resource requests/limits for the sidecar requests: cpu: 100m memory: 128Mi limits: cpu: 500m memory: 256Mi - setBootstrapEnvVar: true # sets KAFKA_BOOTSTRAP_SERVERS on app containers + setBootstrapEnvVar: true # sets KAFKA_BOOTSTRAP_SERVERS on app containers filterDefinitions: - name: my-filter type: io.example.MyFilterFactory config: { ... } - targetClusterTls: - trustAnchorSecretRef: - name: kafka-ca - key: ca.crt plugins: - name: my-plugin image: reference: registry.example.com/my-filter:v1.0@sha256:abc123 pullPolicy: IfNotPresent - delegatedAnnotations: - - sidecar.kroxylicious.io/resources-cpu - - sidecar.kroxylicious.io/resources-memory ``` +The `virtualClusters` list contains per-cluster settings (target bootstrap address, bootstrap port, node ID range, target cluster TLS). The alpha enforces exactly one entry (`minItems: 1`, `maxItems: 1`). Top-level fields (`managementPort`, `proxyImage`, `resources`, `filterDefinitions`, `plugins`, `setBootstrapEnvVar`) are shared across all virtual clusters. + **Why a CRD, not a ConfigMap?** Schema validation by the API server, RBAC separation (admin creates, app owners can't modify), status conditions for observability, consistency with the existing Kroxylicious Kubernetes API. -**Why not reuse the operator's CRDs?** The operator CRDs model a shared proxy deployment with ingress networking, multi-cluster support, and cross-resource references. The sidecar use case is fundamentally simpler — one virtual cluster, localhost binding, no ingress. Coupling them would constrain both models. +**Why not reuse the operator's CRDs?** The operator CRDs model a shared proxy deployment with ingress networking, multi-cluster support, and cross-resource references. The sidecar use case is fundamentally simpler — localhost binding, no ingress, a single virtual cluster in the alpha. Coupling them would constrain both models. #### Status @@ -161,11 +162,11 @@ A typical sidecar config is a few hundred bytes, well within the ~256KB practica | Port | Purpose | Bind address | |------|---------|-------------| -| 9092 | Kafka bootstrap | `localhost` | -| 9093+ | Per-broker ports (one per node ID) | `localhost` | -| 9082 | Management (`/livez`, `/metrics`) | `0.0.0.0` | +| `bootstrapPort` (default 9092) | Kafka bootstrap | `localhost` | +| `bootstrapPort`+1 onwards | Per-broker ports (one per node ID) | `localhost` | +| `managementPort` (default 9082) | Management (`/livez`, `/metrics`) | `0.0.0.0` | -The webhook sets `KAFKA_BOOTSTRAP_SERVERS=localhost:9092` on application containers (configurable, can be disabled). +The webhook sets `KAFKA_BOOTSTRAP_SERVERS=localhost:` on application containers (configurable via `setBootstrapEnvVar`, defaults to `true`). The management endpoint binds to `0.0.0.0` because kubelet HTTP probes target the pod IP, not loopback. This means the application container can also reach `/livez` and `/metrics`, but neither endpoint exposes sensitive data. @@ -189,20 +190,7 @@ The container-level security context is never weakened. If the pod already has a ### Target cluster TLS -When `spec.targetClusterTls.trustAnchorSecretRef` is set, the webhook adds a volume mounting the referenced Secret into the sidecar and configures the proxy to use it as a PEM trust store. The Secret must exist in the pod's namespace. - -### Delegated annotations - -The `delegatedAnnotations` field in `KroxyliciousSidecarConfig` lists which annotations the app owner may set on their pod to override sidecar parameters. The admin sets defaults via the CRD spec (e.g. `spec.resources`); delegation allows the app owner to override those defaults for a specific pod. - -Initially it will support: - -| Annotation | Effect | -|-----------|--------| -| `sidecar.kroxylicious.io/resources-cpu` | Override the CPU request/limit from `spec.resources` | -| `sidecar.kroxylicious.io/resources-memory` | Override the memory request/limit from `spec.resources` | - -Delegation is opt-in per annotation. By default nothing is delegated. +When `spec.virtualClusters[].targetClusterTls.trustAnchorSecretRef` is set, the webhook adds a volume mounting the referenced Secret into the sidecar and configures the proxy to use it as a PEM trust store. The Secret must exist in the pod's namespace. ### Configuration drift detection @@ -317,40 +305,19 @@ When the admin specifies plugin images in `KroxyliciousSidecarConfig.spec.plugin - **Supply chain**: a compromised plugin image contains malicious code with access to Kafka traffic and mounted credentials. Mitigate with image signing and digest-pinned references. - **Dependency conflicts**: as described above under flat classpath limitations. -### Target cluster selection - -In many deployments the admin manages multiple Kafka clusters (e.g. production, staging) and the app owner needs to choose which one their pod connects to. Rather than creating a separate `KroxyliciousSidecarConfig` per cluster, the admin defines an allow-list of named target clusters: - -```yaml -spec: - allowedTargetClusters: - - name: production - bootstrapServers: kafka-prod.internal:9092 - - name: staging - bootstrapServers: kafka-staging.internal:9092 -``` +### Virtual clusters -The app owner selects a cluster by annotation: +The `virtualClusters` list defines the target Kafka clusters that the sidecar proxy will serve. Each entry specifies a name, target bootstrap address, localhost listening port, node ID range, and optional TLS configuration. -```yaml -sidecar.kroxylicious.io/target-cluster: staging -``` - -The admin retains control over which clusters are reachable. The app owner cannot specify an arbitrary bootstrap address — only names from the allow-list are accepted. If the annotation names a cluster not in the list, or is absent when multiple clusters are defined, injection is skipped with a warning. - -When `allowedTargetClusters` is not set, the existing `targetBootstrapServers` field is used directly and there is no cluster selection. - -This is the lowest-risk form of delegation — the app owner chooses a network destination from an admin-controlled set — and is likely the highest-demand delegation feature for app teams. It is included in the initial implementation. +The alpha enforces exactly one virtual cluster entry (`minItems: 1`, `maxItems: 1`). The list structure is forward-looking: future iterations could relax `maxItems` to support multi-cluster applications (e.g. MirrorMaker2) where a single pod connects to multiple Kafka clusters through the same sidecar. App-owner selection of virtual cluster by name or annotation is deferred. ### Future delegation -The delegated annotations mechanism (bootstrap port, node ID range, resource requests) described above provides a general-purpose extension point for further delegation. These are ordered roughly by blast radius: +Annotation-based delegation could allow the app owner to override specific sidecar parameters on a per-pod basis, with the admin explicitly opting in via the `KroxyliciousSidecarConfig`. Possible future delegation, ordered roughly by blast radius: -1. **Port and resource overrides** — app owner adjusts operational parameters. Low risk. +1. **Resource overrides** — app owner adjusts CPU/memory requests and limits. Low risk. 2. **Filter configuration** — app owner adjusts parameters on admin-selected filters. Medium risk: bounded by the filter's config surface. -All delegation beyond target cluster selection requires the admin to explicitly list the delegated annotations. Nothing is delegated by default. - ### Webhook deployment The webhook is packaged as a container image and deployed as a single-replica `Deployment` in a dedicated `kroxylicious-webhook` namespace. Install manifests are provided for: From 0e189fc1494526fc3a7593a14e9061e83b06fd46 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Tue, 5 May 2026 16:32:05 +1200 Subject: [PATCH 13/23] Add secret mounts section and clarify FEATURE_GATES rationale Add design for secretMounts field: motivation (secrets must not be in pod annotations), trust model progression (filesystem isolation now, network isolation later), and rationale for secretMounts over generic volume mounts. Update CRD example. Expand FEATURE_GATES rationale: the API server version does not reveal which feature gates are enabled, so explicit configuration avoids additional RBAC. Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 37 ++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index b3d5a7a..99f6260 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -100,6 +100,9 @@ spec: cpu: 500m memory: 256Mi setBootstrapEnvVar: true # sets KAFKA_BOOTSTRAP_SERVERS on app containers + secretMounts: + - name: kms + secretName: kms-credentials # mounted at /opt/kroxylicious/secrets/kms/ filterDefinitions: - name: my-filter type: io.example.MyFilterFactory @@ -111,7 +114,7 @@ spec: pullPolicy: IfNotPresent ``` -The `virtualClusters` list contains per-cluster settings (target bootstrap address, bootstrap port, node ID range, target cluster TLS). The alpha enforces exactly one entry (`minItems: 1`, `maxItems: 1`). Top-level fields (`managementPort`, `proxyImage`, `resources`, `filterDefinitions`, `plugins`, `setBootstrapEnvVar`) are shared across all virtual clusters. +The `virtualClusters` list contains per-cluster settings (target bootstrap address, bootstrap port, node ID range, target cluster TLS). The alpha enforces exactly one entry (`minItems: 1`, `maxItems: 1`). Top-level fields (`managementPort`, `proxyImage`, `resources`, `filterDefinitions`, `plugins`, `secretMounts`, `setBootstrapEnvVar`) are shared across all virtual clusters. **Why a CRD, not a ConfigMap?** Schema validation by the API server, RBAC separation (admin creates, app owners can't modify), status conditions for observability, consistency with the existing Kroxylicious Kubernetes API. @@ -174,7 +177,7 @@ The management endpoint binds to `0.0.0.0` because kubelet HTTP probes target th On Kubernetes 1.29+ (where the `SidecarContainers` feature gate is enabled by default), the webhook injects the proxy as a native sidecar — an init container with `restartPolicy: Always`. This gives proper startup ordering (proxy starts before the application) and shutdown ordering (proxy stops after the application). On older clusters, the webhook falls back to injecting into `spec.containers`. -The webhook detects the cluster's Kubernetes version at startup and chooses the injection strategy accordingly. For clusters running alpha versions of features (e.g. native sidecars on 1.28, OCI image volumes on 1.31-1.32), the deployer can set the `FEATURE_GATES` environment variable on the webhook (e.g. `FEATURE_GATES=SidecarContainers=true,ImageVolume=true`) to override the version-based defaults. +The webhook detects the cluster's Kubernetes version at startup and uses it to infer which features are available by default. However, the Kubernetes API server version does not reveal which alpha or beta feature gates are actually enabled on the cluster. Rather than requiring additional RBAC to query node or API server configuration, the webhook lets the deployer set the `FEATURE_GATES` environment variable explicitly (e.g. `FEATURE_GATES=SidecarContainers=true,ImageVolume=true`). This overrides the version-based defaults and is the recommended approach for clusters running features ahead of their default-on version (e.g. native sidecars on 1.28, OCI image volumes on 1.31-1.32). ### Sidecar container spec @@ -192,6 +195,36 @@ The container-level security context is never weakened. If the pod already has a When `spec.virtualClusters[].targetClusterTls.trustAnchorSecretRef` is set, the webhook adds a volume mounting the referenced Secret into the sidecar and configures the proxy to use it as a PEM trust store. The Secret must exist in the pod's namespace. +### Secret mounts + +Filter configuration is embedded in the proxy config YAML, which is stored in a pod annotation. Pod annotations are visible to anyone who can `get` pods. Filters that need secrets (e.g. KMS credentials for record encryption) must not have those values in the annotation. + +The `secretMounts` field on `KroxyliciousSidecarConfig` lets the webhook admin mount Kubernetes Secrets into the sidecar container. Each entry mounts all keys from the named Secret as read-only files under `/opt/kroxylicious/secrets//`. Filter config references these paths: + +```yaml +spec: + secretMounts: + - name: kms + secretName: kms-credentials + filterDefinitions: + - name: envelope-encryption + type: io.kroxylicious.filter.encryption.EnvelopeEncryptionFilterFactory + config: + credentialsFile: /opt/kroxylicious/secrets/kms/credentials.json +``` + +The mount path is derived automatically from the `name` field — the admin does not specify `mountPath` directly. This keeps the sidecar's filesystem layout under webhook control, consistent with the operator's approach of using fixed base paths for secret volumes. + +**Why `secretMounts` over generic `sidecarVolumes`/`sidecarVolumeMounts`?** It signals clear intent (admin-controlled secrets for the sidecar), limits the surface to Secrets rather than arbitrary volumes, and is easier to validate. The field can be generalised later without breaking the existing API. + +**Trust model progression:** + +The CRD field for declaring secrets is stable across all the following trust levels — isolation improvements are additive webhook implementation details, not API changes. + +- **Alpha (filesystem isolation)**: Secrets are mounted only on the sidecar container. The app container cannot read the files because it does not have a volume mount for them. This provides defence against accidental leakage but not against a deliberately hostile app container (which could, in principle, access the files via `/proc//root` if the process runs in the same PID namespace). +- **Future (network isolation)**: An opt-in iptables init container (following the Istio model) could prevent the app container from reaching the services the secrets grant access to. This requires relaxing the pod security profile to allow `NET_ADMIN` on the init container, so it would be opt-in rather than default. +- **Further future (container-level network namespaces)**: Kubernetes may add container-level network namespaces, providing network isolation without requiring `NET_ADMIN`. + ### Configuration drift detection The webhook stamps each injected pod with a `sidecar.kroxylicious.io/config-generation` annotation recording the `metadata.generation` of the `KroxyliciousSidecarConfig` at injection time. This annotation serves two purposes: From 8efb9e5d925b9ff0a34525279ec904f356b395c1 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Wed, 6 May 2026 11:14:41 +1200 Subject: [PATCH 14/23] Unify injection label in design doc Use sidecar.kroxylicious.io/injection: disabled for pod opt-out, matching the namespace label key. Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 99f6260..7d621a1 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -17,6 +17,7 @@ A sidecar model is useful when: - The application should connect to Kafka via `localhost` rather than through a shared proxy tier. - Per-pod filter configuration is needed (e.g. different encryption keys per tenant). - The organisation prefers a service mesh-style deployment where each pod carries its own proxy. +- The ownership of policy applied by the proxy is distinct from the ownership of the Kafka application. Manual sidecar construction is error-prone and creates a maintenance burden. A webhook automates injection, enforces a consistent security posture, and gives the webhook administrator control over what runs in the sidecar. @@ -29,9 +30,11 @@ The webhook operates under a two-party trust model: - **Webhook administrator**: controls what gets injected — the proxy image, target Kafka address, filter definitions, security context. These are not overridable by the app owner. - **Application pod owner**: can opt out of injection via pod labels, and can select a specific `KroxyliciousSidecarConfig` by name via the `sidecar.kroxylicious.io/config` annotation. +The initial implementation assumes that the application pod owner is not adversarial in the security sense: They're not actively trying to subvert the policies being enforced by the injected proxy. + Pod annotations in the `sidecar.kroxylicious.io/` namespace form the building blocks for this trust boundary: * Some annotations are always set by the webhook. -For example, the webhook generates proxy configuration YAML from the `KroxyliciousSidecarConfig` and stores it in a `sidecar.kroxylicious.io/proxy-config` pod annotation (see [Config injection](#config-injection)). +For example, the webhook generates proxy configuration YAML based on the `KroxyliciousSidecarConfig` and stores it in a `sidecar.kroxylicious.io/proxy-config` pod annotation (see [Config injection](#config-injection)). This annotation is projected into the sidecar container as a file via a `downwardAPI` volume. The webhook always overwrites `sidecar.kroxylicious.io/proxy-config` on the pod, regardless of any value the app owner may have set. * The `sidecar.kroxylicious.io/config` annotation allows the app owner to select which `KroxyliciousSidecarConfig` applies when multiple exist in the namespace. @@ -46,7 +49,7 @@ Injection is opt-in at the namespace level and opt-out at the pod level, followi | Mechanism | Key | Effect | |-----------|-----|--------| | Namespace label | `sidecar.kroxylicious.io/injection: enabled` | Webhook intercepts pod creates in this namespace | -| Pod label | `sidecar.kroxylicious.io/inject: "false"` | Pod is excluded via `objectSelector` — never reaches the webhook | +| Pod label | `sidecar.kroxylicious.io/injection: disabled` | Pod is excluded via `objectSelector` — never reaches the webhook | The `MutatingWebhookConfiguration` uses `namespaceSelector` to scope interception and `objectSelector` to exclude opted-out pods. The webhook itself is idempotent: if a container named `kroxylicious-proxy` already exists, injection is skipped. From 8155e3776d94f432b501ef0d29cd8535a40f6864 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Wed, 6 May 2026 11:25:48 +1200 Subject: [PATCH 15/23] Document annotation-based idempotency as future improvement The current container-name check is weak against adversarial pods and does not detect config drift during reinvocation. Document comparing the proxy-config annotation against the expected output as a stronger alternative, noting that Jackson serialisation is deterministic. Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 7d621a1..9c001a6 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -51,7 +51,7 @@ Injection is opt-in at the namespace level and opt-out at the pod level, followi | Namespace label | `sidecar.kroxylicious.io/injection: enabled` | Webhook intercepts pod creates in this namespace | | Pod label | `sidecar.kroxylicious.io/injection: disabled` | Pod is excluded via `objectSelector` — never reaches the webhook | -The `MutatingWebhookConfiguration` uses `namespaceSelector` to scope interception and `objectSelector` to exclude opted-out pods. The webhook itself is idempotent: if a container named `kroxylicious-proxy` already exists, injection is skipped. +The `MutatingWebhookConfiguration` uses `namespaceSelector` to scope interception and `objectSelector` to exclude opted-out pods. The webhook itself is idempotent: if a container named `kroxylicious-proxy` already exists, injection is skipped. This is a name-based check, so a pod owner could circumvent injection by pre-adding a container with that name. This is accepted for the alpha — a determined pod owner can also opt out via labels. See [Configuration drift detection](#configuration-drift-detection) for a stronger approach planned for a future iteration. The failure policy of the webhook will be configurable. It will default to fail closed (`failurePolicy: Fail`), which is safe, but sacrifices availability of the Kubernetes control plane to admit workloads in cases where the webhook experiences internal errors. @@ -247,6 +247,14 @@ kubectl get pods -n my-ns -o json | jq '[.items[] | In a future iteration, a reconciler could watch for pods with outdated generations and surface an `UpToDate` condition on the `KroxyliciousSidecarConfig` status, giving operators visibility into configuration drift without requiring manual queries. +#### Annotation-based idempotency (future) + +The current idempotency check (container name) is weak: it can be circumvented by a pod owner pre-adding a container named `kroxylicious-proxy`, and it does not detect configuration drift during reinvocation. A stronger approach is to compare the `sidecar.kroxylicious.io/proxy-config` annotation on the pod against the configuration the webhook would generate right now. If the annotation is absent or differs, the webhook injects (or re-injects); if it matches, injection is skipped. + +This works because `ProxyConfigGenerator.generateConfig()` is deterministic: given the same `KroxyliciousSidecarConfigSpec` inputs, Jackson serialisation produces identical YAML. String equality on the annotation value is therefore a reliable idempotency check within the same webhook version. This also handles the reinvocation case (`reinvocationPolicy: IfNeeded`): if another mutating webhook modifies the pod after initial injection, the webhook can verify that the proxy config annotation is still correct. + +The generation stamp remains useful for fleet-wide drift detection (comparing against the current `KroxyliciousSidecarConfig` generation without reconstructing the expected config), but the annotation comparison becomes the primary idempotency mechanism. + ### Third-party plugin support #### The problem From 55ac3a28dec64d30efafd23f3281c217c359f552 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Wed, 6 May 2026 12:15:28 +1200 Subject: [PATCH 16/23] Move CRD to sidecar.kroxylicious.io group and add skip labelling Update CRD group from kroxylicious.io to sidecar.kroxylicious.io to avoid conflicts with the operator. Add skip labelling section describing the injection-skipped label for operator observability. Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 9c001a6..9fed0a7 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -53,6 +53,24 @@ Injection is opt-in at the namespace level and opt-out at the pod level, followi The `MutatingWebhookConfiguration` uses `namespaceSelector` to scope interception and `objectSelector` to exclude opted-out pods. The webhook itself is idempotent: if a container named `kroxylicious-proxy` already exists, injection is skipped. This is a name-based check, so a pod owner could circumvent injection by pre-adding a container with that name. This is accepted for the alpha — a determined pod owner can also opt out via labels. See [Configuration drift detection](#configuration-drift-detection) for a stronger approach planned for a future iteration. +#### Skip labelling + +When the webhook skips injection for a reason other than pod opt-out, it labels the pod with `sidecar.kroxylicious.io/injection-skipped` so that operators can find affected pods without grepping logs. The label value indicates the reason: + +| Value | Meaning | +|-------|---------| +| `no-config` | No `KroxyliciousSidecarConfig` was found for the pod's namespace | +| `already-injected` | A container named `kroxylicious-proxy` already exists in the pod | + +Pods that opted out via `sidecar.kroxylicious.io/injection: disabled` are not labelled — they already carry a label that identifies them. + +This allows operators to enumerate skipped pods: + +``` +kubectl get pods -l sidecar.kroxylicious.io/injection-skipped +kubectl get pods -l sidecar.kroxylicious.io/injection-skipped=no-config +``` + The failure policy of the webhook will be configurable. It will default to fail closed (`failurePolicy: Fail`), which is safe, but sacrifices availability of the Kubernetes control plane to admit workloads in cases where the webhook experiences internal errors. When configured to fail open and the webhook experiences an internal errors, it will log the error and return `allowed: true`; the pod will be admitted unmodified. @@ -69,7 +87,7 @@ SASL handling (e.g. rejecting downstream SASL handshakes or requiring proxy-init ### CRD: `KroxyliciousSidecarConfig` -A namespaced CRD (group `kroxylicious.io`, version `v1alpha1`) defines the sidecar configuration. The webhook admin creates one per namespace. The following edge cases are handled: +A namespaced CRD (group `sidecar.kroxylicious.io`, version `v1alpha1`) defines the sidecar configuration. The webhook admin creates one per namespace. The following edge cases are handled: 1. **No config in namespace**: the pod is admitted without injection (debug log only). This is the common case for namespaces where the admin has enabled the namespace label but not yet created a config. 2. **Multiple configs in namespace**: the pod is admitted without injection (warning logged). The pod can select a specific config via the `sidecar.kroxylicious.io/config` annotation; without this annotation the webhook cannot choose and skips injection. @@ -77,7 +95,7 @@ A namespaced CRD (group `kroxylicious.io`, version `v1alpha1`) defines the sidec 4. **Config is invalid in a way only the proxy can detect** (e.g. unreachable target Kafka cluster, wrong TLS trust anchor, non-existent filter type): the webhook injects the sidecar normally. The proxy will fail its startup probe and the pod will not become ready, surfacing the problem via standard Kubernetes health-check mechanisms. ```yaml -apiVersion: kroxylicious.io/v1alpha1 +apiVersion: sidecar.kroxylicious.io/v1alpha1 kind: KroxyliciousSidecarConfig metadata: name: my-config From de4bdc2acb13883aef0108065d8089ba6fdc8987 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Wed, 6 May 2026 12:28:05 +1200 Subject: [PATCH 17/23] Fix typo and reorder sections to eliminate forward references Move Config injection and Configuration drift detection before Status, since Status references the config-generation annotation those sections introduce. Fix "an internal errors" typo. Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 121 +++++++++++---------- 1 file changed, 61 insertions(+), 60 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 9fed0a7..ebdd28a 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -53,27 +53,10 @@ Injection is opt-in at the namespace level and opt-out at the pod level, followi The `MutatingWebhookConfiguration` uses `namespaceSelector` to scope interception and `objectSelector` to exclude opted-out pods. The webhook itself is idempotent: if a container named `kroxylicious-proxy` already exists, injection is skipped. This is a name-based check, so a pod owner could circumvent injection by pre-adding a container with that name. This is accepted for the alpha — a determined pod owner can also opt out via labels. See [Configuration drift detection](#configuration-drift-detection) for a stronger approach planned for a future iteration. -#### Skip labelling - -When the webhook skips injection for a reason other than pod opt-out, it labels the pod with `sidecar.kroxylicious.io/injection-skipped` so that operators can find affected pods without grepping logs. The label value indicates the reason: - -| Value | Meaning | -|-------|---------| -| `no-config` | No `KroxyliciousSidecarConfig` was found for the pod's namespace | -| `already-injected` | A container named `kroxylicious-proxy` already exists in the pod | - -Pods that opted out via `sidecar.kroxylicious.io/injection: disabled` are not labelled — they already carry a label that identifies them. - -This allows operators to enumerate skipped pods: - -``` -kubectl get pods -l sidecar.kroxylicious.io/injection-skipped -kubectl get pods -l sidecar.kroxylicious.io/injection-skipped=no-config -``` - The failure policy of the webhook will be configurable. It will default to fail closed (`failurePolicy: Fail`), which is safe, but sacrifices availability of the Kubernetes control plane to admit workloads in cases where the webhook experiences internal errors. -When configured to fail open and the webhook experiences an internal errors, it will log the error and return `allowed: true`; the pod will be admitted unmodified. +When configured to fail open and the webhook experiences an internal error, it will log the error and return `allowed: true`; the pod will be admitted unmodified. + #### Bypass prevention @@ -87,12 +70,7 @@ SASL handling (e.g. rejecting downstream SASL handshakes or requiring proxy-init ### CRD: `KroxyliciousSidecarConfig` -A namespaced CRD (group `sidecar.kroxylicious.io`, version `v1alpha1`) defines the sidecar configuration. The webhook admin creates one per namespace. The following edge cases are handled: - -1. **No config in namespace**: the pod is admitted without injection (debug log only). This is the common case for namespaces where the admin has enabled the namespace label but not yet created a config. -2. **Multiple configs in namespace**: the pod is admitted without injection (warning logged). The pod can select a specific config via the `sidecar.kroxylicious.io/config` annotation; without this annotation the webhook cannot choose and skips injection. -3. **Config is invalid in a way the webhook can detect** (e.g. missing required fields): the webhook logs a warning and admits the pod without injection. Consistent with fail-open semantics. -4. **Config is invalid in a way only the proxy can detect** (e.g. unreachable target Kafka cluster, wrong TLS trust anchor, non-existent filter type): the webhook injects the sidecar normally. The proxy will fail its startup probe and the pod will not become ready, surfacing the problem via standard Kubernetes health-check mechanisms. +A namespaced CRD (group `sidecar.kroxylicious.io`, version `v1alpha1`) defines the sidecar configuration. ```yaml apiVersion: sidecar.kroxylicious.io/v1alpha1 @@ -141,6 +119,64 @@ The `virtualClusters` list contains per-cluster settings (target bootstrap addre **Why not reuse the operator's CRDs?** The operator CRDs model a shared proxy deployment with ingress networking, multi-cluster support, and cross-resource references. The sidecar use case is fundamentally simpler — localhost binding, no ingress, a single virtual cluster in the alpha. Coupling them would constrain both models. +The webhook admin creates one per namespace. The following edge cases are handled: + +1. **No config in namespace**: the pod is admitted without injection (debug log only). This is the common case for namespaces where the admin has enabled the namespace label but not yet created a config. +2. **Multiple configs in namespace**: the pod is admitted without injection (warning logged). The pod can select a specific config via the `sidecar.kroxylicious.io/config` annotation; without this annotation the webhook cannot choose and skips injection. +3. **Config is invalid in a way the webhook can detect** (e.g. missing required fields): the webhook logs a warning and admits the pod without injection. Consistent with fail-open semantics. +4. **Config is invalid in a way only the proxy can detect** (e.g. unreachable target Kafka cluster, wrong TLS trust anchor, non-existent filter type): the webhook injects the sidecar normally. The proxy will fail its startup probe and the pod will not become ready, surfacing the problem via standard Kubernetes health-check mechanisms. + +When the webhook skips injection for a reason other than pod opt-out, it labels the pod with `sidecar.kroxylicious.io/injection-skipped` so that operators can find affected pods without grepping logs. The label value indicates the reason: + +| Value | Meaning | +|-------|---------| +| `no-config` | No `KroxyliciousSidecarConfig` was found for the pod's namespace | +| `already-injected` | A container named `kroxylicious-proxy` already exists in the pod | + +Pods that opted out via `sidecar.kroxylicious.io/injection: disabled` are not labelled — they already carry a label that identifies them. + +This allows operators to enumerate skipped pods: + +``` +kubectl get pods -l sidecar.kroxylicious.io/injection-skipped +kubectl get pods -l sidecar.kroxylicious.io/injection-skipped=no-config +``` + +### Config injection + +The webhook generates proxy configuration YAML from the `KroxyliciousSidecarConfig` spec, using the same `Configuration` model from `kroxylicious-runtime`. The generated config is stored in a pod annotation (`sidecar.kroxylicious.io/proxy-config`) and projected into the sidecar container via a `downwardAPI` volume. + +This avoids creating per-pod ConfigMaps, which would require additional RBAC, lifecycle management for orphaned ConfigMaps, and unique name generation. The annotation approach is self-contained within the pod. + +A typical sidecar config is a few hundred bytes, well within the ~256KB practical annotation size limit. + +### Configuration drift detection + +The webhook stamps each injected pod with a `sidecar.kroxylicious.io/config-generation` annotation recording the `metadata.generation` of the `KroxyliciousSidecarConfig` at injection time. This annotation serves two purposes: + +1. **Idempotency guard**: its presence indicates that the sidecar has already been injected, preventing re-injection when the webhook is reinvoked. +2. **Drift detection**: its value can be compared (equality only) with the current generation of the `KroxyliciousSidecarConfig` to identify pods running stale configuration. + +Because the webhook only mutates pods at creation time, configuration changes to `KroxyliciousSidecarConfig` do not propagate to running pods. This matches how Istio and Linkerd handle sidecar injection. Users must restart pods to pick up new configuration. + +The generation stamp allows operators to identify stale pods: + +``` +kubectl get pods -n my-ns -o json | jq '[.items[] | + select(.metadata.annotations["sidecar.kroxylicious.io/config-generation"] != null) | + {name: .metadata.name, generation: .metadata.annotations["sidecar.kroxylicious.io/config-generation"]}]' +``` + +In a future iteration, a reconciler could watch for pods with outdated generations and surface an `UpToDate` condition on the `KroxyliciousSidecarConfig` status, giving operators visibility into configuration drift without requiring manual queries. + +#### Annotation-based idempotency (future) + +The current idempotency check (container name) is weak: it can be circumvented by a pod owner pre-adding a container named `kroxylicious-proxy`, and it does not detect configuration drift during reinvocation. A stronger approach is to compare the `sidecar.kroxylicious.io/proxy-config` annotation on the pod against the configuration the webhook would generate right now. If the annotation is absent or differs, the webhook injects (or re-injects); if it matches, injection is skipped. + +This works because `ProxyConfigGenerator.generateConfig()` is deterministic: given the same `KroxyliciousSidecarConfigSpec` inputs, Jackson serialisation produces identical YAML. String equality on the annotation value is therefore a reliable idempotency check within the same webhook version. This also handles the reinvocation case (`reinvocationPolicy: IfNeeded`): if another mutating webhook modifies the pod after initial injection, the webhook can verify that the proxy config annotation is still correct. + +The generation stamp remains useful for fleet-wide drift detection (comparing against the current `KroxyliciousSidecarConfig` generation without reconstructing the expected config), but the annotation comparison becomes the primary idempotency mechanism. + #### Status The CRD has a `status` subresource with the following fields: @@ -174,14 +210,6 @@ status: This gives operators visibility into whether the webhook has picked up the latest configuration, complementing the per-pod `sidecar.kroxylicious.io/config-generation` annotation for drift detection. The user documentation should describe how these mechanisms work together and how operators are expected to use them to reason about the state of their sidecar fleet. -### Config injection - -The webhook generates proxy configuration YAML from the `KroxyliciousSidecarConfig` spec, using the same `Configuration` model from `kroxylicious-runtime`. The generated config is stored in a pod annotation (`sidecar.kroxylicious.io/proxy-config`) and projected into the sidecar container via a `downwardAPI` volume. - -This avoids creating per-pod ConfigMaps, which would require additional RBAC, lifecycle management for orphaned ConfigMaps, and unique name generation. The annotation approach is self-contained within the pod. - -A typical sidecar config is a few hundred bytes, well within the ~256KB practical annotation size limit. - ### Port allocation | Port | Purpose | Bind address | @@ -246,33 +274,6 @@ The CRD field for declaring secrets is stable across all the following trust lev - **Future (network isolation)**: An opt-in iptables init container (following the Istio model) could prevent the app container from reaching the services the secrets grant access to. This requires relaxing the pod security profile to allow `NET_ADMIN` on the init container, so it would be opt-in rather than default. - **Further future (container-level network namespaces)**: Kubernetes may add container-level network namespaces, providing network isolation without requiring `NET_ADMIN`. -### Configuration drift detection - -The webhook stamps each injected pod with a `sidecar.kroxylicious.io/config-generation` annotation recording the `metadata.generation` of the `KroxyliciousSidecarConfig` at injection time. This annotation serves two purposes: - -1. **Idempotency guard**: its presence indicates that the sidecar has already been injected, preventing re-injection when the webhook is reinvoked. -2. **Drift detection**: its value can be compared (equality only) with the current generation of the `KroxyliciousSidecarConfig` to identify pods running stale configuration. - -Because the webhook only mutates pods at creation time, configuration changes to `KroxyliciousSidecarConfig` do not propagate to running pods. This matches how Istio and Linkerd handle sidecar injection. Users must restart pods to pick up new configuration. - -The generation stamp allows operators to identify stale pods: - -``` -kubectl get pods -n my-ns -o json | jq '[.items[] | - select(.metadata.annotations["sidecar.kroxylicious.io/config-generation"] != null) | - {name: .metadata.name, generation: .metadata.annotations["sidecar.kroxylicious.io/config-generation"]}]' -``` - -In a future iteration, a reconciler could watch for pods with outdated generations and surface an `UpToDate` condition on the `KroxyliciousSidecarConfig` status, giving operators visibility into configuration drift without requiring manual queries. - -#### Annotation-based idempotency (future) - -The current idempotency check (container name) is weak: it can be circumvented by a pod owner pre-adding a container named `kroxylicious-proxy`, and it does not detect configuration drift during reinvocation. A stronger approach is to compare the `sidecar.kroxylicious.io/proxy-config` annotation on the pod against the configuration the webhook would generate right now. If the annotation is absent or differs, the webhook injects (or re-injects); if it matches, injection is skipped. - -This works because `ProxyConfigGenerator.generateConfig()` is deterministic: given the same `KroxyliciousSidecarConfigSpec` inputs, Jackson serialisation produces identical YAML. String equality on the annotation value is therefore a reliable idempotency check within the same webhook version. This also handles the reinvocation case (`reinvocationPolicy: IfNeeded`): if another mutating webhook modifies the pod after initial injection, the webhook can verify that the proxy config annotation is still correct. - -The generation stamp remains useful for fleet-wide drift detection (comparing against the current `KroxyliciousSidecarConfig` generation without reconstructing the expected config), but the annotation comparison becomes the primary idempotency mechanism. - ### Third-party plugin support #### The problem From 7dc55dfa82f99fa53fa1c2f6ab55ec3032519f63 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Wed, 6 May 2026 13:00:02 +1200 Subject: [PATCH 18/23] Add multiple-configs to injection-skipped label values Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index ebdd28a..8c419a3 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -132,6 +132,7 @@ When the webhook skips injection for a reason other than pod opt-out, it labels |-------|---------| | `no-config` | No `KroxyliciousSidecarConfig` was found for the pod's namespace | | `already-injected` | A container named `kroxylicious-proxy` already exists in the pod | +| `multiple-configs` | Multiple `KroxyliciousSidecarConfig` resources exist in the namespace and no explicit config was selected via annotation | Pods that opted out via `sidecar.kroxylicious.io/injection: disabled` are not labelled — they already carry a label that identifies them. From bb06f73b8595ef96e62503721ed2ac06048e1e03 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Wed, 6 May 2026 13:04:50 +1200 Subject: [PATCH 19/23] Tweak title to placate angry automation gods Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 8c419a3..73cf4da 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -1,4 +1,4 @@ -# Proposal 016 — Sidecar Injection Webhook +# 100 - Sidecar Injection Webhook A Kubernetes mutating admission webhook that automatically injects a Kroxylicious proxy sidecar into application pods. The sidecar intercepts Kafka traffic on localhost, allowing filters to be applied transparently without changes to the application. From bc6e70d6b07bda9502bf432e6647b6e92cfe3a3d Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 7 May 2026 16:43:06 +1200 Subject: [PATCH 20/23] Address review feedback from Sam on sidecar injection proposal Rename injection-skipped label values to reference CRD kind directly, rewrite idempotency section to state requirements not implementation, add Ready=False for invalid configs, rename FEATURE_GATES to K8S_FEATURE_GATES, and add explicit seccompProfile: RuntimeDefault. Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 47 ++++++++++++++++------ 1 file changed, 34 insertions(+), 13 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 73cf4da..9524d5e 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -130,17 +130,17 @@ When the webhook skips injection for a reason other than pod opt-out, it labels | Value | Meaning | |-------|---------| -| `no-config` | No `KroxyliciousSidecarConfig` was found for the pod's namespace | -| `already-injected` | A container named `kroxylicious-proxy` already exists in the pod | -| `multiple-configs` | Multiple `KroxyliciousSidecarConfig` resources exist in the namespace and no explicit config was selected via annotation | +| `no-KroxyliciousSidecarConfig` | No `KroxyliciousSidecarConfig` was found for the pod's namespace | +| `ambiguous-KroxyliciousSidecarConfig` | Multiple `KroxyliciousSidecarConfig` resources exist in the namespace and no explicit config was selected via annotation | +| `container-name-conflict` | A container named `kroxylicious-proxy` already exists in the pod | -Pods that opted out via `sidecar.kroxylicious.io/injection: disabled` are not labelled — they already carry a label that identifies them. +Pods that opted out via `sidecar.kroxylicious.io/injection: disabled` are not labelled — they already carry a label that identifies them. Reinvocation of the webhook (e.g. due to `reinvocationPolicy: IfNeeded`) is expected Kubernetes behaviour and does not set the label. This allows operators to enumerate skipped pods: ``` kubectl get pods -l sidecar.kroxylicious.io/injection-skipped -kubectl get pods -l sidecar.kroxylicious.io/injection-skipped=no-config +kubectl get pods -l sidecar.kroxylicious.io/injection-skipped=no-KroxyliciousSidecarConfig ``` ### Config injection @@ -174,7 +174,9 @@ In a future iteration, a reconciler could watch for pods with outdated generatio The current idempotency check (container name) is weak: it can be circumvented by a pod owner pre-adding a container named `kroxylicious-proxy`, and it does not detect configuration drift during reinvocation. A stronger approach is to compare the `sidecar.kroxylicious.io/proxy-config` annotation on the pod against the configuration the webhook would generate right now. If the annotation is absent or differs, the webhook injects (or re-injects); if it matches, injection is skipped. -This works because `ProxyConfigGenerator.generateConfig()` is deterministic: given the same `KroxyliciousSidecarConfigSpec` inputs, Jackson serialisation produces identical YAML. String equality on the annotation value is therefore a reliable idempotency check within the same webhook version. This also handles the reinvocation case (`reinvocationPolicy: IfNeeded`): if another mutating webhook modifies the pod after initial injection, the webhook can verify that the proxy config annotation is still correct. +For annotation comparison to be a reliable idempotency check, config generation must be deterministic: the same `KroxyliciousSidecarConfigSpec` must always produce byte-identical serialised output within a given webhook version. The implementation must guarantee this; how it does so is not a concern of this proposal. String equality on the annotation value is therefore a reliable idempotency check within the same webhook version. This also handles the reinvocation case (`reinvocationPolicy: IfNeeded`): if another mutating webhook modifies the pod after initial injection, the webhook can verify that the proxy config annotation is still correct. + +A webhook upgrade that changes serialisation output means existing pods' annotations no longer match what the webhook would generate, making them appear as configuration drift even though the `KroxyliciousSidecarConfig` itself hasn't changed. Since nothing proactively re-injects running pods, remediation requires pods to cycle. The generation stamp remains useful for fleet-wide drift detection (comparing against the current `KroxyliciousSidecarConfig` generation without reconstructing the expected config), but the annotation comparison becomes the primary idempotency mechanism. @@ -189,13 +191,16 @@ The CRD has a `status` subresource with the following fields: The webhook maintains a single condition type: -| Condition | Meaning | -|-----------|---------| -| `Ready` | The webhook has observed and accepted this configuration. `observedGeneration` on the condition tracks which generation was acknowledged. | +| Condition | Status | Reason | Meaning | +|-----------|--------|--------|---------| +| `Ready` | `True` | `Accepted` | The webhook has observed and accepted this configuration. | +| `Ready` | `False` | `Invalid` | The webhook has observed this configuration and determined it is invalid. The `message` field describes the problem. | + +The webhook sets `Ready=True` (reason `Accepted`) when it first observes a valid config via its informer. If the webhook can determine that the config is invalid (e.g. missing required fields beyond what the CRD schema enforces), it sets `Ready=False` (reason `Invalid`) with a descriptive message, surfacing the problem directly on the CRD where an operator would naturally look. The condition is only updated when the generation changes, avoiding unnecessary status writes. Status update failures (e.g. conflicts) should be retried; persistent failures should be surfaced via logging so that operators can investigate. A status update failure does not block pod admission, but a `KroxyliciousSidecarConfig` with a stale or missing `Ready` condition indicates a problem that needs attention. -The webhook sets `Ready=True` (reason `Accepted`) when it first observes the config via its informer. The condition is only updated when the generation changes, avoiding unnecessary status writes. Status update failures (e.g. conflicts) should be retried; persistent failures should be surfaced via logging so that operators can investigate. A status update failure does not block pod admission, but a `KroxyliciousSidecarConfig` with a stale or missing `Ready` condition indicates a problem that needs attention. +Per-namespace issues (no config found, multiple ambiguous configs) are not properties of a single CRD and are surfaced via the `sidecar.kroxylicious.io/injection-skipped` label on affected pods. -Example status: +Example status (valid config): ```yaml status: @@ -209,6 +214,20 @@ status: observedGeneration: 3 ``` +Example status (invalid config): + +```yaml +status: + observedGeneration: 2 + conditions: + - type: Ready + status: "False" + reason: Invalid + message: "spec.virtualClusters[0].targetBootstrapServers is required" + lastTransitionTime: "2025-01-15T10:25:00Z" + observedGeneration: 2 +``` + This gives operators visibility into whether the webhook has picked up the latest configuration, complementing the per-pod `sidecar.kroxylicious.io/config-generation` annotation for drift detection. The user documentation should describe how these mechanisms work together and how operators are expected to use them to reason about the state of their sidecar fleet. ### Port allocation @@ -227,18 +246,20 @@ The management endpoint binds to `0.0.0.0` because kubelet HTTP probes target th On Kubernetes 1.29+ (where the `SidecarContainers` feature gate is enabled by default), the webhook injects the proxy as a native sidecar — an init container with `restartPolicy: Always`. This gives proper startup ordering (proxy starts before the application) and shutdown ordering (proxy stops after the application). On older clusters, the webhook falls back to injecting into `spec.containers`. -The webhook detects the cluster's Kubernetes version at startup and uses it to infer which features are available by default. However, the Kubernetes API server version does not reveal which alpha or beta feature gates are actually enabled on the cluster. Rather than requiring additional RBAC to query node or API server configuration, the webhook lets the deployer set the `FEATURE_GATES` environment variable explicitly (e.g. `FEATURE_GATES=SidecarContainers=true,ImageVolume=true`). This overrides the version-based defaults and is the recommended approach for clusters running features ahead of their default-on version (e.g. native sidecars on 1.28, OCI image volumes on 1.31-1.32). +The webhook detects the cluster's Kubernetes version at startup and uses it to infer which features are available by default. However, the Kubernetes API server version does not reveal which alpha or beta feature gates are actually enabled on the cluster. Rather than requiring additional RBAC to query node or API server configuration, the webhook supports a `K8S_FEATURE_GATES` environment variable as an escape hatch (e.g. `K8S_FEATURE_GATES=SidecarContainers=true,ImageVolume=true`). This overrides the version-based defaults for clusters running features ahead of their default-on version (e.g. native sidecars on 1.28, OCI image volumes on 1.31-1.32). Deployers who set this variable are responsible for keeping it in sync with their cluster configuration; version-based detection is the recommended default. ### Sidecar container spec The injected sidecar follows the same patterns as `ProxyDeploymentDependentResource` in the operator: -- Container-level `securityContext`: `allowPrivilegeEscalation: false`, `capabilities: drop ALL`, `readOnlyRootFilesystem: true` +- Container-level `securityContext`: `allowPrivilegeEscalation: false`, `capabilities: drop ALL`, `readOnlyRootFilesystem: true`, `seccompProfile: RuntimeDefault` - Probes: `startupProbe` (initialDelay 5s, period 2s, failure threshold 30), `livenessProbe` (initialDelay 30s, period 10s, failure threshold 3), `readinessProbe` (initialDelay 5s, period 2s, failure threshold 5) — all HTTP GET `/livez` on the management port (default 9082) - `terminationMessagePolicy: FallbackToLogsOnError` The webhook does not set a pod-level `securityContext`. Pod-level security policies (e.g. `runAsNonRoot`, `seccompProfile`) are the responsibility of the cluster admin via Kubernetes `PodSecurity` admission (`pod-security.kubernetes.io/enforce: restricted` namespace label) or equivalent policy enforcement. Setting pod-level security context from a mutating webhook risks ordering conflicts with other webhooks. +The sidecar explicitly sets `seccompProfile: RuntimeDefault` at the container level, so it does not inherit a permissive pod-level profile (e.g. `Unconfined`). Container-level seccomp profiles override pod-level, so this does not affect the application container. Future iterations could allow configuring the sidecar's seccomp profile via `KroxyliciousSidecarConfig` if the proxy itself needs a different profile (e.g. for io_uring). + The container-level security context is never weakened. If the pod already has a stricter security context, it is preserved. ### Target cluster TLS From 486ce341ce753146e77c2c45c76e8f2a7b7a0bec Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 7 May 2026 21:16:53 +1200 Subject: [PATCH 21/23] Address Rob's review feedback on sidecar injection proposal Update proposal to reflect implementation: multi-replica deployment with PDB, "one or more" configs per namespace, replace app-level FAILURE_POLICY with K8s failurePolicy for errors and new UNINJECTED_POD_POLICY for config edge cases. Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 9524d5e..59bf218 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -53,9 +53,9 @@ Injection is opt-in at the namespace level and opt-out at the pod level, followi The `MutatingWebhookConfiguration` uses `namespaceSelector` to scope interception and `objectSelector` to exclude opted-out pods. The webhook itself is idempotent: if a container named `kroxylicious-proxy` already exists, injection is skipped. This is a name-based check, so a pod owner could circumvent injection by pre-adding a container with that name. This is accepted for the alpha — a determined pod owner can also opt out via labels. See [Configuration drift detection](#configuration-drift-detection) for a stronger approach planned for a future iteration. -The failure policy of the webhook will be configurable. -It will default to fail closed (`failurePolicy: Fail`), which is safe, but sacrifices availability of the Kubernetes control plane to admit workloads in cases where the webhook experiences internal errors. -When configured to fail open and the webhook experiences an internal error, it will log the error and return `allowed: true`; the pod will be admitted unmodified. +The `MutatingWebhookConfiguration` sets `failurePolicy: Fail` (fail-closed). If the webhook is unreachable or returns an error, Kubernetes rejects the pod. On unexpected internal errors, the webhook returns HTTP 500 and lets the K8s failure policy govern the outcome — there is no separate application-level failure policy. + +A separate `UNINJECTED_POD_POLICY` environment variable (default `Admit`) controls what happens when the webhook successfully processes a request but cannot inject — for example, because there is no `KroxyliciousSidecarConfig` in the namespace or multiple configs exist without an explicit selection. When set to `Admit`, the pod is admitted without injection (consistent with Istio/Linkerd behaviour). When set to `Deny`, the pod is rejected, ensuring no workload runs un-proxied in a namespace where injection is expected. #### Bypass prevention @@ -119,11 +119,11 @@ The `virtualClusters` list contains per-cluster settings (target bootstrap addre **Why not reuse the operator's CRDs?** The operator CRDs model a shared proxy deployment with ingress networking, multi-cluster support, and cross-resource references. The sidecar use case is fundamentally simpler — localhost binding, no ingress, a single virtual cluster in the alpha. Coupling them would constrain both models. -The webhook admin creates one per namespace. The following edge cases are handled: +The webhook admin creates one or more per namespace. The following edge cases are handled: -1. **No config in namespace**: the pod is admitted without injection (debug log only). This is the common case for namespaces where the admin has enabled the namespace label but not yet created a config. -2. **Multiple configs in namespace**: the pod is admitted without injection (warning logged). The pod can select a specific config via the `sidecar.kroxylicious.io/config` annotation; without this annotation the webhook cannot choose and skips injection. -3. **Config is invalid in a way the webhook can detect** (e.g. missing required fields): the webhook logs a warning and admits the pod without injection. Consistent with fail-open semantics. +1. **No config in namespace**: governed by `UNINJECTED_POD_POLICY`. When `Admit` (default), the pod is admitted without injection (debug log only) — this is the common case for namespaces where the admin has enabled the namespace label but not yet created a config. When `Deny`, the pod is rejected. +2. **Multiple configs in namespace**: governed by `UNINJECTED_POD_POLICY`. The pod can select a specific config via the `sidecar.kroxylicious.io/config` annotation; without this annotation the webhook cannot choose. When `Admit`, the pod is admitted without injection (warning logged). When `Deny`, the pod is rejected. +3. **Config is invalid in a way the webhook can detect** (e.g. missing required fields): governed by `UNINJECTED_POD_POLICY`. When `Admit`, the webhook logs a warning and admits the pod without injection. When `Deny`, the pod is rejected. 4. **Config is invalid in a way only the proxy can detect** (e.g. unreachable target Kafka cluster, wrong TLS trust anchor, non-existent filter type): the webhook injects the sidecar normally. The proxy will fail its startup probe and the pod will not become ready, surfacing the problem via standard Kubernetes health-check mechanisms. When the webhook skips injection for a reason other than pod opt-out, it labels the pod with `sidecar.kroxylicious.io/injection-skipped` so that operators can find affected pods without grepping logs. The label value indicates the reason: @@ -405,7 +405,7 @@ Annotation-based delegation could allow the app owner to override specific sidec ### Webhook deployment -The webhook is packaged as a container image and deployed as a single-replica `Deployment` in a dedicated `kroxylicious-webhook` namespace. Install manifests are provided for: +The webhook is packaged as a container image and deployed as a multi-replica `Deployment` (2 replicas, with a `PodDisruptionBudget` and pod anti-affinity) in a dedicated `kroxylicious-webhook` namespace. Install manifests are provided for: - Namespace, ServiceAccount, ClusterRole, ClusterRoleBinding - Deployment (port 8443) From 455ce2d937b5641e7872ed542f6c16d8b3bc0605 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 11 May 2026 10:14:17 +1200 Subject: [PATCH 22/23] RBAC corrections Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 59bf218..902f030 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -415,7 +415,12 @@ The webhook is packaged as a container image and deployed as a multi-replica `De **TLS**: Kubernetes requires HTTPS for admission webhooks. The primary path uses cert-manager with a self-signed issuer. A manual alternative (admin provides cert/key Secret) is documented. The webhook watches cert files for rotation and reloads the SSLContext. -**RBAC**: The webhook needs only `get`, `list`, `watch` on `KroxyliciousSidecarConfig` resources and `get`, `list`, `watch` on namespaces. No ConfigMap or Secret creation permissions are needed. +**RBAC**: The webhook needs only: +* `get`, `list`, `watch` on `KroxyliciousSidecarConfig` resources, +* `get`, `patch` and `update` on `KroxyliciousSidecarConfig/status`, + +No permissions on namespaces are needed: the selection/filtering of the namespaces is done by the Kubernete's admission controller, using the `MutatingWebhookConfiguration`'s `namespaceSelector`, rather than in the webhook itself. +No `ConfigMap` or `Secret` creation permissions are needed since references to these resources from the `KroxyliciousSidecarConfig` are not validated by the webhook. **HTTP server**: Uses the JDK built-in `HttpsServer` (same pattern as `OperatorMain.java`), serving `POST /mutate` and `GET /livez`. No additional HTTP framework dependencies. From fd472bf50e5ee0a10911f1447b9828165c44d122 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 11 May 2026 14:15:57 +1200 Subject: [PATCH 23/23] Add invalid-KroxyliciousSidecarConfig label and fix idempotency text Address Sam's review comments on the sidecar injection proposal: - Add invalid-KroxyliciousSidecarConfig to the injection-skipped label table with a note that structural checks are defensive (CRD schema catches those) while cross-field semantic checks are primary. - Fix contradictory idempotency text: config-generation annotation serves drift detection only; idempotency relies on container name. Assisted-By: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/100-sidecar-injection-webhook.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/proposals/100-sidecar-injection-webhook.md b/proposals/100-sidecar-injection-webhook.md index 902f030..94ad4fc 100644 --- a/proposals/100-sidecar-injection-webhook.md +++ b/proposals/100-sidecar-injection-webhook.md @@ -133,6 +133,9 @@ When the webhook skips injection for a reason other than pod opt-out, it labels | `no-KroxyliciousSidecarConfig` | No `KroxyliciousSidecarConfig` was found for the pod's namespace | | `ambiguous-KroxyliciousSidecarConfig` | Multiple `KroxyliciousSidecarConfig` resources exist in the namespace and no explicit config was selected via annotation | | `container-name-conflict` | A container named `kroxylicious-proxy` already exists in the pod | +| `invalid-KroxyliciousSidecarConfig` | The resolved `KroxyliciousSidecarConfig` failed webhook-side validation | + +In practice, `invalid-KroxyliciousSidecarConfig` is not expected to occur for structural issues because the CRD schema validation enforced by the API server covers those constraints. However, the webhook also validates cross-field semantic constraints that cannot be expressed in the OpenAPI schema (such as port collisions between `bootstrapPort` and `managementPort`, or broker port ranges exceeding 65535). The label and admission-time check exist as a defensive guard for structural issues and as the primary enforcement point for semantic issues. Pods that opted out via `sidecar.kroxylicious.io/injection: disabled` are not labelled — they already carry a label that identifies them. Reinvocation of the webhook (e.g. due to `reinvocationPolicy: IfNeeded`) is expected Kubernetes behaviour and does not set the label. @@ -153,10 +156,7 @@ A typical sidecar config is a few hundred bytes, well within the ~256KB practica ### Configuration drift detection -The webhook stamps each injected pod with a `sidecar.kroxylicious.io/config-generation` annotation recording the `metadata.generation` of the `KroxyliciousSidecarConfig` at injection time. This annotation serves two purposes: - -1. **Idempotency guard**: its presence indicates that the sidecar has already been injected, preventing re-injection when the webhook is reinvoked. -2. **Drift detection**: its value can be compared (equality only) with the current generation of the `KroxyliciousSidecarConfig` to identify pods running stale configuration. +The webhook stamps each injected pod with a `sidecar.kroxylicious.io/config-generation` annotation recording the `metadata.generation` of the `KroxyliciousSidecarConfig` at injection time. This annotation serves drift detection: its value can be compared (equality only) with the current generation of the `KroxyliciousSidecarConfig` to identify pods running stale configuration. Idempotency currently relies on the container name check described below. Because the webhook only mutates pods at creation time, configuration changes to `KroxyliciousSidecarConfig` do not propagate to running pods. This matches how Istio and Linkerd handle sidecar injection. Users must restart pods to pick up new configuration.