Add sidecar injection webhook proposal#100
Conversation
|
Discussed at Community Call 23rd April. Tom is looking for feedback. |
| | `kroxylicious.io/config-generation` | The `metadata.generation` of the `KroxyliciousSidecarConfig` at injection time | | ||
| | `kroxylicious.io/injection-timestamp` | ISO-8601 timestamp of when the sidecar was injected | | ||
|
|
||
| Because the webhook only mutates pods at creation time, configuration changes to `KroxyliciousSidecarConfig` do not propagate to running pods. This matches how Istio and Linkerd handle sidecar injection. Users must restart pods to pick up new configuration. |
There was a problem hiding this comment.
Not really. Because the proxy configuration is built from the KroxyliciousSidecarConfig at injection time it means that any subsequent changes to the KroxyliciousSidecarConfig are ignored until the next time the pod get created. That creates a lot of problems:
- The user who has RBAC on the
KroxyliciousSidecarConfigmight not have RBAC on thePod,Deploymentetc. So it assumes some kind of communication between those users. Such communication might not be possible in practice. - Erroneous changes to the
KroxyliciousSidecarConfigare only discovered at some unbounded time later.
The right way to solve this would be for the proxy to phone home to obtain a configuration from a control plane, and to receive updates to that configuration without the need for Pod restarts. But I really want to avoid boiling the ocean here by blocking this alpha sidecar feature on having such a control plane. I think the control plane piece is something we could add later (also as an alpha feature) and then maybe they would become stable features together later on.
|
Thanks @tombentley, looks like a solid proposal. I've left a few comments to be considered. |
tombentley
left a comment
There was a problem hiding this comment.
Thanks @k-wall @robobario, I think I've addresses your comments, though with a watering down of some of the initial ambition wrt trust boundaries. PTAL.
| | `kroxylicious.io/config-generation` | The `metadata.generation` of the `KroxyliciousSidecarConfig` at injection time | | ||
| | `kroxylicious.io/injection-timestamp` | ISO-8601 timestamp of when the sidecar was injected | | ||
|
|
||
| Because the webhook only mutates pods at creation time, configuration changes to `KroxyliciousSidecarConfig` do not propagate to running pods. This matches how Istio and Linkerd handle sidecar injection. Users must restart pods to pick up new configuration. |
There was a problem hiding this comment.
Not really. Because the proxy configuration is built from the KroxyliciousSidecarConfig at injection time it means that any subsequent changes to the KroxyliciousSidecarConfig are ignored until the next time the pod get created. That creates a lot of problems:
- The user who has RBAC on the
KroxyliciousSidecarConfigmight not have RBAC on thePod,Deploymentetc. So it assumes some kind of communication between those users. Such communication might not be possible in practice. - Erroneous changes to the
KroxyliciousSidecarConfigare only discovered at some unbounded time later.
The right way to solve this would be for the proxy to phone home to obtain a configuration from a control plane, and to receive updates to that configuration without the need for Pod restarts. But I really want to avoid boiling the ocean here by blocking this alpha sidecar feature on having such a control plane. I think the control plane piece is something we could add later (also as an alpha feature) and then maybe they would become stable features together later on.
Add PRs kroxylicious#96, kroxylicious#98, kroxylicious#99, kroxylicious#100, kroxylicious#101, and kroxylicious#103 which were opened after the initial script was created. Note: PR kroxylicious#100 is already correctly named (100-sidecar-injection-webhook.md). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This change simplifies the proposal numbering system by using PR numbers as proposal identifiers, eliminating number collisions and removing the need for a separate allocation process. Changes: - Simplified proposals/README.md to focus on author workflow - Removed index tables (directory listing serves as the index) - Streamlined instructions for creating and renaming proposals - Updated proposal template with workflow instructions - Require PR number in title format: # <PR#> - <Title> - Moved workflow instructions into comment block - Added GitHub workflow to automatically check proposal numbering - Validates both filename and title format - Updates PR description when proposal files don't match PR number - Provides exact commands to fix naming issues - Removes warning once corrected - Handles both added and renamed files - Runs on all PRs (ready for mandatory status check) - Added notification script for existing open PRs - After merge, run notify-open-prs.sh to ask authors to rebase - Workflow will automatically guide them through renaming - Updated with all current open proposal PRs (kroxylicious#70, kroxylicious#82, kroxylicious#83, kroxylicious#85, kroxylicious#88, kroxylicious#93, kroxylicious#94, kroxylicious#96, kroxylicious#98, kroxylicious#99, kroxylicious#100, kroxylicious#101, kroxylicious#103) Proposals 001-019 retain their original numbers. Assisted-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Keith Wall <kwall@apache.org>
This change simplifies the proposal numbering system by using PR numbers as proposal identifiers, eliminating number collisions and removing the need for a separate allocation process. Changes: - Simplified proposals/README.md to focus on author workflow - Removed index tables (directory listing serves as the index) - Streamlined instructions for creating and renaming proposals - Updated proposal template with workflow instructions - Require PR number in title format: # <PR#> - <Title> - Moved workflow instructions into comment block - Added GitHub workflow to automatically check proposal numbering - Validates both filename and title format - Updates PR description when proposal files don't match PR number - Provides exact commands to fix naming issues - Removes warning once corrected - Handles both added and renamed files - Runs on all PRs (ready for mandatory status check) - Added notification script for existing open PRs - After merge, run notify-open-prs.sh to ask authors to rebase - Workflow will automatically guide them through renaming - Updated with all current open proposal PRs (#70, #82, #83, #85, #88, #93, #94, #96, #98, #99, #100, #101, #103) Proposals 001-019 retain their original numbers. Assisted-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Keith Wall <kwall@apache.org>
|
Hi! We've updated the proposal numbering system to use PR numbers as proposal identifiers. Action required: Please rebase your PR on Once you rebase, you'll need to rename your proposal file and update the title: git mv proposals/100-sidecar-injection-webhook.md proposals/100-injection-webhook.md
# Update title: remove any old number prefix and add PR number
sed -i.bak '0,/^# /{s/^# \([0-9]\{3\}\|xxx\|nnn\|000\) - /# 100 - /; t; s/^# /# 100 - /}' proposals/100-injection-webhook.md && rm proposals/100-injection-webhook.md.bak
git add proposals/100-injection-webhook.md
git commit -m "Rename proposal to use PR number"
git pushThe GitHub workflow will automatically check your proposal file naming after you push and update this PR description if any corrections are still needed. See proposals/README.md for the updated workflow. |
|
Correction: The previous notification had an incorrect filename. Here are the correct commands: git mv proposals/100-sidecar-injection-webhook.md proposals/100-sidecar-injection-webhook.md
# Update title: remove any old number prefix and add PR number
sed -i.bak '0,/^# /{s/^# \([0-9]\{3\}\|xxx\|nnn\|000\) - /# 100 - /; t; s/^# /# 100 - /}' proposals/100-sidecar-injection-webhook.md && rm proposals/100-sidecar-injection-webhook.md.bak
git add proposals/100-sidecar-injection-webhook.md
git commit -m "Rename proposal to use PR number"
git pushSorry for the confusion! |
Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Use Istio-style subdomain prefix (sidecar.kroxylicious.io/) instead of kroxylicious.io/sidecar-* for all sidecar webhook annotations and labels. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Align the design document with the implementation, which uses targetBootstrapServers, targetClusterTls, etc. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Use sidecar.kroxylicious.io/config-generation (recording the KroxyliciousSidecarConfig's metadata.generation) instead of a simple status flag. Serves as both idempotency guard and drift detection mechanism. Drop injection-timestamp. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Document the KroxyliciousSidecarConfig status subresource (Ready condition, observedGeneration) to match the implementation. Update module names and shared dependencies to reflect actual repo structure. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Document pod-level securityContext (runAsNonRoot, seccompProfile: RuntimeDefault). Fix probe port from 9190 to management port (9082). Add actual probe parameters from implementation. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Status update failures should be retried, not silently swallowed. Remove pod-level securityContext from webhook; admins should use PodSecurity admission policies instead to avoid webhook ordering conflicts. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Per review consensus, delegated plugin image selection is too high risk (arbitrary code execution in the proxy JVM). Admin-specified plugins via spec.plugins remain unaffected. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Move targetBootstrapServers, bootstrapPort, nodeIdRange, and targetClusterTls under a virtualClusters list. Remove delegated annotations section. Rewrite target cluster selection as a virtual clusters section. Update trust model, port allocation table, and future delegation to reflect the simplified alpha scope. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Add design for secretMounts field: motivation (secrets must not be in pod annotations), trust model progression (filesystem isolation now, network isolation later), and rationale for secretMounts over generic volume mounts. Update CRD example. Expand FEATURE_GATES rationale: the API server version does not reveal which feature gates are enabled, so explicit configuration avoids additional RBAC. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Use sidecar.kroxylicious.io/injection: disabled for pod opt-out, matching the namespace label key. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
The current container-name check is weak against adversarial pods and does not detect config drift during reinvocation. Document comparing the proxy-config annotation against the expected output as a stronger alternative, noting that Jackson serialisation is deterministic. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Update CRD group from kroxylicious.io to sidecar.kroxylicious.io to avoid conflicts with the operator. Add skip labelling section describing the injection-skipped label for operator observability. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Move Config injection and Configuration drift detection before Status, since Status references the config-generation annotation those sections introduce. Fix "an internal errors" typo. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Rename injection-skipped label values to reference CRD kind directly, rewrite idempotency section to state requirements not implementation, add Ready=False for invalid configs, rename FEATURE_GATES to K8S_FEATURE_GATES, and add explicit seccompProfile: RuntimeDefault. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Update proposal to reflect implementation: multi-replica deployment with PDB, "one or more" configs per namespace, replace app-level FAILURE_POLICY with K8s failurePolicy for errors and new UNINJECTED_POD_POLICY for config edge cases. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Address Sam's review comments on the sidecar injection proposal: - Add invalid-KroxyliciousSidecarConfig to the injection-skipped label table with a note that structural checks are defensive (CRD schema catches those) while cross-field semantic checks are primary. - Fix contradictory idempotency text: config-generation annotation serves drift detection only; idempotency relies on container name. Assisted-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com>
Implements the Kroxylicious admission webhook for Kubernetes sidecar injection, as proposed in kroxylicious/design#100. This adds a Kubernetes mutating admission webhook that automatically injects Kroxylicious proxy sidecars into annotated Pods, enabling transparent Kafka protocol proxying without application code changes. What's delivered: - API module (kroxylicious-admission-api) - Custom Resource Definition for KroxySidecarConfig with comprehensive validation rules - Webhook implementation (kroxylicious-admission) - Mutating admission webhook server with: - Pod mutation logic for sidecar injection - Proxy configuration generation - OCI image volume support for 3rd party filter plugins - TLS certificate handling (PKCS8 format validation) - Fail-closed default with configurable failure policy - Status condition updates on CRD resources - Container image - Docker build workflow for quay.io/kroxylicious/kroxylicious-admission - Distribution artifacts (kroxylicious-admission-dist) - Release tarballs/zips with installation manifests - Installation manifests - RBAC, Deployment, Service, MutatingWebhookConfiguration, PodDisruptionBudget - System tests - End-to-end tests covering webhook installation on Kind and Minikube - Integration tests - CRD validation tests for all constraint violations - Documentation - Complete admission webhook guide with installation and usage procedures - Examples - Sidecar injection demo with message uppercasing filter Note that these features were accumulated into a feature branch using pull requests which have all been individually reviewed. Closes #3890 Co-authored-by: Robert Young <robertyoungnz@gmail.com> Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Tom Bentley <tbentley@redhat.com> Signed-off-by: Robert Young <robertyoungnz@gmail.com>
Assisted-by: Claude Opus 4.6 noreply@anthropic.com