Skip to content

Add sidecar injection webhook proposal#100

Merged
tombentley merged 23 commits into
kroxylicious:mainfrom
tombentley:web-hook
May 11, 2026
Merged

Add sidecar injection webhook proposal#100
tombentley merged 23 commits into
kroxylicious:mainfrom
tombentley:web-hook

Conversation

@tombentley
Copy link
Copy Markdown
Member

@tombentley tombentley commented Apr 17, 2026

Assisted-by: Claude Opus 4.6 noreply@anthropic.com

Comment thread proposals/100-sidecar-injection-webhook.md Outdated
Comment thread proposals/100-sidecar-injection-webhook.md
Comment thread proposals/100-sidecar-injection-webhook.md Outdated
Comment thread proposals/100-sidecar-injection-webhook.md Outdated
@k-wall
Copy link
Copy Markdown
Member

k-wall commented Apr 23, 2026

Discussed at Community Call 23rd April. Tom is looking for feedback.

@tombentley tombentley marked this pull request as ready for review April 23, 2026 08:30
@tombentley tombentley requested a review from a team as a code owner April 23, 2026 08:30
@tombentley tombentley moved this to Want to Do in Release 0.21.0 Apr 24, 2026
@tombentley tombentley moved this from Want to Do to Must Do in Release 0.21.0 Apr 24, 2026
@tombentley tombentley moved this from Must Do to Blocked in Release 0.21.0 Apr 24, 2026
@tombentley tombentley self-assigned this Apr 24, 2026
Comment thread proposals/100-sidecar-injection-webhook.md Outdated
Comment thread proposals/100-sidecar-injection-webhook.md Outdated
Comment thread proposals/100-sidecar-injection-webhook.md Outdated
Comment thread proposals/100-sidecar-injection-webhook.md Outdated
Comment thread proposals/100-sidecar-injection-webhook.md Outdated
Comment thread proposals/100-sidecar-injection-webhook.md Outdated
| `kroxylicious.io/config-generation` | The `metadata.generation` of the `KroxyliciousSidecarConfig` at injection time |
| `kroxylicious.io/injection-timestamp` | ISO-8601 timestamp of when the sidecar was injected |

Because the webhook only mutates pods at creation time, configuration changes to `KroxyliciousSidecarConfig` do not propagate to running pods. This matches how Istio and Linkerd handle sidecar injection. Users must restart pods to pick up new configuration.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds very sensible

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really. Because the proxy configuration is built from the KroxyliciousSidecarConfig at injection time it means that any subsequent changes to the KroxyliciousSidecarConfig are ignored until the next time the pod get created. That creates a lot of problems:

  • The user who has RBAC on the KroxyliciousSidecarConfig might not have RBAC on the Pod, Deployment etc. So it assumes some kind of communication between those users. Such communication might not be possible in practice.
  • Erroneous changes to the KroxyliciousSidecarConfig are only discovered at some unbounded time later.

The right way to solve this would be for the proxy to phone home to obtain a configuration from a control plane, and to receive updates to that configuration without the need for Pod restarts. But I really want to avoid boiling the ocean here by blocking this alpha sidecar feature on having such a control plane. I think the control plane piece is something we could add later (also as an alpha feature) and then maybe they would become stable features together later on.

Comment thread proposals/100-sidecar-injection-webhook.md
Comment thread proposals/100-sidecar-injection-webhook.md
Comment thread proposals/100-sidecar-injection-webhook.md
Comment thread proposals/100-sidecar-injection-webhook.md
@k-wall
Copy link
Copy Markdown
Member

k-wall commented Apr 24, 2026

Thanks @tombentley, looks like a solid proposal. I've left a few comments to be considered.

@tombentley tombentley moved this from Blocked to In Progress in Release 0.21.0 Apr 27, 2026
Copy link
Copy Markdown
Member Author

@tombentley tombentley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @k-wall @robobario, I think I've addresses your comments, though with a watering down of some of the initial ambition wrt trust boundaries. PTAL.

Comment thread proposals/100-sidecar-injection-webhook.md Outdated
Comment thread proposals/100-sidecar-injection-webhook.md
Comment thread proposals/100-sidecar-injection-webhook.md Outdated
Comment thread proposals/100-sidecar-injection-webhook.md Outdated
| `kroxylicious.io/config-generation` | The `metadata.generation` of the `KroxyliciousSidecarConfig` at injection time |
| `kroxylicious.io/injection-timestamp` | ISO-8601 timestamp of when the sidecar was injected |

Because the webhook only mutates pods at creation time, configuration changes to `KroxyliciousSidecarConfig` do not propagate to running pods. This matches how Istio and Linkerd handle sidecar injection. Users must restart pods to pick up new configuration.
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really. Because the proxy configuration is built from the KroxyliciousSidecarConfig at injection time it means that any subsequent changes to the KroxyliciousSidecarConfig are ignored until the next time the pod get created. That creates a lot of problems:

  • The user who has RBAC on the KroxyliciousSidecarConfig might not have RBAC on the Pod, Deployment etc. So it assumes some kind of communication between those users. Such communication might not be possible in practice.
  • Erroneous changes to the KroxyliciousSidecarConfig are only discovered at some unbounded time later.

The right way to solve this would be for the proxy to phone home to obtain a configuration from a control plane, and to receive updates to that configuration without the need for Pod restarts. But I really want to avoid boiling the ocean here by blocking this alpha sidecar feature on having such a control plane. I think the control plane piece is something we could add later (also as an alpha feature) and then maybe they would become stable features together later on.

Comment thread proposals/100-sidecar-injection-webhook.md
Comment thread proposals/100-sidecar-injection-webhook.md
Comment thread proposals/100-sidecar-injection-webhook.md Outdated
Comment thread proposals/100-sidecar-injection-webhook.md Outdated
Comment thread proposals/100-sidecar-injection-webhook.md Outdated
k-wall added a commit to k-wall/design that referenced this pull request Apr 28, 2026
Add PRs kroxylicious#96, kroxylicious#98, kroxylicious#99, kroxylicious#100, kroxylicious#101, and kroxylicious#103 which were opened
after the initial script was created.

Note: PR kroxylicious#100 is already correctly named (100-sidecar-injection-webhook.md).

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
k-wall added a commit to k-wall/design that referenced this pull request Apr 28, 2026
This change simplifies the proposal numbering system by using PR numbers
as proposal identifiers, eliminating number collisions and removing the
need for a separate allocation process.

Changes:
- Simplified proposals/README.md to focus on author workflow
  - Removed index tables (directory listing serves as the index)
  - Streamlined instructions for creating and renaming proposals
- Updated proposal template with workflow instructions
  - Require PR number in title format: # <PR#> - <Title>
  - Moved workflow instructions into comment block
- Added GitHub workflow to automatically check proposal numbering
  - Validates both filename and title format
  - Updates PR description when proposal files don't match PR number
  - Provides exact commands to fix naming issues
  - Removes warning once corrected
  - Handles both added and renamed files
  - Runs on all PRs (ready for mandatory status check)
- Added notification script for existing open PRs
  - After merge, run notify-open-prs.sh to ask authors to rebase
  - Workflow will automatically guide them through renaming
  - Updated with all current open proposal PRs (kroxylicious#70, kroxylicious#82, kroxylicious#83, kroxylicious#85,
    kroxylicious#88, kroxylicious#93, kroxylicious#94, kroxylicious#96, kroxylicious#98, kroxylicious#99, kroxylicious#100, kroxylicious#101, kroxylicious#103)

Proposals 001-019 retain their original numbers.

Assisted-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Keith Wall <kwall@apache.org>
k-wall added a commit that referenced this pull request Apr 28, 2026
This change simplifies the proposal numbering system by using PR numbers
as proposal identifiers, eliminating number collisions and removing the
need for a separate allocation process.

Changes:
- Simplified proposals/README.md to focus on author workflow
  - Removed index tables (directory listing serves as the index)
  - Streamlined instructions for creating and renaming proposals
- Updated proposal template with workflow instructions
  - Require PR number in title format: # <PR#> - <Title>
  - Moved workflow instructions into comment block
- Added GitHub workflow to automatically check proposal numbering
  - Validates both filename and title format
  - Updates PR description when proposal files don't match PR number
  - Provides exact commands to fix naming issues
  - Removes warning once corrected
  - Handles both added and renamed files
  - Runs on all PRs (ready for mandatory status check)
- Added notification script for existing open PRs
  - After merge, run notify-open-prs.sh to ask authors to rebase
  - Workflow will automatically guide them through renaming
  - Updated with all current open proposal PRs (#70, #82, #83, #85,
    #88, #93, #94, #96, #98, #99, #100, #101, #103)

Proposals 001-019 retain their original numbers.

Assisted-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Signed-off-by: Keith Wall <kwall@apache.org>
@k-wall
Copy link
Copy Markdown
Member

k-wall commented Apr 28, 2026

Hi! We've updated the proposal numbering system to use PR numbers as proposal identifiers.

Action required: Please rebase your PR on main.

Once you rebase, you'll need to rename your proposal file and update the title:

git mv proposals/100-sidecar-injection-webhook.md proposals/100-injection-webhook.md
# Update title: remove any old number prefix and add PR number
sed -i.bak '0,/^# /{s/^# \([0-9]\{3\}\|xxx\|nnn\|000\) - /# 100 - /; t; s/^# /# 100 - /}' proposals/100-injection-webhook.md && rm proposals/100-injection-webhook.md.bak
git add proposals/100-injection-webhook.md
git commit -m "Rename proposal to use PR number"
git push

The GitHub workflow will automatically check your proposal file naming after you push and update this PR description if any corrections are still needed.

See proposals/README.md for the updated workflow.

@k-wall
Copy link
Copy Markdown
Member

k-wall commented Apr 28, 2026

Correction: The previous notification had an incorrect filename. Here are the correct commands:

git mv proposals/100-sidecar-injection-webhook.md proposals/100-sidecar-injection-webhook.md
# Update title: remove any old number prefix and add PR number
sed -i.bak '0,/^# /{s/^# \([0-9]\{3\}\|xxx\|nnn\|000\) - /# 100 - /; t; s/^# /# 100 - /}' proposals/100-sidecar-injection-webhook.md && rm proposals/100-sidecar-injection-webhook.md.bak
git add proposals/100-sidecar-injection-webhook.md
git commit -m "Rename proposal to use PR number"
git push

Sorry for the confusion!

tombentley added 23 commits May 11, 2026 14:17
Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Use Istio-style subdomain prefix (sidecar.kroxylicious.io/) instead of
kroxylicious.io/sidecar-* for all sidecar webhook annotations and labels.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Align the design document with the implementation, which uses
targetBootstrapServers, targetClusterTls, etc.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Use sidecar.kroxylicious.io/config-generation (recording the
KroxyliciousSidecarConfig's metadata.generation) instead of a
simple status flag. Serves as both idempotency guard and drift
detection mechanism. Drop injection-timestamp.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Document the KroxyliciousSidecarConfig status subresource (Ready
condition, observedGeneration) to match the implementation. Update
module names and shared dependencies to reflect actual repo structure.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Document pod-level securityContext (runAsNonRoot, seccompProfile:
RuntimeDefault). Fix probe port from 9190 to management port (9082).
Add actual probe parameters from implementation.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Status update failures should be retried, not silently swallowed.
Remove pod-level securityContext from webhook; admins should use
PodSecurity admission policies instead to avoid webhook ordering
conflicts.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Per review consensus, delegated plugin image selection is too high
risk (arbitrary code execution in the proxy JVM). Admin-specified
plugins via spec.plugins remain unaffected.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Move targetBootstrapServers, bootstrapPort, nodeIdRange, and
targetClusterTls under a virtualClusters list. Remove delegated
annotations section. Rewrite target cluster selection as a virtual
clusters section. Update trust model, port allocation table, and
future delegation to reflect the simplified alpha scope.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Add design for secretMounts field: motivation (secrets must not be in
pod annotations), trust model progression (filesystem isolation now,
network isolation later), and rationale for secretMounts over generic
volume mounts. Update CRD example.

Expand FEATURE_GATES rationale: the API server version does not reveal
which feature gates are enabled, so explicit configuration avoids
additional RBAC.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Use sidecar.kroxylicious.io/injection: disabled for pod opt-out,
matching the namespace label key.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
The current container-name check is weak against adversarial pods and
does not detect config drift during reinvocation. Document comparing
the proxy-config annotation against the expected output as a stronger
alternative, noting that Jackson serialisation is deterministic.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Update CRD group from kroxylicious.io to sidecar.kroxylicious.io to
avoid conflicts with the operator. Add skip labelling section describing
the injection-skipped label for operator observability.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Move Config injection and Configuration drift detection before Status,
since Status references the config-generation annotation those sections
introduce. Fix "an internal errors" typo.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Rename injection-skipped label values to reference CRD kind directly,
rewrite idempotency section to state requirements not implementation,
add Ready=False for invalid configs, rename FEATURE_GATES to
K8S_FEATURE_GATES, and add explicit seccompProfile: RuntimeDefault.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Update proposal to reflect implementation: multi-replica deployment with
PDB, "one or more" configs per namespace, replace app-level FAILURE_POLICY
with K8s failurePolicy for errors and new UNINJECTED_POD_POLICY for config
edge cases.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Address Sam's review comments on the sidecar injection proposal:
- Add invalid-KroxyliciousSidecarConfig to the injection-skipped label
  table with a note that structural checks are defensive (CRD schema
  catches those) while cross-field semantic checks are primary.
- Fix contradictory idempotency text: config-generation annotation
  serves drift detection only; idempotency relies on container name.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
@tombentley tombentley merged commit f9d90a9 into kroxylicious:main May 11, 2026
2 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Release 0.21.0 May 11, 2026
tombentley pushed a commit to kroxylicious/kroxylicious that referenced this pull request May 11, 2026
Implements the Kroxylicious admission webhook for Kubernetes sidecar injection, as proposed in kroxylicious/design#100.
  
This adds a Kubernetes mutating admission webhook that automatically injects Kroxylicious proxy sidecars into annotated Pods, enabling transparent Kafka protocol proxying without application code changes.

What's delivered:

- API module (kroxylicious-admission-api) - Custom Resource Definition for KroxySidecarConfig with comprehensive validation rules
- Webhook implementation (kroxylicious-admission) - Mutating admission webhook server with:
  - Pod mutation logic for sidecar injection
  - Proxy configuration generation
  - OCI image volume support for 3rd party filter plugins
  - TLS certificate handling (PKCS8 format validation)
  - Fail-closed default with configurable failure policy
  - Status condition updates on CRD resources
- Container image - Docker build workflow for quay.io/kroxylicious/kroxylicious-admission
- Distribution artifacts (kroxylicious-admission-dist) - Release tarballs/zips with installation manifests
- Installation manifests - RBAC, Deployment, Service, MutatingWebhookConfiguration, PodDisruptionBudget
- System tests - End-to-end tests covering webhook installation on Kind and Minikube
- Integration tests - CRD validation tests for all constraint violations
- Documentation - Complete admission webhook guide with installation and usage procedures
- Examples - Sidecar injection demo with message uppercasing filter

Note that these features were accumulated into a feature branch using pull requests which have all been individually reviewed.

Closes #3890 

Co-authored-by: Robert Young <robertyoungnz@gmail.com>
Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Tom Bentley <tbentley@redhat.com>
Signed-off-by: Robert Young <robertyoungnz@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants