From 32025860cf6fb86fca6dcf05aefbe8f2f155c1ea Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 9 Apr 2026 17:58:03 +1200 Subject: [PATCH 01/13] Config2: multi-file plugin configuration proposal (draft) Initial draft proposing a Kubernetes-inspired multi-file configuration system with explicit plugin references, dependency graph validation, JSON Schema validation, and a migration tool. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- ...config2-multi-file-plugin-configuration.md | 296 ++++++++++++++++++ 1 file changed, 296 insertions(+) create mode 100644 proposals/015-config2-multi-file-plugin-configuration.md diff --git a/proposals/015-config2-multi-file-plugin-configuration.md b/proposals/015-config2-multi-file-plugin-configuration.md new file mode 100644 index 0000000..4dd2f4d --- /dev/null +++ b/proposals/015-config2-multi-file-plugin-configuration.md @@ -0,0 +1,296 @@ +# Config2: Multi-file plugin configuration + +We propose replacing the current monolithic proxy configuration with a multi-file, Kubernetes-inspired +configuration system. Plugin instances are defined in individual YAML files with explicit versioning, +cross-file dependency references, and optional JSON Schema validation. + +## Current situation + +Today, all configuration lives in a single YAML file. Filter definitions embed their nested plugin +configurations inline using `@PluginImplName` / `@PluginImplConfig` annotation pairs on the config +record's constructor parameters: + +```yaml +filterDefinitions: + - name: encrypt + type: RecordEncryption + config: + kms: VaultKmsService + kmsConfig: + vaultTransitEngineUrl: http://vault:8200/v1/transit + vaultToken: + password: my-token + selector: TemplateKekSelector + selectorConfig: + template: "KEK_${topicName}" +``` + +The proxy resolves these at runtime by discovering `@PluginImplName`-annotated fields, looking up +the named plugin implementation via `ServiceLoader`, and deserialising the corresponding +`@PluginImplConfig` field into the plugin's config type. This has several consequences: + +- **No reuse**: two filters that need the same KMS must each embed a copy of the KMS configuration. +- **No independent versioning**: a filter's config schema is monolithic. Changing the KMS config + format requires the filter to understand and migrate its nested portion. +- **No schema validation**: there is no declarative way to validate plugin configurations before + the proxy starts. +- **No dependency ordering**: the proxy discovers plugin dependencies implicitly at initialisation + time, with no upfront visibility into the dependency graph. +- **Opaque to tooling**: since plugin configurations are embedded inside filter configs, external + tools (Kubernetes operators, CI validators, config linters) cannot inspect or validate them + independently. + +## Motivation + +The Kubernetes operator needs to map custom resources to proxy configuration. A multi-file layout +where each plugin instance is a separate document is a natural fit: each custom resource maps to +one file. This also enables: + +- **Shared plugin instances**: a single KMS plugin instance can be referenced by multiple filters, + reducing configuration duplication and ensuring consistent credentials. +- **Declarative dependency validation**: the system can detect missing or circular dependencies + before any plugin is initialised. +- **Schema validation**: each plugin version can ship a JSON Schema, enabling early validation by + the proxy, the operator, CI pipelines, and IDE tooling. +- **Independent evolution**: plugin authors can evolve their config schemas independently, using + Kubernetes-style version progression (v1alpha1 -> v1beta1 -> v1). +- **Migration tooling**: a deterministic mapping from legacy to multi-file format enables automated + migration. + +## Proposal + +### Configuration layout + +The new configuration comprises a `proxy.yaml` and a `plugins.d/` directory: + +``` +config-dir/ + proxy.yaml + plugins.d/ + io.kroxylicious.proxy.filter.FilterFactory/ + encrypt.yaml + io.kroxylicious.kms.service.KmsService/ + vault-kms.yaml + io.kroxylicious.filter.encryption.config.KekSelectorService/ + my-selector.yaml +``` + +`proxy.yaml` contains the global proxy settings (virtual clusters, gateways, management endpoints, +default filter chain, micrometer instance names) plus a `version` field: + +```yaml +version: v1alpha1 +management: + endpoints: + prometheus: {} +virtualClusters: + - name: demo + targetCluster: + bootstrapServers: kafka.example:9092 + gateways: + - name: default + portIdentifiesNode: + bootstrapAddress: localhost:9192 +defaultFilters: + - encrypt +micrometer: + - common-tags +``` + +Each plugin instance file specifies `type`, `version`, and optionally `config`: + +```yaml +type: RecordEncryption +version: v1 +config: + kms: + type: io.kroxylicious.kms.service.KmsService + name: vault-kms + selector: + type: io.kroxylicious.filter.encryption.config.KekSelectorService + name: my-selector +``` + +### Snapshot abstraction + +A `Snapshot` interface abstracts the configuration source. It provides access to `proxy.yaml` +content and the set of plugin instances grouped by interface. This allows the same resolution +logic to work whether configuration comes from a filesystem directory, a Kubernetes ConfigMap, +or an in-memory representation in tests. + +### Plugin references and dependency graph + +Versioned config types implement `HasPluginReferences`, declaring their dependencies as +`PluginReference` values: + +```java +public record RecordEncryptionConfigV1( + PluginReference kms, + PluginReference selector, + Map experimental, + UnresolvedKeyPolicy unresolvedKeyPolicy) + implements HasPluginReferences { + + @Override + public Stream> pluginReferences() { + return Stream.of(kms, selector); + } +} +``` + +`PluginReference` is a record of `(String type, String name)` where `type` is the fully +qualified plugin interface name and `name` is the instance name. This replaces the implicit +`@PluginImplName` / `@PluginImplConfig` pairing with an explicit, typed reference. + +The dependency graph is built from these references using DFS-based topological sort. It detects: + +- **Dangling references**: a plugin references an instance that does not exist. +- **Cycles**: A depends on B depends on A. + +The topological order determines initialisation sequence: dependencies are initialised before +their dependents. + +### Resolved plugin registry + +A `ResolvedPluginRegistry` is built during config2 resolution. It pre-creates non-filter plugin +instances (KMS services, authorisers, etc.) in dependency order. Filter instances are created +later by the proxy runtime because they need per-connection context. + +Versioned filter configs obtain their dependencies from the registry: + +```java +ResolvedPluginRegistry registry = context.resolvedPluginRegistry() + .orElseThrow(() -> new PluginConfigurationException( + "v1 config requires a ResolvedPluginRegistry")); +KmsService kmsPlugin = registry.pluginInstance(KmsService.class, kmsRef.name()); +Object kmsConfig = registry.pluginConfig(kmsRef.type(), kmsRef.name()); +``` + +### Dual `@Plugin` annotations and version dispatch + +Filter implementations carry two `@Plugin` annotations: one for the legacy (unversioned) config +and one for the versioned config: + +```java +@Plugin(configType = RecordEncryptionConfig.class) +@Plugin(configVersion = "v1", configType = RecordEncryptionConfigV1.class) +public class RecordEncryption + implements FilterFactory> { +``` + +The `FilterFactory` type parameter becomes `Object`, and `initialize()` dispatches via +`instanceof`: + +```java +public SharedEncryptionContext initialize( + FilterFactoryContext context, Object config) { + var configuration = Plugins.requireConfig(this, config); + if (configuration instanceof RecordEncryptionConfigV1 v1) { + return initializeV1(context, v1); + } + else if (configuration instanceof RecordEncryptionConfig legacy) { + return initializeLegacy(context, legacy); + } + throw new PluginConfigurationException("Unsupported config type"); +} +``` + +This maintains full backwards compatibility: existing single-file configurations continue to work +via the legacy path. + +### JSON Schema validation + +Plugin authors can ship a JSON Schema for each config version at: + +``` +META-INF/kroxylicious/schemas/{PluginSimpleName}/{version}.schema.yaml +``` + +When a schema is present, config2 validates the raw configuration against it before +deserialisation. This is opt-in: if no schema exists, the config is accepted without validation. +A test utility `SchemaValidationAssert` is provided so plugin authors can verify their schemas +accept valid configs. + +### `@Stateless` annotation + +Plugins annotated `@Stateless` can be shared across multiple consumers. The plugin instance YAML +can set `shared: true` to enable this. The framework rejects `shared: true` on plugins not +annotated `@Stateless`. + +### Migration tool + +A `migrate-config` CLI subcommand converts legacy single-file configurations into the multi-file +format: + +``` +kroxylicious migrate-config -i legacy-config.yaml -o output-dir/ +``` + +The tool: + +1. Parses the legacy configuration via `ConfigParser`. +2. Reflects on each filter's legacy config type to discover `@PluginImplName` / `@PluginImplConfig` + pairs on the canonical constructor parameters. +3. Extracts each nested plugin into its own file under `plugins.d/{interface}/{name}.yaml`. +4. Rewrites the filter config to use `PluginReference`-style maps, selecting the best available + config version. +5. Writes `proxy.yaml` by passing through the raw YAML (stripping `filterDefinitions`, replacing + `micrometer` with instance name references). + +## Affected/not affected projects + +| Project | Affected | Nature of change | +|---|---|---| +| `kroxylicious-api` | Yes | New types: `PluginReference`, `HasPluginReferences`, `ResolvedPluginRegistry`, `@Stateless`. `@Plugin` made `@Repeatable` with `configVersion` attribute. | +| `kroxylicious-runtime` | Yes | New `config2` package: `Snapshot`, `ProxyConfig`, `PluginConfig`, `DependencyGraph`, `ConfigSchemaValidator`, `ProxyConfigParser`, `ResolvedPluginRegistryImpl`. | +| `kroxylicious-filter-test-support` | Yes | New `SchemaValidationAssert` utility. | +| `kroxylicious-app` | Yes | New `ConfigMigrate` picocli subcommand. Dependency scope changes for Jackson. | +| All filter modules | Yes | Dual `@Plugin` annotations, versioned config records, JSON schemas, updated tests. | +| `kroxylicious-operator` | Not yet | Future work: operator generates `Snapshot` from CRDs. | +| `kroxylicious-docs` | Not yet | Future work: document new configuration format. | + +## Compatibility + +### Backwards compatibility + +The legacy single-file configuration format continues to work unchanged. The dual `@Plugin` +annotation approach means the existing `ConfigParser` deserialises legacy configs exactly as +before. The config2 path is only activated when a multi-file configuration source is used. + +### Forward compatibility + +Config version strings follow Kubernetes conventions (v1alpha1, v1beta1, v1). The version +field in plugin YAML files enables the framework to select the correct deserialisation target +as schemas evolve. The `@Repeatable` `@Plugin` annotation allows plugins to support an arbitrary +number of config versions simultaneously. + +### Migration path + +Users migrate at their own pace. The `migrate-config` tool provides a one-shot conversion. +The legacy format is not deprecated by this proposal. + +## Rejected alternatives + +### Embedding version in the config parser rather than plugin annotations + +We considered adding version dispatch logic to the `ConfigParser` itself. This was rejected +because plugin authors know their own config evolution best. The `@Plugin` annotation approach +keeps version knowledge co-located with the plugin implementation. + +### Single versioned config type per plugin + +We considered requiring each plugin to have exactly one config type for the new format, removing +the legacy type entirely. This was rejected because it would break existing configurations and +force a flag-day migration. + +### Generating plugin files from annotations at build time + +We considered generating the `plugins.d/` layout from annotations during the Maven build. This +was rejected because the multi-file layout is a deployment-time concern, not a build-time one. +The configuration source is an operational choice (filesystem, ConfigMap, etc.). + +### Configuration without explicit dependency declarations + +We considered inferring dependencies from the config structure (e.g. scanning for nested objects +that look like plugin references). This was rejected in favour of explicit `HasPluginReferences` +because implicit inference is fragile, hard to validate, and invisible to tooling. From 73adad64a7d8a6b7d652147969ab57fa979997d1 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 9 Apr 2026 18:07:13 +1200 Subject: [PATCH 02/13] Rewrite motivation section from actual project rationale The five numbered motivations now match the original problem statement: per-plugin schemas, uniformity, dependency understanding for dynamic reloading, individual plugin identity for control plane, and principled config versioning. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- ...config2-multi-file-plugin-configuration.md | 62 ++++++++++++++----- 1 file changed, 48 insertions(+), 14 deletions(-) diff --git a/proposals/015-config2-multi-file-plugin-configuration.md b/proposals/015-config2-multi-file-plugin-configuration.md index 4dd2f4d..6ff8e2b 100644 --- a/proposals/015-config2-multi-file-plugin-configuration.md +++ b/proposals/015-config2-multi-file-plugin-configuration.md @@ -42,20 +42,54 @@ the named plugin implementation via `ServiceLoader`, and deserialising the corre ## Motivation -The Kubernetes operator needs to map custom resources to proxy configuration. A multi-file layout -where each plugin instance is a separate document is a natural fit: each custom resource maps to -one file. This also enables: - -- **Shared plugin instances**: a single KMS plugin instance can be referenced by multiple filters, - reducing configuration duplication and ensuring consistent credentials. -- **Declarative dependency validation**: the system can detect missing or circular dependencies - before any plugin is initialised. -- **Schema validation**: each plugin version can ship a JSON Schema, enabling early validation by - the proxy, the operator, CI pipelines, and IDE tooling. -- **Independent evolution**: plugin authors can evolve their config schemas independently, using - Kubernetes-style version progression (v1alpha1 -> v1beta1 -> v1). -- **Migration tooling**: a deterministic mapping from legacy to multi-file format enables automated - migration. +There are five specific drawbacks of the current configuration system that motivate this change. + +### 1. Per-plugin JSON schemas are impossible today + +It is impossible to write a single JSON schema for the configuration file, because what is allowed +depends on which plugins are present at runtime. But having schemas is desirable for documentation, +editor assistance, and automated validation. By giving each plugin instance its own file, each +plugin version can ship its own JSON schema. These per-plugin schemas can be published to schema +catalogues (e.g. schemastore.org) and consumed by IDEs and CI tools independently of the proxy. + +### 2. Lack of uniformity in how plugins are configured + +Each plugin point embeds its plugins differently. Filter factories are explicitly named (via +`NamedFilterDefinition`), but other plugins like `KmsService` are anonymous because they are +referenced once by their parent. This inconsistency extends across the codebase: filters, +authorisers, KMS services, subject builders, and key selectors all follow slightly different +patterns. A uniform model where every plugin instance is a named, versioned resource eliminates +this inconsistency. + +### 3. The runtime cannot understand inter-plugin dependencies + +The lack of uniformity means the runtime has no meaningful understanding of the dependency +relationships between plugins. This matters for dynamic reloading: when a plugin configuration +changes, the runtime needs to know which virtual clusters are affected. Today, dependencies are +discovered implicitly at initialisation time, with no upfront visibility into the graph. Explicit +dependency declarations enable the runtime to reason about the impact of configuration changes. + +Note that the runtime does not (and should not) know about all plugin types. Plugins are allowed +to have plugins of their own (e.g. `RecordEncryption` depends on `KmsService` and +`KekSelectorService`, which are not known to the runtime). This rules out approaches that add +top-level definitions for each plugin type (e.g. `kmsDefinitions`), because the runtime cannot +enumerate plugin types it does not know about. + +### 4. Plugin instances are not individually identifiable + +Eventually we may need to build a control plane for clusters of proxies. Plugin instances would be +important entities in the API of that control plane. For this to work, each plugin instance must +have a unique identity. The current system gives names to filter definitions but not to the plugins +they embed. A model where every plugin instance has a name, a type, and a version makes each one +individually addressable. + +### 5. No principled way to evolve plugin configuration + +None of the configurations (the proxy's or any of the plugins') are explicitly versioned. A plugin +developer can deprecate properties, but cannot remove them in a controlled manner where the API +evolution is obvious to the user. Explicit version strings (following Kubernetes conventions: +v1alpha1, v1beta1, v1) give plugin developers a principled mechanism for config schema evolution, +including support for multiple concurrent versions during migration periods. ## Proposal From d98775c662203db1ecd6580264106b06b4b7f93d Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 9 Apr 2026 21:17:52 +1200 Subject: [PATCH 03/13] Add name field requirement to plugin instance files Plugin instance files must include a `name` field matching the filename. This makes files self-describing for debugging and non-filesystem sources. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- proposals/015-config2-multi-file-plugin-configuration.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/proposals/015-config2-multi-file-plugin-configuration.md b/proposals/015-config2-multi-file-plugin-configuration.md index 6ff8e2b..3e826c7 100644 --- a/proposals/015-config2-multi-file-plugin-configuration.md +++ b/proposals/015-config2-multi-file-plugin-configuration.md @@ -131,9 +131,10 @@ micrometer: - common-tags ``` -Each plugin instance file specifies `type`, `version`, and optionally `config`: +Each plugin instance file specifies `name`, `type`, `version`, and optionally `config`: ```yaml +name: encrypt type: RecordEncryption version: v1 config: @@ -145,6 +146,11 @@ config: name: my-selector ``` +The `name` field must match the filename (without the `.yaml` extension). This is validated during +loading. Including `name` in the file content ensures that plugin instance files are +self-describing: a file can be understood without knowing its path, which matters for debugging, +code review, and non-filesystem configuration sources like ConfigMaps. + ### Snapshot abstraction A `Snapshot` interface abstracts the configuration source. It provides access to `proxy.yaml` From 9b892aa31e825d8e14cffd4c9a9a2c083c5c9c16 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 9 Apr 2026 21:21:06 +1200 Subject: [PATCH 04/13] Expand plugin references, dependency graph, and API boundaries Explain why HasPluginReferences exists (explicit > introspection), why PluginReference is not a breaking change (config versioning), and why we enforce referential integrity unlike Kubernetes. Clarify which types are public API (kroxylicious-api) vs internal. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- ...config2-multi-file-plugin-configuration.md | 53 ++++++++++++++++--- 1 file changed, 45 insertions(+), 8 deletions(-) diff --git a/proposals/015-config2-multi-file-plugin-configuration.md b/proposals/015-config2-multi-file-plugin-configuration.md index 3e826c7..23e7c64 100644 --- a/proposals/015-config2-multi-file-plugin-configuration.md +++ b/proposals/015-config2-multi-file-plugin-configuration.md @@ -158,10 +158,10 @@ content and the set of plugin instances grouped by interface. This allows the sa logic to work whether configuration comes from a filesystem directory, a Kubernetes ConfigMap, or an in-memory representation in tests. -### Plugin references and dependency graph +### Plugin references -Versioned config types implement `HasPluginReferences`, declaring their dependencies as -`PluginReference` values: +When a plugin's configuration needs to refer to another plugin instance (e.g. a filter that +uses a KMS service), the versioned config type uses `PluginReference`: ```java public record RecordEncryptionConfigV1( @@ -179,10 +179,28 @@ public record RecordEncryptionConfigV1( ``` `PluginReference` is a record of `(String type, String name)` where `type` is the fully -qualified plugin interface name and `name` is the instance name. This replaces the implicit -`@PluginImplName` / `@PluginImplConfig` pairing with an explicit, typed reference. +qualified plugin interface name and `name` is the instance name. In the YAML, this serialises +as a `{type, name}` map — a uniform, explicit pattern for all cross-file references. -The dependency graph is built from these references using DFS-based topological sort. It detects: +By introducing `PluginReference` we impose a consistent reference syntax on all plugin +developers. This is a deliberate trade-off: it constrains the shape of versioned config types, +but in return every dependency is visible to the framework without requiring type-specific +introspection. The alternative — scanning config objects for fields that look like references, +or using reflection to discover dependencies at runtime — would be fragile, plugin-specific, +and invisible to tooling. + +This is not a breaking change. Plugin developers adopt `PluginReference` fields only when they +introduce a new versioned config type (e.g. `RecordEncryptionConfigV1`). The legacy config type +(`RecordEncryptionConfig`) continues to use `@PluginImplName` / `@PluginImplConfig` as before. +Because the config versioning mechanism (`@Plugin(configVersion = "v1", configType = ...)`) +allows both old and new config types to coexist, the `PluginReference` pattern is adopted +incrementally, version by version. + +### Dependency graph and referential integrity + +The runtime builds a dependency graph from `HasPluginReferences` declarations. Versioned config +types implement this interface to enumerate their `PluginReference` values. The runtime then +validates the graph using DFS-based topological sort, detecting: - **Dangling references**: a plugin references an instance that does not exist. - **Cycles**: A depends on B depends on A. @@ -190,6 +208,20 @@ The dependency graph is built from these references using DFS-based topological The topological order determines initialisation sequence: dependencies are initialised before their dependents. +This is a deliberate deviation from Kubernetes. The Kubernetes API server does not enforce +referential integrity between resources — a Pod can reference a ConfigMap that does not yet +exist, and eventual consistency resolves the situation over time. We make a different choice +because our needs are different: the proxy cannot start with a dangling KMS reference or a +cyclic dependency. Fail-fast validation at configuration load time is preferable to discovering +broken references at runtime when the first message arrives. The cost is that the runtime must +understand the dependency structure, which is why `HasPluginReferences` exists as an explicit +contract rather than relying on introspection or convention. + +The runtime does not (and should not) know about all plugin types. Plugins can depend on other +plugins that the runtime has never heard of (e.g. `RecordEncryption` depends on `KmsService` +and `KekSelectorService`). The `HasPluginReferences` interface lets each config type declare +its own dependencies without requiring the runtime to enumerate every possible plugin interface. + ### Resolved plugin registry A `ResolvedPluginRegistry` is built during config2 resolution. It pre-creates non-filter plugin @@ -281,14 +313,19 @@ The tool: | Project | Affected | Nature of change | |---|---|---| -| `kroxylicious-api` | Yes | New types: `PluginReference`, `HasPluginReferences`, `ResolvedPluginRegistry`, `@Stateless`. `@Plugin` made `@Repeatable` with `configVersion` attribute. | -| `kroxylicious-runtime` | Yes | New `config2` package: `Snapshot`, `ProxyConfig`, `PluginConfig`, `DependencyGraph`, `ConfigSchemaValidator`, `ProxyConfigParser`, `ResolvedPluginRegistryImpl`. | +| `kroxylicious-api` | Yes | New **public API** types: `PluginReference`, `HasPluginReferences`, `ResolvedPluginRegistry`, `@Stateless`. `@Plugin` made `@Repeatable` with `configVersion` attribute. These are the types that plugin developers depend on and that we commit to supporting long-term. | +| `kroxylicious-runtime` | Yes | New **internal** `config2` package: `Snapshot`, `FilesystemSnapshot`, `ProxyConfig`, `PluginConfig`, `DependencyGraph`, `ConfigSchemaValidator`, `ProxyConfigParser`, `ResolvedPluginRegistryImpl`. These are implementation details not visible to plugin developers. | | `kroxylicious-filter-test-support` | Yes | New `SchemaValidationAssert` utility. | | `kroxylicious-app` | Yes | New `ConfigMigrate` picocli subcommand. Dependency scope changes for Jackson. | | All filter modules | Yes | Dual `@Plugin` annotations, versioned config records, JSON schemas, updated tests. | | `kroxylicious-operator` | Not yet | Future work: operator generates `Snapshot` from CRDs. | | `kroxylicious-docs` | Not yet | Future work: document new configuration format. | +The distinction matters: types in `kroxylicious-api` form the contract with plugin developers. +Changes to `PluginReference`, `HasPluginReferences`, or `ResolvedPluginRegistry` require the +same care as any other public API change. Types in `kroxylicious-runtime` are internal and can +be refactored freely. + ## Compatibility ### Backwards compatibility From 29f5e48f5297e064aa66fc3dbea741bbbded2178 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Fri, 10 Apr 2026 09:48:29 +1200 Subject: [PATCH 05/13] Tom changes Signed-off-by: Tom Bentley --- ...config2-multi-file-plugin-configuration.md | 27 ++++++++++++------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/proposals/015-config2-multi-file-plugin-configuration.md b/proposals/015-config2-multi-file-plugin-configuration.md index 23e7c64..90baea8 100644 --- a/proposals/015-config2-multi-file-plugin-configuration.md +++ b/proposals/015-config2-multi-file-plugin-configuration.md @@ -29,16 +29,16 @@ The proxy resolves these at runtime by discovering `@PluginImplName`-annotated f the named plugin implementation via `ServiceLoader`, and deserialising the corresponding `@PluginImplConfig` field into the plugin's config type. This has several consequences: -- **No reuse**: two filters that need the same KMS must each embed a copy of the KMS configuration. -- **No independent versioning**: a filter's config schema is monolithic. Changing the KMS config - format requires the filter to understand and migrate its nested portion. -- **No schema validation**: there is no declarative way to validate plugin configurations before - the proxy starts. +- **No reuse**: two filters that need the same KMS must each embed a copy of the same KMS configuration. +- **No independent versioning**: a proxy config file is monolithic and lacks explicit version numbers, + so every plugin needs to honour backwards compatibility of its configuration essentially for ever. - **No dependency ordering**: the proxy discovers plugin dependencies implicitly at initialisation time, with no upfront visibility into the dependency graph. +- **No schema validation**: there is no declarative way to validate plugin configurations + except by trying to start the proxy. - **Opaque to tooling**: since plugin configurations are embedded inside filter configs, external tools (Kubernetes operators, CI validators, config linters) cannot inspect or validate them - independently. + independently (see [#2436](https://github.com/kroxylicious/kroxylicious/issues/2436). ## Motivation @@ -135,7 +135,7 @@ Each plugin instance file specifies `name`, `type`, `version`, and optionally `c ```yaml name: encrypt -type: RecordEncryption +type: io.kroxylicious.filter.encryption.RecordEncryption version: v1 config: kms: @@ -149,14 +149,21 @@ config: The `name` field must match the filename (without the `.yaml` extension). This is validated during loading. Including `name` in the file content ensures that plugin instance files are self-describing: a file can be understood without knowing its path, which matters for debugging, -code review, and non-filesystem configuration sources like ConfigMaps. +code review, and non-filesystem configuration sources. + +In old-style proxy configuration files we allowed unqualified type names, +which were resolved based on the loadable plugins. +If we kept with that choice in the new format then plugin configurations would not be fully self-describing. +So to facilitate the unambiguous sharing of configurations the `type` is fully qualified in the new format. ### Snapshot abstraction A `Snapshot` interface abstracts the configuration source. It provides access to `proxy.yaml` content and the set of plugin instances grouped by interface. This allows the same resolution -logic to work whether configuration comes from a filesystem directory, a Kubernetes ConfigMap, -or an in-memory representation in tests. +logic to work whether configuration comes from a filesystem directory, +an in-memory representation in tests or some other source. + +The `Snapshot` interface will not be public API, but internal to the runtime. ### Plugin references From 859d65289d027a736be4989b2dc3382d73984d99 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Fri, 10 Apr 2026 09:48:54 +1200 Subject: [PATCH 06/13] Revert "Tom changes" This reverts commit 83938a79ed0322672661b352426a9686917ee769. Signed-off-by: Tom Bentley --- ...config2-multi-file-plugin-configuration.md | 27 +++++++------------ 1 file changed, 10 insertions(+), 17 deletions(-) diff --git a/proposals/015-config2-multi-file-plugin-configuration.md b/proposals/015-config2-multi-file-plugin-configuration.md index 90baea8..23e7c64 100644 --- a/proposals/015-config2-multi-file-plugin-configuration.md +++ b/proposals/015-config2-multi-file-plugin-configuration.md @@ -29,16 +29,16 @@ The proxy resolves these at runtime by discovering `@PluginImplName`-annotated f the named plugin implementation via `ServiceLoader`, and deserialising the corresponding `@PluginImplConfig` field into the plugin's config type. This has several consequences: -- **No reuse**: two filters that need the same KMS must each embed a copy of the same KMS configuration. -- **No independent versioning**: a proxy config file is monolithic and lacks explicit version numbers, - so every plugin needs to honour backwards compatibility of its configuration essentially for ever. +- **No reuse**: two filters that need the same KMS must each embed a copy of the KMS configuration. +- **No independent versioning**: a filter's config schema is monolithic. Changing the KMS config + format requires the filter to understand and migrate its nested portion. +- **No schema validation**: there is no declarative way to validate plugin configurations before + the proxy starts. - **No dependency ordering**: the proxy discovers plugin dependencies implicitly at initialisation time, with no upfront visibility into the dependency graph. -- **No schema validation**: there is no declarative way to validate plugin configurations - except by trying to start the proxy. - **Opaque to tooling**: since plugin configurations are embedded inside filter configs, external tools (Kubernetes operators, CI validators, config linters) cannot inspect or validate them - independently (see [#2436](https://github.com/kroxylicious/kroxylicious/issues/2436). + independently. ## Motivation @@ -135,7 +135,7 @@ Each plugin instance file specifies `name`, `type`, `version`, and optionally `c ```yaml name: encrypt -type: io.kroxylicious.filter.encryption.RecordEncryption +type: RecordEncryption version: v1 config: kms: @@ -149,21 +149,14 @@ config: The `name` field must match the filename (without the `.yaml` extension). This is validated during loading. Including `name` in the file content ensures that plugin instance files are self-describing: a file can be understood without knowing its path, which matters for debugging, -code review, and non-filesystem configuration sources. - -In old-style proxy configuration files we allowed unqualified type names, -which were resolved based on the loadable plugins. -If we kept with that choice in the new format then plugin configurations would not be fully self-describing. -So to facilitate the unambiguous sharing of configurations the `type` is fully qualified in the new format. +code review, and non-filesystem configuration sources like ConfigMaps. ### Snapshot abstraction A `Snapshot` interface abstracts the configuration source. It provides access to `proxy.yaml` content and the set of plugin instances grouped by interface. This allows the same resolution -logic to work whether configuration comes from a filesystem directory, -an in-memory representation in tests or some other source. - -The `Snapshot` interface will not be public API, but internal to the runtime. +logic to work whether configuration comes from a filesystem directory, a Kubernetes ConfigMap, +or an in-memory representation in tests. ### Plugin references From fccdc7f5943558fe9287cc589d704f1f1443d9b3 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Fri, 10 Apr 2026 10:21:03 +1200 Subject: [PATCH 07/13] Require fully qualified class names for plugin type field Plugin instance files must use FQCNs for the type field, making them self-describing without depending on which plugins are loadable. JSON Schema enum constraints provide editor code completion. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- .../015-config2-multi-file-plugin-configuration.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/proposals/015-config2-multi-file-plugin-configuration.md b/proposals/015-config2-multi-file-plugin-configuration.md index 23e7c64..feadb93 100644 --- a/proposals/015-config2-multi-file-plugin-configuration.md +++ b/proposals/015-config2-multi-file-plugin-configuration.md @@ -135,7 +135,7 @@ Each plugin instance file specifies `name`, `type`, `version`, and optionally `c ```yaml name: encrypt -type: RecordEncryption +type: io.kroxylicious.filter.encryption.RecordEncryption version: v1 config: kms: @@ -151,6 +151,17 @@ loading. Including `name` in the file content ensures that plugin instance files self-describing: a file can be understood without knowing its path, which matters for debugging, code review, and non-filesystem configuration sources like ConfigMaps. +The `type` field must be a fully qualified class name. The legacy single-file format allows +short names (e.g. `RecordEncryption`), but the config2 format requires the FQCN +(e.g. `io.kroxylicious.filter.encryption.RecordEncryption`). This makes each plugin file +completely self-describing: its interpretation does not depend on which plugins happen to be +loadable by a particular proxy binary. Short names are ambiguous when two plugins share a +simple class name; FQCNs eliminate that problem entirely. + +The loss of brevity is mitigated by JSON Schema: each plugin version's schema can constrain +`type` using `const` or `enum`, giving editors code completion and validation without requiring +the user to type the full name. + ### Snapshot abstraction A `Snapshot` interface abstracts the configuration source. It provides access to `proxy.yaml` From 387e2d2e732d95a2e0ed51a27b4b2287cbd83a2e Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Fri, 10 Apr 2026 11:43:18 +1200 Subject: [PATCH 08/13] Update config2 proposal: PluginReference is a runtime type, not YAML serialisation Plugin references in YAML are bare instance name strings. The config type constructs PluginReference internally via HasPluginReferences, combining the statically-known interface type with the instance name from YAML. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- ...config2-multi-file-plugin-configuration.md | 58 +++++++++++-------- 1 file changed, 33 insertions(+), 25 deletions(-) diff --git a/proposals/015-config2-multi-file-plugin-configuration.md b/proposals/015-config2-multi-file-plugin-configuration.md index feadb93..c9774e1 100644 --- a/proposals/015-config2-multi-file-plugin-configuration.md +++ b/proposals/015-config2-multi-file-plugin-configuration.md @@ -138,12 +138,8 @@ name: encrypt type: io.kroxylicious.filter.encryption.RecordEncryption version: v1 config: - kms: - type: io.kroxylicious.kms.service.KmsService - name: vault-kms - selector: - type: io.kroxylicious.filter.encryption.config.KekSelectorService - name: my-selector + kms: vault-kms + selector: my-selector ``` The `name` field must match the filename (without the `.yaml` extension). This is validated during @@ -172,39 +168,51 @@ or an in-memory representation in tests. ### Plugin references When a plugin's configuration needs to refer to another plugin instance (e.g. a filter that -uses a KMS service), the versioned config type uses `PluginReference`: +uses a KMS service), the versioned config type declares its dependencies by implementing +`HasPluginReferences`. The YAML representation of the reference is entirely up to the plugin +author — typically a bare instance name string, since the plugin interface type is statically +known: ```java public record RecordEncryptionConfigV1( - PluginReference kms, - PluginReference selector, + String kms, + String selector, Map experimental, UnresolvedKeyPolicy unresolvedKeyPolicy) implements HasPluginReferences { @Override public Stream> pluginReferences() { - return Stream.of(kms, selector); + return Stream.of( + new PluginReference<>(KmsService.class.getName(), kms), + new PluginReference<>(KekSelectorService.class.getName(), selector)); } } ``` -`PluginReference` is a record of `(String type, String name)` where `type` is the fully -qualified plugin interface name and `name` is the instance name. In the YAML, this serialises -as a `{type, name}` map — a uniform, explicit pattern for all cross-file references. - -By introducing `PluginReference` we impose a consistent reference syntax on all plugin -developers. This is a deliberate trade-off: it constrains the shape of versioned config types, -but in return every dependency is visible to the framework without requiring type-specific -introspection. The alternative — scanning config objects for fields that look like references, -or using reflection to discover dependencies at runtime — would be fragile, plugin-specific, -and invisible to tooling. - -This is not a breaking change. Plugin developers adopt `PluginReference` fields only when they +`PluginReference` is a runtime type — a record of `(String type, String name)` where `type` +is the fully qualified plugin interface name and `name` is the instance name. It is not a +serialisation type: it does not appear in the YAML. Plugin authors construct `PluginReference` +values in their `pluginReferences()` implementation, combining the statically-known interface +type with the instance name read from YAML. + +This separation is deliberate. A plugin like `RecordEncryption` knows statically that its `kms` +field always refers to a `KmsService`. Forcing the user to write +`kms: {type: io.kroxylicious.kms.service.KmsService, name: vault-kms}` in the YAML would be +redundant — the `type` can only ever be one thing. By keeping `PluginReference` out of the +YAML, each plugin is free to choose the most natural representation for its references. + +The `HasPluginReferences` interface is the contract that makes this work. The runtime calls +`pluginReferences()` to discover cross-file dependencies, build the dependency graph, and +validate that all referenced plugin instances exist. The alternative — scanning config objects +for fields that look like references, or using reflection to discover dependencies — would be +fragile, plugin-specific, and invisible to tooling. + +This is not a breaking change. Plugin developers implement `HasPluginReferences` only when they introduce a new versioned config type (e.g. `RecordEncryptionConfigV1`). The legacy config type (`RecordEncryptionConfig`) continues to use `@PluginImplName` / `@PluginImplConfig` as before. Because the config versioning mechanism (`@Plugin(configVersion = "v1", configType = ...)`) -allows both old and new config types to coexist, the `PluginReference` pattern is adopted +allows both old and new config types to coexist, the `HasPluginReferences` pattern is adopted incrementally, version by version. ### Dependency graph and referential integrity @@ -245,8 +253,8 @@ Versioned filter configs obtain their dependencies from the registry: ResolvedPluginRegistry registry = context.resolvedPluginRegistry() .orElseThrow(() -> new PluginConfigurationException( "v1 config requires a ResolvedPluginRegistry")); -KmsService kmsPlugin = registry.pluginInstance(KmsService.class, kmsRef.name()); -Object kmsConfig = registry.pluginConfig(kmsRef.type(), kmsRef.name()); +KmsService kmsPlugin = registry.pluginInstance(KmsService.class, v1.kms()); +Object kmsConfig = registry.pluginConfig(KmsService.class.getName(), v1.kms()); ``` ### Dual `@Plugin` annotations and version dispatch From 217ed40b65b38e6dcbf0e4c8239d39429a2d46db Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Fri, 10 Apr 2026 15:56:36 +1200 Subject: [PATCH 09/13] Consolidate Snapshot description and add non-YAML resource model to config2 proposal Describe the full Snapshot interface (metadata, bytes, passwords, generation numbers) once in the Snapshot abstraction section instead of partially describing it and then re-describing it in Non-YAML resources. Add @ResourceType annotation, TLS plugin interfaces, and filesystem layout for binary resources. Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- ...config2-multi-file-plugin-configuration.md | 178 +++++++++++++++++- 1 file changed, 172 insertions(+), 6 deletions(-) diff --git a/proposals/015-config2-multi-file-plugin-configuration.md b/proposals/015-config2-multi-file-plugin-configuration.md index c9774e1..234c6f7 100644 --- a/proposals/015-config2-multi-file-plugin-configuration.md +++ b/proposals/015-config2-multi-file-plugin-configuration.md @@ -160,10 +160,54 @@ the user to type the full name. ### Snapshot abstraction -A `Snapshot` interface abstracts the configuration source. It provides access to `proxy.yaml` -content and the set of plugin instances grouped by interface. This allows the same resolution -logic to work whether configuration comes from a filesystem directory, a Kubernetes ConfigMap, -or an in-memory representation in tests. +A `Snapshot` interface abstracts the configuration source, allowing the same resolution logic +to work whether configuration comes from a filesystem directory, a Kubernetes ConfigMap, or +an in-memory representation in tests. + +```java +public interface Snapshot { + /** Returns the proxy configuration YAML content. */ + String proxyConfig(); + + /** Returns the plugin interface names for which plugin instances are configured. */ + List pluginInterfaces(); + + /** Returns the names of all plugin instances configured for a given plugin interface. */ + List pluginInstances(String pluginInterfaceName); + + /** Metadata for a plugin instance (name, type, version, shared, generation). */ + PluginInstanceMetadata pluginInstanceMetadata( + String pluginInterfaceName, String pluginInstanceName); + + /** Raw data bytes for a plugin instance (UTF-8 YAML or binary resource). */ + byte[] pluginInstanceData( + String pluginInterfaceName, String pluginInstanceName); + + /** Password for a named resource, or null if no password is configured. */ + @Nullable char[] resourcePassword(String resourceName); +} +``` + +Each plugin instance has **metadata** and **data**: + +- **Metadata** (`PluginInstanceMetadata`): name, type (FQCN), version, shared flag, and a + generation number. The generation is a monotonically increasing value that changes when the + resource content changes, enabling efficient change detection without comparing data bytes + (which is problematic for keystores due to non-deterministic salting and security concerns + around comparing key material). For the Kubernetes operator, generation maps to + `metadata.generation` / `resourceVersion`. For filesystem deployments, it is derived from + file modification time. + +- **Data** (`pluginInstanceData()`): raw bytes. For most plugins, these bytes are UTF-8 YAML — + the proxy parses them with Jackson into the `configType` from the `@Plugin` annotation. For + binary resource plugins (those annotated with `@ResourceType` — see **Non-YAML resources** + below), the bytes are the raw resource content (e.g. a PKCS12 keystore). The runtime checks + `@ResourceType` on the plugin class to determine which deserialisation path to use. + +**Passwords** are provided out-of-band from the resource data via `resourcePassword()`. This +decouples passwords from the data they protect, allowing different access controls (e.g. +Kubernetes Secret vs ConfigMap, or a separate `passwords.yaml` with restricted permissions +for filesystem deployments). ### Plugin references @@ -332,8 +376,8 @@ The tool: | Project | Affected | Nature of change | |---|---|---| -| `kroxylicious-api` | Yes | New **public API** types: `PluginReference`, `HasPluginReferences`, `ResolvedPluginRegistry`, `@Stateless`. `@Plugin` made `@Repeatable` with `configVersion` attribute. These are the types that plugin developers depend on and that we commit to supporting long-term. | -| `kroxylicious-runtime` | Yes | New **internal** `config2` package: `Snapshot`, `FilesystemSnapshot`, `ProxyConfig`, `PluginConfig`, `DependencyGraph`, `ConfigSchemaValidator`, `ProxyConfigParser`, `ResolvedPluginRegistryImpl`. These are implementation details not visible to plugin developers. | +| `kroxylicious-api` | Yes | New **public API** types: `PluginReference`, `HasPluginReferences`, `ResolvedPluginRegistry`, `@Stateless`, `@ResourceType`, `ResourceSerializer`, `ResourceDeserializer`, `KeyMaterialProvider`, `TrustMaterialProvider`. `@Plugin` made `@Repeatable` with `configVersion` attribute. These are the types that plugin developers depend on and that we commit to supporting long-term. | +| `kroxylicious-runtime` | Yes | New **internal** `config2` package: `Snapshot`, `FilesystemSnapshot`, `PluginInstanceMetadata`, `ProxyConfig`, `PluginConfig`, `DependencyGraph`, `ConfigSchemaValidator`, `ProxyConfigParser`, `ResolvedPluginRegistryImpl`. These are implementation details not visible to plugin developers. | | `kroxylicious-filter-test-support` | Yes | New `SchemaValidationAssert` utility. | | `kroxylicious-app` | Yes | New `ConfigMigrate` picocli subcommand. Dependency scope changes for Jackson. | | All filter modules | Yes | Dual `@Plugin` annotations, versioned config records, JSON schemas, updated tests. | @@ -365,6 +409,128 @@ number of config versions simultaneously. Users migrate at their own pace. The `migrate-config` tool provides a one-shot conversion. The legacy format is not deprecated by this proposal. +## Non-YAML resources + +Not all plugin instance data is YAML. Some plugins manage binary resources — most notably TLS +key material stored as Java KeyStores (JKS, PKCS12) or PEM files. For the `Snapshot` to fully +encapsulate a proxy's configuration state, it must include these resources alongside YAML-based +plugin configs. + +### `@ResourceType` annotation + +As described in the Snapshot abstraction, plugin instance data is raw bytes — UTF-8 YAML by +default. For binary resource types (e.g. keystores), the plugin implementation declares its +binary format via a `@ResourceType` annotation: + +```java +@Retention(RetentionPolicy.RUNTIME) +@Target(ElementType.TYPE) +public @interface ResourceType { + Class> serializer(); + Class> deserializer(); +} +``` + +The runtime discovers the serde from the annotation via reflection — the same pattern used +for `@Plugin(configType = ...)`. The `ResourceSerializer` converts a typed resource to bytes +(for snapshot generation), and `ResourceDeserializer` converts bytes back to a typed resource +(for loading). + +### Store type auto-detection + +Keystore format need not be user-specified. Common formats have distinctive magic bytes +(JKS: `0xFEEDFEED`, PKCS12: ASN.1 `0x30`, PEM: `-----BEGIN`). The deserialiser probes the +bytes and selects the appropriate format. PKCS12 is the default keystore type since Java 9; +JKS is legacy. + +### Passwords + +Passwords are provided **out-of-band** from the resource data, not co-located with it. The +`Snapshot` interface includes a `resourcePassword(String resourceName)` method that returns the +password for a named resource. This decouples passwords from the data they protect, allowing +different access controls (e.g. Kubernetes Secret vs ConfigMap for filesystem deployments, a +separate `passwords.yaml` with restricted permissions). + +### Alias selection + +When a keystore contains multiple entries, the consuming plugin's config specifies the alias +via a sibling property — this is a concern of the consumer, not the resource: + +```java +public record VaultKmsConfigV1( + String vaultUrl, + @Nullable String keyMaterial, + @Nullable String keyAlias, + @Nullable String trustMaterial +) implements HasPluginReferences { ... } +``` + +### Change detection: generation numbers + +Each resource in the `Snapshot` carries a **generation number** — a monotonically increasing +value that changes when the resource content changes. Comparing generation numbers rather than +data bytes avoids: + +- Non-determinism from keystore salting (two logically equivalent stores can have different bytes). +- Security concerns around comparing key material. +- Expensive byte-level comparison of large blobs. + +For the operator, generation maps to Kubernetes `metadata.generation` / `resourceVersion`. +For filesystem deployments, it is derived from file modification time. + +### Plugin interfaces for TLS material + +Two marker interfaces in `kroxylicious-api` model TLS resources: + +```java +/** Provides TLS key material (private key + certificate chain). */ +public interface KeyMaterialProvider { + java.security.KeyStore keyStore(); +} + +/** Provides TLS trust material (trusted CA certificates). */ +public interface TrustMaterialProvider { + java.security.KeyStore trustStore(); +} +``` + +Consumers reference these by name in their versioned config, constructing `PluginReference` +values in `pluginReferences()` — the same pattern used for all other cross-file references. + +### Filesystem layout + +Binary resources use a sidecar `.data` file alongside the YAML metadata file: + +``` +config-dir/ + proxy.yaml + passwords.yaml + plugins.d/ + io.kroxylicious.proxy.tls.KeyMaterialProvider/ + my-keystore.yaml # metadata: name, type, version + my-keystore.data # binary keystore bytes + io.kroxylicious.proxy.filter.FilterFactory/ + encrypt.yaml # metadata + YAML config +``` + +### Text-based resources: inline YAML + +For text-based resources like ACL rules, YAML multi-line syntax (`|`) is simpler than the +resource abstraction. The versioned config type uses a `String` field instead of a file path: + +```yaml +name: my-acl-authorizer +type: io.kroxylicious.authorizer.provider.acl.AclAuthorizerService +version: v1 +config: + rules: | + allow User with name = "alice" to READ Topic with name = "foo"; + otherwise deny; +``` + +PEM certificates and private keys are also text and can use the same inline approach. +Binary formats (JKS, PKCS12) require the `@ResourceType` mechanism. + ## Rejected alternatives ### Embedding version in the config parser rather than plugin annotations From 8fc36ce190521ff11a7eb7981de95f7d06a7fa22 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 13 Apr 2026 13:24:54 +1200 Subject: [PATCH 10/13] Aiming for linearity Signed-off-by: Tom Bentley --- ...config2-multi-file-plugin-configuration.md | 654 +++++++++++------- 1 file changed, 387 insertions(+), 267 deletions(-) diff --git a/proposals/015-config2-multi-file-plugin-configuration.md b/proposals/015-config2-multi-file-plugin-configuration.md index 234c6f7..9cb59e2 100644 --- a/proposals/015-config2-multi-file-plugin-configuration.md +++ b/proposals/015-config2-multi-file-plugin-configuration.md @@ -6,9 +6,14 @@ cross-file dependency references, and optional JSON Schema validation. ## Current situation -Today, all configuration lives in a single YAML file. Filter definitions embed their nested plugin -configurations inline using `@PluginImplName` / `@PluginImplConfig` annotation pairs on the config -record's constructor parameters: +Today, all configuration lives in a single YAML file. + +At places where plugins can be configured, `@PluginImplName` / `@PluginImplConfig` annotation pairs are used +so that a plugin's configuration can be included in the YAML tree. +Plugins are allowed to have plugins of their own. + +For example, the proxy runtime uses those annotations for the `type` and `config` properties, and +the `RecordEncryption` filter uses them for `kms` + `kmsConfig` and for `selector` and `selectorConfig`: ```yaml filterDefinitions: @@ -25,31 +30,44 @@ filterDefinitions: template: "KEK_${topicName}" ``` -The proxy resolves these at runtime by discovering `@PluginImplName`-annotated fields, looking up +The proxy runtime resolves these during startup by discovering `@PluginImplName`-annotated fields, looking up the named plugin implementation via `ServiceLoader`, and deserialising the corresponding -`@PluginImplConfig` field into the plugin's config type. This has several consequences: - -- **No reuse**: two filters that need the same KMS must each embed a copy of the KMS configuration. -- **No independent versioning**: a filter's config schema is monolithic. Changing the KMS config - format requires the filter to understand and migrate its nested portion. -- **No schema validation**: there is no declarative way to validate plugin configurations before - the proxy starts. -- **No dependency ordering**: the proxy discovers plugin dependencies implicitly at initialisation - time, with no upfront visibility into the dependency graph. -- **Opaque to tooling**: since plugin configurations are embedded inside filter configs, external - tools (Kubernetes operators, CI validators, config linters) cannot inspect or validate them - independently. +`@PluginImplConfig` field into the plugin's config type. + +The existing APIs work, and plugins-to-plugins has proven to be useful. ## Motivation +Although it works, the current configuration APIs have several consequences: + +1. **No reuse**: The model is tree-like. For example, if two filters need the same KMS + then each must embed a copy of the KMS configuration. +2. **Poor documentation**: a plugin developer ought to be able to document the YAML accepted by their plugin, + but this is difficult because the source of truth is Java code: End users cannot be assumed to know Java. +3. **Opaque to tooling**: similarly, external tools (editors, config linters, CI validators, Kubernetes operators) + cannot inspect or validate plugin configurations. (See [2436](https://github.com/kroxylicious/kroxylicious/issues/2436)) +4. **No schema validation**: there is no declarative way to validate plugin configurations before + the proxy starts. (Somewhat related: [3267](https://github.com/kroxylicious/kroxylicious/issues/3267#issuecomment-4042031788) +5. **No independent versioning**: As a plugin evolves there are limited facilities for end users + and plugin developers to agree common expectations about a plugin configuration. The plugin + implementation either accepts the YAML or it does not. If newer version of a plugin removes + support for a configuration property the user will likely be confused, because the configuration + used to work. But the plug in has no effective way to communicate that the user is speaking + the old language. +6. **No dependency ordering**: the proxy discovers plugin dependencies implicitly at initialisation + time, with no upfront visibility into the dependency graph. + There are five specific drawbacks of the current configuration system that motivate this change. ### 1. Per-plugin JSON schemas are impossible today -It is impossible to write a single JSON schema for the configuration file, because what is allowed -depends on which plugins are present at runtime. But having schemas is desirable for documentation, -editor assistance, and automated validation. By giving each plugin instance its own file, each -plugin version can ship its own JSON schema. These per-plugin schemas can be published to schema +Items 2, 3 and 4 are quite tightly related, and all relate to poorer user experience. + +However it is currently impossible to write a single JSON schema for the configuration file, because what is allowed +depends on which plugins are present at runtime. +By giving each plugin instance its own file, each +plugin version can ship its own JSON schema to be used to validate that file. +These per-plugin schemas can be published to schema catalogues (e.g. schemastore.org) and consumed by IDEs and CI tools independently of the proxy. ### 2. Lack of uniformity in how plugins are configured @@ -93,7 +111,160 @@ including support for multiple concurrent versions during migration periods. ## Proposal -### Configuration layout +### Supporting multiple plugin configuration versions + +The existing `@Plugin` annotation type will be annotated `@Repeatable`, and given a new `configVersion` method. + +This will allow plugin implementations carry multiple `@Plugin` annotations: +* the legacy (unversioned) config will be the one with an empty string as its `configVersion` +* hence forth, each version of a plugin's config will be identified by a non-empty `configVersion` + +The runtime will validate that `configVersion` is unique across all the `@Plugin` annotations on a plugin implementation class. + +```java +@Plugin(configType = RecordEncryptionConfig.class) // The legacy config schema +@Plugin(configVersion = "v1", configType = RecordEncryptionConfigV1.class) // The new config schema, called v1 +public class RecordEncryption + implements FilterFactory> { +``` + +When implementing a plugin like `FilterFactory` which accepts a type parameter representing the configuration type, `Object` will be used as the type argument. +If desired, a plugin developer can declare some common interface type for all their config types to inherit, thus avoiding the Object type argument. + +In either case `initialize()`-like methods will dispatch via `instanceof`: + +```java +public SharedEncryptionContext initialize( + FilterFactoryContext context, Object config) { + var configuration = Plugins.requireConfig(this, config); + if (configuration instanceof RecordEncryptionConfigV1 v1) { + return initializeV1(context, v1); + } + else if (configuration instanceof RecordEncryptionConfig legacy) { + return initializeLegacy(context, legacy); + } + throw new PluginConfigurationException("Unsupported config type"); +} +``` + +This maintains full backwards compatibility: existing single-file configurations continue to work via the legacy path. + +### Plugin references + +`PluginReference` is a new API class used by plugin developers to describe their plugin's configuration dependency on some other plugin instance. + +```java +package io.kroxylicious.proxy.plugin; + +/** + * A typed reference to a named plugin instance. Used by the framework to + * build the dependency graph and determine initialisation order. + *

+ * This is a runtime type, not a serialisation type. Plugin authors choose their + * own YAML representation for references (e.g. a bare instance name string when + * the plugin interface type is statically known) and construct {@code PluginReference} + * instances in their {@link HasPluginReferences#pluginReferences()} implementation. + * + * @param type the fully qualified name of the plugin interface (e.g. {@code io.kroxylicious.kms.service.KmsService}) + * @param name the name of the plugin instance (e.g. {@code aws-kms}) + * @param the plugin interface type + */ +public record PluginReference( + String type, + String name) {} +``` + +In general, a plugin's configuration might have many such individual dependencies. +The runtime needs a way to know about all of a plugin's dependencies so we define `HasPluginReferences`: + +```java +package io.kroxylicious.proxy.plugin; +/** + * Implemented by new-style (versioned) plugin configuration types to declare their + * dependencies on other plugin instances. The framework calls {@link #pluginReferences()} + * to discover cross-file dependencies, build a dependency graph, and validate that + * all referenced plugin instances exist. + */ +public interface HasPluginReferences { + + /** + * Returns all plugin references in this configuration, including references + * from nested structures. Implementations must include all references to + * ensure correct dependency tracking and initialisation ordering. + * + * @return a stream of all plugin references declared by this configuration + */ + Stream> pluginReferences(); +} +``` + +When a plugin's configuration can refer to another plugin instance (e.g. a filter that +uses a KMS service), the plugin's configuration type declares its dependencies by implementing +`HasPluginReferences`. + + +```java +public record RecordEncryptionConfigV1( + String kms, + String selector, + Map experimental, + UnresolvedKeyPolicy unresolvedKeyPolicy) + implements HasPluginReferences { + + @Override + public Stream> pluginReferences() { + return Stream.of( + new PluginReference<>(KmsService.class.getName(), kms), + new PluginReference<>(KekSelectorService.class.getName(), selector)); + } +} +``` + +The YAML representation of the reference is entirely up to the plugin author. +When the plugin interface type is statically known the reference can simply be the name of the depended-upon plugin instance (i.e. a JSON string). +For example, a plugin like `RecordEncryption` knows statically that its `kms` +field always refers to a `KmsService`. Forcing the user to write +`kms: {type: io.kroxylicious.kms.service.KmsService, name: vault-kms}` in the YAML would be +redundant — the `type` can only ever be one thing. +By keeping `PluginReference` out of the +YAML, each plugin is free to choose the most natural representation for its references. + +The `HasPluginReferences` interface is the contract that allows the runtime to resolve the plugin instance graph. +The runtime calls +`pluginReferences()` to discover cross-file dependencies, build the dependency graph, and +validate that all referenced plugin instances exist. + +This is not a breaking change. Plugin developers implement `HasPluginReferences` only when they +introduce a new versioned config type (e.g. `RecordEncryptionConfigV1`). The legacy config type +(`RecordEncryptionConfig`) continues to use `@PluginImplName` / `@PluginImplConfig` as before. +Because the config versioning mechanism (`@Plugin(configVersion = "v1", configType = ...)`) +allows both old and new config types to coexist, the `HasPluginReferences` pattern is adopted +incrementally, version by version. + +### JSON Schema validation + +Plugin developers can ship a JSON Schema for each config version at: + +``` +META-INF/kroxylicious/schemas/{PluginSimpleName}/{version}.schema.yaml +``` + +When a schema is present, the runtime will validate the raw configuration against it before deserialisation. +This is opt-in: if no schema exists, the config is accepted without validation. + +A test utility `SchemaValidationAssert` is provided in `kroxylicious-filter-test-support` so plugin authors can verify their schemas accept valid configs. + +### `@Stateless` annotation + +Plugin developers can appled a new `@Stateless` annotation to their implementation +This signals to the runtime that an instance can be shared across multiple consumers. + +The decision about whether to actually do that belongs to the end user who controls the configuration used to refer to a plugin instance. +The plugin instance YAML can set `shared: true` to enable this. + +The framework will reject `shared: true` on plugins not annotated `@Stateless`. + +### One file per plugin instance The new configuration comprises a `proxy.yaml` and a `plugins.d/` directory: @@ -113,6 +284,7 @@ config-dir/ default filter chain, micrometer instance names) plus a `version` field: ```yaml +# proxy.yaml version: v1alpha1 management: endpoints: @@ -127,13 +299,12 @@ virtualClusters: bootstrapAddress: localhost:9192 defaultFilters: - encrypt -micrometer: - - common-tags ``` Each plugin instance file specifies `name`, `type`, `version`, and optionally `config`: ```yaml +# plugins.d/io.kroxylicious.proxy.filter.FilterFactory/encrypt.yaml name: encrypt type: io.kroxylicious.filter.encryption.RecordEncryption version: v1 @@ -145,7 +316,7 @@ config: The `name` field must match the filename (without the `.yaml` extension). This is validated during loading. Including `name` in the file content ensures that plugin instance files are self-describing: a file can be understood without knowing its path, which matters for debugging, -code review, and non-filesystem configuration sources like ConfigMaps. +code review, and (in the future) non-filesystem configuration sources. The `type` field must be a fully qualified class name. The legacy single-file format allows short names (e.g. `RecordEncryption`), but the config2 format requires the FQCN @@ -158,106 +329,6 @@ The loss of brevity is mitigated by JSON Schema: each plugin version's schema ca `type` using `const` or `enum`, giving editors code completion and validation without requiring the user to type the full name. -### Snapshot abstraction - -A `Snapshot` interface abstracts the configuration source, allowing the same resolution logic -to work whether configuration comes from a filesystem directory, a Kubernetes ConfigMap, or -an in-memory representation in tests. - -```java -public interface Snapshot { - /** Returns the proxy configuration YAML content. */ - String proxyConfig(); - - /** Returns the plugin interface names for which plugin instances are configured. */ - List pluginInterfaces(); - - /** Returns the names of all plugin instances configured for a given plugin interface. */ - List pluginInstances(String pluginInterfaceName); - - /** Metadata for a plugin instance (name, type, version, shared, generation). */ - PluginInstanceMetadata pluginInstanceMetadata( - String pluginInterfaceName, String pluginInstanceName); - - /** Raw data bytes for a plugin instance (UTF-8 YAML or binary resource). */ - byte[] pluginInstanceData( - String pluginInterfaceName, String pluginInstanceName); - - /** Password for a named resource, or null if no password is configured. */ - @Nullable char[] resourcePassword(String resourceName); -} -``` - -Each plugin instance has **metadata** and **data**: - -- **Metadata** (`PluginInstanceMetadata`): name, type (FQCN), version, shared flag, and a - generation number. The generation is a monotonically increasing value that changes when the - resource content changes, enabling efficient change detection without comparing data bytes - (which is problematic for keystores due to non-deterministic salting and security concerns - around comparing key material). For the Kubernetes operator, generation maps to - `metadata.generation` / `resourceVersion`. For filesystem deployments, it is derived from - file modification time. - -- **Data** (`pluginInstanceData()`): raw bytes. For most plugins, these bytes are UTF-8 YAML — - the proxy parses them with Jackson into the `configType` from the `@Plugin` annotation. For - binary resource plugins (those annotated with `@ResourceType` — see **Non-YAML resources** - below), the bytes are the raw resource content (e.g. a PKCS12 keystore). The runtime checks - `@ResourceType` on the plugin class to determine which deserialisation path to use. - -**Passwords** are provided out-of-band from the resource data via `resourcePassword()`. This -decouples passwords from the data they protect, allowing different access controls (e.g. -Kubernetes Secret vs ConfigMap, or a separate `passwords.yaml` with restricted permissions -for filesystem deployments). - -### Plugin references - -When a plugin's configuration needs to refer to another plugin instance (e.g. a filter that -uses a KMS service), the versioned config type declares its dependencies by implementing -`HasPluginReferences`. The YAML representation of the reference is entirely up to the plugin -author — typically a bare instance name string, since the plugin interface type is statically -known: - -```java -public record RecordEncryptionConfigV1( - String kms, - String selector, - Map experimental, - UnresolvedKeyPolicy unresolvedKeyPolicy) - implements HasPluginReferences { - - @Override - public Stream> pluginReferences() { - return Stream.of( - new PluginReference<>(KmsService.class.getName(), kms), - new PluginReference<>(KekSelectorService.class.getName(), selector)); - } -} -``` - -`PluginReference` is a runtime type — a record of `(String type, String name)` where `type` -is the fully qualified plugin interface name and `name` is the instance name. It is not a -serialisation type: it does not appear in the YAML. Plugin authors construct `PluginReference` -values in their `pluginReferences()` implementation, combining the statically-known interface -type with the instance name read from YAML. - -This separation is deliberate. A plugin like `RecordEncryption` knows statically that its `kms` -field always refers to a `KmsService`. Forcing the user to write -`kms: {type: io.kroxylicious.kms.service.KmsService, name: vault-kms}` in the YAML would be -redundant — the `type` can only ever be one thing. By keeping `PluginReference` out of the -YAML, each plugin is free to choose the most natural representation for its references. - -The `HasPluginReferences` interface is the contract that makes this work. The runtime calls -`pluginReferences()` to discover cross-file dependencies, build the dependency graph, and -validate that all referenced plugin instances exist. The alternative — scanning config objects -for fields that look like references, or using reflection to discover dependencies — would be -fragile, plugin-specific, and invisible to tooling. - -This is not a breaking change. Plugin developers implement `HasPluginReferences` only when they -introduce a new versioned config type (e.g. `RecordEncryptionConfigV1`). The legacy config type -(`RecordEncryptionConfig`) continues to use `@PluginImplName` / `@PluginImplConfig` as before. -Because the config versioning mechanism (`@Plugin(configVersion = "v1", configType = ...)`) -allows both old and new config types to coexist, the `HasPluginReferences` pattern is adopted -incrementally, version by version. ### Dependency graph and referential integrity @@ -287,9 +358,9 @@ its own dependencies without requiring the runtime to enumerate every possible p ### Resolved plugin registry -A `ResolvedPluginRegistry` is built during config2 resolution. It pre-creates non-filter plugin -instances (KMS services, authorisers, etc.) in dependency order. Filter instances are created -later by the proxy runtime because they need per-connection context. +A `ResolvedPluginRegistry` is built during plugin resolution. +It pre-creates non-filter plugin instances (KMS services, authorisers, etc.) in dependency order. +Filter instances are created later by the proxy runtime because they need per-connection context. Versioned filter configs obtain their dependencies from the registry: @@ -301,56 +372,6 @@ KmsService kmsPlugin = registry.pluginInstance(KmsService.class, v1.kms()); Object kmsConfig = registry.pluginConfig(KmsService.class.getName(), v1.kms()); ``` -### Dual `@Plugin` annotations and version dispatch - -Filter implementations carry two `@Plugin` annotations: one for the legacy (unversioned) config -and one for the versioned config: - -```java -@Plugin(configType = RecordEncryptionConfig.class) -@Plugin(configVersion = "v1", configType = RecordEncryptionConfigV1.class) -public class RecordEncryption - implements FilterFactory> { -``` - -The `FilterFactory` type parameter becomes `Object`, and `initialize()` dispatches via -`instanceof`: - -```java -public SharedEncryptionContext initialize( - FilterFactoryContext context, Object config) { - var configuration = Plugins.requireConfig(this, config); - if (configuration instanceof RecordEncryptionConfigV1 v1) { - return initializeV1(context, v1); - } - else if (configuration instanceof RecordEncryptionConfig legacy) { - return initializeLegacy(context, legacy); - } - throw new PluginConfigurationException("Unsupported config type"); -} -``` - -This maintains full backwards compatibility: existing single-file configurations continue to work -via the legacy path. - -### JSON Schema validation - -Plugin authors can ship a JSON Schema for each config version at: - -``` -META-INF/kroxylicious/schemas/{PluginSimpleName}/{version}.schema.yaml -``` - -When a schema is present, config2 validates the raw configuration against it before -deserialisation. This is opt-in: if no schema exists, the config is accepted without validation. -A test utility `SchemaValidationAssert` is provided so plugin authors can verify their schemas -accept valid configs. - -### `@Stateless` annotation - -Plugins annotated `@Stateless` can be shared across multiple consumers. The plugin instance YAML -can set `shared: true` to enable this. The framework rejects `shared: true` on plugins not -annotated `@Stateless`. ### Migration tool @@ -372,55 +393,170 @@ The tool: 5. Writes `proxy.yaml` by passing through the raw YAML (stripping `filterDefinitions`, replacing `micrometer` with instance name references). -## Affected/not affected projects -| Project | Affected | Nature of change | -|---|---|---| -| `kroxylicious-api` | Yes | New **public API** types: `PluginReference`, `HasPluginReferences`, `ResolvedPluginRegistry`, `@Stateless`, `@ResourceType`, `ResourceSerializer`, `ResourceDeserializer`, `KeyMaterialProvider`, `TrustMaterialProvider`. `@Plugin` made `@Repeatable` with `configVersion` attribute. These are the types that plugin developers depend on and that we commit to supporting long-term. | -| `kroxylicious-runtime` | Yes | New **internal** `config2` package: `Snapshot`, `FilesystemSnapshot`, `PluginInstanceMetadata`, `ProxyConfig`, `PluginConfig`, `DependencyGraph`, `ConfigSchemaValidator`, `ProxyConfigParser`, `ResolvedPluginRegistryImpl`. These are implementation details not visible to plugin developers. | -| `kroxylicious-filter-test-support` | Yes | New `SchemaValidationAssert` utility. | -| `kroxylicious-app` | Yes | New `ConfigMigrate` picocli subcommand. Dependency scope changes for Jackson. | -| All filter modules | Yes | Dual `@Plugin` annotations, versioned config records, JSON schemas, updated tests. | -| `kroxylicious-operator` | Not yet | Future work: operator generates `Snapshot` from CRDs. | -| `kroxylicious-docs` | Not yet | Future work: document new configuration format. | +### Snapshot abstraction -The distinction matters: types in `kroxylicious-api` form the contract with plugin developers. -Changes to `PluginReference`, `HasPluginReferences`, or `ResolvedPluginRegistry` require the -same care as any other public API change. Types in `kroxylicious-runtime` are internal and can -be refactored freely. +**This subsection concerns non-public APIs. It is included in this proposal to help explain the concepts.** -## Compatibility +A `Snapshot` interface in `kroxylicious-runtime` abstracts the configuration source, allowing the same resolution logic +to work whether configuration comes from a filesystem directory, an in-memory representation in tests, or (in the future) some other source. -### Backwards compatibility +```java + /** + * Returns the proxy configuration YAML content. + * + * @return the proxy configuration as a YAML string + */ + String proxyConfig(); -The legacy single-file configuration format continues to work unchanged. The dual `@Plugin` -annotation approach means the existing `ConfigParser` deserialises legacy configs exactly as -before. The config2 path is only activated when a multi-file configuration source is used. + /** + * Returns the plugin interface names for which plugin instances are configured. + * + * @return list of plugin interface names (fully qualified class names) + */ + // For the filesystem implementation this is the child directories of plugins.d/ + List pluginInterfaces(); -### Forward compatibility + /** + * Returns the names of all plugin instances configured for a given plugin interface. + * + * @param pluginInterfaceName the fully qualified name of the plugin interface + * @return list of plugin instance names + */ + // For the filesystem implementation this is the .yaml files in the given subdirectory of plugins.d/ + List pluginInstances(String pluginInterfaceName); + + /** + * Returns the password for a named resource, if one is configured. Passwords are + * provided out-of-band from the resource data for security. + * + * @param resourceName the name of the resource + * @return the password as a char array, or {@code null} if no password is configured + */ + @Nullable + char[] resourcePassword(String resourceName); + + /** + * Returns the metadata and data bytes for a specific plugin instance as an atomic unit. + * For YAML-based plugins, the data is the UTF-8 encoded YAML content (including the + * metadata envelope). For binary resource plugins (annotated with {@code @ResourceType}), + * the data is the raw resource bytes. + * + *

Callers must not modify the returned data array.

+ * + * @param pluginInterfaceName the fully qualified name of the plugin interface + * @param pluginInstanceName the name of the plugin instance + * @return the plugin instance content (metadata and data) + * @throws IllegalArgumentException if the plugin instance is not found + */ + PluginInstanceContent pluginInstance( + String pluginInterfaceName, + String pluginInstanceName); +``` -Config version strings follow Kubernetes conventions (v1alpha1, v1beta1, v1). The version -field in plugin YAML files enables the framework to select the correct deserialisation target -as schemas evolve. The `@Repeatable` `@Plugin` annotation allows plugins to support an arbitrary -number of config versions simultaneously. +### Passwords -### Migration path +Passwords are provided **out-of-band** from the resource data, not co-located with it. +The `Snapshot` interface includes a `resourcePassword(String resourceName)` method that returns the password for a named resource. +This decouples passwords from the data they protect. +For example, the passwords could be passed in a separate file, via environment variables, or read from a stream at startup. -Users migrate at their own pace. The `migrate-config` tool provides a one-shot conversion. -The legacy format is not deprecated by this proposal. +### `PluginInstanceContent` + +A `PluginInstanceContent` has **metadata** and **data**: + +```java +public record PluginInstanceContent( + PluginInstanceMetadata metadata, + byte[] data) {} +``` + +- **Metadata** (`PluginInstanceMetadata`): name, type (FQCN), version, shared flag, and a + generation number. + +- **Data**: raw bytes. For most plugins, these bytes are UTF-8 YAML — + the proxy parses them with Jackson into the `configType` from the `@Plugin` annotation. For + binary resource plugins (those annotated with `@ResourceType` — see **Non-YAML resources** + below), the bytes are the raw resource content (e.g. a PKCS12 keystore). The runtime checks + `@ResourceType` on the plugin class to determine which deserialisation path to use. + +### `PluginInstanceMetadata` + +```java +public record PluginInstanceMetadata( + String name, + String type, + String version, + boolean shared, + long generation) {} +``` + The `generation` is a monotonically increasing value that changes when the + resource content changes, enabling efficient change detection without comparing data bytes + (which is problematic for keystores due to non-deterministic salting and security concerns + around comparing key material). For the filesystem-based snapshot, it can be derived from + file modification time. + +### Change detection: generation numbers + +Each resource in the `Snapshot` carries a **generation number** — a monotonically increasing +value that changes when the resource content changes. Comparing generation numbers rather than +data bytes avoids: + +- Non-determinism from keystore salting (two logically equivalent stores can have different bytes). +- Security concerns around comparing key material. +- Expensive byte-level comparison of large blobs. + +For the operator, generation maps to Kubernetes `metadata.generation` / `resourceVersion`. +For filesystem deployments, it is derived from file modification time. ## Non-YAML resources -Not all plugin instance data is YAML. Some plugins manage binary resources — most notably TLS -key material stored as Java KeyStores (JKS, PKCS12) or PEM files. For the `Snapshot` to fully -encapsulate a proxy's configuration state, it must include these resources alongside YAML-based -plugin configs. +Some plugins manage binary resources — most notably TLS +key material stored as Java KeyStores (JKS, PKCS12) or PEM files. +For the `Snapshot` to fully encapsulate a proxy's configuration state, it must include these resources alongside YAML-based plugin configs. + +Not all plugin instance data is YAML. +* The existing AclAuthorizer uses a non-YAML text format for its ACL rules. +* Plugins commonly accept `Tls` objects for specifying secret data via Java KeyStores. + To support these it is possible to provide plugin data in any format. + For the filesystem Snapshort this is provided in a separate file in the same directory as the YAML metadata file. + +### Text-based resources: inline YAML + +For text-based resources like ACL rules, YAML multi-line syntax (`|`) is simplest. +The versioned config type can use a `String` field instead of a file path: + +```yaml +name: my-acl-authorizer +type: io.kroxylicious.authorizer.provider.acl.AclAuthorizerService +version: v1 +config: + rules: | + allow User with name = "alice" to READ Topic with name = "foo"; + otherwise deny; +``` + +### Filesystem layout + +Binary resources use a sidecar `.data` file alongside the YAML metadata file: + +``` +config-dir/ + proxy.yaml + passwords.yaml + plugins.d/ + io.kroxylicious.proxy.tls.KeyMaterialProvider/ + my-keystore.yaml # metadata: name, type, version + my-keystore.data # binary keystore bytes + io.kroxylicious.proxy.filter.FilterFactory/ + encrypt.yaml # metadata + YAML config +``` + +Binary formats (including JKS, PKCS12) require the `@ResourceType` mechanism. ### `@ResourceType` annotation -As described in the Snapshot abstraction, plugin instance data is raw bytes — UTF-8 YAML by -default. For binary resource types (e.g. keystores), the plugin implementation declares its -binary format via a `@ResourceType` annotation: +For binary resource types (e.g. keystores), the plugin implementation declares its binary format via a `@ResourceType` annotation: ```java @Retention(RetentionPolicy.RUNTIME) @@ -431,25 +567,15 @@ public @interface ResourceType { } ``` -The runtime discovers the serde from the annotation via reflection — the same pattern used -for `@Plugin(configType = ...)`. The `ResourceSerializer` converts a typed resource to bytes -(for snapshot generation), and `ResourceDeserializer` converts bytes back to a typed resource -(for loading). +The runtime discovers the serde from the annotation via reflection — the same pattern used for `@Plugin(configType = ...)`. +The `ResourceSerializer` converts a typed resource to bytes (for snapshot generation), and `ResourceDeserializer` converts bytes back to a typed resource (for loading). ### Store type auto-detection -Keystore format need not be user-specified. Common formats have distinctive magic bytes -(JKS: `0xFEEDFEED`, PKCS12: ASN.1 `0x30`, PEM: `-----BEGIN`). The deserialiser probes the -bytes and selects the appropriate format. PKCS12 is the default keystore type since Java 9; -JKS is legacy. - -### Passwords - -Passwords are provided **out-of-band** from the resource data, not co-located with it. The -`Snapshot` interface includes a `resourcePassword(String resourceName)` method that returns the -password for a named resource. This decouples passwords from the data they protect, allowing -different access controls (e.g. Kubernetes Secret vs ConfigMap for filesystem deployments, a -separate `passwords.yaml` with restricted permissions). +Keystore format need not be user-specified. +Common formats have distinctive magic bytes (JKS: `0xFEEDFEED`, PKCS12: ASN.1 `0x30`, PEM: `-----BEGIN`). +The deserialiser probes the bytes and selects the appropriate format. +PKCS12 is the default keystore type since Java 9; JKS is legacy. ### Alias selection @@ -465,19 +591,6 @@ public record VaultKmsConfigV1( ) implements HasPluginReferences { ... } ``` -### Change detection: generation numbers - -Each resource in the `Snapshot` carries a **generation number** — a monotonically increasing -value that changes when the resource content changes. Comparing generation numbers rather than -data bytes avoids: - -- Non-determinism from keystore salting (two logically equivalent stores can have different bytes). -- Security concerns around comparing key material. -- Expensive byte-level comparison of large blobs. - -For the operator, generation maps to Kubernetes `metadata.generation` / `resourceVersion`. -For filesystem deployments, it is derived from file modification time. - ### Plugin interfaces for TLS material Two marker interfaces in `kroxylicious-api` model TLS resources: @@ -497,39 +610,46 @@ public interface TrustMaterialProvider { Consumers reference these by name in their versioned config, constructing `PluginReference` values in `pluginReferences()` — the same pattern used for all other cross-file references. -### Filesystem layout -Binary resources use a sidecar `.data` file alongside the YAML metadata file: +## Affected/not affected projects -``` -config-dir/ - proxy.yaml - passwords.yaml - plugins.d/ - io.kroxylicious.proxy.tls.KeyMaterialProvider/ - my-keystore.yaml # metadata: name, type, version - my-keystore.data # binary keystore bytes - io.kroxylicious.proxy.filter.FilterFactory/ - encrypt.yaml # metadata + YAML config -``` +The changes described in this proposal concern the kroxylicious project only. +In particular is only covers the kroxylicious proxy. +Follow-on changes to `kroxylicious-kubernetes-api` are `kroxylicious-operator` will be developed in a future proposal. -### Text-based resources: inline YAML +| Project | Affected | Nature of change | +|---|---|---| +| `kroxylicious-api` | Yes | New **public API** types: `PluginReference`, `HasPluginReferences`, `ResolvedPluginRegistry`, `@Stateless`, `@ResourceType`, `ResourceSerializer`, `ResourceDeserializer`, `KeyMaterialProvider`, `TrustMaterialProvider`. `@Plugin` made `@Repeatable` with `configVersion` attribute. These are the types that plugin developers depend on and that we commit to supporting long-term. | +| `kroxylicious-runtime` | Yes | New **internal** `config2` package: `Snapshot`, `FilesystemSnapshot`, `PluginInstanceMetadata`, `ProxyConfig`, `PluginConfig`, `DependencyGraph`, `ConfigSchemaValidator`, `ProxyConfigParser`, `ResolvedPluginRegistryImpl`. These are implementation details not visible to plugin developers. | +| `kroxylicious-filter-test-support` | Yes | New `SchemaValidationAssert` utility. | +| `kroxylicious-app` | Yes | New `ConfigMigrate` picocli subcommand. Dependency scope changes for Jackson. | +| All filter modules | Yes | Dual `@Plugin` annotations, versioned config records, JSON schemas, updated tests. | +| `kroxylicious-operator` | Not yet | Future work: operator generates `Snapshot` from CRDs. | +| `kroxylicious-docs` | Not yet | Future work: document new configuration format. | -For text-based resources like ACL rules, YAML multi-line syntax (`|`) is simpler than the -resource abstraction. The versioned config type uses a `String` field instead of a file path: -```yaml -name: my-acl-authorizer -type: io.kroxylicious.authorizer.provider.acl.AclAuthorizerService -version: v1 -config: - rules: | - allow User with name = "alice" to READ Topic with name = "foo"; - otherwise deny; -``` +## Compatibility + + +### Backwards compatibility + +The legacy single-file configuration format continues to work unchanged. +The dual `@Plugin` annotation approach means the existing `ConfigParser` deserialises legacy configs exactly as before. +The new configuration handling is only activated when proxy configuration file has a `version` property. +Because current configurations never have this property they will continue to be handled by the current mechanisms. + +### Forward compatibility + +Config version strings follow Kubernetes conventions (v1alpha1, v1beta1, v1). +The `version` field in plugin YAML files enables the framework to select the correct deserialisation target as schemas evolve. +The `@Repeatable` `@Plugin` annotation allows plugins to support an arbitrary number of config versions simultaneously. + +### Migration path + +Users migrate at their own pace. The `migrate-config` tool provides a one-shot conversion. +The legacy format is not deprecated by this proposal. + -PEM certificates and private keys are also text and can use the same inline approach. -Binary formats (JKS, PKCS12) require the `@ResourceType` mechanism. ## Rejected alternatives From ff9b660446c1b7386ff73a462b664e4271b47864 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 13 Apr 2026 14:49:07 +1200 Subject: [PATCH 11/13] Claude + Tom Signed-off-by: Tom Bentley --- ...config2-multi-file-plugin-configuration.md | 498 ++++++++++-------- 1 file changed, 265 insertions(+), 233 deletions(-) diff --git a/proposals/015-config2-multi-file-plugin-configuration.md b/proposals/015-config2-multi-file-plugin-configuration.md index 9cb59e2..019661c 100644 --- a/proposals/015-config2-multi-file-plugin-configuration.md +++ b/proposals/015-config2-multi-file-plugin-configuration.md @@ -6,27 +6,28 @@ cross-file dependency references, and optional JSON Schema validation. ## Current situation -Today, all configuration lives in a single YAML file. +Today, configuration lives in a single YAML file with a few ancillary files used for TLS certificates, keys and ACL rules. -At places where plugins can be configured, `@PluginImplName` / `@PluginImplConfig` annotation pairs are used +At places where plugins can be configured, `@PluginImplName` / `@PluginImplConfig` annotation pairs are used on the relevant Java class so that a plugin's configuration can be included in the YAML tree. Plugins are allowed to have plugins of their own. -For example, the proxy runtime uses those annotations for the `type` and `config` properties, and -the `RecordEncryption` filter uses them for `kms` + `kmsConfig` and for `selector` and `selectorConfig`: +For example, the proxy runtime uses those annotations for the `type` and `config` properties (<1> & <2>), and +the `RecordEncryption` filter uses them for `kms` + `kmsConfig` (<3> & <4>) +and for `selector` and `selectorConfig` (<5> & <6>): ```yaml filterDefinitions: - name: encrypt - type: RecordEncryption - config: - kms: VaultKmsService - kmsConfig: + type: RecordEncryption # <1> + config: # <2> + kms: VaultKmsService # <3> + kmsConfig: # <4> vaultTransitEngineUrl: http://vault:8200/v1/transit vaultToken: password: my-token - selector: TemplateKekSelector - selectorConfig: + selector: TemplateKekSelector # <5> + selectorConfig: # <6> template: "KEK_${topicName}" ``` @@ -34,58 +35,51 @@ The proxy runtime resolves these during startup by discovering `@PluginImplName` the named plugin implementation via `ServiceLoader`, and deserialising the corresponding `@PluginImplConfig` field into the plugin's config type. -The existing APIs work, and plugins-to-plugins has proven to be useful. +The existing APIs work, and the ability for plugins to depend on other plugins has proven useful. ## Motivation -Although it works, the current configuration APIs have several consequences: - -1. **No reuse**: The model is tree-like. For example, if two filters need the same KMS - then each must embed a copy of the KMS configuration. -2. **Poor documentation**: a plugin developer ought to be able to document the YAML accepted by their plugin, - but this is difficult because the source of truth is Java code: End users cannot be assumed to know Java. -3. **Opaque to tooling**: similarly, external tools (editors, config linters, CI validators, Kubernetes operators) - cannot inspect or validate plugin configurations. (See [2436](https://github.com/kroxylicious/kroxylicious/issues/2436)) -4. **No schema validation**: there is no declarative way to validate plugin configurations before - the proxy starts. (Somewhat related: [3267](https://github.com/kroxylicious/kroxylicious/issues/3267#issuecomment-4042031788) -5. **No independent versioning**: As a plugin evolves there are limited facilities for end users - and plugin developers to agree common expectations about a plugin configuration. The plugin - implementation either accepts the YAML or it does not. If newer version of a plugin removes - support for a configuration property the user will likely be confused, because the configuration - used to work. But the plug in has no effective way to communicate that the user is speaking - the old language. -6. **No dependency ordering**: the proxy discovers plugin dependencies implicitly at initialisation - time, with no upfront visibility into the dependency graph. - -There are five specific drawbacks of the current configuration system that motivate this change. - -### 1. Per-plugin JSON schemas are impossible today - -Items 2, 3 and 4 are quite tightly related, and all relate to poorer user experience. - -However it is currently impossible to write a single JSON schema for the configuration file, because what is allowed -depends on which plugins are present at runtime. -By giving each plugin instance its own file, each -plugin version can ship its own JSON schema to be used to validate that file. +Although it works, the current configuration model has several drawbacks that motivate a redesign. + +### 1. No reuse of plugin configurations + +The configuration model is tree-like. If two filters need the same KMS +then each must embed a copy of the KMS configuration. There is no way to define a plugin +instance once and reference it from multiple consumers. + +### 2. Per-plugin JSON schemas are impossible today + +A plugin developer ought to be able to document the YAML accepted by their plugin, +but this is difficult because the source of truth is Java code: end users cannot be assumed to know Java. +Similarly, external tools (editors, config linters, CI validators, Kubernetes operators) +cannot inspect or validate plugin configurations (see [2436](https://github.com/kroxylicious/kroxylicious/issues/2436)), +and there is no declarative way to validate plugin configurations before +the proxy starts (somewhat related: [3267](https://github.com/kroxylicious/kroxylicious/issues/3267#issuecomment-4042031788)). + +These three problems — poor documentation, opaque tooling, and absent validation — share a root cause: +it is currently impossible to write a single JSON schema for the configuration file, because what is allowed +depends on which plugins are present at runtime. +If we give each plugin instance its own file, each +plugin version can ship its own JSON schema to be used to validate that file. These per-plugin schemas can be published to schema catalogues (e.g. schemastore.org) and consumed by IDEs and CI tools independently of the proxy. -### 2. Lack of uniformity in how plugins are configured +### 3. Lack of uniformity in how plugins are configured Each plugin point embeds its plugins differently. Filter factories are explicitly named (via `NamedFilterDefinition`), but other plugins like `KmsService` are anonymous because they are referenced once by their parent. This inconsistency extends across the codebase: filters, authorisers, KMS services, subject builders, and key selectors all follow slightly different -patterns. A uniform model where every plugin instance is a named, versioned resource eliminates +patterns. A uniform model where every plugin instance is a named, versioned resource would eliminate this inconsistency. -### 3. The runtime cannot understand inter-plugin dependencies +### 4. The runtime cannot understand inter-plugin dependencies The lack of uniformity means the runtime has no meaningful understanding of the dependency relationships between plugins. This matters for dynamic reloading: when a plugin configuration changes, the runtime needs to know which virtual clusters are affected. Today, dependencies are discovered implicitly at initialisation time, with no upfront visibility into the graph. Explicit -dependency declarations enable the runtime to reason about the impact of configuration changes. +dependency declarations would enable the runtime to reason about the impact of configuration changes. Note that the runtime does not (and should not) know about all plugin types. Plugins are allowed to have plugins of their own (e.g. `RecordEncryption` depends on `KmsService` and @@ -93,21 +87,21 @@ to have plugins of their own (e.g. `RecordEncryption` depends on `KmsService` an top-level definitions for each plugin type (e.g. `kmsDefinitions`), because the runtime cannot enumerate plugin types it does not know about. -### 4. Plugin instances are not individually identifiable +### 5. Plugin instances are not individually identifiable -Eventually we may need to build a control plane for clusters of proxies. Plugin instances would be -important entities in the API of that control plane. For this to work, each plugin instance must -have a unique identity. The current system gives names to filter definitions but not to the plugins -they embed. A model where every plugin instance has a name, a type, and a version makes each one -individually addressable. +Eventually we may want to build a control plane for clusters of proxies. +Plugin instances would be important entities in the API of that control plane. +For this to work, each plugin instance must have a unique identity. +The current system gives names to filter definitions but typically not to the plugins they embed. +A model where every plugin instance has a name, a type, and a version would make each one individually addressable. -### 5. No principled way to evolve plugin configuration +### 6. No principled way to evolve plugin configuration -None of the configurations (the proxy's or any of the plugins') are explicitly versioned. A plugin -developer can deprecate properties, but cannot remove them in a controlled manner where the API -evolution is obvious to the user. Explicit version strings (following Kubernetes conventions: -v1alpha1, v1beta1, v1) give plugin developers a principled mechanism for config schema evolution, -including support for multiple concurrent versions during migration periods. +None of the configurations (the proxy's or any of the plugins') are explicitly versioned. +A plugin developer can deprecate properties, but cannot remove them in a controlled manner where the API evolution is obvious to the user. +As a plugin evolves, the plugin implementation either accepts the YAML or it does not. +If a newer version of a plugin removes support for a configuration property the user will likely be confused, because the configuration used to work — but the plugin has no effective way to communicate that the user is speaking the old language. +Explicit version strings (following Kubernetes conventions: v1alpha1, v1beta1, v1) would give plugin developers a principled mechanism for config schema evolution, including support for multiple concurrent versions during migration periods. ## Proposal @@ -116,6 +110,7 @@ including support for multiple concurrent versions during migration periods. The existing `@Plugin` annotation type will be annotated `@Repeatable`, and given a new `configVersion` method. This will allow plugin implementations carry multiple `@Plugin` annotations: +* The default value for `Plugin.configVersion` will be the empty string. * the legacy (unversioned) config will be the one with an empty string as its `configVersion` * hence forth, each version of a plugin's config will be identified by a non-empty `configVersion` @@ -129,7 +124,8 @@ public class RecordEncryption ``` When implementing a plugin like `FilterFactory` which accepts a type parameter representing the configuration type, `Object` will be used as the type argument. -If desired, a plugin developer can declare some common interface type for all their config types to inherit, thus avoiding the Object type argument. + +**Design compromise: loss of compile-time type safety.** The plugin loses compile-time type checking on its configuration — the compiler can no longer verify that the config argument matches a specific type. This is unavoidable because multiple config versions form a union type, which Java's generics cannot express directly. The cost is mitigated by the `Plugins.requireConfig()` helper (which validates the config type at runtime) and by `instanceof` pattern matching in `initialize()`, which provides exhaustiveness checking within the method body. If desired, a plugin developer can declare a common `sealed interface` for all their config types, recovering some type safety while avoiding the raw `Object` type argument. In either case `initialize()`-like methods will dispatch via `instanceof`: @@ -149,9 +145,103 @@ public SharedEncryptionContext initialize( This maintains full backwards compatibility: existing single-file configurations continue to work via the legacy path. +### One file per plugin instance + +The new configuration comprises a `proxy.yaml` and a `plugins.d/` directory: + +``` +config-dir/ + proxy.yaml + plugins.d/ + io.kroxylicious.proxy.filter.FilterFactory/ + encrypt.yaml + io.kroxylicious.kms.service.KmsService/ + vault-kms.yaml + io.kroxylicious.filter.encryption.config.KekSelectorService/ + my-selector.yaml +``` + +`proxy.yaml` contains the global proxy settings (virtual clusters, gateways, management endpoints, +default filter chain, micrometer instance names) plus a new `version` field: + +```yaml +# proxy.yaml +version: v1alpha1 +management: + endpoints: + prometheus: {} +virtualClusters: + - name: demo + targetCluster: + bootstrapServers: kafka.example:9092 + gateways: + - name: default + portIdentifiesNode: + bootstrapAddress: localhost:9192 +defaultFilters: + - encrypt +``` + +The presence of the `version` field signals to the runtime that the new configuration system is in use. +When it's absent the old configuration system is used. + +Each plugin instance file specifies `name`, `type`, `version`, and optionally `config`: + +```yaml +# plugins.d/io.kroxylicious.proxy.filter.FilterFactory/encrypt.yaml +name: encrypt +type: io.kroxylicious.filter.encryption.RecordEncryption +version: v1 +config: + kms: vault-kms + selector: my-selector +``` + +The `name` field must match the filename (without the `.yaml` extension). This is a deliberate +redundancy: storing the name in two places means they can diverge, requiring a validation rule +at load time. We accept this cost because it makes plugin instance files self-describing — a file +can be understood without knowing its path, which matters for debugging, code review, and +(in the future) non-filesystem configuration sources. + +The `type` field must be a fully qualified class name. The legacy single-file format allows +short names (e.g. `RecordEncryption`), but the config2 format requires the FQCN +(e.g. `io.kroxylicious.filter.encryption.RecordEncryption`). This makes each plugin file +completely self-describing: its interpretation does not depend on which plugins happen to be +loadable by a particular proxy binary. Short names are ambiguous when two plugins share a +simple class name; FQCNs eliminate that problem entirely. + +The fully qualified names also apply to directory paths under `plugins.d/` +(e.g. `plugins.d/io.kroxylicious.proxy.filter.FilterFactory/`), which creates long paths — +a usability cost for operators working directly on the filesystem. We accept this cost for the +same reason: unambiguity and self-description. The directory structure mirrors the `ServiceLoader` +contract, where the interface FQCN is the natural key. + +The loss of brevity is mitigated by JSON Schema: each plugin version's schema can constrain +`type` using `const` or `enum`, giving editors code completion and validation without requiring +the user to type the full name. +This mitigation relies on plugin developers adding such constraints. + +### JSON Schema validation + +Plugin developers can ship a JSON Schema for each config version at: + +``` +META-INF/kroxylicious/schemas/{PluginSimpleName}/{version}.schema.yaml +``` + +When a schema is present, the runtime will validate the raw configuration against it before deserialisation. +This is opt-in: if no schema exists, the config is accepted without validation. + +A test utility `SchemaValidationAssert` is provided in `kroxylicious-filter-test-support` so plugin authors can verify their schemas accept valid configs. + +Discovery of schemas by external tooling depends on plugin developers publishing the schema so that it can be discovered by tools. +One way to do that is to publish the schema on a public webserver, allowing end users to make use of JSON Schema's `$schema` keyword. +Editors will often allow to download such schemas. +Alternatively schemastore.org provides on mechanism with good editor support. + ### Plugin references -`PluginReference` is a new API class used by plugin developers to describe their plugin's configuration dependency on some other plugin instance. +`PluginReference` is a new API class in `kroxylicious-api` used by plugin developers to describe their plugin's configuration dependency on some other plugin instance. ```java package io.kroxylicious.proxy.plugin; @@ -175,7 +265,8 @@ public record PluginReference( ``` In general, a plugin's configuration might have many such individual dependencies. -The runtime needs a way to know about all of a plugin's dependencies so we define `HasPluginReferences`: +The runtime needs a way to know about all of a plugin's dependencies so we define `HasPluginReferences` +in `kroxylicious-api`: ```java package io.kroxylicious.proxy.plugin; @@ -221,13 +312,13 @@ public record RecordEncryptionConfigV1( ``` The YAML representation of the reference is entirely up to the plugin author. -When the plugin interface type is statically known the reference can simply be the name of the depended-upon plugin instance (i.e. a JSON string). +When the plugin interface type is statically known, the reference can simply be the name of the depended-upon plugin instance (i.e. a JSON string). For example, a plugin like `RecordEncryption` knows statically that its `kms` -field always refers to a `KmsService`. Forcing the user to write -`kms: {type: io.kroxylicious.kms.service.KmsService, name: vault-kms}` in the YAML would be +field always refers to the `KmsService` plugin interface. +Forcing the user to write `kms: {type: io.kroxylicious.kms.service.KmsService, name: vault-kms}` in the YAML would be redundant — the `type` can only ever be one thing. -By keeping `PluginReference` out of the -YAML, each plugin is free to choose the most natural representation for its references. +By keeping `PluginReference` out of the YAML, each plugin is free to choose the most natural representation for its references. +A `type` property can be included where the plugin interface type is _not_ known statically. The `HasPluginReferences` interface is the contract that allows the runtime to resolve the plugin instance graph. The runtime calls @@ -241,95 +332,6 @@ Because the config versioning mechanism (`@Plugin(configVersion = "v1", configTy allows both old and new config types to coexist, the `HasPluginReferences` pattern is adopted incrementally, version by version. -### JSON Schema validation - -Plugin developers can ship a JSON Schema for each config version at: - -``` -META-INF/kroxylicious/schemas/{PluginSimpleName}/{version}.schema.yaml -``` - -When a schema is present, the runtime will validate the raw configuration against it before deserialisation. -This is opt-in: if no schema exists, the config is accepted without validation. - -A test utility `SchemaValidationAssert` is provided in `kroxylicious-filter-test-support` so plugin authors can verify their schemas accept valid configs. - -### `@Stateless` annotation - -Plugin developers can appled a new `@Stateless` annotation to their implementation -This signals to the runtime that an instance can be shared across multiple consumers. - -The decision about whether to actually do that belongs to the end user who controls the configuration used to refer to a plugin instance. -The plugin instance YAML can set `shared: true` to enable this. - -The framework will reject `shared: true` on plugins not annotated `@Stateless`. - -### One file per plugin instance - -The new configuration comprises a `proxy.yaml` and a `plugins.d/` directory: - -``` -config-dir/ - proxy.yaml - plugins.d/ - io.kroxylicious.proxy.filter.FilterFactory/ - encrypt.yaml - io.kroxylicious.kms.service.KmsService/ - vault-kms.yaml - io.kroxylicious.filter.encryption.config.KekSelectorService/ - my-selector.yaml -``` - -`proxy.yaml` contains the global proxy settings (virtual clusters, gateways, management endpoints, -default filter chain, micrometer instance names) plus a `version` field: - -```yaml -# proxy.yaml -version: v1alpha1 -management: - endpoints: - prometheus: {} -virtualClusters: - - name: demo - targetCluster: - bootstrapServers: kafka.example:9092 - gateways: - - name: default - portIdentifiesNode: - bootstrapAddress: localhost:9192 -defaultFilters: - - encrypt -``` - -Each plugin instance file specifies `name`, `type`, `version`, and optionally `config`: - -```yaml -# plugins.d/io.kroxylicious.proxy.filter.FilterFactory/encrypt.yaml -name: encrypt -type: io.kroxylicious.filter.encryption.RecordEncryption -version: v1 -config: - kms: vault-kms - selector: my-selector -``` - -The `name` field must match the filename (without the `.yaml` extension). This is validated during -loading. Including `name` in the file content ensures that plugin instance files are -self-describing: a file can be understood without knowing its path, which matters for debugging, -code review, and (in the future) non-filesystem configuration sources. - -The `type` field must be a fully qualified class name. The legacy single-file format allows -short names (e.g. `RecordEncryption`), but the config2 format requires the FQCN -(e.g. `io.kroxylicious.filter.encryption.RecordEncryption`). This makes each plugin file -completely self-describing: its interpretation does not depend on which plugins happen to be -loadable by a particular proxy binary. Short names are ambiguous when two plugins share a -simple class name; FQCNs eliminate that problem entirely. - -The loss of brevity is mitigated by JSON Schema: each plugin version's schema can constrain -`type` using `const` or `enum`, giving editors code completion and validation without requiring -the user to type the full name. - - ### Dependency graph and referential integrity The runtime builds a dependency graph from `HasPluginReferences` declarations. Versioned config @@ -342,27 +344,36 @@ validates the graph using DFS-based topological sort, detecting: The topological order determines initialisation sequence: dependencies are initialised before their dependents. -This is a deliberate deviation from Kubernetes. The Kubernetes API server does not enforce -referential integrity between resources — a Pod can reference a ConfigMap that does not yet -exist, and eventual consistency resolves the situation over time. We make a different choice -because our needs are different: the proxy cannot start with a dangling KMS reference or a -cyclic dependency. Fail-fast validation at configuration load time is preferable to discovering -broken references at runtime when the first message arrives. The cost is that the runtime must -understand the dependency structure, which is why `HasPluginReferences` exists as an explicit -contract rather than relying on introspection or convention. +Although much of the design described in this proposal is inspired the the Kubernetes model this is a deliberate deviation. +The Kubernetes API server does not enforce referential integrity between resources — a Pod can reference a ConfigMap that does not yet +exist, and eventual consistency resolves the situation over time. +We make a different choice because our needs are different: the proxy cannot start with a dangling KMS reference or a cyclic dependency. +Fail-fast validation at configuration load time is preferable to discovering broken references at runtime when the first message arrives. +The cost is that the runtime must understand ­— and validate ­— the dependency structure, which is why `HasPluginReferences` exists as an explicit contract rather than relying on introspection or convention. + +The runtime does not (and should not) statically know about all plugin types. +Plugins can depend on other plugins that the runtime has never heard of (e.g. `RecordEncryption` depends on `KmsService` and `KekSelectorService`). The `HasPluginReferences` interface lets each config type declare its own dependencies without requiring the runtime to enumerate every possible plugin interface. + +### `@Stateless` annotation + +Plugin developers can apply a new `@Stateless` annotation to their plugin implementation. +This signals to the runtime that an instance may be shared across multiple consumers. + +The decision about whether to actually do that belongs to the end user who controls the configuration used to refer to a plugin instance. +The plugin instance YAML can set `shared: true` to enable this. -The runtime does not (and should not) know about all plugin types. Plugins can depend on other -plugins that the runtime has never heard of (e.g. `RecordEncryption` depends on `KmsService` -and `KekSelectorService`). The `HasPluginReferences` interface lets each config type declare -its own dependencies without requiring the runtime to enumerate every possible plugin interface. +The framework will reject `shared: true` on plugins not annotated `@Stateless`. ### Resolved plugin registry -A `ResolvedPluginRegistry` is built during plugin resolution. -It pre-creates non-filter plugin instances (KMS services, authorisers, etc.) in dependency order. -Filter instances are created later by the proxy runtime because they need per-connection context. +A `ResolvedPluginRegistry` is built during plugin resolution. +It instantiates non-filter plugin instances (KMS services, authorisers, etc.) eagerly, in +dependency order, at startup. These are shared services whose lifecycle is tied to the proxy +process. -Versioned filter configs obtain their dependencies from the registry: +Filter instances are different: the proxy creates a new filter chain for each client connection, +so filter instances cannot be pre-created at startup. Instead, filter factories obtain their +dependencies from the registry when the runtime invokes them: ```java ResolvedPluginRegistry registry = context.resolvedPluginRegistry() @@ -372,36 +383,15 @@ KmsService kmsPlugin = registry.pluginInstance(KmsService.class, v1.kms()); Object kmsConfig = registry.pluginConfig(KmsService.class.getName(), v1.kms()); ``` - -### Migration tool - -A `migrate-config` CLI subcommand converts legacy single-file configurations into the multi-file -format: - -``` -kroxylicious migrate-config -i legacy-config.yaml -o output-dir/ -``` - -The tool: - -1. Parses the legacy configuration via `ConfigParser`. -2. Reflects on each filter's legacy config type to discover `@PluginImplName` / `@PluginImplConfig` - pairs on the canonical constructor parameters. -3. Extracts each nested plugin into its own file under `plugins.d/{interface}/{name}.yaml`. -4. Rewrites the filter config to use `PluginReference`-style maps, selecting the best available - config version. -5. Writes `proxy.yaml` by passing through the raw YAML (stripping `filterDefinitions`, replacing - `micrometer` with instance name references). - - ### Snapshot abstraction -**This subsection concerns non-public APIs. It is included in this proposal to help explain the concepts.** +**This subsection concerns non-public APIs within the runtime. It is included in this proposal to help explain the concepts.** A `Snapshot` interface in `kroxylicious-runtime` abstracts the configuration source, allowing the same resolution logic to work whether configuration comes from a filesystem directory, an in-memory representation in tests, or (in the future) some other source. ```java +interface Snapshot { /** * Returns the proxy configuration YAML content. * @@ -425,16 +415,7 @@ to work whether configuration comes from a filesystem directory, an in-memory re */ // For the filesystem implementation this is the .yaml files in the given subdirectory of plugins.d/ List pluginInstances(String pluginInterfaceName); - - /** - * Returns the password for a named resource, if one is configured. Passwords are - * provided out-of-band from the resource data for security. - * - * @param resourceName the name of the resource - * @return the password as a char array, or {@code null} if no password is configured - */ - @Nullable - char[] resourcePassword(String resourceName); + /** * Returns the metadata and data bytes for a specific plugin instance as an atomic unit. @@ -452,16 +433,15 @@ to work whether configuration comes from a filesystem directory, an in-memory re PluginInstanceContent pluginInstance( String pluginInterfaceName, String pluginInstanceName); + + + // ... +} ``` -### Passwords -Passwords are provided **out-of-band** from the resource data, not co-located with it. -The `Snapshot` interface includes a `resourcePassword(String resourceName)` method that returns the password for a named resource. -This decouples passwords from the data they protect. -For example, the passwords could be passed in a separate file, via environment variables, or read from a stream at startup. -### `PluginInstanceContent` +#### `PluginInstanceContent` A `PluginInstanceContent` has **metadata** and **data**: @@ -480,7 +460,7 @@ public record PluginInstanceContent( below), the bytes are the raw resource content (e.g. a PKCS12 keystore). The runtime checks `@ResourceType` on the plugin class to determine which deserialisation path to use. -### `PluginInstanceMetadata` +#### `PluginInstanceMetadata` ```java public record PluginInstanceMetadata( @@ -490,13 +470,8 @@ public record PluginInstanceMetadata( boolean shared, long generation) {} ``` - The `generation` is a monotonically increasing value that changes when the - resource content changes, enabling efficient change detection without comparing data bytes - (which is problematic for keystores due to non-deterministic salting and security concerns - around comparing key material). For the filesystem-based snapshot, it can be derived from - file modification time. -### Change detection: generation numbers +#### Change detection: generation numbers Each resource in the `Snapshot` carries a **generation number** — a monotonically increasing value that changes when the resource content changes. Comparing generation numbers rather than @@ -509,19 +484,20 @@ data bytes avoids: For the operator, generation maps to Kubernetes `metadata.generation` / `resourceVersion`. For filesystem deployments, it is derived from file modification time. -## Non-YAML resources -Some plugins manage binary resources — most notably TLS -key material stored as Java KeyStores (JKS, PKCS12) or PEM files. +### Non-YAML resources + For the `Snapshot` to fully encapsulate a proxy's configuration state, it must include these resources alongside YAML-based plugin configs. Not all plugin instance data is YAML. * The existing AclAuthorizer uses a non-YAML text format for its ACL rules. * Plugins commonly accept `Tls` objects for specifying secret data via Java KeyStores. - To support these it is possible to provide plugin data in any format. - For the filesystem Snapshort this is provided in a separate file in the same directory as the YAML metadata file. + +It is possible to provide plugin data in any format. +However, this is mainly provides to allow KeyStores to have first class support within the configuration system. +For the filesystem Snapshot non-YAML data is provided in a separate file in the same directory as the YAML metadata file. -### Text-based resources: inline YAML +#### Text-based resources: inline YAML For text-based resources like ACL rules, YAML multi-line syntax (`|`) is simplest. The versioned config type can use a `String` field instead of a file path: @@ -536,7 +512,7 @@ config: otherwise deny; ``` -### Filesystem layout +#### Filesystem layout Binary resources use a sidecar `.data` file alongside the YAML metadata file: @@ -552,11 +528,9 @@ config-dir/ encrypt.yaml # metadata + YAML config ``` -Binary formats (including JKS, PKCS12) require the `@ResourceType` mechanism. +#### `@ResourceType` annotation -### `@ResourceType` annotation - -For binary resource types (e.g. keystores), the plugin implementation declares its binary format via a `@ResourceType` annotation: +For binary resource types, the plugin implementation declares its binary format via a `@ResourceType` annotation: ```java @Retention(RetentionPolicy.RUNTIME) @@ -570,14 +544,53 @@ public @interface ResourceType { The runtime discovers the serde from the annotation via reflection — the same pattern used for `@Plugin(configType = ...)`. The `ResourceSerializer` converts a typed resource to bytes (for snapshot generation), and `ResourceDeserializer` converts bytes back to a typed resource (for loading). -### Store type auto-detection +#### Passwords + +Using `@ResourceType` we can write plugins configuration which depend on binary files, including KeyStores. +But some `KeyStore` types like PKCS#12 can be (or are required to be) password-protected. +If the password was managed as just another piece of metadata, and kept alongside the keystore, then most of its security value is lost. +To provide a security benefit the keystore must be handled separately from the password needed to access it until +the moment that access is needed. +For this reason passwords are provided **out-of-band** from the resource data, not co-located with it. +The `Snapshot` interface includes a `resourcePassword(String resourceName)` method that returns the password for a named resource. +This decouples passwords from the data they protect. + +```java +interface Snapshot { + // ... + + /** + * Returns the password for a named resource, if one is configured. Passwords are + * provided out-of-band from the resource data for security. + * + * @param resourceName the name of the resource + * @return the password as a char array, or {@code null} if no password is configured + */ + @Nullable + char[] resourcePassword(String resourceName); +} +``` + +For the filesystem-based snapshot, passwords are stored in a `passwords.yaml` file in the +configuration directory. This file maps resource names to their passwords: + +```yaml +# passwords.yaml +my-keystore: changeit +``` + +Other `Snapshot` implementations may source passwords differently — for example, the operator +could read them from Kubernetes Secrets, or from Vault. + -Keystore format need not be user-specified. +#### Store type auto-detection + +The keystore format need not be user-specified. Common formats have distinctive magic bytes (JKS: `0xFEEDFEED`, PKCS12: ASN.1 `0x30`, PEM: `-----BEGIN`). The deserialiser probes the bytes and selects the appropriate format. PKCS12 is the default keystore type since Java 9; JKS is legacy. -### Alias selection +#### Alias selection When a keystore contains multiple entries, the consuming plugin's config specifies the alias via a sibling property — this is a concern of the consumer, not the resource: @@ -591,7 +604,7 @@ public record VaultKmsConfigV1( ) implements HasPluginReferences { ... } ``` -### Plugin interfaces for TLS material +#### Plugin interfaces for TLS material Two marker interfaces in `kroxylicious-api` model TLS resources: @@ -610,12 +623,32 @@ public interface TrustMaterialProvider { Consumers reference these by name in their versioned config, constructing `PluginReference` values in `pluginReferences()` — the same pattern used for all other cross-file references. +### Migration tool + +A `migrate-config` CLI subcommand converts legacy single-file configurations into the multi-file +format: + +``` +kroxylicious migrate-config -i legacy-config.yaml -o output-dir/ +``` + +The tool: + +1. Parses the legacy configuration via `ConfigParser`. +2. Reflects on each filter's legacy config type to discover `@PluginImplName` / `@PluginImplConfig` + pairs on the canonical constructor parameters. +3. Extracts each nested plugin into its own file under `plugins.d/{interface}/{name}.yaml`. +4. Rewrites the filter config to use `PluginReference`-style maps, selecting the best available + config version. +5. Writes `proxy.yaml` by passing through the raw YAML (stripping `filterDefinitions`, replacing + `micrometer` with instance name references). + ## Affected/not affected projects The changes described in this proposal concern the kroxylicious project only. -In particular is only covers the kroxylicious proxy. -Follow-on changes to `kroxylicious-kubernetes-api` are `kroxylicious-operator` will be developed in a future proposal. +In particular it only covers the kroxylicious proxy. +Follow-on changes to `kroxylicious-kubernetes-api` and `kroxylicious-operator` will be developed in a future proposal. | Project | Affected | Nature of change | |---|---|---| @@ -650,7 +683,6 @@ Users migrate at their own pace. The `migrate-config` tool provides a one-shot c The legacy format is not deprecated by this proposal. - ## Rejected alternatives ### Embedding version in the config parser rather than plugin annotations From ef24f2782f81149d3cac6d1c06694607008f3f5e Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Mon, 13 Apr 2026 14:53:15 +1200 Subject: [PATCH 12/13] Typos Signed-off-by: Tom Bentley --- .../015-config2-multi-file-plugin-configuration.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/proposals/015-config2-multi-file-plugin-configuration.md b/proposals/015-config2-multi-file-plugin-configuration.md index 019661c..25a69ac 100644 --- a/proposals/015-config2-multi-file-plugin-configuration.md +++ b/proposals/015-config2-multi-file-plugin-configuration.md @@ -109,7 +109,7 @@ Explicit version strings (following Kubernetes conventions: v1alpha1, v1beta1, v The existing `@Plugin` annotation type will be annotated `@Repeatable`, and given a new `configVersion` method. -This will allow plugin implementations carry multiple `@Plugin` annotations: +This will allow plugin implementations to carry multiple `@Plugin` annotations: * The default value for `Plugin.configVersion` will be the empty string. * the legacy (unversioned) config will be the one with an empty string as its `configVersion` * hence forth, each version of a plugin's config will be identified by a non-empty `configVersion` @@ -236,8 +236,8 @@ A test utility `SchemaValidationAssert` is provided in `kroxylicious-filter-test Discovery of schemas by external tooling depends on plugin developers publishing the schema so that it can be discovered by tools. One way to do that is to publish the schema on a public webserver, allowing end users to make use of JSON Schema's `$schema` keyword. -Editors will often allow to download such schemas. -Alternatively schemastore.org provides on mechanism with good editor support. +Editors often support downloading such schemas. +Alternatively schemastore.org provides one mechanism with good editor support. ### Plugin references @@ -344,12 +344,12 @@ validates the graph using DFS-based topological sort, detecting: The topological order determines initialisation sequence: dependencies are initialised before their dependents. -Although much of the design described in this proposal is inspired the the Kubernetes model this is a deliberate deviation. +Although much of the design described in this proposal is inspired by the Kubernetes model, this is a deliberate deviation. The Kubernetes API server does not enforce referential integrity between resources — a Pod can reference a ConfigMap that does not yet exist, and eventual consistency resolves the situation over time. We make a different choice because our needs are different: the proxy cannot start with a dangling KMS reference or a cyclic dependency. Fail-fast validation at configuration load time is preferable to discovering broken references at runtime when the first message arrives. -The cost is that the runtime must understand ­— and validate ­— the dependency structure, which is why `HasPluginReferences` exists as an explicit contract rather than relying on introspection or convention. +The cost is that the runtime must understand — and validate — the dependency structure, which is why `HasPluginReferences` exists as an explicit contract rather than relying on introspection or convention. The runtime does not (and should not) statically know about all plugin types. Plugins can depend on other plugins that the runtime has never heard of (e.g. `RecordEncryption` depends on `KmsService` and `KekSelectorService`). The `HasPluginReferences` interface lets each config type declare its own dependencies without requiring the runtime to enumerate every possible plugin interface. @@ -494,7 +494,7 @@ Not all plugin instance data is YAML. * Plugins commonly accept `Tls` objects for specifying secret data via Java KeyStores. It is possible to provide plugin data in any format. -However, this is mainly provides to allow KeyStores to have first class support within the configuration system. +However, this mainly exists to allow KeyStores to have first class support within the configuration system. For the filesystem Snapshot non-YAML data is provided in a separate file in the same directory as the YAML metadata file. #### Text-based resources: inline YAML From e7f8b3bd6fb83d6651d9b8e6806a81711bcc9020 Mon Sep 17 00:00:00 2001 From: Tom Bentley Date: Thu, 16 Apr 2026 12:52:38 +1200 Subject: [PATCH 13/13] Address self-review comments on config2 proposal - Document version dispatch via instanceof rather than passing version string - Mandate K8s CRD-compatible JSON Schema dialect for plugin schemas - Require schemas for versioned plugins - Add HasPluginReferences validation rule for versioned config types - Note that unreachable plugins are not an error - Expand @Stateless with YAML example and lifecycle detail - Document ResolvedPluginRegistry as public API with full code context - Add dynamic reloading integration subsection - Fix dangling "these resources" reference in non-YAML section - Allow flexible sidecar file extensions (.p12, .jks) instead of .data - Add ANTLR line number caveat for inline resources - Add ResourceSerializer/ResourceDeserializer interface declarations - Document @ResourceType placement and package - Add trust zone limitation and K8s/Vault password sourcing detail - Note ResourceDeserializer receives bytes only, not file extension - Add concrete VaultKmsConfigV1 TLS material example - Add open question on migration tool packaging Assisted-by: Claude Opus 4.6 Signed-off-by: Tom Bentley --- ...config2-multi-file-plugin-configuration.md | 186 ++++++++++++++---- 1 file changed, 150 insertions(+), 36 deletions(-) diff --git a/proposals/015-config2-multi-file-plugin-configuration.md b/proposals/015-config2-multi-file-plugin-configuration.md index 25a69ac..5244db4 100644 --- a/proposals/015-config2-multi-file-plugin-configuration.md +++ b/proposals/015-config2-multi-file-plugin-configuration.md @@ -145,6 +145,8 @@ public SharedEncryptionContext initialize( This maintains full backwards compatibility: existing single-file configurations continue to work via the legacy path. +Note: we deliberately do not change the plugin interface APIs (e.g. `FilterFactory`) to pass the `version` string as an argument to `initialize()`. We could do that, and then the implementation could switch on the version string instead of the config type. However, dispatching via `instanceof` on the config type is more natural in Java and gives the compiler visibility into the dispatch — the version string would be an opaque `String` offering no compile-time checking. + ### One file per plugin instance The new configuration comprises a `proxy.yaml` and a `plugins.d/` directory: @@ -229,8 +231,20 @@ Plugin developers can ship a JSON Schema for each config version at: META-INF/kroxylicious/schemas/{PluginSimpleName}/{version}.schema.yaml ``` -When a schema is present, the runtime will validate the raw configuration against it before deserialisation. -This is opt-in: if no schema exists, the config is accepted without validation. +#### Schema dialect + +Plugin schemas must conform to the JSON Schema subset supported by Kubernetes Custom Resource Definitions (OpenAPI v3.0 structural schemas, derived from JSON Schema draft-04). This choice simplifies the ecosystem: + +1. Contributors and users learn one schema dialect rather than two. +2. Plugin schemas can be directly embedded in Kubernetes CRDs with minimal transformation, allowing validation as early as possible. +3. It opens the door to using Kubernetes itself as a control plane for proxy configuration. +4. The expressiveness trade-off is minor for configuration validation. + +The runtime will validate (or reserve the right to validate) that plugin-provided schemas conform to this subset. If a schema uses unsupported features the runtime will reject it at load time. + +#### Schema requirement for versioned plugins + +For versioned plugin configurations (`configVersion` is non-empty), a JSON schema is **required**. The plugin may provide a trivial schema that accepts any JSON object if it does not wish to constrain the configuration further. For legacy (unversioned) configurations, schema validation remains opt-in. A test utility `SchemaValidationAssert` is provided in `kroxylicious-filter-test-support` so plugin authors can verify their schemas accept valid configs. @@ -332,6 +346,8 @@ Because the config versioning mechanism (`@Plugin(configVersion = "v1", configTy allows both old and new config types to coexist, the `HasPluginReferences` pattern is adopted incrementally, version by version. +The runtime validates that versioned config types (`configVersion` is non-empty) implement `HasPluginReferences`. This ensures plugin developers adopting the new API declare their dependencies, and enables a future tightening of the `@Plugin` annotation to `Class configType()` when legacy support is eventually removed. + ### Dependency graph and referential integrity The runtime builds a dependency graph from `HasPluginReferences` declarations. Versioned config @@ -341,6 +357,8 @@ validates the graph using DFS-based topological sort, detecting: - **Dangling references**: a plugin references an instance that does not exist. - **Cycles**: A depends on B depends on A. +It is not an error for a plugin instance to be declared but not reachable from the proxy configuration. Such instances are simply unused and will not be initialised. This mirrors Kubernetes, where a ConfigMap can exist without being mounted by any Pod. + The topological order determines initialisation sequence: dependencies are initialised before their dependents. @@ -359,28 +377,51 @@ Plugins can depend on other plugins that the runtime has never heard of (e.g. `R Plugin developers can apply a new `@Stateless` annotation to their plugin implementation. This signals to the runtime that an instance may be shared across multiple consumers. -The decision about whether to actually do that belongs to the end user who controls the configuration used to refer to a plugin instance. -The plugin instance YAML can set `shared: true` to enable this. +The decision about whether to actually share an instance belongs to the end user who controls the configuration. +The plugin instance YAML can set `shared: true` to enable sharing: + +```yaml +# plugins.d/io.kroxylicious.kms.service.KmsService/vault-kms.yaml +name: vault-kms +type: io.kroxylicious.kms.service.vault.VaultKmsService +version: v1 +shared: true +config: + vaultTransitEngineUrl: https://vault:8200/v1/transit +``` + +When `shared: true`, the runtime creates a single instance of the plugin during startup (in dependency order, alongside other non-filter plugins). All consumers that reference this plugin instance by name receive the same object. The instance's lifecycle is tied to the proxy process — it is created once and destroyed at shutdown. + +When `shared: false` (the default), each consumer receives its own instance, created when that consumer is initialised. The framework will reject `shared: true` on plugins not annotated `@Stateless`. ### Resolved plugin registry -A `ResolvedPluginRegistry` is built during plugin resolution. -It instantiates non-filter plugin instances (KMS services, authorisers, etc.) eagerly, in +`ResolvedPluginRegistry` is a new **public API** type in `kroxylicious-api`. It is built during plugin resolution and made available to plugins via their context objects. + +The registry instantiates non-filter plugin instances (KMS services, authorisers, etc.) eagerly, in dependency order, at startup. These are shared services whose lifecycle is tied to the proxy process. Filter instances are different: the proxy creates a new filter chain for each client connection, so filter instances cannot be pre-created at startup. Instead, filter factories obtain their -dependencies from the registry when the runtime invokes them: +dependencies from the registry during `FilterFactory.initialize()`: ```java -ResolvedPluginRegistry registry = context.resolvedPluginRegistry() - .orElseThrow(() -> new PluginConfigurationException( - "v1 config requires a ResolvedPluginRegistry")); -KmsService kmsPlugin = registry.pluginInstance(KmsService.class, v1.kms()); -Object kmsConfig = registry.pluginConfig(KmsService.class.getName(), v1.kms()); +public SharedEncryptionContext initialize( + FilterFactoryContext context, Object config) { + var v1 = Plugins.requireConfig(this, config); + if (v1 instanceof RecordEncryptionConfigV1 cfg) { + ResolvedPluginRegistry registry = context.resolvedPluginRegistry() + .orElseThrow(() -> new PluginConfigurationException( + "v1 config requires a ResolvedPluginRegistry")); + KmsService kmsPlugin = registry.pluginInstance(KmsService.class, cfg.kms()); + Object kmsConfig = registry.pluginConfig(KmsService.class.getName(), cfg.kms()); + return initializeV1(context, cfg, kmsPlugin, kmsConfig); + } + // ... legacy path +} ``` ### Snapshot abstraction @@ -473,8 +514,8 @@ public record PluginInstanceMetadata( #### Change detection: generation numbers -Each resource in the `Snapshot` carries a **generation number** — a monotonically increasing -value that changes when the resource content changes. Comparing generation numbers rather than +Each plugin instance in the `Snapshot` carries a **generation number** — a monotonically increasing +value that changes when the plugin instance's content changes. Comparing generation numbers rather than data bytes avoids: - Non-determinism from keystore salting (two logically equivalent stores can have different bytes). @@ -484,19 +525,46 @@ data bytes avoids: For the operator, generation maps to Kubernetes `metadata.generation` / `resourceVersion`. For filesystem deployments, it is derived from file modification time. +#### Integration with dynamic reloading + +Generation numbers are designed to support the dynamic reloading mechanism described in [Proposal 012](https://github.com/kroxylicious/design/pull/83). That proposal defines an `applyConfiguration()` pipeline with `ChangeDetector` implementations that determine which virtual clusters need restarting. With config2, change detection works as follows: + +1. **Identify changed plugin instances.** When a new `Snapshot` arrives, compare each plugin instance's generation number against the currently active `Snapshot`. Changed generations identify which plugin instances have been modified, added, or removed. + +2. **Propagate changes through the dependency graph.** Walk the dependency graph (built from `HasPluginReferences`) to find all transitively affected plugin instances. For example, if a `KmsService` instance's generation changed, all `FilterFactory` instances that depend on it (directly or transitively) are also affected and must be re-initialised. + +3. **Map affected filters to virtual clusters.** The affected filter instances are mapped to the virtual clusters whose filter chains reference them. Only those virtual clusters need to be restarted — unaffected clusters continue serving traffic. + +4. **Detect proxy-level changes.** Changes to `proxy.yaml` itself (virtual cluster definitions, gateways, default filter lists) feed into the `VirtualClusterChangeDetector` from Proposal 012. + +This approach also addresses part of the "plugin resource tracking" gap identified in Proposal 012. That proposal notes that the runtime cannot detect changes to external resources (keystores, ACL rule files) that plugins read during initialisation. The `Snapshot` abstraction makes the runtime aware of all plugin data — including binary sidecar files — so generation changes on those files are visible to the change detection pipeline without requiring plugins to opt in to resource tracking. + ### Non-YAML resources -For the `Snapshot` to fully encapsulate a proxy's configuration state, it must include these resources alongside YAML-based plugin configs. +Not all plugin instance data is YAML. For the `Snapshot` to fully encapsulate a proxy's configuration state, it must support non-YAML data alongside YAML-based plugin configs. For example: -Not all plugin instance data is YAML. * The existing AclAuthorizer uses a non-YAML text format for its ACL rules. * Plugins commonly accept `Tls` objects for specifying secret data via Java KeyStores. It is possible to provide plugin data in any format. However, this mainly exists to allow KeyStores to have first class support within the configuration system. -For the filesystem Snapshot non-YAML data is provided in a separate file in the same directory as the YAML metadata file. - +For the filesystem `Snapshot`, non-YAML data is provided in a separate sidecar file in the same directory as the YAML metadata file: + +``` +config-dir/ + proxy.yaml + passwords.yaml + plugins.d/ + io.kroxylicious.proxy.tls.KeyMaterialProvider/ + my-keystore.yaml # metadata: name, type, version + my-keystore.p12 # binary keystore bytes (any extension except .yaml) + io.kroxylicious.proxy.filter.FilterFactory/ + encrypt.yaml # metadata + YAML config (no sidecar needed) +``` + +For a plugin instance named `foo`, the metadata is always in `foo.yaml`. If a sidecar data file exists, it must be the only other file in the directory whose name without extension is `foo` (e.g. `foo.p12`, `foo.jks`, `foo.pem`). Any extension is permitted (except `.yaml` which is taken), which allows filesystem tools that depend on extensions to work naturally. The runtime validates that at most one sidecar file exists per plugin instance. + #### Text-based resources: inline YAML For text-based resources like ACL rules, YAML multi-line syntax (`|`) is simplest. @@ -511,28 +579,17 @@ config: allow User with name = "alice" to READ Topic with name = "foo"; otherwise deny; ``` - -#### Filesystem layout -Binary resources use a sidecar `.data` file alongside the YAML metadata file: - -``` -config-dir/ - proxy.yaml - passwords.yaml - plugins.d/ - io.kroxylicious.proxy.tls.KeyMaterialProvider/ - my-keystore.yaml # metadata: name, type, version - my-keystore.data # binary keystore bytes - io.kroxylicious.proxy.filter.FilterFactory/ - encrypt.yaml # metadata + YAML config -``` +A limitation of inline text is that parsers (e.g. ANTLR) will report error line numbers relative to the start of the multi-line string, not the enclosing YAML file. This may complicate debugging for large rule sets. #### `@ResourceType` annotation -For binary resource types, the plugin implementation declares its binary format via a `@ResourceType` annotation: +For binary resource types, the plugin implementation declares its binary format via a `@ResourceType` annotation. +`@ResourceType` is applied to the plugin implementation class, alongside `@Plugin`: ```java +package io.kroxylicious.proxy.plugin; + @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.TYPE) public @interface ResourceType { @@ -541,6 +598,18 @@ public @interface ResourceType { } ``` +The serializer and deserializer interfaces are also in `io.kroxylicious.proxy.plugin`: + +```java +public interface ResourceSerializer { + byte[] serialize(T resource); +} + +public interface ResourceDeserializer { + T deserialize(byte[] data, @Nullable char[] password); +} +``` + The runtime discovers the serde from the annotation via reflection — the same pattern used for `@Plugin(configType = ...)`. The `ResourceSerializer` converts a typed resource to bytes (for snapshot generation), and `ResourceDeserializer` converts bytes back to a typed resource (for loading). @@ -579,8 +648,12 @@ configuration directory. This file maps resource names to their passwords: my-keystore: changeit ``` -Other `Snapshot` implementations may source passwords differently — for example, the operator -could read them from Kubernetes Secrets, or from Vault. +The filesystem-based `passwords.yaml` approach is the simple case. It precludes different secret content coming from different trust zones, because someone must assemble a single file where all passwords are in the clear. The `Snapshot` abstraction is designed to support richer implementations: + +* A Kubernetes-based `Snapshot` implementation could read each resource's password from a distinct `Secret` resource, allowing different passwords to originate from different trust zones (e.g. one managed by the platform team, another by the application team). +* A Vault-backed implementation could retrieve passwords from a secrets engine at read time, with per-resource Vault policies controlling access. + +The `resourcePassword()` method is agnostic to the password source. The filesystem implementation is a starting point; production deployments on Kubernetes are expected to use the operator's `Snapshot` implementation which integrates with the platform's secrets management. #### Store type auto-detection @@ -589,6 +662,7 @@ The keystore format need not be user-specified. Common formats have distinctive magic bytes (JKS: `0xFEEDFEED`, PKCS12: ASN.1 `0x30`, PEM: `-----BEGIN`). The deserialiser probes the bytes and selects the appropriate format. PKCS12 is the default keystore type since Java 9; JKS is legacy. +The `ResourceDeserializer` receives only the raw bytes and (optionally) a password. It does not receive the file extension; format detection is content-based. #### Alias selection @@ -623,6 +697,40 @@ public interface TrustMaterialProvider { Consumers reference these by name in their versioned config, constructing `PluginReference` values in `pluginReferences()` — the same pattern used for all other cross-file references. +For example, a Vault KMS plugin that needs TLS material for connecting to the Vault server: + +```yaml +# plugins.d/io.kroxylicious.kms.service.KmsService/vault-prod.yaml +configVersion: v1 +config: + vaultTransitEngineUrl: https://vault.example.com:8200/v1/transit + keyMaterial: vault-client-cert + trustMaterial: vault-ca +``` + +```java +@Plugin(configType = VaultKmsConfigV1.class, configVersion = "v1") +public class VaultKmsService implements KmsService<...> { ... } + +public record VaultKmsConfigV1( + URI vaultTransitEngineUrl, + @Nullable String keyMaterial, + @Nullable String trustMaterial +) implements HasPluginReferences { + @Override + public Stream> pluginReferences() { + return Stream.of( + PluginReference.optional(KeyMaterialProvider.class, keyMaterial), + PluginReference.optional(TrustMaterialProvider.class, trustMaterial) + ); + } +} +``` + +The `DependencyGraph` ensures that `vault-client-cert` and `vault-ca` plugin instances +are initialised before `vault-prod`, so the `ResolvedPluginRegistry` can supply fully +constructed `KeyMaterialProvider` and `TrustMaterialProvider` instances at `initialize()` time. + ### Migration tool A `migrate-config` CLI subcommand converts legacy single-file configurations into the multi-file @@ -643,6 +751,12 @@ The tool: 5. Writes `proxy.yaml` by passing through the raw YAML (stripping `filterDefinitions`, replacing `micrometer` with instance name references). +**Open question:** whether the migration tool should be a subcommand of the proxy entrypoint or +a separate binary. A subcommand is simpler to ship and discover. A separate binary keeps the +proxy image lean and avoids a slippery slope of utility functionality accumulating in the proxy +entrypoint. A Maven plugin or standalone JAR are also options. We welcome input on which approach +the community prefers. + ## Affected/not affected projects