refactor: Flatten ConfigMaps and introduce RoleBuilder by adwk67 · Pull Request #1041 · stackabletech/kafka-operator

adwk67 · 2026-10-12T09:41:16Z

Description

Part of https://github.com/stackabletech/internal-issues/issues/167

Overview of logic flow

%%{init: {"flowchart": {"wrappingWidth": 900, "htmlLabels": true}}}%%
flowchart TB
      START["`**build()** mod.rs:88`"]
      START --> LB
      START --> LC
      
      subgraph broker_track["Broker track"]
          direction TB
          LB["`**loop** over broker_role_group_configs · mod.rs:92`"]
          RB["`**broker::build()** role_group/broker.rs:71
          **role_config_map_data()** :120 · broker.properties, JAAS
          **common_config_map_data()** resource/config_map.rs:47 · shared entries
          **role_pod()** broker.rs:191 · kafka container, broker-only volumes
          **volume_claim_templates()** :166 · incl. bootstrap listener PVC`"]
          RBB["`**RoleGroupBuilder { .. }** broker.rs:96 · PARALLEL, no update strategy
          **build_broker_rolegroup_bootstrap_listener()** :112`"]
          LB --> RB --> RBB
      end 
          
      subgraph controller_track["Controller track"]
          direction TB
          LC["`**loop** over controller_role_group_configs · mod.rs:106`"]
          RC["`**controller::build()** role_group/controller.rs:85
          **role_config_map_data()** :133 · controller.properties, JAAS, admin-client.properties
          **common_config_map_data()** resource/config_map.rs:47 · shared entries
          **role_pod()** controller.rs:193 · kafka + quorum-manager containers, controller-only volumes
          **storage.build_pvcs()** :119`"]
          RCB["`**RoleGroupBuilder { .. }** controller.rs:110 · ORDERED_READY, RollingUpdate`"]
          LC --> RC --> RCB
      end 
          
      RBB --> ADD
      RCB --> ADD
     
      ADD["`**RoleGroupResources::add()** mod.rs:67
      **build_services()** role_group/mod.rs:120 · headless + metrics Service
      **build_config_map()** resource/config_map.rs:100 · wraps builder.config_map_data
      **build_statefulset()** resource/statefulset.rs:69 · builder.role_pod_volumes :80, builder.role_containers :82, then
      **add_common_pod_config()** :153 · config/log volumes, service account
      **add_vector_container()** :191 · **add_graceful_shutdown_config()** :112 · pod overrides last`"]
      ADD --> OUT["`**KubernetesResources { .. }** mod.rs:129
      **build_pdb()** per role :117 · **build_discovery_configmap()** :127 · **build_service_account()** · **build_role_binding()**`"]
      OUT --> APPLY["`**apply()** controller.rs:530 · outside build/`"]

      classDef broker fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
      classDef controller fill:#fde68a,stroke:#b45309,color:#78350f
      classDef shared fill:#e5e7eb,stroke:#4b5563,color:#111827
      classDef edge fill:#ffffff,stroke:#6b7280,color:#111827,stroke-dasharray:4 2
      class LB,RB,RBB broker
      class LC,RC,RCB controller
      class ADD,OUT shared
      class START,APPLY edge
      style broker_track fill:#eff6ff,stroke:#93c5fd
      style controller_track fill:#fffbeb,stroke:#fcd34d

--- PASS: kuttl (1708.82s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/configuration_kafka-latest-3.9.2_openshift-false (11.62s)
        --- PASS: kuttl/harness/cluster-operation_kafka-latest-3.9.2_zookeeper-latest-3.9.6_openshift-false (136.73s)
        --- PASS: kuttl/harness/kerberos_kafka-3.9.2_zookeeper-latest-3.9.6_openshift-false_krb5-1.21.1_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_broker-listener-class-cluster-internal_bootstrap-listener-class-external-unstable (219.61s)
        --- PASS: kuttl/harness/smoke-kraft_kafka-kraft-4.3.1_openshift-false (430.48s)
        --- PASS: kuttl/harness/upgrade_upgrade_old-3.9.2_upgrade_new-4.3.1_use-client-tls-false_use-client-auth-tls-false_openshift-false (187.26s)
        --- PASS: kuttl/harness/kraft-kerberos_kafka-kraft-4.3.1_openshift-false_krb5-1.21.1_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_broker-listener-class-cluster-internal (852.51s)
        --- PASS: kuttl/harness/logging_kafka-3.9.2_zookeeper-latest-3.9.6_openshift-false (146.85s)
        --- PASS: kuttl/harness/delete-rolegroup_kafka-3.9.2_zookeeper-latest-3.9.6_openshift-false (65.49s)
        --- PASS: kuttl/harness/smoke_kafka-3.9.2_zookeeper-3.9.6_use-client-tls-false_openshift-false (84.82s)
        --- PASS: kuttl/harness/tls_kafka-3.9.2_zookeeper-latest-3.9.6_use-client-tls-false_use-client-auth-tls-false_openshift-false (117.13s)
        --- PASS: kuttl/harness/opa_kafka-latest-3.9.2_zookeeper-latest-3.9.6_opa-latest-1.16.2_use-opa-tls-false_openshift-false_krb5-1.21.1 (165.05s)
        --- PASS: kuttl/harness/operations-kraft_kafka-kraft-4.3.1_openshift-false (511.05s)
PASS

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Author

Changes are OpenShift compatible
CRD changes approved
CRD documentation for all fields, following the style guide.
Helm chart can be installed and deployed operator works
Integration tests passed (for non trivial changes)
Changes need to be "offline" compatible
Links to generated (nightly) docs added
Release note snippet added

Reviewer

Code contains useful comments
Code contains useful logging statements
(Integration-)Test cases added
Documentation added or updated. Follows the style guide.
Changelog updated
Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Feature Tracker has been updated
Proper release label has been added
Links to generated (nightly) docs added
Release note snippet added
Add type/deprecation label & add to the deprecation schedule
Add type/experimental label & add to the experimental features tracker

…datedCluster

… role group builder

…omments

…ume per role

Co-authored-by: maltesander <contact@maltesander.com>

Reverts 95553e3 and d60d361. The eight variants are not unused: broker.rs and controller.rs import their `*Snafu` context selectors from `super` and construct them (probes, TLS and Kerberos volumes, listener volume, JVM args, pod descriptors, properties serialisation, secret lifetime). Without them the crate does not compile. The doc comment still holds: `build::Error::RoleGroup` adds the role and role group to every error from this module. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

maltesander

LGTM!

--- PASS: kuttl (1208.86s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/configuration_kafka-latest-3.9.2_openshift-false (11.45s)
        --- PASS: kuttl/harness/delete-rolegroup_kafka-3.9.2_zookeeper-latest-3.9.6_openshift-false (41.21s)
        --- PASS: kuttl/harness/logging_kafka-3.9.2_zookeeper-latest-3.9.6_openshift-false (121.49s)
        --- PASS: kuttl/harness/smoke-kraft_kafka-kraft-3.9.2_openshift-false (281.61s)
        --- PASS: kuttl/harness/opa_kafka-latest-3.9.2_zookeeper-latest-3.9.6_opa-latest-1.16.2_use-opa-tls-true_openshift-false_krb5-1.21.1 (158.76s)
        --- PASS: kuttl/harness/smoke-kraft_kafka-kraft-4.3.1_openshift-false (288.99s)
        --- PASS: kuttl/harness/smoke_kafka-3.9.2_zookeeper-3.9.6_use-client-tls-true_openshift-false (60.91s)
        --- PASS: kuttl/harness/kraft-kerberos_kafka-kraft-3.9.2_openshift-false_krb5-1.21.1_kerberos-realm-CLUSTER.LOCAL_kerberos-backend-mit_broker-listener-class-cluster-internal (480.52s)
        --- PASS: kuttl/harness/operations-kraft_kafka-kraft-4.3.1_openshift-false (382.54s)
        --- PASS: kuttl/harness/kraft-kerberos_kafka-kraft-4.3.1_openshift-false_krb5-1.21.1_kerberos-realm-CLUSTER.LOCAL_kerberos-backend-mit_broker-listener-class-cluster-internal (509.59s)
        --- PASS: kuttl/harness/cluster-operation_kafka-latest-3.9.2_zookeeper-latest-3.9.6_openshift-false (74.07s)
        --- PASS: kuttl/harness/kerberos_kafka-3.9.2_zookeeper-latest-3.9.6_openshift-false_krb5-1.21.1_kerberos-realm-CLUSTER.LOCAL_kerberos-backend-mit_broker-listener-class-cluster-internal_bootstrap-listener-class-external-unstable (94.45s)
        --- PASS: kuttl/harness/tls_kafka-3.9.2_zookeeper-latest-3.9.6_use-client-tls-true_use-client-auth-tls-true_openshift-false (172.78s)
        --- PASS: kuttl/harness/upgrade_upgrade_old-3.9.2_upgrade_new-4.3.1_use-client-tls-true_use-client-auth-tls-true_openshift-false (113.92s)
        --- PASS: kuttl/harness/operations-kraft_kafka-kraft-3.9.2_openshift-false (362.55s)
PASS

adwk67 added 3 commits October 12, 2026 10:42

refactor: Replace AnyConfig with broker and controller fields on Vali…

ebb4599

…datedCluster

refactor: Build each role group through per-role modules and a shared…

33bacdf

… role group builder

remove the render-diff dump harness, remove dead code

27f229c

adwk67 self-assigned this Oct 12, 2026

adwk67 added 4 commits October 12, 2026 13:22

pin the predicted pods to the replica count and correct the replica c…

3ea2a90

…omments

name the volume helpers after what they do and split the Kerberos vol…

85f5758

…ume per role

render the Kafka log config from the validated logging only

12ac9d2

let the role modules fill in RoleGroupBuilder directly

2693b16

adwk67 changed the title ~~refactor: Replace AnyConfig with broker and controller fields~~ refactor: Flatten ConfigMaps and introduce RoleBuilder Oct 12, 2026

adwk67 added 6 commits October 12, 2026 13:37

changelog

29a1e00

Merge branch 'main' into refactor/flatten-configmaps-add-rolebuilder

706b18f

improve doc wording

1b98491

explicit Kerberos volume push and clarify volume-order comments

a28b16e

use explicit array types for collect() output

2702fc7

added comment

58bd85e

adwk67 marked this pull request as ready for review October 13, 2026 11:55

maltesander self-requested a review October 20, 2026 09:29

maltesander requested changes Oct 20, 2026

View reviewed changes

adwk67 and others added 6 commits October 20, 2026 17:11

Update rust/operator-binary/src/controller/build/role_group/mod.rs

95553e3

Co-authored-by: maltesander <contact@maltesander.com>

Update rust/operator-binary/src/controller/build/role_group/mod.rs

d60d361

Co-authored-by: maltesander <contact@maltesander.com>

leave updateStrategy unset for both roles

9c257d3

merge main and fix CL conflict

693a6c3

do not mix broker and controller errors

dbe0a4d

maltesander previously approved these changes Oct 22, 2026

View reviewed changes

merge main and fix CL conflict

7a7dcd4

adwk67 dismissed maltesander’s stale review via 7a7dcd4 October 29, 2026 14:35

adwk67 added this to Stackable Engineering Oct 29, 2026

adwk67 moved this to Development: In Review in Stackable Engineering Oct 29, 2026

reorganise imports

22fff58

adwk67 added 3 commits October 29, 2026 17:16

tests use build() and look up by name

e17c6aa

fix CHANGELOG leftovers from earlier merge

00e9f5e

tighten up validated*config by resolving fields earlier

1fa689f

Navigation Menu

Repository navigation

refactor: Flatten ConfigMaps and introduce RoleBuilder - #1041

refactor: Flatten ConfigMaps and introduce RoleBuilder#1041
adwk67 wants to merge 24 commits into
mainfrom
refactor/flatten-configmaps-add-rolebuilder

adwk67 commented Oct 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maltesander left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

adwk67 commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Overview of logic flow

Definition of Done Checklist

Author

Reviewer

Acceptance

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maltesander left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

adwk67 commented Oct 12, 2026 •

edited

Loading