[GPTEINFRA-16720] Generic multi-tenant loop bridge role

roles/ocp4_workload_multi_tenant_loop/README.md

@@ -0,0 +1,118 @@
+# ocp4_workload_multi_tenant_loop
+
+Generic bridge workload that provisions tenant workloads for N users on a shared OpenShift cluster.
+
+Designed for **combined catalog items** that merge cluster-level and tenant-level provisioning into a single order. This role sits at the end of the `workloads:` list and loops a separate `tenant_workloads:` list once per user.
+
+## How it works
+
+```
+workloads:                              # run once by openshift-workloads config
+├── ocp4_workload_authentication
+├── ocp4_workload_pipelines
+├── ...cluster workloads...
+└── ocp4_workload_multi_tenant_loop     # this role
+        │
+        │  reads: num_users, tenant_workloads
+        │  loops: user1 → userN
+        │
+        ├── tenant_keycloak_user  (user1)
+        ├── tenant_namespace      (user1)
+        ├── tenant_showroom       (user1)
+        ├── >> register user1 with Babylon
+        ├── tenant_keycloak_user  (user2)
+        ├── ...
+        └── >> register userN with Babylon
+```
+
+For each user, the role overrides two variables via `include_role vars:`:
+
+| Variable | Value |
+|---|---|
+| `ocp4_workload_tenant_keycloak_username` | `{{ tenant_user_prefix }}{{ user_number }}` |
+| `common_password` | Auto-generated per user |
+
+All other tenant variables in the catalog item should be Jinja2 expressions that derive from these two roots. Ansible evaluates Jinja2 lazily, so overriding the root causes the entire variable tree to resolve correctly for each user.
+
+After all tenant workloads complete for a user, the bridge registers the user with Babylon via `agnosticd_user_info`. This is the **only** per-user data source for `workshop_user_mode: multi`.
+
+## Required variables
+
+Set these in the AgnosticV catalog item:
+
+| Variable | Type | Description |
+|---|---|---|
+| `num_users` | int | Number of tenant users to provision |
+| `tenant_workloads` | list | Workload FQCNs to run per user (provision order) |
+| `tenant_remove_workloads` | list | Workload FQCNs for destroy (explicit order) |
+
+## Optional variables
+
+| Variable | Default | Description |
+|---|---|---|
+| `tenant_user_prefix` | `user` | Username prefix (users: user1, user2, ...) |
+| `tenant_user_offset` | `1` | Starting user number |
+| `tenant_password_length` | `8` | Per-user password length |
+| `tenant_pre_loop_delay` | `0` | Seconds to wait before starting tenant loop |
+| `tenant_user_info_data` | `{}` | Per-user data dict reported to Babylon (lab-specific) |
+| `tenant_user_info_msg` | `""` | Per-user message (empty = auto-generated) |
+
+## Per-user data for workshops
+
+The bridge reports per-user data to Babylon for `workshop_user_mode: multi`. Base data (`user`, `password`) is always included. Lab-specific data is defined in `tenant_user_info_data`:
+
+```yaml
+# In agnosticv common.yaml:
+tenant_user_info_data:
+  lab_ui_url: >-
+    https://showroom-{{ ocp4_workload_tenant_keycloak_username }}-showroom.{{ openshift_cluster_ingress_domain }}
+```
+
+Jinja2 expressions in `tenant_user_info_data` are evaluated per user — `ocp4_workload_tenant_keycloak_username` resolves to the current user's name.
+
+## Example: AgnosticV catalog item
+
+```yaml
+config: openshift-workloads
+num_users: 16
+
+workloads:
+- agnosticd.core_workloads.ocp4_workload_authentication
+- agnosticd.core_workloads.ocp4_workload_pipelines
+# ...cluster workloads...
+- agnosticd.core_workloads.ocp4_workload_multi_tenant_loop
+
+tenant_workloads:
+- agnosticd.namespaced_workloads.ocp4_workload_tenant_keycloak_user
+- agnosticd.namespaced_workloads.ocp4_workload_tenant_namespace
+- agnosticd.showroom.ocp4_workload_showroom
+
+tenant_remove_workloads:
+- rhpds.litellm_virtual_keys.ocp4_workload_litellm_virtual_keys
+
+remove_workloads:
+- agnosticd.core_workloads.ocp4_workload_multi_tenant_loop
+
+tenant_user_info_data:
+  lab_ui_url: >-
+    https://showroom-{{ ocp4_workload_tenant_keycloak_username }}-showroom.{{ openshift_cluster_ingress_domain }}
+
+__meta__:
+  catalog:
+    workshop_user_mode: multi
+    workshopLabUiRedirect: true
+```
+
+## Destroy behavior
+
+On destroy, the config's `remove_workloads` calls this role first. The role reads `tenant_remove_workloads` and loops users in reverse order (last user destroyed first). All errors are caught and logged — destroy never blocks cluster teardown.
+
+## Passwords
+
+Each user gets a unique password written to `{{ output_dir }}/tenant_user_{{ N }}_password`. The Ansible `password` lookup plugin generates on first call and returns the same value on re-runs — safe for idempotent provisioning.
+
+## Limitations
+
+- Provisioning is sequential (user1 completes all workloads before user2 starts)
+- If a user fails, subsequent users are not provisioned
+- No parallel provisioning in v1 — can be added as a follow-up

roles/ocp4_workload_multi_tenant_loop/defaults/main.yml

@@ -0,0 +1,121 @@
+---
+# =================================================================
+# ocp4_workload_multi_tenant_loop — default variables
+#
+# Generic bridge between cluster-level and tenant-level workloads
+# in a combined catalog item.
+#
+# HOW IT WORKS:
+#   The bridge overrides two root variables per user iteration:
+#     - ocp4_workload_tenant_keycloak_username (e.g. user1)
+#     - common_password (auto-generated per user)
+#   All other tenant vars in the catalog item should be Jinja2
+#   expressions that derive from these roots. Ansible evaluates
+#   Jinja2 lazily, so the entire variable tree resolves correctly
+#   per user without remapping every variable.
+#
+# THREE PHASES:
+#   Phase 0: Run tenant_cluster_workloads once (cluster-level)
+#   Phase 1: Loop users, run tenant_workloads per user
+#   Phase 2: Loop users again, register each with Babylon
+#
+# WHY THREE PHASES (not two):
+#   agnosticd_user_info with user: set writes per-user entries to
+#   user-data.yaml. The showroom role reads user-data.yaml at deploy
+#   time. If it finds a 'users' dict, it switches to multi-user mode
+#   and appends the username to the namespace (user1-showroom becomes
+#   user1-showroom-user1). This is correct behavior for labs that
+#   provision all users in a single play — but conflicts with our
+#   pattern where each user gets their own showroom pod in Phase 1.
+#   Phase 2 defers all agnosticd_user_info calls until after all
+#   showroom pods are deployed, so user-data.yaml is empty when
+#   showroom runs. Fixing showroom's multi-user detection would change
+#   behavior for all existing multi-user labs that rely on it.
+# =================================================================
+
+# Number of tenant users to provision. Must be > 0 when used.
+num_users: 0
+
+# Workload FQCNs to run per user (provision order). Must be non-empty.
+tenant_workloads: []
+
+# Workload FQCNs for destroy (explicit order).
+# Only EXTERNAL resources need cleanup — everything inside the
+# cluster dies when the cluster is destroyed.
+tenant_remove_workloads: []
+
+# Username prefix. Users are named {prefix}{number}: user1, user2, ...
+tenant_user_prefix: user
+
+# First user number.
+tenant_user_offset: 1
+
+# Length of auto-generated per-user passwords.
+# Each user gets a unique password stored at:
+#   {{ output_dir }}/tenant_user_{{ N }}_password
+# The password lookup is deterministic — safe for re-runs.
+tenant_password_length: 8
+
+# Seconds to wait after Phase 0 before starting Phase 1.
+# Gives async cluster resources (ArgoCD syncs, operator CRDs,
+# Keycloak realm init) time to settle.
+tenant_pre_loop_delay: 0
+
+# Per-user data reported to Babylon via agnosticd_user_info.
+# Define in agnosticv with lab-specific data (showroom URL, etc.).
+# Jinja2 expressions are evaluated per user.
+# The bridge always adds 'user' and 'password' automatically.
+#
+# Example:
+#   tenant_user_info_data:
+#     lab_ui_url: >-
+#       https://showroom-{{ ocp4_workload_tenant_keycloak_username }}-showroom.{{ openshift_cluster_ingress_domain }}
+tenant_user_info_data: {}
+
+# Per-user message for Babylon.
+# Default uses the bridge-generated per-user password.
+# Override in agnosticv if the actual Keycloak password differs
+# (e.g. when common_password is a top-level extra_var).
+# Set to "" or omit to suppress the message entirely.
+tenant_user_info_msg: "User: {{ _tenant_username }} / {{ _tenant_password }}"
+
+# Per-workload variable overrides for TENANT workloads.
+# Keys are workload FQCNs, values are dicts of var overrides.
+# Applied via set_fact before each include_role call.
+#
+# USE CASE: When the same workload runs at cluster and tenant level
+# with different vars (e.g. gitops_bootstrap with bootstrap-infra
+# vs bootstrap-tenant).
+#
+# NOTE: These overrides use set_fact, NOT include_role vars:,
+# because AAP rejects Jinja2 dict expressions in include_role vars:.
+# set_fact works here because the conflicting vars are NOT at the
+# AgnosticV top level (not extra_vars).
+tenant_workload_vars: {}
+
+# ---------------------------------------------------------------
+# Cluster workloads run ONCE inside the bridge (Phase 0).
+#
+# WHY THIS EXISTS:
+#   When a workload (e.g. gitops_bootstrap) runs at BOTH cluster
+#   and tenant level with different vars for the SAME variable
+#   names, you can't define either set at the AgnosticV top level.
+#   Top-level vars become extra_vars in AAP, and extra_vars have
+#   the HIGHEST Ansible precedence (#22 of 22). Nothing inside
+#   the playbook can override them — not set_fact, not
+#   include_role vars, nothing.
+#
+#   Solution: move the workload OUT of the config's workloads:
+#   list. Run it inside the bridge where both invocations use
+#   set_fact to apply their respective overrides. Since the
+#   conflicting vars are NOT at the top level, they're NOT
+#   extra_vars, and set_fact overrides role defaults.
+#
+# Workloads listed here must NOT also appear in the config's
+# workloads: list.
+# ---------------------------------------------------------------
+tenant_cluster_workloads: []
+
+# Per-workload variable overrides for cluster workloads.
+# Same mechanism as tenant_workload_vars but for Phase 0.
+tenant_cluster_workloads_vars: {}

roles/ocp4_workload_multi_tenant_loop/meta/main.yml

@@ -0,0 +1,18 @@
+---
+galaxy_info:
+  role_name: ocp4_workload_multi_tenant_loop
+  author: Red Hat, Prakhar Srivastava (psrivast@redhat.com)
+  description: |
+    Generic multi-tenant loop workload. Provisions a list of
+    tenant workloads for N users on a shared OpenShift cluster.
+    Used in combined catalog items that merge cluster and tenant
+    provisioning into a single order.
+  license: MIT
+  min_ansible_version: "2.9"
+  platforms: []
+  galaxy_tags:
+  - ocp
+  - openshift
+  - multitenant
+  - loop
+dependencies: []

roles/ocp4_workload_multi_tenant_loop/tasks/destroy_user.yml

@@ -0,0 +1,37 @@
+---
+# =========================================================================
+# Destroy external resources for a single user.
+#
+# Wrapped in block/rescue — errors are logged but NEVER block teardown.
+# The cluster is about to be destroyed; only EXTERNAL resources
+# (e.g. LiteMaaS virtual keys) need explicit cleanup here.
+# =========================================================================
+
+- name: "Set identity: {{ tenant_user_prefix ~ _tenant_user_num }}"
+  ansible.builtin.set_fact:
+    _tenant_username: "{{ tenant_user_prefix }}{{ _tenant_user_num }}"
+    _tenant_password: >-
+      {{ lookup('password',
+           output_dir ~ '/tenant_user_' ~ _tenant_user_num ~ '_password',
+           length=tenant_password_length | int,
+           chars=['ascii_letters', 'digits']
+         ) }}
+
+- name: "Destroy workloads for {{ _tenant_username }}"
+  block:
+  - name: "Remove workload: {{ _tenant_workload }}"
+    ansible.builtin.include_role:
+      name: "{{ _tenant_workload }}"
+    loop: "{{ tenant_remove_workloads }}"
+    loop_control:
+      loop_var: _tenant_workload
+      label: "{{ _tenant_workload }}"
+    vars:
+      ocp4_workload_tenant_keycloak_username: "{{ _tenant_username }}"
+      common_password: "{{ _tenant_password }}"
+  rescue:
+  - name: "Destroy failed — continuing: {{ _tenant_username }}"
+    ansible.builtin.debug:
+      msg: >-
+        Tenant destroy failed for {{ _tenant_username }}.
+        Error ignored — cluster teardown will clean up remaining resources.

roles/ocp4_workload_multi_tenant_loop/tasks/main.yml

@@ -0,0 +1,11 @@
+---
+# --------------------------------------------------
+# Do not modify this file
+# --------------------------------------------------
+- name: Running workload provision tasks
+  when: ACTION == "provision"
+  ansible.builtin.include_tasks: workload.yml
+
+- name: Running workload removal tasks
+  when: ACTION == "destroy"
+  ansible.builtin.include_tasks: remove_workload.yml

roles/ocp4_workload_multi_tenant_loop/tasks/provision_user.yml

@@ -0,0 +1,44 @@
+---
+# =========================================================================
+# Run all tenant workloads for a single user (Phase 1).
+#
+# VARIABLE SCOPING:
+#   ocp4_workload_tenant_keycloak_username and common_password are set
+#   as facts here (not just via include_role vars:) because:
+#
+#   1. tenant_workload_vars values may contain Jinja2 expressions that
+#      reference them. These expressions are evaluated when
+#      provision_workload.yml applies the set_fact overrides — outside
+#      include_role scope. Without facts, the references are undefined.
+#
+#   2. common_password needs to be resolvable at set_fact time for
+#      the Jinja2 chain in catalog vars (e.g. ocp4_workload_user_password
+#      → common_password) to resolve correctly.
+#
+# No agnosticd_user_info calls here — registration is in Phase 2.
+# See workload.yml header for why two-phase is necessary.
+# =========================================================================
+
+- name: "Set identity: {{ tenant_user_prefix ~ _tenant_user_num }}"
+  ansible.builtin.set_fact:
+    _tenant_username: "{{ tenant_user_prefix }}{{ _tenant_user_num }}"
+    _tenant_password: >-
+      {{ lookup('password',
+           output_dir ~ '/tenant_user_' ~ _tenant_user_num ~ '_password',
+           length=tenant_password_length | int,
+           chars=['ascii_letters', 'digits']
+         ) }}
+    ocp4_workload_tenant_keycloak_username: "{{ tenant_user_prefix }}{{ _tenant_user_num }}"
+    common_password: >-
+      {{ lookup('password',
+           output_dir ~ '/tenant_user_' ~ _tenant_user_num ~ '_password',
+           length=tenant_password_length | int,
+           chars=['ascii_letters', 'digits']
+         ) }}
+
+- name: "Provision workloads: {{ _tenant_username }}"
+  ansible.builtin.include_tasks: provision_workload.yml
+  loop: "{{ tenant_workloads }}"
+  loop_control:
+    loop_var: _tenant_workload
+    label: "{{ _tenant_workload }}"

roles/ocp4_workload_multi_tenant_loop/tasks/provision_workload.yml

@@ -0,0 +1,25 @@
+---
+# =========================================================================
+# Run a single tenant workload for the current user.
+#
+# Per-workload overrides from tenant_workload_vars are applied via
+# set_fact before include_role. This works because the vars are NOT
+# at the AgnosticV top level (not extra_vars), so set_fact can
+# override role defaults.
+# =========================================================================
+
+- name: "Apply tenant overrides: {{ _tenant_workload }}"
+  when: _tenant_workload in tenant_workload_vars
+  ansible.builtin.set_fact:
+    "{{ _tw_item.key }}": "{{ _tw_item.value }}"
+  loop: "{{ tenant_workload_vars[_tenant_workload] | dict2items }}"
+  loop_control:
+    loop_var: _tw_item
+    label: "{{ _tw_item.key }}"
+
+- name: "Run workload: {{ _tenant_workload }}"
+  ansible.builtin.include_role:
+    name: "{{ _tenant_workload }}"
+  vars:
+    ocp4_workload_tenant_keycloak_username: "{{ _tenant_username }}"
+    common_password: "{{ _tenant_password }}"