feat: MLflow operand controller for CR lifecycle and OTEL RBAC

[Bobbins228]

Summary

Adds MLflowOperandReconciler to the kagenti-operator, moving MLflow CR creation and OTEL collector RBAC bootstrap from the setup shell script into the operator. Also fixes several bugs in the OTel bootstrap that prevented traces from reaching MLflow on RHOAI clusters.

Changes

MLflow Operand Controller (original)

• internal/controller/mlflow_operand_controller.go (new) — MLflowOperandReconciler that watches DSC, creates MLflow CR, waits for readiness via requeue, creates OTEL RoleBindings in agent namespaces
• internal/controller/mlflow_operand_controller_test.go (new) — unit tests covering DSC absence, management state handling, MLflow CR creation, readiness checks, OTEL RoleBinding creation, ClusterRole fallback, and idempotency
• internal/mlflow/types.go — added MLflowSpec to MLflow type, added minimal DataScienceCluster types with v2 scheme registration
• cmd/main.go — registered MLflowOperandReconciler under --enable-mlflow gate
• internal/controller/mlflow_controller.go — updated kubebuilder RBAC markers for expanded mlflows verbs
• charts/kagenti-operator/templates/rbac/role.yaml — added datascienceclusters get/list/watch, expanded mlflows to create/update, added clusterroles get + bind (scoped to mlflow-operator roles)

OTel→MLflow Trace Export Fixes

• TLS verification — otel.go: Set ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt on the MLflow exporter so the collector trusts RHOAI's service-serving CA. Without this, exports fail with x509: certificate signed by unknown authority.
• Workspace header — Added mlflow.workspace Helm value (wired as --mlflow-workspace CLI flag). RHOAI requires x-mlflow-workspace header on every request; without it MLflow returns 500. The workspace maps 1:1 to a Kubernetes namespace and is now an explicit configuration input rather than a runtime discovery.
• Experiment ID — Added mlflow.experimentName Helm value (default kagenti-traces, wired as --mlflow-experiment-name). The bootstrap calls the MLflow API at startup to create or retrieve the experiment, then sets the correct x-mlflow-experiment-id header. Without a valid experiment ID, MLflow returns 422.
• ClusterRole preference — mlflow_operand_controller.go: Flipped resolveMLflowClusterRole to prefer mlflow-operator-mlflow-integration over mlflow-operator-mlflow-edit, ensuring gatewayendpoints/use permission is present.
• Removed hardcoded preset values — presets.go: Removed hardcoded x-mlflow-experiment-id: "0" and tls.insecure: true from YAML presets since these are now set programmatically.

Files changed

File	What
charts/kagenti-operator/values.yaml	Added mlflow.workspace and mlflow.experimentName
charts/kagenti-operator/templates/manager/manager.yaml	Wire new CLI flags from Helm values
kagenti-operator/cmd/main.go	Added --mlflow-workspace and --mlflow-experiment-name flags
kagenti-operator/internal/bootstrap/otel.go	TLS fix, config-driven workspace/experiment, MLflow API client for experiment creation
kagenti-operator/internal/bootstrap/otel_test.go	Updated tests for config-driven workspace, added workspace header test
kagenti-operator/internal/bootstrap/presets.go	Removed hardcoded experiment-id and insecure TLS
kagenti-operator/internal/controller/mlflow_operand_controller.go	ClusterRole preference fix

Manual Test Plan (OpenShift + RHOAI)

Prerequisites

• OpenShift cluster with RHOAI installed (MLflow operator available)
• kagenti-system namespace with kagenti-deps deployed
• Access to an LLM endpoint (vLLM or similar)

Step 1: Deploy the operator

helm upgrade --install kagenti-operator charts/kagenti-operator/ \
  -n kagenti-system \
  --set controllerManager.container.image.repository=<your-registry>/kagenti-operator \
  --set controllerManager.container.image.tag=<your-tag> \
  --set controllerManager.container.imagePullPolicy=Always \
  --set mlflow.enable=true \
  --set mlflow.workspace=team1 \
  --set mlflow.experimentName=kagenti-traces \
  --set otelBootstrap.enable=true

Step 2: Create agent namespace and LLM secret

kubectl create namespace team1

kubectl create secret generic llama-stack-inference-model-secret \
  --from-literal INFERENCE_MODEL="<model>" \
  --from-literal VLLM_URL="<url>" \
  --from-literal VLLM_TLS_VERIFY="<true|false>" \
  --from-literal VLLM_API_TOKEN="<token>" \
  -n team1

Step 3: Restart the operator (so bootstrap discovers the new namespace)

kubectl rollout restart deployment/kagenti-controller-manager -n kagenti-system
kubectl rollout status deployment/kagenti-controller-manager -n kagenti-system

Step 4: Verify bootstrap logs

kubectl logs deployment/kagenti-controller-manager -n kagenti-system | grep -E "bootstrap|MLflow|workspace|experiment"

Expected output:

• MLflow CRD detected, discovering MLflow CR
• Found available MLflow CR with correct tracesURL
• Created/found MLflow experiment with name: kagenti-traces, workspace: team1, and an experimentID
• OTel collector bootstrap complete

Step 5: Verify the OTel collector ConfigMap

kubectl get configmap otel-collector-config -n kagenti-system -o jsonpath='{.data.base\.yaml}'

Verify:

• x-mlflow-workspace: team1
• x-mlflow-experiment-id: "<id>" (matches bootstrap log)
• ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
• bearertokenauth/mlflow extension present

Step 6: Deploy a test agent (e.g. weather-service)

Deploy the weather agent and tool from agent-examples/ into team1, adjusting LLM env vars and removing runAsUser: 1000 from securityContext for OpenShift SCC compatibility.

Step 7: Send a request and verify traces

kubectl port-forward svc/weather-service -n team1 8080:8080 &

curl -s -X POST http://localhost:8080/ \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"message/send","id":"test-1","params":{"message":{"role":"user","messageId":"msg-1","parts":[{"type":"text","text":"What is the weather in London?"}]}}}'

Step 8: Confirm no export errors

kubectl logs deployment/otel-collector -n kagenti-system --since=5m | grep -iE "error|fail|reject|404|500|401"

Expected: No errors. Traces should flow cleanly to MLflow.

Test Results

• Unit tests pass (go test ./internal/bootstrap/ -v)
• go build ./... clean
• Pre-commit hooks pass (go-fmt, go-vet, helmlint)
• End-to-end tested on live OpenShift cluster with RHOAI MLflow
• Traces confirmed in MLflow workspace with correct experiment
• Zero export errors in OTel collector logs