Infra/106 advanced devops

infra/helm/README.md

@@ -14,6 +14,8 @@ infra/helm/team-devoops/
     _helpers.tpl              # naming/label/image helpers
     deployment.yaml           # generic Deployment rendered per service
     service.yaml              # generic ClusterIP Service rendered per service
+    hpa.yaml                  # generic HorizontalPodAutoscaler, one per service
+                               #  with autoscaling.enabled -- see "Autoscaling" below
     ingress.yaml              # nginx ingress (prefix-strip + plain rules)
     configmap-db.yaml         # SPRING_DATASOURCE_URL/USERNAME
     secret-db.yaml            # SPRING_DATASOURCE_PASSWORD / POSTGRES_PASSWORD
@@ -225,6 +227,40 @@ kubectl -n ge83mom-devops26 get pods | grep ollama
 kubectl -n ge83mom-devops26 exec deploy/ollama -- ollama list
 ```
 
+## Autoscaling & self-healing
+
+Every app service (the 6 Spring services, `py-genai-helper`, `web-client`,
+`api-docs`) gets a `HorizontalPodAutoscaler` (`templates/hpa.yaml`), driven by
+CPU utilization against each service's `resources.requests.cpu` — the
+in-cluster metrics-server (`v1beta1.metrics.k8s.io`) is already installed and
+working on this cluster. Config lives per-service under `autoscaling:` in
+`values.yaml` (`enabled`, `minReplicas`, `maxReplicas`,
+`targetCPUUtilizationPercentage`); Postgres, Keycloak, Ollama and the
+monitoring stack are rendered from their own templates and intentionally have
+no HPA — they're stateful or singleton and shouldn't scale out.
+
+When a service has `autoscaling.enabled: true`, `templates/deployment.yaml`
+omits `spec.replicas` entirely instead of pinning it to `1` — pinning it would
+make every `helm upgrade` (which runs on every push to `main`) reset the
+replica count and fight the HPA's own scaling decisions.
+
+**Quota caveat:** the namespace's `ResourceQuota` (`limits.cpu: 4`,
+`limits.memory: 6Gi`) is committed almost in full just running one replica of
+everything (`kubectl get resourcequota -n ge83mom-devops26` typically shows
+~90%+ used on both). An HPA scale-up that the quota can't fit simply leaves
+the extra pod `Pending` — it does not affect the already-running replica or
+any other service. `kubectl get hpa -n ge83mom-devops26` shows current
+`TARGETS`/replica counts; `<unknown>` in the `TARGETS` column means
+metrics-server isn't being reached for that resource, not that it's idle.
+
+Self-healing beyond autoscaling is native to Kubernetes and needs no extra
+component here: every service with a `health:` path gets startup, readiness,
+and liveness probes (`templates/deployment.yaml`), so the kubelet restarts a
+container that stops responding, and the ReplicaSet controller replaces any
+pod that's deleted or evicted. `helm upgrade --rollback-on-failure` (used by
+the `deploy-k8s` pipeline job) adds deployment-level self-healing on top —
+a rollout that never becomes healthy is rolled back automatically.
+
 ## Manual deploy
 
 ```bash

infra/helm/team-devoops/templates/deployment.yaml

@@ -7,10 +7,14 @@ metadata:
   labels:
     {{- include "team-devoops.labels" (dict "name" $name "root" $root) | nindent 4 }}
 spec:
+  {{- /* Omitted when HPA-managed: setting it here would make every `helm upgrade`
+        reset replicas back to the default and fight the HPA's scaling decisions. */}}
+  {{- if not (and $svc.autoscaling $svc.autoscaling.enabled) }}
   replicas: {{ $svc.replicas | default 1 }}
+  {{- end }}
   progressDeadlineSeconds: {{ $svc.progressDeadlineSeconds | default 600 }}
   strategy:
-    {{- toYaml $root.Values.strategy | nindent 4 }}
+    {{- toYaml ($svc.strategy | default $root.Values.strategy) | nindent 4 }}
   selector:
     matchLabels:
       {{- include "team-devoops.selectorLabels" (dict "name" $name) | nindent 6 }}
@@ -27,12 +31,21 @@ spec:
       imagePullSecrets:
         {{- toYaml . | nindent 8 }}
       {{- end }}
+      terminationGracePeriodSeconds: {{ $svc.terminationGracePeriodSeconds | default 30 }}
       containers:
         - name: {{ $name }}
           image: {{ include "team-devoops.image" (dict "name" $name "svc" $svc "root" $root) }}
           imagePullPolicy: {{ $root.Values.global.imagePullPolicy }}
           ports:
             - containerPort: {{ $svc.port }}
+          {{- /* Gives the endpoints controller time to remove this pod from Service
+                routing before SIGTERM, so in-flight requests aren't dropped mid-rollout --
+                matters most for maxUnavailable-driven rollouts, which briefly have zero
+                healthy pods for a service otherwise. */}}
+          lifecycle:
+            preStop:
+              exec:
+                command: ["sh", "-c", "sleep 5"]
           {{- if $svc.persistence }}
           volumeMounts:
             - name: data

infra/helm/team-devoops/templates/hpa.yaml

@@ -0,0 +1,25 @@
+{{- range $name, $svc := .Values.services }}
+{{- if and $svc.autoscaling $svc.autoscaling.enabled }}
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: {{ $name }}
+  labels:
+    {{- include "team-devoops.labels" (dict "name" $name "root" $) | nindent 4 }}
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: {{ $name }}
+  minReplicas: {{ $svc.autoscaling.minReplicas }}
+  maxReplicas: {{ $svc.autoscaling.maxReplicas }}
+  metrics:
+    - type: Resource
+      resource:
+        name: cpu
+        target:
+          type: Utilization
+          averageUtilization: {{ $svc.autoscaling.targetCPUUtilizationPercentage }}
+---
+{{- end }}
+{{- end }}

infra/helm/team-devoops/values.yaml

@@ -241,6 +241,8 @@ forwardAuth:
 
 # Rolling update strategy — maxSurge: 0 ensures the old pod is terminated before
 # scheduling the new one, which is required to stay within the namespace CPU quota.
+# Overridable per service via services.<name>.strategy (see web-client/api-docs below
+# for the two services cheap enough to afford a real maxSurge: 1 zero-downtime deploy).
 strategy:
   type: RollingUpdate
   rollingUpdate:
@@ -282,6 +284,11 @@ services:
     dbUser: organization
     health: /actuator/health
     stripPrefix: true
+    autoscaling:
+      enabled: true
+      minReplicas: 1
+      maxReplicas: 2
+      targetCPUUtilizationPercentage: 75
     env:
       SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI: "https://ge83mom-devops26.stud.k8s.aet.cit.tum.de/auth/realms/devops"
       SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI: "http://keycloak:8080/auth/realms/devops/protocol/openid-connect/certs"
@@ -301,6 +308,11 @@ services:
     dbUser: member
     health: /actuator/health
     stripPrefix: true
+    autoscaling:
+      enabled: true
+      minReplicas: 1
+      maxReplicas: 2
+      targetCPUUtilizationPercentage: 75
     env:
       SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI: "https://ge83mom-devops26.stud.k8s.aet.cit.tum.de/auth/realms/devops"
       SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI: "http://keycloak:8080/auth/realms/devops/protocol/openid-connect/certs"
@@ -315,6 +327,11 @@ services:
     dbUser: event
     health: /actuator/health
     stripPrefix: true
+    autoscaling:
+      enabled: true
+      minReplicas: 1
+      maxReplicas: 2
+      targetCPUUtilizationPercentage: 75
     env:
       SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI: "https://ge83mom-devops26.stud.k8s.aet.cit.tum.de/auth/realms/devops"
       SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI: "http://keycloak:8080/auth/realms/devops/protocol/openid-connect/certs"
@@ -326,6 +343,11 @@ services:
     dbUser: feedback
     health: /actuator/health
     stripPrefix: true
+    autoscaling:
+      enabled: true
+      minReplicas: 1
+      maxReplicas: 2
+      targetCPUUtilizationPercentage: 75
     env:
       SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI: "https://ge83mom-devops26.stud.k8s.aet.cit.tum.de/auth/realms/devops"
       SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI: "http://keycloak:8080/auth/realms/devops/protocol/openid-connect/certs"
@@ -337,6 +359,11 @@ services:
     dbUser: finance
     health: /actuator/health
     stripPrefix: true
+    autoscaling:
+      enabled: true
+      minReplicas: 1
+      maxReplicas: 2
+      targetCPUUtilizationPercentage: 75
     env:
       SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI: "https://ge83mom-devops26.stud.k8s.aet.cit.tum.de/auth/realms/devops"
       SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI: "http://keycloak:8080/auth/realms/devops/protocol/openid-connect/certs"
@@ -348,6 +375,11 @@ services:
     dbUser: letter
     health: /actuator/health
     stripPrefix: true
+    autoscaling:
+      enabled: true
+      minReplicas: 1
+      maxReplicas: 2
+      targetCPUUtilizationPercentage: 75
     envFromSecret: letter-env
     env:
       SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI: "https://ge83mom-devops26.stud.k8s.aet.cit.tum.de/auth/realms/devops"
@@ -364,6 +396,8 @@ services:
     # generated controllers which keep their own name in the path -- needs the
     # whole "/api/v1/helper" prefix removed, not just "/api/v1".
     fullStrip: true
+    # No autoscaling: its RWO PVC below can only mount to one node at a time, so a second
+    # replica would never schedule (stuck on a Multi-Attach error) rather than fail cleanly.
     envFromSecret: genai-env
     env:
       KEYCLOAK_ISSUER_URL: "https://ge83mom-devops26.stud.k8s.aet.cit.tum.de/auth/realms/devops"
@@ -401,6 +435,20 @@ services:
     db: false
     health: /
     stripPrefix: false
+    autoscaling:
+      enabled: true
+      minReplicas: 1
+      maxReplicas: 2
+      targetCPUUtilizationPercentage: 75
+    # Overrides the quota-driven global maxSurge: 0 (see .Values.strategy below) -- at
+    # 80Mi/80m limits this is cheap enough to actually surge an extra pod within the
+    # namespace's remaining quota slack, giving this service a genuine zero-downtime
+    # rolling deploy instead of the brief every-other-service gap maxSurge: 0 causes.
+    strategy:
+      type: RollingUpdate
+      rollingUpdate:
+        maxSurge: 1
+        maxUnavailable: 0
     resources:
       requests:
         cpu: 50m
@@ -413,8 +461,23 @@ services:
     path: /docs
     port: 8080
     db: false
+    # Not "/" -- BASE_URL=/docs is baked into api/Dockerfile (swagger-ui serves its
+    # UI under that path, not root), so a bare "/" 404s. Verified live: GET / -> 404,
+    # GET /docs/ -> 200.
+    health: /docs/
     stripPrefix: false
     open: true
+    autoscaling:
+      enabled: true
+      minReplicas: 1
+      maxReplicas: 2
+      targetCPUUtilizationPercentage: 75
+    # See web-client above -- same reasoning, same headroom.
+    strategy:
+      type: RollingUpdate
+      rollingUpdate:
+        maxSurge: 1
+        maxUnavailable: 0
     resources:
       requests:
         cpu: 50m

web-client/.env.development.example

@@ -2,11 +2,13 @@
 # automatically for `pnpm dev`. The real `.env.development` is gitignored so each
 # developer can choose mocks vs. live backend without committing the choice.
 
-# Serve fixtures everywhere instead of calling the backend.
-# Set to false (or delete) to run against the real services.
-VITE_USE_MOCKS=true
+# Defaults to the real backend (requires the full stack running --
+# `docker compose up -d --build` from infra/, not just keycloak).
+# Set to true to serve fixtures everywhere instead, for UI work with no backend running.
+VITE_USE_MOCKS=false
 
-# Demo as a specific persona so member-scoped pages (dashboard feedback, payments,
-# development report) resolve against that fixture identity.
+# Only used when VITE_USE_MOCKS=true. Demo as a specific persona so member-scoped
+# pages (dashboard feedback, payments, development report) resolve against that
+# fixture identity.
 # Options: member | coach | director | admin
 VITE_MOCK_PERSONA=member

web-client/README.md

@@ -28,13 +28,13 @@ The global theme lives in `src/index.css`. Light and dark mode are driven by sem
 ## Prerequisites
 
 - **Node.js 20+** and **pnpm** (`npm install -g pnpm`)
-- **Keycloak running locally** on port 8081 — start it with `docker compose up -d keycloak` from `infra/` (see root README)
+- **Keycloak running locally** on port 8081 — start it (and the rest of the backend, see "Mock data" below) with `docker compose up -d --build` from `infra/` (see root README)
 
 ## Local development
 
 ```bash
 pnpm install
-pnpm dev        # Vite dev server at http://localhost:5173
+pnpm dev        # Vite dev server at http://localhost:3000
 ```
 
 The app requires Keycloak to be reachable at `VITE_KEYCLOAK_URL` (default `http://localhost:8081`) before it renders. On first load it redirects to the Keycloak login page.
@@ -45,6 +45,12 @@ To override the Keycloak URL:
 VITE_KEYCLOAK_URL=http://localhost:8081 pnpm dev
 ```
 
+### Mock data
+
+By default `pnpm dev` calls the real backend services (proxied to `http://localhost` — see `vite.config.ts`), same as every deployed environment (docker-compose, VM, Kubernetes all already run live — none of them ever set `VITE_USE_MOCKS`). To work on the UI without the backend running, copy `.env.development.example` to `.env.development` and set `VITE_USE_MOCKS=true`; every feature then serves fixtures from `src/mocks/fixtures/` instead, scoped to a demo persona selected via `VITE_MOCK_PERSONA` (`member | coach | director | admin`). `.env.development` is gitignored, so this choice is per-developer and never committed.
+
+**Known limitation:** every backend route is also gated by Traefik's `forward-auth` middleware, which needs its own session cookie — established by a full-page login through Traefik, then sent automatically on same-origin requests. `pnpm dev` serves the SPA itself from Vite (port 3000), so that cookie never gets set, and live-mode API calls will fail. This doesn't affect the actual deployment: browsing the docker-compose stack directly at `http://localhost/` (rather than `pnpm dev`) goes through Traefik end-to-end and works correctly. For local UI iteration against real look-and-feel without touching this, use `VITE_USE_MOCKS=true`.
+
 ## Authentication
 
 Authentication is handled by [`src/lib/keycloak.ts`](src/lib/keycloak.ts):