diff --git a/infra/helm/README.md b/infra/helm/README.md index 3c497af..1be5402 100644 --- a/infra/helm/README.md +++ b/infra/helm/README.md @@ -14,6 +14,8 @@ infra/helm/team-devoops/ _helpers.tpl # naming/label/image helpers deployment.yaml # generic Deployment rendered per service service.yaml # generic ClusterIP Service rendered per service + hpa.yaml # generic HorizontalPodAutoscaler, one per service + # with autoscaling.enabled -- see "Autoscaling" below ingress.yaml # nginx ingress (prefix-strip + plain rules) configmap-db.yaml # SPRING_DATASOURCE_URL/USERNAME secret-db.yaml # SPRING_DATASOURCE_PASSWORD / POSTGRES_PASSWORD @@ -225,6 +227,40 @@ kubectl -n ge83mom-devops26 get pods | grep ollama kubectl -n ge83mom-devops26 exec deploy/ollama -- ollama list ``` +## Autoscaling & self-healing + +Every app service (the 6 Spring services, `py-genai-helper`, `web-client`, +`api-docs`) gets a `HorizontalPodAutoscaler` (`templates/hpa.yaml`), driven by +CPU utilization against each service's `resources.requests.cpu` — the +in-cluster metrics-server (`v1beta1.metrics.k8s.io`) is already installed and +working on this cluster. Config lives per-service under `autoscaling:` in +`values.yaml` (`enabled`, `minReplicas`, `maxReplicas`, +`targetCPUUtilizationPercentage`); Postgres, Keycloak, Ollama and the +monitoring stack are rendered from their own templates and intentionally have +no HPA — they're stateful or singleton and shouldn't scale out. + +When a service has `autoscaling.enabled: true`, `templates/deployment.yaml` +omits `spec.replicas` entirely instead of pinning it to `1` — pinning it would +make every `helm upgrade` (which runs on every push to `main`) reset the +replica count and fight the HPA's own scaling decisions. + +**Quota caveat:** the namespace's `ResourceQuota` (`limits.cpu: 4`, +`limits.memory: 6Gi`) is committed almost in full just running one replica of +everything (`kubectl get resourcequota -n ge83mom-devops26` typically shows +~90%+ used on both). An HPA scale-up that the quota can't fit simply leaves +the extra pod `Pending` — it does not affect the already-running replica or +any other service. `kubectl get hpa -n ge83mom-devops26` shows current +`TARGETS`/replica counts; `` in the `TARGETS` column means +metrics-server isn't being reached for that resource, not that it's idle. + +Self-healing beyond autoscaling is native to Kubernetes and needs no extra +component here: every service with a `health:` path gets startup, readiness, +and liveness probes (`templates/deployment.yaml`), so the kubelet restarts a +container that stops responding, and the ReplicaSet controller replaces any +pod that's deleted or evicted. `helm upgrade --rollback-on-failure` (used by +the `deploy-k8s` pipeline job) adds deployment-level self-healing on top — +a rollout that never becomes healthy is rolled back automatically. + ## Manual deploy ```bash diff --git a/infra/helm/team-devoops/templates/deployment.yaml b/infra/helm/team-devoops/templates/deployment.yaml index 5d66f64..60e7e9d 100644 --- a/infra/helm/team-devoops/templates/deployment.yaml +++ b/infra/helm/team-devoops/templates/deployment.yaml @@ -7,10 +7,14 @@ metadata: labels: {{- include "team-devoops.labels" (dict "name" $name "root" $root) | nindent 4 }} spec: + {{- /* Omitted when HPA-managed: setting it here would make every `helm upgrade` + reset replicas back to the default and fight the HPA's scaling decisions. */}} + {{- if not (and $svc.autoscaling $svc.autoscaling.enabled) }} replicas: {{ $svc.replicas | default 1 }} + {{- end }} progressDeadlineSeconds: {{ $svc.progressDeadlineSeconds | default 600 }} strategy: - {{- toYaml $root.Values.strategy | nindent 4 }} + {{- toYaml ($svc.strategy | default $root.Values.strategy) | nindent 4 }} selector: matchLabels: {{- include "team-devoops.selectorLabels" (dict "name" $name) | nindent 6 }} @@ -27,12 +31,21 @@ spec: imagePullSecrets: {{- toYaml . | nindent 8 }} {{- end }} + terminationGracePeriodSeconds: {{ $svc.terminationGracePeriodSeconds | default 30 }} containers: - name: {{ $name }} image: {{ include "team-devoops.image" (dict "name" $name "svc" $svc "root" $root) }} imagePullPolicy: {{ $root.Values.global.imagePullPolicy }} ports: - containerPort: {{ $svc.port }} + {{- /* Gives the endpoints controller time to remove this pod from Service + routing before SIGTERM, so in-flight requests aren't dropped mid-rollout -- + matters most for maxUnavailable-driven rollouts, which briefly have zero + healthy pods for a service otherwise. */}} + lifecycle: + preStop: + exec: + command: ["sh", "-c", "sleep 5"] {{- if $svc.persistence }} volumeMounts: - name: data diff --git a/infra/helm/team-devoops/templates/hpa.yaml b/infra/helm/team-devoops/templates/hpa.yaml new file mode 100644 index 0000000..1c4ec3c --- /dev/null +++ b/infra/helm/team-devoops/templates/hpa.yaml @@ -0,0 +1,25 @@ +{{- range $name, $svc := .Values.services }} +{{- if and $svc.autoscaling $svc.autoscaling.enabled }} +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: {{ $name }} + labels: + {{- include "team-devoops.labels" (dict "name" $name "root" $) | nindent 4 }} +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: {{ $name }} + minReplicas: {{ $svc.autoscaling.minReplicas }} + maxReplicas: {{ $svc.autoscaling.maxReplicas }} + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: {{ $svc.autoscaling.targetCPUUtilizationPercentage }} +--- +{{- end }} +{{- end }} diff --git a/infra/helm/team-devoops/values.yaml b/infra/helm/team-devoops/values.yaml index 1262e91..6e415c3 100644 --- a/infra/helm/team-devoops/values.yaml +++ b/infra/helm/team-devoops/values.yaml @@ -241,6 +241,8 @@ forwardAuth: # Rolling update strategy — maxSurge: 0 ensures the old pod is terminated before # scheduling the new one, which is required to stay within the namespace CPU quota. +# Overridable per service via services..strategy (see web-client/api-docs below +# for the two services cheap enough to afford a real maxSurge: 1 zero-downtime deploy). strategy: type: RollingUpdate rollingUpdate: @@ -282,6 +284,11 @@ services: dbUser: organization health: /actuator/health stripPrefix: true + autoscaling: + enabled: true + minReplicas: 1 + maxReplicas: 2 + targetCPUUtilizationPercentage: 75 env: SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI: "https://ge83mom-devops26.stud.k8s.aet.cit.tum.de/auth/realms/devops" SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI: "http://keycloak:8080/auth/realms/devops/protocol/openid-connect/certs" @@ -301,6 +308,11 @@ services: dbUser: member health: /actuator/health stripPrefix: true + autoscaling: + enabled: true + minReplicas: 1 + maxReplicas: 2 + targetCPUUtilizationPercentage: 75 env: SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI: "https://ge83mom-devops26.stud.k8s.aet.cit.tum.de/auth/realms/devops" SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI: "http://keycloak:8080/auth/realms/devops/protocol/openid-connect/certs" @@ -315,6 +327,11 @@ services: dbUser: event health: /actuator/health stripPrefix: true + autoscaling: + enabled: true + minReplicas: 1 + maxReplicas: 2 + targetCPUUtilizationPercentage: 75 env: SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI: "https://ge83mom-devops26.stud.k8s.aet.cit.tum.de/auth/realms/devops" SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI: "http://keycloak:8080/auth/realms/devops/protocol/openid-connect/certs" @@ -326,6 +343,11 @@ services: dbUser: feedback health: /actuator/health stripPrefix: true + autoscaling: + enabled: true + minReplicas: 1 + maxReplicas: 2 + targetCPUUtilizationPercentage: 75 env: SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI: "https://ge83mom-devops26.stud.k8s.aet.cit.tum.de/auth/realms/devops" SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI: "http://keycloak:8080/auth/realms/devops/protocol/openid-connect/certs" @@ -337,6 +359,11 @@ services: dbUser: finance health: /actuator/health stripPrefix: true + autoscaling: + enabled: true + minReplicas: 1 + maxReplicas: 2 + targetCPUUtilizationPercentage: 75 env: SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI: "https://ge83mom-devops26.stud.k8s.aet.cit.tum.de/auth/realms/devops" SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI: "http://keycloak:8080/auth/realms/devops/protocol/openid-connect/certs" @@ -348,6 +375,11 @@ services: dbUser: letter health: /actuator/health stripPrefix: true + autoscaling: + enabled: true + minReplicas: 1 + maxReplicas: 2 + targetCPUUtilizationPercentage: 75 envFromSecret: letter-env env: SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_ISSUER_URI: "https://ge83mom-devops26.stud.k8s.aet.cit.tum.de/auth/realms/devops" @@ -364,6 +396,8 @@ services: # generated controllers which keep their own name in the path -- needs the # whole "/api/v1/helper" prefix removed, not just "/api/v1". fullStrip: true + # No autoscaling: its RWO PVC below can only mount to one node at a time, so a second + # replica would never schedule (stuck on a Multi-Attach error) rather than fail cleanly. envFromSecret: genai-env env: KEYCLOAK_ISSUER_URL: "https://ge83mom-devops26.stud.k8s.aet.cit.tum.de/auth/realms/devops" @@ -401,6 +435,20 @@ services: db: false health: / stripPrefix: false + autoscaling: + enabled: true + minReplicas: 1 + maxReplicas: 2 + targetCPUUtilizationPercentage: 75 + # Overrides the quota-driven global maxSurge: 0 (see .Values.strategy below) -- at + # 80Mi/80m limits this is cheap enough to actually surge an extra pod within the + # namespace's remaining quota slack, giving this service a genuine zero-downtime + # rolling deploy instead of the brief every-other-service gap maxSurge: 0 causes. + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 resources: requests: cpu: 50m @@ -413,8 +461,23 @@ services: path: /docs port: 8080 db: false + # Not "/" -- BASE_URL=/docs is baked into api/Dockerfile (swagger-ui serves its + # UI under that path, not root), so a bare "/" 404s. Verified live: GET / -> 404, + # GET /docs/ -> 200. + health: /docs/ stripPrefix: false open: true + autoscaling: + enabled: true + minReplicas: 1 + maxReplicas: 2 + targetCPUUtilizationPercentage: 75 + # See web-client above -- same reasoning, same headroom. + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 resources: requests: cpu: 50m diff --git a/web-client/.env.development.example b/web-client/.env.development.example index 5ba4fbf..6e06e28 100644 --- a/web-client/.env.development.example +++ b/web-client/.env.development.example @@ -2,11 +2,13 @@ # automatically for `pnpm dev`. The real `.env.development` is gitignored so each # developer can choose mocks vs. live backend without committing the choice. -# Serve fixtures everywhere instead of calling the backend. -# Set to false (or delete) to run against the real services. -VITE_USE_MOCKS=true +# Defaults to the real backend (requires the full stack running -- +# `docker compose up -d --build` from infra/, not just keycloak). +# Set to true to serve fixtures everywhere instead, for UI work with no backend running. +VITE_USE_MOCKS=false -# Demo as a specific persona so member-scoped pages (dashboard feedback, payments, -# development report) resolve against that fixture identity. +# Only used when VITE_USE_MOCKS=true. Demo as a specific persona so member-scoped +# pages (dashboard feedback, payments, development report) resolve against that +# fixture identity. # Options: member | coach | director | admin VITE_MOCK_PERSONA=member diff --git a/web-client/README.md b/web-client/README.md index 15a5c7f..d154f44 100644 --- a/web-client/README.md +++ b/web-client/README.md @@ -28,13 +28,13 @@ The global theme lives in `src/index.css`. Light and dark mode are driven by sem ## Prerequisites - **Node.js 20+** and **pnpm** (`npm install -g pnpm`) -- **Keycloak running locally** on port 8081 — start it with `docker compose up -d keycloak` from `infra/` (see root README) +- **Keycloak running locally** on port 8081 — start it (and the rest of the backend, see "Mock data" below) with `docker compose up -d --build` from `infra/` (see root README) ## Local development ```bash pnpm install -pnpm dev # Vite dev server at http://localhost:5173 +pnpm dev # Vite dev server at http://localhost:3000 ``` The app requires Keycloak to be reachable at `VITE_KEYCLOAK_URL` (default `http://localhost:8081`) before it renders. On first load it redirects to the Keycloak login page. @@ -45,6 +45,12 @@ To override the Keycloak URL: VITE_KEYCLOAK_URL=http://localhost:8081 pnpm dev ``` +### Mock data + +By default `pnpm dev` calls the real backend services (proxied to `http://localhost` — see `vite.config.ts`), same as every deployed environment (docker-compose, VM, Kubernetes all already run live — none of them ever set `VITE_USE_MOCKS`). To work on the UI without the backend running, copy `.env.development.example` to `.env.development` and set `VITE_USE_MOCKS=true`; every feature then serves fixtures from `src/mocks/fixtures/` instead, scoped to a demo persona selected via `VITE_MOCK_PERSONA` (`member | coach | director | admin`). `.env.development` is gitignored, so this choice is per-developer and never committed. + +**Known limitation:** every backend route is also gated by Traefik's `forward-auth` middleware, which needs its own session cookie — established by a full-page login through Traefik, then sent automatically on same-origin requests. `pnpm dev` serves the SPA itself from Vite (port 3000), so that cookie never gets set, and live-mode API calls will fail. This doesn't affect the actual deployment: browsing the docker-compose stack directly at `http://localhost/` (rather than `pnpm dev`) goes through Traefik end-to-end and works correctly. For local UI iteration against real look-and-feel without touching this, use `VITE_USE_MOCKS=true`. + ## Authentication Authentication is handled by [`src/lib/keycloak.ts`](src/lib/keycloak.ts):