diff --git a/.gitignore b/.gitignore index e5662d8f0..788f3917a 100644 --- a/.gitignore +++ b/.gitignore @@ -137,4 +137,7 @@ scratch/ # Untracked personal Claude Code files (e.g. CLAUDE.local.md, commands/my-command.local.md) .claude/**/*.local.* -CLAUDE.local.md \ No newline at end of file +CLAUDE.local.md + +# Kustomize secrets (real values — never commit) +kustomize/**/secrets.yaml \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md index 74694f84f..b1431ac97 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -28,6 +28,10 @@ When pulling in upstream Apache Superset changes, update the **"Based on"** fiel Wiki pages live in `wiki/` and are synced to the GitHub Wiki on merge to main via `.github/workflows/sync-wiki.yml`. Documentation sync runs automatically as a background process during `/commit-and-push` and `/merge-request` — it audits wiki pages, README.md, and inline documentation against code changes. See `.claude/skills/sync-documentation.md` for details. +## Kubernetes Deployment + +Kustomize overlays live in `kustomize/`. This is GeoSet-specific (not upstream Superset). `base/` is the dev/demo overlay analogous to Docker Compose; `full/` adds Redis, Celery, and Flux GitOps for production. + ## Important Notes - Always use context7 when I need code generation, setup or configuration steps, or diff --git a/README.md b/README.md index de70daa9b..b406a0b0e 100644 --- a/README.md +++ b/README.md @@ -116,6 +116,10 @@ The Dockerfile at the root of the repository uses the same Debian-based image us DOCKERFILE=Dockerfile.rhel docker compose up ``` +### Kubernetes Deployment + +Kustomize overlays are available in [`kustomize/`](./kustomize/) for deploying GeoSet to Kubernetes. The `base` overlay mirrors the Docker Compose stack (dev/demo), while `full` adds Redis, Celery workers, and Flux GitOps for production use. See [`kustomize/README.md`](./kustomize/README.md) for details. + ### Step 3 - Open GeoSet and Explore We've created an example dashboard accessible at [http://localhost:9001/superset/dashboard/geoset-example-dashboard](http://localhost:9001/superset/dashboard/geoset-example-dashboard). diff --git a/kustomize/README.md b/kustomize/README.md new file mode 100644 index 000000000..4f71003be --- /dev/null +++ b/kustomize/README.md @@ -0,0 +1,112 @@ +# GeoSet Kustomize Deployment + +Kubernetes manifests for deploying GeoSet, organized as Kustomize overlays. + +## Overlays + +| Overlay | Description | Use case | +|---------|-------------|----------| +| **base** | PostGIS, metadata DB, Superset web, sample data ingest. No Redis or Celery. | Local/dev clusters, quick demos | +| **full** | Everything in base + Redis, Celery workers, Celery beat, cache warmup configuration, Flux GitOps | Staging/production starting point | + +## Prerequisites + +- A Kubernetes cluster (minikube, kind, EKS, etc.) +- `kubectl` installed and configured +- Container images pushed (default: `jmeegan607/geoset`, `ebienstock/geoset:data-ingest-latest`) + +## Setup + +### 1. Create secrets + +Copy the example and fill in real values: + +```bash +cp kustomize/base/secrets.yaml.example kustomize/base/secrets.yaml +``` + +Edit `kustomize/base/secrets.yaml` and set: + +| Secret | Description | Required | +|--------|-------------|----------| +| `DATABASE_PASSWORD` | Superset metadata Postgres password | Yes | +| `POSTGIS_PASSWORD` | PostGIS (geospatial data) password | Yes | +| `EXAMPLES_PASSWORD` | Superset examples DB password | Yes | +| `SUPERSET_SECRET_KEY` | Flask secret key — generate with `openssl rand -base64 42` | Yes | +| `ADMIN_PASSWORD` | Superset admin user password | Yes | +| `MAPBOX_API_KEY` | Mapbox GL token for map tiles | Yes | + +> **Never commit `secrets.yaml`** — it is gitignored. Only `secrets.yaml.example` is tracked. + +### 2. Review environment variables + +Base env config lives in `base/config/superset.env`. The defaults work out of the box for most setups. Key variables: + +| Variable | Default | Notes | +|----------|---------|-------| +| `SUPERSET_CONFIG_PATH` | `superset_config_docker_light.py` (base) / `superset_config.py` (full) | Auto-switched by overlay | +| `DATABASE_HOST` | `postgres-metadata` | K8s service name | +| `POSTGIS_HOST` | `postgis` | K8s service name | +| `REDIS_HOST` | `redis` | Only used in full overlay | + +You generally don't need to change these unless you're pointing at external databases. + +### 3. Update container images (if needed) + +The base and full overlays use the image tags from the individual manifests. Add an `images` block to the relevant `kustomization.yaml` if you need to pin a different tag for your environment. + +## Deploy + +### Base (dev/demo) + +```bash +kubectl apply -k kustomize/base +``` + +### Full (staging/production) + +```bash +kubectl apply -k kustomize/full +``` + +### Verify + +```bash +kubectl -n geoset get pods +kubectl -n geoset get svc +``` + +Superset web will be available on port `8088` via the `superset-web` service. To access locally: + +```bash +kubectl -n geoset port-forward svc/superset-web 8088:8088 +``` + +### Validate manifests + +The deployment requires a local `secrets.yaml`, which is intentionally gitignored. To render the manifests for review or CI without creating local untracked files, validate against a temporary copy: + +```bash +tmp="$(mktemp -d)" +cp -R kustomize "$tmp/" +cp "$tmp/kustomize/base/secrets.yaml.example" "$tmp/kustomize/base/secrets.yaml" +kubectl kustomize "$tmp/kustomize/base" +kubectl kustomize "$tmp/kustomize/full" +``` + +## Full overlay extras + +The full overlay adds on top of base: + +- **Redis** — caching backend and Celery message broker +- **Celery workers** (2 replicas) — async query execution +- **Celery beat** — scheduled tasks including cache warmup +- **Flux GitOps** — auto-syncs from `raft-tech/GeoSet` main branch + +It also patches `superset-web` to 2 replicas, mounts the full deployment Superset config override, and adds a Redis readiness check to its init container. + +## Teardown + +```bash +kubectl delete -k kustomize/base # or kustomize/full +``` diff --git a/kustomize/base/config/superset.env b/kustomize/base/config/superset.env new file mode 100644 index 000000000..719090cd7 --- /dev/null +++ b/kustomize/base/config/superset.env @@ -0,0 +1,29 @@ +# Superset runtime +PYTHONUNBUFFERED=1 +PYTHONPATH=/app/pythonpath:/app/docker/pythonpath_dev +FLASK_DEBUG=false +SUPERSET_ENV=production +SUPERSET_LOG_LEVEL=info +SUPERSET_LOAD_EXAMPLES=no + +# Superset config — light mode (no Redis/Celery) for base deployment +SUPERSET_CONFIG_PATH=/app/docker/pythonpath_dev/superset_config_docker_light.py + +# Postgres metadata DB +DATABASE_DIALECT=postgresql +DATABASE_HOST=postgres-metadata +DATABASE_PORT=5432 +DATABASE_DB=superset +DATABASE_USER=superset + +# Examples DB (same Postgres instance as metadata) +EXAMPLES_HOST=postgres-metadata +EXAMPLES_PORT=5432 +EXAMPLES_DB=examples +EXAMPLES_USER=examples + +# PostGIS (GeoSet geospatial data) +POSTGIS_HOST=postgis +POSTGIS_PORT=5432 +POSTGIS_DB=geoset +POSTGIS_USER=geoset diff --git a/kustomize/base/kustomization.yaml b/kustomize/base/kustomization.yaml new file mode 100644 index 000000000..abfa6923b --- /dev/null +++ b/kustomize/base/kustomization.yaml @@ -0,0 +1,47 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: geoset + +configMapGenerator: + - name: superset-env + envs: + - config/superset.env + - name: postgres-metadata-initdb + files: + - postgres-metadata/examples-init.sh + - name: postgis-initdb + files: + - postgis/init.sql + +generatorOptions: + disableNameSuffixHash: true + +# images: +# - name: jmeegan607/geoset +# newTag: "6.0.48" + +resources: + # Cluster setup + - namespace.yaml + # NOTE: Copy secrets.yaml.example to secrets.yaml and fill in real values + # secrets.yaml is gitignored — never commit real secrets + - secrets.yaml + + # StatefulSets (data layer) + - postgis/statefulset.yaml + - postgis/service.yaml + - postgres-metadata/statefulset.yaml + - postgres-metadata/service.yaml + + # Jobs (init before app starts) + - superset-init/job.yaml + - sample-data-ingest/job.yaml + + # Deployments (app layer) + - superset-web/deployment.yaml + - superset-web/service.yaml + # superset-frontend is dev-only (webpack dev server) — the production image + # already includes built frontend assets. Uncomment for a dev overlay if needed. + # - superset-frontend/deployment.yaml + # - superset-frontend/service.yaml diff --git a/kustomize/base/namespace.yaml b/kustomize/base/namespace.yaml new file mode 100644 index 000000000..efd9b3122 --- /dev/null +++ b/kustomize/base/namespace.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: geoset diff --git a/kustomize/base/postgis/init.sql b/kustomize/base/postgis/init.sql new file mode 100644 index 000000000..c3757b9f2 --- /dev/null +++ b/kustomize/base/postgis/init.sql @@ -0,0 +1,62 @@ +CREATE EXTENSION IF NOT EXISTS postgis; + +CREATE TABLE IF NOT EXISTS census_state_boundaries ( + id SERIAL PRIMARY KEY, + state_code VARCHAR(2) NOT NULL, + state_gnis_code VARCHAR(8), + state_abbrev VARCHAR(2) NOT NULL, + full_geoid VARCHAR(14), + geoid VARCHAR(2), + legal_statistical_code VARCHAR(2), + land_area BIGINT, + water_area BIGINT, + state_name VARCHAR(100) NOT NULL, + state_boundary TEXT +); + +CREATE TABLE IF NOT EXISTS nifc_wildfire_locations ( + id SERIAL PRIMARY KEY, + fire_id INTEGER, + irwin_id TEXT, + incident_size DOUBLE PRECISION, + containment_time TIMESTAMPTZ, + percent_contained DOUBLE PRECISION, + control_time TIMESTAMPTZ, + incident_description TEXT, + discovery_acres DOUBLE PRECISION, + final_acres DOUBLE PRECISION, + fire_cause TEXT, + origin_coordinate TEXT, + dispatch_center_id TEXT, + fire_discovery_time TIMESTAMPTZ, + nifc_created_time TIMESTAMPTZ, + nifc_modified_time TIMESTAMPTZ, + estimated_cost_to_date DOUBLE PRECISION, + incident_name TEXT, + origin_fips_code CHAR(5), + origin_city_name TEXT, + origin_state_code CHAR(5), + origin_county_name TEXT, + landowner_type TEXT, + is_multijurisdictional BOOLEAN +); + +CREATE TABLE IF NOT EXISTS nhc_best_track ( + id SERIAL PRIMARY KEY, + effective_timestamp TIMESTAMPTZ NOT NULL, + min_sea_level_pressure_mb INTEGER, + max_gust_mph INTEGER, + storm_name TEXT NOT NULL, + nhc_identifier TEXT, + year INTEGER NOT NULL, + observation_point GEOGRAPHY(POINT, 4326) +); + +CREATE INDEX IF NOT EXISTS idx_nhc_identifier ON nhc_best_track (nhc_identifier); +CREATE INDEX IF NOT EXISTS idx_nhc_year ON nhc_best_track (year); + +-- Drop schemas that GeoSet doesn't use so Superset's schema picker only shows public. +DROP SCHEMA IF EXISTS tiger_data CASCADE; +DROP SCHEMA IF EXISTS tiger CASCADE; +DROP SCHEMA IF EXISTS topology CASCADE; +DROP SCHEMA IF EXISTS information_schema CASCADE; diff --git a/kustomize/base/postgis/service.yaml b/kustomize/base/postgis/service.yaml new file mode 100644 index 000000000..e3772fcd8 --- /dev/null +++ b/kustomize/base/postgis/service.yaml @@ -0,0 +1,11 @@ +apiVersion: v1 +kind: Service +metadata: + name: postgis +spec: + clusterIP: None + selector: + app: postgis + ports: + - port: 5432 + targetPort: 5432 diff --git a/kustomize/base/postgis/statefulset.yaml b/kustomize/base/postgis/statefulset.yaml new file mode 100644 index 000000000..7ef87ea29 --- /dev/null +++ b/kustomize/base/postgis/statefulset.yaml @@ -0,0 +1,71 @@ +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: postgis +spec: + serviceName: postgis + replicas: 1 + selector: + matchLabels: + app: postgis + template: + metadata: + labels: + app: postgis + spec: + containers: + - name: postgis + image: postgis/postgis:16-3.4 + ports: + - containerPort: 5432 + env: + - name: POSTGRES_DB + valueFrom: + configMapKeyRef: + name: superset-env + key: POSTGIS_DB + - name: POSTGRES_USER + valueFrom: + configMapKeyRef: + name: superset-env + key: POSTGIS_USER + - name: POSTGRES_PASSWORD + valueFrom: + secretKeyRef: + name: superset-secrets + key: POSTGIS_PASSWORD + resources: + requests: + cpu: 250m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi + volumeMounts: + - name: postgis-data + mountPath: /var/lib/postgresql/data + subPath: pgdata + - name: initdb + mountPath: /docker-entrypoint-initdb.d + readinessProbe: + exec: + command: ["pg_isready", "-U", "geoset", "-d", "geoset"] + initialDelaySeconds: 10 + periodSeconds: 10 + livenessProbe: + exec: + command: ["pg_isready", "-U", "geoset", "-d", "geoset"] + initialDelaySeconds: 30 + periodSeconds: 30 + volumes: + - name: initdb + configMap: + name: postgis-initdb + volumeClaimTemplates: + - metadata: + name: postgis-data + spec: + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 5Gi diff --git a/kustomize/base/postgres-metadata/examples-init.sh b/kustomize/base/postgres-metadata/examples-init.sh new file mode 100644 index 000000000..c41d0dbcc --- /dev/null +++ b/kustomize/base/postgres-metadata/examples-init.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +# Required: Superset needs a separate "examples" database and user in the metadata +# Postgres for `superset load_examples` to work. This is the k8s equivalent of +# docker/docker-entrypoint-initdb.d/examples-init.sh in the Superset repo. +set -e +psql -v ON_ERROR_STOP=1 --username "${POSTGRES_USER}" <<-EOSQL + CREATE USER ${EXAMPLES_USER} WITH PASSWORD '${EXAMPLES_PASSWORD}'; + CREATE DATABASE ${EXAMPLES_DB}; + GRANT ALL PRIVILEGES ON DATABASE ${EXAMPLES_DB} TO ${EXAMPLES_USER}; +EOSQL +psql -v ON_ERROR_STOP=1 --username "${POSTGRES_USER}" -d "${EXAMPLES_DB}" <<-EOSQL + GRANT ALL ON SCHEMA public TO ${EXAMPLES_USER}; +EOSQL diff --git a/kustomize/base/postgres-metadata/service.yaml b/kustomize/base/postgres-metadata/service.yaml new file mode 100644 index 000000000..63728fa12 --- /dev/null +++ b/kustomize/base/postgres-metadata/service.yaml @@ -0,0 +1,11 @@ +apiVersion: v1 +kind: Service +metadata: + name: postgres-metadata +spec: + clusterIP: None + selector: + app: postgres-metadata + ports: + - port: 5432 + targetPort: 5432 diff --git a/kustomize/base/postgres-metadata/statefulset.yaml b/kustomize/base/postgres-metadata/statefulset.yaml new file mode 100644 index 000000000..2c413360b --- /dev/null +++ b/kustomize/base/postgres-metadata/statefulset.yaml @@ -0,0 +1,87 @@ +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: postgres-metadata +spec: + serviceName: postgres-metadata + replicas: 1 + selector: + matchLabels: + app: postgres-metadata + template: + metadata: + labels: + app: postgres-metadata + spec: + containers: + - name: postgres + image: postgres:16-alpine + ports: + - containerPort: 5432 + env: + - name: POSTGRES_DB + valueFrom: + configMapKeyRef: + name: superset-env + key: DATABASE_DB + - name: POSTGRES_USER + valueFrom: + configMapKeyRef: + name: superset-env + key: DATABASE_USER + - name: POSTGRES_PASSWORD + valueFrom: + secretKeyRef: + name: superset-secrets + key: DATABASE_PASSWORD + - name: EXAMPLES_USER + valueFrom: + configMapKeyRef: + name: superset-env + key: EXAMPLES_USER + - name: EXAMPLES_PASSWORD + valueFrom: + secretKeyRef: + name: superset-secrets + key: EXAMPLES_PASSWORD + - name: EXAMPLES_DB + valueFrom: + configMapKeyRef: + name: superset-env + key: EXAMPLES_DB + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 500m + memory: 512Mi + volumeMounts: + - name: metadata-data + mountPath: /var/lib/postgresql/data + subPath: pgdata + - name: initdb + mountPath: /docker-entrypoint-initdb.d + readinessProbe: + exec: + command: ["pg_isready", "-U", "superset", "-d", "superset"] + initialDelaySeconds: 10 + periodSeconds: 10 + livenessProbe: + exec: + command: ["pg_isready", "-U", "superset", "-d", "superset"] + initialDelaySeconds: 30 + periodSeconds: 30 + volumes: + - name: initdb + configMap: + name: postgres-metadata-initdb + defaultMode: 0755 + volumeClaimTemplates: + - metadata: + name: metadata-data + spec: + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 2Gi diff --git a/kustomize/base/sample-data-ingest/job.yaml b/kustomize/base/sample-data-ingest/job.yaml new file mode 100644 index 000000000..b91a5cf4c --- /dev/null +++ b/kustomize/base/sample-data-ingest/job.yaml @@ -0,0 +1,47 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: sample-data-ingest +spec: + backoffLimit: 3 + template: + metadata: + labels: + app: sample-data-ingest + spec: + restartPolicy: OnFailure + initContainers: + - name: wait-for-postgis + image: busybox:1.36 + command: + - sh + - -c + - | + until nc -z postgis 5432; do + echo "waiting for postgis..." + sleep 2 + done + containers: + - name: ingest + image: ebienstock/geoset:data-ingest-latest + env: + - name: DB_HOST + value: "postgis" + - name: DB_PORT + value: "5432" + - name: DB_NAME + value: "geoset" + - name: DB_USER + value: "geoset" + - name: DB_PASSWORD + valueFrom: + secretKeyRef: + name: superset-secrets + key: POSTGIS_PASSWORD + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: 500m + memory: 512Mi diff --git a/kustomize/base/secrets.yaml.example b/kustomize/base/secrets.yaml.example new file mode 100644 index 000000000..0bec75301 --- /dev/null +++ b/kustomize/base/secrets.yaml.example @@ -0,0 +1,14 @@ +apiVersion: v1 +kind: Secret +metadata: + name: superset-secrets + namespace: geoset +type: Opaque +stringData: + # --- CHANGE THESE BEFORE ANY REAL USE --- + DATABASE_PASSWORD: "superset" + POSTGIS_PASSWORD: "geoset" + EXAMPLES_PASSWORD: "examples" + SUPERSET_SECRET_KEY: "CHANGE_ME_TO_A_RANDOM_SECRET" + ADMIN_PASSWORD: "admin" + MAPBOX_API_KEY: "CHANGE_ME" diff --git a/kustomize/base/superset-frontend/deployment.yaml b/kustomize/base/superset-frontend/deployment.yaml new file mode 100644 index 000000000..c03e5a755 --- /dev/null +++ b/kustomize/base/superset-frontend/deployment.yaml @@ -0,0 +1,37 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: superset-frontend + labels: + environment: dev-only +spec: + replicas: 1 + selector: + matchLabels: + app: superset-frontend + template: + metadata: + labels: + app: superset-frontend + component: superset + environment: dev-only + spec: + containers: + - name: frontend + image: jmeegan607/geoset + command: ["npm", "run", "dev-server", "--prefix", "/app/superset-frontend"] + ports: + - containerPort: 9000 + resources: + requests: + cpu: 250m + memory: 512Mi + limits: + cpu: "1" + memory: 1Gi + readinessProbe: + httpGet: + path: / + port: 9000 + initialDelaySeconds: 30 + periodSeconds: 10 diff --git a/kustomize/base/superset-frontend/service.yaml b/kustomize/base/superset-frontend/service.yaml new file mode 100644 index 000000000..3c056f7cf --- /dev/null +++ b/kustomize/base/superset-frontend/service.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: Service +metadata: + name: superset-frontend +spec: + selector: + app: superset-frontend + ports: + - port: 9000 + targetPort: 9000 diff --git a/kustomize/base/superset-init/job.yaml b/kustomize/base/superset-init/job.yaml new file mode 100644 index 000000000..514d63d08 --- /dev/null +++ b/kustomize/base/superset-init/job.yaml @@ -0,0 +1,63 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: superset-init +spec: + backoffLimit: 3 + template: + metadata: + labels: + app: superset-init + component: superset + spec: + restartPolicy: OnFailure + initContainers: + - name: wait-for-db + image: busybox:1.36 + command: + - sh + - -c + - | + until nc -z postgres-metadata 5432; do + echo "waiting for postgres-metadata..." + sleep 2 + done + until nc -z postgis 5432; do + echo "waiting for postgis..." + sleep 2 + done + containers: + - name: init + image: jmeegan607/geoset + command: ["/app/docker/docker-init-geoset.sh"] + envFrom: + - configMapRef: + name: superset-env + env: + - name: DATABASE_PASSWORD + valueFrom: + secretKeyRef: + name: superset-secrets + key: DATABASE_PASSWORD + - name: EXAMPLES_PASSWORD + valueFrom: + secretKeyRef: + name: superset-secrets + key: EXAMPLES_PASSWORD + - name: SUPERSET_SECRET_KEY + valueFrom: + secretKeyRef: + name: superset-secrets + key: SUPERSET_SECRET_KEY + - name: ADMIN_PASSWORD + valueFrom: + secretKeyRef: + name: superset-secrets + key: ADMIN_PASSWORD + resources: + requests: + cpu: 250m + memory: 512Mi + limits: + cpu: "1" + memory: 1Gi diff --git a/kustomize/base/superset-web/deployment.yaml b/kustomize/base/superset-web/deployment.yaml new file mode 100644 index 000000000..91c617e14 --- /dev/null +++ b/kustomize/base/superset-web/deployment.yaml @@ -0,0 +1,69 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: superset-web +spec: + replicas: 1 + selector: + matchLabels: + app: superset-web + template: + metadata: + labels: + app: superset-web + component: superset + spec: + initContainers: + - name: wait-for-db + image: busybox:1.36 + command: + - sh + - -c + - | + until nc -z postgres-metadata 5432; do + echo "waiting for postgres-metadata..." + sleep 2 + done + containers: + - name: superset + image: jmeegan607/geoset + envFrom: + - configMapRef: + name: superset-env + env: + - name: DATABASE_PASSWORD + valueFrom: + secretKeyRef: + name: superset-secrets + key: DATABASE_PASSWORD + - name: EXAMPLES_PASSWORD + valueFrom: + secretKeyRef: + name: superset-secrets + key: EXAMPLES_PASSWORD + - name: SUPERSET_SECRET_KEY + valueFrom: + secretKeyRef: + name: superset-secrets + key: SUPERSET_SECRET_KEY + - name: MAPBOX_API_KEY + valueFrom: + secretKeyRef: + name: superset-secrets + key: MAPBOX_API_KEY + - name: SERVER_WORKER_AMOUNT + value: "4" + ports: + - containerPort: 8088 + readinessProbe: + httpGet: + path: /health + port: 8088 + initialDelaySeconds: 30 + periodSeconds: 10 + livenessProbe: + httpGet: + path: /health + port: 8088 + initialDelaySeconds: 60 + periodSeconds: 30 diff --git a/kustomize/base/superset-web/service.yaml b/kustomize/base/superset-web/service.yaml new file mode 100644 index 000000000..ae437bf64 --- /dev/null +++ b/kustomize/base/superset-web/service.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: Service +metadata: + name: superset-web +spec: + selector: + app: superset-web + ports: + - port: 8088 + targetPort: 8088 diff --git a/kustomize/full/config/superset-env-patch.env b/kustomize/full/config/superset-env-patch.env new file mode 100644 index 000000000..401cbe4ff --- /dev/null +++ b/kustomize/full/config/superset-env-patch.env @@ -0,0 +1,8 @@ +# Override: use full config (with Celery/Redis), not light mode +SUPERSET_CONFIG_PATH=/app/docker/pythonpath_dev/superset_config.py + +# Redis (required for Celery workers/beat) +REDIS_HOST=redis +REDIS_PORT=6379 +REDIS_CELERY_DB=0 +REDIS_RESULTS_DB=1 diff --git a/kustomize/full/config/superset_config_docker.py b/kustomize/full/config/superset_config_docker.py new file mode 100644 index 000000000..94df228a3 --- /dev/null +++ b/kustomize/full/config/superset_config_docker.py @@ -0,0 +1,60 @@ +"""GeoSet overrides for the full Kubernetes deployment (Redis + Celery). + +Mounted as a ConfigMap into the container as superset_config_docker.py +so it is auto-imported by the upstream base config. +""" + +import logging + +from celery.signals import task_failure, task_postrun, task_prerun +from superset.tasks.types import ExecutorType, FixedExecutor + +# Fallback executor for charts without owners (e.g. GeoSet example charts) +CACHE_WARMUP_EXECUTORS = [ExecutorType.OWNER, FixedExecutor("admin")] + +# --------------------------------------------------------------------------- +# Cache warmup logging via Celery signals +# --------------------------------------------------------------------------- +_warmup_logger = logging.getLogger("geoset.cache_warmup") + + +@task_prerun.connect(sender=None) +def _log_cache_warmup_start(sender=None, task_id=None, args=None, kwargs=None, **kw): + if sender and sender.name == "cache-warmup": + strategy = kwargs.get("strategy_name", "unknown") if kwargs else "unknown" + top_n = kwargs.get("top_n", "N/A") if kwargs else "N/A" + _warmup_logger.info( + "[CACHE-WARMUP] Starting | strategy=%s top_n=%s task_id=%s", + strategy, top_n, task_id, + ) + + +@task_postrun.connect(sender=None) +def _log_cache_warmup_done( + sender=None, task_id=None, retval=None, state=None, **kw +): + if sender and sender.name == "cache-warmup": + if isinstance(retval, dict): + scheduled = len(retval.get("scheduled", [])) + errors = len(retval.get("errors", [])) + _warmup_logger.info( + "[CACHE-WARMUP] Finished | scheduled=%d errors=%d state=%s task_id=%s", + scheduled, errors, state, task_id, + ) + else: + _warmup_logger.warning( + "[CACHE-WARMUP] Finished with non-dict result | result=%s state=%s task_id=%s", + retval, state, task_id, + ) + + +@task_failure.connect(sender=None) +def _log_cache_warmup_failure( + sender=None, task_id=None, exception=None, traceback=None, **kw +): + if sender and sender.name == "cache-warmup": + _warmup_logger.error( + "[CACHE-WARMUP] FAILED | exception=%s task_id=%s", + exception, task_id, + exc_info=True, + ) diff --git a/kustomize/full/flux/git-source.yaml b/kustomize/full/flux/git-source.yaml new file mode 100644 index 000000000..3da07c41b --- /dev/null +++ b/kustomize/full/flux/git-source.yaml @@ -0,0 +1,10 @@ +apiVersion: source.toolkit.fluxcd.io/v1 +kind: GitRepository +metadata: + name: geoset + namespace: flux-system +spec: + interval: 1m + url: https://github.com/raft-tech/GeoSet.git + ref: + branch: main diff --git a/kustomize/full/flux/kustomization.yaml b/kustomize/full/flux/kustomization.yaml new file mode 100644 index 000000000..e10448d74 --- /dev/null +++ b/kustomize/full/flux/kustomization.yaml @@ -0,0 +1,13 @@ +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: geoset + namespace: flux-system +spec: + interval: 1m + sourceRef: + kind: GitRepository + name: geoset + path: ./kustomize/full + prune: true + targetNamespace: geoset diff --git a/kustomize/full/kustomization.yaml b/kustomize/full/kustomization.yaml new file mode 100644 index 000000000..8654e9c29 --- /dev/null +++ b/kustomize/full/kustomization.yaml @@ -0,0 +1,34 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: geoset + +# Inherit everything from base +resources: + - ../base + + # Additional services for full deployment + - redis/statefulset.yaml + - redis/service.yaml + - superset-worker/deployment.yaml + - superset-beat/deployment.yaml + +# Override the superset-env configmap to add Redis vars and switch to full config +configMapGenerator: + - name: superset-env + behavior: merge + envs: + - config/superset-env-patch.env + - name: superset-config-overrides + files: + - superset_config_docker.py=config/superset_config_docker.py + +generatorOptions: + disableNameSuffixHash: true + +# Patch superset-web: bump replicas to 2 and wait for Redis +patches: + - path: superset-web-patch.yaml + target: + kind: Deployment + name: superset-web diff --git a/kustomize/full/redis/service.yaml b/kustomize/full/redis/service.yaml new file mode 100644 index 000000000..19e6f17dd --- /dev/null +++ b/kustomize/full/redis/service.yaml @@ -0,0 +1,11 @@ +apiVersion: v1 +kind: Service +metadata: + name: redis +spec: + clusterIP: None + selector: + app: redis + ports: + - port: 6379 + targetPort: 6379 diff --git a/kustomize/full/redis/statefulset.yaml b/kustomize/full/redis/statefulset.yaml new file mode 100644 index 000000000..8d3a2a72a --- /dev/null +++ b/kustomize/full/redis/statefulset.yaml @@ -0,0 +1,41 @@ +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: redis +spec: + serviceName: redis + replicas: 1 + selector: + matchLabels: + app: redis + template: + metadata: + labels: + app: redis + spec: + containers: + - name: redis + image: redis:7-alpine + ports: + - containerPort: 6379 + volumeMounts: + - name: redis-data + mountPath: /data + readinessProbe: + exec: + command: ["redis-cli", "ping"] + initialDelaySeconds: 5 + periodSeconds: 10 + livenessProbe: + exec: + command: ["redis-cli", "ping"] + initialDelaySeconds: 15 + periodSeconds: 20 + volumeClaimTemplates: + - metadata: + name: redis-data + spec: + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 1Gi diff --git a/kustomize/full/superset-beat/deployment.yaml b/kustomize/full/superset-beat/deployment.yaml new file mode 100644 index 000000000..e65ca4d66 --- /dev/null +++ b/kustomize/full/superset-beat/deployment.yaml @@ -0,0 +1,67 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: superset-beat +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: superset-beat + template: + metadata: + labels: + app: superset-beat + component: superset + spec: + initContainers: + - name: wait-for-redis + image: busybox:1.36 + command: + - sh + - -c + - | + until nc -z redis 6379; do + echo "waiting for redis..." + sleep 2 + done + containers: + - name: beat + image: jmeegan607/geoset + imagePullPolicy: Always + command: ["celery", "--app=superset.tasks.celery_app:app", + "beat", "--loglevel=INFO", + "--schedule=/tmp/celerybeat-schedule"] + envFrom: + - configMapRef: + name: superset-env + env: + - name: DATABASE_PASSWORD + valueFrom: + secretKeyRef: + name: superset-secrets + key: DATABASE_PASSWORD + - name: EXAMPLES_PASSWORD + valueFrom: + secretKeyRef: + name: superset-secrets + key: EXAMPLES_PASSWORD + - name: SUPERSET_SECRET_KEY + valueFrom: + secretKeyRef: + name: superset-secrets + key: SUPERSET_SECRET_KEY + - name: MAPBOX_API_KEY + valueFrom: + secretKeyRef: + name: superset-secrets + key: MAPBOX_API_KEY + volumeMounts: + - name: config-overrides + mountPath: /app/docker/pythonpath_dev/superset_config_docker.py + subPath: superset_config_docker.py + volumes: + - name: config-overrides + configMap: + name: superset-config-overrides diff --git a/kustomize/full/superset-web-patch.yaml b/kustomize/full/superset-web-patch.yaml new file mode 100644 index 000000000..15cea4650 --- /dev/null +++ b/kustomize/full/superset-web-patch.yaml @@ -0,0 +1,34 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: superset-web +spec: + replicas: 2 + template: + spec: + initContainers: + - name: wait-for-db + image: busybox:1.36 + command: + - sh + - -c + - | + until nc -z postgres-metadata 5432; do + echo "waiting for postgres-metadata..." + sleep 2 + done + until nc -z redis 6379; do + echo "waiting for redis..." + sleep 2 + done + containers: + - name: superset + imagePullPolicy: Always + volumeMounts: + - name: config-overrides + mountPath: /app/docker/pythonpath_dev/superset_config_docker.py + subPath: superset_config_docker.py + volumes: + - name: config-overrides + configMap: + name: superset-config-overrides diff --git a/kustomize/full/superset-worker/deployment.yaml b/kustomize/full/superset-worker/deployment.yaml new file mode 100644 index 000000000..f73c7c50b --- /dev/null +++ b/kustomize/full/superset-worker/deployment.yaml @@ -0,0 +1,74 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: superset-worker +spec: + replicas: 2 + selector: + matchLabels: + app: superset-worker + template: + metadata: + labels: + app: superset-worker + component: superset + spec: + initContainers: + - name: wait-for-redis + image: busybox:1.36 + command: + - sh + - -c + - | + until nc -z redis 6379; do + echo "waiting for redis..." + sleep 2 + done + containers: + - name: worker + image: jmeegan607/geoset + imagePullPolicy: Always + command: ["celery", "--app=superset.tasks.celery_app:app", + "worker", "--pool=prefork", "-O", "fair", + "-c", "4", "--loglevel=INFO"] + envFrom: + - configMapRef: + name: superset-env + env: + - name: DATABASE_PASSWORD + valueFrom: + secretKeyRef: + name: superset-secrets + key: DATABASE_PASSWORD + - name: EXAMPLES_PASSWORD + valueFrom: + secretKeyRef: + name: superset-secrets + key: EXAMPLES_PASSWORD + - name: SUPERSET_SECRET_KEY + valueFrom: + secretKeyRef: + name: superset-secrets + key: SUPERSET_SECRET_KEY + - name: MAPBOX_API_KEY + valueFrom: + secretKeyRef: + name: superset-secrets + key: MAPBOX_API_KEY + volumeMounts: + - name: config-overrides + mountPath: /app/docker/pythonpath_dev/superset_config_docker.py + subPath: superset_config_docker.py + livenessProbe: + exec: + command: + - sh + - -c + - celery --app=superset.tasks.celery_app:app inspect ping -d celery@$HOSTNAME + initialDelaySeconds: 60 + periodSeconds: 60 + timeoutSeconds: 10 + volumes: + - name: config-overrides + configMap: + name: superset-config-overrides diff --git a/wiki/Development-Guide.md b/wiki/Development-Guide.md index 9b666a189..fa0dd7ec8 100644 --- a/wiki/Development-Guide.md +++ b/wiki/Development-Guide.md @@ -79,6 +79,9 @@ GeoSet/ ├── sample-data/ # Demo data ingestion pipeline ├── docker/ # Docker configuration and init scripts ├── docker-compose.yml # Main stack (includes GeoSet demo data) +├── kustomize/ # Kubernetes deployment (Kustomize overlays) +│ ├── base/ # Dev/demo: PostGIS, metadata DB, Superset web +│ └── full/ # Production: adds Redis, Celery, Flux GitOps └── VERSIONING.md # GeoSet version policy and changelog ``` diff --git a/wiki/Getting-Started.md b/wiki/Getting-Started.md index bfcbaf86a..3844324cb 100644 --- a/wiki/Getting-Started.md +++ b/wiki/Getting-Started.md @@ -75,6 +75,8 @@ docker compose down docker compose down -v ``` +> **Kubernetes:** A Kustomize-based deployment option is also available. See [`kustomize/README.md`](../kustomize/README.md) for details. + ## Next Steps - Load the [[Sample Dashboards]] to see GeoSet in action diff --git a/wiki/Home.md b/wiki/Home.md index 8f6c04ac7..ecc10fd83 100644 --- a/wiki/Home.md +++ b/wiki/Home.md @@ -24,6 +24,7 @@ GeoSet is a geospatial data monitoring and visualization platform built on [Apac - [[Sample Dashboards]] — Loading the example Hurricane and Wildfire dashboards - [[Development Guide]] — Local dev setup, plugin architecture, contributing - [[JSON Config Spec]] — Reference for the GeoSet Map Layer JSON configuration schema +- [Kubernetes Deployment](../kustomize/README.md) — Deploying GeoSet to Kubernetes with Kustomize overlays ## Repository