From d21a688c6b8b7b1537bd7b0b45aa25d5db63772a Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 16 Nov 2026 06:02:59 +1100 Subject: [PATCH 1/2] docs(engine): correct docs against the code Fix or cut every claim in README.md and docs/ that contradicts the code, the dfe-schemas wheel or the named sibling repos. Rewrite four pages about unbuilt or outdated designs to match the code: the edge auth page, the CLI design, the commit standard and the engine-loader schema contract. Remove the hyperdx fork maintenance page because its rebase model is outdated. Point the integrations index at the fork's own runbook. Move the reference index's one link into docs/README.md. Every touched page passes hyperi-ai docs prose check with no deny. hyperi-ci lint-docs passes. --- README.md | 4 +- docs/README.md | 4 +- docs/architecture.md | 54 ++--- docs/archive/README.md | 4 +- docs/archive/helm-compiler-research.md | 4 +- docs/auth-envoy-topology.md | 223 ++++-------------- docs/control-plane/README.md | 20 +- docs/control-plane/dfe-cli-design.md | 202 ++-------------- docs/control-plane/gitops-commit-standard.md | 87 +++---- docs/control-plane/governed-ops-design.md | 149 ++++++------ docs/control-plane/idp-setup/entra.md | 4 +- docs/control-plane/idp-setup/google.md | 4 +- docs/control-plane/idp-setup/okta.md | 4 +- docs/control-plane/oidc-rbac-architecture.md | 113 +++++---- docs/control-plane/rbac-engine-reads.md | 8 +- docs/control-plane/rbac-sign-in.md | 2 +- docs/control-plane/rbac-vocabulary.md | 26 +- docs/control-plane/rbac.md | 120 +++++----- docs/control-plane/type-safety-sync.md | 28 +-- docs/control-plane/ui-api-guide.md | 99 ++++---- docs/data-evolution-acceptance.md | 14 +- docs/data-evolution.md | 34 +-- docs/data-plane/README.md | 6 +- docs/data-plane/expressions-cel.md | 56 ++--- docs/data-plane/hunt-match-delivery.md | 2 +- docs/data-plane/hunt-runner-scaling.md | 48 ++-- docs/data-plane/query-api-python.md | 23 +- docs/data-plane/query-api.md | 23 +- docs/data-plane/schema-bootstrap.md | 6 +- docs/data-plane/schema-classes.md | 32 +-- docs/data-plane/schema-sync.md | 188 ++------------- docs/data-plane/schema.md | 219 +++++++---------- docs/data-plane/sigma.md | 12 +- docs/data-plane/source-definition.md | 23 +- docs/data-plane/source-detections.md | 2 +- docs/data-plane/source-flow.md | 18 +- docs/data-plane/source-registry.md | 20 +- docs/data-plane/source-routing.md | 10 +- docs/data-plane/source.md | 30 +-- docs/data-plane/topic-contract.md | 6 +- docs/deployment/README.md | 48 ++-- docs/deployment/e2e-acceptance.md | 17 -- docs/deployment/keda-scaling.md | 6 +- docs/deployment/observability-standard.md | 8 +- docs/deployment/supply-chain-pinning.md | 16 +- docs/deployment/transports.md | 8 +- docs/integrations/README.md | 3 +- docs/integrations/dfe-ui-guide.md | 109 +++------ docs/integrations/hyperdx-fork-maintenance.md | 61 ----- docs/local-dev.md | 55 ++--- docs/reference/README.md | 5 - 51 files changed, 813 insertions(+), 1454 deletions(-) delete mode 100644 docs/integrations/hyperdx-fork-maintenance.md delete mode 100644 docs/reference/README.md diff --git a/README.md b/README.md index 6e631647d..3099e6d1a 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ Every setting is an environment variable with the `DFE_ENGINE_` prefix, apart fr | `DFE_CONFIG_DIR` | `/app/config` | `./config` with `make dev`, else `/app/config` | The deployment's config: a mounted volume or ConfigMap on Kubernetes, a checkout of the deployment's config repo for local dev. The engine finds `sources/`, `source-builds/`, `source-plans/`, `source-deploys/`, `fieldmaps/`, `services/`, `hunts/`, `rules/` and `alert-destinations/` under it. | | `DFE_ENGINE_SCHEMAS_DIR` | `/app/schemas` | unset | The schema tree root. Unset, the engine reads the trees from the installed `dfe-schemas` package. | -The image bakes the dfe-schemas trees in at `/app/schemas` and an empty `/app/config`, so neither directory variable needs setting in a container. The deployment supplies the config by mount and the secrets and connection settings by env, and a container with none of those refuses to start on the placeholder JWT secret. A per-registry variable such as `DFE_ENGINE_SOURCES_DIR` wins over the subdirectory derived from `DFE_CONFIG_DIR`. The names are exact: a bare `CONFIG_DIR` is not read. ClickHouse and Kafka connection settings follow the same prefix. +The image ships the dfe-schemas trees at `/app/schemas-seed`, copies them into `/app/schemas` on the first run of each engine version, and has an empty `/app/config`, so neither directory variable needs setting in a container. The deployment supplies the config by mount and the secrets and connection settings by env, and a container with none of those refuses to start on the placeholder JWT secret. A per-registry variable such as `DFE_ENGINE_SOURCES_DIR` wins over the subdirectory derived from `DFE_CONFIG_DIR`. The names are exact: a bare `CONFIG_DIR` is not read. ClickHouse and Kafka connection settings follow the same prefix. ## Documentation @@ -59,7 +59,7 @@ The image bakes the dfe-schemas trees in at `/app/schemas` and an empty `/app/co - [docs/control-plane/](docs/control-plane/README.md): the API, auth, RBAC and governed gitops. - [docs/data-plane/](docs/data-plane/README.md): sources, schemas, DDL, the query API and hunts. - [docs/deployment/](docs/deployment/README.md): the chart, the image and the deployment contract. -- [docs/reference/](docs/reference/README.md): the API and CLI reference. +- [docs/reference/api-route-map.md](docs/reference/api-route-map.md): where each API operation moves in the resource hierarchy. The suite's published docs are at , this repository's pages at , and [dfe-infra](https://github.com/hyperi-io/dfe-infra) deploys the whole suite. diff --git a/docs/README.md b/docs/README.md index 36368f1f0..354725f57 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,10 +1,10 @@ # dfe-engine documentation -The DFE engine's documentation: the architecture and data evolution, plus one directory each for the control plane, data plane, deployment and reference. +The DFE engine's documentation: the architecture and data evolution, plus one directory each for the control plane, data plane and deployment, and the API route map. - [DFE architecture](architecture.md) - [The DFE Control-Plane Layer](control-plane/README.md) - [Data evolution](data-evolution.md) - [Data plane](data-plane/README.md) - [Deployment](deployment/README.md) -- [Reference](reference/README.md) +- [API route map](reference/api-route-map.md) diff --git a/docs/architecture.md b/docs/architecture.md index ffd5634a8..aae765957 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -3,8 +3,7 @@ DFE is a deployable product suite: event-data ingest, ClickHouse storage, scheduled detection, and exploration, all governed through one git-backed control plane. This page is the map - where each repo sits, the two principles -that define the system, the invariants, and links into the depth docs. It -describes how DFE works today, grounded in code, not a proposal. +that define the system, the invariants, and links into the depth docs. ## Where the engine sits in the suite @@ -53,9 +52,9 @@ flowchart TB | dfe-engine | config control plane + API server | the engine itself | | dfe-infra | charts, ApplicationSets, bootstrap | consumes engine-authored overlays via the deploy repo | | dfe-ui | web UI | consumes the engine API only | -| dfe-hyperdx | the Observe view: our extended fork of the upstream HyperDX project, explore UI + telemetry sink | embedded in dfe-ui; reads ClickHouse | -| dfe-schemas | every table, view, role and bootstrap topic, plus the apply manifest | the only place they are defined; the engine applies the manifest at boot ([schema bootstrap](data-plane/schema-bootstrap.md)) | -| scalo-py / scalo-rs | shared libraries (neutral-identity OSS) | engine imports scalo-py; Rust services import scalo-rs | +| dfe-hyperdx | the Observe view: our extended fork of the upstream HyperDX project, explore UI + telemetry sink | embedded in dfe-ui, reads ClickHouse | +| dfe-schemas | every table, view, role and bootstrap topic, plus the apply manifest | the only place they are defined. The engine applies the manifest at boot ([schema bootstrap](data-plane/schema-bootstrap.md)) | +| scalo-py / scalo-rs | shared libraries (neutral-identity OSS) | engine imports scalo-py, Rust services import scalo-rs | | dfe-receiver/loader/archiver/fetcher/transform-* | Rust runtime services | read YAML config the engine writes | | dfe-deploy | example deploy/overlay repo | the GitOps hand-off surface | @@ -81,7 +80,7 @@ flowchart LR Adding a service instance is a git write of one `values/--values.yaml` file - the git-files ApplicationSet generator fans out one Argo Application per values file. Removing the file -removes the app. The engine declares instances; the appset deploys them. +removes the app. The engine declares instances. The appset deploys them. An instance removed while its source still exists leaves the files authored into it in `values/--parked.yaml`, which the glob does not match, and its next deploy restores them. Depth: [deployment/README.md](deployment/README.md) and [control-plane/governed-ops-design.md](control-plane/governed-ops-design.md). @@ -104,7 +103,7 @@ flowchart LR MAPS --> HUNTS[hunts] & QAPI[query API] & XPLR[explore UI] ``` -One definition, enforced end to end: the receiver routes an event by its +One definition, read by every stage: the receiver routes an event by its `_source`, the loader inserts into the topology-correct table for that source's schema, and the field-mapping layer projects the same columns back out as standard vocabularies (Sigma, ECS, CIM) for detection and query. There @@ -114,7 +113,7 @@ is no separate ETL config to fall out of sync. Depth: ## The control pyramid Four layers, each a narrower, more guarded window onto the one below. The API -is the single control plane; the CLI and UI only ever consume it; the API +is the single control plane. The CLI and UI only ever consume it. The API never reaches past git to the cluster. ```mermaid @@ -163,51 +162,51 @@ flowchart TB RUST --> SUB ``` -- **config repo** - the engine's working data: service configs, deployment +- The config repo is the engine's working data: service configs, deployment configs, field maps, rules/hunts, OIDC providers, accounts/groups. Engine-written AND directly human-editable. Writes are read-before-write with git blob-SHA ETags: `If-Match` mismatch returns `409 Conflict` with a diff rather than clobbering a human or CI edit. -- **deploy repo** - the GitOps hand-off. Engine writes compiled artifacts; +- The deploy repo is the GitOps hand-off. Engine writes compiled artifacts. Argo reads them. With gitops enabled it also holds the source definitions SSoT (`config/sources/`) and the hunt definitions and detection rules (`config/hunts/`, `config/rules/`), one attributed commit per mutation via - gitcrud; standalone deployments fall back to plain directories in the config + gitcrud. Standalone deployments fall back to plain directories in the config repo. The hunt runner git-syncs those last two directories, so on Kubernetes they are how a hunt reaches the pod that runs it. Provider-agnostic: GitHub / GitLab external git, or the in-cluster Forgejo fallback (see [deployment/README.md](deployment/README.md)). -- **dfe-infra** - the pinned deployment vehicle. Never written by the engine. +- dfe-infra is the pinned deployment vehicle. Never written by the engine. In small deployments the config repo and deploy repo can be one repo with different subtrees (`config/` and `values/` + `ddl/`). ## First principles (invariants) -1. **Generic product, not an internal tool.** Ships into a customer cluster - the vendor never sees. Nothing site-specific in code. -2. **YAML + git is the single source of truth.** No config database. Rust - services read config files directly; the engine reads/writes them through +1. Generic product, not an internal tool. Ships into a customer cluster + we never see. Nothing site-specific in code. +2. YAML + git is the single source of truth. No config database. Rust + services read config files directly. The engine reads/writes them through GitCrud on the deploy repo, or a cached YAML directory (DirectoryConfigStore from scalo) for registries that live off it. Identity comes from the FILENAME, never a field inside the YAML (the store loads YAML 1.1, where `off`/`yes`/`no` become booleans). -3. **Engine writes config, never installs infrastructure.** The hard +3. Engine writes config, never installs infrastructure. The hard engine<->infra boundary: dfe-infra deploys backing services and owns - ApplicationSets; the engine writes config + DDL + OIDC values into the + ApplicationSets. The engine writes config + DDL + OIDC values into the deploy repo and turns dials through overlays. If the engine is about to install ClickHouse or author an Argo Application, the work belongs in dfe-infra. -4. **Assume only a bare cluster.** Layer 0 detect-or-install brings +4. Assume only a bare cluster. Layer 0 detect-or-install brings everything else: a default StorageClass, cert-manager, External Secrets, and Argo CD, installed into `argocd` or, where one already runs, adopted. -5. **ClickHouse is the only operational store.** All non-gitops state - +5. ClickHouse is the only operational store. All non-gitops state - events, results, watermarks, leases - lives in CH via `effective_data_database`. The DocumentDB store is Observe's own, never DFE state. -6. **A commit is not a deployment.** The engine reports what it committed; +6. A commit is not a deployment. The engine reports what it committed. Argo (and any org approval gate) decides when it is live. -7. **Fully operable from the repos alone.** Engine and UI down = running +7. Fully operable from the repos alone. Engine and UI down = running system unaffected, manageable by editing the repos directly. ## Layered deployment model (summary) @@ -215,10 +214,11 @@ different subtrees (`config/` and `values/` + `ddl/`). Layer 0 platform baseline (Argo CD + detect-or-install operators) -> Layer 1 backing services (ClickHouse, optional Kafka, DocumentDB, Observe) -> Layer 2 apps + config (digest-pinned thin charts built from each app's -deployment contract + engine-authored overlays). Five tiers select defaults: +deployment contract + engine-authored overlays). Four tiers select defaults: **dfe-docker** (Compose), **slim** (gRPC, no Kafka), **single** (+ -single-broker Kafka), **scale** (everything clustered), **mesh** (scale's -replicas and ClickHouse cluster over gRPC, no broker). Observe ships in EVERY tier - it is the explore UI over all DFE +single-broker Kafka) and **scale** (everything clustered). A fifth, **mesh** +(scale's replicas and ClickHouse cluster over gRPC, no broker), is in the +charts, but dfe-infra's bootstrap refuses to select it. Observe ships in EVERY tier - it is the explore UI over all DFE data and the self-monitoring sink, not an add-on. Full detail including the per-tier composition table, deploy-repo providers, backing-service modes, and the network model: [deployment/README.md](deployment/README.md). @@ -259,7 +259,7 @@ alert grouping or cooldown. | Ingress / OIDC | Envoy Gateway (k8s) / oauth2-proxy (docker) | native OIDC at the edge | | Expression language | CEL (Python + Rust), DFE subset | one standard, tiered for performance | | Field mapping | YAML, two-tier (default + source) | human-editable, git-tracked, UI-editable | -| Stage transport | one per source, `bus` or `direct`, default from the profile | a flow is never mixed; the bus provider (Kafka) is a deployment fact, so a second provider joins without a model change | +| Stage transport | one per source, `bus` or `direct`, default from the profile | a flow is never mixed. The bus provider (Kafka) is a deployment fact, so a second provider joins without a model change | ## The docs tree @@ -268,7 +268,7 @@ alert grouping or cooldown. | Control-plane surface | [control-plane/README.md](control-plane/README.md) | governed ops, gitops commit standard, RBAC, OIDC, CLI, UI API, type-safety sync | | Data plane | [data-plane/README.md](data-plane/README.md) | sources, schemas, field mapping, CEL, query API, hunts | | Deployment | [deployment/README.md](deployment/README.md) | tiers + composition, backing services, KEDA, observability, supply-chain pinning, e2e acceptance | -| Integrations | [integrations/README.md](integrations/README.md) | dfe-ui guide, dfe-hyperdx fork maintenance | +| Integrations | [integrations/README.md](integrations/README.md) | dfe-ui guide, and where the dfe-hyperdx fork runbook lives | | Archive | [archive/README.md](archive/README.md) | superseded research + snapshots, kept for the decision trail | Edge auth topology notes live in [auth-envoy-topology.md](auth-envoy-topology.md). diff --git a/docs/archive/README.md b/docs/archive/README.md index ac65f47f6..b8f7d13e8 100644 --- a/docs/archive/README.md +++ b/docs/archive/README.md @@ -6,6 +6,6 @@ trail. Nothing here describes the current system - start at | Doc | What it was | Superseded by | |---|---|---| -| [helm-compiler-research.md](helm-compiler-research.md) | 2026-03 ecosystem survey for the Helm values compiler | decision landed: HelmValuesCompiler + vendored deep_merge | -| [monorepo-migration.md](monorepo-migration.md) | plugin entry points + a hypothetical monorepo layout | monorepo not pursued; plugin entry points live in code | +| [helm-compiler-research.md](helm-compiler-research.md) | 2026-03 ecosystem survey for the Helm values compiler | governed helm-var CRUD (`src/dfe_engine/api/v1/helm.py`) over `yaml_utils.deep_merge` | +| [monorepo-migration.md](monorepo-migration.md) | plugin entry points + a hypothetical monorepo layout | neither pursued: no monorepo, and which services exist is the app manifest (`appmgmt/apps.yaml`), not entry points | | [dfe-infra-snapshot.md](dfe-infra-snapshot.md) | 2026-03-31 snapshot of the dfe-infra repo | dfe-infra's own docs + [../deployment/README.md](../deployment/README.md) | diff --git a/docs/archive/helm-compiler-research.md b/docs/archive/helm-compiler-research.md index e67e9c926..4f7e1dba2 100644 --- a/docs/archive/helm-compiler-research.md +++ b/docs/archive/helm-compiler-research.md @@ -29,7 +29,7 @@ The closest tool is **Kapitan** (1,904 stars), but it brings substantial complex Supports Jinja2, Jsonnet, Kadet (Python), and Helm input types. Hierarchical class inheritance via reclass merges YAML fragments. - **Key patterns:** - - **Inventory = classes + targets**: Classes define defaults; targets compose classes + - **Inventory = classes + targets**: Classes define defaults. Targets compose classes and override parameters. Structurally similar to our `base config + environment overlay + service-specific override` pattern. - **`kapitan compile`**: Single command compiles entire inventory into output. @@ -168,7 +168,7 @@ The closest tool is **Kapitan** (1,904 stars), but it brings substantial complex - **URL:** - **Stars:** 2,646 | **Active:** Yes (Grafana Labs production use) -- **What:** K8s config management using Jsonnet. Environments as first-class citizens. +- **What:** K8s config management using Jsonnet. Environments are top-level objects. Go + Jsonnet, not Python. Conceptual pattern (environment overlays -> manifests) identical to ours. diff --git a/docs/auth-envoy-topology.md b/docs/auth-envoy-topology.md index e5223add2..6ee330910 100644 --- a/docs/auth-envoy-topology.md +++ b/docs/auth-envoy-topology.md @@ -1,215 +1,74 @@ -# DFE auth - the Envoy front layer (k8s vs docker) +# DFE auth at the edge (k8s vs docker) -DFE authenticates every request with **one signed JWT**, verified by **one Envoy -front layer** that behaves **identically** in Kubernetes and Docker. Only the -config delivery and the networking plumbing differ - the auth filter chain, -signature verification and path routing are the same in both. So the auth path -is authored once and tested once, and a token minted in one mode verifies in the -other. +The engine is the only issuer of the token the apps trust: an ES384 JWT, published through the engine's JWKS and carried in the `dfe_token` cookie or an `Authorization: Bearer` header. The engine and dfe-hyperdx verify that token themselves. The edge proxy routes requests, and only on Kubernetes can it also check a token, an opt-in for an external IdP's. -This doc covers the Envoy side. For the identity model (who issues the JWT, the -RBAC/group pivot, local-vs-OIDC), see the auth design notes referenced at the end. - -## Where Envoy sits - -DFE is served **single-origin**: one host, three upstreams behind it. - -- `/` -> dfe-ui (the shell) -- `/observe/*` -> hyperdx (embedded chromeless) -- `/api/*` -> dfe-engine (control plane, PDP, JWT issuer) - -Envoy is the single gateway in front of all three. Because it is one origin, the -embed iframe is same-site (no third-party-cookie problem) and the JWT rides every -request - page loads, XHR and streaming alike. - -## Request flow (identical in both modes) +## One issuer, verified by each app ```mermaid flowchart LR - B([browser]) -->|"request + JWT (cookie/Bearer)"| E[Envoy front layer] - E --> V{"jwt_authn:
signature valid?"} - V -->|no| R401[["401"]] - V -->|yes| S[strip client
X-Oidc-* / Authorization] - S --> RT{route by path} - RT -->|"/"| UI[dfe-ui] - RT -->|"/observe/*"| HX[hyperdx] - RT -->|"/api/*"| EN[dfe-engine] - UI -.re-verify JWT.-> J[(JWKS)] - HX -.re-verify JWT.-> J - EN -.re-verify JWT.-> J - E -.remote_jwks fetch+cache.-> J + B([browser]) --> P[edge proxy
Envoy] + P -->|/api/v1/*| EN[dfe-engine
issuer + verifier] + P -->|/| UI[dfe-ui] + P -->|own host or listener| HX[dfe-hyperdx] + EN -->|/.well-known/jwks.json| J[(JWKS)] + HX -.->|verifies dfe_token| J ``` -There is **one issuer: the engine**. The engine mints every token the apps see - -local logins directly, and external OIDC logins by acting as the relying party (it -validates the IdP's token, then re-mints its own engine token). So Envoy's -`jwt_authn` filter verifies a single provider - the engine JWKS - and the external -IdP's token never travels past the engine. +- Local login: `POST /api/v1/auth/login`, and the engine signs the JWT (`src/dfe_engine/api/v1/auth.py`). +- External OIDC: the engine is the relying party. `/api/v1/auth/oidc/{provider}/login` redirects to the IdP, and the callback validates the IdP's token and mints an engine token (`src/dfe_engine/api/v1/oidc_login.py`). The IdP's token stays on the engine-to-IdP leg. +- dfe-hyperdx authenticates from the `dfe_token` cookie dfe-ui shares across the domain, so the fork is its own enforcement point. +- The engine believes proxy-set `X-Oidc-*` identity headers only with `DFE_ENGINE_AUTH_TRUST_PROXY_AUTH_HEADERS=true` (default off, `settings.py` `auth.trust_proxy_auth_headers`). Turn it on only where a proxy strips those headers from the client and the engine is reachable through that proxy alone. -## What the one Envoy does +The engine signs with ES384 (ECDSA P-384 with SHA-384), `api.jwt_algorithm`. Ed25519 is not used: at 128-bit security it is below the CNSA 2.0 target. The `kid` is the RFC 7638 JWK thumbprint and the algorithm is a setting, so a post-quantum signature such as ML-DSA can replace ES384 once JOSE and Envoy can verify it (`src/dfe_engine/auth/jwt_authority.py`). -1. **Verify** - `jwt_authn` checks the JWT signature (**ES384** - ECDSA P-384 - with SHA-384, `EC` keys) against the engine JWKS, plus issuer and - expiry. `remote_jwks` fetches and caches each provider's key set. -2. **Sanitise** - strip any client-supplied `Authorization` and `X-Oidc-*` - headers before they reach an app, so identity cannot be forged by a caller. -3. **Route** - single-origin path routing to the three upstreams. -4. **Forward** - pass the verified JWT upstream; each app re-verifies it - (defence in depth) and reads the claims below. +The engine mints the key on first boot and stores it create-only, as a PKCS#8 PEM, in the scalo.secrets backend at `api.jwt_key_path`. The engine does not encrypt it, so at-rest protection is the backend's. -Envoy never mints a token. Issuance happens elsewhere (see "Issuance" below). +## What the token carries -## k8s vs docker +Every token carries `sub`. A login token adds `email` and `groups`, and the local login also carries `roles`, `org_id` and `org_ids`. The engine re-resolves roles from the group files on every request, so what a token asserts about them never decides an engine answer. -Same binary, same filter chain, same auth behaviour. The differences are where -Envoy sits and how it is configured. +One claim exists for a peer rather than for the engine: -| | Kubernetes | Docker (dfe-docker) | +| Claim | Value | Read by | |---|---|---| -| What Envoy is | the gateway (a Deployment / Gateway-API data plane) behind the cloud load balancer | one `envoy` service in the compose file, on the shared docker network | -| Config delivery | xDS / control plane (or a rendered static config) | a static `envoy.yaml` mounted into the container | -| Upstreams (clusters) | k8s `Service`s (`dfe-engine`, `hyperdx`, `dfe-ui`) | other compose services, by container DNS name | -| JWKS source (`remote_jwks`) | engine `Service` `/.well-known/jwks.json` | engine container `/.well-known/jwks.json` | -| Path routing | Envoy does it (no separate ingress needed) | Envoy does it - this replaces the ingress docker does not have | -| Scale / HA | N Envoy replicas | single container (sufficient for the docker tier) | -| Reachability guard | NetworkPolicy: only Envoy may reach the app pods | docker network isolation: apps not published to the host, only Envoy is | +| `role` | `admin` for `admin` or `infra_admin` at system scope and for the engine's own `svc:dfe-engine` token, `member` for an account whose roles allow `query:write`, `viewer` for any other | dfe-hyperdx | -The single most important consequence: **path routing is free from the ingress in -k8s, but in docker Envoy provides it**. That is why the same component is used in -both - it closes the docker gap rather than adding a second reverse proxy. +The fork reads the same value from `GET /api/v1/auth/me` (`hyperdx_role`), resolved from the account's groups at that request, and falls back to the token's claim only when that answer carries none. `admin` (or `owner`, which the engine never mints) may change what the whole team sees, `member` may save searches and dashboards, and `viewer` is refused every write (`src/dfe_engine/auth/hyperdx_role.py`). -## Issuance vs verification +## Kubernetes: Envoy Gateway -Envoy only verifies, routes, sanitises and forwards. The token is minted by the -identity source: +dfe-infra's `helm/edge/gateway` chart configures Envoy Gateway. -- **Local auth** - the browser posts to `/api/v1/auth/login` (Envoy routes it to - the engine); the engine validates the password and signs an **ES384** JWT. -- **External OIDC** - the engine is the **relying party**: - `/api/v1/auth/oidc/{provider}/login` redirects to the IdP, and the callback - validates the IdP's token then **re-mints an engine ES384 JWT**. The IdP's token - stays on the engine<->IdP leg and never reaches the apps. +| Concern | What the chart does | +|---|---| +| Routing | dfe-ui and the engine share one hostname, the engine on its `/api/v1` path families. dfe-hyperdx has its own hostname. | +| Infra routes | Argo CD and the other infra-class routes carry an OIDC `SecurityPolicy` that requires one of the admin groups. | +| Product routes | dfe-ui, the engine and dfe-hyperdx carry no edge OIDC policy. The apps check the engine token. | +| Edge JWT (opt-in) | `jwtAuthn.enabled` (default `false`) adds Envoy's `jwt_authn` for an external OIDC issuer's Bearer token. It strips client `X-Oidc-*` headers first, then maps the token's claims into them. | +| TLS | One `ClientTrafficPolicy`: TLS 1.2 minimum by default (1.3 for a high-security overlay), AES-256-GCM, P-384 ECDHE, and the hybrid ML-KEM group offered. | +| Reachability | NetworkPolicy keeps the app pods behind the gateway. | -Either way, the token Envoy (and every app) verifies is the **engine's** ES384 -token - one issuer, one JWKS. Merged mode is just both login routes enabled. +## Docker: one static Envoy config -## What the token carries - -Every token carries `sub`; a login token adds `email` and `groups`, and the local -login also carries `roles`, `org_id` and `org_ids`. The engine re-resolves roles -from the group files on every request, so what a token asserts about them never -decides an engine answer. - -One claim exists for a peer rather than for the engine: +dfe-docker's `config/proxy/envoy.yaml` runs the router filter only, with no token check: -| Claim | Value | Read by | -|---|---|---| -| `role` | `admin` for an account holding the engine's `admin` or `infra_admin` role, `member` for any other account, and `admin` on the engine's own `svc:dfe-engine` service token | dfe-hyperdx | - -dfe-hyperdx gates deleting and restoring the shipped dashboards on `role`: it -allows `admin` and `owner`, refuses any other value with 403, and falls back to -plain team membership when there is no claim at all. So the engine always writes -one - `member` is a refusal where a missing claim would be a pass. `owner` is in -the fork's accepted set and the engine never mints it, having no team-owner role -to map onto. - -## Reference config (illustrative) - -This is a reference layout of the shared filter chain. The `jwt_authn` provider -block and the route table are identical across modes; the **only** per-mode delta -is the cluster endpoints (Service DNS vs container names). - -```yaml -# jwt_authn: one provider - the engine is the single issuer -http_filters: - - name: envoy.filters.http.jwt_authn - typed_config: - providers: - engine: - issuer: "https:///api" # the engine is the only issuer - remote_jwks: - http_uri: { uri: "http://dfe-engine/.well-known/jwks.json", cluster: engine, timeout: 5s } - cache_duration: 600s - from_cookies: ["dfe_token"] # browser flow; also from Bearer - forward: true # pass the verified JWT upstream - rules: - # login + JWKS + discovery are public (they issue / expose keys) - - match: { prefix: "/api/v1/auth/login" } - - match: { prefix: "/api/v1/auth/oidc/" } - - match: { prefix: "/.well-known/" } - # everything else requires a valid engine token - - match: { prefix: "/" } - requires: { provider_name: engine } - -# route table: single-origin path routing -route_config: - virtual_hosts: - - name: dfe - domains: ["*"] - request_headers_to_remove: ["x-oidc-subject", "x-oidc-groups"] # trust boundary - routes: - - match: { prefix: "/observe/" } - route: { cluster: hyperdx } - - match: { prefix: "/api/" } - route: { cluster: engine } - - match: { prefix: "/" } - route: { cluster: ui } -``` +- One listener sends `/api/v1/` to `dfe-engine:8000` and everything else to `dfe-ui:3000`. +- A second listener serves the dfe-hyperdx embed from `dfe-hyperdx:8090`. +- The auth profile adds three oauth2-proxy gates (`compose/oauth2-proxy.yml`). -In k8s the `engine` / `hyperdx` / `ui` / `idp` clusters resolve to k8s `Service`s -and the config is delivered by the gateway control plane; in docker they resolve -to compose container names and the same block is mounted as `envoy.yaml`. - -## Cryptographic posture (CNSA 2.0) - -DFE targets CNSA 2.0 compliance where feasible - national-security deployment is -in scope, so algorithm strength is chosen for compliance, not throughput. The -user base is small and high-value, so signing and cipher CPU cost is not a -constraint. - -- **JWT signature: ES384** (ECDSA P-384 + SHA-384) - the CNSA classical / - CNSA-2.0-transitional algorithm, verifiable across the whole chain (Envoy - `jwt_authn`, PyJWT, Node `jose`). Ed25519 is deliberately not used: at 128-bit - security it is below the CNSA target. -- **Post-quantum (roadmap):** CNSA 2.0's target signature is ML-DSA-87 (FIPS - 204), but it is not yet feasible for JWTs - the JOSE binding is an IETF draft, - Envoy `jwt_authn` cannot verify it, and its ~4 KB signature stresses - cookie/header limits. The design keeps algorithm agility (kid-tagged keys, - config-driven `alg`, multi-issuer JWKS) so ML-DSA (or the smaller FN-DSA when - standardised) is a config change, not a rewrite. -- **Transport:** TLS 1.3, AES-256-GCM, SHA-384, P-384 curves, with hybrid ML-KEM - key exchange at the Envoy edge where the TLS stack supports it - the CNSA 2.0 - key-establishment step that is feasible today. -- **At rest:** the signing private key is AES-256 wrapped. - -Transport, certificate (P-384) and at-rest crypto are handled at the deployment -layer; this document covers the app-signing (ES384 JWT) side. - -## Trust boundary - -The security root is the **JWT signature** (ES384), verified at Envoy *and* at each -app - forging identity needs the signing key, so it does not depend on the -deployer's network config being perfect. The network controls are defence in -depth: - -- apps are reachable **only** via Envoy (NetworkPolicy in k8s; unpublished ports + - docker network in docker), -- Envoy strips inbound `X-Oidc-*` / `Authorization` so a caller cannot inject - identity ahead of verification. +The engine API is also published on the host's loopback, port `DFE_ENGINE_PORT` (default 8003), for local tools. ## See also -- The DFE auth design (JWT authority, RBAC group pivot, local-vs-OIDC merged) - - auth design notes / `project_dfe_authz_standard` (agent memory). -- The embedded hyperdx fork - the dfe-hyperdx repo, `docs/fork/` (README, - design, sync-cycle, leaving-upstream). It moved there with the fork itself. +- The embedded HyperDX fork: the dfe-hyperdx repo, `docs/fork/`. +- RBAC and sign-in: [control-plane/rbac-sign-in.md](control-plane/rbac-sign-in.md). diff --git a/docs/control-plane/README.md b/docs/control-plane/README.md index c616f81e8..be45c0625 100644 --- a/docs/control-plane/README.md +++ b/docs/control-plane/README.md @@ -33,9 +33,9 @@ flowchart TB |---|---| | **dfe-engine** | The config control plane + the sole authorisation PDP. Hosts the FastAPI API and the generated `dfe` CLI. Everything else in the layer keys off its API. | | **dfe-schemas** | The schema SSoT (common event header, hunt-results DDL, profiles). The engine reads it (the version-pinned `dfe-schemas` wheel) and auto-creates from it. | -| **dfe-ui** | The operator console. A window over the engine API - it holds no authority of its own; it verifies the engine-minted JWT. | -| **dfe-infra** | The GitOps SSoT and deploy vehicle (Argo CD). Deploys the suite; never a config authority. | -| **dfe-deploy** | The per-deployment read-write gitops repo the running engine commits to. We own only its **initial** state; after handover the deployment owns it. | +| **dfe-ui** | The operator console. A window over the engine API that holds no authority of its own. It carries the engine-minted JWT and reads its expiry, and the engine verifies it. | +| **dfe-infra** | The GitOps SSoT and deploy vehicle (Argo CD). Deploys the suite. Never a config authority. | +| **dfe-deploy** | The per-deployment read-write gitops repo the running engine commits to. We own only its **initial** state. After handover the deployment owns it. | ## The controlling SSoTs @@ -48,8 +48,8 @@ ONE write path - so a change is auditable and consumers can never diverge. |---|---|---|---| | **API surface** - `openapi.json` | dfe-engine `openapi/` | dfe CLI, dfe-ui types, contract mock, reference docs | the FastAPI routes via `scripts/generate_openapi.py` (never hand-edited) | | **Schema** - event header, hunt-results DDL, profiles | dfe-schemas (the `dfe-schemas` wheel, pinned in `uv.lock`) | the engine (auto-creates the CH objects at startup), the CLI, dfe-ui | dfe-schemas commits | -| **Deployment definition** - Helm charts + values | each app's thin chart, built from its deployment contract, plus dfe-infra values + the deploy-repo overlays | Argo CD, the cluster | the thin charts by each app's release, the values by dfe-infra, the overlays by the engine, one `values/--values.yaml` per app instance, each write a gitops commit. Where no chart runs (Compose), `appmgmt/appconfig.py` merges an overlay's `config:` over the deployment's base config into the app's config file | -| **Deployed state** - the gitops YAML | dfe-deploy (per deployment) | Argo CD (syncs to the cluster), the engine (reads current state) | the engine API - every mutation is a git commit (governed ops); `dfe local` break-glass when the daemon is down | +| **Deployment definition** - Helm charts + values | each app's thin chart, built from its deployment contract, plus dfe-infra values + the deploy-repo overlays | Argo CD, the cluster | the thin charts by each app's release, the values by dfe-infra, the overlays by the engine, one `values/--values.yaml` per app instance, each write a gitops commit. Where no chart runs (Compose), `src/dfe_engine/appmgmt/appconfig.py` merges an overlay's `config:` over the deployment's base config into the app's config file | +| **Deployed state** - the gitops YAML | dfe-deploy (per deployment) | Argo CD (syncs to the cluster), the engine (reads current state) | the engine API - every mutation is a git commit (governed ops), and `dfe local` break-glass when the daemon is down | | **AuthZ** - RBAC roles + scopes | dfe-engine `auth/` (YAML) | the API (`require_action`), the CLI | the engine (governed writes) | ```mermaid @@ -93,10 +93,8 @@ do not call it "the dfe-api". Its own settings read as `DFE_ENGINE_API_*`. `openapi/openapi.json` is the **single source of truth for the API surface**. It is generated from the FastAPI app (`uv run python scripts/generate_openapi.py`) -and committed. Every consumer keys off it, never off a hand-maintained copy: - -The spec is the hub; everything radiates out of it. The FastAPI routes feed it, -and every downstream consumer keys off it - never off a hand-kept copy: +and committed. The FastAPI routes feed it, and every consumer keys off it, never +off a hand-kept copy: ```mermaid flowchart TB @@ -104,8 +102,8 @@ flowchart TB routes -->|"scripts/generate_openapi.py"| spec[("openapi.json
THE API SSoT")] spec -->|"command tree at runtime
(x-scalo.cli.hidden opts out)"| cli["dfe CLI"] - spec -->|"openapi-typescript
-> @dfe/dfe-engine-types"| ui["dfe-ui"] - spec -->|"Prism"| mock["contract mock (dev + CI)"] + spec -->|"openapi-typescript
-> @repo/dfe-engine-types"| ui["dfe-ui"] + spec -->|"Prism"| mock["contract mock
docker-compose.dev.yaml"] spec -->|"Swagger UI"| refdocs["API reference docs"] classDef ssot fill:#fff3cd,stroke:#664d03,stroke-width:3px; diff --git a/docs/control-plane/dfe-cli-design.md b/docs/control-plane/dfe-cli-design.md index 53ebdc5fc..40e429118 100644 --- a/docs/control-plane/dfe-cli-design.md +++ b/docs/control-plane/dfe-cli-design.md @@ -1,197 +1,47 @@ -# `dfe` CLI - design (AWS CLI v2 model) +# The `dfe` CLI -The goal: a `dfe` command that installs and behaves like the AWS CLI v2 - a -self-contained binary, `dfe configure` profiles, a stable auth-resolution chain, -and a noun-verb command tree that is DATA-DRIVEN from the API's own contract. This -is the CLIENT half; it speaks HTTP to a deployed dfe-engine, exactly as `aws` speaks -to AWS service endpoints. +`dfe` is the engine's HTTP client. Its command tree is generated from the engine's OpenAPI document, so an endpoint becomes a command with no CLI code. It is scalo's `SpecCli` (`src/dfe_engine/cli/auto/app.py`), shaped like the AWS and gcloud CLIs, and ships as the `dfe` console script in this package (`pyproject.toml`, `[project.scripts]`). -Reference: the AWS CLI v2 source, `aws/aws-cli` on GitHub (the -bundled-installer machinery under `backends/`/`exe/`/`macpkg/`/`docker/`, and the -`aws configure` family under `awscli/customizations/configure/`). +`dfe-engine` is the server. The two are separate entry points: `dfe` talks to an engine over HTTP, and its `local` group works on the deploy repo when no engine is running. -## What exists today (do not confuse the two) +## The command tree -- **`dfe-engine`** - the SERVER entry point (`pyproject [project.scripts]` -> uvicorn). - It also carries a server-side, break-glass CLI (`dfe local governed ...`, - `cli/governed_ops.py`) that calls the SAME GitCrud/services the API routers call - - the standing "CLI is a wrapper over the API" rule. It runs WHERE the engine libs - and the gitops repo live (CI, or on the engine host when the API is down). -- **`dfe`** - the remote HTTP client this document designed. It now ships - from this repo (`cli/auto/`, scalo's `SpecCli` over the OpenAPI spec). +- One group per API tag, one command per operation: `dfe hunts list | get | create | apply | delete | run | status`. +- Path parameters are positional arguments, in path order. Body fields become flags, and `--cli-input-json`, `--cli-input-yaml` and `--generate-cli-skeleton` take or print the whole body. +- The tree is read from the installed engine package's own app factory (`load_live_spec`), so it matches the engine version installed beside it. An endpoint opts out with `x-scalo.cli.hidden`. +- Commands with no operation behind them are mounted on top: `login`, `logout`, `config`, `auth list`, `local`, `schema` and `kafka client-config`. -Keep them distinct binaries. `dfe` talks to an engine over HTTP; `dfe-engine` IS the -engine (and its in-process break-glass path). Everything below is `dfe`. +## Profiles and credentials -## 1. Distribution and install (like AWS CLI v2, NOT pip) +Profiles live under `~/.config/dfe` (`DFE_ENGINE_CONFIG_HOME` moves the root): an INI file per profile in `configurations/` holding `url` and `account`, the active profile's name in `active_config`, and `credentials.json` (mode 0600) keyed by account. Resolution, first wins: -AWS CLI v2 deliberately does not install via pip - it ships a self-contained bundle -(a frozen Python + all deps) so it never collides with a user's Python. Mirror that: +| Setting | Order | +|---|---| +| Endpoint | `--endpoint-url`, `DFE_ENGINE_API_URL`, the profile's `url` | +| Credential | `DFE_ENGINE_API_KEY`, `DFE_ENGINE_API_TOKEN`, the stored credential of the profile's account | +| Profile | `--profile`, `DFE_ENGINE_PROFILE`, `active_config` | -- **Frozen single binary** built with PyInstaller (AWS v2 uses the same approach; - `backends/build_system` + `exe/`). One `dfe` executable per OS/arch, no Python - required on the target. -- **Channels:** - - GitHub Releases: per-OS/arch archive + `install` script - (`curl -sSL https://.../install.sh | sh`), like the AWS v2 Linux zip installer. - - Homebrew tap: `brew install hyperi-io/tap/dfe`. - - Container: `ghcr.io/hyperi-io/dfe` (for CI - `docker run ... hyperi-io/dfe ...`). - - `.pkg` (macOS) / `.msi` (Windows) later, as AWS v2 does (`macpkg/`). -- **Supply chain:** artifacts SHA-pinned and signed; the install script verifies the - digest (matches the repo's pinning standard, docs/deployment/supply-chain-pinning.md). -- **Versioning:** `dfe --version` reports the CLI version AND the negotiated API - version. The CLI is installed independently of the engine but declares a - compatible API range; on mismatch it warns (older CLI vs newer engine is fine - because the command tree is regenerated - see section 4). -- **Self-update:** `dfe --version` checks the release channel; `dfe upgrade` pulls - the latest signed binary (optional, later). +`dfe login` stores an API key (`--api-key`), or exchanges `--username` and `--password` for a JWT through `POST /api/v1/auth/login`. `dfe config set | get | list | configurations` reads and writes profiles. -For local development the same code is also exposed as a `dfe` console-script in -`pyproject`, so contributors run it from the venv without the frozen build. +## Global options -## 2. Configuration and profiles (like ~/.aws/config) +- `--output json|yaml|table|text` (`DFE_ENGINE_OUTPUT`). The default is `json` off a terminal, else `yaml` for a get and `table` for a list. +- `--query ` projects the answer before it is printed. +- A list follows `next_page_token` to the end. `--no-paginate` fetches one page. +- `--verbose`, `--quiet`, `--debug`, `--yes` (skip a delete's confirmation) and `--no-input` (never prompt). -Two files under `~/.dfe/` (env override `DFE_CONFIG_DIR`), splitting config from -secrets exactly as AWS v2 splits `config` and `credentials`: +## Break-glass: `dfe local` -```text -~/.dfe/config - [default] - endpoint_url = https://dfe.example.com - output = json - org = acme # default org/tenant scope (the DFE analogue of region) +`dfe local` reads and writes a clone of the deploy repo through GitCrud with no daemon: `classes`, `ls`, `grep`, `get`, `log`, `versions`, `status`, `set`, `unset`, `rm`, `revert`, `restore` and `push`, plus `ch-cloud` for a ClickHouse Cloud service. `dfe local log` flags the commits made this way. - [profile prod] - endpoint_url = https://dfe.prod.internal - output = table +## Not built -~/.dfe/credentials # 0600, never logged - [default] - api_key = dfe_... # or nothing here if using `dfe login` (OIDC/JWT) -``` - -- **Selection:** `--profile prod` > `DFE_ENGINE_PROFILE` env > `[default]`. -- **`dfe configure`** - interactive first-run setup (prompts endpoint, auth, output). -- **`dfe configure set/get/list/list-profiles`** - the AWS v2 subcommand family - (`customizations/configure/`), same verbs. -- **`dfe configure oidc`** - set up the OIDC login flow for a profile (the analogue - of `aws configure sso`). - -## 3. Authentication resolution chain (like the AWS credential provider chain) - -The engine already has four auth paths (OIDC headers, API key, JWT Bearer, -disabled - `api/deps.py`). The CLI resolves a bearer credential in this fixed order, -first match wins: - -1. Explicit flag: `--token ` or `--api-key `. -2. Env: `DFE_ENGINE_API_TOKEN` / `DFE_ENGINE_API_KEY`. -3. A cached OIDC/JWT session from `dfe login` (`~/.dfe/cache/.json`, - auto-refreshed while valid). -4. Profile credential in `~/.dfe/credentials` (`api_key`). -5. In-cluster: a mounted service-account/JWT token (for CI/Jobs), if present. - -- **`dfe login`** - the browser OIDC flow (device-code or loopback redirect), the - analogue of `aws sso login`; or `dfe login --username u` for the local-auth - JWT path (`POST /api/v1/auth/login`). The resulting JWT is cached and refreshed. -- **`dfe logout`** - clears the cached session. -- 401 from the API -> the CLI prints "run `dfe login`" (or "check `--api-key`"), - never a raw stack trace. - -## 4. The command tree is DATA-DRIVEN from OpenAPI (the core alignment) - -AWS CLI v2's entire command tree is generated from botocore's JSON service models - -that is what makes it consistent and always in step with the services. DFE already -ships the equivalent: `openapi/openapi.json` (74 operations, 23 routers). The -`dfe` command tree is GENERATED from it: - -- **router/tag -> noun group**, **operation -> verb**. So: - - `dfe hunts list | get | create | update | delete | run` - - `dfe rules list | create | validate | publish | rollback | versions` - - `dfe helmvars get | set | list` (Tier-1 dials) - - `dfe deployments get | publish | history` - - `dfe queries run | views list|get|execute` - - `dfe accounts | groups | roles | actions | policies ...` (governance) - - `dfe sources | schemas | fieldmaps | sigma | discovery ...` - - `dfe system health | version` -- **Path/query params -> options** (`--name`, `--page`, `--search`); **request-body - schema -> flags**, plus `--cli-input-json` / `--cli-input-yaml` and - `--generate-cli-skeleton` for the whole body (AWS v2 verbs, invaluable for CRUD - of a large YAML resource). -- **Responses -> the output formatter** (section 5). -- Regenerated whenever the API changes, so the CLI cannot drift from the contract. - An older CLI against a newer engine still works for the operations it knows; new - operations appear when the CLI is regenerated/updated. - -Implementation: scalo's `SpecCli` reads `openapi.json` and builds the Typer tree -(`cli/auto/app.py`). The HTTP layer MUST be scalo's -`HttpClient`/`AsyncHttpClient` (library policy - never raw httpx). Auth, retries, -pagination, and output live in the shared client core; the generated tree is thin. - -## 5. Global options (mirror the AWS CLI) - -Available on every command: - -- `--profile`, `--endpoint-url`, `--region`-analogue `--org`. -- `--output {json,yaml,table,text}` - `json` default (scripts), `table` for humans. -- `--query ` - client-side projection (reuse the `jmespath` lib AWS uses). -- `--no-paginate` / `--page-size N` / `--max-items N`. -- `--debug` (wire log), `--no-verify-ssl`, `--cli-read-timeout`, `--cli-connect-timeout`. -- `--cli-input-json|yaml`, `--generate-cli-skeleton`. - -## 6. Pagination - -The API's list endpoints return scalo's `Page[T]` with an opaque `next_page_token`. -The CLI auto-paginates by following `next_page_token` until exhausted (AWS v2 default), -assembling one result set; `--no-paginate` returns a single page, `--max-items` -caps, `--page-size` sets the server page size. - -## 7. Governed-Ops semantics (DFE-specific, beyond AWS) - -Every mutation in DFE is a git commit over the gitops repo, so the CLI exposes what -AWS has no analogue for: - -- `--message ""` on writes -> the commit message (falls back to a sensible - default); the CLI prints the resulting revision SHA. -- `--dry-run` -> validate + show the diff without committing (maps to the API's - validate path). -- Optimistic concurrency: `--if-match ` sends the base revision; on a 409 - the CLI renders the 3-way view (current-at-head vs yours vs theirs) the - `ConcurrencyConflictError` already carries, instead of a bare error. -- Versioned classes (rules/hunts, via VersionedDoc): the lifecycle verbs - `publish` / `rollback` / `versions` / `diff` map to draft/publish/rollback. A dial - class (helmvars) has no such verbs (it is unversioned) - the generator omits them, - matching the `versioned` opt-in. - -## 8. Ergonomics (AWS CLI v2 features to match) - -- **`--cli-auto-prompt`** - interactive prompting for required args (AWS v2 flagship - feature); `dfe` with no subcommand can drop into it. -- **Shell completion** - `dfe completion bash|zsh|fish` emits a completer. -- **Help system** - `dfe help`, `dfe help`, `dfe help`, - generated from the OpenAPI summaries/descriptions. -- **Non-zero exit codes** by error class (auth, not-found, conflict, validation, - server) so scripts can branch. - -## 9. Build/rollout sketch (phased) - -1. **Client core** - config/profile loader, the auth chain + `dfe login`, the scalo - HTTP client, output formatters (json/yaml/table/text + JMESPath), pagination. -2. **Generator** - `openapi.json` -> click tree; wire a first vertical slice - (`dfe hunts`, `dfe helmvars`, `dfe deployments`) end to end. -3. **Governed-ops verbs** - `--message`/`--dry-run`/`--if-match`, the versioned - lifecycle verbs, the 409 3-way renderer. -4. **Packaging** - PyInstaller frozen build in CI (per OS/arch), `install.sh`, - Homebrew tap, container image; sign + SHA-pin the artifacts. -5. **Ergonomics** - auto-prompt, completion, `dfe configure` family, self-update. - -Each phase ships a usable CLI; the generator (phase 2) is the keystone that keeps -`dfe` in lockstep with the engine's contract the way botocore models keep `aws` in -lockstep with AWS. +A frozen single binary, a Homebrew tap, a container image and a browser OIDC login are not built. diff --git a/docs/control-plane/gitops-commit-standard.md b/docs/control-plane/gitops-commit-standard.md index 8eeae832e..7def3cfc2 100644 --- a/docs/control-plane/gitops-commit-standard.md +++ b/docs/control-plane/gitops-commit-standard.md @@ -1,9 +1,9 @@ # DFE GitOps Commit Standard -Status: PROPOSED (feat/governed-ops-api). Strictly adhered to and ENFORCED IN CODE -by the governed-ops engine (not just prose). Aligned to OpenGitOps + current Argo -CD best practice, amended for the DFE gitops-survivability model. +The governed-ops engine enforces this standard in code (`src/dfe_engine/gitcrud/commit_policy.py`). +It follows the OpenGitOps principles and Argo CD's best practice, amended for DFE's +rule that the repo alone can drive a deployment. ## Scope @@ -11,35 +11,29 @@ This standard governs **machine commits made by dfe-engine** (the governed-ops A and gitops bridge) to the **gitops deploy/config repos**. It does NOT govern human source-code commits to dfe-* repos - those follow the hyperi-ai CI standard (Conventional Commits, `fix:`/`feat:` + semantic-release). Deploy-repo commits are -OPERATIONAL state changes; they never trigger semantic-release. - -Web-researched (2026-06-30), see References. +OPERATIONAL state changes. They never trigger semantic-release. ## 1. Foundational principles (non-negotiable) The OpenGitOps four, plus our survivability rule: -1. **Declarative** - desired state is YAML in git, never imperative cluster calls. -2. **Versioned + immutable** - git history is the SoT; pin every image/chart by +1. Declarative - desired state is YAML in git, never imperative cluster calls. +2. Versioned + immutable - git history is the SoT. Pin every image/chart by tag or SHA. NO `latest`, NO floating refs (immutability principle). -3. **Pulled automatically** - Argo pulls; the engine never `kubectl apply`s. -4. **Continuously reconciled** - auto-sync + self-heal on (drift is reverted). -5. **Survivable** - kill engine + ui and the repo still fully drives + manages DFE - ([[gitops-survivability]]). Every engine mutation is a git commit, full stop. +3. Pulled automatically - Argo pulls. The engine never `kubectl apply`s. +4. Continuously reconciled - auto-sync + self-heal on (drift is reverted). +5. Survivable - kill engine + ui and the repo still fully drives + manages DFE. + Every engine mutation is a git commit, full stop. ## 2. Repository + environment layout -- **Config repo is separate from source repos** (already true) - access separation, - no CI->commit->CI loop. -- **Directory/overlay-per-environment, NOT branch-per-environment.** Permanent env - branches cause merge hell, drift, and lose the single-SoT view. One config repo, - a path per env/cluster (e.g. `clusters//...`, `envs//values/...`), - a shared `base/` + overlays. Argo's ApplicationSet selects the PATH per - env/cluster (not a branch). [AMENDMENT to today's `config_repo_revision`-branch - wiring -> infra alignment task in dfe-infra.] -- **Promotion is a git operation** (PR that bumps the next env's overlay), never an - edit to a shared file. -- The tracked branch is `main`. No env branches. +- The deploy repo is separate from the source repos, so access is separate and + there is no CI->commit->CI loop. +- One deploy repo per deployment. The engine writes `values/` (app-instance + overlays), `infra/` (cluster and platform overlays), `config/` (sources, schemas, + hunts, rules, the library) and `governance/` (`src/dfe_engine/gitcrud/registry.py`). +- The tracked branch is `main`. No environment branches: a second environment is a + second deployment with its own repo. ## 3. Commit conventions (Conventional Commits format, gitops-typed) @@ -50,7 +44,7 @@ Subject: `type(scope): summary` - ASCII only, <= 50 chars, imperative. rollback), `schema`, `seed`, `lifecycle` (a service set running, paused or stopped). - **scope** = the resource class + instance, e.g. `cfg(receiver-default)`, `hunt(bruteforce)`, `rbac(group:soc-ro)`, `action(scale-receiver)`. -- Body wrap 72; explains WHY when non-obvious. +- Body wrap 72. Explains WHY when non-obvious. - `[skip ci]` on every deploy-repo commit (deploy repo has no semantic-release). - **Atomic:** one logical change = one commit. A defined-action that touches N vars is ONE commit. @@ -70,8 +64,8 @@ DFE-Base-Revision: 9c1d... # optimistic-concurrency base SHA DFE-Audit-Id: aud_... # SOC2 audit record id ``` -- **author** = the acting human/principal (blame/attribution). -- **committer** = the `dfe-engine` bot identity (machine provenance). [already split] +- Author and committer are both the engine's configured git identity + (`src/dfe_engine/gitops/repo.py` `_commit`). The acting principal is the `DFE-Actor` trailer. ## 4. Branch + PR policy (direct vs PR is POLICY-DRIVEN) @@ -83,9 +77,9 @@ We honour it via the RBAC + protected-var policy, not by blocking the engine: - **PR mode** (prod env / protected vars / `governance` (RBAC) class / any class flagged `require_pr`): engine creates a short-lived branch `dfe///`, commits, and OPENS A PR via the git - PROVIDER API (Forgejo/GitHub/GitLab - provider-agnostic; REST via + PROVIDER API (Forgejo/GitHub/GitLab - provider-agnostic. REST via AsyncHttpClient, NOT git CLI). A required approval (human or policy bot) merges - to `main`; protected-branch rules + approvals are configured at the GIT level. + to `main`. Protected-branch rules + approvals are configured at the GIT level. - Mode is resolved per change from: environment x class x protected-var-policy x a `require_pr` flag on the defined action. Recorded in the audit trail. - Sources, derived schemas, hunts, rules and app overlays all take this path. With @@ -95,19 +89,15 @@ We honour it via the RBAC + protected-var policy, not by blocking the engine: ## 5. Signing + verification (supply-chain) -- **Sign every machine commit** with the `dfe-engine` bot key (provenance + a second - factor beyond branch protection). -- **Argo verifies via `sourceIntegrity`** (the current mechanism) - NOT the - DEPRECATED `signatureKeys`/GnuPG project field (slated for removal). Tiered: - enforce verification on prod, relax on dev. -- Prefer dulwich-native signing to keep the "no git CLI" rule; if dulwich cannot - sign, signing is the ONE allowed shell-out (to `gpg`/`ssh-keygen`, never `git`). - [implementation decision - verify dulwich signing support at build time.] +The engine does not sign its commits, and neither the engine nor dfe-infra +configures Argo CD's `sourceIntegrity` verification. Provenance rests on the +`DFE-*` trailers, the audit log and the forge's branch protection. ## 6. Concurrency, rollback, no rewriting - **Optimistic concurrency** via the commit SHA as the version token (ETag / - `If-Match`). Stale base -> 409 + 3-way diff. [[governed-ops-api]] Task 1.13. + `If-Match`). Stale base -> 409 carrying the current doc and revision, so the + caller can build a 3-way diff (`src/dfe_engine/gitcrud/engine.py` `ConcurrencyConflictError`). - **Roll-forward only.** Rollback = `git revert` (or pin the prior tag), NEVER Argo rollback (breaks under auto-sync) and NEVER `--force` / history rewrite on `main` (Argo would thrash). Reverts are normal commits under this standard. @@ -116,32 +106,29 @@ We honour it via the RBAC + protected-var policy, not by blocking the engine: ## 7. Self-heal safety (avoid reconciliation loops) -- Do NOT write fields owned by in-cluster controllers under self-heal. KEDA owns `replicas` -> "scale" sets `keda.minReplicaCount`/`keda.maxReplicaCount`, NOT `replicaCount`; declare `ignoreDifferences` for any genuinely-dynamic field. Build the exclusion list BEFORE enabling self-heal, not after. +- Do NOT write fields owned by in-cluster controllers under self-heal. KEDA owns `replicas` -> "scale" sets `keda.minReplicaCount`/`keda.maxReplicaCount`, NOT `replicaCount`. Declare `ignoreDifferences` for any dynamic field. Build the exclusion list BEFORE enabling self-heal, not after. - The exception is a document that explicitly sets `keda.enabled: false`. Nothing owns the count there, and `replicaCount` is the only way to set it. An unset flag is the chart default, which the engine reads off the app manifest (`appmgmt.scaling.keda_by_default`): an overlay in `values/` of a `scale_deployed` app, or one the catalogue cannot place, is read as KEDA-driven, while a platform chart in `infra/` is not unless the document enables KEDA. -- `validate_write` and `validate_revert` (`gitcrud/commit_policy.py`) check the document a write leaves behind and refuse only the violations the write adds, so a document already breaking a rule still takes the write that repairs it. -- auto-sync + self-heal + prune are ON (prune added gradually). Webhooks configured - so commits reconcile in seconds, not on the 3-min poll (tightens the - committed-vs-applied window the API reports). +- `validate_write` and `validate_revert` (`src/dfe_engine/gitcrud/commit_policy.py`) check the document a write leaves behind and refuse only the violations the write adds, so a document already breaking a rule still takes the write that repairs it. +- dfe-infra's ApplicationSets run auto-sync with self-heal on. Prune is on for the + layer-2 app, data, edge and platform sets and off for the add-ons and scale sets + (`argocd/appsets/*.yaml`). ## 8. Enforcement in code (this is a code standard, not just prose) -Implemented in the gitcrud engine (see [[governed-ops-api]] Phase 1): +Implemented in the gitcrud engine: - `CommitPolicy` builds every message + trailers from the request context (actor, role, class, action, request-id, base-SHA, audit-id) - callers cannot hand-write messages. -- Mode resolver (direct vs PR) from env x class x policy; PR via the provider API. +- Mode resolver (direct vs PR) from env x class x policy. PR via the provider API. - Pre-commit validators: ASCII-only, subject <= 50, type in the allowed set, image/ chart refs pinned (reject `latest`/floating), no controller-owned fields. -- Signing + `sourceIntegrity` config emitted for Argo. -- A CI lint (`dfe local gitops lint`) asserts any deploy-repo history conforms, so - hand commits (survivability) are checked too. ## References - OpenGitOps Principles (CNCF): -- Argo CD Best Practices (env folders not branches; config/source repo split): +- Argo CD Best Practices (env folders not branches, config and source repos split): - Argo CD Auto-Sync (self-heal, prune, roll-forward): -- Argo CD anti-patterns (PRs over direct commits; pin bases): +- Argo CD anti-patterns (PRs over direct commits, pin bases): - Argo CD signature verification (GnuPG deprecated -> sourceIntegrity): -- hyperi-ai CI standard (the Conventional Commits format we align to): internal CORE.md / GIT.md +- Conventional Commits: diff --git a/docs/control-plane/governed-ops-design.md b/docs/control-plane/governed-ops-design.md index 34c51e439..437cf1d33 100644 --- a/docs/control-plane/governed-ops-design.md +++ b/docs/control-plane/governed-ops-design.md @@ -21,7 +21,7 @@ whole point, and it is the acceptance test for every piece below. ```mermaid graph TD - CLI["CLI - dfe local governed"] + CLI["CLI - dfe local"] HelmRouter["Tier-1 router - api/v1/helm"] GovRouter["Tier-2 router - api/v1/governance"] ConfigRouter["Config router - api/v1/config"] @@ -32,7 +32,6 @@ graph TD Argo["Argo CD reconciles the commit"] CLI --> GitCrud - CLI --> Actions HelmRouter --> GitCrud GovRouter --> Actions Actions --> GitCrud @@ -58,13 +57,15 @@ surface small and the policy readable. `src/dfe_engine/gitcrud/`. One engine handles every resource class the same way, because every resource is git-backed config. A `ResourceClass` says where a class lives (a directory), how one resource is laid out, and which RBAC prefix governs -it. The default registry covers the deploy repo: `helmvars` (the app-instance -overlays under `values/`), `infravars` (the substrate and platform overlays under -`infra/`), `sources` (the all-in-one source-definition docs under -`config/sources/` - the first datamodel class in the deploy repo), `library` (the -versioned artefact library under `config/library/`) and the `governance` class -(`accounts`, `groups`, `roles`, `actions`, `policies` under `governance/`, plus -the CH RBAC types). +it. The default registry (`src/dfe_engine/gitcrud/registry.py`) covers the deploy repo: +`helmvars` (the app-instance overlays under `values/`), `infravars` (the cluster +and platform overlays under `infra/`), the datamodel classes `sources`, +`compat_profiles` and `derived_schemas` (under `config/sources/`, +`config/compat-profiles/` and `config/schemas/derived/`), `hunts` and `rules` +(`config/hunts/`, `config/rules/`), `library` (the versioned artefact library +under `config/library/`) and the governance classes (`accounts`, `groups`, +`roles`, `actions`, `policies`, `ch_tiers`, `ch_service_roles` and +`gov_settings` under `governance/`). `infravars` shares the `helmvars` RBAC prefix but needs its own directory: Argo hands the app generator's `values/*-values.yaml` glob to git as a pathspec, where @@ -100,6 +101,8 @@ Nothing else moves to `BUNDLE` by default: the other versioned classes hold smal structured payloads that already diff well, so migrating them would cost a data migration in every deployed gitops repo for no gain. +## One commit per mutation + The engine is small on purpose. Read, flatten to dot-paths, set or delete a path, write a whole doc, delete a resource -- every mutation ends in one commit via `GitopsRepo` (which uses dulwich, so there is no shell-out to `git`). Three extras @@ -117,7 +120,7 @@ code: it builds every message and its audit trailers (callers cannot hand-write them), rejects `latest`/unpinned image refs and controller-owned fields like `replicaCount`, and resolves direct-commit vs PR from the environment and class. -## Writing a helm var, end to end +## Writing a helm var ```mermaid sequenceDiagram @@ -169,10 +172,14 @@ sequenceDiagram Crud-->>Op: diff + commit SHA ``` +## Protected vars and refused writes + A `ProtectedPolicy` is the one thing that reaches below a class -- a list of locked `cls:name:path` globs that even Tier-1 must respect unless the caller holds the override grant. It is a policy object, not a per-var ACL. A set or revert at a parent, or a set below a locked scalar, is refused like a write to the var itself, because either one changes it. A whole-document write is gated only on the leaves it adds, changes or removes. Three policies ship and seed into every deploy repo. `baseline` locks image references. `storage-layout` locks the data-layer modes, the storage model, the `objectStore` / `tieredBlock` / `tieredObject` dial blocks, and the disk size and class. `sizing-locks` locks the Kafka sizing inputs, the KRaft controller layout and the cloud. Seeding never rewrites an existing file, so a repo carrying the earlier `storage-model` policy keeps it and gains `storage-layout` beside it -- hence the pre-vocabulary key spellings in the shipped lock. +## Refused writes no grant lifts + Not everything unsafe is a protected var. A lower node or broker count is refused by the backing-services router itself, and by an action that leaves one in the infra overlay, with a 400 that names the reason, and the override grant does NOT get past it: the hazard is data loss rather than @@ -183,7 +190,7 @@ the count, writing one renders nothing. Ahead of the count guard and the policy locks, the router, and an action writing the infra overlay, refuse a write or revert of a `read_only` value on a backing service DFE does not run (`managed_by` `provider` or `external`, per `src/dfe_engine/backing/providers.yaml`) with 409 `managed_by_provider`. The override grant does not lift it: the chart deploys no store there, so the commit would change nothing. -### Constrained params -- a dial with detents, or a bounded knob +## Constrained params -- a dial with detents, or a bounded knob An action may declare `params`, and every param carries a CLOSED constraint: an enum carries its full value list, a numeric carries both bounds. There is @@ -193,7 +200,7 @@ the hole curation closed. A `VarChange.value` references a param as enum value into a per-var literal -- the admin still curates what each detent means. Substitution never reaches `cls`/`name`/`path`, and there is no string interpolation, so params add zero injection surface. Wiring errors (dangling -reference, map not covering the enum) are rejected at DEFINE time; only the +reference, map not covering the enum) are rejected at DEFINE time. Only the caller's supplied values can fail an invoke (422 `invalid_params`). A `value` of `{"$unset": true}`, alone or as a map entry, removes the path instead of setting it, which returns the var to the layer beneath the overlay (a chart default or a tier profile). Removing an absent path is a no-op that creates no file, an emptied parent map is removed with it, and the diff shows `new` as `{"$unset": true}`. A marker with any other content is rejected at define time. @@ -202,7 +209,7 @@ A `value` of `{"$unset": true}`, alone or as a map entry, removes the path inste the convention the shipped roles grant on (`dfe_operator` and `infra_admin` carry `action:invoke:*`). -### The contract exposes the closed sets +## The contract exposes the closed sets Every stringly field on an action is a closed set, so the contract enumerates it rather than trusting hand-typed values: `GET /gitops/classes` (the registry, @@ -214,61 +221,57 @@ fields carry `x-dfe-enum-source` annotations naming these endpoints, so generated clients render selects, not text boxes (the full convention: [ui-api-guide.md](ui-api-guide.md)). -### The shipped action library +## The shipped action library The deploy-repo template (dfe-deploy `governance/`) ships a standard library of dials -- `receiver-surge`/`receiver-normal` (KEDA ceiling detents), `hunts-pause`/`hunts-resume`, `hunts-throttle` (bounded concurrent-run cap) -- plus a `baseline` protected-var policy locking `image.*` (images move through stack pins, not dials). Every shipped action's structure is CI-validated in -that repo (`tools/validate_governance.py`); the chart vars they point at are +that repo (`tools/validate_governance.py`). The chart vars they point at are owned by dfe-infra's chart validation. An action's `description` is its runbook line -- what it does, and which action reverts it. -## ClickHouse data RBAC -- one user per group +## ClickHouse data RBAC -- a tier and a tenant per identity -`governance/ch_rbac.py`. Access control in dfe-hyperdx's app layer cannot contain a bad -query. Quotas and resource limits can, and those live on the ClickHouse user. So a -group maps to exactly one ClickHouse user carrying three controls: +`src/dfe_engine/governance/ch/`. Access control in dfe-hyperdx's app layer cannot +contain a bad query. Quotas and resource limits can, and those live in ClickHouse. +So every ClickHouse identity composes two axes: ```mermaid graph LR - Group["RBAC group - e.g. soc-ro"] - User["ClickHouse user - dfe_grp_soc-ro"] - Grants["GRANTs + row policies - what data"] - Profile["Settings profile - per-query limits"] - Quota["Quota - rate and volume"] + Group["RBAC group"] + User["ClickHouse user
bound to the group"] + Tier["Tier role - grants,
settings profile, quota"] + Tenant["Tenant role - one row policy
per _org_id table"] + Setting["SQL_current_tenant_id
pinned read-only"] Group --> User - User --> Grants - User --> Profile - User --> Quota + User --> Tier + User --> Tenant + Tenant --> Setting ``` -`build_group_sql` renders the `CREATE USER` + `GRANT` + `CREATE SETTINGS PROFILE` + -`CREATE QUOTA` (identifiers backtick-quoted, so a hyphenated group name is a valid -CH name). The primary path is `ddl_artifact`, which emits that as a gitops DDL file -under `ddl/ch-rbac/.sql` -- applied by the same migration runner as the schema -DDL, with the read-only core tiers supplied from dfe-schemas. The imperative -`GroupChProvisioner` is the secondary, direct path. dfe-hyperdx then selects the -connection bound to the user's group, so every query runs as that user and CH itself -is the enforcing boundary. - -This applies to **hunts too**, and it is the same mechanism. A hunt worker does not -connect to ClickHouse as an unbounded account -- it runs its query as a CH user -carrying a quota and a settings profile. The primary job there is a cost guard: a -dumb or runaway hunt query (a full-table scan, an unbounded join) is killed by the -quota or the `max_memory_usage`/`max_execution_time`/`max_rows_to_read` limits -rather than taking the cluster down. The grants are a security scope on top if a -hunt should only ever see certain data. So the per-group CH user has two consumers -- -interactive queries through Observe, and hunt execution -- and CH is the safety -boundary for both. +A group binding ties an RBAC group's ClickHouse user to one tier and at most one +org. A tier is grants, a settings profile and a quota (`dfe__role`, +`_profile`, `_quota`). The tenant axis is one shared role plus a restrictive row +policy on each `_org_id` table, reading the caller's `SQL_current_tenant_id`, which +an org-tied user carries pinned read-only, so a `SETTINGS` override in a query is +refused. The catalogue of tiers and service roles is data in dfe-schemas +(`roles/clickhouse.yaml`), editable through the `ch_tiers` and `ch_service_roles` +classes, and `ChRbacReconciler` renders whatever exists into idempotent DDL and +runs it. + +The hunt runner connects as the `dfe_hunt_runner` service role, which carries +grants and no quota (`roles/clickhouse.yaml`, `src/dfe_engine/hunts/connect.py`). The catalogue's +`hunt_tier_*` tiers carry per-query memory and time limits and an hourly quota for a +group bound to one. ## The hunt runner `src/dfe_engine/hunts/`. A hunt is a rule query on a schedule that writes -matched rows to a table. One pod proved it cannot service them all, so the runner is -a pool -- but hunts are never *assigned* to pods. Workers pull due runs from a +matched rows to a table. The runner is a pool, and hunts are never *assigned* to +pods. Workers pull due runs from a ClickHouse lease table. Add a pod and it just starts pulling. Kill one and its leases expire and get reclaimed. There is no shard map to rebalance and nothing to go split-brain over. @@ -286,7 +289,7 @@ graph TD W1["Worker"] W2["Worker"] CH["ClickHouse - run query, write results"] - WM["Watermark on timestamp_load"] + WM["Watermark on _timestamp_load"] W1 -->|claim| Claim W2 -->|claim| Claim @@ -302,11 +305,14 @@ schedule fan out evenly and never stampede ClickHouse at the boundary. `latest_f names the most recent fire at or before now, and it stays owed until its watermark is written. A global cap hard-limits how much hits CH at once. -A run never doubles up. To claim, a worker INSERTs a lease row for `(hunt, fire)` -then reads the rows back for that key and settles on one deterministic winner (claim -time, then owner id) after a short window -- so a single worker is exactly-once, and -the rare multi-worker race is absorbed because the windowed INSERT is idempotent (the -same window written twice is the same rows). Per-hunt state means an overrun defers +## Claims, windows and coordination tables + +A run never doubles up. To claim, a worker reads the hunt free, INSERTs a lease row +and a marker of its own, waits a settle, and keeps the claim only if its row landed +within that settle and is still the latest. Lease writes reach a majority of +replicas and lease reads go through `FINAL`, so every worker names the same winner +(`src/dfe_engine/hunts/ch_coordinator.py`). Exclusion has to hold, because a window run twice +writes its rows twice. Per-hunt state means an overrun defers rather than starting a second copy -- and flags `too_aggressive` so the UI can tell the user their schedule is too tight. @@ -321,13 +327,16 @@ stateDiagram-v2 deferred --> due: previous run finished ``` -Each run is incremental and crash-safe. The query carries a `{window}` placeholder; -the worker substitutes `timestamp_load >= start AND < end`, runs it, and advances the -watermark ONLY after the query commits. A pod killed mid-run re-runs the same window -from the last committed watermark -- no gap, no loss. The watermark field is fixed to -`timestamp_load`, the common-header column that is always present. +Each run is incremental and crash-safe. The query carries a `{window}` placeholder. +The worker substitutes the window predicate on the hunt's timestamp field, runs it, +and advances the watermark ONLY after the query commits. A pod killed mid-run re-runs +the same window from the last committed watermark -- no gap, no loss. The field +defaults to `_timestamp_load`, the common-header column that is always present, and a +hunt's `timestamp_field` overrides it (`src/dfe_engine/hunts/spec_loader.py`). -The three coordination tables (ReplacingMergeTree, in the effective data database): +The three coordination tables, all ReplacingMergeTree in the effective data database. +`hunt_run` (one row per fire) and `hunt_runner_heartbeat` (one row per runner) sit +beside them: ```mermaid erDiagram @@ -353,21 +362,19 @@ erDiagram `HuntRunner.tick(now)` is one cycle -- find due hunts that are not already active, then claim up to the cap and execute each at its scheduled window. The daemon is -`while: tick(); sleep`. Validated end to end against real ClickHouse: concurrent -disjoint claims, lease reclaim, and incremental resume with no duplicate rows. +`while: tick(); sleep`. ## The smaller pieces -- **Runtime config** -- `GET /api/v1/config/client` (authenticated, any role) hands the UI - its API base, Observe URL, auth mode, and feature flags at runtime. This is what - lets one UI image run in every environment instead of baking the URLs in at build - time. -- **Alerting** -- a scaffold only, deferred by design. A rule model (threshold over a - window, severity to a channel) and a pluggable dispatcher. The real senders and the - library choice wait until we reach that phase. -- **CLI** -- `dfe local governed helm ...` and `action invoke` wrap the same services - the API uses, for CI and break-glass. A CLI is a thin window over the API, not a - second way of doing things. +- `GET /api/v1/config/client` (authenticated, any role) hands the UI its API base, + Observe URL, auth mode and feature flags at runtime, so one UI image runs in every + environment. +- `src/dfe_engine/alerting/` is a scaffold: a rule model (threshold over a window, + severity to a channel) and a dispatcher whose only sender logs. A hunt's own alert + destinations are delivered by `src/dfe_engine/hunts/delivery.py`. +- `dfe local` (`ls`, `get`, `set`, `unset`, `rm`, `revert`, `restore`, `push`) wraps + GitCrud on a clone of the deploy repo, for break-glass when the engine is down. It + has no action invoke ([dfe-cli-design.md](dfe-cli-design.md)). ## What holds it together diff --git a/docs/control-plane/idp-setup/entra.md b/docs/control-plane/idp-setup/entra.md index 0abacd201..2ff200cd1 100644 --- a/docs/control-plane/idp-setup/entra.md +++ b/docs/control-plane/idp-setup/entra.md @@ -2,8 +2,6 @@ DFE asks for two things from your Microsoft Entra ID tenant: your users sign in to DFE with their work account, and DFE reads which security groups each user is in, so your groups decide their DFE roles. You set this up once as an app registration. Nothing is needed per user. -Console paths checked 2026-10-10 against Microsoft's documentation. - The DFE operator who sent you this page gives you the redirect URI. It looks like this, and you paste it exactly as given: ```text @@ -111,7 +109,7 @@ source_id: "" Behind a TLS-terminating proxy, set `api.forwarded_allow_ips` (`DFE_ENGINE_API_FORWARDED_ALLOW_IPS`), or the engine builds the redirect URI with `http://` and Entra refuses it. -## Vendor references +## Microsoft references - Register an app: - Add a redirect URI: diff --git a/docs/control-plane/idp-setup/google.md b/docs/control-plane/idp-setup/google.md index e067fa4bf..22357d33f 100644 --- a/docs/control-plane/idp-setup/google.md +++ b/docs/control-plane/idp-setup/google.md @@ -2,8 +2,6 @@ DFE asks for two things from your Google Workspace: your users sign in to DFE with their Google account, and DFE reads which Google groups each user is in, so your groups decide their DFE roles. You set this up once in Google Cloud. Nothing is needed per user. -Console paths checked 2026-10-10 against Google's documentation. - The DFE operator who sent you this page gives you the redirect URI. It looks like this, and you paste it exactly as given: ```text @@ -114,7 +112,7 @@ source_id: "" Behind a TLS-terminating proxy, set `api.forwarded_allow_ips` (`DFE_ENGINE_API_FORWARDED_ALLOW_IPS`), or the engine builds the redirect URI with `http://` and Google refuses it. -## Vendor references +## Google references - Cloud Identity API setup: - OAuth consent screen: diff --git a/docs/control-plane/idp-setup/okta.md b/docs/control-plane/idp-setup/okta.md index b82184353..d10232167 100644 --- a/docs/control-plane/idp-setup/okta.md +++ b/docs/control-plane/idp-setup/okta.md @@ -2,8 +2,6 @@ DFE asks for two things from your Okta org: your users sign in to DFE with their Okta account, and DFE reads which Okta groups each user is in, so your groups decide their DFE roles. You set this up once as an app integration plus a groups claim. Nothing is needed per user. -Console paths checked 2026-10-10 against Okta's documentation. - The DFE operator who sent you this page gives you the redirect URI. It looks like this, and you paste it exactly as given: ```text @@ -98,7 +96,7 @@ source_id: "" A linked group admits its members only once it carries a role, and until one does no Okta user can sign in, so link the first group as the local admin or the break-glass account. Behind a TLS-terminating proxy, set `api.forwarded_allow_ips` (`DFE_ENGINE_API_FORWARDED_ALLOW_IPS`), or the engine builds the redirect URI with `http://` and Okta refuses it. -## Vendor references +## Okta references - Create an OIDC app integration: - Authorization servers and API Access Management: diff --git a/docs/control-plane/oidc-rbac-architecture.md b/docs/control-plane/oidc-rbac-architecture.md index 0edf9adf9..92418dfcd 100644 --- a/docs/control-plane/oidc-rbac-architecture.md +++ b/docs/control-plane/oidc-rbac-architecture.md @@ -23,21 +23,21 @@ transcribed here where it would drift. ## 1. The auth model - local always, OIDC additive The engine is the RP and the **single token issuer**. An external IdP does the -interactive login; the engine validates the IdP id_token (signature/nonce/aud/ +interactive login. The engine validates the IdP id_token (signature/nonce/aud/ exp against the IdP JWKS, via Authlib) and re-mints its own ES384 token. Downstream (dfe-ui, dfe-hyperdx) only ever sees the engine token - the IdP's -token stays on the engine<->IdP leg. Code: `auth/oidc/rp.py`, -`api/v1/oidc_login.py`. +token stays on the engine<->IdP leg. Code: `src/dfe_engine/auth/oidc/rp.py`, +`src/dfe_engine/api/v1/oidc_login.py`. -By design several auth paths run at once and merge; OIDC never replaces local +By design several auth paths run at once and merge. OIDC never replaces local auth, it augments it: -- **Local accounts** (username/password -> JWT), **API keys**, and the - **disabled/dev** path are always available - the fallback that keeps an +- Local accounts (username/password -> JWT), API keys and the + disabled/dev path are always available - the fallback that keeps an operator in even if the IdP is down or misconfigured. -- **External OIDC** is added when provided (recommended for real deployments). +- External OIDC is added when provided (recommended for real deployments). When present it takes precedence, but the local and key paths stay live. -- **Groups from both sources unify.** An IdP group takes a DFE group's roles +- Groups from both sources unify. An IdP group takes a DFE group's roles only through that group file's `source_provider` + `source_id` link. A matching name grants nothing (rbac-sign-in.md section 2.1). One role model, two merged identity sources. @@ -82,7 +82,7 @@ engine machinery above it: Two principles: 1. **Provider-specific code stops at Layer 2.** Below it is generic OIDC - plumbing; above it, generic machinery that does not care which IdP the user + plumbing. Above it, generic machinery that does not care which IdP the user came from. Adding a provider = one adapter at Layer 2, nothing else changes. 2. **Control-plane and data-plane are orthogonal.** The same group membership feeds both, but control-plane roles decide what you can DO while org_ids @@ -93,7 +93,7 @@ Two principles: ### Provider config Providers are YAML files (one per provider, filename stem = name) managed by -`OIDCProviderRegistry`. The model (`auth/oidc/models.py`, the SSoT) carries: +`OIDCProviderRegistry`. The model (`src/dfe_engine/auth/oidc/models.py`, the SSoT) carries: `type` (generic|google|entra_id|okta), `enabled`, `issuer`, `client_id`, `client_id_env`, `client_secret_path`, `client_secret_env`, `scopes`, and a `groups` block (mode + claim_name + per-provider directory fields). @@ -108,21 +108,20 @@ serving a provider configured that way. ### Group resolution modes -- `token_claim` (the default) - groups arrive as strings in the id_token; direct lookup. -- `api` - the token lacks group names (or carries only ids); an adapter calls +- `token_claim` (the default) - groups arrive as strings in the id_token. Direct lookup. +- `api` - the token lacks group names (or carries only ids). An adapter calls the provider directory API to resolve names and enumerate membership on a - schedule. Each provider syncs every `groups.sync_interval` seconds (60 at least) from a background loop in every engine replica; `DFE_ENGINE_AUTH_OIDC_GROUP_SYNC_ENABLED=false` leaves only `POST /oidc-providers/{name}/sync`. -- `manual` - membership mapped by hand in the engine; the token's groups are ignored at login and nothing syncs. + schedule. Each provider syncs every `groups.sync_interval` seconds (60 at least) from a background loop in every engine replica. `DFE_ENGINE_AUTH_OIDC_GROUP_SYNC_ENABLED=false` leaves only `POST /oidc-providers/{name}/sync`. +- `manual` - membership mapped by hand in the engine. The token's groups are ignored at login and nothing syncs. -Which fields each provider type accepts and requires in each mode is enforced by the provider admin API, from the rules in `auth/oidc/field_rules.py`. A `generic` provider has no `api` mode and a `google` provider has only `api` mode, with `enrich_on_login` on. +Which fields each provider type accepts and requires in each mode is enforced by the provider admin API, from the rules in `src/dfe_engine/auth/oidc/field_rules.py`. A `generic` provider has no `api` mode and a `google` provider has only `api` mode, with `enrich_on_login` on. -### The normalized group +## 3.1 The normalized group Every provider's groups normalise to `NormalizedGroup { id, name, source }`. `GroupInfo { id, name, email, description }` (models.py) already provides this -for api-mode; the token_claim/login path (`extract_identity`, rp.py) still emits -a flat `list[str]`, but the resolution gap it created is now CLOSED at the -matching layer: each group file is linked by `source_provider` and that provider's +for api-mode. The token_claim/login path (`extract_identity`, rp.py) emits a flat +`list[str]`, and the matching layer resolves it: each group file is linked by `source_provider` and that provider's `source_id`, and a login takes the groups whose `source_id` it asserts, for the provider it signed in through (`membership.linked_groups`). A group's name links nothing. So an Entra login carrying object GUIDs - which match no group filename - @@ -131,17 +130,17 @@ The adapter is the ONLY place the remaining quirks live: | Quirk | Real cause | Adapter's job | Status | |-------|-----------|---------------|--------| -| refs are names (dex/okta) vs ids (entra) vs absent (google) | provider design | normalise to {id,name}; resolve GUIDs by source_id, call the directory API when the token lacks the name | id->name resolution DONE (source_id match) | +| refs are names (dex/okta) vs ids (entra) vs absent (google) | provider design | normalise to {id,name}. Resolve GUIDs by source_id, call the directory API when the token lacks the name | id->name resolution by source_id match | | id stable, name mutable | directories rename groups | map roles on `id`, display `name` | source_id is the stable key | -| overage (entra >200 -> a Graph pointer) | token size limits | detect the marker, fall back to the directory API | DONE: `rp.py` detects `_claim_names` and fetches `transitiveMemberOf` with the user's own login token (`/me`: User.Read returns each group's id, which is all role resolution needs; `GroupMember.Read.All` adds names), falling back to app credentials (tenant id + client secret) when Graph refuses that token | -| nested / transitive groups | AD/entra/google nesting | take the closure the provider already emits, or walk it when it does not | MEASURED: entra emits the full closure in-token; honour it (per-provider fact, not a global toggle) | -| name collisions across tenants | bare names aren't unique | namespace the key (source+domain) | a name held by a group not linked to that IdP group is skipped (`name_taken`); namespacing not yet built | +| overage (entra >200 -> a Graph pointer) | token size limits | detect the marker, fall back to the directory API | `rp.py` detects `_claim_names` and fetches `transitiveMemberOf` with the user's own login token (`/me`: User.Read returns each group's id, which is all role resolution needs. `GroupMember.Read.All` adds names), falling back to app credentials (tenant id + client secret) when Graph refuses that token | +| nested / transitive groups | AD/entra/google nesting | take the closure the provider already emits, or walk it when it does not | entra emits the full closure in-token. Honour it (per-provider fact, not a global toggle) | +| name collisions across tenants | bare names aren't unique | namespace the key (source+domain) | a name held by a group not linked to that IdP group is skipped (`name_taken`). Namespacing not yet built | | non-role groups (mailing lists) | not all groups grant access | filter by group type | not yet built | -### Adapters +## 3.2 Adapters -`auth/oidc/adapters/`: `generic` (token_claim/manual, no API), `google` (Cloud -Identity Groups API with the user's own login token; the Admin SDK Directory API +`src/dfe_engine/auth/oidc/adapters/`: `generic` (token_claim/manual, no API), `google` (Cloud +Identity Groups API with the user's own login token. The Admin SDK Directory API as an optional service account), `entra` (Microsoft Graph), `okta` (Groups API). Contract: `list_all_groups`, `resolve_user_groups`, `test_connection`. The two a login can reach never raise: `resolve_user_groups` reports an outage or refusal. An outage @@ -149,9 +148,9 @@ keeps the token's groups, else the account's. Anything else replaces them. `list_all_groups` serves the sync only and raises when a page fails, so the sync never reads a short listing as the whole directory. -### Sync and control-plane independence +## 3.3 Sync and control-plane independence -`auth/oidc/sync.py` enumerates a provider's groups (api mode). It creates a group file for each new one, linked by `source_provider` and `source_id`, and updates only files already linked to that IdP group (their `source_id` is its id and their `source_provider` this provider, or one `auth.source_provider_bindings` joins to it), preserving their roles. A file of the same name with no link is skipped as `name_taken`: a display name is no identity, so linking one is an admin's act through the groups API or dfe-ui's group form (rbac-groups.md section 4.4). A new group whose name is a stored group's `source_id` or whose id is a stored group's name is skipped as `name_taken` too, since an API key resolves an identifier by name, then by `source_id`. A file carrying the id under any other provider, whatever its name, is skipped as `provider_mismatch`, so a sync never moves a group between providers or creates a second group carrying the id. Detaching a provider does NOT delete its groups - they orphan with `source_provider` set, reported on delete. +`src/dfe_engine/auth/oidc/sync.py` enumerates a provider's groups (api mode). It creates a group file for each new one, linked by `source_provider` and `source_id`, and updates only files already linked to that IdP group (their `source_id` is its id and their `source_provider` this provider, or one `auth.source_provider_bindings` joins to it), preserving their roles. A file of the same name with no link is skipped as `name_taken`: a display name is no identity, so linking one is an admin's act through the groups API or dfe-ui's group form (rbac-groups.md section 4.4). A new group whose name is a stored group's `source_id` or whose id is a stored group's name is skipped as `name_taken` too, since an API key resolves an identifier by name, then by `source_id`. A file carrying the id under any other provider, whatever its name, is skipped as `provider_mismatch`, so a sync never moves a group between providers or creates a second group carrying the id. Detaching a provider does NOT delete its groups - they orphan with `source_provider` set, reported on delete. **Sign-in needs the engine.** It is the relying party and the only token issuer, so with the engine down nobody can sign in, and provider CRUD and group @@ -162,20 +161,20 @@ engine. One group resolution (`deps.py _resolve_group_grants` + the `Group` model) feeds both. A group file carries `roles`, `org_ids`, and `scope` -(`system`|`org:`); a user's roles and org_ids are the union across groups. +(`system`|`org:`). A user's roles and org_ids are the union across groups. - **Control plane (3a/4a):** `roles.yaml` + wildcard `permission_matches` gate the dfe-engine API (manage sources/hunts/config/deploy). Static named roles. -- **Data plane (3b/4b):** org_ids drive dynamic, data-only access. A caller fenced to one org (an `org_viewer`, or any role bound at that org's scope) browses Observe as the org's pinned ClickHouse user, whose READONLY tenant setting and the shared RESTRICTIVE row policies filter every query to that org (rbac.md section 5). Read-only; no control-plane power. +- **Data plane (3b/4b):** org_ids drive dynamic, data-only access. A caller fenced to one org (an `org_viewer`, or any role bound at that org's scope) browses Observe as the org's pinned ClickHouse user, whose READONLY tenant setting and the shared RESTRICTIVE row policies filter every query to that org (rbac.md section 5). Read-only. No control-plane power. -Both axes already resolve end to end. The re-minted token carries +Both axes resolve from the group files. The re-minted token carries `{sub, email, groups}` plus the `role` claim dfe-hyperdx reads (see -auth-envoy-topology.md); roles AND org_ids still derive engine-side from the +auth-envoy-topology.md). Roles AND org_ids still derive engine-side from the group files, not the token. Full detail: rbac.md sections 2, 5, 6, 7. ## 5. The management API and CLI -Admin CRUD lives at `/api/v1/auth/oidc-providers` (`api/v1/oidc_providers.py`): +Admin CRUD lives at `/api/v1/auth/oidc-providers` (`src/dfe_engine/api/v1/oidc_providers.py`): create / list / get / update / delete (delete reports orphaned groups) / `{name}/sync` / `{name}/test` / `{name}/verify-login`. RBAC scopes `oidc_read|write|delete`, audit on every mutation, pagination on list. The `dfe` @@ -185,45 +184,45 @@ Okta API token, the Entra client secret, the Google SA JSON), write them to the secret store and record only the path, then re-register the provider with the relying party - so a provider configured purely through the API completes a login without an env edit or a restart. A `*_env` field takes an environment -variable NAME; a credential pasted there is a 422 naming the field. Detach +variable NAME. A credential pasted there is a 422 naming the field. Detach deletes the provider's stored secrets. Verification for a thorough onboard/offboard workflow: -- **Login-config check - DONE.** `GET /{name}/verify-login` reports whether the - client_id and client_secret resolve (value never returned) and whether the - issuer's discovery document is reachable and well-formed. This is the login - half that `{name}/test` (group-DIRECTORY creds only, a no-op for generic - providers) never covered. -- **Consolidated introspection.** No single endpoint returns raw claims + - normalized groups + resolved roles + org_ids for a login. Assemble from the - callback JSON (`{subject,email,groups}`) + `GET /auth/me` (roles + org_ids, - now that /auth/me returns org_ids) - until a dedicated verify endpoint exists. +- `GET /{name}/verify-login` reports whether the client_id and client_secret + resolve (value never returned) and whether the issuer's discovery document is + reachable and well-formed. `{name}/test` checks the group-directory credentials + only, and is a no-op for generic providers. +- No single endpoint returns raw claims, normalized groups, resolved roles and + org_ids for a login. Assemble them from the callback JSON + (`{subject,email,groups}`) and `GET /auth/me` (roles and org_ids). ## 6. Infrastructure requirements -What a deployment must provide (dfe-infra owns these; the engine consumes them). +What a deployment must provide (dfe-infra owns these and the engine consumes them). -- **OIDC client credentials** per provider, as env vars from a K8s Secret +- OIDC client credentials per provider, as env vars from a K8s Secret (materialised by ESO from OpenBao / cloud SM). Env-var NAMES are declared in - the provider config; the chart mounts the values. Per-provider secret keys + the provider config. The chart mounts the values. Per-provider secret keys (client-id, client-secret, and provider extras - Google SA JSON, Entra tenant-id, Okta api-token) are defined by the tofu modules, not here. -- **Cloud IAM provisioning** (tofu): `tf-oidc-google` (OAuth client is +- Cloud IAM provisioning (tofu): `tf-oidc-google` (OAuth client is console-only - the module does the group-resolver SA), `tf-oidc-entra` (app registration + admin-consented `GroupMember.Read.All`), `tf-oidc-okta` (OIDC - app + groups claim + API token). Modules live in dfe-infra; this engine just + app + groups claim + API token). Modules live in dfe-infra. This engine just reads the resulting creds. -- **DNS + TLS** for the callback host. The native RP callback is +- DNS + TLS for the callback host. The native RP callback is `https://{dfe_host}/api/v1/auth/oidc/{provider}/callback` (+ localhost for dev). -- **Network egress** to IdP directory APIs for api-mode group resolution - (`admin.googleapis.com`, `graph.microsoft.com`, `{org}.okta.com`). -- **Envoy Gateway SecurityPolicy - the FALLBACK path only.** The Envoy - header-trust path (injecting `X-Oidc-Subject`/`X-Oidc-Groups`) is kept for - deployments where the engine sits strictly behind Envoy. It is NOT the - primary login path (the native RP is). When used, the engine MUST be reachable - only via Envoy (NetworkPolicy), or headers are forgeable. +- Network egress to IdP directory APIs for api-mode group resolution + (`cloudidentity.googleapis.com`, and the Admin SDK at `www.googleapis.com` when a + service account is set, `graph.microsoft.com`, `{org}.okta.com`). +- The edge header path, a fallback only. dfe-infra's gateway can verify an + IdP token and inject `X-Oidc-Subject` (`jwtAuthn`, off by default), and the + engine trusts it only with `auth.trust_proxy_auth_headers`. Envoy Gateway cannot + map an array claim to a header, so `X-Oidc-Groups` never arrives and such a login + holds no roles (dfe-infra `helm/edge/gateway/values.yaml`, dfe-infra#222). The + native RP is the login path. When the header path is used, the engine MUST be + reachable only via Envoy (NetworkPolicy), or the headers are forgeable. ## 7. Provider-agnostic onboarding recipe diff --git a/docs/control-plane/rbac-engine-reads.md b/docs/control-plane/rbac-engine-reads.md index adc76b487..42cd056e1 100644 --- a/docs/control-plane/rbac-engine-reads.md +++ b/docs/control-plane/rbac-engine-reads.md @@ -4,12 +4,16 @@ Section 5.3 of [rbac.md](rbac.md), which keeps its number so section references ## 5.3 The Engine's Own Reads -The query API does not lean on a ClickHouse user for tenancy: the executor injects the caller's `org_id` as a reserved bind parameter that a client cannot override (`query/executor.py`, `RESERVED_PARAMS`). +The query API does not lean on a ClickHouse user for tenancy: the executor injects the caller's `org_id` as a reserved bind parameter that a client cannot override (`src/dfe_engine/query/executor.py`, `RESERVED_PARAMS`). -The row reads the engine makes for a caller with no platform grant -- the sampler, the transform dry run, and the schema routes `sample-rows`, `json-paths` and `promote-field` -- bind `_org_id` to the tenant ids of the registered orgs a grant holding the read's action reaches (`api/deps.py::held_orgs`), the same ids each org's pinned user carries (`orgs/tenant_scope.py`). A grant bound at an org's scope reaches that org. A tenant role bound system-wide (`org_viewer`, or a `scoped` role) reaches only the orgs its own group names, never one that only another of the caller's groups names, as its group's ClickHouse user does. The sampler, `/operations`, `sample-rows` and `json-paths` accept that action at system scope or at one of the caller's orgs. `promote-field` changes a source every org shares, so it still needs `schema:write` at system scope. Each org marker the caller carries resolves the same way as a group's (`resolve_orgs`): by name, or failing that by the one org declaring it as a tenant id. A marker no org declares, or a tenant id two orgs share, binds nothing, and a caller left with no tenant id, or a table with no `_org_id`, is refused before any row is read. `/operations` shows such a caller only the operations held to its orgs. +The row reads the engine makes for a caller with no platform grant -- the sampler, the transform dry run, and the schema routes `sample-rows`, `json-paths` and `promote-field` -- bind `_org_id` to the tenant ids of the registered orgs a grant holding the read's action reaches (`api/deps.py::held_orgs`), the same ids each org's pinned user carries (`src/dfe_engine/orgs/tenant_scope.py`). A grant bound at an org's scope reaches that org. A tenant role bound system-wide (`org_viewer`, or a `scoped` role) reaches only the orgs its own group names, never one that only another of the caller's groups names, as its group's ClickHouse user does. The sampler, `/operations`, `sample-rows` and `json-paths` accept that action at system scope or at one of the caller's orgs. `promote-field` changes a source every org shares, so it still needs `schema:write` at system scope. Each org marker the caller carries resolves the same way as a group's (`resolve_orgs`): by name, or failing that by the one org declaring it as a tenant id. A marker no org declares, or a tenant id two orgs share, binds nothing, and a caller left with no tenant id, or a table with no `_org_id`, is refused before any row is read. `/operations` shows such a caller only the operations held to its orgs. + +## Source reads `GET /api/v1/sources` and every other source read (`/sources/{name}`, `/columns`, `/flow`, `/signals`, `/export`, `/versions/{version}`, `/catalogue`, `/catalogue-tags`, `/engines`) accept `source:read` at system scope or at one of the caller's orgs. A source definition is shared by every org, so every such caller lists those. A fetcher source belongs to the org it names (the default org when it names none), so a caller without a platform grant lists the fetcher sources of its held orgs only. A named read of a fetcher source bound to an org the caller does not hold gets a 403 `org_not_granted`. A caller with no `source:read` at system scope or in any org gets the 403. The same rule covers `GET /schemas/{source}/columns`, `GET /kafka/topics` (a held caller sees the topics of the sources it may read) and `GET /ingest/rules` (a held caller sees the blocks of those sources). `GET /ingest/compat` and `/ingest/compat-profiles` are the deployment's own bindings and profiles, shared by every source, so any `source:read` holder reads them. `GET /hyperdx/sources` lists every org's dfe-hyperdx team, so it needs `source:read` at system scope and refuses a caller held to its orgs. +## Hunt reads + The hunt runner's ClickHouse user reads every org, so every statement it compiles carries `_org_id IN ()` as its own conjunct, and a rule condition may read no other table except one in `hunts.lookup_databases`. The alert-volume preview a rule write runs carries the same conjunct over the customers of each hunt that runs or names the rule, cut to the orgs the author may read with `rule:write`. A rule no hunt runs is counted over the author's orgs alone, every org for a platform author, and a fence left with no org leaves the rule unmeasured. A listed database holds shared reference data and no org's rows. The runner refuses to start, or stops at its next reload, while one of its tables has an `_org_id` column or is not a plain store of its own rows, as a view is. The engine refuses to start on one too when ClickHouse answers at its start, and starts with the check unrun when it does not ([source-detections.md](../data-plane/source-detections.md#lookup-databases)). ## Usage reads diff --git a/docs/control-plane/rbac-sign-in.md b/docs/control-plane/rbac-sign-in.md index 8b4376703..0411d9e3e 100644 --- a/docs/control-plane/rbac-sign-in.md +++ b/docs/control-plane/rbac-sign-in.md @@ -61,7 +61,7 @@ The linked groups are also the sign-in allowlist. An IdP sign-in, through the OI An admin sets `source_id` and `source_provider` together through the groups API or dfe-ui's group form ([section 4.4](rbac-groups.md#44-group-sync-process)). `source_provider` names the provider whose logins the link answers. Where a `token_claim` provider's claim carries names (Okta's default, dex, Keycloak), `source_id` is the name the IdP sends. For Entra it is the object ID. -A groups claim sent as one string is split on commas, never on whitespace (`auth/oidc/rp.py`), and `X-Oidc-Groups` is split on commas. So a string claim of `SOC Analysts` is the one group `SOC Analysts`. A `source_id` containing a comma matches only an array claim on the relying-party path. +A groups claim sent as one string is split on commas, never on whitespace (`src/dfe_engine/auth/oidc/rp.py`), and `X-Oidc-Groups` is split on commas. So a string claim of `SOC Analysts` is the one group `SOC Analysts`. A `source_id` containing a comma matches only an array claim on the relying-party path. --- diff --git a/docs/control-plane/rbac-vocabulary.md b/docs/control-plane/rbac-vocabulary.md index 178c2550e..d1fc5d7b2 100644 --- a/docs/control-plane/rbac-vocabulary.md +++ b/docs/control-plane/rbac-vocabulary.md @@ -21,9 +21,9 @@ ClickHouse tenant model. Roles split across two orthogonal planes, and conflating them is the most common modelling mistake: -- **Control plane** - what you can DO. Static roles granting API actions: +- Control plane - what you can DO. Static roles granting API actions: configure sources, run hunts, deploy services, manage groups. -- **Data plane** - what you can SEE. Dynamic, driven by `org_ids`, and +- Data plane - what you can SEE. Dynamic, driven by `org_ids`, and restricted to data browsing (Observe, Kibana-style). A user can be a powerful analyst on the control plane and see exactly one org's rows. @@ -67,18 +67,18 @@ it stands, because a parallel name would mean two groups granting one role. ## org_viewer -- the org-tied exception role -`org_viewer` (renamed from `customer_viewer`) is the one role hard-tied to the org_id CRUD: a user or group carrying it is associated with one or more `org_ids`, and that association IS the grant. When such a user browses data through the Observe iframe inside dfe-ui, dfe-hyperdx connects as that org's pinned ClickHouse user, whose READONLY tenant setting and the shared RESTRICTIVE row policies filter every query to the org's `_org_id` values, always. There is no unfiltered path: the scoping lives in the ClickHouse user, not in anything the UI or the user sends. See [rbac.md](rbac.md) sections 5-6 for the mechanism. +`org_viewer` is the one role hard-tied to the org_id CRUD: a user or group carrying it is associated with one or more `org_ids`, and that association IS the grant. When such a user browses data through the Observe iframe inside dfe-ui, dfe-hyperdx connects as that org's pinned ClickHouse user, whose READONLY tenant setting and the shared RESTRICTIVE row policies filter every query to the org's `_org_id` values, always. There is no unfiltered path: the scoping lives in the ClickHouse user, not in anything the UI or the user sends. See [rbac.md](rbac.md) sections 5-6 for the mechanism. ## Naming conventions -- **Roles**: `snake_case`, singular, describing the person - `data_analyst`, not +- Roles: `snake_case`, singular, describing the person - `data_analyst`, not `DataAnalystRole` or `ANALYST_USER`. -- **Groups**: `kebab-case`, plural, `dfe-` prefixed - `dfe-analysts`. The prefix +- Groups: `kebab-case`, plural, `dfe-` prefixed - `dfe-analysts`. The prefix matters because these names land in customers' own IdPs alongside their groups. -- **Per-org groups**: `dfe--viewers`, paired with `scope: org:`. -- **Permissions**: `resource:action`, colon-separated, `*` wildcards - +- Per-org groups: `dfe--viewers`, paired with `scope: org:`. +- Permissions: `resource:action`, colon-separated, `*` wildcards - `hunt:read`, `service:*:config:*`. -- **Orgs / tenants**: the term is **org** everywhere in the product surface (`org_id`, `org_ids`, `scope: org:`). "Tenant" appears only inside the ClickHouse isolation model (the tenant role, the tenant setting and its row policies). One concept, and the boundary between the two words is exactly that model. +- Orgs and tenants: the term is **org** everywhere in the product surface (`org_id`, `org_ids`, `scope: org:`). "Tenant" appears only inside the ClickHouse isolation model (the tenant role, the tenant setting and its row policies). One concept, and the boundary between the two words is exactly that model. ## Who consumes what @@ -87,19 +87,19 @@ it stands, because a parallel name would mean two groups granting one role. | dfe-engine | roles + groups | Defines them, enforces via `_resolve_group_grants` and `RoleConfig` | | dfe-ui | roles | Shows/hides UI by resolved role from `GET /auth/me`, and must not hardcode its own names | | dfe-infra | groups | ArgoCD RBAC group mappings, ESO secret wiring for providers | -| dfe-deploy | groups | Deployment-time group seeding | +| dfe-deploy | nothing | Ships no group files. The engine seeds the default groups at boot | | dfe-schemas | orgs | `org_id` column semantics for tenant row policies | | Test environments | both | The OIDC test fixture renders these names into dex, Entra and Okta | ## Rules -1. **Add roles here first.** A role that exists in a consumer but not in +1. Add roles here first. A role that exists in a consumer but not in `roles.yaml` grants nothing - unknown groups resolve to zero roles, which presents as a broken IdP rather than a config error. -2. **Never rename in a consumer.** Renaming propagates from this document and +2. Never rename in a consumer. Renaming propagates from this document and the two engine files outward, in one change. -3. **Group names are external contract.** They are created in customers' Entra, +3. Group names are external contract. They are created in customers' Entra, Okta and Workspace directories. Renaming one is a customer-facing migration, not a refactor. -4. **Map roles on stable ids where the provider has them**, and display names. +4. Map roles on stable ids where the provider has them, and display names. Entra emits group GUIDs, not names. See the OIDC architecture doc. diff --git a/docs/control-plane/rbac.md b/docs/control-plane/rbac.md index 4f236b6d1..4480c8a34 100644 --- a/docs/control-plane/rbac.md +++ b/docs/control-plane/rbac.md @@ -12,12 +12,12 @@ flowchart TD REQ[Incoming Request] --> OIDC{X-Oidc-Subject header
and auth.trust_proxy_auth_headers?} OIDC -->|Yes| EXTRACT_OIDC[Extract user_id + groups
from OIDC headers] - EXTRACT_OIDC --> RESOLVE_OIDC[GroupStore resolves
groups → roles] + EXTRACT_OIDC --> RESOLVE_OIDC[GroupStore resolves
groups -> roles] RESOLVE_OIDC --> CTX[Build AuthContext] OIDC -->|No| APIKEY{X-API-Key
header?} APIKEY -->|Yes| VERIFY_KEY[APIKeyStore.verify
Parse short+long token
SHA-384 compare] - VERIFY_KEY --> RESOLVE_KEY[GroupStore resolves
key groups → roles] + VERIFY_KEY --> RESOLVE_KEY[GroupStore resolves
key groups -> roles] RESOLVE_KEY --> CTX APIKEY -->|No| JWT{Authorization:
Bearer?} @@ -50,7 +50,7 @@ A JWT carries identity, not authority. Its `org_id`, `roles`, `groups` and `org_ A session's `org_id` is its account's `org`, set by `POST` or `PUT /api/v1/auth/accounts`, or its API key's `org`, set by `POST /api/v1/auth/api-keys` (`422 unknown_org` for an unregistered org), and empty for one bound to none. A view taking `org_id` runs only when the session's groups grant that org or it reads every org, else `403 no_org` or `403 org_not_granted`. A `sub` that binds no account holds nothing -- an API-key subject, a deleted account -- and with auth enabled `POST /api/v1/auth/refresh` answers 401 for it, since each refresh issues a new expiry. -OIDC headers injected by Envoy Gateway SecurityPolicy, read only when `auth.trust_proxy_auth_headers` (`DFE_ENGINE_AUTH_TRUST_PROXY_AUTH_HEADERS`) is true: +OIDC headers a proxy injects, read only when `auth.trust_proxy_auth_headers` (`DFE_ENGINE_AUTH_TRUST_PROXY_AUTH_HEADERS`) is true. dfe-infra's gateway sets them with its opt-in `jwtAuthn`, and Envoy Gateway cannot map the array `groups` claim, so `X-Oidc-Groups` does not arrive from it: | Header | Content | |--------|---------| @@ -79,7 +79,7 @@ dfe-engine MUST only be accessible via the proxy that sets the headers. Direct p blocked by K8s NetworkPolicy. Without this, any client with pod access can forge OIDC headers and impersonate any user. -### 1.4 Credential Storage Architecture +## 1.4 Credential Storage Architecture ```mermaid flowchart LR @@ -105,7 +105,7 @@ flowchart LR ENV -->|JWT signing key,
CH passwords| API ``` -#### What Goes Where +**What goes where:** | Data | Where stored | Format | Safe to commit? | |------|-------------|--------|-----------------| @@ -117,17 +117,17 @@ flowchart LR | JWT signing key | Secrets backend at `api.jwt_key_path` (default `jwt/signing-key`), minted on first boot when absent | ECDSA private key, PEM (P-384 for the default ES384) | **NO** | | OIDC login session cookie key | `DFE_ENGINE_API_SESSION_SECRET`, else `DFE_ENGINE_API_JWT_SECRET` | Random, `DFE_ENGINE_API_JWT_SECRET` at least 32 bytes | **NO** | | CH connection passwords | Env var / K8s Secret | Plaintext | **NO** | -| Role definitions | `rbac/roles.yaml` beside the auth directory, seeded from `auth/resources/roles.yaml` on first boot | Permission patterns | Yes | -| Group→role mapping | `groups/{name}.yaml` | Role list | Yes | +| Role definitions | `rbac/roles.yaml` beside the auth directory, seeded from `src/dfe_engine/auth/resources/roles.yaml` on first boot | Permission patterns | Yes | +| Group->role mapping | `groups/{name}.yaml` | Role list | Yes | -#### Config Directory Layout +**Config directory layout:** ```text config/auth/ accounts/ # One YAML per local user account admin.yaml analyst1.yaml - groups/ # One YAML per group (group → roles) + groups/ # One YAML per group (group -> roles) dfe-admins.yaml soc-analysts.yaml api-keys/ # One YAML per API key @@ -136,11 +136,11 @@ config/auth/ ``` Account filename = username. Group filename = group name. API key filename = -key name. The filename IS the identity — never stored inside the YAML body +key name. The filename IS the identity -- never stored inside the YAML body (avoids DirectoryConfigStore YAML 1.1 boolean coercion on values like `off`, `yes`, `no`). -### 1.5 Password Verification Flow +## 1.5 Password Verification Flow ```python # AccountStore.verify_password() @@ -153,29 +153,29 @@ return bcrypt.checkpw(password.encode(), account.password_hash.encode()) Constant-time rejection for unknown usernames prevents enumeration attacks. -### 1.6 API Key Format +## 1.6 API Key Format ```text -dfe_ak_8a3f2c91_7f3b2c4d8e9a1b5f6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1f +dfe_ak_8a3f2c91_7f3b2c4d8e9a1b5f6c7d8e9f0a1b2c3d | | | prefix short long token (shown once, stored as SHA-384 hash) token (lookup index, shown in UI) ``` -- **Prefix** (`dfe_ak_`): enables secret scanning by GitHub, GitGuardian -- **Short token** (8 hex chars): plaintext in YAML, used for lookup and display -- **Long token** (32 hex chars): SHA-384 hashed. Not bcrypt -- keys are +- Prefix (`dfe_ak_`): enables secret scanning by GitHub, GitGuardian +- Short token (8 hex chars): plaintext in YAML, used for lookup and display +- Long token (32 hex chars): SHA-384 hashed. Not bcrypt -- keys are high-entropy random, bcrypt's slowness adds no security value - Full key shown **once** at creation, never retrievable again -- **Expiry** (`expires_at`, optional ISO-8601 stored as UTC): enforced on every +- Expiry (`expires_at`, optional ISO-8601 stored as UTC): enforced on every verify, so a lapsed key stops authenticating with no sweeper running. The file stays until revoked, and `list` reports `expired: true`. A key created through the API without one expires after `auth.api_key_default_ttl_days` (default 90, `0` for no expiry) -- **Revocation:** Delete the key's YAML file or call the revoke API +- Revocation: delete the key's YAML file or call the revoke API -### 1.7 API Key Verification Flow +## 1.7 API Key Verification Flow ```python # APIKeyStore.verify() @@ -241,7 +241,7 @@ A role write grants only what its caller already holds. Creating a role, or chan A `scoped` role grants only `...:read` patterns and `query:execute`, as at system scope it passes every org-scope check. Anything else fails the roles file's load, and the roles API answers 422. -### 2.3 Permission Taxonomy +## 2.3 Permission Taxonomy | Domain | Actions | Scoped by service? | |--------|---------|---------------------| @@ -263,13 +263,13 @@ A `scoped` role grants only `...:read` patterns and `query:execute`, as at syste The routes enforce more domains than this table lists (accounts, groups, API keys, roles, OIDC providers, governance, and others). The full catalogue is `src/dfe_engine/auth/rbac_scopes/scope_constants.py`. -#### The usage scope +**The usage scope:** `usage:read` opens `GET /api/v1/usage` and `GET /api/v1/usage/overview` ([ui-api-guide.md](ui-api-guide.md#usage-apiv1usage)). Four built-in roles carry it: `infra_admin`, `infra_viewer`, `dfe_operator` and `org_viewer`. `admin` holds it through `*`. A changed built-in grant reaches an existing deployment at its next restart, see [2.2](#22-role-definitions). A grant at system scope or at an org's scope opens both routes. A holder whose grant reads every org sees the whole deployment and may name any org. Any other holder is held to its own orgs that grant `usage:read`. `org_viewer` is scoped, so it is always held. The reads are described in [rbac-engine-reads.md](rbac-engine-reads.md#usage-reads). -### 2.4 Wildcard Permission Matching +## 2.4 Wildcard Permission Matching ```python def permission_matches(permission: str, action: str) -> bool: @@ -277,7 +277,7 @@ def permission_matches(permission: str, action: str) -> bool: | Pattern | Matches | Does NOT match | |---------|---------|----------------| -| `*` | Everything | — | +| `*` | Everything | -- | | `config:*` | `config:read`, `config:write`, `config:read:sub` | `hunt:read` | | `service:*:config:*` | `service:loader:config:read` | `service:loader:metrics:read` | | `service:*:config:read` | `service:loader:config:read` | `service:loader:config:write` | @@ -286,7 +286,7 @@ def permission_matches(permission: str, action: str) -> bool: - Trailing `*` matches any remaining segments - Mid-position `*` matches exactly one segment -### 2.5 Authorization Engine +## 2.5 Authorization Engine ```python # engine.py @@ -303,12 +303,12 @@ def authorize( Decision order: -1. `enabled=False` → allow (dev/test) -2. `auth=None` → root mode, allow +1. `enabled=False` -> allow (dev/test) +2. `auth=None` -> root mode, allow 3. For each of the caller's scoped grants (`auth.grants`, else `auth.roles` as system grants): allow when the grant's scope covers the requested scope and its role's permissions match the action. `scope=None` is a system-scope check 4. Return `AuthzResult(allowed, reason)` -### 2.6 RBAC Enforcement in API +## 2.6 RBAC Enforcement in API ```python # api/deps.py @@ -426,10 +426,10 @@ On first startup (empty `config/auth/` directory), `bootstrap_auth()` seeds: **Groups:** -- `dfe-admins` → roles: `[admin]` -- `dfe-analysts` → roles: `[data_analyst]` -- `dfe-viewers` → roles: `[data_viewer]` -- `dfe-infra` → roles: `[infra_admin]` +- `dfe-admins` -> roles: `[admin]` +- `dfe-analysts` -> roles: `[data_analyst]` +- `dfe-viewers` -> roles: `[data_viewer]` +- `dfe-infra` -> roles: `[infra_admin]` **Accounts:** `admin` and `breakglass`, both in `dfe-admins`. See 3.5. @@ -501,7 +501,7 @@ A Google login reads the user's groups from Cloud Identity with that login's own The customer's IdP admin sets up their side from [idp-setup/google.md](idp-setup/google.md), [idp-setup/entra.md](idp-setup/entra.md) or [idp-setup/okta.md](idp-setup/okta.md), and sends back the client credentials, the issuer and the group identifiers each provider links on. Config holds a secret PATH into the `DfeSecrets` store, or the NAME of an env var -(`client_id_env`, `client_secret_env`, `service_account_json_env`) — never a +(`client_id_env`, `client_secret_env`, `service_account_json_env`) -- never a secret value. Resolution reads the store first, so a credential sent to `POST /api/v1/auth/oidc-providers` takes effect without a restart, then the environment, so an ESO-mounted variable keeps working. @@ -513,18 +513,18 @@ Surrounding whitespace, such as the newline a mounted secret file ends with, is | no `*_env` field | the env var name and any stored copy are kept | | `*_env` set to `""` | the name is cleared, the stored copy is kept | | a name, with no value for the secret, that differs from the one held or sits beside a stored copy | the env var must resolve non-empty in the engine: the name is saved and the stored copy is deleted once the update is saved | -| the same, but the engine does not set that env var | `422`, the field error `env_var_not_set` on that `*_env` field; nothing is saved or deleted | +| the same, but the engine does not set that env var | `422`, the field error `env_var_not_set` on that `*_env` field. Nothing is saved or deleted | | the name already held, with nothing stored | no change | | a value for the secret | the value is stored and the name saved as sent | -A create naming an env var the engine does not set, with no value for that secret, is refused the same way. A stored copy whose delete fails after the update is saved is logged by its store path; the update still answers 200. +A create naming an env var the engine does not set, with no value for that secret, is refused the same way. A stored copy whose delete fails after the update is saved is logged by its store path. The update still answers 200. ### 4.3 Adapter Implementations | Adapter | Status | Admin API | Group sync | |---------|--------|-----------|------------| | Generic | Done | None | No (use `token_claim` mode) | -| Google | Done | Cloud Identity as the user at login; Admin SDK `groups().list()` as an optional service account | Only with a service account | +| Google | Done | Cloud Identity as the user at login. Admin SDK `groups().list()` as an optional service account | Only with a service account | | Entra ID | Done | Graph `/me/transitiveMemberOf` with the user's token on a >200 group overage, falling back to app-only `/users/{oid}/transitiveMemberOf` when Graph refuses that token. Graph `/groups` for sync | Yes, api mode with app credentials -- `$top=999` pagination | | Okta | Done | Groups API `/api/v1/groups` with an API token, api mode only. `token_claim` calls nothing | Yes, api mode -- `Link` header pagination | @@ -540,11 +540,11 @@ Moved to [rbac-groups.md](rbac-groups.md#44-group-sync-process). ### 5.1 Pinned Org Users, One Shared Policy Set -Every registered org gets its own ClickHouse user, and that user is what fences it in. The reconciler in `governance/ch/` renders the whole model from the org registry and the tier catalogue. The role, setting and user names come from the dfe-schemas catalogue, not from engine code. +Every registered org gets its own ClickHouse user, and that user is what fences it in. The reconciler in `src/dfe_engine/governance/ch/` renders the whole model from the org registry and the tier catalogue. The role, setting and user names come from the dfe-schemas catalogue, not from engine code. -- **One shared tenant role.** Every org-pinned user holds it and nothing else does. -- **One RESTRICTIVE row policy per table that carries `_org_id`**, on that role. Its predicate is `has(splitByChar(',', getSetting('')), _org_id)`. -- **The tenant setting, pinned READONLY on each org user**, listing that org's `_org_id` values. A `SETTINGS` clause in query text cannot override it: ClickHouse answers 452 (SETTING_CONSTRAINT_VIOLATION). +- One shared tenant role. Every org-pinned user holds it and nothing else does. +- One RESTRICTIVE row policy per table that carries `_org_id`, on that role. Its predicate is `has(splitByChar(',', getSetting('')), _org_id)`. +- The tenant setting, pinned READONLY on each org user, listing that org's `_org_id` values. A `SETTINGS` clause in query text cannot override it: ClickHouse answers 452 (SETTING_CONSTRAINT_VIOLATION). A platform identity does not hold the tenant role, so no policy targets it and it reads every row. RESTRICTIVE-only is deliberate. A PERMISSIVE policy would flip the table to default-deny for everyone. A holder of the tenant role with no pinned setting makes `getSetting` throw, so a bad grant fails closed. @@ -556,7 +556,7 @@ Each reconcile replaces a pinned or group user's whole role set, so a group that It also replaces each tier, service and tenant role's privileges with the ones configured, so a grant dropped from a tier is revoked. Where ClickHouse refuses the list, the role is cleared and given each entry it accepts. -### 5.2 Who Reads As Whom +## 5.2 Who Reads As Whom ```mermaid flowchart TD @@ -567,9 +567,9 @@ flowchart TD ORG -->|no| DENY["403 no_single_org
fails closed"] ``` -`platform_grants` (`auth/models.py`) is the one filter for "every org". The ClickHouse group bindings, the sampler, the dfe-hyperdx connection read, the fork's role claim and JIT team assignment all go through it. A role bound at an org's scope covers that org alone, a role marked `scoped` in `roles.yaml` is held to its holder's orgs wherever it is bound, a role `roles.yaml` does not declare unfences nothing, and `org_viewer` never unfences anyone. +`platform_grants` (`src/dfe_engine/auth/models.py`) is the one filter for "every org". The ClickHouse group bindings, the sampler, the dfe-hyperdx connection read, the fork's role claim and JIT team assignment all go through it. A role bound at an org's scope covers that org alone, a role marked `scoped` in `roles.yaml` is held to its holder's orgs wherever it is bound, a role `roles.yaml` does not declare unfences nothing, and `org_viewer` never unfences anyone. -Group bindings (`governance/ch/bindings.py`) follow the same rule. An org-scoped group binds to its org's pinned user whatever roles it holds. A system group holding a platform role reads unrestricted, and an `admin` or `infra_admin` group also reads the otel database. A group's claimed org markers resolve to a registered org by name or, failing that, by the one org that declares the marker as a tenant id (`orgs/tenant_scope.py::resolve_orgs`); a marker no org declares, or a tenant id two orgs both declare, resolves to none, so the group gets no ClickHouse user at all -- the alternative is an unrestricted one. +Group bindings (`src/dfe_engine/governance/ch/bindings.py`) follow the same rule. An org-scoped group binds to its org's pinned user whatever roles it holds. A system group holding a platform role reads unrestricted, and an `admin` or `infra_admin` group also reads the otel database. A group's claimed org markers resolve to a registered org by name or, failing that, by the one org that declares the marker as a tenant id (`orgs/tenant_scope.py::resolve_orgs`). A marker no org declares, or a tenant id two orgs both declare, resolves to none, so the group gets no ClickHouse user at all -- the alternative is an unrestricted one. ### 5.3 The Engine's Own Reads @@ -614,9 +614,9 @@ A reconcile that raises, ClickHouse unreachable being the usual cause, is logged A hunt stores the tenant ids each of its customer orgs resolved to, because the hunt runner reads the deploy repo and the org registry is not in it. `POST /api/v1/orgs`, and `PUT` and `DELETE` on `/api/v1/orgs/{name}`, store the org's tenant ids afresh in every hunt naming it, as one deploy-repo commit, before the org itself is written. A hunt write that fails leaves the org as it was, so the same request can be sent again, and a `PUT` with nothing changed repairs a hunt holding stale ids. -- **Delete.** The hunt keeps the org's name and loses its ids, so it fails as `unknown_customers` and holds its watermark until someone edits it. -- **Rename.** An org's name is its identity, so `PUT` cannot change it. -- **Review.** Production with `DFE_ENGINE_GITOPS_MODE=team` puts the hunt change on one review branch, the org answer carries `X-DFE-Review-Required`, and the runner keeps the old ids until it is merged. +- Delete: the hunt keeps the org's name and loses its ids, so it fails as `unknown_customers` and holds its watermark until someone edits it. +- Rename: an org's name is its identity, so `PUT` cannot change it. +- Review: production with `DFE_ENGINE_GITOPS_MODE=team` puts the hunt change on one review branch, the org answer carries `X-DFE-Review-Required`, and the runner keeps the old ids until it is merged. Each hunt rewritten emits `resource.hunt.updated` naming the org. @@ -626,21 +626,21 @@ Each hunt rewritten emits `resource.hunt.updated` naming the org. ### 7.1 Strategy -- **Connections:** the fork asks `GET /api/v1/hyperdx/connection` with the user's own token and seeds that one per-org credential on their team; the engine writes none itself -- **Runtime:** `HyperDXClient` calls the dfe-hyperdx internal API for the team and for per-DFE-source CRUD -- **Failures:** Non-fatal. First failure sets `_connected=False`, subsequent calls logged as warnings +- Connections: the fork asks `GET /api/v1/hyperdx/connection` with the user's own token and seeds that one per-org credential on their team. The engine writes none itself +- Runtime: `HyperDXClient` calls the dfe-hyperdx internal API for the team and for per-DFE-source CRUD +- Failures: non-fatal. First failure sets `_connected=False`, subsequent calls logged as warnings ### 7.2 Teams and Connections -- **Team name.** JIT picks it from the caller's grants (`auth/jit.py`, `resolve_hyperdx_team`): `dfe-admin` for `admin` or `infra_admin` at system scope, `dfe-analysts` for `data_analyst` at system scope, else `customer-` for the caller's first org, else none. -- **Connection.** A team's ClickHouse connection is whatever `GET /api/v1/hyperdx/connection` returned to the user whose token seeded it (section 5.2): the platform reader, or that user's org's pinned user. -- **Role.** `GET /api/v1/auth/me` hands the fork `hyperdx_role` (`auth/hyperdx_role.py`). `admin` goes to `admin` or `infra_admin` at system scope and may change what a whole team sees. `member` goes to anyone whose roles allow `query:write`, the dfe-hyperdx authoring permission, and may save searches and dashboards in the observe embed. Everyone else is `viewer`, and the fork refuses it every write. So `admin` and `data_analyst` author, and the other shipped roles read. +- Team name: JIT picks it from the caller's grants (`src/dfe_engine/auth/jit.py`, `resolve_hyperdx_team`): `dfe-admin` for `admin` or `infra_admin` at system scope, `dfe-analysts` for `data_analyst` at system scope, else `customer-` for the caller's first org, else none. +- Connection: a team's ClickHouse connection is whatever `GET /api/v1/hyperdx/connection` returned to the user whose token seeded it (section 5.2): the platform reader, or that user's org's pinned user. +- Role: `GET /api/v1/auth/me` hands the fork `hyperdx_role` (`src/dfe_engine/auth/hyperdx_role.py`). `admin` goes to `admin` or `infra_admin` at system scope and may change what a whole team sees. `member` goes to anyone whose roles allow `query:write`, the dfe-hyperdx authoring permission, and may save searches and dashboards in the observe embed. Everyone else is `viewer`, and the fork refuses it every write. So `admin` and `data_analyst` author, and the other shipped roles read. --- ## 8. Argo CD Permissions -Roles carry `argo:{resource}:{action}` permissions in `roles.yaml`, and the scope catalogue publishes the `argo:` prefix (`auth/rbac_scopes`). The engine does not export them: it writes no `argocd-rbac-cm` policy and no AppProject roles. +Roles carry `argo:{resource}:{action}` permissions in `roles.yaml`, and the scope catalogue publishes the `argo:` prefix (`src/dfe_engine/auth/rbac_scopes`). The engine does not export them: it writes no `argocd-rbac-cm` policy and no AppProject roles. --- @@ -666,8 +666,8 @@ A refused bearer token emits `auth.login.denied` with `login_method: jwt`, the v Every account, group, API key and role write through `/api/v1/auth` emits one event naming the actor (`admin_id`), the change and the target. Its `details` carry the roles, groups, members or permissions the write set, and the names of contact fields changed, never their values. No event carries a password, a hash, an attribute value or an API key's secret half. -Events flow through the OTel pipeline → JSON → ClickHouse → Observe. -No custom ClickHouse audit table — standard OTel log ingestion is used. +Events flow through the OTel pipeline -> JSON -> ClickHouse -> Observe. +No custom ClickHouse audit table -- standard OTel log ingestion is used. --- @@ -688,7 +688,7 @@ class AuthContext(BaseModel): user_agent: str | None = None ``` -Permissions are resolved at authorisation time from roles — never stored on +Permissions are resolved at authorisation time from roles -- never stored on the context. --- @@ -791,11 +791,11 @@ graph TD --- -## 14. Edge Cases +## 13. Edge Cases | Scenario | Behaviour | |----------|-----------| -| Multi-role user | A platform grant with `query:execute` reads as the platform user; otherwise the caller must resolve to exactly one org, or gets `403 no_single_org` (section 5.2) | +| Multi-role user | A platform grant with `query:execute` reads as the platform user. Otherwise the caller must resolve to exactly one org, or gets `403 no_single_org` (section 5.2) | | dfe-hyperdx unavailable | Non-fatal. Org CRUD succeeds. Sync retried in background | | ClickHouse unavailable | `/readyz` reports NotReady while its ping fails. Liveness is untouched, so the pod is not restarted. The CH RBAC reconcile retries in the background (section 6.2) | | Unknown OIDC group | No linked group with a role, so the sign-in is refused with `401 no_role_granted` and no account is created (section 2.1) | @@ -804,7 +804,7 @@ graph TD --- -## 15. Non-Goals +## 14. Non-Goals - Per-metric RBAC granularity (access is per-service, not per-metric) - Per-setting RBAC granularity (access is per-service config, not per-key) @@ -816,9 +816,9 @@ graph TD --- -## 17. References +## 15. References - [ClickHouse Custom Settings + Row Policy (Highlight)](https://www.highlight.io/blog/row-level-security) -- [API Key Prefix Pattern (Seam)](https://github.com/seamapi/prefixed-api-key) +- [API Key Prefix Pattern (seamapi/prefixed-api-key)](https://github.com/seamapi/prefixed-api-key) - [Multi-Tenant RBAC Design (WorkOS)](https://workos.com/blog/how-to-design-multi-tenant-rbac-saas) - [Envoy Gateway OIDC SecurityPolicy](https://gateway.envoyproxy.io/docs/tasks/security/oidc/) diff --git a/docs/control-plane/type-safety-sync.md b/docs/control-plane/type-safety-sync.md index 0e18339e1..5a41a4a73 100644 --- a/docs/control-plane/type-safety-sync.md +++ b/docs/control-plane/type-safety-sync.md @@ -1,12 +1,12 @@ -# Automagic type safety (dfe-engine → dfe-ui) +# Type safety sync (dfe-engine -> dfe-ui) dfe-engine is the source of truth for the HTTP API contract and the RBAC scope catalog, and `openapi.json` carries both. It is synced into **dfe-ui** so the frontend can use strict TypeScript types and scope constants without hand-maintaining duplicates. -## Goal +## What it covers -- **API types**: Routes, request bodies, and responses defined in FastAPI/Pydantic become TypeScript types in dfe-ui. -- **RBAC scopes**: Permission strings in `scope_constants.py` become the `RbacScope` enum in the spec, and so in dfe-ui. -- **Automation**: Merges to `main`/`master` that touch the spec open a PR in dfe-ui; frontend code can rely on compile-time checks against the latest engine contract. +- API types: Routes, request bodies, and responses defined in FastAPI/Pydantic become TypeScript types in dfe-ui. +- RBAC scopes: Permission strings in `scope_constants.py` become the `RbacScope` enum in the spec, and so in dfe-ui. +- Automation: Merges to `main`/`master` that touch the spec open a PR in dfe-ui. Frontend code can rely on compile-time checks against the latest engine contract. ## Generators in this repo @@ -25,7 +25,7 @@ uv run python scripts/generate_openapi.py The script loads the live FastAPI app and writes the OpenAPI 3 schema. That JSON is used for Prism mocks, contract checks, and (in dfe-ui) `openapi-typescript` code generation. -See also [UI-API-GUIDE.md](ui-api-guide.md) for local mock/typegen workflows. +See also [ui-api-guide.md](ui-api-guide.md) for local mock and typegen workflows. ### RBAC scope constants @@ -49,25 +49,25 @@ Note: Changing only Python API files without regenerating and committing the Ope ### What the job does 1. Checks out dfe-engine. -2. Clones **dfe-ui** (repo from `DFE_UI_REPO`, default `hyperi-io/dfe-ui`) using `DFE_UI_TOKEN`. -3. Copies into dfe-ui’s types package: - - `openapi/openapi.json` → `packages/dfe-engine-types/specs/openapi.json` - - `openapi/openapi.e2e.json` → `packages/dfe-engine-types/specs/openapi.e2e.json` +2. Mints a token from the GitHub App and clones **dfe-ui** (repo from `DFE_UI_REPO`, default `hyperi-io/dfe-ui`). +3. Copies into dfe-ui's types package: + - `openapi/openapi.json` -> `packages/dfe-engine-types/specs/openapi.json` + - `openapi/openapi.e2e.json` -> `packages/dfe-engine-types/specs/openapi.e2e.json` 4. Runs `yarn install --immutable` and `yarn generate` in `packages/dfe-engine-types`, then generates `types/e2e.d.ts` from `openapi.e2e.json` and exports it as `@repo/dfe-engine-types/e2e`. 5. Opens a PR in dfe-ui (closes older open sync PRs on the same branch naming pattern so only one sync PR is active). ### Secrets and configuration -- **`DFE_UI_TOKEN`** (required): PAT with permission to push branches and open/close PRs in dfe-ui. -- **`DFE_UI_REPO`** (optional): `owner/name` for the UI repo (default `hyperi-io/dfe-ui`). +- `vars.GH_APP_CLIENT_ID` and `secrets.GH_APP_PRIVATE_KEY` (required): the GitHub App whose installation token, scoped to dfe-ui with contents and pull-requests write, pushes the branch and opens and closes the PRs. +- `vars.DFE_UI_REPO` (optional): `owner/name` for the UI repo (default `hyperi-io/dfe-ui`). ## Developer workflow (dfe-engine) 1. Change API or scopes in Python. 2. Regenerate the specs: `uv run python scripts/generate_openapi.py`. 3. Commit the updated `openapi/openapi.json` and `openapi/openapi.e2e.json` together with your code change. -4. After merge to `main`/`master`, wait for **Sync dfe-engine-types** to open the dfe-ui PR; merge that PR to pick up types and scopes on the frontend. +4. After merge to `main`/`master`, wait for **Sync dfe-engine-types** to open the dfe-ui PR. Merge that PR to pick up types and scopes on the frontend. ## Developer workflow (dfe-ui) -Consume **`packages/dfe-engine-types`** (generated API types, `RbacScope` among them). Do not copy scope strings or paths by hand—import from the package so renames and new endpoints fail at compile time instead of at runtime. +Consume **`packages/dfe-engine-types`** (generated API types, `RbacScope` among them). Do not copy scope strings or paths by hand. Import from the package so renames and new endpoints fail at compile time instead of at runtime. diff --git a/docs/control-plane/ui-api-guide.md b/docs/control-plane/ui-api-guide.md index d90332773..e1e3d7529 100644 --- a/docs/control-plane/ui-api-guide.md +++ b/docs/control-plane/ui-api-guide.md @@ -20,13 +20,14 @@ docker compose -f docker-compose.dev.yaml up prism ## Authentication -All `/api/v1/*` endpoints require authentication. Three methods supported: +Every `/api/v1/*` endpoint but login and the OIDC sign-in routes needs a credential: | Method | Header | Use Case | |--------|--------|----------| | JWT Bearer | `Authorization: Bearer ` | UI login flow | | API Key | `X-API-Key: dfe_xxxx...` | Machine-to-machine | -| OIDC | `X-Oidc-Subject` (set by Envoy) | Production SSO | +| OIDC | The engine's own sign-in (`/api/v1/auth/oidc/{provider}/login`) mints the JWT above | Production SSO | +| Proxy headers | `X-Oidc-Subject`, read only with `auth.trust_proxy_auth_headers` | An engine reachable only through an OIDC proxy | ### Login Flow @@ -34,14 +35,14 @@ All `/api/v1/*` endpoints require authentication. Three methods supported: // POST /api/v1/auth/login const resp = await fetch('/api/v1/auth/login', { method: 'POST', - headers: { 'Content-Type': 'application/x-www-form-urlencoded' }, - body: new URLSearchParams({ username, password }), + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ username, password }), }); const { access_token } = await resp.json(); // Use: { Authorization: `Bearer ${access_token}` } -// GET /api/v1/auth/me — returns current user info + roles -// POST /api/v1/auth/refresh — refresh token +// GET /api/v1/auth/me -- returns current user info + roles +// POST /api/v1/auth/refresh -- refresh token ``` ## Error Handling @@ -74,7 +75,7 @@ interface FieldError { | 403 | `forbidden` | Insufficient permissions | | 404 | `not_found` | Resource doesn't exist | | 409 | `conflict` | Duplicate name or match conflict | -| 412 | `etag_mismatch` | `If-Match` is stale; `etag` carries the current one | +| 412 | `etag_mismatch` | `If-Match` is stale. `etag` carries the current one | | 413 | `payload_too_large` | Request body over `api.max_request_body_bytes` (4 MiB, `DFE_ENGINE_API_MAX_REQUEST_BODY_BYTES`), or over a file upload's own limit | | 422 | `validation_error` | Request body validation failed | | 500 | `internal_error` | Unexpected server error | @@ -107,7 +108,9 @@ const { data, fetchNextPage } = useInfiniteQuery({ A request that starts slow work answers 202 with an AIP-151 `Operation` and a `Location` to poll: `GET /api/v1/operations/{operation}` until `done`, then read `response` or `error` (a `Problem`). `metadata` carries the progress. -## API Surface (74 Endpoints) +## API surface + +The tables below cover the routes a UI uses most. `openapi/openapi.json` lists every operation. ### Auth & RBAC (`/api/v1/auth/*`) @@ -123,10 +126,10 @@ A request that starts slow work answers 202 with an AIP-151 `Operation` and a `L | GET | `/auth/roles/scopes` | RBAC permission scopes for role editors | | CRUD | `/auth/api-keys` | API key management | | CRUD | `/auth/oidc-providers` | OIDC provider configuration | -| POST | `/auth/oidc-providers/{name}/sync` | Force group sync; `status` says how it ended | +| POST | `/auth/oidc-providers/{name}/sync` | Force group sync. `status` says how it ended | | GET | `/auth/oidc-providers/{name}/test` | Test provider connectivity | -A group's `source_id` and `source_provider` link it to an IdP group; the group form sets both through `POST` and `PUT /auth/groups`. A login through that provider asserting `source_id` takes the group's roles: for a `token_claim` provider sending names (Okta, dex, Keycloak) it is the group name the IdP sends. A `source_id` without a provider answers `422 missing_source_provider`, a taken one `409 conflict`, a malformed one `422 invalid_source_id` and an unknown or overlong provider `422 invalid_source_provider`. A group name that is another group's `source_id` or a `source_id` that is another group's name answers `409 identifier_taken`. +A group's `source_id` and `source_provider` link it to an IdP group. The group form sets both through `POST` and `PUT /auth/groups`. A login through that provider asserting `source_id` takes the group's roles: for a `token_claim` provider sending names (Okta, dex, Keycloak) it is the group name the IdP sends. A `source_id` without a provider answers `422 missing_source_provider`, a taken one `409 conflict`, a malformed one `422 invalid_source_id` and an unknown or overlong provider `422 invalid_source_provider`. A group name that is another group's `source_id` or a `source_id` that is another group's name answers `409 identifier_taken`. A group sync answers 200 with `created`, `updated`, `total`, `groups_skipped`, `error`, `skipped` and `status`. Key on `status` (`ok`, `partial`, `error`, `not_configured`, `user_token` or `skipped`), never on the text of `error` or `skipped`, which are for display. @@ -138,12 +141,12 @@ A group sync answers 200 with `created`, `updated`, `total`, `groups_skipped`, ` | POST | `/sources` | Create source | | GET | `/sources/{name}` | Get source details | | GET | `/sources/{name}/versions/{version}` | Get one version snapshot | -| GET | `/sources/{name}/columns` | Composed schema columns for a source version (a `Page` of columns; `?version=` source id, default `deployed_version`) | -| POST | `/sources/{name}/build` | Build DDL from a source version (`?version=` source id; default `deployed_version`) | -| POST | `/sources/{name}/plan` | Dry-run deploy plan (`?version=` defaults to `current`; not persisted) | +| GET | `/sources/{name}/columns` | Composed schema columns for a source version (a `Page` of columns. `?version=` source id, default `deployed_version`) | +| POST | `/sources/{name}/build` | Build DDL from a source version (`?version=` source id. Default `deployed_version`) | +| POST | `/sources/{name}/plan` | Dry-run deploy plan (`?version=` defaults to `current`. Not persisted) | | POST | `/sources/{name}/deploy` | Apply plan DDL to ClickHouse and set `deployed_version` (`?version=` defaults to `current`) | | PUT | `/sources/{name}` | Update source | -| PATCH | `/sources/{name}` | Set lifecycle state (`state: active\|dormant\|disabled`, or compat `enabled`); no new version; `409 match_conflict` if activating duplicates another non-disabled source's match | +| PATCH | `/sources/{name}` | Set lifecycle state (`state: active\|dormant\|disabled`, or compat `enabled`). No new version. `409 match_conflict` if activating duplicates another non-disabled source's match | | DELETE | `/sources/{name}` | Delete source | | POST | `/sources/bulk` | Bulk action (`enable` / `disable` / `dormant` / `delete` / `deploy`) on named sources | | POST | `/sources/seed` | Seed defaults | @@ -184,7 +187,6 @@ Schema-less service config for Rust services (receiver, loader, archiver, etc.) | GET | `/queries/views/{label}` | View definition + parameters | | POST | `/queries/views/{label}/execute` | Execute view -> JSON | | POST | `/queries/raw` | Raw read-only query -> JSON. Needs `raw_query:execute`, which only `admin` holds | -| GET | `/queries/health` | Query engine health | View execution returns **JSON** (`QueryResponse`) with fields: `rows`, `columns`, `row_count`, `query_duration_ms`, `has_more`, @@ -219,25 +221,10 @@ Long-running operations, such as pipeline builds and synthetic-data streams. | Method | Path | Purpose | |--------|------|---------| | GET | `/operations` | List operations (filter by `kind`) | -| GET | `/operations/{id}` | Get an operation | -| POST | `/operations/{id}/cancel` | Cancel a running operation | -| GET | `/operations/{id}/stream` | SSE progress stream | - -#### SSE Stream Events +| GET | `/operations/{operation}` | Get an operation. Poll it until `done` | +| POST | `/operations/{operation}/cancel` | Cancel a running operation | -```typescript -// EventSource: GET /api/v1/operations/{id}/stream -const es = new EventSource(`/api/v1/operations/${id}/stream`); -es.addEventListener('progress', (e) => { - const { status, progress, message } = JSON.parse(e.data); - // Update progress bar -}); -es.addEventListener('complete', (e) => { - const { status, result, error } = JSON.parse(e.data); - // Task finished — close stream - es.close(); -}); -``` +There is no progress stream: poll the operation, and read `metadata` for progress. ### Other @@ -277,7 +264,7 @@ through the guided flow below - never through free-text fields. Invoke body (only when the action declares params): `{"params": {"level": "2x"}}`. Each param is CONSTRAINED - enum params carry their full `values` list, numerics carry `min`/`max` - so render enum params -as a select and numerics as a bounded input; never a free text box. A +as a select and numerics as a bounded input. Never a free text box. A violation returns 422 `invalid_params`. The flow to render: pick action -> fill params from the constraints -> dry-run @@ -291,7 +278,7 @@ forge is configured to open the PR. | Method | Path | Purpose | |--------|------|---------| | POST | `/governance/admin/actions/validate` | Check a def: EVERY violation + would-be diff, no commit | -| POST | `/governance/admin/actions` | Define (`governance:write`); `required_action` optional | +| POST | `/governance/admin/actions` | Define (`governance:write`). `required_action` optional | | DELETE | `/governance/admin/actions/{name}` | Remove a def | | GET | `/gitops/classes` | Class select (filter on `action_writable`) | | GET | `/gitops/classes/{cls}/resources` | Resource select (gated by that class's `:read`) | @@ -300,7 +287,7 @@ forge is configured to open the PR. Wizard flow: name/description -> per change: class select -> resource select -> path select (show the current value beside each path) -> value input -> validate (render `errors[]` + `diff[]`) -> save. Leave `required_action` -empty - the server derives `action:invoke:`; expose an override only +empty - the server derives `action:invoke:`. Expose an override only behind an advanced toggle. ### Closed sets in the contract (`x-dfe-enum-source`) @@ -309,7 +296,7 @@ House rule: if the server will reject values outside a set, the contract exposes the set. Three mechanisms, in order of preference: 1. **Static set** - a schema enum (e.g. repository `scope`). Generated types - already give a union; render a select. + already give a union. Render a select. 2. **Dynamic set** - the schema field carries an `x-dfe-enum-source` annotation naming the endpoint that enumerates its legal values: @@ -320,7 +307,7 @@ exposes the set. Three mechanisms, in order of preference: `params` maps `{placeholder}` segments in the endpoint template to SIBLING fields of the same object (e.g. the `path` field's source depends on the chosen `cls` and `name`). `value_key` names the option field when the - endpoint returns objects; absent means the items are plain strings. Build + endpoint returns objects. Absent means the items are plain strings. Build ONE generic hook that resolves an annotated field to a select fed by a TanStack query - every annotated field then renders correctly for free. 3. **Derived value** - the field is optional and the server fills a documented @@ -413,7 +400,7 @@ A caller whose grant reads every org sees the whole deployment and may name any Events per second over more than one tenant id (several orgs, or an org that declares several) is the busiest minute of their sum. The minute table keeps 7 days, so days older than that have no entry. -The CLI generates `dfe usage list` for `/usage` and `dfe usage overview list` for `/usage/overview`. The Observe "Overview" dashboard (`src/dfe_engine/hyperdx/dashboards/dfe-overview.json`) reads the same rollup for the whole deployment, without going through the API. +The CLI generates `dfe usage get` for `/usage` and `dfe usage overview list` for `/usage/overview`. The Observe "Overview" dashboard (`src/dfe_engine/hyperdx/dashboards/dfe-overview.json`) reads the same rollup for the whole deployment, without going through the API. ## Repository (UI preferences + small objects) @@ -473,18 +460,18 @@ the object endpoints below (namespace `preferences`, key `default`). |--------|------|---------| | GET | `/repository/objects/{scope}/{scope_id}/{namespace}` | List metadata (key, content_type, size, updated_by, updated_at, etag) | | GET | `/repository/objects/{scope}/{scope_id}/{namespace}/{key}` | Raw bytes with stored `Content-Type` + `ETag` headers | -| PUT | `/repository/objects/{scope}/{scope_id}/{namespace}/{key}` | Store raw body (`Content-Type` header is kept; default `application/octet-stream`) | -| DELETE | `/repository/objects/{scope}/{scope_id}/{namespace}/{key}` | Delete (204; 404 if absent) | +| PUT | `/repository/objects/{scope}/{scope_id}/{namespace}/{key}` | Store raw body (`Content-Type` header is kept. Default `application/octet-stream`) | +| DELETE | `/repository/objects/{scope}/{scope_id}/{namespace}/{key}` | Delete (204. 404 if absent) | Path rules: `scope` is one of `system|org|group|user` (422 otherwise). For `system` the `scope_id` MUST be `-`. `namespace` and `key` match `^[a-zA-Z0-9][a-zA-Z0-9._-]{0,127}$` (422 otherwise). -Access: any authenticated user reads the system layer; members read their -own org/group layers and their own user layer; everything else needs the +Access: any authenticated user reads the system layer. Members read their +own org/group layers and their own user layer. Everything else needs the `repository:read` / `repository:write` RBAC actions at the matching scope (admins system-wide, org-wildcard admins within their org). Reads you -cannot see return 404 (existence stays hidden); forbidden writes return 403. +cannot see return 404 (existence stays hidden). Forbidden writes return 403. ### ETag semantics @@ -525,28 +512,22 @@ type Source = paths['/api/v1/sources/{name}']['get']['responses']['200']['conten ## Mock Server (Prism) -```yaml -# docker-compose.dev.yaml -services: - prism: - image: stoplight/prism:4 - command: mock -h 0.0.0.0 /spec/openapi.json - volumes: - - ./openapi:/spec - ports: - - "4010:4010" +`docker-compose.dev.yaml` runs Prism, digest-pinned, over `./openapi`: + +```bash +docker compose -f docker-compose.dev.yaml up prism ``` -Prism generates realistic mock responses from the OpenAPI spec. -Point the UI at `http://localhost:4010` during development. +Prism generates mock responses from the OpenAPI spec. Point the UI at +`http://localhost:4010` during development. ## Contract Drift Detection -CI validates the committed spec matches the running app: +`tests/unit/test_openapi_spec_committed.py` fails CI when the committed spec, or +the route map built from it, differs from what the app generates. Regenerate and +commit after a route or model change: ```bash uv run python scripts/generate_openapi.py git diff --exit-code openapi/openapi.json openapi/openapi.e2e.json ``` - -If this fails, the spec needs regenerating after route/model changes. diff --git a/docs/data-evolution-acceptance.md b/docs/data-evolution-acceptance.md index fa2912d04..3e3893c7f 100644 --- a/docs/data-evolution-acceptance.md +++ b/docs/data-evolution-acceptance.md @@ -36,16 +36,16 @@ for its number. suite goes red the day that issue is fixed, which is how it gets removed. `part` says the test covers less than the row claims. 2.2 asserts the DDL is -rendered and the table created; that the receiver reloads in place is the flow +rendered and the table created. That the receiver reloads in place is the flow suite's routing check, and that records route by it is 2.3. 2.6 applies the ALTER directly rather than through a derived-schema edit, which is what 2.4 and 2.5 cover. 1.2 re-imports into the SAME deployment under a second path, which proves the bundle carries what the import needs but not that it lands in a deployment that has never seen the schema -- that needs a second stack. -No step is `none` any more. 2.8 drives the source that already has a transform instance rather than creating a second one, because Compose declares one service per transform app and binds it at deploy -- so it proves a different ENGINE on the same corpus, not a second binding. It also reads that source's match rule off the API instead of assuming one, since the source belongs to the deployment. +2.8 drives the source that already has a transform instance, because Compose declares one service per transform app and binds it at deploy. So it proves a different engine on the same corpus, never a second binding. It reads that source's match rule off the API, since the source belongs to the deployment. -Stage 3 was listed here as unable to pass at all, on the grounds that a derived schema had no way to say "stop populating `_json`". It has one: `capture_json: false` with `capture_raw: false`, which the engine compiles to dfe-loader's `extracted_only`. 3.1 passes. 3.2 and 3.3 are xfail on dfe-engine#513 -- the deploy writes `table_capture_modes` and reports `restart_required: []`, but the loader logs no config reload, so the mode only takes effect once it is restarted by hand. +A derived schema stops population with `capture_json: false` and `capture_raw: false`, which the engine compiles to dfe-loader's `extracted_only`. 3.2 and 3.3 are xfail on dfe-engine#513: the deploy writes `table_capture_modes` and reports `restart_required: []`, but the loader logs no config reload, so the mode takes effect only once it is restarted by hand. ## Three carry the weight @@ -71,7 +71,7 @@ One corpus of raw Cisco syslog feeds all three transforms -- transform-vrl, transform-vector and transform-elastic -- and each must produce the same typed columns. -Two things to settle before building: +Two constraints on that corpus: - **Input format.** Our VRL consumes the DFE 2.1 Kafka format (`{message, tags, timestamp}`) with a raw syslog line in `message`. @@ -84,6 +84,6 @@ Two things to settle before building: The VRL also needs the `timezones.csv` enrichment table, and optionally `._conf.tz_offset` and `._conf.tz_map`. -That is the target. 2.7 posts the Cisco IOS module alone today, through whichever -transform app `DFE_E2E_TRANSFORM` names; widening it to all three at once is what -turns it into the fixture above. +2.7 posts the Cisco IOS module alone, through whichever transform app +`DFE_E2E_TRANSFORM` names. Widening it to all three at once turns it into the +fixture above. diff --git a/docs/data-evolution.md b/docs/data-evolution.md index 00641a8af..19b939933 100644 --- a/docs/data-evolution.md +++ b/docs/data-evolution.md @@ -74,18 +74,18 @@ common sources, starting with Elastic in 2.2.0. What makes it different from traditional ETL and from a data lake's schema work: -- **Cover what you USE, not everything you receive.** You are never forced to +- Cover what you USE, not everything you receive. You are never forced to model the full breadth, because `_json` is still there as the slower fallback for anything off the hot path. Partial is a valid answer. -- **Written at the primitive level**, in the types the engine can actually +- Written at the primitive level, in the types the engine can actually store. -- **Written at the use-case level.** A column declares the question it answers +- Written at the use-case level. A column declares the question it answers -- `dimension`, `range`, `word_search` -- and DFE picks the ClickHouse index. The label never names the engine primitive. -- **Readable by the SME who owns the data.** Whoever understands the source and +- Readable by the SME who owns the data. Whoever understands the source and the workflow can write one. It does not require a data engineer, a ClickHouse expert or a Kafka expert. -- **Narrowable to ClickHouse specifics** where you do want that control. +- Narrowable to ClickHouse specifics where you do want that control. A large meta schema is normally NOT deployed as it stands. It is the base a derived schema selects from. @@ -97,7 +97,7 @@ but that is unusual and worth a second look when you see it. ```mermaid flowchart TB - HEADER["common header
_json, _timestamp, timestamp, _tags
partitioning, TTL"] + HEADER["common header
_json, _timestamp, _timestamp_load, _tags
partitioning, TTL"] META["meta schema
full breadth, rarely deployed as-is"] DERIVED["derived schema
selection + one index type per field"] TABLE[("deployed table")] @@ -122,11 +122,11 @@ DFE machinery depends on: - `_json` is the fallback that makes stage 0 work and keeps partial meta schemas honest. -- `_timestamp` and `timestamp` drive partitioning, retention and every - time-bounded query. +- `_timestamp_load` drives partitioning and retention, and `_timestamp`, the + event time read from the record's `timestamp`, every time-bounded query. - `_tags` carries contextual divisions and data routing. -Remove a header field and something concrete breaks: no `_timestamp` and the +Remove a header field and something concrete breaks: no `_timestamp_load` and the table cannot partition or expire, no `_json` and a partial meta schema silently loses every field it did not model. That is why the header is enforced rather than suggested. @@ -142,8 +142,8 @@ table. runs against EVERY inbound record, so the CEL Fast profile refuses regex, iteration and time functions there. -**What is mandatory.** The source name, which is both the topic and the table -name, and the routing condition when the origin is the receiver. A source +**What is mandatory.** The source name, which names the table and the topics +(`_land`, plus `_load` after a transform), and the routing condition when the origin is the receiver. A source landing anywhere other than `main` also needs a meta schema or a derived schema, because that is what defines its table. @@ -202,12 +202,12 @@ net was down, is gone rather than slow. It is a maturity statement. Never a default. Nothing pushes you here and most tables should never arrive. -**The data plane already does this.** dfe-loader's `MetadataConfig` carries -`capture_json` / `capture_raw` with `disable_json_tables` and -`disable_raw_tables` for per-table control, and the column directive framework -reaches the same end with `@skip` on `_json`. The control plane says it through -a derived schema's `capture_json` and `capture_raw`, which the engine compiles -into the loader's per-table `table_capture_modes`. +**The data plane already does this.** dfe-loader's `metadata.capture_mode` +(`full`, `json_only`, `raw_only`, `extracted_only`) decides what it writes, with +`metadata.table_capture_modes` for per-table control, and the column directive +framework reaches the same end with `@skip` on `_json`. The control plane says it +through a derived schema's `capture_json` and `capture_raw`, which the engine +compiles into the loader's per-table `table_capture_modes`. ### Column comments carry loader instructions diff --git a/docs/data-plane/README.md b/docs/data-plane/README.md index b3223e2ea..d611a08dc 100644 --- a/docs/data-plane/README.md +++ b/docs/data-plane/README.md @@ -1,8 +1,8 @@ # Data plane The close-coupled source - ingest - schema - transform path. One source -definition carries routing, schema binding, and field-map bindings together; -the receiver, loader, views, hunts, and query API all read the same +definition carries routing, schema binding, and field-map bindings together. +The receiver, loader, views, hunts, and query API all read the same definition, so there is no separate ETL config to drift. The system map is [../architecture.md](../architecture.md). @@ -19,7 +19,7 @@ flowchart LR | [source.md](source.md) | Source as the top-level data abstraction | | [source-definition.md](source-definition.md) | the source YAML field by field, and what a write is validated against | | [source-registry.md](source-registry.md) | states, storage backends, CRUD, topics on deploy, the API surface | -| [source-flow.md](source-flow.md) | INPUT, optional TRANSFORM, OUTPUT on the Kafka or gRPC transport; what compiles from a source; how a change reaches a pod | +| [source-flow.md](source-flow.md) | INPUT, optional TRANSFORM, OUTPUT on the Kafka or gRPC transport. What compiles from a source. How a change reaches a pod | | [source-routing.md](source-routing.md) | the exact receiver and loader config a source compiles into | | [topic-contract.md](topic-contract.md) | the Kafka topics a source implies: ensure, converge, status, remove, and the dials | | [external-pipelines.md](external-pipelines.md) | the customer's own Vector, Fluentd or Logstash as a source's origin or transform on Kafka | diff --git a/docs/data-plane/expressions-cel.md b/docs/data-plane/expressions-cel.md index d52103972..b97488293 100644 --- a/docs/data-plane/expressions-cel.md +++ b/docs/data-plane/expressions-cel.md @@ -1,4 +1,4 @@ -# DFE Expression Language — CEL +# DFE Expression Language -- CEL DFE uses **CEL (Common Expression Language)** for conditions across the Python engine, the Rust services and the CLI. CEL is not Turing-complete: no loops, no recursion, no side effects, so every expression terminates. An expression is compiled once and evaluated against every row. @@ -6,7 +6,7 @@ DFE uses **CEL (Common Expression Language)** for conditions across the Python e | Language | Package | Notes | |----------|---------|-------| -| Rust | `cel` (renamed from `cel-interpreter`) | Native implementation, through `scalo::expression` | +| Rust | `cel` | Native implementation, through `scalo::expression` | | Python | `common-expression-language` | PyO3 bindings, through `scalo.expression` | scalo-rs's `Cargo.toml` and scalo-py's `pyproject.toml` carry the versions. @@ -24,7 +24,7 @@ scalo-rs's `Cargo.toml` and scalo-py's `pyproject.toml` carry the versions. | System | Reason | |--------|--------| -| `@directive: body` DSL (`@source`, `@generated`, etc.) | Schema metadata — column mapping instructions, not boolean conditions | +| `@directive: body` DSL (`@source`, `@generated`, etc.) | Schema metadata -- column mapping instructions, not boolean conditions | | ClickHouse SQL in `@generated` expressions | Runs in ClickHouse, not our evaluator | | Hunt SQL queries / rule_rewriter | ClickHouse-native SQL | | Duration strings (`"1h"`, `"30m"`) | Simple config values, not expressions | @@ -64,8 +64,8 @@ DFE uses a **subset** of CEL. Expressions outside this profile are rejected at v | Feature | Reason | |---------|--------| -| `map()`, `filter()`, `exists()`, `all()` | Per-element iteration — unpredictable performance on large lists | -| `timestamp()`, `duration()` | ClickHouse handles time natively — no need to duplicate | +| `map()`, `filter()`, `exists()`, `all()` | Per-element iteration -- unpredictable performance on large lists | +| `timestamp()`, `duration()` | ClickHouse handles time natively -- no need to duplicate | | Macros | Complexity with no benefit for DFE use cases | --- @@ -79,7 +79,7 @@ DFE uses a **subset** of CEL. Expressions outside this profile are rejected at v | `string` | `str` | `String` | `String` | | `bool` | `bool` | `bool` | `Bool` | | `list` | `list` | `Vec` | `Array` | -| `map` | `dict` | `HashMap` | — | +| `map` | `dict` | `HashMap` | -- | | `null` | `None` | `Option::None` | `Nullable` | --- @@ -135,35 +135,27 @@ This enables expressions written once in YAML to work both as in-memory evaluati ## Implementation Architecture -```text -┌─────────────────────────────────────────────┐ -│ DFE Expression Profile │ -│ (subset of CEL, documented here) │ -└─────────┬───────────────────┬───────────────┘ - │ │ - ┌─────▼──────┐ ┌──────▼──────┐ - │ scalo-py │ │ scalo-rs │ - │ expression/ │ │ expression/ │ - │ │ │ │ - │ wraps: │ │ wraps: │ - │ common- │ │ cel │ - │ expression- │ │ crate │ - │ language │ │ │ - └─────┬──────┘ └──────┬──────┘ - │ │ - ┌─────▼──────┐ ┌──────▼──────┐ - │ dfe-engine │ │ dfe-loader │ - │ hunts/ │ │ transformer │ - │ alert.py │ │ .rs │ - └─────────────┘ └─────────────┘ +```mermaid +flowchart TB + PROF["DFE expression profile
a subset of CEL"] + PY["scalo-py scalo.expression
wraps common-expression-language"] + RS["scalo-rs scalo::expression
wraps the cel crate"] + ENG["dfe-engine
hunts/alert.py, rule filters"] + LDR["dfe-loader
src/transform/computed.rs"] + PROF --> PY & RS + PY --> ENG + RS --> LDR ``` -Both wrappers enforce the same DFE profile — identical allowed operators, functions, and types. The Rust crate is the single source of truth for parsing and evaluation semantics. +Both wrappers enforce the same DFE profile: the same allowed operators, functions and types. + +### API surface -### API Surface (Both Languages) +scalo-py's names (`scalo.expression`): ```text -compile(expr) → Program # Parse + validate + compile to AST -evaluate(expr, data) → Any # One-shot convenience (compile + execute) -validate(expr) → list[str] # Syntax + profile check, returns errors for UI +compile_expression(expr) -> Program # Parse + validate + compile +evaluate(expr, data) -> Any # One-shot convenience (compile + execute) +validate(expr) -> list[str] # Syntax + profile check, returns errors for UI +transpile_to_clickhouse(expr) -> str # The SQL pushdown below ``` diff --git a/docs/data-plane/hunt-match-delivery.md b/docs/data-plane/hunt-match-delivery.md index 19f95ae70..d9ad3bb6a 100644 --- a/docs/data-plane/hunt-match-delivery.md +++ b/docs/data-plane/hunt-match-delivery.md @@ -57,4 +57,4 @@ After a fire commits its rows and its watermark, the runner checks the hunt's `a - The runner reads each destination by name from the secrets backend its own `settings.secrets` names, which has to be the engine's. On `openbao` and `aws` it needs read access to `alert-destinations/*` and nothing of the engine's volume. - The `file` directory sits on the engine's own volume. On docker the runner mounts `/alert-destinations` read-only at the same path, and the engine creates that directory at startup so the mount finds it. On Kubernetes no runner pod can mount the engine's volume, so `hunts.alert_destinations_secret` (`DFE_ENGINE_HUNTS_ALERT_DESTINATIONS_SECRET`) names a Secret the engine also writes each destination into, one key per name. The engine needs `get` and `update` on that one Secret, and the runner mounts it read-only at `/alert-destinations` with the `file` provider. Every engine startup brings the Secret back into step, so one recreated empty is refilled. -There is no cooldown yet: a hunt that matches on every fire sends a message on every fire. +There is no cooldown: a hunt that matches on every fire sends a message on every fire. diff --git a/docs/data-plane/hunt-runner-scaling.md b/docs/data-plane/hunt-runner-scaling.md index 0d7b6ec35..39ac025e3 100644 --- a/docs/data-plane/hunt-runner-scaling.md +++ b/docs/data-plane/hunt-runner-scaling.md @@ -40,10 +40,10 @@ otherwise lives only in a running worker's memory (loaded from the gitops hunt configs). At zero workers there is nobody to ask. So the schedule is MATERIALISED into a small ClickHouse table, `hunt_schedule`, at -config-deploy time. The scaler then reads it directly. Three tables already exist -for coordination (`ch_coordinator.py`): `hunt_lease`, `hunt_watermark`, -`hunt_state`. `hunt_schedule` is the fourth, and the only one written outside the -worker loop. +config-deploy time. The scaler then reads it directly. The runner's other tables +(`src/dfe_engine/hunts/ch_coordinator.py`) are `hunt_lease`, `hunt_watermark`, +`hunt_state`, `hunt_run` and `hunt_runner_heartbeat`. `hunt_schedule` is the only +one written outside the worker loop. ```text hunt_schedule(hunt_id, interval_seconds, phase_offset, enabled, updated) @@ -88,23 +88,23 @@ Autoscaling (when the BETA is enabled) is only safe if KEDA's view of "due" never diverges from a worker's. It cannot, because both compute the same thing from the same inputs: -1. **Parity, tested.** The scaler predicate equals `runner.tick`'s own gate (via +1. Parity, tested. The scaler predicate equals `runner.tick`'s own gate (via `schedule.latest_fire` + the watermark + lease checks) across a full matrix of (hunt_id, interval, now, watermark, lease), and the SQL implements that predicate against real ClickHouse - counting exactly the runnable hunts, excluding tombstoned / future-watermark / leased hunts, and returning 0 when every hunt is caught up. -2. **Single source of truth for the offset.** The materialiser computes +2. Single source of truth for the offset. The materialiser computes `phase_offset` with the same `schedule.phase_offset` the worker uses, so the two can never drift. Change the offset algorithm once and both move together. -3. **Gate on the signal, not a timer.** KEDA polls the backlog query (via the +3. Gate on the signal, not a timer. KEDA polls the backlog query (via the shim - a real readiness signal), it does not race a clock. Worst-case wake latency is `pollingInterval + pod-start`. For hunts (minute-plus intervals) that is immaterial. This is the "gate on the real condition" rule from the testing standard applied to autoscaling. -4. **Self-healing.** If a worker claims a hunt then dies before writing the watermark, its lease expires (`lease_seconds`, default 300) and the fire is still owed (`watermark < fire`), so the scaler keeps >=1 worker and the hunt is reclaimed. Nothing double-runs (a hunt run twice writes and notifies twice): a claim stands only if acknowledged within a settle of its free read and still the latest row after it. A later claim is never released, as it may sit over the winner, but it holds nothing, so the next poll claims again. -5. **No thundering herd.** The phase offset spreads same-interval hunts across the +4. Self-healing. If a worker claims a hunt then dies before writing the watermark, its lease expires (`lease_seconds`, default 300) and the fire is still owed (`watermark < fire`), so the scaler keeps >=1 worker and the hunt is reclaimed. Nothing double-runs (a hunt run twice writes and notifies twice): a claim stands only if acknowledged within a settle of its free read and still the latest row after it. A later claim is never released, as it may sit over the winner, but it holds nothing, so the next poll claims again. +5. No thundering herd. The phase offset spreads same-interval hunts across the interval, so `due_count` rises gradually and KEDA scales out smoothly rather than all-at-once at the boundary. The offset falls in the first `hunts.phase_spread_fraction` of the interval (default 0.8, env @@ -112,7 +112,9 @@ thing from the same inputs: the materialised schedule all read it, so republish the schedule (`dfe-hunt-runner materialise`) after changing it. -Bounded catch-up: a fire stays owed until its watermark is written, so a worker +## Bounded catch-up + +A fire stays owed until its watermark is written, so a worker that ticks inside `[boundary, boundary+offset)` runs the previous interval's fire if nothing ran it, and the scaler counts it the same way. Without that, a hunt whose offset sits between two poll ticks (a 46s offset polled at :00/:15/:30/:45) @@ -125,14 +127,14 @@ or a resident worker. This is the point of the pull-based design. Once deployed, the running system needs only ClickHouse: -- **Workers** read hunt defs from the gitops config (mounted / cloned), coordinate - through the four CH tables, and execute queries against CH. No engine call on the +- Workers read hunt defs from the gitops config (mounted / cloned), coordinate + through the coordination tables, and execute queries against CH. No engine call on the hot path. -- **KEDA** reads `hunt_schedule` + the coordination tables from CH via the +- KEDA reads `hunt_schedule` + the coordination tables from CH via the `dfe-keda-shim` - a separate, distinctly-labelled pod running the SAME engine container image under a different entrypoint (`dfe-keda-shim run`, not the live `dfe-engine` API service). No call to the running engine API. -- **The engine's only role is at CONFIG-DEPLOY time**, not runtime: materialise +- The engine's only role is at CONFIG-DEPLOY time, not runtime: materialise `hunt_schedule` when the hunt config changes, with the one-shot, idempotent `dfe-hunt-runner materialise`. The charts do not run it (see below). @@ -146,8 +148,8 @@ only ClickHouse is guaranteed present. ### Coordination needs a SINGLE LOGICAL ClickHouse (scale-tier requirement) -The four coordination tables are the shared source of truth every worker and KEDA -read/write. That only works if all of them see the SAME tables. On the **scale +The coordination tables are the shared source of truth every worker and KEDA +read and write. That only works if all of them see the SAME tables. On the **scale tier's multi-node cluster behind a round-robin load balancer**, a plain `ReplacingMergeTree` (and a plain `CREATE DATABASE`) would land on one node, so workers whose connections land on other nodes would see an empty or absent table and @@ -224,7 +226,9 @@ spec: valueLocation: "value" ``` -**The fail-safe** (`dfe_engine/keda_shim/shim.py`, `QueryShim.run`): every query +## The shim's fail-safe + +**The fail-safe** (`src/dfe_engine/keda_shim/shim.py`, `QueryShim.run`): every query caches its last-good integer per (query, params). On ANY failure - CH unreachable, timeout, a bad query - the shim returns the CACHED value, so KEDA sees no change and the pod count FREEZES at current, so it can never run @@ -239,7 +243,7 @@ Other notes: the shim is a separate, distinctly-labelled pod image under a different entrypoint - it is NOT the live engine API, so hunt-runner scaling still has no dependency on the engine service being up. Queries (including the backlog query's clamp/cold-hold/timeout) are -config-driven (`keda_shim/queries.yaml` + an optional mounted override), so a +config-driven (`src/dfe_engine/keda_shim/queries.yaml` + an optional mounted override), so a schema change or a tuning change is a config edit, not a shim rebuild. ### Proving it: the artificial scale test @@ -268,18 +272,18 @@ submit and coordinate concurrently and reliably), not CPU/memory - and the large majority of deployments need exactly one pod. So the default is deliberately simple: -- **`huntRunner.enabled` (default `true` on every tier):** deploys `dfe-hunt-runner` as ONE steady pod +- `huntRunner.enabled` (default `true` on every tier): deploys `dfe-hunt-runner` as ONE steady pod (`replicas: 1`, no ScaledObject at all). This is the resting state for the large majority of deployments - a single always-on pod, not scale-to-zero. -- **`huntRunner.autoscaling.enabled` (default `false`, BETA, keep it off, - see the top of this page):** opts into the ScaledObject described above. `replicas` is then omitted from the +- `huntRunner.autoscaling.enabled` (default `false`, BETA, keep it off, + see the top of this page): opts into the ScaledObject described above. `replicas` is then omitted from the Deployment (KEDA owns the count) and the pod count floats between `minReplicaCount` (default **1**, not 0 - set it to `0` explicitly via values to additionally opt into scale-to-zero) and `maxReplicaCount` (default 5), driven by the due backlog. Work still divvies up safely across N pods via the existing `hunt_lease` mutual exclusion - KEDA picks the pod count, the leases decide who does what, so nothing double-runs. -- **Disabled (operator choice):** set `huntRunner.enabled: false` in the +- Disabled (operator choice): set `huntRunner.enabled: false` in the deploy overlay. No pod and no ScaledObject are rendered, so hunts do not fire. Re-enable by restoring the dial - the schedule + watermarks are still in ClickHouse, so the runner resumes cleanly from the last committed window. diff --git a/docs/data-plane/query-api-python.md b/docs/data-plane/query-api-python.md index f2fbd99ad..484d2f270 100644 --- a/docs/data-plane/query-api-python.md +++ b/docs/data-plane/query-api-python.md @@ -6,10 +6,10 @@ ## Overview -- **No raw SQL from clients** - a view is referenced by label, and its SQL is a ClickHouse parameterised view deployed server-side -- **Mandatory tenant isolation** - `org_id` injected from the org the session's account or API key is bound to, cannot be overridden -- **Scope-based access control** - `query:read` browses the views and `query:execute` runs them -- **Native JSON responses** - dict rows via clickhouse-connect, no Arrow/pyarrow dependency +- No raw SQL from clients - a view is referenced by label, and its SQL is a ClickHouse parameterised view deployed server-side +- Mandatory tenant isolation - `org_id` injected from the org the session's account or API key is bound to, cannot be overridden +- Scope-based access control - `query:read` browses the views and `query:execute` runs them +- Native JSON responses - dict rows via clickhouse-connect, no Arrow/pyarrow dependency --- @@ -51,12 +51,16 @@ df.groupby("event_type").count() ### HTTP Mode -HTTP mode posts to `/api/v1/queries/views/{label}/execute` and sends no credentials, so it is answered only by an engine with auth disabled (a dev `DFE_ENGINE_ENV`): +HTTP mode posts to `/api/v1/queries/views/{label}/execute` and sends the `credential` it is given the way the `dfe` CLI does: an API key in `X-API-Key` (`api_key_header`), a token as `Authorization: Bearer`. With none it sends nothing, which only an engine with auth disabled (a dev `DFE_ENGINE_ENV`) answers: ```python from dfe_engine.query import QueryClient +from scalo.cli.openapi import Credential -client = QueryClient(base_url="http://localhost:8000") +client = QueryClient( + base_url="http://localhost:8000", + credential=Credential(account="ci", api_key="dfe_ak_..."), +) result = client.query( "analytics/user_activity", params={"event_types": ["login"]}, @@ -91,6 +95,9 @@ class QueryClient: base_url: str | None = None, # API URL for HTTP mode direct: bool = False, # Use in-process execution timeout_seconds: int = 30, # Default query timeout + *, + credential: Credential | None = None, # scalo.cli.openapi: a token or an API key + api_key_header: str = "X-API-Key", ): ... ``` @@ -128,7 +135,7 @@ result = client.query( | `query_label` | `str` | Query identifier (e.g., `"analytics/user_activity"`) | | `params` | `dict` | The view's parameters. Undeclared names are dropped, and values are not validated | | `limit` | `int` | Maximum rows (clamped to query/global max) | -| `offset` | `int` | Skip first N rows (offset-based pagination); not combinable with `after_key` | +| `offset` | `int` | Skip first N rows (offset-based pagination). Not combinable with `after_key` | | `order_by` | `str` | Column to order and page by (keyset pagination) | | `order_dir` | `str` | Sort direction: `"asc"` or `"desc"` (default: `"asc"`) | | `after_key` | `str \| int \| float` | `order_by` value of the previous page's last row | @@ -188,7 +195,7 @@ Returned by `query()`. | `num_rows` | `int` | Number of rows | | `num_columns` | `int` | Number of columns | | `metadata` | `QueryMetadata` | Execution metadata | -| `explain` | `ExplainPlan \| None` | Always `None` from `query()`; the client requests no EXPLAIN | +| `explain` | `ExplainPlan \| None` | Always `None` from `query()`. The client requests no EXPLAIN | ### QueryMetadata diff --git a/docs/data-plane/query-api.md b/docs/data-plane/query-api.md index 9462debb2..7cf882c34 100644 --- a/docs/data-plane/query-api.md +++ b/docs/data-plane/query-api.md @@ -6,11 +6,11 @@ A label-based query interface over ClickHouse parameterised views, with multi-te ## Overview -- **No raw SQL from clients** - a view is referenced by label, and its SQL is a ClickHouse parameterised view deployed server-side -- **Mandatory tenant isolation** - `org_id` injected from the org the session's account or API key is bound to, cannot be overridden -- **Scope-based access control** - `query:read` browses the views and `query:execute` runs them -- **Declared parameters only** - the executor passes on only the parameters the view declares, unvalidated, and ClickHouse checks their types -- **Native JSON responses** - dict rows via clickhouse-connect, no intermediate wire format +- No raw SQL from clients - a view is referenced by label, and its SQL is a ClickHouse parameterised view deployed server-side +- Mandatory tenant isolation - `org_id` injected from the org the session's account or API key is bound to, cannot be overridden +- Scope-based access control - `query:read` browses the views and `query:execute` runs them +- Declared parameters only - the executor passes on only the parameters the view declares, unvalidated, and ClickHouse checks their types +- Native JSON responses - dict rows via clickhouse-connect, no intermediate wire format ### Supported Datasources @@ -411,15 +411,14 @@ A failure is an RFC 9457 problem response with a stable `code`: Two distinct things, one implemented, one wire-format-only: -- **View catalog cache (implemented).** The `ViewCatalog` discovers view +- View catalog cache (implemented): the `ViewCatalog` discovers view definitions from ClickHouse and caches them with a TTL (default 60s, - `cache_ttl` on the catalog). A newly deployed view appears within one TTL; - there is no invalidation endpoint - expiry is the mechanism. -- **Result caching (wire format only).** `QueryOptions.cache` and the + `cache_ttl` on the catalog). A newly deployed view appears within one TTL. + There is no invalidation endpoint - expiry is the mechanism. +- Result caching (wire format only): `QueryOptions.cache` and the `cached`/`cache_key` metadata fields exist in the request/response models, but no server-side result cache is implemented - the flag has no - effect. Treat result caching as a client-side concern until the engine - grows one. + effect. Treat result caching as a client-side concern. --- @@ -432,7 +431,7 @@ there is no per-query role list: |---|---| | `query:read` | browse the view catalog (list/get views, namespaces) | | `query:execute` | execute views | -| `raw_query:execute` | run raw datasource queries; only `admin` holds it | +| `raw_query:execute` | run raw datasource queries. Only `admin` holds it | ```mermaid flowchart TD diff --git a/docs/data-plane/schema-bootstrap.md b/docs/data-plane/schema-bootstrap.md index 5e03a9ccc..13e2af0e3 100644 --- a/docs/data-plane/schema-bootstrap.md +++ b/docs/data-plane/schema-bootstrap.md @@ -7,7 +7,7 @@ not a Job, not an init container, not an app at runtime, not the OTel collector. `dfe-schemas` is where those objects are DEFINED. `manifest.yaml` lists every one of them in dependency order, and the engine iterates that list and nothing else. -Adding a table is a manifest edit plus a definition file; it is never an engine +Adding a table is a manifest edit plus a definition file. It is never an engine release. ## What runs, in what order @@ -86,7 +86,7 @@ consulted -- the pod is not restarted out from under whoever is reading it. ## The overlay A deployment's own schema tree is additive. It may declare objects the core does -not; an object of its own whose definition sits under a core directory +not. An object of its own whose definition sits under a core directory (`tables/core/`, `tables/internal/`, `tables/otel/`, `tables/meta/`, `common-header/`, `registries/`, `roles/`, `topics/`, `views/`) is refused by name and reported, and the core apply is unaffected. The overlay is optional: an @@ -98,7 +98,7 @@ install with no deploy repo boots core-only. |---|---|---| | `DFE_ENGINE_CLICKHOUSE_BOOTSTRAP_TABLES` | `true` | Apply the manifest at startup. Off reports the schema state as unknown and gates nothing. | | `DFE_ENGINE_CLICKHOUSE_BOOTSTRAP_WAIT_SECONDS` | `180` | How long to keep retrying an unreachable ClickHouse before reporting the pass failed. | -| `DFE_ENGINE_CLICKHOUSE_DDL_MAX_QUERY_SIZE` | `16777216` | Ceiling in bytes on the `max_query_size` sent with schema DDL. A statement over ClickHouse's 256 KiB default -- the widest Elastic vendor table is 688 KB -- is sent with its own size plus 4 KiB, by the manifest apply and by a source deploy alike. One that would need more is refused before it is sent. A caller's own SQL never gets the setting. | +| `DFE_ENGINE_CLICKHOUSE_DDL_MAX_QUERY_SIZE` | `16777216` | Ceiling in bytes on the `max_query_size` sent with schema DDL. A statement over ClickHouse's 256 KiB default, such as a wide Elastic meta schema's table, is sent with its own size plus 4 KiB, by the manifest apply and by a source deploy alike. One that would need more is refused before it is sent. A caller's own SQL never gets the setting. | | `DFE_ENGINE_CLICKHOUSE_PLATFORM` | `auto` | The ClickHouse platform schema DDL is rendered for: `auto` probes the server, then reads the hostname. `cloud` or `self_hosted` pins it, for a server the probe reads wrongly. Anything else refuses to start. | | `DFE_ENGINE_KAFKA_BOOTSTRAP_TOPICS` | `true` | Create the declared topic set at startup, create-only. | | `DFE_ENGINE_KAFKA_TIERED_STORAGE` | `false` | The brokers tier to object storage, so the landing topic is created with `remote.storage.enable`. Off leaves the key unset rather than false, which is what a broker-level setting needs. | diff --git a/docs/data-plane/schema-classes.md b/docs/data-plane/schema-classes.md index a2335fd5f..b8feb7c41 100644 --- a/docs/data-plane/schema-classes.md +++ b/docs/data-plane/schema-classes.md @@ -3,7 +3,7 @@ The engine classes that turn schema YAML (from the dfe-schemas repo) into ClickHouse DDL. The YAML FORMAT itself (version tree, column fields, types, attributes, use cases, @directives) is documented in the dfe-schemas repo -(); this page is the code-side lookup. Concept +(). This page is the code-side lookup. Concept overview: [schema.md](schema.md). ## File map @@ -12,7 +12,7 @@ overview: [schema.md](schema.md). |------|-----------------| | `src/dfe_engine/source/models.py` | `SchemaColumn`, `SourceSchema`, `SourceHeader`, `Source` Pydantic models | | `src/dfe_engine/source/type_registry.py` | `TypeRegistry` class, `ResolvedType` dataclass | -| dfe-schemas `registries/types.yaml` | 13 primitives, use_case constraints, attribute constraints, ch_override catalogue | +| dfe-schemas `registries/types.yaml` | the primitives, use_case constraints, attribute constraints, ch_override catalogue | | `src/dfe_engine/source/engine_registry.py` | `EngineRegistry` - permitted table engines and their argument rules, from dfe-schemas `registries/engines.yaml` | | `src/dfe_engine/schema/schema_loader.py` | `SchemaLoader` - YAML loading, composition, validation | | `src/dfe_engine/schema/schema_ddl.py` | `DDLGenerator`, `DDLConfig` - column/index/view DDL generation | @@ -38,7 +38,7 @@ flowchart LR ## Key classes -### SchemaColumn (`source/models.py`) +### SchemaColumn (`src/dfe_engine/source/models.py`) ```python from dfe_engine.source.models import SchemaColumn @@ -54,7 +54,7 @@ col = SchemaColumn( errors = col.validate_against_registry(TypeRegistry.default()) ``` -### TypeRegistry (`source/type_registry.py`) +### TypeRegistry (`src/dfe_engine/source/type_registry.py`) ```python from dfe_engine.source.type_registry import TypeRegistry @@ -67,7 +67,7 @@ registry.validate_use_case("integer", "word_search") # raises ValueError registry.validate_cardinality("medium") # raises InvalidCardinalityError ``` -### SchemaLoader (`schema/schema_loader.py`) +### SchemaLoader (`src/dfe_engine/schema/schema_loader.py`) ```python from dfe_engine.schema.schema_loader import SchemaLoader @@ -82,7 +82,7 @@ order_cols = SchemaLoader.get_order_by_columns(full) meta = SchemaLoader.load_version_metadata("meta/syslog.yaml") ``` -### SchemaBuilderV2 (`schema/schema_builder_v2.py`) +### SchemaBuilderV2 (`src/dfe_engine/schema/schema_builder_v2.py`) ```python from dfe_engine.schema.schema_builder_v2 import SchemaBuilderV2 @@ -99,7 +99,7 @@ result.validation_errors # list[str] result.view_ddls # {"sigma": "CREATE VIEW ...", ...} ``` -### DDLGenerator (`schema/schema_ddl.py`) +### DDLGenerator (`src/dfe_engine/schema/schema_ddl.py`) ```python from dfe_engine.schema.schema_ddl import DDLGenerator, DDLConfig @@ -110,7 +110,7 @@ add_ddl = gen.generate_alter_add_column("my_table", column) view_ddl = gen.generate_view("my_table", {"SourceIP": "source_ip"}, "sigma") ``` -### SchemaManager (`schema/schema_manager.py`) +### SchemaManager (`src/dfe_engine/schema/schema_manager.py`) Version WRITE operations - published versions are immutable, so all changes create new version entries: @@ -145,7 +145,7 @@ SchemaManager.clone_version( ) ``` -### The applier (`schema/applier.py`) +### The applier (`src/dfe_engine/schema/applier.py`) One applier brings every live table to its CREATE TABLE. It reads the statement with sqlglot's ClickHouse tokens, and an added column or index is cut from the @@ -162,7 +162,7 @@ plan.statements # CREATE, or ADD COLUMN / index / TTL / settings moves ## Validation rules -1. Primitive must exist in TypeRegistry (one of 15 values). +1. Primitive must exist in TypeRegistry (`registries/types.yaml` `primitives`). 2. Use case must be valid for the primitive (e.g. `word_search` is not valid for `integer`). 3. Attributes must be valid for the primitive (e.g. `lowcardinality` is not @@ -177,8 +177,8 @@ plan.statements # CREATE, or ADD COLUMN / index / TTL / settings moves ## Gotchas - **`expr` vs `comment`**: both end up in the ClickHouse COMMENT clause, - separated by ` — ` (that separator is part of the wire format the loader - parses - do not "fix" it). `expr` carries the `@directive`; `comment` is + separated by ` - ` (dfe-schemas `render.py`). That separator is part of the + wire format the loader parses. `expr` carries the `@directive`. `comment` is the human description. Never put directives in `comment`. - **`default` does triple duty**: `DEFAULT expr` normally, `MATERIALIZED expr` with the `materialized` attribute, `ALIAS expr` with @@ -188,13 +188,13 @@ plan.statements # CREATE, or ADD COLUMN / index / TTL / settings moves - **`timestamp` vs `datetime`**: both map to `DateTime64(3,'UTC')`, but `timestamp` is NOT nullable and uses `Delta, LZ4`. Use `timestamp` for ORDER BY time columns. -- **Nullable ORDER BY**: ORDER BY columns should carry `not_null`; the +- **Nullable ORDER BY**: ORDER BY columns should carry `not_null`. The engine warns but does not reject. - **Shipped schemas are read-only**: `is_shipped_schema()` detects paths inside the resolved schemas root (the `dfe-schemas` package, or the image seed) - custom schemas live outside those directories (`DFE_ENGINE_SCHEMAS_DIR`). - **Profile name matches the FILENAME**: use `timeseries` / `minimal` / `passthrough` in `header.type`, matching the YAML filenames. -- **registries/types.yaml loads via YAML 1.1** (PyYAML through - DirectoryConfigStore semantics): `off`/`yes`/`no` become booleans. Meta - schemas load via ruamel (YAML 1.2) and are unaffected. +- **Schema YAML is YAML 1.2**: `registries/types.yaml` and the meta schemas + load through ruamel (`yaml_utils.yaml_load`, dfe-schemas `loader.py`), so + `off`/`yes`/`no` stay strings there. diff --git a/docs/data-plane/schema-sync.md b/docs/data-plane/schema-sync.md index cfe6e7108..f1ba8d598 100644 --- a/docs/data-plane/schema-sync.md +++ b/docs/data-plane/schema-sync.md @@ -1,180 +1,34 @@ -# Cross-Repo Synchronisation +# The engine and loader schema contract -**Status:** Proposed -**Last Updated:** 2026-02-28 +The engine renders and applies the DDL. dfe-loader reads the deployed table back out of ClickHouse and creates none. So the deployed table is the whole contract between them, and dfe-schemas is where the table's definition comes from. ---- +## dfe-schemas owns the definitions -## Problem +The dfe-schemas package holds the common-header profiles (`common-header/`), the source meta schemas (`meta/`), the hunt output schemas (`hunts/`) and the registries (`registries/`): table engines, the type system (primitive to ClickHouse type) and the default field maps. -Two repos define overlapping schema structures: +The engine reads the version-pinned `dfe-schemas` wheel through `importlib.resources` and ships no schema files of its own. It looks in `DFE_ENGINE_SCHEMAS_DIR`, then the installed package, then the image's seed directory (copied out of that wheel at build), and when none carries what it needs, loading fails with `No dfe-schemas tree carries ...: set DFE_ENGINE_SCHEMAS_DIR or install dfe-schemas` (`src/dfe_engine/schema/schema_loader.py`). -| Concept | dfe-engine (Python) | dfe-loader (Rust) | -|---------|--------------------|--------------------| -| Common header fields | `type_maps.csv` + schema CSVs | `schemas/profiles/*.yaml` + `COMMON-HEADER.md` | -| Field types | `type_maps.csv` (24 logical types) | `src/clickhouse/types.rs` (runtime parsing) | -| Loader directives | Column `comment` in schema CSV | `DDL-EXPRESSION.md` + `field_mapping.rs` | -| Profile definitions | Not yet (proposed in SOURCE.md) | `schemas/profiles/timeseries.yaml` etc. | +## The deployed table is the runtime contract -If these drift apart, the engine generates DDL that the loader can't -correctly populate — or the loader expects columns that don't exist. - ---- - -## What Must Stay In Sync - -### 1. Common Header Profile Definitions - -The common header is the **contract** between the engine (which generates -DDL) and the loader (which populates fields). Both must agree on: - -- Field names (`_timestamp`, `_org_id`, `_source`, etc.) -- Field types (exact ClickHouse types) -- Field order in ORDER BY -- Loader directives (how each field is populated) -- Profile versioning - -### 2. Type Primitives → ClickHouse Type Mapping - -The engine maps primitives to ClickHouse types when generating DDL. -The loader parses ClickHouse types from `system.columns` at runtime. -These must be compatible — the engine's output must be valid input for -the loader's type parser. - -### 3. Loader Directive Syntax - -The engine writes `@source`, `@generated`, `@renamed`, `@captured`, -`@computed` directives as column comments. The loader parses these -directives. The syntax must match. - ---- - -## Synchronisation Strategy - -### Single Source of Truth: the dfe-schemas repo - -The **dfe-schemas repo** owns the canonical schema YAML (this landed after -the original draft of this doc, which had the engine owning it): - -- Common header profile YAML files (`common-header/`) -- Source meta schemas (`meta/`) and hunt output schemas (`hunts/`) -- The registries (`registries/`): table engines, the type system (primitive -> ClickHouse mapping) and the default field maps -- Loader directive syntax examples (documented, not parsed by engine) - -The engine owns the DDL generation. dfe-engine consumes dfe-schemas as the version-pinned `dfe-schemas` wheel, resolved through `importlib.resources`, and keeps no copies of its own: with neither the package nor `DFE_ENGINE_SCHEMAS_DIR` resolving, loading fails with an error naming what was searched. dfe-loader still consumes it as a `schemas/` git submodule. - -```text -dfe-schemas (source of truth) - | - | common-header/{timeseries,minimal,passthrough}.yaml - | - |--> dfe-engine (dfe-schemas wheel) - generates DDL - | - '--> dfe-loader (submodule + bundled fallback) - auto-init tables +```mermaid +flowchart LR + S[dfe-schemas wheel] --> E[dfe-engine
renders + applies DDL] + E --> CH[(ClickHouse table)] + CH -->|system.columns,
column COMMENTs| L[dfe-loader] ``` -dfe-loader reads column metadata from ClickHouse at runtime -(`system.columns`), not from config files. The contract is the DDL itself - -whatever the engine produces, the loader must handle. - -### Runtime Contract: Deployed ClickHouse Schema is the ONLY SSoT - -The Rust K8s services (loader, archiver) at scale **slave from the -deployed ClickHouse schema ONLY**. They do NOT read profile YAMLs, -Source definitions, or any config files at query/ingest time. They read: - -1. **Column types** from `system.columns` — the deployed table -2. **Column comments** from `system.columns` — contain loader directives -3. **Table comment** — contains `@profile`, `@profile_version` tags - -This means there is **never an in-flight mismatch**. The flow is strictly -one-directional: - -```text -Source YAML → engine generates DDL → deploys to ClickHouse → Rust services read from ClickHouse - ↑ - ONLY source of truth at runtime -``` - -The engine writes, ClickHouse stores, Rust services read. No intermediate -state, no config sync, no cache invalidation race. If the engine hasn't -deployed a schema change yet, the Rust services continue operating on the -current deployed schema. Once the DDL is applied, the Rust services pick -up the change on their next `system.columns` refresh (TTL-based cache). - -Profile YAMLs are only needed for the loader's **auto-init** feature -(creating the default table when no table exists). Once the table is -created, the YAML is irrelevant — the deployed DDL is the truth. - -### CI Validation - -Add a CI step that: - -1. Generates DDL from dfe-engine profiles -2. Parses the generated DDL with dfe-loader's type parser -3. Validates that all column comments contain valid loader directives -4. Fails if any incompatibility is detected - -```yaml -# .github/workflows/schema-compat.yml -- name: Generate DDL from profiles - run: python -m dfe_engine.schema.generate_profiles --output /tmp/ddl/ - -- name: Validate loader compatibility - run: cargo test --manifest-path ../dfe-loader/Cargo.toml -- schema_compat - env: - DDL_DIR: /tmp/ddl/ -``` - -This catches drift at PR time, not in production. - ---- - -## Version Contract - -Both repos must agree on version numbers: - -```text -@profile: timeseries -@profile_version: 1 -@schema_version: 2 -``` - -When the engine bumps a profile version: - -1. Engine generates migration DDL (`ALTER TABLE ADD COLUMN`) -2. Loader's `ProfileDiff` detects the version mismatch and applies migration -3. CI validates compatibility before merge - -The version is embedded in the table comment — the loader reads it at -startup and logs a warning if it's behind the current profile version. - ---- - -## What Each Repo Owns +dfe-loader reads three things from the table, never a profile YAML or a source definition: -| Responsibility | Owner | Notes | -|---------------|-------|-------| -| Profile definitions (YAML) | dfe-engine | Canonical source | -| Type primitive mapping | dfe-engine | `type_maps.csv` or equivalent | -| DDL generation | dfe-engine | `SchemaBuilder` / `ClickHouseSchema` | -| Loader directive syntax | dfe-loader | Defines the `@` expression language | -| Loader directive documentation | dfe-engine | Documents what directives are available | -| Column comment parsing | dfe-loader | `field_mapping.rs` | -| Schema caching | dfe-loader | TTL-based from `system.columns` | -| Auto-init DDL | dfe-loader | Uses profile YAMLs (from dfe-schemas) | -| Profile migration | dfe-loader | `ProfileDiff` generates ALTER statements | -| CI compatibility check | Both | Cross-repo CI step | +1. Column types, from `system.columns`. +2. Column COMMENTs, which carry the loader directives. +3. The table COMMENT, which carries `@profile` and `@profile_version` for a header-composed table (dfe-schemas `render.py`). ---- +The loader caches that schema for `schema.cache_ttl_secs` (300 by default, dfe-loader `docs/config-schema.yaml`), so an applied change reaches it on the next refresh, and until then it carries on with the schema it holds. -## Monorepo Path +## What has to agree -If/when the repos merge into a monorepo (see [MONOREPO-MIGRATION.md](../archive/monorepo-migration.md)), -the sync problem disappears. Profile YAMLs live in one place, CI runs -both the Python and Rust tests in a single pipeline, and there's no -package pin, submodule or copy-on-release ceremony. +- Directive vocabulary. The loader parses `@skip`, `@default`, `@source`, `@renamed`, `@computed` and `@coerce` (dfe-loader `src/column_meta/mod.rs`). `tests/unit/test_services/test_loader_directive_contract.py` restates that list as a literal and fails when the engine writes a directive the loader does not carry. +- Types. The engine maps primitives through dfe-schemas `registries/types.yaml`. The loader parses whatever ClickHouse type `system.columns` reports. +- Header columns. Both sides take them from the profile in `common-header/`. -Both consumption paths are stepping stones — when the monorepo happens, -the package dependency and the loader's submodule reference are replaced -by a direct path. +No cross-repo CI job runs the loader's type parser over engine-generated DDL, and the loader has no profile migration of its own: the engine's schema apply adds the columns a new profile version brings. diff --git a/docs/data-plane/schema.md b/docs/data-plane/schema.md index a2c187bb3..f64f18f46 100644 --- a/docs/data-plane/schema.md +++ b/docs/data-plane/schema.md @@ -9,9 +9,8 @@ wheel), registries included, and the engine keeps no copies. This document is written for **data subject matter experts** -- people who understand their data, their queries, and their retention requirements. You -do NOT need to be a ClickHouse expert. The schema system handles ClickHouse -internals (codecs, index tuning, Nullable wrapping, type widths) under the -hood. You pick meaningful types and describe how your data will be queried. +do NOT need to be a ClickHouse expert. The schema system picks the codecs, +indexes, Nullable wrapping and type widths. You pick meaningful types and describe how your data will be queried. The engine does the rest. If you need a specific ClickHouse type that the primitives don't cover, @@ -39,18 +38,15 @@ from `system.columns` at runtime. There is nothing in between. Schema changes flow in one direction: ```text -Source YAML (intent) → engine generates DDL → ClickHouse (deployed truth) - ↑ - Rust K8s services read at runtime - (loader, archiver -- ONLY SSoT) +Source YAML (intent) -> engine generates DDL -> ClickHouse (deployed truth) + ^ + dfe-loader reads it at runtime ``` -The Rust K8s services at scale **slave from the deployed ClickHouse -schema ONLY**. They read column types, comments, and table metadata -from `system.columns` at runtime -- never from YAML, config files, or -any intermediate store. This means there is **never an in-flight -mismatch**: the engine deploys DDL, ClickHouse stores it, Rust services -read it. No sync, no cache invalidation race, no drift. +dfe-loader reads column types, comments and table metadata from +`system.columns` at runtime, never from the YAML. It caches them for +`schema.cache_ttl_secs`, so an applied change reaches it on the next refresh +([schema-sync.md](schema-sync.md)). ### Schema Format: YAML @@ -68,84 +64,44 @@ These are data files, not schema definitions. |-------|--------| | Source created | Engine generates `CREATE TABLE` from common header + source schema | | Schema field added | Engine generates `ALTER TABLE ADD COLUMN` | -| Schema field modified | Engine generates `ALTER TABLE MODIFY COLUMN` (type/codec/comment) | +| Index or TTL changed | The schema apply drops and re-adds the index, or moves the TTL. A changed column type is drift, applied only with `dfe schema apply --allow-drift` ([schema-bootstrap.md](schema-bootstrap.md)) | | Source disabled | No DDL change -- table stays, no new data | | Source deleted | Table preserved -- manual `DROP TABLE` if needed | -The deployed ClickHouse table is always authoritative. If someone modifies -the table directly (via SQL), the engine detects drift on next schema -operation and can reconcile. +The deployed ClickHouse table is authoritative. A change made to it directly is +reported as drift by the next schema plan or apply. --- -## Four axes +## The axes -A schema column definition has **four independent axes**: +A schema column definition has independent axes, each optional where it has a +default, plus the loader directive in `expr`: ```text -type + attribute + use_case + comment (includes loader directives) +type + cardinality + attribute + use_case, with expr (the loader directive) and comment (prose) ``` -Each axis is optional where it has sensible defaults. - --- ## Schema Definition Format (YAML) +The common header columns come from the profile (`common-header/` in +dfe-schemas, see "Common Header Profiles" below). A meta schema declares the rest: + ```yaml # meta_schema.yaml columns: - - name: _timestamp_load - type: timestamp - default: "now64(3)" - order: 0 - comment: "@generated: now64(3)" - - - name: _timestamp - type: datetime - use_case: range - order: 1 - comment: "@source: timestamp | now()" - - - name: _uuid - type: uuid - default: "generateUUIDv7()" - comment: "@generated: generateUUIDv7()" - - - name: _org_id - type: string - cardinality: low - use_case: dimension - comment: "@source: org_id" - - - name: _source - type: string - cardinality: low - use_case: dimension - comment: "@source: first(_source) | topic_name" - - - name: _raw - type: text - use_case: substring_search - comment: "@captured: raw_payload" - - - name: _json - type: json - comment: "@captured: raw_payload as JSON" - - - name: _tags - type: json - comment: "@source: first(tags/_tags/meta/metadata.tags)" - - name: user_name type: string use_case: dimension - comment: "@source: first(user_id/uid/id)" + expr: "@source: first(user_id/uid/id)" + comment: "User identifier" - name: source_ip type: ip use_case: range - comment: "@source: src_ip" + expr: "@source: src_ip" - name: severity type: string @@ -158,7 +114,7 @@ columns: - name: event_id type: integer - comment: "@source: event_id" + expr: "@source: event_id" - name: latency_ms type: float @@ -176,7 +132,8 @@ columns: | `use_case` | No | Query pattern hint -- determines indexing (see below) | | `default` | No | DEFAULT expression | | `order` | No | Position in ORDER BY / PRIMARY KEY | -| `comment` | No | Human description + loader directives | +| `expr` | No | The loader directive (`@source: ...`), see below | +| `comment` | No | Human description. Never a directive | | `ch_override` | No | Exact ClickHouse type -- bypasses primitive mapping | --- @@ -417,18 +374,16 @@ at validation time. Each primitive has a default nullability (defined in the type registry): -- **Nullable by default:** `string`, `text`, `integer`, `float`, `datetime`, `date`, `ip`, `uuid`, `json`, `geo_point` -- **NOT null by default:** `timestamp`, `boolean`, `enum` +- Nullable by default: `string`, `text`, `integer`, `float`, `datetime`, `date`, `ip`, `uuid`, `json`, `geo_point` +- NOT null by default: `timestamp`, `boolean`, `enum`, `map`, `vector` Override with `nullable` or `not_null` in the attribute list. ### Why nullable is not the default for everything ClickHouse stores `Nullable(T)` as two columns -- the data column plus a -UInt8 bitmap tracking which rows are null. This **doubles storage** and -**halves query speed** (measured: 229M rows/s -> 98M rows/s on GROUP BY -with Nullable(Int64)). Booleans and `timestamp` are non-null by default -for this reason. +UInt8 bitmap tracking which rows are null -- so it costs storage and query +speed. Booleans and `timestamp` are non-null by default for this reason. **Design decision: ORDER BY keys must not be Nullable.** ClickHouse refuses a Nullable column in a sorting key unless the merge-tree setting @@ -481,8 +436,8 @@ your query pattern. ### Use Cases A use case names the question you ask the column, never the ClickHouse index -that answers it. You declare the question; the engine picks the primitive, and -can pick a different one on a later ClickHouse without your schema changing. +that answers it. You declare the question. The engine picks the index, and can +pick a different one on a later ClickHouse without your schema changing. | Use Case | The question | Example Columns | |----------|------------|-----------------| @@ -532,21 +487,19 @@ vocabulary. | Use Case | ClickHouse Index Generated | Granularity | Notes | |----------|---------------------------|-------------|-------| -| `dimension` | `set(0)` | 4 | Exact distinct values per granule | +| `dimension` | `set(0)`, or `bloom_filter` on a Nullable column | 4 | Exact distinct values per granule | | `exact_match` | `set(0)` on `cardinality: low`, else `bloom_filter` | 4 | The bloom filter is probabilistic -- false positives, no false negatives | | `range` | `minmax` | 4 | Stores min/max per granule | -| `word_search` | `text(tokenizer=splitByNonAlpha)` | 1 | Native text index (GA v26.2). Deterministic, no false positives, row-level filtering. 45x faster than without index. | +| `word_search` | `text(tokenizer=splitByNonAlpha)` | 1 | Native text index. Deterministic, no false positives, row-level filtering | | `substring_search` | `text(tokenizer=ngrams(3))` | 1 | Character n-gram text index for substring matching | | `key_search` | two indexes, `text(tokenizer=array)` over `mapKeys(col)` and `mapValues(col)` | 1 | A text index refuses a `Map` column itself, so the keys and the values are indexed apart | -| `similarity_search()` | `vector_similarity('hnsw', 'cosineDistance', )` | 1 | `hnsw` is the only method the pinned ClickHouse implements | +| `similarity_search()` | `vector_similarity('hnsw', 'cosineDistance', )` | 100000000 | One index per part. `hnsw` is the only method the pinned ClickHouse implements | | *(empty)* | No index | -- | | **Note on text indexes:** `word_search` and `substring_search` generate the GA -text index (inverted index, v26.2+) instead of the older bloom-filter based -`tokenbf_v1` and `ngrambf_v1`. The text index is deterministic (no false -positives), provides row-level filtering instead of granule-level, and is -10-100x faster for text search workloads. There is no pre-25.10 variant: every -table gets the text index. +text index instead of the older bloom-filter based `tokenbf_v1` and +`ngrambf_v1`. The text index is deterministic (no false positives) and filters +at row level rather than granule level. Every table gets the text index. Every index, partition and TTL clause comes from dfe-schemas, which owns the one table renderer and the DDL catalogue (`registries/ddl.yaml`). The engine builds @@ -566,39 +519,44 @@ deployment sets `DFE_ENGINE_CLICKHOUSE_DEFAULT_TTL_DAYS` (`clickhouse.default_tt `schema.ttl_days`, then the table's own dfe-schemas definition, then the console value, then the deployment default. State tables (the detection checkpoint, the engine's internal and hunt-coordination tables) keep -whatever their definition declares and never take the default. The schema apply refuses a changed TTL on one of the engine's own tables as drift rather than expiring rows at boot; an undeclared TTL never removes a live one. Shortening a TTL expires the rows +whatever their definition declares and never take the default. The schema apply refuses a changed TTL on one of the engine's own tables as drift rather than expiring rows at boot. An undeclared TTL never removes a live one. Shortening a TTL expires the rows older than the new value. -**Changing the default:** an admin sets it with `PUT /api/v1/system/retention` (`system:write`, which only the `admin` role holds by default); `GET` reports the effective value, the console value and the deployment value. The console value is committed to the deploy repo at `governance/settings/retention.yaml` (`default_ttl_days`), so it survives the loss of the engine and every change is an audited commit; `null` clears it. The PUT applies the new value in the same request to every table that follows the default, the engine's own time-series tables and each deployed source's table, with `ALTER TABLE ... MODIFY TTL` (`REMOVE TTL` for 0). Every later build, plan, deploy and boot reads the same value. +**Changing the default:** an admin sets it with `PUT /api/v1/system/retention` (`system:write`, which only the `admin` role holds by default). `GET` reports the effective value, the console value and the deployment value. The console value is committed to the deploy repo at `governance/settings/retention.yaml` (`default_ttl_days`), so it survives the loss of the engine and every change is an audited commit. `null` clears it. The PUT applies the new value in the same request to every table that follows the default, the engine's own time-series tables and each deployed source's table, with `ALTER TABLE ... MODIFY TTL` (`REMOVE TTL` for 0). Every later build, plan, deploy and boot reads the same value. On a restart the engine brings every deployed source's table to the effective default, but the schema apply refuses a changed default on its own tables, so after changing `DFE_ENGINE_CLICKHOUSE_DEFAULT_TTL_DAYS` send the PUT with `null` to apply it there. `dfe schema plan` and `dfe schema apply` read the deployment settings, never the deploy repo, so while a console value is set they report the engine's time-series tables as TTL drift, and `dfe schema apply --allow-drift` rewrites those TTLs to the deployment default. --- -## Comment Axis: Loader Directives +## Loader directives (`expr`) -The `comment` field serves double duty: +A column's `expr` carries the `@` directive that tells dfe-loader how to fill it, +and `comment` the human description. Both land in the ClickHouse column COMMENT, +joined by ` - `. -1. **Human documentation** -- plain text description -2. **Loader directives** -- `@` prefixed expressions that tell dfe-loader - how to populate the field +### The directives dfe-loader parses -### Directive Types (from dfe-loader DDL Expression Language) +dfe-loader's parser (`src/column_meta/mod.rs`) knows six directives and skips any +other word: | Directive | Purpose | Example | |-----------|---------|---------| -| `@source: field` | Copy from source data | `@source: timestamp \| now()` | -| `@source: first(a/b/c)` | First non-null from list | `@source: first(user_id/uid/id)` | -| `@generated: expr` | ClickHouse DEFAULT -- loader omits field | `@generated: now64(3)` | -| `@renamed: field` | Zero-copy field rename | `@renamed: logoriginal` | -| `@captured: payload` | Raw payload sidecar | `@captured: raw_payload as JSON` | -| `@computed: expr` | Derived/enriched value | `@computed: geoip(ip).country_code` | +| `@source: field` | Copy from source data, with an optional `\| fallback` | `@source: timestamp \| now()` | +| `@source: first(a/b/c)` | First present field from a list | `@source: first(user_id/uid/id)` | +| `@renamed: field` | Source field path(s), the hand-written form of `@source` | `@renamed: logoriginal` | +| `@computed: expr` | A CEL expression producing the value | `@computed: risk_score > 80` | +| `@default: value` | Substitute when the field is null or absent | `@default: unknown` | +| `@skip` | Leave the column out of the insert | `@skip` | +| `@coerce: category` | Override the type category for coercion | `@coerce: ` | + +dfe-schemas also writes `@generated: ` on a column ClickHouse fills from +its DEFAULT, such as `_uuid`. The loader does not parse it. ### Precedence -When the loader resolves field mappings, column comments have **highest -precedence** -- above built-in presets and external remap files. This means -the schema definition IS the authoritative field mapping. +The loader's config cascade wins over a column COMMENT: per-table +`column_directives`, then global `column_directives`, then the COMMENT +(dfe-loader `src/column_meta/mod.rs`). ### Comments in DDL @@ -606,17 +564,14 @@ These directives are emitted as ClickHouse column comments in the generated DDL: ```sql -`_timestamp` DateTime64(3,'UTC') CODEC(Delta, ZSTD(1)) - COMMENT '@source: timestamp | now()', -`_raw` Nullable(String) CODEC(ZSTD(3)) - COMMENT '@captured: raw_payload', +`_timestamp` DateTime64(3,'UTC') CODEC(Delta, LZ4) + COMMENT '@source: timestamp | now() - Event timestamp from source data', `user_name` Nullable(String) CODEC(ZSTD(1)) - COMMENT '@source: first(user_id/uid/id)', + COMMENT '@source: first(user_id/uid/id) - User identifier', ``` -The loader fetches column comments from `system.columns` and uses them to -build its per-table field mapping. No separate mapping config needed -- the -schema IS the mapping. +The loader fetches column comments from `system.columns` and builds its +per-table field mapping from them, under whatever its config cascade sets. --- @@ -670,9 +625,12 @@ are added incrementally. Tables are tagged with profile metadata in the table comment: ```sql -COMMENT '@schema_source: core | @schema_version: 2 | @profile: timeseries | @profile_version: 1' +COMMENT '@profile: timeseries | @profile_version:
| @_version: ' ``` +The engine writes these tags on a source table (`DDLGenerator._table_comment`), +and dfe-schemas the first two on a header-composed manifest table. + Profile migrations (adding new common header fields) use `ALTER TABLE ADD COLUMN` -- safe, non-destructive, and automated. @@ -682,9 +640,9 @@ COLUMN` -- safe, non-destructive, and automated. `dfe.all_sources` is a ClickHouse Merge table over `main` and each registered source's MergeTree table, so one Observe source reads every source's rows. Core manifest tables other than `main` (even under a source's name) and tables without `_org_id` are never members. -It declares `_timestamp_load`, `_timestamp`, `_timestamp_received`, `_org_id`, `_source`, `_raw` and `_json`, typed as on `main`. A member lacking one reads its default: epoch 0 for a timestamp, NULL for a Nullable column. `_raw IS NULL` misreads a member without `_raw`, since ClickHouse answers it from the `_raw.null` subcolumn while `optimize_functions_to_subcolumns` is on; test `length(_raw)` or use `coalesce`. The virtual column `_table` names each row's member. +It declares `_timestamp_load`, `_timestamp`, `_timestamp_received`, `_org_id`, `_source`, `_raw` and `_json`, typed as on `main`. A member lacking one reads its default: epoch 0 for a timestamp, NULL for a Nullable column. `_raw IS NULL` misreads a member without `_raw`, since ClickHouse answers it from the `_raw.null` subcolumn while `optimize_functions_to_subcolumns` is on. Test `length(_raw)` or use `coalesce`. The virtual column `_table` names each row's member. -The engine rebuilds it after every source deploy or delete. At startup it rebuilds once the [schema bootstrap](schema-bootstrap.md) converges or is switched off, never after a failed pass. Each rebuild reads the registry as it stands then; concurrent rebuilds in one engine fold into one. A deleted source leaves the members while its table stays. +The engine rebuilds it after every source deploy or delete. At startup it rebuilds once the [schema bootstrap](schema-bootstrap.md) converges or is switched off, never after a failed pass. Each rebuild reads the registry as it stands then. Concurrent rebuilds in one engine fold into one. A deleted source leaves the members while its table stays. Each rebuild applies the tenant row policy ([rbac.md](../control-plane/rbac.md)) to `all_sources` itself. Member tables keep their own policies, which also fence a read through it. @@ -692,7 +650,7 @@ The engine registers it on every team as the Observe source `all_sources`, timed ## The `dlq` Table -`dfe.dlq` holds the dead letters every app writes, kept 30 days. The engine registers it on every team as the HyperDX source `dlq`, timed on `_timestamp`, alongside `all_sources` and with the same retry. A source cannot be named `dlq`, `detection` or `repository`, since its table would be the core table of that name. +`dfe.dlq` holds the dead letters every app writes, kept 30 days. The engine registers it on every team as the HyperDX source `dlq`, timed on `_timestamp`, alongside `all_sources` and with the same retry. A source cannot take the name of a table the core manifest owns, such as `dlq` or `detection`, since its table would be that core table. --- @@ -726,12 +684,12 @@ FROM {db}.{source} - Sigma rules query the view (standard Sigma field names) - Field mapping is per-source (different sources map differently) - Zero storage overhead (view, not materialised table) -- Mappings resolved by `sigma/source_mapper.py` from the source's field map and sigma view +- Mappings resolved by `src/dfe_engine/sigma/source_mapper.py` from the source's field map and sigma view ### Sigma Mapping as Source Metadata The Source definition declares its naming-standard views in the generic -`views` list; the sigma entry carries the logsource binding and overrides: +`views` list. The sigma entry carries the logsource binding and overrides: ```yaml # sources/windows_audit.yaml @@ -749,17 +707,18 @@ The engine generates the Sigma view from source schema + the sigma view entry. ## Elastic Index Template Converter -The existing Elastic/OpenSearch index template converter generates a -source meta schema from a supplied Elastic template JSON. This remains -- -but now outputs YAML in the Source schema format: +The Elastic/OpenSearch index template converter +(`src/dfe_engine/services/schema/elastic_schema_service.py`) generates a source +meta schema from a supplied Elastic template JSON, as YAML in the Source schema +format: ```text Elastic Template JSON │ ▼ Template Converter (dfe-engine) - │ Maps Elastic types → primitives - │ Maps Elastic analyzers → use_cases + │ Maps Elastic types -> primitives + │ Maps Elastic analyzers -> use_cases │ Preserves field hierarchy as flat columns │ ▼ @@ -774,20 +733,16 @@ Attached to a Source definition | Elastic Type | Primitive | Use Case | Notes | |-------------|-----------|----------|-------| -| `keyword` | `string` | `dimension` | Exact match filtering | -| `text` | `text` | `word_search` | Searching for whole words | -| `long` | `integer` | | Default Int64 matches | -| `integer` | `integer` | | | -| `short` | `integer` | | ch_override: Int16 if needed | -| `byte` | `integer` | | ch_override: Int8 if needed | -| `double` | `float` | | | -| `float` | `float` | | ch_override: Float32 if needed | -| `boolean` | `boolean` | | | -| `date` | `datetime` | | | +| `keyword`, `wildcard`, `version` | `string` | | No use case: cardinality is unknown | +| `constant_keyword` | `string` | `dimension` | Bounded by definition | +| `text`, `match_only_text` | `text` | `word_search` | Searching for whole words | +| `long`, `integer`, `short`, `byte`, `unsigned_long` | `integer` | `range` | ch_override narrows if needed | +| `double`, `float`, `half_float`, `scaled_float` | `float` | `range` | | +| `boolean` | `boolean` | `dimension` | | +| `date`, `date_nanos` | `datetime` | `range` | | | `ip` | `ip` | `range` | | | `geo_point` | `geo_point` | | | -| `object` | `json` | | | -| `nested` | `json` | | | +| `object`, `nested`, `flattened`, anything else | `json` (nullable) | | | The converter is invoked when creating a Source from an existing Elastic data source -- it bootstraps the schema so the user doesn't start from diff --git a/docs/data-plane/sigma.md b/docs/data-plane/sigma.md index 1ec5d45b9..3c25e6552 100644 --- a/docs/data-plane/sigma.md +++ b/docs/data-plane/sigma.md @@ -12,8 +12,8 @@ Sigma is off by default: every path under `/api/v1/sigma` answers 404 until the engine runs with `sigma.enabled` set, which is `DFE_ENGINE_SIGMA_ENABLED=true` in its environment. A `sigma` entry in a source's `views` is a field-map view and deploys either way. DFE turns community + commercial Sigma rules into live DFE detections. External -feeds are pulled into an operator-curated catalogue; the operator SELECTS the -rules they want; propagation then generates a DFE detection rule and a per-source +feeds are pulled into an operator-curated catalogue. The operator SELECTS the +rules they want. Propagation then generates a DFE detection rule and a per-source hunt for each selection, running against a per-source Sigma VIEW that exposes the Sigma-aligned columns. Everything is stored in the deploy repo via gitcrud, so every change is one attributed commit (survivable + audited). @@ -45,7 +45,7 @@ signals, tagged with a provenance `origin`). Three kinds ship: A provider config holds only a secret PATH, never a value. Reach is confined at register time (`_validate_provider_reach`): a `local_files` directory must resolve -under the config dir; a `git_repo` url / `valhalla` base_url must be http(s), must +under the config dir. A `git_repo` url / `valhalla` base_url must be http(s), must not target a loopback/link-local host (the `169.254.169.254` metadata endpoint is link-local), and - when `DFE_ENGINE_SIGMA_ALLOWED_HOSTS` is set - must be allow-listed. @@ -55,7 +55,7 @@ link-local), and - when `DFE_ENGINE_SIGMA_ALLOWED_HOSTS` is set - must be allow- gitcrud catalogue (`catalog.py`). The upsert is idempotent and local-edit preserving: an operator can edit a catalogued rule and a later re-sync keeps that edit (via `deep_merge`, where the operator's lists replace upstream's), reporting drift when the upstream -changed. The operator then SELECTS the rules to activate; selection is its own +changed. The operator then SELECTS the rules to activate. Selection is its own gitcrud object. ## Propagation - selection to live detections @@ -70,7 +70,7 @@ hand-edits are detected. Deselecting a rule leaves its binding STALE until prune The conversion is the [pysigma-backend-clickhouse](https://pypi.org/project/pysigma-backend-clickhouse/) backend, which owns the Sigma modifiers, wildcards and LIKE, regex and CIDR escaping. `sigma_backend_clickhouse.py` subclasses it to emit the bare condition, quote field names through the engine's column quoting and match a text-indexed column through `lower(col) LIKE`. What does not convert is recorded as `failed` in the report: - A correlation rule, which is a whole query and not a condition. -- A keyword-only rule (`- needle`), which has no column to search. About 3% of the SigmaHQ set. +- A keyword-only rule (`- needle`), which has no column to search. Plain string equality is case-sensitive, where Sigma matches case-insensitively (`OriginalFileName: CertUtil.exe` misses `certutil.exe`). A JSON-derived view column with no `type` reads as `Nullable(String)`, so `|contains`, `|startswith`, equality and lists work whatever type each row sent, and a missing field is still NULL. Numbers in an equality or a list are compared as quoted text, which suits that column. A `|gt` or `|lt` needs a numeric `type` such as `UInt32`: the view reads it with `toUInt32OrNull`, so a row whose value is not a number reads NULL and matches nothing. Any other declared type reads through `accurateCastOrNull`, with the same NULL for a value that does not convert. @@ -98,7 +98,7 @@ Every ClickHouse query the engine issues carries a JSON `DfeQueryTags` `log_comm (service / feature / tenant / kind / id), stamped by the `ClickHouseClientWrapper` from a request-scoped `tags_context` (feature = the API route) or, for a hunt run, directly by the worker (feature=hunts, id=the hunt). A materialised view over -`system.query_log` (`clickhouse/query_log_archive.py`) lifts the tagged rows into +`system.query_log` (`src/dfe_engine/clickhouse/query_log_archive.py`) lifts the tagged rows into `.query_log_archive`, exploding the JSON into typed columns. dfe-schemas declares the table as optional. `GET /api/v1/queries/cost-leaderboard` ranks the heaviest cost consumers (query count, read rows/bytes, duration, peak memory) grouped by the attribution id - the diff --git a/docs/data-plane/source-definition.md b/docs/data-plane/source-definition.md index 7c36c33ed..c3d571c44 100644 --- a/docs/data-plane/source-definition.md +++ b/docs/data-plane/source-definition.md @@ -11,7 +11,7 @@ One YAML file per source, and it carries the whole stream: identity, where the records come from, whether they are transformed, the table they land in, and the standard views they expose. What a Source IS lives in -[source.md](source.md); this is the file itself. +[source.md](source.md). This is the file itself. ## A full source @@ -148,7 +148,7 @@ receiver's router splits on. It is not a ClickHouse column reference: a `_json.` prefix would be read as a first path segment that no payload has. `ttl_days` is optional. A source that leaves it unset gets the deployment -default (the admin's console value, else `DFE_ENGINE_CLICKHOUSE_DEFAULT_TTL_DAYS`, 90 days shipped, 0 = none); a value +default (the admin's console value, else `DFE_ENGINE_CLICKHOUSE_DEFAULT_TTL_DAYS`, 90 days shipped, 0 = none). A value here wins over that default and over the table's dfe-schemas definition. The next schema apply moves an existing table onto a changed value with `ALTER TABLE ... MODIFY TTL`, and shortening it expires the rows older than the new @@ -160,13 +160,12 @@ meta schema does not define is a hard error, never an appended column, and `index` is the only per-column key it may set -- `type`, `expr`, `comment` and `attribute` all resolve from the base. `additional_fields` is still the append layer. A derived schema also carries `capture_json` and `capture_raw` for this -table, both on by default; turning them off tells dfe-loader to stop populating +table, both on by default. Turning them off tells dfe-loader to stop populating `_json` and `_raw` without removing either column. Its CRUD surface is `/api/v1/schemas/definitions/derived//`. -The 2.1 keys `sigma`, `mapping_standards` and `field_mappings` were a clean -break in 2.2, not a deprecation -- a write carrying one is rejected with a -"removed in 2.2" error. Declare naming-standard views under `views` instead. +A write carrying the 2.1 keys `sigma`, `mapping_standards` or `field_mappings` is +rejected with a "removed in 2.2" error. Declare naming-standard views under `views`. ## The smallest source that works @@ -200,8 +199,8 @@ Source: filebeat ``` The type registry (dfe-schemas `registries/types.yaml`) maps primitives to ClickHouse types -and is global, not per-source. The schema section feeds the `SchemaBuilder` / -`ClickHouseSchema` pipeline. Create and migrate +and is global, not per-source. The schema section feeds `SchemaBuilderV2` +(`src/dfe_engine/schema/schema_builder_v2.py`). Create and migrate are triggered from the source's state rather than from a standalone schema config. @@ -227,7 +226,7 @@ full type system, the primitive-to-ClickHouse mapping and the use cases are in | Source event | Schema action | |-------------|---------------| | created / active | `CREATE TABLE IF NOT EXISTS` (idempotent) | -| schema YAML updated | `ALTER TABLE` via SchemaModifier (add/modify columns) | +| schema YAML updated | the schema applier (`src/dfe_engine/schema/applier.py`) adds the columns the table lacks with their indexes, and moves its TTL | | dormant | none -- the table is retained, pre-positioned for reactivation | | disabled | none -- routing and transforms turn off, the table and its data stay | | deleted | none -- the table is retained. Drop it by hand if the data is not wanted | @@ -246,17 +245,17 @@ emits the schema YAML, which the source then names. The mapping table is in | match and `when` | compile to rows scalo.rules accepts under the receiver's CEL Fast profile, at most 32 of them, else 422 `invalid_rule` with each problem's field and span. `always` only on the reserved `main` source | | compatibility profiles | each binding names a stored profile, a version it publishes and only opt-in rows it offers, one version of a profile per deployment, and every alias is unique per profile across non-disabled sources (409 `alias_conflict`) | | fetcher route | must not name its own source, and compares one field with `equals` | -| fetcher connector type | one type per source, however many connections it polls; the refusal names the source and the types | +| fetcher connector type | one type per source, however many connections it polls. The refusal names the source and the types | | flow | runnable here: the deployment offers the transport, and every app in the flow carries it | | transform | engine must be one the app manifest catalogues (`vrl`, `vector`, `elastic`) | | transform mode and detect | `mode` and `detect` only for an app whose manifest declares `mode_path` and `detect_path` (`elastic`). `detect` only with `mode: agentless`, which takes no `variant`. `detect.fallback` is required when `unmatched` is `fallback` and refused otherwise, and `candidates` holds distinct non-blank names | | catalogue entry | the intake must be one the entry arrives by, the variant one it compiles to and not one a mounted contract of the deployed transform leaves out (409 `missing_variant`), and the source name a legal label | -| schema | files exist and pass SchemaBuilder validation, and carry `_source` as a column (injected if missing) | +| schema | files exist and pass `SchemaBuilderV2` validation, and carry `_source` as a column (injected if missing) | The connector-type rule is the source-schema alignment, enforced: one source is one schema, so it is one type, and the many connections of that type - accounts, tenants, regions, each with its own credential - all land in the one table. `DFE_ENGINE_SOURCE_ALLOW_MIXED_FETCHER_TYPES=true` (`source.allow_mixed_fetcher_types`) turns the enforcement off for a deployment that knowingly wants a mixed fetcher, -and past that dial you are on your own: no suite support, no schema alignment, +and past that dial there is no suite support, no schema alignment, and dfe-fetcher's own many-to-many semantics apply to what lands in the table. diff --git a/docs/data-plane/source-detections.md b/docs/data-plane/source-detections.md index 9d0f67b26..da2a2f908 100644 --- a/docs/data-plane/source-detections.md +++ b/docs/data-plane/source-detections.md @@ -137,5 +137,5 @@ views: A converted Sigma rule becomes a DFE rule tied to the source, querying the `{source}_sigma` view. The provider pipeline that keeps the rule catalogue -current is [sigma.md](sigma.md); the mapping layer itself is +current is [sigma.md](sigma.md). The mapping layer itself is [field-mapping.md](field-mapping.md). diff --git a/docs/data-plane/source-flow.md b/docs/data-plane/source-flow.md index 1737d1c38..94547ce87 100644 --- a/docs/data-plane/source-flow.md +++ b/docs/data-plane/source-flow.md @@ -56,9 +56,9 @@ flowchart LR Direct runs the receiver to loader path alone (the second diagram). A fetcher or a transform is an app of its own per source, and addressing a pool of those over gRPC needs the mesh, which is not GA. A source that needs one is refused at save on a direct deployment, and the compile leaves it out. `GET /api/v1/system/deployment` reports `transports.per_source_apps: false` there, and each `GET /api/v1/sources/catalogue` entry carries `runnable`, `runnable_reason` and `runnable_intakes`, judged by the same resolver as the save. -The archiver keeps the RAW record as it arrived, before any transform. On the bus it reads the landing topic; on direct the receiver fans the record out to it beside the loader. A profile that apps.yaml does not offer dfe-archiver in, or that does not run it unasked (`default_in`) while no archiver instance is deployed, refuses `archive: true` at save, naming the profile, while a source stored with it before still reads and compiles. +The archiver keeps the RAW record as it arrived, before any transform. On the bus it reads the landing topic. On direct the receiver fans the record out to it beside the loader. A profile that apps.yaml does not offer dfe-archiver in, or that does not run it unasked (`default_in`) while no archiver instance is deployed, refuses `archive: true` at save, naming the profile, while a source stored with it before still reads and compiles. -**We parse mechanical transport at max efficiency at the edge.** dfe-receiver and dfe-fetcher parse the transport layer (a syslog header, a beats envelope) into JSON with the payload in `message`. Content-dependent parsing, such as a vendor body through grok, happens in a transform reading `message`. Raw capture (`_raw`) is off by default because it doubles every record. A first-hop parse bug is fixed in the receiver or fetcher, never by turning raw capture on. +The first hop parses only the transport. dfe-receiver and dfe-fetcher parse the transport layer (a syslog header, a beats envelope) into JSON with the payload in `message`. Content-dependent parsing, such as a product's message body through grok, happens in a transform reading `message`. Raw capture (`_raw`) is off by default because it doubles every record. A first-hop parse bug is fixed in the receiver or fetcher, never by turning raw capture on. ## Two transports, one per deployment @@ -68,12 +68,12 @@ A source declares `transport: bus` or `transport: direct`. Left unset, it takes |---|---|---|---| | dfe-docker slim | direct | gRPC | one replica of each | | dfe-docker single | bus | Kafka topics on one broker | one replica of each | -| slim | direct | gRPC | one replica; the KEDA ceiling of two would pin senders to pods | +| slim | direct | gRPC | one replica. The KEDA ceiling of two would pin senders to pods | | single | bus | Kafka topics on one broker | consumer groups | | scale | bus | Kafka topics on a broker cluster | consumer groups | -| mesh | direct | gRPC | a listener per stage pool, see [../deployment/transports.md](../deployment/transports.md) | +| mesh (in the charts, refused by dfe-infra's bootstrap) | direct | gRPC | a listener per stage pool, see [../deployment/transports.md](../deployment/transports.md) | -The source records bus-versus-direct only. Which bus (Kafka today) and which direct protocol (gRPC) are deployment facts, so a second bus provider joins behind the same `transport: bus` value without a change to the source model. +The source records bus-versus-direct only. Which bus (Kafka) and which direct protocol (gRPC) are deployment facts, so a second bus provider joins behind the same `transport: bus` value without a change to the source model. ## Naming is generated, never typed @@ -81,7 +81,7 @@ The source records bus-versus-direct only. Which bus (Kafka today) and which dir |---|---|---| | landing topic | `