Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ deployment asks for it.
| `src/deployment.rs`, `src/vector-layer.dockerfile` | `contract()`, the source the Dockerfile and the released chart are generated from, and the two-binary image override spliced onto scalo's generated Dockerfile |
| `templates/bus.yaml`, `templates/direct.yaml` | Whole runnable Vector topologies, one per transport, commented per field |
| `pipelines/filebeat/` | The shipped filebeat pipeline |
| `docs/` | `architecture.md` for the shape and the invariants, `internals.md` for field-by-field depth, `migration.md`, `librdkafka.md`, the generated `config-schema.*` and `capability-catalog.*` |
| `docs/` | `architecture.md` for why the service is built this way and the invariants, `internals.md` for field-by-field depth, `migration.md`, `librdkafka.md`, the generated `config-schema.*` and `capability-catalog.*` |
| `tests/` | `integration`, `e2e`, `smoke`, and `TESTING.md` for how the broker and Vector binary are resolved |
| `scripts/fetch-vector.sh`, `scripts/pgo-workload.sh` | Downloads the pinned Vector for tests, and drives the PGO/BOLT workload |

Expand Down Expand Up @@ -266,7 +266,7 @@ Outbound -- what depends on this repo:
Two more relationships that are real but are NOT declared edges in the suite
graph, so no gate enforces them:

- **dfe-infra `apps.yaml`** is the SSoT for this app's shape -- multiplicity,
- **dfe-infra `apps.yaml`** is the SSoT for how this app is deployed -- multiplicity,
scaling, transports, endpoints and which compiler derives its routing.
dfe-engine reads that manifest, so adding or changing the app is a manifest
edit, never an engine release.
Expand Down
4 changes: 2 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ already does the per-record transform work well, so building another transform
engine was never the question. The question was the other half.

Vector on its own is opaque to the platform around it. It has its own config
shape, its own metrics endpoint, its own process lifecycle and its own idea of
format, its own metrics endpoint, its own process lifecycle and its own idea of
what "started" means. Every other DFE app presents one control surface: a
big-dial YAML in the config registry, one ops port carrying `/livez`, `/readyz`
and `/metrics`, an OTLP push, a consumer-lag scaling signal and a Helm chart
Expand Down Expand Up @@ -55,7 +55,7 @@ question asked of the wrong layer.
| dfe-engine | What the app IS -- it reads `dfe-infra/apps.yaml`, compiles the Helm values and holds the config in its registry |
| dfe-infra | The image pin and the GitOps deployment |

Nothing about this app's shape is coded into dfe-engine. It is a `per_config`
Nothing about how this app is deployed is coded into dfe-engine. It is a `per_config`
app with `scale_deployed: true`, declared in `dfe-infra/apps.yaml` -- one
deployment per source config, not a singleton, and KEDA drives the replica
count. Its `default_in` list names only `docker-single`, so every other deploy
Expand Down
6 changes: 3 additions & 3 deletions docs/internals.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

dfe-transform-vector wraps Vector.dev as a managed subprocess to provide **Kafka -> transform -> Kafka** pipelines that the DFE platform manages, deploys and observes the same way as dfe-loader and dfe-receiver.

Vector is powerful but opaque. This wrapper makes it behave like every other DFE service: the same health contract, metrics shape and scaling signal, a chart generated from its deployment contract, and the same Argo CD lifecycle. The app's shape is declared in dfe-infra's `apps.yaml`, not coded into dfe-engine.
Vector is powerful but opaque. This wrapper makes it behave like every other DFE service: the same health contract, metric names and scaling signal, a chart generated from its deployment contract, and the same Argo CD lifecycle. How the app is deployed is declared in dfe-infra's `apps.yaml`, not coded into dfe-engine.

---

Expand Down Expand Up @@ -663,7 +663,7 @@ The mounted config is the chart's `config` with `configOverrides` merged over it

### 9.3 Parity With dfe-loader/dfe-receiver Charts

All three are generated on the same scalo-service library chart, so the shape is shared:
All three are generated on the same scalo-service library chart, so they share the same features:

| Feature | dfe-loader | dfe-receiver | dfe-transform-vector |
|---|---|---|---|
Expand Down Expand Up @@ -700,7 +700,7 @@ The app is in the default composition of `docker-single` only. Elsewhere an oper

| Concern | Owner | Notes |
|---|---|---|
| App shape | dfe-infra `apps.yaml` | Multiplicity, scaling, transports, source binding |
| App deployment | dfe-infra `apps.yaml` | Multiplicity, scaling, transports, source binding |
| Config schema + validation | Wrapper (Rust) | `docs/config-schema.json` is generated from the config structs. dfe-engine judges a stored config against that mounted contract |
| K8s manifests | Chart generated from `contract()` | On the scalo-service library chart (section 9) |
| Config assembly (big dials → Vector YAML) | Wrapper (Rust) | The core novel work |
Expand Down
Loading