Skip to content

docs(transform-vrl): correct docs against the code #151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
catinspace-au merged 1 commit into from
Nov 15, 2026
Merged

docs(transform-vrl): correct docs against the code #151

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 41 additions & 59 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ versioning and changelog generation. All commits must follow this format:
| `build` | Changes to build system or dependencies | None |
| `ci` | Changes to CI configuration | None |
| `chore` | Other changes that don't modify src or test files | None |
| `revert` | Reverts a previous commit | Varies |
| `revert` | Reverts a previous commit | Patch (0.0.X) |

### Breaking Changes

Expand All @@ -53,38 +53,18 @@ affected (e.g., `config`, `engine`, `kafka`, `pipeline`).

This project follows [Semantic Versioning 2.0.0](https://semver.org/):

- **MAJOR** (X.0.0): Breaking changes that require users to modify their code
- **MINOR** (0.X.0): New features that are backwards-compatible
- **PATCH** (0.0.X): Bug fixes and minor improvements
- MAJOR (X.0.0): breaking changes that require users to modify their code
- MINOR (0.X.0): new features that are backwards-compatible
- PATCH (0.0.X): bug fixes and minor improvements

Versions are automatically determined by semantic-release based on commit
messages. Do not manually update version numbers.

## Developer Certificate of Origin

This project uses the Developer Certificate of Origin (DCO) to ensure that
contributors have the right to submit their contributions.
This project uses the [Developer Certificate of Origin 1.1](https://developercertificate.org/). A `Signed-off-by` line on a commit certifies, for that contribution, the four statements in that text.

By making a contribution to this project, you certify that:

1. The contribution was created in whole or in part by you and you have the
right to submit it under the license indicated in the file; or

2. The contribution is based upon previous work that, to the best of your
knowledge, is covered under an appropriate open source license and you
have the right under that license to submit that work with modifications;
or

3. The contribution was provided directly to me by some other person who
certified (1), (2) or (3) and you have not modified it.

4. You understand and agree that this project and the contribution are public
and that a record of the contribution (including all personal information
you submit with it, including your sign-off) is maintained indefinitely
and may be redistributed consistent with this project or the license(s)
involved.

## How to Sign Off Your Commits
### How to Sign Off Your Commits

You must sign off each commit to indicate your acceptance of the DCO. Combine
the signoff with your conventional commit message:
Expand Down Expand Up @@ -118,11 +98,11 @@ Change License on the Change Date specified in [LICENSE](LICENSE).

## How to Contribute

1. **Fork the repository** and create your branch from `main`
2. **Make your changes** following the commit message format above
3. **Sign off your commits** with the DCO
4. **Test your changes** to ensure they work as expected
5. **Submit a pull request** with a clear description of what you've done
1. Fork the repository and create your branch from `main`
2. Make your changes following the commit message format above
3. Sign off your commits with the DCO
4. Run the tests that cover your change
5. Submit a pull request with a clear description of what you've done

### Pull Request Checklist

Expand All @@ -145,6 +125,10 @@ cargo build --release
cargo nextest run
```

## CI/CD Workflow

A merge to `main` runs CI and releases nothing by itself. A release is cut when the squash-merge commit ends with a `Release: true` trailer, or when a maintainer dispatches the CI workflow with `from-head: true`. semantic-release then reads the commit messages since the last release and picks the next version, the release commit updates `CHANGELOG.md` and `VERSION`, and hyperi-ci builds and publishes the release.

## Questions

If you have questions about contributing, please open an issue or contact us.
Expand Down Expand Up @@ -174,30 +158,29 @@ turn.
| `src/enrichment/` | Table loading (CSV, JSON, YAML, MMDB, STIX, SQLite), refresh, the custom VRL functions |
| `src/deployment.rs` | `contract()` -- the single source the Dockerfile and the released chart are generated from |
| `Dockerfile` | Generator output, not hand-authored |
| `pipelines/filebeat/` | Opt-in data bundle (212,218 bytes of VRL plus a lookup table), not engine capability |
| `tests/` | `integration/` runs mostly without Kafka, `e2e/` needs a broker, `TESTING.md` explains the modes |
| `pipelines/filebeat/` | Opt-in data bundle (one VRL file plus a lookup table), not engine capability |
| `tests/` | `integration/` runs without a broker, `e2e/` needs a broker container or a live broker, `TESTING.md` explains the tiers |
| `docs/architecture.md` | Why the service is built this way, and the invariants |

### Commands that prove a change

```bash
make check # hyperi-ci check -- quality + test, the pre-push gate
cargo nextest run --features enrichment-mmdb,enrichment-sqlite # what CI actually runs
cargo nextest run --all-features --run-ignored # adds the ignored Kafka e2e tests
cargo run -- config-check --config config.yaml # validate a config without starting
cargo nextest run --features enrichment-mmdb,enrichment-sqlite # the feature set CI tests with
cargo nextest run --all-features --run-ignored all --ignore-default-filter # the full tier, ignored tests included
cargo run --bin dfe-transform-vrl -- --config config.yaml config-check # validate a config without starting
cargo run --bin dfe-transform-vrl -- emit-dockerfile > Dockerfile # regenerate the Dockerfile
```

`cargo run` needs `--bin`, because `pgo-driver` is a second binary, and `--config` goes before the subcommand.

Green lies here in three ways, and all three are on by default.

`default = []` in `Cargo.toml`, so a bare `cargo nextest run` does not compile the MMDB or SQLite enrichment tests in at all. `.hyperi-ci.yaml` adds `enrichment-mmdb` and `enrichment-sqlite` to both the test and the build feature sets -- the build too, because the Dockerfile only copies the binary, so a default build ships a container that rejects `type: mmdb` and `type: sqlite` tables at startup.

Kafka-dependent tests call `skip_if_no_kafka!()` and skip cleanly when no broker
is reachable. They report as not-failed, which is not the same as proven.
The core-tier e2e tests run on broker containers scalo's test support starts. Without Docker they skip locally and fail only under `CI`, so a local run without Docker is green without having reached a broker.

The Kafka e2e tests are `#[ignore]` and need `--run-ignored` plus a broker, and the other e2e tests run by default and start their own containers. Set
`TEST_MODE=docker` for the dfe-docker infra profile on `localhost:19092`, or
`TEST_MODE=remote` for the cluster endpoints.
The tests that may reach a broker somebody else owns are `#[ignore]` and run only under `--run-ignored all` or `only`. They take a live broker first when `KAFKA_BROKERS` answers. `TEST_MODE=docker` points the harness at `localhost:19092` with no auth, the port dfe-docker publishes its broker on, and otherwise it reads `KAFKA_BROKERS`, `KAFKA_SECURITY_PROTOCOL` and the `KAFKA_SASL_*` names from the environment or the repo's `.env`.

One more: the push trigger in `.github/workflows/ci.yml` carries
`paths-ignore: docs/**, **.md`, so a docs-only push runs no jobs. The
Expand All @@ -208,32 +191,34 @@ checked.

| Don't | Do | Why |
|---|---|---|
| Hand-edit `Dockerfile`, or commit a chart | Fix `src/deployment.rs::contract()` and regenerate | The Dockerfile is generator output and the release assembles the chart from the contract, so a hand edit is reverted or never ships, and `test_every_contract_secret_env_var_reaches_the_config` holds every declared name to the field it spells |
| Hand-edit `Dockerfile`, or commit a chart | Fix `src/deployment.rs::contract()` and regenerate | The Dockerfile is generator output, and `checked_in_dockerfile_matches_emit_dockerfile` fails on a hand edit. The release assembles the chart from the contract, so a committed chart never ships, and `test_every_contract_secret_env_var_reaches_the_config` holds every declared name to the field it spells |
| Bump scalo and leave `release.helm.library` behind | Move `release.helm.library` in `.hyperi-ci.yaml` to the same scalo version | A scalo-service release renders only the contract version its scalo release writes |
| Turn the Kafka lag trigger back on in `contract()` | Leave `KafkaLagTrigger::disabled()` and scale on CPU plus scaling pressure | Lag rises when a downstream stage breaks, so a lag trigger adds pods that wait on the same broken stage. The lag trigger was also the one that kept reading a kafka block this app's values do not have |
| Put a scalo section (`metrics`, `logger`, `scaling`, `worker_pool`, `batch_processing`, `self_regulation`, `version_check`, `otel_tracing`) in a `config.yaml` the binary finds in its working directory | Pass the file with `--config`, or set the section through the env layer on a **double** underscore | scalo reads the `--config` file as its settings layer but finds other files only by fixed base name (`settings.yaml`, `defaults.yaml`), so a working-directory `config.yaml` reaches the wrapper alone. Full list under [Environment variable overrides](docs/internals.md#environment-variable-overrides) |
| Turn the Kafka lag trigger back on in `contract()` | Leave `KafkaLagTrigger::disabled()`, so the contract scales on CPU and a deployment adds the scaling-pressure trigger | Lag rises when a downstream stage breaks, so a lag trigger adds pods that wait on the same broken stage. The lag trigger also reads `config.kafka.*`, which this app's values do not have |
| Put a scalo section (`metrics`, `logger`, `memory`, `scaling`, `worker_pool`, `batch_processing`, `self_regulation`, `version_check`, `otel_tracing`) in a `config.yaml` the binary finds in its working directory | Pass the file with `--config`, or set the section through the env layer on a **double** underscore | scalo reads the `--config` file as its settings layer but finds other files only by fixed base name (`settings.yaml`, `defaults.yaml`), so a working-directory `config.yaml` reaches the wrapper alone. Full list under [Environment variable overrides](docs/internals.md#environment-variable-overrides) |
| Trust `pipeline.batch_size`, `pipeline.batch_timeout_ms`, `sink.key_field` or `source.commit_interval_ms` | Size a chunk with `batch_processing.max_chunk_size` | `config::INERT_SETTINGS` -- accepted, validated, reaching nothing. Table under [Reload](docs/internals.md#reload) |
| Call a bare `cargo nextest run` green | Pass `--features enrichment-mmdb,enrichment-sqlite` | `default = []`, so the MMDB and SQLite tests are not compiled in and the run is green without having tested them |
| Set `sasl.enabled` with an empty username or password | Supply both, or neither | librdkafka's SCRAM check is a NULL check that an empty string passes, so that pod authenticated against nothing and still reported Ready. Refused at startup now |
| Add a serialise path that redacts by field name | Keep the password a `scalo::SensitiveString` | Redaction is by type on every path -- `Debug`, the `/config` dump, the emitted schema. The figment round-trip has to be wrapped in `expose_during` or a file-sourced password reaches the broker as the literal `***REDACTED***` |
| Size the container from steady state when the program is large | Leave headroom for the compile | Compilation scales with the VRL, and the bundled filebeat program is OOM-killed under a 32 MiB limit. Below the floor the kernel kills the process mid-compile, seen as an exit-137 restart loop with nothing in the log. `engine::budget` refuses first and names all three numbers |
| Set `sasl.enabled` with an empty username or password | Supply both, or neither | librdkafka's SCRAM check is a NULL check that an empty string passes, so the pod would authenticate against nothing and still report Ready. `validate` refuses it at startup |
| Add a serialise path that redacts by field name | Keep the password a `scalo::SensitiveString` | Redaction is by type on every path -- `Debug`, the `config-check` dump, the emitted schema. The figment round-trip has to be wrapped in `expose_during` or a file-sourced password reaches the broker as the literal `***REDACTED***` |
| Size the container from steady state when the program is large | Leave headroom for the compile | Compilation scales with the VRL, and below the floor `engine::budget` computes the kernel kills the process mid-compile, seen as an exit-137 restart loop with nothing in the log. `engine::budget` refuses first and names all three numbers |
| Pick the optimisation tier with `publish-target` | Use `build.skip_optimize` in `.hyperi-ci.yaml`, or the `skip-optimize` dispatch input for one run | hyperi-ci ignores `publish-target`. A build that ships gets PGO and BOLT from `scripts/pgo-workload.sh` unless optimisation is skipped, and a stable release that skips it is refused unless the dispatch passes `release-unoptimized: true` |

### Where this sits

Inbound -- what this repo depends on:

- **scalo-rs** (`cargo-dep`). `Cargo.toml` declares the `scalo` crate by range, a
- scalo-rs (`cargo-dep`). `Cargo.toml` declares the `scalo` crate by range, a
second range covering the dev dependency. A scalo release arrives through that
range: `cargo update -p scalo` and rebuild if it admits the version, widen the
range first if not.
- **scalo-rs** (`generated-file`, lockstep). The `Dockerfile` is written by
scalo's generator from this crate's `deployment::contract()`, at contract
schema version 4. A generator or schema change upstream means regenerating
with the command in the file's own header and committing the diff. The
released chart is assembled on the scalo-service library chart at
`release.helm.library`, which moves with the scalo version in `Cargo.toml`.
- **dfe-infra** (`apps.yaml`, deploy-time authority). The suite manifest declares
- scalo-rs (`generated-file`, lockstep). The `Dockerfile` is written by
scalo's generator from this crate's `deployment::contract()`, and its header
names the generator and the contract schema version. A generator or schema
change upstream means regenerating with
`cargo run --bin dfe-transform-vrl -- emit-dockerfile > Dockerfile` and
committing the diff. The released chart is assembled on the scalo-service
library chart at `release.helm.library`, which moves with the scalo version in
`Cargo.toml`.
- dfe-infra (`apps.yaml`, deploy-time authority). The suite manifest declares
what this app is: `multiplicity: per_config` (one deployment per source config,
never a singleton), `scale_deployed: true`, both transports, and a source
binding deriving `{source}_land` in, `{source}_load` out and
Expand All @@ -242,10 +227,7 @@ Inbound -- what this repo depends on:

Outbound -- what depends on this repo:

- **dfe-infra** (`image-pin`, lockstep). Pins this repo's ghcr image as a tag plus
- dfe-infra (`image-pin`, lockstep). Pins this repo's ghcr image as a tag plus
the digest that makes it immutable. A release here means bumping the tag and
re-resolving the digest there, with `check_versions_drift.py` confirming the
chart's appVersion and the digest mirror agree.

dfe-transform-splack runs this repo's engine image with a rule-config chart of
its own. It is out of suite scope and no work here is driven by it.
15 changes: 11 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,19 +11,24 @@ Runs your VRL transforms over a DFE deployment's event stream, in-process.
- Kafka in and out, or a direct gRPC push on a deployment with no broker. The transform you write is the same on both.
- At-least-once by default: the source is acknowledged only once the sink has the transformed records. `source.acknowledgements.enabled: false` acknowledges at receipt instead.
- Enrichment tables from CSV, JSON, YAML and STIX, read from VRL, plus MMDB and SQLite with the `enrichment-mmdb` and `enrichment-sqlite` build features. A table with a `refresh` setting reloads while the service runs. MMDB loads once at start.
- The host and network functions (`get_env_var`, `get_hostname`, `http_request`, `dns_lookup`, `reverse_dns`) do not compile, so a transform cannot reach outside the process.
- The host and network functions (`get_env_var`, `get_hostname`, `http_request`, `dns_lookup`, `reverse_dns`) do not compile, so a transform cannot read the host's environment or reach the network.

JSON is the only payload format. A record that is not JSON is dropped, counted in `records_error_total{stage="deserialise"}`, and not redelivered. For Vector's own transforms beyond `remap`, use [dfe-transform-vector](https://github.com/hyperi-io/dfe-transform-vector).

## Quick start

On Linux the build needs `protoc`, a C toolchain, `pkg-config` and librdkafka. scalo compiles its gRPC `.proto` files, `aws-lc-sys` compiles C, and `rdkafka-sys` links librdkafka through pkg-config. It refuses a librdkafka older than the version after the `+` in its `Cargo.lock` entry, and a Debian or Ubuntu package can be older than that, so there [build librdkafka from source](https://github.com/confluentinc/librdkafka#build-from-source) first.

```bash
sudo apt-get install -y protobuf-compiler build-essential pkg-config
cargo build --release
cp config.example.yaml config.yaml # set the brokers, topics and transforms.dir
./target/release/dfe-transform-vrl config-check --config config.yaml
cp config.example.yaml config.yaml # uncomment source, sink and transforms, then set the brokers, topics and transforms.dir
./target/release/dfe-transform-vrl --config config.yaml config-check
./target/release/dfe-transform-vrl --config config.yaml
```

`--config` goes before the subcommand, or `config-check` refuses it as an unexpected argument.

The smallest config names the source, a directory holding at least one `.vrl` file, and the sink:

```yaml
Expand Down Expand Up @@ -53,9 +58,11 @@ del(.message)

Health and metrics answer on port 9090: `/livez`, `/readyz` (503 while memory is under pressure or a transport reports itself unhealthy, and 200 once the pipeline runs, an idle one included) and `/metrics`.

Outside the image, set `dlq.file.path` to a writable directory. The default, `/var/spool/scalo/dlq`, is one the image creates, and without a writable one the service logs that the DLQ has no backend and holds any dead letter unacknowledged at its source.

## Configuration

[config.example.yaml](config.example.yaml) is the annotated reference, and [docs/config-schema.yaml](docs/config-schema.yaml) lists every key with its type and default. The big dials have flat environment overrides for Kubernetes, `DFE_TRANSFORM_VRL_SOURCE_BROKERS`, `DFE_TRANSFORM_VRL_SINK_TOPIC`, `DFE_TRANSFORM_VRL_TRANSFORMS_DIR` and the SASL pair among them. The whole list, Kafka over TLS with a private CA and the scalo sections that share the file on a double underscore are under [Operating notes](docs/internals.md#operating-notes) in docs/internals.md, and the settings that parse but reach nothing under [Reload](docs/internals.md#reload).
[config.example.yaml](config.example.yaml) is the annotated reference, and [docs/config-schema.yaml](docs/config-schema.yaml) lists every key of the transform's own sections with its type and default. The big dials have flat environment overrides for Kubernetes, `DFE_TRANSFORM_VRL_SOURCE_BROKERS`, `DFE_TRANSFORM_VRL_SINK_TOPIC`, `DFE_TRANSFORM_VRL_TRANSFORMS_DIR` and the SASL pair among them. The whole list, Kafka over TLS with a private CA and the scalo sections that share the file on a double underscore are under [Operating notes](docs/internals.md#operating-notes) in docs/internals.md, and the settings that parse but reach nothing under [Reload](docs/internals.md#reload).

`pipelines/filebeat/` ships a pure-VRL port of the DFE 2.1 filebeat templates (Cisco Meraki, IOS, Umbrella) with its `timezones.csv` table. Opt in by pointing `transforms.dir` and `enrichment_tables` at it. dfe-transform-elastic carries the same sources as compiled Rust.

Expand Down
Loading
Loading