docs(transform-vector): correct docs against the code

README.md

@@ -7,20 +7,23 @@ Runs a Vector pipeline as a supervised transform stage in a DFE deployment.
 
 ## What it does
 
-- Vector does the transform, and the supervisor supplies the rest a stage needs: config, credentials, health, metrics and restarts. Reach for it when you need one of Vector's own transforms beyond `remap`. For VRL alone, [dfe-transform-vrl](https://github.com/hyperi-io/dfe-transform-vrl) is smaller and in-process.
+- Vector does the transform, and the supervisor supplies the rest a stage needs: config, credentials, health, metrics and restarts. Reach for it when you need one of Vector's own transforms beyond `remap`. For VRL alone, [dfe-transform-vrl](https://github.com/hyperi-io/dfe-transform-vrl) runs it in-process with no Vector binary.
 - One small YAML (pipeline name, brokers, topics, SASL and TLS) generates Vector's source, sink and observability config, so you write only the `transforms:` and `enrichment_tables:` blocks.
 - Kafka in and out (`bus`), or a direct gRPC push with no broker (`direct`). The transform you author is the same on both.
-- A crashed Vector is restarted with backoff from 1 s to 60 s, and readiness follows it.
+- A crashed Vector is restarted with exponential backoff, and readiness follows it.
 - With reload enabled, a transform change is validated, sent to Vector with a SIGHUP, and read back from Vector's own counters as applied, refused (the running config is put back) or unconfirmed. One that fails validation is logged and never sent.
 - Vector's `vector_*` metrics are merged into the supervisor's, so one endpoint and one OTLP push carry both.
 
 ## Quick start
 
 ```bash
-cargo build --release
-./target/release/dfe-transform-vector run --config config.example.yaml
+cargo build --release --bin dfe-transform-vector
+scripts/fetch-vector.sh        # caches the pinned Vector under .tmp/ and prints its path
+./target/release/dfe-transform-vector --config config.example.yaml config-check
 ```
 
+`--config` goes before the subcommand, and `run` is the default when you leave the subcommand out. To run it off the image, point `vector.binary` at the path `scripts/fetch-vector.sh` printed and `vector.config_dir` and `vector.data_dir` at directories you can write, then `./target/release/dfe-transform-vector --config config.example.yaml`.
+
 [config.example.yaml](config.example.yaml) is the annotated reference. [templates/bus.yaml](templates/bus.yaml) and [templates/direct.yaml](templates/direct.yaml) are the whole Vector topology for each transport, commented per field and runnable as they stand: copy one, replace the `dfe_transform` step, and you have a working pipeline. In a deployment the supervisor generates `sources:` and `sinks:`, and refuses any other top-level key in a transform file.
 
 | | `bus` | `direct` |
@@ -29,20 +32,20 @@ cargo build --release
 | Out | Kafka `<source>_load` | a scalo Push client to `sink.endpoint` |
 | Needs a broker | yes | no |
 
-One port, 9090 by default, carries the ops surface: `/livez`, `/readyz` (200 only while Vector is up and its sink is not stalled) and `/metrics`. Vector's own exporter defaults to `127.0.0.1:9598`, and `metrics.vector_metrics_address` stays on loopback.
+One port, 9090 by default, carries the ops surface: `/livez`, `/readyz` and `/metrics`. `/readyz` answers 503 until Vector has survived its first 500 ms, and again while it is down or its sink has held records for `metrics.sink_stall_secs` without delivering any. A bus config with no topics idles instead: Ready, with no Vector running, until the config file names a topic. Vector's own exporter defaults to `127.0.0.1:9598`. Keep `metrics.vector_metrics_address` on loopback, because nothing checks it.
 
 ## Configuration
 
-Settings load through a cascade: CLI, then `DFE_TRANSFORM_VECTOR_*` environment variables, then the working directory's `.env`, then the config file, then defaults. The transport dials take a single underscore (`DFE_TRANSFORM_VECTOR_SOURCE_TRANSPORT`, `_SOURCE_LISTEN`, `_SINK_ENDPOINT`, the SASL pairs), and a key inside a section takes two (`DFE_TRANSFORM_VECTOR_SOURCE__BROKERS`, `_SINK__TOPIC`, `_TRANSFORMS__DIR`). [docs/config-schema.yaml](docs/config-schema.yaml) lists every key with its type and default.
+The config file (`--config`, else `config.yaml` or `config.yml` in the working directory) sits over the defaults, and `DFE_TRANSFORM_VECTOR_*` environment variables sit over the file. The working directory's `.env` is read into the environment first, and a variable already set wins. The flat names take a single underscore and a comma list (`DFE_TRANSFORM_VECTOR_SOURCE_BROKERS=a:9092,b:9092`, `_SINK_TOPIC`, `_TRANSFORMS_DIR`, the SASL pairs) and beat the double-underscore form. That form reaches any key and takes a list in brackets (`DFE_TRANSFORM_VECTOR_SOURCE__BROKERS='[a:9092,b:9092]'`), and a bare comma list there fails to load. `--log-level`, `--log-format` and `--metrics-addr`, or `LOG_LEVEL`, `LOG_FORMAT` and `METRICS_ADDR`, outrank `logging.*` and `metrics.address`. [docs/config-schema.yaml](docs/config-schema.yaml) lists the app's own keys with their types and defaults.
 
-A transform file change hot-reloads when reload is enabled, after it validates. A change to `source.*`, `sink.*`, `bridge.*`, `pipeline.name`, `vector.*`, `metrics.*` or `logging.*` takes a pod restart, and the watcher logs it as such. The reload path, the readiness rules, the metrics merge and the Kafka tuning are in [docs/internals.md](docs/internals.md) and [docs/librdkafka.md](docs/librdkafka.md).
+With reload on, the default, a transform file change applies once it validates, with no restart. Any other change to the app's own keys, `dfe_source`, `pipeline.name` and `scaling.pressure_threshold` included, is logged as needing a pod restart, and no later transform change applies until the pod restarts or the edit is reverted. scalo's own sections in the file, such as `version_check`, `scaling.enabled` and `metrics.namespace`, are read once at startup: an edit there needs a restart too, and the watcher logs a successful reload without applying it. In a chart deployment a config or file set change rolls the pods anyway, because the chart hashes both into the pod template. The reload path, the readiness rules, the metrics merge and the Kafka settings are in [docs/internals.md](docs/internals.md) and [docs/librdkafka.md](docs/librdkafka.md).
 
 ## Documentation
 
 - [docs/architecture.md](docs/architecture.md): the problem, the ownership boundary and the invariants.
-- [docs/internals.md](docs/internals.md): the config engine, the subprocess manager, health and metrics, the Vector version pin, the chart.
-- [docs/migration.md](docs/migration.md): moving from the official Vector chart.
-- [docs/librdkafka.md](docs/librdkafka.md): Kafka client tuning.
+- [docs/internals.md](docs/internals.md): the config engine, hot reload, the Vector subprocess, health and metrics, dead letters, the Vector version pin, the chart.
+- [docs/migration.md](docs/migration.md): moving a pipeline off the official Vector chart.
+- [docs/librdkafka.md](docs/librdkafka.md): the librdkafka settings Vector's Kafka source and sink are given.
 - [tests/TESTING.md](tests/TESTING.md): the test tiers, and the e2e cases that need Docker and a Vector binary.
 
 The suite's published docs are at <https://dfe.docs.hyperi.io>, and this repository's pages at <https://dfe.docs.hyperi.io/dfe-transform-vector/>.

SECURITY.md

@@ -17,12 +17,10 @@ Please include:
 
 ## What to Expect
 
-- **Acknowledgment**: We will acknowledge receipt of your report within
-  5 business days
-- **Investigation**: We will investigate and keep you informed of our progress
-- **Resolution**: We will work to resolve confirmed vulnerabilities promptly
-- **Disclosure**: We will coordinate with you on an appropriate disclosure
-  timeline
+- We will acknowledge receipt of your report within 5 business days
+- We will investigate and keep you informed of our progress
+- We will work to resolve confirmed vulnerabilities promptly
+- We will coordinate with you on an appropriate disclosure timeline
 
 ## Safe Harbour
 
@@ -39,12 +37,12 @@ We will not pursue legal action against security researchers who:
 ## Recognition
 
 With your permission, we will credit you for the discovery of confirmed
-vulnerabilities. We do not currently offer monetary bounties, but we value
+vulnerabilities. We do not offer monetary bounties, but we value
 and appreciate responsible disclosure.
 
 ## Out of Scope
 
-The following are generally out of scope:
+These are generally out of scope:
 
 - Social engineering or phishing attacks
 - Denial of service (DoS/DDoS) attacks

config.example.yaml

@@ -1,5 +1,5 @@
 # =============================================================================
-# dfe-transform-vector — example configuration
+# dfe-transform-vector -- example configuration
 # =============================================================================
 #
 # Copy to config.yaml and modify as needed.
@@ -20,7 +20,7 @@ pipeline:
 # -----------------------------------------------------------------------------
 # DFE source name
 # -----------------------------------------------------------------------------
-# The one setting a working config cannot go without. It derives the platform
+# Set it, or set source.topics and sink.topic yourself. It derives the platform
 # topic names and the consumer group. With pipeline.name "my-pipeline" above:
 #   source.topics   -> ["my_source_land"]
 #   sink.topic      -> "my_source_load"
@@ -49,7 +49,7 @@ dfe_source: "my_source"
 #   auto_offset_reset: "largest"       # largest, smallest
 #   session_timeout_ms: 30000          # consumer group session timeout
 #   commit_interval_ms: 5000           # offset commit interval
-#   drain_timeout_ms: 15000            # max wait during rebalance (default: half session_timeout)
+#   drain_timeout_ms: 15000            # max wait during rebalance (Vector's default: half session_timeout_ms)
 #   topic_lag_metric: true             # expose consumer lag metric
 #   # Commit an offset, or answer a push, only once the record is delivered.
 #   acknowledgements:
@@ -69,7 +69,7 @@ dfe_source: "my_source"
 #     enabled: true
 #     # DFE_TRANSFORM_VECTOR_KAFKA_SECURITY_PROTOCOL containing SSL (SASL_SSL, SSL)
 #     # turns this on for source and sink. No value turns it off.
-#   # librdkafka overrides — production defaults baked in, override here
+#   # librdkafka overrides over the production profile (docs/librdkafka.md)
 #   # librdkafka_options:
 #   #   fetch.max.bytes: "10485760"
 #   #   queued.min.messages: "100000"
@@ -92,15 +92,15 @@ dfe_source: "my_source"
 #   socket_timeout_ms: 60000           # broker socket timeout
 #   batch:
 #     max_events: 10000                # Vector-level batch size (default 10K)
-#     # max_bytes: 8388608             # optional byte limit; on direct at most 16 MiB, 8 MiB unset
-#     timeout_secs: 1                  # flush timeout; at most 1 on direct
+#     # max_bytes: 8388608             # optional byte limit, at most 16 MiB on direct, where unset means 8 MiB
+#     timeout_secs: 1                  # flush timeout, at most 1 on direct
 #   sasl:
 #     enabled: true
 #     mechanism: "scram_sha_512"
 #     secret_dir: "/var/run/secrets/dfe-kafka"
 #   tls:
 #     enabled: true
-#   # librdkafka overrides — production defaults baked in, override here
+#   # librdkafka overrides over the production profile (docs/librdkafka.md)
 #   # librdkafka_options:
 #   #   batch.size: "8388608"
 #   #   linger.ms: "20"
@@ -123,7 +123,7 @@ dfe_source: "my_source"
 # -----------------------------------------------------------------------------
 # transforms:
 #   dir: "/etc/dfe/transforms/"
-#   # Or specify individual files (loaded in order):
+#   # Or list the files, loaded in order. Read only when dir is unset.
 #   # files:
 #   #   - "/etc/dfe/transforms/parse.yaml"
 #   #   - "/etc/dfe/transforms/enrich.yaml"
@@ -138,15 +138,16 @@ dfe_source: "my_source"
 #   binary: "/usr/local/bin/vector"
 #   data_dir: "/var/lib/vector"
 #   # Where the assembled Vector config is written. The default is the
-#   # container path; a bare install has no write access there.
+#   # container path, which a bare install cannot write.
 #   config_dir: "/var/run/vector/config"
 #   # Vector's API is unauthenticated and nothing here uses it, so it is off.
 #   # Turned on, it binds a loopback address only.
 #   api_enabled: false
 #   api_address: "127.0.0.1:8686"
 #   log_level: "info"
-#   # Version pinning — prevents silent version drift
-#   version: "0.61.0"
+#   # The Vector release the binary must report. The default is the release
+#   # the image ships, so set it only beside a binary you supply.
+#   version: "<vector release>"
 #   version_check: "strict"           # strict, warn, disabled
 #   # The supervisor runs `binary` and never fetches one, so `preshipped` is
 #   # the only value accepted. To run a different Vector, point `binary` at it
@@ -174,23 +175,24 @@ dfe_source: "my_source"
 # -----------------------------------------------------------------------------
 # Logging
 # -----------------------------------------------------------------------------
-# --log-level/LOG_LEVEL and --log-format/LOG_FORMAT outrank these;
-# --verbose/--quiet outrank everything.
+# --log-level/LOG_LEVEL and --log-format/LOG_FORMAT outrank these, and
+# --verbose/--quiet outrank everything. auto is JSON when
+# OTEL_EXPORTER_OTLP_ENDPOINT is set, else text on a terminal or under CI, else JSON.
 logging:
   level: "info"                        # trace, debug, info, warn, error
-  format: "auto"                       # json, text, auto (text on a tty)
+  format: "auto"                       # json, text, auto
 
 # -----------------------------------------------------------------------------
 # Scaling pressure
 # -----------------------------------------------------------------------------
 # NOTHING IN THIS APP READS `pressure_threshold`. It is accepted for parity
 # with the `scaling:` block every DFE service takes, and range-checked. scalo's
 # runtime reads its own `enabled` and `memory_gate_threshold` keys from this
-# block for the pressure gauge the subprocess circuit gates.
+# block for the scaling_pressure gauge. Nothing feeds that gauge a reading here,
+# so it stays 0 and the memory gate never fires.
 #
-# What actually scales this app is the chart's KEDA ScaledObject -- CPU, plus
-# any trigger a deployment adds -- gated by the Vector subprocess circuit, which
-# pins pressure to 0 while the subprocess is down (more pods cannot help a dead
-# subprocess).
+# What scales this app is the chart's KEDA ScaledObject: CPU, plus any trigger
+# a deployment adds. The subprocess circuit pins scaling_pressure to 0 while
+# Vector is down, which matters only to a trigger that reads it.
 # scaling:
 #   pressure_threshold: 0.8

docs/README.md

@@ -3,9 +3,9 @@
 How dfe-transform-vector is built and configured, and how to move a pipeline onto it from Vector.
 
 - [Architecture](architecture.md)
-- [DFE Transform Vector -- Design Document](internals.md)
-- [librdkafka Configuration Reference](librdkafka.md)
-- [Migration Guide: Vector -> dfe-transform-vector](migration.md)
+- [Internals](internals.md)
+- [librdkafka settings](librdkafka.md)
+- [Moving off the Vector chart](migration.md)
 - [config-schema.yaml](config-schema.yaml) -- the configuration as a JSON Schema (YAML)
 - [config-schema.json](config-schema.json) -- the configuration as a JSON Schema (JSON)
 - [capability-catalog.yaml](capability-catalog.yaml) -- what the app can do, as a catalogue (YAML)