Skip to content

docs(transform-vector): correct docs against the code - #164

Merged
catinspace-au merged 4 commits into
mainfrom
fix/docs-reality
Nov 15, 2026
Merged

catinspace-au merged 4 commits into
mainfrom
fix/docs-reality

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

README.md, CONTRIBUTING.md and SECURITY.md now match the code. So do every page under docs/, pipelines/filebeat/README.md, tests/TESTING.md and config.example.yaml's comments.

References: the development binary, scalo-rs v2.14.30, Vector 0.61.0, librdkafka 2.12.1, dfe-infra and hyperi-ci.

The ledger has 246 rows: PASS 118, FAIL 114, UNCHECKABLE 14. All 114 FAIL rows are fixed or cut, 21 of them prose only.

  • --config goes before the subcommand: run --config exits 2. Outside the image, the quick start also needs vector.binary, vector.config_dir and vector.data_dir.
  • A comma list in DFE_TRANSFORM_VECTOR_SOURCE__BROKERS fails to load. Flat names take precedence. DFE_TRANSFORM_VECTOR_CONFIG has no effect.
  • A bus config with no topics stays Ready while idle, with no Vector running.
  • A change to the app's own keys outside transforms blocks later transform reloads until a restart. A scalo-section edit logs a successful reload. The chart rolls the pods on a config or file set change.
  • The migration cutover has a separate output topic. It states where the consumer group starts and when to drain.
  • The docs describe dfe-infra's pressure trigger on three profiles. The contract has dlq among its writable paths.
  • librdkafka.md documents the keys Vector writes itself and the enable.idempotence override. It corrects the socket.nagle.disable default. Unused profiles are cut. Only this app reads the central file.
  • The CI step, the Dockerfile pattern and the parity table are cut.
  • CONTRIBUTING links DCO 1.1. A release needs Release: true or from-head. revert bumps a patch.
  • No code changes and no release trailer.
114 corrected rows
Page Row Was Now
README.md R5 dfe-transform-vrl "is smaller and in-process" runs VRL in-process with no Vector binary (vrl Dockerfile: one COPY)
README.md R11 run --config config.example.yaml --config first, fetch-vector.sh, the binary and the two directories
README.md R17 "/readyz (200 only while Vector is up and its sink is not stalled)" when it answers 503, and the idle case
README.md R18 metrics.vector_metrics_address "stays on loopback" keep it there, nothing checks
README.md R20 Single underscore for transport dials, two for a key in a section, SOURCE__BROKERS flat names with comma lists, the double form with brackets
README.md R21 config-schema.yaml "lists every key" the app's own keys
README.md R23 Restart list source, sink, bridge, pipeline.name, vector, metrics, logging the app's own keys, the block on later reloads, scalo's sections, the chart roll (_deployment.tpl:48-53)
CONTRIBUTING.md C2 revert "Varies" Patch (0.0.X)
CONTRIBUTING.md C4 Semantic Versioning bold leads archiver's plain items
CONTRIBUTING.md C5 DCO block link to DCO 1.1, archiver wording
CONTRIBUTING.md C8 How to Contribute bold leads, to ensure they work Corrected
CONTRIBUTING.md C9 "When your pull request is merged ... This happens automatically" Release: true or from-head, archiver wording
CONTRIBUTING.md C11 "an empty default_in" Corrected
CONTRIBUTING.md C13 src/config/ list Corrected
CONTRIBUTING.md C17 pipelines/filebeat "The shipped filebeat pipeline" not in the image
CONTRIBUTING.md C20 --test e2e # needs Docker or a live broker and a Vector binary
CONTRIBUTING.md C21 "Eight #[ignore] cases (seven ... one) ... six ... one" no count
CONTRIBUTING.md C22 "nothing has run Vector's own validator" before --run-ignored all the variants left unchecked
CONTRIBUTING.md C24 filebeat_kafka "skips without a sibling repo" Corrected
CONTRIBUTING.md C26 Dockerfile row, "reverted or never ships" names the test
CONTRIBUTING.md C34 auto-commit off: "stored offsets never reach the broker" Corrected
CONTRIBUTING.md C35 /readyz "is those two signals and nothing more" Corrected
CONTRIBUTING.md C38 dfe-engine "leaf-name fallback (password, secret, token, api_key, private_key, passphrase)" name-based fallback
CONTRIBUTING.md C40 Bold-lead edge bullets Corrected
CONTRIBUTING.md C41 image-pin "Bump the tag and re-resolve the digest" Corrected
CONTRIBUTING.md C43 vrl relationship "the case skips without it" Corrected
SECURITY.md S2 "currently", "The following", a bold-lead run word-identical to vrl and fetcher main
docs/README.md D2 Link titles ("Design Document", "Migration Guide: Vector -> ...") match the retitled pages
docs/architecture.md A3 Every DFE app has "a consumer-lag scaling signal and a Helm chart compiled by dfe-engine" Corrected
docs/architecture.md A11 emit_dockerfile() "appends ... with the version substituted in" Corrected
docs/architecture.md A12 "The other five DFE Rust apps" count dropped
docs/architecture.md A16 The mermaid diagram valid Mermaid diagram
docs/architecture.md A17 Files: source, transforms, sink, observability. Only transforms: is hand-written Corrected
docs/architecture.md A22 Only transforms hot-reload, "the metrics labels" set at startup, the misclassification sentence labels and history cut. The block on later reloads and the scalo sections added
docs/architecture.md A23 Direct push holds 18/16.5/15 s. An oversize record "released dropped and counted" oversize goes to the DLQ, dropped only with it off
docs/architecture.md A28 "/readyz proves the subprocess exists, not that it carries traffic ... Proving traffic would mean reading Vector's API, which the supervisor does not do" Corrected. "routinely takes longer" now "a large program can take longer" (UNCHECKABLE)
docs/architecture.md A33 "The contract is schema version 4: a 120 s ... and a 90 s grace period", writable paths data and run emit-contract prints them, dlq added
docs/architecture.md A36 1614 words, two sections over 400 passes lint
docs/internals.md I1 Purpose: "Kafka -> transform -> Kafka pipelines" Purpose corrected
docs/internals.md I2 "the same health contract, metric names and scaling signal" Cut
docs/internals.md I4 Architecture box: box characters, "Health polling", bus-only path, "/etc/dfe-transform-vector/transforms/ (empty dir from image)" Replaced by the file table
docs/internals.md I5 Dead letters: a record "the next stage refuses for good" to the DLQ, rejected dropped with the DLQ off Corrected, gaps documented
docs/internals.md I6 Endpoints table: /readyz "200 OK only while the Vector child process is up" Corrected. /livez and /metrics unchanged (scalo metrics/mod.rs:1085-1093)
docs/internals.md I7 Liveness vs readiness block with arrows Prose
docs/internals.md I12 Wrapper metrics list Corrected
docs/internals.md I18 dfe-infra "add no pressure trigger in any profile, so this app scales on CPU alone" Corrected
docs/internals.md I20 The app "loads its own config via the standard scalo 7-layer cascade" with settings.yaml and defaults.yaml the four layers
docs/internals.md I21 The example config sink SASL and TLS. config-check passes. assemble gives the sink SASL_SSL and SECRET[dfe_sink_sasl.*]. Semicolon removed. transforms.dir uses a file set's mountPath
docs/internals.md I22 File layout: 00_global.yaml "data_dir, api switch, SASL secret backends", 50_000_parse, 50_001_enrich, 50_002_filter Corrected
docs/internals.md I23 Generated source example Replaced by the file table and assemble
docs/internals.md I27 Wiring rule 5: outputs "injected as dfe_sink.inputs" Corrected
docs/internals.md I29 Hot-reload tree: config_reloads_total{result=...}, log "config change requires pod restart" Corrected. The block on later reloads, scalo's sections and the chart roll added
docs/internals.md I31 Lifecycle diagram valid Mermaid diagram
docs/internals.md I33 Grace 65 s, "SIGKILL child, then exit(1)", 10 s remaining Corrected
docs/internals.md I34 Crash pseudocode status.success() => break, "Max crashes: unlimited by default" Corrected. Backoff and reset unchanged
docs/internals.md I35 version_check warn: "emit metric" Corrected
docs/internals.md I36 "Vector upgrades are a container image concern, not a wrapper code concern", the cadence table, "Coupling: None" Corrected
docs/internals.md I37 Dockerfile pattern block, USER appuser Cut
docs/internals.md I39 Update steps: cargo run -- emit-dockerfile, "only the vector and runtime stages rebuild", "CI runs integration tests against the new Vector" Corrected, checksums added
docs/internals.md I41 Compatibility CI step Cut
docs/internals.md I43 Failure table: bad config "+ metric" at startup, the DAG message, Kafka unreachable "Scaling pressure stays low (no lag movement)", mismatch "log + metric" Corrected
docs/internals.md I45 Graceful degradation: "prioritizes", "defense" Cut
docs/internals.md I46 Startup sequence: registry published at step 10, "Pydantic-equivalent", ops listener "before step 1", section sign, en dash Corrected
docs/internals.md I47 Data safety: source acknowledgements: true, disk buffers "survive pod restarts via PVC", "does not touch the data path" Corrected
docs/internals.md I48 MemoryGuard "Not applicable ... does not buffer data", "pattern D ... DFE remediation guide" Corrected
docs/internals.md I50 What the contract fixes: writable paths data and run, the literal values emit-contract prints them
docs/internals.md I52 Parity table: ConfigMap config.yaml for dfe-loader Cut
docs/internals.md I53 Multi-instance block, consumer group dfe-transform-vector-syslog Shortened to prose
docs/internals.md I54 Boundary table: "Vector binary version ... Decoupled from wrapper releases" Cut
docs/internals.md I55 Not in scope: "Transform library/marketplace -- future consideration, not MVP", topic creation by dfe-engine Cut
docs/internals.md I56 2041 words, 12 sections, a 565-word section, a 14-bullet list, 30 banned characters passes lint
docs/librdkafka.md L2 "Only set values that differ from librdkafka defaults" Cut
docs/librdkafka.md L4 Every other baseline setting "reverts to the librdkafka default" Corrected
docs/librdkafka.md L7 (the keys Vector writes from its own fields) Added, with the linger.ms alias note
docs/librdkafka.md L8 Example result Corrected
docs/librdkafka.md L9 A central linger.ms=50 gives linger.ms=50 with the instance's 20 still set Corrected
docs/librdkafka.md L12 Left at default: auto.offset.reset latest, session.timeout.ms 45000, heartbeat "(session.timeout / 15)" Corrected
docs/librdkafka.md L13 CONSUMER_DEVTEST and LOW_LATENCY tables, log.connection.close default false Cut
docs/librdkafka.md L14 PRODUCER_PRODUCTION, socket.nagle.disable default false Corrected
docs/librdkafka.md L15 Producer left at default: delivery.timeout.ms 300000, batch.size "(1 MiB)", queue.buffering.max.kbytes and message.max.bytes Corrected
docs/librdkafka.md L16 EXACTLY_ONCE, LOW_LATENCY, DEVTEST producer tables Cut
docs/librdkafka.md L17 Service overrides: no enable.idempotence, "Lag is what KEDA scales on here" Corrected
docs/librdkafka.md L18 dfe-loader and dfe-receiver "No service-specific overrides currently" Cut
docs/librdkafka.md L19 "All DFE services read this file at startup" this app only
docs/librdkafka.md L22 Cached in a OnceLock, "(or future hot-reload support)" Cut
docs/librdkafka.md L24 custom.vendor.setting: "value" Cut
docs/librdkafka.md L26 scalo/src/kafka_config.rs Corrected
docs/librdkafka.md L28 9 sections, currently x2 Corrected
docs/migration.md M1 "the official timberio/vector Helm chart" Corrected
docs/migration.md M2 Before values keys, 0.48.0-debian Example placeholder
docs/migration.md M3 After: replicaCount: 3 under KEDA, no brokers keda.minReplicaCount, brokers, fileSets, secrets.kafka.existingSecret (chart values schema)
docs/migration.md M4 Transforms from "the image's /etc/dfe-transform-vector/transforms" Corrected
docs/migration.md M7 Brokers "come from" the two env vars Corrected
docs/migration.md M12 Changes table: v0.42.1, "no init container ... bake ... or mount", Hot-reload "Not supported" Corrected
docs/migration.md M13 Cutover: run both at once, separate consumer group, switch downstream separate output topic, where the group starts, drain then move, overlap stated
docs/migration.md M14 Arrows, em dashes, bold leads, 10 headings Corrected
pipelines/filebeat/README.md F2 "Point transforms.dir at this directory and the pipeline runs" Corrected
pipelines/filebeat/README.md F6 The wrapper writes the file "verbatim" Corrected
pipelines/filebeat/README.md F8 "the DFE 2.1 Kafka shape" Corrected
pipelines/filebeat/README.md F10 filebeat.yaml "byte-identical" to filebeat.vrl @ e0b7087 Corrected
tests/TESTING.md T1 standards/universal/testing.md in hyperi-ai Cut
tests/TESTING.md T3 Full command and contents Corrected
tests/TESTING.md T8 Vector binary: fetch, PATH, "Without either they skip" Corrected
tests/TESTING.md T11 "The acceptance case" Corrected
tests/TESTING.md T13 "Without one it prints a skip" Corrected
tests/TESTING.md T14 Bold-lead list Corrected
config.example.yaml E1 Header em dash Corrected
config.example.yaml E3 dfe_source derivations. "The one setting a working config cannot go without" topics can also be set directly
config.example.yaml E5 librdkafka comment em dashes Corrected
config.example.yaml E6 Sink block, two semicolons Corrected
config.example.yaml E8 "Or specify individual files" Corrected
config.example.yaml E9 Vector block: version: "0.61.0", a semicolon Corrected
config.example.yaml E11 Logging precedence, semicolon, auto (text on a tty) Corrected
config.example.yaml E12 Scaling: what reads the block, "KEDA ... gated by the Vector subprocess circuit" Corrected

Updates the md files and config.example.yaml comments against the code and development binary. Sources also cover scalo-rs at the Cargo.lock version, Vector 0.61.0, dfe-infra and hyperi-ci.

- run --config exits 2, so --config goes before the subcommand. Outside the image, the quick start also needs vector.binary, vector.config_dir and vector.data_dir.
- A comma list in DFE_TRANSFORM_VECTOR_SOURCE__BROKERS fails to load. The flat names take comma lists and override the double-underscore form. DFE_TRANSFORM_VECTOR_CONFIG has no effect.
- A bus config with no topics idles: /readyz answers 200 with no Vector running until the config file names a topic.
- Changes to the app's own keys outside transforms need a restart. Until that restart, later transform changes do not reload.
- Edits to scalo's sections also need a restart. They log a successful reload.
- The chart rolls the pods on any config or file set change.
- The migration docs give the new pipeline its own output topic. They state where its consumer group starts and how downstream consumers move across.
- dfe-infra adds a pressure trigger on the single, scale and mesh profiles.
- The contract has dlq among its writable paths.
- Shutdown exits 0 even after a SIGKILL.
- librdkafka.md lists the keys Vector writes from its own fields. It documents the enable.idempotence override and librdkafka's default for socket.nagle.disable. It drops the profiles this app never reads. Only this app reads the central file.
- Cuts the CI step, the Dockerfile pattern, the parity table and the scaling claims that matched no file.
- CONTRIBUTING links DCO 1.1. A release needs the Release: true trailer. revert bumps a patch.
- default_in is docker-single.
- The Vector binary check's key is vector.version_check (src/config/loader.rs:649-650). The bare version_check is scalo's own section.
- A restart-only edit blocks transform reloads until the pod restarts or the edit is reverted. A revert lets the next transform change reload.
- The central librdkafka file is read only when DFE_CONFIG_DIR is set. An empty profile is skipped, leaving scalo's constants in force.
- architecture.md's MPL paragraph starts in plain language.
- tests/TESTING.md's filebeat section matches main.
Brings in #165, whose filebeat tests replay dfe-transform-vrl's synthetic cases. tests/TESTING.md keeps main's filebeat paragraph and this branch's edits to the rest of the page.
- CONTRIBUTING says e2e::filebeat_kafka needs dfe-transform-vrl for its cases and the bundled pipeline. It also says the repo holds tests/fixtures/filebeat/cases.yaml.
- pipelines/filebeat/README.md points at vrl's known limitations. It also says both transform apps grade against the same cases.
@catinspace-au
catinspace-au merged commit b13ebca into main Nov 15, 2026
23 checks passed
@catinspace-au
catinspace-au deleted the fix/docs-reality branch November 15, 2026 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant