Skip to content

docs(transform-vrl): correct docs against the code - #151

Merged
catinspace-au merged 1 commit into
mainfrom
fix/docs-reality
Nov 15, 2026
Merged

catinspace-au merged 1 commit into
mainfrom
fix/docs-reality

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

README.md, CONTRIBUTING.md, SECURITY.md and every page under docs/ now match the code. So do the two filebeat READMEs, tests/TESTING.md and config.example.yaml's comments. They also match scalo-rs v2.14.30, vrl 0.35.0, hyperi-ci, dfe-infra, dfe-docker and the development build's binary.

The ledger has 235 rows: PASS 132, FAIL 93, UNCHECKABLE 10. Some rows group similar claims. All 93 FAIL rows are fixed or cut. Of those, 13 fix prose and 9 remove stale numbers or versions.

UNCHECKABLE rows are design rationale, Vector's own behaviour, or claims nothing in the repo measures.

  • dfe-transform-vrl config-check --config config.yaml exits 2. --config goes before the subcommand. cargo run needs --bin to distinguish it from pgo-driver.
  • --run-ignored with no value exits 2. The full tier is --run-ignored all --ignore-default-filter.
  • The idle gate wakes on a change to the config file or a SIGHUP. Adding a program to transforms.dir alone does not wake the gate.
  • The env table lists all 21 flat names. The double-underscore form reaches every key. DFE_TRANSFORM_VRL_CONFIG reaches neither cascade.
  • The contract's one writable path is the DLQ directory. Outside the image, the DLQ needs a writable dlq.file.path.
  • Claims about format auto-sensing, FormatDetector, the /config route, dfe-docker's infra profile, a skip_if_no_kafka! caller and a splack chart are cut. None exists.
  • internals.md's transform example failed to compile with E651. Its Rust snippets matched neither compile_vrl nor run_vrl.
  • CONTRIBUTING links DCO 1.1. A release needs the Release: true trailer or a from-head dispatch. revert bumps a patch.
  • No code or code comment changes. The change is docs: and carries no release trailer.
93 corrected rows
Page Row Was Now
README.md R11 A transform cannot reach outside the process cannot read the host's environment or reach the network
README.md R14 cargo build --release with no prerequisites protoc, a C toolchain, pkg-config and librdkafka at the rdkafka-sys version
README.md R15 Copy the example and set brokers, topics and dir uncomment source, sink and transforms first
README.md R16 config-check --config config.yaml --config config.yaml config-check
README.md R22 (off-image DLQ) dlq.file.path must be writable
README.md R23 config-schema.yaml lists every key every key of the transform's own sections
CONTRIBUTING.md C2 revert "Varies" Patch
CONTRIBUTING.md C4 A paraphrased DCO a link to DCO 1.1
CONTRIBUTING.md C7 Bold-lead contribution steps plain steps
CONTRIBUTING.md C9 (release implied by a merge) the Release: true trailer or a from-head dispatch
CONTRIBUTING.md C12 212,218 bytes of VRL one VRL file plus a lookup table
CONTRIBUTING.md C13 integration/ mostly without Kafka without a broker
CONTRIBUTING.md C16 --run-ignored --run-ignored all --ignore-default-filter
CONTRIBUTING.md C17 cargo run -- config-check --config ... cargo run --bin dfe-transform-vrl -- --config ... config-check
CONTRIBUTING.md C20 Kafka tests call skip_if_no_kafka!() core-tier tests run on containers, skip locally, fail under CI
CONTRIBUTING.md C21 Ignored tests need --run-ignored and a broker --run-ignored all or only, live broker first
CONTRIBUTING.md C22 dfe-docker infra profile localhost:19092, the port dfe-docker publishes
CONTRIBUTING.md C25 A Dockerfile hand edit is reverted checked_in_dockerfile_matches_emit_dockerfile fails on it
CONTRIBUTING.md C27 Scale on CPU plus scaling pressure the contract scales on CPU, a deployment adds scaling pressure
CONTRIBUTING.md C28 scalo section list adds memory
CONTRIBUTING.md C31 "that pod ... Refused at startup now" present tense
CONTRIBUTING.md C32 Redacted in the /config dump the config-check dump
CONTRIBUTING.md C33 OOM-killed under 32 MiB points at engine::budget
CONTRIBUTING.md C36 Contract schema version 4 the header names it
CONTRIBUTING.md C37 The header names the regeneration command the emit-dockerfile command
CONTRIBUTING.md C41 splack runs this image with its own chart cut
CONTRIBUTING.md C42 Bold-lead bullets plain
SECURITY.md S2 "currently", "The following", bold leads plain
docs/README.md D2 "Design Document" Internals
architecture.md A7 Quoted edge labels fail maid unquoted labels pass maid
architecture.md A9 Format auto-sensed, back to the arrival format JSON in, JSON out
architecture.md A17 212,218 bytes of VRL cut
architecture.md A18 96 bytes per byte over 24 MiB the constants' doc comments
architecture.md A20 No topics or no program no topics on the bus, or no program
architecture.md A21 The program that lands later starts the service the gate wakes on the config file or SIGHUP, a new program rolls the pod
architecture.md A22 "Only reaching the loader ... is a failure" cut
architecture.md A25 Config sections adds dlq
architecture.md A26 scalo sections adds memory
architecture.md A27 The wrapper seeds scalo's cascade scalo's run_app does
architecture.md A30 (DFE_TRANSFORM_VRL_CONFIG) reaches neither cascade
architecture.md A32 main/cli "env seeding" the ServiceApp wiring
architecture.md A33 FormatDetector the Kafka configs and DFE names
architecture.md A35 Header names a command, a hand edit is reverted emit-dockerfile and the smoke test
architecture.md A37 Contract schema version 4 emit-contract prints it
architecture.md A39 No writable path beyond /tmp the DLQ directory
architecture.md A40 100m/128Mi, 500m/512Mi, 45 s emit-contract prints them
architecture.md A47 Format detection behaviour the parse and serialise path
architecture.md A50 12 headings 10
internals.md I1 Kafka-to-Kafka both transports
internals.md I2 Owns the Kafka consumer/producer scalo's KafkaTransport from the wrapper's config
internals.md I4 2-4x memory waste cut
internals.md I5 Bus-only diagram both transports
internals.md I7 Per-event KafkaTransport::send(), box characters send_batch per block, a list
internals.md I8 VRL program(s) one program
internals.md I15 vrl::compiler::compile(source, &stdlib::all()) compile_vrl
internals.md I17 runtime.resolve(&mut target, &program) run_vrl with UTC and panic handling
internals.md I19 .labels.env ?? "unknown" (E651) string(.labels.env) ?? "unknown"
internals.md I24 dfe-transform-elastic in beta dropped
internals.md I25 Controls all memory allocation bounds its buffers
internals.md I26 64 MiB defaults config-schema.yaml
internals.md I27 max_chunk_size env only --config or env
internals.md I30 A full producer queue pauses the consumer backpressure, the block retried, prefetch capped
internals.md I31 The wrapper's whole schema six sections
internals.md I32 scalo sections adds memory
internals.md I33 rdkafka, Full memory control scalo's KafkaTransport, the bounds named
internals.md I38 Warns for each one set for each one off its default
internals.md I39 max_chunk_size via the env layer a scalo section
internals.md I40 Restart table adds dlq.*, sink transport and endpoint, enrichment tables
internals.md I41 17 names, the whole list 21 names, the double-underscore form, scalo's memory names
internals.md I55 10 sections 7
filebeat README F9 Every rule needs a leading timestamp column every rule but audit
filebeat README F14 Semicolons, bold leads, "shape" plain
TESTING.md T2 Remote tests skip when unreachable they fall back to a container
TESTING.md T3 dfe-docker infra profile localhost:19092
TESTING.md T6 skip_if_no_kafka!() cut, no caller
TESTING.md T8 Full tier list adds connectivity
TESTING.md T12 alpine/openssl:3.5.9 the image scalo pins
TESTING.md T14 Every broker test fails under CI every core-tier one
TESTING.md T15 hyperi-ai's testing standard cut, a private repo
TESTING.md T16 skip_if_no_kafka!, ensure_docker_infra cut, adds ensure_kafka_or_skip!
TESTING.md T17 Em dashes --
fixtures README X4 Same pin as dfe-transform-elastic pinned in fetch_testdata.py
fixtures README X6 ACCEPTED_MISMATCHES in filebeat_pipeline.rs tests/common/filebeat.rs
fixtures README X7 Keep aligned with dfe-transform-elastic cut
fixtures README X9 Semicolons plain
config.example.yaml E1 Em dash --
config.example.yaml E2 .env always a layer only without --config
config.example.yaml E3 Setting either warns changing either from its default
config.example.yaml E5 Semicolon plain
config.example.yaml E7 dfe-transform-elastic (beta) dropped
config.example.yaml E8 Semicolon plain
config.example.yaml E9 scalo section list adds memory without a warning
config.example.yaml E12 The README's not-applied list internals.md, Reload

Corrects md files and config.example.yaml to match the code, scalo-rs at the Cargo.lock version, dfe-infra and hyperi-ci.

- config-check exits 2 with --config after the subcommand. --config goes before config-check. cargo run needs --bin because there is also a pgo-driver binary.
- --run-ignored needs a value, so the full tier is --run-ignored all --ignore-default-filter.
- The idle gate wakes on a config-file change or SIGHUP, never on a program written into transforms.dir.
- The env table lists all 21 flat names. The double-underscore form reaches every key. DFE_TRANSFORM_VRL_CONFIG reaches neither cascade.
- The contract's one writable path is the DLQ directory. Outside the image, the DLQ needs a writable dlq.file.path.
- Remove references to Format auto-sensing, FormatDetector, a /config route, a dfe-docker infra profile, skip_if_no_kafka callers and the splack chart. None exist.
- The internals.md transform example failed to compile with E651. Its Rust snippets did not match compile_vrl or run_vrl.
- CONTRIBUTING links the DCO 1.1 text. A release needs the Release: true trailer. revert bumps a patch.
- Counts, sizes and version literals point at the file that owns them.
@catinspace-au
catinspace-au merged commit 33f8edd into main Nov 15, 2026
24 checks passed
@catinspace-au
catinspace-au deleted the fix/docs-reality branch November 15, 2026 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant