Repository navigation
docs(transform-vector): correct docs against the code - #164
Merged
Merged
Conversation
Updates the md files and config.example.yaml comments against the code and development binary. Sources also cover scalo-rs at the Cargo.lock version, Vector 0.61.0, dfe-infra and hyperi-ci. - run --config exits 2, so --config goes before the subcommand. Outside the image, the quick start also needs vector.binary, vector.config_dir and vector.data_dir. - A comma list in DFE_TRANSFORM_VECTOR_SOURCE__BROKERS fails to load. The flat names take comma lists and override the double-underscore form. DFE_TRANSFORM_VECTOR_CONFIG has no effect. - A bus config with no topics idles: /readyz answers 200 with no Vector running until the config file names a topic. - Changes to the app's own keys outside transforms need a restart. Until that restart, later transform changes do not reload. - Edits to scalo's sections also need a restart. They log a successful reload. - The chart rolls the pods on any config or file set change. - The migration docs give the new pipeline its own output topic. They state where its consumer group starts and how downstream consumers move across. - dfe-infra adds a pressure trigger on the single, scale and mesh profiles. - The contract has dlq among its writable paths. - Shutdown exits 0 even after a SIGKILL. - librdkafka.md lists the keys Vector writes from its own fields. It documents the enable.idempotence override and librdkafka's default for socket.nagle.disable. It drops the profiles this app never reads. Only this app reads the central file. - Cuts the CI step, the Dockerfile pattern, the parity table and the scaling claims that matched no file. - CONTRIBUTING links DCO 1.1. A release needs the Release: true trailer. revert bumps a patch. - default_in is docker-single.
- The Vector binary check's key is vector.version_check (src/config/loader.rs:649-650). The bare version_check is scalo's own section. - A restart-only edit blocks transform reloads until the pod restarts or the edit is reverted. A revert lets the next transform change reload. - The central librdkafka file is read only when DFE_CONFIG_DIR is set. An empty profile is skipped, leaving scalo's constants in force. - architecture.md's MPL paragraph starts in plain language. - tests/TESTING.md's filebeat section matches main.
Brings in #165, whose filebeat tests replay dfe-transform-vrl's synthetic cases. tests/TESTING.md keeps main's filebeat paragraph and this branch's edits to the rest of the page.
- CONTRIBUTING says e2e::filebeat_kafka needs dfe-transform-vrl for its cases and the bundled pipeline. It also says the repo holds tests/fixtures/filebeat/cases.yaml. - pipelines/filebeat/README.md points at vrl's known limitations. It also says both transform apps grade against the same cases.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
README.md, CONTRIBUTING.md and SECURITY.md now match the code. So do every page under docs/, pipelines/filebeat/README.md, tests/TESTING.md and config.example.yaml's comments.
References: the development binary, scalo-rs v2.14.30, Vector 0.61.0, librdkafka 2.12.1, dfe-infra and hyperi-ci.
The ledger has 246 rows: PASS 118, FAIL 114, UNCHECKABLE 14. All 114 FAIL rows are fixed or cut, 21 of them prose only.
--configgoes before the subcommand:run --configexits 2. Outside the image, the quick start also needsvector.binary,vector.config_dirandvector.data_dir.DFE_TRANSFORM_VECTOR_SOURCE__BROKERSfails to load. Flat names take precedence.DFE_TRANSFORM_VECTOR_CONFIGhas no effect.transformsblocks later transform reloads until a restart. A scalo-section edit logs a successful reload. The chart rolls the pods on a config or file set change.dlqamong its writable paths.enable.idempotenceoverride. It corrects thesocket.nagle.disabledefault. Unused profiles are cut. Only this app reads the central file.Release: trueorfrom-head.revertbumps a patch.114 corrected rows
run --config config.example.yaml--configfirst, fetch-vector.sh, the binary and the two directoriesmetrics.vector_metrics_address"stays on loopback"SOURCE__BROKERS_deployment.tpl:48-53)revert"Varies"to ensure they workRelease: trueorfrom-head, archiver wordingdefault_in"--test e2e # needs Docker or a live broker#[ignore]cases (seven ... one) ... six ... one"--run-ignored allemit_dockerfile()"appends ... with the version substituted in"transforms:is hand-writtenrejecteddropped with the DLQ off/readyz"200 OK only while the Vector child process is up"/livezand/metricsunchanged (scalo metrics/mod.rs:1085-1093)config-checkpasses.assemblegives the sinkSASL_SSLandSECRET[dfe_sink_sasl.*]. Semicolon removed.transforms.diruses a file set's mountPath00_global.yaml"data_dir, api switch, SASL secret backends",50_000_parse,50_001_enrich,50_002_filterassembledfe_sink.inputs"config_reloads_total{result=...}, log "config change requires pod restart"status.success() => break, "Max crashes: unlimited by default"warn: "emit metric"USER appusercargo run -- emit-dockerfile, "only the vector and runtime stages rebuild", "CI runs integration tests against the new Vector"acknowledgements: true, disk buffers "survive pod restarts via PVC", "does not touch the data path"config.yamlfor dfe-loaderdfe-transform-vector-sysloglinger.msalias notelinger.ms=50giveslinger.ms=50with the instance's20still setauto.offset.resetlatest,session.timeout.ms45000, heartbeat "(session.timeout / 15)"log.connection.closedefault falsesocket.nagle.disabledefault falsedelivery.timeout.ms300000,batch.size"(1 MiB)",queue.buffering.max.kbytesandmessage.max.bytesenable.idempotence, "Lag is what KEDA scales on here"custom.vendor.setting: "value"scalo/src/kafka_config.rscurrentlyx2timberio/vectorHelm chart"0.48.0-debianreplicaCount: 3under KEDA, no brokerskeda.minReplicaCount, brokers,fileSets,secrets.kafka.existingSecret(chart values schema)/etc/dfe-transform-vector/transforms"v0.42.1, "no init container ... bake ... or mount", Hot-reload "Not supported"transforms.dirat this directory and the pipeline runs"filebeat.yaml"byte-identical" tofilebeat.vrl@e0b7087standards/universal/testing.mdin hyperi-aiversion: "0.61.0", a semicolonauto (text on a tty)