Skip to content

docs(receiver): correct docs against the code - #268

Merged
catinspace-au merged 2 commits into
mainfrom
fix/docs-reality
Nov 15, 2026
Merged

catinspace-au merged 2 commits into
mainfrom
fix/docs-reality

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

README.md, CONTRIBUTING.md, every page under docs/ and the two fixture READMEs match the code and config schema. They also match scalo and the repos they name. docs/delivery.md takes architecture.md's delivery and buffering sections.

The ledger has 351 rows: PASS 215, FAIL 122, UNCHECKABLE 14. Some rows group similar claims. All 122 FAIL rows are fixed. Claims the repo cannot show, such as a peer's retry rule or a supplier's pricing, stay UNCHECKABLE.

  • When the image starts with --config, only the flat DFE_RECEIVER_* list overrides the file. DFE_RECEIVER_SERVER__BIND_ADDRESS takes effect only without --config.
  • The build needs librdkafka at least the version after the + in rdkafka-sys, and pkg-config. Debian trixie's package is older.
  • auth.mode: mtls on the gRPC and OTLP gRPC ports refuses every RPC. Certificate-only auth there is mode: none with tls.client_auth: required.
  • TLS settings load once at start. A reload into mtls or both needs a restart.
  • docs/metrics.md and docs/log-spamming.md replace historical plans with the receiver's metrics and log limits.
122 corrected rows
Page Row Was Now
README.md R13 HTTP(S) JSON port "configurable" 8080
README.md R14 gRPC Vector sink port "configurable" 6000
README.md R26 "HTTP always runs, because the probes ride its listener" both ports serve the probe paths. The chart uses the metrics port
README.md R29 Build needs protoc, cmake, a C toolchain. rdkafka-sys builds native code build.rs needs protoc. aws-lc-sys 0.47 picks its cc builder (builder/main.rs:786-791). Cargo.toml:66 sets rdkafka dynamic-linking on every target (rdkafka-sys 4.10.0 build.rs:43-72)
README.md R30 apt and brew lines apt and brew lines list pkg-config and librdkafka
README.md R32 "The smallest config that moves data names the bind address, an auth mode, the brokers and the default source" the brokers alone. The rest are written out as defaults
README.md R34 "Every setting is in config.example.yaml" docs/config-schema.yaml lists every setting
README.md R35 DFE_RECEIVER_SERVER__BIND_ADDRESS is how an env override is written documents both env forms and which config load reads each
README.md R36 server.auth.mode is none, header, bearer or mtls server.auth.mode also accepts both
CONTRIBUTING.md C3 revert "Varies" Patch
CONTRIBUTING.md C10 A merge to main releases automatically, no manual step releases need the Release: true trailer
CONTRIBUTING.md C11 "The suite's one external door", "eleven wire protocols" describes the ingest edge without a protocol count
CONTRIBUTING.md C12 "It and dfe-ui are the only components reading untrusted input" threat-model.md:21-25, :73: dfe-receiver, dfe-ui, dfe-hyperdx and, where deployed, culvert
CONTRIBUTING.md C14 "NOT a transform stage (that is dfe-loader)" dfe-loader README: loads Kafka and gRPC into ClickHouse. The transforms are dfe-transform-vrl, -vector, -elastic (suite.yaml)
CONTRIBUTING.md C15 src/main.rs: the --emit-* generators one flag, --emit-dockerfile (main.rs:57-60). Artefacts come from generate-artefacts
CONTRIBUTING.md C16 src/server/ registered at mod.rs:125 function reference replaces mod.rs:125
CONTRIBUTING.md C23 tests/ targets smoke, integration, e2e. common/mod.rs is the harness tests/ also holds orchestrator_lifecycle.rs
CONTRIBUTING.md C26 Without protoc, cmake, C toolchain the build dies naming nothing useful documents librdkafka and pkg-config (R29). rdkafka-sys names librdkafka when it fails (build.rs:66-70)
CONTRIBUTING.md C36 dfe-engine hand-copies Config::validate(), nothing compares, drifted narrows the claim about copied validation
CONTRIBUTING.md C43 dfe-engine mirrored-logic "reimplements Config::validate()" narrows the copied-validation claim (C36). The suite.yaml edge still cites the deleted receiver.py:29
docs/architecture.md A1 "The configuration reference is config.example.yaml" the schema is the reference
docs/architecture.md A2 The pipeline behind "wants one thing: JSON on a Kafka topic" a Kafka topic or the loader's gRPC listener
docs/architecture.md A6 ingest-edge.md "calls the receiver a single-homed airlock" replaces the description of ingest-edge.md
docs/architecture.md A8 Parsing, enrichment and schema work are dfe-loader's the receiver handles mechanical framing. Transforms handle content parsing and enrichment. The loader fits data to tables
docs/architecture.md A10 Diagram: "10 protocol handlers", text "Eleven handlers", "eleven times" removes handler counts
docs/architecture.md A19 HTTP always runs "because /livez and /readyz ride its listener" the chart probes the metrics port (R26)
docs/architecture.md A22 "which dfe-fetcher sends to on the direct transport" it takes Vector's gRPC sink
docs/architecture.md A25 "PB/s-a-day throughput" petabytes a day is the design target
docs/architecture.md A29 "No format!() on the path" the default topic is computed once
docs/architecture.md A35 Records answered at enqueue get their own producer, and librdkafka give-ups "with a message timeout" are queued again revises the producer and resend description
docs/architecture.md A46 Refused for good: "invalid record or timestamp, topic or cluster authorisation failed, invalid topic" lists four codes. An ACL refusal is kept
docs/architecture.md A52 "Secrets are never literals in the config file" auth.bearer.tokens, accepted_headers[].values and kafka.sasl.password take literal values (schema, x-scalo-secret). Spec fields refuse a literal (secrets.rs:30-52)
docs/architecture.md A58 Invariant 4: auth.mode none by default, "limited only by body size, request timeout, the IP filter and the rate limiter" also max_concurrent_requests (http/mod.rs:725-730) and the connection caps (http/mod.rs:505-518)
docs/internals.md I2 "PB/s scale", "PB/s daily throughput" removes throughput claims. architecture.md states the target
docs/internals.md I11 ASCII architecture: HTTP feeds gRPC in series, only two listeners, TieredSink under Kafka only architecture.md's diagram replaces the ASCII diagram
docs/internals.md I16 Hot-path code block and lists, incl. "No format!() on hot path" architecture.md owns the hot-path description
docs/internals.md I19 gRPC uses the "same auth middleware (via Tower)" the interceptor. Certificate-only auth is mode: none with tls.client_auth: required
docs/internals.md I24 SCRAM-SHA-512 works on Apache Kafka, AutoMQ, MSK, Confluent Cloud, Redpanda Confluent Cloud is PLAIN with an API key
docs/internals.md I26 "TieredSink" wraps the sink with breaker, in-memory queue, drain that is InMemoryBuffer (buffer/tiered.rs:96-134). scalo's TieredSink is the disk spool (buffer/mod.rs:32-38)
docs/internals.md I28 "INTERNAL, the old answer to everything" removes the claim
docs/internals.md I29 receiver_records_dropped_total reason "one of unavailable, rejected or shutdown" DropReason also has oversize and kernel (metrics/mod.rs:158-180)
docs/internals.md I34 A reload replaces a listener's whole auth section, reaching server, splunk_hec, prometheus_rw, otlp both ports, grpc TLS settings and moves to mtls or both need a restart
docs/internals.md I37 Example rule matches _source equals crates_audit and sets crates_audit the rule matches data_stream.dataset
docs/internals.md I39 scram_sha_512 "works for ... Confluent Cloud" Confluent Cloud is PLAIN with an API key (I24). The spelling is accepted (config/mod.rs:2421-2422)
docs/internals.md I41 Loader: a payload field named raw lands in _raw with no mapping dfe-loader origin/main pipeline.rs:1042-1054 renames only raw_source_fields (default none). dfe-schemas timeseries.yaml:62-66 gives _raw no @source
docs/internals.md I46 Webhook "its own protocol handler" a WebhookHandler only with webhook.bind_address. Otherwise, routes merge on the HTTP listener (server/mod.rs:218-226, http/mod.rs:378-393)
docs/internals.md I49 runZero "the first caller and the reason" removes the historical explanation
docs/internals.md I51 process_batch_to_topic "skips the router's enrichment, so the handler stamps _source and _timestamp_received itself" process_batch_to_topic stamps both fields (pipeline/mod.rs:1016-1021, :1301 stamp_for_topic, webhook/mod.rs:13-17)
docs/internals.md I58 receiver_kafka_resends_total counts message timeouts also topic not served and authorisation failure (kafka/mod.rs:233-246)
docs/internals.md I60 "MUST NOT be directly exposed to the public internet", always behind edge protection hardening.md carries guidance on protection in front of the receiver
docs/internals.md I65 IP filter on "every accept loop the receiver owns" limits the IP-filter claim to two accept loops
docs/internals.md I71 config.example.yaml has "the full flow: block" docs/config-schema.yaml
docs/internals.md I74 "AsyncFd::readable + recv_from", "Functionally equivalent below ~100K pps" try_recv_from (listener.rs:156). Removes the throughput comparison
docs/hardening.md H1 "March 2026" document date removes the date
docs/hardening.md H3 "Current State" heading removes the relative-time heading
docs/hardening.md H12 TLS "per-protocol, hot-reloadable" only ingest, webhook, HEC, remote write and OTLP HTTP build TlsCertProvider, and only for *_secret material (tls.rs:317-319, http/mod.rs:315-327). Files and the other listeners load once
docs/hardening.md H16 Order "Concurrency -> Timeout -> BodyLimit -> Auth -> Handler" the rate limiter sits outside the concurrency cap (http/mod.rs:725-776)
docs/hardening.md H21 "(Complete)", "All planned application-level hardening is implemented. No remaining items." removes the completion claims
docs/hardening.md H22 Per-listener control table. gRPC and OTLP gRPC take "bearer or mTLS" bearer, or certificates with mode: none and tls.client_auth: required
docs/hardening.md H25 "(March 2026)" status headings removes read dates and keeps the dated EOL event
docs/hardening.md H28 Istio ambient "GA since Istio 1.22" GA in 1.24, beta in 1.22 (istio.io/latest/blog/2024/ambient-reaches-ga/)
docs/hardening.md H29 request_headers_timeout "not yet exposed" in ClientTrafficPolicy, #2598 documents request_headers_timeout. request_headers_timeout, requestReceivedTimeout and EnvoyPatchPolicy are Envoy names
docs/hardening.md H32 ## Part 3 692 words splits the section
docs/hardening.md H33 Billing models "as of March 2026" removes the dated billing models
docs/hardening.md H35 #### variant headings a table replaces the variant headings
docs/hardening.md H36 deployment.rs:80-147 adds webhook 8090 and flow UDP
docs/hardening.md H38 aws-load-balancer-cross-zone-load-balancing-enabled deprecated for aws-load-balancer-attributes: load_balancing.cross_zone.enabled=false (kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/annotations/)
docs/hardening.md H39 "Each layer catches what the previous layer missed." removes the claim
docs/hardening.md H40 "tower-governor 0.8" removes the version from link text
docs/hardening.md H41 ALB vs NLB cost analysis link removes the link
docs/metrics.md M1 Historical planning doc, dfe_ not adopted documents the receiver's metrics
docs/metrics.md M2 "Current State (Problems)" removes the section
docs/metrics.md M4 Labels: protocol on receiver ingest metrics and on records_received_total the receiver labels transport (main.rs:388-442). records_received_total has no labels (manifest)
docs/metrics.md M8 Fleet implementation standard and buckets scalo-rs docs own the fleet standard and buckets
docs/metrics.md M9 KEDA ScaledObject on a prometheus trigger over scaling_pressure{job=...} dfe-infra values.yaml:133-141 uses a metrics-api trigger on dfe-keda-shim
docs/metrics.md M10 Migration phases and per-service map removes both
docs/metrics.md M11 dfe_pipeline_ready removes the metric with M2
docs/performance.md P1 jemalloc + fat LTO "when the project opts in", PGO + BOLT "additionally opt-in" hyperi-ci main rust/optimize.py:28-33: jemalloc every channel, fat LTO from beta, no opt-in. :196-211 PGO on by default on release when scripts/pgo-workload.sh exists, BOLT with it
docs/performance.md P2 hyperi-ci RUST-RELEASE-TRACK-OPTIMISATION.md, PGO-WORKLOAD-GUIDE.md hyperi-ci main docs/ has neither. docs/runtime/pgo-bolt.md exists
docs/performance.md P4 A mimalloc feature and #[cfg] wiring removes both
docs/performance.md P5 Fat LTO, PGO, BOLT as opt-in tiers fat LTO from beta. PGO on by default on release when scripts/pgo-workload.sh exists, BOLT with it (P1)
docs/performance.md P6 "Planned": pre-computed topics, pre-split field paths removes the planned items
docs/performance.md P8 PGO needs build.rust.optimize.pgo.enabled: true. BOLT needs bolt.enabled: true defaults on (optimize.py:196-211). .hyperi-ci.yaml sets only workload_setup_cmd
docs/performance.md P9 Local PGO recipe the recipe builds the driver first
docs/performance.md P11 Binary sizes "v1.14.10", allocator strings removes sizes and allocator strings. Keeps the strings check
docs/performance.md P12 Criterion allocator table removes the table and keeps the decision
docs/performance.md P13 "eight lines: four RFC 3164 ... two RFC 5424 and one CEF" five lines. Removes the host and measured figures
docs/performance.md P14 Audit against dfe-loader, "pending hyperi-ci" removes the audit note. flush_* and max_concurrent_inserts are dfe-loader keys
docs/performance.md P16 "KafkaProducer::HighThroughput ... 256 KiB, 100 ms, LZ4" scalo v2.14.30 kafka/config.rs:259-275: 128 KiB, 20 ms, zstd level 3, 64 MiB. The receiver builds its own producer (kafka/mod.rs:9-18)
docs/performance.md P17 MEMORY_PRESSURE_THRESHOLD 0.80, MEMORY_CGROUP_HEADROOM 0.85, "documented in .env comments" scalo memory/guard.rs:237-240: {PREFIX}_..., so DFE_RECEIVER_MEMORY_.... No .env in the repo
docs/performance.md P19 jeprof with MALLOC_CONF="prof:true" on a release build tikv-jemalloc-sys 0.7.1 needs its profiling feature and reads _RJEM_MALLOC_CONF (README.md:38, :81-82, :128-131). The jemalloc feature enables neither. The recipe's build has no jemalloc
docs/performance.md P20 "Add a dhat-heap feature" removes the instruction
docs/performance.md P21 "Push to a release-channel project and CI applies the full pipeline automatically" a release needs the Release: true trailer (C10)
docs/performance.md P22 "template this doc is modelled on" removes the template reference
docs/performance.md P23 Emoji status column removes the emoji status column
docs/sidecar-transports.md S1 "natively supports these ingest protocols" table links to the README's protocol table
docs/sidecar-transports.md S4 schema the listener refuses to start without client certs or fluent.accept_unauthenticated (config/mod.rs:624-631)
docs/sidecar-transports.md S6 Go collector treats anything but http.StatusOK as an error StatusAccepted
docs/sidecar-transports.md S7 Helm value extraContainers scalo-service v2.14.30 values.schema.json has sidecars, appended to the pod (_deployment.tpl:37), and no extraContainers
docs/sidecar-transports.md S9 "use an initContainer" the same page, line 383, explains why an init container cannot wait for the receiver
docs/sidecar-transports.md S10 Token "in receiver's auth.bearer_tokens list" the key is <listener>.auth.bearer.tokens (schema)
docs/sidecar-transports.md S11 "receiver's auth.mtls" no such key. Uses auth.mode: mtls with tls.client_auth: required (config/mod.rs:807-836)
docs/sidecar-transports.md S12 server.max_body_size limits Vector batches. Set "64MiB" the vector sink is gRPC, bounded by grpc.max_message_size (16 MiB). max_body_size is an integer. The probe refuses "64MiB"
docs/sidecar-transports.md S14 Overloaded receiver "returns 503", Vector buffers to "its disk buffer" the gRPC listener answers UNAVAILABLE (grpc/mod.rs:105, :196). Vector's buffer is memory unless configured
docs/sidecar-transports.md S17 Vector readiness on /health port 8686 Vector's API, which needs api.enabled: true
docs/sidecar-transports.md S18 ghcr.io/hyperi-io/dfe-receiver:latest <version>
docs/sidecar-transports.md S19 em-dashes replaces em-dashes with ASCII punctuation
docs/log-spamming.md L1 Historical banner. "No project implements any form of log rate limiting" documents the receiver's log limits
docs/log-spamming.md L2 Receiver hot spots at pipeline/mod.rs:205, sink/kafka/mod.rs:61, syslog/mod.rs:73, lumberjack/mod.rs:94,135 removes the hot-spot list
docs/log-spamming.md L3 Other repos' file:line refs removes the refs
docs/log-spamming.md L5 dfe_transport_send_errors_total removes the metric
docs/log-spamming.md L6 LOG_THROTTLE_BURST 10, LOG_THROTTLE_RATE 1, LOG_THROTTLE_MAX_SIGNATURES 5000, LOG_THROTTLE_ENABLED true scalo logger/mod.rs:439-466: burst 50, rate 1.0, enabled false, no signatures variable
docs/log-spamming.md L7 Phase 3 helpers to add scalo logger/helpers.rs:33-75 ships all three
docs/webhook-setup.md W1 "end to end" removes the phrase
docs/webhook-setup.md W6 "the shipped precedent" removes the historical explanation
docs/webhook-setup.md W12 401 reasons webhook/auth.rs:71-78 also has missing_timestamp, invalid_timestamp
docs/webhook-setup.md W13 tower_governor 0.8.0 lib.rs:143-144 and error.rs:137-149 adds the 500 filter_error row
docs/testing.md T1 hyperi-ai's standards/universal/testing.md removes the standards reference
docs/testing.md T2 core runs cargo nextest run --all-features CI tests features: default (.hyperi-ci.yaml:24-25), which adds no flag (hyperi-ci main rust/test.py:160-166)
docs/README.md D1 The index lists the pages adds metrics.md and delivery.md. Updates titles for the rewritten pages
tests/fixtures/pgo/README.md F4 "shape" twice replaces both uses
tests/fixtures/flow/external/README.md X5 Semicolon removes the prose semicolon

README.md, CONTRIBUTING.md, docs/ and the fixture READMEs now match the code, the config schema, scalo and the repos they name.

- With `--config`, only the flat `DFE_RECEIVER_*` list overrides the file at image startup. `DFE_RECEIVER_SERVER__BIND_ADDRESS` works only without `--config`.
- The build needs librdkafka and pkg-config, and no cmake.
- docs/metrics.md and docs/log-spamming.md were historical planning docs. They now list what the receiver emits and how it bounds its logs.
- docs/performance.md loses the mimalloc feature Cargo.toml no longer has, the benchmark tables and the wrong producer defaults.
- architecture.md's delivery and buffering sections move to docs/delivery.md.
- Fix wrong keys, statuses and codes in sidecar-transports.md, webhook-setup.md, hardening.md and internals.md. The two mermaid diagrams render.
- sidecar-transports.md: the `vector` sink uses gRPC. `auth.mode: mtls` rejects every RPC. Client certificate authentication needs `grpc.auth.mode: none`, `grpc.tls.client_auth: required` and the CA.
- architecture.md: edge protection or a load balancer allow list limits access to the receiver's ports from the internet. The webhook has its own listener only when `webhook.bind_address` is set.
- hardening.md: the IP filter runs in every accept loop except those for the two gRPC ports. Cloudflare proxies HTTPS only on its listed ports. The HTTP rows need an Origin Rule to the receiver's port. gRPC needs an endpoint with TLS and HTTP/2 on 443.
- internals.md: certificate material from a secret reference refreshes on the five HTTP listeners. The rest of `tls.*` is read once at start.
@catinspace-au
catinspace-au merged commit 8f671eb into main Nov 15, 2026
13 checks passed
@catinspace-au
catinspace-au deleted the fix/docs-reality branch November 15, 2026 08:49
@github-actions

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant