Repository navigation
docs(receiver): correct docs against the code - #268
Merged
Merged
Conversation
README.md, CONTRIBUTING.md, docs/ and the fixture READMEs now match the code, the config schema, scalo and the repos they name. - With `--config`, only the flat `DFE_RECEIVER_*` list overrides the file at image startup. `DFE_RECEIVER_SERVER__BIND_ADDRESS` works only without `--config`. - The build needs librdkafka and pkg-config, and no cmake. - docs/metrics.md and docs/log-spamming.md were historical planning docs. They now list what the receiver emits and how it bounds its logs. - docs/performance.md loses the mimalloc feature Cargo.toml no longer has, the benchmark tables and the wrong producer defaults. - architecture.md's delivery and buffering sections move to docs/delivery.md. - Fix wrong keys, statuses and codes in sidecar-transports.md, webhook-setup.md, hardening.md and internals.md. The two mermaid diagrams render.
- sidecar-transports.md: the `vector` sink uses gRPC. `auth.mode: mtls` rejects every RPC. Client certificate authentication needs `grpc.auth.mode: none`, `grpc.tls.client_auth: required` and the CA. - architecture.md: edge protection or a load balancer allow list limits access to the receiver's ports from the internet. The webhook has its own listener only when `webhook.bind_address` is set. - hardening.md: the IP filter runs in every accept loop except those for the two gRPC ports. Cloudflare proxies HTTPS only on its listed ports. The HTTP rows need an Origin Rule to the receiver's port. gRPC needs an endpoint with TLS and HTTP/2 on 443. - internals.md: certificate material from a secret reference refreshes on the five HTTP listeners. The rest of `tls.*` is read once at start.
|
Released in v1.15.69 -- https://github.com/hyperi-io/dfe-receiver/releases/tag/v1.15.69 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
README.md, CONTRIBUTING.md, every page under docs/ and the two fixture READMEs match the code and config schema. They also match scalo and the repos they name. docs/delivery.md takes architecture.md's delivery and buffering sections.
The ledger has 351 rows: PASS 215, FAIL 122, UNCHECKABLE 14. Some rows group similar claims. All 122 FAIL rows are fixed. Claims the repo cannot show, such as a peer's retry rule or a supplier's pricing, stay UNCHECKABLE.
--config, only the flatDFE_RECEIVER_*list overrides the file.DFE_RECEIVER_SERVER__BIND_ADDRESStakes effect only without--config.+inrdkafka-sys, and pkg-config. Debian trixie's package is older.auth.mode: mtlson the gRPC and OTLP gRPC ports refuses every RPC. Certificate-only auth there ismode: nonewithtls.client_auth: required.mtlsorbothneeds a restart.122 corrected rows
protoc,cmake, a C toolchain.rdkafka-sysbuilds native codedynamic-linkingon every target (rdkafka-sys 4.10.0 build.rs:43-72)DFE_RECEIVER_SERVER__BIND_ADDRESSis how an env override is writtenserver.auth.modeis none, header, bearer or mtlsserver.auth.modealso acceptsbothrevert"Varies"Release: truetrailersrc/main.rs: the--emit-*generators--emit-dockerfile(main.rs:57-60). Artefacts come fromgenerate-artefactssrc/server/registered atmod.rs:125mod.rs:125common/mod.rsis the harnessorchestrator_lifecycle.rsConfig::validate(), nothing compares, driftedConfig::validate()"/livezand/readyzride its listener"format!()on the path"auth.bearer.tokens,accepted_headers[].valuesandkafka.sasl.passwordtake literal values (schema,x-scalo-secret). Spec fields refuse a literal (secrets.rs:30-52)auth.modenone by default, "limited only by body size, request timeout, the IP filter and the rate limiter"max_concurrent_requests(http/mod.rs:725-730) and the connection caps (http/mod.rs:505-518)format!()on hot path"mode: nonewithtls.client_auth: requiredInMemoryBuffer(buffer/tiered.rs:96-134). scalo'sTieredSinkis the disk spool (buffer/mod.rs:32-38)receiver_records_dropped_totalreason "one ofunavailable,rejectedorshutdown"DropReasonalso hasoversizeandkernel(metrics/mod.rs:158-180)mtlsorbothneed a restart_source equals crates_auditand setscrates_auditdata_stream.datasetscram_sha_512"works for ... Confluent Cloud"rawlands in_rawwith no mappingraw_source_fields(default none). dfe-schemas timeseries.yaml:62-66 gives_rawno@sourceWebhookHandleronly withwebhook.bind_address. Otherwise, routes merge on the HTTP listener (server/mod.rs:218-226, http/mod.rs:378-393)process_batch_to_topic"skips the router's enrichment, so the handler stamps_sourceand_timestamp_receiveditself"process_batch_to_topicstamps both fields (pipeline/mod.rs:1016-1021, :1301stamp_for_topic, webhook/mod.rs:13-17)receiver_kafka_resends_totalcounts message timeoutsconfig.example.yamlhas "the fullflow:block"try_recv_from(listener.rs:156). Removes the throughput comparisonTlsCertProvider, and only for*_secretmaterial (tls.rs:317-319, http/mod.rs:315-327). Files and the other listeners load oncemode: noneandtls.client_auth: requiredrequest_headers_timeout"not yet exposed" in ClientTrafficPolicy, #2598request_headers_timeout.request_headers_timeout,requestReceivedTimeoutandEnvoyPatchPolicyare Envoy names## Part 3692 words####variant headingsaws-load-balancer-cross-zone-load-balancing-enabledaws-load-balancer-attributes: load_balancing.cross_zone.enabled=false(kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/annotations/)dfe_not adoptedprotocolon receiver ingest metrics and onrecords_received_totaltransport(main.rs:388-442).records_received_totalhas no labels (manifest)prometheustrigger overscaling_pressure{job=...}metrics-apitrigger on dfe-keda-shimdfe_pipeline_readyscripts/pgo-workload.shexists, BOLT with itRUST-RELEASE-TRACK-OPTIMISATION.md,PGO-WORKLOAD-GUIDE.mddocs/runtime/pgo-bolt.mdexistsmimallocfeature and#[cfg]wiringscripts/pgo-workload.shexists, BOLT with it (P1)build.rust.optimize.pgo.enabled: true. BOLT needsbolt.enabled: trueworkload_setup_cmdstringscheckflush_*andmax_concurrent_insertsare dfe-loader keysKafkaProducer::HighThroughput... 256 KiB, 100 ms, LZ4"MEMORY_PRESSURE_THRESHOLD0.80,MEMORY_CGROUP_HEADROOM0.85, "documented in.envcomments"{PREFIX}_..., soDFE_RECEIVER_MEMORY_.... No.envin the repoMALLOC_CONF="prof:true"on a release buildprofilingfeature and reads_RJEM_MALLOC_CONF(README.md:38, :81-82, :128-131). Thejemallocfeature enables neither. The recipe's build has no jemallocdhat-heapfeature"Release: truetrailer (C10)fluent.accept_unauthenticated(config/mod.rs:624-631)http.StatusOKas an errorStatusAcceptedextraContainerssidecars, appended to the pod (_deployment.tpl:37), and noextraContainersinitContainer"auth.bearer_tokenslist"<listener>.auth.bearer.tokens(schema)auth.mtls"auth.mode: mtlswithtls.client_auth: required(config/mod.rs:807-836)server.max_body_sizelimits Vector batches. Set"64MiB"vectorsink is gRPC, bounded bygrpc.max_message_size(16 MiB).max_body_sizeis an integer. The probe refuses"64MiB"503", Vector buffers to "its disk buffer"UNAVAILABLE(grpc/mod.rs:105, :196). Vector's buffer is memory unless configured/healthport 8686api.enabled: trueghcr.io/hyperi-io/dfe-receiver:latest<version>pipeline/mod.rs:205,sink/kafka/mod.rs:61,syslog/mod.rs:73,lumberjack/mod.rs:94,135dfe_transport_send_errors_totalLOG_THROTTLE_BURST10,LOG_THROTTLE_RATE1,LOG_THROTTLE_MAX_SIGNATURES5000,LOG_THROTTLE_ENABLEDtruemissing_timestamp,invalid_timestampfilter_errorrowstandards/universal/testing.mdcargo nextest run --all-featuresfeatures: default(.hyperi-ci.yaml:24-25), which adds no flag (hyperi-ci main rust/test.py:160-166)