dfe-receiver

The front door of a DFE deployment: takes events in the protocols agents already speak and hands them on as JSON.

What it does

• One service for HTTP(S) JSON, Vector's gRPC sink, OTLP, Beats, Splunk HEC, syslog, Fluent Forward, GELF, Prometheus Remote Write and webhooks, so an agent keeps the protocol it already speaks. Every handler normalises to JSON and hands the record to Kafka or straight to dfe-loader over gRPC.
• Ordered routing rules that set each record's source and org from fields it already carries, such as data_stream.dataset, and pick its Kafka topic from the source.
• A sender answered only once every destination has confirmed its records, on by default for each listener that can hold its answer. POST /ingest answers 202 when the batch is accepted, 503 with Retry-After when it cannot take it, and 413 over the body limit, 10,485,760 bytes by default.
• Each record's org held to what its sender may use. A credential or listener bound in ingest_bindings lands only in its own orgs, and an unbound sender only in the deployment's default org.
• Header, bearer token and mTLS authentication.
• Built on scalo, so config, logs, metrics, health probes and memory backpressure work the way they do in every DFE Rust service.

Protocol	Port	Source
HTTP(S) JSON	8080	Vector, custom agents
gRPC Vector sink	6000	Vector
Webhook (POST /webhook/{caller})	shares HTTP, or 8090	Product alert rules and notification hooks (runZero, ...)
OTLP gRPC	4317	OpenTelemetry collectors
OTLP HTTP	4318	OpenTelemetry collectors
Lumberjack/Beats	5044	Filebeat, Logstash Beats output
Splunk HEC	8088	Splunk forwarders, Fluentd HEC output
Syslog UDP/TCP/TLS	514 / 6514	syslogd, rsyslog, syslog-ng
Fluent Forward	24224	Fluentd, Fluent Bit
GELF TCP	12201	Graylog GELF output, Fluent Bit
Prometheus Remote Write	9091	Prometheus, VictoriaMetrics
Flow (NetFlow + sFlow), experimental	2055 / 4739 / 6343 UDP	NetFlow v5/v9, IPFIX, sFlow v5 exporters (v7 not supported)

HTTP always runs, and the other listeners are opt-in. A listener that answers at enqueue (acknowledgements off, or syslog, GELF and flow) buffers in memory behind a circuit breaker, and disk spillover is opt-in.

Quick start

The build needs protoc, a C toolchain, pkg-config and an installed librdkafka. The gRPC, OTLP and Prometheus Remote Write protocols compile vendored .proto files, aws-lc-sys compiles C, and rdkafka-sys links librdkafka through pkg-config. It refuses a librdkafka older than the version after the + in its Cargo.lock entry, and a Debian or Ubuntu package can be older than that, so there build librdkafka from source first.

sudo apt-get install -y protobuf-compiler build-essential pkg-config   # Debian or Ubuntu
brew install protobuf pkg-config librdkafka                              # macOS
cargo build --release
./target/release/dfe-receiver --config config.yaml

Only the brokers have no default, so this is the smallest config that moves data. The other keys are written out at their defaults:

server:
  bind_address: "0.0.0.0:8080"
  auth:
    mode: none

kafka:
  brokers:
    - "localhost:9092"

routing:
  default_source: "main"
  topic_suffix: "_land"

Configuration

docs/config-schema.yaml lists every setting with its default, generated from the code, and config.example.yaml is an annotated starting point. Started with --config, as the image is, the receiver reads a fixed set of flat environment variables over the file, such as DFE_RECEIVER_BIND_ADDRESS and DFE_RECEIVER_KAFKA_BROKERS. Started without it, scalo's cascade also reads any key as its path with double underscores, DFE_RECEIVER_SERVER__BIND_ADDRESS. server.auth.mode is none, header, bearer, mtls or both. The routing rules, the webhook intake, the metrics and the hot-path design are in docs/internals.md.

Documentation

• docs/architecture.md: where it sits in the suite and why it is built this way.
• docs/delivery.md: when a sender is answered, and what the receiver buffers.
• docs/internals.md: how each part works -- parsing, routing, what a sender is told, auth, configuration, the webhook intake and flow.
• docs/hardening.md: running it where it is reachable from outside.
• docs/sidecar-transports.md and docs/webhook-setup.md: the transport options and the webhook callers.
• docs/metrics.md and docs/performance.md: what it reports and what it costs.

The suite's published docs are at https://dfe.docs.hyperi.io, and this repository's pages at https://dfe.docs.hyperi.io/dfe-receiver/.

Help and contributing

How to work in this repo, the commands that prove a change and what tends to bite are in CONTRIBUTING.md. A question that is not a bug goes in a new issue.

Open bugs: this repo | whole DFE suite

License

BUSL-1.1, see LICENSE.