kubernetes-logs-datalake

Fluent Bit DaemonSet that collects Kubernetes container logs and writes them as Arrow IPC (and, in the demo, Parquet) objects to S3-compatible storage.

Production (Yolean g2) writes arrow only — the parquet output was dropped to halve GCS write operations. The k3d demo below still writes both, because the schema assertions compare them.

Build

images/fluentbit/ compiles Fluent Bit (see FLB_VERSION in its Dockerfile) with Apache Arrow/Parquet support (FLB_ARROW=On), Kafka output and LuaJIT, on a distroless debian13 nonroot runtime. Three patches are applied on top of upstream: a GCS response header lookup fix, an S3 flush on container log close, and the compact-columns Arrow writer.

docker build -t yolean/fluentbit:latest images/fluentbit/

Read images/fluentbit/REQUIREMENTS.md before changing the writer or bumping FLB_VERSION. It states the format invariants consumers depend on, which of them are asserted, and why a 5.x bump is a port rather than a version change.

Example

k3d-example/ provides a complete k3d-based demo: container log forwarding via a DaemonSet, writing to versitygw (S3-compatible gateway with a posix backend).

Prerequisites

• k3d
• Docker
• kubectl
• duckdb (for the query assertions)

Run

./test.sh from the repo root is the entry point: it creates the cluster if needed, deploys everything, waits for objects to land and runs the assertions. k3d-example/setup.sh and teardown.sh manage the cluster on their own if you want to poke at it by hand.

./test.sh

This will:

1. Create a k3d cluster fluentbit-demo
2. Import the locally built yolean/fluentbit:latest image
3. Deploy versitygw (S3-compatible storage with posix backend)
4. Create the fluentbit-logs S3 bucket
5. Deploy the Fluent Bit DaemonSet (tail → S3 arrow + parquet output)
6. Deploy a busybox log-generator that emits JSON every second
7. Assert on the objects that land, via duckdb and pyarrow

All kubectl access goes through --kubeconfig=$(pwd)/k3d-example/kubeconfig.

Teardown

bash k3d-example/teardown.sh

Data flow

busybox (log-generator) → stdout JSON
  → /var/log/pods/**/*.log (k3d node)
  → fluent-bit tail input → compact-columns → S3 output (arrow, parquet)
  → versitygw:7070 → /data/fluentbit-logs/