Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,9 +168,9 @@ runbook](docs/operations.md).

## Performance

Fanout v2026.10.8-alpha accepted **415,859 mixed OpenTelemetry items/s**
over 303.5 seconds on a Hetzner Cloud ccx33 (8 dedicated vCPUs, 32 GB RAM), with
telemetrygen on a separate ccx33. The ingest run on 16 October 2026 passed every
Fanout v2026.11.0-alpha accepted **507,311 mixed OpenTelemetry items/s**
over 306.8 seconds on a Hetzner Cloud ccx33 (8 dedicated vCPUs, 32 GB RAM), with
telemetrygen on a separate ccx33. The ingest run on 6 November 2026 passed every
quality gate. See the [performance methodology](https://fanout.run/docs/concepts/performance).
[Fanout Bench](https://github.com/labstack/fanout-bench) reproduces the measurement on your hardware.

Expand Down
74 changes: 38 additions & 36 deletions site/src/content/docs/docs/concepts/performance.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Performance
description: Measured ingest rates, hardware, methodology and benchmark evidence.
summary: Sustained mixed ingest at 415,859 items/s on a Hetzner ccx33.
summary: Sustained mixed ingest at 507,311 items/s on a Hetzner ccx33.
read_when:
- You want to understand the performance figure on the landing page.
- You need to measure an instance for your own load.
Expand All @@ -11,16 +11,16 @@ status: shipped
import BenchmarkSweep from "../../../../components/landing/BenchmarkSweep.astro";

On a Hetzner Cloud ccx33 (8 dedicated vCPUs, 32 GB RAM), with telemetrygen
on a separate ccx33 over a private network, Fanout v2026.10.8-alpha accepted
**415,859 mixed OpenTelemetry items/s** over **303.5 seconds**.
The composition was 57,037 spans, 179,134 log records and
179,688 metric points per second. The run measured ingest only.
on a separate ccx33 over a private network, Fanout v2026.11.0-alpha accepted
**507,311 mixed OpenTelemetry items/s** over **306.8 seconds**.
The composition was 61,571 spans, 224,112 log records and
221,628 metric points per second. The run measured ingest only. Reads under sustained ingest did not meet the read target and are tracked in [issue #419](https://github.com/labstack/fanout/issues/419).

## Setup and measurement

Both hosts were described as **Hetzner Cloud ccx33, fsn1, ubuntu-24.04**. The Fanout host
reported AMD EPYC-Milan Processor, 8 logical CPUs and
32,857,198,592 bytes of memory. telemetrygen v0.157.0 sent
32,857,186,304 bytes of memory. telemetrygen v0.157.0 sent
plaintext OTLP/gRPC over the private network, using 1,000-item batches
and 8 workers per process.

Expand All @@ -29,21 +29,22 @@ for traces, logs, metrics and mixed ingest, with 3 repetitions per point.
Accepted throughput is the change
in Fanout's `fanout_ingest_rows_total` counter over the measured scrape window.

The suite accepted **575.8 million items** in total: 135,133,541 spans,
210,698,970 log records and 229,999,166 metric points.
The suite accepted **667.3 million items** in total: 145,879,781 spans,
265,470,588 log records and 255,924,526 metric points.

## Measured mixed ingest

The sustained phase ran for **303.5 seconds** at 32 processes per signal.
Fanout averaged **7.24 CPU cores** and reached **11.4 GiB peak RSS**.
The generator host averaged **7.44 of 8 CPU cores**.
The sustained phase ran for **306.8 seconds** at 64 processes per signal.
Fanout averaged **7.73 CPU cores** and reached **10.6 GiB peak RSS**.
The generator host averaged **7.93 of 8 CPU cores**.
The sustained phase is classified generator-limited.

| Signal | Accepted items/s |
|---|---:|
| Spans | 57,037 |
| Log records | 179,134 |
| Metric points | 179,688 |
| Mixed total | 415,859 |
| Spans | 61,571 |
| Log records | 224,112 |
| Metric points | 221,628 |
| Mixed total | 507,311 |

telemetrygen keeps one trace export in flight per process, while log and
metric processes keep eight. Synthetic, highly compressible telemetrygen
Expand All @@ -57,10 +58,10 @@ are median average cores at that count; each host has 8 cores.

| Workload | Median peak items/s | Processes per signal | Range items/s | Fanout CPU cores | Generator CPU cores | Repetitions |
|---|---:|---:|---:|---:|---:|---:|
| Mixed | 402,062 | 32 | 357,956–433,339 | 7.07 | 7.34 | 3 |
| Traces | 248,901 | 32 | 207,920–295,566 | 6.22 | 5.19 | 3 |
| Logs | 387,579 | 64 | 353,862–393,976 | 6.85 | 4.63 | 3 |
| Metrics | 524,878 | 64 | 472,496–561,000 | 6.06 | 5.31 | 3 |
| Mixed | 460,640 | 64 | 453,181–485,193 | 7.22 | 7.39 | 3 |
| Traces | 297,345 | 64 | 192,120–330,450 | 6.97 | 7.24 | 3 |
| Logs | 622,229 | 64 | 558,932–648,672 | 6.02 | 6.38 | 3 |
| Metrics | 488,431 | 64 | 486,759–532,625 | 6.35 | 4.71 | 3 |

## Mixed sweep

Expand All @@ -69,49 +70,50 @@ count. CPU columns are median average cores per host.

| Processes per signal | Median items/s | Min items/s | Max items/s | Fanout CPU cores | Generator CPU cores |
|---|---:|---:|---:|---:|---:|
| 1 | 39,041 | 38,296 | 39,149 | 4.72 | 0.80 |
| 2 | 77,784 | 75,221 | 78,573 | 4.30 | 1.77 |
| 4 | 146,792 | 146,445 | 155,962 | 4.86 | 1.87 |
| 8 | 223,985 | 216,861 | 238,993 | 6.06 | 3.03 |
| 16 | 359,086 | 335,283 | 379,219 | 6.35 | 5.78 |
| 32 | 402,062 | 357,956 | 433,339 | 7.07 | 7.34 |
| 64 | 385,274 | 384,289 | 414,315 | 6.86 | 7.35 |
| 1 | 38,820 | 38,640 | 39,227 | 5.47 | 0.82 |
| 2 | 76,587 | 76,483 | 76,741 | 6.15 | 1.71 |
| 4 | 155,156 | 149,499 | 155,707 | 5.91 | 2.16 |
| 8 | 262,749 | 261,501 | 270,231 | 7.01 | 2.93 |
| 16 | 380,026 | 323,029 | 429,748 | 7.33 | 5.00 |
| 32 | 442,361 | 432,734 | 450,588 | 7.37 | 7.14 |
| 64 | 460,640 | 453,181 | 485,193 | 7.22 | 7.39 |

## Signal sweeps

The charts share a vertical scale and an evenly spaced log2 process axis.
Lines and points show the median; bands show the recorded minimum and maximum.
The diamond marks the sustained phase at 32 processes per signal.
The diamond marks the sustained phase at 64 processes per signal.
The shaded region marks generator CPU of at least 7 cores.

<BenchmarkSweep id="performance" />

## Evidence

The [published evidence](https://github.com/labstack/fanout-bench/tree/main/published/2026-10-16-ccx33x2-v2026.10.8-alpha)
The [published evidence](https://github.com/labstack/fanout-bench/tree/main/published/2026-11-06-ccx33x2-v2026.11.0-alpha)
contains `report.json` and `samples.jsonl`, produced by
[Fanout Bench](https://github.com/labstack/fanout-bench) on 16 October 2026 at
17:36 UTC. All quality gates passed: successful generator exits, zero dropped
[Fanout Bench](https://github.com/labstack/fanout-bench) on 6 November 2026 at
04:15 UTC. All quality gates passed: successful generator exits, zero dropped
rows, no restarts, authoritative accepted-row counters in every phase and
verified Parquet storage.

The harness commit was
[`8f5864ba488d9802f68250d85f67b6658ab13af5`](https://github.com/labstack/fanout-bench/commit/8f5864ba488d9802f68250d85f67b6658ab13af5).
The [Fanout release artifact](https://github.com/labstack/fanout/releases/download/v2026.10.8-alpha/fanout_v2026.10.8-alpha_linux_amd64.tar.gz) has SHA-256:
[`5462cbbcde9c49521c918c45f45312493aa4bb24`](https://github.com/labstack/fanout-bench/commit/5462cbbcde9c49521c918c45f45312493aa4bb24); the harness working tree was clean.
The [Fanout release artifact](https://github.com/labstack/fanout/releases/download/v2026.11.0-alpha/fanout_v2026.11.0-alpha_linux_amd64.tar.gz) has SHA-256:

```text
b314920221693683b7d6ff42787a37e82bee1fbdd7ddcc8647d1905f7629ac6b
1e83657b9eb0ff7e7ad406bd5162642badf087c380448deebd1f88b183656ddc
```

## Reproduce

From the [fanout-bench](https://github.com/labstack/fanout-bench) repository:

```sh
scripts/hetzner.sh \
--fanout-version v2026.10.8-alpha \
FANOUT_BENCH_HCLOUD_SSH_KEY='<key>' scripts/hetzner.sh \
--fanout-version v2026.11.0-alpha \
--preset publish \
--repeat 3
--repeat 3 \
--output results/publish-ingest-ccx33x2-v2026.11.0-alpha
```

Both VMs default to ccx33. The
Expand Down
2 changes: 1 addition & 1 deletion site/src/content/docs/docs/reference/capabilities.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ capabilities uses `preview` if any of its contracts remain unsettled.
| OIDC sign-in | shipped | Provider flow, with group and domain mapping. |
| Generated settings reference | shipped | `cmd/fanout-docgen` writes it; `just check` fails when it drifts from the type. |
| Release checksums and provenance | shipped | `SHA256SUMS` plus GitHub build attestations on every archive. |
| Published ingest benchmark | shipped | [Performance](/docs/concepts/performance) reports v2026.10.8-alpha with three sweep repetitions and [raw evidence](https://github.com/labstack/fanout-bench/tree/main/published/2026-10-16-ccx33x2-v2026.10.8-alpha). |
| Published ingest benchmark | shipped | [Performance](/docs/concepts/performance) reports v2026.11.0-alpha with three sweep repetitions and [raw evidence](https://github.com/labstack/fanout-bench/tree/main/published/2026-11-06-ccx33x2-v2026.11.0-alpha). |
| Signatures and SBOM | planned | Unavailable; see the [release contract](https://github.com/labstack/fanout/blob/main/docs/release.md#supply-chain-scope). |
| Namespace scoping on alert rules | planned | The field is stored but evaluation ignores it; scope with `service`. |

Expand Down
Loading
Loading