Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 38 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@ The public SSH targets are used by your local machine for provisioning. The two

`managed` remains the sole orchestrator on the Fanout host: it binds Fanout to the supplied private IP on a free port, performs setup, samples metrics, issues read load, verifies storage, and writes the report. Every telemetrygen process runs through SSH on the generator host, bounded by a remote timeout. Cleanup removes the marked authorized-key line, the private key, and both stages. Reports record both host descriptions and `generator_on_target_host: false`; reproduction commands redact both public targets and omit the private IP flags.

To measure an unreleased build, pass `--fanout-binary <path>` with a local Linux Fanout binary for the remote architecture and a non-release label such as `--fanout-version explore-fdfbf7f5`. The binary is identified by its SHA-256 as `unreleased-local-binary`, and the report warns that the run is diagnostic and must not be published.

Nothing is installed system-wide. Existing Fanout services and data directories are not touched. Neither remote host needs a Go toolchain or Docker daemon, and Fanout needs no preconfigured account. The Fanout host needs an SSH client; the generator host needs an SSH server and GNU `timeout` (included in the default Ubuntu image).

> `telemetrygen` currently accepts OTLP headers through a command-line flag and prints those headers in its startup configuration. Fanout Bench suppresses and redacts child output, but the short-lived ingest token can still be visible to local process-listing tools while a generator is running. Use a dedicated token and revoke it after the run.
Expand All @@ -89,7 +91,9 @@ Nothing is installed system-wide. Existing Fanout services and data directories
|---|---|---:|---:|---|
| `quick` | 1 topology per signal and mixed | 5s | 10s | installation smoke test |
| `standard` | 1, 2, 4, 8 processes per signal | 15s | 60s | engineering comparison |
| `validate` | up to 1, 2, 4, 8, 16, 32, 64 processes per signal; stops on plateau/collapse | 30s | 5m | faster validation; full quality gates |
| `publish` | 1, 2, 4, 8, 16, 32, 64 processes per signal | 30s | 5m | repeatable public evidence |
| `local` | 8, 16, 32 processes per signal | 30s | 5m | same-host A/B check before a Hetzner run |

The default topology is 8 workers per process at 2,500 items/s per worker with 1,000-item OTLP batches. A mixed step launches the same topology independently for traces, logs, and metrics. Without read load, the sustained proof reruns the mixed topology with the highest **accepted** throughput. With `--query-rate`, it chooses the highest-throughput topology that also completed at least 95% of the requested reads with no errors or scheduler shedding. If no sweep point qualifies, it repeats the best read-completion point and fails the sustained-read quality gate rather than publishing a write-only peak as mixed capacity. Both reports record the selected sustained process count per signal. The wider publish sweep adds independent exporters to get past telemetrygen's per-process limits; see [the source analysis](docs/telemetrygen.md).

Expand All @@ -108,6 +112,28 @@ scripts/hetzner.sh \

Publication runs measure Fanout’s ingest ceiling with all eight ccx33 vCPUs available to Fanout; telemetrygen runs on the separate ccx33 over the private network. A plain `--host` run without the three split flags remains a same-host run, and its report warns about generator contention.

## Faster Hetzner validation

Run ingest and both read groups concurrently on three separate VM pairs:

```sh
scripts/hetzner.sh --parallel-groups \
--fanout-version v2026.10.1-alpha --preset validate --repeat 3 \
--output results/validate-ingest \
-- \
--fanout-version v2026.10.1-alpha --preset validate --repeat 1 \
--signals mixed --query-rate 20 --rate 210 \
--output results/validate-reads-70 \
-- \
--fanout-version v2026.10.1-alpha --preset validate --repeat 1 \
--signals mixed --query-rate 20 \
--output results/validate-reads-saturated
```

Set the Hetzner SSH-key environment variable as in Quick start. Expect **about two hours instead of four**, bounded by the longest group's setup, load, verification and download; early stopping may save more. At peak this uses six VMs. With two started hours for ingest and one for each read pair, budget **8 ccx33 VM-hours**, plus public IPv4 charges and tax—about the previous four-hour pair's VM cost. If both read pairs cross an hour boundary, budget 12 VM-hours. Check current location rates. Each pair is deleted when its group finishes, including failure and Ctrl-C paths; resource names are printed and failed deletion exits nonzero.

`--signals mixed` skips duplicate single-signal sweeps and labels them skipped in reports. `validate` keeps publish's durations and maximum topology, but stops a signal/repetition after two gains below 5% or a collapsing acceptance ratio; reports record the stop and skipped phases. It keeps the sustained proof and quality gates. `publish` remains a full sweep with early stopping disabled. The `210` rate is per worker and must be recalibrated for a different target's approximate 70% load. See the [validation recipe and cost assumptions](docs/publishing.md#faster-validation-runs); validation does not replace publication evidence.

## Read and write together

The write-only suite answers “how quickly can Fanout ingest?” It does not answer “what happens to dashboard latency while ingesting?” Add automatically authenticated read load to mixed phases:
Expand All @@ -119,10 +145,18 @@ scripts/hetzner.sh \
--query-rate 50
```

The harness uses a bounded open-loop scheduler targeting user operations per second at 15-minute, 1-hour, 6-hour, 12-hour, and 24-hour windows. Operations rotate through dashboard refresh, service-map trace drill-down, and trace detail. A refresh sends the full System overview panel batch exactly as the UI's initial all-visible load does, then requests annotations over the returned time-panel bounds. Layout widths model a fixed 1664-pixel viewport and are rounded as in the UI. Drill-down uses `POST /api/panels/exemplars` with the selected service and the panel response's absolute window. Trace detail uses `GET /api/traces/:id` with a discovered ID, namespace, and that same window, retained separately for each range; an empty detail is a failure. Until a trace is discovered, detail slots perform drill-down; until a selectable panel is cached, those slots perform dashboard refresh instead. Each slot performs one user operation. The current System overview has no log drill or standalone log/trace search operation. Each scheduled read measures every request in its operation, and any request or panel error fails the operation. The report records per-operation counts and latency in its methodology alongside the dashboard definition hash; aggregate fields and quality gates retain their schema.
The harness targets only the current Fanout API: `POST /api/queries/run`, `POST /api/queries/exemplars`, and `POST /api/annotations/query`. It uses a bounded open-loop scheduler targeting user operations per second at 15-minute, 1-hour, 6-hour, 12-hour, and 24-hour windows. Operations rotate through dashboard refresh, service-map trace drill-down, and trace detail. A refresh sends the full System overview panel batch exactly as the UI's initial all-visible load does, then requests annotations over the returned time-panel bounds. Layout widths model a fixed 1664-pixel viewport and are rounded as in the UI. Drill-down uses `POST /api/queries/exemplars` with the selected service and the panel response's absolute window. Trace detail uses `GET /api/traces/:id` with a discovered ID, namespace, and that same window, retained separately for each range; an empty detail is a failure. Until a trace is discovered, detail slots perform drill-down; until a selectable panel is cached, those slots perform dashboard refresh instead. Each slot performs one user operation. The current System overview has no log drill or standalone log/trace search operation. Each scheduled read measures every request in its operation, and any request or panel error fails the operation. The report records per-operation counts and latency in its methodology alongside the dashboard definition hash; aggregate fields and quality gates retain their schema.

It reports attempted and completed rates, phase-end cancellations, explicit scheduler shedding, failures, mean, p50, p95, p99, and maximum latency. The concurrency bound prevents a stalled server from creating unbounded requests.

Each phase also reports a **Dashboard panels** table with per-panel outcome
counts, cache-hit percentage, and server-duration p50/p95/p99, including failed
refreshes. Sustained-phase failure contributors are flagged. Structured panel
evidence, including freshness bounds, appears in `trials[].dashboard_panels`;
server execution and admission metrics appear in `samples.jsonl` under
`panel_metrics` when the server exposes them. See the
[panel evidence definitions](docs/metrics.md#dashboard-panel-evidence).

## Output

Every run produces one self-contained directory:
Expand All @@ -136,7 +170,7 @@ results/20261014T190000Z/
└── fanout.log setup-token-scrubbed server log
```

`samples.jsonl` includes cumulative `ingest_rows_total` by signal (`spans`, `logs`, `metrics`), `rollup_seconds` by rollup (`service`, `edge`, `version`, `read_cache`), `write_gate_hold_seconds` and `write_gate_wait_seconds` by operation, `compaction_seconds_total`, `compactions_total`, `publish_wait_seconds_total`, and `ingest_backpressure_seconds_total`, plus the current `compaction_backlog_batches` gauge. Derive per-second accepted-row rates from consecutive samples' `ingest_rows_total` deltas divided by their `at_seconds` difference. Missing metrics from older Fanout releases leave these fields absent or zero. See [sample definitions](docs/metrics.md#background-work-samples) for their sources and interpretation.
`samples.jsonl` includes cumulative `ingest_rows_total` by signal (`spans`, `logs`, `metrics`), `rollup_seconds` by rollup (`service`, `edge`, `version`, `read_cache`), `write_gate_hold_seconds` and `write_gate_wait_seconds` by operation, `compaction_seconds_total`, `compactions_total`, `publish_wait_seconds_total`, `ingest_backpressure_seconds_total`, `ingest_throttled_total` by reason, `ingest_admission_rows_per_second`, `compaction_retired_batches_total`, and `compaction_pass_seconds_total`, plus the current `compaction_backlog_batches` gauge. Derive per-second accepted-row rates from consecutive samples' `ingest_rows_total` deltas divided by their `at_seconds` difference. Missing metrics leave these fields absent or zero. See [sample definitions](docs/metrics.md#background-work-samples) for their sources and interpretation.

The terminal output is optimized for decisions:

Expand Down Expand Up @@ -212,6 +246,8 @@ Presets make comparisons easy. Overrides are intended for diagnostics and ceilin
3 signals × processes-per-signal × workers-per-process × rate-per-worker
```

Retryable throttling is valid when telemetrygen retries and exits successfully. Accepted throughput still comes only from Fanout counters. Any nonzero generator exit, including exhaustion of its export retry timeout, invalidates the phase and fails the generator-processes gate; throttling never excuses exporter death.

Avoid unbounded telemetrygen runs. Its SDK queues may drop locally while the console continues to count generated data. Fanout Bench always uses bounded rates and treats Fanout's accepted counter as authoritative.

## Development
Expand Down
12 changes: 11 additions & 1 deletion cmd/fanout-bench/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package main

import (
"context"
"errors"
"flag"
"fmt"
"os"
Expand Down Expand Up @@ -72,10 +73,12 @@ func commonFlags(name string, args []string) (bench.Config, error) {
fs.StringVar(&cfg.GeneratorAddress, "generator-address", "", "generator host IP reachable from the Fanout host")
fs.StringVar(&cfg.FanoutListen, "fanout-listen", "", "Fanout host IP reachable from the generator host")
fs.StringVar(&cfg.GeneratorHostDescription, "generator-host-description", "", "generator host description recorded in the report")
fs.StringVar(&cfg.FanoutBinary, "fanout-binary", "", "local Linux Fanout binary for a remote run instead of a release; --fanout-version is then a non-release label and the report is marked not publishable")
}
fs.IntVar(&cfg.SSHPort, "ssh-port", 22, "SSH port")
fs.BoolVar(&cfg.KeepRemote, "keep-remote", false, "preserve the isolated remote run directory after downloading artifacts")
fs.StringVar(&cfg.Preset, "preset", "standard", "suite: quick, standard, or publish")
fs.StringVar(&cfg.SignalSelection, "signals", "all", "sweeps: all or mixed (skips single-signal sweeps)")
fs.StringVar(&cfg.Preset, "preset", "standard", "suite: quick, standard, local, validate, or publish")
fs.StringVar(&cfg.Endpoint, "endpoint", "localhost:7520", "Fanout OTLP/gRPC endpoint")
fs.StringVar(&cfg.MetricsURL, "metrics-url", "http://localhost:7520/metrics", "Fanout Prometheus metrics URL (required evidence source)")
fs.BoolVar(&cfg.Insecure, "insecure", false, "use plaintext OTLP/gRPC")
Expand Down Expand Up @@ -104,9 +107,15 @@ func commonFlags(name string, args []string) (bench.Config, error) {
if err := fs.Parse(args); err != nil {
return cfg, err
}
if err := bench.ValidateSignalSelection(cfg.SignalSelection); err != nil {
return cfg, err
}
if err := bench.ValidateGeneratorHosts(cfg); err != nil {
return cfg, err
}
if cfg.FanoutBinary != "" && cfg.Host == "" {
return cfg, errors.New("--fanout-binary requires a remote run with --host")
}
cfg.Token = os.Getenv(tokenEnv)
cfg.MetricsToken = os.Getenv(metricsTokenEnv)
cfg.QueryCookie = os.Getenv(queryCookieEnv)
Expand Down Expand Up @@ -162,6 +171,7 @@ func managed(ctx context.Context, args []string) error {
fs.StringVar(&cfg.OutputDir, "output", "./results", "remote evidence directory")
fs.StringVar(&cfg.FanoutVersion, "fanout-version", "unknown", "Fanout version")
fs.StringVar(&cfg.FanoutArtifact, "fanout-artifact", "", "verified release artifact identity")
fs.StringVar(&cfg.SignalSelection, "signals", "all", "sweeps: all or mixed (skips single-signal sweeps)")
fs.StringVar(&cfg.Preset, "preset", "standard", "suite preset")
fs.IntVar(&cfg.Repeat, "repeat", 1, "sweep repetitions")
fs.IntVar(&cfg.BatchSize, "batch-size", 1000, "OTLP batch size")
Expand Down
38 changes: 37 additions & 1 deletion cmd/fanout-bench/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ func TestSplitReproductionCommandPassesValidationAfterSubstitution(t *testing.T)
cfg, err := commonFlags("run", []string{
"--host", "root@192.0.2.10", "--generator-host", "root@192.0.2.11",
"--generator-address", "10.77.0.3", "--fanout-listen", "10.77.0.2",
"--ssh-port", "2222", "--telemetrygen", executable, "--preset", "quick",
"--ssh-port", "2222", "--telemetrygen", executable, "--preset", "validate", "--signals", "mixed",
})
if err != nil {
t.Fatal(err)
Expand All @@ -41,6 +41,9 @@ func TestSplitReproductionCommandPassesValidationAfterSubstitution(t *testing.T)
if err == nil || report == nil {
t.Fatalf("expected partial report on cancellation, got report=%v err=%v", report, err)
}
if !strings.Contains(report.Settings.Methodology, "were skipped") || !report.Settings.EarlyStop {
t.Fatal("missing validation coverage")
}
command := strings.NewReplacer(
"<host>", "192.0.2.20", "<generator-host>", "192.0.2.21",
"<generator-address>", "10.77.0.13", "<fanout-listen>", "10.77.0.12",
Expand All @@ -55,6 +58,9 @@ func TestSplitReproductionCommandPassesValidationAfterSubstitution(t *testing.T)
if err != nil {
t.Fatalf("substituted reproduction failed validation: %v\n%s", err, command)
}
if reproduced.SignalSelection != "mixed" || reproduced.Preset != "validate" {
t.Fatal("validation flags lost")
}
if reproduced.Host != "root@192.0.2.20" || reproduced.GeneratorHost != "root@192.0.2.21" ||
reproduced.GeneratorAddress != "10.77.0.13" || reproduced.FanoutListen != "10.77.0.12" || reproduced.SSHPort != 2222 {
t.Fatalf("reproduction lost topology: %+v", reproduced)
Expand Down Expand Up @@ -109,3 +115,33 @@ func TestSplitFlagsRequiredTogetherAndRemoteOnly(t *testing.T) {
t.Fatalf("generator description=%q, err=%v", cfg.GeneratorHostDescription, err)
}
}

func TestFanoutBinaryRequiresRemoteHost(t *testing.T) {
if _, err := commonFlags("run", []string{"--fanout-binary", "/tmp/fanout", "--fanout-version", "explore-test"}); err == nil {
t.Fatal("accepted --fanout-binary without --host")
}
cfg, err := commonFlags("run", []string{"--host", "root@192.0.2.10", "--fanout-binary", "/tmp/fanout", "--fanout-version", "explore-test"})
if err != nil {
t.Fatal(err)
}
if cfg.FanoutBinary != "/tmp/fanout" {
t.Fatalf("fanout binary=%q", cfg.FanoutBinary)
}
}

func TestValidationFlags(t *testing.T) {
for _, args := range [][]string{{"--signals", "mixed", "--preset", "validate"}, {"--signals=mixed", "--preset=publish"}} {
cfg, err := commonFlags("run", args)
if err != nil {
t.Fatal(err)
}
if cfg.SignalSelection != "mixed" {
t.Fatal(cfg.SignalSelection)
}
}
for _, args := range [][]string{{"--signals"}, {"--signals", "logs"}, {"--signals", "mixed,traces"}} {
if _, err := commonFlags("run", args); err == nil {
t.Fatalf("accepted %v", args)
}
}
}
Loading
Loading