diff --git a/README.md b/README.md index c768cea..eae00f2 100644 --- a/README.md +++ b/README.md @@ -79,6 +79,8 @@ The public SSH targets are used by your local machine for provisioning. The two `managed` remains the sole orchestrator on the Fanout host: it binds Fanout to the supplied private IP on a free port, performs setup, samples metrics, issues read load, verifies storage, and writes the report. Every telemetrygen process runs through SSH on the generator host, bounded by a remote timeout. Cleanup removes the marked authorized-key line, the private key, and both stages. Reports record both host descriptions and `generator_on_target_host: false`; reproduction commands redact both public targets and omit the private IP flags. +To measure an unreleased build, pass `--fanout-binary ` with a local Linux Fanout binary for the remote architecture and a non-release label such as `--fanout-version explore-fdfbf7f5`. The binary is identified by its SHA-256 as `unreleased-local-binary`, and the report warns that the run is diagnostic and must not be published. + Nothing is installed system-wide. Existing Fanout services and data directories are not touched. Neither remote host needs a Go toolchain or Docker daemon, and Fanout needs no preconfigured account. The Fanout host needs an SSH client; the generator host needs an SSH server and GNU `timeout` (included in the default Ubuntu image). > `telemetrygen` currently accepts OTLP headers through a command-line flag and prints those headers in its startup configuration. Fanout Bench suppresses and redacts child output, but the short-lived ingest token can still be visible to local process-listing tools while a generator is running. Use a dedicated token and revoke it after the run. @@ -89,7 +91,9 @@ Nothing is installed system-wide. Existing Fanout services and data directories |---|---|---:|---:|---| | `quick` | 1 topology per signal and mixed | 5s | 10s | installation smoke test | | `standard` | 1, 2, 4, 8 processes per signal | 15s | 60s | engineering comparison | +| `validate` | up to 1, 2, 4, 8, 16, 32, 64 processes per signal; stops on plateau/collapse | 30s | 5m | faster validation; full quality gates | | `publish` | 1, 2, 4, 8, 16, 32, 64 processes per signal | 30s | 5m | repeatable public evidence | +| `local` | 8, 16, 32 processes per signal | 30s | 5m | same-host A/B check before a Hetzner run | The default topology is 8 workers per process at 2,500 items/s per worker with 1,000-item OTLP batches. A mixed step launches the same topology independently for traces, logs, and metrics. Without read load, the sustained proof reruns the mixed topology with the highest **accepted** throughput. With `--query-rate`, it chooses the highest-throughput topology that also completed at least 95% of the requested reads with no errors or scheduler shedding. If no sweep point qualifies, it repeats the best read-completion point and fails the sustained-read quality gate rather than publishing a write-only peak as mixed capacity. Both reports record the selected sustained process count per signal. The wider publish sweep adds independent exporters to get past telemetrygen's per-process limits; see [the source analysis](docs/telemetrygen.md). @@ -108,6 +112,28 @@ scripts/hetzner.sh \ Publication runs measure Fanout’s ingest ceiling with all eight ccx33 vCPUs available to Fanout; telemetrygen runs on the separate ccx33 over the private network. A plain `--host` run without the three split flags remains a same-host run, and its report warns about generator contention. +## Faster Hetzner validation + +Run ingest and both read groups concurrently on three separate VM pairs: + +```sh +scripts/hetzner.sh --parallel-groups \ + --fanout-version v2026.10.1-alpha --preset validate --repeat 3 \ + --output results/validate-ingest \ + -- \ + --fanout-version v2026.10.1-alpha --preset validate --repeat 1 \ + --signals mixed --query-rate 20 --rate 210 \ + --output results/validate-reads-70 \ + -- \ + --fanout-version v2026.10.1-alpha --preset validate --repeat 1 \ + --signals mixed --query-rate 20 \ + --output results/validate-reads-saturated +``` + +Set the Hetzner SSH-key environment variable as in Quick start. Expect **about two hours instead of four**, bounded by the longest group's setup, load, verification and download; early stopping may save more. At peak this uses six VMs. With two started hours for ingest and one for each read pair, budget **8 ccx33 VM-hours**, plus public IPv4 charges and tax—about the previous four-hour pair's VM cost. If both read pairs cross an hour boundary, budget 12 VM-hours. Check current location rates. Each pair is deleted when its group finishes, including failure and Ctrl-C paths; resource names are printed and failed deletion exits nonzero. + +`--signals mixed` skips duplicate single-signal sweeps and labels them skipped in reports. `validate` keeps publish's durations and maximum topology, but stops a signal/repetition after two gains below 5% or a collapsing acceptance ratio; reports record the stop and skipped phases. It keeps the sustained proof and quality gates. `publish` remains a full sweep with early stopping disabled. The `210` rate is per worker and must be recalibrated for a different target's approximate 70% load. See the [validation recipe and cost assumptions](docs/publishing.md#faster-validation-runs); validation does not replace publication evidence. + ## Read and write together The write-only suite answers “how quickly can Fanout ingest?” It does not answer “what happens to dashboard latency while ingesting?” Add automatically authenticated read load to mixed phases: @@ -119,10 +145,18 @@ scripts/hetzner.sh \ --query-rate 50 ``` -The harness uses a bounded open-loop scheduler targeting user operations per second at 15-minute, 1-hour, 6-hour, 12-hour, and 24-hour windows. Operations rotate through dashboard refresh, service-map trace drill-down, and trace detail. A refresh sends the full System overview panel batch exactly as the UI's initial all-visible load does, then requests annotations over the returned time-panel bounds. Layout widths model a fixed 1664-pixel viewport and are rounded as in the UI. Drill-down uses `POST /api/panels/exemplars` with the selected service and the panel response's absolute window. Trace detail uses `GET /api/traces/:id` with a discovered ID, namespace, and that same window, retained separately for each range; an empty detail is a failure. Until a trace is discovered, detail slots perform drill-down; until a selectable panel is cached, those slots perform dashboard refresh instead. Each slot performs one user operation. The current System overview has no log drill or standalone log/trace search operation. Each scheduled read measures every request in its operation, and any request or panel error fails the operation. The report records per-operation counts and latency in its methodology alongside the dashboard definition hash; aggregate fields and quality gates retain their schema. +The harness targets only the current Fanout API: `POST /api/queries/run`, `POST /api/queries/exemplars`, and `POST /api/annotations/query`. It uses a bounded open-loop scheduler targeting user operations per second at 15-minute, 1-hour, 6-hour, 12-hour, and 24-hour windows. Operations rotate through dashboard refresh, service-map trace drill-down, and trace detail. A refresh sends the full System overview panel batch exactly as the UI's initial all-visible load does, then requests annotations over the returned time-panel bounds. Layout widths model a fixed 1664-pixel viewport and are rounded as in the UI. Drill-down uses `POST /api/queries/exemplars` with the selected service and the panel response's absolute window. Trace detail uses `GET /api/traces/:id` with a discovered ID, namespace, and that same window, retained separately for each range; an empty detail is a failure. Until a trace is discovered, detail slots perform drill-down; until a selectable panel is cached, those slots perform dashboard refresh instead. Each slot performs one user operation. The current System overview has no log drill or standalone log/trace search operation. Each scheduled read measures every request in its operation, and any request or panel error fails the operation. The report records per-operation counts and latency in its methodology alongside the dashboard definition hash; aggregate fields and quality gates retain their schema. It reports attempted and completed rates, phase-end cancellations, explicit scheduler shedding, failures, mean, p50, p95, p99, and maximum latency. The concurrency bound prevents a stalled server from creating unbounded requests. +Each phase also reports a **Dashboard panels** table with per-panel outcome +counts, cache-hit percentage, and server-duration p50/p95/p99, including failed +refreshes. Sustained-phase failure contributors are flagged. Structured panel +evidence, including freshness bounds, appears in `trials[].dashboard_panels`; +server execution and admission metrics appear in `samples.jsonl` under +`panel_metrics` when the server exposes them. See the +[panel evidence definitions](docs/metrics.md#dashboard-panel-evidence). + ## Output Every run produces one self-contained directory: @@ -136,7 +170,7 @@ results/20261014T190000Z/ └── fanout.log setup-token-scrubbed server log ``` -`samples.jsonl` includes cumulative `ingest_rows_total` by signal (`spans`, `logs`, `metrics`), `rollup_seconds` by rollup (`service`, `edge`, `version`, `read_cache`), `write_gate_hold_seconds` and `write_gate_wait_seconds` by operation, `compaction_seconds_total`, `compactions_total`, `publish_wait_seconds_total`, and `ingest_backpressure_seconds_total`, plus the current `compaction_backlog_batches` gauge. Derive per-second accepted-row rates from consecutive samples' `ingest_rows_total` deltas divided by their `at_seconds` difference. Missing metrics from older Fanout releases leave these fields absent or zero. See [sample definitions](docs/metrics.md#background-work-samples) for their sources and interpretation. +`samples.jsonl` includes cumulative `ingest_rows_total` by signal (`spans`, `logs`, `metrics`), `rollup_seconds` by rollup (`service`, `edge`, `version`, `read_cache`), `write_gate_hold_seconds` and `write_gate_wait_seconds` by operation, `compaction_seconds_total`, `compactions_total`, `publish_wait_seconds_total`, `ingest_backpressure_seconds_total`, `ingest_throttled_total` by reason, `ingest_admission_rows_per_second`, `compaction_retired_batches_total`, and `compaction_pass_seconds_total`, plus the current `compaction_backlog_batches` gauge. Derive per-second accepted-row rates from consecutive samples' `ingest_rows_total` deltas divided by their `at_seconds` difference. Missing metrics leave these fields absent or zero. See [sample definitions](docs/metrics.md#background-work-samples) for their sources and interpretation. The terminal output is optimized for decisions: @@ -212,6 +246,8 @@ Presets make comparisons easy. Overrides are intended for diagnostics and ceilin 3 signals × processes-per-signal × workers-per-process × rate-per-worker ``` +Retryable throttling is valid when telemetrygen retries and exits successfully. Accepted throughput still comes only from Fanout counters. Any nonzero generator exit, including exhaustion of its export retry timeout, invalidates the phase and fails the generator-processes gate; throttling never excuses exporter death. + Avoid unbounded telemetrygen runs. Its SDK queues may drop locally while the console continues to count generated data. Fanout Bench always uses bounded rates and treats Fanout's accepted counter as authoritative. ## Development diff --git a/cmd/fanout-bench/main.go b/cmd/fanout-bench/main.go index e33e977..f144c28 100644 --- a/cmd/fanout-bench/main.go +++ b/cmd/fanout-bench/main.go @@ -2,6 +2,7 @@ package main import ( "context" + "errors" "flag" "fmt" "os" @@ -72,10 +73,12 @@ func commonFlags(name string, args []string) (bench.Config, error) { fs.StringVar(&cfg.GeneratorAddress, "generator-address", "", "generator host IP reachable from the Fanout host") fs.StringVar(&cfg.FanoutListen, "fanout-listen", "", "Fanout host IP reachable from the generator host") fs.StringVar(&cfg.GeneratorHostDescription, "generator-host-description", "", "generator host description recorded in the report") + fs.StringVar(&cfg.FanoutBinary, "fanout-binary", "", "local Linux Fanout binary for a remote run instead of a release; --fanout-version is then a non-release label and the report is marked not publishable") } fs.IntVar(&cfg.SSHPort, "ssh-port", 22, "SSH port") fs.BoolVar(&cfg.KeepRemote, "keep-remote", false, "preserve the isolated remote run directory after downloading artifacts") - fs.StringVar(&cfg.Preset, "preset", "standard", "suite: quick, standard, or publish") + fs.StringVar(&cfg.SignalSelection, "signals", "all", "sweeps: all or mixed (skips single-signal sweeps)") + fs.StringVar(&cfg.Preset, "preset", "standard", "suite: quick, standard, local, validate, or publish") fs.StringVar(&cfg.Endpoint, "endpoint", "localhost:7520", "Fanout OTLP/gRPC endpoint") fs.StringVar(&cfg.MetricsURL, "metrics-url", "http://localhost:7520/metrics", "Fanout Prometheus metrics URL (required evidence source)") fs.BoolVar(&cfg.Insecure, "insecure", false, "use plaintext OTLP/gRPC") @@ -104,9 +107,15 @@ func commonFlags(name string, args []string) (bench.Config, error) { if err := fs.Parse(args); err != nil { return cfg, err } + if err := bench.ValidateSignalSelection(cfg.SignalSelection); err != nil { + return cfg, err + } if err := bench.ValidateGeneratorHosts(cfg); err != nil { return cfg, err } + if cfg.FanoutBinary != "" && cfg.Host == "" { + return cfg, errors.New("--fanout-binary requires a remote run with --host") + } cfg.Token = os.Getenv(tokenEnv) cfg.MetricsToken = os.Getenv(metricsTokenEnv) cfg.QueryCookie = os.Getenv(queryCookieEnv) @@ -162,6 +171,7 @@ func managed(ctx context.Context, args []string) error { fs.StringVar(&cfg.OutputDir, "output", "./results", "remote evidence directory") fs.StringVar(&cfg.FanoutVersion, "fanout-version", "unknown", "Fanout version") fs.StringVar(&cfg.FanoutArtifact, "fanout-artifact", "", "verified release artifact identity") + fs.StringVar(&cfg.SignalSelection, "signals", "all", "sweeps: all or mixed (skips single-signal sweeps)") fs.StringVar(&cfg.Preset, "preset", "standard", "suite preset") fs.IntVar(&cfg.Repeat, "repeat", 1, "sweep repetitions") fs.IntVar(&cfg.BatchSize, "batch-size", 1000, "OTLP batch size") diff --git a/cmd/fanout-bench/main_test.go b/cmd/fanout-bench/main_test.go index ede4b02..c3e2fe7 100644 --- a/cmd/fanout-bench/main_test.go +++ b/cmd/fanout-bench/main_test.go @@ -28,7 +28,7 @@ func TestSplitReproductionCommandPassesValidationAfterSubstitution(t *testing.T) cfg, err := commonFlags("run", []string{ "--host", "root@192.0.2.10", "--generator-host", "root@192.0.2.11", "--generator-address", "10.77.0.3", "--fanout-listen", "10.77.0.2", - "--ssh-port", "2222", "--telemetrygen", executable, "--preset", "quick", + "--ssh-port", "2222", "--telemetrygen", executable, "--preset", "validate", "--signals", "mixed", }) if err != nil { t.Fatal(err) @@ -41,6 +41,9 @@ func TestSplitReproductionCommandPassesValidationAfterSubstitution(t *testing.T) if err == nil || report == nil { t.Fatalf("expected partial report on cancellation, got report=%v err=%v", report, err) } + if !strings.Contains(report.Settings.Methodology, "were skipped") || !report.Settings.EarlyStop { + t.Fatal("missing validation coverage") + } command := strings.NewReplacer( "", "192.0.2.20", "", "192.0.2.21", "", "10.77.0.13", "", "10.77.0.12", @@ -55,6 +58,9 @@ func TestSplitReproductionCommandPassesValidationAfterSubstitution(t *testing.T) if err != nil { t.Fatalf("substituted reproduction failed validation: %v\n%s", err, command) } + if reproduced.SignalSelection != "mixed" || reproduced.Preset != "validate" { + t.Fatal("validation flags lost") + } if reproduced.Host != "root@192.0.2.20" || reproduced.GeneratorHost != "root@192.0.2.21" || reproduced.GeneratorAddress != "10.77.0.13" || reproduced.FanoutListen != "10.77.0.12" || reproduced.SSHPort != 2222 { t.Fatalf("reproduction lost topology: %+v", reproduced) @@ -109,3 +115,33 @@ func TestSplitFlagsRequiredTogetherAndRemoteOnly(t *testing.T) { t.Fatalf("generator description=%q, err=%v", cfg.GeneratorHostDescription, err) } } + +func TestFanoutBinaryRequiresRemoteHost(t *testing.T) { + if _, err := commonFlags("run", []string{"--fanout-binary", "/tmp/fanout", "--fanout-version", "explore-test"}); err == nil { + t.Fatal("accepted --fanout-binary without --host") + } + cfg, err := commonFlags("run", []string{"--host", "root@192.0.2.10", "--fanout-binary", "/tmp/fanout", "--fanout-version", "explore-test"}) + if err != nil { + t.Fatal(err) + } + if cfg.FanoutBinary != "/tmp/fanout" { + t.Fatalf("fanout binary=%q", cfg.FanoutBinary) + } +} + +func TestValidationFlags(t *testing.T) { + for _, args := range [][]string{{"--signals", "mixed", "--preset", "validate"}, {"--signals=mixed", "--preset=publish"}} { + cfg, err := commonFlags("run", args) + if err != nil { + t.Fatal(err) + } + if cfg.SignalSelection != "mixed" { + t.Fatal(cfg.SignalSelection) + } + } + for _, args := range [][]string{{"--signals"}, {"--signals", "logs"}, {"--signals", "mixed,traces"}} { + if _, err := commonFlags("run", args); err == nil { + t.Fatalf("accepted %v", args) + } + } +} diff --git a/docs/metrics.md b/docs/metrics.md index ef0f8cf..327fd84 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -48,7 +48,7 @@ Each `samples.jsonl` row also records the following `/metrics` values. The same | `ingest_backpressure_seconds_total` | `fanout_ingest_backpressure_seconds_total` | | `compaction_backlog_batches` | `fanout_compaction_backlog_batches` | -Durations and counts are cumulative values at scrape time, not phase deltas. Subtract adjacent samples to attribute work during an interval, accounting for process restarts. For per-second accepted-row rates, divide each signal's `ingest_rows_total` delta by the difference in `at_seconds` between consecutive samples. `compaction_backlog_batches` is a current gauge rather than a cumulative counter. Series with additional labels are summed, retaining the indicated map key for ingest signals, rollups, and write gates. Older Fanout releases may omit these metrics; absent maps and zero scalar values are omitted from JSON, and missing metrics do not fail a run. These samples provide attribution evidence without changing headline tables or quality gates. +Durations and counts are cumulative values at scrape time, not phase deltas. Subtract adjacent samples to attribute work during an interval, accounting for process restarts. For per-second accepted-row rates, divide each signal's `ingest_rows_total` delta by the difference in `at_seconds` between consecutive samples. `compaction_backlog_batches` is a current gauge rather than a cumulative counter. Series with additional labels are summed, retaining the indicated map key for ingest signals, rollups, and write gates. Absent maps and zero scalar values are omitted from JSON, and missing metrics do not fail a run. These samples provide attribution evidence without changing headline tables or quality gates. ## Generator resources @@ -68,9 +68,9 @@ Same-host runs keep the existing fallback: at least 95% is `target-sustained`, a ## Read workload -Authenticated reads rotate equally across user operations `dashboard-refresh`, `drill-traces` (System overview service-map selection), and `trace-detail`, at 15m, 1h, 6h, 12h, and 24h. A refresh follows `page.tsx`, `use-panel-results.ts`, and `refresh.ts`: one sorted batch of all initially visible panel IDs with the full dashboard spec, range and refresh setting, service All, comparison per spec, and layout widths rounded to 100 pixels for a fixed 1664-pixel viewport. It then sends `POST /api/annotations` with the union of the returned time-panel bounds, when annotations are enabled and those bounds are present. +Authenticated reads rotate equally across user operations `dashboard-refresh`, `drill-traces` (System overview service-map selection), and `trace-detail`, at 15m, 1h, 6h, 12h, and 24h. A refresh follows `page.tsx`, `use-panel-results.ts`, and `refresh.ts`: one sorted batch via `POST /api/queries/run` of all initially visible panel IDs with the full dashboard spec, range and refresh setting, service All, comparison per spec, and layout widths rounded to 100 pixels for a fixed 1664-pixel viewport. It then sends `POST /api/annotations/query` with the union of the returned time-panel bounds, when annotations are enabled and those bounds are present. -Drill-down follows `drill-state.ts` and `drill.tsx`: select the first service-map node, then `POST /api/panels/exemplars` with the original dashboard, panel ID, kind, absolute panel window (`refresh: off`), full-window selection, service dimension, and variables. Trace detail is a separate UI operation using `GET /api/traces/:id` with the discovered namespace, original absolute window, and limit 200. References and selections are retained separately per range. Without a discovered trace, detail slots perform drill-down; without a cached selection, the slot performs dashboard refresh instead. A later slot can select the returned node, so each tick remains one user operation. Fallbacks are reported under the operation actually performed. The current System overview defines no log drill or standalone log/trace search, so synthetic search panels are omitted. +Drill-down follows `drill-state.ts` and `drill.tsx`: select the first service-map node, then `POST /api/queries/exemplars` with the original dashboard, panel ID, kind, absolute panel window (`refresh: off`), full-window selection, service dimension, and variables. Trace detail is a separate UI operation using `GET /api/traces/:id` with the discovered namespace, original absolute window, and limit 200. References and selections are retained separately per range. Without a discovered trace, detail slots perform drill-down; without a cached selection, the slot performs dashboard refresh instead. A later slot can select the returned node, so each tick remains one user operation. Fallbacks are reported under the operation actually performed. The current System overview defines no log drill or standalone log/trace search, so synthetic search panels are omitted. One scheduled read is one operation. Its latency runs until every request completes; any failed request, panel error, missing panel result, or empty discovered trace detail fails the operation. Responses remain bounded at 32 MiB; responses exceeding this cap fail because their contents cannot be validated. The existing methodology string records the fetched dashboard spec SHA-256 and per-phase operation kinds, dispatched/completed counts, failures, cancellations, p95, and p99. Aggregate latency fields retain their schema. @@ -87,6 +87,58 @@ When read load is enabled, the sustained mixed phase must also complete at least The read scheduler is open-loop and bounded. `query_attempts` counts scheduled operations, `query_latency_ms.count` counts valid completed operations, `query_shed` counts ticks rejected because the concurrency bound was full, and `query_canceled_at_phase_end` counts in-flight operations canceled when load stopped. +### Dashboard panel evidence + +Each trial's `dashboard_panels` records per-panel counts for `ok`, `empty`, +`busy`, `error`, `timeout`, and `missing`, including results from failed +refreshes. Both JSON batches and NDJSON streams are supported. An error with +`reason: busy` counts as busy; Fanout's public execution-deadline message +counts as timeout. An explicit HTTP busy rejection or HTTP 504 applies that +outcome to every requested non-text panel. Other unavailable results count as +missing. Complete results in an interrupted stream are retained, and absent +panels count as missing. Duplicate result IDs count once as an error. Text +panels omitted by the server are exempt. + +`elapsed_ms` contains count, mean, maximum, and p50/p95/p99 of the server's +reported duration, including errors and cache hits. Missing durations are +excluded rather than replaced with client latency. All read latency +histograms have 2,048 fixed logarithmic buckets; quantiles use bucket upper +bounds with approximately 1% plus 0.01ms precision (durations above the final +bucket use the observed maximum). Dashboard discovery permits at most 128 +panel IDs of at most 128 bytes each; additional response IDs share a bounded +overflow entry. There is no retained list of individual refreshes. + +`cached` counts returned results with `cached: true`; `cached_percent` divides +by `returned`, excluding missing results and whole-request rejections. When +`fresh_to_ms` is present, `fresh_to_count`, `fresh_to_min_ms`, and +`fresh_to_max_ms` preserve its presence and observed range, including a value +of zero. `failed_refreshes` counts this panel's unsuccessful outcomes, +excluding client-canceled requests. Several panels can cause the same failed +refresh. The per-phase **Dashboard panels** tables flag these panels in the +sustained phase. Annotation and other request failures can also fail a refresh +whose panels all succeeded. + +### Server panel samples + +`samples.jsonl` and `trials[].samples` include `panel_metrics` when available, +keyed by exact metric name, then `panel`, then `status`. Fanout's current +names in `internal/metrics/panels.go` are `fanout_panel_*`, rather than +`fanout_query_panel_*`: + +| Metric | Meaning | +|---|---| +| `fanout_panel_query_duration_seconds_count` | Executed outcomes (`ok`, `empty`, `error`, `other`) | +| `fanout_panel_query_duration_seconds_sum` | Execution seconds by panel and outcome, excluding cache hits and admission wait | +| `fanout_panel_admission_duration_seconds_count` | Admission counts (`admitted`, `rejected`, `canceled`, `timeout`, `other`) | +| `fanout_panel_admission_duration_seconds_sum` | Admission wait seconds by panel and outcome | + +These are cumulative histogram counts and sums. Additional labels are summed; +panel and status labels are retained. Subtract adjacent samples to measure +outcomes and wait during an interval. Admission count series supply the +rejection, cancellation, and timeout counters. A sample without these series +omits `panel_metrics`; present zero counts are retained. These server +metrics exclude result-cache hits, while client panel evidence includes them. + Saturation is not an evidence failure. A valid phase can fail to sustain its offered target, which is the point of a capacity sweep. ## Interpretation limits @@ -97,3 +149,18 @@ Saturation is not an evidence failure. A valid phase can fail to sustain its off - A short peak is not a sustained capacity claim. Publish the long mixed proof alongside signal-specific sweeps. - A stress peak that misses its read target is not serviceable mixed capacity. Use the sustained topology and its read-load gate for that claim. - No benchmark number transfers automatically across CPU models, storage devices, filesystems, retention settings, or Fanout versions. + +### Saturation admission samples + +`ingest_throttled_total` records `fanout_ingest_throttled_total` by reason +(`debt`, `reads`, `budget`). `ingest_admission_rows_per_second` records the +current cap; zero means uncapped. `compaction_retired_batches_total` records +`fanout_compaction_retired_batches_total`; use deltas to measure retirement +throughput. `compaction_pass_seconds_total` records the histogram sum from +`fanout_compaction_pass_seconds_sum`. + +With saturation admission enabled, `ingest_backpressure_seconds_total` measures +seconds with an active cap rather than seconds blocked in the writer. Values +across this semantic change are not directly comparable. Failed generator +quality gates include the first redacted FATAL line from the trial notes when +available, so exporter deadline failures are visible in the summary. diff --git a/docs/publishing.md b/docs/publishing.md index cee491d..90134b9 100644 --- a/docs/publishing.md +++ b/docs/publishing.md @@ -68,3 +68,35 @@ The local provisioner detects both Linux platforms, builds the harness for Fanou A plain `fanout-bench run --host root@` without a generator host remains a same-host run on any clean Linux host. It is useful for diagnostics, but generator CPU contention limits what it says about Fanout’s own ingest ceiling. Before uploading to the ccx33, the local provisioner verifies `fanout_v2026.10.1-alpha_linux_amd64.tar.gz` against the release `SHA256SUMS`. Managed scrapes use private `/metrics` bearer authentication. Read load covers System overview dashboard refresh (batched panels followed by annotations), service-map exemplar drill-down, and trace detail over 15-minute through 24-hour windows. Publish the per-operation methodology, counts and latency, and dashboard spec hash alongside the requested and completed operation rates. + +## Faster validation runs + +Validation keeps the accepted-counter measurements, generator checks, read-completion gate, storage repair verification, warmup, and five-minute mixed sustained proof. It is intended for release/engineering validation; publication still requires the full `publish` sweeps and repetitions above. + +Use three independent VM pairs to overlap ingest, reads near 70% load, and reads at saturation: + +```sh +export FANOUT_BENCH_HCLOUD_SSH_KEY='benchmark-key' +scripts/hetzner.sh --parallel-groups \ + --fanout-version v2026.10.1-alpha \ + --preset validate --repeat 3 \ + --output results/validate-ingest \ + -- \ + --fanout-version v2026.10.1-alpha \ + --preset validate --repeat 1 --signals mixed --query-rate 20 --rate 210 \ + --output results/validate-reads-70 \ + -- \ + --fanout-version v2026.10.1-alpha \ + --preset validate --repeat 1 --signals mixed --query-rate 20 \ + --output results/validate-reads-saturated +``` + +Replace the release and output directories for each run. `--rate 210` is per worker: it approximates 70% for the previously measured target and must be recalibrated when capacity changes; it is not an automatic 70% controller. `--signals mixed` keeps the mixed warmup, mixed sweep and mixed sustained phase, skipping the duplicate traces/logs/metrics sweeps in both read groups. Reports label those single-signal peaks as skipped. + +`validate` has the same maximum process counts (1 through 64), durations, workers, rates and batch sizes as `publish`. Within each signal and repetition it skips remaining sweep points after two consecutive accepted-throughput gains strictly below 5%, or an acceptance ratio below 50% and below half the preceding point. Comparisons require valid points; warmup, other signals and other repetitions cannot trigger a stop. The next repetition starts at the first point again. JSON `sweep_stops` and Markdown record the last measured phase, reason and exact skipped phases. Sustained topology selection still uses the best measured mixed point satisfying the read target, with the existing failing fallback when none qualifies. Early stopping can miss a later throughput recovery, so validation evidence does not replace a full ceiling sweep for publication. `publish` never stops early. + +Budget **about two hours wall time**, down from about four, using the observed 1.5-hour ingest group plus roughly 30 minutes of setup, verification and download as the longest path. Early stopping may shorten that further; it is not a guaranteed duration. Mixed-only read groups should finish sooner and their pairs are deleted as soon as each group finishes. Six VMs run at peak concurrency; ensure account quota and local build/network capacity allow that. + +For cost planning, if ingest occupies its pair for two started hours and each mixed-only read group stays within one started hour, the run bills **8 ccx33 VM-hours**: `2 × (2 + 1 + 1)`. That is about the VM cost of the previous four-hour run on one pair. If both read groups cross into a second hour, budget 12 VM-hours instead. Multiply by the selected location's current ccx33 hourly rate, then add applicable public IPv4 charges and tax. These are estimates based on the observed timings, not live price quotes; check current rates before provisioning. + +`--parallel-groups` must be the first script argument. The parent builds once and launches each group on its own isolated pair and network, printing the group number and all resource names. Each child retains the sequential script's cleanup and deletion retries; the parent waits for every child, including cleanup, and returns nonzero for any run/provisioning/deletion failure. Ctrl-C or TERM signals all active children and waits for their cleanup, including interrupted creates. Failed deletion names resources requiring manual removal. Use distinct `--output` paths; omitted outputs receive unique per-group directories under `results/parallel-*`. Without `--parallel-groups`, groups continue sharing one pair sequentially. diff --git a/internal/bench/hetzner_test.go b/internal/bench/hetzner_test.go new file mode 100644 index 0000000..f8dea6e --- /dev/null +++ b/internal/bench/hetzner_test.go @@ -0,0 +1,171 @@ +package bench + +import ( + "bytes" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "syscall" + "testing" + "time" +) + +// Exercise the real shell entry point in an isolated repo with fake cloud/SSH +// commands. No cloud credentials, shared binaries or actual resources are used. +func TestHetznerParallelLifecycle(t *testing.T) { + for _, mode := range []string{"success", "failure", "create-failure", "delete-failure", "interrupt", "interrupt-create"} { + t.Run(mode, func(t *testing.T) { + dir := t.TempDir() + for _, sub := range []string{"scripts", "bin", "fake", "resources"} { + if err := os.Mkdir(filepath.Join(dir, sub), 0700); err != nil { + t.Fatal(err) + } + } + script, err := os.ReadFile("../../scripts/hetzner.sh") + if err != nil { + t.Fatal(err) + } + write := func(path, body string) { + t.Helper() + if err := os.WriteFile(filepath.Join(dir, path), []byte(body), 0700); err != nil { + t.Fatal(err) + } + } + write("scripts/hetzner.sh", string(script)) + write("fake/hcloud", `#!/usr/bin/env bash +set -eu +printf '%s\n' "$*" >> "$MOCK_ROOT/events" +kind=$1; action=$2 +case "$action" in + create) + shift 2 + while [[ $1 != --name ]]; do shift; done + name=$2 + touch "$MOCK_ROOT/resources/$name" + if [[ $MOCK_MODE == interrupt-create && $kind == server ]]; then /bin/sleep 0.5; fi + if [[ $MOCK_MODE == create-failure && $kind == server ]]; then exit 1; fi + ;; + delete) + if [[ $MOCK_MODE == delete-failure && $kind == network ]]; then exit 1; fi + rm -f "$MOCK_ROOT/resources/$3" + ;; + describe) if [[ $kind == location ]]; then echo eu-central; else echo 10.77.0.2; fi ;; + ip) echo 192.0.2.1 ;; + list) for f in "$MOCK_ROOT"/resources/*; do [[ -e $f ]] && basename "$f"; done; exit 0 ;; +esac +`) + write("fake/just", "#!/bin/sh\necho build >> \"$MOCK_ROOT/builds\"\n") + write("fake/ssh", "#!/bin/sh\nexit 0\n") + write("fake/sleep", "#!/bin/sh\nexit 0\n") + write("bin/fanout-bench", `#!/usr/bin/env bash +printf '%s\n' "$*" >> "$MOCK_ROOT/runs" +if [[ $MOCK_MODE == interrupt ]]; then exec /bin/sleep 60; fi +if [[ $MOCK_MODE == success ]]; then + # Both runs must start before either finishes: sequential execution hangs. + for ((attempt=0; attempt<500; attempt++)); do + if [[ $(wc -l < "$MOCK_ROOT/runs") -eq 2 ]]; then exit 0; fi + /bin/sleep 0.01 + done + exit 9 +fi +if [[ $MOCK_MODE == failure && $* == *fail* ]]; then exit 7; fi +`) + cmd := exec.Command("bash", filepath.Join(dir, "scripts/hetzner.sh"), "--parallel-groups", "--output", "pass", "--", "--output", "fail", "--signals", "mixed") + cmd.Env = append(os.Environ(), "PATH="+filepath.Join(dir, "fake")+":"+os.Getenv("PATH"), "FANOUT_BENCH_HCLOUD_SSH_KEY=test", "FANOUT_BENCH_GROUP_CHILD=", "MOCK_ROOT="+dir, "MOCK_MODE="+mode) + cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} + var output bytes.Buffer + cmd.Stdout = &output + cmd.Stderr = &output + if err := cmd.Start(); err != nil { + t.Fatal(err) + } + defer syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) + done := make(chan error, 1) + go func() { done <- cmd.Wait() }() + if mode == "interrupt" || mode == "interrupt-create" { + deadline := time.Now().Add(10 * time.Second) + for { + runs, _ := os.ReadFile(filepath.Join(dir, "runs")) + events, _ := os.ReadFile(filepath.Join(dir, "events")) + if (mode == "interrupt" && strings.Count(string(runs), "\n") == 2) || (mode == "interrupt-create" && strings.Count(string(events), "server create") == 2) { + break + } + if time.Now().After(deadline) { + _ = cmd.Process.Kill() + t.Fatal("children never reached benchmarks") + } + time.Sleep(10 * time.Millisecond) + } + if err := cmd.Process.Signal(syscall.SIGINT); err != nil { + t.Fatal(err) + } + } + select { + case err := <-done: + if (err == nil) != (mode == "success") { + t.Fatalf("err=%v output=%s", err, output.String()) + } + case <-time.After(15 * time.Second): + _ = cmd.Process.Kill() + t.Fatal("parallel cleanup hung") + } + resources, _ := os.ReadDir(filepath.Join(dir, "resources")) + want := 0 + if mode == "delete-failure" { + want = 2 + } + if len(resources) != want { + t.Fatalf("remaining=%v output=%s", resources, output.String()) + } + builds, _ := os.ReadFile(filepath.Join(dir, "builds")) + if string(builds) != "build\n" { + t.Fatalf("build count: %q", builds) + } + if strings.Count(output.String(), "Hetzner pair:") != 2 { + t.Fatal(output.String()) + } + if mode == "success" || mode == "failure" { + runs, _ := os.ReadFile(filepath.Join(dir, "runs")) + if strings.Count(string(runs), "\n") != 2 || !strings.Contains(string(runs), "--signals mixed") { + t.Fatal(string(runs)) + } + } + if mode == "delete-failure" && !strings.Contains(output.String(), "delete it manually") { + t.Fatal(output.String()) + } + // Every created resource was offered for deletion, servers before network. + events, _ := os.ReadFile(filepath.Join(dir, "events")) + for _, line := range strings.Split(string(events), "\n") { + fields := strings.Fields(line) + if len(fields) < 2 || fields[1] != "create" { + continue + } + for i, f := range fields { + if f == "--name" { + deletion := fmt.Sprintf("%s delete %s", fields[0], fields[i+1]) + if fields[0] == "server" { + network := strings.TrimSuffix(strings.TrimSuffix(fields[i+1], "-fanout"), "-generator") + if strings.Index(string(events), "network delete "+network) < strings.Index(string(events), deletion) { + t.Fatal("network deleted before server") + } + } + if !strings.Contains(string(events), deletion) { + t.Fatalf("missing %s", deletion) + } + } + } + } + }) + } +} + +func TestHetznerRejectsEmptyParallelGroups(t *testing.T) { + for _, args := range [][]string{{"--parallel-groups"}, {"--parallel-groups", "--", "--preset", "quick"}, {"--parallel-groups", "--preset", "quick", "--"}} { + out, err := exec.Command("bash", append([]string{"../../scripts/hetzner.sh"}, args...)...).CombinedOutput() + if err == nil { + t.Fatalf("accepted %v: %s", args, out) + } + } +} diff --git a/internal/bench/managed.go b/internal/bench/managed.go index 15d3c43..e2c4d5e 100644 --- a/internal/bench/managed.go +++ b/internal/bench/managed.go @@ -38,6 +38,7 @@ type ManagedConfig struct { FanoutVersion string FanoutArtifact string Preset string + SignalSelection string Repeat int BatchSize int QueryRate float64 @@ -246,7 +247,7 @@ func managedRunnerConfig(managed ManagedConfig, httpAddr, dataDir string) Config cfg := Config{ Host: managed.ProvisionedHost, HostDescription: managed.HostDescription, SSHPort: managed.ProvisionedSSHPort, SSHIdentity: managed.ProvisionedSSHIdentity, - Preset: managed.Preset, Repeat: managed.Repeat, BatchSize: managed.BatchSize, + SignalSelection: managed.SignalSelection, Preset: managed.Preset, Repeat: managed.Repeat, BatchSize: managed.BatchSize, Endpoint: httpAddr, MetricsURL: "http://" + httpAddr + "/metrics", QueryURL: "http://" + httpAddr, Insecure: true, Telemetrygen: managed.TelemetrygenPath, OutputDir: managed.OutputDir, FanoutVersion: managed.FanoutVersion, FanoutArtifact: managed.FanoutArtifact, DataDir: dataDir, diff --git a/internal/bench/panel_evidence.go b/internal/bench/panel_evidence.go new file mode 100644 index 0000000..29e662e --- /dev/null +++ b/internal/bench/panel_evidence.go @@ -0,0 +1,264 @@ +package bench + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "sort" + "strings" +) + +const maxPanelIDs = 128 +const otherPanelID = "__other__" + +// DashboardPanel contains bounded, phase-local evidence, including unsuccessful +// refreshes. ElapsedMS is the server duration, not the HTTP operation latency. +type DashboardPanel struct { + ID string `json:"id"` + Statuses map[string]int64 `json:"statuses"` + Returned int64 `json:"returned"` + Cached int64 `json:"cached"` + CachedPercent float64 `json:"cached_percent"` + FailedRefreshes int64 `json:"failed_refreshes"` + ElapsedMS Latency `json:"elapsed_ms"` + FreshToCount int64 `json:"fresh_to_count"` + FreshToMinMS *int64 `json:"fresh_to_min_ms,omitempty"` + FreshToMaxMS *int64 `json:"fresh_to_max_ms,omitempty"` +} + +type panelStats struct { + DashboardPanel + hist *histogram +} + +func panelStatus(r panelResult) string { + // The current server returns status=error, reason=busy for admission + // failures, and presents execution deadlines as a public error message. + if r.Reason == "busy" || r.Status == "busy" { + return "busy" + } + if r.Reason == "timeout" || r.Status == "timeout" || + strings.HasPrefix(r.Error, "The query took longer than ") || + strings.HasPrefix(r.Error, "Panels took longer than ") || + r.Error == "context deadline exceeded" { + return "timeout" + } + if r.Error != "" { + return "error" + } + switch r.Status { + case "ok", "empty", "error", "missing": + return r.Status + default: + return "error" + } +} + +// decodePanelResults reads the API's one-result-per-line stream. A truncated +// stream retains its complete results; the remaining panels are missing, and +// the refresh still fails validation. +func decodePanelResults(body []byte) ([]panelResult, bool) { + decoder := json.NewDecoder(bytes.NewReader(body)) + var results []panelResult + for { + var result panelResult + err := decoder.Decode(&result) + if err == io.EOF { + return results, len(results) > 0 + } + if err != nil || result.Status == "" { + return results, false + } + results = append(results, result) + } +} + +// The panel endpoint opts into the UI's stream protocol. HTTP failures have no server +// elapsed_ms, so they never contribute invented latency observations. +func queryPanelHTTP(ctx context.Context, client *http.Client, base, cookie, path string, body []byte) ([]byte, bool, string) { + req, err := http.NewRequestWithContext(ctx, http.MethodPost, strings.TrimRight(base, "/")+path, bytes.NewReader(body)) + if err != nil { + return nil, false, "missing" + } + req.Header.Set("Cookie", cookie) + req.Header.Set("Fanout-Request", "1") + req.Header.Set("Content-Type", "application/json") + req.Header.Set("Accept", "application/x-ndjson") + resp, err := client.Do(req) + if err != nil { + return nil, false, "missing" + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + status := "missing" + if resp.StatusCode == http.StatusServiceUnavailable { + // Only Fanout's explicit busy code attributes rejection to panels. + var failure struct { + Code string `json:"code"` + } + _ = json.NewDecoder(io.LimitReader(resp.Body, maxQueryBody)).Decode(&failure) + if failure.Code == "busy" { + status = "busy" + } + } else if resp.StatusCode == http.StatusGatewayTimeout { + status = "timeout" + } + return nil, false, status + } + response, err := io.ReadAll(io.LimitReader(resp.Body, maxQueryBody+1)) + if len(response) > maxQueryBody { + return nil, false, "missing" + } + return response, err == nil, "missing" +} + +func (q *queryRun) recordPanels(query queryTarget, results map[string]panelResult, absent string, attributeFailures bool) { + text := map[string]bool{} + expected := map[string]bool{} + for _, id := range query.panels { + expected[id] = true + } + var dashboard struct{ Panels []struct{ ID, Viz string } } + _ = json.Unmarshal(query.dashboard, &dashboard) + for _, p := range dashboard.Panels { + text[p.ID] = p.Viz == "text" + } + q.panelsMu.Lock() + defer q.panelsMu.Unlock() + record := func(id string, result panelResult, returned bool) { + if len(id) > 128 || id == "" || len(expected) > 0 && !expected[id] { + id = otherPanelID + } + stats := q.panels[id] + if stats == nil { + named := len(q.panels) + if q.panels[otherPanelID] != nil { + named-- + } + if named >= maxPanelIDs && id != otherPanelID { + id = otherPanelID + } + stats = q.panels[id] + if stats == nil { + stats = &panelStats{DashboardPanel: DashboardPanel{ID: id, Statuses: map[string]int64{}}, hist: newHistogram()} + q.panels[id] = stats + } + } + stats.Statuses[result.Status]++ + if attributeFailures && result.Status != "ok" && result.Status != "empty" { + stats.FailedRefreshes++ + } + if !returned { + return + } + stats.Returned++ + if result.Cached { + stats.Cached++ + } + if result.ElapsedMS != nil { + stats.hist.recordMS(*result.ElapsedMS) + } + if result.FreshToMS != nil { + v := *result.FreshToMS + stats.FreshToCount++ + if stats.FreshToMinMS == nil || v < *stats.FreshToMinMS { + stats.FreshToMinMS = &v + } + if stats.FreshToMaxMS == nil || v > *stats.FreshToMaxMS { + stats.FreshToMaxMS = &v + } + } + } + for id, result := range results { + record(id, result, true) + } + for _, id := range query.panels { + if _, found := results[id]; !found && !text[id] { + record(id, panelResult{Status: absent}, false) + } + } +} + +func (q *queryRun) panelSummary() []DashboardPanel { + q.panelsMu.Lock() + defer q.panelsMu.Unlock() + var panels []DashboardPanel + for _, stats := range q.panels { + panel := stats.DashboardPanel + panel.Statuses = make(map[string]int64) + for _, status := range []string{"ok", "empty", "busy", "error", "timeout", "missing"} { + panel.Statuses[status] = stats.Statuses[status] + } + panel.ElapsedMS = stats.hist.snapshot() + if panel.Returned > 0 { + panel.CachedPercent = round(100*float64(panel.Cached)/float64(panel.Returned), 2) + } + panels = append(panels, panel) + } + sort.Slice(panels, func(i, j int) bool { return panels[i].ID < panels[j].ID }) + return panels +} + +func renderDashboardPanels(b *strings.Builder, trials []Trial) { + for _, trial := range trials { + if len(trial.DashboardPanels) == 0 { + continue + } + fmt.Fprintf(b, "\n### Dashboard panels — %s\n\nServer elapsed_ms includes returned failures and cache hits; missing durations are excluded. Cached %% uses returned panel results. Quantiles use fixed buckets (~1%% plus 0.01ms precision).\n\n", trial.Spec.Name) + fmt.Fprint(b, "| Panel id | ok | empty | busy | error | timeout | missing | Cached % | p50 | p95 | p99 | Failed refreshes |\n|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n") + for _, p := range trial.DashboardPanels { + id := strings.NewReplacer("|", "\\|", "\n", " ", "\r", " ").Replace(p.ID) + if trial.Spec.Kind == "sustained" && p.FailedRefreshes > 0 { + id += " ⚠" + } + fmt.Fprintf(b, "| %s | %d | %d | %d | %d | %d | %d | %.1f%% | %s | %s | %s | %d |\n", id, p.Statuses["ok"], p.Statuses["empty"], p.Statuses["busy"], p.Statuses["error"], p.Statuses["timeout"], p.Statuses["missing"], p.CachedPercent, panelQuantile(p.ElapsedMS.Count, p.ElapsedMS.P50Ms), panelQuantile(p.ElapsedMS.Count, p.ElapsedMS.P95Ms), panelQuantile(p.ElapsedMS.Count, p.ElapsedMS.P99Ms), p.FailedRefreshes) + } + if trial.Spec.Kind == "sustained" { + fmt.Fprint(b, "\n⚠ Panel caused failed sustained refreshes. Counts can overlap when multiple panels fail in one refresh. Request or annotation failures may also fail a refresh.\n") + } + } +} + +func panelQuantile(count int64, ms float64) string { + if count == 0 { + return "n/a" + } + return fmt.Sprintf("%.1fms", ms) +} + +// Preserve both bounded server labels. Admission histogram counts are the +// admitted/rejected/canceled/timeout counters; sums measure admission wait. +type PanelMetrics map[string]map[string]map[string]float64 + +func panelMetricSamples(s *snapshot) PanelMetrics { + var result PanelMetrics + if s == nil { + return nil + } + for _, v := range s.samples { + switch v.name { + case "fanout_panel_query_duration_seconds_count", "fanout_panel_query_duration_seconds_sum", + "fanout_panel_admission_duration_seconds_count", "fanout_panel_admission_duration_seconds_sum": + default: + continue + } + panel, status := v.labels["panel"], v.labels["status"] + if panel == "" || status == "" { + continue + } + if result == nil { + result = map[string]map[string]map[string]float64{} + } + if result[v.name] == nil { + result[v.name] = map[string]map[string]float64{} + } + if result[v.name][panel] == nil { + result[v.name][panel] = map[string]float64{} + } + result[v.name][panel][status] += v.value + } + return result +} diff --git a/internal/bench/panel_evidence_test.go b/internal/bench/panel_evidence_test.go new file mode 100644 index 0000000..f4bd4d3 --- /dev/null +++ b/internal/bench/panel_evidence_test.go @@ -0,0 +1,297 @@ +package bench + +import ( + "context" + "encoding/json" + "fmt" + "io" + "math" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "reflect" + "strings" + "sync" + "testing" +) + +func outcomeTarget() queryTarget { + return queryTarget{name: "dashboard-refresh", body: []byte(`{}`), panels: []string{"requests", "traffic", "latency", "hosts", "containers", "versions", "endpoints", "service_map", "text"}, dashboard: json.RawMessage(`{"panels":[{"id":"text","viz":"text"}]}`)} +} + +func outcomeFixture(t *testing.T) []byte { + t.Helper() + body, err := os.ReadFile("testdata/panel-outcomes.ndjson") + if err != nil { + t.Fatal(err) + } + return body +} + +func panelsByID(panels []DashboardPanel) map[string]DashboardPanel { + result := map[string]DashboardPanel{} + for _, panel := range panels { + result[panel.ID] = panel + } + return result +} + +func TestPanelOutcomeFixturesThroughRefresh(t *testing.T) { + body := outcomeFixture(t) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/api/queries/run" || r.Header.Get("Accept") != "application/x-ndjson" { + t.Errorf("unexpected request: %s", r.URL.Path) + } + w.Header().Set("Content-Type", "application/x-ndjson") + w.Write(body) + })) + defer server.Close() + q := newQueryRun() + q.targets = []queryTarget{outcomeTarget()} + q.request(context.Background(), server.Client(), server.URL, "session=test", 0) + if q.failures != 1 || q.hist.snapshot().Count != 0 { + t.Fatalf("failed refresh escaped validation: %+v", q) + } + panels := q.panelSummary() + got := panelsByID(panels) + for id, status := range map[string]string{"requests": "ok", "traffic": "empty", "latency": "error", "hosts": "busy", "containers": "timeout", "versions": "busy", "endpoints": "timeout", "service_map": "missing"} { + p := got[id] + if p.Statuses[status] != 1 { + t.Errorf("%s: %+v", id, p) + } + failed := int64(1) + if status == "ok" || status == "empty" { + failed = 0 + } + if p.FailedRefreshes != failed { + t.Errorf("%s failures=%d", id, p.FailedRefreshes) + } + } + if len(panels) != 8 || got["requests"].CachedPercent != 100 || got["requests"].FreshToCount != 1 || *got["requests"].FreshToMinMS != 1000 || got["containers"].ElapsedMS.P95Ms != 10000 || got["traffic"].ElapsedMS.Count != 1 || got["service_map"].ElapsedMS.Count != 0 { + t.Fatalf("incorrect evidence: %+v", panels) + } +} + +func TestPanelAggregationConcurrentAndBounded(t *testing.T) { + q := newQueryRun() + query := queryTarget{panels: []string{"requests"}} + var wg sync.WaitGroup + for i := 0; i < 100; i++ { + wg.Add(1) + go func(i int) { + defer wg.Done() + body := []byte(fmt.Sprintf(`{"id":"requests","status":"ok","elapsed_ms":%d,"cached":%t,"fresh_to_ms":%d}`, i+1, i%2 == 0, i)) + if _, ok := q.panelResponse(body, query); !ok { + t.Error("valid response rejected") + } + }(i) + } + wg.Wait() + p := q.panelSummary()[0] + if p.Statuses["ok"] != 100 || p.Returned != 100 || p.Cached != 50 || p.CachedPercent != 50 || p.FreshToCount != 100 || *p.FreshToMinMS != 0 || *p.FreshToMaxMS != 99 || p.ElapsedMS.MeanMs != 50.5 { + t.Fatalf("aggregate=%+v", p) + } + for _, v := range []struct{ got, want float64 }{{p.ElapsedMS.P50Ms, 50}, {p.ElapsedMS.P95Ms, 95}, {p.ElapsedMS.P99Ms, 99}} { + if math.Abs(v.got-v.want) > v.want*.01+.01 { + t.Errorf("quantile=%f want ~%f", v.got, v.want) + } + } + // A snapshot is detached from future status updates. + q.panelResponse([]byte(`{"id":"requests","status":"error","elapsed_ms":10000}`), query) + if p.Statuses["error"] != 0 { + t.Fatal("snapshot mutated") + } + for i := 0; i < 1000; i++ { + q.panelResponse([]byte(fmt.Sprintf(`{"id":"panel-%d","status":"ok","elapsed_ms":1}`, i)), queryTarget{}) + } + if len(q.panels) > maxPanelIDs+1 || q.panels[otherPanelID] == nil { + t.Fatal("panel ID cardinality is unbounded") + } + if len(q.panels["requests"].hist.buckets) != 2048 { + t.Fatal("histogram is not fixed size") + } +} + +func TestPanelPartialMalformedAndDuplicateResponses(t *testing.T) { + for _, tc := range []struct { + name, body, status string + count int64 + }{ + {"partial", "{\"id\":\"a\",\"status\":\"ok\",\"elapsed_ms\":42}\n{\"id\":", "ok", 1}, + {"missing", ``, "missing", 0}, + {"malformed", `broken`, "missing", 0}, + {"duplicate", `{"id":"a","status":"ok","elapsed_ms":1} +{"id":"a","status":"empty","elapsed_ms":2}`, "error", 1}, + {"unknown-status", `{"id":"a","status":"unexpected"}`, "error", 0}, + {"error-overrides-ok", `{"id":"a","status":"ok","error":"failed"}`, "error", 0}, + } { + t.Run(tc.name, func(t *testing.T) { + q := newQueryRun() + if _, ok := q.panelResponse([]byte(tc.body), queryTarget{panels: []string{"a", "b"}}); ok { + t.Fatal("invalid response accepted") + } + got := panelsByID(q.panelSummary()) + if got["a"].Statuses[tc.status] != 1 || got["a"].ElapsedMS.Count != tc.count || got["b"].Statuses["missing"] != 1 { + t.Fatalf("got=%+v", got) + } + }) + } +} + +func TestWholeBatchRejectionAndDeadlineEvidence(t *testing.T) { + for _, tc := range []struct { + code int + body, status string + }{ + {503, `{"code":"busy","message":"server busy"}`, "busy"}, + {503, `{"code":"unavailable"}`, "missing"}, + {504, `{"message":"deadline exceeded"}`, "timeout"}, + {500, `{"message":"failed"}`, "missing"}, + } { + t.Run(fmt.Sprint(tc.code, tc.status), func(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(tc.code); fmt.Fprint(w, tc.body) })) + defer server.Close() + q := newQueryRun() + q.targets = []queryTarget{outcomeTarget()} + q.request(context.Background(), server.Client(), server.URL, "session=test", 0) + if q.failures != 1 { + t.Fatal("HTTP failure counted as success") + } + for _, p := range q.panelSummary() { + if p.Statuses[tc.status] != 1 || p.ElapsedMS.Count != 0 || p.Returned != 0 { + t.Fatalf("got=%+v", p) + } + } + }) + } +} + +func TestCanceledRefreshDoesNotBlameMissingPanels(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + cancel() + q := newQueryRun() + q.targets = []queryTarget{outcomeTarget()} + q.request(ctx, http.DefaultClient, "http://localhost:1", "session=test", 0) + if q.failures != 0 || q.canceled != 1 { + t.Fatalf("failures=%d canceled=%d", q.failures, q.canceled) + } + for _, p := range q.panelSummary() { + if p.Statuses["missing"] != 1 || p.FailedRefreshes != 0 { + t.Fatalf("cancellation blamed server: %+v", p) + } + } +} + +type panelRoundTripper func(*http.Request) (*http.Response, error) + +func (f panelRoundTripper) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) } + +type panelReadError struct{} + +func (panelReadError) Read([]byte) (int, error) { return 0, io.ErrUnexpectedEOF } + +func TestPanelReadFailureRetainsOutcomesButFailsOperation(t *testing.T) { + // A read error after complete streamed results is still an HTTP failure, + // even if the following annotation request succeeds. + client := &http.Client{Transport: panelRoundTripper(func(r *http.Request) (*http.Response, error) { + var body io.Reader = strings.NewReader(`{}`) + if r.URL.Path == "/api/queries/run" { + body = io.MultiReader(strings.NewReader(`{"id":"a","status":"ok","elapsed_ms":42,"from_ms":1000,"to_ms":2000}`), panelReadError{}) + } + return &http.Response{StatusCode: 200, Body: io.NopCloser(body), Header: http.Header{}}, nil + })} + q := newQueryRun() + q.targets = []queryTarget{{name: "dashboard-refresh", panels: []string{"a"}, annotations: true, timePanels: map[string]bool{"a": true}}} + q.request(context.Background(), client, "http://fanout.test", "session=test", 0) + p := q.panelSummary()[0] + if q.failures != 1 || p.Statuses["ok"] != 1 || p.ElapsedMS.P95Ms != 42 || p.FailedRefreshes != 0 { + t.Fatalf("read error evidence lost: failures=%d panel=%+v", q.failures, p) + } +} + +func TestPanelDiscoveryBounds(t *testing.T) { + for _, ids := range [][]string{{"duplicate", "duplicate"}, {strings.Repeat("x", 129)}, {otherPanelID}, make([]string, maxPanelIDs+1)} { + panels := []map[string]string{} + for _, id := range ids { + panels = append(panels, map[string]string{"id": id, "viz": "stat"}) + } + spec, _ := json.Marshal(map[string]any{"name": "System overview", "panels": panels}) + if _, _, err := systemQueryTargets(spec); err == nil { + t.Fatalf("unbounded/invalid discovery accepted: %v", ids) + } + } +} + +func TestPanelReportAndServerMetricArtifacts(t *testing.T) { + metrics := `fanout_panel_query_duration_seconds_count{panel="latency",status="error",instance="a"} 2 +fanout_panel_query_duration_seconds_count{panel="latency",status="error",instance="b"} 3 +fanout_panel_query_duration_seconds_sum{panel="latency",status="error"} 50 +fanout_panel_admission_duration_seconds_sum{panel="latency",status="admitted"} 1.5 +fanout_panel_admission_duration_seconds_count{panel="latency",status="admitted"} 7 +fanout_panel_admission_duration_seconds_count{panel="hosts",status="rejected"} 4 +fanout_panel_admission_duration_seconds_count{panel="hosts",status="canceled"} 1 +fanout_panel_admission_duration_seconds_count{panel="hosts",status="timeout"} 2 +fanout_panel_admission_duration_seconds_count{panel="hosts",status="admitted"} 0 +fanout_panel_query_duration_seconds_bucket{panel="latency",status="error",le="10"} 5 +` + snap, err := parsePrometheus(strings.NewReader(metrics)) + if err != nil { + t.Fatal(err) + } + sample := resourceSample(snap, 1) + want := PanelMetrics{ + "fanout_panel_query_duration_seconds_count": {"latency": {"error": 5}}, + "fanout_panel_query_duration_seconds_sum": {"latency": {"error": 50}}, + "fanout_panel_admission_duration_seconds_sum": {"latency": {"admitted": 1.5}}, + "fanout_panel_admission_duration_seconds_count": {"latency": {"admitted": 7}, "hosts": {"rejected": 4, "canceled": 1, "timeout": 2, "admitted": 0}}, + } + if !reflect.DeepEqual(sample.PanelMetrics, want) { + t.Fatalf("samples=%+v", sample.PanelMetrics) + } + data, _ := json.Marshal(resourceSample(&snapshot{}, 0)) + if strings.Contains(string(data), "panel_metrics") { + t.Fatal("a sample without panel metrics must omit them") + } + q := newQueryRun() + q.panelResponse(outcomeFixture(t), outcomeTarget()) + report := fixtureReport() + report.Trials = []Trial{{Spec: PhaseSpec{Name: "mixed-sustained", Kind: "sustained"}, DashboardPanels: q.panelSummary(), Samples: []ResourceSample{sample}}, {Spec: PhaseSpec{Name: "mixed-sweep", Kind: "sweep"}, DashboardPanels: q.panelSummary()}} + dir := t.TempDir() + if err := WriteArtifacts(dir, report); err != nil { + t.Fatal(err) + } + restored, err := LoadReport(filepath.Join(dir, "report.json")) + if err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(report.Trials, restored.Trials) { + t.Fatal("panel evidence lost in JSON roundtrip") + } + md, err := os.ReadFile(filepath.Join(dir, "report.md")) + if err != nil { + t.Fatal(err) + } + for _, fragment := range []string{"Dashboard panels — mixed-sustained", "Dashboard panels — mixed-sweep", "| containers ⚠ |", "| service_map ⚠ |", "| requests | 1 | 0 |", "Cached %", "Failed refreshes", "n/a"} { + if !strings.Contains(string(md), fragment) { + t.Errorf("Markdown missing %q", fragment) + } + } + if strings.Contains(string(md), "| requests ⚠ |") { + t.Error("healthy panel flagged as failure") + } + raw, err := os.ReadFile(filepath.Join(dir, "samples.jsonl")) + if err != nil { + t.Fatal(err) + } + var row struct { + Trial string + ResourceSample + } + if err := json.Unmarshal(raw, &row); err != nil { + t.Fatal(err) + } + if row.Trial != "mixed-sustained" || !reflect.DeepEqual(row.PanelMetrics, want) { + t.Fatalf("JSONL evidence=%+v", row) + } +} diff --git a/internal/bench/preset.go b/internal/bench/preset.go index 6e26c11..8fc8a77 100644 --- a/internal/bench/preset.go +++ b/internal/bench/preset.go @@ -5,7 +5,17 @@ import ( "time" ) +func ValidateSignalSelection(selection string) error { + if selection != "" && selection != "all" && selection != "mixed" { + return fmt.Errorf("unknown signals %q (want all or mixed)", selection) + } + return nil +} + func Plan(cfg Config) ([]PhaseSpec, error) { + if err := ValidateSignalSelection(cfg.SignalSelection); err != nil { + return nil, err + } if cfg.Repeat <= 0 { return nil, fmt.Errorf("repeat must be positive") } @@ -26,10 +36,14 @@ func Plan(cfg Config) ([]PhaseSpec, error) { s = shape{[]int{1}, 2, 1_000, 5 * time.Second, 10 * time.Second, 2 * time.Second} case "standard": s = shape{[]int{1, 2, 4, 8}, 8, 2_500, 15 * time.Second, 60 * time.Second, 10 * time.Second} - case "publish": + case "publish", "validate": s = shape{[]int{1, 2, 4, 8, 16, 32, 64}, 8, 2_500, 30 * time.Second, 5 * time.Minute, 15 * time.Second} + case "local": + // The publish shape cut to the upper sweep points, so one repetition of + // a same-host A/B comparison finishes well inside 25 minutes. + s = shape{[]int{8, 16, 32}, 8, 2_500, 30 * time.Second, 5 * time.Minute, 15 * time.Second} default: - return nil, fmt.Errorf("unknown preset %q (want quick, standard, or publish)", cfg.Preset) + return nil, fmt.Errorf("unknown preset %q (want quick, standard, local, validate, or publish)", cfg.Preset) } if cfg.ProcessOverride > 0 { s.processes = []int{cfg.ProcessOverride} @@ -55,6 +69,9 @@ func Plan(cfg Config) ([]PhaseSpec, error) { phases := []PhaseSpec{makePhase("warmup", "warmup", Mixed, s.warmup, 1, 1)} for repetition := 1; repetition <= cfg.Repeat; repetition++ { for _, signal := range Signals { + if cfg.SignalSelection == "mixed" { + continue + } for _, processes := range s.processes { name := fmt.Sprintf("%s-p%d-r%d", signal, processes, repetition) phases = append(phases, makePhase(name, "sweep", signal, s.trial, processes, repetition)) diff --git a/internal/bench/preset_test.go b/internal/bench/preset_test.go index 160dadb..9b63fe5 100644 --- a/internal/bench/preset_test.go +++ b/internal/bench/preset_test.go @@ -92,3 +92,92 @@ func TestOverridesProduceOneDiagnosticTopology(t *testing.T) { } } } + +func TestLocalPlanKeepsPublishShapeOnUpperSweepPoints(t *testing.T) { + plan, err := Plan(Config{Preset: "local", Repeat: 1, BatchSize: 1000}) + if err != nil { + t.Fatal(err) + } + if len(plan) != 14 { + t.Fatalf("phases=%d want 14", len(plan)) + } + if p := plan[0]; p.Kind != "warmup" || p.Duration != 15*time.Second { + t.Fatalf("warmup changed: %+v", p) + } + if p := plan[len(plan)-1]; p.Kind != "sustained" || p.Signal != Mixed || p.Duration != 5*time.Minute || p.ProcessesPerSignal != 0 { + t.Fatalf("sustained changed: %+v", p) + } + var total time.Duration + for _, signal := range []Signal{Traces, Logs, Metrics, Mixed} { + var processes []int + for _, phase := range plan { + total += 0 + if phase.Kind != "sweep" || phase.Signal != signal { + continue + } + processes = append(processes, phase.ProcessesPerSignal) + if phase.WorkersPerProcess != 8 || phase.RatePerWorker != 2500 || phase.Duration != 30*time.Second { + t.Fatalf("local shape changed: %+v", phase) + } + } + if !slices.Equal(processes, []int{8, 16, 32}) { + t.Errorf("%s: processes=%v", signal, processes) + } + } + for _, phase := range plan { + total += phase.Duration + } + if total > 12*time.Minute { + t.Errorf("load time %s exceeds the 12 minute budget", total) + } +} + +func TestMixedOnlyPlan(t *testing.T) { + for _, preset := range []string{"publish", "validate"} { + plan, err := Plan(Config{Preset: preset, Repeat: 3, BatchSize: 1000, SignalSelection: "mixed", QueryRate: 20}) + if err != nil { + t.Fatal(err) + } + if len(plan) != 23 { + t.Fatalf("%s phases=%d", preset, len(plan)) + } + for _, p := range plan { + if p.Signal != Mixed { + t.Fatalf("unexpected single-signal phase: %+v", p) + } + if p.Kind != "warmup" && p.QueryRate != 20 { + t.Fatal("missing reads") + } + } + last := plan[len(plan)-1] + if last.Kind != "sustained" || last.Duration != 5*time.Minute || last.ProcessesPerSignal != 0 { + t.Fatal(last) + } + } + if _, err := Plan(Config{Preset: "publish", SignalSelection: "logs", Repeat: 1, BatchSize: 1000}); err == nil { + t.Fatal("invalid selection accepted") + } +} + +func TestValidationUsesPublishShape(t *testing.T) { + cfg := Config{Preset: "validate", Repeat: 3, BatchSize: 1000} + a, err := Plan(cfg) + if err != nil { + t.Fatal(err) + } + cfg.Preset = "publish" + b, err := Plan(cfg) + if err != nil { + t.Fatal(err) + } + if !slices.Equal(a, b) { + t.Fatal("validation changed load shape") + } + if settingsFrom(cfg).EarlyStop { + t.Fatal("publish early stop enabled") + } + cfg.Preset = "validate" + if !settingsFrom(cfg).EarlyStop { + t.Fatal("validate early stop disabled") + } +} diff --git a/internal/bench/query.go b/internal/bench/query.go index efb375a..ccae232 100644 --- a/internal/bench/query.go +++ b/internal/bench/query.go @@ -17,7 +17,12 @@ import ( // maxQueryBody bounds a single response. Oversized responses fail because // their panel errors and discovered trace contents cannot be validated. -const maxQueryBody = 32 << 20 +const ( + maxQueryBody = 32 << 20 + queryRunPath = "/api/queries/run" + queryExemplarsPath = "/api/queries/exemplars" + queryAnnotationsPath = "/api/annotations/query" +) type queryRun struct { targets []queryTarget @@ -26,6 +31,8 @@ type queryRun struct { selections map[string]drillSelection kindsMu sync.Mutex kinds map[string]*operationStats + panelsMu sync.Mutex + panels map[string]*panelStats wg sync.WaitGroup hist *histogram failures int64 @@ -43,7 +50,7 @@ type operationStats struct { } func newQueryRun() *queryRun { - return &queryRun{targets: []queryTarget{{name: "dashboard-refresh", body: []byte(`{}`)}}, traces: map[string]traceReference{}, selections: map[string]drillSelection{}, kinds: map[string]*operationStats{}, hist: newHistogram(), stop: make(chan struct{}), cancel: func() {}} + return &queryRun{targets: []queryTarget{{name: "dashboard-refresh", body: []byte(`{}`)}}, traces: map[string]traceReference{}, selections: map[string]drillSelection{}, kinds: map[string]*operationStats{}, panels: map[string]*panelStats{}, hist: newHistogram(), stop: make(chan struct{}), cancel: func() {}} } func (q *queryRun) start(ctx context.Context, client *http.Client, base, cookie string, rate float64) { @@ -136,15 +143,13 @@ func (q *queryRun) request(ctx context.Context, client *http.Client, base, cooki } }() refresh := func() bool { - body, ok := queryHTTP(ctx, client, base, cookie, http.MethodPost, "/api/panels/query", query.body) - if !ok { - return false - } - annotations, valid := q.panelResponse(body, query) + body, ok, absent := queryPanelHTTP(ctx, client, base, cookie, queryRunPath, query.body) + annotations, valid := q.recordPanelResponse(body, query, absent, ctx.Err() == nil) // refresh.ts requests annotations after the panel batch, even if a // panel reports an error. Count the whole operation as failed either way. if annotations != nil { - _, ok = queryHTTP(ctx, client, base, cookie, http.MethodPost, "/api/annotations", annotations) + _, annotationsOK := queryHTTP(ctx, client, base, cookie, http.MethodPost, queryAnnotationsPath, annotations) + ok = ok && annotationsOK } return valid && ok } @@ -173,7 +178,7 @@ func (q *queryRun) request(ctx context.Context, client *http.Client, base, cooki success = refresh() return } - body, ok := queryHTTP(ctx, client, base, cookie, http.MethodPost, "/api/panels/exemplars", exemplarBody(query, selection)) + body, ok := queryHTTP(ctx, client, base, cookie, http.MethodPost, queryExemplarsPath, exemplarBody(query, selection)) success = ok && q.exemplarResponse(body, query, selection) } @@ -228,44 +233,51 @@ func (q *queryRun) operationSummary(phase string) string { return b.String() } +// Logarithmic fixed buckets bound memory independently of run length. Quantiles +// use bucket upper bounds (within 1% or 0.01ms), clamped to the observed maximum. type histogram struct { - mu sync.Mutex - values []float64 - sum float64 - max float64 + mu sync.Mutex + buckets [2048]int64 + count int64 + sum, max float64 } func newHistogram() *histogram { return &histogram{} } -func (h *histogram) record(d time.Duration) { - ms := float64(d) / float64(time.Millisecond) +func (h *histogram) record(d time.Duration) { h.recordMS(float64(d) / float64(time.Millisecond)) } + +func (h *histogram) recordMS(ms float64) { + if ms < 0 || math.IsNaN(ms) || math.IsInf(ms, 0) { + return + } + index := min(len(h.buckets)-1, int(math.Ceil(math.Log1p(ms)/math.Log(1.01)))) h.mu.Lock() - h.values = append(h.values, ms) + defer h.mu.Unlock() + h.buckets[index]++ + h.count++ h.sum += ms - if ms > h.max { - h.max = ms - } - h.mu.Unlock() + h.max = max(h.max, ms) } func (h *histogram) snapshot() Latency { h.mu.Lock() - values := append([]float64(nil), h.values...) - sum, max := h.sum, h.max - h.mu.Unlock() - if len(values) == 0 { + defer h.mu.Unlock() + if h.count == 0 { return Latency{} } - sort.Float64s(values) quantile := func(q float64) float64 { - index := int(math.Ceil(q*float64(len(values)))) - 1 - if index < 0 { - index = 0 - } - if index >= len(values) { - index = len(values) - 1 + target := int64(math.Ceil(q * float64(h.count))) + var count int64 + for i, n := range h.buckets { + count += n + if count >= target { + if i == len(h.buckets)-1 { + return round(h.max, 2) + } + return round(min(h.max, math.Expm1(float64(i)*math.Log(1.01))), 2) + } } - return round(values[index], 2) + return round(h.max, 2) } - return Latency{Count: int64(len(values)), MeanMs: round(sum/float64(len(values)), 2), P50Ms: quantile(.5), P95Ms: quantile(.95), P99Ms: quantile(.99), MaxMs: round(max, 2)} + return Latency{Count: h.count, MeanMs: round(h.sum/float64(h.count), 2), P50Ms: quantile(.5), P95Ms: quantile(.95), P99Ms: quantile(.99), MaxMs: round(h.max, 2)} } diff --git a/internal/bench/query_test.go b/internal/bench/query_test.go index e6f6670..caffc27 100644 --- a/internal/bench/query_test.go +++ b/internal/bench/query_test.go @@ -13,7 +13,7 @@ func TestQueryRunMaintainsOpenLoopScheduleThroughLatency(t *testing.T) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { time.Sleep(100 * time.Millisecond) w.Header().Set("Content-Type", "application/json") - fmt.Fprint(w, `{"results":[{"status":"empty"}]}`) + fmt.Fprint(w, `{"status":"empty"}`) })) defer server.Close() @@ -41,7 +41,7 @@ func TestQueryRunFinishDrainsInFlightRequestsWithoutFailures(t *testing.T) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { time.Sleep(150 * time.Millisecond) w.Header().Set("Content-Type", "application/json") - fmt.Fprint(w, `{"results":[{"status":"empty"}]}`) + fmt.Fprint(w, `{"status":"empty"}`) })) defer server.Close() @@ -67,7 +67,7 @@ func TestQueryRunFinishDrainsInFlightRequestsWithoutFailures(t *testing.T) { func TestQueryRunDoesNotShedWhileWorkersAreAvailable(t *testing.T) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "application/json") - fmt.Fprint(w, `{"results":[{"status":"empty"}]}`) + fmt.Fprint(w, `{"status":"empty"}`) })) defer server.Close() diff --git a/internal/bench/query_workload.go b/internal/bench/query_workload.go index 77bbc40..4597506 100644 --- a/internal/bench/query_workload.go +++ b/internal/bench/query_workload.go @@ -97,7 +97,7 @@ func systemQueryTargets(spec json.RawMessage) ([]queryTarget, string, error) { if err := json.Unmarshal(spec, &dashboard); err != nil { return nil, "", err } - if dashboard.Name != "System overview" || len(dashboard.Panels) == 0 { + if dashboard.Name != "System overview" || len(dashboard.Panels) == 0 || len(dashboard.Panels) > maxPanelIDs { return nil, "", errors.New("invalid System overview spec") } ids := []string{} @@ -106,6 +106,12 @@ func systemQueryTargets(spec json.RawMessage) ([]queryTarget, string, error) { drills := map[string]string{} var drillIDs []string for _, p := range dashboard.Panels { + if p.ID == "" || len(p.ID) > 128 || p.ID == otherPanelID { + return nil, "", errors.New("invalid dashboard panel ID") + } + if _, exists := widths[p.ID]; exists { + return nil, "", errors.New("duplicate dashboard panel ID") + } ids = append(ids, p.ID) width := p.Width if width == 0 { @@ -185,12 +191,16 @@ func exemplarBody(query queryTarget, selection drillSelection) []byte { func uiTime(ms int64) string { return time.UnixMilli(ms).UTC().Format("2006-01-02T15:04:05.000Z") } type panelResult struct { - ID string `json:"id"` - Status string `json:"status"` - Error string `json:"error"` - From *int64 `json:"from_ms"` - To *int64 `json:"to_ms"` - Frame *struct { + ID string `json:"id"` + Reason string `json:"reason"` + ElapsedMS *float64 `json:"elapsed_ms"` + Cached bool `json:"cached"` + FreshToMS *int64 `json:"fresh_to_ms"` + Status string `json:"status"` + Error string `json:"error"` + From *int64 `json:"from_ms"` + To *int64 `json:"to_ms"` + Frame *struct { Columns []struct { Name string `json:"name"` } `json:"columns"` @@ -199,16 +209,33 @@ type panelResult struct { } func (q *queryRun) panelResponse(body []byte, query queryTarget) ([]byte, bool) { - var response struct { - Results []panelResult `json:"results"` + return q.recordPanelResponse(body, query, "missing", true) +} + +func (q *queryRun) recordPanelResponse(body []byte, query queryTarget, absent string, attributeFailures bool) ([]byte, bool) { + results, parsed := decodePanelResults(body) + // Keep one outcome per panel per refresh. Duplicate IDs invalidate the panel. + unique := make(map[string]panelResult) + duplicates := make(map[string]bool) + for _, result := range results { + if _, exists := unique[result.ID]; exists { + duplicates[result.ID] = true + continue + } + unique[result.ID] = result } - if json.Unmarshal(body, &response) != nil || len(response.Results) == 0 { - return nil, false + for id, result := range unique { + if duplicates[id] { + result.Status, result.Error, result.Reason = "error", "duplicate panel result", "" + } + result.Status = panelStatus(result) + unique[id] = result } + q.recordPanels(query, unique, absent, attributeFailures) seen := map[string]bool{} - valid := true + valid := parsed && len(results) > 0 var from, to int64 - for _, result := range response.Results { + for _, result := range unique { if seen[result.ID] || (result.Status != "ok" && result.Status != "empty") || result.Error != "" { valid = false } diff --git a/internal/bench/query_workload_test.go b/internal/bench/query_workload_test.go index 40ff371..d08f10d 100644 --- a/internal/bench/query_workload_test.go +++ b/internal/bench/query_workload_test.go @@ -1,6 +1,7 @@ package bench import ( + "bytes" "context" "encoding/json" "fmt" @@ -82,9 +83,8 @@ func TestSystemOperationsMatchUIRequestConstruction(t *testing.T) { // Mirrors a panel batch response, including the effective window from discovery // and a selectable service-map node. No invented search dashboard is needed. -func panelBatch(t *testing.T, query queryTarget, from, to int64) []byte { - t.Helper() - results := []any{} +func panelResults(query queryTarget, from, to int64) []map[string]any { + results := []map[string]any{} for _, id := range query.panels { result := map[string]any{"id": id, "status": "empty", "from_ms": from, "to_ms": to} if id == "service_map" { @@ -93,11 +93,25 @@ func panelBatch(t *testing.T, query queryTarget, from, to int64) []byte { } results = append(results, result) } - body, err := json.Marshal(map[string]any{"results": results}) - if err != nil { - t.Fatal(err) + return results +} + +// panelStream encodes results as the API's one-result-per-line stream. +func panelStream(t *testing.T, results []map[string]any) []byte { + t.Helper() + var body bytes.Buffer + encoder := json.NewEncoder(&body) + for _, result := range results { + if err := encoder.Encode(result); err != nil { + t.Fatal(err) + } } - return body + return body.Bytes() +} + +func panelBatch(t *testing.T, query queryTarget, from, to int64) []byte { + t.Helper() + return panelStream(t, panelResults(query, from, to)) } func TestWorkloadDiscoversAuthenticatedBuiltInDashboard(t *testing.T) { @@ -138,9 +152,9 @@ func TestRefreshIncludesAnnotationsAndMeasuresWholeOperation(t *testing.T) { } time.Sleep(15 * time.Millisecond) switch r.URL.Path { - case "/api/panels/query": + case "/api/queries/run": w.Write(panelBatch(t, targets[0], 1000, 2000)) - case "/api/annotations": + case "/api/annotations/query": var body map[string]string _ = json.NewDecoder(r.Body).Decode(&body) if !reflect.DeepEqual(body, map[string]string{"from": "1970-01-01T00:00:01.000Z", "to": "1970-01-01T00:00:02.000Z"}) { @@ -155,7 +169,7 @@ func TestRefreshIncludesAnnotationsAndMeasuresWholeOperation(t *testing.T) { q := newQueryRun() q.targets = targets q.request(context.Background(), server.Client(), server.URL, "fanout_session=test", 0) - if !reflect.DeepEqual(paths, []string{"/api/panels/query", "/api/annotations"}) || q.failures != 0 || q.hist.snapshot().Count != 1 || q.hist.snapshot().MeanMs < 30 { + if !reflect.DeepEqual(paths, []string{"/api/queries/run", "/api/annotations/query"}) || q.failures != 0 || q.hist.snapshot().Count != 1 || q.hist.snapshot().MeanMs < 30 { t.Fatalf("paths=%v failures=%d latency=%+v", paths, q.failures, q.hist.snapshot()) } if summary := q.operationSummary("mixed"); !strings.Contains(summary, "| dashboard-refresh | 1 | 1 | 0 |") { @@ -207,7 +221,7 @@ func TestTraceDetailFallsBackOnlyWithinItsOwnWindow(t *testing.T) { q.traces["15m"] = traceReference{"other-window", "ns", "old-from", "old-to"} q.selections["1h/service_map"] = drillSelection{"2026-10-01T12:00:00.000Z", "2026-10-01T13:00:00.000Z", map[string]string{"service": "checkout"}} server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/api/panels/exemplars" { + if r.URL.Path != "/api/queries/exemplars" { t.Errorf("wrong fallback path %s", r.URL.Path) } var body map[string]any @@ -231,11 +245,11 @@ func TestSustainedQueryGateFailsWhenExemplarsStayEmpty(t *testing.T) { targets, _, _ := systemQueryTargets(systemSpec(t)) server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.URL.Path { - case "/api/panels/query": + case "/api/queries/run": w.Write(panelBatch(t, targets[0], 1000, 2000)) - case "/api/annotations": + case "/api/annotations/query": fmt.Fprint(w, `{}`) - case "/api/panels/exemplars": + case "/api/queries/exemplars": fmt.Fprint(w, `{"traces":[]}`) default: t.Errorf("unexpected request: %s", r.URL.Path) @@ -293,16 +307,16 @@ func TestAnyOperationRequestOrPanelErrorFails(t *testing.T) { targets, _, _ := systemQueryTargets(systemSpec(t)) server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.URL.Path { - case "/api/panels/query": + case "/api/queries/run": body := panelBatch(t, targets[0], 1000, 2000) if defect == "panel-error" { body = []byte(strings.Replace(string(body), `"status":"empty"`, `"status":"error"`, 1)) } if defect == "missing-panel" { - body = []byte(`{"results":[{"id":"traffic","status":"empty"}]}`) + body = []byte(`{"id":"traffic","status":"empty"}`) } w.Write(body) - case "/api/annotations": + case "/api/annotations/query": if defect == "annotation-http" { http.Error(w, "failed", 500) } else if defect == "annotation-json" { @@ -310,7 +324,7 @@ func TestAnyOperationRequestOrPanelErrorFails(t *testing.T) { } else { fmt.Fprint(w, `{}`) } - case "/api/panels/exemplars": + case "/api/queries/exemplars": http.Error(w, "failed", 500) default: t.Errorf("unexpected route %s", r.URL.Path) @@ -337,20 +351,19 @@ func TestEmptyDashboardDoesNotInventDrillSelection(t *testing.T) { requests := 0 server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { requests++ - if r.URL.Path == "/api/annotations" { + if r.URL.Path == "/api/annotations/query" { fmt.Fprint(w, `{}`) return } - if r.URL.Path != "/api/panels/query" { + if r.URL.Path != "/api/queries/run" { t.Errorf("invented selection: %s", r.URL.Path) } - body := decodeBody(t, panelBatch(t, targets[0], 1000, 2000)) - for _, raw := range body["results"].([]any) { - result := raw.(map[string]any) + results := panelResults(targets[0], 1000, 2000) + for _, result := range results { delete(result, "frame") result["status"] = "empty" } - _ = json.NewEncoder(w).Encode(body) + _, _ = w.Write(panelStream(t, results)) })) defer server.Close() q := newQueryRun() @@ -363,7 +376,9 @@ func TestEmptyDashboardDoesNotInventDrillSelection(t *testing.T) { func TestAnnotationWindowUnionAndDisabledAnnotations(t *testing.T) { query := queryTarget{timePanels: map[string]bool{"traffic": true, "latency": true}, annotations: true} - body := []byte(`{"results":[{"id":"traffic","status":"empty","from_ms":2000,"to_ms":5000},{"id":"latency","status":"empty","from_ms":1000,"to_ms":3000},{"id":"requests","status":"empty","from_ms":0,"to_ms":9000}]}`) + body := []byte(`{"id":"traffic","status":"empty","from_ms":2000,"to_ms":5000} +{"id":"latency","status":"empty","from_ms":1000,"to_ms":3000} +{"id":"requests","status":"empty","from_ms":0,"to_ms":9000}`) q := newQueryRun() annotations, ok := q.panelResponse(body, query) if !ok || !reflect.DeepEqual(decodeBody(t, annotations), map[string]any{"from": "1970-01-01T00:00:01.000Z", "to": "1970-01-01T00:00:05.000Z"}) { @@ -390,7 +405,7 @@ func TestUIBatchIncludesTextIDsAndRoundedGridWidths(t *testing.T) { t.Fatalf("batch=%v", body) } q := newQueryRun() - if _, ok := q.panelResponse([]byte(`{"results":[{"id":"a","status":"empty"}]}`), targets[0]); !ok { + if _, ok := q.panelResponse([]byte(`{"id":"a","status":"empty"}`), targets[0]); !ok { t.Fatal("text panel incorrectly required a result") } } @@ -426,11 +441,11 @@ func TestDiscoveryFallbackKeepsOneUserOperationPerSlot(t *testing.T) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { paths = append(paths, r.URL.Path) switch r.URL.Path { - case "/api/panels/query": + case "/api/queries/run": w.Write(panelBatch(t, targets[0], 1000, 2000)) - case "/api/annotations": + case "/api/annotations/query": fmt.Fprint(w, `{}`) - case "/api/panels/exemplars": + case "/api/queries/exemplars": fmt.Fprint(w, `{"traces":[{"trace_id":"trace","namespace":"ns"}]}`) case "/api/traces/trace": fmt.Fprint(w, `{"data":{"spans":[{"span_id":"span"}]}}`) @@ -441,7 +456,7 @@ func TestDiscoveryFallbackKeepsOneUserOperationPerSlot(t *testing.T) { defer server.Close() q := newQueryRun() q.targets = targets - for i, want := range [][]string{{"/api/panels/query", "/api/annotations"}, {"/api/panels/exemplars"}, {"/api/traces/trace"}} { + for i, want := range [][]string{{"/api/queries/run", "/api/annotations/query"}, {"/api/queries/exemplars"}, {"/api/traces/trace"}} { paths = nil q.request(context.Background(), server.Client(), server.URL, "fanout_session=test", 2) if !reflect.DeepEqual(paths, want) { diff --git a/internal/bench/remote.go b/internal/bench/remote.go index 16462c4..57c768b 100644 --- a/internal/bench/remote.go +++ b/internal/bench/remote.go @@ -5,6 +5,7 @@ import ( "compress/gzip" "context" "crypto/sha256" + "debug/elf" "encoding/hex" "errors" "fmt" @@ -22,6 +23,44 @@ import ( var sshTargetPattern = regexp.MustCompile(`^(?:[A-Za-z0-9._-]+@)?(?:[A-Za-z0-9._-]+|\[[0-9A-Fa-f:]+\])$`) var fanoutVersionPattern = regexp.MustCompile(`^v[0-9]{4}\.(?:[1-9]|1[0-2])\.(?:0|[1-9][0-9]*)(?:-(?:alpha|beta|rc))?$`) +var unreleasedLabelPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$`) + +// UnreleasedArtifactPrefix marks a Fanout binary supplied with --fanout-binary. +// Such a build has no release checksum to verify against, so its results are +// diagnostic and never publishable. +const UnreleasedArtifactPrefix = "unreleased-local-binary" + +// ValidateFanoutSource checks the Fanout identity of a remote run: a published +// release by default, or a local binary with a label that cannot be mistaken +// for a release. +func ValidateFanoutSource(cfg Config) error { + if cfg.FanoutBinary == "" { + if cfg.FanoutVersion == "" || cfg.FanoutVersion == "unknown" { + return errors.New("--fanout-version is required for a remote run") + } + if !fanoutVersionPattern.MatchString(cfg.FanoutVersion) { + return fmt.Errorf("invalid Fanout release %q: want vYYYY.M.N[-alpha|-beta|-rc]", cfg.FanoutVersion) + } + return nil + } + if cfg.FanoutVersion == "" || cfg.FanoutVersion == "unknown" { + return errors.New("--fanout-binary requires a --fanout-version label for the unreleased build") + } + if fanoutVersionPattern.MatchString(cfg.FanoutVersion) { + return fmt.Errorf("--fanout-binary label %q looks like a release; use a non-release label such as branch-commit", cfg.FanoutVersion) + } + if !unreleasedLabelPattern.MatchString(cfg.FanoutVersion) { + return fmt.Errorf("invalid unreleased Fanout label %q: want letters, digits, '.', '_' or '-'", cfg.FanoutVersion) + } + info, err := os.Stat(cfg.FanoutBinary) + if err != nil { + return fmt.Errorf("--fanout-binary: %w", err) + } + if !info.Mode().IsRegular() { + return fmt.Errorf("--fanout-binary %s is not a regular file", cfg.FanoutBinary) + } + return nil +} // ValidateGeneratorHosts also runs before local execution, so partial split // configurations cannot silently fall back to same-host load. @@ -56,11 +95,8 @@ func RunRemote(ctx context.Context, cfg Config, version string, out io.Writer) e if cfg.SSHPort < 1 || cfg.SSHPort > 65535 { return fmt.Errorf("invalid SSH port %d", cfg.SSHPort) } - if cfg.FanoutVersion == "" || cfg.FanoutVersion == "unknown" { - return errors.New("--fanout-version is required for a remote run") - } - if !fanoutVersionPattern.MatchString(cfg.FanoutVersion) { - return fmt.Errorf("invalid Fanout release %q: want vYYYY.M.N[-alpha|-beta|-rc]", cfg.FanoutVersion) + if err := ValidateFanoutSource(cfg); err != nil { + return err } if _, err := Plan(cfg); err != nil { return err @@ -118,10 +154,19 @@ func RunRemote(ctx context.Context, cfg Config, version string, out io.Writer) e if err != nil { return fmt.Errorf("build telemetrygen: %w", err) } - fmt.Fprintln(out, " ◌ downloading and verifying the Fanout release") - fanoutPath, artifact, err := fetchFanout(ctx, buildDir, cfg.FanoutVersion, remoteOS, remoteArch) - if err != nil { - return fmt.Errorf("fetch Fanout release: %w", err) + var fanoutPath, artifact string + if cfg.FanoutBinary != "" { + fmt.Fprintln(out, " ◌ staging the unreleased local Fanout binary (not publishable)") + fanoutPath, artifact, err = stageLocalFanout(cfg.FanoutBinary, buildDir, remoteArch) + if err != nil { + return fmt.Errorf("stage local Fanout binary: %w", err) + } + } else { + fmt.Fprintln(out, " ◌ downloading and verifying the Fanout release") + fanoutPath, artifact, err = fetchFanout(ctx, buildDir, cfg.FanoutVersion, remoteOS, remoteArch) + if err != nil { + return fmt.Errorf("fetch Fanout release: %w", err) + } } suffix, err := randomHex(16) @@ -380,6 +425,41 @@ func fetchFanout(ctx context.Context, dir, version, goos, goarch string) (string return binaryPath, assetURL + "#sha256=" + got, nil } +// stageLocalFanout copies a local Linux ELF binary for the remote architecture +// into the upload directory and identifies it by checksum as unreleased. +func stageLocalFanout(source, dir, goarch string) (string, string, error) { + file, err := elf.Open(source) + if err != nil { + return "", "", fmt.Errorf("%s is not a Linux ELF binary: %w", source, err) + } + machine := file.Machine + file.Close() + want := map[string]elf.Machine{"amd64": elf.EM_X86_64, "arm64": elf.EM_AARCH64}[goarch] + if machine != want { + return "", "", fmt.Errorf("%s targets %s, remote host is %s", source, machine, goarch) + } + in, err := os.Open(source) + if err != nil { + return "", "", err + } + defer in.Close() + destination := filepath.Join(dir, "fanout") + out, err := os.OpenFile(destination, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o755) + if err != nil { + return "", "", err + } + hash := sha256.New() + _, copyErr := io.Copy(io.MultiWriter(out, hash), in) + closeErr := out.Close() + if copyErr != nil { + return "", "", copyErr + } + if closeErr != nil { + return "", "", closeErr + } + return destination, UnreleasedArtifactPrefix + "#sha256=" + hex.EncodeToString(hash.Sum(nil)), nil +} + func fanoutAsset(version, goos, goarch string) (string, error) { if !fanoutVersionPattern.MatchString(version) { return "", fmt.Errorf("invalid Fanout release %q", version) @@ -447,7 +527,7 @@ func extractBinary(archivePath, destination, name string) error { } func managedRemoteCommand(stage string, cfg Config, version, artifact string) string { - args := []string{stage + "/fanout-bench", "managed", "--fanout", stage + "/fanout", "--output", stage + "/results", "--fanout-version", cfg.FanoutVersion, "--fanout-artifact", artifact, "--preset", cfg.Preset, "--repeat", fmt.Sprint(cfg.Repeat), "--batch-size", fmt.Sprint(cfg.BatchSize), "--query-rate", fmt.Sprint(cfg.QueryRate), "--sample-interval", cfg.SampleInterval.String(), "--cooldown", cfg.Cooldown.String(), "--harness-version", version} + args := []string{stage + "/fanout-bench", "managed", "--fanout", stage + "/fanout", "--output", stage + "/results", "--fanout-version", cfg.FanoutVersion, "--fanout-artifact", artifact, "--signals", cfg.SignalSelection, "--preset", cfg.Preset, "--repeat", fmt.Sprint(cfg.Repeat), "--batch-size", fmt.Sprint(cfg.BatchSize), "--query-rate", fmt.Sprint(cfg.QueryRate), "--sample-interval", cfg.SampleInterval.String(), "--cooldown", cfg.Cooldown.String(), "--harness-version", version} args = append(args, "--telemetrygen-version", cfg.TelemetrygenVersion) args = append(args, "--provisioned-host", redactedHost(cfg.Host), "--host-description", cfg.HostDescription) args = append(args, "--provisioned-ssh-port", fmt.Sprint(cfg.SSHPort), "--provisioned-ssh-identity", cfg.SSHIdentity) diff --git a/internal/bench/remote_test.go b/internal/bench/remote_test.go index b3c1d86..f26ea13 100644 --- a/internal/bench/remote_test.go +++ b/internal/bench/remote_test.go @@ -13,6 +13,7 @@ import ( "net/http/httptest" "os" "path/filepath" + "runtime" "slices" "strings" "testing" @@ -335,3 +336,93 @@ func TestKnownHostsReachesSSHAndSCPArgs(t *testing.T) { }) } } + +func TestValidateFanoutSource(t *testing.T) { + dir := t.TempDir() + binary := filepath.Join(dir, "fanout") + if err := os.WriteFile(binary, []byte("binary"), 0o755); err != nil { + t.Fatal(err) + } + valid := []Config{ + {FanoutVersion: "v2026.10.8-alpha"}, + {FanoutVersion: "explore-fdfbf7f5", FanoutBinary: binary}, + {FanoutVersion: "main.abc123_dirty", FanoutBinary: binary}, + } + for _, cfg := range valid { + if err := ValidateFanoutSource(cfg); err != nil { + t.Errorf("%+v: %v", cfg, err) + } + } + invalid := []Config{ + {FanoutVersion: "unknown"}, + {FanoutVersion: "explore-fdfbf7f5"}, + {FanoutVersion: "unknown", FanoutBinary: binary}, + {FanoutVersion: "", FanoutBinary: binary}, + {FanoutVersion: "v2026.10.8-alpha", FanoutBinary: binary}, + {FanoutVersion: "-explore", FanoutBinary: binary}, + {FanoutVersion: "explore/evil", FanoutBinary: binary}, + {FanoutVersion: "explore;bad", FanoutBinary: binary}, + {FanoutVersion: "explore", FanoutBinary: filepath.Join(dir, "missing")}, + {FanoutVersion: "explore", FanoutBinary: dir}, + } + for _, cfg := range invalid { + if err := ValidateFanoutSource(cfg); err == nil { + t.Errorf("accepted invalid Fanout source %+v", cfg) + } + } +} + +func TestStageLocalFanoutRejectsNonELFAndWrongArch(t *testing.T) { + dir := t.TempDir() + script := filepath.Join(dir, "fanout") + if err := os.WriteFile(script, []byte("#!/bin/sh\n"), 0o755); err != nil { + t.Fatal(err) + } + if _, _, err := stageLocalFanout(script, t.TempDir(), "amd64"); err == nil { + t.Error("accepted a non-ELF binary") + } + // The test binary itself is an ELF only on Linux; elsewhere skip the arch check. + self, err := os.Executable() + if err != nil || runtime.GOOS != "linux" { + return + } + other := map[string]string{"amd64": "arm64", "arm64": "amd64"}[runtime.GOARCH] + if _, _, err := stageLocalFanout(self, t.TempDir(), other); err == nil { + t.Error("accepted a binary for the wrong architecture") + } + path, artifact, err := stageLocalFanout(self, t.TempDir(), runtime.GOARCH) + if err != nil { + t.Fatal(err) + } + if filepath.Base(path) != "fanout" || !strings.HasPrefix(artifact, UnreleasedArtifactPrefix+"#sha256=") { + t.Errorf("path=%q artifact=%q", path, artifact) + } +} + +func TestUnreleasedBuildIsMarkedNotPublishable(t *testing.T) { + report := Report{Build: Build{FanoutVersion: "explore-fdfbf7f5", FanoutArtifact: UnreleasedArtifactPrefix + "#sha256=00"}} + s := summarize(report, 0, 0) + if len(s.Warnings) == 0 || !strings.Contains(s.Warnings[0], "must not be published") { + t.Errorf("warnings=%q", s.Warnings) + } + report.Build.FanoutArtifact = "https://github.com/labstack/fanout/releases/download/v2026.10.8-alpha/x#sha256=00" + for _, warning := range summarize(report, 0, 0).Warnings { + if strings.Contains(warning, "must not be published") { + t.Errorf("release run marked unpublishable: %q", warning) + } + } +} + +func TestValidationSelectionForwardedToManaged(t *testing.T) { + cfg := Config{Preset: "validate", SignalSelection: "mixed"} + command := managedRemoteCommand("/stage", cfg, "test", "artifact") + for _, want := range []string{"'--signals' 'mixed'", "'--preset' 'validate'"} { + if !strings.Contains(command, want) { + t.Fatalf("missing %s: %s", want, command) + } + } + got := managedRunnerConfig(ManagedConfig{Preset: "validate", SignalSelection: "mixed"}, "127.0.0.1:1", "/stage/data") + if got.SignalSelection != cfg.SignalSelection || got.Preset != cfg.Preset { + t.Fatal("managed flags lost") + } +} diff --git a/internal/bench/report.go b/internal/bench/report.go index d3010c4..a4de037 100644 --- a/internal/bench/report.go +++ b/internal/bench/report.go @@ -118,6 +118,10 @@ func printSummary(w io.Writer, r Report) { name string peak Peak }{{"Traces", r.Summary.PeakTraces}, {"Logs", r.Summary.PeakLogs}, {"Metric points", r.Summary.PeakMetrics}, {"Mixed peak", r.Summary.PeakMixed}, {"Mixed sustained", r.Summary.SustainedMixed}} { + if r.Settings.Signals == "mixed" && !strings.HasPrefix(row.name, "Mixed") && row.peak.TrialName == "" { + fmt.Fprintf(t, " %s\tskipped (--signals mixed)\n", row.name) + continue + } fmt.Fprintf(t, " %s\t%s\t%s\t%.2f cores\t%s\n", row.name, formatNumber(row.peak.AcceptedPerSecond), formatNumber(row.peak.OfferedPerSecond), row.peak.AverageCPUCores, formatBytes(row.peak.PeakRSSBytes)) } t.Flush() @@ -142,6 +146,10 @@ func Markdown(r Report) string { name string peak Peak }{{"Traces (spans)", r.Summary.PeakTraces}, {"Logs (records)", r.Summary.PeakLogs}, {"Metrics (data points)", r.Summary.PeakMetrics}, {"Mixed peak", r.Summary.PeakMixed}, {"Mixed sustained", r.Summary.SustainedMixed}} { + if r.Settings.Signals == "mixed" && !strings.HasPrefix(row.name, "Mixed") && row.peak.TrialName == "" { + fmt.Fprintf(&b, "| %s | skipped (--signals mixed) | — | — | — |\n", row.name) + continue + } fmt.Fprintf(&b, "| %s | %s | %s | %.2f cores | %s |\n", row.name, formatNumber(row.peak.AcceptedPerSecond), formatNumber(row.peak.OfferedPerSecond), row.peak.AverageCPUCores, formatBytes(row.peak.PeakRSSBytes)) } fmt.Fprintf(&b, "\n**Meaning:** accepted throughput is the delta of `fanout_ingest_rows_total` divided by the interval between the two counter scrapes that bracket the phase. It is not telemetrygen's requested or locally generated count.\n\n") @@ -159,6 +167,9 @@ func Markdown(r Report) string { for _, row := range settings { fmt.Fprintf(&b, "| %s | %s |\n", row[0], row[1]) } + for _, stop := range r.SweepStops { + fmt.Fprintf(&b, "\n**Sweep stopped after %s:** %s. Skipped: %s.\n", stop.AfterPhase, stop.Reason, strings.Join(stop.SkippedPhases, ", ")) + } fmt.Fprintf(&b, "\n## Complete phase breakdown\n\n| Phase | Signal | Topology (processes per signal × workers) | Duration | Offered/s | Accepted/s | Ratio | Fanout CPU cores | Generator CPU cores / logical CPUs | Peak RSS | Queue peak | Result |\n|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|\n") for _, t := range r.Trials { generatorCPU := "n/a" @@ -182,16 +193,20 @@ func Markdown(r Report) string { } fmt.Fprintln(&b) } + renderDashboardPanels(&b, r.Trials) fmt.Fprintf(&b, "\n## Methodology and limits\n\n%s\n\n", r.Settings.Methodology) for _, warning := range r.Summary.Warnings { fmt.Fprintf(&b, "- %s\n", warning) } - fmt.Fprintf(&b, "\n## Reproduce\n\n```sh\n%s\n```\n\nRaw evidence: `report.json` (authoritative structured report) and `samples.jsonl` (time-series resource samples, including cumulative `ingest_rows_total` by signal (`spans`, `logs`, `metrics`), `rollup_seconds`, `write_gate_hold_seconds`, `write_gate_wait_seconds`, `compaction_seconds_total`, `compactions_total`, `publish_wait_seconds_total`, and `ingest_backpressure_seconds_total`, plus the current `compaction_backlog_batches` gauge when available).\n", r.ReproductionCommand) + fmt.Fprintf(&b, "\n## Reproduce\n\n```sh\n%s\n```\n\nRaw evidence: `report.json` (authoritative structured report) and `samples.jsonl` (time-series resource samples, including cumulative `ingest_rows_total` by signal (`spans`, `logs`, `metrics`), `rollup_seconds`, `write_gate_hold_seconds`, `write_gate_wait_seconds`, `compaction_seconds_total`, `compactions_total`, `publish_wait_seconds_total`, `ingest_backpressure_seconds_total`, `ingest_throttled_total`, `ingest_admission_rows_per_second`, `compaction_retired_batches_total`, and `compaction_pass_seconds_total`, plus the current `compaction_backlog_batches` gauge when available).\n", r.ReproductionCommand) return b.String() } func reproductionCommand(c Config) string { command := reproductionRunCommand(c) + if c.SignalSelection != "" && c.SignalSelection != "all" { + command += " --signals " + shellQuote(c.SignalSelection) + } if c.HostDescription != "" { command += " --host-description " + shellQuote(c.HostDescription) } diff --git a/internal/bench/runner.go b/internal/bench/runner.go index a856632..e285a3f 100644 --- a/internal/bench/runner.go +++ b/internal/bench/runner.go @@ -68,20 +68,30 @@ func (r *Runner) Run(ctx context.Context) (*Report, error) { } report.Host.Description = r.Config.HostDescription report.Host.GeneratorDescription = r.Config.GeneratorHostDescription + if r.Config.SignalSelection == "mixed" { + report.Settings.Methodology += " Single-signal traces, logs, and metrics sweeps were skipped (--signals mixed); mixed warmup, mixed sweep and mixed sustained remain. Single-signal peaks were not measured." + } + if r.Config.Preset == "validate" { + report.Settings.Methodology += " Validation sweep early stop is enabled: two consecutive accepted-throughput gains below 5%, or acceptance ratio below 50% and below half the previous point. This is validation evidence, not a full publication sweep." + } report.Settings.Methodology += readMethod report.WorkloadSHA256 = workloadHash(report.Settings, plan) report.ReproductionCommand = reproductionCommand(r.Config) printRunHeader(r.Out, report, plan) + skipped := make(map[string]bool) for i := range plan { if err := ctx.Err(); err != nil { return r.abandon(report, err) } spec := plan[i] + if skipped[spec.Name] { + continue + } if spec.Kind == "sustained" && spec.ProcessesPerSignal == 0 { best := bestMixedTopology(report.Trials) if best <= 0 { - return r.abandon(report, errors.New("cannot select sustained topology: mixed sweep accepted no data")) + return r.abandon(report, errors.New("cannot select sustained topology: no valid mixed sweep trial; inspect generator-processes and classification notes")) } spec.ProcessesPerSignal = best plan[i] = spec @@ -94,6 +104,15 @@ func (r *Runner) Run(ctx context.Context) (*Report, error) { report.Trials = append(report.Trials, trial) report.Settings.Methodology += r.readSummary printTrialLine(r.Out, trial) + if r.Config.Preset == "validate" { + if stop := sweepStop(plan[i+1:], report.Trials); stop != nil { + report.SweepStops = append(report.SweepStops, *stop) + for _, name := range stop.SkippedPhases { + skipped[name] = true + } + fmt.Fprintf(r.Out, " sweep stopped: %s; skipped %v\n", stop.Reason, stop.SkippedPhases) + } + } } if r.Config.Cooldown > 0 { @@ -253,6 +272,7 @@ func (r *Runner) runPhase(ctx context.Context, spec PhaseSpec, previous []Trial) ProcessRestarted: restarted, Resources: resources, QueryLatency: queries.hist.snapshot(), QueryAttempts: queries.attempts, QueryFailures: queries.failures, QueryKindCompleted: queries.completedByKind(), + DashboardPanels: queries.panelSummary(), QueryShed: queries.shed, QueryCanceled: queries.canceled, Samples: samples, } @@ -400,7 +420,8 @@ func (r *Runner) sample(ctx context.Context, started time.Time, done chan<- []Re func resourceSample(s *snapshot, at float64) ResourceSample { return ResourceSample{ - AtSeconds: round(at, 3), CPUSeconds: s.total("process_cpu_seconds_total", nil), + PanelMetrics: panelMetricSamples(s), + AtSeconds: round(at, 3), CPUSeconds: s.total("process_cpu_seconds_total", nil), RSSBytes: s.total("process_resident_memory_bytes", nil), HeapBytes: s.total("go_memstats_heap_alloc_bytes", nil), QueueDepth: s.total("fanout_ingest_queue_depth", nil), ParquetFiles: s.total("fanout_parquet_files", nil), ParquetBytes: s.total("fanout_parquet_size_bytes", nil), PublishWaiters: s.total("fanout_parquet_publish_waiters", nil), @@ -413,6 +434,10 @@ func resourceSample(s *snapshot, at float64) ResourceSample { CompactionsTotal: s.total("fanout_telemetry_operation_duration_seconds_count", map[string]string{"operation": "compaction"}), PublishWaitSecondsTotal: s.total("fanout_parquet_publish_wait_seconds_sum", nil), IngestBackpressureSecondsTotal: s.total("fanout_ingest_backpressure_seconds_total", nil), + IngestThrottledTotal: sampleTotalsByLabel(s, "fanout_ingest_throttled_total", "reason"), + IngestAdmissionRowsPerSecond: s.total("fanout_ingest_admission_rows_per_second", nil), + CompactionRetiredBatchesTotal: s.total("fanout_compaction_retired_batches_total", nil), + CompactionPassSecondsTotal: s.total("fanout_compaction_pass_seconds_sum", nil), CompactionBacklogBatches: s.total("fanout_compaction_backlog_batches", nil), } } @@ -512,7 +537,7 @@ func bestMixedTopology(trials []Trial) int { bestRate, processes := -1.0, 0 fallbackCompletion, fallbackRate, fallbackProcesses := -1.0, -1.0, 0 for _, t := range trials { - if t.Spec.Signal != Mixed || t.Spec.Kind != "sweep" || t.GeneratorFailures != 0 { + if t.Spec.Signal != Mixed || t.Spec.Kind != "sweep" || t.GeneratorFailures != 0 || t.RowsDropped > 0 || t.ProcessRestarted || t.Accepted.Total() <= 0 { continue } completion := 1.0 @@ -597,8 +622,9 @@ func summarize(report Report, cooldownRSS, cooldownHeap float64) Summary { restartsOK = restartsOK && !t.ProcessRestarted dataOK = dataOK && t.Accepted.Total() > 0 } + generatorDetail := generatorProcessDetail(genOK, report.Trials) s.Gates = []Gate{ - {"generator-processes", genOK, boolDetail(genOK, "all exited successfully", "at least one failed")}, + {"generator-processes", genOK, generatorDetail}, {"server-dropped-rows", dropsOK, boolDetail(dropsOK, "zero", "non-zero")}, {"server-restarts", restartsOK, boolDetail(restartsOK, "none", "detected")}, {"authoritative-counters", dataOK, boolDetail(dataOK, "Fanout accepted rows in every phase", "a phase accepted no rows")}, @@ -628,6 +654,9 @@ func summarize(report Report, cooldownRSS, cooldownHeap float64) Summary { "telemetrygen payloads are intentionally synthetic and highly compressible; storage size is not representative of production telemetry.", "Offered load is a target. Headline throughput uses Fanout's accepted-row counters, never telemetrygen's intended or generated count.", } + if strings.HasPrefix(report.Build.FanoutArtifact, UnreleasedArtifactPrefix) { + s.Warnings = append([]string{"Fanout " + report.Build.FanoutVersion + " is an unreleased local build without a verifiable release checksum. This run is diagnostic and must not be published."}, s.Warnings...) + } if report.Host.GeneratorLocal { s.Warnings = append(s.Warnings, "Fanout and load generators shared one host; generator contention can make the observed peak lower than Fanout's backend ceiling.") } @@ -642,7 +671,7 @@ func settingsFrom(c Config) Settings { if c.Insecure { transport = "plaintext gRPC" } - return Settings{Preset: c.Preset, Endpoint: c.Endpoint, MetricsURL: c.MetricsURL, Transport: transport, Repeat: c.Repeat, + return Settings{Signals: c.SignalSelection, EarlyStop: c.Preset == "validate", Preset: c.Preset, Endpoint: c.Endpoint, MetricsURL: c.MetricsURL, Transport: transport, Repeat: c.Repeat, SampleInterval: c.SampleInterval, SampleSeconds: c.SampleInterval.Seconds(), GeneratorTimeout: c.GeneratorTimeout, TimeoutSeconds: c.GeneratorTimeout.Seconds(), Cooldown: c.Cooldown, CooldownSeconds: c.Cooldown.Seconds(), DataDir: c.DataDir, QueryURL: c.QueryURL, QueryRate: c.QueryRate, BatchSize: c.BatchSize, TraceChildSpans: 1, @@ -800,3 +829,21 @@ func boolDetail(ok bool, yes, no string) string { } return no } + +// Generator errors in classification notes have already had credentials redacted. +func generatorProcessDetail(ok bool, trials []Trial) string { + detail := boolDetail(ok, "all exited successfully", "at least one failed") + if ok { + return detail + } + for _, trial := range trials { + for _, note := range trial.ClassificationNotes { + for _, line := range strings.Split(note, "\n") { + if strings.Contains(line, "FATAL") { + return detail + "; " + strings.TrimSpace(line) + } + } + } + } + return detail +} diff --git a/internal/bench/runner_test.go b/internal/bench/runner_test.go index 992c3d4..b350cad 100644 --- a/internal/bench/runner_test.go +++ b/internal/bench/runner_test.go @@ -4,9 +4,11 @@ import ( "context" "encoding/json" "errors" + "fmt" "io" "net/http" "os" + "path/filepath" "reflect" "slices" "strings" @@ -101,6 +103,11 @@ fanout_telemetry_operation_duration_seconds_count{operation="ingest"} 100 fanout_parquet_publish_wait_seconds_sum{signal="spans"} 11 fanout_parquet_publish_wait_seconds_sum{signal="logs"} 0.5 fanout_ingest_backpressure_seconds_total 12.5 +fanout_ingest_throttled_total{reason="debt"} 14 +fanout_ingest_throttled_total{reason="reads"} 15 +fanout_compaction_retired_batches_total 128 +fanout_ingest_admission_rows_per_second 20000 +fanout_compaction_pass_seconds_sum 16 fanout_compaction_backlog_batches 13 ` snap, err := parsePrometheus(strings.NewReader(fixture)) @@ -119,6 +126,10 @@ fanout_compaction_backlog_batches 13 PublishWaitSecondsTotal: 11.5, IngestBackpressureSecondsTotal: 12.5, CompactionBacklogBatches: 13, + IngestThrottledTotal: map[string]float64{"debt": 14, "reads": 15}, + CompactionRetiredBatchesTotal: 128, + IngestAdmissionRowsPerSecond: 20000, + CompactionPassSecondsTotal: 16, } if !reflect.DeepEqual(got, want) { t.Fatalf("sample=%+v; want %+v", got, want) @@ -190,7 +201,7 @@ func TestRunPhaseStopsQueriesBeforeFinalGeneratorCPURead(t *testing.T) { Config: Config{GeneratorSSH: "unused", MetricsURL: "http://bench/metrics", QueryURL: "http://bench", SampleInterval: time.Hour}, queryTargets: newQueryRun().targets, client: &http.Client{Transport: releaseTransport(func(req *http.Request) (*http.Response, error) { - body := `{"results":[{"status":"empty"}]}` + body := `{"status":"empty"}` if req.URL.Path == "/metrics" { scrapes++ if scrapes == 2 { @@ -413,7 +424,7 @@ func TestSecretRedactionAndBoundedOutput(t *testing.T) { } func TestBestMixedTopologyUsesAcceptedThroughput(t *testing.T) { - trials := []Trial{{Spec: PhaseSpec{Signal: Mixed, Kind: "sweep", ProcessesPerSignal: 8}, AcceptedPerSecond: 100}, {Spec: PhaseSpec{Signal: Mixed, Kind: "sweep", ProcessesPerSignal: 4}, AcceptedPerSecond: 200}} + trials := []Trial{{Spec: PhaseSpec{Signal: Mixed, Kind: "sweep", ProcessesPerSignal: 8}, Accepted: Counts{Spans: 1}, AcceptedPerSecond: 100}, {Spec: PhaseSpec{Signal: Mixed, Kind: "sweep", ProcessesPerSignal: 4}, Accepted: Counts{Spans: 1}, AcceptedPerSecond: 200}} if got := bestMixedTopology(trials); got != 4 { t.Fatalf("got %d want 4", got) } @@ -421,8 +432,8 @@ func TestBestMixedTopologyUsesAcceptedThroughput(t *testing.T) { func TestBestMixedTopologyRequiresRequestedQueryLoad(t *testing.T) { trials := []Trial{ - {Spec: PhaseSpec{Signal: Mixed, Kind: "sweep", ProcessesPerSignal: 8, QueryRate: 100}, ElapsedSeconds: 10, AcceptedPerSecond: 200, QueryLatency: Latency{Count: 800}, QueryShed: 10}, - {Spec: PhaseSpec{Signal: Mixed, Kind: "sweep", ProcessesPerSignal: 4, QueryRate: 100}, ElapsedSeconds: 10, AcceptedPerSecond: 150, QueryLatency: Latency{Count: 980}}, + {Spec: PhaseSpec{Signal: Mixed, Kind: "sweep", ProcessesPerSignal: 8, QueryRate: 100}, ElapsedSeconds: 10, Accepted: Counts{Spans: 1}, AcceptedPerSecond: 200, QueryLatency: Latency{Count: 800}, QueryShed: 10}, + {Spec: PhaseSpec{Signal: Mixed, Kind: "sweep", ProcessesPerSignal: 4, QueryRate: 100}, ElapsedSeconds: 10, Accepted: Counts{Spans: 1}, AcceptedPerSecond: 150, QueryLatency: Latency{Count: 980}}, } if got := bestMixedTopology(trials); got != 4 { t.Fatalf("got %d want 4", got) @@ -431,8 +442,8 @@ func TestBestMixedTopologyRequiresRequestedQueryLoad(t *testing.T) { func TestBestMixedTopologyFallsBackToBestReadCompletion(t *testing.T) { trials := []Trial{ - {Spec: PhaseSpec{Signal: Mixed, Kind: "sweep", ProcessesPerSignal: 8, QueryRate: 100}, ElapsedSeconds: 10, AcceptedPerSecond: 200, QueryLatency: Latency{Count: 100}}, - {Spec: PhaseSpec{Signal: Mixed, Kind: "sweep", ProcessesPerSignal: 1, QueryRate: 100}, ElapsedSeconds: 10, AcceptedPerSecond: 50, QueryLatency: Latency{Count: 500}}, + {Spec: PhaseSpec{Signal: Mixed, Kind: "sweep", ProcessesPerSignal: 8, QueryRate: 100}, ElapsedSeconds: 10, Accepted: Counts{Spans: 1}, AcceptedPerSecond: 200, QueryLatency: Latency{Count: 100}}, + {Spec: PhaseSpec{Signal: Mixed, Kind: "sweep", ProcessesPerSignal: 1, QueryRate: 100}, ElapsedSeconds: 10, Accepted: Counts{Spans: 1}, AcceptedPerSecond: 50, QueryLatency: Latency{Count: 500}}, } if got := bestMixedTopology(trials); got != 1 { t.Fatalf("got %d want 1", got) @@ -470,3 +481,59 @@ func TestAbandonKeepsCompletedPhasesAndMarksReportFailed(t *testing.T) { t.Fatalf("gate detail leaked the ingest token: %q", reason) } } + +func TestGeneratorProcessDetailIncludesFirstFatalLine(t *testing.T) { + trials := []Trial{{ClassificationNotes: []string{"telemetrygen: startup\nFATAL failed to export batch: DeadlineExceeded\nFATAL later"}}} + got := generatorProcessDetail(false, trials) + if got != "at least one failed; FATAL failed to export batch: DeadlineExceeded" { + t.Fatal(got) + } + if got := generatorProcessDetail(true, trials); got != "all exited successfully" { + t.Fatal(got) + } +} + +func TestRetriedThrottlingDoesNotInvalidateGenerator(t *testing.T) { + for _, tc := range []struct { + name, output string + exit int + }{ + {"retried", "ResourceExhausted: retrying export\nexported batch", 0}, + {"dead", "FATAL failed to export batch: exporter export timeout: ResourceExhausted", 1}, + } { + t.Run(tc.name, func(t *testing.T) { + path := filepath.Join(t.TempDir(), "telemetrygen") + script := fmt.Sprintf("#!/bin/sh\nprintf '%%s\\n' '%s'\nexit %d\n", tc.output, tc.exit) + if err := os.WriteFile(path, []byte(script), 0700); err != nil { + t.Fatal(err) + } + r := Runner{Config: Config{Telemetrygen: path}} + err := r.runGenerator(t.Context(), Logs, PhaseSpec{Duration: time.Second}) + if (err != nil) != (tc.exit != 0) { + t.Fatalf("exit=%d err=%v", tc.exit, err) + } + }) + } +} + +func TestThrottledTrialRemainsValidButExporterDeathDoesNot(t *testing.T) { + trial := Trial{Spec: PhaseSpec{Signal: Mixed, Kind: "sweep", ProcessesPerSignal: 8}, + Accepted: Counts{Spans: 100}, AcceptedPerSecond: 100, AcceptanceRatio: .1, + Samples: []ResourceSample{{IngestThrottledTotal: map[string]float64{"reads": 1000}}}} + if class, _ := classify(trial, nil); class == "invalid" { + t.Fatal("retryable throttling invalidated accepted counters") + } + if got := bestMixedTopology([]Trial{trial}); got != 8 { + t.Fatalf("topology=%d", got) + } + for _, change := range []func(*Trial){func(t *Trial) { t.GeneratorFailures = 1 }, func(t *Trial) { t.RowsDropped = 1 }, func(t *Trial) { t.ProcessRestarted = true }, func(t *Trial) { t.Accepted = Counts{} }} { + failed := trial + change(&failed) + if class, _ := classify(failed, nil); class != "invalid" { + t.Fatal("real failure hidden") + } + if bestMixedTopology([]Trial{failed}) != 0 { + t.Fatal("invalid evidence selected for sustained phase") + } + } +} diff --git a/internal/bench/sweep.go b/internal/bench/sweep.go new file mode 100644 index 0000000..0b18f34 --- /dev/null +++ b/internal/bench/sweep.go @@ -0,0 +1,41 @@ +package bench + +// sweepStop only compares successful points in one signal/repetition. Invalid +// evidence must remain visible to the quality gates, never justify a plateau. +func sweepStop(remaining []PhaseSpec, trials []Trial) *SweepStop { + if len(trials) == 0 { + return nil + } + current := trials[len(trials)-1] + if current.Spec.Kind != "sweep" || !validSweepPoint(current) { + return nil + } + previous := previousSweepTrial(current.Spec, trials) + if previous == nil || !validSweepPoint(*previous) { + return nil + } + reason := "" + if current.AcceptanceRatio < .5 && current.AcceptanceRatio < .5*previous.AcceptanceRatio { + reason = "acceptance ratio collapsed below 50% and below half the previous point" + } else if before := previousSweepTrial(previous.Spec, trials); before != nil && validSweepPoint(*before) && + current.AcceptedPerSecond < 1.05*previous.AcceptedPerSecond && previous.AcceptedPerSecond < 1.05*before.AcceptedPerSecond { + reason = "two consecutive accepted-throughput gains below 5%" + } + if reason == "" { + return nil + } + stop := &SweepStop{AfterPhase: current.Spec.Name, Reason: reason} + for _, spec := range remaining { + if spec.Kind == "sweep" && spec.Signal == current.Spec.Signal && spec.Repetition == current.Spec.Repetition { + stop.SkippedPhases = append(stop.SkippedPhases, spec.Name) + } + } + if len(stop.SkippedPhases) == 0 { + return nil + } + return stop +} + +func validSweepPoint(t Trial) bool { + return t.Classification != "invalid" && t.GeneratorFailures == 0 && t.RowsDropped == 0 && !t.ProcessRestarted && t.Accepted.Total() > 0 && t.AcceptedPerSecond > 0 +} diff --git a/internal/bench/sweep_test.go b/internal/bench/sweep_test.go new file mode 100644 index 0000000..2edd779 --- /dev/null +++ b/internal/bench/sweep_test.go @@ -0,0 +1,74 @@ +package bench + +import ( + "strings" + "testing" +) + +func TestSweepStop(t *testing.T) { + point := func(p int, rate, ratio float64) Trial { + return Trial{Spec: PhaseSpec{Name: "point", Kind: "sweep", Signal: Mixed, Repetition: 1, ProcessesPerSignal: p}, Accepted: Counts{Spans: rate}, AcceptedPerSecond: rate, AcceptanceRatio: ratio} + } + remaining := []PhaseSpec{{Name: "skip", Kind: "sweep", Signal: Mixed, Repetition: 1, ProcessesPerSignal: 8}, {Name: "next-repeat", Kind: "sweep", Signal: Mixed, Repetition: 2}, {Name: "logs", Kind: "sweep", Signal: Logs, Repetition: 1}, {Name: "sustained", Kind: "sustained", Signal: Mixed}} + for _, tc := range []struct { + name string + points []Trial + want string + }{ + {"one point", []Trial{point(1, 100, 1)}, ""}, + {"one flat gain", []Trial{point(1, 100, 1), point(2, 104, .6)}, ""}, + {"plateau", []Trial{point(1, 100, 1), point(2, 104, .6), point(4, 108, .4)}, "two consecutive"}, + {"decline", []Trial{point(1, 100, 1), point(2, 99, .9), point(4, 90, .8)}, "two consecutive"}, + {"exact five percent", []Trial{point(1, 100, 1), point(2, 105, .6), point(4, 108, .4)}, ""}, + {"recovery", []Trial{point(1, 100, 1), point(2, 104, .6), point(4, 120, .4)}, ""}, + {"collapse", []Trial{point(1, 100, 1), point(2, 110, .4)}, "acceptance ratio"}, + {"half boundary", []Trial{point(1, 100, 1), point(2, 110, .5)}, ""}, + } { + t.Run(tc.name, func(t *testing.T) { + stop := sweepStop(remaining, tc.points) + if tc.want == "" { + if stop != nil { + t.Fatal(stop) + } + return + } + if stop == nil || !strings.Contains(stop.Reason, tc.want) || len(stop.SkippedPhases) != 1 || stop.SkippedPhases[0] != "skip" { + t.Fatal(stop) + } + if sweepStop(nil, tc.points) != nil { + t.Fatal("recorded stop without skipped work") + } + }) + } + for _, invalid := range []func(*Trial){func(t *Trial) { t.GeneratorFailures = 1 }, func(t *Trial) { t.RowsDropped = 1 }, func(t *Trial) { t.ProcessRestarted = true }, func(t *Trial) { t.Accepted = Counts{} }, func(t *Trial) { t.Classification = "invalid" }} { + points := []Trial{point(1, 100, 1), point(2, 101, .6), point(4, 102, .4)} + invalid(&points[1]) + if sweepStop(remaining, points) != nil { + t.Fatal("invalid point justified stop") + } + } + points := []Trial{point(1, 100, 1), point(2, 101, .6), point(4, 102, .4)} + points[0].Spec.Repetition = 2 + if sweepStop(remaining, points) != nil { + t.Fatal("crossed repetition") + } + points[0].Spec.Signal = Logs + if sweepStop(remaining, points) != nil { + t.Fatal("crossed signal") + } +} + +func TestMixedCoverageAndStopReport(t *testing.T) { + cfg := Config{SignalSelection: "mixed", Preset: "validate"} + report := Report{Settings: settingsFrom(cfg), SweepStops: []SweepStop{{AfterPhase: "mixed-p4-r1", Reason: "plateau", SkippedPhases: []string{"mixed-p8-r1"}}}} + // Coverage is added by Run; stopping evidence is rendered independently. + md := Markdown(report) + for _, want := range []string{"mixed-p4-r1", "plateau", "mixed-p8-r1"} { + if !strings.Contains(md, want) { + t.Fatal(want) + } + } + if !strings.Contains(reproductionCommand(cfg), "--signals 'mixed'") { + t.Fatal("missing reproduction selection") + } +} diff --git a/internal/bench/testdata/panel-outcomes.ndjson b/internal/bench/testdata/panel-outcomes.ndjson new file mode 100644 index 0000000..55910fd --- /dev/null +++ b/internal/bench/testdata/panel-outcomes.ndjson @@ -0,0 +1,7 @@ +{"id":"requests","status":"ok","elapsed_ms":10,"cached":true,"fresh_to_ms":1000} +{"id":"traffic","status":"empty","elapsed_ms":0} +{"id":"latency","status":"error","error":"query failed","elapsed_ms":50} +{"id":"hosts","status":"error","reason":"busy","error":"Not run: other queries kept the server busy. Refresh to try again.","elapsed_ms":0} +{"id":"containers","status":"error","error":"The query took longer than 10 seconds. Narrow the time range or add filters.","elapsed_ms":10000} +{"id":"versions","status":"busy","elapsed_ms":1} +{"id":"endpoints","status":"timeout","elapsed_ms":12000} diff --git a/internal/bench/types.go b/internal/bench/types.go index a27ff1e..b11e2e9 100644 --- a/internal/bench/types.go +++ b/internal/bench/types.go @@ -31,6 +31,7 @@ type Config struct { SSHPort int KeepRemote bool Preset string + SignalSelection string Endpoint string MetricsURL string Insecure bool @@ -41,6 +42,7 @@ type Config struct { FanoutVersion string FanoutCommit string FanoutArtifact string + FanoutBinary string DataDir string SampleInterval time.Duration GeneratorTimeout time.Duration @@ -107,6 +109,8 @@ type Build struct { } type Settings struct { + Signals string `json:"signals,omitempty"` + EarlyStop bool `json:"early_stop,omitempty"` Preset string `json:"preset"` Endpoint string `json:"endpoint"` MetricsURL string `json:"metrics_url"` @@ -144,6 +148,7 @@ type ResourceSample struct { ParquetBytes float64 `json:"parquet_bytes"` PublishWaiters float64 `json:"publish_waiters"` PublishTimeouts float64 `json:"publish_timeouts_total"` + PanelMetrics PanelMetrics `json:"panel_metrics,omitempty"` IngestRowsTotal map[string]float64 `json:"ingest_rows_total,omitempty"` RollupSeconds map[string]float64 `json:"rollup_seconds,omitempty"` WriteGateHoldSeconds map[string]float64 `json:"write_gate_hold_seconds,omitempty"` @@ -152,6 +157,10 @@ type ResourceSample struct { CompactionsTotal float64 `json:"compactions_total,omitempty"` PublishWaitSecondsTotal float64 `json:"publish_wait_seconds_total,omitempty"` IngestBackpressureSecondsTotal float64 `json:"ingest_backpressure_seconds_total,omitempty"` + IngestThrottledTotal map[string]float64 `json:"ingest_throttled_total,omitempty"` + IngestAdmissionRowsPerSecond float64 `json:"ingest_admission_rows_per_second,omitempty"` + CompactionRetiredBatchesTotal float64 `json:"compaction_retired_batches_total,omitempty"` + CompactionPassSecondsTotal float64 `json:"compaction_pass_seconds_total,omitempty"` CompactionBacklogBatches float64 `json:"compaction_backlog_batches,omitempty"` } @@ -205,6 +214,7 @@ type Trial struct { Resources Resources `json:"resources"` QueryLatency Latency `json:"query_latency_ms,omitempty"` QueryKindCompleted map[string]int64 `json:"query_completed_by_kind,omitempty"` + DashboardPanels []DashboardPanel `json:"dashboard_panels,omitempty"` QueryAttempts int64 `json:"query_attempts,omitempty"` QueryFailures int64 `json:"query_failures,omitempty"` QueryShed int64 `json:"query_shed,omitempty"` @@ -236,16 +246,23 @@ type Summary struct { Warnings []string `json:"warnings,omitempty"` } +type SweepStop struct { + AfterPhase string `json:"after_phase"` + Reason string `json:"reason"` + SkippedPhases []string `json:"skipped_phases"` +} + type Report struct { - SchemaVersion string `json:"schema_version"` - RunID string `json:"run_id"` - StartedAt time.Time `json:"started_at"` - FinishedAt time.Time `json:"finished_at"` - Build Build `json:"build"` - Host Host `json:"host"` - Settings Settings `json:"settings"` - WorkloadSHA256 string `json:"workload_sha256"` - Trials []Trial `json:"trials"` - Summary Summary `json:"summary"` - ReproductionCommand string `json:"reproduction_command"` + SweepStops []SweepStop `json:"sweep_stops,omitempty"` + SchemaVersion string `json:"schema_version"` + RunID string `json:"run_id"` + StartedAt time.Time `json:"started_at"` + FinishedAt time.Time `json:"finished_at"` + Build Build `json:"build"` + Host Host `json:"host"` + Settings Settings `json:"settings"` + WorkloadSHA256 string `json:"workload_sha256"` + Trials []Trial `json:"trials"` + Summary Summary `json:"summary"` + ReproductionCommand string `json:"reproduction_command"` } diff --git a/scripts/hetzner.sh b/scripts/hetzner.sh index faa9660..892d2cc 100755 --- a/scripts/hetzner.sh +++ b/scripts/hetzner.sh @@ -1,9 +1,15 @@ #!/usr/bin/env bash set -euo pipefail +parallel_groups=false +if [[ ${1:-} == --parallel-groups ]]; then + parallel_groups=true + shift +fi + if [[ $# -eq 0 ]]; then - echo "Usage: scripts/hetzner.sh [-- ]…" >&2 - echo "Each nonempty group shares the Fanout and generator VMs; SSH port and identity must agree across groups." >&2 + echo "Usage: scripts/hetzner.sh [--parallel-groups] [-- ]…" >&2 + echo "Groups share one pair by default; --parallel-groups gives each its own pair. SSH settings must agree." >&2 exit 2 fi @@ -86,7 +92,81 @@ done repo_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd) cd "$repo_dir" -just build +# Parallel children reuse the binary built by their parent. +if [[ ${FANOUT_BENCH_GROUP_CHILD:-} != 1 ]]; then + just build +fi + +if [[ $parallel_groups == true ]]; then + pair_pids=() + pair_statuses=() + launching_group=false + interrupted_status= + # Each child owns the existing single-pair EXIT cleanup, including failed + # creates. Signal all children before waiting, so every pair starts cleanup. + # shellcheck disable=SC2317,SC2329 + cleanup_groups() { + local status=$? pid + trap - EXIT + trap '' INT TERM + if [[ ${#pair_pids[@]} -gt 0 ]]; then + for pid in "${pair_pids[@]}"; do + kill -TERM "$pid" 2>/dev/null || true + done + for pid in "${pair_pids[@]}"; do + wait "$pid" 2>/dev/null || true + done + fi + exit "$status" + } + trap cleanup_groups EXIT + # Defer a signal across fork/PID registration so no child is untracked. + # shellcheck disable=SC2317,SC2329 + interrupt_groups() { + if [[ $launching_group == true ]]; then + interrupted_status=$1 + else + exit "$1" + fi + } + trap 'interrupt_groups 130' INT + trap 'interrupt_groups 143' TERM + parallel_output="results/parallel-$(date -u +%Y%m%dT%H%M%SZ)-$(od -An -N4 -tx1 /dev/urandom | tr -d ' \n')" + offset=1 + group_index=0 + for group_length in "${group_lengths[@]}"; do + group_index=$((group_index + 1)) + child_args=("${@:offset:group_length}") + has_output=false + for arg in "${child_args[@]}"; do + case "$arg" in --output|-output|--output=*|-output=*) has_output=true ;; esac + done + if [[ $has_output == false ]]; then + child_args+=(--output "$parallel_output/group-$group_index") + fi + launching_group=true + FANOUT_BENCH_GROUP_CHILD=1 FANOUT_BENCH_GROUP_INDEX=$group_index \ + bash "$repo_dir/scripts/hetzner.sh" "${child_args[@]}" <&0 & + pair_pids+=("$!") + launching_group=false + if [[ -n $interrupted_status ]]; then exit "$interrupted_status"; fi + offset=$((offset + group_length + 1)) + done + status=0 + for index in "${!pair_pids[@]}"; do + child_status=0 + wait "${pair_pids[index]}" || child_status=$? + unset 'pair_pids[index]' + pair_statuses+=("$child_status") + if [[ $child_status -ne 0 ]]; then status=1; fi + done + for index in "${!pair_statuses[@]}"; do + printf 'Parallel group %d/%d: exit %d (includes pair cleanup)\n' "$((index + 1))" "$group_count" "${pair_statuses[index]}" + done + # All children were reaped, including their EXIT traps; avoid PID reuse. + trap - EXIT INT TERM + exit "$status" +fi key_args=() IFS=',' read -r -a keys <<< "$FANOUT_BENCH_HCLOUD_SSH_KEY" @@ -105,6 +185,7 @@ nonce=$(od -An -N4 -tx1 /dev/urandom | tr -d ' \n') name="fanout-bench-$(date -u +%Y%m%dT%H%M%SZ)-$nonce" fanout_name="$name-fanout" generator_name="$name-generator" +printf 'Hetzner pair: group=%s Fanout=%s generator=%s network=%s\n' "${FANOUT_BENCH_GROUP_INDEX:-shared}" "$fanout_name" "$generator_name" "$name" network_attempted=false fanout_attempted=false generator_attempted=false