Skip to content

feat: Explore, FanQL and batch format 5 - #417

Merged
vishr merged 337 commits into
mainfrom
feat/explore
Nov 5, 2026
Merged

vishr merged 337 commits into
mainfrom
feat/explore

Conversation

@vishr

@vishr vishr commented Nov 4, 2026

Copy link
Copy Markdown
Member

What changed

  • Explore: a query page with a FanQL editor and a builder kept in sync with it, plus Logs and Traces presets. State lives in the URL, and results save to dashboards. Dashboards and alert rules can be created and edited without chat.
  • Batch format 5 (ADR 0008): 47 standard semantic-convention keys get dedicated typed columns, with values converted at ingest. The attribute map keeps every value as sent, and attributes['k'] reads the dedicated column. Format 4 data is not read.
  • Reads under load:
    • Span, log, endpoint, resource, host, container and check panels read per-minute summaries.
    • Panel results are cached, and identical runs share one computation.
    • A dashboard refresh is admitted as a whole.
    • Drill-downs and trace detail have their own lane.
    • Compaction publishes without waiting for readers.
  • Ingest at saturation (ADR 0009): exports are refused before decoding, with gRPC RESOURCE_EXHAUSTED + RetryInfo or HTTP 429 + Retry-After, instead of blocking until compaction debt clears.
  • API and MCP names (refactor: clarify API and MCP names and normalize list responses #416):
    • get_overview and get_insights;
    • GET /api/insights and POST /api/annotations/query;
    • named arrays for users, alerts and rules.
  • Offline verification checks every dedicated column against its map value and is about 10× faster.

Benchmarks

Hetzner ccx33, generator on a separate VM, fanout-bench publish preset, 3 repetitions. Both builds ran at the same time on fresh VMs.

Workload v2026.10.8-alpha This branch
Mixed sustained 389,495/s 423,409/s
Mixed peak 463,313/s 459,418/s
Spans 280,894/s 279,180/s
Logs 413,420/s 521,059/s
Metric points 551,632/s 563,778/s
Peak RSS (sustained) 12.1 GiB 9.7 GiB

All quality gates passed for both builds.

Reads at 20 operations/s under saturating ingest:

  • 19.9/s completed, with no reads turned away.
  • System overview refresh: 99.9% completed, p95 2.0 s.
  • Drill-down: 99.4% completed. Trace detail: 100% completed.
  • v2026.10.8-alpha completed no reads under the same load.

Breaking changes

  • Batch format 5 is the only accepted telemetry format. Format 4 data directories must be moved aside.
  • Renamed MCP tools and HTTP routes, and list envelopes, with no aliases (refactor: clarify API and MCP names and normalize list responses #416).
  • Panel results are served from cache for up to FANOUT_QUERY_CACHE_TTL (default 10 s); manual refresh bypasses the cache.
  • Routed panels use per-minute summaries: they lag about one minute, and quantiles are approximate to within one histogram bucket.
  • Saturated ingest returns retryable throttling instead of blocking.

Verification

  • just check
  • just e2e (107 passed, 150 skipped)
  • go test -race across engine, telemetry, query, ingest, api, dashboard, mcp, agent and cmd/fanout
  • Hetzner same-day comparison with v2026.10.8-alpha (above)
  • Hetzner read-under-write at saturation (above)

Closes #416

vishr added 30 commits October 31, 2026 12:00
On a phone the squeezed tables cut state badges to "FIR…"; badge labels no
longer shrink. The history's horizontal scroll area had nothing focusable,
so it is now a named, focusable region. Hovering a rule row tinted it under
dimmed descriptions to 4.3:1; the rows do nothing on hover, so the tint is
gone.
CodeMirror's placeholder grey is 3.5:1 on white, so New dashboard's query
example failed contrast. The shared editor theme now draws it in the grey
the SQL comments use, which clears 4.5:1 on plain and active lines in both
schemes.
Span files sort by trace hash, so every row group covers its file's
whole time range and DuckDB decodes the attributes VARIANT of every row
in any file a window touches, whatever the time filter selects.
Compaction grouped by ingest day, so after a few generations a
one-minute attribute filter over older data decoded a multi-hour file.
Measured on generated files at 2 threads, a one-minute
attributes['http.route'] filter costs CPU in proportion to the file:
0.17 s at 54,000 spans, 3.4 s at 2.1 million, 10 s at 8.3 million, while
the same filter on the scalar http_route column stays under 0.07 s.

Group compaction by ingest hour instead, so the largest file a narrow
window can touch holds one hour (about 325,000 spans at the pico rate)
and older data costs what recent data does. A finished hour merges even
a pair of files, so a quiet install keeps at most one file per hour per
signal. Compaction debt reserves its underfilled group per hour.
A discovery that needed about 20 seconds never finished on a busy host:
the request and the discovery shared a 20-second deadline, and the
discovery was cancelled when its last caller left, so the cache never
warmed. A discovery now runs on, bounded by its own one-minute deadline,
the two-slot cap and the fixed key set. A request waits 15 seconds and
then gets 503 schema_busy with Retry-After; the retry finds it cached.

The schema route also gets its own timeout message, and an unknown
window is a 400 that names the supported windows.
Discovery spent most of its time decoding VARIANT attribute maps. The
sample CTE was referenced twice and inlined, so each reference reran the
window scan, and every key re-parsed its row's whole JSON document. The
sample is now materialized, json_each parses each map once, and the
sample is 1,000 rows (spans sort by trace, so it still spans services).

On a pico-like dataset (24h of 5,400 spans/min with OTel demo sized
attribute maps, 2 threads, 1 GB), the attribute queries for 1h dropped
from 6.6-9.1 s to 1.6-1.8 s of CPU, and a cold 1h discovery from 18-20 s
to under 5 s of server CPU.
An empty panel whose attribute is stored as a number by some rows and as
text by others said only that the filter matches one type. It now tells
the user to quote the value ("308") or drop the quotes, whichever finds
the rows the filter missed.
percentile(duration, 0.95) suggested quantile(duration, 0.0095), and
99.9 suggested 0.9990000000000001: a value below 1 is now already a
fraction, and the fraction is printed without float artifacts.
count() group by service said to separate measures with a comma; it now
says group by is not FanQL, with a fix-it to by, after a measure or a
second |. A measure already reported as foreign no longer adds a false
"by needs measures first". head N and take N right after the source or
a filter get the limit hint and fix-it they get after a stage.
p95(duraton) silently read attributes['duraton'] and drew an empty
chart. A bare name within edit distance of a column (duration included)
still reads the attribute, but its notice now says "did you mean
duration?" and carries a fix-it. "opération" needs brackets now offers
operation and attributes['opération'] as fix-its. A variable used as a
field says so ("by cannot use the variable $x") instead of asking for
brackets. attributes.foo alone stays free text, as a word before no
operator is, with a notice that says how to filter on the field.
After a finished term, and, or, |, sort and limit were cut by the
50-item cap, and "o" never offered or. They now lead the list after a
complete term (and, or inside a group).

http_route, http_status_code and rpc_service take the sampled values of
the attributes ingest copies them from, and operation is scanned for
its common values, so these columns complete like their attributes. A
key whose rows disagree on the type reports both types, and completion
offers each value as a number and as quoted text, saying which rows each
matches. kind, status and type describe the values people type (client,
error) rather than the stored enums.
…mory

Each service rollup pass recomputes every minute bucket touched by an
hour of ingest. A backfill or seed packs days of event time into a few
minutes of ingest, so that pass grouped millions of spans, ran out of
memory at a 1 GB limit, and retried the same window every minute; after
a seven-day backfill service_rollup stayed empty for the whole run and
the health and service map panels showed nothing.

An out-of-memory pass now halves the ingest window, down to a second,
and retries at once; successes double it back to the hour. A tick
drains further passes within half the rollup interval, as the edge
rollup does. On the same backfill the rollup caught up five minutes
after start and stopped failing.
Detached discoveries ran under the request context without its
cancellation, so shutdown never stopped them and an abandoned one held a
read connection for up to a minute. They now run under an executor
lifetime that Close cancels and waits for, and a run is capped at 25
seconds, separate from the one-minute slot wait. The slots and bounds
are per executor rather than package globals, and the cold-burst test
now proves late callers joined the flight.
The window doubled after any pass without an error, including a pass
that skipped because compaction held the Parquet gate, so a narrowed
window grew straight back and ran out of memory again on alternate
passes. It now grows only after three committed passes in a row and
stays put on a pass that did no work. The startup rollup also reports
a fatal engine error, as the loop does.
Compaction kept one file per ingest hour per signal for the whole
retention. Hours more than a day behind the newest that have nothing
left to merge within themselves now merge with adjacent hours of the
same day into files of at most 250,000 rows, in time order. Files
already spanning hours stay out of hour groups, and a day merge's
output sits above every input generation.

The bound keeps narrow queries on old data no slower than a busy hour.
On a 72-hour pico-rate simulation, million-row day files cut span files
from 73 to 33, but slowed a one-minute attribute filter on old data from
0.12 to 0.66 seconds and a 24-hour one from 2.5 to 3.6, while each file
costs a scan about 0.1 ms. With the bound, pico keeps its 73 hourly
files and a quiet install (16,000 spans an hour) holds 31 files after
seven days instead of one per hour, with a one-minute attribute filter
on old data at 0.14 seconds.

The finished-pair test now tests a pair.
FanQL lowers -k:* to a bare k IS NULL, which negatedTerm did not
recognise, so the traces list kept any trace with one span lacking k.
A top-level IS NULL now excludes every trace with a span that has k.
The null-inclusive form counts only when its second branch is one leaf
on the field, as FanQL writes it and lifts it back; hand-written
compounds and calls such as lower(k) stay span conditions.
The Traces page's link to services by error rate reused the search's
filters as span filters, so status:error made every row 100% and an
excluded term dropped single spans instead of whole traces. The link now
formats the search's stored filters and leaves out status terms and
prefix negations, and its label and tooltip say it shows the matching
spans' error rate per service.
A tick that listed a rule as enabled recompiled it after DisableRule
had dropped its program, and wrote a firing or pending alert for a
disabled rule. Disabled and deleted rules are now kept stopped until a
recompile or until a tick no longer lists them, and each rule's
transitions run under the read lock, so disabling waits them out and
cleans up after.
Old rows held SQLite's datetime('now') text and new ones RFC 3339 with
a variable fraction, so listing alerts newest first misordered them, and
an upsert without created_at replaced the episode start with the time
of the update. The store now writes created_at as RFC 3339 UTC with nine
fraction digits, keeps the stored start when an update leaves it empty,
and refuses a new alert without one. A migration rewrites existing
values to the same form.
The schema endpoint used invalid_request, OAuth's spelling and not one
of the documented application codes, and indexed the first problem
without checking there was one. It now returns bad_request with the
first problem's message, or a general one when the list is empty, and
the API reference says so.
Resolved times carry the engine's local offset, so comparing their text
with SQLite's UTC cutoff kept alerts resolved on a later local date.
Retention removes whole files once their newest row passes the cutoff,
as Prometheus, Tempo and Loki remove whole blocks. Day-merged files on
quiet installs can therefore keep rows up to a day past the retention
period. Describe the hour groups and settled-day merges compaction now
performs.
Each metric builder scanned the metrics relation on its own, so avg(value)
and sum(value) read a gauge's rows a second time for the histogram branch,
and bucketed panels a third time for the raw window. Builders and the raw
window now read one shared scan of the authored filters; a scan with several
readers is materialized. Each builder still selects its own window and kinds
from it, so deduplication, lookback and kind semantics are unchanged.

A golden test pins results for every stored kind, alone and mixed.
Alerts were upserted on (rule_id, service), so firing again after a
resolve overwrote the resolved row and history lost episodes. A new
migration drops that unique constraint and adds a partial unique index
that allows one pending or firing alert per rule and service. Firing
again now inserts a new alert, upserts key on the alert id, and
delivery status updates the episode that was notified.
A measure of a field the matching rows never carry returned rows of
nulls with status ok, so the editor saved an empty chart. Structured
span and log panels now report empty with a diagnosis that names the
measure as the missing part, as metric panels already did. State
timelines keep null buckets because they draw them as unknown.
A run the server rejects with problems, such as a SQL binder error,
left the previous result in place, so the save button said to run the
query as it is now although it had just run. The editor now records
the refused run and says it failed until the query or window changes.
The restore button is disabled while it works, so focus fell to the
page body. The confirmation now takes focus when the restore lands.
vishr added 20 commits November 3, 2026 08:50
Close idle pressure groups only after a normal-size batch is ready. Keep
one normal batch per pressure window as the band rate floor, since further
cuts shrink files without reducing their frequency. Seed warm band entry
from measured history across retry gaps and retain the hard debt stop.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Snapshot queued read pressure before scheduling a batch so sibling panels
cannot randomly suppress the filter diagnosis. Report deferred diagnosis
explicitly and keep incomplete results out of the result cache.

Add controlled coverage for sibling queuing, pressure that drains before
execution, and diagnosis recovery after pressure clears.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Optional cache transfers waited behind long analytical and read-cache passes
before the bounded Parquet swap even began. Use idle leases for transfers
and let refresh workers rebuild cold outputs when those leases are busy.
Preserve the durable publication and epoch retirement paths.

Add coverage for repeated publication with pinned warm and cold inputs while
analytical or read-cache leases remain held, then verify cache rebuilding
and exact row counts after the reader finishes.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Skip messaging attribute scans for empty messaging windows and history-wide
child lookup for empty peer candidates. Preserve the complete relationship
queries for nonempty scopes and reuse materialized peer candidates.

Defer contended compaction cache transfers to the cache worker, preserving
complete input contributions without delaying catalog publication. Cover
warm transfers across compaction generations and empty SQL scan paths.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Add log and bounded endpoint rollups, cache resource minutes for exact boundary reads, and skip irrelevant metric files using complete batch name summaries.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Record executed panel duration by the fixed overview panel vocabulary and status; group custom IDs to prevent unbounded labels.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Use the dedicated HTTP route column so endpoint summaries match converted raw values and Parquet can skip route-free files. Cache resource observations by second to keep exact boundary scans small.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Replace correlated map-list percentile queries with grouped bins and one cumulative distribution shared by all quantiles. Preserve totals, top-N folding and sample-based ranking; refresh generated response fixtures.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Keep the pass bounded while processing enough rows to catch up under sustained ingestion and read pressure.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Measure bounded panel admission waits and rejected, canceled, or timed out attempts separately from SQL execution. Keep custom IDs and unknown outcomes in fixed label sets.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Share analytical invalidation and freshness across check, host, container and version panels. Preserve retry ordering, counter resets, reporting cadence and exact inventory semantics while avoiding raw boundary scans and ASOF expansion.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Queue refreshes as bounded FIFO units, cap sibling execution and yield idle leases while preserving interactive capacity and detached-flight cleanup. Reuse validated SQL templates with fresh variables and snapshots on every execution.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Read container IDs and service versions from their authoritative dedicated columns. Rebuild disposable summaries and cover unsupported resource values without changing the sent attribute maps.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Read cadence from only the latest two reporting minutes and carry scalar finite CPU boundaries through window operators. Preserve complete selected-window counter totals and bridge nonfinite minutes.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Collapse repeated raw resources before extracting inventory fields, then regroup projected identities to preserve exact bounds and converted version values. Cover unrelated resource keys and benchmark repeated metric resources.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Decode each new batch resource once into a bounded temporary aggregate, then derive inventory extrema without scanning retained cache contributions. Benchmark the completed metric batch cache separately from publication.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
A refresh could be admitted and then have individual panels refused as
busy, failing the refresh even though every panel was fast. A refresh
now takes one lifetime ticket before emitting any result, and its
panels run under a local parallelism cap on individual global slots,
so an admitted refresh always completes. Cache hits and coalesced runs
hold no connections. Interactive reads keep priority but leave panels
enough slots to make progress.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Rename the overview and insights tools and public read routes, preserve snapshot freshness, and wrap unpaginated collections in named arrays. Update clients, guidance, fixtures, tests, and generated references together.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Rebuild the host assets with the current tool names, annotations query route, and alert collection envelopes.

Closes #416

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Dashboard panels now keep one slot for progress instead of two, so
drill-downs and trace detail regain priority at moderate load.

Read-pressure throttling refuses an exporter for at most one second,
then opens a ten-second window at the debt rate. Default OTLP first
retries (2.5 to 7.5 s) land inside it, so a throttled exporter never
exhausts its export timeout.

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Comment thread internal/telemetry/parquet_merge.go Resolved
@vishr
vishr merged commit 4fe1cf9 into main Nov 5, 2026
17 checks passed
@vishr
vishr deleted the feat/explore branch November 5, 2026 02:36
vishr added a commit that referenced this pull request Nov 6, 2026
…and the CLI (#421)

The site lagged behind what shipped in #410 and #417 and earlier: some
pages
described missing UI as absent, and several user tasks had no page.

Corrected
- Alerts: the Alerts page creates, tests and toggles rules; webhook
payload,
  template fields, delivery timeouts and retries are documented.
- Dashboards: building and editing panels without chat; HTTP-only
deletion.
- Explore: Query, Builder and SQL tabs; Logs and Traces search.
- Backup and restore: upgrade steps moved to a dedicated page, including
the
  batch-format change procedure.
- Ingest token rotation, OTLP/HTTP JSON rejection (415), ingest
throttling,
user management over HTTP, MCP setup for Codex, Docker forms of
commands,
  current version examples.

Added
- Use the built-in agent, Upgrade, Run with Compose or systemd, Harden
an
instance, CLI reference, Insights and anomalies, How Fanout compares,
and
  Get help and contribute.
- Shipped vs planned rows for Explore, FanQL, format 5, the Alerts page,
  search, connected agents, token rotation and insights.

Also ignores the local .wrangler/ cache in Git and Docker.

The performance page is unchanged; it moves with the next published
benchmark.

`just docs-generate-check` and `just site-build` pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refactor: clarify API and MCP names and normalize list responses

1 participant