Repository navigation
feat: Explore, FanQL and batch format 5 - #417
Merged
Merged
Conversation
On a phone the squeezed tables cut state badges to "FIR…"; badge labels no longer shrink. The history's horizontal scroll area had nothing focusable, so it is now a named, focusable region. Hovering a rule row tinted it under dimmed descriptions to 4.3:1; the rows do nothing on hover, so the tint is gone.
CodeMirror's placeholder grey is 3.5:1 on white, so New dashboard's query example failed contrast. The shared editor theme now draws it in the grey the SQL comments use, which clears 4.5:1 on plain and active lines in both schemes.
Span files sort by trace hash, so every row group covers its file's whole time range and DuckDB decodes the attributes VARIANT of every row in any file a window touches, whatever the time filter selects. Compaction grouped by ingest day, so after a few generations a one-minute attribute filter over older data decoded a multi-hour file. Measured on generated files at 2 threads, a one-minute attributes['http.route'] filter costs CPU in proportion to the file: 0.17 s at 54,000 spans, 3.4 s at 2.1 million, 10 s at 8.3 million, while the same filter on the scalar http_route column stays under 0.07 s. Group compaction by ingest hour instead, so the largest file a narrow window can touch holds one hour (about 325,000 spans at the pico rate) and older data costs what recent data does. A finished hour merges even a pair of files, so a quiet install keeps at most one file per hour per signal. Compaction debt reserves its underfilled group per hour.
A discovery that needed about 20 seconds never finished on a busy host: the request and the discovery shared a 20-second deadline, and the discovery was cancelled when its last caller left, so the cache never warmed. A discovery now runs on, bounded by its own one-minute deadline, the two-slot cap and the fixed key set. A request waits 15 seconds and then gets 503 schema_busy with Retry-After; the retry finds it cached. The schema route also gets its own timeout message, and an unknown window is a 400 that names the supported windows.
Discovery spent most of its time decoding VARIANT attribute maps. The sample CTE was referenced twice and inlined, so each reference reran the window scan, and every key re-parsed its row's whole JSON document. The sample is now materialized, json_each parses each map once, and the sample is 1,000 rows (spans sort by trace, so it still spans services). On a pico-like dataset (24h of 5,400 spans/min with OTel demo sized attribute maps, 2 threads, 1 GB), the attribute queries for 1h dropped from 6.6-9.1 s to 1.6-1.8 s of CPU, and a cold 1h discovery from 18-20 s to under 5 s of server CPU.
An empty panel whose attribute is stored as a number by some rows and as
text by others said only that the filter matches one type. It now tells
the user to quote the value ("308") or drop the quotes, whichever finds
the rows the filter missed.
percentile(duration, 0.95) suggested quantile(duration, 0.0095), and 99.9 suggested 0.9990000000000001: a value below 1 is now already a fraction, and the fraction is printed without float artifacts. count() group by service said to separate measures with a comma; it now says group by is not FanQL, with a fix-it to by, after a measure or a second |. A measure already reported as foreign no longer adds a false "by needs measures first". head N and take N right after the source or a filter get the limit hint and fix-it they get after a stage.
p95(duraton) silently read attributes['duraton'] and drew an empty
chart. A bare name within edit distance of a column (duration included)
still reads the attribute, but its notice now says "did you mean
duration?" and carries a fix-it. "opération" needs brackets now offers
operation and attributes['opération'] as fix-its. A variable used as a
field says so ("by cannot use the variable $x") instead of asking for
brackets. attributes.foo alone stays free text, as a word before no
operator is, with a notice that says how to filter on the field.
After a finished term, and, or, |, sort and limit were cut by the 50-item cap, and "o" never offered or. They now lead the list after a complete term (and, or inside a group). http_route, http_status_code and rpc_service take the sampled values of the attributes ingest copies them from, and operation is scanned for its common values, so these columns complete like their attributes. A key whose rows disagree on the type reports both types, and completion offers each value as a number and as quoted text, saying which rows each matches. kind, status and type describe the values people type (client, error) rather than the stored enums.
…mory Each service rollup pass recomputes every minute bucket touched by an hour of ingest. A backfill or seed packs days of event time into a few minutes of ingest, so that pass grouped millions of spans, ran out of memory at a 1 GB limit, and retried the same window every minute; after a seven-day backfill service_rollup stayed empty for the whole run and the health and service map panels showed nothing. An out-of-memory pass now halves the ingest window, down to a second, and retries at once; successes double it back to the hour. A tick drains further passes within half the rollup interval, as the edge rollup does. On the same backfill the rollup caught up five minutes after start and stopped failing.
Detached discoveries ran under the request context without its cancellation, so shutdown never stopped them and an abandoned one held a read connection for up to a minute. They now run under an executor lifetime that Close cancels and waits for, and a run is capped at 25 seconds, separate from the one-minute slot wait. The slots and bounds are per executor rather than package globals, and the cold-burst test now proves late callers joined the flight.
The window doubled after any pass without an error, including a pass that skipped because compaction held the Parquet gate, so a narrowed window grew straight back and ran out of memory again on alternate passes. It now grows only after three committed passes in a row and stays put on a pass that did no work. The startup rollup also reports a fatal engine error, as the loop does.
Compaction kept one file per ingest hour per signal for the whole retention. Hours more than a day behind the newest that have nothing left to merge within themselves now merge with adjacent hours of the same day into files of at most 250,000 rows, in time order. Files already spanning hours stay out of hour groups, and a day merge's output sits above every input generation. The bound keeps narrow queries on old data no slower than a busy hour. On a 72-hour pico-rate simulation, million-row day files cut span files from 73 to 33, but slowed a one-minute attribute filter on old data from 0.12 to 0.66 seconds and a 24-hour one from 2.5 to 3.6, while each file costs a scan about 0.1 ms. With the bound, pico keeps its 73 hourly files and a quiet install (16,000 spans an hour) holds 31 files after seven days instead of one per hour, with a one-minute attribute filter on old data at 0.14 seconds. The finished-pair test now tests a pair.
FanQL lowers -k:* to a bare k IS NULL, which negatedTerm did not recognise, so the traces list kept any trace with one span lacking k. A top-level IS NULL now excludes every trace with a span that has k. The null-inclusive form counts only when its second branch is one leaf on the field, as FanQL writes it and lifts it back; hand-written compounds and calls such as lower(k) stay span conditions.
The Traces page's link to services by error rate reused the search's filters as span filters, so status:error made every row 100% and an excluded term dropped single spans instead of whole traces. The link now formats the search's stored filters and leaves out status terms and prefix negations, and its label and tooltip say it shows the matching spans' error rate per service.
A tick that listed a rule as enabled recompiled it after DisableRule had dropped its program, and wrote a firing or pending alert for a disabled rule. Disabled and deleted rules are now kept stopped until a recompile or until a tick no longer lists them, and each rule's transitions run under the read lock, so disabling waits them out and cleans up after.
Old rows held SQLite's datetime('now') text and new ones RFC 3339 with
a variable fraction, so listing alerts newest first misordered them, and
an upsert without created_at replaced the episode start with the time
of the update. The store now writes created_at as RFC 3339 UTC with nine
fraction digits, keeps the stored start when an update leaves it empty,
and refuses a new alert without one. A migration rewrites existing
values to the same form.
The schema endpoint used invalid_request, OAuth's spelling and not one of the documented application codes, and indexed the first problem without checking there was one. It now returns bad_request with the first problem's message, or a general one when the list is empty, and the API reference says so.
Resolved times carry the engine's local offset, so comparing their text with SQLite's UTC cutoff kept alerts resolved on a later local date.
Retention removes whole files once their newest row passes the cutoff, as Prometheus, Tempo and Loki remove whole blocks. Day-merged files on quiet installs can therefore keep rows up to a day past the retention period. Describe the hour groups and settled-day merges compaction now performs.
Each metric builder scanned the metrics relation on its own, so avg(value) and sum(value) read a gauge's rows a second time for the histogram branch, and bucketed panels a third time for the raw window. Builders and the raw window now read one shared scan of the authored filters; a scan with several readers is materialized. Each builder still selects its own window and kinds from it, so deduplication, lookback and kind semantics are unchanged. A golden test pins results for every stored kind, alone and mixed.
Alerts were upserted on (rule_id, service), so firing again after a resolve overwrote the resolved row and history lost episodes. A new migration drops that unique constraint and adds a partial unique index that allows one pending or firing alert per rule and service. Firing again now inserts a new alert, upserts key on the alert id, and delivery status updates the episode that was notified.
A measure of a field the matching rows never carry returned rows of nulls with status ok, so the editor saved an empty chart. Structured span and log panels now report empty with a diagnosis that names the measure as the missing part, as metric panels already did. State timelines keep null buckets because they draw them as unknown.
A run the server rejects with problems, such as a SQL binder error, left the previous result in place, so the save button said to run the query as it is now although it had just run. The editor now records the refused run and says it failed until the query or window changes.
The restore button is disabled while it works, so focus fell to the page body. The confirmation now takes focus when the restore lands.
Close idle pressure groups only after a normal-size batch is ready. Keep one normal batch per pressure window as the band rate floor, since further cuts shrink files without reducing their frequency. Seed warm band entry from measured history across retry gaps and retain the hard debt stop. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Snapshot queued read pressure before scheduling a batch so sibling panels cannot randomly suppress the filter diagnosis. Report deferred diagnosis explicitly and keep incomplete results out of the result cache. Add controlled coverage for sibling queuing, pressure that drains before execution, and diagnosis recovery after pressure clears. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Optional cache transfers waited behind long analytical and read-cache passes before the bounded Parquet swap even began. Use idle leases for transfers and let refresh workers rebuild cold outputs when those leases are busy. Preserve the durable publication and epoch retirement paths. Add coverage for repeated publication with pinned warm and cold inputs while analytical or read-cache leases remain held, then verify cache rebuilding and exact row counts after the reader finishes. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Skip messaging attribute scans for empty messaging windows and history-wide child lookup for empty peer candidates. Preserve the complete relationship queries for nonempty scopes and reuse materialized peer candidates. Defer contended compaction cache transfers to the cache worker, preserving complete input contributions without delaying catalog publication. Cover warm transfers across compaction generations and empty SQL scan paths. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Add log and bounded endpoint rollups, cache resource minutes for exact boundary reads, and skip irrelevant metric files using complete batch name summaries. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Record executed panel duration by the fixed overview panel vocabulary and status; group custom IDs to prevent unbounded labels. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Use the dedicated HTTP route column so endpoint summaries match converted raw values and Parquet can skip route-free files. Cache resource observations by second to keep exact boundary scans small. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Replace correlated map-list percentile queries with grouped bins and one cumulative distribution shared by all quantiles. Preserve totals, top-N folding and sample-based ranking; refresh generated response fixtures. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Keep the pass bounded while processing enough rows to catch up under sustained ingestion and read pressure. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Measure bounded panel admission waits and rejected, canceled, or timed out attempts separately from SQL execution. Keep custom IDs and unknown outcomes in fixed label sets. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Share analytical invalidation and freshness across check, host, container and version panels. Preserve retry ordering, counter resets, reporting cadence and exact inventory semantics while avoiding raw boundary scans and ASOF expansion. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Queue refreshes as bounded FIFO units, cap sibling execution and yield idle leases while preserving interactive capacity and detached-flight cleanup. Reuse validated SQL templates with fresh variables and snapshots on every execution. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Read container IDs and service versions from their authoritative dedicated columns. Rebuild disposable summaries and cover unsupported resource values without changing the sent attribute maps. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Read cadence from only the latest two reporting minutes and carry scalar finite CPU boundaries through window operators. Preserve complete selected-window counter totals and bridge nonfinite minutes. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Collapse repeated raw resources before extracting inventory fields, then regroup projected identities to preserve exact bounds and converted version values. Cover unrelated resource keys and benchmark repeated metric resources. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Decode each new batch resource once into a bounded temporary aggregate, then derive inventory extrema without scanning retained cache contributions. Benchmark the completed metric batch cache separately from publication. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
A refresh could be admitted and then have individual panels refused as busy, failing the refresh even though every panel was fast. A refresh now takes one lifetime ticket before emitting any result, and its panels run under a local parallelism cap on individual global slots, so an admitted refresh always completes. Cache hits and coalesced runs hold no connections. Interactive reads keep priority but leave panels enough slots to make progress. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Rename the overview and insights tools and public read routes, preserve snapshot freshness, and wrap unpaginated collections in named arrays. Update clients, guidance, fixtures, tests, and generated references together. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Rebuild the host assets with the current tool names, annotations query route, and alert collection envelopes. Closes #416 Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
Dashboard panels now keep one slot for progress instead of two, so drill-downs and trace detail regain priority at moderate load. Read-pressure throttling refuses an exporter for at most one second, then opens a ten-second window at the debt rate. Default OTLP first retries (2.5 to 7.5 s) land inside it, so a throttled exporter never exhausts its export timeout. Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet
…ot against a wall clock
vishr
added a commit
that referenced
this pull request
Nov 6, 2026
…and the CLI (#421) The site lagged behind what shipped in #410 and #417 and earlier: some pages described missing UI as absent, and several user tasks had no page. Corrected - Alerts: the Alerts page creates, tests and toggles rules; webhook payload, template fields, delivery timeouts and retries are documented. - Dashboards: building and editing panels without chat; HTTP-only deletion. - Explore: Query, Builder and SQL tabs; Logs and Traces search. - Backup and restore: upgrade steps moved to a dedicated page, including the batch-format change procedure. - Ingest token rotation, OTLP/HTTP JSON rejection (415), ingest throttling, user management over HTTP, MCP setup for Codex, Docker forms of commands, current version examples. Added - Use the built-in agent, Upgrade, Run with Compose or systemd, Harden an instance, CLI reference, Insights and anomalies, How Fanout compares, and Get help and contribute. - Shipped vs planned rows for Explore, FanQL, format 5, the Alerts page, search, connected agents, token rotation and insights. Also ignores the local .wrangler/ cache in Git and Docker. The performance page is unchanged; it moves with the next published benchmark. `just docs-generate-check` and `just site-build` pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
attributes['k']reads the dedicated column. Format 4 data is not read.RESOURCE_EXHAUSTED+ RetryInfo or HTTP429+Retry-After, instead of blocking until compaction debt clears.get_overviewandget_insights;GET /api/insightsandPOST /api/annotations/query;Benchmarks
Hetzner ccx33, generator on a separate VM, fanout-bench publish preset, 3 repetitions. Both builds ran at the same time on fresh VMs.
All quality gates passed for both builds.
Reads at 20 operations/s under saturating ingest:
Breaking changes
FANOUT_QUERY_CACHE_TTL(default 10 s); manual refresh bypasses the cache.Verification
just checkjust e2e(107 passed, 150 skipped)go test -raceacross engine, telemetry, query, ingest, api, dashboard, mcp, agent and cmd/fanoutCloses #416