Skip to content

Replace DuckLake with authoritative Parquet storage - #207

Merged
vishr merged 31 commits into
mainfrom
feat/telemetry-segment-parquet
Aug 28, 2026
Merged

vishr merged 31 commits into
mainfrom
feat/telemetry-segment-parquet

Conversation

@vishr

@vishr vishr commented Aug 26, 2026 •

Copy link
Copy Markdown
Member

Summary

  • replace DuckLake, WAL, and segment storage with atomic Parquet batch directories
  • use the filesystem as the telemetry catalog and keep SQLite for control-plane state only
  • write spans, logs, and metrics concurrently through bounded durable commit workers
  • serve trace lookups through persistent on-disk indexes while DuckDB handles broad scans and rebuildable rollups
  • compact and retain Parquet with crash recovery, bounded query memory, and immediate commit-failure propagation
  • force every pooled DuckDB connection to UTC so rollup windows remain correct in non-UTC deployments

The exploratory storage comparisons and benchmark POCs remain local under the ignored experiments/ directory.

Breaking change

Existing DuckLake, WAL, and segment telemetry is not migrated. Deployments must start with a clean storage.data_dir. No compatibility or fallback storage path is included.

Verification

  • production binary first boot, admin setup, authenticated OTLP ingest, Parquet queries, compaction, graceful shutdown, restart, persisted queries, and post-restart ingest
  • 376 rows accepted with zero drops in the initial E2E pass; rollup totals increased after restart
  • full local just check and pre-push gate
  • full Go race suite plus repeated storage, query, and UTC regression tests
  • Go and UI dependency audits, generated notices, UI builds/tests, documentation generation, and Astro site build
  • storage benchmarks: approximately 988k spans/s through the production four-worker writer and approximately 1.01 ms median indexed trace lookup on the test host

Move telemetry ingestion to a durable WAL-backed repository that commits indexed hot segments and open Parquet files. Keep DuckDB as the analytical query and rollup engine, and SQLite as control-plane storage.

BREAKING CHANGE: existing DuckLake telemetry data is not migrated. Deployments must start with a clean storage.data_dir.
Comment thread internal/telemetry/store/compaction.go Resolved
Comment thread internal/observability/logs.go Resolved
Comment thread internal/telemetry/store/writer.go Resolved
Comment thread internal/telemetry/store/repository.go Resolved
Comment thread internal/observability/trace.go Resolved
Comment thread internal/query/duck.go Resolved
Comment thread internal/api/health.go Resolved
Fail fast on legacy DuckLake data instead of hiding it. Drain and publish compaction without blocking reads or commits, and surface failed writes immediately.
Comment thread internal/telemetry/store/writer.go Resolved
Comment thread internal/telemetry/store/repository.go Resolved
Comment thread internal/telemetry/store/compaction.go Resolved
Comment thread internal/query/duck.go Resolved
Comment thread internal/telemetry/store/repository.go Resolved
Keep failed ingest batches backpressured until durable commit. Use day-partitioned leveled compaction so retention remains enforceable without rewriting the retained corpus. Hold Parquet snapshot locks through row iteration and bound hot-segment descriptors.
Comment thread internal/telemetry/store/compaction.go Resolved
Comment thread internal/telemetry/store/repository.go Resolved
Comment thread internal/telemetry/store/writer.go Resolved
Comment thread internal/observability/logs.go Resolved
Comment thread internal/query/duck.go Resolved
Comment thread internal/query/duck.go Resolved
Comment thread internal/observability/trace.go Resolved
Comment thread internal/query/duck.go Resolved
vishr added 2 commits August 26, 2026 15:31
Persist staged directory entries before publishing the recovery marker. Validate every required signal output and restore retired inputs instead of publishing an incomplete compaction.
Prevent compacted WAL replay, bound writer shutdown and poison-batch retries, tier cold log reads through Parquet, and serialize DuckDB maintenance with rollups and context-aware query locks.
Comment thread internal/observability/logs.go Resolved
Comment thread internal/observability/trace.go Resolved
Comment thread internal/query/duck.go Resolved
Apply maintenance budgets only between completed publications so slow compactions make durable progress. Bound rollup admission and publication waits without canceling admitted cache work or removing ingest from rotation.
Comment thread internal/query/duck.go Resolved
Comment thread internal/query/duck.go Resolved
Comment thread internal/telemetry/store/repository.go Resolved
Comment thread internal/query/duck.go Resolved
Comment thread internal/telemetry/parquet.go Resolved
Comment thread internal/telemetry/parquet.go Resolved
Comment thread internal/telemetry/parquet.go Resolved
Comment thread internal/query/duck.go Resolved
Comment thread internal/telemetry/store/compaction.go Resolved
Comment thread internal/telemetry/store/repository.go Resolved
vishr added 12 commits August 27, 2026 20:10
Five review rounds relocated the process-wide serialization point four
times (parquetMu, writeGate, publishMu, back to parquetMu) without the
finding rate falling, because each fix bolted a timeout or count onto a
call graph where no constant could see the cost of the work it budgeted.
This bounds that work at admission instead.

The edge rollup stops after maxEdgeSubWindowsPerPass sub-windows,
persists a resume cursor, and leaves the ingested watermark behind until
the window finishes. A seed load or backfill compresses a wide start_time
range into a narrow ingested window, so chunking ingested time alone let
one pass expand into an unbounded number of sub-windows and hold the read
gate for as long as that took. Cancelling mid-transaction only discarded
the work and retried it forever, so the pass now yields instead, and hold
time is a function of the constant rather than the shape of the data.

PublishParquet budgets the reader drain and the directory swap
separately. Sharing one deadline meant a publisher that spent most of it
waiting entered the exclusive window with almost nothing left and
cancelled its own renames after paying the full cost of excluding every
reader.

CommitBatch takes a context, threaded from the writer through
Repository.Commit. It was the last publish-gate waiter using
context.Background(), so a stalled publication blocked ingest
indefinitely and OTLP clients lost rows to their own timeouts.

A compaction marker that fails recovery maxCompactionRecoveryAttempts
times is set aside as COMPACTION.json.failed, and Open sets it aside and
boots rather than exiting. Recovery correctly gates retention,
compaction, and retired cleanup, so without a give-up path one bad marker
latched all maintenance off for the process lifetime and refusing to boot
left no way to run the cleanup that would clear it. Nothing is deleted:
the staged output survives and cleanupRetired now protects the
.retired-<output> sets named by any live or set-aside marker. Cancelled
passes do not count toward giving up.

PruneBefore holds p.mu only to mutate the in-memory set, not across up to
64 renames and an fsync. CommitBatch's first action reads that same
mutex, which made it a second undeadlined serialization point on the
ingest path, invisible to the publish gate's accounting.

Adopting a directory left by an interrupted commit now validates it
before taking the gate rather than inside it.

Removes Repository.Trace and the panicking parquetReadGate.RLock/Lock
wrappers: both were duplicate paths, and Repository.Trace bypassed the
read gate entirely.

Adds invariant tests for the bounded rollup hold, the marker give-up and
its rollback-set protection, booting past an unrecoverable marker, and
ingest surviving a stalled publication, so a future relocation fails CI
instead of waiting for another review round.
Setting a failed compaction marker aside was supposed to end the failure,
but two paths kept it going.

protectedRetiredSuffixes returned an error when a marker would not parse.
The set-aside marker stays on disk by design, so cleanupRetired failed on
it during every subsequent Open and the process exited on each boot —
precisely the outcome that setting it aside exists to prevent, and worse
than the latch it replaced because it survives restarts. An unreadable
marker names no rollback set, so it now retains every retired directory
and logs, deleting nothing.

cleanupCompactionArtifacts decided whether to keep the staged output from
a boolean set on the boot that performed the quarantine. On the next boot
the live marker was gone, the flag was false, and the staged output was
removed — leaving an operator able to roll the compaction back but never
to complete it, though the doc comment promised both. It now keys off the
presence of the set-aside marker, so the stage survives as long as the
marker does.

PublishReplacement held p.mu across the output move, up to 64 input
renames, and an fsync. CommitBatch's first action reads that same mutex,
so this was the ingest-blocking pattern PruneBefore's comment already
documents as forbidden; p.mu now covers only the in-memory map mutations.

CommitBatch no longer leaks its staging directory when the publish rename
loses to an existing final directory.

selectCompactionBatches applies the same MaxIngestedNanos > 0 guard as
the counting loop, so a batch with no ingest timestamp cannot be pulled
into a group it was never counted in.

Adds regression tests for repeated boots against an unreadable marker and
for the staged output surviving every boot the marker survives.
Keep compaction recovery fail-closed instead of bypassing a live marker. Bound only publish admission and edge-rollup work, and read stats under a stable Parquet namespace.
Review of a70592e, reading it directly rather than delegating.

updateParquetStats now takes the Parquet read gate, which is the right
call — it was the one reader that could observe a namespace mid-swap. But
it runs on the rollup and maintenance loop goroutines under the process
context, so unlike every other reader it had no deadline, and a stuck
publication would park both loops indefinitely. The wait is now bounded;
a skipped refresh costs one tick of stale gauges, which is strictly less
than what waiting costs.

validateCompactionID duplicated telemetry.validateBatchID's grammar in a
second package, where the two had already begun to diverge in
implementation. The rule decides which strings may become directory names
under the storage root, so two copies means two gates that can disagree
about what is safe. The telemetry version is now exported and the store
calls it.

cleanupRetired names the coupling it acts on: a live marker is the only
thing that makes a retired directory unreclaimable, so deleting
COMPACTION.json without first renaming its <id>.retired-<output id>
directories back to <id>.batch is not a rollback — it makes those rows
deletable, and this loop removes them on the next pass. The startup log
warns about this; the code that does the deleting did not say so.

Also documents the site-level runbook changes from the review: the
troubleshooting section for a startup-blocking marker, and the data
layout reference, which still described COMPACTION.json as a transient
marker rather than a startup gate.

Adds a regression test for the bounded stats refresh.
Remove both possible replacement locations during manual rollback so restored inputs cannot coexist with a published output. Exercise the stats reader's own wait limit without caller cancellation.
aacffe0 corrected the rollback guidance to delete both possible
replacement locations, since recovery checks for the output either staged
under compaction/ or already published as <output id>.batch. The docs and
the rollback field were updated; the two log fields that describe where
things are were not.

An operator reading the startup log was told the staged replacement was
intact and pointed at compaction/. When the interruption happened after
publication that directory is empty, and the reachable conclusion from
"the log says it is here and it is not" is that the merged rows are gone
— which is the panic that ends in rm -rf of the data directory, the one
outcome failing closed exists to prevent.

The field now names both locations, matching the rollback procedure and
the troubleshooting guide, and staged_replacement is renamed to
compaction_staging so the key stops asserting a location the code does
not guarantee.
A failed publish may have already removed some retired inputs. Require operators to verify the complete rollback set before deleting the replacement or marker.
238f2fe dropped the claim that the retired inputs are all intact, because
a publish that failed partway may already have removed some, and added a
precondition to the startup log: verify every marker input exists as
either <id>.batch or <id>.retired-<output id> before deleting anything.
The troubleshooting guide was not updated with it and still made the
stronger claim.

That left the guide's procedure unsafe in exactly the case the log now
warns about. Its step 2 covered ids present as <id>.retired-<output id>
and ids present as <id>.batch, but said nothing about an id present as
neither — the dangerous one, where the rows survive only inside the
replacement. An operator with a partially removed rollback set would have
followed the steps in order, deleted the replacement, and destroyed those
rows permanently.

The guide now states that the rollback set may be incomplete, verifies it
as an explicit step before any deletion, and tells the operator to stop
and either restore from backup or complete the compaction instead. The
caution covers both ordering constraints rather than only the second.
An input present in both active and retired forms is ambiguous and cannot be safely renamed. Require exactly one copy before deleting the compacted replacement or marker.
The manual rollback lived in both logUnresolvedCompaction and the
troubleshooting guide, and every revision since it was written corrected
one copy and left the other stating something the code no longer
guarantees:

  b8345fb  guide named both replacement locations; the log did not
  238f2fe  log dropped the "all intact" claim and added a verification
            precondition; the guide kept the claim and the old procedure
  7e4ab8c  guide caught up
  33f3cb1  guide required each input in exactly one form; the log still
            said either form, which permits the ambiguous case it was
            added to reject

Each copy was correct when written. Operator guidance that is wrong in
one of the two places an operator might read is worse than guidance in
one place they have to open, so the log now links the runbook and states
only what it alone knows: the absolute paths on this machine, that
nothing has been cleaned up, that a plain retry is the first move, and
the one invariant that must not be got wrong offline — deleting the
marker by itself is not a rollback.

The link is checked by a test that resolves its anchor against the
guide's headings, so the pointer cannot rot the way the copy drifted.

Also rewraps the paragraph 33f3cb1 left broken mid-sentence.
Parallelize group commit and native compaction while preserving atomic publication and bounded query memory.

Add deep offline verification and explicit quarantine for unreadable authoritative batches.
@vishr
vishr merged commit c3dc7d9 into main Aug 28, 2026
8 checks passed
@vishr
vishr deleted the feat/telemetry-segment-parquet branch August 28, 2026 17:56
vishr added a commit that referenced this pull request Sep 8, 2026
…242)

* perf: measure the stall, and stop making reads pay for writes

Three changes for #241, none of which assumes a mechanism, because the first
diagnosis assumed one that does not exist.

**The reader's side of Parquet publication is measured.** The publisher's wait
was already a histogram; the reader's was nothing at all, which is why a
twelve-second query could not be attributed to anything and was blamed on a
lock removed by #207. `fanout_parquet_read_wait_seconds` is labelled by reader
class — a query that waits and a rollup that waits are different problems with
the same symptom — and a reader that gives up is counted separately from one
that waited, because a refusal and a long wait are different failures. The
exclusive half of a publication is measured too, and an explicit checkpoint now
reports both its duration and whether it failed: the non-FORCE form fails
outright while any transaction is open, which on a live instance is most of the
time.

**Rollups and maintenance no longer draw from the read pool.** They take a
dedicated single connection. The pool is sized one per core, so on the
four-core reference host it holds four, shared by every dashboard widget, the
detector, the alert engine, the readiness probe — and, until now, whichever
rollup happened to be running, for as long as it ran. A request that waits for
a connection waits invisibly; database/sql counts that wait and nothing
exported it, so the pool's own gauges are exported now as well. One connection
is all the writers can use, because the write gate already lets one through at
a time.

**The edge rollup yields to a queued publication.** A pass holds the Parquet
snapshot until it commits, and a publication that queues behind it makes every
query arriving after the grace period wait out the rest of that pass — up to
eight sub-windows of work for a request that needed none of it. The pass now
stops at the first sub-window boundary once a publisher is waiting. That costs
nothing to implement: the resume cursor already exists for the per-pass budget,
and the ingested watermark already stays behind an unfinished window. It never
yields before completing one sub-window, so a busy compaction cycle cannot
starve the cache the whole product reads from.

What is deliberately not here: the publication budgets are untouched. They are
the stall's ceiling, not its cause, and tuning them would hide the measurement
that is the point of this change.

Verified against a local instance: the new series appear with a query/rollup/
stats split and an idle baseline of about two microseconds per admission,
against which a nine-second wait is unmistakable. The yield is pinned by a test
that fails when the condition is removed.

Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk

* fix: one owner for the DuckDB instance, and a yield that waits for the grace

Review findings on this branch. Two of them were defects in the change itself.

The write handle could free the DuckDB instance a second time. database/sql
closes a connector that implements io.Closer when its DB is closed, and
duckdb's Connector.Close calls duckdb_close behind an unsynchronized bool — so
two handles over one connector meant two closes of one instance, which kills
the process with a SIGTRAP out of cgo rather than returning an error. The read
pool owns the instance; the write handle takes a connector wrapper that cannot
close it, and Close runs once however many times it is called. The comment
claiming a second connector would fail on a file lock was also wrong: DuckDB
keeps an instance cache and would have shared happily.

The yield gave up eight sub-windows to relieve congestion that had not
happened. A publication queues every ten seconds under compaction, and readers
keep being admitted for the whole thirty-second grace, so yielding the moment
a publisher appeared cut a pass from eight sub-windows to one exactly when
ingest was busiest — an eightfold slower catch-up, plus the fixed cost of the
window scan on every resumed pass. The trigger is now the grace running out,
which is the moment a query would actually start waiting behind the pass. The
test drives the gate's clock rather than a timer, so it pins the ordering
instead of hoping for it.

Smaller ones from the same review: a refused checkpoint no longer lands in the
duration histogram it would drag towards zero, and the comment saying refusals
would be common was wrong — only a concurrent write transaction refuses one,
and the write gate prevents those. Releasing the pool-gauge source now clears
only the source it installed, so a displaced Duck closing cannot blank the
gauges of the one still serving. The two Parquet histograms start a hundred
times lower, because an uncontended admission costs about two microseconds and
every healthy observation was landing in the first bucket. The reader-wait
series go through a recorder for the reason the write gate's do — a label
that must stay a fixed class should not be set through an exported vector.

The pool test now uses the read pool before asserting nothing took a connection
from it, so its failure message is about where the write went rather than about
a pool nobody had touched, and the metrics test no longer asserts on
process-global registry state, so -count works.

Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk

* perf: bound the two statements whose cost grew with the data

Measurement said the stall was pool contention, so the first instinct was to
make the pool bigger. That was wrong: the requests waiting for a connection
were waiting behind two statements doing far more work than they needed.

Log redaction ran on every row in the window. Four chained regexp_replace sat
in the same SELECT as ORDER BY time DESC LIMIT, and a projection there is
evaluated below the top-N — so returning a hundred entries redacted every row
the window held. Measured at 140ms for 100 rows out of 372,000 against 8ms
once the top-N runs first, and independently at 449ms against 11ms over
400,000 rows with more realistic bodies — the plan confirming the projection
moved above the top-N and EXCEPT ALL confirming the rows are identical either
way. It was the one statement whose cost grew
with retention rather than with the answer — with no search term; a search
still has to redact every row it tests, because it must match the text the
reader is shown rather than the secret underneath it.

The endpoint boundary scan read the whole window. Its bounds came from a CTE
joined against the scan, and a bound derived from a joined row is not a
constant DuckDB can push into a Parquet scan — so a query that wants the two
partial minutes at the window's edges read everything between them: 799,333
rows of 800,000 to return 7,333, and 240,000 to return 667 in a smaller
independent reproduction. The bounds are computed in Go and bound as
parameters now; the same queries scan 11,333 and 35,200.

Dashboard queries get a twenty-second deadline. The SQL API has always had one;
these endpoints, which every widget calls, had none, so a pathological query
held its connection until the client gave up. A query that exceeds it answers
503: running out of time under load is not an internal error, and reporting it
as one logs at ERROR and counts towards the 5xx rate that pages someone.

And the half of the picture the last commit lacked: how long a statement takes
to execute. The connection is acquired before the timer starts, because
DB.QueryContext would fold the wait for a free connection into the statement's
duration — and telling queueing from execution is the entire reason to measure
it. A test holds the pool's only connection to prove the two do not overlap.

Measured across the same seed and load — 96 batches of 4,000 spans over ten
hours, continuous ingest, eight concurrent clients, 150s, four-connection pool
in both arms:

                       req     p50     p95     p99   cheap p99
  before            11,970    45ms   275ms   498ms      178ms
  after             15,642    31ms   191ms   358ms      122ms

Throughput up 31%, p95 down 31%. The max column is deliberately absent: with a
closed-loop generator one server-side stall appears as eight slow requests, so
a single run cannot separate a shifted distribution from one bad event.

No pool-sizing change. An earlier version of this commit rewrote
resolveDuckDBMaxConns, and the arm that appeared to justify it compared a
hand-pinned pool of four against an auto-sized seven — which is not what the
rule would have done on any host: unchanged on the four-core reference host,
and a reduction on larger ones that nothing here measured. The evidence says a
cheap query should not queue behind an expensive one; it does not say what the
pool should be. The statement histogram and the pool gauges are what should
answer that, on a real host.

Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
vishr added a commit that referenced this pull request Nov 5, 2026
* fix(ui): keep alert badges whole and the history scrollable by keyboard

On a phone the squeezed tables cut state badges to "FIR…"; badge labels no
longer shrink. The history's horizontal scroll area had nothing focusable,
so it is now a named, focusable region. Hovering a rule row tinted it under
dimmed descriptions to 4.3:1; the rows do nothing on hover, so the tint is
gone.

* fix(ui): give the query editors' placeholder AA contrast

CodeMirror's placeholder grey is 3.5:1 on white, so New dashboard's query
example failed contrast. The shared editor theme now draws it in the grey
the SQL comments use, which clears 4.5:1 on plain and active lines in both
schemes.

* perf(telemetry): keep each compaction inside one ingest hour

Span files sort by trace hash, so every row group covers its file's
whole time range and DuckDB decodes the attributes VARIANT of every row
in any file a window touches, whatever the time filter selects.
Compaction grouped by ingest day, so after a few generations a
one-minute attribute filter over older data decoded a multi-hour file.
Measured on generated files at 2 threads, a one-minute
attributes['http.route'] filter costs CPU in proportion to the file:
0.17 s at 54,000 spans, 3.4 s at 2.1 million, 10 s at 8.3 million, while
the same filter on the scalar http_route column stays under 0.07 s.

Group compaction by ingest hour instead, so the largest file a narrow
window can touch holds one hour (about 325,000 spans at the pico rate)
and older data costs what recent data does. A finished hour merges even
a pair of files, so a quiet install keeps at most one file per hour per
signal. Compaction debt reserves its underfilled group per hour.

* fix(query): let schema discovery finish after its callers leave

A discovery that needed about 20 seconds never finished on a busy host:
the request and the discovery shared a 20-second deadline, and the
discovery was cancelled when its last caller left, so the cache never
warmed. A discovery now runs on, bounded by its own one-minute deadline,
the two-slot cap and the fixed key set. A request waits 15 seconds and
then gets 503 schema_busy with Retry-After; the retry finds it cached.

The schema route also gets its own timeout message, and an unknown
window is a 400 that names the supported windows.

* perf(query): sample attribute keys from 1,000 rows, read once

Discovery spent most of its time decoding VARIANT attribute maps. The
sample CTE was referenced twice and inlined, so each reference reran the
window scan, and every key re-parsed its row's whole JSON document. The
sample is now materialized, json_each parses each map once, and the
sample is 1,000 rows (spans sort by trace, so it still spans services).

On a pico-like dataset (24h of 5,400 spans/min with OTel demo sized
attribute maps, 2 threads, 1 GB), the attribute queries for 1h dropped
from 6.6-9.1 s to 1.6-1.8 s of CPU, and a cold 1h discovery from 18-20 s
to under 5 s of server CPU.

* fix(query): say which value form matches a mixed-type attribute

An empty panel whose attribute is stored as a number by some rows and as
text by others said only that the filter matches one type. It now tells
the user to quote the value ("308") or drop the quotes, whichever finds
the rows the filter missed.

* fix(fanql): give foreign forms exact hints and fix-its

percentile(duration, 0.95) suggested quantile(duration, 0.0095), and
99.9 suggested 0.9990000000000001: a value below 1 is now already a
fraction, and the fraction is printed without float artifacts.
count() group by service said to separate measures with a comma; it now
says group by is not FanQL, with a fix-it to by, after a measure or a
second |. A measure already reported as foreign no longer adds a false
"by needs measures first". head N and take N right after the source or
a filter get the limit hint and fix-it they get after a stage.

* fix(fanql): name field problems and offer the field a typo meant

p95(duraton) silently read attributes['duraton'] and drew an empty
chart. A bare name within edit distance of a column (duration included)
still reads the attribute, but its notice now says "did you mean
duration?" and carries a fix-it. "opération" needs brackets now offers
operation and attributes['opération'] as fix-its. A variable used as a
field says so ("by cannot use the variable $x") instead of asking for
brackets. attributes.foo alone stays free text, as a word before no
operator is, with a notice that says how to filter on the field.

* fix(fanql): rank and/or after a term and offer every column's values

After a finished term, and, or, |, sort and limit were cut by the
50-item cap, and "o" never offered or. They now lead the list after a
complete term (and, or inside a group).

http_route, http_status_code and rpc_service take the sampled values of
the attributes ingest copies them from, and operation is scanned for
its common values, so these columns complete like their attributes. A
key whose rows disagree on the type reports both types, and completion
offers each value as a number and as quoted text, saying which rows each
matches. kind, status and type describe the values people type (client,
error) rather than the stored enums.

* fix(engine): narrow the service rollup window after it runs out of memory

Each service rollup pass recomputes every minute bucket touched by an
hour of ingest. A backfill or seed packs days of event time into a few
minutes of ingest, so that pass grouped millions of spans, ran out of
memory at a 1 GB limit, and retried the same window every minute; after
a seven-day backfill service_rollup stayed empty for the whole run and
the health and service map panels showed nothing.

An out-of-memory pass now halves the ingest window, down to a second,
and retries at once; successes double it back to the hour. A tick
drains further passes within half the rollup interval, as the edge
rollup does. On the same backfill the rollup caught up five minutes
after start and stopped failing.

* docs(api): describe schema_busy from the generator source

* chore(ui): rebuild embedded assets

* test(e2e): update trace baselines for the unitless duration header

* fix(query): stop schema discoveries when the executor closes

Detached discoveries ran under the request context without its
cancellation, so shutdown never stopped them and an abandoned one held a
read connection for up to a minute. They now run under an executor
lifetime that Close cancels and waits for, and a run is capped at 25
seconds, separate from the one-minute slot wait. The slots and bounds
are per executor rather than package globals, and the cold-burst test
now proves late callers joined the flight.

* fix(engine): grow the service rollup window only after committed passes

The window doubled after any pass without an error, including a pass
that skipped because compaction held the Parquet gate, so a narrowed
window grew straight back and ran out of memory again on alternate
passes. It now grows only after three committed passes in a row and
stays put on a pass that did no work. The startup rollup also reports
a fatal engine error, as the loop does.

* perf(store): merge small settled hours across their day

Compaction kept one file per ingest hour per signal for the whole
retention. Hours more than a day behind the newest that have nothing
left to merge within themselves now merge with adjacent hours of the
same day into files of at most 250,000 rows, in time order. Files
already spanning hours stay out of hour groups, and a day merge's
output sits above every input generation.

The bound keeps narrow queries on old data no slower than a busy hour.
On a 72-hour pico-rate simulation, million-row day files cut span files
from 73 to 33, but slowed a one-minute attribute filter on old data from
0.12 to 0.66 seconds and a 24-hour one from 2.5 to 3.6, while each file
costs a scan about 0.1 ms. With the bound, pico keeps its 73 hourly
files and a quiet install (16,000 spans an hour) holds 31 files after
seven days instead of one per hour, with a one-minute attribute filter
on old data at 0.14 seconds.

The finished-pair test now tests a pair.

* fix(query): treat -k:* as one negated term on the traces list

FanQL lowers -k:* to a bare k IS NULL, which negatedTerm did not
recognise, so the traces list kept any trace with one span lacking k.
A top-level IS NULL now excludes every trace with a span that has k.
The null-inclusive form counts only when its second branch is one leaf
on the field, as FanQL writes it and lifts it back; hand-written
compounds and calls such as lower(k) stay span conditions.

* fix(ui): leave status and excluded terms out of span error rates

The Traces page's link to services by error rate reused the search's
filters as span filters, so status:error made every row 100% and an
excluded term dropped single spans instead of whole traces. The link now
formats the search's stored filters and leaves out status terms and
prefix negations, and its label and tooltip say it shows the matching
spans' error rate per service.

* fix(alert): keep a tick from evaluating a rule disabled mid-tick

A tick that listed a rule as enabled recompiled it after DisableRule
had dropped its program, and wrote a firing or pending alert for a
disabled rule. Disabled and deleted rules are now kept stopped until a
recompile or until a tick no longer lists them, and each rule's
transitions run under the read lock, so disabling waits them out and
cleans up after.

* fix(alert): store episode starts in one sortable form

Old rows held SQLite's datetime('now') text and new ones RFC 3339 with
a variable fraction, so listing alerts newest first misordered them, and
an upsert without created_at replaced the episode start with the time
of the update. The store now writes created_at as RFC 3339 UTC with nine
fraction digits, keeps the stored start when an update leaves it empty,
and refuses a new alert without one. A migration rewrites existing
values to the same form.

* fix(api): answer an unknown schema window with bad_request

The schema endpoint used invalid_request, OAuth's spelling and not one
of the documented application codes, and indexed the first problem
without checking there was one. It now returns bad_request with the
first problem's message, or a general one when the list is empty, and
the API reference says so.

* fix(alerts): prune resolved alerts by instant, not by text

Resolved times carry the engine's local offset, so comparing their text
with SQLite's UTC cutoff kept alerts resolved on a later local date.

* test(store): count applied migrations from the migration files

* chore(ui): rebuild embedded assets

* docs(storage): explain file-granular retention and day merges

Retention removes whole files once their newest row passes the cutoff,
as Prometheus, Tempo and Loki remove whole blocks. Day-merged files on
quiet installs can therefore keep rows up to a day past the retention
period. Describe the hour groups and settled-day merges compaction now
performs.

* perf(query): read metrics once per metric panel

Each metric builder scanned the metrics relation on its own, so avg(value)
and sum(value) read a gauge's rows a second time for the histogram branch,
and bucketed panels a third time for the raw window. Builders and the raw
window now read one shared scan of the authored filters; a scan with several
readers is materialized. Each builder still selects its own window and kinds
from it, so deduplication, lookback and kind semantics are unchanged.

A golden test pins results for every stored kind, alone and mixed.

* fix(alert): keep each firing episode in history

Alerts were upserted on (rule_id, service), so firing again after a
resolve overwrote the resolved row and history lost episodes. A new
migration drops that unique constraint and adds a partial unique index
that allows one pending or firing alert per rule and service. Firing
again now inserts a new alert, upserts key on the alert id, and
delivery status updates the episode that was notified.

* fix(query): report a panel whose measures are all null as empty

A measure of a field the matching rows never carry returned rows of
nulls with status ok, so the editor saved an empty chart. Structured
span and log panels now report empty with a diagnosis that names the
measure as the missing part, as metric panels already did. State
timelines keep null buckets because they draw them as unknown.

* fix(ui): say the last run failed when the server refuses it

A run the server rejects with problems, such as a SQL binder error,
left the previous result in place, so the save button said to run the
query as it is now although it had just run. The editor now records
the refused run and says it failed until the query or window changes.

* fix(ui): focus the confirmation after restoring a version

The restore button is disabled while it works, so focus fell to the
page body. The confirmation now takes focus when the restore lands.

* fix(ui): show Updated only when a panel ran

The dashboard header showed the time the last batch returned, even
when the server turned every panel away and each said Not run yet.
It now shows when a panel last ran, and nothing before one has.

* fix(ui): show a running state in Explore and explain an empty run

A rerun or a new time range showed the old result as 'query has changed'
while the run was in flight. The status now says the query is running, and
a result left over from another window says the time range changed.
Enter on an empty editor now says to type a query.

* fix(ui): keep the Builder operator and completion details readable on phones

The filter operator kept shrinking to 48 px at 390 px, hiding = and >; it now
keeps its width while the field and value give way. Wrapped completion
descriptions hang under the label instead of the list's edge.

* fix(ui): say that table rows without a trace_id list exemplar traces

A table with the traces drill said 'click to open trace', but a row without a
trace_id opens the list of exemplar traces. Only rows that carry a trace_id
say they open a trace.

* fix(ui): read each duration axis in one unit

Each tick picked its own unit, so an axis read 0ms, 5s, 10s. Value axes now
pick one unit from their bounds and step, and keep it when annotations
recompute the ticks.

* fix(ui): keep flame graph ticks on one line and fold extra services

The last tick of a long trace wrapped and lost its second line (50m 20s read
as 50m). Services past the six palette slots share the muted colour, so the
legend now names them as one entry instead of several identical grey dots.

* fix(query): label metric panels with the unit the metric reports

Metric value columns had no unit, so a JVM memory stat read 672.4M and a
request-duration axis in seconds was unlabelled. A metric panel filtered
to one metric now gives its value measures the unit the metric reports,
mapped from OpenTelemetry units (s, ms, ns, By, %, and {count} units).
Counts, rates, expressions and an authored panel unit keep their own.
One small lookup per metric name is cached for ten minutes.

* fix(query): name duration as FanQL spells it in diagnoses

An empty panel filtered on duration said "Without duration ms filter";
the diagnosis now names the field duration.

* fix(query): read diagnosis values in one scan and suggest near misses

An empty panel's diagnosis ran two attribute scans after the main
query: the common values, then the rows storing the literal as the other
type. On a slow host the panel deadline ran out before the second, so the
hint to quote the value disappeared on wider windows. One grouped scan
now answers both, and also finds the nearest value within a small edit
distance, so a misspelled metric name or service gets "Did you mean".

On 200,000 local spans the value scans fell from 650-780 ms to
400-430 ms, and the whole empty-panel run from about 1.0 s to 0.7 s.

* fix(fanql): complete only the value forms rows store

For a key whose rows disagree on the type, completion offered every
numeric-looking value both bare and quoted, so 308 was offered as
matching numbers when no row stored it as one. Schema discovery now
counts the sampled values stored as numbers, and completion offers each
form only when rows store it.

A bare column name such as type on metrics also listed the values of an
attribute named type, which the parser then rejected. Completion now
reads a bare column name as the resolver does: column values only, and
vocabulary aliases only on their own signal.

* fix(fanql): explain foreign queries, misspelled functions and metric groups

- A misspelled measure function gets "did you mean" with a fix-it on its
  name; with nothing near, the problem lists the functions. An unknown
  percentile such as p59 offers p95 and quantile(field, 0.59).
- PromQL (a PromQL call, a selector or range after a name, or a
  Prometheus metric name) gets one message instead of a cascade. Kusto's
  | where ... | summarize gets one message; a lone | where gets a fix-it
  that removes the pipe, and the rest parses as usual.
- A metric aggregate that would combine several metrics, which the run
  rejects, is now a parse problem with the run's wording. The two corpus
  fixtures that relied on the run to reject it are now problems.
- A bare key from an OpenTelemetry resource namespace (host, k8s,
  container, cloud, os, service, deployment, telemetry) still reads
  attributes, as on every signal, with a notice and a fix-it toward
  resource.<key>.

* fix(ui): offer FanQL fix-its as buttons under the editor

Problems and notices from the FanQL parser carry fix-its, such as count
for cuont() or resource.host.name for host.name, but the editor only
exposed them in a hover tooltip, out of reach of the keyboard, and
dropped the ones on notices.

The editor now lists problems and notices itself, each fix-it as a
"Replace with ..." button that edits the text, puts the cursor after the
replacement and returns focus to the editor. Notices stay quiet and
apart from the problem alert, and are marked in the text as info. Once
the text is edited, a list's spans no longer index it, so its marks and
fix-its are withdrawn while its messages stay. The search pages map
notice spans back from the combined text, as they did for problems.

* fix(ui): keep Updated when a refresh is turned away for every panel

A panel the server turned away still shows the frame of its last run, and
merge keeps that run's time. Updated now counts it, so a fully shed refresh
no longer hides when the shown data ran.

* fix(ui): make the folded flame legend entry readable without a mouse

The names of services folded into 'N other services' were only in a title
attribute. They are now read with the entry, and a tooltip shows them on
hover, keyboard focus or touch.

* fix(ui): refocus every restore confirmation and announce it once

The focus effect keyed on the confirmation text, so a repeat with the same
text could keep focus where it was. Each confirmation is now a new value.
Focus reads it out, so it is a note rather than a live status.

* fix(query): cache a metric's unit only when the window settles it

The unit lookup reads only the panel's window. A window with no points
or with several units cached an empty unit, so every panel on that
metric lost its unit for ten minutes. Only a single reported unit from
a window with points is cached now.

* fix(ui): give a traces table no click hint until its columns arrive

While loading, a traces-drill table said it listed exemplar traces, then
switched to opening a trace once a trace_id column arrived. It now says
nothing until the columns decide which.

* fix(fanql): match the run's one-metric rule and keep where as an alias

The parse check accepted name in (a) and name in $one as naming one
metric, but the run reads only an equality, so both failed when run. The
parse now reads only an equality too. A measure aliased where lexed the
alias as a misplaced where; after as it is an alias again.

* fix(ui): keep FanQL notices to the text they describe

A notice's span indexes the text that was parsed. The pages kept notices
across edits and stored them even when the text had changed while the
parse was out, and the editor guessed staleness from list identity, which
a failed run or a recomputed list could defeat, leaving a live fix-it
that rewrote the wrong range. The pages now clear notices when the text
changes or a run starts and store them only for the text still in the
editor, as they do for problems; the editor's guess is gone.

Each problem message is now its own alert, and its fix-it buttons sit
beside it, described by it, so an alert reads the message alone.

The empty-run hint in Explore lives in a live region that stays mounted,
shows only while the editor is empty, so text from history or a tab
switch clears it, and adds a new node on every empty Enter so it is
announced each time.

* fix(query): suggest near misses for names only, and bound the search

The diagnosis suggested 500 for a filter on status code 503 and ab for a
one-letter service. Near misses are now offered only for text fields,
for literals of three or more characters that are not numbers.

Edit distance ran over every distinct value, twice. It now runs once,
over at most the 1,000 most common values whose length is within the
allowed distance. On 200,000 local spans with unique IDs the value scan
for a trace_id filter fell from about 265 ms to 25-40 ms, and for an
attribute ID from about 420 ms to 170-220 ms.

* chore(ui): rebuild embedded assets

* test(fanql): name the code-point span test plainly

* test(e2e): update darwin performance baselines for one-unit axes

* test(e2e): wait for the flame tooltip to close before scanning the trace view

* fix(ui): restore an Explore entry's own query on Back after a range change

Back to an entry that differs only in its time window reran the unsaved
draft in the editor and rewrote that entry. Only a window chosen in the
range menu now reruns the editor; any other window change is history
navigation and shows and runs the entry's own query.

* fix(fanql): name a misspelled metric name field before the aggregate problem

A metric aggregate filtered on a misspelling of name, such as nme, read an
attribute, so the run named no metric and parse reported only that metric
aggregates need one metric per group. The notice that explained the typo was
dropped with the problem. Report the misspelling as the problem, with name as
its fix.

* fix(fanql): use http.server.request.duration in metric name examples

http.server.duration is the old, unstable semantic convention name. The
stable name is http.server.request.duration.

* fix(fanql): complete stored severity levels in lowercase

Completion offered levels seen in the window as stored, so logs where
severity: listed UNSPECIFIED beside the lowercase standard bands. FanQL reads
levels case-insensitively; offer them in lowercase.

* fix(fanql): suggest duration for a misspelled duration with a unit

spans where durationn>1s said units apply to duration only and suggested
durationn>1, hiding the typo. Name the misspelling and offer duration as the
fix. On logs and metrics, which have no duration, say the field takes a
number instead of pointing at duration.

* fix(ui): qualify truncated trace timelines beside the chart

The truncation notice sat under the correlated logs while the flame
badge and axis showed the loaded spans' extent as if it were the
trace's. The notice now sits above the waterfall or flame graph and
says how much of the trace's duration the loaded spans cover, and the
badge reads "34h 57m of 4d 6h shown".

Flame axis ticks thin to three or two when their labels would touch,
and Escape on the focused "other services" entry hides its names
before it reaches the drawer (WCAG 1.4.13).

* fix(ui): head table duration columns without a unit

Table cells print each duration in its own unit, so a max_ms column
read "10m" under "Max (ms)". Every table now drops the duration unit
from a header whose cells format durations, as the logs and traces
tables already did. A column without a duration unit keeps its name's
unit, since its cells print bare numbers.

* fix(fanql): suggest the clause keyword for a misspelled clause

spans | count() by service evry 1m said only unexpected evry. A word near a
clause keyword now offers the keyword as a fix: did you mean every?

* fix(fanql): suggest a seen key for a misspelled dotted key

Undotted keys near a column got a did-you-mean notice; dotted keys such as
http.rout got none, because parse knew no attribute keys. Parse now reads the
recent schema when one is already cached, never waiting for a discovery, and
offers the nearest key seen in the same scope. With no cached schema there is
nothing to compare and no notice.

* fix(ui): treat panels the server turned away as skipped, not failed

When a refresh was shed busy while panels still held an earlier run, the
dashboard showed a red "Panels could not be loaded" banner. It now shows
a quiet notice that the server was busy, the latest refresh was skipped,
and the marked panels show data from their earlier run.

A panel that came back busy while out of view had already been
attempted, so scrolling to it did not run it; on a cold System overview
three panels below the fold stayed "Not run yet". Such a panel now runs
once when it scrolls into view. A busy panel still in view waits for the
next refresh or its Retry, so a busy server is not asked again at once.

* fix(ui): keep the other fix-its after applying one, and hide the lint tooltip

Applying a fix-it withdrew every other problem and notice, so two typos
needed two runs. The editor now reports what a fix-it leaves standing:
items it touches are dropped, the rest move by its length change. Any
typed edit still withdraws them all.

Clicking a marked span on a phone opened CodeMirror's lint tooltip over
the range button, repeating the message and fix-its already listed under
the editor. The lint package has no option to turn it off, so the theme
hides it.

* fix(query): tidy empty-panel diagnosis wording and values seen

A diagnosis on the traces viz said no traces match and then counted spans;
it now names spans throughout, since those are what it counts. A filter
without a value reads "Without the duration filter".

Values seen listed arbitrary trace and span IDs. They are skipped for ID
columns and for any field whose five most common values cover under one row
in a hundred, where a sample says nothing. An empty value is written "",
as FanQL writes it, instead of a blank.

* fix(query): report an ungrouped zero count as empty on every viz

A count that matched nothing was an empty stat with a diagnosis, but a table
showed a bare 0 and no reason. An ungrouped count of zero is now empty on
every viz, with the same diagnosis.

* fix(ui): run a shared Explore link on open under StrictMode

The window check remembered the window it had seen, so StrictMode's
remount, whose unmount aborts the first run, skipped the second. The
unmount now forgets it.

* fix(ui): read chart axes the same way across charts

A bytes axis stepped in decimal bytes, so ticks read 95.4, 190.7 and
286.1 MiB; it now steps by 1, 2 or 5 of one binary unit and prints every
tick in it. A missing group value read blank on bars and (none) on time
series; both say (none), and a bar's drill still selects the stored
value. Bar axes and value labels printed 5 s where time series printed
5s; bars now use the time series style. Bar category labels get a few
pixels of room so the chart does not cut a label that fits.

* fix(dashboard): reject restoring a version the dashboard already has

Restoring a version whose spec matched the current one appended a
duplicate version on every click. The service now rejects such a
restore with already_current, as it did for the current version
number, and the history drawer disables the button with "Same as
current". When the server refuses anyway, the drawer says there is
nothing to restore instead of suggesting a conflict.

Tests that saved an identical spec before restoring now save a changed
one; the MCP test that asserted duplicate restores now asserts the
rejection.

* fix(api): keep a busy engine ready

/readyz answered 503 once right after a cold 24-hour schema request.
With two engine connections, long reads held both, so the readiness
probes waited out their timeout and the telemetry probe reported the
engine unhealthy. A probe that runs out its own time now reports the
engine degraded and busy, which keeps a 200, as publication contention
already did. Other errors still fail readiness. All three engine
probes share one timeout.

* fix(ingest): drop requests abandoned before their commit starts

An OTLP sender that times out while its request waits in the writer queue
(for example behind compaction backpressure) resends it. The writer still
committed the original, so every row of that request was stored twice; the
lab showed byte-identical span pairs and triples in exactly the minutes the
collector logged client timeouts.

A commit worker now drops each request whose context is done before the
commit starts, acknowledges it with the context error and counts it in
fanout_ingest_abandoned_total. The rest of a group still commits, and a
commit that has started completes, so at-least-once delivery remains only
inside that window. Group batches are merged by the worker after filtering.

* perf(query): skip totals and the previous range for an empty stat

An empty stat ran its query a second time for totals, and twice more with a
comparison, before the diagnosis started. The diagnosis shares the panel's
deadline, so on a busy server the stat often lost its hint where the same
query as a table kept it. Nothing shows totals or the previous range for an
empty panel, so they are no longer read.

Measured on 320,000 spans in one hour with an attribute filter that matches
nothing (seven runs each): the stat took about 1.01 s before and 0.74 s
after, the same as the table (0.69 to 0.73 s).

* perf(query): cache a metric's unit when its spellings agree

A metric whose series report one unit two ways, such as {packets} and
packets, or {thread} and no unit, never had its unit cached, so every
refresh of its panels looked it up again. Units now agree when they are
spellings of one unit or map to the same panel unit, and an empty unit
defers to the one reported. Agreeing units are applied and cached; units
that disagree are neither, as before.

* fix(api): fail health checks for a stalled engine, not a busy one

A wedged engine times out readiness probes just as a busy one does, so
it read as degraded with a 200 forever while /healthz stayed ok and
nothing restarted it. The engine now records when its read pool last
handed out a connection or finished a statement. A timed-out probe is
degraded only while the pool moved within the last 75 seconds, three of
the longest reads it allows; past that readiness answers 503, and
/healthz fails once a one-second ping confirms the stall, since an idle
pool records no progress either.

The three engine probes ran one after another with 5 s each, so a busy
engine kept /readyz waiting about 15 s, past the container check's 3 s.
They now run concurrently under one 800 ms deadline, below the 1 s a
Kubernetes probe allows by default. A probe counts as timed out on
context.DeadlineExceeded from the error or its context.

* fix(ingest): release an abandoned request's reservation at once

A request whose sender timed out behind compaction backpressure kept its
in-flight bytes until a commit worker reached it, so the sender's immediate
retry could be shed as over budget. Each submission now has one owner
transition: the sender abandoning it releases the reservation and rows at
once, a commit claiming it keeps them, and the dispatcher skips abandoned
requests when grouping.

Abandoned requests are not dropped between commit retries: a failed attempt
can already have published its directory, which the retry adopts by ID. The
abandoned counter's help text now says it includes requests cut off by
server shutdown.

* fix(ingest): make shed exports retryable by OTLP senders

Over-budget shedding answered gRPC with RESOURCE_EXHAUSTED and no RetryInfo,
which OTLP exporters treat as permanent, and OTLP/HTTP with a generic 503.
gRPC now attaches RetryInfo with a one-second delay, and HTTP answers 429
with Retry-After, so a shed request is resent instead of dropped.

* fix(ui): rerun the Explore draft when an absolute range is applied

The range menu recorded the window it asked for with the old relative
range still in it, but the URL drops the range once from and to are set,
so the change looked like navigation and the draft was replaced by the
entry's query. The requested window is now compared as the URL holds it.

* fix(ui): end the bytes axis search past the largest unit

At TiB the step rolled over to 1024 TiB and back again, so the tick
search never ended for axes past about 1000 TiB. The largest unit now
keeps growing, the search is capped, and byte steps are whole bytes, so
an all-zero axis no longer ticks at 0.5 B.

* fix(ui): drop whole-query problems when a fix-it is applied

A problem without a span stayed shown as an error after any fix-it,
even one that resolved it. Such problems now go with the edit; notices
without a span still stay.

* fix(query): show an ungrouped zero count as 0 with the diagnosis as a hint

A count that matched nothing was an empty panel, which hid a real answer.
An ungrouped count of zero is now an ok result on every viz, a stat or gauge
with a 0 total and a table with one 0 row, and carries the empty-result
diagnosis as a hint. A grouped count with no rows stays empty.

MCP still keeps such panels out of agent-built dashboards, since they hold
only panels that return data, and the preview summary names them.

* fix(ui): show a zero count's diagnosis as a panel note and allow saving it

A panel whose count matched nothing now shows 0 with the diagnosis in its
notes. Explore lets a new stat with a 0 total be saved, as a table with a 0
row already could.

* fix(ui): keep a stored (none) apart from a missing group value

Groups were keyed by their display name, so a value that is literally
(none) merged with missing values and a bar's drill took whichever row
came last. Groups are keyed by the stored value again; (none) is only
how a missing one is shown.

* fix(ui): keep the unit in headers of columns printed raw

A duration column's header dropped its unit whenever the unit was a
duration, but service and trace links print the value as it came, so
the unit vanished. The header now drops it only for formats that print
a duration in its own unit.

* fix(ui): dismiss hovered trace legend names with Escape

Escape closed the trace drawer while names opened by hover were still
showing, since only a focused legend entry caught it. A window listener
registered before any drawer now takes the Escape whenever the names
are showing, wherever focus is. Touch opens the names again.

* fix(query): compare a zero count with the previous range

A stat whose count matched nothing now shows 0, but it skipped the previous
range, so a drop from 1,234 to 0 showed no change. The previous range is
read for a zero count as for any other stat; empty panels still skip it.

* fix(engine): count only completed reads as progress

The read pool counted a handed-out connection and any returned statement
as progress, cancelled or not. Readiness probes the engine every few
seconds, so an engine whose statements all hang until their deadline
kept marking itself as moving and stayed degraded forever. Progress now
needs a statement that completes: a scan that succeeds or finds no
row, or rows read to their end without an error. Engine errors do not
count either, since a broken engine also returns them.

* fix(fanql): offer a seen-key fix only when the schema covers the query

The dotted-key suggestion compared with whichever schema was cached, a
recent sample that may miss a rare valid key in a longer window or another
namespace, yet offered a fix that would rewrite the key. Parse now accepts
the query's window and namespace. A cached schema for that window, or the
widest, and that namespace still offers the fix; otherwise the recent
schema only names the likely key. The notice says the key was not seen
recently rather than in this window.

* test(fanql): complete each severity level once whatever its stored case

* fix(ingest): answer OTLP/HTTP failures with a google.rpc.Status body

OTLP/HTTP requires a failed export to carry a serialized google.rpc.Status,
encoded as the request was. The endpoints answered with plain text. Every
failure now writes the status in protobuf, or in JSON for a JSON request,
with the gRPC code that matches the HTTP status.

* fix(ui): send the page's relative range to FanQL parse

Parse offers the dotted-key fix-it only when a schema is cached for the
query's window. Explore and the Logs and Traces pages now send their
relative range as window; an absolute range sends none. No page has a
namespace to send yet.

* fix(engine): tell a hung engine from an overloaded one

Liveness counted only successful reads as progress, so sustained
overload with every read timing out read as a hang: after 75 s /healthz
pinged, the ping could not get a connection, and the supervisor would
restart a working server in a loop.

Any read that returns, by success, error or its own deadline, now
counts as progress, since it shows the engine answers. The engine
tracks reads in flight by deadline and is stalled only when one runs 25
seconds (the longest read allowed) past its deadline: a wedged engine
never returns cancelled reads, an overloaded one returns them on time.
/healthz fails for a stall, or for an idle engine whose ping fails; a
ping that cannot get a connection while reads within their deadlines
hold them all is overload and stays alive. Readiness still reads a busy
engine as degraded, fails a stalled one, and stops waiting for probes
shortly after its deadline.

* chore(ui): rebuild embedded assets

* test(ui): wait for the history list before restoring a version

* fix(engine): count only reads that ran as progress, and track publishers

An engine that failed every read kept recording progress, since any
return counted, so /healthz never pinged it. A read now shows progress
only when it succeeded, found no row, or stopped because its context
ended; other errors may come from a broken engine.

Reads without a deadline have none to overrun and are no longer
tracked; reads that have one are tracked only while the engine runs the
statement, so rows a caller holds afterwards cannot read as a stall.

A Parquet publication that holds readers out past its 30 s swap budget
plus the grace is a stall too: reads then fail at the gate before they
reach the engine, while the ping, which takes no gate, still answers.

* test(e2e): update darwin performance baselines for unitless duration headers

* docs(proposals): accept 0005 dedicated attribute columns and batch format 5

Typed columns for the stable OpenTelemetry keys that rules A and C
select at semantic conventions v1.44.0: 18 attributes of the stable
HTTP and DB duration metrics and 29 stable service, deployment,
container and k8s resource keys. The seven format-4 merged columns go
without aliases, format-4 data is deleted rather than converted, and
the evaluation thresholds are fixed before any measurement.

* fix(engine): exit for a restart when the engine stays stalled

/healthz fails when the engine stops honouring read deadlines or a
Parquet publication holds reads past its budget, but only Kubernetes
acts on a failing liveness probe. Docker, Compose and systemd restart
only on exit, so a stalled Fanout stayed stalled.

A watchdog now checks the stall each second. Once the engine has stayed
stalled for another overrun grace (25 seconds), it logs the reason and
delivers ErrStalled on Fatal, the path that already flushes received
telemetry and exits with status 1. Overload is no stall, the watchdog
fires once, and it stops with the run context.

* docs: run Fanout under a supervisor that restarts on exit

Fanout exits with status 1 on a fatal engine error or a stalled engine.
State the restart policies that act on that (Docker and Compose
unless-stopped, systemd on-failure, Kubernetes), note that Docker's
HEALTHCHECK alone does not restart a container, and add a restart
policy to the docker run samples.

* fix(engine): restart only for stalls in Fanout's own work

A user query whose per-row cost is quadratic, such as levenshtein over
long strings, runs a DuckDB statement long past its deadline because
interrupts are checked only between chunks. Liveness failed on that
overrun and the exit watchdog then restarted the server, so anyone who
can run SQL could loop restarts, including through saved panels.

A read overrunning its deadline by the grace now degrades readiness
(queries check), logs a warning with a hash of the query, never its
text, and counts fanout_duckdb_statement_overruns_total. Liveness and
the watchdog act only on a Parquet publication holding reads for ten
minutes, longer than an NFS hard mount's major timeout so slow storage
finishes its swap, or an idle engine that fails a ping while nothing
holds the read pool. The watchdog pings apart from its ticks so a hung
ping cannot hang it. Every read is now tracked, with or without a
deadline, so a held pool is never mistaken for a dead engine.

On a stall exit the engine handles stay open, since native code may
still run; received telemetry is flushed first as before.

* fix(engine): reject SQL whose per-row cost is quadratic

DuckDB checks for interrupts only between chunks of up to 2,048 rows,
so a function costing the product of two input sizes per row can run
one chunk for most of an hour. Measured per chunk of 20,000-character
values: levenshtein 21 minutes, damerau_levenshtein 60, a computed
SIMILAR TO pattern 25, a literal 10,000-byte regular expression 13 to
27; a 256-byte literal pattern 0.06 s. Linear functions stay bounded by
the memory their values occupy.

Telemetry SQL now blocks levenshtein, editdist3 and
damerau_levenshtein, and requires the pattern of LIKE, ILIKE, GLOB,
SIMILAR TO and the regexp functions to be a literal of at most 256
bytes. FanQL's own typo suggestion still uses levenshtein internally.
User SQL statements are now tracked as reads in flight.

* docs: restarts follow only Fanout's own engine stalls

/healthz and the exit fail only for a stalled Parquet publication or an
idle engine that cannot answer a ping. A query running past its deadline
degrades /readyz, logs a hash of the query and counts an overrun.

* feat(semconv): generate the format 5 dedicated attribute keys

Add internal/telemetry/semconv with the 47 keys that batch format 5 will
store in dedicated columns, and internal/cmd/semconvgen, which applies
rules A and C of proposal 0005 to the semantic conventions v1.44.0
registry as resolved by weaver 0.25.1, the version that tag pins.

`just semconv-generate` downloads the tag archive and the weaver release,
verifies both against pinned sha256 sums, resolves the registry and
rewrites keys_gen.go. The generated header records the tag, the archive
hash and the weaver version. The generator reports added, removed and
changed keys against the committed list, since any change needs a new
batch format version.

Offline tests compare the list with the proposal's keys written out by
hand, check types, bits and column names, and check that no column
collides with a span, log or metric row column. Nothing in the product
reads the keys yet.

* docs(proposals): extend 0005 to logs, metrics, events and links

Format 5 carries every storage change in one break: typed metric kinds,
histogram, exponential histogram and summary lists, typed exemplars,
span events and links, the stable resource columns on every signal, and
a declared sort for logs and metrics after the OpenTelemetry Collector
ClickHouse exporter. No JSON remains in telemetry storage. Evaluation
adds fixed logs and metrics queries with thresholds set before any run.

* fix(engine): count only connections pings hold as theirs

The watchdog treated one in-use connection as its own ping's, but a
ping still waiting for a connection holds none. With a one-connection
pool held by a caller reading rows, the wait looked like an idle engine
that cannot answer, and the watchdog exited. Pings now count the
connections they hold, and the pool is held whenever any other
connection is in use.

Overrun reports hash the query only when it overruns, not on every
read.

* fix(query): bound patterns in filters and variables, block Jaro

Structured filters allowed regexp_matches and LIKE with a column or an
unbounded variable as the pattern, and Jaro similarity is quadratic on
repeated text (146 s for one chunk of 200,000-character values).

Jaro similarities join the blocked edit distances. A filter's pattern
must be a literal of at most 256 bytes or a variable, never a column,
and a variable used as a pattern, in a filter or in panel SQL, fails
with a clear problem when its value is longer than 256 bytes. The
telemetry SQL boundary accepts bound values as patterns, since its
callers now check them; filters with a variable pattern had failed to
check since the boundary required literals.

* perf(engine): take the read connection before the Parquet gate

A reader queued for a pool connection reads no Parquet yet, but it held
the snapshot gate while it waited, so a publisher also waited out the
pool's queue. Readers now take the connection first. No gate holder
waits for a read-pool connection, so the new order cannot deadlock.

* perf(intelligence): read fixed statements on the typed path

Log-pattern detection and alert environments ran their own fixed
statements through the untrusted-SQL path: parsed and guarded on every
pass, read as row maps, counted as user SQL, and, for log patterns,
unbound to a window, so every log file was scanned. They now use
QueryContext with typed scans, and log patterns carry the detector
window so the snapshot binds only overlapping files.

The SQL-comment guard existed only because that validator rejects
comments, so it goes with the row-map conversion helpers.

* test(api): give the readiness contention test a read pool

Reads now take a pool connection before the Parquet gate, so a Duck
without one cannot stand in for the engine.

* refactor(engine): drop branches for a Duck without a repository

NewDuck has required a telemetry repository since #207, so maintenance,
compaction, cache refresh, the version rollup and the window checks no
longer test for one. Trace candidates always use the completed-batch
index; the raw ranking query survives only as the tests' oracle.

Snapshot binding keeps one guard for fixture engines built over mutable
querytest tables, which the query package's metric tests rely on.

* refactor(engine): delete unused query helpers

LatencyOverview, LogsSamples, Throughput, ServiceThroughput, ErrorRoutes
and ErrorRouteDetails had no production caller. They go with their row
types and tests.

* refactor(engine): version every DuckDB cache table together

Rollup tables and anomaly history were rebuilt when a required column
was missing, version history had its own column check, and the batch
caches used a version number. A column check misses a change in what
the rows mean, and three mechanisms can disagree. One version now
covers every table DuckDB keeps: a mismatch drops them all, rollup
progress included, and they rebuild from Parquet. The version moves to
8 so existing rollups rebuild once under the new rule.

* fix(query): make JSON columns with nonfinite numbers parseable

DuckDB writes NaN and infinities inside a serialized column as bare
tokens no JSON parser accepts. Only the direct SQL path normalized
them; panel frames passed the text through. Frames now normalize JSON
columns to the ProtoJSON strings with the engine's NormalizeJSON.

* refactor(engine): remove the direct SQL path

ExecuteSQL and querySQL had no production caller once the detector and
alerts moved to typed reads. User SQL reaches the engine only through
PrepareTelemetrySQL, from SQL panels and MCP query_telemetry. The guard,
row-cap, variant and boundary tests now run through that path the way
panels do, and the string validator, RowMap conversion and the
fanout_query_* metrics that only the direct path recorded go with it.
The read deadline liveness relies on is now named longestRead.

* test(query): commit metric fixtures through the telemetry store

The metric tests inserted rows into mutable querytest tables and read
them through an engine with no repository, which only worked because
snapshot binding skipped such engines. The fixture now buffers metrics
and commits them as real batches before each read, so panels bind and
prune Parquet as in production. The 420k-row distribution fixture is
generated in Go, and the readbench fixture uses a real engine.

Ingestion stores a nonpositive metric time as the ingestion time, so
the two nanosecond-boundary tests now sit just after the epoch rather
than before it.

* test(intelligence): run detector fixtures on a real engine

The recovery and watermark tests ran the detector over a hand-made logs
table through an engine with no repository. They now use a real engine
and commit their logs as batches. The detector depends on a small Engine
interface, which *engine.Duck implements, so a test can fail one
statement by wrapping the real engine instead of the DuckDB driver.

* refactor(engine): bind every windowed read to the snapshot

No engine lacks a repository now that the query and detector fixtures
commit real batches, so snapshot binding drops its last guard.

* fix(engine): keep anomaly history across cache version changes

anomaly_log holds detector output that nothing rebuilds, so dropping it
with the read caches erased chart anomaly history on every version
change. The drop now keeps it; its schema changes need their own step.

* fix(engine): hold no read connection while a publisher blocks the gate

Taking the connection before the gate kept publishers from waiting out
the pool queue, but a reader held back by a publisher then kept a pool
slot for the whole drain, so readiness pings and annotation reads queued
behind it. Readers now wait for the gate without a connection, take
one, and enter the gate without waiting, returning the connection and
waiting again if a publisher slipped in.

* fix(engine): reject GROUP BY ALL in telemetry SQL

DuckDB 2.0.0 groups GROUP BY ALL keys on the raw VARIANT without
collation, so equal attribute values from different key dictionaries
split into separate groups. Explicit grouping keys are collated and
stay allowed. Hash-based aggregates have the same defect; the guide
documents them and a test pins the upstream behaviour.

* refactor(telemetry): write Parquet from one column registry

Each signal's columns are declared once in columns.go with their physical
type, encoding, shredded keys, accessor and clean-view projection. The
writer, compaction, the schema batch, the loader's format check and footer
bounds all derive from it, replacing the tagged row structs and the
reflection-driven writer. Files are byte-identical to format 4 apart from
the footer's root schema name, now the signal name. Golden schemas pin
format 4.

* refactor(engine): derive telemetry views from the column registry

The raw relation projection, the clean views and the mutable query test
tables are generated from the telemetry column registry instead of three
hand-written column lists and a projection string repeated in three
places. The generated view SQL matches the previous text.

* refactor(query): take catalog columns from the telemetry registry

Catalog fields name clean-view columns of the registry, which supplies
their types and each signal's time column; an unknown name fails at
startup. The catalog keeps display order, descriptions and units, so the
schema API and generated FanQL reference are unchanged.

* refactor(engine): read each signal's event-time column from the registry

Resource inventory, deploy annotations and the query catalog named the
clean-view time column by special-casing spans; Signal.Clock derives it
from the registry's event-time column.

* refactor(telemetry): derive per-signal lists from the registry

Snapshot binding, relation aliases, the newest-event scan, compaction
plans, store stats, batch loading and verification iterate
telemetry.Signals instead of repeating the three signal names.
BatchMetadata.Signal maps a signal to its row count and event-time
bounds. The query catalog maps a Bool registry column to a bool field
type instead of rejecting it at startup.

* fix(telemetry): write variant metadata deterministically and dictionary-encoded

Canonical maps are written with sorted keys, so equal rows encode to
equal bytes whatever Go's map order; the cached shared-resource cells
are shredded from pre-encoded sorted bytes for the same reason.

The VARIANT metadata leaf is dictionary-encoded. The variant writer
shares one growing name dictionary across a file, so every row repeated
a blob of up to 4 KiB; compaction interleaved the blobs of its inputs
and the compressor stopped finding them adjacent. On 50 merged span
batches of 20,000 rows with mixed attribute sets, input files shrink
from 76.9 MB to 62.7 MB and the merged file from 125.0 MB to 79.4 MB;
attributes.metadata goes from 46.3 MB to 1.2 MB. Commit throughput is
unchanged.

* refactor(telemetry): order every signal by a key extractor and comparator

Signal.Sort, a footer claim that only spans used, becomes Signal.Order: a
list of key parts, each a stored column or that column's time bucket.
One compiled order sorts a batch at commit, drives the k-way merge in
compaction and is checked by offline verification for every signal. The
footer's sorting_columns claim is derived from it and stays empty for an
order with a bucket part, and Signal.Traces separately marks the signal
that carries trace.fidx.

Logs now sort by five-minute bucket, namespace, service and time, and
metrics by namespace, service, name, hour, series and time, following
the OpenTelemetry Collector's ClickHouse exporter. Compaction merges them
on those keys instead of concatenating inputs.

Commit sorts a permutation over per-row keys instead of cloning the rows;
text parts with few distinct values are ranked so most comparisons are
integer ones. Instants and nanosecond counts are DELTA_BINARY_PACKED: in
sorted log runs plain timestamps compressed 2.3x worse than in arrival
order.

On 50,000-row commits against the previous order: span peak heap falls
from 37 to 17 MiB and bytes per row by 6%; log bytes per row fall 12%
while commit time rises 18%; metric commit time rises 37% at equal size.
The added time is the sort and map access in sorted rather than
allocation order.

* feat(telemetry)!: write batch format 5

Format 5 copies the 47 semconv v1.44.0 keys of proposal 0005 into typed,
nullable, dictionary-encoded columns: _a_<key> for the 18 span keys on
spans and _r_<key> for the 29 resource keys on every signal. A column
holds a value only when its OTLP type matches the key's; any other value
stays in the map, sets the key's bit in the private _spill mask and is
counted in the footer under fanout.spill.<key>. The maps keep every key,
and keys with a column are no longer shredded. BatchMetadata derives a
per-signal spill mask from the footer for later spill-aware reads.

Metric points store the user-visible type (gauge, counter, updown,
histogram, summary), a nullable temporality and value, typed hist_bounds
and hist_counts lists, native exponential layouts (exp_scale, zero
bucket, offsets and counts), summary quantiles, hist_min, hist_max and
flags; an absent histogram sum is NULL. Span events and links and metric
exemplars become parallel typed lists with VARIANT attributes, so no
storage column holds JSON. Ingest rejects explicit histograms whose
bounds are not finite and strictly increasing or whose counts do not
match, and no longer converts exponential histograms or merges
deprecated attribute spellings.

The registry gains nullable scalars, LIST columns of any element type
including VARIANT, dedicated-key columns and parallel list groups. The
merge copies list columns by whole rows and sums spill counts; offline
verification checks every dedicated column against its map, the footer
spill counts, parallel list lengths and the columns of each metric kind.

Removed with no aliases: http_method, http_status_code, http_route,
db_system, rpc_method, service_version, deployment_env, events_json,
links_json, metric_type, monotonic, hist_bounds_json, hist_counts_json
and exemplars_json. Format-4 batches are refused as unsupported.

Consumers move to the new layout: the version rollup reads
_r_service_version, histogram quantiles derive exponential bounds in SQL
with the removed conversion's rules, catalog, filters, FanQL examples,
default dashboards and MCP guidance use attribute keys, the trace detail
view reads the event lists, the read caches rebuild, and the generated
references are regenerated.

* build(ui): rebuild assets for format 5 span details

* fix(query): count histogram points without buckets

OTLP allows a histogram point with neither bounds nor bucket counts, and
ingest accepts it. Point validity dropped it, losing its count and sum.
It now counts toward count, sum, rate and avg; its empty bucket list
adds nothing to quantiles.

* fix(telemetry): leave null key values unspilled and verify spills by type

An empty OTLP value, or an absent array or key-value list, becomes a Go
nil. It has no type, so it no longer sets the key's spill bit; the
column stays NULL. Before, the spilled row then failed verification and a
valid batch read as corrupt.

Spill counts accumulate across every write into one file instead of
keeping only the last call's. The verifier also rejects spill bits outside
the signal's own keys and checks the variant type of each map value: the
key's type with an equal column, another type with the spill bit, or
null with a NULL column.

* docs: describe batch format 5 where format 4 was current

AGENTS.md states format 5 as the only accepted format, with dedicated
columns, spills, typed lists and per-signal sort order, and widens the
verification rule to every signal. The view and merge comments, the
storage concept page and test messages no longer name format 4.

* perf(telemetry): entropy-code zstd literals in Parquet pages

The fastest zstd level stores a block's literals raw when it finds few
matches. Random trace and span IDs are mostly literals, so their hex text
was written at full width. Entropy-coding all literal blocks shrinks it.

* test(telemetry): record each signal's Parquet file digest

Writer changes made for speed must leave the stored bytes unchanged. A
fixed set of rows covering sparse dedicated keys, spills, shared and
absent resources, nested variants, lists, signed times and every metric
layout is written uncompressed and compared against recorded digests.

* perf(telemetry): append dedicated-column NULLs in runs

Most rows hold few dedicated keys, yet every row levelled and appended one
NULL per absent column. A row now visits only the columns it fills, and
each column appends its gap of NULLs from a template levelled once. The
written values, and so the file bytes, are unchanged.

* perf(telemetry): write the trace index alongside the span file

Format 4 wrote trace.fidx concurrently with the span Parquet file; format 5
wrote it first, serially. The index needs only the sorted trace hashes, so
it is written again while the Parquet file is.

* perf(telemetry): radix-sort integer sort keys

When a batch's sort parts are all integers or ranked text and, offset by
their minimums, pack into 128 bits, rows are ordered by stable LSD radix
passes over 16-bit digits instead of a comparison sort. Log and metric
keys fit; span keys, which compare span IDs as text, keep the comparison
sort.

* perf(telemetry): size grouped batches before merging requests

Merging a group's requests appended rows to empty slices, so every signal
regrew and recopied its rows several times. Each signal is now allocated
once at the group's row count.

* perf(telemetry): gather variant map entries in one pass

The variant encoder collected a map's keys, sorted them and looked each
value up again. It now gathers key and value together and sorts only maps
with more than one entry. The encoded bytes are unchanged.

* test(telemetry): cover empty and absent maps in file digests

* perf(telemetry): walk span attributes once per row

The dedicated columns and the attributes variant each walked a span's
attribute map. The key writer now gathers each row's entries once, in key
order, projects the dedicated columns from them and the attributes
variant is written from the same entries. The file bytes are unchanged.

Commit-only span writes measured 2-3% faster; end-to-end span ingest
4% faster (median of five interleaved runs).

* perf(telemetry): hash metric series identities without allocating

Ingest hashes every metric point. The identity encoding allocated its
buffer, a key slice per map and a boxed copy of every string. It is now
built in a stack buffer with strings appended directly; the encoding and
so every series ID are unchanged.

End-to-end metric ingest allocated 30% less and ran about 3% faster
(median of five interleaved runs).

* perf(telemetry): sort small batches by comparison

A radix pass clears and sums 65,536 counters, which costs more than a
comparison sort for a batch of a few rows.

* perf(telemetry): write a file's columns in parallel lanes

Lanes own disjoint columns and walk each row group on their own,
joining only at row group boundaries. The first lane writes the
dedicated columns, _spill and the attributes and resource variants,
which share one gathering of each row's entries; up to three others
share the scalar and list columns. Each lane encodes its columns' last
pages before the row group flush. Output is byte-identical to a single
lane.

* perf(telemetry): keep only the sort key parts an order needs

A batch's sort keys were a fixed 96-byte struct per row. They are now
a flat slice of the order's integer parts, 24 bytes a row for spans,
32 for logs and 48 for metrics, with a text column per unranked text
part allocated only when it has more than maxRanked values. Merge
cursors and verification keep the fixed key.

* test(telemetry): compare two write lanes and tighten the ordering limit

Two lanes take a distinct path, so the lanes test now compares 1, 2
and 4 lanes. The ordering allocation slack drops to what the span ID
rank map measures, so a copy of the keys no longer passes. The writer
notes that early page flushes rely on no bloom filters or encryption.

* feat(engine): read standard keys through their dedicated columns

Snapshot binding rewrites attributes['k'] and resource['k'] subscripts of
the 47 standard keys to their typed columns when the map binds to a
signal relation, directly or through subqueries and CTEs that pass it on
with a star, and to TRY_CAST of the map when a subquery projects the map
alone. Rewritten select items keep their result names.

Snapshot sources split each signal's files by the spill masks of the
keys a statement reads: spill-free files read the columns alone, so
their predicates reach Parquet statistics; the others read
coalesce(column, TRY_CAST(map['k'] AS type)). Unpinned views coalesce
every key. Version history reads service.version through the same path,
so a numeric version is still a version; the read cache version moves
to 10.

Telemetry SQL rejects the storage columns _a_*, _r_*, _spill and
_trace_hash and names the key to write instead.

* fix(query): compare standard integer keys as numbers

A lookup of a standard integer key now has the number type, so a
variable compared with it binds as DOUBLE instead of text, and the
filter gate names the key to write for a storage column. FanQL
http.response.status_code>=500 no longer counts the text "200".

* refactor(query): drop casts of standard keys that are already typed

* docs(proposals): record the format 5 read path

* docs: describe ingest conversion of standard key values

Claude-Session: https://claude.ai/code/session_01XkoYQXHPWxNt4c8sR6DNet

* feat(telemetry)!: convert standard key values to their column types at ingest

A dedicated column now holds its key's value converted to the semconv
type, and the column is the key's read value everywhere. Integer keys
take int64, integral float64 within int64 range, and strings that
strconv.ParseInt reads in base 10 as sent. String keys take strings and
the strconv text of int64, bool and float64. Anything else leaves the
column NULL and is counted in the footer as a spill. The map keeps every
value as sent.

Reads no longer touch the map for a dedicated key: the spill split of
snapshot sources, the coalesce projection, the unpinned-view coalesce
and the TRY_CAST rewrite are gone. A subscript of a map that a subquery
projects without its columns reads the map unchanged. The private
_spill column and the batch metadata s…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant