Repository navigation
feat(dashboards)!: correct metric measures and a system overview (milestone 5) - #297
Merged
Merged
Conversation
…mmaries, system page
…lters do not fix the metric
…hows silence as healthy
…tonicity and start time Ingest now stores each metric point's OTLP data kind (gauge, sum, histogram, exponential_histogram, summary) with its aggregation temporality, whether a sum is monotonic, and its start time. `sum_delta` is gone. A sum or histogram with unspecified temporality is invalid OTLP and rejects the export with InvalidArgument, as other invalid input already does. The metrics view's `type` becomes the semantic kind (counter, updown, gauge, histogram, summary) and gains `temporality` and nanosecond `start_time`. Batch format 4 is the only accepted format. Format 3 batches are refused by their metadata before cleanup, with every file preserved; a partially removed retired batch still opens.
…of failing every query The int64 extremes are DuckDB's timestamp infinity sentinels, so make_timestamp_ns rejects them. One data point carrying such a start time made every query that read the metrics view's start_time fail for its window. TRY maps it to NULL; every other value, including signed nanoseconds, stays exact.
… negatives The old conversion emitted one bound per positive count, so counts never had one more entry than bounds and every exponential histogram was dropped by metric histogram queries. Zero and negative counts were discarded. Negative buckets now keep their own negative bounds below the zero bucket, and counts always have one more entry than bounds and sum to the point's count. A bound that overflows float64 or underflows to zero at extreme scales and offsets is removed, merging the two buckets it separated, so no count is lost and bounds stay strictly ascending.
Dropping each overflowing bound by shifting the slices made a point with many such buckets cost quadratic time, so one export at scale -10 could hold an ingest core for minutes. One pass merges each dropped bound's upper bucket into the bucket below it, with the same result.
…ger stops startup A crash while retention or compaction removed a retired directory could leave it without metadata.json, and the pre-cleanup format check then refused to open the store until the directory was removed by hand. Such a directory has no format left to preserve: cleanup reclaims it, or compaction recovery reports the missing input through its runbook.
…, with stored temporality One SQL builder computes, per series and point, the increase since the previous point and the elapsed time. A series is namespace, service, name, unit, scope, attributes and resource. Points with the same timestamp keep the latest ingested. A cumulative series resets when its value or count falls, when two reported start times differ, or when any histogram bucket decreases; a bounds change re-bins the previous counts onto the new buckets when every earlier bucket fits, and resets otherwise. Delta series contribute their values. NaN, infinite values and unordered bounds are excluded. Metric reads look back 10 minutes, in both the first window predicate and the pinned snapshot window, and clip output to the requested range. Heatmap and histogram metric panels use the builder and the stored temporality, decode bucket JSON only for histogram rows, and rank the "Other" fold among series that contributed. The panel `temporality` option is removed; a spec that still sends it fails validation. Exponential histograms keep a fixed zero bucket, so a new smallest value or a first negative value no longer looks like a reset.
…eshold A NaN, infinite or negative zero_threshold is invalid OTLP. It made the fixed zero bucket fold away, so the moving bounds came back and a new smallest value read as a reset. The export is rejected with InvalidArgument, as ingest does for other invalid input.
…ase B gates a metric fix
…nter rates and stale-series rules Metric measures on `value` work per series, then across the series in each group. rate() is each series' increase divided by the time it covers, so a counter exported every 60 s reads the same per-second rate in 10 s and 1 h buckets. sum, avg, min, max, last and percentiles follow the spec's table for gauges, updown sums and counters; a measure that does not fit a kind returns no value, never zero, and the empty-panel diagnosis names the kind and the measures that fit. Other numeric fields keep per-point aggregation. last(value) sums each series' latest point strictly before the bucket end, counting only series that reported in the last 5 minutes, so stopped or restarted containers drop out. Ingest stores a series_id, an xxh3 hash of a canonical typed encoding of the series identity, so queries partition by one integer. One series builder is shared per kind set; measures that never compare points skip the lookback. An aggregate metric query must fix the metric per group with a single-valued name filter or `name` in `by`; All and multi-value variables do not count. Metric buckets and table trends share the Unix epoch origin; automatic metric buckets are at least one minute.
…ntiles For metric histograms, rate() is each series' observation-count increase over the time it covers, sum(value) the increase of the observations' sum, and avg(value) that sum increase over the count increase. Percentiles interpolate linearly over the per-bound increases summed across the series in each group, as Prometheus does: the lowest bucket starts at 0 when its upper bound is positive, an observation in the overflow bucket returns the highest finite bound, an empty total returns no value, and negative bounds from exponential histograms stay negative. Histogram measures share the existing increases builder. Bucket lists unnest in the select list and pre-aggregate before interpolation, as the histogram and heatmap renderer now does, avoiding a lateral join; on 160-bucket exponential histograms this cuts peak memory for one panel from 2.28 GB to 660 MB. The empty-panel diagnosis suggests rate(), avg(value) or p95(value) for histograms.
get_telemetry_schema reports each metric's semantic type (gauge, counter, updown, histogram, summary) and temporality, and the metrics catalog gains temporality and start_time. The agent's spec guide carries the measure table per kind, the elapsed-time rate, last(value) staleness and the one-metric rule, with a counter rate() example and a histogram p95(value) example that execute. The dashboards guide documents metric kinds, measures, the name rule and the known histogram percentile approximations.
…ss-series totals never dip
… and last against an independent oracle A readbench test, run when FANOUT_PANEL_BENCH_DATA_DIR names a format-4 data directory, compares the panel executor bucket by bucket with SQL written directly against the metrics view: rate() of system.network.io by direction, p95(value) of a histogram, and last(value) of system.memory.usage by state. Derived Parquet lives under the query directory.
… at the live edge
…rvals Each delta of a cumulative series now covers the interval since the previous point. Its rate is spread over every bucket that interval overlaps, as a time-weighted mean, and its increase is prorated by overlap; histogram count, sum and per-bound increases are spread the same way. A series therefore contributes to every bucket its reporting interval spans, so a total across machines that report every minute, every five minutes or at drifting times no longer dips, and a single counter reads the same rate in every bucket. At the live edge, rate() carries a series' latest rate for up to five minutes while reports are in flight; increases are never extended. Gauge and updown sums and averages use a series' latest earlier value, at most five minutes old, in buckets where it has no point. Carry candidates are generated only for uncovered buckets, and one spread expansion is shared per kind set. The demo-data cross-check compares rate() and last(value) exactly and quantiles within a relative 1e-12, because spread shares are summed in a different order.
…metric spreading Spreading every delta over all the buckets its interval spans let two points of one series a day apart expand into thousands of rows per query, so many sparse series could exhaust memory. A delta longer than the 10-minute lookback is now a gap: its whole increase belongs to the bucket of its later point at increase / elapsed, so each delta expands to at most the buckets ten minutes span (61 at 10 s).
…ures ingest separately
Opt-in readbench generation tiles the replayed demo to a target spans per second and duration, scaling span, log and app-metric traffic while host and container metrics expand to a fixed host and container count at their source cadences. Every identity rewrite recomputes series_id, signed times shift (ingested_at and ingested_unix_nano together), a present metric start never shifts onto the absent sentinel, output needs twice its estimated size free, and an atomic manifest marks complete output.
GET /api/system builds a read-only dashboard from the telemetry in the selected range and namespace, in question order: reachability from httpcheck (each check names the host it ran from and shows "no recent result" after two silent intervals), failing or slow services (health trend with deploy markers and the service map, now including external dependencies seen only from client spans), hosts and containers running out (worst first, 80/90 thresholds, "not reporting" after two silent intervals), and versions. A section without data is one text panel naming the Collector receiver to add. Generated SQL runs through the same read-only guard as authored SQL panels. A SQL panel returning NULL or non-integer time cells now returns a panel error instead of crashing the process, and external-peer discovery is bounded to the selected event-time window, so results no longer depend on Parquet file grouping.
…opens it The per-user default dashboard "System overview" already answered whether services fail or slow down, so the separate generated GET /api/system duplicated it. It is removed, with its builder, route, authorization entry and generated reference. "System overview" now follows its existing panels with reachability from httpcheck (check vantage, "no recent result" after two silent intervals), the service map, hosts and containers running out (worst first, 80/90 thresholds, "not reporting" after two silent intervals) and versions. Each new panel's description names its Collector receiver. "/" always opens /dashboards, with or without an agent. SQL panels read their window plus the ten-minute metric lookback, so the default dashboard's CPU-rate and cadence queries see their previous points.
The uncalled-services label and compact card text could extend past the boxes behind them, so axe could not determine their contrast. The label now has an opaque background, line height and padding of its own, and compact cards keep a 0.9 minimum scale with 1.4 line height so names, glyphs and metrics stay inside each card.
…os end to end The e2e seeder now also sends, over OTLP with the Collector's real scope names, a filesystem above 90%, a failing external dependency, an httpcheck target that is down, a check and a host that stop reporting, and a new service.version. A Playwright spec opens /, which reaches the default System overview, pins the seed's hour, and finds each scenario within two refreshes. Global setup lists dashboards before creating fixtures, as a first visit does. The Send telemetry guide gains 'Watch hosts, containers and endpoints': a Collector configuration with hostmetrics, docker_stats and httpcheck (run from a second machine), and the limits, including that Fanout cannot report its own machine failing. System overview panel descriptions link to it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Milestone 5 of agent-built dashboards: metric measures that are correct, and the default dashboard as a view of the whole system.
Correct metrics
series_id. Format 3 is refused before cleanup, with its files preserved.typeis the semantic kind:gauge,counter,updown,histogram,summary.rate()on counters is the per-second increase over the time it covers, handling resets. A delta is spread over the buckets its interval spans, so totals across machines that report at different times do not dip. A delta longer than the 10-minute lookback is a gap.last(value)takes each series' value at the bucket end and drops series silent for more than 5 minutes.rate(),sum,avgand percentiles come from bucket increases, with Prometheus-style interpolation.System overview
httpcheck, naming each check's host and showing "no recent result" after two silent intervals;/always opens/dashboards.hostmetrics,docker_statsandhttpcheck.Fixes
Deferred to a later milestone: a scale benchmark, per-minute summaries and metric query speed.
Contract changes
typevalues change, andtemporality,start_timeandseries_idare added.temporalityoption is removed.InvalidArgument.Verification
just checkjust test-race(29 packages)just e2e: 68 passed, including the new six-scenario System overview spec and the accessibility specsrate(),last(value)and histogram p95 equal an independent SQL oracle bucket for bucket on replayed demo data