Skip to content

perf(query): answer resource lookups from a per-batch inventory - #327

Merged
vishr merged 1 commit into
mainfrom
perf/resource-inventory
Oct 17, 2026
Merged

vishr merged 1 commit into
mainfrom
perf/resource-inventory

Conversation

@vishr

@vishr vishr commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Summary

The default dashboard's versions and check_states panels decoded every span's resource in the window to find a handful of distinct resources. For versions that also meant every log record's and metric point's. On an 8-vCPU host running the OpenTelemetry Demo, each took 5–6.5 s alone and missed the 10-second deadline when four panels ran together.

  • Cache. Completed-batch caching now also records one row per batch, signal, namespace, service and distinct resource, with its earliest and latest event time.
    • The read cache version goes to 6, so all private read tables rebuild once.
    • Rows are written in the same transaction as the batch marker.
    • They move with compaction and leave with retention, through the existing cacheBatches, transferBatchCache and stale-deletion paths.
  • Relation. resource_inventory is a new panel-SQL relation: one row per distinct resource observed in the panel's logical window, with first_seen and last_seen inside that window.
    • Batches wholly inside the window use cached rows.
    • Boundary and uncached files are grouped exactly from Parquet under the same pinned snapshot and read transaction.
    • The window is the panel's own, not the expanded lookback file window.
  • Surface. The relation is described under sql_relations in the telemetry schema, the MCP and AI guidance, and the generated docs. It is dashboard-panel only; unscoped SQL rejects it.
  • Panels. versions and check_states (service_hosts) read it, with identical output.

Evidence

  • One-hour Demo snapshot, frozen window, frames compared row by row: 16 versions rows and 61 check_states rows, all identical.

  • Warm panel time through the real executor:

    Panel Before After
    versions 2.4 s 0.8–0.96 s
    check_states 1.7 s 0.56–0.72 s
  • Cache catch-up built 6,937 inventory rows in 8.6 s across 13 passes.

  • Proof tests cover:

    • inclusive start and exclusive end, including signed nanoseconds;
    • resources that appear only outside the window;
    • unknown footer statistics;
    • late publication;
    • marker and reader races;
    • compaction levels and retention;
    • version rebuild and typed VARIANT identity;
    • SQL boundary rejections.
  • Codex adversarial review: approve, no material findings.

Verification

  • go test and go test -race on ./internal/query/... ./internal/panel/... ./internal/dashboard/...: passed, including after rebase onto main.
  • Docs and dashboard-eval generation checks: passed.
  • just check (pre-push): passed.

Qualification on the 8-vCPU Demo host follows the release.

The default dashboard's versions and check_states panels decoded every
span's (and for versions every log record's and metric point's) resource
in the window to find a handful of distinct resources. On an 8-vCPU host
running the OpenTelemetry Demo each took 5 to 6.5 seconds alone and missed
the 10-second deadline when four panels ran together.

Completed-batch caching now also records one row per batch, signal,
namespace, service and distinct resource, with its earliest and latest
event time (read cache version 6). Rows are built in the same transaction
as the batch marker, move with compaction, and leave with retention.

Panel SQL can read them through a new relation, resource_inventory: one row
per distinct resource observed in the panel's logical window, with
first_seen and last_seen inside that window. Batches wholly inside the
window use cached rows; boundary and uncached files are grouped exactly
from Parquet under the same pinned snapshot. The relation is available to
dashboard panels only, is described in the telemetry schema and MCP
guidance, and is rejected by unscoped SQL.

versions and check_states read it, with identical output. On a one-hour
Demo snapshot the warm panel time fell from about 2.4 to 0.9 seconds
(versions) and 1.7 to 0.6 seconds (check_states).

Claude-Session: https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW
@vishr
vishr merged commit ec0f9bf into main Oct 17, 2026
14 checks passed
@vishr
vishr deleted the perf/resource-inventory branch October 17, 2026 00:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant