Skip to content

feat(dashboards): backlog robustness, SQL boundary, accessibility and performance guards (milestone 4) - #295

Merged
vishr merged 24 commits into
mainfrom
feat/agent-dashboards-m4
Oct 10, 2026
Merged

vishr merged 24 commits into
mainfrom
feat/agent-dashboards-m4

Conversation

@vishr

@vishr vishr commented Oct 10, 2026 •

Copy link
Copy Markdown
Member

What changed

Milestone 4 of agent-built dashboards: performance and finish.

  • Ingest backlog.
    • Under a 4 GB DuckDB cap and a backlog of 4,096 small batches, dashboards stay usable, version and anomaly writes get through, and the backlog drains with exact counts.
    • Peak memory is 3.21 GB, with no out-of-memory error.
    • An edge rollup pass now covers one sub-window per write-gate hold.
    • The detector keeps findings from a write that misses the gate and sends them with its next batch.
  • Panel errors. A failed panel's full redacted error (up to 4 KB) is readable in Data, and the eval's panel checks keep it.
  • SQL panels.
    • Schema-qualified log columns (main.logs.body, telemetry.logs.body) resolve by lexical scope, and log bodies stay redacted.
    • A security review on the pinned engine found three older boundary defects. They are fixed: catalog- and path-returning built-ins are blocked, the statement that runs is the one that was described, and numbers survive exactly.
    • CI now fails when an engine upgrade adds, removes or changes a SQL built-in, until each change has been reviewed.
  • Speed.
    • On replayed demo data over 24 hours, every panel shape stays under 50 ms at p95, against S7's 500 ms. That includes metric panels. No rollup routing is added.
    • At 10x volume, no shape crosses 500 ms, and a twelve-panel request takes 1.1–1.6 s.
    • Larger systems will need per-minute summaries. They are planned for the next milestone.
  • Performance guard in CI.
    • A Playwright spec times the first full render of a 12-panel, 24-hour board in the page. It also checks one panel batch and one annotation request per refresh.
    • Locally it reports p95 against 1.5 s: about 360–390 ms.
    • On replayed demo data it measured 582–634 ms.
    • CI enforces a 1.8 s ceiling, calibrated from three runs on the hosted runner.
    • The performance spec runs in its own CI job.
  • Accessibility.
    • Dashboards and chat panels pass axe-core 4.13.0 WCAG AA scans in both themes, in chart, inspection, menu and full-screen states.
    • The six categorical colours pass a colour-vision-deficiency test.
    • Fixes cover border contrast, scroll-region focus, ARIA roles and labels, the page language and menu focus.
  • Eval. Holdout prompts carry an answer-or-dashboard intent, and there is an optional two-model quality judge. Both share the existing cost ledger.
  • UI fixes.
    • Panel cards keep their rounded bottom corners. An e2e check now guards this.
    • A long empty-panel diagnosis wraps inside the card.
  • Backlog catch-up. After a bulk replay, the edge rollup catches up within one rollup tick.
  • Docs. A new guide to building and refining dashboards.

Contract changes:

  • None to the API, MCP or migrations.
  • One UI behaviour change: opening a menu now focuses its first item.

Verification

  • just check
  • just test-race when auth, API, ingest, query, MCP, or agent paths changed (in CI)
  • just e2e: smoke, accessibility, corners and performance (screenshots are opt-in)
  • Final verification on replayed demo data (traces, logs and 14.3M metric points):
    • eval S1–S5: 10/10 saved, 65/65 panels good, 0 invalid, 21.5 s median first save, 5/5 exact edits;
    • collector: 33/33 checks;
    • S13 parity: both preview boards rebuilt from their prompts, with all eight capabilities present;
    • spend: $3.99 of $5, on claude-sonnet-5-5.
  • User-facing behavior and configuration docs are current
  • No credentials, private telemetry, host details, or enterprise-only source are included
  • API, migration, ingest, MCP/AG-UI, or release-contract changes are called out

Measurements: docs/benchmarks/2026-10-agent-dashboards-m4.md.

vishr added 24 commits October 9, 2026 16:36
Eleven tasks: a criteria audit, the ingest-backlog memory fix, full panel
error text, schema-qualified log columns in SQL panels, measured rollup
routing, the 12-panel performance guard in CI, accessibility checks, the
eval runner's intent scoring, documentation, and final verification with
the preview parity review. No deploy or demo reseed is in scope.
Records current evidence, CI coverage and the remaining gap for each of
S1–S13, maps the #232 regression items, and confirms that the retired
observability routes and widgets are gone. Every verdict stays pending
until the final verification task.
Routing is decided by a raw-path benchmark on replayed demo data and is
built only if S7 fails. The performance test times the page, not the
harness, reports locally and enforces a CI ceiling calibrated from three
runner measurements. The backlog reproduction drives the real rollup,
publication and panel concurrency. The colour-vision check fixes its
method and thresholds, and the eval runner keeps its current CLI.
Two opt-in readbench probes seed a small-batch backlog under a 4 GB
DuckDB cap and drive the real rollup, publication, version and anomaly
loops alongside four concurrent dashboard requests. From 1,024 batches,
anomaly writes time out at the write gate after 10 s while edge rollup
passes hold it for about 22 s; event totals stay exact. Out-of-memory
was not reproduced.

Rollup, version and anomaly errors now name the statement that failed.
An edge rollup pass covered up to eight sub-windows while holding the
write gate, about 22 s under a small-batch backlog, so version and
anomaly writes timed out behind it. A pass now covers one sub-window and
resumes from its committed cursor, so holds stay short and progress is
unchanged.

At 4,096 queued batches under a 4 GB cap, dashboards stay usable, the
backlog drains with exact span and log counts, and peak memory is
3.42 GB. A detector batch that still misses the gate is dropped and
re-detected on the next pass; the opt-in probe counts those drops and
fails on anything else.
A failed panel's executor message was cut at 500 bytes, so long engine
diagnostics lost the part that explains the failure, and the eval's
post-save checks dropped the message entirely.

- Panel errors keep the redacted message up to 4 KB, cut on a rune
  boundary.
- The Data view shows the full error as selectable text above the frame.
- The eval's panel checks retain the executor's error.
The opt-in readbench benchmark runs the production executor on a
generated fixture and on replayed demo telemetry, tiled to 24 hours and
to 10x volume. On demo volume every panel shape stays under 60 ms at
p95, so S7 holds on the raw path and no rollup routing is added. At 10x
volume one shape crosses 500 ms and a twelve-panel request takes about
2 s; both are recorded for later design work.
…cope

SQL panels can now read schema-qualified log columns (main.logs.body,
telemetry.logs.body and their quoted or case variants). Each qualifier is
resolved on the parsed statement in its own lexical scope before logs are
replaced with their redacted projection. Names it cannot resolve are left
for the engine to reject. Log bodies stay redacted on every path.

A security review on the pinned engine also found three older boundary
defects, fixed here with engine-executed tests:

- current_query, current_query_id, pg_get_viewdef, pg_get_constraintdef,
  format_type, get_block_size and write_log are blocked. Built-in macros
  let some of them return catalog contents or absolute Parquet paths as
  row values.
- The statement that runs is the one that was described. Snapshot
  compilation no longer strips main. and telemetry. qualifiers blindly;
  it shares the scope-aware resolver and skips names a CTE shadows.
- The parsed statement keeps exact numbers. Large integer and
  nanosecond literals survive, and USING SAMPLE works.
A Playwright performance spec opens a twelve-panel dashboard over a
deterministic 24-hour seed. The page itself records the moment every
panel has painted with no loader left, so assertions and harness steps
are outside the timed interval. Twenty fresh loads per theme give a p95;
refreshes and a range change assert one panel batch and one annotation
request each.

Locally the spec reports p95 against the spec's 1.5 s (about 420 ms
here) and does not fail on it. CI will enforce a ceiling calibrated from
three runner measurements; until it is set, CI records the measurement.
Browser evidence is now uploaded on every run.
During an ingest backlog an anomaly write can wait past its 10 s limit for
the write gate. Its findings were dropped, so a short spike seen during
the catch-up could vanish from charts for good. The detector now keeps
the findings of a failed write and sends them with its next batch, up to
the 10,000 episodes the store retains; the store already merges
overlapping episodes.
Telemetry SQL blocks dangerous built-ins by name, so a DuckDB upgrade
could add a function or macro that reopens a path or catalog leak.

- The blocklist is one named map.
- A test fails when any scalar macro whose definition runs a query is
  callable. These are the macros that read the catalog, which holds
  absolute Parquet paths.
- A committed snapshot of every built-in in the pinned engine fails the
  build when an upgrade adds, removes or retypes one. The reviewer checks
  each change against the blocklist, then regenerates the snapshot with
  FANOUT_UPDATE_SQL_FUNCTIONS=1.
The eval runner can now score the holdout like the benchmark.

- Holdout prompts carry an explicit intent: build a dashboard or answer
  directly. Each outcome is scored against the prompts with that intent.
  The fixed ten-prompt benchmark and its edits are unchanged.
- An optional two-model quality judge (claude-fable-5-1 and
  gpt-5.6-terra) runs only when both judge keys are configured. Without
  them it skips with no quality score and no spend. Judge calls go
  through the same per-call cost ledger and budget as the agent.
- Tests cover intent scoring, judge failures, budgets and the skip path
  with mocks; no provider is called.
Covers building and editing a dashboard from chat, choosing among the
fifteen panel types, inspecting empty and failed panels (the full error
is in Data), time, variables and comparison, sharing a view, layout,
saved versions and restore, keyboard controls, and the MCP tools. The
MCP guide and the capabilities page link to it.
The e2e suite now scans every dashboard and chat panel type with axe-core
4.13.0 in both themes, in chart, inspection, menu and full-screen states.
A unit test holds the six categorical colours to a colour-vision check:
Machado, Oliveira and Fernandes (2009) simulation, adjacent OKLab
distance of at least 6 under protan and deutan, at least 15 unsimulated,
and chroma of at least 0.10.

Fixes found by the scan:
- Chart borders are darker in both themes to reach 3:1, and service map
  labels stay opaque on hover and focus.
- Scrolling panel bodies paint the card background and take keyboard
  focus, as do scrolling code blocks.
- Table rows no longer carry aria-expanded, the health bar is a labelled
  group, drawer and modal close buttons are labelled, and the panels app
  page declares its language.
- Menus focus their first item when opened, as the ARIA menu-button
  pattern expects, instead of a hidden placeholder.

Status glyphs that repeat their label's meaning are held to WCAG 1.4.11's
3:1 for non-text marks, measured in the page. Review screenshots of every
panel type are captured on request with FANOUT_E2E_SCREENSHOTS=1.
On demo volume the slowest panel shape is 47 ms at p95 and a twelve-panel
request 200 ms; at 10x volume no shape crosses 500 ms and a twelve-panel
request is 1.59 s. The 4,096-batch backlog probe passes on the final
candidate with exact counts and 3.21 GB peak memory.
The single browser job took 11.2 minutes of Playwright's 12-minute
global timeout on the hosted runner. The performance spec now runs in a
separate matrix job and the smoke and accessibility suites in the other,
so both finish well inside the limit. just e2e passes extra arguments to
Playwright, and FANOUT_E2E_SKIP_PERFORMANCE=1 leaves the performance
spec out.
The panel body and the notes footer paint the card's background so text
has an opaque backdrop, but as plain rectangles their square corners
covered the card's rounded border at the bottom. Both now inherit the
card's bottom corner radius. The body regressed with the accessibility
fix; the notes footer, shown on logs and traces panels, had the same
defect since milestone 2.

A new e2e check fails when any opaque element paints over an unclipped
rounded corner, across dashboard and chat panels in every view.
Replay of all three signals, collector, real-data S6, the ten-prompt
benchmark and both preview parity boards on the candidate. The run found
an eval runner defect that hid every save, an edge rollup that catches
up too slowly after a bulk replay, a drill that re-requests all panels
on close, an overflowing empty-state diagnosis and two collector
defects. Fixes and a second run follow.
…aves

- Edge rollup: a pass still holds the write gate for one sub-window, but
  each rollup tick now runs passes back to back, releasing the gate
  between them, until the backlog is caught up or half the interval has
  passed. After a 41-hour bulk replay the service map had shown no
  routes for over an hour, at one sub-window per minute. Before the
  first committed ingest watermark, lag is measured from the window
  start rather than the epoch.
- Eval runner: a saved tool message without an error field is a
  success, as the server sends it. The mock now matches the server, and
  a golden pins the real saved-thread shape.
A long unbroken diagnosis overflowed and was clipped at both card edges.
The opt-in readbench benchmark adds three metric shapes: a gauge average
by service, a counter rate and a histogram p95. Each uses the most
common metric of its type in the replay, and the shapes are skipped,
with a log line, when the data has no metrics.
… scale limit

The rerun on the corrected candidate passes every criterion it measured:
S1–S5, S8, S9, S11, S12 and S13. S7 passes on the raw path, including
metric panels. The document also records where the raw path stops
scaling and that per-minute summaries are planned for the next
milestone.
Three ubuntu-24.04 runs measured a worse-theme p95 of 1,025, 1,410 and
1,239 ms for the twelve-panel first render. The CI ceiling is the
largest plus 25%, rounded up to 50 ms. The spec's 1.5 s target is still
measured locally on real data.
@vishr
vishr marked this pull request as ready for review October 10, 2026 07:46
@vishr
vishr merged commit 03bb28a into main Oct 10, 2026
14 checks passed
@vishr
vishr deleted the feat/agent-dashboards-m4 branch October 10, 2026 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant