Repository navigation
feat(dashboards): backlog robustness, SQL boundary, accessibility and performance guards (milestone 4) - #295
Merged
Merged
Conversation
Eleven tasks: a criteria audit, the ingest-backlog memory fix, full panel error text, schema-qualified log columns in SQL panels, measured rollup routing, the 12-panel performance guard in CI, accessibility checks, the eval runner's intent scoring, documentation, and final verification with the preview parity review. No deploy or demo reseed is in scope.
Records current evidence, CI coverage and the remaining gap for each of S1–S13, maps the #232 regression items, and confirms that the retired observability routes and widgets are gone. Every verdict stays pending until the final verification task.
Routing is decided by a raw-path benchmark on replayed demo data and is built only if S7 fails. The performance test times the page, not the harness, reports locally and enforces a CI ceiling calibrated from three runner measurements. The backlog reproduction drives the real rollup, publication and panel concurrency. The colour-vision check fixes its method and thresholds, and the eval runner keeps its current CLI.
Two opt-in readbench probes seed a small-batch backlog under a 4 GB DuckDB cap and drive the real rollup, publication, version and anomaly loops alongside four concurrent dashboard requests. From 1,024 batches, anomaly writes time out at the write gate after 10 s while edge rollup passes hold it for about 22 s; event totals stay exact. Out-of-memory was not reproduced. Rollup, version and anomaly errors now name the statement that failed.
An edge rollup pass covered up to eight sub-windows while holding the write gate, about 22 s under a small-batch backlog, so version and anomaly writes timed out behind it. A pass now covers one sub-window and resumes from its committed cursor, so holds stay short and progress is unchanged. At 4,096 queued batches under a 4 GB cap, dashboards stay usable, the backlog drains with exact span and log counts, and peak memory is 3.42 GB. A detector batch that still misses the gate is dropped and re-detected on the next pass; the opt-in probe counts those drops and fails on anything else.
A failed panel's executor message was cut at 500 bytes, so long engine diagnostics lost the part that explains the failure, and the eval's post-save checks dropped the message entirely. - Panel errors keep the redacted message up to 4 KB, cut on a rune boundary. - The Data view shows the full error as selectable text above the frame. - The eval's panel checks retain the executor's error.
The opt-in readbench benchmark runs the production executor on a generated fixture and on replayed demo telemetry, tiled to 24 hours and to 10x volume. On demo volume every panel shape stays under 60 ms at p95, so S7 holds on the raw path and no rollup routing is added. At 10x volume one shape crosses 500 ms and a twelve-panel request takes about 2 s; both are recorded for later design work.
…cope SQL panels can now read schema-qualified log columns (main.logs.body, telemetry.logs.body and their quoted or case variants). Each qualifier is resolved on the parsed statement in its own lexical scope before logs are replaced with their redacted projection. Names it cannot resolve are left for the engine to reject. Log bodies stay redacted on every path. A security review on the pinned engine also found three older boundary defects, fixed here with engine-executed tests: - current_query, current_query_id, pg_get_viewdef, pg_get_constraintdef, format_type, get_block_size and write_log are blocked. Built-in macros let some of them return catalog contents or absolute Parquet paths as row values. - The statement that runs is the one that was described. Snapshot compilation no longer strips main. and telemetry. qualifiers blindly; it shares the scope-aware resolver and skips names a CTE shadows. - The parsed statement keeps exact numbers. Large integer and nanosecond literals survive, and USING SAMPLE works.
A Playwright performance spec opens a twelve-panel dashboard over a deterministic 24-hour seed. The page itself records the moment every panel has painted with no loader left, so assertions and harness steps are outside the timed interval. Twenty fresh loads per theme give a p95; refreshes and a range change assert one panel batch and one annotation request each. Locally the spec reports p95 against the spec's 1.5 s (about 420 ms here) and does not fail on it. CI will enforce a ceiling calibrated from three runner measurements; until it is set, CI records the measurement. Browser evidence is now uploaded on every run.
During an ingest backlog an anomaly write can wait past its 10 s limit for the write gate. Its findings were dropped, so a short spike seen during the catch-up could vanish from charts for good. The detector now keeps the findings of a failed write and sends them with its next batch, up to the 10,000 episodes the store retains; the store already merges overlapping episodes.
Telemetry SQL blocks dangerous built-ins by name, so a DuckDB upgrade could add a function or macro that reopens a path or catalog leak. - The blocklist is one named map. - A test fails when any scalar macro whose definition runs a query is callable. These are the macros that read the catalog, which holds absolute Parquet paths. - A committed snapshot of every built-in in the pinned engine fails the build when an upgrade adds, removes or retypes one. The reviewer checks each change against the blocklist, then regenerates the snapshot with FANOUT_UPDATE_SQL_FUNCTIONS=1.
The eval runner can now score the holdout like the benchmark. - Holdout prompts carry an explicit intent: build a dashboard or answer directly. Each outcome is scored against the prompts with that intent. The fixed ten-prompt benchmark and its edits are unchanged. - An optional two-model quality judge (claude-fable-5-1 and gpt-5.6-terra) runs only when both judge keys are configured. Without them it skips with no quality score and no spend. Judge calls go through the same per-call cost ledger and budget as the agent. - Tests cover intent scoring, judge failures, budgets and the skip path with mocks; no provider is called.
Covers building and editing a dashboard from chat, choosing among the fifteen panel types, inspecting empty and failed panels (the full error is in Data), time, variables and comparison, sharing a view, layout, saved versions and restore, keyboard controls, and the MCP tools. The MCP guide and the capabilities page link to it.
The e2e suite now scans every dashboard and chat panel type with axe-core 4.13.0 in both themes, in chart, inspection, menu and full-screen states. A unit test holds the six categorical colours to a colour-vision check: Machado, Oliveira and Fernandes (2009) simulation, adjacent OKLab distance of at least 6 under protan and deutan, at least 15 unsimulated, and chroma of at least 0.10. Fixes found by the scan: - Chart borders are darker in both themes to reach 3:1, and service map labels stay opaque on hover and focus. - Scrolling panel bodies paint the card background and take keyboard focus, as do scrolling code blocks. - Table rows no longer carry aria-expanded, the health bar is a labelled group, drawer and modal close buttons are labelled, and the panels app page declares its language. - Menus focus their first item when opened, as the ARIA menu-button pattern expects, instead of a hidden placeholder. Status glyphs that repeat their label's meaning are held to WCAG 1.4.11's 3:1 for non-text marks, measured in the page. Review screenshots of every panel type are captured on request with FANOUT_E2E_SCREENSHOTS=1.
On demo volume the slowest panel shape is 47 ms at p95 and a twelve-panel request 200 ms; at 10x volume no shape crosses 500 ms and a twelve-panel request is 1.59 s. The 4,096-batch backlog probe passes on the final candidate with exact counts and 3.21 GB peak memory.
The single browser job took 11.2 minutes of Playwright's 12-minute global timeout on the hosted runner. The performance spec now runs in a separate matrix job and the smoke and accessibility suites in the other, so both finish well inside the limit. just e2e passes extra arguments to Playwright, and FANOUT_E2E_SKIP_PERFORMANCE=1 leaves the performance spec out.
The panel body and the notes footer paint the card's background so text has an opaque backdrop, but as plain rectangles their square corners covered the card's rounded border at the bottom. Both now inherit the card's bottom corner radius. The body regressed with the accessibility fix; the notes footer, shown on logs and traces panels, had the same defect since milestone 2. A new e2e check fails when any opaque element paints over an unclipped rounded corner, across dashboard and chat panels in every view.
Replay of all three signals, collector, real-data S6, the ten-prompt benchmark and both preview parity boards on the candidate. The run found an eval runner defect that hid every save, an edge rollup that catches up too slowly after a bulk replay, a drill that re-requests all panels on close, an overflowing empty-state diagnosis and two collector defects. Fixes and a second run follow.
…aves - Edge rollup: a pass still holds the write gate for one sub-window, but each rollup tick now runs passes back to back, releasing the gate between them, until the backlog is caught up or half the interval has passed. After a 41-hour bulk replay the service map had shown no routes for over an hour, at one sub-window per minute. Before the first committed ingest watermark, lag is measured from the window start rather than the epoch. - Eval runner: a saved tool message without an error field is a success, as the server sends it. The mock now matches the server, and a golden pins the real saved-thread shape.
A long unbroken diagnosis overflowed and was clipped at both card edges.
The opt-in readbench benchmark adds three metric shapes: a gauge average by service, a counter rate and a histogram p95. Each uses the most common metric of its type in the replay, and the shapes are skipped, with a log line, when the data has no metrics.
… scale limit The rerun on the corrected candidate passes every criterion it measured: S1–S5, S8, S9, S11, S12 and S13. S7 passes on the raw path, including metric panels. The document also records where the raw path stops scaling and that per-minute summaries are planned for the next milestone.
Three ubuntu-24.04 runs measured a worse-theme p95 of 1,025, 1,410 and 1,239 ms for the twelve-panel first render. The CI ceiling is the largest plus 25%, rounded up to 50 ms. The spec's 1.5 s target is still measured locally on real data.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Milestone 4 of agent-built dashboards: performance and finish.
main.logs.body,telemetry.logs.body) resolve by lexical scope, and log bodies stay redacted.Contract changes:
Verification
just checkjust test-racewhen auth, API, ingest, query, MCP, or agent paths changed (in CI)just e2e: smoke, accessibility, corners and performance (screenshots are opt-in)Measurements:
docs/benchmarks/2026-10-agent-dashboards-m4.md.