Skip to content

feat(dashboards)!: authoring quality, version history and one panel renderer (milestone 3) - #287

Merged
vishr merged 66 commits into
mainfrom
feat/agent-dashboards-m3
Oct 9, 2026
Merged

vishr merged 66 commits into
mainfrom
feat/agent-dashboards-m3

Conversation

@vishr

@vishr vishr commented Oct 9, 2026

Copy link
Copy Markdown
Member

What changed

Milestone 3 of agent-built dashboards: make authoring measurable and dependable.

  • One renderer. Chat and dashboards render panels through the same code. The five MCP apps under ui/apps are replaced by one single-file panel app built from ui/host. The retired /api/observability/* routes and the second workspace are removed, and CI dead-code guards keep them out.
  • Authoring the user can trust. The agent no longer narrates intermediate steps. A quiet live build receipt and edit chips come from executed tools and saved version diffs. Saves return authoritative panel checks, and the guidance forbids pinning saved panels to IDs copied from one result.
  • History and restore. Dashboards keep owner-scoped immutable versions, with history, inspectable changes and restore in the UI and over MCP (list_dashboard_versions, restore_dashboard_version).
  • Full-screen, Explain and keyboard. Full-screen panels work for keyboard users, Explain is precise and read-only, and points, filters and drills work from the keyboard. Range controls and shortcut help are included.
  • Layout and rendering fixes from final acceptance.
    • Packed rows share one height and fill all twelve columns. This applies only to panels the server packs, never to authored or saved grids.
    • Time axes span the panel window.
    • The service map keeps names readable. A cropped map fades at its edge and offers Fit for the whole graph.
    • Failed panels say whether a retry can help.
    • Navigating away no longer logs cancelled queries as errors.
  • Measurement. A promoted eval runner, a checked-in browser smoke test (just e2e, 1100/1440 px, light and dark), and results in docs/benchmarks/2026-10-agent-dashboards-m3.md.

Contract changes:

  • Removed: /api/observability/{overview,topology,services/dependencies,performance,trace,logs}. Trace detail moves to GET /api/traces/{id}.
  • Added: dashboard version routes and the two MCP version tools above.
  • MCP: clients receive one panel app resource instead of five.
  • Migration: 20261008000001_dashboard_origins.sql (Goose).

Verification

Measured results are in docs/benchmarks/2026-10-agent-dashboards-m3.md. Model: claude-sonnet-5-5.

  • Benchmark: S1–S5 pass on the frozen candidate. 10/10 saved, 67/67 panels good, 0 invalid, median 18.7 s to save, 5/5 scoped edits, $1.45.

  • Sealed holdout: never used for tuning; the prompt text was not read.

    • 16/16 correct outcomes: 11/11 dashboards saved, 88/88 panels acceptable, and 5/5 answered without a dashboard.
    • Quality judging was not run, as in milestone 2.
  • Browser collector: 33/33 checks in light and dark, and 30/30 visualization and theme rows.

  • Per-panel check: 33 panels on two boards, light and dark. No failures, truncation or ragged rows, and 0 console errors.

  • Panel app size: 2.38 MB raw and 0.78 MB gzipped, against 7.08 MB and 2.41 MB for the five apps it replaces.

  • just check

  • just test-race when auth, API, ingest, query, MCP, or agent paths changed

  • just e2e (24/24)

  • User-facing behavior and configuration docs are current

  • No credentials, private telemetry, host details, or enterprise-only source are included

  • API, migration, ingest, MCP/AG-UI, or release-contract changes are called out

Milestone 4 follow-ups are listed in the benchmark doc:

  • an intent-aware runner with cost metering and quality judging;
  • memory and timeouts while an ingest backlog drains;
  • reaching the full text of long refresh errors.

Comment thread ui/host/tests/build-graph.test.ts Fixed
Comment thread internal/dashboard/changes.go Fixed
Comment thread internal/dashboard/changes.go Fixed
vishr added 28 commits October 9, 2026 14:32
Fifteen tasks: one renderer for chat and dashboards, zero legacy with
dead-code gates, the eval runner, narration, build receipts, history
and restore, full-screen and Explain, keyboard access, and measured
acceptance.

Claude-Session: https://claude.ai/code/session_018rJZa4mMppj9h54Q6pDQab
…rom host

Adds the package-free fragment and variable models, a FragmentView that
reuses the dashboard panel renderer, the shared trace detail, and a
single-file panels.html app staged from the host workspace. Chat still
uses ui/apps until the server cut-over.

Claude-Session: https://claude.ai/code/session_018rJZa4mMppj9h54Q6pDQab
…nd fragments

FragmentView now uses the dashboard page's variable resolver and one
shared click, drill and brush-zoom helper instead of copies that had
already drifted. Fixtures model the five real presets with rows, the
app build graph rejects every auth module, and the committed SPA assets
match the inlined service-map worker.

Claude-Session: https://claude.ai/code/session_018rJZa4mMppj9h54Q6pDQab
… renderer

MCP view tools and query_telemetry now return panel specs plus results
rendered by the shared dashboard renderer in a single panels.html app.
The model receives a bounded text summary while the app keeps a capped
full fragment. App-only tools are hidden from non-Apps clients and the
host bridge forwards only app-visible tools. The ui/apps workspace and
its five bespoke views, including the dual-axis correlation chart, are
deleted, and the container build has one UI stage.

Claude-Session: https://claude.ai/code/session_018rJZa4mMppj9h54Q6pDQab
The MCP app reports its intrinsic content height in whole pixels, so
chat iframes no longer grow until the cap and the transcript keeps the
reader's scroll position. The 256 KiB fragment cap now holds on the
serialized size, app-only tools are refused to the model, summaries
read as text, and failed app calls render no view. Chat collapses all
but the last view of a turn and drops duplicates; one-panel fragments
show no extra header; single-series charts carry no end label; trace
logs use the dashboard logs table; the service map gets a usable
height.

Claude-Session: https://claude.ai/code/session_018rJZa4mMppj9h54Q6pDQab
The service map uses the M2 left-to-right layered layout everywhere and
fits the whole graph inside the panel in both dimensions, removing the
compact row layout and the clipping it caused on dashboards. Chat keeps
preset views expanded and collapses ad-hoc queries into aligned summary
rows, and trace tables never truncate their time column.

Claude-Session: https://claude.ai/code/session_018rJZa4mMppj9h54Q6pDQab
Short dashboard panels fit the layered map with single-line compact
cards again instead of nameless outlines, full cards size to each
measured service name, and collapsed chat summaries count panels and
non-ok statuses.

Claude-Session: https://claude.ai/code/session_018rJZa4mMppj9h54Q6pDQab
…able

Fragment reduction now drops rows from the largest frame first and trims
trace detail before emptying anything, so small panels survive. The
service map never scales above natural size and starts dense fan-outs
at the readable floor with pan instead of shrinking boxes under their
text. Fragments carry a server-computed view kind and key, failed tool
calls collapse into one quiet line, and null tool arguments act as {}.

Claude-Session: https://claude.ai/code/session_018rJZa4mMppj9h54Q6pDQab
Retires the /api/observability handlers and the browser observability
client in favour of panel routes and GET /api/traces/:id, extracts the
health error trend, deletes the endpoint-histogram and log-count read
caches that no longer have readers (read-cache version 4), drops legacy
OAuth scope aliases, session keys and grid imports, and renames
round-named tests by behaviour. just check now runs knip and a
whole-program Go deadcode gate.

Claude-Session: https://claude.ai/code/session_018rJZa4mMppj9h54Q6pDQab
The rail waits for the schema before showing services, so its search
test no longer races. Review-round IDs are gone from test names and a
check keeps them out, every scripts test runs in just check, deadcode
roots come from go list, retired OAuth refresh grants return
invalid_grant, docs no longer describe the deleted caches, and the doc
lint walker skips dot-directories.

Claude-Session: https://claude.ai/code/session_018rJZa4mMppj9h54Q6pDQab
… scoring

scripts/dashboard-eval scores S1-S5 from persisted saves, executed panel
status and five consecutive edit diffs, meters every provider call into
a ledger with a before-each-prompt spend gate, never retries a mutating
agent request, and keeps holdout artifacts free of prompt text. Mock
transport tests run in just check at no model cost.

Claude-Session: https://claude.ai/code/session_018rJZa4mMppj9h54Q6pDQab
… safely

Golden fixtures come from the real panel executor and agent SSE encoder
and are checked for freshness. Runs fail on a model mismatch, text
panels count as executed, add edits allow server-filled defaults, S4
times the scored save, truncation comes from model_call_usage status,
and failed prompts settle with their metered cost while unknown usage
needs an explicit settlement entry.

Claude-Session: https://claude.ai/code/session_018rJZa4mMppj9h54Q6pDQab
The runtime buffers each provider step's text and publishes it only
when the step ends the run with a final answer, refusal, error or
truncation, so tool-step narration never reaches the live stream or the
persisted transcript. Chat keeps its running status until text arrives.
The eval CLI stops on a model mismatch and settles a stream that drops
after metered usage.

Claude-Session: https://claude.ai/code/session_018rJZa4mMppj9h54Q6pDQab
Provider text streams as AG-UI thinking events and becomes the answer
only when its step ends the run, so chat types live again while tool
step notes collapse into the running status. Stop shows a quiet note
instead of an error, failed runs persist their message, server codes
map to plain language, final answers persist when the client has gone,
and the eval accepts dated model IDs for its label.
…l text

Provisional text uses the current REASONING_* events instead of the
deprecated THINKING_* spelling and renders through the same markdown
component as the answer. Time limits use a sentinel error, a user stop
persists as stopped on every path, and an answer whose delivery failed
reloads without a failure marker.
Saves diff the committed version against its optimistic base inside the
write transaction, so concurrent edits never leak into another save's
change list. Mutation results carry per-panel authored-field changes,
one layout_changed flag for server packing, structured post-save checks
and preview timings, and the eval scores S5 from them.
…a test

Position changes come from the longest common subsequence of the panel
order, so one move tags one panel, and the eval scorer applies the same
rule independently. A file-backed concurrent save test fails if a save
reads its committed row after the transaction.
…ance

Chat renders one compact receipt per build or edit from the server's
committed changes and save checks, with chips only for changed panels.
Dashboards remember the chat request that created them in a new
dashboard_origins table, injected from authenticated in-process context
and never from model input or remote metadata, so the rail links back
across threads.
knip rejects an export used only by tests; receipt reconstruction tests
already cover resolved corrections.
…nce private

Corrections count when a problem's path disappears from a later check,
including panels that execute empty or with an error. Save failure
needs a final failed mutation, chips accumulate the net change across a
turn, unreadable results show no receipt, and the running indicator
stays for the whole run. Build provenance leaves MCP results and is
served only to the owner's rail, and empty message slots reserve no
space.
Thread deletion cascades through dashboard_origins; the migration is
unpublished, so the index joins it instead of a follow-up migration.
A stop can land after the commit, so an interrupted save reads 'Save
interrupted · outcome unknown' unless a later result proves it, and a
failed attempt before a successful save is listed as retried.
list_dashboard_versions and restore_dashboard_version are owner-scoped;
a restore commits a new version and returns the normal receipt, which
chat renders like any save. Tool errors are read only from structured
fields, and the runtime marks every failure path structurally.
…e honest

Failed tool calls render their structured message from Go-generated
fixtures, a missing version gets its own restore error, the eval's
mutating tools come from the server's registry, and a panic after a
committed save reads as interrupted with the stack logged server-side.
…store

A history drawer lists versions with author, agent or person, message
and relative time, shows each version's changes as computed by the
server, previews its spec, and restores it as a new version with a
pending state that blocks double submission and refreshes the board.
…current restores

Restore errors belong to their version and only retryable failures offer
a retry. A host test fails on any colour the theme does not define, the
current version cannot be restored (409 already_current), history
refreshes when the board changes elsewhere, and history and chat share
one change-chip component.
vishr added 27 commits October 9, 2026 14:32
…tle failed panel batches

A pointer click no longer starts keyboard navigation. The chart plot uses
application semantics with a description present before focus, announces
only navigation keys, and drops a pending range on blur. Chat fragments
register no character shortcuts. A failed panel batch turns its panels into
retryable error cards instead of leaving them loading, and the page tests
wait for rendered idle state.
…ttern drill tests

Trace links and the router share one search codec, so a drill has a single
URL form. Hand-typed quoted values stay literal; only values the codec
quoted are unquoted. The scoped pattern test drills through the real
exemplar path, rollup gate cancellation is tested deterministically, and
the model guide states the pattern tie rules once.
The CI smoke test drives the system Chrome channel, so no browser binaries
are downloaded.
…keep failures compact

Keyboard navigation re-bases to the nearest remaining point when its
series disappears, never replays a key pressed on an empty chart, and
keeps a pending range while focus moves into its buttons. Transport
failures offer Retry for exactly the failed batch, queue behind other
work and never ask the agent to fix them; returned query errors keep
their fix action. A failed refresh on a panel with data shows a header
indicator instead of shrinking the chart, and short cards use a one-line
error state.
The analysis guidance forbids filtering saved panels on trace, span or
request IDs copied from a single result. Recent evidence belongs in a
traces panel with drill to its logs.
The baseline and iteration runs of the ten benchmark prompts and five
edits, with the memory-cap environment cause of the baseline S2 failure,
the pinned-identifier guidance fix, and findings for milestone 4. The
Task 1b build measurements stay in the same document.
A Playwright suite drives system Chrome against a disposable Fanout seeded
with deterministic OTLP telemetry. At 1100 and 1440 px in both themes it
checks that all fifteen panel types settle without console errors or
overlapping chrome, that service-map nodes stay unclipped and legible,
and that every Go-generated chat fragment renders in the sandboxed app
with converging size messages. just e2e runs it and CI runs it on every
push and pull request.
…h failures

Panel failures are classified once: transport failures, 408, 429, 5xx,
missing results and execution timeouts get a Retry that re-requests
exactly those panels, while validation errors and other 4xx get the fix
action, inline even in short cards. The refresh-failure indicator is part
of the panel menu button's accessible name and description, with the age
of the data still shown, and lazy batches completing in the same
millisecond no longer collide.
…failure chrome readable

Panel results carry a retryable flag set from the engine's typed errors
(timeouts, the batch deadline, out of memory, I/O, interrupt), so the
browser no longer matches error text. HTTP failures keep their status, the
refresh-failure tooltip wraps and opens on keyboard focus, the menu name
stays short with the detail in its description, and compact cards use an
icon fix action.
…oke test

Global setup waits for /readyz before creating the admin, retries data
readiness until its deadline, cleans up on signals, and prints only
redacted diagnostics. Charts must expose their series or rows, seeded
panels must not be empty, deploy annotations and topology edges must be
present, size messages must stay quiet for five seconds, and the fixture
decoder rejects unknown fields. The CI job pins Node and has a timeout.
…ooltip on focus

A batch-deadline overwrite no longer logs a second error for a panel whose
interrupted query was already logged. The compact Ask Fanout icon's
tooltip opens on keyboard focus, and the error row no longer adds a native
title that competed with it.
…panel chrome readable

Time charts span the observed panel window, so data that ends early and
annotations after it stay in place. The service map starts contain-fitted
with its isolated-node note in its own row. Adjacent anomaly windows share
one label, compact headers move the view switch into the menu instead of
truncating titles, row-panel notes sit in a separate footer, and pattern
tables end with a quiet summary. In chat full screen, Escape keeps focus
inside the panel and keyboard users can tab into it.
The summary under a pattern table is static text, so it no longer
re-announces on every refresh or reads as panel chrome.
…e their plots

Panels the server packs side by side stretch to the tallest panel in
their row when the space below them is free, so agent-built dashboards no
longer leave ragged holes; hand-sized panels keep their size. The service
map shrinks only to its readable floor and pans beyond it, heatmap and
timeline cells clip to the plot, annotation chips sharing the legend row
start from the right edge, and the header reserves the refresh spinner's
slot so cards stop switching layout on every refresh.
…stat height rule

The server packer gives stat panels their four-row minimum and the browser
no longer raises them on its own, so stats and gauges in one row match.
Rows the server packs widen their panels to fill the free columns on the
right, so default layouts no longer leave gaps; hand-sized panels keep
their size.
…d quiet cancelled queries

A service map with no routes wraps its services into centred rows inside
the panel. Annotation chips move to their own lane when the legend's first
row reaches them, the heatmap colour scale shows one label per end, and
queries cancelled by navigation no longer log as errors.
…whole graph

A half-width map whose services don't fit at the readable floor cropped
silently. Overflowing edges now fade, and the Fit control appears whenever
the map overflows, toggling between the whole-graph overview and the
readable view.
Adds the frozen-candidate authoring benchmark, the browser collector,
smoke-test and per-panel results, the panel app size, and the M4
findings. The sealed holdout is recorded as unrun: the runner grades
only dashboard intent.
…nterrupts

DuckDB reports a cancelled query as a bare INTERRUPT error without
context.Canceled in the chain, so navigation away from a dashboard still
logged "panel query failed" at ERROR. Decide from the batch context
instead: a caller cancellation that interrupts the query is not logged;
batch deadlines and unrelated failures still are.
A cancelled or timed-out bind check came back from DuckDB as a bare
INTERRUPT error, which the filter check reported as a spec problem
("does not bind: ... cast variables"). That misleads the agent into
rewriting a valid filter. Interrupts now stop the check as operational
errors, like cancellation and deadlines.
All 16 sealed prompts ran through the intent-aware harness: 11 of 11
dashboards saved with 88 of 88 panels acceptable, and 5 of 5 answers
given without a dashboard. Quality judging was not run.
Two eval tests hard-coded macOS's /private/tmp, which does not exist on
the Linux CI runners. Use the resolved OS temp directory instead.
internal/mcp/apps/panels.html is a minified build of ui/host, which CodeQL
already analyses from source. Its vendored libraries repeat known alerts,
such as zod's compiled validators, under a new path.
knip's parser and resolver install one native binding per platform as an
optional dependency, so the notices listed a darwin binding locally and a
Linux binding in CI, and the CI gate saw the file as stale. Optional
packages pinned to an OS or CPU are now skipped on every machine.
CodeQL flagged the table allocations in field suggestions and panel
order diffing as overflow-prone, and two script-tag regexes as missing
upper-case or spaced closing tags.

- Field suggestions skip names longer than 128 bytes, which are never
  near misses.
- Order diffing checks the 40-panel limit, now the named panel.MaxPanels
  constant that validation also uses.
- The bundle test and the UI boundary check match script tags without
  regard to case or closing-tag attributes.
Under -race on CI the engine startup alone pushed the test to 1.02 s
against its 1 s limit. The bounded history work itself takes about 40 ms.
The UI build type-checks the host tests, and tests/tool-errors.ts reads
scripts/dashboard-eval/testdata/server.json, so the image build failed
with TS2307.
@vishr
vishr force-pushed the feat/agent-dashboards-m3 branch from 7279bd6 to 3309cb4 Compare October 9, 2026 21:35
@vishr
vishr merged commit 3c80055 into main Oct 9, 2026
27 of 29 checks passed
@vishr
vishr deleted the feat/agent-dashboards-m3 branch October 9, 2026 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants