feat(dashboard): unified MIME-bundle rail for rich exec outputs (sortable DataFrames + plots)#591
feat(dashboard): unified MIME-bundle rail for rich exec outputs (sortable DataFrames + plots)#591Andrew Gazelka (andrewgazelka) wants to merge 4 commits into
Conversation
The MCP Python exec board showed every result as a monospace ASCII table: the same text went to the human dashboard and the model's tool result, so a wide polars/pandas frame was both hard to read for the operator and a context sink for the agent. Split the two audiences. The worker now collects a self-contained HTML document for each displayed DataFrame and eval result (reusing the Jupyter rich-display path already used for images): a click-to-sort grid with sticky header, dtype labels, numeric right-align, and light/dark styling, built from the frame's own rows so it is independent of any pl.Config cap. These ride a new html channel on the exec pane (ExecView.html -> hub text -> sandboxed iframe) and reach only the dashboard; worker_response_content never forwards them, so the agent's context is untouched. To keep the agent's captured text compact, the session applies a one-time compact pl.Config (20 rows/cols) the first time polars is used, so an existing print(df)/report() stays terse while the human still gets the full interactive table via display(df). Made with AI (Claude Opus 4.8). Co-authored-by: Andrew Gazelka <andrew@ix.dev>
_apply_polars_compact mutates the process-global pl.Config, so re-applying the compact default would silently undo a pl.Config.set_tbl_rows(...) the user ran in an earlier cell (the once-flag only flips when polars is already imported at a cell's start, missing the cell that imports + configures polars together). Apply the compact repr default only when the user has not set any repr knob (POLARS_FMT_MAX_ROWS/COLS/STR_LEN via Config.state(if_set=True)), so their own config always wins. Found in adversarial review (C1). Made with AI (Claude Opus 4.8).
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e98772dd1b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| <div class="exec-empty">{running ? '· running…' : '· no output'}</div> | ||
| {:else} | ||
| <!-- Rich tables lead: a DataFrame's sortable grid is the answer when present. --> | ||
| <HtmlTables docs={htmlDocs} {expanded} /> |
There was a problem hiding this comment.
This adds the table renderer to the Svelte sources, but the production dashboard served by Rust is still the committed packages/dashboard-core/src/dashboard/dashboard.html (server.rs embeds it with include_str!). I checked that committed HTML with rg and it does not contain HtmlTables, exec-html, or parseExecHtml; packages/dashboard/default.nix also has a dashboardInSync diff check requiring the generated file to match the site build. As a result, packaged/embedded dashboards will not render the new DataFrame tables and the sync check will fail until the rebuilt index.html is copied into dashboard.html.
Useful? React with 👍 / 👎.
| head = "".join( | ||
| f'<th class="{"num" if numeric[i] else "txt"}" data-i="{i}">' | ||
| f"{_html_escape(name)}<span class=dt>{_html_escape(dtypes[i])}</span></th>" | ||
| for i, name in enumerate(columns) |
There was a problem hiding this comment.
When a pandas/polars frame is very wide, this renderer still emits every column into the HTML document (and the row loop below emits every cell for those columns); only the number of rows is capped. Displaying a 500-row frame with thousands of columns can therefore serialize megabytes of srcdoc into the worker response/Loro snapshot and hang or flood the dashboard, even though the text repr is intentionally compacted. Add a column or byte cap before constructing the table document.
Useful? React with 👍 / 👎.
| pl.Config.set_tbl_rows(20) | ||
| pl.Config.set_tbl_cols(20) | ||
| pl.Config.set_fmt_str_lengths(50) |
There was a problem hiding this comment.
In a persistent Python session, if a user imports polars and sets pl.Config in that first cell, _apply_polars_compact has not marked itself applied yet because polars was absent at the start of the call; the next call then overwrites the user's persistent table rows/columns/string-length settings with these hard-coded values. That makes later print(df)/repr output differ from the user's configured session state, despite the comment saying explicit user config should win.
Useful? React with 👍 / 👎.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
AI review found issues in this pull request.
Verdict: patch is incorrect
Confidence: 0.88
The feature is functional for small outputs, but the new rich-HTML path bypasses the intended resource bounds in common cases and can still overwhelm the worker, dashboard, or recording stream.
- P1
packages/mcp/src/python_worker.py:521Rich table HTML is still effectively unbounded - P2
packages/mcp/src/python_worker.py:210MAX_HTML is applied after rendering every candidate
| body_rows = [] | ||
| for row in rows: | ||
| cells = "".join( | ||
| f'<td class="{"num" if numeric[i] else "txt"}">{_html_escape(_cell(v))}</td>' | ||
| for i, v in enumerate(row) | ||
| ) | ||
| body_rows.append(f"<tr>{cells}</tr>") | ||
| body = "".join(body_rows) | ||
| shown = len(rows) | ||
| caption = f"{height} × {len(columns)}" | ||
| if shown < height: | ||
| caption += f" · showing first {shown}" | ||
| return _wrap_html( | ||
| f'<table><caption>{_html_escape(caption)}</caption>' | ||
| f"<thead><tr>{head}</tr></thead><tbody>{body}</tbody></table>{_SORT_JS}" | ||
| ) |
There was a problem hiding this comment.
The row count is capped, but each rendered table still emits every column and the full string value of every cell into a single HTML document. A 500-row frame with thousands of columns, or even one column containing very large strings, can produce a huge JSON-RPC response and Loro text field despite the new caps, blocking the worker/dashboard and bloating recordings. Add column, cell-length, and total HTML-byte limits before returning these docs.
| docs = [doc for obj in candidates if (doc := _object_html_doc(obj)) is not None] | ||
| return docs[:MAX_HTML] |
There was a problem hiding this comment.
This list comprehension renders rich HTML for every displayed object before slicing to MAX_HTML. A cell that calls display() on many DataFrames still materializes and wraps all of them, so the cap does not protect worker latency or memory. Stop iterating once MAX_HTML docs have been collected, and treat render failures like image collection does so one bad rich repr cannot fail the whole response.
| docs = [doc for obj in candidates if (doc := _object_html_doc(obj)) is not None] | |
| return docs[:MAX_HTML] | |
| docs: list[str] = [] | |
| for obj in candidates: | |
| if len(docs) >= MAX_HTML: | |
| break | |
| try: | |
| doc = _object_html_doc(obj) | |
| except Exception: | |
| continue | |
| if doc is not None: | |
| docs.append(doc) | |
| return docs |
Replace the two ad-hoc rich-output rails on the exec pane (images, which went only to the model, and the just-added html, which went only to the dashboard) with a single extensible `outputs` rail modeled on Jupyter's display-data protocol: an ordered list of MIME bundles (mime -> data), one per displayed object / eval result / open figure. The worker builds each bundle from the rich-display protocol it already used for images (text/html for a DataFrame's sortable grid + text/plain repr; image/* for plots and PIL images; whatever _repr_mimebundle_/_repr_*_ offers otherwise). The frontend gains a small renderer registry (DisplayOutputs) that picks the richest representation per bundle: text/html -> sandboxed iframe, image/* -> img, text -> pre. worker_response_content extracts only image/* and the captured text for the model, so rich tables stay operator-only. Why: a third rich type (a chart, LaTeX, audio) would otherwise need yet another hardcoded field threaded through worker, wire, hub, and frontend. One bundle rail makes a new type frontend-only. It also fixes a latent gap: matplotlib/PIL images were sent to the model but never rendered on the dashboard; now every output renders for the operator and the right parts reach the model. Made with AI (Claude Opus 4.8).
Andrew Gazelka (andrewgazelka)
changed the title
There was a problem hiding this comment.
AI review found issues in this pull request.
Verdict: patch is incorrect
Confidence: 0.90
The feature is not actually shipped because the embedded dashboard artifact was not regenerated, and the new DataFrame HTML path can still produce unbounded dashboard payloads despite the stated row cap.
- P1
packages/dashboard-core/site/src/components/ExecBody.svelte:59Embedded dashboard HTML is stale - P2
packages/mcp/src/python_worker.py:507DataFrame HTML output is not bounded by columns or bytes
| <!-- Rich outputs lead: a DataFrame's sortable grid or a plot is the answer. --> | ||
| <DisplayOutputs {outputs} {expanded} /> |
There was a problem hiding this comment.
This wires the new rich-output renderer into the Svelte source, but the PR does not update packages/dashboard-core/src/dashboard/dashboard.html. The dashboard server embeds that committed HTML via include_str!, and the package has a sync test that diffs it against a fresh site build, so the shipped dashboard will still serve the old JS with no outputs renderer and the in-sync check will fail. Rebuild the site and commit the regenerated dashboard.html with this source change.
| if module == "polars" and hasattr(obj, "columns") and hasattr(obj, "rows"): | ||
| columns = list(obj.columns) | ||
| dtypes = [str(t) for t in obj.dtypes] | ||
| height = int(obj.height) | ||
| rows = obj.head(MAX_HTML_ROWS).rows() | ||
| return {"columns": columns, "dtypes": dtypes, "rows": rows, "height": height} | ||
| if module == "pandas" and hasattr(obj, "columns") and hasattr(obj, "itertuples"): | ||
| columns = [str(c) for c in obj.columns] | ||
| dtypes = [str(t) for t in obj.dtypes] | ||
| height = int(len(obj)) | ||
| rows = list(obj.head(MAX_HTML_ROWS).itertuples(index=False, name=None)) | ||
| return {"columns": columns, "dtypes": dtypes, "rows": rows, "height": height} |
There was a problem hiding this comment.
The table preview caps only the number of rows, then materializes every column and stringifies each cell into one HTML document. A wide frame, or a frame with large string/blob cells, can still generate megabytes or more of text/html, bypassing the existing stdout/result capture limits and then storing that payload in the worker response and Loro document. Add a column/cell/output-byte cap before rendering the bundle so one rich display cannot hang or bloat the dashboard.
Blast radius
changed checks
|
There was a problem hiding this comment.
AI review found issues in this pull request.
Verdict: patch is incorrect
Confidence: 0.84
The rich display path introduces a browser script execution surface for arbitrary HTML outputs and can still create unbounded dashboard payloads for wide or large-cell DataFrames.
- P1
packages/dashboard-core/site/src/components/DisplayOutputs.svelte:30Arbitrary rich HTML is executed in the dashboard iframe - P2
packages/mcp/src/python_worker.py:507DataFrame HTML output is still effectively unbounded for wide or large-cell data
| <iframe | ||
| class="display-html" | ||
| class:expanded | ||
| title="output {i + 1}" | ||
| sandbox="allow-scripts" | ||
| srcdoc={bundle[mime]} |
There was a problem hiding this comment.
This renders every text/html bundle with sandbox="allow-scripts". The HTML is not limited to the worker-generated table; _repr_mimebundle_ and _repr_html_ output from arbitrary displayed objects reaches this path too. A sandbox without allow-same-origin prevents host DOM access, but it still lets the frame script read its own rendered output and send network requests, so displaying an untrusted object can execute browser-side JS and exfiltrate operator-only data. Keep arbitrary HTML script-disabled or sanitize/distinguish the trusted table renderer from third-party HTML.
| if module == "polars" and hasattr(obj, "columns") and hasattr(obj, "rows"): | ||
| columns = list(obj.columns) | ||
| dtypes = [str(t) for t in obj.dtypes] | ||
| height = int(obj.height) | ||
| rows = obj.head(MAX_HTML_ROWS).rows() | ||
| return {"columns": columns, "dtypes": dtypes, "rows": rows, "height": height} | ||
| if module == "pandas" and hasattr(obj, "columns") and hasattr(obj, "itertuples"): | ||
| columns = [str(c) for c in obj.columns] | ||
| dtypes = [str(t) for t in obj.dtypes] | ||
| height = int(len(obj)) | ||
| rows = list(obj.head(MAX_HTML_ROWS).itertuples(index=False, name=None)) | ||
| return {"columns": columns, "dtypes": dtypes, "rows": rows, "height": height} |
There was a problem hiding this comment.
MAX_HTML_ROWS only limits row count; this path still collects all columns and later stringifies every cell at full length into the outputs JSON stored in the hub. A 500-row frame with thousands of columns, or even one huge string cell, can produce very large worker responses and Loro text bodies despite the existing stdout/result caps. Add column, cell-length, or total rendered-byte limits before serializing the dashboard table.
2 participants
Exec panes had two ad-hoc rich-output rails:
images(model-only) and, as first drafted here,html(dashboard-only). A third rich type would need yet another hardcoded field through worker → wire → hub → frontend. This collapses them into one extensibleoutputsrail modeled on Jupyter's display-data protocol.What you get
pl.Configcap.display()s a frame and draws a plot produces two ordered bundles, rendered as table + image (screenshot in thread).Design
outputsis an ordered list of MIME bundles (mime -> data), one per displayed object / eval result / open figure. The worker builds each from the rich-display protocol it already used for images:text/html(df grid) +text/plain(repr),image/*base64, or whatever_repr_mimebundle_/_repr_*_offers. The frontend has a small renderer registry (DisplayOutputs):text/html→ sandboxedallow-scriptsiframe,image/*→ img, text → pre. Adding a new rich type is frontend-only.Audience split:
worker_response_contentforwards onlyimage/*+ the captured text to the model; rich tables are operator-only, never context. To keep the agent's captured text compact, the session applies a one-time compactpl.Config(20 rows/cols) the first time polars is used — and only if the user hasn't set those knobs, so their config always wins.Validation
cargo testdashboard-core / ix-mcp / dashboard — outputs bundle round-trips wire + Loro docsvelte-checkcleannix run .#lintgreenLinux-only clippy runs in CI.
Made with AI (Claude Opus 4.8).
Note
Add rich MIME-bundle display rail for exec outputs with sortable DataFrame tables
DisplayOutputsSvelte component that renders rich MIME bundles (HTML, images, plain text) in a sandboxed iframe or as image/text elements, shown in exec panes and feed views.ExecViewin pane.rs with anoutputsfield (Vec<BTreeMap<String, String>>) carrying ordered MIME bundles, serialized to JSON and published via hub.rs.MAX_HTML_ROWS).imagesarray in worker responses with anoutputsbundle list; MCP tool results now extract image blocks from these bundles instead of a separateimagesfield.worker_response_contentin main.rs no longer readsresponse.images; workers must emitoutputsbundles or images will not appear in tool results.Macroscope summarized cf6b0fb.