Skip to content

Commit d9cfd7c

Browse files
Sg312j15zicecrasher321
authored
improvement(mothership): v0.9 (#6815)
* checkpoint * Checkpoint * dot fixes * Make async tool resume delivery recoverable * Support split table tools and option recovery * Harden VFS mutation handling * feat(platform): platform subagent support — docs corpus VFS, search_docs, account context Squash of the feat/platform-agent branch (sim side): mounts the Sim docs corpus in the copilot VFS, wires search_docs and retires the legacy docs search tools, and syncs the generated tool catalog and trace contracts for the platform subagent. * Align Copilot tools and resource handling * Expand workflow log query support * checkpoint * Port desktop-improvements-0 desktop and browser-agent work * fix(desktop): keep browser-agent input alive through live-SPA re-renders * Harden workflow sanitization and Slack setup * feat(desktop): coordinate clicks, caret insertion, and drag for the browser agent * Revert subagent group eager auto-collapse * fix(chat): keep sends FIFO across the streaming-to-idle drain gap * Add the steering backend surface for mid-turn sends * Sync generated contracts for async subagent orchestration Pulls the mothership tool catalog (wait_agents / tail_agent / steer_agent / interrupt_agent), trace spans (chat.async_subagent.*, chat.orchestrate.*), and trace attributes (copilot.async_subagent.*) into the generated TS contracts. * Add display titles for the async subagent orchestration tools wait_agents / tail_agent / steer_agent / interrupt_agent get natural-language running titles (naming the agent id being waited on, tailed, steered, or stopped) and a Steering→Steered completed-verb rewrite. * Show orchestrator-chosen subagent names on agent groups A subagent_start whose payload data carries a name (the orchestrator's new name trigger parameter) now labels the agent group with that mission name — the agent-type icon stays. The name flows through the live stream path, the turn model (AgentNode.displayName) and its serialize/rebuild round-trip, and persisted transcripts (PersistedContentBlock.name), so reloads keep the label. * Improve Copilot error handling and logging * Backfill the subagent display name from the second start event The dispatch-time subagent_start fires before the trigger args (and therefore the name parameter) have streamed; the phase-3 start re-announces the lane with the name. The block builder was dropping that duplicate wholesale, losing the name on streaming providers — now it backfills subagentName onto the existing block instead. (The home turn-model path already reconciled this case.) * Support Slack bot connection flow * Harden Copilot error and VFS handling * Harden VFS resource operations * Show 'Waiting for the first of N agents' for mode-any waits The wait_agents title ignored the mode argument, so an any-mode wait over three agents read 'Waiting for 3 agents' while the model narrated waiting for the first — contradicting the transcript. * Collapsed-by-default agent cards with live intent status lines Subagents now narrate their work through <intent>3-5 words</intent> tags (a fleet-wide prompt protocol on the mothership side). The turn model streams each subagent's text through a split-safe tag parser: complete tags update the agent's currentIntent and disappear from the prose, tags split across deltas are carried until their close arrives, and a tag that never closes flushes back as plain text. The agent card renders as one line — display name (or agent label) plus the latest intent, replaced inline as the agent shifts gears — and never auto-expands; expanding to the full tool log is a deliberate click. Only an outstanding permission prompt or a browser hand-back forces a group open. Intents persist on the subagent block (and through the legacy persisted- message paths) so reloads keep the last status, and a renamed reinvocation now takes the latest name instead of pinning the first. * Add the internal in-band tool execution route for live mothership turns POST /api/copilot/tools/execute (INTERNAL_API_SECRET, Go→Sim) runs one sim-server tool through the same server tool router the resume driver uses and returns the result synchronously — no checkpoint. This is what lets background (async) subagents write files/tables/knowledge, and lets the main lane keep streaming (instead of checkpoint-pausing and killing every background run) while async agents are live. * Persist resource side effects for in-band tool execution Files/tables created through the internal execute route now register on the chat's resources exactly like the resume driver's executions — the route runs the same handleResourceSideEffects pass (persistence only; an out-of-band route has no live event sink, so mid-turn chip pushes are a follow-up). * Extract intents from group text on every path, sync and async The turn-model intent filter only fires for span-scoped subagent lanes, but this surface also delivers subagent text through the legacy block path — so <intent> tags flowed through unparsed and rendered as prose rows. Groups now extract intents from their accumulated text at append time: the last complete tag becomes the card's status line and every complete tag is stripped from the rendered prose. Covers span-scoped, legacy, and persisted-reload paths for both synchronous and background delegations. * Fall back to the live tool title for the agent card status line Persisted data proved tool-first subagents (grok search agents) emit zero prose, so intent tags never stream no matter what the prompt says. The collapsed card now always narrates: the agent's own <intent> tag when present, else the latest tool's display title while the lane is live. * Catch subagent <intent> tags in the server relay The relay's subagent text handler now runs the split-safe intent extraction as chunks stream: the latest complete tag is stamped onto the lane's persisted subagent block (subagentIntent) and stripped from the stored prose, so live, persisted, and replayed views all agree. Per-lane carry handles tags split across chunks; a never-closing tag flushes back as plain text. * Drop the tool-title fallback: the status line is the agent's intent With the intent protocol now injected into every spawn's task message, agents open with an <intent> tag; the card shows that narration or nothing. * Replace intents with live tool-title status lines on agent cards Intent parsing is fully removed (turn model, relay handler, persistence fields, group extraction). The collapsed card's status is the latest tool call in its RUNNING phrasing — never the completed rewrite, which stays in the expanded log. Parallel tools show the most recently started still-running title with a +N for concurrent siblings; between rounds the last title stays frozen; a closed lane shows the bare name. Nested agent cards compute their own status recursively from their own items. * Keep the main Sim lane live-expanded; collapse only real subagent cards The mothership group is the turn's own narration, not a delegation card — collapsing it hid main-lane text and tools until manual expand, which read as mis-ordered streaming while async subagents interleaved. It keeps the original live-expand behavior and no status suffix. * Persist subagent lane lifecycle blocks from the span handler Lane-scoped span events route to the span handler, which only recorded trace side effects — no subagent start block was ever persisted (verified: a seven-agent run stored 104 blocks with zero starts). Grouping then fell back to keying lane content by agent NAME, so a respawned agent of the same type merged invisibly into the first one's card until it resolved. The handler now persists the start block (spanId-keyed and deduped, carrying the display name) and stamps endedAt on close, giving every invocation its own card. * Name agents in orchestration titles; '+ n more' overflow format wait/tail/steer/interrupt titles humanize the slugified agent ids back to their display names ('Waiting for the first of Digest Workflow Build + 4 more'), and the agent card's parallel-tool suffix uses the same '+ n more' format. * Harden in-band tool execution and resources * Route in-band execution through the comprehensive tool dispatcher The internal execute route used the bare server-tool router, which rejects VFS tools with 'Unknown server tool: read/glob/grep' — so nearly every background agent's first discovery call failed (102 in-band calls in one run, dozens rejected). It now uses the relay's executeTool dispatcher: registered handlers (VFS, function execute) with permission checks and param normalization, falling back to the app tool router — the same surface foreground execution gets. * Harden chat stream transition handling * Harden VFS provenance and resource writes * Standardize tool environment references * Harden browser panel and chat cleanup * Descriptive, user-language tool titles across the board House rules applied everywhere: use every argument the call carries, never name internal machinery, and never lead with Getting (the Got rewrite is deleted so it cannot return). - Deployments name the workflow: Deploying {workflow} as API/chat app/MCP tool - Workflow reads name the part: Reading {workflow} meta/state/deployment/notes; generic reads always name the file (Reading {leaf}), never bare Reading file - Block runs name block and workflow: Running {block} in {workflow}, Running from {block} in {workflow}, Running {workflow} until {block}, and Enabling/Disabling {block} in {workflow} - The six split-table tools get per-operation verbs (Adding column {name}, Updating rows, Wiring automation, Creating view {name}) instead of a wall of Querying table - The manage quartet drops X-action system-speak for gerunds - get_* internal names become user language (Checking run settings, Tracing block inputs, Reading the deployed version); web_fetch says Fetching - Scheduled-task titles removed entirely (feature deleted from the Go catalog) - New verb rewrites: Fetched, Traced, Wired, Configured, Looked, Rotated * Deploying {workflow} as chat, not as chat app * Loader gerunds; mv names both ends; mkdir names the folder search_integration_tools -> Finding the right integration; load_integration_tool -> Loading {integration} tools; load_skill -> Loading skill {name}; run_enrichment -> Looking up {subject}. mv prefers the model's phrasing, else reads 'Moving {files} to {destination}'; mkdir reads 'Creating folder {name}' from the path. * Overflow counts read '+ n', dropping 'more' * Unify workspace find and search * Scale desktop title bar with page zoom * Serialize account and organization truth into the copilot VFS Workspace standing, membership, billing, org role, access-control restrictions, published-block provenance, and fork topology were reachable only through three parameterless tools (or not at all). They are ambient read-only facts, so they belong in the VFS where they are greppable, cost no tool round-trip, and every agent that can read gets them — the same move that retired get_blocks_and_tools and list_user_workflows. Adds account/{workspace,workspaces,members,billing}.json (always mounted) and organization/{organization,access-control,custom-blocks,forks}.json (only when the workspace is org-hosted). Every file projects an existing use case or util after getOrMaterializeVFS's access assert — no new queries, no new authorization. One relation per file, cross-referenced by id-and-name stub, so overlapping facts cannot disagree. Volatile content (billing, access control, forks) is lazy, so numbers are read-time fresh and unasked-for reads cost nothing. Projection follows the viewer: member emails are admin-only, fork detail requires workspace admin on a forking-enabled org, and the whole organization/ namespace is absent for a personal workspace — which is itself the answer. Retires get_account_billing, get_enterprise_context, and list_user_workspaces along with their handlers; display titles stay for transcript replay. * Fix insert_text refusing an editable field focused inside a frame describeFocusedEditable descended shadow roots but not frames, while activeElementReadback descends both. Focus inside a same-origin frame therefore surfaced to the first as the FRAME element — not an input, not contentEditable, not a canvas, no textbox role — so it fell through to 'not-editable' and insert_text refused a field that press_key had just typed a character into. Two functions answering 'what is focused' with different answers is the bug; the descent loops now match exactly. The refusal also names what actually held focus (tag, role, contenteditable). A bare 'not-editable' gave the agent nothing to act on, so it guessed at the cause — a real run spent twenty rounds on the wrong theory and had to be stopped by the user. * Keep retired browser takeover renderable in history The tool is gone from the catalog, so its generated constant went with it and every path that referenced it stopped compiling. Deleting those paths instead would have silently downgraded every past transcript containing a takeover card to a generic tool row, and dropped the no-timeout budget that an in-flight takeover still needs while a rolling deploy finishes. retired-tools.ts gives the literal a documented home that says what it is and why it survives its tool. * Follow the agent into a tab it opened to work in browser_open_tab created the page with activate: false, so the agent worked in a tab the user could not see while the panel sat on a page where nothing was happening. The panel now follows a tab the agent deliberately opened. Scoped to that tool only. A page spawning its own tab (popup, target=_blank) is the site grabbing the view rather than the agent choosing a workspace, and stays in the background as before — two existing tests pin that and caught the first version of this change, which moved both. A tab the user claimed still wins over both: the work starts in the background instead of pulling the page out from under them mid-read. * Make the browser tools agree with each other An audit of the module found the frame-descent bug was one instance of a pattern: six independent definitions of 'is this editable' and seven of 'what is focused', disagreeing with each other. A tool refusing what its sibling accepts on identical page state is invisible at runtime — the agent follows a snapshot that says one thing into a tool that says another. - browser_type now accepts role="textbox" like browser_insert_text does. The snapshot advertises those elements as [textbox] with a ref, so refusing them meant rejecting exactly what the outline told the model to type into. Both the native and synthetic paths, and their descendant scans. - pressKeyOnPage descends shadow roots and frames like every other focus reader. It was dispatching synthetic keys at the shadow host or <iframe> element, where they bubble but never reach the editor, while reporting success — and contradicting the activeElement reported beside it. - not-editable and ambiguous-editable name what was found: the element's tag and role, and the candidate fields. Both had the data and discarded it, which is what turns one blocked step into twenty rounds of guessing. - obstructedAfterNavigation requires a dialog that ARRIVED with the navigation. It compared against nothing, so every SPA route change under a persistent role=dialog reported a successful click as obstructed. The test that covered this asserted the false positive; it now pins both directions. - browser_insert_text observes the top document when typing inside a frame, like every other input tool. A submit that navigates the top page was invisible to its frame-scoped observation. * Let hover actually see what it mounted Four independent defects made browser_hover blind to the most common thing a hover produces — a row's action bar — so it reported no effect on a hover that worked, and the agent fell back to clicking pixels off screenshots. - The popup scan matched only role=tooltip/menu/listbox. Slack's message shortcuts bar is a labelled toolbar/group, so it registered as nothing at all. Added toolbar, menubar, labelled group, and [popover]. - The baseline was captured BEFORE prepareElementSurface scrolled the target into view, so scrollChanged was always set by the tool's own probe. That pinned every unproductive hover to 'background DOM churn' instead of the honest 'nothing happened', and hid scrolling the hover really caused. Re-baselined once the scroll settles and before the pointer moves. - The MutationObserver attached only on the first observation, while the roots list is rebuilt every call and grows as shadow roots mount. Components that appeared later were never observed, so their DOM changes raised no revision. Roots are now observed as they show up. - observationTruncated was computed and never read, so a scan capped at 12k nodes reported 'nothing appeared' with the same confidence as a complete one — and portalled overlays live at the end of <body>, exactly what the cap drops. Hover now says the page was too large to scan and to confirm visually. * Stop the browser agent acting on the wrong element, and say why it refused Four findings from the module audit, the first of which could silently do the wrong thing rather than merely fail. - A ref whose node is gone is re-adopted by structural resemblance, matching on ORIGIN only so a pushState between snapshot and act does not kill every ref. That leniency also let a ref to a row control in one view rebind to the identical control in a view the app had since navigated to — acting on the wrong message, signalled by nothing louder than recovered: true. Adoption now requires the same path; a view swap reports the ref stale, and the caller re-snapshots. Revalidating a still-connected node stays lenient, because that is literally the node the model chose. - A hit INSIDE the requested element is its own nested control, not an overlay. Both produced 'covered by X — close or move the overlay', advice that cannot be followed because there is nothing to close. Nested hits now say so and point at retargeting. - browser_click_at, browser_insert_text, and browser_drag listed targetChanged in their effect formulas, but none passes an elementId, so no targetState is ever captured and the term was always false — coverage that read as real. Removed, with a test pinning the dependency. - The seven effect formulas are deliberately NOT collapsed into one predicate: drag must trust domChanged where others must not, hover must ignore field and focus changes, click counts focus only for editables. Forcing one would make each tool wrong differently. The differences are now documented in one place next to the shared computation, so divergence is a declared policy rather than an accident. * Let edit_workflow configure block retries * Updates * Always focus the resource the agent is working on, and its browser tab The resource panel had a carve-out: an already-open browser session declined to replace another selection and only got an attention marker, so agent browser work happened off-screen. The panel now follows the agent to whatever it touches — browser included — and an event can still opt out explicitly. The browser panel also follows the agent BETWEEN tabs: the store already tracked automationTabId (and the strip marked it), but the visible tab never changed. It now switches when the agent's target tab changes, so watching the agent never means hunting for the tab it moved to. Keyed on the target changing rather than on it being set, so a user who browses elsewhere mid-run is only pulled along when the agent itself moves. * Never paint a browser snapshot at stale geometry (the modal-open flash) Opening a modal locks scroll, which removes the window scrollbar and reflows the panel — so a capture taken before the lock describes a rect the panel no longer occupies. The handshake painted that frame anyway and only then retried, so the replacement landed visibly offset from the page it stands in for: the flash. A capture is now checked against the host's live rect before it is painted; a mismatched frame is skipped and re-captured at the settled layout instead (modal retries go 2 -> 3 to absorb the extra settle). * Name the workflow in deployment and workflow-scoped tool titles 'Checked deployment status' never said which workflow — nor did the deployed- state read, run settings, block outputs/inputs, redeploy, promote, or the global-variable write. These tools carry a workflowId (often defaulting to the current workflow), so only the client can resolve a name: the enrichment layer now resolves it for the whole workflow-scoped family and passes it as workflowName, which every workflow title already reads. Titles: Checking {workflow} deployment status, Reading deployed {workflow}, Checking {workflow} run settings, Reading {workflow} block outputs, Tracing {workflow} block inputs, Redeploying {workflow}, Promoting {workflow} version {n} to live, and 'Adding workflow variable {name} in {workflow}' — each falling back to its unnamed form when no workflow resolves. * Name the block that ran; never fall back to a raw block id run_block and set_block_enabled carry only a blockId, so their titles would have printed an opaque UUID ('Running 7f3a2b91-… in Invoice Sync'). The enrichment layer now resolves blockId against the workflow store the same way it already did for run_from_block's startBlockId, and the base titles no longer accept an id as a name — an unresolved block reads 'Running block' rather than a UUID. * Add the missing Removing -> Removed rewrite The table work introduced 'Removing automation'/'Removing enrichment' with no past form, so those rows kept their present tense after completing. * Name the target resource in the remaining tool titles Table tools keep their operands nested under args and identify the table by id, so their rows said 'Adding rows' with no hint where: enrichment now lifts the nested args and resolves tableId against the cached workspace table list, giving 'Adding rows to Runtimes', 'Adding column status in Runtimes', 'Reading views of Runtimes'. Also: a block-schema read names the block instead of the file ('Loading Slack', 'Loading Google Sheets tips'); browser type/insert show the text they send, middle-ellipsized; downloads name the file; library-docs searches name the library and query; knowledge-base searches include the query; generated media names its output file; and diff_workflows, list_deployment_versions, and publish_custom_block joined the workflow-name enrichment set. * Bubble nested agents' tool calls into the parent's status line The collapsed status only scanned a group's OWN tool items and skipped nested agent groups, so a parent that had delegated froze on its last own tool while its child did the actual work — the line described nothing that was running. Status now walks the whole subtree: any tool at any depth counts, the most recently started running one is shown, and the rest become the same '+ n' overflow. With nothing running it falls back to the last tool at any depth, so an idle parent still reflects where its subtree got to. * Align nested tool call status rows * Revert branch-local KB connector error-message edits Restores apps/sim/connectors/ to staging state. Two copilot-focused commits on this branch (3a9fc1c5a5, 24b24e5f8e) drove by nine KB source connectors (airtable, confluence, discord, github, gitlab, google-drive, microsoft-teams, notion, slack) to enrich credential-validation error messages. The env-reference resolution machinery those errors supported is kept; the product connector surface stays unchanged in this branch so the staging promotion remains scoped to copilot work. * Fix the failing audits: NUL escape and route-count ratchet check:source-text: resource-vfs.ts used a raw NUL byte as its folder-index key separator (comment and template literal), which makes git treat the file as binary and hide it from review. Written as the '\u0000' escape — the runtime string is identical. check:api-validation:strict: baseline 1120 -> 1122 for the two routes added on this branch since the last bump. * Add Force Reload (Cmd+Shift+R) to desktop View menu * Route Force Reload through focused-resource boundary; regen docs manifest * Align the title-bar surface audit with the zoom rework 'Scale desktop title bar with page zoom' (833d2f126d) changed the CSS contract in two ways its audit test still pinned the old shape of: the lane vars gained a max() floor around the platform env() terms so page zoom cannot shrink the lane below the OS-drawn lights, and the control square became fixed px — CSS px already scale under zoom — leaving only the centering offset derived from the lane height. The test required the bare env() prefix and calc() on all three control vars, so it failed the commit that implemented its own regression comment. Pins now assert the env() term inside the clamp (still platform-derived, the test's actual intent) and split the control vars: offset must stay computed, size and icon are explicit constants. * Budget the two structurally slow tests explicitly sso-trust imports the whole Better Auth module graph (2.5s on an idle machine) and events.attribution scans call sites across the repo. Under a fully-parallel uncached run on a loaded machine both blow the default timeout while passing in isolation and on CI — a verdict decided by machine load, not by the code. 30s budgets make a loaded local run mean what it says. * Carry the cross-service trace id in sim log lines * Improve nested tool status presentation * Refresh secret schemas and shared test mocks * Open the deployed graph of org-published blocks, read-only, org-wide A consuming workspace could see a published block's interface but never what it does: the backing workflow lives in the publishing workspace, and other workspaces are nameable, not readable. Publishing a block org-wide is the act of sharing it, so the graph it executes is now readable from any org workspace — the DEPLOYED graph, not the publishing workspace's live editor state, so nothing in-progress leaks and what you read is what runs. The namespace also adopts the root's index/detail split: custom-blocks.json slims to names with a detail pointer, organization/custom-blocks/{type}.json carries provenance plus the deployed graph (loaded lazily through the cached loadDeployedWorkflowState; credential ids and env references inside it belong to the publishing workspace and say so), and organization/README.md is the namespace guide WORKSPACE.md is at the root — files, usage, the in-depth block inventory, and forks.json documented only when actually mounted. The block list moves to materialize time (same indexed query the components pass already runs) because the README, the index, and the per-block key-view entries all need it; only the graph stays lazy. * Withhold the deployed graph from external collaborators Workspace access and org membership are different grants: an external collaborator can open the workspace and use the published block, so the names index and the interface schema stay visible to them — but the deployed graph is org implementation internals, and their detail files now simply do not exist. isHostOrganizationMember is the viewer bit the host context already resolves for exactly this distinction. * Fill out the organization namespace: workspaces, permission groups, credential groups Three more read-only files, all lazily loaded — the paths appear in the key view so glob discovers them, but no query runs until a read — and all gated by registration, so an unpermitted viewer's file simply does not exist: - workspaces.json (org members): the org's full workspace map with the viewer's access flag and fork parentage — account/workspaces.json only ever showed what the viewer can reach. Inaccessible workspaces stay nameable, not readable. - permission-groups.json (org admins): every group with member count, targeted workspaces, and the restrictions its config activates. access-control.json remains the per-viewer binding. The queries are lifted into lib/permission-groups/queries.ts because their only prior home was inline drizzle in the route handlers, which the VFS cannot import. - credential-groups.json (entitlement-gated): per-option configuration readiness and enrollment progress — the two facts that decide whether a credential_group workflow will do anything at runtime. The note teaches the contract that bit the audit: an active group with zero completed enrollments yields an empty loop, not an error. Enrollee emails are workspace-admin-only, matching the settings page; counts come from the first enrollment page and say so when truncated. The README documents each file only when mounted for this viewer. * Stop reporting a click that navigated as a failed click The field case: 'Begin Assessment' submits a form. The navigation tears the origin document down while the press completes, so everything after dispatch — the CDP call's own completion, the synthetic dispatch's return value, the postcondition reads — fails against a destroyed context, and a maximally successful click was reported 'Failed clicking element'. The agent's own follow-up investigation in the transcript diagnosed exactly this. navigationRescue detects it at the driver level, where the navigation epoch and URL survive the renderer teardown: when the page provably navigated since dispatch began, a dispatch-path failure becomes a success carrying navigatedDuringDispatch and a note explaining why no postconditions exist. Applied to all four click dispatch paths (unframed CDP, framed native, synthetic in-page, click_at). The soft path had the same blindness: with the after-state unreadable, urlChanged computed false and a navigating click reported 'no observable change'. navigatedByDriver now folds into navigated/effectObserved, which also keeps the new-dialog obstruction check meaningful on real navigations. * Re-hide the browser under a modal after main loses the occlusion lease The punch-through: a modal opens, the native view hides behind a painted snapshot, and the renderer records applied: true. Then one heartbeat commit is skipped — renderer jank past the 2.5s bounds-lease TTL is enough — and main expires the lease, resetting panelOccluded on its side. The next heartbeat finds the modal marker still present and calls setDesired(true), but the lease's dedupe sees applied === desired and sends nothing; the bounds commit that follows lays out an unoccluded native view above the open modal, and no later event ever re-hides it. The comment on this branch already claimed it 'reasserts the lease' — the dedupe made that claim false exactly when the lease had been lost. While the occlusion marker is present, each heartbeat now drops the applied belief (assumeRevealed) before setDesired, so the reassert is a real, forced, idempotent hide IPC — one per second while a modal covers the browser — and any main-side lease loss self-heals within a heartbeat. * Make a stale ref name which of its five causes fired A field run burned five snapshot->click cycles on a STATIC landing page, every one refused with the same sentence — 'the page changed since the last snapshot' — and the agent reasonably concluded the page was regenerating its DOM. It was not; the resolver was refusing, and the message could not say why. Five distinct conditions produced that one string: an id missing from the registry, a connected node whose identity drifted, the view-changed adoption gate, no confident replacement, and a replacement tie. The resolver now stamps the reason (with the drifted node's current identity, or the from->to paths for the view gate) and every stale producer carries it into the driver message. Same pattern as not-editable: a refusal that names its cause costs one round; an opaque one costs a loop and a wrong theory in the bug report. * Pulse throttling off when the browser view is revealed, so it actually paints The blank-page report: a navigation completes while the view is hidden, the page 'finishes loading', and the panel shows white until the user re-navigates by hand. invalidate() on reveal was already there but recomposites the LAST frame — and the last frame is blank, because background throttling suspended the rAF the page's SPA paints its first frame from. The reveal now pulses throttling off (forcing the renderer to produce a real frame), invalidates, and hands the policy back to the session a second later through reassertTabThrottling, which preserves the automation-tab exemption. * Let the agent browser join meetings and finish passkey sign-ins Camera and microphone: 'media' joins the agent partition's allowlist, but every grant is gated on the macOS grant first — asked via systemPreferences.askForMediaAccess so the system prompt appears on first use, and answered from getMediaAccessStatus on checks — so System Settings stays the real authority and a page can never hold a grant the OS refused. Granting site permission without the OS grant produced the misleading NotReadableError Google Meet showed. Packaging gains the camera entitlement and usage string (macOS kills the process on prompt without one) and the mic string now covers meetings. Passkeys: WebAuthn itself is Chromium-native and nothing in our handlers blocks it — USB security keys need no permission at all. The hybrid transport (passkey on a nearby phone via QR) rides Bluetooth, which signed builds silently lacked: the bluetooth entitlement and usage string enable it. iCloud-Keychain platform passkeys remain outside what an entitlement here can grant — Apple restricts that to approved browsers. * Compile and preview Sim-styled pages * Wait for the batched prepare intent instead of instantly failing apply_file_edit The model batches prepare_file_edit and apply_file_edit into one round and the Go loop runs same-round tools concurrently, so apply could reach the executor before its prepare staged the intent. The instant no-intent error cost a model retry round and flashed 'Failed creating …' on the shared file row before the retry succeeded. The apply handler now polls briefly (10s cap) for the intent; a truly missing prepare still errors at the deadline. * Store page source, render docs-styled documents on view The pdf model for agent pages: the .html file keeps the markdown-shaped source (frontmatter + prose + sim: fences) and every surface renders the docs-styled document on demand — preview panel, /api/files/serve, public shares, and downloads all call the same pure compiler, now shared in lib/workspace-files. The docs chrome is reproduced from the real fumadocs source: the left sidebar's exact pill metrics, the clerk TOC with its animated scroll indicator, divider-style tables; cards and stats left the vocabulary. Table cells and kv values render inline markdown, and sim:workflow/table/knowledge/file links resolve to real workspace routes, bridged out of the sandboxed preview to the app router. Hand-written imitations of rendered output are rejected at apply_file_edit with a steer back to source, and a streaming page hides its source behind the live rendered preview (batched ~2s) the way a generating pdf hides its script. * Stagger the page rails so the resource panel keeps the docs sidebar Rails were gated at 1100px of iframe width — the chat resource panel never reaches that, so pages rendered single-column there. The rails now stagger the way the docs do on a laptop: >=640px keeps the section sidebar (240px) beside the content, >=1060px restores the full three-column frame with the clerk TOC, and only a truly narrow pane collapses to one column. * Fix in-page anchors escaping the preview; add the docs' toggle, code frames, pagination, and images Clicking a TOC or section link (or pressing Enter in the section filter, which clicks one) navigated the sandboxed frame off about:srcdoc in Electron and landed on a cookie-less sign-in page — the shell now intercepts every '#' anchor and scrolls directly. The page chrome gains the docs' exact theme toggle (emcn Sun/Moon, 30px rounded-lg, top right), framed code blocks with language label and copy button, and footer previous/next cards from prev/next frontmatter. Workspace images (![alt](sim:file/<id>)) compile to /api/files/view and the preview host inlines them as blob: URLs so the cookie-less frame can render them; sim:accordion joins the vocabulary as the faq component with title keys. * Size the section sidebar to its content so it appears at panel widths The left rail was a fixed 240/300px column, so it only earned its place once the pane was wide. fit-content caps at the docs width but shrinks to the longest section title (150px floor for the pills), and the two-column tier now starts at 560px instead of 640. * Let the clerk TOC join at 860px by sizing it to its content Same move as the section sidebar: the TOC column fits its longest link (capped at the docs' 268px, 150px floor), so the full three-column frame starts at 860px of pane width instead of 1060. * Open external links from pages in a new tab The preview bootstrap cancelled every non-anchor click, so an external link (the Sim docs, a vendor page) did nothing. External http(s) links now compile with target=_blank rel=noopener for the standalone and share surfaces, and the sandboxed preview bridges the click to the host, which window.opens a new tab — same channel the workspace deep links use. * Lock Sim pages to the rendered view via an internal record type The record's contentType is stamped text/x-sim-page when apply_file_edit detects page source — the file stays .html to the user (serving and downloads still emit text/html), but every surface now knows what the file holds before content loads. The viewer forces the rendered view for these files at every moment: the first streamed chunk (whose frontmatter is still partial) no longer flashes raw source, the gaps between an agent's tool calls no longer flip back to raw HTML, and both toggle surfaces (the Files toolbar and the resource-panel tabs) stop offering a code view for them. Mid-stream compiles run lenient — a fence still being written is malformed by definition, so its skip-notice callout is suppressed until the stream settles. * Honor the model's declared page type; default copilot .html to a page An explicit contentType on create_empty_file always wins (the skill now declares text/x-sim-page for pages, text/html for bespoke raw pages); with no declaration a copilot-created .html defaults to the page type. The first apply_file_edit still re-confirms from the actual content, and the category map knows the internal mime explicitly instead of falling through to the extension. * Default undeclared .html back to plain text/html A file is a Sim page only when the model declares it at creation or the first written content proves it — never by extension alone. * Match the docs' PageFooter for page navigation The invented bordered cards with Previous/Next labels are replaced by the docs' actual footer: the destination name with a 14px emcn chevron on a flex-1 hover pill (rounded-lg, px-3 py-3, --surface-active), next right-justified, and a spacer holding the empty half — verified against apps/docs/components/docs-layout/page-footer.tsx. * Scroll the rails invisibly, like the docs The sticky TOC box is overflow-y auto, and the clerk track's absolutely positioned SVGs could tip it a few pixels into overflow — Chromium then painted a full scrollbar beside the rail. Rails now hide their scrollbar chrome entirely (scrollbar-width none + webkit display none), matching how the docs scroll their sidebar and TOC. * Send sim:file links to the Files page, like a markdown link A workspace-file link in a page now navigates exactly as one tagged in a .md does: an in-app SPA push to /workspace/{ws}/files/{id} (the Files page with the file open). The fullscreen /view route stays reserved for the standalone surface; image refs keep the /api/files/view byte route. * Inline workspace images after the page compiles, not before The blob substitution ran on the raw source, where the compiled /api/files/view src it looks for does not exist yet — so the sandboxed cookie-less frame fetched every image itself and got 401s (broken image icons). The substitution now runs on the built document, covering compiled pages, legacy stored-compiled pages, and bespoke HTML alike. * Highlight the section you are AT in the left rail, not the last one visible The rail's current-section pick walked every heading visible in the viewport and kept the last h2 — so clicking a section landed correctly but highlighted whichever later section peeked in from below. Current is now the last h2 at or above the top reading line (matching the 72px scroll-padding a clicked anchor settles at), falling back to the first visible section when everything is below the line. * Stop the TOC jittering sideways as the highlight moves Active TOC links step from weight 430 to 470, and the rail is a fit-content column — every active-section change re-measured the longest link and shifted the rail a pixel or two side to side. Each link now carries a hidden zero-height ghost of itself at the active weight, so the column always occupies its bold width and the highlight moves without the layout moving. * Absolutize links and images in served page documents A downloaded page must behave like a downloaded .md whose links are absolute: clicking a workflow reference opens Sim in the browser at that workflow. The standalone renderer (plain download, fullscreen viewer, shares) now compiles sim: links and workspace image refs against getBaseUrl(); in-app surfaces keep relative paths and SPA navigation. * Drop the eyebrow; add the docs' top page controls The docs have no eyebrow line, so compiled pages no longer render one (old sources still parse; the field is ignored). The title row gains the docs' top controls: Copy page (copies the page text) and prev/next chevrons wired to the same neighbors as the footer cards, disabled-dim when a side is missing. * Read kv keys and table first columns as labels, the docs way kv keys dropped the blanket monospace — they render as the docs' row labels (500, primary, sans), with backticks in the source opting a code-like key (a path, an env var) into the inline-code chip; keys now run through inline markdown to make that work. Table body first columns pick up the same label treatment the docs tables show. * Platform font and emcn chrome for pages Pages live inside the app, and the platform — emcn, every workspace surface — renders the system stack, not the docs' Inter webfont; Inter made pages read as foreign next to the app around them. The face is now the platform stack with weights on the platform scale (400/500/600), the Inter delivery machinery (page-font.ts, the preview data-URI fetch, the public woff2) is gone, and the section filter wears emcn ChipInput's exact chrome (30px rounded-lg, --surface-5 fill flipping to --surface-4 in dark, --border, 14px, no focus ring). The docs' geometry — layout, spacing, rails, tables, code frames — is unchanged. * Color-only active state in the TOC — width can never move again Two attempts at reserving the bold width (a hidden ghost, then freezing measured rail widths) each traded one artifact for another: the ghost did not stop the fit-content column re-measuring, and the freeze made a long link wrap to two lines when it gained weight. The root cause was letting the active state change a width-affecting property at all: the active TOC link now shifts color only (muted to primary — the clerk thumb already carries the emphasis), the search input is a plain text field (the native search clear button is not emcn chrome), and the ghost/freeze machinery is gone. * Drop the Copy page control; keep the top chevrons The title-row actions keep only the previous/next chevrons (rendered when the page has neighbors); the Copy page button is gone. * Set-level sidebar: docs-style groups with the current page expanded Multi-page sets can now carry the whole set's sidebar. nav frontmatter (groups of labelled page links, identical on every page of the set) compiles into hidden set-nav markup whose sim: links resolve like any other; the shell lifts it into the left rail as muted group labels over page links, recognises the current page by title, and nests that page's section list beneath it — the docs sidebar's exact shape. Pages without nav keep the plain section list. * Center the content column at the docs' measure On wide panes the 1fr center cell stretched, so content hugged the left rail with dead space before the TOC. The main column now caps at the docs' ~760px measure and centers in its cell, matching how the docs balance a wide viewport. * Docs Steps, code-tab groups, and API method chips Three docs components join the vocabulary: sim:steps renders the numbered timeline (muted circle markers, hairline connector, title and content per step); sim:tabs renders the docs' grouped code block (mono tab chips, one pane at a time, the icon copy control targeting the visible pane); and a METHOD prefix on a set-nav page entry renders the API-reference chip — sidebar entries only, on the platform badge tokens (blue and purple added to the mirrors and the live bridge). Also: preview images switched from blob: to data: URIs — blob URLs are origin-bound and the sandboxed frame's origin is opaque, so Chromium refused to render them — and the page view-lock is sticky per file so a patch stream cannot flash raw source. * Downloaded pages carry their images Absolute URLs made LINKS survive a download, but an embedded image request from a downloaded file is cross-site and carries no session cookie, so images 401ed outside the app. The standalone renderer now inlines every workspace image the page references as a data: URI at serve time — like a pdf carrying its images — capped at 8MB per image, restricted to the page's own workspace, falling back to the URL reference on any miss. Applies to serve, download, and public shares. * Dead-center the content column between equal gutters The rails are content-sized and unequal, so the old grid (fit-content / 1fr / fit-content) skewed the middle cell toward whichever rail was narrower. The wide tier now uses the docs' geometry: a fixed 760px content column centered between two equal flexible gutters, the sidebar hugging the container's left edge and the TOC its right — rail widths can no longer move the content. * Defer title-bar history state out of currententrychange dispatch The Navigation API fires currententrychange synchronously from the history mutation that caused it, which can originate inside another component's useInsertionEffect (style libraries navigating during commit) — setState there trips React's 'useInsertionEffect must not schedule updates'. The arrow-state sync now defers to a microtask, flushing after the commit unwinds, with a disposal guard. * Show the section sidebar only when there are sections to list Fewer than two sections (and no set nav): the left rail and its filter disappear and the reserved left gutter collapses — the content column leads the container with the TOC trailing. Two or more sections, or a multi-page set, keep the full centered docs frame. * Execute the page shell in a DOM harness jsdom runs the real shell against compiled pages and asserts the layout decisions: both rails on a many-section page, only the left rail dropped on a one-section page. * Medium panes keep the TOC, not the sidebar Between 560 and 860px the frame showed the section sidebar and hid the clerk TOC — backwards by our own reasoning, since the sidebar is the redundant list on a single page. The TOC now survives at medium widths and the sidebar joins only on wide panes. * Sidebar is for doc sets only; stagger the rails like the docs The left rail now exists only for multi-page sets — on a lone page it just repeated the TOC. A set opens its sidebar at 560px with the TOC joining on wide panes (the docs stagger); a lone page waits until 700px and then shows the TOC alone. Set-sidebar spacing tightens to the docs values, with the current page's nested sections styled as small muted entries behind a hairline instead of full chips. * Center the lone-page content and TOC as a pair On a page with no sidebar the content column stretched while the TOC hugged the far edge, leaving a field of dead space between them. The content now caps at reading width with the TOC directly beside it and the pair centered in the pane, and the TOC waits until 800px so narrow panes stay single-column a while longer. * Equalize grid-template specificity across the rail tiers The 560px tier selects .art-cols:not(.no-side-nav) at (0,2,0), so the 860px tier's bare .art-cols template at (0,1,0) could never win and a set page on a wide pane kept the 2-column template — wrapping the TOC to the next grid row, bottom-left. The wide template now carries the same :not() guard, the 860 block's duplicate of the 800px no-side-nav rules is gone, and a test pins every template rule to equal specificity so a future tier can't silently lose the cascade again. * extract_doc_assets: pull a reference deck's assets into the workspace Sim-side handler for the new file-agent tool: given an uploaded .pptx or .docx, unzip it (OOXML is a zip), parse theme1.xml into theme.json (color scheme as hex, major/minor fonts, slide size from presentation.xml) and write every ppt|word/media file into a "<Name> assets" folder with original bytes and real content types. Re-runs overwrite the set in place. Pure extractor unit-tested against in-test-built packages; display label "Extracting assets from <file>". * extract_doc_assets learns .pdf via the doc sandbox PDFs have no zip structure or declared theme, so extraction runs in the same vetted sandbox that compiles and renders documents: poppler's pdfimages dumps every embedded image in its native format (masks filtered via -list), pdfplumber contributes each image's placement rects in page points plus the document's font names, and rendered pages are sampled into an explicitly-inferred color palette. theme.json for a pdf carries fonts, page size and count, the inferred palette, and a per-asset placement map. * Pages have one navigation rail; compile errors go to the agent The left sidebar leaves the renderer and the DSL: the shell builds only the content column and the clerk TOC (pair centered at 800px, one bare .art-cols selector per tier so the cascade cannot invert), the filter box goes with it, and nav frontmatter is tolerated but no longer rendered — sidebar METHOD chips and the set-nav markup are gone. Malformed sim: blocks no longer render a reader-facing "block was skipped" card: the block is omitted and the failure is reported as a diagnostic that apply_file_edit appends to its result, so the authoring agent sees exactly which fence to fix. The lenient flag existed only to suppress those cards mid-stream and is removed. The steps timeline connector now derives its position from the marker size, so it stays centered under the number circles. * Sync tool catalog: extract_doc_assets accepts pdf * Asset extraction yields the rebuild recipe, not just the parts pptx: theme.json now maps every image to its slide-by-slide placements (slide rels resolve rIds to media names; each pic frame's EMU offset and extent convert to inches) plus the slide count. pdf: a second layout.json is written — per page, the text blocks with content, position, font, size, and fill color; the filled rects (backgrounds and scrims); and rect-over-image overlay detection with coverage, which is the "image opacity" effect decks fake with a tinted rect. Stream alpha is unrecoverable, so overlays name the color and the rendered page remains the reference for strength. * Split shared-baseline text runs into separate blocks Two text boxes sitting at the same height merged into one wide line; a gap much wider than a space now starts a new block, so columns and label/value pairs land as distinct entries in layout.json. * Extract faithful document layout recipes * Add .chart files: live interactive ECharts docs, static or table-backed * Size charts by width-driven aspect, not panel height; separate title and legend * Map table chart rows from storage column ids to display names * Inject table rows as datasetIndex 0 so specs can transform; stagger array legends * Give .chart files their own bar-chart icon * Chart table sources gain groupBy/aggregate/pivot shaping; renderer-owned chrome * sim:chart page fence: ECharts SSR to themed inline SVG; shared option builder * sim:chart hydrated embeds: inline or .chart file refs, live table reads per serve * Finish extensionless pages and document staging * Render live charts without server hydration * Cover active-theme page token overrides * Keep artifact tokens synced with the app theme * Use tabs for multi-page Sim docs * Preserve dollar-prefixed tool credentials * Build chart specs from validated fields * Sync new integration docs into Copilot manifest * Add in-document tabs to Sim pages * Rebuild page TOC on tab changes * Keep page tabs with docs chrome * Stabilize tabbed page layout * Regenerate docs manifest for staging's Modal docs page * Share one divider between the bar and the chrome tab row * Pin the keyless OCR path in the unreadable-document test --------- Co-authored-by: Justin Blumencranz <96924014+j15z@users.noreply.github.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
1 parent 6f34dd6 commit d9cfd7c

393 files changed

Lines changed: 30457 additions & 6226 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/test-build.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,9 @@ jobs:
123123
- name: Repo audits
124124
run: bun run check:audits
125125

126+
- name: Verify docs manifest is in sync
127+
run: bun run docs-manifest:check
128+
126129
- name: Migration safety (zero-downtime) audit
127130
run: |
128131
if [ "${{ github.event_name }}" = "pull_request" ]; then

apps/desktop/build/entitlements.mac.plist

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,5 +9,15 @@
99
even after the user grants access in System Settings. -->
1010
<key>com.apple.security.device.audio-input</key>
1111
<true/>
12+
<!-- The agent browser joins real meetings (Google Meet, Zoom web): its
13+
getUserMedia grant is gated on the OS grant, and without this key the
14+
Hardened Runtime denies the camera no matter what the user allowed. -->
15+
<key>com.apple.security.device.camera</key>
16+
<true/>
17+
<!-- WebAuthn hybrid transport (passkey on the user's phone via QR) rides
18+
Bluetooth proximity; without this the QR option silently never
19+
completes in signed builds. -->
20+
<key>com.apple.security.device.bluetooth</key>
21+
<true/>
1222
</dict>
1323
</plist>

apps/desktop/electron-builder.yml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,9 @@ mac:
5858
# macOS refuses to show the microphone prompt at all — it kills the process —
5959
# unless the bundle declares why it wants the device.
6060
extendInfo:
61-
NSMicrophoneUsageDescription: Sim uses your microphone for voice input in Chat.
61+
NSMicrophoneUsageDescription: Sim uses your microphone for voice input in Chat and for meetings you join in the built-in browser.
62+
NSCameraUsageDescription: Sim uses your camera for meetings you join in the built-in browser, such as Google Meet.
63+
NSBluetoothAlwaysUsageDescription: Sim uses Bluetooth to complete passkey sign-ins with a nearby phone in the built-in browser.
6264
entitlements: build/entitlements.mac.plist
6365
entitlementsInherit: build/entitlements.mac.plist
6466
notarize: true

apps/desktop/src/main/browser-agent/cdp.test.ts

Lines changed: 88 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,9 @@ import { describe, expect, it, vi } from 'vitest'
22

33
vi.mock('electron', () => import('@/test/electron-mock'))
44

5-
import { WebContentsView, type WebFrameMain } from 'electron'
5+
import { nativeImage, type WebContents, WebContentsView, type WebFrameMain } from 'electron'
66
import {
7+
captureScreenshot,
78
clickAt,
89
ensureInstrumented,
910
evaluateInIsolatedFrame,
@@ -482,3 +483,89 @@ describe('browser-agent CDP theme', () => {
482483
})
483484
})
484485
})
486+
487+
/**
488+
* The browser panel shows a LIVE view, so a capture must not perturb the page.
489+
* Chromium serves `clip` by applying device-emulation params to the widget and
490+
* syncing visual properties, which the user sees as the page rescaling and
491+
* snapping back. Resolution is bounded on the returned image instead.
492+
*/
493+
describe('browser-agent screenshot capture', () => {
494+
function captureFixture(imageSize: { width: number; height: number } | null) {
495+
const contents = new WebContentsView().webContents
496+
vi.mocked(contents.debugger.sendCommand).mockImplementation((method: string) => {
497+
if (method === 'Page.getLayoutMetrics') {
498+
return Promise.resolve({ cssLayoutViewport: { clientWidth: 2048, clientHeight: 1024 } })
499+
}
500+
if (method === 'Page.captureScreenshot') return Promise.resolve({ data: 'c2lt' })
501+
return Promise.resolve(undefined)
502+
})
503+
const resized = {
504+
toJPEG: vi.fn(() => Buffer.from('resized')),
505+
}
506+
// Shared module-level mock: without this, a later fixture reads the
507+
// earlier test's decoded image.
508+
vi.mocked(nativeImage.createFromBuffer).mockReset()
509+
vi.mocked(nativeImage.createFromBuffer).mockReturnValue({
510+
isEmpty: vi.fn(() => imageSize === null),
511+
getSize: vi.fn(() => imageSize ?? { width: 0, height: 0 }),
512+
resize: vi.fn(() => resized),
513+
toJPEG: vi.fn(() => Buffer.alloc(0)),
514+
} as unknown as ReturnType<typeof nativeImage.createFromBuffer>)
515+
return { contents, resized }
516+
}
517+
518+
function screenshotParams(contents: WebContents): Record<string, unknown> {
519+
const call = vi
520+
.mocked(contents.debugger.sendCommand)
521+
.mock.calls.find(([method]) => method === 'Page.captureScreenshot')
522+
if (!call) throw new Error('no capture was requested')
523+
return call[1] as Record<string, unknown>
524+
}
525+
526+
it('never sends a clip, which would emulate the live page for the capture', async () => {
527+
const { contents } = captureFixture({ width: 4096, height: 2048 })
528+
529+
await captureScreenshot(contents)
530+
531+
expect(screenshotParams(contents)).not.toHaveProperty('clip')
532+
})
533+
534+
/**
535+
* A 2048px CSS viewport bounded to 1024px is scale 0.5, and the capture
536+
* arrives at device resolution (4096px on a 2x display). The resize is what
537+
* lands the image on the CSS-relative size the coordinate contract
538+
* (cssX = imageX / scale) assumes.
539+
*/
540+
it('downscales the returned image to the CSS-relative size', async () => {
541+
const { contents, resized } = captureFixture({ width: 4096, height: 2048 })
542+
543+
const shot = await captureScreenshot(contents)
544+
545+
const image = vi.mocked(nativeImage.createFromBuffer).mock.results[0].value
546+
expect(image.resize).toHaveBeenCalledWith({ width: 1024, height: 512, quality: 'good' })
547+
expect(resized.toJPEG).toHaveBeenCalled()
548+
expect(shot).toEqual({
549+
dataUrl: `data:image/jpeg;base64,${Buffer.from('resized').toString('base64')}`,
550+
scale: 0.5,
551+
})
552+
})
553+
554+
it('skips the re-encode when the capture already matches the target size', async () => {
555+
const { contents } = captureFixture({ width: 1024, height: 512 })
556+
557+
const shot = await captureScreenshot(contents)
558+
559+
const image = vi.mocked(nativeImage.createFromBuffer).mock.results[0].value
560+
expect(image.resize).not.toHaveBeenCalled()
561+
expect(shot).toEqual({ dataUrl: 'data:image/jpeg;base64,c2lt', scale: 0.5 })
562+
})
563+
564+
it('returns the raw capture when the image cannot be decoded', async () => {
565+
const { contents } = captureFixture(null)
566+
567+
const shot = await captureScreenshot(contents)
568+
569+
expect(shot).toEqual({ dataUrl: 'data:image/jpeg;base64,c2lt', scale: 0.5 })
570+
})
571+
})

0 commit comments

Comments
 (0)