Repository navigation
Description
Problem
Creating a conversation in Agent Canvas takes about 1.5 s before the agent even makes its first LLM call. I profiled it with a mock LLM that answers instantly, so everything below is pure agent-server overhead on the create path (POST /api/conversations blocks on agent init when the request carries an initial message, conversation_service.py#L1849). Canvas on the Local backend only navigates to the conversation after that request returns, so the user stares at the home page for the whole time.
There are three causes, in order of how many users they hit.
1. The terminal sets the tmux environment one variable at a time (everyone, every conversation). TmuxPanePool.initialize runs one tmux set-environment subprocess per variable (tmux_pane_pool.py#L139), and _new_owned_session runs one unset-environment per removed variable (tmux_terminal.py#L165-L166); TmuxTerminal has the same loop (tmux_terminal.py#L245). Each call costs about 10.5 ms, so startup grows linearly with the environment: 0.67 s at 55 variables, 2.8 s at 255. A real Canvas agent-server I checked has 103 variables, which is about 1.1 s per new conversation.
2. Worktree mode does git fetch and git worktree add on the event loop (users who pick "new worktree"). _plan_conversation_worktree and _create_conversation_worktree are called synchronously inside _create_conversation (L1676, L1717), and the default start point runs git fetch origin with a 60 s timeout on every create (L171, L197). On a clone of this repo that adds about 0.6 s of fetch and 0.7 s of checkout, and /alive stalls for a median 0.78 s meanwhile, so every other conversation freezes too.
3. MCP servers start inside the create, for every conversation (users with MCP servers). _runtime_mcp_tools starts all enabled servers in parallel and waits for the slowest (local_conversation.py#L1657), bounded only by the 30 s default timeout (mcp/utils.py#L397). One stdio server that starts instantly still adds 0.77 s; one that takes 3 s makes the create take 5.1 s.
Measured with an uninstrumented server and Canvas-shaped requests (warm medians, 95% CI in brackets):
| scenario | create time |
|---|---|
| home create, today | 1.20 s [1.13, 1.28] |
| home create, with the tmux calls batched | 0.33 s [0.33, 0.36] |
| new-worktree mode, repo with origin | 2.56 s [2.49, 2.84] |
| one MCP server that starts instantly | 2.05 s [2.02, 2.14] |
| released 1.62.0, +0 / +100 env vars | 1.34 s / 2.68 s |
End to end in the real Canvas 1.40 UI driven by Playwright, Enter → agent reply visible goes from 1.63 s (n=32) to 0.51 s (n=40) with only the tmux change; the two distributions don't overlap. The number of existing conversations made no difference.
Proposal
- Batch the tmux environment setup into a few
;-chained tmux invocations. A single call is not enough: tmux rejects the message withcommand too longat about 255 variables, so the commands need to be chunked (about 8 KB per call works). I have a 59-line patch that produces an identical session and global environment (checked with 300 values containing spaces,;, quotes and$) and leaves the terminal test failures unchanged versus main. - Move the worktree planning and creation into
asyncio.to_thread. (Skipping the fetch on repeat creates was tried in perf(agent-server): skip the worktree fetch when origin was fetched recently #6873 and dropped: we keep fetching origin for every new worktree.) - Stop blocking the create on MCP startup: start servers in the background and let the agent pick their tools up through the existing
on_tools_changedpath (agent/base.py#L902), waiting at most a short budget before the first LLM call so a first message that needs an MCP tool still sees it. - Warm the public-skill cache and the LLM stack at startup; the first create after launch is about 1 s slower than the rest.
- Later, and together with Canvas: return from
POST /api/conversationsbefore agent init and report startup progress over the WebSocket, so the page can open immediately.
I'd do these as separate PRs, starting with 1.
Regression guard
I'd also like to keep this from coming back. Two layers:
- Deterministic tests in the normal suite. Count tmux invocations during
TmuxPanePool.initializewith 0 and 200 extra environment variables and assert the count does not grow with the environment. This needs no timing thresholds, so it can't flake. For the event loop, create a worktree conversation against a localfile://origin with asyncio debug mode on and assert that no callback blocks the loop for more than ~100 ms during the create. For MCP (after 3), assert that a create with a deliberately slow stdio server returns well before the server is ready. - A conversation-start benchmark script next to
scripts/event_sourcing_benchmarks. It starts a real agent-server and a mock LLM, creates conversations the way Canvas does, and reports p50/p90 for create time, time to first agent event, and the worst/alivelatency during the create. In CI it would run on PRs that touch the agent-server, the terminal tool or the conversation code, and nightly on main, comparing the PR head against its merge base on the same runner (interleaved, so runner noise cancels out) and failing only on a clear regression, e.g. a median at least 25% and 150 ms worse. This harness could also host the concurrent-conversation load test from Add cheap load tests for concurrent conversations #4477.
Acceptance criteria
- Terminal startup makes a number of tmux invocations that does not grow with the environment size, and a test asserts that.
- Worktree planning and creation no longer run on the event loop.
- A create with a slow MCP server is not blocked for the server's full startup time.
- A conversation-start benchmark runs in CI and fails on a clear create-latency regression.
OpenHands AI triage
The following comments and acceptance criteria were added by the OpenHands AI agent.
Triage
Verified against main (2aae48b): the per-variable tmux set-environment/unset-environment loops (tmux_pane_pool.py initialize(), tmux_terminal.py _new_owned_session() and TmuxTerminal), the synchronous _plan_conversation_worktree/_create_conversation_worktree calls in _create_conversation (also on the fork path) with git fetch origin (60 s timeout) on every plan, and the MCP startup join in _runtime_mcp_tools with a 30 s default timeout are all present, and /alive (server_details_router.py) shares the event loop. The measurements reproduce a real slowdown in the actual product workflow (uninstrumented agent-server with Canvas-shaped requests; Canvas UI via Playwright), which warrants priority:medium.
Scope for this issue is items 1–3 of the proposal plus the proposed regression guard, delivered as separate focused PRs:
- Batch tmux environment setup into
;-chained, size-chunked invocations that tolerate the ~255-variable tmuxcommand too longlimit. - Move worktree planning/creation off the event loop (
asyncio.to_thread) and suppressgit fetch originwhenFETCH_HEADis recent, fetching only the default branch with a shorter timeout (cached-refs fallback already exists). - Stop blocking create on MCP startup: start servers in the background, reuse the existing
on_tools_changedpath for late tools, and wait at most a bounded budget before the first LLM call so tools ready within it are present at the first call. - Add the deterministic regression tests and the conversation-start benchmark harness next to
scripts/event_sourcing_benchmarks.
Non-goals: proposal item 4 (skill-cache/LLM stack warming), item 5 (returning from POST /api/conversations before agent init with WebSocket startup progress — cross-repo work with Agent Canvas UI, owned by OpenHands/OpenHands), any REST/OpenAPI or client-side API changes, and changing MCP failure handling (servers that fail startup still log a warning and are skipped).
Acceptance Criteria
- A test in the normal suite asserts that tmux invocation count during terminal/pool environment setup does not grow when the environment grows from 0 to 200+ extra variables, including values containing spaces,
;, quotes, and$. - The resulting tmux session and global environment after batched setup are identical to the current per-variable behavior (same keys and values visible to panes), verified by a test.
- With asyncio debug mode enabled, creating a worktree conversation against a local
file://origin runs no event-loop callback that blocks for more than ~100 ms, asserted by a test. - Creating worktree conversations repeatedly within the
FETCH_HEADrecency window runsgit fetchat most once; a stale or missingFETCH_HEADstill triggers a fetch; and when fetch fails or times out, the create still succeeds from cached refs. - Forked conversations with worktree isolation get the same off-event-loop and fetch-suppression behavior as regular creates.
- Creating a conversation with a deliberately slow-starting stdio MCP server returns before that server finishes starting, and the create succeeds.
- MCP tools from a server that becomes ready within the bounded startup budget are available to the agent's first LLM call; tools from servers still starting after the budget become available during the conversation via the existing tools-changed path, without restarting the conversation.
- The deterministic regression tests (tmux invocation count, event-loop blocking, MCP create-not-blocked) run as part of the normal test suite.
- A conversation-start benchmark script exists next to
scripts/event_sourcing_benchmarks, starts a real agent-server with a mock LLM, creates conversations the way Canvas does, and reports p50/p90 create time, time to first agent event, and worst/alivelatency during create; it is wired to run on PRs touching the agent-server, terminal tool, or conversation code, comparing PR head against merge base interleaved on the same runner, and fails only when a reported metric regresses by ≥25% and ≥150 ms at the median.
Update (2026-11-16): we decided to keep git fetch origin on every new worktree conversation, so the fetch-suppression items above (proposal 2 and the FETCH_HEAD acceptance criteria) no longer apply; #6873 was closed. The fixes are in the stack #6870 → #6871 → #6872 → #6874 → #6877 → #6879 → #6880 → #6881 → #6892.
Activity
VascoSch92 commentedon Nov 16, 2026
Update on this. The fixes are a stack of 9 PRs, all ready for review, and I checked them a few more ways.
On my Mac, main against the stack, with a mock LLM that answers instantly so only the server's own work is measured:
With a slow MCP server, the conversation appears without waiting for it, but the agent's first step does wait, so the first LLM call already has that server's tools; that's why its first event stays close to the server's own startup time. Worktree conversations still run
git fetch originevery time: we dropped #6873, which skipped it when origin had been fetched in the last five minutes.On main the server froze for more than 20 ms during every create; with the stack, none did.
The PRs, in stack order:
git initcheck off the event loop.I modelled the PRs that change behaviour, not just speed, in Lean, on
vasco/conversation-start-lean. That found a regression in #6871: it sized tmux batches in characters while tmux limits bytes, so non-ASCII environment values failed with "command too long" (fixed in b9e55b6). The other one it found was in #6873, which is no longer in the stack.Before asking for review, three independent reviewers went over every PR. They found three more bugs, all fixed with tests that fail without the fix:
exitwas never recovered.The benchmark in #6870 runs on every PR that touches the agent-server or the SDK and comments the comparison with the base.
After start, the other paths I measured hold up:
With several agents running, the UI gets occasional slow spikes. Each terminal command spawned 11 tmux processes, and on macOS every spawn briefly stalls the whole server; #6892 cuts the spikes by about 40%. With a mock LLM that answers in 2 s, a typical UI request stays around 4 ms.
Still open:
VascoSch92 commentedon Nov 16, 2026
All the fixes are merged: #6870, #6871, #6872, #6874, #6877, #6879, #6880, #6881 and #6892 (#6873 was dropped). The remaining ideas (the fixed 0.1 s sleep when the terminal clears its screen, the Changes tab's git cost, worktrees on very large repositories) can get their own issues.