Skip to content

Conversation start spends ~1 s per create on per-variable tmux calls, plus worktree git work on the event loop and MCP startup #6868

Closed

Description

@VascoSch92

Problem

Creating a conversation in Agent Canvas takes about 1.5 s before the agent even makes its first LLM call. I profiled it with a mock LLM that answers instantly, so everything below is pure agent-server overhead on the create path (POST /api/conversations blocks on agent init when the request carries an initial message, conversation_service.py#L1849). Canvas on the Local backend only navigates to the conversation after that request returns, so the user stares at the home page for the whole time.

There are three causes, in order of how many users they hit.

1. The terminal sets the tmux environment one variable at a time (everyone, every conversation). TmuxPanePool.initialize runs one tmux set-environment subprocess per variable (tmux_pane_pool.py#L139), and _new_owned_session runs one unset-environment per removed variable (tmux_terminal.py#L165-L166); TmuxTerminal has the same loop (tmux_terminal.py#L245). Each call costs about 10.5 ms, so startup grows linearly with the environment: 0.67 s at 55 variables, 2.8 s at 255. A real Canvas agent-server I checked has 103 variables, which is about 1.1 s per new conversation.

2. Worktree mode does git fetch and git worktree add on the event loop (users who pick "new worktree"). _plan_conversation_worktree and _create_conversation_worktree are called synchronously inside _create_conversation (L1676, L1717), and the default start point runs git fetch origin with a 60 s timeout on every create (L171, L197). On a clone of this repo that adds about 0.6 s of fetch and 0.7 s of checkout, and /alive stalls for a median 0.78 s meanwhile, so every other conversation freezes too.

3. MCP servers start inside the create, for every conversation (users with MCP servers). _runtime_mcp_tools starts all enabled servers in parallel and waits for the slowest (local_conversation.py#L1657), bounded only by the 30 s default timeout (mcp/utils.py#L397). One stdio server that starts instantly still adds 0.77 s; one that takes 3 s makes the create take 5.1 s.

Measured with an uninstrumented server and Canvas-shaped requests (warm medians, 95% CI in brackets):

scenario create time
home create, today 1.20 s [1.13, 1.28]
home create, with the tmux calls batched 0.33 s [0.33, 0.36]
new-worktree mode, repo with origin 2.56 s [2.49, 2.84]
one MCP server that starts instantly 2.05 s [2.02, 2.14]
released 1.62.0, +0 / +100 env vars 1.34 s / 2.68 s

End to end in the real Canvas 1.40 UI driven by Playwright, Enter → agent reply visible goes from 1.63 s (n=32) to 0.51 s (n=40) with only the tmux change; the two distributions don't overlap. The number of existing conversations made no difference.

Proposal

  1. Batch the tmux environment setup into a few ;-chained tmux invocations. A single call is not enough: tmux rejects the message with command too long at about 255 variables, so the commands need to be chunked (about 8 KB per call works). I have a 59-line patch that produces an identical session and global environment (checked with 300 values containing spaces, ;, quotes and $) and leaves the terminal test failures unchanged versus main.
  2. Move the worktree planning and creation into asyncio.to_thread. (Skipping the fetch on repeat creates was tried in perf(agent-server): skip the worktree fetch when origin was fetched recently #6873 and dropped: we keep fetching origin for every new worktree.)
  3. Stop blocking the create on MCP startup: start servers in the background and let the agent pick their tools up through the existing on_tools_changed path (agent/base.py#L902), waiting at most a short budget before the first LLM call so a first message that needs an MCP tool still sees it.
  4. Warm the public-skill cache and the LLM stack at startup; the first create after launch is about 1 s slower than the rest.
  5. Later, and together with Canvas: return from POST /api/conversations before agent init and report startup progress over the WebSocket, so the page can open immediately.

I'd do these as separate PRs, starting with 1.

Regression guard

I'd also like to keep this from coming back. Two layers:

  • Deterministic tests in the normal suite. Count tmux invocations during TmuxPanePool.initialize with 0 and 200 extra environment variables and assert the count does not grow with the environment. This needs no timing thresholds, so it can't flake. For the event loop, create a worktree conversation against a local file:// origin with asyncio debug mode on and assert that no callback blocks the loop for more than ~100 ms during the create. For MCP (after 3), assert that a create with a deliberately slow stdio server returns well before the server is ready.
  • A conversation-start benchmark script next to scripts/event_sourcing_benchmarks. It starts a real agent-server and a mock LLM, creates conversations the way Canvas does, and reports p50/p90 for create time, time to first agent event, and the worst /alive latency during the create. In CI it would run on PRs that touch the agent-server, the terminal tool or the conversation code, and nightly on main, comparing the PR head against its merge base on the same runner (interleaved, so runner noise cancels out) and failing only on a clear regression, e.g. a median at least 25% and 150 ms worse. This harness could also host the concurrent-conversation load test from Add cheap load tests for concurrent conversations #4477.

Acceptance criteria

  • Terminal startup makes a number of tmux invocations that does not grow with the environment size, and a test asserts that.
  • Worktree planning and creation no longer run on the event loop.
  • A create with a slow MCP server is not blocked for the server's full startup time.
  • A conversation-start benchmark runs in CI and fails on a clear create-latency regression.

Related: #4477, #6553.


OpenHands AI triage

The following comments and acceptance criteria were added by the OpenHands AI agent.

Triage

Verified against main (2aae48b): the per-variable tmux set-environment/unset-environment loops (tmux_pane_pool.py initialize(), tmux_terminal.py _new_owned_session() and TmuxTerminal), the synchronous _plan_conversation_worktree/_create_conversation_worktree calls in _create_conversation (also on the fork path) with git fetch origin (60 s timeout) on every plan, and the MCP startup join in _runtime_mcp_tools with a 30 s default timeout are all present, and /alive (server_details_router.py) shares the event loop. The measurements reproduce a real slowdown in the actual product workflow (uninstrumented agent-server with Canvas-shaped requests; Canvas UI via Playwright), which warrants priority:medium.

Scope for this issue is items 1–3 of the proposal plus the proposed regression guard, delivered as separate focused PRs:

  1. Batch tmux environment setup into ;-chained, size-chunked invocations that tolerate the ~255-variable tmux command too long limit.
  2. Move worktree planning/creation off the event loop (asyncio.to_thread) and suppress git fetch origin when FETCH_HEAD is recent, fetching only the default branch with a shorter timeout (cached-refs fallback already exists).
  3. Stop blocking create on MCP startup: start servers in the background, reuse the existing on_tools_changed path for late tools, and wait at most a bounded budget before the first LLM call so tools ready within it are present at the first call.
  4. Add the deterministic regression tests and the conversation-start benchmark harness next to scripts/event_sourcing_benchmarks.

Non-goals: proposal item 4 (skill-cache/LLM stack warming), item 5 (returning from POST /api/conversations before agent init with WebSocket startup progress — cross-repo work with Agent Canvas UI, owned by OpenHands/OpenHands), any REST/OpenAPI or client-side API changes, and changing MCP failure handling (servers that fail startup still log a warning and are skipped).

Acceptance Criteria

  • A test in the normal suite asserts that tmux invocation count during terminal/pool environment setup does not grow when the environment grows from 0 to 200+ extra variables, including values containing spaces, ;, quotes, and $.
  • The resulting tmux session and global environment after batched setup are identical to the current per-variable behavior (same keys and values visible to panes), verified by a test.
  • With asyncio debug mode enabled, creating a worktree conversation against a local file:// origin runs no event-loop callback that blocks for more than ~100 ms, asserted by a test.
  • Creating worktree conversations repeatedly within the FETCH_HEAD recency window runs git fetch at most once; a stale or missing FETCH_HEAD still triggers a fetch; and when fetch fails or times out, the create still succeeds from cached refs.
  • Forked conversations with worktree isolation get the same off-event-loop and fetch-suppression behavior as regular creates.
  • Creating a conversation with a deliberately slow-starting stdio MCP server returns before that server finishes starting, and the create succeeds.
  • MCP tools from a server that becomes ready within the bounded startup budget are available to the agent's first LLM call; tools from servers still starting after the budget become available during the conversation via the existing tools-changed path, without restarting the conversation.
  • The deterministic regression tests (tmux invocation count, event-loop blocking, MCP create-not-blocked) run as part of the normal test suite.
  • A conversation-start benchmark script exists next to scripts/event_sourcing_benchmarks, starts a real agent-server with a mock LLM, creates conversations the way Canvas does, and reports p50/p90 create time, time to first agent event, and worst /alive latency during create; it is wired to run on PRs touching the agent-server, terminal tool, or conversation code, comparing PR head against merge base interleaved on the same runner, and fails only when a reported metric regresses by ≥25% and ≥150 ms at the median.

Update (2026-11-16): we decided to keep git fetch origin on every new worktree conversation, so the fetch-suppression items above (proposal 2 and the FETCH_HEAD acceptance criteria) no longer apply; #6873 was closed. The fixes are in the stack #6870 → #6871 → #6872 → #6874 → #6877 → #6879 → #6880 → #6881 → #6892.

Activity

  1. added
    priority:mediumFor bugs, a serious source of annoyance, but not blocking a large number of users.
    ready-for-devIssue meets development readiness criteria
    on Nov 15, 2026
  2. VascoSch92 commented on Nov 16, 2026

    @VascoSch92
    MemberAuthor

    Update on this. The fixes are a stack of 9 PRs, all ready for review, and I checked them a few more ways.

    On my Mac, main against the stack, with a mock LLM that answers instantly so only the server's own work is measured:

    startup path create (main → stack) until the agent's first event
    normal session, server already running 1.18 → 0.29 s 1.25 → 0.33 s
    first conversation right after the server starts 1.41 → 0.64 s 1.79 → 0.82 s
    first conversation 3 s after the server starts 1.43 → 0.34 s 1.77 → 0.51 s
    new git worktree 2.04 → 1.17 s 2.12 → 1.21 s
    one MCP server that takes 3 s to start 5.10 → 1.35 s 5.18 → 4.17 s

    With a slow MCP server, the conversation appears without waiting for it, but the agent's first step does wait, so the first LLM call already has that server's tools; that's why its first event stays close to the server's own startup time. Worktree conversations still run git fetch origin every time: we dropped #6873, which skipped it when origin had been fetched in the last five minutes.

    On main the server froze for more than 20 ms during every create; with the stack, none did.

    The PRs, in stack order:

    • #6870: the benchmark.
    • #6871: tmux env setup in batches.
    • #6872: worktree git off the event loop.
    • #6874: warm-up at server start.
    • #6877: create the conversation after at most 1 s of MCP startup; the first step waits for the rest.
    • #6879: poll the lease lock every 2 ms.
    • #6880: git init check off the event loop.
    • #6881: no skill bodies in the agent state sent to sockets.
    • #6892: 5 tmux processes per terminal command instead of 11.

    I modelled the PRs that change behaviour, not just speed, in Lean, on vasco/conversation-start-lean. That found a regression in #6871: it sized tmux batches in characters while tmux limits bytes, so non-ASCII environment values failed with "command too long" (fixed in b9e55b6). The other one it found was in #6873, which is no longer in the stack.

    Before asking for review, three independent reviewers went over every PR. They found three more bugs, all fixed with tests that fail without the fix:

    • The benchmark compared against a stale base commit instead of the merge commit's parent, so newer main commits counted as the PR's change.
    • The tmux pool crashed on environment values that aren't valid UTF-8.
    • With two or more terminal panes, a shell that ran exit was never recovered.

    The benchmark in #6870 runs on every PR that touches the agent-server or the SDK and comments the comparison with the base.

    • It counts how many creates froze the server instead of comparing medians, since the median missed the intermittent lock stall that perf(agent-server): poll the conversation lease lock every 2 ms #6879 fixes.
    • It reports the time to the agent's first event under every startup path.
    • Same-code runs flag nothing, and a planted 50 ms delay or a 60 ms event-loop block is flagged.

    After start, the other paths I measured hold up:

    • 500 conversations add about 1.3 s to server startup.
    • Resume after a restart takes 0.3–0.5 s.
    • Paging a 20k-event history stays around 5 ms.

    With several agents running, the UI gets occasional slow spikes. Each terminal command spawned 11 tmux processes, and on macOS every spawn briefly stalls the whole server; #6892 cuts the spikes by about 40%. With a mock LLM that answers in 2 s, a typical UI request stays around 4 ms.

    Still open:

    • The fixed 0.1 s sleep when the terminal clears its screen.
    • The Changes tab endpoints, which spawn about 10 git processes per call.
    • Worktrees on very large repositories: 13 s for 50k files, which is git's own checkout.
  3. VascoSch92 commented on Nov 16, 2026

    @VascoSch92
    MemberAuthor

    All the fixes are merged: #6870, #6871, #6872, #6874, #6877, #6879, #6880, #6881 and #6892 (#6873 was dropped). The remaining ideas (the fixed 0.1 s sleep when the terminal clears its screen, the Changes tab's git cost, worktrees on very large repositories) can get their own issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestperformancePerformance issuepriority:mediumFor bugs, a serious source of annoyance, but not blocking a large number of users.ready-for-devIssue meets development readiness criteria

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions