Skip to content

Breaking: apply the MCP client's startup_timeout in Strands MCP activities - #188

Open
campoy wants to merge 4 commits into
mainfrom
fix/strands-python-mcp-startup-timeout-ignored
Open

campoy wants to merge 4 commits into
mainfrom
fix/strands-python-mcp-startup-timeout-ignored

Conversation

@campoy

@campoy campoy commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

What was changed

The Strands MCP connection now applies the MCPClient's startup_timeout to opening the transport and initialize(), as native MCPClient.start() does.

  • _ConnectionRecord._run, the task that owns a worker-side MCP connection, starts a timer for startup_timeout when it begins opening the transport. If initialize() has not finished when it fires, waiting callers get MCPClientInitializationError (cause TimeoutError) at once, the record leaves the connection cache, and the owner task is cancelled, which closes the transport.
  • Callers await the shared readiness future through asyncio.shield, so one cancelled Activity (for example at Worker shutdown) cannot cancel the handshake for the others or turn off the deadline.
  • A record leaves the cache only while it is still the cached record, and its idle eviction runs only then. So a late failure, or an idle timer armed by a caller cancelled during the handshake, cannot close the connection that replaced it.
  • Worker shutdown waits for every connection owner that has not finished, including timed-out ones that already left the cache and are still closing their transport. So worker.run() no longer returns before their MCP subprocesses and sessions are closed.
  • README: a paragraph on startup_timeout.

Breaking change: an MCP server whose initialization took longer than startup_timeout (30 seconds unless set) used to connect and now fails. {server}-list-tools fails with MCPClientInitializationError under its retry policy, and {server}-call-tool returns the error as a tool error, as it does for other connection failures. Such applications must raise startup_timeout on the MCPClient or make the server start faster. The title starts with Breaking: because release notes come from commit subjects.

Replay and rollback: only Activity and Worker shutdown code changes. Workflow code, its commands and the recorded results of successful listings are unchanged, so histories replay on either build and no version gate is needed. The rollback target is the code on main.

The branch keeps the first version and the review fixes as separate commits on 5e91c26; review the combined diff against main.

Why?

See #187. The plugin opened the transport and ran ClientSession.initialize() itself and never read startup_timeout, so a server slower than the deadline still connected, and a server that never answered initialize kept the Activity waiting until its own timeout.

Checklist

  1. Closes [Bug] strands_agents: the plugin ignores an MCPClient's startup_timeout #187

  2. How was this tested:

    • New tests in tests/test_mcp.py, with the tests' echo server delayed by ECHO_SERVER_START_DELAY:
      • test_mcp_startup_timeout: a 3 s server with startup_timeout=1 must fail the run with MCPClientInitializationError. On main it fails with AssertionError: the run completed.
      • test_mcp_startup_deadline_survives_cancelled_caller: cancelling the first caller during the handshake must not stop the deadline, and the cancelled caller's idle timer must not evict the replacement.
      • test_mcp_worker_shutdown_during_slow_startup[False]: worker.run() returns within 10 s when the Worker shuts down during a 15 s handshake with startup_timeout=2.
      • test_mcp_worker_shutdown_during_slow_startup[True]: when the Worker shuts down after the deadline, while the timed-out owner is still closing its stdio transport, worker.run() returns only after that owner has finished. With shutdown closing only cached connections, it fails at assert record._owner.done().
    • make lint and make test (56 passed) in python/strands_agents, with uv 0.12.24, on the latest branch head.
    • [Bug] strands_agents: the plugin ignores an MCPClient's startup_timeout #187's reproduction exits 1 on 0.1.0 and main and 0 with the change.
    • On 5e91c26, with a real Worker, a caller cancelled during a 15 s handshake (startup_timeout=1, idle timeout 5 s) no longer closes a replacement connection in the middle of a 4 s tool call; the Workflow gets the tool's result.
  3. Any docs updates needed?
    The plugin README gets a paragraph on startup_timeout.

Not covered:

  • Only the stdio transport was run; streamable HTTP and SSE go through the same owner task.
  • A {server}-call-tool Activity that opens the connection was not run with a delayed server. It waits on the same future as {server}-list-tools.
  • temporalio 1.34.x still bundles the old code as temporalio.contrib.strands. No backport here.

Open question:

Draft while the breaking change and the overlap with #137 are open.

…ities

The worker-side MCP connection opened the client's transport and ran
ClientSession.initialize() without reading the MCPClient's startup_timeout,
so a server slower than the configured deadline still connected, and one
that never answered initialize kept the activity waiting until its own
timeout. Native MCPClient.start() cancels initialization after
startup_timeout (30 seconds unless set).

The connection owner now starts a timer for startup_timeout when it begins
opening the transport. If initialize() has not finished when it fires,
waiting callers get MCPClientInitializationError (cause TimeoutError) at
once, the record leaves the cache, and the owner task is cancelled, which
closes the transport. A record now leaves the cache only if it is still
the cached one, so a late failure cannot drop its replacement.

Breaking change: an MCP server whose initialization took longer than
startup_timeout used to connect and now fails: {server}-list-tools fails
with MCPClientInitializationError under its retry policy, and
{server}-call-tool returns the error as a tool error. Raise
startup_timeout on the MCPClient for slow servers. The change runs only in
activities; workflow histories do not change.
Callers now await the shared readiness future through asyncio.shield, so
a cancelled activity, for example at worker shutdown, no longer cancels
the handshake for the other callers or disables the startup deadline.
The future's failure is marked retrieved so asyncio does not log it when
every caller is gone. An idle timer armed by a record that has since left
the cache no longer evicts the connection that replaced it.

Tests cover a caller cancelled during a slow handshake: the deadline
still fails the other callers, and the replacement connection survives
the old record's idle timer.
@campoy
campoy force-pushed the fix/strands-python-mcp-startup-timeout-ignored branch from 2945041 to 8032e4c Compare October 30, 2026 10:17
In the after_deadline case of test_mcp_worker_shutdown_during_slow_startup,
the startup deadline fails the list-tools activity, and on a slow runner the
workflow can finish before the test terminates it. terminate() then raises
"workflow execution already completed". Ignore only that error.
@campoy
campoy marked this pull request as ready for review November 1, 2026 03:15
@campoy
campoy requested a review from a team as a code owner November 1, 2026 03:15

@DABH DABH left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First review, at head 09dafb3 (four commits on 1e8ae93; a trial merge with current main is clean). On a worktree at the head, make lint is clean and make test passes 174 of 174 (Python 3.14.3, macOS). CI is green on all four cells. The four new test cases fail against main's implementation (CancelledError, the run completed, 14 s against the 10 s bound, and the 60 s pytest timeout) and pass on the head. #187's reproduction exits 1 on main and 0 on the head. Replay: a history recorded on main (echo-list-tools, echo-call-tool) replays on the head. That matches the body: only Activity and Worker shutdown code changed, so no version gate is needed. Each finding below was reproduced with probe scripts against the installed packages (strands-agents 1.56.0, mcp 1.30.0, temporalio 1.34.0). Where a fix is suggested, I applied it to a scratch copy of the head and ran tests/test_mcp.py and tests/test_mcp_connection_eviction.py against it. Checked and dropped: MCPClientInitializationError is not in the failure converter's non-retryable list, so 'its retry policy applies' holds. A {server}-call-tool Activity against a slow server returns Tool execution failed: MCP server initialization did not finish within startup_timeout (1 seconds) after 1.0 s, as the README says. A connect error that is reported as the timeout, because the transport's cleanup outlasts the deadline, matches native MCPClient, which also sets its init error only after that cleanup. The part that differs from native is the first comment. There is no TypeScript parity gap: the TypeScript McpClient (1.19.0, the locked version) has no startup timeout, and that plugin connects through the native connect(). Not run: streamable HTTP and SSE (as in the body); the timing probes ran on macOS only.

Severity: P1 = wrong result or stuck run in realistic use; P2 = correctness or robustness in an edge case; P3 = docs, tests, cleanup.

Where the problems cluster: the deadline does what #187 asks. Waiting callers fail on time. A cancelled caller no longer cancels the shared handshake, which main's await self._ready did. And worker.run() no longer waits indefinitely for a server that never answers. What is left sits at the edges of the new timer. It is disarmed only after the transport has finished closing, so it can cancel an owner that is already shutting down. That skips stdio_client's terminate step, and in this PR's own shutdown-during-startup scenario it leaves the server process running. Two statements before the try can still leave _ready pending forever. Shutdown waits out the full startup_timeout for a handshake that no caller needs. And on Python 3.14 the shield logs the very failure the new done callback was meant to keep quiet.

Decision: comment rather than approve, because the first item is a narrow regression against main with a small fix, and the order relative to #190, which removes startup_timeout support again, is still open.

Related PRs. #190 supersedes this change rather than building on it. It deletes _ConnectionRecord, get_connection and the MCPClient factories this PR changes, and moves the activities to temporalio-mcp. Its body already lists startup_timeout among the MCPClient options that no longer apply, and a trial merge of the two heads conflicts in all five files. If #190 is meant for the same release, merging this first adds a Breaking entry to the release notes that #190 then takes back, and this PR's tests and README paragraph get deleted. If #190 is further out, this PR stands on its own, and #190's rebase removes this code. #184 is independent: it changes the Workflow-side TemporalMCPClient and TemporalAgent, not the connection owner. A trial merge with it conflicts only textually, in README.md (each PR adds a paragraph after 'Each factory returns...') and in tests/test_mcp.py (imports and appended tests). This PR depends on neither. #137, which the body asks about, merged on 2026-10-26 and is already in this PR's base.

PR body (P3): four statements are out of date. The open question about #137 and 'Draft while the breaking change and the overlap with #137 are open' (#137 has merged, and the PR is no longer a draft). 'Separate commits on 5e91c26' (the commits now sit on 1e8ae93). And 'make test (56 passed)' (the head runs 174 tests). The breaking-change and replay paragraphs are accurate. The #190 relationship above belongs in the body too.

self._ready.set_result((client, session))
await self._stop.wait()
except BaseException as err:
deadline.cancel()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The deadline is disarmed only here, after both context managers have finished closing; if it fires while an owner that is already exiting is still closing its transport, _on_startup_timeout cancels the owner a second time and stdio_client skips terminating the server. stdio_client's shutdown closes stdin and waits up to 2 s for the process in anyio.fail_after(PROCESS_TERMINATION_TIMEOUT). It calls _terminate_process_tree only on TimeoutError (mcp/client/stdio/__init__.py:203-209 in mcp 1.30.0). A CancelledError delivered during that wait skips the terminate step, and anyio's Process.aclose() then waits for the process to exit on its own.

Reproduced in this PR's own scenario: a server that answers after 15 s, startup_timeout=3, and the Worker leaving async with Worker(...) 1.5 s into the handshake. Worker.__aexit__ cancels the run task after shutdown(), and the cancellation reaches the owner through aclose()'s await self._owner. On this head the owner finishes 17.6 s after it started, when the server finally reads EOF. On main it finishes after 3.6 s, because the server is terminated after the 2 s wait.

The same mechanism applies to a connect error that arrives within 2 s of the deadline. Probe: a server that rejects initialize after 0.3 s and then ignores EOF for 120 s, with startup_timeout=1. 15 s after the call started, the owner is still running. close_connections, which Worker shutdown now awaits through _OPEN_RECORDS, is still waiting 10 s after that, and the server process is alive. A server that never exits would keep both waiting for good.

Fix: disarm the deadline as soon as the handshake ends either way: try: await session.initialize() / finally: deadline.cancel(). You could also fail _ready there, so callers get the server's error without waiting for the close. With that change, the async with case finishes in 3.7 s. The rejecting server is terminated and its error reaches the caller after 2.8 s, close_connections returns at once, and the 15 MCP tests pass. A variant of test_mcp_worker_shutdown_during_slow_startup that leaves async with Worker shortly before the deadline would pin it.

client = client_factory()
# Apply the client's startup_timeout to opening the transport and
# initialize(), as MCPClient.start() does natively.
deadline = asyncio.get_running_loop().call_later(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Anything that raises before the try leaves _ready pending and the record cached, so every later call for that server waits until its Activity times out; this PR adds a second such statement. MCPClient(..., startup_timeout=None) is outside the int annotation, but it works natively (Future.result(timeout=None) waits without limit) and on main. Here call_later(None, ...) raises TypeError in the owner (delay must not be None on 3.14).

Probe on this head: the first list-tools call is still waiting after 8 s. The record is still cached with _ready pending, and the owner finished with that TypeError. A second call also waits. On main the same factory lists ['echo'] in 0.4 s.

Line 244 (client = client_factory()) has the same effect and predates this PR. A factory that raises hangs every call on main and on the head. Example: lambda: MCPClient(url=os.environ.get('MCP_URL')) with the variable unset raises ValueError: MCPClient: either 'transport_callable' or 'url' must be provided.

Fix: start _run with deadline = None, move client = client_factory() and the call_later inside the try, and guard the cancel() calls. Then any such failure settles _ready and uncaches the record. Optionally, skip the timer for None, as native does. Non-blocking.

"""Return the live client and session, or raise the connect failure."""
return await self._ready
# Callers share _ready; one caller's cancellation must not cancel it.
return await asyncio.shield(self._ready)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: On Python 3.14, asyncio.shield reports a failure of the shared future to the loop's exception handler once any caller has been cancelled, so the done callback at lines 232-234 does not keep asyncio quiet there. When the outer future is cancelled first, 3.14's shield adds _log_on_exception to the inner future. That hook calls loop.call_exception_handler with ' exception in shielded future'; 3.10 to 3.13 have no such hook.

Probe: a bare future carrying the same done callback, the waiter cancelled, then set_exception. 3.10.20 and 3.13.12 report nothing; 3.14.3 reports RuntimeError exception in shielded future. Through the plugin on 3.14.3, a caller cancelled during a handshake that then hit the deadline produced MCPClientInitializationError exception in shielded future, with a traceback, at ERROR. So on CI's newest Python, the Worker-shutdown-during-startup case this PR handles logs an error.

Fix: await asyncio.wait([self._ready]), then return self._ready.result(). A cancelled caller still leaves _ready alone, and none of the three versions reports anything (same probe). Non-blocking.

"""Close the cached connection and wait for every retiring owner to finish."""
await _evict_connection(server)
for record in list(_OPEN_RECORDS.get(server, ())):
await record.aclose()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: Worker shutdown still waits out the full startup_timeout (30 s by default) for a handshake that no Activity is waiting for. aclose() only sets _stop, and the owner reads it only after initialize(). Probe: the only caller cancelled at once, a server that answers after 60 s, startup_timeout=20. close_connections returns after 21.5 s; main waits until the server answers.

Once the deadline is disarmed when the handshake ends (comment on line 262), aclose() can cancel an owner whose _ready is not done: if not self._ready.done(): self._owner.cancel() before awaiting it. The same probe then returns after 2.0 s, and the 15 MCP tests pass. Without the line-262 change it backfires: test_mcp_worker_shutdown_during_slow_startup[False] takes 12 s, because the deadline cancels the owner again while stdio_client is closing. Non-blocking.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] strands_agents: the plugin ignores an MCPClient's startup_timeout

2 participants