Skip to content

feat(engine,plugin,config): MCP engine integration — servers, namespaced tools, client/mcp.call#51

Merged
andybons merged 10 commits into
mainfrom
feat/mcp-engine
Jul 9, 2026
Merged

feat(engine,plugin,config): MCP engine integration — servers, namespaced tools, client/mcp.call#51
andybons merged 10 commits into
mainfrom
feat/mcp-engine

Conversation

@andybons

@andybons andybons commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Implements the last platform blocker for the meeseeks migration: MCP servers as first-class session tools.

  • config: new mcp_servers section declaring named servers — stdio (command/env/dir) and streamable-HTTP (url/headers) — validated at load like plugins
  • engine: sessions connect to each configured server via the mcp/ client, list tools, and register them as engine tools named mcp__<server>__<tool> (Claude Code convention), routing execution through CallTool; a server that fails to connect or list is logged and skipped — never kills the session (fail-open, matching plugin crash philosophy)
  • plugin: client/mcp.call now reaches the same connected clients, replacing ErrMCPNotImplemented (including the late-binding ClientAPI in cmd/harness)
  • lifecycle: MCP connections close with bounded timeouts on session/server shutdown
  • tests: red-first coverage of config validation, namespacing, registration against a stub MCP server, is_error routing, per-server fail-open, and the plugin MCPCall path

🤖 Generated with Claude Code

https://claude.ai/code/session_013B9pwUYhT4o8se9skmS5PP

Harness Agent added 5 commits July 9, 2026 19:01
…tools

MCPManager (engine/mcp.go) is the production MCPRegistry: given named
MCP server configs (stdio or streamable-HTTP), it lazily connects to
each on first use, lists its tools, and namespaces them
mcp__<server>__<tool> (the Claude Code convention). A server that fails
to connect or list tools is logged and skipped rather than failing the
caller — fail-open, matching the plugin crash philosophy. Tool calls
route through the same connected mcp.Client via CallTool; is_error is
threaded through verbatim.

Session.Config gains an MCP field (nil disables MCP entirely, matching
the Hooks pattern) and Session.MCPCall exposes explicit server+tool
routing for the plugin ClientAPI path landing next.
Replace plugin.ErrMCPNotImplemented in both bundled ClientAPI
implementations now that MCP engine integration exists:

- cmd/harness's runClientAPI.MCPCall routes through the run-mode
  session's engine.Session.MCPCall (late-bound, matching how
  SessionMessages already resolves the session lazily).
- server's clientAPI.MCPCall routes through a new Options.MCP field:
  the same *engine.MCPManager shared with every session's
  engine.Config.MCP, called directly by server+tool name since
  MCPCallRequest names neither a session nor a namespaced tool.

Update PROTOCOL.md's client/mcp.call section accordingly; client/
generate remains not-implemented (separate PR, provider-layer
routing).
buildMCPManager converts config.MCPServerSpec into engine.MCPServerConfig
and builds one shared *engine.MCPManager per process (nil when no servers
are configured, mirroring buildPluginHost). Both harness run and harness
serve build it once, thread it into every session's engine.Config.MCP
(and server.Options.MCP, for the plugin ClientAPI's direct server+tool
routing), and close it on shutdown with a bounded timeout — deferred
before the plugin host's own Close so MCP connections outlive it,
symmetric with how the host outlives the server's drain.
@claude

claude Bot commented Jul 9, 2026

Copy link
Copy Markdown
📦 Previous Review (superseded by newer review)

Claude finished @andybons's task in 3m 49s —— View job


Review: MCP engine integration

Reviewed against origin/main. Overall this is clean, well-documented, and closely mirrors the existing plugin-host patterns (process-wide singleton, typed-nil guard, fail-open, lazy connect). Config validation, namespacing, is_error routing, and the plugin client/mcp.call path all look correct. One robustness bug worth fixing before merge, plus a couple of minor notes.

  • Read AGENTS.md and gather context
  • Review config (mcp_servers validation)
  • Review engine MCP connect/register/lifecycle
  • Review plugin client/mcp.call wiring
  • Review cmd/harness + server wiring
  • Post inline comments + summary

🔴 Connect is bound to the first caller's request context (serve mode)

Inline comment → engine/mcp.go:159

ensureConnected fires once via sync.Once and caches success or failure forever, but threads the caller's ctx into connectMCPServer. In serve that ctx is a per-prompt request context. If the first prompt to touch MCP is aborted (client disconnect / POST /abort / drain) during the connect window, every server fails to initialize, is logged-and-skipped, and — because connectOnce already fired — is never retried. A single transient cancellation permanently strips MCP tools from the whole server process. Since the connections are explicitly process-lived and shared (per the package doc), the connect shouldn't be owned by any one request: detach with context.WithoutCancel(ctx) / context.Background() (the per-server ConnectTimeout already bounds it). Fix + a red-first cancellation test in the inline comment.

🟡 Minor notes (non-blocking)

  • Close vs. first-connect race (engine/mcp.go:294)Close reads m.clients under RLock but doesn't call ensureConnected. If Close runs concurrently with an in-flight first Tools() connect, it sees m.clients == nil, returns early, and the clients created moments later are never closed (leaked subprocess/HTTP session). This only bites on a shutdown-racing-startup edge where the process is exiting anyway, so it's low-severity — but a Close that fires after connectOnce.Do returns is the invariant readers will assume.

  • decodeMCPBase64 silently drops malformed data (engine/mcp.go:364) — a bad base64 image/audio/blob payload yields nil Data with no signal to the model or logs, so a corrupt image result looks like an empty one. Consider logging at minimum.

  • Server-name namespacing collision (config/config.go:340)mcp__<server>__<tool> isn't uniquely decodable if a server name contains __ (server a__b/tool c and server a/tool b__c both map to mcp__a__b__c). Routing is a direct map lookup so it's benign today, but a light validateMCPServers guard rejecting __ (or the mcp__ prefix) in server names would prevent a surprising silent shadow.

What's solid

  • Concurrent per-server connect with a local mutex, then a single guarded publish of clients/tools/toolsOrdered — no data race, deterministic sorted registration order.
  • Fail-open on connect/list matches the crashed-plugin philosophy; MaxTokens-style bounded timeouts on both connect and Close.
  • Typed-nil guard (mcpRegistry) correctly avoids the non-nil-interface footgun for the no-servers case, consistent with pluginHooks.
  • Dispatch precedence in executeTool (built-in → MCP-namespaced → plugin) is unambiguous since MCP names are prefix-scoped.
  • is_error (tool-level) vs. err (protocol/connectivity) separation is preserved end-to-end through CallTool/CallServerTool and into MCPCallResult.
  • Test coverage is genuinely red-first and thorough: namespacing, stub-server registration, is_error routing, per-server and session-level fail-open, and the plugin MCPCall path.

Comment thread engine/mcp.go Outdated
Harness Agent added 4 commits July 9, 2026 19:39
ensureConnected is a sync.Once-cached, run-exactly-once-forever
operation shared by every future Session, but it was threading the
very first caller's ctx straight into connectMCPServer. In serve mode
that ctx is per-request: a request that happens to trigger the first
connect being aborted/disconnected mid-connect canceled every
server's connect, which was logged-and-skipped and, being cached,
never retried — one transient cancellation permanently stripped MCP
tools from the whole process.

Detach with context.WithoutCancel before connecting; each server's
own ConnectTimeout still bounds the connect step. Regression test
drives the first caller in with an already-canceled context and
asserts the server still connects (red before the fix: Tools()
returned no entries).
decodeMCPBase64 silently returned nil Data on a base64 decode failure,
losing the tool result's payload with no signal anywhere. Log a slog
warning naming the server and tool the payload came from so the failure
is at least visible — but never the payload bytes themselves, which may
be arbitrarily large and are not diagnostic.

Adds TestDecodeMCPBase64MalformedLogsWarning.
Close read m.clients directly, with no interaction with connectOnce.
connectMCPServer populates m.clients only at the very end of the
one-time connect step, so a Close racing a caller's still-in-flight
first Tools()/CallTool() call would observe the zero-value nil map,
close nothing, and return; the racing connect would then finish moments
later and populate m.clients with a client (or, for a stdio server, a
spawned child process) nobody would ever close again, since connectOnce
never retries or revisits it once run — a silent leak.

Have Close call ensureConnected itself before reading m.clients:
sync.Once serializes concurrent Do calls, so this blocks Close until
any connect already in flight elsewhere has finished and its clients
are visible, closing them as it always did. If no connect was ever
triggered, Close now performs it itself (bounded by each server's own
ConnectTimeout) purely so it can close what it creates.

Adds TestMCPManagerCloseWaitsForInFlightConnect: a gated stub server
that holds the initialize response open until the test has raced Close
in, then asserts (via the DELETE the client's Close issues) that the
client the racing connect created was actually closed. Red before the
fix (Close saw no clients yet).
… decoding

engine/mcp.go namespaces every MCP tool as mcp__<server>__<tool>. That
encoding is only uniquely decodable back into (server, tool) if "__"
cannot occur inside server: server "a__b" tool "c" and server "a"
tool "b__c" both produce the identical namespaced name
mcp__a__b__c, so two unrelated servers' tools would silently collide
and become indistinguishable at call time.

Reject at load, loudly: a server name containing "__" anywhere, or
starting with the "mcp__" namespace prefix itself.

Adds table-test rows to TestLoadMCPServers covering a middle/leading/
trailing "__", a bare "mcp__" prefix, and exactly "mcp__", plus a
control case confirming a single underscore is still accepted. Red
before the fix (all five malformed-name rows loaded without error).
@claude

claude Bot commented Jul 9, 2026

Copy link
Copy Markdown
📦 Previous Review (superseded by newer review)

Claude finished @andybons's task in 3m 33s —— View job


Review: MCP engine integration (re-review after fixes)

  • Read AGENTS.md and gather context
  • Verify the 4 findings from the prior round are addressed
  • Re-review config validation, engine connect/register/lifecycle, plugin wiring
  • Build + vet + targeted -race tests

Reviewed the current branch (aeadd5f) against origin/main. go build ./..., go vet, and the MCP-focused -race suites in engine/, config/, and cmd/harness/ all pass. Every finding from the previous round is now fixed, correctly and with red-first coverage — I found no new blocking issues.

Prior findings — all resolved

  • 🔴 Connect bound to the first caller's ctx → fixed in engine/mcp.go:156. ensureConnected now detaches with context.WithoutCancel(ctx), bounded only by each server's ConnectTimeout. TestMCPManagerConnectSurvivesFirstCallerCancellation (cancels before the first Tools() call, asserts a later caller still sees the tool) locks it in.
  • 🟡 Close vs. first-connect race → fixed in engine/mcp.go:332. Close now calls ensureConnected(ctx) before reading m.clients, so sync.Once serialization guarantees an in-flight connect's clients are visible. TestMCPManagerCloseWaitsForInFlightConnect gates initialize mid-flight over a channel and asserts the racing client is actually DELETEd (not leaked) — a proper channel-synchronized race, no sleeps.
  • 🟡 decodeMCPBase64 silent drop → fixed in engine/mcp.go:413. Malformed payloads now emit an slog.Warn naming server+tool. Covered by TestDecodeMCPBase64MalformedLogsWarning.
  • 🟡 Server-name __ collision → fixed in config/config.go:353. validateMCPServers rejects __ in names (and the mcp__ prefix), so mcp__<server>__<tool> stays uniquely decodable.

What I re-verified as solid

  • Dispatch precedence in executeTool (built-in → isMCPToolName → plugin) is unambiguous; a hallucinated mcp__… name resolves to a tool-level error rather than falling through to plugins.
  • Merge semantics (config.go:561) — MCP servers merge key-by-key with wholesale same-name replacement, deep-copied via copyMCPServerSpec so a merged config never aliases either input layer.
  • Lifecycle ordering — MCP defer closeMCPManager is declared before the plugin host's in both runCmd/serveCmd, so LIFO unwind closes MCP after the host (a plugin's client/mcp.call has nowhere to route once the host is gone).
  • Typed-nil guard (mcpRegistry) correctly avoids the non-nil-interface footgun, matching pluginHooks.
  • Plugin client/mcp.call routes through CallServerTool end-to-end in both server- and run-mode adapters, preserving the is_error (tool-level) vs. err (protocol) split into MCPCallResult. ErrMCPNotImplemented is fully removed.

One non-blocking nit (optional)

  • config/config.go:361strings.HasPrefix(name, "mcp__") is dead relative to the strings.Contains(name, "__") check two lines up (any mcp__… name already contains __). It's harmless and the comment explains it as intentional belt-and-suspenders, so leave it if the explicit error message is the point — just noting it's unreachable.

Verdict: ready to merge from my side. Clean, well-documented, faithful to the existing plugin-host patterns, and the concurrency edges are now closed with tests that would actually catch regressions.

@claude

claude Bot commented Jul 9, 2026

Copy link
Copy Markdown
📦 Previous Review (superseded by newer review)

Claude finished @andybons's task in 3m 27s —— View job


Review: MCP engine integration (re-review after fixes)

Reviewed origin/main..HEAD (MCP-related files only — the server/server.go wait-endpoint deletions in the raw diff are a branch-divergence artifact from #50 landing on main after this branch forked, not part of this PR; the PR itself only adds the Options.MCP field). Built clean, go vet clean, and go test -race ./engine/ ./config/ ./message/ all green.

All five findings from the prior round are resolved and each landed with a red-first regression test:

  • Connect detached from caller ctxensureConnected now wraps with context.WithoutCancel(ctx) (engine/mcp.go:156), covered by TestMCPManagerConnectSurvivesFirstCallerCancellation (cancels before the first Tools() call, asserts a later call still connects). ✔
  • Close vs. first-connect raceClose now calls ensureConnected first so it can never observe the zero-value nil clients map mid-connect (engine/mcp.go:331), covered by TestMCPManagerCloseWaitsForInFlightConnect. ✔
  • Malformed base64 no longer silentdecodeMCPBase64 logs a slog.Warn naming server+tool (engine/mcp.go:413). ✔
  • Server-name namespacing collisionvalidateMCPServers rejects __ in names and the mcp__ prefix (config/config.go:340), so mcp__<server>__<tool> stays uniquely decodable. ✔
  • Bonus: truncated ToolCall.Arguments poisonMessage.Normalize now drops non-empty-but-invalid JSON arguments at the ingest choke point (message/message.go:110), with a thorough incident-reproducing test (engine/tool_call_poison_test.go). ✔

What's solid

  • Concurrency: per-server connect under a local mutex, then a single guarded publish of clients/tools/toolsOrdered; sync.Once correctly serializes Close behind an in-flight connect. -race clean.
  • Fail-open on connect/list matches the crashed-plugin philosophy; bounded timeouts on connect (ConnectTimeout) and Close (mcpCloseTimeout).
  • Typed-nil guard (mcpRegistry) consistent with pluginHooks, avoiding the non-nil-interface footgun for the no-servers case.
  • is_error (tool-level) vs. err (protocol/connectivity) separation preserved end-to-end through CallTool/CallServerTool into MCPCallResult, for both the namespaced engine path and the plugin client/mcp.call path.
  • Config merge deep-copies slice/map fields (copyMCPServerSpec) so the merged config never aliases an input layer.

Minor note (non-blocking)

  • Close on an idle serve process spawns subprocesses just to close them. CloseensureConnected uses context.WithoutCancel, so if a harness serve process is configured with MCP servers but receives no prompt before SIGTERM, shutdown will connect every configured server (spawning stdio children / opening HTTP sessions) and immediately close them — and that connect step is bounded by ConnectTimeout (15s default), not mcpCloseTimeout (10s), so an unreachable server can make idle shutdown block up to 15s. The interlock is the right fix for the leak and the tradeoff is documented in the Close doc comment, so this is acceptable as-is; flagging only in case bounding the whole Close (including the forced connect) under mcpCloseTimeout is desired. Every non-idle path already connects on the first prompt, so this only bites the never-prompted case.

Nothing blocking — clean, well-documented, closely mirrors the existing plugin-host patterns. Nice work. ✅

@claude

claude Bot commented Jul 9, 2026

Copy link
Copy Markdown
📦 Previous Review (superseded by newer review)

Claude finished @andybons's task in 3m 7s —— View job


Review: MCP engine integration

Reviewed the current branch (aeadd5f) against origin/main. go build ./..., go vet, and go test -race ./engine/ ./config/ ./cmd/harness/ ./server/ all pass. The four prior review rounds have already closed the substantive issues (first-caller ctx binding, Close vs. connect race, silent base64 drop, __ name collision), each with a red-first regression test. I found no new blocking issues — this is ready to merge from my side.

  • Read AGENTS.md and gather context
  • Review config (mcp_servers validation + merge)
  • Review engine MCP connect/register/lifecycle
  • Review plugin client/mcp.call + cmd/harness wiring
  • Build + vet + targeted -race tests
  • Post findings

What I re-verified as correct

  • Connect detached from caller ctxensureConnected wraps with context.WithoutCancel(ctx) under sync.Once (engine/mcp.go:154), so an aborted first prompt in serve mode can't permanently cache a failed connect. Bounded per-server by ConnectTimeout.
  • Close interlockClose calls ensureConnected before reading m.clients (engine/mcp.go:332), so a shutdown racing an in-flight first connect can't observe the zero-value nil map and leak a just-spawned client. No lock is held across the connect goroutines, so no deadlock with the interlock.
  • Dispatch precedence in executeTool (built-in → isMCPToolName → plugin, engine/engine.go:633) is unambiguous because MCP names are prefix-scoped; a hallucinated mcp__… name resolves to a tool-level error rather than falling through to plugins.
  • is_error (tool-level) vs. err (protocol/connectivity) preserved end-to-end through CallTool/CallServerTool into MCPCallResult, for both the namespaced engine path and the plugin client/mcp.call path (server- and run-mode adapters). res from mcp.Client.CallTool is non-nil whenever err == nil, so the res.Content/res.IsError reads are safe.
  • Config validation & mergevalidateMCPServers rejects empty names, __-containing names, the mcp__ prefix, and neither/both of command/url (config/config.go:340). merge deep-copies via copyMCPServerSpec so a merged config never aliases an input layer.
  • Typed-nil guard (mcpRegistry, cmd/harness/mcp.go) correctly avoids the non-nil-interface footgun, matching pluginHooks.
  • Lifecycle orderingdefer closeMCPManager is declared before the plugin host's in both runCmd/serveCmd, so LIFO unwind closes MCP after the host (a plugin's client/mcp.call has nowhere to route once the host is gone). Startup stays lazy — NewMCPManager touches no network; connect is deferred to first Tools()/CallTool().
  • ErrMCPNotImplemented fully removed; errors import in plugin/host.go still used by ErrGenerateNotImplemented.

One non-blocking note (already flagged, accepted)

  • Idle serve shutdown connects-then-closes (engine/mcp.go:331) — because CloseensureConnected uses context.WithoutCancel, a harness serve process configured with MCP servers but that receives no prompt before SIGTERM will connect every server just to close it, and that forced connect is bounded by ConnectTimeout (15s), not mcpCloseTimeout (10s). This is the documented tradeoff of the leak-fix interlock and only affects the never-prompted case, so it's fine as-is; flagging only in case bounding the whole Close under mcpCloseTimeout is preferred.

Verdict: clean, well-documented, faithful to the existing plugin-host patterns, concurrency edges closed with tests that would catch regressions. ✅

Close's leak-fix interlock (4075ef4) called ensureConnected(ctx)
unconditionally to synchronize with an in-flight first connect before
reading m.clients. That correctly waits out a connect already underway,
but if nothing had ever triggered one, Close itself became the first
caller: a harness serve process configured with MCP servers that never
received a single prompt before shutdown would still live-dial every
configured server just to immediately close it, up to ConnectTimeout
(default 15s) of pure shutdown latency, bounded by the wrong timeout
entirely (each server's own ConnectTimeout, not mcpCloseTimeout, since
Close was paying for the connect rather than merely waiting on one).

Close now calls connectOnce.Do with a no-op instead of the real connect
closure. sync.Once still serializes every Do call the same way
regardless of which function is passed: if a real connect is already in
flight (or finished) elsewhere, Close's Do call blocks until it
completes and m.clients is fully populated, preserving the original
interlock exactly. If nothing had started, Close's no-op wins instead —
near-instant, no network touched — and because connectOnce is now
permanently consumed, no Tools()/CallTool()/CallServerTool call arriving
after such a Close, even later, can start a connect either.

Adds TestMCPManagerCloseOfNeverPromptedManagerNeverConnects: a gated
stub server (never released until Cleanup) proving Close of an idle
manager makes zero connection attempts and returns near-instantly, plus
that a Tools() call after that Close still makes none. Red before the
fix (one request reached the server). Keeps
TestMCPManagerCloseWaitsForInFlightConnect passing unchanged.
@claude

claude Bot commented Jul 9, 2026

Copy link
Copy Markdown

Claude finished @andybons's task in 3m 46s —— View job


Review: MCP engine integration

Reviewed the branch (9b19ef8) against origin/main, re-reading the full MCP surface independently. go build ./..., go vet, and go test -race ./engine/ ./config/ ./cmd/harness/ ./server/ all pass. The prior five rounds closed every substantive issue; my fresh pass found no new blocking issues and confirms the resolutions hold.

  • Read AGENTS.md and gather context
  • Review config (mcp_servers validation + merge)
  • Review engine MCP connect/register/lifecycle
  • Review plugin client/mcp.call + cmd/harness wiring
  • Build + vet + targeted -race tests
  • Post findings

What I re-verified as correct

  • Connect detached from caller ctxensureConnected wraps context.WithoutCancel(ctx) under sync.Once (engine/mcp.go:156); an aborted first prompt in serve mode can't permanently cache a failed connect. Publish of clients/tools/toolsOrdered is under m.mu.Lock() after wg.Wait(), and the sync.Once happens-before ordering makes it visible to every RLock reader — no data race.
  • Close interlock without initiating a connect — the final fix (engine/mcp.go:351) calls connectOnce.Do(func(){}) with a no-op: it serializes behind an in-flight real connect (closing the just-spawned clients, no leak) but never becomes the first connect, so an idle serve process that never received a prompt shuts down instantly instead of dialing every server just to close it. The doc comment walks through both race arms precisely. Bounded by mcpCloseTimeout.
  • Dispatch precedence (engine/engine.go:641) — built-in → isMCPToolName → plugin is unambiguous because MCP names are prefix-scoped; a hallucinated mcp__… resolves to a tool-level error, not a fall-through.
  • is_error (tool-level) vs err (protocol) preserved end-to-end through CallTool/CallServerTool into MCPCallResult, for both the namespaced engine path and the plugin client/mcp.call path (mcp.Client.CallTool returns a non-nil *CallToolResult whenever err == nil, so the res.Content/res.IsError reads are safe).
  • Config validation & mergevalidateMCPServers rejects empty names, __-containing names, the mcp__ prefix, and neither/both of command/url (config/config.go:353); merge deep-copies via copyMCPServerSpec so a merged config never aliases an input layer (test asserts non-aliasing).
  • Typed-nil guard (mcpRegistry, cmd/harness/mcp.go:51) mirrors pluginHooks, avoiding the non-nil-interface footgun; the same instance is wired into both per-session Config.MCP and server.Options.MCP, so plugin-initiated and namespaced calls hit identical connections.
  • Lifecycle orderingdefer closeMCPManager is declared before the plugin host's in both runCmd/serveCmd, so LIFO unwind closes MCP after the host (a plugin's client/mcp.call has nowhere to route once the host is gone). Startup stays lazy — NewMCPManager touches no network, so the AGENTS.md startup-speed rule is respected.
  • Malformed base64 now logs slog.Warn naming server+tool (engine/mcp.go:432) instead of silently dropping.
  • ErrMCPNotImplemented fully removed; run-mode MCPCall late-binds through getSess().

Non-blocking observations (optional, no action needed)

  • callTool (engine/mcp.go:288) unmarshals args into any then hands it to client.CallTool, which re-marshals it — a validating round-trip. Passing json.RawMessage through directly would skip one marshal, but the unmarshal doubles as an early "invalid arguments" guard, so this is a reasonable tradeoff, not a defect.
  • config/config.go:361strings.HasPrefix(name, "mcp__") is unreachable relative to the strings.Contains(name, "__") check two lines up (already flagged in a prior round; the explicit error message is the stated intent, so leaving it is fine).

Verdict: clean, well-documented, faithful to the existing plugin-host patterns, with the concurrency edges closed by tests that would actually catch regressions. Ready to merge from my side. ✅

@andybons andybons merged commit 1b65188 into main Jul 9, 2026
2 checks passed
@andybons andybons deleted the feat/mcp-engine branch July 9, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant