Serve native MCP over authenticated loopback HTTP

Author: ShadowfetchStudio

.github/workflows/native.yml

@@ -32,11 +32,13 @@ jobs:
         run: node scripts/test-native-cli.mjs
       - name: Exercise the native MCP server without a display
         run: node scripts/test-native-mcp-server.mjs
+      - name: Exercise the authenticated native MCP HTTP server
+        run: SHADOW_MCP_TRANSPORT=http node scripts/test-native-mcp-server.mjs
       - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
         if: always()
         with:
           name: native-mcp-server-diagnostics
-          path: artifacts/native-mcp-server
+          path: artifacts/native-mcp-server*
       - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
         if: always()
         with:
@@ -80,6 +82,13 @@ jobs:
           SHADOW_CLI_ARGS: '["--appimage-extract-and-run"]'
           SHADOW_MCP_ARTIFACTS: artifacts/native-package/mcp-server
         run: node scripts/test-native-mcp-server.mjs
+      - name: Exercise the packaged native MCP HTTP server
+        env:
+          SHADOW_DESKTOP_BINARY: ${{ github.workspace }}/target/release/bundle/appimage/ShadowCode_0.20.0_amd64.AppImage
+          SHADOW_CLI_ARGS: '["--appimage-extract-and-run"]'
+          SHADOW_MCP_ARTIFACTS: artifacts/native-package/mcp-server-http
+          SHADOW_MCP_TRANSPORT: http
+        run: node scripts/test-native-mcp-server.mjs
       - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
         if: always()
         with:

CHANGELOG.md

@@ -2,6 +2,13 @@
 
 ## 0.20.0 — native development (unreleased)
 
+- Native MCP serving supports authenticated loopback Streamable HTTP alongside
+  stdio. Both expose the same project-scoped tools, approvals and owned tasks.
+  HTTP reconnects preserve the gateway's jobs; gateway shutdown cancels unfinished
+  owned work while unrelated engine tasks continue. Mandatory credentials, host
+  checks, denied browser origins, bounded traffic and protocol validation apply.
+  The executable probe runs both transports against source and packaged builds.
+
 - Rust task engine with confined file tools, bounded processes, streaming local
   and compatible models, scoped approvals, checkpoints, persisted queues, and
   recovery. Settings and legacy history are migrated with backups.

Cargo.lock

@@ -22,7 +22,7 @@ workflows, [managed background processes and model tools](docs/NATIVE_BACKGROUND
 [native SQLite inspection](docs/NATIVE_SQLITE.md),
 [queued follow-ups](docs/NATIVE_QUEUE.md),
 [approved MCP stdio and HTTP tools](docs/NATIVE_MCP.md), a
-[native MCP stdio server](docs/NATIVE_MCP.md#connect-another-coding-tool-to-shadowcode), and a
+[native MCP stdio and authenticated HTTP server](docs/NATIVE_MCP.md#connect-another-coding-tool-to-shadowcode), and a
 [native CLI](docs/NATIVE_CLI.md) that shares the active desktop engine or runs
 headlessly with its own profile.
 Development AppImage and Debian

README.md

@@ -150,11 +150,14 @@ Disabling affects new tasks; cancel running and queued tasks to end their
 existing grants. HTTP definitions use `url` and an optional `api_key_env` bearer
 secret reference; remote hosts require network access in Permissions.
 
-To expose ShadowCode itself to another MCP client, use `mcp serve` over stdio.
+To expose ShadowCode itself to another MCP client, use `mcp serve` over stdio, or
+`mcp serve --http 127.0.0.1:8765 --token-env SHADOW_MCP_HTTP_TOKEN` for authenticated
+loopback HTTP. Set that environment/profile secret before starting the gateway.
 `mcp register` prints generic JSON for that executable, project and explicit
 profile. Both default to read-only access. `--allow-write` delegates changes
 within the trusted project's configured permissions; adding `--allow-approvals`
-also delegates individual approvals for that connection's own tasks. Run
+also delegates individual approvals for that MCP owner's tasks. HTTP reconnects
+share the same gateway owner; separate gateways keep clients independent. Run
 `mcp serve` without `--json` so stdout remains protocol-only. See the
 [native server's tools, lifecycle and remaining limits](NATIVE_MCP.md#connect-another-coding-tool-to-shadowcode).
 

docs/NATIVE_CLI.md

@@ -87,7 +87,9 @@ each exact call. HTTP bearer secrets resolve only on connection, and remote
 hosts require network access. Connections close before the task finishes.
 `shadowcode mcp serve` also exposes the selected project to another client over
 stdio, with read-only defaults and explicitly delegated write/approval access.
-The [server guide](NATIVE_MCP.md#connect-another-coding-tool-to-shadowcode) lists
+The native executable also serves authenticated loopback HTTP with a fixed
+project and a gateway owner that survives individual client reconnects. The
+[server guide](NATIVE_MCP.md#connect-another-coding-tool-to-shadowcode) lists
 the implemented tools and remaining migration requirements.
 
 To work on the live interface, run `./ui/node_modules/.bin/tauri dev` from the

docs/NATIVE_DESKTOP.md

@@ -2,14 +2,14 @@
 
 MCP remains an open [native release gate](NATIVE_MIGRATION.md). The Rust stdio and Streamable HTTP
 clients now connect to native Settings, CLI registration, project activation,
-and individually approved agent tool calls. The native stdio server now exposes
-tasks, goals, reviews, memory, SQLite inspection and checkpoints to external clients.
-The remaining server features and broader interoperability/real-model
-verification are still being migrated. Neither direction launches Python.
+and individually approved agent tool calls. Native stdio and authenticated loopback
+HTTP servers expose tasks, goals, reviews, memory, SQLite inspection and checkpoints
+to external clients. Client-specific registration and broader interoperability/real-model
+verification remain release requirements. Neither direction launches Python.
 
 ## Connect another coding tool to ShadowCode
 
-The native executable can serve a single MCP client without a display:
+The native executable can serve a single MCP client over stdio without a display:
 
 ```sh
 shadowcode --workspace /absolute/project mcp serve
@@ -41,7 +41,7 @@ Pending actions can be approved through ShadowCode's desktop or CLI. To delegate
 approval decisions to the connecting client, explicitly add **both**
 `--allow-write --allow-approvals` to `mcp serve` (or to `mcp register` when producing
 its configuration). The `shadow_approve` tool then resolves one exact pending
-approval from a task created by this connection. It cannot approve another
+approval from a task created by this MCP owner (the stdio connection or HTTP gateway). It cannot approve another
 client's task or all pending actions. Denying an owned approval does not require
 the approval flag. The client is responsible for obtaining the user's agreement
 to the displayed action before submitting approval.
@@ -63,7 +63,7 @@ The current server exposes seventeen tools:
 - `shadow_goal` creates, lists, inspects, advances or abandons durable goals.
   Creating or advancing a goal here does not start an agent automatically.
 - `shadow_run` creates a fresh conversation and returns its job ID. `shadow_jobs`
-  inspects that connection's jobs, paginated events and pending approvals, or
+  inspects that owner's jobs, paginated events and pending approvals, or
   cancels one. A returned job ID is not a claim that the task has completed.
 - `shadow_approve` handles the individual decision described above.
 - `shadow_test` submits an owned native test job without calling a model. An
@@ -78,7 +78,7 @@ The plan resource reads recorded events and returns null when no plan is recorde
 The `delegate` prompt accepts a task for the fixed project; `understand` requests
 a read-only project inspection.
 
-The connection owns its delegated jobs. Normal EOF, protocol failure, explicit
+A stdio connection owns its delegated jobs. Normal EOF, protocol failure, explicit
 cancellation and an abandoned server future cancel unfinished owned jobs and await
 cleanup; unrelated tasks in the shared engine continue. A temporary engine closes
 when its server exits. The first delegated task opens a private ownership socket
@@ -101,16 +101,68 @@ close ownership; valid rejected task submissions leave it usable. Submissions
 and cleanup acknowledgements have 30-second response deadlines. Normal close
 waits for the engine's task cleanup before acknowledging it.
 
-Incoming messages share the 1-MiB frame, 32-MiB lifetime and 128-frames-per-second
+Incoming stdio messages share the 1-MiB frame, 32-MiB lifetime and 128-frames-per-second
 transport limits. Initialization has a ten-second deadline. Up to eight tool or
 resource operations and 64 delegated tasks are allowed per connection. Other
 operations have a 45-second deadline; model jobs are asynchronous and follow the
 engine's own limits. Tool/resource results are capped at 2 MB and response writes
 have a five-second deadline. Reconnect after a connection limit is reached.
 
+### Authenticated HTTP gateway
+
+Run a persistent gateway for the selected project with an independently generated
+credential. `--token-env` names an environment variable or private profile secret;
+the actual token is not passed as a command-line argument:
+
+```sh
+export SHADOW_MCP_HTTP_TOKEN="$(openssl rand -hex 32)"
+shadowcode --workspace /absolute/project mcp serve \
+  --http 127.0.0.1:8765 --token-env SHADOW_MCP_HTTP_TOKEN
+```
+
+Configure the calling application's Streamable HTTP transport with
+`http://127.0.0.1:8765/mcp` and `Authorization: Bearer <the secret>`. The gateway
+announces its ready address on stderr; stdout stays empty. Port `0` selects an
+available port, which is reported after the engine is ready. Stdin EOF does not
+stop HTTP serving. SIGINT/SIGTERM stop the gateway and wait for owned cleanup.
+`mcp register` still emits only generic stdio configuration.
+
+The same read-only default, trust requirements and optional
+`--allow-write --allow-approvals` flags apply. A gateway is one fixed-project owner
+shared by every caller holding its credential. Reconnecting over HTTP preserves
+its delegated jobs and their approval scope. Use separate gateway invocations
+and credentials for independent clients. Each gateway has its own engine ownership
+lease; stopping or killing it cancels its unfinished running and queued jobs,
+without stopping another gateway's or the desktop's tasks. Restart after the
+64-job lifetime limit. Project background processes retain their independent
+lifetime described above.
+
+Only loopback IP binds are accepted. Credentials must contain 32–512 bearer-token
+characters; generate them randomly. Authentication is mandatory even on loopback.
+Only the bound IP or `localhost` with the actual port is accepted as Host. Browser
+Origin headers, query parameters and duplicate MCP metadata headers are rejected;
+credential values are removed before requests reach MCP handlers. This endpoint
+does not provide remote deployment, browser CORS, TLS or OAuth.
+
+The official Rust SDK supplies current per-request protocol negotiation and legacy
+Streamable HTTP initialization. The gateway is stateless at the protocol layer:
+it issues no MCP session ID and uses JSON responses, with request-scoped SSE when
+required by the SDK. GET and DELETE return 405. It does not implement the older
+separate `/sse` and message endpoints. See the MCP
+[Streamable HTTP specification](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports/streamable-http)
+and [legacy transport specification](https://modelcontextprotocol.io/specification/2025-11-25/basic/transports).
+
+HTTP permits 32 concurrent TCP connections and 128 requests per second. Each
+connection handles one request, with five-second header/body deadlines, a 1-MiB
+request limit, a 60-second request/response deadline and an 8-MiB transport response
+limit. Existing eight-operation concurrency, 45-second tool deadlines and 2-MB
+result limits still apply. Model/test jobs return an ID and continue asynchronously;
+a closed HTTP request is not a request to cancel an already submitted job. Use
+`shadow_jobs` cancellation or stop its gateway.
+
 This is a development server, not full compatibility with the earlier Python
-server. HTTP serving and client-specific registration formats
-remain unimplemented. Tools use the fixed project and asynchronous owned jobs;
+server. Client-specific registration formats remain unimplemented.
+Tools use the fixed project and asynchronous owned jobs;
 map saving is explicit, and diagnostics do not perform automatic repairs.
 The current tests use the official
 Rust SDK and a real executable protocol probe; broader client interoperability

docs/NATIVE_MCP.md

@@ -161,10 +161,11 @@ transports, inert Settings/CLI registration, explicit project/content activation
 discovery, individually approved calls, and task-owned process/stream cleanup.
 HTTP uses explicit credential references, prevents redirects and automatic call
 retries, and supports modern and legacy Streamable HTTP negotiation. The native
-stdio server exposes seventeen tools, four resources and two prompts through
+stdio and authenticated loopback HTTP servers expose seventeen tools, four resources and two prompts through
 the shared engine, with a fixed project, reduced per-task permissions, explicit
-approval delegation and connection-owned jobs. HTTP serving and broader interoperability/real-model
-verification remain open requirements. Private engine-side ownership now cancels
+approval delegation and owned jobs. HTTP uses a gateway owner across stateless
+requests, with mandatory bearer authentication and bounded traffic. Client-specific
+registration and broader interoperability/real-model verification remain open requirements. Private engine-side ownership now cancels
 MCP jobs after gateway SIGKILL, including unread submission replies, queued tasks
 and active command children; unrelated work continues.
 

docs/NATIVE_MIGRATION.md

@@ -6,7 +6,7 @@ remain in [NATIVE_MIGRATION.md](NATIVE_MIGRATION.md).
 
 ## Automated checks
 
-`cargo fmt --all --check`, Clippy with warnings denied, and all **188 native
+`cargo fmt --all --check`, Clippy with warnings denied, and all **193 native
 integration tests** pass on the development machine. The suite covers:
 
 - Config validation, private secrets, untrusted project overlays, profile locks,
@@ -158,6 +158,21 @@ The seventeen-tool catalog includes project inspection, native diagnostics, SQLi
 approved test execution, which the probe exercises without extra model requests.
 These fixtures do not substitute for broader client and real-model interoperability.
 
+The same executable probe now passes with `SHADOW_MCP_TRANSPORT=http`: six
+scripted model requests, modern per-request negotiation and legacy initialization,
+mandatory authentication, invalid bind/credential rejection, read-only defaults,
+real edits/tests, exact approvals, checkpoint rollback, and gateway SIGKILL cleanup.
+Each HTTP request opens a fresh connection; stdin EOF preserves the gateway and
+its delegated jobs. SIGTERM closes temporary owners cleanly. CI repeats both
+transports against the native binary and packaged AppImage.
+
+Five Rust HTTP-server tests additionally exercise official-SDK interoperability,
+project and owner isolation, duplicate/mismatched metadata, invalid credentials,
+browser origins and hosts, malformed and oversized bodies, slow partial requests,
+reconnects, queued cancellation, child cleanup, and abandoned-future profile/socket
+release. Dependency-notice generation passes with 540 application dependencies,
+including the HTTP server's newly resolved `httpdate` dependency.
+
 The host's distro `rustdoc` needs its LLVM library directory in the loader path
 for doc tests. The full suite was run with:
 

docs/NATIVE_VERIFICATION.md

@@ -25,5 +25,11 @@ libc = "0.2"
 tracing = "0.1"
 cap-std = "3"
 clap = { version = "4.6.7", features = ["derive"] }
-rmcp = { version = "=3.4.0", default-features = false, features = ["client", "server", "transport-streamable-http-client"] }
+rmcp = { version = "=3.4.0", default-features = false, features = ["client", "server", "transport-streamable-http-client", "transport-streamable-http-server"] }
 sse-stream = "=0.2.6"
+bytes = "1"
+http-body = "1"
+http-body-util = "0.1"
+hyper = { version = "1", features = ["http1", "server"] }
+hyper-util = { version = "0.1", features = ["tokio"] }
+tower-service = "0.3"