#1045: ✨ Complete MCP Server Scaffold

.dev/sessions.md

@@ -91,17 +91,29 @@ Newest first.
 - Added `description` as an optional catalogue config property (`configOptionalProperties.DESCRIPTION` in `modules/types`) — surfaces in both the root `/introspection` response and the per-catalogue `/introspection/:catalogId` response via conditional spread (key absent when not configured, not `undefined`)
 - Restructured `CatalogIntrospectionResponse`: removed `validOperators` from individual fields; added top-level `operators: Record<string, string[]>` keyed by field type; `buildFieldOperators()` in `buildCatalogueIntrospection.ts`
 - Updated unit tests to match new shape; added coverage for `description` present/absent and `operators` deduplication
+- Added `mcp-server` Docker target to both `docker/Dockerfile.local` and `docker/Dockerfile.jenkins`, mirroring the existing `server` target structure
+    - Both targets `COPY --from=scaffolding` the shared `node_modules` and `apps/mcp-server` source; no internal modules are copied because the MCP server is standalone (talks to Arranger over HTTP, not ES directly)
+    - CMD runs `node_modules/.bin/tsx ./apps/mcp-server/src/index.ts` (no shell-level pre-check; `validateArrangerConnection` runs in-app at startup)
+    - EXPOSE 3100 matches the `MCP_PORT` default
+- Updated `docker/Dockerfile.jenkins` scaffolding stage to include `--workspace apps/mcp-server` in the `npm ci --omit=dev` install, so the jenkins production image installs the MCP server's runtime deps
+- Moved `tsx` from `devDependencies` to `dependencies` in `apps/mcp-server/package.json` (load-bearing for the jenkins build, which runs `npm ci --omit=dev`; mirrors what `apps/search-server` already does)
+- Updated both `.dockerignore` files (`Dockerfile.local.dockerignore`, `Dockerfile.jenkins.dockerignore`) to add `!apps/mcp-server`, allowing the folder through the build context
 
 **Decisions:**
 
 - `operators` (not `typeOperators`) — cleaner, consistent with existing "operators" vocabulary in SQON introspection
 - `buildFieldOperators` (not `buildTypeOperators`) — "field operators" is the established naming family in `modules/sqon` (`SqonFieldOp`, `SqonFieldOperatorDetail`, `getSqonFieldOperatorDetails`)
 - `description` on per-catalogue response too (not just root listing) — complete data at the endpoint; LLM context optimization is the MCP layer's responsibility
 - `getValidOperators` → `modules/sqon` consolidation is out of scope: requires redesigning `applicableTo` data in `getSqonFieldOperatorDetails` (range types incorrectly include `filter`, `some-not-in`, `all` at present); separate roadmap item
+- MCP server target does NOT use a shell wrapper or pre-flight script. The app's own `validateArrangerConnection` handles the Arranger readiness check at startup — duplicating that at the shell level would be redundant.
+- Kept tsx-from-source at runtime (not a tsc pre-build) to match `apps/search-server`'s pattern. Revisiting that for both apps together is a future concern.
+- Did not modify `docker-compose.yml` — that's a separate "is the MCP server part of the dev stack?" decision.
 
 **Open threads:**
 
 - `getValidFieldOperators` → `modules/sqon` consolidation: follow-up when sqon consolidation roadmap item is picked up
+- `package-lock.json` will need an `npm install` to reflect tsx moving sections in `apps/mcp-server/package.json`.
+- `docker-compose.yml` does not include the MCP server. If the local Compose dev stack should boot MCP alongside server/ES/UI, that needs a follow-up (port 3100, depends_on server, ARRANGER_BASE_URL pointed at the `server` service).
 
 ---
 
@@ -115,6 +127,12 @@ Newest first.
 - Added roadmap item: consolidate field-type-to-operator rules into `modules/sqon`
 - Fleshed out `sqon-builder` absorption into `modules/sqon` as a detailed roadmap item: what to keep (builder API, `reduceSQON`, filter manipulation, `from()`), what to fix (operator coverage gap — only `in`/`gt`/`lt` today), what to leave behind (the `& SQON` anti-pattern), and migration path
 - Added anchor links to all cross-references between `tech-debt.md` and `roadmap.md`
+- Added `integration-tests/mcp-server` workspace with end-to-end tests for `apps/mcp-server`
+    - Spins up Arranger search-server in-process (multicatalog mode) with two test catalogs, then starts the MCP server pointed at it, then drives it over Streamable HTTP via the official MCP SDK Client
+    - Connection assertion is implicit: `validateArrangerConnection` runs before `app.listen`, so the suite reaching the test phase proves the MCP→Arranger contract works
+    - Test coverage: spinup/active (ping, capabilities, resource/tool listings), MCP resources (`arranger://introspection/server`, `arranger://introspection/sqon`, `arranger://introspection/catalog/{id}` via template), MCP tools (`list-catalogs`, `get-sqon-schema`, `get-catalog-fields` happy + 404 paths)
+    - 13 tests in 4 suites; runs against the same local ES used by `integration-tests/server`
+- Added `integration-tests/mcp-server` to the root `package.json` workspaces list
 
 **Decisions:**
 
@@ -123,11 +141,18 @@ Newest first.
 - Boolean values should be supported in SQON (not just string `"true"`); fix is additive — add `zod.boolean()` to `SqonScalarValueSchema`; confirmed this is an oversight, not deliberate
 - The `/introspection/fields` endpoint is the canonical LLM context source — the evaluation document should reference it specifically rather than "GraphQL introspection"
 - JSDoc/TSDoc should be added to functions and types as code is written or touched — not deferred to a documentation pass; inline docs are the safety net when `/docs` lags
+- Backend: spin up search-server in-process (mirrors `integration-tests/server`), not external nor mocked — keeps the harness self-contained while exercising the real Arranger contract.
+- Coverage: multicatalog only — exercises the catalog resource template and `list-catalogs` with >1 catalog. Single-catalog is a subset and not worth doubling runtime for.
+- Catalog field introspection (`/introspection/{catalogId}`) reads from `catalogConfigs.extended`, which is empty in the existing `integration-tests/server` multiconfigs. New fixtures under `integration-tests/mcp-server/multiconfigs/` include populated `extended` arrays so the tests can assert real field metadata.
+- Test files live under `test/` and the entry point is `test/index.test.ts`. Node 24's test runner auto-discovers `.ts` files in `test/`; node 20 does not, so this suite requires node 24+ (consistent with the project's `engines.node >= 20` but practically aligned with the dev shell setup).
+- Lazy MCP client access via `getClient: () => Client` — `node:test` suite factories run at registration time, before `before()` hooks have populated state. Each test resolves the client when it actually runs.
 
 **Open threads:**
 
 - Boolean support in SQON schema: fix is clear but not yet implemented (two schema files, one in `sqon-builder`, one in `modules/sqon`)
 - `reduceSQON` extension for full operator set needs deliberate design (e.g. what does reducing two `between` ranges under `and` mean?)
+- Single-catalog coverage is not exercised; can be added if MCP server adds single-catalog-specific behavior (currently it doesn't differentiate).
+- Negative test for `validateArrangerConnection` failure on startup is covered by unit tests (`apps/mcp-server/src/arranger/validation.test.ts`); not duplicated as an integration test because the production startup path calls `process.exit(1)`, which is awkward to exercise in-process.
 
 ---
 

apps/mcp-server/.env.schema

@@ -0,0 +1,9 @@
+ARRANGER_BASE_URL=http://localhost:5050
+ARRANGER_CATALOGUES=server
+ARRANGER_REQUEST_TIMEOUT_MS=10_000
+
+MCP_HOST=0.0.0.0
+MCP_PORT=3100
+MCP_PATH=/mcp
+
+LOG_LEVEL=info

apps/mcp-server/README.md

@@ -1,68 +1,153 @@
-# Arranger MCP Server Foundation
+# Arranger MCP Server
 
-This app is the starting point for an MCP server that learns how to talk to Arranger by consuming Arranger's introspection endpoints.
+This app is an MCP server that learns how to talk to Arranger by consuming Arranger's introspection endpoints.
 
-The current scaffold does not implement a full MCP transport or SDK binding yet. Instead, it defines:
+The current scaffold implements the Streamable HTTP MCP transport using **v1.x** of the official [MCP TypeScript SDK](https://github.com/modelcontextprotocol/typescript-sdk/tree/v1.x).
 
-- how to read Arranger introspection
-- how to model that information as MCP-friendly resources
-- which MCP tools are natural to expose first
+## Folder Structure
 
-## Why this exists
+```text
+src/
+├── arranger/
+│   ├── client.ts               # fetches Arranger introspection endpoints
+│   ├── types.ts                # response types for introspection payloads
+│   └── validation.ts           # validates the connection to Arranger
+├── http/
+│   └── app.ts                  # MCP express app with Streamable HTTP transport
+├── mcp/
+│   ├── resources.ts            # registers MCP resources
+│   └── tools.ts                # registers MCP tools
+├── utils/
+│   ├── config.ts               # env/config parsing
+│   ├── inMemoryEventStore.ts   # in-memory storage util for dev
+│   └── logger.ts               # pino logger wrapper
+├── index.ts                    # entrypoint for the application
+└── server.ts                   # creates the MCP server
+```
 
-The intended flow is:
+## Quick Start
 
-1. call Arranger's `/introspection`
-2. call Arranger's `/introspection/sqon`
-3. call Arranger's `/introspection/:catalogId`
-4. expose those results to MCP clients as resources and tool-backed lookups
+1. Install dependencies:
 
-## Tools vs skills
+```bash
+# from project root
+npm ci
+```
 
-In this context:
+2. Configure environment variables:
 
-- **tools** are callable operations an MCP client can invoke, such as "list catalogs" or "get SQON schema"
-- **resources** are readable documents the MCP client can fetch, such as the server introspection payload or a catalog field listing
-- **skills** are not part of the MCP standard itself; they are higher-level agent behaviors or authored instructions layered on top of tools/resources
+> [!NOTE]
+> See [Configuration](#configuration) for more details.
 
-For this scaffold, the important MCP building blocks are tools and resources.
+```bash
+# from apps/mcp-server
+cp .env.schema .env
+```
 
-## Folder layout
+3. Build Arranger modules:
 
-```text
-src/
-  arranger/
-    client.ts      # fetches Arranger introspection endpoints
-    types.ts       # response types for introspection payloads
-  mcp/
-    resources.ts   # MCP-facing resource definitions backed by introspection
-    tools.ts       # MCP-facing tool definitions backed by introspection
-    types.ts       # simple MCP-ish types for the scaffold
-  config.ts        # env/config parsing for the future MCP server
-  index.ts         # composition entrypoint for the scaffold
+```bash
+# from project root
+npm run modules:build
 ```
 
-## Intended first MCP features
+4. (Optional) Ensure Elasticsearch and Arranger Server are running.
 
-- `list_catalogs`
-- `get_sqon_schema`
-- `get_catalog_fields`
+> [!NOTE]
+> This is only necessary if you are developing against a local Arranger Server. See [Testing](#testing) for more details.
 
-Those all map directly to already-implemented Arranger introspection endpoints.
+```bash
+# from project root
+make start-es
+ES_INDEX=file_centric DOCUMENT_TYPE=file CONFIGS_PATH=$(pwd)/docker/server npm run dev:server
+```
 
-## Not implemented yet
+5. Start the MCP Server:
 
-- actual MCP SDK bootstrap
-- stdin/stdout server transport
-- authentication
-- query execution tools
-- SQON generation helpers beyond introspection exposure
+```bash
+# from project root
+npm run mcp-server:dev
+```
+
+## Configuration
+
+Configuration of this application is done by providing [environment variables](#environment-variables) to the application at run time.
+
+> [!WARNING]
+> If **required** environment variables are not available or misconfigured at run time, the application will shut down immediately.
+
+An example environment variables file is located at [`.env.schema`](./.env.schema). This example file lists all available configuration variables and is prepopulated with default values that should work to run the application locally. You can copy the contents of this file to populate a `.env`:
+
+```bash
+# from apps/mcp-server
+cp .env.schema .env
+```
+
+### Environment Variables
+
+| Name                       | Description                                                             | Type     | Required     | Default                 |
+| -------------------------- | ----------------------------------------------------------------------- | -------- | ------------ | ----------------------- |
+| `ARRANGER_BASE_URL`        | URL for the Arranger Server                                             | `string` | **Required** | `http://localhost:5050` |
+| `ARRANGER_CATALOGUES`      | Comma-separated list of Arranger catalogues to expose to the MCP Server | `string` | **Required** | `server`                |
+| `ARRANGER_REQUEST_TIMEOUT_MS` | Timeout for requests to Arranger                                        | `number` | Optional     | `10_000`               |
+| `MCP_HOST`                 | Host URL for the MCP server                                             | `string` | Optional     | `0.0.0.0`               |
+| `MCP_PORT`                 | Port the MCP Server will listen for requests on                         | `number` | Optional     | `3100`                  |
+| `MCP_PATH`                 | Endpoint for the MCP Streamable HTTP transport                           | `string` | Optional     | `/mcp`                  |
+| `LOG_LEVEL`                | Pino [log level](https://getpino.io/#/docs/api?id=level-1)              | `string` | Optional     | `info`                  |
+
+## Testing
+
+### Local Arranger
 
-## Suggested next step
+To test the MCP Server against a **local** instance of Arranger Server:
 
-When you're ready to implement the actual MCP server, keep this app focused on:
+1. Confirm your [`apps/mcp-server/.env`](.env) configuration aligns with your local Arranger server.
 
-- fetching and caching Arranger introspection
-- exposing catalog and SQON metadata to MCP clients
+2. Ensure ES and Arranger Server are running:
 
-Then add query execution tools only after the metadata contract feels stable.
+```bash
+# from project root
+
+# start ES (note: you may need to seed ES with `make seed-es` after if this is your first time)
+make start-es
+
+# start Arranger Server (config may vary)
+ES_INDEX=file_centric DOCUMENT_TYPE=file CONFIGS_PATH=$(pwd)/docker/server npm run dev:server
+```
+
+3. Start the MCP Server:
+
+```bash
+# from project root
+npm run mcp-server:dev
+```
+
+4. Start the [MCP Inspector](https://github.com/modelcontextprotocol/inspector):
+
+```bash
+# from project root
+npm run mcp-server:inspect
+```
+
+5. You can then open the MCP Inspector URL in your web browser (`http://localhost:6274/?MCP_PROXY_AUTH_TOKEN={AUTH_TOKEN}`), connect to the MCP Server via Streamable HTTP, and test the Resources and Tools.
+
+### Remote Arranger
+
+To test against a **remote** instance of Arranger Server:
+
+1. Update the `ARRANGER_BASE_URL` and `ARRANGER_CATALOGUES` in your MCP Server `.env` file to point to and reflect the state of your remote Arranger.
+2. Follow steps 3-5 of the [**local**](#local-arranger) testing instructions.
+
+### LM Studio
+
+To test with **LM Studio** instead of MCP Inspector:
+
+- Follow the LM Studio instructions to add an MCP server configuration: https://lmstudio.ai/docs/app/mcp
+    - Provide the config JSON in [`apps/mcp-server/mcp-inspector.json`](./mcp-inspector.json)
+
+## Not Implemented Yet
+
+- stdin/stdout server transport
+- authentication
+- query execution tools
+- SQON generation helpers beyond introspection exposure

apps/mcp-server/mcp-inspector.json

@@ -0,0 +1,8 @@
+{
+	"mcpServers": {
+		"mcp-server": {
+			"type": "streamable-http",
+			"url": "http://127.0.0.1:3100/mcp"
+		}
+	}
+}

apps/mcp-server/package.json

@@ -1,7 +1,45 @@
 {
 	"$schema": "https://json.schemastore.org/package.json",
 	"name": "@overture-stack/arranger-mcp-server",
+	"description": "Arranger MCP Server",
 	"version": "0.0.0-dev",
+	"homepage": "https://github.com/overture-stack/arranger#readme",
+	"bugs": {
+		"url": "https://github.com/overture-stack/arranger/issues"
+	},
+	"main": "src/index.ts",
+	"types": "src/index.ts",
+	"repository": {
+		"directory": "apps/mcp-server",
+		"type": "git",
+		"url": "git+https://github.com/overture-stack/arranger.git"
+	},
 	"private": true,
+	"scripts": {
+		"dev": "NODE_ENV=development tsx watch --include './src/**/*' ./src/index.ts",
+		"start": "tsx ./src/index.ts",
+		"test": "tsx --test --experimental-test-module-mocks ./src/**/*.test.ts",
+		"inspect": "npx @modelcontextprotocol/inspector --config ./mcp-inspector.json --server mcp-server"
+	},
+	"dependencies": {
+		"@modelcontextprotocol/sdk": "^1.29.0",
+		"cors": "^2.8.6",
+		"dotenv": "^16.6.1",
+		"express": "^4.21.2",
+		"pino": "^10.3.1",
+		"pino-pretty": "^13.1.3",
+		"tsx": "^4.21.0",
+		"zod": "^3.25.76"
+	},
+	"devDependencies": {
+		"@tsconfig/node22": "^22.0.5",
+		"@types/cors": "^2.8.12",
+		"@types/express": "^4.17.14",
+		"@types/node": "^25.6.2",
+		"typescript": "^5.8.3"
+	},
+	"imports": {
+		"#*": "./src/*"
+	},
 	"type": "module"
 }

apps/mcp-server/src/arranger/client.ts

@@ -1,4 +1,4 @@
-import type { ArrangerMcpConfig } from '#config.js';
+import type { ArrangerMcpConfig } from '#utils/config.js';
 
 import type {
 	ArrangerCatalogIntrospection,