feat: read the engine's unfulfilled-keys report instead of diffing declared vs delivered

[ralphstodomingo]

Issue for this PR

Closes #1307

Type of change

• Bug fix
• New feature
• Refactor / code improvement
• Documentation

What does this PR do?

When a workspace is bound, the attach toast says "N of M declared integration tools available" and lists what is declared but absent. Until now that list came from a client-side diff: fetch the workspace's allowlist from the API, subtract the tool names the engine served. A diff can name keys, never reasons — an expired Jira token, an MCP server whose binary is not installed, an integration the tenant removed from the catalog, an extension tool with no VS Code window, and a key the provider does not offer all read the same.

@altimateai/datamate 0.7.2 (AltimateAI/altimate-mcp-engine#248) reports every declared-but-unserved key with a reason under _meta["ai.altimate/unfulfilled"] on each tools/list response. This PR reads it:

• MCP catalog keeps the _meta of a server's last tools/list page per client. paginate keeps only each page's items, so the result object — the only carrier of _meta — was dropped. A listing starts with none; any page that carries one sets it; a listing without one clears it. Exposed as MCP.listMeta(name) (undefined while not connected).
• Attach takes the gaps from the report instead of the diff. no-bridge entries stay out of the "missing" line, as absent extension tools without an IDE were already treated as expected; every other reason is named, grouped, with the engine's detail (e.g. spawn docker ENOENT) — Declared but not available — no usable connection: jira_search_issues; server failed to start (spawn docker ENOENT): gh_list_prs, gh_create_pr. The "N of M" headline still counts declared keys that are present. The attached outcome carries the full report for later surfaces.
• No report means no claim. An engine that sends no _meta (nothing at or above the floor does) yields an outcome with neither missing nor unfulfilled, and a toast with no gap line — not "all served".
• Floor moves to 0.7.2 (MIN_ENGINE_VERSION), the first engine that emits the report. Do not merge before @altimateai/datamate@0.7.2 is on npm (AltimateAI/altimate-mcp-engine#249 is the bump); until then every attach would refuse with "needs 0.7.2 or newer".

The client no longer reads what it cannot know: the allowlist lookup (declared()) is kept only for the headline's denominator and the extension-tool count, and a report without a reachable allowlist still names the gaps.

Claims

• C1 — Nothing is invented. missing and unfulfilled exist on the outcome only when the engine sent a well-formed report; a malformed or absent _meta yields neither (test: "an engine that sends no report is not read as having no gaps"; parseUnfulfilled cases).
• C2 — no-bridge never counts as missing, and every other reason does — including unknown-key on an extension key while a bridge is connected (reportedMissing; test "no-bridge entries in the report are expected, never missing").
• C3 — The catalog keeps the report across the paths that list tools: initial connect, the tools/list_changed refresh, the post-OAuth reconnect all go through McpCatalog.defs → listTools, which is the only writer (catalog-list-meta.test.ts covers first page, multi-page, and clearing).
• C4 — A gap whose reason changes is announced again; an identical report is not (signature carries key=reason; test "a gap whose reason changed is announced again").
• C5 — Servers other than the engine are unaffected: _meta is retained per client but read only for datamate; tool conversion and the stored defs are unchanged.

Residuals

• R1 — Reasons outside the engine's current set are shown verbatim (a newer engine may add one) rather than dropped.
• R2 — The toast shows at most five keys across groups and truncates a detail at 60 characters; the full report is on the outcome.
• R3 — The floor bump refuses 0.7.1 engines; that is the intended contract, and the reason is in the MIN_ENGINE_VERSION comment.

How did you verify your code works?

• bun run typecheck clean; prettier clean on the files this PR touches (the files that were already non-conforming on main are left as they were).

• test/altimate/workspace (all), test/mcp/catalog-list-meta.test.ts, test/altimate/precedence-guard-order.test.ts: 460 pass. test/mcp and the two session suites whose MCP stubs gained listMeta: 279 pass; the 5 mcp.headers failures and 1 oauth-auto-connect failure reproduce identically on an untouched main checkout (environmental, not this change).

• New tests: 6 attach cases (reasons in the toast, no-bridge exclusion, no-report, report-without-allowlist, reason-change re-announce, the existing inventory case now stating the engine's report), describeMissing/parseUnfulfilled/reportedMissing unit cases, 3 catalog cases over a real in-memory MCP server.

• End to end through the real MCP service (test/mcp/engine-unfulfilled.e2e.test.ts, env-guarded, skipped in CI): the engine at AltimateAI/altimate-mcp-engine#248's head built as 0.7.2 is spawned over stdio by MCP.add exactly as the overlay spawns it, against a fake Altimate API, a real second MCP server and a missing binary; MCP.listMeta("datamate") returns the five-entry report with the expected reasons and the toast text reads Declared but not available — no usable connection: jira_search_issues; not offered by the integration: ghost; server failed to start (spawn altimate-e2e-missing-binary ENOENT): whatever; no longer in the catalog: retired_tool. — 1 pass. Run it with ALTIMATE_ENGINE_E2E_ROOT=<engine checkout with dist/> bun test test/mcp/engine-unfulfilled.e2e.test.ts from packages/opencode.

• Engine → CLI through the real attach path (evidence): bootstrap + beforeTurn on a bound directory against the 0.7.2 release candidate (engine PRs 250 + 248 merged, built locally, on PATH as datamate). Settled outcome attached with declared: 5, missing: [jira_search_issues, ghost, whatever, retired_tool], the full report incl. the no-bridge entry, and the exact toast text; 8/8 checks. A 0.7.1 build is refused as engine-too-old with the install line; 2/2.

Screenshots / recordings

Not a UI change beyond toast text; the exact strings are asserted in the tests above.

Checklist

• I have tested my changes locally
• I have not included unrelated changes in this PR

🤖 Generated with Claude Code

https://claude.ai/code/session_01GHBUvb843k1R7UAGi8Ya9b

Appendix — complexity delta (altimate-code: engine unfulfilled report)

e8c21c2af7 → 93c879af8 · only functions this diff touches · advisory, not a gate.

✅ No touched function changed in complexity (12 touched, 4 new, all under 10).

ℹ️ How to read these numbers

Cognitive (Sonar spec) counts breaks in linear reading flow — each if/loop/catch/ternary/boolean-operator switch adds 1, and nesting makes every further break cost more. It approximates how much you must hold in your head to follow the function: 0–5 trivial · 6–10 easy · 11–15 moderate (15 = Sonar's recommended per-function cap) · 16–25 hard to follow · >25 needs decomposition.

CCN (cyclomatic) counts independent paths — also the minimum number of test cases for full branch coverage of the function.

Only functions this diff touches are measured, as deltas — pre-existing complexity is not counted against this change. Rising numbers aren't automatically wrong; they're where review attention should go. Test files excluded.

---

Summary by cubic

Replaces the client-side diff of declared vs delivered tools with the engine's own unfulfilled-keys report, so the attach toast can say why a tool is missing (expired connection, failed spawn, etc.) instead of just naming it. Closes #1307.

MCP catalog

• Keeps each server's tools/list _meta per client, exposed via MCP.listMeta(name).

Behavior

• no-bridge entries never count as missing; every other reported reason does.
• An engine that sends no report claims no gaps.
• MIN_ENGINE_VERSION moves to 0.7.2, the first engine that emits the report.
• A gap whose reason changed is announced again; an identical report is not.
• The attached outcome carries the full report.
• Numeric integration ids (custom integrations) are read as strings, so one of them no longer voids the whole report.

^{Written for commit 8253154. Summary will update on new commits.}

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ralphstodomingo]

Engine → altimate-code, through the real attach path

This runs the CLI's production attach code under a real instance with no model turn: the binding cache, the datamate lookup on PATH, the --version probe, the allowlist lookup, the MCP spawn, the report and the toast are all the real code (bootstrap(dir, …) → beforeTurn(sessionID)). Only two things are not production: the SaaS API is a local fake (it serves both the CLI's endpoints and the engine's, and every call is asserted 200), and the toast sink is captured instead of published to a TUI bus. The engine is the 0.7.2 release candidate: AltimateAI/altimate-mcp-engine#250's head with AltimateAI/altimate-mcp-engine#248 merged in, built locally (datamate --version → 0.7.2), reached through a shim on PATH exactly as a global install would be.

Declared by the workspace: jira (no connection on this machine), vscode-power-user (extension), mcp-ok (a real second MCP server offering only echo), mcp-missing-binary (command absent), retired-integration (not in the catalog).

Result — release candidate (head engine-rc-0.7.2, 6170 ms from beforeTurn to settled outcome)

Toast (variant warning):

1 of 5 declared integration tools available. Declared but not available — no usable connection: jira_search_issues; not offered by the integration: ghost; server failed to start (spawn altimate-e2e-missing-binary ENOENT): whatever; no longer in the catalog: retired_tool.

Settled outcome:

{
  "kind": "attached",
  "available": 1,
  "declared": 5,
  "missing": [
    "jira_search_issues",
    "ghost",
    "whatever",
    "retired_tool"
  ],
  "unfulfilled": [
    {
      "key": "jira_search_issues",
      "integrationId": "jira",
      "reason": "invalid-connection"
    },
    {
      "key": "pu_lineage",
      "integrationId": "vscode-power-user",
      "reason": "no-bridge"
    },
    {
      "key": "ghost",
      "integrationId": "mcp-ok",
      "reason": "unknown-key"
    },
    {
      "key": "whatever",
      "integrationId": "mcp-missing-binary",
      "reason": "spawn-failed",
      "detail": "spawn altimate-e2e-missing-binary ENOENT"
    },
    {
      "key": "retired_tool",
      "integrationId": "retired-integration",
      "reason": "catalog-missing"
    }
  ]
}

• PASS outcome is attached
• PASS declared counted from the allowlist (5 CLI-servable keys)
• PASS missing = every gap the engine reported except no-bridge
• PASS outcome carries the full report incl. the no-bridge entry
• PASS spawn-failed detail carries the engine's error
• PASS exactly one toast, warning
• PASS toast text
• PASS every API call served (no 404)

API calls, in order: GET /skills -> 200, GET /datamates/77/summary -> 200, GET /datamate_integrations/ -> 200, GET /dbt/v3/validate-credentials -> 200, GET /datamates -> 200, GET /datamate_integrations -> 200, GET /mask -> 200, GET /datamate_integrations/custom -> 200, GET /connections -> 200 — the first three are the CLI's, the rest the engine's.

Result — floor negative (engine 0.7.1, headless)

The same run against a 0.7.1 build settles engine-too-old (found 0.7.1) and prints one stderr line:

Workspace "e2e-rc": 5 integration tools need datamate 0.7.2+ (found 0.7.1). Update with: npm i -g @altimateai/datamate@0.7.2

• PASS engine below the floor is refused as engine-too-old
• PASS refusal names the found version and the floor

Not covered

Windows; a real tenant; the TUI rendering of the toast (the text is asserted, the widget is not).

Reproduce — .e2e/engine-to-cli.ts + .e2e/run.sh (run from packages/opencode)

.e2e/run.sh <engine checkout with dist/> out.json            # attach
HEADLESS=1 .e2e/run.sh <0.7.1 engine> out.json too-old      # floor negative

run.sh (starts bun with an isolated HOME — Bun caches os.homedir() at startup, so the script cannot set it itself):

#!/usr/bin/env bash
# usage: .e2e/run.sh <engine-root> <out.json> [EXPECT]
set -uo pipefail
H=$(mktemp -d /tmp/e2e-engine-to-cli-home-XXXXXX)
env -i HOME="$H" PATH="$H/bin:/usr/local/bin:/usr/bin:/bin:$(dirname "$(command -v bun)"):$(dirname "$(command -v node)")" TERM=dumb ALTIMATE_WORKSPACE=1 ${HEADLESS:+ALTIMATE_CODE_HEADLESS=1} ENGINE_ROOT="$1" EXPECT="${3:-}" \
  timeout 180 bun .e2e/engine-to-cli.ts > "$2" 2> "${2%.json}.stderr"
echo "RC=$? HOME=$H"

engine-to-cli.ts:

// Engine → altimate-code, through the CLI's real attach path. Nothing is
// stubbed except the toast sink (so its text can be captured) and the SaaS
// API (served locally). The binding, the `datamate` lookup on PATH, the
// version probe, the allowlist lookup, the MCP spawn and the report all run
// the production code under a real instance — no model turn.
import http from "node:http"
import path from "node:path"
import { mkdtempSync, mkdirSync, writeFileSync, chmodSync } from "node:fs"
import { tmpdir } from "node:os"
import { execFileSync } from "node:child_process"

const engineRoot = process.env["ENGINE_ROOT"]!
const node = Bun.which("node")!
const DATAMATE_ID = 77

// ---- fake SaaS: engine endpoints + the CLI's own ----------------------------
const catalog = [
  { id: "jira", type: "tool", name: "Jira", description: "", url: "", supportsLocalConnectionTest: true, supportsSaasConnectionTest: false,
    config: [{ key: "url", name: "URL", type: "string", required: true }, { key: "email", name: "Email", type: "string", required: true }, { key: "token", name: "Token", type: "string", required: true }],
    tools: [{ key: "jira_search_issues", name: "Search issues" }] },
  { id: "vscode-power-user", type: "extension", name: "Power User for dbt", description: "", url: "", supportsLocalConnectionTest: false, supportsSaasConnectionTest: false, config: [],
    tools: [{ key: "pu_lineage", name: "Lineage" }] },
]
const custom = [
  { id: "mcp-ok", type: "mcp", name: "Echo MCP", description: "", url: "", config: [],
    toolConfig: [{ key: "type", name: "type", type: "string", required: false, value: "stdio" }, { key: "command", name: "command", type: "string", required: true, value: node },
      { key: "arguments", name: "arguments", type: "array", required: false, value: [path.join(import.meta.dir, "../test/mcp/fixtures/echo-mcp-server.mjs")] }],
    tools: [{ key: "echo" }, { key: "ghost" }] },
  { id: "mcp-missing-binary", type: "mcp", name: "Missing MCP", description: "", url: "", config: [],
    toolConfig: [{ key: "type", name: "type", type: "string", required: false, value: "stdio" }, { key: "command", name: "command", type: "string", required: true, value: "altimate-e2e-missing-binary" }],
    tools: [{ key: "whatever" }] },
]
const datamate = {
  id: String(DATAMATE_ID), name: "e2e-rc", description: "", privacy: "private", memory_enabled: false, knowledge_engine_enabled: false, knowledge_bases: [],
  integrations: [
    { id: "jira", type: "tool", name: "Jira", description: "", url: "", tools: [{ key: "jira_search_issues" }] },
    { id: "vscode-power-user", type: "extension", name: "PU", description: "", url: "", tools: [{ key: "pu_lineage" }] },
    { id: "mcp-ok", type: "mcp", name: "Echo MCP", description: "", url: "", tools: [{ key: "echo" }, { key: "ghost" }] },
    { id: "mcp-missing-binary", type: "mcp", name: "Missing MCP", description: "", url: "", tools: [{ key: "whatever" }] },
    { id: "retired-integration", type: "tool", name: "Retired", description: "", url: "", tools: [{ key: "retired_tool" }] },
  ],
}
const hits: string[] = []
const api = http.createServer((req, res) => {
  const url = new URL(req.url ?? "/", "http://x"); const p = url.pathname.replace(/\/+$/, "") || "/"
  const json = (code: number, body?: unknown) => { hits.push(`${req.method} ${url.pathname} -> ${code}`); res.writeHead(code, { "content-type": "application/json" }); res.end(body === undefined ? "" : JSON.stringify(body)) }
  if (p === "/dbt/v3/validate-credentials") return json(200, { ok: true })
  if (p === "/datamates") return json(200, { datamates: [datamate] })
  if (p === `/datamates/${DATAMATE_ID}/summary`) return json(200, { datamate })
  if (p === "/datamate_integrations") return json(200, catalog)
  if (p === "/datamate_integrations/custom") return json(200, { items: custom })
  if (p === "/mask") return json(200, { mask_data: [] })
  if (p === "/connections") return json(200, { connections: [] })
  if (p === `/datamates/${DATAMATE_ID}/knowledge_bases`) return json(200, { knowledge_bases: [] })
  if (p === `/datamates/${DATAMATE_ID}/knowledge_engine_description`) return json(200, {})
  if (p === "/datamates/audit/create_batch") return json(204)
  if (p === "/skills") return json(200, { skills: [] })
  return json(404, { detail: `unhandled ${p}` })
})
await new Promise<void>((r) => api.listen(0, "127.0.0.1", r))
const apiUrl = `http://127.0.0.1:${(api.address() as { port: number }).port}`

// ---- isolated HOME: the wrapper starts this process with HOME already pointing
// at a fresh directory (Bun caches os.homedir() at startup, so setting it here
// would be too late for Global.Path); this script only fills it in.
const home = process.env["HOME"]!
if (!home.includes("e2e-engine-to-cli-home-")) throw new Error(`refusing to run against a real HOME: ${home}`)
mkdirSync(path.join(home, ".altimate"), { recursive: true })
writeFileSync(path.join(home, ".altimate/altimate.json"), JSON.stringify({ altimateUrl: apiUrl, altimateInstanceName: "e2e", altimateApiKey: "e2e-key" }))
writeFileSync(path.join(home, ".altimate/settings.json"), "{}")
writeFileSync(path.join(home, ".altimate/connections.json"), "[]")
// `datamate` on PATH → the engine under test, on node (the published bin is a node shebang script)
const bin = path.join(home, "bin"); mkdirSync(bin)
writeFileSync(path.join(bin, "datamate"), `#!/bin/sh\nexec "${node}" "${path.join(engineRoot, "dist/cli.js")}" "$@"\n`); chmodSync(path.join(bin, "datamate"), 0o755)
process.env["PATH"] = `${bin}:${process.env["PATH"]}`
process.env["ALTIMATE_WORKSPACE"] = "1"
const resolved = Bun.which("datamate")
const engineVersion = execFileSync(resolved!, ["--version"], { encoding: "utf8" }).trim().split("\n").pop()

// a bound project directory
const project = mkdtempSync(path.join(tmpdir(), "e2e-engine-to-cli-project-"))
execFileSync("git", ["init", "-q", project])

// ---- production modules, imported only after the environment is shaped ----
const { bootstrap } = await import("../src/cli/bootstrap")
const { recordApprovedBinding } = await import("../src/altimate/workspace/state")
const { beforeTurn, settledOutcome } = await import("../src/altimate/workspace/engine-overlay")
const { syncInternals } = await import("../src/altimate/workspace/engine-seams")
const toasts: { title: string; message: string; variant: string }[] = []
syncInternals.notify = async (t) => { toasts.push(t) }   // capture only; no TUI bus here
const lines: string[] = []
syncInternals.printLine = (l) => { lines.push(l) }

await recordApprovedBinding(project, { datamateId: DATAMATE_ID, datamateName: "e2e-rc", repoRemote: null, projectPath: project, linkedAt: Date.now() })
const t0 = Date.now()
const result = await bootstrap(project, async () => {
  await beforeTurn("s1")
  return settledOutcome("s1")
})
const elapsedMs = Date.now() - t0
api.close()

const checks: string[] = []
const check = (label: string, ok: boolean) => checks.push(`${ok ? "PASS" : "FAIL"} ${label}`)
const attached = result?.kind === "attached" ? result : undefined
if (process.env["EXPECT"] === "too-old") {
  check("engine below the floor is refused as engine-too-old", result?.kind === "engine-too-old")
  check("refusal names the found version and the floor", lines.concat(toasts.map((t) => t.message)).some((l) => l.includes(engineVersion!) && l.includes("0.7.2")))
} else {
  check("outcome is attached", !!attached)
  check("declared counted from the allowlist (5 CLI-servable keys)", attached?.declared === 5)
  check("missing = every gap the engine reported except no-bridge", JSON.stringify(attached?.missing) === JSON.stringify(["jira_search_issues", "ghost", "whatever", "retired_tool"]))
  check("outcome carries the full report incl. the no-bridge entry", !!attached?.unfulfilled?.some((u) => u.key === "pu_lineage" && u.reason === "no-bridge"))
  check("spawn-failed detail carries the engine's error", /ENOENT/.test(attached?.unfulfilled?.find((u) => u.key === "whatever")?.detail ?? ""))
  check("exactly one toast, warning", toasts.length === 1 && toasts[0].variant === "warning")
  check("toast text", toasts[0]?.message === "1 of 5 declared integration tools available. Declared but not available — no usable connection: jira_search_issues; not offered by the integration: ghost; server failed to start (spawn altimate-e2e-missing-binary ENOENT): whatever; no longer in the catalog: retired_tool.")
  check("every API call served (no 404)", !hits.some((h) => / -> 404$/.test(h)))
}
console.log(JSON.stringify({ engineRoot, home, resolvedDatamate: resolved, engineVersion, elapsedMs, outcome: result, toasts, lines, apiHits: hits, checks, verdict: checks.every((c) => c.startsWith("PASS")) ? "ALL PASS" : "FAILURES" }, null, 2))
process.exit(checks.every((c) => c.startsWith("PASS")) ? 0 : 1)

[ralphstodomingo]

One more commit on this head, 82531540c — accept numeric integration ids in the engine report. A tenant-created (custom) MCP integration's id arrives from the engine as a number; the parser treated the whole report as malformed over that one field and the attach announced no gaps at all. The id is now taken as a string (unit test added). Found by the real-chain run for the attach-report post (#1310); the engine side stringifies too (AltimateAI/altimate-mcp-engine#248 64e5bb4), so either side alone is enough.