Skip to content

Commit 4d326c3

Browse files
committed
feat(mcp): let agent tools return MCP content blocks
The MCP adapter sent every tool result as one JSON text block, so a tool could not send an image. withMcpContent (from devframe) now attaches MCP content blocks (text, image or audio) to a tool result under a non-enumerable symbol key, and tools/call sends them in place of the JSON text block. The value itself does not change, so the RPC transport, returns validation and structuredContent keep the plain result. Invalid blocks give an isError result with DF0081.
1 parent 3c85f76 commit 4d326c3

17 files changed

Lines changed: 325 additions & 8 deletions

File tree

‎docs/content/1.guide/15.agent-native.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -189,6 +189,29 @@ ctx.agent.registerTool({
189189
})
190190
```
191191

192+
## Returning images and other content
193+
194+
The MCP adapter sends a tool result as one text block that holds the result as JSON. To send other MCP content, such as a screenshot next to its description, attach content blocks to the result with `withMcpContent`:
195+
196+
```ts
197+
import { defineRpcFunction, withMcpContent } from 'devframe'
198+
199+
export const captureComponent = defineRpcFunction({
200+
name: 'my-plugin:capture-component',
201+
type: 'query',
202+
agent: { description: 'Take a screenshot of one component. Call it when the layout of the component matters.' },
203+
handler: async () => {
204+
const { markdown, png } = await capture()
205+
return withMcpContent({ markdown }, [
206+
{ type: 'text', text: markdown },
207+
{ type: 'image', data: png, mimeType: 'image/png' },
208+
])
209+
},
210+
})
211+
```
212+
213+
The adapter sends the blocks in place of the JSON text block. A block is `text`, `image` or `audio`, and `data` is base64. The blocks are stored under a hidden key, so the returned value does not change: the SPA gets `{ markdown }` over RPC, without the image data. If the tool has an output schema, `structuredContent` is also that plain value. Invalid blocks make the call fail with [`DF0081`](/errors/DF0081).
214+
192215
## Structured errors
193216

194217
A coded diagnostic thrown from a handler crosses the MCP boundary as JSON:

‎docs/content/6.errors/DF0081.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
---
2+
title: 'DF0081: Invalid MCP Content Blocks'
3+
description: 'Tool "{name}" returned invalid MCP content blocks: {reason}'
4+
---
5+
6+
## Message
7+
8+
> Tool "`{name}`" returned invalid MCP content blocks: `{reason}`
9+
10+
## Cause
11+
12+
A tool result carries content blocks from `withMcpContent`, but the blocks are not a non-empty array of valid blocks. A valid block is `{ type: 'text', text }`, `{ type: 'image', data, mimeType }` or `{ type: 'audio', data, mimeType }`, and each field is a string. The MCP adapter does not send invalid blocks. It returns this error as an `isError` result of the `tools/call` request.
13+
14+
## Example
15+
16+
```ts
17+
import { withMcpContent } from 'devframe'
18+
19+
ctx.agent.registerTool({
20+
id: 'my-plugin:snapshot',
21+
description: 'Take a snapshot.',
22+
// The image block has no `mimeType`.
23+
handler: () => withMcpContent({}, [{ type: 'image', data: png }]),
24+
})
25+
```
26+
27+
## Fix
28+
29+
Give `withMcpContent` at least one block, and give each block all of its fields: `text` for a text block, base64 `data` and a `mimeType` (for example `image/png`) for an image or audio block.
30+
31+
## Source
32+
33+
- [`packages/agentic/src/mcp/build-server.ts`](https://github.com/devframes/devframe/blob/main/packages/agentic/src/mcp/build-server.ts): `mcpContentOf()` throws this when the content blocks of a `tools/call` result are invalid.

‎docs/content/6.errors/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,7 @@ Emitted by `devframe`: the framework-neutral host, RPC, streaming, assets, servi
8686
| [DF0077](/errors/DF0077) | error | In-Page Channel Function Not Registered |
8787
| [DF0078](/errors/DF0078) | warn | Agent Surface Without @devframes/agentic |
8888
| [DF0079](/errors/DF0079) | error | MCP Enabled Without @devframes/agentic |
89+
| [DF0081](/errors/DF0081) | error | Invalid MCP Content Blocks |
8990

9091
## Hub: context & lifecycle (DF80xx)
9192

‎packages/agentic/src/mcp/__tests__/mcp-http.test.ts‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
11
import type { StartedServer } from 'devframe/internal'
22
import type { DevframeDefinition, McpRouteOptions } from 'devframe/types'
33
import { Client, StreamableHTTPClientTransport } from '@modelcontextprotocol/client'
4+
import { withMcpContent } from 'devframe'
45
import { createDevServer } from 'devframe/adapters/dev'
56
import { afterEach, describe, expect, it } from 'vitest'
67

78
const TOKEN = 'a-high-entropy-test-bearer-token'
9+
/** A 1x1 PNG. */
10+
const PNG_BASE64 = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8DwHwAFBQIAX8jx0gAAAABJRU5ErkJggg=='
811

912
function defineTestDef(overrides?: Partial<DevframeDefinition>): DevframeDefinition {
1013
return {
@@ -95,6 +98,38 @@ describe('mcp adapter (streamable http route)', () => {
9598
}
9699
})
97100

101+
it('sends the content blocks attached with withMcpContent', async () => {
102+
const started = await boot(true, defineTestDef({
103+
setup(ctx) {
104+
ctx.agent.registerTool({
105+
id: 'snapshot',
106+
description: 'Take a snapshot.',
107+
safety: 'read',
108+
handler: () => withMcpContent({ markdown: 'A red pixel.' }, [
109+
{ type: 'text', text: 'A red pixel.' },
110+
{ type: 'image', data: PNG_BASE64, mimeType: 'image/png' },
111+
]),
112+
})
113+
},
114+
}))
115+
const client = new Client(
116+
{ name: 'test-client', version: '0.0.0' },
117+
{ versionNegotiation: { mode: 'auto' } },
118+
)
119+
try {
120+
await client.connect(originTransport(started))
121+
const result = await client.callTool({ name: 'snapshot', arguments: {} })
122+
expect(result.isError).toBeFalsy()
123+
expect(result.content).toEqual([
124+
{ type: 'text', text: 'A red pixel.' },
125+
{ type: 'image', data: PNG_BASE64, mimeType: 'image/png' },
126+
])
127+
}
128+
finally {
129+
await client.close()
130+
}
131+
})
132+
98133
it('answers a bare GET with 405 (no session lifecycle)', async () => {
99134
const started = await boot()
100135
// Stateless serving has no session stream to open, so the SDK answers a

‎packages/agentic/src/mcp/__tests__/mcp-server.test.ts‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,27 @@
1+
import type { StandardSchemaV1 } from '@standard-schema/spec'
12
import type { DevframeHost } from 'devframe/types'
23
import { Client, InMemoryTransport } from '@modelcontextprotocol/client'
4+
import { withMcpContent } from 'devframe'
35
import { createHostContext } from 'devframe/node'
46
import * as v from 'valibot'
57
import { describe, expect, it } from 'vitest'
68
import { buildMcpServerFromContext } from '../build-server'
79

10+
/** A 1x1 PNG. */
11+
const PNG_BASE64 = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8DwHwAFBQIAX8jx0gAAAABJRU5ErkJggg=='
12+
13+
/** A Standard Schema that accepts any value and converts to `jsonSchema`. */
14+
function objectSchema(jsonSchema: Record<string, unknown>): StandardSchemaV1 {
15+
return {
16+
'~standard': {
17+
version: 1,
18+
vendor: 'test',
19+
validate: (value: unknown) => ({ value }),
20+
jsonSchema: { input: () => jsonSchema, output: () => jsonSchema },
21+
} as StandardSchemaV1['~standard'],
22+
}
23+
}
24+
825
function nullHost(): DevframeHost {
926
return {
1027
mountStatic: () => { /* no-op */ },
@@ -166,6 +183,81 @@ describe('mcp adapter (in-memory)', () => {
166183
}
167184
})
168185

186+
it('sends the content blocks attached with withMcpContent in place of the JSON text', async () => {
187+
const { ctx, client, cleanup } = await bootPair()
188+
try {
189+
ctx.agent.registerTool({
190+
id: 'snapshot',
191+
description: 'Take a snapshot.',
192+
handler: () => withMcpContent({ markdown: 'A red pixel.' }, [
193+
{ type: 'text', text: 'A red pixel.' },
194+
{ type: 'image', data: PNG_BASE64, mimeType: 'image/png' },
195+
{ type: 'audio', data: 'AAAA', mimeType: 'audio/wav' },
196+
]),
197+
})
198+
199+
const result = await client.callTool({ name: 'snapshot', arguments: {} })
200+
expect(result.isError).toBeFalsy()
201+
expect(result.content).toEqual([
202+
{ type: 'text', text: 'A red pixel.' },
203+
{ type: 'image', data: PNG_BASE64, mimeType: 'image/png' },
204+
{ type: 'audio', data: 'AAAA', mimeType: 'audio/wav' },
205+
])
206+
expect(result.structuredContent).toBeUndefined()
207+
}
208+
finally {
209+
await cleanup()
210+
}
211+
})
212+
213+
it('keeps the plain result as structured content for an rpc-backed tool with an output schema', async () => {
214+
const { ctx, client, cleanup } = await bootPair()
215+
try {
216+
ctx.rpc.register({
217+
name: 'snapshot',
218+
type: 'query',
219+
jsonSerializable: true,
220+
args: [],
221+
returns: objectSchema({ type: 'object', properties: { markdown: { type: 'string' } }, required: ['markdown'] }),
222+
agent: { description: 'Take a snapshot.' },
223+
handler: () => withMcpContent({ markdown: 'A red pixel.' }, [
224+
{ type: 'image', data: PNG_BASE64, mimeType: 'image/png' },
225+
]),
226+
} as never)
227+
228+
const listed = await client.listTools()
229+
const tool = listed.tools.find(t => t.name.endsWith('snapshot'))!
230+
expect(tool.outputSchema).toBeDefined()
231+
232+
const result = await client.callTool({ name: tool.name, arguments: {} })
233+
expect(result.isError).toBeFalsy()
234+
expect(result.content).toEqual([{ type: 'image', data: PNG_BASE64, mimeType: 'image/png' }])
235+
expect(result.structuredContent).toEqual({ markdown: 'A red pixel.' })
236+
}
237+
finally {
238+
await cleanup()
239+
}
240+
})
241+
242+
it('returns DF0081 when the attached content blocks are invalid', async () => {
243+
const { ctx, client, cleanup } = await bootPair()
244+
try {
245+
ctx.agent.registerTool({
246+
id: 'broken',
247+
description: 'Returns a bad block.',
248+
handler: () => withMcpContent({}, [{ type: 'image', data: PNG_BASE64 } as never]),
249+
})
250+
251+
const result = await client.callTool({ name: 'broken', arguments: {} })
252+
expect(result.isError).toBe(true)
253+
const content = result.content as Array<{ type: string, text: string }>
254+
expect(content[0]!.text).toContain('DF0081')
255+
}
256+
finally {
257+
await cleanup()
258+
}
259+
})
260+
169261
it('calls an rpc-backed tool whose return schema has no native converter', async () => {
170262
const { ctx, client, cleanup } = await bootPair()
171263
try {

‎packages/agentic/src/mcp/build-server.ts‎

Lines changed: 29 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
import type { Tool } from '@modelcontextprotocol/server'
22
import type { StandardSchemaV1 } from '@standard-schema/spec'
33
import type { RpcFunctionDefinitionAnyWithContext } from 'devframe/rpc'
4-
import type { AgentTool, CreateMcpServerOptions, DevframeDefinition, DevframeHost, DevframeNodeContext, McpServerHandle } from 'devframe/types'
4+
import type { AgentTool, CreateMcpServerOptions, DevframeDefinition, DevframeHost, DevframeNodeContext, McpContentBlock, McpServerHandle } from 'devframe/types'
55
import { homedir } from 'node:os'
66
import process from 'node:process'
77
import { Server } from '@modelcontextprotocol/server'
88
import { DEVFRAME_EVENTS } from 'devframe/constants'
9-
import { argsToJsonSchema, diagnostics, formatMcpError, returnToJsonSchema, stringifyForMcp } from 'devframe/internal'
9+
import { argsToJsonSchema, diagnostics, formatMcpError, getMcpContent, returnToJsonSchema, stringifyForMcp } from 'devframe/internal'
1010
import { createHostContext } from 'devframe/node'
1111
import { toAgentToolName } from 'devframe/utils/agent-tool-name'
1212
import { join } from 'pathe'
@@ -273,13 +273,12 @@ function registerToolHandlers(
273273
? usableOutputSchema(tool.outputSchema ?? computeOutputSchema(tool, ctx))
274274
: undefined
275275
const result = await ctx.agent.invoke(tool?.id ?? name, args ?? {})
276+
// Blocks attached with `withMcpContent` replace the default JSON text
277+
// block. `structuredContent` stays the plain result either way, so a
278+
// tool with an output schema still matches it.
279+
const content = mcpContentOf(name, result) ?? [{ type: 'text', text: stringifyForMcp(result) }]
276280
return {
277-
content: [
278-
{
279-
type: 'text',
280-
text: stringifyForMcp(result),
281-
},
282-
],
281+
content,
283282
...(outputSchema ? { structuredContent: result as Record<string, unknown> } : {}),
284283
}
285284
}
@@ -368,6 +367,28 @@ function registerResourceHandlers(
368367
})
369368
}
370369

370+
/**
371+
* The content blocks a tool result carries through `withMcpContent`, checked
372+
* and copied field by field, or `undefined` when it carries none. Invalid
373+
* blocks throw DF0081, which the `tools/call` handler turns into an
374+
* `isError` result.
375+
*/
376+
function mcpContentOf(name: string, result: unknown): McpContentBlock[] | undefined {
377+
const blocks = getMcpContent(result)
378+
if (blocks === undefined)
379+
return undefined
380+
if (!Array.isArray(blocks) || blocks.length === 0)
381+
throw diagnostics.DF0081({ name, reason: 'expected a non-empty array' })
382+
return blocks.map((block: unknown, index): McpContentBlock => {
383+
const b = (block ?? {}) as Record<string, unknown>
384+
if (b.type === 'text' && typeof b.text === 'string')
385+
return { type: 'text', text: b.text }
386+
if ((b.type === 'image' || b.type === 'audio') && typeof b.data === 'string' && typeof b.mimeType === 'string')
387+
return { type: b.type, data: b.data, mimeType: b.mimeType }
388+
throw diagnostics.DF0081({ name, reason: `block ${index} is not a valid text, image or audio block` })
389+
})
390+
}
391+
371392
/**
372393
* MCP constrains a tool's `outputSchema` to a JSON Schema of `type:
373394
* "object"`; clients (the SDK included) reject anything else. Non-object
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
import { describe, expect, it } from 'vitest'
2+
import { getMcpContent, withMcpContent } from '../mcp-content'
3+
4+
describe('withMcpContent', () => {
5+
const blocks = [
6+
{ type: 'text', text: 'A red pixel.' },
7+
{ type: 'image', data: 'iVBORw0KGgo=', mimeType: 'image/png' },
8+
] as const
9+
10+
it('attaches the blocks without changing the value', () => {
11+
const value = { markdown: 'A red pixel.' }
12+
const result = withMcpContent(value, blocks)
13+
expect(result).toBe(value)
14+
expect(getMcpContent(result)).toBe(blocks)
15+
expect(Object.keys(result)).toEqual(['markdown'])
16+
expect(JSON.stringify(result)).toBe('{"markdown":"A red pixel."}')
17+
expect(structuredClone(result)).toEqual({ markdown: 'A red pixel.' })
18+
expect(getMcpContent(structuredClone(result))).toBeUndefined()
19+
})
20+
21+
it('reads nothing from values without blocks', () => {
22+
expect(getMcpContent({ markdown: 'x' })).toBeUndefined()
23+
expect(getMcpContent('text')).toBeUndefined()
24+
expect(getMcpContent(null)).toBeUndefined()
25+
expect(getMcpContent(undefined)).toBeUndefined()
26+
})
27+
})
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
import type { McpContentBlock } from '../types/mcp'
2+
3+
/**
4+
* Registry symbol for the content blocks attached by {@link withMcpContent}.
5+
* `Symbol.for` keeps the key the same across duplicate copies of devframe.
6+
*/
7+
const MCP_CONTENT = Symbol.for('devframe:mcp-content')
8+
9+
/**
10+
* Attach MCP content blocks to a tool result, for example an image next to
11+
* its text. The MCP adapter sends these blocks as the `content` of the
12+
* `tools/call` result, in place of the default single JSON text block.
13+
*
14+
* The blocks are stored under a non-enumerable symbol key, so `value` itself
15+
* does not change: the RPC transport to the SPA, `returns` validation, and the
16+
* `structuredContent` of a tool with an output schema all see the plain value
17+
* and never the (often large) block data.
18+
*
19+
* `value` must be an object that is not frozen. Returns `value`.
20+
*/
21+
export function withMcpContent<T extends object>(value: T, content: readonly McpContentBlock[]): T {
22+
Object.defineProperty(value, MCP_CONTENT, {
23+
value: content,
24+
enumerable: false,
25+
configurable: true,
26+
writable: true,
27+
})
28+
return value
29+
}
30+
31+
/**
32+
* Read the blocks attached by {@link withMcpContent}, or `undefined` when the
33+
* value carries none. The value is not validated here; the MCP adapter checks
34+
* each block before it sends it.
35+
*
36+
* @internal
37+
*/
38+
export function getMcpContent(value: unknown): unknown {
39+
if (value === null || typeof value !== 'object')
40+
return undefined
41+
return (value as Record<symbol, unknown>)[MCP_CONTENT]
42+
}

‎packages/devframe/src/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,5 +2,6 @@
22
// `defineRpcFunction`) as values, plus every public type. `devframe/types`
33
// remains available as the type-only subpath, but `devframe` is the canonical
44
// import for both values and types.
5+
export { withMcpContent } from './agent/mcp-content'
56
export * from './define'
67
export type * from './types'

‎packages/devframe/src/internal/index.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,10 +39,13 @@
3939
// `formatMcpError`: the pure agent-surface projections shared between the
4040
// browser WebMCP registration and `@devframes/agentic/mcp`, so the two
4141
// surfaces cannot drift.
42+
// - `getMcpContent`: reads the content blocks that `withMcpContent` (from
43+
// `devframe`) attaches to a tool result, for `@devframes/agentic/mcp`.
4244
// - `probeDevframeOrigin`: the registry's origin-candidate probe,
4345
// `@devframes/agentic/connect` reuses it for explicit `--port` probes.
4446
export { loadAutoMcpAdapter, normalizeBasePath, resolveBasePath, resolveMcpConfig } from '../adapters/_shared'
4547
export type { ResolvedMcpConfig } from '../adapters/_shared'
48+
export { getMcpContent } from '../agent/mcp-content'
4649
export { formatMcpError, stringifyForMcp } from '../agent/stringify'
4750
export { argsToJsonSchema, returnToJsonSchema } from '../agent/to-json-schema'
4851
export { importAgenticMcp } from '../node/agentic'

0 commit comments

Comments
 (0)