From c7db627f1b4ddafa68549d8f76437eef00fd8b34 Mon Sep 17 00:00:00 2001 From: Arnav Dadarya Date: Sat, 22 Aug 2026 13:35:17 -0700 Subject: [PATCH] feat: native tool calling, end to end MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All three providers reported `supportsFunctionCalling: true` while implementing nothing, and `RespAct` ran ReAct purely by text prompting. This makes the flag honest and gives the loop a native path. Provider layer: - `LLMCallOptions` gains `tools` and `toolChoice`; `ILanguageModel` gains `chatWithTools`, returning text, tool calls, and a normalised finish reason from one turn. `BaseLM` supplies a text-only default. - Each provider translates declarations into its own request shape and reads the calls back: OpenAI `tools`/`tool_calls`, Anthropic `input_schema`/`tool_use`, Gemini `functionDeclarations`/`functionCall`. - Fixes the silent role collapse in all three converters: `tool` and `function` turns were downgraded to `user` text, and Anthropic could then merge a tool result into the preceding user turn. - Anthropic also dropped `tool_use` blocks on the floor and ignored `input_json_delta` while streaming; both are now surfaced. Module layer: - `RespAct` uses the native path when the model supports it and tools are declared, keeping the text-parsing loop as the fallback. Both paths run the same tools and emit the same events. - Tools may declare a JSON Schema or Zod schema for their arguments and receive a validated object; bare functions keep working unchanged. - Parallel tool calls in one turn are executed and reported individually. - `forceTextMode` pins a tool-capable model to the text loop. BREAKING: `ToolCall` is reshaped for cross-provider use — it was a copy of OpenAI's encoding that no other provider could populate faithfully. It is now `{ id?, name, arguments, rawArguments? }` with `arguments` always a parsed object. The dead `ChatMessage.functionCall` is removed; `ChatMessage` gains `toolCallId`. Co-Authored-By: Claude Opus 5 --- .changeset/native-tool-calling.md | 45 ++ README.md | 23 + packages/anthropic/src/anthropic-lm.test.ts | 204 ++++++++ packages/anthropic/src/anthropic-lm.ts | 176 ++++++- packages/core/src/core/base-lm.ts | 14 + packages/core/src/index.ts | 9 +- packages/core/src/modules/respact.test.ts | 537 ++++++++++++++++++++ packages/core/src/modules/respact.ts | 497 ++++++++++++++++-- packages/core/src/types/language-model.ts | 77 ++- packages/gemini/src/gemini-lm.test.ts | 149 ++++++ packages/gemini/src/gemini-lm.ts | 163 +++++- packages/openai/src/openai-lm.test.ts | 213 ++++++++ packages/openai/src/openai-lm.ts | 136 ++++- site/docs.html | 79 ++- 14 files changed, 2260 insertions(+), 62 deletions(-) create mode 100644 .changeset/native-tool-calling.md diff --git a/.changeset/native-tool-calling.md b/.changeset/native-tool-calling.md new file mode 100644 index 0000000..646f17a --- /dev/null +++ b/.changeset/native-tool-calling.md @@ -0,0 +1,45 @@ +--- +'@ts-dspy/anthropic': minor +'@ts-dspy/gemini': minor +'@ts-dspy/openai': minor +'@ts-dspy/core': minor +--- + +Native tool calling, end to end. + +All three providers reported `supportsFunctionCalling: true` while implementing +nothing, and `RespAct` ran ReAct purely by text prompting — regex-extracting +`Action:`/`Action Input:` from raw completions. That capped every tool at exactly +one string argument, ruled out parallel calls, and left the loop at the mercy of +the model formatting its output correctly. The flag is now honest. + +`LLMCallOptions` gains `tools` and `toolChoice`, and `ILanguageModel` gains +`chatWithTools`, which returns text, tool calls, and a normalised finish reason +from one turn. `BaseLM` supplies a text-only default, so the capability flag — +not feature detection — is what callers branch on. Each provider translates the +declarations into its own request shape (OpenAI `tools`/`tool_calls`, Anthropic +`input_schema`/`tool_use`, Gemini `functionDeclarations`/`functionCall`) and +reads the calls back out. + +`RespAct` uses that path whenever the model supports it and tools are declared, +and keeps the text-parsing loop as the fallback for local models and providers +without native tool calling — the same task completes either way. Tools can now +declare a JSON Schema or Zod schema for their arguments and receive a validated +object instead of a single string; bare functions and `{ description, function }` +keep working unchanged. Parallel tool calls in one turn are executed and reported +individually, and the whole `RespActEvent` surface stays meaningful on both +paths. `forceTextMode` pins a tool-capable model to the text loop. + +**Breaking:** `ToolCall` is reshaped for cross-provider use. It was a copy of +OpenAI's encoding — a required `id`, a `type: 'function'` literal, and a nested +`function.arguments` JSON *string* — which no other provider can populate +faithfully. It is now `{ id?, name, arguments, rawArguments? }`, where +`arguments` is always a parsed object and `id` is optional because Gemini's +function calls have none. The dead `ChatMessage.functionCall` field is removed; +`ChatMessage` gains `toolCallId` to correlate a tool result with its call. + +That correlation also fixes a silent role collapse in all three converters: +`tool` and `function` turns were downgraded to `user` text, and Anthropic could +then merge a tool result into the preceding user turn. Anthropic additionally +dropped `tool_use` blocks on the floor (`textOf` keeps only `text` blocks) and +ignored `input_json_delta` while streaming; both are now surfaced. diff --git a/README.md b/README.md index aa2eed0..a080929 100644 --- a/README.md +++ b/README.md @@ -180,6 +180,29 @@ const agent = new RespAct(AnswerQuestion, { }); ``` +Give a tool a `parameters` schema — JSON Schema or Zod — and it takes named, +validated arguments instead of one string: + +```ts +const agent = new RespAct(AnswerQuestion, { + tools: { + flights: { + description: 'Find flights between two airports on a date.', + parameters: z.object({ from: z.string(), to: z.string(), date: z.string() }), + function: ({ from, to, date }) => search(from, to, date), + }, + }, +}); +``` + +`RespAct` picks its execution path from the model's capabilities. Against a +provider reporting `supportsFunctionCalling: true` — all three of ours do — tools +are declared in the request and the model's calls come back as structured data, +so several tools can run in one turn. Against anything else, the loop falls back +to prompting for `Action:` / `Action Input:` and parsing the reply, which works on +any completion model. Both paths run the same tools and emit the same events; pass +`forceTextMode: true` to pin a tool-capable model to the text loop. + Tool descriptions are what the model uses to decide when to call each tool, so they earn the detail. Never pass model output to `eval()` — see [`examples/utils.ts`](examples/utils.ts) for a bounded arithmetic evaluator. diff --git a/packages/anthropic/src/anthropic-lm.test.ts b/packages/anthropic/src/anthropic-lm.test.ts index b56acf2..ddd47c3 100644 --- a/packages/anthropic/src/anthropic-lm.test.ts +++ b/packages/anthropic/src/anthropic-lm.test.ts @@ -235,6 +235,133 @@ describe('AnthropicLM', () => { usage: { promptTokens: 14, completionTokens: 9, totalTokens: 23 }, }); }); + + it('forwards tool-argument deltas and the assembled calls', async () => { + const events = [ + { type: 'content_block_delta', delta: { type: 'text_delta', text: 'ok' } }, + { + type: 'content_block_delta', + index: 1, + delta: { type: 'input_json_delta', partial_json: '{"a":' }, + }, + { + type: 'content_block_delta', + index: 1, + delta: { type: 'input_json_delta', partial_json: '1}' }, + }, + ]; + mocks.stream.mockReturnValue({ + async *[Symbol.asyncIterator]() { + yield* events; + }, + finalMessage: async () => + message('ok', { + content: [ + { type: 'text', text: 'ok' }, + { type: 'tool_use', id: 'toolu_1', name: 'add', input: { a: 1 } }, + ], + stop_reason: 'tool_use', + }), + }); + + const chunks = []; + for await (const chunk of new AnthropicLM({ apiKey: 'k' }).generateStream('Hi')) { + chunks.push(chunk); + } + + // Argument fragments are not text, so they travel in metadata. + expect(chunks.filter((c) => !c.done).map((c) => c.content)).toEqual(['ok', '', '']); + expect(chunks[1].metadata).toEqual({ + toolInputDelta: { index: 1, partialJson: '{"a":' }, + }); + expect(chunks.at(-1)?.metadata).toEqual({ + toolCalls: [{ id: 'toolu_1', name: 'add', arguments: { a: 1 } }], + }); + }); + }); + + describe('tool calling', () => { + it('sends tool declarations as name/description/input_schema', async () => { + mocks.create.mockResolvedValue(message('ok')); + + await new AnthropicLM({ apiKey: 'k' }).chatWithTools( + [{ role: 'user', content: 'hi' }], + { + tools: [ + { + name: 'add', + description: 'Add two numbers', + parameters: { type: 'object', properties: {} }, + }, + ], + } + ); + + expect(mocks.create.mock.calls[0][0].tools).toEqual([ + { + name: 'add', + description: 'Add two numbers', + input_schema: { type: 'object', properties: {} }, + }, + ]); + }); + + it('spells a forced tool call as tool_choice any', async () => { + mocks.create.mockResolvedValue(message('ok')); + + await new AnthropicLM({ apiKey: 'k' }).chatWithTools( + [{ role: 'user', content: 'hi' }], + { tools: [{ name: 'add', parameters: {} }], toolChoice: 'required' } + ); + + expect(mocks.create.mock.calls[0][0].tool_choice).toEqual({ type: 'any' }); + }); + + it('names a specific tool when the choice is an object', async () => { + mocks.create.mockResolvedValue(message('ok')); + + await new AnthropicLM({ apiKey: 'k' }).chatWithTools( + [{ role: 'user', content: 'hi' }], + { tools: [{ name: 'add', parameters: {} }], toolChoice: { name: 'add' } } + ); + + expect(mocks.create.mock.calls[0][0].tool_choice).toEqual({ + type: 'tool', + name: 'add', + }); + }); + + it('surfaces tool_use blocks alongside the text', async () => { + mocks.create.mockResolvedValue( + message('ignored', { + content: [ + { type: 'text', text: 'Let me add those.' }, + { type: 'tool_use', id: 'toolu_1', name: 'add', input: { a: 1, b: 2 } }, + ], + stop_reason: 'tool_use', + }) + ); + + const result = await new AnthropicLM({ apiKey: 'k' }).chatWithTools([ + { role: 'user', content: 'hi' }, + ]); + + // textOf() keeps only text blocks; the tool_use block used to be + // silently discarded here with nothing to replace it. + expect(result.content).toBe('Let me add those.'); + expect(result.finishReason).toBe('tool_calls'); + expect(result.toolCalls).toEqual([ + { id: 'toolu_1', name: 'add', arguments: { a: 1, b: 2 } }, + ]); + }); + + it('omits tool parameters entirely when no tools are offered', async () => { + mocks.create.mockResolvedValue(message('ok')); + await new AnthropicLM({ apiKey: 'k' }).chat([{ role: 'user', content: 'hi' }]); + + expect(mocks.create.mock.calls[0][0]).not.toHaveProperty('tools'); + expect(mocks.create.mock.calls[0][0]).not.toHaveProperty('tool_choice'); + }); }); describe('toAnthropicMessages', () => { @@ -270,6 +397,83 @@ describe('AnthropicLM', () => { expect(input).toHaveLength(2); }); + it('turns an assistant tool call into text and tool_use blocks', () => { + const { messages } = toAnthropicMessages([ + { + role: 'assistant', + content: 'Let me add those.', + toolCalls: [{ id: 'toolu_1', name: 'add', arguments: { a: 1, b: 2 } }], + }, + ]); + + expect(messages).toEqual([ + { + role: 'assistant', + content: [ + { type: 'text', text: 'Let me add those.' }, + { type: 'tool_use', id: 'toolu_1', name: 'add', input: { a: 1, b: 2 } }, + ], + }, + ]); + }); + + it('gives parallel calls to one tool distinct synthesized ids', () => { + // A Gemini-sourced turn carries no ids at all; two calls to the same + // tool must not collapse onto one tool_use_id. + const { messages } = toAnthropicMessages([ + { + role: 'assistant', + content: '', + toolCalls: [ + { name: 'lookup', arguments: { id: 1 } }, + { name: 'lookup', arguments: { id: 2 } }, + ], + }, + ]); + + const ids = (messages[0].content as any[]).map((block) => block.id); + expect(new Set(ids).size).toBe(2); + }); + + it('turns a tool result into a user turn holding a tool_result block', () => { + const { messages } = toAnthropicMessages([ + { role: 'tool', name: 'add', toolCallId: 'toolu_1', content: '3' }, + ]); + + expect(messages).toEqual([ + { + role: 'user', + content: [{ type: 'tool_result', tool_use_id: 'toolu_1', content: '3' }], + }, + ]); + }); + + it('merges a tool result onto a preceding user turn as blocks, not text', () => { + // String merging is still the behaviour for two plain user turns + // (see above), but concatenating a tool result into prose would + // destroy the `tool_use_id` the API correlates on, so a mixed pair + // is promoted to block form instead. + const { messages } = toAnthropicMessages([ + { role: 'user', content: 'context' }, + { role: 'tool', name: 'add', toolCallId: 'toolu_1', content: '3' }, + ]); + + expect(messages).toEqual([ + { + role: 'user', + content: [ + { type: 'text', text: 'context' }, + { type: 'tool_result', tool_use_id: 'toolu_1', content: '3' }, + ], + }, + ]); + }); + + it('degrades an uncorrelated tool result to plain user text', () => { + const { messages } = toAnthropicMessages([{ role: 'tool', content: '3' }]); + expect(messages).toEqual([{ role: 'user', content: '3' }]); + }); + it('passes the system parameter through on a chat call', async () => { mocks.create.mockResolvedValue(message('ok')); await new AnthropicLM({ apiKey: 'k' }).chat([ diff --git a/packages/anthropic/src/anthropic-lm.ts b/packages/anthropic/src/anthropic-lm.ts index 8d9b8c5..99f0650 100644 --- a/packages/anthropic/src/anthropic-lm.ts +++ b/packages/anthropic/src/anthropic-lm.ts @@ -2,12 +2,19 @@ import { BaseLM, LMError, type ChatMessage, + type ChatResult, + type FinishReason, type LLMCallOptions, type ModelCapabilities, type StreamChunk, + type ToolCall, } from '@ts-dspy/core'; import Anthropic, { APIError } from '@anthropic-ai/sdk'; -import type { Message, MessageParam } from '@anthropic-ai/sdk/resources/messages'; +import type { + ContentBlockParam, + Message, + MessageParam, +} from '@anthropic-ai/sdk/resources/messages'; /** Current Claude Opus. Model IDs are exact — never append a date suffix. */ export const DEFAULT_ANTHROPIC_MODEL = 'claude-opus-5'; @@ -69,6 +76,13 @@ export class AnthropicLM extends BaseLM { } async chat(messages: ChatMessage[], options?: LLMCallOptions): Promise { + return (await this.chatWithTools(messages, options)).content; + } + + async chatWithTools( + messages: ChatMessage[], + options?: LLMCallOptions + ): Promise { const { system, messages: converted } = toAnthropicMessages(messages); const startedAt = Date.now(); @@ -81,6 +95,7 @@ export class AnthropicLM extends BaseLM { messages: converted, ...(system ? { system } : {}), ...samplingParams(options), + ...toolParams(options), }, requestOptions(options) ); @@ -96,7 +111,13 @@ export class AnthropicLM extends BaseLM { }); this.assertNotRefused(message); - return textOf(message); + + const toolCalls = toolCallsOf(message); + return { + content: textOf(message), + ...(toolCalls.length > 0 ? { toolCalls } : {}), + finishReason: finishReasonOf(message.stop_reason), + }; } async generateStructured( @@ -176,14 +197,32 @@ export class AnthropicLM extends BaseLM { messages: converted, ...(system ? { system } : {}), ...samplingParams(options), + ...toolParams(options), }, requestOptions(options) ); try { for await (const event of stream) { - if (event.type === 'content_block_delta' && event.delta.type === 'text_delta') { + if (event.type !== 'content_block_delta') continue; + + if (event.delta.type === 'text_delta') { yield { content: event.delta.text, done: false }; + } else if (event.delta.type === 'input_json_delta') { + // Tool arguments stream as JSON fragments on their own event + // type. They are not text, so they travel in metadata rather + // than being spliced into `content`; the assembled calls also + // arrive whole on the final chunk. + yield { + content: '', + done: false, + metadata: { + toolInputDelta: { + index: event.index, + partialJson: event.delta.partial_json, + }, + }, + }; } } @@ -195,9 +234,11 @@ export class AnthropicLM extends BaseLM { }); this.assertNotRefused(final); + const toolCalls = toolCallsOf(final); yield { content: '', done: true, + ...(toolCalls.length > 0 ? { metadata: { toolCalls } } : {}), usage: { promptTokens: final.usage?.input_tokens ?? 0, completionTokens: final.usage?.output_tokens ?? 0, @@ -239,6 +280,13 @@ export class AnthropicLM extends BaseLM { } } +/** + * The assistant's prose. + * + * This deliberately keeps only `text` blocks: `tool_use` blocks are not text and + * are surfaced separately by {@link toolCallsOf}, so a tool-calling turn returns + * whatever the model said alongside the call rather than a JSON blob. + */ function textOf(message: Message): string { return message.content .filter( @@ -248,12 +296,36 @@ function textOf(message: Message): string { .join(''); } +/** Extract the `tool_use` blocks a turn requested. Anthropic sends `input` already parsed. */ +function toolCallsOf(message: Message): ToolCall[] { + return message.content + .filter( + (block): block is Extract => + block.type === 'tool_use' + ) + .map((block) => ({ + id: block.id, + name: block.name, + arguments: + block.input && typeof block.input === 'object' && !Array.isArray(block.input) + ? (block.input as Record) + : {}, + })); +} + /** * Convert ts-dspy messages into the Messages API shape. * * System messages become the top-level `system` parameter — Anthropic has no * system role inside `messages`. Consecutive same-role turns are merged, since * the API requires strict alternation. + * + * Tool traffic is structural rather than textual: an assistant turn carrying + * tool calls becomes `text` + `tool_use` blocks, and a `tool` result turn becomes + * a user turn holding a `tool_result` block keyed by `tool_use_id`. Merging + * therefore happens at the block level whenever either side is block-shaped — + * concatenating a tool result onto a plain user string, as the previous + * implementation did, would have destroyed the correlation the API needs. */ export function toAnthropicMessages(messages: ChatMessage[]): { system?: string; @@ -269,12 +341,18 @@ export function toAnthropicMessages(messages: ChatMessage[]): { } const role: 'user' | 'assistant' = message.role === 'assistant' ? 'assistant' : 'user'; + const content = toAnthropicContent(message); const previous = converted.at(-1); - if (previous?.role === role && typeof previous.content === 'string') { - previous.content = `${previous.content}\n\n${message.content}`; + if (previous?.role !== role) { + converted.push({ role, content }); + continue; + } + + if (typeof previous.content === 'string' && typeof content === 'string') { + previous.content = `${previous.content}\n\n${content}`; } else { - converted.push({ role, content: message.content }); + previous.content = [...asBlocks(previous.content), ...asBlocks(content)]; } } @@ -284,6 +362,92 @@ export function toAnthropicMessages(messages: ChatMessage[]): { }; } +/** A plain turn stays a string; anything carrying tool traffic becomes blocks. */ +function toAnthropicContent(message: ChatMessage): string | ContentBlockParam[] { + if (message.role === 'assistant' && message.toolCalls?.length) { + const blocks: ContentBlockParam[] = []; + if (message.content) { + blocks.push({ type: 'text', text: message.content }); + } + message.toolCalls.forEach((call, index) => { + blocks.push({ + type: 'tool_use', + // Anthropic requires an id; a call relayed from a provider + // without one (Gemini) gets a placeholder. The position is part + // of it because two parallel calls to the same tool would + // otherwise share an id, mispairing their results. + id: call.id ?? `toolu_${index}_${call.name}`, + name: call.name, + input: call.arguments ?? {}, + }); + }); + return blocks; + } + + if (message.role === 'tool' || message.role === 'function') { + // Without a `tool_use_id` there is nothing to correlate against, so the + // result degrades to ordinary user text rather than a rejected request. + if (!message.toolCallId) return message.content; + return [ + { + type: 'tool_result', + tool_use_id: message.toolCallId, + content: message.content, + }, + ]; + } + + return message.content; +} + +function asBlocks(content: string | ContentBlockParam[]): ContentBlockParam[] { + if (typeof content !== 'string') return content; + return content ? [{ type: 'text', text: content }] : []; +} + +/** Translate tool declarations into the Messages API request shape. */ +function toolParams(options?: LLMCallOptions): Record { + if (!options?.tools?.length) return {}; + + const params: Record = { + tools: options.tools.map((tool) => ({ + name: tool.name, + ...(tool.description ? { description: tool.description } : {}), + input_schema: tool.parameters, + })), + }; + + const choice = options.toolChoice; + if (choice !== undefined) { + if (typeof choice === 'object') { + params.tool_choice = { type: 'tool', name: choice.name }; + } else if (choice === 'required') { + // Anthropic spells "you must call some tool" as `any`. + params.tool_choice = { type: 'any' }; + } else { + params.tool_choice = { type: choice }; + } + } + + return params; +} + +function finishReasonOf(reason: Message['stop_reason']): FinishReason { + switch (reason) { + case 'end_turn': + case 'stop_sequence': + return 'stop'; + case 'tool_use': + return 'tool_calls'; + case 'max_tokens': + return 'length'; + case 'refusal': + return 'content_filter'; + default: + return 'other'; + } +} + /** * Build sampling parameters. * diff --git a/packages/core/src/core/base-lm.ts b/packages/core/src/core/base-lm.ts index a1b36ed..91e80b2 100644 --- a/packages/core/src/core/base-lm.ts +++ b/packages/core/src/core/base-lm.ts @@ -1,5 +1,6 @@ import type { ChatMessage, + ChatResult, ILanguageModel, LLMCallOptions, ModelCapabilities, @@ -38,6 +39,19 @@ export abstract class BaseLM implements ILanguageModel { return this.chat([{ role: 'user', content: prompt }], options); } + /** + * Default tool-calling implementation: run the turn as plain chat and report + * no tool calls. Providers with native tool calling override this; a provider + * that does must also report `supportsFunctionCalling: true`, since that flag + * is what callers such as `RespAct` branch on. + */ + async chatWithTools( + messages: ChatMessage[], + options?: LLMCallOptions + ): Promise { + return { content: await this.chat(messages, options), finishReason: 'stop' }; + } + /** * Default structured-output implementation: ask for JSON in the prompt and * parse the reply. Providers with a native JSON-schema mode should override diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 6911b30..9765cfd 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -17,7 +17,14 @@ export type { FieldValidationIssue } from './core/errors'; export { Predict } from './modules/predict'; export { ChainOfThought } from './modules/chain-of-thought'; export { RespAct } from './modules/respact'; -export type { ToolFunction, ToolWithDescription, ToolDefinition } from './modules/respact'; +export type { + ToolFunction, + ToolWithDescription, + ToolDefinition, + ToolParameterSchema, + RespActEvent, + RespActOptions, +} from './modules/respact'; // Utilities export { buildPrompt, parseOutput } from './utils/parsing'; diff --git a/packages/core/src/modules/respact.test.ts b/packages/core/src/modules/respact.test.ts index 1c92b83..cf3850f 100644 --- a/packages/core/src/modules/respact.test.ts +++ b/packages/core/src/modules/respact.test.ts @@ -1,3 +1,4 @@ +import { z } from 'zod'; import { RespAct, type ToolFunction, @@ -5,7 +6,57 @@ import { type RespActEvent, } from './respact'; import { Signature, OutputField, InputField } from '../core/signature'; +import { BaseLM } from '../core/base-lm'; import { MockLM } from '../test-utils'; +import type { + ChatMessage, + ChatResult, + LLMCallOptions, + ModelCapabilities, +} from '../types/language-model'; + +/** + * A language model that advertises native tool calling and replies from a + * script of `ChatResult`s. `MockLM` deliberately reports + * `supportsFunctionCalling: false`, which is what keeps the rest of this suite + * on the text path. + */ +class ToolCallingLM extends BaseLM { + readonly turns: ChatMessage[][] = []; + readonly options: Array = []; + + constructor(private readonly script: ChatResult[]) { + super('fake', 'fake-model'); + } + + async chat(messages: ChatMessage[], options?: LLMCallOptions): Promise { + return (await this.chatWithTools(messages, options)).content; + } + + async chatWithTools( + messages: ChatMessage[], + options?: LLMCallOptions + ): Promise { + // Snapshot: RespAct appends to the same array across turns. + this.turns.push(messages.map((message) => ({ ...message }))); + this.options.push(options); + if (this.script.length === 0) { + throw new Error('ToolCallingLM: no more scripted results'); + } + return this.script.shift()!; + } + + getCapabilities(): ModelCapabilities { + return { + supportsStreaming: false, + supportsStructuredOutput: false, + supportsFunctionCalling: true, + supportsVision: false, + maxContextLength: 8192, + supportedFormats: ['text'], + }; + } +} // These tests drive the real Module base class and the real parser. The previous // suite mocked both, so it could not catch parsing or validation regressions. @@ -292,4 +343,490 @@ describe('RespAct', () => { expect(types).toContain('tool_result'); }); }); + + describe('typed tools', () => { + it('derives a JSON Schema from a Zod argument schema', () => { + const agent = new RespAct('question -> answer', { + lm: new MockLM(), + tools: { + add: { + description: 'Add two numbers', + parameters: z.object({ a: z.number(), b: z.number() }), + function: ({ a, b }: { a: number; b: number }) => a + b, + }, + }, + }); + + const schema = (agent as any).tools.add.parameters; + expect(schema.type).toBe('object'); + expect(Object.keys(schema.properties)).toEqual(['a', 'b']); + expect(schema.required).toEqual(['a', 'b']); + }); + + it('synthesizes a single-string schema for an untyped tool', () => { + const agent = new RespAct('question -> answer', { + lm: new MockLM(), + tools: { echo: (input: string) => input }, + }); + + const schema = (agent as any).tools.echo.parameters; + expect(Object.keys(schema.properties)).toEqual(['input']); + expect((agent as any).tools.echo.typed).toBe(false); + }); + + it('accepts a raw JSON Schema without wrapping it in Zod', () => { + const parameters = { + type: 'object', + properties: { city: { type: 'string' } }, + required: ['city'], + }; + const agent = new RespAct('question -> answer', { + lm: new MockLM(), + tools: { + weather: { description: 'Weather', parameters, function: () => 'sunny' }, + }, + }); + + expect((agent as any).tools.weather.parameters).toBe(parameters); + expect((agent as any).tools.weather.validator).toBeUndefined(); + }); + + it('parses a JSON Action Input into named arguments in text mode', async () => { + const add = vi.fn(({ a, b }: { a: number; b: number }) => a + b); + const agent = new RespAct('question -> answer', { + lm: new MockLM(), + tools: { + add: { + description: 'Add two numbers', + parameters: z.object({ a: z.number(), b: z.number() }), + function: add, + }, + }, + }); + + const result = await (agent as any).executeTool('add', '{"a": 2, "b": 3}', 0); + + expect(add).toHaveBeenCalledWith({ a: 2, b: 3 }); + expect(result).toBe('5'); + }); + + it('accepts a bare value in text mode when the schema has one property', async () => { + const shout = vi.fn(({ text }: { text: string }) => text.toUpperCase()); + const agent = new RespAct('question -> answer', { + lm: new MockLM(), + tools: { + shout: { + description: 'Uppercase text', + parameters: z.object({ text: z.string() }), + function: shout, + }, + }, + }); + + expect(await (agent as any).executeTool('shout', 'hello', 0)).toBe('HELLO'); + expect(shout).toHaveBeenCalledWith({ text: 'hello' }); + }); + + it('turns a Zod validation failure into a correctable observation', async () => { + const add = vi.fn(); + const events: RespActEvent[] = []; + const agent = new RespAct('question -> answer', { + lm: new MockLM(), + onEvent: (event) => events.push(event), + tools: { + add: { + description: 'Add two numbers', + parameters: z.object({ a: z.number(), b: z.number() }), + function: add, + }, + }, + }); + + const result = await (agent as any).executeTool('add', '{"a": 2}', 0); + + expect(add).not.toHaveBeenCalled(); + expect(result).toContain('Invalid arguments'); + expect(result).toContain('b'); + expect(events.some((event) => event.type === 'tool_error')).toBe(true); + }); + + it('coerces a bare value to the declared scalar type in text mode', async () => { + const double = vi.fn(({ n }: { n: number }) => n * 2); + const agent = new RespAct('question -> answer', { + lm: new MockLM(), + tools: { + double: { + description: 'Double a number', + parameters: z.object({ n: z.number() }), + function: double, + }, + }, + }); + + // The text loop only ever produces a line of text, so a numeric + // argument would otherwise fail validation on every attempt. + expect(await (agent as any).executeTool('double', '21', 0)).toBe('42'); + expect(double).toHaveBeenCalledWith({ n: 21 }); + }); + + it('lists the required argument keys in the text-mode prompt', () => { + const agent = new RespAct('question -> answer', { + lm: new MockLM(), + tools: { + add: { + description: 'Add two numbers', + parameters: z.object({ a: z.number(), b: z.number() }), + function: () => 0, + }, + }, + }); + + const prompt = (agent as any).buildInitialPrompt({ question: 'Q' }); + expect(prompt).toContain('Action Input must be a JSON object with keys: a, b'); + }); + }); + + describe('native tool calling', () => { + it('calls a multi-argument tool and returns the final answer', async () => { + const add = vi.fn(({ a, b }: { a: number; b: number }) => a + b); + const lm = new ToolCallingLM([ + { + content: 'Let me add those.', + toolCalls: [{ id: 'call_1', name: 'add', arguments: { a: 6, b: 7 } }], + finishReason: 'tool_calls', + }, + { content: 'The result is 13', finishReason: 'stop' }, + ]); + + const agent = new RespAct('question -> answer', { + lm, + tools: { + add: { + description: 'Add two numbers', + parameters: z.object({ a: z.number(), b: z.number() }), + function: add, + }, + }, + }); + + const result = await agent.forward({ question: 'What is 6 plus 7?' }); + + expect(add).toHaveBeenCalledWith({ a: 6, b: 7 }); + expect(result.answer).toBe('The result is 13'); + expect(result.steps).toBe(2); + }); + + it('advertises every tool schema on each request', async () => { + const lm = new ToolCallingLM([{ content: 'done', finishReason: 'stop' }]); + const agent = new RespAct('question -> answer', { + lm, + tools: { + add: { + description: 'Add two numbers', + parameters: z.object({ a: z.number(), b: z.number() }), + function: () => 0, + }, + }, + }); + + await agent.forward({ question: 'Q' }); + + expect(lm.options[0]?.tools).toEqual([ + { + name: 'add', + description: 'Add two numbers', + parameters: expect.objectContaining({ type: 'object' }), + }, + ]); + }); + + it('records the tool result as a correlated tool turn', async () => { + const lm = new ToolCallingLM([ + { + content: '', + toolCalls: [{ id: 'call_9', name: 'echo', arguments: { input: 'hi' } }], + }, + { content: 'hi', finishReason: 'stop' }, + ]); + const agent = new RespAct('question -> answer', { + lm, + tools: { echo: (input: string) => input }, + }); + + await agent.forward({ question: 'Q' }); + + const secondTurn = lm.turns[1]; + expect(secondTurn.at(-2)).toMatchObject({ + role: 'assistant', + toolCalls: [{ id: 'call_9', name: 'echo' }], + }); + expect(secondTurn.at(-1)).toEqual({ + role: 'tool', + name: 'echo', + toolCallId: 'call_9', + content: 'hi', + }); + }); + + it('passes a bare string through to an untyped tool', async () => { + const echo = vi.fn((input: string) => `echo:${input}`); + const lm = new ToolCallingLM([ + { + content: '', + toolCalls: [{ id: 'c1', name: 'echo', arguments: { input: 'hi' } }], + }, + { content: 'echo:hi', finishReason: 'stop' }, + ]); + const agent = new RespAct('question -> answer', { lm, tools: { echo } }); + + await agent.forward({ question: 'Q' }); + + expect(echo).toHaveBeenCalledWith('hi'); + }); + + it('runs parallel tool calls from a single turn', async () => { + const lm = new ToolCallingLM([ + { + content: '', + toolCalls: [ + { id: 'c1', name: 'left', arguments: { input: 'a' } }, + { id: 'c2', name: 'right', arguments: { input: 'b' } }, + ], + }, + { content: 'ab', finishReason: 'stop' }, + ]); + const left = vi.fn((input: string) => `L${input}`); + const right = vi.fn((input: string) => `R${input}`); + + const agent = new RespAct('question -> answer', { lm, tools: { left, right } }); + await agent.forward({ question: 'Q' }); + + expect(left).toHaveBeenCalledWith('a'); + expect(right).toHaveBeenCalledWith('b'); + const toolTurns = lm.turns[1].filter((message) => message.role === 'tool'); + expect(toolTurns.map((message) => message.content)).toEqual(['La', 'Rb']); + }); + + it('does not repeat an identical native tool call', async () => { + const fetchTool = vi.fn(() => 'data'); + const lm = new ToolCallingLM([ + { content: '', toolCalls: [{ id: 'c1', name: 'fetch', arguments: { q: 1 } }] }, + { content: '', toolCalls: [{ id: 'c2', name: 'fetch', arguments: { q: 1 } }] }, + { content: 'done', finishReason: 'stop' }, + ]); + const events: RespActEvent[] = []; + + const agent = new RespAct('question -> answer', { + lm, + onEvent: (event) => events.push(event), + tools: { + fetch: { + description: 'Fetch', + parameters: z.object({ q: z.number() }), + function: fetchTool, + }, + }, + }); + + const result = await agent.forward({ question: 'Q' }); + + expect(fetchTool).toHaveBeenCalledTimes(1); + expect(result.answer).toBe('done'); + expect(events.some((event) => event.type === 'repeated_tool_call')).toBe(true); + }); + + it('emits the same event surface as the text loop', async () => { + const lm = new ToolCallingLM([ + { + content: 'thinking', + toolCalls: [{ id: 'c1', name: 'echo', arguments: { input: 'hi' } }], + }, + { content: 'hi', finishReason: 'stop' }, + ]); + const events: RespActEvent[] = []; + + const agent = new RespAct('question -> answer', { + lm, + tools: { echo: (input: string) => input }, + onEvent: (event) => events.push(event), + }); + + await agent.forward({ question: 'Q' }); + + // The final turn's prose is a thought too, exactly as in text mode. + expect(events.map((event) => event.type)).toEqual([ + 'thought', + 'tool_call', + 'tool_result', + 'thought', + ]); + expect(events[1]).toMatchObject({ tool: 'echo', input: 'hi' }); + }); + + it('asks again when the final answer is missing a required field', async () => { + class Scored extends Signature { + @OutputField({ description: 'the answer' }) + answer!: string; + + @OutputField({ description: 'confidence', type: 'number' }) + confidence!: number; + } + + const lm = new ToolCallingLM([ + { content: 'answer: Paris', finishReason: 'stop' }, + { content: 'answer: Paris\nconfidence: 0.8', finishReason: 'stop' }, + ]); + const agent = new RespAct(Scored, { + lm, + tools: { echo: (input: string) => input }, + }); + + const result = await agent.forward({ question: 'Capital of France?' }); + + expect(result.confidence).toBe(0.8); + const nudge = lm.turns[1].at(-1); + expect(nudge?.role).toBe('user'); + expect(nudge?.content).toContain('missing or malformed for: confidence'); + }); + + it('surfaces a failing tool as an observation rather than throwing', async () => { + const lm = new ToolCallingLM([ + { + content: '', + toolCalls: [{ id: 'c1', name: 'boom', arguments: { input: 'x' } }], + }, + { content: 'recovered', finishReason: 'stop' }, + ]); + const agent = new RespAct('question -> answer', { + lm, + tools: { + boom: () => { + throw new Error('kaboom'); + }, + }, + }); + + const result = await agent.forward({ question: 'Q' }); + + expect(result.answer).toBe('recovered'); + expect(lm.turns[1].at(-1)?.content).toContain('Error executing boom: kaboom'); + }); + + it('falls back to the text loop when the model cannot call tools', async () => { + const add = vi.fn(({ a, b }: { a: number; b: number }) => a + b); + const lm = new MockLM({ + responses: [ + 'Action: add\nAction Input: {"a": 6, "b": 7}', + 'Final Answer: answer: The result is 13', + ], + }); + + const agent = new RespAct('question -> answer', { + lm, + tools: { + add: { + description: 'Add two numbers', + parameters: z.object({ a: z.number(), b: z.number() }), + function: add, + }, + }, + }); + + const result = await agent.forward({ question: 'What is 6 plus 7?' }); + + expect(add).toHaveBeenCalledWith({ a: 6, b: 7 }); + expect(result.answer).toBe('The result is 13'); + }); + + it('stays on the text loop when the model inherited chatWithTools', async () => { + // The flag existed long before anything read it, so a custom LM can + // report it while inheriting BaseLM's text-only `chatWithTools`. + // Taking the native path there would silently never call a tool. + const lm = new MockLM({ + capabilities: { supportsFunctionCalling: true }, + responses: ['Action: echo\nAction Input: hi', 'Final Answer: answer: hi'], + }); + const echo = vi.fn((input: string) => input); + const agent = new RespAct('question -> answer', { lm, tools: { echo } }); + + const result = await agent.forward({ question: 'Q' }); + + expect(echo).toHaveBeenCalledWith('hi'); + expect(result.answer).toBe('hi'); + }); + + it('does not record an empty assistant turn when the model says nothing', async () => { + const lm = new ToolCallingLM([ + { content: '', finishReason: 'stop' }, + { content: 'done', finishReason: 'stop' }, + ]); + const agent = new RespAct('question -> answer', { + lm, + tools: { echo: (input: string) => input }, + }); + + await agent.forward({ question: 'Q' }); + + // Providers reject a non-final assistant turn with empty content, + // so the nudge goes in on its own. + expect(lm.turns[1].some((message) => message.role === 'assistant')).toBe(false); + expect(lm.turns[1].at(-1)?.role).toBe('user'); + }); + + it('reports unparseable provider arguments instead of running the tool', async () => { + const echo = vi.fn((input: string) => input); + const lm = new ToolCallingLM([ + { + content: '', + toolCalls: [ + { + id: 'c1', + name: 'echo', + arguments: {}, + rawArguments: '{not json', + }, + ], + }, + { content: 'recovered', finishReason: 'stop' }, + ]); + const agent = new RespAct('question -> answer', { lm, tools: { echo } }); + + await agent.forward({ question: 'Q' }); + + expect(echo).not.toHaveBeenCalled(); + expect(lm.turns[1].at(-1)?.content).toContain('not valid JSON'); + }); + + it('rejects a non-string argument for an untyped tool', async () => { + const echo = vi.fn((input: string) => input); + const lm = new ToolCallingLM([ + { content: '', toolCalls: [{ id: 'c1', name: 'echo', arguments: { q: 1 } }] }, + { content: 'recovered', finishReason: 'stop' }, + ]); + const agent = new RespAct('question -> answer', { lm, tools: { echo } }); + + await agent.forward({ question: 'Q' }); + + expect(echo).not.toHaveBeenCalled(); + expect(lm.turns[1].at(-1)?.content).toContain('Expected a single string argument'); + }); + + it('honours forceTextMode on a tool-calling model', async () => { + const lm = new ToolCallingLM([ + { content: ' Final Answer: answer: done', finishReason: 'stop' }, + ]); + const agent = new RespAct('question -> answer', { + lm, + forceTextMode: true, + tools: { echo: (input: string) => input }, + }); + + const result = await agent.forward({ question: 'Q' }); + + expect(result.answer).toBe('done'); + // The text loop sends one user turn and declares no tools. + expect(lm.turns[0]).toHaveLength(1); + expect(lm.options[0]?.tools).toBeUndefined(); + }); + }); }); diff --git a/packages/core/src/modules/respact.ts b/packages/core/src/modules/respact.ts index 67b5953..e16a072 100644 --- a/packages/core/src/modules/respact.ts +++ b/packages/core/src/modules/respact.ts @@ -1,7 +1,15 @@ +import { z } from 'zod'; +import { BaseLM } from '../core/base-lm'; import { Module } from '../core/module'; import { Prediction } from '../core/prediction'; import { type Signature } from '../core/signature'; -import type { ILanguageModel, LLMCallOptions } from '../types/language-model'; +import type { + ChatMessage, + ILanguageModel, + LLMCallOptions, + ToolCall, + ToolSpec, +} from '../types/language-model'; import type { SignatureOutput } from '../types/signature'; import { parseOutput as utilParseOutput } from '../utils/parsing'; import { ValidationError } from '../core/errors'; @@ -10,13 +18,60 @@ export interface ToolFunction { (...args: any[]): Promise | any; } +/** + * A tool's argument schema: either a JSON Schema object or a Zod schema. + * + * A Zod schema buys argument validation on top of the declaration — the + * model's arguments are parsed through it before the tool runs, and a failure + * comes back as an observation the model can correct. + */ +export type ToolParameterSchema = Record | z.ZodType; + export interface ToolWithDescription { description: string; function: ToolFunction; + /** + * Argument schema. When present the tool is called with a single object of + * named, validated arguments; when absent it keeps the historical + * single-string signature. + */ + parameters?: ToolParameterSchema; } export type ToolDefinition = ToolFunction | ToolWithDescription; +/** A tool after normalisation, as stored on the agent. */ +interface NormalizedTool { + description: string; + function: ToolFunction; + /** JSON Schema advertised to the provider. Synthesised for untyped tools. */ + parameters: Record; + /** Present only when the caller declared a Zod schema. */ + validator?: z.ZodType; + /** False for tools kept on the historical single-string signature. */ + typed: boolean; +} + +/** Schema advertised for a tool that did not declare one. */ +function untypedToolSchema(name: string): Record { + return { + type: 'object', + properties: { + input: { type: 'string', description: `Input to the ${name} tool.` }, + }, + required: ['input'], + additionalProperties: false, + }; +} + +function isZodSchema(value: unknown): value is z.ZodType { + return ( + typeof value === 'object' && + value !== null && + typeof (value as { safeParse?: unknown }).safeParse === 'function' + ); +} + /** Events emitted as the reasoning loop runs, for logging or debugging. */ export type RespActEvent = | { type: 'thought'; step: number; text: string } @@ -33,25 +88,46 @@ export interface RespActOptions { lm?: ILanguageModel; /** Observe the reasoning loop. Replaces the previous console logging. */ onEvent?: (event: RespActEvent) => void; + /** + * Force the text-prompting loop even on a model that supports native tool + * calling. Useful for comparing the two paths, or when a provider's tool + * mode misbehaves on a particular model. + */ + forceTextMode?: boolean; } export class RespAct extends Module { - private tools: Record; + private tools: Record; private maxSteps: number; private onEvent?: (event: RespActEvent) => void; + private forceTextMode: boolean; constructor(signature: string | TSignature, options: RespActOptions) { super(signature, options.lm); this.tools = {}; for (const [name, tool] of Object.entries(options.tools)) { - this.tools[name] = + const described = typeof tool === 'function' ? { description: `Tool: ${name}`, function: tool } : tool; + const declared = described.parameters; + + this.tools[name] = { + description: described.description, + function: described.function, + typed: declared !== undefined, + validator: isZodSchema(declared) ? declared : undefined, + parameters: !declared + ? untypedToolSchema(name) + : isZodSchema(declared) + ? (z.toJSONSchema(declared, { io: 'input' }) as Record) + : declared, + }; } this.maxSteps = options.maxSteps ?? 6; this.onEvent = options.onEvent; + this.forceTextMode = options.forceTextMode ?? false; } async forward( @@ -60,6 +136,161 @@ export class RespAct ext ): Promise< Prediction & { steps: number }> & SignatureOutput & { steps: number } + > { + if (this.usesNativeTools()) { + return this.forwardNative(inputs, options); + } + return this.forwardText(inputs, options); + } + + /** + * Native tool calling needs a model that advertises it, a real + * `chatWithTools` implementation, and at least one tool to offer. Anything + * else falls back to the text-prompting loop, which works on any completion + * model. + * + * The identity check against `BaseLM.prototype` matters: the base class + * supplies a text-only `chatWithTools`, so a plain `typeof` test passes for + * every subclass. A model that inherited the default but reported + * `supportsFunctionCalling: true` — free to do before anything read the flag + * — would otherwise take the native path, never see its tools, and answer + * from the first reply without calling one. + */ + private usesNativeTools(): boolean { + if (this.forceTextMode) return false; + if (Object.keys(this.tools).length === 0) return false; + const lm = this.lm as ILanguageModel | undefined; + if (!lm || typeof lm.chatWithTools !== 'function') return false; + if (lm.chatWithTools === BaseLM.prototype.chatWithTools) return false; + return lm.getCapabilities().supportsFunctionCalling === true; + } + + // ---------------------------------------------------------------- native + + private async forwardNative( + inputs: Record, + options?: LLMCallOptions + ): Promise< + Prediction & { steps: number }> & + SignatureOutput & { steps: number } + > { + const messages: ChatMessage[] = [ + { role: 'system', content: this.buildNativePrompt() }, + { role: 'user', content: this.questionOf(inputs) }, + ]; + const previousToolCalls = new Set(); + const callOptions: LLMCallOptions = { ...options, tools: this.toolSpecs() }; + + for (let step = 0; step < this.maxSteps; step++) { + const result = await this.lm.chatWithTools!(messages, callOptions); + const text = result.content ?? ''; + if (text.trim().length > 0) { + this.emit({ type: 'thought', step, text }); + } + + const toolCalls = result.toolCalls ?? []; + if (toolCalls.length > 0) { + messages.push({ role: 'assistant', content: text, toolCalls }); + for (const call of toolCalls) { + const observation = await this.runNativeCall(call, step, previousToolCalls); + messages.push({ + role: 'tool', + name: call.name, + toolCallId: call.id, + content: observation, + }); + } + continue; + } + + if (text.trim().length === 0) { + // No assistant turn is recorded here on purpose: an empty + // assistant message is rejected outright by some providers, + // which would turn this recovery branch into a hard failure. + messages.push({ + role: 'user', + content: + 'You returned neither a tool call nor an answer. Call a tool or give the final answer.', + }); + continue; + } + + messages.push({ role: 'assistant', content: text }); + + let parsed: Record | null = null; + try { + parsed = this.parseOutput(this.extractFinalAnswer(text) || text); + } catch (error) { + this.emit({ type: 'parse_failed', step, error }); + if (error instanceof ValidationError && step < this.maxSteps - 1) { + messages.push({ + role: 'user', + content: this.malformedAnswerMessage(error), + }); + continue; + } + throw error; + } + + return this.predict(parsed, step + 1); + } + + throw this.exhaustedError(); + } + + /** Execute one model-requested call, honouring the repeat guard. */ + private async runNativeCall( + call: ToolCall, + step: number, + previousToolCalls: Set + ): Promise { + const description = describeArguments(call.arguments); + const key = `${call.name}:${description}`; + + if (previousToolCalls.has(key)) { + this.emit({ + type: 'repeated_tool_call', + step, + tool: call.name, + input: description, + }); + return 'You have already made this tool call. Please move to the next step.'; + } + + previousToolCalls.add(key); + this.emit({ type: 'tool_call', step, tool: call.name, input: description }); + + // A provider that ships arguments as a JSON string (OpenAI) reports a + // payload it could not parse as empty arguments with the original text + // kept. Running the tool on those empty arguments would feed the model a + // plausible-looking observation derived from nothing, so say what went + // wrong instead and let it retry. + if (unparsedArguments(call)) { + const error = new Error(`Tool arguments were not valid JSON: ${call.rawArguments}`); + this.emit({ type: 'tool_error', step, tool: call.name, error }); + return `Error executing ${call.name}: ${error.message}`; + } + + return this.runTool(call.name, call.arguments, step); + } + + /** The tool declarations sent to the provider on every native turn. */ + private toolSpecs(): ToolSpec[] { + return Object.entries(this.tools).map(([name, tool]) => ({ + name, + description: tool.description, + parameters: tool.parameters, + })); + } + + // ------------------------------------------------------------------ text + + private async forwardText( + inputs: Record, + options?: LLMCallOptions + ): Promise< + Prediction & { steps: number }> & + SignatureOutput & { steps: number } > { let conversation = this.buildInitialPrompt(inputs); const previousToolCalls = new Set(); @@ -111,25 +342,20 @@ export class RespAct ext // A malformed final answer is recoverable: tell the model what // shape it owes us and let it try again on the next step. if (error instanceof ValidationError && step < this.maxSteps - 1) { - const fieldList = error.issues.map((issue) => issue.field).join(', '); - conversation += `\n\nObservation: Your Final Answer was missing or malformed for: ${fieldList}. Provide a Final Answer with every required field on its own "field: value" line.`; + conversation += `\n\nObservation: ${this.malformedAnswerMessage(error)}`; continue; } throw error; } - const combinedOutput = { ...parsed, steps: step + 1 }; - return new Prediction(combinedOutput) as Prediction< - SignatureOutput & { steps: number } - > & - SignatureOutput & { steps: number }; + return this.predict(parsed, step + 1); } - throw new Error( - `RespAct exceeded maximum steps (${this.maxSteps}) without producing a valid final answer` - ); + throw this.exhaustedError(); } + // --------------------------------------------------------------- shared + protected parseOutput(rawOutput: unknown): Record { if (!this.signature) { throw new Error('No signature provided for RespAct parsing'); @@ -139,31 +365,55 @@ export class RespAct ext return utilParseOutput(this.signature, outputText); } + private predict(parsed: Record, steps: number) { + const combinedOutput = { ...parsed, steps }; + return new Prediction(combinedOutput) as Prediction< + SignatureOutput & { steps: number } + > & + SignatureOutput & { steps: number }; + } + + private exhaustedError(): Error { + return new Error( + `RespAct exceeded maximum steps (${this.maxSteps}) without producing a valid final answer` + ); + } + + private malformedAnswerMessage(error: ValidationError): string { + const fieldList = error.issues.map((issue) => issue.field).join(', '); + return `Your Final Answer was missing or malformed for: ${fieldList}. Provide a Final Answer with every required field on its own "field: value" line.`; + } + private emit(event: RespActEvent): void { this.onEvent?.(event); } + private questionOf(inputs: Record): string { + return String(inputs.question ?? JSON.stringify(inputs)); + } + + private outputFormatInstruction(): string { + if (typeof this.signature === 'string' || !this.signature) return ''; + const fieldNames = Object.keys(this.signature.getOutputFields()); + if (fieldNames.length === 0) return ''; + + let instruction = + '\n\nWhen providing your Final Answer, include all of the following fields, each on its own line:\n\n'; + for (const field of fieldNames) { + instruction += `${field}: [your response for ${field}]\n`; + } + return instruction; + } + private buildInitialPrompt(inputs: Record): string { const toolDescriptions = Object.entries(this.tools) - .map(([name, tool]) => `- ${name}: ${tool.description}`) + .map(([name, tool]) => `- ${name}: ${tool.description}${textInputHint(tool)}`) .join('\n'); - let outputFormatInstruction = ''; - if (typeof this.signature !== 'string' && this.signature) { - const fieldNames = Object.keys(this.signature.getOutputFields()); - if (fieldNames.length > 0) { - outputFormatInstruction = - '\n\nWhen providing your Final Answer, include all of the following fields, each on its own line:\n\n'; - for (const field of fieldNames) { - outputFormatInstruction += `${field}: [your response for ${field}]\n`; - } - } - } - return `You have access to the following tools: ${toolDescriptions} -Question: ${inputs.question ?? JSON.stringify(inputs)} +Question: ${this.questionOf(inputs)} Use the available tools to gather what you need before answering. @@ -176,11 +426,24 @@ Action Input: [input to the tool] When you have everything you need, respond with: Thought: [why you now have enough] -Final Answer: [complete answer to the original question]${outputFormatInstruction} +Final Answer: [complete answer to the original question]${this.outputFormatInstruction()} Begin.`; } + /** + * System prompt for the native path. + * + * The tools and their schemas travel in the request rather than the prompt, + * so this only has to cover what the tool protocol does not: how the final + * answer should be shaped. + */ + private buildNativePrompt(): string { + return `Answer the user's question. You have tools available; call them as needed to gather what you need before answering. + +When you have everything you need, reply with the final answer and no tool call.${this.outputFormatInstruction()}`; + } + private extractToolCall(response: string): { tool: string; input: string } | null { const actionMatch = response.match(/Action:\s*(.+?)(?=\n|$)/m); const inputMatch = response.match(/Action Input:\s*(.+?)(?=\n|$)/m); @@ -191,14 +454,65 @@ Begin.`; return null; } + /** Run a tool from the text path, where arguments arrive as one string. */ private async executeTool(toolName: string, input: string, step: number): Promise { - if (!(toolName in this.tools)) { - return `Error: Tool '${toolName}' not found. Available tools: ${Object.keys(this.tools).join(', ')}`; + const tool = this.tools[toolName]; + if (!tool) { + return this.unknownToolMessage(toolName); } + let args: Record; try { - const result = await this.tools[toolName].function(input); - const output = String(result); + args = tool.typed ? parseTextArguments(tool, input) : { input }; + } catch (error) { + this.emit({ type: 'tool_error', step, tool: toolName, error }); + return `Error executing ${toolName}: ${error instanceof Error ? error.message : String(error)}`; + } + + return this.runTool(toolName, args, step); + } + + /** Validate arguments, call the tool, and turn whatever happens into an observation. */ + private async runTool( + toolName: string, + args: Record, + step: number + ): Promise { + const tool = this.tools[toolName]; + if (!tool) { + return this.unknownToolMessage(toolName); + } + + try { + let callArgs: unknown = args; + if (tool.validator) { + const validated = tool.validator.safeParse(args); + if (!validated.success) { + throw new Error( + `Invalid arguments: ${validated.error.issues + .map( + (issue) => + `${issue.path.join('.') || '(root)'}: ${issue.message}` + ) + .join('; ')}` + ); + } + callArgs = validated.data; + } + + if (!tool.typed && typeof args.input !== 'string') { + // An untyped tool is declared as taking one string called + // `input`. A model that sends something else would otherwise + // reach the tool as `undefined`. + throw new Error( + `Expected a single string argument named "input". Received: ${JSON.stringify(args)}` + ); + } + + const result = await (tool.typed + ? tool.function(callArgs) + : tool.function(args.input)); + const output = typeof result === 'string' ? result : stringifyResult(result); this.emit({ type: 'tool_result', step, tool: toolName, output }); return output; } catch (error) { @@ -207,9 +521,128 @@ Begin.`; } } + private unknownToolMessage(toolName: string): string { + return `Error: Tool '${toolName}' not found. Available tools: ${Object.keys(this.tools).join(', ')}`; + } + private extractFinalAnswer(response: string): string { // Keep everything after the marker: multi-field answers span lines. const match = response.match(/Final Answer:\s*([\s\S]*)$/i); return match ? match[1].trim() : ''; } } + +/** Objects and arrays are worth sending back as JSON; everything else stringifies. */ +function stringifyResult(result: unknown): string { + if (result === null || typeof result !== 'object') return String(result); + try { + return JSON.stringify(result); + } catch { + return String(result); + } +} + +/** + * Did the provider fail to parse this call's arguments? + * + * Only providers that transmit arguments as a JSON string can hit this, and they + * signal it by reporting empty `arguments` while keeping the text they could not + * parse in `rawArguments`. + */ +function unparsedArguments(call: ToolCall): boolean { + const raw = call.rawArguments?.trim(); + if (!raw || raw === '{}') return false; + return Object.keys(call.arguments ?? {}).length === 0; +} + +/** + * Turn a text-mode `Action Input:` line into named arguments. + * + * A typed tool wants an object, but the text loop only ever produces a line of + * text. JSON is the documented form; a single-property schema also accepts the + * bare value, which is what models tend to write for a one-argument tool. That + * bare value is coerced to the declared scalar type, so a `number` argument does + * not fail validation purely because the loop only speaks text. + */ +function parseTextArguments(tool: NormalizedTool, input: string): Record { + const trimmed = input.trim(); + if (trimmed.startsWith('{')) { + try { + const parsed = JSON.parse(trimmed); + if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) { + return parsed as Record; + } + } catch { + // Fall through to the single-property case. + } + } + + const names = propertyNames(tool); + if (names.length === 1) { + return { [names[0]]: coerceScalar(trimmed, propertyType(tool, names[0])) }; + } + + throw new Error( + `Arguments must be a JSON object with these keys: ${names.join(', ')}. Received: ${trimmed}` + ); +} + +function coerceScalar(value: string, type: string | undefined): unknown { + switch (type) { + case 'number': + case 'integer': { + const parsed = Number(value); + return Number.isNaN(parsed) ? value : parsed; + } + case 'boolean': + if (value === 'true') return true; + if (value === 'false') return false; + return value; + default: + return value; + } +} + +/** Argument names declared by a tool's JSON Schema, in declaration order. */ +function propertyNames(tool: NormalizedTool): string[] { + return Object.keys(schemaProperties(tool)); +} + +/** The declared JSON Schema `type` of one argument, when it is a simple scalar. */ +function propertyType(tool: NormalizedTool, name: string): string | undefined { + const property = schemaProperties(tool)[name]; + if (!property || typeof property !== 'object') return undefined; + const type = (property as { type?: unknown }).type; + return typeof type === 'string' ? type : undefined; +} + +function schemaProperties(tool: NormalizedTool): Record { + const properties = tool.parameters.properties; + if (!properties || typeof properties !== 'object') return {}; + return properties as Record; +} + +/** Tell the text-mode model what an `Action Input:` for a typed tool looks like. */ +function textInputHint(tool: NormalizedTool): string { + if (!tool.typed) return ''; + const names = propertyNames(tool); + if (names.length === 0) return ''; + return ` (Action Input must be a JSON object with keys: ${names.join(', ')})`; +} + +/** Stable, human-readable rendering of a tool call's arguments. */ +function describeArguments(args: Record): string { + const keys = Object.keys(args ?? {}); + if (keys.length === 1 && typeof args[keys[0]] === 'string') { + return args[keys[0]] as string; + } + const sorted: Record = {}; + for (const key of keys.sort()) { + sorted[key] = args[key]; + } + try { + return JSON.stringify(sorted); + } catch { + return String(args); + } +} diff --git a/packages/core/src/types/language-model.ts b/packages/core/src/types/language-model.ts index 866deed..2e72a48 100644 --- a/packages/core/src/types/language-model.ts +++ b/packages/core/src/types/language-model.ts @@ -16,27 +16,74 @@ export interface LLMCallOptions { * ts-dspy does not add a second retry layer on top. */ retries?: number; + /** + * Tools the model may call on this turn. Providers advertising + * `supportsFunctionCalling` translate these into their own request shape; + * providers without native tool calling ignore them. + */ + tools?: ToolSpec[]; + /** How hard to push the model towards calling a tool. Defaults to the provider's own default. */ + toolChoice?: ToolChoice; metadata?: Record; } +/** + * A tool offered to the model. + * + * `parameters` is a JSON Schema object describing the arguments. Every provider + * accepts JSON Schema here, so this is the one representation that survives the + * trip through all three SDKs unchanged. + */ +export interface ToolSpec { + name: string; + description?: string; + parameters: Record; +} + +export type ToolChoice = 'auto' | 'none' | 'required' | { name: string }; + +/** + * A tool call requested by the model. + * + * The shape is deliberately provider-neutral rather than a copy of any one SDK: + * + * - `arguments` is always a **parsed object**. OpenAI sends a JSON string, which + * is parsed on the way in; Anthropic (`input`) and Gemini (`args`) already + * send objects. + * - `id` is optional because Gemini's function calls have no identifier — + * results there are correlated by function name. + * - `rawArguments` keeps the provider's original encoding when there was one, so + * an assistant turn can be replayed byte-for-byte and so a call whose arguments + * failed to parse is still inspectable. + */ +export interface ToolCall { + /** Provider-assigned identifier. Absent on Gemini. */ + id?: string; + name: string; + arguments: Record; + /** The unparsed argument payload, when the provider sent one (OpenAI only). */ + rawArguments?: string; +} + export interface ChatMessage { role: 'system' | 'user' | 'assistant' | 'function' | 'tool'; content: string; + /** On a `tool`/`function` turn, the name of the tool that produced the result. */ name?: string; - functionCall?: { - name: string; - arguments: string; - }; + /** On a `tool`/`function` turn, the id of the call being answered. */ + toolCallId?: string; + /** On an `assistant` turn, the tool calls the model requested. */ toolCalls?: ToolCall[]; } -export interface ToolCall { - id: string; - type: 'function'; - function: { - name: string; - arguments: string; - }; +/** Why the model stopped, normalised across providers. */ +export type FinishReason = 'stop' | 'tool_calls' | 'length' | 'content_filter' | 'other'; + +/** A full chat turn, including any tool calls the model asked for. */ +export interface ChatResult { + content: string; + toolCalls?: ToolCall[]; + finishReason?: FinishReason; } export interface UsageStats { @@ -86,6 +133,14 @@ export interface ILanguageModel { options?: LLMCallOptions ): Promise; chat(messages: ChatMessage[], options?: LLMCallOptions): Promise; + /** + * Run one chat turn and return the tool calls alongside the text. + * + * Providers advertising `supportsFunctionCalling` implement this natively; + * {@link BaseLM} supplies a text-only default for everyone else, so callers + * can rely on the capability flag rather than feature-detecting the method. + */ + chatWithTools?(messages: ChatMessage[], options?: LLMCallOptions): Promise; generateStream?( prompt: string, options?: LLMCallOptions diff --git a/packages/gemini/src/gemini-lm.test.ts b/packages/gemini/src/gemini-lm.test.ts index 91a8a0b..5631365 100644 --- a/packages/gemini/src/gemini-lm.test.ts +++ b/packages/gemini/src/gemini-lm.test.ts @@ -124,6 +124,155 @@ describe('GeminiLM', () => { expect(systemInstruction).toBe('be terse'); expect(contents).toHaveLength(1); }); + + it('turns an assistant tool call into a functionCall part', () => { + const { contents } = toGeminiContents([ + { + role: 'assistant', + content: 'adding', + toolCalls: [{ name: 'add', arguments: { a: 1, b: 2 } }], + }, + ]); + + expect(contents).toEqual([ + { + role: 'model', + parts: [ + { text: 'adding' }, + { functionCall: { name: 'add', args: { a: 1, b: 2 } } }, + ], + }, + ]); + }); + + it('turns a tool result into a functionResponse part', () => { + // Gemini has no tool-call id: results are correlated by name. + const { contents } = toGeminiContents([ + { role: 'tool', name: 'add', content: '3' }, + ]); + + expect(contents).toEqual([ + { + role: 'user', + parts: [{ functionResponse: { name: 'add', response: { result: '3' } } }], + }, + ]); + }); + + it('folds parallel tool results into one user content', () => { + // Gemini requires the replies to a parallel call turn to arrive + // together, one functionResponse per functionCall. + const { contents } = toGeminiContents([ + { + role: 'assistant', + content: '', + toolCalls: [ + { name: 'left', arguments: {} }, + { name: 'right', arguments: {} }, + ], + }, + { role: 'tool', name: 'left', content: 'L' }, + { role: 'tool', name: 'right', content: 'R' }, + ]); + + expect(contents).toHaveLength(2); + expect(contents[1]).toEqual({ + role: 'user', + parts: [ + { functionResponse: { name: 'left', response: { result: 'L' } } }, + { functionResponse: { name: 'right', response: { result: 'R' } } }, + ], + }); + }); + + it('does not fold ordinary user text into a tool result turn', () => { + const { contents } = toGeminiContents([ + { role: 'tool', name: 'left', content: 'L' }, + { role: 'user', content: 'and now?' }, + ]); + + expect(contents).toHaveLength(2); + }); + + it('degrades an unnamed tool result to plain text', () => { + const { contents } = toGeminiContents([{ role: 'tool', content: '3' }]); + expect(contents).toEqual([{ role: 'user', parts: [{ text: '3' }] }]); + }); + }); + + describe('tool calling', () => { + it('sends tool declarations under config.tools', async () => { + mocks.generateContent.mockResolvedValue(response('ok')); + + await new GeminiLM({ apiKey: 'k' }).chatWithTools( + [{ role: 'user', content: 'hi' }], + { + tools: [ + { + name: 'add', + description: 'Add two numbers', + parameters: { type: 'object', properties: {} }, + }, + ], + } + ); + + expect(mocks.generateContent.mock.calls[0][0].config.tools).toEqual([ + { + functionDeclarations: [ + { + name: 'add', + description: 'Add two numbers', + parametersJsonSchema: { type: 'object', properties: {} }, + }, + ], + }, + ]); + }); + + it('maps a forced choice onto functionCallingConfig', async () => { + mocks.generateContent.mockResolvedValue(response('ok')); + + await new GeminiLM({ apiKey: 'k' }).chatWithTools( + [{ role: 'user', content: 'hi' }], + { tools: [{ name: 'add', parameters: {} }], toolChoice: { name: 'add' } } + ); + + expect(mocks.generateContent.mock.calls[0][0].config.toolConfig).toEqual({ + functionCallingConfig: { mode: 'ANY', allowedFunctionNames: ['add'] }, + }); + }); + + it('reads functionCall parts off the candidate', async () => { + mocks.generateContent.mockResolvedValue( + response('', { + candidates: [ + { + content: { + parts: [{ functionCall: { name: 'add', args: { a: 1 } } }], + }, + }, + ], + }) + ); + + const result = await new GeminiLM({ apiKey: 'k' }).chatWithTools([ + { role: 'user', content: 'hi' }, + ]); + + expect(result.finishReason).toBe('tool_calls'); + // No `id` key at all — Gemini does not issue one. + expect(result.toolCalls).toEqual([{ name: 'add', arguments: { a: 1 } }]); + }); + + it('omits tool config entirely when no tools are offered', async () => { + mocks.generateContent.mockResolvedValue(response('ok')); + await new GeminiLM({ apiKey: 'k' }).chat([{ role: 'user', content: 'hi' }]); + + const config = mocks.generateContent.mock.calls[0][0].config; + expect(config).not.toHaveProperty('tools'); + expect(config).not.toHaveProperty('toolConfig'); + }); }); describe('safety blocking', () => { diff --git a/packages/gemini/src/gemini-lm.ts b/packages/gemini/src/gemini-lm.ts index 363595a..68997cf 100644 --- a/packages/gemini/src/gemini-lm.ts +++ b/packages/gemini/src/gemini-lm.ts @@ -2,9 +2,12 @@ import { BaseLM, LMError, type ChatMessage, + type ChatResult, + type FinishReason, type LLMCallOptions, type ModelCapabilities, type StreamChunk, + type ToolCall, } from '@ts-dspy/core'; import { GoogleGenAI, @@ -13,6 +16,7 @@ import { type Content, type GenerateContentConfig, type GenerateContentResponse, + type Part, type SafetySetting, } from '@google/genai'; @@ -72,9 +76,27 @@ export class GeminiLM extends BaseLM { } async chat(messages: ChatMessage[], options?: LLMCallOptions): Promise { + return (await this.chatWithTools(messages, options)).content; + } + + async chatWithTools( + messages: ChatMessage[], + options?: LLMCallOptions + ): Promise { const { contents, systemInstruction } = toGeminiContents(messages); - const response = await this.send(contents, systemInstruction, options); - return response.text ?? ''; + const response = await this.send( + contents, + systemInstruction, + options, + toolConfig(options) + ); + + const toolCalls = toolCallsOf(response); + return { + content: response.text ?? '', + ...(toolCalls.length > 0 ? { toolCalls } : {}), + finishReason: finishReasonOf(response, toolCalls.length > 0), + }; } async generateStructured( @@ -120,7 +142,10 @@ export class GeminiLM extends BaseLM { stream = await this.client.models.generateContentStream({ model: options?.model ?? this.model, contents, - config: this.buildConfig(systemInstruction, options), + config: { + ...this.buildConfig(systemInstruction, options), + ...toolConfig(options), + }, }); } catch (error) { this.recordError(); @@ -244,6 +269,11 @@ function usageFrom(response: GenerateContentResponse | undefined) { * system role in `contents`. This does not mutate the caller's array — the * previous implementation called `messages.pop()`, destroying the last turn of * any array a caller reused. + * + * Consecutive `functionResponse` turns are folded into one user content: Gemini + * requires the replies to a parallel call turn to arrive together, matching the + * `functionCall` parts one for one, and rejects them spread across separate + * turns. */ export function toGeminiContents(messages: ChatMessage[]): { contents: Content[]; @@ -257,9 +287,20 @@ export function toGeminiContents(messages: ChatMessage[]): { systemParts.push(message.content); continue; } + + const parts = toGeminiParts(message); + const previous = contents.at(-1); + + if (previous?.role === 'user' && isFunctionResponses(previous.parts)) { + if (isFunctionResponses(parts)) { + previous.parts = [...(previous.parts ?? []), ...parts]; + continue; + } + } + contents.push({ role: message.role === 'assistant' ? 'model' : 'user', - parts: [{ text: message.content }], + parts, }); } @@ -269,6 +310,120 @@ export function toGeminiContents(messages: ChatMessage[]): { }; } +/** + * Build the parts of one turn. + * + * Tool traffic is structural in Gemini too: an assistant turn's tool calls + * become `functionCall` parts, and a `tool` result turn becomes a + * `functionResponse` part. Both are keyed by function *name* — Gemini has no + * tool-call identifier, so results are correlated by name and position. + */ +function toGeminiParts(message: ChatMessage): Part[] { + if (message.role === 'assistant' && message.toolCalls?.length) { + const parts: Part[] = []; + if (message.content) parts.push({ text: message.content }); + for (const call of message.toolCalls) { + parts.push({ + functionCall: { name: call.name, args: call.arguments ?? {} }, + }); + } + return parts; + } + + if (message.role === 'tool' || message.role === 'function') { + // Without a name there is nothing to correlate against, so the result + // degrades to ordinary text rather than an unaddressed response part. + if (!message.name) return [{ text: message.content }]; + return [ + { + functionResponse: { + name: message.name, + // `response` must be an object, not a bare string. + response: { result: message.content }, + }, + }, + ]; + } + + return [{ text: message.content }]; +} + +function isFunctionResponses(parts: Part[] | undefined): boolean { + return Boolean(parts?.length) && parts!.every((part) => Boolean(part.functionResponse)); +} + +/** Extract the `functionCall` parts of a response. Gemini sends `args` already parsed. */ +function toolCallsOf(response: GenerateContentResponse): ToolCall[] { + const parts = response.candidates?.[0]?.content?.parts ?? []; + const calls = parts + .map((part) => part.functionCall) + .filter((call): call is NonNullable => Boolean(call?.name)); + + return calls.map((call) => ({ + // Gemini omits `id` on the Gemini API and populates it on some Vertex + // configurations; `ToolCall.id` is optional precisely for this. + ...(call.id ? { id: call.id } : {}), + name: call.name!, + arguments: (call.args ?? {}) as Record, + })); +} + +/** Translate tool declarations into the `generateContent` config shape. */ +function toolConfig(options?: LLMCallOptions): Partial { + if (!options?.tools?.length) return {}; + + const config: Partial = { + tools: [ + { + functionDeclarations: options.tools.map((tool) => ({ + name: tool.name, + ...(tool.description ? { description: tool.description } : {}), + parametersJsonSchema: tool.parameters, + })), + }, + ], + }; + + const choice = options.toolChoice; + if (choice !== undefined) { + config.toolConfig = + typeof choice === 'object' + ? { + functionCallingConfig: { + mode: 'ANY' as never, + allowedFunctionNames: [choice.name], + }, + } + : { + functionCallingConfig: { + mode: (choice === 'required' ? 'ANY' : choice.toUpperCase()) as never, + }, + }; + } + + return config; +} + +function finishReasonOf( + response: GenerateContentResponse, + hasToolCalls: boolean +): FinishReason { + if (hasToolCalls) return 'tool_calls'; + switch (response.candidates?.[0]?.finishReason) { + case 'STOP': + return 'stop'; + case 'MAX_TOKENS': + return 'length'; + case 'SAFETY': + case 'PROHIBITED_CONTENT': + return 'content_filter'; + case undefined: + return 'stop'; + default: + return 'other'; + } +} + function toLMError(error: unknown): LMError { if (error instanceof LMError) return error; const message = error instanceof Error ? error.message : String(error); diff --git a/packages/openai/src/openai-lm.test.ts b/packages/openai/src/openai-lm.test.ts index 578f133..570bec8 100644 --- a/packages/openai/src/openai-lm.test.ts +++ b/packages/openai/src/openai-lm.test.ts @@ -264,5 +264,218 @@ describe('OpenAILM', () => { toOpenAIMessages(messages); expect(messages).toHaveLength(1); }); + + it('keeps a tool result on the tool role, keyed by its call id', () => { + expect( + toOpenAIMessages([ + { role: 'tool', name: 'lookup', toolCallId: 'call_1', content: '42' }, + ]) + ).toEqual([{ role: 'tool', tool_call_id: 'call_1', content: '42' }]); + }); + + it('downgrades an uncorrelated tool result to user content', () => { + // The API rejects a tool turn without `tool_call_id`, so a result + // that carries no id is surfaced rather than making the call fail. + expect(toOpenAIMessages([{ role: 'tool', content: '42' }])).toEqual([ + { role: 'user', content: '42' }, + ]); + }); + + it('re-encodes assistant tool calls as JSON-string arguments', () => { + expect( + toOpenAIMessages([ + { + role: 'assistant', + content: '', + toolCalls: [{ id: 'call_1', name: 'add', arguments: { a: 1, b: 2 } }], + }, + ]) + ).toEqual([ + { + role: 'assistant', + content: null, + tool_calls: [ + { + id: 'call_1', + type: 'function', + function: { name: 'add', arguments: '{"a":1,"b":2}' }, + }, + ], + }, + ]); + }); + + it('gives parallel calls to one tool distinct synthesized ids', () => { + // A Gemini-sourced turn carries no ids at all; two calls to the same + // tool must not collapse onto one tool_call_id. + const [message] = toOpenAIMessages([ + { + role: 'assistant', + content: '', + toolCalls: [ + { name: 'lookup', arguments: { id: 1 } }, + { name: 'lookup', arguments: { id: 2 } }, + ], + }, + ]) as any[]; + + const ids = message.tool_calls.map((call: any) => call.id); + expect(new Set(ids).size).toBe(2); + }); + + it('replays the provider original argument bytes when present', () => { + const [message] = toOpenAIMessages([ + { + role: 'assistant', + content: 'ok', + toolCalls: [ + { + id: 'call_1', + name: 'add', + arguments: { a: 1 }, + rawArguments: '{"a": 1}', + }, + ], + }, + ]) as any[]; + + expect(message.tool_calls[0].function.arguments).toBe('{"a": 1}'); + }); + }); + + describe('tool calling', () => { + it('sends tool declarations in the function-wrapped request shape', async () => { + mocks.create.mockResolvedValue(completion('ok')); + + await new OpenAILM({ apiKey: 'k' }).chatWithTools( + [{ role: 'user', content: 'hi' }], + { + tools: [ + { + name: 'add', + description: 'Add two numbers', + parameters: { type: 'object', properties: {} }, + }, + ], + toolChoice: 'required', + } + ); + + const body = mocks.create.mock.calls[0][0]; + expect(body.tools).toEqual([ + { + type: 'function', + function: { + name: 'add', + description: 'Add two numbers', + parameters: { type: 'object', properties: {} }, + }, + }, + ]); + expect(body.tool_choice).toBe('required'); + }); + + it('names a specific tool when the choice is an object', async () => { + mocks.create.mockResolvedValue(completion('ok')); + + await new OpenAILM({ apiKey: 'k' }).chatWithTools( + [{ role: 'user', content: 'hi' }], + { + tools: [{ name: 'add', parameters: {} }], + toolChoice: { name: 'add' }, + } + ); + + expect(mocks.create.mock.calls[0][0].tool_choice).toEqual({ + type: 'function', + function: { name: 'add' }, + }); + }); + + it('omits tool parameters entirely when no tools are offered', async () => { + mocks.create.mockResolvedValue(completion('ok')); + await new OpenAILM({ apiKey: 'k' }).chat([{ role: 'user', content: 'hi' }]); + + expect(mocks.create.mock.calls[0][0]).not.toHaveProperty('tools'); + expect(mocks.create.mock.calls[0][0]).not.toHaveProperty('tool_choice'); + }); + + it('parses the JSON argument string into an object', async () => { + mocks.create.mockResolvedValue( + completion('', { + choices: [ + { + message: { + content: null, + tool_calls: [ + { + id: 'call_1', + type: 'function', + function: { + name: 'add', + arguments: '{"a": 1, "b": 2}', + }, + }, + ], + }, + finish_reason: 'tool_calls', + }, + ], + }) + ); + + const result = await new OpenAILM({ apiKey: 'k' }).chatWithTools([ + { role: 'user', content: 'hi' }, + ]); + + expect(result.finishReason).toBe('tool_calls'); + expect(result.toolCalls).toEqual([ + { + id: 'call_1', + name: 'add', + arguments: { a: 1, b: 2 }, + rawArguments: '{"a": 1, "b": 2}', + }, + ]); + }); + + it('keeps unparseable arguments inspectable rather than throwing', async () => { + mocks.create.mockResolvedValue( + completion('', { + choices: [ + { + message: { + content: null, + tool_calls: [ + { + id: 'call_1', + type: 'function', + function: { name: 'add', arguments: '{not json' }, + }, + ], + }, + finish_reason: 'tool_calls', + }, + ], + }) + ); + + const result = await new OpenAILM({ apiKey: 'k' }).chatWithTools([ + { role: 'user', content: 'hi' }, + ]); + + expect(result.toolCalls?.[0].arguments).toEqual({}); + expect(result.toolCalls?.[0].rawArguments).toBe('{not json'); + }); + + it('reports no toolCalls key on an ordinary reply', async () => { + mocks.create.mockResolvedValue(completion('plain')); + + const result = await new OpenAILM({ apiKey: 'k' }).chatWithTools([ + { role: 'user', content: 'hi' }, + ]); + + expect(result).toEqual({ content: 'plain', finishReason: 'stop' }); + }); }); }); diff --git a/packages/openai/src/openai-lm.ts b/packages/openai/src/openai-lm.ts index b2257ef..48de99a 100644 --- a/packages/openai/src/openai-lm.ts +++ b/packages/openai/src/openai-lm.ts @@ -2,9 +2,12 @@ import { BaseLM, LMError, type ChatMessage, + type ChatResult, + type FinishReason, type LLMCallOptions, type ModelCapabilities, type StreamChunk, + type ToolCall, } from '@ts-dspy/core'; import OpenAI, { APIError } from 'openai'; import type { ChatCompletionMessageParam } from 'openai/resources/chat/completions'; @@ -65,6 +68,13 @@ export class OpenAILM extends BaseLM { } async chat(messages: ChatMessage[], options?: LLMCallOptions): Promise { + return (await this.chatWithTools(messages, options)).content; + } + + async chatWithTools( + messages: ChatMessage[], + options?: LLMCallOptions + ): Promise { const startedAt = Date.now(); try { @@ -73,6 +83,7 @@ export class OpenAILM extends BaseLM { model: options?.model ?? this.model, messages: toOpenAIMessages(messages), ...samplingParams(options), + ...toolParams(options), }, requestOptions(options) ); @@ -83,7 +94,14 @@ export class OpenAILM extends BaseLM { latencyMs: Date.now() - startedAt, }); - return completion.choices[0]?.message?.content ?? ''; + const choice = completion.choices[0]; + const toolCalls = fromOpenAIToolCalls(choice?.message?.tool_calls); + + return { + content: choice?.message?.content ?? '', + ...(toolCalls.length > 0 ? { toolCalls } : {}), + finishReason: finishReasonOf(choice?.finish_reason), + }; } catch (error) { this.recordError(); throw toLMError(error); @@ -229,11 +247,27 @@ export function toOpenAIMessages(messages: ChatMessage[]): ChatCompletionMessage case 'system': return { role: 'system', content: message.content }; case 'assistant': + if (message.toolCalls?.length) { + return { + role: 'assistant', + // The API rejects an empty string alongside tool_calls. + content: message.content || null, + tool_calls: message.toolCalls.map(toOpenAIToolCall), + }; + } return { role: 'assistant', content: message.content }; case 'tool': case 'function': - // The core ChatMessage shape has no tool_call_id, so a tool - // result is surfaced as user content rather than dropped. + // `tool_call_id` is what pairs a result with its call. Without + // one the API would reject the turn, so an uncorrelated result + // is still surfaced as user content rather than dropped. + if (message.toolCallId) { + return { + role: 'tool', + tool_call_id: message.toolCallId, + content: message.content, + }; + } return { role: 'user', content: message.content }; default: return { role: 'user', content: message.content }; @@ -241,6 +275,102 @@ export function toOpenAIMessages(messages: ChatMessage[]): ChatCompletionMessage }); } +function toOpenAIToolCall(call: ToolCall, index: number) { + return { + // OpenAI requires an id; a call relayed from a provider without one + // (Gemini) gets a placeholder. The position is part of it because two + // parallel calls to the same tool would otherwise share an id, and + // results would then pair up with the wrong call. + id: call.id ?? `call_${index}_${call.name}`, + type: 'function' as const, + function: { + name: call.name, + // The wire format is a JSON *string*. Replay the provider's original + // bytes when we have them, so a round trip is lossless. + arguments: call.rawArguments ?? JSON.stringify(call.arguments ?? {}), + }, + }; +} + +function fromOpenAIToolCalls( + toolCalls: + Array<{ id?: string; function?: { name?: string; arguments?: string } }> | undefined +): ToolCall[] { + if (!toolCalls?.length) return []; + + return toolCalls + .filter((call) => typeof call.function?.name === 'string') + .map((call) => { + const raw = call.function?.arguments ?? ''; + return { + id: call.id, + name: call.function!.name!, + arguments: parseArguments(raw), + rawArguments: raw, + }; + }); +} + +/** + * OpenAI sends arguments as a JSON string. A model can emit one that does not + * parse; that is a tool-argument problem for the caller to report back to the + * model, not a transport failure, so it yields empty arguments with the original + * text preserved in `rawArguments`. + */ +function parseArguments(raw: string): Record { + if (!raw.trim()) return {}; + try { + const parsed = JSON.parse(raw); + return parsed && typeof parsed === 'object' && !Array.isArray(parsed) + ? (parsed as Record) + : {}; + } catch { + return {}; + } +} + +/** Translate tool declarations into the Chat Completions request shape. */ +function toolParams(options?: LLMCallOptions): Record { + if (!options?.tools?.length) return {}; + + const params: Record = { + tools: options.tools.map((tool) => ({ + type: 'function', + function: { + name: tool.name, + ...(tool.description ? { description: tool.description } : {}), + parameters: tool.parameters, + }, + })), + }; + + const choice = options.toolChoice; + if (choice !== undefined) { + params.tool_choice = + typeof choice === 'string' + ? choice + : { type: 'function', function: { name: choice.name } }; + } + + return params; +} + +function finishReasonOf(reason: string | null | undefined): FinishReason { + switch (reason) { + case 'stop': + return 'stop'; + case 'tool_calls': + case 'function_call': + return 'tool_calls'; + case 'length': + return 'length'; + case 'content_filter': + return 'content_filter'; + default: + return 'other'; + } +} + /** * Build sampling parameters. * diff --git a/site/docs.html b/site/docs.html index e78ae13..687f10d 100644 --- a/site/docs.html +++ b/site/docs.html @@ -253,9 +253,9 @@

RespAct

06

Tools and RespAct

- Tools are plain functions taking a string and returning a string, or a - promise of one. Give each a description — the model picks from those - descriptions, so write them as instructions. + In their simplest form tools are plain functions taking a string and + returning a string, or a promise of one. Give each a description — the + model picks from those descriptions, so write them as instructions. 

import { RespAct } from '@ts-dspy/core'
@@ -266,11 +266,11 @@ 
Tools and RespAct

     tools: {
       lookupOrder: {
         description: 'Look up an order by its ID. Input: the order ID.',
-        fn: async (id) => JSON.stringify(await db.orders.find(id)),
+        function: async (id) => JSON.stringify(await db.orders.find(id)),
       },
       today: {
         description: 'Return today’s date. Input: ignored.',
-        fn: () => new Date().toISOString().slice(0, 10),
+        function: () => new Date().toISOString().slice(0, 10),
       },
     },
     maxSteps: 8,
@@ -281,6 +281,75 @@ 
Tools and RespAct

 const out = await agent.forward({ question: 'Has order A-4182 shipped?' })
 console.log(out.answer, out.steps)
+

Typed tool arguments

+

+ One string is a thin pipe for a tool that really wants three fields. Add a + parameters schema — a JSON Schema object, or a Zod schema + — and the tool receives named arguments instead. Zod schemas are + validated before the tool runs, and a failure comes back to the model as an + observation it can correct rather than an exception you have to catch. +

+ + 
import { z } from 'zod'
+
+const agent = new RespAct(
+  'question -> answer',
+  {
+    tools: {
+      findFlights: {
+        description: 'Find flights between two airports on a date.',
+        parameters: z.object({
+          from: z.string().describe('departure IATA code'),
+          to: z.string().describe('arrival IATA code'),
+          date: z.string().describe('ISO date, e.g. 2026-03-14'),
+        }),
+        function: ({ from, to, date }) => flights.search(from, to, date),
+      },
+    },
+  }
+)
+ +

+ Bare functions and { description, function } keep working as + they always did; a tool without parameters is still called with + a single string. +

+ +

Native vs. text-mode tool calling

+

+ RespAct chooses its path from the model, not from configuration. + When the language model reports + supportsFunctionCalling: true — every provider in this + repo does — the tool schemas travel in the request itself and the + model's calls come back as structured data. When it does not, the loop falls + back to prompting for Action: / Action Input: lines + and parsing them out of the completion, which works on any model that can + produce text. +

+ +
+ + + + + + + + + +
 NativeText mode
RequiressupportsFunctionCallingany model that emits text
Tool schemassent with the requestdescribed in the prompt
Argumentsa parsed object from the providerJSON on the Action Input line
Calls per turnseveral, executed in orderexactly one
Typical failurearguments fail your schemathe model mis-formats and burns a step
+
+ +

+ Both paths run the same tools, honour the same repeat guard, and emit the + same events, so a program written against one works against the other. + That is the point of keeping the text loop: a local model behind an + OpenAI-compatible endpoint still runs your agent. Pass + forceTextMode: true to pin a tool-capable model to the text + loop — useful for comparing the two, or when a particular model's tool + mode misbehaves. +

+
Loop behaviour