From 628deede076f3388012b34bdef8f55bfc143ab35 Mon Sep 17 00:00:00 2001 From: Arnav Dadarya Date: Sat, 22 Aug 2026 13:35:17 -0700 Subject: [PATCH 1/2] feat: send images through every provider `ChatMessage.content` was a plain `string` and each converter passed it straight through, so `supportsVision: true` was a claim the library could not honour. Content is now `string | ContentPart[]`, where a part is text or an image carried as a URL, a data URI, or base64 plus a media type. Plain strings stay valid, so text-only code is untouched. Signature inputs can be declared images with `@ImageField` (or the `image` type in a string signature), and `buildPromptContent()` renders such a signature as parts, returning a string when every input is text. Per provider: OpenAI `image_url` parts, role-aware because only the user variant of its message union accepts them; Anthropic `image` blocks, with same-role merging moved onto block arrays so a text turn beside an image turn still satisfies strict alternation; Gemini `inlineData`, refusing a URL `fileData` cannot dereference rather than earning a 400. `supportsVision` is now reported per model instead of hardcoded. Co-Authored-By: Claude Opus 5 --- .changeset/vision-multimodal-input.md | 34 +++++ README.md | 33 +++++ packages/anthropic/src/anthropic-lm.test.ts | 86 +++++++++++- packages/anthropic/src/anthropic-lm.ts | 100 ++++++++++++-- packages/core/src/core/signature.test.ts | 21 ++- packages/core/src/core/signature.ts | 38 ++++++ packages/core/src/index.ts | 22 ++- packages/core/src/test-utils.ts | 8 +- packages/core/src/types/language-model.ts | 62 ++++++++- packages/core/src/utils/content.test.ts | 140 ++++++++++++++++++++ packages/core/src/utils/content.ts | 139 +++++++++++++++++++ packages/core/src/utils/parsing.test.ts | 80 ++++++++++- packages/core/src/utils/parsing.ts | 138 ++++++++++++++++--- packages/gemini/src/gemini-lm.test.ts | 53 +++++++- packages/gemini/src/gemini-lm.ts | 53 +++++++- packages/openai/src/openai-lm.test.ts | 81 +++++++++++ packages/openai/src/openai-lm.ts | 69 +++++++++- site/docs.html | 120 +++++++++++++++++ 18 files changed, 1228 insertions(+), 49 deletions(-) create mode 100644 .changeset/vision-multimodal-input.md create mode 100644 packages/core/src/utils/content.test.ts create mode 100644 packages/core/src/utils/content.ts diff --git a/.changeset/vision-multimodal-input.md b/.changeset/vision-multimodal-input.md new file mode 100644 index 0000000..34181ba --- /dev/null +++ b/.changeset/vision-multimodal-input.md @@ -0,0 +1,34 @@ +--- +'@ts-dspy/anthropic': minor +'@ts-dspy/gemini': minor +'@ts-dspy/openai': minor +'@ts-dspy/core': minor +--- + +Send images, not just text. `ChatMessage.content` is widened from `string` to +`string | ContentPart[]`, where a `ContentPart` is either text or an image +carried as an `https://` URL, a `data:` URI, or base64 plus a media type. Plain +strings remain valid content and behave exactly as before, so text-only code — +`generate()`, `generateStructured()`, and every module — is untouched. + +Signature inputs can now be declared as images with `@ImageField` (or the +`image` type in a string signature), and the new `buildPromptContent()` renders +such a signature as content parts, returning a plain string when every input is +text. `buildPrompt()` still returns a string, rendering an image input as an +`[image: image/png]` placeholder. + +Each provider maps parts onto its own SDK shape: OpenAI `image_url` parts (only +on user turns, since system and assistant messages accept text alone), +Anthropic `image` blocks with a base64 or URL source, and Gemini `inlineData` or +`fileData`. Anthropic's merging of consecutive same-role turns now concatenates +block arrays rather than strings; it previously merged only when both turns were +strings, which silently skipped the merge for image turns and produced two +adjacent user messages that the Messages API rejects. + +Widening `ChatMessage.content` is a breaking change to a public type — code +that treats it as a `string` without narrowing will need a narrowing step. Per +the pre-1.0 convention this ships as a minor. + +`supportsVision` is reported per model rather than hardcoded to `true`: false +for `gpt-3.5`, `o1-mini` and `o3-mini`, for `claude-3-5-haiku` and older Claude +models, and for Gemini embedding models. diff --git a/README.md b/README.md index aa2eed0..a8da9e2 100644 --- a/README.md +++ b/README.md @@ -184,6 +184,39 @@ Tool descriptions are what the model uses to decide when to call each tool, so they earn the detail. Never pass model output to `eval()` — see [`examples/utils.ts`](examples/utils.ts) for a bounded arithmetic evaluator. +### Images + +`ChatMessage.content` is `string | ContentPart[]`, so one message can carry text +and images together. Declare an image input with `@ImageField` — or the `image` +type in a string signature — and render the message with `buildPromptContent`: + +```ts +import { Signature, ImageField, OutputField, buildPromptContent } from '@ts-dspy/core'; + +class ReadSign extends Signature { + static description = 'Read the sign in the photo.'; + + @ImageField({ description: 'photo of the sign' }) + photo!: string; + + @OutputField({ description: 'the words on the sign' }) + words!: string; +} + +const content = buildPromptContent(ReadSign, { photo: 'data:image/png;base64,...' }); +await lm.chat([{ role: 'user', content }]); +``` + +`imagePart()` accepts an `https://` URL, a `data:` URI, or an explicit +`{ kind: 'base64', data, mediaType }` source. OpenAI receives `image_url` parts, +Anthropic `image` blocks, and Gemini `inlineData` (Gemini fetches no arbitrary +web URLs, so pass bytes or a Files API URI). Only user turns can carry an image, +so system and assistant content is flattened to text. Modules still build string +prompts, where an image input renders as an `[image: image/png]` placeholder — +send images through `lm.chat()` for now. A plain string content behaves exactly +as it did, and `supportsVision` is reported per model instead of being hardcoded +to `true`. + ## Examples ```bash diff --git a/packages/anthropic/src/anthropic-lm.test.ts b/packages/anthropic/src/anthropic-lm.test.ts index b56acf2..791724a 100644 --- a/packages/anthropic/src/anthropic-lm.test.ts +++ b/packages/anthropic/src/anthropic-lm.test.ts @@ -1,4 +1,4 @@ -import { LMError } from '@ts-dspy/core'; +import { LMError, imagePart, textPart } from '@ts-dspy/core'; import { AnthropicLM, AnthropicRefusalError, @@ -20,6 +20,9 @@ const mocks = vi.hoisted(() => { const { MockAPIError } = mocks; +const PNG = 'iVBORw0KGgo='; +const DATA_URI = `data:image/png;base64,${PNG}`; + vi.mock('@anthropic-ai/sdk', () => ({ default: class { messages = { create: mocks.create, stream: mocks.stream }; @@ -255,12 +258,85 @@ describe('AnthropicLM', () => { { role: 'assistant', content: 'reply' }, ]); + // Merged content is now a block array rather than a joined string: + // image turns can only be expressed as blocks, and merging on one + // representation instead of two is what keeps a text turn followed + // by an image turn from being sent as two adjacent user messages. + // The blank line between the two texts is preserved. expect(messages).toEqual([ - { role: 'user', content: 'one\n\ntwo' }, + { role: 'user', content: [{ type: 'text', text: 'one\n\ntwo' }] }, { role: 'assistant', content: 'reply' }, ]); }); + it('merges a text turn and an adjacent image turn into one user message', () => { + const { messages } = toAnthropicMessages([ + { role: 'user', content: 'what is this?' }, + { role: 'user', content: [imagePart(DATA_URI)] }, + { role: 'assistant', content: 'a logo' }, + ]); + + // Two adjacent user messages would be rejected outright: the + // Messages API requires strict user/assistant alternation. + expect(messages).toHaveLength(2); + expect(messages[0]).toEqual({ + role: 'user', + content: [ + { type: 'text', text: 'what is this?' }, + { + type: 'image', + source: { type: 'base64', media_type: 'image/png', data: PNG }, + }, + ], + }); + expect(messages[1].role).toBe('assistant'); + }); + + it('sends an image as a base64 block', () => { + const { messages } = toAnthropicMessages([ + { role: 'user', content: [textPart('look'), imagePart(DATA_URI)] }, + ]); + + expect(messages[0].content).toEqual([ + { type: 'text', text: 'look' }, + { + type: 'image', + source: { type: 'base64', media_type: 'image/png', data: PNG }, + }, + ]); + }); + + it('sends a remote image as a url block', () => { + const { messages } = toAnthropicMessages([ + { role: 'user', content: [imagePart('https://example.com/a.png')] }, + ]); + + expect(messages[0].content).toEqual([ + { type: 'image', source: { type: 'url', url: 'https://example.com/a.png' } }, + ]); + }); + + it('flattens an image in a system message, which is text-only', () => { + const { system } = toAnthropicMessages([ + { role: 'system', content: [textPart('logo: '), imagePart(DATA_URI)] }, + { role: 'user', content: 'hi' }, + ]); + + expect(system).toBe('logo: [image: image/png]'); + }); + + it('leaves a text-only conversation as plain strings', () => { + const { messages } = toAnthropicMessages([ + { role: 'user', content: 'hi' }, + { role: 'assistant', content: 'hello' }, + ]); + + expect(messages).toEqual([ + { role: 'user', content: 'hi' }, + { role: 'assistant', content: 'hello' }, + ]); + }); + it('does not mutate the caller array', () => { const input = [ { role: 'user' as const, content: 'a' }, @@ -308,6 +384,12 @@ describe('AnthropicLM', () => { expect(capabilities.supportsStreaming).toBe(true); expect(capabilities.supportsStructuredOutput).toBe(true); expect(capabilities.supportsFunctionCalling).toBe(true); + expect(capabilities.supportsVision).toBe(true); expect(capabilities.maxContextLength).toBe(1_000_000); }); + + it('reports no vision for the models that cannot read images', () => { + const haiku = new AnthropicLM({ apiKey: 'k', model: 'claude-3-5-haiku-latest' }); + expect(haiku.getCapabilities().supportsVision).toBe(false); + }); }); diff --git a/packages/anthropic/src/anthropic-lm.ts b/packages/anthropic/src/anthropic-lm.ts index 8d9b8c5..4cb392d 100644 --- a/packages/anthropic/src/anthropic-lm.ts +++ b/packages/anthropic/src/anthropic-lm.ts @@ -1,13 +1,24 @@ import { BaseLM, LMError, + contentToText, + normalizeImageSource, type ChatMessage, + type ImageContentPart, type LLMCallOptions, + type MessageContent, type ModelCapabilities, type StreamChunk, } from '@ts-dspy/core'; import Anthropic, { APIError } from '@anthropic-ai/sdk'; -import type { Message, MessageParam } from '@anthropic-ai/sdk/resources/messages'; +import type { + Base64ImageSource, + ContentBlockParam, + ImageBlockParam, + Message, + MessageParam, + TextBlockParam, +} from '@anthropic-ai/sdk/resources/messages'; /** Current Claude Opus. Model IDs are exact — never append a date suffix. */ export const DEFAULT_ANTHROPIC_MODEL = 'claude-opus-5'; @@ -216,7 +227,7 @@ export class AnthropicLM extends BaseLM { supportsStreaming: true, supportsStructuredOutput: true, supportsFunctionCalling: true, - supportsVision: true, + supportsVision: supportsVisionFor(this.model), maxContextLength: 1_000_000, supportedFormats: ['text', 'json_schema'], }; @@ -248,12 +259,21 @@ function textOf(message: Message): string { .join(''); } +/** Claude models that take text only; every other current model reads images. */ +const TEXT_ONLY_MODELS = [/^claude-3-5-haiku/, /^claude-2/, /^claude-instant/]; + +function supportsVisionFor(model: string): boolean { + return !TEXT_ONLY_MODELS.some((pattern) => pattern.test(model)); +} + /** * Convert ts-dspy messages into the Messages API shape. * * System messages become the top-level `system` parameter — Anthropic has no - * system role inside `messages`. Consecutive same-role turns are merged, since - * the API requires strict alternation. + * system role inside `messages`, and that parameter is text-only, so an image + * addressed to it is flattened to its placeholder rather than silently dropped. + * Consecutive same-role turns are merged, since the API requires strict + * alternation. */ export function toAnthropicMessages(messages: ChatMessage[]): { system?: string; @@ -264,17 +284,23 @@ export function toAnthropicMessages(messages: ChatMessage[]): { for (const message of messages) { if (message.role === 'system') { - systemParts.push(message.content); + systemParts.push(contentToText(message.content)); continue; } const role: 'user' | 'assistant' = message.role === 'assistant' ? 'assistant' : 'user'; + const content = toAnthropicContent(message.content); const previous = converted.at(-1); - if (previous?.role === role && typeof previous.content === 'string') { - previous.content = `${previous.content}\n\n${message.content}`; + // Merging happens on block arrays, not by string concatenation. The + // previous implementation merged only when both sides were strings, + // which meant a text turn followed by an image turn was pushed as two + // adjacent user messages — and the API rejects anything but strict + // alternation. + if (previous?.role === role) { + previous.content = mergeBlocks(toBlocks(previous.content), toBlocks(content)); } else { - converted.push({ role, content: message.content }); + converted.push({ role, content }); } } @@ -284,6 +310,64 @@ export function toAnthropicMessages(messages: ChatMessage[]): { }; } +function toAnthropicContent( + content: MessageContent +): string | Array { + if (typeof content === 'string') return content; + return content.map((part) => + part.type === 'text' + ? { type: 'text' as const, text: part.text } + : toAnthropicImage(part) + ); +} + +/** + * Images are either inline base64 with an explicit media type, or a URL the API + * fetches. A `data:` URI handed in as a URL is rewritten to the base64 form, + * which is the only shape `URLImageSource` will not accept. + */ +function toAnthropicImage(part: ImageContentPart): ImageBlockParam { + const source = normalizeImageSource(part.source); + if (source.kind === 'url') { + return { type: 'image', source: { type: 'url', url: source.url } }; + } + return { + type: 'image', + source: { + type: 'base64', + // The SDK narrows media_type to the four types the API accepts; + // ours stays open so a new one needs no core release. + media_type: source.mediaType as Base64ImageSource['media_type'], + data: source.data, + }, + }; +} + +function toBlocks(content: MessageParam['content']): ContentBlockParam[] { + return typeof content === 'string' ? [{ type: 'text', text: content }] : [...content]; +} + +/** + * Concatenate two turns' blocks, folding a text block that meets another text + * block into one. Without the fold the two turns would abut with no separator; + * the blank line is what the string merge used to provide. + */ +function mergeBlocks( + left: ContentBlockParam[], + right: ContentBlockParam[] +): ContentBlockParam[] { + const last = left.at(-1); + const first = right[0]; + if (last?.type === 'text' && first?.type === 'text') { + return [ + ...left.slice(0, -1), + { ...last, text: `${last.text}\n\n${first.text}` }, + ...right.slice(1), + ]; + } + return [...left, ...right]; +} + /** * Build sampling parameters. * diff --git a/packages/core/src/core/signature.test.ts b/packages/core/src/core/signature.test.ts index 0ba9584..8107942 100644 --- a/packages/core/src/core/signature.test.ts +++ b/packages/core/src/core/signature.test.ts @@ -1,4 +1,4 @@ -import { Signature, InputField, OutputField } from './signature'; +import { Signature, ImageField, InputField, OutputField, isImageFieldType } from './signature'; describe('Signature', () => { describe('parseStringSignature', () => { @@ -135,6 +135,25 @@ describe('Signature', () => { }); }); + describe('ImageField', () => { + it('records an input field typed image', () => { + class Caption extends Signature { + @ImageField({ description: 'the picture' }) + picture!: string; + + @OutputField({ description: 'one sentence' }) + caption!: string; + } + + const picture = Caption.getInputFields().picture; + expect(picture.type).toBe('image'); + expect(picture.description).toBe('the picture'); + expect(isImageFieldType(picture.type)).toBe(true); + // Images are inputs only: a model replies in text. + expect(Caption.getOutputFields().caption.type).toBe('string'); + }); + }); + describe('decorator mode', () => { it('explains what to change when legacy decorators are disabled', () => { // Under TC39 stage-3 decorators a field decorator is called with diff --git a/packages/core/src/core/signature.ts b/packages/core/src/core/signature.ts index 82662d0..4f55c83 100644 --- a/packages/core/src/core/signature.ts +++ b/packages/core/src/core/signature.ts @@ -1,5 +1,18 @@ import type { FieldConfig, ParsedSignature } from '../types/signature'; +/** + * Field type marking an input as an image rather than text. + * + * Only inputs may be images: a model returns text, so an output field declared + * `image` would be a promise nothing can keep. + */ +export const IMAGE_FIELD_TYPE = 'image'; + +/** True when a field config declares an image input. */ +export function isImageFieldType(type: string | undefined): boolean { + return type === IMAGE_FIELD_TYPE; +} + // Symbol keys for decorator metadata const INPUT_FIELDS = Symbol('inputFields'); const OUTPUT_FIELDS = Symbol('outputFields'); @@ -62,6 +75,31 @@ export function InputField(config: Partial = {}) { }; } +/** + * Declare an image input. Sugar for `@InputField({ type: 'image' })`. + * + * The decorated property holds an {@link ImageInput}: an `https://` URL, a + * `data:` URI, or an explicit source object. + * + * ```ts + * class DescribeReceipt extends Signature { + * @ImageField({ description: 'photo of the receipt' }) + * receipt!: ImageInput; + * + * @OutputField({ description: 'total charged', type: 'number' }) + * total!: number; + * } + * ``` + */ +export function ImageField(config: Omit, 'type'> = {}) { + return function (target: any, propertyKey: string | symbol | any) { + defineField('input', INPUT_FIELDS, target, propertyKey, { + ...config, + type: IMAGE_FIELD_TYPE, + }); + }; +} + export function OutputField(config: Partial = {}) { return function (target: any, propertyKey: string | symbol | any) { defineField('output', OUTPUT_FIELDS, target, propertyKey, config); diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 6911b30..2a98972 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -2,7 +2,14 @@ export * from './types'; // Core classes -export { Signature, InputField, OutputField } from './core/signature'; +export { + Signature, + InputField, + OutputField, + ImageField, + IMAGE_FIELD_TYPE, + isImageFieldType, +} from './core/signature'; export { Module } from './core/module'; export { BaseLM } from './core/base-lm'; export { Prediction } from './core/prediction'; @@ -20,5 +27,16 @@ export { RespAct } from './modules/respact'; export type { ToolFunction, ToolWithDescription, ToolDefinition } from './modules/respact'; // Utilities -export { buildPrompt, parseOutput } from './utils/parsing'; +export { buildPrompt, buildPromptContent, parseOutput } from './utils/parsing'; +export { + textPart, + imagePart, + isImagePart, + imageToUrl, + imageMediaType, + normalizeImageSource, + toContentParts, + hasImageContent, + contentToText, +} from './utils/content'; export { fieldConfigToZod, buildOutputSchema, buildOutputJsonSchema } from './utils/schema'; diff --git a/packages/core/src/test-utils.ts b/packages/core/src/test-utils.ts index 29be682..15570a2 100644 --- a/packages/core/src/test-utils.ts +++ b/packages/core/src/test-utils.ts @@ -1,4 +1,5 @@ import { BaseLM } from './core/base-lm'; +import { contentToText } from './utils/content'; import type { ChatMessage, LLMCallOptions, ModelCapabilities } from './types/language-model'; export interface MockLMOptions { @@ -71,9 +72,12 @@ export class MockLM extends BaseLM { return this.capabilities; } - /** The prompt text of the most recent chat call. */ + /** + * The prompt text of the most recent chat call. Image parts are flattened to + * their `[image: …]` placeholder so this stays a string. + */ lastPrompt(): string { const last = this.calls.at(-1); - return last?.messages.map((message) => message.content).join('\n') ?? ''; + return last?.messages.map((message) => contentToText(message.content)).join('\n') ?? ''; } } diff --git a/packages/core/src/types/language-model.ts b/packages/core/src/types/language-model.ts index 866deed..dc93863 100644 --- a/packages/core/src/types/language-model.ts +++ b/packages/core/src/types/language-model.ts @@ -19,9 +19,69 @@ export interface LLMCallOptions { metadata?: Record; } +/** + * Media type of an image. The four listed types are the intersection every + * provider accepts; the open arm keeps the union accepting anything a provider + * adds later, without giving up autocompletion on the four. + */ +export type ImageMediaType = + 'image/jpeg' | 'image/png' | 'image/gif' | 'image/webp' | (string & Record); + +/** An image fetched by the provider from a URL. */ +export interface ImageUrlSource { + kind: 'url'; + /** An `https://` URL, or a `data:image/…;base64,…` URI. */ + url: string; + /** Optional hint; providers that need one infer it from a data URI. */ + mediaType?: ImageMediaType; +} + +/** An image carried inline as base64. */ +export interface ImageBase64Source { + kind: 'base64'; + /** Base64 payload only — no `data:` prefix. */ + data: string; + mediaType: ImageMediaType; +} + +export type ImageSource = ImageUrlSource | ImageBase64Source; + +export interface TextContentPart { + type: 'text'; + text: string; +} + +export interface ImageContentPart { + type: 'image'; + source: ImageSource; + /** + * Fidelity hint. Only OpenAI has an equivalent (`image_url.detail`); other + * providers ignore it. + */ + detail?: 'auto' | 'low' | 'high'; +} + +/** One piece of a multimodal message. */ +export type ContentPart = TextContentPart | ImageContentPart; + +/** + * The content of a chat message. + * + * Plain `string` is still a valid content: every text-only call site keeps + * working unchanged, and providers keep sending a bare string when that is all + * they were given. + */ +export type MessageContent = string | ContentPart[]; + +/** + * What callers may hand to an image field or {@link imagePart}: an `https://` + * URL, a `data:` URI, an explicit {@link ImageSource}, or a ready-made part. + */ +export type ImageInput = string | ImageSource | ImageContentPart; + export interface ChatMessage { role: 'system' | 'user' | 'assistant' | 'function' | 'tool'; - content: string; + content: MessageContent; name?: string; functionCall?: { name: string; diff --git a/packages/core/src/utils/content.test.ts b/packages/core/src/utils/content.test.ts new file mode 100644 index 0000000..ad56535 --- /dev/null +++ b/packages/core/src/utils/content.test.ts @@ -0,0 +1,140 @@ +import { TsDspyError } from '../core/errors'; +import { + contentToText, + hasImageContent, + imageMediaType, + imagePart, + imageToUrl, + isImagePart, + normalizeImageSource, + textPart, + toContentParts, +} from './content'; + +const PNG = 'iVBORw0KGgo='; +const DATA_URI = `data:image/png;base64,${PNG}`; + +describe('imagePart', () => { + it('splits a data URI into its media type and payload', () => { + expect(imagePart(DATA_URI)).toEqual({ + type: 'image', + source: { kind: 'base64', mediaType: 'image/png', data: PNG }, + }); + }); + + it('treats an http(s) string as a URL source', () => { + expect(imagePart('https://example.com/cat.jpg')).toEqual({ + type: 'image', + source: { kind: 'url', url: 'https://example.com/cat.jpg' }, + }); + }); + + it('rejects a bare base64 blob, which carries no media type', () => { + expect(() => imagePart(PNG)).toThrow(TsDspyError); + expect(() => imagePart(PNG)).toThrow(/media type/); + }); + + it('rejects a data URI that is not an image', () => { + expect(() => imagePart('data:application/pdf;base64,JVBERi0=')).toThrow( + /not an image media type/ + ); + }); + + it('parses a data URI carrying extra parameters', () => { + expect(imagePart(`data:image/jpeg;name=sign.jpg;base64,${PNG}`).source).toEqual({ + kind: 'base64', + mediaType: 'image/jpeg', + data: PNG, + }); + }); + + it('says what it expected when handed something that is not an image at all', () => { + expect(() => imagePart(42 as never)).toThrow(TsDspyError); + expect(() => imagePart(null as never)).toThrow(/Expected an image URL/); + }); + + it('wraps an explicit source', () => { + expect(imagePart({ kind: 'base64', data: PNG, mediaType: 'image/webp' })).toEqual({ + type: 'image', + source: { kind: 'base64', data: PNG, mediaType: 'image/webp' }, + }); + }); + + it('returns a ready-made part unchanged', () => { + const part = imagePart(DATA_URI); + expect(imagePart(part)).toBe(part); + }); + + it('applies a detail hint without mutating the original part', () => { + const part = imagePart(DATA_URI); + expect(imagePart(part, 'low').detail).toBe('low'); + expect(part.detail).toBeUndefined(); + }); +}); + +describe('image sources', () => { + it('renders a base64 source as a data URI', () => { + expect(imageToUrl({ kind: 'base64', data: PNG, mediaType: 'image/png' })).toBe( + DATA_URI + ); + }); + + it('passes a URL through untouched', () => { + expect(imageToUrl({ kind: 'url', url: 'https://example.com/a.png' })).toBe( + 'https://example.com/a.png' + ); + }); + + it('rewrites a data URI given as a URL source into base64 form', () => { + expect(normalizeImageSource({ kind: 'url', url: DATA_URI })).toEqual({ + kind: 'base64', + mediaType: 'image/png', + data: PNG, + }); + }); + + it('leaves a real URL alone', () => { + const source = { kind: 'url' as const, url: 'https://example.com/a.png' }; + expect(normalizeImageSource(source)).toBe(source); + }); + + it('reads the media type out of a data URI, and reports none for a bare URL', () => { + expect(imageMediaType({ kind: 'url', url: DATA_URI })).toBe('image/png'); + expect(imageMediaType({ kind: 'url', url: 'https://example.com/a' })).toBeUndefined(); + }); +}); + +describe('content helpers', () => { + it('wraps a string in a single text part', () => { + expect(toContentParts('hi')).toEqual([{ type: 'text', text: 'hi' }]); + }); + + it('reports whether content carries an image', () => { + expect(hasImageContent('hi')).toBe(false); + expect(hasImageContent([textPart('hi')])).toBe(false); + expect(hasImageContent([textPart('hi'), imagePart(DATA_URI)])).toBe(true); + }); + + it('identifies image parts', () => { + expect(isImagePart(imagePart(DATA_URI))).toBe(true); + expect(isImagePart(textPart('hi'))).toBe(false); + }); + + it('flattens parts to text, standing an image in for its pixels', () => { + const flattened = contentToText([ + textPart('look: '), + imagePart(DATA_URI), + textPart('\nwhat is it?'), + ]); + + expect(flattened).toBe('look: [image: image/png]\nwhat is it?'); + }); + + it('falls back to a bare placeholder when the media type is unknown', () => { + expect(contentToText([imagePart('https://example.com/a')])).toBe('[image]'); + }); + + it('returns a string content unchanged', () => { + expect(contentToText('plain')).toBe('plain'); + }); +}); diff --git a/packages/core/src/utils/content.ts b/packages/core/src/utils/content.ts new file mode 100644 index 0000000..427a9c7 --- /dev/null +++ b/packages/core/src/utils/content.ts @@ -0,0 +1,139 @@ +import { TsDspyError } from '../core/errors'; +import type { + ContentPart, + ImageContentPart, + ImageInput, + ImageMediaType, + ImageSource, + MessageContent, + TextContentPart, +} from '../types/language-model'; + +/** + * `data:image/png;base64,iVBOR…` — media type, any further parameters, payload. + * The middle group exists so a URI carrying e.g. `;name=sign.png` still parses + * rather than being mistaken for something that is not a data URI at all. + */ +const DATA_URI = /^data:([^;,]+)(;[^,]*)?;base64,([\s\S]*)$/; + +/** Build a text part. Mostly sugar, but it keeps call sites symmetrical. */ +export function textPart(text: string): TextContentPart { + return { type: 'text', text }; +} + +/** + * Normalise anything a caller may reasonably hand us into an image part. + * + * Accepts an `https://` URL, a `data:` URI, an explicit {@link ImageSource}, or + * an already-built part. A bare base64 blob is rejected: without a media type + * no provider can be told what it is, and guessing from the payload's first + * bytes is the kind of cleverness that fails silently in production. + */ +export function imagePart( + image: ImageInput, + detail?: ImageContentPart['detail'] +): ImageContentPart { + const part = toImagePart(image); + return detail === undefined ? part : { ...part, detail }; +} + +function toImagePart(image: ImageInput): ImageContentPart { + if (typeof image === 'string') { + return { type: 'image', source: sourceFromString(image) }; + } + // Inputs arrive from user-supplied records, so a number or null can reach + // here; say what was expected rather than failing inside `in`. + if (typeof image !== 'object' || image === null) { + throw new TsDspyError( + `Expected an image URL, a data URI, or an image source object, got ${typeof image}.` + ); + } + // `kind` is the source discriminant; a part has `type: 'image'` instead. + if ('kind' in image) { + return { type: 'image', source: image }; + } + return image; +} + +function sourceFromString(value: string): ImageSource { + const dataUri = value.match(DATA_URI); + if (dataUri) { + const mediaType = dataUri[1]; + if (!mediaType.startsWith('image/')) { + throw new TsDspyError( + `"${mediaType}" is not an image media type. Only images can be sent as ` + + 'image content; a provider would reject anything else.' + ); + } + return { kind: 'base64', mediaType, data: dataUri[3] }; + } + if (/^https?:\/\//i.test(value)) { + return { kind: 'url', url: value }; + } + throw new TsDspyError( + 'Image strings must be an http(s) URL or a "data:;base64,…" URI. ' + + 'For raw base64, pass { kind: "base64", data, mediaType } so the media type is known.' + ); +} + +/** + * Rewrite a `data:` URI carried in a URL source as a base64 source. + * + * Providers that take inline bytes (Anthropic, Gemini) need the media type and + * payload separately, and a caller is free to hand us a data URI either way + * round. Anything else passes through untouched. + */ +export function normalizeImageSource(source: ImageSource): ImageSource { + if (source.kind !== 'url') return source; + const dataUri = source.url.match(DATA_URI); + if (!dataUri) return source; + return { kind: 'base64', mediaType: dataUri[1], data: dataUri[3] }; +} + +export function isImagePart(part: ContentPart): part is ImageContentPart { + return part.type === 'image'; +} + +/** Render an image source as a `data:` URI, or pass a URL straight through. */ +export function imageToUrl(source: ImageSource): string { + return source.kind === 'url' + ? source.url + : `data:${source.mediaType};base64,${source.data}`; +} + +/** + * Media type of an image source, inferred from a `data:` URI when the source is + * a URL that carries one. `undefined` when it genuinely cannot be known. + */ +export function imageMediaType(source: ImageSource): ImageMediaType | undefined { + if (source.kind === 'base64') return source.mediaType; + if (source.mediaType) return source.mediaType; + return source.url.match(DATA_URI)?.[1]; +} + +/** Content as an array of parts, wrapping a plain string in a single text part. */ +export function toContentParts(content: MessageContent): ContentPart[] { + return typeof content === 'string' ? [textPart(content)] : content; +} + +/** True when the content carries at least one image. */ +export function hasImageContent(content: MessageContent): boolean { + return typeof content !== 'string' && content.some(isImagePart); +} + +/** + * Flatten content to text, replacing images with a short placeholder. + * + * Used wherever a channel cannot carry an image at all — Anthropic's `system` + * parameter, OpenAI's system and assistant roles — so an image degrades to a + * visible marker rather than `[object Object]`. + */ +export function contentToText(content: MessageContent): string { + if (typeof content === 'string') return content; + return content.map((part) => (isImagePart(part) ? placeholder(part) : part.text)).join(''); +} + +function placeholder(part: ImageContentPart): string { + const mediaType = imageMediaType(part.source); + return mediaType ? `[image: ${mediaType}]` : '[image]'; +} diff --git a/packages/core/src/utils/parsing.test.ts b/packages/core/src/utils/parsing.test.ts index b8886c9..9e3a5bb 100644 --- a/packages/core/src/utils/parsing.test.ts +++ b/packages/core/src/utils/parsing.test.ts @@ -1,5 +1,5 @@ -import { buildPrompt, parseOutput } from './parsing'; -import { Signature, InputField, OutputField } from '../core/signature'; +import { buildPrompt, buildPromptContent, parseOutput } from './parsing'; +import { Signature, ImageField, InputField, OutputField } from '../core/signature'; import { ValidationError } from '../core/errors'; describe('Parsing Utils', () => { @@ -192,4 +192,80 @@ describe('Parsing Utils', () => { ); }); }); + + describe('buildPromptContent', () => { + const PNG = 'iVBORw0KGgo='; + const DATA_URI = `data:image/png;base64,${PNG}`; + + class ReadSign extends Signature { + static description = 'Read the sign in the photo.'; + + @ImageField({ description: 'photo of the sign' }) + photo!: string; + + @InputField({ description: 'the language to answer in' }) + language!: string; + + @OutputField({ description: 'the words on the sign' }) + words!: string; + } + + it('returns a plain string when every input is text', () => { + const content = buildPromptContent('question -> answer', { question: 'why?' }); + + expect(content).toBe(buildPrompt('question -> answer', { question: 'why?' })); + expect(typeof content).toBe('string'); + }); + + it('emits content parts when a class input field is an image', () => { + const content = buildPromptContent(ReadSign, { + photo: DATA_URI, + language: 'French', + }); + + expect(content).toEqual([ + { type: 'text', text: 'Read the sign in the photo.\n\nphoto: ' }, + { + type: 'image', + source: { kind: 'base64', mediaType: 'image/png', data: PNG }, + }, + { + type: 'text', + text: '\nlanguage: French\n\nProvide:\nwords (the words on the sign):', + }, + ]); + }); + + it('emits content parts for a string signature declaring an image input', () => { + const content = buildPromptContent('photo: image, question -> answer', { + photo: 'https://example.com/sign.png', + question: 'what does it say?', + }); + + expect(Array.isArray(content)).toBe(true); + expect(content[1]).toEqual({ + type: 'image', + source: { kind: 'url', url: 'https://example.com/sign.png' }, + }); + }); + + it('skips an image input that was not supplied', () => { + const content = buildPromptContent(ReadSign, { language: 'French' }); + + expect(content).toBe( + 'Read the sign in the photo.\n\nlanguage: French\n\n' + + 'Provide:\nwords (the words on the sign):' + ); + }); + }); + + describe('buildPrompt with images', () => { + it('flattens an image input to a placeholder, since a string has no pixels', () => { + const prompt = buildPrompt('photo: image -> caption', { + photo: 'data:image/png;base64,iVBORw0KGgo=', + }); + + expect(prompt).toContain('photo: [image: image/png]'); + }); + }); }); diff --git a/packages/core/src/utils/parsing.ts b/packages/core/src/utils/parsing.ts index ca80da0..8e5c0df 100644 --- a/packages/core/src/utils/parsing.ts +++ b/packages/core/src/utils/parsing.ts @@ -1,69 +1,165 @@ -import { Signature } from '../core/signature'; +import { Signature, isImageFieldType } from '../core/signature'; import { ValidationError, type FieldValidationIssue } from '../core/errors'; +import type { ContentPart, ImageInput, MessageContent } from '../types/language-model'; +import { contentToText, imagePart, textPart } from './content'; import { buildOutputSchema, getOutputFieldConfigs } from './schema'; +/** + * Render a signature and its inputs as a plain-text prompt. + * + * Image inputs are flattened to a `[image: …]` placeholder, because a string + * cannot carry pixels. Use {@link buildPromptContent} to send them for real. + */ export function buildPrompt( signature: typeof Signature | string, inputs: Record ): string { - if (typeof signature === 'string') { - return buildPromptFromString(signature, inputs); + return contentToText(buildPromptContent(signature, inputs)); +} + +/** + * Render a signature and its inputs as chat message content. + * + * Returns a plain `string` when every input is text — identical to what + * {@link buildPrompt} produces — and an array of {@link ContentPart}s when any + * input field is declared `image`, so the image travels as an image: + * + * ```ts + * const content = buildPromptContent(DescribeReceipt, { receipt: dataUri }); + * await lm.chat([{ role: 'user', content }]); + * ``` + */ +export function buildPromptContent( + signature: typeof Signature | string, + inputs: Record +): MessageContent { + const parts = + typeof signature === 'string' + ? buildPartsFromString(signature, inputs) + : buildPartsFromClass(signature, inputs); + return collapse(parts); +} + +/** + * Accumulates prompt text, breaking it into parts wherever an image lands. + * + * Text is buffered so that a prompt without images ends up as exactly one part + * holding exactly the string the old string-only builder produced. + */ +function partBuilder() { + const parts: ContentPart[] = []; + let buffer = ''; + + const flush = () => { + if (buffer !== '') { + parts.push(textPart(buffer)); + buffer = ''; + } + }; + + return { + text(chunk: string): void { + buffer += chunk; + }, + image(value: ImageInput): void { + flush(); + parts.push(imagePart(value)); + }, + done(): ContentPart[] { + flush(); + return trimEnds(parts); + }, + }; +} + +/** Mirror the trailing `prompt.trim()` the string builders used to end with. */ +function trimEnds(parts: ContentPart[]): ContentPart[] { + const trimmed = [...parts]; + const first = trimmed[0]; + if (first?.type === 'text') { + trimmed[0] = textPart(first.text.replace(/^\s+/, '')); } - return buildPromptFromClass(signature, inputs); + const last = trimmed[trimmed.length - 1]; + if (last?.type === 'text') { + trimmed[trimmed.length - 1] = textPart(last.text.replace(/\s+$/, '')); + } + return trimmed.filter((part) => part.type !== 'text' || part.text !== ''); } -function buildPromptFromString(signatureStr: string, inputs: Record): string { - const parsed = Signature.parseStringSignature(signatureStr); +/** One text part is just a string; anything else stays a part array. */ +function collapse(parts: ContentPart[]): MessageContent { + if (parts.length === 0) return ''; + if (parts.length === 1 && parts[0].type === 'text') return parts[0].text; + return parts; +} - let prompt = ''; +function buildPartsFromString( + signatureStr: string, + inputs: Record +): ContentPart[] { + const parsed = Signature.parseStringSignature(signatureStr); + const prompt = partBuilder(); for (const inputKey of parsed.inputs) { if (inputs[inputKey] !== undefined) { - prompt += `${inputKey}: ${inputs[inputKey]}\n`; + prompt.text(`${inputKey}: `); + if (isImageFieldType(parsed.types[inputKey])) { + prompt.image(inputs[inputKey]); + } else { + prompt.text(`${inputs[inputKey]}`); + } + prompt.text('\n'); } } if (parsed.outputs.length === 1) { const outputKey = parsed.outputs[0]; - prompt += `\nProvide the ${outputKey} in this format:\n${outputKey}: [your response]`; + prompt.text( + `\nProvide the ${outputKey} in this format:\n${outputKey}: [your response]` + ); } else { - prompt += '\nProvide the following fields:\n'; + prompt.text('\nProvide the following fields:\n'); for (const outputKey of parsed.outputs) { const typeInfo = parsed.types[outputKey] ? ` (${parsed.types[outputKey]})` : ''; - prompt += `${outputKey}${typeInfo}: [your response]\n`; + prompt.text(`${outputKey}${typeInfo}: [your response]\n`); } } - return prompt.trim(); + return prompt.done(); } -function buildPromptFromClass( +function buildPartsFromClass( signatureClass: typeof Signature, inputs: Record -): string { +): ContentPart[] { const inputFields = signatureClass.getInputFields(); const outputFields = signatureClass.getOutputFields(); - - let prompt = ''; + const prompt = partBuilder(); if (signatureClass.description) { - prompt += `${signatureClass.description}\n\n`; + prompt.text(`${signatureClass.description}\n\n`); } Object.entries(inputFields).forEach(([key, config]) => { if (inputs[key] !== undefined) { const prefix = config.prefix || `${key}:`; - prompt += `${prefix} ${inputs[key]}\n`; + prompt.text(`${prefix} `); + if (isImageFieldType(config.type)) { + prompt.image(inputs[key]); + } else { + prompt.text(`${inputs[key]}`); + } + prompt.text('\n'); } }); - prompt += '\nProvide:\n'; + prompt.text('\nProvide:\n'); Object.entries(outputFields).forEach(([key, config]) => { const desc = config.description ? ` (${config.description})` : ''; - prompt += `${key}${desc}:\n`; + prompt.text(`${key}${desc}:\n`); }); - return prompt.trim(); + return prompt.done(); } /** diff --git a/packages/gemini/src/gemini-lm.test.ts b/packages/gemini/src/gemini-lm.test.ts index 91a8a0b..2886061 100644 --- a/packages/gemini/src/gemini-lm.test.ts +++ b/packages/gemini/src/gemini-lm.test.ts @@ -1,4 +1,4 @@ -import { LMError } from '@ts-dspy/core'; +import { LMError, imagePart, textPart } from '@ts-dspy/core'; import { GeminiLM, toGeminiContents, DEFAULT_GEMINI_MODEL } from './gemini-lm'; const mocks = vi.hoisted(() => ({ @@ -26,6 +26,9 @@ vi.mock('@google/genai', () => ({ HarmBlockThreshold: { BLOCK_MEDIUM_AND_ABOVE: 'BLOCK_MEDIUM_AND_ABOVE' }, })); +const PNG = 'iVBORw0KGgo='; +const DATA_URI = `data:image/png;base64,${PNG}`; + function response(text: string, extra: Record = {}) { return { text, @@ -124,6 +127,48 @@ describe('GeminiLM', () => { expect(systemInstruction).toBe('be terse'); expect(contents).toHaveLength(1); }); + + it('sends inline bytes as inlineData alongside the text part', () => { + const { contents } = toGeminiContents([ + { role: 'user', content: [textPart('what is this?'), imagePart(DATA_URI)] }, + ]); + + expect(contents).toEqual([ + { + role: 'user', + parts: [ + { text: 'what is this?' }, + { inlineData: { mimeType: 'image/png', data: PNG } }, + ], + }, + ]); + }); + + it('sends a Files API URI as fileData', () => { + const uri = 'https://generativelanguage.googleapis.com/v1beta/files/abc123'; + const { contents } = toGeminiContents([ + { role: 'user', content: [imagePart(uri)] }, + ]); + + expect(contents[0].parts).toEqual([{ fileData: { fileUri: uri } }]); + }); + + it('refuses an arbitrary web URL, which fileData cannot dereference', () => { + expect(() => + toGeminiContents([ + { role: 'user', content: [imagePart('https://example.com/a.png')] }, + ]) + ).toThrow(LMError); + }); + + it('flattens an image in a system message, which is text-only', () => { + const { systemInstruction } = toGeminiContents([ + { role: 'system', content: [textPart('logo: '), imagePart(DATA_URI)] }, + { role: 'user', content: 'hi' }, + ]); + + expect(systemInstruction).toBe('logo: [image: image/png]'); + }); }); describe('safety blocking', () => { @@ -268,9 +313,15 @@ describe('GeminiLM', () => { expect(capabilities.supportsStreaming).toBe(true); expect(capabilities.supportsStructuredOutput).toBe(true); expect(capabilities.supportsFunctionCalling).toBe(true); + expect(capabilities.supportsVision).toBe(true); // The old implementation hardcoded 32768 with a "Gemini 1.0 Pro" comment. expect(capabilities.maxContextLength).toBe(1_000_000); }); + + it('reports no vision for embedding models', () => { + const lm = new GeminiLM({ apiKey: 'k', model: 'gemini-embedding-001' }); + expect(lm.getCapabilities().supportsVision).toBe(false); + }); }); it('wraps SDK failures in LMError and counts them', async () => { diff --git a/packages/gemini/src/gemini-lm.ts b/packages/gemini/src/gemini-lm.ts index 363595a..cf24230 100644 --- a/packages/gemini/src/gemini-lm.ts +++ b/packages/gemini/src/gemini-lm.ts @@ -1,8 +1,13 @@ import { BaseLM, LMError, + contentToText, + imageMediaType, + normalizeImageSource, type ChatMessage, + type ImageContentPart, type LLMCallOptions, + type MessageContent, type ModelCapabilities, type StreamChunk, } from '@ts-dspy/core'; @@ -13,6 +18,7 @@ import { type Content, type GenerateContentConfig, type GenerateContentResponse, + type Part, type SafetySetting, } from '@google/genai'; @@ -46,6 +52,11 @@ const DEFAULT_SAFETY_SETTINGS: SafetySetting[] = [ HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT, ].map((category) => ({ category, threshold: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE })); +/** Embedding models take text only; every generative Gemini model reads images. */ +function supportsVisionFor(model: string): boolean { + return !model.includes('embedding'); +} + /** Context windows by model family; the 1M default matches current Gemini models. */ function contextLengthFor(model: string): number { if (model.includes('flash-lite')) return 1_000_000; @@ -145,7 +156,7 @@ export class GeminiLM extends BaseLM { supportsStreaming: true, supportsStructuredOutput: true, supportsFunctionCalling: true, - supportsVision: true, + supportsVision: supportsVisionFor(this.model), maxContextLength: contextLengthFor(this.model), supportedFormats: ['text', 'json_object', 'json_schema'], }; @@ -254,12 +265,14 @@ export function toGeminiContents(messages: ChatMessage[]): { for (const message of messages) { if (message.role === 'system') { - systemParts.push(message.content); + // `systemInstruction` is text-only, so an image aimed at it is + // flattened to its placeholder rather than silently dropped. + systemParts.push(contentToText(message.content)); continue; } contents.push({ role: message.role === 'assistant' ? 'model' : 'user', - parts: [{ text: message.content }], + parts: toGeminiParts(message.content), }); } @@ -269,6 +282,40 @@ export function toGeminiContents(messages: ChatMessage[]): { }; } +function toGeminiParts(content: MessageContent): Part[] { + if (typeof content === 'string') return [{ text: content }]; + return content.map((part) => + part.type === 'text' ? { text: part.text } : toGeminiImage(part) + ); +} + +/** URI schemes `fileData` can actually dereference. */ +const FILE_URI_PATTERN = /^(gs:\/\/|https:\/\/generativelanguage\.googleapis\.com\/)/i; + +/** + * Inline bytes go in `inlineData`; a URI the API can resolve goes in `fileData`. + * A `data:` URI arriving as a URL is rewritten to inline bytes, since `fileData` + * cannot dereference one either. + * + * Unlike OpenAI and Anthropic, Gemini will not fetch an arbitrary web URL, so + * one is refused here with an explanation rather than sent on to earn a 400. + */ +function toGeminiImage(part: ImageContentPart): Part { + const source = normalizeImageSource(part.source); + if (source.kind === 'url') { + if (!FILE_URI_PATTERN.test(source.url)) { + throw new LMError( + 'gemini', + `Gemini cannot fetch "${source.url}": fileData accepts a Files API or ` + + 'Cloud Storage URI. Pass the image as base64 bytes instead.' + ); + } + const mimeType = imageMediaType(source); + return { fileData: { fileUri: source.url, ...(mimeType ? { mimeType } : {}) } }; + } + return { inlineData: { mimeType: source.mediaType, data: source.data } }; +} + function toLMError(error: unknown): LMError { if (error instanceof LMError) return error; const message = error instanceof Error ? error.message : String(error); diff --git a/packages/openai/src/openai-lm.test.ts b/packages/openai/src/openai-lm.test.ts index 578f133..0eaadba 100644 --- a/packages/openai/src/openai-lm.test.ts +++ b/packages/openai/src/openai-lm.test.ts @@ -1,5 +1,9 @@ import { LMError } from '@ts-dspy/core'; import { OpenAILM, toOpenAIMessages, DEFAULT_OPENAI_MODEL } from './openai-lm'; +import { imagePart, textPart } from '@ts-dspy/core'; + +const PNG = 'iVBORw0KGgo='; +const DATA_URI = `data:image/png;base64,${PNG}`; // Everything the hoisted vi.mock factory touches must itself be hoisted. const mocks = vi.hoisted(() => { @@ -222,6 +226,23 @@ describe('OpenAILM', () => { expect(capabilities.supportsStreaming).toBe(true); expect(capabilities.supportsStructuredOutput).toBe(true); expect(capabilities.supportsFunctionCalling).toBe(true); + expect(capabilities.supportsVision).toBe(true); + }); + + it('reports no vision for text-only model families', () => { + const models = [ + 'gpt-3.5-turbo', + 'gpt-4-0613', + 'gpt-4-32k', + 'o1-mini', + 'o1-preview', + 'o3-mini', + ]; + for (const model of models) { + expect( + new OpenAILM({ apiKey: 'k', model }).getCapabilities().supportsVision + ).toBe(false); + } }); it('reports a context length matching the configured model', () => { @@ -264,5 +285,65 @@ describe('OpenAILM', () => { toOpenAIMessages(messages); expect(messages).toHaveLength(1); }); + + it('sends a user image as an image_url part', () => { + expect( + toOpenAIMessages([ + { role: 'user', content: [textPart('what is this?'), imagePart(DATA_URI)] }, + ]) + ).toEqual([ + { + role: 'user', + content: [ + { type: 'text', text: 'what is this?' }, + { type: 'image_url', image_url: { url: DATA_URI } }, + ], + }, + ]); + }); + + it('passes a remote URL and a detail hint straight through', () => { + const [message] = toOpenAIMessages([ + { + role: 'user', + content: [imagePart('https://example.com/a.png', 'low')], + }, + ]); + + expect(message.content).toEqual([ + { + type: 'image_url', + image_url: { url: 'https://example.com/a.png', detail: 'low' }, + }, + ]); + }); + + it('flattens images in system and assistant turns, which take text only', () => { + // ChatCompletionMessageParam is a discriminated union: only the user + // variant accepts image parts, so anything else has to degrade to text. + expect( + toOpenAIMessages([ + { role: 'system', content: [textPart('logo: '), imagePart(DATA_URI)] }, + { role: 'assistant', content: [textPart('seen')] }, + ]) + ).toEqual([ + { role: 'system', content: 'logo: [image: image/png]' }, + { role: 'assistant', content: 'seen' }, + ]); + }); + + it('sends an image through a chat call', async () => { + mocks.create.mockResolvedValue(completion('a logo')); + await new OpenAILM({ apiKey: 'k' }).chat([ + { role: 'user', content: [imagePart(DATA_URI)] }, + ]); + + expect(mocks.create.mock.calls[0][0].messages).toEqual([ + { + role: 'user', + content: [{ type: 'image_url', image_url: { url: DATA_URI } }], + }, + ]); + }); }); }); diff --git a/packages/openai/src/openai-lm.ts b/packages/openai/src/openai-lm.ts index b2257ef..5c465b2 100644 --- a/packages/openai/src/openai-lm.ts +++ b/packages/openai/src/openai-lm.ts @@ -1,13 +1,21 @@ import { BaseLM, LMError, + contentToText, + imageToUrl, type ChatMessage, + type ImageContentPart, type LLMCallOptions, + type MessageContent, type ModelCapabilities, type StreamChunk, } from '@ts-dspy/core'; import OpenAI, { APIError } from 'openai'; -import type { ChatCompletionMessageParam } from 'openai/resources/chat/completions'; +import type { + ChatCompletionContentPart, + ChatCompletionContentPartImage, + ChatCompletionMessageParam, +} from 'openai/resources/chat/completions'; /** * Current default. Confirm against `client.models.list()` if you need a specific @@ -48,6 +56,25 @@ function contextLengthFor(model: string): number { return 128_000; } +/** + * Model families that take text only. Everything else in the current lineup + * accepts images, so this is a deny-list: a new vision model works on the day it + * ships, and the flag stops claiming vision for models that never had it. + */ +const TEXT_ONLY_PREFIXES = [ + 'gpt-3.5', + 'gpt-4-0', // gpt-4-0314 / gpt-4-0613, before vision + 'gpt-4-32k', + 'o1-mini', + 'o1-preview', + 'o3-mini', + 'text-', +]; + +function supportsVisionFor(model: string): boolean { + return !TEXT_ONLY_PREFIXES.some((prefix) => model.startsWith(prefix)); +} + export class OpenAILM extends BaseLM { private readonly client: OpenAI; @@ -211,7 +238,7 @@ export class OpenAILM extends BaseLM { supportsStreaming: true, supportsStructuredOutput: true, supportsFunctionCalling: true, - supportsVision: true, + supportsVision: supportsVisionFor(this.model), maxContextLength: contextLengthFor(this.model), supportedFormats: ['text', 'json_object', 'json_schema'], }; @@ -223,24 +250,54 @@ export class OpenAILM extends BaseLM { } } +/** + * Convert ts-dspy messages into the Chat Completions shape. + * + * `ChatCompletionMessageParam` is a discriminated union in which only the + * `user` variant accepts image parts, so the conversion is role-aware: system + * and assistant content is flattened to text (an image there would be rejected + * by the API), and only user turns keep their parts. + */ export function toOpenAIMessages(messages: ChatMessage[]): ChatCompletionMessageParam[] { return messages.map((message) => { switch (message.role) { case 'system': - return { role: 'system', content: message.content }; + return { role: 'system', content: contentToText(message.content) }; case 'assistant': - return { role: 'assistant', content: message.content }; + return { role: 'assistant', content: contentToText(message.content) }; case 'tool': case 'function': // The core ChatMessage shape has no tool_call_id, so a tool // result is surfaced as user content rather than dropped. - return { role: 'user', content: message.content }; + return { role: 'user', content: toOpenAIUserContent(message.content) }; default: - return { role: 'user', content: message.content }; + return { role: 'user', content: toOpenAIUserContent(message.content) }; } }); } +/** User content: a plain string stays a string, parts become the SDK's part union. */ +function toOpenAIUserContent(content: MessageContent): string | ChatCompletionContentPart[] { + if (typeof content === 'string') return content; + return content.map((part) => + part.type === 'text' ? { type: 'text' as const, text: part.text } : toOpenAIImage(part) + ); +} + +/** + * Images travel in `image_url.url`, which accepts an `https://` URL or a + * `data:image/…;base64,…` URI — so a base64 source is rendered as a data URI. + */ +function toOpenAIImage(part: ImageContentPart): ChatCompletionContentPartImage { + return { + type: 'image_url', + image_url: { + url: imageToUrl(part.source), + ...(part.detail ? { detail: part.detail } : {}), + }, + }; +} + /** * Build sampling parameters. * diff --git a/site/docs.html b/site/docs.html index e78ae13..b8ed7ae 100644 --- a/site/docs.html +++ b/site/docs.html @@ -64,6 +64,7 @@

Everything the
library does.

  • Testing
  • API reference
  • Migrating to 0.5
    • +
  • Images & multimodal
    • @@ -591,6 +592,125 @@

    Migrating to 0.5

    +
    +

    19

    +

    Images and multimodal

    +

    + All three providers read images, and until now there was no way to send + one. ChatMessage.content was a string and every + converter passed it straight through, so supportsVision: true + was a claim the library could not honour. Content is now + string | ContentPart[] — a plain string still means + exactly what it always did, and an array carries text and images in order. +

    + 
    import { imagePart, textPart, type ChatMessage } from '@ts-dspy/core'
+
+const messages: ChatMessage[] = [
+  { role: 'user', content: [
+    textPart('What does this sign say?'),
+    imagePart('data:image/png;base64,iVBORw0KGgo…'),
+  ] },
+]
+
+await lm.chat(messages)
    +

    + imagePart() accepts an https:// URL, a + data: URI, or an explicit source such as + { kind: 'base64', data, mediaType }. A bare base64 blob is + refused: nothing in it says whether the bytes are a PNG or a JPEG, and + every provider insists on being told. +

    + +

    Image input fields

    +

    + A signature can declare an input as an image — with + @ImageField, or the image type in a string + signature. buildPromptContent() then renders the prompt as + content parts, with the image sitting where its label falls. +

    + 
    import { Signature, ImageField, OutputField, buildPromptContent } from '@ts-dspy/core'
+
+class ReadSign extends Signature {
+  static description = 'Read the sign in the photo.'
+
+  @ImageField({ description: 'photo of the sign' })
+  photo!: string
+
+  @OutputField({ description: 'the words on the sign' })
+  words!: string
+}
+
+const content = buildPromptContent(ReadSign, { photo: dataUri })
+await lm.chat([{ role: 'user', content }])
+
+// string signatures take a type too
+buildPromptContent('photo: image, question -> answer', { photo, question })
    +
    + Modules send text +

    + buildPromptContent() returns a plain string + when every input is text, byte for byte what buildPrompt() + produces. buildPrompt() always returns a + string, rendering an image input as an + [image: image/png] placeholder — and that placeholder + is what Predict, ChainOfThought and + RespAct send, because they build string prompts. Until the + modules carry content parts, send images through lm.chat() + yourself. +

    +
    + +

    What each provider sends

    +
    + + + + + + + +
    ProviderInline bytesRemote URL
    OpenAIimage_url with a data: URIimage_url.url, plus detail when set
    Anthropicimage block, source.type: "base64"image block, source.type: "url"
    GeminiinlineData with mimeTypefileData.fileUri, Files API or Cloud Storage only
    +
    +

    + A data: URI handed to Anthropic or Gemini as a URL is rewritten + to inline bytes, because neither can dereference one. Gemini will not fetch + an arbitrary web URL at all — fileData resolves only a + Files API or Cloud Storage URI — so anything else is refused with an + LMError rather than sent on to earn a 400. detail + is an OpenAI-only fidelity hint; the others ignore it. +

    + +

    Roles and alternation

    +

    + Only a user turn may carry an image. OpenAI's message type is a + discriminated union in which system and assistant + accept text alone, and Anthropic's system and Gemini's + systemInstruction are top-level strings — so an image + addressed to any of those is flattened to its placeholder rather than + dropped without trace. +

    +

    + Anthropic additionally requires strict user/assistant alternation, so + consecutive same-role turns are merged. That merge now concatenates block + arrays: it used to run only when both turns were strings, which meant a + text turn followed by an image turn was sent as two adjacent user messages + and rejected outright. Merged content is therefore a block array, with + two adjoining text blocks folded into one so the blank line between the + turns survives. +

    + +
    + Capability, honestly +

    + supportsVision is now reported per model rather than + hardcoded to true: false for + gpt-3.5, o1-mini and o3-mini, for + claude-3-5-haiku and older Claude models, and for Gemini + embedding models. Check it before sending pixels. +

    +
    +
    + From f11025ed5f53e720e60b0fbc406973820724284c Mon Sep 17 00:00:00 2001 From: Arnav Dadarya Date: Sat, 22 Aug 2026 15:21:47 -0700 Subject: [PATCH 2/2] fix(vision): resolve merge against tool calling, demos and zod signatures --- packages/anthropic/src/anthropic-lm.ts | 1 - packages/core/src/utils/parsing.ts | 3 +++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/packages/anthropic/src/anthropic-lm.ts b/packages/anthropic/src/anthropic-lm.ts index b34e735..1114f9c 100644 --- a/packages/anthropic/src/anthropic-lm.ts +++ b/packages/anthropic/src/anthropic-lm.ts @@ -11,7 +11,6 @@ import { type FinishReason, type ImageContentPart, type LLMCallOptions, - type MessageContent, type ModelCapabilities, type StreamChunk, type ToolCall, diff --git a/packages/core/src/utils/parsing.ts b/packages/core/src/utils/parsing.ts index 4420bb0..1272d1c 100644 --- a/packages/core/src/utils/parsing.ts +++ b/packages/core/src/utils/parsing.ts @@ -468,6 +468,9 @@ function extractFieldValue( * rules runs exactly as it does for a text-only prompt, and the image is spliced * back in at the end. NUL is used because no prompt legitimately contains one. */ +// NUL is the point here: it is the one character a prompt can never +// legitimately contain, so the marker cannot collide with caller input. +// eslint-disable-next-line no-control-regex const IMAGE_MARKER = /\u0000ts-dspy:image:(\d+)\u0000/; /** Input fields declared as images. Only class signatures can declare one. */