feat: vision and multimodal input across all providers - #18
Merged
Merged
Conversation
`ChatMessage.content` was a plain `string` and each converter passed it straight through, so `supportsVision: true` was a claim the library could not honour. Content is now `string | ContentPart[]`, where a part is text or an image carried as a URL, a data URI, or base64 plus a media type. Plain strings stay valid, so text-only code is untouched. Signature inputs can be declared images with `@ImageField` (or the `image` type in a string signature), and `buildPromptContent()` renders such a signature as parts, returning a string when every input is text. Per provider: OpenAI `image_url` parts, role-aware because only the user variant of its message union accepts them; Anthropic `image` blocks, with same-role merging moved onto block arrays so a text turn beside an image turn still satisfies strict alternation; Gemini `inlineData`, refusing a URL `fileData` cannot dereference rather than earning a 400. `supportsVision` is now reported per model instead of hardcoded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts: # README.md # packages/anthropic/src/anthropic-lm.test.ts # packages/anthropic/src/anthropic-lm.ts # packages/core/src/index.ts # packages/core/src/test-utils.ts # packages/core/src/types/language-model.ts # packages/core/src/utils/parsing.ts # packages/gemini/src/gemini-lm.test.ts # packages/gemini/src/gemini-lm.ts # packages/openai/src/openai-lm.test.ts # packages/openai/src/openai-lm.ts # site/docs.html
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ModelCapabilities.supportsVisionwas reportedtrueby all three providers, butChatMessage.contentwas a plainstringand every converter passed it straight through — there was no way to send an image. This makes the flag true.ChatMessage.contentis nowstring | ContentPart[]. Keepingstringin the union is what keeps the blast radius small:BaseLM.generate(),generateStructured(),parseJsonResponseand all three modules build string prompts and compile untouched.ContentPartis text or image; an image source is a URL (https://ordata:) or base64 plus a media type. Helpers:imagePart(),textPart(),contentToText(),toContentParts(),hasImageContent(),imageToUrl(),imageMediaType(),normalizeImageSource().@ImageField({ … }), orphoto: imagein a string signature.buildPromptContent()emits content parts when any input is an image and a plain string otherwise — byte for byte whatbuildPrompt()produces.buildPrompt()still returns a string, rendering an image as an[image: image/png]placeholder.supportsVisionis per model: false forgpt-3.5/gpt-4-0*/gpt-4-32k/o1-mini/o1-preview/o3-mini, forclaude-3-5-haikuand older Claude, and for Gemini embedding models.Per-provider mapping
image_urlwith adata:URIimage_url.url, plusdetailimageblock,source.type: "base64"imageblock,source.type: "url"inlineDatawithmimeTypefileData.fileUri, Files API / GCS onlyOpenAI's conversion is role-aware:
ChatCompletionMessageParamis a discriminated union where onlyuseraccepts image parts, so system and assistant content is flattened to text. Anthropic'ssystemparameter and Gemini'ssystemInstructionare flattened the same way.Gemini refuses an arbitrary web URL with a clear
LMErrorinstead of sending it on —fileDataonly dereferences Files API and Cloud Storage URIs.The Anthropic risk
toAnthropicMessagesmerged consecutive same-role turns by string concatenation, guarded bytypeof previous.content === 'string'. With array content that guard silently stops merging, and the Messages API requires strict alternation — so a text turn followed by an image turn would have been sent as two adjacent user messages and rejected. The merge now concatenates block arrays, folding two adjoining text blocks into one so the blank line between turns survives. The existing test asserting the string-merged shape was updated with a comment explaining why, and a new test covers the text-then-image adjacency directly.Verification
npm run build,lint,typecheck,format:check,test(206 pass) andverify:packagingall green. A throwaway consumer script (deleted before commit) imported the built packages through the workspace symlinks and asserted each converter's SDK shape for text-only, mixed, and remote-URL messages, plus the alternation case — the text-only path is byte-identical to before.Docs: new section 19
#visioninsite/docs.html(verified rendering locally, TOC entry included) and an Images section in the README.🤖 Generated with Claude Code