Skip to content

feat: per-model image limits in model capabilities - #244

Draft
erzaehlsalex wants to merge 8 commits into
agdevhq:mainfrom
erzaehlsalex:feat/per-model-image-limits
Draft

erzaehlsalex wants to merge 8 commits into
agdevhq:mainfrom
erzaehlsalex:feat/per-model-image-limits

Conversation

@erzaehlsalex

Copy link
Copy Markdown
Contributor

What

Adds ModelCapabilities.input with maxImages and an optional maxMedia. Each chat model now reports the most images it accepts in one request, so apps can trim or batch images before sending instead of finding out from ImageCountExceededError (#229).

Like output.maxTokens, the field is present only when the provider has a verified limit for the model id on the host it serves. If it is missing, the limit is unknown. It does not mean there is no limit.

Provider maxImages Source
Anthropic, Anthropic on Vertex 600 for 1M-context models, 100 for 200k models; images and PDF pages share the limit, so maxMedia is set too platform.claude.com vision and context-windows docs; matches the captured … > 100 errors
OpenAI (Responses and Chat Completions) 1,500 for image-capable models developers.openai.com images-and-vision guide
Gemini API 3,600 for known Gemini models ai.google.dev image-understanding docs; matches the captured error
Vertex AI Gemini 3,000 for known Gemini models every Vertex AI model page
Mistral 8 for Mistral's vision models docs.mistral.ai vision FAQ; matches the captured rejection
Azure OpenAI, OpenAI-compatible, Omnifact, xAI, Kimi not set Azure's docs contradict each other (50 vs 10). The other hosts enforce their own limits; xAI and Kimi document none.

Design notes

  • Expose only, no client-side check. Providers also count document pages and video frames against the limit, so counting locally would disagree with them. ImageCountExceededError handling stays as it is.
  • Unknown ids get no limit. This covers fail-open registries, self-hosted endpoints and image-generation models, which accept far fewer images (Gemini Flash Image allows 14).
  • Google platform. getGoogleModelCapabilities(modelId, platform) and a platform factory option on createGoogleGenAIProvider. @core-ai/google-vertex passes 'vertex', and its getGoogleModelCapabilities export now returns Vertex AI's limits.
  • OpenAI wrappers. When a provider has no registry of its own, the shared factory falls back to OpenAI's registry, and that fallback now applies omitOpenAIInputLimits. This covers Azure v1, openai-compat and Omnifact. Azure classic applies it explicitly.

Not covered

  • Mistral's 8 is not live-probed for each family. One Mistral docs page says the limit "depends on the model".
  • Only image counts are covered. Per-image byte and pixel limits are out of scope.

Testing

  • npm run build, turbo lint check-types test, npm run docs:check-links
  • New capability tests for each provider, plus provider tests checking that Vertex reports 3,000 and that Azure (v1 and classic), openai-compat and Omnifact report no limit.

🤖 Generated with Claude Code

Alex Augustin and others added 7 commits October 27, 2026 21:26
Add optional capabilities.input with maxImages and maxMedia, present only when a provider has a verified per-request limit for the model id on the host it serves. Mirrors output.maxTokens and the ImageCountExceededError / MediaCountExceededError fields.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
600 for models with a 1M-token context window, 100 for 200k models, per Anthropic's vision and context-windows docs. Unknown ids report no limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Known Gemini models report 3,600 images on the Gemini API and 3,000 on Vertex AI. getGoogleModelCapabilities and createGoogleGenAIProvider take a platform; google-vertex passes 'vertex' and its getGoogleModelCapabilities export returns Vertex limits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Image-capable registry models report 1,500 images per request. Providers that fall back to the OpenAI registry on another host (Azure OpenAI, OpenAI-compatible servers, Omnifact) drop it via omitOpenAIInputLimits, as do unknown ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mistral's vision families report 8 images per request (vision docs FAQ, matching the API rejection). Unknown and self-hosted ids keep vision input without a limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both providers document no image count limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Describe capabilities.input in the types reference, add an image limits section to the multi-modal guide, and note Vertex AI's limit on the provider page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mintlify

mintlify Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit ebdf83a.

Approve preview

@erzaehlsalex
erzaehlsalex marked this pull request as draft October 27, 2026 20:55
Google GenAI chat models now pick Vertex AI's limit when the client was
built with `vertexai: true`. OpenAI's 1,500-image limit is kept only when
the client's baseURL is api.openai.com or a regional OpenAI host, instead
of being inferred from whether a registry was passed. createOpenAI drops
its duplicate registry, so fine-tuned ids resolve to their base model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* context-windows docs). Anthropic documents the same split on Vertex AI and
* Bedrock.
*/
const MAX_MEDIA_PER_REQUEST: Record<string, number> = {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can add 600 and 100 into constants

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant