Anthropic translation forwards data: URI image_url as a URL-type source, breaking base64 vision requests

[ctresco]

Summary

When routing an OpenAI-format chat completion request containing a data: URI in an image_url.url content part to an anthropic_messages-format target, Switchyard forwards the string verbatim into Anthropic's image content block as a URL-type source (source: {type: "url", url: "data:image/png;base64,..."}). Anthropic's API correctly rejects this, since a URL-type source must be a real http(s):// link, not a base64 data URI. The resulting error is opaque to callers: it's returned as a generic upstream 400 with message "Only HTTPS URLs are supported.", giving no indication that the actual problem is the missing data-URI translation.

Expected behavior

Per OpenAI's chat completions API, image_url.url may be either a real http(s):// URL or a data:<mime>;base64,<data> URI (this is the standard way OpenAI-compatible clients send inline images without external hosting). When translating to Anthropic's Messages API format, Switchyard should detect the data: prefix, parse out the media type and base64 payload, and emit Anthropic's base64-type image source instead:

{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": "<payload>"}}

URLs that are already http(s):// should presumably continue to pass through as Anthropic's URL-type source unchanged.

Steps to reproduce

Configuration: a target with format = "anthropic_messages" pointing at the real Anthropic API (base_url = "https://api.anthropic.com").

Request (OpenAI-format, sent to Switchyard's /v1/chat/completions):

{
  "model": "<anthropic-backed route>",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "What color is this?"},
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,<...>"}}
    ]
  }]
}

Observed response: HTTP 400, {"error": {"code": "upstream_error", "message": "Only HTTPS URLs are supported.", "type": "upstream_error"}}.

Confirmation this is translation-side, not client/config error

• The same request against a format = "openai_chat" target (Google's OpenAI-compatible Gemini endpoint, passthrough with no format translation) succeeds and correctly describes the image — so the base64 payload itself is well-formed and the issue is specific to the Anthropic translation path.
• The literal error string "Only HTTPS URLs are supported." does not appear anywhere in the compiled Switchyard binary (checked via binary grep), confirming it's Anthropic's own API error being relayed through, not a Switchyard-side validation message — i.e. Switchyard is constructing a request Anthropic considers malformed, rather than rejecting it before the request is sent.

Environment

• Switchyard built from main via cargo build --release (Docker image built same-day as this report).
• Reached via a target with llm_client pointing at format = "anthropic_messages", base_url = "https://api.anthropic.com".

[bharadwaj-pendyala]

Reproduced on main at 1700f62 as a translation-layer unit test, no Anthropic key needed.

decode_image_source (crates/switchyard-translation/src/codecs/openai_chat/buffered.rs:546) maps every image_url.url to ImageSource::Url whatever the scheme, and the Anthropic encoder renders that variant as {"type": "url", "url": ...} at crates/switchyard-translation/src/codecs/anthropic/buffered.rs:853. A data: URI therefore reaches Anthropic as a URL source, which is what produces "Only HTTPS URLs are supported.".

Translating {"type": "image_url", "image_url": {"url": "data:image/png;base64,aW1hZ2U="}} from openai_chat to anthropic_messages gives:

{"type": "image", "source": {"type": "url", "url": "data:image/png;base64,aW1hZ2U="}}

There are two places to fix this, and I went with the second:

1. Normalize in decode_image_source so the neutral IR carries ImageSource::Base64. That variant has no detail field, so an openai_chat to openai_chat translation would start silently dropping "detail": "high" on inline images.
2. Convert in the Anthropic encoder, where the constraint actually lives. ImageSource::Base64 is already encoded correctly there, so this is one match arm plus a data: splitter, and it also covers the Anthropic tool-result image path, which funnels through the same function.

Option 2 leaves ImageSource::Url untouched for every other target, so nothing changes for OpenAI or Responses. PR coming shortly under the "focused bug fix, no issue needed" path in CONTRIBUTING. Happy to move it to the decoder instead if you would rather normalize at the boundary and add detail to the Base64 variant.