From 00e62884c80cb2cba068334c7350b6bd7cb578de Mon Sep 17 00:00:00 2001 From: Aakash Wijesekara Date: Tue, 4 Aug 2026 17:00:05 +0530 Subject: [PATCH 1/9] Document multi-provider LLM routing capabilities --- .../next/llm-proxy/multi-provider-routing.md | 381 ++++++++++++++++-- 1 file changed, 345 insertions(+), 36 deletions(-) diff --git a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md index 78c0d19ec..074b69ca6 100644 --- a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md +++ b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md @@ -8,7 +8,7 @@ tags: - llm - routing author: WSO2 API Platform Documentation Team -last_updated: 2026-07-30 +last_updated: 2026-08-04 content_type: "guide" --- @@ -16,9 +16,9 @@ content_type: "guide" ## Overview -Multi-provider routing lets one large language model (LLM) proxy expose a single OpenAI-compatible endpoint while routing each request to a selected LLM provider. Applications continue to use the same endpoint and OpenAI-compatible request and response format, even when the upstream provider changes. +Multi-provider routing lets one large language model (LLM) proxy expose a single OpenAI-compatible endpoint while routing each request to a selected LLM provider. Applications continue to use the same endpoint and OpenAI-compatible request format when the upstream provider changes. Non-streaming responses are normalized where supported; streaming compatibility varies by provider. -For example, an application can send all requests to `/openai-multi/chat/completions` and select OpenAI or Anthropic with the `x-provider` request header. +For example, an application can send all requests to `/openai-multi/chat/completions` and select OpenAI or Anthropic with the `x-provider` request header. The proxy can also distribute requests automatically across provider and model pairs by using round-robin or weighted round-robin routing. This is useful when you want to: @@ -26,9 +26,97 @@ This is useful when you want to: - Compare provider responses using the same OpenAI-compatible request - Keep vendor credentials in the gateway instead of distributing them to applications - Apply proxy-level authentication, rate limits, and guardrails consistently across providers -- Introduce provider fallback or selection logic through a routing policy +- Introduce provider selection and model suspension through a routing policy -## How It Works +Multi-provider transformation is scoped to the OpenAI Chat Completions request and response model. It does not add cross-provider support for the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs. + +## Choose a Routing Strategy + +Choose one provider-selection strategy for each operation unless you have explicitly designed and tested the precedence between multiple routing policies. + +| Capability | Header router | Model round robin | Model weighted round robin | +|------------|---------------|-------------------|----------------------------| +| Explicit client or provider choice | Yes | No | No | +| Selects a provider | Yes | Optional per model entry | Optional per model entry | +| Selects and rewrites a model | No | Yes | Yes | +| Uses the primary provider when no provider is selected | Yes | Yes | Yes | +| Suspends a provider/model pair after `429` or `5xx` | No | Yes | Yes | +| Weighted traffic distribution | No | No | Yes | +| Latency-, cost-, or semantic-based routing | No | No | No | +| Retries the failed request on another target | No | No | No | + +### Header-based routing + +Use `llm-header-router` when the application or an earlier policy must explicitly choose a provider. The router reads a request header, matches its value against an ordered mapping, and publishes the selected provider name. + +| Parameter | Required | Default | Description | +|-----------|----------|---------|-------------| +| `mappings` | Yes | None | Ordered list of header values and effective provider names. At least one mapping is required. | +| `headerName` | No | `x-provider` | Header used for provider selection. Header-name lookup is case-insensitive. | +| `defaultProvider` | No | Unset | Provider selected when the header is missing, empty, or unmatched. If unset, the primary provider is used. | + +The router has the following selection behavior: + +- Uses only the first value when the header appears more than once +- Trims leading and trailing whitespace from the value +- Matches configured values case-insensitively +- Rejects duplicate mapping values case-insensitively +- Preserves a non-empty provider selection made by an earlier policy +- Leaves the routing header on the upstream request + +The header router publishes provider-selection metadata but does not by itself override the named upstream. An additional provider therefore needs a matching inline transformer, or another policy that explicitly sets its upstream. + +### Model round robin + +Use `model-round-robin` to cycle deterministically through a list of models. A model entry can include a `provider` to route that model to an additional provider. When `provider` is omitted, the model uses the primary provider. + +```yaml +operationPolicies: + - name: model-round-robin + version: v1 + paths: + - path: /chat/completions + methods: [POST] + params: + models: + - model: gpt-4o + - model: claude-sonnet-4-5-20250929 + provider: anthropic-provider + suspendDuration: 30 +``` + +The policy rewrites the model at the location defined by the provider template. It can rewrite a model in the request payload, a header, a query parameter, or a path parameter. + +See [Model Round Robin](load-balancing/model-round-robin.md) for its complete configuration. + +### Model weighted round robin + +Use `model-weighted-round-robin` to distribute requests in a deterministic weighted cycle. Each entry requires an integer `weight` of at least `1`. + +```yaml +operationPolicies: + - name: model-weighted-round-robin + version: v1 + paths: + - path: /chat/completions + methods: [POST] + params: + models: + - model: gpt-4o + weight: 2 + - model: claude-sonnet-4-5-20250929 + provider: anthropic-provider + weight: 1 + suspendDuration: 30 +``` + +This example produces the repeating sequence `gpt-4o`, `gpt-4o`, `claude-sonnet-4-5-20250929` while both targets are available. It provides proportional deterministic distribution, not random or performance-based load balancing. + +See [Model Weighted Round Robin](load-balancing/model-weighted-round-robin.md) for its complete configuration. + +## Configure Providers + +### How provider selection works A multi-provider LLM proxy has: @@ -63,7 +151,16 @@ OpenAI-compatible client response The router writes the selected provider name to request metadata. The gateway conditionally applies only the authentication and transformer associated with that provider. When the selection header is missing, empty, or does not match a configured mapping, the router uses `defaultProvider` when configured; otherwise, the proxy's primary provider is used. -## Before You Begin +The effective provider name connects routing, transformation, authentication, and upstream selection: + +- The primary provider is identified by `spec.provider.id`. +- An additional provider uses `additionalProviders[].as` when an alias is configured; otherwise, it uses `additionalProviders[].id`. +- Router mappings and model-routing entries must use the effective provider name. +- When no provider is selected, the proxy uses its primary provider. +- Authentication and transformation for an additional provider execute only when that provider is selected. +- The controller injects the effective provider name into an inline transformer's `providerId`; do not configure it manually. + +### Before you begin Make sure that: @@ -74,7 +171,7 @@ Make sure that: This guide configures OpenAI as the primary provider and Anthropic as an additional provider. The same configuration model can be extended to Azure OpenAI, Mistral, Gemini, AWS Bedrock, and other providers supported by your AI Gateway version. -## Understand the Authentication Layers +### Understand the authentication layers Multi-provider routing can involve three different kinds of credentials: @@ -86,11 +183,11 @@ Multi-provider routing can involve three different kinds of credentials: Do not use a vendor API key as a loopback or consumer key. Do not commit any of these credentials to source control. -## Step 1: Deploy the LLM Providers +### Step 1: Deploy the LLM providers Each provider must exist before a proxy can reference it. -### Deploy the OpenAI provider +#### Deploy the OpenAI provider Replace `` with an OpenAI API key. @@ -131,7 +228,7 @@ spec: EOF ``` -### Deploy the Anthropic provider +#### Deploy the Anthropic provider Replace `` with an Anthropic API key. @@ -174,7 +271,7 @@ EOF The vendor credentials under `spec.upstream.auth` are added only when the provider calls its external service. -## Step 2: Create Provider Loopback Keys +### Step 2: Create provider loopback keys Because both providers in this example use the `api-key-auth` policy, create an API key for each provider. The proxy uses these keys when routing to the providers through the gateway's internal loopback route. @@ -203,7 +300,7 @@ test -n "$ANTHROPIC_LOOPBACK_KEY" && test "$ANTHROPIC_LOOPBACK_KEY" != "null" API key values are returned only when they are created or regenerated. Store them securely. -## Step 3: Deploy the Multi-Provider LLM Proxy +### Step 3: Deploy the multi-provider LLM proxy The following proxy exposes one `/chat/completions` operation. OpenAI is the primary and default provider. Anthropic is an additional selectable provider with an inline request and response transformer. @@ -268,7 +365,7 @@ EOF The controller automatically passes the additional provider's effective upstream name to its transformer. Do not add a `providerId` under `transformer.params`; it is injected from `additionalProviders[].id` or `additionalProviders[].as`. -## Step 4: Create a Proxy Consumer Key +### Step 4: Create a proxy consumer key The proxy uses `api-key-auth` to protect its public endpoint. Create a key for the application that will invoke it: @@ -287,11 +384,11 @@ Verify that a key was returned: test -n "$PROXY_CONSUMER_KEY" && test "$PROXY_CONSUMER_KEY" != "null" ``` -## Step 5: Invoke Different Providers +### Step 5: Invoke different providers All requests use the same URL and OpenAI Chat Completions payload. -### Invoke the default provider +#### Invoke the default provider If `x-provider` is omitted, the router uses `defaultProvider`, which is `openai-provider` in this example. @@ -310,7 +407,7 @@ curl -k -X POST https://localhost:8443/openai-multi/chat/completions \ }' ``` -### Invoke Anthropic +#### Invoke Anthropic Set `x-provider` to the configured `headerValue`: @@ -334,25 +431,28 @@ The Anthropic transformer replaces the request's `model` value with the model co Header names and mapped header values are matched case-insensitively. Leading and trailing whitespace in the header value is ignored. If the header is missing, empty, or does not match a mapping, the router selects `defaultProvider`. -## Add More Providers +### Add more providers Add each selectable provider under `additionalProviders`, then add a corresponding mapping under the LLM Header Router policy (`llm-header-router`). -### Supported provider transformers +#### Supported provider transformers Use a transformer when an additional provider does not accept and return the OpenAI wire format. -| Target provider | Transformer type | Purpose | -|-----------------|------------------|---------| -| Anthropic | `openai-to-anthropic` | Converts OpenAI-compatible requests to the Anthropic Messages format and converts responses back to the OpenAI format. | -| Azure OpenAI | `openai-to-azure-openai` | Adapts OpenAI-compatible requests for Azure OpenAI deployments and API versions. | -| Mistral | `openai-to-mistral` | Adapts OpenAI-compatible requests and responses for Mistral. | -| Gemini | `openai-to-gemini` | Converts OpenAI-compatible requests and responses for Google Gemini. | -| AWS Bedrock | `openai-to-bedrock-transformer` | Converts OpenAI-compatible requests and supported AWS Bedrock responses. | +| Target provider | Transformer type used in this guide | Purpose | +|-----------------|-------------------------------------|---------| +| Anthropic | `openai-to-anthropic` | Converts OpenAI-compatible requests to Anthropic Messages and converts non-streaming responses back to OpenAI format. | +| Azure OpenAI | `openai-to-azure-openai` | Adapts the request path for an Azure OpenAI deployment and API version. | +| Mistral | `openai-to-mistral` | Normalizes OpenAI-compatible requests and responses for Mistral. | +| Gemini | `openai-to-gemini` | Converts OpenAI-compatible requests and non-streaming responses for Gemini. | +| AWS Bedrock | `openai-to-bedrock-transformer` | Converts OpenAI-compatible requests and Bedrock Converse responses, including streaming responses. | A transformer is not required when the selected provider already exposes an OpenAI-compatible API. -### Azure OpenAI +!!! note "Transformer names and versions" + Transformer names and major versions can differ between AI Gateway releases. Inspect the policy catalog installed with your gateway and use the name and version exposed there. The examples on this page use the policy names supported by this documentation baseline. + +#### Azure OpenAI ```yaml - id: azure-openai-provider @@ -368,7 +468,7 @@ A transformer is not required when the selected provider already exposes an Open apiVersion: "2024-02-15-preview" ``` -### Mistral +#### Mistral ```yaml - id: mistral-provider @@ -383,7 +483,7 @@ A transformer is not required when the selected provider already exposes an Open model: mistral-large-latest ``` -### Gemini +#### Gemini ```yaml - id: gemini-provider @@ -399,7 +499,7 @@ A transformer is not required when the selected provider already exposes an Open apiVersion: v1beta ``` -### AWS Bedrock +#### AWS Bedrock ```yaml - id: aws-bedrock-provider @@ -428,7 +528,7 @@ mappings: provider: aws-bedrock-provider ``` -## Use Provider Aliases +### Use provider aliases Use `as` when the logical upstream name used by routing policies should differ from the deployed provider ID: @@ -462,9 +562,9 @@ The alias must: - Be unique within the proxy - Not match the primary provider ID or another additional provider's effective name -## Configuration Reference +### Configuration reference -### `additionalProviders` +#### `additionalProviders` | Field | Required | Description | |-------|----------|-------------| @@ -473,7 +573,7 @@ The alias must: | `auth` | No | API key authentication used by the proxy when calling the provider's internal route | | `transformer` | No | Request and response transformer applied only when this provider is selected | -### `transformer` +#### `transformer` | Field | Required | Description | |-------|----------|-------------| @@ -481,17 +581,212 @@ The alias must: | `version` | Yes | Major policy version, such as `v1` | | `params` | No | Transformer-specific parameters, such as `model` or `apiVersion` | -### LLM Header Router parameters +#### LLM Header Router parameters Use `llm-header-router` as the policy name in the configuration. | Parameter | Required | Default | Description | |-----------|----------|---------|-------------| | `headerName` | No | `x-provider` | Request header used for selection | -| `defaultProvider` | No | Primary provider | Effective provider name selected when no mapping matches. When omitted, the proxy's primary provider is used. | +| `defaultProvider` | No | Unset | Effective provider name selected when no mapping matches. When omitted, selection remains unset and the proxy's primary provider is used. | | `mappings` | Yes | None | Header value to effective provider name mappings; the first match wins | -## Validation and Troubleshooting +## Provider Capability Matrix + +All transformer capabilities on this page refer to the OpenAI Chat Completions contract implemented by the gateway. They do not imply that every model offered by a provider supports the corresponding feature. + +| Capability | Anthropic | Azure OpenAI | AWS Bedrock | Gemini | Mistral | +|------------|-----------|--------------|-------------|--------|---------| +| Request body handling | Full conversion | Pass-through | Full conversion | Full conversion | OpenAI-compatible normalization | +| Request path | `/v1/messages` | Deployment path with `api-version` | Converse or Converse Stream | `generateContent` or `streamGenerateContent` | `/v1/chat/completions` | +| Model behavior | Policy model required | Optional deployment override; otherwise body model | Optional policy model; otherwise body model | Policy model required | Policy model required | +| System and developer messages | Converted to top-level system text | Pass-through | Converted to system blocks | Converted to `systemInstruction` | Pass-through | +| Text messages | Converted | Pass-through | Converted | Converted | Pass-through | +| Image input | Base64 data URI and remote URL | Pass-through | Base64 data URI only | Base64 data URI and remote URL | Pass-through | +| Function tools | Converted | Pass-through | Converted | Converted | Pass-through | +| Tool-call history and results | Converted | Pass-through | Converted | Converted | Pass-through | +| Non-streaming response | Converted to OpenAI format | Already OpenAI-compatible | Converted to OpenAI format | Converted to OpenAI format | Normalized OpenAI-compatible response | +| Error response | Converted | Passed through | Converted | Converted | Converted | +| Streaming request selection | Passed to Anthropic | Passed through | Selects Converse Stream | Selects `streamGenerateContent` SSE | Passed through | +| OpenAI-compatible streaming response | No | Yes, subject to deployment and API version | Yes | No | Expected, subject to model and API behavior | +| Usage conversion | Converted, including available cache details | Passed through | Converted, including cache details | Converted, including cached and reasoning tokens | Passed through | + +### Input capability matrix + +`Converted` means that the transformer explicitly maps a field to the provider-native format. `Pass-through` means that it deliberately retains the OpenAI field. `Omitted` means that a full-conversion transformer does not copy the field. A passed-through field is still subject to support by the selected provider model and API version. + +| OpenAI Chat Completions input | Anthropic | Azure OpenAI | AWS Bedrock | Gemini | Mistral | +|--------------------------------|-----------|--------------|-------------|--------|---------| +| `model` | Replaced by policy | Passed through and used as deployment fallback | Used in path and omitted from body | Replaced and used in path | Replaced by policy | +| Text messages | Converted | Pass-through | Converted | Converted | Pass-through | +| `system` role | Top-level system text | Pass-through | System blocks | `systemInstruction` | Pass-through | +| `developer` role | Treated as system | Pass-through | Treated as system | Treated as system | Pass-through | +| `assistant.tool_calls` | Converted | Pass-through | Converted | Converted | Pass-through | +| `tool` role results | Converted | Pass-through | Converted | Converted | Pass-through | +| Image data URI | Converted | Pass-through | Converted | Converted | Pass-through | +| Remote image URL | Converted | Pass-through | Omitted | Converted to `fileData` | Pass-through | +| `max_completion_tokens` | Mapped to `max_tokens` | Pass-through | Mapped to `max_tokens` | Mapped to `maxOutputTokens` | Pass-through | +| `max_tokens` | Mapped to `max_tokens` | Pass-through | Mapped to `max_tokens` | Mapped to `maxOutputTokens` | Pass-through | +| `temperature` | Converted | Pass-through | Converted | Converted | Pass-through | +| `top_p` | Converted | Pass-through | Converted | Converted | Pass-through | +| `stop` string or array | Converted | Pass-through | Converted | Converted | Pass-through | +| `stream` | Passed to Anthropic | Pass-through | Selects streaming path | Selects streaming path | Pass-through | +| `n` | Omitted | Pass-through | Omitted | Mapped to `candidateCount` | Removed | +| `seed` | Omitted | Pass-through | Omitted | Converted | Pass-through | +| `frequency_penalty` | Omitted | Pass-through | Omitted | Converted | Pass-through | +| `presence_penalty` | Omitted | Pass-through | Omitted | Converted | Pass-through | +| `tools` and `tool_choice` | Converted | Pass-through | Converted | Converted | Pass-through | +| `response_format` | Omitted | Pass-through | Omitted | Omitted | Pass-through | +| `logprobs` and `top_logprobs` | Omitted | Pass-through | Omitted | Omitted | Removed | +| `logit_bias` | Omitted | Pass-through | Omitted | Omitted | Removed | +| `service_tier`, `store`, `metadata`, and `user` | Omitted | Pass-through | Omitted | Omitted | Removed | + +For Anthropic, AWS Bedrock, and Gemini, the transformer constructs a new provider-native request body. Fields not explicitly converted are omitted. + +### Response capability matrix + +| Output feature | Anthropic | Azure OpenAI | AWS Bedrock | Gemini | Mistral | +|----------------|-----------|--------------|-------------|--------|---------| +| OpenAI `chat.completion` envelope | Generated | Native and passed through | Generated | Generated | Native and normalized | +| Multiple choices | No; one choice | Upstream-dependent | No; one choice | Yes; all candidates | Upstream-dependent | +| Text output | Converted | Pass-through | Converted | Converted; thought parts excluded | Pass-through | +| Tool calls | Converted | Pass-through | Converted | Converted | Pass-through | +| Finish reason | Converted | Pass-through | Converted | Converted | Pass-through | +| Token usage | Converted, including available cache-read and cache-creation details | Pass-through | Converted, including cache-read and cache-write details | Converted, including cached and reasoning tokens | Pass-through | +| Error envelope | Converted | Pass-through | Converted | Converted | Converted | +| OpenAI SSE conversion | No | Native and passed through | Yes | No | Native and passed through | + +Anthropic produces one OpenAI choice and preserves available cache-read and cache-creation counts in prompt token details. AWS Bedrock produces one choice and retains cache-read and cache-write information for cost calculation. Gemini converts every candidate, preserves candidate indices, excludes `thought: true` parts from visible assistant text, and exposes thought tokens as reasoning tokens. + +## Streaming + +Streaming behavior is not uniform across provider routes. Select a provider based on the response contract required by the client, not only on whether the upstream accepts `stream: true`. + +| Provider route | Upstream stream | Response returned to the client | OpenAI SSE compatible | +|----------------|-----------------|---------------------------------|-----------------------| +| Anthropic | Anthropic SSE | Native Anthropic events are passed through | No | +| Azure OpenAI | Azure OpenAI SSE | Passed through | Yes, subject to deployment and API version | +| AWS Bedrock | Amazon binary event stream | Decoded into OpenAI `chat.completion.chunk` SSE | Yes | +| Gemini | Gemini SSE | Native Gemini events are passed through | No | +| Mistral | OpenAI-compatible SSE | Passed through | Expected, subject to model and API behavior | + +!!! warning "Anthropic and Gemini streams are not OpenAI SSE" + The Anthropic and Gemini transformers select the correct upstream streaming endpoint, but they do not translate the returned provider-native SSE events into OpenAI Chat Completions chunks. Use non-streaming requests for clients that require one uniform OpenAI response contract. + +The Bedrock transformer provides full cross-protocol streaming conversion. It decodes Amazon event-stream frames, converts text and tool-call deltas, maps stream errors, emits usage data, and terminates the stream with `data: [DONE]`. + +## Tools and Multimodal Input + +### Images + +Image support varies by provider: + +- Anthropic accepts OpenAI image parts containing base64 data URIs or remote image URLs. +- AWS Bedrock accepts base64 image data URIs. Remote image URLs are omitted because Bedrock Converse requires image bytes. +- Gemini converts base64 data URIs to inline data and remote URLs to file data. +- Azure OpenAI and Mistral receive image parts unchanged. Support depends on the selected deployment or model. + +The gateway does not inspect a model's capabilities before routing. A successfully transformed request can still be rejected when the selected model does not support image input. + +### Function tools + +Anthropic, AWS Bedrock, and Gemini convert OpenAI function declarations into provider-native tool declarations. Azure OpenAI and Mistral receive `tools` unchanged. + +The transformers accept OpenAI function tools in the following shape: + +```json +{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the weather for a city", + "parameters": { + "type": "object", + "properties": { + "city": { + "type": "string" + } + }, + "required": ["city"] + } + } +} +``` + +| OpenAI field | Anthropic | AWS Bedrock | Gemini | Azure OpenAI and Mistral | +|--------------|-----------|-------------|--------|--------------------------| +| `function.name` | `tools[].name` | `toolConfig.tools[].toolSpec.name` | `tools[].functionDeclarations[].name` | Passed through | +| `function.description` | `description` | `toolSpec.description` | `description` | Passed through | +| `function.parameters` | `input_schema` | `inputSchema.json` | `parameters` | Passed through | +| Missing parameter schema | Empty object schema | Empty object schema | Omitted | Passed through | + +Only OpenAI tools with `type: function` are explicitly converted. Provider-native tools, hosted tools, computer-use tools, web-search tools, MCP tool declarations, and OpenAI custom tools are not translated. + +### Tool choice + +| OpenAI `tool_choice` | Anthropic | AWS Bedrock | Gemini | +|----------------------|-----------|-------------|--------| +| `auto` | Automatic selection | Automatic selection | `AUTO` mode | +| `required` | Any tool | Any tool | `ANY` mode | +| `none` | Tool definitions omitted | Tool configuration omitted | Tools omitted and mode set to `NONE` | +| Named function | Named tool | Named tool | `ANY` mode restricted to the named function | +| Unknown or malformed value | Defaults to automatic selection | Omits `toolChoice` | Defaults to `AUTO` mode | + +Azure OpenAI and Mistral receive `tools` and `tool_choice` unchanged. Their acceptance depends on the selected model and API version. + +### Tool-call conversations + +The Anthropic, AWS Bedrock, and Gemini transformers support multi-turn function-tool conversations: + +- Assistant `tool_calls` are converted into provider-native tool-use or function-call blocks. +- The JSON string in `function.arguments` is decoded into an object. +- A `role: tool` message is converted into a provider-native tool result or function response. +- Consecutive tool results are grouped into a provider-compatible user turn where required. +- Provider tool calls in non-streaming responses are converted back into OpenAI `message.tool_calls`. +- Bedrock streaming tool-call deltas are converted into OpenAI chunk deltas. + +Invalid JSON in historical assistant tool arguments is replaced with an empty object. OpenAI-specific strict schemas, `parallel_tool_calls`, provider-specific tool caching, and non-function tool types are not explicitly translated. + +## Failure Behavior + +### Routing failures and suspension + +The round-robin policies track failures per provider/model pair. The same model name configured for two providers is therefore suspended independently. + +- A `429` response or any `5xx` response suspends the selected pair. +- `suspendDuration` defaults to 30 seconds. +- Setting `suspendDuration` to `0` disables failure suspension. +- Suspended entries are skipped on later requests until their suspension expires. +- If every entry is suspended, the policy returns HTTP `503` with `All models are currently unavailable`. +- Rotation counters and suspension state are held by the policy instance in memory and are not coordinated across gateway replicas. + +!!! important "Suspension is not a retry" + The request that receives a `429` or `5xx` response is returned to the client. The policy does not replay that request on another provider. Suspension affects only later requests. + +### Transformation failures + +- Empty or invalid JSON request bodies return HTTP `400` in transformers that perform full request conversion. +- Missing required transformer parameters cause policy validation or initialization to fail. +- Azure OpenAI returns HTTP `400` when neither the policy nor the request supplies a deployment ID. +- AWS Bedrock returns HTTP `400` when neither the policy nor the request supplies a model ID. +- A non-JSON successful provider response is generally passed through instead of being replaced with a gateway-generated `500` response. +- Converted provider errors retain the upstream HTTP status and use an OpenAI-style error envelope where supported by the transformer. + +## Limitations + +- **Chat Completions only:** Cross-provider translation targets the OpenAI `/chat/completions` model. +- **No universal streaming abstraction:** Only AWS Bedrock has cross-protocol conversion to OpenAI SSE. Anthropic and Gemini streams remain provider-native. +- **No automatic capability negotiation:** The gateway does not query the selected model for support for vision, tools, schemas, or individual generation parameters. +- **No automatic routing validation:** Router mappings must match the primary provider ID or an additional provider's effective name. +- **No request retry or immediate failover:** Suspension removes an unhealthy target from later rotations but does not retry the failing request. +- **Instance-local state:** Round-robin counters and suspension maps are maintained in memory by each policy instance. +- **Field loss during full conversion:** Anthropic, AWS Bedrock, and Gemini omit request fields that their transformers do not explicitly map. +- **Provider restrictions still apply:** Successful conversion does not guarantee that a model accepts images, tools, tool choice, penalties, candidate counts, or other mapped values. +- **No primary inline transformer:** The inline `transformer` field is available on `additionalProviders`, not on the primary `provider` object. A transformer for another layout must be attached as an operation policy. +- **One routing strategy is recommended:** Combining routing policies can produce precedence-dependent behavior and should be tested explicitly. +- **Header selection needs an upstream override:** A header-routed additional provider without a transformer does not automatically change the named upstream. + +## Troubleshooting ### The additional provider is not found @@ -522,6 +817,20 @@ Check that: An unknown header value intentionally falls back to `defaultProvider`. +If the mapping selects an additional provider that has no transformer, confirm that another operation policy explicitly sets the named upstream. The header router alone publishes selection metadata. + +### The model router does not move to another provider after a failure + +Model suspension does not retry the current request. Confirm the behavior with a later request after the first target returns `429` or `5xx`. Also confirm that `suspendDuration` is greater than `0` and that each model entry uses the correct effective provider name. + +### Streaming is not in OpenAI chunk format + +Anthropic and Gemini streaming responses are provider-native SSE. Use non-streaming mode, choose Azure OpenAI, AWS Bedrock, or an OpenAI-compatible Mistral stream, or adapt the provider-native stream in the client. + +### An image or tool request is rejected by the provider + +Transformation support and model support are separate. Check the capability matrix, then verify that the exact selected model supports images, function tools, the requested `tool_choice`, and the supplied JSON Schema. + ### The provider returns `401 Unauthorized` Confirm which authentication layer rejected the request: From c9727905629b249b6bcd2c69a5f37a81b22eab33 Mon Sep 17 00:00:00 2001 From: Aakash Wijesekara Date: Wed, 5 Aug 2026 16:15:13 +0530 Subject: [PATCH 2/9] Reorganize multi-provider capability reference --- .../next/llm-proxy/multi-provider-routing.md | 411 +++++++++++------- 1 file changed, 264 insertions(+), 147 deletions(-) diff --git a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md index 074b69ca6..ee513a18d 100644 --- a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md +++ b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md @@ -8,7 +8,7 @@ tags: - llm - routing author: WSO2 API Platform Documentation Team -last_updated: 2026-08-04 +last_updated: 2026-08-05 content_type: "guide" --- @@ -28,8 +28,6 @@ This is useful when you want to: - Apply proxy-level authentication, rate limits, and guardrails consistently across providers - Introduce provider selection and model suspension through a routing policy -Multi-provider transformation is scoped to the OpenAI Chat Completions request and response model. It does not add cross-provider support for the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs. - ## Choose a Routing Strategy Choose one provider-selection strategy for each operation unless you have explicitly designed and tested the precedence between multiple routing policies. @@ -593,159 +591,278 @@ Use `llm-header-router` as the policy name in the configuration. ## Provider Capability Matrix -All transformer capabilities on this page refer to the OpenAI Chat Completions contract implemented by the gateway. They do not imply that every model offered by a provider supports the corresponding feature. - -| Capability | Anthropic | Azure OpenAI | AWS Bedrock | Gemini | Mistral | -|------------|-----------|--------------|-------------|--------|---------| -| Request body handling | Full conversion | Pass-through | Full conversion | Full conversion | OpenAI-compatible normalization | -| Request path | `/v1/messages` | Deployment path with `api-version` | Converse or Converse Stream | `generateContent` or `streamGenerateContent` | `/v1/chat/completions` | -| Model behavior | Policy model required | Optional deployment override; otherwise body model | Optional policy model; otherwise body model | Policy model required | Policy model required | -| System and developer messages | Converted to top-level system text | Pass-through | Converted to system blocks | Converted to `systemInstruction` | Pass-through | -| Text messages | Converted | Pass-through | Converted | Converted | Pass-through | -| Image input | Base64 data URI and remote URL | Pass-through | Base64 data URI only | Base64 data URI and remote URL | Pass-through | -| Function tools | Converted | Pass-through | Converted | Converted | Pass-through | -| Tool-call history and results | Converted | Pass-through | Converted | Converted | Pass-through | -| Non-streaming response | Converted to OpenAI format | Already OpenAI-compatible | Converted to OpenAI format | Converted to OpenAI format | Normalized OpenAI-compatible response | -| Error response | Converted | Passed through | Converted | Converted | Converted | -| Streaming request selection | Passed to Anthropic | Passed through | Selects Converse Stream | Selects `streamGenerateContent` SSE | Passed through | -| OpenAI-compatible streaming response | No | Yes, subject to deployment and API version | Yes | No | Expected, subject to model and API behavior | -| Usage conversion | Converted, including available cache details | Passed through | Converted, including cache details | Converted, including cached and reasoning tokens | Passed through | - -### Input capability matrix - -`Converted` means that the transformer explicitly maps a field to the provider-native format. `Pass-through` means that it deliberately retains the OpenAI field. `Omitted` means that a full-conversion transformer does not copy the field. A passed-through field is still subject to support by the selected provider model and API version. - -| OpenAI Chat Completions input | Anthropic | Azure OpenAI | AWS Bedrock | Gemini | Mistral | -|--------------------------------|-----------|--------------|-------------|--------|---------| -| `model` | Replaced by policy | Passed through and used as deployment fallback | Used in path and omitted from body | Replaced and used in path | Replaced by policy | -| Text messages | Converted | Pass-through | Converted | Converted | Pass-through | -| `system` role | Top-level system text | Pass-through | System blocks | `systemInstruction` | Pass-through | -| `developer` role | Treated as system | Pass-through | Treated as system | Treated as system | Pass-through | -| `assistant.tool_calls` | Converted | Pass-through | Converted | Converted | Pass-through | -| `tool` role results | Converted | Pass-through | Converted | Converted | Pass-through | -| Image data URI | Converted | Pass-through | Converted | Converted | Pass-through | -| Remote image URL | Converted | Pass-through | Omitted | Converted to `fileData` | Pass-through | -| `max_completion_tokens` | Mapped to `max_tokens` | Pass-through | Mapped to `max_tokens` | Mapped to `maxOutputTokens` | Pass-through | -| `max_tokens` | Mapped to `max_tokens` | Pass-through | Mapped to `max_tokens` | Mapped to `maxOutputTokens` | Pass-through | -| `temperature` | Converted | Pass-through | Converted | Converted | Pass-through | -| `top_p` | Converted | Pass-through | Converted | Converted | Pass-through | -| `stop` string or array | Converted | Pass-through | Converted | Converted | Pass-through | -| `stream` | Passed to Anthropic | Pass-through | Selects streaming path | Selects streaming path | Pass-through | -| `n` | Omitted | Pass-through | Omitted | Mapped to `candidateCount` | Removed | -| `seed` | Omitted | Pass-through | Omitted | Converted | Pass-through | -| `frequency_penalty` | Omitted | Pass-through | Omitted | Converted | Pass-through | -| `presence_penalty` | Omitted | Pass-through | Omitted | Converted | Pass-through | -| `tools` and `tool_choice` | Converted | Pass-through | Converted | Converted | Pass-through | -| `response_format` | Omitted | Pass-through | Omitted | Omitted | Pass-through | -| `logprobs` and `top_logprobs` | Omitted | Pass-through | Omitted | Omitted | Removed | -| `logit_bias` | Omitted | Pass-through | Omitted | Omitted | Removed | -| `service_tier`, `store`, `metadata`, and `user` | Omitted | Pass-through | Omitted | Omitted | Removed | - -For Anthropic, AWS Bedrock, and Gemini, the transformer constructs a new provider-native request body. Fields not explicitly converted are omitted. - -### Response capability matrix - -| Output feature | Anthropic | Azure OpenAI | AWS Bedrock | Gemini | Mistral | -|----------------|-----------|--------------|-------------|--------|---------| -| OpenAI `chat.completion` envelope | Generated | Native and passed through | Generated | Generated | Native and normalized | -| Multiple choices | No; one choice | Upstream-dependent | No; one choice | Yes; all candidates | Upstream-dependent | -| Text output | Converted | Pass-through | Converted | Converted; thought parts excluded | Pass-through | -| Tool calls | Converted | Pass-through | Converted | Converted | Pass-through | -| Finish reason | Converted | Pass-through | Converted | Converted | Pass-through | -| Token usage | Converted, including available cache-read and cache-creation details | Pass-through | Converted, including cache-read and cache-write details | Converted, including cached and reasoning tokens | Pass-through | -| Error envelope | Converted | Pass-through | Converted | Converted | Converted | -| OpenAI SSE conversion | No | Native and passed through | Yes | No | Native and passed through | - -Anthropic produces one OpenAI choice and preserves available cache-read and cache-creation counts in prompt token details. AWS Bedrock produces one choice and retains cache-read and cache-write information for cost calculation. Gemini converts every candidate, preserves candidate indices, excludes `thought: true` parts from visible assistant text, and exposes thought tokens as reasoning tokens. - -## Streaming - -Streaming behavior is not uniform across provider routes. Select a provider based on the response contract required by the client, not only on whether the upstream accepts `stream: true`. - -| Provider route | Upstream stream | Response returned to the client | OpenAI SSE compatible | -|----------------|-----------------|---------------------------------|-----------------------| -| Anthropic | Anthropic SSE | Native Anthropic events are passed through | No | -| Azure OpenAI | Azure OpenAI SSE | Passed through | Yes, subject to deployment and API version | -| AWS Bedrock | Amazon binary event stream | Decoded into OpenAI `chat.completion.chunk` SSE | Yes | -| Gemini | Gemini SSE | Native Gemini events are passed through | No | -| Mistral | OpenAI-compatible SSE | Passed through | Expected, subject to model and API behavior | - -!!! warning "Anthropic and Gemini streams are not OpenAI SSE" - The Anthropic and Gemini transformers select the correct upstream streaming endpoint, but they do not translate the returned provider-native SSE events into OpenAI Chat Completions chunks. Use non-streaming requests for clients that require one uniform OpenAI response contract. - -The Bedrock transformer provides full cross-protocol streaming conversion. It decodes Amazon event-stream frames, converts text and tool-call deltas, maps stream errors, emits usage data, and terminates the stream with `data: [DONE]`. - -## Tools and Multimodal Input - -### Images - -Image support varies by provider: - -- Anthropic accepts OpenAI image parts containing base64 data URIs or remote image URLs. -- AWS Bedrock accepts base64 image data URIs. Remote image URLs are omitted because Bedrock Converse requires image bytes. -- Gemini converts base64 data URIs to inline data and remote URLs to file data. -- Azure OpenAI and Mistral receive image parts unchanged. Support depends on the selected deployment or model. - -The gateway does not inspect a model's capabilities before routing. A successfully transformed request can still be rejected when the selected model does not support image input. - -### Function tools - -Anthropic, AWS Bedrock, and Gemini convert OpenAI function declarations into provider-native tool declarations. Azure OpenAI and Mistral receive `tools` unchanged. - -The transformers accept OpenAI function tools in the following shape: - -```json -{ - "type": "function", - "function": { - "name": "get_weather", - "description": "Get the weather for a city", - "parameters": { - "type": "object", - "properties": { - "city": { - "type": "string" - } - }, - "required": ["city"] - } - } -} -``` +Expand a provider to see its complete transformation behavior. `Converted` means that the transformer explicitly maps a field to the provider-native format. `Pass-through` means that it retains the OpenAI field, subject to support by the selected model and API version. + +??? info "Anthropic" + **Transformer:** [`openai-to-anthropic`](https://wso2.com/api-platform/policy-hub/policies/openai-to-anthropic-transformer) in the Policy Hub. + + **Scope:** The Anthropic transformer targets OpenAI Chat Completions request and response shapes. It does not translate the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs. + + **Capability summary** + + | Capability | Anthropic support | + |------------|-------------------| + | Request handling | Full conversion | + | Image input | Base64 and remote URL | + | Function tools | Converted | + | Non-streaming OpenAI response | Yes | + | OpenAI-compatible streaming | No | + + **Request conversion** + + | OpenAI input | Anthropic behavior | + |--------------|--------------------| + | Request path | Rewritten to `/v1/messages` | + | `model` | Replaced by the required policy model | + | Text messages | Converted to Anthropic message content | + | `system` and `developer` roles | Combined into top-level system text; developer messages are treated as system messages | + | `assistant.tool_calls` and `tool` results | Converted to tool-use and tool-result blocks | + | Image data URI | Converted to an Anthropic base64 image source | + | Remote image URL | Converted to an Anthropic URL image source | + | `max_completion_tokens` and `max_tokens` | Mapped to `max_tokens` | + | `temperature`, `top_p`, and `stop` | Converted | + | `stream` | Passed to Anthropic | + | `tools` and `tool_choice` | Converted | + | `n`, `seed`, `frequency_penalty`, `presence_penalty`, `response_format`, `logprobs`, `top_logprobs`, `logit_bias`, `service_tier`, `store`, `metadata`, and `user` | Omitted | + + The transformer constructs a new Anthropic request body. Any field that is not explicitly converted is omitted. + + **Response conversion** + + - Generates one OpenAI `chat.completion` choice containing converted text, tool calls, and finish reason. + - Converts token usage, including available cache-read and cache-creation counts in prompt token details. + - Converts Anthropic errors to an OpenAI-style error envelope while retaining the upstream HTTP status. + + **Streaming** + + The transformer selects the Anthropic streaming endpoint and passes native Anthropic SSE events through without converting them to OpenAI Chat Completions chunks. OpenAI SSE compatibility is **No**. Use non-streaming requests when the client requires a uniform OpenAI response contract, or handle Anthropic events in the client. + + **Tools and multimodal input** + + | OpenAI function field | Anthropic field | + |-----------------------|-----------------| + | `function.name` | `tools[].name` | + | `function.description` | `tools[].description` | + | `function.parameters` | `tools[].input_schema` | + | Missing parameter schema | Empty object schema | + + | OpenAI `tool_choice` | Anthropic behavior | + |----------------------|--------------------| + | `auto` | `{ "type": "auto" }` | + | `required` | `{ "type": "any" }` | + | `none` | Drops `tools` | + | Named function | `{ "type": "tool", "name": "" }` | + | Unknown or malformed value | Defaults to automatic selection | + + Only tools with `type: function` are translated. Assistant tool calls and tool-result messages are supported across multiple turns. The transformer decodes the JSON string in `function.arguments`; invalid JSON becomes an empty object. Consecutive tool results are grouped into a provider-compatible user turn. Provider-native tools, hosted tools, computer-use tools, web-search tools, MCP declarations, OpenAI custom tools, `parallel_tool_calls`, strict structured-output flags, and provider-specific tool caching are not explicitly translated. + + Image support in the transformer does not guarantee image support in every Anthropic model. The gateway does not negotiate model capabilities before routing. + +??? info "Azure OpenAI" + **Transformer:** [`openai-to-azure-openai`](https://wso2.com/api-platform/policy-hub/policies/openai-to-azure-openai-transformer) in the Policy Hub. + + **Scope:** The Azure OpenAI transformer targets the OpenAI Chat Completions request and response shape exposed by Azure OpenAI. It does not add support for the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs through this route. + + **Capability summary** + + | Capability | Azure OpenAI support | + |------------|----------------------| + | Request handling | Pass-through | + | Image input | Pass-through | + | Function tools | Pass-through | + | Non-streaming OpenAI response | Native | + | OpenAI-compatible streaming | Yes, subject to deployment and API version | + + **Request conversion** + + | OpenAI input | Azure OpenAI behavior | + |--------------|-----------------------| + | Request path | Rewritten to the Azure deployment path with `api-version` | + | `model` | Passed through and used as the deployment fallback when the policy does not override it | + | Messages, images, tools, tool history, and generation parameters | Passed through | + | `stream` | Passed through | + + Azure OpenAI requires a deployment ID from the transformer configuration or the request model. If neither is available, the transformer returns HTTP `400`. + + **Response conversion** + + - Passes through the native OpenAI-compatible completion envelope, choices, text, tool calls, finish reason, usage, and errors. + + **Streaming** + + Azure OpenAI SSE is passed through. OpenAI SSE compatibility is **Yes, subject to the selected deployment and API version**. + + **Tools and multimodal input** + + `tools`, `tool_choice`, tool-call history, tool results, and image content are passed through unchanged. Their acceptance depends on the selected deployment, model, and API version. No fallback behavior is added for an unsupported or malformed `tool_choice`. + +??? info "AWS Bedrock" + **Transformer:** [`openai-to-bedrock-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-bedrock-transformer) in the Policy Hub. + + **Scope:** The AWS Bedrock transformer targets OpenAI Chat Completions requests and responses and the Bedrock Converse APIs. It does not translate the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs. + + **Capability summary** + + | Capability | AWS Bedrock support | + |------------|----------------------| + | Request handling | Full conversion | + | Image input | Base64 only | + | Function tools | Converted | + | Non-streaming OpenAI response | Yes | + | OpenAI-compatible streaming | Yes | + + **Request conversion** + + | OpenAI input | AWS Bedrock behavior | + |--------------|----------------------| + | Request path | Selects Converse or Converse Stream according to `stream` | + | `model` | Uses the policy model when configured; otherwise uses the body model in the path and omits it from the body | + | Text messages | Converted to Converse message content | + | `system` and `developer` roles | Converted to system blocks; developer messages are treated as system messages | + | `assistant.tool_calls` and `tool` results | Converted to tool-use and tool-result blocks | + | Image data URI | Converted to Bedrock image bytes | + | Remote image URL | Omitted because Converse requires image bytes | + | `max_completion_tokens` and `max_tokens` | Mapped to `max_tokens` | + | `temperature`, `top_p`, and `stop` | Converted | + | `tools` and `tool_choice` | Converted | + | `n`, `seed`, `frequency_penalty`, `presence_penalty`, `response_format`, `logprobs`, `top_logprobs`, `logit_bias`, `service_tier`, `store`, `metadata`, and `user` | Omitted | + + The transformer constructs a new Bedrock request body. Any field that is not explicitly converted is omitted. A missing model in both the policy and request returns HTTP `400`. + + **Response conversion** + + - Generates one OpenAI `chat.completion` choice containing converted text, tool calls, and finish reason. + - Converts usage, including cache-read and cache-write details used for cost calculation. + - Converts Bedrock errors to an OpenAI-style error envelope while retaining the upstream HTTP status. + + **Streaming** + + The transformer selects Converse Stream and decodes Amazon binary event-stream frames into OpenAI `chat.completion.chunk` SSE. It converts text and tool-call deltas, maps stream errors, emits usage, and terminates the stream with `data: [DONE]`. OpenAI SSE compatibility is **Yes**. + + **Tools and multimodal input** + + | OpenAI function field | AWS Bedrock field | + |-----------------------|-------------------| + | `function.name` | `toolConfig.tools[].toolSpec.name` | + | `function.description` | `toolConfig.tools[].toolSpec.description` | + | `function.parameters` | `toolConfig.tools[].toolSpec.inputSchema.json` | + | Missing parameter schema | Empty object schema | + + | OpenAI `tool_choice` | AWS Bedrock behavior | + |----------------------|----------------------| + | `auto` | `{ "auto": {} }` | + | `required` | `{ "any": {} }` | + | `none` | Drops `toolConfig` | + | Named function | `{ "tool": { "name": "" } }` | + | Unknown or malformed value | Omits `toolChoice` | + + Only tools with `type: function` are translated. Multi-turn tool calls and results are converted, and streaming tool-call starts and argument deltas are converted to OpenAI chunk deltas. Invalid JSON in historical assistant tool arguments becomes an empty object. Non-function tools, `parallel_tool_calls`, strict structured-output flags, and provider-specific tool caching are not explicitly translated. + + Bedrock accepts base64 image data URIs through this transformer. It does not fetch remote image URLs, and the selected Bedrock model must support the supplied image format and tool features. + +??? info "Gemini" + **Transformer:** [`openai-to-gemini`](https://wso2.com/api-platform/policy-hub/policies/openai-to-gemini-transformer) in the Policy Hub. + + **Scope:** The Gemini transformer targets OpenAI Chat Completions requests and responses and Gemini `generateContent`. It does not translate the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs. + + **Capability summary** + + | Capability | Gemini support | + |------------|----------------| + | Request handling | Full conversion | + | Image input | Base64 and remote URL | + | Function tools | Converted | + | Non-streaming OpenAI response | Yes | + | OpenAI-compatible streaming | No | + + **Request conversion** + + | OpenAI input | Gemini behavior | + |--------------|-----------------| + | Request path | Uses `generateContent` or `streamGenerateContent` with the required policy model | + | `model` | Replaced by the policy model and used in the path | + | Text messages | Converted to Gemini contents and parts | + | `system` and `developer` roles | Converted to `systemInstruction`; developer messages are treated as system messages | + | `assistant.tool_calls` and `tool` results | Converted to function-call and function-response parts | + | Image data URI | Converted to `inlineData` | + | Remote image URL | Converted to `fileData` | + | `max_completion_tokens` and `max_tokens` | Mapped to `maxOutputTokens` | + | `temperature`, `top_p`, `stop`, `seed`, `frequency_penalty`, and `presence_penalty` | Converted | + | `n` | Mapped to `candidateCount` | + | `tools` and `tool_choice` | Converted | + | `response_format`, `logprobs`, `top_logprobs`, `logit_bias`, `service_tier`, `store`, `metadata`, and `user` | Omitted | + + The transformer constructs a new Gemini request body. Any field that is not explicitly converted is omitted. + + **Response conversion** + + - Generates an OpenAI `chat.completion` choice for every Gemini candidate and preserves candidate indices. + - Converts text, tool calls, and finish reasons, while excluding parts marked `thought: true` from visible assistant text. + - Converts usage, including cached tokens and thought tokens exposed as reasoning tokens. + - Converts Gemini errors to an OpenAI-style error envelope while retaining the upstream HTTP status. + + **Streaming** + + The transformer selects `streamGenerateContent` and passes native Gemini SSE events through without converting them to OpenAI Chat Completions chunks. OpenAI SSE compatibility is **No**. Use non-streaming requests when the client requires a uniform OpenAI response contract, or handle Gemini events in the client. + + **Tools and multimodal input** + + | OpenAI function field | Gemini field | + |-----------------------|--------------| + | `function.name` | `tools[].functionDeclarations[].name` | + | `function.description` | `tools[].functionDeclarations[].description` | + | `function.parameters` | `tools[].functionDeclarations[].parameters` | + | Missing parameter schema | Omitted | + + | OpenAI `tool_choice` | Gemini behavior | + |----------------------|-----------------| + | `auto` | Mode `AUTO` | + | `required` | Mode `ANY` | + | `none` | Drops tools and sets mode `NONE` | + | Named function | Mode `ANY` with `allowedFunctionNames` restricted to that function | + | Unknown or malformed value | Defaults to mode `AUTO` | + + Only tools with `type: function` are translated. Multi-turn function calls and responses are converted. Invalid JSON in historical assistant tool arguments becomes an empty object. Non-function tools, `parallel_tool_calls`, strict structured-output flags, and provider-specific tool caching are not explicitly translated. + + Gemini image and tool support still depends on the selected model. The gateway does not check those model capabilities before routing. + +??? info "Mistral" + **Transformer:** [`openai-to-mistral`](https://wso2.com/api-platform/policy-hub/policies/openai-to-mistral-transformer) in the Policy Hub. + + **Scope:** The Mistral transformer targets OpenAI Chat Completions request and response shapes supported by Mistral's OpenAI-compatible API. It does not add support for the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs through this route. + + **Capability summary** -| OpenAI field | Anthropic | AWS Bedrock | Gemini | Azure OpenAI and Mistral | -|--------------|-----------|-------------|--------|--------------------------| -| `function.name` | `tools[].name` | `toolConfig.tools[].toolSpec.name` | `tools[].functionDeclarations[].name` | Passed through | -| `function.description` | `description` | `toolSpec.description` | `description` | Passed through | -| `function.parameters` | `input_schema` | `inputSchema.json` | `parameters` | Passed through | -| Missing parameter schema | Empty object schema | Empty object schema | Omitted | Passed through | + | Capability | Mistral support | + |------------|-----------------| + | Request handling | OpenAI-compatible normalization | + | Image input | Pass-through | + | Function tools | Pass-through | + | Non-streaming OpenAI response | Native and normalized | + | OpenAI-compatible streaming | Yes, subject to model and API behavior | -Only OpenAI tools with `type: function` are explicitly converted. Provider-native tools, hosted tools, computer-use tools, web-search tools, MCP tool declarations, and OpenAI custom tools are not translated. + **Request conversion** -### Tool choice + | OpenAI input | Mistral behavior | + |--------------|------------------| + | Request path | Rewritten to `/v1/chat/completions` | + | `model` | Replaced by the required policy model | + | Messages, system and developer roles, images, tool history, `max_completion_tokens`, `max_tokens`, `temperature`, `top_p`, `stop`, `stream`, `seed`, `frequency_penalty`, `presence_penalty`, `tools`, `tool_choice`, and `response_format` | Passed through | + | `n`, `logprobs`, `top_logprobs`, `logit_bias`, `service_tier`, `store`, `metadata`, and `user` | Removed | -| OpenAI `tool_choice` | Anthropic | AWS Bedrock | Gemini | -|----------------------|-----------|-------------|--------| -| `auto` | Automatic selection | Automatic selection | `AUTO` mode | -| `required` | Any tool | Any tool | `ANY` mode | -| `none` | Tool definitions omitted | Tool configuration omitted | Tools omitted and mode set to `NONE` | -| Named function | Named tool | Named tool | `ANY` mode restricted to the named function | -| Unknown or malformed value | Defaults to automatic selection | Omits `toolChoice` | Defaults to `AUTO` mode | + **Response conversion** -Azure OpenAI and Mistral receive `tools` and `tool_choice` unchanged. Their acceptance depends on the selected model and API version. + - Normalizes the native OpenAI-compatible completion response and model value. + - Passes through upstream choices, text, tool calls, finish reasons, and usage. + - Converts Mistral errors to an OpenAI-style error envelope while retaining the upstream HTTP status. -### Tool-call conversations + **Streaming** -The Anthropic, AWS Bedrock, and Gemini transformers support multi-turn function-tool conversations: + OpenAI-compatible SSE is passed through. OpenAI SSE compatibility is **Yes, subject to model and API behavior**. -- Assistant `tool_calls` are converted into provider-native tool-use or function-call blocks. -- The JSON string in `function.arguments` is decoded into an object. -- A `role: tool` message is converted into a provider-native tool result or function response. -- Consecutive tool results are grouped into a provider-compatible user turn where required. -- Provider tool calls in non-streaming responses are converted back into OpenAI `message.tool_calls`. -- Bedrock streaming tool-call deltas are converted into OpenAI chunk deltas. + **Tools and multimodal input** -Invalid JSON in historical assistant tool arguments is replaced with an empty object. OpenAI-specific strict schemas, `parallel_tool_calls`, provider-specific tool caching, and non-function tool types are not explicitly translated. + `tools`, `tool_choice`, tool-call history, tool results, and image content are passed through unchanged. Their acceptance depends on the selected Mistral model and API behavior. No fallback behavior is added for an unsupported or malformed `tool_choice`. ## Failure Behavior From 25f97cb576441d9e84b9713da25dcb368ec7bcb7 Mon Sep 17 00:00:00 2001 From: Aakash Wijesekara Date: Thu, 6 Aug 2026 09:43:48 +0530 Subject: [PATCH 3/9] Correct transformer reference details --- .../next/llm-proxy/multi-provider-routing.md | 60 ++++++++++--------- 1 file changed, 33 insertions(+), 27 deletions(-) diff --git a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md index ee513a18d..fe2f25a1c 100644 --- a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md +++ b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md @@ -133,7 +133,7 @@ OpenAI-compatible client request Multi-provider LLM proxy | | LLM Header Router selects anthropic-provider - | openai-to-anthropic transforms the request + | openai-to-anthropic-transformer transforms the request | provider loopback authentication is added v Anthropic LLM provider @@ -330,8 +330,8 @@ spec: header: X-API-Key value: ${ANTHROPIC_LOOPBACK_KEY} transformer: - type: openai-to-anthropic - version: v1 + type: openai-to-anthropic-transformer + version: v0 params: model: claude-sonnet-4-5-20250929 @@ -346,7 +346,7 @@ spec: in: header - name: llm-header-router - version: v1 + version: v0 paths: - path: /chat/completions methods: [POST] @@ -439,10 +439,10 @@ Use a transformer when an additional provider does not accept and return the Ope | Target provider | Transformer type used in this guide | Purpose | |-----------------|-------------------------------------|---------| -| Anthropic | `openai-to-anthropic` | Converts OpenAI-compatible requests to Anthropic Messages and converts non-streaming responses back to OpenAI format. | -| Azure OpenAI | `openai-to-azure-openai` | Adapts the request path for an Azure OpenAI deployment and API version. | -| Mistral | `openai-to-mistral` | Normalizes OpenAI-compatible requests and responses for Mistral. | -| Gemini | `openai-to-gemini` | Converts OpenAI-compatible requests and non-streaming responses for Gemini. | +| Anthropic | `openai-to-anthropic-transformer` | Converts OpenAI-compatible requests to Anthropic Messages and converts non-streaming responses back to OpenAI format. | +| Azure OpenAI | `openai-to-azure-openai-transformer` | Adapts the request path for an Azure OpenAI deployment and API version. | +| Mistral | `openai-to-mistral-transformer` | Normalizes OpenAI-compatible requests and responses for Mistral. | +| Gemini | `openai-to-gemini-transformer` | Converts OpenAI-compatible requests and non-streaming responses for Gemini. | | AWS Bedrock | `openai-to-bedrock-transformer` | Converts OpenAI-compatible requests and Bedrock Converse responses, including streaming responses. | A transformer is not required when the selected provider already exposes an OpenAI-compatible API. @@ -459,8 +459,8 @@ A transformer is not required when the selected provider already exposes an Open header: X-API-Key value: transformer: - type: openai-to-azure-openai - version: v1 + type: openai-to-azure-openai-transformer + version: v0 params: model: gpt-4o apiVersion: "2024-02-15-preview" @@ -475,8 +475,8 @@ A transformer is not required when the selected provider already exposes an Open header: X-API-Key value: transformer: - type: openai-to-mistral - version: v1 + type: openai-to-mistral-transformer + version: v0 params: model: mistral-large-latest ``` @@ -490,8 +490,8 @@ A transformer is not required when the selected provider already exposes an Open header: X-API-Key value: transformer: - type: openai-to-gemini - version: v1 + type: openai-to-gemini-transformer + version: v0 params: model: gemini-2.5-flash apiVersion: v1beta @@ -507,7 +507,7 @@ A transformer is not required when the selected provider already exposes an Open value: transformer: type: openai-to-bedrock-transformer - version: v1 + version: v0 params: model: anthropic.claude-3-5-sonnet-20240620-v1:0 ``` @@ -539,8 +539,8 @@ additionalProviders: header: X-API-Key value: transformer: - type: openai-to-anthropic - version: v1 + type: openai-to-anthropic-transformer + version: v0 params: model: claude-sonnet-4-5-20250929 ``` @@ -575,8 +575,8 @@ The alias must: | Field | Required | Description | |-------|----------|-------------| -| `type` | Yes | Installed transformer policy name, such as `openai-to-anthropic` | -| `version` | Yes | Major policy version, such as `v1` | +| `type` | Yes | Installed transformer policy name, such as `openai-to-anthropic-transformer` | +| `version` | Yes | Major policy version, such as `v0` for the current provider transformers | | `params` | No | Transformer-specific parameters, such as `model` or `apiVersion` | #### LLM Header Router parameters @@ -594,10 +594,12 @@ Use `llm-header-router` as the policy name in the configuration. Expand a provider to see its complete transformation behavior. `Converted` means that the transformer explicitly maps a field to the provider-native format. `Pass-through` means that it retains the OpenAI field, subject to support by the selected model and API version. ??? info "Anthropic" - **Transformer:** [`openai-to-anthropic`](https://wso2.com/api-platform/policy-hub/policies/openai-to-anthropic-transformer) in the Policy Hub. + **Transformer:** [`openai-to-anthropic-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-anthropic-transformer) in the Policy Hub. **Scope:** The Anthropic transformer targets OpenAI Chat Completions request and response shapes. It does not translate the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs. + **Configuration:** `model` is required. `anthropicVersion` is optional and defaults to `2023-06-01`. + **Capability summary** | Capability | Anthropic support | @@ -619,7 +621,7 @@ Expand a provider to see its complete transformation behavior. `Converted` means | `assistant.tool_calls` and `tool` results | Converted to tool-use and tool-result blocks | | Image data URI | Converted to an Anthropic base64 image source | | Remote image URL | Converted to an Anthropic URL image source | - | `max_completion_tokens` and `max_tokens` | Mapped to `max_tokens` | + | `max_completion_tokens` and `max_tokens` | Mapped to `max_tokens`; defaults to `4096` when neither field is supplied | | `temperature`, `top_p`, and `stop` | Converted | | `stream` | Passed to Anthropic | | `tools` and `tool_choice` | Converted | @@ -635,7 +637,7 @@ Expand a provider to see its complete transformation behavior. `Converted` means **Streaming** - The transformer selects the Anthropic streaming endpoint and passes native Anthropic SSE events through without converting them to OpenAI Chat Completions chunks. OpenAI SSE compatibility is **No**. Use non-streaming requests when the client requires a uniform OpenAI response contract, or handle Anthropic events in the client. + The transformer selects the Anthropic streaming endpoint and passes native Anthropic server-sent events (SSE) through without converting them to OpenAI Chat Completions chunks. OpenAI SSE compatibility is **No**. Use non-streaming requests when the client requires a uniform OpenAI response contract, or handle Anthropic events in the client. **Tools and multimodal input** @@ -654,15 +656,17 @@ Expand a provider to see its complete transformation behavior. `Converted` means | Named function | `{ "type": "tool", "name": "" }` | | Unknown or malformed value | Defaults to automatic selection | - Only tools with `type: function` are translated. Assistant tool calls and tool-result messages are supported across multiple turns. The transformer decodes the JSON string in `function.arguments`; invalid JSON becomes an empty object. Consecutive tool results are grouped into a provider-compatible user turn. Provider-native tools, hosted tools, computer-use tools, web-search tools, MCP declarations, OpenAI custom tools, `parallel_tool_calls`, strict structured-output flags, and provider-specific tool caching are not explicitly translated. + Only tools with `type: function` are translated. Assistant tool calls and tool-result messages are supported across multiple turns. The transformer decodes the JSON string in `function.arguments`; invalid JSON becomes an empty object. Consecutive tool results are grouped into a provider-compatible user turn. Provider-native tools, hosted tools, computer-use tools, web-search tools, Model Context Protocol (MCP) declarations, OpenAI custom tools, `parallel_tool_calls`, strict structured-output flags, and provider-specific tool caching are not explicitly translated. Image support in the transformer does not guarantee image support in every Anthropic model. The gateway does not negotiate model capabilities before routing. ??? info "Azure OpenAI" - **Transformer:** [`openai-to-azure-openai`](https://wso2.com/api-platform/policy-hub/policies/openai-to-azure-openai-transformer) in the Policy Hub. + **Transformer:** [`openai-to-azure-openai-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-azure-openai-transformer) in the Policy Hub. **Scope:** The Azure OpenAI transformer targets the OpenAI Chat Completions request and response shape exposed by Azure OpenAI. It does not add support for the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs through this route. + **Configuration:** `apiVersion` is required. `model` is optional and falls back to the request body model. `pathSuffix` is optional and defaults to `/chat/completions`. + **Capability summary** | Capability | Azure OpenAI support | @@ -761,10 +765,12 @@ Expand a provider to see its complete transformation behavior. `Converted` means Bedrock accepts base64 image data URIs through this transformer. It does not fetch remote image URLs, and the selected Bedrock model must support the supplied image format and tool features. ??? info "Gemini" - **Transformer:** [`openai-to-gemini`](https://wso2.com/api-platform/policy-hub/policies/openai-to-gemini-transformer) in the Policy Hub. + **Transformer:** [`openai-to-gemini-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-gemini-transformer) in the Policy Hub. **Scope:** The Gemini transformer targets OpenAI Chat Completions requests and responses and Gemini `generateContent`. It does not translate the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs. + **Configuration:** `model` is required. `apiVersion` is optional and defaults to `v1beta`. + **Capability summary** | Capability | Gemini support | @@ -827,7 +833,7 @@ Expand a provider to see its complete transformation behavior. `Converted` means Gemini image and tool support still depends on the selected model. The gateway does not check those model capabilities before routing. ??? info "Mistral" - **Transformer:** [`openai-to-mistral`](https://wso2.com/api-platform/policy-hub/policies/openai-to-mistral-transformer) in the Policy Hub. + **Transformer:** [`openai-to-mistral-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-mistral-transformer) in the Policy Hub. **Scope:** The Mistral transformer targets OpenAI Chat Completions request and response shapes supported by Mistral's OpenAI-compatible API. It does not add support for the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs through this route. @@ -918,7 +924,7 @@ Every effective provider name must be unique. The effective name is `as` when it Make sure that: - `transformer.type` names a transformer supported by your AI Gateway version. -- `transformer.version` uses a major-only version such as `v1`. +- `transformer.version` uses the installed policy's major-only version, such as `v0` for the current provider transformers. - All parameters required by that transformer are present. The gateway resolves the major version to an installed full policy version and rejects invalid transformer configuration during deployment. From 77023da6075c2705b4cae36ffca5fadeddf85b64 Mon Sep 17 00:00:00 2001 From: Aakash Wijesekara Date: Thu, 6 Aug 2026 10:00:04 +0530 Subject: [PATCH 4/9] Adressing code rabbit comments --- .../next/llm-proxy/multi-provider-routing.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md index fe2f25a1c..b874a13da 100644 --- a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md +++ b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md @@ -564,6 +564,8 @@ The alias must: #### `additionalProviders` +This table defines the additional LLM providers that the proxy can route requests to. + | Field | Required | Description | |-------|----------|-------------| | `id` | Yes | ID of an already deployed `LlmProvider` | @@ -573,10 +575,12 @@ The alias must: #### `transformer` +This table defines the transformer configuration for an additional provider. + | Field | Required | Description | |-------|----------|-------------| | `type` | Yes | Installed transformer policy name, such as `openai-to-anthropic-transformer` | -| `version` | Yes | Major policy version, such as `v0` for the current provider transformers | +| `version` | Yes | Major policy version, such as `v0` for the installed provider transformers | | `params` | No | Transformer-specific parameters, such as `model` or `apiVersion` | #### LLM Header Router parameters @@ -637,7 +641,7 @@ Expand a provider to see its complete transformation behavior. `Converted` means **Streaming** - The transformer selects the Anthropic streaming endpoint and passes native Anthropic server-sent events (SSE) through without converting them to OpenAI Chat Completions chunks. OpenAI SSE compatibility is **No**. Use non-streaming requests when the client requires a uniform OpenAI response contract, or handle Anthropic events in the client. + The transformer selects the Anthropic streaming endpoint and passes native Anthropic server-sent events (SSE) through without converting them to OpenAI Chat Completions chunks. OpenAI SSE compatibility is No. Use non-streaming requests when the client requires a uniform OpenAI response contract, or handle Anthropic events in the client. **Tools and multimodal input** @@ -924,7 +928,7 @@ Every effective provider name must be unique. The effective name is `as` when it Make sure that: - `transformer.type` names a transformer supported by your AI Gateway version. -- `transformer.version` uses the installed policy's major-only version, such as `v0` for the current provider transformers. +- `transformer.version` uses the installed policy's major-only version, such as `v0` for the installed provider transformers. - All parameters required by that transformer are present. The gateway resolves the major version to an installed full policy version and rejects invalid transformer configuration during deployment. From 9bf4e1e896dcb4d9bccd9ab687faa5d5ab16332f Mon Sep 17 00:00:00 2001 From: Aakash Wijesekara Date: Thu, 6 Aug 2026 10:21:15 +0530 Subject: [PATCH 5/9] fix limitations issue --- en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md index b874a13da..a3da0c1c0 100644 --- a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md +++ b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md @@ -902,7 +902,7 @@ The round-robin policies track failures per provider/model pair. The same model ## Limitations - **Chat Completions only:** Cross-provider translation targets the OpenAI `/chat/completions` model. -- **No universal streaming abstraction:** Only AWS Bedrock has cross-protocol conversion to OpenAI SSE. Anthropic and Gemini streams remain provider-native. +- **No universal OpenAI streaming conversion:** Only AWS Bedrock converts provider-specific streaming events into OpenAI Chat Completions chunk objects. Anthropic and Gemini return valid SSE streams, but their provider-native event payloads are passed through unchanged. - **No automatic capability negotiation:** The gateway does not query the selected model for support for vision, tools, schemas, or individual generation parameters. - **No automatic routing validation:** Router mappings must match the primary provider ID or an additional provider's effective name. - **No request retry or immediate failover:** Suspension removes an unhealthy target from later rotations but does not retry the failing request. From 8bd9a5692fc272ff087b2b1e951134e144a48281 Mon Sep 17 00:00:00 2001 From: Aakash Wijesekara Date: Thu, 6 Aug 2026 16:50:13 +0530 Subject: [PATCH 6/9] Adress streaming issues --- .../next/llm-proxy/multi-provider-routing.md | 50 +++++++++---------- 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md index a3da0c1c0..edf751c4a 100644 --- a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md +++ b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md @@ -127,23 +127,23 @@ The request flow is: ```text OpenAI-compatible client request - | - | x-provider: anthropic - v - Multi-provider LLM proxy - | - | LLM Header Router selects anthropic-provider - | openai-to-anthropic-transformer transforms the request - | provider loopback authentication is added - v - Anthropic LLM provider - | - | vendor authentication is added - v - Anthropic API - | - | response is transformed to OpenAI format - v + | + | x-provider: anthropic + v + Multi-provider LLM proxy + | + | LLM Header Router selects anthropic-provider + | openai-to-anthropic-transformer transforms the request + | provider loopback authentication is added + v + Anthropic LLM provider + | + | vendor authentication is added + v + Anthropic API + | + | response is transformed to OpenAI format + v OpenAI-compatible client response ``` @@ -612,7 +612,7 @@ Expand a provider to see its complete transformation behavior. `Converted` means | Image input | Base64 and remote URL | | Function tools | Converted | | Non-streaming OpenAI response | Yes | - | OpenAI-compatible streaming | No | + | Streaming response | Native Anthropic SSE passthrough | **Request conversion** @@ -641,7 +641,7 @@ Expand a provider to see its complete transformation behavior. `Converted` means **Streaming** - The transformer selects the Anthropic streaming endpoint and passes native Anthropic server-sent events (SSE) through without converting them to OpenAI Chat Completions chunks. OpenAI SSE compatibility is No. Use non-streaming requests when the client requires a uniform OpenAI response contract, or handle Anthropic events in the client. + Streaming is supported. The transformer selects the Anthropic streaming endpoint and passes native Anthropic server-sent events (SSE) through unchanged. It does not convert the event payloads to OpenAI Chat Completions chunks, so streaming clients must handle Anthropic event payloads. **Tools and multimodal input** @@ -679,7 +679,7 @@ Expand a provider to see its complete transformation behavior. `Converted` means | Image input | Pass-through | | Function tools | Pass-through | | Non-streaming OpenAI response | Native | - | OpenAI-compatible streaming | Yes, subject to deployment and API version | + | Streaming response | OpenAI-compatible SSE passthrough, subject to deployment and API version | **Request conversion** @@ -717,7 +717,7 @@ Expand a provider to see its complete transformation behavior. `Converted` means | Image input | Base64 only | | Function tools | Converted | | Non-streaming OpenAI response | Yes | - | OpenAI-compatible streaming | Yes | + | Streaming response | Converted to OpenAI SSE | **Request conversion** @@ -783,7 +783,7 @@ Expand a provider to see its complete transformation behavior. `Converted` means | Image input | Base64 and remote URL | | Function tools | Converted | | Non-streaming OpenAI response | Yes | - | OpenAI-compatible streaming | No | + | Streaming response | Native Gemini SSE passthrough | **Request conversion** @@ -813,7 +813,7 @@ Expand a provider to see its complete transformation behavior. `Converted` means **Streaming** - The transformer selects `streamGenerateContent` and passes native Gemini SSE events through without converting them to OpenAI Chat Completions chunks. OpenAI SSE compatibility is **No**. Use non-streaming requests when the client requires a uniform OpenAI response contract, or handle Gemini events in the client. + Streaming is supported. The transformer selects `streamGenerateContent` and passes native Gemini SSE events through unchanged. It does not convert the event payloads to OpenAI Chat Completions chunks, so streaming clients must handle Gemini event payloads. **Tools and multimodal input** @@ -849,7 +849,7 @@ Expand a provider to see its complete transformation behavior. `Converted` means | Image input | Pass-through | | Function tools | Pass-through | | Non-streaming OpenAI response | Native and normalized | - | OpenAI-compatible streaming | Yes, subject to model and API behavior | + | Streaming response | OpenAI-compatible SSE passthrough, subject to model and API behavior | **Request conversion** @@ -952,7 +952,7 @@ Model suspension does not retry the current request. Confirm the behavior with a ### Streaming is not in OpenAI chunk format -Anthropic and Gemini streaming responses are provider-native SSE. Use non-streaming mode, choose Azure OpenAI, AWS Bedrock, or an OpenAI-compatible Mistral stream, or adapt the provider-native stream in the client. +Anthropic and Gemini support streaming through provider-native SSE passthrough. If the client expects OpenAI Chat Completions chunks, adapt the provider-native event payloads in the client or choose a route that returns OpenAI-compatible chunks. ### An image or tool request is rejected by the provider From 3c085658999fc1dad046868a03a4a4cb0b979681 Mon Sep 17 00:00:00 2001 From: Aakash Wijesekara Date: Thu, 6 Aug 2026 17:06:13 +0530 Subject: [PATCH 7/9] Adress the coderabbit comment --- en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md index edf751c4a..fbffa114e 100644 --- a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md +++ b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md @@ -902,7 +902,7 @@ The round-robin policies track failures per provider/model pair. The same model ## Limitations - **Chat Completions only:** Cross-provider translation targets the OpenAI `/chat/completions` model. -- **No universal OpenAI streaming conversion:** Only AWS Bedrock converts provider-specific streaming events into OpenAI Chat Completions chunk objects. Anthropic and Gemini return valid SSE streams, but their provider-native event payloads are passed through unchanged. +- No universal OpenAI streaming conversion: Only AWS Bedrock converts provider-specific streaming events into OpenAI Chat Completions chunk objects. Anthropic and Gemini return valid SSE streams, but their provider-native event payloads are passed through unchanged. - **No automatic capability negotiation:** The gateway does not query the selected model for support for vision, tools, schemas, or individual generation parameters. - **No automatic routing validation:** Router mappings must match the primary provider ID or an additional provider's effective name. - **No request retry or immediate failover:** Suspension removes an unhealthy target from later rotations but does not retry the failing request. From 4ff9eaf28c2958a096848cbb16d379afb93bfaf2 Mon Sep 17 00:00:00 2001 From: Aakash Wijesekara Date: Fri, 7 Aug 2026 12:29:33 +0530 Subject: [PATCH 8/9] Add multi-provider routing details for 1.2.0 --- .../1.2.0/llm-proxy/multi-provider-routing.md | 572 +++++++++++++++--- 1 file changed, 504 insertions(+), 68 deletions(-) diff --git a/en/docs/ai-gateway/1.2.0/llm-proxy/multi-provider-routing.md b/en/docs/ai-gateway/1.2.0/llm-proxy/multi-provider-routing.md index 78c0d19ec..fbffa114e 100644 --- a/en/docs/ai-gateway/1.2.0/llm-proxy/multi-provider-routing.md +++ b/en/docs/ai-gateway/1.2.0/llm-proxy/multi-provider-routing.md @@ -8,7 +8,7 @@ tags: - llm - routing author: WSO2 API Platform Documentation Team -last_updated: 2026-07-30 +last_updated: 2026-08-05 content_type: "guide" --- @@ -16,9 +16,9 @@ content_type: "guide" ## Overview -Multi-provider routing lets one large language model (LLM) proxy expose a single OpenAI-compatible endpoint while routing each request to a selected LLM provider. Applications continue to use the same endpoint and OpenAI-compatible request and response format, even when the upstream provider changes. +Multi-provider routing lets one large language model (LLM) proxy expose a single OpenAI-compatible endpoint while routing each request to a selected LLM provider. Applications continue to use the same endpoint and OpenAI-compatible request format when the upstream provider changes. Non-streaming responses are normalized where supported; streaming compatibility varies by provider. -For example, an application can send all requests to `/openai-multi/chat/completions` and select OpenAI or Anthropic with the `x-provider` request header. +For example, an application can send all requests to `/openai-multi/chat/completions` and select OpenAI or Anthropic with the `x-provider` request header. The proxy can also distribute requests automatically across provider and model pairs by using round-robin or weighted round-robin routing. This is useful when you want to: @@ -26,9 +26,95 @@ This is useful when you want to: - Compare provider responses using the same OpenAI-compatible request - Keep vendor credentials in the gateway instead of distributing them to applications - Apply proxy-level authentication, rate limits, and guardrails consistently across providers -- Introduce provider fallback or selection logic through a routing policy +- Introduce provider selection and model suspension through a routing policy -## How It Works +## Choose a Routing Strategy + +Choose one provider-selection strategy for each operation unless you have explicitly designed and tested the precedence between multiple routing policies. + +| Capability | Header router | Model round robin | Model weighted round robin | +|------------|---------------|-------------------|----------------------------| +| Explicit client or provider choice | Yes | No | No | +| Selects a provider | Yes | Optional per model entry | Optional per model entry | +| Selects and rewrites a model | No | Yes | Yes | +| Uses the primary provider when no provider is selected | Yes | Yes | Yes | +| Suspends a provider/model pair after `429` or `5xx` | No | Yes | Yes | +| Weighted traffic distribution | No | No | Yes | +| Latency-, cost-, or semantic-based routing | No | No | No | +| Retries the failed request on another target | No | No | No | + +### Header-based routing + +Use `llm-header-router` when the application or an earlier policy must explicitly choose a provider. The router reads a request header, matches its value against an ordered mapping, and publishes the selected provider name. + +| Parameter | Required | Default | Description | +|-----------|----------|---------|-------------| +| `mappings` | Yes | None | Ordered list of header values and effective provider names. At least one mapping is required. | +| `headerName` | No | `x-provider` | Header used for provider selection. Header-name lookup is case-insensitive. | +| `defaultProvider` | No | Unset | Provider selected when the header is missing, empty, or unmatched. If unset, the primary provider is used. | + +The router has the following selection behavior: + +- Uses only the first value when the header appears more than once +- Trims leading and trailing whitespace from the value +- Matches configured values case-insensitively +- Rejects duplicate mapping values case-insensitively +- Preserves a non-empty provider selection made by an earlier policy +- Leaves the routing header on the upstream request + +The header router publishes provider-selection metadata but does not by itself override the named upstream. An additional provider therefore needs a matching inline transformer, or another policy that explicitly sets its upstream. + +### Model round robin + +Use `model-round-robin` to cycle deterministically through a list of models. A model entry can include a `provider` to route that model to an additional provider. When `provider` is omitted, the model uses the primary provider. + +```yaml +operationPolicies: + - name: model-round-robin + version: v1 + paths: + - path: /chat/completions + methods: [POST] + params: + models: + - model: gpt-4o + - model: claude-sonnet-4-5-20250929 + provider: anthropic-provider + suspendDuration: 30 +``` + +The policy rewrites the model at the location defined by the provider template. It can rewrite a model in the request payload, a header, a query parameter, or a path parameter. + +See [Model Round Robin](load-balancing/model-round-robin.md) for its complete configuration. + +### Model weighted round robin + +Use `model-weighted-round-robin` to distribute requests in a deterministic weighted cycle. Each entry requires an integer `weight` of at least `1`. + +```yaml +operationPolicies: + - name: model-weighted-round-robin + version: v1 + paths: + - path: /chat/completions + methods: [POST] + params: + models: + - model: gpt-4o + weight: 2 + - model: claude-sonnet-4-5-20250929 + provider: anthropic-provider + weight: 1 + suspendDuration: 30 +``` + +This example produces the repeating sequence `gpt-4o`, `gpt-4o`, `claude-sonnet-4-5-20250929` while both targets are available. It provides proportional deterministic distribution, not random or performance-based load balancing. + +See [Model Weighted Round Robin](load-balancing/model-weighted-round-robin.md) for its complete configuration. + +## Configure Providers + +### How provider selection works A multi-provider LLM proxy has: @@ -41,29 +127,38 @@ The request flow is: ```text OpenAI-compatible client request - | - | x-provider: anthropic - v - Multi-provider LLM proxy - | - | LLM Header Router selects anthropic-provider - | openai-to-anthropic transforms the request - | provider loopback authentication is added - v - Anthropic LLM provider - | - | vendor authentication is added - v - Anthropic API - | - | response is transformed to OpenAI format - v + | + | x-provider: anthropic + v + Multi-provider LLM proxy + | + | LLM Header Router selects anthropic-provider + | openai-to-anthropic-transformer transforms the request + | provider loopback authentication is added + v + Anthropic LLM provider + | + | vendor authentication is added + v + Anthropic API + | + | response is transformed to OpenAI format + v OpenAI-compatible client response ``` The router writes the selected provider name to request metadata. The gateway conditionally applies only the authentication and transformer associated with that provider. When the selection header is missing, empty, or does not match a configured mapping, the router uses `defaultProvider` when configured; otherwise, the proxy's primary provider is used. -## Before You Begin +The effective provider name connects routing, transformation, authentication, and upstream selection: + +- The primary provider is identified by `spec.provider.id`. +- An additional provider uses `additionalProviders[].as` when an alias is configured; otherwise, it uses `additionalProviders[].id`. +- Router mappings and model-routing entries must use the effective provider name. +- When no provider is selected, the proxy uses its primary provider. +- Authentication and transformation for an additional provider execute only when that provider is selected. +- The controller injects the effective provider name into an inline transformer's `providerId`; do not configure it manually. + +### Before you begin Make sure that: @@ -74,7 +169,7 @@ Make sure that: This guide configures OpenAI as the primary provider and Anthropic as an additional provider. The same configuration model can be extended to Azure OpenAI, Mistral, Gemini, AWS Bedrock, and other providers supported by your AI Gateway version. -## Understand the Authentication Layers +### Understand the authentication layers Multi-provider routing can involve three different kinds of credentials: @@ -86,11 +181,11 @@ Multi-provider routing can involve three different kinds of credentials: Do not use a vendor API key as a loopback or consumer key. Do not commit any of these credentials to source control. -## Step 1: Deploy the LLM Providers +### Step 1: Deploy the LLM providers Each provider must exist before a proxy can reference it. -### Deploy the OpenAI provider +#### Deploy the OpenAI provider Replace `` with an OpenAI API key. @@ -131,7 +226,7 @@ spec: EOF ``` -### Deploy the Anthropic provider +#### Deploy the Anthropic provider Replace `` with an Anthropic API key. @@ -174,7 +269,7 @@ EOF The vendor credentials under `spec.upstream.auth` are added only when the provider calls its external service. -## Step 2: Create Provider Loopback Keys +### Step 2: Create provider loopback keys Because both providers in this example use the `api-key-auth` policy, create an API key for each provider. The proxy uses these keys when routing to the providers through the gateway's internal loopback route. @@ -203,7 +298,7 @@ test -n "$ANTHROPIC_LOOPBACK_KEY" && test "$ANTHROPIC_LOOPBACK_KEY" != "null" API key values are returned only when they are created or regenerated. Store them securely. -## Step 3: Deploy the Multi-Provider LLM Proxy +### Step 3: Deploy the multi-provider LLM proxy The following proxy exposes one `/chat/completions` operation. OpenAI is the primary and default provider. Anthropic is an additional selectable provider with an inline request and response transformer. @@ -235,8 +330,8 @@ spec: header: X-API-Key value: ${ANTHROPIC_LOOPBACK_KEY} transformer: - type: openai-to-anthropic - version: v1 + type: openai-to-anthropic-transformer + version: v0 params: model: claude-sonnet-4-5-20250929 @@ -251,7 +346,7 @@ spec: in: header - name: llm-header-router - version: v1 + version: v0 paths: - path: /chat/completions methods: [POST] @@ -268,7 +363,7 @@ EOF The controller automatically passes the additional provider's effective upstream name to its transformer. Do not add a `providerId` under `transformer.params`; it is injected from `additionalProviders[].id` or `additionalProviders[].as`. -## Step 4: Create a Proxy Consumer Key +### Step 4: Create a proxy consumer key The proxy uses `api-key-auth` to protect its public endpoint. Create a key for the application that will invoke it: @@ -287,11 +382,11 @@ Verify that a key was returned: test -n "$PROXY_CONSUMER_KEY" && test "$PROXY_CONSUMER_KEY" != "null" ``` -## Step 5: Invoke Different Providers +### Step 5: Invoke different providers All requests use the same URL and OpenAI Chat Completions payload. -### Invoke the default provider +#### Invoke the default provider If `x-provider` is omitted, the router uses `defaultProvider`, which is `openai-provider` in this example. @@ -310,7 +405,7 @@ curl -k -X POST https://localhost:8443/openai-multi/chat/completions \ }' ``` -### Invoke Anthropic +#### Invoke Anthropic Set `x-provider` to the configured `headerValue`: @@ -334,25 +429,28 @@ The Anthropic transformer replaces the request's `model` value with the model co Header names and mapped header values are matched case-insensitively. Leading and trailing whitespace in the header value is ignored. If the header is missing, empty, or does not match a mapping, the router selects `defaultProvider`. -## Add More Providers +### Add more providers Add each selectable provider under `additionalProviders`, then add a corresponding mapping under the LLM Header Router policy (`llm-header-router`). -### Supported provider transformers +#### Supported provider transformers Use a transformer when an additional provider does not accept and return the OpenAI wire format. -| Target provider | Transformer type | Purpose | -|-----------------|------------------|---------| -| Anthropic | `openai-to-anthropic` | Converts OpenAI-compatible requests to the Anthropic Messages format and converts responses back to the OpenAI format. | -| Azure OpenAI | `openai-to-azure-openai` | Adapts OpenAI-compatible requests for Azure OpenAI deployments and API versions. | -| Mistral | `openai-to-mistral` | Adapts OpenAI-compatible requests and responses for Mistral. | -| Gemini | `openai-to-gemini` | Converts OpenAI-compatible requests and responses for Google Gemini. | -| AWS Bedrock | `openai-to-bedrock-transformer` | Converts OpenAI-compatible requests and supported AWS Bedrock responses. | +| Target provider | Transformer type used in this guide | Purpose | +|-----------------|-------------------------------------|---------| +| Anthropic | `openai-to-anthropic-transformer` | Converts OpenAI-compatible requests to Anthropic Messages and converts non-streaming responses back to OpenAI format. | +| Azure OpenAI | `openai-to-azure-openai-transformer` | Adapts the request path for an Azure OpenAI deployment and API version. | +| Mistral | `openai-to-mistral-transformer` | Normalizes OpenAI-compatible requests and responses for Mistral. | +| Gemini | `openai-to-gemini-transformer` | Converts OpenAI-compatible requests and non-streaming responses for Gemini. | +| AWS Bedrock | `openai-to-bedrock-transformer` | Converts OpenAI-compatible requests and Bedrock Converse responses, including streaming responses. | A transformer is not required when the selected provider already exposes an OpenAI-compatible API. -### Azure OpenAI +!!! note "Transformer names and versions" + Transformer names and major versions can differ between AI Gateway releases. Inspect the policy catalog installed with your gateway and use the name and version exposed there. The examples on this page use the policy names supported by this documentation baseline. + +#### Azure OpenAI ```yaml - id: azure-openai-provider @@ -361,14 +459,14 @@ A transformer is not required when the selected provider already exposes an Open header: X-API-Key value: transformer: - type: openai-to-azure-openai - version: v1 + type: openai-to-azure-openai-transformer + version: v0 params: model: gpt-4o apiVersion: "2024-02-15-preview" ``` -### Mistral +#### Mistral ```yaml - id: mistral-provider @@ -377,13 +475,13 @@ A transformer is not required when the selected provider already exposes an Open header: X-API-Key value: transformer: - type: openai-to-mistral - version: v1 + type: openai-to-mistral-transformer + version: v0 params: model: mistral-large-latest ``` -### Gemini +#### Gemini ```yaml - id: gemini-provider @@ -392,14 +490,14 @@ A transformer is not required when the selected provider already exposes an Open header: X-API-Key value: transformer: - type: openai-to-gemini - version: v1 + type: openai-to-gemini-transformer + version: v0 params: model: gemini-2.5-flash apiVersion: v1beta ``` -### AWS Bedrock +#### AWS Bedrock ```yaml - id: aws-bedrock-provider @@ -409,7 +507,7 @@ A transformer is not required when the selected provider already exposes an Open value: transformer: type: openai-to-bedrock-transformer - version: v1 + version: v0 params: model: anthropic.claude-3-5-sonnet-20240620-v1:0 ``` @@ -428,7 +526,7 @@ mappings: provider: aws-bedrock-provider ``` -## Use Provider Aliases +### Use provider aliases Use `as` when the logical upstream name used by routing policies should differ from the deployed provider ID: @@ -441,8 +539,8 @@ additionalProviders: header: X-API-Key value: transformer: - type: openai-to-anthropic - version: v1 + type: openai-to-anthropic-transformer + version: v0 params: model: claude-sonnet-4-5-20250929 ``` @@ -462,9 +560,11 @@ The alias must: - Be unique within the proxy - Not match the primary provider ID or another additional provider's effective name -## Configuration Reference +### Configuration reference + +#### `additionalProviders` -### `additionalProviders` +This table defines the additional LLM providers that the proxy can route requests to. | Field | Required | Description | |-------|----------|-------------| @@ -473,25 +573,347 @@ The alias must: | `auth` | No | API key authentication used by the proxy when calling the provider's internal route | | `transformer` | No | Request and response transformer applied only when this provider is selected | -### `transformer` +#### `transformer` + +This table defines the transformer configuration for an additional provider. | Field | Required | Description | |-------|----------|-------------| -| `type` | Yes | Installed transformer policy name, such as `openai-to-anthropic` | -| `version` | Yes | Major policy version, such as `v1` | +| `type` | Yes | Installed transformer policy name, such as `openai-to-anthropic-transformer` | +| `version` | Yes | Major policy version, such as `v0` for the installed provider transformers | | `params` | No | Transformer-specific parameters, such as `model` or `apiVersion` | -### LLM Header Router parameters +#### LLM Header Router parameters Use `llm-header-router` as the policy name in the configuration. | Parameter | Required | Default | Description | |-----------|----------|---------|-------------| | `headerName` | No | `x-provider` | Request header used for selection | -| `defaultProvider` | No | Primary provider | Effective provider name selected when no mapping matches. When omitted, the proxy's primary provider is used. | +| `defaultProvider` | No | Unset | Effective provider name selected when no mapping matches. When omitted, selection remains unset and the proxy's primary provider is used. | | `mappings` | Yes | None | Header value to effective provider name mappings; the first match wins | -## Validation and Troubleshooting +## Provider Capability Matrix + +Expand a provider to see its complete transformation behavior. `Converted` means that the transformer explicitly maps a field to the provider-native format. `Pass-through` means that it retains the OpenAI field, subject to support by the selected model and API version. + +??? info "Anthropic" + **Transformer:** [`openai-to-anthropic-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-anthropic-transformer) in the Policy Hub. + + **Scope:** The Anthropic transformer targets OpenAI Chat Completions request and response shapes. It does not translate the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs. + + **Configuration:** `model` is required. `anthropicVersion` is optional and defaults to `2023-06-01`. + + **Capability summary** + + | Capability | Anthropic support | + |------------|-------------------| + | Request handling | Full conversion | + | Image input | Base64 and remote URL | + | Function tools | Converted | + | Non-streaming OpenAI response | Yes | + | Streaming response | Native Anthropic SSE passthrough | + + **Request conversion** + + | OpenAI input | Anthropic behavior | + |--------------|--------------------| + | Request path | Rewritten to `/v1/messages` | + | `model` | Replaced by the required policy model | + | Text messages | Converted to Anthropic message content | + | `system` and `developer` roles | Combined into top-level system text; developer messages are treated as system messages | + | `assistant.tool_calls` and `tool` results | Converted to tool-use and tool-result blocks | + | Image data URI | Converted to an Anthropic base64 image source | + | Remote image URL | Converted to an Anthropic URL image source | + | `max_completion_tokens` and `max_tokens` | Mapped to `max_tokens`; defaults to `4096` when neither field is supplied | + | `temperature`, `top_p`, and `stop` | Converted | + | `stream` | Passed to Anthropic | + | `tools` and `tool_choice` | Converted | + | `n`, `seed`, `frequency_penalty`, `presence_penalty`, `response_format`, `logprobs`, `top_logprobs`, `logit_bias`, `service_tier`, `store`, `metadata`, and `user` | Omitted | + + The transformer constructs a new Anthropic request body. Any field that is not explicitly converted is omitted. + + **Response conversion** + + - Generates one OpenAI `chat.completion` choice containing converted text, tool calls, and finish reason. + - Converts token usage, including available cache-read and cache-creation counts in prompt token details. + - Converts Anthropic errors to an OpenAI-style error envelope while retaining the upstream HTTP status. + + **Streaming** + + Streaming is supported. The transformer selects the Anthropic streaming endpoint and passes native Anthropic server-sent events (SSE) through unchanged. It does not convert the event payloads to OpenAI Chat Completions chunks, so streaming clients must handle Anthropic event payloads. + + **Tools and multimodal input** + + | OpenAI function field | Anthropic field | + |-----------------------|-----------------| + | `function.name` | `tools[].name` | + | `function.description` | `tools[].description` | + | `function.parameters` | `tools[].input_schema` | + | Missing parameter schema | Empty object schema | + + | OpenAI `tool_choice` | Anthropic behavior | + |----------------------|--------------------| + | `auto` | `{ "type": "auto" }` | + | `required` | `{ "type": "any" }` | + | `none` | Drops `tools` | + | Named function | `{ "type": "tool", "name": "" }` | + | Unknown or malformed value | Defaults to automatic selection | + + Only tools with `type: function` are translated. Assistant tool calls and tool-result messages are supported across multiple turns. The transformer decodes the JSON string in `function.arguments`; invalid JSON becomes an empty object. Consecutive tool results are grouped into a provider-compatible user turn. Provider-native tools, hosted tools, computer-use tools, web-search tools, Model Context Protocol (MCP) declarations, OpenAI custom tools, `parallel_tool_calls`, strict structured-output flags, and provider-specific tool caching are not explicitly translated. + + Image support in the transformer does not guarantee image support in every Anthropic model. The gateway does not negotiate model capabilities before routing. + +??? info "Azure OpenAI" + **Transformer:** [`openai-to-azure-openai-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-azure-openai-transformer) in the Policy Hub. + + **Scope:** The Azure OpenAI transformer targets the OpenAI Chat Completions request and response shape exposed by Azure OpenAI. It does not add support for the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs through this route. + + **Configuration:** `apiVersion` is required. `model` is optional and falls back to the request body model. `pathSuffix` is optional and defaults to `/chat/completions`. + + **Capability summary** + + | Capability | Azure OpenAI support | + |------------|----------------------| + | Request handling | Pass-through | + | Image input | Pass-through | + | Function tools | Pass-through | + | Non-streaming OpenAI response | Native | + | Streaming response | OpenAI-compatible SSE passthrough, subject to deployment and API version | + + **Request conversion** + + | OpenAI input | Azure OpenAI behavior | + |--------------|-----------------------| + | Request path | Rewritten to the Azure deployment path with `api-version` | + | `model` | Passed through and used as the deployment fallback when the policy does not override it | + | Messages, images, tools, tool history, and generation parameters | Passed through | + | `stream` | Passed through | + + Azure OpenAI requires a deployment ID from the transformer configuration or the request model. If neither is available, the transformer returns HTTP `400`. + + **Response conversion** + + - Passes through the native OpenAI-compatible completion envelope, choices, text, tool calls, finish reason, usage, and errors. + + **Streaming** + + Azure OpenAI SSE is passed through. OpenAI SSE compatibility is **Yes, subject to the selected deployment and API version**. + + **Tools and multimodal input** + + `tools`, `tool_choice`, tool-call history, tool results, and image content are passed through unchanged. Their acceptance depends on the selected deployment, model, and API version. No fallback behavior is added for an unsupported or malformed `tool_choice`. + +??? info "AWS Bedrock" + **Transformer:** [`openai-to-bedrock-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-bedrock-transformer) in the Policy Hub. + + **Scope:** The AWS Bedrock transformer targets OpenAI Chat Completions requests and responses and the Bedrock Converse APIs. It does not translate the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs. + + **Capability summary** + + | Capability | AWS Bedrock support | + |------------|----------------------| + | Request handling | Full conversion | + | Image input | Base64 only | + | Function tools | Converted | + | Non-streaming OpenAI response | Yes | + | Streaming response | Converted to OpenAI SSE | + + **Request conversion** + + | OpenAI input | AWS Bedrock behavior | + |--------------|----------------------| + | Request path | Selects Converse or Converse Stream according to `stream` | + | `model` | Uses the policy model when configured; otherwise uses the body model in the path and omits it from the body | + | Text messages | Converted to Converse message content | + | `system` and `developer` roles | Converted to system blocks; developer messages are treated as system messages | + | `assistant.tool_calls` and `tool` results | Converted to tool-use and tool-result blocks | + | Image data URI | Converted to Bedrock image bytes | + | Remote image URL | Omitted because Converse requires image bytes | + | `max_completion_tokens` and `max_tokens` | Mapped to `max_tokens` | + | `temperature`, `top_p`, and `stop` | Converted | + | `tools` and `tool_choice` | Converted | + | `n`, `seed`, `frequency_penalty`, `presence_penalty`, `response_format`, `logprobs`, `top_logprobs`, `logit_bias`, `service_tier`, `store`, `metadata`, and `user` | Omitted | + + The transformer constructs a new Bedrock request body. Any field that is not explicitly converted is omitted. A missing model in both the policy and request returns HTTP `400`. + + **Response conversion** + + - Generates one OpenAI `chat.completion` choice containing converted text, tool calls, and finish reason. + - Converts usage, including cache-read and cache-write details used for cost calculation. + - Converts Bedrock errors to an OpenAI-style error envelope while retaining the upstream HTTP status. + + **Streaming** + + The transformer selects Converse Stream and decodes Amazon binary event-stream frames into OpenAI `chat.completion.chunk` SSE. It converts text and tool-call deltas, maps stream errors, emits usage, and terminates the stream with `data: [DONE]`. OpenAI SSE compatibility is **Yes**. + + **Tools and multimodal input** + + | OpenAI function field | AWS Bedrock field | + |-----------------------|-------------------| + | `function.name` | `toolConfig.tools[].toolSpec.name` | + | `function.description` | `toolConfig.tools[].toolSpec.description` | + | `function.parameters` | `toolConfig.tools[].toolSpec.inputSchema.json` | + | Missing parameter schema | Empty object schema | + + | OpenAI `tool_choice` | AWS Bedrock behavior | + |----------------------|----------------------| + | `auto` | `{ "auto": {} }` | + | `required` | `{ "any": {} }` | + | `none` | Drops `toolConfig` | + | Named function | `{ "tool": { "name": "" } }` | + | Unknown or malformed value | Omits `toolChoice` | + + Only tools with `type: function` are translated. Multi-turn tool calls and results are converted, and streaming tool-call starts and argument deltas are converted to OpenAI chunk deltas. Invalid JSON in historical assistant tool arguments becomes an empty object. Non-function tools, `parallel_tool_calls`, strict structured-output flags, and provider-specific tool caching are not explicitly translated. + + Bedrock accepts base64 image data URIs through this transformer. It does not fetch remote image URLs, and the selected Bedrock model must support the supplied image format and tool features. + +??? info "Gemini" + **Transformer:** [`openai-to-gemini-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-gemini-transformer) in the Policy Hub. + + **Scope:** The Gemini transformer targets OpenAI Chat Completions requests and responses and Gemini `generateContent`. It does not translate the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs. + + **Configuration:** `model` is required. `apiVersion` is optional and defaults to `v1beta`. + + **Capability summary** + + | Capability | Gemini support | + |------------|----------------| + | Request handling | Full conversion | + | Image input | Base64 and remote URL | + | Function tools | Converted | + | Non-streaming OpenAI response | Yes | + | Streaming response | Native Gemini SSE passthrough | + + **Request conversion** + + | OpenAI input | Gemini behavior | + |--------------|-----------------| + | Request path | Uses `generateContent` or `streamGenerateContent` with the required policy model | + | `model` | Replaced by the policy model and used in the path | + | Text messages | Converted to Gemini contents and parts | + | `system` and `developer` roles | Converted to `systemInstruction`; developer messages are treated as system messages | + | `assistant.tool_calls` and `tool` results | Converted to function-call and function-response parts | + | Image data URI | Converted to `inlineData` | + | Remote image URL | Converted to `fileData` | + | `max_completion_tokens` and `max_tokens` | Mapped to `maxOutputTokens` | + | `temperature`, `top_p`, `stop`, `seed`, `frequency_penalty`, and `presence_penalty` | Converted | + | `n` | Mapped to `candidateCount` | + | `tools` and `tool_choice` | Converted | + | `response_format`, `logprobs`, `top_logprobs`, `logit_bias`, `service_tier`, `store`, `metadata`, and `user` | Omitted | + + The transformer constructs a new Gemini request body. Any field that is not explicitly converted is omitted. + + **Response conversion** + + - Generates an OpenAI `chat.completion` choice for every Gemini candidate and preserves candidate indices. + - Converts text, tool calls, and finish reasons, while excluding parts marked `thought: true` from visible assistant text. + - Converts usage, including cached tokens and thought tokens exposed as reasoning tokens. + - Converts Gemini errors to an OpenAI-style error envelope while retaining the upstream HTTP status. + + **Streaming** + + Streaming is supported. The transformer selects `streamGenerateContent` and passes native Gemini SSE events through unchanged. It does not convert the event payloads to OpenAI Chat Completions chunks, so streaming clients must handle Gemini event payloads. + + **Tools and multimodal input** + + | OpenAI function field | Gemini field | + |-----------------------|--------------| + | `function.name` | `tools[].functionDeclarations[].name` | + | `function.description` | `tools[].functionDeclarations[].description` | + | `function.parameters` | `tools[].functionDeclarations[].parameters` | + | Missing parameter schema | Omitted | + + | OpenAI `tool_choice` | Gemini behavior | + |----------------------|-----------------| + | `auto` | Mode `AUTO` | + | `required` | Mode `ANY` | + | `none` | Drops tools and sets mode `NONE` | + | Named function | Mode `ANY` with `allowedFunctionNames` restricted to that function | + | Unknown or malformed value | Defaults to mode `AUTO` | + + Only tools with `type: function` are translated. Multi-turn function calls and responses are converted. Invalid JSON in historical assistant tool arguments becomes an empty object. Non-function tools, `parallel_tool_calls`, strict structured-output flags, and provider-specific tool caching are not explicitly translated. + + Gemini image and tool support still depends on the selected model. The gateway does not check those model capabilities before routing. + +??? info "Mistral" + **Transformer:** [`openai-to-mistral-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-mistral-transformer) in the Policy Hub. + + **Scope:** The Mistral transformer targets OpenAI Chat Completions request and response shapes supported by Mistral's OpenAI-compatible API. It does not add support for the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs through this route. + + **Capability summary** + + | Capability | Mistral support | + |------------|-----------------| + | Request handling | OpenAI-compatible normalization | + | Image input | Pass-through | + | Function tools | Pass-through | + | Non-streaming OpenAI response | Native and normalized | + | Streaming response | OpenAI-compatible SSE passthrough, subject to model and API behavior | + + **Request conversion** + + | OpenAI input | Mistral behavior | + |--------------|------------------| + | Request path | Rewritten to `/v1/chat/completions` | + | `model` | Replaced by the required policy model | + | Messages, system and developer roles, images, tool history, `max_completion_tokens`, `max_tokens`, `temperature`, `top_p`, `stop`, `stream`, `seed`, `frequency_penalty`, `presence_penalty`, `tools`, `tool_choice`, and `response_format` | Passed through | + | `n`, `logprobs`, `top_logprobs`, `logit_bias`, `service_tier`, `store`, `metadata`, and `user` | Removed | + + **Response conversion** + + - Normalizes the native OpenAI-compatible completion response and model value. + - Passes through upstream choices, text, tool calls, finish reasons, and usage. + - Converts Mistral errors to an OpenAI-style error envelope while retaining the upstream HTTP status. + + **Streaming** + + OpenAI-compatible SSE is passed through. OpenAI SSE compatibility is **Yes, subject to model and API behavior**. + + **Tools and multimodal input** + + `tools`, `tool_choice`, tool-call history, tool results, and image content are passed through unchanged. Their acceptance depends on the selected Mistral model and API behavior. No fallback behavior is added for an unsupported or malformed `tool_choice`. + +## Failure Behavior + +### Routing failures and suspension + +The round-robin policies track failures per provider/model pair. The same model name configured for two providers is therefore suspended independently. + +- A `429` response or any `5xx` response suspends the selected pair. +- `suspendDuration` defaults to 30 seconds. +- Setting `suspendDuration` to `0` disables failure suspension. +- Suspended entries are skipped on later requests until their suspension expires. +- If every entry is suspended, the policy returns HTTP `503` with `All models are currently unavailable`. +- Rotation counters and suspension state are held by the policy instance in memory and are not coordinated across gateway replicas. + +!!! important "Suspension is not a retry" + The request that receives a `429` or `5xx` response is returned to the client. The policy does not replay that request on another provider. Suspension affects only later requests. + +### Transformation failures + +- Empty or invalid JSON request bodies return HTTP `400` in transformers that perform full request conversion. +- Missing required transformer parameters cause policy validation or initialization to fail. +- Azure OpenAI returns HTTP `400` when neither the policy nor the request supplies a deployment ID. +- AWS Bedrock returns HTTP `400` when neither the policy nor the request supplies a model ID. +- A non-JSON successful provider response is generally passed through instead of being replaced with a gateway-generated `500` response. +- Converted provider errors retain the upstream HTTP status and use an OpenAI-style error envelope where supported by the transformer. + +## Limitations + +- **Chat Completions only:** Cross-provider translation targets the OpenAI `/chat/completions` model. +- No universal OpenAI streaming conversion: Only AWS Bedrock converts provider-specific streaming events into OpenAI Chat Completions chunk objects. Anthropic and Gemini return valid SSE streams, but their provider-native event payloads are passed through unchanged. +- **No automatic capability negotiation:** The gateway does not query the selected model for support for vision, tools, schemas, or individual generation parameters. +- **No automatic routing validation:** Router mappings must match the primary provider ID or an additional provider's effective name. +- **No request retry or immediate failover:** Suspension removes an unhealthy target from later rotations but does not retry the failing request. +- **Instance-local state:** Round-robin counters and suspension maps are maintained in memory by each policy instance. +- **Field loss during full conversion:** Anthropic, AWS Bedrock, and Gemini omit request fields that their transformers do not explicitly map. +- **Provider restrictions still apply:** Successful conversion does not guarantee that a model accepts images, tools, tool choice, penalties, candidate counts, or other mapped values. +- **No primary inline transformer:** The inline `transformer` field is available on `additionalProviders`, not on the primary `provider` object. A transformer for another layout must be attached as an operation policy. +- **One routing strategy is recommended:** Combining routing policies can produce precedence-dependent behavior and should be tested explicitly. +- **Header selection needs an upstream override:** A header-routed additional provider without a transformer does not automatically change the named upstream. + +## Troubleshooting ### The additional provider is not found @@ -506,7 +928,7 @@ Every effective provider name must be unique. The effective name is `as` when it Make sure that: - `transformer.type` names a transformer supported by your AI Gateway version. -- `transformer.version` uses a major-only version such as `v1`. +- `transformer.version` uses the installed policy's major-only version, such as `v0` for the installed provider transformers. - All parameters required by that transformer are present. The gateway resolves the major version to an installed full policy version and rejects invalid transformer configuration during deployment. @@ -522,6 +944,20 @@ Check that: An unknown header value intentionally falls back to `defaultProvider`. +If the mapping selects an additional provider that has no transformer, confirm that another operation policy explicitly sets the named upstream. The header router alone publishes selection metadata. + +### The model router does not move to another provider after a failure + +Model suspension does not retry the current request. Confirm the behavior with a later request after the first target returns `429` or `5xx`. Also confirm that `suspendDuration` is greater than `0` and that each model entry uses the correct effective provider name. + +### Streaming is not in OpenAI chunk format + +Anthropic and Gemini support streaming through provider-native SSE passthrough. If the client expects OpenAI Chat Completions chunks, adapt the provider-native event payloads in the client or choose a route that returns OpenAI-compatible chunks. + +### An image or tool request is rejected by the provider + +Transformation support and model support are separate. Check the capability matrix, then verify that the exact selected model supports images, function tools, the requested `tool_choice`, and the supplied JSON Schema. + ### The provider returns `401 Unauthorized` Confirm which authentication layer rejected the request: From 5113399f6eb3a36d05ca6112f37d85f3d0cfb2e0 Mon Sep 17 00:00:00 2001 From: Aakash Wijesekara Date: Fri, 7 Aug 2026 13:12:26 +0530 Subject: [PATCH 9/9] Address multi-provider routing review comments --- .../1.2.0/llm-proxy/multi-provider-routing.md | 139 +++++++++--------- .../next/llm-proxy/multi-provider-routing.md | 2 +- 2 files changed, 73 insertions(+), 68 deletions(-) diff --git a/en/docs/ai-gateway/1.2.0/llm-proxy/multi-provider-routing.md b/en/docs/ai-gateway/1.2.0/llm-proxy/multi-provider-routing.md index fbffa114e..1d212b390 100644 --- a/en/docs/ai-gateway/1.2.0/llm-proxy/multi-provider-routing.md +++ b/en/docs/ai-gateway/1.2.0/llm-proxy/multi-provider-routing.md @@ -8,11 +8,11 @@ tags: - llm - routing author: WSO2 API Platform Documentation Team -last_updated: 2026-08-05 +last_updated: 2026-08-07 content_type: "guide" --- -# Multi-Provider Routing for LLM Proxies +# Multi-provider routing for LLM proxies ## Overview @@ -22,13 +22,13 @@ For example, an application can send all requests to `/openai-multi/chat/complet This is useful when you want to: -- Switch providers without changing application code or endpoint URLs +- Switch providers without changing application code or endpoint Uniform Resource Locators (URLs) - Compare provider responses using the same OpenAI-compatible request - Keep vendor credentials in the gateway instead of distributing them to applications - Apply proxy-level authentication, rate limits, and guardrails consistently across providers - Introduce provider selection and model suspension through a routing policy -## Choose a Routing Strategy +## Choose a routing strategy Choose one provider-selection strategy for each operation unless you have explicitly designed and tested the precedence between multiple routing policies. @@ -112,7 +112,7 @@ This example produces the repeating sequence `gpt-4o`, `gpt-4o`, `claude-sonnet- See [Model Weighted Round Robin](load-balancing/model-weighted-round-robin.md) for its complete configuration. -## Configure Providers +## Configure providers ### How provider selection works @@ -142,7 +142,7 @@ OpenAI-compatible client request v Anthropic API | - | response is transformed to OpenAI format + | non-streaming response is transformed to OpenAI format v OpenAI-compatible client response ``` @@ -151,7 +151,7 @@ The router writes the selected provider name to request metadata. The gateway co The effective provider name connects routing, transformation, authentication, and upstream selection: -- The primary provider is identified by `spec.provider.id`. +- The primary provider is identified by its provider identifier (ID), `spec.provider.id`. - An additional provider uses `additionalProviders[].as` when an alias is configured; otherwise, it uses `additionalProviders[].id`. - Router mappings and model-routing entries must use the effective provider name. - When no provider is selected, the proxy uses its primary provider. @@ -165,6 +165,7 @@ Make sure that: - The AI Gateway is running and the management API is available at `http://localhost:9090/api/management/v1`. - You are using an AI Gateway version that supports multi-provider routing and includes the required router and transformer policies. - You have credentials for each external LLM provider. +- `ADMIN_USERNAME` and `ADMIN_PASSWORD` contain the management API credentials. - `curl` and `jq` are installed if you want to follow the command-line examples. This guide configures OpenAI as the primary provider and Anthropic as an additional provider. The same configuration model can be extended to Azure OpenAI, Mistral, Gemini, AWS Bedrock, and other providers supported by your AI Gateway version. @@ -191,7 +192,7 @@ Replace `` with an OpenAI API key. ```bash curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -u admin:admin \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ -H "Content-Type: application/yaml" \ --data-binary @- <<'EOF' apiVersion: gateway.api-platform.wso2.com/v1 @@ -232,7 +233,7 @@ Replace `` with an Anthropic API key. ```bash curl -X POST http://localhost:9090/api/management/v1/llm-providers \ - -u admin:admin \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ -H "Content-Type: application/yaml" \ --data-binary @- <<'EOF' apiVersion: gateway.api-platform.wso2.com/v1 @@ -276,14 +277,14 @@ Because both providers in this example use the `api-key-auth` policy, create an ```bash OPENAI_LOOPBACK_KEY=$(curl -s -X POST \ http://localhost:9090/api/management/v1/llm-providers/openai-provider/api-keys \ - -u admin:admin \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ -H "Content-Type: application/json" \ -d '{"name":"openai-proxy-loopback"}' \ | jq -r '.apiKey.apiKey') ANTHROPIC_LOOPBACK_KEY=$(curl -s -X POST \ http://localhost:9090/api/management/v1/llm-providers/anthropic-provider/api-keys \ - -u admin:admin \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ -H "Content-Type: application/json" \ -d '{"name":"anthropic-proxy-loopback"}' \ | jq -r '.apiKey.apiKey') @@ -304,7 +305,7 @@ The following proxy exposes one `/chat/completions` operation. OpenAI is the pri ```bash curl -X POST http://localhost:9090/api/management/v1/llm-proxies \ - -u admin:admin \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ -H "Content-Type: application/yaml" \ --data-binary @- <` to each `curl` command. + ```bash -curl -k -X POST https://localhost:8443/openai-multi/chat/completions \ +curl -X POST https://localhost:8443/openai-multi/chat/completions \ -H "Content-Type: application/json" \ -H "X-API-Key: ${PROXY_CONSUMER_KEY}" \ -d '{ @@ -410,7 +413,7 @@ curl -k -X POST https://localhost:8443/openai-multi/chat/completions \ Set `x-provider` to the configured `headerValue`: ```bash -curl -k -X POST https://localhost:8443/openai-multi/chat/completions \ +curl -X POST https://localhost:8443/openai-multi/chat/completions \ -H "Content-Type: application/json" \ -H "X-API-Key: ${PROXY_CONSUMER_KEY}" \ -H "x-provider: anthropic" \ @@ -425,7 +428,7 @@ curl -k -X POST https://localhost:8443/openai-multi/chat/completions \ }' ``` -The Anthropic transformer replaces the request's `model` value with the model configured under `transformer.params.model`. It also translates the request to the Anthropic Messages format and translates the response back to the OpenAI response shape. +The Anthropic transformer replaces the request's `model` value with the model configured under `transformer.params.model`. It also translates the request to the Anthropic Messages format and translates non-streaming responses back to the OpenAI response shape. Header names and mapped header values are matched case-insensitively. Leading and trailing whitespace in the header value is ignored. If the header is missing, empty, or does not match a mapping, the router selects `defaultProvider`. @@ -593,18 +596,18 @@ Use `llm-header-router` as the policy name in the configuration. | `defaultProvider` | No | Unset | Effective provider name selected when no mapping matches. When omitted, selection remains unset and the proxy's primary provider is used. | | `mappings` | Yes | None | Header value to effective provider name mappings; the first match wins | -## Provider Capability Matrix +## Provider capability matrix Expand a provider to see its complete transformation behavior. `Converted` means that the transformer explicitly maps a field to the provider-native format. `Pass-through` means that it retains the OpenAI field, subject to support by the selected model and API version. ??? info "Anthropic" - **Transformer:** [`openai-to-anthropic-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-anthropic-transformer) in the Policy Hub. + Transformer: [`openai-to-anthropic-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-anthropic-transformer) in the Policy Hub. - **Scope:** The Anthropic transformer targets OpenAI Chat Completions request and response shapes. It does not translate the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs. + Scope: The Anthropic transformer targets OpenAI Chat Completions request and response shapes. It does not translate the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs. - **Configuration:** `model` is required. `anthropicVersion` is optional and defaults to `2023-06-01`. + Configuration: `model` is required. `anthropicVersion` is optional and defaults to `2023-06-01`. - **Capability summary** + Capability summary | Capability | Anthropic support | |------------|-------------------| @@ -612,9 +615,9 @@ Expand a provider to see its complete transformation behavior. `Converted` means | Image input | Base64 and remote URL | | Function tools | Converted | | Non-streaming OpenAI response | Yes | - | Streaming response | Native Anthropic SSE passthrough | + | Streaming response | Native Anthropic server-sent events (SSE) passthrough | - **Request conversion** + Request conversion | OpenAI input | Anthropic behavior | |--------------|--------------------| @@ -623,7 +626,7 @@ Expand a provider to see its complete transformation behavior. `Converted` means | Text messages | Converted to Anthropic message content | | `system` and `developer` roles | Combined into top-level system text; developer messages are treated as system messages | | `assistant.tool_calls` and `tool` results | Converted to tool-use and tool-result blocks | - | Image data URI | Converted to an Anthropic base64 image source | + | Image data Uniform Resource Identifier (URI) | Converted to an Anthropic base64 image source | | Remote image URL | Converted to an Anthropic URL image source | | `max_completion_tokens` and `max_tokens` | Mapped to `max_tokens`; defaults to `4096` when neither field is supplied | | `temperature`, `top_p`, and `stop` | Converted | @@ -633,17 +636,17 @@ Expand a provider to see its complete transformation behavior. `Converted` means The transformer constructs a new Anthropic request body. Any field that is not explicitly converted is omitted. - **Response conversion** + Response conversion - Generates one OpenAI `chat.completion` choice containing converted text, tool calls, and finish reason. - Converts token usage, including available cache-read and cache-creation counts in prompt token details. - Converts Anthropic errors to an OpenAI-style error envelope while retaining the upstream HTTP status. - **Streaming** + Streaming Streaming is supported. The transformer selects the Anthropic streaming endpoint and passes native Anthropic server-sent events (SSE) through unchanged. It does not convert the event payloads to OpenAI Chat Completions chunks, so streaming clients must handle Anthropic event payloads. - **Tools and multimodal input** + Tools and multimodal input | OpenAI function field | Anthropic field | |-----------------------|-----------------| @@ -660,18 +663,20 @@ Expand a provider to see its complete transformation behavior. `Converted` means | Named function | `{ "type": "tool", "name": "" }` | | Unknown or malformed value | Defaults to automatic selection | - Only tools with `type: function` are translated. Assistant tool calls and tool-result messages are supported across multiple turns. The transformer decodes the JSON string in `function.arguments`; invalid JSON becomes an empty object. Consecutive tool results are grouped into a provider-compatible user turn. Provider-native tools, hosted tools, computer-use tools, web-search tools, Model Context Protocol (MCP) declarations, OpenAI custom tools, `parallel_tool_calls`, strict structured-output flags, and provider-specific tool caching are not explicitly translated. + Only tools with `type: function` are translated. Assistant tool calls and tool-result messages are supported across multiple turns. The transformer decodes the JSON string in `function.arguments`. Invalid JSON becomes an empty object. Consecutive tool results are grouped into a provider-compatible user turn. + + The transformer does not explicitly translate provider-native tools, hosted tools, computer-use tools, or web-search tools. It also does not translate Model Context Protocol (MCP) declarations, OpenAI custom tools, `parallel_tool_calls`, strict structured-output flags, or provider-specific tool caching. Image support in the transformer does not guarantee image support in every Anthropic model. The gateway does not negotiate model capabilities before routing. ??? info "Azure OpenAI" - **Transformer:** [`openai-to-azure-openai-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-azure-openai-transformer) in the Policy Hub. + Transformer: [`openai-to-azure-openai-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-azure-openai-transformer) in the Policy Hub. - **Scope:** The Azure OpenAI transformer targets the OpenAI Chat Completions request and response shape exposed by Azure OpenAI. It does not add support for the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs through this route. + Scope: The Azure OpenAI transformer targets the OpenAI Chat Completions request and response shape exposed by Azure OpenAI. It does not add support for the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs through this route. - **Configuration:** `apiVersion` is required. `model` is optional and falls back to the request body model. `pathSuffix` is optional and defaults to `/chat/completions`. + Configuration: `apiVersion` is required. `model` is optional and falls back to the request body model. `pathSuffix` is optional and defaults to `/chat/completions`. - **Capability summary** + Capability summary | Capability | Azure OpenAI support | |------------|----------------------| @@ -681,7 +686,7 @@ Expand a provider to see its complete transformation behavior. `Converted` means | Non-streaming OpenAI response | Native | | Streaming response | OpenAI-compatible SSE passthrough, subject to deployment and API version | - **Request conversion** + Request conversion | OpenAI input | Azure OpenAI behavior | |--------------|-----------------------| @@ -692,24 +697,24 @@ Expand a provider to see its complete transformation behavior. `Converted` means Azure OpenAI requires a deployment ID from the transformer configuration or the request model. If neither is available, the transformer returns HTTP `400`. - **Response conversion** + Response conversion - Passes through the native OpenAI-compatible completion envelope, choices, text, tool calls, finish reason, usage, and errors. - **Streaming** + Streaming Azure OpenAI SSE is passed through. OpenAI SSE compatibility is **Yes, subject to the selected deployment and API version**. - **Tools and multimodal input** + Tools and multimodal input `tools`, `tool_choice`, tool-call history, tool results, and image content are passed through unchanged. Their acceptance depends on the selected deployment, model, and API version. No fallback behavior is added for an unsupported or malformed `tool_choice`. ??? info "AWS Bedrock" - **Transformer:** [`openai-to-bedrock-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-bedrock-transformer) in the Policy Hub. + Transformer: [`openai-to-bedrock-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-bedrock-transformer) in the Policy Hub. - **Scope:** The AWS Bedrock transformer targets OpenAI Chat Completions requests and responses and the Bedrock Converse APIs. It does not translate the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs. + Scope: The AWS Bedrock transformer targets OpenAI Chat Completions requests and responses and the Bedrock Converse APIs. It does not translate the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs. - **Capability summary** + Capability summary | Capability | AWS Bedrock support | |------------|----------------------| @@ -719,7 +724,7 @@ Expand a provider to see its complete transformation behavior. `Converted` means | Non-streaming OpenAI response | Yes | | Streaming response | Converted to OpenAI SSE | - **Request conversion** + Request conversion | OpenAI input | AWS Bedrock behavior | |--------------|----------------------| @@ -737,17 +742,17 @@ Expand a provider to see its complete transformation behavior. `Converted` means The transformer constructs a new Bedrock request body. Any field that is not explicitly converted is omitted. A missing model in both the policy and request returns HTTP `400`. - **Response conversion** + Response conversion - Generates one OpenAI `chat.completion` choice containing converted text, tool calls, and finish reason. - Converts usage, including cache-read and cache-write details used for cost calculation. - Converts Bedrock errors to an OpenAI-style error envelope while retaining the upstream HTTP status. - **Streaming** + Streaming - The transformer selects Converse Stream and decodes Amazon binary event-stream frames into OpenAI `chat.completion.chunk` SSE. It converts text and tool-call deltas, maps stream errors, emits usage, and terminates the stream with `data: [DONE]`. OpenAI SSE compatibility is **Yes**. + The transformer selects Converse Stream and decodes Amazon binary event-stream frames into OpenAI `chat.completion.chunk` SSE. It converts text and tool-call deltas. It maps stream errors, emits usage, and terminates the stream with `data: [DONE]`. OpenAI SSE compatibility is **Yes**. - **Tools and multimodal input** + Tools and multimodal input | OpenAI function field | AWS Bedrock field | |-----------------------|-------------------| @@ -769,13 +774,13 @@ Expand a provider to see its complete transformation behavior. `Converted` means Bedrock accepts base64 image data URIs through this transformer. It does not fetch remote image URLs, and the selected Bedrock model must support the supplied image format and tool features. ??? info "Gemini" - **Transformer:** [`openai-to-gemini-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-gemini-transformer) in the Policy Hub. + Transformer: [`openai-to-gemini-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-gemini-transformer) in the Policy Hub. - **Scope:** The Gemini transformer targets OpenAI Chat Completions requests and responses and Gemini `generateContent`. It does not translate the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs. + Scope: The Gemini transformer targets OpenAI Chat Completions requests and responses and Gemini `generateContent`. It does not translate the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs. - **Configuration:** `model` is required. `apiVersion` is optional and defaults to `v1beta`. + Configuration: `model` is required. `apiVersion` is optional and defaults to `v1beta`. - **Capability summary** + Capability summary | Capability | Gemini support | |------------|----------------| @@ -785,7 +790,7 @@ Expand a provider to see its complete transformation behavior. `Converted` means | Non-streaming OpenAI response | Yes | | Streaming response | Native Gemini SSE passthrough | - **Request conversion** + Request conversion | OpenAI input | Gemini behavior | |--------------|-----------------| @@ -804,18 +809,18 @@ Expand a provider to see its complete transformation behavior. `Converted` means The transformer constructs a new Gemini request body. Any field that is not explicitly converted is omitted. - **Response conversion** + Response conversion - Generates an OpenAI `chat.completion` choice for every Gemini candidate and preserves candidate indices. - Converts text, tool calls, and finish reasons, while excluding parts marked `thought: true` from visible assistant text. - Converts usage, including cached tokens and thought tokens exposed as reasoning tokens. - Converts Gemini errors to an OpenAI-style error envelope while retaining the upstream HTTP status. - **Streaming** + Streaming Streaming is supported. The transformer selects `streamGenerateContent` and passes native Gemini SSE events through unchanged. It does not convert the event payloads to OpenAI Chat Completions chunks, so streaming clients must handle Gemini event payloads. - **Tools and multimodal input** + Tools and multimodal input | OpenAI function field | Gemini field | |-----------------------|--------------| @@ -837,11 +842,11 @@ Expand a provider to see its complete transformation behavior. `Converted` means Gemini image and tool support still depends on the selected model. The gateway does not check those model capabilities before routing. ??? info "Mistral" - **Transformer:** [`openai-to-mistral-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-mistral-transformer) in the Policy Hub. + Transformer: [`openai-to-mistral-transformer`](https://wso2.com/api-platform/policy-hub/policies/openai-to-mistral-transformer) in the Policy Hub. - **Scope:** The Mistral transformer targets OpenAI Chat Completions request and response shapes supported by Mistral's OpenAI-compatible API. It does not add support for the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs through this route. + Scope: The Mistral transformer targets OpenAI Chat Completions request and response shapes supported by Mistral's OpenAI-compatible API. It does not add support for the OpenAI Responses API, embeddings, image generation, audio, assistants, batches, or fine-tuning APIs through this route. - **Capability summary** + Capability summary | Capability | Mistral support | |------------|-----------------| @@ -851,7 +856,7 @@ Expand a provider to see its complete transformation behavior. `Converted` means | Non-streaming OpenAI response | Native and normalized | | Streaming response | OpenAI-compatible SSE passthrough, subject to model and API behavior | - **Request conversion** + Request conversion | OpenAI input | Mistral behavior | |--------------|------------------| @@ -860,27 +865,27 @@ Expand a provider to see its complete transformation behavior. `Converted` means | Messages, system and developer roles, images, tool history, `max_completion_tokens`, `max_tokens`, `temperature`, `top_p`, `stop`, `stream`, `seed`, `frequency_penalty`, `presence_penalty`, `tools`, `tool_choice`, and `response_format` | Passed through | | `n`, `logprobs`, `top_logprobs`, `logit_bias`, `service_tier`, `store`, `metadata`, and `user` | Removed | - **Response conversion** + Response conversion - Normalizes the native OpenAI-compatible completion response and model value. - Passes through upstream choices, text, tool calls, finish reasons, and usage. - Converts Mistral errors to an OpenAI-style error envelope while retaining the upstream HTTP status. - **Streaming** + Streaming OpenAI-compatible SSE is passed through. OpenAI SSE compatibility is **Yes, subject to model and API behavior**. - **Tools and multimodal input** + Tools and multimodal input `tools`, `tool_choice`, tool-call history, tool results, and image content are passed through unchanged. Their acceptance depends on the selected Mistral model and API behavior. No fallback behavior is added for an unsupported or malformed `tool_choice`. -## Failure Behavior +## Failure behavior ### Routing failures and suspension -The round-robin policies track failures per provider/model pair. The same model name configured for two providers is therefore suspended independently. +The round-robin policies track failures per model across all requests. A failure suspends every matching model entry, including entries configured for different providers. -- A `429` response or any `5xx` response suspends the selected pair. +- A `429` response or any `5xx` response suspends the selected model. - `suspendDuration` defaults to 30 seconds. - Setting `suspendDuration` to `0` disables failure suspension. - Suspended entries are skipped on later requests until their suspension expires. @@ -901,8 +906,8 @@ The round-robin policies track failures per provider/model pair. The same model ## Limitations -- **Chat Completions only:** Cross-provider translation targets the OpenAI `/chat/completions` model. -- No universal OpenAI streaming conversion: Only AWS Bedrock converts provider-specific streaming events into OpenAI Chat Completions chunk objects. Anthropic and Gemini return valid SSE streams, but their provider-native event payloads are passed through unchanged. +- **Chat Completions only:** Cross-provider translation targets the OpenAI Chat Completions API. +- No universal OpenAI streaming conversion: Only AWS Bedrock converts provider-specific streaming events into OpenAI Chat Completions chunk objects. Anthropic and Gemini return valid SSE streams. Their provider-native event payloads are passed through unchanged. - **No automatic capability negotiation:** The gateway does not query the selected model for support for vision, tools, schemas, or individual generation parameters. - **No automatic routing validation:** Router mappings must match the primary provider ID or an additional provider's effective name. - **No request retry or immediate failover:** Suspension removes an unhealthy target from later rotations but does not retry the failing request. @@ -970,7 +975,7 @@ Confirm which authentication layer rejected the request: The AI Gateway distribution includes the router and transformer policies supported by that version. Use a supported `transformer.type` and major version, or upgrade the AI Gateway to a version that includes the required transformer. -## Security Recommendations +## Security recommendations - Store vendor credentials and loopback keys in a secret manager or Kubernetes `Secret` instead of committing plain-text values. - Protect the proxy with an authentication policy so applications cannot invoke it anonymously. @@ -978,7 +983,7 @@ The AI Gateway distribution includes the router and transformer policies support - Apply rate limiting and guardrails at the provider or proxy level according to your governance requirements. - Use explicit router mappings. Do not accept a client-provided value as an unrestricted upstream name. -## Complete Example +## Complete example For a larger configuration containing OpenAI, Anthropic, Azure OpenAI, Mistral, Gemini, and AWS Bedrock, see [`gateway/examples/openai-multi-provider-proxy.yaml`](https://github.com/wso2/api-platform/blob/main/gateway/examples/openai-multi-provider-proxy.yaml). diff --git a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md index fbffa114e..b2134054e 100644 --- a/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md +++ b/en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md @@ -902,7 +902,7 @@ The round-robin policies track failures per provider/model pair. The same model ## Limitations - **Chat Completions only:** Cross-provider translation targets the OpenAI `/chat/completions` model. -- No universal OpenAI streaming conversion: Only AWS Bedrock converts provider-specific streaming events into OpenAI Chat Completions chunk objects. Anthropic and Gemini return valid SSE streams, but their provider-native event payloads are passed through unchanged. +- No universal OpenAI streaming conversion: Only AWS Bedrock converts provider-specific streaming events into OpenAI Chat Completions chunk objects. Anthropic and Gemini return valid SSE streams. Their provider-native event payloads are passed through unchanged. - **No automatic capability negotiation:** The gateway does not query the selected model for support for vision, tools, schemas, or individual generation parameters. - **No automatic routing validation:** Router mappings must match the primary provider ID or an additional provider's effective name. - **No request retry or immediate failover:** Suspension removes an unhealthy target from later rotations but does not retry the failing request.