Add multi-provider LLM routing capabilities - #404
Conversation
|
Warning Review limit reached
Next review available in: 10 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe multi-provider routing guides now document routing strategies, provider selection, setup, transformers, aliases, provider capabilities, failure handling, limitations, and troubleshooting. ChangesMulti-provider routing guide
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md (3)
567-580: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAdd lead-in sentences before the tables.
The tables under
additionalProviders,transformer, and the provider capability sections begin immediately after headings or labels. Add one sentence that explains each table before it starts.As per coding guidelines, introduce every table with meaningful text before it. As per path instructions, report this violation for files under
en/docs/**/*.md.Also applies to: 601-628, 640-657, 666-685, 704-730, 768-795, 808-827, 834-851
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md` around lines 567 - 580, Add a meaningful lead-in sentence immediately before each table under additionalProviders, transformer, and the provider capability sections identified in the review, including the tables near the visible additionalProviders and transformer definitions. Ensure each sentence explains what the following table documents, without changing the table content or surrounding configuration guidance.Sources: Coding guidelines, Path instructions
115-115: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winUse sentence case in the heading.
Change
## Configure Providersto## Configure providers.As per coding guidelines, Markdown headings use sentence case. As per path instructions, report this violation for files under
en/docs/**/*.md.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md` at line 115, Update the “Configure Providers” heading in the multi-provider routing documentation to sentence case: use “Configure providers,” preserving the heading level and surrounding content.Sources: Coding guidelines, Path instructions
448-448: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winClarify the no-transformer case.
An OpenAI-compatible additional provider still needs another policy to set the named upstream. The header router only publishes selection metadata, as stated at Lines 65 and 937. Add that condition here to prevent readers from routing unintentionally to the primary provider.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md` at line 448, Clarify the transformer guidance to state that OpenAI-compatible additional providers still require a policy that sets the named upstream, even when no transformer is needed. Reference the header router’s selection metadata behavior and warn that omitting this policy can route requests to the primary provider.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md`:
- Around line 638-657: Expand the acronyms at their first occurrences in this
documentation section: change SSE to “server-sent events (SSE)” in the streaming
endpoint description and MCP to “Model Context Protocol (MCP)” in the
provider-specific tools list, while preserving the surrounding wording.
---
Outside diff comments:
In `@en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md`:
- Around line 567-580: Add a meaningful lead-in sentence immediately before each
table under additionalProviders, transformer, and the provider capability
sections identified in the review, including the tables near the visible
additionalProviders and transformer definitions. Ensure each sentence explains
what the following table documents, without changing the table content or
surrounding configuration guidance.
- Line 115: Update the “Configure Providers” heading in the multi-provider
routing documentation to sentence case: use “Configure providers,” preserving
the heading level and surrounding content.
- Line 448: Clarify the transformer guidance to state that OpenAI-compatible
additional providers still require a policy that sets the named upstream, even
when no transformer is needed. Reference the header router’s selection metadata
behavior and warn that omitting this policy can route requests to the primary
provider.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 93613c37-8fd4-458d-9097-ea0268fc07ae
📒 Files selected for processing (1)
en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md (1)
563-565: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winIntroduce each configuration table before the table.
The
additionalProvidersandtransformertables begin immediately after their headings. Add one short sentence before each table that states what it defines.As per coding guidelines, introduce every table with meaningful text before it.
Also applies to: 574-578
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md` around lines 563 - 565, Add a brief, meaningful introductory sentence between the `additionalProviders` and `transformer` headings and their respective configuration tables, stating what each table defines. Keep the existing headings and table content unchanged.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md`:
- Line 579: Replace the time-dependent term “current” in both version guidance
locations with the stable phrase “installed provider transformers,” preserving
the existing meaning and formatting.
- Line 640: Update the OpenAI SSE compatibility statement in the transformer
documentation to remove bold formatting from the capability value, leaving it as
plain “No” while preserving the surrounding guidance.
---
Outside diff comments:
In `@en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md`:
- Around line 563-565: Add a brief, meaningful introductory sentence between the
`additionalProviders` and `transformer` headings and their respective
configuration tables, stating what each table defines. Keep the existing
headings and table content unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 30f5191d-df1f-48c5-89c2-641eb28201e4
📒 Files selected for processing (1)
en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md`:
- Line 905: Update the limitation label in the streaming conversion bullet to
remove bold Markdown formatting while preserving the label text, colon, and
surrounding explanation unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 3800e5ed-4aa1-4021-9ab6-a8212675402b
📒 Files selected for processing (1)
en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md
There was a problem hiding this comment.
Actionable comments posted: 8
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
en/docs/ai-gateway/1.2.0/llm-proxy/multi-provider-routing.md (2)
394-394: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick winDo not make TLS verification bypass the default.
Both invocation examples use
-kwhile sendingPROXY_CONSUMER_KEY. The option disables server certificate verification. Remove it for verified connections. If a local self-signed certificate requires it, mark the example as development-only and show a--cacertalternative. (curl.se)Also applies to: 413-413
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@en/docs/ai-gateway/1.2.0/llm-proxy/multi-provider-routing.md` at line 394, Update both curl examples for the multi-provider routing documentation to remove the default `-k` TLS verification bypass while retaining `PROXY_CONSUMER_KEY`. If self-signed local certificates must remain supported, explicitly label that invocation as development-only and provide a `--cacert` alternative.
193-194: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winDo not embed
admin:adminin copy-paste commands.These commands expose a default management password in documentation and command arguments. They also conflict with the instruction at Line 182 to avoid committing credentials. Use environment variables or another approved credential mechanism. The official policy examples use
$ADMIN_USERNAME:$ADMIN_PASSWORD. (wso2.com)Use environment-based management credentials
- -u admin:admin + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD"Based on static analysis, these six command sites contain basic-auth credentials.
Also applies to: 234-235, 277-279, 284-286, 306-307, 371-373
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@en/docs/ai-gateway/1.2.0/llm-proxy/multi-provider-routing.md` around lines 193 - 194, Replace the hard-coded admin:admin credentials in all six curl command sites, including the commands near the referenced sections, with the approved environment-based form using $ADMIN_USERNAME:$ADMIN_PASSWORD. Keep the command behavior unchanged and ensure no default username or password remains embedded in copy-paste documentation.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@en/docs/ai-gateway/1.2.0/llm-proxy/multi-provider-routing.md`:
- Around line 601-607: Remove bold Markdown formatting from body labels
throughout every provider capability block in this document, including
Transformer, Scope, Configuration, Response conversion, and Streaming labels.
Convert them to plain labels or appropriate heading syntax while preserving the
existing text and structure.
- Line 15: Update all Markdown headings in multi-provider-routing.md to sentence
case, including the headings at the referenced locations such as “Multi-provider
routing for LLM proxies,” “Choose a routing strategy,” and “Provider capability
matrix.” Apply the same casing consistently to every title-case heading in the
document.
- Line 663: Split the long capability sentence near the transformer description
into separate short, declarative sentences or bullets, keeping each under 26
words. Apply the same sentence-length adjustment to the streaming-operations
sentence near the corresponding streaming section, preserving all existing
details and meaning.
- Line 904: Update the cross-provider translation note to refer to OpenAI
`/chat/completions` as an API or endpoint, not a model; use “OpenAI Chat
Completions API” or “OpenAI `/chat/completions` endpoint.”
- Line 25: Define each acronym at its first prose occurrence in the
documentation: use “Uniform Resource Locators (URLs),” “provider identifier
(ID),” “server-sent events (SSE),” “Uniform Resource Identifier (URI),” and
“JavaScript Object Notation (JSON),” then retain the abbreviations for later
references throughout the affected sections.
- Line 905: Split the shared streaming limitation into separate sentences in
both en/docs/ai-gateway/1.2.0/llm-proxy/multi-provider-routing.md:905-905 and
en/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md:905-905: state the
AWS Bedrock conversion exception separately from the Anthropic and Gemini
provider-native SSE behavior, preserving the existing meaning.
- Around line 128-147: Qualify response-conversion descriptions in the
multi-provider routing flow diagram and Anthropic/Gemini invocation sections as
applying only to non-streaming responses. Preserve the capability-matrix
behavior that streaming payloads may remain provider-native, and update both the
diagram text and the additional response-conversion description noted in the
comment.
- Around line 881-888: Align the suspension identity across the round-robin
policy implementation and documentation: verify whether suspension is keyed by
provider/model pair or model alone, add coverage for the same model configured
with two providers, and update the guide and linked Model Round Robin
documentation to consistently describe the implemented behavior.
---
Outside diff comments:
In `@en/docs/ai-gateway/1.2.0/llm-proxy/multi-provider-routing.md`:
- Line 394: Update both curl examples for the multi-provider routing
documentation to remove the default `-k` TLS verification bypass while retaining
`PROXY_CONSUMER_KEY`. If self-signed local certificates must remain supported,
explicitly label that invocation as development-only and provide a `--cacert`
alternative.
- Around line 193-194: Replace the hard-coded admin:admin credentials in all six
curl command sites, including the commands near the referenced sections, with
the approved environment-based form using $ADMIN_USERNAME:$ADMIN_PASSWORD. Keep
the command behavior unchanged and ensure no default username or password
remains embedded in copy-paste documentation.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 16aaf743-7e6d-4905-a820-131c4b3384e9
📒 Files selected for processing (2)
en/docs/ai-gateway/1.2.0/llm-proxy/multi-provider-routing.mden/docs/ai-gateway/next/llm-proxy/multi-provider-routing.md
|
@coderabbitai review |
|
1 participant
Purpose
Improve the Multi-Provider Routing for LLM Proxies documentation by providing a complete reference for routing strategies, provider capabilities, request and response transformations, streaming behavior, function tools, multimodal input, failure handling, and known limitations.
The existing guide explained basic provider configuration but did not clearly document behavioral differences across Anthropic, Azure OpenAI, AWS Bedrock, Gemini, and Mistral.
No related issue.
Checklist
llms.txt(located aten/docs/llms.txt) is updated for AI readiness content.Goals
Approach
Reorganized and expanded the existing multi-provider routing guide with:
tool_choicebehavior and fallback handlingNo UI changes were introduced.
User stories
Release note
Expanded the Multi-Provider Routing for LLM Proxies guide with routing strategy guidance, provider capability matrices, streaming and tool-calling compatibility, failure behavior, limitations, and troubleshooting information.
Documentation
Multi-Provider Routing for LLM Proxies
Training
N/A. This documentation update does not require changes to WSO2 training content.
Certification
N/A. This documentation clarification does not introduce certification exam changes.
Marketing
N/A. No marketing content is required for this documentation update.
Automation tests
Unit tests
Integration tests
Security checks
Samples
The guide includes YAML and
curlexamples covering:No new standalone sample project was added.
Related PRs
N/A.
Migrations (if applicable)
N/A. This documentation update does not require migration steps.
Test environment
git diff --checkcompleted successfullyLearning
The documentation was developed by reviewing the existing multi-provider routing guide, the related model round-robin documentation, and implementation-level capability details for the Anthropic, Azure OpenAI, AWS Bedrock, Gemini, and Mistral transformers.
The review focused on differences between explicit field conversion and OpenAI-compatible pass-through behavior, streaming response compatibility, tool-call translation, token usage conversion, and failure suspension semantics.