Multi-LLM backend for /api/ai/triage (Anthropic / OpenAI / Doubao / Grok)#9
Merged
Conversation
…Grok
Adds an `LlmProvider` enum and a single dispatcher so the triage
endpoint can run against any of four backends, selected at startup via
the `LLM_PROVIDER` env var. The same prompt and response schema work
across all four; operators pick the provider that fits their stack.
Why this matters:
- The endpoint was previously hard-coded to Anthropic. For
Volcengine-hosted deployments, operators want Doubao; for users
with existing OpenAI/Grok keys, no Anthropic signup overhead;
for the rest, Anthropic remains the default with its prompt-cache
advantage.
- Adds a clean talking point: the AI brain is provider-agnostic by
design. Locking the platform to one LLM vendor would have been a
bad call.
Implementation:
- `LlmProvider` enum (Anthropic / OpenAI / Doubao / Grok) with
`parse`, `as_str`, `default_model`, `Default = Anthropic`. Aliases
accepted (claude, gpt, volcengine, ark, xai).
- `call_llm` dispatches to `call_anthropic` (existing, preserves
`cache_control: ephemeral`) or `call_openai_compatible` (new,
shared by OpenAI/Doubao/Grok — same chat/completions wire format,
same `Authorization: Bearer` auth).
- `TriageResponse` gains a `provider` field so the caller can see
which backend produced a response (useful when rotating providers
during testing).
- Config switches from `ANTHROPIC_API_KEY` to `LLM_PROVIDER` +
`LLM_API_KEY`. Back-compat: `ANTHROPIC_API_KEY` still honored
when `LLM_PROVIDER` is anthropic or unset — existing deployments
don't need to rotate env vars.
Tests:
- 4 existing parse tests still pass.
- 3 new tests: provider label propagation across all parse paths,
`LlmProvider::parse` alias matrix, default-model + Default impl.
Zero new crate dependencies — still just reqwest + serde_json.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the AI triage endpoint provider-agnostic. The LLM backend is now selected at startup via
LLM_PROVIDERenv var; four providers ship out of the box:LLM_PROVIDERanthropic(default)api.anthropic.com/v1/messagesclaude-sonnet-4-6openaiapi.openai.com/v1/chat/completionsgpt-4o-minidoubaoark.cn-beijing.volces.com/api/v3/chat/completionsdoubao-pro-32kgrokapi.x.ai/v1/chat/completionsgrok-3Same prompt, same response schema. Operators pick whichever fits their stack — Doubao for Volcengine-hosted deployments, OpenAI/Grok for users with existing keys, Anthropic for the prompt-cache savings.
Why
The endpoint was previously hard-coded to Anthropic. Adding a provider abstraction:
LLM_API_KEY, differentLLM_PROVIDER.Implementation notes
LlmProviderenum inroutes/ai_triage.rs. Anthropic keeps its ownmessagesendpoint andcache_control: ephemeralprompt caching; OpenAI / Doubao / Grok share onecall_openai_compatibleimplementation that speaks the standard chat-completions wire format withAuthorization: Bearerauth.TriageResponsegains aproviderfield so callers can see which backend produced a response — useful when rotating providers during testing.LLM_PROVIDER+LLM_API_KEY. Back-compat: existingANTHROPIC_API_KEYstill works whenLLM_PROVIDERisanthropicor unset, so existing deployments don't need env-var rotation.claude→ anthropic,gpt→ openai,volcengine/ark→ doubao,xai→ grok.reqwest+serde_jsonfor HTTP and JSON.Test plan
cargo checkcleancargo test routes::ai_triage— 7/7 pass (4 original parse tests + 3 new: provider label propagation,LlmProvider::parsealias matrix, default-model + Default impl)cache_read_input_tokensin logs (provider-specific signal)prompt_tokens_details.cached_tokensfor providers that expose it