feat(providers): support multiple credential profiles with weighted load balancing

[jischeng]

Problem

A model connection currently has one effective provider credential. When that credential reaches a rate, usage, or billing limit—or is invalid for one model—the whole connection becomes unavailable even when the user has another API key or provider account for the same endpoint and model configuration.

The current workaround is to create duplicate connections and switch manually. That duplicates endpoint/model/header configuration, loses one stable connection identity, does not provide turn-level stickiness, and cannot safely distinguish credential failover from model/provider fallback.

This is related to the multi-account key-shape question raised in #32, but that RFC only unified credential storage. It is also downstream of the failure taxonomy in #2516 / #2521: automatic account failover must not guess from raw provider error text.

Desired outcome

Allow one connection to own multiple Credential Profiles (a primary profile plus optional secondary profiles), with explicit opt-in weighted load balancing.

The design should preserve these invariants:

• Existing single-credential connections remain an implicit primary profile and keep byte-for-byte legacy behavior while routing mode is disabled.
• Each profile has an independent credential locator, revision/CAS lifecycle, label, weight, enabled state, verification evidence, and health state.
• Balanced routing is explicit; adding a secondary credential must not silently enable it.
• Only enabled, configured profiles with current credential + execution-basis + model-support evidence are eligible.
• Selection uses deterministic smooth weighted round robin, with one sticky binding per turn and credential resolution repeated for each physical provider attempt.
• Credential-global failures and credential+model failures remain separate. One model permission failure must not disable the profile for unrelated models.
• Open circuits allow at most one half-open probe. Abort, failure, replacement, disable, and deletion must not leave a permanently claimed circuit.
• Failover only consumes normalized failure kinds and conservative routing scope. It must remain inactive until fix(runtime): distinguish usage limits from authentication failures #2516 / fix(runtime): distinguish usage limits from auth errors #2521 or an equivalent structured taxonomy lands.
• No credential failover occurs after visible output or tool-call side effects.
• Every physical model-call attempt records the actual selected profile and selection reason, without exposing secrets or mutable labels.
• Desktop exposes profile CRUD, credential replacement, test/discovery, readiness, cooldown state, and explicit routing activation.
• Configuration export/import uses stable export-local profile references, never exports verification/health state, imports profiles disabled, re-verifies them, and restores balanced mode only when activation gates still hold.
• Headless and OAuth-profile support can land in later, independently reviewable slices.

Acceptance criteria

• Legacy primary-only connections have no routing or retry behavior change.
• Weighted distribution and deterministic tie-breaking are covered for multiple ratios.
• Disabled, unconfigured, stale-evidence, unsupported, open, invalid, and half-open profiles are excluded correctly.
• Turn stickiness, per-attempt credential freshness, bounded retry count, single-flight probes, abort cleanup, and model-scoped health isolation have concurrency coverage.
• Balanced activation fails closed unless current verification proves at least one eligible profile for every enabled model and at least one model has two eligible profiles.
• Credential rotation, endpoint/header/protocol changes, profile removal, connection removal, config import, and process restart have lifecycle/crash-safety tests.
• API, protocol, logs, diagnostics, accounting, and exported config never leak credential material.

Alternatives or workarounds

• Duplicate connections + manual switching: works today but duplicates configuration and cannot provide safe turn binding or structured failover.
• Comma-separated API keys: rejected because it has no identity, CAS, verification, health, attribution, or safe import/export semantics.
• Random selection per request: rejected because it breaks turn consistency and makes failures/tool-call continuations difficult to reason about.
• Parsing provider error messages to rotate accounts: rejected because wording is unstable and ambiguous; unknown scope must remain fail closed.
• Quota-aware scheduling: useful only where an official provider API exposes trustworthy quota/reset data; it should be an optional enhancement, not a prerequisite for basic weighted routing.

Related work

• RFC: unify credential storage for desktop + headless (pure-Node CredentialStore) #32 — credential storage and the original multi-account key-shape question
• fix(runtime): distinguish usage limits from authentication failures #2516 — distinguish usage limits from authentication failures
• fix(runtime): distinguish usage limits from auth errors #2521 — proposed normalized provider failure taxonomy
• [Tracking] Runtime Host remote access and production hardening #2522 — standalone Runtime Host tracking; useful for later headless exposure but not a duplicate

Contribution request

I have prepared a detailed design and am willing to implement this in small, independently reviewable PRs, beginning with contracts/storage and keeping automatic failover gated on the structured failure taxonomy.

Maintainers: if this direction is acceptable, please assign this issue to @jischeng.

[me2seeks]

I’m not convinced weighted load balancing is the right abstraction here.

The problem I hit was Kimi Code Plan quota exhaustion being reported as an authentication failure (#2516 / #2521). In that situation, I’d rather see the active connection and explicitly switch through /model. Even when opt-in, weighted routing makes it harder to know which account a turn will use.

The TUI already supports cross-connection switching and shows the active connection slug. The smaller missing piece seems to be making multiple named connections for the same provider easy to configure and showing an actionable “run /model to switch connections” hint.

Weighted round robin is not quota-aware and may consume multiple plans in parallel. The observed Kimi 403 permission_error also remains ambiguous in #2521, so it cannot safely trigger automatic failover.

I’d keep each account as a separate connection. If automatic multi-account routing is needed, I think it belongs in an upstream gateway or relay. Could we narrow this issue to explicit multi-connection configuration and switching, and discuss weighted routing separately?

中文对照

我不太确定加权负载均衡是不是这里合适的抽象。

我实际遇到的问题是 Kimi Code Plan 额度耗尽后被错误显示为鉴权失败（#2516 / #2521）。这种情况下，我更希望看到当前使用的连接，并通过 /model 明确切换。即使负载均衡需要显式启用，它仍会让用户难以知道某个 Turn 实际用了哪个账号。

TUI 已经支持跨连接切换，状态栏也会显示当前连接。现在更小、更直接的缺口，可能是让用户能方便地为同一供应商配置多个命名连接，并在额度错误后提示“运行 /model 切换连接”。

加权轮询并不了解真实额度，还可能同时消耗多个套餐。实测 Kimi 的 403 permission_error 在 #2521 中仍属于语义不明确的错误，因此也不能安全地触发自动 failover。

我倾向于让每个账号保持为独立连接。如果确实需要自动多账号路由，更适合由上游 gateway 或 relay 负责。是否可以把这个 Issue 收缩为显式的多连接配置与切换，再单独讨论加权路由？

[jischeng]

Thanks — I agree with the core safety point: an ambiguous provider 403 must not be treated as a safe quota signal, and automatic distribution should never be the default.\n\nThe implementation I am preparing keeps the explicit path: one provider connection can contain named credential profiles; users can reorder accounts for deterministic priority failover, see per-account quota/reset information, and leave automatic balancing off. Load balancing is an explicit opt-in only; it reacts to classified account-level failures and cooldowns rather than treating an arbitrary 403 as quota exhaustion.\n\nI also agree that /model switching across independent connections remains the right cross-provider/cross-configuration mechanism. I am leaving this issue open because the remaining product question is whether opt-in automatic routing belongs here or at a relay/gateway boundary; the profile-management and explicit ordering pieces are useful independently.