You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
RouterProvider: signal-aware shared-flight model discovery so a cancelled waiter can stop the fetch #2202
LiteLLMHandler.createMessage and completePrompt settle a cancelled request promptly during model discovery via rejectOnAbort(this.fetchModel(), signal, "LiteLLM"), but the underlying discovery fetch is owned by RouterProvider.fetchModel() (src/api/providers/router-provider.ts:58), which takes no signal. The listener side is clean — rejectOnAbort detaches on settle — yet the single-flight fetch itself keeps running to warm the shared catalog cache after every waiter has gone away.
RouterProvider.fetchModel intentionally shares one in-flight getModels/refreshModels call across concurrent requests (modelFetchPromise), so simply passing one caller's signal through would let a single cancellation reject discovery for the other requests waiting on it.
Proposed direction
Give RouterProvider.fetchModel() a signal-aware shared-flight API: the first caller creates the underlying controller/promise; every caller registers as a waiter and detaches on settle or its own abort.
When the last waiter detaches without a successful resolution, abort the underlying discovery request (propagate the controller signal into getModels/refreshModels).
Normalize the aborted discovery to the provider AbortError contract; preserve the existing negative-cache and missing-model retry semantics.
Acceptance criteria
LiteLLMHandler (and other router providers) pass their request signal into the shared-flight discovery API instead of racing it with rejectOnAbort.
A cancelled sole waiter aborts the underlying discovery fetch; a mix of surviving and cancelled waiters still resolves for the survivors.
Tests cover: single-waiter cancel aborts the fetch, last-waiter-detach aborts, survivor keeps the fetch alive, listener cleanup on both paths.
Split out of the #1303 review (lifecycle finding on lite-llm.ts): the fix belongs in the shared router-provider infrastructure, not in the LiteLLM unit. Current behavior is documented in the #1303 comment: the request settles promptly and listeners detach; only the cache-warming fetch continues.
Activity