Skip to content

RouterProvider: signal-aware shared-flight model discovery so a cancelled waiter can stop the fetch #2202

Open

Description

@easonLiangWorldedtech

Problem

LiteLLMHandler.createMessage and completePrompt settle a cancelled request promptly during model discovery via rejectOnAbort(this.fetchModel(), signal, "LiteLLM"), but the underlying discovery fetch is owned by RouterProvider.fetchModel() (src/api/providers/router-provider.ts:58), which takes no signal. The listener side is clean — rejectOnAbort detaches on settle — yet the single-flight fetch itself keeps running to warm the shared catalog cache after every waiter has gone away.

RouterProvider.fetchModel intentionally shares one in-flight getModels/refreshModels call across concurrent requests (modelFetchPromise), so simply passing one caller's signal through would let a single cancellation reject discovery for the other requests waiting on it.

Proposed direction

  1. Give RouterProvider.fetchModel() a signal-aware shared-flight API: the first caller creates the underlying controller/promise; every caller registers as a waiter and detaches on settle or its own abort.
  2. When the last waiter detaches without a successful resolution, abort the underlying discovery request (propagate the controller signal into getModels/refreshModels).
  3. Normalize the aborted discovery to the provider AbortError contract; preserve the existing negative-cache and missing-model retry semantics.

Acceptance criteria

  • LiteLLMHandler (and other router providers) pass their request signal into the shared-flight discovery API instead of racing it with rejectOnAbort.
  • A cancelled sole waiter aborts the underlying discovery fetch; a mix of surviving and cancelled waiters still resolves for the survivors.
  • Tests cover: single-waiter cancel aborts the fetch, last-waiter-detach aborts, survivor keeps the fetch alive, listener cleanup on both paths.

Split out of the #1303 review (lifecycle finding on lite-llm.ts): the fix belongs in the shared router-provider infrastructure, not in the LiteLLM unit. Current behavior is documented in the #1303 comment: the request settles promptly and listeners detach; only the cache-warming fetch continues.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions