Skip to content

Bound queued model-info probes and keep proxy aliases authoritative on lookup failure #5385

Open

Description

@srausser

Remaining model-info discovery failure modes

PR #3912 addresses an indefinite model-info lookup by adding per-probe deadlines and moving profile loads off the agent-server event loop. A downstream patch for the same area also protects two cases that PR does not cover:

  1. A stalled litellm.get_model_info call can remain alive after the caller times out. Repeated LLM constructions can enqueue more work behind a saturated fixed thread pool. Those queued calls still consume a full timeout each, and the queue can grow without a bound.
  2. If a configured litellm_proxy/<alias> lookup fails, falling through to provider-level discovery can issue another slow or unwanted request for the wrong route. Known local model metadata and pricing should remain available for the alias even when the proxy lookup fails.

Reproduction / evidence

For (1), make four provider probes stall, let callers time out, then make many more model-info requests. PR #3912's ThreadPoolExecutor(max_workers=4) still accepts those requests into its queue. A downstream test verifies that saturated probes return promptly and recover capacity when the stalled calls finish.

For (2), make /v1/model/info fail for a proxy alias while the local registry contains the underlying model. The downstream tests assert that no provider fallback begins, known metadata remains available, and a provider-prefixed underlying name can be priced from the local registry.

Expected behavior

Bound both active and queued optional discovery work (for example, acquire a slot without waiting before starting a probe). Keep proxy aliases authoritative: a proxy failure should return available local metadata without launching a different provider route. Every proxy HTTP request should have a timeout.

The downstream patch is currently needed for these safeguards in addition to a deadline. This issue tracks only the remaining behavior so it does not duplicate #3912; removal can happen after a release includes the deadline and these safeguards.


OpenHands AI triage

The following comments and acceptance criteria were added by the OpenHands AI agent.

Triage

This tracks two residual failure modes in model-info discovery in openhands-sdk/openhands/sdk/llm/utils/model_info.py, layered on PR #3912 (per-probe deadline plus off-event-loop profile loads). Scope is limited to the discovery path; it does not cover the agent-server event-loop change (already in #3912) and does not otherwise change the LLM call path. Non-goals: no new configurable surface, no changes to the /v1/model/info matcher, and no work outside model-info discovery.

Evidence: in the current tree, get_litellm_model_info runs its fallbacks against the last path segment via get_model_info(model.split("/")[-1]). For a configured litellm_proxy/<custom-alias>, a failed proxy lookup therefore falls through to provider-level discovery under a different route; and PR #3912's fixed ThreadPoolExecutor(max_workers=4) still accepts unbounded submissions behind stalled probes because it never rejects excess work.

Desired Behavior

  • Optional model-info discovery work is bounded in both active and queued dimensions: starting a new probe must not wait behind saturated workers, and submissions must not accumulate without bound.
  • A failed litellm_proxy/<alias> lookup stays authoritative: return any known local metadata for the alias without launching a provider-level request.
  • Every network request made during discovery carries a finite timeout.

Acceptance Criteria

  • Model-info discovery never lets queued probes grow without bound: once the bounded capacity is saturated, further probes return promptly (degrading to no model info) instead of waiting in an unbounded queue.
  • Capacity recovers automatically when stalled probes complete, and subsequent probes succeed again.
  • When a configured litellm_proxy/<alias> lookup fails or finds no matching entry, discovery returns available local registry metadata for the alias and does not initiate a fallback request for a provider-level route.
  • A proxy HTTP request that does not complete within the configured timeout is abandoned rather than blocking the caller.
  • Existing non-proxy fallbacks still resolve known models when the proxy lookup yields no metadata.
  • Automated tests cover both failure modes: saturated/queued probe bounding with recovery, and the proxy-authoritative path (no provider fallback, local metadata retained, provider-prefixed underlying name still priced from the local registry).
  • Existing model-info and proxy-lookup tests continue to pass unchanged.

Activity

  1. added
    priority:lowFor bugs, affects only non-mainstream cases, or is annoying but with a clear workaround.
    ready-for-devIssue meets development readiness criteria
    on Sep 29, 2026
  2. all-hands-bot commented on Nov 17, 2026

    @all-hands-bot
    Collaborator

    [Automatic Post]: This issue has had no activity for 48 days (labels: priority:low, ready-for-dev). Could someone take a look when they have a chance and help figure out next steps?

    Generated by OpenHands automation on behalf of the user.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority:lowFor bugs, affects only non-mainstream cases, or is annoying but with a clear workaround.ready-for-devIssue meets development readiness criteria

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions