Skip to content

Avoid unauthenticated model-info lookup for managed OpenHands models during LLM initialization #4098

Closed as not planned

Description

@aivong-openhands

Summary

Constructing or validating an LLM configured with an openhands/* model can synchronously call the managed LiteLLM proxy's GET /v1/model/info endpoint even when no API key is available.

The SDK correctly omits the Authorization header when the key is empty, but it still sends the request. The managed proxy returns 401, the SDK silently falls back to static LiteLLM metadata, and production accumulates roughly 14K–30K unauthenticated model-info responses per day.

For the known managed OpenHands proxy, a request without credentials cannot return key-scoped proxy metadata and should be skipped.

Current behavior

The initialization path is:

  1. LLM._post_init()
  2. LLM._init_model_info_and_caps()
  3. get_litellm_model_info()
  4. _get_model_info_from_litellm_proxy()
  5. httpx.get(f"{base_url}/v1/model/info", headers=headers)

For a missing key, headers remains empty but step 5 still runs.

This commonly occurs during startup, profile loading, validation, or deserialization—not only before a completion. The process-local LRU cache reduces identical calls, but pod restarts, hourly cache keys, concurrency, and key/model cardinality still produce substantial traffic.

PR #1022 previously fixed a blank Authorization: Bearer header by omitting it. That changed the failure from 422 to 401 but retained the request. PR #1148 added caching, but the lookup remains eager during LLM initialization.

Expected behavior

When all of the following are true:

  • the model resolves to the managed OpenHands LiteLLM proxy;
  • the proxy URL is the default or recognized managed OpenHands proxy, rather than an arbitrary custom proxy; and
  • no API key is available;

the SDK should skip /v1/model/info discovery and use its existing static LiteLLM metadata fallback without making an HTTP request.

Unauthenticated discovery must remain supported for generic litellm_proxy/* configurations and custom proxy URLs, because private proxies may intentionally expose model information without authentication.

Authenticated managed-proxy discovery must also remain unchanged because it supplies key-scoped aliases and capability/token-limit overrides.

Suggested implementation

  • Preserve enough pre-translation context to distinguish the managed OpenHands provider from a generic LiteLLM proxy.
  • Before the HTTP request, return None when the target is the managed OpenHands proxy and the resolved API key is empty.
  • Preserve the current static fallback.
  • Consider explicitly handling non-2xx responses instead of treating arbitrary error JSON as an empty model list, without propagating discovery failures from initialization.
  • Coordinate with draft PR fix(llm): bound model-info discovery with a timeout to prevent deadlocks #3912, which adds model-info timeouts but does not address missing credentials or 401 volume.

Acceptance criteria

  • Constructing LLM(model="openhands/<model>") without an API key and without a custom base URL performs no model-info HTTP request.
  • A recognized managed OpenHands proxy URL without an API key also skips the request.
  • An authenticated managed OpenHands configuration still calls /v1/model/info with its Bearer token and applies returned overrides.
  • A custom or authless litellm_proxy/* configuration without an API key can still query model info.
  • A custom openhands/* proxy URL is not incorrectly classified as the managed production proxy.
  • Regression tests cover each case above.

Related work

Activity

  1. all-hands-bot commented on Jul 15, 2026

    @all-hands-bot
    Collaborator

    [Automatic Post]: This issue has been waiting for triage. @xingyaoww @neubig @enyst, could you
    please take a look and add the appropriate priority label when you have a chance?

    This comment was created by an AI agent (OpenHands) on behalf of the user.

  2. all-hands-bot commented on Jul 16, 2026

    @all-hands-bot
    Collaborator

    [Automatic Post]: This issue has been waiting for triage. @xingyaoww @neubig @enyst, could you
    please take a look and add the appropriate priority label when you have a chance?

    This comment was created by an AI agent (OpenHands) on behalf of the user.

  3. all-hands-bot commented on Jul 18, 2026

    @all-hands-bot
    Collaborator

    [Automatic Post]: This issue has been waiting for triage. @xingyaoww @neubig @enyst, could you
    please take a look and add the appropriate priority label when you have a chance?

    This comment was created by an AI agent (OpenHands) on behalf of the user.

  4. all-hands-bot commented on Jul 19, 2026

    @all-hands-bot
    Collaborator

    [Automatic Post]: This issue has been waiting for triage. @xingyaoww @neubig @enyst, could you
    please take a look and add the appropriate priority label when you have a chance?

    This comment was created by an AI agent (OpenHands) on behalf of the repository maintainers.

  5. all-hands-bot commented on Jul 20, 2026

    @all-hands-bot
    Collaborator

    [Automatic Post]: This issue has been waiting for triage. @xingyaoww @neubig @enyst, could you
    please take a look and add the appropriate priority label when you have a chance?

    This comment was generated by an AI agent (OpenHands) on behalf of the user.

  6. all-hands-bot commented on Jul 22, 2026

    @all-hands-bot
    Collaborator

    [Automatic Post]: This issue has been waiting for triage. @xingyaoww @neubig @enyst, could you
    please take a look and add the appropriate priority label when you have a chance?

    This comment was generated by an AI agent (OpenHands) on behalf of the user.

  7. all-hands-bot commented on Jul 24, 2026

    @all-hands-bot
    Collaborator

    [Automatic Post]: This issue has been waiting for triage. @xingyaoww @neubig @enyst, could you
    please take a look and add the appropriate priority label when you have a chance?

    This comment was created by an AI agent (OpenHands) on behalf of the user.

  8. all-hands-bot commented on Jul 29, 2026

    @all-hands-bot
    Collaborator

    [Automatic Post]: This issue has been waiting for triage. @xingyaoww @neubig @enyst, could you
    please take a look and add the appropriate priority label when you have a chance?

    This comment was created by an AI agent (OpenHands) on behalf of the user.

  9. all-hands-bot commented on Jul 31, 2026

    @all-hands-bot
    Collaborator

    [Automatic Post]: This issue has been waiting for triage. @xingyaoww @neubig @enyst, could you please take a look and add the appropriate priority label when you have a chance?

    This comment was created by an AI agent (OpenHands) on behalf of the user.

  10. all-hands-bot commented on Aug 2, 2026

    @all-hands-bot
    Collaborator

    [Automatic Post]: This issue has been waiting for triage. @xingyaoww @neubig @enyst, could you
    please take a look and add the appropriate priority label when you have a chance?

    This comment was posted by an AI agent (OpenHands) on behalf of the user.

  11. all-hands-bot commented on Aug 3, 2026

    @all-hands-bot
    Collaborator

    [Automatic Post]: This issue has been waiting for triage. @xingyaoww @neubig @enyst, could you
    please take a look and add the appropriate priority label when you have a chance?

    This comment was created by an AI agent (OpenHands) on behalf of the user.

  12. all-hands-bot commented on Sep 4, 2026

    @all-hands-bot
    Collaborator

    [Automatic Post]: This issue was automatically closed due to 53 days of inactivity. We do this to help keep the issue tracker manageable and focused on active issues. If this is still relevant, feel free to reopen it.

    Generated by OpenHands automation on behalf of the user.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions