Skip to content

[Performance]: Cache the OpenRouter model catalog with a TTL #56

Description

@shauryagangrade

Problem

list_free_models hits https://openrouter.ai/api/v1/models on every call.
/models and startup both fetch the full catalog each time — a wasted
network round-trip and a slow menu on a flaky connection.

Proposed approach

  • Add a short-lived in-process cache (e.g. 10-60 min TTL) so repeated calls
    within a session reuse the last response.
  • Keep a 0-TTL escape hatch for anything that needs fresh data.
  • Respect the existing best-effort contract (empty list + error on failure).

Where to look

  • gcode/models.py:18-55 (list_free_models).

Acceptance criteria

  • Two consecutive list_free_models() calls inside the TTL make one HTTP request.
  • Cache invalidation on expiry is tested.

Difficulty

Low — great first issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    good first issueGood for newcomersperformanceLatency, throughput, and efficiency improvements

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions