Skip to content

fix: isolate provider-rate tests from mutable model catalogs #358

Description

@gloryfromca

Summary

The Python unit job is red on main and on pull requests that do not touch Python because provider-rate tests depend on mutable LiteLLM and OpenRouter catalog state. The exact main base and PR #355 fail with the same 19 assertions while all TUI checks pass.

The tests mock an OpenRouter response containing deepseek/deepseek-v4-pro with a 163,840-token context window and fixed prices. In CI, another catalog tier or process state answers first with a 1,048,576-token window and different prices. The mocked fetch is therefore never called, and disk-cache assertions observe no write.

Steps to reproduce

  1. Check out main at 0e544ecb3493cfd51fdc910dedd037f71188ccd0.

  2. Run:

    uv run pytest tests/test_agent_loop_usage_sink.py tests/test_provider_rates.py
  3. Compare the full CI runs:

The CI run reports 19 failures and 6,700 passes. A current local targeted run also remains catalog-sensitive: 65 tests pass and two MiniMax catalog assertions fail.

Expected behavior

Provider-rate unit tests should be deterministic and isolated from mutable LiteLLM and OpenRouter metadata, test order, process-wide caches, and local disk state. A TUI-only pull request should not inherit unrelated Python failures from the same base.

Actual behavior

The test fixtures expect mocked values such as context window 163840, but CI resolves 1048576 and different prices before reaching the mock tier. Fetch counters stay at zero, cache files are not written, and allow_fetch=False assertions receive catalog data instead of None.

The exact base commit fails with the same 19 tests as PR #355, so rerunning or rebasing the pull request onto the unchanged main does not address the cause.

Environment

  • CI: GitHub-hosted ubuntu-24.04
  • Python: 3.12
  • Dependency command: uv run --frozen
  • Raven: main@0e544ecb3493cfd51fdc910dedd037f71188ccd0
  • PR evidence: #355@7a7dc1a34c6769550f12eccf4c80f2cb2be67f61

Logs or screenshots

Representative failures:

assert 1048576 == 163840
assert 1048576 is None
assert 0.0033 == 0.00125
assert counter["calls"] == 1  # actual: 0
FileNotFoundError: .../model-catalog.json
19 failed, 6700 passed, 33 skipped

Acceptance criteria

  • Provider catalog tests patch every upstream tier that can answer before the tier under test.
  • Process-wide caches and timestamps are restored between tests.
  • Disk-cache tests use only their per-test path.
  • The focused files pass in isolation and in full-suite order.
  • The full Python unit job passes on main.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpython:uvPull requests that update python:uv code

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions