Summary
The Python unit job is red on main and on pull requests that do not touch Python because provider-rate tests depend on mutable LiteLLM and OpenRouter catalog state. The exact main base and PR #355 fail with the same 19 assertions while all TUI checks pass.
The tests mock an OpenRouter response containing deepseek/deepseek-v4-pro with a 163,840-token context window and fixed prices. In CI, another catalog tier or process state answers first with a 1,048,576-token window and different prices. The mocked fetch is therefore never called, and disk-cache assertions observe no write.
Steps to reproduce
-
Check out main at 0e544ecb3493cfd51fdc910dedd037f71188ccd0.
-
Run:
uv run pytest tests/test_agent_loop_usage_sink.py tests/test_provider_rates.py
-
Compare the full CI runs:
The CI run reports 19 failures and 6,700 passes. A current local targeted run also remains catalog-sensitive: 65 tests pass and two MiniMax catalog assertions fail.
Expected behavior
Provider-rate unit tests should be deterministic and isolated from mutable LiteLLM and OpenRouter metadata, test order, process-wide caches, and local disk state. A TUI-only pull request should not inherit unrelated Python failures from the same base.
Actual behavior
The test fixtures expect mocked values such as context window 163840, but CI resolves 1048576 and different prices before reaching the mock tier. Fetch counters stay at zero, cache files are not written, and allow_fetch=False assertions receive catalog data instead of None.
The exact base commit fails with the same 19 tests as PR #355, so rerunning or rebasing the pull request onto the unchanged main does not address the cause.
Environment
- CI: GitHub-hosted
ubuntu-24.04
- Python: 3.12
- Dependency command:
uv run --frozen
- Raven:
main@0e544ecb3493cfd51fdc910dedd037f71188ccd0
- PR evidence:
#355@7a7dc1a34c6769550f12eccf4c80f2cb2be67f61
Logs or screenshots
Representative failures:
assert 1048576 == 163840
assert 1048576 is None
assert 0.0033 == 0.00125
assert counter["calls"] == 1 # actual: 0
FileNotFoundError: .../model-catalog.json
19 failed, 6700 passed, 33 skipped
Acceptance criteria
- Provider catalog tests patch every upstream tier that can answer before the tier under test.
- Process-wide caches and timestamps are restored between tests.
- Disk-cache tests use only their per-test path.
- The focused files pass in isolation and in full-suite order.
- The full Python unit job passes on
main.
Summary
The Python unit job is red on
mainand on pull requests that do not touch Python because provider-rate tests depend on mutable LiteLLM and OpenRouter catalog state. The exactmainbase and PR #355 fail with the same 19 assertions while all TUI checks pass.The tests mock an OpenRouter response containing
deepseek/deepseek-v4-prowith a 163,840-token context window and fixed prices. In CI, another catalog tier or process state answers first with a 1,048,576-token window and different prices. The mocked fetch is therefore never called, and disk-cache assertions observe no write.Steps to reproduce
Check out
mainat0e544ecb3493cfd51fdc910dedd037f71188ccd0.Run:
Compare the full CI runs:
The CI run reports 19 failures and 6,700 passes. A current local targeted run also remains catalog-sensitive: 65 tests pass and two MiniMax catalog assertions fail.
Expected behavior
Provider-rate unit tests should be deterministic and isolated from mutable LiteLLM and OpenRouter metadata, test order, process-wide caches, and local disk state. A TUI-only pull request should not inherit unrelated Python failures from the same base.
Actual behavior
The test fixtures expect mocked values such as context window
163840, but CI resolves1048576and different prices before reaching the mock tier. Fetch counters stay at zero, cache files are not written, andallow_fetch=Falseassertions receive catalog data instead ofNone.The exact base commit fails with the same 19 tests as PR #355, so rerunning or rebasing the pull request onto the unchanged
maindoes not address the cause.Environment
ubuntu-24.04uv run --frozenmain@0e544ecb3493cfd51fdc910dedd037f71188ccd0#355@7a7dc1a34c6769550f12eccf4c80f2cb2be67f61Logs or screenshots
Representative failures:
Acceptance criteria
main.