Skip to content

test(flash): isolate the summarizer's model factories from provider credentials - #86

Closed
basil-k-aji-dev wants to merge 1 commit into
google:mainfrom
basil-k-aji-dev:fix/flash-tests-isolate-model-factories
Closed

basil-k-aji-dev wants to merge 1 commit into
google:mainfrom
basil-k-aji-dev:fix/flash-tests-isolate-model-factories

Conversation

@basil-k-aji-dev

Copy link
Copy Markdown

Closes #41.

Problem

Four Flash test modules cannot run without Gemini credentials, contrary to the deterministic-suite contract in CONTRIBUTING.md. On this host (Linux, GOOGLE_API_KEY / GEMINI_API_KEY / OPENAI_API_KEY all unset) they account for 64 of the 89 failures in make test.

VisualStepSummarizer.__init__ resolves its utility model eagerly:

try:
    if model_name:
        self._llm = get_google_llm(model_name=target_model, temperature=0.0)
    else:
        self._llm = get_llm(ctx, name="summarizer", is_utils=True)
except Exception:
    self._llm = get_google_llm(model_name=target_model, temperature=0.0)

Both branches — including the except fallback, which catches any failure of the configured path — end in a real ChatGoogleGenerativeAI construction that raises API key required for Gemini Developer API. The tests assign their fake to summarizer._llm after construction, which is too late: the object never gets built. test_flash_runner* and test_flash_scrub_edge reach the same constructor indirectly through FlashRunner and ScrubEdgeCompressor.

Separately, test_flash_config_and_builder only asserts Flash configuration values but ran the builder's provider validation, failing with Planner requires GOOGLE_API_KEY in .env.

Change

tests/unit/agents/conftest.py (new) — a stub_summarizer_model_factories fixture replacing both factories at the summarizer module boundary, and a small StubChatModel that:

  • exposes model, the one attribute the constructor reads back, so the model-name resolution branch stays exercised rather than silently skipped;
  • raises from ainvoke, so a test that needs generation and forgets to install its own fake fails loudly instead of quietly reaching for the network.

Each of the four modules opts in with a one-line autouse fixture. The real summarizer, its bounded retry loop, the compressor, the ledger and the runner all stay under test — this replaces only the two credential-resolving factories.

test_flash_config_and_builder now calls .build(validate_profiles=False), the option the builder already exposes for configuration-only use.

No production code is touched and no credential validation is relaxed.

Why the module boundary rather than an autouse fixture for tests/unit/agents

A directory-wide autouse fixture would also cover test_explorer.py and test_video_analyzer.py, whose failures are a different bug (#33, MagicMock provider values). Keeping the opt-in explicit matches the issue's scope and keeps the two fixes independently reviewable.

Verification

Targeted, exactly the reproduction in the issue:

$ uv run pytest -q tests/unit/agents/test_flash_runner.py \
                  tests/unit/agents/test_flash_runner_ledger.py \
                  tests/unit/agents/test_flash_scrub_edge.py \
                  tests/unit/agents/test_flash_step_summarizer.py
77 passed in 1.30s

matching the 77 passed the issue predicts (from 64 failed, 13 passed).

Full deterministic suite on this host:

Result
main 89 failed, 2096 passed, 10 skipped, 8 deselected
this branch 26 failed, 2159 passed, 10 skipped, 8 deselected

Exact failed-node comparison: 64 resolved, 0 new failures. The 26 that remain are the groups the issue already attributes elsewhere — #31 (turn-index snapshot, 4), #33 (explorer / video-analyzer provider values, 12), #18/#52 and adb-dependent tests.

Command Result
uv run ruff check tests/unit/agents/ All checks passed
uv run ruff format --check tests/unit/agents/ 41 files already formatted

make typecheck could not be run in my environment — the pyright wheel downloads a Node runtime on first use and my sandbox blocks it. No touched file is in pyright-core.json.

…redentials

Four Flash test modules could not run without Gemini credentials, contrary to
the deterministic-suite contract in CONTRIBUTING.md.

`VisualStepSummarizer.__init__` resolves its utility model through `get_llm`
and falls back to `get_google_llm` on any exception, so with no GOOGLE_API_KEY
both paths end in a real ChatGoogleGenerativeAI construction that raises. The
tests assign their fake to `summarizer._llm` after construction, which is too
late -- the object never gets built. The runner and compressor modules reach
the same constructor indirectly.

Add a `stub_summarizer_model_factories` fixture that replaces both factories at
the summarizer module boundary, and opt the four modules into it. The real
summarizer, its bounded retry loop, the compressor, the ledger and the runner
all stay under test, and every existing assertion is unchanged.

`test_flash_config_and_builder` only asserts Flash configuration values but ran
the builder's provider validation; it now uses the `validate_profiles=False`
option the builder already exposes for configuration-only use.

No production credential validation is relaxed.

Closes google#41.
@basil-k-aji-dev

Copy link
Copy Markdown
Author

Closing as a duplicate. #42 by @wellorbetter already covers issue #41 and was opened first — that one should get the review, not this.

My mistake: I didn't check the open PR queue for an existing claim before sending this. Sorry for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flash unit tests require Gemini credentials before installing their model mocks

1 participant