test(flash): isolate the summarizer's model factories from provider credentials - #86
Closed
basil-k-aji-dev wants to merge 1 commit into
Closed
basil-k-aji-dev wants to merge 1 commit into
basil-k-aji-dev wants to merge 1 commit into
Conversation
…redentials Four Flash test modules could not run without Gemini credentials, contrary to the deterministic-suite contract in CONTRIBUTING.md. `VisualStepSummarizer.__init__` resolves its utility model through `get_llm` and falls back to `get_google_llm` on any exception, so with no GOOGLE_API_KEY both paths end in a real ChatGoogleGenerativeAI construction that raises. The tests assign their fake to `summarizer._llm` after construction, which is too late -- the object never gets built. The runner and compressor modules reach the same constructor indirectly. Add a `stub_summarizer_model_factories` fixture that replaces both factories at the summarizer module boundary, and opt the four modules into it. The real summarizer, its bounded retry loop, the compressor, the ledger and the runner all stay under test, and every existing assertion is unchanged. `test_flash_config_and_builder` only asserts Flash configuration values but ran the builder's provider validation; it now uses the `validate_profiles=False` option the builder already exposes for configuration-only use. No production credential validation is relaxed. Closes google#41.
Author
|
Closing as a duplicate. #42 by @wellorbetter already covers issue #41 and was opened first — that one should get the review, not this. My mistake: I didn't check the open PR queue for an existing claim before sending this. Sorry for the noise. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #41.
Problem
Four Flash test modules cannot run without Gemini credentials, contrary to the deterministic-suite contract in
CONTRIBUTING.md. On this host (Linux,GOOGLE_API_KEY/GEMINI_API_KEY/OPENAI_API_KEYall unset) they account for 64 of the 89 failures inmake test.VisualStepSummarizer.__init__resolves its utility model eagerly:Both branches — including the
exceptfallback, which catches any failure of the configured path — end in a realChatGoogleGenerativeAIconstruction that raisesAPI key required for Gemini Developer API. The tests assign their fake tosummarizer._llmafter construction, which is too late: the object never gets built.test_flash_runner*andtest_flash_scrub_edgereach the same constructor indirectly throughFlashRunnerandScrubEdgeCompressor.Separately,
test_flash_config_and_builderonly asserts Flash configuration values but ran the builder's provider validation, failing withPlanner requires GOOGLE_API_KEY in .env.Change
tests/unit/agents/conftest.py(new) — astub_summarizer_model_factoriesfixture replacing both factories at the summarizer module boundary, and a smallStubChatModelthat:model, the one attribute the constructor reads back, so the model-name resolution branch stays exercised rather than silently skipped;ainvoke, so a test that needs generation and forgets to install its own fake fails loudly instead of quietly reaching for the network.Each of the four modules opts in with a one-line autouse fixture. The real summarizer, its bounded retry loop, the compressor, the ledger and the runner all stay under test — this replaces only the two credential-resolving factories.
test_flash_config_and_buildernow calls.build(validate_profiles=False), the option the builder already exposes for configuration-only use.No production code is touched and no credential validation is relaxed.
Why the module boundary rather than an autouse fixture for
tests/unit/agentsA directory-wide autouse fixture would also cover
test_explorer.pyandtest_video_analyzer.py, whose failures are a different bug (#33, MagicMock provider values). Keeping the opt-in explicit matches the issue's scope and keeps the two fixes independently reviewable.Verification
Targeted, exactly the reproduction in the issue:
matching the
77 passedthe issue predicts (from64 failed, 13 passed).Full deterministic suite on this host:
mainExact failed-node comparison: 64 resolved, 0 new failures. The 26 that remain are the groups the issue already attributes elsewhere — #31 (turn-index snapshot, 4), #33 (explorer / video-analyzer provider values, 12), #18/#52 and adb-dependent tests.
uv run ruff check tests/unit/agents/uv run ruff format --check tests/unit/agents/make typecheckcould not be run in my environment — thepyrightwheel downloads a Node runtime on first use and my sandbox blocks it. No touched file is inpyright-core.json.