Skip to content

test(flash): build the turn-index runner without provider credentials - #91

Closed
basil-k-aji-dev wants to merge 1 commit into
google:mainfrom
basil-k-aji-dev:fix/turn-index-snapshot-offline
Closed

basil-k-aji-dev wants to merge 1 commit into
google:mainfrom
basil-k-aji-dev:fix/turn-index-snapshot-offline

Conversation

@basil-k-aji-dev

Copy link
Copy Markdown

Closes #31.

Problem

tests/unit/agents/flash/test_turn_index_snapshot.py fails 4 of its 5 tests without GOOGLE_API_KEY, contrary to the deterministic-suite contract in CONTRIBUTING.md:

$ uv run pytest -q tests/unit/agents/flash/test_turn_index_snapshot.py
4 failed, 1 passed
E   pydantic_core._pydantic_core.ValidationError: 1 validation error for ChatGoogleGenerativeAI
E     Value error, API key required for Gemini Developer API.

FlashRunner.__init__ constructs a VisualStepSummarizer, whose constructor resolves a real model client. The _runner helper already intends to discard it:

runner = FlashRunner(mock_context, goal="Open Wi-Fi and Display")
runner.summarizer = None   # never reached -- the line above raised

but that assignment runs after construction, so it never takes effect.

Change

Patch VisualStepSummarizer for the construction itself, in the with block that already patches get_driver:

with (
    patch("artemis.controllers.unified_controller.get_driver"),
    patch("artemis.agents.flash.runner.VisualStepSummarizer"),
):
    runner = FlashRunner(mock_context, goal="Open Wi-Fi and Display")
runner.summarizer = None

Patching the whole class is safe here precisely because the helper was already throwing the instance away: these tests assert index resolution across a turn and never summarize anything, so nothing under test is mocked out.

Test-only; no production code is touched.

Relationship to #41

#41 covers the same root cause in four other Flash modules and I've sent a fix for it separately (#86). That one introduces a stub_summarizer_model_factories fixture in tests/unit/agents/conftest.py, which would also cover this file once merged.

I kept this PR self-contained instead of depending on it, so the two can land in either order or independently. If #86 lands first and you'd rather this file used the shared fixture too, say so and I'll rebase it onto that.

Verification

Command main this branch
uv run pytest -q tests/unit/agents/flash/test_turn_index_snapshot.py 4 failed, 1 passed 5 passed
uv run ruff check tests/ — All checks passed
uv run ruff format --check on the touched file — already formatted

Run with GOOGLE_API_KEY, GEMINI_API_KEY and OPENAI_API_KEY all unset.

make typecheck could not be run in my environment — the pyright wheel downloads a Node runtime on first use and my sandbox blocks it. The touched file is not in pyright-core.json.

`tests/unit/agents/flash/test_turn_index_snapshot.py` failed 4 of its 5 tests
without GOOGLE_API_KEY. `FlashRunner.__init__` constructs a
`VisualStepSummarizer`, whose constructor resolves a real model client and
raises `ValidationError: API key required for Gemini Developer API`.

The `_runner` helper already discards the summarizer with
`runner.summarizer = None`, but that runs after construction has raised, so it
never takes effect. Patch the class for the construction itself.

These tests assert index resolution across a turn and never summarize
anything, so nothing they cover is mocked away.

Closes google#31.
@basil-k-aji-dev

Copy link
Copy Markdown
Author

Closing as a duplicate. #32 by @Fire162 already covers issue #31 and was opened first — that one should get the review, not this.

My mistake: I didn't check the open PR queue for an existing claim before sending this. Sorry for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_turn_index_snapshot fails offline when GOOGLE_API_KEY is not set

1 participant