Skip to content

fix(tests): isolate Flash model clients and MCP spawn watchdogs - #42

Open
wellorbetter wants to merge 3 commits into
google:mainfrom
wellorbetter:fix/flash-offline-tests
Open

wellorbetter wants to merge 3 commits into
google:mainfrom
wellorbetter:fix/flash-offline-tests

Conversation

@wellorbetter

@wellorbetter wellorbetter commented Sep 11, 2026 •

Copy link
Copy Markdown

Summary

Several deterministic tests depend on real model clients or local runtime state despite the credential-free test contract in CONTRIBUTING.md. This fixes test isolation in four Flash modules and one MCP module, without changing production behavior.

Fixes #41 and #43.

  • Patch both model factories at the summarizer module boundary before construction. Keep the real runner, summarizer, retry, ledger, and compression implementations and their assertions. The default async model stub raises on an unconfigured invocation.
  • Use the existing validate_profiles=False builder option in the configuration-only test.
  • Mock the spawn-watchdog launcher for fake MCP subprocesses and assert its dispatch arguments. Previously, four tests left real watchdog threads running after fixture teardown, causing failure notifications for simulated tasks. Dedicated watchdog behavior tests remain active with external effects mocked.
  • Create a temporary SQLite database for the invalid-action inspection test, so it does not depend on a database left by earlier local task runs.

The turn-index snapshot file addressed by #32/#35 is unchanged; this covers additional test modules.

Reproduction and before/after evidence

Environment: Windows, Python 3.12.14, committed lockfile, no Google/Gemini/OpenAI API keys. Baseline: 0860788. Latest PR head: cfcfc8d.

uv run pytest -q tests/unit/agents/test_flash_runner.py tests/unit/agents/test_flash_runner_ledger.py tests/unit/agents/test_flash_scrub_edge.py tests/unit/agents/test_flash_step_summarizer.py tests/unit/mcp/test_mcp_tools.py tests/unit/mcp/test_spawn_watchdog.py
Four Flash modules:
Before: 64 failed, 13 passed
After:  77 passed

MCP watchdog isolation guard:
Before: 25 passed, 4 teardown errors (real watchdog thread-start attempts)
After:  25 passed, no real watchdog thread-start attempts
Dedicated watchdog behavior tests: 4 passed

All six target modules in a fresh worktree:
Before the database fixture fix: 1 failed, 105 passed
Latest PR:                      106 passed in 2.18s

Full deterministic suite (pytest -q):
Upstream: 82 failed, 2109 passed, 4 skipped, 8 deselected
Latest PR: 18 failed, 2173 passed, 4 skipped, 8 deselected
Latest full run: 91.00s; 64 baseline failures resolved, no new failed nodes

The temporary watchdog guard intercepted thread starts without executing them, avoiding real notifications or process termination during reproduction. The Flash modules also passed with shuffled execution order (seed 42) and a guard forbidding real provider-client construction.

#41 and #43 contain the root-cause analysis and representative failure excerpts. Commands above use the portable uv run form; local execution used the locked virtual environment's Python executable in isolated worktrees.

Contribution checks and limitations

Executed the Windows equivalents of the contribution checks:

No tests are skipped or removed by this change. During extended testing, an unchanged device FIFO reservation test failed intermittently and could leave pytest running; the same behavior was reproduced on clean upstream. It did not fail in the latest full run and remains an unresolved validation limitation.

Private screenshots, UI XML, and raw device traces are not included. Public evidence consists of sanitized text excerpts.

@wellorbetter wellorbetter changed the title fix(tests): isolate Flash model construction from credentials fix(tests): isolate Flash model clients and MCP spawn watchdogs Sep 11, 2026
@Fire162

Fire162 commented Sep 11, 2026

Copy link
Copy Markdown

the mods are lazy or whatever they are not accepting pr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flash unit tests require Gemini credentials before installing their model mocks

2 participants