Skip to content

Honor the configured provider when a node pins its own model - #151

Open
basil-k-aji-dev wants to merge 1 commit into
google:mainfrom
basil-k-aji-dev:fix/flash-honors-configured-provider
Open

basil-k-aji-dev wants to merge 1 commit into
google:mainfrom
basil-k-aji-dev:fix/flash-honors-configured-provider

Conversation

@basil-k-aji-dev

Copy link
Copy Markdown

Closes #138

Why

get_google_llm hard-codes ModelProvider.GOOGLE. Any caller that pins a specific lightweight model therefore ignores the configured provider, and an OpenAI-compatible deployment gets a Gemini client built with an empty key:

1 validation error for ChatGoogleGenerativeAI
Value error, API key required for Gemini Developer API.
input_value={'model': '<configured-model>', 'google_api_key': ''}

Two call sites do this, both reached on a normal Flash run:

  • agents/flash/summarizer.py — VisualStepSummarizer is built eagerly at runner init when agent.flash.step_summarizer.enabled is on, which is the default. That is why the run dies before the first step rather than at the first summary.
  • memory/chunking.py — StepCapsuleLens._get_llm and _get_fallback_llm, pinning chunking.model.

What

get_node_model(ctx, name, *, model_name, is_utils, temperature) in services/llm.py. It resolves the endpoint through _resolve_endpoint, the same path get_llm already uses, and overrides only model_name. The provider, and anything else the operator configured, survive.

Two things it deliberately does not do:

  • It returns a raw model, not a RobustChatModelWrapper. VisualStepSummarizer._meter_lens_call skips metering when the model is already wrapped, so returning a wrapped model here would have double-counted every lens call. Matching what get_google_llm returned keeps that guard correct.
  • With no ctx it still builds Google. There is no configuration to read and therefore no provider to honour; StepCapsuleLens is constructed with ctx=None in places and that path is unchanged.

What I left alone

agents/flash/runner.py:287 also falls back to get_google_llm("gemini-2.5-flash"), as the issue notes. I did not touch it: it only runs after get_llm(ctx, "operator") has already raised, so resolving a provider from the same configuration would be circular. The issue argues that path should fail with a clear message instead of building a Gemini client with an empty key, and I agree, but that is a behaviour change on an error path and it is your call rather than mine. Happy to add it here.

The Pro-profile judge nodes the issue mentions (planner_validation, validator_pixel_safety_net via lightweight_judge_default()) are the same class of problem but a different shape, since the default+nodes config format has no way to override them. Worth its own issue.

Testing

tests/unit/test_llm_node_model.py, five cases. The two that matter assert at the real call site rather than on the new helper, because a test that only imported get_node_model would fail on an unpatched tree with ImportError, which proves the function is new and nothing else.

Against the unpatched tree, with the test pointed at get_google_llm:

AssertionError: a pinned compression model must not force the Gemini client
assert <ModelProvider.GOOGLE: 'google'> is <ModelProvider.OPENAI: 'openai'>
5 failed

With the change:

5 passed

make lint and make typecheck are clean:

ruff format --check    4 files already formatted
ruff check             All checks passed!
pyright                0 errors, 0 warnings, 0 informations

The suite, honestly

tests/unit/memory and tests/unit/agents report 80 failed, 535 passed, 4 skipped on this machine, and the identical 80 named tests fail on a clean tree with my three files stashed. I compared the sorted failure lists rather than the totals, because equal counts would also be the signature of breaking one test while an unrelated one happened to pass.

They are environmental, not the change:

artemis/config/llm.py:91: in validate_provider
    raise Exception(f"{name} requires GOOGLE_API_KEY in .env")
E   Exception: Planner requires GOOGLE_API_KEY in .env

I have no GOOGLE_API_KEY here, so every test that validates providers fails at config time. That also means I could not run the reporter's end-to-end Flash task; the verification above is the targeted unit tests plus the reporter's own confirmation that the same patch shape completed a Flash run against a custom endpoint with usage recorded as openai:<model>.

get_google_llm hard-codes ModelProvider.GOOGLE, so the Flash step summarizer
and the chunk capsule lens sent an OpenAI-compatible deployment to the Gemini
client and failed on an empty GOOGLE_API_KEY. The summarizer is built eagerly
at runner init, so the run died before its first step.

get_node_model resolves the endpoint the same way get_llm does and overrides
only the model name, leaving the operator's provider intact.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flash profile fails with OpenAI-compatible (non-Gemini) providers: step summarizer & memory chunker hard-code the Gemini API

1 participant