Skip to content

feat(models): add native Google GenAI generateContent dialect - #147

Open
jack-scitix-ai wants to merge 11 commits into
scitix:mainfrom
jack-scitix-ai:feat/google-genai
Open

jack-scitix-ai wants to merge 11 commits into
scitix:mainfrom
jack-scitix-ai:feat/google-genai

Conversation

@jack-scitix-ai

Copy link
Copy Markdown
Contributor

Type

  • feature — new benchmark, task, or capability

Summary

  • Add an active google_genai dialect for the Gemini Developer API's native generateContent protocol, with audited request lowering and strict streaming/non-streaming response lifting.
  • Support explicit v1 and v1beta wire contracts, including version-specific operations, function schemas, final-request validation, and distinct runtime provenance.
  • Support text and image input, structured output, thinking controls, logprobs, request seeds, function tools, Google Search, URL Context, and opaque continuation.
  • Enforce model-family constraints and reject unsupported combinations before I/O, including Gemini 2.5 mixed tools and Gemini 3 thinking, sampling, candidate-count, call-ID, and signature requirements.
  • Preserve provider-issued tool history and signatures across continuation replay, and verify the final PreparedRequest, including an exact audit of requested hosted tools.
  • Recognize both MAX_TOKENS and CONTINUATION as truncation while preserving partial output and provider-verbatim finish reasons.
  • Keep model-dependent capabilities gated on model-profile outcomes; Interactions API and Vertex AI / Agent Platform authentication and resource paths remain outside this dialect.

Related Issues

Refs #25

Test Plan

Automated

  • Lint/format clean (pdm run ruff check && pdm run ruff format --check)
  • Type check clean (pdm run ty check sieval/)
  • Unit and integration tests pass (pdm run python -m pytest tests/unit tests/integration -q): 8420 passed, 84 skipped

Checklist

Required (all PRs)

  • PR title follows conventional format (type(scope): description)
  • No internal paths, credentials, or personal info in committed files
  • AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
  • No new upper-layer dependencies added to core/
  • Deleted code verified — no remaining call sites depend on it

Allow dialect evidence to verify the final PreparedRequest rather than only JSON body fields. This lets operation-selected protocols prove stream and endpoint control flow without inventing a body parameter.
Keep Google generateContent finish reasons provider-verbatim while ensuring MAX_TOKENS contributes to both anomaly detection and truncated-rollout reporting.
Bind the native generateContent protocol with audited request lowering, strict streaming and non-streaming lifting, capability declarations, deterministic seed support, and provider-native continuation evidence.
Reject lossy chat and stream shapes before they can be reinterpreted, preserve terminal usage and citation evidence, and make multi-candidate and continuation behavior explicit.

Snapshot the request used for response lifting and strengthen regression tests around final-wire provenance, stable call identities, replay, and required output channels.
Surface prompt blocking reasons, preserve compatible streamed text deltas, and validate hosted-tool call state in opaque continuation history. Add fail-loud regression coverage for contradictory metadata, HTTP error status handling, and malformed or forged provider responses.
Treat each streamed candidate finishReason as terminal while continuing to allow response-level usage-only tail events. Add regression coverage and mutation verification for the terminal-state guard.
Add discriminating coverage for request rejection, opaque replay, malformed provider responses, streaming state, wire verifiers, prepared-request integrity, and final request evidence.\n\nExercise both high-level I/O boundaries and lower-level verifier contracts so guard removal or mutation fails loudly.

@ethan-scitix ethan-scitix left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes. The design is solid and the tests are strong (unit + integration pass; ruff and ty are clean). Two accounting bugs put wrong numbers in reports without raising, and coverage is below the core/ gate.

Blocking

  1. Bug: wrong usage semantics (google_genai.py:2039). candidatesTokenCount excludes thoughts (totalTokenCount = prompt + toolUsePrompt + thoughts + candidates). But UsageStats.reasoning_tokens must be a breakdown of output_tokens. With prompt=20, candidates=150, thoughts=3000, the profiler reports a 2000% reasoning share, and every thinking call trips the total-mismatch counter. Fix: output_tokens = candidates + thoughts and input_tokens = prompt + toolUsePrompt.
  2. Bug: Google filter stops aren't counted as truncation (meta.py:24). The set includes content_filter, but not Google's equivalents: SAFETY, RECITATION, BLOCKLIST, PROHIBITED_CONTENT, SPII, IMAGE_SAFETY, … So a filtered Gemini rollout reads n_truncated = 0. Add them, or document why they're left out.
  3. Coverage: google_genai.py is at 90%, below the 95% required for core/. Most uncovered lines are the fail-loud raise branches (e.g. :878–942, :1383–1447, :1743–1765).

Non-blocking

  • Question: CONTINUATION is counted as truncation and continuationToken is silently dropped. That's fine for scoring, but please note it in infer.md and add a comment in the code.
  • Nit: cast(Any, group) and int(cast(int, minimum)) in dialect.py aren't needed; ty passes without them. The AssertionError at dialect.py:~429 is unreachable.
  • Nit: the body of commit 6f81a283 contains a literal \n\n.
  • Follow-up: model-family limits are matched by regex on the model name, so gateway aliases skip them. Consider moving them into model profiles, and splitting the 3.8k-line module.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants