diff --git a/docs/guide/infer.md b/docs/guide/infer.md index d1543d58..684cd778 100644 --- a/docs/guide/infer.md +++ b/docs/guide/infer.md @@ -117,6 +117,67 @@ Replace the placeholder above through your configuration/secret-management workflow; omit `api_key` only when the target endpoint does not require authentication. +### Google GenAI + +To call the native Gemini `generateContent` REST API, select the dialect +explicitly and provide the Google AI endpoint and API key: + +```yaml +models: + gemini: + name: gemini-3.8-flash + type: chat + dialect: google_genai + api_base: https://generativelanguage.googleapis.com + api_key: "" +``` + +The model name may be `gemini-...` or `models/gemini-...`. An endpoint root +uses `v1beta`; an explicit `/v1` or `/v1beta` suffix selects that version, and +gateway path prefixes are preserved. The key is sent only in the +`x-goog-api-key` header. Vertex AI resource paths and OAuth are not supported. + +Supported: text and image input, function tools, the `google_search` and +`url_context` hosted tools, JSON output, thinking controls, logprobs, +streaming, request seeds, and opaque continuation. Ordinary YAML bindings do +not yet supply Google model-profile outcomes, so model-dependent capabilities +(images, reasoning, tools, structured output, logprobs, opaque continuation) +currently fail reconciliation with `model_outcome_missing`; plain text +generation, streaming, seeds, and `dialect_options` work today. + +Rejected before any network I/O: + +- per-image `detail`, assistant prefill, input scoring, + `previous_response_id`, and `developer` messages (Google has no such role); +- on `v1`, function schemas using constructs outside Google's OpenAPI subset + (`additionalProperties`, `$ref`, type arrays, numeric enums) — use `/v1beta`; + on both versions, a function needs a non-empty description and an explicit + `type: object` parameter root; +- Gemini model-family limits, matched on the model name: on Gemini 3, + `sampling.n > 1`, explicit `temperature`/`top_p`/`top_k`, and + `reasoning.effort` values outside the model's + [thinking-level table](https://ai.google.dev/gemini-api/docs/generate-content/thinking); + on Gemini 2.5, `reasoning.effort` and out-of-range `reasoning.budget_tokens`; + on Gemini 1.x/2.x, mixing function tools with hosted tools. A gateway alias + that does not start with `gemini-` skips these checks. + +Provider-only root controls such as `safetySettings`, `serviceTier`, or `store` +go through `dialect_options` (`cachedContent` and `labels` are `v1beta`-only). +They cannot override `contents`, `generationConfig`, `tools`, `model`, or +`stream`, and there is no HTTP-header passthrough. + +Accounting: + +- `MAX_TOKENS` and `CONTINUATION` keep the partial text and count toward + `n_truncated`. The dialect does not replay `continuationToken`; raise + `max_tokens` instead. +- Candidate policy stops (`SAFETY`, `RECITATION`, …) count toward + `n_truncated`; a prompt blocked under `promptFeedback.blockReason` does not. + Both keep the raw reason, and `finish_reason_sources` records which it was. +- `input_tokens` is `promptTokenCount + toolUsePromptTokenCount` and + `output_tokens` is `candidatesTokenCount + thoughtsTokenCount`, so + `reasoning_tokens` stays a breakdown of `output_tokens`. + ## Environment Variables Custom environment variables can be passed through the YAML config's `infer.env` section. Values are injected into the inference engine process. diff --git a/docs/guide/metrics.md b/docs/guide/metrics.md index 391a607e..2b64cd17 100644 --- a/docs/guide/metrics.md +++ b/docs/guide/metrics.md @@ -509,12 +509,22 @@ at **every** budget, `n = 1` included, and by the single-draw tasks below that have no `n` at all. It measures the parser, not the draw, and `n = 1` is where a silently-stopped extractor survives longest: no second rollout to disagree with. -`n_truncated` — **rollouts** whose generation stopped because it ran out of -tokens rather than because the model was done — and `n_scored_rollouts`, the -rollouts scored in total, are reported by **every `gen` task**, on the same terms -as `n_unextracted`: they describe the generation rather than the draw, so neither -is confined to `n > 1`. Unlike every other key here the *runner* injects them, so -a task cannot report zero truncation by having forgotten to look. +`n_truncated` — **rollouts** whose output was interrupted before completion, +either by a token/context limit, output filtering, or refusal — and +`n_scored_rollouts`, the rollouts scored in total, are reported by **every `gen` +task**, on the same terms as `n_unextracted`: they describe the generation rather +than the draw, so neither is confined to `n > 1`. Unlike every other key here +the *runner* injects them, so a task cannot report zero truncation by having +forgotten to look. + +What counts: token/context-limit stops (`length`, `max_tokens`, +`max_output_tokens`, `model_context_window_exceeded`, Google `MAX_TOKENS` / +`CONTINUATION`), OpenAI `content_filter`, Anthropic `refusal`, and Google +candidate policy stops (`SAFETY`, `RECITATION`, `LANGUAGE`, …). A Google prompt +block (`promptFeedback.blockReason`) does not count — no candidate was +generated — and older Google records that lack `finish_reason_sources` cannot +tell the two apart, so their policy stops are left uncounted. A Chat +`message.refusal` without a `content_filter` finish reason is not counted. They arrive as a **pair or not at all**. The count alone says nothing about how much of a score it explains — `26` is a different fact at 600 rollouts than at 30 @@ -532,14 +542,11 @@ finals without per-stage metadata and a `0` would claim a clean run rather than unmeasured one. `sieval_versions` reports that state in band as `"unknown"`; a count has no such value. -Read `n_truncated` as a **bound on how much of a score is budget, not -capability** — a truncated rollout is scored wrong whether or not the model was -on its way to the right answer, and the fix is `max_tokens`. It is independent of -`n_unextracted`: an answer can be cut short and still parse (the truncation lands -after the boxed answer), or parse cleanly and be wrong for reasons of its own. -The two are worth reading together only in the direction where they coincide — a -rollout that is both truncated *and* unextracted is the case where raising the -budget is most likely to move the score. +Read `n_truncated` as a count of interrupted outputs, **not** a budget-only +diagnosis: raising `max_tokens` fixes only the token-limit stops, and the raw +`finish_reasons` say which kind each was. It is independent of `n_unextracted` — +an interrupted answer can still parse when the stop lands after the boxed +answer — so when both occur, check the stop reason before changing the budget. ### Pairs that must be read together diff --git a/sieval/core/models/__init__.py b/sieval/core/models/__init__.py index 023d6a7e..a1d8e3df 100644 --- a/sieval/core/models/__init__.py +++ b/sieval/core/models/__init__.py @@ -46,6 +46,8 @@ from .exceptions import CapabilityError from .gen_model import GenModel from .ir import ( + FINISH_SOURCE_CANDIDATE, + FINISH_SOURCE_PROMPT_BLOCK, CapabilityEvidence, ChatInput, ChatMessage, @@ -130,6 +132,8 @@ "DialectOptions", "DialectSpec", "Engine", + "FINISH_SOURCE_CANDIDATE", + "FINISH_SOURCE_PROMPT_BLOCK", "FunctionToolCall", "GenModel", "GroundingChunk", diff --git a/sieval/core/models/_legacy_bridge.py b/sieval/core/models/_legacy_bridge.py index e0d1dd3f..10cc4176 100644 --- a/sieval/core/models/_legacy_bridge.py +++ b/sieval/core/models/_legacy_bridge.py @@ -77,6 +77,7 @@ class ModelCallMeta(TypedDict): usage: NotRequired[ModelUsage] request_params: NotRequired[dict[str, JSONValue]] finish_reasons: NotRequired[list[str]] + finish_reason_sources: NotRequired[list[str]] response_model: NotRequired[str] system_fingerprint: NotRequired[str | None] @@ -101,6 +102,7 @@ class ModelOutput: request_params: dict[str, JSONValue] | None = None response_model: str | None = None system_fingerprint: str | None = None + finish_reason_sources: list[str] | None = None def _optional_float(value: object, name: str) -> float | None: @@ -475,6 +477,11 @@ def response_to_model_output(model_meta: ModelMeta, response: Response) -> Model if response.finish_reasons is not None else None ), + finish_reason_sources=( + list(response.finish_reason_sources) + if response.finish_reason_sources is not None + else None + ), reasoning_texts=reasoning_texts, tool_calls=response.tool_calls, logprobs_tokens=logprobs_tokens, diff --git a/sieval/core/models/dialect.py b/sieval/core/models/dialect.py index b849445a..b3b5f477 100644 --- a/sieval/core/models/dialect.py +++ b/sieval/core/models/dialect.py @@ -13,7 +13,7 @@ from copy import deepcopy from dataclasses import dataclass, fields from enum import StrEnum -from typing import Protocol, cast, runtime_checkable +from typing import TYPE_CHECKING, Protocol, cast, runtime_checkable from sieval.core.types import JSONValue @@ -35,6 +35,9 @@ response_field_contract, ) +if TYPE_CHECKING: + from _typeshed import DataclassInstance + class DialectError(ValueError): """Base class for deterministic dialect contract failures.""" @@ -74,7 +77,31 @@ class WireVerifier(Protocol): def verify(self, body: Mapping[str, JSONValue]) -> str | None: ... -type WireEvidence = WireObservation | WireVerifier +class PreparedRequestVerifier(Protocol): + """Independently verify request evidence outside the JSON body.""" + + def verify_prepared(self, prepared: "PreparedRequest") -> str | None: ... + + +@dataclass(frozen=True) +class PreparedRequestEvidence: + """Opt a verifier into checking the whole prepared request, not just the body. + + An explicit wrapper, so a ``WireVerifier`` never switches path by happening + to have a ``verify_prepared`` method. + """ + + verifier: PreparedRequestVerifier + + def __post_init__(self) -> None: + if not callable(getattr(self.verifier, "verify_prepared", None)): + raise TypeError( + "PreparedRequestEvidence.verifier must provide callable " + "verify_prepared()" + ) + + +type WireEvidence = WireObservation | WireVerifier | PreparedRequestEvidence @dataclass(frozen=True) @@ -220,7 +247,7 @@ def active_request_leaves(req: Request) -> dict[str, object]: """Return exact active semantic leaf paths for one concrete request.""" active = _input_leaves(req) - grouped = { + grouped: dict[str, DataclassInstance] = { "sampling": req.sampling, "scoring": req.scoring, "reasoning": req.reasoning, @@ -309,11 +336,14 @@ def consumed(self, path: str, *wire_evidence: WireEvidence) -> None: raise RequestAuditError( f"consumed request leaf {path!r} requires wire evidence" ) - if not all( - isinstance(evidence, (WireObservation, WireVerifier)) - for evidence in wire_evidence - ): - raise TypeError("wire evidence must be WireObservation or WireVerifier") + for evidence in wire_evidence: + if isinstance(evidence, WireObservation | PreparedRequestEvidence): + continue + if not callable(getattr(evidence, "verify", None)): + raise TypeError( + "wire evidence must be WireObservation, a verifier with " + "callable verify(), or PreparedRequestEvidence" + ) self._record(path, Consumed(path, tuple(wire_evidence))) def rejected(self, path: str, reason: str) -> None: @@ -344,7 +374,7 @@ def finish(self, prepared: PreparedRequest) -> None: "unaccounted request leaves: " + ", ".join(sorted(missing)) ) - verified: set[int] = set() + verified: set[tuple[str, int]] = set() for path, decision in self._decisions.items(): if isinstance(decision, Consumed): evidence_items = decision.wire_evidence @@ -378,20 +408,39 @@ def finish(self, prepared: PreparedRequest) -> None: ) continue - verifier_id = id(evidence) + verifier = ( + evidence.verifier + if isinstance(evidence, PreparedRequestEvidence) + else evidence + ) + verifier_id = ( + "prepared" + if isinstance(evidence, PreparedRequestEvidence) + else "body", + id(verifier), + ) if verifier_id in verified: continue verified.add(verifier_id) - mismatch = evidence.verify(prepared.body) + if isinstance(evidence, PreparedRequestEvidence): + mismatch = evidence.verifier.verify_prepared(prepared) + else: + verify = getattr(evidence, "verify", None) + if not callable(verify): + raise AssertionError( + "admitted wire verifier lost its verify() method" + ) + mismatch = verify(prepared.body) if mismatch is not None and ( not isinstance(mismatch, str) or not mismatch.strip() ): raise TypeError( - "WireVerifier.verify() must return None or a non-empty string" + "request evidence verifier must return None or a non-empty " + "string" ) if mismatch is not None: raise RequestAuditError( - f"prepared wire mismatch for {path!r}: {mismatch}" + f"prepared request evidence mismatch for {path!r}: {mismatch}" ) @@ -606,9 +655,21 @@ class OutputRule: @dataclass(frozen=True) class OutputContract: rules: Mapping[str, OutputRule] + empty_outcome_finish_reasons: frozenset[str] = frozenset({"refusal"}) def __post_init__(self) -> None: object.__setattr__(self, "rules", dict(self.rules)) + empty_outcome_finish_reasons = frozenset(self.empty_outcome_finish_reasons) + if not empty_outcome_finish_reasons or any( + not isinstance(reason, str) or not reason + for reason in empty_outcome_finish_reasons + ): + raise ValueError("empty-outcome finish reasons must be non-empty strings") + object.__setattr__( + self, + "empty_outcome_finish_reasons", + empty_outcome_finish_reasons, + ) channels = { name for name, (role, _) in response_field_contract().items() @@ -628,16 +689,18 @@ def validate(self, plan: RuntimePlanView, req: Request, response: Response) -> N # response channels this particular call must return. required = set(required_response_channels(req)) finish_reasons = response.finish_reasons - refused = ( + empty_outcome = ( finish_reasons is not None and bool(finish_reasons) - and all(reason == "refusal" for reason in finish_reasons) + and all( + reason in self.empty_outcome_finish_reasons for reason in finish_reasons + ) ) for name, rule in self.rules.items(): value = getattr(response, name) if rule.guarantee is Guarantee.NEVER and value is not None: raise OutputContractError(f"dialect promised {name!r} would be absent") - if name in required and not refused: + if name in required and not empty_outcome: if rule.guarantee is not Guarantee.PRESENT_OR_ERROR: raise OutputContractError( f"{name!r} is required but has {rule.guarantee.value} guarantee" @@ -649,7 +712,7 @@ def validate(self, plan: RuntimePlanView, req: Request, response: Response) -> N if value is not None: rule.validator(value) - if req.session.opaque_continuation is not None and not refused: + if req.session.opaque_continuation is not None and not empty_outcome: reasoning = response.reasoning if reasoning is None or any( item is None diff --git a/sieval/core/models/dialect_registry.py b/sieval/core/models/dialect_registry.py index 9340eb34..b57ef5d6 100644 --- a/sieval/core/models/dialect_registry.py +++ b/sieval/core/models/dialect_registry.py @@ -34,6 +34,10 @@ CAPABILITY_DECISIONS as ANTHROPIC_MESSAGES_CAPABILITY_DECISIONS, ) from .dialects.anthropic_messages import AnthropicMessagesDialect +from .dialects.google_genai import ( + CAPABILITY_DECISIONS as GOOGLE_GENAI_CAPABILITY_DECISIONS, +) +from .dialects.google_genai import GoogleGenAIDialect from .dialects.openai_chat import ( CAPABILITY_DECISIONS as OPENAI_CHAT_CAPABILITY_DECISIONS, ) @@ -329,6 +333,10 @@ def _bind_anthropic_messages(connection: Any, requested_model_id: str) -> Dialec return AnthropicMessagesDialect(connection, requested_model_id) +def _bind_google_genai(connection: Any, requested_model_id: str) -> Dialect: + return GoogleGenAIDialect(connection, requested_model_id) + + def _binder( function: Callable[[Any, str], Dialect], decisions: Mapping[str, DialectCapabilityDecision], @@ -370,6 +378,13 @@ def _binder( ANTHROPIC_MESSAGES_CAPABILITY_DECISIONS, ), ), + "google_genai": _binder( + _bind_google_genai, + cast( + Mapping[str, DialectCapabilityDecision], + GOOGLE_GENAI_CAPABILITY_DECISIONS, + ), + ), } ) @@ -418,7 +433,8 @@ def _registered_decisions( "google_genai": _spec( "google_genai", "async_http_json", - request_seed_support=RequestSeedSupport.RESERVED, + decisions=_registered_decisions("google_genai"), + request_seed_support=RequestSeedSupport.SUPPORTED, input_kinds=("chat",), input_modalities=("text", "image", "tool_call", "tool_result"), ), diff --git a/sieval/core/models/dialects/_google_genai_wire.py b/sieval/core/models/dialects/_google_genai_wire.py new file mode 100644 index 00000000..2c559171 --- /dev/null +++ b/sieval/core/models/dialects/_google_genai_wire.py @@ -0,0 +1,383 @@ +"""Versioned GenerateContent routes and function-schema representations. + +Contracts follow Google's v1/v1beta Discovery revision 20260928: +https://generativelanguage.googleapis.com/$discovery/rest?version=v1 +https://generativelanguage.googleapis.com/$discovery/rest?version=v1beta +No runtime discovery or schema-key dropping is used. v1beta accepts both +function schema forms; this adapter selects parametersJsonSchema to preserve +the shared IR's JSON Schema. v1 requires the narrower Google Schema form. +Lossless conversion preserves the supported schema's structure and constraints; +annotations are carried unchanged, not promised to affect model generation. +Google explicitly ignores default for validation. Its example documentation +describes root population, not an explicit rejection of nested input examples. + +AI-Generated Code - GPT-6 (OpenAI) +""" + +import math +import re +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from types import MappingProxyType +from typing import Literal, cast + +import httpx + +from sieval.core.models._shared import copy_json_value +from sieval.core.models.dialect import DialectError +from sieval.core.types import JSONValue + + +@dataclass(frozen=True) +class GoogleWireContract: + api_version: Literal["v1", "v1beta"] + function_schema_field: Literal["parameters", "parametersJsonSchema"] + unavailable_fields: Mapping[str, frozenset[str]] + + +_V1_UNAVAILABLE_FIELDS = MappingProxyType( + { + "request": frozenset({"cachedContent", "labels"}), + "generationConfig": frozenset( + { + "audioTranscriptionConfig", + "enableAffectiveDialog", + "speechConfig", + "translationConfig", + } + ), + "tool": frozenset({"computerUse", "mcpServers"}), + "functionDeclaration": frozenset( + {"behavior", "parametersJsonSchema", "responseJsonSchema"} + ), + "part": frozenset({"audioTranscription", "speechMetadata"}), + "functionResponse": frozenset({"parts", "scheduling", "willContinue"}), + "inlineData": frozenset({"displayName"}), + "fileData": frozenset({"displayName"}), + } +) +GOOGLE_WIRE_CONTRACTS: Mapping[str, GoogleWireContract] = MappingProxyType( + { + "v1": GoogleWireContract("v1", "parameters", _V1_UNAVAILABLE_FIELDS), + "v1beta": GoogleWireContract( + "v1beta", "parametersJsonSchema", MappingProxyType({}) + ), + } +) + + +def resolve_wire_endpoint(base_url: httpx.URL) -> tuple[GoogleWireContract, httpx.URL]: + """Resolve a root or versioned gateway URL without mutating its client.""" + + if base_url.query or base_url.fragment: + raise ValueError("Google api_base must not contain query data or fragments") + path = base_url.path.rstrip("/") + segments = path.split("/") + # Only the terminal segment selects the Google API version. Earlier + # version-like segments may belong to a gateway's own routing scheme. + if "models" in segments or ( + segments[-1] not in GOOGLE_WIRE_CONTRACTS + and re.match(r"v[0-9]", segments[-1], re.IGNORECASE) + ): + raise ValueError( + "Google api_base must be an endpoint root or end in /v1 or /v1beta" + ) + version = "v1beta" + if segments[-1] in GOOGLE_WIRE_CONTRACTS: + version = segments.pop() + path = "/".join(segments) + return GOOGLE_WIRE_CONTRACTS[version], base_url.copy_with(path=f"{path}/") + + +_SCHEMA_TYPES = frozenset( + {"string", "number", "integer", "boolean", "array", "object", "null"} +) +_SCHEMA_STRINGS = frozenset({"title", "description", "format", "pattern"}) +_SCHEMA_COUNTS = frozenset( + {"minItems", "maxItems", "minLength", "maxLength", "minProperties", "maxProperties"} +) +_SCHEMA_NUMBERS = frozenset({"minimum", "maximum"}) +_SCHEMA_STRING_LISTS = frozenset({"required", "propertyOrdering", "enum"}) +_SCHEMA_KEYS = ( + _SCHEMA_STRINGS + | _SCHEMA_COUNTS + | _SCHEMA_NUMBERS + | _SCHEMA_STRING_LISTS + | {"type", "properties", "items", "anyOf", "nullable", "default", "example"} +) +_INT64_MAX = 2**63 - 1 + + +def validate_function_parameters(raw: object) -> None: + """Require an explicit object root for function arguments, not results. + + This is SiEval's conservative admission rule for Google's object-schema + requirement. It does not infer the root type from refs or composition, nor + rewrite a schema to insert a type. Omitting parameters entirely is valid. + """ + + if ( + not isinstance(raw, Mapping) + or cast(Mapping[str, object], raw).get("type") != "object" + ): + raise DialectError( + "Google function parameters must describe an object with explicit " + "type: object" + ) + + +def _schema_object(raw: object, path: str) -> dict[str, JSONValue]: + if not isinstance(raw, Mapping): + raise DialectError(f"{path}: Google v1 Schema requires an object") + try: + copied = copy_json_value(raw, path) + except (TypeError, ValueError) as exc: + raise DialectError(f"{path}: invalid Google v1 Schema: {exc}") from exc + assert isinstance(copied, dict) + unsupported = set(copied) - _SCHEMA_KEYS + if unsupported: + raise DialectError( + f"{path}: Google v1 Schema cannot represent keyword(s) " + f"{', '.join(sorted(unsupported))}; use /v1beta for JSON Schema" + ) + return dict(copied) + + +def _check_scalar(schema: Mapping[str, JSONValue], key: str, path: str) -> None: + value = schema[key] + if key in _SCHEMA_STRINGS and not isinstance(value, str): + raise DialectError(f"{path}.{key}: Google v1 Schema requires a string") + if key == "nullable" and not isinstance(value, bool): + raise DialectError(f"{path}.{key}: Google v1 Schema requires a boolean") + if key in _SCHEMA_NUMBERS: + try: + exact = ( + isinstance(value, (int, float)) + and not isinstance(value, bool) + and math.isfinite(value) + and float(value) == value + ) + except OverflowError: + exact = False + if not exact: + raise DialectError( + f"{path}.{key}: Google v1 Schema requires an exactly representable " + "finite double" + ) + if key in _SCHEMA_STRING_LISTS: + if not isinstance(value, list) or any(not isinstance(x, str) for x in value): + raise DialectError(f"{path}.{key}: Google v1 Schema requires strings") + if len(value) != len(set(cast(list[str], value))): + raise DialectError( + f"{path}.{key}: Google v1 Schema requires unique strings" + ) + if key == "enum" and not value: + raise DialectError( + f"{path}.enum: Google v1 Schema requires a non-empty enum" + ) + + +def lower_function_schema( + raw: object, path: str = "tools.functions.parameters" +) -> dict[str, JSONValue]: + """Encode a lossless subset, requiring an explicit type on every node.""" + + source = _schema_object(raw, path) + type_name = source.get("type") + if not isinstance(type_name, str) or type_name not in _SCHEMA_TYPES: + raise DialectError( + f"{path}.type: Google v1 Schema requires a single explicit JSON Schema type" + ) + result: dict[str, JSONValue] = {"type": type_name.upper()} + for key, value in source.items(): + if key == "type": + continue + if key in _SCHEMA_COUNTS: + if ( + isinstance(value, bool) + or not isinstance(value, int) + or not 0 <= value <= _INT64_MAX + ): + raise DialectError( + f"{path}.{key}: Google v1 Schema requires a non-negative int64" + ) + result[key] = str(value) + elif key == "properties": + if not isinstance(value, Mapping): + raise DialectError( + f"{path}.properties: Google v1 Schema requires an object" + ) + property_schemas: dict[str, JSONValue] = {} + for name, child in value.items(): + assert isinstance(name, str) # _schema_object checked JSON keys. + property_schemas[name] = lower_function_schema( + child, f"{path}.properties.{name}" + ) + result[key] = property_schemas + elif key == "items": + result[key] = lower_function_schema(value, f"{path}.items") + elif key == "anyOf": + if not isinstance(value, list) or not value: + raise DialectError( + f"{path}.anyOf: Google v1 Schema requires a non-empty array" + ) + result[key] = [ + lower_function_schema(child, f"{path}.anyOf[{index}]") + for index, child in enumerate(value) + ] + else: + _check_scalar(source, key, path) + if key == "enum" and type_name != "string": + raise DialectError( + f"{path}.enum: Google v1 Schema only maps string enums" + ) + result[key] = value + return result + + +def inspect_function_schema( + raw: object, path: str = "final function parameters" +) -> dict[str, JSONValue]: + """Decode the actual v1 wire form independently of request lowering. + + The audit compares this result to the original JSON Schema. Calling the + encoder here would let an encoder mutation also rewrite its own oracle. + Only schema nodes are traversed; annotation values and property names are + opaque, even when they happen to contain keys such as ``type``. + """ + + wire = _schema_object(raw, path) + wire_type = wire.get("type") + if not isinstance(wire_type, str) or wire_type not in { + "STRING", + "NUMBER", + "INTEGER", + "BOOLEAN", + "ARRAY", + "OBJECT", + "NULL", + }: + raise DialectError( + f"{path}.type: Google v1 wire Schema requires an uppercase type" + ) + decoded: dict[str, JSONValue] = {} + for name, value in wire.items(): + if name == "type": + decoded[name] = wire_type.lower() + elif name in _SCHEMA_COUNTS: + if ( + not isinstance(value, str) + or re.fullmatch(r"0|[1-9][0-9]{0,18}", value) is None + ): + raise DialectError( + f"{path}.{name}: Google v1 wire Schema requires an int64 string" + ) + count = int(value) + if count > _INT64_MAX: + raise DialectError( + f"{path}.{name}: Google v1 wire Schema int64 is out of range" + ) + decoded[name] = count + elif name == "properties": + if not isinstance(value, Mapping): + raise DialectError( + f"{path}.properties: Google v1 wire Schema requires an object" + ) + properties: dict[str, JSONValue] = {} + for property_name, property_schema in value.items(): + assert isinstance( + property_name, str + ) # _schema_object checked JSON keys. + properties[property_name] = inspect_function_schema( + property_schema, f"{path}.properties.{property_name}" + ) + decoded[name] = properties + elif name == "items": + decoded[name] = inspect_function_schema(value, f"{path}.items") + elif name == "anyOf": + if not isinstance(value, list) or not value: + raise DialectError( + f"{path}.anyOf: Google v1 wire Schema requires a non-empty array" + ) + decoded[name] = [ + inspect_function_schema(child, f"{path}.anyOf[{index}]") + for index, child in enumerate(value) + ] + else: + _check_scalar(wire, name, path) + if name == "enum" and wire_type != "STRING": + raise DialectError( + f"{path}.enum: Google v1 wire Schema only maps string enums" + ) + decoded[name] = value + return decoded + + +def validate_wire_body( + body: Mapping[str, JSONValue], contract: GoogleWireContract +) -> None: + """Validate versioned fields and parameter roots at actual schema locations. + + Free-form JSON (function arguments, results, annotations, JSON Schema) + must not be searched by key name: a user may legitimately name a property + ``cachedContent`` or ``parametersJsonSchema``. + """ + + def check(raw: object, shape: str, path: str) -> None: + if isinstance(raw, Mapping): + forbidden = contract.unavailable_fields.get(shape, frozenset()) & raw.keys() + if forbidden: + raise DialectError( + f"{path}: field(s) {', '.join(sorted(forbidden))} are unavailable " + f"in Google {contract.api_version}" + ) + + def sequence(raw: object) -> Sequence[object]: + return ( + raw + if isinstance(raw, Sequence) and not isinstance(raw, str | bytes) + else () + ) + + check(body, "request", "request") + check(body.get("generationConfig"), "generationConfig", "generationConfig") + for index, tool in enumerate(sequence(body.get("tools"))): + check(tool, "tool", f"tools[{index}]") + if not isinstance(tool, Mapping): + continue + tool_mapping = cast(Mapping[str, JSONValue], tool) + for declaration in sequence(tool_mapping.get("functionDeclarations")): + check( + declaration, + "functionDeclaration", + f"tools[{index}].functionDeclarations", + ) + if not isinstance(declaration, Mapping): + continue + declaration_mapping = cast(Mapping[str, JSONValue], declaration) + for field in ("parameters", "response"): + if field in declaration and f"{field}JsonSchema" in declaration: + raise DialectError( + f"{field} and {field}JsonSchema are mutually exclusive" + ) + if field in declaration: + schema = inspect_function_schema( + declaration_mapping[field], f"function.{field}" + ) + if field == "parameters": + validate_function_parameters(schema) + if "parametersJsonSchema" in declaration: + validate_function_parameters( + declaration_mapping["parametersJsonSchema"] + ) + for content in (*sequence(body.get("contents")), body.get("systemInstruction")): + if not isinstance(content, Mapping): + continue + content_mapping = cast(Mapping[str, JSONValue], content) + for part in sequence(content_mapping.get("parts")): + if not isinstance(part, Mapping): + continue + check(part, "part", "contents.parts") + part_mapping = cast(Mapping[str, JSONValue], part) + for field in ("functionResponse", "inlineData", "fileData"): + check(part_mapping.get(field), field, f"contents.parts.{field}") diff --git a/sieval/core/models/dialects/google_genai.py b/sieval/core/models/dialects/google_genai.py new file mode 100644 index 00000000..0ac3571a --- /dev/null +++ b/sieval/core/models/dialects/google_genai.py @@ -0,0 +1,3821 @@ +"""Google Gemini native ``generateContent`` dialect for the model IR. + +The adapter speaks the public REST protocol through the provider-neutral +``async_http_json`` connection. Request leaves are translated only where the +shared IR has a lossless Google representation; provider-only controls remain +audited dialect options and ambiguous response shapes fail loudly. + +AI-Generated Code - GPT-5.6 (OpenAI) +""" + +import hashlib +import json +import math +import re +from collections.abc import Callable, Mapping, Sequence +from copy import deepcopy +from dataclasses import dataclass, field +from datetime import UTC, datetime +from types import MappingProxyType +from typing import cast +from urllib.parse import quote + +import httpx + +from sieval.core.models._shared import copy_json_value +from sieval.core.models.capabilities import ( + CAPABILITY_KEYS, + Capability, + CapabilityKey, + DialectCapabilityBinding, + DialectCapabilityDecision, + HostedToolsOptions, + MultimodalInputOptions, + ReasoningOptions, + Supported, + TopLogprobsOptions, + Unsupported, +) +from sieval.core.models.connection_factory import AsyncHTTPJSONConnection +from sieval.core.models.deployment import BINDING_RESOURCE_KEYS +from sieval.core.models.dialect import ( + DialectError, + Guarantee, + OutputContract, + OutputContractError, + OutputRule, + PreparedRequest, + PreparedRequestEvidence, + RequestAudit, + RuntimePlanView, + WireEvidence, + WireObservation, + active_request_leaves, + validate_reasoning, + validate_request_invariants, + validate_runtime_binding_plan, + validate_structured_output, + validate_tool_calls, + validate_top_logprobs, +) +from sieval.core.models.ir import ( + FINISH_SOURCE_CANDIDATE, + FINISH_SOURCE_PROMPT_BLOCK, + ChatInput, + ChatMessage, + Citation, + FunctionToolCall, + GroundingChunk, + GroundingMetadata, + ImagePart, + ReasoningOutput, + Request, + Response, + ServerToolUse, + StructuredOutput, + TextPart, + TokenLogprob, + ToolCallPart, + ToolResultPart, + TopKEntry, + UsageStats, +) +from sieval.core.types import JSONValue + +from ._google_genai_wire import ( + GoogleWireContract, + inspect_function_schema, + lower_function_schema, + resolve_wire_endpoint, + validate_function_parameters, + validate_wire_body, +) + +_THINKING_LEVELS = frozenset({"minimal", "low", "medium", "high"}) +_HOSTED_TOOL_KINDS = frozenset({"google_search", "url_context"}) +_IMAGE_MEDIA_TYPES = frozenset( + { + "image/avif", + "image/gif", + "image/heic", + "image/heif", + "image/jpeg", + "image/jpg", + "image/png", + "image/webp", + } +) +_OPAQUE_VERSION = 1 +_INT32_MIN = -(2**31) +_INT32_MAX = 2**31 - 1 +_CHAT_ROLES = frozenset({"system", "developer", "user", "assistant", "tool"}) +_FUNCTION_NAME_PATTERN = re.compile(r"[A-Za-z0-9_:.\-]{1,128}") +_RFC3339_TIMESTAMP_PATTERN = re.compile( + r"(?P\d{4}-\d{2}-\d{2})T" + r"(?P