Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
161 changes: 161 additions & 0 deletions docs/guide/infer.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,167 @@ Replace the placeholder above through your configuration/secret-management
workflow; omit `api_key` only when the target endpoint does not require
authentication.

### Google GenAI

To call the native Gemini `generateContent` REST API, select the dialect
explicitly and provide the Google AI endpoint and API key:

```yaml
models:
gemini:
name: gemini-3.8-flash
type: chat
dialect: google_genai
api_base: https://generativelanguage.googleapis.com
api_key: "<your Gemini API key>"
```

The model name may be either `gemini-...` or `models/gemini-...`. The dialect
supports the Google AI Gemini API's `v1` and `v1beta` versions. An endpoint root
defaults to `v1beta`; an explicit `/v1` or `/v1beta` suffix selects that version:

```yaml
# Stable API; use /v1beta explicitly for the beta version.
api_base: https://generativelanguage.googleapis.com/v1
```

Requests use `POST /<version>/models/<model>:generateContent`, or
`POST /<version>/models/<model>:streamGenerateContent?alt=sse` for streaming.
Gateway path prefixes are preserved. Unknown API versions and full resource
URLs are rejected at dialect construction before I/O. The final HTTP URL is
checked against the selected version and recorded in request evidence; explicit
`/v1` and `/v1beta` endpoints also have distinct runtime/provenance fingerprints.
Vertex AI resource paths and OAuth authentication are not mapped by this dialect.

Function parameter schemas have a version-specific wire contract, based on
Google's [v1 Discovery schema](https://generativelanguage.googleapis.com/$discovery/rest?version=v1)
and [v1beta Discovery schema](https://generativelanguage.googleapis.com/$discovery/rest?version=v1beta)
(revision `20260928`). `v1beta` permits both `parameters` and
`parametersJsonSchema`, mutually exclusively; SiEval selects
`parametersJsonSchema` to preserve the supplied JSON Schema. For either version,
function parameters must describe an object. SiEval requires an explicit root
`type: object`: scalar/array roots, type unions, and roots whose type is only
implied by `$ref` or composition are rejected before I/O. This is SiEval's
conservative supported subset, not a claim that Google rejects every referenced
object schema. A function with no parameters may omit `parameters`, or declare
`{"type": "object", "properties": {}}`.

`v1` exposes only `parameters`, using Google's OpenAPI `Schema` subset.
SiEval converts explicit
types, nested properties/items/`anyOf`, string enums, and supported scalar
constraints without discarding keywords. Unsupported constructs such as
`additionalProperties`, `$ref`/`$defs`, type arrays, and numeric enums are rejected
before I/O; every converted schema node must declare an explicit `type`.
Use `/v1beta` for other schema constructs, retaining the explicit object root.
The `v1` conversion encodes Schema types as uppercase enums and size limits as
int64 strings. Structured output
continues to use `generationConfig.responseJsonSchema` on both API versions.
Lossless conversion here means preserving supported structural and validation
information, not promising that Google uses every annotation during generation.
`default` and `example` are preserved unchanged; Google explicitly ignores
`default` for validation. Nested `example` is not rejected solely on the basis
of Google's root-population description. Fixed Discovery contract tests run
offline; neither evaluation nor tests fetch a changing schema at runtime.

The credential is sent only in the `x-goog-api-key` header. As with the
Anthropic example, YAML does not expand environment-variable expressions and
the provider-neutral HTTP connection does not infer a Google-specific
environment variable. Omit `api_key` only for a compatible endpoint that does
not require authentication.

The wire adapter implements text and image input, function tools,
`google_search` and `url_context` hosted tools, JSON output, thinking controls,
logprobs, streaming, deterministic request seeds, and opaque continuation.
Function declarations require a non-empty description and a Google-compatible
name of at most 128 characters. When hosted tools are combined with an explicit
`tools.choice: auto`, the dialect sends Google's `VALIDATED` mode because
server-side tool invocation circulation does not support `AUTO`.
Google documents combining custom functions with hosted tools as a Gemini 3
feature. Known Gemini 1.x/2.x model identifiers therefore reject this combination
before I/O, without disabling standalone function tools, Google Search, or URL
Context where supported. Unknown provider aliases do not acquire a model-family
constraint by name; model support still requires capability admission.
Historical function calls are validated turn by turn: every model turn that
emits calls must be followed immediately by one user/tool turn containing
exactly one matching `FunctionResponse` for each call. Matching the calls and
responses only across the conversation as a whole is rejected before I/O, as
required by Google's
[function-calling guidance](https://ai.google.dev/gemini-api/docs/generate-content/function-calling).
Gemini 3 responses must also provide Google's exact non-empty function-call ID,
which is replayed unchanged in the matching `FunctionResponse`; a missing ID is
an output-contract error. The first function call in each Gemini 3 step must
also preserve its provider-issued `thoughtSignature`; unsigned canonical tool
history is rejected for the current turn, while completed earlier turns remain
accepted. Older Gemini families retain a deterministic local fallback for
responses from endpoints that omit the formerly optional ID.

Gemini 3 tool-context circulation has an additional state contract: when
`includeServerSideToolInvocations` is enabled, each returned `toolCall` and
`toolResponse` must carry its own non-empty `thoughtSignature`. Missing signatures
fail response validation rather than producing incomplete replay state. These
parts and signatures must also survive opaque replay and final HTTP serialization,
even if a later request no longer enables hosted tools. This does not require
signatures on every text part or every parallel custom function call. See Google's
[tool-combination contract](https://ai.google.dev/gemini-api/docs/generate-content/tool-combination).

Hosted-tool configuration follows the Google Discovery schema. `url_context`
accepts only an empty object. `google_search` accepts `timeRangeFilter` and
`searchTypes`; time ranges require `startTime` and `endTime` together as
RFC3339 timestamps with explicit UTC offsets, and the start must not be after
the end. `webSearch` and `imageSearch` are empty marker objects. Unknown
configuration keys are rejected locally rather than being deferred to a
provider 400. The schema source is Google's public
[Generative Language Discovery document](https://generativelanguage.googleapis.com/$discovery/rest?version=v1beta).
Ordinary YAML bindings do not yet supply Google model-profile outcomes, so
model-dependent capabilities such as images, reasoning, tools, structured
output, logprobs, and opaque continuation fail reconciliation with
`model_outcome_missing` instead of being assumed to work on every Gemini model
or compatible endpoint. Plain text generation, streaming, seeds, and audited
provider options are currently reachable without those profile outcomes.

Per-image `detail`, assistant prefill, input scoring, and
`previous_response_id` are rejected before I/O. Google has no distinct
developer-message role, so a `developer` message is also rejected instead of
being silently weakened to a system instruction.

Gemini 3 rejects `candidateCount` greater than one, so the dialect rejects
`sampling.n > 1` for Gemini 3 model identifiers before network I/O. Earlier
Gemini families retain the provider's multiple-candidate behavior where no
singular response channel makes it ambiguous.

The dialect also applies Google's documented model-family request domains.
Gemini 2.5 models reject `reasoning.effort` and validate positive
`reasoning.budget_tokens` against the published range for Pro, Flash, or
Flash-Lite. Gemini 3 models validate `reasoning.effort` against the model's
published thinking-level table; for example, 3.1 Pro and 3.8 Flash reject
`minimal`, 3 Pro Preview accepts only `low` and `high`, and 3.1 Flash-Lite
Image accepts only `minimal` and `high`. Gemini 3 continues to accept
`reasoning.budget_tokens` for provider backward compatibility, although Google
recommends `reasoning.effort` / `thinkingLevel` for more predictable behavior.
Google's current Gemini 3 migration guidance says to remove `temperature`,
`top_p`, and `top_k`, so explicit values are rejected for Gemini 3 rather than
being recorded as effective sampling controls. These policies follow the
[Google thinking guide](https://ai.google.dev/gemini-api/docs/generate-content/thinking)
and the
[current-model migration guide](https://ai.google.dev/gemini-api/docs/latest-model).

`reasoning.summary: auto` requests visible thoughts with `includeThoughts`.
Google documents those summaries as optional when no thought is available, so
their absence is valid; returned `thoughtSignature` values and signed history
are preserved without rewriting provider parts in the opaque continuation
channel for later turns.
When `finishReason` is `MAX_TOKENS`, partial text is preserved, requested
channels may be absent, and incomplete JSON is not misreported as a malformed
structured response; downstream truncation accounting receives the provider's
finish reason unchanged.

Provider-only root controls can be supplied through matching
`dialect_options`, for example `safetySettings`, `serviceTier`, or `store`.
`cachedContent` and `labels` are available only on `v1beta` and are rejected on
`v1`. Options are audited against the final JSON request. Canonical fields
such as `contents`, `generationConfig`, `tools`, `model`, and `stream` cannot be
overridden there, and there is no arbitrary HTTP-header passthrough.

## Environment Variables

Custom environment variables can be passed through the YAML config's `infer.env` section. Values are injected into the inference engine process.
101 changes: 82 additions & 19 deletions sieval/core/models/dialect.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
from copy import deepcopy
from dataclasses import dataclass, fields
from enum import StrEnum
from typing import Protocol, cast, runtime_checkable
from typing import Any, Protocol, cast, runtime_checkable

from sieval.core.types import JSONValue

Expand Down Expand Up @@ -74,7 +74,33 @@ class WireVerifier(Protocol):
def verify(self, body: Mapping[str, JSONValue]) -> str | None: ...


type WireEvidence = WireObservation | WireVerifier
class PreparedRequestVerifier(Protocol):
"""Independently verify request evidence outside the JSON body."""

def verify_prepared(self, prepared: "PreparedRequest") -> str | None: ...


@dataclass(frozen=True)
class PreparedRequestEvidence:
"""Explicitly select verification against the complete prepared request.

A wrapper keeps structural ``WireVerifier`` objects on their historical
body-only path even when they happen to expose another similarly named
method. It also validates the callable at evidence-admission time rather
than failing later with an unhelpful ``TypeError``.
"""

verifier: PreparedRequestVerifier

def __post_init__(self) -> None:
if not callable(getattr(self.verifier, "verify_prepared", None)):
raise TypeError(
"PreparedRequestEvidence.verifier must provide callable "
"verify_prepared()"
)


type WireEvidence = WireObservation | WireVerifier | PreparedRequestEvidence


@dataclass(frozen=True)
Expand Down Expand Up @@ -231,7 +257,7 @@ def active_request_leaves(req: Request) -> dict[str, object]:
}
for group_name, group in grouped.items():
default = type(group)()
for item in fields(group):
for item in fields(cast(Any, group)):
value = getattr(group, item.name)
if value != getattr(default, item.name):
active[f"{group_name}.{item.name}"] = value
Expand Down Expand Up @@ -309,11 +335,14 @@ def consumed(self, path: str, *wire_evidence: WireEvidence) -> None:
raise RequestAuditError(
f"consumed request leaf {path!r} requires wire evidence"
)
if not all(
isinstance(evidence, (WireObservation, WireVerifier))
for evidence in wire_evidence
):
raise TypeError("wire evidence must be WireObservation or WireVerifier")
for evidence in wire_evidence:
if isinstance(evidence, WireObservation | PreparedRequestEvidence):
continue
if not callable(getattr(evidence, "verify", None)):
raise TypeError(
"wire evidence must be WireObservation, a verifier with "
"callable verify(), or PreparedRequestEvidence"
)
self._record(path, Consumed(path, tuple(wire_evidence)))

def rejected(self, path: str, reason: str) -> None:
Expand Down Expand Up @@ -344,7 +373,7 @@ def finish(self, prepared: PreparedRequest) -> None:
"unaccounted request leaves: " + ", ".join(sorted(missing))
)

verified: set[int] = set()
verified: set[tuple[str, int]] = set()
for path, decision in self._decisions.items():
if isinstance(decision, Consumed):
evidence_items = decision.wire_evidence
Expand Down Expand Up @@ -378,20 +407,39 @@ def finish(self, prepared: PreparedRequest) -> None:
)
continue

verifier_id = id(evidence)
verifier = (
evidence.verifier
if isinstance(evidence, PreparedRequestEvidence)
else evidence
)
verifier_id = (
"prepared"
if isinstance(evidence, PreparedRequestEvidence)
else "body",
id(verifier),
)
if verifier_id in verified:
continue
verified.add(verifier_id)
mismatch = evidence.verify(prepared.body)
if isinstance(evidence, PreparedRequestEvidence):
mismatch = evidence.verifier.verify_prepared(prepared)
else:
verify = getattr(evidence, "verify", None)
if not callable(verify):
raise AssertionError(
"admitted wire verifier lost its verify() method"
)
mismatch = verify(prepared.body)
if mismatch is not None and (
not isinstance(mismatch, str) or not mismatch.strip()
):
raise TypeError(
"WireVerifier.verify() must return None or a non-empty string"
"request evidence verifier must return None or a non-empty "
"string"
)
if mismatch is not None:
raise RequestAuditError(
f"prepared wire mismatch for {path!r}: {mismatch}"
f"prepared request evidence mismatch for {path!r}: {mismatch}"
)


Expand Down Expand Up @@ -463,10 +511,11 @@ def validate_runtime_binding_plan(plan: RuntimePlanView, req: Request) -> None:
)

minimum = plan.capability_minimums.get("top_logprobs", {}).get("minimum")
minimum_value = int(cast(int, minimum)) if minimum is not None else None
if (
minimum is not None
minimum_value is not None
and req.scoring.top_logprobs > 0
and req.scoring.top_logprobs < int(minimum)
and req.scoring.top_logprobs < minimum_value
):
raise DialectError(
f"top_logprobs={req.scoring.top_logprobs} weakens required "
Expand Down Expand Up @@ -606,9 +655,21 @@ class OutputRule:
@dataclass(frozen=True)
class OutputContract:
rules: Mapping[str, OutputRule]
empty_outcome_finish_reasons: frozenset[str] = frozenset({"refusal"})

def __post_init__(self) -> None:
object.__setattr__(self, "rules", dict(self.rules))
empty_outcome_finish_reasons = frozenset(self.empty_outcome_finish_reasons)
if not empty_outcome_finish_reasons or any(
not isinstance(reason, str) or not reason
for reason in empty_outcome_finish_reasons
):
raise ValueError("empty-outcome finish reasons must be non-empty strings")
object.__setattr__(
self,
"empty_outcome_finish_reasons",
empty_outcome_finish_reasons,
)
channels = {
name
for name, (role, _) in response_field_contract().items()
Expand All @@ -628,16 +689,18 @@ def validate(self, plan: RuntimePlanView, req: Request, response: Response) -> N
# response channels this particular call must return.
required = set(required_response_channels(req))
finish_reasons = response.finish_reasons
refused = (
empty_outcome = (
finish_reasons is not None
and bool(finish_reasons)
and all(reason == "refusal" for reason in finish_reasons)
and all(
reason in self.empty_outcome_finish_reasons for reason in finish_reasons
)
)
for name, rule in self.rules.items():
value = getattr(response, name)
if rule.guarantee is Guarantee.NEVER and value is not None:
raise OutputContractError(f"dialect promised {name!r} would be absent")
if name in required and not refused:
if name in required and not empty_outcome:
if rule.guarantee is not Guarantee.PRESENT_OR_ERROR:
raise OutputContractError(
f"{name!r} is required but has {rule.guarantee.value} guarantee"
Expand All @@ -649,7 +712,7 @@ def validate(self, plan: RuntimePlanView, req: Request, response: Response) -> N
if value is not None:
rule.validator(value)

if req.session.opaque_continuation is not None and not refused:
if req.session.opaque_continuation is not None and not empty_outcome:
reasoning = response.reasoning
if reasoning is None or any(
item is None
Expand Down
Loading
Loading