Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions docs/guide/infer.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,40 @@ used, not whether the service starts.

Models with a `path` field (and no `api_base`) or an `infer` section in the YAML config are automatically launched by `sieval run` and stopped after evaluation completes.

### Anthropic Messages

To call the native Anthropic Messages API, select the dialect explicitly and
provide an external endpoint and credential:

```yaml
models:
claude:
name: claude-sonnet-4-5
dialect: anthropic_messages
api_base: https://api.anthropic.com/v1
api_key: "<your Anthropic API key>"
args:
max_tokens: 1024
```

The dialect sends JSON or SSE requests to `POST /v1/messages`. Every request
must specify `max_tokens`; set a model-wide default under `models.<name>.args`,
as above, or override it for one task under `tasks.<name>.infer_args`.

Anthropic Messages has no per-request `seed`. Deterministic mode therefore does
not inject one for this dialect, and an explicit `seed` is rejected before any
network I/O.

Raw request passthrough is intentionally unavailable. `dialect_options` cannot
inject arbitrary Anthropic body fields or HTTP headers; in particular, there is
currently no way to send the `anthropic-beta` header. Features that require a
beta header are unsupported until they receive an explicit, audited mapping.

The YAML loader does not expand environment-variable expressions in `api_key`.
Replace the placeholder above through your configuration/secret-management
workflow; omit `api_key` only when the target endpoint does not require
authentication.

## Environment Variables

Custom environment variables can be passed through the YAML config's `infer.env` section. Values are injected into the inference engine process.
10 changes: 8 additions & 2 deletions sieval/core/models/dialect.py
Original file line number Diff line number Diff line change
Expand Up @@ -627,11 +627,17 @@ def validate(self, plan: RuntimePlanView, req: Request, response: Response) -> N
# must be able to serve. Only the projected request decides which
# response channels this particular call must return.
required = set(required_response_channels(req))
finish_reasons = response.finish_reasons
refused = (
finish_reasons is not None
and bool(finish_reasons)
and all(reason == "refusal" for reason in finish_reasons)
)
for name, rule in self.rules.items():
value = getattr(response, name)
if rule.guarantee is Guarantee.NEVER and value is not None:
raise OutputContractError(f"dialect promised {name!r} would be absent")
if name in required:
if name in required and not refused:
if rule.guarantee is not Guarantee.PRESENT_OR_ERROR:
raise OutputContractError(
f"{name!r} is required but has {rule.guarantee.value} guarantee"
Expand All @@ -643,7 +649,7 @@ def validate(self, plan: RuntimePlanView, req: Request, response: Response) -> N
if value is not None:
rule.validator(value)

if req.session.opaque_continuation is not None:
if req.session.opaque_continuation is not None and not refused:
reasoning = response.reasoning
if reasoning is None or any(
item is None
Expand Down
18 changes: 17 additions & 1 deletion sieval/core/models/dialect_registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@
from .connection_factory import CONNECTION_FACTORY_REGISTRY
from .deployment import ConnectionPool, Deployment, RouteIntent, resolve_route
from .dialect import Dialect, active_request_leaves
from .dialects.anthropic_messages import (
CAPABILITY_DECISIONS as ANTHROPIC_MESSAGES_CAPABILITY_DECISIONS,
)
from .dialects.anthropic_messages import AnthropicMessagesDialect
from .dialects.openai_chat import (
CAPABILITY_DECISIONS as OPENAI_CHAT_CAPABILITY_DECISIONS,
)
Expand Down Expand Up @@ -321,6 +325,10 @@ def _bind_openai_responses(connection: Any, requested_model_id: str) -> Dialect:
return OpenAIResponsesDialect(connection, requested_model_id)


def _bind_anthropic_messages(connection: Any, requested_model_id: str) -> Dialect:
return AnthropicMessagesDialect(connection, requested_model_id)


def _binder(
function: Callable[[Any, str], Dialect],
decisions: Mapping[str, DialectCapabilityDecision],
Expand Down Expand Up @@ -355,6 +363,13 @@ def _binder(
OPENAI_RESPONSES_CAPABILITY_DECISIONS,
),
),
"anthropic_messages": _binder(
_bind_anthropic_messages,
cast(
Mapping[str, DialectCapabilityDecision],
ANTHROPIC_MESSAGES_CAPABILITY_DECISIONS,
),
),
}
)

Expand Down Expand Up @@ -395,7 +410,8 @@ def _registered_decisions(
"anthropic_messages": _spec(
"anthropic_messages",
"async_http_json",
request_seed_support=RequestSeedSupport.RESERVED,
decisions=_registered_decisions("anthropic_messages"),
request_seed_support=RequestSeedSupport.UNSUPPORTED,
input_kinds=("chat",),
input_modalities=("text", "image", "tool_call", "tool_result"),
),
Expand Down
Loading
Loading