Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 45 additions & 1 deletion packages/agentenv-protocol/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1388,6 +1388,39 @@ Extensions come from explicit definitions, configured activations, and
decorated handlers in one validated registry, so routes and card advertisement
cannot drift.

`Usage` reports what a task used: `tool_call_count`, `input_tokens`,
`output_tokens`, `total_tokens`, `cost_usd`, the prompt-cache counts
`cache_read_tokens` and `cache_write_tokens`, and `input_includes_cache`, which
says whether `input_tokens` counts both cache counts (`True`), neither
(`False`), or is unknown (`None`, the default). Set the flag whenever you set
the cache counts: a price can't read them without it. Runtime-specific counts go
in `provider_details`, but cache counts go in their own fields. `Usage.from_dict`
reads what `to_dict` wrote under any protocol version, a field it leaves out
being `None` and a key it doesn't know ignored, and `TaskResult.failure(...,
usage=...)` carries usage on a failure as `success` does.

Whoever makes a model call knows what its counts hold, so an agent builds its own
`Usage`. `Usage.from_model_response(usage)` is the default reader for a model
response's own usage, with `input_tokens` counting the cache
(`input_includes_cache` true), for OpenAI's Chat Completions (`prompt_tokens`)
and Responses (`input_tokens`) shapes and Anthropic's:

- An OpenAI count, or a LiteLLM client's, counts the cache its
`prompt_tokens_details` or `input_tokens_details` names: `cached_tokens`, and
`cache_write_tokens` beside it.
- Anthropic's `input_tokens` counts neither `cache_read_input_tokens` nor
`cache_creation_input_tokens`, and neither does the `prompt_tokens` a LiteLLM
proxy sends with those two beside it, so they're added, to `total_tokens` too.
- A provider's split of its cache writes by how long they're kept, priced apart
(Anthropic's `cache_creation`, OpenAI's `cache_write_tokens_details`), is kept
in `provider_details`, and so are the keys of a usage in none of these shapes,
as `unrecognized`. A value that isn't a count is left out.

Through a gateway, whether a response's input counts the cache depends on the
model behind it: a LiteLLM proxy's Claude answers through Chat Completions leave
it out, its GPT answers don't. Set `input_includes_cache` per call, never once
per agent.

`run(request)` is the required execution contract. It is an ordinary method
and needs no decorator.

Expand Down Expand Up @@ -1866,7 +1899,18 @@ operations:
records a call, each action must be one the last recorded call made, by name and
arguments (a `bash` call by its command alone, a whole float equal to its int), and all of
them must run before the next `record` or the `reply`, else 409.
Calls to tools the turn doesn't offer aren't checked.
Calls to tools the turn doesn't offer aren't checked. `usage` is the call's: a `Usage` the
driver reports under `REPORTED_USAGE_KEY` (`{"agentenv_usage": usage.to_dict()}`), taken as
given, or else the model response's own, read by `Usage.from_model_response`. It rides in
`usage` rather than a field of its own, so a proxy on an earlier protocol still takes the
record. A reported `Usage` that isn't one gets 400 and leaves the turn as it was. The call's
input is made to count the cache, its `chat` span shows that reading, and its turn sums it.
The turn's `Usage`, on its reply or its error, sums each count over the calls that gave it,
`cost_usd` included. Its `provider_details` give `model_calls`, how many were recorded;
`unreported`, how many left out each count only some gave, so such a sum reads as a lower
bound; `cache_unsettled`, how many had cache counts with no word on whether their input held
them, which leaves the turn's flag unknown; `unrecognized`, how many usages were in no shape
the reader knows; and each provider's cache-write split, summed.
- `reply` (`{"token", "task_id", "parts"}`, or `"error"` in place of `"parts"`) ends the
turn: parts complete the task, and an error fails it as `proxy.driver_error`.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
INSTALL_V1,
MCP_CONFIG_V1,
PEER_AGENTS_V1,
REPORTED_USAGE_KEY,
SKILL_CONFIG_V1,
SNAPSHOT_V1,
STANDARD_EXTENSIONS,
Expand Down Expand Up @@ -147,6 +148,7 @@
"INSTALL_V1",
"MCP_CONFIG_V1",
"PEER_AGENTS_V1",
"REPORTED_USAGE_KEY",
"SKILL_CONFIG_V1",
"SNAPSHOT_V1",
"STAGING_ENDPOINT",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -450,6 +450,13 @@ class ProxyActionState(ExtensionResponse):
result: Annotated[ProxyBashResult | ProxyToolResult, Field(discriminator="kind")] | None = None


# The key under which a record's ``usage`` carries a ``Usage`` its driver reported, as ``Usage.to_dict`` wrote it; any
# other ``usage`` is a model response's own, which the proxy reads with ``Usage.from_model_response``. It rides in
# ``usage`` rather than a field of its own so a proxy on an earlier protocol, whose request refuses unknown fields,
# still takes the record.
REPORTED_USAGE_KEY = "agentenv_usage"


class ProxyRecordRequest(ExtensionRequest):
token: str
task_id: str
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@
import shutil
import signal
from collections import defaultdict
from collections.abc import AsyncIterator
from dataclasses import dataclass, field
from collections.abc import AsyncIterator, Mapping
from dataclasses import dataclass, field, fields, replace
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
Expand All @@ -31,6 +31,7 @@
from .extensions import (
MCP_CONFIG_V1,
PROXY_V1,
REPORTED_USAGE_KEY,
SKILL_CONFIG_V1,
TRAJECTORY_V1,
BundleSkillRequest,
Expand Down Expand Up @@ -111,6 +112,7 @@ class _Turn:
pending: list[tuple[str, str]] = field(default_factory=list) # (tool, its canonical arguments) not yet run
last_event: str = field(default_factory=lambda: _now())
model: str | None = None
model_usage: list[Usage] = field(default_factory=list) # each recorded model call's, its input counting the cache


@contextlib.asynccontextmanager
Expand Down Expand Up @@ -174,11 +176,11 @@ async def run(self, request: TaskRequest[ProxyConfig]) -> TaskResult:
if reply.error is not None:
turn.trajectory.append(_span("turn", "chain", turn.started, turn.model, prompt=prompt,
completion={"error": reply.error}, failed=True))
return TaskResult.failure("proxy.driver_error", reply.error)
return TaskResult.failure("proxy.driver_error", reply.error, usage=_turn_usage(turn))
turn.trajectory.append(_span("turn", "chain", turn.started, turn.model, prompt=prompt, completion={
"content": [{"type": "text", "text": part.text} for part in reply.parts]}))
return TaskResult.success(parts=tuple(TextPart(text=part.text) for part in reply.parts),
usage=Usage(tool_call_count=len(turn.actions)))
usage=_turn_usage(turn))

@extension(PROXY_V1.claim)
async def claim(self, request: ProxyClaimRequest) -> None:
Expand Down Expand Up @@ -238,6 +240,7 @@ async def result(self, request: ProxyResultRequest) -> ProxyActionState:
@extension(PROXY_V1.record)
async def record(self, request: ProxyRecordRequest) -> None:
content, calls = _read_message(request.message)
usage = _counting_cache(_recorded_usage(request.usage))
offered = await self._tools(self._open_turn(request.token, request.task_id))
turn = self._open_turn(request.token, request.task_id)
if turn.pending:
Expand All @@ -247,10 +250,9 @@ async def record(self, request: ProxyRecordRequest) -> None:
turn.recorded = True
turn.model = request.model or turn.model
span = _span("chat", "chat", turn.last_event, turn.model, completion={"content": content})
usage = request.usage or {}
for key, names in (("input_tokens", ("input_tokens", "prompt_tokens")),
("output_tokens", ("output_tokens", "completion_tokens"))):
if (count := next((usage[name] for name in names if name in usage), None)) is not None:
turn.model_usage.append(usage)
for key in ("input_tokens", "output_tokens"):
if (count := getattr(usage, key)) is not None:
span["attributes"][f"gen_ai.usage.{key}"] = count
turn.trajectory.append(span)
turn.last_event = span["end_time"]
Expand Down Expand Up @@ -519,6 +521,69 @@ def _tool_error(message: str) -> ProxyToolResult:
return ProxyToolResult(content=[{"type": "text", "text": f"error: {message}"}], is_error=True)


_USAGE_SUMS = (*(f.name for f in fields(Usage) if f.name.endswith("_tokens")), "cost_usd")
# Each provider's split of its cache writes by how long they're kept, which ``Usage.from_model_response`` keeps.
_CACHE_WRITE_SPLITS = ("cache_creation", "cache_write_tokens_details")


def _recorded_usage(usage: dict[str, Any] | None) -> Usage:
"""A recorded call's usage: the ``Usage`` its driver reported under ``REPORTED_USAGE_KEY``, taken as given, or
else read from the model response's own by ``Usage.from_model_response``."""
if usage is None:
return Usage()
if REPORTED_USAGE_KEY not in usage:
return Usage.from_model_response(usage)
try:
return Usage.from_dict(usage[REPORTED_USAGE_KEY])
except (TypeError, ValueError) as e:
raise HTTPException(status_code=400, detail=f"usage[{REPORTED_USAGE_KEY!r}]: {e}") from e


def _counting_cache(usage: Usage) -> Usage:
"""``usage`` with its ``input_tokens`` counting the cache, as its span shows it and its turn sums it: one that
leaves the cache out has it added, to its total too, and one with no cache to count counts it. One whose flag is
unknown while it has cache counts stays as given."""
cache = (usage.cache_read_tokens or 0) + (usage.cache_write_tokens or 0)
if usage.input_tokens is None or usage.input_includes_cache is True:
return usage
if usage.input_includes_cache is False:
total = None if usage.total_tokens is None else usage.total_tokens + cache
return replace(usage, input_tokens=usage.input_tokens + cache, total_tokens=total, input_includes_cache=True)
return replace(usage, input_includes_cache=True) if cache == 0 else usage


def _turn_usage(turn: _Turn) -> Usage:
"""The turn's usage: the calls to its tools, and each count summed over the recorded model calls that gave it,
None when none did, with ``input_tokens`` counting the cache. ``provider_details`` says how many calls were
recorded (``model_calls``); for a count only some gave, how many left it out (``unreported``), so the sum reads
as the lower bound it is; how many had cache counts with no word on whether their input held them
(``cache_unsettled``), which leaves the turn's flag unknown; how many usages were in a shape none of the readers
know (``unrecognized``); and each provider's split of the cache writes, summed, a reported ``Usage``'s split
counted only where it holds counts, as its ``provider_details`` may hold anything."""
calls = turn.model_usage
sums, unreported = {}, {}
for name in _USAGE_SUMS:
values = [value for call in calls if (value := getattr(call, name)) is not None]
sums[name] = sum(values) if values else None
if values and len(values) < len(calls):
unreported[name] = len(calls) - len(values)
unsettled = sum(call.input_tokens is not None and call.input_includes_cache is None for call in calls)
unrecognized = sum("unrecognized" in call.provider_details for call in calls)
splits: dict[str, dict[str, int]] = {}
for call in calls:
for name in _CACHE_WRITE_SPLITS:
if isinstance(split := call.provider_details.get(name), Mapping):
for part, count in split.items():
if isinstance(count, int) and not isinstance(count, bool) and count >= 0:
splits.setdefault(name, {})[part] = splits.get(name, {}).get(part, 0) + count
details = {"model_calls": len(calls), **({"unreported": unreported} if unreported else {}),
**({"cache_unsettled": unsettled} if unsettled else {}),
**({"unrecognized": unrecognized} if unrecognized else {}), **splits}
return Usage(tool_call_count=len(turn.actions), **sums,
input_includes_cache=True if sums["input_tokens"] is not None and not unsettled else None,
provider_details=details if calls else {})


def _span(name: str, operation: str, started: str, model: str | None, *, prompt: Any = None,
completion: Any = None, failed: bool = False) -> dict[str, Any]:
"""An OTel GenAI span, the trajectory format rubrics and the explorer read."""
Expand Down
Loading
Loading