Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,49 @@ field added in a later minor names the version it arrived in. Proposed for a lat
`input_includes_cache` on the root and `cost_status` (`reported`, `priced` or `unpriced`), with the usage
cross-check that reads them; per-part origins, when a reader needs them; and an always-on harness's fields.

## Validator

`validate(document)` returns findings, each with a rule id below, a severity and where in the document it is. It holds a
producer's output to profile 1.0. `reader=True` checks someone else's document for reading instead: what a producer
must not write only warns, and the document fails only where agentenv couldn't read it. `media` is the folder the
document's media paths are relative to, the one holding `media/`, or the files by path; without it only the paths' form
is checked. `validate_live(pages, final)` checks what a live reader was served, page by page, with the final read. A sub-agent's task is its spawning call's, found through the references, so an embedded sub-agent that no served step references isn't held to `live-unanswered`. On
the command line: `python -m agentenv_protocol.atif validate FILE [--media DIR] [--reader]`, which prints each finding
and exits 1 when one is an error.

| Rule | Holds | Producer | Reader |
| --- | --- | --- | --- |
| `json` | The document is a JSON object: UTF-8, with no NaN or infinity | error | error |
| `atif` | ATIF's own models, vendored from Harbor, accept it | error | error |
| `profile` | Each `extra.agentenv` parses with its profile model | error | error |
| `profile-key` | Each `extra.agentenv` holds only the fields profile 1.0 defines | error | warning |
| `profile-version` | The root's `extra.agentenv.version`, when given, is `1.0` | error | warning |
| `vocabulary` | A vocabulary field holds one of its values | error | warning |
| `outside-key` | No `extra` holds a key outside `agentenv` but `ToolCall.extra.raw_arguments` | error | warning |
| `call-id` | Each `tool_call_id` is unique across the document and its sub-agents | error | warning |
| `reference` | Each `subagent_trajectory_ref` names an embedded sub-agent by `trajectory_id` | error | warning |
| `totals` | A `final_metrics` total equals the sum over the model steps of the root and its embedded sub-agents, when every one has the count | error | warning |
| `total-steps` | `final_metrics.total_steps` counts the root's steps, or the root's `notes` explain the difference | error | warning |
| `usage` | A step's cache reads and writes fit within its `prompt_tokens` | error | warning |
| `media-path` | A media path is `media/<sha256>.<ext>` | error | warning |
| `media-bytes` | Given the media, each file the document names exists, with the sha256 its name and profile give and the profile's size | error | error |
| `timestamp` | A timestamp is UTC in the builder's form: `YYYY-MM-DDTHH:MM:SSZ`, with no, 3 or 6 fractional digits | error | not checked |
| `harbor-save` | Harbor's `to_json_dict` gives the document back unchanged | error | warning |
| `text-message` | A step message whose parts are all text is one string | error | warning |
| `live-prefix` | Every live page is a prefix of the next page and of the final read | error | not checked |
| `live-unanswered` | No live page serves a step with an unanswered call before its task ends | error | not checked |

The adapter kit, `check_adapter(adapter, recorded, expected=, from_atif=)`, is for an agent's CI. It runs an adapter
over a context's recorded tasks as the SDK runs it live, applies the live rule after every record and the validator to
the final document, and adds its own rules. `kit-from-atif` only warns when the adapter has no `from_atif`.

| Rule | Holds | Severity |
| --- | --- | --- |
| `kit-run` | The adapter takes every recorded record without raising | error |
| `kit-expected` | The document equals the checked-in expected document | error |
| `kit-unmatched` | Every native record has a handler; one that none takes is only counted | warning |
| `kit-from-atif` | `from_atif` turns a document without the profile into a native log | error |

## Fixtures

`fixtures/` holds the profile's canonical examples, built through the adapter framework from a toy format's records:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,16 +44,19 @@
profile_of,
read_profile,
)
from .kit import KIT_RULES, KitResult, check_adapter
from .reader import Task, check, is_prefix, load, media_paths, strip_profile, tasks
from .runner import AdapterRun, Fed, RecordedTask, replay
from .validator import RULES, Finding, Rule, errors_in, validate, validate_live

__all__ = [
"Adapter", "AdapterRef", "AdapterRun", "AdapterSpec", "Agent", "AgentProfile", "Arrival", "AtifError", "AtifTurn",
"AudioSource", "ContentPart", "Document", "FORMAT", "Fed", "FinalMetrics", "FinalMetricsProfile", "ImageSource",
"Media", "MediaInfo", "Metrics", "MetricsProfile", "Native", "Observation", "ObservationResult",
"ObservationResultProfile", "PROFILES", "PROFILE_KEY", "PROFILE_VERSION", "RAW_ARGUMENTS", "ReasoningBlock",
"RecordedTask", "SCHEMA_VERSION", "Selector", "Step", "StepError", "StepProfile", "SubagentTrajectoryRef",
"SubagentTrajectoryRefProfile", "Task", "ToolCall", "ToolCallProfile", "Trajectory", "TrajectoryProfile",
"TurnContext", "VOCABULARIES", "atif_adapter", "aux", "check", "dump_profile", "finish", "is_prefix", "load",
"media_paths", "native", "profile_of", "read_profile", "replay", "strip_profile", "tasks", "timestamp",
"Adapter", "AdapterRef", "AdapterRun", "AdapterSpec", "Agent", "AgentProfile", "Arrival", "atif_adapter",
"AtifError", "AtifTurn", "AudioSource", "aux", "check", "check_adapter", "ContentPart", "Document", "dump_profile",
"errors_in", "Fed", "FinalMetrics", "FinalMetricsProfile", "Finding", "finish", "FORMAT", "ImageSource",
"is_prefix", "KIT_RULES", "KitResult", "load", "Media", "media_paths", "MediaInfo", "Metrics", "MetricsProfile",
"Native", "native", "Observation", "ObservationResult", "ObservationResultProfile", "PROFILE_KEY", "profile_of",
"PROFILE_VERSION", "PROFILES", "RAW_ARGUMENTS", "read_profile", "ReasoningBlock", "RecordedTask", "replay", "Rule",
"RULES", "SCHEMA_VERSION", "Selector", "Step", "StepError", "StepProfile", "strip_profile", "SubagentTrajectoryRef",
"SubagentTrajectoryRefProfile", "Task", "tasks", "timestamp", "ToolCall", "ToolCallProfile", "Trajectory",
"TrajectoryProfile", "TurnContext", "validate", "validate_live", "VOCABULARIES",
]
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
"""``python -m agentenv_protocol.atif validate FILE [--media DIR] [--reader]``: check an ATIF document, printing each
finding; exit 1 when any is an error, 2 when the file can't be read."""
from __future__ import annotations

import argparse
import sys
from pathlib import Path
from typing import Sequence

from .validator import errors_in, parse, validate


def main(argv: Sequence[str] | None = None) -> int:
parser = argparse.ArgumentParser(prog="python -m agentenv_protocol.atif",
description="Check ATIF documents against ATIF and the agentenv profile.")
commands = parser.add_subparsers(dest="command", required=True)
check = commands.add_parser("validate", help="check one document; exit 1 when it has an error")
check.add_argument("file", type=Path, help="the trajectory.json to check")
check.add_argument("--media", type=Path, metavar="DIR",
help="the folder the document's media paths are relative to, which holds media/; "
"given, each file is checked against its name and profile")
check.add_argument("--reader", action="store_true",
help="check someone else's document for reading: what a producer must not write warns")
args = parser.parse_args(argv)
try:
document, findings = parse(args.file)
except OSError as exc:
print(f"can't read {args.file}: {exc.strerror or exc}", file=sys.stderr)
return 2
if not findings:
findings = validate(document, reader=args.reader, media=args.media)
for finding in findings:
print(finding)
failed = errors_in(findings)
print(f"{args.file}: {'invalid' if failed else 'valid'}, {len(failed)} errors, "
f"{len(findings) - len(failed)} warnings")
return 1 if failed else 0


if __name__ == "__main__":
sys.exit(main())
106 changes: 106 additions & 0 deletions packages/agentenv-protocol/src/agentenv_protocol/atif/kit.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
"""The adapter kit, for an agent's CI: run an adapter over recorded tasks, as the SDK runs it live, and check what it
writes. The live rule holds after every record, the final document passes the validator as a producer's, it equals
the checked-in document when one is given, and ``from_atif`` turns ATIF without the profile back into a native log."""
from __future__ import annotations

import json
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Mapping, Sequence

from .adapter import Adapter, Native
from .profile import profile_of
from .reader import strip_profile
from .runner import AdapterRun, RecordedTask
from .validator import ERROR, WARNING, Finding, Rule, _difference, errors_in, validate_live

KIT_RULES: dict[str, Rule] = {rule.id: rule for rule in (
Rule("kit-run", ERROR, None, "The adapter takes every recorded record without raising"),
Rule("kit-expected", ERROR, None, "The document equals the checked-in expected document"),
Rule("kit-unmatched", WARNING, None, "Every native record has a handler; one that none takes is only counted"),
Rule("kit-from-atif", ERROR, None, "`from_atif` turns a document without the profile into a native log"),
)}


@dataclass
class KitResult:
"""What the kit found, with the document and every live page the adapter's run served."""

document: dict[str, Any] | None
pages: list[dict[str, Any] | None] = field(default_factory=list)
media: dict[str, bytes] = field(default_factory=dict)
findings: list[Finding] = field(default_factory=list)

@property
def errors(self) -> list[Finding]:
return errors_in(self.findings)

def raise_for_errors(self) -> None:
"""Raise AssertionError listing every finding when any is an error."""
if self.errors:
raise AssertionError("the adapter kit found:\n" + "\n".join(str(f) for f in self.findings))


def check_adapter(adapter: type[Adapter], recorded: Sequence[RecordedTask], *,
expected: Mapping[str, Any] | Path | None = None,
from_atif: Sequence[Mapping[str, Any]] | None = None,
media: Mapping[str, bytes] | None = None) -> KitResult:
"""Run ``adapter`` over ``recorded``, one A2A context's tasks in order, and check it. ``expected`` is the document
the run should write, or the file holding it. ``from_atif`` are the documents its ``from_atif`` is tried on; by
default, the run's own document with the profile stripped. ``media`` holds, by path, the files of any Media in the
recorded tasks' inbound or final content; the adapter's own media come from the run."""
run = AdapterRun(adapter)
result = KitResult(document=None)
where = ""
try:
for t, task in enumerate(recorded):
where = f"task {t}, its start"
run.start_task(task.context)
result.pages.append(run.page())
for r, fed in enumerate(task.records):
where = f"task {t}, record {r}" + (f" of aux {fed.aux!r}" if fed.aux is not None else "")
if fed.aux is None:
run.native(fed.record)
else:
run.aux(fed.aux, fed.record)
result.pages.append(run.page())
where = f"task {t}, its end"
run.end_task(state=task.state, final=task.final, ended_at=task.ended_at)
result.pages.append(run.page())
except Exception as exc:
result.findings.append(Finding("kit-run", ERROR, where, f"{type(exc).__name__}: {exc}"))
return result
result.document = run.document()
result.media = {**(media or {}), **run.media_files}
if result.document is None:
result.findings.append(Finding("kit-run", ERROR, "", "the run wrote no step"))
return result
result.findings.extend(validate_live(result.pages, result.document, media=result.media))
Comment thread
greptile-apps[bot] marked this conversation as resolved.
Resolved
unmatched = profile_of(result.document).get("unmatched")
if unmatched:
counts = ", ".join(f"{kind} ({count})" for kind, count in sorted(unmatched.items()))
result.findings.append(Finding("kit-unmatched", WARNING, "extra.agentenv.unmatched",
f"no handler takes these native records: {counts}"))
if expected is not None:
want = json.loads(expected.read_text(encoding="utf-8")) if isinstance(expected, Path) else expected
if result.document != want:
result.findings.append(Finding("kit-expected", ERROR, _difference(want, result.document),
"the document differs from the expected one here"))
for index, document in enumerate(from_atif if from_atif is not None else [strip_profile(result.document)]):
_from_atif(adapter, document, index, result)
return result


def _from_atif(adapter: type[Adapter], document: Mapping[str, Any], index: int, result: KitResult) -> None:
where = f"from_atif document {index}"
try:
native = adapter.from_atif(document)
except NotImplementedError as exc:
result.findings.append(Finding("kit-from-atif", WARNING, where, str(exc)))
return
except Exception as exc:
result.findings.append(Finding("kit-from-atif", ERROR, where, f"raised {type(exc).__name__}: {exc}"))
return
if not isinstance(native, Native) or not isinstance(native.log, list):
result.findings.append(Finding("kit-from-atif", ERROR, where,
f"returned {type(native).__name__}, not a Native holding a list log"))
Loading
Loading