Repository navigation
feat(protocol): the ATIF validator, its CLI and the adapter kit (Y1b, FD-6075) - #793
Merged
Merged
Conversation
… FD-6075) agentenv_protocol.atif.validate checks a document against ATIF's vendored models, the profile's models, and what neither checks: unknown profile keys and versions, vocabularies, keys outside extra.agentenv, unique tool_call_ids, references that resolve, final_metrics totals over the root and the sub-agents it references, total_steps, cache counts within prompt_tokens, media paths and bytes, timestamps, Harbor's save and all-text messages as strings. Each finding carries a rule id PROFILE.md names; a producer is held to profile 1.0, a reader only warned where a producer would fail. validate_live checks the pages a live reader was served: each a prefix of the next and of the final read, with no unanswered call served before its task ends. python -m agentenv_protocol.atif validate FILE [--media DIR] [--reader] prints the findings. check_adapter runs an adapter over recorded tasks for an agent's CI: the live rule after every record, the validator on the document, the expected document, unmatched records and from_atif. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sub-agent's task through its spawn, and reads a null count as absent Chaos on the validator found three things. final_metrics totals summed only the sub-agents a result references, while the builder, and ATIF's "across all steps", sum every embedded one: a sub-agent whose spawning call never answered is embedded but referenced by nothing. live-unanswered couldn't place such a sub-agent's steps in a task, and flagged them; a sub-agent's task now comes through its chain of spawning calls, nested ones included, and one no served step references isn't checked. A count present as null, which ATIF allows, crashed the totals sum. Difference paths are now bare paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… ATIF's models read them, and the kit takes recorded media Greptile's first review: nested subagent_trajectories, which ATIF allows, escaped every check past the first level; the walk now goes to any depth with full paths, and a reference resolves to a sub-agent embedded in its trajectory or one around it. A count given as a numeric string, which the models accept, reached arithmetic in usage; counts are now read through the models' own int coercion. check_adapter takes media= for the files of Media in the recorded tasks' inbound or final content, so a case with attached files, odd_parts, replays through the kit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h, not its bare id ATIF makes a trajectory_id unique only within its parent's array, so two sub-agents may each embed a child of the same id. References now resolve to a sub-agent's path, its own embedded children first and then those around it, and the live check walks the spawning chain by path, so each child keeps its own task. The reference rule uses the same resolver. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rors is errors_in validate_live skips a page equal to the one before it: most records close no step, so a recorded session's pages repeat. The adapter kit over a 400-step session drops from 5.6 s to 2.6 s. The helper that filters findings to errors is renamed errors_in, before a release makes its generic name hard to change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Y1b of FD-6075: the ATIF validator, its CLI and the adapter kit, on top of Y1a (#778). Agents' CI, the SDK (Y2b) and the worker (Y4) can now check a document, check what a live reader was served, and check an adapter over recorded tasks.
What it adds
agentenv_protocol.atif.validate(document, reader=False, media=None)returns findings. Each has a rule id, a severity and a path into the document, such assteps[2].tool_calls[0]. It checks:atif), JSON itself (json), and the profile's models (profile);profile-key), and the profile version (profile-version);vocabulary), andextrakeys outsideagentenv(outside-key);tool_call_idunique across the document and its sub-agents (call-id), and references that resolve to an embedded sub-agent (reference);final_metricstotals over the root and every sub-agent it references, comparing a count only when every model step has it (totals), andtotal_steps(total-steps);prompt_tokens(usage);media/<sha256>.<ext>(media-path), and withmediagiven, each file's bytes against its name and profile (media-bytes);timestamp);to_json_dictgiving the document back unchanged (harbor-save), and an all-text message written as one string (text-message).Two modes. A producer is held to profile 1.0, and every rule is an error.
reader=Truechecks someone else's document for reading, such as a Harbor task's trajectory.json:json,atif,profile,media-bytes;timestampisn't checked at all.PROFILE.md's new "Validator" table lists every rule with both severities, generated from the rule definitions, and a test keeps the two equal.
validate_live(pages, final), the live rule (Decision 9):live-prefix);live-unanswered).errors_in(findings)keeps the findings that fail a check. It's named so, noterrors, since a generic public name is hard to change after a release.The CLI:
python -m agentenv_protocol.atif validate FILE [--media DIR] [--reader]prints each finding. It exits 1 on an error, 0 when there are only warnings, and 2 when the file can't be read.The adapter kit,
check_adapter(adapter, recorded, expected=, from_atif=), for an agent's CI. It runs an adapter over a context's recorded tasks, through the sameAdapterRunthe SDK uses live. It applies the live rule after every record and the validator to the final document. It adds four rules of its own:kit-run: the adapter raised;kit-expected: the document differs from the checked-in one, naming the first differing path;kit-unmatched: a warning for native records no handler takes (Y1a-5);kit-from-atif:from_atifdoesn't give a native log for a document without the profile. By default it's tried on the run's own document with the profile stripped. It only warns when the adapter has nofrom_atif.KitResult.raise_for_errors()makes the result a one-line pytest assertion.validate_livechecks a page equal to the one before it only once, since most records close no step. Over a 400-step session the kit went from 5.6 s to 2.6 s. Its cost still grows with the square of the session's length (about 0.6 s at 200 steps), because each new page is validated whole. Validating only each page's new steps, which the prefix check makes safe, would make it linear; that waits until a recorded session needs it.Choices to review
extra.agentenvstill validates" rule is left out. With ATIF 1.8's models everyextrais a free-form dict, so removing the profile can't make a valid document invalid. The rule could never fire. Y1a's fixture tests still check the stripped fixtures validate.reference, which the live rule needs ("every reference it is served resolves");usage, which catches an adapter with itsinput_includes_cacheconstant wrong.totalssums the root and every embedded sub-agent. The issue says "every sub-agent trajectory their calls reference", but the builder, and ATIF's "across all steps", sum every embedded one. They differ only for a sub-agent whose spawning call never answered (below).profile-keyandtotal-stepswarn for a reader. I extended that to every producer-only rule, following Decision 5's strict producers and tolerant readers. A Harbor document with a file-ref sub-agent or totals computed its own way stays readable. Harbor's own ATIF of a real Claude Code session reads with onlyoutside-keywarnings, for itscwdsandgit_branchesextras.total-stepsaccepts any non-empty rootnotes, since ATIF says only that a difference "must be documented in the root-level notes field".tool_call_idwritten through an adapter surfaces askit-run. The builder already refuses one when it's written. The validator'scall-idrule covers documents from other producers.media=, the files ofMediain recorded inbound or final content, by path, since a recordedMediadoesn't hold its bytes.Found by chaos, for Y1a's builder
A sub-agent whose spawning call never gets a result before its task ends is written as an embedded trajectory that nothing references. Its reference goes on that result, and the result never comes. So a reader can't tell which call or task the sub-agent belongs to. The validator handles this by not holding such a sub-agent to
live-unanswered, and by summing it intotalsas the builder does. Recording the spawning call on the sub-agent's own profile would close the gap, but that's a new profile field, so I've left it for a follow-up.Testing
nullcount and a numeric-string count;odd_partswith its attached files included;from_atifwith no log;python -min a subprocess.The ATIF tests: 181 pass on Python 3.10 with pydantic 2.11.10, and on 3.13 with 2.14.3.
e2e (PASS on all three venvs):
claude -prun, as one context with two tasks: no error, the live rule after every record;python -m agentenv_protocol.atif validate --mediaon all eight shipped fixtures: valid;--reader: no error.Harbor 0.26.0 parity: Harbor's
TrajectoryValidatoraccepts every shipped fixture, media included. Of the 16 negatives, Harbor rejects onlyatifandmedia-path, and the validator rejects all 16.Chaos, 15 minutes on each of three venvs (5 minutes more on an earlier commit was stopped): 145,807 random builder scripts, 5,179,137 live pages, 2,604,560 random mutations and 770,000 targeted breaks, with 0 failures. Its checks:
validate_liveclean;On the code before 21a4da1 and 919884d, chaos and Greptile found the sub-agent placement and totals mismatch, the
nullcount crash and the nested walk, all fixed in those two commits. The host is stopped.ruff check .passes.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; no new blocking issue was found.Summary
Adds ATIF document checks, a command-line validator, and a kit for testing adapters against recorded tasks.
No new actionable issues found. The five earlier findings are addressed in the current code.
Reviews (9) · Last reviewed commit: "perf(protocol): the live check validates..." · Reviewed by Greptile