Skip to content

feat(protocol): the ATIF validator, its CLI and the adapter kit (Y1b, FD-6075) - #793

Merged
earakely-scale merged 5 commits into
mainfrom
edgararakelyan/fd-6075-y1b-atif-validator
Nov 19, 2026
Merged

earakely-scale merged 5 commits into
mainfrom
edgararakelyan/fd-6075-y1b-atif-validator

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Y1b of FD-6075: the ATIF validator, its CLI and the adapter kit, on top of Y1a (#778). Agents' CI, the SDK (Y2b) and the worker (Y4) can now check a document, check what a live reader was served, and check an adapter over recorded tasks.

What it adds

  • agentenv_protocol.atif.validate(document, reader=False, media=None) returns findings. Each has a rule id, a severity and a path into the document, such as steps[2].tool_calls[0]. It checks:

    • ATIF's vendored models (atif), JSON itself (json), and the profile's models (profile);
    • what neither checks:
      • profile keys that profile 1.0 doesn't define (profile-key), and the profile version (profile-version);
      • vocabulary values (vocabulary), and extra keys outside agentenv (outside-key);
      • a tool_call_id unique across the document and its sub-agents (call-id), and references that resolve to an embedded sub-agent (reference);
      • final_metrics totals over the root and every sub-agent it references, comparing a count only when every model step has it (totals), and total_steps (total-steps);
      • cache reads and writes within prompt_tokens (usage);
      • media paths of the form media/<sha256>.<ext> (media-path), and with media given, each file's bytes against its name and profile (media-bytes);
      • timestamps in the builder's form (timestamp);
      • Harbor's to_json_dict giving the document back unchanged (harbor-save), and an all-text message written as one string (text-message).
  • Two modes. A producer is held to profile 1.0, and every rule is an error. reader=True checks someone else's document for reading, such as a Harbor task's trajectory.json:

    • errors only where agentenv couldn't read it: json, atif, profile, media-bytes;
    • warnings where a producer would fail;
    • timestamp isn't checked at all.

    PROFILE.md's new "Validator" table lists every rule with both severities, generated from the rule definitions, and a test keeps the two equal.

  • validate_live(pages, final), the live rule (Decision 9):

    • every page a reader was served validates;
    • each page is a prefix of the next page and of the final read (live-prefix);
    • no page serves a step with an unanswered call before its task ends (live-unanswered).
  • errors_in(findings) keeps the findings that fail a check. It's named so, not errors, since a generic public name is hard to change after a release.

  • The CLI: python -m agentenv_protocol.atif validate FILE [--media DIR] [--reader] prints each finding. It exits 1 on an error, 0 when there are only warnings, and 2 when the file can't be read.

  • The adapter kit, check_adapter(adapter, recorded, expected=, from_atif=), for an agent's CI. It runs an adapter over a context's recorded tasks, through the same AdapterRun the SDK uses live. It applies the live rule after every record and the validator to the final document. It adds four rules of its own:

    • kit-run: the adapter raised;
    • kit-expected: the document differs from the checked-in one, naming the first differing path;
    • kit-unmatched: a warning for native records no handler takes (Y1a-5);
    • kit-from-atif: from_atif doesn't give a native log for a document without the profile. By default it's tried on the run's own document with the profile stripped. It only warns when the adapter has no from_atif.

    KitResult.raise_for_errors() makes the result a one-line pytest assertion.

    validate_live checks a page equal to the one before it only once, since most records close no step. Over a 400-step session the kit went from 5.6 s to 2.6 s. Its cost still grows with the square of the session's length (about 0.6 s at 200 steps), because each new page is validated whole. Validating only each page's new steps, which the prefix check makes safe, would make it linear; that waits until a recorded session needs it.

Choices to review

  • The issue's "the document without extra.agentenv still validates" rule is left out. With ATIF 1.8's models every extra is a free-form dict, so removing the profile can't make a valid document invalid. The rule could never fire. Y1a's fixture tests still check the stripped fixtures validate.
  • Two rules beyond the issue's list:
    • reference, which the live rule needs ("every reference it is served resolves");
    • usage, which catches an adapter with its input_includes_cache constant wrong.
  • totals sums the root and every embedded sub-agent. The issue says "every sub-agent trajectory their calls reference", but the builder, and ATIF's "across all steps", sum every embedded one. They differ only for a sub-agent whose spawning call never answered (below).
  • Sub-agents are walked at any depth. A reference resolves to a sub-agent's path: one embedded in its own trajectory first, then the nearest one around it. That covers ATIF's nesting (an id is unique only within its parent's array) and the builder's embedding of every sub-agent in the root.
  • Reader-mode severities. The issue says only profile-key and total-steps warn for a reader. I extended that to every producer-only rule, following Decision 5's strict producers and tolerant readers. A Harbor document with a file-ref sub-agent or totals computed its own way stays readable. Harbor's own ATIF of a real Claude Code session reads with only outside-key warnings, for its cwds and git_branches extras.
  • total-steps accepts any non-empty root notes, since ATIF says only that a difference "must be documented in the root-level notes field".
  • A duplicate tool_call_id written through an adapter surfaces as kit-run. The builder already refuses one when it's written. The validator's call-id rule covers documents from other producers.
  • The kit takes media=, the files of Media in recorded inbound or final content, by path, since a recorded Media doesn't hold its bytes.
  • The Harbor parity check runs in the VM run, not in CI. HI1 (FD-6109), the Python 3.12 job with the harbor extra that the issue names, is still in Backlog.

Found by chaos, for Y1a's builder

A sub-agent whose spawning call never gets a result before its task ends is written as an embedded trajectory that nothing references. Its reference goes on that result, and the result never comes. So a reader can't tell which call or task the sub-agent belongs to. The validator handles this by not holding such a sub-agent to live-unanswered, and by summing it in totals as the builder does. Recording the spawning call on the sub-agent's own profile would close the gap, but that's a new profile field, so I've left it for a follow-up.

Testing

  • 72 new protocol tests:
    • every fixture clean in both modes, media checked;
    • every case's live pages pass the live rule;
    • a negative fixture for each rule, failing with that rule's id; the CLI reports it, and a reader sees it at the table's severity;
    • sub-agent totals, including a null count and a numeric-string count;
    • a sub-agent nested in a sub-agent, a reference resolving through an outer trajectory, and two children sharing an id under different parents keeping their own tasks;
    • a page that differs from the final read by one step;
    • a page serving an unanswered call;
    • the kit passing the toy adapter against all seven fixtures, odd_parts with its attached files included;
    • the kit failing a duplicating adapter, a wrong expected document, and a from_atif with no log;
    • python -m in a subprocess.
  • CI time: no integration tests were added, since nothing in Y1b deploys anything, so e2e CI time doesn't change.
  • On the arm64 test host, against the installed wheel (919884d):
    • The ATIF tests: 181 pass on Python 3.10 with pydantic 2.11.10, and on 3.13 with 2.14.3.

    • e2e (PASS on all three venvs):

      • the kit over two real Claude Code sessions, from a real claude -p run, as one context with two tasks: no error, the live rule after every record;
      • python -m agentenv_protocol.atif validate --media on all eight shipped fixtures: valid;
      • Harbor's own ATIF of a real session, read with --reader: no error.
    • Harbor 0.26.0 parity: Harbor's TrajectoryValidator accepts every shipped fixture, media included. Of the 16 negatives, Harbor rejects only atif and media-path, and the validator rejects all 16.

    • Chaos, 15 minutes on each of three venvs (5 minutes more on an earlier commit was stopped): 145,807 random builder scripts, 5,179,137 live pages, 2,604,560 random mutations and 770,000 targeted breaks, with 0 failures. Its checks:

      • no finding on any builder document or page, and validate_live clean;
      • no crash on any mutation, in either mode;
      • on the Harbor venv, 830,400 mutations checked against Harbor: its models accept exactly what the vendored ones do, and nothing Harbor rejects passes the validator;
      • each targeted break reported under its rule.

      On the code before 21a4da1 and 919884d, chaos and Greptile found the sub-agent placement and totals mismatch, the null count crash and the nested walk, all fixed in those two commits. The host is stopped.

  • Greptile: its five comments are fixed and resolved; the fifth, children sharing an id, in 0179042, which a local chaos run also passed. ruff check . passes.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new blocking issue was found.

Summary

Adds ATIF document checks, a command-line validator, and a kit for testing adapters against recorded tasks.

  • ATIF documents and live pages now receive rule-based validation findings.
  • The ATIF CLI reports findings and exits with a status code.
  • Adapter authors can check recorded runs against ATIF output.

No new actionable issues found. The five earlier findings are addressed in the current code.

Reviews (9) · Last reviewed commit: "perf(protocol): the live check validates..." · Reviewed by Greptile

… FD-6075)

agentenv_protocol.atif.validate checks a document against ATIF's vendored models, the profile's models, and what
neither checks: unknown profile keys and versions, vocabularies, keys outside extra.agentenv, unique tool_call_ids,
references that resolve, final_metrics totals over the root and the sub-agents it references, total_steps, cache
counts within prompt_tokens, media paths and bytes, timestamps, Harbor's save and all-text messages as strings. Each
finding carries a rule id PROFILE.md names; a producer is held to profile 1.0, a reader only warned where a producer
would fail. validate_live checks the pages a live reader was served: each a prefix of the next and of the final read,
with no unanswered call served before its task ends. python -m agentenv_protocol.atif validate FILE [--media DIR]
[--reader] prints the findings. check_adapter runs an adapter over recorded tasks for an agent's CI: the live rule
after every record, the validator on the document, the expected document, unmatched records and from_atif.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 18, 2026 23:14
earakely-scale and others added 2 commits November 18, 2026 15:20
…sub-agent's task through its spawn, and reads a null count as absent

Chaos on the validator found three things. final_metrics totals summed only the sub-agents a result references, while
the builder, and ATIF's "across all steps", sum every embedded one: a sub-agent whose spawning call never answered is
embedded but referenced by nothing. live-unanswered couldn't place such a sub-agent's steps in a task, and flagged
them; a sub-agent's task now comes through its chain of spawning calls, nested ones included, and one no served step
references isn't checked. A count present as null, which ATIF allows, crashed the totals sum. Difference paths are
now bare paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… ATIF's models read them, and the kit takes recorded media

Greptile's first review: nested subagent_trajectories, which ATIF allows, escaped every check past the first level;
the walk now goes to any depth with full paths, and a reference resolves to a sub-agent embedded in its trajectory or
one around it. A count given as a numeric string, which the models accept, reached arithmetic in usage; counts are now
read through the models' own int coercion. check_adapter takes media= for the files of Media in the recorded tasks'
inbound or final content, so a case with attached files, odd_parts, replays through the kit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h, not its bare id

ATIF makes a trajectory_id unique only within its parent's array, so two sub-agents may each embed a child of the same
id. References now resolve to a sub-agent's path, its own embedded children first and then those around it, and the
live check walks the spawning chain by path, so each child keeps its own task. The reference rule uses the same
resolver.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rors is errors_in

validate_live skips a page equal to the one before it: most records close no step, so a recorded session's pages
repeat. The adapter kit over a 400-step session drops from 5.6 s to 2.6 s. The helper that filters findings to errors
is renamed errors_in, before a release makes its generic name hard to change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit ddc82e5 into main Nov 19, 2026
66 of 69 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fd-6075-y1b-atif-validator branch November 19, 2026 02:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant