[sophia-ramsey]

Motivation & Context

The agent evaluation previously omitted tools injected by context providers, causing Foundry tool-selection and tool-input-accuracy evaluators to return null scores because tool definitions were missing. This change captures the tool definitions available during execution and preserves schema-only snapshots across response serialization, so they remain available for evaluation. It supports evaluating agents with dynamically provided tools, both immediately after execution and from saved responses. Tool snapshots are preserved through streaming reconstruction, approval continuations, and loops.

Description & Review Guide

• What are the major changes? Capturing run-specific tool definitions, including provider-injected and dynamically exposed tools, and preserving schema-only snapshots across JSON/pickle serialization and agent-loop aggregation. Evaluation automatically uses these definitions, with an optional tools= override.
• What is the impact of these changes? Evaluation receiving the definitions needed to score tool calls without persisting executable callbacks or connections. Serialized responses gain tool-schema metadata, and schema capture introduces earlier JSON-compatibility validation that can reject non-serializable custom schemas.
• What do you want reviewers to focus on? Potential changes in serialization, specifically whether the metadata and schema validation introduces compatibility changes. The addition of tools= was not explicitly required to resolve this issue, but it seemed like it would improve the developer experience. Similarly, response additional_properties metadata now contains agent_framework.evaluation with version and tools (which can include an empty "tools": []).

Related Issue

Fixes #9160

AI Assistance

• No material AI assistance was used.
• This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: Used copilot with GPT-6.1 Sol for solution planning, design evaluation, implementation, and testing.

Contribution Checklist

• The code builds clean without any errors or warnings
• All unit tests pass, and I have added new tests where possible
• The PR follows the Contribution Guidelines
• This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
• This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

[github-code-quality]

Code coverage overview

Languages: Python

Python / code-coverage/python

main 9e971db	sramsey/python-tool-... 80ec512	+/-
92%	92%	0%

View coverage of the 8 most impacted files.

File	main 9e971db	sramsey/python-tool-... 80ec512	+/-
packages/core/a.../_evaluation.py	88%	88%	0%
packages/ag-ui/...i/_agent_run.py	90%	90%	0%
packages/core/a...ework/_types.py	96%	96%	0%
packages/core/a...ool_approval.py	91%	91%	0%
packages/ag-ui/...workflow_run.py	89%	89%	0%
packages/core/a...ork/_clients.py	94%	94%	0%
packages/core/a...arness/_loop.py	98%	98%	0%
packages/core/a...work/_agents.py	95%	96%	+1%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.

---

_{Updated October 29, 2026 23:35 UTC}

[Copilot]

🟡 Changes recommended

Serialization and supported streaming/client paths can still lose or misrepresent tool definitions.

4 open findings

• Preserve tool snapshots in middleware streaming responses · New
• Capture dynamically added tools at each model call · New
• Preserve tuple values in serialized snapshots · New
• Persist empty tool snapshots across serialization · New

What changed in this PR

Addresses #9160 by capturing run-specific tool definitions for evaluation, including tools supplied by context providers.

Changes:

• Captures schema-only tool snapshots and preserves them through serialization.
• Adds an explicit tools= evaluation override and loop aggregation support.
• Adds regression tests and documents snapshot behavior.

File	Description
python/​packages/​core/​tests/​core/​test_evaluation.py	Tests tool discovery, overrides, and serialization.
python/​packages/​core/​AGENTS.md	Documents evaluation snapshots.
python/​packages/​core/​agent_framework/​_types.py	Stores and restores tool schemas.
python/​packages/​core/​agent_framework/​_harness/​_loop.py	Aggregates definitions across loop iterations.
python/​packages/​core/​agent_framework/​_evaluation.py	Uses captured definitions and explicit overrides.
python/​packages/​core/​agent_framework/​_agents.py	Captures tools during execution.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

MAF Automated Review — Iteration 1

Result: Findings reported
Scope: full PR (2 commit(s)): 813d7c0994f1, 36a84a8608a6
Model: gpt-5.6-sol

Overview

The change captures run-specific tool declarations with deep-copied, JSON-validated schemas, restores them as non-executable tools, and carries them through normal response serialization and loop aggregation. Per-run capture, malformed-metadata validation, explicit overrides, and broad streaming/non-streaming tests are strong guardrails. Three serialization and lifecycle gaps remain: empty snapshots are erased, same-name progressive replacements retain stale schemas, and update-only streaming round trips omit the snapshots.

Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
3 verified findings remained after source verification (3 medium) across 2 files. Details are attached to the affected lines below.

Affected areas: python/packages/core/agent_framework/_agents.py, python/packages/core/agent_framework/_types.py

[Eduard van Valkenburg (eavanvalkenburg)]

sophia-ramsey Two additional evaluation-metadata cases still misrepresent what the model could call:

1. _agents.py:1787-1790 seeds the snapshot from prepared options before chat middleware runs. If middleware removes a configured tool from every outgoing request, subsequent captures cannot remove that seed: the model receives no tools, but EvalItem.tools still includes the removed tool, including after serialization. Please make observed model-call exposure authoritative rather than unioning it with unobserved provisional configuration.
2. _types.py:75-78 overwrites declarations by tool name. If one function-loop iteration calls a no-argument search({}), then replaces it with a same-name declaration requiring city, the later valid search({"city":"Seattle"}) leaves only the city-required schema in evaluation. That schema is then incorrectly applied to the earlier {} call. This is distinct from the fixed stale-replacement-schema case: updating the declaration fixes later calls but mischaracterizes earlier ones. Preserve the declaration associated with each model-call occurrence, or explicitly reject evaluation when the evaluator cannot represent incompatible same-name versions instead of silently using the last schema for every call.

Both cases reproduced against exact-head capture and evaluation logic. The already-raised continuation-token loss is separate and is not repeated here.

[sophia-ramsey]

Updated the PR for snapshots to capture only tools exposed after chat middleware and to check for schema changes so that declarations with the same name but other changes will raise an error. Right now, I have it checking for any input-schema change, which means compatible changes would be rejected too. Can you confirm if this is the behavior we want? Eduard van Valkenburg (@eavanvalkenburg)