[Qiming Bao (14H034160212)]

Motivation & Context

tool_called_check, tool_calls_present and tool_call_args_match scanned item.conversation, the whole conversation, for function calls. Every other derived value on EvalItem (query, response) and the Foundry evaluator follow the item's split strategy.

As a result, in multi-turn evaluation a tool call from an earlier turn satisfied the check for a later turn. With EvalItem.per_turn_items(...), a turn in which the agent made no tool call still passed all three checks, because the earlier call sits on the query side of that turn's split. This is a silent false pass on the turns that per-turn evaluation is meant to isolate, and it contradicts ADR-0023 ("Changing the split_strategy consistently changes all derived values").

This implements the plan agreed in #10091.

Description & Review Guide

• What are the major changes?
  • One private helper, _response_function_calls, returns the function calls on the response side of item.split_messages() (the item's strategy, else LAST_TURN). tool_called_check, tool_calls_present and tool_call_args_match all use it, so they look at the same messages as item.response.
  • The docstrings of the three checks, ConversationSplit and EvalItem.split_strategy now say that the tool checks follow the split, and that ConversationSplit.FULL checks the whole trajectory.
• What is the impact of these changes? No API change. Single-turn items with the default LAST_TURN split (for example from evaluate_agent) give the same results, because the query side holds only the user message. With FULL, every call after the first user message still counts. The behavior change is for multi-turn items: tool calls on the query side, such as calls from earlier turns, no longer count. Callers who want whole-conversation matching can use ConversationSplit.FULL. A custom splitter decides which calls count, the same way it decides item.response.
• What do you want reviewers to focus on? Whether the docstring wording is enough as the migration note for multi-turn callers. I did not edit python/CHANGELOG.md, following the repo rules.

Tests. New TestToolChecksUseConversationSplit in python/packages/core/tests/core/test_local_eval.py. Each test covers all three checks:

• test_per_turn_tool_checks_ignore_calls_from_prior_turn: only turn 1 calls the tool. Turn 1 passes and turn 2 fails.
• test_full_split_counts_whole_trajectory: with FULL, calls from every turn count.
• test_single_turn_last_turn_unchanged
• test_conversation_without_user_message_counts_all_calls
• test_custom_split_decides_which_calls_count: a custom splitter that puts the call on the response side or on the query side.
• test_calls_in_the_current_turn_count_and_match_arguments: two calls with arguments in the current turn.

The per-turn, custom-split and current-turn tests fail on main and pass with this change. The others pass on both and guard the behavior that should not change.

After merging the latest main:

• test_local_eval.py: 107 passed
• Foundry eval tests: 144 passed
• ruff check and ruff format --check: clean
• pyright on _evaluation.py: 0 errors

Related Issue

Fixes #10091

#10186 also targets this issue. It was opened before the plan in #10091 was posted. This PR follows that plan: one shared helper, docstrings for ConversationSplit and EvalItem, and tests for all three checks, including custom splits.

AI Assistance

• No material AI assistance was used.
• This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: I used Claude (Anthropic) to help with the analysis, implementation, tests and this description. I reviewed the change and ran the tests and checks listed above myself.

Contribution Checklist

• The code builds clean without any errors or warnings
• All unit tests pass, and I have added new tests where possible
• The PR follows the Contribution Guidelines
• This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
• This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

I left the breaking-change box unticked. There is no API change, but multi-turn results change as described above. Evals are experimental, so I did not add the breaking change label myself. Please add it if you treat this as breaking.

[Copilot]

🟡 Changes recommended

The conversation factory shared by six tests remains a test-class static method instead of a conftest.py fixture or explicitly imported helper, contrary to the repository’s coding standard.

1 open issue

• Move shared conversation factory to a test helper or fixture · New

What changed in this PR

This PR fixes #10091 by aligning Python’s local tool checks with EvalItem’s conversation split, preventing earlier-turn tool calls from falsely satisfying later-turn checks.

Changes:

• Adds a shared helper that extracts function calls only from response-side messages.
• Documents split-aware checks and FULL for whole-trajectory evaluation.
• Adds regression coverage for per-turn isolation, custom splits, argument matching, and unchanged single-turn behavior.

File	Description
python/​packages/​core/​tests/​core/​test_local_eval.py	Adds six tests covering split-aware behavior across all three tool checks.
python/​packages/​core/​agent_framework/​_evaluation.py	Centralizes response-side function-call extraction and updates split and check docstrings.

_{Review effort: Balanced}

[Qiming Bao (14H034160212)]

Good point. Fixed in 5ebd65e: the conversation is now the two_turn_conversation fixture in tests/core/conftest.py, and the six tests take it through pytest injection, as CODING_STANDARD.md asks. test_local_eval.py passes, and ruff and pyright are clean.