Python: fix(orchestrations): preserve multimodal content during agent handoff

[Manideep Malyala (manideep-malyala)]

Motivation & Context

Multi-modal workflows (like GroupChat and HandoffAgentExecutor) were silently dropping image and file attachments during agent handoffs. This prevented downstream Vision models from receiving the necessary context to complete their tasks. This fix ensures that multi-modal data is properly preserved across handoffs.

Description & Review Guide

• What are the major changes?
  I updated the filtering logic in clean_conversation_for_handoff. The previous logic aggressively dropped all non-text content. I replaced this with a safer allowlist (text, data, uri, hosted_file, etc.) that preserves multi-modal context while still successfully stripping out tool-control payloads. I updated the test_handoff.py tests to cover this behavior, and added a regression test to test_sequential.py to explicitly confirm SequentialBuilder natively preserves this data.
• What is the impact of these changes?
  Vision models will now correctly receive image context and file attachments from prior steps during sequential handoffs.
• What do you want reviewers to focus on?
  Please take a quick look at the ALLOWED_CONTENT_TYPES set to ensure you agree with the types I allowed through. Thank you for your time and review!

Related Issue

Fixes #7822

Contribution Checklist

• The code builds clean without any errors or warnings
• All unit tests pass, and I have added new tests where possible
• The PR follows the Contribution Guidelines
• This PR is linked to an issue and there is no other open PR for this issue
• This is not a breaking change.

[Copilot]

Pull request overview

Preserves multimodal content during orchestration handoffs, while also adding an unrelated AG-UI snapshot option.

Changes:

• Adds a handoff content allowlist.
• Tests URI preservation and tool-content removal.
• Adds configurable AG-UI snapshot suppression.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
python/packages/orchestrations/agent_framework_orchestrations/_orchestrator_helpers.py	Retains allowlisted content during handoffs.
python/packages/orchestrations/tests/test_handoff.py	Tests URI retention.
python/packages/ag-ui/agent_framework_ag_ui/_agent.py	Adds snapshot emission configuration.
python/packages/ag-ui/agent_framework_ag_ui/_agent_run.py	Applies snapshot suppression.
python/packages/ag-ui/tests/ag_ui/test_run.py	Tests snapshot suppression.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

python/packages/orchestrations/agent_framework_orchestrations/_orchestrator_helpers.py:31

• The updated return description still conflicts with the function summary and bullets above, which say the helper is text-only and drops every non-text item. Update the complete contract to describe retained multimodal content and removed tool-control payloads.

        Cleaned conversation history with semantic multimodal content preserved, suitable for handoff routing

[Evan Mattson (moonbox3)]

Please re-open when failing CI/CD checks are addressed.

[Evan Mattson (moonbox3)]

One item for you to look at as well.

[Evan Mattson (moonbox3)]

What happens when an agent returns data or uri content? This keeps the assistant role, so the OpenAI clients serialize those parts as input_file or input_image, which are input-only content and make the next handoff request fail. Could we retain multimodal parts only from user messages here, while preserving text on the existing roles?

[Manideep Malyala (manideep-malyala)]

Evan Mattson (@moonbox3) Thanks for the feedback! I've updated the handoff logic to only retain multimodal content for user messages, keeping assistant messages text-only to avoid the OpenAI input serialization issue. Reopening for review!