Python: AG-UI chat client drops multimodal message content

[CoralGarden52]

Description

The Python AGUIChatClient drops non-text content when serializing Agent Framework messages into AG-UI request messages.

In python/packages/ag-ui/agent_framework_ag_ui/_message_adapters.py, agent_framework_messages_to_agui currently handles text, function calls, and function results, but Content.from_uri() and Content.from_data() content is skipped. _client.py then sends the already-truncated message list to the HTTP service.

Reproduction

On the current main branch (26832bed1fa192febbb0ceec16d5a2900a39eb27), sending a normal user message through AGUIChatClient:

Message(
    role="user",
    message_id="user-1",
    contents=[
        Content.from_text("describe this"),
        Content.from_uri("https://example.com/cat.png", media_type="image/png"),
        Content.from_data(b"abc", media_type="image/png"),
    ],
)

produces this outbound request message:

{"id": "user-1", "role": "user", "content": "describe this"}

Both media items are absent. The same loss occurs through the public Agent -> AGUIChatClient -> HTTP path, before the request reaches the server. This was reproduced with an httpx.MockTransport, so no external model or service is involved.

Expected behavior

The client should preserve text and supported media content in their original order using AG-UI InputContent parts, for example:

{
  "role": "user",
  "content": [
    {"type": "text", "text": "describe this"},
    {
      "type": "image",
      "source": {
        "type": "url",
        "value": "https://example.com/cat.png",
        "mimeType": "image/png"
      }
    },
    {
      "type": "image",
      "source": {
        "type": "data",
        "value": "YWJj",
        "mimeType": "image/png"
      }
    }
  ]
}

AG-UI documents ordered multimodal input content for images, audio, video, and documents: https://docs.ag-ui.com/sdk/js/core/multimodal-inputs

Impact

Any Python Agent using AGUIChatClient to forward a multimodal user message to a stateless AG-UI server loses the media before the server can process it. This affects the regular client request path, not an exceptional or low-probability condition.

Proposed fix

• Convert Agent Framework URI and data content to AG-UI InputContent parts.
• Preserve the order of text and media parts.
• Keep the existing text-only and tool-call behavior backward compatible.
• Restrict InputContent lists to user messages because the AG-UI AssistantMessage content shape remains text-only.
• Add adapter and request-level regression tests covering URL, base64 data, and mixed text/media input.

This concerns the outbound Agent Framework-to-AG-UI conversion. It is distinct from #8083, which addresses multimodal content during orchestrator handoff.

[github-actions]

🤖 Automated triage reproduction notes (agent-authored — trust but verify)

Agent analysis

The bug occurs in python/packages/ag-ui/agent_framework_ag_ui/_message_adapters.py::agent_framework_messages_to_agui at lines 998-1059 when a Message contains Content.from_uri or Content.from_data alongside text. Minimal repro: convert one user Message containing text, an image URI, and image bytes, then observe that the result contains only the text string; _client.py passes this truncated value to post_run unchanged.

• Failing test: python/packages/ag-ui/tests/ag_ui/test_message_adapters.py::test_agent_framework_to_agui_preserves_multimodal_content
• Files examined: python/packages/ag-ui/agent_framework_ag_ui/_message_adapters.py, python/packages/ag-ui/agent_framework_ag_ui/_client.py, python/packages/ag-ui/tests/ag_ui/test_message_adapters.py, python/packages/ag-ui/pyproject.toml
• Tests run: test_agent_framework_to_agui_preserves_multimodal_content (failed), test_agent_framework_to_agui_basic (passed)
• Current version: 1.2.2

[manjunathshiva]

I can reproduce this by inspection: agent_framework_messages_to_agui (_message_adapters.py:1002-1017) branches on text, function_call and function_result only, with no fallback, so uri and data contents are dropped silently rather than raising or degrading. The outbound message is truncated before _client.py posts it, as described.

Worth noting for whoever picks this up: the pinned ag-ui-protocol==0.1.19 already exports InputContent, TextInputContent, ImageInputContent, InputContentUrlSource and InputContentDataSource, so the shape proposed here is achievable without touching the dependency floor.

I'd be glad to take this, but I want to ask first rather than assume. The function this fix lands in also serializes function_call and function_result, which puts the file on the AG-UI transport serialization path covered by the Contribution ownership clause in docs/specs/004-python-function-calling-loop.md — even though the change itself would only add uri/data handling and wouldn't touch call/result behaviour. Is that inside or outside the clause for your purposes?

Eduard van Valkenburg (@eavanvalkenburg), this is the same boundary question moonbox3 tagged you on in #8078, so a single answer would cover both.

[vedantsonkar]

Hi Eduard van Valkenburg (@eavanvalkenburg), I’d like to contribute a fix for this issue as my first contribution here. Since it’s already assigned to you, is this available for me to work on?
My proposed approach is TDD: first add failing outbound adapter tests and a request-level regression test using httpx.MockTransport, then add serialization for ordered multimodal user content. I’ll preserve existing text-only and tool-call/result behavior and run the relevant regression, coverage, formatting, and typing checks.
I’m planning to keep this focused on outbound user-message serialization. Are there any ownership or function-calling-loop validation requirements you’d like me to account for before implementation?

[vedantsonkar]

Following up after reviewing #8116: I’d like to propose an alternative that also preserves non-base64 data URIs and tests the public Agent.run() path in streaming and non-streaming modes.
In an isolated execution of #8116’s helper, data:text/plain,hello%20world becomes an inline data source containing hello%20world, although that source requires base64. Preserving the complete URI as a URL source would avoid this.
Eduard van Valkenburg (@eavanvalkenburg), would you welcome this alternative, and can you confirm the contribution ownership boundary and required validation scope for the shared serializer? The local environment is ready, all 128 existing adapter tests pass, and I’ve reproduced #8114 against unchanged code.

[vedantsonkar]

I have created a draft PR with my understanding of the issue and the fix if this fits what you're looking for then great or else Im open to any changes you might suggest for the same.

#8117

[CoralGarden52]

Thanks Vedant Sonkar (@vedantsonkar) for pointing out the non-base64 data: URI case. I’ve added commit 6515c9612 to PR #8116 to preserve non-base64 data URIs as complete AG-UI URL sources, while keeping base64 data URIs as inline data sources, along with a regression test.

Eduard van Valkenburg (@eavanvalkenburg), I’d appreciate it if you could review the updated PR when you have a chance. If you have any concerns or requested changes, I’m happy to revise it. I opened #8114 and am the original issue author; if appropriate, could you assign the issue to me?