Python: preserve multimodal messages in AG-UI chat client requests - #8116
Python: preserve multimodal messages in AG-UI chat client requests#8116CoralGarden52 wants to merge 6 commits into
Conversation
agent-framework-automation
Bot
added
the
python
|
@microsoft-github-policy-service agree |
Evan Mattson (moonbox3)
added
the
ag-ui
|
One file conflict, too. |
884cfd2 to
e562c95
Compare
agent-framework-automation
Bot
added
the
documentation
|
Thanks for the heads-up, Evan Mattson (@moonbox3). I rebased the branch onto the current upstream main and resolved the conflict. The PR should now be conflict-free. |
| @@ -293,22 +294,34 @@ def _extract_state_from_messages(self, messages: Sequence[Message]) -> tuple[lis | |||
|
|
|||
| last_message = messages[-1] | |||
There was a problem hiding this comment.
Following up on the state-marker change: could we strip every marked carrier from messages_to_send and use the most recent carrier as state, instead of inspecting only messages[-1]? Once a carrier is retained in client-controlled history, the next turn makes it non-final, so _extract_state_from_messages() returns it and _encode_agui_segment() serializes the state as a model-visible document. I reproduced {"secret":"SENSITIVE"} in the next request's messages while request state was None.
There was a problem hiding this comment.
I’ve addressed this in a follow-up. State extraction now scans the full client-controlled history, removes every dedicated explicitly marked carrier, and uses the most recent successfully decoded carrier as the request state. Historical carriers therefore cannot reach the AG-UI converter as model-visible documents. I also added a regression test for a carrier retained from an earlier turn.
| if (content.additional_properties or {}).get(STATE_CARRIER_KEY) is not True: | ||
| return list(messages), None |
There was a problem hiding this comment.
Have we considered a compatibility path for existing callers that send a final base64 application/json content as request state? The new STATE_CARRIER_KEY guard makes every unmarked legacy carrier bypass extraction, and the multimodal path sends its payload as a model-visible document instead, silently removing request state in a Production/Stable package. Could we preserve legacy extraction behind an option or deprecation window before requiring state_carrier()?
There was a problem hiding this comment.
Evan Mattson (@moonbox3) I’ve added a compatibility path in a follow-up. Callers can set allow_legacy_state_carrier=True to retain the existing final base64 application/json extraction behavior; it emits a DeprecationWarning and remains client-only. The option defaults to False, so unmarked JSON documents are treated as model-visible documents by default. I also added a request-level regression test. If you prefer preserving legacy behavior by default during a deprecation window, I can adjust that policy.
| if allow_legacy_state_carrier and messages and not _is_state_carrier_message(messages[-1]): | ||
| legacy_state = _extract_legacy_json_state(messages[-1]) | ||
| if legacy_state is not None: | ||
| messages_to_send.pop() |
There was a problem hiding this comment.
Following up on the earlier mixed-JSON fix: could legacy extraction keep the dedicated state-only boundary? With allow_legacy_state_carrier=True, _extract_legacy_json_state() accepts JSON inside a final text-plus-document message, then messages_to_send.pop() drops the entire prompt and sends the document as state. Could the fallback require exactly one unmarked JSON content so opting into migration does not reintroduce the message loss fixed earlier?
There was a problem hiding this comment.
Evan Mattson (@moonbox3) I’ve addressed this in a follow-up. The legacy fallback now requires the final message to contain exactly one unmarked application/json content, so mixed text-plus-document messages remain intact and are sent through the AG-UI converter. The single-content legacy convention remains available with allow_legacy_state_carrier=True, including the deprecation warning. A request-level regression test covers the mixed-message case.
Summary
AGUIChatClientserializes user messages for AG-UI.InputContentsource fields (source.type,source.value, andmimeType) with MIME-aware image/audio/video/document part types.httpx.MockTransportrequest JSON regression test.Validation
PYTHONPATH="$PWD/python/packages/core:$PWD/python/packages/ag-ui:$PWD/python/packages/orchestrations" /home/nyc/agent-framework/python/.venv/bin/pytest python/packages/ag-ui/tests/ag_ui -q— 1100 passed, 1 skipped.uvx --from ruff==0.16.4 ruff check ...— passed.uvx --from ruff==0.16.4 ruff format --check ...— 3 files already formatted.ag_ui.core.UserMessage; the captured HTTP JSON contains the ordered URL and data parts.Related issue
Fixes #8114
This is the outbound Agent Framework-to-AG-UI conversion path. It is distinct from #8083, which addresses multimodal content during orchestrator handoff.