Skip to content

Keep multimodal content inline in v1 rollouts - #2417

Draft
eligotts wants to merge 1 commit into
mainfrom
feat/v1-inline-mm-vllm
Draft

Keep multimodal content inline in v1 rollouts#2417
eligotts wants to merge 1 commit into
mainfrom
feat/v1-inline-mm-vllm

Conversation

@eligotts

@eligotts eligotts commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Summary

  • keep base64 image content in canonical v1 message nodes instead of carrying processed multimodal sidecars
  • use renderer logical prompt tokens only for bridge extension, then store the effective expanded prompt tail returned by vLLM
  • remove multimodal kwargs and metadata bookkeeping from graph, trace, and training client state
  • preserve strict prefix validation and response masking across multimodal tool turns

Stack

Validation

  • uv run pytest -q tests/v1/test_graph.py (7 passed)
  • uv run ruff check on all touched files

Note

Keep multimodal content inline in v1 rollouts and track logical renderer tokens

Replaces separate multi_modal_data passing with inline raw multimodal content in the prompt. Removes multi_modal_data from TurnTokens, MessageNode, Branch, and serialization logic.

  • Adds renderer_prompt_ids to TurnTokens and renderer_token_ids to MessageNode to carry a separate logical renderer token stream distinct from canonical prompt tokens; both are excluded from serialization.
  • TrainClient.get_response now detects raw multimodal prompts via _has_multimodal_content and requires renderer.supports_raw_multimodal; passes raw_multimodal to generate() and sets process_multimodal=False in render and bridge paths.
  • PendingTurn.previous_token_ids and _commit_turn use the renderer token stream when available, and _commit_turn enforces exact prefix extension of renderer_prompt_ids against stored renderer_token_ids, raising ValueError on mismatch.
  • Risk: renderers that do not implement supports_raw_multimodal now raise NotImplementedError for multimodal prompts in get_response; _commit_turn raises ValueError when incoming renderer_prompt_ids diverge from the stored rollout prefix, affecting callers that previously relied on loose prefix matching.

Macroscope summarized 6f5dccb.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant