Keep multimodal content inline in v1 rollouts - #2417
Draft
eligotts wants to merge 1 commit into
Draft
Conversation
This was referenced Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stack
Validation
Note
Keep multimodal content inline in v1 rollouts and track logical renderer tokens
Replaces separate
multi_modal_datapassing with inline raw multimodal content in the prompt. Removesmulti_modal_datafromTurnTokens,MessageNode,Branch, and serialization logic.renderer_prompt_idstoTurnTokensandrenderer_token_idstoMessageNodeto carry a separate logical renderer token stream distinct from canonical prompt tokens; both are excluded from serialization.TrainClient.get_responsenow detects raw multimodal prompts via_has_multimodal_contentand requiresrenderer.supports_raw_multimodal; passesraw_multimodaltogenerate()and setsprocess_multimodal=Falsein render and bridge paths.PendingTurn.previous_token_idsand_commit_turnuse the renderer token stream when available, and_commit_turnenforces exact prefix extension ofrenderer_prompt_idsagainst storedrenderer_token_ids, raisingValueErroron mismatch.supports_raw_multimodalnow raiseNotImplementedErrorfor multimodal prompts inget_response;_commit_turnraisesValueErrorwhen incomingrenderer_prompt_idsdiverge from the stored rollout prefix, affecting callers that previously relied on loose prefix matching.Macroscope summarized 6f5dccb.