Skip to content

Add raw multimodal content parts support - #133

Draft
eligotts wants to merge 1 commit into
mainfrom
feat/vllm-content-parts
Draft

Add raw multimodal content parts support#133
eligotts wants to merge 1 commit into
mainfrom
feat/vllm-content-parts

Conversation

@eligotts

@eligotts eligotts commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add an opt-in raw multimodal path that preserves renderer-owned text and tool formatting without running image processors
  • send ordered raw media as content_parts to the vLLM token-generation endpoint
  • use the effective prompt_token_ids returned by vLLM while retaining the logical renderer prompt for strict RL extension
  • preserve the existing eager multimodal path by default, including SFT callers

Stack

Validation

  • uv run pytest -q tests/test_client.py (25 passed)

Note

Add raw multimodal content parts support to client.generate and multimodal renderers

  • Adds raw_multimodal flag to client.generate; when true, a new _content_parts helper flattens image_url, audio_url, and video_url parts from messages into content_parts and sends them in the POST body
  • When raw multimodal is used, the engine response must include prompt_token_ids; the return now carries renderer_prompt_ids (renderer-side tokens) and prompt_ids (engine-provided tokens). A missing prompt_token_ids raises MalformedGenerateResponseError
  • Marks KimiK25Renderer, Qwen35Renderer, and Qwen3VLRenderer with supports_raw_multimodal = True and adds a process_multimodal flag to their render and bridge_to_next_turn methods; when false, placeholder tokens are still emitted but multimodal tensors and metadata are skipped
  • Risk: renderers called with process_multimodal=False no longer populate mm_hashes, mm_placeholders, or mm_items; any caller relying on those structures for tensor construction must handle the absent data

Macroscope summarized 26f4ea3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant