Skip to content

[Feature] Support image-text multimodal input for training #1

Description

@slepshep106

Motivation

Training in AngelSpec is text-only today, so vision-language targets (Qwen3.5,
Qwen3-VL, Hy3-VL) can't be used. We'd like to train DFlash/DSpark drafters for
VLM targets on image-text multimodal input.

Some of the plumbing already exists (media extraction in the dataset, an
image_placeholder in templates, enable_multimodal in the inference config,
multimodal_inputs passed to vLLM). The gap is on the draft-training side,
where block-parallel drafters have no mRoPE support.

Proposed Solution

Support image-text records end to end: accept images in the training data, carry
the target's mRoPE positions through to the draft training forward, and make the
block-parallel draft RoPE mRoPE-aware.

The drafter core shouldn't need to change — KV injection, anchor sampling, and
the losses are all modality-agnostic — and since DFlash, DSpark, and DFly share
a backbone, one implementation should cover all three.

Happy to help implement or test.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions