Experimental Qwen3.6 vision DFlash - #1
Merged
Conversation
Load mmproj alongside draft, parse image_url in HTTP server, inject vision embeddings in Qwen35 prefill, and keep speculative decode for text-only requests while forcing AR on multimodal turns.
Agent payloads (38+ tools) tokenize into a single large mtmd text chunk. Prefilling it in one graph build reserved ~8.7GB VRAM and failed on dual-GPU deploys. Sub-chunk text segments with DFLASH27B_PREFILL_UBATCH (same as text prefill) and release scratch buffers before multimodal prefill starts.
Wire mmproj through the layer-split adapter and backend so dflash_server can run vision prefill across sharded GPUs with mRoPE/bidirectional masks.
…per. Forward-declare LayerSplitAttnPrefillOpts via layer_split_forward.h and use the shared build_bidirectional_mask from attn_masks.h in qwen35_backend.
GENERATE_MULTIMODAL checked ModelBackend::supports_multimodal(), but LayerSplitBackend never delegated to the adapter, so vision failed with vision_not_configured even after mmproj loaded successfully.
vision_ only exists when mmproj is enabled; ggml-only builds must not reference it from the inline override.
Layer-split and Qwen35 overrides need a base virtual; without it MMPROJ builds fail with 'marked override, but does not override'.
Nuked88
marked this pull request as ready for review
August 1, 2026 20:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
DFLASH_VISION_DFLASH=1;Provenance
The vision implementation is taken directly from PR
Luce-Org#571 by David Roth, using head commit
122393626cf7bc488e6a27d2ea7069ff15316312fromdavidmroth/lucebox-hub:feat/vision-native-mmproj.The DFlash opt-in and final-prefill-row corrections are the additional changes
in this fork.
Validation
DFLASH27B_MMPROJ=ONon sm_86;Q4_0 K/V cache, 61,440 context, and prefill chunk/micro-batch 256;
git diff --checkpassed.Limitations
Image acceptance was often only 10-17%, and one deterministic comparison
diverged semantically from AR. Vision DFlash remains disabled by default and
is intended for evaluation. Layer-split/multi-GPU vision remains AR-only.