Repository navigation
Sync upstream ml-explore/mlx-swift-lm f5b0677 (real merge, 9 commits) - #92
Merged
Merged
Conversation
* mlx-swift 0.33.0 - use compile/@sendable implementation in mlx-swift
* Read Gemma 3 RoPE scaling from rope_parameters
transformers 4.57+ exports set rope_scaling to null and write the
scaling to rope_parameters, either keyed by layer type
({"full_attention": {...}, "sliding_attention": {...}}) or flat.
Gemma3TextConfiguration in MLXLLM and MLXVLM only read rope_scaling, so
for those checkpoints (e.g. TranslateGemma) the global-attention layers
ran without the x8 linear factor. Both now fall back to rope_parameters
when rope_scaling is absent or null.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BertConfiguration turned each id2label string key into an Int and built the dictionary with uniqueKeysWithValues. Keys such as "1" and "01" both become 1, and Dictionary(uniqueKeysWithValues:) traps. Read the keys one at a time. Skip a key that is not an integer. Throw DecodingError.dataCorrupted when an integer id is already in the map. A test covers colliding keys and a normal id2label. Co-authored-by: Krish Garg <181051779+krishhgg@users.noreply.github.com>
…l-explore#735) Keep image sentinels out of the repetition-penalty history VLM prompts carry negative placeholder IDs (FastVLM uses -200). The penalty processors loaded them into their token history and then indexed logits with them. loadPrompt now drops negative IDs, so sampled tokens are unaffected. To keep loading cheap on long prompts, it reads the prompt back to front, starting with `capacity` tokens and then doubling the window size, until it has `capacity` tokens. Each window is converted on the CPU stream to skip a GPU round trip. Loading no longer scales with prompt length (0.01 to 0.04 ms, down from about 2 ms per processor for a 1M-token prompt). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
) * 🐞 Fix attention masks sized from the absolute cache offset A capacity-limited cache evicts keys while the absolute offset keeps growing, so a mask sized from that offset describes more keys than the cache presents and attention rejects the shapes. Qwen3VL fixed its own case in ml-explore#706. createAttentionMask(h:cache:) still built the array from the raw offset, and Qwen3VLMoE used that and then sliced it again. Both cache-list overloads now delegate to the cache makeMask, and Qwen3VLMoE builds one shared mask from the retained window, as ml-explore#706 does. M-RoPE positions stay absolute. Qwen3.5-VL no longer slices its mask to an offset-derived length. That slice ended past the mask and MLX clamps it, so the results do not change. The Qwen3 embedder folds key padding across the same retained width. Regressions cover warm continuation after rotation for Qwen3VLMoE and Qwen3.5-VL, the padding fold, and the mask width of the shared helper. * Gemma2: suppress boolean attention masks like mlx-lm Gemma2 attention added its mask to the scores unconditionally, so a boolean mask boosted the kept scores instead of hiding the masked keys and multi-token prefill never matched token-by-token decode. Route it through applyAttentionMask (bool masks suppress, float masks add), matching mlx-lm gemma2.py, and expose that helper to model code. * Gemma2: build the attention mask from the first layer cache
…xplore#703) attentionWithCacheUpdate sent every query length to quantizedScaledDotProductAttention when the cache is quantized. That path materializes [B, H, L, N] scores through quantized matmuls, which is the right trade at decode (L = 1 reads a quarter of the key bytes) but loses to the fused kernel once the query block is long enough to be compute-bound. For L >= 64 and head dims the fused kernel handles on every GPU generation (64, 72, 80, 96, 128), dequantize the returned keys and values once and call MLXFast.scaledDotProductAttention. Head dims 192 and 256 stay on the quantized route because MLX runs them unfused outside NAX, where the quantized matmuls are within ~1.1x. MLA-style caches with narrower values than queries also stay on the quantized route. Measured on M4 Pro with Qwen3-14B-4bit, kvBits 4, step 512: - 16k prefill from an empty cache: 171 -> 199 tok/s (fp16 cache: 200), peak memory 11.3 -> 9.8 GB - 2k turn over 8k history: 170 -> 195 tok/s (8-bit cache: 173 -> 182) - decode at 8k unchanged (quantized route still used at L = 1) - SDPA microbenchmark, D = 128, L = 512: quantized path 1.45-1.7x slower than fused at N = 2.5k..33k, and 135 MB..1.4 GB of scores vs 24..99 MB
…re#563) Add reasoning support: Gemma 4 channels and Generation.reasoning * Support Gemma 4's labeled reasoning channels Gemma 4 delimits thoughts with special tokens (<|channel>, <channel|>) and a role label after the opener (<|channel>thought\n ... <channel|>). A delimiter pair cannot express that. Add an optional ReasoningConfig.channel (ReasoningChannel) and a label state in ReasoningEventEmitter. nil keeps the pair protocol unchanged, so Qwen3 and DeepSeek-R1 are unaffected. - The label is consumed as metadata; unknown labels route to reasoning. - Hold-back covers the longest watched delimiter. - An unterminated label settles into a visible body. - A stray end delimiter with no opener is swallowed. - "content" is listed as a response label defensively, not from observation. The four Gemma 4 model types declare the protocol through ChatConventionsProviding (ml-explore#502). Also add a token-id overload of promptEndsInsideReasoning. * Resolve the think-then-call gate through the template default The constrained-generation path read the reasoning level as `thinkingEnabled(for:) != false` for the gate, but rendered the prompt with `?? defaultOn`. For Gemma 4 (`defaultOn: false`) these disagree: the gate opens a thinking phase, the template prefills an empty closed channel, and a `.required` tool call never arrives. Add `thinkingEnabled(for:defaultOn:)` and use it for both. No change for `defaultOn: true` models. The tests need a macOS 27 host to run. * Expose reasoning as Generation.reasoning Nothing on the standard path read ModelConfiguration.reasoningConfig, so raw reasoning reached Generation.chunk with delimiters attached and ChatSession replayed it into later prompts. - StandardTokenStreamDecoder takes an optional ReasoningEventEmitter, applied after the stop-string filter and before the tool-call parser, so a tool call in a scratchpad cannot fire. finish() drains the scanner first. - Add Generation.reasoning(String) and update all in-repo switches. This is a source-breaking change for downstream exhaustive switches. - Framed protocols (Harmony, Onyx) keep decoding their own reasoning. - generate() computes priming from the prompt (needed for DeepSeek-R1, which prefills
…#712) Support Nemotron 3.5 (Lightning, Bolt) text models - Read the layer stack from layers_block_type or block_configs when hybrid_override_pattern is absent; omit num_hidden_layers if the config does. - Add moe_latent_size: routed experts run in a latent space between fc1_latent_proj and fc2_latent_proj, as in mlx-lm. - Add per-layer block_configs overrides (expert count, expert size, experts per token) and register nemotron_h_puzzle (Bolt). - Skip MTP head weights (mtp.*) at load. - Declare the Nemotron-H reasoning config (enable_thinking), which the Nemotron 3 and 3.5 chat templates use; previously the flag was ignored. Tested: Nemotron-3.5-Lightning-30B-A3B-4bit and Bolt-74B-A9B-MLX-3bit load and generate; Lightning with thinking off now answers directly. AI-assisted (Claude Code); reviewed against mlx-lm.
* Expose Qwen 3.5 text geometry read-only Applications that plan memory for a Qwen 3.5 target and its MTP drafter need the text dimensions before running either. Make the dimension fields of Qwen35TextConfiguration and the models' configuration readable, as the VLM's TextConfiguration already is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Real merge of 9 upstream commits. Conflicts resolved by hand: - Package.swift: keep the ../mlx-swift path dependency. - CompiledTrace.swift / CompiledTraceTests.swift: take upstream (compile shadows removed). - AttentionUtils.swift: use quantizedCacheAttention, keep the fork's mask clamp. - Evaluate.swift: keep the fork's .chunk(text, tokenId:) and DualModelMTP, add upstream's .reasoning event and prompt-priming read; pass state/components/tools through generate(draftModel:). - Gemma4.swift: keep fork audio parts, add addReasoning. - Tests and docs: adapt to .chunk(text, _) plus .reasoning. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ict resolution Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The job still checked out the temporary branch fix/mlx-swift-lm-throwing-newcache. That fix (SwiftLM#167) is merged, and the branch is now 167 commits behind main, so it lacks later SwiftLM changes such as the Generation `.reasoning` tolerance (SwiftLM#252) and fails to build against this sync. downstream_integration.yml already uses main. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Real merge of
ml-explore/mlx-swift-lmmain (f5b0677, 9 commits sinceefcedcd) into this fork. Merge it with a merge commit (not squash/rebase) so upstream stays an ancestor and downstream submodule pins stay valid.Upstream: ml-explore/mlx-swift-lm@efcedcd...f5b0677
The sync bot found 11 conflicting files and pushed nothing; they were resolved by hand:
Package.swift: keep the../mlx-swiftpath dependency (not upstream's0.33.0requirement).CompiledTrace.swift,CompiledTraceTests.swift: take upstream (the@Sendablecompileshadows are gone upstream).AttentionUtils.swift: call upstream'squantizedCacheAttention, keep the fork's mask clamp.Evaluate.swift: keep the fork's.chunk(text, tokenId:)andDualModelMTP; add upstream's.reasoningevent and prompt-priming read.generate(draftModel:)now also passesstate,components,tools,stopStringsand the reasoning config to the speculative iterator (the fork's copy dropped them).Gemma4.swift: keep the fork-only audio parts, add upstream'saddReasoning..chunk(text, _)plus.reasoning.Silent-revert check: for every file upstream changed, the remaining difference from upstream matches the fork's earlier difference, apart from the intentional items above.
Tests (local, merged tree)
swift build --build-tests:EXIT=0.TokenLoopHandlerTests,CompiledTraceTests,ChatSessionTests(98 XCTest) andGemma4Tests(30 Swift Testing): passed,EXIT=0.Needs attention after merge
Generationgains a.reasoningcase. SwiftLM'sSources/SwiftLM/Server.swiftandCalibrator.swiftswitch overGeneration, so bumping the SwiftLM submodule to this merge needs a change there.Depends on SharpAI/mlx-swift#30 (merged), which raises the manifest to tools-version 6.1 for upstream's JACCL trait.
🤖 Generated with Claude Code