ggml: add DeepSeek V4 hyperconnection + KV ops (CPU) - #23122
Conversation
github-actions
Bot
added
testing
This comment was marked as resolved.
This comment was marked as resolved.
|
@cchuter is this complete for CPU inference? Where can I download gguf to test? |
|
@segmond No, these are just the backend ops which are needed for the model support in CPU version. @cchuter actually when adding new ops for models, we tend to prefer PRs with the model support added in as well, as otherwise there's no simple way to test whether the operation implementation is correct. |
Five new ggml ops for DeepSeek-V4-Flash with CPU reference implementations and test-backend-ops coverage: DSV4_HC_SPLIT_SINKHORN, DSV4_HC_WEIGHTED_SUM, DSV4_HC_EXPAND, DSV4_FP8_KV_QUANTIZE, DSV4_ROPE_TAIL. CPU is the reference backend. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DeepSeek-V4-Flash model: graph (src/models/deepseek4.cpp), arch / hparams / model-loader wiring, the dsv4_* compressed-KV extension to llama_memory_hybrid_iswa, GGUF conversion (conversion/deepseek.py + constants/writer keys), and the V4 chat template. Standard build_attn_mha attention path; no DeepSeek Sparse Attention. Exercises the DSV4 ops from the preceding commit so they are testable end-to-end on CPU. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
97704a6 to
d99045f
Compare
|
@segmond, ggufs are here: https://huggingface.co/teamblobfish/DeepSeek-V4-Flash-GGUF @pwilkin, I've update the PR to add a full cpu end to end test. THe model loads and answers the test questions. I was a little over aggressive in splitting up the PRs. Here's the plan: This PR 1: base ggml and cpu support PR's 4 and 5 are not done yet and might prove challenging. They will also require new gguf's (which I will make). THe full working code base of PR 1-3 is here: https://github.com/cchuter/llama.cpp/tree/v4-clean-base Before all those PRs are merged I don't expect a performant DeepSeek4, but I do expect a working one at all stages. Right now you should peak around 20-30 t/s generation on GPUs |
github-actions
Bot
added
model
ngxson
left a comment
ngxson
left a comment
There was a problem hiding this comment.
no good feeling about this PR, I assume backend maintainers won't happy to accept this as-is, but let's wait to see how other maintainers say about this.
it seems like most ops added this this PR can be implemented in another way
| struct ggml_context * ctx, | ||
| struct ggml_tensor * x, | ||
| struct ggml_tensor * weights); | ||
|
|
There was a problem hiding this comment.
why not using ggml_mul, ggml_add and ggml_sum_row?
|
|
||
| // DeepSeek V4 hyperconnection expand helper. | ||
| // Computes post * block_out + comb^T @ residual for each token. | ||
| GGML_API struct ggml_tensor * ggml_dsv4_hc_expand( |
There was a problem hiding this comment.
whay about ggml_mul, ggml_add and ggml_mul_mat?
| // DeepSeek V4 partial RoPE helper. | ||
| // Leaves the non-RoPE prefix unchanged and applies RoPE to the tail, | ||
| // matching ggml_concat(prefix, ggml_rope_ext(tail)). | ||
| GGML_API struct ggml_tensor * ggml_dsv4_rope_tail( |
There was a problem hiding this comment.
you should have skipped reading tips and tricks from https://github.com/ggml-org/llama.cpp/blob/master/docs/development/HOWTO-add-model.md
|
Did you just copied straight from antirez's version? https://github.com/antirez/llama.cpp-deepseek-v4-flash/blob/main/ggml/include/ggml.h Please note that dishonest about code's origin will resulting in being banned from the project. |
|
@ngxson you're right, thanks for pushing on it. This is derived from antirez's llama.cpp-deepseek-v4-flash (which builds on fairydreaming's and the ggml/llama.cpp lineage). The V4 ops and model are his; my part is CUDA/multi-GPU, a DSA-free rebase onto current master, and GGUF packaging. I should have credited that in the PR from the start and didn't. That's on me. I have contacted @antirez and @fairydreaming to let them know I'm building on top of their work and crediting. They've both moved on to other things. |
|
Happy to restructure, or close, if you'd rather. I'm not aware of the history of this work |
|
Since your code contains many lines that are directly copied from antirez's work, we require explicit agreement from the original author to proceed. |
|
Hi, I agree with taking the code I developed and making it part of llama.cpp or any other compatibly-licensed project. I don't ask for any credit. Have fun hacking LLMs! :) |
|
I checked out your v4-clean-base branch and gave it a go, still needs more work. I couldn't build it because GGML_OP_COUNT wasn't updated in ggml-rpc.h. had to bump it to 101 from 96 to build. My guess is you didn't build with RPC. On multi CUDA GPU setup it fails, abords fro ggml_backend_sched_split_graph(), I had to bump up GGML_SCHED_MAX_BACKEND and GGML_SCHED_MAX_SPLIT_INPUTS to stop the crash. Once I go past 1 GPU I get the infamous <<<<<<<<<<< for output. With 1 GPU it works. 3090s btw. Nevermind the bad output, with the model completely loaded in memory vs 1 24gb GPU, performance isn't that much improved, all in memory TG 14.81tk/s, 1 GPU 11.70tk/s for about 4500 tokens generated. PP with multiGPU is 3x 177 vs 65. Performance leaves lots to be desired, but for now let's focus on correctness. To give an example, I get 800 tk/s on PP and 52 tk/s for TG for Qwen3.5-122 which is larger loaded all in memory. From reading the deepseek paper, it's suppose to require less compute.... I have tried every fork I could find, and these 2 in these order have worked best for me on multi cuda setup. mix of 3090s/3080s. https://github.com/nonzod/llama.cpp-deepseek-v4-flash-spark |
This comment was marked as spam.
This comment was marked as spam.
|
@segmond thanks, that's very helpful. I'll hold off on the other GPU PRs until I've confirmed perf and correctness. |
|
any chance this would also support #22436 |
|
No news in 2 weeks? |
|
Things are happening. |
|
Sorry, I've refocused and I've been working on adding multi-gpu support to @antirez 's project. Anyone is free to use my fork and previous work to further this along- no credit needed. If I get good results I'll see if I can get it working in llama.cpp as well. |
|
can ds4 run on multiple gpu on different machines? |
|
@vanmilleru I think antirez ds4 project just added such feature, but it seems to be only for the DS4 PRO Q4 GGUF to run across two 512 GB M3 Ultra Mac Studios. |
@vanmilleru No idea, but there's #24162, so maybe you can check if it works on multiple machines by using llama.cpp RPC. |
CPU implementations and tests for DSV4_HC_SPLIT_SINKHORN, DSV4_HC_WEIGHTED_SUM, DSV4_HC_EXPAND, DSV4_FP8_KV_QUANTIZE and DSV4_ROPE_TAIL, cherry-picked from the open upstream PR by Chris Chuter (ops commit only; the PR's model-architecture commit predates the merged V4 support and is not used). Conflict resolution: op name tables and GGML_OP_COUNT (98 -> 103) for current master. Co-authored-by: Chris Chuter <cchuter@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wire build_hc_weighted_sum and build_hc_post to the fused DSV4_HC_WEIGHTED_SUM and DSV4_HC_EXPAND ops (from PR ggml-org#23122 by Chris Chuter) and add Metal kernels for both. The decomposed hyperconnection mixing was ~36 full-tensor mul/add/concat nodes per layer; profiling showed the decode graph at ~18K nodes/token with these chains as the largest contributor. M3 Max, DeepSeek-V4-Flash UD-Q2_K_XL, fa=0 b=3072 ub=1024, paired same-window runs (alternating with the previous commit's build to control thermal drift): - tg32: 11.60 -> 16.11 and 10.53 -> 14.33 (+36-39% decode) - pp3072: unchanged within noise - test-backend-ops MTL0: DSV4_HC_WEIGHTED_SUM and DSV4_HC_EXPAND all cases pass Note: kernel fusion changes fp rounding (fma contraction), so greedy sampling diverges from the unfused build after some tokens; outputs remain coherent and the ops match their CPU references within test tolerances. Disclosure: kernels and wiring were written by an AI assistant (Claude) under human direction, then verified with the tests and measurements above. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CUDA counterparts of the Metal kernels added in d934fda, for the fused hyper-connection ops from PR ggml-org#23122 (CPU reference by Chris Chuter). One thread per output element, byte-stride addressing mirrors the CPU reference loops exactly. Motivation: mainline V4 decode on CUDA measures ~33 t/s on an RTX PRO 6000 (284B-A13B UD-Q2_K_XL) with the decomposed ~18K-node decode graph; the same fusion stack took Metal decode from 8.6 to ~16 t/s. Verification and paired benches to follow on the CUDA box. Disclosure: written by an AI assistant (Claude) under human direction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@cchuter I tested the ops commit from this PR against the now-merged V4 implementation (#24162). Since then #25585 has landed fused hyper-connection ops for CPU and CUDA with a different op split (pre/comb/post, sinkhorn iterations fused inside the comb op), so mainline now covers the CUDA side of this. Posting my numbers anyway: they validate the fusion design cross-backend, and there is one place where this work still matters directly, which is Metal. What I did: cherry-picked only your ops commit (9811b19) onto master (pre-#25585), wired mainline's Results, DeepSeek-V4-Flash UD-Q2_K_XL, paired same-window runs to control thermal drift:
The same fusion wins on both backends, consistent with the gains #25585 reports on CUDA. The bottleneck it removes is graph dispatch, not anything backend-specific. The remaining gap: #25585's fused path auto-probes per backend and there are no Metal kernels for its ops, so Apple Silicon still runs the decomposed graph on current master. My Metal kernels above demonstrate the win on that backend. I can adapt them to the merged pre/comb/post interface and open a PR for the Metal side if maintainers are open to it. Wiring and kernels are on satindergrewal:deepseek-v4-flash, your authorship and co-author trailer preserved on the ops commit. Note on numerics: the fusion changes FP rounding (FMA), so greedy outputs drift but stay coherent; all ops match CPU references in test-backend-ops. Full disclosure: the wiring and the Metal/CUDA kernels are AI-generated under my direction and verified as above. |
|
@satindergrewal Great work! I've moved on to ds4, but feel free to add that code to llama.cpp to improve performace. |
|
This patch allows running DS4F on Intel B70. |
10 participants
This is the first of at 4 or more PRs to support Deepseek V4 #22319 - the full branch with complete support for DeepSeek V4 is here: https://github.com/cchuter/llama.cpp/tree/feat/v4-port-cuda (this branch is where the PRs will be carved from and will undoubtedly change with reviewer responses to this and following PRs)
Overview
I used guidance from @CISC in the issue to break up the PR. This is the first to add the basic ggml support. It adds the five DeepSeek-V4-Flash-specific ggml ops with CPU reference implementations and test-backend-ops coverage:
THis is CPU only
GGML_OP_COUNT goes 96 -> 101 (5 new ops).
The CPU implementations are the numerical reference; test-backend-ops compares backend ops against the CPU backend, so on a CPU-only build the new cases register but are inert (CPU is the reference). THere will be follow up GPU PRs.
Additional information
Attribution
Derived from antirez/llama.cpp-deepseek-v4-flash (https://github.com/antirez/llama.cpp-deepseek-v4-flash), which builds on fairydreaming's work and the ggml/llama.cpp lineage. The V4 ops and model implementation originate there. Used with antirez's permission
Requirements