feat(dsv4): DeepSeek V4 Flash support — CUDA/VRAM expert tiering, DSML tool use, cross-session KV persistence

[rafpigna]

feat(dsv4): DeepSeek V4 Flash support — CUDA/VRAM expert tiering, DSML tool use, cross-session KV persistence

Motivation

Until now, DeepSeek V4 Flash on colibri has been a CPU-only citizen: every
expert streams from disk (O_DIRECT), dense/attention run on the CPU, and the
OpenAI-compatible layer has no DSML tool calling and no cross-session KV
persistence. The engine works, but it is missing the three things that make a
model actually usable day-to-day on the hardware people own:

1. Use the CUDA GPU people already have. A large share of colibri users
   run CUDA-capable NVIDIA GPUs (10-24 GB class). This PR adds a VRAM expert
   tier so the hottest routed experts per layer stay resident on the GPU
   (usage-ranked via .coli_usage) instead of being re-read from disk — with
   a silent CPU fallback when CUDA is absent or disabled, so the
   dependency-free default path is untouched. The honest numbers: on a
   10.7 GB VRAM card the measured gain of CPU+GPU over CPU-only is ~+4-5%
   decode, because the bottleneck at that VRAM size is the disk read volume,
   not the matmul. The gain is small but real, it grows with the VRAM budget,
   and the tier is arch-agnostic (sm_86 → sm_120) and opt-in.
2. Tool use. The checkpoint's native ｜DSML｜ format is now wired
   end-to-end (render, stream, parse, Anthropic translation). Without it the
   model cannot call functions, which rules it out for agent workflows.
3. Cross-session KV persistence. Without .coli_kv every restart
   re-prefills the entire history; with it a conversation resumes from disk
   in seconds (prefix_reused>0).

In short: this PR turns DeepSeek V4 Flash from a CPU-only engine into a
complete colibri model — GPU tiering when you have the VRAM, tool calling,
and conversation state that survives restarts.

Summary

Full DeepSeek V4 Flash support for colibri, in one PR:

1. CUDA/VRAM expert tiering — the top-M hottest routed experts per layer
   (usage-ranked, .coli_usage) stay resident in VRAM on top of the existing
   RAM hot-store; the rest streams cold from disk (O_DIRECT). Arch-agnostic
   kernels (sm_86 → sm_120), CUDA DLL split on Windows (coli_dsv4_cuda.dll,
   silent CPU fallback via backend_loader_dsv4.c), direct CUDA link on
   Linux. Dense/attention stay on CPU/RAM. Numerics identical to the CPU path
   (verified A/B).
2. Tool use (DSML) — the checkpoint's native ｜DSML｜ tool format wired
   end-to-end in openai_server.py: streaming with multi-marker suppression
   (tool_calls/invoke), multi-turn tool results merged into user turns,
   tool_choice none/auto/required/forced, tolerant parser for truncated
   blocks, Anthropic /v1/messages translation (content_blocks). Builds on
   the DSML primitives vendored by feat(v4): DSML tool calling for DeepSeek V4 (#916) #948.
3. Cross-session KV persistence (.coli_kv) — attention state
   (window + compressed + recurrent indexer) snapshotted after every turn
   (atomic temp+rename, KVSAVE=0 disables) and resumed at the next start:
   no re-prefill of the history. Verified by prefix_reused>0 in the serve
   DONE frame.
4. Performance guide (docs/deepseek-v4-cuda-tiering.md) — baseline
   launch commands for CPU-only and CPU+CUDA, tuning knobs, reference
   measurements.

Build

• Linux: make -f Makefile.deepseek-v4 deepseek-v4 CUDA=1 CUDA_ARCH=sm_86
• Windows: make -f Makefile.deepseek-v4 dsv4-cuda-dll CUDA=1 CUDA_ARCH=sm_86
  (nvcc/MSVC DLL) + make -f Makefile.deepseek-v4 deepseek-v4 CUDA=1 CUDA_ARCH=sm_86 LTO=0
• Parent make deepseek-v4 forwards CUDA/CUDA_ARCH to the DS4 sub-makefile.

Testing

• Numerics CPU vs CUDA: token-identical on real checkpoints (rows16-pack
  ordering pitfall fixed).
• Tiering: banner [DSV4 CUDA] device 0 ... sm_86, hit_rate climbs with
  autopin, clean VRAM teardown; silent CPU fallback with the DLL absent.
• Tool use: unit 25/25, E2E 14/14 (DSML, strict checkpoint parser).
• KV persist: 15/15 (resume prefix_reused>0, corrupt-file ignored,
  KVSAVE=0 disables).
• Performance (reference: 5900X, 64 GB RAM, RTX 3080 10.7 GB, NVMe): ~1.0 tok/s
  decode with --ram 52 --vram 4 + COLI_V4_PREWARM=1; details in the guide.

Notes / limitations

• Serve is greedy, one KV slot; MTP/DSpark stays opt-in (V4_MTP=0).
• Only routed experts are GPU-eligible; dense/attention remain CPU (the
  "inverted" placement vs GLM, documented in the guide).
• The bottleneck is expert read volume from disk (~1 GB/token at 83% hit);
  measured tuning levers are documented in the guide (--ram ≈ RAM − 12 GB,
  --vram ≈ 50% VRAM, COLI_V4_PREWARM=1).

Acknowledgements

The CUDA kernels (backend_cuda_dsv4.cu/.h, dsv4_mhc.h, dsv4_quant.h) are
ported from ZacharyZcR's colibri fork, branch
feat/deepseek-v4-long-context (PRs #772/#773), adapted from the all-resident
expert layout to the streaming/offload tier used by this engine. Everything on
top of those kernels — the tiering integration, DSML tool use, cross-session KV
persistence, and the Windows DLL loader — is original work on this branch.
Same Apache-2.0 license as colibri.

Files

14 files, +3555/−25. New: backend_cuda_dsv4.cu/.h, backend_loader_dsv4.c,
dsv4_mhc.h, dsv4_quant.h, kv_persist_dsv4.h,
docs/deepseek-v4-cuda-tiering.md. Modified: deepseek_v4.c,
deepseek_v4_internal.h, openai_server.py, coli, Makefile,
Makefile.deepseek-v4, .gitignore. Synced with upstream/dev by merge.

[RobertKoval]

Reproducible CUDA build portability issue on Jetson AGX Orin (aarch64, sm_87).

The host compiler issue is resolved by using GCC 13 explicitly with nvcc, but the CUDA backend then fails to compile because backend_cuda_dsv4.cu unconditionally references newer cuBLASLt MXFP8 block-scaling APIs that are absent from the Jetson CUDA/cuBLASLt headers:

backend_cuda_dsv4.cu(1247): error: identifier "cublasLtMatmulMatrixScale_t" is undefined
backend_cuda_dsv4.cu(1247): error: identifier "CUBLASLT_MATMUL_MATRIX_SCALE_VEC32_UE8M0" is undefined
backend_cuda_dsv4.cu(1251): error: identifier "CUBLASLT_MATMUL_DESC_A_SCALE_MODE" is undefined
backend_cuda_dsv4.cu(1252): error: identifier "CUBLASLT_MATMUL_DESC_B_SCALE_MODE" is undefined

Repro command:

make -f Makefile.deepseek-v4 \
  deepseek-v4 \
  CUDA=1 \
  CUDA_ARCH=sm_87 \
  NVCCFLAGS="-O3 -std=c++17 -arch=sm_87 -ccbin /home/linuxbrew/.linuxbrew/bin/g++-13" \
  -j$(nproc)

Looking at the source, these symbols are used by the optional tc_plan() / tc_fp4_matvec() path behind the runtime DSV4_CUDA_TC switch. DSV4_CUDA_TC defaults to 0, while the normal custom FP4 path (mv_fp4_grouped etc.) does not require these APIs.

So the optional TC implementation is currently a compile-time dependency even when it cannot/will not be selected at runtime.

This seems inconsistent with the PR's stated arch-agnostic sm_86 -> sm_120 support: an Orin (sm_87) should be able to build the ordinary FP4 CUDA expert path without requiring the newer cuBLASLt UE8M0 scale-mode API.

A minimal fix would be to compile-gate the cuBLASLt TC helper path on availability of the required API/toolkit version (and make DSV4_CUDA_TC unavailable there), leaving the existing custom FP4 CUDA fallback compiled normally.

I can test a patch on physical AGX Orin if useful.

[dcutugno]

This is what i have in my implementation that i'm still testing, what about those numbers,
Turn 1: time to first token 305 s
Turn 2 (tool result / follow-up) (full re-prefill) 6.4 s
New session, same system prompt (from the 2nd new session on) 5.8 s
Decode 1.5–1.6 tok/s

System is Ryzen 9 3900x, 32GB ram, 5080 16GB, 2 nvme sdd in model mirror mode

[dcutugno]

squeezed 1.7 tok/sec
also brain and profilings works

[rafpigna]

This is what i have in my implementation that i'm still testing, what about those numbers, Turn 1: time to first token 305 s Turn 2 (tool result / follow-up) (full re-prefill) 6.4 s New session, same system prompt (from the 2nd new session on) 5.8 s Decode 1.5–1.6 tok/s

System is Ryzen 9 3900x, 32GB ram, 5080 16GB, 2 nvme sdd in model mirror mode

From my testings, the real nob for performance is the nvme. Having two fast nvme in mirror mode makes a HUGE difference in performance, greater than having more VRAM or RAM or a faster CPU/GPU. The compute is disk-bounded. As comparison, using llama.cpp that has no tiering optimization, no hot or warm experts but only disk streaming, I have DOUBLE the performances than colibri, but no kv persistance is a real pain for a "production" usage exceot than small chats. Also the prefills took ages, because have to read and execute ALL the experts, not the pinned one like colibri, so it takes A LOT, 10x than colibri.

[ANBAL534]

I cannot get to reproduce your findings, other than vram filling I see no GPU compute usage and also no tok/s gains vs CPU only.

I checked out your branch feat/ds4-cuda-tier and built the engine with:

make -f Makefile.deepseek-v4 deepseek-v4 CUDA=1 CUDA_ARCH=native

And did the comparison using these commands:

# CPU-only
OMP_NUM_THREADS=8 COLI_V4_PREWARM=1 ./coli web --model ~/ssd-m2/DeepSeek-V4-Flash --gpu none --ram 82

# GPU
OMP_NUM_THREADS=8 COLI_V4_PREWARM=1 ./coli web --model ~/ssd-m2/DeepSeek-V4-Flash --gpu 0 --ram 82 --vram 6

Both instances gets me 0.6tok/s

This is the nvidia-smi output mid-generating with GPU enabled:

Sat Aug 15 19:50:58 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 610.57.04              KMD Version: 610.57.04     CUDA UMD Version: 13.3     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4070        Off |   00000000:04:00.0  On |                  N/A |
|  0%   49C    P2             35W /  200W |    8590MiB /  12282MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
...
|    0   N/A  N/A           21475      C   ...ri-dsv4-cuda-pr/c/deepseek_v4       6654MiB |
+-----------------------------------------------------------------------------------------+

My system is:
Ryzen 7 5800X - 96GB RAM - RTX 4070 (12GB)

The model itself is in a dedicated m.2 ssd

[dcutugno]

i will post my PR soon, so if you can take a test to it and report back would be good, dual SSD is not the only performance bound gap for the 4x series and onward i have used DeepGem for other the standard CUDA one.
Currently it's missing the multiple GPU shared work setup as i have only one 5080...
Or if you want to checkout my branch here https://github.com/dcutugno/colibri/tree/feature/dsv4-cuda-kernels

[rafpigna]

Reproducible CUDA build portability issue on Jetson AGX Orin (aarch64, sm_87).

The host compiler issue is resolved by using GCC 13 explicitly with nvcc, but the CUDA backend then fails to compile because backend_cuda_dsv4.cu unconditionally references newer cuBLASLt MXFP8 block-scaling APIs that are absent from the Jetson CUDA/cuBLASLt headers:

backend_cuda_dsv4.cu(1247): error: identifier "cublasLtMatmulMatrixScale_t" is undefined
backend_cuda_dsv4.cu(1247): error: identifier "CUBLASLT_MATMUL_MATRIX_SCALE_VEC32_UE8M0" is undefined
backend_cuda_dsv4.cu(1251): error: identifier "CUBLASLT_MATMUL_DESC_A_SCALE_MODE" is undefined
backend_cuda_dsv4.cu(1252): error: identifier "CUBLASLT_MATMUL_DESC_B_SCALE_MODE" is undefined

Repro command:

make -f Makefile.deepseek-v4 \
  deepseek-v4 \
  CUDA=1 \
  CUDA_ARCH=sm_87 \
  NVCCFLAGS="-O3 -std=c++17 -arch=sm_87 -ccbin /home/linuxbrew/.linuxbrew/bin/g++-13" \
  -j$(nproc)

Looking at the source, these symbols are used by the optional tc_plan() / tc_fp4_matvec() path behind the runtime DSV4_CUDA_TC switch. DSV4_CUDA_TC defaults to 0, while the normal custom FP4 path (mv_fp4_grouped etc.) does not require these APIs.

So the optional TC implementation is currently a compile-time dependency even when it cannot/will not be selected at runtime.

This seems inconsistent with the PR's stated arch-agnostic sm_86 -> sm_120 support: an Orin (sm_87) should be able to build the ordinary FP4 CUDA expert path without requiring the newer cuBLASLt UE8M0 scale-mode API.

A minimal fix would be to compile-gate the cuBLASLt TC helper path on availability of the required API/toolkit version (and make DSV4_CUDA_TC unavailable there), leaving the existing custom FP4 CUDA fallback compiled normally.

I can test a patch on physical AGX Orin if useful.

Hello @RobertKoval
Thank you for the detailed report and for the precise diagnosis, it saved time. You are right on both counts: the TC path is a compile-time dependency even though it is selected at runtime (DSV4_CUDA_TC, default 0), and that is inconsistent with the arch-agnostic sm_86 → sm_120 claim.
I've prepared a fix: the cuBLASLt MXFP8 tensor-core helper path (tc_plan/tc_prepare_x/tc_fp4_matvec and its dispatch in
dsv4_cuda_expert_group) is now compile-gated. The custom FP4 path (mv_fp4_grouped etc.) always builds, unchanged.
On toolkits without the block-scaling APIs (cuBLASLt < 12.8, e.g. Jetson JetPack with CUDA 12.6), build with:

make -f Makefile.deepseek-v4 \
  deepseek-v4 \
  CUDA=1 \
  CUDA_ARCH=sm_87 \
  NVCCFLAGS="-O3 -std=c++17 -arch=sm_87 -ccbin /home/linuxbrew/.linuxbrew/bin/g++-13 -DCOLI_DSV4_NO_TC" \
  -j$(nproc)

With -DCOLI_DSV4_NO_TC the TC path is not compiled at all (so DSV4_CUDA_TC is inert) and the ordinary FP4 CUDA expert path compiles normally. Without the flag the build is byte-for-byte the same as before, no behavior change on CUDA 13.x / 12.8+ toolkits.

Fix commit: 4199e1b

I would very much welcome a test on your physical AGX Orin, exactly the hardware I cannot reach from here. If the build passes and the FP4 tier runs ([DSV4 CUDA] device ... sm_87, uploads>0), I'll fold any adjustment you suggest into the PR.

I also added a Jetson-specific note to the build docs, but please confirm the -DCOLI_DSV4_NO_TC invocation works as-is.

Thanks again for offering to test.

[rafpigna]

i will post my PR soon, so if you can take a test to it and report back would be good, dual SSD is not the only performance bound gap for the 4x series and onward i have used DeepGemm for other the standard CUDA one. Currently it's missing the multiple GPU shared work setup as i have only one 5080... Or if you want to checkout my branch here https://github.com/dcutugno/colibri/tree/feature/dsv4-cuda-kernels

@dcutugno I dint follow the DeepGem route deliberately, because you cant use with Ampere / RTX 30XX (and I have only that). My goal was to provide a cuda arch-agnostic usage. Optimizations for other archs are welcome, this is obvious, but in my understanding of the Colibri project, having an older GPU (or no GPU at all) has to not stop you from compile and use a model. But this is just my opinion :)

[dcutugno]

@rafpigna that's why it will compile two dll, if no deepgem hardware found it will fallback to CUDA one :)

[rafpigna]

New commit on this branch: optional dense-on-GPU tier (8b9aa52).

Following the placement discussion, this adds an opt-in tier that moves the five per-layer dense matmuls (wq_a/wq_b/wkv/wo_a/wo_b) to the GPU, with numerics identical to the CPU path. It is off by default (COLI_DSV4_DENSE_CUDA=1 enables it); any failure falls back to CPU silently.

Measured on the reference hardware (RTX 3080 sm_86, --ram 52 --vram 4, same raw-serve harness, .coli_usage restored between runs):

	dense CPU	dense GPU
decode tok/s	0.955	1.126 (+18%)
TTFT	95 s	105 s
CPU cores (decode)	5.54	3.97
GPU util (decode)	5.6%	10.0%
VRAM peak	5.3 GB	9.7 GB

• Integrity: 42,785 dense-GPU calls ok=1, 0 CPU fallbacks over a 200-token run (43 layers × 5 matmuls); outputs token-identical to the CPU path.
• Cost: ~9.6 GB VRAM (dense 6.27 GiB + expert tier) and ~+10 s TTFT (H2D preload of the 43 layers at startup) — amortized on long sessions, hence default OFF.
• The docs (docs/deepseek-v4-cuda-tiering.md) are updated: tuning table gains COLI_DSV4_DENSE_CUDA / COLI_V4_DENSE_DEBUG, plus a "Dense-on-GPU tier" measurements section.

[rafpigna]

I cannot get to reproduce your findings, other than vram filling I see no GPU compute usage and also no tok/s gains vs CPU only.

I checked out your branch feat/ds4-cuda-tier and built the engine with:

make -f Makefile.deepseek-v4 deepseek-v4 CUDA=1 CUDA_ARCH=native

And did the comparison using these commands:

# CPU-only
OMP_NUM_THREADS=8 COLI_V4_PREWARM=1 ./coli web --model ~/ssd-m2/DeepSeek-V4-Flash --gpu none --ram 82

# GPU
OMP_NUM_THREADS=8 COLI_V4_PREWARM=1 ./coli web --model ~/ssd-m2/DeepSeek-V4-Flash --gpu 0 --ram 82 --vram 6

Both instances gets me 0.6tok/s
My system is: Ryzen 7 5800X - 96GB RAM - RTX 4070 (12GB)

The model itself is in a dedicated m.2 ssd

Thank you @ANBAL534 for the detailed report, a 0% GPU-util with the VRAM tier filling is really strange and Ada / native Linux is a combination I cannot test from here, so this is very valuable.

Quick sanity note: on my hardware (RTX 3080, sm_86) the tier does execute and I measure uploads>0/drops>0, 10-15% GPU util during decode, and token-identical output vs CPU-only. So a full 0% on your side is not the "small but expected" disk-bound effect; something is falling back to CPU for every expert. The uploads still happen (that is why your VRAM fills), but the matvec kernels are apparently never launched.

I have just pushed a diagnostic commit to the branch (d745bcc) that prints the exact reason whenever the expert dispatch falls back to CPU.

Please could you:

1. Update and rebuild the branch:

   git fetch origin
   git checkout feat/ds4-cuda-tier
   git pull origin feat/ds4-cuda-tier
   make -f Makefile.deepseek-v4 deepseek-v4 CUDA=1 CUDA_ARCH=native
   

   (check with git log -1 --oneline that you are on d745bcc` or a following one)

2. Re-run the GPU command and look at the terminal output for:

   • the startup banner [DSV4 CUDA] device 0: ... sm_89 does it appear?
   • any repeated lines [DSV4 CUDA] expert dispatch failed (layer=.. expert=..); falling back to CPU
   • any [DSV4 CUDA] ...: <error> lines
     Copy the first few lines of each kind.

3. Also useful:

   • your model's config.json (model_type, model_version) to rule out a different checkpoint layout than the original "DeepSeek V4 Flash 0731"
   • a re-run with CUDA_LAUNCH_BLOCKING=1 (same command): if an async CUDA error is the cause, it becomes synchronous and prints to stderr
   • re-runs with DSV4_CUDA_BATCHED=0 and DSV4_CUDA_TC=1 to isolate which expert path (batched custom kernel vs tensor-core path) does anything
   • while generating, a short nvidia-smi dmon -s u -d 2 (30 s) to see if there are any utilization spikes at all
   • test with coli chat and coli serve to see if it's something related only to coli web (quite sure it's not, but worth trying)
   • test using the new layer placement introduced with the commit 8b9aa52, using the new env_var COLI_DSV4_DENSE_CUDA=1 before the run.

One note: the engine's shutdown counters (v4_cuda_tier uploads=.. drops=..) only print on a clean serve-protocol shutdown, which Ctrl-C on coli web skips, the new per-dispatch log above is the more direct signal anyway.

If the new log shows the fallback reason, that should pinpoint the cause directly.

Thanks again for taking the time to test.

[dcutugno]

On my bracnh i get:
turn after serve start ttft decode
1st (cold VRAM mirrors + RAM cache) 8.4 s 1.60 tok/s
2nd 5.5 s 1.76 tok/s
3rd 5.6 s 1.76 tok/s

[dcutugno]

We've been building the same thing from a different angle and just opened it against dev, so the overlap is visible side by side rather than discovered at merge time:

• feat(v4): prefix checkpoints (memory + disk), resumable 4096-token prefill segments, cancellable prefill, gateway prefix hint #1053 — prefix checkpoints (memory + disk, any matching prefix, not one session), resumable 4096-token prefill segments, cancellable prefill, gateway prefix hint (CPU-only; greedy text byte-identical to dev)
• feat(v4/cuda): the CUDA tier wired into the DeepSeek V4 engine — generic kernels for any sm_80+, Windows runtime DLL loader, Linux CUDA=1 #1055 — the CUDA tier wired into the engine on top of DeepSeek V4: the CUDA kernels, as a tier rather than an engine #772's rescoped kernels: two DLL flavours picked at runtime (generic sm_80+ / DeepGEMM sm_120a), GPU prefill attention on a device KV ring, route-aware VRAM expert bank for prefill MoE, VRAM-sized decode mirrors, Linux CUDA=1 (verified under WSL2)
• build(v4/cuda): fetch the pinned DeepGEMM sm120 headers on first build — cuda-dsv4-dg-dll / DEEPGEMM=1 need no external checkout, nothing vendored #1056 — optional vendoring of the DeepGEMM sm120 headers

On an RTX 5080 16 GB + 2× NVMe: 3.3k-token prefill 90 s, 8.3k opencode first turn ~4 min once and then 6–9 s per session/turn, decode 1.76 tok/s at 3k context. The dense-on-GPU tier and .coli_kv here overlap with the dense mirrors / prefix checkpoints there; happy to reconcile whichever lands first — the numbers and design notes are in the PR descriptions.

[rafpigna]

We've been building the same thing from a different angle and just opened it against dev, so the overlap is visible side by side rather than discovered at merge time:

• feat(v4): prefix checkpoints (memory + disk), resumable 4096-token prefill segments, cancellable prefill, gateway prefix hint #1053 — prefix checkpoints (memory + disk, any matching prefix, not one session), resumable 4096-token prefill segments, cancellable prefill, gateway prefix hint (CPU-only; greedy text byte-identical to dev)
• feat(v4/cuda): the CUDA tier wired into the DeepSeek V4 engine — generic kernels for any sm_80+, Windows runtime DLL loader, Linux CUDA=1 #1055 — the CUDA tier wired into the engine on top of DeepSeek V4: the CUDA kernels, as a tier rather than an engine #772's rescoped kernels: two DLL flavours picked at runtime (generic sm_80+ / DeepGEMM sm_120a), GPU prefill attention on a device KV ring, route-aware VRAM expert bank for prefill MoE, VRAM-sized decode mirrors, Linux CUDA=1 (verified under WSL2)
• build(v4/cuda): vendor the DeepGEMM sm120 headers — cuda-dsv4-dg-dll / DEEPGEMM=1 need no external checkout #1056 — optional vendoring of the DeepGEMM sm120 headers

On an RTX 5080 16 GB + 2× NVMe: 3.3k-token prefill 90 s, 8.3k opencode first turn ~4 min once and then 6–9 s per session/turn, decode 1.76 tok/s at 3k context. The dense-on-GPU tier and .coli_kv here overlap with the dense mirrors / prefix checkpoints there; happy to reconcile whichever lands first — the numbers and design notes are in the PR descriptions.

Thanks @dcutugno for the detailed status and congrats on the merge, the sequencing (kernels first, then wiring) was the right call and @JustVugg summary makes that easy to see.

Since you offered to help reconciling, here is where I land, based on what is now in dev after #1054, #988 and #1055:

What I will drop from #1031 (now covered by the merged tree):

• the expert CUDA tier and its kernels (backend_cuda_dsv4.cu/.h, dsv4_mhc.h, dsv4_quant.h) — superseded by the merged COLI_V4_GPU_TIER implementation;
• the Windows loader (backend_loader_dsv4.c) and the dense-on-GPU tier (COLI_DSV4_DENSE_CUDA) — covered by the DLL flavours and the dense mirrors there.

What I would keep (not in dev today, as far as I can tell):

• DSML tool use in openai_server.py on top of feat(v4): DSML tool calling for DeepSeek V4 (#916) #948's vendored primitives: Anthropic /v1/messages content_blocks rendering, the tolerant parser for truncated blocks, multi-marker streaming suppression (tool_calls + invoke). Verified 25/25 unit + 14/14 E2E.
• Cross-session KV persistence (.coli_kv) — per-layer snapshots (window + compressed + recurrent indexer), atomic temp+rename, resume with prefix_reused>0 in the serve DONE frame. This is the piece with the most conceptual overlap with feat(v4): prefix checkpoints (memory + disk), resumable 4096-token prefill segments, cancellable prefill, gateway prefix hint #1053's prefix checkpoints: yours are CPU-only, prefix-agnostic and chunked, mine are per-conversation attention-state snapshots. I would like to reconcile that specifically — happy to drop mine if feat(v4): prefix checkpoints (memory + disk), resumable 4096-token prefill segments, cancellable prefill, gateway prefix hint #1053 is meant to cover the use case, or to keep mine and let feat(v4): prefix checkpoints (memory + disk), resumable 4096-token prefill segments, cancellable prefill, gateway prefix hint #1053 focus on the prefill machinery.
• Diagnostics: fallback-reason logging (d745bcc, the 0%-GPU-util report from @ANBAL534).
• Engine telemetry for the web dashboard: VRAM-tier expert reporting in the TIERS/EMAP serve frames + native Windows HWINFO (b2c8eb7) — the per-column VRAM/RAM/disk tier view in coli web.
• The user guide (docs/deepseek-v4-cuda-tiering.md) with my measurements — can be merged into your docs if you prefer a single page or totally be discarded, but I think a full, clear and very detailed guide is really a must-have to not stop the project to be easily usable only from people that already knows how to build and debug, in my opinion a software project is really "open" when also a newbie can compile and run in 5 minutes without having to learn for a day how to compile on linux or windows, or dig in the docs to find the env_vars or the cli flags.

Questions

1. On .coli_kv vs feat(v4): prefix checkpoints (memory + disk), resumable 4096-token prefill segments, cancellable prefill, gateway prefix hint #1053: which one should own conversation-state persistence? If feat(v4): prefix checkpoints (memory + disk), resumable 4096-token prefill segments, cancellable prefill, gateway prefix hint #1053 lands first with disk checkpoints, I'll drop mine; if you'd rather we keep it, tell me and I'll align the interfaces.
2. Do you see anything else in my diff that the merged tree already covers?
3. @JustVugg: do you want me to rebase-and-shrink feat(dsv4): DeepSeek V4 Flash support — CUDA/VRAM expert tiering, DSML tool use, cross-session KV persistence #1031 now (keeping only the items above), or would you rather sequence it after feat(v4): prefix checkpoints (memory + disk), resumable 4096-token prefill segments, cancellable prefill, gateway prefix hint #1053/build(v4/cuda): fetch the pinned DeepGEMM sm120 headers on first build — cuda-dsv4-dg-dll / DEEPGEMM=1 need no external checkout, nothing vendored #1056?

I'll start the rebase as soon as the direction is clear.
Thanks for the help.

[JustVugg]

Heads-up, and an apology: this went dirty because of us, not you.

#1063 landed on dev — it makes model families registry-owned, so c/coli, c/openai_server.py, c/doctor.py and c/resource_plan.py now all read one descriptor table (c/family_registry.py) instead of each carrying its own branches. That was a deliberate structural change agreed in Discussion #1057, and it rewrote exactly the files your branch touches.

A rebase on current dev should be mechanical — the conflicts will be in the family/dispatch branches that no longer exist, and the replacement is usually "read it from the registry" rather than "add another branch". If a conflict isn't obvious, say so on the PR and I'll work through it with you rather than leave you guessing at the new contract.

Two things worth knowing while you're in there:

• Unknown model_type is now refused explicitly instead of silently falling through to the GLM engine. If your change relied on that fallback anywhere, it won't behave the same.
• resource_plan.py now refuses to plan for families without a measured adapter (Kimi, OLMoE, Inkling, V4) rather than returning a plausible-looking zero. That was the bug that hit two model PRs independently in the same day.

Sorry for the churn. Landing the registry before your branches was the right call for the project, but it does mean the cost of it fell on the people with open work.

[JustVugg]

Before you spend an evening rebasing 4,019 lines, you deserve an honest map of what dev did while this sat — because most of this PR's first pillar landed independently, and rebasing it as-is would mostly be re-fighting merged code.

Superseded — already on dev: the CUDA/VRAM expert tiering. c/backend_cuda_dsv4.cu, c/backend_cuda_dsv4.h, c/backend_loader_dsv4.c, c/dsv4_mhc.h, c/dsv4_quant.h all exist there now, via #1054/#1055 (@dcutugno) — "the DeepSeek V4 CUDA kernels the engine tier needs" plus "the CUDA tier wired into the engine, every stage CPU-canonical with per-stage fallback" — and #1056 added the pinned-commit DeepGEMM fetch on top. That is the same ground your PR covers, and the file names collide exactly.

Still unique to you, as far as I can see: kv_persist_dsv4.h (deepseek_v4.c on dev has no kv_persist reference at all — it has in-process prefix reuse from #1051, but nothing cross-session) and the DSML tool-use path (no dsml anywhere in the tree).

So the ask is not "rebase" — it is rescope. Two smaller PRs against current dev, each standing on its own:

1. Cross-session KV persistence for V4 — this is genuinely missing and genuinely useful; on a 156 GB streamed model, not re-prefilling a repeated conversation is worth real seconds.
2. DSML tool use — independent of the engine internals, reviewable on its own terms.

Both would go in clean and get merged on green CI; the tiering part I would simply drop rather than have you reconcile it against an implementation that already ships. If you think your tiering does something #1054/#1055 does not — different placement policy, different fallback shape — say what, and we will compare rather than assume.

And to be explicit about whose problem this is: your PR is four days old and the collision happened inside those four days. That is the cost of a fast-moving dev, not a judgement on your work.

[rafpigna]

Thanks again @JustVugg for the honest map of dev — and no apology needed: a structural change landing on open PRs is the natural cost of the sequencing decision, and the registry refactor (#1063) was clearly the right call for the project.

I've gone through current dev (v1.7.0, 940ea50) against the branch. Here's what I find and what I intend to do, so we're explicit before I spend the effort:

Dropping from #1031 — agreed, already on dev:

• the expert CUDA/VRAM tier and its kernels (backend_cuda_dsv4.cu/.h, dsv4_mhc.h, dsv4_quant.h) — superseded by DeepSeek V4: the CUDA kernels, as a tier rather than an engine (rebase of #772) #1054/feat(v4/cuda): the CUDA tier wired into the DeepSeek V4 engine — generic kernels for any sm_80+, Windows runtime DLL loader, Linux CUDA=1 #1055 (+build(v4/cuda): fetch the pinned DeepGEMM sm120 headers on first build — cuda-dsv4-dg-dll / DEEPGEMM=1 need no external checkout, nothing vendored #1056 DeepGEMM), file names collide;
• the Windows loader (backend_loader_dsv4.c) and the dense-on-GPU tier (COLI_DSV4_DENSE_CUDA) — covered by the DLL flavours and dense mirrors in feat(v4/cuda): the CUDA tier wired into the DeepSeek V4 engine — generic kernels for any sm_80+, Windows runtime DLL loader, Linux CUDA=1 #1055.

Two factual notes before deciding scope (I checked the tree, not just the title):

• Cross-session KV persistence: confirmed genuinely missing. deepseek_v4.c has no kv_persist reference at all; it carries only in-process prefix reuse (kv_prefix.h, "no snapshot, no rewind"). The generic kv_persist.h is wired only into colibri.c (GLM), not the V4 engine. So .coli_kv appears to be a real gap, and a useful one on a 156 GB streamed model that streams a conversation's experts off disk again on every restart. I'd like to keep this as a single, self-contained PR.
• DSML tool use: here I need to gently push back. Current dev already ships the full DSML tool path — c/v4_dsml.py (vendored from feat(v4): DSML tool calling for DeepSeek V4 (#916) #948) is wired into openai_server.py (render_chat_v4 dispatches on deepseek_v4, tolerant parse_completion_text, two-marker streaming suppression for tool_calls+invoke, and the Anthropic /v1/messages translation + content_blocks rendering). So I don't think a separate DSML PR would add anything — it would mostly re-add merged code. If you mean a specific gap that isn't in the tree today, point me at it and I'll scope just that.

What I propose: close #1031 and open one clean PR against current dev, scoped to cross-session KV persistence for DeepSeek V4 (.coli_kv) — self-contained, on top of kv_prefix.h. Possibly with the web-dashboard telemetry (TIERS/EMAP/HWINFO) and my user guide folded in, since both are engine/docs-side and currently absent from dev; say the word if you'd rather I keep the PR to the persistence only and send the guide as a docs contribution.

A scope question so we align on ownership with #1053: its "prefix checkpoints (memory + disk)" overlap conceptually with an attention-state snapshot. Yours is prefix-agnostic and chunked for the prefill machinery; mine is a per-conversation on-disk snapshot of the full attention state. Do you want #1053 to own that domain (then I drop .coli_kv), or is there room for both — one as the prefill/prefix-cache layer, the other as the cross-session resume layer?

I'll start the new branch as soon as the direction is clear. Thanks for taking the time to write this up.

[JustVugg]

Closing as you proposed, and here are the answers you were waiting on. Sorry these took a week.

Your DSML pushback was right. I checked the tree rather than the titles: c/v4_dsml.py is wired through openai_server.py (_dsv4_tools_block, _dsv4_tool_calls, the tolerant completion parse and the two-marker streaming suppression) with e2e coverage. Nothing to add there. Thank you for checking before spending the effort rather than after.

Ownership question, #1053 vs .coli_kv. #1053 merged on 2026-08-16 and its disk half is real, not aspirational: V4_PREFIX_CKPT_DISK writes <model>/.coli_ckpt/ckpt_<fingerprint>_<n>.bin and reloads lazily on the first generate, so a system prefix does survive a process restart. That covers the expensive shared part of the case you made. A per-conversation full attention snapshot would add resume of the turns after the prefix boundary, which on this workload is the short end, against per-conversation state files and their invalidation. So: #1053 owns the domain, drop .coli_kv. That is a call about marginal value, not about your design.

What is still a genuine gap, and what I would rather have from you. V4 emits HWINFO and nothing else: v4_hwinfo_emit() at deepseek_v4.c:13503 is the only emitter in the engine, while tiers_emit() and emap_emit() live in colibri.c and are GLM-only. The web brain map is therefore blank for V4, on the engine where expert residency is the thing you most want to see. That is a clean, self-contained PR against current dev and it is the one I would take first.

Two notes if you pick it up: telemetry.h has traffic right now (#1235 is reworking the Vulkan residency path in it), so pull dev before you start; and V4 tiers are not GLM tiers, so the interesting question is what the wire format should say about a streamed-from-disk expert, not just how to reuse the GLM emitter.

The user guide: docs/deepseek-v4.md exists and is fairly complete now. Send yours as additions to that file where it goes deeper, rather than as a parallel document.

Thank you for the whole exchange on this one. You checked dev against your own branch, dropped what was superseded without being asked, pushed back where I was wrong, and asked the scope question before writing code instead of after. That is more care than most contributors spend on a PR that ends up closed, and it is exactly the reason the parts that were still real are easy to name.