Name and Version
$ llama-cli --version
version: 9924 (90e0f5c)
built with GNU 13.3.0 for Linux x86_64
Operating systems
Linux
GGML backends
CUDA
Hardware
GPU: 5x NVIDIA CUDA (2x RTX 3090, RTX 5060 Ti, 2x RTX 4060 Ti), 96 GiB total VRAM
CPU: Intel Core i5-13600K (14 cores / 20 threads)
RAM: 128 GB DDR4-3200
OS: Linux x86_64 (Ubuntu 24.04)
Models
DeepSeek-V4-Flash (DSV4-Flash), GGUF, MXFP4 quant, 5 shards.
Problem description & steps to reproduce
During an agentic coding session, after several turns, I get a crash with a false "Context size has been exceeded" at ~1250 tokens against a 16k context. It's SWA find_slot exhaustion of the 768-cell window, not a real context limit, with 0 CUDA errors (no OOM/abort).
Also, before the crash, throughout the session every divergent turn stalls. Every divergent turn re-prefills thousands of tokens from a checkpoint boundary instead of just the delta, so the agent sits on "Working" with nothing streaming. The same churn eventually crashes.
Both fall out of the same no-partial-rollback limitation; one proper seq_rm of the diverged suffix fixes both.
The window full so no slot even at ~1250 tokens, not a real ctx limit, 0 CUDA errors proves it's not a GPU fault.
The root cause:
DSV4-Flash raw attention is a 128-token sliding window on a 768-cell SWA KV cache, the only cache find_slot runs on. find_slot reuses an occupied cell only if SWA-masked. Also the same sequence causal reuse path is commented out (src/llama-kv-cache.cpp ~1043-1047) on the invariant "purge futures beforehand."
Normal models satisfy that with seq_rm(seq_id, n_past, -1) before re-decoding a divergent suffix. DSV4 can't do a partial seq_rm (src/llama-kv-cache-dsv4.cpp ~1153). The compressor ring state can't roll back mid-block. So the server rewinds via a checkpoint-restore that repopulates the SWA window without purging futures.
The crash: re decode refills positions inside the full 768-cell window and find_slot can neither reuse the occupied same-seq cells nor reclaim the live top ~128 window cells resulting in a false "context exceeded."
The stall: The same no partial rollback. Every divergent turn re prefills from a checkpoint boundary, no delta path.
The defect is in shared KV / server code not a CUDA kernel. Having no CUDA errors backs this so it's expected across backends.
A monotonic growing prompt never jams and the band probe reached 119,788 tokens clean. The crash needs reuse that re-fills occupied cells, not raw depth.
Reproduce with:
churn.py:
import urllib.request, urllib.error, json, random, time
URL="http://127.0.0.1:10099/v1/completions"
random.seed(11)
def sent(i): return f"Paragraph {i}: quick brown fox {i} leaps over lazy dog {i}, counting {i} apples, {i} pears, {i} plums in crate {i} now. "
def comp(prompt,n=8):
data=json.dumps({"prompt":prompt,"max_tokens":n,"cache_prompt":True,"temperature":0.6}).encode()
req=urllib.request.Request(URL,data=data,headers={"Content-Type":"application/json"})
try:
with urllib.request.urlopen(req,timeout=600) as r: return r.status, r.read().decode()
except urllib.error.HTTPError as e: return e.code, e.read().decode()
except Exception as e: return -1, str(e)
MAXLINES=340; lines=[]; peak=0; t0=time.time()
for it in range(1200):
if len(lines)>=MAXLINES:
lines=lines[:random.randint(30, 90)] # DEEP rewind from high depth -> strands high-pos cells
elif lines and random.random()<0.6:
lines[-1]=("d",it) # shallow divergence -> checkpoint rewind
lines.append(("a",it))
prompt="".join(sent(hash(x)%99999) for x in lines)
st,body=comp(prompt)
if st==-1: print(f"[NETERR] it={it} {body[:150]}",flush=True); continue
if st>=500 or "exceed" in body.lower():
print(f"[JAM] it={it} elapsed={time.time()-t0:.0f}s lines={len(lines)} peak_ptok={peak} status={st} body={body[:200]}",flush=True); break
try:
u=json.loads(body).get("usage",{}); pt=u.get("prompt_tokens",0); peak=max(peak,pt)
if it%10==0: print(f"it={it} t={time.time()-t0:.0f}s lines={len(lines)} ptok={pt} peak={peak}",flush=True)
except Exception as e: print(f"[PARSE] it={it} {body[:100]}",flush=True)
else: print(f"[NOJAM] done 1200 iters peak_ptok={peak}",flush=True)
print("FIN",flush=True)# 1. isolated debug server: small ctx, KV-cache debug on (substitute your own DSV4-Flash GGUF path)
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1,2,3,4 GGML_CUDA_NO_PINNED=1 LLAMA_KV_CACHE_DEBUG=2 ./llama-server --port 10099 --host 127.0.0.1 --flash-attn on --jinja --parallel 1 --model <DSV4-Flash GGUF, shard 1 of 5> --ctx-size 16384 --ubatch-size 512 --fit on --fit-ctx 16384 --fit-target 1024,1024,2048,2048,2048 --no-mmap --temp 0.6 --top-p 0.95 -lv 5
# wait for {"status":"ok"} on /health
# 2. drive the churn (grows depth, then deep-rewinds into the checkpoint-restore path)
python3 churn.py # seed=11, targets :10099/v1/completionsFix branch
EDIT: I did another rebase and rework. The current branch is https://github.com/TacoTakumi/llama.cpp/tree/dsv4-rs-rollback
A fix branch is up at https://github.com/TacoTakumi/llama.cpp/tree/dsv4-swa-churn-fix
A proper seq_rm of the diverged suffix so only the delta re-prefills, which removes both symptoms. This builds on @danielhanchen's own DeepSeek-V4 Checkpointing fix (PR #25402) and is forked from commit 15f7321. A PR may follow depending on maintainer interest.
AI assistance
I'm a 30+ year experience software developer and sysadmin. Claude Fable xhigh meaningfully assisted the root cause analysis, the reproduction, and iterating the fix, under my direction. I reviewed every line, understand it, and can explain any part.
Not a duplicate
This is not a duplicate of any of these issues:
First Bad Commit
No response
Relevant log output
Logs
# Crash (version 9924, 90e0f5cfc) -- churn jams deterministically at iteration 29:
[JAM] it=29 elapsed=303s lines=30 peak_ptok=1248 status=500 body={"error":{"code":500,"message":"Context size has been exceeded.","type":"server_error"}}
# No CUDA / abort / OOM anywhere in the server log:
$ grep -acE "CUDA error|ggml_abort|out of memory|GGML_ASSERT" step1.server.log
0
# find_slot state at the jam (LLAMA_KV_CACHE_DEBUG=2). Cell-map is from the deterministic
# repro run; the version-9924 master run reproduces the identical signature at head=231:
find_slot: stream[0], n = 768, used = 768, head = 225, size = 768, n_swa = 128
find_slot: stream[0] min[0] = 225, max[0] = 992
decode: failed to find a memory slot for batch of size 1
srv decode: Context size has been exceeded. off = 219, n_batch = 1, ret = 1
# Stall - per-turn KV routing over the 30 churn turns:
routing (30 turns): 1 append, 0 delta, 27 restore, 2 reset
Name and Version
$ llama-cli --version
version: 9924 (90e0f5c)
built with GNU 13.3.0 for Linux x86_64
Operating systems
Linux
GGML backends
CUDA
Hardware
GPU: 5x NVIDIA CUDA (2x RTX 3090, RTX 5060 Ti, 2x RTX 4060 Ti), 96 GiB total VRAM
CPU: Intel Core i5-13600K (14 cores / 20 threads)
RAM: 128 GB DDR4-3200
OS: Linux x86_64 (Ubuntu 24.04)
Models
DeepSeek-V4-Flash (DSV4-Flash), GGUF, MXFP4 quant, 5 shards.
Problem description & steps to reproduce
During an agentic coding session, after several turns, I get a crash with a false "Context size has been exceeded" at ~1250 tokens against a 16k context. It's SWA find_slot exhaustion of the 768-cell window, not a real context limit, with 0 CUDA errors (no OOM/abort).
Also, before the crash, throughout the session every divergent turn stalls. Every divergent turn re-prefills thousands of tokens from a checkpoint boundary instead of just the delta, so the agent sits on "Working" with nothing streaming. The same churn eventually crashes.
Both fall out of the same no-partial-rollback limitation; one proper seq_rm of the diverged suffix fixes both.
The window full so no slot even at ~1250 tokens, not a real ctx limit, 0 CUDA errors proves it's not a GPU fault.
The root cause:
DSV4-Flash raw attention is a 128-token sliding window on a 768-cell SWA KV cache, the only cache find_slot runs on. find_slot reuses an occupied cell only if SWA-masked. Also the same sequence causal reuse path is commented out (src/llama-kv-cache.cpp ~1043-1047) on the invariant "purge futures beforehand."
Normal models satisfy that with seq_rm(seq_id, n_past, -1) before re-decoding a divergent suffix. DSV4 can't do a partial seq_rm (src/llama-kv-cache-dsv4.cpp ~1153). The compressor ring state can't roll back mid-block. So the server rewinds via a checkpoint-restore that repopulates the SWA window without purging futures.
The crash: re decode refills positions inside the full 768-cell window and find_slot can neither reuse the occupied same-seq cells nor reclaim the live top ~128 window cells resulting in a false "context exceeded."
The stall: The same no partial rollback. Every divergent turn re prefills from a checkpoint boundary, no delta path.
The defect is in shared KV / server code not a CUDA kernel. Having no CUDA errors backs this so it's expected across backends.
A monotonic growing prompt never jams and the band probe reached 119,788 tokens clean. The crash needs reuse that re-fills occupied cells, not raw depth.
Reproduce with:
churn.py:
Fix branch
EDIT: I did another rebase and rework. The current branch is https://github.com/TacoTakumi/llama.cpp/tree/dsv4-rs-rollback
A fix branch is up at https://github.com/TacoTakumi/llama.cpp/tree/dsv4-swa-churn-fix
A proper seq_rm of the diverged suffix so only the delta re-prefills, which removes both symptoms. This builds on @danielhanchen's own DeepSeek-V4 Checkpointing fix (PR #25402) and is forked from commit 15f7321. A PR may follow depending on maintainer interest.
AI assistance
I'm a 30+ year experience software developer and sysadmin. Claude Fable xhigh meaningfully assisted the root cause analysis, the reproduction, and iterating the fix, under my direction. I reviewed every line, understand it, and can explain any part.
Not a duplicate
This is not a duplicate of any of these issues:
First Bad Commit
No response
Relevant log output
Logs