kv: replace eligible dense causal masks with compact prefixes - #7
Open
GenerelSchwerz wants to merge 23 commits into
Open
kv: replace eligible dense causal masks with compact prefixes#7GenerelSchwerz wants to merge 23 commits into
GenerelSchwerz wants to merge 23 commits into
Conversation
GenerelSchwerz
force-pushed
the
exp/compact-causal-mask-pre-pr4
branch
from
August 21, 2026 05:45
a71c9f6 to
d4183ad
Compare
Give each eligible compact causal descriptor consumer its own graph view so scheduler copies die with their attention split instead of fragmenting the CUDA compute arena. Include the two-turn reproduction runner and the allocation, exactness, PPL, and 4K/30K/34K evidence.
GenerelSchwerz
force-pushed
the
exp/compact-causal-mask-pre-pr4
branch
from
August 21, 2026 06:50
d4183ad to
565233f
Compare
Preserve the exact-source 30K prefill and 128K decode gates, production profiler findings, and the configuration-matched 240K load-to-idle allocation check. Document the measured-versus-inferred causal conclusion and the bounded symbolic/tile-scheduling redesign plan while PR 7 remains draft.
Reuse the proven consecutive KV write-index contract directly as the compact causal descriptor. Load one bound per query tile or warp, derive later bounds by query offset, and keep fully visible CUDA attention tiles on the unmasked inner path. Cover partial vector, tile, and MMA query tiles bit-exactly and record the rejected designs, direct screens, clean-process performance, memory, PPL, profiler, and KVarN compatibility evidence.
GenerelSchwerz
changed the base branch from
beellama-kv-cpu-offload
to
beellama/main
August 22, 2026 22:38
…/compact-causal-mask-pre-pr4
Move the compact prefix bound out of aliased dynamic shared-memory padding and into block-owned storage. Broadcast it per warp and cover D256 F16 MMA layouts with and without GQA on the isolated branch.
Store causal-prefix presence and its retained KV span in one optional value. This removes the invalid bool/count combinations while leaving graph construction, reuse checks, and attention kernels unchanged.
Require the compiled CUDA backend capability whenever the compact descriptor oracle is selected, and remove compact-only source-text assertions from the shared KVarN route-policy test.
Replace the attempt transcript with the durable design, accepted evidence, reproducible gates, source boundaries, and limitations. Require the lifecycle runner model path explicitly instead of embedding a host-specific location.
GenerelSchwerz
marked this pull request as ready for review
August 23, 2026 17:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status
Official isolated head:
ea066a68ad6eb6db701251d5ae23dfea3948ee26Measured runtime head:
222eec2e7c91296b5f9335389dcff0cfc8a08129Exact base:
4b86269fdf001de44dd96e9c9ae26a9e25091cca(beellama/main)This PR is ready for review after a normal merge of current
beellama/main.It remains unmerged. The evidence-only commit after the runtime head changes no
binary source.
Scope
Eligible single-stream contiguous standard-KV graphs pass the existing I64
write-index tensor directly to FlashAttention as an exclusive causal-prefix
descriptor. Unsupported layouts fail closed to the existing dense mask.
threshold, or prompt/decode policy;
host-staging, speculation, or development-branch feature;
<..., false>dense and<..., true>compact vectorspecializations retained across the existing cache-format matrix;
and
removing an invalid boolean/count state without hot-path work.
The main merge is
c542f137c; the isolated lifetime repair is1a3e6fcf3.Focused compiled-capability coverage replaces compact-specific source-text
assertions. The retained two-turn runner now takes model and artifact paths as
arguments rather than embedding a machine model path.
Correctness and quality
12 build jobs under
/tmp/beellama-cuda-build.lock.with no GQA and GQA 6.
replay CTests: 10/10.
-b 512 -ub 256 --chunks 4: identical chunks1.9315, 2.1279, 2.2498, 2.1674and final2.1674 +/- 0.03849; measured increase zero.Every GPU lifecycle used the complete
flock /tmp/beellama-single-gpu.lock -c 'COMMAND'form, fresh processes, andnative progress. The one 128K placement setup failure is explicitly invalid
and contributes no result.
Matched runtime and resources
Qwen3.8 27B, q8_0 K/V, batch/ubatch 512/256, RTX 5070 Ti, CUDA 13.3, driver
610.57.04:
The user-requested closing A4 anchor makes 4K prefill chronological
A B A B A B A. Baseline falls monotonically from 1810.6711 to 1798.3230tok/s. A linear elapsed-time/source model estimates -1.3282 tok/s (-0.0736%)
with 1.7246 tok/s standard error; local interpolated effects are -0.1665%,
-0.0330%, and -0.2066%. This is neutral within observed time/process variation,
not a speedup and not a detected regression. The 128K decode series retains all
four processes/source; its +0.4923% aggregate has wide process variance and is
not promoted to a strict speedup claim.
Independent 100 ms resource sampling measured process-VRAM deltas of
-2 / 0 / -58 MiB at 4K/30K/128K prefill. At 128K, total compute backing
falls 123.5 MiB, while pinned KV context, resident KV, ordinary-host buffers,
and CUDA VMM high-water are unchanged. Allocation-class telemetry and the
process sampler are separate from timing.
Evidence
The durable design, commands, identities, limitations, and results are in
docs/compact-causal-mask-vram-investigation.md. The 135-file local artifactmanifest has SHA-256
d4246230238f39cf643a2a2a1037835cc4a20dd490844abc1dc976d1180206a1.The final main-relative audit is 27 files and contains no PR 4, workspace,
allocator-policy, VMM-policy, server, or development-only source. This PR must
not be merged by the readiness maintainer; normal review owns landing.