Skip to content

Add KVCacheEntry.{key,value}_scale - #882

Merged
akihironitta merged 10 commits into
mainfrom
fp8-stack/cache-types
Nov 19, 2026
Merged

akihironitta merged 10 commits into
mainfrom
fp8-stack/cache-types

Conversation

@aw471

@aw471 aw471 commented Sep 15, 2026 •

Copy link
Copy Markdown
Collaborator

No description provided.

@copy-pr-bot

copy-pr-bot Bot commented Sep 15, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@aw471

aw471 commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

@CodeRabbit full review

@coderabbitai

coderabbitai Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review.

Summary

Summary by CodeRabbit

  • New Features
    • Added support for quantized attention cache entries, including key, value, and query scaling information and output data type handling.
    • Cache entries can now select a requested number of attention heads. Selecting fewer heads retains the leading heads in contiguous copies; selecting all available heads returns the existing entry.
    • Quantized cache head selection preserves query scales and carries over the corresponding key and value scales. Tensor device and data type are preserved during selection.

The PR changes KVCacheEntry to a dataclass and adds leading-head selection. It adds QuantizedKVCacheEntry with scale metadata. ICLBlock uses entry-level selection when recording cache entries and preparing entries for the query pass.

Changes: KV cache head selection

Layer / File(s) Summary
Cache entry contracts
sdm/cache.py, test/test_cache.py
KVCacheEntry is now a dataclass with a leading-head selection method. A helper creates contiguous key and value copies. Tests check selection and tensor storage.
Quantized cache entry
sdm/cache.py, test/test_cache.py
QuantizedKVCacheEntry stores key, value, and query scales and output dtype metadata. Head selection slices key/value scales and retains all query scales. CUDA-decorated tests check selected tensor values, device, dtype, contiguity, and cache size.
ICL cache integration
sdm/models/kumo/tabular/icl.py
ICLBlock uses select_heads when recording cache entries and reducing entries for the query pass.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: Minimal · up to aaf47

The cache-entry changes are mergeable with normal checks. Quantized attention remains outside this PR’s supported behavior.

Pre-merge checks: Passed 2 · Failed 1 · Inconclusive 1

Failed Failed checks: 1 warning · 1 inconclusive

Check name Status Explanation Resolution
Title check Warning The title "Add KVCacheEntry.{key,value}_scale" does not match the changeset. The summary shows KVCacheEntry becoming a dataclass and a new QuantizedKVCacheEntry class carrying key, value, and query … Retitle the PR to describe the main change, for example: "Refactor K/V cache types and add QuantizedKVCacheEntry" (matching PR #882's stated purpose). If the scale fields are meant to live on KVCacheEntry, the code or the title must change …
Description check Inconclusive The author left no pull request description. The check cannot find any description text to compare with the changeset, so the description does not describe the changes. Add a pull request description that summarizes the changes. Name the main changes, such as the KVCacheEntry dataclass conversion, the new QuantizedKVCacheEntry fields, head selection behavior, and the new tests in test/test_cache.py.
Passed Passed checks: 2 passed

Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title "Add KVCacheEntry.{key,value}_scale" does not match the changeset. The summary shows KVCacheEntry becoming a dataclass and a new QuantizedKVCacheEntry class carrying key, value, and query scales plus a dtype. The base KVCacheEntry gains no key_scale or value_scale fields. The title names a change that is not present, and it omits the main refactor.

Resolution

Retitle the PR to describe the main change, for example: "Refactor K/V cache types and add QuantizedKVCacheEntry" (matching PR #882's stated purpose). If the scale fields are meant to live on KVCacheEntry, the code or the title must change so they agree.

Finishing Touches

🧪 Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between c64ae9b and aaf4735.

Files selected for processing: 3

  • sdm/cache.py
  • sdm/models/kumo/tabular/icl.py
  • test/test_cache.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@aw471
aw471 force-pushed the fp8-stack/cache-types branch 2 times, most recently from 3b65b26 to 9889395 Compare October 7, 2026 01:27
@aw471
aw471 marked this pull request as ready for review October 8, 2026 07:35
@aw471
aw471 force-pushed the fp8-stack/cache-types branch 2 times, most recently from e1fa8d5 to c64ae9b Compare October 12, 2026 02:50
@aw471 aw471 changed the title [FP8 1/4] Refactor K/V cache types and add quantized entries [FP8 1/5] Refactor K/V cache types and add quantized entries Oct 13, 2026
@aw471
aw471 force-pushed the fp8-stack/cache-types branch from c64ae9b to aaf4735 Compare October 24, 2026 16:14
@codecov

codecov Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.59%. Comparing base (82c4d79) to head (f56da4c).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #882   +/-   ##
=======================================
  Coverage   84.59%   84.59%           
=======================================
  Files         267      267           
  Lines       14212    14220    +8     
=======================================
+ Hits        12022    12029    +7     
- Misses       2190     2191    +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

akihironitta and others added 8 commits November 18, 2026 18:38
KVCacheEntry gains optional key and value scales and a dequantize method that returns the projections in a given dtype. Scales move with the entry across devices and count toward Cache.size().
Decoding moves to the per-dtype classes that come with the KV cache dtype option, so KVCacheEntry only carries the optional scales for now.
@akihironitta akihironitta changed the title [FP8 1/5] Refactor K/V cache types and add quantized entries Add KVCacheEntry.{key,value}_scale Nov 19, 2026
@akihironitta
akihironitta enabled auto-merge (squash) November 19, 2026 02:50
@akihironitta
akihironitta merged commit d0080aa into main Nov 19, 2026
6 of 7 checks passed
@akihironitta
akihironitta deleted the fp8-stack/cache-types branch November 19, 2026 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants