Repository navigation
Add KVCacheEntry.{key,value}_scale - #882
Conversation
|
@CodeRabbit full review |
✅ Action performedFull review finished. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. Summary
Merge Risk:
|
3b65b26 to
9889395
Compare
e1fa8d5 to
c64ae9b
Compare
c64ae9b to
aaf4735
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #882 +/- ##
=======================================
Coverage 84.59% 84.59%
=======================================
Files 267 267
Lines 14212 14220 +8
=======================================
+ Hits 12022 12029 +7
- Misses 2190 2191 +1 ☔ View full report in Codecov by Harness. |
KVCacheEntry gains optional key and value scales and a dequantize method that returns the projections in a given dtype. Scales move with the entry across devices and count toward Cache.size().
Decoding moves to the per-dtype classes that come with the KV cache dtype option, so KVCacheEntry only carries the optional scales for now.
aaf4735 to
f56da4c
Compare
KVCacheEntry.{key,value}_scale
No description provided.