Add get_tier_prefix_hit_rate(cpu only) - #20
Conversation
58ab026 to
865f4f5
Compare
bc2b3ff to
ddcd635
Compare
ddcd635 to
e082697
Compare
| raw_medium = cls._opt_str(fields, 5) # vLLM medium: "GPU" / "cpu" | ||
| medium = raw_medium.lower() if raw_medium is not None else None |
There was a problem hiding this comment.
为什么要全小写，而不是直接保留原始的字符串。如果只是为了方便转化为Layer枚举没有必要
There was a problem hiding this comment.
确认了一下，KVCacheEvent.medium 当前只用于转换为 Layer 和输出 unsupported-medium 日志。由于 _medium_to_layer() 已经通过 medium.lower() 处理大小写，这里提前转小写是重复的。
已经移除解析阶段的 .lower()处理，并修改对应的单元测试断言
| medium = cls._opt_str(fields, 1) | ||
| block_hashes = [_normalize_block_hash(bh) for bh in fields[0]] | ||
| raw_medium = cls._opt_str(fields, 1) | ||
| medium = raw_medium.lower() if raw_medium is not None else None |
| def get_layer_prefix_hit_rate(self, node_id: str, prompt_ids: list[int], | ||
| layer: Layer = Layer.GPU) -> float: | ||
| """Prefix-cache hit rate for a node at a layer, ∈ [0.0, 1.0]. | ||
|
|
||
| GPU: walk the local reverse index (``replicas_by_block``) along the | ||
| prompt's prefix-hash chain until a hash isn't cached on this node. | ||
| CPU/SSD: placeholder 0.0 (mooncake /batch_query not wired yet). | ||
| """ | ||
| def get_layer_prefix_hit_rate( | ||
| self, | ||
| node_id: str, | ||
| prompt_ids: list[int], | ||
| layer: Layer = Layer.GPU, | ||
| ) -> float: | ||
| """Return the node's longest contiguous prefix hit rate for a layer.""" | ||
| normalized_layer = self._normalize_layer(layer) |
| self.block_size: int | None = None | ||
| self.replicas_by_block: dict[str, set[str]] = {} | ||
| # Per-layer per-replica block counts, maintained alongside replicas_by_block. | ||
| self._replicas_by_layer: dict[Layer, dict[str, set[str]]] = { |
There was a problem hiding this comment.
self._replicas_by_layer 这个变量的意义是什么
There was a problem hiding this comment.
该变量用于把原来的 GPU-only block 反向索引扩展为 GPU/CPU/SSD 分层索引，从而支持按层查询和增删，避免不同缓存层相互影响。GPU 层仍复用 replicas_by_block，主要是为了保持原有replicas_by_block调用兼容。这里是可以不额外加这个的，直接修改replicas_by_block的数据结构也行。
There was a problem hiding this comment.
block-hash的str作为key，一次查询，获取所有Layer的结果
| self._on_all_blocks_cleared(event, update) | ||
| self._on_all_blocks_cleared(update) |
| self._block_size = event.block_size | ||
| update.set_block_size(event.block_size) | ||
| elif layer == Layer.CPU: | ||
| local_hashes = self._translate_cpu_offload_hashes(event, update.node_id) |
There was a problem hiding this comment.
整体排查下，不要出现cpu-offload字样和概念，这个event事件不跟cpu-offload绑定
| to locally-computed prefix hash (str). Used for chained | ||
| hash computation. | ||
| _block_size: Learned block size from first event. | ||
| Remote block hashes are scoped by cache layer and node so CPU offload |
| """Normalize vLLM event block hashes across GPU and CPU events. | ||
|
|
||
| GPU KV events publish the low 64 bits of the internal SHA256 block hash as | ||
| an int. Native CPU offload events can publish the raw internal bytes hash. |
| """Translate token-less CPU offload hashes through the GPU mapping.""" | ||
| local_hashes: list[str] = [] | ||
| for remote_hash in event.block_hashes: | ||
| local_hash = self._get_remote_mapping(Layer.GPU, node_id, remote_hash) |
There was a problem hiding this comment.
为什么这个方法名叫_get_remote_mapping。本意应该是自己计算的hash和vllm计算的hash。引入remote这个概念有歧义，不利于可读性。全局排查下remote这个概念
There was a problem hiding this comment.
这里的 remote 意思是：
- remote_hash：vLLM 进程计算并通过 ZMQ event 发来的 block hash。
- local_hash：router 根据 token IDs 用 compute_hash() 计算的 prefix hash。
_get_remote_mapping() 实际不是“获取远端 mapping”，而是在 decoder 本地字典中，用 vLLM hash 查找对应的 router prefix hash。
这个看起来确实是有歧义的，我把这个命名改一下。
| ) | ||
| return | ||
|
|
||
| if event.token_ids is not None: |
There was a problem hiding this comment.
这个判断很诡异，如果是可hash计算条件判断，那么这个判断条件是不足的。如果是按照gpu cpu判断，那不如直接判断layer即可
There was a problem hiding this comment.
这里不是按 GPU/CPU 分支，而是想区分“能够通过 token IDs 计算 local hash”和“只能复用已有 mapping”两种事件形态。不过 token_ids is not None 确实不足以完整表达可计算条件（没有验证 block 数量和 parent mapping）。计划将其封装为显式的 _can_compute_local_hashes() 判断，并在条件不满足时走 mapping 解析路径。
e082697 to
130f20c
Compare
| self.remote_to_local_block_hash: dict[str, str] = {} | ||
| self.vllm_to_local_block_hash: dict[tuple[Layer, str, str], str] = {} |
There was a problem hiding this comment.
只查询 GPU 命中率时，vllm_block_hash 到本地 prefix hash 的映射主要用于单一缓存层，且事件处理默认处在同一个 vLLM 实例上下文中，因此没有显式加入 node_id；增加 CPU 命中率查询后，同一个 block 会在不同 replica 以及 GPU、CPU 层之间迁移，而 BlockRemoved、AllBlocksCleared 等事件只对其来源实例生效，所以必须将 node_id 加入映射 key，用来隔离不同 vLLM replica 的 hash 命名空间和缓存生命周期，避免一个实例的删除或清空事件误删其他实例仍在使用的映射
| self.block_size: int | None = None | ||
| self.replicas_by_block: dict[str, set[str]] = {} | ||
| # Per-layer per-replica block counts, maintained alongside replicas_by_block. | ||
| self._replicas_by_layer: dict[Layer, dict[str, set[str]]] = { |
There was a problem hiding this comment.
block-hash的str作为key，一次查询，获取所有Layer的结果
3 participants
What does this PR do?
Checklist Before Starting
[{modules}] {type}: {description}(checked by CI){modules}may includecore,interaction,model,env,tools,deployment,reward,dashboard,docs,examples,data,train,ci,build,deps,misc,like[interaction, tools, docs]{type}must be one offeat,fix,refactor,chore,test[BREAKING]to the beginning of the title[1/N][BREAKING][deployment, docs] feat: simplify runtime env configurationTest
--ut
--st-gpu
--st-cpu
--e2e
API and Usage Example
# Add a short example here when the PR changes public behaviorDesign & Code Changes
Checklist Before Submitting
pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always