Skip to content

[Frontend] vllm-rs: port prompt hashing and content prefix memoization (Rubin) - #170

Merged
YibaiMeng merged 1 commit into
CentML:mlperf-end-multiturn-v1.0-qwen-v2from
JulienDarve:jdarve/native-feh-mlperf-centml-v2
Oct 27, 2026
Merged

YibaiMeng merged 1 commit into
CentML:mlperf-end-multiturn-v1.0-qwen-v2from
JulienDarve:jdarve/native-feh-mlperf-centml-v2

Conversation

@JulienDarve

Copy link
Copy Markdown

Summary

Let vllm-rs, the Rust OpenAI-compatible front end, compute each request's prompt prefix-cache block hashes and send them
with the request, as the Python front end already does with FEH=1 (#72) and VLLM_FEH_MEMO=1 (#141). Without this,
requests from vllm-rs reach EngineCore unhashed and EngineCore hashes every prompt itself. The Dynamo vLLM sidecar
deployment of Qwen3.6 depends on it.

Design

  • EngineCore reports the hashing it accepts in its ready response through a new optional field,
    native_frontend_hash_config: algorithm sha256-pickle5-feh1, block size and none_hash. It is set only when FEH is
    enabled, prefix caching uses sha256 with pickle protocol 5, and the hash block size equals FEH's block
    (vllm/v1/engine/frontend_block_hashing.py, vllm/v1/engine/core.py).
  • vllm-rs (rust/src/engine-core-client/src/frontend_hashing.rs) hashes only with VLLM_NATIVE_FEH=1 and when every
    engine reports the same config. Otherwise it logs why and sends requests unhashed, as before.
  • Hashes are the same pickle-5 SHA-256 block hashes as the Python front end, including cache salt and partial blocks.
    A content-keyed memo with 1024-token checkpoints reuses hashes across turns that share a prefix, like VLLM_FEH_MEMO.
    Hashing runs on a blocking thread, off the async request path.
  • vllm/utils/hashing.py gains a SHA256_PICKLE_PROTOCOL constant. The edits to
    tests/v1/engine/test_frontend_block_hashing.py and the blank-line moves in vllm/v1/engine/core.py are
    ruff formatting only.

Configuration

Env key Read by Default Effect
VLLM_NATIVE_FEH vllm-rs unset (off) 1 enables native hashing when EngineCore advertises a matching config.

FEH and VLLM_FEH_MEMO are unchanged.

Numerics

Exact by design. The hashes are bit-identical to the Python front end's, which unit tests check against CPython pickle-5
output. Prefix-cache behavior is unchanged.

Testing

  • cargo test -p vllm-engine-core-client: 120 passed, including 6 frontend_hashing tests.
  • pytest tests/v1/engine/test_frontend_block_hashing.py: 18 passed.
  • pre-commit run --files <changed files>: no new findings. Pre-existing on the base: mypy (MainThreadRepin
    assignments in core.py, an index at frontend_block_hashing.py:246) and check-root-lazy-imports (vllm.lcd_pdl).
  • End to end: this commit's source tree is identical to JulienDarve/vllm c7228a5a, which was validated in our
    Qwen3.6 serving runs behind Dynamo vLLM sidecars. Results are not shared here.

Not a duplicate

#72 and #141 add this hashing and its memo to the Python front end; this ports the same hashing to vllm-rs. No open
CentML PR touches rust/src/engine-core-client.

Merging

Please merge with a merge commit, not squash or rebase. Our benchmark harness pins this commit ID (f6c69f1c).

AI assistance

AI assistance was used: Codex for the implementation and Claude for preparing this PR. I reviewed every changed line
and the test results above.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
@YibaiMeng
YibaiMeng merged commit 6c7001e into CentML:mlperf-end-multiturn-v1.0-qwen-v2 Oct 27, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants