[llm][kv][doc] Add KV-aware routing guide and native KV cache offloading docs

[jeffreywang88]

Description

Documents the KV-aware router and native vLLM CPU offloading. Adds a KVAwareRouter guide covering install, configuration, router comparison, scoring, and tuning, and extends the KV cache offloading guide with the native backend, backend compatibility, and the new Grafana observability panels.

AI assistance was used; I reviewed every line.

go

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

[gemini-code-assist]

Code Review

This pull request introduces comprehensive documentation for KV-aware routing (KVAwareRouter) and native vLLM KV cache offloading in Ray Serve LLM, including a new user guide and updates to existing documentation. The review feedback focuses on improving the clarity, grammar, and professionalism of the newly added text, such as fixing parallel verb structures and rephrasing awkward or colloquial sentences.

[dstrodtman]

Docs style review (style and grammar only)

Reviewed by Douglas Strodtman (@dstrodtman) on the Anyscale docs team, with Claude Code assisting. I read every comment below and stand behind each one.

Scope: style, grammar, and Ray docs conventions, against Ray's documentation style guide. I reviewed the prose, not the behavior it describes. Every technical claim on these pages is yours and sits inside your own team's domain, so nothing below should be read as vouching for or questioning the accuracy of the feature description.

On the whole this is strong writing and closer to the style guide than most docs PRs I see. The KV-aware routing page in particular does what the guide asks for and most pages don't: it explains the mechanism in prose rather than a numbered list, states counts before lists, and keeps the limitations in one section at the end. Most of what's below is line-level.

What's in the review

Twenty inline comments, almost all with a click-acceptable suggestion. Grouped:

• Mechanical: two Unicode curly apostrophes this PR introduces (kv-cache-offloading.md L13, routing-policies.md L89), via -> through, session id -> session ID, input/output -> input or output, three missing contractions, 1. on every ordered-list item.
• One subject-verb agreement bug: kv-aware-routing.md L195 reads "when tokenizing and scoring at the ingress bounds throughput," where the parallel sentence at L112 correctly reads "bound."
• One judgment call: the "Benefits of KV cache offloading:" list at kv-cache-offloading.md L17-21. The guide is explicit that benefits lists are usually a cut, and the paragraph right below it already does the job better. Suggestion folds it into prose. Push back if you'd rather keep it.
• Voice: a few passives and allowing/simply instances where naming the actor is shorter.
• Consistency: bold-vs-italic on the first definition of token load, sentence capitalization after bold labels, and a toctree order in index.md that no longer matches the reordered prose list.

Already covered, not reopening

The bot flagged improving TTFT (offloading L20), how much prefill each candidate remains (routing L18), and credits that match less than (offloading L111), and you agreed to all three. I haven't refiled any of them. Two of my suggestions touch those exact lines, so I've folded your agreed wording into the suggestion text rather than clobbering it. The two bot comments you haven't answered yet, the one part (routing L130) and bound as a verb (routing L191), I'm deliberately leaving to you; for what it's worth I read both as fine as written.

Two questions for you, not style findings

1. Grafana panel names. Left as a note on the panel table rather than a suggestion. The doc's seven rows collapse the nine panels the dashboard actually defines, and every title in the source carries a KV Offload: prefix the table drops. Details and the verified list are in that comment. Readers match those strings literally, so it's worth a decision either way.
2. Alpha warning placement. KVAwareRouter carries an alpha warning at the top of its own guide, but the routing-policies.md architecture section and the user-guides/index.md bullet both introduce it with no stability signal. Intentional?

What I checked, so you don't have to re-verify it

Everything below resolved clean against master, so nothing in this review turns on it:

• All twelve environment variables and experimental_configs keys exist, and every default in both tables matches the source: RAY_SERVE_LLM_KV_TOKEN_STAGING_TTL_S 60, MAX_ENTRIES 8192, MAX_BYTES 1024**3 (1 GiB), RAY_SERVE_INGRESS_ROUTER_REPLICAS_PER_NODE 1, RAY_SERVE_LLM_ENABLE_DECODE_BLOCK_PROGRESS 0, KV_INDEXER_THREADS 4, KV_EVENTS_PORT_BASE 5557, KV_TOKEN_PORT_BASE 7557.
• Both KVAwareRouter import forms match the tests: from ray.serve.llm.request_router import KVAwareRouter and the YAML ray.serve.llm.request_router.KVAwareRouter.
• kv_offloading_size and kv_offloading_backend: "native" match routing_policies/kv_aware/vllm/kv_events.py.
• Every ray_vllm_kv_offload_*, ray_vllm_external_prefix_cache_hits_total, and ray_vllm_kv_offload_lookup_sync_delay_seconds metric exists in the dashboard panel source.
• Every cross-reference target resolves, including the two labels this PR adds (native-kv-cache-offloading, lmcache-kv-offloading).

I ran Vale by hand for signal even though .vale.ini doesn't scope it to these paths. It agreed on via, the three missing contractions, and the undefined TTFT; the rest of its output was vocabulary noise on prefill and LMCache.

Approving on the docs side. None of the twenty comments is a blocker, so don't let them hold the PR. Take the ones you agree with, skip the rest, and merge whenever the remaining checks come back green. If you'd rather land as-is and fold the mechanical items into a follow-up, that's fine too.

[eicherseiji]

Add a note reference for RAY_SERVE_HAPROXY_INGRESS_REQUEST_ROUTER_BUFSIZE?

[eicherseiji]

Double checking that kv_offloading_size doesn't apply with native?

[jeffreywang88]

Hmm it does. kv_offloading_size works with kv_offloading_backend:"native".

[jeffreywang88]

Does this make it sound like they’re incompatible?

[eicherseiji]

clarification: lmcache still requires kv_offloading_size to be set to enable offloading, but the size is managed by LMCache

[jeffreywang88]

Removed the confusing part: a1c0fe2

[jeffreywang88]

Add a note reference for RAY_SERVE_HAPROXY_INGRESS_REQUEST_ROUTER_BUFSIZE?

Agree, done in ec3765b.

[jeffreywang88]