[doc][llm] Add docs for serving LLMs with TPUs

[jeffreywang88]

Description

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

[gemini-code-assist]

Code Review

This pull request adds documentation and a Python example for topology-aware TPU serving in Ray Serve LLM, introducing support for TPU-V6E and multi-host TPU slices. The review feedback points out a runtime configuration error where max_num_batched_tokens is smaller than max_model_len, and recommends replacing the non-existent google/gemma-4-31B-it model with a valid identifier like google/gemma-2-27b-it across both the code and documentation.

[github-actions]

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[ryanaoleary]

agree with the outstanding comments, once those are resolved LGTM

[dstrodtman]

Docs style review (style and grammar only)

Review from Douglas Strodtman on the Anyscale documentation team. I ran this with Claude Code's help, and I've read and stand behind every comment below.

Scope: style, grammar, and Ray docs conventions, checked against Ray's own writing style guide. I don't own this area. Every technical claim on this page, and final approval, belongs to @eicherseiji and @ryanaoleary. Posting as a comment, not a change request.

Nice page overall. The topology-to-bundles explanation carries the concept well, and the prose is already close to the guide: contractions where they belong, active voice, sentence-case headings, no benefits list, tagged code fences, {doc} and {class} cross-references instead of hardcoded docs.ray.io/en/latest/ URLs. I verified that the SlicePlacementGroup class reference resolves; ray.util.tpu.SlicePlacementGroup is autodoc'd in doc/source/ray-core/api/utility.rst. What's left is small.

The one finding worth calling out above nit level is the missing myst.html_meta.description front matter. All 15 existing pages in serve/llm/user-guides/ have one, so this page would be the only page in the directory without a search-result description.

On the open threads

I read the existing threads before writing anything, and I'm deliberately not re-raising any of them:

• @eicherseiji's suggestions on L63 and L67 are unapplied (the single commit predates the review), and I'd take both as written. My comments avoid those lines.
• The questions about prerequisites, defining "TPU slice", moving the topology note lower, the per-chip bundle use case, whether the SlicePlacementGroup section is an implementation detail, and the tpu-inference executor link are all yours to settle. They're content decisions, not style.
• On the model ID: @ryanaoleary already confirmed the Gemini bot was wrong, so I left it alone everywhere.
• I did not touch max_num_batched_tokens. That's still open above and it's a correctness question, not mine.

Technical questions, not style findings

Two things I noticed while verifying my own suggestions. Both are questions for you, and neither has a suggestion attached.

1. Which Ray package a reader ends up with. The Dockerfile comment on L85 says to use "a released Ray version or wheel URL that contains TPU topology support," and the default is ARG RAY_PACKAGE="ray". @ryanaoleary's successful run used an explicit Ray nightly wheel instead. If topology support isn't in a released Ray yet, a reader who copies this Dockerfile as-is gets a stable ray that may lack the feature. Worth making that pairing explicit either way.
2. blocking=True versus blocking=False. The example uses serve.run(app, blocking=True); the verified run used blocking=False. Probably intentional for a standalone script, just confirming.

Sign-off

Happy to give a docs-side approval once the content questions above settle. Nothing in my list should hold the PR up on its own.

Also, one out-of-diff nit with no comment attached because it's outside a hunk: index.md L3 reads "If you are new" where the guide asks for contractions ("If you're new"). Pre-existing, so entirely optional to fold in here.

[jeffreywang88]

Rebuilt the docs and verified they render cleanly:

[eicherseiji]

Minor typo