Add NVIDIA Dynamo upstream guide, cassettes, and CI replay tests by franciscojavierarceo · Pull Request #212 · vllm-project/agentic-api

franciscojavierarceo · 2026-08-27T10:40:25Z

Summary

Agentic API works in front of NVIDIA Dynamo with no code changes. This PR documents how, records real Dynamo traffic as cassettes, and adds CI coverage so the Dynamo upstream stays working without a GPU in CI.

Verified locally against Dynamo 1.4.1 (dynamo.frontend + dynamo.vllm worker, vLLM 0.26.0) serving openai/gpt-oss-20b on a GB10.

What Dynamo changes vs. `vllm serve`

Dynamo's /v1/responses is stateless: previous_response_id returns 501 Validation: previous_response_id is not supported. The gateway never forwards it (it rehydrates the full item history), which is what makes stateful conversations work anyway. The new tests pin that upstream request shape.
Reasoning/tool parsing is done by the Dynamo frontend: the worker needs --dyn-reasoning-parser / --dyn-tool-call-parser; vLLM's --tool-call-parser is rejected and --reasoning-parser is ignored.

Changes

docs/guides/dynamo-upstream.md — install, launch flags, verification curls, troubleshooting table; linked from README and mkdocs nav.
tests/cassettes/dynamo/ — stateful two-turn and function-call cassettes (streaming + non-streaming) recorded from the live Dynamo, plus record_dynamo_cassettes.sh (turn 2 is recorded from the hydrated item array since the recorder's own chaining can't be used against a stateless upstream).
tests/dynamo_cassette_test.rs — 4 replay tests: outputs match the recordings (OK → APPLE, get_stock_price with NVDA), and the upstream requests carry the full history with no previous_response_id.
.github/workflows/rust.yml — new dynamo-upstream job that validates the cassette files and runs the replay tests (they also run under the existing cargo test job).
CHANGELOG.md — Unreleased entry.

Test plan

cargo test -p agentic-server-core --test dynamo_cassette_test — 4 passed
cargo fmt --check, cargo clippy --all-targets -- -D warnings, pre-commit hooks
Live: gateway on :9001 → Dynamo on :8001 → gpt-oss-20b; two-turn previous_response_id and function tool call succeed; same second request straight to Dynamo returns 501
CI dynamo-upstream job green

🤖 Generated with Claude Code

https://claude.ai/code/session_01BGqhGxWPxyip8XF6DmZ7rc

Copilot

Pull request overview

Adds documented and replay-tested NVIDIA Dynamo support as a stateless inference upstream.

Changes:

Adds Dynamo setup, verification, and troubleshooting guidance.
Records streaming and non-streaming stateful/tool-call traffic.
Adds four replay tests and dedicated CI validation.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`README.md`	Links the Dynamo guide.
`mkdocs.yaml`	Adds guide navigation.
`docs/guides/dynamo-upstream.md`	Documents Dynamo deployment and usage.
`crates/agentic-server-core/tests/dynamo_cassette_test.rs`	Tests rehydration and function calls.
`crates/agentic-server-core/tests/cassettes/record_dynamo_cassettes.sh`	Records Dynamo scenarios.
`crates/agentic-server-core/tests/cassettes/README.md`	Documents cassette recording.
`crates/agentic-server-core/tests/cassettes/dynamo/stateful_turn2_input.json`	Defines hydrated turn-two input.
`crates/agentic-server-core/tests/cassettes/dynamo/dynamo-stateful-openai-gpt-oss-20b-streaming.yaml`	Records streaming stateful traffic.
`crates/agentic-server-core/tests/cassettes/dynamo/dynamo-stateful-openai-gpt-oss-20b-nonstreaming.yaml`	Records non-streaming stateful traffic.
`crates/agentic-server-core/tests/cassettes/dynamo/dynamo-tool-call-auto-openai-gpt-oss-20b-streaming.yaml`	Records streaming function-call traffic.
`crates/agentic-server-core/tests/cassettes/dynamo/dynamo-tool-call-auto-openai-gpt-oss-20b-nonstreaming.yaml`	Records non-streaming function-call traffic.
`CHANGELOG.md`	Notes Dynamo support and tests.
`.github/workflows/rust.yml`	Adds Dynamo cassette CI.

Suppressed comments (2)

crates/agentic-server-core/tests/dynamo_cassette_test.rs:153

Use the normative “client-executed function tools” wording here; TERMINOLOGY.md:16 explicitly lists “client-owned tool” as terminology to avoid.

    assert_eq!(requests.len(), 1, "client-owned tools take one model call");

docs/guides/dynamo-upstream.md:78

Update the expected field name to reasoning_content. Also, TERMINOLOGY.md:317-320 calls this exposed plaintext “reasoning text,” distinguishing it from raw chain of thought.

`content` should hold the answer and `reasoning` the chain of thought. If the answer starts with `analysis`, the

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    assert_eq!(requests.len(), 2, "one upstream call per turn");
+    for request in requests {
+        assert!(
+            request.get("previous_response_id").is_none_or(Value::is_null),


+    assert_eq!(
+        request_input_texts(&requests[1]),
+        vec![TURN1_PROMPT, "OK", TURN2_PROMPT],


+    run_stateful_two_turn(true).await;
+}
+
+async fn run_client_function_call(streaming: bool) {


 | `record_custom_tool_cassettes.sh` | Matching two-turn custom-tool flows (streaming + non-streaming) | gateway and OpenAI reference |
 | `record_mcp_cassettes.sh` | Native MCP counter tool discovery and calls (streaming + non-streaming) | gateway and OpenAI reference |
 | `record_web_search_cassettes.sh` | Matching web-search calls (streaming + non-streaming) | gateway and OpenAI reference |
+| `record_dynamo_cassettes.sh` | Stateful two-turn and client function-call cassettes (streaming + non-streaming) | NVIDIA Dynamo frontend |


Dynamo's frontend speaks the same /v1/responses surface as vLLM but is stateless: it rejects previous_response_id with 501. The gateway already rehydrates the item history and sends it upstream, so it works in front of Dynamo unchanged. This records that behavior against a live Dynamo 1.4.1 + gpt-oss-20b worker and pins it in CI. - docs/guides/dynamo-upstream.md: install, worker flags (--dyn-* parsers, file discovery, memory sizing), verification, troubleshooting - cassettes/dynamo: stateful two-turn and client-executed function tool recordings (streaming + non-streaming) plus record_dynamo_cassettes.sh, which builds turn 2 from turn 1's recorded assistant message - tests/dynamo_cassette_test.rs: replays the cassettes and asserts the upstream requests carry the exact hydrated item history, never a previous_response_id key - rust.yml: dedicated dynamo-upstream job validating the cassettes and running the replay tests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BGqhGxWPxyip8XF6DmZ7rc Signed-off-by: Francisco Javier Arceo <arceofrancisco@gmail.com>

franciscojavierarceo · 2026-08-27T10:55:19Z

Addressed the Copilot review:

previous_response_id assertion now requires the key to be absent (is_none()), not merely null.
Turn-2 upstream request is compared structurally against the recorded turn-2 input (type/role/status/id/content), with the assistant id taken from turn 1's payload. The recorder now derives turn 2 from turn 1's recorded assistant message instead of a static fixture; cassettes re-recorded.
Renamed helpers/tests to run_function_tool_call / dynamo_function_tool_call_* and switched prose to the TERMINOLOGY.md wording (client-executed function tool, gateway-executed built-in tool).
Fixed the verification jq to select reasoning_content (confirmed against Dynamo 1.4.1).

Commit is squashed and signed off for DCO.

Replace the Dynamo-specific inline validation in CI with scripts/validate-cassettes.py, which checks every cassette under tests/cassettes: request path/method/body, response status, exactly one of body or sse, JSON-decodable SSE data lines, and a terminal event or [DONE] marker for 2xx streams. Dynamo-specific behavior stays in the Rust replay test. Messages cassettes for Dynamo are tracked in #213. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BGqhGxWPxyip8XF6DmZ7rc Signed-off-by: Francisco Javier Arceo <arceofrancisco@gmail.com>

The guide implied a separate vllm serve was required next to Dynamo. It is not: ai-dynamo[vllm] installs the vLLM engine and dynamo.vllm runs it. Drop the coexistence framing and use port 8000 for the frontend. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BGqhGxWPxyip8XF6DmZ7rc Signed-off-by: Francisco Javier Arceo <arceofrancisco@gmail.com>

- Install with "ai-dynamo[vllm]==1.4.1" instead of --prerelease=allow, which resolved an unpinned ai-dynamo to a 1.5.0.dev build. Cassettes re-recorded from a real 1.4.1 install. - Dynamo /health returns 200 healthy with an empty instances list before any worker registers, so the gateway's readiness probe does not imply a loaded model; document /v1/models/{model}/ready (404 until registered). - Drop the host-specific --gpu-memory-utilization 0.15 from the launch command and explain sizing; only claim the verified gpt_oss/harmony parser pair and point at --help for other model families. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BGqhGxWPxyip8XF6DmZ7rc Signed-off-by: Francisco Javier Arceo <arceofrancisco@gmail.com>

franciscojavierarceo · 2026-08-27T11:11:54Z

Follow-up from an external review pass (Codex) — all verified against a live install:

Release pinning / uv prerelease: --prerelease=allow with an unpinned ai-dynamo had actually installed 1.5.0.dev20260816, not 1.4.1. The guide now pins ai-dynamo[vllm]==1.4.1 (resolves without the prerelease flag), explains the trap, and the cassettes were re-recorded from a real 1.4.1 install (dynamo_cassette_test 4/4, validator 88/88).
Health-check semantics: Dynamo /health is 200 {"status":"healthy","instances":[]} before any worker registers, so the gateway's LLM ready probe does not mean a model is loaded. Documented /v1/models/{model}/ready (404 until registration, then ready: true) plus a troubleshooting row.
Frontend configuration: noted the frontend defaults (port 8000, round-robin over registered workers) and removed the earlier implication that a separate vllm serve is needed — dynamo.vllm is the vLLM engine.
gpt-oss memory: dropped the host-specific --gpu-memory-utilization 0.15 from the launch command; the guide now says gpt-oss-20b fits a 24 GB GPU at vLLM's default and explains when/why to lower it.
Reasoning parsing: only the verified gpt_oss + harmony pair is asserted; other families point at python -m dynamo.vllm --help.

franciscojavierarceo requested review from bbrowning, jiahuei, leseb, maralbahari, noobHappylife, qandrew and tjtanaa as code owners August 27, 2026 10:40

franciscojavierarceo requested a balanced review from Copilot August 27, 2026 10:46

Copilot started reviewing on behalf of franciscojavierarceo August 27, 2026 10:46 View session

Copilot AI reviewed Aug 27, 2026

View reviewed changes

franciscojavierarceo force-pushed the dynamo-upstream-guide branch from e5bb120 to a907480 Compare August 27, 2026 10:55

franciscojavierarceo mentioned this pull request Aug 27, 2026

Record Messages API cassettes against NVIDIA Dynamo #213

Open

franciscojavierarceo and others added 3 commits August 27, 2026 07:04

maralbahari approved these changes Aug 27, 2026

View reviewed changes

franciscojavierarceo merged commit ee2f04a into main Aug 27, 2026
8 checks passed

Navigation Menu

Uh oh!

Add NVIDIA Dynamo upstream guide, cassettes, and CI replay tests - #212

Add NVIDIA Dynamo upstream guide, cassettes, and CI replay tests#212
franciscojavierarceo merged 4 commits into
mainfrom
dynamo-upstream-guide

franciscojavierarceo commented Aug 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

franciscojavierarceo commented Aug 27, 2026

Uh oh!

franciscojavierarceo commented Aug 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

franciscojavierarceo commented Aug 27, 2026

Summary

What Dynamo changes vs. vllm serve

Changes

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

franciscojavierarceo commented Aug 27, 2026

Uh oh!

franciscojavierarceo commented Aug 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

What Dynamo changes vs. `vllm serve`