fix(rollout-gateway): parse tool calls with model-native response schemas, auto-detected from the chat template

[luyuzhe111]

The issue

The gateway's default derender parsed tool calls with a regex for one specific XML format (<tool_call><function=...>). Qwen3 emits JSON tool calls (<tool_call>{"name": ..., "arguments": ...}</tool_call>), so the regex never matched — and the failure was invisible three times over:

1. parse_tool_uses hardcoded ill_formed=False, so no flag was raised;
2. the raw <tool_call> block flowed back to the agent as plain text, so Strands saw a turn with no tool calls and simply ended the conversation;
3. GSM8K reward only reads the final #### <answer>, and Qwen3-4B can solve GSM8K mentally — so the previous "validated" run reached ~0.93 val reward while never firing a single tool call. Every trajectory was single-turn; the multi-turn concatenation and tool loss-mask paths (the gateway's core purpose) had never actually been exercised.

The engine_kwargs.vllm.tool_call_parser=hermes flags in the run script were no-ops: verl's token-in/token-out channel never touches vLLM's chat-completions layer where that parser lives.

The fix

Make the default correct instead of adding configuration:

1. Schema-based derender

• New rollout_gateway/response_schemas.py: response-schema dicts vendored from huggingface/trl (Apache-2.0, baseline 7073af94, see NOTICE) covering Qwen2.5 / Qwen3 / Qwen3-Instruct-2507 / Qwen3-VL (JSON <tool_call>), Qwen3.5 / Qwen3.6 / Nemotron-3 (XML <function=>), GLM4-MoE, and GPT-OSS — plus a sha256(chat_template) → schema table.
• HfTemplateRenderer hashes the tokenizer's chat template at construction; on a match it derenders the whole output in one tokenizer.parse_response(text, schema=...) pass — reasoning, text, and tool calls together, ordered by the schema's anchored regex (a literal <tool_call> inside a think block cannot corrupt the split). A schema is pure data; the parsing engine ships in transformers (>= 5.0).
• Detection is byte-exact and loud: fine-tuned checkpoints inherit the chat template verbatim and keep resolving; an unrecognized template with tools in play now raises instead of silently guessing (the old regex remains available by explicit injection, tool_parser=parse_tool_uses). Malformed model output degrades in place to raw text with ill_formed=True — flagged, never a 500, never partial extraction.
• Explicitly injected stage parsers disable detection entirely, so the slime backend's SGLang-detector path is unchanged.
• The verl integration needed zero changes: gateway_host's existing bare HfTemplateRenderer(tokenizer) construction now does the right thing.

2. Session-key contract cleanup

The agent previously derived its gateway session key from the ACR runtime session id (context.session_id). The trainer now passes it explicitly as _rollout.api_key (it still equals the ACR runtimeSessionId, so already-deployed agent images keep working), and rl_app.py reads it from the payload — removing a dead assignment left by the #91/#93 merge and keeping the agent free of any inference-infrastructure awareness.

Validation

• 46 gateway + 45 verl-backend tests (the schema paths are exercised against transformers' real parsing engine; a new always-on test_template_hashes_live.py verifies every vendored template hash against the actual hub templates — one repo per registered hash, incl. both Qwen3.5 think/nothink variants). Real-tokenizer probes: Qwen3-4B-Instruct-2507 auto-detects qwen3, Qwen3.6-35B-A3B auto-detects qwen3_5, both parse their native formats correctly.
• E2E GSM8K GRPO run (stock verl.trainer.main_ppo, Qwen3-4B-Instruct-2507 full-FT on 8 GPUs, live AgentCore Runtime rollouts):
• Tool calls now fire: step-0 validation num_turns mean 2.20 / max 5 (verl counts the user message, so num_turns=2 = one LLM turn with no tool call; >2 = tool loop). The previous run was uniformly 2 — no tool call ever — so this is the first real exercise of multi-turn concatenation and tool loss-mask boundaries.
• Zero ill_formed turns, zero parse errors across all rollouts (~70k over two full epochs).
• Step-0 val reward 0.585, reaching ~0.92 in one epoch. Note: training num_turns rises early (mean ~3.1 by step 6) then collapses back to 2.0 by step ~16 — the policy learns to drop the calculator, since the answer-only GSM8K reward gives tool use no advantage for a model that solves GSM8K mentally. That is a task/reward-design property, orthogonal to this fix; the parsing fix is what made those early tool turns possible (and trainable) at all.
• agent.num_workers=2 also validated: two AgentLoopWorkers, each with its own gateway on an auto-assigned port, identical metrics.

Also in this PR

• Run script: removed the two no-op vLLM tool-parser flags (with a comment on where parsing actually happens).
• gateway extra: transformers floor >=4.44 → >=5.0 (tokenizer.parse_response); jmespath declared (schema transforms). verl-experimental extras: transformers pinned 5.12.1 (compatible with the pinned verl 78bba31d and vllm 0.23.0).
• NOTICE: TRL attribution. AGENTS.md: replaced a stale output_parser
  paragraph describing a never-implemented seam; added the TRL row to the
  vendoring/re-sync table.
• Review cleanups: render() requests bare token ids (return_dict=False) instead of unwrapping a BatchEncoding; llama3 schema dropped (not a fine-tuning target); tautological renderer/schema tests removed in favor of golden-path + degradation-contract coverage.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[Linbo-Liu]

Why copy them over from: https://github.com/huggingface/trl/blob/main/trl/chat_template_utils.py?

[luyuzhe111]

otherwise we will need to add TRL as a dependency : )