[roborluo]

Draft. Depends on a dataset release that sets max_tokens on every turn. With the current recordings the loader will reject the dataset, by design. It is opened now so the client side is visible next to the dataset discussion.

Problem

Live replies are discarded and the next turn is built from the recording. A longer or shorter live reply therefore changes only the output token count, and with it the output tokens per second, without changing anything the server has to do next. The scenario allows EOS and does not lock output-shaping --extra-inputs (max_tokens, min_tokens, stop, reasoning effort, chat-template options, sampling), so reply length varies with sampling and with submitter settings.

Change

• agentperf-v2 sets require_ignore_eos=True, so each reply runs to its turn's max_tokens.
• New ScenarioSpec.require_turn_max_tokens: the DAG loader rejects a dataset in which any request turn lacks max_tokens, or whose per-turn extra sets ignore_eos=false or a forbidden key (per-turn extra is merged last). Turns that send no request, such as orchestrator turns, are skipped. This is a load error, so --unsafe-override does not bypass it.
• New ScenarioSpec.forbid_extra_input_keys: agentperf-v2 rejects --extra-inputs keys that would override output length or stopping, reasoning, chat templates, constrained decoding or the main sampling settings. Extra inputs are merged into each request body after the per-turn max_tokens, so they would otherwise replace it.
• New ScenarioSpec.require_endpoint_types: agentperf-v2 accepts only --endpoint-type chat, so a custom payload template cannot replace max_tokens.
• The agentperf-v2 tutorial documents the behavior.

Dataset side (needed before merge)

• Set each turn's max_tokens to that turn's recorded output token count (the completion_tokens of the recorded response, reasoning and tool-call tokens included). That is more exact than any client-side retokenization.
• The last turn of each session has no recorded reply, so it needs one fixed rule, for example a fixed max_tokens.

Follow-up (not in this PR)

A per-run check that each reply's streamed token count matches its turn's max_tokens would catch servers that stop early despite ignore_eos.

Tests

• tests/unit/dataset/loader/test_dag_jsonl_turn_max_tokens.py: agentperf-v2 accepts turns with max_tokens, rejects a request turn without it or with an overriding per-turn extra, and accepts orchestrator sessions. No requirement without a scenario or under another scenario.
• tests/unit/common/scenario/test_agentperf_v2_output_length.py: agentperf-v2 injects ignore_eos, rejects each output-shaping --extra-inputs key, and rejects the template and completions endpoints.

🤖 Generated with Claude Code

[github-actions]

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@0acfcfe1de3c818f3ccbd6075a0ad7535dfb1172

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@0acfcfe1de3c818f3ccbd6075a0ad7535dfb1172

---

Last updated for commit: 0acfcfe • Browse code