Repository navigation
Conversation
Context-overflow responses in agentic replay are skipped: the session ends and the skipped turns never reach the metrics, while the validity gate only checks overflow events against a 1% rate. A server started with a context window shorter than the dataset needs could skip the longest turns and still report submission_valid=true. Every request in the agentperf-v2 recordings fits a 1M-token window, so a correctly configured server never overflows. Add a per-scenario max_context_overflow_rate ceiling and set it to 0.0 for agentperf-v2. The AIPERF_AGENTX_CONTEXT_OVERFLOW_RATE_LIMIT env var can only tighten a scenario's ceiling, never loosen it. Other scenarios keep the 1% rule. Signed-off-by: Bofeng Luo <bofengl@nvidia.com>
Try out this PRQuick install: pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@6dd4f37cdba1740c3561ea7c7a08ffd274b8e5c4Recommended with virtual environment (using uv): uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@6dd4f37cdba1740c3561ea7c7a08ffd274b8e5c4Last updated for commit: |
…rement Signed-off-by: Bofeng Luo <bofengl@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In agentic replay, a context-overflow response is skipped: the session ends, the record stays out of the metrics, and the turns after it are never sent. The only validity check is the overflow rate against
AIPERF_AGENTX_CONTEXT_OVERFLOW_RATE_LIMIT(1%), and it counts overflow events, not the turns that were skipped.So a server started with a shorter context window than the dataset needs (for example 512k) can skip the longest, most expensive turns and still report
submission_valid=true. One overflow ends a whole session, so a few dozen overflows per run stay far below 1%.Every request in the agentperf-v2 recordings fits in a 1M-token window (the largest prompts are about 804k, 873k and 845k tokens for the three models). A correctly configured server never overflows.
Change
ScenarioSpec.max_context_overflow_rate(defaultNone, which keeps the env limit).agentperf-v2sets it to0.0, so any context overflow flipssubmission_validtofalsewithcontext_overflow_rate_exceeded.Known gaps (not changed here)
Tests
tests/unit/exporters/test_submission_valid_field.py: zero overflows stay valid; one overflow in 10,000 is invalid for agentperf-v2; inferencex-agentx-mvp keeps the 1% rule; the env var cannot loosen the agentperf-v2 ceiling. The agentperf-v2 tutorial now states the 1M-token context requirement.🤖 Generated with Claude Code