Repository navigation
Conversation
The agentperf-v2 scenario neither locks nor forbids --trace-idle-gap-cap-seconds or --system-idle-gap-cap-seconds, so a submitter can compress or skip recorded idle time and still report submission_valid=true. Set forbid_trace_idle_gap_cap for agentperf-v2, and add a matching forbid_system_idle_gap_cap scenario field that the validator enforces on every profiling phase. The per-turn --inter-turn-delay-cap-seconds is left unchanged here. Signed-off-by: Bofeng Luo <bofengl@nvidia.com>
Try out this PRQuick install: pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@37e4b9ad208cc0941cfecc262b4cde57c6efc95eRecommended with virtual environment (using uv): uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@37e4b9ad208cc0941cfecc262b4cde57c6efc95eLast updated for commit: |
…t them Signed-off-by: Bofeng Luo <bofengl@nvidia.com>
roborluo
force-pushed
the
fix/agentperf-v2-forbid-idle-gap-caps
branch
from
November 12, 2026 23:40
b176e77 to
37e4b9a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The agentperf-v2 scenario neither locks nor forbids
--trace-idle-gap-cap-secondsor--system-idle-gap-cap-seconds. Both are accepted by the validator, so a submitter can compress or skip recorded idle time and still reportsubmission_valid=true. The system idle-gap cap applies on both the--public-datasetand--input-filepaths.Change
agentperf-v2setsforbid_trace_idle_gap_cap=True.ScenarioSpec.forbid_system_idle_gap_cap, enforced by the validator on every profiling phase, and set foragentperf-v2.submission_valid=falseunder--unsafe-override.--inter-turn-delay-cap-secondsis not touched here; how per-turn pauses are capped is covered separately (feat(dataset): smooth tanh inter-turn delay cap; honor the cap on public DAG datasets #1).Tests
tests/unit/common/scenario/test_scenario_validator.py: a clean agentperf-v2 run stays valid; each idle-gap cap flag is rejected.🤖 Generated with Claude Code