[v-shobhit]

Adds a smooth tanh option for the inter-turn delay cap, and fixes the cap being ignored on the agentperf-v2 public dataset.

Recorded delays in the dataset

These are the pauses the harness actually sleeps: 24,936 delays, 47.6 h in total. Subagent join turns are left out because the harness fires them as soon as the children finish. The median pause is 0.34 s, but the 9% of pauses over 10 s hold 87% of the time (p99 120 s, max 28 min). Those long pauses set a run's wall time.

Change

• New --inter-turn-delay-cap-mode {hard,tanh} (default hard). tanh sleeps cap * tanh(d / cap): close to the recorded delay for short pauses, bending smoothly into the cap and never above it. hard stays min(d, cap).
• Fix: with --public-dataset agentperf_v2, DagJsonlLoader only read the cap from FileDataset, so --inter-turn-delay-cap-seconds was silently ignored. Cap and mode now also resolve from PublicDataset.

Recorded	hard (10 s)	tanh (10 s)
5 s	5.00 s	4.62 s
10 s	10.00 s	7.62 s
20 s	10.00 s	9.64 s

With a 10 s cap, the 47.6 h of slept pauses drops to 12.3 h (hard) or 11.5 h (tanh). Hard puts the 8.9% of delays over 10 s at exactly 10 s; tanh spreads them out.

How the cap shortens a run

An agentperf-v2 run ends when every user has replayed its burn-in and measured slice. Both are fixed stretches of the recorded timeline (15 min, then up to 75 min), so the requests sent don't depend on the cap. What changes is how long each user takes to get through them:

user wall time  =  generation (server)  +  slept pauses  +  subagent join waits
                                            ^ cap shrinks only this term
run wall time   =  slowest user

The cap helps a run only when its slowest user is mostly waiting on recorded pauses. Estimates from our uncapped DeepSeek-V4.1-Flash runs on GB200 (TP2, agentperf-v2 scenario), each user's measured idle gaps replaced by their 10 s tanh-capped value:

Concurrency	Slowest user, uncapped	Its idle time	Join waits (not capped)	Cappable pauses → after cap	Estimated run with cap
8	96 min	87 min	10 min	77 → 10 min	43 min (−55%)
16	60 min	56 min	12 min	44 → 6 min	32 min (−46%)
32	65 min	56 min	0 min	56 → 4 min	62 min (−4%)
64	122 min	62 min	61 min	1 → 1 min	122 min (0%)

• Low concurrency roughly halves. The server is fast, so the slowest users mostly sit in recorded pauses.
• c32 barely moves. Its slowest user would drop to about 13 min, but another user, bound by generation time, then becomes the slowest.
• c64 doesn't move. The saturated server makes subagents slow, and the main thread spends 61 of its 62 idle minutes waiting on them. Capping subagents' own pauses could shorten those waits; that isn't modeled, so the 0% is a lower bound.
• Hard and tanh differ by at most a minute. The estimates assume the server behaves the same with the cap. Pauses that disappear also raise the load each user offers, so capped and uncapped runs don't produce comparable numbers.

The default stays uncapped; the agentperf-v2 scenario is unchanged.

Testing

• New tests/unit/dataset/loader/test_inter_turn_delay_soft_cap.py (curve, monotonicity, pass-throughs, tracker, DAG loader in both modes, PublicDataset resolution), plus routing and cache-key tests.
• pytest tests/unit/{config,dataset,common,timing,cli}: 10,346 passed, 105 skipped, 1 xfailed. ruff and generate_config_schema --check clean.
• End to end on aiperf-mock-server with --public-dataset agentperf_v2 --inter-turn-delay-cap-seconds 10: both modes cap 2,382 delays (tanh removes 248,895.5 s in total). Before the fix, neither applied.

🤖 Generated with Claude Code

[github-actions]

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@ecc30701f475b40f8c1b479fc6c84c0a689b1444

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@ecc30701f475b40f8c1b479fc6c84c0a689b1444

---

Last updated for commit: ecc3070 • Browse code