Repository navigation
Conversation
…ming submittable Refusing a measurement stamped submission_valid=False is what stops a short smoke from being read as a leaderboard number, and that stays the default. The same refusal also blocks the search the smoke exists for: every candidate on a deviating workload carries the stamp, so no variant is comparable to a baseline measured the same way, and a session spends its budget on verdicts it discards. Measured on Kimi-K3 agentic: the only gradeable workload was the canonical 613-trajectory set at ~33h per round, and a 150-trajectory variant round aborted as benchmark_report_invalid_metric with a healthy report on disk. HYPERLOOM_ALLOW_NONCANONICAL_MEASUREMENT buys grading and nothing else -- the verdict stays False, its reasons travel with it, and an unknown verdict is still refused. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Closing this one because main has moved past what it changes. #1922 (merged as 75df551) replaced the AIPerf AgentX client with Magpie's native AgentX contract. It removed aiperf_client.sh, aiperf_phase_gate.py and map_aiperf.py, along with the |
Description
is_valid_measurementrefuses any AgentX measurement stampedsubmission_valid=False. That refusal is load-bearing — it is what stops a shortened smoke from being read as a leaderboard number — and it stays the default here.But the stamp is applied to every run on a deviating workload, including the smoke that search is meant to run on. So the refusal also rejects the A/B the smoke exists for: a variant measured on 150 trajectories cannot be compared against a baseline measured on 150 trajectories, because both are non-canonical. The result is a session that spends its budget producing measurements it then discards.
Measured on a Kimi-K3 agentic session:
benchmark_report_invalid_metric (rc=0): benchmark_report missing valid throughput/completed requests, with a complete report on disk written 31s before the abort.What this changes
HYPERLOOM_ALLOW_NONCANONICAL_MEASUREMENTlets a declared deviation be graded. It buys comparability and nothing else:submission_validstaysFalsein the result andsubmission_invalid_reasonstravel with it, so no artifact becomes submittable.None) is still refused; that is a different claim from declared-and-deviating, and keeps its own existing knob.Observable effect
With the opt-in set, a session can search on a reduced workload and have its variants graded against a like-for-like baseline. Without it, nothing changes.
Test plan
test_agentx_budget_and_guards.py— 38 passed, including three new cases: opt-in grades a declared deviation while leaving the verdict False, the opt-in does not admit an unknown verdict, and the default still refuses-k 'benchmark_result or agentx or grid_runner or valid_measurement') — 849 passed, 2 skipped (one pre-existing failure intest_agentx_repair.pyreproduces on a clean tree)Breaking changes: no. PR addresses single concern: yes.
Made with Cursor