Issues

• Null inside a named accuracy metric is dropped silently

  mlcommons/endpoints-submission-cli#178 · Filter by author anandhu-eng opened on Nov 9, 2026

• NaN accuracy score passes the gate

  mlcommons/endpoints-submission-cli#177 · Filter by author anandhu-eng opened on Nov 9, 2026

• Single-turn accuracy average includes the inline performance entry

  mlcommons/endpoints-submission-cli#176 · Filter by author anandhu-eng opened on Nov 9, 2026

• Null SWE-bench result gives only a warning

  mlcommons/endpoints-submission-cli#175 · Filter by author anandhu-eng opened on Nov 9, 2026

• Accuracy gate ignores complete: false

  mlcommons/endpoints-submission-cli#174 · Filter by author anandhu-eng opened on Nov 9, 2026

• No check that samples issued is a whole multiple of the dataset size (§6.4)

  mlcommons/endpoints-submission-cli#167 · Filter by author mkankana opened on Nov 4, 2026

• Run requirements: duration keyed by region (rules key on concurrency), agentic 2400 s not encoded, values still "subject to ratification"

  mlcommons/endpoints-submission-cli#165 · Filter by author nv-alicheng opened on Oct 30, 2026
  1

• Seed adoption window: docstring quotes §4.6 wording that is not in the current rules

  mlcommons/endpoints-submission-cli#164 · Filter by author nv-alicheng opened on Oct 30, 2026

• seed-config-legacy: no-seed_set points pass with a warning when seeds are 42, while §4.6 requires a recorded bound set

  mlcommons/endpoints-submission-cli#163 · Filter by author nv-alicheng opened on Oct 30, 2026

• steady_state block: the checker reads it from point.yaml, but the client writes it elsewhere

  mlcommons/endpoints-submission-cli#162 · Filter by author nv-alicheng opened on Oct 30, 2026

• Dedicated Offline run: which region value? submitters_choice currently passes the checker

  mlcommons/endpoints-submission-cli#161 · Filter by author nv-alicheng opened on Oct 30, 2026

• Load pattern: §6.1 defines no wire tokens; the checker requires concurrency / agentic_inference

  mlcommons/endpoints-submission-cli#160 · Filter by author nv-alicheng opened on Oct 30, 2026

• Availability field spelling: the checker accepts three spellings; the rules use one (system_availability_status)

  mlcommons/endpoints-submission-cli#159 · Filter by author nv-alicheng opened on Oct 30, 2026

• system_info: the checker documents system_desc.json as submitter-authored; Submission Rules §5.2 tells submitters to configure a system_info section

  mlcommons/endpoints-submission-cli#158 · Filter by author nv-alicheng opened on Oct 30, 2026

• Warmup salt: the checker warns when salt is enabled; §6.3.1 requires salting exactly when performance data is used for warmup

  mlcommons/endpoints-submission-cli#157 · Filter by author nv-alicheng opened on Oct 30, 2026

• Checker docs/messages cite rule sections that do not exist (§10, §11, §12, §14, §15)

  mlcommons/endpoints-submission-cli#156 · Filter by author nv-alicheng opened on Oct 30, 2026

• Drafter approval lead time: §2.9.4 anchors on the review eligibility date, §9.1 on target_cohort; the checker uses target_cohort

  mlcommons/endpoints-submission-cli#155 · Filter by author nv-alicheng opened on Oct 30, 2026

• Accuracy gate enforced from MLPerf Inference constants while §2.9.8 is WIP and no benchmark definition publishes Endpoints targets

  mlcommons/endpoints-submission-cli#154 · Filter by author nv-alicheng opened on Oct 30, 2026

• tps_utilization: §8.2's "max of all reported_system_tps for all runs" is unscoped; the checker normalizes per curve

  mlcommons/endpoints-submission-cli#153 · Filter by author nv-alicheng opened on Oct 30, 2026

• Power tolerance/precision: "computed disagrees" has no threshold in the rules; the checker defines its own tolerances

  mlcommons/endpoints-submission-cli#152 · Filter by author nv-alicheng opened on Oct 30, 2026

• D.1 scale-up fallback cannot run: the descriptor does not declare the link protocol

  mlcommons/endpoints-submission-cli#151 · Filter by author nv-alicheng opened on Oct 30, 2026

• E.7 power schema: two structural choices are enforced as rejections while E.7 marks them "recommendations, not settled policy"

  mlcommons/endpoints-submission-cli#150 · Filter by author nv-alicheng opened on Oct 30, 2026

• Cooling vocabulary: E.2 requires liquid/air consistent with §8.2's free-text cooling; "passive cooling" has no overhead fraction

  mlcommons/endpoints-submission-cli#149 · Filter by author nv-alicheng opened on Oct 30, 2026

• Power descriptor scope: §4.5.2 says every submission MUST include system_power.json; the checker makes it optional for RDI/Serviced

  mlcommons/endpoints-submission-cli#148 · Filter by author nv-alicheng opened on Oct 30, 2026

• feat: list should print out system/model + filter by staus

  mlcommons/endpoints-submission-cli#135 · Filter by author arekay-nv opened on Oct 15, 2026