[kaulastudies]

Addresses #167.

Adds a whole-pass check using runtime_settings.n_samples_from_dataset from the resolved benchmark settings. Explicitly supplied zero or negative issued counts are errors even when the dataset size is unavailable. Partial passes are errors when the size is known. An omitted issued count, or a positive count with no dataset size, produces an explicit warning. The dataset-size field must be positive when supplied.

The minimum-query table is deliberately not used as dataset size: some accuracy counts include repeats, and LiveCodeBench's per-model run shape differs from that table. The guide explains the recorded field and makes clear that reviewers still need to verify full-dataset use or working-group approval for a subset. Existing minimum-completed-query enforcement is unchanged.

Validation for b03da4d on Python 3.12:

• All 18 dataset-pass regression cases pass. The two cases for zero/negative issued counts with an unknown size fail on the previous PR head 744e040 and pass with this fix.
• Ruff checks/format and mypy pass (67 source files).
• Full local suite: 1,818 passed, 5 failed. Four CLI display/rollback failures also occur in the unchanged parent checkout (1,815 passed, 4 failed). The fifth display failure also reproduces with the unchanged parent code when run from the same test path; its result depends on the checkout path used in rendered output.
• Hosted validation is tracked in the PR checks.

#170 is merged, and this PR has been marked ready for review by a maintainer. It has no dependency on #170.

Thanks to @nv-alicheng for reporting the gap. Prepared with Codex assistance; the local checks above were executed.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[Copilot]

🟡 Changes recommended

Explicitly reported zero or negative n_samples_issued values bypass the required positivity error when dataset size is unavailable, producing only an unknown-size warning.

1 open issue

• Reject nonpositive issued counts without dataset size · New

What changed in this PR

This PR addresses #167 by adding whole-dataset-pass validation to the submission checker using the recorded dataset size, without changing minimum-completed-query enforcement.

Changes:

• Adds an optional, positive n_samples_from_dataset setting and issued-count validation.
• Adds diagnostics and regression tests for whole passes, partial passes, non-positive counts, and missing metadata.
• Documents the recorded size and the remaining manual checks for full-dataset use or approved subsets.

File	Description
tests/​submission_checker/​test_sample_multiples.py	Adds regression coverage for pass counts and missing metadata.
src/​submission_checker/​models/​file/​point_config.py	Defines the positive, optional dataset-size field.
src/​submission_checker/​models/​aggregate/​point_result.py	Adds validation against the recorded dataset size.
src/​submission_checker/​data/​messages.yaml	Adds validation results and corrective guidance.
docs/​endpoints-submission-cli.md	Explains size recording and subset-approval responsibilities.

_{Review effort: Balanced}

[kaulastudies]

@anandhu-eng The issued-count fix is in b03da4d, and the review thread is resolved. GitHub now shows all seven checks passing. Could you take another look when you have time?