Repository navigation
feat(config): register MLPerf Endpoints v1.0 seed set B (cohort 2026-11-C0) - #564
Conversation
…11-C0) Registers mlperf-endpoints-v1.0-2026-11-C0-B, the set published in mlcommons/endpoints_policies#155, alongside set A. mlperf-endpoints-current stays on set A (policies §4.6 binding lifetime).
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
|
recheck |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #564 +/- ##
=======================================
Coverage ? 82.63%
=======================================
Files ? 170
Lines ? 24332
Branches ? 0
=======================================
Hits ? 20107
Misses ? 4225
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@rebklee please sign the CLA form so we can review the PR.
|
|
recheck |
|
@arekay-nv: This may be ready to go |
What does this PR do?
Registers seed set B (cohort
2026-11-C0, mlcommons/endpoints_policies#155) asmlperf-endpoints-v1.0-2026-11-C0-Balongside set A, following #508. The seedsare pinned to
seedset.yaml@ aa7537c.Today a config bound to set B fails at load (
submission_ref.ruleset ... is not registered), and from2026-12-C1set B is the only set a new submission canadopt.
mlperf-endpoints-currentstays on set A. Policies §4.6 binds a submission toone set for its update window, so advancing the alias would re-seed runs that
bind through it.
Out of scope
2026-11-C0cohort for the in-repo checker.cohorts/2026-10-C1acceptsonly set A, which is correct for that cohort.
Type of change
Related issues
Relates to mlcommons/endpoints_policies#155
Testing
test_endpoints_v1_0_set_b_official_seedsfails onmainand passeshere. The seeds-differ test now also compares the Endpoints sets with
each other.
-m "not slow and not performance") on Linux:3240 passed, 8 skipped, 1 failed. The failure,
test_run_timeout.py::test_metrics_drain_timeout_fails_run, also failson
mainon this machine.inference-endpoint benchmark from-configagainst the echo server withset B:
config.yamlrecords17312635550606430962/6679150691998295617. Onmainthe same command exits 2.Checklist