Skip to content

feat(config): register MLPerf Endpoints v1.0 seed set B (cohort 2026-11-C0) - #564

Merged
arekay-nv merged 1 commit into
mlcommons:mainfrom
rebklee:feat/endpoints-seed-set-b
Nov 14, 2026
Merged

arekay-nv merged 1 commit into
mlcommons:mainfrom
rebklee:feat/endpoints-seed-set-b

Conversation

@rebklee

@rebklee rebklee commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

Registers seed set B (cohort 2026-11-C0, mlcommons/endpoints_policies#155) as
mlperf-endpoints-v1.0-2026-11-C0-B alongside set A, following #508. The seeds
are pinned to
seedset.yaml @ aa7537c.

Today a config bound to set B fails at load (submission_ref.ruleset ... is not registered), and from 2026-12-C1 set B is the only set a new submission can
adopt.

mlperf-endpoints-current stays on set A. Policies §4.6 binds a submission to
one set for its update window, so advancing the alias would re-seed runs that
bind through it.

Out of scope

  • A 2026-11-C0 cohort for the in-repo checker. cohorts/2026-10-C1 accepts
    only set A, which is correct for that cohort.

Type of change

  • New feature

Related issues

Relates to mlcommons/endpoints_policies#155

Testing

  • test_endpoints_v1_0_set_b_official_seeds fails on main and passes
    here. The seeds-differ test now also compares the Endpoints sets with
    each other.
  • CI's test selection (-m "not slow and not performance") on Linux:
    3240 passed, 8 skipped, 1 failed. The failure,
    test_run_timeout.py::test_metrics_drain_timeout_fails_run, also fails
    on main on this machine.
  • inference-endpoint benchmark from-config against the echo server with
    set B: config.yaml records 17312635550606430962 /
    6679150691998295617. On main the same command exits 2.

Checklist

  • Code follows project style
  • Pre-commit hooks pass
  • Documentation updated (if needed): n/a, no docs list ruleset names

…11-C0)

Registers mlperf-endpoints-v1.0-2026-11-C0-B, the set published in
mlcommons/endpoints_policies#155, alongside set A.
mlperf-endpoints-current stays on set A (policies §4.6 binding lifetime).
@rebklee
rebklee requested a review from a team November 12, 2026 02:08
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@github-actions github-actions Bot added the size/normal PR Review Policy: <=500 non-test lines & <=20 files label Nov 12, 2026
@rebklee

rebklee commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

recheck

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@745ac91). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #564   +/-   ##
=======================================
  Coverage        ?   82.63%           
=======================================
  Files           ?      170           
  Lines           ?    24332           
  Branches        ?        0           
=======================================
  Hits            ?    20107           
  Misses          ?     4225           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@arekay-nv

Copy link
Copy Markdown
Collaborator

@rebklee please sign the CLA form so we can review the PR.

MLCommons CLA bot:
Thank you very much for your submission; we really appreciate it. Before we can accept your contribution,
we ask that you sign the MLCommons CLA (Apache 2). Please submit your GitHub ID to our onboarding form to initiate
authorization. If you are from a MLCommons member organization, we will request that you be added to the CLA.
If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact
support@mlcommons.org.
0 out of 1 committers have signed the MLCommons CLA.
❌ @rebklee

@mrmhodak

Copy link
Copy Markdown
Collaborator

recheck

@mrmhodak

Copy link
Copy Markdown
Collaborator

@arekay-nv: This may be ready to go

@arekay-nv arekay-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rebklee

@arekay-nv
arekay-nv merged commit c4b797b into mlcommons:main Nov 14, 2026
9 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/normal PR Review Policy: <=500 non-test lines & <=20 files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants