Skip to content

feat: add lab result submission contract - #3

Merged
0xnavarro merged 1 commit into
mainfrom
feat/labs-submission-contract
Sep 24, 2026
Merged

0xnavarro merged 1 commit into
mainfrom
feat/labs-submission-contract

Conversation

@0xnavarro

Copy link
Copy Markdown
Contributor

Summary

Makes ReflexBench easier to integrate from model-lab and evaluation-platform CI.

  • adds FOR_LABS.md
  • adds a versioned JSON Schema for result-submission envelopes
  • adds a canonical template
  • adds a dependency-free validator CLI
  • adds unit tests for validation and disclosure requirements
  • links the lab integration path from the README

Why

A benchmark becomes easier to adopt when third parties can produce machine-readable evidence directly from CI instead of relying on screenshots or bespoke issue text. This keeps raw model results, model+harness results, corpus identity, engine identity and development-contamination disclosure explicit.

QA

  • 50/50 unit tests PASS
  • python3 tools/validate_submission.py templates/result-submission-v1.json PASS
  • python3 tools/check_public_release.py . PASS
  • python3 tools/check_claims.py . PASS
  • git diff --check PASS

No benchmark tasks, scoring, v1 corpus identity or existing result values change.

@0xnavarro
0xnavarro merged commit ddf0d2e into main Sep 24, 2026
2 checks passed
@0xnavarro
0xnavarro deleted the feat/labs-submission-contract branch September 24, 2026 00:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant