diff --git a/.github/ISSUE_TEMPLATE/model-result-submission.yml b/.github/ISSUE_TEMPLATE/model-result-submission.yml new file mode 100644 index 0000000..dd23a91 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/model-result-submission.yml @@ -0,0 +1,64 @@ +name: Model / result submission +description: Propose a model adapter or reproducible result for a frozen ReflexBench version. +title: "[result] " +labels: [] +body: + - type: markdown + attributes: + value: | + Thanks for contributing a reproducible System One / typed-decision result. Please do not include credentials, private customer data, or unpublished model artifacts in this issue. + - type: input + id: engine + attributes: + label: Engine / model + placeholder: owner/model-name + validations: + required: true + - type: input + id: revision + attributes: + label: Immutable revision + description: Model/checkpoint revision, commit, image digest, or hosted revision if exposed. + validations: + required: true + - type: dropdown + id: deployment + attributes: + label: Deployment boundary + options: + - Hosted API + - Local CPU + - Local GPU + - Other + validations: + required: true + - type: input + id: benchmark + attributes: + label: Benchmark version + value: v1.0.0 + validations: + required: true + - type: textarea + id: hardware + attributes: + label: Hardware / provider metadata + description: Hardware for local runs, or provider/region boundary for hosted runs. Do not include secrets. + - type: textarea + id: procedure + attributes: + label: Exact run procedure + description: Commands, adapter revision and any retry policy. + validations: + required: true + - type: checkboxes + id: integrity + attributes: + label: Evidence integrity + options: + - label: I retained failed/unsupported rows and retries in the submitted evidence. + required: true + - label: I did not silently change benchmark task semantics for this engine. + required: true + - label: I will disclose whether ReflexBench influenced model/prompt/threshold development. + required: true diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c9a10a4..8138dbd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -12,3 +12,5 @@ For a new v1 engine result: 6. include machine-readable receipt(s) and the command/procedure used. Changes to v1 tasks/scoring are not accepted as silent fixes; propose v2 instead. + +For the full model/result submission workflow, see [SUBMIT_A_MODEL.md](SUBMIT_A_MODEL.md). diff --git a/README.md b/README.md index d629312..c1041f1 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,12 @@ bounded state + canonical typed questions A model-specific prompt rewrite, semantic adapter or extra inference pass is **not** counted as Core uplift. See [docs/system-one-contract-v1.md](docs/system-one-contract-v1.md). +## Benchmark your model + +Maintaining a System One or typed-decision model? Run the frozen protocol and submit a reproducible result. See **[SUBMIT_A_MODEL.md](SUBMIT_A_MODEL.md)** for the adapter contract, identity/provenance requirements, receipt rules and result-PR workflow. + +Accepted public results may use the **Evaluated on ReflexBench v1** badge; acceptance records reproducible evidence and is not an endorsement or universal ranking. + ## Quick start Requires Python 3.11+ for the core harness. Core v1 uses only the standard library. diff --git a/SUBMIT_A_MODEL.md b/SUBMIT_A_MODEL.md new file mode 100644 index 0000000..629bd95 --- /dev/null +++ b/SUBMIT_A_MODEL.md @@ -0,0 +1,90 @@ +# Submit a model to ReflexBench + +ReflexBench accepts reproducible third-party model results against frozen benchmark versions. + +The goal is simple: if you maintain a System One model, typed decision model, probabilistic decision engine, or compatible hosted API, you should be able to run the same public protocol and submit evidence without changing benchmark semantics. + +## 1. Pick the frozen benchmark version + +For v1, do not edit tasks or scoring. Use the committed v1 corpora/manifests and verify their hashes before running. + +```bash +python3 tools/check_public_release.py . +``` + +A material task/scoring change belongs in a future benchmark version, not a v1 result submission. + +## 2. Implement or configure an adapter + +A compatible engine must map the canonical ReflexBench task shapes to the engine's public interface without changing task meaning: + +- Binary / Noul +- Choice +- Score + +Provider-specific transport/authentication is allowed. Provider-specific semantic rewriting that changes the task is not considered a comparable v1 run. + +Use the existing provider-neutral HTTP runner as the reference contract: + +```bash +PYTHONPATH=reflexbench python3 reflexbench/systemone_reflex_runner.py --help +``` + +## 3. Pin identity + +Every submitted result must identify, when observable: + +- model/engine owner; +- canonical model/engine name; +- immutable model/checkpoint revision; +- adapter/runner code revision; +- benchmark version; +- corpus/manifests SHA-256; +- local hardware or hosted deployment boundary; +- date of execution; +- retry/failure behavior. + +Do not use an ambiguous wire alias as model identity. + +## 4. Keep failures + +Do not drop failed rows, unsupported task shapes, timeouts, provider errors or retries from the evidence. Completion rate is part of the result. + +## 5. Produce machine-readable receipts + +Submit the raw ReflexBench result receipt(s), not only a screenshot or aggregate table. The receipt should preserve per-row outputs/probabilities when the engine exposes them and enough metadata to reproduce the aggregation. + +Before submitting: + +```bash +python3 tools/check_claims.py . +python3 tools/check_public_release.py . +``` + +## 6. Open a result PR + +A result PR should contain: + +1. raw receipt(s) under the appropriate versioned results directory; +2. the exact command/procedure used; +3. adapter code if a new public adapter is required; +4. regenerated public result tables/assets, if applicable; +5. a note describing whether the benchmark was used during model/prompt/threshold development. + +If the benchmark influenced model selection, tuning, calibration or thresholds, say so. That result may still be useful development evidence, but should not be represented as pristine held-out evidence. + +## 7. Optional verified badge + +After a result is accepted, model authors may use: + +```markdown +[![Evaluated on ReflexBench v1](https://img.shields.io/badge/ReflexBench-v1-2eaadc)](https://github.com/brida-ai/reflexbench) +``` + +Use the badge to indicate that a result exists in the public ReflexBench record, not as an endorsement or universal quality ranking. + +## Questions / pre-release models + +For public questions, open a GitHub issue using the **Model / result submission** template. + +For a pre-release model where public disclosure is not yet possible, contact Brida through the public contact surface at https://www.brida.ai/ and reference ReflexBench. Private evaluation does not automatically become a public canonical result.