Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions .github/ISSUE_TEMPLATE/model-result-submission.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
name: Model / result submission
description: Propose a model adapter or reproducible result for a frozen ReflexBench version.
title: "[result] "
labels: []
body:
- type: markdown
attributes:
value: |
Thanks for contributing a reproducible System One / typed-decision result. Please do not include credentials, private customer data, or unpublished model artifacts in this issue.
- type: input
id: engine
attributes:
label: Engine / model
placeholder: owner/model-name
validations:
required: true
- type: input
id: revision
attributes:
label: Immutable revision
description: Model/checkpoint revision, commit, image digest, or hosted revision if exposed.
validations:
required: true
- type: dropdown
id: deployment
attributes:
label: Deployment boundary
options:
- Hosted API
- Local CPU
- Local GPU
- Other
validations:
required: true
- type: input
id: benchmark
attributes:
label: Benchmark version
value: v1.0.0
validations:
required: true
- type: textarea
id: hardware
attributes:
label: Hardware / provider metadata
description: Hardware for local runs, or provider/region boundary for hosted runs. Do not include secrets.
- type: textarea
id: procedure
attributes:
label: Exact run procedure
description: Commands, adapter revision and any retry policy.
validations:
required: true
- type: checkboxes
id: integrity
attributes:
label: Evidence integrity
options:
- label: I retained failed/unsupported rows and retries in the submitted evidence.
required: true
- label: I did not silently change benchmark task semantics for this engine.
required: true
- label: I will disclose whether ReflexBench influenced model/prompt/threshold development.
required: true
2 changes: 2 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,5 @@ For a new v1 engine result:
6. include machine-readable receipt(s) and the command/procedure used.

Changes to v1 tasks/scoring are not accepted as silent fixes; propose v2 instead.

For the full model/result submission workflow, see [SUBMIT_A_MODEL.md](SUBMIT_A_MODEL.md).
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,12 @@ bounded state + canonical typed questions

A model-specific prompt rewrite, semantic adapter or extra inference pass is **not** counted as Core uplift. See [docs/system-one-contract-v1.md](docs/system-one-contract-v1.md).

## Benchmark your model

Maintaining a System One or typed-decision model? Run the frozen protocol and submit a reproducible result. See **[SUBMIT_A_MODEL.md](SUBMIT_A_MODEL.md)** for the adapter contract, identity/provenance requirements, receipt rules and result-PR workflow.

Accepted public results may use the **Evaluated on ReflexBench v1** badge; acceptance records reproducible evidence and is not an endorsement or universal ranking.

## Quick start

Requires Python 3.11+ for the core harness. Core v1 uses only the standard library.
Expand Down
90 changes: 90 additions & 0 deletions SUBMIT_A_MODEL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Submit a model to ReflexBench

ReflexBench accepts reproducible third-party model results against frozen benchmark versions.

The goal is simple: if you maintain a System One model, typed decision model, probabilistic decision engine, or compatible hosted API, you should be able to run the same public protocol and submit evidence without changing benchmark semantics.

## 1. Pick the frozen benchmark version

For v1, do not edit tasks or scoring. Use the committed v1 corpora/manifests and verify their hashes before running.

```bash
python3 tools/check_public_release.py .
```

A material task/scoring change belongs in a future benchmark version, not a v1 result submission.

## 2. Implement or configure an adapter

A compatible engine must map the canonical ReflexBench task shapes to the engine's public interface without changing task meaning:

- Binary / Noul
- Choice
- Score

Provider-specific transport/authentication is allowed. Provider-specific semantic rewriting that changes the task is not considered a comparable v1 run.

Use the existing provider-neutral HTTP runner as the reference contract:

```bash
PYTHONPATH=reflexbench python3 reflexbench/systemone_reflex_runner.py --help
```

## 3. Pin identity

Every submitted result must identify, when observable:

- model/engine owner;
- canonical model/engine name;
- immutable model/checkpoint revision;
- adapter/runner code revision;
- benchmark version;
- corpus/manifests SHA-256;
- local hardware or hosted deployment boundary;
- date of execution;
- retry/failure behavior.

Do not use an ambiguous wire alias as model identity.

## 4. Keep failures

Do not drop failed rows, unsupported task shapes, timeouts, provider errors or retries from the evidence. Completion rate is part of the result.

## 5. Produce machine-readable receipts

Submit the raw ReflexBench result receipt(s), not only a screenshot or aggregate table. The receipt should preserve per-row outputs/probabilities when the engine exposes them and enough metadata to reproduce the aggregation.

Before submitting:

```bash
python3 tools/check_claims.py .
python3 tools/check_public_release.py .
```

## 6. Open a result PR

A result PR should contain:

1. raw receipt(s) under the appropriate versioned results directory;
2. the exact command/procedure used;
3. adapter code if a new public adapter is required;
4. regenerated public result tables/assets, if applicable;
5. a note describing whether the benchmark was used during model/prompt/threshold development.

If the benchmark influenced model selection, tuning, calibration or thresholds, say so. That result may still be useful development evidence, but should not be represented as pristine held-out evidence.

## 7. Optional verified badge

After a result is accepted, model authors may use:

```markdown
[![Evaluated on ReflexBench v1](https://img.shields.io/badge/ReflexBench-v1-2eaadc)](https://github.com/brida-ai/reflexbench)
```

Use the badge to indicate that a result exists in the public ReflexBench record, not as an endorsement or universal quality ranking.

## Questions / pre-release models

For public questions, open a GitHub issue using the **Model / result submission** template.

For a pre-release model where public disclosure is not yet possible, contact Brida through the public contact surface at https://www.brida.ai/ and reference ReflexBench. Private evaluation does not automatically become a public canonical result.
Loading