decision(evals): ratify a privacy-safe model-output benchmark contract

[WalksWithASwagger]

Summary

Ratify a privacy-safe, human-review benchmark contract for model and prompt output quality.

Context

• Roadmap phase: Phase 2 - model quality
• Relevant files: a new docs/MODEL-EVALS.md, docs/MODEL-POLICY.md, docs/INDEX.md, and a small synthetic schema/fixture under tests/fixtures/model-evals/
• Current behavior: generation has operational metadata and human review surfaces, but no versioned benchmark prompts, anchored rubric, golden dataset contract, or repeatable comparison report.
• Desired behavior: maintainers approve what can be benchmarked publicly and how human scores are recorded before any eval runner or dataset is implemented.

Implementation Notes

• This is a policy and rubric decision issue; no provider calls or generated asset commits.
• Use public synthetic inputs only.
• Define stable prompt IDs, versioned metadata, anchored 1-5 scores, and approve/revise/reject rules.
• Include prompt adherence, composition, typography, reference fidelity, and artifact severity unless maintainers revise the dimensions.

Acceptance Criteria

• Maintainers ratify the benchmark prompt set and stable prompt IDs.
• Each rubric dimension has anchored 1-5 scoring guidance.
• Approve, revise, and reject decision rules are explicit.
• Recorded metadata covers model, provider, style, parameters, prompt version, and reviewer.
• Public fixtures are synthetic and contain no private likenesses, artwork, outputs, or paths.
• Provider spend and benchmark refreshes require an explicit human gate.
• The approved schema fixture validates in the normal test suite.

Tests/Evals

• Validate the machine-readable schema using synthetic records only.
• No generation, automated vision judging, or real provider credentials.

Verification

• npm run docs:check
• npm test
• Validate one synthetic approved record, one revise record, and one rejected record against the schema.

Agent Instructions

• Do not generate images or call providers.
• Do not commit private prompts, likenesses, reference assets, or generated outputs.
• Do not implement the reporting command before the rubric is approved.

Human Checkpoints

• KK must approve the prompts, rubric, public metadata boundary, and spend/refresh policy.

Out of Scope

• Provider calls or benchmark generation.
• Automated model judging.
• Default-model changes.
• Committing golden images.

Linear

Not applicable. Rafiki delivery is GitHub-only; do not create or update a Linear issue.