Skip to content

Explicit structured rubrics (version 2) grounded in the frozen source - #25

Merged
TheCryptoDonkey merged 1 commit into
mainfrom
docs/rubric-v2
Sep 22, 2026
Merged

TheCryptoDonkey merged 1 commit into
mainfrom
docs/rubric-v2

Conversation

@TheCryptoDonkey

Copy link
Copy Markdown
Member

Summary

  • Rewrites the six structured D5 rubrics (docs/experiments/rubric-v2-20260923/) so each dimension lists its required points, with "fails if" conditions, accepted alternatives and four review rules (placement, stated not implied, lists, structure). Dimension ids, required evidence, task prompts and the checker are unchanged; locked version 1 files are untouched.
  • diagnosis-context correction. The existing test that version 1 asked to "restore" is present and passes on the seeded tree, because the commit-time reinspectPolicy in search() still blocks. A committed probe test (diagnosis-context-probe.test.ts) shows that status() reports policy current on the seeded tree and not once the seed is reverted. Version 2 requires a regression test of that kind and says explicitly that the commit-time check is neither the lost invariant nor the repair.
  • impact-context. Corrects the type name (NavigationSearchOptions), accepts a separate case-preserving index as an alternative, and allows a separate mode field in the response.
  • The harness applies version 2 only when a protocol sets "rubricDir", and prints the review rules above the rubric; the rest of the reviewer prompt is byte-identical to recorded prompts. REPEATED-RUNS.md adopts version 2.

Validation

  • node build.mjs refuses to build unless dimension ids match version 1 and the task's required findings.
  • Rendered a version 2 reviewer prompt through the harness's own reviewerPrompt against a recorded answer: the rules and rubric are inserted, and everything else matches the recorded prompt.
  • Probe test run on a scratch copy of a prepared diagnosis-context workspace, seeded and reverted.

Whether version 2 makes the reviewer more consistent is not measured; that needs a paid re-review.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KzXbuBSq88i3Re82qt6c4L

@TheCryptoDonkey
TheCryptoDonkey merged commit 7d436fc into main Sep 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant