Add decisions-v1 corpus of real GitHub decisions

[MaggieAppleton]

Note

Holdout check passed (2026-10-19). None of the 8 eval-hub holdout threads (5 validation, 3 sealed final) is in this corpus, and no kept thread links to one. The holdout URLs are deliberately not recorded.

Since review: the leak check covers every free-text field, the validator reports malformed timestamps instead of crashing, the holdout check also scans document records and discussion URLs, merged statuses no longer depend on annotation order, and bun test evals/datasets runs the validators.

What

evals/datasets/decisions-v1/ contains 127 real software decisions from public GitHub threads. It has two uses:

• Coverage critique for questions. Each of the 132 decision records gives:
  • the question shape (explore, choice, tune or check)
  • the flows it exercised
  • chopinFit.gaps: what the decision needed that the design doesn't offer
• Evals. Each case uses the v2 async-discussion format, with 2–4 cutoffs placed at these moments:
  • the question is clear
  • the options are on the table
  • just before the decision
  • just after the decision

flowchart LR
  S[gh search + seeds<br/>192 candidates] --> F[fetch.py<br/>raw API → .cache/ (ignored)]
  F --> N[normalise.py<br/>v2 event skeleton]
  N --> A[subagent annotation<br/>paraphrased summaries,<br/>cutoffs, decision record]
  A --> C[check_ann.py<br/>schema + 8-word leak check]
  C --> B[assemble.py + package.py]
  B --> O[cases/*.json<br/>decisions.jsonl<br/>manifest.json, stats.json]
  O --> V[validate.py]

Loading

Highlights

Quotas (target vs kept):

Quota	Target	Kept
Product, UI and behaviour	30	35
PR-review debates	20	26
Formal processes	15	24
Engineering issues	15	20
Small teams	10	11
Open choice	10	11

Most common gaps (share of the 132 decisions):

Gap	Share
Decided off-platform	42%
Staged acceptance	41%
Votes or reactions as evidence	40%
Coupled decisions	39%
Decider not in thread	35%
Reversed after shipping	22%
Experiments or flags as evidence	22%

Shapes: choice is the primary shape 70% of the time. Check, explore and tune are each a common secondary shape.

Licensing

Committed files contain attributed paraphrases only, with no quotes. Raw GitHub responses stay in the git-ignored .cache/decisions/. validate.py fails on any run of 8 or more words that also appears in the source.

Checks

• python3 -I evals/datasets/decisions-v1/validate.py: 127 cases, 132 decisions, 0 problems (leak check run against the local raw cache)
• bun run ci: pass
• bun run types: pass
• bun test evals/datasets: pass

v2 and registry.json are unchanged.

Known limits

• Each thread was annotated once, by a model, with no second annotator.
• 2 cases are truncated by the 150-event cap.
• 7 threads have a close event rebuilt from closed_at, because the timeline API omitted it.
• Evidence from outside GitHub is recorded only as reported in the threads.

See the README for the full list.

🤖 Generated with Claude Code