epic: benchmark historical retrieval before admitting project memory

[sungjunlee]

Outcome

Measure whether agents actually need a compiled historical retrieval layer, and admit only the smallest layer that demonstrably outperforms live GitHub/Git/spec search.

Baseline

Collect 20 representative historical questions from dev-backlog, dev-relay, and consuming-repository work. Compare:

1. raw task-mirror grep,
2. gh + Git history + existing spec/ADR/context sources,
3. a non-committed, on-demand compiled report.

Guardrails

• No committed topic graph or automatic per-PR memory update in the initial experiment.
• Derived output contains no authoritative current status.
• Every material claim links to an Issue, PR, commit, ADR, or spec source.
• Freshness distinguishes compiled_at, sources_through, and human_verified_at.
• Partial source/API failure preserves existing output and performs no write.

Go criteria

• Top-3 source recall is at least 90%.
• Material claims have source pointers and no major factual error is observed.
• Median retrieval time or tool calls improve by at least 20%.
• Monthly maintenance cost is at most 15 minutes.
• The result is reused for real historical questions during a 4–6 week shadow period.

Decision

• If the baseline sources are sufficient, close with no compiler.
• If the on-demand experiment wins, propose a separate project-memory skill and human-gated charter amendment.
• A committed artifact requires a separate follow-up decision with a single integration writer, source cursor/CAS, dry-run diff, and regenerate-not-merge semantics.

Dependencies

The baseline can start immediately after the authority contract. Any productization decision waits until the mirror-off pilot has produced real usage evidence.

[coderabbitai]

🔗 Related PRs

#174 - feat(backlog-triage): add spec-aware Decision Review [closed]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[sungjunlee]

Shadow benchmark started — 2026-07-31

The 4–6 week evidence clock starts today. Earliest 4-week decision date: 2026-08-28; 6-week boundary: 2026-09-11. No compiler or committed memory artifact is admitted during the shadow.

Arms

• A: raw legacy task-mirror grep
• B: live gh + Git history + existing spec/ADR/sprint context
• C: non-committed on-demand compiled report

Pre-registered gates

• 20 gold historical questions spanning dev-backlog, dev-relay, and at least one consuming repository
• macro top-3 source recall >= 90%
• zero major factual errors; every material claim has a source pointer
• arm C improves median retrieval time or tool calls by >= 20% over the winning baseline
• measured maintenance <= 15 minutes/month
• at least 10 organic shadow questions, across 3 repository classes, with at least 2 weeks of real reuse before decision

Per-question log

Record question ID/class, observed_at, arm, elapsed_ms, tool_calls, top-3 source pointers, gold-source hits, major/minor errors, freshness (compiled_at, sources_through, human_verified_at), maintenance minutes, and whether the question was organic or benchmark-only. Partial source/API failure preserves prior output and records no write.

The initial source-presence audit found the legacy mirror corpus cannot answer most planned historical questions, while live GitHub/Git/spec sources remain the authority-bearing baseline. Exact 20-question gold set and measurements will be attached as implementation evidence; no go/no-go conclusion is claimed on day 0.

[sungjunlee]

Day-0 benchmark support merged — PR #356

The frozen protocol, 20 gold questions, per-question observations, scoring rules, and failure-atomicity evidence are now on main in docs/historical-retrieval-shadow.md.

Results

• Arm A, legacy mirrors: 2/20 direct source coverage; unavailable as a complete baseline.
• Arm B, live sources: 20/20 top-3 question hits, zero major errors, median 5.952 s and 2 calls.
• Arm C, non-committed disposable report: 20/20 question hits, 37/41 eligible Issue/PR pointers, zero major errors, conservative median 0.794 ms and 1 local query call.
• Projected maintenance: 6.27 min/month.
• Missing-source probe: non-zero exit with identical report SHA-256, mtime, and size and no candidate residue.
• Environment: gh 2.97.0; the earlier epic: validate GitHub Projects as an optional planning projection #349 pilot remains recorded against its original gh 2.85.0 environment.

Linux and Windows CI passed. An independent review found no remaining issues.

Still open

These are synthetic Day-0 results. They do not satisfy the reuse gate. #350 remains open until there are at least 10 organic historical questions across dev-backlog, dev-relay, and consumer repositories, at least two weeks between first and last reuse, actual maintenance is measured, and the earliest decision date 2026-08-28 has arrived. No compiler, topic graph, or committed memory artifact is admitted by this PR.