[Plan 2026-04] AI quality/strength/diversity/production tracking

[an0mium]

Overview

Tracking issue for the plan at docs/planning/AI_QUALITY_STRENGTH_DIVERSITY_PLAN_2026-04-16.md.

Week 1 issues

• [A1] Add per-seat WR tracking to quality gate #78 — A1 Per-seat WR tracking in quality gate (shipped 98736c5; deployed to gh200-13)
• [A2] Add plateau detector with optional auto-threshold relaxation #79 — A2 Plateau detector (shipped 24bf557)
• [C2] Expose heuristic personas in production UI #80 — C2 Expose personas in production UI (shipped 5aa0fe3 + 186a74b + 1d9393a; flag-gated; ready to flip)
• [D1] Model-version telemetry on /ai/move #81 — D1 Model-version telemetry on /ai/move (shipped cbcd73b; live in production)
• [D5] Silent-fallback observability and alerting #82 — D5 Silent-fallback observability (shipped 6ec5c8e; live in production)

Review follow-ups (closed)

• [D5 follow-up] Circuit breaker isOpen=true during half-open allows concurrent trial requests #83 Circuit breaker half-open concurrency (shipped f7df4cd)
• [A2 follow-up] Plateau detector re-reads its own metrics.jsonl every iteration #84 Plateau detector in-memory history (shipped 81d3295)
• [D5 follow-up] Collapse CircuitBreaker isOpen + state into single state machine #85 CircuitBreaker state-machine consolidation (shipped f7df4cd together with [D5 follow-up] Circuit breaker isOpen=true during half-open allows concurrent trial requests #83)
• [D5 follow-up] getLocalFallbackMove labels ai_type and difficulty as 'unknown' #86 getLocalFallbackMove tier context (shipped 61621ef)

Other work shipped along the way

• A3 training-probe model_version regression test (b714c63)
• Security sweep 19 → 4 LOW npm findings (30c73b6)
• v5-heavy compatibility fixes: bootstrap (c9a4302) + runtime infer (5764c26)
• Test-stub unblock after game-granular resume (bcbf395)
• Evidence-artifact automation (787f4a2 by Codex)

Week 2–3 next

• B2 v5-heavy pilot — running on gh200-11 now (iter 1 selfplay post-fix)
• C1 Ensemble serving for D9–D10 tiers
• C3 Varied multiplayer seating — leverages the personaIds[] array C2 shipped
• D2 Hot reload for new checkpoints
• B3 Seat-stratified value loss (if gh200-12 iter 26 seat_wr confirms imbalance)

Success metrics

• At least one config above 2000 Elo within 8 weeks
• square8_3p above 1600 Elo within 4 weeks
• Production serves ≥ 4 distinguishable personas — ready via C2; pending flag flip
• p95 inference latency within per-tier SLO
• Fallback rate < 1% under normal operation (baseline observable via D5 telemetry)

Related

• Plan doc: docs/planning/AI_QUALITY_STRENGTH_DIVERSITY_PLAN_2026-04-16.md
• Security triage: docs/security/npm_audit_triage_2026-04-17.md
• Independent review: docs/reviews/week1_commits_review_2026-04-17.md