Establish a genuinely pristine Phase 1 test benchmark

[NeoLorenzo]

Audit Priority: P1
Audit Type: VALIDATE

Problem / Opportunity

The current 250-map Phase 1 test pool is no longer pristine evidence for final generalization claims because it has been evaluated repeatedly across successive reference models (v3, v4, and v5) and those results have informed ongoing development. The repository already describes it as a fixed held-out benchmark rather than newly pristine evidence, but the active evaluation workflow still uses this same pool for each frozen reference.

Why This Matters

Repeatedly inspecting the same test set creates adaptive overfitting risk at the research-process level even when PPO itself never trains on those maps. As Phase 1 optimization continues, improvements can become implicitly selected against known test behavior. Before treating a future model as a Phase 1 completion/generalization result, PolyVision needs an untouched benchmark whose results have not influenced model, reward, observation, or curriculum design.

Evidence

• README.md states that the current fixed test pool has been used for prior reference benchmarks and is therefore "a fixed held-out benchmark rather than newly pristine evidence."
• docs/evaluation.md records repeated canonical test evaluations for the v3, v4, and v5 reference models on the same 250-map pool.
• docs/evaluation.md still names current test artifacts with pristine_test, despite documenting prior reuse of the pool.
• tools/evaluate_phase1.py still describes --confirm-test as opt-in for "pristine test evaluation" and refuses test runs with the message Refusing pristine test evaluation..., so stale pristine terminology remains in executable tooling as well as documentation.
• docs/results/Phase1_V5_PARITY002_Seed3_16M_TerminalSPT_Reference_Run.md explicitly notes that the current test pool is fixed across reference iterations.
• Completed issue Document the human-benchmark gate before multi-seed Phase 1 runs #1 only documented the sequencing gate for multi-seed Phase 1 work; it did not establish or protect a new untouched test benchmark. Open issue Establish the current Phase 1 human benchmark gate #3 separately owns establishing the human-relative benchmark gate, so neither issue duplicates this final-evidence benchmark work.

Proposed Outcome

Create and freeze a new Phase 1 final-evidence test benchmark from genuine maps that have not previously been used for training, validation, human benchmarking, reference comparison, debugging, or model-selection decisions. Treat the existing 250-map test split as a reusable historical benchmark from that point forward. Add a fail-closed mechanism that makes accidental access to the new final-evidence split materially harder than the current --confirm-test flag and records any authorized access in machine-readable provenance.

Definition of Done

• A new untouched Phase 1 final-evidence map set is created with stable hashes and documented provenance showing no prior experimental use.
• The existing 250-map test split is explicitly reclassified as a reusable historical/fixed benchmark rather than pristine evidence.
• Maintained evaluator CLI/help/error text and artifact naming no longer describe the reused 250-map historical test pool as pristine.
• The canonical evaluator supports the new final-evidence split but fails closed unless an explicit one-time/final-evaluation mechanism is used.
• Any access to the new split is recorded in machine-readable provenance sufficient to determine whether pristine status has been consumed.
• Documentation clearly states which claims may use validation, the historical fixed test benchmark, and the new final-evidence benchmark.
• Automated tests verify split identity, separation from all existing pools, and the access guard.