feat(memory):feedback-ranking-tie-breaker-evaluation

[SunwardL]

P3-B feedback tie-breaker closeout / PR description

Date: 2026-09-23 15:36 Asia/Shanghai
Branch: codex/prepare-memory-feedback-tiebreaker
Base: KunAgent/Kun:develop
P3-A dependency: PR #1324 merged as 8974700b

PR title

feat(memory): evaluate feedback ranking tie-breakers

PR description

Summary

This PR adds an offline, versioned evaluation of whether explicit Memory confirmation and correction evidence can safely break lexical near-ties. The frozen development evaluation found no eligible feedback candidate: the best feedback configurations improved the small local pair-ordering set, but still violated the pre-registered zero-explicit-forbidden-selection gate. The result is a development no-go, not a production-ranking change and not a holdout decision.

The PR also synchronizes and archives the already-merged P3-A feedback-evolution specification. The P3-A ledger and correction infrastructure remain intact; feedback stays disabled by default, and no feedback signal is wired into production retrieval ranking.

Changes

• Add synthetic, stratified Memory/feedback fixtures, a frozen manifest and checksums, development-only calibration, a finite candidate grid, a locked plan, and reproducible development reports.
• Compare foundation-only, confirmation-aware, and confirmation-plus-correction-aware near-tie candidates. Retrieval frequency is recorded as a shadow feature only and cannot change candidate ordering.
• Add paired-bootstrap, safety, privacy, deterministic-replay, resource-ceiling, correction/replay, and evaluator-isolation tests.
• Retire the v1 holdout runner after an execution-integrity audit. Preserve the historical output unchanged, but explicitly exclude it from decision-grade evidence.
• Synchronize the canonical P3-A feedback, Memory-record, and retrieval specifications and archive the completed P3-A change in this same PR.

Development decision

The frozen development partition contains 18 synthetic cases, including four preference pairs. The foundation baseline has pair accuracy 0, Recall/Precision/MRR 0.4375, and six explicit forbidden selections. The strongest confirmation-plus-correction candidates have pair accuracy 0.75, Recall/Precision/MRR 0.625, and three explicit forbidden selections. Their authorization/lifecycle violation count is zero, but the hard gate allows zero explicit forbidden selections, so every candidate fails the complete gate. The fallback lock selects foundation-control; a valid lock is not a passing decision.

The confidence-bound and gate data remain frozen. No threshold, fixture label, or gate was changed to obtain this outcome. These synthetic development metrics are not estimates of production traffic.

Holdout integrity disclosure

The originally planned one-time holdout execution is not claimed as complete. A later audit found at least five successful test invocations plus the repository write that scored the real frozen holdout through temporary output directories; the exact count including internal deterministic replays is not recoverable, and its recorded holdoutRunCount: 1 did not represent all attempts. The runner also reported some privacy, determinism, and resource flags without measuring all of them. Therefore the historical artifact is retained only as an execution record, not as a valid holdout result. The v1 runner now rejects execution, and this PR does not rerun or reinterpret that version.

The final v1 disposition is to close at the development no-go and disclose the deviation from the planned holdout step. Any renewed experiment requires a new decision version, fresh execution controls, and independent review.

Frozen development inputs

• Fixture SHA-256: a6a37e35a5da6afdfbdae42b9abe8a9f182dcad19167fc82361f662873d7e259
• Manifest SHA-256: 17ab0ea2186d11bbdfb38e0f76b75f3547144c74412183e8118cb2e525d6124a
• Calibration SHA-256: b34b6d995a2230aac543885d4173c07f8e3024b4fd8603637d5f49e4258a7964
• Decision-plan SHA-256: 27dfc5852afa9d86e0ff52bf549fb154e5b788ac00690758c0c8accd15460ab7
• Development-report SHA-256: c66eecdd069a6507bf158a980e25a556f87fb39f69b6674bede77470d70b1542

Validation

• P3-B focused tests: 19 files / 71 tests passed.
• Kun Memory tests: 57 files / 296 tests passed.
• npm run build:kun: passed.
• npm exec -- eslint kun/src/memory: passed.
• Strict OpenSpec validation for evaluate-kun-memory-feedback-tiebreaker: passed.
• git diff --check: passed.
• Rebased on upstream/develop at 59e121018; the hosted typecheck and file-line failures are addressed by the upstream fix(rooms): align model settings state with the saved modelRef contract #1341 baseline fix.
• Local focused renderer test: 5/5 passed; changed-file ESLint and npm run check:file-lines passed.

Baseline rebase and follow-up

The first PR check exposed two upstream baseline problems: the AgentModelSettings.tsx saved-model-reference type mismatch and two files over the 700-line gate. Latest develop now contains #1341 (163984da3), which fixes the type contract and splits the oversized files. After rebasing, this PR keeps only a focused regression test in commit c5207b804, verifying that a provider-less fallback option is treated as inheritance (modelRef: null) rather than sent as an invalid persisted model reference. No duplicate production fix is included.

Scope and follow-up

• No production retrieval ranking, Memory runtime composition, renderer toggle, or feedback-weight change is included.
• feedback.enabled remains off by default. P3-B no-go does not invalidate P3-A ledger/correction infrastructure, but it does not authorize using feedback to rank production memories.
• A production-ranking proposal requires a separate, newly versioned decision that passes its own relevance, uncertainty, safety, privacy, determinism, and resource gates. P4-A relation suggestions remain a separate future stage.

Follow-up: room model-reference typecheck fix

The initial PR check exposed a mismatch in AgentModelSettings.tsx: its broad AgentModelBinding allowed a missing providerId, while the persistence API requires the stricter AgentIdentity model-reference shape. Commit 4291b324a narrows the editor state to that persisted contract and disables options that cannot be saved without a provider ID. This adjacent fix does not change Memory behavior.

Validation: AgentModelSettings.test.ts passed (5/5); ESLint for the two changed files and git diff --check passed. A fresh hosted CI run will verify the complete dependency-installed typecheck.