KS78: Recency tie-breaker for score equality (#13)#28
Conversation
- Add microsecond-precision recency epsilon (step 7c7) after inflation cap so newer memories win ties when final_score values are identical - Fix test matcher for google_result to exclude the Meta memory content (both memories contain "Google", causing ambiguous match) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Liorrr has reached the 50-review limit for trial accounts. To continue receiving code reviews, upgrade your plan.
Liorrr
left a comment
There was a problem hiding this comment.
pr-review Summary
PR #28 adds a recency tie-breaker (7c7) to the echo scoring pipeline. After all boosts and the 7c6 inflation cap, a negligible epsilon derived from created_at is added so newer memories win score ties. The tie-breaker logic is correct — the delta between two memories is ~3e-5 per year of age, which is genuinely negligible. One P2 finding on a misleading comment.
Confidence Score: 4/5
P2 only — mergeable with awareness. The single finding is a comment correction; the production logic is sound.
Findings
| Sev | File | Lines | Title |
|---|---|---|---|
| P2 | crates/shrimpk-memory/src/echo.rs |
1632–1636 | Comment misstates cap exceedance as ~3e-5; actual absolute epsilon is ~1.75e-3 |
Changed Files
| File | Risk | Note |
|---|---|---|
crates/shrimpk-memory/src/echo.rs |
Low | Tie-breaker logic correct; comment misleads on cap exceedance magnitude |
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Liorrr
left a comment
There was a problem hiding this comment.
pr-review Summary
This PR adds a recency tie-breaker (step 7c7) to in : after all boosts and the inflation cap, a small epsilon derived from is added to each result's , so that newer memories win when scores are otherwise equal. The test change is a correctness fix: the original could accidentally match the Meta memory (which mentions leaving Google), so the updated predicate adds .
The core logic is sound for the common case (same-year memories, epsilon differential ~3e-5). One issue: the constant produces an epsilon of ~1.7e-3 per memory, and the differential between memories from temporally distant epochs can reach ~3e-4 — large enough to silently override a genuine (if small) score difference. The code comment acknowledges the magnitude (~1.75e-3) but then asserts it only breaks ties, which is self-contradictory.
Confidence Score: 4/5
P2 only — one non-blocking finding. Mergeable with awareness; suggest addressing the epsilon scale or switching to a comparator before the next stable release.
Findings
| Severity | File | Lines | Title |
|---|---|---|---|
| P2 | 1630–1639 | Epsilon overrides meaningful score differences between temporally distant memories |
Changed Files
| File | Issue |
|---|---|
| P2: epsilon constant too large; can reverse real ranking for memories years apart |
Liorrr
left a comment
There was a problem hiding this comment.
pr-review Summary (run 2)
This PR adds a recency tie-breaker (KS78, #13): after all boosts and the inflation cap, a created_at-derived epsilon (timestamp_micros * 1e-18) is added to each result's final_score so that, when scores are otherwise equal, the newer memory sorts first. The tie-breaker mechanism is functionally sound — the ordering-relevant quantity is the delta between two epsilons (~1e-12 per second of separation), which is far above f64 machine-epsilon at score ~1.0, so newer memories reliably win genuine ties without spurious reordering of meaningfully different scores.
The one real risk is that the epsilon is applied after the similarity + 0.50 inflation cap (step 7c6), so a clamped result ends up at max_allowed + ~1.78e-3, breaching the invariant the cap just established and contradicting the comment's "only breaks ties" claim. This is bounded and low-impact, hence P2 not P1.
The only other hunk is a test assertion change (&& !r.content.contains("Meta")) — the author disambiguating the Google lookup to match the new ordering. Both is_some() checks still hold, so it is expected benign adaptation, not a masked regression; per the correctness-only rubric it is not flagged.
Previous P2: STILL OPEN
The run-1 P2 (epsilon overrides the inflation cap / contradicts the "only breaks ties" comment) is unchanged — the PR-head file is byte-identical to run 1. It remains open. Mechanism wording is refined here: the override-of-real-differences framing applies only at decades-scale separation; the defensible defect is the cap breach.
Confidence Score: 4/5
One P2, no P1s. Tie-breaker logic is correct; the only concern is the bounded (~1.78e-3) breach of the inflation-cap invariant.
| Severity | File | Line | Issue |
|---|---|---|---|
| P2 | crates/shrimpk-memory/src/echo.rs | 1636 | Recency epsilon applied after inflation cap, breaches similarity + 0.50 invariant (STILL OPEN) |
Liorrr
left a comment
There was a problem hiding this comment.
pr-review Summary (run 3)
Previous P2 (epsilon applied after the 7c6 inflation cap): RESOLVED
The code keeps the after-cap placement, but this is correct-by-design, not a tolerated wart:
- Run-2's suggested fix (move epsilon above the cap) would actually break the feature — any result clamped to
similarity + 0.50would have its epsilon erased by the clamp, so two capped results with equal similarity would tie exactly and the tie-breaker would do nothing. After-cap placement is the only way the tie-breaker functions on capped results. - The cap breach is bounded, uniform (~1.78e-3 added near-equally to every result), and cannot reintroduce the unbounded boost stacking the cap exists to prevent.
- The comment's claim "only breaks ties, never meaningful score differences" is verifiable: the ranking-relevant quantity is the differential
(t1 - t2) * 1e-18, which is ~3e-4 even for memories a decade apart — far below cosine score gaps in the 0.5–0.9 range. Genuinely ties-only.
The author also added an explicit comment acknowledging the cap breach, satisfying the documented-acknowledgement criterion.
Confidence Score: 5/5 — No open findings. The flagged behavior is functionally necessary and benign-by-construction; the only other diff hunk is a test-matcher tightening (contains("Google") && !contains("Meta")), which improves robustness with no correctness impact.
No findings — recommend merge.
Addresses PR #28 review (P2, all runs): the 7c7 epsilon added `created_at.timestamp_micros() * 1e-18` to final_score *after* the 7c6 inflation cap, breaching the `similarity + 0.50` invariant the cap just set (every current-era score lifted ~1.75e-3 above the cap) and, in principle, reordering memories separated by decades. - Remove the 7c7 epsilon block; final_score is no longer mutated post-cap. - Add pure `cmp_score_then_recency` comparator: order by descending final_score, break *exact* f64 ties by descending created_at. A genuine score difference is never overridden by age, so the cap invariant holds. - Exact ties are real here: the cap clamps two equal-similarity memories to the same value (issue #13 panic: 0.771923 == 0.771923). Applying the tie-break after the cap is what makes it fire; before the cap it would be clamped away. - Rewrite the #13 e2e test to assert ranking ORDER (Meta above Google), not a strict score inequality that is unsatisfiable for a genuine tie. - Add model-free unit test `cmp_score_then_recency_breaks_exact_ties_by_recency` so the tie-break has executed coverage in CI (the e2e test is #[ignore]'d). Verified: clippy clean, 392 passed / 0 failed, e2e #13 test passes with the real BGE model. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses PR #28 review (P2, all runs): the 7c7 epsilon added `created_at.timestamp_micros() * 1e-18` to final_score *after* the 7c6 inflation cap, breaching the `similarity + 0.50` invariant the cap just set and, in principle, reordering memories separated by decades. - Remove the 7c7 epsilon block; final_score is no longer mutated post-cap. - Add pure `cmp_score_then_recency` comparator used in the 7d sort: order by descending final_score, break *exact* f64 ties by descending created_at. A genuine score difference is never overridden by age, so the cap invariant holds by construction. - Add model-free unit test `cmp_score_then_recency_breaks_exact_ties_by_recency` — this is where the exact-tie behaviour is actually exercised in CI. - Rewrite the #13 e2e test to assert ranking ORDER (newer ranks above older). NOTE: under current scoring the original #13 tie no longer reproduces. Issue #13 reported both memories at exactly 0.771923; today supersession demotion (KS78 #11) drops the superseded "Google" memory to ~0.32, so Meta wins on score outright (meta=0.7719 vs google=0.3213, observed locally). The comparator is therefore defensive logic for genuine exact ties and is verified by the unit test, not by the e2e test (which now passes on score separation, not on the tie-break). Verified: clippy clean, 392 passed / 0 failed, e2e ordering test passes with the real BGE model. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
a6da44f to
8de767a
Compare
Summary
Adds a recency tie-breaker (KS78, #13) so that when the echo ranking produces an exact
final_scoretie, the newer memory sorts first.The tie-break is a pure comparator in the 7d sort step — it never mutates
final_score:final_score; on an exact f64 tie (partial_cmp == Equal), order by descendingcreated_at.final_scoreis untouched after the 7c6 inflation cap, thesimilarity + 0.50cap invariant is preserved.Status of the original #13 symptom
Issue #13 reported both memories scoring exactly
0.771923. Under current scoring that tie no longer reproduces: supersession demotion (KS78 #11) drops the superseded "Google" memory to ~0.32, so Meta now wins on score outright (meta=0.7719vsgoogle=0.3213, observed locally with the real BGE model). The comparator is therefore defensive logic for genuine exact ties — its behaviour is verified by a model-free unit test, not by the e2e test (which now passes on score separation).Why not the earlier epsilon approach
The previous commit added
created_at.timestamp_micros() * 1e-18tofinal_scoreafter the cap. That lifted every current-era score ~1.75e-3 above the cap (breaching the invariant 7c6 had just set) and, at decades-scale separation, could reorder genuinely different scores. The comparator removes both problems by construction.Changes
cmp_score_then_recency.cmp_score_then_recencyhelper (extracted so it is unit-testable without an engine/embedding model).cmp_score_then_recency_breaks_exact_ties_by_recency— this is where the exact-tie behaviour is actually exercised in CI.&& !r.content.contains("Meta")finder disambiguation (the Meta memory also contains "Google").Test plan
cargo clippy -p shrimpk-memory --all-targets -- -D warnings— cleancargo test -p shrimpk-memory— 392 passed, 0 failed, 21 ignoredcargo check --workspace— all crates compilerecency_boost_newer_memory_ranks_higher— run locally with the real BGE model: passes (Meta ranks above Google)Closes #13.
Generated with Claude Code