fix: dedupe overlapping archive segments in the rotated corpus

[buluoray]

Closes #8484. Split out of #7916 under a human override: the finding was real, but low-probability, so it did not hold that PR.

The seam

read_rotated_messages concatenated archive segments with a bare rows.append(row) and no cross-segment identity check. drop_persisted_tail_prefix guards the archive↔live seam — it cannot reach this one, because the duplicate is already inside rotated before that comparison runs.

How two archive segments come to overlap

HistoryRewriter._maybe_rotate writes the archive before rewriting the live file, deliberately: until the archive lands, the live file is the only copy of those rows, so archiving is a precondition of the rewrite rather than a best-effort side effect.

A hard crash between those two steps leaves dropped in both places:

1. archive segment N is written;
2. the process dies before the live-file rewrite;
3. the live file still begins with those same rows;
4. the next rotation computes dropped' = message_lines[:-keep_count], which starts with that same prefix, and archives it again as segment N+1;
5. segments N and N+1 overlap, and the reader concatenates both.

The header-damage branch a few lines above already named this hazard — "a retry rotation can archive the same rows successfully, and then a partial read of this segment duplicates them" — but nothing guarded it.

Why a duplicate is worse than showing a row twice

read_rotated_messages is the pagination index space: before / next_before cursors and the fork index path both resolve a rendered row's position against it. A duplicated row shifts every index above it, silently. A reader pages positions that no longer mean what they meant, and an index-addressed fork copies a different cutoff than the one on screen. Nothing raises.

The fix

Accumulate each segment's rows, then apply drop_persisted_tail_prefix against the corpus so far before extending. Same rule, same helper, one level down — no new identity logic. Both producers persist a prefix, never an interior slice, so the duplication is always a segment's head against what the corpus already ends with, which is exactly the shape that helper was written for.

Test power — verified in both directions

test/test_archive_segment_overlap.py, 6 cases. Three pin the fix and three pin its limits, so the suite fails if the guard is missing or too greedy:

mutation	reddens
drop the dedup (rows.extend(seg_rows))	wholly_duplicated → ['1','2','1','2','3'], partially_duplicated → ['1','2','2','3'], overlap_matched_by_stable_id
widen the identity rule to (ts, role) only	distinct_rows_sharing_ts_and_role_are_both_kept

The three upper-bound cases stay green under the first mutation by design — they assert the guard does not eat rows — which is why the second mutation is what proves they are not vacuous.

non_overlapping_segments_are_all_served keeps a healthy archive intact; an_interior_repeat_is_not_treated_as_overlap pins that a row reappearing mid-segment is the reader's own repeated message, not a re-archived prefix.

Cost

The helper's all(...) short-circuits on the first mismatch, so the healthy no-overlap path costs one comparison per candidate offset, and the whole read is cached on the segment files' stat signature — it runs once per archive change, not once per page.

Checks run locally

pytest on the new file plus the five adjacent archive/fork/pagination suites (51 passed), isort, flake8, mypy --platform linux, and the black gate. The full suite is left to CI.

Why no screenshot: backend-only change to a transcript read projection. There is no user-visible surface in this diff — the observable effect is the absence of duplicated rows in a corpus that only appears after a crash-interrupted rotation, which is not reproducible in a screenshot.

[github-actions]

Design Review (Fable 5) — ✅ PASS

Design-level review of 4056e2ba5691ca8bab8f329fecdd9169a5d1a223 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Right layer, right rule: the one function assembling the rotated corpus reuses the existing seam helper, and reader-side dedup also heals corpora already damaged on disk.

The reader-side placement is the correct root-cause choice, not a symptom patch: a writer-side guard in _maybe_rotate could only prevent future overlaps, while this heals segments already duplicated by past crashes, and it mirrors how the archive↔live seam is already handled (live = drop_persisted_tail_prefix(rotated, live) at history_projection.py:537). The failure mode when the prefix shape doesn't hold is append-everything — no worse than pre-fix — and the incomplete-read path still raises before caching, so the guard never composes with a partially-read archive. Tests pin both the fix and its upper bound (interior repeats and same-second distinct rows survive), which is what keeps the deleting-matcher honest.

[DESIGN-REVIEWED] 4056e2b

[github-actions]

Opus 4.8 Review — ✅ no blocking findings

Reviewed 4056e2ba5691ca8bab8f329fecdd9169a5d1a223 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 4056e2b

Verdict parsed from the review's SHA-scoped output markers for commit 4056e2ba5691ca8bab8f329fecdd9169a5d1a223.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 4056e2ba5691ca8bab8f329fecdd9169a5d1a223: <one-sentence reason>

[github-actions]

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 4056e2ba5691ca8bab8f329fecdd9169a5d1a223 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All claims verified: the writer ordering in history_rewrite.py:182-224 (archive first, rewrite second), the existing helper drop_persisted_tail_prefix at history_projection.py:58 designed for exactly this shape, and the fact that read_rotated_messages is the single shared segment-concatenation point (both read_messages_chained_full and read_rotated_messages_chained route through it). One sibling consumer of overlapping segments exists (session_digest.py), which only counts turns. Final review:

First-Principles-Verdict: PASS

A reported crash-window defect fixed at the one shared reader with the existing helper — no new surface, no new identity rule, tests pin both directions.

What this change ships

Intent: after a crash-interrupted rotation, serve each archived row once so pagination cursors and fork cutoffs land on the row on screen — a FIX.

1. Rows re-archived by a retry rotation now appear once in the transcript corpus — justified (reported defect, split from perf(chat): reachable archived history and stable phone scrolling #7916).
2. A segment head matching the corpus tail is silently dropped, bounded by the existing meta.mid / (ts, role, content) rule — justified, declared.
3. Six-test file pinning the guard and its over-greed limits — rides with the fix, declared.

No new config key, flag, or public symbol; the guard reuses drop_persisted_tail_prefix (history_projection.py:58), the same rule already applied at the archive↔live seam (history_projection.py:537), so lens 4's answer is "it uses the existing one." The fix sits at the shared mechanism: both chained readers route through read_rotated_messages, so this is one fix, not a point patch — and the true cause (two file writes cannot be atomic; archiving deliberately precedes the rewrite to fail closed, history_rewrite.py:189-207) is a physical constraint, with reader-side healing the only fix that also repairs corpora already on disk.

Watch

One counted sibling of the root cause: grepped ARCHIVE_SEGMENT_DELIMITER consumers — session_digest.py:65-88 concatenates segments plus the live file with no dedupe at either seam, so an overlapping corpus double-counts turns. A cosmetic count, not an index space; accepted-and-deferred, not a demand.

[FIRST-PRINCIPLES-REVIEWED] 4056e2b

[github-actions]

GPT 5.6 Review — 🔴 changes requested (blocking)

GPT 5.6 found at least one blocking issue that must be resolved before merging 4056e2ba5691ca8bab8f329fecdd9169a5d1a223. 1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- src/kiro_crew/history_projection.py:471 -- Reused message IDs delete distinct archived rows

rows.extend(drop_persisted_tail_prefix(rows, seg_rows))
Distinct /api/chat messages sharing caller-supplied meta.mid -> rotation places them across segment boundaries -> the latter row disappears from pagination and fork indices.
Anchor: residual/crash-data-loss-corruption
Fix: Corroborate matching IDs with equal body or timestamp before dropping the prefix.
[BLOCK-MERGE] 4056e2b
[GPT-REVIEWED] 4056e2b
False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4056e2ba5691ca8bab8f329fecdd9169a5d1a223: <one-sentence reason>