Bound IMAP body indexing and terminalize unsupported messages

Companion/Decisions/V3/Active/adr-ios-080.md

@@ -0,0 +1,87 @@
+## ADR-IOS-080: Body Indexing Is Bounded, Payload-Selective, and Truthfully Terminal
+
+**Date:** 2026-08-31
+
+**Status:** Active.
+
+**Context.** An IMAP message part can be larger than SwiftMail's ordinary response parser limit.
+Fetching the complete part therefore fails deterministically with `PayloadTooLargeError`; reducing
+the number of messages in a batch cannot split that one literal. Affected rows have complete
+headers but bodies that are neither indexed nor confirmed empty, so body queues stop making progress
+while Fast Sync continues to count the rows as pending forever.
+Increasing the response limit merely moves the failure threshold and raises memory use. Marking the
+rows empty or complete would lie about server content that was never fetched.
+
+**Decision.**
+
+1. **IMAP MIME parts are fetched with validated partial `BODY.PEEK` ranges.** The SwiftMail fork
+   exposes `fetchPart(section:of:offset:count:)`, encodes
+   `BODY.PEEK[section]<offset.count>`, and accepts bytes only from the requested message identity,
+   exact section kind and origin. It rejects ignored/malformed ranges, impossible lengths, extra
+   body literals and responses larger than the requested count while draining the wire safely.
+2. **The app fetches required parts in one-MiB chunks.** Transfer-encoded bytes are concatenated
+   before base64 or quoted-printable decoding, because encoded units can straddle a chunk boundary.
+   The metadata preflight explicitly requests ENVELOPE, INTERNALDATE, FLAGS and BODYSTRUCTURE; it
+   does not use SwiftMail's default full-header option because `BODY.PEEK[HEADER]` is itself an
+   unbounded literal and existing stored headers remain authoritative for non-ENVELOPE fields.
+   A BODYSTRUCTURE size is a planning hint, not truncation authority: after reaching it, the app
+   probes one byte at the advertised endpoint and accepts completion only when the server returns
+   an empty range. Where size is unavailable, a short nonempty response is not treated as EOF and
+   fetching continues until an empty range. The server-reported size does not trigger a full-size
+   allocation before bytes arrive.
+3. **Background body indexing downloads only render ingredients.**
+   `IMAPFetchMapping.isRequiredBodyPart` selects visible `text/plain` and `text/html`, calendar
+   content, and non-attachment CID images. BODYSTRUCTURE metadata is sufficient for normal
+   attachments and opaque `.eml` payloads. Flattened descendants of an attached `message/rfc822`
+   part are excluded component-wise as attachment payload too. Attachment bytes are fetched on
+   demand through the same bounded path. Attached HTML is metadata, not a display body.
+4. **The ordinary four-MiB response parser limit and NSE memory limit stay unchanged.** The bounded
+   transport is the fix; there is no larger-buffer fallback. The NSE uses the same payload selection
+   and chunking, never downloads ordinary attachment payloads, and declines active body rendering
+   before fetching when required BODYSTRUCTURE sizes are unknown or exceed its aggregate admission
+   budget.
+5. **A server that cannot honor partial ranges creates an explicit terminal-unindexed state.**
+   `MessageHeader.bodyIndexingFailureReason = "partial_fetch_unsupported"` means the row is neither
+   indexed nor empty, but is no longer runnable by automatic body queues. Only deterministic partial
+   protocol/assembly failures take this path; transient connection and database failures remain
+   retryable. The write is guarded by the row's full provider address plus positive identity proof
+   from the failed fetch: its exact SELECT UIDVALIDITY epoch must match the stored row whenever both
+   epochs exist; only when epoch evidence is unavailable may a matching fetched RFC 5322 Message-ID
+   serve as fallback proof. A move, re-key, or reused UID therefore cannot attach the outcome to
+   another message, and a duplicate Message-ID cannot override an explicit epoch contradiction.
+6. **Completion reports runnable work and truthful omissions separately.** `pendingBodyCount`
+   excludes terminal-unindexed rows; `unindexedBodyCount` reports them. Once the header walk and all
+   runnable body work finish, the UI says `Sync complete with N messages not indexed` and displays a
+   completed progress bar instead of holding indexing active forever. Smart Reindex clears terminal
+   reasons so a later app or server upgrade can try again.
+
+**Rationale.** A finite chunk bound closes the arbitrary-size-literal class without weakening the
+parser's memory boundary. Payload selection avoids paying attachment memory and network cost during
+background indexing. The terminal state separates three facts that must never be conflated:
+indexed content, confirmed-empty content, and content that exists but cannot be indexed with the
+server's current protocol behaviour.
+
+**Consequences.**
+
+- Large text/calendar/CID parts can be reconstructed beyond 32 MiB while every response literal
+  stays below the ordinary parser limit.
+- Background indexing uses more IMAP commands for large required parts, in exchange for bounded
+  memory and deterministic progress.
+- Normal attachment payloads are absent from background `MessagePart` values and are downloaded
+  only when requested through the same chunked path. BODYSTRUCTURE still supplies their filename,
+  MIME type, section and size; `.eml` attachments are parsed for preview only after that tap-time
+  download.
+- A non-compliant server may leave messages unindexed, but the state is visible, terminal, and
+  retryable by explicit Smart Reindex rather than silently empty, falsely complete, or infinite.
+
+**Tests / evidence.** SwiftMail wire tests reconstruct a body larger than 32 MiB under a four-MiB
+parser buffer and cover ignored/malformed ranges, wrong identity/section/origin, oversized and
+duplicate literals, UID ordering, and unsolicited FETCH updates. iOS tests cover encoded-boundary
+assembly, oversized raw-header exclusion, understated and unknown BODYSTRUCTURE sizes, payload
+selection on the provider's wire hot path, on-demand `.eml` fetching, NSE admission/state
+transitions, identity-safe terminal writes,
+queue convergence, migration/default state, queue exclusion, Smart Reindex recovery, and exact
+completion wording.
+
+**Relates:** ADR-IOS-050 (`bodyComplete` is FTS truth), ADR-IOS-072 (content ownership),
+ADR-IOS-075 (acknowledge only committed cache state), bounded-memory absolute.

Companion/Memory/Current/030-backfill-fast-sync-completion-gate-on-pendingbodycount-never-a-server-to.md

@@ -1,3 +1,13 @@
+<!-- COMPANION-CURRENT-NOTE-BEGIN -->
+> **Current amendment (2026-08-31, ADR-IOS-080):** `pendingBodyCount` now selects
+> `headerComplete=1 AND bodyComplete=0 AND bodyEmptyConfirmed=0 AND
+> bodyIndexingFailureReason IS NULL`. A deterministic failure to honor validated partial IMAP
+> ranges is persisted as terminal-unindexed, counted separately by `unindexedBodyCount`, and shown
+> as `Sync complete with N messages not indexed` once runnable work drains. Such rows are never
+> confirmed empty and never marked indexed; Smart Reindex clears the reason for an explicit retry.
+> This supersedes the preserved body's claim that every eligible row is self-terminating through
+> success/empty and its reference to oversized bodies as confirmed-empty.
+<!-- COMPANION-CURRENT-NOTE-END -->
 
 ### Backfill / Fast Sync Completion — gate on `pendingBodyCount`, NEVER a server total
 - **`BackfillProgress.isFullyComplete` gates on `headersDone && totalEmails > 0 && pendingBodyCount == 0`** (`AccountManager.swift`). `pendingBodyCount` = body-eligible headers still awaiting fetch (`headerComplete=1 AND bodyComplete=0 AND bodyEmptyConfirmed=0`), the same criteria `BackfillBodyQueue`/`ActiveBodyQueue` select on. It is local and self-terminating (empty/404/oversized bodies confirm-empty), so it reaches 0 once the body queues have nothing fetchable left.

Companion/Memory/Current/065-screen-keep-awake-chat-pill.md

@@ -2,4 +2,5 @@
 ### Screen Keep-Awake (chat pill)
 - `Theme/ScreenKeepAwake.swift` — reference-counted wrapper around `UIApplication.isIdleTimerDisabled` + `.keepScreenAwake(while:)` view modifier. The idle-timer flag is a single global, so holders are counted; the modifier tracks its own held state (`@State holding`) so acquire/release fire exactly once per transition regardless of `onChange` vs `onDisappear` teardown ordering.
 - Applied once in `DynamicIslandChatButton.swift` on the pill root: `isExpanded || isWorking || ActiveAgentTracker.shared.anyWorking || speechRecognizer.isRecording` (same scope as the wand-glow indicator). Covers all three host screens (Inbox/Compose/MessageDetail) with no per-screen wiring.
+- `FastSyncView` passes `true` for its entire presentation lifetime. This is intentionally video-playback style: header/body progress, queue idleness, and completion do not release the hold while the view remains open; the modifier releases it on view disappearance.
 - No `scenePhase` handling needed — iOS only honors `isIdleTimerDisabled` while the app is foreground; the hold resumes automatically on return. Reuse `.keepScreenAwake(while:)` for any future keep-awake need (never set `isIdleTimerDisabled` directly).

DECISIONS.md

@@ -127,7 +127,7 @@ These records were authored after `v1.6.38`, so the pinned compaction has no byt
 - **[ADR-IOS-026B — the v3 supersession record](Companion/Decisions/V3/Superseded/adr-ios-026b-v3-superseded-by-068.md)** — *PendingOperation Uses Stable IDs (rfc822MessageId)*, **SUPERSEDED 2026-08-02 by ADR-IOS-068** and retained verbatim as evidence: `MessageHeader.stableId`, `IMAPProvider.resolveUID`'s Message-ID `SEARCH`, dual-match pending-op filtering, the UIDVALIDITY rationale. **Only its durable-mutation-authority layer is superseded** — fetch, normalize, dedup, stage, the AI cross-device cache probe, threading/`References`, and Outbox send de-duplication all SURVIVE; ADR-IOS-068's exempt list is normative. Authored under the colliding number `ADR-IOS-026`, so both search terms find it. The byte-identical `v1.6.38` twin, without the supersession banner, is [`Companion/Decisions/Superseded/adr-ios-026b.md`](Companion/Decisions/Superseded/adr-ios-026b.md).
 - **[Compaction drift list](Companion/Decisions/V3/retained-inline-no-byte-identical-routed-twin.md)** — the retired *Retained inline — no byte-identical routed twin* preamble: check the routed twin before editing a post-`v1.6.38` amendment.
 
-# v3 records (ADR-IOS-068 … 076)
+# v3 records (ADR-IOS-068 … 080)
 
 > **Numbering note.** This file jumps from ADR-IOS-057 to ADR-IOS-068. That gap is deliberate and is
 > itself a record: **ADR-IOS-058, 059, 060, 061, 062, 063, 064, 065, 066 and 067 were authored on a
@@ -149,3 +149,4 @@ These records were authored after `v1.6.38`, so the pinned compaction has no byt
 - **[ADR-IOS-077](Companion/Decisions/V3/Active/adr-ios-077.md)** — Active. Hostile attachment filenames are **REJECTED, not reduced** (`c35cfdca2`, net −476): one shared `AttachmentFilename.isSafeFileComponent` predicate, throw before `createDirectory` on save and refuse before the fetch on download, generic `"Unsupported file name"` for all six rules. Reducer + co-edit twin DELETED — all five confirmed defects lived in the *transformation*, none in the classification. ⚠️ **Rejecting at save does NOT make the loaders safe** — `metaBase`/`afterIndexPrefix` stay load-bearing; type-spoof is bounded, not closed; the combining test is `ccc != 0` on NFD, **not** category `Mn`/`Mc`/`Me`. ⚠️ **Consequence 5 retracts the MIGRATION GUARANTEE — there was never a reducer to migrate FROM** (`v1.7.6`/`v1.7.7`/`v1.7.8` write the name verbatim, so legacy on-disk names are RAW sender-authored; stranded set = refused ∩ writable-by-v1.7.8, 3 narrow shapes). `IOS-ATTACH-001` — forward-only by owner verdict: **no migration, rename-on-load or grandfathering path.** Pre-compaction bullet, byte-for-byte: [pre-compaction-index-lines.md](Companion/Decisions/V3/pre-compaction-index-lines.md).
 - **[ADR-IOS-078](Companion/Decisions/V3/Active/adr-ios-078.md)** — Active. Newest-100 bounds sync-origin AI processing only; existing summaries always display, while action tags remain Inbox-only. `ActiveAIQueue.recentInboxWindowContains`, `AIJob.windowExempt`, `MIS-IOS-018`, #68. [Prior catalog wording](Companion/Decisions/V3/pre-compaction-index-lines-078-079.md#source-line-150--adr-ios-078)
 - **[ADR-IOS-079](Companion/Decisions/V3/Active/adr-ios-079.md)** — Active. Scheduled tasks and `taskCache` are deleted from iOS, remain live on Thunderbird; `[Task]` prose and `disabledReminders` `t:` hashes are retained. [Prior catalog wording](Companion/Decisions/V3/pre-compaction-index-lines-078-079.md#source-line-151--adr-ios-079)
+- **[ADR-IOS-080](Companion/Decisions/V3/Active/adr-ios-080.md)** — Active. IMAP body indexing uses validated bounded partial fetches, skips normal attachment payloads in background, and reports terminal-unindexed rows as `Sync complete with N messages not indexed` instead of retrying forever.

PROJECT_MEMORY.md

@@ -82,7 +82,7 @@ Search the topic text below as subsystem keywords. Each link is mandatory when i
 | Manual Tag Teaching (Long-Press Context Menu) | [read in full](Companion/Memory/Current/062-manual-tag-teaching-long-press-context-menu.md) |
 | Tool Registry (Scaffold) | [read in full](Companion/Memory/Current/063-tool-registry-scaffold.md) |
 | Agent Chat (ADR-IOS-022, ADR-IOS-023) | [read in full](Companion/Memory/Current/064-agent-chat-adr-ios-022-adr-ios-023.md) |
-| Screen Keep-Awake (chat pill) | [read in full](Companion/Memory/Current/065-screen-keep-awake-chat-pill.md) |
+| Screen Keep-Awake (chat pill + Fast Sync full-view hold) | [read in full](Companion/Memory/Current/065-screen-keep-awake-chat-pill.md) |
 | Agent Compose FIFO Queue (ADR-IOS-030) | [read in full](Companion/Memory/Current/066-agent-compose-fifo-queue-adr-ios-030.md) |
 | Outgoing Threading — reply/forward stay in-thread on every provider (ADR-IOS-043, 2026-06-23) | [read in full](Companion/Memory/Current/067-outgoing-threading-reply-forward-stay-in-thread-on-every-provider-adr-io.md) |
 | Incoming Thread Detection — `ThreadDetection.findRelatedMessages` has NO subject-based fallback (by design) | [read in full](Companion/Memory/Current/068-incoming-thread-detection-threaddetection-findrelatedmessages-has-no-sub.md) |