Make the oversized-metadata quarantine durable so sync can complete

[tabmail-kmyi]

Closes #104. Companion to #74, which stays open as the tracker for the real (upstream parser) fix — do not close #74 on this.

This does not make an oversized body fetchable. It stops the app paying for a fetch that cannot succeed, and stops it asking the user to wait for one.

The bug

handlePayloadTooLarge quarantines an oversized message in oversizedDeferredThisSession, an in-memory, process-lifetime set. Every launch rebuilds it empty, so every affected message is offered to the body queues again, fails again, and is quarantined again. Each failure is expensive: withFolderConnection classifies PayloadTooLargeError as unhealthy and releases the connection, so the retry pays a full TCP + TLS + LOGIN + SELECT.

Consequences: the body is never indexed; BackfillProgress.pendingBodyCount never reaches 0, so the sync banner never clears and the progress bar parks short of 100%; and opening the message spends a connection to land on "Unable to load message", with a 2s poll repeating it.

Correcting the premise the old comments carried

Several comments described the overflow as size-deterministic per binary. It is not — the parser's bound is on unread aggregate bytes measured after the decode loop stops, so it depends on wire fragmentation. The same message can overflow on a lossy link and parse fine on WiFi.

So an overflow is an observation about one wire attempt, never a verdict that the body is unfetchable, and nothing here treats it as one.

The change

A durable messageHeader.bodyMetadataOversized flag.

• v88_addBodyMetadataOversized adds only the column. The supporting index messageHeader_bodyRepopulateV2 is built off the blocking launch path by SyncEngine.deferredIndexes, per ADR-IOS-029's 2026-08-05 amendment — its absence degrades performance and nothing else, which is that ADR's own eligibility test, and v83_markAllAsReadUnreadSweepIndex is the precedent (its body is intentionally empty for exactly this reason).
• One writer symbol, four callers. Every path that observes an overflow records it through BodyFetchProcessor.markBodyMetadataOversized: the singleton branch of handlePayloadTooLarge on both queues, BodyFetchProcessor.fetch's PayloadTooLargeError branch, and the inbox snippet loader's network tier. That last one matters — it calls the same provider.fetchMessage the queues do, and on a scrolling user it is frequently the first path to reach a deep-history message, since backfill admission is date DESC. A tier that observed an overflow and only remembered it in process memory would make the flagged population "whatever a background queue happened to reach first", which is precisely what this column must not be. A multi-item overflow still isolates rather than flagging: the batch error does not say which item was too large.
• All four marks go through ONE serialized write chain, shared with the UIDVALIDITY reset's clear, so a mark and a clear can never commit out of order. Without that, a mark enqueued just before a folder turnover could land after the reset's clear and the resync, and quarantine a fresh-epoch row that reused the UID — passing both guards below, because that row genuinely lives at that address and genuinely has no body yet.
• The writer carries two guards, both load-bearing. AND bodyComplete = 0 keeps a completed row from being handed a stale flag. AND id = accountId || ':' || folderPath || ':' || messageId refuses to write inside an optimistic-move window — optimisticMoveToFolder rewrites a row's columns to the destination while its primary key still encodes the source folder and UID, so bytes fetched at the columns' address in that window belong to a different message, and an overflow observed there is not evidence about the row its key names.
• Eight read-side initiators, one predicate, one authoritative gate. Every path that can start a fetch for a flagged row is gated on the single symbol MessageHeader.isBodyQuarantined (bodyMetadataOversized && !bodyComplete) or, for the SQL half, on the single hoisted Active/BackfillBodyQueue.admissionSQL:

#	initiator	half	role
1–4	repopulateFromDatabase and repopulateOnDrain, on each queue	SQL	the guard
5	AccountManagerFetch.fetchBody — the funnel every on-demand fetch goes through	predicate	the guard
6	MessageDetailViewModel.loadBody (the user open)	predicate	UI state
7	MessageDetailViewModel.startBodyPoll — the 2s body poll	predicate	keep-polling decision
8	InboxViewModel.loadSnippetBatch tier 2	predicate	skip + record

Row 5 is the authoritative one: the network gate lives at the funnel, not only in the callers, so a future caller cannot open a hole by forgetting it. Rows 6–8 are not the guard — they decide which UI state to show and whether to keep polling. The poll and the snippet loader are the two initiators this change adds.

• And one state the flag structurally cannot cover. Both bodyComplete terms are deliberate: the writer's AND bodyComplete = 0 stops a completed row acquiring a stale flag, and isBodyQuarantined's && !bodyComplete is the eviction fail-safe. Together they make one row invisible at every site — fetched once, messageBody later evicted by BodyAssetMaintenance (which leaves bodyComplete = 1 by design), re-fetch now overflows. Nothing records it; nothing gates on it; it polls forever. BodyFetchRefusal gives the funnel's four refusal classes an identity, and endsPolling is what loadBody and the poll's catch consult before continuing. It also collapses three hand-copied user-facing strings and four bare NSError codes into one place. Both mattered: loadBody's cancelled-read exits all call startBodyPoll(); return before its quarantine branch, so a flagged row was retried every 2s indefinitely, each attempt paying a full TCP + TLS + LOGIN + SELECT; and reloadMessages clears snippetFailed and re-queues the visible window, so tier 2 re-attempted the same fetchMessage that overflowed on every reload. The poll reads the flag fresh from the database each tick, because a background queue can flag the row while the poll is running.
• Nothing is classified, terminalized or deleted. bodyComplete and bodyEmptyConfirmed stay 0, emptyFetchCount and missFetchCount are untouched, and the header stays FTS-indexed — the message is still findable by subject and sender.

Two product decisions, made deliberately

Both reverse a non-change an earlier revision of this work had built and defended; the superseded reasoning is kept in the source and tests rather than deleted.

The sync banner is allowed to complete. Backfill progress counts a flagged row as settled, so pendingBodyCount can reach 0. A banner that can never clear, over work the build cannot perform, is worse than rounding an unfetchable message up to done. The indexed numerator moves with pending, or the bar contradicts the check.

Opening a flagged message reports failure immediately. loadBody presents the same state a failed fetch leaves behind — error shown, no spinner, header rendered — with no wire attempt and no poll. Pull-to-refresh is deliberately unchanged and still performs a genuine fetch, which is what keeps the flag an observation rather than a verdict.

Releases

• a UIDVALIDITY reset for the folder (the address no longer names the same message);
• any successful body write — all four of them. A written body is positive evidence refuting the observation, and this is load-bearing: BodyAssetMaintenance evicts a messageBody row while deliberately leaving bodyComplete = 1, and the detail view's cache-miss fetch is the only recovery. A stale flag would delete that recovery and brick a message already fetched once. The open path carries && !msg.bodyComplete as the fail-safe for any residual;
• Smart Reindex — its statement needs the flag in both the SET and the WHERE, or it silently skips the rows the gesture was invoked for;
• pull-to-refresh on the open message;
• and, when the raised parser bound ships, one statement in that migration: UPDATE messageHeader SET bodyMetadataOversized = 0 WHERE bodyMetadataOversized = 1. Exact by construction, because every row carrying the flag was written by this code.

That last point is why this earns a dedicated column rather than reusing an existing state: a reused value cannot name its own population.

Verification

Full suite green on a dedicated simulator and a dedicated derived-data directory, both created for this branch and deleted afterwards: 9,404 tests in 1,260 suites passed, one pre-existing known issue (SyncFolderEpochPersistenceTests, unchanged by this branch), no test-host restarts, and no actionable compiler warnings — only the three documented-benign appintentsmetadataprocessor lines (one per target that runs the processor).

The run is corroborated as complete rather than truncated: a static census of @Test declarations in TabMailTests/ returns 9,404, exactly the number executed. The same census on the base commit returns 9,337, so this branch adds 67 tests net. (The census must be anchored — ^\s*@Test — because eight doc comments in TabMailTests/ mention @Test in prose and an unanchored count reads 9,410, an eight-test phantom surplus. No test carries a .disabled trait, so the identity is exact rather than a coincidence.) A truncated run cannot match its own census, which is the check that matters here — an earlier run in this work reported a plausible-looking 236 tests because the test host was crash-restarting on a polluted simulator container.

Red-first proved by focused mutation for every new invariant — invert the fix and the test that pins it fails while its control stays green:

mutation	test that goes red
drop !msg.bodyComplete from the open path	evicted-row recovery
drop bodyMetadataOversized = 0 from the flushBatch success write	success retracts the observation
narrow resetCrawlState's WHERE to bodyEmptyConfirmed = 1	Smart Reindex releases the quarantine
put the CREATE INDEX back in v88	migration does not build the index
drop the conjunct from either repopulateFromDatabase	that queue's relaunch test
flag on a stale generation	generation guard
drop await previous?.value from enqueueDurableWrite	durable write ordering
drop the re-minted-key guard from markBodyMetadataOversized	mid-move misattribution
delete the funnel's quarantine block	the funnel's refusal
stop the snippet loader recording what it observed	snippet-loader durability (control: non-overflow failure)
drop bodyMetadataOversized = 0 from the NSE batch flip	NSE body clears the observation
drop , bodyMetadataOversized = 0 from oneTimeBodyCompleteRestore's heal	the restore releases the quarantine on healed rows only
if BodyFetchRefusal.endsPolling(error) → if false in the poll's catch	a running poll ends on an overflow the durable flag cannot see
drop the bodyMetadataOversized conjunct from backfill progress	a quarantined row lets an account reach Sync Complete
drop AND m.bodyEmptyConfirmed = 0 from the diagnostics predicate	the bodyless buckets stay an exact partition
drop AND folderPath = ? from the durable clear	the clear releases one folder, not the account
drop AND accountId = ? from the durable clear	…and not the device
the funnel's fresh DB re-read → the caller's in-memory copy	the funnel gates on the fresh row
add || ns.code == retryable to endsPolling	the poll survives a transient refusal

Every mutation ran with a control test in the same run that had to stay green, so a mutation that simply broke the suite cannot be read as a red-first proof; and each mutated file was restored by hash against a pre-mutation backup, not by git checkout.

⚠️ That mutation run also caught one of these tests being vacuous, which is the reason it was worth running rather than asserting. Removing AND m.bodyMetadataOversized = 0 from the diagnostics' pending predicate left the bucket-partition test GREEN: its single fixture row had emptyFetchCount = 2 and failed emptyFetchCount = 0 either way, so the mutation was invisible to it. It now seeds two quarantined rows — one with strikes, one without — because the four buckets subtract the quarantine from two different siblings and one fixture cannot exercise both.

Six further hand-copied admission predicates in the test suite now run the production Active/BackfillBodyQueue.admissionSQL instead. Three of them carry never-drop claims — "a row that satisfies this is one the body fetch WILL pick up" — and had silently stopped being that predicate the moment production gained AND bodyMetadataOversized = 0.

Tests drive production symbols rather than replicas: the relaunch tests call the real repopulateFromDatabase(), Smart Reindex calls the real SyncEngine.resetCrawlState(), the success-clears test calls the real BodyFetchProcessor.flushBatch. The query-plan gate asserts all five equality columns in the seek with no temp B-tree, against the deferred index, with a negative control.

What the audit rounds changed

Ten rounds of four-angle review (architecture, correctness, robustness-security, test coverage) ran
against this branch, each angle on a fresh-context specialist in its own detached reviewer worktree.
The final round carried architecture, correctness and robustness-security forward as clean on a
machine-checked proof that the intervening source diff was comment-only (every added and removed
line in all six touched source files matches ^\s*(///|//)), and re-ran test coverage, whose
substance had changed. The code changes they produced, and the claims they retracted:

• The clear-side SQL is now one symbol too. The mark side already funnelled through
  BodyFetchProcessor.markBodyMetadataOversized; the clear was hand-copied into both queues'
  clearOversizedDurably. It is now BodyFetchProcessor.clearBodyMetadataOversized, and both
  queues call it. The comment that justified the duplication claimed a shared helper was
  impossible — that was false (Mutex<Task<Void, Never>?> is Sendable and withLock is
  synchronous), and it is replaced by the two real reasons the write chains stay per-queue:
  per-instance isolation, because suites construct their own ActiveBodyQueue() /
  BackfillBodyQueue() against temporary pools, and different priority tiers (syncPool vs
  backgroundPool).
• "All four marks share this one serialized chain" was too strong. BackfillBodyQueue's
  handlePayloadTooLarge marks through Backfill's chain, not Active's. The invariant that
  actually holds — and the one the ordering depends on — is that every mark shares a serialized
  chain with the clear issued on that same chain, and the UIDVALIDITY reset clears on both.
• The "one remaining bodyComplete = 1 writer" absolute now carries its negative case.
  Migrations v31_addHasBodyInFTS and v57_repairOptimisticSentBodyComplete also write that
  column; they are excluded because they run before v88 and are frozen, not because they do not
  exist.
• The release bound is first-party, not upstream. IMAPFetchMapping.responseBufferLimit
  (4 MiB) is ours — upstream PR #179 made it a constructor parameter — so raising it needs no
  upstream work. Several comments had described it as an upstream constant.
• The refusal strings are a DEBUG surface. MessageCardView renders viewModel.error only
  under DebugModeManager.isLoggingEnabled(); a release user sees the generic
  "Unable to load message. Pull to retry." The refusal texts are documented as such rather than as
  user-facing copy, and ProviderError.networkError's "Network error: " prefix is documented as
  inherited and deliberately not unwrapped.
• The two body queues no longer carry a copy of the durable-write chain.
  BodyFetchProcessor.DurableWriteChain replaces 63 byte-identical lines on each queue,
  which had sat under an "edit both copies or neither" comment over an ordering the file
  itself labels a correctness requirement. Each queue still holds its own INSTANCE — that
  is load-bearing for test isolation and for the two write-priority tiers — but no longer
  its own copy, and the database pool became a call parameter, which turns the
  eager-resolution rule from a comment into a signature.
• The ~90-line accepted-limitations enumeration was also duplicated verbatim on both
  queues under the same instruction. It lives once now, on the single writer both call.
• BodyFetchRefusal.addressInFlight is gone. It carried a byte-identical copy of
  ProviderError.addressPendingMove's user-facing sentence, while fetchAttachment — in
  the same file — already threw the typed case that ComposeView matches by TYPE. The
  funnel throws it too now. Behaviour is unchanged at both fetchBody call sites, and
  endsPolling's exclusion of the mid-move refusal becomes structural rather than a
  listed omission.
• Four enumerating sentences were replaced by the properties they were caching. Each
  had been correct when written and had gone stale as the branch grew: a census count of
  "three" that returned four, "four surfaces" that omitted two ftsIndexed consumers,
  "five consumers, three of them code" that omitted the authoritative funnel, and a
  self-heal comment still promising a next dispatch that the quarantine had removed.
• A status claim in the source contradicted the register this branch adds. Three sites headed
  the accepted-limitations enumeration "owner-blessed" while IOS-BODY-006 says the opposite —
  filed open, NOT accepted. That gap is the whole failure mode the register exists to prevent:
  a reader greps the source, reads "owner-blessed", and stops asking the question. The headers now
  name the register and state that exactly two items carry an owner decision (let "Sync
  Complete" fire; fail fast when an affected message is opened — both 2026-09-01), and that the
  other five do not. The register's own blanket "none has the owner's blessing yet" now exempts
  the two that do.
• Three records a diff cannot show were added to the migration banner. The EXPLAIN QUERY PLAN
  figures now state the statistics regime they were measured under, which IOS-PERF-012 requires
  and which decides whether the numbers mean anything; an earlier revision on this branch built
  the deferred index inside v88 before it moved to deferredIndexes, so a dev database that ran
  the old body already carries the index and converges with a fresh install only because
  createDeferredIndexes is IF NOT EXISTS; and the migration number collides with the sibling
  branch, which breaks no migration but does break this file's own equal-counts self-check after a
  merge.
• The funnel's typed refusal had no producer-side test. The classification test constructs the
  error by hand and the predicate test never runs the funnel, so reverting the throw to the
  pre-change wrapped NSError left the entire suite green while the refusal started classifying as
  a connection error — a wasted retry and a "check your connection" message for something that has
  nothing to do with the network. Both halves green, contract dead. A mutation reproducing exactly
  that revert now turns the new test red while its control stays green in the same run.
• Two conjuncts were satisfied vacuously, in the two requests this branch introduces, and both
  are now pinned. Every fixture in the partition suite seeded one account, so accountId could be
  deleted from either request with the whole suite green; a second account is now seeded with one
  pending-shaped and one settled-shaped row, and two mutations drop the conjunct from each request
  in turn. The second was headerComplete, which pendingBodyRequest carries and
  bodySettledRequest deliberately omits — every fixture that measured either request set it by
  construction, so both of the opposite edits left all 9,402 tests green. That one was proved
  against the full suite rather than a targeted subset, because the claim under test was not
  "does my new test catch it" but "does anything else": each mutation reddened exactly one test of
  9,403, the new one. The rows that distinguish it are reachable — NSEDataBridge stages headers
  with headerComplete = false and flips the flag in a separate statement, so an extension wake
  terminated in between leaves one behind. Dropping the term from pending re-creates the
  never-clearing banner and pinned wake lock this branch exists to remove; adding it to settled
  parks "N / M indexed" permanently below its denominator.
• Two funnel tests explained their own safety with the wrong mechanism. They said no provider is
  registered, so the call cannot reach the wire — backwards, since the quarantine gate sits before
  the provider block and an absent provider is what causes connectAccount to be called. What
  actually keeps them offline is that the fixture's account has no imapHost, so
  createIMAPProvider throws before any provider is constructed, registered in the process-wide
  singleton, or asked to connect. The tests were safe for an unstated reason that one plausible
  edit — adding host and port to that fixture — would have removed silently, turning them into live
  connection attempts from the unit suite. Both comments now name the real guard and both fixtures
  assert it.
• The funnel's outcome→refusal-class mapping was reachable but unpinned, and it is the headline
  behaviour. fetchBody's tail turns a processor outcome into a refusal — .payloadTooLarge to a
  terminal class, .retry to a retryable one — and both loadBody and the 2s poll branch on exactly
  that distinction. Both halves were tested and the join was not: the classifier test constructs
  every refusal by hand, every MessageDetailViewModel test injects _fetchBodyOverride, and all
  six other test call sites of fetchBody exit before the tail. Swapping the two arms left all
  9,403 tests green while restoring the original defect in full — an oversized body would classify
  as retryable, the poll would start, and the row would be re-fetched every two seconds forever. The
  poll's own quarantine gate cannot save it in the eviction case, where the row reads
  bodyComplete = 1. A new test now drives the funnel to completion against a registered mock
  provider; under the swap both of its legs go red. It asserts the classification, never the
  integer code, since pinning -1/-2 would be a mechanism-pinning test.
• Two residual properties are registered rather than mechanised. See below.

Accepted residuals — IOS-BODY-006, filed open

Registered in the known-issues register (post-freeze amendment channel) and documented at the
mechanisms themselves, not only in the register:

1. The flag is a one-strike latch on a non-deterministic signal. One overflow on one wire
   attempt quarantines the row until something positively releases it; there is no strike counter
   and no automatic expiry. Because the parser bound is on unread aggregate bytes measured after
   the decode loop stops, a message that overflows on a lossy link can parse fine on WiFi — so a
   single unlucky attempt can quarantine a message this build could have fetched. The recovery is
   one ordinary user gesture (pull-to-refresh on the open message, or Smart Reindex), which is why
   this is filed as a residual rather than built around.
2. Two writers reach the flag outside the serialized chain. SyncEngine.resetCrawlState
   (Smart Reindex) writes through AppDatabase.backgroundPool, and markOversizedDurably's
   non-queue callers are ordered against the reset's clear only by that same chain. A mark
   dispatched just before a Smart Reindex can therefore commit after its clear, leaving a row
   quarantined that the gesture was invoked to release. It fails closed — the row is retryable
   by the same gesture, repeated — per THE MANTRA, and the window is one dispatch wide.

Both are filed with class open, not accepted: they need the owner's decision, and this PR does
not assert one.

The full decision being asked for. Seven limitations are enumerated on
BodyFetchProcessor.markBodyMetadataOversized. Two already carry an owner decision, both dated
2026-09-01 and both reversals of an earlier stance in this branch: let "Sync Complete" fire on an
account that still holds an unfetchable body, and fail fast when such a message is opened rather
than spending a doomed wire attempt. The other five do not, and are what this PR puts in front
of you:

1. the body is unsearchable by content until IMAPFetchMapping.responseBufferLimit is raised
   (the header stays FTS-indexed, so the message is still findable by subject and sender);
2. the FTS self-heal deliberately does not carry the flag, because it re-indexes headers and
   this row's header is healthy;
3. affected rows show no snippet preview in the inbox;
4. expanding a collapsed thread bubble for a flagged message yields nothing — no body, no error, no
   wire attempt — with recovery by opening it as the focused message;
5. the one-strike latch itself (item 1 above), which is the one I would most like a decision on.

Until you rule, the source says open and not accepted in every place it mentions them, and an
earlier revision of this branch that called the set "owner-blessed" was corrected by the audit.

Merge order

⚠️ This branch and #103 (draft) both register a migration named v88 — here v88_addBodyMetadataOversized, there v88_addBodyIndexingFailureReason. They are merge-exclusive by construction, not conflicting in substance. This one merges first; #103 renumbers to v89 when it resumes. Verified: #103 does not yet carry that prerequisite in its body or comments, so it is being noted there separately.

One unrelated commit rides along

Wait for the fake IMAP transport to settle before asserting no live session fixes a pre-existing race in the test harness, not in the app. assertIMAPTeardown read FakeIMAPServer.liveSessionCount() the instant provider.disconnect() returned, but disconnect returns when the client closes its socket while the server drops the fd on its own thread when it observes that close. Instrumented on a failing run: the count read 1, healed to 0 after 50ms, and abandonedSessionCount() — the monotonic oracle that never heals — was 0, so nothing had actually been abandoned. The assertion now waits for the transport to settle, bounded at 2s, and still fails on a session genuinely left live.