Chunk prefetcher re-downloads every subset snapshot: getNextChunkUrl doesn't exclude subset-bearing requests

[jwaltz]

Summary

createFetchWithChunkBuffer's getNextChunkUrl (packages/typescript-client src/fetch.ts) decides whether to prefetch a "next log chunk" from three signals: the response carries electric-handle + electric-offset, has no electric-up-to-date header, and the request isn't live=true. It never checks for subset__* params — although createFetchWithResponseHeadersCheck in the same file explicitly encodes that "Snapshot responses (subset params) return a JSON object and do not include Electric chunk headers" (through some proxies they do echo handle/offset headers, which arms the prefetcher).

A subset snapshot response (requestSnapshot in changes_only mode, e.g. driven by @tanstack/electric-db-collection's on-demand loadSubset) therefore triggers a speculative prefetch of the same subset — the rebuilt URL is the snapshot URL with handle/offset set from the response headers, which for an established stream is byte-identical to the request that just completed (searchParams.sort() in getNextChunkUrl). The prefetched body (up to the full subset, again) lands in a PrefetchQueue that is never consumed for snapshot flows, and is discarded.

Observed impact

TanStack Start app (TanStack DB on-demand collections over Electric, @electric-sql/client@1.5.25, @tanstack/electric-db-collection@0.3.15): on a cold route load issuing ~15 subset snapshots, every subset was fetched twice — e.g. a 1.25 MB subset downloaded as 1,251 KB + 1,253 KB back-to-back at identical offset+handle (no cache buster, so not the 409 retry path). ~1.3–2.1 MB of duplicate transfer per cold load depending on how much of the speculative body completes before the queue aborts.

Diagnosis evidence: one requestSnapshot call → one snapshot application, two HTTP fetches; the second fetch's call stack runs through new PrefetchQueue → #prefetch from createFetchWithChunkBuffer; no must-refetch events; both duplicate shapes reproduce (cold stream: offset=now snapshot → prefetch at the response's log offset re-delivers the subset; established stream: identical-URL prefetch).

Direct-server confirmation (no proxy in the chain)

A subset snapshot request straight against vanilla Electric (ElectricSQL/1.4.13, GET /v1/shape?table=...&log=changes_only&offset=now&subset__where=... — and equally a where-less subset__order_by+subset__limit variant) returns:

HTTP/1.1 200 OK
electric-snapshot: true
electric-handle: 22092372-1785199320854135
electric-offset: 840041648_144
(no electric-up-to-date header)

That is exactly the header combination that arms getNextChunkUrl (handle + offset present, up-to-date absent), so the prefetch fires on every subset snapshot by construction. Note the server already labels these responses with a dedicated electric-snapshot: true header, which the prefetcher currently ignores.

Suggested fix

Skip prefetch for subset-bearing requests in getNextChunkUrl, mirroring the isSnapshotRequest check createFetchWithResponseHeadersCheck already uses:

// in getNextChunkUrl, alongside the live=true exclusion
if ([SUBSET_PARAM_WHERE, SUBSET_PARAM_WHERE_PARAMS, SUBSET_PARAM_LIMIT,
     SUBSET_PARAM_OFFSET, SUBSET_PARAM_ORDER_BY].some((p) => nextUrl.searchParams.has(p))) {
  return
}

Alternatively (server-authoritative, no URL sniffing): skip prefetch when the response carries electric-snapshot: true, since the server already marks snapshot responses explicitly.

Workaround we shipped

A fetchClient wrapper that adds electric-up-to-date to subset snapshot responses (its only consumer in the client is the prefetch gate), which suppresses the prefetcher on subsets entirely. Happy to PR the fix if useful.