[BUG] Indexing gives up on a file after one transient network error (batch retries removed in PR #34) #58

Type

Bug — regression from PR #34

Problem / Value

A single transient network error while writing a file to the vector store now fails that file for the rest of the session. Before PR #34, the scanner retried each embed + upsert batch up to 3 times with exponential backoff. The new write path has no retry at all, so a one-off blip leaves the index partial and (through #57) blocks codebase_search for hours.

Context

Observed 2026-11-02 09:18:13 PST during a full scan (renderer.log): [Extension Host] Failed to upsert points: TypeError: fetch failed. Qdrant v1.16.3 runs on localhost in Docker.

• verified: docker logs qdrant has no matching error and no non-2xx response in that window, so the request failed client-side at the connection level and never got a response.
• The scan finished at 09:40:37 as partial → Standby → 28 failed searches from 10:10 to 11:57 ([BUG] One failed file keeps the code index in Standby and blocks codebase_search until a manual restart #57).
• Upstream retried: origin/main scanner.ts#L385-L463 loops while (attempts < MAX_BATCH_RETRIES && !success) with INITIAL_RETRY_DELAY_MS * 2^(attempts - 1) backoff (3 attempts, 500 ms base).
• verified: git log -S MAX_BATCH_RETRIES points at a21bd59 ("fix(code-index): preserve progress and fence cancelled indexing attempts", part of PR [code-index] Isolate parsing so parser failures stop indexing safely #34) as the commit that removed it. git grep finds no retry logic in fork file-completion.ts, scanner.ts, file-watcher.ts, or qdrant-client.ts.
• Fork write path, no retry anywhere: FileIndexWriter.replace → createEmbeddings → upsertPoints (logs at L386 and rethrows) → completion: deletePointsByFilePath + countPointsForFile verification (file-completion.ts#L79). Any throw fails the file: in a scan via orchestrator.ts#L141, in the watcher via file-watcher.ts#L244.
• Embedders still retry their own HTTP calls (for example 429 in openai-compatible.ts), but the Qdrant upsert, delete, and count calls do not.

Reproduction

1. Start a scan or a file-watcher update.
2. Inject one transient failure: make the first client.upsert call in upsertPoints throw new TypeError("fetch failed"), or docker stop qdrant briefly during a write and start it again.
3. Expected: the batch is retried and the file completes. Actual: the file fails immediately and the scan or batch is marked incomplete.

Acceptance criteria

• Transient failures in a file's embed, upsert, delete, or verify steps are retried with bounded exponential backoff (keep 3 attempts / 500 ms base unless there is a reason to change it).
• "Transient" means connection-level errors (fetch failed, ECONNRESET, ECONNREFUSED, ETIMEDOUT, closed socket), HTTP 5xx, and 429. Validation/4xx errors and parser failures are not retried.
• Retries respect the abort signal, the write gate, and generation fencing: a superseded or cancelled attempt never writes after cancellation.
• Each retry and the final failure are logged with the file path and error ([ENHANCEMENT] Log code-index failures and state changes so a stuck index can be diagnosed #60).
• Tests: one transient upsert failure followed by success → file completes; persistent failure → fails after the maximum attempts; abort during backoff → stops promptly with no further writes.

Constraints/Preferences

• Keep PR [code-index] Isolate parsing so parser failures stop indexing safely #34's fencing and progress-preservation guarantees.
• Don't stack another retry loop on top of the embedders' internal retries for the same error class.

Related: PR #34, #57, #60