[simurg79]

Fixes #23

What is the problem?

A failed or stuck Tree-sitter parse can disrupt indexing and editor responsiveness.

How does this PR solve the problem?

Indexing parses in a terminable per-workspace worker thread. Crashed or timed-out parsers are never reused. Indexing fail-stops to Error with one localized message and restarts only on explicit retry. A per-workspace write gate prevents stale attempts from writing to the vector store or cache. Tree cleanup (tree.delete()) is adopted from upstream draft PR RooCodeInc#10390.

Human review: the user approved the spec, design, and implementation. Design docs are maintained privately.

How did you test the PR?

Build status: PASS. Verified commands: root pnpm build (5 successful, cached), pnpm knip (exit 0), changed-file prettier --check (45 files, all matched); translations check (all complete). From src: pnpm bundle, pnpm check-types, pnpm lint (all exit 0). Emitted worker/WASM checks passed: 38 valid WASM headers, worker smoke test returned one block without fallback; asset tests: 37 passed.

Verified: npx vitest run services/code-index services/tree-sitter: 801 passed, 0 failed, 12 skipped. The 12 skips predate this change. Reported base revision: 762 passed (not rerun here).

Known limits

Not verified: packaged VSIX on remote/WSL hosts; real termination of a non-finishing WASM parse; live editor responsiveness. The 10 s / 2 s limits are provisional. Only the OpenAI embedder honours cancellation. Cancelling mid-scan before the completion phase saves no hashes. The Stopping state is no longer set on stop.

Pre-PR Code Review

• Reviewer: Code Reviewer mode (automated)
• Result: ✅ Passed with non-blocking suggestions
• Summary: Round 2 re-review passed after a fix loop (write-gate cleanup, single stop deadline, single recovery path, dead API removal). Remaining 🟢: uncertainFiles() is test-only; defensive throw in waitForPriorWrites.

Agent notes

Validation: tester and builder child tasks; review outcome and human approvals supplied by the workflow handoff. Base: b32c5cf. Scope: 45 files; private docs and protected items excluded.

Round 2 commit: a21bd59; 67 committed files. Verified: git rev-parse HEAD and git ls-remote fork refs/heads/fix/issue-23-isolated-parse-worker both returned that SHA. Commit and whole-PR scope checks passed; protected paths and the stray root file were excluded. Existing round-1 sections and deferred findings are retained as history; Round 2 below supersedes resolved limitations. Runtime comparison against the base was not measured.

Deferred Items

Reviewed commit: 8c38f1c.

Findings 1 and 2 were fixed in PR #34 and are not deferred.

• Finding 🟡3: [BUG] Keep completed indexing progress when a scan is cancelled #35 - preserve completed scan progress on cancellation.
• Finding 🟡4: [BUG] Make indexing stop status reflect what actually stopped #36 - indexing stop status and partial-progress messages.
• Finding 🟡5: [BUG] Cancel abandoned embedding requests for all indexing providers #37 - cancellation for all embedding providers.
• Finding 🟡6: [ENHANCEMENT] Validate packaged parser shutdown on supported hosts #38 - packaged worker validation on Windows and remote/WSL hosts.
• Findings 🟢7, 🟢8, 🟢11: [ENHANCEMENT] Remove misleading indexing leftovers and comments #39 - redundant code and misplaced/stale comments.
• Finding 🟢9: [BUG] Translate parser failure reasons shown to users #40 - localized parser failure reasons.
• Finding 🟢10: [ENHANCEMENT] Keep fallback decisions with each parser result #41 - per-request fallback result.
• Finding 🟢12: [BUG] Preserve parser failure state when retry setup fails #42 - preserve parser-failure guard if retry setup fails.
• Finding 🟢13: [BUG] Report idle parser worker crashes without waiting for another file #43 - immediate idle-worker failure notification.

Round-2 deferred finding

Reviewed commit: a21bd59.

• Finding green N1: [ENHANCEMENT] Test indexing recovery through its grace deadline #49 - test indexing recovery through its grace deadline; remove test-only tracking unless a concrete production use justifies it.

Review-addresser round-2 deferred finding

Reviewed commit: 78b3a10; checked against head 9922e83.

• Finding yellow 1: [BUG] A stalled index clear can leave retry waiting indefinitely #50 - bound the wait for a stalled index clear so retry does not remain pending; the database client timeout is not yet verified.

Round 2

Summary

• Per-file progress survives stop and retry. A new Stopping state shows partial/stopped messages; cancelling while idle does nothing.
• Generation-scoped point IDs use a durable counter and fenced deletes. Late writes or deletes from old attempts cannot overwrite or remove newer results. Upgrading forces a one-time re-index.
• Completion is verified by exact counts. Uncertain files receive a 30-second grace period and a re-check: the user-accepted bounded exception for saves that land late.
• All 8 embedders forward cancellation.
• Real WASM-loop fault injection is covered by tests. The 39-byte fault-loop.wasm ships in the distribution.
• The CI matrix now uses fail-fast: false.
• The gauge test expectation was wrong; the constant 266789 is correct.

Clear-index and retry service recreation are now serialized, so a retry cannot write while the old collection is still being deleted. The regression test now waits for an explicit signal and flushes microtasks instead of relying on a 20 ms sleep.

Follow-up issues

#35, #36 and #37 are addressed. #38 is partially addressed: Windows is done; Remote-WSL is pending and will be run by the user. This updates the historical Deferred Items list above rather than removing it.

Test results

Verified by actual commands in the submission gate:

• pnpm test, with TURBO_FORCE=true: 7,976 passed, 0 failed, 53 skipped; 8 successful tasks, 0 cached. All skips existed before this change; none are in code-index. The skip-history check found no added skip markers; the base suite was not rerun.
• From the backend workspace, npx vitest run services/code-index: 762 passed, no skips.
• Real-Qdrant T23: 4/4 passed on Qdrant 1.16.3, verified by the server version endpoint. This suite is opt-in and documented as a required manual step in the integration instructions. The client/server compatibility warning was retained.
• Build, lint, types, knip, prettier and translations: PASS. Actual pnpm build, pnpm lint and pnpm check-types each passed without turbo cache; pnpm knip and node scripts/find-missing-translations.js exited 0. Changed-file pnpm exec prettier --check passed for 71 files after formatting the acceptance log.

Packaged-host acceptance

• Local Windows: 7/8 PASS, 0 FAIL. Stop responsiveness was NOT RUN because stop completes in about 3 ms, too quickly to place an editor action inside that phase.
• Remote-WSL: pending, to be run by the user. Overall cross-host acceptance remains incomplete.
• Evidence: test plan and acceptance log.

User-waived suites

The CLI integration suite and editor e2e suite are waived for this PR by user decision. Both need an OpenRouter key unavailable locally and in fork CI.

Agent notes

Review-fix commit: 78b3a10. Test-hardening commit: 9922e83; only manager.spec.ts changed (+14/-1).

Updated test totals are supplied by the workflow handoff: pnpm test 7,976 passed / 0 failed / 53 skipped; code-index 762. Tester task: 01a19ac1-98e1-74d1-938d-db14b8e30863; the tester also reported that removing the production wait makes the regression test fail. Tests were not rerun by this PR-update task. Review on the review-fix commit: PASS WITH NOTES (01a19ab9-e18d-7748-b3ae-1b5177f5849e); build: PASS (01a19abb-4a2c-722f-b9c0-206f844e9be9).

verified: git rev-parse HEAD and git ls-remote fork refs/heads/fix/issue-23-isolated-parse-worker both returned 9922e83. Scope command output: "PUSH_MATCH; COMMIT_SCOPE_PASS: exactly one authorized test file". Expected untracked entries were excluded.

verified: changed-file pnpm exec eslint returned "ESLINT_PASS"; pnpm exec prettier --check returned "All matched files use Prettier code style!". Normal commit lint and push type-check hooks each reported "10 successful, 10 total" (9 cached). Existing content is retained by explicit request, overriding the general description-length limit.

[Copilot]

🟡 Changes recommended

Unresolved retry and write-gating races can leave indexing workers running untracked and allow index writes to overlap collection deletion.

5 open issues

• Clear Index and retry can mutate the collection concurrently · New
• Concurrent recreation can leak indexing workers and watchers · New
• Watcher batch failures leave the UI stuck in Indexing · New
• Path-and-line IDs cause same-line chunks to overwrite each other · New
• Deleted-file cleanup misses paths with incomplete replacement vectors · New

What changed in this PR

This PR isolates code-index parsing in per-workspace worker threads to contain parser crashes and hangs, addressing #23 without blocking the shared extension host.

Changes:

• Adds worker-based parsing with timeouts, tree cleanup, and explicit recovery after fatal failures.
• Adds cancellation and write coordination across scans, file watching, vector storage, and cache updates.
• Adds lifecycle regression tests, worker build support, and translated failure messages.

File	Description
src/​services/​tree-sitter/​languageParser.ts	Supports worker-local parser runtime assets.
src/​services/​code-index/​vector-store/​qdrant-client.ts	Propagates deletion failures.
src/​services/​code-index/​vector-store/​__tests__/​qdrant-client.spec.ts	Tests deletion error propagation.
src/​services/​code-index/​service-factory.ts	Shares workspace parser and write gate.
src/​services/​code-index/​processors/​scanner.ts	Gates writes and handles cancellation and fatal parsing.
src/​services/​code-index/​processors/​parser.ts	Adds worker delegation, tree cleanup, and chunking optimizations.
src/​services/​code-index/​processors/​file-watcher.ts	Integrates worker parsing and guarded replacement writes.
src/​services/​code-index/​processors/​__tests__/​scanner.spec.ts	Tests cancellation, reconciliation, and batch completion.
src/​services/​code-index/​processors/​__tests__/​parser.spec.ts	Tests tree cleanup and chunking performance.
src/​services/​code-index/​processors/​__tests__/​file-watcher.spec.ts	Tests watcher shutdown and replacement lifecycle.
src/​services/​code-index/​parser-errors.ts	Defines fatal parser and cancellation errors.
src/​services/​code-index/​parse-worker.ts	Adds the parsing worker entry point.
src/​services/​code-index/​parse-worker-client.ts	Queues parses and terminates failed or timed-out workers.
src/​services/​code-index/​orchestrator.ts	Coordinates fatal shutdown and guarded indexing writes.
src/​services/​code-index/​manager.ts	Tracks parser failures and recreates stopped services.
src/​services/​code-index/​interfaces/​file-processor.ts	Adds cancellation and fatal-error event contracts.
src/​services/​code-index/​interfaces/​embedder.ts	Adds optional embedding cancellation.
src/​services/​code-index/​index-write-gate.ts	Tracks outstanding writes and uncertain file outcomes.
src/​services/​code-index/​embedders/​openai.ts	Forwards cancellation to embedding requests.
src/​services/​code-index/​embedders/​__tests__/​openai.spec.ts	Tests cancellation without retries.
src/​services/​code-index/​constants/​index.ts	Adds parser and stop timeout constants.
src/​services/​code-index/​__tests__/​parse-worker-client.spec.ts	Tests worker ordering, failures, cancellation, and isolation.
src/​services/​code-index/​__tests__/​orchestrator.spec.ts	Tests fatal shutdown and late-callback suppression.
src/​services/​code-index/​__tests__/​manager.spec.ts	Tests failure guards and explicit restart.
src/​services/​code-index/​__tests__/​index-write-gate.spec.ts	Tests cancellation barriers and uncertain writes.
src/​i18n/​locales/​zh-TW/​embeddings.json	Adds Traditional Chinese parser-failure guidance.
src/​i18n/​locales/​zh-CN/​embeddings.json	Adds Simplified Chinese parser-failure guidance.
src/​i18n/​locales/​vi/​embeddings.json	Adds Vietnamese parser-failure guidance.
src/​i18n/​locales/​tr/​embeddings.json	Adds Turkish parser-failure guidance.
src/​i18n/​locales/​ru/​embeddings.json	Adds Russian parser-failure guidance.
src/​i18n/​locales/​pt-BR/​embeddings.json	Adds Brazilian Portuguese parser-failure guidance.
src/​i18n/​locales/​pl/​embeddings.json	Adds Polish parser-failure guidance.
src/​i18n/​locales/​nl/​embeddings.json	Adds Dutch parser-failure guidance.
src/​i18n/​locales/​ko/​embeddings.json	Adds Korean parser-failure guidance.
src/​i18n/​locales/​ja/​embeddings.json	Adds Japanese parser-failure guidance.
src/​i18n/​locales/​it/​embeddings.json	Adds Italian parser-failure guidance.
src/​i18n/​locales/​id/​embeddings.json	Adds Indonesian parser-failure guidance.
src/​i18n/​locales/​hi/​embeddings.json	Adds Hindi parser-failure guidance.
src/​i18n/​locales/​fr/​embeddings.json	Adds French parser-failure guidance.
src/​i18n/​locales/​es/​embeddings.json	Adds Spanish parser-failure guidance.
src/​i18n/​locales/​en/​embeddings.json	Adds English parser-failure guidance.
src/​i18n/​locales/​de/​embeddings.json	Adds German parser-failure guidance.
src/​i18n/​locales/​ca/​embeddings.json	Adds Catalan parser-failure guidance.
src/​esbuild.mjs	Builds and watches the parser worker bundle.
knip.json	Registers the parser worker entry point.

_{Review effort: Balanced}