fix: prepare sparse post-processing for lazy parallel embedding

[jipeng6036-del]

All Submissions:

• Followed the contributing guidelines.
• Checked open PRs for the same fix.

Summary

With lazy_load=True and parallel sparse embedding, workers return raw ONNX output but the parent post-processes it without loading model-specific assets. BM42 then has no tokenizer or inverted vocabulary, and miniCOIL has no vocabulary resolver or encoder.

Prepare only those post-processing assets in the parent before consuming worker output. The parent still does not create an ONNX session. Both models reuse the same preparation after ordinary model loading.

Fixes #749.

Validation

• uv run --no-project --with-editable . --with pytest --with ruff pytest -q tests/test_sparse_embeddings.py -k 'parallel_processing or lazy_parallel_sparse_post_processing' — 4 passed on Windows/Python 3.13, including real BM42 and miniCOIL model runs.
• uv run --no-project --with-editable . --with 'ruff==0.3.4' ruff check fastembed/sparse/bm42.py fastembed/sparse/minicoil.py fastembed/text/onnx_text_model.py tests/test_sparse_embeddings.py — passed (repository pre-commit Ruff version).
• Ruff format check and git diff --check — passed.

The full test suite was not run locally.

AI assistance

Codex prepared the change and ran the checks above. The PR author is responsible for the contribution and review replies.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: db73bce5-5cb2-4f34-ac11-79829c058e4f

📥 Commits

Reviewing files that changed from the base of the PR and between e0447a2 and 80fadbb.

📒 Files selected for processing (4)

• fastembed/sparse/bm42.py
• fastembed/sparse/minicoil.py
• fastembed/text/onnx_text_model.py
• tests/test_sparse_embeddings.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

---

📝 Walkthrough

Walkthrough

OnnxTextModel now calls a post-processing preparation hook before parallel document processing. BM42 and MiniCOIL add model-specific preparation that loads tokenizer-related state under the conditions described in their implementations. A parameterized regression test checks that lazy models remain unloaded during parallel embedding and that their embedding counts, indices, and values match serial results.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 80fad

The changed path prepares sparse post-processing state before parallel work, and the regression test checks lazy loading and serial-parallel output parity. No actionable merge-blocking risk is established.

Security Architecture Review

Security architecture risk: 🟡 Moderate · up to 80fad

The new preparation phase fixes the successful lazy-parallel path, but an interrupted MiniCOIL preparation can leave an instance unable to recover on retry. No new security boundary bypass was identified.

Retained concerns

• Medium · reliability · inferred: MiniCOIL publishes its vocabulary resolver before the complete post-processing state is ready, then uses that resolver alone as the retry and concurrent-call guard. Failure or interruption after publication can strand the instance with missing assets; another caller can proceed with incomplete state.

Security review details

Security Blast Radius

• inferred — The identifiable new exposure is parent-process reading of already selected sparse-model assets. The incomplete-state failure is bounded to a model instance in the examined code; tenant or service-wide exposure cannot be established from the available runtime context.

Trust Boundaries and Controls

• inferred — No new path from document text to an asset filename, credential, or ONNX session-loading operation was identified in preparation. Caller-selected model directories remain the relevant asset trust boundary.

Resilience and Maintainability Implications

• inferred — MiniCOIL’s unsynchronized, partial readiness marker weakens failure containment: retry can skip repair, while a concurrent call can consume a resolver before its vocabulary or companion assets are ready.

Hardening Proposals

• proposed — Construct the complete MiniCOIL post-processing state before publishing readiness, and make first-use preparation safe against concurrent callers and retry after failure.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 9.09% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 4 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: preparing sparse post-processing for lazy parallel embedding.
Description check	✅ Passed	The description directly explains the lazy parallel embedding issue, the BM42 and miniCOIL fix, the implementation approach, and validation results.
Linked Issues check	✅ Passed	Issue [#749] requires lazy parallel embedding to match eager embedding for BM42 and miniCOIL. OnnxTextModel._embed_documents now calls _prepare_post_processing() before parent-side worker-output p…
Out of Scope Changes check	✅ Passed	The changes stay within issue [#749]. The base hook, model-specific preparation, import reordering, and regression test directly support lazy parallel post-processing. No unrelated product behavior or…

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[joein]

Hey @jipeng6036-del

Thanks, but no need to fix it, I have added a fix to #640