[Bug]: Bm42 and miniCOIL fail in parallel embedding with lazy_load=True #749

What happened?

With lazy_load=True, embed(..., parallel=2) crashes for Qdrant/bm42-all-minilm-l6-v2-attentions and Qdrant/minicoil-v1. The same call works with lazy_load=False, and SPLADE (prithivida/Splade_PP_en_v1) works either way. The lazy_load docstring on both classes recommends this exact combination: "Should be set to True when using multiple-gpu and parallel encoding."

In the parallel branch of OnnxTextModel._embed_documents, the workers run inference and send raw ONNX outputs back, and the parent process runs _post_process_onnx_output on them. The parent never calls load_onnx_model() on that path, so any state that post-processing reads and that only load_onnx_model() builds is missing:

• Bm42 needs self.tokenizer (read by _reconstruct_bpe) and self.invert_vocab.
• miniCOIL needs self.vocab_resolver, self.encoder, and self.sparse_vector_converter, which load_onnx_model() builds from the vocab JSON and the .npy weights in the model directory.

SPLADE's post-processing only reads the ONNX output, which is why it isn't affected.

The existing test_parallel_processing tests don't catch this: the model_cache fixture builds models with the default lazy_load=False, so the parent has already loaded everything.

What is the expected behaviour?

lazy_load=True with parallel returns the same embeddings as lazy_load=False.

A minimal reproducible example

from fastembed import SparseTextEmbedding

if __name__ == "__main__":
    docs = ["Paris is the capital of France.", "hello world"] * 8
    model = SparseTextEmbedding("Qdrant/bm42-all-minilm-l6-v2-attentions", lazy_load=True)
    list(model.embed(docs, batch_size=2, parallel=2))

Use Qdrant/minicoil-v1 for the miniCOIL failure. The __main__ guard is needed because workers start with forkserver or spawn, and len(docs) has to be at least batch_size, otherwise _embed_documents takes the serial path.

What Python version are you on?

3.10.10

FastEmbed version

main at 8fcf0fe (0.8.1)

What os are you seeing the problem on?

macOS 15.7 (arm64)

Relevant stack traces and/or logs

Abridged to fastembed frames, line numbers from 8fcf0fe.

Bm42:

File "fastembed/text/onnx_text_model.py", line 168, in _embed_documents
File "fastembed/sparse/bm42.py", line 250, in _post_process_onnx_output
File "fastembed/sparse/bm42.py", line 193, in _reconstruct_bpe
    continuing_subword_prefix = self.tokenizer.model.continuing_subword_prefix
AttributeError: 'Bm42' object has no attribute 'tokenizer'

miniCOIL:

File "fastembed/sparse/minicoil.py", line 219, in embed
File "fastembed/text/onnx_text_model.py", line 168, in _embed_documents
File "fastembed/sparse/minicoil.py", line 282, in _post_process_onnx_output
    assert self.vocab_resolver is not None
AssertionError

Possible fix

#640 routes tokenizer loading through _load_tokenizer() and _ensure_tokenizer(), and moves Bm42's special_tokens into that hook. For Bm42, building invert_vocab in the same hook and calling _ensure_tokenizer() before post-processing would be enough. miniCOIL also needs its vocab resolver, encoder, and converter in the parent. None of those need the ONNX session, so they could be loaded separately from it.