Support inference-free SPLADE models (asymmetric doc-side encoding, tokenizer-only queries) #648

Summary

FastEmbed currently ships only one SPLADE model — prithivida/Splade_PP_en_v1 — which is a symmetric SPLADE variant: the ONNX encoder runs on both documents and queries. There is no support for inference-free SPLADE (IF-SPLADE), where document vectors are produced by neural expansion at index time, but queries are processed by plain tokenization into raw BERT token IDs — no model, no GPU, no neural inference at query time.

This is a meaningful gap because inference-free SPLADE gives essentially the same retrieval quality as full SPLADE while making queries dramatically cheaper, which is exactly the regime FastEmbed targets (lightweight, ONNX, CPU-friendly serving).

Motivation

Benchmarks of inference-free vs. full SPLADE on BEIR scifact (writeup):

• ~13× faster queries — median query time dropped from ~57ms to ~4.3ms.
• <1.3% relative quality loss — only 0.0093 NDCG@10 lower than full SPLADE.
• Both substantially outperform BM25 at comparable query speed.
• The cost shifts entirely to index time (neural expansion of documents), which for static / slowly-changing corpora is an acceptable one-time investment.

Because queries skip the transformer entirely, IF-SPLADE removes the 50–100ms per-query encoder latency that currently makes Splade_PP_en_v1 expensive to serve, while keeping learned sparse retrieval quality.

What's missing in FastEmbed

1. No inference-free SPLADE models are registered. fastembed/sparse/splade_pp.py only lists prithivida/Splade_PP_en_v1 (and a duplicate misspelled prithvida/... alias).
2. No inference-free query path. SpladePP runs the ONNX model for both embed() and query_embed(). There is no code path that tokenizes a query and emits raw BERT token-id sparse vectors (with IDF/learned token weights) without invoking the encoder.

Proposal

• Add support for asymmetric / inference-free SPLADE models, e.g.:
  • opensearch-neural-sparse-encoding-doc-v3-gte (asymmetric)
• Implement an inference-free query path: tokenize the query into BERT token IDs and assign token weights without running the ONNX encoder, so query_embed() requires no model inference, while embed() (documents) continues to run neural expansion.

Notes:

Unfortuantely, naver/splade-v3-doc (asymmetric, inference-free optimized) cannot be integrated in Fastembed due to license issues.

References

• Blog: https://kshivendu.dev/blog/if-splade
• naver/splade-v3-doc, naver/splade-v3 on Hugging Face