Repository navigation
[Bug]: tokenizer_config max_length caps several models at 128 tokens #689
Description
What happened?
load_tokenizer takes min(model_max_length, max_length) when both keys exist in tokenizer_config.json (fastembed/common/preprocessor_utils.py:47). Several Qdrant-hosted ONNX repos carry a "max_length": 128 that the upstream model repo does not, so input is truncated at 128 tokens with no warning.
For two models this contradicts the shipped description:
| model | description says | actual limit |
|---|---|---|
thenlper/gte-large |
512 input tokens truncation | 128 |
sentence-transformers/all-MiniLM-L6-v2 |
256 input tokens truncation | 128 |
Two more repos carry the same key where the upstream repo has none: Qdrant/Splade_PP_en_v1 (upstream sentence_bert_config.json is max_seq_length: 512) and Qdrant/all_miniLM_L6_v2_with_attentions (BM42). Their descriptions carry no token count.
qdrant/paraphrase-multilingual-MiniLM-L12-v2-onnx-Q also caps at 128, and there it matches the upstream. Whether the four above are deliberate is a question for maintainers.
What is the expected behaviour?
The effective limit matches the description, and a limit below the upstream default is a deliberate choice.
Related: there is no way to change the limit through the API. enable_truncation is called once from the config file, and no max_length kwarg reaches the tokenizer, so the only override today is specific_model_path with an edited tokenizer_config.json.
A minimal reproducible example
from fastembed import TextEmbedding, SparseTextEmbedding
print(TextEmbedding("sentence-transformers/all-MiniLM-L6-v2").model.tokenizer.truncation["max_length"]) # 128, description says 256
m = SparseTextEmbedding("prithivida/Splade_PP_en_v1")
doc = "lorem ipsum " * 200 + "zebra"
print(m.model.tokenizer.token_to_id("zebra") in list(m.embed([doc]))[0].indices) # FalseThe second check shows the effect: a term past position 128 is absent from the sparse vector.
What Python version are you on? e.g. python --version
Python 3.11, uv
FastEmbed version
0.8.0
What os are you seeing the problem on?
MacOS
Activity
Dylancouzon commentedon Aug 24, 2026
Related open PRs in this code path. None of them covers the repo configs above.
enable_truncationcall, for thesys.maxsizesentinel case. The review there asks for the real limit to be read fromconfig.json(max_position_embeddings) instead of clamped, which would givethenlper/gte-largeits 512.max_length: 0in the same function ([Bug]: models with max_length=0 in tokenizer config break #685).sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2to 128, matching that model's upstreammax_seq_length: 128. That fix is right for that model. The two models in the table here are the reverse case: their descriptions match upstream and the config sits below it.denis-samatov commentedon Aug 31, 2026
Thanks for the detailed reproducer and the cross-links. I traced the same loader path and see three distinct choices here: correct the affected model metadata, expose an explicit
max_lengthoverride while preserving the current default, or change the automatic precedence usingconfig.json:max_position_embeddings. The last option looks most likely to change existing behavior unexpectedly. Would maintainers prefer a focused PR for the metadata path or for a validated public override? I’d be happy to implement the chosen scope with regression tests covering the affected models and the neighboring#500/#687cases.joein commentedon Sep 27, 2026
@Dylancouzon this has to be fixed now
Dylancouzon commentedon Sep 30, 2026
Confirming fixed, thanks!