[Bug]: models with max_length=0 in tokenizer config break #685

What happened?

In load_tokenizer, the tokenizer config is checked to set max_context.

        if "model_max_length" not in tokenizer_config:
            max_context = tokenizer_config["max_length"]
        elif "max_length" not in tokenizer_config:
            max_context = tokenizer_config["model_max_length"]
        else:
            max_context = min(tokenizer_config["model_max_length"], tokenizer_config["max_length"])

However, in the tokenizer spec by Hugging Face, it is possible for either of these values to be 0. This does not impact their functioning in transformers tokenizers. When such a model is loaded in fastembed, it will not truncate at all, and crash on longer inputs.

I think an easy fix is that, instead of checking for existence, the tokenizer is checked for non-zero values.

I also have to admit that it takes a little bit of editing to get to a state where you create a tokenizer with max_length = 0, so it's definitely a corner case.

What is the expected behaviour?

Change the check to reject a key if it is not present or 0.

A minimal reproducible example

This is a bit difficult to reproduce, but happens when a tokenizer has a padding module with a fixed length of 0. This can be done to be able to store the padding token on a tokenizer.json without resorting to an external file.

What Python version are you on? e.g. python --version

Python 3.12, uv

FastEmbed version

v0.8.0

What os are you seeing the problem on?

MacOS

Relevant stack traces and/or logs