[Bug]: Loading a tokenizer resets serialized left truncation to right #774

What happened?

load_tokenizer resets serialized left truncation to the tokenizers default, right truncation, when it reapplies the context limit. Consequently, a saved tokenizer that keeps the end of a long input instead keeps its beginning after loading through FastEmbed. This also affects the tokenizer loaded by OnnxTextModel.

Expected behaviour

Keep the truncation direction saved in tokenizer.json while applying the existing context-limit resolution. Tokenizers without serialized truncation should continue to default to right truncation.

This concerns which end of the input survives, independently of the length-metadata issue #689 and the batch-padding fix #716. It does not require a new max_length API argument, as discussed in #722.

Minimal reproducible example

The following uses a tiny locally created tokenizer and requires no model download or ONNX inference:

import json
from pathlib import Path
from tempfile import TemporaryDirectory
from tokenizers import Tokenizer, models, pre_tokenizers
from fastembed.common.preprocessor_utils import load_tokenizer

with TemporaryDirectory() as directory:
    path = Path(directory)
    words = ["[UNK]", "[PAD]", "one", "two", "three", "four", "five"]
    original = Tokenizer(models.WordLevel(dict(zip(words, range(len(words)))), "[UNK]"))
    original.pre_tokenizer = pre_tokenizers.Whitespace()
    original.enable_truncation(max_length=3, direction="left")
    original.enable_padding(pad_id=1, pad_token="[PAD]")
    original.save(str(path / "tokenizer.json"))
    (path / "tokenizer_config.json").write_text(json.dumps({"model_max_length": 3, "pad_token": "[PAD]"}))
    loaded, _ = load_tokenizer(path)
    print("Saved:", original.encode("one two three four five").tokens)
    print("Loaded:", loaded.encode("one two three four five").tokens)

Expected: both lines show ['three', 'four', 'five'].
Actual on upstream: Saved shows ['three', 'four', 'five']; Loaded shows ['one', 'two', 'three'].

Environment

FastEmbed 0.8.1 from source, upstream 58264f7fe53ad5886d7489f4ce65c50ded239d76; Python 3.13.3 on macOS arm64, tokenizers 0.23.2. Dependencies installed with Poetry/uv.

AI assistance (OpenAI Codex) was used to inspect the implementation, prepare the reproduction and regression coverage, and validate the proposed fix.