fix: preserve ColBERT tokenizer truncation direction

[j45856021-dev]

ColBERT loads its document and query tokenizers through load_tokenizer, then reapplies the marker-adjusted context limit. Calling enable_truncation without direction resets both to right truncation, undoing the direction resolved by #775. This also affects JinaColBERT, LateOn and mLateOn, which share the loader.

Pass the already-resolved direction to both limit updates. Context-limit resolution, marker reservation, query expansion, and default/right truncation are unchanged.

The regressions use real local WordLevel tokenizer files and lazy/local-path initialization, without an ONNX session. For example, with left truncation, a 40-token limit and word0 through word49, ColBERT should retain word13 through word49 between its special tokens; unchanged main instead retains word0 through word36.

Validation

• Before the fix: 16 failures, 16 passing controls across both tokenizers and all four model classes. Only the left-truncation cases fail.
• After the fix: 40 passed, including the 32 new cases and eight existing model-free controls. Tests check exact tokens, IDs, attention masks, marker-adjusted lengths, short inputs, and tokenizer reloads.
• Pre-commit-pinned Ruff 0.3.4 lint/format, the CI mypy command (66 files), and the CI Pyright command pass. Pyright was run with the test virtual environment activated; an initial unactivated invocation's 12 unknown-member diagnostics disappear on both unchanged and patched source when that interpreter is selected.
• Sdist/wheel build passes; the wheel's changed module matches the tested source byte-for-byte.

python -m pytest tests/test_common.py \
  tests/test_late_interaction_embeddings.py::test_get_embedding_size \
  tests/test_late_interaction_embeddings.py::test_colbert_tokenizer_preserves_truncation_direction -q

Environment: macOS arm64, Python 3.13.16, tokenizers 0.23.2, NumPy 2.5.3. Validation is tokenizer-only: no model-weight downloads, pretrained inference, full model suite, or OS/Python matrix. Pre-commit hooks were not installed; their pinned Ruff checks ran directly.

All Submissions

• Followed the contribution guidelines.
• Checked open and closed issues/PRs and upstream branches for the same change. fix: preserve serialized tokenizer truncation direction #775 fixes the shared loader, not these subsequent ColBERT calls.

AI disclosure: investigation, implementation, tests and this PR text were developed autonomously with AI assistance. No human pre-review is claimed.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: f42904be-db06-4884-bf44-9905641cbd87

---

📥 Commits

Reviewing files that changed from the base of the PR and between d076f08 and 6dd1129.

---

📒 Files selected for processing (2)

• fastembed/late_interaction/colbert.py
• tests/test_late_interaction_embeddings.py

---

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

---

---

📝 Walkthrough

Walkthrough

Colbert._load_tokenizer now passes the truncation direction resolved by load_tokenizer to both tokenizers when it reduces the maximum length for reserved marker tokens. New parametrized tests cover document and query tokenization, truncation settings, retained tokens, attention masks, short inputs, and tokenizer reload.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

---

Merge Risk: ⚪ Minimal · up to 6dd11

This fixes ColBERT-family tokenizers losing left truncation when marker tokens reduce the context limit. Right-truncation and default behavior are unchanged. No actionable merge risk remains.

Pre-merge checks | 4 | 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check		The title clearly and concisely describes the main change: preserving the ColBERT tokenizer truncation direction.
Description check		The description directly explains the truncation-direction bug, the affected tokenizer flow, the implementation, regression coverage, and validation results.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

---

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{Comment @coderabbitai help to get the list of available commands.}

[Coding-Professional]

I reviewed the tokenizer change and ran the PR's regression test against this exact commit. Preserving the direction already resolved by load_tokenizer is the right fix when ColBERT reserves marker space; the tests cover document and query tokenizers, left and right truncation, and all four affected model classes.

Local validation: 33 passed (test_colbert_tokenizer_preserves_truncation_direction plus test_get_embedding_size). This was tokenizer-only; I did not run model inference or the full suite.