fix: prevent order-dependent BM25 collision weights

[RAMZI0TO99]

Different BM25 token strings can share an absolute hash ID. kitchens and prostaglandins have signed hashes 358434922 and -358434922, respectively. With k=1.2, b=0.75 and avg_len=256, the same bag kitchens kitchens prostaglandins produces weight 1.6786885245901642 or 1.904311073541843 depending on order, because later entries overwrite earlier weights.

This draft proposes summing separately calculated lexical-token weights at their shared ID, yielding 3.582999598132007 for either ordering. Related: #369. Maintainer input is requested on the collision policy: aggregating counts before the nonlinear BM25 formula is a different approach and yields 1.9936283185840709 for this example. The tests establish approximate order independence and ordinary noncolliding behavior without selecting between those policies.

Existing IDs and hashing are unchanged. This does not eliminate hash aliases; affected stored document vectors need recomputation for consistent corrected weights. No bitwise invariance, Rust-runtime agreement, or retrieval-quality improvement is claimed. The query-index overflow fix is separate and is not included here.

Validation on Windows 11, Python 3.13.5, NumPy 2.3.5, FastEmbed 0.8.1:

• 15 selected tests passed: python -m pytest -q tests/test_bm25_hash_collisions.py.
• Local diff-scoped AST docstring coverage: 11/11 functions (100%, above the reported 80% threshold). CodeRabbit's own result is separate.
• Ruff 0.3.4 lint/format and staged whitespace checks passed.
• Repository mypy and pyright checks passed on Python 3.13.5.

The full model/OS/Python CI matrix was not run locally.

Split from #766 following the maintainer's request for one PR per problem. This branch is based directly on current main and contains only this problem's production change and regression module.

[RAMZI0TO99]

Thanks for reviewing and merging the other fixes. I rechecked this draft against main at db785ee93077fce5da745dca921ff9c40863f18f with the actual FastEmbed package, Python 3.13.5 and NumPy 2.3.5. Before the candidate, nine collision regression cases fail; with it, all 22 selected tests pass (15 collision regression cases and seven existing common tests). The repository-pinned Ruff 0.3.4 checks pass, and local docstring coverage is 11/11 functions.

Which collision policy would you prefer for this PR: summing independently calculated lexical-token weights (the current draft), or aggregating occurrence counts by token ID before applying the BM25 formula? For the example in the description, those give 3.582999598132007 and 1.9936283185840709, respectively.

I'll keep this as a draft pending that decision and coordination with core BM25 behavior. Existing token IDs stay unchanged; affected stored document vectors would need recomputation to use the corrected weights consistently. No retrieval-quality improvement or Rust-runtime agreement has been established by these local checks.