fix(chunkers): fall back to 'character' tokenizer when configured one is unavailable#1804

Open

qianjindexiaozu wants to merge 3 commits into

MemTensor:dev-20260604-v2.0.19from

qianjindexiaozu:fix/sentence-chunker-tokenizer-fallback

Commits on May 25, 2026

fix(chunkers): fall back to 'character' tokenizer when configured one is unavailable

qianjindexiaozu
and
claude
committed

Commits on May 27, 2026

Merge branch 'main' into fix/sentence-chunker-tokenizer-fallback
qianjindexiaozu
authored
fix(chunkers): harden 'character' fallback against chonkie API changes

qianjindexiaozu
and
claude
committed