Add splintr as an engine - #4
Open
farhan-syah wants to merge 1 commit into
Open
Conversation
splintr is a Rust tokenizer that loads `tokenizer.json` directly and covers BPE, Unigram, SentencePiece and WordPiece behind one loader, so it needs no derived artifact and is measured on the same file as the reference. The adapter calls `encode_raw`, not `encode`: it returns the backend's content tokens without the post-processor template (BOS/EOS, `[CLS] ... [SEP]`), which is what `add_special_tokens = false` asks the reference for, while still matching special tokens that appear in the text. It is also the serial per-document path -- `encode_batch` and `encode_rayon` are never called, so the cell measures one thread like every other native engine. Phases are measured by subtracting rungs, the way `pipeline` does it, since splintr does not instrument its own stages: whole encode, then the normalizer alone, then the pre-tokenizer over the normalized text, with core encoding as what is left. The pre-tokenizer rung is `for_each_pre_token` rather than `pre_tokenize`, which allocates a `String` per piece and undoes the ByteLevel mapping for its caller -- charging pre-tokenization for work `encode` never does would misplace the time the chart exists to locate. The cache is cleared first, so the merge loop is not credited for chunks an earlier rep resolved, and only the BPE backend reports: Unigram and WordPiece split inside the model with no rung between the normalizer and the ids. Pinned to `=0.19.1` at default features -- what `cargo add splintr` gives you, so the throughput and the package size are the ones a reader would get. 352 of 352 cells verified against the reference: every family x corpus it runs, ids identical.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds splintr as an engine, if you'd like another one in the matrix.
It reads
tokenizer.jsondirectly and handles BPE, Unigram, SentencePiece and WordPiece through one loader, so there is no derived artifact to build first — it runs on the same file the reference gets.The change is small: a little adapter crate under
engines/splintr, plus the usual wiring in the four places every other engine is wired, one README row, and the pin inscripts/package_size.py. Pinned to=0.19.1at default features, so it measures whatcargo add splintrgives you.On a local run it verified on every cell it attempts — 352/352, ids identical to the reference. That is the same set the reference itself covers; the three models it sits out (
albert,bert-wiki,llama-2) have notokenizer.jsonindata/models, so no json-based engine attempts them either. Worth re-running on your side rather than taking my word for it.A couple of adapter notes:
encode_raw, which returns content tokens without the post-processor template — matchingadd_special_tokens = false— and is the serial per-document path, so the cell stays single-threaded like the other native engines.pipelinedoes it, since splintr doesn't instrument its own stages. Only the BPE backend reports; Unigram and WordPiece split inside the model with no rung to measure.Disclosure: splintr is my own project, so please treat any numbers it produces with the same scepticism you'd apply to any self-submitted engine — the verification gate and the ranking rules here are the repo's, not mine, and I haven't touched them.
Happy to adjust anything, or to drop it if you'd rather keep the engine list where it is.