Bump sentencepiece from 0.2.1 to 0.2.2

[dependabot]

Bumps sentencepiece from 0.2.1 to 0.2.2.

Release notes

Sourced from sentencepiece's releases.

v0.2.2

Release Notes v0.2.2

This release contains major updates, including the migration of the Python wrapper to pybind11, native NumPy support, enhanced offset mapping, performance optimizations, security hardening, and internal code modernization.

1. Python Wrapper (pybind11 Migration & Features)

• SWIG to pybind11 Migration: Fully migrated the Python wrapper from SWIG to pybind11, enabling better performance, safer memory management, and cleaner code integration (#1266, Commit 528dc9e).
• Native NumPy Support:
  • direct encode/decoder from/to numpy object
• Enhanced Offset Mapping:
  • Added new offset mapping features supporting both Unicode character offsets and raw byte offsets.
  • Added bounds checking to prevent OOB reads in offset mapping (Commit c5b0ce0).
  • Fixed crash issues and OOB spans in alignment conversions (#1231, #1211).
• Type Stubs (PEP 561): Added inline type stubs (.pyi files) and packaged PEP 561 marker to provide auto-completion and static type safety in IDEs (#1258, Commit 8b7f97b).
• Parallel Encoding API: Introduced parallel_encode and ThreadPool APIs, allowing Python users to tokenize single large documents across multiple CPU cores in parallel (#1254, Commit 3c37c39).
• Direct Proto Loading: Added support for loading processors and normalizers directly from in-memory Proto instances (Commit 12788ba).
• Concurrency Enhancements: Added GIL release during C++ execution to improve multi-threaded Python performance (#1251, Commit 0a044ae).
• BOS/EOS Behavior Hardening: Aligned BOS/EOS adding behavior with C++ constraints, raising explicit ValueError instead of silent failures or crashes (Commit 780618a).

2. Performance Optimizations

• BPE Training Optimization: Ported O(log n) lazy priority queue update optimization to BPE trainer to speed up training (Commit 8db9878).
• Encode/Decode Optimizations: Decoding/Encoding optimization for integer/string input case by skipping SentencePieceText conversion, resulting in ~2x speedup (Commit 4b9231b).
• Unigram Pruning: Improved unigram pruning using token-normalized frequency calculations (#1135, Commit 1cd0c59).

3. Security & Hardening

• Trie Validation at Load: Added verification of darts-clone trie offsets when loading models to prevent segfaults on corrupted or malicious model files (#1224, Commit 6b2f05d).
• Bounds Checking in Maps: Added bounds checks to piece accessors and validated value offsets in DecompileCharsMap (#1265, Commit 82b7c6d).
• Crash Prevention: Replaced internal CHECK macros with DCHECK or returned errors to prevent host application crashes on invalid inputs (Commit 66b2587).
• Null Character Disallowment: Explicitly disallow vocabulary pieces containing null characters (Commit eb3ba49).
• NBest Timeout: Implemented a global timeout for NBest A* search to prevent hangs on pathological inputs (Commit ba57749).

4. C++ API & Internals Modernization

• Abseil Migration: Extensively migrated from custom utilities to Abseil primitives (#1270
• Normalizer Maps: Supported loading normalizer from raw maps in C++ (#1273, Commit e476e5e).

5. Build & Packaging

• Deprecate Win32 Wheels: Stopped support and distribution of 32-bit Windows wheels (win32) (Commit 24fd00c).
• Modernized Python builds: Switched from calling setup.py directly to the standard pypa/build PEP 517 frontend (#1167, Commit f300c7d).
• PEP 639 License: Added PEP 639 license metadata to pyproject.toml (#1173).

6. Documentation

• API Cheat Sheet: Added comparison tables, zero-copy, and sampling comparison guides.
• Refactored Options: Fully reorganized options.md and special_symbols.md with verified code examples.

Commits

• e0cce7d Merge pull request #1279 from google/dependabot/github_actions/github-actions...
• a9e2b5e Bump the github-actions group with 2 updates
• 20ab5fb Fix Unigram float score overflow and NaN/Inf corruption
• 1f1483c Add use_reserved_id_map parameter and control token merge tests
• 69d8562 Add performance benchmark suite and reports
• 24fd00c Stop support for Win32 wheels
• 4b9231b Optimize Decode/Encode paths using absl::Span and add benchmarks
• b91b764 Merge pull request #1277 from google/new_doc
• e93413b Update documentation: Modernize README and merge CLI/Build guides
• 7ce7669 Merge pull request #1276 from kkew3/fix-dockerfile-20260620
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud