Migrate deepcut tokenizer from TensorFlow to built-in ONNX inference#1372
Conversation
|
@copilot Try again |
- Bundle deepcut.onnx model (from LEKCut) in pythainlp/corpus/ - Add deepcut_onnx entry to default_db.json - Rewrite pythainlp/tokenize/deepcut.py with ONNX inference; no longer depends on the TF-based deepcut package - Move deepcut tests from noauto_tensorflow/ to noauto_onnx/ - Remove deepcut from pyproject.toml TF deps and mypy overrides - Update CHANGELOG.md Agent-Logs-Url: https://github.com/PyThaiNLP/pythainlp/sessions/ff10e57e-b105-4263-9f34-12b2e67cd4ab Co-authored-by: wannaphong <8536487+wannaphong@users.noreply.github.com>
Done in e6ca3fa. Here's what was implemented:
The |
|
|
Alternatively, we can call the DeepcutTokenizer in nlpo3 |
Yes but I think it should call from python because we already used onnxruntime in pythainlp. |
|
This PR is stale because it has been open for 30 days with no activity. |
Removed custom_dict parameter from segment function for API compatibility.
Removed tests for sefr_cut, deepcut, and oskut tokenizers from the test suite.
Renamed DetokenizeDeepcutTestCaseN to tokenizeDeepcutTestCaseN and removed unused imports.
|
This pull request was drop support |
Removed unused import of 'Union' from typing.
Removed unused imports from the test file.
Simplify deepcut segment handling by removing custom_dict checks.
Removed unused import of 'deepcut' from the test file.
|
Coverage is 66.73% — copilot/remove-lekcut-insert-deepcut-onnx into dev. No base build found for |
|
What do these changes do
Replaces the external TensorFlow-based
deepcutpackage dependency with a built-in ONNX inference engine. The DeepCut ONNX model (ported from LEKCut) is now bundled directly with PyThaiNLP, removing the need for TensorFlow and reducing dependencies.What was wrong
The
pythainlp.tokenize.deepcutmodule depended on the externaldeepcutpackage which requires TensorFlow (~1–2 GB), making it a heavy optional dependency. The project already usesonnxruntimefor other models (e.g.,thai2rom,sefr_cut), so the TensorFlow dependency was unnecessary.How this fixes it
pythainlp/corpus/deepcut.onnx(new): Bundles the DeepCut ONNX model (2.1 MB) ported from LEKCut directly in the package corpus directory.pythainlp/corpus/default_db.json: Adds adeepcut_onnxentry soget_corpus_path()resolves the bundled model file.pythainlp/tokenize/deepcut.py: Completely rewritten with direct ONNX inference usingonnxruntimeandnumpy. Implements character/type feature encoding from the original DeepCut model. Thesegment()API is unchanged;custom_dictis kept for backward compatibility.pyproject.toml: Removesdeepcut(TF package) fromnoauto-tensorflowdeps,fulldeps, and mypy overrides.tests/noauto_tensorflow/totests/noauto_onnx/, reflecting the new ONNX-only dependency.CHANGELOG.md: Change documented under[Unreleased].Your checklist for this pull request
📱 Kick off Copilot coding agent tasks wherever you are with GitHub Mobile, available on iOS and Android.