Skip to content

Added new memory efficient conversion script for hf to ggml format, tested on bloomz 176B + Added token conversion script to convert from tokenizer.json format to tokenizer.model#867

Closed
akumaburn wants to merge 4 commits into
ggml-org:masterfrom
akumaburn:feature/tokenconvert

Add sentencepiece processor

9020757
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs