size of package

[weejulius]

tiktoken/lib/src/core_bpe_constructor.dart

Line 112 in f2538e7

"gpt2": CoreBPEConstructor.gpt2,

firstly thanks for your tiktoken library, and there might be a consideration regarding the const variable "all". It will refer all the ranks, so that the treeshaking does not work well in case not all of the models are used.

The diff :
All model: 21.7MB
Only cl100k_base: 20.6MB

The number might not be too much(~1M) however it might be a step of improvement.

[mvitlov]

Yup, you're right.

I did a little refactoring, and tested with small script example.dart containing:

import 'package:tiktoken/tiktoken.dart';

void main(List<String> args) {
  final encoding = getEncoding('cl100k_base');
  print(encoding.encode("hello world")); // [15339, 1917]
}

When run $ dart compile aot-snapshot example\example.dart , these are the results:

// old (all models):
example.aot : ~10.00 MB ❌  

// refactored (account for tree-shaking)
example.aot: ~7.18 MB ✅ 

So, I'd say it a win! 💪