[Question] Tokenizer is not counted in submission size

[DouglasOrr]

The submission size calculation doesn't count persisting the tokenizer itself - is this right? Two challenges to this: first, the size includes everything necessary (modulo generic Python requirements) to fully specify and inference the model, so it seems a shame to lose this property by omitting the tokenizer. Second, not counting this allows large-vocabulary models an artificial advantage, since they get the strings "for free".

(However, I appreciate this may be a pragmatic decision, as although the fineweb_1024_bpe.vocab file is small, fineweb_1024_bpe.model is large, e.g. 150 kB compressed & is presumably larger than it needs to be. I presume the model isn't uniquely reconstructable from the vocab(?))

[kyleliang919]

Not sure about your first point but to your second point you would end up paying for the large vocab size with large embedding table so it's not free advantage.

[DouglasOrr]

True. I can certainly think of ways that could make use of an arbitrarily large set of strings via this side-channel to gain an unfair advantage, but these would all hopefully fail against "Submissions that edit the tokenizer will be examined much more carefully, since bugs may unjustly improve your score."

[0hq]

This is a good point. I can imagine some ridiculous N-gram tokenizers perhaps that let you sneak in lots of bits into training. For now, let's just assume things like this are 'not in the spirit of the competition' and can be evaluated on a case-by-case basis.

[pszemraj]

hey - was thinking about this 'independently' and in the process of checking/investigating found this thread. Am I understanding this correctly that tokenizer files/code "don't count", putting training an optimized 1024 vocab (or more, if embeddings size cost is 'worth') tokenmonster optimized jointly across the train set -> eval set text (both known) is tenable?? You could probably even have a whole optimization branch of back calculating what the merges and so on should have been given the model size and other constraints...

imo that is both:

1. against the spirit of the competition (unless you really want people to optimize vocabs and/or you are fairly certain that yours is close to optimal already)
2. nonsensical in terms of "16mb artifact contains the things needed for my model" because.. any tokenizers other than the default (or some spicy charLM/ ord() etc) are clearly core files needed to use it

I think that having any tokenizer files/data other than the default count towards the total 16mb size is one way to mitigate/discourage this

[sharpobject]

Hello @0hq, I would appreciate clarification about what sorts of tokenizers are legal.

I think it's like this: first, a contestant may use any text encoding. a "text encoding" is defined to be a scheme by which a large alphabet of symbols can be represented using a smaller one, such that each sequence of symbols from the large alphabet can be represented as a distinct sequence of symbols from the smaller one. then, the contestant may use any lossless text preprocessing procedure, where a lossless text preprocessing procedure is any algorithm that computes an injective function from strings to strings (and for practical reasons, an algorithm for computing the inverse should also be provided). then, the contestant may construct by any means a set of strings from the small alphabet used by their chosen text encoding, and use any algorithm to convert their encoded, preprocessed documents to a sequence of token ids, where the concatenation of the strings represented by the token ids must be equal to the encoded, preprocessed document.

that is to say, if you begin with the token ids for a document, you should be able to decode the tokens and concatenate the results of decoding them, then apply the inverse of whatever text preprocessing was used, then transcode the text to utf-8, in order to recover the original utf-8 bytes, and this process must work for any document that is valid unicode.

1 special token must be present in the vocabulary, which is used as a document delimiter.

regardless of the chosen text encoding and preprocessing scheme, "bytes" in "bits per byte" is the length in bytes of the original utf-8 text. for the validation set, this happens to be 151,080,363 bytes.

is that reasonable?

thanks.

aside: note that the default sentencepiece tokenizer does not actually meet these standards for arbitrary documents because it uses various forms of lossy text normalization, but because these are idempotent, it happens to meet these standards for the particular documents in the provided tokenized data.