Was My 2,103-Token Tokenizer a Mistake?

Automated excerpt

Scope note: 12 LLaMA-style decoders trained from scratch on TypeScript/JavaScript, 3 sizes × 4 vocabulary sizes, equal training FLOPs. Vocabulary beats backbone: the 10M tag with 8k vocab (8. 0M total params, 1. 168 bpb) beats the 50M tag with 2k vocab (33. 5M total params, 1. 436 bpb): 4. 2× fewer total parameters, 18. 7% lower bits-per-byte. Directionally yes — 2k costs ~19% bpb vs 8k at 10M.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.