Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models

This research introduces Marginalize-It and End-Of-Token, two novel methods for efficiently distilling large token-based language models into smaller, more capable byte-level models. By evaluating dense transformers across various compute budgets, the study reveals that while token models perform better with limited resources, byte models achieve a significantly higher performance ceiling as training data increases. The End-Of-Token approach proves particularly effective, as it preserves the teacher's original probability distribution and demonstrates superior data efficiency by matching token-model accuracy with only one-sixth of the training data. These byte-level architectures also provide a five-fold reduction in logit storage costs because they operate on a much smaller vocabulary of roughly 256 values. Scaling laws developed in the paper predict that these distilled byte models will asymptotically outperform prominent open-weight models like Llama 3.2-1B and Gemma 2B. Ultimately, the work suggests that moving beyond traditional tokenization can "break the token ceiling" to create smaller models with greater long-term potential.