From Tokens to Bytes: AI Learns Its Own Building Blocks

Most language models begin with a tokenizer that cuts text into predetermined pieces. What if the model could learn those pieces for itself? Cartesia’s hierarchical networks, or H-Nets, start with raw bytes and build useful chunks inside the model. We explore how this learned hierarchy can compress familiar patterns, preserve important detail, and even change its segmentation as new words and repeated structures appear within a passage. Cartesia reports promising efficiency gains in controlled comparisons with tokenized Transformers, including a larger experiment against SmolLM3. We unpack why those results depend on the particular models, training mixtures, and measures being compared, while following the central idea: learning how to represent information can matter as much as adding more computation. The research also reaches into DNA sequences and image generation, offering an intriguing glimpse of systems that discover useful structure directly from their inputs. Note: This podcast was AI-generated, and sometimes AI can make mistakes. Please double-check any critical information. Sponsored by Embersilk LLC