Do we really need to break text into tokens, or could we work directly with raw bytes?
First, let’s think about how do LLMs currently handle text. They first chop it up into chunks called tokens using rules about common word pieces. This tokenization step has always been a bit of an odd one out. While the rest of the model learns and adapts during training, tokenization stays fixed, based on those initial rules. This can cause problems, especially for languages that aren’t well-represented in the training data or when handling unusual text formats.
AIModels.fyi is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
