Glossary
Tokenization
Tokenization is the step where text is cut into tokens, the small pieces an AI model actually reads.
What it means in practice
It explains why a model can stumble on spelling, rare names or counting letters, and why different languages can use different amounts of space. Check names and numbers it writes.
Example
A long unusual name being split into several pieces before the model reads it.