How Tokenization Works: Words, Subwords and Bytes in NLP

Methods & evaluation · Published 2026-10-26

Close-up black-and-white photograph of round typewriter keys arranged in staggered rows — Typewriter keys, photographed by Ralph Steiner for the Clarence H. White School of Photography. Library of Congress, public domain.

Before a language model reads a single sentence, a much simpler program has already decided what the units of that sentence are. Tokenization is rarely discussed outside technical papers, yet it shapes vocabulary size, model cost, multilingual fairness and even evaluation scores. This guide explains how it works and what to check.

Why not just use words?

The intuitive unit of text is the word, and early statistical NLP systems did split text on spaces and punctuation. Two problems appear quickly. First, vocabularies explode: every inflected form, name, typo, number and hashtag becomes a separate entry, and a model has to store a vector for each. Second, any word not seen during training becomes "unknown", which is fatal for translation and search, where names and new terms matter most.

Spaces are also a poor guide in many languages. Chinese and Japanese are normally written without spaces between words; Thai does not separate words either. Agglutinative languages such as Turkish or Finnish build long words from many meaningful parts, so a single "word" may carry what English expresses in a whole phrase. A word-level vocabulary that is reasonable for English becomes unmanageable for these languages.

The opposite extreme, reading text character by character, solves the unknown-word problem but makes sequences much longer, which increases computation and makes it harder for a model to relate distant parts of a sentence. Subword tokenization is the compromise most systems settled on.

Subword tokenization in one example

A subword tokenizer keeps frequent strings intact and breaks rare ones into pieces it already knows. A trained tokenizer might, for instance, keep "the" and "language" as single tokens but split a rarer word such as "untranslatability" into something like:

untranslatability

The exact split depends entirely on the corpus the tokenizer was trained on, which is why two models can segment the same word differently. The pieces are not designed to be linguistic morphemes; they are statistical units that happen to be frequent. Sometimes they coincide with prefixes and suffixes, often they do not.

The main algorithms

Byte pair encoding (BPE)

Byte pair encoding was first described as a data compression method by Philip Gage in 1994. Rico Sennrich, Barry Haddow and Alexandra Birch adapted it to neural machine translation in a 2016 ACL paper, "Neural Machine Translation of Rare Words with Subword Units", showing that translating subword units lets a system handle rare words and names it never saw whole. Training starts from individual characters and repeatedly merges the most frequent adjacent pair into a new symbol until the vocabulary reaches a chosen size. At inference time the learned merges are applied in the same order.

A widely used variant works on bytes rather than Unicode characters: OpenAI's GPT-2 model (2019) used byte-level BPE so that any string, in any script, can be encoded without an "unknown" token, because every text is ultimately a sequence of bytes.

WordPiece

WordPiece was introduced by Mike Schuster and Kaisuke Nakajima at Google in 2012 for Japanese and Korean voice search, and became widely known through BERT (Devlin and colleagues, NAACL 2019). It also builds a vocabulary by merging units, but chooses merges according to how much they improve the likelihood of the training data rather than by raw frequency. In BERT's English vocabulary, pieces that continue a word are marked with a "##" prefix.

Unigram language model

Taku Kudo's unigram method (ACL 2018) works the other way round: it starts from a large set of candidate pieces and prunes those that contribute least to a probabilistic model of the corpus. Because it keeps probabilities for alternative segmentations, it can also sample different splits of the same word during training, a technique Kudo called subword regularisation, which can make models more robust.

SentencePiece

SentencePiece, described by Taku Kudo and John Richardson in an EMNLP 2018 system demonstration, is a library rather than a new algorithm: it implements both BPE and the unigram model. Its practical innovation is to treat the input as a raw stream, including spaces, which are replaced by a visible marker symbol. The tokenizer therefore needs no language-specific pre-splitting, and the original text can be restored exactly from the tokens. This is why it is popular for multilingual models.

Vocabulary size is a design choice

Every tokenizer is trained to a fixed vocabulary size, chosen by the developers. A small vocabulary produces longer token sequences; a large one shortens sequences but adds rows to the model's embedding table and spreads training data thinner across more entries. Published models use very different settings, and there is no universally best value. When you compare two models, remember that "tokens" are not a common unit: a context window of a given number of tokens holds more or less text depending on the tokenizer and the language.

Tokenization is not neutral across languages

Because the vocabulary is learned from a training corpus, languages that are well represented in that corpus get long, efficient tokens, while others are broken into many short fragments. Several studies have measured this. Phillip Rust and colleagues, in "How Good is Your Tokenizer?" (ACL 2021), compared multilingual models with monolingual ones and found that the quality of the tokenizer for a given language was an important part of the performance gap; replacing a multilingual tokenizer with a dedicated monolingual one improved results in many of their experiments. Aleksandar Petrov and colleagues (NeurIPS 2023) measured how many tokens commercial and open tokenizers need for the same parallel text in different languages and reported very large differences, with some languages needing many times more tokens than English for the same content.

The consequences are practical. Longer sequences mean that less text fits in a model's context, that processing is slower, and, where a service charges by the token, that the same message costs more in some languages than in others. Fragmented tokens also give a model less meaningful units to learn from, which is one of the reasons discussed in our article on low-resource languages.

Beyond subwords: characters and bytes

Some research models drop learned vocabularies altogether. CANINE (Clark and colleagues, 2022) operates directly on Unicode characters, and ByT5 (Xue and colleagues, 2022) on raw UTF-8 bytes; both were published in the Transactions of the Association for Computational Linguistics. These "token-free" models avoid the language imbalance built into a learned vocabulary and are more robust to spelling noise, but they process longer sequences and need architectural adjustments to stay efficient. Whether they will replace subword tokenizers is an open research question.

Practical checklist

Use the model's own tokenizer. A model only understands the token numbers it was trained with. Loading a different tokenizer, even one with the same vocabulary size, silently produces nonsense.
Normalise text consistently. Unicode offers several ways to encode the same visible character (for instance an accented letter as one code point or as a letter plus a combining accent). Apply the same normalisation form at training and inference time.
Inspect a sample. Tokenize a few sentences from your own data, including names, numbers and any languages or scripts you care about, and look at the pieces. Excessive fragmentation is an early warning sign.
Do not compare token counts across models. Context lengths, perplexity and per-token prices are only comparable between systems that share a tokenizer.
Keep evaluation tokenization fixed. Some metrics, notably BLEU in machine translation, are sensitive to how references and outputs are tokenized; see our guide to machine translation evaluation and our wider article on how NLP models are evaluated.
Document it. When you publish, state the tokenizer, its library version, vocabulary size and the corpus it was trained on.

Sources

Sennrich, Haddow and Birch, "Neural Machine Translation of Rare Words with Subword Units", ACL 2016 (ACL Anthology P16-1162).
Kudo and Richardson, "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing", EMNLP 2018 demonstrations (ACL Anthology D18-2012).
Kudo, "Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates", ACL 2018.
Schuster and Nakajima, "Japanese and Korean Voice Search", ICASSP 2012.
Rust, Pfeiffer, Vulić, Ruder and Gurevych, "How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models", ACL 2021.
Petrov, La Malfa, Torr and Bibi, "Language Model Tokenizers Introduce Unfairness Between Languages", NeurIPS 2023.
Clark, Garrette, Turc and Wieting, "CANINE", TACL 2022; Xue and colleagues, "ByT5", TACL 2022.