Loading...
Spaceplay← →stepRreset
See how a language model turns text into tokens with an interactive visualization. Watch byte pair encoding merge characters into subwords, map tokens to ids, and see why token counts differ from word counts.
Loading visualization...
Tokenization is the step that turns raw text into the sequence of integer ids a language model actually reads. Modern models use subword schemes such as byte pair encoding, so a token is usually a word fragment rather than a whole word or a single character.
A language model cannot consume strings. Its first layer is an embedding table indexed by integers, so every input has to be converted into a list of ids drawn from a fixed vocabulary. The tokenizer is the deterministic function that performs that conversion, and its inverse, decoding, turns generated ids back into text.
Splitting on whitespace would make the vocabulary unbounded and leave every misspelling or new name out of vocabulary. Splitting into characters keeps the vocabulary tiny but makes sequences very long, and each position then carries almost no meaning. Subword tokenization sits between the two: frequent strings such as common words become one token, rare strings are broken into pieces the model has seen before.
Byte pair encoding learns those pieces from data. Training starts from base symbols, counts every adjacent pair across the corpus, merges the most frequent one into a new symbol, and repeats until the vocabulary reaches a target size. The ordered list of merges is the tokenizer. Encoding replays the same merges on new text, so training and inference always segment a string the same way.
Byte-level variants, used by GPT-2 and most models after it, take UTF-8 bytes rather than characters as the base symbols. That guarantees every possible input can be encoded with 256 base tokens and no unknown-token placeholder, at the cost that scripts which were rare in the training corpus, or which need several bytes per character, split into many more tokens than English does.
Interview framing: define Tokenization in one sentence, then explain one concrete runtime behavior and one common pitfall with a short code example.
Context windows and API pricing are measured in tokens. English prose averages roughly four characters per token, but code, numbers, unusual names, and non-Latin scripts can run at one to three tokens per word.
Fix: Count tokens with the model's own tokenizer before truncating or budgeting. Never assume a fixed words-to-tokens ratio across languages or content types.
In GPT-style vocabularies the space belongs to the following word, so 'hello' and ' hello' are different ids with different embeddings. Prompts that end with a trailing space or start a line without one can shift which token the model sees.
Fix: Keep the boundary consistent with training data: end prompts without a trailing space and let the model produce the space-prefixed token itself.
Ids only mean something relative to the vocabulary they came from. Feeding ids from one tokenizer into another model, or comparing token counts across model families, silently produces nonsense.
Fix: Load the tokenizer that ships with the exact model checkpoint and treat token counts as model-specific numbers.
A model sees 'strawberry' as one or two ids, not as a run of letters, so questions about letter counts or reversing strings work only when the model has memorised the spelling.
Fix: Break the string into visible characters in the prompt, or hand the task to code. Do not rely on the model to see inside tokens.
Common interview prompts with concise model answers.
Whole-word vocabularies are unbounded and any unseen word becomes an unknown token. Character vocabularies are tiny but produce very long sequences with little meaning per position. Subword tokenization keeps the vocabulary fixed while still encoding any string, and it lets common words be one token while rare words fall back to shared fragments such as prefixes and suffixes.
Start with each word split into base symbols and a frequency count per word. Count all adjacent symbol pairs weighted by frequency, merge the most frequent pair into a new symbol everywhere it occurs, add that symbol to the vocabulary, and record the merge. Repeat until the vocabulary reaches the target size. The ordered merge list is what encoding replays later.
const merges = [];
while (merges.length < NUM_MERGES) {
const pairs = countPairs(splits, freq); // { "e s": 9, "s t": 9, ... }
const best = argmax(pairs); // ["e", "s"]
if (!best) break;
merges.push(best);
vocab.add(best[0] + best[1]); // "es"
for (const [word, syms] of splits) {
splits.set(word, applyMerge(syms, best));
}
}Each pre-tokenized piece is split into base symbols, then the encoder repeatedly merges the adjacent pair with the lowest rank in the merge list until no pair has a rule. Applying merges in rank order reproduces the segmentation seen during training. The surviving symbols are mapped to ids through the vocabulary.
function encodePiece(piece) {
let syms = [...piece];
while (syms.length > 1) {
const pair = lowestRankPair(syms, ranks);
if (!pair) break;
syms = applyMerge(syms, pair);
}
return syms.map((s) => vocab.get(s));
}The tokenizer works on bytes and has no notion of case or word boundaries beyond what the pre-tokenizer regex encodes. A leading space is part of the byte sequence, so ' hello' is a different string with its own merge path and id. The same holds for a different first byte in 'Hello'. Each variant gets its own embedding and the model learns their relationship only from co-occurrence.
In a byte-level tokenizer the 256 possible byte values are always in the vocabulary, so any UTF-8 input can be encoded without an unknown token. When no merge rules were learned for a script, every byte becomes its own token. A character that needs three bytes in UTF-8 then costs three tokens, which is why the same sentence can be several times more expensive in Georgian or Thai than in English.
Continue with these concepts to strengthen your mental model.
const vocab = new Set(words.flatMap((w) => [...w]));const splits = new Map(words.map((w) => [w, [...w]]));const merges = [];while (merges.length < NUM_MERGES) { const pairs = countPairs(splits, freq); const best = argmax(pairs); if (!best) break; merges.push(best); vocab.add(best[0] + best[1]); for (const [word, syms] of splits) { splits.set(word, applyMerge(syms, best)); }}waiting
waiting
waiting
waiting