Loading...
Spaceplay← →stepRreset
See how embeddings turn tokens into vectors with an interactive visualization. Watch a lookup table map token ids to numbers, compare vectors with cosine similarity, and see how meaning becomes geometry.
Loading visualization...
An embedding is a learned vector of numbers that stands in for a token, so that tokens used in similar contexts end up close together and the model can do arithmetic on meaning instead of on ids.
A tokenizer turns text into integer ids, but an id is only an index: id 1 is not closer to id 2 than to id 500. The embedding layer replaces each id with a row from a matrix E of shape vocabulary size by embedding dimension. That row is the vector the rest of the network works with.
The values in E are parameters, learned by gradient descent alongside every other weight. Training starts them at small random numbers and adjusts the rows that appear in each batch so the model predicts text better. Tokens that show up in similar contexts receive similar gradients over time, which pushes their rows toward the same region of the space.
Because the vectors live in a geometric space, similarity becomes a measurement. Cosine similarity divides the dot product of two vectors by the product of their lengths and returns a value in [-1, 1]: 1 for the same direction, 0 for perpendicular, -1 for opposite. Retrieval, semantic search, and deduplication all rank candidates by this number.
In a transformer, the token embedding is not the whole input. Attention treats its inputs as a set, so the model adds a position signal to each token vector before the first layer. In GPT-2 that is a second learned table indexed by position, the original transformer used fixed sinusoids, and many recent models apply a rotation inside attention instead. Typical embedding widths run from 768 in GPT-2 small to 12288 in the largest GPT-3.
Interview framing: define Embeddings in one sentence, then explain one concrete runtime behavior and one common pitfall with a short code example.
Ids are assigned by the tokenizer, often by merge order or frequency, so numeric closeness between ids says nothing about the tokens. Feeding raw ids into a dense layer forces the model to learn around an arbitrary ordering.
Fix: Always pass ids through an embedding lookup first. The learned rows, not the ids, are the representation.
The raw dot product grows with vector length, so a long vector scores high against everything. Frequent tokens often have larger norms, which biases nearest neighbour results toward them.
Fix: Normalise both vectors or use cosine similarity, which divides by the norms and compares direction only. If you store normalised vectors, the dot product and cosine similarity become the same number.
Attention is permutation equivariant: shuffling the input rows shuffles the outputs the same way. With token embeddings only, the model cannot tell the cat sat from sat the cat.
Fix: Add a position embedding to each token vector, or use a positional scheme such as RoPE inside attention, before the first transformer layer.
Each model learns its own space. The dimension may match, but the axes mean different things, so a vector from one model compared with a vector from another produces a number that is not a similarity at all.
Fix: Embed queries and documents with the same model and version, and re-embed the whole corpus when the model changes.
Common interview prompts with concise model answers.
It is a matrix with one row per vocabulary token and one column per dimension. Given a token id, it returns the row at that index. There is no multiplication involved, it is a lookup, and the rows are trained parameters that the rest of the network updates through backpropagation.
// E has shape [vocabSize, d]
const ids = tokenize("cat dog car"); // [1, 2, 3]
const X = ids.map((id) => E[id]); // shape [3, d]Cosine similarity is the dot product divided by the product of the two norms, so it measures the angle between vectors and ignores their length. It matters when vectors have different magnitudes, which is common in learned embeddings, because the raw dot product would favour long vectors regardless of direction.
const dot = (a, b) => a.reduce((s, x, i) => s + x * b[i], 0);
const norm = (a) => Math.sqrt(dot(a, a));
const cosine = (a, b) => dot(a, b) / (norm(a) * norm(b));
cosine([0.8, 0.5, 0.1], [0.6, 0.7, 0.1]); // 0.95Self attention computes a weighted sum over the set of input vectors, and the weights depend only on the vectors themselves. Permuting the input permutes the output identically, so word order would be invisible. Adding a position dependent vector to each token gives the model a way to tell slots apart.
const X = ids.map((id, i) =>
tokEmb[id].map((v, k) => v + posEmb[i][k])
);They are initialised randomly and updated by gradient descent like any other weight. The gradient of the loss with respect to the input vectors flows back into the rows that were looked up, so each training example nudges only the rows for the tokens it contains. Over many examples, tokens with similar contexts converge toward similar rows.
Each model learns its own coordinate system. Even with equal dimension, a direction that encodes one property in model A encodes something unrelated in model B, so cosine similarity across models is meaningless. A retrieval index has to be rebuilt whenever the embedding model changes.
Continue with these concepts to strengthen your mental model.
const V = 6; // vocabulary sizeconst d = 4; // embedding dimensionconst E = initMatrix(V, d); // learned, V x d const ids = tokenize("cat dog car"); // [1, 2, 3] const X = [];for (const id of ids) { const vec = E[id]; // row lookup, no math X.push(vec);}// X is 3 x 4, the input to the first layerwaiting
waiting
waiting
waiting