Tokens and Embeddings

A transformer is a long chain of additions and multiplications. It has no notion of letters or words, only numbers. So before any learning can happen, every piece of text must become numbers. This first day is that bridge: text goes in, a grid of numbers comes out, and everything later in the course operates on that grid.

The plan has two steps. First, cut the text into tokens and give each token an integer id. Second, look up each id in a big table and replace it with a short vector of numbers called an embedding.

The model works from a fixed vocabulary — a list of every token it recognizes, each pinned to an integer id. Turning text into tokens means matching the text against that list and writing down the ids.

Splitting on whole words is wasteful: the vocabulary would have to store every word form separately, and any unseen word would have no id at all. Instead, modern models use BPE (Byte-Pair Encoding), which learns common sub-word pieces. The word running might become two tokens, run and ning, so a never-before-seen word can still be built from familiar pieces.

For this course we will keep a tiny toy vocabulary of five whole-word tokens so the numbers stay small enough to check by hand:

So the phrase the cat becomes the id list [3, 0].

An id like 1 is just a name; it carries no meaning on its own (id 2 is not "twice" id 1). To give tokens meaning, we store an embedding matrix: a table with one row per vocabulary token. Each row is that token's vector. Our toy model uses 3 dimensions, so the matrix is 5 rows by 3 columns:

An embedding lookup is nothing more than picking a row. To embed token id 1 (dog), read row 1:

embed(1) = row 1 of the matrix = [0.8, 0.9, 0.1]

There is a tidy way to see this "pick a row" as multiplication. Write the id as a one-hot vector — all zeros except a single 1 at the id's position — and multiply it by the matrix. For id 1 the one-hot is [0, 0, 1, 0, 0]. Column by column:

dim 0:  0*0.9 + 0*0.8 + 1*0.8 + 0*0.0 + 0*0.3 = 0.8
dim 1:  0*0.8 + 0*0.9 + 1*0.9 + 0*0.0 + 0*0.1 = 0.9
dim 2:  0*0.1 + 0*0.1 + 1*0.1 + 0*0.0 + 0*0.7 = 0.1

The one-hot times the matrix gives [0.8, 0.9, 0.1] — exactly row 1. Lookup and matrix multiply are the same operation; real models just skip straight to the row.

Each of those three numbers is one axis of a small space. It helps to see one token's vector as an arrow in 3-D space, its length and direction set by the three dimensions.

Here is the payoff. Embeddings are learned so that tokens used in similar ways end up with similar vectors — close together in the space. Look back at the table: cat is [0.9, 0.8, 0.1] and dog is [0.8, 0.9, 0.1]. Those two arrows point in almost the same direction, while car at [0.1, 0.2, 0.9] points elsewhere.

We measure "similar direction" with cosine similarity: the cosine of the angle between two vectors. Vectors pointing the same way score near 1; unrelated ones score near 0. So cat and dog score high, cat and car score low — the geometry encodes meaning. The widget below plots handcrafted word vectors in 2-D so you can pick a word and watch which neighbours are close.

This is why the embedding step matters: it does not just number the tokens, it arranges them so that closeness in the number space means closeness in meaning. Everything the transformer does next builds on that arrangement.

• Text must become numbers before a transformer can touch it.
• Tokenization cuts text into vocabulary tokens; BPE uses reusable sub-word pieces.
• An embedding lookup replaces a token id with its row in the embedding matrix.
• Row-lookup equals multiplying a one-hot vector by the matrix.
• Embeddings place similar tokens near each other; cosine similarity measures it.

• [ ] Can you turn a short phrase into a list of token ids using the toy vocabulary?
• [ ] Can you read the embedding of a token id straight off the matrix as a row?
• [ ] Can you explain why row-lookup is the same as a one-hot matrix multiply?
• [ ] Can you say what a high cosine similarity between two word vectors means?